# Local AI Development in .NET with Ollama and GitHub Models: A Setup Checklist

Getting a model running on your laptop is the easy part. Two commands and a NuGet package and you have a chat endpoint answering questions with no API key and no bill. The hard part, and the part that quietly costs teams weeks, is keeping local AI development in .NET honest: making sure the thing you built against a small local model still behaves when it meets the model you actually ship. In production I've seen a feature that worked beautifully against a local 8B model fall apart on the hosted model it was deployed with, because the prompt had been tuned to compensate for weaknesses the production model did not have.

This checklist is the setup I would give a team starting today. It assumes you want a free local loop for the fast inner cycle and a real model in the path before anything merges. The complete dev-environment configuration, including the Compose file and the provider-switching tests, is on [Patreon](https://www.patreon.com/CodingDroplets).

Getting a free environment working is a chapter in itself, and [Chapter 2 of AI-Powered .NET APIs](https://aiapis.codingdroplets.com/) walks through installing Ollama, pulling a small model, wiring up the GitHub Models free tier, and running the same C# against either provider - plus the setup failures that catch everyone the first time.

[![AI-Powered .NET APIs](https://newsletter.codingdroplets.com/images/ai-api-course-banner-1.jpg align="center")](https://aiapis.codingdroplets.com/)

## 1\. Choose the Model by Your Machine's Memory, Not the Leaderboard

Model size is bounded by RAM or VRAM, not by ambition. A quantised model in the 7B to 8B range runs comfortably on a typical developer laptop; a 70B model does not, and attempting it produces either an out-of-memory failure or swap-driven inference so slow it is unusable. Pick the largest quantised model that leaves headroom for the rest of your machine, then stop optimising - local model quality is not what you are testing.

## 2\. Put Every Provider Behind IChatClient

This is the item everything else depends on. Register the local client through the same abstraction as the hosted one so switching providers is a registration change:

```csharp
// OllamaSharp's client implements IChatClient directly
builder.Services.AddChatClient(
    new OllamaApiClient(new Uri("http://localhost:11434"), "llama3.2:3b"));
```

Because [`Microsoft.Extensions.AI`](https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai) normalises the surface, your endpoints, tools, and tests never learn which provider is behind them. [OllamaSharp](https://github.com/awaescher/OllamaSharp) is the .NET client that provides the implementation. Our guide to [IChatClient in enterprise ASP.NET Core APIs](https://codingdroplets.com/microsoft-extensions-ai-ichatclient-aspnet-core-enterprise-2026) covers the abstraction in more depth.

## 3\. Drive the Switch From Configuration, Never From #if DEBUG

A compile-time switch means the local path and the production path are different builds, and the one you tested is not the one you ship. Read the provider from configuration, default it to local in `appsettings.Development.json`, and make sure a developer can point their local build at the hosted model by changing one setting. That single capability is what lets you diagnose "works locally, fails in staging" in minutes.

## 4\. Pin the Model Tag Explicitly

`llama3.2` is a moving target; `llama3.2:3b` is not. An unpinned tag means a teammate who pulls the model a month later gets different weights and different behaviour, and nothing in your repository records the difference. Pin the exact tag in configuration and treat a change to it like a dependency upgrade, because that is what it is.

## 5\. Add GitHub Models for the Frontier-Model Path

Ollama gives you a fast offline loop. GitHub Models gives you access to hosted frontier models against a free tier using a GitHub personal access token, through an OpenAI-compatible endpoint, which means the same `Microsoft.Extensions.AI` registration pattern works. Use it for the checks that a small local model cannot answer honestly - instruction following, structured output reliability, tool selection. Respect the rate limits; the free tier is for development, not for a load test.

## 6\. Verify Feature Support Before You Design Around It

Structured outputs and tool calling are not universal across local models. Some support constrained JSON output natively, some approximate it, and some ignore the schema entirely and return prose. Confirm what your chosen local model actually does before building a feature that assumes it, and never conclude from a local failure that the feature does not work - it may simply be the model. Our comparison of [structured outputs versus tool calling in .NET](https://codingdroplets.com/structured-outputs-vs-tool-calling-dotnet) covers which mechanism to reach for.

## 7\. Never Tune Prompts Against the Local Model Alone

Small models need more hand-holding: more explicit instructions, more examples, more repetition of the output format. A prompt tuned to compensate for that is over-specified for a stronger model and can actively degrade its output. Write prompts against the production model, and treat the local model as a smoke test for plumbing rather than a proxy for quality.

## 8\. Watch the Context Window Difference

Local models frequently expose a smaller effective context than the hosted model you deploy with, and Ollama applies its own default context length that may be lower than the model's maximum. A RAG pipeline that fits comfortably in production will silently truncate locally, producing answers that look like a retrieval bug. Log the token count you are sending and the model's configured limit so truncation is visible rather than inferred.

## 9\. Keep Ollama Out of CI

Continuous integration agents have no GPU and no patience. Running a real model in CI makes builds slow, flaky, and non-deterministic, and it tests the model rather than your code. Use a fake `IChatClient` returning canned responses for unit and integration tests, and run genuine model calls only in a scheduled evaluation job against the production provider. This is exactly the split that makes the `IChatClient` abstraction worth having.

## 10\. Standardise the Runtime With Compose, and Mount a Volume

Running Ollama as a Compose service alongside your API gives every developer the same endpoint and the same setup steps. Mount a named volume for the model store: weights are measured in gigabytes, and re-downloading them on every container rebuild wastes an afternoon per developer. Remember that from inside another container the host is not `localhost`, which is the single most common reason a containerised API cannot reach a containerised Ollama.

## Bonus: Know When Local Models Are the Production Answer

Local models are usually the wrong answer for production cost, because self-hosting inference at scale is rarely cheaper than per-token pricing once you account for hardware and operations. They are the right answer when data cannot leave your perimeter - regulated data, strict residency requirements, PII you have decided not to send to a third party. That is a compliance decision, not a cost optimisation, and it is worth separating the two in any discussion. If cost is the actual driver, our post on [runaway LLM costs in a .NET API](https://codingdroplets.com/runaway-llm-costs-dotnet-api) covers the levers that work better.

## Summary Checklist

*   \[ \] Model size chosen against available memory, quantised, with headroom
    
*   \[ \] All providers registered behind `IChatClient`
    
*   \[ \] Provider selected by configuration, not compilation
    
*   \[ \] Model tag pinned explicitly and treated as a dependency
    
*   \[ \] GitHub Models wired up for frontier-model checks
    
*   \[ \] Structured output and tool support verified per model
    
*   \[ \] Prompts authored and tuned against the production model
    
*   \[ \] Token counts and context limits logged
    
*   \[ \] CI runs against a fake client, never a real model
    
*   \[ \] Compose service with a mounted volume for model weights
    

## FAQ

### Can I use Ollama with Microsoft.Extensions.AI in .NET?

Yes. OllamaSharp's client implements `IChatClient` directly, so it registers exactly like any other provider and everything downstream is unchanged. That is the entire point of the abstraction: the same endpoint code, tools, and tests run against a local model or a hosted one depending only on which client you registered.

### Is a local model good enough to develop an AI feature against?

Good enough for plumbing, not for quality. A local model will tell you whether your DI wiring, streaming, serialisation, and error handling work, which is most of what you iterate on day to day. It will not tell you whether your prompt is good, whether tool selection is reliable, or whether structured output holds, because small models behave differently on all three.

### What model size can I run on a typical developer laptop?

A quantised model in the 7B to 8B parameter range is the usual comfortable ceiling on a machine with 16 GB of memory, and smaller 3B models run well on less. The constraint is memory rather than raw compute: exceed it and the system swaps, and inference becomes slow enough to break your development loop entirely.

### Should I run real model calls in my CI pipeline?

No. Build agents lack GPUs, model calls are non-deterministic, and a failing test then tells you the model changed rather than that your code broke. Use a fake `IChatClient` for tests that run on every commit, and schedule a separate evaluation job that exercises the real production provider against a labelled dataset.

### Why does my containerised API fail to reach Ollama running in Docker?

Because `localhost` inside a container refers to that container, not the host or a sibling. On a Compose network, use the service name as the hostname. If Ollama runs on the host while the API runs in a container, use the host gateway address your platform provides rather than `localhost`.

### Do local models support structured outputs and tool calling?

It varies by model and version, so verify rather than assume. Some local models handle constrained JSON output and tool selection well, others return prose where you expected a schema. Test the specific model and tag you have pinned, and never conclude that a feature is broken from a local failure alone until you have confirmed the same behaviour against your production model.

* * *

## About the Author

I'm Celin Daniel, Co-founder of [Coding Droplets](https://codingdroplets.com/). I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

*   GitHub: [codingdroplets](http://github.com/codingdroplets/)
    
*   YouTube: [Coding Droplets](https://www.youtube.com/@CodingDroplets)
    
*   Website: [codingdroplets.com](https://codingdroplets.com/)