# How to Evaluate LLM Output in .NET APIs: A Real-World Walkthrough

The demo went perfectly. Your ASP.NET Core support-desk endpoint took a customer question, called an LLM, and returned a clean, helpful answer. Everyone nodded. Two weeks later someone tweaked the system prompt to "Make it Friendlier", and the model quietly started inventing refund policies that do not exist. No exception was thrown. No test went red. The endpoint returned `200 OK` the entire time. This is the core reason you need to evaluate LLM output in .NET as a first-class engineering step, not a manual eyeball check before every deploy.

Traditional testing assumes a deterministic answer: given input X, assert output equals Y. Language models break that assumption. The same prompt can produce different wording every call, and "correct" is a spectrum rather than a string match. In production I have seen teams ship AI features with zero automated quality signal, then discover a regression only when a customer complains. The patterns in this walkthrough go deeper on our [Patreon](https://www.patreon.com/CodingDroplets), where the full evaluation harness - wired into a real support API with edge cases and a passing test suite - is ready to run and adapt.

We will build this using the `Microsoft.Extensions.AI.Evaluation` libraries, scoring responses for relevance and groundedness, then wiring the whole thing into CI so a quality drop fails the build instead of your customers. Getting evaluation right means thinking about response caching, cost, and non-determinism at the same time, and that is exactly what [Chapter 17 of the AI-Powered .NET APIs course](https://aiapis.codingdroplets.com/) covers against one running codebase you can execute end to end.

[![AI-Powered .NET APIs](https://newsletter.codingdroplets.com/images/ai-api-course-banner-1.jpg align="center")](https://aiapis.codingdroplets.com/)

## What Does It Mean to Evaluate LLM Output in .NET?

Evaluating LLM output means scoring a model response against measurable quality criteria - relevance, groundedness, coherence, completeness - instead of checking it for an exact string. Each criterion produces a numeric score plus a human-readable reason, so "good enough" becomes a number you can track over time.

In the [`Microsoft.Extensions.AI.Evaluation`](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) world, that scoring is done by evaluators, and they come in three flavors:

*   **Quality evaluators** (LLM-based): a second model acts as a judge and scores the answer. Examples: `RelevanceEvaluator`, `GroundednessEvaluator`, `CoherenceEvaluator`, `CompletenessEvaluator`.
    
*   **NLP evaluators** (no AI): classic text-similarity metrics that compare a response to a reference answer. Examples: `BLEUEvaluator`, `GLEUEvaluator`, `F1Evaluator`.
    
*   **Safety evaluators**: content-harm and protected-material checks that run against the Microsoft Foundry evaluation service, such as `ContentHarmEvaluator` and `HateAndUnfairnessEvaluator`.
    

The libraries target .NET 8 or later. All snippets here use current stable tooling (.NET 10 and C# 14 at the time of writing), and they slot into any test framework - xUnit, NUnit, or MSTest.

## Why String Assertions Fail for AI Responses

If you have ever written `Assert.Equal("The Moon is 238,855 miles away", response)`, you already know how this ends. The next model version phrases it as "about 238,900 miles," and your test fails despite the answer being perfectly correct. The failure is in the test, not the model.

LLM evaluation replaces brittle equality with semantic scoring. Rather than asking "does this text match exactly", you ask "is this answer relevant to the question" and "is it grounded in the context I provided". Those questions survive rewording, model upgrades and temperature changes - the three things that make raw string comparison useless for AI output.

## The Building Blocks: Microsoft.Extensions.AI.Evaluation

Start by adding the packages you actually need. For LLM-based quality scoring plus disk reporting, that is three evaluation packages on top of the core `Microsoft.Extensions.AI` abstractions:

```bash
dotnet add package Microsoft.Extensions.AI.Evaluation
dotnet add package Microsoft.Extensions.AI.Evaluation.Quality
dotnet add package Microsoft.Extensions.AI.Evaluation.Reporting
```

Every evaluator talks to a model through an `IChatClient` - the same abstraction you already use to call the LLM in your API - wrapped in a `ChatConfiguration`. If you are new to `IChatClient`, our post on how to [stream LLM responses in ASP.NET Core](https://codingdroplets.com/stream-llm-responses-aspnet-core-ichatclient) covers the same client from the request side.

```csharp
// The judge model that scores your responses.
// Reuse any IChatClient - Ollama, GitHub Models, OpenAI, or Azure OpenAI.
ChatConfiguration judge = new(chatClient);
```

## A Real-World Scenario: Evaluating a Support-Desk Answer

Say our support API answers billing questions. We want to assert that its answers stay relevant and coherent as prompts and models evolve. First, gather the evaluators into a reporting configuration. The configuration decides which evaluators run, which judge model they use, where results are stored, and whether responses are cached.

```csharp
// Relevance + coherence, scored by the judge model, stored on disk.
ReportingConfiguration reporting = DiskBasedReportingConfiguration.Create(
    storageRootPath: "eval-results",
    evaluators: [new RelevanceEvaluator(), new CoherenceEvaluator()],
    chatConfiguration: judge,
    enableResponseCaching: true,
    executionName: buildId);   // group results per CI run
```

Each test creates a `ScenarioRun`, gets a real model response, and evaluates it. The `await using` matters - disposing the scenario run is what persists results to the store.

```csharp
await using ScenarioRun run = await reporting.CreateScenarioRunAsync(scenarioName);

ChatResponse answer = await run.ChatConfiguration!.ChatClient
    .GetResponseAsync(messages);

EvaluationResult result = await run.EvaluateAsync(messages, answer);
```

Now the payoff. Pull a metric by name and read its score, its interpretation, and the reason the judge gave:

```csharp
NumericMetric relevance =
    result.Get<NumericMetric>(RelevanceEvaluator.RelevanceMetricName);

Assert.False(relevance.Interpretation!.Failed, relevance.Reason);
Assert.True(relevance.Interpretation.Rating
    is EvaluationRating.Good or EvaluationRating.Exceptional);
```

That `relevance.Reason` string is gold during triage: when a score drops, the judge explains why in plain English, so you are not guessing what changed.

## How Do You Catch AI Quality Regressions in CI?

Run the evaluation as an ordinary test in your pipeline and let scores - not exact strings - gate the build. Three features make this practical:

1.  **Response caching.** With `enableResponseCaching: true`, the model response for a given prompt is cached (14 days by default) and reused until the prompt or model changes. Reruns are fast and cheap, and you are not billed for identical calls on every commit.
    
2.  **Execution names.** Passing a build id as the `executionName` groups each run's results separately, so you can compare this build against the last one instead of overwriting history.
    
3.  **The report.** The [`Microsoft.Extensions.AI.Evaluation.Console`](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) tool turns stored results into an HTML report:
    

```bash
dotnet tool run aieval report --path eval-results --output report.html
```

A word of hard-won advice: do not fail a build on a single test dropping from "Exceptional" to "Good." Model scores wobble. In production the pattern that worked for us was tracking the trend across many scenarios and failing only on a broad, sustained drop. This mindset pairs naturally with the metrics side of AI observability - see our walkthrough on [OpenTelemetry for AI endpoints in ASP.NET Core](https://codingdroplets.com/opentelemetry-ai-endpoints-aspnet-core) for wiring live scores into dashboards.

## Grounding and RAG: Did the Model Stick to Your Data?

If your endpoint uses retrieval-augmented generation, the most important question is not "does this sound right" but "did the model answer only from my documents." That is exactly what `GroundednessEvaluator` measures - how well a response aligns with the context you supplied - and `RetrievalEvaluator` scores how good the retrieved context was in the first place.

Groundedness is your hallucination alarm. A high relevance score with a low groundedness score is the classic tell that the model wrote a confident, on-topic answer that your source data never supported. If you are building the retrieval side, our guide to [the RAG pattern in ASP.NET Core](https://codingdroplets.com/rag-pattern-aspnet-core) pairs directly with grounded evaluation.

## Custom Evaluators for Your Own Rules

The built-in evaluators do not know your business rules. When you need one - "every billing answer must cite a ticket id," or "responses must stay under 100 words" - implement `IEvaluator` and return your own metric. Non-AI evaluators like this are fast and free because no model call is involved.

```csharp
public sealed class TicketIdEvaluator : IEvaluator
{
    public IReadOnlyCollection<string> EvaluationMetricNames => ["TicketCited"];

    public ValueTask<EvaluationResult> EvaluateAsync(
        IEnumerable<ChatMessage> messages,
        ChatResponse modelResponse,
        ChatConfiguration? chatConfiguration = null,
        IEnumerable<EvaluationContext>? additionalContext = null,
        CancellationToken cancellationToken = default)
    {
        bool cited = modelResponse.Text.Contains("TCK-");
        var metric = new BooleanMetric("TicketCited", value: cited);
        return new(new EvaluationResult(metric));
    }
}
```

Custom evaluators sit in the same reporting pipeline as the built-in ones, so their scores land in the same report alongside relevance and groundedness.

## Trade-offs and What to Watch Out For

*   **LLM-as-judge costs tokens.** Quality evaluators call a model to score each response. Caching absorbs most of that in CI, but budget for it and consider a cheaper judge model where accuracy allows.
    
*   **The judge is not infallible.** A weak judge model gives noisy scores. Use a capable model for evaluation even if your production endpoint runs a smaller one.
    
*   **Non-determinism is real.** Set a low temperature for the responses under test, and rely on trends, not single-run pass/fail.
    
*   **Safety evaluators need Foundry.** Content-harm and protected-material evaluators run against the Microsoft Foundry service, not a local model - factor that into environment setup.
    

Evaluation is one control among several. It fits neatly into the broader [AI-Powered .NET API production readiness checklist](https://codingdroplets.com/ai-powered-dotnet-api-production-readiness-checklist) alongside cost caps, guardrails, and observability.

## What to Do Next

Pick one AI endpoint. Write three or four representative questions with known-good expectations. Wire up `RelevanceEvaluator` and `GroundednessEvaluator`, enable response caching, and run the evaluation in your pipeline. Generate the report and watch the scores for a week before you gate anything. Once you trust the signal, tighten it: add custom evaluators for your domain rules and fail the build on sustained drops. That is the difference between hoping your AI feature still works and knowing it does.

## Frequently Asked Questions

### How do I evaluate LLM output in a .NET API without an exact expected answer?

Use quality evaluators such as `RelevanceEvaluator` and `CoherenceEvaluator` that score semantic quality against the question rather than a fixed string. They return a numeric score and a reason, so you assert on the interpretation (Good, Exceptional, Unacceptable) instead of on exact text. This survives rewording and model upgrades.

### What is the difference between relevance and groundedness evaluators?

`RelevanceEvaluator` measures whether the answer addresses the user's question. `GroundednessEvaluator` measures whether the answer is supported by the context you supplied, which is what catches hallucinations in RAG systems. A high-relevance, low-groundedness result usually means a confident answer your source data never backed up.

### Does Microsoft.Extensions.AI.Evaluation work with xUnit and NUnit?

Yes. The libraries are test-framework agnostic and integrate with xUnit, NUnit, or MSTest, and with `dotnet test` in any CI/CD pipeline. You create a `ScenarioRun`, call `EvaluateAsync`, and assert on the resulting metrics like any other test.

### How do I stop LLM evaluation from slowing down and inflating my CI bill?

Enable response caching in the reporting configuration. Responses are cached (14 days by default) and reused as long as the prompt and model are unchanged, so repeated runs read from the cache instead of billing a fresh model call. Use a distinct execution name per build to keep results comparable across runs.

### Should a failing evaluation score block my build?

Not on a single test. Model scores fluctuate, so blocking on one metric dropping a level produces flaky pipelines. Track the trend across many scenarios using the generated report and fail the build only on a broad, sustained decline in scores rather than a one-off wobble.

* * *

## About the Author

I'm Celin Daniel, Co-founder of [Coding Droplets](https://codingdroplets.com/). I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

*   GitHub: [codingdroplets](http://github.com/codingdroplets/)
    
*   YouTube: [Coding Droplets](https://www.youtube.com/@CodingDroplets)
    
*   Website: [codingdroplets.com](https://codingdroplets.com/)