# .NET Container OOMKilled in Kubernetes: Root Cause and Fix

The pod restarts at 3 AM. No exception, no stack trace, no shutdown log - the process is simply gone. `kubectl describe pod` tells the whole story in three lines:

```text
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
```

When a **.NET container is OOMKilled in Kubernetes**, the kernel killed it. Your application never got a chance to log anything, which is why this failure feels so opaque compared to every other production problem. Exit code 137 is 128 plus signal 9: SIGKILL, delivered by the cgroup memory controller the instant the container crossed its limit.

In production I've chased this on three different services, and only one of them turned out to be an actual memory leak. The other two were the runtime behaving exactly as documented against a memory limit nobody had reasoned about. This article separates those cases, because the diagnosis determines whether you spend an afternoon on config or a week on a heap dump. The instrumented setup - counters, dump capture, and the runtime config that goes with it - is on [Patreon](https://www.patreon.com/CodingDroplets) if you would rather start from a working baseline.

Most of what follows is invisible until you containerise, which is why teams meet it for the first time in production rather than on a laptop. Getting the container itself right - the multi-stage build, the non-root user, the production readiness checklist that surrounds it - is what [Chapter 15 of the Zero to Production course](https://aspnetcoreapi.codingdroplets.com/) walks through, and a correctly built image removes a whole category of noise before you start tuning anything.

[![ASP.NET Core Web API: Zero to Production](https://newsletter.codingdroplets.com/images/aspnet-core-api-course-banner-1.jpg align="center")](https://aspnetcoreapi.codingdroplets.com/)

Verified against .NET 10.

## Why the .NET GC Makes This Worse Than You Expect

The critical thing to internalise: **the .NET GC sizes its heap against the container's memory limit, not the node's memory.** When it detects a memory-constrained environment, the heap hard limit defaults to a percentage of that limit.

That percentage is **75 percent**. If you set `limits.memory: 512Mi`, the GC will happily grow the managed heap to roughly 384 MB before it feels any pressure at all.

Now add everything the managed heap does not include:

*   The runtime itself, the JIT, and loaded assemblies
    
*   Thread stacks - one megabyte each by default, and a busy ASP.NET Core app has plenty
    
*   Native buffers held by Kestrel, sockets, and TLS
    
*   Native memory in database drivers and any unmanaged dependency
    
*   GC bookkeeping structures
    

That is your remaining 25 percent - 128 MB in the example above. When the sum crosses 512 MB, the kernel kills the container. From the GC's point of view nothing went wrong; it stayed inside its budget. The budget was just the wrong size for the whole process.

This is why "we're only using 380 MB of heap" and "we got OOMKilled at 512 MB" are both true statements about the same incident.

## How to Diagnose It

Do these three things in order. They take about twenty minutes together and they tell you which of the causes below you are actually looking at.

### Step 1: Confirm What Limit the Runtime Detected

Half of all cases are a mismatch between the limit you think you set and the limit the runtime sees. Ask the runtime directly:

```csharp
app.MapGet("/diag/memory", () => new
{
    DetectedLimitMb = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes / 1024 / 1024,
    HeapMb          = GC.GetTotalMemory(false) / 1024 / 1024,
    WorkingSetMb    = Environment.WorkingSet / 1024 / 1024,
    Processors      = Environment.ProcessorCount,
    ServerGc        = System.Runtime.GCSettings.IsServerGC
});
```

Put it behind auth. `TotalAvailableMemoryBytes` is what the GC is actually budgeting against. If it reports the node's memory rather than your pod limit, the runtime never saw the cgroup limit and every assumption below is void - check that `limits.memory` is set on the container and not only on a namespace default.

### Step 2: Compare Managed Heap Against Working Set

This single comparison splits the problem cleanly:

*   **Heap is large and growing, working set tracks it.** You have a managed memory problem - either a real leak or a heap budget set too high for the limit.
    
*   **Heap is small and stable, working set is much larger and growing.** Your problem is native, and no GC setting will fix it. Look at undisposed handles, native driver buffers, or unbounded connection pools.
    

Attach `dotnet-counters` to a running pod and watch `GC Heap Size`, `Working Set`, `LOH Size` and `Gen 2 GC Count` under real traffic. That takes minutes and eliminates the wrong half of the search space.

### Step 3: Capture a Heap Dump Only If Step 2 Says Managed

`dotnet-gcdump` is much lighter than a full dump and usually enough to identify a leak's retention path. Grab two, several minutes apart, under load, and compare. A type whose instance count grows monotonically and never falls is your answer.

Do not start here. A heap dump on a native memory problem tells you nothing and costs an afternoon.

## Cause 1: The 75 Percent Default Leaves Too Little Headroom

The most common configuration cause. The GC's default budget is fine for a process that is mostly managed memory, and too generous for an API doing heavy TLS, large uploads, or holding native driver buffers.

**The fix** is to lower the heap hard limit percent and give native memory room:

```json
{
  "runtimeOptions": {
    "configProperties": {
      "System.GC.HeapHardLimitPercent": 65
    }
  }
}
```

Start at 65 or 70 and measure. Too low and you trade OOMKills for constant gen 2 collections and higher CPU, which is a different production problem. The right value depends on how native-heavy your workload is, so change it, watch gen 2 counts and p99 latency for a day, and adjust.

You can set the same thing as `DOTNET_GCHeapHardLimitPercent`, but read the next section before you do.

## Cause 2: The Hexadecimal Environment Variable Trap

This one is worth its own section because I have watched it burn a full day.

**GC settings expressed as environment variables are interpreted as hexadecimal. The same settings in** `runtimeconfig.json` **are decimal.**

So a team wanting a 500 MB heap limit sets:

```yaml
env:
  - name: DOTNET_GCHeapHardLimit
    value: "500000000"     # intended 500 MB
```

The runtime reads that as `0x500000000`, which is roughly 21.5 GB. The limit is effectively removed, the heap grows without restraint, and the container gets OOMKilled faster than before the "fix" was applied. The correct hex value for 500 MB is `1DCD6500`.

If you can, prefer `runtimeconfig.json` or the MSBuild properties precisely so this cannot happen. If you must use environment variables, write the intended decimal value in a comment next to the hex.

## Cause 3: Server GC With Many Heaps in a Small Pod

Server GC creates one heap per logical processor, each with its own allocation budget. On a 32-core node with a 512Mi pod limit, that arithmetic does not work in your favour.

The runtime default is Workstation GC, but Server GC is enabled in a great many ASP.NET Core deployments - sometimes deliberately, sometimes inherited from a template nobody reviewed. Do not assume; the diagnostic endpoint above reports `IsServerGC`, so check.

Two fixes, and the first is usually right on modern runtimes:

*   **Make sure DATAS is on.** Dynamic Adaptation To Application Sizes adjusts heap count and budgets to what the app actually needs instead of to the core count, and it is **enabled by default from .NET 9**. If you are on .NET 8, opt in with `System.GC.DynamicAdaptationMode: 1`. This is the single most effective change for Server GC in a small container.
    
*   **Or cap the heap count** with `System.GC.HeapCount` if you have a specific reason to keep Server GC without DATAS.
    

For genuinely small services - a sidecar, a lightweight worker - Workstation GC is often simply the better fit and costs you nothing you were using.

## Cause 4: A Real Managed Leak

Sometimes it is exactly what it looks like. The usual suspects in an ASP.NET Core API:

*   A `static` dictionary or `IMemoryCache` with no size limit and no expiration
    
*   Event handlers subscribed and never unsubscribed, keeping their targets alive
    
*   A `BackgroundService` accumulating state across the process lifetime
    
*   An unbounded `Channel` whose consumer is slower than its producer
    
*   A scoped service captured by a singleton, keeping a `DbContext` and its change tracker alive
    

The tell is a heap that grows monotonically and never returns to baseline after a gen 2 collection. GC configuration cannot fix a leak; it only changes how long you survive before the kernel notices. Our post on [memory leaks in ASP.NET Core background services](https://codingdroplets.com/aspnet-core-memory-leak-background-service-fix) walks the most common one in detail.

## Cause 5: Large Object Heap Pressure

File uploads, large JSON payloads and big `byte[]` buffers go on the Large Object Heap, which is not compacted by default and fragments. Working set climbs, the managed heap looks larger than the live data justifies, and the container dies.

Two responses. **Fix the allocation** by streaming instead of buffering, and by renting from `ArrayPool<byte>.Shared` rather than allocating fresh arrays per request - that is the durable fix. As a mitigation, `System.GC.ConserveMemory` accepts 0 to 9 and compacts the LOH automatically when fragmentation is high; the Microsoft guidance is to start between 5 and 7. It costs CPU and pause time, so treat it as a stopgap while you fix the allocations.

## How to Stop It Recurring

*   **Always set** `limits.memory` **explicitly**, and set `requests.memory` equal to it. That gives the pod Guaranteed QoS so it is not evicted first under node pressure, and gives the GC an unambiguous number to size against.
    
*   **Alert on working set as a fraction of the limit**, not on restarts. Crossing 85 percent is a warning; a restart is a post-mortem.
    
*   **Pin GC settings in** `runtimeconfig.json`**,** in source control, reviewed. Not in a Helm value someone edits during an incident.
    
*   **Load-test in a container with production limits.** Almost every case in this article is invisible on a developer machine with 32 GB of RAM, and obvious within ten minutes of load against a 512Mi limit.
    
*   **Configure probes so a struggling pod is restarted gracefully** rather than SIGKILLed mid-request. Our guide on [health checks and Kubernetes probes](https://codingdroplets.com/aspnet-core-health-checks-kubernetes-readiness-liveness-startup-probes) covers the liveness and readiness split.
    
*   **Track it on the release checklist.** The [production readiness checklist](https://codingdroplets.com/aspnet-core-production-readiness-checklist-dotnet-teams) is where this belongs, not in tribal memory.
    

## Frequently Asked Questions

### What Does Exit Code 137 Mean for a .NET Container?

It means the process received SIGKILL (128 + 9). In Kubernetes this is almost always the cgroup memory controller enforcing `limits.memory`. Because SIGKILL cannot be handled, your application gets no chance to log, flush telemetry or shut down cleanly - which is exactly why the failure looks like the process vanished.

### Does the .NET GC Respect Kubernetes Memory Limits Automatically?

Yes, and that is the source of the confusion. The GC reads the cgroup limit and sets its heap hard limit to 75 percent of it by default. The problem is not that it ignores the limit, it is that the remaining 25 percent must cover the runtime, thread stacks, and all native allocations - which for some workloads is not enough.

### Should I Use Workstation GC or Server GC in a Container?

Server GC gives better throughput and is the right choice for a busy API with reasonable memory headroom. Workstation GC uses less memory and suits small services and many-small-pods deployments. On .NET 9 and later, DATAS is enabled by default and substantially narrows the memory gap, so measure both under production-shaped load rather than picking on principle.

### Will Increasing the Memory Limit Fix the Problem?

It buys time and sometimes that is the right call during an incident. It does not fix a leak, and because the GC sizes its heap as a percentage of the limit, raising the limit also raises the heap budget - so a process that was OOMKilled at 512Mi may simply be OOMKilled at 1Gi a few hours later. Diagnose first, then decide whether more memory is the answer.

### How Do I Tell a Managed Leak From a Native One?

Compare the managed heap size against the working set. If both grow together, it is managed and a `dotnet-gcdump` comparison will find it. If the heap stays flat while working set climbs, the leak is native - undisposed handles, driver buffers, connection pools - and no heap dump or GC setting will help.

### Where Are the GC Settings Officially Documented?

The [garbage collector config settings reference on Microsoft Learn](https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector) documents every setting named here, including the exact defaults, the introducing .NET version, and the decimal-versus-hexadecimal rule that catches so many teams out.

* * *

## About the Author

I'm Celin Daniel, Co-founder of [Coding Droplets](https://codingdroplets.com/). I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

*   GitHub: [codingdroplets](http://github.com/codingdroplets/)
    
*   YouTube: [Coding Droplets](https://www.youtube.com/@CodingDroplets)
    
*   Website: [codingdroplets.com](https://codingdroplets.com/)