.NET Container OOMKilled in Kubernetes: Root Cause and Fix

The pod restarts at 3 AM. No exception, no stack trace, no shutdown log - the process is simply gone. kubectl describe pod tells the whole story in three lines:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

When a .NET container is OOMKilled in Kubernetes, the kernel killed it. Your application never got a chance to log anything, which is why this failure feels so opaque compared to every other production problem. Exit code 137 is 128 plus signal 9: SIGKILL, delivered by the cgroup memory controller the instant the container crossed its limit.

In production I've chased this on three different services, and only one of them turned out to be an actual memory leak. The other two were the runtime behaving exactly as documented against a memory limit nobody had reasoned about. This article separates those cases, because the diagnosis determines whether you spend an afternoon on config or a week on a heap dump. The instrumented setup - counters, dump capture, and the runtime config that goes with it - is on Patreon if you would rather start from a working baseline.

Most of what follows is invisible until you containerise, which is why teams meet it for the first time in production rather than on a laptop. Getting the container itself right - the multi-stage build, the non-root user, the production readiness checklist that surrounds it - is what Chapter 15 of the Zero to Production course walks through, and a correctly built image removes a whole category of noise before you start tuning anything.

Verified against .NET 10.

Why the .NET GC Makes This Worse Than You Expect

The critical thing to internalise: the .NET GC sizes its heap against the container's memory limit, not the node's memory. When it detects a memory-constrained environment, the heap hard limit defaults to a percentage of that limit.

That percentage is 75 percent. If you set limits.memory: 512Mi, the GC will happily grow the managed heap to roughly 384 MB before it feels any pressure at all.

Now add everything the managed heap does not include:

• The runtime itself, the JIT, and loaded assemblies

• Thread stacks - one megabyte each by default, and a busy ASP.NET Core app has plenty

• Native buffers held by Kestrel, sockets, and TLS

• Native memory in database drivers and any unmanaged dependency

• GC bookkeeping structures

That is your remaining 25 percent - 128 MB in the example above. When the sum crosses 512 MB, the kernel kills the container. From the GC's point of view nothing went wrong; it stayed inside its budget. The budget was just the wrong size for the whole process.

This is why "we're only using 380 MB of heap" and "we got OOMKilled at 512 MB" are both true statements about the same incident.

How to Diagnose It

Do these three things in order. They take about twenty minutes together and they tell you which of the causes below you are actually looking at.

Step 1: Confirm What Limit the Runtime Detected

Half of all cases are a mismatch between the limit you think you set and the limit the runtime sees. Ask the runtime directly:

app.MapGet("/diag/memory", () => new
{
    DetectedLimitMb = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes / 1024 / 1024,
    HeapMb          = GC.GetTotalMemory(false) / 1024 / 1024,
    WorkingSetMb    = Environment.WorkingSet / 1024 / 1024,
    Processors      = Environment.ProcessorCount,
    ServerGc        = System.Runtime.GCSettings.IsServerGC
});

Put it behind auth. TotalAvailableMemoryBytes is what the GC is actually budgeting against. If it reports the node's memory rather than your pod limit, the runtime never saw the cgroup limit and every assumption below is void - check that limits.memory is set on the container and not only on a namespace default.

Step 2: Compare Managed Heap Against Working Set

This single comparison splits the problem cleanly:

• Heap is large and growing, working set tracks it. You have a managed memory problem - either a real leak or a heap budget set too high for the limit.

• Heap is small and stable, working set is much larger and growing. Your problem is native, and no GC setting will fix it. Look at undisposed handles, native driver buffers, or unbounded connection pools.

Attach dotnet-counters to a running pod and watch GC Heap Size, Working Set, LOH Size and Gen 2 GC Count under real traffic. That takes minutes and eliminates the wrong half of the search space.

Step 3: Capture a Heap Dump Only If Step 2 Says Managed

dotnet-gcdump is much lighter than a full dump and usually enough to identify a leak's retention path. Grab two, several minutes apart, under load, and compare. A type whose instance count grows monotonically and never falls is your answer.

Do not start here. A heap dump on a native memory problem tells you nothing and costs an afternoon.

Cause 1: The 75 Percent Default Leaves Too Little Headroom

The most common configuration cause. The GC's default budget is fine for a process that is mostly managed memory, and too generous for an API doing heavy TLS, large uploads, or holding native driver buffers.

The fix is to lower the heap hard limit percent and give native memory room:

{
  "runtimeOptions": {
    "configProperties": {
      "System.GC.HeapHardLimitPercent": 65
    }
  }
}

Start at 65 or 70 and measure. Too low and you trade OOMKills for constant gen 2 collections and higher CPU, which is a different production problem. The right value depends on how native-heavy your workload is, so change it, watch gen 2 counts and p99 latency for a day, and adjust.

You can set the same thing as DOTNET_GCHeapHardLimitPercent, but read the next section before you do.

Cause 2: The Hexadecimal Environment Variable Trap

This one is worth its own section because I have watched it burn a full day.

GC settings expressed as environment variables are interpreted as hexadecimal. The same settings in runtimeconfig.json are decimal.

So a team wanting a 500 MB heap limit sets:

env:
  - name: DOTNET_GCHeapHardLimit
    value: "500000000"     # intended 500 MB

The runtime reads that as 0x500000000, which is roughly 21.5 GB. The limit is effectively removed, the heap grows without restraint, and the container gets OOMKilled faster than before the "fix" was applied. The correct hex value for 500 MB is 1DCD6500.

If you can, prefer runtimeconfig.json or the MSBuild properties precisely so this cannot happen. If you must use environment variables, write the intended decimal value in a comment next to the hex.

Cause 3: Server GC With Many Heaps in a Small Pod

Server GC creates one heap per logical processor, each with its own allocation budget. On a 32-core node with a 512Mi pod limit, that arithmetic does not work in your favour.

The runtime default is Workstation GC, but Server GC is enabled in a great many ASP.NET Core deployments - sometimes deliberately, sometimes inherited from a template nobody reviewed. Do not assume; the diagnostic endpoint above reports IsServerGC, so check.

Two fixes, and the first is usually right on modern runtimes:

• Make sure DATAS is on. Dynamic Adaptation To Application Sizes adjusts heap count and budgets to what the app actually needs instead of to the core count, and it is enabled by default from .NET 9. If you are on .NET 8, opt in with System.GC.DynamicAdaptationMode: 1. This is the single most effective change for Server GC in a small container.

• Or cap the heap count with System.GC.HeapCount if you have a specific reason to keep Server GC without DATAS.

For genuinely small services - a sidecar, a lightweight worker - Workstation GC is often simply the better fit and costs you nothing you were using.

Cause 4: A Real Managed Leak

Sometimes it is exactly what it looks like. The usual suspects in an ASP.NET Core API:

• A static dictionary or IMemoryCache with no size limit and no expiration

• Event handlers subscribed and never unsubscribed, keeping their targets alive

• A BackgroundService accumulating state across the process lifetime

• An unbounded Channel whose consumer is slower than its producer

• A scoped service captured by a singleton, keeping a DbContext and its change tracker alive

The tell is a heap that grows monotonically and never returns to baseline after a gen 2 collection. GC configuration cannot fix a leak; it only changes how long you survive before the kernel notices. Our post on memory leaks in ASP.NET Core background services walks the most common one in detail.

Cause 5: Large Object Heap Pressure

File uploads, large JSON payloads and big byte[] buffers go on the Large Object Heap, which is not compacted by default and fragments. Working set climbs, the managed heap looks larger than the live data justifies, and the container dies.

Two responses. Fix the allocation by streaming instead of buffering, and by renting from ArrayPool<byte>.Shared rather than allocating fresh arrays per request - that is the durable fix. As a mitigation, System.GC.ConserveMemory accepts 0 to 9 and compacts the LOH automatically when fragmentation is high; the Microsoft guidance is to start between 5 and 7. It costs CPU and pause time, so treat it as a stopgap while you fix the allocations.

How to Stop It Recurring

• Always set limits.memory explicitly, and set requests.memory equal to it. That gives the pod Guaranteed QoS so it is not evicted first under node pressure, and gives the GC an unambiguous number to size against.

• Alert on working set as a fraction of the limit, not on restarts. Crossing 85 percent is a warning; a restart is a post-mortem.

• Pin GC settings in runtimeconfig.json, in source control, reviewed. Not in a Helm value someone edits during an incident.

• Load-test in a container with production limits. Almost every case in this article is invisible on a developer machine with 32 GB of RAM, and obvious within ten minutes of load against a 512Mi limit.

• Configure probes so a struggling pod is restarted gracefully rather than SIGKILLed mid-request. Our guide on health checks and Kubernetes probes covers the liveness and readiness split.

• Track it on the release checklist. The production readiness checklist is where this belongs, not in tribal memory.

Frequently Asked Questions

What Does Exit Code 137 Mean for a .NET Container?

It means the process received SIGKILL (128 + 9). In Kubernetes this is almost always the cgroup memory controller enforcing limits.memory. Because SIGKILL cannot be handled, your application gets no chance to log, flush telemetry or shut down cleanly - which is exactly why the failure looks like the process vanished.

Does the .NET GC Respect Kubernetes Memory Limits Automatically?

Yes, and that is the source of the confusion. The GC reads the cgroup limit and sets its heap hard limit to 75 percent of it by default. The problem is not that it ignores the limit, it is that the remaining 25 percent must cover the runtime, thread stacks, and all native allocations - which for some workloads is not enough.

Should I Use Workstation GC or Server GC in a Container?

Server GC gives better throughput and is the right choice for a busy API with reasonable memory headroom. Workstation GC uses less memory and suits small services and many-small-pods deployments. On .NET 9 and later, DATAS is enabled by default and substantially narrows the memory gap, so measure both under production-shaped load rather than picking on principle.

Will Increasing the Memory Limit Fix the Problem?

It buys time and sometimes that is the right call during an incident. It does not fix a leak, and because the GC sizes its heap as a percentage of the limit, raising the limit also raises the heap budget - so a process that was OOMKilled at 512Mi may simply be OOMKilled at 1Gi a few hours later. Diagnose first, then decide whether more memory is the answer.

How Do I Tell a Managed Leak From a Native One?

Compare the managed heap size against the working set. If both grow together, it is managed and a dotnet-gcdump comparison will find it. If the heap stays flat while working set climbs, the leak is native - undisposed handles, driver buffers, connection pools - and no heap dump or GC setting will help.

Where Are the GC Settings Officially Documented?

The garbage collector config settings reference on Microsoft Learn documents every setting named here, including the exact defaults, the introducing .NET version, and the decimal-versus-hexadecimal rule that catches so many teams out.

---

About the Author

I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

• GitHub: codingdroplets

• YouTube: Coding Droplets

• Website: codingdroplets.com