Kubernetes troubleshooting

OOMKilled: what to check before raising the memory limit

Published September 25, 2026 · Fortem engineering

OOMKilled identifies a container that the kernel killed after an out-of-memory condition. It does not tell you whether the configured limit was too small, the application's memory grew unexpectedly, or the node was under pressure. First preserve the termination fact, then compare the memory boundary, previous-container logs, and timing. Raising a limit can be a mitigation, but it is not a diagnosis.

1. Confirm which container terminated

Use the exact context and namespace. In Containers, look for Last State: Terminated, Reason: OOMKilled, the finish time, and the restart count. Check the Pod's Events too. Exit code 137 alone is not a substitute for the termination reason.

kubectl --context YOUR_CONTEXT -n YOUR_NAMESPACE describe pod YOUR_POD

If the Pod was replaced, the old Pod object and its last termination state may no longer be available. A new healthy Pod does not reconstruct what happened to the deleted one. Preserve the old Pod name or consult an existing historical source if you have one.

2. Read the request and limit separately

A memory request informs scheduling and node-pressure decisions. A memory limit, if set, is the container's enforced boundary. The two numbers answer different questions. Inspect the Pod that failed, not just the current Deployment template: admission policies or a later rollout may have changed the values.

kubectl --context YOUR_CONTEXT -n YOUR_NAMESPACE get pod YOUR_POD \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests.memory}{"\t"}{.resources.limits.memory}{"\n"}{end}'

The three columns are container name, memory request and memory limit. An empty value means that field is not set in this Pod spec; it is not zero measured usage.

Limit set

Compare it with workload needs and historical usage, if available. A limit breach is plausible; the termination reason by itself does not name the application code path.

No limit shown

Do not invent a limit breach. Inspect Node conditions, other affected Pods and any node-level OOM evidence your access permits.

Pod says Evicted

Follow the eviction message and node-pressure path instead of treating it as the same event as a container's OOMKilled last state.

3. Recover the evidence that can disappear

Read the previous container's last lines while its Pod still exists. This is a Kubernetes log read, not a replay of every earlier restart; --previous can be unavailable if no prior container instance remains. Treat log lines as context, not proof that a particular message caused the kill.

kubectl --context YOUR_CONTEXT -n YOUR_NAMESPACE logs YOUR_POD \
  -c YOUR_CONTAINER --previous --tail=100 --timestamps

You can check current usage, but interpret its timestamp:

kubectl --context YOUR_CONTEXT -n YOUR_NAMESPACE top pod YOUR_POD --containers

kubectl top needs a working Metrics API. Its value after a restart describes the new container, not the peak before the OOM. A low current reading does not disprove a previous spike. If you have an existing time-series backend, compare its container-level memory curve around the termination time; without one, mark the historical peak unknown.

4. Check what changed—without declaring a root cause

Compare the termination time with image changes, rollout history, config changes and traffic shape. For a Deployment, this command can show available revision history:

kubectl --context YOUR_CONTEXT -n YOUR_NAMESPACE rollout history deployment/YOUR_DEPLOYMENT

A rollout near the first OOM is a useful lead, not proof that the rollout caused it. Revision history can be incomplete; Kubernetes does not expose a universal, trustworthy “last deployed at” timestamp for every workload. If several Pods on one Node were affected, inspect Node conditions and events before changing one container's limit:

kubectl --context YOUR_CONTEXT describe node YOUR_NODE

Find the Pod's Node: field in the earlier describe pod output. Node reads may be forbidden by your RBAC; that means this branch is unverified, not that pressure was absent. Do not mutate a production workload just to test this article.

A small example: what the facts support

Synthetic example: payments-api has 2/3 ready Pods. One container's last state says OOMKilled; its configured limit is 1 GiB. Current usage after restart is 620 MiB, and a new image appeared shortly before the failure.

You can state the termination, configured boundary, current readiness and change timing. You cannot conclude that usage peaked at 1 GiB from the 620 MiB snapshot, that the rollout caused the OOM, or that raising the limit will fix a leak. The next useful check is the previous container log plus any retained memory history around the finish time.

Fortem's workload view puts readiness, restarts, last termination reason, requests/limits, available usage, images, events and current/previous logs together. Its incident brief keeps observations apart from hypotheses. Optional metrics can be unavailable; Fortem does not backfill memory usage from before it was running. For a separate permission diagnosis, see why node CPU and memory may be missing.

Sources and scope

Checked September 25, 2026. The commands are read-only and use placeholders. The example is synthetic; no customer cluster, live OOM or historical metric series was used to validate this article.