Warrior, this is one of the most important chapters in the entire Kubernetes course.
In real projects, knowing Kubernetes isn't just about creating Deployments and Services.
The real skill is:
When something breaks, can you quickly identify where the problem is and fix it?
We'll build a systematic troubleshooting method instead of randomly running commands.
When something doesn't work, don't immediately change YAML.
First ask:
What exactly is broken?
โ
Which layer is broken?
โ
Can I reproduce it?
โ
What evidence do I have?
โ
What is the smallest fix?Think in layers:
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Application โ
โ Node.js / Next.js / etc. โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Container / Pod โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Deployment / Job โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Service โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Ingress / DNS โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Node / Cluster โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Cloud / Infrastructure โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโFind the broken layer first.
Memorize this:
1. kubectl get
โ
2. kubectl describe
โ
3. kubectl logs
โ
4. kubectl logs --previous
โ
5. Check Events
โ
6. Check Service
โ
7. Check Endpoints
โ
8. Check DNS / Network
โ
9. Check Node / ResourcesThe three commands you should instinctively remember are:
kubectl get
kubectl describe
kubectl logskubectl get#Suppose your application isn't working.
Start with:
kubectl get podsExample:
NAME READY STATUS RESTARTS
node-api-7d9f8c6b-x2k4m 1/1 Running 0
node-api-7d9f8c6b-q8m2p 0/1 CrashLoopBackOff 7Immediately you know:
One Pod is healthy and another is repeatedly crashing.
Don't modify anything yet.
Investigate.
Useful command:
kubectl get allYou'll see resources such as:
NAME READY
pod/node-api-xxx 1/1
NAME TYPE CLUSTER-IP
service/node-api ClusterIP 10.96.20.10
NAME READY
deployment.apps/node-api 2/2For production, namespace matters:
kubectl get all -n productionRun:
kubectl describe pod <pod-name>Example:
kubectl describe pod node-api-xxxThe most important section is usually:
Events:Example:
Warning FailedScheduling
0/3 nodes are available:
2 Insufficient memory
1 node has untolerated taintNow you have evidence.
Events are extremely useful.
Run:
kubectl get eventsFor a namespace:
kubectl get events -n productionSort by time:
kubectl get events \
--sort-by=.lastTimestampEvents can reveal:
FailedScheduling
FailedMount
Failed
BackOff
Unhealthy
Pulling
Pulled
Created
Started
KillingFor beginners:
When something unexpected happens in Kubernetes, check Events.
One of the most common Kubernetes errors.
You see:
NAME READY STATUS
node-api 0/1 CrashLoopBackOffWhat does it mean?
It means roughly:
The container starts, crashes, Kubernetes restarts it, it crashes again, and Kubernetes increasingly backs off between restart attempts.
It does not itself tell you the root cause.
First:
kubectl logs <pod>Then:
kubectl logs <pod> --previousThen:
kubectl describe pod <pod>Look for:
Last State:
Terminated
Reason:
Exit Code:Logs:
Error: Cannot connect to database
at connectDatabase (/app/db.js:42:11)Now you know:
Kubernetes isn't necessarily the root problem.
Your application is crashing because DB connection failed.Potential causes:
Wrong DATABASE_URL
Wrong Secret
Database unavailable
DNS failure
NetworkPolicy
Credentials invalidSuppose:
Error: listen EADDRINUSE: address already in useThis is an application/container problem.
Don't immediately blame Kubernetes.
Check:
Application
โ
Port configuration
โ
Container configurationYou see:
Last State:
Terminated
Reason: OOMKilledOOM means:
Out Of Memory.
Suppose your Deployment has:
resources:
limits:
memory: "512Mi"But the application needs more memory.
The container can be terminated when it exceeds its memory limit.
Run:
kubectl describe pod <pod>Look for:
Reason: OOMKilledThen:
kubectl top pod <pod>if metrics are available.
Check:
resources:
requests:
memory: "256Mi"
limits:
memory: "512Mi"Questions to ask:
Is the memory limit too low?
Is the application leaking memory?
Did traffic increase?
Is Node.js heap growing?
Are large files/models being loaded?This is especially relevant for Node.js applications.
Suppose:
Node.js API
โ
512Mi memory limit
โ
Large JSON processing
โ
Memory usage increases
โ
Container exceeds limit
โ
OOMKilledYou may need to investigate:
Don't blindly increase memory.
First understand why memory increased.
Suppose:
kubectl get podsshows:
node-api 0/1 ImagePullBackOffThis usually means Kubernetes cannot successfully pull the container image.
Check:
kubectl describe pod <pod>Look at Events.
You might see:
Failed to pull imageimage: my-node-appp:1.0instead of:
image: my-node-app:1.0image: my-node-api:999when tag doesn't exist.
The cluster doesn't have permission to pull it.
You may need:
imagePullSecretsor cloud workload/node identity depending on your environment.
Run:
kubectl describe pod <pod>Find:
Events:Example:
Failed to pull image "myrepo/node-api:5.0"
manifest unknownNow the issue is clear:
Image/tag doesn't exist.
ErrImagePull vs ImagePullBackOff#You may see:
ErrImagePullThis means image pulling failed.
Then Kubernetes retries.
Eventually you may see:
ImagePullBackOffThink:
Image pull fails
โ
ErrImagePull
โ
Retry
โ
Backoff
โ
ImagePullBackOffExample:
NAME READY STATUS
node-api 0/1 PendingA Pending Pod hasn't been successfully scheduled and started.
Run:
kubectl describe pod <pod>Look at Events.
You might see:
0/3 nodes are available:
3 Insufficient cpuInsufficient cpuInsufficient memoryPod requires:
workload=gpubut no node has that label.
Node has:
workload=gpu:NoScheduleand Pod doesn't have the appropriate toleration.
Pod requires storage that isn't available.
Pod cannot be placed according to topology rules.
Use:
kubectl describe pod <pod>Then ask:
Is there enough CPU?
Is there enough memory?
Are nodeSelector/affinity rules valid?
Are there taints?
Does the Pod tolerate them?
Is PVC available?
Are topology constraints too strict?Remember Chapter 16.
Suppose:
HPA
โ
Pods increase
โ
No node has enough capacity
โ
Pods Pending
โ
Cluster Autoscaler
โ
New node
โ
Scheduler
โ
Pod RunningIf your cloud cluster has a Cluster Autoscaler configured, a Pending Pod caused by insufficient capacity may trigger node scaling.
But:
Not every Pending Pod is a Cluster Autoscaler problem.
If the reason is:
untolerated taintadding another generic node may not solve it.
You may see:
STATUS
ContainerCreatingfor a long time.
This means Kubernetes is still trying to prepare/start the container.
Potential causes:
Image pulling
Volume mounting
Secret/ConfigMap issues
CNI/network setup
Container runtime issuesRun:
kubectl describe pod <pod>Again:
Events are your friend.
Example:
CreateContainerConfigErrorOften indicates a configuration dependency problem.
For example, your Deployment says:
envFrom:
- secretRef:
name: database-secretBut:
database-secretdoesn't exist.
Check:
kubectl get secretsand:
kubectl describe pod <pod>You may find:
secret "database-secret" not foundSuppose:
envFrom:
- configMapRef:
name: node-api-configbut ConfigMap doesn't exist.
Check:
kubectl get configmapsor:
kubectl get cmThen:
kubectl describe pod <pod>Suppose:
NAME READY STATUS
node-api 0/1 RunningThis is an important situation.
The container is running.
But:
READY = 0means the Pod isn't currently ready to receive normal Service traffic.
Check:
kubectl describe pod <pod>You may see:
Readiness probe failed:
HTTP probe failed with statuscode: 503Remember Chapter 14.
Container running
โ
Readiness probe
โ
Success?
โโโโโดโโโโ
Yes No
โ โ
Traffic No normal
allowed Service trafficSo:
Running โ ReadyThis is one of the most important Kubernetes concepts.
Application:
3000Probe:
port: 8080Application:
/healthProbe:
path: /healthzFor container networking, your server generally needs to listen on:
0.0.0.0rather than only:
127.0.0.1Use an appropriate:
startupProbeSuppose:
Frontend
โ
backend-service
โ
No responseFirst:
kubectl get svcThen:
kubectl describe svc backend-serviceCheck:
Selector
Port
TargetPort
EndpointsService:
selector:
app: backendPod:
labels:
app: node-apiThey don't match.
Therefore:
Service
โ
selector app=backend
โ
No matching Pods
โ
No endpointsCheck:
kubectl get pods --show-labelsand:
kubectl describe svc backend-serviceModern Kubernetes uses EndpointSlices.
Run:
kubectl get endpointslicesOr:
kubectl get endpointslices \
-l kubernetes.io/service-name=backend-serviceIf your Service has no ready backend endpoints, investigate:
Selector
Pod labels
Pod readiness
Port configurationSuppose your Node.js app listens on:
3000Service:
ports:
- port: 80
targetPort: 3000Correct.
Traffic:
Client
โ
Service :80
โ
Pod :3000But if you accidentally write:
targetPort: 8080your Service may route traffic to the wrong port.
Result:
Connection refusedSuppose backend should be reachable at:
http://backend-servicebut your frontend gets:
getaddrinfo ENOTFOUND backend-serviceFirst verify:
kubectl get svcThen test DNS from inside the cluster.
For example:
kubectl run dns-test \
--image=busybox:1.36 \
-it --rm \
--restart=Never \
-- nslookup backend-serviceDepending on the image, nslookup may not be available; you can use an image with DNS tools such as busybox or dnsutils.
Suppose Service:
backend-serviceexists in:
productionYou can use:
backend-service.productionor:
backend-service.production.svc.cluster.localThe standard pattern is:
<service>.<namespace>.svc.cluster.localSuppose:
https://example.comreturns:
404or:
502 Bad GatewayDon't immediately blame the application.
Debug layer by layer:
DNS
โ
Load Balancer
โ
Ingress Controller
โ
Ingress
โ
Service
โ
EndpointSlices
โ
PodsFirst:
kubectl get ingressThen:
kubectl describe ingress <ingress-name>Check:
Host
Path
Backend Service
Backend Port
EventsThen:
kubectl get svcThen:
kubectl get endpointslicesThen:
kubectl get podsThese can give clues, although exact behavior depends on your Ingress Controller.
Potentially:
Wrong host
Wrong path
Ingress routing rule
Controller configurationPotentially:
Service has no healthy endpoints
Backend unreachable
Wrong targetPort
Application unavailableUse actual controller logs and events to confirm.
For example, if using an NGINX-based controller:
kubectl get pods -AFind the controller Pods.
Then:
kubectl logs <ingress-controller-pod> -n <namespace>The exact namespace and controller depend on your installation.
Suppose:
kubectl get pvcshows:
NAME STATUS
postgres-pvc PendingYour Pod may also remain Pending.
Run:
kubectl describe pvc postgres-pvcLook for Events.
Possible causes:
No StorageClass
No suitable provisioner
Invalid access mode
Insufficient storage
Cloud storage problem
Zone/topology constraintsRun:
kubectl get storageclassThen:
kubectl describe storageclass <name>Remember:
PVC
โ
StorageClass
โ
Provisioner
โ
PV
โ
Actual storageCheck:
kubectl get nodesExample:
NAME STATUS
node-1 Ready
node-2 NotReadyInvestigate:
kubectl describe node node-2Look for:
Conditions
Events
Taints
Allocatable resources
CapacityYou may see conditions such as:
Ready
MemoryPressure
DiskPressure
PIDPressure
NetworkUnavailableFor example:
DiskPressure=Truecould indicate a disk-space issue.
This can affect Pods and scheduling.
Use:
kubectl top nodesif metrics are available.
Look at:
CPU
MemoryThen:
kubectl describe node <node>to understand:
Capacity
Allocatable
Allocated resources
Conditions
Taints
PodsSuppose you changed:
image: node-api:2.0but Pods still appear to run the old version.
Check:
kubectl rollout status deployment/node-apiThen:
kubectl get deployment node-apiand:
kubectl get rsThen:
kubectl get podsRun:
kubectl describe deployment node-apiLook for:
Image:You can also inspect:
kubectl get deployment node-api \
-o=jsonpath='{.spec.template.spec.containers[*].image}'Suppose you use:
image: node-api:latestYou build a new image with the same tag.
Kubernetes may not necessarily pull the new image in the way you expect, depending on image pull policy and node cache.
For production deployments, immutable versioned tags are much safer:
node-api:1.0.0
node-api:1.0.1
node-api:1.0.2Even better, deployments can use immutable image digests.
Check:
kubectl rollout status deployment/node-apiMaybe:
Waiting for deployment "node-api" rollout to finish...Check:
kubectl get podsMaybe new Pods are:
0/1 RunningThen:
kubectl describe pod <new-pod>Maybe readiness probe is failing.
Or:
ImagePullBackOffOr:
CrashLoopBackOffThis demonstrates an important point:
A Deployment rollout problem often has a lower-level Pod problem underneath it.
Check:
kubectl rollout history deployment/node-apiIf a new version is broken:
kubectl rollout undo deployment/node-apiThen:
kubectl rollout status deployment/node-apiWe'll study production rollouts and rollback strategies in Chapter 24.
Suppose:
Frontend Pod
โ
Backend Service
โ
TimeoutDon't immediately modify NetworkPolicy.
Test from inside the cluster.
Run a temporary debugging Pod:
kubectl run network-debug \
--image=curlimages/curl \
-it --rm \
--restart=Never \
-- shThen:
curl http://backend-serviceTest a specific port:
curl http://backend-service:80Inside the debugging container:
nslookup backend-serviceThen:
curl http://backend-serviceThis separates:
DNS problemfrom:
Application/network connectivity problemRemember Chapter 20.
Suppose:
Frontend โ Backendused to work.
Then you add:
default-denyand suddenly it fails.
Check:
kubectl get networkpolicy -AThen:
kubectl describe networkpolicy <policy>Ask:
Is frontend allowed to talk to backend?
Is backend port allowed?
Is DNS allowed?
Is egress allowed?
Are namespace selectors correct?NetworkPolicy debugging requires understanding the actual source and destination Pods.
If service DNS doesn't work:
kubectl get pods -n kube-systemLook for your cluster DNS components, commonly CoreDNS.
Then inspect:
kubectl logs -n kube-system <coredns-pod>You can also inspect:
kubectl get svc -n kube-systemLook for the cluster DNS Service.
Suppose your app says:
Authentication failedBut the Pod is Running.
Potentially:
Secret value incorrectCheck that the Secret exists:
kubectl get secretDon't casually print sensitive values into terminal history or logs.
You can inspect metadata:
kubectl describe secret database-secretRemember:
Kubernetes Secrets are not automatically equivalent to a full external secret-management system.
You can inspect environment variables inside a container when appropriate:
kubectl exec -it <pod> -- envBut be careful:
This can expose secrets in your terminal output.
For production secrets, avoid indiscriminate dumping.
Instead, verify configuration through safer targeted checks.
kubectl exec#Sometimes logs aren't enough.
You can enter a running container:
kubectl exec -it <pod> -- /bin/shThen:
lsor:
envor:
cat /etc/resolv.confor:
wget ...depending on available tools.
Your production image may be:
node:alpineor an even smaller hardened image.
It might not contain:
curl
ping
nslookup
bash
netstatSo don't assume every debugging command exists inside your application container.
That's why temporary debug Pods are useful.
Modern Kubernetes environments may support ephemeral containers for troubleshooting running Pods.
Conceptually:
Application Pod
โโโ Node.js container
โโโ Temporary debug containerThis lets you troubleshoot without rebuilding the application image just to add debugging tools.
Availability and permissions depend on your cluster/version/setup.
This is worth saving.
Application broken
โ
โ
kubectl get pods
โ
โโโโโโโโโโโโผโโโโโโโโโโโ
โ โ โ
Pending Running Failed/Crash
โ โ โ
โ โ โ
describe logs logs
โ โ โ
โ โ --previous
Scheduling? Ready? โ
Resources? โ โ
PVC? โ Exit code?
Taints? โ OOMKilled?
โ App error?
โ
Service check
โ
โ
EndpointSlices
โ
โ
DNS / curl
โ
โ
NetworkPolicy
โ
โ
Ingress / DNS
โ
โ
Node / CloudMemorize this table:
| Status | First suspicion | First command |
|---|---|---|
CrashLoopBackOff |
App/container crash | kubectl logs --previous |
OOMKilled |
Memory exceeded | kubectl describe pod |
ImagePullBackOff |
Image/registry issue | kubectl describe pod |
ErrImagePull |
Image pull failed | kubectl describe pod |
Pending |
Scheduling/resource/PVC | kubectl describe pod |
ContainerCreating |
Volume/image/network setup | kubectl describe pod |
CreateContainerConfigError |
Config/Secret dependency | kubectl describe pod |
Running 0/1 |
Readiness failure | kubectl describe pod |
Service no response |
Selector/port/endpoints | kubectl describe svc |
PVC Pending |
Storage provisioning | kubectl describe pvc |
Node NotReady |
Node health/resources | kubectl describe node |
Imagine your company deploys:
Next.js
โ
Ingress
โ
Node.js API
โ
PostgreSQLUsers report:
"The website is loading, but checkout fails."
You investigate.
kubectl get pods -n productionEverything is:
RunningSo don't conclude everything is healthy.
Check API logs:
kubectl logs -n production <api-pod>You find:
Database connection timeoutCheck PostgreSQL Pod:
kubectl get pods -n productionPostgres is:
RunningCheck PostgreSQL Service:
kubectl get svc -n productionExists.
Check endpoints:
kubectl get endpointslices -n productionPostgres has no ready endpoints.
Inspect Postgres:
kubectl describe pod postgres-xxx -n productionYou discover:
Readiness probe failedInspect logs:
kubectl logs postgres-xxx -n productionYou discover:
Database storage is fullNow you have a chain:
Disk full
โ
PostgreSQL unhealthy
โ
Readiness fails
โ
No ready endpoints
โ
Node.js cannot connect
โ
Checkout failsThis is real troubleshooting.
You didn't randomly change the API.
You followed evidence.
When troubleshooting Kubernetes:
Don't jump directly to the solution. Follow the evidence.
Bad troubleshooting:
Pod broken
โ
Increase CPU
โ
Restart
โ
Delete Pod
โ
Hope it worksGood troubleshooting:
Symptom
โ
Status
โ
Events
โ
Logs
โ
Dependencies
โ
Metrics
โ
Root cause
โ
Fix
โ
VerifySuppose you fix the problem.
Don't stop at:
"It seems fixed."
Verify.
kubectl get podsThen:
kubectl rollout status deployment/node-apiThen:
kubectl get svcThen test:
curl ...Then inspect logs:
kubectl logs <pod>And if monitoring exists:
Error rate โ
Latency โ
Restarts stable
Pods ReadyA fix isn't complete until you verify the system recovered.
kubectl logs <pod>
kubectl logs <pod> --previous
kubectl describe pod <pod>Then inspect application errors, exit code, probes, resource limits, and Events.
kubectl describe pod <pod>Look for:
kubectl describe pod <pod>Check:
The container was terminated because it exceeded its memory limit or otherwise encountered an out-of-memory condition.
Investigate actual usage, memory limits, application behavior, and node conditions.
Check:
kubectl get svc
kubectl describe svc <service>
kubectl get endpointslices
kubectl get pods --show-labelsVerify:
selector
Pod labels
port
targetPort
readinessBecause:
Running โ ReadyThe readiness probe may be failing.
Check:
DNS
Ingress
Ingress Controller
Service
EndpointSlices
PodsCommands:
kubectl get ingress
kubectl describe ingress <ingress>
kubectl get svc
kubectl get endpointslices
kubectl logs <ingress-controller>kubectl describe pvc <pvc>
kubectl get storageclassCheck provisioning, StorageClass, access modes, topology and storage backend.
kubectl describe node <node>Check:
Conditions
Events
DiskPressure
MemoryPressure
PIDPressure
kubelet
networkingA very good answer:
kubectl get
kubectl describe
kubectl logsThen use Events, metrics, Services, EndpointSlices and network debugging depending on the symptom.
kubectl get pods
kubectl describe pod <pod>
kubectl logs <pod>
kubectl logs <pod> --previouskubectl get events --sort-by=.lastTimestampkubectl get deployment
kubectl describe deployment <deployment>
kubectl rollout status deployment/<deployment>
kubectl rollout history deployment/<deployment>kubectl get svc
kubectl describe svc <service>
kubectl get endpointsliceskubectl top pods
kubectl top nodeskubectl get nodes
kubectl describe node <node>kubectl get pvc
kubectl describe pvc <pvc>
kubectl get storageclasskubectl run network-debug \
--image=curlimages/curl \
-it --rm \
--restart=Never \
-- shThen:
curl http://service-nameWhen your Kubernetes application breaks:
SOMETHING BROKE
โ
โ
kubectl get
โ
โ
What's wrong?
โ
โโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโ
โ โ โ
Pod issue Service issue Node issue
โ โ โ
โ โ โ
describe describe describe
โ โ โ
โ โ โ
logs endpoints resources
โ โ โ
โ โ โ
--previous DNS/curl kubelet/events
โ โ โ
โโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโ
โ
Metrics
โ
Root Cause
โ
Fix
โ
VerifyIf you become comfortable with this flow, you'll be able to troubleshoot a surprisingly large percentage of everyday Kubernetes incidents.
Remember these 10 rules:
kubectl get โ understand the current state.kubectl describe โ inspect details and Events.kubectl logs โ inspect application output.--previous โ inspect the previous crashed container.CrashLoopBackOff โ find why the application is crashing.Pending โ investigate scheduling/resources/storage/constraints.ImagePullBackOff โ investigate image/registry/authentication.Running โ Ready โ check readiness probes.We'll now move from troubleshooting into production deployment strategies.
You'll learn:
maxSurgemaxUnavailableMark it as complete to track your mastery.