Warrior, now we enter Observability.
Until now, we learned how Kubernetes runs and manages applications. Now we need to answer:
“My Pod is running, but my application is not working. How do I find out what happened?”
The first tool is logging.
A log is a record of something that happened inside your application.
For example, your Node.js API may generate:
Server started on port 3000
User login request received
Database connection established
Payment request failed
Redis connection timeoutLogs help us understand:
Think of it like this:
Kubernetes Cluster
│
┌──────────────┼──────────────┐
│ │ │
Pod A Pod B Pod C
│ │ │
Node.js API Node.js API Next.js
│ │ │
stdout stdout stdout
│ │ │
└──────────────┼──────────────┘
↓
Container Runtime
↓
Node-level log files
↓
Logging Agent
↓
┌──────────────┼──────────────┐
↓ ↓ ↓
Loki Elasticsearch Cloud LogsThe important idea:
Your application should generally write logs to stdout/stderr, and the platform handles collecting them.
This is extremely important.
Linux processes commonly have:
stdout → normal output
stderr → errorsFor example:
console.log("Server started");goes to stdout.
And:
console.error("Database connection failed");goes to stderr.
In Kubernetes, these streams are captured by the container runtime and made available through Kubernetes logging mechanisms.
kubectl logs#Suppose you have:
node-api-7d9f8c6f5b-x7k2mYou can see its logs:
kubectl logs node-api-7d9f8c6f5b-x7k2mExample:
Server started on port 3000
Database connected
Redis connected
GET /users
GET /users/123Similar to:
tail -fyou can use:
kubectl logs -f node-api-7d9f8c6f5b-x7k2m-f means:
Follow the log stream.
Now when new requests arrive:
GET /users
POST /login
GET /productsyou'll see them immediately.
This is extremely useful during debugging.
Suppose:
kubectl get podsreturns:
node-api-6d4f7c8d9b-abc12
node-api-6d4f7c8d9b-def34
node-api-6d4f7c8d9b-ghi56You can select the Pods using labels.
For example:
kubectl logs -l app=node-apiThis is useful when you have multiple replicas.
You can also use:
kubectl logs deployment/node-apiDepending on the situation, Kubernetes will retrieve logs from a Pod associated with the Deployment; for multiple replicas, label-based selection is often more useful for examining all instances.
A Pod can contain multiple containers.
Example:
Pod
│
├── node-api
│
└── logging-sidecarThen:
kubectl logs my-pod -c node-apiAnd:
kubectl logs my-pod -c logging-sidecarThe -c option specifies the container.
Example:
apiVersion: v1
kind: Pod
metadata:
name: app-pod
spec:
containers:
- name: api
image: my-api:1.0
- name: sidecar
image: fluent-bit:latestNow:
kubectl logs app-podmay not be enough because Kubernetes needs to know which container you mean.
Use:
kubectl logs app-pod -c apior:
kubectl logs app-pod -c sidecar--previous#Imagine your Node.js application crashes.
Kubernetes restarts it.
You run:
kubectl logs my-podBut the logs you're looking for aren't there because you're seeing the current container instance.
Use:
kubectl logs my-pod --previousThis shows logs from the previous terminated container instance.
Very useful for:
CrashLoopBackOff
OOMKilled
Application crash
Unhandled exception
Startup failureSuppose your application contains:
throw new Error("Database connection failed");Container crashes.
You may see:
Error: Database connection failed
at connectDatabase (/app/db.js:24:11)
at startServer (/app/server.js:12:3)Then Kubernetes restarts the container.
You can inspect:
kubectl logs my-pod --previousThis is a very important distinction.
kubectl logs#Answers:
What did the application/container print?
kubectl logs my-podkubectl describe#Answers:
What is Kubernetes saying about this Pod?
kubectl describe pod my-podFor example:
Events:
Warning BackOff
Warning Unhealthy
Normal Pulled
Normal StartedSo:
Application problem
↓
kubectl logs
Kubernetes problem
↓
kubectl describeIn practice, you often use both.
kubectl logs Useful Options#kubectl logs my-pod --tail=100kubectl logs my-pod --tail=20kubectl logs my-pod --since=1hkubectl logs my-pod --since=10mkubectl logs my-pod --timestampskubectl logs -f my-podkubectl logs my-pod --previouskubectl logs my-pod -c apiYou can combine options:
kubectl logs my-pod \
-c api \
--previous \
--timestampsSuppose your application is in:
productionThen:
kubectl logs my-podmight fail because you're currently looking in another namespace.
Use:
kubectl logs -n production my-podFor example:
kubectl logs -n production node-api-7d9f8c6f5b-x7k2mThis is a very important production concept.
Many beginners think:
Kubernetes is my logging system.
Not exactly.
Kubernetes provides mechanisms to access container logs, but production systems commonly use a centralized logging system.
Why?
Imagine:
Node 1
├── Pod A logs
└── Pod B logs
Node 2
├── Pod C logs
└── Pod D logs
Node 3
├── Pod E logs
└── Pod F logsIf you have 100 nodes and 2,000 Pods, manually checking:
kubectl logsisn't practical.
You need centralized logging.
Typical architecture:
Application
│
↓
stdout/stderr
│
↓
Container Runtime
│
↓
Node
│
↓
Logging Agent
│
↓
Central Log System
│
↓
Dashboard / SearchExamples of central log systems:
One popular log collector is:
Fluent Bit
Think of Fluent Bit as:
A small agent that collects logs and forwards them somewhere else.
Architecture:
Pod
│
↓
stdout
│
↓
Node log
│
↓
Fluent Bit
│
├────────→ Elasticsearch
│
├────────→ Loki
│
├────────→ CloudWatch
│
└────────→ OpenSearchFluent Bit is commonly deployed as a DaemonSet.
Why DaemonSet?
Because we want one logging agent on each node.
Node 1 → Fluent Bit
Node 2 → Fluent Bit
Node 3 → Fluent Bit
Node 4 → Fluent BitRemember our earlier chapters.
A DaemonSet ensures a Pod runs on each eligible node.
So:
DaemonSet
│
├── Fluent Bit → Node 1
├── Fluent Bit → Node 2
├── Fluent Bit → Node 3
└── Fluent Bit → Node 4Each Fluent Bit instance collects logs from its node.
One famous logging architecture is:
Stores and searches logs.
Processes/transforms logs.
Provides UI for searching and visualizing logs.
Architecture:
Kubernetes Pods
│
↓
Fluent Bit
│
↓
Logstash
│
↓
Elasticsearch
│
↓
KibanaThis is called:
ELK Stack
Another architecture:
Kubernetes
↓
Fluent Bit
↓
Elasticsearch
↓
KibanaThis is commonly referred to as:
EFK
The exact components can vary; Fluent Bit can often handle collection and some processing without Logstash.
Another popular modern approach:
Kubernetes
↓
Fluent Bit / Promtail / Alloy
↓
Loki
↓
GrafanaLoki is designed specifically for log aggregation and works particularly well with Grafana-based observability stacks.
Conceptually:
Logs
↓
Loki
↓
GrafanaYou can search logs and correlate them with metrics.
This is extremely important for production Node.js applications.
Avoid logs like:
User login failedPrefer structured JSON:
{
"level": "error",
"message": "User login failed",
"userId": "12345",
"service": "auth-service",
"environment": "production"
}Why?
Because machines can parse JSON easily.
You can search:
service = auth-service
level = error
userId = 12345A production Node.js application commonly uses a logging library such as:
For high-throughput Node.js services, Pino is a popular choice.
Example:
import pino from "pino";
const logger = pino();
logger.info({
service: "node-api",
environment: "production"
}, "Server started");
logger.error({
userId: "123",
errorCode: "DB_CONNECTION_FAILED"
}, "Database connection failed");The output can be JSON:
{
"level": 30,
"time": 1780000000000,
"service": "node-api",
"msg": "Server started"
}Your Kubernetes logging pipeline can then collect this JSON.
Common log levels:
| Level | Meaning |
|---|---|
trace |
Extremely detailed |
debug |
Debugging information |
info |
Normal application events |
warn |
Something unusual |
error |
Failure/error |
fatal |
Critical failure |
Example:
logger.debug("Cache lookup");
logger.info("User logged in");
logger.warn("Slow database query");
logger.error("Payment failed");In production, you usually don't want extremely verbose debug logging everywhere.
This is extremely important.
Never casually log:
password
JWT token
API key
credit card number
private key
database passwordBad:
console.log({
username,
password
});Also dangerous:
logger.info({
authorization: req.headers.authorization
});Because you might expose bearer tokens.
Instead:
logger.info({
userId,
action: "login"
}, "User login attempt");Now we reach an important production concept.
Imagine a user sends:
POST /checkoutThe request goes through:
Frontend
↓
API Gateway
↓
Node.js API
↓
Payment Service
↓
DatabaseEach service produces logs.
How do we know which logs belong to the same request?
Use a:
Correlation ID / Request ID
Example:
requestId = abc-123Then:
{
"requestId": "abc-123",
"service": "api",
"message": "Checkout started"
}Payment service:
{
"requestId": "abc-123",
"service": "payment",
"message": "Payment initiated"
}Database service:
{
"requestId": "abc-123",
"service": "database",
"message": "Transaction committed"
}Now you can search:
requestId = abc-123and reconstruct the request flow.
Let's imagine your application:
Next.js Frontend
│
↓
Ingress
│
↓
Node.js API
│
┌───┴────┐
↓ ↓
Redis PostgreSQLRunning in Kubernetes:
Internet
│
↓
Ingress
│
↓
┌─────────────────┐
│ Node.js API │
│ │
│ Pod 1 │
│ Pod 2 │
│ Pod 3 │
└────────┬────────┘
│
stdout/stderr
│
↓
Node log files
│
↓
Fluent Bit
│
┌────────┴────────┐
↓ ↓
Loki Elasticsearch
│ │
└────────┬────────┘
↓
GrafanaFor a Node.js API:
Server started
Database connected
Redis connectedGET /users
POST /ordersOrder created
Payment initiated
Email sentDatabase query failed
Redis timeout
External API failedDatabase query took 850ms
External API took 1.2sAvoid:
Passwords
JWT tokens
API keys
Credit card numbers
Session cookies
Private keys
Personal data unnecessarilyUse:
userId
requestId
orderId
traceId
errorCode
service
endpoint
durationwhere appropriate.
For Next.js, logging depends on where code executes.
For server-side code:
console.log("Server request received");will appear in the server/container logs.
But logs from browser-side code:
console.log("Button clicked");appear in the user's browser console, not your Kubernetes container logs.
This distinction is important:
Next.js Server
↓
Kubernetes logs
Next.js Browser
↓
Browser DevToolsDon't confuse them.
Your code produces:
logger.error(...)Kubernetes/container runtime makes those container logs accessible:
kubectl logsA logging system stores/searches them:
Loki
Elasticsearch
Cloud Logging
etc.So:
Application
↓
stdout/stderr
↓
Kubernetes/container runtime
↓
Log collector
↓
Centralized storage
↓
DashboardSuppose:
kubectl get podsshows:
node-api-abc123 0/1 CrashLoopBackOffWhat should you do?
kubectl logs node-api-abc123If the current container isn't useful:
kubectl logs node-api-abc123 --previousThen:
kubectl describe pod node-api-abc123Look at:
EventsThen:
Check status:
kubectl get pod node-api-abc123You may discover:
Last State:
Terminated
Reason: OOMKilledNow you know:
The application/container exceeded its memory limit.
This connects Chapter 13 with Chapter 21.
Suppose:
Pod: Running
Ready: 0/1Check:
kubectl logs my-podMaybe:
Database connection timeoutThen:
kubectl describe pod my-podMaybe readiness probe is failing.
So the application is running, but Kubernetes doesn't consider it ready for traffic.
Remember:
Running ≠ ReadyImagine your application produces:
10 GB logs/dayIf you keep everything forever on a node, disk usage can become a problem.
Production logging systems therefore need:
For example:
Hot logs → 7 days
Archived logs → 30 days
Delete → after retention periodExact retention depends on operational and regulatory requirements.
This is an important engineering principle.
More logs ≠ better observability.
Suppose:
1 request
↓
100 log linesWith:
100,000 requests/secyou could generate enormous log volume.
Instead, prefer useful structured events.
For example:
{
"level": "info",
"requestId": "abc123",
"route": "/checkout",
"statusCode": 200,
"durationMs": 145
}Much more useful than printing huge objects everywhere.
Now we are approaching the bigger concept:
Observability
Three major pillars:
Observability
│
┌──────────┼──────────┐
↓ ↓ ↓
Logs Metrics Traces
│ │ │
Events Numbers RequestsTell you:
What happened?
Tell you:
How much / how often?
Tell you:
Where did this request spend its time?
Example:
Request abc123
│
├── API: 20ms
│
├── Redis: 5ms
│
├── PostgreSQL: 80ms
│
└── Payment API: 500msWe'll study metrics in Chapter 22 and later connect this to tracing/observability.
kubectl get podskubectl logs <pod>kubectl logs -f <pod>kubectl logs <pod> --previouskubectl logs <pod> -c <container>kubectl logs <pod> --tail=100kubectl logs <pod> --since=10mkubectl logs <pod> --timestampskubectl logs -n production <pod>kubectl logs -l app=node-apikubectl describe pod <pod>kubectl logs <pod> -c api --previousMemorize this:
Application issue
│
↓
kubectl get pods
│
↓
Is Pod Running?
│
┌───┴───┐
No Yes
│ │
↓ ↓
describe logs
pod │
│ ↓
│ error?
│ │
│ ↓
│ --previous
│
↓
EventsA more practical workflow:
kubectl get pods -n production
kubectl logs -n production <pod>
kubectl logs -n production <pod> --previous
kubectl describe pod -n production <pod>Then inspect:
Status
Events
Last State
Restart Count
Probe failures
Resource failureskubectl logs <pod>kubectl logs -f <pod>kubectl logs <pod> --previouskubectl logs <pod> -c <container>Generally:
stdout / stderrrather than relying on application-specific files inside ephemeral container filesystems.
Because large Kubernetes clusters can have:
100s of nodes
1000s of Pods
millions/billions of log eventsCentralized logging allows:
A lightweight log collector/forwarder commonly deployed as a DaemonSet.
Elasticsearch
Logstash
KibanaCommonly:
Elasticsearch
Fluent Bit
KibanaBecause JSON logs are easier for machines to:
parse
search
filter
aggregate
analyzeA unique identifier used to connect logs/events belonging to the same request across services.
kubectl logs give centralized logging?#No.
It gives access to container logs through Kubernetes.
Centralized logging requires an additional logging pipeline/system.
stdout/stderr
↓
Container Runtime
↓
Node
↓
Log Collector
↓
Centralized Logging
↓
Search / Dashboardkubectl logs POD
kubectl logs -f POD
kubectl logs POD --previous
kubectl logs POD -c CONTAINER
kubectl logs POD --tail=100
kubectl logs POD --since=10m
kubectl logs POD --timestamps
kubectl logs -l app=node-apiFluent Bit
Fluentd
Loki
Elasticsearch
OpenSearch
CloudWatch
Cloud Logging
Splunk
Grafana
KibanaLogs → What happened?
Metrics → How much/how often?
Traces → Where did request time go?If your Node.js application is running inside Kubernetes:
Node.js
│
│ console.log / logger.info
│ console.error / logger.error
↓
stdout / stderr
↓
Container Runtime
↓
Kubernetes Node
↓
Fluent Bit / Collector
↓
Loki / Elasticsearch / Cloud Logging
↓
Grafana / KibanaAnd when something breaks:
kubectl logs
↓
kubectl logs --previous
↓
kubectl describe pod
↓
Events + application errorThat is the basic Kubernetes logging workflow.
We'll learn the second major pillar of observability:
kubectl topAnd we'll build the mental model:
Logs → What happened?
Metrics → How is the system performing?
Traces → Where did the request spend time?Mark it as complete to track your mastery.