Warrior, in Chapter 21 we learned Logging.
Now we move to the second major pillar of observability:
Metrics — numerical measurements that tell us how the system is performing.
A simple way to remember:
Logs → What happened?
Metrics → How much / how often?
Traces → Where did the request spend time?Monitoring means continuously collecting measurements from your system and using them to understand its health and performance.
For a Kubernetes application, you might want to know:
CPU usage
Memory usage
Pod count
Restart count
Request rate
Error rate
Response latency
Network traffic
Disk usage
Node healthFor example:
node-api
────────────────────────────
CPU 72%
Memory 61%
Requests 2,450/sec
Errors 12/sec
P95 latency 180 ms
Pods 6/10This gives you a much faster understanding of system health than reading individual logs.
Imagine your API is slow.
You search logs and find:
Request received
Request completedNothing obviously wrong.
But metrics show:
CPU
100% ──────────────────────
80% ──────────────────────
60% ───────────────
40%
20%
10AM 11AMYou immediately know:
CPU utilization increased significantly around 11 AM.
Another example:
Requests: 10,000/min
Errors: 800/minThat means your error rate is roughly:
800 / 10,000 = 8%Metrics make this kind of system-level behavior easy to see.
You should now start thinking about observability as:
OBSERVABILITY
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Logs Metrics Traces
│ │ │
Events Numbers Request flow
│ │ │
"What?" "How much?" "Where?"Example:
Database query failedDatabase error rate = 7%API → PostgreSQL
↓
2.4 secondsTogether they give you a much better picture.
This needs an important clarification.
Kubernetes itself exposes information about the cluster, but commands such as:
kubectl top pods
kubectl top nodesnormally require a metrics provider such as Metrics Server.
So if you run:
kubectl top podsand get something like:
error: Metrics API not availableit often means Metrics Server isn't available/configured.
Metrics Server collects resource usage data from Kubernetes nodes/kubelets and exposes it through the Kubernetes Metrics API.
Conceptually:
Kubernetes Cluster
│
┌────────────┴────────────┐
↓ ↓
Node 1 Node 2
│ │
kubelet kubelet
│ │
└────────────┬────────────┘
↓
Metrics Server
│
↓
Metrics API
│
↓
kubectl topIt is mainly useful for basic resource metrics and autoscaling use cases.
It is not a complete observability platform.
kubectl top#Once a metrics provider is available:
kubectl top nodesmight show:
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
node-1 850m 42% 3.2Gi 40%
node-2 1200m 60% 5.1Gi 64%This tells you current resource usage.
kubectl top podsExample:
NAME CPU MEMORY
node-api-7d8f6b4c9-x2k7m 120m 250Mi
node-api-7d8f6b4c9-q8l4p 450m 600Mi
redis-5d7f8c9d7-k2m4x 80m 180MiImmediately you can see:
One Node.js Pod is consuming significantly more CPU/memory than another.
Suppose your application is in:
productionUse:
kubectl top pods -n productionYou can also use:
kubectl top pods -Ato inspect Pods across namespaces.
Remember Chapter 13.
You learned:
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"Now monitoring lets you compare actual usage.
Example:
CPU Request: 250m
CPU Usage: 220m
CPU Limit: 500mSo:
Usage / Request
= 220 / 250
= 88%This can matter for HPA because CPU utilization targets are commonly calculated relative to CPU requests.
Suppose:
resources:
requests:
cpu: "500m"Actual usage:
400mThen utilization is approximately:
400 / 500 = 80%If HPA target is:
70%HPA may want more replicas.
But if your request were:
1000mthe same 400m usage would be:
400 / 1000 = 40%So resource requests affect autoscaling behavior.
That's why:
Bad resource requests can produce bad autoscaling decisions.
This distinction is extremely important for interviews.
Primarily provides:
CPU
Memoryresource metrics for:
kubectl topProvides a much broader metrics collection and monitoring system.
It can collect:
CPU
Memory
Request rate
Error rate
Latency
Database metrics
Redis metrics
Application metrics
Business metrics
Kubernetes metrics
Custom metricsThink:
Metrics Server
↓
Basic Kubernetes resource metrics
Prometheus
↓
Full monitoring/metrics platformPrometheus is a popular open-source monitoring and alerting system.
Its main job is:
Collect and store time-series metrics.
Time-series means:
metric + value + timestampFor example:
cpu_usage = 72%
timestamp = 10:30Then:
10:30 → 72%
10:31 → 75%
10:32 → 81%
10:33 → 90%Now you can graph the metric.
Basic architecture:
Kubernetes
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Node Exporter API Server App Metrics
│ │ │
└─────────────┼─────────────┘
↓
Prometheus
│
┌──────┴──────┐
↓ ↓
Queries Alerts
│
↓
GrafanaOne of the most important concepts:
Prometheus generally pulls/scrapes metrics from targets.
For example:
Prometheus
│
│ GET /metrics
↓
Node.js applicationThe application exposes:
/metricsPrometheus periodically fetches it.
This is different from simply having your application push every metric directly to Prometheus.
/metrics#Imagine your Node.js application exposes:
http://node-api:3000/metricsIt might return:
http_requests_total 15420
http_request_errors_total 321
process_cpu_seconds_total 1832Prometheus scrapes this endpoint.
A popular Node.js library is:
prom-clientExample:
import client from "prom-client";
const register = new client.Registry();
client.collectDefaultMetrics({
register
});
app.get("/metrics", async (req, res) => {
res.set("Content-Type", register.contentType);
res.end(await register.metrics());
});Now:
GET /metricsreturns Prometheus-compatible metrics.
A metric is a measurable value.
Common application metrics:
http_requests_totalhttp_errors_totalhttp_request_duration_secondshttp_requests_in_progressdatabase_connectionsPrometheus commonly works with four metric types:
Counter
Gauge
Histogram
SummaryLet's understand each.
A Counter is a value that generally only increases.
Example:
HTTP requests100
101
102
103
...It can reset when the application restarts.
Examples:
http_requests_total
errors_total
orders_created_totalExample:
const requests = new client.Counter({
name: "http_requests_total",
help: "Total HTTP requests"
});Gauge can go:
up
down
up
downExample:
Active users
Memory usage
Queue length
TemperatureExample:
const activeUsers = new client.Gauge({
name: "active_users",
help: "Current active users"
});If:
100 → 120 → 90 → 140that's perfectly valid for a Gauge.
Histogram is very important for application performance.
Suppose you want to measure:
How long do API requests take?
You might see:
50ms
80ms
100ms
150ms
300ms
2sA Histogram groups observations into buckets.
Example:
request duration
≤ 0.1s → 800 requests
≤ 0.5s → 950 requests
≤ 1s → 990 requests
≤ 5s → 1000 requestsThis is extremely useful for latency analysis.
These are extremely important in production systems.
Suppose 100 requests were measured.
50% of requests completed within this time.
Example:
P50 = 100ms95% completed within:
P95 = 400ms99% completed within:
P99 = 1.2sThis means a small percentage of users are experiencing much slower responses.
Imagine:
99 requests = 100ms
1 request = 10 secondsAverage:
~199msLooks reasonable.
But one user waited:
10 secondsPercentiles reveal this much better.
You may see:
P50 = 100ms
P95 = 110ms
P99 = 10sThat's much more informative.
Metrics can have labels.
Example:
http_requests_total{
method="GET",
route="/users",
status="200"
}Another:
http_requests_total{
method="POST",
route="/orders",
status="500"
}Now you can query:
all POST requestsor:
all 500 errorsor:
/orders requestsThis is a critical production concept.
Bad:
userId="123456"as a Prometheus label for every user.
Why?
Suppose:
10 million usersYou could create millions of unique time series.
That can become extremely expensive.
Avoid labels such as:
userId
requestId
sessionId
email
random UUIDin most Prometheus metrics.
Better:
method
route
status
service
environmentFor example:
http_requests_total{
method="GET",
route="/users/:id",
status="200"
}This is where Kubernetes becomes interesting.
You don't want to manually configure every Pod.
Prometheus can discover Kubernetes resources.
Conceptually:
Kubernetes API
│
↓
Prometheus
│
↓
Discover targets
│
↓
Scrape metricsFor example:
Pod A
Pod B
Pod CPrometheus can discover them according to its configuration.
If you're using the Prometheus Operator / kube-prometheus-stack, you'll commonly encounter:
ServiceMonitor
It tells Prometheus how to discover and scrape metrics exposed by a Kubernetes Service.
Conceptually:
ServiceMonitor
│
↓
Service
│
↓
Pods
│
↓
/metrics
│
↓
PrometheusExample:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: node-api
spec:
selector:
matchLabels:
app: node-api
endpoints:
- port: metrics
path: /metricsThe exact labels and port configuration depend on your Prometheus Operator setup.
Imagine:
Node.js API
│
└── /metrics
│
↓
Service
│
↓
Prometheus
│
↓
GrafanaNow Grafana can display:
Requests/sec
Errors/sec
P95 latency
CPU
Memory
Pod countPrometheus stores and queries metrics.
Grafana is commonly used to visualize them.
Think:
Prometheus
↓
Data
↓
Grafana
↓
DashboardExample dashboard:
┌─────────────────────────────────────────────┐
│ Node API Production │
├─────────────┬─────────────┬─────────────────┤
│ Requests/s │ Error Rate │ P95 Latency │
│ 2,340 │ 0.8% │ 180ms │
├─────────────┴─────────────┴─────────────────┤
│ CPU Usage │
│ ▁▂▃▃▄▅▆▇▇█ │
├─────────────────────────────────────────────┤
│ Memory Usage │
│ ▂▂▃▃▄▅▅▆▆▇ │
└─────────────────────────────────────────────┘You may want metrics like:
CPU
Memory
Disk
Network
Load
FilesystemA common component is:
Node Exporter
It runs on nodes and exposes machine-level metrics.
Typically it is deployed as a DaemonSet.
Node 1 → Node Exporter
Node 2 → Node Exporter
Node 3 → Node ExporterPrometheus scrapes these exporters.
An exporter converts information into Prometheus-readable metrics.
For example:
Node
↓
Node Exporter
↓
/metrics
↓
PrometheusOther exporters exist for systems such as:
A realistic architecture:
Kubernetes
│
┌─────────────────────┼─────────────────────┐
↓ ↓ ↓
Node Exporter kube-state-metrics App /metrics
│ │ │
└─────────────────────┼─────────────────────┘
↓
Prometheus
│
┌─────────┴─────────┐
↓ ↓
Grafana Alertmanager
│ │
↓ ↓
Dashboards AlertsThis is an extremely common architecture.
This one is worth understanding.
Node Exporter tells you about the machine.
kube-state-metrics tells you about the state of Kubernetes objects.
For example:
Deployment replicas
Pod status
DaemonSets
StatefulSets
Jobs
CronJobsExample question:
Deployment wants 10 replicas. How many are actually available?
Kubernetes object-state metrics can help answer that.
Monitoring isn't useful if humans have to stare at Grafana all day.
You want alerts.
For example:
CPU > 90% for 10 minutesor:
Error rate > 5%or:
Pod restart count increasingor:
P95 latency > 1 secondArchitecture:
Prometheus
│
↓
Alert Rule
│
↓
Alertmanager
│
├── Slack
├── Email
├── PagerDuty
└── Other integrationsConceptually:
groups:
- name: node-api
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 5
for: 10m
labels:
severity: critical
annotations:
summary: "High API error rate"This says roughly:
If the matching error metric remains above the threshold for 10 minutes, fire an alert.
The exact expression must match your actual metric schema.
Don't create alerts for everything.
Bad:
CPU = 70%
→ ALERT!Maybe that's normal.
Better:
CPU > 90%
AND
sustained for 10 minutesOr more meaningful:
5xx error rate > 5%
for 5 minutesGood alerts should generally represent something that requires action.
A very useful monitoring concept is the Four Golden Signals:
Latency
Traffic
Errors
SaturationHow long requests take.
P95 = 300msHow much demand exists.
5,000 requests/secHow many requests fail.
2% HTTP 5xxHow close the system is to capacity.
CPU 95%
Memory 90%These are excellent starting points for application monitoring.
For request-driven services, another useful model is:
R → Rate
E → Errors
D → DurationFor your Node.js API:
Rate → requests/sec
Errors → 5xx/sec
Duration → P95 latencyExample:
Rate: 3,200 req/s
Errors: 1.2%
P95: 250msFor infrastructure, another useful framework is:
U → Utilization
S → Saturation
E → ErrorsFor a Kubernetes node:
CPU utilization
Memory saturation
Network errors
Disk errorsThese frameworks help you decide what to monitor, rather than randomly collecting thousands of metrics.
For your typical architecture:
Next.js
↓
Ingress
↓
Node.js API
├── PostgreSQL
├── Redis
└── External APIsI'd want at least:
Request rate
Error rate
P50 latency
P95 latency
P99 latency
Active requestsCPU
Memory
Heap usage
Event loop behavior
GC behaviorConnections
Query latency
Errors
Connection pool usageMemory
Commands/sec
Latency
Connections
ErrorsPod count
Restart count
CPU
Memory
Pending Pods
Node capacity
Deployment availabilityRemember Chapter 15.
Traffic
↓
CPU increases
↓
HPA
↓
More PodsMonitoring helps you understand whether that actually happened.
Example:
10:00
Pods = 2
CPU = 40%
10:05
Traffic increases
10:06
CPU = 78%
10:07
HPA increases Pods
Pods = 4
10:10
CPU = 45%Grafana can show this entire story visually.
This is a common interview topic.
For standard resource-based HPA:
HPA
↓
Metrics API
↓
Metrics Server
↓
CPU / MemoryPrometheus is different:
Application
↓
Prometheus
↓
Custom metrics / monitoringWith the appropriate adapters/integrations, Prometheus-derived metrics can also be used for autoscaling.
So don't say:
HPA always uses Prometheus.
That's incorrect.
Let's make the distinction crystal clear.
| Logging | Monitoring |
|---|---|
| Events | Measurements |
| Text/structured records | Time-series numbers |
Database failed |
Error rate = 5% |
User login failed |
Login failure rate |
Redis timeout |
Redis latency |
kubectl logs |
Prometheus/Grafana |
Example:
LOG:
Database query failed
timeout after 3000msMetric:
database_query_errors_total = 152Dashboard:
DB error rate
│
10% ────────╮
5% ──────╮ │
0% ───────╰─╯Imagine your production API becomes slow.
You see:
P95 latency
200ms → 3 seconds1,000 req/s → 5,000 req/sCPU
40% → 95%2 Pods → 10 PodsYou find:
Database connection pool exhaustedConnections
100 → 500Now you've connected:
Metrics
↓
Logs
↓
Root cause investigationThis is what observability is about.
Eventually, your course project will look something like:
Internet
│
↓
Ingress
│
┌──────────────┴──────────────┐
↓ ↓
Next.js Service Node.js API Service
│ │
Pods × N Pods × N
│
┌────────────┼────────────┐
↓ ↓ ↓
Redis PostgreSQL External APIs
OBSERVABILITY
│
┌────────────┼────────────┐
↓ ↓ ↓
Logs Metrics Traces
│ │
Fluent Bit Prometheus
│ │
Loki Grafana
│ │
└────────────┴───────→ Dashboardkubectl top nodeskubectl top podskubectl top pods -n productionkubectl top pods -Akubectl get podskubectl get deploymentskubectl get hpakubectl describe hpa node-api-hpaWhen something looks wrong:
kubectl get nodesThen:
kubectl top nodesThen:
kubectl get pods -AThen:
kubectl top pods -AThen:
kubectl get hpa -AThen inspect problematic Pods:
kubectl describe pod <pod>And logs:
kubectl logs <pod>This gives you a basic operational workflow.
A Kubernetes component that provides basic resource usage metrics, commonly used by kubectl top and resource-based HPA.
Metrics Server
→ basic CPU/memory resource metrics
Prometheus
→ general-purpose time-series monitoringAn open-source monitoring and alerting system that collects and stores time-series metrics.
Prometheus primarily uses a pull/scrape model.
/metricsA visualization and dashboarding platform commonly used to query and display metrics from Prometheus and other data sources.
A metric that generally increases over time.
Example:
http_requests_totalA metric that can increase or decrease.
Example:
active_connectionsA metric type useful for observing distributions such as request latency.
The value below which approximately 95% of observations fall.
It helps identify tail latency and user experiences that an average can hide.
A metric label creating a very large number of unique time series.
For example:
userId
requestId
sessionIdcan create problematic cardinality when used indiscriminately as labels.
A Prometheus exporter that exposes host/node-level system metrics.
A component that exposes metrics representing the state of Kubernetes objects.
Latency
Traffic
Errors
SaturationMonitoring
│
├── Metrics Server
│ └── CPU / Memory
│
├── Prometheus
│ └── Time-series metrics
│
├── Grafana
│ └── Dashboards
│
├── Alertmanager
│ └── Notifications
│
├── Node Exporter
│ └── Node metrics
│
└── kube-state-metrics
└── Kubernetes object statekubectl top
↓
Basic resource usage
Prometheus
↓
Metrics collection/storage/query
Grafana
↓
Visualization
Alertmanager
↓
NotificationsYour production Kubernetes system should eventually look like:
APPLICATION
│
┌───────────────┼───────────────┐
↓ ↓ ↓
Logs Metrics Traces
│ │ │
↓ ↓ ↓
Fluent Bit Prometheus OpenTelemetry
│ │ │
↓ ↓ ↓
Loki Grafana Tempo/Jaeger
│ │ │
└───────────────┼───────────────┘
↓
OBSERVABILITYAnd when an incident happens:
Grafana
↓
"What changed?"
↓
Prometheus metrics
↓
"Where is the problem?"
↓
Logs
↓
"What exactly happened?"
↓
Traces
↓
"Which component is slow?"That is the foundation of production-grade Kubernetes observability.
This is one of the most important Kubernetes chapters for interviews and real production work.
We'll learn how to systematically debug:
CrashLoopBackOffImagePullBackOffPendingOOMKilledContainerCreatingCreateContainerConfigErrorErrImagePullReadiness probe failedPendingAnd we'll build a Kubernetes troubleshooting decision tree you can use in real projects.
Mark it as complete to track your mastery.