Warrior, now we reach one of the most important Kubernetes internals.
You already know:
But one critical question remains:
When a Pod is created, how does Kubernetes decide which worker node should run it?
The answer is:
The kube-scheduler is a control-plane component responsible for selecting a suitable node for Pods that don't already have a node assigned.
Think of it as a:
Job placement system
For example:
Pod
│
▼
kube-scheduler
│
┌───────┼────────┐
▼ ▼ ▼
Node 1 Node 2 Node 3
❌ ✅ ❌
│
▼
Pod runsThe scheduler considers things such as:
Suppose you create:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
containers:
- name: app
image: nginxThe flow is:
kubectl apply
│
▼
API Server
│
▼
Pod created
│
▼
Pod has no node assigned
│
▼
Scheduler
│
▼
Find suitable nodes
│
▼
Select node
│
▼
Bind Pod to node
│
▼
kubelet
│
▼
Container startsInitially, a Pod might look like:
NAME READY STATUS NODE
my-app 0/1 Pending <none>Notice:
NODE = <none>That means the scheduler hasn't assigned it to a node yet.
After scheduling:
NAME READY STATUS NODE
my-app 1/1 Running node-2Now:
NODE = node-2Beginners often think:
Pod → random nodeThat's incorrect.
The scheduler evaluates nodes against constraints.
Conceptually:
Pod
│
▼
Can Node 1 run it?
│
├── No
│
Can Node 2 run it?
│
├── Yes
│
Can Node 3 run it?
│
├── Yes
│
▼
Rank suitable nodes
│
▼
Choose a nodeA useful mental model is:
Filtering
↓
Scoring
↓
SelectionRemove nodes that cannot run the Pod.
Example:
5 Nodes
Node 1 → insufficient CPU ❌
Node 2 → wrong architecture ❌
Node 3 → nodeSelector mismatch ❌
Node 4 → suitable ✅
Node 5 → suitable ✅Remaining:
Node 4
Node 5Among feasible nodes, Kubernetes considers scheduling preferences and scores candidates.
Node 4 → score 70
Node 5 → score 85Node 5 may be selected.
The actual scheduler framework has many plugins and details, but this filter → score → select model is an excellent way to understand it.
This connects directly to Chapters 13 and 16.
Suppose:
Node 1:
4 CPUExisting workloads request:
3 CPUAvailable allocatable capacity might only allow:
~1 CPUNow your Pod requests:
resources:
requests:
cpu: "2"Node 1 isn't suitable.
Pod requires 2 CPU
Node has insufficient allocatable capacity
↓
❌Scheduler checks other nodes.
This is extremely important:
Scheduling is primarily based on resource requests, not current real-time CPU usage.
Example:
Node:
8 CPU
Existing Pods:
Requests = 6 CPU
Actual usage = 2 CPUNew Pod:
Request = 3 CPUScheduler sees approximately:
6 + 3 = 9 CPUSo the Pod doesn't fit based on requested capacity, even though current observed usage might be only 2 CPU.
nodeSelector#Now let's say you have:
Node 1 → general
Node 2 → general
Node 3 → gpuYou want your AI workload on GPU nodes.
You can label the node:
kubectl label node node-3 workload=gpuCheck:
kubectl get nodes --show-labelsThen Pod:
spec:
nodeSelector:
workload: gpuNow:
Pod
│
└── workload=gpu
│
▼
Node 3A label is a key-value pair attached to a Kubernetes object.
Example:
workload=gpu
environment=production
team=payments
disk=ssdNode:
node-3
labels:
workload=gpu
environment=productionYou can inspect:
kubectl get nodes --show-labelsnodeSelector Example#apiVersion: v1
kind: Pod
metadata:
name: gpu-app
spec:
nodeSelector:
workload: gpu
containers:
- name: app
image: my-ai-app:1.0This means:
Only schedule this Pod onto nodes with
workload=gpu.
If no suitable node exists:
Pod → PendingSuppose:
nodeSelector:
workload: gpubut your cluster has:
Node 1 → workload=general
Node 2 → workload=generalNo GPU node.
Result:
Pod
↓
Scheduler
↓
No matching node
↓
PendingThis is why labels and scheduling rules must be designed carefully.
nodeSelector is simple.
For more expressive rules, Kubernetes provides:
Node Affinity
Example:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: workload
operator: In
values:
- gpuThis is more powerful than a simple nodeSelector.
This distinction is important.
Must satisfy this ruleExample:
requiredDuringSchedulingIgnoredDuringExecution:If the condition isn't satisfied:
Pod → PendingPrefer this node if possibleExample:
preferredDuringSchedulingIgnoredDuringExecution:If preference can't be satisfied, Kubernetes can still schedule the Pod elsewhere if other requirements are met.
Think:
required = MUST
preferred = SHOULDIgnoredDuringExecution#This phrase looks complicated:
requiredDuringSchedulingIgnoredDuringExecutionBreak it down:
During scheduling
↓
Rule must be satisfied
During execution
↓
Changes to the node label don't necessarily evict the already-running PodSo the rule primarily affects scheduling decisions.
Suppose you have:
Node 1 → zone=india-north
Node 2 → zone=india-west
Node 3 → zone=india-southYou want the application in:
india-northYou can use node affinity.
Conceptually:
Pod
│
▼
nodeAffinity
│
▼
zone=india-north
│
▼
Node 1Now we get to one of the most important scheduling concepts:
Taints repel Pods from nodes.
Suppose you have a GPU node:
Node GPU-1You don't want normal workloads accidentally using it.
You can taint it:
kubectl taint nodes gpu-1 workload=gpu:NoScheduleNow:
Normal Pod
│
▼
GPU Node
│
▼
❌ RejectedImagine:
GPU nodeGPU resources are expensive.
You want only AI workloads there.
Without protection:
Frontend Pod → GPU node ❌
Backend Pod → GPU node ❌
Redis Pod → GPU node ❌
AI Pod → GPU node ✅Taint helps reserve the node.
A Pod can be given permission to tolerate a taint.
Node:
workload=gpu:NoSchedulePod:
tolerations:
- key: workload
operator: Equal
value: gpu
effect: NoScheduleNow:
GPU Node
│
│ taint
▼
workload=gpu
│
▲
│ toleration
AI PodThe Pod is allowed to be considered for that node.
Memorize this:
Taint is on the Node. Toleration is on the Pod.
Node
↓
Taint
↓
"Don't schedule ordinary Pods here."
Pod
↓
Toleration
↓
"I'm allowed to tolerate this taint."But an important nuance:
A toleration allows a Pod to be scheduled onto a tainted node; it does not by itself force the Pod onto that node.
You may combine toleration with node affinity/selector if you want to both allow and target a specific node group.
There are three common taint effects:
NoSchedule
PreferNoSchedule
NoExecuteNoSchedule#New Pods without matching tolerations won't be scheduled there.
New Pod
↓
No toleration
↓
❌Existing Pods generally aren't automatically removed merely because this taint is added.
PreferNoSchedule#Kubernetes tries to avoid placing Pods there, but it's a preference rather than a hard prohibition.
Think:
"Please avoid this node if possible."NoExecute#Stronger.
It can affect existing Pods that don't tolerate the taint and may cause them to be evicted.
Node gets NoExecute taint
↓
Existing incompatible Pod
↓
EvictedNode:
kubectl taint nodes gpu-1 workload=gpu:NoSchedulePod:
apiVersion: v1
kind: Pod
metadata:
name: ai-worker
spec:
tolerations:
- key: workload
operator: Equal
value: gpu
effect: NoSchedule
containers:
- name: worker
image: my-ai-worker:1.0The Pod tolerates the taint.
But again, if you specifically want it on GPU nodes, combine that with:
nodeSelector:
workload: gpuor appropriate node affinity.
So far we talked about:
Pod → NodeNow let's discuss:
Pod → Pod
Pod affinity means:
Prefer or require placing a Pod near other Pods matching certain labels.
Example:
Frontend Pod
│
│ affinity
▼
Backend PodWhy might you want this?
Potentially:
This is extremely useful for high availability.
Suppose you have:
3 replicasYou don't want all replicas on one node.
Without anti-affinity:
Node 1
├── Pod A
├── Pod B
└── Pod C
Node 2
└── emptyIf Node 1 fails:
All replicas lostYou can tell Kubernetes:
Prefer not to place replicas together.
Conceptually:
Node 1
└── Pod A
Node 2
└── Pod B
Node 3
└── Pod CNow if Node 1 fails:
Pod A lost
Pod B → still running
Pod C → still runningThis improves resilience.
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- node-apiThis says, roughly:
Prefer not to place this Pod on the same node as another
app=node-apiPod.
topologyKey#This is an important concept.
It tells Kubernetes the topology domain across which the rule applies.
Common examples:
kubernetes.io/hostnamemeans:
different nodesAnother common topology dimension:
topology.kubernetes.io/zonemeans:
different availability zonesImagine:
Zone A
├── Node 1
└── Node 2
Zone B
├── Node 3
└── Node 4
Zone C
├── Node 5
└── Node 6You have:
3 API PodsA resilient placement might be:
Zone A → Pod A
Zone B → Pod B
Zone C → Pod CNow losing one zone doesn't necessarily take down every replica.
This is one reason topology-aware scheduling matters in production.
Modern Kubernetes also provides:
Topology Spread Constraints
These let you express how evenly Pods should be distributed across topology domains.
Example:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: node-apiConceptually:
Zone A → 2 Pods
Zone B → 2 Pods
Zone C → 1 PodThe goal is to keep the distribution reasonably balanced according to the configured constraint.
DoNotSchedule vs ScheduleAnyway#With topology spread constraints, you may see:
DoNotSchedule
ScheduleAnywayIf the constraint cannot be satisfied:
Don't schedule the Pod.Try to honor the constraint, but schedule the Pod anyway if necessary.
Think:
DoNotSchedule = hard
ScheduleAnyway = softSuppose:
Node.js API
3 replicasYou want:
Node 1 → Pod A
Node 2 → Pod B
Node 3 → Pod CYou could use anti-affinity or topology spread constraints.
A simplified Deployment might look like:
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-api
spec:
replicas: 3
selector:
matchLabels:
app: node-api
template:
metadata:
labels:
app: node-api
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: node-api
containers:
- name: node-api
image: my-node-api:1.0
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1"
memory: "512Mi"This encourages replicas to spread across different nodes.
Suppose we have:
Node 1
CPU available: 1
zone: A
workload: general
Node 2
CPU available: 2
zone: B
workload: general
Node 3
CPU available: 4
zone: A
workload: gpuPod:
resources:
requests:
cpu: "2"
nodeSelector:
workload: generalScheduler:
Node 1
CPU 1 < required 2
❌
Node 2
CPU 2 >= required 2
workload=general
✅
Node 3
CPU sufficient
BUT workload=gpu
❌Result:
Pod → Node 2Here's the mental model you should remember:
Pod
│
▼
Scheduler
│
▼
┌─────────────────┐
│ Filter Nodes │
└─────────────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
Resources Selector Taints
│ │ │
└────────────┼────────────┘
▼
Feasible Nodes
│
▼
Score Nodes
│
▼
Select Node
│
▼
Bind Pod
│
▼
kubelet
│
▼
Container runsNow you know many possible reasons.
Insufficient cpuInsufficient memoryNo node matches selectorAffinity requirements not satisfiedNode has taint
Pod doesn't tolerate itCannot satisfy topology ruleStorage may only be available in certain zones/nodes.
So:
Pending does not automatically mean "need more nodes."
First:
kubectl get podsIf:
STATUS = Pendingrun:
kubectl describe pod <pod-name>Look at:
Events:For example:
Warning FailedScheduling
0/3 nodes are available:
2 Insufficient cpu,
1 node(s) didn't match Pod's node affinity.Now you have the reason.
List nodes:
kubectl get nodesDetailed:
kubectl get nodes -o wideLabels:
kubectl get nodes --show-labelsDescribe node:
kubectl describe node <node-name>Add label:
kubectl label node <node-name> workload=gpuRemove label:
kubectl label node <node-name> workload-Add taint:
kubectl taint nodes <node-name> workload=gpu:NoScheduleRemove taint:
kubectl taint nodes <node-name> workload=gpu:NoSchedule-cordon#Another useful node command:
kubectl cordon <node-name>This marks a node as:
Unschedulable for new Pods.
Existing Pods aren't automatically removed.
Conceptually:
Node 1
├── Existing Pod A ✅
├── Existing Pod B ✅
└── New Pod ❌drain#Now:
kubectl drain <node-name>is more aggressive.
It attempts to safely evict supported workloads from the node so the node can be maintained.
Conceptually:
Node
├── Pod A
├── Pod B
└── Pod C
drain
↓
Pod A → another node
Pod B → another node
Pod C → another nodeThen the node can be taken out of service.
drain has important options and exceptions, so don't blindly run it against production nodes.
Very common interview question:
| Command | Meaning |
|---|---|
cordon |
Don't schedule new Pods |
drain |
Evict/move suitable existing workloads and make node unschedulable |
Simple memory trick:
cordon = STOP NEW
drain = MOVE EXISTING + STOP NEWDon't confuse these.
Decides:
"Pod should run on Node 2."Runs on Node 2 and ensures:
"Pod actually runs here."Architecture:
Scheduler
│
▼
Node 2
│
▼
kubelet
│
▼
Container Runtime
│
▼
ContainerA control-plane component that selects suitable nodes for unscheduled Pods.
No.
The scheduler assigns the Pod to a node. The kubelet and container runtime handle execution.
A simple way to restrict a Pod to nodes having specific labels.
A more expressive mechanism for defining required or preferred node placement rules.
A node-side mechanism that repels Pods unless they have a matching toleration.
A Pod-side declaration allowing it to be considered for nodes with matching taints.
No.
It only permits it to tolerate the taint.
A scheduling rule used to avoid placing certain Pods close together, commonly to spread replicas across nodes.
A mechanism for distributing Pods across topology domains such as nodes or zones.
The Pod generally remains:
Pendinguntil scheduling conditions change.
Scheduler
│
├── Resources
├── nodeSelector
├── Node Affinity
├── Taints
├── Tolerations
├── Pod Affinity
├── Pod Anti-Affinity
└── Topology SpreadnodeSelector
→ Simple node requirement
nodeAffinity
→ Advanced node placement
taint
→ Node repels Pods
toleration
→ Pod can tolerate taint
podAffinity
→ Put Pods near matching Pods
podAntiAffinity
→ Keep Pods apart
topologySpread
→ Distribute Pods across topologyIf an interviewer asks:
"How does Kubernetes decide where to run a Pod?"
Think:
Pod
│
▼
Scheduler
│
▼
Filter Nodes
│
┌───────────┼───────────┐
▼ ▼ ▼
Resources Affinity Taints
│ │ │
└───────────┼───────────┘
▼
Feasible Nodes
│
▼
Scoring
│
▼
Best Node
│
▼
Bind Pod
│
▼
kubelet
│
▼
ContainerAnd when things don't fit:
Pod
↓
No suitable node
↓
Pending
↓
Cluster Autoscaler may add nodes
↓
Scheduler retries
↓
Pod gets placedWe'll move from long-running applications to one-time and scheduled workloads:
Deployment
→ continuously running application
Job
→ run task until completion
CronJob
→ run Job on a scheduleWe'll build practical examples such as database migrations, background processing, cleanup jobs, scheduled reports, and Node.js scripts.
Mark it as complete to track your mastery.