In Chapter 06, we learned horizontal scaling.
Instead of:
Client
↓
App Serverwe can have:
┌── App 1
├── App 2
Client → ??? ├── App 3
└── App 4The missing component is usually a load balancer.
A load balancer sits between clients and backend servers and decides where traffic should go.
Clients
↓
Load Balancer
├── App 1
├── App 2
├── App 3
└── App 4But a load balancer does much more than simply "split traffic."
A production load-balancing system may need to handle:
This chapter builds the complete mental model.
A load balancer is a system that distributes incoming traffic across multiple backend instances.
For example:
4,000 requests/sec
│
▼
Load Balancer
/ | \
↓ ↓ ↓
App 1 App 2 App 3If each instance can safely handle approximately:
1,500 req/secthe load balancer can distribute traffic across them.
The objective is not necessarily equal distribution.
The objective is:
Route traffic safely and efficiently to healthy backend capacity.
Without a load balancer, clients need some way to discover individual application servers.
Imagine:
Client
↓
App 1Now App 1 becomes overloaded.
Adding App 2 does not automatically help:
Client
↓
App 1
App 2The client still needs to know when and how to use App 2.
A load balancer provides a stable entry point:
Client
↓
Load Balancer
├── App 1
├── App 2
└── App 3The client does not need to know the individual backend addresses.
Think of the load balancer as a traffic controller.
┌── App 1
│
Internet → LB ────┼── App 2
│
└── App 3The LB decides:
Which backend?
When?
Based on what information?
What if the backend is unhealthy?These decisions are the core of load balancing.
These concepts overlap.
A reverse proxy:
Client
↓
Reverse Proxy
↓
BackendA load balancer:
Client
↓
Load Balancer
├── Backend 1
├── Backend 2
└── Backend 3A reverse proxy can perform load balancing.
For example, Nginx can act as:
Therefore:
"Reverse proxy" describes a role, while "load balancer" describes a traffic-distribution function.
Modern infrastructure components can perform both.
A common architecture:
Internet
↓
DNS
↓
Load Balancer
↓
Application Servers
↓
Cache / DatabaseA more advanced architecture:
Internet
↓
CDN / Edge
↓
Load Balancer
↓
API Gateway
↓
Application ClusterThe exact ordering depends on the system.
There are two major categories to understand:
Layer 4 Load Balancing
Layer 7 Load BalancingThe distinction comes from the networking layers.
Layer 4 operates primarily using transport-level information.
Typical information includes:
It does not need to understand the full HTTP request.
Conceptually:
Client
↓ TCP
L4 Load Balancer
↓ TCP
BackendL4 load balancing can be:
It can handle protocols other than HTTP.
Examples:
TCP
UDP
TLS pass-through
Database connections
Other network protocolsSuppose:
Client IP: 10.0.0.10
Destination: 203.0.113.10:443The L4 load balancer can choose:
Backend 1 → 10.0.1.10:443It does not necessarily need to inspect:
GET /users/123Layer 7 load balancing operates at the application layer.
For HTTP traffic, it can inspect:
Example:
GET /api/userscould be routed to:
User Servicewhile:
GET /api/orderscould go to:
Order Service L7 Load Balancer
│
┌─────────────┼─────────────┐
↓ ↓ ↓
/users /orders /payments
↓ ↓ ↓
User Service Order Service Payment ServiceThis is much more application-aware than simple connection-level routing.
| L4 | L7 |
|---|---|
| Transport-level | Application-level |
| TCP/UDP aware | HTTP-aware |
| Less request inspection | Can inspect HTTP details |
| Often lower overhead | More processing |
| Protocol-agnostic at app layer | Application-specific |
| Good for raw TCP/UDP | Good for HTTP routing |
Neither is universally better.
The choice depends on requirements.
Once a load balancer receives a request, it needs a routing strategy.
Common algorithms include:
Different algorithms solve different problems.
The simplest algorithm.
Suppose:
Servers:
A
B
CRequests:
Request 1 → A
Request 2 → B
Request 3 → C
Request 4 → A
Request 5 → B
Request 6 → CPattern:
A → B → C → A → B → CIt is:
It works well when backend servers have approximately equal capacity and requests have similar cost.
Suppose:
Server A → powerful
Server B → medium
Server C → weakRound robin still sends:
A = 1/3
B = 1/3
C = 1/3This may be inefficient.
Another issue:
Request 1 → 1 ms
Request 2 → 10 secondsAll requests are not equally expensive.
Round robin does not know this.
Assign weights:
A → weight 5
B → weight 3
C → weight 2Approximate distribution:
A → 50%
B → 30%
C → 20%Useful when servers have different capacities.
Route the new connection/request to the backend with the fewest active connections.
Example:
App 1 → 100 connections
App 2 → 40 connections
App 3 → 70 connectionsNew request:
→ App 2This can work better when requests have significantly different durations.
The number of connections is not always the same as actual workload.
For example:
App 1 → 10 expensive requests
App 2 → 30 cheap requestsApp 1 has fewer connections but may be more CPU-intensive.
Therefore connection count is only an approximation of load.
Combines:
Current connections
+
Server capacityA more powerful server can receive more traffic while still considering active connections.
The load balancer randomly chooses a backend.
Request 1 → B
Request 2 → A
Request 3 → C
Request 4 → BWith enough requests, traffic can approach a balanced distribution.
Randomization is useful in many distributed systems, especially when combined with other strategies.
The load balancer hashes the client's IP.
Conceptually:
hash(client_ip) % NExample:
User A → hash → Server 1
User B → hash → Server 3
User C → hash → Server 2This can provide a form of session affinity.
Suppose an application has local session state.
You might want:
User A → App 1
User A → App 1
User A → App 1IP hashing can provide approximate affinity.
But it has limitations.
Users may appear behind the same NAT.
Example:
Office
├── User A
├── User B
├── User C
└── User D
↓
Same public IPThey may all hash to the same backend.
Also, when servers are added or removed:
N changesand many users may map to different servers.
Sticky sessions mean the load balancer attempts to keep a client associated with the same backend.
Example:
User A
↓
App 1
Later:
User A
↓
App 1This can be implemented using:
Suppose:
Session stored in App 1 memorySticky sessions ensure:
User
↓
App 1instead of:
User
↓
App 2where the session does not exist.
They create dependency on specific instances.
If:
User → App 1and App 1 crashes:
User → App 1
Xthe user may lose local state.
They can also cause uneven load:
App 1 → 80%
App 2 → 10%
App 3 → 10%Therefore a shared session store is often preferable when possible.
Consistent hashing is useful when we want stable mapping between keys and backend nodes while allowing the node set to change.
Instead of:
hash(key) % Nwhich can remap many keys when N changes, consistent hashing uses a logical hash ring.
Conceptually:
Hash Ring
A
/ \
D B
\ /
CKeys are mapped to positions on the ring and assigned to the next appropriate node.
Suppose:
Server A
Server B
Server CA new server is added:
Server DWith simple modulo hashing:
hash(key) % Nmany keys may move.
With consistent hashing:
Ideally, only a subset of keys needs to move.
This is particularly useful for:
We will revisit consistent hashing later.
Consistent hashing can use virtual nodes.
Instead of one position per physical server:
Server A → A1, A2, A3...
Server B → B1, B2, B3...This improves distribution around the ring.
Conceptually:
A1
B1
A2
C1
B2
A3
C2
...If a physical server is removed, its virtual-node ranges can be reassigned.
A load balancer should not route traffic to an unhealthy backend.
Example:
App 1 → healthy
App 2 → unhealthy
App 3 → healthyTraffic:
Load Balancer
/ \
↓ ↓
App 1 App 3App 2 is temporarily removed from rotation.
The load balancer periodically checks the backend.
For example:
GET /healthResponse:
200 OKmeans the server is considered healthy according to that check.
If:
timeout
500
connection refusedthe backend may be marked unhealthy.
The load balancer can also observe actual traffic.
For example:
Backend returns repeated 503sThe load balancer may temporarily reduce or stop sending traffic to it.
Passive and active health mechanisms can be combined.
These are especially important in modern application platforms.
"Is the process alive?"
"Can this instance safely receive traffic?"
Example:
Process = running
Database = unavailableThe process is alive.
But it may not be ready to serve requests.
Therefore:
Liveness → yes
Readiness → noThe load balancer/orchestrator can use readiness to control traffic.
Do not make a health endpoint unnecessarily dependent on every downstream service.
Bad:
/health
↓
Database
↓
Redis
↓
External API
↓
QueueIf any dependency fails, the whole instance may be declared unhealthy.
Sometimes that causes a large-scale failure:
Database temporary issue
↓
All instances fail health checks
↓
All instances removed
↓
No traffic servedHealth checks should reflect the intended readiness semantics carefully.
A system may have:
/health/live
/health/readyFor example:
live:
Process running
ready:
Application initialized
Required dependencies availableThe exact readiness requirements depend on the service.
Suppose:
App 1
App 2
App 3We want to remove App 2 for deployment.
Bad:
Remove App 2
↓
Immediately terminate
↓
Active requests failConnection draining:
Stop new requests → App 2
↓
Existing requests finish
↓
Close connections
↓
Terminate App 2This is also called:
Connection draining / graceful deregistration
It is important during:
Without draining, users can experience unnecessary request failures.
Two designs:
Sticky Sessions
+
Local session memoryStateless Apps
+
Shared Session StoreDesign B usually provides more flexibility for horizontal scaling.
Example:
App 1 ─┐
App 2 ─┼──→ Redis
App 3 ─┘Now any backend can handle a request.
HTTPS can be terminated at the load balancer.
Client
↓ HTTPS
Load Balancer
↓ HTTP/internal TLS
ApplicationThis is called:
TLS termination
Benefits:
Alternatively:
Client
↓ HTTPS
Load Balancer
↓ HTTPS
ApplicationThe load balancer forwards encrypted traffic without terminating TLS.
The backend handles TLS.
This may be useful when end-to-end encryption requirements or application-level certificate handling require it.
| Termination | Passthrough |
|---|---|
| LB decrypts traffic | LB does not decrypt |
| Easier centralized certificate management | Backend handles TLS |
| LB can inspect HTTP if L7 | Cannot inspect encrypted HTTP content |
| Can reduce backend TLS work | End-to-end encryption remains intact to backend |
Some systems use re-encryption:
Client
↓ HTTPS
LB
↓ HTTPS
BackendThe LB terminates the first TLS connection and establishes another TLS connection to the backend.
An L7 load balancer can route based on hostname.
Example:
api.example.com
↓
API Cluster
admin.example.com
↓
Admin ClusterIt can also route by path:
/api/users
↓
User Service
/api/orders
↓
Order ServiceThis is often called:
Content-based routing
For example:
X-Region: INcould influence routing.
Or:
X-Experiment: Acould route users to a particular deployment.
This can support:
Routing policies must be carefully controlled because headers may be client-controlled unless added or validated by trusted infrastructure.
Suppose:
Version A → 95%
Version B → 5%The load balancer can send a small percentage of traffic to Version B.
Load Balancer
/ \
↓ ↓
Version A Version B
95% 5%If metrics are healthy, traffic can gradually increase:
95/5
↓
90/10
↓
75/25
↓
50/50
↓
100/0Canary deployment reduces blast radius.
Two environments:
Blue → Current
Green → NewTraffic initially goes to Blue.
LB
↓
BlueAfter Green is validated:
LB
↓
GreenRollback can potentially switch traffic back to Blue.
This requires both environments to be maintained during the transition.
A load balancer can distribute traffic by weights.
Example:
Version A → 90
Version B → 10Approximately:
90% → A
10% → BWeights can be used for:
Multiple backend clusters actively serve traffic.
Traffic
/ \
↓ ↓
Region A Region B
Active ActiveAdvantages:
Challenges:
One environment serves traffic.
Another is ready for failover.
Primary → Active
Secondary → StandbyDuring failure:
Primary fails
↓
Traffic → SecondaryThis can be simpler but may leave standby capacity underutilized.
A load balancer itself cannot become a single point of failure.
Bad:
Single LB
X
↓
All trafficIf the LB fails:
Entire application unreachableTherefore production load-balancing infrastructure is usually redundant.
Conceptually:
LB Cluster
/ \
LB1 LB2
\ /
Backend PoolManaged cloud load balancers typically hide much of this infrastructure.
Suppose:
LB
↓
App 1
App 2
App 3If the LB crashes, application servers may still be healthy but unreachable.
Therefore high-availability design must consider:
DNS
Load Balancer
Application
Databasenot just the application servers.
For global systems:
Users
├── India
├── Europe
├── USA
└── Asiawe may route users to nearby regions.
Global Routing
/ | \
↓ ↓ ↓
India Europe USA
Region Region RegionRouting can consider:
DNS, Anycast, CDN, and global load-balancing services can participate in this architecture.
A simplified architecture:
User
↓
DNS
↓
Load Balancer
↓
Application ServersDNS chooses the destination infrastructure.
The load balancer chooses the backend instance.
This creates two different routing layers.
DNS
↓
Region A
or
Region BLB
↓
App A
App B
App CDNS usually operates at a coarser level.
The load balancer can make much more frequent backend-level decisions.
Consider:
App 1 → unhealthy
App 2 → healthy
App 3 → healthyThe LB can remove App 1.
Therefore:
One instance failure
↓
Traffic redirected
↓
Service continuesThis is a major availability benefit.
Then the load balancer cannot manufacture capacity.
LB
├── App 1 X
├── App 2 X
└── App 3 XPossible behavior:
503 Service UnavailableThe load balancer should fail clearly rather than pretending the service is healthy.
Some load balancers can retry failed upstream connections.
Potential flow:
Request
↓
App 1 → failure
↓
Retry
↓
App 2
↓
SuccessThis can improve resilience for transient failures.
But retries can also amplify load.
Therefore:
Load-balancer retries must be bounded and carefully coordinated with application-level retries.
Suppose:
Client retry = 2 attempts
LB retry = 2 attempts
Service retry = 2 attemptsOne logical request can potentially create many physical attempts.
A simplified worst-case multiplication can become:
2 × 2 × 2 = 8 attemptsActual behavior depends on implementation and where failures occur.
This is why retry ownership should be designed explicitly.
For HTTP systems, connection reuse can reduce overhead.
Instead of:
Request
↓
New TCP connection
↓
Request
↓
Closepersistent connections can be reused.
The load balancer may maintain:
Client connectionsand separately:
Backend connectionsThis allows it to manage frontend and backend connection lifecycles independently.
A load balancer can itself be constrained by:
Large-scale systems must monitor these limits.
Suppose:
100,000 clientsand the load balancer maintains backend connections.
If backend connection pooling is poorly configured, the number of connections can become enormous.
Therefore:
Client connections
↓
Load Balancer
↓
Backend connection poolsmust be designed together.
The load balancer can help enforce capacity boundaries.
Example:
Traffic
↓
LB
↓
Healthy backendsIf all backends are overloaded, additional mechanisms may be needed:
Rate limiting
Queueing
Load shedding
Circuit breakersThe LB is not a substitute for end-to-end overload protection.
Suppose:
Service Bis failing.
Service A keeps sending requests.
A circuit breaker can stop repeated calls temporarily.
Service A
↓
Circuit Breaker
↓
Service BThis is different from load balancing.
Load balancing chooses:
Which healthy backend?A circuit breaker decides:
Should we call this dependency at all?We will study circuit breakers in the distributed-systems section.
In dynamic environments, backend instances can change.
Example:
App 1
App 2
App 3App 3 is replaced:
App 4The load balancer needs current backend information.
This can come from:
Kubernetes, for example, provides service discovery and routing mechanisms.
Conceptually:
Client
↓
Ingress / Load Balancer
↓
Kubernetes Service
↓
Pods
├── Pod 1
├── Pod 2
└── Pod 3Pods can be created or removed dynamically.
The service layer abstracts changing pod IPs.
We will study Kubernetes system design later.
Cloud systems commonly spread application instances across failure domains.
Example:
Load Balancer
/ \
↓ ↓
Zone A Zone B
App 1 App 3
App 2 App 4If Zone A fails:
Zone B
↓
Continue servingThis improves availability.
A failure domain is a group of resources that can fail together.
Examples:
Good architecture avoids placing all capacity in one failure domain.
Load balancers can distribute traffic across multiple domains.
Suppose:
Traffic = 9,000 req/secServers:
App 1 = 3,000
App 2 = 3,000
App 3 = 3,000Total:
9,000 req/secNow App 2 fails.
Remaining capacity:
3,000 + 3,000
= 6,000 req/secTraffic:
9,000 req/secThe system is now overloaded.
Therefore high availability requires failure capacity, not merely multiple instances.
A common resilience concept is N+1 capacity.
If normal required capacity is:
Nprovision:
N + 1so the system can tolerate one failure.
Example:
Required = 3 instances
Provision = 4If one fails:
4 → 3and normal capacity remains.
At larger scale, more sophisticated redundancy targets may be used.
Requirements:
20,000 req/sec peak
99.99% availability
Multiple application serversArchitecture:
Internet
│
▼
DNS
│
▼
Load Balancer Cluster
/ | \
↓ ↓ ↓
App1 App2 App3
│ │ │
└─────┼─────┘
▼
Redis
│
▼
DatabaseLB responsibilities:
Application responsibilities:
Database remains a separate scaling concern.
When choosing a strategy, ask:
If yes:
Round Robinmay be sufficient.
Consider:
Weighted Round RobinConsider:
Least ConnectionsConsider:
Sticky Sessions
IP HashBut first ask whether the application can be made stateless.
Consider:
Service Discovery
Consistent Hashingwhere appropriate.
Use:
L7The goal is appropriate distribution, not necessarily exact equality.
Traffic should not continue going to failed backends.
They can reduce flexibility and create uneven load.
A single LB can become a single point of failure.
It only distributes traffic to backends.
Immediate instance termination can fail active requests.
Multiple retry layers can amplify traffic.
L4 works with transport-level information.
L7 can inspect application-level protocol details.
A bad health check can either:
send traffic to broken instancesor:
remove healthy instances unnecessarilyA component that distributes incoming traffic among backend instances according to routing and health policies.
L4 operates primarily at the transport layer and routes based on connection/network metadata.
L7 understands application-layer protocols such as HTTP and can route based on request properties.
Round Robin cycles through backends.
Least Connections chooses a backend with fewer active connections.
A mechanism that attempts to route a client repeatedly to the same backend.
They can create uneven load, reduce failover flexibility, and make local state dependencies harder to manage.
Allowing an instance to finish existing requests before removing it from service.
A mechanism for determining whether a backend should receive traffic.
Because if it becomes a single point of failure, backend redundancy does not help.
A hashing approach that minimizes key remapping when backend nodes are added or removed.
Provisioning one additional unit beyond the normal required capacity so the system can tolerate one failure.
Design the load-balancing layer for:
50,000 requests/sec
20 application servers
99.99% availabilityRequirements:
Answer:
Design:
Internet
│
▼
DNS
│
▼
┌─────────────────────┐
│ LB / LB Cluster │
└──────────┬──────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
App 1 App 2 App 3
│ │ │
└────────────┼────────────┘
▼
Redis
│
▼
DatabaseAdd:
Health Checks
Connection Draining
TLS
Autoscaling
Request IDs
Metrics
LogsThen simulate:
App 2 crashesApp 3 is being deployedTraffic doublesOne availability zone failsNew version should receive 5% trafficExplain how the load-balancing layer handles each case.
When designing load balancing, think:
CLIENTS
│
▼
DNS / EDGE
│
▼
LOAD BALANCER
│
┌───────────┼───────────┐
▼ ▼ ▼
App 1 App 2 App 3
│ │ │
└───────────┼───────────┘
│
BackendThen ask:
Are backends healthy?
↓
Which backend should receive traffic?
↓
How much traffic should it receive?
↓
What if it fails?
↓
What if it is being deployed?
↓
What if traffic increases?
↓
What if an entire failure domain fails?
↓
What if the load balancer itself fails?This is the core load-balancing mindset.
You should now understand:
Before moving forward, make sure you can explain:
We now know how to distribute dynamic requests across application servers.
Next, we ask:
Can we prevent many requests from reaching our application servers in the first place?
That is where CDNs and edge infrastructure become important.
We will learn:
User
↓
Nearest Edge Location
├── Cache Hit → Response
└── Cache Miss
↓
Origin
↓
ApplicationTopics will include:
Mark it as complete to track your mastery.