Chapter 15 introduced the fundamentals of caching:
Now we move to production-grade caching.
At large scale, the cache itself becomes a distributed system.
A typical architecture may look like:
Users
|
v
Load Balancer
|
v
Application Tier
/ \
/ \
Local Cache Redis
| |
| v
| Cache Cluster
| |
+----------+----------+
|
v
DatabaseAt this level we need to think about:
The central idea is:
A cache is not merely a faster key-value store. At scale, it is a distributed infrastructure component with its own availability, consistency, capacity, and failure requirements.
Redis is commonly used as a fast in-memory data store.
Conceptually:
Application
|
v
Redis
|
v
MemoryRedis can support many use cases beyond simple caching, including:
For system design, do not think of Redis only as:
"A cache."
Think of it as:
A high-speed data structure server that can be used as a cache and for several coordination/state workloads.
Redis supports multiple data structures.
Common ones include:
String
Hash
List
Set
Sorted Set
StreamThe choice depends on the data model.
A string stores a value associated with a key.
Example:
user:123:name -> "Anubhav"Or:
product:123 -> "{...json...}"Common use cases:
Conceptually:
SET user:123 "Anubhav"
GET user:123A hash stores fields under a key.
Example:
user:123
name = Anubhav
city = Delhi
role = engineerConceptually:
user:123
|
+-- name
+-- city
+-- roleUseful when accessing individual fields rather than replacing the entire object.
A set stores unique values.
Example:
online-users
|
+-- user:101
+-- user:102
+-- user:103Useful for:
A sorted set associates values with scores.
Example:
leaderboard
Anubhav -> 9800
Rahul -> 9500
Amit -> 9100This is useful for:
Lists provide ordered collections.
Conceptually:
queue
|
+-- task A
+-- task B
+-- task CThey can be useful for certain queue/work-list patterns.
For production messaging, however, the correct choice depends on delivery guarantees and workload requirements.
Streams provide an append-oriented data structure with consumer-oriented processing features.
Conceptually:
Event Stream
|
+-- Event 1
+-- Event 2
+-- Event 3They can be used for:
Redis Streams are different from simply storing cached values.
A cache entry can have an expiration time.
Conceptually:
product:123
TTL = 300 secondsAfter expiration, the key becomes eligible to disappear according to Redis's expiration behavior.
Expiration is important for:
Redis can be used purely as an ephemeral cache, but Redis deployments can also use persistence mechanisms.
This creates an important distinction:
Pure Cache
Redis
|
X restart
|
Data can disappearversus:
Redis with persistence
Redis
|
v
Persistence
|
v
RecoveryIf Redis is being used as a cache, losing its contents may be acceptable if the database can repopulate it.
If Redis stores important application state, durability requirements become much more important.
Never assume:
"Redis means data is always temporary."
The architecture determines whether Redis data is disposable.
A large cache may need more memory and throughput than one machine can provide.
Instead of:
One Rediswe can use:
Redis Cluster
|
+-- Node A
+-- Node B
+-- Node C
+-- Node DThe key idea is:
Distribute keys across multiple nodes.
This is cache partitioning.
Suppose we have:
1 billion keysA single node may not have enough memory.
We can partition keys:
Key Space
|
+---------+---------+
| | |
v v v
Node A Node B Node CFor example:
user:1 -> Node A
user:2 -> Node C
user:3 -> Node BThe routing mechanism determines where each key belongs.
Many distributed cache systems use a deterministic partitioning mechanism.
A simplified model:
slot = hash(key) % NThen:
slot -> cache nodeThe actual implementation can use a more sophisticated fixed slot space.
The important system-design concept is:
Key
|
v
Hash / Partition Function
|
v
Partition
|
v
Cache NodeSuppose one cache server provides:
100 GB memoryand we need:
1 TBWe can distribute data:
Node 1 -> 100 GB
Node 2 -> 100 GB
Node 3 -> 100 GB
...Partitioning allows the cache to scale horizontally.
Partitioning improves capacity.
Replication improves availability.
A common architecture is:
Partition 1
|
+-- Primary
+-- Replica
Partition 2
|
+-- Primary
+-- Replica
Partition 3
|
+-- Primary
+-- ReplicaNow we have:
Partitioning -> scale
Replication -> redundancyThese solve different problems.
Suppose:
Partition A Primary
XA replica may be promoted.
Partition A Replica
|
v
New PrimaryApplications need to continue routing requests to the correct partition.
This requires cluster-aware routing and failure detection.
A simplified failover process:
1. Detect primary failure
2. Determine whether failure is real
3. Promote replica
4. Update cluster metadata
5. Redirect traffic
6. Reconnect clients
7. Rebuild/repair redundancyThe actual implementation depends on the cache technology.
Suppose a cache node fails.
If cached data is disposable:
Cache failure
|
v
Database fallbackBut if millions of requests miss simultaneously:
Cache failure
|
v
Millions of misses
|
v
Database overloadTherefore:
Cache failure can become a database failure if fallback traffic is uncontrolled.
Do not confuse:
Database replicationwith:
Cache replicationDatabase replication protects persistent data.
Cache replication primarily protects cached state and availability.
Example:
Database
|
+-- DB Replica 1
+-- DB Replica 2
Cache
|
+-- Cache Replica 1
+-- Cache Replica 2The requirements are different.
Sometimes many application instances need to coordinate around one operation.
Example:
100 servers
|
v
same cache key expiredAll servers may attempt to regenerate it.
A distributed lock can ensure:
Server A -> gets lock
Server B -> waits
Server C -> waitsThen:
Server A
|
v
Database
|
v
Cache
|
v
Release lockSuppose:
product:123expires.
Without coordination:
Request 1 -> DB
Request 2 -> DB
Request 3 -> DB
...
Request 10,000 -> DBWith a lock:
Request 1
|
gets lock
|
v
Database
|
v
Cache
Requests 2-10,000
|
wait/retry
|
v
Cache HITThis can protect the database.
A distributed lock should consider:
A simple lock implementation is not automatically safe for every distributed workflow.
Suppose:
Server A
gets lockThen Server A crashes.
Without expiration:
Lock remains foreverThe system can become stuck.
Therefore locks commonly have a bounded lifetime.
But expiration creates another problem.
Suppose:
Lock TTL = 5 sec
Operation = 10 secThe lock may expire while Server A is still working.
Then:
Server B
gets same lockNow both may believe they own the operation.
This is why distributed locking requires careful design.
A stronger approach is to assign monotonically increasing fencing tokens.
Example:
Server A -> token 41
Server B -> token 42The protected resource can reject stale operations:
token 41 -> reject
token 42 -> acceptThis prevents an old lock holder from continuing to modify the protected resource after a newer owner has taken over.
Fencing is particularly important when operations can outlive lock leases.
Request coalescing means combining multiple simultaneous requests for the same missing value into one backend operation.
Example:
100 requests
|
v
Same cache key
|
v
One database query
|
v
Result sharedThis is different from simply having a cache.
It specifically protects against concurrent misses.
Request A ----\
Request B -----\
Request C ------> Loader
Request D -----/ |
Request E ----/ v
Database
|
v
Cache
|
+----------+----------+
| | |
v v v
A B CThis is one of the most effective ways to reduce stampedes for very hot keys.
A powerful strategy is:
Serve stale data immediately while refreshing it in the background.
Example:
Cache entry:
Fresh for 60 sec
Stale-but-servable for another 300 secWhen stale:
Request
|
v
Cache
|
+---- return stale value
|
+---- background refreshThis avoids making the user wait for the refresh.
0s -------- 60s -------- 360s
| | |
Fresh Stale Expired
but |
usable |During the stale period:
User -> stale response
+
background refreshThis is useful when:
Refresh-ahead means refreshing data before it expires.
Example:
TTL = 10 minutes
Refresh at = 9 minutesFlow:
Cache
|
| approaching expiry
v
Background refresh
|
v
Fresh valueThe next user never experiences a cold miss.
This works particularly well for predictable popular keys.
If millions of entries are inserted simultaneously with:
TTL = 300 secondsthey may all expire around the same time.
Instead:
TTL = 300 + random(0..60)This spreads expiration times.
Conceptually:
Without jitter:
||||||||||||||||||
same expiration
With jitter:
| | || | | || | |
spread expirationTTL jitter reduces synchronized cache misses.
Another advanced technique is to refresh entries probabilistically as they approach expiration.
Instead of:
At exact expiration -> refreshthe system can gradually increase the probability of refreshing earlier.
This spreads backend work over time.
The exact algorithm depends on the implementation.
The goal is:
Avoid synchronized regeneration of popular values.
Suppose:
product:123receives:
500,000 requests/secOne cache node or one logical key can become hot.
Strategies:
App memory
|
v
RedisHot values are served locally.
Store the same logical value under multiple physical cache keys.
product:123:replica1
product:123:replica2
product:123:replica3Requests can be distributed.
Move extremely popular public content closer to clients.
Reduce duplicate backend work.
Replicating a hot key reduces read concentration but increases:
For example:
One copy:
1 key
Ten replicas:
10 keysWhen the value changes:
Update 10 copiesTherefore hot-key mitigation is a trade-off.
Global applications may deploy caches near users.
Global Users
/ | \
v v v
India US Europe
| | |
v v v
Cache Cache Cache
\ | /
\ | /
v v v
DatabaseBenefits:
Challenges:
Suppose:
India Cache:
price = 100
US Cache:
price = 100A product price changes:
Database:
price = 120Now:
India Cache -> 100
US Cache -> 100The system needs to invalidate or refresh both regions.
Possible approaches:
Database Update
|
v
Global Event
/ \
v v
India US
Cache Cache
InvalidateThe propagation itself may be asynchronous.
Suppose the India cache cluster fails.
Possible fallback:
India Users
|
v
India Cache
X
|
v
Regional fallback
|
v
Database / alternate cacheBut cross-region fallback increases latency.
Therefore a regional cache should ideally have enough redundancy to handle local failures.
After a cache cluster restart:
Cache = emptyMillions of requests may immediately cause misses.
A warming strategy may preload:
Architecture:
Database
|
v
Warm-up Worker
|
v
CacheThe worker can prioritize the most valuable keys.
Bad warming can be harmful.
Suppose you attempt to preload:
10 billion recordsThe database may become overloaded.
Therefore warming should consider:
Warm the data that provides the most value.
A production cache needs monitoring.
Important metrics include:
hits / total requestsmisses / total requestsMeasure:
High eviction rates may indicate insufficient capacity or poor key/TTL strategy.
Monitor:
used memory
max memory
fragmentationIdentify disproportionately accessed keys.
Track:
Average latency can hide problems.
Example:
Average = 2 msBut:
p50 = 1 ms
p95 = 4 ms
p99 = 80 msA small percentage of requests are very slow.
At large scale, p99 matters significantly.
Always monitor cache latency percentiles.
Suppose:
Max memory = 100 GB
Used = 98 GBThe cache is close to capacity.
Potential consequences:
Therefore capacity should be monitored before the system reaches the limit.
Different workloads benefit from different policies.
Possible policies include:
Questions to ask:
Are recent items more valuable?
Are frequent items more valuable?
Is data disposable?
What happens when memory is full?The answer determines the appropriate policy.
A distributed cache may contain:
sessions
tokens
personal data
authorization state
financial informationSecurity considerations include:
Never expose an internal cache directly to untrusted clients.
Instead of storing sessions in one application server:
Server A
|
+-- Sessionuse shared session storage:
Server A \
Server B ---> Redis
Server C /Then any application server can retrieve the session.
This supports horizontal scaling.
Example:
session:abc123
|
+-- user_id
+-- expires_atSuppose we need:
page views = 1,000,000Multiple application servers update the same counter.
A distributed cache can provide atomic counter operations.
Conceptually:
Server A --\
Server B ---+--> Counter
Server C --/This avoids storing every individual increment in the database.
The durability requirements must still be considered.
Caching systems can also support rate limiting.
Example:
user:123:requestsIncrement:
1
2
3
...
100Policy:
100 requests / minuteIf count exceeds the limit:
429 Too Many RequestsA more complete rate-limiting architecture will be covered later in the system-design course.
A cache is usually optimized for:
A database is optimized for:
Therefore:
Cache != DatabaseEven if both can store key-value data.
A cache answers:
"What is the current value?"
A queue answers:
"What work/event needs to be processed?"
Example:
Cache:
user:123 -> profile
Queue:
send-email -> job123They solve different problems.
A CDN primarily caches content close to users at the network edge.
An application cache usually caches application-level data.
Example:
Browser
|
v
CDN
|
v
Application
|
v
Redis
|
v
DatabaseAll are caching layers, but they serve different purposes.
A large production system might look like:
Users
|
v
CDN
|
v
Load Balancer
|
v
Application Servers
/ \
/ \
L1 Local Cache L2 Redis Cluster
|
+---------+---------+
| | |
v v v
Node A Node B Node C
| | |
+---------+---------+
|
v
DB ClusterDifferent layers absorb different types of traffic.
A mature cache architecture should define what happens when:
Use:
Use:
Use:
Use:
Never allow cache calls to hang indefinitely.
Example:
Application
|
v
Redis
|
timeoutWithout a timeout:
Request threads
|
v
waiting...
waiting...
waiting...This can exhaust application resources.
A cache timeout should be aligned with the application's latency budget.
Suppose Redis is failing.
Without protection:
Every request
|
v
Redis
|
X
failureThe application repeatedly waits for the failing dependency.
A circuit breaker can temporarily stop requests to the failing cache:
Redis failure
|
v
Circuit OPEN
|
v
Fast fallbackThis can protect application resources.
Suppose recommendation cache fails.
Instead of:
Whole page = ERRORthe application can return:
Page
|
+-- Product information
+-- Reviews
+-- Purchase
|
+-- Recommendations unavailableThis is graceful degradation.
Caching often enables optional features to fail independently.
For every cache, explicitly define:
Source of truth
Freshness requirement
Invalidation mechanism
Fallback behaviorExample:
Source of truth:
PostgreSQL
Freshness:
< 30 seconds
Invalidation:
TTL + event
Fallback:
Database
Failure:
Serve stale value if safeThis is much better than simply saying:
"We'll use Redis."
Suppose we are building an e-commerce product API.
Requirements:
Traffic = 300,000 reads/sec
Product data changes relatively infrequently
Product details can tolerate small staleness
High availability requiredPossible design:
Client
|
v
CDN
|
v
Application
|
v
L1 Local Cache
|
v
Redis Cluster
|
v
Database ReplicasFlow:
1. CDN hit -> return
2. CDN miss -> application
3. L1 hit -> return
4. L1 miss -> Redis
5. Redis hit -> populate L1 -> return
6. Redis miss -> DB
7. DB result -> Redis -> L1 -> returnThis creates a multi-level cache.
Before deploying a large cache, answer:
Requirements:
1,000,000 requests/sec
Average object = 2 KB
Hit ratio target = 98%Database fallback:
2% × 1,000,000
=
20,000 requests/secApproximate cache traffic:
980,000 requests/secNow ask:
Can one node handle this?
How much memory is required?
How many shards?
How many replicas?
What is the network bandwidth?
What happens during a node failure?If average object size is 2 KB:
980,000 × 2 KB
≈ 1.96 GB/secof response payload traffic at the cache boundary under a simplified assumption.
This demonstrates why cache sizing requires both:
Question:
Redis suddenly becomes unavailable. What happens?
Bad answer:
"The application reads from the database."
A stronger answer considers:
Redis failure
|
+-- timeout quickly
|
+-- circuit breaker
|
+-- controlled DB fallback
|
+-- request coalescing
|
+-- rate limiting
|
+-- graceful degradationThen ask:
Can the database handle the fallback load?If not, the system needs protection.
Question:
One cache key receives 500,000 requests/sec. What do you do?
Possible reasoning:
1. Confirm it is a hot key.
2. Check whether data is public/cacheable.
3. Add local L1 caching.
4. Use request coalescing.
5. Replicate the key if needed.
6. Use CDN if appropriate.
7. Monitor invalidation cost.Do not immediately add more random cache nodes.
If every request hashes to the same logical key, simply adding nodes may not solve the problem.
Question:
A popular key expires every five minutes and causes database overload.
Possible solution:
TTL
+
TTL jitter
+
request coalescing
+
background refreshFor example:
At 4m30s:
background refresh
At 5m:
still fresh
or:
First request after expiry:
gets refresh lock
Other requests:
wait/share resultAvailability depends on the deployment architecture.
A cache outage can overload the database.
One key can overload a node.
Incorrect lock semantics can cause duplicate or conflicting work.
Slow cache dependencies can consume application resources.
Large objects can be expensive even if the cache lookup itself is fast.
This can cause cache stampedes.
They solve different problems.
Multi-region caches can return different values temporarily.
This can create severe security issues.
Build an API:
GET /products/:id
POST /products
PUT /products/:id
DELETE /products/:idArchitecture:
Client
|
v
API
|
+---- L1 Local Cache
|
+---- Redis
|
+---- PostgreSQLImplement:
L1
|
+-- HIT -> response
|
MISS
|
v
Redis
|
+-- HIT -> L1 -> response
|
MISS
|
v
PostgreSQL
|
v
Redis
|
v
L1
|
v
responsePostgreSQL update
|
v
Invalidate Redis
|
v
Invalidate L1Add:
Test:
Request -> Cache -> responseRequest -> Cache MISS -> DB -> Cache -> responseTTL expires -> next request missesSend many concurrent requests for one expired key.
Expected:
One backend load
Many shared responsesStop Redis.
Expected:
Fast failure/fallback
No uncontrolled database explosionTest behavior when both cache and database cannot provide fresh data.
Expected behavior should be defined by business requirements.
At advanced level, think about caching as six layers:
CACHE SYSTEM
|
+-------------+-------------+
| | |
SPEED CAPACITY FRESHNESS
| | |
L1 Partitioning TTL
L2 Sharding Invalidation
CDN Replication Refresh
| | |
+-------------+-------------+
|
RELIABILITY
|
+----------+----------+
| | |
Failover Timeout Fallback
|
PROTECTION
|
+----------+----------+
| | |
Stampede Hot Key Rate Limit
|
OBSERVABILITY
|
Hit Ratio / Latency /
Memory / Evictions /
Errors / Hot KeysWhen designing a cache, do not stop at:
"We'll use Redis."
Instead explain:
What
Why
Where
How
TTL
Invalidation
Partitioning
Replication
Failure
Capacity
Consistency
Observability
SecurityThat is production-grade cache thinking.
In this chapter, we learned:
The central idea is:
At scale, the cache itself becomes a distributed system. You must design its partitioning, replication, consistency, failure handling, capacity, and observability—not merely choose a cache technology.
Make sure you can explain:
Next we move beyond databases and caches into large-scale storage systems.
We will learn:
Mark it as complete to track your mastery.