A distributed system is a collection of independent computers that work together to provide a larger system.
A simple system might be:
Client
|
v
ServerA distributed system might look like:
Load Balancer
|
+-----------+-----------+
| | |
v v v
Server A Server B Server C
| | |
+-----------+-----------+
|
Database ClusterAt first glance, this looks like:
More servers = more power.
But distributed systems are difficult because those computers communicate over networks that can fail.
A server can be alive while another server cannot reach it.
A message can be delayed.
A response can be lost even though the operation succeeded.
Two nodes can temporarily disagree about state.
A clock on one machine can differ from another machine's clock.
This chapter builds the foundation needed to reason about these problems.
A distributed system is a system in which multiple independent computing nodes cooperate to perform work or provide a shared service.
Examples include:
A typical architecture:
Internet
|
v
Load Balancer
|
+-----------+-----------+
| | |
v v v
Node A Node B Node C
| | |
+-----------+-----------+
|
Shared ServicesThe nodes do not share one CPU or one memory space.
They communicate through networks.
That distinction creates most of the complexity.
Server
/ | \
/ | \
Client Client ClientOne main server owns the computation/state.
Node A ←→ Node B
↕ ↕
Node C ←→ Node DMultiple nodes participate.
Distributed architecture can improve:
But introduces:
There are several major reasons.
One server may not handle:
1 million requests/secMultiple servers can distribute the load.
If one server fails:
Server A ✗
Server B ✓
Server C ✓traffic can continue.
Users may be located worldwide.
India → India region
Europe → Europe region
US → US regionThis can reduce latency.
Different services can specialize:
API
Search
Payments
Notifications
AnalyticsIn a single machine:
function callis usually fast and predictable.
In a distributed system:
Service A → Network → Service Bhas uncertainty.
The network can:
Therefore:
A remote call is fundamentally different from a local function call.
This is one of the most important system-design principles.
Suppose:
Client
|
| request
v
ServerThe client sends:
Charge ₹1000The server processes it successfully.
But the response is lost:
Server → Client
XThe client sees:
timeoutWhat does the client know?
It does not know whether:
operation failedor:
operation succeeded but response was lostThis ambiguity is fundamental.
A partial failure occurs when some components fail while others continue working.
Example:
Node A ✓
Node B ✓
Node C ✗
Node D ✓The system as a whole may continue.
This differs from a single-machine failure where the whole process may simply stop.
Distributed systems must reason about:
Who failed?
Who can still communicate?
Who knows who failed?Suppose:
Node A → Node BA sends a request.
No response.
Possible explanations:
A cannot always distinguish these cases immediately.
Therefore:
A timeout is evidence of uncertainty, not proof that the remote operation never happened.
Every distributed call should have an appropriate timeout.
Without a timeout:
Service A
|
v
Service B
|
| hangs
v
Service A waits foreverThreads/connections can accumulate.
With a timeout:
request
|
v
wait
|
timeout
v
fallback/error/retryTimeouts prevent indefinite waiting.
This distinction is extremely important.
Suppose:
Client → Payment ServicePayment processing takes:
5 secondsClient timeout:
2 secondsThe client sees:
timeoutBut the payment may still succeed at:
5 secondsTherefore:
timeout ≠ operation definitely failedThis is why retrying non-idempotent operations can be dangerous.
Retries can recover from transient failures.
Example:
Request
|
X timeout
|
v
Retry
|
v
SuccessBut retries introduce a new problem:
duplicate operationsExample:
Charge ₹1000
|
v
payment succeeds
|
X response lost
|
v
client retriesThe payment might happen twice.
Therefore:
retry
+
idempotencymust be considered together.
Never blindly retry immediately:
retry
retry
retry
retryInstead:
1s
2s
4s
8s
16sThis gives the dependency time to recover.
Use a maximum retry limit.
Suppose:
10,000 clientsall experience a timeout.
If all retry immediately:
10,000 retriesThe already-failing service receives even more traffic.
This can make the failure worse.
This is a retry storm.
Use:
exponential backoff
+
jitter
+
retry limitsJitter adds randomness to retry delays.
Instead of:
all retry at 4 secondsyou may get:
3.7s
4.2s
4.8s
5.1sThis spreads load over time.
Consider:
API
|
v
Service A
|
v
Service B
|
v
DatabaseDatabase becomes slow.
Then:
Service B slows
↓
Service A waits
↓
API requests wait
↓
connections increase
↓
memory increases
↓
API becomes unhealthyOne dependency failure can cascade through the system.
Common techniques include:
timeouts
retries
circuit breakers
bulkheads
rate limiting
backpressure
load shedding
caching
fallbacksThese mechanisms protect services from downstream failures.
A circuit breaker monitors failures.
Conceptual states:
CLOSED
|
many failures
v
OPEN
|
wait
v
HALF-OPEN
|
success → CLOSED
failure → OPENRequests flow normally.
Requests are rejected or failed quickly.
A limited number of test requests are allowed.
Without one:
Service A
|
+→ B
+→ B
+→ B
+→ BIf B is down, A wastes resources repeatedly trying.
With a circuit breaker:
A
|
X
B unavailableA can fail fast.
This protects resources.
Bulkheads isolate resources.
Suppose one service has:
100 connection slotsDo not allow one dependency to consume all 100.
Instead:
Payment → 30
Search → 30
Other → 40If Search fails, it cannot consume all resources.
This is analogous to compartments in a ship.
When the system is overloaded, it may deliberately reject lower-priority work.
Example:
High priority:
payments
Low priority:
recommendationsDuring extreme load:
accept payments
reject/defer recommendationsThis preserves critical functionality.
Backpressure means controlling incoming work when processing capacity is limited.
Example:
Producer
|
v
Queue
|
v
WorkersIf workers are overloaded:
slow producer
or
buffer work
or
reject workBackpressure prevents unbounded resource consumption.
A distributed system consists of nodes.
A node may be:
Example:
Node A
Node B
Node CNodes communicate over a network.
A network partition occurs when groups of nodes cannot communicate.
Example:
Node A ←→ Node B
X
Node C ←→ Node DThe nodes may all still be running.
But:
A/B cannot reach C/DThis is different from node failure.
Suppose all nodes believe they are healthy.
Group 1:
A
BGroup 2:
C
DIf both sides accept writes independently, the system may create conflicting state.
Example:
Account balance
Group 1 → ₹500
Group 2 → ₹300After connectivity returns:
Which value is correct?Distributed systems need explicit consistency and conflict-resolution strategies.
Split brain occurs when multiple nodes or groups incorrectly believe they are authoritative.
Example:
Cluster
|
+---- Group A → Leader A
|
+---- Group B → Leader BBoth leaders may accept writes.
This can corrupt state.
Avoiding split brain is a major reason distributed coordination and quorum mechanisms exist.
A quorum is a minimum number of nodes required for an operation or decision.
Suppose:
5 nodesA majority quorum is:
3If a decision requires 3 nodes:
A ✓
B ✓
C ✓
D ✗
E ✗the system can still make progress if its protocol permits it.
Suppose:
N = 5
quorum = 3Two separate groups cannot both have a majority:
Group A = 3
Group B = 3because that would require 6 distinct nodes.
Majority overlap helps prevent conflicting decisions.
The exact protocol matters, but this is the intuition behind quorum systems.
Replication means keeping copies of data or state on multiple nodes.
Example:
Primary
|
+---- Replica 1
|
+---- Replica 2Benefits:
Costs:
A common model:
Leader
/ \
v v
Replica A Replica BWrites go to the leader.
Followers replicate.
If leader fails:
Replica A
|
v
New LeaderLeader election is then required.
Leader election is the process of choosing one node to coordinate or act as leader.
Conceptually:
Node A
Node B
Node COne becomes:
LeaderThe others become:
FollowersIf the leader fails, another eligible node can be elected.
The cluster needs to answer:
Who is the leader?while dealing with:
Two nodes must not incorrectly act as leader at the same time.
This is why consensus algorithms are important.
Consensus means a group of nodes agrees on a value or sequence of decisions despite failures.
Examples of consensus algorithms include:
Raft
PaxosAt this stage, focus on the intuition:
nodes
↓
communicate
↓
agree
↓
commit decisionConsensus is much more than simple replication.
These are not the same.
Replication:
copy dataConsensus:
agree on authoritative decisions/orderA replicated system may use consensus to decide:
which write is committed
who is leader
what log entry comes nextDistributed nodes have separate clocks.
Example:
Node A: 10:00:00.100
Node B: 10:00:00.300They differ by:
200 msThis is clock skew.
Clock synchronization is never perfectly guaranteed.
Suppose:
Node A says:
event happened at 10:00:01
Node B says:
event happened at 09:59:59Sorting events purely by local timestamps can be incorrect.
This affects:
Wall clock:
2026-10-01 10:00:00can move forward or backward due to synchronization adjustments.
Monotonic clocks are useful for measuring durations:
start
|
v
operation
|
v
end
duration = end - startFor timeouts and elapsed time, monotonic time is generally safer than relying on wall-clock timestamps.
Distributed systems sometimes use logical clocks to reason about event ordering without relying entirely on physical clocks.
One example is:
Lamport clockBasic idea:
event → logical counterIf A sends a message to B:
A event = 5
B receives → counter becomes at least 6This can represent causal ordering.
Suppose:
A creates order
|
v
B processes payment
|
v
C creates shipmentThere is a causal relationship:
OrderCreated
→
PaymentCompleted
→
ShipmentCreatedCausality matters when events can arrive out of order.
Availability describes how often a system is operational and able to serve requests.
Common targets:
99%
99.9%
99.99%
99.999%Higher availability usually requires more infrastructure and careful failure handling.
Conceptually:
Availability =
uptime / total timeFor example:
99.9%means approximately:
0.1%of the measurement period is unavailable.
Availability targets must be associated with a defined measurement window and service scope.
Reliability is broader than availability.
A system can be:
availablebut still return incorrect results.
Reliability includes the probability that the system performs its intended function correctly over time.
Examples:
correct payment
correct balance
correct order stateFault tolerance means the system continues operating correctly despite certain failures.
Example:
3 replicas
Replica A ✗
Replica B ✓
Replica C ✓The service can continue.
Fault tolerance requires knowing:
which failures are tolerated?
how many?
under what assumptions?High availability:
Keep the service accessible.
Fault tolerance:
Continue correct operation despite specified failures.
They overlap but are not identical.
A system may remain reachable while producing incorrect results, which is not acceptable fault tolerance.
Scalability describes how a system handles increased workload.
Horizontal scaling:
1 server
↓
10 serversVertical scaling:
4 CPU
↓
32 CPUDistributed systems commonly favor horizontal scaling.
Latency is the time taken for an operation.
Example:
request → response = 100 msIn distributed systems, latency accumulates across network hops.
Example:
API
↓ 20ms
Service A
↓ 30ms
Service B
↓ 50ms
DatabaseTotal may approach:
100ms+depending on concurrency and overhead.
Average latency can hide bad experiences.
Suppose:
99 requests → 50 ms
1 request → 5 secondsAverage may not look terrible.
But the 99th percentile:
P99 ≈ 5 secondsreveals the problem.
Important metrics include:
P50
P90
P95
P99
P99.9Suppose one request calls:
10 downstream servicesEach service has a small chance of being slow.
The probability that at least one dependency is slow increases.
Therefore fan-out can increase tail latency.
Architecture should minimize unnecessary synchronous fan-out.
Sometimes multiple nodes need exclusive access to a resource.
Example:
Job SchedulerTwo workers should not both process the same singleton job.
A distributed lock can coordinate ownership.
Conceptually:
Worker A
|
v
Lock
|
X
Worker BDistributed locks are tricky because leases can expire and network partitions can occur.
A distributed lock may have an expiration.
Example:
lock owner = Worker A
lease = 30 secondsIf Worker A stops renewing:
lease expiresAnother worker may acquire the lock.
But this creates a subtle danger.
Suppose:
Worker A
acquires lockThen the network isolates A.
The lock expires.
Worker B
acquires lockNow A may still continue executing.
A thinks it owns lock
B thinks it owns lockThis is why robust distributed locking sometimes requires fencing tokens.
A coordination service can issue increasing tokens:
Worker A → token 10
Worker B → token 11The protected resource accepts only the newest valid token.
If A continues after its lease expires:
A → token 10 → reject
B → token 11 → acceptThis prevents stale owners from performing protected operations.
A distributed system often needs to suspect failed nodes.
But remember:
no responsedoes not prove:
node crashedThe system therefore uses timeouts and health signals to make a best-effort failure suspicion.
False suspicions are possible.
Nodes may send heartbeats:
A → "I am alive"
B → "I am alive"
C → "I am alive"If a node stops sending heartbeats:
suspect nodeHeartbeats are useful but not proof of global health.
A node may be alive but unable to serve requests.
Different health checks can answer different questions.
Is the process alive?
Can the instance serve traffic?
Can it reach required dependencies?
Do not blindly restart a healthy application because a temporary downstream dependency is unavailable.
When a dependency fails, return reduced functionality when possible.
Example:
Recommendation Service ✗Main product page can still work:
Product ✓
Price ✓
Cart ✓
Recommendations unavailableThis is often better than failing the entire request.
A fallback can use:
cache
default response
stale data
simplified computationExample:
Recommendation API
|
X
|
v
Cached recommendationsFallbacks should be deliberate.
Serving incorrect data can be worse than returning an error.
State may be spread across nodes.
Example:
Node A → state X
Node B → state X
Node C → state XKeeping replicas consistent is difficult.
Questions include:
Who can write?
Who can read?
When do replicas update?
What if one is unavailable?
What if writes conflict?A strong consistency model aims to make successful reads observe the latest committed state according to the system's consistency definition.
Example:
Write X
|
v
Read
|
v
XThis can require coordination.
With eventual consistency:
Write X
|
+→ Replica A = X
|
+→ Replica B = old valueAfter propagation:
Replica A = X
Replica B = XTemporary divergence is accepted.
Real systems may choose different guarantees:
strong
session/read-your-writes
causal
eventualThe exact semantics matter.
Do not simply label a system:
consistentwithout specifying what reads are guaranteed to observe.
During a network partition, a distributed system may have to choose how it behaves.
For some operations:
continue accepting writesmay produce divergent state.
For others:
reject writesmay preserve consistency but reduce availability.
This is the foundation for the CAP theorem, which we will study in depth in Chapter 23.
A transaction across multiple independent systems is difficult.
Example:
Service A DB
Service B DB
Service C DBA single local transaction cannot normally atomically cover all three.
Approaches include:
The right choice depends on requirements.
Suppose:
Client → ServerServer performs operation.
Response is lost.
Client retries.
Now the system must determine:
Did the first operation happen?Without an idempotency mechanism, exactly-once business behavior is difficult.
This is a recurring distributed-systems theme:
Ambiguous outcomes require explicit protocol design.
Important principles:
Do not assume nodes are always healthy.
Never wait forever.
Use idempotency.
Use backoff and jitter.
Use bulkheads and circuit breakers.
Do not leave it implicit.
Coordination adds latency and failure dependencies.
Logs, metrics, traces, events.
Suppose every request requires:
A → B
B → C
C → DThe system now depends on all of them.
More coordination means:
more latency
+
more failure points
+
more operational complexityAsynchronous communication can reduce some dependencies but introduces eventual consistency.
System design is therefore a trade-off.
Consider:
Client
|
v
Payment API
|
v
Payment Service
|
v
BankFailure cases:
Bank rejects payment.
Simple failure.
Bank succeeds but response is lost.
Ambiguous result.
Payment API times out but bank is still processing.
Retry could create duplicate payment.
Solution:
idempotency key
+
payment status lookup
+
safe retryArchitecture:
Client
|
v
Load Balancer
|
+---- App A
+---- App B
+---- App C
|
v
DatabaseScaling introduces:
shared stateIf URL mappings are cached:
App A → Cache
App B → Cache
App C → Cachecache consistency becomes relevant.
If the database is replicated:
Primary
|
+→ Replica A
+→ Replica Bread-after-write behavior becomes relevant.
One simple application now becomes a distributed system with multiple failure modes.
Notification API
|
v
Queue
|
+---- Worker A
+---- Worker B
+---- Worker CFailures include:
worker crash
queue delay
provider outage
duplicate message
rate limitDesign:
retry
+
backoff
+
idempotency
+
DLQ
+
provider rate limitingA useful design exercise is to list:
| Component | Failure | Effect | Mitigation |
|---|---|---|---|
| API server | crash | traffic lost | load balancing |
| Database | unavailable | writes fail | replication/failover |
| Network | partition | nodes cannot communicate | quorum/fallback |
| Queue | unavailable | async work delayed | durable HA deployment |
| Worker | crash | job incomplete | retry |
| External API | timeout | uncertain result | idempotency |
| Cache | failure | slower requests | DB fallback |
| Region | outage | regional traffic lost | multi-region |
This is a powerful system-design interview technique.
Instead of asking only:
How does the system work when everything is healthy?
also ask:
What happens when each component fails?
For every dependency:
What if it is slow?
What if it is unavailable?
What if response is lost?
What if it returns an error?
What if it returns stale data?
What if it recovers?This mindset separates robust distributed designs from happy-path designs.
Failure handling is incomplete without recovery.
Ask:
How does the system return to normal?Examples:
failed replica
→ rebuild
consumer crash
→ restart and resume offset
database failure
→ failover
network partition
→ reconnect
queue backlog
→ scale consumers and drainRecovery should be part of the design.
Some failures are larger:
entire region unavailablePossible strategies:
backup
replication
multi-region
warm standby
active-activeImportant metrics:
RPO
RTOWe will revisit these in the High Availability and Disaster Recovery chapters.
Recovery Point Objective answers:
How much data loss can the business tolerate?
Example:
RPO = 5 minutesmeans the recovery strategy should aim to lose no more than approximately five minutes of data under the defined disaster scenario.
Recovery Time Objective answers:
How quickly must the service recover?
Example:
RTO = 30 minutesmeans the business requires restoration within approximately 30 minutes for the defined scenario.
A service with:
99.99% availability targethas a finite error budget.
The exact allowed downtime depends on the measurement period.
This concept helps teams balance:
feature delivery
vs
reliability workWe will study operational reliability more deeply later.
Classic assumptions that often cause problems include:
These assumptions are dangerous.
Packets can be:
lost
duplicated
delayed
reorderedApplications need appropriate protocols.
Every network hop costs time.
Across regions:
Mumbai → Singaporeis not equivalent to:
process within one machineArchitecture must account for network distance.
Large payloads consume:
network
CPU
memory
storagePrefer:
compression
caching
CDNs
object storage
pagination
streamingwhere appropriate.
Traffic can be intercepted or manipulated without proper security controls.
Use:
TLS
authentication
authorization
network isolation
secrets managementCloud systems constantly change:
instances start
instances stop
pods move
IPs change
regions fail
load balancers rerouteService discovery and dynamic configuration are important.
Modern systems involve:
cloud provider
database team
platform team
application team
third-party servicesDifferent components may have different ownership and failure domains.
Every byte has a cost in:
latency
bandwidth
CPU
cloud billingLarge-scale system design must consider data movement.
Different paths may have different:
latency
bandwidth
packet loss
routing
security
availabilityDesign for variability.
When designing a system, repeatedly evaluate:
Consistency
Availability
LatencyIncreasing coordination for stronger consistency may increase latency or reduce availability during failures.
Reducing coordination may improve latency and availability but introduce stale data.
There is no universal best configuration.
Synchronous:
A → B
wait
responseGood when:
caller needs immediate answerAsynchronous:
A → queue/event
|
v
BGood when:
work can happen laterBut asynchronous architecture introduces:
eventual consistencyand additional operational complexity.
Stateless service:
Request 1 → Server A
Request 2 → Server BNo critical session state exists only in Server A.
This makes horizontal scaling easier.
Stateful systems maintain important state locally or on specific nodes.
They require additional mechanisms for:
Stateless application servers can be:
added
removed
restarted
replacedwithout losing important application state.
State is moved to shared systems:
Database
Cache
Object Storage
QueueThis improves horizontal scalability.
App A ─┐
App B ─┼→ Redis Cluster
App C ─┘The cache itself becomes distributed.
Now questions appear:
What if Redis node fails?
How is data partitioned?
How is replication handled?
What if cache is unavailable?Every distributed dependency creates another failure domain.
A failure domain is a set of resources that can fail together.
Examples:
process
VM
rack
availability zone
region
cloud providerGood high-availability designs spread replicas across independent failure domains.
Bad:
Replica A → same VM
Replica B → same VM
Replica C → same VMOne VM failure destroys all replicas.
Better:
AZ-A → Replica A
AZ-B → Replica B
AZ-C → Replica CRedundancy means having additional capacity or copies.
Examples:
3 application servers
3 database replicas
2 network paths
2 regionsBut redundancy only helps if components fail independently enough.
Three replicas in one failure domain may not provide three independent failure protections.
Suppose:
Replica A → AZ-A
Replica B → AZ-A
Replica C → AZ-AA whole availability-zone failure causes:
A ✗
B ✗
C ✗Therefore redundancy must consider correlated failures.
CAP theorem will be covered in detail next.
The intuition is:
During a network partition, a distributed system cannot simultaneously guarantee both:
strong consistencyand:
full availabilityunder the standard CAP model.
The important point is that the network partition is the trigger.
CAP does not simply mean:
pick any two of C, A, PWe will examine the exact meaning in Chapter 23.
Treating remote calls like local function calls.
No timeout.
Retrying without idempotency.
Retrying immediately.
Assuming timeout means operation failed.
Ignoring partial failure.
Assuming replicas automatically solve consistency.
Ignoring network partitions.
Using global locks unnecessarily.
Ignoring clock skew.
Scaling without considering failure domains.
Optimizing average latency while ignoring tail latency.
Using asynchronous processing without defining consistency expectations.
Having redundancy inside one failure domain.
When designing any distributed system, ask:
Requests/sec?
Users?
Data volume?
Peak traffic?Where is state stored?
Who owns it?HTTP?
gRPC?
Queue?
Events?What if each dependency fails?
What if it becomes slow?
What if response is lost?Strong?
Eventual?
Read-after-write?
Per-user?
Per-object?Single region?
Multi-AZ?
Multi-region?Retry?
Failover?
Replay?
Rebuild?
Restore?Metrics?
Logs?
Traces?
Alerts?Partial failure occurs when some components of a distributed system fail while other components continue operating. Because failures are not always globally visible, surviving nodes may have incomplete or stale information about the system's state.
A timeout prevents a caller from waiting indefinitely for a remote operation. However, a timeout only means the caller did not receive a response within the deadline; it does not prove that the remote operation did not succeed.
A retry can repeat a business operation whose first attempt may already have succeeded but whose response was lost. Therefore retries should be bounded, use backoff and jitter, and be paired with idempotency for operations where duplicates are dangerous.
Split brain occurs when multiple nodes or partitions of a distributed system independently believe they are authoritative and both make decisions that should have been made by a single leader or quorum.
A quorum is a sufficient subset of nodes required to make or accept a decision. Majority quorums are useful because two different majorities of the same fixed node set must overlap.
Replication creates multiple copies of data or state. Consensus provides a protocol for nodes to agree on decisions or an ordered sequence despite certain failures. Replicated systems may use consensus to coordinate leaders and committed state.
Averages can hide a small percentage of extremely slow requests. Percentiles such as P95 and P99 show the experience of slower requests and are especially important when a request depends on multiple downstream services.
Build a fault-tolerant distributed API.
Client
|
v
Load Balancer
|
+--------+--------+
| | |
v v v
App A App B App C
| | |
+--------+--------+
|
v
Redis Cache
|
v
DatabaseAdd:
timeouts
retries
circuit breaker
rate limiting
health checks
metrics
distributed tracingSimulate:
App A crashes
Database becomes slow
Cache becomes unavailable
Network request times out
Database connection pool exhausts
One replica becomes stale
One service returns 500Observe:
Does the system recover?
Does traffic move?
Does retry amplify load?
Does the user receive a safe response?Add:
3 application nodes
3 database replicas
leader election
health checks
automatic failover
distributed lock
idempotent requests
request tracingThen simulate a network partition:
Node A/B
X
Node CStudy what happens when nodes cannot communicate.
Do not attempt to implement a production consensus algorithm from scratch as a first exercise; focus first on understanding the failure model and protocol behavior.
[ ] Remote-call timeouts
[ ] Retry policy
[ ] Exponential backoff
[ ] Jitter
[ ] Retry limits
[ ] Idempotency
[ ] Circuit breakers
[ ] Bulkheads
[ ] Backpressure
[ ] Load shedding
[ ] Health checks
[ ] Graceful degradation
[ ] Replication
[ ] Failure domains
[ ] Quorum strategy
[ ] Leader election where required
[ ] Split-brain protection
[ ] Consistency model
[ ] Recovery strategy
[ ] Observability
[ ] Distributed tracing
[ ] Tail-latency monitoring
[ ] Disaster recoveryThe most important mental model from this chapter is:
Distributed System
|
+---- Network can fail
|
+---- Nodes can fail
|
+---- Messages can be delayed
|
+---- Responses can be lost
|
+---- Clocks can differ
|
+---- State can temporarily diverge
|
+---- Failures can be partialTherefore:
Timeout
+
Retry
+
Idempotency
+
Consistency
+
Coordination
+
Recovery
+
Observabilityare fundamental distributed-system tools.
The most important concepts are:
The core principle is:
Assume the network will fail.
Assume components will fail.
Assume responses can be lost.
Design the protocol so the system
remains correct under those failures.You should now be able to explain:
In the next chapter we will deeply understand CAP rather than memorizing the phrase "pick two."
We will learn:
Mark it as complete to track your mastery.