Applications eventually need to store data.
So far, we studied:
But databases are not the only storage systems.
Modern applications often use several storage layers:
Application
|
+-------------------+-------------------+
| | |
v v v
Database Object Storage Cache
| | |
Structured Files / Media Fast Temporary
Data Data DataA large production system may store:
The key idea is:
Different types of data require different storage characteristics.
A storage system provides a way to persist and retrieve data.
At a high level:
Application
|
v
Storage
|
+---- Write
|
+---- ReadA storage system can be evaluated using:
There is no single storage technology that is optimal for every workload.
A simplified storage hierarchy looks like:
Fastest
|
v
CPU Cache
Memory / RAM
Local NVMe SSD
SSD
HDD
Network Storage
Object Storage
Archival Storage
|
v
Slowest / CheapestGenerally:
This is why large systems use storage tiers.
Local storage is directly attached to the machine.
Example:
Application Server
|
v
Local SSDAdvantages:
Disadvantages:
Local storage is often useful for:
Network storage is accessed over a network.
Application Server
|
| Network
v
Storage SystemAdvantages:
Disadvantages:
Block storage exposes storage as blocks that an operating system can treat like a disk.
Conceptually:
Application
|
v
Operating System
|
v
Block Device
|
v
Storage SystemThe application generally sees:
/dev/deviceor an equivalent block device.
The operating system can place a filesystem on it.
Block storage is useful for workloads such as:
Example:
Database Server
|
v
Filesystem
|
v
Block StorageA database may depend heavily on predictable storage latency and IOPS.
File storage exposes files and directories.
Conceptually:
/
├── users
│ ├── user1
│ └── user2
├── uploads
└── reportsApplications interact using filesystem-like operations:
open
read
write
rename
deleteFile storage is useful when applications need:
Object storage stores data as objects.
Conceptually:
Bucket
|
+-- object-1
+-- object-2
+-- image.jpg
+-- video.mp4An object usually consists of:
Object
|
+-- Data
+-- Metadata
+-- Identifier / KeyUnlike a traditional filesystem, object storage is generally accessed through APIs rather than arbitrary filesystem operations.
A simplified architecture:
Application
|
| API
v
Object Storage
|
+---- Object
+---- Metadata
+---- ReplicationCommon object-storage characteristics include:
Object storage is extremely useful for large unstructured data.
| Property | Block | File | Object |
|---|---|---|---|
| Interface | Blocks | Files/directories | Objects/API |
| Typical use | Databases/VM disks | Shared files | Media/backups/data lakes |
| Hierarchical filesystem | Usually via filesystem | Yes | Not traditionally |
| Scalability | High | High | Extremely high |
| Access model | Low-level | Filesystem | API |
| Large unstructured data | Possible | Good | Excellent |
Choosing storage depends on the application's access pattern.
Two important storage media are:
Hard Disk Drive.
Uses spinning magnetic disks.
Advantages:
Disadvantages:
Solid State Drive.
Uses flash memory.
Advantages:
Disadvantages:
NVMe is a protocol designed for high-speed access to non-volatile memory, commonly SSDs.
A simplified hierarchy:
HDD
|
v
SSD
|
v
NVMe SSDNVMe devices can provide very high throughput and low latency.
They are useful for:
Storage latency is the time required to complete a storage operation.
Example:
Application
|
v
Storage
|
v
ResponseIf:
Storage latency = 5 msthen a storage operation takes approximately that amount of time under the relevant measurement conditions.
Latency matters heavily for transactional workloads.
Throughput measures how much data can be transferred over time.
Example:
500 MB/secmeans approximately 500 megabytes of data can be transferred per second under the relevant conditions.
High throughput is important for:
IOPS means:
Input/Output Operations Per Second
Suppose a storage system provides:
100,000 IOPSIt can handle a large number of individual I/O operations per second under the relevant workload.
IOPS is especially important for workloads involving many small random operations.
Consider two workloads.
1,000,000 tiny operations/secNeeds high IOPS.
1 large sequential file/secMay be dominated by throughput.
Therefore:
IOPS ≠ ThroughputA storage system can have:
Workload characteristics matter.
Sequential:
Block 1
Block 2
Block 3
Block 4
Block 5Random:
Block 7
Block 100
Block 2
Block 900
Block 41Sequential workloads often achieve higher throughput.
Random workloads generally depend more heavily on IOPS and latency.
Durability answers:
What is the probability that stored data remains available over time despite failures?
Example:
Data written
|
v
Storage
|
+---- Disk failure
+---- Node failure
+---- Rack failure
+---- Region failureA durable storage system uses techniques such as:
These concepts are related but different.
Data is not lost.
The system can serve requests.
Example:
Data still exists
but
storage service is temporarily unreachableThen:
Durability = preserved
Availability = temporarily reducedA strong architecture considers both.
Storage systems may replicate data:
Object
|
+---- Copy A
+---- Copy B
+---- Copy CIf one copy fails:
Copy A -> FAILED
Copy B -> available
Copy C -> availableThe system can continue operating.
Replication consumes additional storage capacity.
Suppose:
Replication factor = 3A logical dataset of:
100 TBcould require roughly:
300 TBof raw replicated capacity before accounting for metadata and other overhead.
Replication improves redundancy but increases cost.
Instead of storing complete copies, storage can use erasure coding.
Conceptually:
Original Data
|
v
Encode
|
+---- Data Chunk A
+---- Data Chunk B
+---- Data Chunk C
+---- Parity Chunk P
+---- Parity Chunk QIf some chunks are lost, the original data can be reconstructed from the remaining chunks.
Erasure coding can provide redundancy with less storage overhead than full replication for some workloads.
| Property | Replication | Erasure Coding |
|---|---|---|
| Simplicity | Higher | More complex |
| Storage overhead | Higher | Often lower |
| Read/rebuild complexity | Lower | Higher |
| Failure recovery | Straightforward | Requires reconstruction |
| Common use | Hot/critical data | Large durable datasets |
The exact trade-offs depend on the implementation.
Storage systems need to detect corrupted data.
A checksum can be computed:
Data
|
v
ChecksumLater:
Stored Data
|
v
Recalculate checksum
|
v
CompareIf they differ:
Possible corruptionChecksums are important because storage failures are not always obvious.
An object may contain:
Key
Size
Content-Type
Creation Time
Version
Custom Metadata
ChecksumExample:
Key:
users/123/profile.jpg
Metadata:
Content-Type = image/jpeg
Size = 2 MBMetadata enables systems to locate, classify, and process objects efficiently.
Object storage commonly organizes data into buckets/containers and keys.
Example:
bucket:
nextgen-assets
key:
users/123/photos/profile.jpgThe key acts as the object's identifier.
Although the key may contain /, object storage does not necessarily implement a traditional filesystem hierarchy.
This distinction matters.
Suppose a user uploads:
5 GB videoA bad architecture may send the entire file through application servers:
Client
|
v
Application Server
|
v
Object StorageNow the application server handles:
This can be inefficient.
A better architecture often allows direct upload to object storage.
Architecture:
Client
|
| 1. Request upload authorization
v
Application
|
| 2. Signed upload URL
v
Client
|
| 3. Upload directly
v
Object StorageThe application does not carry the full file payload.
Benefits:
A pre-signed URL gives a client temporary permission to access an object-storage operation.
Simplified flow:
Client
|
| "I want to upload file"
v
Application
|
| Generate signed URL
v
Client
|
| PUT file
v
Object StorageThe signed URL can have:
The exact capabilities depend on the object-storage provider.
A production upload flow might be:
1. Authenticate user
2. Validate file metadata
3. Generate unique object key
4. Generate short-lived signed upload URL
5. Client uploads directly
6. Storage event generated
7. Processing worker validates/processes file
8. Database records file metadataThis separates:
File bytesfrom:
Application metadataSuppose a user uploads an image.
Client
|
v
Object Storage
|
v
Storage Event
|
v
Processing Queue
|
v
Image Worker
|
+---- resize
+---- thumbnail
+---- validation
+---- metadata extraction
|
v
Processed ObjectsThis is an event-driven storage architecture.
Large files can be split into parts.
Example:
10 GB file
Part 1
Part 2
Part 3
...
Part NThe client uploads parts independently.
Advantages:
After all parts arrive:
Parts
|
v
Object Storage
|
v
Complete ObjectSuppose:
5 GB uploadfails at:
4.7 GBWithout resumability:
Start againWith resumability:
Resume from 4.7 GBThis is especially important for:
For large objects:
Client
|
v
CDN
|
v
Object StorageThe CDN can cache popular content.
This prevents every download from reaching the origin storage.
A common architecture:
Users
|
v
CDN
/ \
HIT MISS
| |
v v
Response Object Storage
|
v
ResponseOn a cache miss:
Object Storage -> CDN -> UserFuture requests may be served directly from the CDN.
For a video platform:
User
|
| Upload
v
Object Storage
|
v
Processing Queue
|
v
Transcoding Workers
|
+---- 360p
+---- 720p
+---- 1080p
+---- 4K
|
v
Object Storage
|
v
CDN
|
v
UsersThis architecture separates:
Not all data needs the same performance.
A system may classify:
Hot
Warm
Cold
ArchiveFrequently accessed.
Needs:
Occasionally accessed.
Can use lower-cost storage.
Rarely accessed.
Optimized for cost.
Very rarely accessed.
Optimized heavily for long-term retention cost.
Data can automatically move between tiers.
Example:
Day 0
|
v
Hot Storage
Day 30
|
v
Warm Storage
Day 180
|
v
Cold Storage
Day 365
|
v
ArchiveThis can reduce storage costs.
Every storage system should answer:
How long do we need to keep this data?
Examples:
Logs -> 30 days
Reports -> 1 year
Audit records -> several years
Temporary uploads -> 24 hours
Backups -> according to recovery policyRetention policies prevent unlimited storage growth.
Unused objects can accumulate.
Example:
Temporary upload
|
X processing failed
|
v
Orphaned objectA cleanup worker can find objects that are no longer referenced.
Object Storage
|
v
Cleanup Job
|
v
Delete OrphansThis is especially important when storage costs scale with data volume.
A common architecture is:
PostgreSQL
|
+-- file_id
+-- owner_id
+-- object_key
+-- size
+-- status
+-- created_at
Object Storage
|
+-- actual file bytesThe database stores metadata.
Object storage stores large binary content.
This separation is often much more scalable than storing huge blobs directly in the application database.
Small binary objects can sometimes be stored in databases.
But for large media workloads, object storage is often a better fit.
A common design is:
Database:
metadata
Object Storage:
binary contentThis keeps transactional metadata separate from large file payloads.
The correct choice depends on workload and database capabilities.
Backups themselves require storage.
Architecture:
Database
|
v
Backup Process
|
v
Object Storage
|
v
Archive TierA backup strategy should define:
A backup that has never been restored successfully should not be treated as proven recovery capability.
For disaster recovery:
Region A
|
v
Object Storage A
|
| Replication
v
Object Storage B
|
v
Region BThis can protect against regional failures.
But cross-region replication can involve:
Storage systems can have different consistency semantics.
Questions include:
After writing an object,
when should a read see it?
After deleting an object,
when should the deletion become visible?
If two clients update metadata,
which value wins?The exact behavior depends on the storage technology.
For system design, always identify the consistency guarantees required by the application.
Suppose a storage system has:
99.9% availabilityThat means downtime can still occur.
Applications should consider:
For critical files, redundant storage locations may be appropriate.
Storage architecture is also a cost problem.
Suppose:
100 TB dataWith replication factor 3:
~300 TB raw capacityWith erasure coding, the raw overhead may be lower depending on the coding scheme.
Then add:
Therefore:
Capacity planning must include total lifecycle cost, not just disk size.
Suppose:
10,000 userseach upload:
10 MB/secPeak ingest:
10,000 × 10 MB/sec
= 100,000 MB/sec
≈ 100 GB/secThis is an enormous ingest workload.
The system needs to consider:
Back-of-the-envelope estimation is essential.
Suppose uploads arrive faster than processing can handle them.
Uploads
|
v
Queue
|
v
WorkersIf:
Incoming = 10,000 files/sec
Processing = 5,000 files/secthe queue grows.
Eventually:
Queue
###########
###########
###########The system needs backpressure mechanisms such as:
Large file processing is often asynchronous.
Upload
|
v
Object Storage
|
v
Event
|
v
Queue
|
v
WorkersThe queue decouples:
Upload speedfrom:
Processing speedThis prevents temporary processing slowdowns from immediately blocking uploads.
A distributed file system spreads files across multiple machines.
Conceptually:
Client
|
v
Metadata Service
|
+----------+----------+
| | |
v v v
Node A Node B Node C
data data dataThe system may maintain:
Distributed file systems are useful when applications need filesystem-like semantics at large scale.
Suppose:
10 billion filesThe data bytes may be distributed easily enough, but metadata such as:
file -> location
file -> permissions
file -> owner
file -> timestampscan become a bottleneck.
Therefore large storage systems often design metadata services carefully.
Potential techniques include:
Suppose object storage contains:
1 billion fileseach only:
1 KBThe total payload is only about:
1 TBbut metadata overhead can be enormous.
Many tiny files can create:
Sometimes applications bundle small records into larger objects or use specialized storage formats.
The opposite problem is:
1 file = 10 TBChallenges include:
Solutions include:
For large objects, clients may need only part of the file.
Example:
10 GB videoClient requests:
Bytes 100 MB - 110 MBInstead of downloading the entire object.
Range reads are useful for:
Storage often contains sensitive data.
Security measures can include:
Stored data -> encryptedClient <--- encrypted connection ---> StorageOnly authorized identities can access objects.
Applications should receive only the permissions they require.
Record:
Avoid putting sensitive information directly into object keys.
Bad:
users/987654321/medical-record.pdfif keys can be exposed.
A safer design may use opaque identifiers:
objects/7f3a91c2...and keep sensitive metadata in the protected database.
A secure architecture may use:
Private Bucket
|
X
Direct public access disabled
Application
|
v
Authorization
|
v
Short-lived signed URL
|
v
UserThis allows temporary access without making the entire bucket public.
Production systems should consider:
Disk -> FAILEDStorage Node -> DOWNMultiple nodes may fail.
Entire regional infrastructure may become unavailable.
Data may exist but be incorrect.
Valid data may be removed.
Unauthorized access may occur.
Each requires different protection.
Suppose:
User accidentally deletes fileIf deletion is replicated:
Primary
|
v
DELETE
|
+---- Replica A
+---- Replica BAll copies may disappear.
Therefore we still need:
Replication is not a substitute for backup.
Object versioning can preserve previous versions.
Example:
file.pdf
|
+-- Version 1
+-- Version 2
+-- Version 3If Version 3 is accidentally deleted or corrupted, an earlier version may remain recoverable.
Versioning increases storage consumption, so lifecycle policies are important.
Instead of immediately deleting:
DELETE filethe system can mark it as deleted:
deleted = trueand physically remove it later.
This creates a recovery window.
Example:
Day 0:
Deleted
Day 1:
Recoverable
Day 30:
Permanent deletionThis is useful for user-facing recovery and operational safety.
A mature storage system may look like:
Upload
|
v
Hot Storage
|
| after 30 days
v
Warm Storage
|
| after 180 days
v
Cold Storage
|
| after 1 year
v
Archive
|
| after retention period
v
DeleteThis controls cost while preserving required retention.
Suppose users upload photos.
Architecture:
Mobile/Web Client
|
v
API Server
|
Generate upload URL
|
v
Object Storage
|
Storage Event
|
v
Queue
|
+---------+---------+
| | |
v v v
Resize Thumbnail Metadata
| | |
+---------+---------+
|
v
Object Storage
|
v
CDN
|
v
UsersDatabase stores:
photo_id
user_id
object_key
thumbnail_key
status
created_atObject storage stores:
original photo
processed photo
thumbnailThis separates metadata from large binary content.
Uploader
|
v
Signed Upload URL
|
v
Object Storage
|
v
Event
|
v
Queue
|
v
Transcoding Workers
|
+---- 360p
+---- 720p
+---- 1080p
+---- 4K
|
v
Object Storage
|
v
CDN
|
v
ViewersDatabase stores:
video_id
owner_id
title
status
duration
object referencesThe video bytes remain in object storage.
Design a file-storage service with:
10 million users
100 million files
Average file = 5 MBRequirements:
Estimate:
100 million × 5 MB
= 500 million MB
≈ 500 TBThen consider:
Your actual storage requirement will be significantly larger than the raw 500 TB.
Build a local file-storage service.
Features:
POST /upload
GET /files/:id
DELETE /files/:id
GET /files/:id/downloadArchitecture:
Client
|
v
Node.js / Next.js API
|
+---- Metadata DB
|
+---- Local Object DirectoryImplement:
Then extend it with:
Storage
|
v
Queue
|
v
Thumbnail WorkerThis prepares you for real media-storage architectures.
Users upload videos up to 20 GB. Your application servers are becoming overloaded. How would you redesign the upload architecture?
A strong approach:
Client
|
| Request upload authorization
v
Application
|
| Short-lived signed URL
v
Client
|
| Multipart upload
v
Object Storage
|
| Event
v
Queue
|
v
Processing WorkersBenefits:
Think of storage design in this order:
STORAGE DESIGN
|
+-------------+-------------+
| | |
ACCESS PERFORMANCE DURABILITY
| | |
Block/File Latency Replication
Object IOPS Erasure Coding
Throughput Backups
| | |
+-------------+-------------+
|
CAPACITY
|
+--------+--------+
| | |
Size Growth Cost
|
v
LIFECYCLE
|
Hot -> Warm -> Cold
|
v
SECURITY
|
+-------------+-------------+
| | |
Encryption IAM/Auth AuditingThe core questions are:
1. What kind of data are we storing?
2. How will it be accessed?
3. How much data will we have?
4. How fast must reads/writes be?
5. What happens when storage fails?
6. How durable must the data be?
7. How long must we keep it?
8. How much will it cost?
9. How will users upload/download large objects?
10. How will we recover from accidental deletion?In this chapter, we learned:
The central idea is:
Storage architecture is about matching the data and access pattern to the right storage model while balancing latency, throughput, durability, availability, capacity, security, and cost.
Make sure you can explain:
Next we specialize the storage concepts into a complete production-grade file and media platform.
We will design:
Mark it as complete to track your mastery.