Nextgen Blog
HomeLearnAIQuantumRoboticsSecuritySpaceAbout
Nextgen Blog

By Anubhav — full stack engineer, IIT Mandi CSE. Deep dives into AI, software engineering, and the technologies shaping the next generation of tech.

Navigate

  • Home
  • Learn
  • Blog
  • Archive
  • About

Topics

  • Artificial Intelligence
  • Quantum Computing
  • Robotics
  • Cybersecurity
  • Space Tech

Connect

Built with ❤️ & By - Anubhav

© 2026 Nextgen Blog by Anubhav. All rights reserved.

Privacy PolicyTerms of Service
Phase 02 · Scaling, Traffic & Data StoresChapter 11 of 33
10.
Phase 02Read time ~34 minDifficulty advanced

Database Fundamentals

In this chapter
  • 01What You Will Learn
  • 02Why Do We Need Databases?
  • 03What Is a Database?
  • 04Database vs File System
  • 05File-based approach
  • 06What Problems Does a Database Solve?
  • 07Persistence
  • 08Efficient retrieval
  • 09Concurrent access
  • 10Data integrity
  • 11Transactions
  • 12Recovery
  • 13The Basic Relational Database Mental Model
  • 14What Is a Row?
  • 15What Is a Column?
  • 16What Is a Table?
  • 17What Is a Schema?
  • 18Primary Key
  • 19Why Primary Keys Matter in System Design
  • 20Types of Identifiers
  • 21Auto-increment integer
  • 22UUID
  • 23Application-generated IDs
  • 24Foreign Keys
  • 25Relationships
  • 26One-to-One
  • 27One-to-Many
  • 28Many-to-Many
  • 29Constraints
  • 30PRIMARY KEY
  • 31UNIQUE
  • 32NOT NULL
  • 33CHECK
  • 34FOREIGN KEY
  • 35CRUD
  • 36Create
  • 37Read
  • 38Update
  • 39Delete
  • 40Querying Data
  • 41Indexes
  • 42What Is an Index?
  • 43Example Without an Index
  • 44Example With an Index
  • 45The Trade-Off of Indexes
  • 46Composite Indexes
  • 47Selectivity
  • 48Query Pattern Comes First
  • 49Transactions
  • 50ACID
  • 51Atomicity
  • 52Consistency
  • 53Isolation
  • 54Durability
  • 55ACID Mental Model
  • 56Isolation Problems
  • 57Dirty Read
  • 58Non-Repeatable Read
  • 59Phantom Read
  • 60Why Transactions Matter in System Design
  • 61Normalization
  • 62Why Normalize?
  • 63Normalization Trade-Off
  • 64Denormalization
  • 65Denormalization
  • 66Normalization vs Denormalization
  • 67SQL Databases
  • 68Strengths of Relational Databases
  • 69NoSQL Databases
  • 70Document Databases
  • 71Key-Value Databases
  • 72Wide-Column Databases
  • 73Graph Databases
  • 74SQL vs NoSQL
  • 75Database Choice Is a Workload Decision
  • 76Read-Heavy vs Write-Heavy Workloads
  • 77Read-heavy
  • 78Write-heavy
  • 79Database Connection Pool
  • 80Why Connection Pools Matter
  • 81Connection Pool Is Not Database Scaling
  • 82Common Database Bottlenecks
  • 83CPU
  • 84Memory
  • 85Disk I/O
  • 86Lock contention
  • 87Connections
  • 88Poor indexes
  • 89Bad queries
  • 90Hot keys
  • 91The Database Is Often the Hardest Component to Scale
  • 92Database Replication — Preview
  • 93Database Partitioning — Preview
  • 94Primary Key vs Index
  • 95Primary key
  • 96Index
  • 97Index vs Cache
  • 98Index
  • 99Cache
  • 100Database vs Cache
  • 101Transactions vs Cache
  • 102Database as the Source of Truth
  • 103Read Path
  • 104Write Path
  • 105Database Availability
  • 106Backups vs Replication
  • 107Replication
  • 108Backup
  • 109Database Failure Modes
  • 110Timeouts Matter
  • 111Retries Can Make Database Failures Worse
  • 112Hot Rows and Hot Keys
  • 113Database Design Starts With Access Patterns
  • 114Example: E-Commerce Database
  • 115Important Indexes for This Example
  • 116Example: URL Shortener Database
  • 117Database Selection Checklist
  • 118What data model do we need?
  • 119What are the main queries?
  • 120What consistency is required?
  • 121What transaction requirements exist?
  • 122What is the workload?
  • 123What is the scale?
  • 124What are latency requirements?
  • 125What availability is required?
  • 126What operational capabilities are available?
  • 127Common Beginner Mistakes
  • 128Mistake 1: "NoSQL is always faster."
  • 129Mistake 2: "More indexes are always better."
  • 130Mistake 3: "Replication means backup."
  • 131Mistake 4: "Database scaling means adding more servers."
  • 132Mistake 5: "ACID consistency means distributed strong consistency."
  • 133Mistake 6: "Connection pool size should be as large as possible."
  • 134Mistake 7: "A cache can replace the database."
  • 135Mistake 8: "Every query should use an index."
  • 136Database Mental Model
  • 137System Design Decision Example
  • 138Database as Part of a Larger Architecture
  • 139Interview Questions
  • 140Beginner
  • 141Q1. What is a database?
  • 142Q2. What is a primary key?
  • 143Q3. What is a foreign key?
  • 144Q4. What is an index?
  • 145Q5. What is a transaction?
  • 146Intermediate
  • 147Q6. Explain ACID.
  • 148Q7. Why not index every column?
  • 149Q8. SQL vs NoSQL?
  • 150Q9. Why use a connection pool?
  • 151Q10. What is normalization?
  • 152Advanced Interview Questions
  • 153Q11. Why can a database become the bottleneck after application servers scale horizontally?
  • 154Q12. How would you scale a read-heavy database?
  • 155Q13. How would you scale a write-heavy database?
  • 156Q14. Why can increasing database connections make performance worse?
  • 157Q15. What happens if a database query becomes slow?
  • 158Practical Exercise
  • 159Mini Project — E-Commerce Database
  • 160Requirements
  • 161Query 1
  • 162Query 2
  • 163Query 3
  • 164Query 4
  • 165Query 5
  • 166Production Checklist
  • 167Data model
  • 168Query performance
  • 169Transactions
  • 170Capacity
  • 171Availability
  • 172Recovery
  • 173Operations
  • 174Chapter Summary
  • 175Master Checklist
  • 176What Comes Next?
  • 177Chapter 10 — SQL Databases
In this chapter (177 sections)
01What You Will Learn02Why Do We Need Databases?03What Is a Database?04Database vs File System05File-based approach06What Problems Does a Database Solve?07Persistence08Efficient retrieval09Concurrent access10Data integrity11Transactions12Recovery13The Basic Relational Database Mental Model14What Is a Row?15What Is a Column?16What Is a Table?17What Is a Schema?18Primary Key19Why Primary Keys Matter in System Design20Types of Identifiers21Auto-increment integer22UUID23Application-generated IDs24Foreign Keys25Relationships26One-to-One27One-to-Many28Many-to-Many29Constraints30PRIMARY KEY31UNIQUE32NOT NULL33CHECK34FOREIGN KEY35CRUD36Create37Read38Update39Delete40Querying Data41Indexes42What Is an Index?43Example Without an Index44Example With an Index45The Trade-Off of Indexes46Composite Indexes47Selectivity48Query Pattern Comes First49Transactions50ACID51Atomicity52Consistency53Isolation54Durability55ACID Mental Model56Isolation Problems57Dirty Read58Non-Repeatable Read59Phantom Read60Why Transactions Matter in System Design61Normalization62Why Normalize?63Normalization Trade-Off64Denormalization65Denormalization66Normalization vs Denormalization67SQL Databases68Strengths of Relational Databases69NoSQL Databases70Document Databases71Key-Value Databases72Wide-Column Databases73Graph Databases74SQL vs NoSQL75Database Choice Is a Workload Decision76Read-Heavy vs Write-Heavy Workloads77Read-heavy78Write-heavy79Database Connection Pool80Why Connection Pools Matter81Connection Pool Is Not Database Scaling82Common Database Bottlenecks83CPU84Memory85Disk I/O86Lock contention87Connections88Poor indexes89Bad queries90Hot keys91The Database Is Often the Hardest Component to Scale92Database Replication — Preview93Database Partitioning — Preview94Primary Key vs Index95Primary key96Index97Index vs Cache98Index99Cache100Database vs Cache101Transactions vs Cache102Database as the Source of Truth103Read Path104Write Path105Database Availability106Backups vs Replication107Replication108Backup109Database Failure Modes110Timeouts Matter111Retries Can Make Database Failures Worse112Hot Rows and Hot Keys113Database Design Starts With Access Patterns114Example: E-Commerce Database115Important Indexes for This Example116Example: URL Shortener Database117Database Selection Checklist118What data model do we need?119What are the main queries?120What consistency is required?121What transaction requirements exist?122What is the workload?123What is the scale?124What are latency requirements?125What availability is required?126What operational capabilities are available?127Common Beginner Mistakes128Mistake 1: "NoSQL is always faster."129Mistake 2: "More indexes are always better."130Mistake 3: "Replication means backup."131Mistake 4: "Database scaling means adding more servers."132Mistake 5: "ACID consistency means distributed strong consistency."133Mistake 6: "Connection pool size should be as large as possible."134Mistake 7: "A cache can replace the database."135Mistake 8: "Every query should use an index."136Database Mental Model137System Design Decision Example138Database as Part of a Larger Architecture139Interview Questions140Beginner141Q1. What is a database?142Q2. What is a primary key?143Q3. What is a foreign key?144Q4. What is an index?145Q5. What is a transaction?146Intermediate147Q6. Explain ACID.148Q7. Why not index every column?149Q8. SQL vs NoSQL?150Q9. Why use a connection pool?151Q10. What is normalization?152Advanced Interview Questions153Q11. Why can a database become the bottleneck after application servers scale horizontally?154Q12. How would you scale a read-heavy database?155Q13. How would you scale a write-heavy database?156Q14. Why can increasing database connections make performance worse?157Q15. What happens if a database query becomes slow?158Practical Exercise159Mini Project — E-Commerce Database160Requirements161Query 1162Query 2163Query 3164Query 4165Query 5166Production Checklist167Data model168Query performance169Transactions170Capacity171Availability172Recovery173Operations174Chapter Summary175Master Checklist176What Comes Next?177Chapter 10 — SQL Databases

Course: System Design — Zero to Hero
Level: Beginner → Intermediate
Phase: Database Fundamentals
Status: Complete


1. What You Will Learn#

In this chapter, we build the database foundation required for system design.

You will learn:

  1. What a database is
  2. Why applications need databases
  3. Database vs file system
  4. Data, records, rows, columns, and tables
  5. Schemas
  6. Primary keys
  7. Foreign keys
  8. Relationships
  9. One-to-one, one-to-many, and many-to-many relationships
  10. Constraints
  11. CRUD operations
  12. Indexes and why they matter
  13. Transactions
  14. ACID properties
  15. SQL vs NoSQL
  16. Relational vs non-relational data models
  17. Normalization and denormalization
  18. Read-heavy vs write-heavy workloads
  19. Database connection pools
  20. Database bottlenecks
  21. Database scaling overview
  22. Replication and sharding preview
  23. How to choose a database in system design
  24. Common mistakes and interview traps
  25. Practical exercises
  26. A database design mini-project

The goal is not merely to learn SQL syntax.

The goal is to understand how databases behave as system-design components.


2. Why Do We Need Databases?#

Imagine an application that stores users.

Without persistent storage, the application might keep data in memory:

users = [
    { id: 1, name: "Anubhav" },
    { id: 2, name: "Rahul" }
]

This works temporarily.

But what happens when:

  • the server restarts?
  • another server needs the data?
  • 1 million users exist?
  • multiple requests update the same user?
  • you need to search users?
  • you need transactions?
  • you need backups?
  • multiple application servers access the same data?

In-memory data is not enough.

We need a persistent storage system.

That is where databases come in.


3. What Is a Database?#

A database is a system for storing, organizing, retrieving, and modifying data.

At a high level:

Application
     |
     v
 Database
     |
     +---- Store data
     +---- Retrieve data
     +---- Update data
     +---- Delete data
     +---- Protect data integrity
     +---- Manage concurrent access

A database provides much more than storage.

It also provides mechanisms for:

  • querying
  • indexing
  • transactions
  • concurrency control
  • durability
  • constraints
  • recovery
  • replication
  • backup

4. Database vs File System#

A beginner may ask:

Why not simply save everything into files?

You can.

For a small application:

users.json
products.json
orders.json

might work.

But large applications need more sophisticated capabilities.

File-based approach#

Suppose we have:

users.json

containing:

[
  {
    "id": 1,
    "name": "Anubhav",
    "email": "anubhav@example.com"
  }
]

Searching for a user may require reading a large portion of the file.

Now imagine:

100 million users

The problem becomes much harder.

A database can maintain indexes that allow efficient lookup.

It can also coordinate concurrent reads and writes.


5. What Problems Does a Database Solve?#

A production database helps solve several problems.

5.1 Persistence#

Data survives application restarts.

Application restart
       |
       v
Database still contains data

5.2 Efficient retrieval#

Instead of scanning every record, indexes can make targeted queries much faster.


5.3 Concurrent access#

Many users can access the system simultaneously.

User A ----\
User B -----\
User C ------> Database
User D -----/
User E ----/

The database manages concurrent operations according to its consistency and transaction model.


5.4 Data integrity#

Databases can enforce rules such as:

email must be unique
user_id cannot be NULL
order must reference an existing user

5.5 Transactions#

A database can group multiple operations into one logical unit.

Example:

Transfer ₹1,000
 
Debit Account A
Credit Account B

We generally don't want:

Debit succeeds
Credit fails

A transaction can make the operations succeed or fail together according to the database's transaction semantics.


5.6 Recovery#

Databases provide mechanisms for recovering from failures.

Examples include:

  • logs
  • checkpoints
  • backups
  • replication
  • recovery procedures

6. The Basic Relational Database Mental Model#

A relational database organizes data into tables.

For example:

Users
 
+----+----------+---------------------+
| id | name     | email               |
+----+----------+---------------------+
| 1  | Anubhav  | anubhav@example.com  |
| 2  | Rahul    | rahul@example.com    |
| 3  | Priya    | priya@example.com    |
+----+----------+---------------------+

A table contains:

  • rows
  • columns

7. What Is a Row?#

A row represents one record.

Example:

1 | Anubhav | anubhav@example.com

This represents one user.

You can think of a row as:

One instance of an entity.

For example:

User #1
User #2
User #3

8. What Is a Column?#

A column represents an attribute.

For example:

id
name
email
created_at

Each user has values for those attributes.

Example:

id = 1
name = Anubhav
email = anubhav@example.com

9. What Is a Table?#

A table is a structured collection of records.

Example:

users
orders
products
payments
reviews

Each table normally represents an entity or a useful relationship.


10. What Is a Schema?#

A schema describes the structure and rules of stored data.

For example:

users
 
id          INTEGER
name        VARCHAR
email       VARCHAR
created_at  TIMESTAMP

The schema defines things such as:

  • columns
  • data types
  • constraints
  • relationships

Conceptually:

Database
   |
   +-- users
   |     +-- id
   |     +-- name
   |     +-- email
   |
   +-- orders
   |     +-- id
   |     +-- user_id
   |     +-- amount
   |
   +-- products

11. Primary Key#

A primary key uniquely identifies a row.

Example:

users
 
id | name
---+---------
1  | Anubhav
2  | Rahul
3  | Priya

Here:

id

is the primary key.

The important property is uniqueness.

id = 1

should identify exactly one user.


12. Why Primary Keys Matter in System Design#

Suppose an application receives:

GET /users/123

The database needs an efficient way to locate user 123.

A primary-key lookup is usually an important access pattern.

Primary keys also become references from other tables.

Example:

users
-----
id = 123

Then:

orders
------
id = 9001
user_id = 123

The order belongs to user 123.


13. Types of Identifiers#

Common identifiers include:

Auto-increment integer#

1
2
3
4

Advantages:

  • compact
  • easy to index
  • simple

Potential concerns in distributed systems:

  • coordination
  • predictable IDs
  • allocation across multiple writers

UUID#

Example:

550e8400-e29b-41d4-a716-446655440000

Advantages:

  • globally unique identifiers can be generated independently
  • useful in distributed systems

Potential concerns:

  • larger storage
  • index characteristics
  • less human-readable

Application-generated IDs#

Applications may use identifiers such as:

order_01J...
user_...
payment_...

The important system-design question is not:

Which ID is universally best?

Instead ask:

What are the application's generation, ordering, indexing, storage, and distribution requirements?


14. Foreign Keys#

A foreign key represents a relationship between records.

Example:

users
 
id
---
1
2
3

And:

orders
 
id | user_id
---+--------
10 | 1
11 | 1
12 | 2

orders.user_id references users.id.

Conceptually:

User
 |
 +---- Order
 |
 +---- Order

15. Relationships#

There are three common relationship types.

15.1 One-to-One#

One record corresponds to one record.

Example:

User ---- Profile

A user may have one profile.


15.2 One-to-Many#

One record can have many related records.

Example:

User
 |
 +---- Order
 |
 +---- Order
 |
 +---- Order

This is extremely common.


15.3 Many-to-Many#

Many records relate to many other records.

Example:

Students <----> Courses

A student can enroll in many courses.

A course can contain many students.

Usually, a junction table is introduced:

students
courses
student_courses

Example:

student_courses
 
student_id | course_id
-----------+----------
1          | 101
1          | 102
2          | 101

16. Constraints#

Constraints help protect data integrity.

Common constraints include:

PRIMARY KEY#

Ensures a unique row identifier.

UNIQUE#

Example:

email UNIQUE

Prevents duplicate values.

NOT NULL#

Requires a value.

name NOT NULL

CHECK#

Enforces a condition.

Example:

price >= 0

FOREIGN KEY#

Maintains a relationship between tables.


17. CRUD#

CRUD represents the four basic data operations.

C = Create
R = Read
U = Update
D = Delete

Example:

Create#

INSERT INTO users (name, email)
VALUES ('Anubhav', 'anubhav@example.com');

Read#

SELECT * FROM users;

Update#

UPDATE users
SET name = 'Anubhav Chaudhary'
WHERE id = 1;

Delete#

DELETE FROM users
WHERE id = 1;

These operations form the foundation of application data access.


18. Querying Data#

A database becomes powerful when we can retrieve exactly what we need.

Example:

SELECT id, name
FROM users
WHERE email = 'anubhav@example.com';

The database may use an index to avoid scanning every row.

This brings us to one of the most important system-design concepts:

Indexes#


19. What Is an Index?#

An index is an additional data structure that helps a database find rows efficiently.

Think of a book.

Without an index:

Read every page
until you find the topic

With an index:

Topic
  |
  v
Page number

A database index works with the same broad idea.


20. Example Without an Index#

Suppose:

users = 100 million rows

We run:

SELECT *
FROM users
WHERE email = 'anubhav@example.com';

Without a suitable index, the database may need to examine many rows.

Conceptually:

Row 1   check
Row 2   check
Row 3   check
...
Row 100M check

This is a scan.


21. Example With an Index#

Suppose we create:

CREATE INDEX idx_users_email
ON users(email);

The database now has an additional structure organized for lookup.

Conceptually:

email
  |
  +---- anubhav@example.com --> row location
  +---- rahul@example.com   --> row location
  +---- priya@example.com   --> row location

The exact implementation depends on the database and index type.


22. The Trade-Off of Indexes#

Indexes are not free.

They can improve reads but increase costs elsewhere.

More indexes
    |
    +--> faster eligible reads
    |
    +--> more storage
    |
    +--> additional write/update work
    |
    +--> more memory/cache pressure

Therefore:

Do not create an index for every column.

Design indexes around real query patterns.


23. Composite Indexes#

Suppose the application frequently runs:

SELECT *
FROM orders
WHERE user_id = 123
AND created_at >= '2026-01-01';

A composite index may be useful:

(user_id, created_at)

The order of columns matters.

Indexes should be designed around access patterns rather than simply listing every field.


24. Selectivity#

An index is often more useful when the indexed condition narrows the result set substantially.

Suppose:

gender

has only:

Male
Female

while:

email

is almost unique.

An email lookup generally has much higher selectivity than a gender lookup.

This does not mean a low-cardinality column can never be indexed.

The actual usefulness depends on:

  • query shape
  • data distribution
  • database engine
  • workload
  • table size
  • optimizer decisions

25. Query Pattern Comes First#

A common beginner mistake is:

"Let's add indexes to important columns."

A better approach is:

  1. Identify important queries.
  2. Measure their frequency.
  3. Inspect query plans.
  4. Determine filtering/sorting/join requirements.
  5. Design appropriate indexes.
  6. Measure again.

System design is workload-driven.


26. Transactions#

A transaction groups operations into a logical unit of work.

Consider money transfer:

Account A = ₹10,000
Account B = ₹5,000

Transfer:

₹1,000

Desired final state:

A = ₹9,000
B = ₹6,000

We do not want:

A = ₹9,000
B = ₹5,000

because money disappeared.

Nor:

A = ₹10,000
B = ₹6,000

because money appeared.

A transaction helps coordinate the operations.

Conceptually:

BEGIN
 
Debit A
Credit B
 
COMMIT

If something fails:

ROLLBACK

27. ACID#

ACID is one of the most important database concepts for system design interviews.

ACID stands for:

A = Atomicity
C = Consistency
I = Isolation
D = Durability

28. Atomicity#

Atomicity means a transaction is treated as one logical unit.

Conceptually:

Transaction
 
Operation A
Operation B
Operation C
 
All succeed
     OR
Transaction fails/rolls back

It prevents an incomplete transaction from being treated as successfully completed.


29. Consistency#

Consistency means transactions move the database between valid states according to its defined constraints and rules.

For example:

balance >= 0

if such a constraint/business rule is enforced.

A transaction should not leave the database violating applicable integrity constraints.

Important:

Database consistency in ACID is not the same concept as distributed-system consistency discussed later in this course.

This distinction is an important interview point.


30. Isolation#

Multiple transactions may execute concurrently.

Example:

Transaction A
Transaction B
Transaction C

Isolation controls how concurrent transactions interact and what intermediate states they can observe.

Different isolation levels provide different guarantees.

Common SQL isolation levels include:

Read Uncommitted
Read Committed
Repeatable Read
Serializable

Exact behavior varies by database engine.


31. Durability#

After a transaction is successfully committed, its effects are intended to survive failures according to the database's durability guarantees.

Conceptually:

COMMIT
   |
   v
Database failure
   |
   v
Committed data remains recoverable

Durability commonly involves mechanisms such as:

  • transaction logs
  • WAL/write-ahead logging
  • stable storage
  • checkpoints
  • recovery procedures
  • replication and backups

Replication alone should not automatically be equated with durability.


32. ACID Mental Model#

A useful interview memory trick:

Atomicity  -> All or nothing
Consistency -> Valid state
Isolation  -> Concurrent transactions don't improperly interfere
Durability -> Committed data survives failures according to guarantees

33. Isolation Problems#

Concurrent transactions can produce anomalies depending on the isolation model.

Important examples include:

Dirty Read#

Transaction B reads data written by transaction A before A commits.

A writes X
B reads X
A rolls back

B observed data that did not ultimately commit.


Non-Repeatable Read#

A transaction reads the same row twice and gets different committed values because another transaction changed it between reads.


Phantom Read#

A transaction repeats a query and sees a different set of matching rows because another transaction inserted or deleted rows.


34. Why Transactions Matter in System Design#

Not every operation needs a large transaction.

Consider:

Create user

versus:

Place order
Reserve inventory
Create payment record
Update order state

The second workflow has stronger coordination requirements.

System design requires asking:

Which operations must succeed together?

That question helps determine transaction boundaries.


35. Normalization#

Normalization organizes relational data to reduce unnecessary duplication and update anomalies.

Suppose we store:

orders
 
order_id
customer_name
customer_email
product_name
product_price

If a customer has 1,000 orders, their information may be repeated many times.

We might separate:

customers
orders
products

and connect them with identifiers.


36. Why Normalize?#

Normalization can reduce:

  • duplicate data
  • inconsistent updates
  • storage waste
  • update anomalies

For example, if a customer changes their email:

Without normalization:

Update 1,000 order records

With normalization:

Update customer record once

37. Normalization Trade-Off#

Normalization is not automatically always better.

Highly normalized schemas may require more joins.

For read-heavy systems, joins can become expensive or inconvenient depending on workload and architecture.

This leads to:

Denormalization#


38. Denormalization#

Denormalization intentionally duplicates some data to improve access patterns.

Example:

Instead of:

orders
  |
  +--> customers

we might store:

orders
 
order_id
customer_id
customer_name
customer_email

This duplicates customer data.

Why?

Perhaps the application frequently needs:

Order + customer display information

and wants to avoid an additional lookup.


39. Normalization vs Denormalization#

Concept Normalization Denormalization
Duplication Lower Higher
Consistency management Often simpler More complex
Reads May require joins Can be faster/simpler
Writes Less duplicated data May require multiple updates
Storage Usually lower Usually higher
Typical use Transactional models Read-heavy/access-pattern optimized models

The important system-design lesson:

Model the data around correctness and access patterns.


40. SQL Databases#

SQL databases are relational databases that generally organize data into tables and support SQL for querying.

Examples include:

  • PostgreSQL
  • MySQL
  • MariaDB
  • Microsoft SQL Server
  • Oracle Database

A typical model:

Users
Orders
Products
Payments

with relationships between them.


41. Strengths of Relational Databases#

Relational databases are especially useful when you need:

  • structured data
  • relationships
  • transactions
  • constraints
  • complex queries
  • joins
  • strong integrity guarantees

Example:

E-commerce
 
Customer
   |
   +---- Orders
            |
            +---- Order Items
                       |
                       +---- Products

Relational modeling naturally represents these relationships.


42. NoSQL Databases#

NoSQL is a broad family of non-relational database technologies.

Common categories include:

Document
Key-Value
Wide-Column
Graph

Examples:

MongoDB       -> Document
Redis         -> Key-Value
Cassandra     -> Wide-Column
Neo4j         -> Graph

These systems have different data models and operational characteristics.

"NoSQL" does not mean:

No structure.

It generally means the system is not based on the traditional relational model.


43. Document Databases#

A document database stores records as documents.

Example:

{
  "id": 101,
  "name": "Anubhav",
  "email": "anubhav@example.com",
  "skills": [
    "JavaScript",
    "Node.js",
    "System Design"
  ]
}

This can be useful when an entity naturally maps to a document.


44. Key-Value Databases#

The model is approximately:

key -> value

Example:

user:123 -> {...}
session:abc -> {...}

They are often useful for:

  • caching
  • sessions
  • counters
  • simple lookups
  • ephemeral state

Redis is a common example.


45. Wide-Column Databases#

Wide-column databases are designed around distributed storage and query patterns involving large datasets.

A common example is Cassandra.

They can be useful for workloads requiring:

  • high write throughput
  • horizontal scaling
  • distributed storage
  • predictable query patterns

The schema and query model should be designed around access patterns.


46. Graph Databases#

Graph databases represent:

Nodes
Edges
Properties

Example:

Anubhav
   |
   | follows
   v
Rahul
   |
   | follows
   v
Priya

They can be useful for relationship-heavy queries such as:

  • social networks
  • recommendation relationships
  • fraud networks
  • dependency graphs

47. SQL vs NoSQL#

Avoid the beginner rule:

SQL is old and NoSQL is modern.

That is incorrect.

The correct question is:

Which data model and operational characteristics fit the workload?

Consider:

Requirement Often a natural fit
Complex relationships Relational
Strong transactional workflows Relational
Flexible document-shaped data Document
Extremely fast key lookup/cache Key-value
Very large distributed write workload with known access patterns Wide-column
Relationship traversal Graph

These are general tendencies, not absolute rules.


48. Database Choice Is a Workload Decision#

Suppose you are designing a payment system.

Important requirements might include:

Correctness
Transactions
Auditability
Consistency
Durability

A relational database may be a natural candidate.

Now consider a session store:

session_id -> session_data

A key-value system may be a natural candidate.

Now consider social relationships:

User A follows User B
User B follows User C
User C follows User D

A graph-oriented approach may be useful for certain traversal-heavy workloads.


49. Read-Heavy vs Write-Heavy Workloads#

System design depends heavily on the read/write ratio.

Read-heavy#

Example:

100,000 reads/sec
10,000 writes/sec

Possible techniques:

  • caching
  • read replicas
  • denormalization
  • indexes
  • CDNs for suitable content

Write-heavy#

Example:

10,000 reads/sec
100,000 writes/sec

Possible techniques:

  • batching
  • asynchronous processing
  • partitioning
  • queue-based buffering
  • write-optimized data models
  • carefully designed indexes

The exact solution depends on workload characteristics.


50. Database Connection Pool#

Applications generally do not want to establish a brand-new database connection for every request.

That would create overhead.

Instead, applications often use a connection pool.

Application
     |
     v
Connection Pool
  |  |  |  |
  v  v  v  v
Database connections

Requests borrow connections from the pool and return them when finished.


51. Why Connection Pools Matter#

Suppose:

10,000 application requests/sec

but the database can only handle a much smaller number of concurrent connections efficiently.

If every request creates a connection:

Connection storm
      |
      v
Database overload

A bounded pool can provide backpressure.

Example:

Pool size = 100

Requests beyond the available connections may wait, time out, or be rejected according to application behavior.


52. Connection Pool Is Not Database Scaling#

An important distinction:

Connection pool

controls how the application uses database connections.

It does not magically increase database capacity.

If the database is already overloaded:

Increase pool from 100
to
1,000

may make the problem worse.

More concurrency can create more database contention.


53. Common Database Bottlenecks#

Databases can become bottlenecks because of:

CPU#

Complex queries consume CPU.

Memory#

Insufficient memory can increase disk I/O.

Disk I/O#

Large scans or write-heavy workloads can stress storage.

Lock contention#

Concurrent transactions may wait for one another.

Connections#

Too many active connections can overload the database.

Poor indexes#

Queries may perform expensive scans.

Bad queries#

Examples:

SELECT *

when only a few fields are needed.

Or queries that unintentionally scan huge datasets.

Hot keys#

A disproportionate amount of traffic targets the same record or partition.


54. The Database Is Often the Hardest Component to Scale#

Application servers are frequently easy to scale horizontally:

Server 1
Server 2
Server 3
Server 4

The database is harder because state must be coordinated.

You cannot always solve database scaling by simply adding another identical server.

This is why system design eventually requires:

  • replication
  • partitioning
  • sharding
  • caching
  • asynchronous processing
  • read/write separation

We will study these in later chapters.


55. Database Replication — Preview#

Replication means maintaining copies of data across multiple database nodes.

Conceptually:

             +--> Replica 1
Primary -----+
             +--> Replica 2

Potential benefits:

  • read scaling
  • high availability
  • disaster resilience

But replication introduces questions:

  • How quickly do replicas receive updates?
  • Can reads be stale?
  • What happens when primary fails?
  • How is a new primary selected?
  • How are conflicting writes handled?

These are later system-design topics.


56. Database Partitioning — Preview#

Partitioning divides data into smaller pieces.

For example:

Users 0–99M
       |
       +---- Partition A
       +---- Partition B
       +---- Partition C

Partitioning can improve scalability and manageability.

When partitioning is distributed across database nodes, it is commonly called sharding.

We will study this deeply later.


57. Primary Key vs Index#

This is a common interview question.

Primary key#

Defines the primary identity of a row.

Index#

Provides an access path that can make queries faster.

A primary key is generally backed by an index in relational database systems, but:

Not every index is a primary key.

Example:

Primary key:
id
 
Additional indexes:
email
created_at
status
(user_id, created_at)

58. Index vs Cache#

Another common confusion.

Index#

Helps the database locate data efficiently.

Query
  |
  v
Database index
  |
  v
Stored row/data

Cache#

Stores frequently used results/data closer to the application.

Application
     |
     v
Cache
  |
  +--> hit -> return
  |
  +--> miss -> Database

An index and cache solve different problems.

They are often used together.


59. Database vs Cache#

Suppose:

User profile

is requested 100,000 times.

The database can serve it.

But if the same data is repeatedly requested, caching can reduce database load.

Without cache:
 
100,000 requests
       |
       v
Database
       |
       v
High load

With cache:

100,000 requests
       |
       v
Cache
       |
       +---- Most requests -> cache
       |
       +---- Misses -> database

This connects directly to the previous CDN/caching chapter.


60. Transactions vs Cache#

A cache is generally not a replacement for the system of record.

For example:

Database -> source of truth
Cache    -> performance layer

If cached data disappears:

Cache failure
    |
    v
Fetch from database

This is one reason cache architecture should define:

  • source of truth
  • cache invalidation
  • TTL
  • stale-data tolerance
  • failure behavior

61. Database as the Source of Truth#

For many transactional applications:

Application
    |
    +--> Cache
    |
    +--> Database

The database may be the authoritative source.

For example:

Product price = ₹999

The cache might contain:

₹999

but the database is generally the durable record.

This distinction becomes critical when designing consistency behavior.


62. Read Path#

A common read architecture:

Client
  |
  v
Application
  |
  v
Cache
  |
  +---- Hit ----> Response
  |
  +---- Miss
          |
          v
       Database
          |
          v
       Cache
          |
          v
       Response

This is called a cache-aside pattern when the application explicitly loads data into the cache after a miss.


63. Write Path#

A simplified write flow:

Client
  |
  v
Application
  |
  v
Database
  |
  v
Cache update/invalidation

The exact order matters.

If:

Database updated
Cache not invalidated

the application may serve stale data.

If:

Cache updated
Database write fails

the cache could temporarily contain data that was never committed.

Therefore, cache/database consistency requires deliberate design.


64. Database Availability#

A single database instance creates a single failure domain.

Application
     |
     v
Database
     X
   failure

Potential result:

Application unavailable

Production architectures may use:

Primary
  |
  +---- Replica
  |
  +---- Replica

along with failover mechanisms.

Availability design introduces trade-offs around:

  • failover time
  • replication lag
  • consistency
  • split-brain
  • operational complexity

65. Backups vs Replication#

This is an important distinction.

Replication#

Maintains additional copies, often for availability or read scaling.

Backup#

Creates recoverable historical copies for restoration.

If bad data is accidentally written:

Primary
   |
   +--> Replica

the bad change may replicate.

A backup may allow recovery to an earlier point.

Therefore:

Replication is not a substitute for backups.


66. Database Failure Modes#

Consider:

Application
     |
     v
Database

Possible failures include:

Database process crashes
Disk fails
Network partition
CPU saturation
Memory exhaustion
Connection exhaustion
Long-running query
Lock contention
Replication lag
Corrupted data

A robust system considers failure explicitly.


67. Timeouts Matter#

Never assume a database query will always return quickly.

A query can become slow because of:

  • locks
  • overloaded CPU
  • disk pressure
  • network problems
  • bad query plans
  • huge scans

Applications should use appropriate:

Connection timeout
Query timeout
Request timeout

Without timeouts:

Slow database
    |
    v
Requests wait
    |
    v
Connection pool fills
    |
    v
More requests wait
    |
    v
Application becomes unhealthy

This is a cascading failure pattern.


68. Retries Can Make Database Failures Worse#

Suppose the database is overloaded.

Application receives:

timeout

and immediately retries:

request
retry
retry
retry
retry

Now database load increases further.

This can produce a retry storm.

Use:

  • bounded retries
  • exponential backoff
  • jitter
  • idempotency where appropriate
  • sensible timeouts

We will revisit this in distributed systems.


69. Hot Rows and Hot Keys#

Imagine:

product_id = 123

is extremely popular.

Millions of requests target:

product:123

or a single database row.

Even if the database is large, one record or partition can become a bottleneck.

This is called a hot key/hot partition pattern.

Potential solutions include:

  • caching
  • replication
  • request coalescing
  • partitioning strategies
  • workload redesign
  • counters designed for distributed updates

The correct solution depends on the workload.


70. Database Design Starts With Access Patterns#

Do not start with:

What tables should I create?

Start with:

What does the application need to do?

For an e-commerce application:

Find product by ID
Search products
Find orders by user
Find order by ID
Update inventory
Create payment
List recent orders

Then design:

  • schema
  • indexes
  • transactions
  • caching
  • partitioning strategy

around those access patterns.


71. Example: E-Commerce Database#

A simplified relational model:

Users
-----
id
name
email
 
Products
--------
id
name
price
inventory
 
Orders
------
id
user_id
status
created_at
 
OrderItems
----------
id
order_id
product_id
quantity
price

Relationships:

User
 |
 +---- Orders
         |
         +---- OrderItems
                    |
                    +---- Product

72. Important Indexes for This Example#

Possible access patterns:

Find user by email
Find orders by user
Find order by ID
Find order items by order
Find product by ID

Possible indexes:

users(email)
orders(user_id, created_at)
order_items(order_id)

The exact index design should be validated against actual queries and database behavior.


73. Example: URL Shortener Database#

Suppose:

short_code -> long_url

A table could look like:

urls
 
id
short_code
long_url
created_at
user_id

Critical query:

Find URL by short_code

Therefore:

short_code

needs an efficient lookup path, usually with a uniqueness constraint/index.

Read workload may be much larger than write workload.

That suggests possible future techniques:

Database
   |
   +--> Cache
   |
   +--> Read replicas

This connects database design to system scaling.


74. Database Selection Checklist#

When choosing a database, ask:

1. What data model do we need?#

Relational?
Document?
Key-value?
Wide-column?
Graph?

2. What are the main queries?#

Point lookup?
Range query?
Join?
Aggregation?
Graph traversal?

3. What consistency is required?#

Strong?
Eventual?
Application-defined?

4. What transaction requirements exist?#

Single-row?
Multi-row?
Multi-table?
Cross-service?

5. What is the workload?#

Read-heavy?
Write-heavy?
Mixed?

6. What is the scale?#

10K rows?
100M rows?
10B rows?

7. What are latency requirements?#

Milliseconds?
Seconds?
Batch?

8. What availability is required?#

Single region?
Multi-zone?
Multi-region?

9. What operational capabilities are available?#

Backups
Monitoring
Failover
Migration
Capacity management

75. Common Beginner Mistakes#

Mistake 1: "NoSQL is always faster."#

Not necessarily.

Performance depends on:

  • workload
  • schema
  • query patterns
  • indexes
  • hardware
  • implementation
  • scale

Mistake 2: "More indexes are always better."#

No.

Indexes consume storage and add work to writes.


Mistake 3: "Replication means backup."#

No.

Replication and backup solve different problems.


Mistake 4: "Database scaling means adding more servers."#

Sometimes.

But database scaling can involve:

Vertical scaling
Caching
Read replicas
Partitioning
Sharding
Query optimization
Denormalization
Storage optimization

Mistake 5: "ACID consistency means distributed strong consistency."#

Not exactly.

The word "consistency" appears in multiple contexts.

Always explain which consistency model you mean.


Mistake 6: "Connection pool size should be as large as possible."#

No.

Too many connections can overwhelm the database.


Mistake 7: "A cache can replace the database."#

Usually not for durable transactional state.


Mistake 8: "Every query should use an index."#

Not necessarily.

For small tables or queries returning a large percentage of rows, a sequential scan can sometimes be more appropriate.

The database optimizer decides based on statistics and cost estimates.


76. Database Mental Model#

Remember this architecture:

                  +----------------+
                  |  Application   |
                  +-------+--------+
                          |
                          v
                  +----------------+
                  | Connection     |
                  | Pool           |
                  +-------+--------+
                          |
                          v
                  +----------------+
                  | Database       |
                  +----------------+
                    |     |      |
                    v     v      v
                 Tables Indexes Transactions
                    |
                    v
                 Storage

At scale:

                    Application
                         |
                 +-------+-------+
                 |               |
               Cache        Connection Pool
                                 |
                                 v
                         +---------------+
                         | Primary DB    |
                         +-------+-------+
                                 |
                      +----------+----------+
                      |                     |
                      v                     v
                  Replica 1             Replica 2

Later:

                    Application
                         |
                         v
                     Router
                         |
              +----------+----------+
              |          |          |
            Shard 1    Shard 2    Shard 3

This is the progression we will build throughout the course.


77. System Design Decision Example#

Suppose we are designing:

Food delivery application

Requirements:

Millions of users
High read traffic
Frequent order writes
Payment transactions
Restaurant inventory
Location updates

We might use different storage technologies for different workloads.

Conceptually:

Users / Orders / Payments
          |
          v
      Relational DB
 
Hot product/restaurant data
          |
          v
        Cache
 
Location / high-frequency event data
          |
          v
   Specialized system
 
Analytics
          |
          v
 Data warehouse/lake

The key principle:

One system does not have to store every kind of data.

A large architecture can use multiple specialized storage systems.


78. Database as Part of a Larger Architecture#

A database rarely exists alone.

A realistic architecture might be:

Client
  |
  v
CDN
  |
  v
Load Balancer
  |
  v
Application Servers
  |
  +---------> Cache
  |
  +---------> Database
  |
  +---------> Message Queue
                    |
                    v
               Worker Services

The database is one component in the complete system.


79. Interview Questions#

Beginner#

Q1. What is a database?#

A system for persistent storage, retrieval, modification, and management of data.

Q2. What is a primary key?#

A column or set of columns that uniquely identifies a row.

Q3. What is a foreign key?#

A field that references a key in another table to represent a relationship.

Q4. What is an index?#

An auxiliary data structure that can make eligible queries more efficient.

Q5. What is a transaction?#

A logical unit of database work with defined transactional guarantees.


Intermediate#

Q6. Explain ACID.#

Atomicity
Consistency
Isolation
Durability

Q7. Why not index every column?#

Because indexes consume storage and can increase write/update cost and maintenance overhead.

Q8. SQL vs NoSQL?#

Compare based on:

  • data model
  • relationships
  • consistency
  • transactions
  • query patterns
  • scale
  • operational characteristics

Q9. Why use a connection pool?#

To reuse a bounded set of database connections and avoid creating a new connection for every request.

Q10. What is normalization?#

Structuring relational data to reduce unnecessary duplication and anomalies.


80. Advanced Interview Questions#

Q11. Why can a database become the bottleneck after application servers scale horizontally?#

Because all application servers may converge on the same database for shared state.

10 App Servers
      |
      v
One Database
      |
      v
Potential bottleneck

Q12. How would you scale a read-heavy database?#

Potential techniques:

Caching
Read replicas
Indexes
Query optimization
Denormalization
CDN for suitable data
Partitioning where appropriate

Q13. How would you scale a write-heavy database?#

Potential techniques:

Batching
Asynchronous processing
Partitioning
Sharding
Write-optimized data modeling
Reducing unnecessary indexes
Queue-based buffering

Q14. Why can increasing database connections make performance worse?#

Because more concurrent work can increase:

  • CPU contention
  • memory usage
  • lock contention
  • context switching
  • disk pressure

Q15. What happens if a database query becomes slow?#

Potential chain:

Slow query
   |
   v
Request waits
   |
   v
Connection remains occupied
   |
   v
Pool fills
   |
   v
New requests wait
   |
   v
Timeouts
   |
   v
Retry storm
   |
   v
System instability

This is a classic distributed-system failure pattern.


81. Practical Exercise#

Design the database model for a blogging platform.

Requirements:

Users can create posts.
 
Posts can have comments.
 
Users can like posts.
 
Posts can have multiple tags.
 
Users can follow other users.

Create:

users
posts
comments
likes
tags
post_tags
follows

Then identify:

  1. Primary keys
  2. Foreign keys
  3. One-to-many relationships
  4. Many-to-many relationships
  5. Important indexes
  6. Queries that need optimization
  7. Which operations require transactions

82. Mini Project — E-Commerce Database#

Design a simplified e-commerce database.

Requirements#

Users can:

Register
Browse products
Create orders
Add products to orders
Pay for orders
View order history

Create these entities:

users
products
orders
order_items
payments

Design:

Primary keys
Foreign keys
Constraints
Indexes

Then answer:

Query 1#

Find product by ID.

Query 2#

Find all orders for a user.

Query 3#

Find a specific order.

Query 4#

Find all items in an order.

Query 5#

Find recent orders for a user.

For each query, ask:

What index would support this access pattern?


83. Production Checklist#

Before considering a database design production-ready, ask:

Data model#

  • Is the schema understandable?
  • Are relationships correct?
  • Are constraints defined?

Query performance#

  • What are the most common queries?
  • Are indexes appropriate?
  • Have query plans been examined?

Transactions#

  • Which operations must be atomic?
  • What isolation level is appropriate?

Capacity#

  • How many reads/sec?
  • How many writes/sec?
  • How much data?
  • How quickly will data grow?

Availability#

  • What happens if the database fails?
  • Is replication needed?
  • How does failover work?

Recovery#

  • Are backups configured?
  • Can point-in-time recovery be performed if required?
  • Has restoration been tested?

Operations#

  • Are metrics monitored?
  • Are slow queries tracked?
  • Are connection pools bounded?
  • Are timeouts configured?

84. Chapter Summary#

You now have the foundation needed to reason about databases in system design.

Key concepts:

Database
   |
   +-- Tables
   |     +-- Rows
   |     +-- Columns
   |
   +-- Keys
   |     +-- Primary
   |     +-- Foreign
   |
   +-- Constraints
   |
   +-- Indexes
   |
   +-- Transactions
   |     +-- ACID
   |
   +-- Data Models
         +-- Relational
         +-- Document
         +-- Key-Value
         +-- Wide-Column
         +-- Graph

At system-design scale:

Database
   |
   +-- Query optimization
   +-- Indexing
   +-- Connection pooling
   +-- Caching
   +-- Replication
   +-- Partitioning
   +-- Sharding
   +-- Backups
   +-- Recovery
   +-- Availability

The most important mindset is:

Do not choose a database because it is popular. Choose the data model and architecture based on workload, access patterns, consistency, transaction, availability, and scaling requirements.


85. Master Checklist#

Before moving forward, you should be able to explain:

  • What a database is
  • Why applications need databases
  • Database vs file system
  • Table
  • Row
  • Column
  • Schema
  • Primary key
  • Foreign key
  • One-to-one relationship
  • One-to-many relationship
  • Many-to-many relationship
  • Constraints
  • CRUD
  • Index
  • Composite index
  • Selectivity
  • Transactions
  • ACID
  • Isolation
  • Dirty reads
  • Non-repeatable reads
  • Phantom reads
  • Normalization
  • Denormalization
  • SQL databases
  • NoSQL databases
  • Document databases
  • Key-value databases
  • Wide-column databases
  • Graph databases
  • Read-heavy workloads
  • Write-heavy workloads
  • Connection pooling
  • Database bottlenecks
  • Replication basics
  • Partitioning basics
  • Backup vs replication
  • Database vs cache
  • Source of truth
  • Hot keys
  • Database timeouts
  • Retry storms
  • Database selection criteria

86. What Comes Next?#

Chapter 10 — SQL Databases#

Next we move from database fundamentals into SQL databases in depth.

We will learn:

SQL database architecture
        ↓
Tables and schemas
        ↓
SELECT / INSERT / UPDATE / DELETE
        ↓
WHERE / ORDER BY / GROUP BY
        ↓
JOINs
        ↓
Aggregations
        ↓
Subqueries
        ↓
Transactions
        ↓
Isolation levels
        ↓
Query execution
        ↓
EXPLAIN / query plans
        ↓
Indexes in SQL databases
        ↓
Constraints
        ↓
Database design
        ↓
Production SQL patterns

We will also start connecting SQL concepts directly to system-design performance and scalability decisions.

Finished this chapter?

Mark it as complete to track your mastery.

Previous ChapterChapter 08 — CDN & Edge ComputingNext ChapterChapter 10 — SQL Databases
PrevNext