Scalability

What Is Scalability?

A system is scalable if it can handle increased load by adding resources, without requiring a redesign. Scalability is not about being big today — it's about being able to grow tomorrow without a crisis.

Load can mean many things: more users, more requests per second, more data stored, more concurrent connections, or geographically distributed users. A scalable design handles all of these gracefully.

Vertical Scaling (Scaling Up)

Vertical scaling means adding more power to the existing machine: more CPU cores, more RAM, faster disks.

Advantages:

• Simple — no architectural changes
• No coordination overhead between machines
• Works immediately

Disadvantages:

• Hard ceiling — there is a maximum machine size
• Expensive at the top end (the cost curve is non-linear)
• Single point of failure — one machine means one failure domain
• Downtime required for hardware upgrades

When it's the right choice: Early-stage systems, databases that are hard to distribute, or workloads that require large amounts of shared memory (e.g., in-memory caches, OLAP queries over large datasets).

Horizontal Scaling (Scaling Out)

Horizontal scaling means adding more machines and distributing the load across them.

Advantages:

• Theoretically unlimited — add as many machines as needed
• Commodity hardware — cheap, replaceable, no vendor lock-in
• High availability — one machine failing doesn't take down the system
• Can scale down as well as up

Disadvantages:

• Complexity — requires load balancers, service discovery, distributed coordination
• State becomes a problem — data must be shared or replicated across nodes
• Network becomes a bottleneck and failure domain

When it's the right choice: Web servers, API servers, stateless compute layers — anything that can be designed to be stateless.

Stateless vs Stateful Services

The key to horizontal scaling is statelessness.

A stateless service holds no data between requests. Every request carries all the information needed to process it. Any instance can handle any request.

Client → Load Balancer → Server A
                       → Server B   ← any of these can handle the next request
                       → Server C

A stateful service holds session data locally. Routing a request to the wrong instance produces incorrect behavior.

Client → Load Balancer → Server A (has session for user 42)
                       → Server B (does not know user 42)  ← wrong answer

How to make stateful services scalable:

1. Externalize state — move sessions into a shared store (Redis, DynamoDB). Servers become stateless; the store scales independently.
2. Sticky sessions — configure the load balancer to always route a given client to the same server. Works, but reduces flexibility and creates uneven load.
3. Client-side state — store state in signed cookies or JWTs. No server-side storage needed. Limited by cookie size and security constraints.

In practice: externalize state. Sticky sessions are an antipattern at scale.

The Capacity Planning Mindset

Before designing for scale, estimate what you're designing for.

A useful estimation framework:

1. Requests per second (RPS):

   • 1M daily active users × 10 actions/day = 10M actions/day
   • 10M / 86,400 seconds ≈ 116 RPS average
   • Peak is usually 3–5× average → ~400–600 RPS peak

2. Data volume:

   • 1M users × 100 KB profile data = 100 GB
   • 1M users × 10 posts/day × 1 KB/post = 10 GB/day

3. Read/write ratio:

   • Twitter: ~100:1 read-heavy (reads dominate)
   • Logging system: write-heavy (writes dominate)
   • The ratio determines which path to optimize

Scale your estimates by 10× or 100× to see where you break.

Scaling the Database Layer

The web tier is easy to scale horizontally because it can be stateless. The database tier is harder because it holds state. Common strategies:

Read Replicas

One primary handles writes; multiple replicas handle reads. Works well for read-heavy workloads (social feeds, product catalogs).

             ┌─────────────┐
Writes ──→   │   Primary   │ ──→ replicate ──→ Replica 1
             └─────────────┘                 ↘ Replica 2
                                               Replica 3
Reads ──→ any replica

Tradeoff: replicas may be slightly behind the primary (replication lag). Reads may see stale data.

Caching Layer

Put a cache (Redis, Memcached) in front of the database. Frequently read data is served from memory — orders of magnitude faster than disk.

Client → App Server → Cache HIT  → return cached value
                    → Cache MISS → Database → cache result → return

Cache hit rates of 90–99% dramatically reduce database load. Covered in depth in the Caching article.

Sharding (Horizontal Partitioning)

Split the database into multiple shards, each holding a subset of the data. Route requests to the correct shard.

Example: users 0–999999 → shard A, users 1000000–1999999 → shard B.

Sharding adds significant complexity (cross-shard queries, rebalancing, hotspots). Use it only when vertical scaling and read replicas are insufficient. Covered in the Replication & Sharding article.

Application-Level Techniques

Beyond infrastructure, application design choices affect scalability:

Async processing: Don't make users wait for slow operations (sending email, resizing images, updating search indexes). Put work in a queue; process it asynchronously. The user gets a fast response; the work happens in the background.

Connection pooling: Database connections are expensive to open. Pool them and reuse them across requests. Without pooling, each request opens and closes a connection — this collapses quickly under load.

Pagination: Never return unbounded result sets. Every API that returns a list should paginate. A query returning 1 row takes milliseconds; a query returning 100,000 rows takes seconds and can OOM the server.

Efficient queries: N+1 queries are the most common scalability killer at the database layer. Fetching 100 posts, then making 100 separate queries for each post's author, is 101 queries. One query with a JOIN is 1 query.

Scalability Is a Spectrum

Most systems never need to handle Twitter-scale traffic. Over-engineering for scale you'll never reach is waste. The right approach:

1. Start simple — monolith, single database, no caching
2. Measure — find the actual bottleneck, not the imagined one
3. Fix the bottleneck — targeted intervention, not wholesale redesign
4. Repeat

"Premature optimization is the root of all evil" applies to architecture as much as code. Build for the next 10× of growth, not the next 1000×.

Summary

Next: Load Balancing — how traffic is distributed across instances, and what happens when a node fails.