System design foundations

HOW IT FITS TOGETHER

KEY TERMS

SLI, SLO and error budget

An SLI is a ratio of good events to valid events, such as requests answered successfully under 300 ms. The SLO is the target for that ratio over a window, and 1 minus the SLO is the error budget: the unreliability you may spend on releases and incidents before reliability work takes priority.

Back-of-envelope estimation

Daily volume divided by 86,400 gives average per-second load; multiply by a peak factor of 2 to 10. Multiply writes by record size and retention for storage, and responses by size for bandwidth. The goal is the order of magnitude, which decides whether you need one database or a fleet.

L4 vs L7 load balancing

An L4 balancer routes TCP or UDP connections by address and port without reading requests, which is fast and protocol-agnostic. An L7 balancer terminates HTTP and routes each request by host, path or header, enabling retries, per-request balancing over long-lived HTTP/2 connections and TLS offload.

Stateless services and horizontal scaling

If no request depends on memory in a specific instance, instances are interchangeable: you scale by adding copies and survive losing one. Sticky sessions and in-process caches quietly break that and make deploys and autoscaling lose state.

Read replicas and partitioning

Replicas multiply read capacity but not write capacity and serve slightly stale data. When writes or data size outgrow one primary, partition by a key so each shard owns a subset, at the cost of cross-shard queries and transactions.

Queues for async work

A queue decouples accepting work from doing it: the request returns quickly, workers process at their own pace, bursts become backlog instead of errors, and failures are retried. The price is eventual completion, idempotent consumers and a dead-letter path.

Rate limiting and load shedding

Token buckets cap each client's sustained rate while allowing short bursts, protecting shared capacity from one noisy caller. Load shedding rejects excess work early with 429 or 503 so the requests you do accept still meet their latency SLO.

IN YOUR STACK

// A photo-sharing feed: 10M daily active users.
const DAU = 10_000_000;
const SECONDS_PER_DAY = 86_400;
const PEAK_FACTOR = 3;

const readsPerSec = (DAU * 20) / SECONDS_PER_DAY;  // 20 feed loads/user/day ~ 2,315/s
const writesPerSec = (DAU * 2) / SECONDS_PER_DAY;  // 2 posts/user/day ~ 231/s
const peakReads = readsPerSec * PEAK_FACTOR;       // ~ 6,900/s

const postMetaBytes = 1_000;                        // row in the database
const photoBytes = 500_000;                         // object in blob storage
const metaPerYearGB = (DAU * 2 * postMetaBytes * 365) / 1e9;  // ~ 7,300 GB = 7.3 TB
const photosPerYearTB = (DAU * 2 * photoBytes * 365) / 1e12;  // ~ 3,650 TB = 3.65 PB

const feedResponseBytes = 20_000;                   // JSON for one page of feed
const peakEgressMbps = (peakReads * feedResponseBytes * 8) / 1e6; // ~ 1,100 Mbit/s

// Conclusions the numbers force:
// - 231 writes/s fits one Postgres primary; 6,900 reads/s wants a cache and replicas.
// - Photos never go in the database: object storage plus a CDN.
// - 7.3 TB/year of metadata is a planned partitioning or archiving decision, not an emergency.
console.table({ readsPerSec, writesPerSec, peakReads, metaPerYearGB, photosPerYearTB, peakEgressMbps });

import Redis from "ioredis";
import { Pool } from "pg";

const redis = new Redis(process.env.REDIS_URL!);
const db = new Pool({ connectionString: process.env.DATABASE_URL });
const TTL_SECONDS = 300;

type Product = { id: string; name: string; priceCents: number };

export async function getProduct(id: string): Promise<Product | null> {
  const key = `product:v1:${id}`;
  const cached = await redis.get(key).catch(() => null); // cache down: fall through
  if (cached) return JSON.parse(cached) as Product;
  const { rows } = await db.query<Product>(
    'SELECT id, name, price_cents AS "priceCents" FROM products WHERE id = $1', [id]);
  const product = rows[0] ?? null;
  const ttl = product ? TTL_SECONDS : 30;                // short TTL for negative results
  await redis.set(key, JSON.stringify(product), "EX", ttl).catch(() => {});
  return product;
}

export async function updatePrice(id: string, priceCents: number) {
  await db.query("UPDATE products SET price_cents = $1 WHERE id = $2", [priceCents, id]);
  await redis.del(`product:v1:${id}`);                 // invalidate after the commit
}

class TokenBucket {
  private tokens: number;
  private last = Date.now();

  constructor(private capacity: number, private refillPerSec: number) {
    this.tokens = capacity;
  }

  tryTake(cost = 1): boolean {
    const now = Date.now();
    this.tokens = Math.min(this.capacity, this.tokens + ((now - this.last) / 1000) * this.refillPerSec);
    this.last = now;
    if (this.tokens < cost) return false;
    this.tokens -= cost;
    return true;
  }
}

// Burst of 20, sustained 5 requests/second per API key.
const buckets = new Map<string, TokenBucket>();
export function allow(apiKey: string): boolean {
  let bucket = buckets.get(apiKey);
  if (!bucket) buckets.set(apiKey, (bucket = new TokenBucket(20, 5)));
  return bucket.tryTake(); // false -> respond 429 with Retry-After
}

upstream app {
    least_conn;                                   # fewest active connections wins
    server 10.0.1.10:3000 max_fails=3 fail_timeout=10s;
    server 10.0.1.11:3000 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:3000 max_fails=3 fail_timeout=10s;
    keepalive 32;                                 # idle upstream connections per worker
}

server {
    listen 443 ssl;
    server_name api.example.com;

    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;                   # needed for upstream keepalive
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 2s;
        proxy_read_timeout 15s;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
    }
}

import { createServer } from "node:http";
import { Pool } from "pg";

const db = new Pool({ connectionString: process.env.DATABASE_URL, max: 10 });
let draining = false;

async function dbReachable(): Promise<boolean> {
  const timeout = new Promise<boolean>((resolve) => setTimeout(resolve, 500, false));
  const ping = db.query("SELECT 1").then(() => true, () => false);
  return Promise.race([ping, timeout]);
}

const server = createServer(async (req, res) => {
  if (req.url === "/livez") return res.writeHead(200).end("ok"); // process alive, no dependency checks
  if (req.url === "/readyz") {
    const ready = !draining && (await dbReachable());
    return res.writeHead(ready ? 200 : 503).end(ready ? "ready" : "not ready");
  }
  res.writeHead(404).end();
});
server.listen(3000);
process.on("SIGTERM", () => {
  draining = true;                                   // fail readiness so the LB stops sending traffic
  setTimeout(() => server.close(() => process.exit(0)), 10_000);
});

const SLO = 0.999;                    // 99.9% of requests succeed
const WINDOW_DAYS = 30;
const budget = 1 - SLO;               // 0.1% of requests may fail

// Time-based view: minutes of total outage the budget allows.
const budgetMinutes = WINDOW_DAYS * 24 * 60 * budget;   // 43.2 minutes

// Request-based view, which is what most SLIs count.
const expectedRequests = 3_000_000;
const allowedFailures = expectedRequests * budget;      // 3,000 failed requests

// Burn rate: how fast the current error rate spends the budget.
// 1 means the budget runs out exactly at the end of the window.
function burnRate(errorRatio: number): number {
  return errorRatio / budget;
}

// Page when the last hour burned at >= 14.4x (2% of a 30-day budget in 1 h)
// and the last 5 minutes confirm it is still happening.
function shouldPage(errorRatio1h: number, errorRatio5m: number): boolean {
  return burnRate(errorRatio1h) >= 14.4 && burnRate(errorRatio5m) >= 14.4;
}

console.log({ budgetMinutes, allowedFailures, page: shouldPage(0.02, 0.03) });

WHEN TO REACH FOR WHAT

WHERE IT BITES

• Designing before estimating: adding sharding, Kafka and microservices for a load one Postgres instance would serve comfortably.
• Stating averages instead of percentiles; p99 latency and peak-hour load are what break SLOs.
• A cache without a plan for cold starts and hot-key expiry, so the first deploy or a popular key stampedes the database.
• Liveness probes that check the database, turning a dependency outage into every instance restarting at once.
• Retries at every layer with no budget, multiplying load during exactly the incident they were meant to absorb.
• Reading from replicas on paths where the user just wrote, producing read-your-writes bugs that look like lost data.

WHEN IT BREAKS IN PRODUCTION

• The database spikes when a hot cache key expires
• A brief blip turns into a long outage
• A third-party API starts returning 429
• Requests hang waiting for a database connection
• One bad message blocks or floods the queue
• The team wants microservices too early

SOURCES

• Google SRE book: Embracing Risk (opens in new tab)Checked 2026-10-11
• Google SRE workbook: Alerting on SLOs (opens in new tab)Checked 2026-10-11
• Google SRE book: Load Balancing in the Datacenter (opens in new tab)Checked 2026-10-11
• Amazon Builders' Library: Caching challenges and strategies (opens in new tab)Checked 2026-10-11
• nginx: ngx_http_upstream_module (opens in new tab)Checked 2026-10-11
• Kubernetes: Liveness, readiness and startup probes (opens in new tab)Checked 2026-10-11