Replication, sharding and consistency

HOW IT FITS TOGETHER

KEY TERMS

Session guarantees

Read-your-writes: after you write, your reads reflect it. Monotonic reads: once you have seen a value you never see an older one, which breaks when consecutive reads hit replicas at different lag. Both are per-client promises and can be provided by routing (sticky replica, LSN checks, primary after write) without making the whole system strongly consistent.

Quorums: R + W > N

With N replicas, a write acknowledged by W and a read from R overlap in at least one replica when R + W > N, so the read can see the latest acknowledged write. It still is not linearizable: concurrent writes resolved by timestamps, sloppy quorums with hinted handoff and failed partial writes can all yield anomalies. Cassandra applies this per query with QUORUM or LOCAL_QUORUM.

Consistency models

Linearizable: every operation appears to take effect atomically at one instant between its start and end, respecting real time. Sequential: one total order all processes agree on, without the real-time constraint. Causal: causally related operations appear in the same order everywhere, concurrent ones may not. Eventual: replicas converge if writes stop, with no promise about what you read meanwhile.

CAP and PACELC, precisely

CAP (Gilbert and Lynch) says that when the network partitions, a system cannot be both linearizable and available, where available means every request to a non-failing node gets a response. It says nothing about normal operation. PACELC adds the everyday trade-off: Else, with no partition, choose between Latency and Consistency, because waiting for replicas costs round trips.

Hot partitions and resharding

A shard key with low cardinality or skewed traffic (one celebrity, today's date) caps throughput at one shard's limit; DynamoDB, for example, limits each partition to 3,000 read units and 1,000 write units per second. Fixes include high-cardinality keys, write sharding with a suffix, and splitting ranges. Resharding moves data while serving traffic, so plan it as a migration with double reads or a copy-then-cutover.

2PC vs sagas

Two-phase commit makes participants vote then commit atomically, but blocks while the coordinator is down and holds locks across services. A saga runs local transactions in sequence and undoes earlier steps with compensating actions on failure. Sagas stay available but give up isolation, so other readers can observe intermediate states.

Consensus (Raft)

Raft elects a leader per term, the leader appends entries to followers, and an entry is committed once a majority has stored it. A cluster of 2f + 1 nodes tolerates f failures: 3 nodes survive 1, 5 survive 2. Distributed SQL databases run one consensus group per range so they can shard and still offer serializable, linearizable access.

IN YOUR STACK

import { Pool } from "pg";

const primary = new Pool({ connectionString: process.env.PRIMARY_URL });
const replica = new Pool({ connectionString: process.env.REPLICA_URL });
type Session = { lastWriteLsn?: string }; // persisted in the session or a cookie

export async function write(session: Session, sql: string, params: unknown[]) {
  const client = await primary.connect();
  try {
    await client.query(sql, params);
    const { rows } = await client.query("SELECT pg_current_wal_lsn()::text AS lsn");
    session.lastWriteLsn = rows[0].lsn;
  } finally { client.release(); }
}

export async function read(session: Session, sql: string, params: unknown[]) {
  if (!session.lastWriteLsn) return replica.query(sql, params);
  const client = await replica.connect(); // check and read on the same replica connection
  try {
    const { rows } = await client.query(
      "SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn AS caught_up", [session.lastWriteLsn]);
    return rows[0].caught_up ? await client.query(sql, params) : await primary.query(sql, params);
  } finally { client.release(); }
}

import cassandra from "cassandra-driver";

const { consistencies } = cassandra.types; // keyspace replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3}
const client = new cassandra.Client({ contactPoints: ["10.0.0.1", "10.0.0.2"], localDataCenter: "dc1", keyspace: "shop" });
const quorum = { prepare: true, consistency: consistencies.localQuorum };

// LOCAL_QUORUM write (2 of 3) + LOCAL_QUORUM read (2 of 3): 2 + 2 > 3, so reads overlap the write.
export async function placeOrder(customerId: string, orderId: string, totalCents: number) {
  await client.execute(
    "INSERT INTO orders_by_customer (customer_id, created_at, order_id, total_cents) VALUES (?, ?, ?, ?)",
    [customerId, new Date(), orderId, totalCents], quorum);
}

export async function recentOrders(customerId: string) {
  const rs = await client.execute(
    "SELECT order_id, total_cents FROM orders_by_customer WHERE customer_id = ? LIMIT 20", [customerId], quorum);
  return rs.rows;
}

// Compare-and-set needs Paxos (a lightweight transaction), not just a quorum.
export async function claimUsername(name: string, userId: string) {
  const rs = await client.execute("INSERT INTO usernames (name, user_id) VALUES (?, ?) IF NOT EXISTS",
    [name, userId], { prepare: true, serialConsistency: consistencies.localSerial });
  return rs.wasApplied();
}

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, GetCommand, QueryCommand } from "@aws-sdk/lib-dynamodb";

const db = DynamoDBDocumentClient.from(new DynamoDBClient({}));

// Right after a write the user is waiting on: strongly consistent, twice the read cost.
export async function getAccount(accountId: string) {
  const { Item } = await db.send(new GetCommand({
    TableName: "accounts",
    Key: { PK: `ACCOUNT#${accountId}`, SK: "PROFILE" },
    ConsistentRead: true,
  }));
  return Item;
}

// Feeds and lists: the default eventually consistent read is usually fine.
export async function listTransactions(accountId: string) {
  const { Items } = await db.send(new QueryCommand({
    TableName: "accounts",
    KeyConditionExpression: "PK = :pk AND begins_with(SK, :tx)",
    ExpressionAttributeValues: { ":pk": `ACCOUNT#${accountId}`, ":tx": "TX#" },
    ScanIndexForward: false,
  }));
  return Items ?? [];
}

import { createHash } from "node:crypto";

const hash = (s: string) => parseInt(createHash("md5").update(s).digest("hex").slice(0, 8), 16);

export class HashRing {
  private ring: { point: number; node: string }[] = [];
  constructor(nodes: string[], private vnodes = 128) {
    for (const node of nodes) this.add(node);
  }

  add(node: string) {
    for (let i = 0; i < this.vnodes; i++) this.ring.push({ point: hash(`${node}#${i}`), node });
    this.ring.sort((a, b) => a.point - b.point);
  }

  remove(node: string) { this.ring = this.ring.filter((v) => v.node !== node); }

  nodeFor(key: string): string {
    const h = hash(key);
    let lo = 0, hi = this.ring.length;
    while (lo < hi) { const mid = (lo + hi) >> 1; if (this.ring[mid].point < h) lo = mid + 1; else hi = mid; }
    return this.ring[lo % this.ring.length].node; // first point clockwise, wrapping to the start
  }
}

-- On the primary: one row per directly connected standby.
SELECT application_name,
       client_addr,
       state,                 -- startup, catchup, streaming, backup, stopping
       sync_state,            -- async, potential, sync, quorum
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes,
       write_lag,
       flush_lag,
       replay_lag
FROM pg_stat_replication;

-- On a standby: is it a standby, and how old is the last replayed commit?
SELECT pg_is_in_recovery() AS is_standby,
       pg_last_wal_replay_lsn() AS replayed_to,
       now() - pg_last_xact_replay_timestamp() AS since_last_replayed_commit;

CREATE TABLE orders (
    customer_id bigint      NOT NULL,
    order_id    bigint      NOT NULL,
    created_at  timestamptz NOT NULL DEFAULT now(),
    total_cents bigint      NOT NULL,
    PRIMARY KEY (customer_id, order_id)   -- must include the partition key
) PARTITION BY HASH (customer_id);

CREATE TABLE orders_p0 PARTITION OF orders FOR VALUES WITH (MODULUS 4, REMAINDER 0);
CREATE TABLE orders_p1 PARTITION OF orders FOR VALUES WITH (MODULUS 4, REMAINDER 1);
CREATE TABLE orders_p2 PARTITION OF orders FOR VALUES WITH (MODULUS 4, REMAINDER 2);
CREATE TABLE orders_p3 PARTITION OF orders FOR VALUES WITH (MODULUS 4, REMAINDER 3);

-- Created on every partition.
CREATE INDEX ON orders (customer_id, created_at DESC);

-- Pruned to a single partition because the key is in the filter.
EXPLAIN SELECT * FROM orders WHERE customer_id = 42 ORDER BY created_at DESC LIMIT 20;

-- No partition key in the filter: every partition is scanned.
EXPLAIN SELECT * FROM orders WHERE order_id = 9001;

WHEN TO REACH FOR WHAT

WHERE IT BITES

• Sending every read to replicas, then debugging 'lost' edits that are really read-your-writes violations from replication lag.
• Treating QUORUM reads and writes as linearizable; timestamp-based conflict resolution and partial failures still produce anomalies.
• Quoting CAP as 'pick two of three' and choosing a database by its CAP label instead of by its behaviour during partitions and its latency without them.
• Choosing a shard key by what is unique (an auto-increment id, a timestamp) rather than by what queries filter on and how load spreads.
• Dual-writing to two services or stores and calling it a transaction; use one database transaction, an outbox, or a saga with compensations.
• Failing over an async primary automatically without fencing the old one, so two primaries accept writes and the data splits.

WHEN IT BREAKS IN PRODUCTION

• The UI shows old data after a save
• Balances and nonces flip between refreshes
• Offline edits conflict or vanish on reconnect
• A retry creates two payments
• Webhooks arrive twice, late, or out of order

SOURCES

• PostgreSQL 18: Log-shipping standby servers (streaming and synchronous replication) (opens in new tab)Checked 2026-10-11
• Apache Cassandra: Dynamo (consistency levels, replication, token ring) (opens in new tab)Checked 2026-10-11
• Amazon DynamoDB: Read consistency (opens in new tab)Checked 2026-10-11
• Jepsen: Linearizability (opens in new tab)Checked 2026-10-11
• Daniel Abadi: Consistency Tradeoffs in Modern Distributed Database System Design (IEEE Computer, 2012) (opens in new tab)Checked 2026-10-11
• CockroachDB: Replication layer (opens in new tab)Checked 2026-10-11