One JSON HTTP server, eight runtimes

HOW IT FITS TOGETHER

KEY TERMS

Event loop servers

One thread runs all handlers; I/O waits are registered with the kernel (epoll, kqueue, IOCP) and resumed as callbacks or awaits. Cheap per connection, but any synchronous CPU work is a global pause. Node, Bun, Deno and each uvicorn worker work this way.

Thread per request

Each in-flight request owns an OS thread from a bounded pool (Tomcat defaults to 200). Code is plain blocking code, but when every thread waits on a slow dependency new requests queue even though CPUs are idle.

Green threads: goroutines, virtual threads, BEAM processes

The runtime schedules many cheap user-space threads over a few OS threads and parks them on I/O, so blocking-style code scales to hundreds of thousands of concurrent requests. BEAM adds preemption by reduction count, so even a CPU-bound process cannot hog a scheduler.

Async tasks on a work-stealing runtime

Tokio and .NET turn async functions into state machines scheduled on a thread pool; idle workers steal queued tasks from busy ones. Awaiting is cheap, but a blocking call inside async code holds a worker thread, so offload it (spawn_blocking, Task.Run for CPU work).

Scaling across cores

Single-threaded runtimes scale out with processes (replicas, cluster, SO_REUSEPORT, uvicorn workers); Go, Rust, JVM, .NET and BEAM use every core in one process. In Kubernetes the usual answer for all of them is more pods, with CPU limits that match what the runtime thinks it has.

Graceful shutdown

Orchestrators send SIGTERM, wait a grace period (30 s by default in Kubernetes), then SIGKILL. A correct server stops accepting, drains in-flight requests and closes pools inside that window. Spring Boot, ASP.NET Core and uvicorn do it by default; Node, Go and axum need a few lines.

IN YOUR STACK

import { createServer } from "node:http";

async function readJson(req, limit = 1_000_000) {
  const chunks = [];
  let size = 0;
  for await (const chunk of req) {
    size += chunk.length;
    if (size > limit) throw Object.assign(new Error("body too large"), { status: 413 });
    chunks.push(chunk);
  }
  return JSON.parse(Buffer.concat(chunks).toString("utf8")); // SyntaxError -> 400 below
}
const send = (res, status, body) =>
  res.writeHead(status, { "content-type": "application/json" }).end(JSON.stringify(body));

const server = createServer(async (req, res) => {
  try {
    if (req.method === "GET" && req.url === "/health") return send(res, 200, { ok: true });
    if (req.method === "POST" && req.url === "/echo") return send(res, 200, await readJson(req));
    send(res, 404, { error: "not found" });
  } catch (err) {
    send(res, err.status ?? 400, { error: err.message });
  }
}).listen(3000);
process.once("SIGTERM", () => server.close()); // stop accepting, drain, then exit

const server = Bun.serve({
  port: 3000,
  routes: {
    "/health": { GET: () => Response.json({ ok: true }) },
    "/echo": {
      POST: async (req) => {
        try {
          return Response.json(await req.json());
        } catch {
          return Response.json({ error: "invalid JSON" }, { status: 400 });
        }
      },
    },
  },
  // Anything the routes table does not match.
  fetch: () => Response.json({ error: "not found" }, { status: 404 }),
});

process.on("SIGTERM", async () => {
  await server.stop(); // stop accepting; in-flight requests finish
  process.exit(0);
});
console.log(`listening on ${server.url}`);

import time
from typing import Any

from fastapi import FastAPI

app = FastAPI()  # unknown paths -> 404 {"detail": "Not Found"} automatically


@app.get("/health")
async def health() -> dict[str, bool]:
    return {"ok": True}


@app.post("/echo")
async def echo(payload: dict[str, Any]) -> dict[str, Any]:
    return payload  # body parsed and validated; malformed JSON -> 422


@app.get("/slow")
def slow() -> dict[str, str]:
    time.sleep(2)  # plain def runs in a thread pool, so the event loop keeps serving
    return {"done": "yes"}

# fastapi run main.py --workers 4   (or: uvicorn main:app --workers 4)

// package main; imports: encoding/json, log, net/http, time
func writeJSON(w http.ResponseWriter, status int, v any) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)
	json.NewEncoder(w).Encode(v)
}

func main() {
	mux := http.NewServeMux() // unmatched path -> 404, wrong method -> 405
	mux.HandleFunc("GET /health", func(w http.ResponseWriter, r *http.Request) {
		writeJSON(w, http.StatusOK, map[string]bool{"ok": true})
	})
	mux.HandleFunc("POST /echo", func(w http.ResponseWriter, r *http.Request) {
		var body any
		if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&body); err != nil {
			writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()})
			return
		}
		writeJSON(w, http.StatusOK, body)
	})
	srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
	log.Fatal(srv.ListenAndServe())
}

use axum::{http::StatusCode, routing::{get, post}, Json, Router};
use serde_json::{json, Value};
use tokio::signal::unix::{signal, SignalKind};

async fn health() -> Json<Value> { Json(json!({ "ok": true })) }

// Json<Value>: bad JSON -> 400, wrong Content-Type -> 415, before this runs.
async fn echo(Json(body): Json<Value>) -> Json<Value> { Json(body) }

async fn not_found() -> (StatusCode, Json<Value>) {
    (StatusCode::NOT_FOUND, Json(json!({ "error": "not found" })))
}

async fn shutdown_signal() {
    let mut term = signal(SignalKind::terminate()).expect("install SIGTERM handler");
    tokio::select! { _ = term.recv() => {}, _ = tokio::signal::ctrl_c() => {} }
}

#[tokio::main]
async fn main() {
    let app = Router::new().route("/health", get(health)).route("/echo", post(echo)).fallback(not_found);
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();
    axum::serve(listener, app).with_graceful_shutdown(shutdown_signal()).await.unwrap();
}

// dependency: spring-boot-starter-webmvc (named spring-boot-starter-web before Boot 4)
// application.properties: spring.threads.virtual.enabled=true   (Java 21+)
package demo;

import java.util.Map;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.*;

@SpringBootApplication
@RestController
public class App {
  public static void main(String[] args) {
    SpringApplication.run(App.class, args);
  }

  @GetMapping("/health")
  Map<String, Boolean> health() { return Map.of("ok", true); }

  // Malformed JSON -> 400; unknown paths -> 404 JSON error body from Boot.
  @PostMapping("/echo")
  Map<String, Object> echo(@RequestBody Map<String, Object> body) { return body; }
}

using System.Text.Json;
using Microsoft.AspNetCore.Mvc;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/health", () => TypedResults.Ok(new { ok = true }));

// Body binding uses System.Text.Json: malformed JSON -> 400,
// non-JSON Content-Type -> 415. Unmatched routes -> 404.
app.MapPost("/echo", ([FromBody] JsonElement body) => TypedResults.Ok(body));

app.MapGet("/slow", async (CancellationToken ct) =>
{
    await Task.Delay(2000, ct); // frees the thread while waiting; never .Result or .Wait()
    return TypedResults.Ok(new { done = true });
});

app.Run(); // SIGTERM: stop accepting, drain in-flight requests, then exit

# deps: {:bandit, "~> 1.0"} (pulls in Plug); Elixir 1.18+ for the JSON module
defmodule Echo.Router do
  use Plug.Router

  plug :match
  plug Plug.Parsers, parsers: [:json], pass: ["application/json"], json_decoder: JSON
  plug :dispatch

  get "/health" do
    send_json(conn, 200, %{ok: true})
  end

  # Malformed JSON raises Plug.Parsers.ParseError (plug_status 400).
  post "/echo" do
    send_json(conn, 200, conn.body_params)
  end

  match _ do
    send_json(conn, 404, %{error: "not found"})
  end

  defp send_json(conn, status, data),
    do: conn |> put_resp_content_type("application/json") |> send_resp(status, JSON.encode!(data))
end
# application.ex: children = [{Bandit, plug: Echo.Router, port: 4000}]

WHEN TO REACH FOR WHAT

WHERE IT BITES

• Doing blocking work inside an event loop or async runtime: a sync DB driver in async def, readFileSync in Node, std::fs or a mutex held across awaits in Tokio, .Result in C#. Each stalls far more than the one request.
• Reading request bodies without a size limit. node:http and many minimal servers buffer whatever the client sends unless you cap it (MaxBytesReader, DefaultBodyLimit, a limit in your read loop).
• Missing graceful shutdown: if the server exits as soon as SIGTERM arrives, every rolling deploy drops in-flight requests and you see 502s at the load balancer.
• Running N workers or threads that each open a full database pool. Total connections are workers × pool size × replicas, which exhausts the database long before the CPU.
• Assuming virtual threads or goroutines remove backpressure. Cheap concurrency lets 10,000 requests hit a dependency at once; bound it with semaphores, pools and timeouts.
• Misreading container CPU limits: some runtimes size their thread pools or workers from the host's core count rather than the cgroup quota. Check what yours detects and set it explicitly (GOMAXPROCS, workers, pool sizes) when they disagree.

WHEN IT BREAKS IN PRODUCTION

• Requests hang waiting for a database connection

SOURCES

• Go: Routing enhancements for Go 1.22 (opens in new tab)Checked 2026-10-11
• FastAPI: Concurrency and async / await (opens in new tab)Checked 2026-10-11
• Tokio: runtime module (opens in new tab)Checked 2026-10-11
• Oracle: Virtual Threads (Java SE 25) (opens in new tab)Checked 2026-10-11
• Bun: HTTP server (opens in new tab)Checked 2026-10-11
• Spring Boot: Graceful shutdown (opens in new tab)Checked 2026-10-11