AI System Design
Designing the systems these companies actually run — gateways, retrieval, agents, evals, and the bill.
11 topics
Designing an LLM Gateway
One service between your product and the model providers — auth, routing, quotas, caching, retries and observability in a single place.
core25 minInference Serving & Capacity
Why LLM capacity behaves unlike any other service — token-rate limits, batching, and the queueing you need when demand exceeds throughput.
core25 minStreaming at Scale
Long-lived SSE connections through your whole stack — proxies, load balancers, cancellation, resumption, and what breaks at a hundred thousand of them.
core20 minRAG at Scale
The retrieval pipeline as a system — ingestion, indexing, hybrid retrieval and reranking — and the freshness and permission problems nobody mentions in the diagram.
core25 minVector Search Infrastructure
Running an ANN index in production — memory as the real constraint, the recall/latency dial, filtered search, and staying in sync with the source of truth.
core25 minDesigning an Agent Platform
Running many long-lived agent loops as infrastructure — durable state, sandboxed tool execution, approvals, and bounding what a loop can spend or break.
core25 minEval & Experiment Infrastructure
The system that tells you whether a prompt or model change made things better — datasets, graders, CI gating, and online experiments.
core25 minPrompt & Model Versioning
Treating prompts and model choices as deployed artifacts — versioned, gradually rolled out, attributable per request, and rollback-able in seconds.
core20 minMulti-Tenant Quotas & Cost Control
Metering tokens per tenant, enforcing limits before the spend happens, and designing a feature whose unit cost scales with usage.
core25 minGuardrails & Abuse in the Architecture
Where safety checks, prompt-injection defenses and abuse controls actually sit in the system — and why the model is not the only place to put them.
core25 minDesign an Assistant Over Company Data
The whole section as one worked prompt — a multi-tenant assistant that answers questions over a company's documents, end to end.
core25 min