Notes on building systems
Longer-form writing about AI application engineering, backend architecture, and the parts of production engineering that only show up in year two.
·11 min read
Idempotency Is a Scaling Problem, Not a Correctness Footnote
Double charges, phantom inventory, ghost orders: all the same bug. The operation was safe to run once and your infrastructure ran it twice.
Backend / Architecture / Distributed Systems / E-commerce / PostgreSQL / Node.js
·13 min read
You Cannot Prompt Away a Hallucination
My system prompt said every number must come from a tool. With that rule in place, an eval run still flagged 14 of 38 answers. A rule in a prompt is a request. A check in code is a control.
AI / LLM / Testing / Reliability / Architecture
·14 min read
A Pause Is Not the End of an Answer
Voice agents end your turn when you go quiet. In an interview, quiet is where the answer gets built. Let the person end the turn, then engineer the silence that follows.
AI / LLM / Voice / WebSockets / Streaming / Architecture
·12 min read
Rate Limits Are an Architecture Problem
Rate limits are a throughput constraint, and the fix is a queue, a concurrency cap, and a degradation path. A longer backoff only resubmits work the provider has already refused.
LLM / AI / Architecture / Backend / Rate Limiting / Node.js
·11 min read
Observability for a Small Team
You do not need a vendor to know your system is broken. Structured logs, a request ID, and three dashboards get a small team further than most observability platforms.
Observability / Backend / Node.js / DevOps / Architecture
·11 min read
Building an Agent Loop That Knows When to Stop
Starting an agent loop is the easy part. The hard part is making it stop before it has burned forty dollars retrying a step that will never work.
Agents / LLM / AI / Architecture / Backend
·11 min read
A Pipeline Where No Node Is Allowed to Raise
An exception at stage nine throws away the model calls you already bought. Make every node declare what it returns when it cannot do its job, and make that answer a value rather than a log line.
AI / LLM / Reliability / Architecture / Backend
·11 min read
Every Model Call Should Have an Owner
The invoice says one number. Your pipeline has fourteen stages, and one of them is most of the bill. Attribution is what turns model spend from a surprise into a design decision.
LLM / AI / Cost / Architecture / Backend / PostgreSQL
·11 min read
Ranking Is Not a Prompt
Asking a model to rank a list gives you an order with no explanation, no stability and no dial to turn. Use it for the judgements no formula can make, then do the ordering in code.
AI / LLM / Ranking / Architecture / Backend
·12 min read
The Interesting Part of MCP Is the Client
Writing an MCP server is a weekend. Writing one that behaves well in a client you do not control is the actual job, and the protocol's most interesting half runs in the other direction.
MCP / Agents / AI / LLM / Protocols / Architecture
·11 min read
The RAG System With No Embeddings
An embedding pipeline is a background system you own forever, not a one-time decision. I built retrieval without one, and spent the model budget on the query instead.
RAG / AI / LLM / Embeddings / PostgreSQL / Search / Backend
·11 min read
Retrieval Should Be a Tool, Not a Pipeline Stage
A fixed pipeline retrieves before every answer, including the ones that need arithmetic, and the ones that need nothing. Making retrieval a tool lets the model decide, and lets it look twice.
RAG / AI / LLM / Agents / Architecture / Backend
·11 min read
A2A Is Not a Better Function Call
Reading A2A as RPC with extra steps is the mistake. It is built for a callee that is opaque, long-running, and not yours, and that assumption is what should change your design.
A2A / Agents / AI / Protocols / Architecture
·10 min read
The Gym Is Part of the Job
We profile everything except the machine doing the reasoning. The hard bugs are found in hour six of concentration, and hour six is a physical capacity, not a character trait.
Career / Health / Focus / Developer Life / Habits
·11 min read
Postgres Is the Rest of Your AI Stack
The vector column was the easy part. An AI feature accumulates a queue, a cache, a trace log and a tenant boundary, and every one of those is a table you already know how to write.
PostgreSQL / AI / LLM / Architecture / Backend
·12 min read
Making a Postgres App Faster, in Order
Indexes, N+1s, round trips, caching: in that order, because caching a bad query just buys you a fast wrong answer and a stampede at expiry.
PostgreSQL / Performance / Indexing / Caching / Backend
·10 min read
Prompt Injection Is an Authorisation Problem
Filtering for malicious prompts is an arms race you lose. The model cannot distinguish instructions from data, so the boundary has to be enforced where it always was: permissions.
Security / LLM / AI / Agents / Architecture
·10 min read
Testing Something That Answers Differently Every Time
You cannot assert equality on a model's output, so most LLM features ship with no regression safety at all. Evals are the substitute, and most of them are built wrong.
LLM / Testing / AI / Evals / CI/CD
·10 min read
Two Files That Give an AI Agent a Memory
Coding agents start every session cold, and code records what it does but never why. Two files in the repo (one append-only, one living) fix more than a better prompt ever will.
AI Agents / Claude Code / Cursor / Documentation / Workflow
·10 min read
Streaming Tokens to a Browser Without Breaking Everything
Streaming looks like a frontend nicety. It is mostly a backend problem: cancellation, buffering proxies, execution limits, and persisting a response that arrives in fragments.
LLM / Node.js / Streaming / AI / Backend
·12 min read
Cache Invalidation with Debezium: Let the Database Tell You
Invalidating from application code only works if application code is the only thing that writes. Postgres logical replication turns every committed row change into the signal instead.
Debezium / PostgreSQL / Redis / CDC / Backend
·10 min read
Getting Reliable JSON Out of a Language Model
"Respond only with JSON" is a request, not a guarantee. The failures show up in production at a rate nobody measured, and the fix is to stop asking.
LLM / AI / TypeScript / API Design / Backend
·11 min read
Why Notifications Need an Integrated Event Source
Preferences, digests, dedup and audit are all functions over a stream of facts, and you cannot write a function over a stream you never materialised.
Architecture / Event-Driven / Notifications / Backend
·10 min read
Keeping a Vector Index in Sync With a Moving Codebase
Every RAG demo indexes once. Real corpora change hourly, and a stale index answers confidently from code that no longer exists. Deletions are the case that hurts most.
RAG / Embeddings / AI / Git / Architecture
·11 min read
Where to Draw a Service Boundary
Most microservice pain is a boundary drawn along the wrong axis. Data ownership is the rule; team structure and technical layers are how you get a distributed monolith.
Microservices / Architecture / NestJS / Distributed Systems
·10 min read
Vector Search Alone Is Not Retrieval
Semantic search fails hardest on the queries users are most sure about: an error code, a function name, an SKU. No embedding model fixes that; a keyword index does.
RAG / Search / AI / Embeddings / Backend
·9 min read
Structured Content for a Real Storefront
A headless CMS fails when content is modelled as pages rather than data. Model entities, and every surface (web, app, email) composes them.
Sanity / CMS / Next.js / Architecture / Content
·10 min read
Vector Search Without a New Database
Retrieval stacks usually add a vector database. For most applications Postgres already does this, and keeping vectors beside your rows removes an entire class of consistency problem.
pgvector / PostgreSQL / RAG / AI / Embeddings
·9 min read
A Design System, Not Class Soup
The unreadable-markup complaint about Tailwind is nearly always a symptom of skipping the config. Tokens first, and the class lists get short.
Tailwind CSS / Frontend / Design Systems / CSS
·10 min read
Chunking Is the Whole Ballgame in RAG
Most RAG failures blamed on the model are chunking failures. How you cut a document decides what can ever be retrieved, and character-count splitting destroys the thing you are searching for.
RAG / LLM / Embeddings / AI / Architecture
·9 min read
End-to-End Tests You Can Trust
A flaky suite is worse than none: it teaches everyone to re-run failures instead of reading them. Flakiness is a design defect with specific, fixable causes.
Testing / Playwright / Cypress / CI/CD
·10 min read
What to Test When There Is No Time
The pyramid is advice about cost, not virtue. Test the boundary you promise, and accept that some code is not worth testing at all.
Testing / Jest / Code Quality / Backend
·9 min read
Uploads That Never Touch Your Server
Proxying uploads through your API burns memory and request time for nothing. Presigned URLs move the bytes directly and leave your server one job: authorisation.
AWS / S3 / Backend / Architecture / Security
·10 min read
What Serverless Changes About Your Backend
Serverless moves your operational concerns somewhere your instincts do not look: connections, cold starts, and nowhere to put background work.
Serverless / Vercel / Architecture / PostgreSQL
·9 min read
A Monorepo Without the Slow Builds
Teams adopt a monorepo for shared code, then find every change rebuilds everything. That is a graph problem, not a monorepo problem.
Monorepo / NX / Architecture / CI/CD
·10 min read
A Pipeline That Fails for the Right Reasons
A twenty-minute CI run does not cost twenty minutes. It costs the context switch of everyone waiting, and a flaky one costs the habit of believing it.
CI/CD / GitHub Actions / DevOps / Testing
·10 min read
Probes, Limits, and Whether You Need Kubernetes
Liveness probes and memory limits are the two most misconfigured settings in Kubernetes, and both fail in ways that look like application bugs.
Kubernetes / DevOps / Reliability / Infrastructure
·10 min read
Images That Build Fast and Ship Small
Image size is deploy latency, cold-start time and attack surface. Most Node images are several times too large, and layer ordering is usually why.
Docker / DevOps / Node.js / CI/CD
·11 min read
Sessions, JWTs, and the Logout Problem
JWTs are chosen for statelessness and then made stateful by the first real requirement: logout. Better to know that before choosing than after.
Authentication / Security / JWT / Backend
·10 min read
Real-Time That Survives a Second Instance
WebSocket code works perfectly on one instance and breaks on the second, because the connection is state and everything around it assumed otherwise.
WebSockets / Socket.io / Redis / Architecture / Backend
·11 min read
Five Things Redis Does That Aren't Caching
Redis gets adopted as a cache and stays one. Most of its value is in rate limiting, locks, sessions, streams and sorted sets.
Redis / Backend / Distributed Systems / Architecture
·10 min read
Judge an ORM by Its Escape Hatch
Every ORM bets you will not need the database's real features. That bet always loses eventually, so evaluate the exit, not the query builder.
Prisma / Mongoose / PostgreSQL / TypeScript / Backend
·10 min read
Modeling For Documents, Not Against Them
Most MongoDB pain is a relational schema in document clothing. Embed versus reference is the only modeling decision that really matters, and it has rules.
MongoDB / Databases / Data Modeling / Backend
·10 min read
GraphQL Solves a Client Problem
GraphQL moves cost from the client to the server. Often the right trade, but the bill arrives later, and it is paid by whoever operates the API.
GraphQL / API Design / Backend / Performance
·9 min read
Express, Fastify, or NestJS
Pick a Node framework by how much structure it will force on you, not by throughput. That is a team-size question wearing technical clothes.
Node.js / NestJS / Express / Fastify / Architecture
·11 min read
The Event Loop Is a Budget
Node is fast until one synchronous function spends 200ms: then every concurrent request waits behind it. Concurrency is not parallelism, and the loop is a budget you spend.
Node.js / Performance / Backend
·11 min read
Choosing a Rendering Strategy in the App Router
The client boundary is a bundling decision, not a syntax detail. One "use client" at the wrong level ships an entire import tree to the browser.
Next.js / React / Performance / Frontend
·10 min read
State That Doesn't Belong in React
Most React bugs are ownership bugs. Server data copied into useState, values derived and then stored, effects synchronising things that never needed synchronising.
React / Frontend / State Management / Performance
·10 min read
Types That Catch Real Bugs
A codebase can be fully annotated and still catch nothing. Types pay in two places (the untrusted boundary and the domain model) and are ceremony almost everywhere else.
TypeScript / Backend / Frontend / Code Quality
·12 min read
Kafka in a Microservices Architecture
Kafka is a log, not a queue. Consumers read at their own pace and nothing is removed when read: almost every pattern and every pitfall follows from that.
Kafka / Microservices / Event-Driven / Architecture / Backend
·11 min read
SQL vs NoSQL: What Actually Differs, and How to Choose
The comparison is usually made on scale, which is the axis that matters least for most applications. The one that decides it is whether you already know your access patterns.
Databases / PostgreSQL / NoSQL / Architecture / Backend