·11 min read
Observability for a Small Team
You do not need a vendor to know your system is broken. Structured logs, a request ID, and three dashboards get a small team further than most observability platforms.
Observability / Backend / Node.js / DevOps / Architecture
At a two-to-five-person team, observability usually means console.log in development and “check the cloud provider's logs tab” in production. When something breaks, someone SSHs in (or opens CloudWatch, or scrolls through Vercel's function logs) and searches for the error. It works until the first time two things break at once, or until a problem is intermittent enough that grepping a log file finds nothing.
The opposite extreme is a full observability platform: Datadog, New Relic, or Grafana Cloud with a bill that scales with log volume. At the scale of a small team's traffic, the bill is often more than the compute itself, and most of the dashboards go unread.
This post is about the middle ground: the smallest set of practices that lets a small team diagnose production problems quickly, without a vendor contract or a dedicated SRE.
What you are diagnosing
When production breaks at a small company, the question is almost never “what is the p99 latency of this endpoint.” It is one of three things:
- Is it down? The entire service, or one route, or one dependency (the database, a third-party API, a payment provider).
- Why is it slow? A specific request that used to take 200ms now takes 8 seconds, and a customer noticed before you did.
- What happened to this specific request? A user says they submitted a form and nothing happened. You need to find that request and see where it stopped.
Everything in this post is aimed at answering those three questions as fast as possible. Anything that does not help answer them is overhead you do not need yet.
Structured logs, not console.log
The single most useful change is switching from free-text log lines to structured JSON. JSON is worse to read than plain text, but it is filterable. console.log("user signed up", email) produces a string that no tool can reliably parse. A structured log produces a JSON object with named fields that any log viewer can filter, aggregate, and alert on.
// Before
console.log("Payment failed", userId, err.message);
// After
logger.error({
event: "payment_failed",
userId,
provider: "stripe",
error: err.message,
stripeCode: err.code,
});
In Node.js, pino is the standard choice. It logs JSON to stdout, which is what every container runtime and cloud platform expects. Winston works too, but it is slower and the plugin ecosystem encourages transport complexity you do not need at this scale.
Three rules that make structured logs useful rather than noisy:
- Log events, not commentary. Every log line should have an
eventfield that is a stable, greppable string:request_started,db_query_slow,payment_failed. Not “Processing the request...” or “Something went wrong.” - Include the identifiers. User ID, request ID, order ID, session ID. Whatever lets you find every log line related to a single operation. Without identifiers, structured logs are just prettier noise.
- Control the volume. Log at
infofor events worth keeping (request boundaries, state changes, failures) anddebugfor everything else. Runinfoin production,debuglocally. A service that logs every database query atinfois producing enough volume that finding anything useful requires its own search infrastructure, which defeats the point of staying small.
The request ID that ties everything together
A request ID is a random string generated at the edge (your API gateway, your load balancer, or the first middleware in your Express or Fastify server) and attached to every log line, database query, and outgoing HTTP call for the duration of that request. It is the single field that lets you reconstruct what happened.
// Fastify example with async local storage
import { randomUUID } from "node:crypto";
import { AsyncLocalStorage } from "node:async_hooks";
const store = new AsyncLocalStorage<{ requestId: string }>();
app.addHook("onRequest", (req, _reply, done) => {
const requestId =
(req.headers["x-request-id"] as string) || randomUUID();
store.run({ requestId }, done);
});
// In any module, at any depth:
export function getRequestId(): string {
return store.getStore()?.requestId ?? "no-request-context";
}
AsyncLocalStorage is the Node.js mechanism that makes this work without passing the request ID through every function signature. It is stable, it is in the standard library, and it costs effectively nothing at runtime.
When your service calls another service, forward the request ID as a header. When the downstream service logs its own work, it includes that same ID. Now a single search across both services' logs reconstructs the full request path, which is the core of distributed tracing without any tracing vendor.
Three metrics before any dashboard
Before building a dashboard, decide what it should show. For a small team, three metrics cover almost every production conversation:
- Request rate (requests per second, per route). Tells you whether traffic is normal. A sudden drop means something upstream broke or DNS changed. A sudden spike means you are about to find out whether your autoscaling works.
- Error rate (5xx responses per second, per route). The first thing to look at when something is wrong. A flat error rate during a traffic spike is healthy. A rising error rate at constant traffic means you broke something.
- Latency distribution (p50, p95, p99 per route). Averages are useless here. A p50 of 80ms and a p99 of 12 seconds means one in a hundred users is waiting 12 seconds, and the average will not show it. The p99 is the number that matters.
These three, together, answer the “is it down” and “why is it slow” questions immediately. You can collect them with Prometheus (free, self-hosted, a 50MB binary), with the cloud provider's built-in metrics (CloudWatch, Cloud Monitoring), or even with a custom middleware that writes to a time-series table in your existing Postgres.
Do not start with application-level metrics (cache hit rate, queue depth, model call latency). Those become valuable once you have ruled out the infrastructure-level answer, and ruling it out is what the three above are for.
Dashboards that answer one question
A dashboard with twelve panels is a screenshot in a postmortem document. It is not a diagnostic tool. A useful dashboard answers one question, and the answer is visible in under five seconds.
For a small team, three dashboards are enough:
- Service health. Request rate, error rate, and p99 latency, per service. One glance tells you whether the system is healthy. This is the one you put on a TV in the office, if you have an office.
- Slow requests. The ten slowest requests in the last hour, with their request ID, route, and total duration. Click through to the logs for that request ID. This is the one you open when a customer says “it is slow.”
- Errors. Recent 5xx responses grouped by route and error message, with request IDs. This is the one you open when the alert fires.
The shared pattern: every dashboard links back to a request ID, and the request ID links to the structured logs. The dashboard tells you where to look. The logs tell you what happened.
Tracing across services, without the platform
If you run more than one service (and at a small team, “more than one” often means the API server and a background worker), you need to follow a request across the boundary. Full distributed tracing with OpenTelemetry, a collector, Jaeger or Tempo is the proper solution and it is more infrastructure than a small team should operate.
The pragmatic version: propagate the request ID header, log it on both sides, and search by it. That gives you the trace without the trace infrastructure.
- Outgoing HTTP calls: attach
x-request-idfrom the current context. - Queue messages: include
requestId(orcorrelationId) in the message payload. The consumer logs it when it picks up the job. - Database queries: if your ORM or query builder supports query comments, append the request ID as a SQL comment. Now
pg_stat_activityor the slow query log shows which request is responsible for which query.
You lose the waterfall timeline and automatic span detection, and keep what matters more at this scale: the ability to find everything related to one user's broken experience in under a minute.
When to buy the vendor tool
Everything above stops being enough at a specific, recognisable point: when the log volume makes searching slow, when the number of services makes propagation error-prone, or when you hire the fifth engineer and need everyone to diagnose problems without tribal knowledge of which logs to search.
Signs you have reached that point:
- Searching logs takes more than 30 seconds. At this point you need an index, and running your own Elasticsearch or Loki cluster is its own operational burden.
- More than three services are involved in a single request path. The manual request-ID-grep approach still works, but the person doing it needs to know which three services to search, and a new hire does not.
- You want alerting on log patterns (not just metrics). “Alert me when this error message appears more than ten times in five minutes” requires a log pipeline, not a dashboard.
- Postmortem investigations take longer than the incident. The raw material is there, but assembling it takes expertise that should be automated.
At that point, Datadog, Grafana Cloud, or Axiom are worth what they cost. But the practices above (structured logs, request IDs, the three core metrics, dashboards that link to logs) are prerequisites, not alternatives. A vendor tool on top of console.log gives you a searchable index of noise. The structure has to come first.
The short version
Switch from console.log to structured JSON with pino. Generate a request ID at the edge and propagate it everywhere with AsyncLocalStorage. Collect three metrics (request rate, error rate, latency distribution) and build three dashboards (health, slow requests, errors), each linking back to the request ID. Propagate the ID across service boundaries through headers and message payloads. This is enough to diagnose any production issue at a small team's scale, and it costs nothing beyond compute you are already paying for. Buy the vendor tool when log search becomes slow or the team outgrows tribal knowledge. Not before.