Streaming Tokens to a Browser Without Breaking Everything

Streaming a model's response token by token is presented as a UI improvement, and the reason it matters is more specific: it converts a wait into a read. Ten seconds of blank screen is a broken feature; ten seconds of text arriving as fast as someone reads is a normal one, with identical total latency.

Almost all the difficulty is on the server, and most of it is not about tokens at all.

Server-Sent Events, not WebSockets

The reflex is a WebSocket. For this shape it is the wrong tool: the traffic is entirely server-to-client, one response at a time, over ordinary HTTP.

Server-Sent Events fit exactly, and bring things you would otherwise build:

• Plain HTTP — no upgrade, no sticky sessions, no separate adapter to scale out.
• Automatic reconnection in the browser, with a resume header for free.
• Ordinary auth: cookies and headers work as they do everywhere else.
• Cancellation is just closing the request, which propagates naturally.

Use a WebSocket when the client genuinely needs to send during generation — live interruption, collaborative sessions, voice. For a request that produces one streamed answer, SSE is less machinery and fewer failure modes.

Cancellation is the one that costs money

The most common production bug in streaming endpoints: a user stops reading, closes the tab, or edits their question and re-sends — and the original generation keeps running to completion on the provider, billed in full, occupying a connection.

The client disconnecting does not stop your upstream call by itself. You have to propagate it:

export async function POST(req: Request) {
  const stream = await client.chat.completions.create(
    { model, messages, stream: true },
    { signal: req.signal },        // <- abort upstream when the client goes away
  );

  const encoder = new TextEncoder();
  return new Response(
    new ReadableStream({
      async start(controller) {
        try {
          for await (const chunk of stream) {
            if (req.signal.aborted) break;
            const delta = chunk.choices[0]?.delta?.content ?? "";
            if (delta) controller.enqueue(encoder.encode(`data: ${JSON.stringify({ delta })}\n\n`));
          }
          controller.enqueue(encoder.encode("data: [DONE]\n\n"));
        } finally {
          controller.close();
        }
      },
      cancel() { /* client went away: nothing else to do, signal handles it */ },
    }),
    { headers: {
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache, no-transform",
        "Connection": "keep-alive",
        "X-Accel-Buffering": "no",
    }},
  );
}

Passing the request's abort signal upstream is one argument and it is the difference between paying for abandoned generations and not. On a product where people frequently rephrase mid-answer, this is a substantial share of spend.

The proxy that eats your stream

The classic report: streaming works perfectly locally and in production the whole response appears at once after ten seconds. Nothing in your code is wrong. Something between you and the browser is buffering.

Three culprits, in order of likelihood:

• A reverse proxy buffering responses. Hence X-Accel-Buffering: no above, which nginx respects — and the equivalent setting on whatever else sits in front.
• Compression middleware. Gzip wants a buffer to compress. Cache-Control: no-transform asks intermediaries not to, and you may need to disable compression on this route explicitly.
• A CDN treating it as a cacheable response. Streaming endpoints must bypass the cache entirely.

The tell that distinguishes this from a code bug: run curl -N against the endpoint directly. If tokens arrive incrementally there and not through your domain, it is infrastructure, and no amount of rewriting the handler will help.

Persisting a response that arrives in pieces

A non-streamed call gives you the whole answer to save. Streaming gives you fragments, and the connection may die at any point — which raises a question with no default answer: what do you store when generation stops halfway?

The arrangement that holds up:

• Persist the user's message before generating. It is the part that must never be lost, and it costs one insert.
• Accumulate the assistant response server-side as you forward chunks. Do not rely on the client sending back what it received — it may not be there.
• Write the final text when the stream ends, including when it ends early, marked as incomplete.
• Store a stop reason. Completed, aborted by the user, hit the token limit, upstream error. Without it a truncated answer is indistinguishable from a short one, and you cannot offer to continue.

Marking incomplete responses matters beyond bookkeeping: a truncated answer fed back as conversation history teaches the model that stopping mid-sentence is normal in this conversation.

Serverless makes this sharper

Streaming endpoints sit awkwardly on function platforms, and the constraints are worth knowing before you discover them under load.

Execution limits apply to the whole stream, so a long generation can be cut off mid-response by the platform rather than by the model — which looks exactly like a truncation bug. Some runtimes support streaming and others buffer the response before returning it, which silently defeats the entire feature. And a function held open for the duration of a generation is billed for that duration, so a long-running stream is a different cost profile from a normal request.

If generations routinely run long, the honest answer is a persistent process for this route, or an asynchronous design: kick off generation, return an id, and have the client subscribe to a stream that a worker writes into. More moving parts, and it stops being at the mercy of a timeout.

What the client has to handle

• A visible stop control that aborts the request. This is what makes cancellation reach the server at all.
• Errors mid-stream. The response was 200 and the failure arrives later, so errors must be a message type in the stream rather than an HTTP status.
• Markdown rendered from partial text. A half-received code fence or table is malformed by definition; render incrementally in a way that tolerates it rather than throwing.
• Reconnection semantics. SSE reconnects automatically, which for a one-shot generation is usually not what you want — it will restart. Signal completion explicitly and close, so the browser does not retry a finished response.

The short version

Use SSE — it is plain HTTP with reconnection and cancellation already solved. Pass the request's abort signal upstream or you pay for every abandoned generation. If streaming works locally and not in production, it is a buffering proxy, and curl -N proves it in seconds. Accumulate server-side and record why the stream stopped, so truncated answers are recognisable. Then check your platform actually streams rather than buffering, before building on the assumption.