Chunked Transfer Encoding — How HTTP/1.1 Streams Without Knowing the Size

"HTTP/1.1 needs to know either how long a response body is (Content-Length) or that the body ends when the connection closes (HTTP/1.0 style). Chunked transfer is the third option: a self-delimiting body of unknown final size. It's what makes streaming possible without keeping a connection open forever."

The Length Problem

HTTP/1.1 needs to know when one response ends so it can start reading the next (keep-alive connection reuse). Two options for fixed-size bodies:

Content-Length — send the exact byte count as a header. Client reads N bytes, knows the response is done.
Connection: close — body ends when the server closes the connection. Defeats keep-alive.

Neither works when the body is generated on the fly and the final size isn't known until done. AI streaming, log tailing, large query results — all need to start writing before knowing the total. That's what Transfer-Encoding: chunked solves.

The Chunked Format

Each chunk is preceded by its size in hex, then CRLF, then the chunk bytes, then CRLF. The end is signaled by a chunk of size 0:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Transfer-Encoding: chunked

2A\r\nevent: message\ndata: {"content":"Hi"}\n\n\r\n
2C\r\nevent: message\ndata: {"content":"아빠"}\n\n\r\n
0\r\n\r\n

2A hex = 42 bytes of chunk content. 2C hex = 44 bytes. 0 with the final CRLF marks the end. Most clients and servers handle this transparently — you write your async generator's yield, the HTTP library frames it as chunks.

What Chunked Enables

SSE — body is a stream of events of unknown count and size.
Large file downloads — server can start sending before computing the total size.
Real-time logs — server tails a file and writes chunks as new lines arrive.
Pipelined query results — DB streams rows; server writes each row as a chunk.

HTTP/2 and HTTP/3 don't expose chunked encoding explicitly — they have their own binary framing — but the semantic is identical: a streaming body of unknown total length.

Chunked transfer is the substrate that makes 'streaming over HTTP' work without violating HTTP's request/response model. SSE rides on it, file downloads ride on it, AI token streams ride on it. Most of the time you don't write chunked-encoded bytes by hand — your HTTP library or framework does it when you yield from an async generator. But knowing it's there explains why streaming over HTTP/1.1 works at all.

The Gotchas

1. Trailers. Chunked responses can include headers AFTER the body, declared by Trailer: in the leading headers. Useful for HMAC signatures over streamed content. Rarely supported by intermediaries; use with caution.

2. Buffering breaks streaming. Same gotcha as SSE — proxies that buffer chunked responses collect everything before forwarding, defeating the streaming. Always set X-Accel-Buffering: no or equivalent for streaming responses.

3. Don't set Content-Length on chunked responses. They're mutually exclusive. Some old servers do both; some old clients get confused.

cwkPippa's Chunked Reality

cwkPippa's StreamingResponse from FastAPI/Starlette uses chunked transfer automatically. Every yield from the async generator becomes a chunk on the wire. The healing layer relies on this: the JSONL log is written BEFORE the chunk is yielded to the response, so if the client disconnects mid-stream, the durable record exists. The HTTP/1.1 chunked encoding is also why Uvicorn (without nginx in front) works fine for SSE — the chunked framing IS the streaming.

Code

Chunked encoding on the wire — hex size + CRLF + content + CRLF·http

# A chunked response on the wire (with literal CRLF and hex sizes shown)
HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

14\r\n
The first part body\r\n
B\r\n
 second part\r\n
0\r\n
\r\n

# Translates to body: 'The first part body second part'
# 14 hex = 20 bytes; B hex = 11 bytes; 0 marks end.

FastAPI: StreamingResponse + async generator yields chunked bytes·python

# FastAPI — chunked encoding is automatic when you use StreamingResponse
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio

app = FastAPI()

async def streamed_content():
    # Each yield becomes one chunk on the wire (after framing by Starlette)
    for i in range(10):
        yield f'chunk {i}\n'
        await asyncio.sleep(0.5)

@app.get('/stream')
async def stream():
    # Starlette writes Transfer-Encoding: chunked automatically when body length is unknown
    return StreamingResponse(streamed_content(), media_type='text/plain')

# curl -N (no buffering) lets you see it chunk by chunk:
# curl -N http://localhost:8000/stream
# chunk 0   (appears after 0.5s)
# chunk 1   (appears after 1.0s)
# ... etc

Client: httpx.stream + iter_text shows chunks as they arrive·python

# Client side — httpx and curl handle chunked transparently
import httpx

with httpx.stream('GET', 'http://localhost:8000/stream') as resp:
    print('headers:', dict(resp.headers))  # Transfer-Encoding: chunked
    for chunk in resp.iter_text():
        print(f'received chunk: {chunk!r}')
# received chunk: 'chunk 0\n'
# (0.5s)
# received chunk: 'chunk 1\n'
# ...

# Curl: -N disables curl's own buffering
# curl -N http://localhost:8000/stream

Exercise

Build a FastAPI endpoint /count that streams numbers 1 to 100, one per second, using StreamingResponse. Test it three ways: (1) curl -N http://localhost:8000/count to see chunks arrive in real-time, (2) curl http://localhost:8000/count (no -N) to see how curl buffers, (3) Python httpx.stream with iter_text to consume the stream programmatically. Then add a print of dict(resp.headers) and verify you see Transfer-Encoding: chunked.

Hint

Without -N, curl waits to print until it has enough output to fill a line buffer — masking the streaming. With -N, you SEE each chunk arrive at its sleep interval. httpx.stream's iter_text yields decoded text as soon as bytes arrive. The Transfer-Encoding header is set automatically by Starlette when the response body is an iterator/generator and Content-Length isn't computed.

Chunked Transfer Encoding — How HTTP/1.1 Streams Without Knowing the Size

The Length Problem

The Chunked Format

What Chunked Enables

The Gotchas

cwkPippa's Chunked Reality

Code

External links

Exercise

Progress

Comments 0