Latency: Batching, Async, and Where the Time Goes

~22 min · ops, latency, performance

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Where time is spent in a typical RAG turn

Query embedding — 30–200 ms (hosted) or 5–20 ms (local small model)
Vector search — 5–50 ms (Chroma, single-node) or 1–10 ms (pgvector with HNSW)
(optional) BM25 — 5–30 ms
(optional) Cross-encoder rerank on 50 — 100–500 ms (CPU) or 30–150 ms (GPU)
LLM generation — 500 ms (small) to 10 s (large + long output). Dominates everything else.

The order of optimization

Profile first, optimize the dominant cost. Cutting query embedding from 100 ms to 10 ms is wasted effort if the LLM call is 4 seconds. Streaming the LLM response is usually a bigger UX win than cutting retrieval latency in half.

Async, not threading

Retrieval is I/O bound. Use async (asyncio in Python, async/await everywhere else) so you can issue the embedding call, vector search, and BM25 in parallel when the architecture allows. Threading is the wrong abstraction for this work.

Code

Run vector + BM25 in parallel·python

import asyncio

async def parallel_retrieve(question: str, k: int = 20):
    vec_task  = asyncio.create_task(asyncio.to_thread(vector_search, question, k))
    bm25_task = asyncio.create_task(asyncio.to_thread(bm25_search,   question, k))
    vec, bm25 = await asyncio.gather(vec_task, bm25_task)
    return reciprocal_rank_fusion([
        [r['id'] for r in vec],
        [r['id'] for r in bm25],
    ])

results = asyncio.run(parallel_retrieve('how do I cancel?'))

Time each stage explicitly·python

import time
from contextlib import contextmanager

@contextmanager
def stage(name: str, log: dict):
    start = time.perf_counter()
    try:
        yield
    finally:
        log[name] = round((time.perf_counter() - start) * 1000, 1)

log = {}
with stage('embed', log):
    qvec = embed(question)
with stage('search', log):
    cands = collection.query(query_embeddings=[qvec], n_results=20)
with stage('rerank', log):
    final = rerank(question, cands)
print(log)   # {'embed': 42.3, 'search': 11.8, 'rerank': 187.4}

External links

Exercise

Add explicit per-stage timing to your RAG pipeline. Log 100 real queries. Compute median and p95 for each stage. Identify the biggest contributor — that is your next optimization target. Anything else is premature.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousCost: Embedding Budgets and Call Patterns Next →Versioning Embeddings Without Breaking Production

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.