Ship a Local AI Fallback

The shape of this final lesson

Tie the quest together by shipping one real thing: a small backend service that answers user prompts using your cloud provider of choice, with Ollama as a transparent fallback when the cloud is unreachable. This is the smallest production-shape proof of the patterns from every prior track.

Production check-list

Adapter pattern. Cloud and Ollama both implement stream().
Health checks. Both have health_check() with a fast timeout.
Orchestrator. Tries cloud first; falls back to Ollama on failure or degraded health. Priority is an argument, not a default — the adapters track builds the same class local-first, so a service that means cloud-first has to say so.
Streaming. Frontend sees one continuous SSE stream regardless of which vessel answered.
Pinned model tags. Both Ollama and cloud model versions are explicit.
Observability. Log what answered each request — the skeleton below records the model tag, which is what the chunks carry; if you need the vessel key, have the orchestrator yield it — and read that log once before you trust it — a name field that is empty rather than absent will happily record nothing forever. You'll want this when you wonder later why some answers feel different.
Mini mode. If your local model is small, the system prompt is stripped down.
Warm-up. A startup task pre-loads the local model with keep_alive: -1 so first-fallback latency is acceptable — reading the tag off the adapter, so the warmed model and the called model cannot drift apart.

What you've built when this works

A service that's privacy-aware, cost-aware, and outage-aware. Cloud goes down → users keep getting answers. Cloud rate-limits you → users keep getting answers. Cloud gets expensive → bulk batch jobs route local. The orchestrator is dumb and the adapters are narrow; that's why it's robust.

EV v3 · 2026-09-18: The skeleton below contradicted its own check-list in three places, all since publication. Item 3 promised “tries cloud first” while the code instantiated the adapters track's AIOrchestrator unchanged, whose priority list starts with ollama — the shipped service was local-first, the opposite of the lesson's premise; priority is now passed explicitly. Item 8's warm-up pre-loaded qwen2.5:7b while the orchestrator's Ollama adapter was constructed with qwen2.5:32b, so it warmed a model the service never called and left the cold first-fallback it exists to prevent; it now reads the tag off the adapter. And item 6's vessel log latched chunk.model on the first chunk under an is None guard — StreamChunk.model defaults to the empty string, which is not None, so the guard was satisfied immediately and every request logged vessel=. The skeleton also moved from the deprecated @app.on_event("startup") to a lifespan context manager, which is what FastAPI 0.140 asks for. Run as written it now prints a real model tag. Review caught two follow-ons in this same block: the warm-up was sending keep_alive as the string "-1", which ollama answers 400 to — so the fix for the wrong-model warm-up was itself inert until the value became the number -1 — and the log records chunk.model, which is the model tag rather than the vessel key, so the variable says model now instead of implying otherwise. The orchestrator's second vessel is also named codex here, matching the fallback lesson's prose.

Code

End-to-end skeleton (FastAPI + orchestrator)·python

from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json

# Cloud first, local as the outage net. The default priority in the
# adapters track is the other way round, so say which one you mean here.
ORCHESTRATOR = AIOrchestrator(priority=["claude", "codex", "gemini", "ollama"])

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Pre-load the local fallback so the first cold fallback isn't a 30s wait.
    # Read the tag off the adapter instead of repeating it: a warm-up that
    # names its own model warms the wrong one the moment the adapter changes.
    local = ORCHESTRATOR.vessels["ollama"]
    try:
        await local.client.post(
            f"{local.base_url}/api/generate",
            json={"model": local.model, "keep_alive": -1},   # the number, not "-1"
            timeout=120.0,
        )
    except Exception:
        pass  # ok if local isn't installed
    yield

app = FastAPI(lifespan=lifespan)

@app.post("/chat/stream")
async def chat_stream(req: dict):
    """Stream answers to the user, with cloud→local fallback."""
    answered_by = {"model": None}

    async def gen():
        async for chunk in ORCHESTRATOR.stream(req["messages"]):
            # Record the first model tag we actually see. Guarding on
            # `is None` alone would latch the empty-string default from the
            # first chunk and log an empty value forever. Note what this
            # gives you: the model tag, not the vessel key — if you need the
            # key, have the orchestrator yield it.
            if not answered_by["model"] and chunk.model:
                answered_by["model"] = chunk.model
            payload = {"content": chunk.content, "done": chunk.done}
            yield f"data: {json.dumps(payload)}\n\n"
        # log which vessel answered (for ops)
        print(f"[/chat/stream] model={answered_by['model']}")

    return StreamingResponse(gen(), media_type="text/event-stream")

# Run with: uvicorn main:app --host 0.0.0.0 --port 9000

Exercise

Build the FastAPI service. Run it. Test 4 scenarios: (1) cloud reachable, (2) cloud blocked at firewall, (3) cloud reachable but slow, (4) Ollama daemon stopped. Confirm the user-visible behavior is graceful in all four. Write down what 'graceful' means in the failure modes — that's your real ops contract.