Ship a Local AI Fallback

The shape of this final lesson

Tie the quest together by shipping one real thing: a small backend service that answers user prompts using your cloud provider of choice, with Ollama as a transparent fallback when the cloud is unreachable. This is the smallest production-shape proof of the patterns from every prior track.

Production check-list

Adapter pattern. Cloud and Ollama both implement stream().
Health checks. Both have health_check() with a fast timeout.
Orchestrator. Tries cloud first; falls back to Ollama on failure or degraded health.
Streaming. Frontend sees one continuous SSE stream regardless of which vessel answered.
Pinned model tags. Both Ollama and cloud model versions are explicit.
Observability. Log which vessel answered each request. You'll want this when you wonder later why some answers feel different.
Mini mode. If your local model is small, the system prompt is stripped down.
Warm-up. A startup task pre-loads the local model with keep_alive: -1 so first-fallback latency is acceptable.

What you've built when this works

A service that's privacy-aware, cost-aware, and outage-aware. Cloud goes down → users keep getting answers. Cloud rate-limits you → users keep getting answers. Cloud gets expensive → bulk batch jobs route local. The orchestrator is dumb and the adapters are narrow; that's why it's robust.

Code

End-to-end skeleton (FastAPI + orchestrator)·python

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json, asyncio

app = FastAPI()

# Build the orchestrator at startup; pre-warm the local fallback
ORCHESTRATOR = AIOrchestrator()

@app.on_event("startup")
async def warm_local():
    # Pre-load the local fallback model so first cold-fallback isn't a 30s wait
    try:
        await ORCHESTRATOR.vessels["ollama"].client.post(
            "http://localhost:11434/api/generate",
            json={"model": "qwen2.5:7b", "keep_alive": "-1"},
            timeout=120.0,
        )
    except Exception:
        pass  # ok if local isn't installed

@app.post("/chat/stream")
async def chat_stream(req: dict):
    """Stream answers to the user, with cloud→local fallback."""
    chosen_vessel = {"name": None}

    async def gen():
        async for chunk in ORCHESTRATOR.stream(req["messages"]):
            if chosen_vessel["name"] is None:
                chosen_vessel["name"] = chunk.model
            payload = {"content": chunk.content, "done": chunk.done}
            yield f"data: {json.dumps(payload)}\n\n"
        # log which vessel answered (for ops)
        print(f"[/chat/stream] vessel={chosen_vessel['name']}")

    return StreamingResponse(gen(), media_type="text/event-stream")

# Run with: uvicorn main:app --host 0.0.0.0 --port 9000

Exercise

Build the FastAPI service. Run it. Test 4 scenarios: (1) cloud reachable, (2) cloud blocked at firewall, (3) cloud reachable but slow, (4) Ollama daemon stopped. Confirm the user-visible behavior is graceful in all four. Write down what 'graceful' means in the failure modes — that's your real ops contract.