Multi-Vessel Fallback

Why fallback is the point

The most useful local AI pattern is not local everywhere — it's local-first with cloud fallback. Local handles 80–95% of work for free, with privacy. Cloud handles the rest when local can't. Your orchestrator picks per-call, transparently.

The multi-vessel pattern (Pippa)

cwkPippa runs seven AI vessels behind one orchestrator: Claude (primary, frontier reasoning), Codex (ChatGPT Pro OAuth), Gemini (Cloud Code Assist), then Grok, Kimi and GLM, with Ollama as the local fallback. It shipped with four; the other three arrived over 2026 without the orchestrator changing shape, which is the argument for this pattern in one sentence. The orchestrator picks based on:

Explicit user choice ("use Codex for this turn").
Health check ("Claude is degraded right now → try Codex").
Cost / privacy ("this is a heartbeat job → local").
Capability ("vision needed → Gemini or local Gemma").

The orchestrator is dumb on purpose

It doesn't know about NDJSON or SSE or tool argument formats. It only knows: try vessel A; if A's health check is bad or A throws before the first chunk, try vessel B. The adapters absorb provider details so the orchestrator stays simple.

Fallback has a deadline: the first token

That italicised clause is the whole contract. A generator that has already yielded cannot take its output back — those chunks are rendered, or written to a socket, or read by a human. If vessel A dies after ten tokens and the orchestrator quietly moves to vessel B, the caller receives A's ten tokens with B's complete answer spliced onto the end, and nothing in the stream marks the seam. So the switch is allowed only while nothing has been emitted; after that the orchestrator raises VesselFailedMidStream and the caller decides whether to restart from scratch on another vessel. Transparent fallback and a partial answer are mutually exclusive — pick which one you are offering, and say so in the type.

EV v3 · 2026-09-18: The orchestrator here used to yield vessel A's chunks from inside the try and fall through to vessel B on any exception — including one raised halfway through A's answer. Chunks already yielded are gone, so the caller got A's partial output with B's full answer appended, and no marker between them. That was wrong from the day it was written; nothing in any provider changed to cause it. It stayed invisible partly because the Ollama adapter two lessons back never raised mid-stream at all — it returned quietly on a truncated stream, which the same pass fixed, and fixing it is exactly what would have started producing spliced answers here. The orchestrator now tracks whether anything has been emitted and raises VesselFailedMidStream once it has. It also takes an explicit priority argument, because the ops track's shipped service wants cloud-first while this lesson's default is local-first, and until now both were reading the same hard-coded list. The vessel names now match the prose too — it calls the second vessel Codex while the code registered openai/OpenAIAdapter. The vessel count is the world moving rather than an error: four was right when this quest shipped, and Grok (July), Kimi (July) and GLM (August) joined afterwards.

Code

Multi-vessel orchestrator with fallback·python

class AllVesselsFailed(RuntimeError): ...
class VesselFailedMidStream(RuntimeError):
    """A vessel died after its first chunk reached the caller."""

class AIOrchestrator:
    """Try vessels in priority order; fall back on failure."""

    def __init__(self, priority: list[str] | None = None):
        self.vessels: dict[str, AIAdapter] = {
            "ollama": OllamaAdapter(model="qwen2.5:32b"),
            "claude": ClaudeAdapter(model="claude-opus-4-7"),
            "codex": CodexAdapter(model="gpt-5-codex"),
            "gemini": GeminiAdapter(model="gemini-3-pro"),
        }
        # Local-first by default. Pass priority=["claude", ..., "ollama"]
        # for a cloud-first service that keeps local as the outage net.
        self.priority = priority or ["ollama", "claude", "codex", "gemini"]
        self.active: str | None = None  # explicit override; None = use priority

    async def stream(self, messages: list[dict], **kwargs):
        order = [self.active] if self.active else self.priority
        last_err: Exception | None = None
        for name in order:
            vessel = self.vessels.get(name)
            if vessel is None:
                continue
            yielded = False
            try:
                if not await vessel.health_check():
                    continue
                async for chunk in vessel.stream(messages, **kwargs):
                    yielded = True
                    yield chunk
                return
            except Exception as e:
                # Fallback is only transparent BEFORE the first chunk. Once a
                # token has reached the caller, switching vessels would splice
                # a second, complete answer onto a partial one. Fail loudly and
                # let the caller decide whether to retry from scratch.
                if yielded:
                    raise VesselFailedMidStream(
                        f"{name} failed after streaming had started") from e
                last_err = e
                continue
        raise AllVesselsFailed(f"all vessels failed; last error: {last_err}")

Per-task vessel selection·python

async def route_task(orchestrator: AIOrchestrator, task: dict, prompt: str):
    """Route a task to the right vessel based on its shape."""
    if task.get("private") or task.get("heartbeat"):
        orchestrator.active = "ollama"          # never leave the machine
    elif task.get("needs_vision"):
        orchestrator.active = "gemini"          # vision-strong cloud
    elif task.get("needs_thinking"):
        orchestrator.active = "claude"          # frontier reasoning
    else:
        orchestrator.active = None              # default priority

    async for chunk in orchestrator.stream([{"role": "user", "content": prompt}]):
        if chunk.done:
            break
        yield chunk.content

Exercise

Implement a 2-vessel orchestrator with Ollama + one cloud adapter (use a stub if you don't have a real key). Stop the Ollama daemon and confirm the orchestrator falls back to cloud automatically. Restart Ollama and confirm priority returns to local on the next call. Then force the hard case: make the local vessel fail *after* a few chunks (raise from inside its stream()) and confirm you get VesselFailedMidStream rather than two answers glued together.