Edge Deployment & Multi-Adapter Patterns

S-LoRA: thousands of adapters on one base

S-LoRA enables serving thousands of LoRA adapters simultaneously on the same base model. Two innovations:

Unified paging: manages adapter weights in a paged memory system, like vLLM's PagedAttention but for adapters.
Heterogeneous batching: batches requests using different adapters in the same forward pass.

Can handle 2,000+ simultaneous adapters with minimal overhead. The right pick when you have one fine-tuned variant per customer/tenant.

Edge deployment

Platform	Format	Best for
Apple (macOS / iOS)	MLX / Core ML	Mac apps, iPhone inference
Android	GGUF via llama.cpp	On-device inference
Browser	ONNX / WebGPU	Web-based inference, no server
Embedded	GGUF (heavily quantized)	IoT, robotics

When to retrain

Quality degradation — monitoring shows declining output quality.
Domain shift — user inputs have changed since training.
New data available — you've collected enough new production data to matter.
New base model — a better foundation model is released.

For incremental updates, train from the previous checkpoint. For major updates or new base models, start fresh.

Code

Multi-adapter retrieval at request time·python

# Server (S-LoRA-style; vLLM 0.6+ supports many adapters)
# python -m vllm.entrypoints.openai.api_server \
#     --model meta-llama/Llama-3.1-8B-Instruct \
#     --enable-lora \
#     --max-loras 64 \
#     --max-lora-rank 32 \
#     --lora-modules \
#         tenant-acme=./adapters/acme \
#         tenant-globex=./adapters/globex \
#         tenant-stark=./adapters/stark \
#         (... up to max-loras ...)

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

def chat_for_tenant(tenant_id: str, user_msg: str) -> str:
    r = client.chat.completions.create(
        model=f"tenant-{tenant_id}",   # routes to that tenant's adapter
        messages=[{"role": "user", "content": user_msg}],
        temperature=0.7,
    )
    return r.choices[0].message.content

Exercise

Sketch a deployment plan for one fine-tuned model in your stack: which serving framework (Ollama / vLLM / TGI), single or multi-adapter, what the API surface looks like, what 5 metrics you'd monitor, what triggers retraining. This is the doc that takes you from 'I have a trained adapter' to 'it's serving real users'.