The model server you don't have to write
TensorFlow Serving is a production-grade model server built for one job: serving SavedModels at low latency and high throughput. It's the bridge between your trained model and apps that need predictions — web APIs, mobile backends, data pipelines, real-time systems.
You could load a Keras model in a Flask app and call model.predict(). It works for prototypes. But TF Serving solves problems you'll hit in production:
- Model versioning — deploy a new version without restart, hot-swap live
- Request batching — auto-groups concurrent requests to use GPU efficiently
- Multi-model — serve dozens of models from one server via a config file
- gRPC + REST — both protocols natively, REST for simplicity, gRPC for performance
- C++ runtime — uses TF directly with no Python overhead