REST and gRPC APIs

Two protocols, two use cases

TF Serving exposes both REST (port 8501) and gRPC (port 8500). REST is easiest — standard HTTP JSON requests, integrate from anything. gRPC is faster — binary Protocol Buffers serialization, lower latency, lower bandwidth for high-volume production traffic.

REST URL pattern: http://host:8501/v1/models/{MODEL_NAME}:predict. Add /versions/{N} to target a specific version.

For Python clients, REST is two lines (requests.post). gRPC needs the tensorflow-serving-api package and Protocol Buffer message types.

When to use which: REST for development, debugging, low-volume production, integration with web frameworks. gRPC for high-throughput production where serialization overhead actually shows up in latency budgets.

Code

REST — curl로 한 줄·bash

# Check model status
curl http://localhost:8501/v1/models/my_model
# {"model_version_status": [{"version": "2", "state": "AVAILABLE", ...}]}

# Predict (row format — one instance per element)
curl -d '{"instances": [[1.0, 2.0, 3.0, 4.0]]}' \
  -X POST http://localhost:8501/v1/models/my_model:predict
# {"predictions": [[0.1, 0.8, 0.1, ...]]}

# Target a specific version
curl -d '{"instances": [[1.0, 2.0]]}' \
  -X POST http://localhost:8501/v1/models/my_model/versions/1:predict

Python REST client·python

import requests
import json
import numpy as np

url = "http://localhost:8501/v1/models/my_classifier:predict"
input_data = np.random.rand(2, 784).tolist()
payload = json.dumps({"instances": input_data})
headers = {"content-type": "application/json"}

response = requests.post(url, data=payload, headers=headers)
predictions = response.json()['predictions']
print(f"Top class for sample 0: {np.argmax(predictions[0])}")

Python gRPC client — production용·python

import grpc
import numpy as np
import tensorflow as tf
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc

# pip install tensorflow-serving-api

channel = grpc.insecure_channel('localhost:8500')
stub    = prediction_service_pb2_grpc.PredictionServiceStub(channel)

request = predict_pb2.PredictRequest()
request.model_spec.name           = 'my_classifier'
request.model_spec.signature_name = 'serving_default'

input_data = np.random.rand(2, 784).astype(np.float32)
request.inputs['keras_tensor'].CopyFrom(
    tf.make_tensor_proto(input_data, dtype=tf.float32))

result_future = stub.Predict.future(request, timeout_seconds=10.0)
result = result_future.result()
predictions = tf.make_ndarray(result.outputs['output_0'])
print("Predictions shape:", predictions.shape)

channel.close()

Two protocols, two use cases

Code

Exercise

Progress

Comments 0