When inference is "slow", the first reflex is to scale GPUs. Expensive. Half of those stories are cured by an ordinary nginx in front of the service — a layer that solves nothing about the model but honestly takes care of all the chores around it.
What the gateway takes over
- Rate limit per key. The model should not suffer because one integrator wrote a loop without a sleep.
- Timeouts.
proxy_read_timeoutshorter than the client's patience. A 503 in 300 ms beats a 200 in a minute — the latter also pins a worker the whole time. - Caching idempotent requests. For classifiers with stable inputs a 30–60% hit rate is routine, and it is inference-queue capacity stolen back for free.
- Honest status codes. 429 with
Retry-Afterinstead of "it will answer eventually, just slowly".
location /infer {
limit_req zone=infer burst=20 nodelay;
proxy_read_timeout 0.3s;
proxy_next_upstream error timeout http_502 http_503;
proxy_cache infer_cache;
proxy_cache_key "$cache_key";
proxy_cache_valid 200 5m;
add_header X-Cached $upstream_cache_status;
}
The honesty check
A month after turning the cache on, look at what actually lands in X-Cached. If it is a HIT on personal data or on anything that clearly must be recomputed, the cache needs fixing, not congratulating. A cache that stores the wrong thing is worse than no cache: it lies silently.
The cheapest inference is the one the model never had to run.
Blue-green at the upstream level, grey 429s on the dashboard instead of red pages, and GPUs that stay cold when nobody needs them — the entire profit of a layer nobody ever praises. It does not need praising. It just needs to not be turned off "temporarily" on a Friday.