The App Called OpenAI Directly. The Gateway Should Have Been in Between.

A SaaS runs Go microservices on EKS with PostgreSQL and Kafka. This spring the team shipped an AI assistant on Python and FastAPI that calls OpenAI. Under load the workers queue without bounds, a noisy customer can spend the budget for everyone else, and a slow provider blocks a worker for the lifetime of the stream. The fix is not a new model. It is a boundary in Go between the app and every provider.

The first version works and that is why it stays too long. The app calls one provider from inside the request handler. Timeouts are defaults. Retries have no place to live. When load goes up, synchronous calls become a queue nobody instrumented.

The direct call hides the decisions you need to make

When the app imports an OpenAI client, which model answers, with what timeout, at what cost, for which customer, is scattered across call sites. Routing is a branch. Budgets are a comment. Observability is a wrapper that someone meant to add.

A gateway inverts that. The app calls one endpoint with a streaming response. The gateway decides the provider and records the cost. The app keeps product logic. The gateway keeps provider logic.

Every provider fails differently. One returns 429 on a burst. One hangs without a first byte. One streams fifty tokens and stalls. The gateway speaks those three dialects so the app does not have to.

Streaming is where the proxy earns its complexity

The useful path is a stream. The gateway opens an upstream stream and relays Server-Sent Events while holding both connections. The complexity is the lifetimes.

A Go handler must flush, respect cancellation, and not buffer the whole answer to decide whether to retry. The gateway must propagate client cancellation upstream rather than holding a worker until a global timeout fires.

The rule is the first byte. Before the gateway has written a header and a chunk, switching providers is invisible. After that the client has rendered a prefix. Retrying silently would resend that prefix or leave a gap. The gateway holds the upstream response until a first byte arrives or a first-byte timeout fires, only then commits the client stream, and after commitment treats failures as client-visible errors.

func (h *Gateway) Stream(w http.ResponseWriter, r *http.Request) {
    flusher, _ := w.(http.Flusher)
    ctx, cancel := context.WithTimeout(r.Context(), h.cfg.FirstByteTimeout)
    defer cancel()
    upstream, err := h.router.OpenStream(ctx, r)
    if err != nil {
        h.writeProviderError(w, err)
        return
    }
    defer upstream.Close()
    first, err := upstream.Peek(ctx)
    if err != nil {
        h.writeProviderError(w, err)
        return
    }
    w.Header().Set("Content-Type", "text/event-stream")
    w.WriteHeader(http.StatusOK)
    w.Write(first.Bytes())
    flusher.Flush()
    for {
        chunk, err := upstream.Next(ctx)
        if err != nil {
            if errors.Is(err, io.EOF) {
                return
            }
            h.logStreamError(r, err)
            return
        }
        w.Write(chunk)
        flusher.Flush()
        select {
        case <-r.Context().Done():
            return
        default:
        }
    }
}

Retry and fallback are not the same operation

Retry repeats the same provider. Fallback changes the provider. They share code and should not share policy.

A 429 with Retry-After, a 503, or a timeout can be retried once with jitter inside the first-byte window. An auth error, a 400, or a content filter block should not be retried at all. A sustained stall is the case for fallback.

Fallback is not a URL change. Each provider has its own shape, token rules, and framing. The gateway needs a normalized request and three translators. LiteLLM and Portkey solve that normalization and are worth evaluating before writing the third translator. Their limit is the edge you care about most, like precise token accounting for a budget. Keep the translation behind an interface you own.

A token budget is a product boundary, not a counter

A request carries a customer id. The gateway checks remaining budget, reserves an estimate for input tokens, and settles on true usage after the stream completes. If exceeded, it returns 429 with a Retry-After that matches the budget window and says when it resets. A generic rate limit cannot be acted on.

Input tokens can be estimated before the call and corrected after. Output tokens are counted as the stream arrives. Settlement must happen once, even on cancellation or partial failure. The store is a durable counter, typically PostgreSQL with a per-customer row or Redis with atomic increments. If the provider never produced a completion, refund. If it produced half an answer, keep that cost. A budget that only counts successes subsidizes retries.

A cache speeds up the second question, not the first

Caching the same prompt saves money and latency. Caching a prompt grounded in one set of documents and replaying it after the docs changed returns a fluent lie.

The key must include what made the answer: a hash of the normalized prompt, a hash of the retrieval result, the model identifier and parameters, and the customer scope if tenancy matters. A prompt-only key silently serves stale grounding.

TTL should be short and size bounded. A per-tenant LRU with an explicit cap is easier to reason about than a global cache one customer can flood. On a write to the index that changes rank or content, invalidate by document set rather than by time.

The Python workers still needed hardening without a rewrite

First, a queue with a bound. The gateway enqueues jobs rather than calling workers synchronously, and workers consume from Kafka or a task queue with max concurrency. When depth grows, the gateway sheds load with 429 rather than queueing forever.

Second, timeouts that mean something. A route that awaits retrieval then generation inherits the slowest part. Each hop gets its own budget, and the overall request gets one. Every request carries a request id from the gateway, and every log line includes customer, model, latency split, input and output tokens, and cache decision. If you cannot see which hop spent the budget after an incident, you did not instrument the boundary.

@app.post("/rag/query")
async def rag_query(req: RAGRequest, request_id: str = Header(alias="x-request-id")):
    logger.info("rag.query.start", extra={"request_id": request_id, "customer": req.customer_id})
    try:
        docs = await asyncio.wait_for(retriever.search(req.query, top_k=req.top_k), timeout=4.0)
    except asyncio.TimeoutError:
        logger.warning("rag.retrieval.timeout", extra={"request_id": request_id})
        raise HTTPException(status_code=504, detail="retrieval timeout")
    return {"docs": docs, "index_version": docs.index_version}

Prompt and retrieval need a regression check before deploy

GitHub Actions is the right place for this, and the check should be boring.

Hold a set of golden prompts paired with the document sets they should retrieve and assertions that catch drift. Not an LLM grading an LLM. Checks like the expected document appears in top results and the citation span is present. The data is fixtures. The failure is a diff someone can read in a pull request.

name: rag-regression
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python -m rag.eval --suite fixtures/golden.json --against staging

That suite blocks the innocent change that still breaks grounding, like trimming a system prompt and dropping the citation instruction, or re-embedding an index with a new chunk size and shifting rank. The model still returns tokens. Only the regression knows they are grounded in the wrong paragraph.

The deploy lives in the same cluster as everything else

The gateway is a Docker image deployed by Helm to the existing EKS cluster. It sits on the same network, Prometheus, and Grafana as the Go services it fronts. The chart sets requests and limits, a HorizontalPodAutoscaler on stream concurrency, and a PodDisruptionBudget so a node roll does not drop every stream.

replicaCount: 3
resources:
  requests: { cpu: "500m", memory: "512Mi" }
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 12
  targetCPUUtilizationPercentage: 70

Grafana gets four panels before anything fancier: p50 and p90 time to first byte per provider, full stream duration per route, per-customer token spend with budget state, and cache hit rate split by prompt versus retrieval cache. Fallback counts and circuit breaker state sit next to latency so a routing change is visible as a routing change.

What I would tell someone building this

Put the gateway in front on day one, even if it only proxies. A passthrough with logging, metrics and one timeout teaches you the shape of your traffic. Routing, budgets and cache can follow without changing the caller.

Hold the stream, do not buffer it. Commit the client response only after the provider has committed bytes. Retrying before first byte is free. Retrying after is a second answer.

Count cost where the stream ends, and make the denial useful. Reserve, settle, refund on failure, and return a budget error that tells the caller when it can retry.

Key the cache on what made the answer, not just the question. Prompt hash plus docs hash plus model and params. If the docs can change under the prompt, the cache has to know.

Keep the RAG service in the language its authors own, harden the boundary around it. A queue, per hop timeouts, structured logs that split retrieval from generation, and a regression suite that blocks drift in CI are enough.