Every Deploy Reported Success. The Endpoint Had Been Unreachable for Two Weeks.

An internal endpoint on the platform I work on had been returning 404 for about two and a half weeks. Nobody had filed a ticket, every deployment in that window had reported success, and the service behind it was healthy the whole time, running, warm, serving.

I only found it because I was auditing an architecture diagram against the live system, expecting to correct the diagram.

This is the fourth post in a series about an internal developer platform that generates infrastructure and deploys into customers’ cloud projects. The first covers the architecture. This one is much smaller and much more portable: it is about reading a 404 properly, and about a class of configuration bug that no amount of green CI will surface.

A 404 is not one thing

The moment that turned this from confusion into a diagnosis was noticing that two different paths 404’d differently:

POST /api/v1/git/repo         → 404   text/html
POST /api/v1/git/create-repo  → 404   text/plain

Same status code. Completely different meanings.

text/html, the request never reached the service at all. The hosting layer has a /** catch-all that serves the frontend single-page app, and the SPA returned its own not-found page. No rewrite existed for this path.
text/plain, the request did reach the service. This is the Go router 404ing internally, because that exact path is not registered on it. The rewrite exists and points at a live service, but the service disagrees about the path.
application/json, the service is running and this is a real application response. Once I had the route fixed, hitting it returned 400 application/json from the controller’s own field validation, which is the sound of success.

So the two 404s were telling me two halves of the same story: the old path had a rewrite that no longer matched anything the service served, and the new path had no rewrite at all. Somewhere, a rename had happened on one side of a boundary and not the other.

That content-type check costs nothing and I now reach for it first:

curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://api.example.com/some/path

The route lived in two unlinked places

Here is the actual shape of the problem.

A request to the public API domain hits a hosting layer that holds a list of rewrites, path → Cloud Run service. That list is generated by a deploy script and stored in a secrets manager. Separately, the service itself registers its routes in its Go router. Both must agree, and nothing enforces that they do.

The git history is unambiguous:

19 Jun   in sync    router: /v1/git/create-repo    rewrite: /api/v1/git/create-repo
20 Jul   RENAME     router: /v1/git/repo           rewrite: unchanged
06 Aug   found      during an unrelated diagram audit

One commit, one side of the boundary. The commit was reviewed. It was also entirely reasonable in isolation. It renamed a route in a Go file, and there is nothing in that Go file that hints a copy of the path is stored in a secrets manager.

Why it never healed

This is the part I find genuinely interesting, and the reason the bug survived seventeen days of active development.

The deploy script does not rewrite the whole route table. It merges the entry for the service being deployed into the existing table, keyed by service id:

map(select(.run.serviceId != $svc and .source != "/**"))
  + [{source: $src, run: {serviceId: $svc, region: $region}}]
  + map(select(.source == "/**"))

Read that carefully: every entry that is not the service being deployed is carried forward verbatim. That is a sensible design. It means deploying one service cannot clobber another’s route.

It also means a wrong entry is immortal. It will be faithfully preserved by every future deploy of every other service, forever, until somebody redeploys that one specific service’s hosting config.

The version history bore this out exactly. Across ten consecutive versions of the route table, the broken entry is byte-identical in every one, including two versions written by my own deploys of an unrelated service earlier that week. My deploys succeeded. They also carefully preserved the bug.

The smoke test that could not have caught it

The deploy script is not naive. It has a guard that refuses to deploy if the /** catch-all is not the last rule, because a catch-all above an API route shadows it. That guard is correct, and it is well-reasoned.

It also has a smoke test, which does exactly the content-type check I described above:

SMOKE_URL="https://${PROJECT_ID}.web.app${SRC_PATH}"
CONTENT_TYPE=$(curl -s -o /dev/null -w '%{content_type}' --max-time 10 "$SMOKE_URL")
if [[ "$CONTENT_TYPE" == *"text/html"* ]]; then
  log_warn "route may still be shadowed by /**"
fi

Three things are wrong with it, and they are all instructive:

It probes the wrong host. The default hosting domain is not the custom domain that real traffic uses. Those can resolve to different hosting configurations.
It warns rather than fails. A warning in a CI log that already printed “deployed successfully” is not a signal anybody reads.
It only checks the route being deployed right now. This is the fatal one. The broken route belonged to a service nobody was deploying. A check scoped to the change can never catch a regression in something you did not change.

That third point is the general lesson. Most deploy-time validation asks “did my change land?” Almost none asks “is everything still reachable?”, and the second question is the one that catches drift.

A smaller trap in the same layer

One more, because it cost me an hour separately and it is not obvious: a /** rewrite does not match its own bare parent.

rewrite:  /api/projects/v1/detail/**

/api/projects/v1/detail/123   → matches
/api/projects/v1/detail       → does NOT match, falls through to the frontend catch-all

If your API has an endpoint that is both a collection and a prefix, you need both rules. I verified this against a live deployment rather than trusting the docs, and I would suggest you do the same, because the behaviour differs between hosting providers.

What actually fixes this

Derive, don’t duplicate. The real fix is for the route to exist in exactly one place. The service already knows its own path. It is in an environment variable the service reads at startup. A deploy that asked the service for its route, instead of being told the route separately, could not drift.

Check the whole surface, not the delta. A job that walks every registered route and asserts each returns something other than text/html from the public domain would have caught this on day one. It is a loop over a list you already have.

Fail, don’t warn. If an assertion is worth writing, it is worth breaking the build over. A warning printed after “deployed successfully” is decoration.

What I would take from this

Read the whole response, not the status code. 404 text/html, 404 text/plain and 404 application/json are three different bugs in three different places. The status code told me nothing; the content-type told me exactly which layer to open.

“Merge” is a synonym for “carry forward the mistakes.” Any config assembled by merging into previous state will preserve errors indefinitely, and every subsequent successful deploy becomes evidence that things are fine. If you merge config, you need a periodic check of the merged result, not of the merge.

Validation scoped to the change has a blind spot exactly the size of everything else. This bug lived in the gap between “my deploy worked” and “the system works.”

Silence is not health. Seventeen days, no ticket. The endpoint was called by one internal flow that people had stopped exercising. Absence of complaints measured attention, not correctness, which is the same lesson the second post in this series reached from the opposite direction, where the pain was real and still invisible because nobody was looking at the right percentile.