Two Environments, One Resource. The Label Said 'dev'. The Name Said Otherwise.

We added a dev environment to a platform that had, until then, deployed straight to production. Every branch declared dev, every resource in the generated Terraform carried env: dev as a billing label, and every deploy reported success.

Which is why it took a while to notice that a push to dev was replacing the production service.

This is the fifth post in a series about an internal developer platform that generates infrastructure and applies it into customers’ cloud projects. The first covers the architecture; post three covered why the state file must never be the source of truth. This one is about the cheapest mistake in the series: an environment that existed only as a label.

Two states, one resource

On Google Cloud, the environment concept we wanted was already everywhere except where it mattered. The generated Terraform sets labels = { env = "<environment>" } on every resource that supports it, billing attribution works, cost reports split cleanly by environment. The state file got an environment segment in its prefix, so dev and main state could never clobber each other.

Two states, cleanly separated, in both places that handle accounting.

Here is the problem: neither of those participates in identity. GCP identifies a Cloud Run service by its name within the project, not by its labels and not by the state file that describes it. A gcloud run deploy with the same service name in the same project replaces the existing service, always, silently, no conflict error. Labels are metadata about a resource. The name is the resource.

So dev and main looked like two environments:

dev    state: …/dev/deploy/default.tfstate     label: env:dev     service: checkout
main   state: …/main/deploy/default.tfstate    label: env:main    service: checkout

And ran as one. Every dev deploy targeted the same project, used the same service name, and landed on top of whatever main had running. The platform reported success because the deploy did succeed. It succeeded at destroying the other environment.

Labels do not participate in identity. The name is the resource.

The fix that sounds trivial and isn’t

The fix is to make the environment part of the resource name: dev-checkout instead of checkout, via a prefix applied at deploy time:

deployedServiceName(prefix, service):
  empty prefix   → name unchanged     (production stays production)
  non-empty      → {prefix}-{service} (dev-checkout, staging-checkout, …)

“Prefix the name” is a one-line idea. The implementation is where it bites, in four ways:

1. Idempotency. The name is computed on every deploy, from the same prefix and service name, so re-deploying can never produce dev-dev-checkout. The function is pure: same input, same output, no state to drift.

2. Substring safety. A prefix is not a substring check. With prefix d, the service delete-user must become d-delete-user, not stay delete-user because d is “already there”. You do not test for HasPrefix(service, prefix). You prepend, always, and let the resulting name be what it is.

3. The name is used in exactly two places, and both are the same place. gcloud run deploy names the service; gcloud run services describe reads it back. If the describe path used the raw name while deploy used the prefixed name, the smoke test would probe a service that never existed, and we would have traded a silent overwrite for a silent 404.

4. Zero-change guarantees. Eleven of twelve projects had no prefix configured. For them the function returns the name unchanged, byte-identical to what production has always deployed. The fix had to be provably invisible everywhere it was not enabled, or it would have renamed every service in every project at once.

The smaller trap in the same story

The environment value itself needed normalising before it ever reached a name or a label: normalizeEnvironment trims, lowercases, and defaults to dev. That is a correctness requirement, not cosmetics, because the platform does not invent names for you. A client typing Dev on one deploy and dev on the next creates two environments in the state prefix:

…/Dev/deploy/default.tfstate
…/dev/deploy/default.tfstate

…both deploying to the same resource name if the prefix path is fed the raw value. One character of case, two states, one resource, the exact collision this whole post is about, manufactured by a typo.

A typo in case creates two states and one resource. Normalize before the value touches anything.

What I would take from this

Labels are attribution. Names are identity. Anything that must distinguish environments, tenants, or deployments has to be in the name, the label tells you who paid for it, the name says what is actually running. Designing “multi-environment” and only touching labels is designing accounting, not infrastructure.

A deploy that succeeds is not evidence. Every apply in that window was green. The system will happily deploy you onto the other environment and tell you it worked. The worst failures in this series share that shape: the 404 in the fourth post lived under two weeks of green deploys, and this collision lived under every branch.

Idempotency is a deploy-time property, not a code-quality one. The prefix function had to be pure because it runs on every deploy and its output feeds both the name and the smoke test. A function that “sometimes” returns a different name is not a bug waiting to happen. It is a bug that already happened, repeatedly, silently.

Normalise at the boundary, before the value touches anything. Case, whitespace, defaults: pick the canonical form once, at the request edge. The rest of the system (names, state prefixes, labels, subscriptions) then has exactly one representation to reason about. We store dev, not Dev, not dev.

Check the thing you changed and the thing you did not. The smoke test that deploys dev-checkout and probes dev-checkout is correct and would have caught nothing, because it never deploys the unprefixed service that main still owns. Multi-environment validation has to cover the unchanged environment too, the one you are not touching right now is the one being overwritten.

This series: the platform architecture · one flag, 71 minutes · delete the state file · every deploy reported success · this one.