On the platform I work on, nobody writes Terraform by hand. A customer designs their backend on a canvas, a generator turns that into HCL, and CI applies it into the customer’s own cloud project. The largest tenant is 613 resources from a 10,000-line generated config, 131 of them Cloud Run services.
That arrangement has a failure mode that took me a while to name properly, because nothing about it errors.
Declared is not running
A Cloud Run service and the container it runs are two different objects with two different owners. Terraform owns the service. CI owns the image. And Terraform will happily create a service whose image does not exist yet, because at generation time it frequently does not: the customer just drew a new service on the canvas, and no build has ever run for it.
So the generator emits a placeholder image, the apply succeeds, and CI replaces it a few minutes later when the build finishes.
Two independent events. The first one is reported as success. If the second one never lands, you have a Cloud Run service that reports READY, passes its health check, accepts traffic, and answers with a placeholder container. Terraform will not tell you, because Terraform created exactly what it was asked to create.
The nastiest version of this I have seen was not an HTTP endpoint at all. It was a Pub/Sub push subscription pointed at a placeholder consumer. The placeholder returned 200 to everything, Pub/Sub read that as successful delivery, and acknowledged and deleted the messages. Nothing errored anywhere. The data was simply gone, cleanly and on purpose, because every component behaved exactly as documented.
Then state loss makes it worse
Generated infrastructure re-emits the whole graph on every deploy. That is normally fine. It stops being fine the moment you combine it with recovery.
If state is lost and you re-apply, the generator produces the same config it always produces, which includes the placeholder image, because the generator has no idea what is currently running. A recovery you performed to restore production quietly rewrites 131 live services back to placeholders. The apply succeeds. The dashboard goes green. The product stops working.
The fix is that the generator has to read before it writes. For any Cloud Run service that already exists, the image field is not generated from the canvas, it is read from the live service and written back unchanged. The canvas describes the service; only CI decides what image it runs. Generation stops being purely declarative for exactly this one field, and that is a deliberate exception, not an oversight.
A tag is a pointer, not a version
Once you are reading the live image, the next question is what you are actually storing. This is where a lot of pipelines have a rollback story that is not true.
app:v2.1 is a mutable pointer. Somebody re-pushes that tag, deliberately or by re-running a pipeline, and the same string now resolves to different content. Every service that pulls it later gets the new bytes. Nothing in your config changed, so nothing in your diff shows it.
Now roll back to v2.1. You did not roll back to the build you were thinking of. You rolled back to whatever v2.1 means today.
app@sha256:aa1f... cannot do that. The digest is the content address. If the bytes differ, the digest differs. Pin to the digest and rollback stops being a hopeful gesture and becomes an actual guarantee: you get those bytes or you get an error, and both of those are fine outcomes.
The cost is that digests are unreadable and your config stops being human-diffable in a satisfying way. That is a real cost. It is much smaller than the cost of a rollback that silently did not roll anything back.
Promotion, once the artefact is immutable
Immutable artefacts are what make multi-environment promotion worth having rather than just ceremony.
If dev, staging and production each build their own image from the same commit, you have not promoted anything. You have three builds that you hope are equivalent. Base images move, dependency resolution is not reproducible, and the thing you tested is not the thing you shipped.
Promotion should move a digest, not trigger a build. Staging runs sha256:aa1f, the gate passes, production runs sha256:aa1f. The bytes that were tested are the bytes that serve traffic, and that is the entire point.
Building this on top of a generator had a side effect I did not expect. Because environments are described as configuration rather than as hand-built infrastructure, standing up a new one stopped being a project. It went from hours to about five minutes, and existing customers could be migrated into the model as metadata, with no downtime, because nothing about their running infrastructure had to change.
What I would tell someone building this
Write down which fields your generator owns and which it must read. For us it is exactly one field, and it is the one that decides what code is running. Everything else is generated from the design. Making that boundary explicit is what stops a recovery from becoming an outage.
Assume the build and the apply will get out of order. Not as an edge case. They are separate systems with separate failure modes and no shared transaction. Design for the state where one has happened and the other has not.
Treat a health check as evidence of liveness, not of correctness. A placeholder passes health checks. A placeholder returns 200. On a message consumer, returning 200 is worse than crashing, because crashing preserves the message.
Pin to digests. If your rollback target can change under you, you do not have a rollback.