Why our Gitea went down every Sunday at midnight

For a while, our self-hosted Gitea would blink out around midnight WIB. Not every night, but often enough to be annoying, and always around the same time. Here’s what was actually happening.

Symptom

The gitea pod would show a fresh start time in the small hours, but with a restart count of zero. That’s the tell: the container didn’t crash. The whole pod was recreated, which means the node it lived on went away.

Root cause

Two things combined:

1. A GKE maintenance window anchored at 17:00 UTC, which is exactly 00:00 in WIB (UTC+7).
2. A single-replica StatefulSet with a ReadWriteOnce disk.

When GKE performs a node upgrade during the maintenance window, it drains the node. For a single-replica stateful workload, draining means:

cordon node → evict pod → wait for PersistentDisk to detach
            → reschedule on new node → reattach disk → start

That detach/reattach dance is most of the downtime. There’s no second replica to serve traffic while it happens.

The lesson

If a service must not blink, it needs either:

• More than one replica (with a PodDisruptionBudget), or
• To not be on the cluster at all when it doesn’t need to be.

This very blog is an example of the second option, it’s static files in a bucket, so a node drain can’t touch it.

Quick checklist

Keep your stateful services boring and redundant. Keep your static content off the cluster entirely.