I wrote a small Go service that sits between Google Cloud’s alerting and the people who have to do something about it. An alert arrives, the service works out who is on call, pulls the relevant source files out of Gitea or GitHub, runs root-cause analysis against them, and posts to Mattermost with a probable cause attached rather than a bare stack trace. If nobody acknowledges within a window, or the same error keeps recurring, it escalates by phone.
It is not a large service. It was, however, the first thing I had built where the central design constraint was a mismatch between two numbers, and I think that mismatch is the most transferable thing about it.
The sender is not waiting for you to be useful
Cloud Monitoring posts the alert over HTTP and expects a prompt 2xx. If it does not get one, it treats the delivery as failed and retries. That is the correct behaviour for a notification channel, and it means a slow handler does not delay an alert. It duplicates it.
Now look at what the service actually needs to do. Resolving who is on call is a cheap lookup. Fetching source from a git host is a network call subject to rate limits. Running an LLM over those files takes seconds at best and tens of seconds when the context is large. Posting to chat is another network call.
Every one of those is slower than the budget, and the last one is slower by an order of magnitude. There is no version of this where you do the work and then reply.
So the handler does the only three things that are both fast and mandatory: validate the payload, persist it, and acknowledge. Everything after that belongs to a worker. The HTTP endpoint’s job is to make the alert durable and get out of the way.
This is unglamorous and it is most of the design.
Acknowledging early means you now have two sources of duplicates
The moment you split receive from process, you have taken on at-least-once semantics twice over. The sender retries. Your own queue redelivers. A single failure can therefore arrive many times, and each arrival looks exactly like a fresh problem.
The tempting fix is a dedupe cache with a TTL. That is a guess dressed as a mechanism: too short and you page twice, too long and you miss a genuine recurrence.
The better answer is that the sender already told you what this is. Cloud Monitoring alerts carry an incident identifier that is stable across retries and across the lifetime of the condition. Key your incident record on that, upsert rather than insert, and duplicates become a counter increment instead of a second page. No cache, no TTL, nothing to tune.
The general form of this: when you need an identity for something, look for one the upstream system has already committed to before you invent your own. Anything you generate on receipt is an artefact of your transport, not of the event.
Resolve the assignee once, then stop asking
Here is a bug I designed in and had to take out.
The obvious way to escalate is to notice an unacknowledged incident, look up who is on call, and phone them. The problem is that “who is on call” is a function of time. If the incident was raised at 08:55 and the rota rolls over at 09:00, the escalation reaches a different person than the one who was notified, and the person actually looking at it never gets the call.
The fix is that the assignee is resolved once, at notification time, and written onto the incident. Escalation reads that field, it does not re-derive it. The rota answers “who was on call when this fired”, and once that question has an answer it is not asked again.
There is a deeper version of this that applies well beyond on-call rotas. Any value you derive from a time-varying source and then act on later has to be captured at decision time, not recomputed at action time. Recomputing looks cleaner and is a source of very confusing bugs.
The clever part must be allowed to fail
The root-cause analysis is the reason the service exists, and it is also the least reliable component in it. The git host can rate-limit. The model can time out, or return something useless.
If the page waits for the analysis, then an LLM outage becomes a paging outage. That is a strictly worse system than the one with no AI in it at all.
So the notification goes out first, carrying what is already known: what fired, when, which service, who is on call. The enrichment is a second, independent step that edits the message in place once it has something to add. If it never has anything to add, the human still got paged within seconds and has everything the raw alert would have given them.
Stated generally: an enhancement must not be able to break the thing it enhances. If it can, it is not an enhancement, it is a dependency you did not mean to take.
What I would tell someone building something like this
Work out your response budget before your architecture. The gap between what the caller will wait for and what the job actually takes is the shape of the whole service. Here it forced a queue, a worker, an incident table and a state machine, and none of those were choices so much as consequences.
Take your idempotency key from upstream. They have already solved the identity problem for their own retries. Reusing their answer is free; inventing your own costs you a cache and a TTL you will keep guessing at.
Freeze time-varying decisions at the moment you act on them. Store the assignee, not the rule for finding the assignee.
Escalation timers have to be durable. A timer in process memory is a promise you break every time you deploy. If the escalation matters, it lives in the datastore alongside the incident, and something polls it.
Make the best part optional. The AI is what makes the first message useful. It is not allowed to be what makes the first message late.