Every on-call engineer knows the ritual. A page lands in Slack: the 5xx rate on a service is above the threshold. The message tells you that something is wrong and nothing about what. So you open Kibana or Loki in another tab, set the time window to the last few minutes, filter by the service, and start reading. In my team this took two or three minutes on every alert, and these are the worst minutes of the incident, because nothing is being diagnosed yet.
The obvious request is to put the last few error lines into the alert message itself. I tried to do this inside Alertmanager, and it cannot be done there. It helps to understand why before building anything around it.
Why Alertmanager Cannot Do It
Alertmanager renders notifications through Go templates, and a template can only use what is already attached to the alert: the labels and the annotations which Prometheus put there when the rule fired. There is no mechanism which would call Elasticsearch, Loki or any other HTTP API at the moment a notification is sent. Annotations are rendered from the alerting rule at evaluation time, inside Prometheus, and Prometheus does not talk to log stores either. Fetching data at send time is simply not a capability of the design.
So the logs have to come from a separate process which stands next to Alertmanager.
The Sidecar
What runs in production now is a small service next to Alertmanager, and its shape came out of two failed attempts, so I will describe it through them.
My first attempt was a bot which received alerts and posted them to Slack itself, with the logs already attached. It looked good in a demo, but it was a mistake because the bot had become the paging path. Any bug in my code, any timeout of the log store, and nobody gets paged. I threw that version away quickly. Alertmanager has to stay the thing which posts the alert, exactly as before, and whatever adds context has to be allowed to die without anyone noticing. So the enricher hangs off a second receiver: the route which sends the alert to Slack also duplicates it to a webhook, and behind the webhook stands my service. When it is down, the page arrives as it always did, just without the extra.
The second attempt posted the logs as its own message into the channel, right after the alert. Within a day the channel was hard to read: two messages per alert, and during an alert storm they interleaved, so the logs stood three messages away from their page. The fix was to reply into the thread of the message which Alertmanager had already posted. The channel stays as it was, one message per alert, and whoever opens the incident an hour later finds the logs from the exact minute of the page sitting under it.
Finding the Message to Reply To
That thread reply is the one interesting engineering problem in the whole service. The enricher receives the alert from Alertmanager, and it has to find the Slack message which corresponds to this alert, and nobody tells it which one that is. When the webhook fires, the service reads the last couple of minutes of channel history and looks for a message which carries the same alert fingerprint. Our Slack template rendered the fingerprint into the message, so the match was exact; before I added that, I matched on the label set which appears in the message text, and that worked too. Once the message is found, the reply is one API call with thread_ts.
Then production corrected the details. Alertmanager re-sends a firing alert on repeat_interval, and my version at first posted the same log dump under the same message again and again, so now the service remembers which fingerprints it has already enriched. The lookback for matching is limited to about two minutes, since replying to some old message which happens to carry the same labels is worse than staying silent. The query into the log store is built from the labels of the alert (service, namespace, severity) and takes the last few error-level lines, ten or so. I keep it that short on purpose: the reply is a hint for the person, and the full picture is in the log store anyway. One more thing I did not expect: an empty result is also worth posting. When the reply says there were no matching error logs in the last five minutes, the person on call can broaden the investigation to the infrastructure or the alerting rule itself, and this saves the same minutes.
What It Changed
The service is a few hundred lines, and in operation it has been boring, and boring is what I want from anything standing near paging. The visible change for the team: when you open Slack on a page, the first evidence is already under it, and the phase of “which dashboard, which index, which time window” is gone from incidents. Time to diagnosis dropped on every incident which had logs behind it, without any change to the services themselves.
Logs turned out to be only the first thing worth attaching. The same service can reply with anything which Alertmanager cannot know at render time: a runbook link picked by label, the marker of the latest deploy of the affected service (the first question in half of the incidents anyway), or a dashboard link already scoped to the incident window. Alertmanager alerts, and the sidecar carries what the responder reaches for in the first minute.
If you run Prometheus with Slack, the whole thing is one small service plus one extra webhook receiver in the Alertmanager config. Do the idempotency bookkeeping first; it is the first thing which bites.
Daniil Romashov is a Site Reliability / DevOps engineer working on reliability and observability for high-load systems. Open-source work: github.com/youngpabl0.