9 services went down at once, and my status page called it 9 outages
On October 7, 9 services stopped responding together for 80 minutes. Not a single down mail went out, and only 9 recovery mails arrived. I changed the status page to group failures in the same time window into a common outage and show them separately.
#Monitoring #Incident response #Operations #Node.js #React
On the evening of October 7 I opened my inbox and found 9 mails from the monitoring server. All of them were recovery mails. Couple App recovered, Blog recovered, Travel API recovered, and so on, 9 services recovering one after another within 2 minutes. But there wasn't a single down mail. In my 100 days of monitoring retrospective I wrote down "if everything fails together, it's not a service outage" as something to fix, and it actually happened three weeks later. In this post I'll go over what happened that day and how I changed the status page to show it. The record from that day I started with the alert history table. triggered_at rule_name message notified 2026-10-07 16:40:52 Couple App API Down (2 consecutive failures) 0 2026-10-07 16:43:51 Game Down (2 consecutive failures) 0 2026-10-07 16:43:51 Travel API Down (2 consecutive failures) 0 2026-10-07 16:44:25 Coin Tracker Down (2 consecutive failures) 0 2026-10-07 16:45:17 Ieum Down (2 consecutive failures) 0 2026-10-07 16:47:56 Blog Down (2 consecutive failures) 0 2026-10-07 16:48:00 Monitoring Down (2 consecutive failures) 0 2026-10-07 16:48:01 taro Down (2 consecutive failures) 0 2026-10-07 16:49:42 Dapjeongneo(fishing) Down (2 consecutive failures) 0 2026-10-07 17:58:14 Couple App API Recovered (downtime about 82 min) 1 ... 2026-10-07 18:01:15 Dapjeongneo(fishing) Recovered (downtime about 84 min) 1 9 services got marked down one by one within 9 minutes, and about 80 minutes later they all recovered within 3 minutes. notified is 0 for every down row and 1 for every recovery row. That means not a single down mail went out. The reason was in the error log. [Scheduler] healthcheck Dapjeongneo(fishing): connect ETIMEDOUT [Health] down mail error: connect ENETUNREACH (mail server):587 ENETUNREACH is the error you get when the network itself can't be reached. The server's way out was cut off. The health check calls each service through its real domain, so the request has to go out and come back in, and since that didn't work everything timed out. The down mails had to go out the same way, so they couldn't go out either. Once the way opened up again, only the recovery mails went out all at once. The services were up and fine the whole time. Looking at when each process last started with pm2 jlist , the blog hadn't restarted since October 1, the couple app since September 22, and taro since August 18. It wasn't 9 apps dying at once. The server's network was cut off. The status page showed this as 9 incidents On the public status page, hovering over the October 7 bar showed this. Not just the blog, the bars for all 9 services were orange, and each one had its own "Response timeout" attached. Anyone looking at it would read it as 9 services each having their own problem that day. Even I opened each service and looked at its logs one by one at first. In reality there was one cause, but the screen made it look like there were 9. It was in the past records too I wondered if this was a one-time thing, so I lined up the 171 failure records left in chronological order and grouped failures that came within 10 minutes of each other, regardless of service. Oct 7 16:36 ~ 17:55 9 services 134 failures Response timeout Sep 24 04:07 3 services 3 failures DNS lookup failed Aug 14 02:04 ~ 02:08 7 services 8 failures Response timeout Aug 8 00:47 4 services 4 failures Response timeout Jun 20 18:04 3 services 3 failures Response timeout There were five times when three or more services failed together. The August 14 one I had already seen in the 100 day retrospective, but this time it lasted much longer. The rest of the groups were mostly a single service failing once or twice on its own. How to group them I kept the rule simple. Line up failures in time order regardless of service, and if a failure is within 10 minutes of the previous one, it's in the same group. If that group has three or more services in it, it's a common outage. const COMMON_GAP_SEC = 10 * 60; const COMMON_MIN_SERVICES = 3; function markCommonOutages(failures, nameById) { const sorted = [...failures].sort((a, b) = a.ts - b.ts); const clusters = []; for (const f of sorted) { const last = clusters[clusters.length - 1]; if (last f.ts - last.endTs = COMMON_GAP_SEC) { last.rows.push(f); last.endTs = f.ts; } else { clusters.push({ rows: [f], endTs: f.ts }); } } const outages = []; for (const c of clusters) { const ids = [...new Set(c.rows.map(r = r.endpoint_id))]; if (ids.length COMMON_MIN_SERVICES) continue; for (const r of c.rows) r.common = true; // ... collect start, end, service count and the most common failure reason into outages } return outages.reverse(); } I went with 10 minutes because of the check intervals. Each service is checked every 3 or 5 minutes, so even when they fail for the same reason, the recorded times are off by a few minutes. On October 7 the first and last down marks were 9 minutes apart. Too short and one incident gets split into several, too long and unrelated failures get attached. I went with three because two can overlap by chance. The records had three groups where only two services failed together, and two of those were just single timeouts that happened to land at the same time. Three or more failing together in the same window is hard to call a coincidence. I didn't touch uptime There was one thing I thought about here. Since I grouped it as a common outage, should I take it out of each service's uptime? It wasn't the service's fault. But it's true that visitors couldn't get into the blog for those 80 minutes. Whether the cause was the app or the network, from the user's side it was the same site that wouldn't open. If I start subtracting from the uptime on a public status page depending on the cause, that's no different from fudging the numbers to look better. So I left the uptime and bar colors as they were and only added a note that this was a common outage . The new screen I added a separate common outage section above the service list. It shows in one line when it happened, for what reason, and how many services failed together. It only shows the latest 5, and if there are no common outages the section doesn't show up at all. The tooltip on each service's bar also gets a mark if that outage was part of a common outage. So they don't get mixed up with individual outages, failures that belong to a common outage and ones that don't are grouped separately even on the same day. Looking at 90 days again after the change, 23 of the 39 per-service outages belonged to 4 common outages. More than half of the outages on the screen were actually 4 events. The uptime of the 10 services is the same as before down to the third decimal place. What I haven't fixed yet All I fixed this time was how it's shown. The real problem is still there. When the server's network goes down, the alert mails have to go out the same way, so they can't. This time too, I knew nothing for the 80 minutes of the outage and only found out from 9 recovery mails after it was all over. This can't be solved from inside the server. Something outside has to be watching my server. The most realistic way is to send a signal to an external heartbeat service every minute and have it alert me when the signal stops. I wrote this down in the 100 day retrospective too, and this incident definitely pushed it up the list. The 9 recovery mails can also be combined into one. For recoveries in a window judged to be a common outage, one mail saying "9 services recovered, downtime about 80 minutes" is enough. I plan to do this next time I work on the alerts. If your own monitoring sends alerts per service, I recommend grouping your records once to see whether several services have ever failed at the same time. What looked like a service problem might have been a server or network problem.