Introduction

Of all the incidents that occur day to day, a full system outage has an exceptionally large impact on service. That makes it one of the incidents we most need to prevent in the first place, and, when it does happen, to keep from happening again.

Suppose the following incident occurs. The number of database connections in production hits its limit, the system goes down, and the service becomes unavailable. Investigation reveals that a particular process was not releasing its connections. The team fixes the program, verifies the fix in testing, and releases it to production. “That should keep it from happening again.” Relieved, they go back to running the system as usual.

A few months later, a similar incident occurs. This time it involves a different program, a different person in charge, and a different time of day. Yet when you trace how the incident unfolded, something about it resembles the last one.

Why?

Perhaps it is because the incident was viewed only as a technical problem.

Why a technically correct fix can still fall short

Let’s go back to the earlier example. Fixing the bug that failed to release connections is a necessary measure in itself. If the technical cause is clear, responding with a technical fix is only natural.

But if the analysis ends there, one question remains unanswered: why did this problem make it all the way into production? The situation might have looked something like this:

The performance test environment did not sufficiently match the conditions in production
The pre-release checklist did not include database connection count as a perspective at all
Monitoring that could have detected the anomaly after release was also insufficient

In that case, fixing the program solves only part of the problem. Behind the technical issue of “connections not being released,” there may also be a testing process issue, a release process issue, and a monitoring issue.

Dig further, and you may arrive at organizational questions. Why wasn’t that test carried out? Why wasn’t the need for monitoring shared? Why couldn’t development and operations catch this together?

In other words, a single incident can contain several overlapping problems across technology, process, and organization. In my view, this is far from unusual.

Looking at incidents through three layers: technology, process, and organization

When thinking about the causes of an incident, it is important to look at the problem from multiple angles. If you look only at the technical side, as in the opening example, you will miss the other contributing factors.

A useful reference here is the Swiss cheese model, a concept widely used in safety engineering. Even if one slice of cheese has a hole, as long as several more slices are stacked behind it, nothing passes all the way through to cause an accident. Accidents happen when the holes in multiple layers happen to line up. System incidents work the same way. Even if there is a hole in the technical layer of defense, the outcome should not be serious as long as the other layers, process and organization, are solid.

At a minimum, splitting the analysis into the following three perspectives makes it easier to see where the problem lies.

The first is technology. These are problems that exist in the system itself: program bugs, misconfigurations, insufficient performance, network failures, data inconsistencies, and so on. Here, the response centers on technical measures such as fixing the program, changing settings, reviewing the architecture, or adding monitoring.

The second is process. Testing was not thorough enough, the release procedure lacked a checkpoint, there were no escalation criteria for when an incident occurs, information about similar past incidents was not reused. These are problems that are hard to solve through individual technical skill alone. They call for revising procedures, adding checkpoints, and clarifying who is responsible for what.

The third is organization. Collaboration between development and operations is weak, system staff and business units do not share the same priorities, and decisions end up concentrated on specific individuals whenever something goes wrong. When you look at the challenges surrounding incident response, problems involving people and organizations, such as poor cross-team collaboration, skill gaps among team members, and weak working relationships, are endless.

What matters here is not to treat these three as entirely separate. It is closer to reality to think of technology, process, and organization as connected.

Why “operator error” should not be the end of the story

Another thing that often happens in incident reviews is that the discussion fixates on who made the mistake.

For example, suppose a configuration change in production was made incorrectly and the service went down. Let’s say the conclusion was that “the person in charge failed to double-check.”

It may well be true that the person failed to check. But that conclusion alone may not prevent recurrence.

Why did they fail to check? Perhaps there was no checklist. Perhaps there was a checklist, but it was not clear who was supposed to verify it. Or perhaps the time allotted for the work was so short that there was no room to check. Beyond that, the fact that a single person could make changes to production on their own may itself be the problem.

Thinking it through this way, it becomes clear that “raising the operator’s attentiveness” is not a sufficient countermeasure.

This connects to the idea of Just Culture from the field of safety culture (a culture that seeks out flaws in the system rather than punishing individuals). Of course, there are cases where individual care and training are necessary. But if “please be careful” and “let’s pay more attention” were enough to prevent recurrence, the same incidents would have disappeared from the world much sooner.

Humans will always make mistakes. That is exactly why we need to design mechanisms that keep mistakes from turning into serious incidents.

Misplace the problem, and the countermeasure misses too

This is where identifying where the problem actually lies becomes important.

Even for the same event, a “configuration change error,” there is more than one possible countermeasure. Correcting the configuration value is a technical measure. Adding verification steps before and after the change is a process measure. Making sure the change is not left to one person and is checked by someone else is a measure at the level of operational structure. And if you set up a forum where development, operations, and business units review the business impact of changes in advance, that would be an organizational measure.

How far you take your view of the problem changes what you do about it. Put the other way, if you misplace the problem, the countermeasure will miss as well. If you think “it’s a technical issue, so let’s fix the system” when the real problem lies in the release process, the same kind of incident may keep occurring no matter how many technical fixes you make.

That is why, in incident reviews, it is important not to settle on a single cause from the start. Instead of asking only “why did the incident happen?”, try widening the questions: “Why couldn’t we prevent it?” “Why couldn’t we detect it sooner?” “Why didn’t the information reach the right people?” Doing so can reveal problems with process and organization that sit apart from the technical cause.

Leaning too hard on “organizational problems” is also risky

Having read this far, you may feel that the essential takeaway is: “Most system incidents ultimately stem from organizational problems, so we should just improve the organization.” After all, if you abstract the problem and keep running a why-why analysis, you will usually end up at organization or culture. But this, too, is a little dangerous.

If the technical cause is clear and all anyone discusses is organizational theory, the necessary technical fixes will be delayed. If an incident was caused by insufficient database capacity, the first step is to secure that capacity. Only then do you widen the analysis: why couldn’t the capacity shortage be detected in advance, were the capacity management rules appropriate, were the people responsible clearly defined?

What matters is not a binary choice between technology and organization. It is about addressing technical causes with technical measures while digging into the underlying process and organizational issues as needed. In other words, it is about looking at the problem one layer deeper at a time. Even if an incident is technically minor, if it stems from process or organization, it may one day lead to a much larger one.

Next time an incident occurs, stop and ask again

Recurrence prevention does not end once you have found the cause and written down a countermeasure. However good a countermeasure looks on paper, it will not deliver the expected effect if it takes too long to implement, is never carried out due to poor follow-up, or becomes a formality as people come and go. Recurrence prevention has to include correctly identifying where the problem lies, turning it into something that can actually be implemented, and confirming that the organization’s behavior has actually changed.

The next time an incident occurs, try pausing partway through the root cause analysis. Rather than rushing to conclude “this is a technical problem,” ask yourself again: “Was there a problem with the process?” “Was there a problem with how we acted as an organization?”

What used to look like “another similar incident” may then be reframed as “a mechanism we have not yet improved, now brought to light.” Starting from that question is likely to be the first step not only toward fixing the incident in front of you, but toward building an organization that is resilient to incidents.