Introduction

When we look back on system incident response, certain phrases come up again and again.

“The judgment was too slow.”
“We should have decided sooner.”
“No, at that point we didn’t have enough information.”

Each of them sounds correct.

In practice, incident response spends a great deal of time on decision-making activity. Investigation and discussion all come down to deciding “what should we do.” Once the response is decided, the actual work often finishes relatively smoothly.

What kinds of decisions are these? For example: whether to stop the service or keep it partially running. Whether to roll back or not. Whether to notify users, or to hold off on notification to prevent confusion.

Incident response is often pictured as technical activity, such as log analysis and root cause investigation. Yet when we review real situations, what determines success or failure is frequently not technical skill itself, but decision-making.

And the tricky part is that at the moment of decision during an incident, no “correct answer” exists.

Looking back later, it is easy to say “that judgment was wrong.” But at that point, no one knows the future.

In this piece, we would like to explore what a “good decision” in incident response really means.

Is a Good Decision About “Producing the Right Result”?

What is a good decision?

Before answering, there is something worth keeping in mind: a good decision and a good result are not the same thing.

Suppose an incident occurs on an e-commerce site. The impact on sales is large, and user inquiries are increasing. After discussion, the technical team judges that “the most recent release is the likely cause” and performs a rollback.

As it turns out, the cause was elsewhere, and the rollback was meaningless.

Was this a bad decision?

Not necessarily. Given the information available at that point, the rollback was the most reasonable choice.

The reverse can also happen. Someone responds on a hunch, and it happens to work out. The result was good. Still, it is hard to call that a good decision.

In other words, when we evaluate a decision, we may need to look not at the result, but at the information available at that moment and the reasoning process.

Incident Response Is a “Battle With Uncertainty”

Why is decision-making difficult?

The reason is simple: information is often lacking.

Right after an incident begins, it is normal:

not to know the cause
not to know the scope of impact
not to know the recovery outlook
not to know the user impact
And yet a decision still has to be made. This is a highly unusual environment.

For a management decision, there is time to conduct market research and prepare analysis. For project management, there is time to perform risk analysis.

In incident response, that time does not exist. Incident response can be described as decision-making activity under a shortage of information.

The Trade-off Between Speed and Accuracy
This is where the trade-off between speed and accuracy always appears.

Consider the moment of deciding whether to stop a service. Rush the judgment, and you may make the wrong call to stop. Gather information carefully, and the damage may spread in the meantime. There is risk on both sides.

Anyone who has worked in operations will recognize this. The system is unstable, but the cause is still unknown. In that state, you have to decide whether to stop or to continue, and either choice can be criticized.

The important fact here is that speed and accuracy sometimes cannot both be achieved. Waiting for a fully accurate judgment takes time. Deciding instantly leaves room for error.

In that sense, the demand to “make a fast and correct judgment” is contradictory.

A Good Decision Uses “Necessary and Sufficient Information”

So what can be done?

A useful reference here is the service perspective. What users want to know is not necessarily the technical root cause.

Consider a train delay. When a delay occurs, passengers want to know:

whether it affects them
whether they should wait
whether they should take an alternate route
Would they want the details: that the cause was an electrical signal failure, that the signal sits between crossing X and crossing Y, that part Z is needed for the repair? These are not what passengers want to know.

Incident response is the same. What a decision requires is not “all information,” but “the information needed to decide.”

For example, if you know:

the scope of impact
the potential for damage to spread
whether a workaround exists
the recovery outlook
the business impact
a reasonable judgment becomes possible.

In other words, a good decision means deciding at the stage where necessary and sufficient information has come together.

The “Lack of Options” That Often Causes Failure

One pattern of failure in incident response is when the options are not visible.

Take the case of “stop the service or not.” In reality, there are several options:

stop only some functions
limit traffic
switch to alternative operation
When the situation is framed as a binary choice, the discussion tends to stall.

To make a good decision, the first step is to increase the options, then compare the merits and risks of each.

In other words, a good decision is not only about searching for the correct answer, but also about creating better options.

These options, the actions taken during incident response, can be prepared in advance as part of daily practice. Listing them only after an incident begins already puts you a step behind.

A Template for Decisions That Do Not Fail

So what kind of template is useful in practice? For incident response, we find the following four steps effective.

Step 1: Gather facts
Start with facts, not guesses.

What is happening. What is not happening. How far you have been able to confirm. Organize these.

Step 2: Organize decision information
Next, consider what information is needed in order to decide.

The key here is the purpose of information gathering. You do not collect information vaguely; you collect it in order to decide. Because incident response is a race against time, there is no room to gather meaningless information.

Step 3: Create options
Only now do you lay out the options. For example: stop the service, continue operation, partial stop, alternative operation, rollback.

Then compare the candidates.

Step 4: Evaluate against criteria
Finally, the decision criteria. For example:

minimize customer impact
limit revenue impact
protect data integrity
err on the safe side

When something like customer impact or revenue can be expressed quantitatively, setting a threshold in advance speeds things up. For example, “when it affects 10 percent of customers.” Operating that 10 percent line strictly is often difficult, but having a rough guide (whether the line is 1 percent, 10 percent, or 50 percent) makes the judgment easier.

When this is vague, the discussion never ends. When the criteria are shared, decision-making becomes surprisingly fast.

Why Are Veterans So Fast to Judge?

Watching a veteran of incident response, you may wonder “how can they decide so quickly?” This is not just intuition.

The more experienced a person is, the more they know:

what to look at
what to ignore
what options exist
what to prioritize
In other words, they have internalized a decision template. That is why they are fast, and why they do not hesitate. And the important point is that this template is not an individual talent; it can be shared across an organization.

What information they deliberately obtain, from what angle they evaluate it, and what action they choose: all of this can be written down and passed on as know-how.

Once it becomes habitual, this decision-making happens unconsciously, so it is hard to draw out by simply asking. But using this template as a basis for the conversation should help put it into words.

A Good Decision Is Not About “Predicting the Future”

Finally, there is one thing we most want to convey about decision-making in incident response.

It is that a good decision is not about predicting the future.

In incident response, the future is unknown. However skilled the engineer, no one can predict with 100 percent accuracy.

For that very reason, a good decision is not something evaluated in hindsight. It is gathering the information available at that moment, clarifying the decision criteria, comparing several options, and then deciding.

If your organization ever holds a review after an incident, try looking back at “how the judgment was made.” There you will find many hints your organization can learn from.

The maturity of incident response cannot be measured by the number of incidents alone. It lies in how high the quality of decision-making can be under uncertain conditions.

This is where real strength lies. And that strength is not a special talent; it can be cultivated through daily review and design.

The next time an incident occurs, instead of trying to “find the true correct answer,” try thinking “let us make a better judgment.”

That shift in perspective can be the first step in turning incident response from an individual skill into an organizational capability.