Part 1 of 3: Decision, Authority and Accountability in AI
Updated: Sep 8
Like most professionals in technology, I often receive the same questions about Artificial Intelligence. What will AI replace? What will it change? Will it take people's jobs? How much autonomy will we eventually give it?
Initially, my response was that I hadn't seen AI replacing people. Instead, I observed AI augmenting them. It helps individuals analyse information, search, correlate events, generate ideas, and work considerably faster. However, I now realise there is a significant issue with that perspective.
AI is already making decisions for us.
Cybersecurity serves as a perfect example. Security platforms make countless automated decisions daily. They assess whether an email is malicious, if a transaction or login appears suspicious, whether an endpoint should be isolated, or if particular behaviour signifies a threat. Increasingly, these decisions lead to automated actions, often without human review.
So, perhaps the more pertinent question isn't whether we will allow AI to make decisions. We already have. The better question is: Which decisions are we prepared to give it — and who ultimately makes the final call?
AI vs NI: Understanding the Distinction
We all know what AI stands for: Artificial Intelligence. However, we must also consider its counterpart: NI — Natural Intelligence. By Natural Intelligence, I refer not just to the processing power of the human brain but to a broader concept. It encompasses knowledge, experience, context, judgement, intent, values, and accountability.
It also includes something humans use constantly but rarely think about: "I know what you mean." This understanding is crucial.
What I Said vs What I Meant: The Importance of Context
Consider a simple data-protection objective. Imagine instructing an AI: "Make sure none of our sensitive data ever leaves the organisation." Taken literally, there are effective ways to achieve this. You could block every external email containing sensitive information, disable file sharing, prevent uploads to external services, and more.
Congratulations. Data loss problem solved.
However, there's a significant issue. The business may no longer function. Organisations don't just need to protect data; they also need to use it. For instance, a finance team may need to send financial information to an auditor. HR may need to exchange employee information with a benefits provider.
The same piece of information leaving the same organisation could represent either a legitimate business process or a serious data breach. The data alone doesn't tell you which. Context does.
When I say: "Don't allow sensitive information to leave the organisation," what I really mean is: "Protect sensitive information from leaving the organisation inappropriately while allowing authorised business processes to continue." This includes considering who is sending it, where it's going, why it's being sent, what controls are in place, and the associated risk.
This is a considerably harder problem and highlights the importance of understanding intent.
To Be Fair, Humans Get This Wrong Too
It would be easy to conclude that humans understand intent while machines do not. However, that's not entirely true. Anyone who has worked in a Security Operations Centre (SOC) or with Data Loss Prevention (DLP) knows that humans can also misinterpret context. An analyst can investigate an incident and reach the wrong conclusion. A poorly designed DLP policy can generate thousands of false positives.
AI didn't create these problems, and it may eventually outperform humans in many areas. The difference lies in speed and scale. A human analyst misunderstanding an objective might make several poor decisions before someone notices. An autonomous AI could potentially make thousands.
The characteristics that make AI valuable — speed, consistency, and scale — become risks when the objective, context, or boundaries are incorrect.
Who Actually Made the Decision?
Imagine an AI-enabled SOC platform analysing thousands of events and determining that a user's account has likely been compromised. It correlates an unusual login, suspicious endpoint activity, access to sensitive files, and abnormal data transfer. It concludes: High-confidence account compromise. Disable the account and isolate the endpoint.
An analyst reviews the recommendation and clicks Approve. Who made the decision? Technically, the human did. But consider what happened. The AI collected the information, correlated the events, identified the pattern, assessed the risk, and recommended the response. The human may have contributed little more than approval.
Now, imagine the same system is configured to automatically disable the account and isolate the endpoint without waiting for approval. The human didn't make that decision anymore. The AI did.
Systems capable of automated response are not a distant theoretical future. We already allow security technology to act automatically under defined circumstances because sometimes waiting for a person creates more risk than acting immediately. So, who made the final call? Perhaps the answer is that a human made a different call. A human decided: "Under these circumstances, this system has authority to act." This distinction is extremely important.
Intelligence, Authority, and Accountability
Historically, intelligence, authority, and accountability resided in roughly the same place. A person considered the information, made the decision, and was responsible for the outcome. AI begins to separate these elements. The intelligence may increasingly come from the machine. The authority may have been delegated to the machine by a human. But accountability still ultimately resides somewhere in the human organisation.
Nobody is inviting the AI into the incident review the following morning and asking: "Why did you make that decision?" Someone still has to answer for it. This is where Natural Intelligence remains critically important.
The Illusion of the "Human in the Loop"
One phrase frequently used when discussing autonomous AI is: "There will always be a human in the loop." This sounds reassuring, but it may not address the problem effectively. A modern SOC can process enormous volumes of telemetry. Now, imagine an AI system analysing that environment and making tens of thousands of decisions every hour.
A human supervisor technically has the authority to override any of them. But perhaps that person reviews only a fraction of the decisions. Is that human truly making the final call? I would argue they are not. They are supervising a machine that is making the calls.
Human oversight and human decision-making are not the same. Our progression increasingly looks like this: Human investigates → AI assists → AI recommends → Human approves → AI decides within boundaries → Human supervises → Human reviews exceptions. Eventually, the human decision may no longer concern the individual security incident at all. The human decision becomes: "We authorise this AI to make this class of decision within these boundaries." This represents a fundamentally different kind of final call.
But What If AI Is Better Than Us?
There is another uncomfortable problem with insisting that humans must always make the decision. Humans aren't perfect. Analysts get tired, have biases, miss information, and become overwhelmed by alerts. Two experienced analysts can look at the same incident and reach different conclusions. AI doesn't automatically have those same limitations.
Imagine an AI system that correctly identifies a particular type of security incident 99.7% of the time. Experienced analysts achieve a success rate of 94%. What exactly is the argument for requiring human approval of every action? Because we're human? At some point, that becomes difficult to defend.
In cybersecurity, requiring human intervention can create risk. If an attacker is actively exfiltrating sensitive data, waiting fifteen minutes for an analyst to review the incident may be far more dangerous than allowing a highly reliable system to stop it automatically.
This produces an interesting reversal. Today, after an incident, we might ask: "Why did you allow the AI to take that action?" Perhaps one day the question will instead be: "The system correctly identified the threat. Why did the human override it?"
Not Every Security Decision Has a Correct Answer
Cybersecurity isn't simply about identifying threats; it's about managing risk. A Chief Information Security Officer (CISO) wants security, but the business wants productivity. Users want flexibility, executives want growth, and customers want privacy. Technology teams desire manageable systems. The organisation has limited money and resources.
These objectives often conflict. The most secure decision isn't necessarily the best business decision. The most restrictive DLP policy isn't always the best data-protection strategy. Sometimes, someone must decide: "We understand the risk, but we're prepared to accept it because the business benefit justifies it."
Can AI analyse that decision? Absolutely. Can it quantify the risk better than a human? Quite possibly. But should AI ultimately decide what level of risk an organisation is prepared to accept? This isn't purely an intelligence question. It involves judgement, priorities, values, and responsibility.
AI May Replace Tasks Before It Replaces Responsibility
This changes how I view AI and jobs. In cybersecurity, AI can already perform tasks that previously required significant human effort. It can correlate events, summarise incidents, analyse behaviour, recommend responses, write queries, and identify patterns across vast datasets.
One analyst equipped with capable AI may eventually perform work that previously required several people. So, perhaps AI is replacing tasks faster than it is replacing responsibility. But this raises another question. What happens as we become comfortable giving it more responsibility as well? If the AI can detect the incident, investigate it, determine the appropriate response, and execute that response more accurately than the analyst, what exactly is left for the analyst to do?
So Who Makes the Final Call?
After contemplating all of this, my answer — for now — is NI. Not because I believe Natural Intelligence will always be smarter than Artificial Intelligence. It probably won't be. And not because AI shouldn't be trusted to make decisions. It already is — and increasingly it should be.
AI will make more individual decisions because it will often be faster, more consistent, and eventually, in some areas, demonstrably better than we are. However, Natural Intelligence must still decide which decisions we are prepared to delegate, what boundaries AI operates within, what level of risk we are prepared to accept, and who owns the consequences when things go wrong.
So, perhaps there is an important distinction between the call and the final call. AI can make the call. NI should decide when AI is allowed to make it. At least for now.
Because there is an uncomfortable paradox at the end of this argument. What happens when NI concludes that AI is simply better at making decisions? What if our final call is: "You make the final call from now on."
If we progressively hand over detection, investigation, analysis, decision-making, and response to AI because it performs all these tasks better, faster, and more consistently, another uncomfortable question follows: At what point have we also handed over the job?
That's the question I'll explore in Part 2.
And then there is another possibility altogether. Perhaps framing the future as AI vs NI is itself wrong. Perhaps the future is AI + NI — humans becoming more capable by combining Natural Intelligence with Artificial Intelligence rather than attempting to compete against it. That's Part 3.
Coming next
Part 2 — If AI Does the Work, What Do We Do?
When augmentation becomes autonomy, what happens to the human job?
Part 3 — AI + NI: Is the Future Integration Rather Than Competition?
Perhaps the future isn't about which intelligence wins, but what happens when we combine them.
For now, though, my answer remains: AI can make the call. NI should decide when it is allowed to. At least for now.
Roman Kreychman
Ready to reduce data risk?
https://www.shadowlens.com.au | sales@shadowlens.com.au
