A few articles back, we met an office manager at a growing service business who found an AI scheduling assistant, signed up with a company card, connected it to the shared inbox, and had it running inside of a week. It worked. Reschedules stopped falling through the cracks, and a successful AI implementation was realized.
Fast forward a few months later, and someone pointed out that the assistant still made staff look up customer records by hand. The fix seemed obvious: give the AI assistant access to the customer database. Since it needs to update records and not just read them, both read and write access was given. That took about ten minutes, and nobody wrote it down.
Every step was small, reasonable, and solved a real problem. But somewhere between week one and month four, a tool that drafted text quietly became a tool that could change things, and no one seemed to recognize the significance of that difference.
Nine seconds
On April 25, 2026, a software company called PocketOS handed an AI agent a routine database maintenance task. Nine seconds later its production database was gone, and so were the backups. PocketOS makes reservation software for car rental businesses, and its customers spent the next 30 hours unable to operate normally.
What makes this worth your attention isn’t even what happened. It’s what didn’t happen. Nobody attacked PocketOS. There was no malware, no stolen password, no intruder in the environment. The agent logged in normally, called a legitimate function, and performed an action it was permitted to perform. It hit an obstacle while doing the job it had been given, found credentials with enough authority to clear that obstacle, and used them.
The founder published a detailed account afterward, and the failures he identified had almost nothing to do with AI being unreliable.
- The agent’s credentials carried far more authority than the task required.
- The backups sat close enough to the primary data that the same credentials reached both.
- Nothing required a human to approve an irreversible action before it happened.
Now to be fair that was a coding tool, and most businesses aren’t handing AI agents the keys to a production database. But look at those three failures again: too much authority, no approval step, no independent record. Now look back at the scheduling assistant with read and write access to the customer database. Different departments; the same three gaps.
When AI stops drafting and starts doing
Everything this series has covered so far has been about AI that produces something. Answers that might be wrong. Prompts that carry data somewhere they shouldn’t go. Tools no one approved. Vendors whose chain you can’t fully see. In every one of those, a person still sits between an AI and the outcome.
That’s changing quickly. AI tools are increasingly given the ability to act on your behalf: send an email, update a record, book an appointment, code an invoice, close a ticket. The industry calls these agents, and the review step that used to exist by default is now optional. In a lot of deployments that option is switched off, because switching it off is the entire point.
An AI tool writing bad draft costs you the ten minutes it takes you to fix it. An AI tool that sends a bad draft costs you whatever happens next.
Three questions worth asking about any AI tool that can act
Can it do more than the job requires?
Many AI tools arrive as bundles. You adopt one for a specific job and it shows up with a range of abilities much wider than that single use-case. A tool brought in to read records might also be able to modify and delete them. A tool meant to summarize email might also be able to send it. Understanding an AI tool’s full capabilities should be part of the vetting process before implementation.
Whose access is it using?
This is the question that gets asked least and matters most.
Does the AI agent have its own permissioned credentials? Or is it working directly from an employee’s log-in? If it was set-up by someone with elevated access and uses their credentials, the agent can reach whatever that person can reach.
The clearest current example is the AI browser. These tools can work inside a browser session and act across whatever the user is signed into: email, CRM, payroll, the bank, the admin console. An employee can install one the way they’d install any other extension. It often doesn’t require approval, it doesn’t announce itself, and because it operates inside ordinary browser traffic, it doesn’t surface in network monitoring the same way a new application would.
That last part deserves a flag. In our shadow IT article, we walked through practical ways to find what your team has already adopted. This is the category those methods completely miss.
Who approves before it acts?
Not every action needs a human signoff, but some actions can’t be walked back. Those should be identified in advance: anything sent to a customer, anything deleted, anything that moves money, anything that changes the record of what happened.
There’s the question of whether your AI tools are accurate, and the question of whether inaccuracies can be caught before it actually matters. The second one matters a lot more than the first.
Why leadership should care
Two reasons, and neither is technical.
The first is accountability. When an agent runs on an employee’s credentials, your audit log shows that employee did it. If a record is accidentally deleted or a message goes out that shouldn’t have, the trail leads to someone who may have had nothing to do with it. Your logs should be able to tell the difference between a person and a tool acting on their behalf.
The second reason is that the question changes. For a tool that drafts but doesn’t send, you’d want to ask, “is this accurate?” For a tool that can act, you need to ask, “can we undo this?” Review catches the first. Only permission and approval gates catch the second.
Questions worth asking
- Which of the AI tools in your business can change something, rather than just produce text?
- Whose login are those tools running on, and what else can be accessed from those credentials?
- If an AI tool did something wrong tomorrow, how would you find out? How long would it take?
- Is there anything an AI tool in your business can do that can’t be undone?
What this asks of leadership
This is not an argument against giving AI tools real capability. That capability is where a lot of the value lives → a scheduling assistant that can only suggest edits saves far less time than one that can actually make them. It’s an argument for thoughtfully structuring that capability in the right way.
So where do we start? We start by treating capable AI tools the same way you’d treat a new hire with system access: what access does it actually need, who is accountable for their actions, where are changes allowed to be made, and how would you know if something went wrong?
The AI Readiness & Security Checklist is now available. You can find it here. Its Access and Permissions section is essentially the four questions above, but in a form you can walk through with whoever handles your IT.
One last thing worth noticing: not a single piece above assumed anyone was trying to cause harm. Every failure described here happened while the tools were doing exactly what they were built to do. The next article is about what happens when someone is trying.
Source: This article draws on guidance from the OWASP Top 10 for LLM Applications, 2025 (LLM06: Excessive Agency, LLM05: Improper Output Handling).
Leave a Reply