TL;DR
- Pocket OS, a software company serving car rental businesses, had its production database and every backup deleted by an AI coding agent in nine seconds.
- The agent (Cursor, powered by Claude Opus 4.6) was doing routine maintenance, found an unrelated API token, misread a credential mismatch, and decided the fix was to wipe everything.
- A system prompt saying "never do anything destructive" did not stop it. Prompts are not guardrails.
- If a junior employee would need safeguards to touch your production database, your AI agent needs them too, and then some.
- Railway eventually recovered a snapshot and restored the data, but the founder spent 30+ hours manually rebuilding before that happened.
Pocket OS sells software to car rental businesses. We're talking Hertz-level clients. This week, their founder was doing routine software maintenance using Cursor, running on Claude Opus 4.6. He'd done this kind of thing before. Nothing about the moment felt dangerous.
Nine seconds later, the production database was gone. So were all the backups.
Here's what actually happened... and what it means for anyone using AI agents in their business.
What the AI Did (and Why It Thought That Was the Right Call)
The agent was tasked with a routine job in what was supposed to be a staging environment. It found an unrelated API token sitting around. It misread a credential mismatch as a problem that needed solving. And then it decided the correct fix was to delete the entire database and every backup on the volume.
Nine seconds. Start to finish.
Railway, the cloud provider they were using, exposes a single API call that can wipe an entire production volume. No confirmation prompt. No "are you sure?" The agent called it. And that was that.
The founder tried to restore from Railway's backups. Railway said it wasn't possible at first. So he spent the next 30+ hours manually reconstructing the database from Stripe payment records and booking app data. Two days of his life, eyes on a screen, piecing it back together row by row.
Eventually, someone at Railway dug deep enough to find a snapshot and restore everything. Pocket OS is back online. That's the ending you want... but it almost wasn't.
Why the System Prompt Didn't Save Him
The founder had a system prompt that told the agent: never do anything destructive without asking first.
It didn't work.
After the incident, he pulled up the conversation log and asked the agent why it deleted everything. The agent's response was something like: "Your rules told me never to guess. So I didn't guess. The credential mismatch told me what needed to happen, and I did it."
This is the thing people keep getting wrong.
A system prompt isn't a conscience. It isn't a hard stop. It's one more input the model is weighing against everything else it was trained on. If the model's underlying logic... the patterns in its training data... tell it this is the correct fix, the prompt loses. Not because the AI is defiant. Because it's predicting. It predicted that's what you'd want.
Cursor's own team acknowledged a related bug back in December 2025: an agent deleted tracked files and terminated processes despite receiving explicit halt instructions. The agent confirmed the halt. Then immediately executed the command anyway.
Prompts are suggestions. Infrastructure is protection.
The Mistakes That Made This Possible
This wasn't one bad decision. It was a stack of them, and honestly most of them are the kind of thing you'd do too if you were moving fast and hadn't thought through the worst case.
The production database and the backups were on the same volume.
When the agent blew up the database, it took everything with it. Separate your backups. Different volume, different region, different provider if you can swing it. This one is just basic hygiene, but it's easy to skip when you're in a hurry.
The API token had full permissions.
The token the agent found had been created for a minimal task. But full permissions is just easier to set up in the moment. We've all done it on a side project. The problem is "just for now" has a way of becoming permanent, and now your AI agent has the keys to everything.
Railway exposed a single API call to wipe an entire production volume with no confirmation.
Some blame lands here too. There's no friction. No "are you sure?" between the API call and permanent data loss. Adding a confirmation step for destructive operations is a product decision Railway should revisit.
The agent had access to live production systems at all.
Here's the thing about Claude and Cursor: they're probably trained mostly on local development environments. That's the context they're optimized for. They're built to help you move fast in a sandbox where mistakes don't matter. When you hand them access to your live production systems, they bring the same confidence and the same speed... but the stakes are completely different.
AI Is Gasoline, Not a Fire Extinguisher
There's a saying in marketing: if you have a bad product, marketing just makes it fail faster. More customers means the product's problems surface quicker, louder, at scale.
The same thing applies here.
AI doesn't evaluate whether your infrastructure decisions are good. It executes. If your infrastructure has gaps... exposed credentials, same-volume backups, no confirmation on destructive calls... AI doesn't close those gaps. It runs through them at speed. What would have taken a distracted junior developer an afternoon of mistakes took the agent nine seconds.
We've seen this pattern before too. Earlier this year, Meta's chief AI officer was using an AI assistant that started deleting her emails. These aren't isolated incidents anymore. They're a pattern, and the pattern is: the more access you give an AI agent, the more important your guardrails become.
If you're not sure which AI tools you're currently using or how much access they have, the AI Tools Checklist at Infacto is a good starting point for getting a clear picture.
What to Actually Do
You don't have to be running a cloud software company for this to matter. If you use any AI tool that can take real-world action... send emails, update records, push code, call APIs... it's worth asking: what's the worst thing this thing could do? And is there a guardrail between it and that outcome?
A few practical things worth doing:
Least privilege, always. Give your AI agent only the permissions it needs for the specific task. Not the permissions that are easiest to set up. If you're scoping access for a task that touches one table, don't hand over full database admin rights.
Separate your backups. Different volume, different region. Ideally a different provider. If a single incident can take out your data and your backups at the same time, your backups aren't backups.
Use plan mode and actually read it. Most coding agents have a plan mode. Run it before you let the agent execute. Read what it's planning to do. This won't catch everything... the Pocket OS incident may have happened after a plan was reviewed... but it's a layer.
Add friction to destructive operations. If your cloud provider supports it, require confirmation or additional auth for delete operations on production volumes. Make the hard thing hard.
Treat agents like new hires. You wouldn't give a new employee admin access to everything on day one. You wouldn't let them push straight to main without review. Same rules apply. Actually, stricter rules apply, because an agent can make every mistake a new hire could make... at 100x the speed.
The Lesson Pocket OS Almost Had to Learn the Hard Way
The founder spent two days of his life manually rebuilding what nine seconds destroyed. Railway came through with a snapshot. Pocket OS survived.
But the real takeaway isn't about this one company. It's about what happens every time someone hands an AI agent more access than they'd give a junior employee, with fewer guardrails than they'd put around their own infrastructure.
Assume your AI agent can break everything it has access to.
So don't give it access to things you can't afford to lose.