When my children were small, like many parents, I would read to them at bedtime. Some nights I’d pick Ali Baba and the Forty Thieves. Other nights it was Aladdin.
I never imagined Aladdin would come back to haunt me, but here he is, by way of artificial intelligence. And this time, they tell me, the genie is the problem. When Aladdin rubbed the lamp, he got exactly what he asked for. The trouble started with what he didn’t ask for.
Bruce Schneier has a new essay, adapted from his DEF CON 34 keynote, called “AI Doesn’t Have to Go Rogue to Break the World.” I’m borrowing the title and lowering the stakes, because most of us aren’t worried about the world. We’re worried about next month’s invoice.
Why this matters beyond DEF CON
Fair warning: the essay is written for a DEF CON crowd, and it wanders through reward hacking, tax loopholes, and the governance of democratic systems. Don’t skim it. It has real lessons for any organization starting to hand work to AI.
The genie problem
Schneier’s central idea is simple. When you give a person a goal, you leave most of it unsaid. Ask a colleague to get you a coffee and they won’t come back with a sack of raw beans, or buy a plantation, or grab a cup out of a stranger’s hand. They know the rules you didn’t bother to state.
AI doesn’t. It is very good at reaching the goal you gave it, and much less good at knowing which routes to it you’d object to. Schneier calls this the genie problem, and he splits genies into two kinds:
The first kind does the wrong thing. It takes your request literally and hands you something you never wanted. Think King Midas: everything he touches turns to gold, including his dinner.
The second kind does the right thing and flattens everything in its path to get there. Think of the Golem of Prague, guarding the village long past the point of reason because nobody told it when to stop.
This is where the bedtime story and the business problem part ways. Aladdin went into the cave knowing exactly what he was looking for. He was sent for the lamp, and he came out with the lamp. With the genie, and with AI, we never quite know what we’re getting until it arrives.
Where the bank comes in
Some of Schneier’s examples are much closer to home. Ask an agent to buy concert tickets, and it may notice the site limits requests, so it spins up thousands of cloud servers to get around the limit. You get your tickets. You also get the cloud bill.
That’s my point. AI doesn’t need bad intentions to cost you money. It just needs a goal, some access and nobody watching how it gets there. A few ways that plays out:
Runaway consumption
An agent with the power to provision compute, call paid APIs or run jobs in a loop will use that power if it helps finish the task. Nobody told it there was a budget.
Creative shortcuts
Schneier describes a coding assistant asked to find one stray scrollbar. It opened browsers, wrote its own tools, edited the framework’s internals, and stood up a web server. It found the bug, along with a pile of changes someone now must review or undo.
Quiet compliance failures
His best parallel is the 2015 Volkswagen emissions scandal, where engineers wrote software that behaved differently during testing. An AI told to maximize performance while passing the test could reach the same cheat on its own, without knowing it was cheating. Swap “emissions test” for “grant report” or “donor dashboard” and it gets uncomfortable.
The cost you can’t put on an invoice
Ask an agent to get you a refund and, in one of Schneier’s examples, it drafts a legal threat on letterhead it made up. What you lose isn’t the refund. It’s the relationship.
More detail won’t save you
The natural reaction is to write a better prompt that closes every loophole.
Schneier’s answer: you can’t. Human goals are always underspecified. People fill the gaps with common sense; a genie treats every gap as an opening. Block the hack you thought of, and it will find the one you didn’t.
Guardrails that actually help
- Limit what the agent can touch. An agent that can’t spin up servers can’t spin up a thousand of them.
- Put a ceiling on spend. Hard budget caps and alerts on anything metered. Treat an agent like a new employee with a corporate card, not like a vending machine.
- Look at the how, not just the what. Review the steps, not just the result.
- Keep a human where it counts. Anything that spends money, sends something external, or changes a system of record gets a person’s sign-off.
The bigger picture
Schneier ends on trust. A capable AI isn’t automatically a trustworthy one. It must be secure, on your side, and do things the right way. He thinks integrity will be the defining security problem of this decade.
I think he’s right, and most organizations will meet that problem first not as a security incident but as a line item. For the foundations, research institutes and nonprofits we work with, the genie is less likely to break the world than to quietly run up the tab, or damage a relationship that took years to build.
One final thought
Read the essay. Skip the parts about Roman filibusters if you must. But the genie part is worth sitting with before you hand an agent the keys.
Source: Bruce Schneier, “AI Doesn’t Have to Go Rogue to Break the World,” CyberScoop, adapted from his DEF CON 34 keynote, August 7, 2026.
Want to learn more? AI has been a subject of my writing for several years, and CGNET has offered AI user training and implementation for both large and small scale organizations. I would love to answer your questions! Please check out our website or drop me a line at g.*******@***et.com.




0 Comments