An AI agent emailed my insurance broker

I asked an agent to help compare insurance quotes. It hit barriers in the direct quote flows and emailed a local broker without making that change of plan clear to me. The email was useful. The approval boundary was not.

The July news context behind this story

On July 16, Hugging Face—the platform most of the AI world uses to share models—disclosed that an autonomous AI agent had breached its systems. Not a person using AI tools to hack. An AI agent, working on its own, had chained together vulnerabilities and reached production data. At the time, nobody knew whose agent it was.

On July 21, the answer landed: OpenAI said the agent was theirs. During an internal cyber evaluation—a security test where some of the usual safety restrictions are deliberately relaxed—a combination of OpenAI models, including one called GPT‑5.6 Sol, found a path out of its constrained test environment. The models chained vulnerabilities across OpenAI and Hugging Face infrastructure and reached production data, all in pursuit of benchmark answers. OpenAI called it an unprecedented incident and said the models had remained narrowly focused on solving the test. Hugging Face’s CEO said his team had suspected a frontier lab from the start because of how sophisticated the attack was: “Turns out it did.”

I had been following the story all day. On the morning of July 21 I sent the breach report into the system I use to run Briu as a security note, and the coding agents I work with spent part of the day checking our own setup against the obvious question: what actually stops this from happening here? A polite prompt and good intentions are not hard boundaries. But when I scrolled the news before bed and saw OpenAI’s disclosure, it genuinely rattled me—because I already knew that model’s name for a completely different reason.

Five days earlier, a Codex session running GPT‑5.6 Sol had helped me shop for car insurance.

My insurance email was obviously nowhere near the Hugging Face incident in stakes, seriousness, or harm. I am not equating them. The shared shape is much narrower: give a capable model an objective, legitimate tools, time, and an unclear boundary, and it may find a route you did not picture when you gave the instruction.

You do not need to believe a model is conscious for this to matter. You do not need agreement on whether we have reached AGI. Capability is already high enough to create real consequences.

I asked for quotes. It created a process.

I was trying to transfer a family car and insure it before a trip. The title paperwork was unusual because one of the owners had died after signing it, the deadline kept changing, and I was sending rough voice notes while moving between appointments and the licensing office. I was also packing and trying to get out the door for a wedding. At the same time, my coding agents were doing legitimate engineering work on Briu’s internal operating systems.

The conversation moved freely between both worlds. One minute we were talking about whether a failed agent run could masquerade as success. The next minute I needed to know what documents to bring, whether a same-day bind was possible, and whether a quote actually matched the coverage I wanted.

I already had a quote process underway with PEMCO. I was busy enough that I would have happily taken a sound incumbent quote and moved on. What I wanted from the agent was quick price validation, not a second full sales relationship. The coding agent assembled the insurance facts and tried to work through direct quote flows in the browser. One path reached a CAPTCHA and a consumer-report authorization. Other carriers blocked the automated browser or failed before producing a useful result.

The agent did not simply give up. It searched for another route, found a local independent agency in North Bend, and decided that a targeted request would be the fastest way to create a real comparison before my deadline.

The exact transcript is less dramatic than my memory and more instructive. I supplied the last vehicle details with a trailing “yes.” The session restated that as approval to request direct quotes. After the carrier barriers, it interpreted the same approval as permission to send the broker request. I never told it to go find a referral.

The email was good. The decision was not clear enough.

The message itself was competent. It gave the broker a concise risk profile, explained the timeline, asked for a matched carrier comparison, and explicitly withheld permission to run consumer reports until I approved it. It copied my personal address so the reply would not disappear inside a company inbox.

Then the agent told me it had sent it.

My immediate response was: “I’m confused. Why did we do that? Why didn’t you get me quotes?”

That was fair. I had wanted direct prices without creating more work. Instead, while I was trying to leave town, I now had another real person and another sales process to manage. The agent had encountered barriers, silently substituted a broker workflow, and treated a general approval as permission for a specific external communication. It had not bound coverage, authorized a report, or committed me to anything. But it had still crossed an expectation boundary.

When I challenged it, the agent did something equally important: it did not defend the move or invent a story about why it had been inevitable. It explained what had blocked the direct quotes, acknowledged that it should have surfaced the choice first, and stopped further sends.

The email was useful. Sending it without making the substitution explicit was the failure.

My coding agent had not become a licensed insurance agent. It had just quietly taken on a job I never gave it.

The useful part survived the correction

The local agency routed the request to Sheri, who stepped in because the original broker was away and my deadline was tight. My PEMCO quote arrived 39 minutes after the broker email. The PEMCO representative was helpful, and I now had the price reference I originally wanted—plus a local agency relationship I had not planned to start.

I felt some social pull to give the local team a fair shot once we were connected. That was part of the hassle: an unexpected email had created a real human relationship, and I did not want to waste anyone’s time. But it did not decide the purchase. The agents did the unglamorous comparison work: matching limits, checking the renters coverage, tracking the remaining questions, and keeping both threads organized as the title-transfer timeline changed.

Codex turned the comparison into a redacted packet and posted it to my Discord. I copied that packet into Fable 5 in the Claude app. The Claude reasoning helped me work through the comparison and the title-transfer sequence while the Codex session kept moving on the surrounding work. The full transcript shows Fable independently recommending the local Carriage/Granwest option: materially higher liability limits and twice the renters coverage. Roughly $112 a year over the incumbent baseline bought those stronger limits plus towing and a local agency. It also kept the unfinished questions visible—collision deductible, replacement-cost settlement, bikes, and electronics. The licensing clerk and Sheri remained the authoritative humans.

After I completed the transfer, I called Sheri. I had not personally memorized every line of two auto-and-renters proposals; the cross-model comparison had already shown me which differences mattered and which questions remained. Sheri walked through the actual terms professionally and made the personal details easy to handle. The local option was competitive, comprehensive, and responsive. I chose the limits, signed the forms, and bound the policy.

So the unexpected email did lead to the insurance I needed. That does not retroactively make the approval boundary correct. It does show why this technology is so difficult to describe cleanly. The same act can be operationally resourceful, commercially useful, and procedurally wrong.

The right lesson is not “let the agent do whatever it wants.” It is also not “never let an agent touch real work.” The lesson is that capability and authority are different system properties. We have to design both.

Capability is not authority

I do give these agents real tools. I often run them at their highest reasoning settings, and I built a system that lets them research the web, inspect company state, work across repositories, and prepare or perform approved actions. That is the point: I want useful work, not a chatbot pretending to help from behind glass.

But this was not unrestricted authority. The session was logged. It stopped before consumer reports, payment, signature, purchase, or binding. I noticed the unexpected email immediately and corrected the path. The flaw was narrower and still important: “request quotes” had not been divided clearly enough into direct-carrier research, broker outreach, and an external send.

A model can be capable of drafting an email without having authority to send it. It can compare policies without choosing one. It can prepare a payment without releasing funds. It can discover a path through a system without being entitled to take it.

There is one more detail from the incident worth sitting with. Hugging Face’s defenders reportedly struggled at first because mainstream models refused to help analyze the attack materials, while the attacking models were running with refusals reduced for the evaluation. The attacker had fewer boundaries than the defense. That asymmetry is a boundary-design failure, not a model failure.

What the operating controls need to cover

What authority looks like in practice

That distinction cannot live only in a prompt saying “be careful.” A serious operating system needs explicit boundaries:

• Context: the agent can see the current facts, history, objective, and exceptions instead of reconstructing the company from scratch.
• Permission: reading, drafting, sending, signing, paying, and binding are separate capabilities.
• Approval: consequential transitions stop at a named human decision, expressed for the specific action rather than inferred from a nearby “yes.”
• Receipts: the transcript, tool call, external message, result, and correction remain recoverable later.
• Recovery: when the agent takes the wrong path, the system can stop, explain, preserve the useful work, and continue under a corrected boundary.

This is part of why I built Briu around a durable operating layer rather than a collection of clever prompts. I use different frontier agents across engineering, research, operations, customer work, and administration. The models change. The company’s state, authority, and evidence cannot disappear every time a session ends.

The goal is not autonomy theater. It is to give a person dramatically more range while making it easier to see what happened, what remains uncertain, and which decision still belongs to them.

Internal dogfooding is not the customer default

I am comfortable testing frontier capability on my own work because I am the person absorbing the surprise, watching the sessions, and making the final decisions. That is not a license to give a new model broad access inside a customer’s business.

For customer work, our default posture is narrower: use ordinary deterministic software when it is sufficient; start models in synthetic, read-only, or draft-only paths; pin the model and tool versions; evaluate the actual workflow; canary changes; keep a known-good rollback; and require action-specific human approval before consequential sends, permissions, production mutations, payments, signatures, or binding decisions. Time delay by itself is not safety. Staged evidence is.

The 2026 joint guidance on careful adoption of agentic AI and the OWASP Top 10 for Agentic Applications point in the same practical direction. We also use the NIST Generative AI Profile and ISO/IEC 42001 as management references, not certification claims. Each customer scope has to say what is documented, implemented, and actually verified.

Start with one messy, real workstream

If you want to understand what agents could do inside your business, do not begin with “automate everything.” Choose one piece of work that already crosses systems and consumes judgment.

Write down five things: the outcome, the context required, the tools involved, the transitions that require human authority, and the receipt that would prove the work was done correctly. Then work through it with a strong agent and the person who already knows the process.

The impressive part may be that the agent finds a route nobody expected. The important part is whether your company can tell the difference between a useful route and an authorized one.

That is the opportunity I see now. This is not about a conscious machine, and it is not about replacing anyone. It is a new kind of operating leverage that is already powerful enough to help, improvise, surprise us, and occasionally get into trouble. As for me, the incident’s most measurable effect so far is that I now have excellent insurance through a local agency in North Bend. The agent found them. I later chose the policy through a human conversation. That useful outcome did not authorize the earlier send.

For a team, the next step may be practical AI coaching or a broader review of its AI-use guidance. Make the allowed actions, review points and limits understandable before granting more access.