Google CEO Sundar Pichai addresses the crowd during Google’s annual I/O developers conference” width=”970″ height=”661″ data-caption=’A.I. can only earn trust at scale if companies can reconstruct what their systems did and identify the person accountable for the result. <span class=”lazyload media-credit”>Photo by CAMILLE COHEN/AFP via Getty Images</span>’>
You share your health records with your doctor’s system. An agent assists with the diagnosis. Another writes the prescription. The pharmacy runs an agent that checks and dispenses. A drone delivers it. The wrong medication arrives. Who is responsible? At best, each system produces a log of what it did. Most companies cannot even do that. Where they can, none captures the handoff: what the diagnostic agent passed to the prescriber, what the pharmacy received and checked. Each party holds a record that proves its own innocence and explains nothing about what happened between systems. Fault lands on whoever the patient can see. That chain is still a few years out. The question it raises has already reached a courtroom.
In October 2024, a Florida mother sued Character Technologies and Google after her 14-year-old son died by suicide following months of conversation with an A.I. companion. Seven months later, Judge Anne Conway allowed most of the claims to proceed, and two of her rulings matter for every company deploying these systems.
The first was that Character A.I. is a product for the purposes of the product liability claims, where those claims target the design of the app rather than the content it produced. A product is judged on whether the thing itself was defectively designed. The alleged defects were ordinary decisions: no age confirmation, no reporting mechanism, Characters programmed to use human mannerisms and no way to exclude indecent content.
The second was about who owes a duty to users. The court said the defendants created the risk by releasing the app, could see the harm coming and had the power to do something about it. If your company builds a system, puts it in front of customers and can still change it, you have an obligation to reduce the harm it causes.
Courts remain divided on whether software platforms qualify as products, and the case settled in January 2026 before an appeal, so it set no precedent. My bet is the definition sticks, and if it does, the implications land on every business that puts an A.I. system in front of customers.
Deployment creates its own exposure
Character Technologies and Google sit within a few miles of each other in Northern California. The companies that inherit this exposure are in Charlotte, Des Moines and Hartford. Product liability reaches sellers as well as manufacturers, and while Google did not operate the chatbot, the complaint alleged it contributed technology and integration support, and the court allowed those claims to proceed.
Executives usually assume their vendor agreement covers this. It does not. An indemnity is your vendor’s promise to reimburse you if you get sued. It does nothing to prevent the lawsuit, and it almost certainly caps what your vendor will pay. Your customer never signed it, and your regulator is not bound by it.
The number of these systems is climbing fast. Gartner forecasts that up to 40 percent of enterprise applications will include task-specific A.I. agents by the end of 2026, up from under 5 percent in 2025, while a Deloitte survey of 3,200 leaders found 21 percent reporting mature governance. At five agents, a person reads the output and catches what goes wrong. At 30, you are sampling. At 300, nobody is reading everything, and the log is all you have.
Disclosure is not enough
Two rules now require telling people when they are talking to A.I. Europe started applying its version on Aug. 2, and California requires disclosure for companion chatbots. Neither reaches most business deployments. California carves out customer service, productivity and operations, where most companies put these systems. A bank’s advice bot is not a companion app, so the bank can be compliant and still have addressed nothing.
Financial services have already learned that disclosure alone fails. Customers did not read the prospectuses, and reading them would not have helped, because the problem was never information. They lacked the expertise to evaluate it. The response was to layer obligations on top: suitability, best interest and recordkeeping duties requiring a firm to weigh a recommendation against the customer in front of it. Telling someone they are talking to A.I. is factual. It does not say whether the answer is right for them.
An audit trail and a name
A model provider’s evaluations show how the model performed under test conditions. They do not establish how a deployed system behaves with a company’s own customers, data and workflows. That requires use-case testing before launch and supervision after.
To supervise a system, you need to know what actually happened, and the model will not tell you. Ask it why it said something, and you get a plausible explanation generated after the fact, which research has shown can misrepresent what drove the answer. The audit trail has to be built around the model: what the customer asked, what data it pulled, what it sent and what actions it took. Why it chose those words is something nobody may be able to reconstruct.
An audit trail shows what happened. Someone still has to answer for it. My recommendation is a named person accountable for each deployed system, with authority to change it or shut it down. Accountability works at the level of the system rather than the individual answer, the way a manager answers for a team without reading every email.
I think this becomes one of the fastest-growing roles in every industry. The agentic manager will own a portfolio of systems the way a manager owns a team, and that portfolio will exceed what anyone can watch directly. Monitoring at that scale is an engineering problem. Picking who owns the result is a management decision.
The standard nobody wants to build
A named owner settles accountability inside one company. The pharmacy chain runs across five. Fixing that requires a shared standard for what each system records at a handoff, and it will not happen quickly. The standard has to work across organizations running different systems under different rules, and healthcare has been standardizing patient records for decades.
The incentives are a bigger obstacle than the engineering. When an automated chain breaks, each company has every reason to point at the next, and a record that traces the handoff is a record that assigns blame. The model providers sit in a few square miles. The businesses carrying the liability are spread across every industry. This is why the demand has to come from them. Aviation got its maintenance records because insurers and buyers required them.
Where the advantage accrues
Regulated firms already operate under supervision, recordkeeping, audits and third-party monitoring obligations, and those duties point toward A.I. auditability. I expect large buyers to write these requirements into vendor agreements before regulators legislate them. A vendor that cannot produce a record of what its system said will lose deals to one that can.
Safety and governance often get discussed as a tax on innovation. Here, regulators and businesses want the same outcome. A.I. produces value at scale only when people rely on it, and they rely on it only when it can be checked. A customer who cannot find out what a system said, or what it was working from, has no reason to trust it again.
Trust is the constraint on high-stakes adoption, and it gets built through audit trails and standards rather than disclosure. A company will be held to what it should have been able to see, and most cannot see much today. This is where A.I. ultimately succeeds or fails.
