Confronting Hallucinatory AI: The End of Accountability?

By Sebastian Smith '26

---

When one makes the conscious decision to end another person's life, they are swiftly held accountable by the hand of the law, often with grave consequences. But let's imagine a world in which, if you are hallucinating, then you cannot be held accountable at all. If you are hallucinating when you pull the trigger, then accountability is essentially waived. In this world, psychedelics, or any hallucinogens for that matter, relieve you of any and all culpability—but only if you were hallucinating. While seemingly dystopian, this elusiveness of accountability is the current reality of “hallucinatory” artificial intelligence. In this article, I argue that the liability allocation frameworks within the US legal system are ill-equipped to answer two integral questions: who is accountable for AI-generated misinformation, and what legal standard of accountability can apply?

Hallucinatory Systems: The Gravity of AI-Generated Misinformation

While first rising to widespread societal prominence in late 2022 with the public release of ChatGPT-3.5, artificial intelligence (AI) originally traces back to the infamous 1950 paper published by Alan Turing, Computing Machinery and Intelligence. (1) Since then, it is no secret that the field has experienced unprecedented advancement, specifically in Machine Learning (ML). ML is the broad field of AI where computers learn from data to render decisions, operationalized today by Large Language Models (LLMs), which are predictive engines trained on vast amounts of text to understand, summarize, and generate natural language. (2)

While a remarkable innovation, LLMs are by no means omnipotent, plagued by pervasive “hallucinations.” Hallucinations are a phenomenon that occurs when LLMs perceive patterns or objects in their training data that drive the creation of nonsensical or inaccurate responses. (3) Considering the monumental influence and reliance on AI in contemporary human life, the widespread emergence of hallucinations poses an imminent danger. From increasing connections to critical infrastructure such as power grids, stock markets, and roadways, to diagnosing patients for healthcare professionals, to overtaking traditional search engines, AI-generated misinformation will only become more costly, with human lives potentially at stake. (4)

As in other industries, legal and regulatory frameworks that provide accountability are key to ensuring responsible development and protecting consumers from harm. Accountability regimes strive to provide the same for AI innovation, aspiring to mitigate hallucinations by mandating prompt discipline, comprehensive disclaimers, and continual system improvement. Nevertheless, AI hallucination poses a uniquely difficult responsibility problem, first raised by the question: “who’s accountable in the first place?”

The Missing Defendant

When we think of the responsible parties for “AI,” it’s easy to think exclusively of the prominent frontier model developers such as OpenAI or Anthropic. In reality, though, AI consists of far more than a single entity, existing within a complex chain of actors. A hallucination attributed to “the AI” may actually originate at any of the nodes along this chain, from the developer who trained the underlying model, to an entity that fine-tuned it for a specific application, to even the company deploying the model for the end-user. (5)

Of course, complex supply chains are not new to liability law. Courts navigate them constantly, and in industries with far more moving parts than AI, they still manage to isolate the responsible party. (6) Take the oil industry: a spill can implicate a drilling operator, a pipeline company, a refinery, and a shipping carrier, each occupying a distinct link in the chain. Nevertheless, due to the traceability and explainability of such defects, liability frameworks are able to effectively navigate the intricate chain of actors, pinpointing the accountable party. (7)

Unfortunately, AI offers no equivalent. Modern LLMs process natural language through a vector space of thousands of dimensions, hundreds of billions of parameters, and dozens of processing layers. (8) In other words, these systems are intractable “black boxes.” Where in the computational process the error occurred, and which actor of the supply chain is responsible for facilitating such an error, is effectively impossible to determine. Unlike the oil spill, where the causal sequence that led to the harm is explainable, in AI systems such transparency is simply unfeasible. Therefore, the plaintiff will have an insurmountably difficult time identifying the “defendant” responsible for their incurred harm, and with nowhere to point a finger, accountability remains elusive.

Three Classes of Accountability, Three Systematic Gaps

Even if we could identify a single relevant actor, the legal theories of liability are ill-equipped to assign blame and facilitate relief. To see this, it helps to separate accountability into three distinct classes: criminal, civil, and regulatory. Each operates under different rules and, as I will argue, fails to meaningfully engage with AI in its current form.

Criminal Accountability

As the prevalence of autonomous AI has grown, so has the fear of Machine Crime, occurring when an automated system is the proximate cause of a criminal offense. But for an entity to be held criminally accountable, statutes require the culpable party to possess a criminal intent, otherwise known as mens rea. (9) However, with the advent of autonomous learning machines powered by AI, these systems possess the ability to act completely beyond the intentions of any human party present in the supply chain. This is what Andreas Matthias calls the responsibility gap, where the manufacturer/operator of the machine is in principle not capable of predicting the future machine behavior, and thus cannot be held liable for it. (10) In other words, the intent of the individuals developing, designing, and deploying these systems may be entirely pure, rendering the mens rea element of criminal accountability impossible to attain. And, of course, attributing criminal intent to the AI system itself is also infeasible, as it lacks the consciousness to intend anything at all, producing only outputs deemed statistically most likely after being processed through the model's transformer layers. Therefore, criminal accountability isn’t the solution to ascribing blame to hallucinatory systems.

Civil Accountability

The apparent solution, then, would appear to be an accountability framework that doesn’t require intent at all. After all, harm may occur as a result of hallucinatory AI without any ill intent. This is a notable feature of negligence, the most common form of civil liability, which only requires a duty of care, a breach of that duty, causation, and resulting injury. (11) Still, negligence also fails to ascribe accountability for AI-generated harm, as the requirement of establishing a “breach” mandates that there be a party who failed to uphold their standard of care. (12) This becomes problematic because a developer can uphold such standards rigorously, while harmful hallucinations persist. Consider a developer that curates its training data with care, maintains human-centricity, and thinks critically about the potential negative impacts of their model. This, per NIST’s guidelines, adheres to a standard of care recognized by the industry, and thereby the courts. (13) Yet, even a model built this way can promulgate misinformation and cause irreparable harm. So, without a breach of duty occurring, no party can be held liable for negligence, again leaving the harmed consumer without recourse.

Still, another form of civil liability remains feasible: strict products liability. Strict product liability applies when a defective product for which an appropriate defendant is responsible causes injury to an appropriate plaintiff, regardless of intent or breach of duty. (14) This framework, at first glance, looks to be the best option yet; it doesn’t question whether the system or any individual in the supply chain “intended” to hallucinate, or whether said entities failed to uphold a standard of care, only whether the defect caused harm. Unfortunately, though, products liability has traditionally encapsulated only tangible goods with stable, identifiable defects. (15). In fact, while courts have not yet reached consensus on whether AI counts as a “product” (16), others, like the U.S. Court of Appeals for the Third Circuit, have reaffirmed the “tangible” product requirement, thereby excluding AI. (17) Moreover, the Uniform Commercial Code, a model law many states have adopted, also refuses to classify software as a product. (18) Therefore, under the current posture of strict products liability, it’s simply inapplicable to the attribution of liability to modern ML systems.

Administrative and Regulatory Accountability

Regulatory accountability is meant to fill the gaps left by criminal and civil law. But our current regulatory initiatives, both domestic and international, approach accountability in ways that assume a human actor is always in control and overlook the shortcomings of current accountability frameworks.

In 2023, President Biden issued the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence order to lay the groundwork for a future marked by responsible AI. While an admirable effort, the order possesses a fundamental shortcoming. When addressing accountability, the order frames “existing consumer protection laws” as sufficient to ensure “safeguards against fraud, prevent unintended bias, discrimination, infringements on privacy, and other harms from AI. (19) However, as demonstrated above, existing liability allocation frameworks are not, in fact, sufficient to attribute responsibility to appropriate actors and grant harmed consumers relief. While consumers injured by typical products or negligence possess clear recourse and avenues of private action, those inflicted harm at the hand of hallucinatory systems lack any clear remediation options.

This problem isn’t reserved to the United States. The European Union issued the Declaration on Digital Rights and Principles for the Digital Decade, a document regarded as one of the most comprehensive outlines for managing the revolutionary impact of AI. The declaration affirms that it will “hold accountable those that seek to undermine security online and the integrity of the digital environment.” (20) Once again, we are presented with the issue of intentionality, or lack thereof. LLMs are non-sentient entities, operated by their underlying transformer layers; they simply cannot intend anything. Consequently, if an autonomous system truly does “undermine the security and integrity of the digital environment,” as the declaration forbids, it still eludes accountability, as it does not “seek” to do so at all.

Conclusion

As I’m sure many of you do, I hope policymakers, tech developers, and judges can reach a compromise on the critical issues of hallucinatory AI and accountability, paving the way for a future marked by responsible AI. That compromise will likely need to begin by attributing AI the status of a product, so that strict product liability can serve as an applicable framework for assigning accountability. But even this is not a decisive solution, with the First Amendment potentially protecting speech-based outputs and uncertainties surrounding Section 230 of the Communications Decency Act of 1996, which has absolved technology companies of liability in the past (21). Either way, until further progress is achieved, we are largely on our own. We must therefore be cognizant of our reliance on AI and cautiously analyze its risks, to avoid becoming yet another victim of hallucinations.

Endnotes

1. Turing, Alan. “Computing Machinery and Intelligence.” Mind, vol. 59, no. 236, 1950. Discussed in “The Birth of Artificial Intelligence (AI) Research,” Lawrence Livermore National Laboratory, st.llnl.gov/news/look-back/birth-artificial-intelligence-ai-research.

2. “The Difference Between AI vs. Machine Learning vs. LLM vs. Generative AI.” Toloka, 2023, toloka.ai/blog/difference-between-ai-ml-llm-and-generative-ai/.

3. “What Are AI Hallucinations?” IBM, 2024, www.ibm.com/think/topics/ai-hallucinations.

4. Metz, Cade. “How Could A.I. Destroy Humanity?” The New York Times, The New York Times, 10 June 2023, www.nytimes.com/2023/06/10/technology/ai-humanity.html.

5. National Telecommunications and Information Administration. Artificial Intelligence Accountability Policy Report (Ellen P. Goodman, principal author). U.S. Department of Commerce, 27 Mar. 2024, www.ntia.gov/issues/artificial-intelligence/ai-accountability-policy-report.

6. “Products Liability.” Wex, Legal Information Institute, Cornell Law School, www.law.cornell.edu/wex/products_liability. Accessed Mar. 2024.

7. Nichols, James E. Oil Pollution Act of 1990 (OPA): Liability of Responsible Parties. Congressional Research Service, Rep. No. R41266, 2 June 2010, www.everycrsreport.com/reports/R41266.html.

8. Brown, Tom B., et al. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33, NeurIPS 2020, arxiv.org/abs/2005.14165.

9. “Mens Rea.” Wex, Legal Information Institute, Cornell Law School, www.law.cornell.edu/wex/mens_rea.

10. Matthias, Andreas. “The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata.” Ethics and Information Technology, vol. 6, no. 3, 2004, pp. 175–183, doi.org/10.1007/s10676-004-3422-1.

11. “Negligence.” Wex, Legal Information Institute, Cornell Law School, www.law.cornell.edu/wex/negligence.

12. Ibid.

13. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, Jan. 2023, doi.org/10.6028/NIST.AI.100-1.

14. “Strict Liability.” Wex, Legal Information Institute, Cornell Law School, www.law.cornell.edu/wex/strict_liability.

15. American Law Institute. Restatement of the Law, Third: Torts: Products Liability. 2010, §19.

16. Raji, Inioluwa Deborah, I. Elizabeth Kumar, Aaron Horowitz, and Andrew D. Selbst. "The Fallacy of AI Functionality." Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT '22), 21 June 2022, pp. 959–972, doi.org/10.1145/3531146.3533158.

17. American Law Institute. Restatement of the Law, Third: Torts: Products Liability. 2010, §19 (defining “product” as tangible personal property).

18. Uniform Commercial Code, § 9-102, Definitions and Index of Definitions, Item 42, “General Intangible.”

19. Executive Order 14110: Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. President Biden, 2023.

20. European Union. “European Declaration on Digital Rights and Principles for the Digital Decade.” European Commission, 2022.

21. Brannon, Valerie C., and Eric N. Holmes. Section 230: An Overview. Congressional Research Service, R46751, 4 Jan. 2024, crsreports.congress.gov/product/pdf/R/R46751.