Fake AI use in courts could be catastrophic, Supreme Court warns - PRESS Insider

The Supreme Court of India has set aside insolvency orders passed by the National Company Law Tribunal (NCLT) and the National Company Law Appellate Tribunal (NCLAT) after discovering they were based on non-existent, AI-generated legal precedents. Justices P.S. Narasimha and Alok Aradhe warned that the use of such hallucinated material constitutes a serious lapse in the judicial process and renders a decision invalid in the eyes of the law.

What happened

The controversy emerged during insolvency proceedings against Essel Infraprojects Ltd. The NCLT Mumbai Bench had admitted an insolvency plea from Jammu and Kashmir Bank Ltd on August 28, 2024, a decision later upheld by the NCLAT. During the appeal, senior advocate Madhavi Divan revealed that several precedents cited by the NCLT were either entirely fabricated or contained passages that did not exist in the actual judgments.

Upon examination of six cited decisions, the Supreme Court found a mix of wrong citations, non-existent cases, and instances where the citation existed but the specific paragraph relied upon was false. J&K Bank confirmed that these disputed cases were not cited by their counsel, indicating the material originated from the tribunal’s own research process. The Court described the introduction of such fabricated material into the legal system as "catastrophic," comparing it to the release of a toxic substance that is invisible and insidious.

In response, the Supreme Court ordered the matter back to the NCLT for fresh consideration. The Bench emphasized that adjudication must remain under the absolute control of humans and that any decision influenced by even an "iota" of hallucinated material must be set aside. The Court has directed the Bar Council of India to form a committee to establish guidelines and disciplinary protocols to prevent lawyers from submitting unverified AI-generated material. This incident highlights the growing risks of AI in the judiciary, occurring shortly before the Supreme Court's AI Committee proposed new draft regulations to ensure human primacy, transparency, and accountability in legal technology.