How AI Works in the DISCO eDiscovery Platform

DISCO has integrated a proprietary AI system called Cecilia into its eDiscovery platform. I received a demonstration of it this past week, and Cecilia AI shows how artificial intelligence is transforming electronic discovery and how attorneys work with document productions.

Cecilia AI can answer questions based on an analysis of the data set loaded into DISCO, and it can further narrow down the data it considers in its answers based on a smaller subset of data. In this example you can see that it identifies the position of an executive whose name appears in the Enron email data set.

Similarly, Cecilia AI can provide definitions for unfamiliar terms with a simple right-click function:

Cecilia AI does not however allow a user to provide feedback, instructing it to correct a mistake or hallucination so it will not get repeated in the future for other users who may not recognize the mistake. So if an attorney knew that in fact Kenneth Lay had become CEO of Enron in 1984 rather than in 1985, it cannot tell Cecilia AI to give that answer going forward.

There is an option to reset Cecilia so that it will not base its results on previous inputs made by users of the database.

It does list the documents that it uses as the basis for its answers.

You can ask Cecilia questions about individual documents, such as whether or not a contract has a clause addressing potential damages. DISCO provides Cecilia as a free feature on all databases, and up to 50 questions based on a single document or document summaries can be generated each day without opting for Cecilia to be fully enabled.

Cecilia will not answer questions about documents which are less than 300 characters, or more than 250,000 characters. The upper limit is surprising - a short novel like The Adventures of Huckleberry Finn is about 455,000 characters.

DISCO's Auto Review uses Cecilia to identify relevant documents for production based on tags in which a subject matter expert simply explains in ordinary language the kind of documents he or she wants to be identified:

Auto Review provides percentages for precision, recall, and prevalence, and breaks down what fraction of document results are associated with a given tag.

Cecilia can tell a user the custodian for a specific document, and it can run searches for document-based queries made in ordinary English such as, "Show all documents between July 4, 2021 and December 25, 2023", or, "How many email messages are in this database?"

Relativity Study on AI

Relativity commissioned a study last year on how lawyers are using artificial intelligence. Here are some key points that I found interesting:

1. While 38% of law firm study participants used AI software, significantly more — 50 % — of government employees did.

2. AI software was most often used by legal teams for document review.

3. Two-thirds of study participants have implemented training programs to help employees learn how to use AI.

4. Paralegals actually use AI more often than lawyers.

5. AI is more often used as a way to automate low-level tasks, and with the goal of cutting costs - two times more frequently than as a means to enhance risk compliance or legal analysis.

6. There was more concern about the loss of confidential data, than there was about misleading AI hallucinations.

7. IT professionals tend to be concerned about the loss of confidential data that is input into large language models (LLMs).

8. Law firms were twice as likely to use in-house proprietary models or software provided by vendors as they were to rely on publicly available AI software.

Attorneys are citing to cases which don't exist

It's not only becoming common for law firms to submit briefs to courts which they drafted with the assistance of artificial intelligence software, courts are catching them in the act and finding that some of the caselaw cited to in these briefs is completely fictional. AI 'hallucinations' are instances where AI software generates fallacious information in response to a question.

In January, the Second Circuit issued a per curiam decision, Op., Park v. Kim, No. 22-2057 (2d Cir. Jan. 30, 2024), ECF No. 178-1, in which the court found that:

The attorney who filed the brief was referred to the Court's Grievance Panel. The Court cited the 5th Circuit's amendment to its rules which requires that attorneys certify that no generative artificial intelligence was used for a filing, or that at least that the filing was reviewed for accuracy by a human.

In May 2023, Judge Castel of the United States District Court for the Southern District of New York, issued an Order to Show Cause why the plaintiff's counsel should not be sanctioned pursuant to Fed. R. Civ. P. 11 and the Court's inherent authority for submitting an affirmation in opposition to a motion to dismiss which included citations to six "bogus judicial decisions with bogus quotes and bogus internal citations." Order to Show Cause at 1, Mata v. Avianca, Inc., No. 22-cv-1461 (PKC) (S.D.N.Y. May 4, 2023), ECF No. 31. The brief used citations to reporters which actually refer to other cases:

Id. at 2.

Not only did the plaintiff's affirmation refer to Varghese v China South Airlines Ltd., 925 F.3d 1339 (11th Cir. 2019), and other cases which are completely made up, but when the court issued an order asking that plaintiff's counsel submit copies of these cases (Order, Mata v. Avianca, Inc., No. 22-cv-1461 (PKC) (S.D.N.Y. Apr. 11, 2023), ECF No. 25) the counsel in turn filed an affidavit submitting AI generated copies of the imaginary cases! Affidavit, Mata v. Avianca, Inc., No. 22-cv-1461 (PKC) (S.D.N.Y. Apr. 25, 2023), ECF No. 29.

A copy of Varghese was filed on PACER, but it's something that AI simply invented:

PACER doesn't lie!

At a subsequent hearing, Judge Castel excoriated the plaintiff's attorney for submitting the fictitious case:

Hr'g Tr. at 15:17-17:6, Mata v. Avianca, Inc., No. 22-cv-1461 (PKC) (S.D.N.Y. Apr. 11, 2023), ECF No. 52. So the case that ChatGPT invented was not even one which had an internal logic of its own.

The court sanctioned the plaintiff's attorney under Rule 11, fined him $5,000 and ordered him to send a letter to each judge listed as the author of the false cases the affirmation cited to. Mata has been dismissed for being untimely under an international convention that covered a claim for an injury suffered by the plaintiff during an international flight.