Catalyst Study on Technology Assisted Review of Family Batches

Today I participated in a webinar hosted by ACEDS and conducted Thomas Gricks and Jermey Pickens of Catalyst, entitled, Just Say No to Family Batching in Technology-Assisted Review. Gricks and Pickens are the author, along with Andrew Bye, of a paper entitled, Break up the Family: Protocols for Efficient Recall-Oriented Retrieval Under Legally-Necessitated Dual Constraints. Gricks, et al. challenge the standard notion that since families of documents are produced together they should also be reviewed together. The authors advocate a 'broken family' review protocol, and 'dual phase workflow' with an initial expedited review for relevancy.

TAR algorithms will be more effective when trained with individual documents rather than complete families. Catalyst employed a continuous active learning protocol. A Full Family continuous active learning protocol will pull all documents in a family into the review queue irrespective of whether or not they are highly ranked. This is the approach favored by most attorneys.

In a Positive Family protocol any time one relevant document from a family is found to be relevant, any documents from the same family found to be non-relevant are not re-reviewed.

In an Individual Padded continuous active learning protocol, once a relevancy level is determined for any one document in a family, the rest of the documents in the family are added to the review queue.

Catalyst got results from eight different e-discovery projects. Its study shows how much additional review is needed to achieve recall rates of 75% or 90%. Positive Family and Individual Padded continuous active learning are shown to be clearly more efficient than Full Family review.

Phased continuous active learning involves reviewing documents first only for relevance, and removing the entire family from the queue when any one document has been determined to be relevant. In the second phase every document family with a relevant document is reviewed for both relevancy and privilege This approach is superior to both Full Family and Individual Padded continuous active learning.

Amendment to New York State Court Rules Encourages the use of TAR

Last week the Commercial Division of the New York State Supreme Court implemented a new rule designed to encourage parties to use technology assisted review. Rule 11-e concerns responses and objections to document requests. As the request for public comment on the proposed amendment indicated the rule has been modified to include this language:

The parties are encouraged to use the most efficient means to review documents, including electronically stored information ("ESI"), that is consistent with the parties' disclosure obligations under Article 31 of the CPLR and proportional to the needs of the case. Such means may include technology-assisted review, including predictive coding, in appropriate cases.

The Commercial Division Advisory Council's memorandum on the proposed rule (available at the above link) notes that parties are also encouraged to use email threading; near duplicate detection, and clustering in addition to TAR to speed up document review and reduce its cost.

The memo specifies that it is the producing party which is best suited to determine whether or not it should use TAR. While the court notes that a requesting party can challenge the method of review, and a judge will retain oversight of discovery, it cites a decision by a federal court strongly endorsing the producing party's discretion. “Unless [the responding party’s]choice is manifestly unreasonable or the requesting party demonstrates that the resulting production is deficient, the court should play no role in dictating the design of the search.” Mortg. Resolution Servicing, Inc. v. JP Morgan Chase Bank, N.A., 15-CV-0293,2017 WL 2305398, at *2 (S.D.N.Y. May 18, 2017).

No preference is given to a particular kind of technology assisted review. "[T]he appropriateness of a given methodology can only be determined in the context of the particular case and the data set to be reviewed." The Council points out that the preamble to the rules directs, "proportionality in discovery".

In re Broiler Chicken - Federal Court Order Sets TAR Protocol

This past January, Magistrate Judge Jeffrey Gilbert and Special Master Maura Grossman, issued an order setting a protocol for technology assisted review. See, In re Broiler Chicken Antitrust Litig., 1:16-cv-08637 (N.D. Ill. Jan. 3, 2018). While there have been many court decisions approving the use of TAR in discovery, few courts have given detailed instructions on how TAR is to be performed. This matter involves 30 different defendants, a fact which may have prompted the court to set the protocol.

The Order Regarding Search Methodology for Electronically Stored Information addresses document source disclosures; search methods, and means to validate the process. Here's a list of key aspects of the protocol:

1. De-duplication by hash values across all custodians is required.

2. Parties may choose to conduct email threading and only produce inclusive emails.

3. Categories of documents to be produced in their entirety (e.g., a network folder containing agricultural reports) are to be excluded from data sets used to test search terms.

4. Processing exceptions must be disclosed if they are not system files. An example would include encrypted files.

5. If a party uses TAR (or Continuous Active Learning), they have to disclose the vendor and software; a general description of the training algorithm; the quality control measures; and the categories included or excluded from TAR. Requesting parties have 7 days to propose exemplars for training or alternate keyword search strings.

6. Search software is also to be disclosed, along with the stop words the program uses and other information about its capabilities.

7. False positives can be proposed in addition to keywords, but contextual examples must also be provided.

8. The QC sample must include 500 random documents responsive to at least one request for production; 500 documents marked non-responsive by a human reviewer if they were selected for review by TAR or 2500 documents marked non-responsive by a human reviewer if they were selected by manual review; and 2000 documents excluded from manual review by the TAR process. This validation sample is to then be reviewed by a subject matter expert.

9. The parties have to meet and confer on whether or not the review of the sample indicates that a substantial amount of responsive documents are being identified.

10. A recall of 70-80% is consistent with, but not a sole indicator, of adequate review. The recall estimation method for TAR is:

# of Responsive Docs Found

__________________________________________________________________

# of Responsive Docs Found + # of Responsive Docs Coded Wrong + # of Responsive Docs Not Reviewed

The recall estimation method for Manual Review is:

# of Responsive Docs Found ___________________________________________________

# of Responsive Docs Found + # of Responsive Docs Coded Wrong