TAR for Smart People Outline

More than a year ago now I posted an outline of the first three chapters of John Tredennick's 'TAR for Smart People'. This is a continuation of that outline using the first edition. An updated edition has been posted at: http://www.catalystsecure.com/tarforsmartpeople .

4. TAR 2.0 Capabilities Allow Use in Even More E-Discovery Tasks

• Document Review o Classification - responsive / non-responsive o Protection - privileged / trade secrets o Knowledge Generation - specific issues / deposition witnesses

• Metrics o Recall – the percentage of relevant documents actually recalled. o Precision – the percentage of retrieved documents that are actually relevant.

• Classification Tasks o FRCP and Sedona Principles – e-discovery is limited by principles of reasonableness and proportionality. o 80% recall is a common TAR target. Gold standard of linear review can’t do better than this and costs more. o Recall usually gets more attention than precision.

• Protection Tasks o For confidential information, 100% recall is necessary. o Use TAR; keyword searching; and human review – stack techniques. TAR systematic errors; human random.

• Knowledge Generation Tasks o Precision is the most important metric. o Prioritize document population by issue. o Concentrate most interesting documents for review first. o TAR imperfectly concentrates interesting documents near top of responsiveness ranking.

5. Measuring Recall for E-Discovery Review: An Introduction to Recall Sampling

• Review high level of recall (75%) after only reviewing small percentage of documents (5%). Discard pile include so few relevant documents that more review not economically justified.

• Hypo: 1M document production. 1% relevant – 10K documents. o Using Sampling to Estimate Richness  Statistical sampling to estimate richness.  Randomly selected subset  Concepts • 1. Point Estimate – most likely value for a population characteristic. • 2. Confidence Interval – ranges of values around point estimate that we believe contains true value. e.g. 8K to 12K. • 3. Margin of Error – max. by which a point value might deviate from true value. • 4. Confidence Level – chance confidence interval will include true value. • 5. Sample size – higher confidence level more docs must be reviewed.  Determine sample size with Raosoft calculator. Inputs: • Document set size 1,000,000 • Confidence level • Margin of Error 4% • RESULT: 600 documents. o Initial Sampling Results  If 6 relevant documents found; estimate 1% richness. o Determining the Exact Confidence Interval  Binomial Calculator to determine confidence interval

We multiply these decimal values against the total number of documents in our collection (1,000,000) to calculate our exact confidence interval. In this case, it runs from 3,700 to 21,600. We believe there are 10,000 relevant documents in our collection (our point estimate) but it could be as high as 21,600 (or as low as 3,700). Let’s move on to our review.

• The Review - if we find 7,500 relevant docs in 50K may have to review 950K docs to get 2,500 more – not reasonable. But what if 21,600 relevant documents? 35% recall not sufficient.

• Sampling Discard pile – again 600 sample size. E.g. 2 relevant documents.

• Use binomial calculator again – find could be as many as 11,400 relevant documents in 950K docs left. Only getting 7502/18900 – only 40%.

• Narrow margin of error again and use Raosoft calculator – e.g. 1% margin of error – must review 9,508 documents.

• Find 31 documents relevant documents. estimate there are 3,097 relevant documents in the discard pile, about the same as before (950,000*(31/9508)). Range could be 2090 to 4370 documents. Using these values for our exact confidence interval, the range goes from 63% (7,500/11,870) to 78% (7,500/9,590).

Sedona TAR Case Law Primer

This past August, the Sedona Conference published its TAR Case Law Primer, which is available for download here. It mentions Da Silva Moore v. Publicis Group, 287 F.R.D. 182 (S.D.N.Y. 2012) as the landmark decision which finally provided court approval of Technology Assisted Review and meant that lawyers could proceed with the use of the technology without worrying about being guinea pigs in a test case. In this decision, Judge Peck criticized the common notion of manual review being the gold standard, and endorsed studies showing TAR would be more accurate. Da Silva recommend seven iterative rounds of training for predictive software and specifically cited FRCP 26(b) as a basis for TAR to be used because of the need for costs to be proportional to the amounts at stake and to encourage transparency.

Showing that TAR is making the discovery big time, in Gabriel Technologies Corporation v. Qualcomm Inc. the U.S. District Court for the Southern District of California, awarded $2.8 million in costs related to use of predictive coding in the review of 12 million documents. No. 09-cv-1992, 2013 WL 410103 (S.D. Cal. Feb. 1, 2013).

The Primer discusses cases addressing the issue of whether or not a court will grant a motion requiring a party to use TAR. A federal court in the Northern District of Illinois required the parties to meet and confer on the issue; the Southern District of California denied a request to order the use of TAR where a party had already conducted regular keyword searching; the Delaware Chancery Court declined to order the use of TAR by a party that only had a low volume of documents to search; but the Central District of California did order the use of TAR where the parties has spent months arguing over which search terms to use.

The District of Nevada declined to allow a party to use TAR on its own initiative when it tried to do so late in the discovery process without consulting opposing counsel, but the Middle District of Tennessee allowed a change to TAR in the middle of the discovery process. The Northern District of Indiana denied a motion to make a party redo its review with TAR after it had already performed keyword searches, and let it proceed by applying TAR to a set culled down through the use of the keyword searches, but the District of Nevada stated that it was not a best practice to use TAR on documents found with the use of traditional search terms. Judge Peck his Rio Tinto decision did allow this approach when it was specified in an initial protocol, as did the Middle District of Tennessee.

Judge Peck recommended the disclosure of training seed sets in Da Silva, as did the Northern District of Indiana - but that court states that Federal Rules of Civil Procedure prevent it from requiring the disclosure of such sets. The Northern District of Indiana rejected the approach of Western District of Louisiana that in using experts to judge the quality of seed sets.

A Virginia State Circuit Court approved a TAR protocol aiming at 75% recall (or getting 3/4 of the total responsive documents.), as did the U.S. District Court for the Central District of California.

The Primer also notes that the Federal Trade Commission and the Antitrust Division of the Department of Justice have approved the use of TAR.

Important S.D.N.Y. Ruling on Ordering the Use of TAR

This month, Magistrate Judge Andrew Peck of the S.D.N.Y. issued an important decision on the use of Technology Assisted Review, or predictive coding. In Hyles v. New York City, 10 Civ. 3119 (S.D.N.Y.), Judge Peck denied the plaintiff's application to force the defendant to use TAR even though he acknowledged that, "TAR is cheaper, more efficient and superior to keyword searching". As previously noted on this blog, Judge Peck issued Da Silva Moore v. Publicis Groupe & MSL Group, one of the first judicial decisions to approve the use of TAR. However in this recent decision, Judge Peck rules that Sedona Principle 6, which states that parties are best able to judge for themselves which technologies are best for the preservation and production of ESI, should supersede the obvious advantages of TAR.

It should be noted that this is an employment discrimination suit. The Review was staged, beginning first with only 9 custodians, and then if necessary, only expanding to an additional 6. Presumably very large amounts of ESI are not involved.

Judge Peck closed his decision by noting that it's possible as TAR becomes more widely used, it will be unreasonable for a party to decline to use it in the electronic discovery process.