Bloomberg Legal Webinar on Advanced Techniques for Managing Digital Discovery

On November 16, 2016 I attended a webinar hosted by Bloomberg Legal entitled, Advanced Techniques for Managing Digital Discovery. You can watch a free online demand recording of the webinar here. Magistrate Judge David Waxse (D. Kan.); Susan Leffert, a partner with Mayer Brown LLP; and Mark Noel of Catalyst where the panelists for the webinar.

The presentation was interesting primarily because of the insights of Judge Waxse, a sitting United States District Court Judge. Judge Waxse began by noting that while less than 1% of civil cases go to trial, attorneys are not necessarily inclined to settle matters very quickly. In his words attorneys have candidly admitted to him that they postpone the settlement of cases because they need to bill hours for the practical demands of life such as putting away money for the children's college education. The judge mentioned the District of Kansas's Guidelines for Cases Involving Electronically Stored Information [ESI] and specifically referenced Section 25 addressing discovery on party's preservation and collection actions. The judge pointed out that these guidelines state such discovery about discovery should not be routine and may violate the FRCP 26(g) ban on unreasonable or burdensome discovery. He recommended that parties come to an agreement on discovery before issuing interrogatories and preparing requests for production.

Judge Waxse observations are that while the use of Technology Assisted Review is growing, it is still not broadly used. People have still not grasped that TAR is cheaper, faster and more accurate. The lawyers that come before Judge Waxse still claim that they are comfortable until humans have reviewed the ESI. He bemoaned the idea that having staff attorneys review one document every 60 seconds was an effective means of review. He advises parties to share the services of single electronic discovery vendor.

Judge Waxse said that he wasn't sure of the general trend in the use of FRCP 37(e) sanctions since the December 2015 amendments, but he did notice that judges are not regarding this as the only basis for sanctions. They still assert the inherent power to issue sanctions. One example is the decision of Judge Francis (author of several noted electronic discovery decisions) in Cat3, LLC v. Black Lineage, No. 14-civ-5511 (S.D.N.Y.).

Judge Waxse pointed out GN Netcom, Inc. v. Plantronics, Inc., No. 12-1318-LPS (D. Del. July 12, 2016) as an especially scary case. Punitive damages in the amount of $3 million imposed by the court for spoliation. This case illustrates Judge Waxse's maxim that the most trouble and biggest sanctions come from attempting to cover up electronic discovery misdeeds, rather than the misdeeds themselves.

The group also touted Mayer Brown's white paper, Preparing for Electronic Discovery Litigation .

TAR for Smart People Outline

More than a year ago now I posted an outline of the first three chapters of John Tredennick's 'TAR for Smart People'. This is a continuation of that outline using the first edition. An updated edition has been posted at: http://www.catalystsecure.com/tarforsmartpeople .

4. TAR 2.0 Capabilities Allow Use in Even More E-Discovery Tasks

• Document Review o Classification - responsive / non-responsive o Protection - privileged / trade secrets o Knowledge Generation - specific issues / deposition witnesses

• Metrics o Recall – the percentage of relevant documents actually recalled. o Precision – the percentage of retrieved documents that are actually relevant.

• Classification Tasks o FRCP and Sedona Principles – e-discovery is limited by principles of reasonableness and proportionality. o 80% recall is a common TAR target. Gold standard of linear review can’t do better than this and costs more. o Recall usually gets more attention than precision.

• Protection Tasks o For confidential information, 100% recall is necessary. o Use TAR; keyword searching; and human review – stack techniques. TAR systematic errors; human random.

• Knowledge Generation Tasks o Precision is the most important metric. o Prioritize document population by issue. o Concentrate most interesting documents for review first. o TAR imperfectly concentrates interesting documents near top of responsiveness ranking.

5. Measuring Recall for E-Discovery Review: An Introduction to Recall Sampling

• Review high level of recall (75%) after only reviewing small percentage of documents (5%). Discard pile include so few relevant documents that more review not economically justified.

• Hypo: 1M document production. 1% relevant – 10K documents. o Using Sampling to Estimate Richness  Statistical sampling to estimate richness.  Randomly selected subset  Concepts • 1. Point Estimate – most likely value for a population characteristic. • 2. Confidence Interval – ranges of values around point estimate that we believe contains true value. e.g. 8K to 12K. • 3. Margin of Error – max. by which a point value might deviate from true value. • 4. Confidence Level – chance confidence interval will include true value. • 5. Sample size – higher confidence level more docs must be reviewed.  Determine sample size with Raosoft calculator. Inputs: • Document set size 1,000,000 • Confidence level • Margin of Error 4% • RESULT: 600 documents. o Initial Sampling Results  If 6 relevant documents found; estimate 1% richness. o Determining the Exact Confidence Interval  Binomial Calculator to determine confidence interval

We multiply these decimal values against the total number of documents in our collection (1,000,000) to calculate our exact confidence interval. In this case, it runs from 3,700 to 21,600. We believe there are 10,000 relevant documents in our collection (our point estimate) but it could be as high as 21,600 (or as low as 3,700). Let’s move on to our review.

• The Review - if we find 7,500 relevant docs in 50K may have to review 950K docs to get 2,500 more – not reasonable. But what if 21,600 relevant documents? 35% recall not sufficient.

• Sampling Discard pile – again 600 sample size. E.g. 2 relevant documents.

• Use binomial calculator again – find could be as many as 11,400 relevant documents in 950K docs left. Only getting 7502/18900 – only 40%.

• Narrow margin of error again and use Raosoft calculator – e.g. 1% margin of error – must review 9,508 documents.

• Find 31 documents relevant documents. estimate there are 3,097 relevant documents in the discard pile, about the same as before (950,000*(31/9508)). Range could be 2090 to 4370 documents. Using these values for our exact confidence interval, the range goes from 63% (7,500/11,870) to 78% (7,500/9,590).

Sedona TAR Case Law Primer

This past August, the Sedona Conference published its TAR Case Law Primer, which is available for download here. It mentions Da Silva Moore v. Publicis Group, 287 F.R.D. 182 (S.D.N.Y. 2012) as the landmark decision which finally provided court approval of Technology Assisted Review and meant that lawyers could proceed with the use of the technology without worrying about being guinea pigs in a test case. In this decision, Judge Peck criticized the common notion of manual review being the gold standard, and endorsed studies showing TAR would be more accurate. Da Silva recommend seven iterative rounds of training for predictive software and specifically cited FRCP 26(b) as a basis for TAR to be used because of the need for costs to be proportional to the amounts at stake and to encourage transparency.

Showing that TAR is making the discovery big time, in Gabriel Technologies Corporation v. Qualcomm Inc. the U.S. District Court for the Southern District of California, awarded $2.8 million in costs related to use of predictive coding in the review of 12 million documents. No. 09-cv-1992, 2013 WL 410103 (S.D. Cal. Feb. 1, 2013).

The Primer discusses cases addressing the issue of whether or not a court will grant a motion requiring a party to use TAR. A federal court in the Northern District of Illinois required the parties to meet and confer on the issue; the Southern District of California denied a request to order the use of TAR where a party had already conducted regular keyword searching; the Delaware Chancery Court declined to order the use of TAR by a party that only had a low volume of documents to search; but the Central District of California did order the use of TAR where the parties has spent months arguing over which search terms to use.

The District of Nevada declined to allow a party to use TAR on its own initiative when it tried to do so late in the discovery process without consulting opposing counsel, but the Middle District of Tennessee allowed a change to TAR in the middle of the discovery process. The Northern District of Indiana denied a motion to make a party redo its review with TAR after it had already performed keyword searches, and let it proceed by applying TAR to a set culled down through the use of the keyword searches, but the District of Nevada stated that it was not a best practice to use TAR on documents found with the use of traditional search terms. Judge Peck his Rio Tinto decision did allow this approach when it was specified in an initial protocol, as did the Middle District of Tennessee.

Judge Peck recommended the disclosure of training seed sets in Da Silva, as did the Northern District of Indiana - but that court states that Federal Rules of Civil Procedure prevent it from requiring the disclosure of such sets. The Northern District of Indiana rejected the approach of Western District of Louisiana that in using experts to judge the quality of seed sets.

A Virginia State Circuit Court approved a TAR protocol aiming at 75% recall (or getting 3/4 of the total responsive documents.), as did the U.S. District Court for the Central District of California.

The Primer also notes that the Federal Trade Commission and the Antitrust Division of the Department of Justice have approved the use of TAR.