Northen District of Illinois: Random Sampling of Null Set Reasonable and Proportional
On Tuesday Judge Iain D. Johnston issued a ESI protocol order in City of Rockford v. Mallinckrodt ARD, Inc., No. 17 CV 50107; No. 18 CV 379, 2018 U.S. Dist. LEXIS 133332 (N.D. Ill. Aug. 7, 2018). This is a breach of contract case concerning Acthar, a drug used to treat multiple sclerosis. The case involves millions of documents, and the Defendants will produce most of them. The parties in this case agreed to use keyword searching, and conferred on specific search terms. Hit reports were exchanged, and the producing party permitted the requesting party to review a statistical sample to confirm the percentage of responsive documents that the keyword searches returned.
The Defendants want to resolve a dispute about whether or not responsive documents were included in the search results with a meet and confer, and the Plaintiffs proposed reviewing a random sample of the 'null set' (the documents not included in the search results) to determine how many documents in it are relevant - a process called elusion.
Judge Johnston acknowledged the growing consensus that TAR is superior to keyword searching, but also cited several sources that asserted that keyword searching could more effectively identify responsive documents. "Indeed, when being pitched on the virtues of TAR, some parties, attorneys and courts may feel —albeit without any evidence-based reason — as though they are being sold a monorail." Id. at *12.
The Court concluded that sampling a null set is reasonable means under Fed. R. Civ. P. 26(g) to verify that discovery is complete and correct. The review of the null set in this case confirmed that the review process was defensible. "The defendants provided that less than 5 percent of the documents reviewed each defendant's 'null set' sample(the documents not hit upon in their search efforts) were deemed potentially responsive upon additional QC. This would mean that 95 percent were not relevant and therefore appropriately cast aside. Certainly that should be considered a reasonable result under the rules." Id. at *15.
Judge Johnston also concluded that sampling a null set was proportional and not burdensome under Fed. R. Civ. P. 26(b)(1). He noted the extraordinary circumstances of this case:
1. Acthar was prescribed to infants at the cost of tens of thousands of dollars.
2. The case has attracted national media attention.
3. The amount in controversy consisted of a potential sum of damages which made this a 'bet the company' case.
4. The Defendants have access to the vast majority of relevant documents.
5. ESI plays a key role in the case.
6. The burden of random sampling of the null set is outweighed by the benefit of reasonable, but not perfect production.
The Court ruled that a confidence level of 95% and a margin of error of 2% was valid to calculate the size of the random sample set. This means one can be confident, at the 95% level, that out of every 100 simple random samples taken from a given population, 95 of them will contain the true percentage, within a margin of error of plus or minus 2 %.