Rand Report on Electronic Discovery Costs

In 2012, the Rand Institute on Civil Justice published, Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery. The report consisted of case studies of eight different corporations, and review of additional electronic discovery literature. The study is well known for finding that 73% of the cost of producing documents is taken up by Review, with Collection accounting for 8%, and Processing 19%. Outside counsel was found to charge for 70% of the overall costs, vendors 26%, and only 4% of the cost came from internal activities. In 45 of 49 cases outside counsel conducted Review; while in 42 of 44 cases vendors handled processing, and in 31 of 42 cases the vendors performed collection. [The median cost of production in the cases in the study was $1.8 million.]

The total cost of production per GB was less than $40,000 per GB for most of the cases, and less than $20,000 in a third of the cases. Collection costs ranged as a percentage of totals costs from about 1% up to 25%; Processing costs were from 5% to as much as 45%. In all of the cases Review took up at least 35% and up to 90%. The average cost of Collection per GB was $940. The average cost of Processing was $2931 per GB. The average cost of Review was $13636 per GB.

The report includes a chart listing document review rates per hour and the amount charged for an hour of a reviewer's work from nine different surveys. Between 50 to 100 documents could be reviewed per hour, and billing rates varied between $50 and $300 per hour. [some Review conducted offshore was cheaper].

A table showing reported rates of review suggests that 50 documents per hour is standard pace.

A study of a review team using a predictive coding application showed variances between 66% - 81% in the Recall rate, and 68& - 81% in Precision rates.

What goes into the NIST list?

A list of the software or operating systems from which the hash values of system files are collected for the master list (or the National Software Reference Library) prepared by the National Institute of Standards and Technology can be found here. This is a tab delimited file. You can import it into Excel, and see that more than 21,000 operating systems are included. Everything from Mac OS X Snow Leopard; Linux Ubuntu and Kubuntu 9.10; to Sierra's Space Quest IV Roger Wilco and the Time Rippers and Virgin Games' Monty Pythons Flying Circus.

CEDS Exam - How to Gut It Out

Tonight, after a lot of studying, I passed the CEDS exam - a 145 multiple choice test which allows you to quality as a Certified Electronic Discovery Specialist, after also submitting some other background qualifications to the Association of Certified Electronic Specialists. Here are my random thoughts on the test - right after passing and, admittedly, after doing quite a bit of celebrating immediately afterwards.

First off when taking the test, be sure to take into consideration whether or not you'll have enough time to actually complete the exam. Only 125 of the 145 questions are actually counted towards your final score - and it's a basic multiple choice exam where you have four answers to choose from. Each question not only tests your ability to absorb the materials on electronic discovery that ACEDS provides you with [the 91 page CEDS Examination Preparation Manual; the CEDS Prep Seminar conducted by Helen Bergman Moure, a former partner at K&L Gates; and the ACEDS e-Discovery Essentials On-line course] but also on other information which is not referenced in these materials. I could not point you to a single reference which would cover all of the information that is required to answer each question on the exam - which is part of what makes the test a legitimate means of testing an individuals' expertise in the electronic discovery field. However my preparation for the exam solely focused on the three sources listed above and I was able to pass.

The main shortcoming in my preparation for the test was my failure to take into account the amount of time I would need to complete all 145 questions. Test takers are given four hours to complete the exam, and I assumed this would be more than enough time to consider each question, enter an answer, and still have plenty of time to review questions I was unsure about. [The computer based exam gives you the option to mark questions you are uncertain about for review latter on. You need to click directly in check box - not just over the 'label' for the check box, but directly in the box, whenever you want to mark a question that you want to review later on. When you have actually answered each question, the online exam will give you the option to review all of your answers - there will be a list showing the number of each question and the letter corresponding to your answer. If you checked an answer for further review there will be an asterisk next to the number of the question.] To my surprise, I actually struggled to answer all of the questions in the allotted time. While I did ultimately manage to do so, with 15 minutes to spare to review the questions I marked for further review, I initially set off at a rate which would not have allowed me to answer all of the questions in four hours. I had to pick up the rate at which I was answering the questions. 145 divided by 4 is 36.25, so you want to answer at least that many questions at the very least each hour - and really ideally quite a bit more. If after the first hour, you've only answered 40 questions, you may [as I did] wonder whether or not you'll have enough time to review each question and submit an answer. Not doing so would be a disaster on any multiple choice exam. The timing of entering answers is something I did not give a lot of thought to before taking the exam, and I would highly recommend that you come into the exam already knowing how many answers you want to complete in each hour before taking the test. The test provides a comment section for test takes to enter their opinions on any one question. If you have time to make entries in this field you can mark me down as some very impressed with your test taking skills and electronic discovery skills!

The format of the CEDS exam requires the test takers to review a factual scenario consisting of 2 to 10 sentences (by my estimate) and then apply their knowledge of electronic discovery to that scenario. Essentially the test is a mix of reading comprehension - and the application of electronic discovery to the scenarios presented in the 'reading comprehension' part. However the knowledge of electronic discovery that you need is not necessarily included in the three separate materials that you are provided with. In the test that I took [and I assume that each test taker does not see the same questions in the same order and ACEDS draws from a large pool of questions that are not all presented each test taker] the last question covered the collection of meta data from social media sites. If someone is aware of where this subject is covered in the materials provided by ACEDS they would need to point it out to me - and I think I did a very good job of mastering those materials.

My recollection of the substance of the questions of the test that I took just a few hours ago is not very good, because I had to devote all of my energy to answering each question. I had no time to make 'mental notes' of the questions that I found especially challenging, or which entirely departed from the study materials ACEDS provided. However I can note that there seemed to be an especially heavy emphasis on litigation holds, and the concept of not reasonably accessible data. There did not seem to be a lot of questions on cross border discovery. The

EU Data Protection Directive was specifically referenced, but ABA Rule 1 - comment 8; Directive 95/46/EC [a reference to EU Data Protection Directive itself]; ABA Resolution 103; Council Regulation Number 1206/2001; Working Document 1/2009 [but perhaps the Data Protection Working Group was???]; 28 U.S.C. 1782; Article 5 of the Hague Convention; Model Rules of Professional Conduct 1.1; 1.6; 3.3 and 3.4; and Sedona Principle 6 were not referenced. [and all of these 'rules' are specifically referenced in the ACEDS review materials]. I also did not come across a reference in any questions to a specific court decision [Zubulake - which not not mentioned in the ACEDS materials or Qualcomm v. Broadcom or any other case referenced in the ACEDS materials].

After you finish answering each question, and reviewing any questions you marked for review, you can choose the option to submit your answers. After you do this, you should immediately review a report which indicates the percentage of questions you answered correctly (66% is a passing grade), whether or not you passed, and a breakdown of the percentage of questions you got correct in various subject matter areas covered by the test. This is the breakdown that I received:

9 Information Management and Litigation Readiness (Pre Trigger)

6 Project Planning (Post Trigger Events)

8 Hold Implementation

11 Collection Planning and Implementation

6 Data Processing

5 Data Culling

11 Review Planning

11 Document Review

8 Data Production

6 Project Management

14 Legal Framework and Obligation

4 International Discovery

5 Ethics

17 Technology

4 Budgeting

. . . although I assume each CEDS test is different and the questions in each subject matter are can vary.

In any event, my main tip is to gut this sucker out, and make damn sure you submit an answer for each question. You can't ruminate over individual questions. You need to read quickly and make sure you give answers at a very rapid rate. Getting bogged down in pondering the questions can be fatal. This is really a problem because the test in structured in such a way, as to require the taker to enter the 'best' answer. You should see many questions on the test for which there is more than one conceivable answer. You need to have an instinct on what context ACEDS is asking specific questions in, and not get distracted by making considerations which essentially begin by thinking "Well, if they thought about this problem this way . . . ".

Finally if you are lucky enough to pass the test, for fuck's sake go out an celebrate like a mad dog! This is a grueling, nerve wracking test, and you deserve to unwind. Order a cool cocktail named something like Figgy Stardust and kick back! You did it!