Email Threading and Near Dupe Identification in Relativity

Here's a brief demo on how to run email threading and near duplicate analyses together in Relativity to group email conversations together, and identify near duplicates among the attachments and individual files.

Note that email threading and near duplicate operation should never be run in an analytics set at the same time with a repeated content and/or a language identification operation.

1. The first step is to create a saved search containing all of the files to be analyzed. The threading analysis is run on the extracted text field. Documents with more than 30 MB of extracted text will be excluded from the analysis.

2. At Indexing &Analytics . . . Analytics Profiles, click 'New Analytics Profile'.

3. Map the email header fields - (this step can actually be omitted, and the analysis can be run on the extracted text only.)

4. In the Email metadata fields section select the Parent Document ID. The Parent Document ID has to be set for attachments but can be left blank for the parent emails. The Parent Document ID for the attachments should be the same as the Control Number for the parent email. As noted in the Tip of the Night for May 30, 2018, the Conversation ID field should not be selected in most situations.

5. Next go to Indexing & Analytics, and select Structured Analytics Set. Click the New Structured Analytics Set button.

6. Set the saved search as the set to analyze and check off 'Email threading' and 'Textual near duplicate identification' as the operations.

7. In the Email Threading section under 'Select profile for field mappings' select from the drop down menu the profile you just created. In this example, select the radial button to use email header fields. On the right the Destination Email Thread Group and Destination Email Duplicate ID fields will be used to show emails from the same group and duplicate emails.

8. In the Textual Near Duplicate Identification section, we set the minimum similarity percentage to a value between 80 and 100. This indicates how similar a near dupe must be to a principal document. Setting to a value lower than 90 will slow down the operation. You should choose to ignore numbers to keep the operation running quickly. The Destination Textual Near Duplicate Group field shows the related items for your near duplicate groups.

9. Next save the set and then click 'Run Structured Analysis' on the console to the right. You'll be given the option to either run the operation on all documents or only new documents added to the set.

The document set will be synced; file sizes will be calculated; and new documents will be added.

10. When the operation is completed, you'll see a summary indicating how many documents and emails were analyzed, and how many emails were 'inclusive', or have unique content not shown in other emails.

11. Back on the console to the right, you can click, 'View Email Threading Summary' to get an even better overview.

12. New fields are generated with the Prefix set for the Analytics Set. In this view we can see that the field named, 'SA S003 Textual Near Duplicate Group' shows a principal document whose control ID is listed in this field for two other documents which have between 98-99% similarity to the principal document. The field named, 'SA S03: Textual Near Duplicate Principal' will be listed 'No' for the two documents which match the principal document.

13. The email threads are grouped with sequential numbers in the 'SA S03:: Email Duplicate ID' field (showing how many total threads are in the set); a common ID in the 'SA S::Email Thread Group' field; and IDs unique for each thread in the individual groups in the 'SA S03:: Email Threading ID'.

Analytics Profile: Don't Select Outlook Conversation ID

The Tip of the Night for September 14, 2016 discussed how Outlook Conversation IDs can be used to group emails from the same thread together in a Relativity workspace. While working with Conversation IDs can be very helpful, note that Relativity recommends that the Conversation ID field not be selected when setting up an Analytics profile for email threading.

If an admin sets up a profile with the Conversation ID field selected, Relativity will group together emails with the same IDs even if their extracted text is different. The Analytics Guide for version 9.5 states that, "The Conversation ID is not typically recommended, as inaccurate Conversation ID data can harm email threading results (and even when it works correctly, it can join together very different email conversations). It is recommended to run email threading without mapping this field."

Be sure that your admin doesn't the very understandable mistake of selecting a field for each of the email meta data fields in a draft analytics profile:

Creating an Analytics Index

Okay Relativity admins, here's how we create an Analytics Index.

Analytics is based on the mathematical review of documents and does not rely on word lists or the word order in the documents. Documents are mapped in a spatial index. Analytics indexes are used for categorization sets; clustering; concept searching; keyword expansion; and similar document detection.

Under Indexing & Analytics, select Analytics Index. Click on 'New Analytics Index'. Be sure to click open the 'Advanced Settings' section.

1. Enter a name. Existing indexes can be selected in a drop down menu in the index field.

2. Order - this will simply set the order of the index in the drop down menu.

3. Index Type - Classification uses Support Vector Machine to show how terms are related to categories. Conceptual - uses Latent Semantic Indexing to discover concepts between documents.

4. Searchable Set - these are the documents to be searched - they should be in an optimized saved search. Documents greater than 30 MB should not be included.

5. Training Set - the document set from which the Analytics engine will learn word relationships to create the concept index. It will only use documents whose extracted text field is less than 2 MB. Excel spreadsheets; system files; zip files; image files; and documents with poor OCR or no text should be excluded. Metadata fields should not be included. It should be a saved search with only extracted text.

6. Optimized Training Set - if set to 'Yes' only conceptually useful documents will be used in the training set to generate the index. Systems files; documents composed mainly of numbers; and very short or very long documents will automatically removed.

Email signatures and footers can also be excluded from the training set.

After saving the index, click 'Populate Index: Full'. This will add the documents in the training set to the ready to index list.

After the population process is completed, when new documents are added to be reviewed, an incremental build can be performed. The next step is to build the index.

Lastly queries should be enabled . . .

. . . and then the Index should be activated. The index will then be searchable.