The Electronic Discovery Reference Model

To best explain how Advanced eDiscovery can help us solve these challenges, let us use the Electronic Discovery Reference Model (EDRM), an open source model explaining the discovery process shown in Figure 14-2. Think of the EDRM as the detailed subprocess for the discovery box in Figure 14-1 covering the litigation process.

Before we even get into eDiscovery, you want to manage your organization’s data volume through excellent information protection and governance practices. Implementing processes like managing retention and deletion of your content reduces risk in the discovery process. Retention ensures people do not delete content prematurely, which would result in fines. Deletion reduces risk because if you can find it and it is relevant, you must produce it in discovery. It would not be enjoyable to lose a case based on information that you could have legally deleted. Deletion also reduces the volume of discoverable information, which reduces costs in the review process. We covered information governance in Chapter 8.

The next step in the EDRM is the processing stage. In the processing stage, we try to convert electronically stored information into forms suitable for reviewing and analyzing the data for this case. For example, in Microsoft 365, Advanced eDiscovery breaks apart emails into the message and any attachments. This way, we can search for each of them. Processing also conducts more in-depth indexing of locations on hold to search on all the metadata. Finally, Microsoft 365 Advanced eDiscovery will attempt to process and reindex any partially indexed or unindexed items and provides a report for anything unsuccessful in the Processing tab.

Now let us look at the preservation step. You might also call this step a legal hold. In this step, we identify the data related to our case to preserve it, meaning that people cannot delete it or modify it. Advanced eDiscovery allows you to put custodians on hold and customize the specific locations. For example, you could select their email, OneDrive, Teams of which they are a member, Yammer private messages, and SharePoint sites. You can also place locations on hold separately from a custodian. This option is useful if you release a custodian from a hold but want to keep some locations on hold. You can also use Advanced eDiscovery features to manage custodial hold notifications.

Next is the collection step. This step is where we search for data that might be relevant to our case. Typically, your legal team will tell you what data to search for and how. Advanced eDiscovery uses the same search options as content search and Core eDiscovery. However, we will not run into as many issues with unindexed and partially indexed items because of the previous step’s processing, resulting in a more thorough search.

Throughout the search and review process, your team may perform early case analysis to ensure you are on the right track with your discovery. This analysis finds critical patterns, topics, people, and discussion items in the data. Advanced eDiscovery supports early case analysis in almost every step of the EDRM process, even though the model shows it as a separate step. This analysis can include statistics about your holds and searches. It also includes charts and graphs showing content metadata, using artificial intelligence to identify themes in your dataset, and detailed reports and statistics. This analysis can help you refine search queries, identify other custodians, and more.

Next, you will reach a point where the discovery process is complete and you are ready to export your data. Some organizations may export the content and go directly to producing it to the opposing counsel. Other organizations may export Microsoft 365 data into another eDiscovery tool, where they combine data from all the electronic content sources in their company. Advanced eDiscovery can handle both scenarios and has many options for how to configure the export. It can even create load files with tagging information that you can use for your import into the other system. The export process is the last step in the EDR model involving Microsoft 365.

The next step is to produce the discovery to the opposing counsel. Advanced eDiscovery does not help with this step. There are specialized tools that the legal team uses for production.

The final step in the EDRM process is to produce your evidence in court. Again, there are specialized tools for this step, and you would not use Microsoft 365 for this purpose.

Now that you understand the overall process for discovery, the rest of the chapter will review complete details for each step and how Advanced eDiscovery works. With that, let us start by discussing how we create a new case.

Case Settings

When you create a case and choose yes for whether you want to configure additional settings for this case, the system will bring you to the case settings page in Figure 14-5. You could also navigate here by clicking the Settings tab in the case. On the case settings page, you have three sections. First is Case information, where you can edit general information about the case, such as case number, case name, description, status, and whether the case is open or closed, and you can also delete the case. The next section is Access & permissions. Here is where you manage who can access the case and what they can do. It is where you will go to configure those permissions and add or remove case members.

Lastly is Search & analytics. The search and analytics section is where you configure the processing options for the case. It is where you can set analytics settings such as identifying duplicate information. You can also thread emails and Microsoft Teams conversations. There are settings for machine learning and artificial intelligence capabilities such as theming and OCR. You can also specify text that you would like to ignore in the case.

Click Case information. Here you will see the unique case ID. You are also able to edit the case name, case number, and case description. You can also see that the case type is Advanced eDiscovery. You can see whether the case is active or closed and the date the case was created. If you would like to close the case, this is where you would come to perform that action. You could also choose to delete the case entirely. Click Settings to exit.

The next section in settings is Access & permissions. Click Select to access the options. Here you will see individual members of the case and role groups that you have added to the case, shown in Figure 14-6. For example, if you have designated role groups based on geography for Advanced eDiscovery, instead of adding individual members to the case, you could add that geography’s role group to grant everyone in that geography access. See the “Licensing and Permissions” section earlier in this chapter to learn more about role groups and Advanced eDiscovery permissions.

If you would like to change case permissions, click Add under either the Members or Role groups heading. Here, you will see a list of all users or compliance role groups in your organization. If you would like to narrow down the list, you can use the search functionality to show a subset of role groups. Check the box next to a name and click Add. Once you have added your users and your role groups, click Exit.

The next section on the settings page is Search & analytics. On this page, we configure our settings related to analytics and optical character recognition. The system will use these settings to process the case data when you add a search to a review set. I highly recommend that you configure these settings before you perform any other actions in your case. Unfortunately, there is not a way to configure these settings globally for all cases. Let us go through each one of these settings and talk about what they do.

First is the checkbox in Figure 14-7 for near-duplicate/email threading. This checkbox allows artificial intelligence (AI) in Advanced eDiscovery to process what emails, Microsoft Teams chats and conversations, and Yammer messages are a part of a thread or conversation. Threading helps to decrease the time required for manual review. The idea is that it is easier for someone to review a conversation thread than individual messages. It is also easier to review all near-duplicates at one time, highlighting the documents’ differences.

For Teams conversations in a channel and Yammer messages in a group, a thread is a post and its replies. For Teams chats, it groups threads by the time stamp. The system groups discussions that occurred around the same time. Emails are grouped in conversations by threads. Near-duplicate detection parses every document with text. It compares every document to determine whether the similarity is greater than the set threshold. If it is, the system groups the documents. You set the threshold using the document and email similarity threshold percentage under the checkbox.

Next are themes. Themes use AI to help you find topics related to your dataset, of which you were not aware. For example, a theme might be a project, customer, or keyword. You can then modify your search to include discovered topics. You can set the max number of themes that you would like the AI to find. Another choice is whether to include numbers as themes. For example, you might have the fiscal year of 2020 that the system would represent as a theme number. Next, you can choose whether to adjust the maximum number of themes dynamically. If you check this box, the artificial intelligence module will decide the number of appropriate themes for your dataset. If unchecked, it will find the number of themes from the Max number of themes box.

There is a checkbox to automatically create a query named “For Review” after running analytics on a review set. This query filters out duplicate items from the review set, so you only review unique items. You will need to run analytics on the review set, covered in the “Review Set Settings” section of this chapter, for this view to work.

Then, you have options to set a min number of words and a max number of words. These fields are related to the near-duplicate and email threading. They ensure that near-duplicate and email threading analyses are performed only on documents that have at least the minimum number of words and, at most, the maximum number of words.

Finally, we can ignore text. This feature is useful for things like an email disclaimer footer, which can throw off your searches and near-duplicate detection. If you add this disclaimer to ignore a text, the system will not use it in search and analysis in your dataset.

The last set of options is for optical character recognition (OCR). OCR allows us to extract text from images. To make this feature performant, we can set a maximum image size that should be analyzed. Large images can increase the processing time and slow down your results. Images with sizes above the set size will not have their text extracted.

Similarly, you could set whether you want a low or a high accuracy for the text extraction. This accuracy setting will affect your processing times. Finally, you can select your timeout for OCR, meaning that if the system cannot extract the text from an image in this number of seconds, it will stop and move on. If this happens, the system will list that image as a processing error that you can remediate later. We will discuss processing and error remediation in a later section of this chapter.

Once you are happy with your settings, click Save. Now that we have configured all the case settings, we can either put people and content on hold or perform a search if you do not need the hold functionality.

Add Custodians to the Case

To add one or a few custodians, click Add custodians. The first screen will ask you to choose custodians. You need to type at least three letters of the custodian’s name for the search to start. It will then bring up the custodian names that match the search. You can click their name to add them to the hold. Click Next when there are no more custodians to add.

On the next screen, shown in Figure 14-9, you will need to choose the custodian locations. By default, the system will check both the Exchange and OneDrive locations, which will add the Exchange mailboxes and OneDrive accounts to the hold. Go ahead and check and uncheck the boxes as needed to make the hold. Click Next.

On the next screen, you can set additional locations for each custodian, shown in Figure 14-10. We associate these locations with the custodian. When we release a custodian from a hold, it will include associated locations. You can add additional locations for each custodian, such as Exchange mail accounts, including user mailboxes, group email accounts, or Microsoft Teams chats. You can also add SharePoint sites, Microsoft Teams, and Yammer groups.

Let us start by adding additional Exchange email locations. Click Choose users, groups, or teams. This click will bring you to a screen to choose which user or group mailboxes to add to the hold. Type a minimum of three characters to get the list of mailboxes that match that search. Check the boxes next to the group mailboxes or individual users that you would like to add to the hold. When done, click Choose and then Add.

Next, let us add a SharePoint site to the custodian hold. Click Choose sites and then Choose sites again. Here, the system will bring you to a screen to paste the entire URL for the SharePoint site. If you want to add an additional OneDrive account, you could do that as well by pasting the URL of that person’s OneDrive here. Check the boxes next to the sites; click Choose and then Done.

To add a Microsoft Team, click Choose Teams and then Choose Teams again. Here, it will show you a list of all the Teams where the custodian is a member or owner, shown in Figure 14-11. This filter is helpful because you do not have to use a different process to find Teams where the custodian interacts. Check the boxes next to any Microsoft Teams you would like to add to the hold. Click Choose and then Add. Please note that if you add a custodian to a new Team, it will not automatically place that Team on hold. You will need to update the hold locations on an ongoing basis to account for new team memberships.

Now let us add a Yammer location. Click Choose Yammer and Choose Yammer again. Again, the system will show you a list of all the Yammer group memberships for that individual user. Check the boxes next to the groups you want to add; click Choose and then Done.

When you finish adding locations, click Save. The screen will summarize the additional locations for the users, and you can click Next. The next page is where you place the hold. Uncheck the boxes if you are not ready to put the person on hold to save the configuration. Click Complete. Please note the system will put new content items, such as emails, within a location on hold. But as a reminder, for entirely new locations, such as a new Microsoft Team, the system will not place that location on hold automatically.

Custodian Hold Details

Once you place a custodian on hold, it will appear in the custodian list on the Data sources tab. Click the custodian’s name to view the hold details. You will see summary information about the custodian, such as their title, manager, location, and more, on the details screen. You can also double-check that the hold went through correctly by seeing hold status = true. Scroll down to see a summary of the data sources, like the number of mailboxes and sites in the hold. If you would like to edit the custodian’s data sources, click Edit and add or remove data source locations.

At the top of the details window, you can see options: Update the index, View custodian activity, or Release the hold. First, we will cover Update the index. Click Update the index. This button will remediate any added partially indexed items since you last ran the last index. This click will also update the case information and information about the data sources. Please do not update the index more than once a day as it is a long-running process.

To view custodian activity, make sure the person viewing the activity has View-Only Audit Logs or Audit Logs permissions. The audit log also must be enabled in your tenant. Click View custodian activity to see a screen showing you the custodian’s audit log activities, shown in Figure 14-12. Here you can filter options by date, IP address, activity, or item name. You can also show results for all activities in the audit log, or you can show specific audit log activities. You can also export the audit log. If you are interested in this feature, please see Chapter 16.

The last action on the custodian details page is Release custodians. When you click this button, it will give you a warning that asks if you are sure you want to remove the custodian from the hold. You can choose yes or no. If you release this custodian from the case, all holds placed on the custodian’s data sources will be removed. Any holds placed on the custodian’s data sources in other cases will still apply. You can also release custodians from their holds in bulk. To do this action, select multiple custodians and click the Release source button that appears on the Custodians toolbar when you select multiple people. You can also update the index for those custodians or view their activities.

Add Custodians in Bulk

Now let us look at how to add custodians in bulk. From the Data sources tab, click Import custodians. On the page that appears, click Download a blank template to download the CSV file you need to complete, shown in Figure 14-13.

Add the custodial information to the CSV file and save it on your local computer. See the next section for information about the properties in the CSV file. On the Data sources tab, click Import custodians again. On the page that appears, click Browse and upload your CSV file.

After the CSV file is uploaded, the system creates a BulkAddCustodian job, which you can find on the Jobs tab. The job validates the custodians and their corresponding data sources and then adds them to the Custodians tab on the case’s Sources page.

Currently, you can only import custodians who are in Azure Active Directory (AAD). The system validates and finds custodians using the UPN value in the Custodian ContactEmail column in the CSV file. If it cannot validate a custodian, it will show them as Not validated in the Validated column on the Data sources ➤ Custodians tab.

Put Data Locations on Hold

If you would like to put content on hold that is not associated with a custodian, you can use Data locations. From within the case, click the Data sources tab and then Data locations. Click Add data locations.

This click will bring you to a pop-out window that asks you to choose locations, shown in Figure 14-14. For example, you can select specific users, groups, or teams based on Exchange mailboxes to include in the hold. Or you can add URLs to SharePoint sites or OneDrive accounts. Once you are happy with your location, click Next. The final screen asks you to place the hold. You can uncheck the box if you want to save the configuration but not activate the hold. Click Submit.

The hold will now appear in the data locations list. Click the name of the hold to view the hold details. Here you can see the information about the hold, update the index, or release the hold.

Create a Query-Based Hold

Query-based holds are a bit different than the other types of holds. For one, you create them from a separate tab. Second, only they put the items on hold that match the search query rather than the entire location.

To create a query-based hold, click the Hold tab in an Advanced eDiscovery case, shown in Figure 14-15. Here you will see a list of your existing query-based holds and the date they were last modified. You can also export a list of the holds. To make a new hold, click Create.

A window will appear and ask you to name and optionally describe the hold. Click Next. The next screen asks you to choose locations. These locations will scope your search query. For example, if you later choose a date range when you create your query, it will only find the items within that date range for the locations you choose here – let us say Exchange. Choosing only Exchange returns mailbox items that fall within the date range .

These locations are the same as in content search. We review them in depth in Chapter 12, in the “Create a Content Search” section. As a reminder, the first location group corresponds to Exchange mailboxes. You can click Choose users, groups, or teams to select individual or group mailboxes. If you choose an individual mailbox, it captures that person’s emails, other mailbox items, Teams chats, Yammer private messages, and To Do tasks. Selecting a group mailbox covers any group emails and other mailbox items and Teams conversations if the mailbox is associated with a Team. However, it does not include Yammer group messages.

The next group of locations covers site locations, such as SharePoint and OneDrive accounts. You can add these items by using the site’s URL. You can also put an entire Yammer network on hold by adding the URL of your Yammer network.

Let us pause for a moment and discuss Yammer group holds. Suppose you want only to put individual Yammer groups on hold and not the entire network. You can do that by associating a Yammer group with a custodian as described in the “Add Custodians to the Case” section. If you want to place your Yammer network on hold permanently, I recommend creating a separate query-based hold only for that purpose, so you can easily understand how to disable the hold later.

The last section in the Choose locations screen is for Exchange public folders. You can either place all public folders on hold or none of them. My advice is like Yammer networks. If you want to put public folders on hold, do that in a separate query-based hold. When you finish selecting locations, click Next.

The Create query screen is where you create the search conditions for what content you place on hold. Figure 14-16 shows an example of a query. This query looks for Teams chat messages (keyword =kind:im) created between February 1, 2020, and June 30, 2020, sent or received by Adele Vance. In the location step, I choose Adele’s Exchange mailbox as the location. Again, for more details on creating a search query, see Chapter 12. Click Next when you are finished. On the next screen, you can review your settings for the hold. Click Create this hold when you are happy.

Once you add a hold, you can see it in the list on the Hold tab. If you would like to see more details about the hold, double-click its name. The details screen will pop up. Here you can edit or delete the hold. You can also see hold details such as the locations, statistics, last modified information, and any errors .

Additional Hold Details

What happens to the content we have put on hold? Let us start with Exchange Online. When we put content on hold in Exchange Online, if someone edits a message or attachment, the modified version will stay in the mailbox, and the custodian can view it. The original message is moved to the Versions subfolder and is kept intact in an immutable state. If someone deletes an email message, it is moved to the Deletions subfolder in the Recoverable Items partition. If the item is also under a retention policy, then the hold will always override the retention policy, meaning that we cannot delete content prematurely if it is on hold, and then once the hold is released, the retention policy would then take over as to whether the item is kept or deleted.

When a custodian’s email content is on hold, if a custodian tries to delete an item in the Deletions folder, the item will be moved into the Purges subfolder. The system preserves it according to the hold settings.

The system purges a message that is on targeted hold. When a targeted hold has been placed on specific messages or attachments, rather than on the entire mailbox, a cleanup process runs roughly every week, which acts on items that exceed the single item recovery limit. The cleanup process moves the items that matched the targeted query criteria to the Discovery Hold folder for ongoing preservation. The system also moves items with any index errors to Discovery Hold for continued preservation. The cleanup process then expunges the rest of the items that do not match the criteria and have no index errors.

If your organization uses Exchange Online archiving, which we covered in Chapter 8, the system moves mailbox items to an Archive folder after two years by default. When you place a custodian’s Exchange mailbox on hold, the hold takes effect for this archive, the same as for the primary mailbox. If the custodian tries to delete archive content on hold, the system preserves the content in the Recoverable Items folder of their Exchange Online mailbox. Microsoft 365 eDiscovery searches include the Exchange Online archive, so you do not have to include it as a secondary source location. Every time you search a custodian’s mailbox, it also searches the archive.

The last thing I want to mention in this section is that it is possible to use PowerShell to script eDiscovery holds. You can also use the Microsoft Graph API to create and manage custodian holds. Microsoft will release more APIs in the future, so please check the documentation for the latest information.

Set Up Custodian Communications

To use custodian communications, open an Advanced eDiscovery case. Click the Communications tab. Here you will see a list of any previous communications that you sent to custodians, shown in Figure 14-17. You can see the name of the communication, the status (draft or published), the date it was last modified, the number of custodians associated with the communication, and the number of people who acknowledged the notice. Additionally, you can export a list of communications.

To create a new communication, click the New communication button. In the wizard that appears, name your communication – for example, initial notification. You will also choose your issuing officer. Any member of the case or an eDiscovery administrator assigned to the case can be the issuing officer. Click Next.

You will define the portal content on the next screen, shown in Figure 14-18. The portal content is your hold notice that you send to custodians. You will usually get the language for this notice from your legal department. You can insert dynamic fields, such as display name, the acknowledgment link for the custodians, the portal link, the issuing officer’s email address, and the issuing date. You can also use formatting options to use different fonts and settings like you would when editing a Word document. These dynamic fields will update with the relevant information when you send the notice to custodians. Click Next when done.

There are several types of notices, explained in Table 14-2 above. First, we configure the required issuance notification. The issuance notification tells the custodian that they are on hold. The email sender is the issuing officer we designated in the first step. You can add additional people in the Cc or Bcc line. Enter a subject and the body. Again, you have full formatting capabilities as well as dynamic content fields that you can add. Click Save.

Next is the reissue notification. The system will automatically trigger this notification if the portal content changes. Now we configure the release notification. The system automatically sends this notification when you release a custodian from hold. It has the same options as previous notices.

Once you have configured these notices, click Next.

Next, you can set optional notifications, shown in Figure 14-20. These optional notifications remind the custodian to acknowledge the notice and send an escalation to the custodian’s manager if they have not acknowledged it. The system starts the escalation workflow when it runs out of reminders to send to the custodian.

First, let us configure the reminder notification. On the configuration screen, you can toggle the reminder on or off. You can set the reminder interval, which is the number of days between each reminder, and the count, which is the number of reminders. For example, suppose you set the reminder interval for three days and the count to three. In that case, it will send the first reminder email three days after the initial notice if the custodian does not acknowledge the notice. Then, it will wait three days and send another reminder for three total cycles. You have the usual notice configuration fields for the subject and body. Click Save.

Now let us configure the escalation notice. The system sends the escalation to the person’s manager as defined in Azure Active Directory. Click Edit next to Escalation. The options are the same as reminders. Toggle escalations on or off and set the escalation interval and the number of escalations. Configure the text for the subject and body.

Click Next when you finish with the optional notifications. The next page asks you to Choose the custodians you want to notify, shown in Figure 14-21. You can choose to inform all custodians or only some custodians. Click Select custodians and check the boxes next to the names you want to add. Click Next.

The last page asks you to review the settings for your communication. When you are happy, click Submit. The system will immediately send the notification to all the selected custodians.

The custodians will receive the issuance email asking them to acknowledge the hold, shown in Figure 14-22 above. Once they click the acknowledgment, it will bring them to a page that says, “Thank you. Your acknowledgment has been recorded.” The system records their acknowledgment, and you can see it counted on the Communications tab and in Data sources ➤ Custodians.

You can also view details about each communication on the Communications tab. Click the name of the notice to open the details pane. Here you can see how many people have acknowledged the hold notice and how many are awaiting acknowledgment. You can also see the list of custodians and their responses. You can also edit or delete the communication from the details pane or resend the notice.

Finally, you can download the complete text and configuration of the communication for your records.

Bulk Remediation

The remediation process will start to process the selected data to prepare it for the download process. This process will take several minutes. Once the data is ready, you can click Next ➤ Download files.

To download the data, you need to use a tool called AzCopy. Before you start, you will need to install it. You can find AzCopy here: https://erica.news/AzCopy81. Something important to note is that there is AzCopy version 8.1 or AzCopy version 10. For eDiscovery, you need to use AzCopy version 8.1 because it integrates with the Advanced eDiscovery UI.

To install AzCopy, download it to your Downloads folder on your computer or similar. Double-click the executable. This click will bring you to an install screen. Click Next and go through the install process.

When installed, start AzCopy by searching for it in Windows. Click the name to open and run it. AzCopy will open in a window that looks like a command prompt. It will be blue and text only.

Go back to Advanced eDiscovery to the screen where we left off. Fill out the destination path for the download, shown in Figure 14-25. This path is where you want your downloaded remediation files to go. By default, files will be downloaded to your Downloads folder on your local machine in a folder called Errors. Filling out the destination path for download is going to populate the next field automatically.

Under the heading Copy this command and run it in a Windows command prompt, click Copy to clipboard, go back to AzCopy, paste it, and press Enter. This step is why we want to use AzCopy 8.1. With AzCopy version 10, you cannot easily copy and paste that command. With 8.1, you can paste the tool’s information, and you do not have to know how to write PowerShell.

Then, the files will download to the destination you specified, and you can go to Downloads ➤ Errors on your computer to view the files.

Now you need to remediate each item. The exact steps will depend on the error. For example, if the error is a password-protected file, like when someone puts a personal password on their Excel sheet, you will need to get the person’s password or run a password cracking tool to break into the file. If it was an unrecognized file type, you need to save it as a file type that eDiscovery can process. Or, if it is a text-based file, you could copy and paste it into something like Notepad, so at least the text is discoverable to get around an unknown file type. Those are just some examples of how to remediate files, but do what makes sense.

Once you remediate the files, save them back into the Errors folder or create a new folder in your Downloads area called Remediated, and then save the content with the same file name and folder structure. The file name and folder structure needs to match the download exactly .

Go back to eDiscovery and click Next on the upload file screen. You will put the path to the remediated files’ location in the first field, shown in Figure 14-26. Again, copy the resulting command to the clipboard, go back to AzCopy, paste it, and press Enter to upload your remediated files. Then Advanced eDiscovery will process the files. This process may take several minutes, depending on the amount of data. When the processing completes, the remediated files will now be available in the search index.

These fields are all related to Advanced eDiscovery and do not change the files themselves .

Single File Error Remediation

The second way to remediate files is through single file error remediation . We will first need to add search results to a review set, which we will go over in the next two sections of this chapter. Once you have added your data to a review set, you can filter your view or search the review set to show only files with processing errors. To do that, open your review set. Click New query and create a name for the query. Add the condition called “processing status,” and then choose the errors that interest you. I like to use the “equals any of” conditions to check the boxes for multiple error types. Leave “Success” unchecked because that option corresponds to successfully processed items. Save the query. Now, you can click that saved query anytime to view files with processing errors. Click one of the files to view its metadata to help decide if you want to remediate it.

Above the file metadata , you will see a red box that says “Processing failed: [error type]”, shown in Figure 14-27. Here I have two choices. First, I can ignore the file, which removes it from the bulk remediation list, and I will not see it again in this view. Second, I can click Remediation. When I click that, it allows me to upload a remediated file. First, download the individual file by clicking the downward-facing arrow with the line under it, located in the document pane’s upper-right corner. I added a box around it in Figure 14-27. You then perform the file remediation and click the Remediation button to upload the new file. When the upload is complete, the system reprocesses it and lets you know when processing is complete.

Another usual review set query related to processing remediation is a query to view remediated files. To set this up, click New query and name the query. Add the condition “was remediated.” Choose equals true. This query shows a list of all the files that you have remediated through either the single item remediation or bulk remediation method.

After performing file remediation using either option, you now have a fully searchable set of files for either your custodial or search data. The files are fully indexed and ready for the full discovery process. In the next section, we will examine how to perform an eDiscovery search to gather files for review, analysis, and export .

How to Run a Search

The eDiscovery case manager often works jointly with the legal department to get a list of relevant custodians and search queries to run against the content source locations. In that case, you simply need to run the query that they have given you. If you do not have a pre-built query, you can use trial and error, informed by the search statistics, to refine your search.

To create a search, go to the Searches tab in the Advanced eDiscovery case, shown in Figure 14-29 above. You can see a list of your searches, any query text, and the last time someone ran the search. You can also see the search estimate status, the preview status, and the add to the review set process.

To create a new search, click New search. The first screen asks for a name and a description. Click Next. If you would like to run the search against custodian data locations, select a custodian here. You also have the option to choose all custodians. Selecting a custodian will run a search against all the data sources mapped to that custodian. You do not have to add a custodian. This step is optional. Click Next.

Now you are asked to add non-custodian locations to the search. Again, you can select some or all locations. This step is also optional. Click Next.

Next, the wizard asks if you would like to add additional locations to the search, shown in Figure 14-30 above. This step allows you to search locations that are unrelated to the custodian or non-custodian data sources. You can choose to search all Exchange email mailboxes and SharePoint sites or specific locations. You can also search all Exchange public folders. Click Next.

The next screen asks for the search criteria. This search criterion is the same as in content search. You can add keywords, a keyword list, KQL, or no keywords. You can also add conditions to the search, such as a date range or participants in an email. Click Next. Review your search settings and click Submit. There are some limits that affect search. Microsoft increases these limits frequently, so please see the latest information here: http://erica.news/AeDLimits.

Search Statistics

Searches are a long-running process. You can view the status of your search from the Searches tab. For example, this page’s information will tell you if the search is still in progress or has been successful or has failed. You can see whether the search preview status was successful or is still processing. Then, when you add the search to a review set, you can see that status as well.

It can be useful to view the search details to learn more about your results. This information in the search details area can help you ensure the search aligns with your expectations. Click the name of a search, and the search details screen will appear. On that display, you will be able to view the description of the search. You will also see the query and the custodian, non-custodian, and additional locations. It also shows the last time you ran the search with an option to rerun the search. Finally, the screen displays when you created the search, the last modified date, and the estimated search results.

Under the estimate, you will see the estimated number and size of items that match the search criteria. You will see the number and size of partially indexed items and the number of mailboxes and searched sites. Look at this estimate and make sure that those numbers seem reasonable for what you expected.

Click Statistics in the upper-left corner of the search details screen. This click brings you to a page that provides information about the search. It includes a summary, top locations, and detailed statistics about the search query. First, in the summary view, you can see the search results broken down by a location type, such as Exchange or SharePoint, and then see the following information displayed for each location type. It will show you the number of locations with items that matched the search query, the total number of items from each location type that matched the search criteria, and the overall size of items from each location type that matched the search criteria. You can also download a report of this data.

Next, for Type, choose Top locations. In this view, shown in Figure 14-31 above, you see the individual content locations with the most items that matched the search query. For each content location, it displays the following information. First, it shows the name of the location. This name could be the email address for mailboxes or the URL for SharePoint sites. Next, it shows the location type, which is Exchange or SharePoint. Then, the number of items that matched the search criteria for that location is displayed. Lastly, the total size of all items that met the search criteria for that location is shown. You can click Download to get a report on this view. This report can be useful because if your search is hitting any search limitations, you can use it to see what locations have the most significant size in your search. You can use that information to break the locations out into multiple searches.

Lastly, for Type, choose Queries. You can see the detailed search statistics for each component of the search query in the Queries view. For example, if you used the keyword list in the search query, you could see enhanced statistics in the Queries view, showing how many items matched each keyword or keyword phrase. This view helps you identify which parts of the query are the most and least significant. You can then use that information to modify your query as needed.

The system displays the following information in the Queries view – first, the location type, which is SharePoint or Exchange. The next column will display one of the following values – primary or keyword. If the column’s value is primary, the statistics for the entire search query are displayed. If the value is a keyword, it shows the query component’s statistics. For example, if you use a keyword list, then the statistics for each of the keywords are displayed.

Here are some other things to know about the statistics displayed in the Queries view. When you search for all content in mailboxes, by not specifying any keywords, the actual query’s size is (size >= 0) so that it returns all items. When you search SharePoint and OneDrive sites, it adds the two following components to each search query:

• NOT IsExternalContent:1: This term excludes any content from the on-premises SharePoint organization.

• NOT isOneNotePage:1: This term excludes all OneNote files because these are duplicates of any content that matches the search query.

The number of locations shows the number of content locations that matched the search query for the part or condition displayed in the row. Note that archive mailboxes are counted as a separate location if they contain items that match the search query:

• Items: Displays the total number of things that matched the search criteria for the part or condition in the row.

• Size: Shows the total number of items that matched the search query for the part or condition in this row.

Click the back arrow to return to the search details screen.

Next, let us click Preview in the search details window. This option allows us to see a sample of our search results. You can click an item to view the file’s contents in full fidelity or the file metadata. Click the back arrow to return to the search details pane.

The next option is to add results to a review set. We will review this option in the next section. The Edit option allows you to modify the search query. You can change anything except the name of the search. Click More to see further options. Here you can delete or copy the search. You can also sample the search.

The sample option allows you to add a statistically significant sample of the search to a review set for a more detailed analysis. When you click Sample, the system will bring you to a wizard that asks you to select the sampling parameters. You can specify a confidence level and confidence interval or take a random sample of the data and specify a percentage for how much data to include. Select which option you would like and click Next.

You can then choose the review set where you would like to add the data. If you have existing review sets, they will show up as options. You can also create a new review set. Then you can choose the collection options. We will discuss these options in the next section about creating a review set. When you finish your selections, click Next and then Submit. Now you can use the options we discuss in the next section to analyze the sample.

Now let us go over the options for adding a search or sample to a review set.

Create a Review Set

To create a new empty review set, click the Review sets tab in an Advanced eDiscovery case. Click Add review set. In the window that appears, create a name and optional description.

To create a review set by adding search results, visit the Searches tab. Click the search name. In the search details pane that appears, shown in Figure 14-32 above, click Add results to review set. You then have a few options to configure.

First, choose the review set where you want to put the search results. If you have existing review sets, you can see them here and select one. You also have the option to create a new review set.

When you finish, click Add. Adding data to a review set from a search is a long-running process. You can track the progress of this process in the Jobs tab. Please note that there are limits to the amount of data you can add to a case and a review set. Microsoft frequently increases these limits, so see the latest information here: http://erica.news/AeDLimits.

Review Set Settings

Like the case settings, you may want to configure review set settings before proceeding with the data review. These settings allow you to create additional analytics and reports, add non–Office 365 data to the case, configure your review tags, and use machine learning to predict document relevance.

Now let us look at the details for the review set settings. First, let us start with analytics. Before you can view analytics for the review set, you need to run an additional process. Click Run analytics for the review set. A warning will pop up that says, “This will take some time. Are you sure?” That is because running analytics is a long-running job that can take from several minutes to several hours.

Running analytics also reindexes the data with the near-duplicates considered. The system will rerun the search index, taking the email threads into account. It will also look for document themes, which we will talk about in just a second, and then it will run another index after it finds the themes to include that in the search results. After you run analytics, a query called “For Review” will appear. You can use this to only review unique information in your review set.

Once you run the analytics, you can click View report under the Analytics heading in the review set settings. The report will give you additional data about your review set, shown in Figure 14-33. For example, it will tell you the target population, including the number of errors, emails, attachments, and documents. It will also show you documents, excluding attachments, and how many are unique vs. exact duplicates. Other useful information includes statistics about the emails and how many are inclusive vs. other types.

For attachments, it will show you how many are duplicate attachments vs. unique attachments. You will also see a graph with the number of documents by file type and then the source location documents. You can also see the documents aggregated by the different processes, including unique documents, tagged documents, and so on. The point of the analytics screen is to allow you to do some pre-case analysis.

Now’s let us talk about the summary report, shown in Figure 14-34 above. Go back to the settings screen. Under the Summary report heading, click View summary. The review set summary will give you some additional summary data about your review set. It will show you how many load sets are in the case. We will talk about load sets in the next paragraph. The statistics show you things like the total extracted documents, total unique items, duplicate items, total processed items, the total number of failed extracted documents, review set indexed documents, and review set not indexed documents. Again, this information lets you see an overview of what content is in your review set to help with pre-case analysis.

Next, let us cover load sets. Go back to the settings screen. Under Load sets setting, click Manage load sets. A load set is a collection of documents that you added to a review set at a single time from Microsoft 365 or ingested into a review set from non–Office 365 sources. For example, every time you run a search and click Add results to review set, that would be considered one load set. If you then later added another search, the system would view that as a second load set. This area will show you all the load sets added to the review set. If you like, you can download a report, and that report will show you the same information you see on the screen. Each load set has a unique load ID, and it will show you some information about the source and from where it came.

Suppose I double-click one of these load sets. I can see even more detailed statistics about the load set, such as total errors to skipped items, data processing information, and search index information. It is a good idea to look at this report to ensure that there are no errors that you need to remediate and that everything looks reasonable.

I can also select two load sets and compare them, shown in Figure 14-35 above. This view allows you to understand what files are added or removed from load set to load set. This information can be useful if you have a long-running case and you reran the search to get new search results and then added them to the review set. It might be useful to see how many new files you added need to be reviewed and processed.

Now let us manage tags. The tags are what you will use to tag the documents during the actual review process. These tags are completely customizable based on your business needs. Go back to the settings screen and under the Tags heading, click Manage tags.

Here you will see a blank screen where we can add groupings and tags. First, click Add section and select Add smart tag group. This area is where some of the cool artificial intelligence features of Advanced eDiscovery come into play. When we include a smart tag, we will eventually have several models from which to choose. As of the writing of this book, the available one is the attorney-client privilege smart tag.

Suppose you have the attorney-client privilege detection setting on. Remember that we configured this tool at the beginning of the chapter. In that case, the system will run a model on your data and flag documents that are likely to be privileged based on the content and compare participants against the provided attorney list. This feature does not replace the need for privilege review. This feature helps to get you started on the privilege review and saves you time. Select that if you would like to add that smart tag. If you do, it will create a section called attorney-client privilege, and the selection options are negative and positive.

Now let us add more tags to the review set. Click Add section. Here you will be able to add a section title and click Save. Then, from there, you can click the ellipsis that is to the right of the description for the section title, and you have the option to add a button if you need to have mutually exclusive tags. You can add a checkbox if people can select more than one option. You can move the section up so it is above other sections, or you can delete the section. You can also see a live preview on the right side of the screen that shows you these changes and additions in real time. Add your sections and tags as needed. An example of a complete tagging panel is shown in Figure 14-36 above.

If you go back to the review set settings screen, we have gone over all the options except for the non–Office 365 data import, which we will go over in the next section, and the relevance model we will go through near the end of this chapter.

How to Add Non–Office 365 Data into a Case

You may want to add non–Office 365 data to a review set for a couple of reasons. First, if you have old PST files that you are keeping for discovery purposes, your organization may sometimes store those in a file share or other locations. You can add those to your discovery by uploading the PST files to your case.

Second, this process also works for any documents or other files that you might have sitting around. We will add the non–Office 365 data to custodian mailboxes and associate them with a custodian. They need to have an active online mailbox in your organization for it to work.

To add non–Office 365 data to a case, open an Advanced eDiscovery case. Click the Review sets tab and click a review set to open it. Click Manage review set, look for the Non–Office 365 data heading, and click View uploads. This click brings you to the upload screen. This screen shows you any load sets already created for non–Office 365 data, including the creation date, status, number of files, and size of all the files. Click Upload files. A wizard will pop up, shown in Figure 14-37 below.

The first step is to prepare the non–Office 365 data for upload. Organize all the files you will upload into folders, naming each folder as a custodian email address. The format for the folder needs to be [email protected] or alias@domainname without .com. You can put all those folders into a root folder, but the root folder can only contain these custodian folders and not any loose files. Otherwise, this process will error out. As of March 2021 you cannot import more than 300 GB at one time. Microsoft is working to increase these limits, so view the latest information here: http://erica.news/AeDLimits. If you have more than 300 GB of data, please divide the custodian folders into more than one root file and import them in multiple batches.

Once you have prepared all these files, click Next ➤ Upload files. The resulting screen is shown in Figure 14-38 above. We will upload the files using AzCopy.exe. If you have not yet installed it, you can find the instructions for installing AzCopy in the “Processing and Error Remediation” section of this chapter. For the path to the files’ location, put in the path of the root folder containing the custodian folders you prepared in the preceding text. This process will then auto-generate the script you can run in AzCopy. Copy the script to the clipboard, paste it into AzCopy, and press Enter.

Then it will show you the processing of the files. This processing could take several minutes to several hours, depending on the amount of data you are uploading. You cannot pause or cancel this process, and you can see the status in the Jobs tab. Once the non–Office 365 data upload process completes, you will see the success message shown in Figure 14-39 above. The import will appear in the list of non–Office 365 data load sets.

When you search against the review set, you can search the files by the individual custodian location, which is their Exchange mailbox.

Reviewing Data in a Review Set

Now that you have all your data in your review set and have analyzed it, let’s discuss how we will review the data. From your Advanced eDiscovery case, click the Review sets tab and then click the name of your review set. Here, it shows you all the files included in the review set. First, let us go over the options we have for this view, shown in Figure 14-40.

In the Search profile view, run the review set analytics first, described in the section “Review Set Settings.” If you click Search profile view, it will bring you to a new screen that allows you to filter all the charts on the page based on different criteria, shown in Figure 14-41. You can also either add new filters or add new chart widgets to the page.

First, let us look at the widgets. We can see things like the custodian column chart, which shows you the different number of files available for each custodian. We could see the file class pie chart, which will show you how many emails, documents, or attachments are in the review set. We could look at a recipient bar chart, which will show us the number of files per email recipient.

To add a new widget, click New widget ➤ Create custom widget. Create a title, choose a pivot, and then choose the chart type and click Add, shown in Figure 14-42 above. There are also some pre-built charts you can access by clicking New widget ➤ Choose from library.

You can also add filters that will affect the data in all the widgets. Click Filters in the upper right-hand corner. There are 75 filters available for the Search profile view. Go ahead and check any boxes that you would like to see and click OK.

Click the X in the upper right-hand corner of the screen to go back to the view of documents to review content.

Now we will review the content. Click the name of a file to view the contents. Click the diagonal arrows on the upper-right side of the document to make the document full screen.

You can view a file’s metadata by expanding the gray bar above the file viewer, shown in Figure 14-44 above. Here you can see all the metadata for the file, including information added by Advanced eDiscovery. To see a full list of the metadata fields and their meaning, visit this link: http://erica.news/AeDFileProperties.

Document family includes an email and the email’s attachments, shown in Figure 14-46. It can also include an Office document and any embedded files. Viewing content as a document family allows you to see all the files in one another’s context. It also allows you to bulk tag all the files in the family. To do this, select all the documents and click Tag documents.

You can also open the documents in explorer view. This view will open a new window with a list of the documents, shown in Figure 14-47.

The conversation view shows an email thread conversation. The system creates this view when you run analytics on the review set, covered in this chapter’s “Review Set Settings” section.

This view allows you to view the email thread in context, bulk tag items, or open them in explorer to view each file in detail.

The near-duplicate display shows you documents that are almost the same. You define what percentage of the files must be similar in the case settings. The analysis to find duplicates happens when you run analytics on the review set.

The text differences view highlights the differences between files for faster review, shown in Figure 14-48. You can access this view by clicking one of the files in the near-duplicates list. Then click the icon that looks like two rectangles next to each other, located above the document viewer.

As with the other views, you can bulk tag the files and open them in the explorer view.

Lastly, we have the exact duplicate display, shown in Figure 14-49. Using this view, you can review one file and then bulk tag all the copies in one action. To use bulk tagging, click the circle checkbox in the list header. Click Tag documents. The tagging panel will appear, and you can select the documents. Click Done.

Query and Filter Data in a Review Set

When working with documents in a review set, you might need to filter or refine the search to work with the files. To do this, you can use the search and filter functions within the review set.

To create a new search, click New query from within the review set. This click will bring you to the query wizard, where you can add conditions and condition groups, shown in Figure 14-50 above.

Click Add condition group. The condition group evaluates the conditions, with AND or OR statements connecting the group’s conditions. You then add conditions to the group.

Click Add condition. Adding a condition allows you to choose from many different metadata fields in Microsoft 365 and Advanced eDiscovery. There are over 50 fields from which to choose. For example, if we want to search for documents authored by a specific individual or email that they participated in, we will do the following. First, choose the condition “author” and select “equals any of,” and check the box next to the custodian’s name. Then we will create a second condition and choose the “participants” condition. We will select “contains all” and put the individual’s email address in the box below. We will connect these two statements with an “OR” operator. If we are only interested in finding documents and emails within a specific date range, we would add a new condition group, select date, and choose the date range we want to search.

We will connect the two condition groups with the AND operator. Our query says that if the author or the participant on the email is Adele Vance and the date range is within the last year, return the result. We would then save the search query so that we can use it later.

Next, let us talk about filters. Filters are useful when you want to query data temporarily. They are an addition to the search query, meaning you are filtering the data within the query view. You also do not have the flexibility to group conditions with AND and OR statements with a filter. If you click a different search query, you will lose the filter. In comparison, the system saves searches for reuse, and you can build complexity in the query with multiple conditions in groups, using AND and OR.

To create a filter, click Filters ➤ Add condition. This click will bring up the same list of conditions that were available for a search. Check the boxes next to the conditions you want to use and click Add. Add the values to the conditions and click Search.

Remember, back at the beginning when we created the case, we set up the case settings. Well, one of the things that we configured was case themes. Now, you may be wondering, “What are those themes, and how can I use them?” The answer is that the theme has been added to files as a piece of metadata. Therefore, we can use them as a condition in our search and filter queries inside a review set. I show an example of using themes as a filter in Figure 14-51.

First, go ahead and create a new filter. Add a condition and scroll down until you see the condition called themes list. Click Add. Now you are going to see the list of themes that the analytics engine created for your case. You can browse through the list of themes and check the boxes next to ones that seem interesting to explore them further.

The Relevancy Process

At its core, these steps build a machine learning model to determine what documents are relevant to a specific case and, more specifically, to a particular case issue. For each step in this process, there are both mandatory steps and optional settings. I will explain each part of the process’s required actions first and then optional steps.

To access the relevancy model, click Advanced eDiscovery in the Compliance Center. Click Review sets and the name of a review set. Click Manage review set and Show relevance.

Within the relevancy module, there will be new navigation. The gray bars along the top are the overall steps in the process. The settings for each step are on the left navigation with the blue bar highlights.

Prepare

The first process step is to prepare. These are simply the steps we need to do to set up our case and case issues appropriately. Our required steps are to run pre-processing on the container of documents and to set up our case issues. Our optional steps are the process advanced settings, process task status, process results, process custodians, set uploads, set up keywords, and set up advanced keywords.

The first required step is to run pre-processing on our containers. Pre-processing prepares the files, so they are ready to go for our machine learning and machine training process later. To do this, click Process on the first gray bar at the top of the page. Click Setup on the left navigation. Make sure you do not click Setup on the gray bar. On this page, we will see a list of all the containers in your review set. A container corresponds to a load of files into the review set. For example, adding a search to the review set will result in one container. Click the name of a container to highlight it in blue. Then click the Process button at the bottom of the page. Eventually, under the Pre-processing column, the bar will turn orange and show as complete (shown in Figure 14-53), and you can move on.

The other required step in this process is to set up issues. A case issue is the precise legal questions or conflicts you are addressing in the case. The legal department will provide issues for this situation. Click the setup link at the top in the gray bar to access issues. You will need to create issues on this page and assign a person to perform the training. By default, one issue, called Case issue, is created for you. You can click the issue name and then the pencil on the right side of the page just above the issue to edit it, shown in Figure 14-54. To create a new issue, click the plus icon. You can have more than one issue per case and run the train and apply step for each issue. Here you can name the issue, add a description, and add someone to the issue.

To add someone, click their name in the left box. Click the arrow to add them to the right box. Once added, you can designate the person as on, off, or idle. This status defines whether they can train the model. You can only add one person to an issue, and this person is the only one who can perform training. To add a user to an issue, you first need to add them individually to the case. You cannot add them through a role group. If someone does not appear as an option for your issue, this is most likely why.

Those are the two required steps for the prepare phase. Next, we will review the optional steps. The first optional step is process task status. To view this page, click Process on the gray bar at the top. Then click Task status on the left navigation. This page shows you the status of whatever tasks you are performing at that time, such as pre-processing or adding files to a case, which we will go over later. If you are unsure if processing was successful, check this page.

The next optional step is results. Click Results in the left navigation. This page will show you the process summary results, how many files it extracted, how many were processed and processed with errors, erroneous file types, and more. It is good to check this page after you run pre-processing, but before you proceed to training to make sure everything worked and you received the expected results.

The next optional step is to set up keywords. Keywords are useful to highlight certain words in different colors, shown in Figure 14-55. When your attorney trains your model, those words will show in a different color to easily see them. This highlight helps them determine what is relevant and not relevant. To set up keywords, click Setup on the top gray bar. Then click Keywords in the left navigation. To add a keyword, click the plus sign located on the right side of the page above the keyword list. You can put multiple keywords in here, separated by a comma. You can choose from a list of colors, which will be the highlight color for those keywords, and then you can select which issue to which these keywords apply. For example, you could have three keywords show as red and one as blue.

Finally, the last optional step is to set up advanced settings. Click Setup on the top gray bar and then Advanced settings in the left navigation. Here there are two cost parameters. These parameters are the cost per hour for one of your low-cost attorneys to review the files and the number of files they can review per hour. Once you have trained your model, when you are trying to decide what percent relevancy makes sense for your case, these parameters help you. For example, 35% relevancy means reviewing all the files with a 35 and above relevancy score, which would cost $500,000 based on these cost parameters. This information helps you determine where to set that percent relevancy and even if the case is worth it to pursue or settle based on the costs required to go through discovery.

Train and Apply

Now, we will move on to the train and apply phase of the relevancy process. The mandatory steps in train and apply are

• Tracking your progress

• Loading files

• Running an assessment

• Assessment tagging

Optionally you can skip steps or adjust the assessment settings.

First, let us understand how the Track area works. To get to this area, click Track on the gray bar at the top. That click brings you to the Track page, shown in Figure 14-56. The Track page shows you where you are in the train and apply process. You can always come back to this page if you get lost in the user interface. The Track page shows you the next step in this process. Click the issue name to expand it and see the process details.

The first step in train and apply is to load files. Loading files adds the content you have pre-processed to the training process for this issue. From the Track page, click the Add files button. The button is located near the bottom of the page and can be challenging to see. This click takes you to the Loads page that contains a list of all your load files from the review set. As shown in Figure 14-57, please ensure that the Number of files column does not have zero. If it has a zero, you did not do your pre-processing correctly, and you need to go back to that step in the “Prepare” section and run pre-processing again. However, if it shows anything other than zero, you are good to go. Click the row corresponding to your files, under Loads management. Make sure the row is highlighted in blue and click the Add files button. Wait for that process to run. When the operation finishes, it will say, “Files were added to relevance successfully,” and you can click OK.

The next step in the train and apply process is to run the assessment, shown in Figure 14-58. The assessment starts with 500 documents to build your machine teaching model. We need the relevancy model’s statistics to be defensible, meaning that they are in an acceptable range of significance. We will configure these statistics with our first 500 documents, and then we may need to add more assessment documents later to meet our statistical goals.

The number of documents we need to tag for the entire process depends on various things, such as the assessment’s richness. Richness is the number of relevant documents in the sample of 500. If you have a high level of richness, the training will go a lot faster than if you have a low level of richness. If your richness is less than 2%, then this process could take so long that it is just not worth the time. The assessment process’s output will determine the estimated richness; the recall level, which is the percentage of relevant documents that we want to find; and an error margin, which we want to be 10% or less. The error margin is determined based on the richness of the data. A higher richness means it is easier for the system to find relevant documents, which means you will have a higher recall level with a lower error margin.

The system selects a random sample of 500 documents from the files in our review set. The best-case scenario is the system will give us additional sets of 500 files to review until we see enough to provide us with an error margin of 10% or less. When we tag things as relevant or irrelevant, we need to be consistent in our assessment and have a high enough number of documents that the machine learning model can understand what is relevant.

Let us start tagging documents. From the Track page, click Assessment to start the evaluation. That click brings you to a screen. The screen displays the file on the right side of the page, and the tagging panel is on the left. View the document and mark it as either R for relevant or NR for not relevant, or skip if you do not know, shown in Figure 14-59. As a reminder, we defined keywords earlier that the system displays in color to make it easier for them to find these relevant files while they are tagging.

As you tag these documents, follow these tagging best practices:

1. 1.

   Be consistent. If you make errors during training, you need to return to those files to correct them. If you have too many errors, you need to abort and start over. I will cover how you can clear the data and start over later in this section.

    
2. 2.

   Tag the files based on content only. Do not consider metadata, such as the custodian, a date, or any other file metadata. You should not consider a date range or graphical images when tagging the files. Also, do not consider the formatting of the text. Only consider the actual text and words in the document.

    
3. 3.

   Use the skip tagging option only when necessary. The relevancy module does not train based on skipped files, and if you pass on too many, it is hard for the model to tell what is relevant and not relevant.

    

You can browse sample files, for example, if you need to go back and correct a tagging error. To do that, select sample files from the dropdown in the upper-left corner of the page, and this will give you a list of all the files in your current set, shown in Figure 14-60. You can find the document in the scroll menu or put the file number in the file field, which will let you jump directly to that file.

Once you review all 500 of the assessment files, the assessment phase will be complete. In the next step, called the training phase, the model will predict what it thinks is relevant and not relevant, and you will start the actual training by telling the model if it is correct.

To start, go back to the Track tab and click Training. The process is going to be the same as in the assessment phase. You will go through the batches of 40 documents, and you need to decide whether the file is relevant or not relevant. When you finish the sets of 40 files, click calculate in the lower left of the screen, shown in Figure 14-61. The model will determine whether your training is complete, meaning that we have hit our statistical targets for error margin. If we have not hit our targets, the system will give us another set of 40 documents to review for training.

Eventually, you will hit your statistics goals, which will end the training. The amount of training sets you need to do is going to be entirely different for every case. But once the training bar is all orange and up to 40, the next step is to run the batch calculation.

The batch calculation scores documents on how likely they are to be relevant. Files tagged as relevant or not relevant by the expert will receive a score of either 0 or 100. The batch calculation scores the relevancy of the remaining documents that you did not review on a scale of 0–100. You will then run some reports and view some graphs on the cost to review the rest of the items manually, and you can decide how to proceed from there.

Once you have run your batch calculation, you can move on to the Decide tab. The Decide tab gives us the information we need to make a proportional decision about the data. You will see a few charts and sections of data on the page, shown in Figure 14-62. Please keep in mind these numbers are entirely fake and unrealistic because I used a demo environment:

• Review-recall ratio: A comparison of results according to relevance scores in a hypothetically linear review. The recall is estimated given the review set size. Move the slider at the bottom of the graph to change the review percentage, which affects the cost information.

  • Review: The X-axis is the percentage of files to review based on this cutoff.

  • Recall: The y-axis is the percentage of relevant files in the review set.

• Parameters: Cumulative calculated statistics about the review set concerning the file population for the entire case. The cost shows the projected cost of discovery based on the parameters you entered in the prepare ➤ advanced settings step earlier in this section. It shows the cost to review one document and for the overall discovery process.

• Distribution by relevance score: A graph of the relevance score of the files in the load set.

• Details: Data about the file collection (load set), the statistics, parameters, costs, and document families.

These graphs give us a calculation of how much it costs to review a single document. It allows us to decide what percentage of documents we want to review and at what cost. We need to weigh the risk of missing documents based on the case specifics with the cost of reviewing more documents. For example, if it is a litigation case and costs you $100,000 to settle, you will not spend $150,000 to review all the documents. Our machine learning model module that you just built by reviewing those documents will identify what documents are likely relevant and export those for manual review.

Validate and Test

The last step in the relevancy process is to validate our model through testing. We do this test through an illusion sample. We do the test by taking a statistically significant random sample of the unreviewed documents to make sure that we did this process correctly. We want to ensure that the files we think are below our review percentage are, in fact, irrelevant. This test proves to the court that our model is valid.

Start a test by clicking the Test tab in the gray bar at the top of the page. There are two test options:

1. 1.

   Test the rest: Used to validate culling decisions, such as reviewing only files above a specific relevance cutoff score based on the final Advanced eDiscovery results.

    
2. 2.

   Test a slice: Performs testing like the “Test the rest” test but focuses on a segment of the file set specified by Relevance Read %.

    

The purpose of the tests is similar for both types. You have decided to only review documents with a relevance score of 35 or higher in our example. We want to make sure that 35 is the correct number. To accomplish this, we will take a random sample of documents with a score of less than 35 and have an expert review them for relevancy. After the review, we will compare the statistics of the test with those of the overall model.

To run a test, click Test on the gray bar at the top of the page. At the bottom of the page, click the blue button that says New test. A screen, shown in Figure 14-63, appears.

• Enter any name and description for the test.

• Under Parameters, for test type, choose Test the rest or Test a slice.

• For Issue, choose the model you want to test.

• For Load, choose the load set you used when training the model.

• For Read percentage

  • If you chose Test the rest, choose the relevance score under which you want to collect the sample.

  • If you chose Test a slice, choose the relevancy score range you want to test.

• For Test size, 500 is the default number of documents to sample. You can change this number.

Click Start tagging to begin the tagging process for the test – tag in the same way you did in the train and apply phase. When you complete tagging, click Calculate on the bottom left of the screen. Click Test on the gray bar. Click View results next to your test.

The table’s Sample parameters section contains details about the number of files in the sample tagged by the expert and the number of relevant files found in that sample.

The table’s Population parameters section contains the test results, including the review set population of files with a score below the selected cutoff and “The Rest” population of files with a score above the selected cutoff. For each population, it displays the following results:

• Includes files with read % – stated cutoff

• The total number of files

• The estimated number of relevant files

• The estimated richness

• The average review cost of finding another relevant file

In this section, we reviewed the relevancy module. There are a couple more settings and things that you can do with it, but those extra settings confuse people when you are just getting started with it. This walk-through helps you understand how relevancy might be useful in your organization, how you can test it out, and, if you do want to use it, how to get started.

Review the Export Files

In the next chapter, we will look at the Data Investigations tool to run an internal investigation, using functionality like Advanced eDiscovery.