Today’s post is about AI-Driven Batches, a document-batching technology introduced with Reveal 11 that enables the platform to automatically select the best documents for reviewers to assess as they train AI models.

Batching Documents: A Short History

For decades, documents have been grouped into batches for review. In the 1980’s, I oversaw projects where reviewers were assigned stacks of sequentially numbered paper documents. If I remember correctly, the stacks typically were about three inches high. A reviewer would (1) take a document from the top of the stack, (2) grab a coding form (ideally a single-sided sheet of paper; if the coding extended to a second page, review quality tended to decline precipitously), (3) complete the form for that document, (4) put the reviewed document on a pile of completed documents, (5) take the next document from the top of the batch of documents to be reviewed, and (6) continue until there were no more documents in the batch. Then a new batch would appear, and the reviewer would start all over again.

Reviewers could have been given much higher stacks, even entire boxes, to review. The theory behind parceling out smaller stacks was that as reviewers reached the ends of their stacks, they would have a sense of accomplishment. Too high a stack, and they might feel they were stuck in an endless grind.

In the early 1990’s, reviewers began to switch from filling out paper forms to entering on computers. At first, they continued to work with batches of paper documents. Over time, the batches moved online.

The batching process, too, began to change. With documents delivered online instead of as paper copies, they could more readily be sorted before being batched. Documents still could be batched by page number, but they also could be batched by custodian, document date, document type, or any other data associated with them.

With the advent of predictive coding, a new way of batching documents arose. For the first time, it became practical to organize batches based on the likely relevance of documents. Early on, this was a cumbersome and not especially reliable process. With time, as you would expect, the process improved.

Why Use AI-Driven Batches

With AI-Driven Batches, you can let the platform rely on predictive scores to select the best documents to use for training classifiers. This approach can reduce the overall time needed to prioritize documents, meaning quicker access to results along with lower review costs.

(You also can use the AI-Driven Batch feature to create a control set. You then can use that control set to measure the progress of classifier training against target metrics such as precision, recall, and F1 score.)

How to Create AI-Driven Batches

This will open an AI-Driven Batches modal (a modal is a user-interface element that sits on top of the application’s main window):

2. Select the Classifier you want to train. A classifier is a type of machine learning algorithm. It receives input, such as a designation of Responsive made by a reviewer using an AI Tag (for more about AI Tags, see What are AI Tags?). The classifier “learns” from the input. The more examples it is given of responsive documents, the better it can do at predicting how likely it is that each of the remaining documents also is responsive. In this example, I chose the Responsiveness classifier:

3. Select the Tag Profile that includes the AI Tags you want used when coding documents to train the classifier. A Tag Profile is a set of panes containing more or more Tags (for more about Tag Profiles, see What are AI Tags?). Here, I chose the GS Review Tag Profile:

4. Select the AI-Driven queue to create a new AI-Driven Batch. For you to select this option, it must say Available. The option will only say Available if the platform is ready for you to create an AI-Driven Batch for the selected Classifier:

Reveal will create a new batch of documents for you, based on the Classifier and Tag Profile you selected. In that batch, it will place the preset number of documents it thinks best match the classifier. In this example, it will choose the 20 documents it thinks are most likely to be responsive.

Notify you both on the screen and via email:

Start Training the System

Once you have created a batch of documents using AI-Driven Batches, you can start reviewing them, making decisions about their responsiveness that will train the system to do a better job of finding what you are looking for. To do this:

By selecting the assignment, you create a search that returns the documents from that assignment:

2. From that list of documents, select a document to begin your training of the platform:

If you have the right permissions, you can see the platform’s prediction as to the responsiveness of the document. You then enter your own decision, further training the classifier. If you choose Responsive, you tell the system you agree with it. If you choose Non-Responsive, you let the system know that you disagree with its assessment – teaching the system that this document, which it thought was 65% likely to be responsive, really is not:

Learn More

Using Reveal’s AI-Driven Batches feature, you can more efficiently train the platform the find the documents most like the ones you are looking for.

For more information about how Reveal can streamline your review, contact us for a demo.