Document reconciliation

When you align documents, you merely view a variety of annotation sets associated with a common signal. You can also reconcile the conflicts among these sets, and produce a new document reflecting your decisions.

Note: this mechanism currently reflects the original MAT 2.0 approach to reconciliation, which only supports spans. We are currently working on a new, more general reconciliation approach, related to our approaches to comparison and scoring, which will replace the reconciliation strategy described here, and will, crucially, be based on annotation pairs, rather than annotation segments (see below). This new approach will be available in an upcoming version of MAT.

• The reconciliation process
• Selecting documents for reconciliation
• Basic voting
• Menu options
  
• Processing and viewing the votes
• Saving and reloading reconciliation documents
• Exporting
  

The reconciliation process

In order to understand the reconciliation UI, it's important to understand something first about the reconciliation process itself.

The basis of the reconciliation process is the reconciliation document. A reconciliation document is a rich annotated document which is created from a set of input annotated documents. This document contains all the annotations from the input annotated documents, but also contains information about which regions are consistently annotated, and which regions are not. The regions are represented by SEGMENT annotations, and the conflicts within the inconsistent regions are represented by special annotations, specific to reconciliation, called VOTEs. Each VOTE is associated with exactly one SEGMENT, and specifies a subset of the annotations in that segment, as well as the annotator or annotators associated with the relevant SEGMENTs in the input documents. So each VOTE knows which pattern of annotations it corresponds to, and who produced that pattern.

The process of reconciliation is, essentially, a process of voting. In the UI, the reviewer is presented with the VOTEs for each SEGMENT, and makes her choice, or proposes a new VOTE. The MAT server processes those votes, and decides whether the SEGMENT has been reconciled.

This basic model is remarkably powerful and flexible, and raises the possibility of some very sophisticated reconciliation processes. In MAT 2.0, only the most basic of these processes is enabled: there's only one voter, and that voter's decisions are absolute. However, due to the nature of the process, the impact of those votes can't be completely reflected immediately in the UI; the votes must still be processed in order to reconcile the SEGMENT.

The final step of reconciliation is export. If a reconciliation document has been completely reconciled, you can export it, via the UI, to a new window or document which is a normal, rich annotated document, which reflects all the decisions which have been made during the reconciliation process.

One of the most appealing properties of this process is that it's stateful. The way reconciliation is implemented, the SEGMENT and VOTE annotations bear all of the information about the state of reconciliation: which SEGMENTs are consistent, which are inconsistent, which have been reconciled, and who's voted for which alternative. You can save these reconciliation documents and reload them later. And because the changes are made operational by exporting rather than by modifying the reconciliation document, you always have a record of the voting process.

Selecting documents for reconciliation

In the MAT desktop, select "File" in the top menubar, and then "Reconcile files...". You'll be presented with a dialog:

This dialog works very similarly to the document comparison dialog; you must select a task, and then either select loaded documents or load documents via the "Load document..." option. These documents will be listed in the dialog, and once there's at least two, the "Reconcile" button will be enabled. Like document comparison, the signals of all the documents must match.

Here's an example of a populated dialog:

Unlike document comparison, your only option is to remove the document, via the "X" to the left of each listed document; the visual position of the documents is controlled by the reconciliation UI.

You can also invoke document reconciliation in a document alignment window, by selecting "Menu" -> "Reconcile these documents". In this case, the documents being aligned will be the ones selected for reconciliation.

Basic voting

When you press "Reconcile", the MAT server will create a new reconciliation document and present it to you:

In this document, agreed-upon regions are shown with "normal" presentation of the annotations, behind the annotated text. The first occurrence here of "World Health Organization" appears in such a region; we've already reconciled this region via the process we're about to describe. Any regions in the document which are common to all the input documents will also be displayed in this way.

Those regions which are in conflict are shown with yellowish (e.g., "caution") square brackets delimiting the region, with the pattern for each VOTE represented on a separate line below the text. You can hover your mouse over these conflicting annotation bars and get information about the annotation, just like you can with normally presented annotations. You can also view information about the SEGMENT annotations by hovering your mouse over the left or right bracket for that conflicting region. Unreconciled SEGMENTs will have the status "human gold"; reconciled SEGMENTs will have the status "reconciled".

To vote on a conflicting region, click on any portion of the region: the text, the conflicting annotations, or the delimiting brackets. The rest of the document will be grayed out, and your voting options will be presented in the "Voting" tab in the detail pane at the bottom:

You can vote for one of the existing patterns, or vote for the region having no annotations, or propose a new annotation pattern for the segment. If you select "new annotation pattern", you'll be able to hand annotate in that region only:

Once you're satisfied with your choice, select "Done". The document will no longer be grayed out, a check mark will appear to the right of the closing bracket, and you can select another region for voting. If you've proposed a new annotation pattern, that pattern will be displayed behind the text.

Until you process your votes, you can return to any region and change your vote, clear your vote, or edit your new annotation pattern.

Menu options

The "Reconciliation" item in the top menubar is activated when viewing a reconciliation document:

In addition, "File -> Save..." is active, and the "View" menu is changed to reflect the items which are relevant to reconciliation:

The meanings of these elements is as follows:

• "Process votes", "Show processed votes" and "Show all segment boundaries" are described here.
• "File -> Save.." is described here.
• "Export" and "Export to file" are described here.
• "Show character offsets" behaves as it does in other windows; it includes the start and end offsets of an annotation in the information provided when you hover over it.

The "Auto-advance" option modifies voting interaction. When you enable "Auto-advance", the reconciliation UI will automatically advance you to the next region that doesn't have a vote registered, and scroll that region into view. The "Auto-advance" option will have a check mark after it if it's enabled. If you enable "Auto-advance" when no region is selected, it will select the first region. You can disable "Auto-advance" at any time from the menu; at that point, once you're completed with the current region under review, you'll be returned to the normal, unguided mode of voting.

Processing and viewing the votes

When you review a region and make a vote, you're just voting; the region is not yet reconciled. As long as you're not reviewing a region, you can tell the MAT server to apply your votes by selecting "Process votes" from the menu. Once the votes in a document are processed, those regions which are now reconciled can no longer be reviewed; the votes cannot be changed or cleared. These regions now look like any other reconciled region.

For instance let's say we reviewed the first conflicting segment, containing the word "South", in the example above, and then selected "Process votes". The result looks like this:

Like "World Health Organization"  in the first line, "South" in the second line now appears with the selected annotation pattern behind the text, instead of below, and the yellowish brackets have been removed.

That doesn't mean the voting information has been discarded, though; it's just not visible by default. You can control the display with the "Show processed votes" and "Show all segment boundaries" options in the menu. Here's the same document display with both these options enabled:

You can see that all reconciled regions are delimited with green ("ok") square brackets, and that the region containing "South" that we voted on still has a check mark to the right of the closing bracket, indicating that it was reconciled by voting rather than by being already consistent when the reconciliation document was created. You can also see that the first occurrence of "World Health Organization" was also agreed upon via reconciliation.

Saving and reloading reconciliation documents

As long as you're not reviewing a region, you can save the reconciliation document for reloading later. Before you're prompted for a save location, the UI will process the votes in your document.

These documents are always saved as MAT JSON documents; the only difference between these documents and normal MAT JSON documents is that these documents contain VOTE annotations, and are marked as reconciliation documents in their document metadata.

To load a saved reconciliation document, use "File" -> "Open file..." in the MAT desktop. In the dialog, select "(reconciliation)" as the workflow/mode; "mat-json" will be automatically selected as the document type, and can't be changed:

Exporting

As long as you're not reviewing a region, you can export the reconciliation document to a file or window. Before the document is exported, the UI will  process the votes in your document. If not all the regions are reconciled, the UI will tell you this and refuse to export.

By default, the UI will export to a new read-only document window, which you can save If you want. If you want to export to a file, enable the "Export to file" option in the menu before you select "Export", and you'll be able to select an output annotation format, and then a save location.