When you align documents, you merely
view a variety of annotation sets associated with a common signal.
You can also reconcile
the conflicts among these sets, and produce a new document
reflecting your decisions.
Note: this mechanism currently reflects the original MAT
2.0 approach to reconciliation, which only supports spans. We are
currently working on a new, more general reconciliation approach,
related to our approaches to comparison
and scoring, which will
replace the reconciliation strategy described here, and will,
crucially, be based on annotation pairs, rather than annotation
segments (see below). This new approach will be available in an
upcoming version of MAT.
In order to understand the reconciliation UI, it's important to
understand something first about the reconciliation process
The basis of the reconciliation process is the reconciliation document. A
reconciliation document is a rich annotated document which is
created from a set of input annotated documents. This document
contains all the annotations from the input annotated documents,
but also contains information about which regions are consistently
annotated, and which regions are not. The regions are represented
annotations, and the conflicts within the inconsistent
regions are represented by special annotations, specific to
reconciliation, called VOTEs. Each VOTE is associated with exactly
one SEGMENT, and specifies a subset of the annotations in that
segment, as well as the annotator or annotators associated with
the relevant SEGMENTs in the input documents. So each VOTE knows
which pattern of annotations it corresponds to, and who produced
The process of reconciliation is, essentially, a process of
voting. In the UI, the reviewer is presented with the VOTEs for
each SEGMENT, and makes her choice, or proposes a new VOTE. The
MAT server processes
those votes, and decides whether the SEGMENT has been reconciled.
This basic model is remarkably powerful and flexible, and raises
the possibility of some very sophisticated reconciliation
processes. In MAT 2.0, only the most basic of these processes is
enabled: there's only one voter, and that voter's decisions are
absolute. However, due to the nature of the process, the impact of
those votes can't be completely reflected immediately in the UI;
the votes must still be processed
in order to reconcile the SEGMENT.
The final step of reconciliation is export. If a reconciliation
document has been completely reconciled, you can export it, via
the UI, to a new window or document which is a normal, rich
annotated document, which reflects all the decisions which have
been made during the reconciliation process.
One of the most appealing properties of this process is that it's
stateful. The way reconciliation is implemented, the SEGMENT and
VOTE annotations bear all of the information about the state of
reconciliation: which SEGMENTs are consistent, which are
inconsistent, which have been reconciled, and who's voted for
which alternative. You can save these reconciliation documents and
reload them later. And because the changes are made operational by
exporting rather than by modifying the reconciliation document,
you always have a record of the voting process.
In the MAT desktop, select "File" in the top menubar, and then
"Reconcile files...". You'll be presented with a dialog:
This dialog works very similarly to the document
comparison dialog; you must select a task, and then either
select loaded documents or load documents via the "Load
document..." option. These documents will be listed in the dialog,
and once there's at least two, the "Reconcile" button will be
enabled. Like document comparison, the signals of all the
documents must match.
Here's an example of a populated dialog:
Unlike document comparison, your only option is to remove the
document, via the "X" to the left of each listed document; the
visual position of the documents is controlled by the
You can also invoke document reconciliation in a document alignment window, by
selecting "Menu" -> "Reconcile these documents". In this case,
the documents being aligned will be the ones selected for
When you press "Reconcile", the MAT server will create a new
reconciliation document and present it to you:
In this document, agreed-upon regions are shown with "normal"
presentation of the annotations, behind the annotated text. The
first occurrence here of "World Health Organization" appears in
such a region; we've already reconciled this region via the
process we're about to describe. Any regions in the document which
are common to all the input documents will also be displayed in
Those regions which are in conflict are shown with yellowish
(e.g., "caution") square brackets delimiting the region, with the
pattern for each VOTE represented on a separate line below the
text. You can hover your mouse over these conflicting annotation
bars and get information about the annotation, just like you can
with normally presented annotations. You can also view information
about the SEGMENT annotations by hovering your mouse over the left
or right bracket for that conflicting region. Unreconciled
SEGMENTs will have the status "human gold"; reconciled SEGMENTs
will have the status "reconciled".
To vote on a conflicting region, click on any portion of the
region: the text, the conflicting annotations, or the delimiting
brackets. The rest of the document will be grayed out, and your
voting options will be presented in the "Voting" tab in the detail
pane at the bottom:
You can vote for one of the existing patterns, or vote for the
region having no annotations, or propose a new annotation pattern
for the segment. If you select "new annotation pattern", you'll be
able to hand annotate in that
Once you're satisfied with your choice, select "Done". The
document will no longer be grayed out, a check mark will appear to
the right of the closing bracket, and you can select another
region for voting. If you've proposed a new annotation pattern,
that pattern will be displayed behind the text.
Until you process
your votes, you can return to any region and change your
vote, clear your vote, or edit your new annotation pattern.
The "Reconciliation" item in the top menubar is activated when
viewing a reconciliation document:
In addition, "File -> Save..." is active, and the "View" menu
is changed to reflect the items which are relevant to
The meanings of these elements is as follows:
The "Auto-advance" option modifies voting interaction. When you
enable "Auto-advance", the reconciliation UI will automatically
advance you to the next region that doesn't have a vote
registered, and scroll that region into view. The "Auto-advance"
option will have a check mark after it if it's enabled. If you
enable "Auto-advance" when no region is selected, it will select
the first region. You can disable "Auto-advance" at any time from
the menu; at that point, once you're completed with the current
region under review, you'll be returned to the normal, unguided
mode of voting.
When you review a region and make a vote, you're just voting; the
region is not yet reconciled. As long as you're not reviewing a
region, you can tell the MAT server to apply your votes by
selecting "Process votes" from the menu. Once the votes in a
document are processed, those regions which are now reconciled can
no longer be reviewed; the votes cannot be changed or cleared.
These regions now look like any other reconciled region.
For instance let's say we reviewed the first conflicting segment,
containing the word "South", in the example above, and then
selected "Process votes". The result looks like this:
Like "World Health Organization" in the first line, "South"
in the second line now appears with the selected annotation
pattern behind the text, instead of below, and the yellowish
brackets have been removed.
That doesn't mean the voting information has been discarded,
though; it's just not visible by default. You can control the
display with the "Show processed votes" and "Show all segment
boundaries" options in the menu. Here's the same document display
with both these options enabled:
You can see that all reconciled regions are delimited with green
("ok") square brackets, and that the region containing "South"
that we voted on still has a check mark to the right of the
closing bracket, indicating that it was reconciled by voting
rather than by being already consistent when the reconciliation
document was created. You can also see that the first occurrence
of "World Health Organization" was also agreed upon via
As long as you're not reviewing a region, you can save the
reconciliation document for reloading later. Before you're
prompted for a save location, the UI will process the votes in
These documents are always saved as MAT JSON documents; the only
difference between these documents and normal MAT JSON documents
is that these documents contain VOTE annotations, and are marked
as reconciliation documents in their document metadata.
To load a saved reconciliation document, use "File" -> "Open
file..." in the MAT desktop. In the dialog, select
"(reconciliation)" as the workflow/mode; "mat-json" will be
automatically selected as the document type, and can't be changed:
As long as you're not reviewing a region, you can export the
reconciliation document to a file or window. Before the document
is exported, the UI will process the votes in your document.
If not all the regions are reconciled, the UI will tell you this
and refuse to export.
By default, the UI will export to a new read-only document
window, which you can save If you want. If you want to export to
a file, enable the "Export to file" option in the menu before you
select "Export", and you'll be able to select an output annotation
format, and then a save location.