General Deidentification Customizations

• The task
  
• The annotations
• Extensions to the available steps
• Additional step implementations
• MATEngine example
• Training, lexicons, and case sensitivity
  
• Running deidentification experiments
  
• Customizing deidentification replacement
  
• Accessing deidentification from Java
  
• Extensions to the workspaces
• Extensions to the UI
• Flagged unparseable seeds and forced review
• Replacement, redaction and resynthesis
• Extensions to the task.xml file
• Defining your own deidentification task
• Using deidentification without MIST
• Version history
  

The task

The core MIST deidentification system is implemented as a task within the MAT toolkit. You will not use this core deidentification task directly; rather, you'll be using a further specialization of the core deidentification task, e.g., the AMIA or HIPAA task. Each task is a child task of the parent task class, named "Deidentification", and the implementation of each task is a subclass of the Deidentification.DeidTaskDescriptor Python class. You can see evidence of both these parent-child relationships in the task.xml file for your particular deidentification task.

Creating new deidentification tasks is very, very complicated, and we don't have the resources to document the process. Please consult the specific deidentification tasks you have in your distribution and use them as models if you absolutely need to build your own task.

The annotations

The zone and token annotations you'll use (see the discussion of annotations here) are just the default zone and token annotations. The content annotations, which are the ones you'll add or correct, correspond to the PII categories for your task: e.g., the HIPAA categories for medical record deidentification. Refer to the documentation for your specific deidentification task for more details.

Extensions to the available steps

As we describe here, steps are the basic activities that you undertake in a document. E.g., the tokenize step typically identifies the word boundaries in the text. The typical deidentification task has the following steps:

• zone: find appropriate the regions of the document to process, and find the word boundaries in those regions. This step is a concatenation of the core MAT zone step and the core MAT tokenize step, for accidental historical reasons.
• tag: add PII annotations, either automatically or by hand.
• nominate: create replacement phrases for the regions of text tagged with PII annotations, and store them in the PII annotations.
• transform: create a new document using the replacement phrases.

Note that the tokenize step found in the sample MAT task is not used explicitly in MIST.

The nominate step requires a replacer, which is a strategy for generating replacement phrases. We discuss replacers below.

The typical deidentification task arranges these steps into a number of workflows:

• Hand annotation: zone, (hand) tag, nominate, transform
• Review/repair: nominate, transform
• Demo: zone, tag, nominate, transform
• Resynthesize: tag, nominate, transform

What's unusual about these four workflows is that the last of the workflows applies to a different class of documents than the first three. The resynthesize workflow is intended to apply to documents which have been transformed into an obscure form, e.g., documents which contain fillers like [PERSON] in place of the person names in the original document. Under the resynthesize workflow, the tag step is a simple pattern-matching operation, and the replacers convert obscured documents into resynthesized documents. We discuss resynthesis below.

Additional step implementations

The steps of the deidentification task require the following step implementations:

As described here, steps can also take key-value pair arguments which can be specified in the task.xml file or in the invocation of the MAT engine. We document the arguments for these steps here.

Deidentification.NominateStep

The following options might be generally useful for the nominate step:

The nominate step also accepts the following options, but they're pretty exotic and you're very unlikely to use them (they're also not particularly well tested or supported):

Deidentification.TransformStep

The transform step allows you to insert a prologue into your file.

MATEngine example

Here's a standard use of MATEngine in this task. Let's say you want to prepare a deidentified copy of a document with fake but realistic English replacement PII elements, in rich format, and you have a prologue in prologue.txt that you want to insert, and you have a default model in your task. Here's how it works:

% $MAT_PKG_HOME/bin/MATEngine --task 'My Deid Task' --workflow Demo \
--steps 'zone,tag,nominate,transform' --replacer 'clear -> clear' \
--input_file /path/to/my/file.txt --input_file_type raw \
--output_file /path/to/my/resynth/file.txt.json --output_file_type mat-json \
--tagger_local --prologue_file prologue.txt

The --tagger_local option is required because by default,  the Carafe tagging task attempts to contact the tagging server.

Training, lexicons, and case sensitivity

MIST is distributed with a default feature specification file, which provides reasonable performance. If you received MIST via a zip distribution, the default Carafe feature specification for MIST is found in src/tasks/core/resources/amia.fspec. You may want to try to improve your performance by customizing these training features. The Carafe documentation will help you customize this feature specification. You can provide this customized feature specification to the trainer via the --feature_spec option.

The Carafe engine supports the specification of lexicons to aid in model training. Although the behavior in MIST doesn't differ from that in the core MAT system, it's worth mentioning that the capitalization in your lexicons interacts with the Carafe feature specification. Please follow the instructions carefully.

Running deidentification experiments

The sample experiment described here is a good place to start for constructing experiments for the MATExperimentEngine. The primary differences you should keep in mind are:

• The name of the task in the experiment XML file must correspond to the name of your deidentification task
• There is no tokenize step in the deidentification family of tasks

So if your task is named 'My Deid Task', then your simple experiment might look like this:

<experiment task='My Deid Task'>
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <corpus name="deid">
      <pattern>*.json</pattern>
    </corpus>
  </corpora>
  <model_sets dir="model_sets">
    <model_set name="deid_model">
      <training_corpus corpus="deid" partition="train"/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tag" workflow="Demo"/>
    </run_settings>
    <run name="test_run" model="deid_model">
      <test_corpus corpus="deid" partition="test"/>
    </run>
  </runs>
</experiment>

Customizing deidentification replacement

In MIST 1.2, we added a (very experimental) capability to customize the deidentification replacement. This capability is available only with clear -> clear replacement. To understand how to use it, you'll need to know a bit more about how the replacement works.

Each replacement strategy consists of a digester and a renderer. The digester produces a pattern which describes the features of the digested element (e.g., for a phone number, was an area code present). The digested pattern also contains the raw source filler, and, in the case of the clear digester, the parsed form of the input in some cases (e.g., for names and locations). The renderer uses the pattern to generate its replacement, and, in the case of the clear renderer, attempts to apply any customization rules it finds.

The customization rules are provided to the nominate step either with the replacement_map or the replacement_map_file option. The replacement map is a JSON string which has the format described immediately below; the replacement_map_file provides a filename which contains such a string.

The JSON string described a JSON hash (object) whose keys are the file basenames which you're trying to deidentify; e.g., if you're trying to deidentify /path/to/my/file.txt as in the example above, the key for that file would be "file.txt". As a special case, if you're accessing the MIST capability via the Web service, the name of the file should be "<cgi>". The values of these keys should be another JSON object whose keys are the names of the labels you're targeting. So if you have rules for file.txt which target the DATE tag, your replacement map will look like this so far:

{"file.txt": {"DATE": ...}}

The label key values are also JSON objects. These objects can have two keys: caseSensitive (either true or false) and rules, which should be a list of 2-element lists, where the first element of each sublist is the antecedent and the second is the consequent. The antecedent and consequent are each themselves JSON objects. The antecedent is a recursive structure which describes a subset of the digested pattern; the consequent contains two keys, seed and pattern, each of which describe updates to the seed and pattern, respectively.

Much of the details of this needs to be derived from the source code, but we provide a couple examples here.

Let's say you want to replace all occurrences of the last name "Marshall" for the NAME tag in the file file.txt with the last name "Bigelow". Your specification will look like this:

{"file.txt":
 {"NAME": 
  {"rules": [[{"parse": {"lastName": "Marshall"}}, {"seed": {"lastName": "Bigelow"}}]]}}}

All eligible rules apply. So if you want to replace the first name "Betty" with the first name "Phyllis", you can add another rule:

{"file.txt":
 {"NAME": 
  {"rules": [[{"parse": {"lastName": "Marshall"}}, {"seed": {"lastName": "Bigelow"}}],
             [{"parse": {"firstName": "Betty"}}, {"seed": {"firstNameAlts": ["Phyllis"]}}]]}}}

If you want to replace them only when they're together, do this:

{"file.txt":
 {"NAME": 
  {"rules": [[{"parse": {"lastName": "Marshall", "firstName": "Betty"}}, 
              {"seed": {"lastName": "Bigelow", "firstNameAlts": ["Phyllis"]}}]]}}}

If you only want to replace the literal string "Betty Marshall", do this:

{"file.txt":
 {"NAME": 
  {"rules": [[{"input": "Betty Marshall"}, 
              {"seed": {"lastName": "Bigelow", "firstNameAlts": ["Phyllis"]}}]]}}}

(Note that the input key is available for all tags, while the details of the parse and pattern keys differ from tag to tag).

Let's say you want to control how much dates are shifted (dates are shifted by a consistent amount throughout a single document). Do this:

{"file.txt":
 {"DATE":
  {"rules": [[{}, {"pattern": {"deltaDay": 5}}]]}}}

The deltaDay attribute of the pattern controls the date shift, and this rule applies the shift to all dates (because the antecedent is empty).

Finally, let's say you want to do a consistent substitution of certain IDs, e.g., patient IDs, for correlation with an external data escrow application which manages deidentification for your structured records. If your tag is IDNUM, you can do this:

{"file.txt":
 {"IDNUM":
  {"rules": [[{"input": "PATNO67897"}, {"seed": {"id": "ID9938273"}}]]}}}

In the future, we hope to flesh out and further document this capability.

Accessing deidentification from Java

The MAT toolkit comes with a Java API which allows you to read and write MAT JSON documents, and access services provided by the MATWeb application. You can find documentation for the Java API under the "Core developer documentation" in your documentation sidebar.

One common use of the Java API is in incorporating deidentification capabilities in a larger Java application. Here's a Java fragment which shows you how to do that. Remember, you must make sure you have all the jars in the following directories in your class path:

• src/MAT/lib/mat/java/lib
• src/MAT/lib/mat/java/java-mat-core/dist
• src/MAT/lib/mat/java/java-mat-engine-client/dist

import java.util.*;
import org.mitre.mat.core.*;
import org.mitre.mat.engineclient.*;

String res = "";
/* Modify the URL as needed. Only the host and port are required. */
String url = "http://localhost:7801";

/* This should be the name of your task, as expted by MATEngine. */
String task = "HIPAA Deidentification";
/* This should be the name of the workflow, as expected by MATEngine. */
String workflow = "Demo";
/* This should be the step sequence to perform, as expected by MATEngine. */
String steps = "zone,tag,nominate,transform";
HashMap<String, String> attrMap = new HashMap<String, String>();
/* This should be the name of the replacer you want to use. */
attrMap.put("replacer", "clear -> clear");

MATDocument doc = new MATDocument();

/* Your input string is the argument to setSignal(). */
doc.setSignal("Hello World");

/* Here's where you connect to the server. */
MATCgiClient client = new MATCgiClient(url);
try {
    MATDocument resultDoc = (MATDocument) client.doSteps(doc, task,
       workflow, steps, attrMap);
    res = resultDoc.getSignal();
} catch (MATEngineClientException ex) {
    /* Handle the error. */
    System.out.println("Processing failed: " + ex.getMessage());
}
/* Do something with the retrieved deidentified text. */
System.out.println(res);

Extensions to the workspaces

As we describe here, workspaces are actively-managed directory structures which encapsulate the standard workflows in MAT. Workspaces in the deidentification task have three extra document folders:

• redacted, rich: documents corresponding to documents in the core folder which have been redacted. These documents are in the rich MAT JSON format which encodes the annotations.
• redacted, raw: documents corresponding to documents in the core folder which have been redacted, as raw UTF-8 text.
• nominated: documents which bear replacement nominations.
  

In addition, the core folder has two new operations:

redact clears and populates the "redacted, rich" and "redacted, raw" folders in parallel, by applying a replacer to the specified documents in the core folder. This operation is intended to apply only to original documents, to produce redacted documents; the inverse resynthesize operation isn't available via the workspace. This operation is the equivalent of performing both the nominate and transform steps described above. It has the following options:

nominate performs just the nominate step and inserts a copy of the document which is decorated with nominations into the "nominated" folder. This latter folder supports the transform operation, which removes the document from the "nominated" folder and performs the remainder of the redact operation. This alternative, two-step workflow is provided for situations where the user wishes (or needs) to review the nominations. It has the following options:

As of version 2.0.2, the "nominated" folder now locks its files for editing when they're shown in the UI. Accordingly, this folder now supports the "force_unlock" operation on the command line, analogous to the "core" folder.

Extensions to the UI

File mode enhancements

The normal file and workspace mode document window configurations described here and here. The nomination and transformation steps require enhancements to the UI, as shown in this section. The document exhibited is drawn from the CMC free text corpus of deidentified radiology reports.

First, we show the result of the automated tag step:

Note not just the additional steps in the workflow, but also the menu labeled "Replacer" at the bottom of the Controls area. This menu allows you to select the replacement method for the PII elements. The nominate step requires a replacer; here we've selected "clear -> []". We correct the automated tagging output and mark it gold:

Now, we apply the nominate step:

Once the nominate step is completed, a table appears in a tab in the details pane below the document showing each PII, along with its type, its location in the document, and its proposed replacement. The final transform step splits the document pane:

You'll also discover that the "File" menu in the menubar now has a "Save replacement..." option which allows you to save the replacement in the same way you can save the original document.

Workspace enhancements

If you're in a workspace, the relevant point you'll notice a difference is when you're viewing a document in the core folder which you've marked gold:

Like the file mode case, the "Controls" area now has a replacer menu, and the operations menu allows you to access the new redact and nominate operations. Unlike file mode, the result of the redact operation will be a new document tab, rather than a split in the existing tab.

The new operations are available only when the document status is gold. Note that your task may have a default replacer specified in the workspace operation declaration in task.xml; if so, the redact and nominate operations will not require that the replacer be specified (but they will respect a menu selection if one is made).

Working with replacements

The UI has two important enhancements available at the conclusion of the nominate step. First, when you mouse over an annotation, the redaction is shown in the status line below the text as well as in the table below the main document pane:

In addition, if you're not satisfied with the replacement, you can edit the replacement by clicking on the appropriate cell in the "Replacement" column in the table below the main document pane:

If you modify the replacement and click "OK", the result will be what's used as the replacement (at least, as long as you don't undo and redo the "nominate" step).

Note: this is not the procedure for marking a previously unmarked PII; if you want to do that, you have the opportunity after the "tag" step to review and modify the automated annotations.

Flagged unparseable seeds and forced review

If you set up your task to flag unparseable seeds in the nominate step, the document will be marked in such a way that whenever it's presented, it will warn you that unparseable seeds are present and must be reviewed, and the nomination table 'Replacement' column will contain at least one cell with a red background:

If you attempt to transform the document before you clear all these flags, the transform step will fail and you'll get an error.

The process in workspaces is quite similar. If you attempt the "Redact" operation in the "core" folder, you'll get an error if the nominate step generated any of these unparseable flags; in this situation, you should employ the alternative two-step workflow, by performing the "Nominate" operation instead.

Replacement, redaction and resynthesis

The nomination step generates replacement fillers for the PII elements identified by the content annotations. We identify two types of replacement strategies:

• redaction, where the PII element is converted from clear text to some obscured form
• resynthesis, where the PII element is converted to clear text, from either an obscured form or clear text

The core deidentification system provides three basic types of redaction:

• replacement with a bracketed PII name: "John Smith" -> "[PERSON]"
• replacement with a slightly more elaborate pattern-oriented format, compatible with the output of the University of Pittsburgh's De-ID system: "John Smith" -> "**NAME[AAA BBB]"
• replacement of alphanumeric characters with characters of the same type: "John Smith" -> "Pqty Muwqd"

The replacement engine first gathers a set of features from the input PII. For instance, in the case of names, it attempts to reproduce the number of tokens, the capitalization pattern, whether they correspond to a name with the last name first, etc. For dates, it attempts to preserve the offset from the earliest date in the document, as well as the specific details of how the date was formatted. Any features which cannot be determined from the input are assigned randomized values based on a weighted, hand-crafted estimation of the frequency of the possible values.

Once the feature values are determined, the engine generates replacement fillers. In the case of redaction, the fillers are trivially produced from the gathered features; in the case of resynthesis, the problem is considerably more complex, because our target is realistic English clear text. In the resynthesis case, the tokens for the replacement fillers are drawn from a variety of sources, including weighted lists of first and last names provided by the US Census, and lists of cities, states, streets, medical facilities and ZIP codes derived from various on-line resources. The replacement fillers are assembled based on the features the engine has already gathered; so, for instance, if the engine has determined that a name consists of a last name followed by a first name and an initial (as it might determine from a pattern like **NAME[AAA, BBB M] or a PII such as "Philips, Bruce R."), it will generate a new filler like “Ahmad, Jane Q”. Similarly, date offsets are preserved, so that the pair of dates Jan. 17 and Jan. 20 are shifted, but the 3 day difference is preserved. The engine also caches replaced name tokens on a document-by-document basis; so "AAA" will correspond to "Ahmad" throughout the document shown.

The resynthesis engine can be modified in a number of ways: where the replacement fillers are drawn from (e.g., an external lexicon or the corpus itself), how the fillers are constructed (whether they're replaced whole or reconstructed token-by-token), and how the fillers are cached for repetition (by document, by corpus, or without any cacheing). We will not document those customizations right now.

Extensions to the task.xml file

Each workflow in this task can be associated with replacers. This is specified in the <settings> section of task.xml. The specification pairs attribute-value settings, where the first attribute-value pair is an arbitrary name of a replacer set and a comma-separated list of replacer implementations, and the second attribute-value pair is "<name>_workflows" and a comma-separated list of workflow names. The interpretation is that the specified replacers are available during the nomination step of the specified workflows. Here's an example:

    <settings>
      <setting>
        <name>redaction_replacers</name>
        <value>BracketReplacementEngine.BracketReplacementEngine,CharacterReplacementEngine.CharacterReplacementEngine,ClearReplacementStrategy.ClearReplacementEngine</value>
      </setting>
      <setting>
        <name>redaction_replacers_workflows</name>
        <value>Demo,Hand annotation,Review/repair</value>
      </setting>      
      <setting>
        <name>resynthesis_replacers</name>
        <value>BracketReplacementEngine.BracketResynthesisEngine</value>
      </setting>
      <setting>
        <name>resynthesis_replacers_workflows</name>
        <value>Resynthesize</value>
      </setting>
    </settings>

So the interpretation here is that the "Resynthesize" workflow has the replacer BracketReplacementEngine.BracketResynthesisEngine available, and the other three workflows have the other replacers available.

Here are the available default replacers:

There are also replacers available to manage the De-ID-style replacement, but those have to be customized in a child task.

Defining your own deidentification task

At some point, you may want to define your own deidentification task. We really don't recommend this, because much of the magic to make this work is undocumented. If you must do this, start by copying the entire directory structure of an existing deidentification task (your distribution will contain at least one). Do not copy the core task.

Once you've copied the task directory, you'll need to make the following changes to that task to so that MIST can recognize it:

• Change the name attribute of the <task> element in the task.xml file to something new.
• Change the name of all Python libraries and classes. In order to do this, rename all the files in the python/ subdirectory of the new task, rename all the classes within them, rename any module references which correspond to Python file names you've changed, and change all the corresponding names in the task.xml file to match. These class names can be found in the various class attributes of the elements in that file, and in the settings described in the extensions section above. Note that only some of the redaction and resynthesis class names will have to be changed; some of the classes will have been provided by the core task. Only change the names of mentioned classes whose Python names you've changed.

Using deidentification without MIST

The replacement engine can be run independently of MIST, from either Python or Java. The instructions for setting up this option, as well as supporting code, can be found in src/tasks/core/standalone. See the README file in that directory for further details.

Version history

2.0.2

• Fixed a wide range of bugs related to assigned documents and the "nominated" and "redacted, raw" and "redacted, rich" folders in workspaces.
• Added edit locks for the "nominated" folder for the case where files are opened in the UI.
• Added the force_unlock operation to the "nominated" folder.
  

2.0.1

• Updated the default MIST feature specification file to include a lexicon feature, without which --lexicon_dir has no effect.
  

2.0

• Code updates for compatibility with MAT 2.0.
  

1.3.1

• Created resources for running the replacement engine without MIST.
• Created resources for running the replacement engine under Jython.
• Modified LOCATION replacement to use the input pattern as a seed, in parallel with other replacement types.
• Improved LOCATION processing to eliminate the possibility of null replacements.
• Introduced the COUNTRY pattern.
• Refactored management of replacement frequencies and pattern specification to simply the engine slightly.
• Fixed a bug with replacement key cacheing where names which share a last name ended up sharing a replacement.
• Added hospital initials.
• Expanded the OTHER category to capture capitalization status and abbrevation status, and support arbitrary resource replacement files.
• Ensured that the replacement engine can pass through annotations which have no replacement category.
• Ensured that the replacement engine can pass through annotations which it is explicitly instructed not to replace.
• Ensured that AGE fillers will never be replaced with themselves, and will never be negative.
• Ensured that street numbers never start with 0.
• Ensured that tokenization applies in batch mode to batch replacements.
• Removed spurious entries from the nickname file.
• Fixed a long-standing, subtle bug where patterns which are repopulated from frequency information were not respecting the specified dependencies between pattern attributes when determining the order in which these attributes would be assigned values.
  

1.3

• Improved test facility for AMIA.
• Added the ability to flag unparseable elements (e.g., of DATE) for review. Added review capabilities to UI.
• Added heuristics for determining name gender.
  

1.2

• Added the AMIA task to the distribution.
• Improved and enhanced replacement cacheing.
• Added the ability to replace resource files.
• Added replacement map capabilities for PERSON, DATE, ID.
• Added the ability to include a prologue file.