Experiment XML Use Cases

Use cases for the XML format for the experiment files (see MATExperimentEngine) are described here. The reference document is found here. Click here for a split-screen view.

In all the examples below, we're going to use the sample "Named Entity" task.

A simple experiment

The simplest possible experiment involves a single corpus, a single model, and a single run. Assume you have a set of "gold" documents in /documents/newswire/*.json.

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <corpus name="ne">
      <pattern>/documents/newswire/*.json</pattern>
    </corpus>
  </corpora>
  <model_sets dir="model_sets">
    <model_set name="ne_model">
      <training_corpus corpus="ne" partition="train"/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
    </run_settings>
    <run name="test_run" model="ne_model">
      <test_corpus corpus="ne" partition="test"/>
    </run>
  </runs>
</experiment>

This experiment takes a single set of documents, and designates 80% of the set for training and the remaining 20% for test. It then generates a single model from the training documents, and executes a single run using this model against the test documents.

If all your documents have the ".json" extension, and you want to reuse this experiment XML file, just change the <pattern> element entry to a relative pathname and use the --pattern_dir argument when you call MATExperimentEngine.

...
    <corpus name="ne">
      <pattern>*.json</pattern>
    </corpus>

...

A simple experiment with a set-aside corpus

Let's say you've set aside a test corpus which you want to hold constant across a set of experiments, in /documents/newswire-test/*.json. You can use an experiment XML file such as this one:

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <corpus name="train_nw">
      <pattern>/documents/newswire/*.json</pattern>
    </corpus>
    <corpus name="test_nw">
      <pattern>/documents/newswire-test/*.json</pattern>
    </corpus>
  </corpora>
  <model_sets dir="model_sets">
    <model_set name="train">
      <training_corpus corpus="train_nw"/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
    </run_settings>
    <run name="test" model="train">
      <test_corpus corpus="test_nw"/>
    </run>
  </runs>
</experiment>

Here, we have two separate corpora, which are not split; one is used as a training corpus, and the other as a testing corpus. We generate one model, and one run.

A simple experiment run against a workspace

You can run experiments against workspaces. There are special workspace capabilities devoted to experiments, and special experiment engine capabilities devoted to workspaces.

Let's say you have a workspace, and you've introduced two basename sets: test_1 and test_2. Furthermore, there are documents in the workspace which are in neither set. Let's also suppose that you want to restrict the documents to the gold and reconciled documents in the workspace. Finally, let's suppose you have multiple workspaces with these basename set names, and you want to use bindings to specify the specific workspace to use. Here's how you use test_1 and test_2 together as your test set, and use the rest of the documents in the workspace as your training set:

<experiment task='Named Entity'>
  <workspace_corpora dir="corpora" workspace_dir="$(WS)"
                     document_statuses="gold,reconciled">
    <workspace_corpus name="test" basename_sets="test_1,test_2"/>
    <workspace_corpus name="train" use_remainder="yes"/>
  </workspace_corpora>
  <model_sets dir="model_sets">
    <build_settings partial_training_on_gold_only="yes"/>
    <model_set name="train">
      <training_corpus corpus="train"/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
      <score_args gold_only="yes"/>
    </run_settings>
    <run name="test" model="train">
      <test_corpus corpus="test"/>
    </run>
  </runs>
</experiment>

Note, too, that we've enhanced the experiment file as recommended here.

Using multiple corpora

Let's say you have two corpora, and you want to split each of them 4-to-1, and use the larger slice of each of them, together, to build a single model, and test against the smaller slice of each of them, in a single run:

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <corpus name="nw1">
      <pattern>/documents/newswire-1/*.json</pattern>
    </corpus>
    <corpus name="nw2">
      <pattern>/documents/newswire-2/*.json</pattern>
    </corpus>
  </corpora>
  <model_sets dir="model_sets">
    <model_set name="train">
      <training_corpus corpus="nw1" partition="train/>
      <training_corpus corpus="nw2" partition="train/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
    </run_settings>
    <run name="test" model="train">
      <test_corpus corpus="nw1" partition="test"/>
      <test_corpus corpus="nw2" partition="test"/>
    </run>
  </runs>
</experiment>

Validating against the training corpus

Sometimes, you want to run the model against the corpus that produced it. In the example in the previous use case, you can modify the <runs> as follows:

...
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
    </run_settings>
    <run name="test" model="train">
      <training_corpus corpus="train_nw"/>
    </run>
  </runs>
...

An experiment enhanced with model size iteration

To find out what happens when you use more and more training data, we add a <corpus_settings> element to <model_sets>, as follows:

...
  <model_sets dir="model_sets">
    <corpus_settings>
      <iterator type="corpus_size" increment="50"/>
    </corpus_settings>
    <model_set name="test">
      <training_corpus corpus="test"/>
    </model_set>
  </model_sets>
...

In this case, we're telling the experiment engine to build a model at 50-document increments. So if the corpus contains 150 documents, the experiment engine will build three models, and produce one set of three runs.

If your corpus has more than 100 documents, but fewer than 150, the above values for <build_settings> will only build two models. If you want a model built for the remainder, use the "force_last" attribute:

...
  <model_sets dir="model_sets">
    <corpus_settings>
      <iterator type="corpus_size" increment="50" force_last="yes"/>
    </corpus_settings>
    <model_set name="test">
      <training_corpus corpus="test"/>
    </model_set>
  </model_sets>
...

Comparing precision/recall biases

The Carafe tagger has the option of biasing precision and recall differently during automated tagging, using the --prior_adjust flag. If you want to compare two decoding strategies, one which biases heavily toward recall and one toward precision, you might do this:

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <corpus name="newswire">
      <pattern>/documents/newswire/*.json</pattern>
    </corpus>
  </corpora>
  <model_sets dir="model_sets">
    <model_set name="newswire">
      <training_corpus corpus="newswire" partition="train"/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo" prior_adjust="-3.0"/>
    </run_settings>
    <run name="recall_bais" model="newswire">
      <test_corpus corpus="newswire" partition="test"/>
    </run>
  </runs>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo" prior_adjust="3.0"/>
    </run_settings>
    <run name="precision_bias" model="newswire">
      <test_corpus corpus="newswire" partition="test"/>
    </run>
  </runs>
</experiment>
In this case we have two different <runs> elements, because the run settings differ for the two runs. So we end up with one corpus, one model, and two runs.

An experiment with multiple iteration types

Let's say that in addition to increasing the size of the model, you also want to know what happens when you increment the number of model iterations during model building, and you also want to vary the recall/precision bias of the decoder. You can do all these things at once, as follows:

...
  <model_sets dir="model_sets">
    <build_settings>
      <iterator type="increment" attribute="max_iterations"
                start_val="3" end_val="9" increment="2"/>
    <build_settings>
    <corpus_settings>
      <iterator type="corpus_size" increment="50"/>
    </corpus_settings>
    <model_set name="train">
      <training_corpus corpus="nw1" partition="train/>
      <training_corpus corpus="nw2" partition="train/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
      <iterator type="increment" attribute="prior_adjust"
                start_val="-3.0" end_val="3.0" increment="1.0"
                force_last="yes"/>
    </run_settings>
    <run name="test" model="train">
      <test_corpus corpus="nw1" partition="test"/>
      <test_corpus corpus="nw2" partition="test"/>
    </run>
  </runs>
...

If your training corpus is 150 documents, this experiment will generate 12 models - for each of 50, 100 and 150 documents, a model for each of the max_iterations values (3, 5, 7, 9). For each model, the experiment will conduct 7 runs, one for each of the values of prior_adjust. Note that we've used force_last to force the final value to be used, even if it's not exactly 3.0 (due to the issues with how floats are implemented).

A simple cross-comparison experiment with two corpora

Let's say you have two sets of completed documents: a set of newswire documents, in /documents/newswire/*.json, and a set of chat transcripts, in /documents/chat/*.json. Both these document sets are tagged with the same tag set. If you want to know how a model built against each will work on the other, here's an experiment XML file that accomplishes that:

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <corpus name="newswire">
      <pattern>/documents/newswire/*.json</pattern>
    </corpus>
    <corpus name="chat">
      <pattern>/documents/chat/*.json</pattern>
    </corpus>
  </corpora>
  <model_sets dir="model_sets">
    <model_set name="newswire">
      <training_corpus corpus="newswire" partition="train"/>
    </model_set>
    <model_set name="chat">
      <training_corpus corpus="chat" partition="train"/>
    </model_set>
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
    </run_settings>
    <run name="nw_train_nw_test" model="newswire">
      <test_corpus corpus="newswire" partition="test"/>
    </run>
    <run name="nw_train_chat_test" model="newswire">
      <test_corpus corpus="chat" partition="test"/>
    </run>
    <run name="chat_train_chat_test" model="chat">
      <test_corpus corpus="chat" partition="test"/>
    </run>
    <run name="chat_train_nw_test" model="chat">
      <test_corpus corpus="newswire" partition="test"/>
    </run>
  </runs>
</experiment>

This experiment XML file will split each corpus 80%/20%, and build two models, one from each corpus. Finally, it performs a four-way comparison between the models and the test subsets of the corpora.

Model comparison

Let's say that you have a Carafe lexicon directory, as described in the documentation for MATModelBuilder. You want to know whether using this lexicon results in a better model. Here's an experiment XML file which accomplishes that:

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <corpus name="newswire">
      <pattern>/documents/newswire/*.json</pattern>
    </corpus>
  </corpora>
  <model_sets dir="model_sets">
    <model_set name="newswire">
      <training_corpus corpus="newswire" partition="train"/>
    </model_set>
  </model_sets>
  <model_sets dir="model_sets">
    <build_settings lexicon_dir="/documents/newswire_lexicon/"/>
    <model_set name="newswire_w_lex">
      <training_corpus corpus="newswire" partition="train"/>
    </model_set>  
  </model_sets>
  <runs dir="runs">
    <run_settings>
      <args steps="zone,tokenize,tag" workflow="Demo"/>
    </run_settings>
    <run name="w_lex" model="newswire_w_lex">
      <test_corpus corpus="newswire" partition="test"/>
    </run>
    <run name="wo_lex" model="newswire">
      <test_corpus corpus="newswire" partition="test"/>
    </run>
  </runs>
</experiment>

In this case, there are two different <model_sets> elements, because the build settings for the enclosed models differ. We have one corpus, two models, and two runs.

You can further specify any of the advanced settings for the trainer, if you know what you're doing. See MATModelBuilder for whatever documentation is available.

Preprocessing a corpus

Sometimes, you may need to do some preprocessing of a corpus. Let's assume:

To do this during the experiment, you'd use the <prep> element:

...
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <prep steps="zone,tokenize,align" workflow="Preprocess" input_file_type="xml-inline" xml_input_is_overlay="yes"/>
    <corpus name="test">
      <pattern>/documents/newswire/*.xml</pattern>
    </corpus>
  </corpora>
...

Preserving zone tags during run preparation

Let's say, for instance, that you're working with the MUC (Message Understanding Conference) corpus, and you're not tagging the header portions of the documents. Under normal circumstances, when it prepares an experiment run, the experiment engine converts the test documents to raw text, and processes them starting from raw text. However, in this case, you can't actually recreate the zoning with your own zoner; you need the zoning as it was provided in the MUC documents. In this situation, you can use the <prep_args> element in the <run> element to specify a set of parameters to MATEngine to modify the default test document preparation:

...
  <runs dir="runs">
    <run_settings>
      <prep_args output_file_type="mat-json" undo_through="tag" workflow="Demo"/>
      <args steps="tag" workflow="Demo"/>
    </run_settings>
    <run name="test" model="test">
      <test_corpus corpus="test" partition="test"/>
    </run>
  </runs>
...

Here, instead of undoing all steps by using an output_file_type of "raw" (which is the default), we undo the "tag" step and use MAT JSON documents as the inputs to the run; we see that the <args> for the run only does the "tag" step.

Sharing a corpus

Sometimes, you might want to prepare a corpus ahead of time, with a fixed partition, a fixed prep phase, or the like. You can use the experiment engine to create a corpus alone, and then refer to that corpus elsewhere.

For instance, you might prepare the corpus in the previous use case with nothing in the <experiment> element except the <corpora>:

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <partition name="train" fraction=".8"/>
    <partition name="test" fraction=".2"/>
    <prep steps="zone,tokenize,align" workflow="Preprocess" input_file_type="mat-json"/>
    <corpus name="test">
      <pattern>/documents/newswire/*.json</pattern>
    </corpus>
  </corpora>
</experiment>

Assume we save this XML file to /experiments/xml/corpus.xml, and output the experiment into /experiments/corpus1:

% cd $MAT_PKG_HOME
% bin/MATExperimentEngine --exp_dir /experiments/corpus1 /experiments/xml/corpus.xml

The corpus will be in the "corpora" subdirectory, in the subdirectory named "test" (the name of the corpus).

Now, let's refer to it in a different experiment XML file:

<experiment task='Named Entity'>
  <corpora dir="corpora">
    <corpus name="local_test" source_corpus_dir="/experiments/corpus1/corpora/test"/>
  </corpora>
  ...
</experiment>

Instead of including a <pattern> element, we use the "source_corpus_dir" attribute. The corpus referred to can itself have a "source_corpus_dir" attribute (i.e., you can chain them). Local <prep> or <partition> elements can augment or override remote elements; the combinations are complex, and you can find more documentation on them in the experiment XML reference.