Processing Engine

Description

The processing engine manages the execution of a sequence of steps against a set of files.

Note that you should never use MATEngine to save files into workspace; use MATWorkspaceEngine instead.

Usage

Unix:

% $MAT_PKG_HOME/bin/MATEngine

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd

Usage: MATEngine [core options] [input/output/task options] [other options]

Options:
  -h, --help            show this help message and exit

  Core options:
    --other_app_dir=dir
                        additional directory to load a task from. Optional and repeatable.
    --settings_file=file
                        a file of settings to use which overwrites existing settings. The file should a
                        Bourne shell file which sets variables. Optional.
    --task=task         name of the task to use. Obligatory if the system knows of more than one task.
                        Known tasks are: ...
    --debug             Enable debug output.
...

If no arguments are provided to MATEngine, the help message above is presented. The complete list of options is presented once a task argument is provided. Note that the core options must precede the input, output and task options, which must precede any other options (this is because the later options are added progressively as the earlier options are discovered).

The MAT engine is embedded in a number of other locations, such as the specification of workspace operations and the preprocessing and test corpus processing in the experiment engine specifications. Both of these specifications use XML as their configuration language. Accordingly, we describe both the command line options and their XML equivalents here (with the exception of the core options immediately below, which don't have any XML equivalents).

Core options

MATEngine also makes the common options available.

Once a task argument is present, MATEngine summarizes the workflow structure for the task before it prints out the full option list:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task 'Named Entity'

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task 'Named Entity'

Error: workflow must be specified
Usage: MATEngine [core options] [input/output/task options] [other options]

Named Entity :
  available workflows:
    Hand annotation : steps zone, tokenize, tag
    Review/repair : steps 
    Demo : steps zone, tokenize, tag

...

The remainder of the options can be grouped into a number of categories.

Task options

The task options control what is done to each input file. A workflow must be specified. You can either apply new steps (with the --steps flag), or undo existing steps (with the --undo_through flag). If neither is specified, the tool operates as a (somewhat expensive) transducer between the input and output formats.

Input options

The input options specify the input files. You can specify individual files, or directories (possibly filtering their contents using a regular expression). You must specify a file type. For raw files, you can also specify an input character encoding.

Output options

The output options specify how the result is saved. If you don't specify any output options, the result will be ignored. You can specify an output file for an input file, or an output directory and/or name mapping for an input directory. You must also specify the output format; usually, you'll want this to be one of the rich formats, but "raw" is useful in some rare circumstances. Finally, you can specify an output character encoding for raw files.

Other options

The readers and writers described above may introduce additional options, which are described here. These options must follow the input and output options.

Finally, it's possible for the implementation of individual steps to contribute command-line arguments to MATEngine. There may be default values for these arguments defined in your task, and these command-line specifications override those defaults.

The general options for automated tagging are:

In addition, the Carafe tagger provides other tagging options.

Examples

Example 1

Let's say you have a task named "My Task", with a workflow named "All" which contains steps "zone", "tokenize" and "tag" as in the sample 'Named Entity' task. In order to zone and tokenize a raw document /path/to/my/document.txt and save the result to a MAT JSON document:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize' --input_file /path/to/my/document.txt \
--input_file_type raw --output_file /path/to/my/document.txt.json \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize" --input_file c:\path\to\my\document.txt \
--input_file_type raw --output_file c:\path\to\my\document.txt.json \
--output_file_type mat-json

Example 2

Let's say you want to undo the tokenize step from the document above:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--undo_through 'tokenize' --input_file /path/to/my/document.txt.json \
--input_file_type mat-json --output_file /path/to/my/document.txt.notoks.json \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--undo_through "tokenize" --input_file c:\path\to\my\document.txt.json \
--input_file_type mat-json --output_file c:\path\to\my\document.txt.notoks.json \
--output_file_type mat-json

Example 3

Let's say you want to process the document as in example 1, but you don't have any interest in saving the results (e.g., you're just testing to see if it breaks):

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize' --input_file /path/to/my/document.txt \
--input_file_type raw

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize" --input_file c:\path\to\my\document.txt \
--input_file_type raw

Example 4

Let's say you want to process the document as in example 1, and you want to see the result, but you don't want to bother saving it to a file:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize' --input_file /path/to/my/document.txt \
--input_file_type raw --output_file - --output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize" --input_file c:\path\to\my\document.txt \
--input_file_type raw --output_file - --output_file_type mat-json

Example 5

Let's say you want to process the document as in example 1, but you want to see the intermediate results:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize' --input_file /path/to/my/document.txt \
--input_file_type raw --output_file /path/to/my/document.txt.json \
--output_file_type mat-json --print_steps 'zone,tokenize'

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize" --input_file c:\path\to\my\document.txt \
--input_file_type raw --output_file c:\path\to\my\document.txt.json \
--output_file_type mat-json --print_steps "zone,tokenize"

Example 6

Let's say you have the output of example 1, but you want to retokenize it. You can simultaneously specify the undo and redo steps. The undo steps will be performed first.

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--undo_through 'tokenize' --steps tokenize \
--input_file /path/to/my/document.txt.json \
--input_file_type mat-json --output_file /path/to/my/document.txt.retoks.json \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--undo_through "tokenize" --steps tokenize \
--input_file c:\path\to\my\document.txt.json \
--input_file_type mat-json --output_file c:\path\to\my\document.txt.retoks.json \
--output_file_type mat-json

Example 7

Let's say you have a directory full of text files in /path/to/my/documents which you want to process, and you want the results to have the identical names, but in /path/to/my/jsondocuments:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize' --input_dir /path/to/my/documents \
--input_file_type raw --output_file /path/to/my/jsondocuments \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize" --input_dir c:\path\to\my\documents \
--input_file_type raw --output_file c:\path\to\my\jsondocuments \
--output_file_type mat-json

Example 8

Let's say you want to process your documents as in example 7, but you want to save them back to /path/to/my/documents, with an additional suffix:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize' --input_dir /path/to/my/documents \
--input_file_type raw --output_file /path/to/my/documents \
--output_file_type mat-json --output_fsuff '.json'

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize" --input_dir c:\path\to\my\documents \
--input_file_type raw --output_file c:\path\to\my\documents \
--output_file_type mat-json --output_fsuff ".json"

Example 9

Let's say you have a directory like the one that would be created in example 8, with raw and MAT JSON documents intermixed. But all the files you want to process end with ".txt":

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize' --input_dir /path/to/my/documents \
--input_file_type raw --output_file /path/to/my/documents \
--output_file_type mat-json --output_fsuff '.json' \
--input_file_re '.*[.]txt'

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize" --input_dir c:\path\to\my\documents \
--input_file_type raw --output_file c:\path\to\my\documents \
--output_file_type mat-json --output_fsuff ".json" \
--input_file_re ".*[.]txt"

Note that the regular expression is a Python regular expression, and that it must be enclosed in single quotes on the Unix command line to suppress any bash command-line preprocessing (double quotes should be used in Windows, as usual).

Example 10

Let's say your "tag" step in the "All" workflow is implemented as a Carafe tag step. By default, MATEngine attempts to contact the MATWeb server for tagging (because the MATWeb server doubles as a proxy for a tagging service, which doesn't need to be started up every time you want to tag a document). You can provide a Carafe model, and ensure that the engine starts up Carafe itself rather than trying to contact MATWeb, as follows:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'zone,tokenize,tag' --input_file /path/to/my/document.txt \
--tagger_local --tagger_model /path/to/my/model \
--input_file_type raw --output_file /path/to/my/document.txt.json \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "zone,tokenize,tag" --input_file c:\path\to\my\document.txt \
--tagger_local --tagger_model c:\path\to\my\model \
--input_file_type raw --output_file c:\path\to\my\document.txt.json \
--output_file_type mat-json

Note that this model overrides any model specified in the task file.

Example 11

Like example 10, except on the output of example 1 (that is, zoning and tokenization are already done):

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow All \
--steps 'tag' --input_file /path/to/my/document.txt.json \
--tagger_local --tagger_model /path/to/my/model \
--input_file_type mat-json --output_file /path/to/my/document.txt.tagged.json \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow All \
--steps "tag" --input_file c:\path\to\my\document.txt.json \
--tagger_local --tagger_model c:\path\to\my\model \
--input_file_type mat-json --output_file c:\path\to\my\document.txt.tagged.json \
--output_file_type mat-json

Example 12

Let's say that you have some XML documents which contain XML content annotations, and you have an "align" step which will align the annotation boundaries with token boundaries after you've tokenized the document. Furthermore, you want all the tags which aren't names of annotations in your task to be preserved in the signal, so you provide the --xml_input_is_overlay option, which is an option enabled by the xml-inline reader:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow Import \
--steps 'zone,tokenize,align' --input_file /path/to/my/document.xml \
--input_file_type xml-inline --xml_input_is_overlay \
--output_file /path/to/my/document.xml.json \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow Import \
--steps "zone,tokenize,align" --input_file c:\path\to\my\document.xml \
--input_file_type xml-inline --xml_input_is_overlay \
--output_file c:\path\to\my\document.xml.json \
--output_file_type mat-json

Example 13

Let's say that you want to do example 12 in two steps: first convert to MAT JSON format, then process. To do the conversion, simply call MATEngine without any steps:

Unix:

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow Import \
--input_file /path/to/my/document.xml \
--input_file_type xml-inline --xml_input_is_overlay \
--output_file /path/to/my/document_unprocessed.xml.json \
--output_file_type mat-json

% $MAT_PKG_HOME/bin/MATEngine --task "My Task" --workflow Import \
--steps 'zone,tokenize,align' \
--input_file /path/to/my/document_unprocessed.xml.json \
--input_file_type mat-json \
--output_file /path/to/my/document.xml.json \
--output_file_type mat-json

Windows native:

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow Import \
--input_file c:\path\to\my\document.xml \
--input_file_type xml-inline --xml_input_is_overlay \
--output_file c:\path\to\my\document_unprocessed.xml.json \
--output_file_type mat-json

> %MAT_PKG_HOME%\bin\MATEngine.cmd --task "My Task" --workflow Import \
--steps "zone,tokenize,align" \
--input_file c:\path\to\my\document_unprocessed.xml.json \
--input_file_type mat-json \
--output_file c:\path\to\my\document.xml.json \
--output_file_type mat-json