Experiment output directory structure

The experiment engine records its output in a directory, which is specified when you call the experiment engine. Here, we describe the structure of the output directory.

The output directory has three toplevel subdirectories, one for the corpora ("corpora", by default), one for the models that are constructed ("model_sets", by default), and one for the test runs that are executed ("runs", by default). The names of the toplevel subdirectories might vary (or there may be more such directories), if you've provided a non-default value for "dir" to any of the elements in the XML file. The <format> value in the names of the CSV files are determined by the --csv_formula_output command-line option of the experiment engine.

When the engine completes its experiment runs, it produces a pair of toplevel spreadsheets, allbytag_<format>.csv and allbytoken_<format>.csv. These files contain the tag-level and token-level scoring results for all the runs. The format and interpretation of these results is found in the documentation for MATScore, except that the initial columns are different:

run family
The name of the run in the experiment XML file.
run
The actual directory of the run (which may be affected by whatever iterators have applied)
model family
The name of the model set in the experiment XML file.
model
The actual directory of the model (which may be affected by whatever iterators have applied)
train corpora
The name of the training corpora (and their partitions, if appropriate) for this run. Comma-separated.
test corpora
The name of the test corpora (and their partitions, if appropriate) for this run. Comma-separated.
tag
The label being scored, as described in MATScore.
train docs
The number of documents in the training corpus for this run.
train toks
The total number of tokens in the documents in the training corpus for this run.
train items
The total number of training "items" for this label in the documents in the training corpus for this run. For tag-level scoring, this is the number of annotations; for token-level scoring, this is the number of tokens in those annotations.

In the directory for each run, you'll find the individual scoring files bytag_<format>.csv and bytoken_<format>.csv, which are (approximately) the subset of the corresponding overall scoring files which is relevant to this run. Of greater interest is details.csv, which is the detail spreadsheet for this run. These detail spreadsheets are not aggregated at the top level because they contain an entry for each tag, and the volume of data would likely be too great.