Explore AIM repos

An AIM repository is merely a folder where AIM stores a rocksdb database (see AIM reference doc for more info).

AIM has great functionality for tracking metrics but has very little support for tracking general artifacts outside console output and nor has native support for storing trained models files.

Hence tcbench complement AIM by collecting runs artifacts into run-specific folders.

Specifically, a tcbench repository has the following structure

<root>
├── .aim
├── artifacts
│   ├── 001baa39ed8d4b8bb9966e94
│   ├── 025830cb840b4f3f8f0a1625
│   ├── 050bae064b5246f88e821a29
...
└── campaign_summary
    └── <campaign_id>

  • Each subfolder of /artifacts collects the artificats of a specific run and is named with the hash of the run itself.

  • The /campaign_summary subfolder collects reports generated by the aimrepo report subcommand.

Investigating the content of one run artifact folder 

ls -1 <root>/artifacts/001baa39ed8d4b8bb9966e94

Output

log.txt
params.yml
test-human_class_rep.csv
test-human_conf_mtx.csv
test-script_class_rep.csv
test-script_conf_mtx.csv
test-train-val-leftover_class_rep.csv
test-train-val-leftover_conf_mtx.csv
train_class_rep.csv
train_conf_mtx.csv
val_class_rep.csv
val_conf_mtx.csv
best_model_weights_split_2.pt

For each run tcbench creates the following artifacts:

  • params.yml is a YAML file collecting parameters used when triggering a run, i.e., both the arguments explicitly defined on the command line, as well the ones with default values.

  • log.txt collects the console output generated by the run.

  • <context>_class_rep.csv contains a classification report. The filename is bounded to the context (i.e., train, val, test) used to generate it.

  • <context>_conf_mtx.csv contains confusion matrix. The filename is bounded to the context (i.e., train, val, test) used to generate it.

  • best_model_weights_split_<split-index>.pt stores the weights of the best trained pytorch model (for a deep learning model). The filename is bounded to the specific split index configured when triggering the run.

  • xgb_model_split_<split-index>.json stores an XGBoost model (when training via xgboost). The filename is bounded to the specific split index configured when triggering the run.