Developing Benchmarks: Glossary

Snakemake allows one to run their data analyses and share them with others

Create setup.config to switch between using the simulation campaign and re-simulating events

Each stage of the benchmark pipeline is defined in config.yml

config.yml takes normal bash scripts as input

Copy resulting figures over to the results directory to turn them into artifacts

Status flags are used to indicate detrimental changes to software/detector design

Add a status flag to your benchmark to alert developers to changes in performance

Create paper-ready benchmark figures whenever possible

Clearly label plots with simulation details, and large axis labels and legends

If possible, augment the benchmark with an additional explainer document which describes figures in greater detail