Skip to content

GitLab

Last edited by Jochem Snuverink
Page history

Home

HowTo Create a Regression Test

note

The content of this page is outdated

1. Introduction

We run nightly regression tests for OPAL quality assurance. The results are published on http://amas.web.psi.ch/opal/regressionTests/master/. The following sections explain how a regression test can be included in our nightly runs.

2. Regression Tests Files and Directory Layout

To create a new regression test the following files have to be created in a directory called RegressionTestName

  1. RegressionTestName.rt: Specify the variables tested against reference data

  2. RegressionTestName.local: The local script executed to run the test when --run-local is specified

  3. RegressionTestName.in: The input file of the simulation

Any deviation from this naming scheme will result in a failure. Additionally there has to be a directory called reference holding the reference data to compare the nightly runs against. This directory contains the following files:

  1. RegressiontestName.lbal: The reference load balancing file

  2. RegressionTestName.lbal.md5: The md5sum of the reference load balancing file

  3. RegressionTestName.out: The simulation output file

  4. RegressionTestName.out.md5: The md5sum of the simulation output file

  5. RegressionTestName.stat: The stat file generated by the simulation

  6. RegressionTestName.stat.md5: The md5sum of the stat file

Creating md5sums shows how md5sums can be generated for reference files. All these files have to be placed in regression-tests/RegressionTests/ and be committed to the repository in order be run nightly.

Examples of working regression tests are located in regression-tests/RegressionTests/.

2.1. Summary

The directory tree should look like this:

regression-tests/RegressionTests/RegressionTestName
   RegressionTestName.local
   RegressionTestName.in
   RegressionTestName.rt
   reference
      RegressiontestName.lbal
      RegressionTestName.lbal.md5
      RegressionTestName.out
      RegressionTestName.out.md5
      RegressionTestName.stat
      RegressionTestName.stat.md5

3. Specifying Regression Tests

A sample rt file looks like this:

"External field test (without space charge)"
stat "rms_x" last 1E-15    #this is a comment
stat "rms_y" last 1E-15    #this is a comment
stat "rms_s" last 1E-15    #this is a comment

They start with a description of the regression test that will be displayed on the web-page results. The rest of the file specifies what will be compared against the reference data (one line per test). In the above example we test the variable rms_x from the stat file by using the last value found in the file with a precision of 1E-15. Everything after a # sign will be treated as comment and therefore will be ignored.

Choices for files are: lbal, out, stat

Choices for variables are defined in the files

Choices for data points to compare are: last, avg

In case of data from the PROBE element, you can compare x, y, z, px, py, pz, track_id, turn and time.

  • quantity: string that defines how the variable should be handled. Options are all (other options not implemented) all test fails if any particles in any plane in the loss file have variable variable_(ref) > tolerance

  • tolerance: floating point tolerance (absolute)

  • file_name: name of the loss file to be checked

Note that

  • Output in the loss file is assumed to be that of a PROBE element.

  • If a line of output is not compatible with PROBE output, test will ignore the line (not fail).

  • Test will always fail if no valid data was found in the loss file or the loss file could not be opened.

  • Particles are grouped into plane according to a unique combination of <Turn id> and <Element id>

A sample rt file looks like this:

"One hundred turn RK-4 tracking in ERIT FFAG using SBend3D; no space charge"
PROBE1.loss "x" all 1E-15
PROBE1.loss "y" all 1E-15

3.1. Example local file

A sample local file looks like this:

#!/bin/bash
cp TwoStep.h5 ExternalFieldTest.h5
mpirun -np 4 $OPAL_EXE_PATH/opal ExternalFieldTest.in --commlib mpi --info 0 --warn 0 2>&1

It is very important to make the local file executable by running chmod +x file.local.

3.2. Creating md5sums

To create the md5sums of the reference files execute the following in the terminal:

md5sum ExternalFieldTest.out > ExternalFieldTest.out.md5

4. Locally Run Regression Tests

  1. git clone git@gitlab.psi.ch:OPAL/NightlyBuild.git

  2. goto the scripts directory, where run-reg-tests.py is present (you get this from the clone above)

  3. set the environment variable OPAL_EXE_PATH

  4. set the $REG_TEST_DIR for example export REG_TEST_DIR=$OPAL_ROOT/regression-tests/RegressionTests

  5. set the $REGTEST_WWW for example export REGTEST_WWW=$HOME/regtest-www

  6. python run-reg-tests.py --opal-exe-path=${OPAL_EXE_PATH} --base-dir=${REG_TEST_DIR} RingCyclotron

To run more than one test, simply add additional names separated by a space : test1 test2 test3