|
|
= HowTo Create a Regression Test
|
|
|
:TOC:
|
|
|
|
|
|
We run nightly regression tests for OPAL quality assurance. The
|
|
|
results are published on a [http://amas.web.psi.ch/regressiontests/
|
|
|
webpage]. The following sections explain how a regression test can be
|
|
|
included in our nightly runs.
|
|
|
|
|
|
We provide a python script generating regression tests (beta version)
|
|
|
([attachment:generate-regressiontest.py
|
|
|
generate-regressiontest.py]). Please read the
|
|
|
[#AutoGeneratingRegressionTests instructions] carefully.
|
|
|
|
|
|
== Regression Tests Files and Directory Layout
|
|
|
|
|
|
To create a new regression test the following files have to be created
|
|
|
in a directory called `RegressionTestName`
|
|
|
|
|
|
1. `RegressionTestName.rt`: [#SpecifyingRegressionTests Specify] the variables tested against reference data
|
|
|
2. `RegressionTestName.sge`: The [#ExamplesgeFile sge script] submitted to the batch system to run the regression test
|
|
|
3. `RegressionTestName.local`: The [#ExamplelocalFile local script] executed to run the test when `--run-local` is specified
|
|
|
4. `RegressionTestName.in`: The input file of the simulation
|
|
|
|
|
|
Any deviation from this naming scheme will result in a failure when the cron script is trying to run your regression test. Additionally there has to be a directory called `reference` holding the reference data to compare the nightly runs against. This directory contains the following files:
|
|
|
|
|
|
1. `RegressiontestName.lbal`: The reference load balancing file
|
|
|
2. `RegressionTestName.lbal.md5`: The md5sum of the reference load balancing file
|
|
|
3. `RegressionTestName.out`: The simulation output file
|
|
|
4. `RegressionTestName.out.md5`: The md5sum of the simulation output file
|
|
|
5. `RegressionTestName.stat`: The stat file generated by the simulation
|
|
|
6. `RegressionTestName.stat.md5`: The md5sum of the stat file
|
|
|
|
|
|
[#Creatingmd5sums Creating md5sums] shows how md5sums can be generated for reference files. All these files have to be placed in `opal-Tests/RegressionTests/` and be committed to the svn repository in order be run nightly. You can omit the svn commit to only use the regression test locally by running
|
|
|
`
|
|
|
run-regression-test.py --user --run-local
|
|
|
}}}
|
|
|
|
|
|
Examples of working regression tests are located in `opal-Tests/RegressionTests/`.
|
|
|
|
|
|
=== Summary
|
|
|
|
|
|
The directory tree should look like this:
|
|
|
|
|
|
----
|
|
|
opal-Tests/RegressionTests/RegressionTestName
|
|
|
RegressionTestName.rt
|
|
|
RegressionTestName.sge
|
|
|
RegressionTestName.in
|
|
|
reference
|
|
|
RegressiontestName.lbal
|
|
|
RegressionTestName.lbal.md5
|
|
|
RegressionTestName.out
|
|
|
RegressionTestName.out.md5
|
|
|
RegressionTestName.stat
|
|
|
RegressionTestName.stat.md5
|
|
|
----
|
|
|
|
|
|
== Specifying Regression Tests
|
|
|
|
|
|
A sample `rt` file looks like this:
|
|
|
|
|
|
----
|
|
|
"External field test (without space charge)"
|
|
|
stat "rms_x" last 1E-15 #this is a comment
|
|
|
stat "rms_y" last 1E-15 #this is a comment
|
|
|
stat "rms_s" last 1E-15 #this is a comment
|
|
|
----
|
|
|
|
|
|
They start with a description of the regression test that will be
|
|
|
displayed on the web-page results. The rest of the file specifies what
|
|
|
will be compared against the reference data (one line per test). In
|
|
|
the above example we test the variable "rms_x" from the `stat` file by
|
|
|
using the `last` value found in the file with a precision of
|
|
|
`1E-15`. Everything after a `#` sign will be treated as comment and
|
|
|
therefore will be ignored.
|
|
|
|
|
|
Choices for files are: `lbal`, `out`, `stat` <br/>
|
|
|
Choices for variables are defined in the files <br/>
|
|
|
Choices for data points to compare are: `last`, `avg`
|
|
|
|
|
|
In case of data from the '''PROBE''' element, you can compare "x",
|
|
|
"y", "z", "px", "py", "pz", "track_id", "turn" and "time".
|
|
|
|
|
|
* quantity: string that defines how the variable should be handled. Options are "all" (other options not implemented) "all" test fails if any particles in any plane in the loss file have variable variable_(ref) > tolerance
|
|
|
* tolerance: floating point tolerance (absolute)
|
|
|
* file_name: name of the loss file to be checked
|
|
|
|
|
|
Note that
|
|
|
* Output in the loss file is assumed to be that of a PROBE element.
|
|
|
* If a line of output is not compatible with PROBE output, test will ignore the line (not fail).
|
|
|
* Test will always fail if no valid data was found in the loss file
|
|
|
or the loss file could not be opened.
|
|
|
* Particles are grouped into plane according to a unique combination
|
|
|
of <Turn id> and <Element id>
|
|
|
|
|
|
A sample `rt` file looks like this:
|
|
|
|
|
|
----
|
|
|
"One hundred turn RK-4 tracking in ERIT FFAG using SBend3D; no space charge"
|
|
|
PROBE1.loss "x" all 1E-15
|
|
|
PROBE1.loss "y" all 1E-15
|
|
|
----
|
|
|
|
|
|
|
|
|
|
|
|
=== Example `sge` file
|
|
|
|
|
|
A sample `sge` file looks like this:
|
|
|
|
|
|
----
|
|
|
#!/bin/bash
|
|
|
#$ -cwd
|
|
|
#$ -j y
|
|
|
#$ -pe mpi 4
|
|
|
#$ -N ExternalFieldTest-RT
|
|
|
#$ -v OPENMPI,LD_LIBRARY_PATH,OPAL_EXE_PATH,REG_TEST_DIR
|
|
|
|
|
|
MACHINE_FILE=$TMPDIR/machinefile
|
|
|
awk '/^felsim/ {print $1" slots="$2}' $PE_HOSTFILE > $MACHINE_FILE
|
|
|
cp $MACHINE_FILE machinefile.last
|
|
|
|
|
|
echo "PE_HOSTFILE:"
|
|
|
cat $PE_HOSTFILE
|
|
|
echo "MACHINE_FILE:"
|
|
|
cat $MACHINE_FILE
|
|
|
echo "SLOTS=$NSLOTS"
|
|
|
|
|
|
cd $REG_TEST_DIR
|
|
|
cp TwoStep.h5 ExternalFieldTest.h5
|
|
|
OPAL="$OPAL_EXE_PATH/opal ExternalFieldTest.in --commlib mpi --info 0 --warn 0 2>&1"
|
|
|
CMD="$MPIHOME/bin/mpirun -machinefile $MACHINE_FILE -np $NSLOTS --mca ras localhost --mca pls rsh $OPAL "
|
|
|
$CMD
|
|
|
----
|
|
|
|
|
|
|
|
|
It is very important to include the `OPAL_EXE_PATH` and `REG_TEST_DIR`
|
|
|
on line 6 and using these variables as shown in the example. Otherwise
|
|
|
the regressiontests will not run properly.
|
|
|
|
|
|
=== Example `local` file
|
|
|
|
|
|
A sample `local` file looks like this:
|
|
|
|
|
|
----
|
|
|
#!/bin/bash
|
|
|
cp TwoStep.h5 ExternalFieldTest.h5
|
|
|
mpirun -np 4 $OPAL_EXE_PATH/opal ExternalFieldTest.in --commlib mpi --info 0 --warn 0 2>&1
|
|
|
----
|
|
|
|
|
|
It is very important to make the `local` file executable by running `chmod +x file.local`.
|
|
|
|
|
|
|
|
|
=== Creating md5sums ===
|
|
|
|
|
|
To create the md5sums of the reference files execute the following:
|
|
|
|
|
|
----
|
|
|
md5sum ExternalFieldTest.out > ExternalFieldTest.out.md5
|
|
|
----
|
|
|
|
|
|
in the terminal.
|
|
|
|
|
|
== Auto Generating Regression Tests
|
|
|
|
|
|
*This currently only works with simulations not requiring any pre-run setup (e.g. restarted runs wont work)*
|
|
|
|
|
|
1. Download the script [attachment:generate-regressiontest.py].
|
|
|
2. Put it in the directory containing a working OPAL simulation (needs inputfile and all T7's, dat and phases files)
|
|
|
3. execute:
|
|
|
----
|
|
|
./generate-regressiontest.py SimulationName.in
|
|
|
----
|
|
|
|
|
|
This will generate a new folder called `SimulationName` containing a
|
|
|
template `rt` file and generated `sge` and reference files. To report
|
|
|
bugs please send an email to yves.ineichen@psi.ch.
|
|
|
|
|
|
== Locally Run Regression Tests
|
|
|
|
|
|
1. set the environment variable `OPAL_EXE_PATH`
|
|
|
2. set the `$REG_TEST_DIR` for example `export REG_TEST_DIR=$OPAL_ROOT/opal-Test/RegressionTests`
|
|
|
3. set the `$REGTEST_WWW` for example `export REGTEST_WWW=$HOME/regtest-www`
|
|
|
4. `cd $REG_TEST_DIR/run`
|
|
|
5. `./run-reg-tests.py --user --run-local --tests=ParallelPlateFurman1`
|
|
|
|
|
|
To run more than one test, simply add addition names separated by coma: `--tests=test1,test2,test3` |