... | ... | @@ -12,23 +12,18 @@ |
|
|
== Indroduction
|
|
|
|
|
|
We run nightly regression tests for OPAL quality assurance. The
|
|
|
results are published on [http://amas.web.psi.ch/opal/regressionTests/master/].
|
|
|
results are published on http://amas.web.psi.ch/opal/regressionTests/master/.
|
|
|
The following sections explain how a regression test can be
|
|
|
included in our nightly runs.
|
|
|
|
|
|
We provide a python script generating regression tests (beta version)
|
|
|
([attachment:generate-regressiontest.py
|
|
|
generate-regressiontest.py]). Please read the
|
|
|
[#AutoGeneratingRegressionTests instructions] carefully.
|
|
|
|
|
|
== Regression Tests Files and Directory Layout
|
|
|
|
|
|
To create a new regression test the following files have to be created
|
|
|
in a directory called `RegressionTestName`
|
|
|
|
|
|
1. `RegressionTestName.rt`: [#SpecifyingRegressionTests Specify] the variables tested against reference data
|
|
|
2. `RegressionTestName.sge`: The [#ExamplesgeFile sge script] submitted to the batch system to run the regression test
|
|
|
3. `RegressionTestName.local`: The [#ExamplelocalFile local script] executed to run the test when `--run-local` is specified
|
|
|
1. `RegressionTestName.rt`: <<specifying-regression-tests,Specify>> the variables tested against reference data
|
|
|
2. `RegressionTestName.sge`: The <<example-sge-file,sge script>> submitted to the batch system to run the regression test
|
|
|
3. `RegressionTestName.local`: The <<example-local-file,local script>> executed to run the test when `--run-local` is specified
|
|
|
4. `RegressionTestName.in`: The input file of the simulation
|
|
|
|
|
|
Any deviation from this naming scheme will result in a failure when the cron script is trying to run your regression test. Additionally there has to be a directory called `reference` holding the reference data to compare the nightly runs against. This directory contains the following files:
|
... | ... | @@ -40,7 +35,7 @@ Any deviation from this naming scheme will result in a failure when the cron scr |
|
|
5. `RegressionTestName.stat`: The stat file generated by the simulation
|
|
|
6. `RegressionTestName.stat.md5`: The md5sum of the stat file
|
|
|
|
|
|
[#Creatingmd5sums Creating md5sums] shows how md5sums can be generated for reference files. All these files have to be placed in `opal-Tests/RegressionTests/` and be committed to the svn repository in order be run nightly. You can omit the svn commit to only use the regression test locally by running
|
|
|
<<creating-md5sums,Creating md5sums>> shows how md5sums can be generated for reference files. All these files have to be placed in `opal-Tests/RegressionTests/` and be committed to the svn repository in order be run nightly. You can omit the svn commit to only use the regression test locally by running
|
|
|
`
|
|
|
run-regression-test.py --user --run-local
|
|
|
}}}
|
... | ... | @@ -172,17 +167,7 @@ in the terminal. |
|
|
|
|
|
== Auto Generating Regression Tests
|
|
|
|
|
|
*This currently only works with simulations not requiring any pre-run setup (e.g. restarted runs wont work)*
|
|
|
|
|
|
1. Download the script [attachment:generate-regressiontest.py].
|
|
|
2. Put it in the directory containing a working OPAL simulation (needs inputfile and all T7's, dat and phases files)
|
|
|
3. execute:
|
|
|
----
|
|
|
./generate-regressiontest.py SimulationName.in
|
|
|
----
|
|
|
|
|
|
This will generate a new folder called `SimulationName` containing a
|
|
|
template `rt` file and generated `sge` and reference files.
|
|
|
TODO
|
|
|
|
|
|
== Locally Run Regression Tests
|
|
|
0. `git clone git@gitlab.psi.ch:OPAL/NightlyBuild.git`
|
... | ... | |