... | @@ -12,69 +12,69 @@ symbol in the template file will look like **\_CHARGE\_**. |
... | @@ -12,69 +12,69 @@ symbol in the template file will look like **\_CHARGE\_**. |
|
|
|
|
|
# Recognized Environment Variables by the Scripts
|
|
# Recognized Environment Variables by the Scripts
|
|
|
|
|
|
There are 5 environment variables recognized by the {{{runOPAL.py}}}:
|
|
There are 5 environment variables recognized by the *runOPAL.py*:
|
|
|
|
|
|
1. {{{TEMPLATES}}}
|
|
1. *TEMPLATES*
|
|
2. {{{FIELDMAPS}}}
|
|
2. *FIELDMAPS*
|
|
3. {{{OPAL_EXE_PATH}}}
|
|
3. *OPAL_EXE_PATH*
|
|
4. {{{SGE_QUEUE}}}
|
|
4. *SGE_QUEUE*
|
|
5. {{{SGE_RAM}}}
|
|
5. *SGE_RAM*
|
|
|
|
|
|
In Bash parlance:
|
|
In Bash parlance:
|
|
|
|
|
|
{{{
|
|
*
|
|
export TEMPLATES=`pwd`/tmpl/
|
|
export TEMPLATES=`pwd`/tmpl/
|
|
export FIELDMAPS=`pwd`/fieldmaps/
|
|
export FIELDMAPS=`pwd`/fieldmaps/
|
|
export OPAL_EXE_PATH=/gpfs/home/adelmann/build/opal-1.2.0/src/
|
|
export OPAL_EXE_PATH=/gpfs/home/adelmann/build/opal-1.2.0/src/
|
|
export SGE_QUEUE=all.q
|
|
export SGE_QUEUE=all.q
|
|
export SGE_RAM=8
|
|
export SGE_RAM=8
|
|
}}}
|
|
*
|
|
|
|
|
|
From the TEMPLATES directory the {{{foo.tmpl}}} file is taken and the values are replaced.
|
|
From the TEMPLATES directory the *foo.tmpl* file is taken and the values are replaced.
|
|
|
|
|
|
The field maps from the {{{FIELDMAPS}}} directory are linked to the directory where the simulation is executed.
|
|
The field maps from the *FIELDMAPS* directory are linked to the directory where the simulation is executed.
|
|
|
|
|
|
{{{OPAL_EXE_PATH}}} maybe depends on the loaded opal module. If a {{{setup.sh}}} script is available and executable, runOPAL.py will source it.
|
|
*OPAL_EXE_PATH* maybe depends on the loaded opal module. If a *setup.sh* script is available and executable, runOPAL.py will source it.
|
|
|
|
|
|
{{{SGE_QUEUE}}} is the queue used for the simulation. Different queues may have different numbers of nodes and CPU's available as well as different run-time limitations.
|
|
*SGE_QUEUE* is the queue used for the simulation. Different queues may have different numbers of nodes and CPU's available as well as different run-time limitations.
|
|
|
|
|
|
{{{SGE_RAM}}} contains the number of GB of RAM that each CPU will allocate (if not specified, default is 4). If there is not enough memory available on one node, the node will not
|
|
*SGE_RAM* contains the number of GB of RAM that each CPU will allocate (if not specified, default is 4). If there is not enough memory available on one node, the node will not
|
|
be fully loaded. Instead the number of CPU's will be distributed on as many nodes as needed to fulfil the RAM requirement. Merlin has nodes with 64 and 128 GB of RAM. Each node has 16 CPU's.
|
|
be fully loaded. Instead the number of CPU's will be distributed on as many nodes as needed to fulfil the RAM requirement. Merlin has nodes with 64 and 128 GB of RAM. Each node has 16 CPU's.
|
|
|
|
|
|
|
|
|
|
== Here a snippet of a data and tmpl file ({{{foo.data}}})
|
|
== Here a snippet of a data and tmpl file (*foo.data*)
|
|
|
|
|
|
{{{
|
|
*
|
|
# This is a comment
|
|
# This is a comment
|
|
Q -1 # the charge
|
|
Q -1 # the charge
|
|
NPART 100000 # the number of simulation particles
|
|
NPART 100000 # the number of simulation particles
|
|
...
|
|
...
|
|
CORES 1 # how many cores are used by OPAL
|
|
CORES 1 # how many cores are used by OPAL
|
|
}}}
|
|
*
|
|
|
|
|
|
The corresponding part of the template ({{{foo.tmpl}}}) file would look like:
|
|
The corresponding part of the template (*foo.tmpl*) file would look like:
|
|
|
|
|
|
{{{
|
|
*
|
|
beam1: BEAM, PARTICLE=ELECTRON, pc=P0, NPART=_NPART_, BFREQ=BFREQ, BCURRENT=BCURRENT, CHARGE=_Q_;
|
|
beam1: BEAM, PARTICLE=ELECTRON, pc=P0, NPART=_NPART_, BFREQ=BFREQ, BCURRENT=BCURRENT, CHARGE=_Q_;
|
|
}}}
|
|
*
|
|
|
|
|
|
You can run the {{{runOPAL.py}}} script without arguments, if the two files ({{{foo.tmpl}}} and {{{foo.data}}}) are present, a directory {{{foo}}} will be
|
|
You can run the *runOPAL.py* script without arguments, if the two files (*foo.tmpl* and *foo.data*) are present, a directory *foo* will be
|
|
created and all files for the simulation are either copied or linked to that directory. The simulation is then started from this directory
|
|
created and all files for the simulation are either copied or linked to that directory. The simulation is then started from this directory
|
|
where also all results are stored.
|
|
where also all results are stored.
|
|
|
|
|
|
== The general syntax is: ==
|
|
== The general syntax is: ==
|
|
|
|
|
|
{{{
|
|
*
|
|
runOPAL.py [--restart-file=FILE [--restart-step=STEPNR | --restart-pos=POS]] [--test] [--block] [--keep] [--nobatch] [ATTR=SCANVALUE] {[ATTR=VALUE]}
|
|
runOPAL.py [--restart-file=FILE [--restart-step=STEPNR | --restart-pos=POS]] [--test] [--block] [--keep] [--nobatch] [ATTR=SCANVALUE] {[ATTR=VALUE]}
|
|
}}}
|
|
*
|
|
|
|
|
|
* {{{--test}}} exercises everything except for the submission of the job.
|
|
* *--test* exercises everything except for the submission of the job.
|
|
* {{{--restart-pos}}} specifies the position (in meter) defining the restart of the simulation. If no data has been dumped at that position {{{runOpal}}} will use the nearest position stored in the restart file as restart position.
|
|
* *--restart-pos* specifies the position (in meter) defining the restart of the simulation. If no data has been dumped at that position *runOpal* will use the nearest position stored in the restart file as restart position.
|
|
* {{{--restart-step}}} specifies the restart step of the simulation.
|
|
* *--restart-step* specifies the restart step of the simulation.
|
|
* {{{--block}}} runs opal local not using the batch system and waits until the job is done.
|
|
* *--block* runs opal local not using the batch system and waits until the job is done.
|
|
* {{{--keep}}} if same simulation has been run before, keep old data and abort.
|
|
* *--keep* if same simulation has been run before, keep old data and abort.
|
|
* {{{--nobatch}}} uses plain mpirun on the local machine.
|
|
* *--nobatch* uses plain mpirun on the local machine.
|
|
* {{{SCANVALUE=start:end:step}}} scans a parameter space, e.g., {{{TFWHM=0.85:0.90:0.01}}}.
|
|
* *SCANVALUE=start:end:step* scans a parameter space, e.g., *TFWHM=0.85:0.90:0.01*.
|
|
|
|
|
|
=== Cautions ===
|
|
=== Cautions ===
|
|
|
|
|
... | @@ -83,7 +83,7 @@ where also all results are stored. |
... | @@ -83,7 +83,7 @@ where also all results are stored. |
|
|
|
|
|
|
|
|
|
== Example 1: Regular Run ==
|
|
== Example 1: Regular Run ==
|
|
{{{
|
|
*
|
|
dude:foo adelmann$ source setup.sh
|
|
dude:foo adelmann$ source setup.sh
|
|
dude:foo adelmann$ runOPAL.py --test
|
|
dude:foo adelmann$ runOPAL.py --test
|
|
Simulation directory is foo using OPAL at /gpfs/home/adelmann/build/opal-1.2.0/src/
|
|
Simulation directory is foo using OPAL at /gpfs/home/adelmann/build/opal-1.2.0/src/
|
... | @@ -122,70 +122,70 @@ Parameter set in foo.in are: |
... | @@ -122,70 +122,70 @@ Parameter set in foo.in are: |
|
:::: DPOS1= 0.02
|
|
:::: DPOS1= 0.02
|
|
:::: EDES= 0.250
|
|
:::: EDES= 0.250
|
|
Done with setup of the OPAL simulation but not submitting the job (--test)
|
|
Done with setup of the OPAL simulation but not submitting the job (--test)
|
|
}}}
|
|
*
|
|
After that you will have a directory foo with this content:
|
|
After that you will have a directory foo with this content:
|
|
{{{
|
|
*
|
|
dude:foo adelmann$ ls
|
|
dude:foo adelmann$ ls
|
|
fieldmaps foo foo.data setup.sh tmpl
|
|
fieldmaps foo foo.data setup.sh tmpl
|
|
|
|
|
|
dude:foo adelmann$ ls foo
|
|
dude:foo adelmann$ ls foo
|
|
foo.in run.sge
|
|
foo.in run.sge
|
|
}}}
|
|
*
|
|
|
|
|
|
'''Note''': is setup.sh is available and executable you do not have to source it.
|
|
'''Note''': is setup.sh is available and executable you do not have to source it.
|
|
|
|
|
|
== Example 2: 1D Parmeter Scan ==
|
|
== Example 2: 1D Parmeter Scan ==
|
|
|
|
|
|
{{{
|
|
*
|
|
runOPAL.py --test EDES=0.050:0.250:0.050
|
|
runOPAL.py --test EDES=0.050:0.250:0.050
|
|
...
|
|
...
|
|
dude:foo adelmann$ ls
|
|
dude:foo adelmann$ ls
|
|
fieldmaps foo.data foo_EDES=0.05:0.25:0.05 setup.sh tmpl
|
|
fieldmaps foo.data foo_EDES=0.05:0.25:0.05 setup.sh tmpl
|
|
|
|
|
|
dude:foo adelmann$ ls foo_EDES=0.05:0.25:0.05
|
|
dude:foo adelmann$ ls foo_EDES=0.05:0.25:0.05
|
|
fooEDES=0.05 fooEDES=0.1 fooEDES=0.15 fooEDES=0.2 fooEDES=0.25
|
|
fooEDES=0.05 fooEDES=0.1 fooEDES=0.15 fooEDES=0.2 fooEDES=0.25
|
|
|
|
|
|
}}}
|
|
*
|
|
|
|
|
|
== Example 3: 2D Parmeter Scan ==
|
|
== Example 3: 2D Parmeter Scan ==
|
|
{{{
|
|
*
|
|
|
|
|
|
runOPAL.py --test EDES=0.050:0.250:0.050 POS1=0.4:0.5:0.01
|
|
runOPAL.py --test EDES=0.050:0.250:0.050 POS1=0.4:0.5:0.01
|
|
|
|
|
|
dude:foo adelmann$ ls
|
|
dude:foo adelmann$ ls
|
|
fieldmaps foo.data foo_EDES=0.05:0.25:0.05_POS1=0.4:0.5:0.01 setup.sh tmpl
|
|
fieldmaps foo.data foo_EDES=0.05:0.25:0.05_POS1=0.4:0.5:0.01 setup.sh tmpl
|
|
|
|
|
|
|
|
|
|
dude:foo adelmann$ ls foo_EDES=0.05:0.25:0.05_POS1=0.4:0.5:0.01
|
|
dude:foo adelmann$ ls foo_EDES=0.05:0.25:0.05_POS1=0.4:0.5:0.01
|
|
fooEDES=0.05POS1=0.4 fooEDES=0.05POS1=0.45 fooEDES=0.15POS1=0.4 fooEDES=0.15POS1=0.45 fooEDES=0.1POS1=0.4 fooEDES=0.1POS1=0.45 fooEDES=0.25POS1=0.4 fooEDES=0.25POS1=0.45 fooEDES=0.2POS1=0.4 fooEDES=0.2POS1=0.45
|
|
fooEDES=0.05POS1=0.4 fooEDES=0.05POS1=0.45 fooEDES=0.15POS1=0.4 fooEDES=0.15POS1=0.45 fooEDES=0.1POS1=0.4 fooEDES=0.1POS1=0.45 fooEDES=0.25POS1=0.4 fooEDES=0.25POS1=0.45 fooEDES=0.2POS1=0.4 fooEDES=0.2POS1=0.45
|
|
fooEDES=0.05POS1=0.41 fooEDES=0.05POS1=0.46 fooEDES=0.15POS1=0.41 fooEDES=0.15POS1=0.46 fooEDES=0.1POS1=0.41 fooEDES=0.1POS1=0.46 fooEDES=0.25POS1=0.41 fooEDES=0.25POS1=0.46 fooEDES=0.2POS1=0.41 fooEDES=0.2POS1=0.46
|
|
fooEDES=0.05POS1=0.41 fooEDES=0.05POS1=0.46 fooEDES=0.15POS1=0.41 fooEDES=0.15POS1=0.46 fooEDES=0.1POS1=0.41 fooEDES=0.1POS1=0.46 fooEDES=0.25POS1=0.41 fooEDES=0.25POS1=0.46 fooEDES=0.2POS1=0.41 fooEDES=0.2POS1=0.46
|
|
fooEDES=0.05POS1=0.42 fooEDES=0.05POS1=0.47 fooEDES=0.15POS1=0.42 fooEDES=0.15POS1=0.47 fooEDES=0.1POS1=0.42 fooEDES=0.1POS1=0.47 fooEDES=0.25POS1=0.42 fooEDES=0.25POS1=0.47 fooEDES=0.2POS1=0.42 fooEDES=0.2POS1=0.47
|
|
fooEDES=0.05POS1=0.42 fooEDES=0.05POS1=0.47 fooEDES=0.15POS1=0.42 fooEDES=0.15POS1=0.47 fooEDES=0.1POS1=0.42 fooEDES=0.1POS1=0.47 fooEDES=0.25POS1=0.42 fooEDES=0.25POS1=0.47 fooEDES=0.2POS1=0.42 fooEDES=0.2POS1=0.47
|
|
fooEDES=0.05POS1=0.43 fooEDES=0.05POS1=0.48 fooEDES=0.15POS1=0.43 fooEDES=0.15POS1=0.48 fooEDES=0.1POS1=0.43 fooEDES=0.1POS1=0.48 fooEDES=0.25POS1=0.43 fooEDES=0.25POS1=0.48 fooEDES=0.2POS1=0.43 fooEDES=0.2POS1=0.48
|
|
fooEDES=0.05POS1=0.43 fooEDES=0.05POS1=0.48 fooEDES=0.15POS1=0.43 fooEDES=0.15POS1=0.48 fooEDES=0.1POS1=0.43 fooEDES=0.1POS1=0.48 fooEDES=0.25POS1=0.43 fooEDES=0.25POS1=0.48 fooEDES=0.2POS1=0.43 fooEDES=0.2POS1=0.48
|
|
fooEDES=0.05POS1=0.44 fooEDES=0.05POS1=0.49 fooEDES=0.15POS1=0.44 fooEDES=0.15POS1=0.49 fooEDES=0.1POS1=0.44 fooEDES=0.1POS1=0.49 fooEDES=0.25POS1=0.44 fooEDES=0.25POS1=0.49 fooEDES=0.2POS1=0.44 fooEDES=0.2POS1=0.49
|
|
fooEDES=0.05POS1=0.44 fooEDES=0.05POS1=0.49 fooEDES=0.15POS1=0.44 fooEDES=0.15POS1=0.49 fooEDES=0.1POS1=0.44 fooEDES=0.1POS1=0.49 fooEDES=0.25POS1=0.44 fooEDES=0.25POS1=0.49 fooEDES=0.2POS1=0.44 fooEDES=0.2POS1=0.49
|
|
|
|
|
|
|
|
|
|
}}}
|
|
*
|
|
|
|
|
|
|
|
|
|
== Run a Restart ==
|
|
== Run a Restart ==
|
|
|
|
|
|
In this particular example we assume that a simulation called ''Phase3'' has successfully been executed. The output of the simulation is located
|
|
In this particular example we assume that a simulation called ''Phase3'' has successfully been executed. The output of the simulation is located
|
|
at {{{/gpfs/homefelsim/adelmann/Phase3/FinPhase3/}}}.
|
|
at */gpfs/homefelsim/adelmann/Phase3/FinPhase3/*.
|
|
|
|
|
|
A restart simulation starting from ''step=10'' until ''ZSTOP=13.1'' m would then look like:
|
|
A restart simulation starting from ''step=10'' until ''ZSTOP=13.1'' m would then look like:
|
|
|
|
|
|
{{{
|
|
*
|
|
runOPAL.py --restart-file=/gpfs/homefelsim/adelmann/Phase3/FinPhase3/FinPhase3.h5 --restart-step=10 ZSTOP=13.1
|
|
runOPAL.py --restart-file=/gpfs/homefelsim/adelmann/Phase3/FinPhase3/FinPhase3.h5 --restart-step=10 ZSTOP=13.1
|
|
}}}
|
|
*
|
|
|
|
|
|
|
|
|
|
In this example instead of a restart step, a restart position is specified.
|
|
In this example instead of a restart step, a restart position is specified.
|
|
|
|
|
|
{{{
|
|
*
|
|
runOPAL.py --restart-file=/gpfs/homefelsim/adelmann/Phase3/FinPhase3/FinPhase3.h5 --restart-pos=10.0 ZSTOP=13.1
|
|
runOPAL.py --restart-file=/gpfs/homefelsim/adelmann/Phase3/FinPhase3/FinPhase3.h5 --restart-pos=10.0 ZSTOP=13.1
|
|
}}}
|
|
*
|
|
|
|
|
|
If there exists no phase space dump at the specified position (10.0 meters here), the nearest position will be located and used.
|
|
If there exists no phase space dump at the specified position (10.0 meters here), the nearest position will be located and used.
|
|
|
|
|
|
|
|
|
|
== CAVE: At the moment restart is only possible from the last step dumped in the .h5 file. Also, be careful to use the number of dumps with --restart-step, not the actual number of steps (i.e. if in the original run you used PSDUMPFREQU = 10, you will get 10 dumps per 100 steps. to restart from STEP 100 you will write --restart-step=10). == |
|
== CAVE: At the moment restart is only possible from the last step dumped in the .h5 file. Also, be careful to use the number of dumps with --restart-step, not the actual number of steps (i.e. if in the original run you used PSDUMPFREQU = 10, you will get 10 dumps per 100 steps. to restart from STEP 100 you will write --restart-step=10). |
|
\ No newline at end of file |
|
\ No newline at end of file |