Sampler cleanup with KEEP option not working when directory is present
Summary
Reported by @ext-piot_p on the OPAL mailing list. With the option SAMPLE, KEEP non-empty, non-wanted files should be deleted, but an error message is shown instead when a non-empty subdirectory is present in the simulation directory.
Steps to reproduce
An OPAL Sampler simulation whereby also files in the auxiliary directory (/data) are produced together with the KEEP option of the SAMPLER.
What is the current bug behavior?
An error message is shown and the unwanted files are not (all) deleted.
What is the expected correct behavior?
No error message and all but the files with extensions in the KEEP option should be deleted.
Relevant logs and/or screenshots
OPAL{0}> opal varyTDCphase.in --inputfile=varyTDCphase.tmpl --outfile=phiscan --outdir=phiscan --num-masters=1 --num-coworkers=2 --restartstep=-2147483648 --jsonDumpFreq=1 --nsamples=11 --simtmpdir=/home/piot/OpalSandBox/scan_example/phiscan --templates=/home/piot/OpalSandBox/scan_example/template
7 (PID: 214335) ▶ Worker
1 (PID: 214329) ▶ Sampler
2 (PID: 214330) ▶ Worker
3 (PID: 214331) ▶ Worker
4 (PID: 214332) ▶ Worker
5 (PID: 214333) ▶ Worker
6 (PID: 214334) ▶ Worker
✔ 40 objectives
✔ 1 dvars
0 (PID: 214328) ▶ Pilot
Warning: argument "one-pilot-converge" not found! Using default value (0).
Warning: argument "restartfile" not found! Using default value ().
Warning: argument "restartfile" not found! Using default value ().
Ippl> CommMPI: Started job 1 on host `localhost.localdomain'.
Ippl> CommMPI: Parent process waiting for children ...
Ippl> CommMPI: Child 1 ready.
Ippl> CommMPI: Initialization complete.
Warning: argument "restartfile" not found! Using default value ().
Warning: argument "restartfile" not found! Using default value ().
Ippl> CommMPI: Started job 1 on host `localhost.localdomain'.
Ippl> CommMPI: Parent process waiting for children ...
Ippl> CommMPI: Child 1 ready.
Ippl> CommMPI: Initialization complete.
Warning: argument "restartfile" not found! Using default value ().
Warning: argument "restartfile" not found! Using default value ().
Ippl> CommMPI: Started job 1 on host `localhost.localdomain'.
Ippl> CommMPI: Parent process waiting for children ...
Ippl> CommMPI: Child 1 ready.
Ippl> CommMPI: Initialization complete.
Can't remove file in directory '/home/piot/OpalSandBox/scan_example/phiscan/1', (boost::filesystem::remove: Directory not empty: "/home/piot/OpalSandBox/scan_example/phiscan/1/data")Possible fixes
The function OpalSimulation::Cleanup(const std::vector<std::string>& keep)
does not take into account the possibility that a subdirectory might be present. It could check if the path is a directory and if so use boost::filesystem::remove_all (or the c++17 equivalent) instead of boost::filesystem::remove to remove these non-empty subdirectories. Possibly even better would be if the subdirectories are traversed and files with proper extensions are kept.