Sampler: Allow multiple masters

For small simulations (e.g. a few seconds), we cannot use the full set of cores efficiently since one master is not able to keep up.

changed the description

closed via commit f563c4ae

The sampler writes now as many JSON files as masters. The ID is however still unique. So, a file may start e.g. with ID 10.

Did you check that random sequences aren't the same on all masters when determining the samples?

reopened

I knew I forgot something. Good point!

For the random case I will just increment the seed. You implemented the sequence, so what's the easiest to do?

I think we can use the allocate function. There the communicator for the sampler is already created.

mentioned in commit 8130684a

mentioned in commit fc978dec

The missing generators that have to be updated are

• SampleGaussianSequence
• SampleSequence

The others should work now. Not yet tested.

mentioned in commit 37b0fba6

mentioned in commit 04061a59

I have to think about how we could split the sequences. Probabely along one (the first) dimension. But I don't know how this has to be done yet.

Maybe we can use the same approach as in FromFile.h

In case of SampleSequence, we need to modify line 37 to something like

sampleNr_m = sampleNr_m % nLocSamples_m + shift

where nLocSamples_m is the number of local samples and shift brings it to the correct global sample number.

That will only work if RASTER=FALSE or with a single dimension (DVAR).

Yes, that's true.

mentioned in commit 8ad7fa3c

mentioned in commit 40505a19

@kraus We have a similar problem (i.e. not same bins for all DVARs) in case of LatinHyperCube. Fixed it by 40505a19. All masters initialize the full set of bins and shuffle it the same way. Then each master selects its range. In this case we have different bins for different DVARs.

I think what you proposed, i.e. splitting just in one dimension, is the simplest.

Or in case of multi-master we follow the approach of the LatinHyperCube, i.e. a list of bins.

@kraus What do you think about a static public counter of SamplingMethod that is used to figure out the id of each sequence dimension? We could then split id == 0 into nMaster bins if the static counter variable is greater than 1.

I haven't looked at your code changes yet, but I think this will not work the way as you described here with multi-dimensional samplings with Latin hypercube. The samplings will look like a block-diagonal matrix (if it were a matrix).

I think the only way to solve this is to split the sampling space the same way as the sc solver splits the grid. The easiest way is along one dimension.

I don't think so because each master selects randomly Nloc bins out of all Ntot bins. So, the full range for all masters. But it is then still guaranteed that each bin occurs only once in each dimension.

Ok, I'll have to look at your changes but can't do it now...

Sure ... here's an example in 2D for LatinHyperCube.

Just looked at the code and it's clear how this works in the case of Latin Hypercube. But the same approach wont work for sequences with RASTER = TRUE. Latin Hypercube works similarly to RASTER = FALSE where in each column and row there is only a single sample.

I'm currently trying to implement the one dimension split, as you proposed.

I see two approaches:

• parallelize along one dimension or
• linearize the sample space and split then evenly using this index.

BTW the masters should still have different seeds. In the case of Latin Hypercube there will be n samples which are at the same position within a hypercube just in different cubes. n is here the number of masters.

They have different seeds line 50 together with line 43. In order to shuffle the indices, there, it requires the same seed.

Sorry I didn't see this.

In case of parallelizing the first dimension, there I fail finding a solution for modulo.

RASTER=FALSE is wrong. RASTER=TRUE fails for 3 masters. I tested with N = 11 (DVAR 1) and N = 8 (DVAR 2)

RASTER=TRUE fails for 3 masters because the Sampler itself does a "fair" splitting of jobs. In case of 2 masters the splitting luckily matched the Sampler one.

RASTER=TRUE

RASTER=FALSE

In the case of RASTER = FALSE all dimensions should be split among the masters. And in the case of RASTER = TRUE only one dimension.

Could you push the changes to the SampleSequence and SampleGaussianSequence classes?

I've only changed SampleSequence so far. Since the Sampler does a splitting itself, it won't work when the sequence is split independently.

mentioned in commit 9d0d36cb

Each sampler master knows its global ids. Maybe we can use this somehow. The unique ids are also stored by the individuals.

RASTER=TRUE

mentioned in commit b7b30609

RASTER=FALSE

RASTER=FALSE	RASTER=TRUE

An example also in 3D. N = 4x8x11

RASTER=FALSE	RASTER=TRUE

I will change the SampleGaussianSequence accordingly and after cleaning the code, close the issue.

mentioned in commit 78b2007d

closed via commit fa3c0f01

closed via commit 78b2007d