Randomness is lost in parfor loop on GPU?

I have a code with a following structure:

% my_code_seed#.m
myoutputs1 = my_function1(myinputs1); 

seed = #; 
rng(seed); 

parfor iter = 1:50
    myouputs2 = my_gpu_function2(myoutputs1, my_function_of_Gaussian);
    result(iter).name = myoutputs2; 
end

save(file_name, 'result'); 

What I want to do is to model some random behavior of something by using this code repeatedly. What I'm doing is to copy this code and change the value of the seed. For example, the "my_code_seed1.m" uses "seed = 1", and "my_code_seed2.m" uses "seed = 2". I run these codes in a parallel way on a GPU cluster to save the computation time. However, it seems like the results from the codes are very similar, even though I'm giving different values of seed.

Do you have an idea why rng(seed); is not working properly in this case? I appreciate any help.

Answer 1

[Steven Lord]

What I'm doing is to copy this code and change the value of the seed. For example, the "my_code_seed1.m" uses "seed = 1", and "my_code_seed2.m" uses "seed = 2".

So if later on you find a bug in my_code_seed42.m you're going to go back and modify the previous 41 files? That's inefficient. Instead write the code once as a function that accepts a seed value and call that function with the various seeds as input.

As for the randomness in a parfor, see this documentation page on repeating random numbers in a parfor loop and the page on controlling randomness to which its first paragraph links.

[Daigo]

Thanks for your advice. Yes, I will make a function that takes a seed as an input.

As for the randomness, I still couldn't figure out. The randomness in the parfor loop is not a problem. The 50 results from the parfor loops are different from each other, which is exactly what I want. My problem is, the jobs are using the same random number stream even though I'm giving the different seed values. I'd like to know how to fix that.

[Edric Ellis]

In the example you show, you're calling rng(seed) on the client, this does not affect the workers operating on the body of the parfor loop. If you're using gpuArray in the body of the parfor loop, then you'll need a combination of these two bits of documentation:

• https://www.mathworks.com/help/parallel-computing/repeat-random-numbers-in-parfor-loops.html - general page showing you how to get reproducible random numbers inside parfor. If you want to have each "job" get a different set, you'll need to adapt that to offset the streams per job
• https://www.mathworks.com/help/parallel-computing/gpurng.html describes how to set up rng state for gpuArray - it is separate from the CPU rng state.

[Daigo]

I appreciate a lot, Edric! I didn't fully understand what the client and workers are, but now it's all clear. I will take a look at the documentations.

[Daigo]

@Edric Ellis I ended up using parallel.gpu.RandStream.create to control the rng state for gpuArray inside of forloop. Here is a toy example:

function [r, time] = example_random_gpu(val_seed)
    tic
    % Example data to be used in parfor loop 
    n = 100; 
    data = reshape(1:n^2, [n, n]); 
    % Create random streams for gpuArray
    streams = parallel.gpu.RandStream.create('Threefry',...
                        'NumStreams', n,...
                        'Seed', val_seed,...
                        'CellOutput', true); 
    r = zeros(3,n,'gpuArray'); 
    % Loop 
    parfor ii = 1:n
        a = sum(data(ii,:))/n; 
        r(ii,:) = func_random(a, n, streams{ii}); 
    end
    % Get the elapsed time
    time = toc; 
end

function c = func_random(a, n, stream)
    c = randn(stream, 1, n) + 1i*a*randn(stream)*gpuArray(linspace(-2,2,n)); 
end

The function example_random_gpu performs as exactly desired - the same r for the same val_seed, but a different r for a different val_seed.

However, I have another question regarding the data access. In this toy example, I have multiple parfor-loops accessing the same constant: data. Is there anyway to allow each worker to access the value of data more efficiently? I tried doing the following but the computation became much slower:

function [r, time] = example_random_gpu2(val_seed)
    tic
    % Example data to be used in parfor loop 
    n = 100; 
    data = reshape(1:n^2, [n, n]); 
    C = parallel.pool.Constant(data); % <----- ADDED 
    % Create random streams for gpuArray
    streams = parallel.gpu.RandStream.create('Threefry',...
                        'NumStreams', n,...
                        'Seed', val_seed,...
                        'CellOutput', true); 
    r = zeros(3,n,'gpuArray'); 
    % Loop 
    parfor ii = 1:n
        a = sum(C.Value(ii,:))/n; % <---- Modified 
        r(ii,:) = func_random(a, n, streams{ii}); 
    end
    % Get the elapsed time
    time = toc; % ---> Computation became slower ... 
end