Reading update of variable in parallel function?

I need to implement two function in parallel but one of them reads update value of another function .

Details:

i read CSV file " 200" row as stream data (sensor data) in matrix and at same time another function need to read updated matrix

both need to be simulated in parallel

Two parallel function will be implemented

1- function read() read each raw from file in matrix "Datachunk1"

2- function( out ) has if condition

"if condition" will implement sequentional with another function and take too much time than function read

"else condition" will need to read updated matrix Datachunk1" 50 raw" at a time do operations then deleted them from matrix .this process continue till reading all 200 raw."4 chunks"

i have matlab >2016 i didn't try parallel function before.

part of code

function read()
fid = fopen('t.csv');
filename = fullfile(pwd, 't.csv');
while(i ~=200)
   
      i = i + 1;
    next_line = fgetl(fid);
     D=Datachunk1; 
    Datachunk1=cell2mat(textscan(next_line,'%f %f %f %f','Delimiter', ',','CollectOutput', true))
    Datachunk1=[D;Datachunk1];
    
end
end
function out(inputs,position)
if position==0
   call by another function
   /*do some operations 
      
end
else
    Datachunk1
    /*do some operations in Datachunk1
    
end

end

Answer 1

[Walter Roberson]

You have several options:

1. instead of parfor use spmd. spmd permits you to use labSend to send data to another worker
2. you can carefully arrange a series of three parallel queue objects such that you can send data from one worker back to the client, and then have the client send it to the other worker (the workers cannot communicate directly for this purpose)
3. you can look in the File Exchange for the Shared Matrix contribution
4. you can use a shared file, especially if you use memory map (but that is not strictly needed)

When the second worker goes to delete items out of the matrix, that is going to be a problem, as most of these methods will not reflect the deletion back to the first worker. Matlab arrays always live in exactly one process, and the memory management routines are not able to allocate or deallocate on a different process.

[E.B]

thanks so much

i didn't try this options before which one is simplest and get the result ?

if i read row and write in file "x" in stead of matrix then read from this file "x" this will be easier and can be deleted from file every updated row?

[Walter Roberson]

MATLAB itself does not permit you to delete from a file, not in the sense of ending up with a shorter file. MATLAB permits you to copy later information down to earlier in the file, eliminating what was between, but once you get to the end of that and want to say "Okay, now remove the junk from here to end of file" then MATLAB itself does not permit that. The C or C++ code to do file truncation is simple in itself but getting a proper connection between MATLAB's idea of files and C/C++'s idea of files can be a bit tricky.

Because of these factors, it can be easier to reserve a column in each row to indicate whether the row is active or deleted. When you want to write more rows, you can go looking for a large enough block marked deleted. You could potentially also periodically synchronize the two threads, doing a "compacting" .

Or you could use matFile, which is able to handle deleting within a variable. However, the overhead for matFile can really add up; the flexibility can be expensive.

Sometimes you should just use a ring buffer of fixed-length blocks, implemented in a file.

[E.B]

i am trying to use spmd but i got this message and how i control that Receive 50 raw

that i got

"read_chunck1" and "out" two m files functions

Error detected on worker 2.

Caused by:
    Error using sp (line 1)
    Source for operation receive or probe is out of range (valid values: 1 <= source <= 2 or 'any').

spmd(2) 
    switch labindex
        case 1
            % Code for worker 1
            
            Datachunk1 = read_chunck1()
          
            labSend(Datachunk1, 2);
            labBarrier;
        case 2
            % Code for worker 2
          
          labReceive(Datachunk1, 1);
           out(datachunk1,2);    
        ...
    end
end

[Walter Roberson]

        Datachunk1  = labReceive(1);

[E.B]

thanks so much

both lab1 ,2 implemented at same time so why number 6 printed after finish lab 1 as sequential ?

That another error i get

An invalid Composite or distributed array was passed to an SPMD block.
It might have been saved and then loaded, or the parallel pool might have been shut down.
Use parpool to start a new parallel pool.

poolobj = gcp;
addAttachedFiles(poolobj,{'read_chunck1.m','out.m'})
spmd(2)
    
    switch labindex
        case 1
            % Code for worker 1
            
            Datachunk1 = read_chunck1()
          
            labSend(Datachunk1, 2);
            %labBarrier;
        case 2
            % Code for worker 2
           6
           Datachunk1  = labReceive(1)
          %labReceive(Datachunk1, 1);
            out(Datachunk1,1)
            
            
            
        ...
    end
end

[Walter Roberson]

The timing of displaying back to the client is not all that well defined. If you need something more immediately you should use a DataQueue

https://www.mathworks.com/matlabcentral/answers/375808-what-is-the-difference-between-dataqueue-and-pollabledataqueue#answer_298932

[E.B]

ok DataQueue for which problem?

An invalid Composite or distributed array was passed to an SPMD block.?

or implementing lab2 with lab 1?

remove spmd?

[Walter Roberson]

" both lab1 ,2 implemented at same time so why number 6 printed after finish lab 1 as sequential ?"

That problem to be solved with DataQueue

[Walter Roberson]

I do not see anything being passed to spmd, but on the other hand there is that "..." that could be hiding the problem.

[E.B]

if i want timing to use DataQueue with parfor and remove spmd right?

if i want to use spmd and need to send recevied data to another function "out" how i handle this error "invalid Composite or distributed array" it printed well but not send to function ?

Datachunk1  = labReceive(1)
          %labReceive(Datachunk1, 1);
            out(Datachunk1,1)

the problem that it first time to write spmd or know about DataQueue so i am confused.

thanks

[E.B]

i tried this

Q = parallel.pool.DataQueue;
afterEach(Q,@out);

spmd
    
    switch labindex
        case 1
            % Code for worker 1
            
            Datachunk1 = read_chunck1();
          
            labSend(Datachunk1, 2);
            %labBarrier;
        case 2
            % Code for worker 2
           6
           Datachunk1  = labReceive(1)
          %labReceive(Datachunk1, 1);
          send(Q, Datachunk1 );        
        ...
    end
end

[Walter Roberson]

Sorry, I think you are going to need to use a different programming language.

[E.B]

what i need to do .i can't implement with matlab.

or using matlab with another language?

[Walter Roberson]

We seem to be unable to figure out how to assist you in implementing your needs in MATLAB.