Storing Live streamed data

Hi, so i am currently working on a project which is taking streamed data from a device to matlab. Now i am using this data to control a prosthesis but i am struggling to write some code that will store the data in an appropriate variable that can run classification (Machine learning) on the data. I have 4 channels worth of data, and the easiest part of this is i dont even need all the data saved, just the most significant feature i.e. only above a threshold.

The streamed data is coming in at a rate of 2000Hz but as i said before i only need the values after the thresholding. I simply am struggling with storing this data, the threhsold is of course easy. I dont know the size of the data to create as its live data which is a problem in itself (Not being able to run machine learning on the whole data).

I have been trying this for a while, if anyone can help it would be greatly appreciated.

Answer 1

[dpb]

Open a file before beginning the process and write whatever it is that is wanted...formatted data is simple to read but bulky, stream data is compact and the fastest way...not having been given any of your code variables can't try to fit into whatever it is you do have, but the idea is

fid=fopen('LoggingFile.dat','w');
collecting=true;
while collecting
  x=readdatafromsource;
  y=threshold(x);
  fwrite(fid,y,'double')
end
fid=fclose(fid);

The above presumes some other manner such as a pushbutton callback that sets the variable collecting to false to stop the collection process; you've apparently got all that worked out already.

Then just

fid=fopen('LoggingFile.dat','r');
data=fread(fid);
fid=fclose(fid);

will give you the saved data in a 2D array, assuming the x data are a 4-vector as described for each collection cycle.

ADDENDUM:

Alternatively, if you have sufficent memory, hold the data in memory and use a .mat file at the end.

[BionicP]

This is absolutely amazing, i think i have got the general idea you are presenting me with and will try to implement!! I am really grateful for this, just for reference i will add my code (sorry for not giving it before i just didnt think haha XD)

%% instantiate the library
disp('Loading the library...');
lib = lsl_loadlib();

%% resolve a stream...
disp('Resolving an EEG stream...');
result = {};
while isempty(result)
    result = lsl_resolve_byprop(lib,'type','EEG'); end

%% create a new inlet
disp('Opening an inlet...');
inlet = lsl_inlet(result{1});

%% Display Data
disp('Now receiving chunked data...');
while true
    % get chunk from the inlet
    [chunk,stamps] = inlet.pull_chunk();
    for s=1:length(stamps)
        % and display it
        fprintf('%.1f\t',chunk(:,s));
        fprintf('%.1f\n',stamps(s));
    v = chunk(:,s);
    s = stamps(s);
    pause(0.0005);   
%% Storing data in a variable

   c1 = v(1,:);
   c2 = v(2,:);
   c3 = v(3,:);
   c4 = v(4,:);
    end
 %% Moving RMS 
len1 = length(chunk(1,:));
len2 = length(chunk(2,:));
len3 = length(chunk(3,:));
len4 = length(chunk(4,:));
lenv = length(chunk);

array_w=[];
A=zeros(1,lenv);
B=zeros(1,lenv);
C=zeros(1,lenv);
D=zeros(1,lenv);
w = length(chunk);

%% Edited window for RMS on the vec(i)
% for i=1:lenv
%     if (w < lenv)
%         
%         if (i<=w)
%             %When data smaller than window
%             %take RMS of no of samples available
%             array_w = [array_w rms(vec(i))];
%                 Y(i) = rms(vec(i));
%         else 
%             Y(i) = Y(i-1) + (rms(vec(i))-array_w(1))/w;
%             array_w = [array_w(2:end) rms(vec(i))];
%         end
%     elseif (w == lenv)
%         Y = rms(vec(i));
%         break;
%     end
%% RMS for channel 1
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c1(i))];
                Y(i) = rms(c1(i));
        else 
            Y(i) = Y(i-1) + (rms(c1(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c1(i))];
        end
    elseif (w == lenv)
        A = rms(c1(i));
        break;
    end
    end
plot(s,A,'.');
hold on

%% RMS for channel 2
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c2(i))];
                Y(i) = rms(c2(i));
        else 
            Y(i) = Y(i-1) + (rms(c2(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c2(i))];
        end
    elseif (w == lenv)
        B = rms(c2(i));
        break;
    end
    end
plot(s,B,'.');
hold on

%% RMS for channel 3
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c3(i))];
                Y(i) = rms(c3(i));
        else 
            Y(i) = Y(i-1) + (rms(c3(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c3(i))];
        end
    elseif (w == lenv)
        C = rms(c3(i));
        break;
    end
    end
plot(s,C,'.');
hold on

%% RMS for channel 4
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c4(i))];
                Y(i) = rms(c4(i));
        else 
            Y(i) = Y(i-1) + (rms(c4(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c4(i))];
        end
    elseif (w == lenv)
        D = rms(c4(i));
        break;
    end
    end
plot(s,D,'.');
hold on   

end

[BionicP]

Hey, so I have tried implementing this into my code but i am coming into the issue that when I run the code, after its done i go into a text file i created instead and its still not writing anything into it! Any advice would be appreciated, code is as below. I have incorporated partially what you said, the while collecting statement is already in my data aquisition section when recieivng the data so i thought it would be best at the end of the code if i write the feature extracted data into a file and see if this works, but i dont get anything in the file!

%% instantiate the library
disp('Loading the library...');
lib = lsl_loadlib();

%% resolve a stream...
disp('Resolving an EEG stream...');
result = {};
while isempty(result)
    result = lsl_resolve_byprop(lib,'type','EEG'); end

%% create a new inlet
disp('Opening an inlet...');
inlet = lsl_inlet(result{1});

%% Display Data
disp('Now receiving chunked data...');
while true
    % get chunk from the inlet
    [chunk,stamps] = inlet.pull_chunk();
    for s=1:length(stamps)
        % and display it
        fprintf('%.1f\t',chunk(:,s));
        fprintf('%.1f\n',stamps(s));
    v = chunk(:,s);
    s = stamps(s);
    pause(0.0005);   
%% Storing data in a variable

   c1 = v(1,:);
   c2 = v(2,:);
   c3 = v(3,:);
   c4 = v(4,:);
    end
 %% Moving RMS 
len1 = length(chunk(1,:));
len2 = length(chunk(2,:));
len3 = length(chunk(3,:));
len4 = length(chunk(4,:));
lenv = length(chunk);

array_w=[];
A=zeros(1,lenv);
B=zeros(1,lenv);
C=zeros(1,lenv);
D=zeros(1,lenv);
w = length(chunk);

%% Edited window for RMS on the vec(i)
% for i=1:lenv
%     if (w < lenv)
%         
%         if (i<=w)
%             %When data smaller than window
%             %take RMS of no of samples available
%             array_w = [array_w rms(vec(i))];
%                 Y(i) = rms(vec(i));
%         else 
%             Y(i) = Y(i-1) + (rms(vec(i))-array_w(1))/w;
%             array_w = [array_w(2:end) rms(vec(i))];
%         end
%     elseif (w == lenv)
%         Y = rms(vec(i));
%         break;
%     end
%% RMS for channel 1
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c1(i))];
                Y(i) = rms(c1(i));
        else 
            Y(i) = Y(i-1) + (rms(c1(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c1(i))];
        end
    elseif (w == lenv)
        A = rms(c1(i));
        break;
    end
    end
    subplot(4,1,1)
plot(s,A,'.');
hold on

%% RMS for channel 2
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c2(i))];
                Y(i) = rms(c2(i));
        else 
            Y(i) = Y(i-1) + (rms(c2(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c2(i))];
        end
    elseif (w == lenv)
        B = rms(c2(i));
        break;
    end
    end
    subplot(4,1,2)
plot(s,B,'.');
hold on

%% RMS for channel 3
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c3(i))];
                Y(i) = rms(c3(i));
        else 
            Y(i) = Y(i-1) + (rms(c3(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c3(i))];
        end
    elseif (w == lenv)
        C = rms(c3(i));
        break;
    end
    end
    subplot(4,1,3)
plot(s,C,'.');
hold on

%% RMS for channel 4
    for i=1:lenv
    if (w < lenv)
        
        if (i<=w)
            %When data smaller than window
            %take RMS of no of samples available
            array_w = [array_w rms(c4(i))];
                Y(i) = rms(c4(i));
        else 
            Y(i) = Y(i-1) + (rms(c4(i))-array_w(1))/w;
            array_w = [array_w(2:end) rms(c4(i))];
        end
    elseif (w == lenv)
        D = rms(c4(i));
        break;
    end
    end
    subplot(4,1,4)
plot(s,D,'.');
hold on   
       
fid=fopen('LoggingFile.dat','wt+');
data = horzcat(A,B,C,D);
fwrite(fid, data, 'double')
fid = fclose(fid);

[dpb]

Set a breakpoint at the point you open the output file and see what the various arrays contain.

How did you determine there's nothing in the file? Since you wrote it as a stream file, it won't show anything readable in a text editor, for example.

[BionicP]

problem is, if i put a break in at the end, it just cycles through one cycle of streamed info therefore its not useful enough for the whole thing. Even so when i try to simply run the file my way i assessed nothing is coming up when i try to open the dat file in matlab and it says that it cant open an empty file :( Some backstory; im using this for a real time application in prosthetic control so all the classification and everything i am doing is ideally in real time too, breaks will cause alot of latency right?

Once again, thank you so much for the help, i look forward to hearing back

[dpb]

I'm just telling you to use breakpoints to debug your code...you've got to see what you're really getting there that are trying to write. The fwrite itself looks fine so if there's nothing in the file that implies the data arrays are empty...if that is so, then you've got to backtrack to find out what's going wrong.

[BionicP]

ohh ok sorry i understand! I will attempt to debug and see what the problem is

[dpb]

This is tougher for us to help with; we don't have the means to run the code so you've got to do the heavy lifting. We can give advice or answer syntax errors or even help diagnose further symptoms, but that's about all...

[BionicP]

No of course, i think you have helped alot anyways haha but if i do come across further issues pertaining to this part, ill try ask and see if you can help! For now though, i should be able to go from here or atleast with the idea you have given me !!

Once again, Thanks