Large array

Hi I would like to plot and process large data array with a GUI. The problem is that with small data array, the user interface works correctly but with large array the plotting (1D and 2D plot) encounters big delays and the interactivity (scrolling, updating...) becomes impossible. What's the best way to plot, process and interact with large data arrays ? Thanks.

Answer 1

[Sean de Wolski]

How are you plotting? What is the best way to visualize your data that requires minimal memory?

The best way would probably include a few steps:

• Instead of plotting repeatedly, update the xdata and ydata of the existing plots
• Downsample your data to just the required pieces
• Use drawnow to flush events that have built up in the queue.

[Vincent Gigot]

I need to plot and analysse spikes trains, coming from electrophysiological device. It coulb be 2-3 seconds of data recording (less than one Mo text file) or 4-5 minutes of data recording (btween 50 and 100 Mo).

I need to vizualize these spikes trains before and after various process, like peaks detection (spikes detection), denoising, spikes clustering...

[Vincent Gigot]

Downsampling could something to try...

The plotting could also depending of the zoom I choose. I don't really know how to do, but to visualize all the data I could downsample by a factor 10 for example, and for a big zoom I could keep the original sampling...honnestly I don't know how to get the zooming level.

Answer 2

[Jan]

Please post any details, e.g. the used code and the absolute size of what you call "large". A 2D plot of 1e9 points will be slow under all conditions, while a plot of 1e5 lines can be impeded by a variety of factors. Without seeing the code, we cannot guess the cause of the slowdown or suggest any improvements.

[Vincent Gigot]

Thanks for your answer.

Here is the importation code. Data are stored in a structure called SIGNAL wich is a global variable (this is maybe the problem)

SIGNAL.Source contains source data of train spikes, first column for time, second column for amplitude. It could be more than 3 millions lines for the 1D plot (2D is another pb, it's time/frequency map, i have a solution)

%------------------------- data structure creation ------------------------

SIGNAL=struct('Directory',{{}},'Filename',{{}},'Source',{{}},'PeaksBoundaries',{{}},...

'TimeBoundaries',{{}},'Step',{{}},'Header',{{}},'Spikes',{{}},...

'Interspikes',{{}},'Peaks',{{}},'Halfpeak',{{}},'Clusters',{{}},'Scalogram',{{}},...

'FFT',{{}},'Denoised',{{}});

%--------------------------------------------------------------------------

SIGNAL.Spikes.data=[];

SIGNAL.Spikes.length=[];

SIGNAL.Clusters.clusters=[];

SIGNAL.Clusters.tx=[];

[FILENAME,pathname] = uigetfile('*.*','Spikes train file ');

addpath(pathname);

s=uiimport(FILENAME);

[r,c]=size(s.data);

SIGNAL.Source=s.data;

SIGNAL.Header=s.textdata;

SIGNAL.Filename=FILENAME;

SIGNAL.Directory=genpath(pathname);

%----------------------------------------------------------

[Vincent Gigot]

Then a specific function for updating the plotting is called after every process.

This function, redraw the spikes train and threshold bars after every slider move for example.

[Jan]

You draw 3e6 lines on the screen? How many pixels do you expect to appear?

[Vincent Gigot]

no, no, I plot 3e6 points sorry, a 1D plot.

[Jan]

Even 3e6 points are a lot for standard display resolutions.

Answer 3

[Vincent Gigot]

I am also wondering if the plotting updating is delayed not only by the visualisation but also by the calculation or the modification apllied just before replot. It's certainly both.

[Jan]

You can use the "profile" command to find the bottlenecks of your program.