Logical indexing based on intervals

I have a matrix of time intervals (attached 'intervals') and time stamps (attached 'times'). I want to logical index for all the time stamps contained in the intervals.

Please let me know if this needs clarification. I have attached example matrices.

Answer 1

[Bruno Luong]

any(EMG_times>=t_comb_5(:,1)' &  EMG_times<=t_comb_5(:,2)',2)

Or if you run for old matlab release:

any(bsxfun(@ge,EMG_times,t_comb_5(:,1)') & bsxfun(@le,EMG_times,t_comb_5(:,2)'),2)

[Matt J]

Note, this will consume a very large amount of memory, unless sparse calculations are used, like in my first answer. Also, using discretize will be faster if the intervals are sequential and disjoint.

[Bruno Luong]

@Matt: your claim is not true with SPARSE, it actually takes 4 times more memory in OP's example.

>> S=intervals(1,:)<=EMG_times & intervals(2,:)>=EMG_times;
>> L=EMG_times>=t_comb_5(:,1)' &  EMG_times<=t_comb_5(:,2)';
>> whos
  Name                 Size                Bytes  Class      Attributes

    EMG_times      1697014x1              13576112  double               
    L              1697014x57             96729798  logical              
    S              1697014x57            379906583  logical    sparse    
    intervals            2x57                 2288  double     sparse    
    t_comb_5            57x2                   912  double

[Matt J]

Hmmm. Something has changed in the way Matlab stores sparse matrices. The transpose is much more compact for some reason,

>> St=S.';
>> whos L S St

Name   Size           Kilobytes     Class     Attributes

L      1697014x57         94463     logical             
S      1697014x57        371003     logical   sparse    
St     57x1697014         14487     logical   sparse

In any case, it has to be possible to profit from sparse storage. The density of L,S is considerably small,

>> nnz(L)/numel(L)

ans =

      0.0014

[Bruno Luong]

Nothing has fundamentally changes recently, except for complex sparse that now have interleaved storage, but that doesn't change the memory, and here we discuss about logical.

Sparse matrix, in matlab is columns-major ordering so if you transpose the amount of RAM changes. But already know that I'm sure.

[Matt J]

Sparse matrix, in matlab is columns-major ordering so if you transpose the amount of RAM changes. But already know that I'm sure.

Yes, but it always used to be the case that between A and A.', the most memory-compact would be the one with the least columns. But here, you can see that S.' is the most compact.

[Matt J]

Ah, now I see why. It's an nzmax thing:

 >> S(:)=S(:); whos L S

Name   Size           Kilobytes     Class     Attributes

L      1697014x57         94463     logical             
S      1697014x57          1229     logical   sparse

[Systematically Neural]

This did what I needed. Thank you both for all your effort. I needed the a logical vector the same length as EMG_times that was logically indexed for all the time points of the intervals contained in t_comb_5. For instance the first interval is 54.9566 to 59.5566, so the logical index vector (or L termed above) would correspond to rows 54,913 to 59.513 (the time stamps started at .0444 and sampling rate is a 1000 data points per second so 54.957-.044*1000= 54,913 and 59.557-.044*1000=59,513), which is what is logically indexed.

Thank you!

[Systematically Neural]

@Matt J sorry for the lack of clarity. Please let me know if you believe you have a faster way, I am more than willing to post another question similar to the one above that is answering it in a quicker manner.

[Matt J]

The approach with discretize will be faster, again assuming that your intervals are sequential and disjoint.

Answer 2

[Matt J]

I want to logical index for all the time stamps contained in the intervals. Please let me know if this needs clarification.

Yes, almost certainly. Do you mean the result should be a logical matrix L such that L(i,j)=true if the i-th time stamp lies in the j-th interval. If so, then it would be as follows:

intervals=sparse(t_comb_5).';

L= intervals(1,:)<=EMG_times & intervals(2,:)>=EMG_times;

[Matt J]

Based on conversation with Bruno, best to do

intervals= t_comb_5.';
L= sparse( intervals(1,:)<=EMG_times & intervals(2,:)>=EMG_times );

if a logical matrix is desired.

[Bruno Luong]

The above still generate an intermediate full matrix.

[Matt J]

Yes, but it's type logical so it's not so bad. If the intervals are sequential and disjoint, then you could do

     edges= reshape(t_comb_5.',[],1);
     J = (discretize(EMG_times, edges)+1)/2;
     I=(1:numel(EMG_times)).';

     idx = abs(J-round(J))<.1;

     L=sparse(I(idx),J(idx),true, numel(EMG_times), numel(t_comb_5)/2);

[Bruno Luong]

It's not a question of bad or not, it's just using SPARSE would not be benefit if the result is used once afterwards, will all the dangerous subtlety details of MAXNNZ.

Another observation: SPARSE stores the internal Ir array like command FIND() applied on each input interval.

So one can do that to build sparse without passing by full matrix with FIND on for-loop of interval, and doesn't require the requirement of sorted/non-overlap intervals (where the OP never state, but I agree it would be faster to compute using histogram binning).

But then the question is why use sparse at all? Just doing a basic for-loop on intervals and build L by "&" operator if the memory is the concern.

Answer 3

[Matt J]

Or, do you mean the result should be a logical vector L the same length as EMG_times and L(i)=true if EMG_times(i) lies in any of the intervals? If so, and if the intervals are sequential and disjoint, you can do

    edges= reshape(t_comb_5.',[],1);

    L= mod(discretize(EMG_times, edges),2)==1;

Answer 4

[Bruno Luong]

For general case, meaning without any assumption about the intervals; you can use a function in this FEX file to then reorganize the intervals, then use the HISTOGRAM binning algoritm to find out if it falls into the pairs (similar to Matt's dicretize method)

[left, right] = IntervalUnion(t_comb_5(:,1), t_comb_5(:,2)); % FEX
edges = [left,right]';
edges = edges(:);
[~,~,loc] = histcounts(EMG_times,edges);
L = mod(loc,2)==1;

This would be fast method without any assumption.

A small modification of Matt's method.