Multidimensional Array deleting entire max record keeping information across in order

I have the following loop:

sz = 20;
m = zeros(size(sz));
for i = 1:sz
    [max_val, position] = max(sh(:));  %Find Max Position
    m(i,:) = [position];                          %Store Max Position in "m"
    sh(sh==max(sh)) = [];                   %Delete current max value of "sh"
end

What I am trying to do here is pretty much to extract to "m" the top 20 highest values in "sh" (sh is a 3D matrix - 100x60x95). What i am not sure is happening is if " sh(sh==max(sh)) = []; " deletes only the value or the entire "row" containig the data of the max value.

What i ultimately need is once i delete the value, i need to delete everything else pertaining to that value in the matrix so when values shift they keep their order. Thanks so much for the help!

[Voss]

Let's do some tests with a smaller 3D matrix:

format('compact');

sh = [1 3;2 4]+reshape([0 4 8],[1 1 3]);
display(sh);

sh = 
sh(:,:,1) =
     1     3
     2     4
sh(:,:,2) =
     5     7
     6     8
sh(:,:,3) =
     9    11
    10    12

[max_val, position] = max(sh(:));  %Find Max Position
display(max_val);

max_val = 12

display(position);

position = 12

sh(sh==max(sh)) = [];
display(sh);

sh = 1×6

1 3 5 7 9 11

Maybe that's not what we expected. Let's investigate what's going on there:

sh = [1 3;2 4]+reshape([0 4 8],[1 1 3]); % restore the original value of sh
display(max(sh));

(:,:,1) =
     2     4
(:,:,2) =
     6     8
(:,:,3) =
    10    12

So max(sh) gives more than one max value. That is because max() by default does the maximum along the first non-singleton dimension (i.e., the first dimension whose size is not 1), which in this case is dimension 1, so max(sh) gives us the maximum of each column. That's not what we want. We want the maximum of the whole thing, so we can do max(sh(:)) instead, which makes sh into a column vector before doing the max() operation (like you have in a previous line), and see what that does:

sh = [1 3;2 4]+reshape([0 4 8],[1 1 3]); % restore the original value of sh
display(max(sh(:)));

    12

That's the maximum we expected. Let's delete it (actually this will delete any element of sh that equals its maximum value (i.e., 12), of which there is just one in this case, but when we try the 100x60x95 matrix, we'll have to account for the situation where the maximum occurs more than once in the matrix):

sh(sh==max(sh(:))) = [];
display(sh);

sh = 1×11

1 2 3 4 5 6 7 8 9 10 11

Now the maximum value is gone, but sh is a row vector with 11 elements. This is because deleting an element out of a matrix (even a 2D matrix) is an ambiguous situation: how should the remaining elements be shifted to try to preserve a matrix structure? Can they even be shifted to preserve a matrix structure? As opposed to, say, deleting a row (or column) from a 2D matrix, in which case it is clear that the remaining rows can be shifted up (or the remaining columns can be shifted left). (In this case, clearly, with 11 elements remaining, there is no 2D or higher-dimensional matrix than can be made with 11 elements, as 11 is prime.) So MATLAB just deletes the element(s) and returns a row vector of the element(s) that remain.

These questions that arise when deleting individual elements from matrices have to be answered by you, the programmer. If you had a 2D matrix and you want to delete one element from it, do you want to keep every other element and deal with the row vector MATLAB gives you? Or do you instead want to delete the entire row where the element is? Or the entire column? Or both? With a 3D matrix, there are even more possibilities: delete the entire x-y slice? the entire y-z slice? the entire x-z slice? both the x-y and x-z slices? both the x-y and y-z slices? both the x-z and y-z slices? all three x-y, y-z and x-z slices? In other words, in which dimension(s) should the the size of the matrix be reduced by one when the element is deleted?

You allude to this type of consideration in your question: "i need to delete everything else pertaining to that value in the matrix". But it is not clear at all (to me anyway) how to interpret this statement and get a set of dimensions in which 2D slices need to be removed from the 3D matrix, so that I could hope to write down actual MATLAB commands to do what you have in mind.

And more questions arise when thinking about deleting not just the single maximum value but the 20 highest values in a 3D matrix. If you want to delete a slice in each of the 3 dimensions each time, say, that's going to delete a bunch of other elements. What if some of those were in the top 20? Should they count towards the top 20 to be deleted or should the top 19 maximum values among what's left of the matrix be deleted next (which is the way your code is written now)? (And what if the maximum element occurs twice? Does that count as 2 of the top 20 or only 1?)

Anyway, sorry for the long lecture, but answering some of these questions about exactly what you want the code to do will help you get an answer on how you can go about doing it.

[IDN]

Thanks so much this is actually great as I am just starting to learn matlab. So "sh" is the result of a 3 variables x1,x2,x3 computation. Therefore, what I am really after is the top 10 values of sh and their respective coefficients. I have not been able to find a way to do this...so I figured if I delete the max and its row of coefficients and do this 10 times I would get the top 10 max sh values and their respective coefficients.

[IDN]

Ah sorry I forgot to tell you I was checking top 10 unique sh...

Answer 1

[Voss]

If all you need is the 20 maximum values and their positions (either linear index or subscript index), then you can do this:

sh_temp = sh;
sh_temp(isnan(sh_temp)) = -Inf;
[sh_sort,ii] = sort(sh_temp(:),'descend');

sz = 20;
max_vals = sh_sort(1:sz);
m = ii(1:sz); % linear index of max_vals

[r,c,p] = ind2sub(size(sh),m); % row, column, 'page' index for each of max_vals

[IDN]

Thanks so much!

[IDN]

Hi Benjamin,

After and indepth look at this...it looks like its not working its not giving me the right numbers given what i am after "Thanks so much this is actually great as I am just starting to learn matlab. So "sh" is the result of a 3 variables x1,x2,x3 computation. Therefore, what I am really after is the top 10 values of sh and their respective coefficients. I have not been able to find a way to do this...so I figured if I delete the max and its row of coefficients and do this 10 times I would get the top 10 max sh values and their respective coefficients."

So when i picked the top 10 and i validated the answer 1 by 1, it should be the case that sh as i insert the different values becomes smaller and smaller as i go down the top 10. What i am seeing is something like the first one is fine, the the second one is really worse off, then the third one goes back to a similar outcome like the first and so on....so therefore the coefficiente x1,x2,x3 are not following correctly sh. Hope this helps! Thanks!

[Voss]

Can you save the variable sh and any other necessary variables in a mat file (you can use the save function) and attach it here (using the paperclip icon)?

In my answer, sh does not change size. The code just sorts sh and grabs the top (20 in that case) elements and their locations in sh, then converts the locations to their indexes in 3 dimensions. So I'm not sure why sh would get smaller and smaller using that approach.

[IDN]

Attached sh

[IDN]

Thanks for taking the time to help me out!

[Voss]

No problem. OK, let me load that mat file and run the code (with top 10 this time):

S = load('RSI SMARSI 300-400 Results Sharpe Full Data 12-22-21.mat');
sh = S.sh;

sh_temp = sh;
sh_temp(isnan(sh_temp)) = -Inf;
[sh_sort,ii] = sort(sh_temp(:),'descend');

sz = 10;
max_vals = sh_sort(1:sz);
m = ii(1:sz); % linear index of max_vals

[r,c,p] = ind2sub(size(sh),m); % row, column, 'page' index for each of max_vals


format('long');

display(max_vals);

max_vals = 10×1

0.033709543930498 0.033709543930498 0.033666860548129 0.033666860548129 0.033348518415672 0.033314820207525 0.033133702928952 0.033133702928952 0.033105767313503 0.033105767313503

display(m);

m = 10×1

1471999 1472000 1495999 1496000 1471996 1495996 1471997 1471998 1495997 1495998

display([r c p]);

   399    20    62
   400    20    62
   399    20    63
   400    20    63
   396    20    62
   396    20    63
   397    20    62
   398    20    62
   397    20    63
   398    20    63

So, we got:

1) the 10 maximum sh values are 0.0337095... and so on

2) the linear indexes of these values in sh are 1471999 and so on

3) those linear indexes correspond to row, column, page indexes of 399, 20, 62 (respectively) and so on

Seems reasonable to me. Let's do some sanity check for the first few:

display(sh(399,20,62));

   0.033709543930498

display(sh(400,20,62));

   0.033709543930498

display(sh(399,20,63));

   0.033666860548129

Seems ok. What is the problem?

[IDN]

Let me look in depth and provide some samples of what I see...thanks so much Benjamin!

[IDN]

Hello Benjamin, please see file attached which is same here below...

Below grabbed the top 10 unique sh values from the attached top 1,000 results table (script from initial answer)

Manual sh Position sh r c p

0.036176 1471999 0.033710 399 20 62

0.011005 1495999 0.033667 399 20 63

0.036176 1471996 0.033349 396 20 62

0.011005 1495996 0.033315 396 20 63

0.036176 1471997 0.033134 397 20 62

0.011005 1495997 0.033106 397 20 63

0.035607 1471599 0.033057 399 19 62

0.011068 1495599 0.033026 399 19 63

0.032129 1471196 0.032745 396 18 62

0.009171 1495196 0.032719 396 18 63

So I manually ran sh using the same function that generated this matrix and it very much matched every other position which i find rather strange, not sure if its in the process of sorting or what...hope this can provide color as to what i see on my end. Thanks!

[Voss]

You mention you are looking at the top 10 unique sh values. The code in my answer gets the top 10, regardless of whether they are unique or not (notice how the top two appear to be the same in the output from my previous comment and the third one there is your second one).

[IDN]

Right, so I got the top 1,000 with your script (attached) and did in excel a sh unique filtering and just mapped that sh to a position which then with that position I look for r,c,p thats how I built that table.

[Voss]

Ah, ok. So everything looks good, right?

[IDN]

Yes, thanks so much!