taking mean of certain range ignorning NaN in matrix

I have a matrix Phi of size 500x359 with NaNs interspersed throughout. I need to take the column wise mean of the first 25 rows of non-NaN values for each column, but unlike nonmean() I dont want to consider rows with NaNs as rows at all. I want them to be skipped over completely. So I want to kinda of do the following nanmean(Phi(1:25)) where 1 is the first nonmean value regardless of its actual location in the matrix whether its the 1, 100, 200 etc and take the mean of the following 25 non-Nan values regardless of the location.

For example:

X=magic(4); X([1 3 7:9 14]) = repmat(NaN,1)

X =

   NaN     2   NaN    13
     5    11    10   NaN
   NaN   NaN     6    12
     4   NaN    15     1

nanmean(X(1:2,3)) ans =

    10

but what I want is (10+6)/2 = 8

and... so my end result would look like this...

phi_mean = [4.5000 6.5000 8 12.5000]

thank you

[Adam]

nanmean(X(2:3,3))

would be what you want if you are selectively choosing just the two rows your expectation suggests you want.

That doesn't answer your bigger question, just the confusion of your example.

[Stephen23]

This explanation is not very clear: "I dont want to consider rows with NaNs as rows at all. I want them to be skipped over completely", but in your example all of the rows of X contain NaN s, so should they all be "skipped over" ?

[Adam]

Yeah, I also found your example to actually be an example of something completely separate to your question which is why I initially got thrown off by just pointing out the obvious problem above before realising that was irrelevant to the actual question!

[Brandon]

What I mean (and try to explain in my example) is the nanmean still considers the rows with NaN for indexing, but ignores them for actually taking the mean. What I need is to ignore the whole the NaN altogether. So for the example nanmean(X(1:2,3)) returns the mean of the first two rows, not the first two values, which is what I need. But in a large matrix I cant go through each row and select the first 25 values manually.

[Adam]

nanmean(X(1:2,3)) returns the mean of the first two values in the 3rd column, ignoring NaNs, but since that is just two values, one of which is a NaN it simply returns the second value of 10.

Answer 1

[Andrei Bobrov]

nn = ~isnan(X);
ii = cumsum(nn).*nn;
out = mean(reshape(X(ii >= 1 & ii <= 25),25,[]));

other way

X = phidp;
k = 25;
ll = ~isnan(X);
[~,j0] = find(ll);
out = accumarray(j0,X(ll),[1 size(X,2)], @(x)mean(x(1:min(k,numel(x)))) );

[Brandon]

This seems like it would work but it gives me an error. As stated my matrix is 500x359, I only used the magic(4) to illustrate what I meant.

Error using reshape Product of known dimensions, 25, not divisible into total number of elements, 6136.

[Andrei Bobrov]

see "other way" in my answer and please attach file with your array (500x359)

[Brandon]

Hi, the second way worked! thank you. Also I didnt know you could attah the array to the question, will do next time.

[Brandon]

Not sure if anyone will see this since I set thread as answered, but I found a problem with this "other way" method. if there is a column of all NaN this method ignores the whole column.

For example in the attached file when you apply this method the out becomes 355 long, instead of 359.

This is a problem because I need the length of the "out" to stay the same as the original matrix so I can remove the mean from the original value. I have attached a array if anyone has any ideas.

thanks

[Andrei Bobrov]

corrected

Answer 2

[Adam]

hasNans = any( isnan(Phi), 2 );

validIdx = find( ~hasNans, 25 );

will give you the first 25 row indices which you can then use for your mean calculation.

[Brandon]

problem with that is that each column has a NaN in it somewhere so it just gives me an empty validIdx

[Adam]

It would probably help if you can give an example of what you actually want output to look like for a given small input that represents the problem you have, but on a small scale. The example you gave does not seem to do that and your explanation of what you want included the phrase 'I dont want to consider rows with NaNs as rows at all' which I took to mean that you wish to ignore all rows with a NaN anywhere in them. If that is the case then if every row has a NaN somewhere you would get an empty result.

Answer 3

[Stephen23]

As far as I understand, you wish to calculate the mean of the first 25 non-NaN values in each column. This can be achieved easily using a couple of find calls. Using your example matrix X :

>> N = 2;
>> K = arrayfun(@(k)find(c==k,N),1:size(X,2),'UniformOutput',false);
>> K = horzcat(K{:});
>> [r,c] = find(~isnan(X));
>> mean(X(sub2ind(size(X),r(K),c(K))),1)
ans =
        4.5          6.5            8         12.5

Or a much tidier solution is with some indexing:

>> N = 2;
>> Y = ~isnan(X);
>> mean(reshape(X(cumsum(Y,1)<=N & Y),N,[]),1)
ans =
        4.5          6.5            8         12.5