Computation of mean and standard deviation after supressing NaNs of an array

nnmax = 4;

for nn=2:nnmax

            D_filt(nn,:) = D_filt(nn-1,:);

            [Dmax(nn,aa),Imax(nn,aa)] = max(D_filt(nn-1,:));

            D_filt(nn,Imax(nn,aa)) = NaN ;
end

#Comments (loop indicates)

1. Scans from 0 to 360 degrees with 10 degree interval.

2. aa represents the same [0:10:360]

3. At each value of angle, the detector collects 10 data points

4. OBJECTIVE: To discard 4 data points out of the 10 and compute mean and SD

5. nn represnts the iteration number while supressing the bad data points

6. The output of the above loop displays

       1 2 3 4 5 6 7 8 9 10

       1 2 3 4 5 6 7 8 9 0

       1 2 3 4 5 6 7 8 0 0

       1 2 3 4 5 6 7 0 0 0

Instead of 4 rows, a single row matrix after supressing four bad points is desired, so that mean and SD can be computed with ease.

Answer 1

[Thomas Koelen]

Hi Naga,

The term "average" usually encompasses several ways to measure what value best represents a sample. There are various measurements that are used and at times the term and measurement used depends on the situation.

A statistician or mathematician would use the terms mean and average to refer to the sum of all values divided by the total number of values, what you have called the average. This especially true if you have a list of numbers. In fact even in mathematics there are different "averages" or "means" and this one is more properly called the arithmetic mean.

I think the thing you are trying to accomplish is straight up using the function:

mean()

http://nl.mathworks.com/help/matlab/ref/mean.html

For the standard deviation use this function:

std()

http://nl.mathworks.com/help/matlab/ref/std.html

Thomas Koelen

[Naga]

Hi Thomas, Thank you. Actually, I am running a measurement for 37 points, at each point the detector measures 10 times, my goal is to identify 4 bad points (I treat them as NaN), supress them and compute the average and SD for the rest of the data.

I find it bit complicated to put my above problem in a loop and generate the required output

[Thomas Koelen]

a(isnan(a(:,1)),:)=[]

this should give you your vector without NaN, then you can just apply the mean() and the std() function.

have a look at this, a is your vector here.

[Naga]

When I run the program, the output is zeroes instead of NaN, so when I use your above function, there's no change in the output.

Just to give you an idea, When I run the above loop the following is being displayed in the output

D_filt

       1 2 3 4 5 6 7 8 9 10

       1 2 3 4 5 6 7 8 9 0

       1 2 3 4 5 6 7 8 0 0

       1 2 3 4 5 6 7 0 0 0

So, there are no NaNs in principle

Answer 2

[Greg Dionne]

You can use mean(x,'omitnan') in R2015a. If you have the Statistics Toolbox, you can use nanmean.

Otherwise, try this:

n = sum(~isnan(x));
x(isnan(x)) = 0;
s = sum(x);
avg = s./n;

[Naga]

Thank you Greg. The code computes the average correctly. But, it excutes the original array, not the one which contained NaNs. 1 2 3 4 5 6 7 8 9 10

       1 2 3 4 5 6 7 8 9 0

       1 2 3 4 5 6 7 8 0 0

       1 2 3 4 5 6 7 0 0 0

In the above output, average of the last row should be computed. Being an amateur programmer, I believe I am wrongly assigning 'x' in the formula. According to the code above( kindly, see my original question), 'x' has a dimension x(nnmax,:), but this generates an obvious error. So, in the loop I should be able to assign only 'x'.

[Naga]

Also, surprisingly mean(x, 'omitnan') doesn't generate the desired output; this may be because, instead of treating the number as 'Not a Number' it is assigning zero for the same, hence zero is being included while computing the average.

[Greg Dionne]

Are you wanting to operate along rows and not columns? In that case try:

mean(x,2,'omitnan')

where x is the matrix that contains NaN values.

Your example shows zeros at the end of your vectors, not NaN. Do you want to take the average of all values except zero? or all values except NaN?

[Naga]

Yes, in the for loop after the second iteration a vector is generated(with a single zero, which actually is a NaN but I don't know for some reason NaN is being assigned as a zero).

Third iteration generates the subsequent vector with two zeros (which are again NaNs). so on and so forth...

So,my goal is to take the average of the remaining elements (of the LAST VECTOR ONLY) except zeros.

[Greg Dionne]

Ok.

So you want the mean of the fourth row excluding the last three elements.

How about: mean(x(4,1:end-3),2)

[Naga]

Actually, I cited you an example here. But, in reality the zeros are not at the end always, they can be at any position depending on the occurence of the 'NaN'.

[Greg Dionne]

In that case you could do:

sum(x(4,:),2) ./ sum(0~=x(4,:),2)

But it seems you are still confused as to why you are not getting NaN in your output like you are expecting. Perhaps if you posted your data (save it to a MAT file and post it to your question) and a quick blurb about what you want to see happen (i.e. what you expect the answer to be) we'll be able to point you in the right direction.

Good luck!