What is the best way to handle missing data?

Hello,

So, basically, our data analysis courses request how to handle data properly, and there's one file in particular that contains missing data (ie. showing -99 while other numbers were positive). In order to proceed to effective statistical analysis, we have to take them into account.

Suppose our file is named 'DATA.dat' from now. Here's my method of resolution :

data = load('DATA.dat');
data(data<=0) = NaN;

But here's the problem: not only I can't compute my data properly, but I'm unable from now on to compute the mean, the median or the standard value. So I was wondering if there was another method to treat the file properly without going through the NaNs?

The main solution that was offered is the following :

NegVal = data<=0;  
Years = [1:size(data,1)];
for i=size(data,2)
plot(Years(NegVal(:,i)), data(NegVal(:,i),i))
end

But I really don't understand what it's supposed to do and from the looks of it, it's too complicated to use for the following instructions (dressing a bar plot, etc...).

Thanks in advance!

Answer 1

[Jos (10584)]

You can take a look at the functions nanmean, nanstd and the like (available in the Statistics Toolbox).

You can also through them out, as in:

originalData = [1 2 3 NaN 5 NaN 7]
isOk = ~isnan(originalData)
newData = originalData(isOk)
mean(newData)
nanmean(originalData) % check

[Shadi Boomi]

Thank you! It worked just fine on my code.

Are you sure this is the best method? Because the one used above doesn't give an irregular line between the data when I'm trying to plot it. From the looks of it, I'm skipping values whenever a NaN is present on a line (I'm plotting column by column in function of the time).

[Jos (10584)]

The best method depends on what you want to do. Note that plot will ignore NaNs:

x = [1 2 3 4 5]
y = [2 4 NaN 8 10] 
plot(x,y,'bo-')