How to plot and filter some values from a csv file?

Hello,

I have some csv files where one column is time and the other is distance.

How do I plot these columns and filter out the irrelevant values I don't want?

I have made the following code to read and output the columns in x and y axes.

sonar_F_030 = readtable('sonar_F_030.csv');
x1 = sonar_F_030(:,2);
y1 = sonar_F_030(:,1);

Will I need the least squares method? I'm not sure that's why I'm asking you.

Thanks in advance!

[Walter Roberson]

sonar_F_030 = readtable('sonar_F_030.csv');
x1 = sonar_F_030{:,2};
y1 = sonar_F_030{:,1};

However you have not given us anything to go by to know which points are irrelevant or not. Nothing in what you posted suggests a need for least squares methods.

[RoBoTBoY]

I want to exclude those that are quite far from the 0.30m point. E.g. From 0.29 to 0.31 are acceptable.

I would like something like this:

https://www.mathworks.com/help/matlab/ref/polyfit.html

Linear regression and linear regresssion with estimate error.

Answer 1

[Star Strider]

Use the rmoutliers function to remove the outliers.

The easiest way to implement a polynomial fit to data like these is with the Savitzky-Golay filter (sgolayfilt function in the Signal Processing Toolbox).

Try this —

sonar_F_030 = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1127720/sonar_F_030.csv')

sonar_F_030 = 950×2 table
     range       time  
    _______    ________

     0.3044    0.062361
     0.3044     0.12892
    0.30696     0.19563
    0.30376     0.26228
    0.30696     0.32898
    0.30376      0.3957
    0.30376     0.46234
    0.30376     0.52899
    0.30759     0.59565
     0.3044     0.66232
    0.30759     0.72895
    0.30312     0.79598
     0.3012     0.86278
     0.3044     0.92966
     0.3044     0.99618
     0.3012      1.0629

t = sonar_F_030.time;
range = sonar_F_030.range;

[rangee,TFrm,TFoutlier,L,U,C]  = rmoutliers(range, 'percentiles',[1 99]);
Lower_Limit_Retained = L

Lower_Limit_Retained = 0.2967

Centre_Value = C

Centre_Value = 0.3025

Upper_Limit_Retained = U

Upper_Limit_Retained = 0.3082

range_filt = sgolayfilt(rangee, 3, 51);

figure
plot(t, range, 'DisplayName','Original Data')
hold on
plot(t(~TFrm), range_filt, '-r', 'LineWidth',2, 'DisplayName',['Savitzky-Golay Filtered Data' newline 'With Outliers Removed From Original'])
hold off
grid
legend('Location','best')

Make appropriate changes to get different results.

.

[RoBoTBoY]

My data does not come from any signal processing, it is simply from an infrared sensor that measures distance. I don't know if what you wrote to me is correct.

[Star Strider]

The origin of the data is not important. Signal processing techniques can be used on any data that meet the criteria for the functions (usually that means having constant sampling intervals, although that is not always a requirement). The Savitzky-Golay filter fits a polynomial (here defined as having degree 3) to a moving frame length of data to smooth it. This is likely more appropriate than polyfit, although if you want to use polyfit, that is an option.

EDIT — (17 Sep 2022 at 16:58)

The polynomial fit would work like this —

sonar_F_030 = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1127720/sonar_F_030.csv')

sonar_F_030 = 950×2 table
     range       time  
    _______    ________

     0.3044    0.062361
     0.3044     0.12892
    0.30696     0.19563
    0.30376     0.26228
    0.30696     0.32898
    0.30376      0.3957
    0.30376     0.46234
    0.30376     0.52899
    0.30759     0.59565
     0.3044     0.66232
    0.30759     0.72895
    0.30312     0.79598
     0.3012     0.86278
     0.3044     0.92966
     0.3044     0.99618
     0.3012      1.0629

t = sonar_F_030.time;
range = sonar_F_030.range;

[p,S,mu] = polyfit(t, range, 21)

p = 1×22

-0.0006 -0.0004 0.0126 0.0073 -0.1062 -0.0561 0.4863 0.2237 -1.3333 -0.5082 2.2649 0.6608 -2.3697 -0.4569 1.4613 0.1310 -0.4816 -0.0002 0.0690 -0.0020 -0.0027 0.3035

S = struct with fields:
        R: [22×22 double]
       df: 928
    normr: 0.0654

mu = 2×1

31.6961 18.2925

range_fit = polyval(p,t,S,mu);

figure
plot(t, range, 'DisplayName','Original Data')
hold on
plot(t, range_fit, '-r', 'LineWidth',2, 'DisplayName','Polynomial Fit To Data')
hold off
grid
legend('Location','best')

It may not be necessary to remove the outliers, because they do not have a noticable effect on the polynomial fit (so I did not remove them here). I used the centreing and scaling options to polyfit to improve the fit.

Change the polynomial order (last argument to polyfit) to get different results.

.

[RoBoTBoY]

Ok...I will try them!

[RoBoTBoY]

How did you find that the order is 21?

[Star Strider]

I just experimented until I got a result that seemed to work.

Signal processing is frequently heuristic!

Answer 2

[KSSV]

sonar_F_030 = readtable('https://in.mathworks.com/matlabcentral/answers/uploaded_files/1127720/sonar_F_030.csv');
x1 = sonar_F_030.(2);
y1 = sonar_F_030.(1);

y2 = filloutliers(y1,"nearest","mean") ;
y3 = smooth(y2) ;
plot(x1,y1,'r',x1,y2,'b',x1,y3,'g')
legend('original','Removed outliers','smoothed')