How can I approximate a certain section of a vector to remove noise?

I have some plots of S-parameters from a network analyzer, and they have a large amount of noise around 2.4 GHz. I am relatively certain that this noise is due to the Wi-Fi signal in the lab where I am working, so I would like to remove this section of the vector entirely and replace it with an approximation of what the line should look like. These vectors are complex, which complicates the issue because if I use local linear approximation, the real and imaginary components turn out alright, but the magnitude still looks pretty bad. I've tried using some sort of interpolation function (like spline or pchip) but I can't seem to get it to do what I want, which is to entirely remove one section of the vector and replace it with values that fit with the curve before and after. I've attached a few plots to show the issue. The [Raw] plots are straight from the network analyzer, and the [Corrected] plots have simply been "fixed" with local linear approximation on both the real and imaginary components. Thanks!

Answer 1

[Image Analyst]

I wouldn't use polyfit - I'd use a Savitzky-Golay filter. It's like polyfit, but does polyfit in a window of limited width as it slides along the entire signal. You can use function sgolayfilt() in the Signal Processing Toolbox. Since a sine wave or cosine wave is pretty well fit from a 4th or 5th order polynomial (as we know from the Taylor series representation of it), you could try 4 or 5 for the order of the Savitzky-Golay filter. Fitting a 5th order polynomial to a small sliding window will be better than fitting the entire signal with polyfit.

[Image Analyst]

See attached demo.

[Adam Horton]

This looks like it would work well, thank you!. As far as using sgolayfilt(), do I need to load the vector as a "signal"? Is the signal processing toolbox a separate app that I would have to load within Matlab? I'm still pretty new to Matlab so I don't fully understand how this works. If possible, I would like to remove this noise from within a function, as those S-parameters are then used to compute material properties (complex permittivity, permeability, and conductivity if you're interested).

[Image Analyst]

It's a "Toolbox" that is a separate plug-in product that you have to buy. To see if it's included in your current installation, you can type "ver" on the command line. If you don't have it, you can get a 30 day trial version from the Mathworks.

Your "vector" is fine. It doesn't have to be loaded as a "signal". In essence, they're all just 1-dimensional numerical arrays regardless of what you call them.

[Image Analyst]

You can use conv() if you want to do a sliding window to give a mean. Just use linear weights in the kernel. No toolbox is required:

windowWidth = 5; % Some odd number.
outputSignal = conv(inputSignal, ones(1, windowWidth)/windowWidth);

It will "blur" out the signal and smooth the spikes, though it will change the shape of the signal slightly. You can do a modified one where you do a first pass to get a smooth signal then subtract the original signal and threshold to find the outliers. Then get a logical vector of the outliers and replace only those:

originalSignal(outlierLocations) = smoothSignal(outlierLocations);

That will keep the original values in all locations except near the outliers. Also see this: http://www.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers

[Adam Horton]

Thanks for all your help! I ended up using polyfit in the region, but I'll definitely keep the sgolayfilt() function in mind for later cases, as well as the sliding window with conv(). Thanks again.

Answer 2

[Andrew Reibold]

A easy solution is to either use polyfit or take a moving line average of your data, but you lose some resolution.

Try

data = smooth(yourdata)

Smoothing twice dramatically reduces noise:

data = smooth(smooth(yourdata))

By default, its is a 5 point moving line average.

I wouldn't recommend doing it too many times, especially if you have very few data points. Once or twice should be fine in most situations.

(Taking the moving line average 100 times would results in a flat line)

[Adam Horton]

I ended up using polyfit only in the region of the 2.4 GHz noise, which worked pretty well. Thanks for your response! smooth() doesn't work very well for this case because I know that the data around 2.4 GHz are pretty much garbage, so I don't want them affecting the "fixed" data too much.