Get derivatives from noisy data

Hi everyone, here i had a problem when i want to get derivatives from a surface.

For my data, first i use sft to get a fitting model (cubic interpolant was used when data have no noise) and then use Differentiate(f,x,y) function to get the derivatives. it worked well for the non-noisy data. but when i add noise, i have to use other method to smooth and fit the data, i used Lowess. it gave me a good fitting. but when i use differentiate(f,x,y) to get derivatives, the results are terrible. there are many big peaks and valleies even when they should be very smooth. I tried to decrease the noise level. But for noise higher than 2%, I still cannot get good results.

I wonder is there any other method to smooth noisy data and recover the real data and their derivatives. And I think how much smoothing yield accurate derivative is difficult to know. Any suggestion? Your answer will be greatly appreciated.

Cheers Hui

Answer 1

[Matt Fig]

Numerical differentiation with noisy data is notoriously unreliable. Have a look at this article to see one method for addressing the issue.

[Zhenhui]

Thanks for your quick answer, i will first read the article you provided. hope i cant find the way ^^

[Zhenhui]

Hi Matt, I read the paper and it helps a lot. From the paper, regularization is needed before denosing.It seems to be a complicated process. i wonder how can we do that in matlab or should I write a code for that.sorry that i just learn to use matlab, your advice will help a lot~

Cheers

[Isabel Llorente Garcia]

I am afraid the link seems to be broken. Can somebody provide another link? Many thanks.

[Tobias]

Hi,

Here is some code available to perform total variation:

https://github.com/JeffreyEarly/GLNumericalModelingKit/blob/master/Matlab/TVRegDiff.m https://sites.google.com/site/dnartrahckcir/home/tvdiff-code

Link to the article, Best, Toby

[shamraiz shamraiz]

The paper discussed by @Loginatorist has a broken link -- it can be found here:

https://www.researchgate.net/publication/255907171_Numerical_Differentiation_of_Noisy_Nonsmooth_Data

OR here: https://www.hindawi.com/journals/isrn/2011/164564/

OR here: https://pdfs.semanticscholar.org/8124/2160d75be7d0f65eb29204609a7d41282447.pdf

Answer 2

[Rory Staunton]

I am working on a similar problem, but only on univariate data, and I am still exploring different approaches. One that has worked for me reasonably well is essentially what could be called 'automatic piecewise linear fitting by threshold-limited iterative fit range extension'. Extend the range of your fit until your deviation gets too high, then begin fitting in the adjacent region. There are a number of tricks that can be used to improve on this, but that's the basic idea. Translating this to surface fitting though could be a much harder problem...