detrend using cubic splines

How can I detrend a time series using cubic spline interpolation? I would like to get this done over for eg., 0.2 year bins.

Answer 1

[Shashank Prasanna]

Venkatessh, detrending is just a process of removing long term deterministic patterns or in short trends. For example your data may look like it is exponentially increasing, but with variation. You are more interested in the variation because the exponential increase renders the data non-stationary.

It is as simple as fitting your data to the best possible curve and subtracting it from your data.

Simple example:

Generate data and get a spline fitted trend:

x = 0:100;
y = 100*randn(1,length(x))+x.^2;
xx = 0:10:100;
yy = spline(x,y,xx);

Substract the trend from your original data and plot the detrended variation sequence.

ytrend = interp1(xx,yy,x,'spline');
subplot(2,1,1)
plot(x,y,'.',x,ytrend);
subplot(2,1,2)
plot(x,y-ytrend)

[Venkatessh]

Thanks, I had messed up with the "interp1" initially. The example served the purpose.

[Image Analyst]

Maybe I don't understand, but you're merely subsampling the data, basically throwing away the original data, and then interpolating a spline between the subsampled points. Then subtracting that from the full, original data. So since you're ignoring most of the data, you're only detrending the spline knots - the data you trained the spline with. Whatever happens in between the subsample points it totally ignored as far as calculating the trend goes. That doesn't sound right to me. See this illustration:

% Create sample data:
x = 0:100;
y = 100*randn(1,length(x))+x.^2;
% Reduce every 10th data point.
y(1:10:end) = 0.5 * y(1:10:end);
% Now we have our sample data.

% Now subsample.
xx = 0:10:100;
% Fit a spline between the subsampled points only.
yy = spline(x,y,xx);

% Subtract the trend from your original data and plot the detrended variation sequence.
ytrend = interp1(xx,yy,x,'spline');
subplot(2,1,1)
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);
plot(x,y,'.',x,ytrend);
grid on;
subplot(2,1,2)
plot(x,y-ytrend, 'bd-')
grid on;
title('Spline knots are detrended, other data are not', 'FontSize', 25);

Of course I guess you could say the crazy data I used doesn't really show a trend, as in the slowly varying sense of the word. As long as your data sort of lies along the spline, like your original data did, then it would probably be good enough.

[Venkatessh]

Yes, discarding data having high significance is a problem. Infact, my next problem is in de-trending an unequally spaced time-series. One of the references in my work, have utilized the cubic splines to de-trend a similar unequally spaced data using equal bins. I am confused over this approach of de-trending.

[Shashank Prasanna]

IA, I agree with you. In practice I have never seen detrending with a spline which as you mention does not make much sense. I am not sure what kind of time series Venkatessh is working on, but I have never seen detrending using nonparametric regression methods such as splines. You either detrend using an FIR filter or parametric regression which is intuitive. I am however referring to this.

Venkatessh, if you could provide your area of study and the application, we could assist you in a more formal solution.

[Image Analyst]

Maybe you might try my suggested approach or even take a look at this: SLM, or at the Curve Fitting Toolbox.

[Venkatessh]

Sorry for keeping my field of work a secret until now. I am in the field of astronomy currently with "light curves" showing arbitrary behaviour. As per your discussion, I need to be very careful in discarding the data. Infact, there is always a problem of handing the uncertainties associated with each variable.

[Matt J]

@Benji,

but I have never seen detrending using nonparametric regression methods such as splines.

Spline smoothing can be posed as a linear parametric regression, where the unknowns are the coefficients of a linear combination of spline basis functions, e.g.,

http://en.wikipedia.org/wiki/Smoothing_spline#Derivation_of_the_smoothing_spline

My Answer used a similar formulation, though without the regularization, and valid only for equi-spaced time samples.

@Venkatessh

As per your discussion, I need to be very careful in discarding the data. Infact, there is always a problem of handing the uncertainties associated with each variable.

It's becoming a bit confusing whether you have the answer you need. You accepted Benji's suggestion even though we now agree that it discards the data, something you don't want to do. In your response to my Answer, you also seemed to say that it served your needs, but now you say you have an unequally spaced time series... Do you have what you came for, or don't you?

[Venkatessh]

Only moments back was I given a new task to deal with an unequally spaced data. Added to which, I have to handle those uncertainties. I am freaked out with this new task. My question to you with regards to your suggested code is, will it be possible to define an interval instead of number of points for the control points?

[Matt J]

I don't know what you mean by a spline without control points. Knots & control points are what make a spline a spline.

[Venkatessh]

I wanted to know if it is possible to define a bin width in this case.

[Shashank Prasanna]

@Matt J, I may not entirely agree with your definition under the accepted definition of parametric regression. Splines come under a general class of non-parametric regression models although each piecewise polynomial is parametrically fitted. This is the reason I phrased specifically as "nonparametric regression methods such as splines"

Several literature categorize the method as below:

http://www.stats.uwo.ca/faculty/braun/ss9924/additionalnotes/nonparametricreg.pdf

http://en.wikipedia.org/wiki/Non-parametric_statistics#Non-parametric_models

http://books.google.com/books?id=p5E1LVJ2C-0C&lpg=PA247&pg=PA246#v=onepage&q=page%20246&f=false

Not to argue that each piecewise polynomial is parametrized.

[Matt J]

There are more flexible spline fit tools that let you place the spline knots unevenly. ImageAnalyst pointed you to some. There are many more on the FEX

http://www.mathworks.com/matlabcentral/fileexchange/index?utf8=%E2%9C%93&term=spline+smooth

This one looks particularly straightforward, does not require toolboxes, and is reasonably popular

http://www.mathworks.com/matlabcentral/fileexchange/13812-splinefit

I haven't used it myself, though.

[Venkatessh]

Thanks everyone for a good discussion on the topic.

[Matt J]

@Benji,

It's a little murky to me why your refs view spline fitting as nonparametric when the spline basis functions, e.g., in (1.6) in the .pdf, aren't derived from the data in any way.

But regardless, going back to your original point, why would splines (parametric or no) not be a sensible tool for determining a trend? They constitute a family of curves that can be fit to a function just like any other.

[Shashank Prasanna]

@Matt J

A collection of local models is non-parametric, such as the one summarized in 1.6. Since it can exploit a global trend in a set of local models it can characterize some behavior in the data better than a single global model. Splines fall into this framework and is non-parametric and an accepted categorization.

I mentioned "but I have never seen detrending using nonparametric regression methods such as splines." I didn't discuss the sensibility of using this tool at all, since I haven't come across it being used. It may very well be the best tool, but I haven't seen it in the applications I have worked on and is exactly what I wanted to convey in my comment.

[Matt J]

I'll assume that "a collection of local models" means "basis functions with localized support" and "exploit a global trend" means "capable of bending as needed to fit different regions of the curve."

So the only thing that distinguishes parametric modeling from non-parametric modeling is locality, i.e, that non-parametric offers more region-by-region local flexibility over the fit? Seems like peculiar distinction. What happens, for example, if you fit using only 1 or 2 basis functions. Then you're using the same fitting family, but you have much less local control...? But if that's the accepted terminology, I guess that's what it is.

[Shashank Prasanna]

Honestly, I don't know if there is a definition written on stone somewhere east of here, but that is atleast the accepted convention. All the following methods fit into the non-parametric model framework: Splines, Neural Networks, SVM (and other kernel estimators), regression trees etc

Anyway this made for a nice discussion. I wish this topic wasn't embedded in another post, would loved to have see more responses.

[Image Analyst]

I'm not sure what "I need to be very careful in discarding the data. " means. Of course any type of fitting, regression, or detrending is going to discard data. Even though the data may go into creating the fit, eventually the data is going to be discarded and replaced by the fitted/estimated/smoothed values. If you didn't want to do that, you'd just keep your original data. So you are going to discard data and replace it - the only issue to decide is how much to smooth the data.

Answer 2

[Matt J]

Here's an example of how to do a cubic spline regression using interpMatrix. You would have to choose how finely you wanted to space the control points, which would affect the stiffness of the spline fit. In the example, the control points occur every 9 samples. After obtaining the 'trend' you would of course subtract it off the original time series to detrend it.

    s = @(t) cos(2*pi*t).*exp(-abs(2*t))+ 2;  %timeseries to fit

    cubicBspline = @(t) (t>-1 & t<1).*(2/3 - t.^2 +abs(t).^3/2) +...
                        (abs(t)>=1 & abs(t)<2).*((2-abs(t)).^3/6);

    tCtrlPts=linspace(-1.2, 1.2,9);     %CtrlPts sample locations on t-axis
       dtCtrlPts=tCtrlPts(2)-tCtrlPts(1);
    tFine=linspace(-1.2, 1.2,81);      %Fine sample locations on t-axis
       dtFine=tFine(2)-tFine(1);

    timeseries=s(tFine(:));

    %create regression matrix
    SampRatio=round(dtCtrlPts/dtFine); %Sampling ratio
    kernel=cubicBspline(-2:1/SampRatio:2 );
    nCtrlPts=length(tCtrlPts);

    A=interpMatrix(kernel, 'max', nCtrlPts, SampRatio, 'mirror');

    %%Do the fit!!! 
    trend = A*(A\timeseries); 

    plot(tFine,timeseries,tFine,trend,'*-'); 
                    legend('Time Series','Fitted Trend')

[Venkatessh]

This is a clinical approach to the solution. Thanks.

[Image Analyst]

What does that mean?

Answer 3

[Image Analyst]

Because a spline is an interpolation rather than a regression, and so it goes through all the points, I don't see how it could detrend. Why not use detrend() or sgolay()?

[SAAlly]

Hi there, if you look at Shashank Prasanna's answer you'll notice x has a step of 1, while xx has a step of 10. Therefore the spline would only go through every tenth point.

[Bruno Luong]

It's still an arbitrary and poor choice IMO.