How do I fit two data sets to each other?

Hey everyone

I have a problem regarding an experiment data.

I have two data sets, one is reference and the other one is a measurement.

The data set from the measurement has a shift in x, every measurement a different shift.

To say it clearly I want to fit a measurement data set f(x+c) to a reference data set denoted f(x) and at each measurement to discover the variable 'c'. note: the data set is not a function but a pair {Y(i),X(i)} for both the reference and the measurement.

more small details:

The reference will be a high resolution measurement (small spacing, lots of points) and the measurement will have a higher spacing between points and a smaller amount of points.

The range of the x axis of the measurement and the reference might be different a bit.

Thanks for your help

[dpb]

Clear as mud... :)

I think for anybody to have any chance at all of doing anything specific to the problem would take having the data to look at to see what it is are looking at. Attach a mat file containing at least a representative sample of both...with some more context of what it is that x represents and from whence is it to be obtained? Your posting only mentions having the y, but not the x.

Answer 1

[John D'Errico]

This is a classic problem of calibration. You have two curves, one of which must be shifted (translated) to the other. In some cases, there is also a scale factor in x. And of course, sometimes there is a y shift or scale too.

It sounds like all you have indicated is a translation in x.

In any case, the problem is nonlinear. You will need to choose a simple model for the reference curve. My suggestion is a spline. Make sure the fit is a monotonic one. If necessary, you may need to smooth the reference curve, forcing it to be monotonic. For this purpose the best tool may be something like my SLM toolbox, (found on the file exchange for free download. I'll post a link.)

https://www.mathworks.com/matlabcentral/fileexchange/24443-slm-shape-language-modeling?s_tid=srchtitle

You may also want to insure the reference curve covers a wide domain. You may need to extrapolate it. And again, monotonicity will probably be important. Since you don't show any data, I'll make something up.

xref = linspace(-20,20,100);
F = @(x) atan(x)/pi + 1/2;
yref = F(xref)

yref = 1×100

0.0159 0.0162 0.0166 0.0169 0.0173 0.0177 0.0181 0.0185 0.0190 0.0194 0.0199 0.0204 0.0210 0.0216 0.0222 0.0228 0.0235 0.0242 0.0250 0.0258 0.0266 0.0276 0.0286 0.0296 0.0308 0.0320 0.0334 0.0349 0.0365 0.0382

plot(xref,yref,'b.')

It is a nice smooth and strictly monotonic curve. That is good here, so I will just use an interpolating spline.

refspl = spline(xref,yref);

Now, suppose I have a second curve, shifted from the first along the x axis.

xs = sort(rand(20,1)*20 - 10);
shift = randn(1)*5;
ys = F(xs + shift);

I have no idea what the shift is here in advance. Can I recover it, and shift the new curve onto the target? Assume I don't know the functional form itself of F, only the points on the reference curve. Even worse, the new curve is not even equally spaced.

hold on
plot(xs,ys,'ro')
hold off
grid on

Since the curve is monotonic, I'll use a tool from my SLM toolbox, slmsolve.

https://www.mathworks.com/matlabcentral/fileexchange/24443-slm-shape-language-modeling?s_tid=srchtitle

xest = zeros(size(xs));
for i = 1:numel(ys);
  xest(i) = slmsolve(refspl,ys(i)); % slmsolve is not vectorized, oh well
end

Here is my estimate of the shift, along with the true shift

shiftest = median(xest - xs);
[shiftest, shift]

ans = 1×2

-4.3976 -4.3976

plot(xref,yref,'b.',xs,ys,'ro',xs + shiftest,ys,'ms')
legend('Reference curve','New curve','After translation')

Of course, there is no noise in this system, so the estimate should be almost perfect.

If the problem is more complicated, where there is also a scale change, or a linear shift or scale in y, with some effort, that too could be solved.

And if there is noise in the problem, where the reference curve needs to be smoothed, the SLM toolbox can help with that too.

[roy wexler]

Hey John

this was very helpfull, already part of it is implented and helped me understand better some values of the experiment.

in order to use it in real time though, I need to also rescale the x and the y components, I still care only on the shift value(Im not sure the scaling of the x is that crucial)

in mydata_question file that is attached there are 4 variables xref yref and xs ys

xref and yref are the reference and cleaner signal and xs ys is the measured signal

I would like to understand what is the shift in x in my measured signal(xs,ys) next to the reference signal(xref,yref)

they look something like this:

do you know how to use the slmsolve also to do rescailing?

Thanks in advance

Roy

Answer 2

[Matt J]

Sounds like a 1D image registration problem. Maybe consider imregtform.