How to Filter a Real-Time Signal With Rapidly Changing Frequency? (Possible?)

Hi everyone,

I need to precisely filter a real-time signal with rapid amplitude and frequency changes (from 20kHz to less than 1Hz).

Changes in the frequency are fast to the extent that FFT methods can not be used.

For detecting a 1Hz frequency there should be a 1s window of data.

The sweep between 20 kHz and 1 Hz occurs in a few milliseconds.

Besides the system requires one microsecond updating.

The latency of the system is extremely important. Because it is a real-time system and I have to implement it in an FPGA device.

Further Information:

Indeed, there are disturbances in the system that cause instantaneous variation in frequency and amplitude of the signal compared to the steady-state.

I have to detect these abrupt changes in the real-time process and change the filter coefficient accordingly to be able to provide proper output (smooth as much as possible) that can represent actual system behavior.

If I use a constant coefficient filter, which is designed for steady-state, for whole the process it causes a significant error and malfunctioning of the system.

An example from the beginning of the signal; there are many ones in the middle of the signal.

The Yellow Signal is the respons of the lowpass filter at main frequency in which the system produces in the steady-state (I mean without disturbances).

Whenever a disturbance occurs, the system always brings about transitions to high frequencies and then back to the main frequency.

...

[Mathieu NOE]

hello (again !)

what is the intention ? you want to make a bandpass filter that tracks the signal frequency ? (seems this is a fast sine sweep)

you can use a trigger and a counter to estimate the signal "almost" instantaneous frequency and tune the filter coefficients on the fly

[Minerva Bionic]

Hi dear Mathieu,

thank you for your reply.

I'm tunning the filter coefficients dynamically. However there are many rapid changes in the signal in which I can not estimate the frequency to tune the filter coefficients.

"use a trigger and a counter to estimate the signal 'almost' instantaneous frequency"

I don't understan what you mentioned. May I ask you to explain more?

An example from the beginning of the signal. There are many unpredictable ones in the system process.

[Mathieu NOE]

hello Minerva

so this plot is quite different from the first one ! or is it just a zoom on the start ?

does it has to do with your previous post regarding PID control with filter in the loop ?

what is the target in terms of data filtering ?

[Minerva Bionic]

It is a zoom on the begining of the signal.

It's a complete different problem.

The raw signal introduces excessive noises into the system that cause malfunctioning. However, inappropriately filtered signals also cause malfunctioning. Unfortunately, these changes occur exactly in critical points for the system.

[Minerva Bionic]

...

Let me present you the difference between the required filtering at the beginning and the destination filter (frequency).

[Mathieu NOE]

ok

I see 3 curves , the raw noisy data, then a bit of smoothing (with smoothdata ?) then a response through a very low fc lowpass filter

so what is the target ? can you elaborate ?

can you also share the data if we want to try on my side ?

[Mathieu NOE]

also can you explain what you mean by "the destination filter (frequency)." ?

tx

[Minerva Bionic]

The destination frequency is the main frequency in which the system produces in the steady-state (I mean without disturbances).

Whenever a disturbance occurs, the system always brings about transitions to high frequencies and then back to the main (destination) frequency.

Sorry for the weird naming :-)

[Mathieu NOE]

ok , so the y data of this plot is a frequency ?

so, again, what do you expect for the "ideal" filter effect ? can you simply draw it on top of your data ?

[Minerva Bionic]

The first is the raw signal.

The second one is the response of a lowpass filter (passband frequency: 2kHz)

The third is the response of a lowpass filter with the passband frequency of 5Hz.

I want to implement a digital filtering system on an FPGA that its output be smooth and as close as possible to the raw signal.

I will upload the signal in few minutes.

Thank you.

[Minerva Bionic]

Sampling frequency: 1e6

Thank you.

[Minerva Bionic]

The blue one is the raw signal

The red one is the thing I want.

[Mathieu NOE]

ok

do you look for a "real time" solution or it's to post process once the data are stored ?

can you share the data ?

[Mathieu NOE]

sorry , I didn't see you already posted the data

still the question : do you look for a "real time" solution or it's to post process once the data are stored ?

[Minerva Bionic]

I'm looking for real time solution and I have to implement it on an FPGA device.

And the latency is critical to the system.

[Mathieu NOE]

do you know how to implement a IIR buterworth filter ?

seems to do the job quite well :

and maybe you could do the filtering in the analog section before it enters the FPGA

clc
clearvars

load('Data1_Sept_21.mat');


y = Data;
Fs = 1e6;
dt = 1/Fs;
t = (0:length(y)-1)*dt;


%%%%%%%%%%%%%%%%
NN = 2;
Wn = 0.0002;
[B,A] = butter(NN,Wn);

figure(1)
ys = filter(B,A,y);
plot(t,y,t,ys);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with butterworth LP' ]);
grid on

[Minerva Bionic]

Dear Mathieu,

Thank you in advance.

Both of those filter responses that I posted previously are the output of IIR Butterworth filters.

You filtered the signal at 100Hz . Let me zoom in to see what can come out of it.

as you can see, there is a considerable difference between the 100Hz output with the raw signal.

additionally, these small variations in the signal have to be omitted. Because they cause a considerable error in the system (the acceptable filtering frequency for the steady state signal is at least 5Hz).

[Minerva Bionic]

Let me show you the error at the out put of system with 100Hz and 5Hz lowpass filters respectively:

[Mathieu NOE]

well ! the second plot is even more disturbing to me... I don't know what is your "process" , but I would not feel ok with such spikes in the output. I understand your search for a better input filter but maybe there is also something to do between the input and output (your process) . What creates those spikes ? if that is your main issue

all the best

[Minerva Bionic]

Dear Mathieu,

Thank you for your reply.

Fortunately, hose outliers (spikes) in the second plot can be easily handled within the signal processing system by adding a few mathematics. But this just works fine in steady state.

The signal processing solution that I'm going to implement, must work in every condition.

Unfortunately, there is not any way to change the main process.

It is not a simple case at all, at least for me.

[Mathieu NOE]

hello Minerva

tried today another approach with time varying smoothing factor.... low at the begining then increasing up to the max allowed value. This makes little effect of the initial peak, but after a while the smooting effect is much stronger. So very little noise in the steady state time frame. You cannot avoid the time lag as we ask for a very strong low pass filter.

plot

code

clc
clearvars

load('Data1_Sept_21.mat');


y = Data;
Fs = 1e6;
samples = length(y);
dt = 1/Fs;
t = (0:samples-1)*dt;

ind = find(t<0.4);
t = t(ind);
y = y(ind);
samples = length(y);

%%%%%%%%%%%%%%%%
% "smart filter" : simple first order low pass filter (2 stages in series = 2nd order filter), 
% the smoothing factor varies with time (low at the beginning , then
% increases to reach max allowed value

alpha = 0.99; % initial value
alpha_max = 0.999975; % final (max) value - must always be slightly below 1


%% main loop 
flag = 0;
ys = zeros(size(y));
ys2 = zeros(size(y));
ys(1) = y(1);
ys2(1) = ys(1);
alphaplot = zeros(size(y));
alphaplot(1) = alpha;

for ci = 2:samples
    if y(ci) >1 % start time to modify (increase alpha)
        flag = 1;
    end
    
     if flag == 1 % increase progressively alpha but saturate to alpha_max value
        alpha = alpha + 1.5*1e-7;
        if alpha>alpha_max
            alpha=alpha_max;
        end
     end   
    
    % two first order low pass IIR filter recursion
    ys(ci) = alpha.*ys(ci-1)  + (1-alpha).*y(ci);
    ys2(ci) = alpha.*ys2(ci-1)  + (1-alpha).*ys(ci);
    
    alphaplot(ci) = alpha; % just for plotting purposes 
 end
clear ys

figure(1)
subplot(211),plot(t,y,t,ys2);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with home made filter' ]);
subplot(212),plot(t,alphaplot);legend('alpha');

[Minerva Bionic]

Hello dear Mathieu,

Undoubtedly, the solution you provided is intelligent and amazing.

But there are still missing parts, about how can I detect the beginning of the variation in coefficient and the initial coefficient.

The time of occurring disturbances in the system is unpredictable as well as the frequency it sweeps to.

However, I really appreciate your intelligent solution.

[Mathieu NOE]

hello Minerva

if we make the coefficient of smoothing dependant of the signal characteristics (amplitude and or frequency) may be we can still using this king of approach . We only have to know when we are in steady state vs transient. That should be managable ? maybe you can tell more about this than I can on my side.

another trial of the same code on your second data file (Data2_Sept_21.mat)

[Minerva Bionic]

Thanks a lot for your reply.

Unfortunately, all information that we can use to implement this stand-alone signal processing system are these signals.

If we need more information. we have to extract it from the signal.

Without considering hardware and latency limitations, the best solution could be using FFT to extract the main frequency and then manipulate the filter coefficient.

For detecting a 20kHz frequency, we need at least a 40k sampling rate, and the minimum required number of points is 40k/5 = 8k (5 is the minimum frequency). It means That we have to implement 8k points FFT and then cope with implementing another algorithm to find the mean frequency.

Due to the hardware and latency limitations, it is unacceptable.

I will upload another data set as soon as possible.

Thank you.

[Mathieu NOE]

hello

do I have to understand that all your focus is to know wheter there is a 20 kHz strong signal or not ? the first data record is not really showing this behaviour spike followed by a 3 Hz rumble + noise) , the second shows a strong 300 hz tone + lot of spurious frequencies above 3 kHz . But you wanted also the first one to be heavily smoothed (once the peak is gone) so I am a bit lost about what is the target or criteria that drives if I have to apply a strong or weak filter ...

I stay at your disposal if there is more I can do

Data1_Sept_21 FFT plot

Data2_Sept_21 FFT plot

[Minerva Bionic]

Dear Mathieu,

There are disturbances in the system that cause instantaneous variation in frequency and amplitude of the signal compared to the steady-state.

I have to detect these abrupt changes in the real-time process and change the filter coefficient accordingly to be able to provide proper output (smooth as much as possible) that can represent actual system behaviour.

In other words, I need to know the main frequency at each time.

If I use a constant filter for whole the process it causes a significant error and malfunctioning of the system.

I have uploaded a new data set and I will upload another one with 20kHz frequency as soon as possible.

Sorry for poor description.

[Mathieu NOE]

hello again

so for the second and third data set , could you draw the "ideal" signal as you did on the first one :

[Minerva Bionic]

Here you are.

I'm not good at all in painting.

However, I think that we should look for a way to detect the main frequency at each time.

Kind regards.

[Mathieu NOE]

Hmmm , not sure if it's so easy to decide based only on frequency info. In the first data , there is this rumble at 3 Hz after the spike. do we consider this frequency as "noisy" or not ? what are we supposed to do ? it was decided in the beginning that the sharp peak should be lightly filtered then after the decay we need strong filtering (exactly where we have this3 Hz rumble).

according to your paintings (they are pretty , believe me !) , I was more thinking about taking small buffers of data and analysing the mean value , deviation (will increase with strong noise) and average slope (like a sharp peak should show high slope condition)

my 2 cents for this evening (in France)

[Minerva Bionic]

In the first data set, there is an impulse at the frequency of 5KHz that gradually sweeps to the steady-state frequency of 3Hz ( the steady-state frequency also varies between 0 and 500 Hz, but we know the steady-state frequency).

I will try to implement your valuable suggestions.

Thank you.

[Mathieu NOE]

Ok , I'll be glad to help further if I can; this is a tricky but interesting challenge !

have a good day

[Minerva Bionic]

Of course, it is.

Especially, if we look for a general solution.

I'm trying different methods.

Thank you, dear Mathieu.

[Mathieu NOE]

hello Minerva

so how are you doing today ?

did you manage to find the "ideal" filter by now ?

all the best

[Minerva Bionic]

Hello dear Mathieu.

I hope you have a great time in beautiful France.

Talking about the problem, I think it is not just about filtering, but it is about "how" and "when" using a proper filter.

As I mentioned before it is not a simple case at all, at least for me.

Thank you.

Kind regards.

[Mathieu NOE]

hello again

thank you , I hope you have a good time also at your location

yes, your problem is not simple , I don't know if I will be of any help from now - let me know

keep the faith !

all the best

[Minerva Bionic]

Of course,

I will update you about this case with my findings or questions.

Thanks a lot.