Unstable LPF filter response in Simulink

Hello,

I'm trying to simulate an RF system. I have an analog sinewave carrier at 700 MHz which has analog sinewave phase modulation at 700kHz:

Fs = 100e9;               % Sampling rate
Fc = 700e6;               % Carrier frequency in Hz
t = (0:1/Fs:0.000004)';   % Time points for sampling
f = 0.7e6;
phi = 0.0025*sin(2*pi*f*t);
y = sin(2*pi*Fc*t+phi);
y = [t, y];

This signal then goes through this system:

The idea is to demodulate this signal into I and Q streams, digitize them with a 12 bit adc with 100 MHz clock (here I just downsample the signal simulating a FIR compiler) low pass filter the I and Q streams and then calculate the phase modulation with an atan2.

My problem is that up to the LPF everything works fine, from Fs = 100e9 I downsample to Fs = 100e6, and before the LPF the I stream is modulated with the imposed modulating signal (bottom left square):

However, at the output of a low pass filter (upper right square) I get a signal with a huge amplitude swings, which causes the atan2 to go crazy. What am I doing wrong?

There's nothing special in the filter parameters which would cause it to be unstable: