Initialization of small model slow

I've got a pretty small dsp model basically consisting of a few transposes, a reshape, a flip, an fft and a few FIR filters. The input and output ports have fixed sizes and data types. For the FIR filters I wrote a custom MATLAB System Object holding a cell array of dsp.FIRFilters in order to achieve multi-channel FIR filtering for frame-based simulations (see my other question here for the objects code).

Now when I increase the number of channels to above 16 (first dimension of taps in the example code), the model build slows down by quite a while. With 64 channels it is already well above 10 minutes on my machine (i7 4790k, 32GiB ram, SSD).

According to the status bar, the tasks taking the most time are signal type and size propagation.

What can I do to speed up the model build?