Real-Time Audio Processing: Varying Delay Depending on Initial Under/Overruns

Hi,

i am using the Audio System Toolbox to do real-time processing. When running my program multiple times I experienced that the delay between input and output was not always the same though i did not change the code between the runs. The only difference i could observe was a different amount of underruns when starting the program. After a few frames the system was then in a 'stable loop' and running without any underruns - however, with different delays.

A simple test set-up: Put a microphone to the soundcard (Fireface UC in my case). Use the audioPlayerRecorder from the Audio System Toolbox. Within the processing-loop map the input vector to the output vector (so basically just a talkthrough). Force some underruns within the first frames (i observed this often before the system then runs smoothly). When system is stable, check the delay between input and output. Repeat - but with different amount of underruns.

Observation: With different amounts of initial underruns, the delay when the system is stable is different.

Expected Behaviour: To my understanding i would expect the following framing steps: 1. Frame A is collected at the soundcard. 2. When done, frame B is collected at the soundcard and frame A is processed within Matlab. 3. After that, frame C is collected, frame B is processed and frame A is read to the output of the soundcard. 4. Frame D is collected, frame C is processed, frame B is read to the output, and so on...

This should result in a delay of 2 frames. When overruns occur, i would expect input frames to be dropped. When underruns occur, the processing was too slow and probably the old output buffer content is read out again - the processing should be interrupted and start processing the following frame. However, in both cases the framing should remain the same.

Summary/Question: Since the processing after the initial under/overruns is stable, the complexity of the processing is not too high and must therefore be processable within one frame. However, the initial under/overruns seem to influence the resulting framing and can cause output frames to be (constantly) read out later than expected - which in turn yields a (constant) higher delay.

What is happening here? Is my understanding of the framing wrong (or too much influenced from Digital Signal Processors?)? Is this behaviour a bug or in the end even desired?

Best regards and thank you very much for helping out!

Answer 1

[plewanig]

Hello. this is not an answer. I just want to add here that I experience the exact same problem. I get dropouts in the beginning of an audio loop from time to time (underruns), which cause a permanent change in the round trip latency. In my case I use a buffer size of 2048 samples, an asio driver with the "dante virtual soundcard" and there is no processing involved. I play back and record (multi channel) in order to measure transfer functions. So the conditions should be optimal in order to have no dropouts. Are there any new regarding this subject? Is there any documentation about how to properly "setup()" an audio system object, especially the audioPlayerRecoder? wbr

Answer 2

[Zhao Wang]

As a first step, you can run the Measure Audio Latency example to verify that your audio I/O system is configured correctly: https://www.mathworks.com/help/audio/examples/measure-audio-latency.html

For profiling your code: https://www.mathworks.com/help/audio/examples/measure-performance-of-streaming-real-time-audio-algorithms.html

Output signal silence (underrun) occurs when the device buffer is empty and it is time for digital-to-analog conversion. This results when the processing loop in MATLAB does not supply samples at the rate the sound card demands. The number of samples underrun is returned when you call your audioPlayerRecorder or audioDeviceWriter object.

Input signal drops (overrun) occur when the processing stage does not keep pace with the acquisition of samples. The number of samples overrun is returned when you call your audioPlayerRecorder or audioDeviceReader object.

Generally, since you encounter overrun or underrun, I recommend you to try improving your I/O system:

1. Identify when the overrun or underrun occurs. If it occurs in the first few iterations, consider calling setup on your System objects before the loop where real-time processing is required. You can also run the I/O system with dummy data for a few frames before starting the real processing. For more information, see Measure Performance of Streaming Real-Time Audio Algorithms.

2. If you are using a DirectSound driver on a Windows platform, consider switching to a WASAPI or ASIO driver. ASIO drivers have the least overhead. If you are using an ASIO driver, make sure to match the frame size in MATLAB to the ASIO buffer size.

3. If you can afford to add more latency to your application, consider increasing the buffer size of your object. By default, the buffer size is the frame size of the data processed by the audio object.

4. If you can afford to decrease signal resolution, consider decreasing the sample rate.

5. Close all nonessential processes on your machine, such as mail checkers and file sync utilities. These processes can asynchronously ask for CPU time through interrupts and disturb the audio processing loop.

6. To maximize performance, remove all plotting and visualization from your real-time loop. If you require a visualization to update in your processing loop, use a DSP System ToolboxT scope such as dsp.TimeScope, , dsp.SpectrumAnalyzer, , or dsp.ArrayPlot, . Follow the recommendations listed in point 1 to setup and pre-run your scopes. If you require custom graphics or are processing callbacks in the loop, use the drawnow command and specify a limited update rate to optimize your event queue.

7. If the processing loop is algorithm-heavy, try profiling your loop to locate the bottlenecks, and then apply appropriate measures: o Replace handwritten code with MATLAB features that have been optimized for speed. o Follow best practices for performance: Techniques to Improve Performance. o Generating MATLAB executables (MEX files) using MATLAB CoderT may result in faster execution. See Desktop Real-Time Audio Acceleration with MATLAB Coder for an example. You can also generate standalone executables (EXE files). See Generate Standalone Executable for Parametric Audio Equalizer for an example. o If you are considering turning your algorithm into a VST plugin, then try running it as a VST plugin within MATLAB. VST plugin generation uses C code generation technology under the hood, and running the generated VST plugin within MATLAB may result in faster execution than with your original MATLAB code. See Design an Audio Plugin and Host External Audio Plugins to learn how to design, generate, and then host a VST plugin.

About Latency, there are output latency and input latency. If properties and frame size remain consistent, the ratio of input latency (between when audio enters the sound card to when the frame is output by the processing stage) to output latency (between the generation of an audio frame in MATLAB to the time that audio is heard through the speaker) is consistent between calls to an audioPlayerRecorder object. To minimize latency, you can:

1. Optimize the processing stage. If your processing stage has reached a peak algorithmically, compiling your MATLAB code into C code using MATLAB Coder may result in faster execution.

2. Decrease the sample rate.

3. Decrease the frame size.

[JFr]

Thank you very much for your answer and recommendations.

First of all, let me briefly comment on your recommended points:

Points 2-4: We do indeed use a RME Fireface UC soundcard with ASIO drivers. Matlab frame size and ASIO buffer size are matched. Since latency is very important for us, we have to chose the buffer size very small, and the sample rate is already set to the possible minimum.

Point 5-7: All nonessential processes are closed and we did some optimizations. Since our processing runs smoothly after the few initial under/overruns, this seems to be alright.

Point 1: This point seemed very promising to me. Since our setup is already called before starting the real-time loop, we tried running the I/O system with empty dummy data for a few frames before starting the real processing. The result is interesting: the initial under/overruns are still there, but now during the processing with the dummy data. After the few initial under/overruns the dummy data is processed smoothly and the subsequent real processing runs smoothly as well. However: the resulting final delay still depends on the initial under/overruns!

So my question on this remains: Why (and how) do the initial under/overruns cause output frames to be (constantly) read out later than expected and therefore cause a (constant) higher delay?

Best regards and thanks again for helping out!

(PS: In the Measure Audio Latency example you linked, it says for the final table 'the latency values in the table are the average of 10 runs'. Does this averaging imply that there is no deterministic framing?)

Answer 3

[JFr]

Since this topic is not finally answered, i may warm up the discussion with some more or less new input.

With the release of MATLAB R2017b the documentation of the Audio System Toolbox now contains an interesting sentence: 'When using ASIO (or CoreAudio with Mac OS), the latency measurements are consistent as long as no dropouts occur. For small buffer sizes, it is possible to get a clean measurement in one instance and dropouts the next.' (https://de.mathworks.com/help/audio/examples/measure-audio-latency.html, section 'Some Tips When Measuring Latency').

This behaviour is basically what i described in my initial question and, with that sentence, it seems to be either tolerated or even wanted.

Since i'm still very taken by the idea of using the Audio System Toolbox for my development it would be important for me to know what i can expect in the near future. Are there any (soon) updates planned that would allow for a different (and consistent) low-delay handling as described in my previous posts or is this the final behaviour i can expect from the toolbox?

Best regards and thanks again for an answer.

Answer 4

[James Rex]

I have experienced the same issue using the audioPlayerRecorder object in MATLAB R2017a, and in my latest version MATLAB R2020b (both under Windows). Observed behaviour is that live audio round-trip latency, with loopback from audio device output to input, is not constant, but tends to increase whenever audioPlayerRecorder detects multiple under/overruns within a short time. I believe this is caused by MATLAB software, as round-trip latency does NOT vary, even after under/overruns, when using the same audio interface hardware (Focusrite Scarlett 8i6) with Max 8 software performing the same function (simultaneous live audio input + output).

I believe this is an undocumented feature of the MATLAB audio interface driver software, and I would like MathWorks staff to confirm this, and tell me how I can achieve constant low-latency live audio processing in MATLAB, even after occasional under/overruns (heard as audio dropouts). Ideally, the MATLAB audio interface driver software would be updated to allow users to choose either (a) current behaviour, where latency may increase automatically, or (b) guaranteed constant latency, even after under/overruns.

Note that I have already applied all the solutions suggested by Zhao Wang above in this thread, where they are possible in my application. I can achieve the low latency I need with MATLAB for a short time after starting audio processing, but after a few tens or hundreds of milliseconds latency usually increases significantly. Thereafter, latency remains constant, but is now too high for my application.

Hoping for an answer and a fix :-)

Answer 5

[Jimmy Lapierre]

Some updates since these issues were raised:

• Starting 21b, Audio Toolbox I/O has a new buffering mechanism to prevent the latency from increasing when there is a frame underrun.
• The Impulse Response Measurer application is another alternative for measuring loopback latency.

[James Rex]

Thanks for this good news!

I have used MATLAB R2021b to repeat my measurements of live audio round-trip latency, with loopback from audio device output to input, and confirmed that the issue I had with earlier versions is no longer present!

Earlier versions would sometimes automatically increase latency, following under/overruns, but when under/overruns occur in R2021b, the latency returns to its previous value, once the under/overruns have finished. This is very useful for my application, where latency must be low and remain constant.

Occasional audio dropouts, caused by under/overruns when another Windows task interrupts MATLAB, are not a big problem, but persistent increases in latency ARE a big problem for my application.

[Jimmy Lapierre]

Thank you very much for checking it out, this is exactly the pain point we wanted to eliminate after getting customer feedback.