Random lags while running app

I have an App Designed application that collects data from an Arduino Pico.

The Pico runs individual trials with accurate to submillisecond timing. MATLAB starts each trial as soon as the previous one is complete. Data is saved in tables as I found that writing to disk after each trial took too long. Even so, the time between trials is longer than it should be AND it can be extremely long on occasion. I've plotted each trial duration and the timing of the end of one to the start of the next. Any ideas as to why MATLAB occasionally takes long breaks? I've tried clearing variables between each trial (doesn't seem to affect my app tables). Removed calls to get time of day (useing Pico Timer for data) various other attemps with Profiler but no luck since the error is rare. Copilot wasn't much help either.

I tried updating to version R2025a and that was much worse!

Any help much appreciated. Code for the app is here, it's big:

https://drive.google.com/file/d/1mwCtRJ-l_2fhk26JYzz2IxidDqkU3IV1/view?usp=sharing

[dpb]

It's probably not MATLAB itself but the OS behind the scenes. Windows is not a realtime OS; it does task switching and garbage cleanup on its own and who knows what all else is in the background in the startup tasks and processes...including apps calling home to see if any updates are available or the like; Adobe in particularl is terrible about that if you have any of their apps installed if you don't stop them manually from automatically updating in the background.

It is somewhat interesting that there seems to be the beginnings at least of a pattern; there are two peaks roughy the same time interval apart when the lag does occur and in both cases the first is larger than the second. Wonder if that might point to what actually is...

You could use the resource monitor tool and log and perhaps find out what is the culprit that way...perfmon.exe is the underlying tool, resmon.exe is a user interface to it...

[Gavin]

Thanks for the pointer to resmon and perfmon. I don't allow the users to run any other apps but looks like I need to check what's in startup. There does not seem to be a pattern that was just one example.

I can disconnect the internet and see if that helps but I suspect stuff will still try to "phone home" and lock things up.

Is there any way to increase the priority of my app to block interruptions from other crap?

[dpb]

There are a zillion background processes loaded by the OS besides just what is in the startup folder.

You can raise the priority of a process, but you cannot block other interruptions entirely, no, the OS is going to do what it thinks it needs to do, regardless.

Is this compiled or are you running inside MATLAB?

[Gavin]

I'm running in MATLAB, I didn't realize I could compile it. (How?)

ResourceMon looks like my app isn't a big hog.

Problem is very sporadic. Today several runs had no issues, then another one had several.

I'm working on version to tighten up the average lag but can't seem to control the big ones. Users have seen lags up to 30 seconds or so long they gave up.

[dpb]

The compiler <mcc<> is another product/TB. I doubt it will solve your problem, I was just trying to find out which way you were/are running. My experience with a compiled app is that it is actually quite a bit slower than the same code running interactively. You'd think otherwise, but it doesn't really compile compile, it builds a call into the MATLAB runtime engine that then executes the code. That, it appears, then doesn't get the benefit of the JIT compiler interactive functions do (hypothesis).

To find your problem, you would have to run a log of the resource monitor at the same time such that you would have a chance to then find the other process that is running at the same time as your delays. May not be an easy thing to find what that is. I wondered about a memory reallocation issue, but given your repetitive nature, I think that would be pretty regular as the same memory amounts would be used repetitively.

[Gavin]

Compiling the app was quick, it took a while to install the runtime platform. Sadly it didn't help at all. In fact the beginning part of loading the app and setting default files took much longer.

Not consistent but still a problem.

[Walter Roberson]

That, it appears, then doesn't get the benefit of the JIT compiler interactive functions do (hypothesis).

Code compiled with MATLAB Compiler is pre-parsed and tokenized, and then at run-time, it uses the same execution engine as interactive MATLAB does. It does use the JIT compiler.

[dpb]

Then why does it run so much slower????

I understand the startup to an extent, but my experience has been the same code compiled runs slower than the same code run interactively on the same machine.

ADDENDUM

OK, I just timed it and it is closer than I felt it seemed to be.comparision.

On the home machine, it is actually reasonably close; the comparison data

Run  Compiled   Interactive Ratio
  1     41.65   31.35       1.32
  2     32.23   31.96       1.01
  3     32.36   31.47       1.03

I'll grant the 2nd and 3rd are the same; not sure why the first compiled case was so much longer? Unfortunately, the way the code is used, it generally is only one run at a time so that advantage is normally lost. I did not try restarting to see if this is reproducible.

Perhaps the ladies' office machines aren't as quick as this one at home so am influenced by an artificial difference; MATLAB isn't installed there so can't do direct comparison on their machines but I know from watching them it takes longer there; the list that is shown in the progress dialog to give indications that something is happening is noticeably slower than the list I just watched go by...but, it does appear that the two are basically the same.

So, if they could do something about the startup time and that black hole that seems interminable after the double-click and one thinks nothing is happening...

[Gavin]

Downloading the RunTime environment, compiling didn't speed it up. I tried it before your other comments Walter was right.

Thank you for all your work on this @dpb I'm not sure what code you were testing with since mine needs an Arduino to run. The first load of a program takes longer as it's loading all those DLL or whatever code files are these days. Later runs will speed up since everything is already in memory.

I made a new version shortening some of the pauses to 0.01 sec. (I have to allow interrupts to get processed.) They take 2 msec. It didn't make much difference for the slow runs but did speed up the fastest to 20msec.

I ran the profiler on one session to see what's going on. My code looks fast enough. More testing shows that in one session it will run fast (20ms between trials) while another will take half a second for most trials and 1 or more seconds for others. Seems random when it will work fine and when it runs slow but now I know after the first couple of trials whether it's going fast or slow.

I would attach File0.html but HTML files aren't allowed to attach!

OK rename File0 to a .txt file you can switch it back to view.

Second profile is for one that ran fast as it should.

FYI I'm running Windoze on an i5-4690 3.5 GHz with 16GB, sadly, MATLAB only uses one core. It would be nice if at least Windows extra crap could run on the other cores and leave me alone! With the app running there is 7.4GB physical memory remaining so there shouldn't be any swapping. Should I turn off VM and page file since I think Windows might use it even when not necessary?

Thank you all for viewing and especially those with comments.

[dpb]

I just added timing code into an app I built; it is immaterial what the code is for the test to show whether the compiled code and interpreted code run at essentially the same speed. You may be right about other needed DLLs although I only timed calculational code after the UI was loaded to avoid all the startup and initial rendering of the UI; my initial thinking was that would be everything needed with the runtime, but since this code also does COM stuff reading/writing Excel files, there probably are some of those other OS and MS pieces needed that weren't already loaded the first time.

I'm totally confused by the discussion about whether it runs slow or runs fast overall; if there is a total difference in behavior from one invocation to another, that is truly bizarre and I would have no explanation at all unless it has something to do with the Arduino interface communications. I thought from the prior graphs the issue is that it generally runs pretty consistently between acquisitions with then an occasional lag; that type of behavior is consistent with OS context switching and/or other background processes that are scheduled or interrupt driven. A behavior wherein the main MATLAB code has two different timing behaviors is something I can't even think about what would be the cause other than as above since are using the Arduino that there's something going on there, not just in MATLAB itself.

Base MATLAB is single-threaded, that is true although a fair number of the base computation-heavy functions in the matrix library, etc., are built to take advantage of mulitiple cores automatically if the internal logic determines it appears to be a case in which it will improve performance. Your code would have to be using such functions for that to ever be the case.

If your application could run pieces in parallel on the same data, not requiring the results from one part to be completed before the next phase can start, then <the parallell processing toolbox> could possibly help, but with it, the base code does have to be able to be parallelized and you have to code it to take advantage yourself. A white paper on <MATLAB Multicore>.

[Gavin]

@dpb thanks for sticking with me on this.

The problem is indeed from one run of a session to the next.

Here are the basics to make it more clear:

The Arduino runs a single trial with parameters sent by MATLAB based on user GUI input (spinners and list boxes etc) What is sent to the Arduino are characters followed by an integer parameter value e.g. for example: I2500O5P3A0B3D4T2400W2000U20S1

Then a 'G' for go character is sent and Arduino Pico runs the trial. The Pico runs very fast and has lots of memory (for a micro controller) I recommend it for any new Arduino RT project.

As the trial progresses Pico sends characters and timestamps back to MatLab as the trial progresses. This is not bundled up and all sent at the end of the trial because the user wants to see the progress as it happens. Events are not very fast it can take several seconds between one and the next or maybe a few per second at most. Anyway I have a circular buffer (probably not needed) to store up the data from the MATLAB completion routine when bytes are available.

The returned data is saved in various arrays and at the end of the session written to files (writing after each trial was definitly too slow). One of the files includes the time of the start of the trial and the time of the end of a trial. So I subtract the end of the last trial from start of the next to get that lag between trials. I'm now also running the profiler inside the program to capture what goes on in a single session. The profile saving takes the most time so I only save it at the end of the session. I can then run another session while the program is still running.

The Pico code has not changed in a while and it's timing is rock solid I was measuring trial duration to compare with inter trial time in above graphs but it doesn't vary so I'm just saving the ITIs

Trials take from 1 to 2 seconds ( usually closer to 1.1- 1.3) depending on the timing of a random input to Pico that simulates a "response" in my test setup.

It's not necessarily the first run session that is fast, that would be too easy! For this debugging purpose I run 8 trials (users run hundreds) and then display to Command Window the (end-next start) times. I'm also saving the profiles for each run.

Inter trial msec examples

Slow 1628 577 531 543 541 1651 532

Slow 1567 598 540 511 525 531 533

Fast 20 20 21 20 20 20 21

Fast 20 20 20 20 20 20 20

Profile of RunOneTrial (not from same trials as above)

calls Total time self time

8 24.990 s 15.397 s Slow run

8 24.305 s 16.578 s Slow run

8 17.741 s 9.842 s Fast run

8 18.189 s 8.152 s Fast run

Let me point out that when it's running well every trial is fast but when it's slow the time from trial to trial can vary greatly. Half a second could be tolerated I suppose but is can be greater than 2 seconds and users have seen is as long as 30s. After that they give up and restart MATLAB and the program.

While writing this I've run 10 sessions and not seen a fast one so I'm going to reboot windoze and see if that helps.

[dpb]

I've no klew, sorry. Is this running in a tight loop or are there timers or other ways to try to schedule the trials?

I can't dream up an explanation of the above behavior unless there is some other process hogging CPU cycles during the "slow" runs that isn't when "fast"

[Gavin]

I'v checked the while loops to be sure they aren't hogging CPU, put in pauses as needed. Hard to think that it's in the code when it runs OK sometimes. But now I'm finding it harder to run correctly with this version.

So I went back to an earlier version of my code that used to always run correctly, and now it's slow too! Is a Windows update screwing me?

[dpb]

Anything is possible, I suppose, but I keep coming back to that the CPU has to be doing _something_ while the time is being consumed; figuring out what that something is is the problem. If the the task running the MATLAB code is not idle, it has to be doing something, even if that something is a pause...

[Gavin]

Using tic toc I've been tracking down where the lag occurs and it looks like it's between when I send a command to the Pico and when it gets it.

Pico is a USB connected arduino module. I'm running Windows 11. Connections is as follows:

 app.PicoCom = serialport(ports(1,2),57600,"Timeout",0.1);
configureTerminator(app.PicoCom,"CR/LF");

I have no problem with incoming data, the issue is when I send the go command 'G' it varies how long it takes for Windoze to get around to sending it to the Pico.

str="G"; then call proc that has

        if ~isempty(str)
            write(app.PicoCom, str, "string");

While typically 20-30 msec which is fine, ever few trials it can take 500ms or longer.

Is there a faster way to do this than write()? Shortening Timeout didn't help. Is that just for incoming?

[dpb]

Good spelunking to have uncovered the "where"...can you tell whether it is a delay from the time the MATLAB line is encountered before the transmission or a delay in actually executing the code itself?

I'm not sure there is any way you can control that, at least with MATLAB-level code; you're pretty-much at the mercy of the OS there, I think.

Only a couple of ideas to try otomh; first would be to use the low-level char() array for the data instead of the string class; this is just the string of bytes without the overhead of the string wrapper to be dereferenced on sending.

str='G';
if ~isempty(str)
  write(app.PicoCom, str, 'char');

Second would be writeline which sends the terminator, too.

I don't know that if you dropped into mex and C to do the communications you could ensure better performance or not; the internals are all builtin functions already, so it would only be whether that could prevent some sort of delay in the MATLAB code itself between lines of code being dispatched.

What would it take to just build a command line interface that does nothing but ping the Arduino without any other distractions and see if that behaves more reliably?

Answer 1

[Gavin]

Turns out it takes a LONG time to update a graph. I moved the screen updates to happen AFTER the start of the next trial and that seems to have resolved it, or at least vastly improved the problem.

AI had some recomendataions but basically said I was on the right track. It didn't catch the screen updating issue as I couldn't give it the whoe program at once.

Thanks to all that gave this some thought and had some suggestions.

[dpb]

Ah, so! Glad you were able to resolve it...

"...it takes a LONG time to update a graph"

Are you redrawing or updating the data in place? For speed, unless it is mandtory for other things to be updated, just modifying 'X|YData' is much faster, although if rendering itself is an issue, that won't be helped.

For realtime graphics, don't forget to look into animatedline and friends...

Answer 2

[Image Analyst]

See if the problem goes away if you set the priority of any of your program's processes to High. Type Control-Shift-Esc to bring up Task manager. On the "Details" panel (Window 11, slightly different in Windows 10) find your processes (any MATLAB processes, your executable, and anything else related to your running of your program. Then right click and go to "Set priority" and set it to High. See if that avoids the problem.

[dpb]

Raising priorities was discussed early on, IA, I do not know whether Gavin actually experimented with doing so or not, however...

It would still be interesting to see if could somehow tie in what is happening with OS context switching at the same time these delays occur...