get MATLAB-inside-Python output redirected to Python in real-time

when i use the matlabengine Python package to launch MATLAB in Python code and run a MATLAB function, I redirect its output to a string stream that gets displayed on my Python GUI application's console like this:

import matlab.engine
adapter = MatlabStdoutToLogger()
engine: MatlabEngine = matlab.engine.start_matlab()
engine.func(..., nargout=0, stdout=adapter)

however, i don't get real-time output from the MATLAB script: it only gets printed at the end of MATLAB function's execution.

i need to see the output of fprintfs or disps i put in my MATLAB function which report execution progress in real-time. i need to know how much time will it take to finish, to be able to ditch experiments that take an eternity to compute.

so far i see only the vice-versa question like here: https://www.mathworks.com/matlabcentral/answers/2019676-return-print-statement-output-from-python-file-in-real-time-when-using-pyrunfile however, it deals with the same problem but for Python running in MATLAB, not MATLAB running in Python.

what can you recommend to fix this? it appears like something to do with buffering. i asked AI assisstants already but to no avail, they recommended drawnow('update') to flush MATLAB's output, but when i put these after printing statements, nothing gets displayed in my Python real-time, i again need to wait for the MATLAB execution to finish.

[Torsten]

Can you give a link on how execution progress of a MATLAB function (only in MATLAB itself, without Python communication) can be reported ? I've never heard of such a facility - most probably because execution time will be strongly problem-dependent and usually cannot be foreseen. Or do I misunderstand your question ?

[Anna]

@Torsten i use fprintfs/disps to report progress in form of a log to standard output. likely built-in MATLAB methods for that don't exist, you need to tailor these for your task. i have an image processing method that goes over all pixels in an image in a nested loop, so i count the number of processed pixels and print statements whenever a 10%-portion gets processed. furthermore, my Python GUI captures the output and logs it with a timestamp so i get an approximate idea of how long it takes to process a 10%-chunk.

Answer 1

[Anna]

• customizing the stdout argument of type io.StringIO() with inheritance on Python's side doesn't work
• keeping a percent variable on MATLAB's side and sending the MATLAB engine subprocess to run in background with engine.func(..., background=True) on Python's side, periodically polling engine's workspace for that percent variable, also doesn't work

the only solution that works for me (as suggested by ChatGPT) is using a log-file as an intermediary. it looks like this is the only valid way to avoid buffering in either Python or MATLAB. you need to make sure you have a certain path to keep the log-file in. your MATLAB script/function launched from Python must work in the background and must write your desired output to that log-file (most probably you need to pass the log-file's path to MATLAB from Python). meanwhile Python should poll the log-file for new content that the MATLAB script supplies.

here's an example how to setup MATLAB's side:

%% progress setup
logfile = fullfile(cache_dir, 'matlab_progress.log');
fw = java.io.FileWriter(logfile, false);
bw = java.io.BufferedWriter(fw);
cleanupObj = onCleanup(@() bw.close());
%% an example output inside a nested loop (tailor it for your long-running procedure)
bw.write(sprintf('Progress: %d%% - %d/%d\n', pct, processed_count, total_to_process));
bw.flush();

then the corresponding setup in Python is:

import matlab.engine
engine: matlab.engine.MatlabEngine = matlab.engine.start_matlab()
logfile = os.path.join(cache_dir, 'matlab_progress.log')
if os.path.exists(logfile):
    os.remove(logfile)
future = engine.func(cache_dir, ..., background=True)
last_pos = 0
while not future.done():
    try:
        with open(logfile, 'r', encoding='utf-8') as f:
            f.seek(last_pos)
            for line in f:
                logger.info('MATLAB: ' + line.rstrip('\n'))
            last_pos = f.tell()
    except FileNotFoundError:
        pass
    time.sleep(0.8)  # polling frequency
future.result()