Run cmd faster and parallelly in background

Hello!

I have this code which runs an executable going in to each folders. The executable opens a command prompt window. Executable also needs couple parameter inputs to make sure it runs.

Even though each individual run takes only few seconds total number of folders/executions could be upto 20000. So I am looking for a way to execute this faster in Matlab, and if possible in parallel and in background, - without a pop up window.

Any suggestion would be greatly appreciated.

code=['@ECHO off',newline,...
'setlocal enabledelayedexpansion',newline,...
'set dir=%1',newline,...
'SET "var=1"',newline,...
'for /D %%a in (%dir%\*) do (',newline,...
'echo processing: %%a',newline,...
'start /D "%%a" /min "Dummy" cmd.exe /c "Dummy.exe < parameters.txt"',newline,...
'IF !var!==3 (PING localhost -n 3 >NUL) & SET "var=1" ',newline,...
'SET /A "var=!var!+1")'];

dlmwrite('Dummy.bat',code,'delimiter',''); % writes out a batch file 

for run=1:4 % 4 folders each containing around 5000 subfolders
name = (sprintf('Doe%i' ,run));
CmdStr=sprintf('Dummy.bat %s',name);
system(CmdStr);
end

Answer 1

[Benjamin Thompson]

Due to the overhead of launching and running CMD so many times, you may not gain much or it may run slower. After first optimizing the algorithm, the second principal of parallelization is to keep the batch size as large as possible. So maybe if you have 4 CPU cores or less, break it up into running one job per each of your folders.

Instead of running the start and cmd processes, try using the system function in MATLAB, and then maybe parfor or parfeval to parallelize it into a parallel pool on your system.