While using "batchsim" command in MATLAB Parallel Server, how can I transfer the simulation results directly to Amazon S3 bucket?

I am using MATLAB Parallel Server with AWS using the "batchsim" commands. At the end of the simulation, I want to post-process the data and save the data in the Amazon S3 bucket directly, without fetching the data in the local machine to save memory using the command line

Can you please let me know if this possible and how to trigger these commands using "batchsim"?

I don't want to use "fetchOutputs" command, but do some post-processing and directly write the outputs to S3 bucket in AWS.

Answer 1

[MathWorks Support Team]

Since you can save files to S3 using MATLAB Code, it is possible to do this in the "setPostSimFcn" of the Simulation Input object that you pass to "batchsim".

The "setPostSimFcn" is run on the workers and not the client. Thus we can put code in the "setPostSimFcn" to both:

a) Save the data to S3,

b) Delete the data that we don't need from the Simulation Output object so that we don't transfer it to the client if we use "fetchOutputs".

The documentation link below shows an example of using the "setPostSimFcn" for simulations:

https://www.mathworks.com/help/simulink/slref/simulink.simulationinput.setpostsimfcn.html

This way you will not have to transfer the data back from the workers to the client so that you can reduce memory consumption on the client and you can perform various post-processing operations as well using the "setPostSimFcn".

You can use the code shown below to transfer your data using MATLAB's copyfile API.

To do this, the user is first required to set their S3 credentials as environment variables.

From inside of MATLAB:

setenv('AWS_ACCESS_KEY_ID', '1234');
setenv('AWS_SECRET_ACCESS_KEY', '1234');

(The above commands should first be updated to reflect the user's actual S3 credentials).

Then to actually do the copying, again from inside of the same MATLAB session:

% recursively copy all content from inside a local folder to a folder on a given s3 bucket
folderToCopy = 'some/folder/containing/data/to/upload'
s3Location = 's3://bucket/copiedFolder';
copyfile(folderToCopy, s3Location);

Or to transfer a single file:

% copy a single local file to copiedFile.dat on a given s3 bucket:
fileToCopy = 'some/single/file.dat'
s3Location = 's3://bucket/copiedFile.dat';
copyfile(fileToCopy, s3Location);

Obviously, folderToCopy (or fileToCopy) and s3Location will need updating to reflect genuine locations prior to actually attempting a copy.