Script to access Amazon S3 files
조회 수: 32 (최근 30일)
이전 댓글 표시
I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?
Thanks
댓글 수: 0
답변 (3개)
Jhon Wine
2018년 1월 24일
I found datastore to be very easy to use. Run the code below in each worker.
%Set S3 Credentials
setenv('AWS_ACCESS_KEY_ID', 'ID');
setenv('AWS_SECRET_ACCESS_KEY','Key');
setenv('AWS_REGION', 'us-west-2');
%Load Data
fp= 's3://mybucket/data/file.data';
ds=fileDatastore(spectralFilePath,'ReadFcn',@AWSRead);
data=ds.read;
Also define this function
function data= AWSRead(fileName)
fid = fopen(fileName);
data= fread(fid,inf,'short');
fclose(fid);
end
댓글 수: 0
David Fink
2017년 9월 28일
Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.
댓글 수: 1
Jhon Wine
2018년 11월 17일
Is there an example, how to use webread and webwrite with S3? There is no good tutorial..
Harish Sundaresh
2021년 6월 26일
can MATLAB read .json files from aws s3? I get these errors
%% webread
webread(fname)
Error using webread (line 119)
The 's3' protocol specified in URL, 'url', is not supported. Specify the URL with the protocol 'http://' or 'https://'.
%% jsondecode
Error using jsondecode
JSON syntax error: expected value but found end of text.
댓글 수: 2
Dmitry Kaplan
2022년 2월 8일
I could not get webread to work. I am sure there is a way, but I could not figure it out.
Instead, I use something like this
fds = fileDatastore('s3://blah/blah/filename.json','FileExtensions',{'.json'},'readfcn',@fileread);
file_data = read(fds);
This defines the read function for the datastore to be fileread(). Invoking the read() actually runs fileread() and passes to it the filename -- because the fds.ReadMode is 'file'.
참고 항목
카테고리
Help Center 및 File Exchange에서 Web Services에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!