fwrite and MATLAB for a raid0 disk - Only one lane?
조회 수: 2(최근 30일)
표시 이전 댓글
Hello everyone,
I have a raid0 NVMe disk (made up of 4 NVMe disks connected together through a PCIe card adaptator).
The disk works great (up to 12GB/s OUTSIDE MATLAB, PCIe 3.0) but I cannot reach such speed in MATLAB.
It looks like MATLAB is using a single bus lane (aka 3.5GB/s) to write the data to the disk (simple example):
data = randn(1024, 1024, 1024, 'double'); %8 GB
fid = fopen('test.bin', 'W');
tic;
fwrite(fid, data(:), 'double');
toc;
fclose(fid);
Takes about 2.3 seconds which is about 3.5 GB/s so like using one lane... where the raid0 uses 4 lanes (4x4 PCIe).
I am running out of solution, this is not related to the disk/raid0 itself; I tested a lot of raid0 configuration (bios, VROC, Windows raid), the issue only occur in MATLAB. Using hd5f files does not solve that issue, it seems to be related to MATLAB itself.
FYI: I need such speed, in my field/lab we are creating about 1TB data per 5 min the bottleneck is always related to saving the data.
EDIT 1: Removed "b" argument from "fopen"
EDIT 2: Added type "double" to "fwrite"
Thank you a lot.
답변(2개)
Jan
2022년 3월 29일
편집: Jan
2022년 3월 29일
What about trying it as C-Mex?
data = randn(1024, 1024, 1024, 'double'); %8 GB
tic
uglyCWrite(data);
toc
// Short hack, UNTESTED!!!
// uglyCWrite.c
#include "mex.h"
#include <stdio.h>
#include <stdlib.h>
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
double *data;
size_t n, w;
File *fid;
data = (double *) mxGetData(prhs[0]);
n = mxGetNumberOfElements(prhs[0]);
w = mxGetElementSize(prhs[0]);
fid = fopen("test.bin", "w");
fwrite(data, n, w, fid);
fclose(fid);
}
Jeremy Hughes
2022년 3월 29일
I was playing around with this and found that this is much faster (by a factor of 3 on my machine):
fwrite(fid,data(:),"double");
참고 항목
범주
Find more on Large Files and Big Data in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!