Writing a tall array / datastore to a .dat

조회 수: 1 (최근 30일)
Stefan Oline
Stefan Oline 2021년 3월 9일
댓글: Divya Gaddipati 2021년 5월 13일
Hello, I'm having trouble writing a large .dat file. It's composed of 64 x 1GB files. I attempted to follow the method here, however I'm having trouble writing the output (a 64 x 407297536 matrix) to a .dat which will end up being ~64GB.
Is there an easy way to write a giant .dat?
Thanks very much.
%% User inputs
channels = 1:64;
demed_flag = 1;
store_flag = 1;
% Choose a directory to store the files
outDir = 'H:\Falkner_lab\Ephys\2020.07.16_Mouse2357\2020.08.28\tall_eg';
writeDir = 'H:\Falkner_lab\Ephys\2020.07.16_Mouse2357\2020.08.28\tall_eg\write';
%% Setup
n_channels = length(channels);
% Check how many samplesa are in a single channel
[Sample_check, Header] = Nlx2MatCSC(['CSC' num2str(channels(1),'%02.f') '.ncs'],...
[0 0 0 0 1], 1, 1, [] );
temp_a = reshape(Sample_check,[],1)';
n_samples = length(temp_a);
clear Header Sample_check temp_a
%% Import .ncs data files from the channels list to individual .mat files
fprintf('*Importing data*\n')
for i=1:n_channels
disp(['Importing channel ' num2str(i) ' of ' num2str(n_channels) ' (' ...
num2str(i/n_channels*100,2) '%)'])
%fprintf('i = %.0f\n', i )
[Samples, Header] = Nlx2MatCSC(['CSC' num2str(channels(i),'%02.f') '.ncs'],...
[0 0 0 0 1], 1, 1, [] );
data = reshape(Samples,[],1)';
% Choose a file name - ensure these progress in order
fname = fullfile(outDir, sprintf('data_%05d.mat', channels(i)));
% Save the data and increment counters
save(fname, 'data', '-v7.3');
clear Samples Header data fname
end
clear i
fprintf('*Importing data complete*\n');
%% Create a datastore from the files
% Read the data back in as a tall array. First create a datastore ...
fprintf('*Creating a datastore*\n');
ds = fileDatastore(fullfile(outDir, '*.mat'), ...
'ReadFcn', @(fname) getfield(load(fname), 'data'), ...
'UniformRead', true);
fprintf('*Creating a datastore complete*\n');
% ... and then a tall array
fprintf('*Storing the datastore in a tall array*\n');
tdata = tall(ds);
fprintf('*Storing the datastore in a tall array complete*\n');
%% Write to file for KS2
if store_flag == 1
fprintf('*Writing data*\n');
fid = fopen('myNewFile_zeros.dat', 'w');
fwrite(fid,tdata, 'int16'); % Code breaks here********************
fclose(fid);
fprintf('*Writing data complete*\n');
beep
end
  댓글 수: 1
Divya Gaddipati
Divya Gaddipati 2021년 5월 13일
Could you also mention the error you get?

댓글을 달려면 로그인하십시오.

답변 (0개)

카테고리

Help CenterFile Exchange에서 Tall Arrays에 대해 자세히 알아보기

태그

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by