Combining/Merging Output for Multiple NetCDF Files

Good afternoon! I have a series of 31 netCDF files (.nc4), each representing a day of the month. Variables for each of the files include latitude, longitude, and time among other things (note: there are 4 time-steps included in each .nc file, so the variable "time" is a 4x1 column vector representing four distinct times during the day [0h 6h 12h 18h] for each day). I am trying to run a for loop to combine all of my netCDF files and their respective variables into one single file I can easily work with in the future. However, I am having difficulty doing so because I'm not sure how to combine the output of each of the variables. Any suggestions? Here's what I have so far...

ncfiles = dir('*.nc4')      %this identifies all files within my folder ending in '.nc4'
Nfiles = length(ncfiles)    %this indicates the number of files in my folder (i.e. 31)
for i = 1:Nfiles
    ncfiles(i).name         %this just helps my OCD in listing the name of the file
    time = ncread(ncfiles(i).name, 'time')          %this reads the variable "time" for each of my 31 .nc files
end

Answer 1

[Walter Roberson]

ncfiles = dir('*.nc4');      %this identifies all files within my folder ending in '.nc4'
Nfiles = length(ncfiles);    %this indicates the number of files in my folder (i.e. 31)
all_times = cell(Nfiles,1);
for i = 1:Nfiles
    ncfiles(i).name         %this just helps my OCD in listing the name of the file
    all_times{i} = ncread(ncfiles(i).name, 'time')          %this reads the variable "time" for each of my 31 .nc files
end

This postpones all consideration of the best way to combine the information until after the data has been read in and stored. Afterwards you could

time = horzcat(all_times{:});

for example if each individual entry is a column vector of consistent length.

[Michelle De Luna]

I appreciate your help, Walter! This worked fantastically! Again, thank you.

[ANIE]

Hi everyone,

I also want to merge multiple nc files. But my files are unequal in longitude. Some are 406x271 (lon x lat) and some are (408 x 271) (lon x lat). How to do merging in such a case?

Looking forward to any kind of suggestions or help...

Answer 2

[L Chi]

I wrote a function (FUN_nc_merge) for this purpose previously. It is a part of my package for handling netcdf files and can be downloaded from here: https://github.com/HappySpring/Easy_NetCDF

An example is shown below:

% input_dir: path for the folder containing the files
    input_dir = '.';
% filelist
    filelist  = dir(fullfile(input_dir,'Merge_Demo*.nc'));
% output filename
    output_fn = 'Merged_Output.nc';
    
% name of the demension to be merged.
merge_dim_name = 'time';

% compatibility_mode:
%     compatibility_mode = 1: write netCDF in 'CLOBBER'; Compression would be disabled.
%     compatibility_mode = 0: write netCDF in 'NETCDF4'.
compatibility_mode = 0;

strvcat( filelist(:).name )

FUN_nc_merge( input_dir, filelist, output_fn, merge_dim_name, compatibility_mode )

[Hannes Nevermann]

L Chi, I have seen your toolbox on github. Are you still working on it?

[Eli]

Wow, this was SO HELPFUL!! Thank you so much @L Chi !!!

[Augusto Gabriel da Costa Pereira]

your function contains an error, see:,

When I run the commands, at the end the following message appears:

Unrecognized field name "all".
Error in FUN_nc_merge (line 223)
       MV.all = [ MV.all ; tem(:)       

The script I used is just below and the link with google drive is the NetCDF files I used https://drive.google.com/drive/folders/1yts62Bbd5P1N_O3we3R2iQi0PgYbkJNT?usp=share_link

    % input_dir: path for the folder containing the files
input_dir = '/home/augusto/Documentos/Dados_Mensais_CHIRPS';
% filelist
filelist  = dir(fullfile(input_dir,'Merge_Demo*.nc'));
% output filename
output_fn = 'Merged_Output.nc';
    
% name of the demension to be merged.
merge_dim_name = 'time';

% compatibility_mode:
%     compatibility_mode = 1: write netCDF in 'CLOBBER'; Compression would be disabled.
%     compatibility_mode = 0: write netCDF in 'NETCDF4'.
compatibility_mode = 0;

strvcat( filelist(:).name )
cd /home/augusto/Downloads/Easy_NetCDF-main
FUN_nc_merge( input_dir, filelist, output_fn, merge_dim_name, compatibility_mode )

[L Chi]

Hi @Augusto Gabriel da Costa Pereira,

I have tried your codes and it works after editing "input_dir" and "filelist".

I put the "data_chirps*.nc" in "./Data/" and here are the codes work for me:

% input_dir: path for the folder containing the files
input_dir = './Data/';
% filelist
filelist  = dir(fullfile(input_dir,'data_chirps*.nc'));
% output filename
output_fn = 'Merged_Output.nc';
    
% name of the demension to be merged.
merge_dim_name = 'time';
% compatibility_mode:
%     compatibility_mode = 1: write netCDF in 'CLOBBER'; Compression would be disabled.
%     compatibility_mode = 0: write netCDF in 'NETCDF4'.
compatibility_mode = 0;
strvcat( filelist(:).name )
FUN_nc_merge( input_dir, filelist, output_fn, merge_dim_name, compatibility_mode )

A mlx file with outputs is also attached.

[L Chi]

@Hannes Neverma, I am still using the toolbox frequrently and I do plan to maintain it for a long time since handling NetCDF files is kind of the fundation of research in my field. You're welcome to try it. I do not check my Matlab Community account frequently. Please submit an issue in github if you have any questions/suggestions.

[Augusto Gabriel da Costa Pereira]

ok, thanks.

[Afiq]

Thank you so much bro

Answer 3

[Abdullah Azzam]

Hi @Michelle De Luna I am new with matlab. i have run the above code and i couldn't save the merged or combine file. Can you please help me to merg multiple Nc file into the same directory by a specified name. Thanks

[Walter Roberson]

netcdf files can contain hierarchies of variables and attributes and groups, and the time base for various files is not necessarily the same. You will need to be more specific about what kind of content is in your files.

[Michelle De Luna]

@Abdullah Azzam: Hi Abdullah! I hope you're well. Can you please provide more details about the code you ran and the files you'd like to work with? Maybe we can look into the issue a bit deeper to help you resolve your question. Best regards...

[Abdullah Azzam]

Thank you very much dear @Walter Roberson and @Michelle De Luna. I am working with GCM climate model which contains multi-NetCDF files. I need to merg multi-Netcdf file to a single to ease in processing using matlab. The variable included in all Netcdf files are same i.e. time, lat, lon and variables such as pr (precipiation), tasmax (Maximum temperature), and tas (Mean temperature) etc. Thank you both for helping.