Why won't jsonencode encode my entire structure array?

I have a 1x2 struct named "Vortices" that contains data outputs for two objects in a simulation, and is approximately 1.2 GB in size. Each struct in the struct array has 6 fields, each of which are 1x6000 struct arrays as well. I want to encode "Vortices" in JSON and write out to a "vortices.json" file, using the jsonencode() function, as below:

file_handle = fopen('vortices.json','w');
str_vortices = jsonencode(Vortices);
fprintf(file_handle,str_vortices);

"Vortices" structure:

However, I have tried this with structure outputs from several different simulations, and found that my file write-out never results in a JSON file larger than 2,097,152 KB. I have backtracked and found that this is because the jsonencode(Vortices) is not encoding the entire structure, as you can see the encoded string output from jsonencode() does not end with any JSON termination characters like "}" or "]":

I know that the structure I'm trying to encode is large, but I'm running MATLAB 64-bit (R2020a) and the sizes of these objects are well below what inputting "memory" indicates is my largest possible array size, 43.8 GB.

Since I haven't exceeded the maxium array size (or my allocated RAM), why would jsonencode() not be encoding the entire structure?

Answer 1

[Gaurav Garg]

Hey Andrew,

You can divide your data into small chunks and then enode those chunks (one-by-one) into a JSON file (using jsonencode).

You can start by opening a file in write mode, loading a chunk of data, writing the data into JSON, and repeating the same steps but opening the file in append mode this time.

It will lead to some discrepancies because the resultant JSON file will show the variable names multiple times.

For example -

I assume a struct 'data' with variable 'x' in it. For 2 data chunks, I would get something like -

{"x": values} {"x": values}

But, you can easily write a script to handle this case. HINT: Start off with small example to remove the second "x", "}{" and ":" symbols and then generalize the script for the whole JSON file (with all the chunks of data).

Using this approach, you can encode even larger data without any errors.

[Andrew Mistele]

Thanks! Based on your suggestion, I found an even better way to do this when working with structure arrays so that there aren't any overlaps or discrepancies, by encoding each structure in the array separately and adding the delimiting and grouping characters between them as below (in my code I also used newline characters that I have omitted here):

Vortices = struct();
Vortices(1).%fields -- field assignment, etc
Vortices(2).%fields -- field assignment, etc 

fhand = fopen('Vortices-out.json','w')
fprintf(fhand,'[');
fprintf(fhand,jsonencode(Vortices(1)));
fprintf(fhand,',');
fprintf(fhand,jsonencode(Vortices(2)));
fprintf(fhand,']')

Of course, with structures long than 2 elements this could easily be done within a loop.

Interestingly, once this is all encoded and written out in JSON with the above method, I've found it can be read back in and decoded with just a single jsondecode() call -- so one jsondecode() call can decode more data than a single jsonencode() call can encode.

I have no pretensions of being a computer scientist, but this kind of piqued my curiosity and I wonder why it's the case.