ArrayDatastore 2x1 combine to 1 column

Hello, I would like to combine 2 arrayDatastores vertically.

I tried it with combine and permute, but it always gives a 1x2 instead of a 2x1 back:

A = cat(3,magic(10),magic(10),magic(10));
B = cat(3,magic(10),magic(10),magic(10));

a1= arrayDatastore(A);
a2= arrayDatastore(B);

a3= combine(a1,a2);
a4=tall(a3);
a5=cellfun(@(im) permute(im,[2,1]),a4, 'UniformOutput', false);

Need to combine it with other datastores later.

Answer 1

[Sreehari Hegden]

If you are looking to combine 2 ArrayDatastores vertically to essentially merge their data, one option would be to do a vertcat of readall outputs from both the ArrayDatastores a1 and a2.

A = cat(3,magic(10),magic(10),magic(10));
B = cat(3,magic(10),magic(10),magic(10));

a1= arrayDatastore(A);
a2= arrayDatastore(B);

data = vertcat(readall(a1), readall(a2))

You can also consider writing these data to files and then construct a single TabularTextDatastore or SpreadsheetDatastore from the files.

Another sustainable option would be to write a custom datastore that would essentially do a vertical concatenation of the datastores, instead of the horizontal concatenation done by combine.

The datastore will have 2 properties:

1. a list of the underlying datastores
2. index indicating which is the current underlying datastore to read from

For this datastore,

• At the beginning, we set the index to 1 and start reading from the first ArrayDatastore a1 till it has no more data.
• When hasdata for a1 is false, increment the index by 1 to point to the second ArrayDatastore a2.

The readall for this custom datastore would do a vertcat of the readall from all underlying datastores. This is extensible for more number of ArrayDatastores as we can just pass them to the custom datastore constructor.

Please see attached a sample VertcatDatastore.

Now, construct the VertcatDatastore using the ArrayDatastores a1 and a2 and do a readall.

vds = VertcatDatastore({a1, a2});
data = readall(vds)

NOTE: As we are seeing more similar queries, we are working on enabling vertically merging multiple datastores. That would essentially provide a means to read from the underlying datastores sequentially.

[Odo Luo]

This solution is so simple and effective, I am ashamed I did not think of it!

Thank you for your help!