Hi, I have large text files that I am trying get through "datastore". Can we get multiple files into a single datastore? If so, How I can pull out data from each file into separate tall arrays?

I tried to get the data from two files. It says that i has got bot files, but I cannot get them into separate tall arrays.

ttds = tabularTextDatastore('K:\geo-cloud-files\home\NayaniIlangakoon','FileExtensions',{'.csv','.txt'})

But the datastore structure shows only the data from one file, not both. Anyone has any ideas how can I pull out data from datastore?

[Kojiro Saito]

You mean your datastore only contains one file? Could you type the following to see which files are stored in your datastore?

ttds.Files

[Nayani Ilangakoon]

Hi,

Thanks for the reply. Here is what I got. It shows both files (the test case) in datastore.Files ans =

    'K:\geo-cloud-files\home\NayaniIlangakoon\140823_161804_1.pls_PULSE.txt'
    'K:\geo-cloud-files\home\NayaniIlangakoon\L11 - Scanner 1 - 140823_161804_1 - originalpoints.txt'

But all other descriptions are only for one file. I dod not know how I can pull out the information of the other.

[Kojiro Saito]

Your datastore seems to contain 2 files. Could you confirm after converting from datastore to tall array, tall array contains the summary of length of both files by the following?

 tt = tall(ttds);
 s = size(tt);
 tableSize = gather(s)

[Jiro Doke]

Also, note that a datastore is meant to represent a "collection" of files. To use datastore (and tall), all the files in the collection must abide by the same structure, hence only one of each descriptions, such as NumHeaderLines, Delimiter, etc.

That means all the files you include in the datastore must have the same number of columns (in the same order), same number of header lines, same delimiter, etc. Take a look at this page from the documentation:

Getting Started with Datastore

[Nayani Ilangakoon]

Hi Jiro,

Thanks for the reply. Yes, my files are not in the same structure. Their columns are not same, not even the header lines.

[Jiro Doke]

In that case, you shouldn't be creating a single datastore for both files. Instead, you would create one for each:

ds1 = tabularTextDatastore('K:\geo-cloud-files\home\NayaniIlangakoon\140823_161804_1.pls_PULSE.txt');
ds2 = tabularTextDatastore('K:\geo-cloud-files\home\NayaniIlangakoon\L11 - Scanner 1 - 140823_161804_1 - originalpoints.txt');

[Nayani Ilangakoon]

Thank you so much Jiro. I will try in this way.