readtable() interpreting HEX as scientific

I have a script that is importing a csv file that contains decimal and hexadecimal values. Currently I am only utilizing the HEX values. The issue I have is occassionally the data has a value of something like 6E60 which matlab interprets as scientific and then errors on the hex2dec function. Is there a function that would suit better to avoid this issue or is there a format setting which I can apply prior to the readtable function to interpret the csv data as text?

Here are the relevant lines:

clear
clc
format short
warning('OFF', 'MATLAB:table:ModifiedVarnames')
[baseName, folder] = uigetfile({'*.csv';'*.xls';'*.xlsx';'*.txt'},'Select File for Table Data');
file = fullfile(folder, baseName);

The resulting variable is used to create a dataset from a search function

DataSet=csvData(r,:);
avgval=3.3*mean(hex2dec(DataSet.measurement))/65536;

avgval is the line that has issues.

[dpb]

"Here are the relevant lines:"

Excepting for the only one, truly relevant line is the one that was used to actually read the file...

Answer 1

[Guillaume]

In recent versions of matlab, the easiest would be to use detectImportOptions and override the relevant variable type to force it to char. However, R2015a predates detectImportOptions (introduced in R2016b). The only way for you to override the automatic type detection is to specify the 'Format' property in the readtable call. Unfortunately, that means you have to specify the format of all columns and know the exact number of columns rather than letting readtable work that out for you.

It would be something like:

data = readtable('yourfile', 'Format', '%f%f%s%s%f');  %for a file with 6 columns

where '%s' forces the corresponding column to be read as text regardless of its content.

[Michael Bernico]

Thanks! I was afraid that column-by-column sorting or format assignment was going to be needed.

This has worked correctly with the problem children in my data set.

[dpb]

If using readtable and doing this often for a number of these files that are similar in construction you may want to build an import options object for the specific file structure using detectImportOptions In it you can set variable types and other specific features of the file specifically that otherwise readtable has to explore and determine on its own, lacking guidance.

You'll find that if you take one of these input files that has a hex string containing a letter as the first element that then the column will be imported as the cellstr array; readtable tries to convert the first row (or maybe a first few, I've not pursued the details to that depth) and makes the determination of the column data type from that effort pass/fail result. If you pass the import options object that gives readtable the heads up, it can bypass that. The string format does the same but there are other specifics that can be set for the file structure as well. NB: You don't need to call detectImportOptions on every file; just create one object and save it for use with all files of the given structure.

It's a glaring weakness imo that textscan, readtable cannot deal with hex input data (nor complex), only fscanf can handle hex; C doesn't know anything about complex.

[Guillaume]

The only problem with detectImportOptions is that it's not available in the OP version...

[dpb]

Is OP limited? Doesn't seem to say altho (after the fact) I did see your note re: it...I certainly didn't see anything in the Q that said so. Well, maybe it'll help somebody else even if not OP! :)

I did give your answer a vote for having mentioned it...

I really do not understand the lack of support for HEX and COMPLEX by TMW for what is supposed to be scientific toolset. :(

complex is a little easier to understand the intertia of C not supplying so it's all from scratch; %X is in C and fscanf so why it wasn't/isn't carried over is harder to fathom.

[Guillaume]

For once, the OP did fill the matlab version on the right -> (below the tags).

Totally agree about the lack of support for hex in textscan (and therefore readtable).

[dpb]

"OP did fill the matlab version on the right -> (below the tags)."

Huh. Took me about a minute to find it even with the pointer... :) I never look outside the text of the Q? and/or Comments so whiffed on that entirely.

[Guillaume]

"It's a glaring weakness imo that textscan, readtable cannot deal with hex input data (nor complex), only fscanf can handle hex; C doesn't know anything about complex."

Now that 2020a is out, you'll be pleased to know that this glaring weakness has finally been fixed.

textscan now has '%x' and '%b' format specifiers, readtable automatically recognises the 0x and 0b prefixes and there's various options for detectImportOptions to support direct conversion from hexadecimal or binary (with or without prefixes).

Hooray!