Regular expressions on uint8 or single byte characters

I have a 200 MB text file encoded in UTF-8. My maximum array size is around 350 MB, so I can safely read it in using fread('filename','*uint8'). For using regular expressions, I need to turn this into a char array, which blows up the array size by at least a factor of two (depending on encoding, but for my application I can ignore all fancy characters), and thus leads to an "out of memory" error.

I wrote some code that breaks up the original array, so that the matching of the regular expressions works on smaller chunks, but I am still wondering: Can I somehow run regular expressions on the uint8 array? Or is there a char-like variable type that only uses 1 byte per character?

[Cedric]

I don't think that it is possible (it is not an answer though), and I guess that I would have gone for this block-based solution that you implemented, with a little extra logic to ensures that splits are made at places which do not belong to the pattern (e.g. fseek base position, read 50MB, determine negative offset from the end of first char which is not in the pattern, truncate block at this place, and use this negative offset for computing next base position).

[Walter Roberson]

How fancy are your regular expressions?

[Martin Hoecker]

I am relatively new to regular expressions, so I have no idea how they rank on a scale of "fanciness". But here is an example: regexpi(chunk_str,'(?m)^\s*?==[^\n=]+?==[^\n=]*?$','start')

[dpb]

Instead of 'unit8', try 'uchar' Not sure it'll help but it is at least a character class, not an integer.

[Cedric]

Actually, it is simpler to ask what you are trying to match instead of the pattern (copy/paste of chunk of file content or string, and an explanation of what you want to extract). With a little luck, we can perform this using STRFIND (which works on uint8 arrays) or some numeric test on uint8's.