Extract and replace data in the middle of a large text file

I have a large (several GB) text file (.vtk) file that contains several different sections of numeric data separated by headers or different lengths that separate the sections. I only want to change the data in one of the sections in the middle. From using grep I already found where that section starts (line N) and how long its data section is (P lines).

As an example I have a text file with a 10x1 vector of interest starting on Line 29:

HEADER_1

data

more Headers & data

...

HEADER_of_Section_I_want

Line 29: 0 ( beginningPx1 vector over the next P lines)

1

2

3

...

Line 38: 9

more Headers & data

I want to be able to extract the Px1 vector, change some of its values and then replace the Px1 vector in the text file with the updated vector (or create a new text file). I know I can use fgetl() to read in the data line by line, but that takes a long time since the vector I am interested in begins around line 100,000,000 and is a 50,000,000x1 vector. Are there any faster ways to extract the data in the middle of the text file, and how can I go about replacing it later?

Thanks for the help,

Brandon

Answer 1

[dpb]

I don't suppose is a fixed-length record file by any chance, is it?

You can, of course, have grep return the byte offset to the lines of interest and fseek() to the beginning location of the file and read from there...that's very quick and probably the most expedient route to take.

You fread() that many bytes first, and fwrite() them to the new file, then write the new data section.

Then fseek() to the end of the section to skip and fread() from there to end to finish writing the new file.

[Brandon Snow]

No, they are of variable length. But your idea works great. I had never used the --byte-offset option of grep or fread() to go byte by byte, but it totally works and is really fast (I only had to wait a few minutes, when reading line by line took almost an hour and didn't even finish the file). Thanks!

[dpb]

Only slight advantage of fixed-length is you can compute the offset knowing the line numbers.

Yes, it's amazingly quick -- last year did a poor man's memmapfile to read arbitrary pieces of data from a very large Fortran output file. That file had fixed structure so could compute offsets and locations of the desired data...it was something like 30X faster than best we could figure out how to do otherwise and by far the simplest to code.