Loading in Data from large files

조회 수: 24 (최근 30일)
Mitch Lautigar
Mitch Lautigar . 2022년 5월 19일
편집: dpb . 2022년 5월 23일
Hello everyone,
Going to try and explain what i'm trying to do below and I welcome any suggestions the community can provide.
  • Problem: I am trying to read in a large (>10GB) binary file and parse specific data. I can already parse the data, but I run out of RAM on MATLAB when parsing such large amounts of data.
  • Current Logic: I have been using memmap to load in the data, and it worked up until I started having to deal with large file sizes. I am aware that memmap can skip a specific amount of memory and start at a later point, but i need to load a specific amount of bits at a time. I'm trying to avoid using fread since it takes so long.
  • Goal: I am looking to parse through these binary files in smaller sections by using a command or some programmed logic to read in a small percentage of the binary file at a time, run the computations, grab what I need and store it elsewhere, then grab the next small percentage of the binary file and repeat. I've included some detailed notes below for my intent to try and help; but this is a project I am not allowed to share code too (copyright).
Procedure for what I want to do:
  1. Load in the header info of the file. this is static info that is easy to load and I can do this pretty well already.
  2. Load in a percentage of the file. For this specific logic, let's assume my file is 10 GB and I want to read it in in 500 MB sections.
  3. Parse the 500 MB I read in. I'm aware that I may only read 492 MB, in which case I need to make sure I read in 508 MB the next time.
  4. Store the parsed data in a structure
  5. clear used variables to reuse for next section of code logged.
  6. repeat
I hope this helps. I'll try to keep an eye on this post moving forward, but it might take me some time to respond as i'll be traveling.
  댓글 수: 11
dpb 2022년 5월 23일
We're in that big D4 area that stretches from KS all down E NM and W TX.

댓글을 달려면 로그인하십시오.

답변 (0개)


Help CenterFile Exchange에서 Data Type Identification에 대해 자세히 알아보기




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by