Incremental median across pages of a 3D array
조회 수: 1 (최근 30일)
이전 댓글 표시
I am trying to compute the inter-page median,
B=median(A,3)
of a 3D array A except that A is too large to be held in memory in its entirety and its pages A(:,:,k) occupy separate files on disk (EDIT: and I do not have the means to read in strict subchunks of a page). Is there an algorithm, and ideally also a Matlab implementation somewhere, that will compute B incrementally by looping over successive pages A(:,:,k), or chunks of pages?
댓글 수: 4
Cris LaPierre
2021년 1월 25일
Looking as well. I see that reshape is supported for tall arrays, as is cat. How do you envision loading your data? Could you load each page as a tall array and use cat to turn it into a 3D array?
답변 (2개)
Gaurav Garg
2021년 1월 28일
Hi Matt,
You can compute the median of each column by converting the column into tall column and then calculating its median. You can repeat the step for each column (in your case).
T=tall(A(:,1))
m=median(T);
answer = gather(m);
Or, you can also convert the array into distributed array and then compute median (though the former solution might be more useful).
A=zeros(100000,3);
D = distributed(A);
e = median(D);
참고 항목
카테고리
Help Center 및 File Exchange에서 Large Files and Big Data에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!