Parfeval memory leak? Clear memory on parallel workers?
조회 수: 10 (최근 30일)
이전 댓글 표시
Hi,
I have some sort of memory leak when using parfeval.
I have a function that starts a number of parfeval-computations and then uses fetchNext in a loop to retrieve all results.
The memory taken up by the Matlab instances in my parallel pool grows with each execution of the function that uses parfeval.
My question: Am I doing something wrong? Do I have to explicitly clear the memory of the workers after using parfeval? How do I do this? I tried "pctRunOnAll clear all" but it did not free up any memory on the workers (just the client instance).
Right now, the only way I can clear this memory is to shut down and re-start my parallel pool, but that's not a feasible solution for my application.
Thanks, Matthias
댓글 수: 3
Atakan Atay
2020년 3월 25일
편집: Atakan Atay
2020년 3월 25일
Hi Matthias,
Currently I experience the very same problem. How did you manage to solve it? I would really appreciate your help.
Thanks for your time.
Francis Plamondon
2020년 11월 18일
Hello,
I also recently struggled with parfeval. I would get 'out of memory' errors after several hours of running a huge batch of jobs.
Here is what I found out: each parallel.FevalFuture object holds a Diary property that contains everything that was displayed in the console window for each job. Since I had a function use disp() for monitoring purpose, it was being recorded in the Diary variable, eventually filling all the memory.
After commenting out the disp function, the memory stopped slowly increasing.
채택된 답변
Edric Ellis
2014년 12월 15일
Please could you try applying the workaround from this bug report. (The bug report mentions mapreduce, but it includes a fix that affects only parfeval).
댓글 수: 1
Seth
2015년 2월 13일
I have a similar problem and tried using this patch and now get errors indicating "no method 'createInstance' with matching signatures found for class 'com.mathworks.toolboox.distcomp.objects.EntityFileProxy'
Do you have any other suggestions? I was running two workers with PARFEVAL and seeing 3gb increase in RAM used each call to them. (The PARFEVAL commands are in a loop)
Thanks.
추가 답변 (0개)
참고 항목
카테고리
Help Center 및 File Exchange에서 Startup and Shutdown에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!