regression learner running long time over 3 weeks now

Hi,

I have two 2019 matlab instances are running. Both running a regression learner from Machine learning and deep learning toolbox, one is running Squered Exponention Gauss Process Regression 650k rows 30 features, the other Rational Quadratic GPR 250k rows with 10 features. Both running over 3 weeks now on the same laptop 16GB RAM, Processor Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz, 2601 Mhz, 2 Core(s), 4 Logical Processor(s)

For the first two weeks the CPU ran 100%, and for a half a week for about 60% which dropped down to 0.5 to 2% on both app. The training indicator bar is still moving on both, memory usages are still changing. The only thing I dont like is that both matlab command window is busy, in one i entered 1+1 days ago. but all window buttons etc are responsive saying busy at the bottom.

I ran squered exponential gpr on 250k rows and 10 features 1.1 days before, stating 140obj/sec.

Do you think its normal that cpu dropps to 0,5-2 % but it is still calculating for days, untill I get the model?

Do you think I j ust need to be patient until it finishes? What is your opinion?

I appreciate your advice if you have had similar experience.

Thank you

[Rakshat]

I have been facing the same issue where I have been running rational gpr model with 35k rows with 11 features in a laptop of 16gb ram , i7-10870h processor , rtx 3060 graphics card and my model is still training after 48 hours . Please help

Answer 1

[Shivansh]

Hi Gabor!

It seems like you are stuck in a performance bottleneck in your Gaussian Process Regression (GPR) models in MATLAB. The size of datasets along with the complexity of the models can lead to longer training duration but the time taken in your cause seems a bit excessive.

A few potential reasons for this behavior can be:

1. It is possible that your system is running out of RAM and resorting to swapping memory to disk, which is significantly slower. This could also explain the drop in CPU usage, as the bottleneck shifts from processing power to disk I/O speed. You can try methods like learning rate scaling to speed up the training or consider training on a machine with higher RAM.
2. GPR can be computationally intensive with large datasets and certain kernel functions. You can try using a less complex model or decreasing the size of the dataset using feature selection techniques like PCA.

You can also consider moving to the latest version of MATLAB as it may contain optimizations and algorithmic improvements that could significantly reduce computation time. MATLAB's parallel processing features can also be utilized for leveraging parallel processing in training GPR-based models.

I don't think patience might be the solution here as three weeks and this drop in CPU utilization suggests some issue with the process. You can implement the above suggestions and share more details about the problem if the issue persists. The situation can get more clearer after looking at the problem statement and the training environment.

You can refer to the following documentation to learn more about parallel processing toolbox: https://www.mathworks.com/help/parallel-computing/index.html.

I Hope it helps!