Change objective function for hyperparameter optimization in regression ensembles

Hi

I want to change the objective function which is minimized during hyperparameter optimization for regression ensembles. The objective function is by default: log(1 + mse). Is it possible to change it to just Mean Absolute Error (MAE)? This value is more intuitive i think.

If possible, how should I call fitrensemble() with 'mae' as objective function?

BR /Tobias Pahlberg

Answer 1

[Don Mathis]

'fitrensemble' doesn't support MAE, but you can use 'TreeBagger' to do what you want. TreeBagger fits a random forest and supports quantile prediction, which can be used to compute MAE. Using MAE doesn't affect how the tree is built, but you can still optimize hyperparameters with MAE as the objective.

TreeBagger doesn't have built-in hyperparameter optimization so you need to use the 'bayesopt' function. Here's a script that does it. Paste it into an editor to run it. Note that it contains a local function definition.

% load a dataset
load carsmall
X = table(Acceleration,Cylinders,Displacement,Horsepower,Mfg,Model_Year,Weight,MPG);

% Define some variables to optimize
Hyperparameters = [optimizableVariable('MinLeafSize',[1,20], 'Type','integer');
                   optimizableVariable('NumPredictorstoSample', [1,width(X)-1], 'Type','integer')];

% Define the objective function to be the out-of-bag MAE. 
fun = @(hparams)oobMAE(hparams,X);          % It must be a function of only the hyperparameters.

% Call bayesopt to optimize MAE
results = bayesopt(fun, Hyperparameters);

% Look at the best hyperparameters found
bestHyperparameters = bestPoint(results)

% Fit a "final" model to the full dataset using the best hyperparameters
FinalModel = TreeBagger(300,X,'MPG','Method','regression',...
                        'MinLeafSize', bestHyperparameters.MinLeafSize,...
                        'NumPredictorstoSample', bestHyperparameters.NumPredictorstoSample)

% Define a function to fit a random forest of 300 trees and return out-of-bag MAE
function MAE = oobMAE(params,X)
randomForest = TreeBagger(300,X,'MPG','Method','regression',...
                          'OOBPrediction','on',...
                          'MinLeafSize', params.MinLeafSize,...
                          'NumPredictorstoSample', params.NumPredictorstoSample);
MAE = oobQuantileError(randomForest);   % Uses the median by default.
end