FeatureSelectionNCARegression

Feature selection for regression using neighborhood component analysis (NCA)

Description

FeatureSelectionNCARegression contains the data, fitting information, feature weights, and other model parameters of a neighborhood component analysis (NCA) model. fsrnca learns the feature weights using a diagonal adaptation of NCA and returns an instance of FeatureSelectionNCARegression object. The function achieves feature selection by regularizing the feature weights.

Creation

Create a FeatureSelectionNCARegression object using fsrnca.

Properties

NCA Properties

ModelParameters — Model parameters

structure

This property is read-only.

Model parameters used for training the model, specified as a structure.

You can access the fields of ModelParameters using dot notation.

For example, for a FeatureSelectionNCARegression object named mdl, you can access the LossFunction value using mdl.ModelParameters.LossFunction.

Data Types: struct

Lambda — Regularization parameter

scalar

This property is read-only.

Regularization parameter used for training this model, specified as a scalar. For n observations, the best Lambda value that minimizes the generalization error of the NCA model is expected to be a multiple of 1/n.

Data Types: double

FitMethod — Name of the fitting method used to fit this model

'exact' | 'none' | 'average'

This property is read-only.

Name of the fitting method used to fit this model, specified as one of the following:

• 'exact' — Perform fitting using all of the data.

• 'none' — No fitting. Use this option to evaluate the generalization error of the NCA model using the initial feature weights supplied in the call to fsrnca.

• 'average' — The software divides the data into partitions (subsets), fits each partition using the exact method, and returns the average of the feature weights. You can specify the number of partitions using the NumPartitions name-value argument.

Solver — Name of the solver used to fit this model

'lbfgs' | 'sgd' | 'minibatch-lbfgs'

This property is read-only.

Name of the solver used to fit this model, specified as one of the following:

• 'lbfgs' — Limited memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) algorithm

• 'sgd' — Stochastic gradient descent (SGD) algorithm

• 'minibatch-lbfgs' — stochastic gradient descent with LBFGS algorithm applied to mini-batches

GradientTolerance — Relative convergence tolerance on gradient norm

positive scalar

This property is read-only.

Relative convergence tolerance on the gradient norm for the 'lbfgs' and 'minibatch-lbfgs' solvers, specified as a positive scalar value.

Data Types: double

IterationLimit — Maximum number of iterations for optimization

positive integer

This property is read-only.

Maximum number of iterations for optimization, specified as a positive integer value.

Data Types: double

PassLimit — Maximum number of passes

positive integer

This property is read-only.

Maximum number of passes for 'sgd' and 'minibatch-lbfgs' solvers, specified as a positive integer. Every pass processes all of the observations in the data.

Data Types: double

InitialLearningRate — Initial learning rate

positive real scalar

This property is read-only.

Initial learning rate for 'sgd' and 'minibatch-lbfgs' solvers, specified as a positive real scalar. The learning rate decays over iterations starting at the value specified for InitialLearningRate.

Use the NumTuningIterations and TuningSubsetSize to control the automatic tuning of initial learning rate in the call to fsrnca.

Data Types: double

Verbose — Verbosity level indicator

nonnegative integer

This property is read-only.

Verbosity level indicator, specified as a nonnegative integer. Possible values are:

• 0 — No convergence summary

• 1 — Convergence summary, including norm of gradient and objective function value

• >1 — More convergence information, depending on the fitting algorithm. When you use the 'minibatch-lbfgs' solver and verbosity level > 1, the convergence information includes the iteration log from intermediate mini-batch LBFGS fits.

Data Types: double

InitialFeatureWeights — Initial feature weights

p-by-1 vector of positive real scalars

This property is read-only.

Initial feature weights, specified as a p-by-1 vector of positive real scalars, where p is the number of predictors in X. For more information about feature weights, see Neighborhood Component Analysis (NCA) Feature Selection.

Data Types: double

FeatureWeights — Feature weights

numeric vector | numeric matrix

This property is read-only.

Feature weights, specified as a p-by-1 numeric vector or a p-by-m numeric matrix, where p is the number of predictor variables after dummy variables are created for categorical variables (for more details, see ExpandedPredictorNames).

If FitMethod is 'average', then FeatureWeights is a p-by-m matrix. m is the number of partitions specified via the NumPartitions name-value argument in the call to fsrnca.

The absolute value of FeatureWeights(k) is a measure of the importance of predictor k. A FeatureWeights(k) value that is close to 0 indicates that predictor k does not influence the response in Y. For more information about feature weights, see Neighborhood Component Analysis (NCA) Feature Selection.

Data Types: double

FitInfo — Fit information

structure

This property is read-only.

Fit information, specified as a structure with the following fields.

Field Name	Meaning
Iteration	Iteration index
Objective	Regularized objective function for minimization
UnregularizedObjective	Unregularized objective function for minimization
Gradient	Gradient of regularized objective function for minimization

• For classification, UnregularizedObjective represents the negative of the leave-one-out accuracy of the NCA classifier on the training data.

• For regression, UnregularizedObjective represents the leave-one-out loss between the true response and the predicted response when using the NCA regression model.

• For the 'lbfgs' solver, Gradient is the final gradient. For the 'sgd' and 'minibatch-lbfgs' solvers, Gradient is the final mini-batch gradient.

• If FitMethod is 'average', then FitInfo is an m-by-1 structure array, where m is the number of partitions specified via the NumPartitions name-value argument.

You can access the fields of FitInfo using dot notation. For example, for a FeatureSelectionNCARegressionobject named mdl, you can access the Objective field using mdl.FitInfo.Objective.

Data Types: struct

Other Regression Properties

NumObservations — Number of observations in the training data

scalar

This property is read-only.

Number of observations in the training data (X and Y) after removing NaN or Inf values, specified as a scalar.

Data Types: double

Mu — Predictor means

p-by-1 vector | []

This property is read-only.

Predictor means, specified as a p-by-1 vector for standardized training data. In this case, the predict method centers predictor matrix X by subtracting the respective element of Mu from every column.

If data is not standardized during training, then Mu is empty.

Data Types: double

Sigma — Predictor standard deviations

p-by-1 vector | []

This property is read-only.

Predictor standard deviations, specified as a p-by-1 vector for standardized training data. In this case, the predict method scales predictor matrix X by dividing every column by the respective element of Sigma after centering the data using Mu.

If data is not standardized during training, then Sigma is empty.

Data Types: double

X — Predictor values

matrix | table

This property is read-only.

Predictor values used to train this model, specified as a matrix or a table. Each column of X represents one predictor (variable), and each row represents one observation.

Data Types: single | double | table

Y — Response values

numeric vector of size n

This property is read-only.

Response values used to train this model, specified as a numeric vector of size n, where n is the number of observations.

Data Types: double

W — Observation weights

numeric vector of size n

This property is read-only.

Observation weights used to train this model, specified as a numeric vector of size n. The sum of observation weights is n.

Data Types: double

CategoricalPredictors — Categorical predictor indices

vector of positive integers | []

This property is read-only.

Categorical predictor indices, specified as a vector of positive integers. CategoricalPredictors contains index values indicating that the corresponding predictors are categorical. The index values are between 1 and p, where p is the number of predictors used to train the model. If none of the predictors are categorical, then this property is empty ([]).

Data Types: single | double

ResponseName — Response variable name

character vector

This property is read-only.

Response variable name, specified as a character vector.

Data Types: char

PredictorNames — Predictor variable names

cell array of unique character vectors

This property is read-only.

Predictor variable names in order of their appearance in the predictor data, specified as a cell array of unique character vectors. The length of PredictorNames is equal to the number of variables in the training data X used as predictor variables.

Data Types: cell

ExpandedPredictorNames — Expanded predictor names

cell array of unique character vectors

This property is read-only.

Expanded predictor names, specified as a cell array of unique character vectors.

If the model uses encoding for categorical variables, then ExpandedPredictorNames includes the names that describe the expanded variables. Otherwise, ExpandedPredictorNames is the same as PredictorNames.

Data Types: cell

Object Functions

loss	Evaluate accuracy of learned feature weights on test data
predict	Predict responses using neighborhood component analysis (NCA) regression model
refit	Refit neighborhood component analysis (NCA) model for regression
selectFeatures	Select important features for NCA classification or regression

Examples

Explore FeatureSelectionNCARegression Object

Load the sample data.

load imports-85

The first 15 columns contain the continuous predictor variables, whereas the 16th column contains the response variable, which is the price of a car. Define the variables for the neighborhood component analysis model.

Predictors = X(:,1:15);
Y = X(:,16);

Fit a neighborhood component analysis (NCA) model for regression to detect the relevant features.

mdl = fsrnca(Predictors,Y);

The returned NCA model, mdl, is a FeatureSelectionNCARegression object. This object stores information about the training data, model, and optimization. You can access the object properties, such as the feature weights, using dot notation.

Plot the feature weights.

plot(mdl.FeatureWeights,"o")
xlabel("Feature Index")
ylabel("Feature Weight")
grid on

The weights of the irrelevant features are zero. The Verbose=1 option in the call to fsrnca displays the optimization information on the command line. You can also visualize the optimization process by plotting the objective function versus the iteration number.

plot(mdl.FitInfo.Iteration,mdl.FitInfo.Objective,"o-")
grid on
xlabel("Iteration Number")
ylabel("Objective")

The ModelParameters property is a struct that contains more information about the model. You can access the fields of this property using dot notation. For example, see if the data was standardized or not.

mdl.ModelParameters.Standardize

ans = logical
   0

0 means that the data was not standardized before fitting the NCA model. You can standardize the predictors when they are on very different scales using the Standardize=true name-value argument in the call to fsrnca.

Version History

Introduced in R2016b