reluLayer

Rectified Linear Unit (ReLU) layer

Description

A ReLU layer performs a threshold operation to each element of the input, where any value less than zero is set to zero.

This operation is equivalent to

$$f\left(x\right)=\{\begin{array}{cc}x,& x\ge 0\\ 0,& x<0\end{array}.$$

Creation

Syntax

layer = reluLayer

layer = reluLayer('Name',Name)

Description

layer = reluLayer creates a ReLU layer.

layer = reluLayer('Name',Name) creates a ReLU layer and sets the optional Name property using a name-value pair. For example, reluLayer('Name','relu1') creates a ReLU layer with the name 'relu1'.

Properties

Name — Layer name

"" (default) | character vector | string scalar

Layer name, specified as a character vector or string scalar. For Layer array input, the trainnet and dlnetwork functions automatically assign names to layers with the name "".

The ReLULayer object stores this property as a character vector.

Data Types: char | string

NumInputs — Number of inputs

1 (default)

This property is read-only.

Number of inputs to the layer, returned as 1. This layer accepts a single input only.

Data Types: double

InputNames — Input names

{'in'} (default)

This property is read-only.

Input names, returned as {'in'}. This layer accepts a single input only.

Data Types: cell

NumOutputs — Number of outputs

1 (default)

This property is read-only.

Number of outputs from the layer, returned as 1. This layer has a single output only.

Data Types: double

OutputNames — Output names

{'out'} (default)

This property is read-only.

Output names, returned as {'out'}. This layer has a single output only.

Data Types: cell

Examples

Create ReLU Layer

Create a ReLU layer with the name relu1.

layer = reluLayer(Name="relu1")

layer = 
  ReLULayer with properties:

    Name: 'relu1'

Include a ReLU layer in a Layer array.

layers = [ ...
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    reluLayer
    maxPooling2dLayer(2,Stride=2)
    fullyConnectedLayer(10)
    softmaxLayer]

layers = 
  6x1 Layer array with layers:

     1   ''   Image Input       28x28x1 images with 'zerocenter' normalization
     2   ''   2-D Convolution   20 5x5 convolutions with stride [1  1] and padding [0  0  0  0]
     3   ''   ReLU              ReLU
     4   ''   2-D Max Pooling   2x2 max pooling with stride [2  2] and padding [0  0  0  0]
     5   ''   Fully Connected   10 fully connected layer
     6   ''   Softmax           softmax

More About

ReLU Layer

A ReLU layer performs a threshold operation to each element of the input, where any value less than zero is set to zero.

Convolutional and batch normalization layers are usually followed by a nonlinear activation function such as a rectified linear unit (ReLU), specified by a ReLU layer. A ReLU layer performs a threshold operation to each element, where any input value less than zero is set to zero, that is,

$$f\left(x\right)=\{\begin{array}{cc}x,& x\ge 0\\ 0,& x<0\end{array}.$$

The ReLU layer does not change the size of its input.

There are other nonlinear activation layers that perform different operations and can improve the network accuracy for some applications. For a list of activation layers, see Activation Layers.

References

[1] Nair, Vinod, and Geoffrey E. Hinton. "Rectified linear units improve restricted boltzmann machines." In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814. 2010.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.

GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Version History

Introduced in R2016a