Visualize Features of a Convolutional Neural Network

This example shows how to visualize the features learned by convolutional neural networks.

Convolutional neural networks use features to classify images. The network learns these features itself during the training process. What the network learns during training is sometimes unclear. However, you can use the deepDreamImage function to visualize the features learned.

The convolutional layers output a 3D activation volume, where slices along the third dimension correspond to a single filter applied to the layer input. The channels output by fully connected layers at the end of the network correspond to high-level combinations of the features learned by earlier layers.

You can visualize what the learned features look like by using deepDreamImage to generate images that strongly activate a particular channel of the network layers.

The example requires Deep Learning Toolbox™ and Deep Learning Toolbox Model for GoogLeNet Network support package.

Load Pretrained Network

Load a pretrained GoogLeNet network and the corresponding class names. This requires the Deep Learning Toolbox™ Model for GoogLeNet Network support package. If this support package is not installed, then the software provides a download link. For a list of all available networks, see Pretrained Deep Neural Networks.

[net,classNames] = imagePretrainedNetwork("googlenet");

Visualize Early Convolutional Layers

There are multiple convolutional layers in the GoogLeNet network. The convolutional layers towards the beginning of the network have a small receptive field size and learn small, low-level features. The layers towards the end of the network have larger receptive field sizes and learn larger features.

Using analyzeNetwork, view the network architecture and locate the convolutional layers.

analyzeNetwork(net)

Features on Convolutional Layer 1

Set layer to be the first convolutional layer. This layer is the second layer in the network and is named 'conv1-7x7_s2'.

layer = 2;
name = net.Layers(layer).Name

name = 
'conv1-7x7_s2'

Visualize the first 36 features learned by this layer using deepDreamImage by setting channels to be the vector of indices 1:36. Set 'PyramidLevels' to 1 so that the images are not scaled. To display the images together, you can use imtile.

deepDreamImage uses a compatible GPU, by default, if available. Otherwise it uses the CPU. Using a GPU requires Parallel Computing Toolbox™ and a supported GPU device. For information on supported devices, see GPU Computing Requirements (Parallel Computing Toolbox).

channels = 1:36;
I = deepDreamImage(net,name,channels, ...
    'PyramidLevels',1);

|==============================================|
|  Iteration  |  Activation  |  Pyramid Level  |
|             |   Strength   |                 |
|==============================================|
|           1 |        15.84 |               1 |
|           2 |       423.25 |               1 |
|           3 |       849.50 |               1 |
|           4 |      1275.84 |               1 |
|           5 |      1702.21 |               1 |
|           6 |      2128.59 |               1 |
|           7 |      2554.97 |               1 |
|           8 |      2981.36 |               1 |
|           9 |      3407.74 |               1 |
|          10 |      3834.13 |               1 |
|==============================================|
Training finished: Max epochs completed.

figure
I = imtile(I,'ThumbnailSize',[64 64]);
imshow(I)
title(['Layer ',name,' Features'],'Interpreter','none')

These images mostly contain edges and colors, which indicates that the filters at layer 'conv1-7x7_s2' are edge detectors and color filters.

Features on Convolutional Layer 2

The second convolutional layer is named 'conv2-3x3_reduce', which corresponds to layer 6. Visualize the first 36 features learned by this layer by setting channels to be the vector of indices 1:36.

To suppress detailed output on the optimization process, set 'Verbose' to 'false' in the call to deepDreamImage.

layer = 6;
name = net.Layers(layer).Name

name = 
'conv2-3x3_reduce'

channels = 1:36;
I = deepDreamImage(net,name,channels, ...
    'Verbose',false, ...
    'PyramidLevels',1, ...
    'OutputScaling',"clipped");
figure
I = imtile(I,'ThumbnailSize',[64 64]);
imshow(I)
name = net.Layers(layer).Name;
title(['Layer ',name,' Features'],'Interpreter','none')

Filters for this layer detect more complex patterns than the first convolutional layer.

Visualize Deeper Convolutional Layers

The deeper layers learn high-level combinations of features learned by the earlier layers.

Increasing the number of pyramid levels and iterations per pyramid level can produce more detailed images at the expense of additional computation. You can increase the number of iterations using the 'NumIterations' option and increase the number of pyramid levels using the 'PyramidLevels' option.

layer = 97;
name = net.Layers(layer).Name

name = 
'inception_4e-1x1'

channels = 1:4;
I = deepDreamImage(net,name,channels, ...
    'Verbose',false, ...
    "NumIterations",20, ...
    'PyramidLevels',2, ...
    'OutputScaling', "clipped");
figure
I = imtile(I,'ThumbnailSize',[224 224]);
imshow(I)
name = net.Layers(layer).Name;
title(['Layer ',name,' Features'],'Interpreter','none')

Notice that the layers which are deeper into the network yield more detailed filters which have learned complex patterns and textures.

Visualize Fully Connected Layer

To produce images that resemble each class the most closely, select the fully connected layer, and set channels to be the indices of the classes.

Select the fully connected layer (layer 142).

layer = 142;
name = net.Layers(layer).Name

name = 
'loss3-classifier'

Select the classes you want to visualize by setting channels to be the indices of those class names.

channels = [114 293 563 950];

The classes are stored in the Classes property of the output layer (the last layer). You can view the names of the selected classes by selecting the entries in channels.

classNames(channels)

ans = 4×1 string
    "snail"
    "tiger"
    "fountain"
    "strawberry"

Generate detailed images that strongly activate these classes. Set 'NumIterations' to 100 in the call to deepDreamImage to produce more detailed images. The images generated from the fully connected layer correspond to the image classes.

I = deepDreamImage(net,name,channels, ...
    'Verbose',false, ...
    'NumIterations',100, ...
    'PyramidLevels',2, ...
    'OutputScaling', "clipped");
figure
I = imtile(I,'ThumbnailSize',[224 224]);
imshow(I)
name = net.Layers(layer).Name;
title(['Layer ',name,' Features'])

The images generated strongly activate the selected classes. The image generated for the ‘zebra’ class contain distinct zebra stripes, whilst the image generated for the ‘castle’ class contains turrets and windows.