This example shows how to emulate the behavior of a quantized network for GPU deployment. Once you quantize your network for a GPU execution environment, you can emulate the GPU target behavior without the GPU hardware. Doing so allows you to examine your quantized network structure and behavior without generating code for deployment.
Emulated quantized networks are not smaller than the original network.
Load the pretrained network. net is a SqueezeNet convolutional neural network that has been retrained using transfer learning to classify images in the MerchData data set.
net =
dlnetwork with properties:
Layers: [67×1 nnet.cnn.layer.Layer]
Connections: [74×2 table]
Learnables: [52×3 table]
State: [0×3 table]
InputNames: {'data'}
OutputNames: {'prob'}
Initialized: 1
View summary with summary.
Define calibration and validation data to use for quantization.
Use the calibration data to collect the dynamic ranges of the weights and biases in the convolution and fully connected layers of the network and the dynamic ranges of the activations in all layers of the network. For the best quantization results, the calibration data must be representative of inputs to the network.
Use the validation data to test the network after quantization to understand the effects of the limited range and precision of the quantized convolution layers in the network.
For this example, use the images in the MerchData data set. Split the data into calibration and validation data sets.
Create a dlquantizer object and specify the network to quantize. How the network is quantized depends on the execution environment. Set ExecutionEnvironment to GPU to perform quantization specific to GPU target hardware.
Use the calibrate function to exercise the network object with sample inputs and collect range information.
Use the quantize method to quantize the network object and return a simulatable quantized network.
qNet =
Quantized dlnetwork with properties:
Layers: [67×1 nnet.cnn.layer.Layer]
Connections: [74×2 table]
Learnables: [52×3 table]
State: [0×3 table]
InputNames: {'data'}
OutputNames: {'prob'}
Initialized: 1
View summary with summary.
Use the quantizationDetails function to extract quantization details.
You can use the quantizationDetails method to see that the network is now quantized.
qDetails = struct with fields:
IsQuantized: 1
TargetLibrary: "cudnn"
QuantizedLayerNames: [55×1 string]
QuantizedLearnables: [35×3 table]
The TargetLibrary field shows that the quantized network emulates the CUDA® Deep Neural Network library (cuDNN).
The QuantizedLayerNames field displays a list of layers that have been quantized.
ans = 5×1 string
"conv1"
"relu_conv1"
"pool1"
"fire2-squeeze1x1"
"fire2-relu_squeeze1x1"
The QuantizedLearnables field contains additional details on quantized network learnable parameters. In this example, the 2-D convolutional layer, conv1, has had the weights scaled and cast to int8. The bias is scaled and remains in single precision. The values of quantized learnables are returned as stored integer values.
ans=35×3 table
Layer Parameter Value
__________________ _________ ___________________
"conv1" "Weights" {3×3×3×64 int8 }
"conv1" "Bias" {1×1×64 single}
"fire2-squeeze1x1" "Weights" {1×1×64×16 int8 }
"fire2-squeeze1x1" "Bias" {1×1×16 single}
"fire2-expand1x1" "Weights" {1×1×16×64 int8 }
"fire2-expand3x3" "Weights" {3×3×16×64 int8 }
"fire3-squeeze1x1" "Weights" {1×1×128×16 int8 }
"fire3-squeeze1x1" "Bias" {1×1×16 single}
"fire3-expand1x1" "Weights" {1×1×16×64 int8 }
"fire3-expand3x3" "Weights" {3×3×16×64 int8 }
"fire4-squeeze1x1" "Weights" {1×1×128×32 int8 }
"fire4-squeeze1x1" "Bias" {1×1×32 single}
"fire4-expand1x1" "Weights" {1×1×32×128 int8 }
"fire4-expand3x3" "Weights" {3×3×32×128 int8 }
"fire5-squeeze1x1" "Weights" {1×1×256×32 int8 }
"fire5-squeeze1x1" "Bias" {1×1×32 single}
⋮
You can use the quantized network to emulate how a network quantized for GPU target hardware would perform a classification task.
Make predictions using the original, single-precision floating-point network. To accelerate the computation by compiling and executing a MEX function on the GPU, use the acceleration option "mex" of the predict function.
Generating MEX for cudnn target.
Make predictions using the quantized INT8 network. Use the acceleration option "mex" of the predict function. MEX acceleration is supported for quantized networks based on quantization objects with ExecutionEnvironment set to GPU.
Generating MEX for cudnn target.
Compute the relative accuracy of the quantized network as compared to the original network.
The quantized network shows no drop in accuracy.
