Characterizing Parameter Scaling with Quantization for Deployment of CNNs on Real-Time Systems

2.1 Neural-Network Performance Metrics

The goal behind neural-network metrics is to capture some of the complexity of a model to understand its inference performance on a particular hardware system. Two common metrics, parameters and FLOPs, are described. The background on these metrics is followed by Critical Datapath Length (CDL), which is a recently developed metric for understanding model performance.

2.2 Neural Architecture Search

Designing neural networks that perform well on resource-constrained devices has become a research topic of interest in the machine-learning and computer-vision fields. Recently, there has been a significant amount of interest in NAS, where a neural-network architecture is evolved algorithmically to adapt to specific constraints. Within NAS, there is research interest in creating networks optimized for specific devices. Cai et al. perform a proxyless NAS by generating an over-parameterized network and then conducting path-level pruning to create a network that is efficient for a given platform [3]. This method is performed while avoiding simpler tasks that do not relate to a given network, dataset, or target platform. Cai et al. demonstrate NAS by choosing a sub-network from a supernet for deployment on a given device [2]. They do this operation by training the supernet and then fine-tuning subsets of the supernet. With this method, they were able to achieve a new state of the art on ImageNet. Another method for pruning is ResRep [7]. This method uses structural pruning to remove entire filters from CNN layers to achieve a minimal loss in accuracy for a target FLOPs reduction. The authors of ResRep claim to achieve near lossless accuracy performance with a ResNet-50 model that has only 45% of the FLOPs of the original model.

2.3 NeuralScale

Another NAS-based reduction method called NeuralScale was proposed by Lee and Lee [18]. With NeuralScale, only the width component of the network is scaled. NeuralScale attempts to select the best number of filters for each individual layer at a given scaling factor rather than scaling uniformly across layers. The authors showed accuracy improvements over other scaling methods [18].

To target a specific number of parameters, NeuralScale first performs an iterative pruning method to determine the importance of each filter in the network [18]. The importance is defined as a measurement of the increase in error caused by the removal of that filter. When removing a filter causes a large decrease in accuracy, the filter is considered to be more important. This metric of importance is explained in the research of Molchanov et al. [21].

As the pruning method is performed across many iterations, the change in the number of filters per layer compared with the number of parameters in the network is learned. With this knowledge, a model of the number of filters in a layer given the total number of parameters can be generated. Essentially, a curve is fit which shows how the parameter count changes as the number of filters is increased for each layer. Using the learned curve, stochastic gradient descent is then used to refine the model until the target parameter count is reached. An in-depth, mathematical explanation can be found in the NeuralScale article [18]. The steps of iteratively pruning the model, searching for the parameters for the curve, and then generating the network with stochastic gradient descent represent one iteration of what the authors define as architecture descent. Architecture descent can then be run iteratively for a set number of steps or until convergence.

After completing several iterations of architecture descent, the method requires a scaling ratio to be set. The filter layout for the model is then generated using the previously learned curve. The target parameter count is set so that the number of parameters in the generated model is similar to the number of parameters that would exist in the original model when each layer is uniformly multiplied by the scaling ratio. The final step is to train the newly scaled model architecture on the target dataset. The entire NeuralScale process is depicted in Figure 1.

Fig. 1.

View Figure

2.4 MobileNets and Uniform Scaling

MobileNets are a group of networks aimed at making an efficient architecture for mobile devices [12]. MobileNetV1 and MobileNetV2 use a depthwise-separable convolution to reduce the computational complexity of the models [12, 32]. Hyperparameters of the network include the width multiplier and image resolution. The effect of the width multiplier on the network is the same as the effect of the scaling ratio with uniform scaling in this research. The individual number of filters in each layer are all multiplied, uniformly, by the scaling ratio to shrink the model to target fewer parameters and make it more amenable to resource-constrained systems. Figure 1 highlights the main steps in uniform scaling.

2.5 Quantization for Improved Inference

Modern computer-vision models tend to be over-parameterized [6], meaning they are a good candidate for quantization. Through quantization, the precision of a model can be lowered from standard 32-bit floating-point (float32) values to half-precision floating point (float16), 8-bit integer (int8), or even binary. Quantization can improve the runtime performance of models on target platforms [10]. Additionally, some hardware accelerators like Xilinx’s DPU run only int8 models [37]. In our research, int8 quantization is used as it is supported by the tools of all tested platforms.

Quantization can take many forms. Of interest to this research is post-training quantization (PTQ). PTQ has exceptionally low engineering overhead as models are not retrained or fine-tuned. Part of the training dataset may be used to determine scaling values for the activations in the model. This process is known as calibration. If PTQ does not achieve satisfactory performance, further methods like quantization-aware training (QAT) can be used to fine-tune the model on the training dataset [10].

2.6 Real-Time Computer Vision

There are many cases where a computer-vision application would need to be performant on an embedded system. One of the most relevant examples is the vision processing system for a self-driving car. Consider a car moving at 80 miles per hour, roughly 117 feet per second. If an obstruction suddenly enters the roadway 20 feet ahead of the vehicle, an object detector with 100 milliseconds of processing time will produce its first result when the car has already traversed half of the distance to the object in question. Therefore, it is critically important that low latency is achievable on embedded devices. Furthermore, the high-resolution vision necessary for detecting small obstructions will be even slower due to the added computational cost from the increase in the number of pixels and limited memory bandwidth on the embedded devices.

For this research, we consider a real-time latency constraint of \({16.67}\) ms, or a frame rate of 60 FPS. Many state-of-the-art computer-vision apps target this rate even with more complex goals than classification [14, 20]; it follows that simple classification models should also be able to reach this real-time goal. Ideally, models will only use a fraction of this time as often the processing system needs to perform other operations with this time budget. Therefore, achieving smaller latencies is prudent as less of the time budget spent on the model enables more of the budget to be used for other important applications in the pipeline.

3.1 Test Platforms

Details about the three embedded device platforms are included below. An overview of the test platforms is shown in Table 1. Each device runs a separate inference framework designed for improved performance on its architecture. A brief overview of the three frameworks studied is also included.

3.2 Model Training and Setup

In this subsection, we summarize the training device, framework, and hyperparameters. First, we outline the method for training models on the CIFAR-10 dataset. Next, we detail the additional steps for training MobileNetV2 on tinyImageNet. Model training and achieving the highest accuracy is not the primary focus of our study. Instead, we consider the impact of model structure on device performance. Therefore, before using scaling to deploy a model, additional investigation into the best hyperparameters should be performed to improve model accuracy.

3.3 CPU Benchmarking

The first CPU-specific task is the generation of quantized ONNX models. For simplicity, quantization was performed on the same device as the original training. ONNX Runtime static PTQ [28] was used to convert the float32-based ONNX model into an int8 model. To perform quantization, 1,000 random training image samples were used as a calibration dataset. The ONNX Runtime quantizer was then run on the model, and a final accuracy on the test dataset was gathered.

A separate Docker container with ONNX Runtime version 1.12.1 was built for testing on the ARM Cortex-A72 device. The Raspberry Pi 4 default maximum clock rate of 1.5 GHz was used for testing. We gathered latency measurements by sending random data through the models with 20 cycles of warmup followed by 80 cycles of measured inferences. Memory measurements were gathered using the Python Memory Profiler [29]. This tool measures the amount of runtime memory by querying the OS to determine the amount of memory used. It also can report the specific amount of memory used by a specific line of Python code. The profiler was used to record the amount of memory needed for the ONNX Runtime InferenceSession generation. Both latency and memory measurements were averaged over 50 trials to better sample inference latency and runtime-memory utilization. All measurements were gathered using a batch size of 1 to represent a streaming app performing inference on 1 image at a time, a common use case on embedded systems.

3.4 GPU Benchmarking

NVIDIA JetPack version 5.1, which includes TensorRT version 8.5.2, was prepared on the NVIDIA AGX Xavier. The AGX Xavier includes several different power modes. For this testing, we configured the device for an approximate max power budget of 30 W, with two CPU cores enabled at a frequency of 2,100 MHz, a GPU frequency of 900 MHz, and a memory frequency of 1,600 MHz. This mode was chosen to enable the fastest clock frequencies for the GPU.

Using the TensorRT Python API, we converted the original float32 ONNX models into serialized TensorRT engines. We also enabled the conversion of the model using int8 operations by creating a calibration cache using 1,000 training images as described in the TensorRT documentation, which allows TensorRT to perform implicit quantization [23]. Note that TensorRT engine generation is non-deterministic due to the use of profiling to optimize kernel selection at the time of generation. The accuracy of both unquantized and quantized engines was measured using the test datasets through the Python TensorRT API.

Latency measurements were gathered on all serialized engines using NVIDIA’s trtexec program, details of which can be found in the TensorRT documentation [23]. We used this program to load the serialized engines generated through the Python API. This program then profiled the latency of inference using a batch size of 1 by sampling the performance for a total of 10 seconds with 2 seconds of warmup. Memory measurements were also gathered using trtexec. To estimate the runtime memory required, the TensorRT documentation recommends summing the size of the serialized engine, the persistent memory allocated, and the memory allocated to store the activations [22]. We gather the size of the serialized engine using the du -b Bash command. The size of persistent memory and activation memory are reported in the verbose mode of the trtexec program. Since all steps from engine generation to runtime performance can have variance, we performed this pipeline of engine generation and analysis 50 times to sample the overall latency and memory requirements of each model when using TensorRT for inference.

3.5 FPGA Benchmarking

The Xilinx-released Vitis-AI Docker container, version 3.0.0.106, was used for converting the PyTorch models into DPU models. This container was run on the same systems used for model training. The pipeline found in Xilinx’s sample code at [36, 39] was used for reference. The DPU only runs on quantized models, therefore no float32 results are gathered. We use 500 sample images from the training dataset to calibrate the quantizer. Then we export the model and set it to target the ZCU104.

A modified version of the app from Reference [36] was created to load the 10,000 test images in the test datasets. This app reports the latency of inference when using only 1 thread, essentially setting the batch size to 1 as the thread concurrency represents the number of inferences that can be performed in parallel on the DPU. Accuracy on the test dataset was recorded on the device. Again, the latency performance was averaged over 50 trials.

3.6 Power Benchmarks

Power results are reported in terms of the dynamic power to compare how power utilization varies across the models. Additionally, the results are reported as the throughput (images/sec) per watt of peak load power. This metric helps to identify the overall efficiency of inference on the devices. To avoid device-dependent frameworks, an external power meter was used to measure the total device power for the three devices under test. Specifically, we used the Poniie PN2000 Electricity Usage Monitor. Idle power measurements were gathered when no foreground processes were running on the devices. Then, we measured the power during the execution of each model and recorded the peak load power. The dynamic power consumed during the execution of each model can be found by subtracting the idle power from the peak load power measurements. We found these measurements to be precise enough to identify trends in power across scaling methods and scaling ratios.

4.1 Model Metrics on CIFAR-10

The maximum accuracy achieved on each model versus the number of parameters are displayed in Figures 2(a), 2(b), 2(c). Due to the wide range in the number of parameters in each model, each graph is displayed separately. Note that the marked points from left to right represent the four scaling ratios in order: \({0.25\times }\), \({0.50\times }\), \({0.75\times }\), and \({1.00\times }\). The goal of the NeuralScale method is to achieve a higher accuracy for the same number of parameters. This goal was achieved for VGG-11 and MobileNetV2 as the NeuralScale line is always above the uniform-scaling line. However, that is not the case for the ResNet-50 model tested. While likely a limitation of using the same hyperparameters for all models, it is important to note that NeuralScale alone is not guaranteed to generate a more accurate model than uniform scaling.

Fig. 2.

View Figure

Figures 2(d), 2(e), 2(f) show the accuracy compared with the number of FLOPs in each model. For VGG-11 and MobileNetV2, the two lines begin to overlap. When comparing the NeuralScale \({0.50\times }\) VGG-11 model to the original, equivalent to uniform-scaling \({1.00\times }\), the large reduction in parameters only leads to a \({0.9\times }\) increase in FLOPs with a near identical accuracy, only \(0.14\%\) different. For MobileNetV2, the NeuralScale \({0.25\times }\) may reduce the parameter count over the uniform-scaling \({0.75\times }\) model, but the NeuralScale model is actually \({0.10\%}\) less accurate with \({1.3\times }\) more FLOPs. With ResNet-50, all uniform-scaling models used far fewer FLOPs than their NeuralScale counterparts, often at significant accuracy improvements. Overall, these results indicate that simply scaling to target a number of parameters ignores the potential impact on the number of FLOPs in the model, which is influential to the overall accuracy of the models.

Figures 2(g), 2(h), 2(i) show the number of FLOPs in a model versus the number of parameters. By fitting a line to the number of FLOPs required for each parameter, we can see the rate at which the FLOPs increase relative to the parameter count. All lines have an \({R^{2} \gt 0.999}\). Compared with uniform scaling, the NeuralScale models increase the FLOPs per parameter at a rate of \({3.2\times }\) for VGG-11, \({10.5\times }\) for MobileNetV2, and \({9.8\times }\) for ResNet-50. This result indicates that NeuralScale learns to increase the number of FLOPs in a model for a target parameter count to try to achieve a more accurate model.

Finally, the CDL for each model and the FLOPs range for the models generated by uniform scaling and NeuralScale are displayed in Table 3. Parameter scaling does not affect the depth of the networks; therefore, all models with the same base have the same CDL. As noted in [16], CDL is a more meaningful metric for memory-bound devices such as GPUs, while FLOPs are more meaningful on compute-bound devices such as CPUs.

4.2 CPU Results

The CPU results are first presented with a comparison of scaling methods while varying the best model. Next, a comparison of scaling methods while varying the dataset is presented using the MobileNetV2 model. Finally, a summary is provided for the reader’s benefit. For the CPU device, the idle power was measured to be 3.1 W.

4.2.1 Comparing Scaling Methods with Three Models on CIFAR-10.

The inference results for the float32-based models on the ARM Cortex-A72 cores are shown in Figure 3. Across all three devices, the dashed teal lines represent the performance of the original models, while the dashed red line represents a real-time target of 60 FPS. Latency results for a batch size of 1 are shown in Figure 3(a). Note that the uniform-scaling models are always faster than their NeuralScale counterparts. On the CPU, many models struggle to meet real-time performance, represented by the dashed red line on the latency graph. For uniform scaling, one VGG-11 model, three MobileNetV2 models, and one ResNet-50 model achieve this constraint. However, with NeuralScale models, only the \({0.25\times }\) models achieve real-time latency. NeuralScale is designed for resource-constrained systems, but it does not optimize for latency, thus it is important to consider the impact of the scaling method toward reaching real-time performance.

Fig. 3.

View Figure

Also note that, as shown in Figure 3(b), the memory utilization between the two scaling types is always approximately equivalent. The memory consumption has a Pearson correlation coefficient of 0.999 (\({\it p}=1.6 \cdot 10^{-30}\)) with the number of parameters in the model. This result indicates that, by adjusting the number of parameters, it is possible to estimate the amount of memory that will be used by the device. However, given the larger number of FLOPs in the NeuralScale models, they tend to run more slowly on a less parallel device like a CPU.

In terms of power, uniform-scaling models always use less dynamic power than their NeuralScale counterparts, though not by large margins. Given the much faster inference speeds of the uniform-scaling models, throughput per watt of peak load power is significantly higher for uniform-scaling models over NeuralScale. Therefore, we conclude that uniform-scaling models are more efficient for processing on a CPU than their NeuralScale counterparts. For example, to achieve any improvement over the baseline model with MobileNetV2, the scaling ratio has to be set to \({0.25\times }\) for NeuralScale, while all ratios tested are more efficient for uniform scaling.

Figures 4(a), 4(b), 4(c) show the accuracy versus average latency performance on the three models tested. For VGG-11 and MobileNetV2, we see that the increases in accuracy come at the clear tradeoff of latency on the CPU device. As the uniform scaling and NeuralScale lines overlap, we can see how higher accuracies are only achieved with higher latencies. For the ResNet-50 model, the uniform scaling implementations are always more accurate, often at lower latencies. Figures 4(d), 4(e), 4(f) show the accuracy versus average memory utilization on the three models tested. Given that parameter count is highly correlated with memory utilization on the CPU, we do not see a tradeoff between memory and accuracy. On the CPU, NeuralScale models achieve higher accuracies for the same memory for VGG-11 and MobileNetV2, while uniform scaling achieves a higher accuracy for the same memory for ResNet-50.

Fig. 4.

View Figure

Next are the results for the int8-based models on the ARM Cortex-A72, shown in Figure 5. Again, similar trends are evident to those found in the float32 models. The uniform-scaling models always, and quite often significantly, outperform their NeuralScale counterparts in terms of inference latency. On average, the quantized NeuralScale models are \({3.3\times }\) slower than their uniform-scaled counterparts. Several more models now achieve real-time performance. For VGG-11, the three smallest uniform-scaling models are faster than 16.67 ms, while only one NeuralScale model is. For MobileNetV2, all uniform-scaling models meet the real-time constraint, while only the smallest NeuralScale model does. Finally, for ResNet-50, three uniform-scaling models are real-time, while only one NeuralScale model is.

Fig. 5.

View Figure

Memory utilization between the two scaling types again tends to be approximately the same. However, an unexpected result in terms of model accuracy occurred for the NeuralScale \({0.25\times }\) ResNet-50 model. Its uniform-scaling counterpart was able to quantize well, but the NeuralScale version of the model was not able to do so, with an accuracy drop of approximately 48% due to quantization. This drop is likely due to the change in model structure resulting from the NeuralScale process, discussed in Section 5.1.

Different from the float32 models, the int8 uniform-scaling VGG-11 models use more power than their NeuralScale counterparts. Uniform-scaling int8 MobileNetV2 and ResNet-50 models still use less power than NeuralScale versions, as shown in Figure 5(d). The uniform-scaling VGG-11 model can operate more efficiently on the hardware than NeuralScale, as shown by its significantly higher throughput per watt in Figure 5(e). Additionally, the other uniform-scaling models always remain significantly more efficient in terms of throughput per watt than their NeuralScale counterparts.

The accuracy versus latency charts shown in Figures 6(a), 6(b), 6(c) are similar to the trends shown on the float32 models. Again we see that accuracy gains for NeuralScale VGG-11 and MobileNetV2 models come at the cost of latency, while the uniform-scaling ResNet-50 models outperform their NeuralScale counterparts. In Figures 6(d), 6(e), 6(f), we see memory utilization again is highly correlated with parameter count, and therefore, the most accurate model should be used when targeting memory utilization.

Fig. 6.

View Figure

4.2.2 Comparing Scaling Methods with Two Datasets on MobileNetV2.

The results for CIFAR-10 and tinyImageNet with float32 MobileNetV2 models are shown in Figure 7. In terms of latency, NeuralScale models are again slower than their uniform-scaling counterparts. For CIFAR-10, NeuralScale models are \({3.6\times }\) slower, while for tinyImageNet, they are \({3.3\times }\) slower. The difference between the memory consumptions of NeuralScale and uniform scaling is similar and not significantly impacted by the dataset.

Fig. 7.

View Figure

Interestingly, in this study, the NeuralScale models are less accurate on tinyImageNet than the uniform-scaling models. This trend differs from the results of [18]. While it is likely caused by non-optimized hyperparameters, it does indicate that the NeuralScale method is potentially more sensitive to hyperparameter choices, making it more difficult to use than uniform scaling.

Finally, for the power measurements, there are no significant differences in dynamic power, though NeuralScale models do tend to use slightly more power than their uniform-scaling counterparts. Our results also indicate that dataset has essentially no effect on throughput per watt of peak load power. Uniform scaling always significantly outperforms NeuralScale.

Similar results were gathered for the quantized models, shown in Figure 8. NeuralScale models were \({3.9\times }\) and \({3.5\times }\) slower than their uniform-scaling counterparts for CIFAR-10 and tinyImageNet, respectively. The trends in memory utilization, accuracy, power, and throughput per watt match the trends seen on float32 models.

Fig. 8.

View Figure

4.3 GPU Results

In this section, we report the results from the NVIDIA Jetson AGX Xavier. Similar to the CPU results, we first report the results for the three base models trained on CIFAR-10, followed by a comparison of MobileNetV2 trained on the two datasets. Finally, we provide a summary for review. For the GPU device, the idle power was measured to be 8.1 W.

4.3.1 Comparing Scaling Methods with Three Models on CIFAR-10.

On the NVIDIA Jetson AGX Xavier, the uniform-scaling models once again tend to perform faster than their NeuralScale counterparts, but often by a small margin, as shown in Figure 9(a). As the device has greater parallel computational capacity than the CPU testbed, the CDL is the more impactful metric. The GPU is therefore less sensitive to the additional FLOPs of the NeuralScale models, and the execution time difference is less pronounced. The slowest model is roughly 4.5 ms, placing its percent of the overall real-time processing budget at approximately 27%. Therefore, in the worst case, these models leave plenty of processing time for other tasks across all three model types. The memory results on the AGX are more irregular, as shown in Figure 9(b). The trend on the VGG-11 model shows that NeuralScale models tend to use slightly more memory than their uniform-scaling counterparts. The ResNet-50 results also similarly follow this trend, though there is more variance in the memory usage for the uniform-scaling models. This variance is caused by the differing memory usage of the 50 trials of TensorRT engine generation, showing that TensorRT will conduct tradeoffs between memory usage and runtime depending on the kernels it selects for the engine to optimize performance.

Fig. 9.

View Figure

For MobileNetV2, the uniform-scaling models at \({0.25\times }\) and \({0.75\times }\) use an unexpectedly large amount of memory. This large utilization is attributed to large amounts of memory being used for the activations, one of the three components summed to provide the estimate of overall runtime memory on the GPU devices. This results indicate that, especially given the large variance at \({0.25\times }\), sometimes the engine generated by TensorRT uses far more memory than even the same model on different runs of the engine generation. We generated 50 separate engines for each individual model, resulting in large differences in estimated memory consumption for some of the models, shown by the large error bars in Figure 9(b). Given the non-deterministic nature of engine generation, it is therefore wise to attempt engine generation multiple times to confirm the most efficient engine is created. It also indicates that TensorRT, on certain attempts to generate the engines, concluded that allocating additional memory for activations was worth the cost for improvements to latency given which kernels it decided to use.

Next are the dynamic-power results, shown in Figure 9(d). On VGG-11, there is not a consistent difference in dynamic-power utilization between uniform scaling and NeuralScale. However, uniform scaling always achieves consistently better throughput per watt than NeuralScale as shown in Figure 9(e). On MobileNetV2 and ResNet-50, the uniform-scaling models always use less power and have better throughput per watt than their NeuralScale equivalents.

Figures 10(b), 10(a), 10(c) show the accuracy versus latency of the models. Once again, we see that the increase in accuracy comes at the cost of latency for VGG-11 and MobileNetV2. Additionally, the NeuralScale line tends to be slightly above the uniform scaling line for the same latency, though the values are close. For ResNet-50, we again show that uniform scaling is best due to its higher accuracy. Figures 10(d), 10(e) show the accuracy versus memory trends. Note that MobileNetV2 is not included due to the large variations in memory. Based on these graphs, NeuralScale achieves better accuracy for the same memory utilization for VGG-11, while uniform scaling outperforms NeuralScale on ResNet-50.

Fig. 10.

View Figure

Quantized-model inference results are shown in Figure 11. TensorRT failed to generate a quantized engine for the NeuralScale \({0.25\times }\) ResNet-50 model as it was unable to calibrate the activations in the model. For the NeuralScale \({1.00\times }\) ResNet-50 model, it failed to generate an engine as it ran out of memory on the device. These results are therefore not included in the bar charts. Again, the inference latency of the uniform-scaling models is less than that of the NeuralScale models. Across all model types, inference latency is now near or below 1.2 ms. In the worst case, the models are only using about 7.2% of the real-time latency budget, leaving plenty of processing time for other tasks. In general, the NeuralScale versions are \(1.2\times\) slower than their uniform-scaling counterparts. In terms of runtime-memory usage, the NeuralScale models use more memory than their uniform-scaling counterparts, but not by as large of a margin as seen in the float32 models.

Fig. 11.

View Figure

The dynamic power of the quantized uniform-scaling VGG-11 models, displayed in Figure 11(d), is consistently higher than their NeuralScale counterparts. This larger dynamic power is likely caused by the model better utilizing the device, as uniform scaling is more efficient in terms of throughput per watt, shown in Figure 11(e). Similar to the float32 models, the uniform-scaling MobileNetV2 and ResNet-50 models consistently use less power and are more efficient than the NeuralScale versions of these models.

When quantized, Figure 12(a) indicates that uniform scaling and NeuralScale have similar accuracy for each millisecond of latency for VGG-11. Figures 12(b),12(c), however, show that uniform scaling outperforms NeuralScale in terms of accuracy versus latency on MobileNetV2 and ResNet-50. The results are similar for accuracy versus memory as Figure 12(d) shows NeuralScale outperforming uniform scaling for VGG-11 and the opposite for MobileNetV2, Figure 12(e), and ResNet-50, Figure 12(f). Also note that for NeuralScale \(0.75\times\) MobileNetV2, the tools struggled to quantize as well, leading to a drop in accuracy. This result indicates that QAT would likely be needed to improve accuracy performance.

Fig. 12.

View Figure

Figure 13 shows the memory usage per parameter for all models on the AGX Xavier. Note the log scale on the y-axis. With float32 models, NeuralScale uses an average of \({4.6\times }\) the runtime memory of uniform-scaling models. For the int8 models, the increase is not nearly as large at \({1.3\times }\) the memory usage. Essentially, on the GPU, the modified parameter layout of the NeuralScale models becomes less efficient than simply scaling the layers uniformly with float32 values. This result is further evidence that, especially on the GPU, simply scaling to a set number of parameters does not guarantee equivalent or efficient runtime-memory usage when using float32 values, while the trend observed on the CPU showed parameter count correlated well with memory usage.

Fig. 13.

View Figure

Another result that is important to note is that neither of the two working NeuralScale ResNet-50 models, \({0.50\times }\) and \({0.75\times }\), could be quantized with high accuracies shown in Figure 11(c). Their uniform-scale counterparts had no quantization issues. Even when we attempted to use all 50,000 training images for the PTQ calibration dataset, the achieved accuracy was only 25% for the NeuralScale \({0.75\times }\) model. Given this result, uniform-scaled models are the better option when using quantization on the AGX Xavier.

4.3.2 Comparing Scaling Methods with Two Datasets on MobileNetV2.

When comparing the two scaling methods across the two datasets, uniform-scaling float32 models continue to be faster than their NeuralScale counterparts, displayed in Figure 14(a). For CIFAR-10, the NeuralScale models are an average of \({1.6\times }\) slower. With the larger images of tinyImageNet, NeuralScale models are \({1.8\times }\) slower than their uniform-scaling counterparts.

Fig. 14.

View Figure

The estimated runtime-memory usage for the tinyImagNet MobileNetV2 model contains similar variations to those seen with the CIFAR-10 models, Figure 14(b). These results indicate that the memory utilization can vary in terms of activation memory when using MobileNetV2 with this version of TensorRT. However, for models with lower variation, uniform-scaling models tend to use less memory than the NeuralScale models.

In terms of power utilization, the dataset does not change the relationship between NeuralScale and uniform scaling, Figure 14(d). Once again, the uniform-scaling models consistently use less power than their NeuralScale counterparts. However, the dataset itself can change dynamic power as the tinyImageNet models use more power than the CIFAR-10 models. With a more complex dataset, the models have more operations to do, likely leading to an increase in power. In terms of throughput per watt, Figure 14(e), uniform scaling is more efficient than NeuralScale. Additionally, performance across datasets is similar.

Finally, the results for the int8 models are shown in Figure 15. NeuralScale models are an average of \({1.2\times }\) and \({1.4\times }\) slower for CIFAR-10 and tinyImageNet, respectively. Additionally, there was significant memory variation, though NeuralScale models consistently use more memory than uniform scaling. Finally, power results are similar to the float32 models with approximately similar throughput per watt between the scaling methods.

Fig. 15.

View Figure

4.4 FPGA Results

This subsection presents the results gathered on the Xilinx ZCU104. Again, we first present results comparing the scaling methods across three models with CIFAR-10, followed by results across datasets and a device summary. For the FPGA device, the idle power was measured to be 16.6 W.

4.4.1 Comparing Scaling Methods with Three Models on CIFAR-10.

The Xilinx ZCU104 using Vitis AI and Xilinx’s DPU only support int8-based models. Inference results are shown in Figure 16. For VGG-11 and MobileNetV2, the uniform-scaling models are slightly faster than the NeuralScale models, displayed in Figure 16(a). However, for ResNet-50, the performance of some of the uniform-scaling models, \({0.25\times }\) and \({0.50\times }\), is slightly slower than the NeuralScale models. Overall, the number of parameters in the model is highly correlated with the runtime on the FPGA with a Pearson correlation of 0.924 (\({\it p}=1.2\cdot 10^{-10}\)). This result indicates that parameter scaling is an appropriate technique for FPGA-based devices.

Fig. 16.

View Figure

Given that the ZCU104 FPGA fabric is not reconfigured to run different models and always has the same DPU-based bitstream, memory measurements were less sensible for this device. On the FPGA, the accuracy performance of the generated models is shown in Figure 16(b). The NeuralScale ResNet-50 models did not reach the expected accuracies. Based on these results, the structure of the NeuralScale models was more challenging for Vitis AI to quantize satisfactorily using PTQ. Techniques like QAT are therefore needed for improved accuracy performance with NeuralScale.

Next, the power results are presented in Figure 16(c). The dynamic-power trends on the FPGA are again similar to the other devices with uniform scaling consistently outperforming NeuralScale. However, in terms of throughput per watt, Figure 16(d), there is no clear advantage between uniform scaling and NeuralScale. The efficiency of the models tends to be very similar, with NeuralScale being more efficient than uniform scaling for some of the ResNet-50 models.

Finally, we compare the accuracy versus latency results for the ZCU104. Figure 17(a) shows NeuralScale outperforming uniform scaling as it is always more accurate for the same latency on VGG-11. For MobileNetV2, at the two smaller scaling ratios, Figure 17(b) indicates that NeuralScale is more accurate than uniform scaling, though performance becomes similar at larger values. Given that the latencies of these models are only a minuscule part of the overall real-time processing budget, it is evident that NeuralScale models are the best option for FPGAs. Finally, Figure 17(c) indicates that uniform scaling is once again faster with higher accuracies than NeuralScale on ResNet-50.

Fig. 17.

View Figure

4.4.2 Comparing Scaling Methods with Two Datasets on MobileNetV2.

The results comparing scaling methods on MobileNetV2 across datasets are shown in Figure 18. In terms of latency, the NeuralScale models are more impacted by the growth in image size of tinyImageNet compared with CIFAR-10. NeuralScale models are only \({1.1\times }\) slower than their uniform-scaling counterparts on CIFAR-10. However, with the larger image size and additional growth in operations, they are \({1.6\times }\) slower than uniform scaling on tinyImageNet. These results indicate that the growth in image size was more detrimental for the more complicated NeuralScale models.

Fig. 18.

View Figure

Finally, in terms of dynamic power, NeuralScale models continue to use more power than uniform scaling, Figure 18(c). Interestingly, the dynamic power of the tinyImageNet models is lower than the CIFAR-10 models. In terms of throughput per watt, the dataset does not have a large impact on the scaling methods as efficiency is similar.

4.5 Parameter Scaling versus Quantization Performance on CIFAR-10

In this subsection, we compare an original float32 model to uniform and NeuralScale float32 models at \({0.50\times }\). We also compare these models to an int8 version of the original model. We chose a scaling ratio of \({0.50\times }\) for this case study as it represents the middle of the range for the ratios tested. We calculate the speedup for each of the optimized models over the original float32 model. We also compare the reduction in memory usage by dividing the original model’s memory usage by the value for each of the three optimizations. The dynamic-power improvement is also compared by dividing the original model’s dynamic power by the reduced model’s power. As a reminder, the dynamic power is the peak load power minus the idle power. The improvement to the throughput per watt (TpW) is also presented. Additionally, we report the accuracy of all models in these comparisons. Please note that the accuracy range depicted on the charts in this section is reduced to 70–100% to make it easier to compare the variations between the optimization methods.

Figure 19 shows these results on the ARM Cortex-A72 device. The NeuralScale models see the least improvement in latency, including a slowdown for the MobileNetV2 model. Uniform scaling improves the latency more than quantization for VGG-11, but quantization has a larger speedup for MobileNetV2 and ResNet50. Across the three models tested, quantization improves latency performance by \({3.1\times }\). Uniform scaling improves latencies by slightly more at \({3.2\times }\), while NeuralScale only improves latencies by \({1.4\times }\). However, in terms of memory usage, Figure 19(a), both NeuralScale and uniform scaling have a larger improvement. In terms of dynamic power, none of the reduction methods provide a significant improvement, Figure 19(c). Finally, in terms of throughput per watt, shown in Figure 19(d), NeuralScale is consistently the worst. Uniform scaling performs best with VGG-11, but quantization produces the best improvement for the other two models. These results indicate that quantization is generally better for improving latencies on the ARM Cortex-A72 while the scaling methods are better for improving memory usage.

Fig. 19.

View Figure

On the AGX Xavier, quantization always outperforms the scaling methods in terms of latency, Figure 20(a), and memory usage, Figure 20(b). Latencies are improved by \({2.7\times }\) with quantization alone, while uniform scaling improves performance by \({1.8\times }\), and NeuralScale improves by \({1.3\times }\). Dynamic power is improved by \({1.3\times }\) with quantization. With uniform scaling it is also improved by \({1.3\times }\), while with NeuralScale it is improved by \({1.2\times }\). Note that if the peak load power were to be considered, these differences would be smaller. Additionally, quantization always produces the largest improvement to throughput per watt, as shown in Figure 20(e). These results indicate that TensorRT and the AGX are well optimized for quantization. There is also minimal overall change to accuracy, shown in Figure 20(c), though a more complex scheme like QAT could be used to improve the performance of the quantized MobileNetV2 model.

Fig. 20.

View Figure

Essentially, quantization was able to achieve similar or better results to that of both uniform scaling and NeuralScale at a ratio of \({0.50\times }\) on the three models tested. This result indicates that quantization can be considered as an alternative to parameter scaling for model deployment. It is equally important, therefore, to consider the practicality of training models with the two different optimizations to decide which tool should be used first, discussed in Section 5.5.

5.1 Model Layout

One of the interesting aspects of the NeuralScale method is how it adjusts the widths of the layers in models, as shown in Figure 21. In this figure, the y-axis represents the width of a layer, while the x-axis represents the layer number, or depth, in the network. The layer widths are shown for a scaling ratio of \({1.00\times }\) as this ratio depicts the model layout with the number of parameters chosen by the original model designers. As the scaling ratio is \({1.00\times }\), the uniform-scaling method simply returns the original network layout.

Fig. 21.

View Figure

Figure 21 is insightful because it shows the structure for a more efficient architecture, in terms of accuracy per parameter, as determined by NeuralScale. NeuralScale tends to shift the parameters from the end of the network to the beginning or middle. In CNNs, earlier layers capture more general features like textures while later layers capture more specific details [41]. Figure 21 displays how NeuralScale infers that general features are more useful on CIFAR-10. This result is interesting because most of the models as designed by the original sources tend to place more parameters near the end of the network.

A possible reason why NeuralScale may shift so many parameters to the beginning of the network is that when it trains and attempts to prune an early layer, it discovers a larger drop in accuracy, and therefore identifies it as more important. As these earlier layers tend to hold more fundamental shapes [41], removing one of them would likely remove a main component of higher-level features. NeuralScale perhaps overparameterizes the early layers even further as it continues to place more emphasis on having duplicates of more fundamental features rather than learning more complex high-level features. We also hypothesize that the vastly different model layouts, especially for ResNet-50, meant that some of the quantization tools struggled with the NeuralScale versions of these models. Given that the uniform-scaled ResNet-50 models did not struggle to quantize with ONNX Runtime, TensorRT, or Vitis AI, it seems likely that the non-standard structure in the NeuralScale models made it more difficult for these tools to perform PTQ well.

For the NeuralScale models, there is an observed large increase in FLOPs for the same parameter count as uniform scaling. This result can be explained by the relationship between FLOPs and the model layout. Earlier in the network, there is also a larger spatial dimension. The relationship of FLOPs in a convolutional layer is on the order of \(O(n^2)\) to the spatial dimension due to the height and width of the image factoring into the FLOPs calculation. Therefore, there is a large increase in the number of FLOPs to be performed when more of the parameters are used for these earlier filters.

5.2 Accuracy Tradeoffs

Given that the same hyperparameters were used for training all models, we cannot guarantee that all models are the most accurate possible. However, it is also important to note that several tools, namely TensorRT and Vitis AI, failed to generate accurate models with NeuralScale ResNet-50 models when attempting quantization with CIFAR-10. Therefore, extra care needs to be taken in the form of QAT to try to improve performance. Doing so, though, will result in increased complexity, which shows the value of a simpler scaling method like uniform scaling.

Accuracy in this study is further limited because parameter scaling often removes the ability to perform transfer learning. When using transfer learning, weights from a model pre-trained on a more complex dataset like ImageNet [5] are used as an initialization for the model being trained on a new dataset. However, without using standard model sizes, it is often difficult to use pretrained weights to help improve the accuracy of a model. There are several methods [4, 8] that attempt to transfer weights between models of different sizes, but these methods have other limitations and are outside the scope of this study. Not using transfer learning can increase the challenge of creating the most accurate model for a given task.

Additionally, NeuralScale ResNet-50 models were less accurate than their uniform-scaling counterparts. This result is perhaps most interesting because NeuralScale is designed to produce more accurate models per parameter, but it failed to do so on CIFAR-10 in this study. Therefore, the additional complexity of the NeuralScale pipeline is not always justified. NeuralScale was also less accurate than uniform scaling on MobileNetV2 with tinyImageNet, differing from the results of Reference [18]. While additional accuracy could likely be gained via an exhaustive hyperparameter search, this would be prohibitively expensive given the training times of architecture descent and NeuralScale as discussed in Section 5.4. It also shows that NeuralScale is sensitive to hyperparameter selection in practice, making it more difficult to recommend for other models and datasets due to these tradeoffs.

Finally, we consider VGG-11 and MobileNetV2, which both showed consistent accuracy versus latency tradeoffs on our test platforms with CIFAR-10. On the embedded CPU, when targeting latency, both methods produced similar results. When targeting runtime-memory utilization, NeuralScale outperforms uniform scaling. For the embedded GPU, again both methods perform similarly in terms of latency, while results are mixed for memory utilization. Finally, on the embedded FPGA, NeuralScale slightly outperforms uniform scaling with generally higher accuracies at the same latency.

5.3 Scaling Ratio Selection

A major challenge with parameter scaling methods for deployment is setting the scaling ratio. While a smaller ratio reduces the parameter count, in the case of NeuralScale, it does not necessarily lead to better on-device performance. One of our main contributions is, therefore, guidance on setting a scaling ratio based on the empirical results of this study.

Table 4 shows the estimated linear relationship between latency and the number of parameters, while Table 5 shows the estimated linear relationship between device memory and parameter count. Using one of the equations from the tables, we can solve for a scaling ratio, S, with Equation (1). \(P_t\) represents the target parameter count, which can be estimated using the tables. In this equation, y represents the target latency (in ms) or memory usage (in MB). \(P_o\) is the value of the parameter count in the original model. (1) \(\begin{equation} S \approx \sqrt {\frac{P_t}{P_o}} = \sqrt {\frac{(y-b)/m}{P_o}} . \end{equation}\)

It is, of course, important to note the Pearson correlation coefficients between the latency and parameters and memory utilization and parameters. Given a lower Pearson correlation coefficient, the less useful the linear equation will be at estimating a scaling ratio. In terms of latency requirements and estimating a scaling ratio, the linear equations are best for estimating float32 performance on the GPU and int8 performance on the FPGA. For targeting a set memory utilization, the linear equations are useful at both precisions on the CPU and with int8 on the GPU. Given these equations, and the accompanying correlation values, we can now estimate a target scaling ratio given a latency or memory constraint.

To compare the usability of these equations, we performed a brief case study with the CIFAR-100 dataset [15], which is similar to CIFAR-10 but instead contains 100 classes. We attempted to achieve real-time performance with uniform scaling on the ARM Cortex-A72 embedded CPU. The result of Equation (1) is a scaling ratio of \({\approx 0.82}\). Therefore, we chose a slightly more conservative ratio of \({0.80\times }\). We then trained a MobileNetV2 model with uniform scaling at this ratio and also attempted quantization on the unscaled model. Once the models were trained, we found that the uniform scaling achieved a latency of \({18.2}\) ms and an accuracy of 64 %. Therefore, even with an approximation for the optimal scaling ratio, we still failed to meet real-time performance of 16.67 ms and would need to attempt training again with a smaller scaling ratio. On the other hand, the quantized base MobileNetV2 model reached real-time performance on CIFAR-10 and continued to do so with CIFAR-100 with a latency of \({9.1}\) ms and an accuracy of \({65.3}\) %, an improvement over the float32 uniform-scaling model at a ratio of \({0.80\times }\). Again, we see the challenges of using scaling methods for deployment. If quantization alone is not capable of meeting the latency requirements, scaling is a useful tool. However, we conclude that quantization remains the best first step for improving runtime performance.

5.4 Training Challenges

While this article is focused on the deployment performance of scaled models, it is also important to note the changes to the training process when using these scaling methods. NeuralScale includes the pre-scaling step of architecture descent, which can be costly depending on the number of parameters in a model. Architecture descent is dependent upon a dataset, and therefore, cannot be performed once per model, but needs to be performed once per dataset+model combination. For reference, Table 6 shows the times for architecture descent for each of the three models on CIFAR-10 and the time of MobileNetV2 with tinyImageNet. As is evident on MobileNetV2, the larger image sizes have a big impact on architecture descent time as the tinyImageNet dataset is \({3.8\times }\) slower than the CIFAR-10 version. ResNet-50 has significantly more parameters and is theoretically therefore a model that would be more likely to be scaled to achieve better on-device performance. However, having more parameters leads to much longer pruning times with architecture descent.

For both uniform scaling and NeuralScale, the models must be completely retrained if changing the scaling ratio or changing the dataset. This could be costly if you do not know what scaling ratio to use, though Section 5.3 provides intuition for selecting a value. Figure 22 shows the average training time per epoch for each of the models. Note that since the NeuralScale models tend to have more FLOPs, they are significantly costlier to train than their uniform-scaling counterparts. For example, with \({1.00\times }\) MobileNetV2, the NeuralScale model takes over \({2.05\times }\) as long to train as the uniform-scaling model on CIFAR-10. This tradeoff in training time is important to consider depending on how many scaling ratio combinations may need to be tested to reach a target latency or memory utilization and how often retraining is expected.

Fig. 22.

View Figure

5.5 Recommendations for Model Deployment

When considering model deployment, reduction methods aim at improving a model’s inference performance. As studied in this research, the results of the parameter scaling reduction method can be of mixed utility. A simple version like uniform scaling tends to give good latency and memory improvements over the original model, but often at the cost of accuracy. NeuralScale generates models that can be more accurate than the original but can perform worse in terms of latency and memory usage.

One of the main challenges of parameter scaling is that training is performed after scaling the model. Other reduction methods like structured pruning use a fully trained model that is then pruned to reduce the model size [7]. This difference means that training techniques like transfer learning can still be effectively used for achieving more accurate models.

In contrast to parameter scaling and similar to structured pruning, quantization is also done after training the model. As shown in Figure 19, quantization is able to achieve similar speedups to uniform scaling and better speedups compared with NeuralScale on the CPU, though it is not as efficient in terms of memory usage. On the GPU in Figure 20, quantization is always better than the two scaling methods in terms of speedup and memory-use reduction. All three of the frameworks tested in this research had native support for quantization through PTQ. While parameter scaling requires changing the model definition in the PyTorch code, quantization can be easily used to reduce the size of a model. Quantization is therefore a natural first reduction method to use since parameter scaling and structured pruning both require additional setup in the model definition to enable the removal of filters.