DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Models^††thanks: Authors, Ajitha Rajan and Nikolaos Louloudakis, would like to acknowledge the support received from funding sources, UKRI Trustworthy Autonomous Systems Node in Governance and Regulation (EP/V026607/1) and Royal Society Industry Fellowship, for this work.

Deep Learning Frameworks (the third layer in Figure 4), provide utilities such as model declaration, training, and inference to machine learning engineers. For our study, we consider four widely used DL frameworks: Keras, PyTorch, TensorFlow (TF), and TensorFlow Lite (TFLite).

Keras [12] is a high-level DL framework that provides APIs for effective deep learning usage. It acts as an interface for TensorFlow, and we aim to observe potential overheads and bug introductions from the extra layer of complexity.

PyTorch [13] is an open source machine learning framework based on the Torch library. It supports hardware acceleration for tensor computing operations.

TensorFlow (TF) [14] is an open-source DL framework developed by Google, widely used for training and inference of DNNs.

TensorFlow Lite (TFLite) [14] is a lightweight version of TF, and part of the full TF library, focused only on the inference of DNNs on mobile and lightweight devices.

Developers and researchers convert DNN models from one DL framework to another for one of the following two reasons, 1) portability, to enable the compilation and execution of models on devices of varying capabilities, and 2) differing framework capabilities. For example, some DL frameworks may provide detailed debugging and profiling capabilities for development while others may be better suited for optimized deployment.

Conversion of DNN models between DL frameworks is facilitated by automated conversion processes enabled by tools such as tf2onnx [15], onnx2keras [16], onnx2torch [17], and MMdnn [18]. The conversion process can, however, suffer from errors in the model parameters and graph representation which can potentially affect the output labels. We refer to models defined within a given DL framework as a “native model”, and models that have been converted to another DL framework as a “converted model”. Systems such as ONNX [19] and MMdnn [18] attempt to provide common intermediate formats for conversion between DL frameworks. However, these systems can still be error-prone and have issues supporting bespoke operators, motivating our investigation of framework conversion errors.

Apache TVM [10] is an end-to-end machine learning compiler framework for CPUs, GPUs, and specialized accelerators, actively used and supported by a wide community of developers and researchers. It generates optimized code for specific DNN models and hardware backends, allows us to import DNN models from a range of DL frameworks, and provides profiling utilities such as per-layer inference times. A simplified representation of Apache TVM can be seen in Figure 3. TVM’s support of several DL frameworks, optimization settings, and hardware accelerators makes it a suitable choice to leverage within DeltaNN. It also provides direct importers for models from most popular DL frameworks, which load the models as a TVM computation graph that can be optimized and compiled. TVM also provides a set of graph-based optimizations for DNNs, such as operator fusion, elimination of common subexpressions, loop re-ordering and unrolling, tiling, vectorization, and potentially unsafe precision optimizations such as fast math. TVM applies such optimizations automatically via the use of an -o[0-4] flag. Finally, TVM supports third-party operator kernel libraries such as cuDNN [20] and the Arm Compute Library [21].

A common benchmark for Perception AI models is the ImageNet image classification dataset [9], which requires assigning one of 1000 possible class labels to RGB images of $224\times 224$ pixels. For solving Perception AI problems, such as classification and semantic segmentation, convolutional neural networks (CNNs) are commonly used, which are DNNs characterized by convolutional layers. Transformer-based architectures [22] have begun to provide competitive results in recent years [23, 24], however are still maturing. Thus, for our evaluation we explore three widely used CNN models: MobileNetV2 [6], ResNet101V2 [25], and InceptionV3 [8]. These models are widely known and extensively used for classification and semantic segmentation operations, and are the “backbone network” for other tasks such as object detection [26]. All three models have native definitions within the DL frameworks under study. The accuracy of the native version of each model is shown in Table I. It is expected that the same model may have varying accuracy between frameworks, as each framework will define and train their own version of the model from scratch, which produce different parameters (since training is stochastic), and there may even be small differences in the graph definition (e.g., different padding parameters). We observe that TF and TFLite models have the same accuracy, suggesting that the latter models were converted from the former by developers.

As seen on the left-hand side of Figure 5, the Model Variant Generator takes as input a pre-trained image recognition model (e.g., InceptionV3) sourced from a given DL framework (e.g., PyTorch). If we use one of these pre-trained models “as-is”, and pass it directly to the Model Importer, we refer to it as a native model. However, if we convert it using the DL Framework converter, we refer to the original model as the source, and the converted model as the target. For example, we could convert an InceptionV3 model sourced from PyTorch to the TensorFlow model format. Across the four DL frameworks we support in DeltaNN, we have conversion paths from every framework to every other one.

To implement the conversions, we use the popular ONNX format [19] as an intermediate representation when a direct conversion is not possible. Some DL frameworks, such as PyTorch and TFLite, have native tools for this conversion; whereas for others, such as TensorFlow, we leverage popular third-party conversion tools like tf2onnx [15]. We then convert from ONNX to the target DL framework model format using a number of widely used libraries, such as onnx2torch [17], onnx2keras [16].

The next stage of DeltaNN is Differential Execution of the model variants from the previous stage. It consists of three main steps: (1) the Pre-processing module responsible for normalizing inputs for better model performance (with a variety of pre-processing functions to choose from); (2) the Regular Execution module that executes the model for different target devices; and (3) the Debug Execution module, that executes the model similar to the Regular Execution module but additionally generates execution profiling information that can be used for deeper performance and error insights.

For every pair of model variants, the Analysis stage compares labels and inference time from Model metrics for all images in the dataset. To compare labels, we compare the top ranked predictions between the model variants or performing rank-biased overlap to compare rankings for top-K elements. This means not only can we detect divergence, but also measure the level of divergence. When an image generates different labels or inference time between a pair of model variants, the Analysis module compares the debug traces from the model variants inspecting differences in per-layer activations, weights, and the graph structure. For per-layer activations, we compare mean, max, and standard deviations statistics of the layer activations between the pairs of models. The Analysis module also provides the capability to visualize the differences observed in layer activations and weights.

Our experiments are aimed at evaluating: (1) Robustness of model output, by recording the top-1 output label for every combination of environment parameters and performing pairwise comparisons; and (2) Robustness of model execution time, by measuring average inference time across executions in our dataset and comparing across different configurations. We investigate the following research questions for evaluating robustness:

Output Label Robustness

 RQ1. Label Sensitivity to DL Framework Conversions

Are the output labels of an image recognition model affected when converting the model from a source to a target DL framework? Both source and target frameworks are one among PyTorch, TF, TFLite, or Keras with $\mathit{source}\neq\mathit{target}$. All conversions are through the intermediate ONNX format. We compare output labels of target against the source for each image to check if any errors were introduced by model conversion. We do this for each of the three image recognition models – MobileNetV2, ResNet101V2, and InceptionV3.

 RQ2. Label Sensitivity to Compiler Optimizations

Are the output labels of an image recognition model affected when changing the level of compiler optimization? We vary the optimization level within TVM between Basic, Default, and Extended and observe if there are any difference in the output label for images in the dataset.

Inference Time Robustness

 RQ3. Time Sensitivity to DL Framework Conversion

Are the inference times of an image recognition model affected when converting the model from a source to a target DL framework?

 RQ4. Time Sensitivity to Compiler Optimizations

Are the inference times of an image recognition model affected when changing the level of compiler optimization? We are aware that differences in inference time is to be expected to some extent when changing compiler optimizations. The goal here is to identify unexpected performance degradation and extent of change with the different compiler optimization levels.

We used four different hardware devices, featuring high-end to low-end GPU accelerators:

• •

  an Intel-based server featuring an Nvidia Tesla K40c (GK11BGL) GPU (Server),

• •

  a Nvidia AGX Xavier featuring an Nvidia Volta GPU (Xavier),

• •

  a Laptop featuring an Intel(R) GEN9 HD Graphics NEO (Local),

• •

  and a mobile-class Hikey 970 board featuring an Arm Mali-G72 GPU (Hikey).

For all GPU devices, we generate OpenCL device code, except for the Xavier device where we generate CUDA code, since it does not support OpenCL. We found no accuracy impact between the two programming paradigms, and OpenCL vs. CUDA trade-offs are already explored[52]. We run the test dataset through the model configurations, and take the average inference time.

We use the ImageNet object detection test dataset [9] in our experiments, consisting of 5500 RGB images that are generally resized to 224$\times$224 pixels, and perform classification of 1000 possible labels and measure inference time on each image. For models native to TensorFlow and TFLite, we observed that models actually used input size of 299, rather than the typical 244. In general, using larger input sizes could increase the potential accuracy of the models, but also the computational requirements they need to perform.

All environment parameter combinations could not be executed with all models due to the following incompatibility issues. First, for ResNet101 sourced from PyTorch, we selected the V1 version the model instead of V2 as the V2 version was not provided in the official PyTorch repository. The version difference may have a larger effect on model inference time when we compare across DL frameworks. Second, regarding MobileNetV2, we experienced problems when executing it on the Xavier device, as we received a CUDA_ERROR_INVALID_PTX error, in all cases except when natively sourced from PyTorch. Thus, we do not consider this device configuration for MobileNetV2 in our experiments. Third, while utilizing the conversion process, we encountered various cases where the source model conversion failed, as presented in Figure 6. This happened due to incompatibility either of the source model with the conversion tool or the generated model operations in TVM.

There are five main threats to validity in our experiments:

1. 1.

   We only evaluate robustness using three image recognition models that are widely used. The results are model dependent as seen in our experiments and will likely vary on other models;

2. 2.

   We use the ImageNet [9] object detection test dataset for our experiments, which we believe adequately stresses configurations. Other datasets may yield different robustness results on the models considered;

3. 3.

   Model pre-processing is crucial for model performance [55], and models may give suboptimal performance if given data with ineffective and erroneous pre-processing. We use the recommended pre-processing for each model and DL framework from the official repositories extracted;

4. 4.

   Beyond the DL framework conversions explored in our results, we also have a “hidden” conversion step, i.e., importing models into Apache TVM, which itself may introduce errors. To ensure that errors are not introduced before loading each model into TVM, we generate “target outputs” from their source framework using an indicative number of random image samples. After importing into TVM, we confirmed that we match the target outputs, however this may not guarantee that the import process is entirely bug-free;

5. 5.

   We consider the potential deviations of inference time measurement. To ensure that time deviations are taken into account, we repeat inferences 10 times for each image and use the average inference time across each run across a small-scale test dataset, verifying that no deviations happen on scaling. Note that non-trivial medium-term cache behavior may cause the inference time to change over time with repeated inferences, and depending on the deployment scenario of interest, only the “first” inference time may be of interest, or the inference time of the ‘$N$th’ sample where $N$ is a large value.