{"id":14079,"date":"2018-10-01T13:52:21","date_gmt":"2018-10-01T11:52:21","guid":{"rendered":"https:\/\/www.inovex.de\/blog\/?p=14079"},"modified":"2025-02-26T07:24:46","modified_gmt":"2025-02-26T06:24:46","slug":"image-classification-deployment-gap","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/","title":{"rendered":"From Exploration to Production\u200a\u2014\u200aBridging the Deployment Gap for Deep Learning"},"content":{"rendered":"<p>This is the first part of a series of two blogposts on deep learning model exploration, translation, and deployment. Both involve many technologies like PyTorch, TensorFlow, TensorFlow Serving, Docker, ONNX, NNEF, GraphPipe, and Flask. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset. The first part introduces EMNIST, we develop and train models with PyTorch, translate them with the Open Neural Network eXchange format ONNX and serve them through GraphPipe. Part two will cover TensorFlow Serving and Docker as well as a rather hobbyist approach in which we build a simple web application that serves our model. You can find all the related source code on\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\">GitHub<\/a>.<!--more--><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_83 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Bridge-the-Gap-to-Move-towards-an-AI-powered-Society\" >Bridge the Gap to Move towards an AI-powered Society<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#EMNIST-Image-Classification-Models-with-PyTorch-Translation-with-ONNX-Deployment-with-GraphPipe\" >EMNIST Image Classification Models with PyTorch, Translation with ONNX, Deployment with GraphPipe<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Data-Exploration-the-EMNIST-Dataset\" >Data Exploration: the EMNIST\u00a0Dataset<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Model-Exploration-Develop-Neural-Network-Models-with-PyTorch\" >Model Exploration: Develop Neural Network Models with\u00a0PyTorch<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#A-short-Excursus-on-how-we-can-fail-in-Data-Science\" >A short Excursus on how we can fail in Data\u00a0Science<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Selecting-the-right-Deep-Learning-Framework\" >Selecting the right Deep Learning Framework<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Modeling-Neural-Networks-in-PyTorch\" >Modeling Neural Networks in\u00a0PyTorch<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Linear-Model\" >Linear Model<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Deep-Neural-Network\" >Deep Neural\u00a0Network<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Model-Translation-Using-ONNX%E2%80%8A%E2%80%94%E2%80%8Athe-Open-Neural-Network-eXchange-Format\" >Model Translation: Using ONNX\u200a\u2014\u200athe Open Neural Network eXchange\u00a0Format<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Open-Neural-Network-eXchange-format-ONNX\" >Open Neural Network eXchange format\u00a0(ONNX)<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Model-Deployment-GraphPipe-and-Docker-for-Efficient-Model-Server-Implementations\" >Model Deployment: GraphPipe and Docker for Efficient Model Server Implementations<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#Where-are-we-now-and-where-do-we-go-next-time\" >Where are we now and where do we go next\u00a0time?<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h2 id=\"2336\" class=\"graf graf--h3 graf-after--p\"><span class=\"ez-toc-section\" id=\"Bridge-the-Gap-to-Move-towards-an-AI-powered-Society\"><\/span>Bridge the Gap to Move towards an AI-powered Society<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p id=\"71b0\" class=\"graf graf--p graf-after--h3\">Andrew Ng is an artificial intelligence rockstar. Being professor at Stanford, co-founder of Coursera and deeplearning.ai he tremendously pushes AI education and application. He can be assumed to be the teacher with the most students in this field which earns him great authority. With this he coined the term of\u00a0<strong class=\"markup--strong markup--p-strong\">AI as the new electricity that lights a new industrial revolution<\/strong>. But, in contrast to the ease of this claim it took massive efforts to take electricity from labs to millions of households and factories. It took manpower, heavy investments and practical solutions to take it from theory into practice and make it a matter of course for the masses. And Andrew Ng also knows about this.<\/p>\n<p id=\"34e8\" class=\"graf graf--p graf-after--p\">With AI\u200a\u2014\u200agoing back to his analogy\u200a\u2014\u200athe case is similar. There is a myriad of research in AI and there is also growing practical application, in healthcare, transportation, or commerce\u200a\u2014\u200ajust to name a few. However, most people are still far from actively perceiving AI as an integral part of their daily life. There may be political, societal or cultural factors slowing adaption. To be fair, we also need to be more precise about what we mean when talking about\u00a0<em class=\"markup--em markup--p-em\">Artificial Intelligence.<\/em>\u00a0This\u00a0distinction\u00a0between narrow and general AI illustrates that we haven\u2019t passed the point of what many people mean when they actually talk about AI.<\/p>\n<p id=\"2ea0\" class=\"graf graf--p graf-after--p\">Either way, to solve lacking mass engagement and understanding,\u00a0<strong class=\"markup--strong markup--p-strong\">we need to\u00a0<\/strong><strong class=\"markup--strong markup--p-strong\">make AI more accessible<\/strong>. In\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.youtube.com\/watch?v=JsGPh-HOqjY&amp;t=\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.youtube.com\/watch?v=JsGPh-HOqjY&amp;t=\">his speech at the AI Frontiers Conference in 2017<\/a>\u00a0Andrew Ng shared his vision of an AI-powered society and the need to give people all the tools to make that vision becoming reality. In my view, this goes hand in hand with better bridging the gap between exploration and production. With this I mean the steps to take a machine learning model from its concept and development stage to deployment. Model deployment brings models to life, makes them accessible for the masses and allows us to see their realistic performance. Staying with Andrew Ng\u2019s analogy we can compare\u00a0<strong class=\"markup--strong markup--p-strong\">exploration-translation-deployment of AI\u00a0<\/strong>with generation-transportation-consumption of electricity. To empower people to do this and bridge the gap, I decided to write a blogpost that incorporates different technologies along the way from exploring to deploying neural network models.<\/p>\n<h2 id=\"bc8f\" class=\"graf graf--h3 graf-after--p\"><span class=\"ez-toc-section\" id=\"EMNIST-Image-Classification-Models-with-PyTorch-Translation-with-ONNX-Deployment-with-GraphPipe\"><\/span>EMNIST Image Classification Models with PyTorch, Translation with ONNX, Deployment with GraphPipe<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>For this exploration-to-production tutorial, we will solve a supervised learning task with neural networks and serve predictions through a web service. We split this into the following 4 steps:<\/p>\n<ul class=\"postList\">\n<li id=\"740c\" class=\"graf graf--li graf-after--p\"><strong class=\"markup--strong markup--li-strong\">Data Exploration<\/strong><\/li>\n<li id=\"8039\" class=\"graf graf--li graf-after--li\"><strong class=\"markup--strong markup--li-strong\">Model Development<\/strong><\/li>\n<li id=\"60c7\" class=\"graf graf--li graf-after--li\"><strong class=\"markup--strong markup--li-strong\">Model Translation<\/strong><\/li>\n<li id=\"3943\" class=\"graf graf--li graf-after--li\"><strong class=\"markup--strong markup--li-strong\">Model Deployment<\/strong><\/li>\n<\/ul>\n<p>Starting with the exploration phase, I will first introduce the EMNIST dataset and outline its characteristics. Secondly, we continue with building and exploring neural networks to learn a proper mapping between input and output by assigning correct digits and character labels to images. A few iterations will quickly provide us with a satisfactory accuracy level. Just notice: I will not cover data science topics in detail here as this is not the focus of this blogpost, so data and model exploration will be rather superficial and further model tweaking is left for you. Third, it becomes time to leave the lab and roll up the sleeves: therefore, we translate the model into a framework-independent representation. In the final part, we import this representation and turn it into a deployable model and configure the inference interfaces to make it available for users.<\/p>\n<h2 id=\"d565\" class=\"graf graf--h3 graf-after--p\"><span class=\"ez-toc-section\" id=\"Data-Exploration-the-EMNIST-Dataset\"><\/span>Data Exploration: the EMNIST\u00a0Dataset<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p id=\"6dc0\" class=\"graf graf--p graf-after--h3\">The\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"http:\/\/yann.lecun.com\/exdb\/mnist\/\" target=\"_blank\" rel=\"noopener\" data-href=\"http:\/\/yann.lecun.com\/exdb\/mnist\/\">MNIST<\/a>\u00a0dataset of handwritten digits is the de facto\u00a0<em class=\"markup--em markup--p-em\">Hello World<\/em>\u00a0for data science (among Iris flower data and Boston pricing).\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.nist.gov\/itl\/iad\/image-group\/emnist-dataset\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.nist.gov\/itl\/iad\/image-group\/emnist-dataset\">EMNIST<\/a>\u00a0is offered by the US National Institute of Standards and Technology (NIST). It extends the digits by images of handwritten characters\u200a\u2014\u200auppercase and lowercase. With 28&#215;28 pixels and a single color channel (greyscale) the images have the same format as in MNIST. The whole dataset contains 814,255 images from 62 classes (10 digits, 26 lowercase and 26 uppercase characters). For those of you that are interested in more details, have a look at the corresponding\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/arxiv.org\/pdf\/1702.05373v1.pdf\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/arxiv.org\/pdf\/1702.05373v1.pdf\">paper<\/a> from 2017. MNIST has become really widespread for all kinds of deep learning tutorials due to its easy accessibility, limited size and easy to grasp structure. However, it has also become overused in my view. Therefore, I decided to take a step further for the practical part of this blogpost that is not too far from existing content, but far enough to add some diversity.<\/p>\n<p id=\"e116\" class=\"graf graf--p graf-after--p\">NIST offers the EMNIST dataset in two different formats: binary and Matlab. I decided to stick with the Matlab format and if you like to take the same road, you are welcome to use the proper\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\/blob\/master\/src\/emnist_dl2prod\/utils.py\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\/blob\/master\/src\/emnist_dl2prod\/utils.py\">loaders<\/a>. Reading the images and labels provides us with 697\u00b4932 training and 116\u00b4323 test examples which is almost a 6:1 train-test split that we keep here. Below, you can see some examples from the dataset:<\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_digit_8_187000.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-14082\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_digit_8_187000.png\" alt=\"Image Classification Example digit 8\" width=\"200\" height=\"198\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_digit_8_187000.png 255w, https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_digit_8_187000-150x150.png 150w\" sizes=\"auto, (max-width: 200px) 100vw, 200px\" \/><\/a> <a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_letter_a_305999.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-14083\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_letter_a_305999.png\" alt=\"Image Classification example letter a\" width=\"200\" height=\"198\" \/><\/a> <a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_letter_N_492000.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-14084\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/example_letter_N_492000.png\" alt=\"Image Classification example letter N\" width=\"200\" height=\"198\" \/><\/a><\/p>\n<p>A problem that comes with EMNIST is the highly unbalanced class distribution. Not only between digits and letters, but also within letters. Thus, the 10 digits constitute about half of the instances in both, training and test datasets. Within letters there is also a large disbalance in two ways. First, across upper- and lowercase letters. This particularly applies to some of the most frequently used letters in English language like\u00a0<em class=\"markup--em markup--p-em\">e<\/em>,\u00a0<em class=\"markup--em markup--p-em\">s<\/em>,\u00a0<em class=\"markup--em markup--p-em\">t<\/em>\u00a0or\u00a0<em class=\"markup--em markup--p-em\">o<\/em>. Secondly, within upper- or lowercase letters there is a large disbalance reflecting this differing usage frequency in language. Nevertheless, there is a good aspect left. The letter shares stay approximately constant when we compare train with test data distributions.<\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/pie_plot_emnist_train_classes.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-14088\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/pie_plot_emnist_train_classes-255x300.png\" alt=\"Character distribution of the training set\" width=\"255\" height=\"300\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/pie_plot_emnist_train_classes-255x300.png 255w, https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/pie_plot_emnist_train_classes-400x471.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/pie_plot_emnist_train_classes.png 467w, https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/pie_plot_emnist_train_classes-360x424.png 360w\" sizes=\"auto, (max-width: 255px) 100vw, 255px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/bar_chart_uppercase_classes.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14089\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/bar_chart_uppercase_classes.png\" alt=\"Upper case classes\" width=\"906\" height=\"603\" \/><\/a><\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/bar_chart_lowercase_classes.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14090\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/bar_chart_lowercase_classes.png\" alt=\"Lower case classes\" width=\"906\" height=\"603\" \/><\/a><\/p>\n<p>This unbalanced data can impede classification models from correctly learning the right pattern-to-label matching. There are plenty of methods with their pros and cons to anticipate this. We can downsample classes to achieve a uniform distribution. We can upsample data by duplicating underrepresented class instances or creating synthetic samples. But, we can also adjust our later training loss calculation in a way such that underrepresented classes receive higher weights such that classification errors equally affect the loss regardless of their underlying frequency. But, staying with my remark, this should be none of our concerns here.<\/p>\n<h2 id=\"5153\" class=\"graf graf--h3 graf-after--p\"><span class=\"ez-toc-section\" id=\"Model-Exploration-Develop-Neural-Network-Models-with-PyTorch\"><\/span>Model Exploration: Develop Neural Network Models with\u00a0PyTorch<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Following our first impression of the data we can start exploring useful models. This is generally an iterative process that involves building a model, training it and assessing its performance on our test data to estimate its generalization capacity. Afterwards, we select the best one(s) and bring them into production.<\/p>\n<h3 id=\"31a4\" class=\"graf graf--h4 graf-after--p\"><span class=\"ez-toc-section\" id=\"A-short-Excursus-on-how-we-can-fail-in-Data-Science\"><\/span>A short Excursus on how we can fail in Data\u00a0Science<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>This process also involves a lot of failing which is absolutely fine if you fail fast and learn quickly. Therefore, I generally distinguish between three kinds of failing:<\/p>\n<ul class=\"postList\">\n<li id=\"be0e\" class=\"graf graf--li graf-after--p\">technical<\/li>\n<li id=\"7ca6\" class=\"graf graf--li graf-after--li\">mathematical<\/li>\n<li id=\"08c4\" class=\"graf graf--li graf-after--li\">performance-wise<\/li>\n<\/ul>\n<p id=\"143e\" class=\"graf graf--p graf-after--li\"><strong class=\"markup--strong markup--p-strong\">Technical<\/strong>\u00a0failing is due to incorrect usage of an API leading to uncompilable models or runtime errors. For example, implementing a fully-connected layer in a neural network, we have to define the number of units the layer\u2019s weight matrix expects and the number of output units. If we now stack this layer on top of an input layer that does not meet the defined dimensions the model definition process or instantiation will lead to an error. This kind of problem category is normally straightforward and more or less easy to detect and correct. It concentrates on the implementation of an existing concept for a model that we have in mind.<\/p>\n<p id=\"e949\" class=\"graf graf--p graf-after--p\"><strong class=\"markup--strong markup--p-strong\">Mathematical<\/strong>\u00a0failing can become more complicated to identify and fix. Models that show mediocre prediction quality due to high bias and \/ or high variance mostly fall into this category. This means that a model is correctly defined and successfully runs on training data\u200a\u2014\u200afrom a technical point of view\u200a\u2014\u200abut leads to illogical or totally poor results which makes us believe that the model is learning no proper mapping from input to output data. We may attribute these problems to adverse hyperparameters or inadequate model architectures. For example, we can choose a learning rate that is too high and see our loss functions heavily oscillating instead of converging. These kinds of problems generally involve more intense cause analysis and sometimes a lot trial-and-error. Their solutions also benefits a lot from experience, domain knowledge and solid understanding of theoretical foundations.<\/p>\n<p id=\"e38f\" class=\"graf graf--p graf-after--p\"><strong class=\"markup--strong markup--p-strong\">Performance-wise<\/strong>\u00a0failing characterizes the models that fall short of our expectations. These expectations can be quality- or efficiency-oriented and relate to classification accuracies, the tolerable amount of false positives, but also training duration, model complexity or inference times with respect to later scalability. In this highest stage of failing, we see models that are technically correct and that successfully learn from our data. But we are still unsatisfied with what or how they learn. For example, we may train a shallow neural network on images to classify them correctly and our model converges at 60% classification accuracy. Thus, we are probably unsatisfied with its results and may increase the model complexity to improve on the accuracy.<\/p>\n<h3 id=\"5aea\" class=\"graf graf--h4 graf-after--p\"><span class=\"ez-toc-section\" id=\"Selecting-the-right-Deep-Learning-Framework\"><\/span>Selecting the right Deep Learning Framework<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p id=\"e8cb\" class=\"graf graf--p graf-after--h4\">As we can see, a model needs to pass different quality gates. Generally, it takes many iterations to drive our approach to pass through them. Especially for beginners, I would therefore recommend using frameworks that support fast failing and learning as well as supports our understanding of why we fail. When I started building my first neural networks\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.tensorflow.org\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.tensorflow.org\/\">TensorFlow<\/a>\u00a0was at version 1.0. Now we are approaching version 2.0. It is the most widely adopted deep learning framework with a strong community and supported by Google. I got used to the sometimes rather un-pythonic way of solving problems with it. As opposed to it,\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/pytorch.org\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/pytorch.org\/\">PyTorch<\/a>\u00a0has become a strong competitor with also large support from the community and backed up by Facebook. Where TensorFlow stands as the mature framework with useful features like TensorBoard for visualization and TensorFlow Serving for model deployment and production, it also suffers a little from its static graph paradigm which can slow down failing and learning. Here, PyTorch better integrates with the Python data science ecosystem and uses a dynamic graph paradigm that lets you easily and quickly explore variables and respond to failures. Nevertheless, it&#8217;s missing some functionality and maturity and one also has to mention that TensorFlow is anticipating the dynamic graph with its eager execution mode that is announced to become\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/groups.google.com\/a\/tensorflow.org\/forum\/m\/#!topic\/announce\/qXfsxr2sF-0\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/groups.google.com\/a\/tensorflow.org\/forum\/m\/#!topic\/announce\/qXfsxr2sF-0\">\u201ca central feature of 2.0\u201c<\/a>. I took this blogpost as a chance to become more acquainted with PyTorch and in particular to explore how we can get the most out of different worlds. Doing so, I will focus on TensorFlow and PyTorch as the two most significant frameworks and refrain from comparing other frameworks like CNTK or Caffee.<\/p>\n<h3 id=\"f2ab\" class=\"graf graf--h4 graf-after--p\"><span class=\"ez-toc-section\" id=\"Modeling-Neural-Networks-in-PyTorch\"><\/span>Modeling Neural Networks in\u00a0PyTorch<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p id=\"97c8\" class=\"graf graf--p graf-after--h4\">Back to EMNIST and our problem of digit and letter classification:<\/p>\n<p class=\"graf graf--p graf-after--h4\">In this section, we build two different models and train them on the EMNIST training dataset. First, a simple linear model that fully connects input to output units and applies a softmax function to the output units to predict the class that relates to the highest output value. Secondly, we increase the depth of this shallow neural network by adding two hidden layers which are also fully connected to each other. As loss function, we use the mean cross-entropy. To back-propagate the loss and perform the gradient updates, we use the Adam-Optimizer which uses an adaptive momentum for the gradually diminishing learning rate within stochastic gradient descent. Furthermore, we use mini batches with 128 images each and train for five epochs. After seeing my Jupyter notebook kernel die having almost 100 GB compressed memory, I decided NOT to use all testing instances for evaluation, and rather sampled a random subset of 5% to use for each evaluation step. Nevertheless, we should use the whole test dataset and evaluate batch-wise to get a more reliable perception on the generalization capacity of both models after we trained them. Refer to\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\/blob\/master\/notebooks\/2_exploration_model.ipynb\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\/blob\/master\/notebooks\/2_exploration_model.ipynb\">this notebook<\/a>\u00a0for the code and to try it yourself.<\/p>\n<h3 id=\"8725\" class=\"graf graf--h4 graf-after--p\"><span class=\"ez-toc-section\" id=\"Linear-Model\"><\/span>Linear Model<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p id=\"da80\" class=\"graf graf--p graf-after--h4\">First, we load our EMNIST dataset and set some informative variables for our latter training procedure:<\/p>\n<p>Next, we easily create PyTorch tensors as basic data building blocks from our NumPy Arrays. This also shows the nice integration of PyTorch. In addition, we normalize the pixel values of our grayscale images to the unit interval and flatten them. In PyTorch, we don\u2019t have to one-hot encode our labels for classification tasks. This is internally handled by PyTorch. The only thing we have to take care of here is to provide them in integer format:<\/p>\n<p>Now, we define the linear model as a new class that inherits from\u00a0<code class=\"markup--code markup--p-code\">torch.nn.Module<\/code>. We usually define the model in the constructor and override the\u00a0<code class=\"markup--code markup--p-code\">forward<\/code>\u00a0method. During training we pass it input data, it performs the forward propagation and returns the results of the output layer. This is a minimal configuration for a model in PyTorch with easy extendability as we will see in the second model:<\/p>\n<p>After we set up our model, we create a model instance and define the loss function (the criterion to minimize). Furthermore, we set up the Adam optimizer by passing the model parameters and a proper learning rate for the optimization process:<\/p>\n<p>To control the length of our training, we define the number of epochs, i.e. full passes through the training data, along with a proper batch size which is the number of training instances that are used for a single iteration. In order to track and visualize the model performance throughout training, we also provide some lists that keep losses and accuracies for both, training and test dataset. Finally, we also define the frequency of evaluations:<\/p>\n<p>Finally, we set up our training routine that runs through the epochs and batches respectively. Every iteration, we draw a batch from the training data, reset the gradients accumulated before, perform a forwards pass to obtain predictions and compute the loss resulting from the deviation between predictions and true values. We backpropagate this loss by calling the\u00a0<code class=\"markup--code markup--p-code\">backward()<\/code>\u00a0function on it and perform a parameter update that adjusts the network\u2019s parameters accordingly. Optionally, we take the current model and apply it to the test data to track its progress and estimate the generalization performance. Now we are all set to start the training procedure.<\/p>\n<p>Here is what we get from it: the test accuracy converges to about 68%. Normally, we would now step into some deeper error analysis, for example by identifying for which classes the model was particularly wrong consulting the confusion matrix or by adjusting our loss function weighting underrepresented classes according to their inverse frequency.<\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/line_plot_lm_accuracy.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14092\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/line_plot_lm_accuracy.png\" alt=\"Test accuracy line plot\" width=\"906\" height=\"608\" \/><\/a><\/p>\n<p>In order to illustrate how easy it can be to extend from existing model definitions in PyTorch, I will rather extend the existing architecture to a deep neural network by adding two hidden layers. Without a respectable discussion on what constitutes DNNs to be actually deep, let\u2019s go for it and simply adjust our network a little. However, if you like to be prepared, consult Goodfellow, Bengio and Courville\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.deeplearningbook.org\/contents\/intro.html\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.deeplearningbook.org\/contents\/intro.html\">here<\/a>.<\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/more_layers_please.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14094\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/more_layers_please.png\" alt=\"Statistical learning vs. Neural Networks comic\" width=\"663\" height=\"662\" \/><\/a><\/p>\n<h3 id=\"03be\" class=\"graf graf--h4 graf-after--figure\"><span class=\"ez-toc-section\" id=\"Deep-Neural-Network\"><\/span>Deep Neural\u00a0Network<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p id=\"7a9e\" class=\"graf graf--p graf-after--h4\">Once we set up the surrounding parameters and tracking capacities, exchanging or adjusting models is pretty easy. We may define further classes based on existing or define totally new models following the structure that was already shown above. Thus, we similarly define a DNN by just extending the number of layers as well as defining the operations that connect those layers with each other. Here, we apply the\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/arxiv.org\/pdf\/1511.07289.pdf\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/arxiv.org\/pdf\/1511.07289.pdf\">exponential linear unit function<\/a>\u00a0to activate the logits of the hidden layers.<\/p>\n<p>Finally, we observe the DNN to achieve more accurate classification results as the linear model. Thus, we stick to the more accurate model as it performs significantly better on the test dataset with an approximate accuracy of 78%. his performance serves as proxy for generalization capacity and we want to use models that we expect to generalize well on new data.<\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/line_plot_lm_vs_dnn_test_accuracy.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14095\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/line_plot_lm_vs_dnn_test_accuracy.png\" alt=\"lm vs. dnn test accuracy\" width=\"906\" height=\"608\" \/><\/a><\/p>\n<p id=\"fbdb\" class=\"graf graf--p graf-after--figure\">Of course, these results are far from being really satisfying and rigorous, instead they serve illustrative purposes. We could extend our approach with hierarchical model structures that first distinguish digits from upper- from lowercase letters and do the recognition secondly. We could also shift to using convolutional neural networks. However, the focus here is not model tweaking and exploration, the focus is to better connect exploration with production. Thus, we finish model development at this point and proceed with the model translation.<\/p>\n<h2 id=\"e46c\" class=\"graf graf--h3 graf-after--p\"><span class=\"ez-toc-section\" id=\"Model-Translation-Using-ONNX%E2%80%8A%E2%80%94%E2%80%8Athe-Open-Neural-Network-eXchange-Format\"><\/span>Model Translation: Using ONNX\u200a\u2014\u200athe Open Neural Network eXchange\u00a0Format<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p id=\"3eca\" class=\"graf graf--p graf-after--h3\">After model exploration and selection, we have to consider how to put the model into production. Deploying framework-dependent models can be challenging. In general, there are two paradigms to approach this on a higher level: on the one hand, one can stay with the framework that was used for modelling and training, for example by implementing a web service that embeds a forward propagation. On the other hand, one can cross framework-boundaries and mix technologies. I personally prefer to use the best of different worlds and hope to convince you to do the same\u200a\u2014\u200awith ease and some guidance. With that said, let\u2019s cross the gap and move beyond boundaries.<\/p>\n<p id=\"a912\" class=\"graf graf--p graf-after--p\">First, we elaborate on some criteria for machine learning model deployment in production:<\/p>\n<ul class=\"postList\">\n<li id=\"74f0\" class=\"graf graf--li graf-after--p\"><strong class=\"markup--strong markup--li-strong\">High-performance and Scalability<\/strong><\/li>\n<li id=\"1f4f\" class=\"graf graf--li graf-after--li\"><strong class=\"markup--strong markup--li-strong\">Flexibility<\/strong><\/li>\n<li id=\"be29\" class=\"graf graf--li graf-after--li\"><strong class=\"markup--strong markup--li-strong\">Interoperability<\/strong><\/li>\n<li id=\"3c1e\" class=\"graf graf--li graf-after--li\"><strong class=\"markup--strong markup--li-strong\">Maturity<\/strong><\/li>\n<\/ul>\n<p id=\"9e37\" class=\"graf graf--p graf-after--li\">The solution needs to deal with concurrency and frequency of requests and serve them at high speed regarding transmission, preprocessing, forward propagation and post-processing. If resources deplete, further need to quickly scale to the incoming demand, especially with respect to fault tolerance (reliability). Flexibility refers to language support, configuration and model handling. Interoperability relates to supporting multiple frameworks and formats with the least effort. Finally, maturity can pay off from the very beginning if some criteria are not fulfilled. Maturity fosters adoption which promotes discussion, problem solving as well as the diversity and quantity of ideas. This make it easier to start with something new. Nevertheless, its not crucial if the solution is already very good at some points which may compensate sufficiently in the beginning when there is few adoption.<\/p>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/onnx_icon_new.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-14096\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/onnx_icon_new.png\" alt=\"The ONNX logo\" width=\"2002\" height=\"515\" \/><\/a><\/p>\n<h3 id=\"210e\" class=\"graf graf--h4 graf-after--figure\"><span class=\"ez-toc-section\" id=\"Open-Neural-Network-eXchange-format-ONNX\"><\/span>Open Neural Network eXchange format\u00a0(ONNX)<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p id=\"c685\" class=\"graf graf--p graf-after--h4\"><strong class=\"markup--strong markup--p-strong\">ONNX<\/strong>\u00a0stands for\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/onnx.ai\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/onnx.ai\/\"><strong class=\"markup--strong markup--p-strong\">Open Neural Network eXchange format<\/strong><\/a><strong class=\"markup--strong markup--p-strong\">\u00a0<\/strong>and claims to be the \u201cnew open ecosystems for interchangeable AI models\u201c. It provides a standard format to represent deep learning models and allows interoperability by easily sharing models between frameworks. It is supported by Amazon, Facebook and Microsoft and exists since 2017. The ONNX format defines a computational graph model. There are many tutorials referenced on their\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/onnx\/tutorials\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/onnx\/tutorials\">GitHub page<\/a>\u00a0for exporting and importing from and to different frameworks. Caffee2, Microsoft Cognitive Toolkit, MXNet and PyTorch natively support ONNX. There are also connectors for\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/onnx\/onnx-tensorflow\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/onnx\/onnx-tensorflow\">TensorFlow<\/a>\u00a0and\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/onnx\/onnx-coreml\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/onnx\/onnx-coreml\">CoreML<\/a>. And with the\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/groups.google.com\/a\/tensorflow.org\/forum\/m\/#!topic\/announce\/qXfsxr2sF-0\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/groups.google.com\/a\/tensorflow.org\/forum\/m\/#!topic\/announce\/qXfsxr2sF-0\">TensorFlow 2.0 announcement<\/a>\u00a0pointing out the \u201cstandardization on exchange formats\u201c we may also hope for TensorFlow natively supporting ONNX soon. However, we can see some rather hesitant reactions from officials in\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/tensorflow\/tensorflow\/issues\/12888\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/tensorflow\/tensorflow\/issues\/12888\">this<\/a>\u00a0ongoing discussion on GitHub. In general, ONNX can be seen as a modern, accessible and deep learning focused successor of PMML, the\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"http:\/\/dmg.org\/\" target=\"_blank\" rel=\"noopener\" data-href=\"http:\/\/dmg.org\/\">Predictive Model Markup Language (PMML)<\/a>, which is used to represent predictive models along with data transformations in a framework-unspecific way.<\/p>\n<p id=\"f39b\" class=\"graf graf--p graf-after--p\">Fortunately, PyTorch has already integrated ONNX and thus provides\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/pytorch.org\/docs\/stable\/onnx.html\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/pytorch.org\/docs\/stable\/onnx.html\">functionality<\/a>\u00a0to export models into ONNX protobuf format. Therefore, we export our model with\u00a0<code class=\"markup--code markup--p-code\">torch.onnx.export<\/code>\u00a0and pass it the model, a path where to save it and an example input. Since the model export itself works via\u00a0<em class=\"markup--em markup--p-em\">tracing<\/em>\u00a0we need to provide this example input. This means that invoking the export triggers the model to perform a forward pass using this input and records a trace of operators that were involved in computing the output. Thus, the example can also be random data, but needs to match the shape we specified for model input. Finally, we may also specify names for model parameters. Now, we can apply it to the trained deep neural network\u00a0<code class=\"markup--code markup--p-code\">dnn_model<\/code>\u00a0to obtain the export file:<\/p>\n<h2 id=\"a79e\" class=\"graf graf--h3 graf-after--figure\"><span class=\"ez-toc-section\" id=\"Model-Deployment-GraphPipe-and-Docker-for-Efficient-Model-Server-Implementations\"><\/span>Model Deployment: GraphPipe and Docker for Efficient Model Server Implementations<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p id=\"569a\" class=\"graf graf--p graf-after--h3\">Oracle\u00a0recently published\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/oracle.github.io\/graphpipe\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/oracle.github.io\/graphpipe\/\">GraphPipe<\/a>\u00a0to \u201csimplify machine learning model deployment and decouple it from framework-specific model implementations.\u201c GraphPipe\u2019s machine learning transport specification uses\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/google.github.io\/flatbuffers\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/google.github.io\/flatbuffers\/\">Google\u2019s flatbuffers<\/a>. It provides reference model servers for TensorFlow, caffee2 and ONNX as well as client implementations for Go, Java and Python. Integrating ONNX support broadens its support for even more deep learning frameworks. Although it accepts ONNX as a framework-independent model format, GraphPipe uses framework-specific model servers. Tweaking the model server configurations and standardizing client-server communication, GraphPipe excels in server efficiency and performance. Respective model servers come embedded into Docker containers offered on their website. The ONNX model server accepts ONNX models as well as models in caffee2 NetDef format. The TensorFlow model server handles both TensorFlow models, the SavedModel and the GraphDef format. Here is a summary of how a GraphPipe handles a request:<\/p>\n<blockquote id=\"68c9\" class=\"graf graf--blockquote graf-after--p\"><p>In essence, a GraphPipe request behaves like a TensorFlow-serving predict request, but using flatbuffers as the message format. Flatbuffers are similar to google protocol buffers, with the added benefit of avoiding a memory copy during the deserialization step. The flatbuffer definitions provide a request message that includes input tensors, input names and output names. A GraphPipe remote model accepts the request message and returns one tensor per requested output name. The remote model also must provide metadata about the types and shapes of the inputs and outputs that it supports.<\/p><\/blockquote>\n<p><a href=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/graphpipe_overview.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-14097 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/graphpipe_overview.jpg\" alt=\"GraphPipe overview flow chart\" width=\"512\" height=\"384\" \/><\/a><\/p>\n<p id=\"0cda\" class=\"graf graf--p graf-after--figure\">ONNX and GraphPipe are not the only technologies that promote interoperability and deployment ease. Almost at the same time as GraphPipe was released, the Khronos Group published its\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.khronos.org\/nnef\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.khronos.org\/nnef\">Neural Network Exchange Format (NNEF)<\/a>\u00a0as a new standard to support interoperability. In addition, with\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/pytorch\/glow\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/pytorch\/glow\">Glow\u200a\u2014\u200aa Compiler for Neural Network hardware accelerators<\/a>\u00a0there is another way of transforming models from different frameworks into a common standard. Feel free to check them and don\u2019t forget to share your experiences. For this blogpost, we will concentrate on ONNX and GraphPipe for now on and heading back to the practical part. With this said, let\u2019s get back to our EMNIST image classification model and serve it through GraphPipe. You can refer to\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\/blob\/master\/notebooks\/3_translation_ONNX_GraphPipe.ipynb\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/squall-1002\/emnist_dl2prod\/blob\/master\/notebooks\/3_translation_ONNX_GraphPipe.ipynb\">this Jupyter notebook<\/a>\u00a0for the code.<\/p>\n<p id=\"496c\" class=\"graf graf--p graf-after--p\">First, make sure you have Docker installed on your machine. Secondly, pull the\u00a0<em class=\"markup--em markup--p-em\">graphpipe-tf<\/em>\u00a0and\u00a0<em class=\"markup--em markup--p-em\">graphpipe-onnx<\/em>\u00a0container images with\u00a0<code class=\"markup--code markup--p-code\">docker pull sleepsonthefloor\/graphpipe-onnx:cpu<\/code>\u00a0and\u00a0<code class=\"markup--code markup--p-code\">docker pull sleepsonthefloor\/graphpipe-tf:cpu<\/code>\u00a0. Third, use\u00a0<code class=\"markup--code markup--p-code\">pip install graphpipe\u00a0<\/code>to install the GraphPipe client to test our model subsequently. Consult the\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/oracle.github.io\/graphpipe\/#\/guide\/user-guide\/overview\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/oracle.github.io\/graphpipe\/#\/guide\/user-guide\/overview\">user guide<\/a>\u00a0for further information. The references are simple to use and let us quickly serve a deep learning model through a running model server. We just go ahead with our ONNX model and start our model server from the root of the repository and using port 9000. In order to do that, we have to create the\u00a0<code class=\"markup--code markup--p-code\">value-inputs<\/code>\u00a0first which are required for serving models with\u00a0<em class=\"markup--em markup--p-em\">graphpipe-onnx<\/em>. Unfortunately, the user guide contains very little information on how to set up the\u00a0<code class=\"markup--code markup--p-code\">value_inputs.json<\/code>\u00a0:<\/p>\n<pre name=\"14eb\" id=\"14eb\" class=\"graf graf--pre graf-after--p\">--value-inputs string value_inputs.json for the model. Accepts local file or http(s) url.<\/pre>\n<p id=\"34d0\" class=\"graf graf--p graf-after--pre\">However, we can follow the structure for the exemplary\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/oracle.github.io\/graphpipe\/models\/squeezenet.value_inputs.json\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/oracle.github.io\/graphpipe\/models\/squeezenet.value_inputs.json\">Squeezenet input\u00a0<\/a>assuming the outer list to describe the batch size per request and the inner list to hold the dimensions of the input:<\/p>\n<pre name=\"63ab\" id=\"63ab\" class=\"graf graf--pre graf-after--p\">{\"data_0\": [1, [1, 3, 227, 227]]}<\/pre>\n<pre name=\"3fab\" id=\"3fab\" class=\"graf graf--pre graf-after--figure\">{\"flattened_rescaled_img_28x28\": [1, [1, 784]]}<\/pre>\n<p id=\"5fcc\" class=\"graf graf--p graf-after--pre\">Now it is time to start the model server with the following command:<\/p>\n<pre name=\"2fba\" id=\"2fba\" class=\"graf graf--pre graf-after--p\">docker run -it \u2014 rm \\\r\n\r\n -v \u201c$PWD\/models:\/models\/\u201c \\\r\n\r\n -p 9000:9000 \\\r\n\r\n sleepsonthefloor\/graphpipe-onnx:cpu \\\r\n\r\n \u2014 value-inputs=\/models\/dnn_model_pt.value_inputs.json \\\r\n\r\n \u2014 model=..\/models\/dnn_model_pt.onnx \\\r\n\r\n \u2014 listen=0.0.0.0:9000<\/pre>\n<p id=\"643d\" class=\"graf graf--p graf-after--pre\">Unfortunately, we fail with the following log message (for the complete log, see the notebook):<\/p>\n<pre name=\"b758\" id=\"b758\" class=\"graf graf--pre graf-after--p\">terminate called after throwing an instance of \u2018caffe2::EnforceNotMet\u2019\r\n\r\n what(): [enforce fail at tensor.h:147] values.size() == size_. 784 vs 1229312<\/pre>\n<p id=\"18c6\" class=\"graf graf--p graf-after--pre\">This seems like some tensor shapes are not set as expected. Nevertheless, without proper knowledge of Caffee2, this can be hard to debug. Therefore, we alternatively try to load both resources (<code class=\"markup--code markup--p-code\">dnn_model_pt.value_inputs.json<\/code>and\u00a0<code class=\"markup--code markup--p-code\">dnn_model_pt.onnx<\/code>) directly from GitHub which also fails. Despite the fact that the Squeezenet example worked, trying to replicate this for our own ONNX model is currently a big hassle in GraphPipe. However, with\u00a0<em class=\"markup--em markup--p-em\">graphpipe-tf<\/em>\u00a0as the TensorFlow model server there seems to be a way out. Thanks to ONNX we can easily generate a TensorFlow model export from our ONNX-model and try to serve that one through GraphPipe. Therefore, we only have to install the\u00a0<a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/onnx\/tensorflow-onnx\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/onnx\/tensorflow-onnx\">ONNX TensorFlow connector<\/a>. Hence, let\u2019s give it another try:<\/p>\n<p id=\"ec39\" class=\"graf graf--p graf-after--figure\">After translating our ONNX model into a TensorFlow protobuf, we start the Docker container with:<\/p>\n<pre name=\"78e6\" id=\"78e6\" class=\"graf graf--pre graf-after--p\">docker run -it \u2014 rm \\\r\n\r\n -v \u201c$PWD\/models:\/models\/\u201c \\\r\n\r\n -p 9000:9000 \\\r\n\r\n sleepsonthefloor\/graphpipe-tf:cpu \\\r\n\r\n \u2014 model=\/models\/dnn_model_tf.pb \\\r\n\r\n \u2014 listen=0.0.0.0:9000<\/pre>\n<p id=\"23b6\" class=\"graf graf--p graf-after--pre\">Brings the following to our terminal:<\/p>\n<pre name=\"3931\" id=\"3931\" class=\"graf graf--pre graf-after--p\">INFO[0000] Starting graphpipe-tf version 1.0.0.10.f235920 (built from sha f235920)\r\n\r\nINFO[0000] Model hash is \u2018e3ee2541642a8ef855d49ba387cee37d5678901f95e8aa0d3ed9a355cf464fb2\u2019\r\n\r\nINFO[0000] Using default inputs [flattened_rescaled_img_28x28:0]\r\n\r\nINFO[0000] Using default outputs [Softmax:0]\r\n\r\nINFO[0000] Listening on \u20180.0.0.0:9000<\/pre>\n<p id=\"0e64\" class=\"graf graf--p graf-after--pre\">This looks much better now. Despite some initial difficulties, we deployed our model quickly in just a few lines of code. That is why we know want to know whether what we deployed behaves similar to what we trained.<\/p>\n<p id=\"32b2\" class=\"graf graf--p graf-after--p\">Therefore, we finally validate the deployment using some test data queries against the REST interface of our containerized model server. To do this, we use the GraphPipe client implementation which we have already installed:<\/p>\n<section class=\"section section--body section--first\">\n<div class=\"section-content\">\n<div class=\"section-inner sectionLayout--insetColumn\">\n<pre name=\"87fa\" id=\"87fa\" class=\"graf graf--pre graf-after--figure\">Predicted Label \/ True Label: 2 == z ? \u2014 False !\r\n\r\nPredicted Label \/ True Label: r == r ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: 3 == 3 ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: h == h ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: 2 == 2 ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: j == j ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: 5 == 5 ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: 2 == 2 ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: 7 == 7 ? \u2014 True !\r\n\r\nPredicted Label \/ True Label: 8 == 8 ? \u2014 True !<\/pre>\n<p id=\"e533\" class=\"graf graf--p graf-after--pre\">This is what happens in the backend:<\/p>\n<pre name=\"1412\" id=\"1412\" class=\"graf graf--pre graf-after--p\">\u2026\r\n\r\nINFO[0113] Request for \/ took 773.621\u00b5s\r\n\r\nINFO[0113] Request for \/ took 859.584\u00b5s\r\n\r\nINFO[0113] Request for \/ took 810.67\u00b5s\r\n\r\n\u2026<\/pre>\n<p id=\"eebe\" class=\"graf graf--p graf-after--pre\">Great, our model is alive and kicking, quickly responds to requests and shows to be similarly accurate as it was during training. Feel free to try more examples for some less questionable statistical significance\u00a0\ud83d\ude09<\/p>\n<h3 id=\"bf4c\" class=\"graf graf--h3 graf-after--p\"><span class=\"ez-toc-section\" id=\"Where-are-we-now-and-where-do-we-go-next-time\"><\/span>Where are we now and where do we go next\u00a0time?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p id=\"2f09\" class=\"graf graf--p graf-after--h3\">That was a lot to read, but hopefully also a lot to learn. This is what I took from this work and I hope you can share the experience or contribute additional feedback:<\/p>\n<ul class=\"postList\">\n<li id=\"d04b\" class=\"graf graf--li graf-after--p\">PyTorch excels in ease and natively supports ONNX interoperability, though it lacks an integrated deployment solution.<\/li>\n<li id=\"e0c0\" class=\"graf graf--li graf-after--li\">TensorFlow excels by maturity and efficiency, and we hope that it will also excel at interoperability making ONNX support a matter of course.<\/li>\n<li id=\"7dbb\" class=\"graf graf--li graf-after--li\">ONNX is a convincing mediator that promotes model interoperability. I wish to see it integrating some more connectors in the future, like\u00a0<em class=\"markup--em markup--li-em\">onnx-tf<\/em>.<\/li>\n<li id=\"16c7\" class=\"graf graf--li graf-after--li\">GraphPipe is useful and neat, but comes with some teething trouble. Integration of TensorFlow works right of the box which isn\u2019t the case for ONNX models.<\/li>\n<\/ul>\n<p id=\"ebbe\" class=\"graf graf--p graf-after--li graf--trailing\">Stay tuned and don\u2019t miss the second part of this series that will be released in just a week. Thanks for reading, clapping, and sharing and don\u2019t forget to\u00a0<strong class=\"markup--strong markup--p-strong\">mind the gap<\/strong>.<\/p>\n<\/div>\n<\/div>\n<\/section>\n<section class=\"section section--body section--last\">\n<div class=\"section-content\">\n<div class=\"section-inner sectionLayout--insetColumn\">\n<p id=\"32a3\" class=\"graf graf--p graf--leading graf--trailing\"><em class=\"markup--em markup--p-em\">Thanks to my colleagues\u00a0<\/em><a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/florianwilhelm.info\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/florianwilhelm.info\/\"><em class=\"markup--em markup--p-em\">Florian Wilhelm<\/em><\/a><em class=\"markup--em markup--p-em\">, Jan Bender, and Michael Timpelan for their valuable feedback.<\/em><\/p>\n<p>This blog first appeared on Towards Data Science.<\/p>\n<\/div>\n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"<p>This is the first part of a series of two blogposts on deep learning model exploration, translation, and deployment. Both involve many technologies like PyTorch, TensorFlow, TensorFlow Serving, Docker, ONNX, NNEF, GraphPipe, and Flask. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset. [&hellip;]<\/p>\n","protected":false},"author":89,"featured_media":14104,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[206,225],"service":[431],"coauthors":[{"id":89,"display_name":"Marcel Kurovski","user_nicename":"mkurovski"}],"class_list":["post-14079","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-data-science","tag-data-science-in-production","service-data-science"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Bridging the Deployment Gap for Deep Learning [Tutorial]<\/title>\n<meta name=\"description\" content=\"This article introduces EMNIST, we develop and train models with PyTorch, translate them with the Open Neural Network eXchange format ONNX and serve them through GraphPipe. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Bridging the Deployment Gap for Deep Learning [Tutorial]\" \/>\n<meta property=\"og:description\" content=\"This article introduces EMNIST, we develop and train models with PyTorch, translate them with the Open Neural Network eXchange format ONNX and serve them through GraphPipe. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2018-10-01T11:52:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-02-26T06:24:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Marcel Kurovski\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero-1024x576.jpg\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marcel Kurovski\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"23\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Marcel Kurovski\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/\"},\"author\":{\"name\":\"Marcel Kurovski\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/3944aeed829f9e8621835c12d5c6b433\"},\"headline\":\"From Exploration to Production\u200a\u2014\u200aBridging the Deployment Gap for Deep Learning\",\"datePublished\":\"2018-10-01T11:52:21+00:00\",\"dateModified\":\"2025-02-26T06:24:46+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/\"},\"wordCount\":4394,\"commentCount\":2,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2018\\\/10\\\/mind-the-gap-data-science-hero.jpg\",\"keywords\":[\"Data Science\",\"Data Science in Production\"],\"articleSection\":[\"Analytics\",\"English Content\",\"General\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/\",\"name\":\"Bridging the Deployment Gap for Deep Learning [Tutorial]\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2018\\\/10\\\/mind-the-gap-data-science-hero.jpg\",\"datePublished\":\"2018-10-01T11:52:21+00:00\",\"dateModified\":\"2025-02-26T06:24:46+00:00\",\"description\":\"This article introduces EMNIST, we develop and train models with PyTorch, translate them with the Open Neural Network eXchange format ONNX and serve them through GraphPipe. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2018\\\/10\\\/mind-the-gap-data-science-hero.jpg\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2018\\\/10\\\/mind-the-gap-data-science-hero.jpg\",\"width\":1920,\"height\":1080,\"caption\":\"mind the gap deep learning hero\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/image-classification-deployment-gap\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"From Exploration to Production\u200a\u2014\u200aBridging the Deployment Gap for Deep Learning\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/3944aeed829f9e8621835c12d5c6b433\",\"name\":\"Marcel Kurovski\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/60619b0c05f0a2c53493da496f506ec95329d5e1834df91c2de2917876284b7d?s=96&d=retro&r=g9f07d90a15e7954fbacf07b430bdcf51\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/60619b0c05f0a2c53493da496f506ec95329d5e1834df91c2de2917876284b7d?s=96&d=retro&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/60619b0c05f0a2c53493da496f506ec95329d5e1834df91c2de2917876284b7d?s=96&d=retro&r=g\",\"caption\":\"Marcel Kurovski\"},\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/author\\\/mkurovski\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Bridging the Deployment Gap for Deep Learning [Tutorial]","description":"This article introduces EMNIST, we develop and train models with PyTorch, translate them with the Open Neural Network eXchange format ONNX and serve them through GraphPipe. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/","og_locale":"de_DE","og_type":"article","og_title":"Bridging the Deployment Gap for Deep Learning [Tutorial]","og_description":"This article introduces EMNIST, we develop and train models with PyTorch, translate them with the Open Neural Network eXchange format ONNX and serve them through GraphPipe. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset.","og_url":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2018-10-01T11:52:21+00:00","article_modified_time":"2025-02-26T06:24:46+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero.jpg","type":"image\/jpeg"}],"author":"Marcel Kurovski","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero-1024x576.jpg","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Marcel Kurovski","Gesch\u00e4tzte Lesezeit":"23\u00a0Minuten","Written by":"Marcel Kurovski"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/"},"author":{"name":"Marcel Kurovski","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/3944aeed829f9e8621835c12d5c6b433"},"headline":"From Exploration to Production\u200a\u2014\u200aBridging the Deployment Gap for Deep Learning","datePublished":"2018-10-01T11:52:21+00:00","dateModified":"2025-02-26T06:24:46+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/"},"wordCount":4394,"commentCount":2,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero.jpg","keywords":["Data Science","Data Science in Production"],"articleSection":["Analytics","English Content","General"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/","url":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/","name":"Bridging the Deployment Gap for Deep Learning [Tutorial]","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero.jpg","datePublished":"2018-10-01T11:52:21+00:00","dateModified":"2025-02-26T06:24:46+00:00","description":"This article introduces EMNIST, we develop and train models with PyTorch, translate them with the Open Neural Network eXchange format ONNX and serve them through GraphPipe. We will orchestrate these technologies to solve the task of image classification using the more challenging and less popular EMNIST dataset.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero.jpg","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2018\/10\/mind-the-gap-data-science-hero.jpg","width":1920,"height":1080,"caption":"mind the gap deep learning hero"},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/image-classification-deployment-gap\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"From Exploration to Production\u200a\u2014\u200aBridging the Deployment Gap for Deep Learning"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/3944aeed829f9e8621835c12d5c6b433","name":"Marcel Kurovski","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/secure.gravatar.com\/avatar\/60619b0c05f0a2c53493da496f506ec95329d5e1834df91c2de2917876284b7d?s=96&d=retro&r=g9f07d90a15e7954fbacf07b430bdcf51","url":"https:\/\/secure.gravatar.com\/avatar\/60619b0c05f0a2c53493da496f506ec95329d5e1834df91c2de2917876284b7d?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/60619b0c05f0a2c53493da496f506ec95329d5e1834df91c2de2917876284b7d?s=96&d=retro&r=g","caption":"Marcel Kurovski"},"url":"https:\/\/www.inovex.de\/de\/blog\/author\/mkurovski\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/14079","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/89"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=14079"}],"version-history":[{"count":4,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/14079\/revisions"}],"predecessor-version":[{"id":61014,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/14079\/revisions\/61014"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/14104"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=14079"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=14079"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=14079"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=14079"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}