{"id":31914,"date":"2021-10-06T06:02:51","date_gmt":"2021-10-06T05:02:51","guid":{"rendered":"https:\/\/www.inovex.de\/?p=31914"},"modified":"2022-11-21T11:37:55","modified_gmt":"2022-11-21T10:37:55","slug":"neural-networks-pruning-sparsification","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/","title":{"rendered":"Pruning and Sparsification of Neural Networks"},"content":{"rendered":"<p>Just as the practice of pruning helps to shape and direct the growth of trees and bushes, pruning of synaptic connections in the brain plays a critical role in learning. The use of pruning in neural network models is gaining popularity due to the many benefits it offers.<!--more--><\/p>\n<blockquote>\n<div class=\"bg-color-blue\">\n<p>The pruning practice entails targeted removal of diseased, damaged, dead, non-productive, structurally unsound, or otherwise unwanted tissue from crop and landscape plants. [<a href=\"https:\/\/en.wikipedia.org\/wiki\/Pruning\">Wikipedia<\/a>]<\/p>\n<\/div>\n<\/blockquote>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_83 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#Pruning-and-sparcification-in-Artificial-Neural-Networks\" >Pruning and sparcification in Artificial Neural Networks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#Types-of-sparsity\" >Types of sparsity<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#When-to-prune\" >When to prune?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#Pruning-and-Sparsification-State-of-the-art\" >Pruning and Sparsification: State of the art<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Pruning-and-sparcification-in-Artificial-Neural-Networks\"><\/span>Pruning and sparcification in Artificial Neural Networks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Artificial neural networks (ANNs) are traditionally dense and fit the current computing paradigm and architecture of GPUs well. This match between the available hardware and ANN architectures was the major factor that led to the exciting developments in AI within the last decade.<\/p>\n<p>In recent years, novel ANN models have been increasing in size and complexity to such an extent that the memory and computation costs have become a major issue and the focus of the research community [<a href=\"https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/view\/7123\">Strubell et al. 2020<\/a>]. Another consequence is that large models are over-parametrized which might cause issues with generalization as the models get too sensitive to subtle input changes [<a href=\"https:\/\/arxiv.org\/abs\/1912.03925\">Kohler and Krzyzak 2019<\/a>, <a href=\"https:\/\/arxiv.org\/abs\/2102.00554\">Hoefler et al. 2021<\/a>]. These issues are usually worked around by using various methods during the training phase, such as:<\/p>\n<ul>\n<li aria-level=\"1\">L1 or L2 weight regularization<\/li>\n<li aria-level=\"1\">dropout regularization variants<\/li>\n<li aria-level=\"1\">data augmentation<\/li>\n<\/ul>\n<p>However, none of these methods deal with the fundamental issues of over-parametrization: inefficient use of memory and computation.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-31918\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-300x175.png\" alt=\"ANN before and after pruning\" width=\"300\" height=\"175\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-300x175.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-768x447.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-400x233.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-360x210.png 360w, https:\/\/www.inovex.de\/wp-content\/uploads\/pruning.png 872w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p>Pruning or sparsification is a family of methods to produce sparse neural networks. Similar to other regularization methods, pruning often leads to a better generalization of the networks [<a href=\"https:\/\/arxiv.org\/abs\/1906.03728\">Bartoldson et al. 2020<\/a>]. Many recent works have also shown that sparse models tend to be more robust against adversarial attacks [<a href=\"https:\/\/arxiv.org\/abs\/1912.02386\">Cosentino et al. 2019<\/a>;<a href=\"https:\/\/arxiv.org\/abs\/1803.03880\"> Gopalakrishnanet al.2018<\/a>;<a href=\"https:\/\/papers.nips.cc\/paper\/2018\/hash\/4c5bde74a8f110656874902f07378009-Abstract.html\"> Guo et al.2018<\/a>;<a href=\"https:\/\/arxiv.org\/abs\/1908.04355\"> Madaan et al.2020<\/a>;<a href=\"https:\/\/dl.acm.org\/doi\/10.1145\/3386263.3407651\"> Rakin et al.2020<\/a>;<a href=\"https:\/\/arxiv.org\/abs\/2002.10509\"> Sehwag et al.2020<\/a>;<a href=\"https:\/\/arxiv.org\/abs\/2006.00896\"> Verdenius et al.2020<\/a>]. Sparse models have also significantly lower memory and computation costs compared to dense counterparts (often in the range of 10-100x reduction). This is especially relevant for the training phase because the large and dense models are not only very expensive during inference but often are prohibitively costly to train [<a href=\"https:\/\/arxiv.org\/abs\/2005.14165\">Brown et al. 2020<\/a>]. Thus, the pruning methods that are applied during training are the most interesting and beneficial for reducing computational costs.<\/p>\n<p><figure id=\"attachment_31920\" aria-describedby=\"caption-attachment-31920\" style=\"width: 904px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-31920 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/test-error.png\" alt=\"Pruning Test Error Graph\" width=\"904\" height=\"438\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/test-error.png 904w, https:\/\/www.inovex.de\/wp-content\/uploads\/test-error-300x145.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/test-error-768x372.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/test-error-400x194.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/test-error-360x174.png 360w\" sizes=\"auto, (max-width: 904px) 100vw, 904px\" \/><figcaption id=\"caption-attachment-31920\" class=\"wp-caption-text\">Typical test error vs. sparsity showing Occam&#8217;s hill (results: ResNet-50 on Top-1 ImageNet). Reproduction of Fig. 4 from [Hoefler et al. 2021].<\/figcaption><\/figure>It is important to keep in mind that computation and memory efficiency depend largely on the implementation of the sparse data types and operations on them. Different hardware and different libraries offer a variety of features and support for sparse network models.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Types-of-sparsity\"><\/span>Types of sparsity<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Different pruning methods can be characterized by three main aspects:<\/p>\n<ul>\n<li aria-level=\"1\">which parts of the model are pruned<\/li>\n<li aria-level=\"1\">when is the pruning applied<\/li>\n<li aria-level=\"1\">a heuristic that decides which network weight should be pruned (or regrown)<\/li>\n<\/ul>\n<p>The sparsification or pruning can be permanently applied to the model, also called structural pruning, or temporarily during computation, also called ephemeral sparsification.<\/p>\n<p>Structural pruning in its basic form relates to the fine-grained pruning of weights of a neural network but can also be applied over larger building blocks of models such as neurons, convolutional filters and attention heads. Neural architecture search (NAS) is a family of such block pruning methods that aim to automate the search for better deep learning architectures [<a href=\"https:\/\/arxiv.org\/abs\/1808.05377\">Elsken et al. 2019<\/a>].<\/p>\n<p>The advantage of coarser element (block) style pruning is that the block and the architecture of the model stay dense and efficiently executable on common hardware. In contrast, fine-grained pruning of weights can result in unstructured layers which can result in computation and memory overhead depending on the specific architecture and data types used. Fine-grained pruning, however, is much more flexible and when paired with specialized hardware and sparse activations (like neuromorphic chips and spiking neural networks) can result in a much stronger reduction in computation and memory requirements.<\/p>\n<p>Ephemeral sparsification refers to the sparsification happening online, usually dependent on the current input data samples. This type of sparsification is very common in deep learning models and includes sparse activations like ReLU, which clamp the output of a neuron to zero for input values below zero, and regularization methods such as Dropout which effectively clamp the output of a random sample of neurons to zero during training. Less common methods that belong to this category include sparsification of gradients during training and conditional computation.<\/p>\n<figure id=\"attachment_31922\" aria-describedby=\"caption-attachment-31922\" style=\"width: 904px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-31922 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification.png\" alt=\"Pruning possibilities\" width=\"904\" height=\"312\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification.png 904w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-300x104.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-768x265.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-400x138.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-360x124.png 360w\" sizes=\"auto, (max-width: 904px) 100vw, 904px\" \/><figcaption id=\"caption-attachment-31922\" class=\"wp-caption-text\">Overview of neural network elements that can be pruned. Reproduction of Fig. 6 from [Hoefler et al. 2021].<\/figcaption><\/figure>\n<h2><span class=\"ez-toc-section\" id=\"When-to-prune\"><\/span>When to prune?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Pruning can also be applied at different points during the training of the neural networks. The most trivial and the least beneficial application of pruning is after the training. In this setting, the model is sparsified after the training, which usually results in significant performance degradation. It also means that the full dense model is trained, which is the most costly process in terms of memory and computation.<\/p>\n<p>A better approach is using a schedule of sparsification during training. This way, training is started with a dense model which is gradually sparsified during the training procedure. Different pruning schedules and implementation details [<a href=\"https:\/\/arxiv.org\/abs\/1906.00586\">Wortsmanet al. 2019<\/a>;<a href=\"https:\/\/arxiv.org\/abs\/2006.07253\"> Lin et al. 2020<\/a>] make this method very flexible. The advantage of this method is that sparsification can prevent overfitting gradually during training. But it can also make convergence more unstable.<\/p>\n<p><a href=\"https:\/\/www.sciencedirect.com\/science\/article\/abs\/pii\/S0925231296000549?via%3Dihub\">Prechelt [1997<\/a>] used the generalization loss to quantify the amount of overfitting and adjust the pruning rate dynamically during training. <a href=\"https:\/\/arxiv.org\/abs\/1607.05423\">Jin et al. [2016<\/a>] combined the dense and sparse training in a single schedule with a method called iterative hard thresholding (IHT). <a href=\"https:\/\/arxiv.org\/abs\/1607.04381\">Han et al. [2017<\/a>] used a three-step schedule (dense training to convergence, magnitude-pruning followed by retraining, and dense training) and showed that this schedule leads to significantly higher generalization performance. A follow-up extension to this method is presented in [<a href=\"https:\/\/ieeexplore.ieee.org\/document\/8578988\">Carreira-Perpinan and Idelbayev 2018<\/a>]. A major downside to all these approaches is that they still start with dense models which need to be held in memory and thus do not enable the use of smaller-memory devices.<\/p>\n<p>The last category is fully-sparse training. The training starts with an already sparse model and trains it in the sparse regime by adding and removing weights during its course. These methods often have more hyperparameters but enable very high-dimensional models to be trained on smaller memory hardware. There is a variety of pruning and regrowth techniques that can be combined to implement a fully-sparse training scheme. For example, <a href=\"https:\/\/arxiv.org\/abs\/1902.05967\">Mostafa and Wang [2019<\/a>] use random regrowth and magnitude pruning to maintain sparsity throughout training.<\/p>\n<p><figure id=\"attachment_31931\" aria-describedby=\"caption-attachment-31931\" style=\"width: 904px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-31931\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-schedules.png\" alt=\"3 graphs of structural sparsification schedules\" width=\"904\" height=\"236\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-schedules.png 904w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-schedules-300x78.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-schedules-768x200.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-schedules-400x104.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/sparsification-schedules-360x94.png 360w\" sizes=\"auto, (max-width: 904px) 100vw, 904px\" \/><figcaption id=\"caption-attachment-31931\" class=\"wp-caption-text\">Overview of structural sparsification schedules. Reproduction of Fig. 7 from Hoefler et al. [2021].<\/figcaption><\/figure>Overview of structural sparsification schedules. Reproduction of Fig. 7 from <a href=\"https:\/\/arxiv.org\/abs\/2102.00554\">Hoefler et al. [2021<\/a>].<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Pruning-and-Sparsification-State-of-the-art\"><\/span>Pruning and Sparsification: State of the art<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Using a sparse evolutionary training approach [<a href=\"https:\/\/www.nature.com\/articles\/s41467-018-04316-3?amp=1\">Mocanu et al., 2018<\/a>] demonstrates training of sparsely initialized networks by dynamically changing the connectivity with a magnitude-based pruning and random growth strategy. A related method [<a href=\"https:\/\/arxiv.org\/abs\/1711.05136\">Bellec et al., 2018<\/a>] demonstrates the advantage of training sparse networks with stochastic parameter updates by sampling the sparse connectivity pattern based on a posterior which is theoretically shown to converge to a stationary distribution.<\/p>\n<p><a href=\"http:\/\/proceedings.mlr.press\/v97\/mostafa19a.html\">Mostafa &amp; Wang [2019<\/a>] propose dynamic sparse parameterization to train sparse networks via adaptive threshold-based pruning. <a href=\"https:\/\/arxiv.org\/abs\/1907.04840\">Dettmers &amp; Zettlemoyer, [2019<\/a>] develop a sparse learning method using momentum-based regrowth of pruned weights.<\/p>\n<p>More recently, with the rising concerns about adversarial attacks, <a href=\"http:\/\/proceedings.mlr.press\/v139\/ozdenizci21a.html\">\u00d6zdenizci and Legenstein [2021<\/a>] propose an intrinsically sparse rewiring approach based on Bayesian connectivity sampling to train neural networks by simultaneously optimizing the sparse connectivity structure and the robustness-accuracy trade-off.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In conclusion, the pruning and sparsification of neural networks can provide the following benefits:<\/p>\n<ul>\n<li aria-level=\"1\">improved generalization<\/li>\n<li aria-level=\"1\">robustness to adversarial attacks<\/li>\n<li aria-level=\"1\">memory and compute efficiency<\/li>\n<\/ul>\n<p>Many popular deep learning libraries already provide many built-in pruning methods:<\/p>\n<ul>\n<li aria-level=\"1\"><a href=\"https:\/\/www.tensorflow.org\/model_optimization\/guide\/pruning\">TensorFlow<\/a><\/li>\n<li aria-level=\"1\"><a href=\"https:\/\/pytorch.org\/tutorials\/intermediate\/pruning_tutorial.html\">PyTorch<\/a><\/li>\n<\/ul>\n<p>A large part of the content of this article is based on <a href=\"https:\/\/arxiv.org\/abs\/2102.00554\">Hoefler et al. [2021<\/a>] and <a href=\"http:\/\/proceedings.mlr.press\/v139\/ozdenizci21a.html\">\u00d6zdenizci and Legenstein [2021<\/a>].<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Just as the practice of pruning helps to shape and direct the growth of trees and bushes, pruning of synaptic connections in the brain plays a critical role in learning. The use of pruning in neural network models is gaining popularity due to the many benefits it offers.<\/p>\n","protected":false},"author":199,"featured_media":31940,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[511,385,140],"service":[76,422],"coauthors":[{"id":199,"display_name":"Darjan Salaj","user_nicename":"dsalaj"}],"class_list":["post-31914","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-artificial-intelligence-2","tag-data-engineering","tag-machine-learning","service-artificial-intelligence","service-it-engineering"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Pruning and Sparsification of Neural Networks - inovex GmbH<\/title>\n<meta name=\"description\" content=\"Pruning and sparsification is a family of methods to produce sparse neural networks, which often leads to a better generalization.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Pruning and Sparsification of Neural Networks - inovex GmbH\" \/>\n<meta property=\"og:description\" content=\"Pruning and sparsification is a family of methods to produce sparse neural networks, which often leads to a better generalization.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2021-10-06T05:02:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-11-21T10:37:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Darjan Salaj\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification-1024x576.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Darjan Salaj\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"7\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Darjan Salaj\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/\"},\"author\":{\"name\":\"Darjan Salaj\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/3eb5d9c43358af38d1e463c024df3da5\"},\"headline\":\"Pruning and Sparsification of Neural Networks\",\"datePublished\":\"2021-10-06T05:02:51+00:00\",\"dateModified\":\"2022-11-21T10:37:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/\"},\"wordCount\":1328,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/pruning-and-sparsification.png\",\"keywords\":[\"Artificial Intelligence\",\"Data Engineering\",\"Machine Learning\"],\"articleSection\":[\"Analytics\",\"English Content\",\"General\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/\",\"name\":\"Pruning and Sparsification of Neural Networks - inovex GmbH\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/pruning-and-sparsification.png\",\"datePublished\":\"2021-10-06T05:02:51+00:00\",\"dateModified\":\"2022-11-21T10:37:55+00:00\",\"description\":\"Pruning and sparsification is a family of methods to produce sparse neural networks, which often leads to a better generalization.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/pruning-and-sparsification.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/pruning-and-sparsification.png\",\"width\":1920,\"height\":1080,\"caption\":\"Illustration of scissors pruning an stylized ANN\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/neural-networks-pruning-sparsification\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Pruning and Sparsification of Neural Networks\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/3eb5d9c43358af38d1e463c024df3da5\",\"name\":\"Darjan Salaj\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/cropped-profile_dsalaj_2024_hk-96x96.jpg9fcfeffc54642cddbc08ffd0e0638b80\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/cropped-profile_dsalaj_2024_hk-96x96.jpg\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/cropped-profile_dsalaj_2024_hk-96x96.jpg\",\"caption\":\"Darjan Salaj\"},\"description\":\"As a Data\\\/ML Engineer at inovex, I specialize in building robust pipelines for large-scale data processing in distributed systems. I focus on delivering high-impact data products by prioritizing data quality and master data governance, ensuring that machine learning models are built on a foundation of reliable, well-governed information. I have extensive experience collaborating with SAP customers to modernize their data landscapes and integrate intelligent solutions into their enterprise workflows.\",\"sameAs\":[\"https:\\\/\\\/dsalaj.com\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/darjan-salaj-78599818b\\\/\"],\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/author\\\/dsalaj\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Pruning and Sparsification of Neural Networks - inovex GmbH","description":"Pruning and sparsification is a family of methods to produce sparse neural networks, which often leads to a better generalization.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/","og_locale":"de_DE","og_type":"article","og_title":"Pruning and Sparsification of Neural Networks - inovex GmbH","og_description":"Pruning and sparsification is a family of methods to produce sparse neural networks, which often leads to a better generalization.","og_url":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2021-10-06T05:02:51+00:00","article_modified_time":"2022-11-21T10:37:55+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification.png","type":"image\/png"}],"author":"Darjan Salaj","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification-1024x576.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Darjan Salaj","Gesch\u00e4tzte Lesezeit":"7\u00a0Minuten","Written by":"Darjan Salaj"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/"},"author":{"name":"Darjan Salaj","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/3eb5d9c43358af38d1e463c024df3da5"},"headline":"Pruning and Sparsification of Neural Networks","datePublished":"2021-10-06T05:02:51+00:00","dateModified":"2022-11-21T10:37:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/"},"wordCount":1328,"commentCount":0,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification.png","keywords":["Artificial Intelligence","Data Engineering","Machine Learning"],"articleSection":["Analytics","English Content","General"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/","url":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/","name":"Pruning and Sparsification of Neural Networks - inovex GmbH","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification.png","datePublished":"2021-10-06T05:02:51+00:00","dateModified":"2022-11-21T10:37:55+00:00","description":"Pruning and sparsification is a family of methods to produce sparse neural networks, which often leads to a better generalization.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/pruning-and-sparsification.png","width":1920,"height":1080,"caption":"Illustration of scissors pruning an stylized ANN"},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/neural-networks-pruning-sparsification\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"Pruning and Sparsification of Neural Networks"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/3eb5d9c43358af38d1e463c024df3da5","name":"Darjan Salaj","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-profile_dsalaj_2024_hk-96x96.jpg9fcfeffc54642cddbc08ffd0e0638b80","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-profile_dsalaj_2024_hk-96x96.jpg","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-profile_dsalaj_2024_hk-96x96.jpg","caption":"Darjan Salaj"},"description":"As a Data\/ML Engineer at inovex, I specialize in building robust pipelines for large-scale data processing in distributed systems. I focus on delivering high-impact data products by prioritizing data quality and master data governance, ensuring that machine learning models are built on a foundation of reliable, well-governed information. I have extensive experience collaborating with SAP customers to modernize their data landscapes and integrate intelligent solutions into their enterprise workflows.","sameAs":["https:\/\/dsalaj.com\/","https:\/\/www.linkedin.com\/in\/darjan-salaj-78599818b\/"],"url":"https:\/\/www.inovex.de\/de\/blog\/author\/dsalaj\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/31914","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/199"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=31914"}],"version-history":[{"count":5,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/31914\/revisions"}],"predecessor-version":[{"id":31916,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/31914\/revisions\/31916"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/31940"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=31914"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=31914"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=31914"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=31914"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}