{"id":14673,"date":"2019-04-03T12:27:26","date_gmt":"2019-04-03T10:27:26","guid":{"rendered":"https:\/\/www.inovex.de\/blog\/?p=14673"},"modified":"2026-03-17T07:59:21","modified_gmt":"2026-03-17T06:59:21","slug":"machine-learning-model-management","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/","title":{"rendered":"Frameworks for Machine Learning Model Management"},"content":{"rendered":"<p>In my previous <a href=\"https:\/\/www.inovex.de\/blog\/how-to-manage-machine-learning-models\/\">blog post &#8222;how to manage machine learning models&#8220;<\/a>\u00a0I explained the difficulties within the process of developing a good machine learning model and motivated using a tool to support data scientists with this challenge. This blog post will follow up by comparing three different tools developed to support reproducible machine learning model development:<!--more--><\/p>\n<ul>\n<li><a href=\"https:\/\/mlflow.org\/\">MLFlow<\/a> developed by DataBricks (the company behind Apache Spark)<\/li>\n<li><a href=\"https:\/\/dvc.org\/\">DVC<\/a>, an open-source project that gets its main support by the San Francisco-\/ London based startup iterative.ai<\/li>\n<li><a href=\"https:\/\/github.com\/IDSIA\/sacred\">Sacred<\/a>, an academical project developed by different researchers<\/li>\n<\/ul>\n<div class=\"bg-color-blue\">\n<p>Have a look at our 2022 update comparing <a href=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-experiment-tracking\/\">frameworks for machine learning experiment tracking<\/a> and see how MLflow, ClearML, neptune.ai and DAGsHub hold up!<\/p>\n<\/div>\n<p>First there will be one paragraph per framework that describes the project and shows some code examples. In the end of the article you will find a framework comparison and recommendations when to use which framework. As with my previous post the sklearn dataset on Boston-Housing prices will be used as basis. You can find a notebook to play with in this <a href=\"https:\/\/github.com\/inovex\/machine-learning-model-management\">github repo<\/a>. This notebook also includes instructions how to install the frameworks as well as some other functions we will use within the code examples below, but that won&#8217;t be discussed further, to place focus on the framework specific parts and omit boilerplate code.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_83 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#DVC\" >DVC<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#MLFlow\" >MLFlow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#Sacred\" >Sacred<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#Conclusion\" >Conclusion<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#Read-on\" >Read on<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"DVC\"><\/span>DVC<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>DVC means &#8222;data (science) version control&#8220; and aims to do for data science what git already does for software development: Making development processes traceable and reproducible. Therefore it strongly relies on git to track the source files and also uses it as an inspiration for its own CLI: To start a project, run &#8222;dvc init&#8220;, which will\u00a0 initialise some hidden files within a .dvc folder\u2014just as git does. After initialisation you will need to write your program in such a way that everything will be stored within a file. DVC is agnostic to the programming language used, since it is controlled via terminal commands and does not have access to python variable values or similar.<\/p>\n<p>DVC basically has three types of objects:<\/p>\n<ul>\n<li>dependencies: Everything your code relies on to run, such as other libraries, external scripts or the data source<\/li>\n<li>output files: Everything that will be created by your program and should be tracked, this can be preprocessing results as well as final predictions<\/li>\n<li>A metric file to store KPIs on the action taken place.<\/li>\n<\/ul>\n<p>So the source code used in the example looks like this:<\/p>\n<pre class=\"lang:python decode:true\">def logDVC(model,param=dict(),metrics=dict(),features=None, tags=dict()):\r\n\r\n    # Imports\r\n\r\n    import json\r\n\r\n    from sklearn.externals import joblib\r\n\r\n    # Get some general information\r\n\r\n    type = model.__module__.split(\".\")[0]\r\n\r\n    # No option to set some tags to identify the experiment\r\n\r\n    # Save Model\r\n\r\n    if type==\"sklearn\":\r\n\r\n        _ = joblib.dump(model,\"tmp\/mymodel\")\r\n\r\n    if type==\"lgb\":\r\n\r\n        model.save_model(\"tmp\/mymodel\")\r\n\r\n    # Log metrics\r\n\r\n    with open('tmp\/metrics.txt', 'w') as f:\r\n\r\n        f.write(json.dumps(metrics))\r\n\r\n    # plot Feature importances if avaible\r\n\r\n    plotFeatureImportances(model, features, type)\r\n\r\n    # Create file about features\r\n\r\n    if features is not None:\r\n\r\n        with open(\"tmp\/features.txt\", \"w+\") as f:\r\n\r\n            f.write(\",\".join(features))\r\n\r\nif __name__ == \"__main__\":\r\n\r\n    from sklearn.model_selection import train_test_split\r\n\r\n    from sklearn.linear_model import Lasso\r\n\r\n    # We need to import utils here, since it is an own script and the execution environment has no access to the jupyter execution environment\r\n\r\n    from utils import *\r\n\r\n    # Do a train_test_split\r\n\r\n    data = getData()\r\n\r\n    x_train, x_test, y_train, y_test = train_test_split(data.iloc[:,:-1], data.iloc[:,-1], test_size=10, random_state=42)\r\n\r\n    # Define the details of our run\r\n\r\n    params=dict(alpha=0.4)\r\n\r\n    clf = Lasso(**params)\r\n\r\n    clf.fit(x_train, y_train)\r\n\r\n    predictions = clf.predict(x_test)\r\n\r\n    metrics = eval_metrics(y_test, predictions)\r\n\r\n    logDVC(clf,params,metrics,features=x_test.columns.values)<\/pre>\n<p>We load the data, train an sklearn regression on it and store interesting information.<\/p>\n<p>We will run the program like most other python programs but use the dvc run command in front to keep track of the results. Therefore we declare the three different types of objects we mentioned above:<\/p>\n<pre class=\"\">dvc run \\\r\n\r\n -d dvc_simple.py \\\r\n\r\n -f simple.dvc \\\r\n\r\n -o tmp\/featureimportance.png \\\r\n\r\n -M tmp\/metrics.txt \\\r\n\r\n -o tmp\/features.txt \\\r\n\r\n -o tmp\/mymodel\r\n\r\n python dvc_simple.py<\/pre>\n<p>The f-flag specifies the name of the file dvc will use to track this command. Within this file it will store a hash value for alle mentioned files, so that dvc can track, if any of those have changed. With the command:<\/p>\n<pre class=\"\">dvc repro simple.dvc<\/pre>\n<p>you can reproduce the same experiment. DVC will check if any dependencies have changed and if so re-run the experiment. It will even detect if a dependent file was also specified as the output of another DVC run and if so, check those dependencies. This way you can build pipelines of different runs.<\/p>\n<p>Using a command like<\/p>\n<pre class=\"\">dvc metrics show<\/pre>\n<p>you can check the results and compare them to an earlier run (but make sure you commited the state of the metrics file, since dvc will not do this out of the box for you)<\/p>\n<p>Furthermore it is possible to version control even big dependent files such as input data using dvc. Therefore you should configure a remote storage such as a cloud bucket and then use the git-like syntax to add and push the relevant files:<\/p>\n<pre class=\"\">dvc remote add myremote s3:\/\/bucket\/path\r\n\r\ndvc add data.xml\r\n\r\ndvc push<\/pre>\n<p>While DVC is great at reproducible experiments and sharing those with a team, it only provides little support to analyse the results. If you want to do ten runs using different settings, you will need to build some kind of comparison yourself. More complicated jobs with multiple dependencies and outputs make the cli quite verbose, but there are <a href=\"https:\/\/github.com\/iterative\/dvc\/issues\/995\">ideas to change this<\/a>. Until those are implmenented, a good trick is to place all dependencies within a directory and then just link the whole folder. It is nice to be able to mix multiple languages but if you only use e.g. python anyway, it feels weird to write all information you want to store into files. Finally this design also hinders a tighter integration with notebooks, which makes dvc less usable for\u00a0rapidly prototyping ideas.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"MLFlow\"><\/span>MLFlow<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>When I first heard about Databricks creating MLFlow, I expected a heavy-weight tool with a tight spark integration, but I was completely wrong about that. MLFlow provides an API for python, R and Java as well as REST calls\u2014of which I only tested the python version. It can be installed via pip and contains three sub-projects:<\/p>\n<ul>\n<li>Tracking: Function-calls for logging and a visualisation server (This is what we will focus on)<\/li>\n<li>Project: A definition of a project and dependencies to make them easily executable (we will only look at this briefly)<\/li>\n<li>Model:\u00a0 A standard for\u00a0model storage and deployment (exploring this is left to you)<\/li>\n<\/ul>\n<p>Let&#8217;s dive into the code. We&#8217;ll keep the main function similar and only use a new log-function:<\/p>\n<pre class=\"lang:default decode:true\">def logMlflow(model,data,output_folder=\"mlflow_out\", param=dict(),metrics=dict(),features=None, tags=dict(),run_name=None):\r\n\r\n    # Imports\r\n\r\n    from sklearn.externals import joblib\r\n\r\n    import mlflow\r\n\r\n    import os\r\n\r\n    if not os.path.exists(output_folder):\r\n\r\n        os.makedirs(output_folder)\r\n\r\n    # Get some general information\r\n\r\n    type = model.__module__.split(\".\")[0]\r\n\r\n    modelname = model.__class__.__name__\r\n\r\n    sha, remoteurl = getGitInfos()\r\n\r\n    # Start actual logging\r\n\r\n    mlflow.set_experiment(experiment_name=\"demo\")\r\n\r\n    if not run_name:\r\n\r\n        run_name = modelname\r\n\r\n    with mlflow.start_run(source_name=remoteurl,source_version=sha, run_name=run_name):\r\n\r\n        # Log Parameters\r\n\r\n        for k,v in param.items():\r\n\r\n            mlflow.log_param(k, v)\r\n\r\n        # Track dependencies\r\n\r\n        import pkg_resources\r\n\r\n        with open(\"{}\/dependencies.txt\".format(output_folder), \"w+\") as f:\r\n\r\n            for d in pkg_resources.working_set:\r\n\r\n                f.write(\"{}={}\\n\".format(d.project_name,d.version))\r\n\r\n        mlflow.log_artifact(\"{}\/dependencies.txt\".format(output_folder))\r\n\r\n        # Track data\r\n\r\n        data.to_csv(\"{}\/data\".format(output_folder))\r\n\r\n        mlflow.log_artifact(\"{}\/data\".format(output_folder))\r\n\r\n        if type==\"sklearn\":\r\n\r\n            _ = joblib.dump(model,\"{}\/sklearn\".format(output_folder))\r\n\r\n            mlflow.log_artifact(\"{}\/sklearn\".format(output_folder))\r\n\r\n        if type==\"lgb\":\r\n\r\n            model.save_model(\"{}\/lghtgbm.txt\".format(output_folder))\r\n\r\n            mlflow.log_artifact(\"{}\/lghtgbm.txt\".format(output_folder))\r\n\r\n        # Log metrics\r\n\r\n        for k,v in metrics.items():\r\n\r\n            mlflow.log_metric(k,v)\r\n\r\n        # plot Feature importances if avaible\r\n\r\n        featurePlot = plotFeatureImportances(model, features, type)\r\n\r\n        if featurePlot:\r\n\r\n            mlflow.log_artifact(\"{}.png\".format(featurePlot))\r\n\r\n        # Set some tags to identify the experiment\r\n\r\n        mlflow.set_tag(\"model\",modelname)\r\n\r\n        for tag, v in tags.items():\r\n\r\n            mlflow.set_tag(t,v)<\/pre>\n<p>Using a with-statement combined with <span class=\"lang:python decode:true crayon-inline\">mlflow.start_run<\/span>\u00a0you can create a new run if there isn&#8217;t one yet and log the results of the experiment to it. Therefore three different log functionalities exist: Parameters for model configuration, metrics for evaluation and artifacts, for all files worth storage, input as well as output.<\/p>\n<p>All this information is saved in a local folder named mlruns, which also is the information source for mlflows visualisation frontend. Running<\/p>\n<pre class=\"lang:default decode:true \">mlflow ui<\/pre>\n<p>will start the user interface that shows the different runs within an experiment as well as the parameters used and metrics created. You can also filter those based on parameters, metrics or tags. Furthermore it is possible to compare multiple runs in detail using a simple but effective scatter plot.<\/p>\n<p>MLFlow does not offer a simple reproduce command such as DVC to re-run a predefined setting. How ever its &#8222;project&#8220; component offers a way to increase standardisation. Therefore you can place a file called &#8222;MLProject&#8220; into the root of you folder which serves as an entrypoint to your project. It looks like this:<\/p>\n<pre class=\"lang:default decode:true\">name: tutorial\r\n\r\nconda_env: conda.yaml\r\n\r\nentry_points:\r\n\r\n  main:\r\n\r\n    parameters:\r\n\r\n      alpha: float\r\n\r\n      l1_ratio: {type: float, default: 0.1}\r\n\r\n    command: \"python train.py {alpha} {l1_ratio}\"<\/pre>\n<p>It references an environment file for dependencies and provides a main command with some modifiable parameters, in this case the <span class=\"lang:python decode:true crayon-inline \">alpha<\/span>\u00a0 and the <span class=\"lang:python decode:true crayon-inline \">l1_ratio<\/span>\u00a0 hyperparameter to control an ElasticNet model. Using the mlflow client you can then reproduce an experiment even without checking out its source code but directly accessing the github url:<\/p>\n<pre class=\"\">mlflow run git@github.com:inovex\/machine-learning-model-management.git -P alpha=0.4<\/pre>\n<p>In total, I found mlflow easy to understand due to its explicit logging functionality.\u00a0Its integration does need a few extra lines of code, but its visualisation and comparison features are worth it, even in a quick prototyping phase.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Sacred\"><\/span>Sacred<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Sacred was developed in the academic environment and introduced at <a href=\"https:\/\/www.youtube.com\/watch?v=qqg7RO0o1OE\">SciPy conference 2017<\/a>. It only provides a python interface and uses annotations to define which parts and metrics to track. This way users don&#8217;t need to define every parameter that should be tracked manually and can thus minimise overhead. Sacred does not support capturing information developed <a href=\"https:\/\/github.com\/IDSIA\/sacred\/issues\/100\">within a notebook<\/a> since the interactive nature of such notebooks makes code less reproducible, which is the primary goal of the framework. While you can test your Sacred code within a notebook, it does not really track the information but only checks the syntax.\u00a0For this reason we again use the <span class=\"lang:python decode:true crayon-inline \">write_file<\/span>\u00a0 magic of Jupyter to create a script from within the notebook and call it afterwards.<\/p>\n<p>In the <a href=\"https:\/\/github.com\/inovex\/machine-learning-model-management\">notebook<\/a> you will find two versions of similar code to track experiments with Sacred. The first one is similar to the mlflow tracking code and uses dedicated logging functions. The second version (displayed below) is probably closer to the Sacred-way developers intended and tries to use annotations and special Sacred features.<\/p>\n<pre class=\"lang:default decode:true\">#!\/usr\/bin\/env python\r\n\r\nfrom __future__ import division, print_function, unicode_literals\r\n\r\nfrom sacred import Experiment\r\n\r\n# Imports need to be done in the beginning of the file, since sacred won't recognize them, if they occur within a function\r\n\r\nfrom sklearn.externals import joblib\r\n\r\nex = Experiment('Boston Housing Prices')\r\n\r\nfrom utils import *\r\n\r\n@ex.capture\r\n\r\ndef capturestuff(_seed):\r\n\r\n    print(_seed)\r\n\r\ndef getData():\r\n\r\n    from sklearn.datasets import load_boston\r\n\r\n    boston = load_boston()\r\n\r\n    data = pd.DataFrame(boston.data,columns=boston.feature_names)\r\n\r\n    data['target'] = pd.Series(boston.target)\r\n\r\n    return data\r\n\r\n@ex.config\r\n\r\ndef cfg(_log):\r\n\r\n    alpha= 0.5\r\n\r\ndef logSacred(run,model,data,output_folder=\"sacred_out\", param=dict(),metrics=dict(),features=None, tags=dict()):\r\n\r\n    # Get some general information\r\n\r\n    import os\r\n\r\n    if not os.path.exists(output_folder):\r\n\r\n        os.makedirs(output_folder)\r\n\r\n    type = model.__module__.split(\".\")[0]\r\n\r\n    modelname = model.__class__.__name__\r\n\r\n    # Config will be tracked automatically\r\n\r\n    # Dependencies will also be tracked automatically\r\n\r\n    # Track source code\r\n\r\n    data.to_csv(\"{}\/data\".format(output_folder))\r\n\r\n    #ex.add_source_file(\"{}\/data\".format(output_folder))\r\n\r\n    ab = ex.open_resource(\"{}\/data\".format(output_folder))\r\n\r\n    # Create file about features\r\n\r\n    if features is not None:\r\n\r\n        with open(\"{}\/features.txt\".format(output_folder), \"w+\") as f:\r\n\r\n            f.write(\",\".join(features))\r\n\r\n        ex.add_artifact(\"{}\/features.txt\".format(output_folder))\r\n\r\n    # plot Feature importances if avaible\r\n\r\n    if plotFeatureImportances(model, features, type):\r\n\r\n        ex.add_artifact(\"{}\/featureimportance.png\".format(output_folder))\r\n\r\n    # Track Model binary\r\n\r\n    if type==\"sklearn\":\r\n\r\n        _ = joblib.dump(model,\"{}\/sklearn\".format(output_folder))\r\n\r\n        ex.add_artifact(\"{}\/sklearn\".format(output_folder))\r\n\r\n    if type==\"lgb\":\r\n\r\n        model.save_model(\"{}\/lghtgbm.txt\".format(output_folder))\r\n\r\n        ex.add_artifact(\"{}\/lghtgbm.txt\".format(output_folder))\r\n\r\n    # Log metrics\r\n\r\n    for k,v in metrics.items():\r\n\r\n        ex.log_scalar(k,v)\r\n\r\n    # Set some tags to identify the experiment\r\n\r\n    for tag, v in tags.items():\r\n\r\n        ex.add.set_tag(t,v)\r\n\r\n@ex.automain\r\n\r\ndef run(_run, alpha):\r\n\r\n    # Setup\r\n\r\n    from sklearn.model_selection import train_test_split\r\n\r\n    from sklearn.linear_model import Lasso\r\n\r\n    from sklearn.metrics import mean_absolute_error\r\n\r\n    # Do a train_test_split on my Data\r\n\r\n    data = getData()\r\n\r\n    x_train, x_test, y_train, y_test = train_test_split(data.iloc[:,:-1], data.iloc[:,-1], test_size=10, random_state=42)\r\n\r\n    # Define my params\r\n\r\n    params=dict(alpha=alpha)\r\n\r\n    clf = Lasso(**params)\r\n\r\n    clf.fit(x_train, y_train)\r\n\r\n    predictions = clf.predict(x_test)\r\n\r\n    metrics = eval_metrics(y_test, predictions)\r\n\r\n    logSacred(_run,clf,data,param=params,metrics=metrics,features=x_test.columns.values)<\/pre>\n<p>Maybe it is just because I use Sacred for the first time and already wrote the code for the two other frameworks, but using the Sacred annotations did not feel very natural. And while the Sacred magic of auto-collecting important information seems compelling at first, it is far from perfect and I found myself frequently controlling whether my dependencies and files got tracked properly. For example Sacred will track dependencies if you import them at the top of your script, but not if you do this within a function. It also provides some commands such as <span class=\"lang:python decode:true crayon-inline \">open_resource<\/span>\u00a0 to read files and track them at the same time, but you will need to remember those special commands and if your data is not stored in a file but received from an API for example you will need to log the dependency explicitly nevertheless. To me, the Sacred way of annotating the code was more of a burden than a relieve compared to explicit logging.<\/p>\n<p>That said, Sacred probably is a good candidate if you want to create your own framework for your organisation, since sacred uses a modular design approach. Its\u00a0API provides many functions and hooks to create custom wrappers of your own. While the number of options will\u00a0overwhelm normal developers that just want to keep track of some libraries and dependencies, it will give you the power to shape the tool according to your needs if you want to do so. Not only can its functionality be extended in multiple ways, Sacred also supports different storage backends like file storage,\u00a0MongoDB, TinyDB, SQL and others. The examples in the notebook use MongoDB since this is the only one supported by the existing <a href=\"https:\/\/github.com\/IDSIA\/sacred#omniboard\">visualisation tools<\/a>, which are external components as well.<\/p>\n<p>While none of the three listed visualisation tools offers as much functionality as mlflow, the different backends should make it easy to create your own visualisation as well. The screenshots below show omniboard which allows you to create new columns for metrics logged and supports ordering, but lacks a comparison of different runs in detail.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15393 size-large\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-1024x552.png\" alt=\"\" width=\"1024\" height=\"552\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-1024x552.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-300x162.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-768x414.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-1536x828.png 1536w, https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-2048x1104.png 2048w, https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-1920x1035.png 1920w, https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-400x216.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/02\/Screenshot-2019-02-02-at-21.17.04-360x194.png 360w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The three different frameworks leverage different purposes. While DVC targets universal applicability and prioritises reproducibility over ease-of-use, mlflow is a nice and convenient tool to track your parameters and compare model performance while prototyping. Sacred&#8217;s use case is somewhat limited, it only provides a python interface, does not support usage within notebooks and does not offer a simple command to reproduce an experiment. Nevertheless it shines with its open standards and extensibility.<\/p>\n<table width=\"906\">\n<tbody>\n<tr>\n<td width=\"319\"><\/td>\n<td width=\"196\"><b>DVC<\/b><\/td>\n<td width=\"196\"><b>mlFlow<\/b><\/td>\n<td width=\"196\"><b>Sacred<\/b><\/td>\n<\/tr>\n<tr>\n<td width=\"319\">Language Support<\/td>\n<td width=\"196\">Universal using Shell<\/td>\n<td width=\"196\">Python, R, Java, REST<\/td>\n<td width=\"196\">Python<\/td>\n<\/tr>\n<tr>\n<td width=\"319\">Control<\/td>\n<td width=\"196\">DAG using\u00a0Scripts<\/td>\n<td width=\"196\">Python Package<\/td>\n<td width=\"196\">Annotations<\/td>\n<\/tr>\n<tr>\n<td width=\"319\">Storage Backend<\/td>\n<td width=\"196\">File storage<\/td>\n<td width=\"196\">File storage<\/td>\n<td width=\"196\">MongoDB + others<\/td>\n<\/tr>\n<tr>\n<td width=\"319\">Github Stars<\/td>\n<td width=\"196\">2400<\/td>\n<td width=\"196\">3400<\/td>\n<td width=\"196\">1700<\/td>\n<\/tr>\n<tr>\n<td width=\"319\">License<\/td>\n<td width=\"196\">Apache 2<\/td>\n<td width=\"196\">Apache 2<\/td>\n<td width=\"196\">MIT<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>(last updated 4.4.2019)<\/p>\n<p>If you are looking for a tool that gives you an easy way to log important attributes and helps you organise your models while prototyping, go for mlflow.<\/p>\n<p>If you work in a team and want to make your experiments absolutely reproducible even if it takes some extra steps: check out DVC.<\/p>\n<p>Delve into Sacred if you want to build your own tool and profit from previous work, or if the others don&#8217;t offer you enough flexibility.<\/p>\n<table width=\"1053\">\n<tbody>\n<tr>\n<td style=\"width: 419px;\"><b>Features<\/b><\/td>\n<td style=\"width: 201px;\"><b>DVC<\/b><\/td>\n<td style=\"width: 204px;\"><b>mlFlow<\/b><\/td>\n<td style=\"width: 201px;\"><b>Sacred<\/b><\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Notebook support<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<td style=\"width: 204px;\">Yes<\/td>\n<td style=\"width: 201px;\">Only Dry-Run<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">model storage<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<td style=\"width: 204px;\">As Artefacts<\/td>\n<td style=\"width: 201px;\">Only Artefacts<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Input data storage<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<td style=\"width: 204px;\">Only Artefacts<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Visualisation<\/td>\n<td style=\"width: 201px;\">No<\/td>\n<td style=\"width: 204px;\">Yes<\/td>\n<td style=\"width: 201px;\">Yes (External)<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Parameter Tracking<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<td style=\"width: 204px;\">Yes<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Performance Metrics<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<td style=\"width: 204px;\">Yes<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Execution statistics<\/td>\n<td style=\"width: 201px;\">Little<\/td>\n<td style=\"width: 204px;\">Little<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Source Code Tracking<\/td>\n<td style=\"width: 201px;\">Via Git<\/td>\n<td style=\"width: 204px;\">Manually (Git Hashes)<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 419px;\">Dependencies<\/td>\n<td style=\"width: 201px;\">Manually (Env-File)<\/td>\n<td style=\"width: 204px;\">Manually\u00a0(Env-File)<\/td>\n<td style=\"width: 201px;\">Yes<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2><span class=\"ez-toc-section\" id=\"Read-on\"><\/span>Read on<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Find our complete <a href=\"https:\/\/www.inovex.de\/de\/leistungen\/data-science-deep-learning\/\">Data Science portfolio<\/a> on our website or have a look at our job openings and consider joining us as a data scientist!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In my previous blog post &#8222;how to manage machine learning models&#8220;\u00a0I explained the difficulties within the process of developing a good machine learning model and motivated using a tool to support data scientists with this challenge. This blog post will follow up by comparing three different tools developed to support reproducible machine learning model development:<\/p>\n","protected":false},"author":75,"featured_media":16066,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[206,226],"service":[431],"coauthors":[{"id":75,"display_name":"Nico Kreiling","user_nicename":"nkreiling"}],"class_list":["post-14673","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-data-science","tag-model-management","service-data-science"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Frameworks for Machine Learning Model Management - inovex GmbH<\/title>\n<meta name=\"description\" content=\"This blog post will compare three different tools developed to support reproducible machine learning model development: MLFlow developed by DataBricks (the company behind Apache Spark), DVC, a software product of the London based startup iterative.ai, and Sacred, an academic project developed by different researchers.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Frameworks for Machine Learning Model Management - inovex GmbH\" \/>\n<meta property=\"og:description\" content=\"This blog post will compare three different tools developed to support reproducible machine learning model development: MLFlow developed by DataBricks (the company behind Apache Spark), DVC, a software product of the London based startup iterative.ai, and Sacred, an academic project developed by different researchers.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2019-04-03T10:27:26+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-17T06:59:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Nico Kreiling\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1-1024x576.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nico Kreiling\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"15\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Nico Kreiling\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/\"},\"author\":{\"name\":\"Nico Kreiling\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/445f04b88618788ea0fd469ab9363ce6\"},\"headline\":\"Frameworks for Machine Learning Model Management\",\"datePublished\":\"2019-04-03T10:27:26+00:00\",\"dateModified\":\"2026-03-17T06:59:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/\"},\"wordCount\":1975,\"commentCount\":5,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2019\\\/04\\\/Machine-Learning-Management-1.png\",\"keywords\":[\"Data Science\",\"Model Management\"],\"articleSection\":[\"Analytics\",\"English Content\",\"General\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/\",\"name\":\"Frameworks for Machine Learning Model Management - inovex GmbH\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2019\\\/04\\\/Machine-Learning-Management-1.png\",\"datePublished\":\"2019-04-03T10:27:26+00:00\",\"dateModified\":\"2026-03-17T06:59:21+00:00\",\"description\":\"This blog post will compare three different tools developed to support reproducible machine learning model development: MLFlow developed by DataBricks (the company behind Apache Spark), DVC, a software product of the London based startup iterative.ai, and Sacred, an academic project developed by different researchers.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2019\\\/04\\\/Machine-Learning-Management-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2019\\\/04\\\/Machine-Learning-Management-1.png\",\"width\":1920,\"height\":1080,\"caption\":\"Three Cards machine learning model management tool logos\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/machine-learning-model-management\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Frameworks for Machine Learning Model Management\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/445f04b88618788ea0fd469ab9363ce6\",\"name\":\"Nico Kreiling\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/668c761da41e2273f8b8c1ff2abd8c941ee173645c87c6e9f992e0912baa5549?s=96&d=retro&r=g23148f0bf8e343c0425edf8bd9582674\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/668c761da41e2273f8b8c1ff2abd8c941ee173645c87c6e9f992e0912baa5549?s=96&d=retro&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/668c761da41e2273f8b8c1ff2abd8c941ee173645c87c6e9f992e0912baa5549?s=96&d=retro&r=g\",\"caption\":\"Nico Kreiling\"},\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/author\\\/nkreiling\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Frameworks for Machine Learning Model Management - inovex GmbH","description":"This blog post will compare three different tools developed to support reproducible machine learning model development: MLFlow developed by DataBricks (the company behind Apache Spark), DVC, a software product of the London based startup iterative.ai, and Sacred, an academic project developed by different researchers.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/","og_locale":"de_DE","og_type":"article","og_title":"Frameworks for Machine Learning Model Management - inovex GmbH","og_description":"This blog post will compare three different tools developed to support reproducible machine learning model development: MLFlow developed by DataBricks (the company behind Apache Spark), DVC, a software product of the London based startup iterative.ai, and Sacred, an academic project developed by different researchers.","og_url":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2019-04-03T10:27:26+00:00","article_modified_time":"2026-03-17T06:59:21+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1.png","type":"image\/png"}],"author":"Nico Kreiling","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1-1024x576.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Nico Kreiling","Gesch\u00e4tzte Lesezeit":"15\u00a0Minuten","Written by":"Nico Kreiling"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/"},"author":{"name":"Nico Kreiling","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/445f04b88618788ea0fd469ab9363ce6"},"headline":"Frameworks for Machine Learning Model Management","datePublished":"2019-04-03T10:27:26+00:00","dateModified":"2026-03-17T06:59:21+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/"},"wordCount":1975,"commentCount":5,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1.png","keywords":["Data Science","Model Management"],"articleSection":["Analytics","English Content","General"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/","url":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/","name":"Frameworks for Machine Learning Model Management - inovex GmbH","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1.png","datePublished":"2019-04-03T10:27:26+00:00","dateModified":"2026-03-17T06:59:21+00:00","description":"This blog post will compare three different tools developed to support reproducible machine learning model development: MLFlow developed by DataBricks (the company behind Apache Spark), DVC, a software product of the London based startup iterative.ai, and Sacred, an academic project developed by different researchers.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2019\/04\/Machine-Learning-Management-1.png","width":1920,"height":1080,"caption":"Three Cards machine learning model management tool logos"},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/machine-learning-model-management\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"Frameworks for Machine Learning Model Management"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/445f04b88618788ea0fd469ab9363ce6","name":"Nico Kreiling","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/secure.gravatar.com\/avatar\/668c761da41e2273f8b8c1ff2abd8c941ee173645c87c6e9f992e0912baa5549?s=96&d=retro&r=g23148f0bf8e343c0425edf8bd9582674","url":"https:\/\/secure.gravatar.com\/avatar\/668c761da41e2273f8b8c1ff2abd8c941ee173645c87c6e9f992e0912baa5549?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/668c761da41e2273f8b8c1ff2abd8c941ee173645c87c6e9f992e0912baa5549?s=96&d=retro&r=g","caption":"Nico Kreiling"},"url":"https:\/\/www.inovex.de\/de\/blog\/author\/nkreiling\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/14673","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/75"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=14673"}],"version-history":[{"count":3,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/14673\/revisions"}],"predecessor-version":[{"id":66501,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/14673\/revisions\/66501"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/16066"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=14673"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=14673"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=14673"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=14673"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}