{"id":50743,"date":"2024-02-05T12:44:50","date_gmt":"2024-02-05T11:44:50","guid":{"rendered":"https:\/\/www.inovex.de\/?p=50743"},"modified":"2024-02-13T10:04:29","modified_gmt":"2024-02-13T09:04:29","slug":"orchestrating-snowpark-scripts-with-apache-airflow","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/","title":{"rendered":"Orchestrating Snowpark Scripts With Apache Airflow"},"content":{"rendered":"<p>The following blog article describes one possible way to automate and orchestrate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow. I will show some ideas, concepts, and code snippets that we implemented at a project with one of our customers dmTECH. All the examples in this article are based on a simple \u201cHello World&#8220; example.<!--more--><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_79_2 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Project-context\" >Project context<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Motivation\" >Motivation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Implementation\" >Implementation<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Install-the-Snowflake-connector\" >Install the Snowflake connector<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Configure-a-Snowflake-connection\" >Configure a Snowflake connection<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Wrap-your-code-into-a-stored-procedure\" >Wrap your code into a stored procedure<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Write-your-DAG\" >Write your DAG<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Monitoring-and-logging\" >Monitoring and logging<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Run-your-Airflow-behind-a-proxy-server\" >Run your Airflow behind a proxy server<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#Final-thoughts\" >Final thoughts<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Project-context\"><\/span>Project context<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>dmTECH has been using Snowflake successfully for some time to migrate its existing data warehouse to a modern cloud-based solution. We support a team in another department with the implementation of various analytical use cases mainly on an on-premise Hadoop cluster in PySpark. Now we wanted to migrate one of the existing analytics use cases to Snowpark as a proof of concept to get to know the technology and its potential better. This blog article is mainly about the integration of Snowflake into our existing automation tooling.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Motivation\"><\/span>Motivation<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>We already showed in a <a href=\"https:\/\/www.inovex.de\/de\/blog\/datenpipeline-mit-streams-und-tasks-in-snowflake\/\">previous article<\/a> how you can use <a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/tasks-intro\">tasks<\/a> to easily automate Snowflake scripts. So why should we use an external tool like Airflow at all?<\/p>\n<p>First of all, Airflow is a specialized orchestration tool that has much richer functionalities to build the sequence of tasks and their dependencies (we call this a DAG \u2013 Directed acyclic graph). For example, you can use so-called <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow\/stable\/core-concepts\/dags.html#trigger-rules\">trigger rules<\/a> to define under which circumstances a task should be triggered, depending on its predecessor tasks. Many more powerful features allow you to build complex DAG logic, which is currently missing in Snowflake.<\/p>\n<p>Furthermore, Airflow&#8217;s web frontend is more suited for operational aspects like browsing and filtering historical runs, rerunning single tasks or a chain of tasks, browsing the logs, and so on. There are basic filter and sorting controls in Snowflakes web UI (a.k.a. Snowsight), but not as powerful as in Airflow.<\/p>\n<p>The most important argument is that we use Airflow in the current project for almost every automated process, regardless of the technology the tasks run on. That means it is technology-agnostic and can schedule processes on almost every runtime technology you can think of. Thus, we do not have two separate places where we automate things but have it in one central place. Together with the fact that it&#8217;s probably the scheduling tool with the largest community, this makes it easier for the operation of all the processes in the first-level support, because they do not need a Snowflake account and Snowflake knowledge to restart a process from Airflow. Furthermore, at the customer company in this project, we have multiple other data processing and storage systems we want to integrate with, for example, an on-prem Hadoop system where we run various Spark jobs. Okay, we could migrate all those from Hadoop to Snowflake \u2013 but that is nothing you do in a few days or weeks.<\/p>\n<p>Those are the reasons why we decided to use Airflow for automation over the Snowflake tasks.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Implementation\"><\/span>Implementation<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In the following subchapters, I want to show you which steps are necessary to integrate an Airflow installation with your Snowflake account and how to schedule stored procedures on it.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Install-the-Snowflake-connector\"><\/span>Install the Snowflake connector<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The first step is to install the <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow-providers-snowflake\/stable\/index.html\">Snowflake provider<\/a> on your Airflow machine. This can simply be done with the following command on your Airflow host:<\/p>\n<pre>export SNOWFLAKE_PROVIDER_VERSION=\"5.0.0\"\r\nexport AIRFLOW_VERSION=\"2.7.3\"\r\nexport PYTHON_VERSION=\"3.8\"\r\n\r\npip install \"apache-airflow-providers-snowflake==${SNOWFLAKE_PROVIDER_VERSION}\" --constraint \/tmp\/airflow_${AIRFLOW_VERSION}_py${PYTHON_VERSION}_constraints.txt<\/pre>\n<p>You have to set the three mentioned environment variables depending on your needs and your given environment.<\/p>\n<p>At the customer project, we are very lucky and have a platform team that provides the Airflow machine as a service. They put Airflow-specific <a href=\"https:\/\/pip.pypa.io\/en\/stable\/user_guide\/#constraints-files\">pip constraint files <\/a>on the filesystem of the remote machine so that you can use them to install further dependencies on your own without breaking the dependency tree. That\u2019s done with the additional &#8211;constraint parameter.<\/p>\n<p>Please note that in reality we do not simply SSH to our machine and manually install dependencies on the CLI. In fact, we have an automated process in place that runs all the provisioning steps in a reproducible CI pipeline.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Configure-a-Snowflake-connection\"><\/span>Configure a Snowflake connection<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>After the installation of the Snowflake provider is done we can add and configure a Snowflake connection. This can either be done from the web frontend or the Airflow CLI. This is also something we automated in provisioning steps in a CI pipeline, where we use the following CLI command:<\/p>\n<pre>airflow connections add \\\r\n  --conn-type snowflake \\\r\n\u00a0\u00a0--conn-schema SBX_JUSEITHER \\\r\n\u00a0\u00a0--conn-login \"$SNOWFLAKE_SERVICE_USERNAME\" \\\r\n\u00a0\u00a0--conn-password \"$SNOWFLAKE_SERVICE_USER_PASSWORD\" \\\r\n\u00a0\u00a0--conn-extra \"{\\\"account\\\": \\\"$ACCOUNT_NAME\\\", \\\"warehouse\\\": \\\"$SNOWFLAKE_DEFAULT_WAREHOUSE\\\", \\\"database\\\": \\\"$SNOWFLAKE_DATABASE\\\", \\\"role\\\": \\\"$SNOWFLAKE_ROLE\\\"}\" \\\r\n\u00a0\u00a0snowflake<\/pre>\n<p>Again, all those variables need to be set before. In our case, they are injected from Gitlab as CI variables.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Wrap-your-code-into-a-stored-procedure\"><\/span>Wrap your code into a stored procedure<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>To call a Snowflake script \u2013 either written in SQL or any Snowpark-supported language (in our case Python) \u2013 you need to encapsulate it in a <a href=\"https:\/\/docs.snowflake.com\/en\/developer-guide\/stored-procedure\/stored-procedures-overview\">stored procedure<\/a>. This is usually done with an SQL statement like this:<\/p>\n<pre>CREATE OR REPLACE PROCEDURE DEV.SBX_JUSEITHER.HELLO_WORLD(name string)\r\nRETURNS VARCHAR(16777216)\r\nLANGUAGE PYTHON\r\nRUNTIME_VERSION = '3.8'\r\nPACKAGES = ('snowflake-snowpark-python')\r\nHANDLER = 'main'\r\nEXECUTE AS OWNER\r\nAS '#This is a simple Hello World script\r\nimport snowflake.snowpark as snowpark\r\ndef main(session: snowpark.Session, name: str):\u00a0\u00a0\u00a0\r\n  return f\"Hello {name}\"';<\/pre>\n<p>As you can see, the Python logic is implemented inside the procedure definition itself. This may work for very small amounts of code but does not make much sense for larger or more complex code. What you usually want to do is deploy the Python files separately on a Snowflake stage and reference those files from your procedure definition.<\/p>\n<p>So let\u2019s first create a stage and make sure you have access to it (we assume you use the role \u201cDEV_DEVELOPER\u201c in this example):<\/p>\n<pre>USE ROLE SYSADMIN;\r\n\r\nCREATE STAGE DEV.SBX_JUSEITHER.DEPLOY\u00a0\r\nDIRECTORY = (ENABLE=TRUE);\r\n\r\nGRANT ALL PRIVILEGES ON STAGE DEV.SBX_JUSEITHER.DEPLOY TO ROLE DEV_DEVELOPER;<\/pre>\n<p>The DIRECTORY\u00a0 part of the above statement ensures that you can view folders and files on your stage in Snowsight.<\/p>\n<p>Next, let&#8217;s create a Python file with the following content and save it as hello_world.py:<\/p>\n<pre># This is a simple Hello World script\r\n\r\nimport snowflake.snowpark as snowpark\r\n\r\ndef main(session: snowpark.Session, name: str):\u00a0\u00a0\u00a0\r\n\u00a0\u00a0\u00a0\u00a0return f\"Hello {name}\u201c<\/pre>\n<p>Now upload the Python file to the previously created stage. This can be done with <a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/snowsql\">SnowSQL<\/a>, the Snowflake CLI tools:<\/p>\n<pre>export WORKING_DIR=$(pwd)\r\n\r\nsnowsql \\\r\n  --accountname $ACCOUNT_NAME \\\r\n  --dbname $DB_NAME \\\r\n  --schemaname $SCHEMA_NAME \\\r\n  --rolename $ROLE_NAME \\\r\n  --warehouse $WAREHOUSE \\\r\n  --username $USERNAME \\\r\n  --query \"PUT file:\/\/\/${WORKING_DIR}\/hello_world.py @DEV.SBX_JUSEITHER.DEPLOY\/hello_world AUTO_COMPRESS=FALSE OVERWRITE=TRUE<\/pre>\n<p>After that you can see in Snowsight that the file was placed on the stage:<\/p>\n<figure id=\"attachment_50760\" aria-describedby=\"caption-attachment-50760\" style=\"width: 1063px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-50760\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow.png\" alt=\"\" width=\"1063\" height=\"393\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow.png 2320w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-300x111.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-1024x379.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-768x284.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-1536x568.png 1536w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-2048x757.png 2048w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-1920x710.png 1920w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-400x148.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/deploy_stage_snowflake_airflow-360x133.png 360w\" sizes=\"auto, (max-width: 1063px) 100vw, 1063px\" \/><figcaption id=\"caption-attachment-50760\" class=\"wp-caption-text\">The Python file is deployed on the stage. Preview with the directory feature.<\/figcaption><\/figure>\n<p>We can now adapt the definition of our stored procedure to use a reference to the Python file instead of the inline code:<\/p>\n<pre>CREATE OR REPLACE PROCEDURE DEV.SBX_JUSEITHER.HELLO_WORLD(name string)\r\nRETURNS VARCHAR(16777216)\r\nLANGUAGE PYTHON\r\nRUNTIME_VERSION = '3.8'\r\nPACKAGES = ('snowflake-snowpark-python')\r\nIMPORTS = (\u2018@DEV.SBX_JUSEITHER.DEPLOY\/hello_world\/hello_world.py\u2019)\r\nHANDLER = 'hello_world.main'\r\nEXECUTE AS OWNER;<\/pre>\n<p>The necessary changes are colored. In the IMPORTS part, you tell the procedure to import the hello_world.py file from your stage. Please note that you need to adapt the HANDLER definition to be prefixed with the name of the corresponding Python module. The code definition in the AS part can simply be skipped.<\/p>\n<p>Again, this is all wrapped in CI pipelines in reality.<\/p>\n<p>You can test your procedure with the following command:<\/p>\n<pre>CALL DEV.SBX_JUSEITHER.HELLO_WORLD(name =&gt; \u2018inovex\u2019);<\/pre>\n<p>As you can see on the following screenshot, the return value is shown as result of this query:<\/p>\n<figure id=\"attachment_50758\" aria-describedby=\"caption-attachment-50758\" style=\"width: 1049px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-50758 \" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow.png\" alt=\"\" width=\"1049\" height=\"274\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow.png 1554w, https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow-300x78.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow-1024x268.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow-768x201.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow-1536x401.png 1536w, https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow-400x105.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/call_procedure_snowflake_airflow-360x94.png 360w\" sizes=\"auto, (max-width: 1049px) 100vw, 1049px\" \/><figcaption id=\"caption-attachment-50758\" class=\"wp-caption-text\">The output of the procedure call. Useful for quick debugging in Snowsight.<\/figcaption><\/figure>\n<h3><span class=\"ez-toc-section\" id=\"Write-your-DAG\"><\/span>Write your DAG<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Now that your code is packed into a stored procedure, and Airflow can connect to your Snowflake account, we can prepare a DAG which runs periodically and simply calls it. The Snowflake Airflow provider mentioned above has a special operator for running arbitrary SQL code. At the time of writing, according to the docs, this was <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow-providers-snowflake\/stable\/_api\/airflow\/providers\/snowflake\/operators\/snowflake\/index.html#airflow.providers.snowflake.operators.snowflake.SnowflakeOperator\">already marked as deprecated<\/a>. The suggested alternative is to use the generic <a href=\"https:\/\/airflow.apache.org\/docs\/apache-airflow-providers-common-sql\/stable\/operators.html#execute-sql-query\">SQLExecuteQueryOperator<\/a> .<\/p>\n<p>The task definition in our DAG looks like this:<\/p>\n<pre>hello_world_snowflake = SQLExecuteQueryOperator(\r\n  dag=dag,\r\n\u00a0 task_id=\"hello_world_snowflake\",\r\n\u00a0\u00a0sql='CALL DEV.SBX_JUSEITHER.DEPLOY(name =&gt; {{ run_id }})',\r\n\u00a0\u00a0conn_id=\"snowflake\",\r\n\u00a0\u00a0show_return_value_in_logs=True,\r\n)<\/pre>\n<p>Here are a few things to note:<\/p>\n<ul>\n<li>You can use any pre-defined Airflow macros in your SQL code, in the above example I put the run ID of the current DAG run as a name into the procedure call<\/li>\n<li>The parameter conn_id references the Snowflake connection that we created a few steps before<\/li>\n<li>show_return_value_in_logs can be used to print the output of the operator to the Airflow logs. Caution: This should only be used for small amounts of data, for example, while debugging\/developing<\/li>\n<\/ul>\n<h2><span class=\"ez-toc-section\" id=\"Monitoring-and-logging\"><\/span>Monitoring and logging<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>When it comes to automating production jobs, one important aspect is monitoring the running scripts and finding logs that let you investigate errors.<\/p>\n<p>The monitoring part is at least at a basic level covered by Airflow itself, because in its web UI you have a good overview of the tasks that ran and the corresponding logs, status, and so on. Many of us have experience in the operation of Airflow DAGs and there is a lot of material out there covering this topic in detail. So I will not dive deeper into it. But what can you do in addition from Snowflake side?<\/p>\n<p>First, you can simply use the query history in Snowsight to get a feeling of the runtime of your procedure, the single steps that are executed (Tab \u201cQuery Profile\u201c), and possible failures:<\/p>\n<figure id=\"attachment_50762\" aria-describedby=\"caption-attachment-50762\" style=\"width: 805px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-50762\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow.png\" alt=\"\" width=\"805\" height=\"519\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow.png 2320w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-300x193.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-1024x660.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-768x495.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-1536x990.png 1536w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-2048x1321.png 2048w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-1920x1238.png 1920w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-400x258.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/query_history_snowflake_airflow-360x232.png 360w\" sizes=\"auto, (max-width: 805px) 100vw, 805px\" \/><figcaption id=\"caption-attachment-50762\" class=\"wp-caption-text\">Show query details and profile of your procedure call<\/figcaption><\/figure>\n<p>Second, you can add <a href=\"https:\/\/docs.snowflake.com\/en\/developer-guide\/logging-tracing\/logging-python\">logging to your Python script <\/a>and find the log messages in the so-called event table. The event table is a special table in your account (you can define exactly one) that stores all logs and traces from your applications. Please note that you need to <a href=\"https:\/\/docs.snowflake.com\/en\/developer-guide\/logging-tracing\/event-table-setting-up\">create and configure the event table initially<\/a>.<\/p>\n<p>To add logging to your code, adapt the Python script as follows:<\/p>\n<pre># This is a simple Hello World script\r\nimport snowflake.snowpark as snowpark\r\nimport logging\r\n\r\nlogger = logging.getLogger(\"hello_world\")\r\n\r\ndef main(session: snowpark.Session, name: str):\u00a0\u00a0\r\n\u00a0  logger.warn(f\"Hello {name}\")\r\n\u00a0\u00a0\u00a0return f\"Hello {name}\"<\/pre>\n<p>As you can see, we simply use the Python standard logging mechanism.<\/p>\n<p>After upgrading the procedure with the new code and running it again, you can find your logs in the event table for example with the following query:<\/p>\n<pre class=\"lang:tsql decode:true\">SELECT timestamp, value FROM COMMON.COMMON.EVENT_TABLE\r\nWHERE resource_attributes['snow.schema.name'] = 'SBX_JUSEITHER'\r\nAND record['severity_text'] = 'WARN'\r\nORDER BY TIMESTAMP DESC;<\/pre>\n<figure id=\"attachment_50764\" aria-describedby=\"caption-attachment-50764\" style=\"width: 786px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-50764\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Bildschirmfoto-2024-01-08-um-07.37.08.png\" alt=\"\" width=\"786\" height=\"361\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Bildschirmfoto-2024-01-08-um-07.37.08.png 1258w, https:\/\/www.inovex.de\/wp-content\/uploads\/Bildschirmfoto-2024-01-08-um-07.37.08-300x138.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/Bildschirmfoto-2024-01-08-um-07.37.08-1024x470.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/Bildschirmfoto-2024-01-08-um-07.37.08-768x353.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/Bildschirmfoto-2024-01-08-um-07.37.08-400x184.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/Bildschirmfoto-2024-01-08-um-07.37.08-360x165.png 360w\" sizes=\"auto, (max-width: 786px) 100vw, 786px\" \/><figcaption id=\"caption-attachment-50764\" class=\"wp-caption-text\">Query the application logs right from Snowsight<\/figcaption><\/figure>\n<h2><span class=\"ez-toc-section\" id=\"Run-your-Airflow-behind-a-proxy-server\"><\/span>Run your Airflow behind a proxy server<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Especially in a larger corporate context, servers often run behind a proxy server for security reasons. To route the Snowflake connection through the proxy, you can simply set the following environment variables inside your DAG definition:<\/p>\n<pre>import os\r\nos.environ[\"HTTP_PROXY\"] = \"http:\/\/your.proxy.host.com:8000\"\r\nos.environ[\"HTTPS_PROXY\"] = \"http:\/\/your.proxy.host.com:8000\"<\/pre>\n<p>Of cause you have to adapt the hostname and port to your actual values.<\/p>\n<p>The proxy brings one more challenge into the game: Snowflake uses <a href=\"https:\/\/docs.snowflake.com\/en\/user-guide\/ocsp\">OCSP<\/a> for certificate validation by default. You could disable it in the client (for example in your Airflow Snowflake connection definition), but it&#8217;s strongly recommended to leave it enabled for security reasons. Depending on the configuration of your proxy, there is a chance that it will intercept the SSL certificate chain (also known as SSL termination). This will break the OCSP communication between your Airflow server and the OCSP responder. Your network administrators need to bypass the Snowflake hostnames on the proxy, otherwise, you will get OCSP errors (or even worse: depending on the mode, it won&#8217;t work at all).<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Final-thoughts\"><\/span>Final thoughts<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>I showed multiple steps on how we use Airflow to automate Snowpark Python scripts in one of our customer projects at dmTECH. Please note that there might be other options that could better suit your requirements. I want to mention especially the Snowflake internal way in the form of tasks. At the time of writing Snowflake has a very interesting preview feature available in its Python API which allows you to define more complex <a href=\"https:\/\/docs.snowflake.com\/en\/developer-guide\/snowflake-python-api\/snowflake-python-managing-tasks\">DAGs as Python code<\/a>, almost like in Airflow. If you are exclusively on Snowflake, don&#8217;t have the need to schedule tasks on multiple technology stacks from one common orchestration tool, and have simpler DAG logic, check that out!<\/p>\n<p>This article focused mainly on the automation and production aspects of a Snowpark application. If you want to learn more about Snowpark itself and its capabilities please check out <a href=\"https:\/\/www.inovex.de\/de\/blog\/next-stop-insights-how-streamlit-and-snowflake-power-up-deutsche-bahn-data-adventures\/\">this blog post<\/a> with a more sophisticated code example.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The following blog article describes one possible way to automate and orchestrate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow. I will show some ideas, concepts, and code snippets that we implemented at a project with one of our customers dmTECH. All the examples in this article are [&hellip;]<\/p>\n","protected":false},"author":206,"featured_media":51491,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[783,77,181,71,733,385,377,66,1108],"service":[446,414,411],"coauthors":[{"id":206,"display_name":"Julian Seither","user_nicename":"jseither"}],"class_list":["post-50743","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-airflow","tag-big-data","tag-business-intelligence","tag-cloud","tag-cloud-en-2","tag-data-engineering","tag-development","tag-devops","tag-snowflake","service-business-intelligence","service-cloud","service-data-engineering"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Orchestrating Snowpark Scripts With Apache Airflow - inovex GmbH<\/title>\n<meta name=\"description\" content=\"This article describes one possible way to automate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Orchestrating Snowpark Scripts With Apache Airflow - inovex GmbH\" \/>\n<meta property=\"og:description\" content=\"This article describes one possible way to automate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2024-02-05T11:44:50+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-13T09:04:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"880\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Julian Seither\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts-1024x601.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Julian Seither\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"11\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Julian Seither\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/\"},\"author\":{\"name\":\"Julian Seither\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/person\/0a3268d44503c4d0d32fbeb6f1129b94\"},\"headline\":\"Orchestrating Snowpark Scripts With Apache Airflow\",\"datePublished\":\"2024-02-05T11:44:50+00:00\",\"dateModified\":\"2024-02-13T09:04:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/\"},\"wordCount\":1868,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png\",\"keywords\":[\"Airflow\",\"Big Data\",\"Business Intelligence\",\"Cloud\",\"Cloud\",\"Data Engineering\",\"Development\",\"DevOps\",\"Snowflake\"],\"articleSection\":[\"Analytics\",\"English Content\",\"General\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/\",\"url\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/\",\"name\":\"Orchestrating Snowpark Scripts With Apache Airflow - inovex GmbH\",\"isPartOf\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png\",\"datePublished\":\"2024-02-05T11:44:50+00:00\",\"dateModified\":\"2024-02-13T09:04:29+00:00\",\"description\":\"This article describes one possible way to automate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage\",\"url\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png\",\"contentUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png\",\"width\":1500,\"height\":880,\"caption\":\"Grafische Darstellung der Orchestrierung von Snowpark-Skripten, die eine Cloud-Umgebung zeigt, aus der Datenstr\u00f6me flie\u00dfen. Zwei Entwickler arbeiten an Laptops, w\u00e4hrend Diagramme und Datenvisualisierungen abgebildet sind, alles auf einem hellen Hintergrund.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.inovex.de\/de\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Orchestrating Snowpark Scripts With Apache Airflow\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.inovex.de\/de\/#website\",\"url\":\"https:\/\/www.inovex.de\/de\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.inovex.de\/de\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.inovex.de\/de\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\/\/www.inovex.de\/de\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/inovexde\",\"https:\/\/x.com\/inovexgmbh\",\"https:\/\/www.instagram.com\/inovexlife\/\",\"https:\/\/www.linkedin.com\/company\/inovex\",\"https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/person\/0a3268d44503c4d0d32fbeb6f1129b94\",\"name\":\"Julian Seither\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/www.inovex.de\/de\/#\/schema\/person\/image\/35f978bb618834bfd2353e7390e16e33\",\"url\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-36713720_1746291525467658_8163086856494252032_n-96x96.jpg\",\"contentUrl\":\"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-36713720_1746291525467658_8163086856494252032_n-96x96.jpg\",\"caption\":\"Julian Seither\"},\"description\":\"I'm a Data Engineer and Architect, interested in designing and implementing various types of data platforms and streaming applications in the cloud as well as on premise.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/julian-seither-34ba40139\/\"],\"url\":\"https:\/\/www.inovex.de\/de\/blog\/author\/jseither\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Orchestrating Snowpark Scripts With Apache Airflow - inovex GmbH","description":"This article describes one possible way to automate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/","og_locale":"de_DE","og_type":"article","og_title":"Orchestrating Snowpark Scripts With Apache Airflow - inovex GmbH","og_description":"This article describes one possible way to automate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow.","og_url":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2024-02-05T11:44:50+00:00","article_modified_time":"2024-02-13T09:04:29+00:00","og_image":[{"width":1500,"height":880,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png","type":"image\/png"}],"author":"Julian Seither","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts-1024x601.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Julian Seither","Gesch\u00e4tzte Lesezeit":"11\u00a0Minuten","Written by":"Julian Seither"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/"},"author":{"name":"Julian Seither","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/0a3268d44503c4d0d32fbeb6f1129b94"},"headline":"Orchestrating Snowpark Scripts With Apache Airflow","datePublished":"2024-02-05T11:44:50+00:00","dateModified":"2024-02-13T09:04:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/"},"wordCount":1868,"commentCount":0,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png","keywords":["Airflow","Big Data","Business Intelligence","Cloud","Cloud","Data Engineering","Development","DevOps","Snowflake"],"articleSection":["Analytics","English Content","General"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/","url":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/","name":"Orchestrating Snowpark Scripts With Apache Airflow - inovex GmbH","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png","datePublished":"2024-02-05T11:44:50+00:00","dateModified":"2024-02-13T09:04:29+00:00","description":"This article describes one possible way to automate Snowpark scripts written in Python with the help of the generic orchestration tool Apache Airflow.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/Blogheader-Orchestrating-Snowpark-Scripts.png","width":1500,"height":880,"caption":"Grafische Darstellung der Orchestrierung von Snowpark-Skripten, die eine Cloud-Umgebung zeigt, aus der Datenstr\u00f6me flie\u00dfen. Zwei Entwickler arbeiten an Laptops, w\u00e4hrend Diagramme und Datenvisualisierungen abgebildet sind, alles auf einem hellen Hintergrund."},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/orchestrating-snowpark-scripts-with-apache-airflow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"Orchestrating Snowpark Scripts With Apache Airflow"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/0a3268d44503c4d0d32fbeb6f1129b94","name":"Julian Seither","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/image\/35f978bb618834bfd2353e7390e16e33","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-36713720_1746291525467658_8163086856494252032_n-96x96.jpg","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-36713720_1746291525467658_8163086856494252032_n-96x96.jpg","caption":"Julian Seither"},"description":"I'm a Data Engineer and Architect, interested in designing and implementing various types of data platforms and streaming applications in the cloud as well as on premise.","sameAs":["https:\/\/www.linkedin.com\/in\/julian-seither-34ba40139\/"],"url":"https:\/\/www.inovex.de\/de\/blog\/author\/jseither\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/50743","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/206"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=50743"}],"version-history":[{"count":6,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/50743\/revisions"}],"predecessor-version":[{"id":51596,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/50743\/revisions\/51596"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/51491"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=50743"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=50743"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=50743"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=50743"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}