a picture of Alan with some ornaments and the talk's title

Fully Orchestrating Databricks from Airflow [Video]

Lesezeit
1 ​​min

In this talk given at WeAreDevelopers‘ Python Day we will introduce how to use the popular cloud service Databricks to host Apache Spark applications for distributed data processing in combination with Apache Airflow, an orchestration framework for ETL batch workflows.

YouTube

By loading the video, you agree to YouTube’s privacy policy.
Learn more

Load video

PGlmcmFtZSB0aXRsZT0iRnVsbHkgT3JjaGVzdHJhdGluZyBEYXRhYnJpY2tzIHdpdGggQWlyZmxvdyBbUHJlc2VudGF0aW9uXSIgd2lkdGg9IjY0MCIgaGVpZ2h0PSIzNjAiIHNyYz0iaHR0cHM6Ly93d3cueW91dHViZS1ub2Nvb2tpZS5jb20vZW1iZWQvTE12OURvU3NfV0k/ZmVhdHVyZT1vZW1iZWQiIGZyYW1lYm9yZGVyPSIwIiBhbGxvdz0iYWNjZWxlcm9tZXRlcjsgYXV0b3BsYXk7IGNsaXBib2FyZC13cml0ZTsgZW5jcnlwdGVkLW1lZGlhOyBneXJvc2NvcGU7IHBpY3R1cmUtaW4tcGljdHVyZSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPg==

After a brief exploration of the Databricks Workspace and the fundamentals of Airflow we will take a
deeper look into the functionality Databricks provides in Airflow for orchestrating its workspace. Afterwards, we will find out how to extend and customize that functionality to manage virtually every aspect of the Databricks Workspace from Airflow.

This talk does not require any prior knowledge of Databricks, Spark or Airflow but it does assume familiarity with the fundamentals of the Python programming language especially object oriented programming and REST api requests. The actual distributed data processing with Apache Spark itself is not the focus of this talk.

About Alan Mazankiewicz

Alan finished his Master’s degree from Karlsruhe Institute of Technology in Information Engineering and Management in 2020 before starting his career as a Machine Learning Engineer at inovex GmbH in Cologne, Germany. He (co-) authored two scientific papers in the area of machine learning published at major journals and conferences and is a regular contributor to the open source community.

Hat dir der Beitrag gefallen?

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

Ähnliche Artikel