{"id":40924,"date":"2023-05-23T14:52:56","date_gmt":"2023-05-23T12:52:56","guid":{"rendered":"https:\/\/www.inovex.de\/?p=40924"},"modified":"2023-05-23T14:52:56","modified_gmt":"2023-05-23T12:52:56","slug":"data-orchestration-is-airflow-still-the-best-part-1","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/data-orchestration-is-airflow-still-the-best-part-1\/","title":{"rendered":"Data Orchestration: Is Airflow Still the Best? (Part 1)"},"content":{"rendered":"

Nowadays, we rely a lot on technology. As such every second a tremendous amount of data is being collected and processed. Companies can only utilize this tremendous amount of data by building reliable, maintainable, and robust data pipelines. A large company, especially a technology-oriented company, can have more than a thousand data pipelines. How can companies manage so many pipelines?<\/p>\n

\"Morning<\/p>\n

This is only possible due to data orchestration tools like Airflow whose primary task is to simplify the management of data pipelines. Data orchestration is a subfield of workflow orchestration. Workflow orchestration is responsible for the management of related tasks. For instance, when you wake up in the morning, you will probably follow your routine: tooth brushing, making yourself a sandwich, hopping into the shower, and so on. Every activity represents a task. You will usually do them in order, but you could also do some of them simultaneously, e.g. toasting your toast and going in the shower. All of these tasks would be related.<\/p>\n

The same concepts apply to data orchestration, e.g. pulling data out of some database, storing raw data into some file on a file system, validating your data, aggregating your data, and so forth. All of these activities can be viewed as tasks that we need to work off to get our desired result. A typical result could be a cleaned and validated dataset for machine learning model training.<\/p>\n

If you have only one pipeline, you would not need any orchestration tools. Orchestration gets more interesting when you have several pipelines which need to be managed and which need to be run on schedule. E.g. one pipeline should run every hour, the second should run daily at 6 a.m. and the third one needs to run every 10 minutes. Of course, you could set up some cron job that executes your pipelines at the right time but you would need to manually manage all of these cron jobs. Moreover, the cost of maintenance increases dramatically with every new feature you add to your pipelines, e.g., data governance features, logging, or monitoring. All of these requirements can potentially be handled & facilitated by a data orchestrator. But before diving into the world of data orchestration, let us have a look at the history of workflow orchestration.<\/p>\n

\n

<\/p>\n<\/div>