Down the event-driven road: Experiences of integrating streaming into analytic data platforms

Beim Confluent Meetup am 08.10.2018 in München, das von inovex gehostet wurde, sprach Dominik Benz über einige wichtige Aspekte der Einführung von Streaming-basierten Ansätzen in produktiven Datenplattformen.


The requirements of many modern data platforms develop along two directions: (1) Low latency, i.e. the shift from batch-oriented to event-driven processes, which facilitate much more timely and reactive insights; and (2) complex analytics, i.e. the ability to efficiently apply analytic functions or models to the incoming data streams. However, many companies don't start from scratch, and already have well-established data infrastructure and processes with various degrees of affinity and compatibility to these novel paradigms. Based on extensive experience of building data platforms with customers, we describe in this talk some key challenges and aspects of introducing streaming-based approaches in real-world productive environments. These include e.g. integrating existing batch-oriented data sources and APIs, checking consistency when using event sourcing to exchange data, and building realtime analytical visualizations. For all cases, architectural options are discussed, and the final solution is explained, including technologies like Apache Nifi, Airflow, Phoenix, Druid and the Confluent Platform. We close the talk by describing non-technical aspects like building up an event-driven mindset among analysts.

Event: Confluent Meetup

Datum: 08.10.2018

Speaker: Dr. Dominik Benz

Dr. Dominik Benz

Dominik Benz works as Head of Machine Learning Engineering at inovex GmbH. His research background lies in the field of Data Mining from the Social Web, where he obtained a PhD at the Chair of Knowledge and Data Engineering, University of Kassel. Since 2012, he was involved at inovex in engineering and architecting analytic data platforms in various projects for major companies. He is most experienced in tools around the Hadoop ecosystem, and has hands-on experience in productionizing analytical applications, with a special focus on streaming and realtime approaches.

Wie können wir Sie unterstützen?

Nehmen Sie Kontakt mit uns auf – ganz einfach per Telefon oder E-Mail.

Wir freuen uns auf Ihre Nachricht!

inovex Events

Hier geht's zu unseren aktuellen Messen, Konferenzen und Meetups.

Zur Event-Liste

inovex Blog

Florian Wilhelm I 17.01.2018

Data Science in Production: Packaging, Versioning and Continuous Integration

Blog-Artikel lesen

inovex Blog

C. Mense I 13.09.2018

Time Series Forecasting with Machine Learning Models

Blog-Artikel lesen

Aktuelle News über inovex und die Branche

inovex Newsletter

Jetzt hier anmelden!