Down the event-driven road: Experiences of integrating streaming into analytic data platforms
Beim Confluent Meetup am 08.10.2018 in München, das von inovex gehostet wurde, sprach Dominik Benz über einige wichtige Aspekte der Einführung von Streaming-basierten Ansätzen in produktiven Datenplattformen.
The requirements of many modern data platforms develop along two directions: (1) Low latency, i.e. the shift from batch-oriented to event-driven processes, which facilitate much more timely and reactive insights; and (2) complex analytics, i.e. the ability to efficiently apply analytic functions or models to the incoming data streams. However, many companies don't start from scratch, and already have well-established data infrastructure and processes with various degrees of affinity and compatibility to these novel paradigms. Based on extensive experience of building data platforms with customers, we describe in this talk some key challenges and aspects of introducing streaming-based approaches in real-world productive environments. These include e.g. integrating existing batch-oriented data sources and APIs, checking consistency when using event sourcing to exchange data, and building realtime analytical visualizations. For all cases, architectural options are discussed, and the final solution is explained, including technologies like Apache Nifi, Airflow, Phoenix, Druid and the Confluent Platform. We close the talk by describing non-technical aspects like building up an event-driven mindset among analysts.
Event: Confluent Meetup
Speaker: Dr. Dominik Benz
Dominik Benz works as Head of Machine Learning Engineering at inovex GmbH. His research background lies in the field of Data Mining from the Social Web, where he obtained a PhD at the Chair of Knowledge and Data Engineering, University of Kassel. Since 2012, he was involved at inovex in engineering and architecting analytic data platforms in various projects for major companies. He is most experienced in tools around the Hadoop ecosystem, and has hands-on experience in productionizing analytical applications, with a special focus on streaming and realtime approaches.
Hier geht's zu unseren aktuellen Messen, Konferenzen und Meetups.Zur Event-Liste
Florian Wilhelm I 17.01.2018
Data Science in Production: Packaging, Versioning and Continuous IntegrationBlog-Artikel lesen
C. Mense I 13.09.2018