inovex and ProSiebenSat.1 Digital Celebrate 100th Sprint in Agile Big Data Project
Agile working methods prove key to successful long-term collaboration
(Karlsruhe, 14.12.2017) Since 2012, IT project experts inovex have been supporting ProSiebenSat.1 Digital GmbH in creating, operating and continually developing a complex big data solution to analyse the ProSiebenSat.1 group's web traffic. After five years of successful collaboration, the agile project reached its 100th sprint, a milestone to celebrate.
ProSiebenSat.1 Digital is the multimedia subsidiary of ProSiebenSat.1 Media SE, handling the group's online and video businesses as well as those strategic sectors closely linked to the group?s core TV business (mobile, social TV and HbbTV). The subsidiary's portfolio includes, for example, the websites for the individual channels, such as ProSieben.de, SAT1.de, kabeleins.de and sixx.de, as well as the TV apps for all seven channels.
In order to evaluate, control and optimise its numerous online platforms, the company needs detailed data analyses of its web traffic, with a particular view to scalability. As early as 2012, it was predicted that the data volume would rise by 300% annually, with the (already extremely heterogeneous) data sources predicted to increase considerably, both in type and number. Based on these requirements, therefore, ProSiebenSat.1 Digital wanted a specially designed data warehouse to generate key figures showing the reach and marketing revenues of the company's online content.
Bearing in mind the long-term nature of the project, ProSiebenSat.1 Digital needed a reliable partner with in-depth big data expertise. The company chose inovex. From the very beginning, the project?s goal was to use a data warehouse application to create and implement centralised key figure reporting for the entire ProSiebenSat.1 group.
ProSiebenSat.1 Digital combined Apache Hadoop with a relational data warehouse to form the backend and analysis platform for the future cross-media big data platform. What were initially individual reports have now become a consolidated reporting database supplied with data from more than 30 sources, including social media portals and online marketing.
The raw data from these sources is deposited into and pre-processed by a Hadoop cluster boasting more than 10 terabytes of RAM, 1,400 vcores and over a petabyte of memory. The aggregated data is stored in a relational database (PostgreSQL) and prepared for reporting. The project now contains around 50 ETL paths (most with Pentaho data integration) which process all the raw data from the various vendor systems on a daily basis – and, in the case of social media – in near real-time.
Agile working methods
Right from the start it was apparent that the project's long duration and increasing number of stakeholders necessitated a project management approach that could react rapidly to new demands. The key to the project's success was ? and, indeed, still is – its agile working methods, which involved applying Scrum techniques throughout. Rather than planning a large project in detail through from beginning to end, Scrum is based on the concept of incremental development. This means that the project team performs all the requisite quality assurance and documentation steps and delivers a fully operational version of the software at regular intervals (known as "sprints"). This working method maximises transparency and ensures the high quality which is inherent in Scrum-based projects.
The ProSiebenSat.1 Digital project team, who were supported throughout by (on average) four inovex employees, were able to use two-week sprints to assess the project's progress at regular intervals and to react flexibly to changing customer requirements. The work was carried out remotely and performed in accordance with agile methods, with the team conferring via daily video conferences. At each sprint end, planning, review and retrospective meetings were conducted in person at ProSiebenSat.1 Digital's premises.
Particular emphasis was placed on the collaboration between the software development and infrastructure operations teams. This concept, which is known as "DevOps", describes a corporate culture devoid of hierarchies and silo mentalities, one which focuses on effective collaboration in order to rapidly implement stable, high-quality software.
In October 2017 the team celebrated a major milestone: the completion of the project?s 100th sprint. This has been an extremely successful project, and it is one that will continue to be developed. Although the application was initially the preserve of ProSiebenSat.1 Digital's Business Intelligence Team, its scalable architecture means that more and more of the media group's specialised and internal departments are now benefiting from its results. It has already been geographically expanded to Austria, and expansion of the multi-client capability to the group?s other German subsidiaries is currently underway.
Head of Marketing & Communications
Fresnostraße 14 – 18
Telefon: +49 251 275 800 00
Mobil: +49 173 3181006