DevOps Operations for ProSiebenSat.1’s Channel Portals – inovex Supports IT Infrastructure for Germany’s Largest TV Conglomerate
ProSiebenSat.1’s online portals are among the most frequently visited networks in the German Internet market. The sites belonging to the group’s popular TV programmes receive more than 1.3 million visits every day. Behind this success is a sophisticated IT landscape which owes its setup, control and operations to the knowledge of inovex’s IT Engineering & Operations specialists. A close collaborative partnership between the two companies ensures a high level of quality.
ProSiebenSat.1’s channel portals are built on inovex’s expertise.
When it comes to their online presence, Germany’s biggest TV company is leaving nothing to chance: interactive offerings and additional content for popular formats like “Germany’s Next Top Model” expand and enhance ProSiebenSat.1’s presence on the worldwide web. There, visitors can find information on all the TV programmes, watch whole episodes of their favourites, as well as clips and film trailers; and read news, advice and specials on a tremendous variety of topics. A separate multimedia subsidiary, ProSiebenSat.1 Digital GmbH, manages these – and the other digital activities which now represent an important growth market for the Munich-based concern.
In addition to the portals for the group’s major channels – pro7.de, sat1.de, kabeleins.de and Sixx.de – the online offerings include separate websites for highly successful formats like “The Voice of Germany”, “Schlag den Raab”, “ran” and “Bully” Herbig’s website, bullybase.de. In addition, there are portals for the new, growing TV channels ProSieben MAXX and Sat.1 Gold and the new online music service Ampya.com. ProSiebenSat.1 has centralised the necessary IT infrastructure, which is now handled by the Central Operations department. This is the heart of the corporation’s entire IT operations, where IT processes, infrastructures, tools and cooperation partners are identified and managed.
Behind the Scenes: Server Architecture Seeks Top Engineers
Successfully setting up and operating a comprehensive online landscape whose total hits place it in Germany’s IVW Top 20 ranking of domestic Internet offerings requires ProSiebenSat.1 Digital GmbH’s Central Operations to possess two technical keys. The first is the detailed knowledge of the Linux system and comprehensive, in-depth experience required for the precise planning, testing and configuration of the IT infrastructure. Secondly (and simultaneously) the utterly seamless, integrated collaboration from development and testing right through to deployment and operation requires intelligent processes, excellent automation and wide-ranging, across-the-board expertise. The first factor ensures stable availability even under high (and highly fluctuating) loads comprising large data volumes and millions of hits weekly. The second is particularly important in improving cost efficiency and flexibility in accompanying new innovations on their journey from the development environment to the productive one. It also ensures a high level of sustainability. These requirements combine to place the highest of demands on ProSiebenSat.1’s IT engineers.
For the overall planning, configuration, installation, automation, control and monitoring of the server landscapes needed for ProSiebenSat.1's channel portals, ProSiebenSat.1 Digital benefits from support provided by inovex GmbH at its Karlsruhe location. The company also ensures close collaboration between its own specialist departments and external service providers. Using the specifications set out by ProSiebenSat.1's internal development department – which, in addition to technical specifications like system versions and the requisite web, database and other servers and modules, also includes the expected visitor numbers – the inovex engineers build and configure the customised test environments, the pre-production staging environment, and, ultimately, the highly available live environment. inovex was also substantially involved in setting up the entire configuration management system, as well as an automated deployment infrastructure.
For ProSiebenSat.1's channel portals, there is a specific factor involved in TV platforms that must be taken into account as far as scalability is concerned. Specific TV campaigns, unexpectedly popular TV programmes, or websites mentioned during broadcast programmes can cause considerable loads on the appropriate online platforms. This often has little in common with “normal” everyday site access patterns – traffic fluctuates much more strongly and sometimes in unpredictable ways. Already, the inovex IT engineers can scale up temporarily as needed, for example at weekends or during specific campaigns with audiences in the millions. In future, particularly in view of ProSiebenSat.1’s growth plans, however, an even higher level of automation will be implemented.
Long-Term Cooperation Ensures Continuity
The collaboration between ProSiebenSat.1 and inovex dates back to 2009. Since then, it has led to a wide range of follow-up contracts for inovex. Back then, the inovex experts were responsible for Maxdome, a popular video-on-demand platform. Today, the platform is wholly owned by ProSiebenSat.1 and still – successfully – operated by inovex. The long-term partnership with the media concern will continue into the future. inovex has also taken over productive operations for ProSiebenSat.1’s channel portals. Since the beginning of 2013, these have been operated by dedicated teams which are on call around the clock, seven days a week, to resolve any operational problems. Moreover, inovex was involved in setting up the test and staging environments and in supporting the initial configuration for the increasingly significant MyVideo.de free video platform, which also belongs to ProSiebenSat1 Digital.
“Our close, collaborative partnership with ProSiebenSat.1 illustrates how we are able to react appropriately and with flexible staffing to the individual needs of our customers. We serve as ProSiebenSat.1’s dedicated operator in order to preserve the high quality of this comprehensive landscape over the long term. Our integrated IT Engineering & Operations department helps us to efficiently blend cross-departmental operations, while a dedicated, customer-specific team ensures continuity.”
– Matthias Albert, Head of IT Engineering & Operations, inovex GmbH
Testing, Configuration and Deployment in Major Web Projects
There are a great many factors to be considered when taking a major web project from a new development to its deployment in a productive environment, and customers still tend to underestimate the complexity of these considerations. Every new feature added also poses a potential risk to the stability and availability of the system. Correctly designed testing and staging environments, as well as a well established continuous integration and deployment process, enable these risks to be effectively and efficiently managed. These factors allow significant risks to be identified and prevented before they enter the productive system, thereby ensuring the project’s success among a discerning online audience. The sustainable construction of server infrastructures begins with the targeted setup and configuration of the appropriate test environment – which may include separate frontend and backend systems and different testing levels. These tasks require a great deal of practical knowledge and experience in order to produce meaningful results. An incorrectly configured test is as “good” as no test at all. inovex then deploys the finished development in the staging environment for more advanced testing and ultimate customer acceptance. These staging environments are also closer in scope to the subsequent live environment, a fact which enables load-testing to be carried out – no trivial task with traffic averaging millions of hits per week. Only once the staging tests have been successfully completed can the system be made live. Heterogeneous online landscapes comprising many individual portals (like the ProSiebenSat.1 environment, which was newly reconstructed in collaboration with the inovex experts) require a very large number of deployments at the outset.
Technology and Automation
The latest technologies are evaluated by inovex and are, where applicable, rapidly deployed. In doing this, inovex places clear emphasis on open-source technologies, a policy which facilitates future-proof, manufacturer-independent investments. Using open-source automation technologies like Puppet and Rex, in which inovex has a great deal of expertise, results in considerable savings during the configuration process, as well as helping to standardise processes. Web server alternatives (Nginx), in-memory databases (Redis), cache servers (memcached) and web accelerators (Varnish) ensure better performance. When it comes to search servers (Solr, Elasticsearch) and relational and document-orientated databases (such as PostgreSQL, MySQL and CouchDB) open-source products play a prominent role. With regard to configuration and deployment processes, the greatest possible level of automation is desirable – and not just because of the time savings involved. In large-scale projects, automation also reduces error-proneness at various levels. It allows changes or new versions to be reliably and reproducibly delivered, and test quality increases due to the elimination of potential manual, undocumented deviations. In addition, manual deployment and configuration management carry the risk that critical knowledge is divided among multiple employees – who may not all be available when required. Lastly, automation also ensures standardisation by considerably reducing sources of error and the need for coordination. In this significant area of automation, initial added expenses are quickly amortised. inovex has a great deal of experience with automated Linux data centres, and industry-leading knowledge of the open-source Rex automation solution. Rex’s core committer works for inovex, and the company offers specialised training and presentations.
Work Organisation: inovex Breathes Life Into DevOps
In keeping with the DevOps approach, inovex works very closely with the development department involved to ensure that the testing and administration views are taken into account from the very outset. It also, however, cooperates with the Central Operations department. In particular, the collaboration with ProSiebenSat.1 Digital GmbH’s Central Operations experts illustrates how DevOps, properly applied, is more than just a buzzword. Instead, it’s a real-life method of improving processes, promoting cooperation, and eliminating developmental errors in areas of intensive collaboration. It therefore considerably improves efficiency. Instead of encouraging silos with separate responsibilities and goals, it allows communication across the board. Processes run more smoothly – and therefore faster and more cheaply. In the case of ProSiebenSat.1, for example, one inovex employee functions as a permanent member of the ProSiebenSat.1 development team, and the departments remain in daily contact with one another – thus continuing agile development across departments which were originally separated.
As part of its long-standing partnership with ProSiebenSat.1 Digital GmbH’s Central Operations department, inovex supports the setup, configuration and operation of the online portals for the various channels belonging to the ProSiebenSat.1 group. This means that inovex is involved with the development department (DevOps), sets up the various environments (testing/staging/live), and manages the live data centre operations, including the 24x7 on-call technical support and problem resolution. The result is a stable, yet flexible, highly available, highly scalable system for the ProSiebenSat.1 group’s channel portals.
- Setup of all environments (TESTING/STAGING/LIVE)
- Setup of the entire configuration management
- Automated deployment infrastructure
- 24x7 website operation with on-call technical support
- Puppet, Rex
- Apache, Nginx
- Solr, Elasticsearch
- MySQL, CouchDB