DevOps has been a subject of interest for quite a while now, although — at least in my opinion — it never got the mainstream adoption it should have. But maybe this can change. Very recently Puppet Labs published a report that links overall company performance to the performance of the companies’ IT department and, here’s the kicker, the IT departments’ performance to the utilisation of DevOps practices. So, let’s have closer look at DevOps.
What is DevOps, anyway?
From my perspective it’s the agile idea of a cross-functional team taken to the next level. The idea of a cross-functional team has always been to include all people that are needed to generate value (see e.g. the first Scrum paper by Schwaber & Sutherland from 1995). That means if you need to operate your product in order to generate value from it (e.g. because your product is a web application), a cross-functional team must also include the people who deploy, operate and manage it.
This is not a a very new idea. Facebook has been operating with this mindset from the very beginning (as far as I can tell). And as an example for a big enterprise doing it, we have the famous 2-pizza teams from Amazon: their engineering force is comprised of truly cross-functional teams in the above sense, with each team being small enough so it can be fed with two pizzas.
Now, there are two aspects I want to talk about. Why is this idea becoming relevant now (as opposed to a couple of years ago) and if it’s such a good idea, why isn’t everybody just doing it?
Why is this becoming relevant now?
In my opinion, there are a couple of factors contributing to the fact that many companies feel the pain of not having integrated software development and operations on a team level.
More and more enterprise applications are becoming web applications (or mobile applications requiring web service backends), resulting in a software development lifecycle which doesn’t end with shipping the software to the customer. These companies now have to start providing a service instead of a product to their customers and thereby become operators of their own software (more on this here).
The rise of agile
It may appear a bit paradoxical, but the rise of agile methods in software development solved for many teams the communication problem between product management or the business side and software development. Oftentimes however, an implementation of an agile method, say Scrum, is limited to these two areas. The Scrum Development Team includes software developers and testers, but no one from operations. The Scrum Product Owner deals with features and customers and doesn’t care much about the operational side of things (It just has to work).
When a Scrum team set up in this fashion starts to perform, it stresses the interface between development and operations by continuously releasing new software versions. At this point, the interface becomes the new bottleneck and these teams will have to start thinking about solutions.
The increasing use of off-the-shelve components
When you develop web software, there are a number of components that you have to rely on in order to get anything done: web servers, application containers, databases, etc. Such components share an interesting property: they are not integrated into your software like libraries or frameworks. Instead they provide an environment which enables your software to run in the first place. The difference between a component like Tomcat and a framework like Spring is that Tomcat needs a non-trivial amount of deployment, configuration and maintenance of itself. The skills needed to do this properly are not skills a typical software developer has. Take the aspect of networking as an example: not only do you need to know the basics of networking (say, the difference between broadcast and multicast), but also the current physical setup of your infrastructure (say, if there’s more than one class D network).
So, as products include more and more components that need to be operated by specialists, the question of how to integrate these specialists with the development teams becomes more and more relevant.
Why isn’t everybody just doing it, then?
The idea of DevOps at its core is quite simple (see above). The execution is not. DevOps has many facets which you all need to get working in order to become successful. There’re the technical aspects of automated deployment & configuration management. There’s also the aspect of the technical process, i.e. how to use version control, build tools and ticket systems effectively (see e.g. Agile ALM). And then there’s the aspect of organisational and cultural integration…
It’s quite easy to say we need one cross-functional team that has all the skills and competencies necessary to deliver customer/company value. It’s quite hard to implement. Just to give you an idea, here’s a list of things that make implementing DevOps hard from an organisational and cultural perspective:
- Merging a development and operations team simply into one creates a team that is usually too large to be effective
- If you have n development teams and one operations team, how can you create m DevOps teams?
- How to deal with (unpredictable) maintenance tasks? Do you put them on the backlog?
- How to deal with firefighting?
- How do you integrate on-call duty or 24/7 on-site presence into agile frameworks like Scrum?
- How do you deal with regulatory requirements, like restricted access to certain systems?
- On the cultural side: developers value “new stuff” while sysadmins value stability (very generally speaking).
As I don’t want to end this article with a bunch of open questions or problems, let’s discuss the first one.
Merging Dev and Ops
Let’s create a concrete, albeit textbook, example: You have a Scrum Development Team consisting of 7 developers. In addition you have a 5 person operations team. If you just form a DevOps team from these two, you end up with 12 people, which is too large for effective collaboration (7±2 is the magic number). That means you need to form two (DevOps!) teams from 12 people. It also means you’re entering the realm of Scrum at scale, i.e. Scrum with more than one team (see e.g. Scaling Lean and Agile Development).
One common idea for scaling agile development are feature teams. A feature team is end to end responsible for a set of features, which typically involves all software layers/components. (The traditional counterpart of a feature team is the component team, which is responsible for a layer or component.)
The main benefit of feature teams is that there are fewer inter-team dependencies. If you have component teams, (nearly) every new feature that needs to be implemented needs collaboration from all teams. With feature teams you still need collaboration, but there will be a significant amount of features that can be developed by one team on it’s own (to learn know more about feature vs. component teams, see e.g. Agile Software Requirements)
This of course, fits very well into the idea of DevOps. If you regard infrastructure as simply the lowest layer of your software, this layer should be included in every feature team. And there you are: a DevOps team!
Applying this to our example means that we have to identify (at least) two feature-areas and build the teams accordingly. Let’s say you develop a software that allows small business owners to create their own web-shops. One team could work on all features related to the visitors of the individual shops and the other team on features available for the shop creators. Or one team could deal with everything that is related to social (like Facebook, Twitter integration, etc.) while the other team works on everything else.
Ok. Now we have an idea for one piece of the puzzle. But, as we’re agile, one next step is all we need… Let’s get to work!