Fundamentals of DevOps and Software Delivery

Almost every piece of software depends on software delivery practices for deployment, maintenance, and security. And yet, I’m not aware of any hands-on guides that teach software delivery end-to-end. Just about everyone that learns software delivery today is learning it the hard way: that is, through trial and error. Unfortunately, errors in software delivery can be costly: they involve outages, data loss, and security breaches. The lack of a good way to learn software delivery is making the entire software industry slower, less effective, and less secure.

I experienced this first hand. Back in 2011, I was working at LinkedIn. From the outside, everything looked great: the company had just had its IPO, the share price was up by over 100%, revenue was growing by more than 100% year over year, and the website had over 100M members, with 2 new members joining every single second. But from the inside, the company was in turmoil. Why? Because our software delivery practices had gotten so bad that we could no longer deploy.

Back then, we would do deployments once every two weeks, and it was always a painful, tedious, slow, and error-prone affair. In 2011, we had a deployment that went so badly, that we couldn’t complete it, no matter how hard we tried. We rolled out some new changes, which caused system instability; we pushed some fixes, but those caused new bugs; we pushed more fixes, but that only led to more issues. Teams worked through the night, into the next day, and we still couldn’t get things stable. In the end, after a several day deployment nightmare, we had to roll everything back.

Here was a company worth nearly $10 billion, and we could not deploy code. To get out of this mess, we kicked off Project Inversion, which was a complete freeze on all new feature development for several months while the entire engineering, product, and design team reworked all the underlying infrastructure, tooling, and practices. The result was a huge success: months later, we were able to deploy dozens of times per day, with far fewer issues and outages, and that allowed the whole company to move faster.

Today, we might call this a "DevOps transformation" (though back then, the term "DevOps" had just appeared on the scene, so we didn’t call it that), and to get there, we had to go through a lot of pain and outages. The truth is that we didn’t know what we didn’t know. We had to go out and chat with companies across the industry, learning about trunk based development from one company, canary deployments from another, feature toggles from another, and so on.

Sadly, even now, as I write this blog post series nearly 15 years later, relatively few developers know about these DevOps and software delivery practices. After leaving LinkedIn, I co-founded Gruntwork, where I had the opportunity to work with hundreds of companies on their DevOps and software delivery practices. What I saw was LinkedIn’s DevOps nightmare repeated over and over again at companies of all sizes. The techniques that a handful of the top tech companies had figured out were not filtering down to the rest of the industry. Most developers out there still don’t know what they don’t know.

So I decided to write a blog post series.

I hope that this series can be a small step in improving this situation. I hope that a comprehensive, hands-on overview of DevOps and software delivery will help the next generation of software companies get off on the right foot, and avoid some of the DevOps nightmares I’ve seen. I hope that instead of just hacking things together and learning things the hard way, this blog post series will allow you to learn from the experience of others, and perhaps the result will be a software industry that can build software faster, more reliably, and more securely.

If that happens, it could have a profound impact on the industry, as described next.

The vast majority of developers have never had the opportunity to see what world-class software delivery looks like first hand. If you’re one of them, you’ll be astonished by the gap between companies with world-class software delivery processes and everyone else. It’s not a 1.1x or 1.5x improvement: it’s 10x, 100x, or more.

Table 1 shows the difference between elite performers and low performers at the four key DevOps Research and Assessment (DORA) metrics, which are a quick way to assess the performance of a software development team:

These are staggering differences. To put them into perspective, we’re talking the difference between:

It’s almost a meme that developers who leave companies with world-class software delivery processes, such as Google, Facebook, Amazon, or LinkedIn (after Project Inversion), complain bitterly about how much they miss the infrastructure and tooling. That’s because they are used to a world where:

What do the equivalent numbers look like at your organization?

If you’re not even in the ballpark, don’t fret. The first thing you need to know is that it’s possible to achieve these results, even if you’re not a multi-billion dollar company. In fact, there are many ways to achieve these sorts of results, and each of those world-class companies does it a bit differently. That said, they also have a lot in common, and DevOps is an attempt to capture some of the common patterns.

DevOps emerged in the late 2000s as a response to inefficiencies in how companies delivered software. Historically, most companies had a Developer ("Dev") team responsible for writing the software, and an Operations ("Ops") team responsible for managing the hardware. In many companies, these teams worked in silos with conflicting objectives. The Ops team’s core objectives were usually security and reliability, so they were the only ones with access to production systems, and were on the hook for doing deployments and dealing with outages. The Dev team’s core objectives were usually to ship features, and to do so as quickly as possible. This arrangement often led to problems.

The Dev team, who was typically working under time pressure, with no access to production systems or on-call duties, would write the code, and with little testing, "toss it over the wall" to the Ops team. The Ops team would then deploy that code, typically via a manual process, and either from an error in this deployment process, or a bug in the code, there would often be failures and outages. The Ops team, tired of waking up at 3 a.m. to deal with outages, would end up reducing the release cadence. Unfortunately, this led to a vicious cycle: fewer releases meant each release was bigger and more complicated, which led to more outages, which led to an even slower release cadence, and so on, until the company would grind to a halt.

Although the problems with manual processes and siloing were known for many years, it wasn’t until around 2007-2008 that the software industry started to discuss these issues more openly in various conferences, meetups, and articles. These discussions coincided with, and were big drivers of, several major shifts that happened over the next decade:

All these trends together became known as the DevOps movement. DevOps isn’t the name of a team or a job title or a particular technology (though you’ll see it used for all three). Instead, it’s a set of processes, ideas, and techniques. Everyone has a slightly different definition of DevOps, but for this blog post series, I’m going to define DevOps as a methodology with the following goal:

You heard how LinkedIn’s DevOps transformation saved the company, but there are many other examples. Nordstrom found that after applying DevOps practices to its organization, it was able to increase the number of features it delivered per month by 100%, reduce defects by 50%, reduce lead times by 60%, and reduce the number of production incidents by 60% to 90%. After HP’s LaserJet Firmware division began using DevOps practices, the amount of time its developers spent on developing new features went from 5% to 40%, and overall development costs were reduced by 40%. Etsy used DevOps practices to go from stressful, infrequent deployments that caused numerous outages to deploying 25 to 50 times per day, with fewer outages.^[1]

All that said, a fair warning: while the results from adopting DevOps can be wonderful, the experience along the way can be anything but wonderful, as described next.

I’m going to let you in on a little secret: we use a single word, "DevOps," to describe what’s actually dozens and dozens of largely unrelated concepts. What does the cryptography behind a TLS certificate have to do with defining a deployment pipeline in GitHub Actions YAML or backing up data from a PostgreSQL database? Not much. And yet, your typical SRE or DevOps Engineer has to deal with all of these, and countless other concepts, too.

What makes DevOps hard is not that any one of these concepts is incredibly complicated by itself, but that there are so many concepts to master—and you have to connect them all together just right. The TLS certificate must be configured just right, or your users will get scary errors that prevent them from accessing your website; your deployment pipeline must be configured just right, or your team won’t be able to deploy; your database backup must be set up just right, or you are at risk of data loss, and if you lose all your data, you may go out of business entirely. DevOps is remarkable combination of an incredibly broad surface area, but also one where you have to sweat every single detail, for either you get everything connected together correctly, or nothing works at all.

I often use the analogy of a box of cables: you reach into the box, hoping to pull out just one cable, but you inevitably end up pulling out a giant mess where everything is tangled together. Unfortunately, that’s the state of DevOps today. It’s a relatively new industry, the tools and techniques we have just aren’t that mature, and it often feels like everything is broken and frustrating and hopelessly tangled.

My hope in this blog post series is, as much as I can, to untangle this mess of cables for you. To show you that these are, in fact, separate cables—separate concepts—that, in isolation, are something you can readily understand, begin to work with, and ultimately become proficient in.

But sometimes, this is hard to do. Sometimes, reaching into this box of cables feels more like reaching into a box of snakes. You just end up getting bitten. If you find yourself sitting there, staring at some nonsense error message, tearing your hair out, stressed, angry, and afraid, know this:

You are not alone.

There are thousands of other developers reaching into that box of snakes and getting bitten, every day. I’m one of them. I’ve lost more hair to DevOps than I care to admit. Even while writing this blog post series, I frequently found myself frustrated, or confused, or yelling at my screen, even though I’ve done most of these things a thousand times. That’s just how it is today.

In fact, there are a few places in this series where I haven’t been able to untangle the wires as much as I’d like: e.g., some example code that’s just too complicated and long to include in the blog post series, so I have to settle for a simpler and less realistic version, or a concept I can’t explain without introducing ten other concepts that come later, so I can only give you a partial explanation for now. In cases like these, I’ve added a "box of snakes" warning that looks like this:

Whenever you see such a warning, understand that you’re going to see part of the picture now, but perhaps won’t be able to get the full picture until later. In fact, this is true of DevOps in general. If you’re new to DevOps, initially, it’ll all seem strange, confusing, and full of incomprehensible buzzwords. And each time you go to learn a new buzzword, you’re hit with ten more unfamiliar buzzwords, so you never feel like you’re getting the whole picture. But I promise you that if you give it enough time, you’ll eventually get over a hump, and suddenly, the pieces will start to make sense, and really come together. You need to build up a big enough base of knowledge and experience, and it’s hard going at first, but at some point, it starts to get easier. It never becomes completely easy, but you get to a point where you always feel confident that you can figure it out.

So stick with it. And watch out for those snakes.

This blog post series is for anyone responsible for deploying and managing apps in production—that is, anyone responsible for software delivery. This includes:

This blog post series does not assume that you’re already an expert coder or expert sysadmin. A basic familiarity with programming, the command line, and server-based software (e.g., websites) should suffice. Everything else you need you’ll be able to pick up as you go. The only tools you need are a computer, an internet connection, and the desire to learn.

Table 2 shows a part-by-part outline of what the blog post series covers, including the key ideas you’ll explore and the hands-on examples you’ll try in each part:

Feel free to read the blog post series from beginning to end or jump around to the parts that interest you the most. Note that the examples in each part reference and build upon the examples from the previous parts, so if you skip around, use the open source code examples (as described in Open Source Code Examples) to catch up.

On the blog post series’s website, you’ll find a dedicated page for each blog post (for example, here is the dedicated page for Chapter 1), which includes a list of recommended resources (books, blog posts, courses) you can use to go deeper on the topics covered in that blog post, as well as a list of all the tools mentioned in that blog post.

Given the breadth of DevOps, this blog post series covers a lot of ground and includes a lot of detail. To help you avoid missing the forest for the trees, I try to call out the key takeaways in each blog post as follows:

Pay special attention to these items, as they typically highlight the most important lessons in that post.

This blog post series is meant to fill a specific gap: a hands-on guide to DevOps and software delivery, targeted at practitioners. This is already a huge amount of content to cover, which means there are some DevOps and software delivery topics that this series will either skip or only touch on lightly:

This blog post series includes many examples for you to work through. You can find all these code samples in the following GitHub repository:

https://github.com/brikis98/devops-book

You might want to check out this repo before you begin reading so you can follow along with all the examples on your own computer:

The code samples are organized by part (e.g., ch1, ch2, etc.), and within each part, by tool (e.g., ansible, kubernetes, tofu). For example, the example Packer template in Part 2 will be in the folder ch2/packer, and the example OpenTofu module called lambda in Part 3 will be in the folder ch3/tofu/modules/lambda.

It’s worth noting that most of the examples show you what the code looks like at the end of a part. If you want to maximize your learning, you’re better off writing the code yourself, from scratch, and checking the "official" solutions only at the end.

Now that you have a basic understanding of what this blog post series is all about, it’s time to get started. And where better to begin than at the beginning: head over to Part 1, How to Deploy Your App to learn the basics of deploying apps.

Update, June 25, 2024: This blog post series is now also available as a book called Fundamentals of DevOps and Software Delivery: A hands-on guide to deploying and managing software in production, published by O’Reilly Media!