Fundamentals of DevOps and Software Delivery

Almost every piece of software today depends on software delivery practices to be able to deploy, maintain, and secure that software. And yet, I’m not aware of any hands-on guides that teach software delivery end-to-end. Just about everyone that learns software delivery today is learning it the hard way: that is, through trial and error. Unfortunately, errors in software delivery can be very costly: they involve outages, data loss, and security breaches. The lack of a good way to learn software delivery is making the entire software industry slower, less effective, and less secure.

I experienced this first hand. Back in 2011, I was working at LinkedIn. From the outside, everything looked great: the company had just had its IPO, the share price was up by over 100%, revenue was growing by more than 100% year over year, the website had over 100M members, and it was growing fast, with more than 2 new members joining every single second of the day. But from the inside, the company was in turmoil. Why? Because our software delivery practices had gotten so bad that we could no longer deploy changes to the website.

Back then, we would do deployments once every two weeks, and it was always a painful, tedious, slow, and error-prone affair. But in 2011, we had a deployment that went so badly, that we couldn’t complete it, no matter how hard we tried: we rolled out some new changes, which caused outages and bugs; we pushed some fixes, but those caused new issues; we pushed more fixes, which only led to more issues. Teams worked through the night, into the next day, and still we couldn’t get things stable. In the end, after a several day deployment nightmare, we had to roll everything back.

We were stuck. Here was a company worth nearly $10 billion—and we could not deploy code. This was the cost of not having proper software delivery practices in place. To get out of this mess, we kicked off Project Inversion: a complete freeze on all new feature development for several months while the entire engineering, product, and design team reworked all the underlying infrastructure, tooling, and techniques. The result was a huge success: months later, we were able to deploy dozens of times per day, with far fewer issues and outages, and that allowed the whole company to move much faster.

Today, we might call this a "DevOps transformation" (though back then, the term "DevOps" had just appeared on the scene, so we didn’t call it that), and to get there, we had to go through a lot of pain and outages. The honest truth is that we didn’t know what we didn’t know. We had to seek out best practices from all over the industry, learning about trunk based development from one company, canary deployments from another, feature toggles from another, and so on.

Sadly, even now, as I write this blog post series nearly 15 years later, relatively few developers know about these DevOps and software delivery best practices. After leaving LinkedIn, I co-founded Gruntwork, where I had the opportunity to work with hundreds of companies on their DevOps and software delivery practices. What I saw was LinkedIn’s DevOps nightmare repeated over and over again at companies of all sizes. The best practices that a handful of the top tech companies had figured out were not filtering down to the rest of the industry. Most developers out there still don’t know what they don’t know.

So I decided to write a blog post series.

I hope that this series can be a small step in improving this situation. I hope that a comprehensive, hands-on overview of DevOps and software delivery will help the next generation of DevOps Engineers and Software Engineers get off on the right foot, and avoid some of the DevOps nightmares I’ve seen. I hope that instead of just hacking things together and learning things the hard way, this blog post series will allow you to learn from the experience of others, and perhaps the result will be a software industry that can build software faster, more reliably, and more securely.

All that said, a fair warning: while the results from adopting DevOps can be wonderful, the experience along the way can be anything but wonderful, as described next.

I’m going to let you in on a little secret: we use a single word, "DevOps," to describe what’s actually dozens and dozens of largely unrelated concepts. What does the cryptography behind a TLS certificate have to do with defining a deployment pipeline in GitHub Actions YAML or backing up data from a PostgreSQL database? Not much. And yet, your typical SRE or DevOps Engineer has to deal with all of these, and countless other concepts, too.

What makes DevOps hard is not that any one of these concepts is incredibly complicated by itself, but that there are so many concepts to master—and you have to connect them all together just right. The TLS certificate must be configured just right, or your users will get scary errors; your deployment pipeline must be configured just right, or your team won’t be able to deploy; your database backup must be set up just right, or you are at risk of data loss, and if you lose all your data, you may go out of business entirely. DevOps is remarkable combination of an incredibly broad surface area, but also one where you have to sweat every single detail, for either you get everything connected together correctly, or nothing works at all.

I often use the analogy of a box of cables: you reach into the box, hoping to pull out just one cable, but you inevitably end up pulling out a giant mess where everything is tangled together. Unfortunately, that’s the state of DevOps today: it’s a relatively new industry, the tools and techniques we have just aren’t that mature, and it often feels like everything is broken and frustrating and hopelessly tangled.

My hope in this blog post series is, as much as I can, to untangle this mess of cables for you. To show you that these are, in fact, separate cables—separate concepts—that, in isolation, are well within your grasp.

But sometimes, this is hard to do. Sometimes, reaching into this box of cables feels more like reaching into a box of snakes. You just end up getting bitten. You end up wondering, why the hell am I doing this for a living? If you find yourself sitting there, staring at some nonsense error message, tearing your hair out, stressed, angry, and afraid, know this:

You are not alone.

There are thousands of other developers reaching into that box of snakes and getting bitten, every day. I’m one of them. I’ve lost more hair to DevOps than I care to admit. Even while writing this blog post series, I frequently found myself frustrated, or confused, or yelling at my screen, even though I’ve done most of these things a thousand times. That’s just how it is today.

In fact, there are a few places in this series where I haven’t been able to untangle the wires as much as I’d like: e.g., some example code that’s just too complicated and long to include in the blog post series, so I have to settle for a simpler and less realistic version, or a concept I can’t explain without introducing ten other concepts that come later, so I can only give you a partial explanation for now. In cases like these, I’ve added a "box of snakes" warning that looks like this:

Whenever you see such a warning, understand that you’re going to see part of the picture now, but perhaps won’t be able to get the full picture until later. In fact, this is true of DevOps in general. If you’re new to DevOps, initially, it’ll all seem strange, confusing, and full of incomprehensible buzzwords. And each time you go to learn a new buzzword, you’re hit with ten more unfamiliar buzzwords, so you never feel like you’re getting the whole picture. But I promise you that if you give it enough time, you’ll eventually get over a hump, and suddenly, the pieces will start to make sense, and really come together. You need to build up a big enough base of knowledge and experience, and it’s hard going at first, but at some point, it starts to get easier. It never becomes completely easy, but you get to a point where you always feel confident that you can figure it out.

So stick with it. And watch out for those snakes.

This blog post series is for anyone responsible for deploying and managing apps in production—that is, anyone responsible for software delivery. This includes:

This blog post series does not assume that you’re already an expert coder or expert sysadmin: a basic familiarity with programming, the command line, and server-based software (e.g., websites) should suffice. Everything else you need you’ll be able to pick up as you go. The only tools you need are a computer, an internet connection, and the desire to learn.

Table 1 shows a part-by-part outline of what the blog post series covers, including the key ideas you’ll explore and the hands-on examples you’ll get to try out in that part:

Feel free to read the blog post series from beginning to end or jump around to the parts that interest you the most. Note that the examples in each part reference and build upon the examples from the previous parts, so if you skip around, use the open source code examples (as described in Open Source Code Examples) to get your bearings.

Given the breadth of DevOps, this book covers a lot of ground and includes a lot of detail. To help you avoid missing the forest for the trees, I try to call out the key takeaways in each blog post as follows:

Pay special attention to these items, as they typically highlight the most important lessons in that post.

This blog post series is meant to fill a specific gap: a hands-on guide to DevOps and software delivery, targeted at practitioners. This is already a huge amount of content to cover, which means there are some DevOps and software delivery topics that this series will either skip or only touch on lightly:

This blog post series includes many examples for you to work through. You can find all these code samples in the following GitHub repository:

https://github.com/brikis98/devops-book

You might want to check out this repo before you begin reading so you can follow along with all the examples on your own computer (if you are new to Git, check out the Git tutorial in Part 4):

The code samples are organized by part (e.g., ch1, ch2, etc.), and within each part, by tool (e.g., ansible, kubernetes, tofu). For example, the example Packer template in Part 2 will be in the folder ch2/packer, and the example OpenTofu module called lambda in Part 3 will be in the folder ch3/tofu/modules/lambda.

It’s worth noting that most of the examples show you what the code looks like at the end of a part. If you want to maximize your learning, you’re better off writing the code yourself, from scratch, and checking the "official" solutions only at the very end.

Now that you have a basic understanding of what this blog post series is all about, it’s time to get started. And where better to begin than at the beginning: head over to Part 1, An Introduction to DevOps and Software Delivery to learn where DevOps came from and the basics of deploying apps.

Update, June 25, 2024: This blog post series is now also available as a book called Fundamentals of DevOps and Software Delivery: A hands-on guide to deploying and managing production software, published by O’Reilly Media!