Part 10. How to Monitor Your Systems

Most logging libraries support multiple log levels, which allow you to specify the severity of each log message. Table 21 shows the typical log levels you’ll see in most libraries (sometimes with slightly different names), ordered from least to most severe:

Example 179 shows how to use the Winston library to do logging in JavaScript:

Here’s what this code does:

If you run this code, you’ll see:

This might not seem like a big improvement over using something like console.log, but using a logging library with log levels gives you a few advantages right away. First, just about all logging libraries allow you to specify the minimum log level, and they will automatically discard any messages that have a lower severity. For example, consider what happens when you have the log statements shown in Example 180:

When you run this code, you’ll see the following output:

Notice how the debug message is missing. Since the logger in Example 179 sets the log level to info, only messages at info and above are going to be recorded. Log levels allow you to add lots of trace and debug logging throughout your code without making the logs too noisy during production usage, while still having the ability to use a lower log level in dev, or even in prod (temporarily) while troubleshooting.

Log levels also make it easier to scan the logs, both visually, and with tools such as grep. For example, if you had a log file with thousands of lines of logs, you could use grep to filter the log to just the error messages as shown in Example 181:

Log levels are only one of the benefits you get from using a logging library. Another big benefit is the ability to standardize the format of all your log messages, which can make it even easier to read and scan your logs, as discussed in the next section.

Just about all logging libraries allow you to define the format to use for your log messages. Example 182 shows how to apply basic formatting to a Winston logger:

The preceding code applies the following formatting to each log message:

If you run this code, you’ll see the following:

Using log formatting ensures every message uses the same standardized pattern, which makes it easier to read and parse the logs. It also allows you to include useful contextual information, often referred to in logging libraries as metadata, to help you understand what was happening when each message was logged. The timestamp is one useful piece of contextual information; Example 183 shows a few other pieces of useful context:

This code has two changes:

When you run this code, you see the following:

Now, each log message tells a little self-contained story: at this time, a certain user sent an HTTP request for a specific path, and here’s a message about it, at a specific severity. These stories become even more useful if you use multiple loggers.

Most logging libraries allow you to create multiple loggers, each with their own name. Example 184 shows how you can create multiple Winston loggers:

This code does the following:

The output from the code is as follows:

Using multiple loggers allows you to see what part of your code is doing the logging. It also allows you to configure different logging settings (such as different log levels) for different parts of your code, so if you’re seeing strange behavior in one subsystem, you can turn up logging just for that part of the code, or if another subsystem is spamming the logs too much, you can turn down the log level for just that subsystem. You can also choose to include different contextual data with the logs for each subsystem, making the self-contained stories more relevant to their particular domain.

One trend you may have noticed is that the logs become more useful as you add more and more "metadata" to them: e.g., IP address, module name, timestamp, and so on. This evolves into structured logging, as described in the next section.

For years, logging text strings and defining fancy formats was the gold standard. Unfortunately, the result was that every piece of software, every library, every language, and every developer formatted their logs a bit differently. This made it hard to build tooling that could automatically parse, search, and filter logs.

In recent years, there has been a push to move to structured logging, where you output logs in a well-defined data format such as JSON or XML, rather than as arbitrary strings. This makes it easier to build a variety of tooling that can parse and interpret the logs. Example 185 shows how to configure the Winston to output logs as JSON:

There’s only one change from the previous example:

Here’s the output from this code (note, the output has been reformatted to fit this blog post series; it would normally all be on one line):

The advantage of structured logging is that it is both human-readable—arguably, it’s more readable than the raw text format, as every field has a clear label—and machine-readable. It’s now easier to build tooling to parse, search, filter, and analyze logs. Just as importantly, structured logging changes how you think about logging from just plain-text messages to well-defined structures, as shown in Example 186:

The preceding code produces the following output (again, reformatted to multiple lines to fit in this blog post series):

Instead of thinking of arbitrary strings with a little bit of relevant information, you start thinking of logging as a way to capture as much context as you can, in the form of key-value pairs, about what’s happening. As you’ll see later in this blog post, these ideas are closely related to the shift to structured events, observability, and tracing.

So far, all the examples in this blog post have logged directly to the terminal, but most software stores logs in files, as per the next section.

Logging to the terminal (to stdout and stderr streams) is great in the development environment, when there’s a person looking directly at the terminal, but it’s not as useful in a server environment. Therefore, outside of an interactive terminal session, most software writes log data to files. This way, your log history is persisted, so you can go back any time, and see what happened in the past. Example 187 shows an example of how to send the log output to a file:

There is only one change:

When you run this code, you end up with two files on disk, moduleA.log and moduleB.log. moduleA.log will have just the logs from module A and moduleB.log will have just the logs from module B. Now you can check the file for the module you’re interested in and use the timestamps to go back to whatever point in time you need.

Note that, over time, log files can become huge. To ensure the files don’t become so large that it’s too hard to browse them, and to prevent running out of disk space, you should typically configure log rotation, where you set up an automated process to rename, compress, and possibly delete log files once they hit a certain file size or age. On most Linux systems, you can use logrotate to do this for you. Alternatively, some logging libraries have log rotation features built-in, as shown in Example 188:

This code uses the Winston library to do the following:

In addition to configuring log rotation to ensure your servers don’t run out of disk space, you should also configure log aggregation, as discussed in the next section.

If your software is on a single server, looking at its log files is no problem. But when you run your software across dozens or hundreds of different servers, any one of which may contain the log messages you need, finding the right log file can be a significant challenge. The solution is to use log aggregation tools that send logs from all of your servers to a single, central destination that can store the log data and provide an interface to analyze (search and filter) that data.

Some of the major players in the log storage and analysis space include Loggly, Papertrail, Sumo Logic, Splunk, Graylog, Elasticsearch (with Logstash) / OpenSearch (with OpenSearch Logstash), Datadog, New Relic, Amazon CloudWatch Logs, Google Cloud Logging, and Azure Monitor Logs. Most of these tools provide their own custom data collector tools that you can run on your servers to ship logs to the central data store. There are also several data-store agnostic data collector tools that can ship logs to many different destinations, including Syslog, Fluentd, Fluent Bit, Beats, Vector, and OpenTelemetry Collector (you’ll learn more about OpenTelemetry later in this blog post).

For example, the Lambda functions you deployed in previous blog posts automatically aggregate their logs in CloudWatch Logs. Logs in CloudWatch are organized into log groups, which for a Lambda function named function-name will end up being a log group named /aws/lambda/<function-name>. If you head to the CloudWatch Logs Console, search for the name of a Lambda function, such as "lambda-rds-app" from Part 9, and click on its log group (/aws/lambda/lambda-rds-app), you’ll see all the log streams that represent the different instances of that Lambda function, as shown in Figure 108:

Even though this Lambda function has had many instances, and these may have been deployed across many different servers, all the logs are aggregated into CloudWatch Logs, so you can see and search them in a single place. If you click the "Search all log streams" button in the top-right corner, you can search across all your logs, as shown in Figure 109:

Try searching for a term such as "knex," since the lambda-rds-app used Knex.js to access the database. If you see some results with database queries, congrats, you’re seeing log aggregation in action! Note that Lambda also automatically includes contextual information in each log message, such as timestamp and request ID.

Log aggregation has several major advantages:

Logs are often your first step when you’re investigating a problem. But how do you know a problem happened in the first place? You could wait for users to complain, but a better way to go about it is to rely on metrics, as discussed in the next section.

Broadly speaking, you can categorize metrics into the following buckets:

The following sections will dig into each of these, starting with availability metrics.

To make use of all the different kinds of metrics you just saw, there are three steps:

Some monitoring tools are designed to do all three of these steps, while others are focused on just one or two. The following sections will take a brief look at each of these steps, starting with how to collect metrics.

In this section, you’ll see an example of how to use CloudWatch, which is a monitoring tool built into AWS, to get metrics for EC2 instances, ASGs, and ALBs. Out-of-the-box, CloudWatch provides application and server metrics for most AWS resources; you’ll also deploy Route 53 health checks to get availability metrics. Finally, you’ll gather all these metrics in a custom CloudWatch dashboard.

To get started, head into the folder you’ve been using for all the examples in this blog post series, and create a new ch10/tofu/live folder:

Next, copy your asg-sample module from Part 3 into the tofu/live folder:

The asg-sample module deploys an ASG with an ALB in front of it. It’s configured to deploy an AMI built from the Packer template in ch3/packer. If you still have that AMI ID, you can reuse it; if not, see the Packer section of Part 3 for how to build it again. Copy the ID of the AMI into the ami_id parameter in main.tf of the asg-sample module.

Next, add a Route 53 health check in main.tf, as shown in Example 189:

This code configures a Route 53 health check that will test your service from multiple endpoints around the world, with the following configuration:

Next, you can create a CloudWatch dashboard using a module called cloudwatch-dashboard, which is in the blog post series’s sample code repo in the ch10/tofu/modules/cloudwatch-dashboard folder. This module creates a dashboard that highlights a few specific metrics for the ASG, ALB, and Route 53 health checks. Example 190 shows how to update main.tf to use the cloudwatch-dashboard module:

The preceding code configures the dashboard as follows:

You should also add the dashboard URL as an output variable in outputs.tf, as shown in Example 191:

Deploy as usual, authenticating to AWS as described in Authenticating to AWS on the command line, and running init and apply in the asg-sample folder:

It will take 3-10 minutes for the ASG to deploy and for the apps to boot up, so be patient. When everything is done, you should see two output variables:

Test the URL in the alb_dns_name output to check the app is working (make sure to use HTTP, as this app from back in Part 3 does not support HTTPS):

If you see the usual "Hello, World!" text, that means the app is working. Next open up the URL in the dashboard_url output variable in your web browser. This should take you to a CloudWatch dashboard that looks similar to Figure 110:

If the dashboard is empty, wait a few minutes for the Route 53 health checks to make requests to your app, and you should start seeing the following metrics:

Congrats, you’re now seeing a small set of useful application, server, and availability metrics! If you want to see all available metrics in CloudWatch, head over to the CloudWatch metrics dashboard; if you want to see more info on the availability metrics, head over to the Route 53 health checks dashboard.

When you’re done testing, you may wish to run tofu destroy to clean up your infrastructure, so you don’t accumulate any charges. Alternatively, you may want to wait until later in this blog post, when you update this example code to enable alerts in CloudWatch. Either way, make sure to commit your latest code to Git.

Let’s now move on to the next section, where we’ll discuss structured events, and how they compare to other types of monitoring you’ve seen so far.

As your architecture becomes more complicated, and it evolves along the lines you saw in Section 1.1.2—e.g., you move from a monolith to microservices, from single-node systems to distributed systems, and so on—two things will start to happen:

By the time you get to the scale of massive companies like Google and Facebook, you’ll constantly be dealing with problems for which you don’t have, and never would’ve thought to have, automated tests, metrics, or dashboards. While you can anticipate that it’s worth monitoring the four golden signals (latency, traffic, errors, and saturation) in almost any company, what do you do for dealing with complex, unanticipated, and even unimaginable problems? How do you deal with the unknown unknowns? This is where observability comes into the picture.

Observability is a property of a system: in control theory, it’s a measure of how well the internal states of that system can be inferred from knowledge of its external outputs; in DevOps, it’s a measure of how well you can understand any state your software may have gotten itself into solely by interrogating that software with external tools. That means you can figure out what caused a problem, no matter how weird or unpredictable, without shipping new code, but solely through an ad hoc, iterative investigation of monitoring data. If the only data you have access to is the metrics you gathered (because you could anticipate they would be useful), then when facing novel problems, your investigations would quickly hit a brick wall. This is why companies are starting to turn to structured events.

Structured events are essentially hashmaps your service publishes that contain interesting key-value pairs describing what that service is doing. This may sound familiar: the events from the event-driven architectures you read about in Part 9 are structured events; the structured logs from earlier in this blog post are structured events; and you could even publish all the types of metrics you read about earlier in this blog post as structured events. The idea is to instrument your services to publish structured events that contain all the interesting data about that service and to set up tooling to consume these events, and store them in such a way that you can efficiently analyze what’s happening in your system. What sort of tooling is that?

Note that structured events have many dimensions (many key-value pairs): e.g., what time the event happened, what service the event happened in, what host the service was running on, the ID of the user who was accessing that service, what response code the user got from the service, how long it took to process the request, and so on. Moreover, many of these dimensions have a high cardinality, which is a measure of how many distinct values a dimension can have. For example, the user ID dimension could be set to millions of different values, depending on how many users you have in your system. So you need tooling that can efficiently store and analyze data with high dimensionality and cardinality.

I haven’t found a widely-accepted name for this sort of tooling—I’ve heard "observability platform," "application visibility platform," "software debugging tool," and others—so for the purposes of this book, I’ll use the term observability tooling specifically to refer to tools for storing and analyzing structured event data with high dimensionality and cardinality. Honeycomb released one of the first publicly-available observability tools in 2016 and did a lot of the leg work to popularize the concept of observability. Since then, other observability tools have appeared on the market, such as SigNoz, ServiceNow Observability (formerly LightStep), Chronosphere, OpenObserve, and IBM Instana, and many existing monitoring tools have started to add observability features (or at least, to use "observability" in their marketing), such as Dynatrace, New Relic, and Datadog.

The ability of observability tools to efficiently slice and dice data with high dimensionality and cardinality is what allows you to iteratively interrogate your systems without knowing in advance what questions you’ll need to ask. For example, the following shows the steps of a sample debugging session using an observability tool to iteratively figure out the cause of latency spiking on your website:

While you can set up metrics and alerts ahead of time for something like latency, there’s no way to know ahead of time all the queries you needed to do in the debugging session. And even if you added metrics and dashboards for some of those queries, in a complicated architecture, the next issue you have to debug might require a bunch of novel queries again. This sort of iterative search process is where observability tools shine, as they give you the ability to find answers when you don’t know, and can’t know, what questions you’ll have to ask ahead of time.

Another class of tools that falls under the observability umbrella are tracing tools, as discussed in the next section.

When you have an architecture with many services, a single request from a user may result in dozens of internal requests within your architecture. For example, when your frontend service receives a request, it may send requests to services A, B, and C; service A, in turn, may send requests to services D, E, and F, while service B sends requests to services G, H, and I; and so on, as shown in Figure 111.

The diagram in Figure 111 is static, but in the real world, your services and request flows are going to be changing all the time, and beyond a certain scale, understanding the request flow, and debugging problems within it, can become a serious challenge. To solve this problem, many companies turn to distributed tracing, which is a method of tracking requests as they flow through a distributed system.

The basic idea is that when you get the request that initiates the flow, such as a user’s request to a frontend, you instrument your code to assign a unique ID to that request, called a trace ID, and you propagate that trace ID throughout all the downstream requests. Each of your services can then publish structured events that include the trace ID, along with other useful data such as a timestamp and how long the request took to process, and a distributed tracing tool can stitch all these events back together. That way, for every request, you can get a dynamically-generated diagram similar to what you saw in Figure 111, or a waterfall diagram similar to Figure 112:

Waterfall diagrams consists of a series of nested spans, one for each request, which show you which services were involved, the order of requests (which reveals dependencies between services), how long the requests took, and other useful metadata. These sorts of diagrams help with debugging problems and performance tuning in a microservices architecture.

There are a number of tools available for instrumenting your code and collecting and analyzing distributed traces. Some of the early players in the space were Zipkin, Jaeger, Honeycomb, and LightStep, which was acquired by ServiceNow and is now known as ServiceNow Observability. Later on, many other distributed tracing tools have appeared on the scene, and many existing monitoring tools added varying levels of support for distributed tracing as well, including SigNoz, Grafana Tempo, Uptrace, Dynatrace, New Relic, Datadog, AWS X-Ray, and Google Cloud Trace. These days, using OTel is often the best approach, as it’s open source, works with many different programming languages, orchestration tools, and metrics backends, and it’s a single toolkit you can use to collect all your telemetry data (i.e., metrics, logs, and traces). Also, as you saw in Part 7, some service mesh tools provide distributed tracing out-of-the-box, such as Istio, which uses OTel under the hood.

You’ve now seen a number of tools for understanding what’s happening in production. If you get good enough with these tools, that opens the door to an approach known as testing in production, which is the focus of the next section.

One way to think of monitoring is as the yin to the automated testing yang. That is, you use automated tests to catch some types of errors, and you use monitoring to catch other types. Automated testing is great for catching errors when (a) you can anticipate the error and (b) you can simulate production sufficiently to replicate the error. As you saw in the observability section, as your company scales, you’ll hit more and more errors that you can’t anticipate. Similarly, as your company scales, you’ll find it harder and harder to simulate production. If you ever get to the scale of a Google or Facebook, where you have billions of users and petabytes of data, there’s just no practical way to have a faithful copy of production.

As a result, some larger companies don’t bother. That is, they don’t have staging environments. Instead, they do what has been called testing in production (TIP), where you test your software by deploying it to production, and seeing what happens. This may sound reckless and dangerous, and if not done correctly, it can be. However, the reality is that we all test in production to some extent: every deployment, every configuration change, every single user request is a test. It might work, or might hit a bug. The idea with TIP is that, at a certain scale, the effort to do automated testing in pre-production environments becomes so high, that you may actually be better off investing that effort into making your systems more resilient. Instead of spending thousands of hours trying to catch every bug in testing, you spend that time reducing the impact of bugs.

This doesn’t mean you throw all automated testing out the window, and just thoughtlessly start tossing untested code into production while screaming Leeeeeeerooooooy Jenkins. If anything, TIP requires more discipline to do well, including the following:

If you do TIP correctly, you’ll see significant improvements in all your DORA metrics. Deployment frequency will go up and lead times will go down, as deploying will be something every team does dozens or hundreds of times as part of their normal coding process, rather than only at the end. Also, change failure rates and recovery time will go down, as your tooling will be designed to limit the blast radius and quickly recover from errors.

That said, testing in production is not without risks and challenges. For each feature you work on, you will constantly need to balance how much time you invest in testing before production versus how much testing you want to do in production. When the cost of bugs is high, you’ll want to lean more heavily towards automated testing before production, such as for the following types of features:

For these sorts of features, do as much testing before production as you possibly can. On the other hand, when the cost of bugs is low, you’ll want to lean more heavily towards testing in production. For example, if an employee at Facebook is testing in production, and that leads to a bug that breaks the news feed for a small number of users, it’s just not that big of a deal. For this very reason, Facebook’s motto used to be "move fast and break things." They eventually changed this motto to "move fast with stable infra," which in some ways, does an even better job of encapsulating the TIP philosophy of speed and resiliency.

That said, things will go wrong even with the most resilient systems, and when they do, you’ll want to be alerted about it. This is the focus of the next section.

The idea with alerts is to define triggers or rules for when a notification should be sent to someone. You typically define triggers in terms of various metrics, events, or logs exceeding one of the following types of thresholds:

Once you’ve defined a bunch of triggers, the next question is what to do when a trigger goes off. Whenever possible, the default action should be to deal with it using automation, such as auto healing (e.g., replace a server or container) or auto scaling (e.g., scale the number of servers or containers up or down). If you can’t handle a trigger automatically, the next step is to notify a human, as discussed in the next section.

Although this entire section is about alerts, the reality is that you should use alerts sparingly. I’m specifically referring to the kinds of alerts that notify a person, interrupting whatever they are doing, such as an alert on a pager or, as is more common these days, a smartphone. Alerts of this kind have a significant cost. First, they interrupt work. I’ve seen some companies where alerts go off all the time, so their operations team never has time for anything other than "fire fighting." Second, they interrupt personal life. Since alerts may come up at any time—in the middle of dinner, in the middle of the night, or during your Christmas holiday—they can significantly interfere with your personal life.

If alerts go off too often, your team will experience alert fatigue, where they will become stressed, unproductive, and unhappy. In the best case, they’ll start to ignore alerts, like in The Boy Who Cried Wolf parable. In the worst case, your team members will burn out and leave. To avoid alert fatigue, you should only use an alert if the trigger meets all the following criteria:

If a trigger doesn’t meet all of these criteria, then instead of an alert, you should send a non-urgent notification, such as filing a ticket in an issue tracking system or sending a message to a chat room (e.g., in Slack or Teams) or mailing list that your operations team checks from time-to-time. That way, you know someone will see and investigate the issue, without forcing them to drop everything to do it.

If a trigger does meet all of these criteria, then you send an alert, typically to the team members who are on-call, as per the next section.

When you’re a tiny startup, if there’s an incident, you might alert everyone. As you grow, this becomes inefficient and counter-productive, so it’s more common to have an on-call rotation, where you create a schedule that assigns specific team members to be responsible for responding to alerts at specific times. In the past, you might have managed the on-call rotation with a spreadsheet and had the on-call team members carry pagers, but these days, it’s more common to use dedicated software to manage the rotation and send alerts to smartphones. The major players in this space include PagerDuty, Splunk On-Call (formerly known as VictorOps), Opsgenie, Blameless, Squadcast, All Quiet, and Rootly.

Being on-call is stressful, so here are a few practices to consider:

If you’re on-call, and an alert goes off, you’ll need to resolve the incident by going through an incident response process, as defined in the next section.

Whenever there is an incident, your team should follow an incident response plan, which details the steps to resolve that issue. In small startups, this is often an ad hoc process, but as your company grows, you’ll want to get the plan down in writing; in fact, this is a requirement of many compliance standards (e.g., SOC 2). This gives your on-call teams a systematic, repeatable way to resolve incidents, which should give them more confidence and allow them to resolve issues faster (as compared to, say, randomly pushing buttons and hoping it works). Every incident response plan is different, but most plans will include the following items:

Now that you’ve seen the basics of alerts, let’s try an example using CloudWatch.

Earlier in this blog post, you created an asg-sample module that deployed an ASG with an ALB, Route 53 health checks, and a CloudWatch dashboard with some metrics. Let’s set up an alert for one of those metrics—the Route 53 health check—using CloudWatch alarms. You’ll want the trigger to go off when the HealthCheckStatus metric drops below 1, as that indicates the website is down. When the trigger goes off, it will publish a message to Amazon Simple Notification Service (SNS), and you can subscribe to SNS to get notifications via email or SMS.

Add a CloudWatch alarm to main.tf in the asg-sample module as shown in Example 192:

Here’s what this code does:

To make it easier for you to test out this alarm, you can also update the asg-sample module to automatically subscribe you to the SNS topic as shown in Example 193:

The preceding code does the following:

Run apply to deploy these changes:

After a minute or two, you’ll get an email confirming your SNS subscription. Make sure to click the link in the Subscribe URL. Now that you’re subscribed, and the alarm is deployed, let’s see if it works. To do that, you need to do something fun: break your own sample app! You can get creative with how you do this. One option is to go into the AWS Load Balancers Console, select the sample-app ALB, and click Actions > "Delete load balancer." After a few minutes, the Route 53 health checks should start failing, and you should get an email similar to Figure 113:

If you see an email with subject "ALARM: sample-app-is-down," then congrats, alerts are working! You now have a way to monitor your software 24x7, and to be notified as soon as anything goes wrong.

If you want to fix the alert you’re seeing, you can run apply one more time to redeploy whatever infrastructure you destroyed. If you’re done testing, commit your latest code to Git, and then run destroy to clean everything up.