A simple way to manage any kind of log messages from containers: AWS CloudWatch Logs

Gone are the days when administrators logged into their machines to access log files. Containers and virtual machines are launched and terminated dynamically to scale based on demand, to deploy new versions, or to recover from failure nowadays. Collecting, monitoring and analyzing log messages in a centralized data storage has become a minimum requirement for production-ready systems.

You will learn how to use CloudWatch Logs to manage log messages from multiple containers in the following.

What are the CloudWatch Logs?

CloudWatch Logs is a managed service offered by AWS providing scalable, easy-to-use, and highly available log management. I do like to use CloudWatch Logs to collect, monitor, and analyze your log messages because of its simplicity. AWS covers the basics of log management.

The following figure shows the main elements of CloudWatch Logs:

  • Group a bucket for your log messages, comparable with an S3 bucket.
  • Stream processes up to 5 MB per second, use multiple streams to scale log data ingestion.
  • Event contains the log message.

Additionally, you can use metric filters to monitor incoming log messages. CloudWatch logs are priced per amount of ingested data, stored data and transferred data. See CloudWatch Pricing for details.

Docker logging drivers

Docker can forward log messages from stdout and stderr to different targets. You can use the following built-in logdrivers: none,local, json-file, syslog, journald, gelf, fluentd, awslogs, splunk, etwlogs, logentriesand gcplogs. See Supported logging drivers for details.

Use the awslogs logging driver to send logs from your container to CloudWatch Logs without the need of installing any additional log agents. As shown in the following figure the container writes log messages to stdout and stderr. By default, docker logs or docker service logs shows the command’s output just as it would appear if you ran the command interactively in a terminal. UNIX and Linux commands typically open three I/O streams when they run, called STDIN, STDOUT, and STDERR. STDIN is the command’s input stream, which may include input from the keyboard or input from another command. STDOUT is usually a command’s normal output, and STDERR is typically used to output error messages. By default, docker logs shows the command’s STDOUT and STDERR. To read more about I/O and Linux, see the Linux Documentation Project article on I/O redirection.

In some cases, docker logs may not show useful information unless you take additional steps.

  • If you use a logging driver which sends logs to a file, an external host, a database, or another logging back-end, docker logs may not show useful information.
  • If your image runs a non-interactive process such as a web server or a database, that application may send its output to log files instead of STDOUT and STDERR.

In the first case, your logs are processed in other ways and you may choose not to use docker logs. In the second case, you can use the following command in your Dockerfile.

# forward request and error logs to docker log collectorRUN  ln -sf /dev/stdout /var/log/nginx/access.log \&& ln -sf /dev/stderr /var/log/nginx/error.log

This creates a symbolic link from /var/log/nginx/access.log to /dev/stdout, and creates another symbolic link from /var/log/nginx/error.log to /dev/stderr, overwriting the log files and causing logs to be sent to the relevant special device instead.

That’s all you need to send log messages from a single container to CloudWatch Logs. But how to send log messages from hundreds of containers to CloudWatch Logs? Learn how to integrate CloudWatch Logs with ECS (Fargate).

Summary

When looking for an easy way to manage your container logs on AWS, CloudWatch Logs is a good choice. Docker comes with a built-in logging driver for CloudWatch Logs: awslogs. No need to install and run any additional log collecting agents. CloudWatch Logs scales automatically so you can use it for a single container or thousands of containers running on ECS.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dilshan Fernando

Quality Engineering | Test Automation Engineer | AWS Certified Solutions Architect | Problem Solver