How to Manage Your RabbitMQ Logs: Tips and Best Practices

Let’s break down our best tips for the most effective ways to manage your RabbitMQ logs…

RabbitMQ is an open-source message broker that allows you to build distributed systems and implement messaging-based architectures. It’s a reliable and scalable messaging system that enables efficient communication between different parts of your application.

Many teams start out with a small, single node RabbitMQ installation, where failure of the system is easily trackable. As time goes on, the applications rely more and more on RabbitMQ, which necessitates increasing the size of the installation, clustering, or deploying more installations. At the same time, managing and monitoring these installations becomes more crucial.

In this post we’ll take a look at the logging configuration of RabbitMQ, how logging works in RabbitMQ, what are the different options for outputting these logs to different backends and shipping logs to external systems.

Understanding RabbitMQ Logs

The life of logs.

RabbitMQ logs are a form of mostly unstructured information generated by RabbitMQ’s internal subsystems. For example, a log message is produced whenever a node goes down or comes up, or if any errors are encountered during processing a message or by any internal process, but these messages are generally not related to any messages flowing through RabbitMQ.

Before managing your RabbitMQ logs, you should first understand the different types of logs that RabbitMQ produces. There are few types of RabbitMQ logs:

Crash dumps
Regular logs
Crash logs
Internal events
Tracing logs

Crash Dumps

Crash dumps are text files generated by RabbitMQ (or any Erlang based system) anytime the application crashes. These files will show up as erl_crash.dump in the log directory of RabbitMQ. This information will not show up in the regular log files. You can open this file with a text editor, and see the header for the cause of the crash.

Saving this file for later investigation when the system crashes is very important to be able to track down the reasons why the process got into the failure state. Care needs to be taken because the size of these files can be in the tens of gigabytes, potentially causing the host to run out of disk space.

If your system created these files and you are looking for help to investigate the reasons, contact Seventh State for expert RabbitMQ Support.

Regular Logs

The log entries include timestamps, severity levels, and descriptive messages about the events that occurred. These logs can help administrators and developers diagnose problems, identify bottlenecks, analyse system performance, and detect any anomalies or misconfiguration within the RabbitMQ infrastructure.

RabbitMQ logs are categorised based on the severity and nature of the information they contain. Crash logs document unrecoverable or unhandled errors, while error Logs capture runtime errors, and warning logs highlight potential runtime issues. Info logs provide general runtime information. Debug logs offer detailed information about the RabbitMQ server’s internal processes and operations, such as message routing decisions, channel operations, and internal state changes, and may contain sensitive information.

Crash Logs

Anytime you see a multiline log message which displays the “crasher” and “initial call” information, we are talking about a process crash log. This signals an internal process crash, which may be caused by bugs in RabbitMQ, however in many cases they can be ignored as well. These logs are included in the regular log outputs. For example:

			
[error] <0.27962.6>   crasher:
[error] <0.27962.6> initial call: rabbit_channel:init/1
[error] <0.27962.6> pid: <0.27962.6>
[error] <0.27962.6> registered_name: []
[error] <0.27962.6> exception exit: noproc
[error] <0.27962.6>   in function  gen_server2:terminate/3 (gen_server2.erl, line 1183)
[error] <0.27962.6> ancestors: [<0.27958.6>,<0.27935.6>,<0.27891.6>,<0.27889.6>,<0.16617.0>,
[error] <0.27962.6>               <0.16616.0>,<0.16615.0>,<0.16613.0>,<0.16612.0>,rabbit_sup,
[error] <0.27962.6>               <0.218.0>]
[error] <0.27962.6> message_queue_len: 1
[error] <0.27962.6> messages: [{'EXIT',<0.27958.6>,shutdown}]
[error] <0.27962.6> links: []

		

If your installation displays such messages and you are looking for help to investigate the reasons, contact our Seventh State RabbitMQ Support.

RabbitMQ Logging Location

Use RabbitMQ management UI or rabbitmq-diagnostics -q log_location to find where a node stores its log file(s).

By default, the logging directory is: /var/log/rabbitmq

The value for the log file location is picked up from an environment variable named RABBITMQ_LOGS.

RABBITMQ_LOGS=/data/logs/my-rabbit-logs.log

Changing this environment variable will change the RabbitMQ log file name.

The value for the whole log directory can be changed by utilising the RABBITMQ_LOG_BASE environment variable:

RABBITMQ_LOG_BASE=/data/rabbitmq-logs You can find more information about the other available environment variables here.

RabbitMQ Logging Location on Windows

On Windows logs are produced by default to a directory which belongs to the user who installed RabbitMQ.

If you are logged in by the user who installed RabbitMQ, then you can reach this directory by opening the following path: %AppData%\RabbitMQ in Explorer. For example if RabbitMQ is started by a user Administrator, the %AppData% directory is set to C:\Users\Administrator\AppData\Roaming.

For this reason, in many environments the location may be changed as outlined above.

RabbitMQ Log Categories

Categories are the first level filters for logs in RabbitMQ. Each log message is sorted into one category.

connection: connection lifecycle events for AMQP 0-9-1, AMQP 1.0, MQTT and STOMP.
channel: channel logs. Mostly errors and warnings on AMQP 0-9-1 channels.
queue: queue logs. Mostly debug messages.
mirroring: queue mirroring logs. Queue mirrors status changes: starting/stopping/synchronising.
federation: federation plugin logs.
upgrade: verbose upgrade logs. These can be excessive.
ra: Log related to the Quorum Queue backend.
default: all other log entries. You cannot override file location for this category.

For each category, it is possible to configure the level. For example, it is common to disable info messages for connections, as they can increase the size of the logs and affect performance in certain scenarios (However, we definitely do not recommend this if not needed!).

You can achieve this, by placing the following into rabbitmq.conf:

log.connection.level = warning

By configuring none as the level, you can turn off all logs from the category:

log.<category>.level = none

For each category, it is also possible to redirect the outputs to a different file, for example:

log.ra.file = /data/rabbitmq-logs

This can be useful for the same reason outlined above for the levels, performance, or the amount of logs produced.

RabbitMQ Log Outputs

In RabbitMQ log outputs allow you to configure where the logs produced will be sent. Most commonly the file and console outputs are used.

For each output it is possible to configure the log level separately. This means that it is possible to output debug logs to a file, while not printing them to the console.

The File Output

Logging to a file is the default option.

Logging to a file can be disabled by inserting the following line into rabbitmq.conf:

log.file = false

Console or Stdout

On Unix based systems RabbitMQ will log to the stdout for a short time on startup. If RabbitMQ does not start up, most probably the reason will end up there.

You can query these logs by using the following command:

journalctl -u rabbitmq-server --no-pager

When you are using containers, such as with Docker, or Podman, Kubernetes, all logs will go to the stdout by default.

You can configure RabbitMQ to log to stdout by placing the following into the config file:

log.console = true

As with all changes to the config files of RabbitMQ, you must restart the instance.

Syslog

In most installations RabbitMQ produces logs to files or the console and then these logs are shipped by an external tool, however you can configure RabbitMQ to send directly to the log storage using the Syslog Protocol.

You must set up a syslog server to receive the logs as well. You can use Graylog or Logtash to create a syslog listener.

You can turn on the syslog output by placing the following line to the rabbitmq.conf:

log.syslog = true

Make sure to configure the correct protocol RFC for the system, otherwise logs might not make it through, and have to make sure to use the correct transport protocol as well:

log.syslog.protocol = rfc5424 or rfc3164 log.syslog.transport = tls or tcp or udp

You can find more information on the other required information in the RabbitMQ documentation.

Advanced configuration of the syslog logging module allows you to configure fields, verbosity, and formatter for fields.

Journald

RabbitMQ also supports sending logs to journald by enabling the output:

log.journald = true

Setting Log Levels Dynamically and Persistently

As mentioned earlier, RabbitMQ logs are sorted into multiple severities. By default, RabbitMQ logs at the info level.

It can be useful for debugging to set the log level temporarily to debug:

rabbitmqctl set_log_level debug

If you want to reduce the amount of logs RabbitMQ produces, it is best to set the log level to error. This can be useful when there is a suspicion that RabbitMQ is being overloaded by the volume of logs it generates. However, it significantly limits the ability to debug effectively.

Log Rotation

Log Rotation of RabbitMQ on Unix System With Logrotate

If you’ve installed RabbitMQ from a package on a Unix based system, you get log rotation out of the box, because the installer will set up the required configuration files in /etc/logrotate.d/rabbitmq-server.

By default, it will keep 20 files, rotated weekly. Make sure to configure this according to the disk space allocated to your RabbitMQ instances to not run out of disk space.

If you’ve configured a different log location, don’t forget to change these settings.

Log Rotation Built Into RabbitMQ

In some cases you may want to set up log rotation inside RabbitMQ rather than logrotate or some other external tool. This is especially useful on Windows.

Rotation Based on date

It is possible to rotate logs based on time:

# rotate every night at midnight log.file.rotation.date = $D0

In case midnight is not suitable for you, you can configure 10 AM the following way:

log.file.rotation.date = $D10

Based On File Size

It is possible to rotate the logs based on the size of the file as well. This can be useful if you want to make sure to limit the disk space used by logs.

# rotate when the file reaches 10 MiB log.file.rotation.size = 10485760

Keeping Old Logs

It is important to keep a reasonably long (multiple weeks) of log history to be able to investigate errors. In many cases errors are not noticed or logs are not collected in a timely manner when issues happen. If logs rotate out by the time you notice an issue, it may be difficult or impossible to investigate what went wrong.

RabbitMQ allows you to configure how many past files you want to keep. Works best with time based rotation!

# keep 14 files (including the active log file) log.file.rotation.count = 14

JSON Formatting of Logs

In most production environments, you would want to use a third-party tool to analyse all of your logs. This becomes easier when the logs are formatted in a way that does not require special parsing of log lines by an outside tool. In most cases, this is the JSON format.

For each output, it is possible to configure a formatter separately. For example:

			
log.file.formatter = json
log.console.formatter = json
log.syslog.formatter = json

Other Advanced Logging Features

We are not going to go into a lot of details about more advanced logging features of RabbitMQ, however at the level of mentioning there are some additional useful logging features of RabbitMQ which you may consider checking out.

RabbitMQ is able to generate its own logs as messages into the queues as well using the exchange output. This can be useful for some use cases but places more load on RabbitMQ. You can find more information about it here.

RabbitMQ produces some structured internal events as well, such as an event when a queue is created or deleted, or a user was edited. These events can be consumed as messages. You can find more information about this feature here.

RabbitMQ also has a tracing feature, which can greatly help you track down issues with applications and RabbitMQ itself. You can find more information about tracing here.

All of these advanced features place additional load on RabbitMQ, therefore their use must be carefully considered to not overload the broker.

Maintaining RabbitMQ

Managing RabbitMQ logs is an essential part of maintaining a reliable and efficient RabbitMQ setup. However, even with the best logging practices in place, issues can still arise.

I usually work with RabbitMQ following these steps:

– Check the Logs Regularly: Checking the logs on a regular basis can help you identify any issues as they arise, allowing you to address them promptly, also help you to track trends and identify patterns of behaviour that may indicate potential issues.

– Look for Error Messages: When reviewing your RabbitMQ logs, pay close attention to any error messages that appear. Error messages can provide valuable clues about potential issues with your RabbitMQ setup. Address any warnings or errors promptly to prevent them from becoming larger issues down the road.

– Monitor Restarts: Look for any restarts of the RabbitMQ instances in the logs. Verify that all these were planned restarts and not due to, for example, Out-of-Memory.

– Monitor Resource Usage: Monitor resource usage on your RabbitMQ server, including CPU, memory, and disk space. If any of these resources are running low, it can cause issues with your RabbitMQ instance.

Conclusion

In some installations ample amounts of logs are generated, while some use cases are very silent. Remember to set appropriate log levels, use a monitoring tool to keep track of your logs, and be proactive in identifying and resolving issues.

If you still need help to manage your RabbitMQ logs, contact our RabbitMQ troubleshooting experts today

“With these practices in place, you can make sure that your RabbitMQ system is reliable and performing at its best. If you require any help with your RabbitMQ environment reach out to myself or the team. “

Lajos Gerecs

RabbitMQ Consultant at Seventh State