Why Is the RabbitMQ Disk Space Alarm Triggered and How Do You Fix It?

A RabbitMQ disk alarm is one of the most disruptive incidents a broker can produce. When the alarm is active, every publishing connection in the cluster is blocked, not just on the affected node. Production traffic stops upstream, queues stop growing on disk, but they also stop being drained because the broker is busy refusing publishes. Most teams hit a disk alarm at least once before they understand how disk_free_limit is meant to be configured.

This guide explains what triggers the disk alarm, what to check when one fires, how to safely clear it, and how to prevent recurrence. The four main causes covered are persistent message backlogs, quorum queue Raft log growth, broker logs sharing the data volume, and a disk_free_limit left at the default.

Quick Answer

The RabbitMQ disk alarm fires when free disk on the broker’s data partition falls below the configured disk_free_limit (default 50 MB, which is unusable in production); while the alarm is active, publishing is blocked cluster-wide. First checks are rabbitmq-diagnostics alarms, rabbitmq-diagnostics status (which prints “Free disk space” and “Low free disk space watermark”), and the management UI cluster overview. The most common causes are persistent message backlogs, quorum queue Raft log growth, broker log files on the same volume as the data directory, and disk_free_limit left at the default. Fix it by freeing disk and addressing the underlying cause; treat lowering disk_free_limit as a last resort.

What the Disk Alarm Actually Does

When RabbitMQ detects that free space on the data partition is at or below disk_free_limit, it raises a disk resource alarm. The behaviour of this alarm is different from the memory alarm in three important ways:

• It is cluster-wide. If one node is over the limit, every node blocks publishing.
• It cannot be cleared by paging messages in memory; it can only be cleared by actual disk coming free.
• The check frequency increases as the broker approaches the limit, from every 10 seconds down to 10 times per second, so the alarm fires quickly once the disk is almost full.

Consumers and management traffic are unaffected. The broker is deliberately refusing new work so it can keep flushing existing state to disk without running out of room mid-write.

Categories Covered

• Persistent message backlog (the most common cause).
• Quorum queue Raft logs growing faster than they compact.
• Broker logs, audit logs, or core dumps sharing the data volume.
• disk_free_limit left at the default 50 MB and disk genuinely full.

Issue 1: Persistent Message Backlog Filling the Disk

What it looks like

• Alarm fires after a long period of normal operation.
• Queue depth on persistent queues has been rising steadily.
• The management UI shows the disk alarm under the cluster overview.
• du -sh on the data directory points at the message store or quorum queue Raft logs as the dominant consumer.

What is happening

Persistent messages are written to disk by design. When publish rate exceeds consume rate on persistent queues, the broker accumulates those messages on disk and the data directory grows. If queue length is not bounded, the only ceiling is the disk itself.

First checks

Bash

How to fix it

Mitigate by draining the backlog if you have control of the producer or consumer side: scale consumers, restart stalled consumers, or temporarily throttle producers at the application level. If the backlog cannot be drained, free disk by other means (rotate or move broker logs off the data volume, expand the volume).

The durable fix is to bound the queues. Apply queue length limits with x-max-length or x-max-length-bytes so the broker drops or dead-letters the oldest messages instead of letting the disk fill. Plan disk capacity around steady-state queue depth plus a comfortable safety margin.

Common mistake

Lowering disk_free_limit to clear the alarm. The limit exists to give the broker headroom to write. Setting it lower removes the protection and risks the broker running out of disk mid-write, which can corrupt state. Cleared-by-lowering is a symptom-management move that often becomes the next incident.

Prevention

Monitor disk_free and queue length on every persistent queue. Alert when free disk falls below twice the configured limit, not at the limit itself. Apply length limits to any queue that does not have a known upper bound. Right-size the data volume for expected backlog under failure conditions, not just under happy-path load.

Issue 2: Quorum Queue Raft Logs Growing Faster Than They Compact

What it looks like

• Cluster runs quorum queues.
• Data directory growth is concentrated in the quorum subdirectory.
• Disk usage climbs even when queue depth metrics look stable.
• The alarm fires under heavy load and clears slowly after load drops.

What is happening

Quorum queues use a Raft log on disk. Under heavy publish load, the Raft log grows continuously and compaction (which reclaims disk by removing acknowledged entries) runs in the background. If the log grows faster than compaction can keep up, disk usage rises even though queue depth metrics may not reflect it directly.

First checks

Bash

How to fix it

Mitigate by reducing publish rate, scaling consumers to accelerate ack flow (compaction depends on acked entries), or temporarily moving traffic off the affected queue. The durable fix is capacity: quorum queues require materially more disk headroom than classic queues did. The RabbitMQ team’s published recommendation is to provision disk that comfortably exceeds the WAL footprint; verify the current guidance for the deployed version before committing to a specific multiple.

Common mistake

Sizing disk based on average message backlog. Quorum queue disk footprint is driven by the WAL and pending acks, not by visible queue depth. A queue with low messages_ready can still hold a large Raft log if unacked messages are accumulating.

Prevention

Monitor disk usage on the data directory directly, not just messages_ready. Alert on messages_unacknowledged growth as a leading indicator of Raft log growth. When migrating from classic mirrored queues to quorum queues, plan a disk capacity increase as part of the migration.

Version note

Classic mirrored queues were removed in RabbitMQ 4.0. Quorum queue disk behaviour and compaction performance have improved across 4.x; verify the latest guidance against the deployed version.

Issue 3: Logs and Other Files on the Data Volume

What it looks like

• Disk usage on the data partition climbs steadily even when broker traffic is light.
• du -sh shows broker logs, audit logs, or older core dumps consuming significant space alongside the data directory.
• The alarm fires unexpectedly after a long-running session.

What is happening

The broker writes operational logs, sometimes audit logs, and occasionally core dumps. If these are written to the same volume as the data directory, they compete for the same disk budget. A long-running broker with verbose logging will eventually fill the volume.

First checks

Bash

How to fix it

Mitigate by truncating or rotating logs. The durable fix is to move broker logs off the data volume entirely, configure log rotation, and review log levels. Audit logs should be shipped off-broker rather than retained locally.

Common mistake

Leaving info-level logs without rotation on a small data volume. Over weeks, a default install can accumulate gigabytes of logs even on a quiet broker.

Prevention

Configure logrotate or equivalent. Mount broker logs on a separate volume from the data directory. Ship logs to a central log store so local retention is short.

Issue 4: disk_free_limit at the Default 50 MB

What it looks like

• The alarm fires when there is still substantial free disk available, because the limit itself is so low that the disk genuinely is approaching exhaustion before any alarm can help.
• Alternatively, the alarm fails to fire until the disk is dangerously full, leaving no headroom for recovery.

What is happening

The default disk_free_limit is 50 MB. This is too low for any production workload. The official production checklist recommends a minimum disk_free_limit roughly equal to the configured memory high watermark (for example, 4 GB if the broker has a 4 GB memory budget) so the broker has room to flush memory to disk during an alarm scenario.

First checks

How to fix it

Durable fix: set disk_free_limit.absolute in rabbitmq.conf to a value matched to the broker’s memory budget. For containerised deployments, prefer absolute over relative because the relative value is computed against host memory rather than the cgroup limit.

Ini, TOML

# /etc/rabbitmq/rabbitmq.conf

disk_free_limit.absolute = 4GB

When both disk_free_limit.absolute and disk_free_limit.relative are set, the absolute value takes precedence in supported RabbitMQ versions.

If an alarm is already active, the immediate mitigation is to free disk first (see Issue 1); do not lower the limit to clear the alarm.

Common mistake

Setting disk_free_limit.relative = 1.0 inside a container expecting the broker to use the container’s memory limit. The relative calculation uses host memory, which on a cgroup-constrained container can produce a much larger value than expected and effectively disable the alarm.

Prevention

Set disk_free_limit.absolute explicitly in production. Confirm the effective value with rabbitmq-diagnostics -q status after every config change. Document the chosen value alongside the memory watermark and the volume size so capacity reviews stay consistent.

Summary Table

Issue	Primary signal	Likely cause	First check	Immediate mitigation	Durable fix	Version note
Persistent backlog	Queue depth rising on persistent queues	Publish rate exceeds consume rate on unbounded persistent queues	list_queues messages_persistent	Drain backlog, free disk	Queue length limits, capacity	n/a
Quorum Raft log growth	quorum subdirectory growing	Raft log grows faster than compaction reclaims acked entries	du -sh .../quorum, quorum_status	Reduce publish rate, scale consumers	Provision more disk headroom	Quorum queues need more disk than 3.x mirrored queues
Logs on data volume	Logs in du -sh output dominate	Broker logs, audit logs, or core dumps share the data volume	find -size +100M on data volume	Truncate or rotate logs	Move logs to separate volume	n/a
Default disk_free_limit	Alarm fires unexpectedly or too late	Default 50 MB limit is too low for production	rabbitmq-diagnostics status(free disk watermark)	Set disk_free_limit.absolute	Match limit to memory watermark	Absolute beats relative in containers

Metrics to Monitor

Metric	What it tells you	Related issue
disk_free	Remaining disk capacity	All disk alarm causes
disk_free_limit	The threshold for alarm	Configuration drift
messages_persistent	Persistent backlog volume	Persistent backlog
messages_unacknowledged	Unacked work driving Raft log	Quorum queue growth
Data directory size on disk	Actual on-disk footprint	Logs, Raft growth