SAS Viya Platform: Metric Monitoring

rabbitmq_queue_messages

This metric represents the total number of messages waiting in RabbitMQ queues on a node, including both messages that are ready to be delivered and messages that have been delivered but not yet acknowledged.

rabbitmq_queue_messages > N

N is your organization's threshold. It usually indicates that messages are accumulating faster than they are being processed. The expectation is that this is near to 0.

rabbitmq_consumers < C

C is specific to your deployment.

rabbitmq_queue_messages_ready

Total messages ready to be delivered to consumers across all queues

rabbitmq_queue_messages_unacked

Total unacknowledged messages across all queues

RabbitMQ queues should normally stay mostly empty because messages are consumed quickly. When the total queue depth (ready + unacknowledged messages) grows beyond a deployment-specific threshold, it indicates a backlog and may signal slow consumers, failed consumers, overloaded systems, or downstream processing problems.

RabbitMQ messages ready versus RabbitMQ messages unacknowledged tells you where a problem might be.

• High ready messages might mean that consumers are not keeping up with incoming message volume. Most messages are waiting in queues rather than being actively processed.
• High unacknowledged messages might mean that consumers have accepted many messages but are processing them slowly or are stuck waiting on some dependency.

For the following cluster-wide counter metrics, calculate the rate of change over time using Prometheus rate() over a time window equal to 4 scrape intervals. This converts the counters into useful metrics such as messages received per second, acknowledged per second, delivered per second, and redelivered per second, making it easier to identify backlogs, consumer failures, or crash loops.

rabbitmq_global_messages_received_total

Messages accepted from publishers

rabbitmq_global_messages_acknowledged_total

Messages acknowledged by consumers

rabbitmq_global_messages_delivered_consume_manual_ack_total

Messages delivered to consumers

rabbitmq_global_messages_redelivered_total

example: rate(rabbitmq_global_messages_redelivered_total[60s]) > 0

Redeliveries

If the number of message redeliveries continues increasing at a steady or high rate over time, it often indicates that one or more consumers are repeatedly failing after receiving messages, causing RabbitMQ to redeliver those messages again and again.

When one or more monitored resource thresholds in a cluster exceed their configured limits, the system prevents publishers from sending additional data, messages, or events across the entire cluster until conditions return to acceptable levels.

Set alerts when resources are low.

rabbitmq_process_resident_memory_bytes

rabbitmq_process_resident_memory_bytes > .08

Node memory that is in use

rabbitmq_resident_memory_limit_bytes

rabbitmq_resident_memory_limit_bytes > 0.8

Memory high watermark in bytes

rabbitmq_disk_space_available_bytes

rabbitmq_disk_space_available_bytes < 2.0

Free disk on data partition in bytes

rabbitmq_disk_space_available_limit_bytes

rabbitmq_disk_space_available_limit_bytes < 2.0

Free disk space low watermark in bytes

When RabbitMQ reaches its file descriptor limits, new network connections are not accepted. (RabbitMQ uses file descriptors for virtually every client connection and many internal operations.)

rabbitmq_process_open_fds

rabbitmq_process_open_fds > 0.8

rabbitmq_process_max_fds

rabbitmq_process_max_fds > 0.8

RabbitMQ is built on Erlang, which uses lightweight processes rather than operating system threads. RabbitMQ creates many Erlang processes to handle the following:

• Connections
• Channels
• Queues
• Message routing
• Replication
• Management operations

Erlang scheduler threads run these processes on available CPU cores. The following metric returns the number of runnable processes waiting for a scheduler thread:

rabbitmq_erlang_scheduler_run_queue

If the value remains greater than about one waiting process per CPU core, RabbitMQ is likely overloaded and cannot keep up with incoming work, which may lead to increased message latency, growing queues, and reduced throughput.

Use this threshold or alert condition to indicate RabbitMQ may be severely CPU-constrained:

rabbitmq_erlang_scheduler_run_queue > (num_cpus * 1.5)

rabbitmq_connections_opened_total

Cumulative connections opened

If the number of new RabbitMQ connections being opened increases rapidly over time, clients might be repeatedly disconnecting and reconnecting instead of maintaining stable connections.

Use a rate() calculation to determine how quickly new connections are being created. If the rate() is high, a reconnect loop occurs.

rabbitmq_channels_opened_total

rate(rabbitmq_channels_opened_total)

Cumulative channels opened; A channel is a lightweight virtual connection that exists inside a RabbitMQ connection.

If RabbitMQ is continuously opening channels at a high rate, applications may be creating channels but not properly closing or reusing them, resulting in a channel leak.