When a Queue Table Isn't Enough: Moving to RabbitMQ

A surprising number of production systems handle background work the same way: a jobs table, a status column (pending, processing, done, failed), and one or more workers that poll it on a loop, picking up pending rows and updating their status as they go. This is a completely reasonable way to start. It needs no new infrastructure, it's easy to inspect (SELECT * FROM jobs WHERE status = 'failed' is a debugging tool everyone already knows how to use), and for low volumes it works without issue for a long time.

The signs that it's outgrown this aren't usually "too much volume" in the abstract. They're more specific than that.

The polling itself becomes the load

Every worker checking the table every few seconds is a query, regardless of whether there's any work to do. At low volume this is negligible. As the table grows and more workers get added for throughput, that polling becomes a steady background load on the database, competing with the actual application queries it's meant to support, and SELECT ... FOR UPDATE SKIP LOCKED (the right way to do this safely) still means every worker is hitting the same table on every cycle.

Retries and failure handling get reinvented, badly

A queue table's retry logic is usually a retry_count column and some application code that increments it and re-queues the job, until it doesn't, edge cases around jobs that partially completed before failing, jobs that get picked up twice because two workers polled at the same moment, or failed jobs that pile up with no clear way to inspect why, tend to accumulate as special cases over time. These aren't hard problems, but they're problems a message queue solved years ago, and re-solving them in application code means re-solving their edge cases too.

What RabbitMQ actually changes

RabbitMQ flips the model from polling to pushing: a producer publishes a message to an exchange, RabbitMQ routes it to a queue based on bindings, and consumers receive messages as they arrive rather than asking repeatedly:

# Producer
channel.basic_publish(
    exchange='',
    routing_key='email_notifications',
    body=json.dumps({'user_id': 123, 'template': 'welcome'}),
    properties=pika.BasicProperties(delivery_mode=2)  # persistent
)

# Consumer
def callback(ch, method, properties, body):
    process_notification(json.loads(body))
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue='email_notifications', on_message_callback=callback)

basic_ack only fires after the message is successfully processed, if the consumer crashes mid-processing, RabbitMQ redelivers the message, because it was never acknowledged. Failed messages can route to a dead-letter exchange automatically, giving you a separate queue of "things that failed" to inspect, without writing that logic yourself.

What stays the same, and what doesn't transfer

The mental model of "a job has a payload and a status" doesn't go away, RabbitMQ doesn't replace the concept, it replaces the delivery mechanism. Things that don't transfer directly: SELECT * FROM jobs WHERE status = 'failed' as a debugging habit becomes checking a dead-letter queue instead, which has a different (though arguably better, messages there are genuinely undeliverable rather than just slow) shape. And anything that relied on the queue table being queryable alongside other application data in the same transaction, atomically inserting a row and a related queue entry together, needs rethinking, since the queue is now a separate system.

When the table is still the right call

If the volume is low, the jobs are simple, and a handful of workers polling every few seconds doesn't register as load on the database, a queue table isn't a mistake, it's appropriately sized for the problem. The switch to RabbitMQ is worth making when polling load becomes visible in database metrics, when retry and failure handling in application code has grown its own set of edge cases, or when multiple different services need to produce or consume the same events, at which point a message broker designed for exactly that stops being "extra infrastructure" and starts being the simpler option.