Using Postgres as a Message Queue

These days, I’ve been given a lot to think about message queues. While their utility is undeniable, the most important architectural question is: which messaging service/backend do you use?

If we don’t get into the managed vs. self-hosted debate, the impulsive answer is “Redis!”. It’s simple, it’s ridiculously fast, it’s mature, and it’s more popular than energy drinks. Other answers are RabbitMQ, SQS, ZeroMQ, your favorite higher-level abstraction library (BullMQ, etc.), and so forth.

While all the answers are good, I think any messaging/queue backend must provide a few characteristics if we’re to rely on it for creating stable systems:

If you start thinking like this, Redis is the most likely candidate for rejection. Sure, you can tune its performance characteristics but nothing can be done about the acknowledgement part. SQS seems like a good idea but simulating it locally and getting sucked into the AWS ecosystem (SNS, at the very least) are real concerns. RabbitMQ seems like a very nice thing but there are two problems: it’s a push-based system (which isn’t that bad in itself) and the conceptual complexity seems ridiculous (way too many abstractions for the simple job of messaging!).

I’d still choose RabbitMQ over anything else and pay the price for complexity, but my current use case at work is making me lean towards Postgres! Yes, a very nice messaging system with all the characteristics listed above can be designed in Postgres. Let’s see how:

Of course, everything in the end depends on the use case and the details, but I think using such a system will create a highly reliable messaging backend and can handle situations where less than 50/100 messages are being processed per second and a delay of 100ms isn’t life-or-death. Another huge positive outcome is that nobody will have to learn another library or introduce another point of failure in the system.

I’m yet to take a final call on this, and over the next 2-3 months will build and put in production such a messaging system. Whether all goes well or not, time will tell, but I intend to record my experience here.