Hacker News new | past | comments | ask | show | jobs | submit login
NSQ: Open-source realtime distributed messaging, billions of messages / day (github.com/nsqio)
67 points by seansh on Jan 9, 2024 | hide | past | favorite | 10 comments



Billions of messages per day is 10-100k per second on average. If you do this in a vertical on one box with something like a ring buffer abstraction, you can achieve MPSC messaging rates in excess of millions per second without trying very hard across any arbitrary # of clients.

E.g. https://aeron.io/aeron-open-source/#aeron_transport


I find it mildly devious when numbers are inflated by changing the period from seconds to days.

Aeron and NSQ have slightly different design principles. Which can easily be identified in their feature list. Aeron’s origins are exchanges for fintechs and focus on predictable low latency with a tight standard deviation. NSQ puts heavy emphasis on being a distributed broker less message queue. Performance alone probably isn’t a good basis to measure their utility.


When you would use this over Kafka, Pulsar, Kinesis, NATS, RedPanda, Fluvio, etc.?

I found this comparison from 2023 https://gcore.com/learning/nats-rabbitmq-nsq-kafka-compariso...


I think those product can be sliced based on the typical use case.

Kafka and Azure EventHub and RedPanda are EventStream system use Retention policy. They can handle very high throughput efficiently but are not transactional and cannot be safely used when data consistency is the priority.

While system like ActiveMQ, Azure service bus, RabbitMQ are transactional message broker each subscriber to a topic has its own mailbox and message are removed from the mailbox when the client acknowledge that the message has been successfully processed. And deleted from the broker when no mailbox still contain the message. They can be used for workflow involving financial transaction (money) and are very safe. But they sometime dont maintain ordering, client might receive message in different order than the publisher put them inside the topic.


NATS especially seems very similar but more advanced.


Probably that’s the scenario. You want something simple and narrowly focused. Advanced and infinitely configurable isn’t always a virtue.


This project has been around for a long time. I made a few contributions in like 2014 or 2015. It is a great distributed stack and a semi-comfortable alternative to things like RabbitMQ. Did something new happen?


Looks like a release about 2 weeks ago but otherwise not sure.


Used NSQ a bunch at an old job, it was simple and reliable, never gave us any trouble, at the scale of, yes, billions of messages per day.


how is this different from zeromq, rabbitmq?




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: