Joking aside: since I've used Kafka for the reference implementation it naturally looks very similar, yes but it's really about the semantics. For example, does your 'normal' Kafka consumer start from the beginning or use latest index, etc.
> ... if the same consumer pulls the same dnpipe again, does it get the same message
No. Each message is delivered at most once. But good point, need to make that clearer!
> Do messages ever get removed from dnpipes without a reset.
No. Again, something I need to clarify as it seems.
> Does pull block, support async completions, or just return an error when no data is available?
It blocks.
> Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset.
I don't follow. Reset empties the underlying queue.
In general, since the consumers start to consume not from the beginning of time (as in Kafka's `--from-beginning`) but wherever they happen to be (that is, in Kafka terminology from `latest` index) this shouldn't be a problem.
I tried to model as close as possible and as it makes sense after the semantics of (local) named pipes. I might have fudged up here but I'm not 100% clear on where :)
> Do messages ever get removed from dnpipes without a reset.
No. Again, something I need to clarify as it seems.
So what happens after the system has been running for a while? Why doesn't the dnpipe system fill up with old messages that will never be read again? If new subscribers don't see them, and old subscribers have read them, why are they not removed? It would seem that once every subscriber who subscribed before a message was sent has received that message, the message is dead and can be removed. Why keep the history? Did you really mean that?
Also, what happens if one of many subscribers stops making PULL requests? Maybe it's blocked on something, or hung. Do the queues start to build up? Does PUSH eventually block?
> Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset.
I don't follow. Reset empties the underlying queue.
On most queuing systems, when a publisher wants to shut down, they close the channel's sending end or send and EOF message. When all subscribers have read up to the EOF, they close their receiving end. The last messages get processed, and then the subscribers stop. If the only shutdown mechanism is a reset, that can lose messages not yet received. That's OK if the intent is just "kill everything and terminate", but not if you need a clean shutdown.
(ROS, the Robot Operating System, has a publish/subscribe system something like this. It's a soft real time system, so old data is discarded and you never want to block a publisher. They chose to lose messages if a subscriber isn't reading often enough. That's appropriate to a robotics use case. It probably wouldn't be for a containerized web backend. See [1].)
Author here. Thanks for the feedback and I agree, a lighter ref impl would be ideal (it's just that I happen to use DC/OS on a daily basis so for me this was the most natural thing to do)
I'm all open for GitHub issues that capture this request and/or PR ;)
DC/OS [1] supports services (long-running tasks) and jobs (one-offs or scheduled) which don't necessarily have to be container images. See also [2] for a small write-up on this topic.
