Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Good point, Consensus at high data scales is not achievable. We are building a cloud-native log analytics solution at logiq.ai. Our initial design had RAFT at the core. Though it is relatively easy to implement when we started hitting high data rates (50+ Gb per hour), the RAFT APIs started to slow down the entire system. We removed it and made each pod stateless, which allowed us to scale exponentially. Now we can go from 1 Gb per hour to hundreds with a single `kubectl scale sts` command. Eventual consistency is the way forward for applications that generates high volumes of data.

Of course, some applications do require Consensus; for those, I would start with RAFT as a starting point.



Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: