So far I don't see that many talent in software engineering to be honest. The moment it has become a profitable job, it started to suffer from the same syndromes as other jobs - there are more people, but more incompetent people too.
I'm curious about how Indexify handles fault tolerance during ingestion spikes and compute infrastructure failures.
Could you please provide some examples?
Sorry just seeing this! There are various aspects of how it handles ingestion spikes -
1. The ingestion api writes to blob stores, which are horizontally scalable. Only when an ingestion finishes, we write the metadata to the replicated state machine.
2. The replicated state machine takes 100k IOPs on most commodity machines, and they can vertically scale.
3. The extractors can autoscale based on the number of tasks in the system.
4. The ingestion server cluster can autoscale also based on the amount of IOPs they are doing.
