Hello, congratulations. Danswer looks really interesting and the name is simply great.
We are building something similar (internal enterprise search using llm) and I am thinking whether we should jump to Danswer codebase. I would have a couple of questions, if you could answer:
- how would you compare Danswer and privateGPT? Do you see it as a direct competitor?
- you posted below that you have not use llama hub connectors because they do not allow incremental updates? Can you maybe elaborate on that, examples?
- instead of pulling data from different sources, did you consider knowledge graph approach (push) where data and vector index would live together in the single graph database? What would be the advantages of your approach?
- you posted below that you had to implement a custom search. Could that possibly be avoided (with the different architecture?)
Thanks for the kind words! Sorry for the delayed response, this post drew a lot more interest than anticipated and we've been swamped working with new folks coming in.
Regarding:
- PrivateGPT: they're for individual use where you ingest your own data. We're for teams to use with access controls, connectors to typical business SaaS tools, works at scale (incremental updates and scalable container architecture), different user roles, etc. So basically I see very little overlap between the two projects at least in terms of "competition", we're just different.
- LlamaHub: Our connectors pull all of the documents in the first run, then every following run, it only pulls in documents that have changed since the last run. For large teams, the first run may take many hours but following that, it will only take seconds each time. Without this, it becomes untenable to keep all information up to date. Also we pull in additional metadata and permissions, which not all LlamaHub connectors support.
- Push flow for indexing: Yes, we also have APIs that you can push to for indexing documents. For event based "push", we didn't go that route because most tools we connect to don't support this.
- Knowledge Graph: We will certainly be building this, it is only a matter of prioritization and timelines.
- Replacing the custom search: We think our search is much better than a basic RAG pipeline out of the box that someone can get from Langchain/LlamaIndex. What's the motivation for wanting to remove/replace it?
Thanks and best