Congrats on the milestone! I've been following for a while - awesome to see the progress you've made.
Q: What are your plans for sharding / scale-out?
At Notion, we run ~480 logical schemas spread across ~32 Postgres databases [1]. Our data model [2] (sorry for the blog post spam) has a recursive / graph-like structures, and we make use of columns or jsonb attributes like `{ table: Table, id: UUID }` which sounds like your polymorphic links feature. It seems like EdgeDB's model lines up well with how we already use our database.
I did a quick cmd-f here, on the linked announcement, and in your docs looking for "shard" and "scale" but didn't find any relevant results. Postgres needs a Vitess!
Q: Do you have plans to support EdgeQL embedding or SQLite?
I am always looking for a way to compile "better than SQL" languages down to SQL. I like Datalog in this area because it's a composable way to define relationships/facts/derivations but no one is putting serious business effort to this idea (honorable mention to logica [3]). EdgeQL also fits the bill -- queries aren't logical, but they are composable -- plus looks easier to teach than most Datalog variants.
I took a peek in the repo and saw that a few EdgeQL components are written in Rust [4]; are you considering porting more logic to Rust? That would make EdgeQL much more embeddable - it could run in WASM or linked into an Android/iOS app. My pie in the sky dream is to use a single composable query/logic language to define all my relations and queries, and then compile that stuff so it works the same on both the client, server DB, and data streams (for incremental materialized views, ideally on both client & server).
If EdgeDB had a Lite version that ran on SQLite, we'd be 66% of the way there.
(To get the materialized view bits on the server, a mad scientist might already be able to point EdgeDB at Materialize [5])
> Q: What are your plans for sharding / scale-out?
Sharding is planned, though there is no set design yet, this area is in early research phase currently. Thanks for sharing your experience by the way! Learnings from the field definitely help. A traditional read replica scale-out is already supported and we are building integrations with Postgres orchestrators (for failover, replica discovery etc). Oh, and automatically routing read-only queries to read replicas (with some controls for lag) is something that we plan as well.
> Q: Do you have plans to support EdgeQL embedding or SQLite?
Possibly. Depends on the application and performance expectations :-) PostgreSQL is really special in its ability to deal with complex queries. We already have a toy EdgeQL interpreter in the codebase [1], which is mostly used to quickly prototype syntax and validate semantics. It would be great to scale it up to something that can work with persistent stores (even if dumb and slow).
Q: What are your plans for sharding / scale-out?
At Notion, we run ~480 logical schemas spread across ~32 Postgres databases [1]. Our data model [2] (sorry for the blog post spam) has a recursive / graph-like structures, and we make use of columns or jsonb attributes like `{ table: Table, id: UUID }` which sounds like your polymorphic links feature. It seems like EdgeDB's model lines up well with how we already use our database.
I did a quick cmd-f here, on the linked announcement, and in your docs looking for "shard" and "scale" but didn't find any relevant results. Postgres needs a Vitess!
Q: Do you have plans to support EdgeQL embedding or SQLite?
I am always looking for a way to compile "better than SQL" languages down to SQL. I like Datalog in this area because it's a composable way to define relationships/facts/derivations but no one is putting serious business effort to this idea (honorable mention to logica [3]). EdgeQL also fits the bill -- queries aren't logical, but they are composable -- plus looks easier to teach than most Datalog variants.
I took a peek in the repo and saw that a few EdgeQL components are written in Rust [4]; are you considering porting more logic to Rust? That would make EdgeQL much more embeddable - it could run in WASM or linked into an Android/iOS app. My pie in the sky dream is to use a single composable query/logic language to define all my relations and queries, and then compile that stuff so it works the same on both the client, server DB, and data streams (for incremental materialized views, ideally on both client & server).
If EdgeDB had a Lite version that ran on SQLite, we'd be 66% of the way there.
(To get the materialized view bits on the server, a mad scientist might already be able to point EdgeDB at Materialize [5])
[1]: https://www.notion.so/blog/sharding-postgres-at-notion
[2]: https://www.notion.so/blog/data-model-behind-notion
[3]: https://opensource.googleblog.com/2021/04/logica-organizing-...
[4]: eg https://github.com/edgedb/edgedb/tree/master/edb/edgeql-pars...
[5]: https://materialize.com/docs/overview/architecture/