Does anyone happen to know which country the new company is incorporated in? I'm still looking for a chance to use ClickHouse because it sounds so excellent!

This is so fast!! If anybody is using Pandas to keep rows in order and has hesitated to use DuckDB for that reason, hesitate no more! Give it a shot!

Seems like you're affiliated with DuckDB? Knowing what paper inspired the software [0], and regularly commenting about it's performance/recommending it [1][2][3][4][5][6][7]. Perhaps you should consider being more forthcoming?

[0]: https://news.ycombinator.com/item?id=24669902 [1]: https://news.ycombinator.com/item?id=27878401 [2]: https://news.ycombinator.com/item?id=26825096 [3}: https://news.ycombinator.com/item?id=26588029 [4]: https://news.ycombinator.com/item?id=26476649 [5]: https://news.ycombinator.com/item?id=24534795 [6]: https://news.ycombinator.com/item?id=24534721 [7]: https://news.ycombinator.com/item?id=24338671

Hey, this is HN you know :)

It's not unusual for happy, unaffiliated users to post enthusiastically, or even evangelise a bit, about the software they love most.

Case in point, I often pop up in TimescaleDB threads to sing it's praise regarding real-world usage, but I'm not affiliated with the TimescaleDB folks in any way.

I am not affiliated, just a happy user! I have spoken with one of the developers, but that's it!

I've just come up through the SQL side of analytics and I'm moving into Data Science and I feel like DuckDB is a superpower for people with my background.

Does that help? Happy to answer any other questions!

How would you compare the goals, vision, and current status of DataFusion with DuckDB? (www.duckdb.org) Could DuckDB be an execution engine for Ballista?

DataFusion committee here.

DuckDB can work on Arrow data, so I think it could coexist with DataFusion / Ballista quite well. I am not sure whether having different execution engines for Ballista is on the roadmap, but it's certainly a possibility!

I'm not sure if it would help in your case, but could you process all categories at once with a larger SQL query?

If so, DuckDB can process bulk queries about 20x faster than SQLite per CPU core because it is vectorized and column oriented. Then with multiple cores you can easily reach 100x SQLite speed. DuckDB has node bindings and is an in-process DB like SQLite.

If reading from disk is your bottleneck, I would recommend storing your data in compressed parquet files and reading them with DuckDB's parquet reader.

One drawback is that indexes are not persistent to the filesystem in DuckDB yet, but full table scans are much faster than SQLite since it is columnar.

https://github.com/duckdb/duckdb/tree/master/tools/nodejs

https://duckdb.org/

Came here just to recommend DuckDB! :-) Huge fan. It's unreasonably fast for how easy it is to use.

Odd that DuckDB didn't work for you on Windows! I only use it on Windows and love it!

My use case is a bit more complex. I have to compile it with an electron app and etc.

I filed a few GitHub issues with them as well.

This has great potential. The dialect is much more comprehensive than the Sqlite one.

Here is a DuckDB FDW for Postgres! I have not used it, but it sounds like what you need! https://github.com/alitrack/duckdb_fdw

The best part of this is just how easy it is! Just a pip install and you're up and running using industry standard Postgres SQL!

The DuckDB folks are migrating from pull to push and put together this interesting documentation of their reasoning! They use a vectorized model instead of a compiled one, so it's another interesting comparison point. It seems like push will simplify how they handle parallelism.

https://github.com/duckdb/duckdb/issues/1583

(I'm the author of this post) That's a great document I hadn't seen, thanks for sharing!

I completely agree!! We see 20-100x performance from DuckDB over SQLite for OLAP style queries.