Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Why not use parquet files + AWS Athena?


The ability to use an index to seek directly to a handful of consecutive rows without processing the whole file was very important for our use case. Athena doesn't support indexing like this; it only has partitioning on a single column. It has to scan whole partitions every time. Both S3 Select and Athena are more useful when you want to aggregate massive data sets, but that's not what we're doing. We want to jump in and pull out rows from the middle of big data sets with reasonably low latency, not aggregate the whole thing.


> it only has partitioning on a single column

You can partition using several columns. But I get your point, it's not optmized for row level operations in general.


To avoid depending on a AWS product.


Athena = managed Presto + managed Hive Metastore

I use a presto containers to mock Athena for local and CI tests + a psql container for metastore.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: