It reminds me quite a bit of collision engines for 2D physics/games. Could probably find some additional clever optimisations for the lookup/overlap (better than kd-trees) if you dive into those. Not that it matters too much. Very cool.
There used to be a joint online project to compute these tables in a SETI like distributed system. Everyone who contributed their CPU cycles, could use the tables. And yeah, around 2005-2008.
I've had to test out various networked filesystems this year for a few use cases (satellite/geo) on a multi petabyte scale. Some of my thoughts:
* JuiceFS - Works well, for high performance it has limited use cases where privacy concerns matter. There is the open source version, which is slower. The metadata backend selection really matters if you are tuning for latency.
* Lustre - Heavily optimised for latency. Gets very expensive if you need more bandwidth, as it is tiered and tied to volume sizes. Managed solutions available pretty much everywhere.
* EFS - Surprisingly good these days, still insanely expensive. Useful for small amounts of data (few terabytes).
* FlexFS - An interesting beast. It murders on bandwidth/cost. But slightly loses on latency sensitive operations. Great if you have petabyte scale data and need to parallel process it. But struggles when you have tooling that does many small unbuffered writes.
Did you happen to look into CephFS? CERN (folks that operate Large Hadron Collider) use it to store ~30PB of scientific data. Their analysis cluster is serving ~30GB/s reads
Sure, so the use case I have requires elastic storage and elastic compute. So CephFS really isn't a good fit in the cloud environment for that case. It would get prohibitively expensive.
While I do love Postgres and use it daily on AWS and Google Cloud, I will add that the managed Postgres on Google Cloud is a mess in some areas. For example they use some EOL extensions outdated for 10+ years (a specific example is GEOS) and refuse to update it and give no control for you to upgrade it either.
I wonder where things will stand in 10 years from now. Will many orgs still be consuming vanilla Postgres, or will most workloads have shifted to ~proprietary implementations behind cloud services like "Aurora PostgreSQL Limitless Database" and "Google AlloyDB for PostgreSQL" due to unrivalled price-performance? In other words, can progress in OSS Postgres keep up with cloud economics or will things devolve into an even messier ecosystem centred purely around the wire protocol and SQL dialect?
Databases seem to grow much slower than other assets, so maybe this price advantage just won't be worth the vendor lockin. Hell my current extremely valuable postgres database worth literally millions of dollars is about thirteen gigs and could be hosted on my mac mini if I really wanted to. Still, the managed hosting is worth it—only without vendor lockin!
DBMS (and DBAs) tend to be more conservative, so extensions and implementations diverge much more slowly than, say, js.
There's also the incoming business argument in favor of not diverging too far from baseline.
If I'm AWS/Azure/GCP trying to attract a customer from a competitor service, 'completely rewrite your app to be able to use us' isn't a compelling pitch.
MS SQL Server and Oracle have different incentives, but the cloud services would probably prefer portability and decreased maintenance / fork support load.
We need to update the home page with these details, but $0.05 is only charged on transfer between Regatta and S3. We calculate your cache usage minutely and tally it into a monthly usage amount that we then bill for.
> You don't actually directly charge for storage itself, so I assume this a "bring your own s3 bucket" type of deal, correct?
That's correct -- we store data in the customer S3 bucket.
> How long does data, that is no longer being accessed sit in the cache and count towards billing?
We keep data in the cache for up to 1 hour after you've stopped accessing it.
> As for availability, are you in the process or do you have plans to also support Google Cloud?
We have plans to support Google Cloud. If you're interested in using us from GCP, I'd recommend setting up some time to chat (either use the website or email me at hleath [at] regattastorage.com). We are prioritizing where we launch our infrastructure next based on customer demand.
reply