See it's the exact opposite for me, although my experience is mostly a) building giant cubes in giant enterprise orgs with hourly data volumes you couldn't fit in memory, and b) 10-15 years old (so the hardware sucked and we didn't have duckDB). But yeah, I don't think the O in OLAP standing for 'online' ever really made sense.
I'm curious to know how much of this article is OLAP specific vs just generic good practice for tuning batch insert chunk size. The whole "batch your writes, use 100k rows or 1s worth of data" thing applies equally to pretty much any database, they're just ignoring the availability of builtin bulkload methods so they can arguing that INSERTs are slow so they can fix it by adding Kafka, for reasons? Maybe I'm missing something.
Author here—this article was meant to highlight how you can optimize writes to CH with streams.
If you want to directly insert data into ClickHouse with MooseStack, we have a direct insert method that allows you to use ClickHouse's bulkload methods.
It's a tradeoff. Analytics databases are often filled with periodic dumps of transactional databases and this feels so dirty that it's easy to accidentally forget that it isn't just a hack, it's actually a structural workaround for the poor random-write performance of analytics DBs:
OLTP = more read amplification on analytics workflows, less write amplification of random insert
OLAP = less read amplification on analytics workflows, more write amplification of random insert
If that's too theoretical, the other day I saw 1-row updates of about 10kb data lead to 1GB of writes in Redshift: 1MB block size times 300 columns times a log+shuffle factor of about 3. That's a write amplification factor of 100000. Crazy stuff.
There are a few realtime analytic DBs that let you query the in memory streaming buffer. That lets you have the best of both worlds but comes at the risk of inconsistency because you’re querying data that isn’t durable.
Once you replicate the data, you replicate the APIs and generate the sdks. Got it. But then how much work to change my front end components on the like 30 visualizations I have to actually use them? And to validate them?
If your new APIs use the same endpoint names and the same request/response schema (same field names, types, shape), then swapping the base URL in your fetch calls could get you most of the way there without touching your visualization components.
In practice I get that there’s usually some drift, like maybe a field is renamed or you have different error handling. Depending on how strictly your front-end components assume the old schema, you’ll need to update those areas.
When I worked at bigCo , it was a totally different world. Data engineers used data platform tools to do data work, usually for data’s sake. Software teams trying to build stuff with data had to finagle their way onto roadmaps.
Why? carbon goes into the seaweed, seaweed gets turned into bioplastic which contains that carbon.
Bioplastic is used and then discarded into a composter. Composter degrades the bioplastic, the carbon (and other nutrients) are in the compost that is created?
