Great read. I always find these turn a feature of a computer into a database an interesting read/idea. It certainly has upside, and downsides. For example having an entire filesystem as a sqlite database etc, or in this case an executable. It would be nice to have a structured way of talking to our utilities, rather than interpreting various commands as json through a few layers of tools to then get the output in a common format. That said I'd probably rather have a split up tool-set, than everything having to be in a single database for it to interoperate.
We do have a single database, it's the filesystem. But quite importantly it's a very composable kind of database, because you can "section off" a part of the database, calling it a folder, and the folder looks like a whole filesystem of its own. Working in the folder is like working in a separate database that's visible from the outside but the outside isn't visible from the inside unless you deliberately choose to go out through the escape hatch. SQL isn't like that - it's a flat namespace and that is like the original filesystems that didn't have folders. Those were usable when a computer was only big enough to hold a few activities at a time and when you had a stack of disks next to your computer, each with a different program's filesystem.
A flat namespace might actually work with an app-based OS model (like on android) where everything that happens is associated with an app ID and apps mostly can't talk to each other. But there's a reason nerds don't do nerd stuff on smartphones.
Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
> this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.
Great article. I find that I learn something every time I read a post about linux kernel work.
I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific.
What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
I've built my own workflow for using agents on git, as i now often have to do changes across repositories, or in the same repository for different tasks. I could use worktrees, but I'd rather invert it, give agents the ability to have a workspace, that they pull repositories into, create branches as they want, commit on main it doesn't matter. the agents don't bother each other, and when i finally have to merge, conflicts are either resolved, or it is just smooth sailing.
The tool is called gitnow. it is honestly quite simple, just create a project, add the repositories you want and get to building. I've found having another claude chat or whatever use the tool to great success coupled with zellij, but could also be zed, tmux or whatever.
Secondly it also pretty much solves the problem of the agent dumping memory files everywhere, they now basically have a scratch space that is theirs, where they can keep their tasks, and just update the repositories as needed.
Use gn the shell after eval if you use it, it will actually invoke cd, instead of creating a subshell.
Love clickhouse it is just super performant. I've had to tweak a few queries for performance here and there but it has been more than stellar. I'd initially set up a real time pipeline ingest for us to handle our larger incremental ingests when i started (used redshift in the past, very expensive and quite slow in comparison), it has been unnecessary so far, as clickhouse has just been able to chew through tons of data and large transformations without breaking a sweat.
Only issue i had was that the default configuration had some quite heavy handed tracing turned on which tanked performance on the relatively little machine we had for it, it has since been scaled up and is the core of our data stack.
I can't recommend clickhouse enough. If you were truly large I'd probably choose something else, but as long as we're staying on a few nodes it is manageable complexity and we enjoy using it.
Probably trace_log (we had a similar experience), but you'll only notice with limited storage space, and a regular keen eye on how your database fills up.
Interesting Mistral sort of knew something about me, both gpt and deepseek produced the same answer more or less. I wonder why xD, only gemini knew my online handle mostly github and rust which is interesting.
I tried out PgDog a while ago, but couldn't find a good way of handling the config except for having this users / pgdog toml file, which makes it a bit awkward to handle in kubernetes where we often do multi-tenancy in postgres - or rather having many databases on the same instance(s), and have them come and go at will.
Also had an issue with it because it cached authentication requests when doing passthrough it seems, I'd changed the roles password, but it kept using the old one, which was no bueno ;).
PgDog seems to make more sense when you really care about a few databases that need massive scale, rather than a simple proxy in front of postgres. I'll keep following the development though, it is much needed in this space, postgres can use all the investment it can get to get it past the single machine scale that it excels at currently.
Not the place and not the time, but we are building an enterprise edition that "just works" out of the box. Not saying that the open source experience cannot be better - it always can and we'll keep improving. What you've experienced is definitely a known issue with our specific implementation of passthrough auth. Scram made things a bit harder, since we can't validate user's passwords at login time anymore (that's what makes scram secure fwiw).
Happy to chat about this, but we use the AWS secrets manager flowing into External Secrets Operator to generate a pgdog_users.toml. We then kick off a workflow to refresh things, but our rate of change here is much smaller than a super dynamic multi-tenant system.
You could also build a watcher side car that watches for changes of the pgdog_users.toml and have pgdog refresh itself then too with this combination. We thought about that but prefer to control the reloads for our needs.
We successfully did this with pgdog at $JOB using our own "controller" -- the same service that handles deploying new instances of our application (instancing an argoCD Application that fires Crossplane DB creation, making new Deployments of bricks, etc) will also, at the end of that process, scan the cluster for Database CRDs, use those to generate a new pgdog.toml + users.toml, update the Secrets in the cluster, enable maintenance mode on all pgdog pods, do a live config reload on each of them, then disable maintenance mode (this is to make the change atomic between all the pgdog instances). Downtime there is about 2-3 seconds and all it does is make new SQL requests from existing clients wait, it doesn't break the connection or anything.
At this point it is too high of a risk to store my password elsewhere. I've been screwed over by dashlane, lastpass, potentially bitwarden now, I am with 1password now, but I've had my passwords in all these places, and I've had to change them each time, probably missing a few.
I like 1password, it is by far the highest quality product I've used in this category. I moved from BitWarden back then because their browser integration was quite poor.
I think I'll move to something custom, or a selfhosted keepass server, with the rugpulls, incidents, and whatnot, it is becoming too high of a risk.
Keepass has been my go to since forever, highly recommend. I never jumped on the SaaS password manager train when they started coming out, always just kept it local. There were times I thought I was missing out on some convenience but I'm glad I never moved.
Depending on your threat model, you can even just keep the .kdbx in cloud storage somewhere and point your keepass client to that. I'd recommend using a keyfile in addition to your master password though so that if anyone does happen to get a hold of the database they can't just make brute force attempts against it.
I’ve found being able to share passwords with my spouse very valuable which we couldn’t easily do with keepass. Also the syncing strategy on iOS is a disaster and corrupted my wife’s keepass db causing her to lose everything.
keepass files + syncthing works very nicely for me.
For non technical people, I just recommend to use the browser built in password managers. traviso has a good writeup why: https://lock.cmpxchg8b.com/passmgrs.html
I was doing this too until recently.
The problem with this setup is more at Syncthing.
More specifically, Syncthing Android app has seen some troubling changes in maintainers.
The latest maintainer has a very sparse Github profile and an AI generated avatar, so I noped out of installing it right then.
It is limited to ... well ... Firefox! Sometimes you need passwords elsewhere. Besides that Firefox (or other browser password managers) doesn't support more advanced use cases like shared vaults.
If you only need to manage online passwords, only use Firefox, and aren't using an iOS device, then it's probably fine. But most people may also need to use native apps, other browsers, and iOS devices.
Any malware or LLM with user-level filesystem access can attack the outdated KDF [1] and/or wait for Firefox to be running with an unlocked credential store and read the decrypted passwords from Firefox's process memory.
Rug-pulls, security incidents, lost passwords, I also don't know if they've kept my passwords behind when i deleted my accounts. The risk of them having them is too high, so i had to swap all of them.
Interesting! I've been a LastPass and then 1Password user since 2009ish.
I left LastPass because of UX paper-cuts, but I've never lost passwords on either of them.
Honestly, it's something I don't want to think about and just need it to work on mobile and desktop, so the switching friction is very high for me. I'm not going to shop around and try different password managers.
Is "rug pull" a cost thing? I'm generally frugal, but pay for a family plan and don't think twice.
When I started my professional work, Visual Studio was the recommended editor where I was, it was terrible, the Laptops we had were incredibly shit. My phone scored better on benchmarks than it, as such Visual Studio was not a good experience, I convinced my boss to let us try Rider, it was incredible, I no longer had to sit for 10s of minutes for a project to load, it was relatively snappy. My next job I started using Goland and was quite happy with it, at this point we had more high-power macs, but still great editor. I then moved on to Neovim, and then Helix in search of better ergonomics. I now have gone full circle and pretty much develop on a laptop of the same caliper as when i started working, however, because of a more lightweight editor helix, it doesn't feel like a slog, I wished I'd gotten it recommended back then, or been curious enough to give it a try, I'd saved myself many coffee breaks, and pain.
So if anyone is out there sitting in a similar position, give it a shot, you can get a better editor experience, whether you build it yourself with emacs, neovim, or use a more curated approach like helix, or zed for that matter. I mix and match Helix now with Claude Code, and it works really well. I don't want a single AI feature in my editor, only navigation, and auto complete. I'll have my AI on the side thank you ;)
I am using docker-compose everywhere. I really enjoy using it. I have a single thing that is annoying for normal production deployments, and that is that it isn't super easy to have a rolling deployment, I just need two replicas for zero downtime deployment, and I don't really want docker swarm. I think it is the networking which breaks at that point, and you have to have a more involved setup, and at that point I'd just use kubernetes, as I know how that works.
Could i survive with 10 seconds of downtime, probably, but I'd really like if I could avoid it.
Reading the article over, it really feels like Docker should be targeting Swarm as (instead of being its own platform) a set of incremental enhancements to Docker Compose. "I need healthcheck-restarts" "I need off-host logging", etc.
They've basically lost the war against Kubernetes but they could easily claim a lot of ground when it's just one more tweak you're adding to your docker-compose file as it scales.
if you use Caddy as your reverse proxy (instead of nginx for example which does not do this), when requests come in and your service is missing because it's being deployed, Caddy waits for a timeout before giving up. this means that visitors during the brief deploy period don't see errors - they just get a slightly longer wait, which often is not obvious depending on how long your service takes to boot.
