This just goes to show how crucial it was for Anthropic and OpenAI to hire first class product leads. You can’t just pay the AI engineers $100M. Models alone don’t generate revenue.
I got the exact opposite lesson. The parent and grandparent comments seem to be talking about dropping one product for another purely on the strength of the model.
I tested this feature out today, applying the same prompt and CSV data to both Claude Opus 4.1 and GPT-5-Thinking. They both chugged away writing Pandas code and produced similar output. It's nice to have another option for data analysis to act as a second opinion on GPT, if nothing else.
It’s not free at all. Most of the voltage drop along the cable is caused by conversion of electrical energy into photons within the erbium-doped fiber amplifiers. A relatively small fraction of the voltage drop is caused by losses in the copper cable that carries the current along the route. The high supply voltage allows a relatively small amount of current to carry thousands of kilowatts of power to the amplifiers without causing much loss in the copper.
I took that as referring to how over large distances the results of driving a metal rod into the dirt don't always match, so if you do things like tie both ends of a shielded cable's shielding to separate ground rods you can get odd problems sometimes.
Although I hadn't thought the differences were usually anywhere close to that large.
The challenge as I understand it, is that yes you will get ‘free’ power (not actually free, as you had to create the low resistance electrical path for it to exist), but you have no control over the properties or values of what you get - and it will vary unpredictably.
It’s also unlikely you’ll consistently get much actual net power out of it, as you’re competing against an entire planets worth of reasonably conductive (in bulk) parallel paths.
Even the best optical fiber transceivers and glass are limited (practically) to about 100km; repeaters are typically placed every 60-70km. The technology for delivering power to the repeaters is fascinating. They inject 5,000-10,000VDC at one end and each repeater shunts off a tiny amount of current to power the amplifier. All of this is embedded in the cable itself before being loaded onto the cable ship.
The history behind TAT-1, the first transatlantic telephone cable, and the repeaters used, is fascinating. Bell Labs designed the repeaters. The repeaters used vacuum tubes for amplification and were designed for extreme reliability. The flexible repeaters were integrated into the cable like modern cables.
The tubes were tested to an extremely high standard. Only a small fraction of the manufactured tubes were selected after testing: Bell Labs designed a test regime over 18 years to detect minute flaws in manufactured tubes
The cable and its 306 tubes operated for 22 years with no failures.
Note that this is in stark contrast to the first transatlantic tele_graph_ cable, which did not really have a ground line and consisted of seven copper wires covered with three coats of gutta-percha (natural latex rubber) and then hemp and tar. Many breaks and failures later, the first messages were sent in August 1858. The bandwidth was such that Queen Victoria's message to the US president, James Buchanan, that contained 98 words took 16 hours to send. It ultimately died during a famous dispute between William Thomson – later Lord Kelvin – yes, _that_ Kelvin – and the project's main engineer that ultimately ended in disaster (when the engineer put 2k VDC on the cable, destroying the insulation, against Thomson's advice) and a famous court case that basically saw the role of "the scientist" (the physicist!) as a competition professional for the first time.
It's all fascinating history. By the time of Bell Labs, an awful _lot_ had already been learned from previous failures.
Yeah, trying to build a thousands of km long undersea cable without a good theory of transmission lines is gonna be a painful experience (a lot of this theory was developed to fix these problems!)
Back in 2001/2002, I was charged with building a WiFi hotspot box. I was a fan of OpenBSD and wanted to slim down our deployment, which was running on Python, to avoid having to copy a ton of unnecessary files to the destination systems. I also wanted to avoid dependency-hell. Naturally, I turned to `chroot` and the jails concept.
My deployment code worked by running the software outside of the jail environment and monitoring the running processes using `ptrace` to see what files it was trying to open. The `ptrace` output generated a list of dependencies, which could then be copied to create a deployment package.
This worked brilliantly and kept our deployments small and immutable and somewhat immune to attack -- not that being attacked was a huge concern in 2001 as it is today. When Docker came along, I couldn't help but recall that early work and wonder whether anyone has done a similar thing to monitor file usage within Docker containers and trim them down to size after observing actual use.
The best CI/CD pipeline I ever used was my first freelance deployment using Django. I didn't have a clue what I was doing and had to phone a friend.
We set up a git post receive hook which built static files and restarted httpd on a git receive. Deployment was just 'git push live master'.
While I've used Docker a lot since then, that remains the single easiest deployment I've ever had.
I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.
Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?
> I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.
Sounds great if you're only running a single web server or whatever. My team builds a fairly complex system that's comprised of ~45 unique services. Those services are managed by different teams with slightly different language/library/etc needs and preferences. Before we containerized everything it was a nightmare keeping everything in sync and making sure different teams didn't step on each others dependencies. Some languages have good tooling to help here (e.g. Python virtual environments) but it's not so great if two services require a different version of Boost.
With Docker, each team is just responsible for making sure their own containers build and run. Use whatever you need to get your job done. Our containers get built in CI, so there is basically a zero percent chance I'll come in in the morning and not be able to run the latest head of develop because someone else's dev machine is slightly different from mine. And if it runs on my machine, I have very good confidence it will run on production.
OK, this seems like an absolutely valid use case. Big enterprise microservice architecture, I get it. If you have islands of dev teams, and a dedicated CI/CD dev ops team, then this makes more sense.
But this puts you in a league with some pretty advanced deployment tools, like high level K8, Ansible, cloud orchestration work, and nobody thinks those tools are really that appropriate for the majority of devteams.
People are out here using docker for like... make install.
almost. there is still an issue with selinux. i just had that case. because the client develops with selinux turned off, the docker containers don't run on my machine if i have selinux turned on.
Imagine you have a team of devs, some using macOS, some using Debian, some using NixOS, some on Windows + WSL. Go ahead and try to make sure that everyone's development environment by simply running "git pull" and "make dev"
Ha, I've written a lot of these Makefiles and the "make dev" command even became a personal standard that I added to each project. I don't know if I read about that, or if it just developed into that because it just makes sense.
In the last few years, these commands very often started a docker container, though.
I do tend to work on Windows with WSL and I most of my colleagues use macOS or Linux, so that's definitely one of the reasons why docker is just easier there.
15 years ago i has a customer who ran a dozen different services on one machine, php, python, and others. a single dev team. upgrading was a nightmare. you upgraded one service, it broke another. we hadn't yet heard about docker, and used proxmox. but the principle is the same. this is definitely not just big enterprise.
That is wild. I have been maintaining servers with many services and upgrading never broke anything, funnily enough: on Arch Linux. All the systems where an upgrade broke something were Ubuntu-based ones. So perhaps the issue was not so much about the services themselves, but the underlying Linux distribution and its presumably shitty package manager? I do not know the specifics so I cannot say, but in my case it was always the issue. Since then I do not touch any distribution that is not pacman-based, in fact, I use Arch Linux exclusively, with OpenBSD here and there.
i used "broken" generously. it basically means that for example for multiple php based services, we had to upgrade them all at once, which lead to a large downtime until everything was up and running again. services in containers meant that we could deal with them one at a time and dramatically reduce the downtime and complexity of the upgrade process.
Would there still have been a problem if you were able to install multiple php versions side-by-side? HPC systems also have to manage multiple combinations of toolchains and environments and they typically use Modules[1] for that.
probably not, but it wasn't just php, and also one of the goals was the ability to scale up. and so having each service in its own container meant that we could move them to different machines and add more machines as needed.
> so there is basically a zero percent chance I'll come in in the morning and not be able to run the latest head of develop because someone else's dev machine is slightly different from mine.
It seems you never had to deal with timezone-dependent tests.
I once had to work with a legacy Java codebase and they hardcoded the assumption their software would run in the America/New_York timezone, except some parts of the codebase and tests used the _system_ timezone, so they would fail if run in a machine with a different timezone.
I like how you didn't even ask for any context that would help you evaluate whether or not their chosen architecture is actually suitable for their environment before just blurting out advice that may or may not be applicable (though you would have no idea, not having enquired).
I never said it was applicable to them, in fact I said the opposite:
> Obviously that ship has sailed for you, but I mean in the general sense.
In the general sense, no, you don't need a distributed system. Even if you have billions of dollars worth of revenue - no, you don't need a distributed system. I know, because I've worked on monoliths that service hundreds of thousands of users and generate billions in revenue.
If you're making YouTube, maybe you need a distributed system. Are you making YouTube? Probably not.
You can, of course, choose to make a distributed system anyway. If you want to decrease your development velocity 1000x and introduce unbelievable amounts of complexity and risk.
Yeah most enterprise software barely works and is an absolute maintenance nightmare because they're sprawling distrivuted systems.
Ask yourself: how does an enterprise with 1000 engineers manage to push a feature out 1000x slower than two dudes in a garage? Well, processes, but also architecture.
Distributed systems slow down your development velocity by many orders of magnitude, because they create extremely fragile systems and maintenance becomes extremely high risk.
We're all just so used to the fragility and risk we might think it's normal. But no, it's really not, it's just bad. Don't do that.
Enterprises are frequently antipattern zoos. If you have many teams you can use the modular monolith pattern instead of microservices, that way you have the separation but not the distributed system.
> I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.
For me, as an ex-ops, the value proposition is to be able to package a complex stack made of one or more db, several services and tools (ours and external), + describe the interface of these services with the system in a standard way (env vars + mounts points).
It massively simplify the onboarding experience, make updating the stack trivial, and also allow devs, ci and prod to run the same version of all the libraries and services.
That said, I'm not a nix guy, but to me, intuitively NixOS wins for this use case. It seems like you could either
A. Use declarative OS installs across deployments
B. Put your app into a container which sometimes deploys it's own kernel and then sometimes doesn't and this gets pushed to a third party cloud registry, or you can set up your own registry, and then this container runs on a random ubuntu container or cloud hosting site where you basically don't administer or do any ops you just kind of use it as an empty vessel which exists to run your Docker container.
I get that in practice, these are basically the same, and I think that's a testament to the massive infrastructure work Docker, Inc has done. But it just doesn't make any sense to me
you can actually declare containers directly in Nix. they use the same config/services/packages machinery as you'd use to declare a system config. and then you can embed them in the parent machine's config, so they all come online and talk to each other with the right endpoints and volumes and such.
or you can use the `build(Layered)Image` to declaratively build an oci image with whatever inside it. I think you can mix and match the approaches.
but yes I'm personally a big fan of Nix's solution to the "works on my machine" problem. all the reproducibility without the clunkiness of having to shell into a special dev container, particularly great for packaging custom tools or weird compilers or other finnicky things that you want to use, not serve.
The end result will be the same but I can give 3 docker commands to a new hire and they will be able to set up the stack on their MacBook or Linux or Windows system in 10 minutes.
Nix is, as far as I know, not there and we would probably need weeks of training to get the same result.
Most of the time the value of a solution is not in its technical perfection but in how many people already know it, documentation, and more important all the dumb tooling that's around it!
Not having to regularly rebuild the whole dev environment because I need to work on one particular Python app once a quarter and its build chain reliably breaks other stuff? Priceless.
This false sense of reproducability is why I funded https://docs.stablebuild.com/ some years ago. It lets you pin stuff in dockerfiles that are normally unpinnable like OS package repos, docker hub tags and random files on the internet. So you can go back to a project a year from now and actually get the same container back again.
Isn't this problem usually solved by building an actual image for your specific application, tagging that and pushing to some docker repo? At least that's how it's been at placec I've worked at that used docker. What am I missing?
Build and tag internal base images on a regular cadence that individual projects then use in their FROM. You’ll have `company-debian-python:20250901` as a frozen-in-time version of all your system level dependencies, then the Dockerfile using it handles application-level dependencies with something that supports a lockfile (e.g. uv, npm). The application code itself is COPY’d into the image towards the end, such that everything before it is cached, but you’re not relying on the cache for reproducibility, since you’re starting from a frozen base image.
The base image building can be pretty easily automated, then individual projects using those base images can expect new base images on a regular basis, and test updating to the latest at their leisure without getting any surprise changes.
At that point you're doing most of the work yourself, and the value add from Docker is pretty small (although not zero) - most of the gains are coming from using a decent language-level dependency manager.
It's not, but at that point you're giving up on most of the things Docker was supposed to get you. What about when you need to upgrade a library dependency (but not all of them, just that one)?
I'm not sure what the complication here is. If application code changes, or some dependency changes, you build a new docker image as needed, possibly with an updated Dockerfile as well if that's required. The Dockerfile is part of the application repo and versioned just like everything else in the repo. CICD helps build and push a new image during PRs, or tag creation, just like you would with any application package / artifact. Frequent building and pushing of docker images can over time start taking up space of course but you can take care of that by maybe cleaning out old images from time to time if you can determine they're no longer needed.
Docker re-uses layers as needed and can detect when a new layer needs to be added. It's not like images grow in size without bound each time something is changed in the Dockerfile.
I know this pain, and Docker absolutely makes sense for this use case, but I feel like we would both agree that this is a duct tape and bubble gum solution? Though a totally justifiable one
Someone wrote a PHP7 script to generate some of our daily reports a while back that nobody wants to touch. Docker happily runs the PHP7 code in the container and generates the reports on any system. its portable, and it doesnt require upkeep.
Docker in and of itself does not do you much good. Its strength comes from the massive amounts of generic tooling that is built around the container as the standard deployable unit.
If you want to handle all your deployments the same way, you can basically only choose between Nix and containers. Unfortunately, containers are far more popular and have more tooling.
> Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?
Yes. Everything on my box is ephemeral and can be deleted and recreated or put on another box with little-to-no thought. Infrastructure-as-code means my setup is immutable and self-documented.
It's a little more time to set up initially, but now I know exactly what is running.
I don't really understand the 24/7 comment, now that it is set up there's very very little maintenance. Sometimes an upgrade might go askew but that is rare.
Any change to it is recorded as a git commit, I don't have to worry about logging what I've done ever because it's done for me.
Changes are handled by a GitHub action, all I have to do to change what is running is commit a file, and the infra will update itself.
I don't use docker-compose, I use a low-overhead microk8s single-node cluster that I don't think about at all really, I just have changes pushed to it directly with Pulumi (in a real environment I'd use something like ArgoCD) and everything just works nicely. Ingress to services is done through Cloudflare tunnels so I don't even have to port-forward or think about NAT or anything like this.
To update my personal site, I just do a git commit/push, the it's CI/CD builds builds a container and then updates the Pulumi config in the other repo to point to the latest hash, which then kicks off an action in my infra repo to do a Pulumi apply.
Currently it runs on Ubuntu but I'm thinking of using Talos (though it's still nice to be able to just SSH to the box and mess around with files).
I'm not sure why people struggle with this, or the benefits of this approach, so much? It seems like a lot of complexity if you're inexperienced, but if you've been working with computers for a long time, it isn't particularly difficult—there are far more complicated things that computers do.
I could throw the box (old macbook) in a lake and be up and running with every service on a new box in an hour or so. Or I could run it on the cloud. Or a VPS, or metal, or whatever really, it's a completely portable setup.
Just for my home server, I have more than 10 containers for home assistant, vpn, library management for movies/tv/music, photos backup, password manager, and a notes server. I started without knowing what docker was, and in less than a year realized running services directly on my OS was more hassle than I wanted both with compatibility between services dependencies, networking setup for them, and configuring reboots and upgrades. I would say the reproducibility and configurability is easily worth the slight overhead and in my experience even reduced it.
Suddenly you're in a team with 2-3 people and one of them likes to git push broken code and walk-off.
Okay, lets make this less about working with a jack-ass, same setup, but each 5 minutes of downtime cost you millions of dollars. One of your pushes work locally but don't work on the server.
The point of a more structed / complex CI/CD process is to eliminate failures. As the stakes become higher, and the stack becomes more complex, the need for the automation grows.
Docker is just a single part of that automation that makes other things / possible / lowers specific class of failures.
I know well what you are talking about since I did something similar, but I finally moved to Docker with Kamal (except one project I still have to move). The advantage of Docker's reproducibility is to have a peace of mind when comes to rollbacks and running exact versions due to system dependencies. If anyone is curious I wrote Kamal Handbook to help people adopt Kamal which I think brings all the niceness to Docker deployment so it's not annoying.
> Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?
Why wouldn't it be, containers are super easy to manage, dockerd uses bugger all resources in dev (on Linux anyway) and docker compose files are the simplest setup scripts I've ever used
Sorry, sir, I didn't realize nobody should ever spend any time learning anything or, failing that, describe what happened to them during that time. I'm no neckbeard savant but I do have a dozen years of deploying web apps and also using Docker during that time, so I think I'm allowed to have an opinion. Go drink some warm milk, you will feel better.
You have 12 years of deployment experience and some of that using docker, would have been a more useful thing to say in your OC. I was literally just pointing out your argument was pretty weak - this context would have made it stronger.
In OpenWrt there is ujail, you give it an ELF (or multiple) to run, it'll parse them to find all the libraries they need, then it creates a tmpfs and mount bind read only the required files.
https://github.com/openwrt/procd/blob/dafdf98b03bfa6014cd94f...
I interviewed for a startup that does exactly this, except also for syscalls etc. They're mainly focused on security and not size. https://bifrostsec.com/
(I ended up taking another offer but I still think they're onto something.)
I can’t agree more. I spent a lifetime trying to master every productivity hack, reading books, doing hundreds of hours of therapy (this is not a joke - hundreds of hours), and organizing my desk and using tools and … everything.
And then I did a psychoeducational assessment and found out I have ADHD - and not just a little hint of ADHD, but really quite profoundly terrible ADHD. I learned about how much my brain had short changed me in my personal life too; because real ADHD affects you in many areas of life. It’s not just about “getting things done.”
Medication is a must have to make any progress. If you’re like me, you already tried everything else. Maybe you’re also really intelligent and even managed to get a great job and somehow maintain it. But you can’t ever follow a plan for long and never go to bed on time and always seem to be burning the candle at both ends?
Yeah, this is where you need some help from meds. And good god do they help. That being said, your brain is a responsive self-correcting system. So my advice to anyone taking this journey post-diagnosis is to not give up if things stop working. You may need to pivot, change dose, switch to something else, or add a non-stimulant option.
But, don’t ignore meds. ADHD is neurochemical in nature and it’s a joke to expect anyone will manage it without drugs.
The Roku example perfectly illustrates the deeper issue here - we're seeing a systemic breakdown in capital allocation that goes beyond the passive/active debate.
The real problem is a feedback loop: large companies get cheaper capital → they can afford to hoard cash and make defensive acquisitions → this reduces competition and innovation → which paradoxically makes them even "safer" investments → reinforcing their cost-of-capital advantage.
Meanwhile, the "missing middle" gets squeezed from both ends. Small companies can access some capital through VC/growth equity, but medium enterprises ($10M-$1B revenue) face a brutal gap. They're too big for most VCs, too small for institutional debt markets, and banks are increasingly consolidated and risk-averse.
(I am so personally familiar with the missing middle in my day job)
This isn't just about market efficiency - it's about market structure. When a streaming company parks $500M in bank accounts instead of investing in content or technology, that's not rational capital allocation. It's defensive positioning enabled by cheap capital and regulatory capture. There are many many lazy companies sitting on a cash machine structure with no decent ideas on how to grow.
Some potential fixes:
- Tax policy that penalizes excessive cash hoarding; eliminating the tax deduction on interest would encourage companies to hold less cash by making cash more expansive
- Regulatory limits on horizontal acquisitions above certain market share thresholds
- Public development banks focused on the missing middle (like Germany's KfW)
- Capital gains tax advantages for investments held in companies under certain size thresholds
The irony is that this concentration might ultimately hurt passive investors too - less competition means less innovation and slower long-term growth across the entire economy.
Nearly every startup I've talked to recently (last year or so) has been cash flow positive after just series A (or sometimes after just a seed round!)
Companies are pursuing smart business strategies and prioritizing healthy rates of growth over "hire everyone we can and figure out how to make a profit later". I'm seeing a lot of very focused businesses that are meeting customers demand right away and solving real problems. They aren't billion dollar problems, but they are profitable problems to solve. I wonder how the startup ecosystem will change if VCs start seeing 60% of companies making a healthy profit after a couple years, vs the historical trend of waiting a decade and hoping 1 company strikes it big. 60% of companies growing 10-20x ROI seems ... Not bad?
One problem of large mega corps is they don't even bother to go after new ideas that aren't multi-billion dollars markets. It used to be Microsoft would make a bunch of senior engineers filthy rich, they'd go boot strap a new company, and if the idea took off, Microsoft would aquire them again. Some people in the 90s/early 2000s pulled that off multiple times!
If some business is defensively buying off all the competition, that’s an easy ATM for a serial entrepreneur, especially if the large business is price gouging: found a competitor, grow it e.g. with some like-minded funding, sell it at a premium. Wait for the clauses of the contract to run out, and rinse and repeat.
The defensive acquisitions can work on domains with gravity effects to a degree, but those are domains are far from being the total market.
> found a competitor, grow it e.g. with some like-minded funding, sell it at a premium. Wait for the clauses of the contract to run out, and rinse and repeat.
Unrealistic, that strategy has no history of success. MariaDB tried... still wimpy.
> The defensive acquisitions can work on domains with gravity effects to a degree, but those are domains are far from being the total market.
Not so. Market size and inertia, connections, red tape and political weight are always present, fundamental forces of "gravity". They are the total market, at least the part of it that matters.
More importantly, even if your strategy of "feeding the machine for personal gain" did work, it amounts only to feeding the machine without changing its nature or direction. That strategy can benefit few individuals but it can't fix any of the issues discussed here.
I would guess that using an efficient embedding model to organize files is probably forthcoming in the next year or two.
Rather than moving similar files into folders, I can see the OS suggesting related files based on similarity to another file, or permitting search by concept rather than keyword.
reply