People like Geofabrik are why we can (sometimes) have nice things, and I'm very thankful for them.
Level of irresponsibility/cluelessness you can see from developers if you're hosting any kind of an API is astonishing, so downloads are not surprising at all...If someone, a couple of years back, told me things that I've now seen, I'd absolutely dismiss them as making stuff up and grossly exaggerating...
However, on the same token, it's sometimes really surprising how API developers rarely ever think in terms of multiples of things - it's very often just endpoints to do actions on single entities, even if nature of use-case is almost never on that level - so you have no other way than to send 700 requests to do "one action".
> Level of irresponsibility/cluelessness you can see from developers if you're hosting any kind of an API is astonishing
This applies to anyone unskilled in a profession. I can assure you, we're not all out here hammering the shit out of any API we find.
With the accessibility of programming to just about anybody, and particularly now with "vibe-coding" it's going to happen.
Slap a 429 (Too Many Requests) in your response or something similar using a leaky-bucket algo and the junior dev/apprentice/vibe coder will soon learn what they're doing wrong.
> Slap a 429 [...] will soon learn what they're doing wrong.
Oh how I wish this was true. We have customers sending 10-100s requests per second and they will complain if even just one gets 429. As in, they escalate to their enterprise account rep. I always tell them to buy the customer a "error handling for dummies" book but they never learn.
Another commenter said essentially the same thing, I sympathise, it's painful when the "customer" can't understand something clearly telling them they're doing it wrong.
I don't have an answer, but I wonder, for the sake of protecting your own org, is some sort of "abuse" policy the only approach; as in, deal with the human problem, be clear with them in writing somewhere that if they're seeing response X or Y (429 etc) that they're essentially abusing the API and need to behave.
The only thing that reliably works is to push the cost to the customer - so they can do whatever insanity they want, and they get charged accordingly.
And we’ve had cases where we had “fuckoff” charges in contracts (think $1 per API call after X thousand calls a day) and the customer just gladly pays tens of thousands of dollars and thanks us for fixing their problem.
The money is nice but sometimes you just want to shake them and say “look we have notifications you don’t need to poll the endpoint ten times a second, fuck it give us the code and we’ll fix it …”
Thanks for the reply - I did not mean to rant, but, unfortunately, this is in context of a B2B service, and the other side are most commonly IT teams of customers.
There are, of course, both very capable and professional people, and also kind people who are keen to react / learn, but we've also had situations where 429s result in complaints to their management how our API "doesn't work", "is unreliable" and then demanding refunds / threatening legal action etc...
One example was sending 1.3M update requests a day to manage state of ~60 entities, that have a total of 3 possible relevant state transitions - a humble expectation would be several requests/day to update batches of entities.
Not at all, I sympathise, we're all like minded people here!
> One example was sending 1.3M update requests a day to manage state of ~60 entities, that have a total of 3 possible relevant state transitions
> but we've also had situations where 429s result in complaints to their management how our API "doesn't work", "is unreliable" and then demanding refunds / threatening legal action etc
That's painful, and at this point, we're beyond technical solutions, this need human solutions. If they can't realise that they're rate limited because they're basically abusing the API, they need to be told in no uncertain terms.
Of course I understand that it's not that simple, as a backend dev, my "customers" are usually other devs so I can be frank, but when dealing with B2B customers we often have to act like they're not in the wrong.
But that is a question that should be escalated to management right? If they charge the customer enough that allowing them to make 1.3M requests to update 60 entities makes sense, why not let them?
If they want the service stupidly overprovisioned to deal with these nutjobs, then that’s what we’ll do. I find that they’re generally receptive to the message of $10k per month to serve nutjobs, $100 per month to serve everyone else, though.
That’s the key - have that in place from the beginning.
Because many “enterprise” customers can spend literally millions doing shit the absolute wrong way, but have $0 budget for a developer to make it work right.
I don't understand why features like S3's "downloader pays" isn't more widely used (and available outside AWS). Let the inefficient consumer bear their own cost.
Major downside is that this would exclude people without access to payment networks, but maybe you could still have a rate-limited free option.
Their download service does not require authentication, and they are kind enough to be hesitant about blocking IPs (one IP could be half of a university campus, for example). So that leaves chasing around to find an individual culprit and hoping they'll be nice and fix it.
They could however rate-limit per IP with possibly a whitelist or higher limits for cooperative universities that are willing to prevent abuse from their own network. Unless they are facing a DDOS but then this appeal is even less likely to help.
This "it's only right that we, the humble and fair politicians, are exempt from this forceful control we're exerting over everyone" aspect of ChatControl is beyond ridiculous.
I'm not usually of a "revolutionist" kind in the slightest, but, when you combine this small example to a lot of things currently happening across Europe and the US - it does increasingly seem like people in power are less and less wary of heavy and serious responsibility their positions hold to the people, and are more and more brazen when it comes to trying to isolate themselves from scrutiny over their self-profiting endeavours.
Historically, there were somewhat regular "correction" events happening somewhere sufficiently close, that made sure that responsibility is stuck in politician's minds for longer into the future, but it's been a long time since.
Edit: My comment is partially fueled by everything that's currently happening in Serbia (grand-scale systemic corruption), but I do think you can see similar movement in much more orderly countries in Europe as well, and all this is unconnected to ChatControl, but I see it as a small ripple from the same source.
I also dare say that current state of affairs in US has emboldened such people everywhere.
Nepal is probably not felt as close enough to have an effect.
I think Bill Maher did a good summary about Europe getting worse in every perceivable way on civil liberties.
And people wonder why young people leave the old continent. Surely not only because of this, but the political class in Europe is more and more dysfunctional.
Eschewing responsibility through these kinds of "tricks", where the person obviously thinks themselves so above everyone else that they can make them idiots to their face, makes my blood boil.
It's always either public "servants" in power, or the rich people, putting themselves outside of the rules. If you are an elected official, and make a stunt like this, it should be grounds for immediate dismissal, IMO. But, alas, nowadays these kinds of things are so minor and irrelevant, in the sea of ridiculously horrible stuff they do.
It's at least refreshing that there are still places, like the Netherlands in this case, where there are some (even when it's surface-level) repercussions of such behavior.
I don't think off-the-record communication always implies corruption. I imagine it to be impossibly hard sometimes to get people to agree on anything (which is basically a PM's entire job), if all communication must happen out in the open.
I'm genuinely curious about how you and other people with similar outlook see this playing out, as it would kind of provide hope.
Scenario: You are a medium level engineer, who got laid off from a company betting on AI to replace a significant portion of their junior/medium level developers. You were also employing a middle-aged woman, to help with the kids after school and around the house, until you and your wife come back from work. She now needed to be let go as well, as you can't afford her anymore. The same thing happened to a large portion of your peers and work in the same industry/profession is practically no longer available. This has ripple effects on your local market (restaurants, caffes, clothing stores etc).
How do you see this as empowering and a net positive thing for these people individually, and for the society? What do they do that replaces their previous income and empowers them to get back to the same level at least?
Well, if everyone is unemployed there won't be much of a market for these newly AI enabled companies to sell into. Also, in the extreme, you'd have deflation such that it's worth hiring again. This would be very painful.
More likely automatic stabilizers and additional stimulative spending would have to happen in order to fully utilize all the new productive capacity (or reduce it, as people start to work less). It's politically hard to sustain double digit unemployment, and ultimately the government can always spend enough or cut enough taxes to get everyone employed or get enough people to leave the labor force.
I totally share your concern, but I think there's reason for hope assuming it's not Terminator-style AGI that destroys the world (bigger problems than unemployment in that case). Specific to your scenario, it seems like companies are laying people off today in the name of AI efficiency gains (that in itself is debatable, but let's assume that is why they're doing it--they think they can do the same if not more with less). But if you play out those same efficiency gains companies that are in growth mode ought to be able to use those efficiency gains to accelerate product development. So instead of laying people off, companies will be able to build product that much faster because their employees, and engineers in particular, can move so much more quickly. We're so early, though, and c-suite folks are so myopic that the troops haven't yet had time to show them that revenue growth is the real prize of AI/LLM's (and believe me it's always the some troops that show them the way).
On a larger level, I would just ask your fictitious medium-level engineer what are they able to do today, with an AI/LLM, that they were unable to do before? As a very basic example, and one that is already true with existing LLM's, a mid-level engineer who wanted to build an app might've formerly struggled with building a UI for their app. Now, sans designer, a mid-level engineer can spin up an app UI much more quickly, and without the labor of finding and actually paying a designer. That's not to say there's no value left in design, but if you're starting out it's similar to how bootstrap (dating myself here) was an enabler because you were no longer in need of a designer to build a website (was still a huge time suck and pain in the ass though). You can multiple that by a bunch of roles and tasks today because LLM's make it possible to do things you just formerly wouldn't have been able to do on your own.
Last thing is the much more high level. Every time some new tech is introduced there's a lot of concern about displacement. I think, again, that's valid and perhaps moreso with AI. But it does seem to me like major new tech always seems to create a lot of opportunity. It might not be for the exact same people like your mid-level engineer (although I think it might for him/her), but I stay hopeful that the amount of opportunity created will offset the amount of suffering it will cause. And I don't say that in some kind of "suffering is ok" way, but just like revenue growth is the be all end all for so many companies, tech brings change and some suffering is a part of that. Prior skills become less important, new skills are preferred. Some folks adapt. Others thrive. Some are left behind.
If you're still checking in on this thread, and you actually read my diatribe, do you think I'm totally full of it? Again, I don't know that I would bet it would work out this way. Actually I probably would bet on that. But I'm definitely hopeful it will.
I finally got to do something more extensive and serious with Claude Code / Gemini. It's basically a more complex CRUD app for multiple data entities, with some additional functionality.
I'm hoping that sharing my experience, amongst all others, can:
A) help someone understand more / set their expectations
B) get someone to point out how to do it better
On one hand, I managed, in 10 days, to get the amount of functionality would take ~2 months of coding "by hand". If I started the same project now - after learning, realising what works and not, and adapting - it would probably be possible in 5. The amount done was incredible - and it's working.
On the other hand:
- you need to be already very experienced in knowing how things should be built well, how they need to work together, and what is a good way to organize the user interface for the functionality
- you then need to have some practical experience with LLMs to know the limitations, and guide it through the above gradually, with proper level of detail provided and iteration. Which takes attention and process and time - it won't be a couple of sentences and hitting enter a couple of times, no matter how smart your prompts are
- otherwise, if you didn't think it through and planned it first, and did it with consideration of LLM itself, and you just give it high level requirements for an app with multiple functionalities - you'll just get a mess. You can try and improve your prompts over and over, and you'll get a different kind of mess every time, but mess nevertheless
- even with doing all of the above, you'll get a very very mediocre result in terms of "feeling of quality" - thoughtfulness of design, how information is laid out and things are organised - UX and polish. It might be more than fine for a lot of use-cases, but if you're building something that people need to use productively every day, it's not passable...
- the problem is that, at least in my experience, you can't get it to high level with LLM in an automated way - you still need to craft it meticulously. And doing that will require manually tearing down a lot of what LLM generated. And that way you'll still end up with something at least a bit compromised, and messy when it comes to code
In summary, it's amazing how far it's come and how much you can do quickly - but if you need quality, there's no going around it, you still need most of the effort and time do invest in it. Considering both together, I think it's still a great position to be in currently for people who can provide that needed level of quality - sometimes you can do things very easily and quickly and sometimes you do your proud work with a bit of assistance along the way.
I'm not sure until when that will work, or what happens later, or how does current state already bodes for less experienced people...
In countries where it comes to government blocking/censoring internet traffic, traditional media is cleared of all dissent and fully controlled long before. Last stages of that are happening in my country, Serbia, currently.
It's a mixed bag apparently, free press is technically legal since 1998 but selective prosecution and harassment of those actually uncovering issues (mainly becomes clear in the last section, "Safety")
Tried looking up Serbia next on that website but got a cloudflare block. I'm a robot now...
Level of irresponsibility/cluelessness you can see from developers if you're hosting any kind of an API is astonishing, so downloads are not surprising at all...If someone, a couple of years back, told me things that I've now seen, I'd absolutely dismiss them as making stuff up and grossly exaggerating...
However, on the same token, it's sometimes really surprising how API developers rarely ever think in terms of multiples of things - it's very often just endpoints to do actions on single entities, even if nature of use-case is almost never on that level - so you have no other way than to send 700 requests to do "one action".