Hacker News new | past | comments | ask | show | jobs | submit login
Statement regarding the ongoing Sourcehut outage (sr.ht)
382 points by rapnie on Jan 12, 2024 | hide | past | favorite | 189 comments



I run a service which was recently DDOS'd. It came immediately after a request from the police services in India to give them data on a specific user. They claimed it was terrorism related. We explained our policy on that (there is a procedure but it requires lengthy legal formalities in our jurisdiction). Within hours our service was taken down. It lasted about 12 hours. We never received any ransom request. We were left with 3 possibilities:

1. Indian police/government

2. Original user

3. Total coincidence

When it happens you just realize: this should not be possible. This is a game that hugely benefits the most powerful players whether they just don't bother solving the problem or actively play in it.


How sure are you that the request came from the actual police in India, and not somebody else? I.e. someone who might get irrationally upset when you did not give them what you wanted?


"someone who might get irrationally upset when you did not give them what you wanted"

Like the police in India?


I would assume that the actual police have their own traditional methods of being irrationally vindictive, and would not normally restort to DDoS.


Those traditional methods don't often work very well across borders like a DDoS does. The post at the top of the thread didn't state whether or not they were based in India too.


Indian police will have a lot of trouble beating you if you live in another country.


Google Hardeep Singh Nijjar.


Still a lot of trouble.


Their government might though..


Yes we went through a bit of procedure to establish that. They have also been in touch with the authorities in our jurisdiction since.

In fact we were contacted by multiple police authorities from different regions.


Agreed, I've been through something like this before and it was not legit. They'd compromised a Bangladeshi police email account and used it to try to get data out of us.


We were recently DDOS'd, too. Every time it happens, I wonder, who benefits from it (or even cares about us)? Competitors? Someone's bored? A disgruntled customer? And we'll probably never know.


> Every time it happens, I wonder, who benefits from it

Cloudflare. And any other DDoS protection vendors.

Not saying they caused it but obviously they benefit the same way roof tile manufacturers benefit from hurricanes.


Cloudflare benefits so long you buy protection from them. It doesn't need to be you. So long as you think you need their services it doesn't matter if you actually do. They also have the ability to keep you running if your project explodes in popularity.


Yes my org was recent DDoSed too repeatedly. It was honestly inexplicable. We're still not sure why.


Which police service? Some Military and Internal Security forces have the name "Police", eg. CRPF, ITBP.

If it was one of these forces within the MHA, I wouldn't be surprised, because it falls under "National Security"


IMO what is surprising about this is that the service didn't already have DDOS protection of some sort (or did it?) It's been a pretty standard practice to add that to any public-facing service for a decade or two now, hasn't it? Cloudflare is free/cheap for smaller services and there are many other options too.

I'd just assume that any service is going to get randomly DDOSed soon after launch, even without any sort of blackmail/targeted attack. Even if you can figure out who did it, there's pretty much no chance of chasing them down.

My takeaway from this wouldn't be "this shouldn't be possible" but "this has been commonplace for decades, and will keep happening, and we should prepare for the next one".

It doesn't really matter whether it's some corrupt local agency or some script kiddie... half the world's connected, on various shitty devices, and DDOSes are gonna keep happening.


Sourcehut is big enough that Cloudfare is neither cheap nor free, I think there was a discussion about this in another post.

Nor is it a good thing that Cloudfare has keys to half of the internet.


Wasn't talking about Sourcehut, but this comment's parent.

And there are alternatives to Cloudflare.

The point is just that DDOSes have been common for decades and they shouldn't be a surprise for anyone. They're an inherent part of the internet protocols we have and the freedom of routing. Saying "they shouldn't exist" is like saying bad actors shouldn't exist. Sure, that's a nice thought, but they do exist (and have existed long before Cloudflare) and we can't just pretend otherwise and believe that our service won't be affected. It's just a matter of time/luck.


Is sr.ht free?


> I'd just assume that any service is going to get randomly DDOSed soon after launch, even without any sort of blackmail/targeted attack. Even if you can figure out who did it, there's pretty much no chance of chasing them down. My takeaway from this wouldn't be "this shouldn't be possible" but "this has been commonplace for decades, and will keep happening, and we should prepare for the next one".

I treat this as a foundational flaw in the protocols powering the internet. People shouldn't need to pay off DDoS protection companies as a standard line of business.


> Connections from CloudFlare’s reverse proxy are dropped. Do not help one private company expand its control over all internet traffic.

https://web.archive.org/web/20231005165737/https://srht.site...


IMO. There is no way Indian Police is competent enough to DDOS your service.


They definitely are.

Beat cops definitely aren't, but most police forces in India are state level and a couple are federal, and will usually have a Task Force devoted to offensive and defensive security, and if not, will contract out to companies like Appin Security Group (who were able to force Reuters and SentinelOne to remove their report into how the private-public cyber model in India operated [0])

Also, after the 2008 Mumbai Attacks, the Indian Ministry of Home Affairs began working on coordinating and centralizing Metadata Analysis and Monitoring [1][2][3], because they unable to trace the VoIP calls used by LeT [4].

If it was actually National Security related, it would have went through a couple fusion centers that the Indian government formed [5][6]. If it was some local PD asking the platform, they probably wouldn't have the capabilities to DDOS

[0] - https://www.reuters.com/investigates/special-report/usa-hack...

[1] - https://en.m.wikipedia.org/wiki/Central_Monitoring_System

[2] - https://en.m.wikipedia.org/wiki/NATGRID

[3] - https://en.m.wikipedia.org/wiki/DRDO_NETRA

[4] - https://www.wired.com/2008/12/mumbai-and-voip/

[5] - https://en.m.wikipedia.org/wiki/National_Cyber_Coordination_...

[6] - https://en.m.wikipedia.org/wiki/Indian_Cyber_Crime_Coordinat...


So, Indian police is actually able to do quite the lot of pressure when terrorism is involved... but when Americans like Scammer Payback report giant scam callcenters with credible evidence, they take well over a year to raid it, and leak the first raid to the scammers beforehand? [1]

[1] https://www.youtube.com/watch?v=UdEELggaY5Q


> Indian police is actually able to do quite the lot of pressure when terrorism is involved

Because terrorism related stuff falls under the Indian equivalent of Homeland Security, even if it's local PD triaging.

> Americans like Scammer Payback report giant scam callcenters with credible evidence, they take well over a year to raid it

Raids that are not National Security related need coordination between the Federal Government and State Government in India. The scamming call centers are located in a state called West Bengal. West Bengal is ruled by a regional opposition party called the TMC. West Bengal removed it's consent for Federal Police in India to raid without the consent of West Bengal Police. This ended up in the Supreme Court for a couple years [0].

On top of that, the scamming call centers are closely tied to the ruling party machinery in that state, as you need to get a license from the state government to operate a call center, and these kinds of call centers will often be donating to local political parties to look the other way.

Watch Jamtara on Netflix. It's a good overview on the economics of scam calls in India.

[0] - https://www.deccanherald.com/india/cbi-independent-legal-ent...


Ah, so that is the reason why it's always West Bengal that's mentioned in SP and the other scambaiter videos? I had wondered about that before - I thought that it's mostly because he has infiltrated some local scammer coordination group on Whatsapp.

Many thanks for the context!


Np. India is a federal democracy like the US. The same kind of state-federal clashes that happen in the US happen in India.

Think of Indian democracy as being similar to American democracy in the 1890s-1930s, when local despots like Huey Long and populists like William Jennings Bryan roamed the planet.

> that is the reason why it's always West Bengal

Yep. In other states they either will get raided by the Federal Police (eg. CBI) or the economics of running a call center doesn't make sense.

To run a scamming call center you need a low cost English speaking population AND Political Backing. Most states in India will have 1, but not 2 (or at least, not for call centers).


Don't forget hacking activists computers and phones to plant files to convict them of terrorism. [0]

[0] https://www.wired.com/story/modified-elephant-stan-swamy-hac...


That's local/state level police.

That entire apparatus is rotten due to the incentive structure - if you as a cop don't listen to politicians, you'll get a last minute transfer to some village in the middle of nowhere with no running water


It's not too hard to DDOS a website. Here's the 9th google result for booter service:

https://nightmarestresser.net/


DDoS-for-hire services exist. They don't have to build their own.


Government authorities purchasing shady cyberweapons is a well documented issue. It would a pleasant surprise if there was any government on Earth that didn't do such things out of respect for basic human rights.

https://hn.algolia.com/?q=nso


Oof, my heart goes out to them. My very first week at [large popular public code forge], we were attacked by [state level actor], probably for hosting something that [state] didn't like. In a way, SourceHut and Codeberg getting this kind of attention is an encouraging sign that these alternative forges are starting to gain traction.


I wonder how you could figure out it was [state]. Was there some clear threat made? A blackmail "do this, or else.."?


This is likely a reference to GitHub, which was DoSed by the Chinese government for hosting a repository that was mirroring uncensored Western media on a domain that could not be blocked by the Great Firewall without hobbling the Chinese software industry.

The attribution wasn't subtle - a substantial fraction of Baidu's ads/analytics traffic served to domestic Chinese users was rewritten to hammer that specific repository directly.

NYT coverage at the time: https://www.nytimes.com/2015/03/31/technology/china-appears-...


Interesting to learn the reason HN was down:

> You may have noticed that Hacker News was down on January 10th; we believe that was ultimately due to Cogent’s heavy handed approach to mitigating the DDoS targetting SourceHut (sorry, HN, glad you got it sorted).

It was also in https://news.ycombinator.com/item?id=38939532 but I did not see it earlier.


i’m confused; is he saying that Cogent black-holed sourcehut’s IP addresses and that somehow affected HN’s IP addresses? they’re in completely separate IP allocations and ASNs...


to be fair, you reach a certain point when the only options left for DDoS protection are Remote Triggered blackholing on your edge.

This results in an entire /24 network not being routed to your network and being dropped by your peer instead.


That's not true. I work for a company that sells DDoS mitigation products to large network operators like Cogent for helping them deal with exactly these kinds of attacks in a much more sane manner than, "oy, just blackhole the destination and head out to lunch."

Either Cogent didn't buy our product (or a competitor's equivalent), or they have a network op who's a fresher and only knows how to blackhole things. Either way, it's a bad look for Cogent.


That is not a new look for cogent.


I remember at one point back in 2009 or so I wasn’t able to download from one of SourceForge’s mirrors. Turned out the reason for that was that Cogent had cut off my ISP (Telia) for some weird reason. IIRC Cogent and Telia hadn’t agreed on a peering policy, so Cogent just said whatever, we’ll cut you off (:


We were a Cogent customer for years. Blackhole was there one and only tool for DDoS. Eventually we just left.


Are there upstream providers in the US or EU that are known to deal with this kind of attacks in a more thoughtful way?


For a blackhole /32 (single IP) I believe is allowed in most ISP/INP.

Many ISP also have DDOS mitigation using bgp flowspec to block dirty traffic only and let valid traffic pass through.


Which is a totally reasonable way of going about the problem, of course. /s


Nobody has ever accused Cogent of being reasonable or competent.


Surprised he put that shout out in, considering he doesn't like Hacker News.


Likewise for the reach out to CloudFlare given their reverse proxy is actively blocked from source hut and anyone using source hut pages.


Only DD knows, but I guess even if he dislikes HN, he still might not wish for them to be blackholed.


Feel bad for the team to have to deal with this, but I have to say its a great example of how to communicate with your customers so hats off to you!


I wish the sr.ht employees the best of luck with the AMS migration. Luckily git is decentralized enough that I've been able to practically do everything the last few days except that I can't push new releases of software to the canonical upstream repository.


Also, the fact sourcehut does not baby users about git send-email means that collaborative development happening on the platform is uniquely posed to continue


sure, i can `git send-email` to the particular developers i've collaborated with before, but by default my send-email goes to just the list-serve at `~user/project@lists.sr.ht`. pulling out `nc lists.sr.ht 25` seems to show that it's also offline.

so, yes for really important stuff i could get patches through to projects i care about. but in practice the handful of projects i'm involved in on sr.ht seem to be mostly stalled.


Yeah. My comment was largely optimistic, in practice people would rather just wait for service.

That being said, I did send a small patch directly by email just for the fun of it.


Erm... AWS migration? What am I missing?

A DDoS on AWS would most likely bankrupt Drew and Sourcehut, even if Amazon managed to absorb the traffic. Not to mention the principial issues that they have with AWS in the first place.


> Erm... AWS migration? What am I missing?

It's A M S, not A W S.


AMS ≠ AWS :P


Erm, apologies, thank you for the correction. I'm embarrassed. :)


> We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means, but we are investigating other solutions which may be more affordable and have a few avenues for research today, though we cannot disclose too many details without risking alerting the attackers to our plans.

Maybe someone higher up in Cloudflare here can escalate this subject and help them along for a proper quote?


I don’t think people realize that a lot of Cloudflare’s free/cheap services are squarely focused on HTTP. Multi protocol stuff is on the enterprise plans where they make their money.


Just because the quote is expensive doesn't make it not a proper quote. Either CloudFlare does it for PR for free or they need paid for a proper quote.


Let me rephrase: Maybe they can negotiate a different quote?

The PR gained in the development community to help a service beloved by hackers in this site could be very beneficial.


So you want them to be paid in exposure? If a company offers a service that another needs, but doesn't pay for, should they provide the service for free for them? If GitHub goes down, should CF provide the service to them for good PR?

I feel awful for sourcehut, but that doesn't mean that cloudflare should be obliged to support them for free or below cost rates.


Paid for in exposure


[flagged]


Yeah, that post is not at all related to getting a discount from Cloudflare on a product they need to buy from somewhere regardless.

The owner is in general against slowly having all traffic on the internet routed through Cloudflare, but considering that they asked Cloudflare they seem to be interested in that compromise to stay online.


I don't understand how the linked image had anything to do with Cloudflare. Are DDoS attacks happening specifically to marginalized people, and Cloudflare is the techbro getting rich, and Cloudflare is "scanning eyeballs?"

IMO that quote is more about, say, Facebook blimping in internet for parts of Africa. The person posting it here as if the author is against all tech companies seems like a deliberate misinterpretation.


Why should cloudflare be expected to foot the bill instead of sourcehut?


Helping out a useful service? Though who knows really. Cost of services and the price they are sold at are only loosely correlated!

Though honestly it’s totally possible that cloudflare was offering a real discount but it’s still just very high for something like sourcehut.


Yes, that could be the case.

I am unable to edit my original post anymore to reflect that is also my opinion :(


Maybe they were misquoted, who knows. Connections are a nice thing to have and I've never felt bad about asking, knowing the answer could full well be no, but a second opinion is useful when your business is on the line.


> Maybe they were misquoted

99.999999% chance they were not.


This might be due to how outrageous Cloudflare’s pricing is for protecting non-HTTP services like SSH (Cloudflare Spectrum.)


Why not showing the quote? Perhaps other companies could offer a better deal.


Godot, codeberg, sourcehut… any other websites? Looks like somebody is targeting open source related websites.


Somebody is demonstrating their DDOS capabilities to be able to sell capacity to malicious buyers


Statements like this don't help either

"However, this is not an ordinary DDoS attack; the attacker posesses considerable resources and is operating at a scale beyond that which we have the means to mitigate ourselves."

:(


The Chinese government routinely runs DDoS against GitHub because it hosts VPN software that can be used to circumvent the Great Firewall of China. They use the Great Cannon of China, which is the offensive side of the Great Firewall, effectively turning ordinary Chinese Internet users into an unwitting army of DDoS-ers.

Of course, GitHub and Microsoft have significantly higher resources than SourceHut to endure the DDoS.


> Of course, GitHub and Microsoft have significantly higher resources than SourceHut to endure the DDoS.

At that point I wonder why peering partners of Chinese ISPs used in these attacks don't go and drop connectivity unless the abusive traffic gets stopped.


Pardon my lack of knowledge, is it possible to block all requests that originate from a particular country?


Not really, because your neighbor's security camera is likely participating in the attack. See "Mirai botnet" for example.


Also why it's called DDoS and not DoS.


I work in reliability and I've sat at the intersection of this question before.

Kind of, but not really. A lot of times you'll see UDP blocked from EMEA, which stops a good amount of attacks but doesn't solve the problem. It also creates problems for services that rely on UDP like VOIP. These days, even if the command originates from EMEA many of the participants are IOT devices that've been compromised - and those may live in the host country!

Blocking an entire country can do something, sometimes, but it also opens up a wormhole of optics when users who are not knowingly part of malicious activity complain they can't access a service that the rest of the world can. Of course, the host country that operates with a decent degree of CYA acts like they have no idea why someone would do such a thing.

Mitigating this stuff long term is often a game of 4D chess on a rotating board.


While the attack might be sponsored by one country, the servers often come from a wide number of places.


We did just (Americans and allies) Iranian proxies…non-news really when run through the lens of current political environment.

The things under attack are the message, not the attack itself. XD


They should use some crappy websites like Twitter to do that.


There is a -high- probability that twitter is where they disseminate their propaganda, so that would be like punching themselves in the face.


Oh… didn’t think of that. Makes sense.


Or someone is trying to make something being hosted there unavailable.


Yeah that seems most likely. The targets will have been chosen for being likely to attract the attention of the relevant communities.


Microsoft's Github?


Microsoft has very little to gain from DDoSing sourcehut as at the end of the day GitHub’s competition is like flies, AND Microsoft has a lot to loose if it was discovered they did it


Companies are made of people. Those whose wealth and wellbeing depend on Github being the monopoly may be interested in destroying the competition before it grows into a concern.


It would be really stupid for Microsoft, the country's richest company, to go after a tiny competitor via extralegal means. Absolutely would make no sense. The liability they would incur would be astronomical


Not to mention the free publicity for their competitor if they're caught. "We're so good, Microsoft was scared of us"


Cloudflare should just run a public ddos sink with live traffic dashboard so the attackers can demonstrate their power there and leave the rest of us alone. /s


The funniest thing is that this doesn’t strike me as a terrible idea, though maybe it looks bad from a marketing perspective:

“Look, if you’re trying to sell DDoS services, go right ahead, demonstrate your ability on our infrastructure. That way you’ll also know not to target our infrastructure”

But at the same time that might genuinely be a positive for the net. Just like the drug epidemic — you can’t stop people from doing it, but you can reduce the potential harm


Good for the internet but then there would be less attacks on real websites, who is cloudflare going to charge people for protection? Could be a good nonprofit idea though, get some hosting companies together and donate band with to a central target to get people to stop targeting their actual customers.


This is straight out of the onion. I love it so much.


> My name is Drew, I’m the founder of SourceHut and one of three SourceHut staff members working on the outage, alongside my colleagues Simon and Conrad. As you have noticed, SourceHut is down. I offer my deepest apologies for this situation. We have made a name for ourselves for reliability, and this is the most severe and prolonged outage we have ever faced. We spend a lot of time planning to make sure this does not happen, and we failed. We have all hands on deck working the problem to restore service as soon as possible.

Drew, you're great, Simon and Conrad are great, you'll get through this, and you will be fine.

Keep doing your great work, forever.


What is the motivation of ddos attacks, in general? I assume it comes with some risk of being criminally prosecuted, so there must be some upside. Is it shakedown attempts, or competitor's sabotage? Neither seems too plausible in this case.


Often it's as simple as unhinged individuals renting an attack because they feel wronged by their victim. That's a big reason why Discord became so popular (Skype and Teamspeak revealed your IP address) and why the vast majority of online games stopped using direct P2P networking.


script-kiddies, state actors, misconfigured botnets...

Motivation can often be like why everything is re-written: "Because we can" or just "Because it's there".

I'm not in the circle of these things, but history/news suggests that many are performed by those under the age of adult prosecution, or from countries that don't care, so there is minimal risk, and even when there is those involved are not the sort to believe they will get caught.


Ransom, taking out competitors (e.g. gambling sites), censorship.

The Chinese government regularly targets GitHub because it hosts VPN software.


> it comes with some risk of being criminally prosecuted

Has there been actual cases of prosecution?


One recent example would be Zeekill, who ran a lot of DDoS attacks:

https://krebsonsecurity.com/2023/02/finlands-most-wanted-hac...


Specifically, have there been cases of prosecution for those who hire the services? All of the links given so far appear to be for operators.



Most of lulzsec, for example. https://en.wikipedia.org/wiki/LulzSec


Most obvious is: someone doesn't like part of a code hosted on the website, and it is easy to take down. There was probably a cursory warning that we don't know about.


For dark net markets, it was commonly used to extract a ransom or to make other websites aware of their DDoS capabilities so that others pay the ransom too. But for a clear web website it's probably not a monetary incentive


Motivation? The "quoted a number we cannot reasonably achieve within our financial means" is all the motivation "someone" needs. So yeah "shakedown" or "protection racket".


Are you suggesting that CloudFlare is behind the same ddos attack that they're selling protection against? Because that's an absurd accusation.


Not the parent, but obviously if the cost of CloudFlare protection is $XXX million, a protection racket can say: pay us 10% of $XXX M to make this problem go away. It's routinely deployed against highly profitable online businesses like gambling, but I doubt even at a 90% discount (and assuming SourceHut were willing to pay off criminals, which I doubt), they are not likely to be a profitable target for extortion and some other motive must be at play.


Why is it absurd? Any such provider has something to gain by scaring us into believing we need their protection.


No, its more for the "fun" of making firms lose money, kinda LulzSec [0].

[0]: https://en.wikipedia.org/wiki/LulzSec


Codeberg went down to DDoS ~12h ago too:

https://status.codeberg.eu/status/codeberg


I wonder why they're thinking about moving to EU as their compatriot seems to be having the same issue and moving to Europe wouldn't fix it. It was mentioned in their public notice.


Their plan to move to the EU via their AMS site is not related to the attacks, but seems to be a general plan they have had for a while. At least this is how I understand their notice.


They were planning to moving to EU for the last 6-8 months. They were making it gracefully.

This attack expedited it greatly.


The EU move wouldn't exactly help, if the attack is specifically directed at Sourcehut. That is unless the ISP and hosting provider in AMS is offering some form of protection as part of their offering. I can't see why the attack wouldn't just follow the services.


I assumed that the addresses for the EU server are not publicly available, so if they get some sort of DDOS protection before bringing them up, then the attackers will not know where to target their attack.


This is certainly possible, but it seems they are anticipating this as well.

> One of our main concerns right now is finding a way of getting back online on a new network without the DDoS immediately following us there, and we have reason to believe that it will.


That doesn't make sense though, as all the DDOS people have to do is aim their bot army at the new site. Probably just a single parameter in their attack scripts. I mean I'm an idiot and I can launch a DDoS on someone, I just don't have the $$ or compromised army of iot devices to aim at accomplishing that, nor really the will to do others harm. Whether you're in Europe or NA doesn't matter.


If their plan is to get online via the second location, it's likely that said location has a much beefier upstream or built-in filtering, allowing them to absorb these amounts of traffic without being null-routed.

More likely, though, they're restoring the service to a fresh IP range and put the servers behind some kind of DDOS-protection or, alternatively, they simply choose to do the switch now as they need to do a full restore anyway and it's not related to the DDOS mitigation.


You're right, but they were already planning it, so why not?

Maybe they have another plans under that, a better server or set of servers, some hand-rolled mitigations, etc. I have no idea. I'm a user as everybody else.


1. Do "PHL, FRE, and AMS" mean something? Or are these just codenames for each site?

2. If I host a service on AWS, Azure, Linode, DigitalOcean am I also susceptible to layer 3 DDoS?


They're generally airport codes, so Philadelphia and Amsterdam... unsure about FRE.


The IATA code FRE is assigned to Fera in the Solomon Islands. Freemont Airport does not have an IATA code.


Naming conventions can be inconsistent, and not every datacenter location has an airport. ASD is often used instead of AMS for Amsterdam.


ASD is the IATA code assigned to Andros Town Airport, in the Bahamas. It might also be a commonly used abbreviation for a medical condition.


To answer your 2nd question, yes you can be DDoSd, Azure specifically offers a DDoS protection plan which is quite expensive. https://learn.microsoft.com/en-us/azure/ddos-protection/ddos...


Their topology page has information about PHL/AMS and FRE (seems like Fremont datacenter):

https://web.archive.org/web/20240111132224/https://man.sr.ht...


Linode went through a rather long DDoS attack a few years back with a few of their data enters being offline for a few days, so I would guess yes there.


1. PHL means Philadelphia, AMS means Amsterdam. Yes, just codenames for sites.

2. Depends if DDoS protection is part of the offer, I suppose.


>“what if the primary datacenter just disappeared tomorrow?” We ask this question of ourselves seriously, and make serious plans for what we’d do if this were to pass, and we are executing those plans now – though we had hoped that we would never have to.

Respectfully, if you guys asked this question...how come you don't have a cluster slave as a replica in another data center where the whole thing is synced?

A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.


> A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.

If you want a Klugey non-production IoT solution that bodges up something really important THESE days, all the cool kids are using ESP32s.

And as much as I think that would be a totally inappropriate solution for src.ht, I kinda wanna go make a "black-hole" switch for my office.


I suspect a possible DDoS attack from something Kiwi Farms related, it seems they really don't like the guy over there?

https://archive.is/yOObX


If kiwifarms ddosed everyone that they didn't like... plus, if anything KF has been surprisingly resilient to massive DDoS without any 3rd party protection service since they lost CloudFlare and other providers. So for this one specific thing, I think lessons can be learnt from kf on how to mitigate DDoS without 3rd party providers.


They developed their own proof-of-work system to rate limit requests, using SHA256, implementation details here: https://archive.is/dfBVN

They also have servers in multiple countries. Some of these servers being blackholed by Cogent, by the way.


HugOps to the Drew, the rest of the Sourcehut crew, and anyone else working on this. Compliments on the clear blog post.


> We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means

I don't get it. Cloudflare proudly advertises unmetered DDoS protection on any plan level. Is that just a lie, or what am I missing here? They don't need to be on a custom Enterprise plan.


They are only doing unmetered DDoS for HTTP traffic - for websites. File storage etc. is not allowed. And git falls into that realm.


I see, that explains it.


To expand on that, Cloudflare's standard product is their HTTP reverse proxy. To proxy arbitrary TCP/UDP traffic, you need to use their Cloudflare Spectrum service (https://www.cloudflare.com/en-gb/application-services/produc...), which is metered.


I think they might be looking in something more similar to Magic Transit, but that depends on how they set up their infrastructure.


wow $1/GB?

would be cheaper to pay a developer to add websocket support to openssh


Unlimited free is almost always a lie.


See also "unlimited paid time off".


But in this case it literally is? Cloudflare provides unlimited HTTP traffic on all plans, exactly what they claim. Sourcehut needs a different service.


HTTP(S)-only _is_ the limit.


Yeah, because protecting other protocols would require a totally different approach and product. This isn't a conspiracy.

The Cloudflare HTTP CDN cannot protect SSH any more than a condom will make a good umbrella just because they both are designed for protection!


There is no chance of unlimited http even - otherwise startups would free ride that for their video sites etc


every / Netflix / porn site etc would use the unlimited free http traffic service if they could. I’d start a YouTube competitor. At some point if you use a lot of bandwidth you will get a call. So unlimited http traffic is a lie. Heck - my guess if you dumped a few petabytes into New Zealand or similar with high term costs you’d get a call


https://www.cloudflare.com/en-gb/plans/

Looks like level 3 ddos protection is only available on the enterprise plan, it's not included in the unmetered ddos protection.


Layer 3*


Am I correct in assuming these kind of problems are not possible if you are using a major cloud provider instead of renting rack space?


No, that is not a correct assumption. It's also worth remembering that cloud providers are also servers in racks, but they own the building around it to.

For DDoS attacks, you need to have enough capacity to absorb the attack. Major cloud providers tend to have that, as do DDoS mitigation services (Cloudflare amongst others).

> We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means [..]

Typically what you want to do though is stop the traffic from reaching you at all, so ideally your network provider, who is upstream from you, blocks the illegitimate traffic so your servers never see it and don't get overwhelmed.

What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage. Imagine if your ISP decided to stop routing traffic to Google. Being hosted on GCP, a major cloud provider, would be of no use, since there wouldn't be a network path to them in the first place.

In general, Cogent seems to be doing a rather bad job at dealing with this attack and there's been fallout for many services beyond Sourchut. Google or AWS or Microsoft might've handled it more gracefully, or might not. Though major cloud providers tend to have their own connectivity between their datacenters, they too have peering/transit agreements with other major network providers. If those upstreams stop forwarding traffic to them, the same thing would happen. It's just less likely to go unnoticed.

Cogent is a massive provider, so you'd think they'd be a bit better at this. But they also have a reputation for being awful.


> What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage.

That's not what happened based on my understanding. The provider nullrouted their traffic (which is common if a customer is under attack), but Sourcehut couldn't talk to the customer support as their support panel wasn't working for them.


Cogent is also a low-cost provider, and it shows in their customer service.


For any company with a complicated cloud footprint, how do they avoid "denial-of-money" by having the attacker pick some externally reachable piece of the cloud setup where the attacker can take actions that the company pays for (e.g. download large files from a cloud storage bucket, generating huge bandwidth charges)?


This is a major problem, particularly for those offering any kind of outbound SMS capability, whether for 2FA or just phone number verification.

That service is the easiest and most profitable to abuse, there are certain providers in certain countries that price inbound SMS very steeply, and are willing to share the profits with you. If you if you can get an attack going.


Still very much possible, in most cases all you're doing at the cloud provider is running virtual hardware through load balancers so they can very much become overwhelmed. Even with more advanced setups, it's still all just hitting a variety of servers, each of which can be affected in different ways.

There are (expensive) ways of mitigating but a project like Sourcehut couldn't afford or justify what will likely be a 5 to 6 figure sum.


If you're using a major cloud provider, a DDoS might mean either that your service hits scaling limits or that you get a massive bill.


You are probably correct in assuming these kinds of problems are not possible given enough $$$


Depends if you buy their ddos protection services or not.


Sad to see, best of luck to them


This site can’t be reached

DNS_PROBE_FINISHED_NXDOMAIN


I can reach it but here's an archive: https://archive.is/wVtqN


I does work for me, but more people seem to experience that problem:

https://fosstodon.org/@drewdevault/111742324107487646


[flagged]


When has Drew said he was a fan of DDoS attacks? When you make claims like these you should really bring receipts.

SourceHut doesn't blackhole content. It is in their terms of service and documentation what they will not accept on their website.


[flagged]


Yes, he doesn't like cryptocurrency projects. I don't see much vague about it.

The other clauses about bigotry are pretty much aligned with GitHub's ToS, which has "GitHub does not tolerate speech that attacks or promotes hate toward an individual or group of people on the basis of who they are, including age, body size, ability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, sexual identity, or sexual orientation", and "We have the right to refuse or remove any User-Generated Content that, in our sole discretion, violates any laws or GitHub terms or policies."

While people who don't like these policies call it "deplatforming", in the US the freedom of association is a protected First Amendment right, with only limited exceptions for certain protected classes like race, ethnicity, and national origin where there is a strong tension between the First Amendment protections and other constitutional protections.

I don't know about Dutch law though.


In private communication, he told me that he would ban me for being Christian.

Yeah, his policies are "aligned" with GitHub's (and I avoid that too), but Drew is on the record saying he will ban people for their opinions, not their actions.


Good morning, Gavin. You were shown the door because you are an outspoken transphobe and we were not interested in helping you voice those opinions. You seem to be unable to disentangle this with your Christian identity, but most Christians seem to manage alright.


I'm missing a lot of context.

A whole lot of people are Christian, so if this were true we should see evidence of this happening by now on Sourcehut.

If the ban was for membership in an "I am not a Christian" club, then that ban would be entirely reasonable.

How does he learn about any of your opinions without any sort of action? You must express the opinion first, and the act of expressing an opinion is an action.

I can think of many times when people were fired or "de-platformed" due to their opinion, like Jimmy Snyder (a.k.a. "Jimmy the Greek") - https://en.wikipedia.org/wiki/Jimmy_Snyder_(sports_commentat... , so it isn't like banning someone due to expressing a specific opinion is inherently unconscionable.


There are a whole lot of different kinds of Christians. I am a traditional Christian (not a fundamentalist) and tend to be more outspoken.

But doesn't that prove my point? You say that if I said an opinion, I made an action, so you are saying that Source Hut is right to ban me for opinions.

But you and I fundamentally disagree: banning someone for their opinion is not inherently conscionable.

You may find yourself with a "wrong" opinion someday and be banned from places you find important. Will you find it conscionable then?

By the way, expressing an opinion was not the kind of action I meant. I meant doing something that might harm others. My opinions do not call for harm (in fact, they call out what I see as harm), so my opinions cannot cause harm except by some broad definition that "speech is violence."

So why is it okay to ban me? Free Software needs Free Speech. SourceHut, as Drew has admitted, is not the place for that.


> I am a traditional Christian

"Traditional Christian" means so many things that it's practically meaningless. The fact you don't want to bring up the specific issue or denomination says much.

In the US context, I know some "traditional Christians" agree with Paul's advice "I do not permit a woman to teach or to exercise authority over a man", and want to live their lives that way.

Most people in the US work somewhere covered by EEOC or the equivalent state civil rights laws. If a "traditional Christian" expresses that opinion at work - not do it, just express it - there's a chance that sexist Christian will have a special talk with HR. Continued expression of that opinion can subject the company to a claim of a hostile workplace environment due to discrimination on the basis of sex.

Yes, expressing certain sexist "traditional Christian" opinions at work can result in reprisals. Yes, companies must make some allowances for religion in the workplace, but nothing in the law says that "traditional" Christian opinions override the religious beliefs of other Christians who do not heed Paul's advice and indeed are members of a church with a woman priest.

(I trust you understand how loaded the term "traditional Christian" can be, yes? As a non-evangelical you are not in the Southern Baptist Convention, but you should surely know how historically they held the racist view concerning the "mark of Cain"; see https://en.wikipedia.org/wiki/Curse_and_mark_of_Cain#America... for details. Some in the SBC still believe this centuries old tradition, and may regard themselves as "traditional Christian". Without knowing what you mean by "traditional Christian", you leave others to wonder just what traditions you believe in that you don't want to make public.)

So yes, if you are in the US then you should expect that expressing certain religious opinions in the wrong place can get you in trouble.

> so you are saying that Source Hut is right to ban me for opinions.

If you have violated the terms of service in the actions you used to express your opinions, then they are well within their rights.

You act like this is surprising, but terms of service are extremely commonplace, and a company doesn't need a written document to exercise a right they already had.

We generally agree that companies have the right to not do business with someone, and clubs have the right to reject someone as a member. A store can ban a customer for being too rude to employees, and does not need to provide those written rules to the customer first.

There are certain protected classes where a company is not allowed to discriminate, but that protection cannot be used as a carte blanche to express any and all religious beliefs at work.

For example, an evangelical Christian can get in trouble for proselytizing at work, even during break time, and even if that Christian argues their faith requires it of them. Quoting the EEOC, "if an employee complained about proselytizing by a co-worker, the employer can require that the proselytizing to the complaining employee cease. ... An employer can restrict religious expression ... where the item or message in question is harassing or otherwise disruptive." https://www.eeoc.gov/laws/guidance/questions-and-answers-rel...

> Free Software needs Free Speech

"Free Speech" is the bullshit argument always used to justify saying mean things and expecting no consequences. Right there in same First Amendment with Free Speech is the right of Free Assembly, that is, freedom of association - of being with who you want to be with, and of excluding those you do not want to be with.

Free Software needs liberty. Free speech is a part of liberty.

As John Stuart Mills wrote in "On Liberty": "We have a right, also, in various ways, to act upon our unfavourable opinion of any one, not to the oppression of his individuality, but in the exercise of ours. We are not bound, for example, to seek his society; we have a right to avoid it (though not to parade the avoidance), for we have a right to choose the society most acceptable to us. We have a right, and it may be our duty, to caution others against him, if we think his example or conversation likely to have a pernicious effect on those with whom he associates. We may give others a preference over him in optional good offices, except those which tend to his improvement. In these various modes a person may suffer very severe penalties at the hands of others, for faults which directly concern only himself; but he suffers these penalties only in so far as they are the natural, and, as it were, the spontaneous consequences of the faults themselves, not because they are purposely inflicted on him for the sake of punishment."

There is nothing there which says those unfavourable opinions must be due to harm.

There is nothing in the laws about workplace harassment which limit harassment to threats of harm.

By not acting upon those unfavourable opinions, we suppress our own liberty.

The fact that you focus specifically on harm means you don't understand any of the relevant issues ... or you do and are trying to redirect the topic to a strawman.

> So why is it okay to ban me?

Because of the freedom of association, a protected right in the US Constitution, in Article 11 of the European Convention on Human Rights, and supported in other legal systems.

> Will you find it conscionable then?

Then it's probably a good indication that that isn't a place for me. If it happened, and I was as outspoken as you say you are, I would want to present the details so others could get insight beyond "free speech" and "traditional Christian", and so they could perhaps inform me of things I don't understand.


Reminder that source hut makes a value judgement on your source code and will ban repositories they don't like https://news.ycombinator.com/item?id=33403780


I followed that link to see what "don't like" means, and it means "no cryptocurrency projects". That's +1 to SourceHut in my book.


[flagged]


My karma farming is working better than a lot of people's crypto farming these days :D


You have an obvious bias given your username. Why should SourceHut involve itself with projects that are at a high risk of being a scam?


And that's a good thing. A bootstrapped company doesn't have to be a slave to money.


Why do you need to convert "goes against their principles" to "doesn't like" to make your argument? Is it because it is a weak argument without twisting words?

They're allowed to decide what they'll host, and it looks like they're just being clear about a boundary. Unless they're playing favorites and allowing some crypto, what is the problem?


I don't want my source code vendor to tell me what source code I am allowed to write


They ain't doing that, they are just telling you which code they don't like to host.


What does that have to do with getting DDoSed? Because "you don't like them" they deserve to be DDoSed? Or is this just, "BTW, I am salty." Nobody owes you a centralized location for your distributed source code repositories.


Goes against their principles is an explanation for why they don't like it, so I don't see anything wrong with saying "if they don't like it". OP is correct and under no obligation for explaining their motives.


They and you are both incorrect. You both wish to pretend that because all dolphins are mammals, all mammals are dolphins.

They aren't removing code simply because they don't like it. They have defined a term of service. You can pretend it's more capricious than that, but it isn't the truth unless you have some additional evidence beyond that one thing.


So it turns out that running your own servers isn't always the best idea




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: