Ah this is so shit. I want to support Linode, I've had nothing but a good experience. But I just had to check my credit card to be sure they hadn't lost my details. I've NEVER had to do that before with anyone - they've got to respond fast here because if I don't trust them with my CC then I can't leave five-figure contracts at jeopardy hosted on their servers.
I've been living comfortably on Linode servers for over three years. This is like suddenly being evicted and having to pack my stuff up and find another apartment.
I have to wait for some sort of verification for this but if true then I have to leave Linode. I have client sites hosted here - not for cost reasons, just because I like Linode.
For the sake of $5 a month I can't even take the slightest risk of being criticised for using Linode. And this lack of transparency could be a nail in the coffin here.
I don't want to waste a couple of days on this but that's what's going to be involved if this is true.
I've now heard from a number of people using Linode that have suspicious activities on the cc which they used with Linode.
I just called up my bank to tell them to 'block' it as a precaution (I will now have to give them a visit later today to get a new card). I encourage all other Linode customers to do the same, because it'll be easier to just spend half an hour doing this instead of spending hours upon hours disputing specific transactions.
Linode customer support keeps saying they have "no comment" on this issue (which I suppose does make sense -- I'm assuming they've been ordered by law enforcement persons to not share details), so as we're not being given much information to work with... just treat this as a worst-case scenario (all names, addresses, credit card numbers, etc. have been compromised). Do operate now with the assumption that all of this data has been compromised and may very well be public soon.
It depends on how sophisticated the identify theft is. I had a good friend who was taken for about $9000 in credit card fraud in 1998/1999, with Well Fargo. It took him the better part of six months, and endless correspondence with WF to prove all of the purchases were not his. There are lots of stories of people who were financially wiped out, to the point of bankruptcy, because of Credit Card/Identify fraud.
With that said - almost everyone seems to feel comfortable handing out their credit card to random taxi drivers, waiters, sales staff - with no idea whether a copy of their information is being taken down. Heck - if you give them the Credit Card, they even get your CCV as well.
That makes me wonder why the credit card system is so insecure in the first place. Why are credit card systems not secured with a password that the merchant never gets to see? Yet at the same time credit card suppliers keep bragging about how "secure" their cards are.
Funny that you mention that. After a few bouts with fraud (because as i mention I have my CC out to many services), I was wanting a way to track down the offending service.. I was thinking something along the lines of a vendor-specific set of numbers to be run through.
I even wrote a blog post about it. I probably don't know what I'm talking about, but these were my thoughts at the time:
Wishlist – A Method to Pre-Approve and Track Credit Card Transactions
The issue:
A business using a credit card doing business with a relatively small number of vendors wanting to first avoid credit card fraud (stolen numbers) and secondly wanting to easily track down the offending business.
The idea:
The business would like to approve particular vendors to use the credit card with number 0000-0000-0000-0000 with each individual business pre-approved to run the transaction with a 5th set of identifiable numbers, so something like 0000-0000-0000-0000-0001.
If the credit card is used to make a fraudulent transaction, then ideally, they would have had to have used the 5th set of identifying numbers. This 5th set of identifiable numbers would then allow for easy tracking of the offending vendor, which would allow the business to either re-think doing business with them, or to serve as a starting point discuss security issues with the vendor’s credit card transaction processes.
Summary:
Basically, I believe there may be a need for a new or value added credit card type service. This transaction type would require a 5th set of numbers which have been assigned to pre-approved vendors. This 5 number set (ie. 0000-0000-0000-0000-0002) credit card transaction would most likely prevent theft right off (because the vendor is pre-approved and should provide their own private key (ie. CCV) to put through the transaction). Secondly, if and when the credit card number is stolen and used to make a fraudulent purchase, then, at least with the 5th number set a vendor can be identified and security policy with them can be re-evaluated.
> Why are credit card systems not secured with a password that the merchant never gets to see?
My bank in Sweden requires MasterCard SecureCode for all online transactions on their debit cards. Stores that don't support it simply won't work with the card.
So, it's up to the bank how secure they want it to be. The technology is there.
I've had roughly the same experience: in the last 8 years, I've had suspicious activity on my CC about 5 times. Each time, the bank caught and trapped it before I noticed and issued me a new card quickly. I've only had to fill out paperwork for a disputed charge once, and it was a 2-page, 2 question, sign-and-mail-it-in deal.
My advice is different, though. I notice that I tend to get lucky in places where people can have very aggravating experiences. I'd say that if you've had problems before, then anticipate problems this time around too. If you haven't, then don't bother.
Wow, four times? You should probably be more careful about who you give your number to. Personally, I usually get a new card every 3-5 months. If someone ever sat on my card number, it's useless to them now. Never had any issues either.
@kansface It's not bad for your credit rating. A number is simply a representation of the account. The account doesn't change. It's not like getting a whole new item of credit issued. Just the means to access it.
Also, great idea. But a pain, because most of my bills - cell, internet, insurance(s), etc all go through my credit cards. Is a gigantic pain to change the numbers.
I can be in some cases. I got mugged and my card was used to pay for parking garages for 1.5 years until it expired even though it was canceled and blocked by the issuing bank. They said that for some transactions, the blocking mechanisms are so expensive its more economically sane to them to refund whatever was drawn.
I also do that, but it's because I'm scared of recurring subscriptions that I've forgotten about, especially those that decide to sneak into my pocket after I've deliberately canceled them.
He's not tearing down and setting back up the entire credit line, just the card number associated with it. It won't be reflected on any credit reports.
I would be utterly shocked if nobody using Linode had suspicious activity on their CC. Linode has lots of customers, and at any given time, some of them probably have suspicious activity going on.
There's baseless speculation and then there's I have some information speculation. I'm operating on heuristics which rely on information that is handily available. Yes, in the end you're right, I'm just speculating. But hey, it's better to err on the side of caution.
Credit card numbers are of pretty low value. Like way less than a buck in medium volume and still just a few bucks for the super premium ones. And there is way, way more inventory of them than interested buyers. The likelyhood of a coordinated break in of a large hosting service with the intention of stealing credit cards is pretty low, and the chance that they'd be exploited so quickly is even lower.
Unless the attacker dumped them all (semi) publicly, the more likely explanation is that the breakin caused people to check their accounts and a statistically normal percentage of them showed fraud from another origin. But anybody who sees it will be sure to get online and find others in a similar situation.
Everybody would be doing themselves a big favor if they stopped treating CC info as the #1 scary OMG data theft. The banks programmed you to care because congress made sure they're liable instead of you. Theoretically you might owe $50 due to fraud but practically you never pay a dime. Sure it's a bit of a pain in the ass to get resolved, but it's not worth stressing about until it happens.
I'd be way more concerned if my hoster lost my contact info, ip logs and identity challenge questions & answers.
I contacted Linode support and they've said in clear terms that they have no evidence that payment information of customers was accessed. I initially signed up for Linode because my friends spoke highly of the tech people working at Linode. Right now amidst all the commotions it's ryan's words (some anonymous dude who joined #linode/irc.oftc.net) vs. an established company's. I'm just going to now stop worrying and get back to my work.
If it is indeed true that credit card numbers were compromised, it would behove Linode to tell their customers quickly so they can take the proper action.
With this lack of transparency, I feel like I had no choice but to block my card.
There's no lack of transparency here. Linode expressly said in their blog post that no CC details were leaked.
> In addition, we have found no evidence that payment information of any customer was accessed.
The question isn't transparency, but trustworthiness. Either Linode is telling the truth, and this anonymous IRC person with a pastebin is trolling everyone, or Linode is lying (or alternatively, Linode is incompetent and simply didn't detect the CC access). At the moment I'm going with Linode is telling the truth, because honestly, am I going to believe an anonymous person on IRC over a company I do business with?
I'm thinking here why Linode holds CC data on its servers in the first place. Anyone care to weight in here?
Secondly, if they hold that data, it's possible one day someone will find a security breach and will access that data. The best solution is to never hold that data.
Since I don't trust most of the systems I use AND Linode has not denied it holds that data... I'm more inclined to believe in this anonymous IRC guy and err on the side of caution.
If Linode had come out and said "Look, we don't hold your CC number in our database" then I think there would be very little reason to be concerned. However...
No weird activities on mine either but I will give a call to my CC company anyway. I have had to cancel the card I use on linode twice in the past few months because of suspicious activities. I just didn't think it would be coming from Linode
I asked about the security measures and they answered with:
"We appreciate the response, and we can assure you that we have implemented all appropriate measures to provide the maximum amount of protection to our customers."
Yeah I did the same thing, but new card takes 5-7 days in my case. Very disappointed that I had to find out about this incident here, imagine all the customers who dont happen to look on slashdot or hacker news.
If it's a debit card that can be used as a credit card (and it must be, otherwise it couldn't have been used to pay for Linode), then it enjoys the same protection as regular credit cards when it's used as one.
Very true. If your bank gives you a separate savings account you can transfer the bulk of your money there and just use the card from the current account to limit your exposure.
What about for people who did not use a credit card to pay for linode but instead relied on PayPal. Should they follow the same steps? What about other cautious steps?
I've got a PayPal business mastercard which is connected to PayPal Smart Connect. They may not allow PayPal proper but you could pay using PayPal by way of their card. I use my PayPal card hooked into Smart Connect for a good number of recurring payments like this.
So I guess the answer would be, if you ended up hooking your PayPal account up to Linode in the way I described, yeah follow the same steps as other cards, otherwise it's not even possible to have a problem.
I'd say support tickets or posting on their forum[1] may help try to get a response. But based one one of the support ticket responses posted in the comments in this HN story already, it sounds like Linode isn't allowed to release that kind of information yet. They may be waiting on the police and/or their lawyers to allow them to talk publicly about it. And if that isn't the gating factor, they are probably trying to determine what they are required and should share about the incident.
They may be waiting on the police and/or their lawyers to allow them to talk publicly about it.
Waiting for their own lawyers would be a particularly weak excuse. This is a priority, and they are responsible from conveying that urgency to their lawyers.
If your using lvm you can create a snapshot to do this while online, if not just read from your disk (assuming sda here):
1. (offline) Boot new and old VM servers from live CD
2. old server: dd if=/dev/sda bs=8M | pbzip2 -c | netcat <newhost> <random high port>
3. new server: netcat -l <same port> | pbzip2 -cd | dd of=/dev/sda bs=8M
Compression: You can use something besides pbzip2, maybe pigz of if you only have a single core use bzip2 or gzip.
Security: You probably want to add encryption to this pipeline.
Yeah, I mentioned that at the bottom of my post. Using ssh or some other inline encryption would be a good idea if it is a system you care about. If you have a site to site VPN tunnel between your systems, you can skip adding the encryption.
I thought my only edit had been to replace might with probably. If this was not the case I'm sorry. Maybe I edited it to add that right after posting and forgot-
You don't want to leave out the compression or block size however, so add those back in. Most WAN links are low bandwidth enough that compression will not slow things down (this is usually true even on 1 GBE LAN links for pigz) and in my experience the speed-up is substantial.
pigz is much faster than using ssh compression as it is multicore. apt-get it or http://zlib.net/pigz/
It's pretty necessary if you aren't moving between colocated boxes. "less than a couple hundred GB of data" is still a lot of data to be moving around, even on a 100Mb/s link, which almost no one in America has residentially, and isn't even a guarantee for colo'd machines.
In most cases, compression at the raw block level will result in HUGE size savings, especially since a lot of that may be free space. It might even make this transfer tenable.
That's a great way to screw up the filesystem on the destination. You'll get mounted fs write combined with low-level block writes, filesystem in a blender.
Your procedure already requires booting out of live cd, and then uses "unix way" that might be easy to screw up for less advanced people, so using G4U simplifies it substantially, does the same good job and by no means is a 'overkill' ;)
Generally speaking, this is one example where having a good deployment system starts to look extremely valuable (along with tested backups and restores for non-deployed data).
How about you guys cool it and stop organizing a lynching mob devoid of any real data? It's embarrassing. HN is supposed to be populated with lots of very smart, data-driven analytical folks. Yet, every time something like this happens out of the woodwork come people who would ran you and your children down in the event of an emergency rather than turn around, carefully evaluate the situation, and help you. Don't be a moron. Stop it. For all you know there's a serious law enforcement effort under way that prevents Linode from talking.
For the record, I am a Linode customer and just got a new server to migrate a couple of sites into. My plans have not been altered at all by this. I have no data to suggest I should.
I am a customer too, and I wield no torch or pitchfork, but I grow increasingly frustrated at the lack of response from Linode.
I understand your point about the idea that they may be unable to speak to the issue due to law enforcement efforts, but for the moment, acknowledgement would be satisfactory. I would be happy with, "We're aware of the rumors regarding the intrusion at Linode this past week. We are working with law enforcement and cannot comment on details at this time. However, we will provide a full postmortem once we are able to do so."
The problem is when the explanation never comes. It's OK if it's not this second, but tell us it's coming, and then follow through. Complete silence is frustrating.
>Thank you for your inquiry, and I certainly understand your concern. We are still conducting an active investigation and unable to disclose most information at this time. This being said, we do not yet have any evidence that any payment information of any customers have been compromised. We will be releasing further information regarding the incident soon, so please keep watch of our website and blog for said information. If you have any further questions, please feel free to ask.
That actually sounds pretty close to what you're asking for, although I have to say it didn't make me feel much better. It would be nice if they would make a public statement too.
I get it. It's just that these things keep getting exaggerated, twisted and blown out of proportion in the best Fox News, CNN, MSNBC, et al. style. It's sad to see it happen on HN, where things ought to be far more analytical and data (aka: facts) driven.
Before it was the widespread hacking that resulted in Bitcoins being stolen. And users were never told exactly what happened and what was done about it.
People have every right to expect the worse with a company with a track record as poor as Linodes.
I don't think that quite explains it. Someone who calls himself "ryan" on IRC makes unsubstantiated claims, and in response many dozens of people say loathsome, sneering things about the security practices of this company. The appearance of a mob emerges after a thread exists, it doesn't create the thread.
Reading through the IRC chat log it didn't look like he'd proved he had the CC numbers.
I mean, if I was a linode customer I'd definitely be on the phone to the bank, but this guy presented no evidence I'm aware of that he had the kind of access he claims.
From a purported abridged chatlog with the alleged hacker:
> 05:42 < ryan||> credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security
> 06:00 < ryann> They did try to encrypt them, but using public key encryption doesn't work if you have the public and private key in the same directory
Here is what Linode replied to me when I asked them about that chat log in a support ticket:
Hello,
Thank you for reaching out. We appreciate and understand your concerns. At this time the evidence suggest that this activity was targeting a specific customer. We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
We have no comment regarding ryan*'s comments in #linode. You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence.
I am sorry that we cannot provide more information at this time. As always feel free to contact us at any time with any future concerns.
Regards,
Quintin
These guys are looking totally incompetent at this point.
If you believe this Ryan guy, credit cards stored on the same server as the key to decrypt them, Lish passwords stored in plain text, they've known for some time and lied about what actually happened and now they're saying "we won't do anything about it" via email?
"You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence."
Unbelievable.
Edit: not to mention they "made a deal" with the hacker not to tell anyone? What the hell?
That's a rather key assumption. If you don't believe him, then all you have is a trolling (or at least self-aggrandizing) hacker whose credentials consist solely of logging into an IRC channel, refusing to identify who he was working with, and offering no tangible proof of having compromised any CC info.
On the other hand, it's conceivable that if ryan managed to get into the files a customer was hosting on Linode, and that customer was improperly storing CC info, then their customers' info would have been vulnerable, and ryan's claims would be sort of half-true. Even so, that wouldn't directly affect other Linode customers or put liability in Linode's lap.
for the record, i am the person who started the WHT thread.
there is a mixture of truth and lies on both sides, to be honest.
i am annoyed with it, because i reached out to several linode employees privately to given them an opportunity to explain what was going on -- they either said 'no comment' or said my linode was fine.
based on the irc log, that is clearly not the case. which is why i decided to raise my concerns publically.
luckily for me, my linode was not doing anything mission-critical, just some secondary monitoring and running an ircd for a network i like using, but there are others who are using linode for mission-critical work, and they deserve more transparency than this.
To be fair the hacker didn't say the keys were stored on the same server as the credit card numbers, he said they were stored on the web server. It's most likely the database containing the CC numbers resides on a separate set of boxes than the web servers.
Despite what the other replies here are saying, this seems like a perfectly acceptable response to me. This comes off to me not as they're refusing to talk about it, but they _can't_ talk about it, presumably because of an ongoing investigation. I'm not sure what else people here are expecting them to say.
That is an absurd response. I don't care if they believe a specific customer was targeted, I want to know what happened and what information may have been compromised.
They probably can't say anything just yet. It's fair to be mad about them not realizing (or worse, covering up) the breadth of this breach, but do keep in mind that if they're working with the FBI, they've probably been asked to keep a lid on their official response for a few hours.
That is not the way to handle this issue. I've found my one problem with Linode is they are arrogant. It comes off pretty strong if you ever ask them questions in chat.
One thing to note is that the irc channel if that is what you mean by chat, is mostly populated by non linode staff. And many of us there tend to be sarcastic as we idle there and chat about all sorts of stuff while we are bored at work or whatever. Unless it's someone with ops, you aren't getting an official reply and even then for something official they usually refer to ticket system.
Can't think of the exact word to describe that practice (what you did in response to the long list of questions which I've seen) but on the part of the company requesting you to answer the questions it's more or less a "absence of malice" type of thing that allows them to appear that they are doing the right thing while fully knowing that people are doing what you are doing. It's a "we will look the other way until we need to show that it's not our fault because we have passed the liability to you - look you acknowledge doing all the right things".
It does not mean anything until they decide you are not compliant and you need to prove you are compliant. (I've never had to but I'd appreciate insight from people who have)
Actually, if you're processing cards directly, you do in fact need to have an PCI-qualified outside firm† (a QSA) audit you for PCI compliance. But those audits are notoriously superficial; PCI audits are a race-to-the-bottom affair.
The quality of PCI security audits is a continual aggravation to everyone I routinely talk to in my industry. I've told more than one client: if you need a QSA audit, get the cheapest one you can. If you need a software security assessment, don't use a QSA firm.
Will add that just having gone through an ICANN registrar audit (which by the way were specified and supposed to be done literally 10 or 12 years ago but never requested by ICANN) with a third party company hired (accounting firm) it's total compliance theater.
Add: "hired by ICANN after a bidding process". Same happened with data escrow which was just implemented a few years ago and is operated by Iron Mountain.
Companies are split into PCI Levels based on how much money/customers they handle. Level 1 are big companies like amazon, level 2 are medium sized online retailers generally, and level 3 are smaller retailers.
The 'lower' your level, the easier the PCI audits are. If you are level 1 you have mandatory external audits. If you are level 2 you have a 'self assessment' which is basically a checklist which says "Yes, I promise I'm in compliance".
If you have a confirmed breach, you are upgraded to Level 1 merchant audit requirements. This is generally quite costly as the external audit is extensive and must be paid for.
The audits are toothless and ultimately the audit only happens once (if ever) and people keeping pub/secret key in the same place unprotected... well.... They're just unlikely to get security at all...
Am I the only one who is more confused about why there are compiled java classes and AMI BIOS updates in the www directory than about the hacking itself?
> Am I the only one who is more confused about why there are compiled java classes and AMI BIOS updates in the www directory than about the hacking itself?
Sysadmins being lazy.
Somebody needs to get a file from a workstation to a remote machine. There's a firewall in the way somewhere that prevents SSH directly between them or one of them is a Windows box that isn't running an SSH server.
The "correct" solution is complicated and takes 5 minutes to setup. So the sysadmin just copies the file to the web server and downloads it with a browser on the other end. Because port 80 is always open.
Something that took me a long time to notice was that you can actually copy files directly over a RDP session from/to the local machine or other RDP sessions using the clipboard.. Even nested RDPs. Has been an absolute timesaver to know about when doing Windows admin work.
I think it depends on how you mount your clipboard/drives on rdp connect. To get it right with windows you just check the share clipboard/share drive checkboxes and off to the races. With rdesktop you have to throw the -r flag and mount a clip board and then the -r flag and mount a drive. Not sure about other Linux clients but I'm sure there's a similar option in all of them.
Its also why we invoice and take wire payments rather than storing CC details. There's just so much to go wrong.
Also PKI is shit for this sort of thing. As demonstrated, the moment that public key is gone, then the whole system falls like a house of cards. For the non believers of this fact, why else would there be a certificate revocation list and root CA updates for windows periodically...
Even better to use someone that isn't your payment processor, so that should you need to change payment processors you don't also have to re-acquire all the billing info from your customers. You can use Stripe today, PayPal tomorrow, and Braintree the next if that's what works best for your business.
There's always going to be single points of failure, but which is more likely: you want or have to change payment processors (you've been terminated, your fees have gone up, you want to switch to a lower cost provider) or you want to change flat-rate vaulting services? Plus, Spreedly will give you your data if you leave, whereas there is no way to get stored billing info out of most payment processors.
Storing credit card info just helps make you a bigger target. If your a small company, better let someone else store card info, let them be the target.
Also you're fined by the credit card companies if you lose card information. I believe it's a per card fine, so it get expensive really quickly.
Actually I don't get why any company would choose to store credit card information, when most payment providers will do it for you.
True, but I'd rather fix the problem that got them in, force reset of passwords, and delete all customer keys and require them to create new ones than be like "uhhhh, our data was hacked and your credit card is safely encrypted... But we had the encryption key on the server too, oops"
Following the traditional responsibility/accountability dichotomy: They are responsible for storing the cc number securely but you are accountable when something goes wrong (because you gave them that task)
Much like Linode are responsible for hosting my clients site but I sigh am accountable when something goes wrong.
In what ways are wire payments better than using credit cards? In wire payments aren't you using the actual bank account numbers along with routing numbers which is also very sensitive information ?
Also I do not think, but I am not sure, that fraudulent wire payments/transfers are reversible.
Wire transfers often are not able to be undone once they happen (and are accepted by the other bank). This is the reason why there's so much verification that happens in wire transfers. (I helped develop 2nd factor authentication used for authenticating wire transfers for a financial company)
My Visa card that I used with Linode was stolen and used on an Amazon order I didn't authorise last week, my bank successfully blocked the charge. Someone else reported their Visa had also been compromised in the thread 2 days ago, looks like that confirms the suspicions: https://news.ycombinator.com/item?id=5542015
Poor show Linode. (edit: worth noting I use the card with other things too, I have no confirmation it was leaked through Linode other than the compromise happening at the same time these supposed leaks happened).
I have two accounts with them, one for my day job and one through my LLC. Right now neither card is showing unusual charges.
My day job had some Google Apps account compromises last month, and this is making me paranoid that the database that contained hashed/salted passwords for our Intranet hosted on Linode was the culprit. The time frame doesn't seem to line up, but we didn't see any evidence of phishing or compromised desktops being involved.
(Don't want to spread fear - I haven't been able to find any evidence our Linodes were compromised either.)
I'm normally huge a Linode evangelist, but I'm severely disappointed with the lack of transparency on this. I'm debating right now whether to rebuild all our nodes from scratch as I'm not sure they can be trusted.
I had a CC problem, and I use it for a very restricted number of online services. Luckily the fraudulent attempt was blocked by the CC company. Linode was not at the top of my list of suspects (there was another company that seemed to be storing passwords in cleartext), but now I'm wondering. This was a couple of months ago though... I wonder if that's the right time frame?
Every site can be hacked. It's just a matter of time. You just have to properly react: call your CC provider, check for any charges on your bill, and move on.
I do not think anyone here is actually worried about their funds; as mentioned below, any reputable provider will have such charges promptly reversed. The problem is instead with their response to the situation.
Linode has addressed the breach, but assured customers nothing of value had been compromised. This infers two thoughts. One: they knew of the breach and lied, thereby unveiling a unforthcoming and dishonest nature. Or two: they did not properly investigative the severity of the issue, thereby suggesting incompetence. Both equally reprehensible.
Wait a second.. you consider a provider giving data from your VMs to another random customer a SMALL issue?
Maybe you don't have anything of importance on your VMs, but plenty of people do. That data could contain credit card data, passwords, etc, etc, etc. It is very much a large issue.
Sure. My thoughts don't apply to everyone here, and I certainly can't claim to be unbiased since I like DO so much.
According to DigitalOcean, they stated that this impacts 3% of all machines, only the largest and most expensive servers. None of the smaller plans were leaking data.
I don't know how many credit card numbers were leaked from linode, but I'd guess more than 3%.
Second, if security is important to you, you can use 'dd' to clear the machine yourself before shutting it off. (In fact, good data destruction policies mandate the use of 'shred' et al anyway). On Linode, affected users don't even have a workaround (like this) to avoid information compromise.
There is no evidence to suggest that Linode leaked credit card data.
Personally I took precautionary measures and just called my bank to replace my credit card, which I think is the sane approach, as when it comes to hacking you have to assume the worst.
However, your statement on "has lots of small issues but fixes them the same day" is just stupidly childish. Linode's issues are bigger just because they are a bigger target.
Great, now I am feeling paranoid although I don't see any unauthorized charges on my card. Does anyone know if debit cards are legally protected the same way as credit cards with 0% liability.
These are the FTC's rules [1], I'm not sure if Visa or Mastercard can make them 'better' (give you a larger window). They have an interesting tidbit below their chart -
>If someone makes unauthorized transactions with your debit card number, but your card is not lost, you are not liable for those transactions if you report them within 60 days of your statement being sent to you.
Isn't it free to get a new card? That'd be the easier way than worrying.
Generally these days they are, but the problem is the money is removed is from your account by the time the charge appears, (at least a period of time), whereas on a credit card you have 30 days to review charges. This could cause overdrafts, etc. depending on timing and amount. Those too can generally be reversed, but the whole thing becomes more of a headache. I never use debit cards for any kind of recurring charges, and I avoid using them online in general.
The same is possible with a credit card - you can have your card maxed, get nailed with overage charges, declined transactions, etc. It's still a headache, but when I've had it happen to me I think I had the money restored within a few minutes of making the call.
they typically have the same protections, the only difference is that while credit cards only check that the charge can be made during the authorization period, debit cards lock up the bank's funds immediately which may cause your checking account to be temporarily unusable even if all the charges are successfully disputed.
a good practice that bankers constantly tell me is to have a separate credit card for online purchases for the fact alone that it is one step removed from your checking account.
I know someone who got their debit card cloned. While the bank eventually repaid him, that did nothing to repay him the additional fees he owned his normal debtors (e.g. rent, utilities, etc).
With a credit card you aren't losing "actual" money. You are losing the bank's borrowed money which the bank pays back. With a debit card you're losing cash which you won't be able to replace yourself and which the bank might take days to weeks to replace.
Even if you NEED to borrow while your credit card is out of commission you can either use the overdraft facility on your debit card or other quick sources of credit. Hard to get quick cash without going to a pawn shop.
I would cancel that card right now. IMHO You should never ever ever ever use a debit card anywhere else other than the ATM. Credit cards give you way more financial protection.
You should talk to your bank as it depends on the network your bank uses. Most of the time it's something like a 48 hour window to challenge charges but it's far less at some banks.
> [With a debit card,] "Until the bank provides provisional credit, you could temporarily be out of pocket for the amount in dispute," said Richard Foley, an FDIC attorney who specializes in consumer issues. "This would not typically happen with a credit card because consumers can withhold payment of the amount in dispute."
> Also, as discussed on the next page, consumers have better federal protections when they purchase faulty goods with credit cards.
They are. Protections are exactly the same (in the US at least.) You have protection from the moment that you learn of the problem, not when it happens.
Not that having your account drained doesn't suck, but your worst case scenario there isn't terrible unless you fail to check stuff and be responsible.
If this is true then all the trust that Linode has built up over the years was just thrown out the window. According to the hacker they've known for 2 weeks and made a deal with the hackers. Ultimately, they were as far from transparent as it gets and on top of that they did a horrible job with their security.
Hopefully, they own up and start being transparent.
If this is true then what alternative hosts should I look at, besides AWS?
Two alternatives often mentioned on here are DigitalOcean and RamNode.
I've only used DigitalOcean. My anecdotal experience from running a Chef Server on a 1GB instance has been pretty mixed. The price is good, but network and CPU performance feels very variable to me. A month ago their Amsterdam servers were unable to be resized, and there was nothing about it on their status page. I tweeted and was told they'd be working "some time later today". Doesn't fill me with much confidence in general.
I'd still choose Linode for anything of importance - their long reputation is well earned in my opinion. But, if this breach is true, I hope they handle it well.
"credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security"
That's just poor security and 100% they're own fault. I accept that there are security issues with every platform, but basic security measures and being transparent is still expected. My biggest issue with them in all of this is not being transparent.
Looks like someone who likes attention on some random IRC channel who is apparently a hacker may have hacked our system and we don't know who/when/where/why/how or what they may have got. Nor are we sure we were even hacked???
It takes time for people to investigate stuff. It's not just a couple hours. Also some random guys words on IRC (who could very well own INSERT RANDOM HOSTING COMPANY for all we know and looking to scare people off Linode) should be taken with a grain of salt.
They are supposed to say that they would never ever store the CC numbers this way. Otherwise their customers (like me) have really no better option than to block their cards, which is quite an inconvenience. This is exactly the trouble I was hoping to avoid by not using a cheap VPS hosting.
If the information he offered is accurate (e.g. the public and private keys were stored together on the webserver), that wouldn't take a long time to confirm.
"Logical" is not a fancy synonym for "severe" or "unforgiving", and is probably not the word you wanted here. There are sometimes good, logical arguments that you can make for cutting people some slack.
I've been using digitalocean for a hobby project, and am planning on launching a more serious project with them. For the past two months I've used them (so, not much experience, but some) I haven't had any issues, and they are very reasonably priced.
Just like I can have application-specific passwords for my Google account, I wish I could have application-specific credit card numbers from my CC issuer.
If I had these, I would immediately cancel my Linode-specific CC# and reissue a new one. I would not have to worry that my other recurring bills will go unpaid, or spend hours dealing with tracking them down and changing them.
Crazy, thanks for sharing. I use BofA and am interested, but here is the issue:
> Set your Valid through date for up to 1 year in the future
I'd really like it to be up to the expiration date of my card. It's probably worth doing anyway, I suppose. Thanks for the heads up. Given the zero-liability status, I'm surprised that banks don't promote this feature more visibly.
I've used the BoA and it's fantastic for one-time purchases. Like another commenter mentioned, it's only good for 1 year and you give a global cap. It would be nice if it was more permanent and allowed for a weekly or monthly cap so you could use it with subscription services or something like Amazon.
The portuguese ATM network operator provides this for free (its called mbnet btw). You can even set expiry times and value limit, it's the best thing to use when paying for stuff online. Want to buy a 9€ game? Just create a 10€ card and use it.
Visa have this (probably others too) that you can generate and either set a time or value limit on. Don't think you can set a monthly limit though unfortunately. They call it e-cards (in sweden at least).
Because the CVV is used to indicate the "presence" of the customer at a transaction. CVV1 (which is on the magstripe) is used to indicate "card-present" physical transactions, CVV2 (printed on the back) is used for "customer just typed this in" non-physical transactions.
In that case, specific Bitcoin users were targeted.
It's pretty much why I don't trust Linode.
You can't trust a company which puts random AMI BIOS files on the main index directory on the main web site. You can't trust a company that can't even lock down their own Linode customer service portal (which could lead to a breach of each and every customer's VPS).
Perhaps history is fuzzy for people when new announcements come out or low prices are around.
Off topic but still relevant, but doesn't it seem a bit primitive that companies have to store you CC# for recurring payments? The one number that uniquely identifies your account and everyone you want to re-use it has to keep a copy. Couldn't the credit card company issue some unique ID to each vendor for recurrent payments? Ex. the vendor issues your CC# to the CC Company for charge and recurring process. The CC Co. responds by replying back with authorization and an ID unique to that vendor that says "Use this number for charging this customer again, but it will only work coming from you, so if you lose it, it can't be used elsewhere". The vendor then discards your real CC number.
What if you want to change processors? If you weren't storing the CC details, wouldn't you have to have customers enter all their details again? I imagine this could cause a drop in revenues due to people either forgetting, procrastinating, or just not bothering.
It's never cool to be actually- or quasi-locked into a vendor.
Most decent processors have processes in place for the transfer of CC numbers. I was involved with this process at a decent-sized magazine, it involved armed security, an encrypted hard drive in a locked container and millions of dollars of insurance. It's not an easy process, but is possible.
Some will let you export the data, but it's a hassle. They burn it to a DVD and ship it to you and you have to sign tons of stuff freeing them of any liability.
Otherwise you just ask customers to re-authenticate. Often you have a few months headway for a switch like that.
It seems there are a LOT of incompetents then. Are the payment processors you're talking about something the vendor interfaces with in the background and not a third party that is directly utilized by the customer (eg. Stripe, Paypal)?
This is basically what Stripe does. The "CC Company" basically doesn't offer more than a simple yes/no API -- "approved w/ #" or "declined". It'd be great if they could do more, but that's where the opportunity for folks like Stripe lies.
I guess it just surprises me that the CC companies don't do it themselves. Is the additional overhead of such a feature so high such as to warrant the choice between no payment processor (x% fee per transaction) vs. third party middle guy (x% + a bit more)? Not to mention the money the CC companies would save by not having so many fraudulent transactions (assuming they come out of their pocket, I honestly don't know).
Well, you can get AVS checks back too. But in general the entire system is so far behind the times. It really needs to be overhauled with security as the focal point.
It's amazing that it doesn't work that way. You could argue that it's because of the amount of infrastructure already in place, but why couldn't a new system be gradually and optionally rolled out?
Stripe (and probably others) has functionality like this where the seller's server never sees the CC number, and developers can store a unique token to re-charge the customer at a later date.
Depends on what part of the chain you are. Most gateways/processors offer some sort of token that the end user uses.
You then put your trust in the gateway/processor to store the credit card. Which I assume is most likely behind the best possible security stuff money can afford. Since that's their entire business. One screw up and their gone.
The majority of CC names have "Virtual CC numbers" which is precisely this - You generate a new number, which links to your account, but can only be used for the merchant you specify.
But isn't that just between Stripe and the company requesting payment?
e.g: Acme, Inc. sends Stripe your CC#, Stripe sends them some unique token, and they store that; correct?
So Stripe still has your CC#, and is at risk.
So this is really just risk mitigation; what I think TP is suggesting we need is unique authorizations at the banking level.
Something on the order of virtual credit cards, or temporary tokens, which are ultimately verified by your bank [or in other words: the lender(s) making anti-fraud guarantees, etc.]
(e.g: this token is authorized for 24 hours up to this limit; this token is authorized indefinitely up to $xx/mo.; this token is authorized for 1 year; etc.)
No. Customer sends Stripe their CC number, via AJAX in the browser. Acme, Inc. never has it even transiently. Stripe return a token to the browser, which is sent in a POST to Acme, Inc., then they verify it server side with a private API key.
Edit: yes Stripe has your number, but since their sole business is about securing that information, they probably do a better job of it than your typical online merchant.
I think that Linode did a big mistake here. Let's wait for a formal communication.
But this is the moment to support them.
Yes, maybe sounds crazy.
When you host on any third party datacenter, you take risks that something like this could happen. So, deal with it. Check your credit card, if your receive something wrong, call to your card and that's all. But we need to support also the good work, and this guys do great work in the hosting business. Just my opinion.
Well said. It's a fact of life that companies get hacked. So it's no surprise that it eventually happened to Linode. If you flee somewhere else, all you're doing is hoping that the other company you run to won't get hacked rather than using any logical thought.
I can think of two good reasons why you should flee Linode. It remains to be seen if either are actually true, and until indications say yes, then panic is unwarranted:
1. If it becomes apparent that Linode is far more vulnerable to hacking than other hosting providers. But one hack alone does not prove this.
2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.
Linode has already grossly mishandled the situation by not coming out with a complete statement about what exactly happened. I only read this news because it was posed here -- no email notification, no update on their homepage, no twitter, no nothing.
The alleged hacker has made serious and specific claims, and Linode has done jack shit; without more information, how should I proceed? I don't want to call my bank and waste time getting a new credit card (not to mention replacing a million and two services) without a confirmation and I can't get a confirmation because Linodes people are having a circle jerk (or whatever the hell they do).
> 2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.
Linode's handling of the Bitcoin incident last year was sub-optimal. This too has been sub-optimal, given that credit cards were exposed but all we heard on Friday was to change our passwords, and even that was claimed to just be a super-careful precaution.
Linode needs to start giving us some frank talk ASAP. They've already burned through a very generous helping of benefit-of-the-doubt.
[Warning: imperfect analogy follows.] It's one thing if Linode is like someone who gets drunk and crashes their vehicle. That's 100% their fault and they've burned any goodwill. In this case, however, Linode is like someone who was carjacked. Perhaps Linode shouldn't have been driving that type of vehicle in an area known to have people attempting to carjack every single vehicle that drives by. Perhaps they should have installed thicker bullet-proof glass. Or even have taken measures not to trust any locks that the manufacturer insists are secure but have zero-day exploits. Regardless, Linode is still the victim of unscrupulous criminals. Maybe they could and should have done more but the bigger question is now that they've been carjacked, what are they doing to ensure that the carjackers haven't installed anything malicious that still remains in the vehicle?
I think a more accurate analogy is to say that Linode is moving your important files from one location to another in their moving van. They park at a 7-11 to run inside and grab a snack, leaving the van unlocked. An intruder comes along, opens the unlocked doors, makes a few copies of your files and leaves. Linode gets back in the van, notices the intrusion, does nothing except tell you that "you have nothing to worry about, but you may as well change your locks" and then when the truth comes to light, they basically stop returning your phone calls.
If the allegations are true, then Linode was keeping encrypted CC numbers, with the decryption key in nearly the same place.
Trying to make the analogy more sufficient by incorporating this type of fact would only make the carjacking analogy more absurd. At the end of the day, an analogy is not needed.
Seems like it usually takes an event like this for a company to really crack down on their security practices. I would wager that Linode will be one of the most secure providers in the coming months. Whether or not people will trust them is another story.
It seems to me that the fundamental problem is not that they got hacked (although it seems that storing a decryption key in the same directory as the encrypted data is over the top careless), but their response to the disclosure that they got hacked. I realize that there may be limitations on exactly what they can say, but they should be as open as possible on what may have happened, what they are doing to protect their customers, and what their customers should do to protect themselves. Customers taking action when there wasn't a breach is less of a problem than not taking action when there was, in fact, a breach.
Linode really needs to make a statement about what happened with this hack, stating if credit card information was taken. A lack of communication does not help me trust them. I'd rather have them speak up as to what happened and know if I need to have my CC reissued.
FYI, I just learned from Linode support that accounts have both a default LISH password AND a default API key, which means that even if you've never set a LISH password or generated an API key before, you still should still go and reset them. This is not what I would consider expected or desirable behavior.
I've never generated an API key, and Linode showed it as blank. I generated a new one anyway, but I can't imagine how they'd have a default key and somehow not show it in the UI.
I know. I've asked for further clarification, especially since their email on Friday said API keys should be reset "if applicable."
Edit: Groan, here's their clarification. It's starting to look like they don't know what the heck they're talking about:
"Thanks for getting back to us. To be extra cautious it would not hurt to regenerate your Linode API key. You can do that in your user profile. Please let us know if you have any other concerns we can address."
After seeing your original post here, I also asked for clarification, and received a similar reply from support:
The Lish password is set to a random string by default, however we would still recommend resetting this password even if you had not set one manually previously.
I had expected that if the password was not set, then password auth was disabled. I've told them that's what I want and have asked when it will be implemented.
I'm kind of upset they didn't clarify this in the initial email/blog entry. The way it was worded ("if applicable") implies that resetting the API might not be necessary in some cases. I think it is reasonable to assume that those who never generated an API key in the first place would've fallen under such a bucket.
Now it sounds like basically everyone should have reset their API key. Bleh.
Just rang my bank to cancel my debit card. Hate doing that. Now I have a week or two of failing payments, bills, etc to look forward to.
I will probably be moving away from Linode after this. The poor response to this and lack of full disclosure, plus reading that they're using ColdFusion (wtf?), means I don't feel I'll be able to trust them any longer. It's a shame because their UI and service is generally fantastic.
There's nothing wrong with ColdFusion, especially if you've had it around for a while. It's not as glitzy as Rails, but it works and it's still supported and modern. Besides, this isn't ColdFusion's fault. Leave because Linode violated your trust, but not because of the programming language they wrote their site in.
It's closed-source, made by Adobe and seems to have a bad security record - there are 3 things wrong with it.
Besides, it's not the reason I'm leaving - it just makes me question them. I'm not after glitzy. If anything, I'd have expected Linode to have been written in Perl or something.
Not with a debit card - you don't get the same protections as a credit card and I'd rather just have a few days of hassle and then know my card is secure than be unsure and have to constantly check my account for odd transactions. Also, whilst I may get money refunded if taken from my account, if I miss bill payments as a result - that would affect my credit report and I don't know if that would be removed when I report the transactions as fraudulent.
The Debit Card was used in a Credit transaction, so Visa's general protections still apply. You can dispute any of the transactions if they were done through credit (which online ones are nearly 100% of the time).
Had $3-4k stripped off a debit Visa while in Europe on holiday. Sent a list of fraudulent transactions to my bank and they had them sorted and refunded within an hour.
I'll be watching my card more closely after this Linode incident, but not cancelling it preemptively.
I'm curious how he's going to backdoor the binaries running on my computer built by the launchpad servers without anyone noticing. Granted, I'm not checking the commit logs every time apt gives me a new version, but all the same.
So you're unfortunate enough to be a customer who had their CC leaked. So you spend 5 minutes changing your password (you use unique, non-formulaic passwords, right?) and 15 minutes on the phone to CC company to ask for a new card. Then you use your backup card for 2 weeks (you have a backup card, right?)
A month later, spend 30 minutes on the phone with CC company only if strange transactions appeared.
Not the end of the world. The CC industry is set up well to handle this kind of thing.
To dismiss this breach seems odd to me. The tech community in general has placed a lot of trust and faith in Linode over the years. The shareowners at Linode have surely been great beneficiaries to that. Part of that "unspoken agreement", if you will, is that Linode be competent at what they do and that means keeping your data and information secure.
If even an iota of what I read in the abridged IRC log is true, Linode doesn't seem to care much about security or protecting Linode customer data. I mean, storing "encrypted" card numbers alongside private/public keys? Really.
Sigh, really? Ok, you typed your credit card number into a web browser at some point. If your sole reason for doing so was "I absolutely trust the people on the other end of this socket not to do what 99% of all people handling credit card data do whether they pretend otherwise or not", instead of something like "hmm that reminds me, I haven't scanned last month's statement yet", then the problem lies squarely with you, the uninformed consumer.
I will happily dismiss this breach, not because they didn't make some amateur crypto mistake, or because they weren't using freaking ColdFusion, or because they were storing data in some nice compartmentalized form, I reject because this happens every single day and has done for decades, and there is an entire sub-industry built around its after-effects. If you don't understand this you shouldn't own a credit card.
If you type a credit card number in online not expecting to recuperate any damage caused from your card company, call them up now for clarification or cancel the damn card. That's equivalent to stuffing cash in an envelope and posting it to Nigeria because some prince promises he'll keep it in a safe for you. It's 90% the reason you should be using credit cards in the first place. Think.
Linode should not be rubbished here. They've got one of the largest VPS installs around, so they most likely know their shit. They make an ultra-common CC mistake that has happened daily for almost 20 years now, by companies large and small, got pwned due to a bug in someone else's software, and you think I'm going to play along with the righteous indignation bullshit here? GTFO.
Let he without sin cast the first stone. Despite 20+ years' experience I still cannot cast that first stone. I make bullshit mistakes like this every day, and despite your grandiose delusions you probably do too.
As for whiners complaining about their data suddenly being insecure, well, data security 101: you're making the same bullshit mistake Linode are making, and despite that you're complaining about it. If you care about data security in the "cloud", hosting it on a freaking VPS is not the way to do things.
So because companies A through X are irresponsible with data, customers should regard that as acceptable and give company Y a free pass to do the same? I don't understand how a reasonable analysis of the situation can come to that conclusion.
You don't know the names of companies A through X, or supposedly safe Z for that matter. All you'll be doing is an enormous amount of work and bother to move from Y to, lets say, A, because you think you'll be more secure but unfortunately if anything its probably the other way around, its just that A hasn't been hacked... yet... so far as they know...
none of them get a free pass they all suck, but the one that just got busted is probably going to be a little more security focused in the near future.
Hmm stay at a place that just got burned, or expend lots of effort to move to a place that hasn't been burned yet...
"Because Nigerian princes A through X are irresponsible with your cash, budding lottery winners should regard that as acceptable and give Nigerian prince Y a free pass to do the same?"
Of course not, and this drives to the very core of risk management. I've signed up for some very shady online services in my time, doing so in the full knowledge that should a product or service not be rendered as advertised, I am guaranteed to be able to reverse the relevant charge. Even when I the consumer am doing something shady (in a case last month, attempting to import goods I knew weren't certified for the EU), the system still works for me. This is the sole reason I use a credit card rather than, say, my current account's Visa number.
It's not even about assessing the risk of whether or not you're going to get ripped off, but whether or not a particular company will cause you the inconvenience of the aforementioned phone calls.
If you work on the assumption that you card data is safe, you quite simply aren't safe enough to be in possession of a computer or card. Credit cards aren't built on that assumption, instead their entire motivation is based on risk profiling both the consumer and merchant, and terminating agreements when various thresholds are reached. In return the industry guarantees that in the minority of cases where things go wrong for the consumer, the problem can be corrected swiftly.
It's understandably upsetting that their customer database might have leaked, and I can genuinely understand peoples' concern over that. But as 4chan has taught us, there are very few people left in the west whose address and telephone number aren't available within even an hour's Googling.
As for locating confidential data on machines shared with other customers and managed by a piece of unaudited software, I have no sympathy for that. That's the price of a VPS, and why it's so heavily discounted compared to real hardware.
> If your sole reason for doing so was "I absolutely trust the people on the other end of this socket not to do what 99% of all people handling credit card data do whether they pretend otherwise or not", instead of something like "hmm that reminds me, I haven't scanned last month's statement yet", then the problem lies squarely with you, the uninformed consumer.
If your expectation when taking credit card numbers was "I'm confident in my abilities to keep this information safe, and if I get hacked I expect my customers not to move to another service and never, ever touch mine with a ten foot pole", then the problem lies squarely with you, the uninformed business.
You missed the step where you have to find all charges going to your old CC and then deal with moving every one of those accounts to your new one when it gets there. Hopefully you don't incur any late fees while you're going through the process!
Kind of sucks to have to spend hours doing that for someone else's oversight. It's not the end of the world, but it paints a clear picture about where a company's priorities are.
Maybe I'm weird, but I know exactly which binding credit agreements I'm in and how they're paid, and definitely none of them get paid using another binding credit agreement. :)
I mean, I know all that I'm in, but I can definitely see myself missing one or two if asked on the spot to recall all of them. Phone, cable, AWS, gym, power, garbage/etc, insurance,... I'm sure there are one or two others. Even if there aren't, I'd feel compelled to go through everything to make sure there aren't more.
Even just dealing with all of those is going to be a pain in the ass though. I'd probably end up spending hours all together just on hold with some of those people.
You don't pay for Netflix, Adwords, Amazon Prime, AWS, etc. using a card? If so, yes, I think you're weird. What do you do? Give them all your bank details?
I'm confused. The person you were replying to said:
> you have to find all charges going to your old CC and then deal with moving every one of those accounts to your new one when it gets there. Hopefully you don't incur any late fees while you're going through the process!
I've never been charged a late fee by a firm I didn't have a credit agreement with. Perhaps other parts of the world are more insane, but here that is definitely not commonplace.
I don't know who does and who doesn't, honestly. I try to avoid being delinquent. I know from the last time I had to change my card that my phone provider and ISP certainly do. Anybody who charges you on a recurring basis certainly can — they just add the amount to your next bill. They won't take you to court for it, but it will be added to the amount that you must pay or be disconnected.
Sorry, I should have said "fees". I've recently had to deal with this process, and I've found a few vendors who charge a penalty if the attempt to charge my card doesn't go through for any reason.
Its worth noting that charges can be credited to cancelled cards under certain circumstances. Happened to me and the bank said it happens regularly. True, its usually a painless process; but its not that simple.
Found it interesting that Linode uses Coldfusion. Wonder if Adobe has anything to say about the apparent 0-day.
If the hacker's claims are true (Would appear so, the directory listing checks out) then Linode really need to address this ASAP. Passwords are one thing but to have CC details leaked is even worse. I'm not familiar with CC processing but it seems like bad practice to store the encryption keys on the web server.
It wouldn't take a zero-day flaw in the Coldfusion stack for a CF application to have an undocumented vulnerability; in fact, it's much more likely that the vulnerability is in the application code than in the stack itself.
A patch has recently been issued (09 APR 2013) by Adobe for the various versions of ColdFusion:
"This hotfix resolves a vulnerability that could be exploited to impersonate an authenticated user (CVE-2013-1387).
"This hotfix resolves a vulnerability that could be exploited by an unauthorized user to gain access to the ColdFusion administrator console (CVE-2013-1388)."
Depending on who you're talking to, an app-level vulnerability in a Linode management console might be called a "0-day". But it's true that a CF stack flaw is not impossible.
The problem I have balancing the likelihood of CF stack bugs vs. CF app bugs is that I've had to assess a bunch of CF apps, and they're uniformly coded to mid-1990s best practices. No matter how many bugs have been announced in the CF stack, as a betting man my money would always be on CF app bugs.
This is not a brand new 0-day. This is a bug that Adobe communicated and patched months ago, a bug that affected a lot of folks who didn't follow the standard practice of locking down an administrative directory on the website.
The basic overview is this: CF servers have an administrative portal at /cfide/. A bug in the scheduler code (think cron) allowed remote attackers to upload arbitrary code to the server and then execute it. Savvy attackers could upload their own backdoors directly into the administrative folder on the site and then execute that code to gain additional access.
As a Linode customer (admittedly only for a small VM I play around with) I have to say I've been impressed with their service and their prices of course, and I'm waiting for further confirmation about the depth of this hack. I was unaware Linode was using ColdFusion. It should be pointed out that CF is a very mature language, akin to ASP.NET. It is actively maintained by Adobe and used by a huge number of websites globally.
I wonder if Linode has a requirement to have their CF admin site accessible outside their network (assuming, of course, that the attacker didn't first gain entry into the corporate network, and then attacked the CF installation)?
As someone who still maintains a very old CF application, I am sure to lock down access to the admin site via IP restrictions.
I had really hoped that they had changed their stance on incident management. If it's true that they suppressed information about a possible wide-scale compromise where customer data could have been affected, then despite everything else about their service that's so great, there's no way anyone should want to continue to be a customer there.
Given Linode's past behavior and the information provided in the IRC chat, I think there's reasonable suspicion that customers' password hashes were stolen and Linode wasn't completely honest in their recent email to customers.
Is it just me or is each passing minute without an acknowledgment of this issue bad for Linode? There's tens of thousands of customers right now who would kind of like to know if they need to request new credit cards or not, or don't know about this and deserve to know that their VPS provider's credit card database has been compromised.
The chatlog does provide some evidence that it is indeed the hacker, but does little to convince me that he got CC info and Linode is not telling us the whole truth. The evidence he provides is just simple source code snips and the directory listing, which would be expected based on what Linode has told us.
This could very well be the hackers own submission to /. trying to get more attention for his hack by claiming he has CC numbers which I doubt he has.
That's a fair point. I'm speaking more as a judge/jury view on the situation though. I don't think Linode users should panic and run for the hills just yet based off this alone.
I'm looking forward to seeing an official response from Linode on this. Hopefully they are fast and honest about it. I've been a happy customer for quite a while, but this is definitely a concern.
They weren't exactly fast and honest the last time a break-in happened. In fact even to this day nobody is quite sure what went down aside from tons of Bitcoins going missing!
They are already way overdue. Their support staff is pretty damn fast, so I'm sure they could have commented on this if they wanted to.
Alas, they have not.
I just blocked my credit card, and Linode will not get anymore of my money. Too bad, i enjoyed my stay very much, but no matter what their response will be, I doubt I will be able to believe them. =/
Not sure how useful that reset was. All I had to do was type in my old password and then choose a new one. No email verification, no reset token, nothing. So if the password was indeed compromised, couldn't the attacker do the same?
Yes, he could. But he didn't. If he did that, the page would tell you that your password is wrong, and you'd contact support. The attacker probably didn't change any password, because it would be futile.
Also, the attaker could also change the account email. Neither tokens nor email confirmation would help.
The biggest problem is for people that used the same password on Linode and any other important service. (And, of course, all the CC stuff...)
Not only that, it also makes me wonder about the free RAM upgrade from almost a week ago. Some people are reporting their Linode credit cards being used for fraudulent purchases as far as a week ago, so this might have been a move to gain some pre-emptive goodwill.
I don't know though... will wait until more details are available but will be keeping an eye on CC statements / VPS alternatives.
Logically, they couldn't have planned it all in such a short period of time. However if they did, from a business perspective it is very good plan. Without the upgrade, today some customers would have had two reasons to leave Linode, now they only have one.
I tend to think the timing is just a coincidence. They also upgraded CPU and bandwidth over the last month and there were rumors of upgraded RAM for a while too.
I believe that to be a coincidence. They have been performing upgrades for a while now. (Increased hard drive space, increased CPU, and finally increased ram.)
I think it was just an unfortunately timed third phase of their upgrade plan.
If that's the same one, then that link implies Linode already fixed the issue on or around the 13th. So, I'm wondering if the silence that has some folks here up in arms is indeed because they've been instructed by law enforcement to keep quiet pending the investigation.
Now if we could only stay the torches and pitchforks for a while before this gets sorted out...
That tweet would suggest to me that someone's credit card was stolen via another method and used to purchase Linode services, rather than the other way around. Ie, I don't think that was related to Linode breach.
Putting aside any opinions on his morals or maturity-level displayed in the chat, it's a fun read. I love how ryan keeps reconnecting after each attempt to kick him out. He must have a pre-verified list of proxies/compromised-boxes he can connect from so he's just burning through them as he gets banned. I found that amusing.
So, they asked us to change the password for reasons they cannot yet tell. The fact that they cannot lay it all out for our curiosity is driving most of people here off the rails. I don't have any problem with letting them verify & deal with the issue properly before releasing any public statement. Anyone who's ever been with them has never received less than perfect customer service. Can we give them a break, wait and hold our judging wands for a moment? It's not like they are staying mum out of spite towards their customers and the press…
I am an ex-customer of Linode and I'm still worried about this incident - Do they still store your card after you've quit the service? This is terrible :(
I've asked them this question. Here is the answer:
Credit card information continues to be stored in our database in an encrypted format, and the decryption key is not stored electronically. We are working on a process on remove the credit card details of past customers on request and can handle this for you soon if you would like. If you have any further questions or concerns please let us know.
As someone who has been a very happy and loyal Linode customer for a long time now, this whole situation paints an image of Linode I otherwise would never have thought. The fact they apparently had both the private and public keys for the credit card hashes in the same location as one another is beyond belief. The very fact that Linode failed to mention they made a deal with the attackers and then reneged on it all without telling anyone makes me sick. I don't want to bash Linode purely because everyone else is, I am legitimately concerned here that my personal details have been compromised.
I thought Linode was different but based on their lack of transparency in this matter, I'm seriously considering just moving all of my sites to DigitalOcean, Rackspace or even AWS instead. This makes me wonder who originally cleared them for PCI compliance in the first place. This is a huge violation of trust and now I've got to keep my eyes focused on my credit card statement for fraudulent transactions, the bank I am with ANZ however has great fraud detection systems and considering I'm in Australia any transaction should be easily reversible, but the fact there is a possibility my card could be fraudulently used saddens me.
Linode needs to come clean about this situation now.
Really off-topic, but still sad: This is a link to slashdot, but it's on HN's frontpage before it's on slashdot's front-page (if it'll ever get there). (And IMHO that's sad, because /. used to be top notch).
I've noticed before that stuff from the HN frontpage appears on /. one to three days after, but I've never seen it for links to slashdot :-)
I rather wish this was a link instead to the original thread rather than to Slashdot. It isn't a big deal, but it certainly would have saved an extra click.
Also off-topic: I've noticed that as well with Slashdot, which is why I lurk HN pretty regularly now. Plus, some of the front page material on /. does more to insight angry discussion, and the community has become increasingly more vitriolic.
At least here, even if someone's brash, they're fairly honest about it (in general). I've even seen a number of disagreements that have been respectful and cordial. It's sad to say, but that's a rare thing these days.
Just got a response from linode:
somethings not adding up?
----------------------
dportalatin
30 minutes ago Hello,
Thanks for getting into contact with us about this. Linode has found no evidence that payment information of any customer was accessed. We have implemented all appropriate measures to provide the maximum amount of protection to our customers. If you have any other concerns we can address, please let us know.
To those of you who have claimed that your CCs have been abused -- I checked mine (which I used to pay for Linode) and it hasn't been used to do anything funny.
(not knowing anything) - wouldn't it be possible to give the cc company a white list of clients that can continue to use it and block all other requests?
The best part is at the very bottom of the log. A customer enters the IRC channel to ask for support after ryann (the hacker) finishes explaining the attack.
Customer: "hello, i forgot my password and linode's email reminder service doesn't work. i checked spam box but there's no email from linode."
Linode Guy: "ryannn: can you give him the password?"
And right now we got confirmed on IRC that data is out, we do not know how much, 4 last numbers of CCs are out but that doesn't mean that full CCs are out.
Last four digits of CC#s are often used to identify them to users ("Visa ending in 1234"), and are specifically OK to store in cleartext. So that's not necessarily a big deal.
<tjfontaine> this is what I'm going to say, as a network representative
<tjfontaine> regardless of what has or has not happened with linode, OFTC cannot tolerate release of sensitive information with itself as that mechanism
<tjfontaine> this channel is moderated until staff determines otherwise
Is this why Linode doubled the RAM? To bribe us and make us stay. I'm pretty pissed off about this and will be exploring other options. I'm not pissed off they got hacked, I'm pissed off they are hiding and not being forth coming about it. A simple, "We fucked up, we are going to take steps 1, 2 3 to fix it and reduce the likely hood of this ever happening again" will make me happy. I understand that any server can be hacked. I'm stunned that they are storing CC details on the servers, there are ways to go about this without storing them if you want recurring billing.
I am not sure that "gospel truth" is a fair characterization.
Anonymous IRC person has provided verifiable details that strongly suggest he or she had access to Linode administrative systems. Fyodor's post to nmap-dev supports the notion that customer nodes were accessed as well.
Linode has provided no details or evidence of anything.
I don't think one has to take that IRC log as gospel truth to be reasonably concerned about the security of their data stored by Linode.
The only "verifiable detail" I saw in the chatlog was the output of `ls` in the http root. And that's only verifiable because you can try to access that weirdly-named HTML file and get a 200 back. Honestly, that doesn't tell me a whole lot.
Everything else, such as the password hashes, don't seem at all verifiable (even if someone were to crack any of the hashes, you can't verify that the password worked at the time of the hack because Linode has presumably changed them all anyway).
Since Linode has proven in the past that they aren't the worst at communication, I can only assume some entity really has them over a barrel, considering the curt and callous responses there.
I just requested a new CC. Can never be too safe. Not sure I'm going to stick with Linode now... sucks because they seemed to be doing so many things right.
This is disappointing and scary. A friend on another forum posted that some guy on IRC told him the last 4 digits of his CC and his e-mail address. I just called my bank and cancelled my current CC and give me a new number. I really liked Linode too :(
FYI, the last 4 and your e-mail address are both visible in plaintext from your /account page in Linode Manager. Obviously, still disappointing and scary, but it doesn't necessarily mean that whoever has that information also has the full CC number.
I echo everything everybody has said. This sucks because we don't know one way or the other. This may be some asshat FUDing away, or this may be a genuine hack.
It's hard to get angry at anybody but Linode needs better auditing around sensitive data so they can tell people one way or the other.
I resigned to the fact that I'd find it easier to change my card details in 30-odd online shops than it is fight my bank to get my money back. Now I can't make any purchases for 7-10 days.
I don't know the skill level of the Linode folks, but could it be that they left a honey-pot CC DB out for this hacker to discover on purpose? Keys sitting right there?
I wonder if they've deleted CC details of previous clients. Is Linode going to contact all relevant customers? Seems like the right thing to do. Not everyone reads HN.
I was just told by a customer service rep at linode that I "shouldn't trust everything I read on the internet" when I inquired about the possibility of deleting my personal information from their server. This seems like an extremely inappropriate way to handle this situation...
By the sounds of it, they probably don't know the full extent of who and what was taken, otherwise they could just email everyone involved and say: "there's been a breach and it affects you" ... or they could contact CC companies.
I stopped using linode in mid February and I have not received an email. They also still appear to have my credit card attached to my account (and I had to change my password when I logged in).
From: "Linode" <support@linode.com>
Date: Sat, 13 Apr 2013 00:11:09 -0000
Precedence: bulk
Return-Path: 6723614.1706014@e2ma.net
Message-ID: <knuab.c9dae.xxx@e2ma.net>
List-Unsubscribe: <http://e2.ma/optout/c9dae/xxx>
X-Test-Mailing: no
Dear Linode customer,
Linode administrators have discovered and blocked suspicious activity on th=
e Linode network.=C2=A0 This activity appears to have been a coordinated at=
tempt to access the account of one of our customers.=C2=A0 This customer is=
aware of this activity and we have determined its extent and impact.=C2=A0=
We have found no evidence that any Linode data of any other customer was a=
ccessed.=C2=A0 In addition, we have found no evidence that payment informat=
ion of any customer was accessed.
We have been advised that law enforcement officials are aware of the intrus=
ion into this customer=E2=80=99s systems. We have implemented all appropria=
te measures to provide the maximum amount of protection to our customers. O=
ut of an abundance of caution, however, we have decided to implement a Lino=
de Manager password reset. In so doing, we have immediately expired all cur=
rent passwords. You will be prompted to create a new password the next time=
that you log into the Linode Manager. We also recommend changing your LISH=
passwords and, if applicable, regenerating your API key.
The following represent best practices in creating new passwords:
-- Avoid using simple passwords based on dictionary words
-- Never use the same password on multiple sites or services
-- Never click on 'reset password' requests in unsolicited emails - instead=
go directly to the service
We apologize for the inconvenience. If you have any questions, please do no=
t hesitate to contact our support team at support@linode.com.
I'm most concerned about Lish access. I'm not liable for fraudulent charges to my CC, so all I'll suffer is inconvenience. I'm most concerned about access to my VMs, which can be a business terminating event if things go extremely badly.
No suspicious charges on my end, yet, but I did just find out that my bank (USAA) is finally offering cards with Chip-and-Pin, which causes the card number to get re-provisioned. No time like the present, eh?
I recently sent an email to linode support, and got a very murky response. I used my debit card on linode, and it was recently used on transactions I didn't make in random parts of the world that I'm not in, so I had to cancel it. My first guess was that it was linode, and all of the posts here make it more likely.
Essentially: I am a linode customer. My cc details were somehow leaked. Adds a data point here.
the card with my linode account has a minimal amount total allowed to charge. I'll go through my cc usage, but at worst they could've gotten a couple of ebooks. as a long time customer though, Im conflicted and motivated to move off linode to another provider.
Doesn't look like it. If you believe the IRC logs, the attacker broke in using a "0day" exploit of ColdFusion, which is being used on the site. Not because of a flaw in RAM upgrades. Although I believe that Linode beginning to offer free trial accounts is probably responsible for the increase in attacks lately...
I dropped linode two months ago. Hope they have purged my cc information from their system. Well I logged in just now with old credentials. Cc information is still there, not so good...
Sad day to see that Linode is not immune to data breaches. That being said, I don't necessarily support the witch hunt that seems to have taken place on its reputation.
Being a former customer I decided to contact Linode support. I wanted to know if we have anything to worry about as we used to have a credit card on file with them. Notice how they say "Decrypted" which to me is not the same as NOT COMPROMISED.
Here is their response:
Thank you for reaching out to us. We do archive customer credit card details. At this point there is no evidence that customer credit cards have been decrypted.
I've been living comfortably on Linode servers for over three years. This is like suddenly being evicted and having to pack my stuff up and find another apartment.
I have to wait for some sort of verification for this but if true then I have to leave Linode. I have client sites hosted here - not for cost reasons, just because I like Linode.
For the sake of $5 a month I can't even take the slightest risk of being criticised for using Linode. And this lack of transparency could be a nail in the coffin here.
I don't want to waste a couple of days on this but that's what's going to be involved if this is true.