1) choosing that particular eMMC, likely not something specced for very high write endurance, or not speccing a larger amount of flash to spread the write endurance out over a period of up to 20, 25 years.
2) linux/operating system/software engineering implementation, for a certain volume of MB of writes per day, that would be known to wear out the flash write endurance over a short number of years. possibly a result of the software team not communicating properly with the hardware team. the software team very well may have been operating in a vacuum of assuming that they could do whatever they wanted within the CPU, I/O, RAM, and disk space limits of the architecture, and weren't even thinking at all of the consequences of their logging setup.
3) not making that a socketed/removable part.
4) asking the owners to pay for the replacement due to tesla's own screwup.
Always thought of this as a fascinating race to watch. What will happen first: (1) Tesla figuring out how to really build & manufacture cars or (2) the legacy automotive companies really figuring out build & deploy software?
(It's not that black and white, all automotive companies have varying degrees of expertise in both software engineering and mechanical engineering. But it's certainly interesting to watch the industries learn from each other.)
I can offer a somewhat unusual perspective on Tesla and SpaceX, from the manufacturing POV, because I never did sign their production NDAs, although I have made parts for them.
Producing for Musk companies is a fucking shit show. Specifications change mid-production on a constant basis, to a ridiculously comical extent. I honestly can’t imagine what the hell is wrong with these people that they change hole locations and part geometry for every single order, knowing full well that 1000 of 1000 units have already been produced and boxed and loaded onto pallets.
As an Elon Musk fan, it was rude surprise that their shit is so fucked.
I tried holding off on production until the last possible nanosecond, but no. No good. Changes happened the day orders are due.
The only Musk work I do at this point is emergency corrections for other local outfits who made parts with rapidly evolving and mutating requirements.
I thought that the designers would settle down as Tesla and SpaceX get through production hell, but they seem committed to mass production of no-two-things-ever-the-same.
I drive a Model S and I figure probably most things people make and do are objectively retarded, I just happen to know more than I want about this particular retardation.
Software/design issues requiring hardware recalls are routine in the industry, though. There's really nothing notable about this particular issue at all, except that it involves "consumer-like" devices that posters here understand and feel expert on. We all think that "we'd never have done something this dumb", so therefore it's an easy shot to take where esoterica about engine control PCBs isn't something we have the confidence to second guess.
(Needless to say, we totally would have made the same kind of mistake, and probably do so every day.)
The only meaningful ding on Tesla here is #4 above: clearly this was a design flaw and should have been a recall from the start, not a customer-financed repair.
>There's really nothing notable about this particular issue at all, except that it involves "consumer-like" devices that posters here understand and feel expert on
I think you're off base here; it's the very fact of Tesla using "consumer grade" hardware that is the (notable) problem. The people who "would have made the same mistake" are qualified to pass judgment if they know they are not qualified to develop automotive grade systems.
I don't think that holds. In fact while longer-wear flash devices do exist in the market, the design mistake here would have driven any feasible part to failure. While you can argue that choosing consumer parts was "a" mistake, it wasn't the one that caused this failure, which was a software/integration thing.
I think the biggest mistake here is the drivetrain depending on a device which is known to fail (eMMC) even without the logging problem (why not using a separate device for the logging?) it would eventually fail, other manufacturers have similar problems, (Mazda, Ford, Mercedes) but you just lose the audio system.
> drivetrain depending on a device which is known to fail
Uh... the drive train itself is "known to fail". Everything wears out. Everything breaks. You just aren't surprised when your transmission needs rebuilding or your brakes or shocks need replacement because those are "car things" you've been conditioned are normal. Well, electromigration is just another kind of normal wear and tear to be designed into the device.
And it's the design here that was at fault, having incorrectly planned for the replacement rate. But there's absolutely nothing weird about building cars out of parts that fail over time!
I certainly have shipped dodgy stuff knowingly for lack of time to do better. "This is a prototype, we'll apply the Chinese warranty, if it's broken we'll replace the unit". I would ship non-UV cables to outdoors stuff, because I had no time to select and order a better one, I shipped an outdoor temperature sensor cover that was not open to the air enough and was just tracking the sun. The only red lines were thunder and fire. I don't want to start a fire, and I don't want to conduct the thunder.
But I did that way earlier than Tesla in the process.
Yeah, but it's not a single person or single decision. They went with consumer eMMC, and probably had a justification showing that they'd never need replacement for 30 years of use or whatever. But then someone else overfilled the partition, leaving not enough space for the hardware wear leveling to work in, which over-wears the available space. (No idea what the filesystem choice was, some are more susceptible to this than others). And then a third party decided to ship with the logging settings set to dump all sorts of junk to storage that wasn't needed on a production device.
None of those are particularly boneheaded in isolation. It's just a bug. Bugs just happen.
(But again: design bugs should be recalls and not repairs)
> Chinese warranty, if it's broken we'll replace the unit"
That's the mass production warranty for integrated objects.
Calling it "Chinese" is just weird unless you are calling attention to a claim that only china does mass production anymore.
For cars, we have the "extended warranty" which is just a bet that the car won't fail to much on average.
> Always thought of this as a fascinating race to watch. What will happen first: (1) Tesla figuring out how to really build & manufacture cars or (2) the legacy automotive companies really figuring out build & deploy software?
Yes! And (3) is potentially: A company who's even better at building and deploying software and hardware enters the market.
The Chevy Spark EV was supposedly nearly as cheap after rebates, I think. But I believe it was discontinued, and I assume that means it probably was being produced at a loss, so it doesn't really count.
I believe that is before subsidies and including VAT, so it will be more like € 11 000 after subsidies. The point still stands: there are many other manufacturers with long experience in making cheap cars.
I see a lot of speculation that apple will produce a car.
But apple at the moment just designs hardware, doesn't really make it (as in they don't have their own factories), so presumably they would either need to build factory for their cars or hire some other factory(or factories) to build cars for them ?
As far as I understand most cars sold in USA are build in USA (I am not sure if that's only because of shipping costs or if there are any import taxes or something).
Even if most parts were from China finally assembly would probably have to be done in USA, and factory quite big and expensive, and have lots of trained people (to have apple standards).
Given that its apple that we are talking about car would almost certainly be an instant hit, so they would need to produce a lot of them, which again means big factory with lots of people.
How likely is it that everybody missed such factory ? Especially with press so obsessed with apple. Even if some other company like Toyota were building it for them, they would likely need a lot more additional capacity than they have now ?
And we have seen with Tesla, that scaling out is really the big problem here.
I genuinely interested how likely is it that apple cars will be on streets anytime soon.
There was speculation that they were partnering with Kia/Hyundai to do the manufacturing. This was big news a week ago, but Kia/Hyundai have backpedaled since.
Every established company I've worked for seems to have as many processes that get in the way as those that help.
also - nobody mentioned the amount of data collected. Is it good diagnostics or is it hoarding?
I remember the story of the guy who tried to return his tesla under the lemon law thing because of some weird failure. Tesla diagnosed it and found he would pop the trunk before every failure happened, and they used that to prove he was doing it. After that, I can see an engineering push to log more, not less.
Gm has this covered I believe. I think they have a less totalizijg view of over the air undates than Tesla. Tesla uses it as a marketing tool to attract “tech savvy” consumers. Gm has good tech look at Onstar.
In Model3 you can move front passenger seat forward and up (to clean up in the back) all the way to the point when it pushes on sun visor and cracks the mirror in it. Just by using handles intended to move the seat, no manual pressure or anything, there are no limitation on how far it goes and it's strong enough..
Tesla service centers blame owners for that and charge full replacement cost for the visor :\
One thing people don't understand about cost saving is that it almost always looks good in the short term. You save a lot of money by using commercial grade parts (like eMMC, the dashboard screen, etc.), or by hiring SW and HW engineers who have never worked in the automotive or embedded environment. It's sacrificing quality for quantity and price. But it's usually the customers who pay for it all, one way or another.
From Tesla's perspective the bet might have payed off, by the time this becomes an actual problem for them, they're already over the hump and weathered the worst of times, and can afford to spec them better (one would hope they learned something from this although I have my doubts). In reality probably the real reason this worked for Tesla is Musk's ability to build a fanatical fanbase that can drown out any other voice.
Funny thing is while I was okay with buying Tesla, it was Musk's personality attached to it that really bugged me.. So while there is a (quite loud) fanbase, there is also a group of potential Tesla owners who may go other route just due to desire to not be associated with "meme king"
The idea was that there is a critical mass of people which are vocal enough to help Musk push Tesla's public image past any blunder. No other car manufacturer has this kind of clout.
Could be but the clout I'm talking about is how easily they're let off the hook after repeated blunders. I personally doubt Toyota could get away with Tesla's QA for a few years. Hard to judge in practice though.
Tesla learned from Toyota's playbook! At least they were settling their "unintended acceleration" issue as Tesla were getting started: Deny and minimise systemic faults, recall quietly if ever, and blame it on the user wherever possible. I think they paid billion dollar fine in the US?
I had a 2002 Celica that tried to kill me (gas pedal that got stuck under the floor mat at 90... 95... 100... on the M62) but I loved it too much to hold a grudge.
I had a Honda years ago that had a gas pedal that hinged at the floor. And when the Toyota gas pedal thing was in the news, I was like "ha, they cheaped out" and was glad I had a Honda. Recently, I believe Honda has switched to the pedal hinging from above though.
But I have a hard time taking many complaints seriously. If you have a 600 hp sports car, sure, a stuck gas pedal could kill you before you know what happened. But a 2002 Celica at 90 (mph or kph) - you expect me to believe that the brakes can't overcome the engine, and you can't shift to neutral before it goes into orbit?
What is "the issue"? Back when Toyota was in the news, there were multiple alleged issues.
I commented on the issue of loose floor mats, on an underpowered car, on a high speed road. Even if it is potentially fatal, I don't see what's specific to Toyota.
Toyota has fixed the frame on the Tacoma and pinky promised the new ones won't rust out every year since 1990ish and recalled something every year since 2000ish. People find all sorts of ways to hand-wave it away.
And then there was that decade where they made trucks and SUVs with ball joints that would pop out with levels of wear that would be inconsequential had they designed them conventionally.
I guess the above are engineering examples. Their QA is definitely spot on.
Don't leave out the scale. Toyota sells almost 10 million vehicles per year across some dozens of models. Tacoma makes about 2% of that. Any manufacturer might have a few problematic models in their lineup even if mostly specific model years. But when it's across the board you have a problem. It's not just one factory, one model, one year, one batch. It's not an exception, it's the rule. You can twist and turn it until the situation looks the same between the 2 brands but if we're being honest, it's not.
If anything your readiness to excuse Toyota's faults is proving my point.
They've had 40yr to fix the problem. Still not fixed. We'll have to wait a few years to see if they fixed it this year.
Yes the Tacoma is small compared to their global presence.
The Focus is small compared to Ford's global presence and they still got a ton of crap for that transmission. Nissan gets crap for their CVTs that aren't really that bad anymore. Subaru gets crap about head gasket and transmission issues they mostly cleaned up a decade ago.
My point is that Tesla is at say a 5 and the fanboys pump up their reputation so that they look like a 6. Toyota is at say a 7 and the blind fanboys pump it up to an 8. It's not about who's better. My point is that this blind fanboyism that guards a reputation in the face of mis-steps isn't something unique to Tesla or the cult of Musk.
If anything the common factor is upper middle class allure. Nothing rounds out a driveway in a nice school district like a Model 3 paired with a Toyota pickup or SUV. Though I have no idea whether it's the cause or the effect or maybe a feedback loop.
> If anything your readiness to excuse Toyota's faults is proving my point.
I'm not defending Toyota, I'm only attacking your argument, which I hope is fair game. The problem with it is that it fails to clear the bar for basically any major manufacturer out there. Put Renault or Volkswagen in there and the situation stays the same. Everyone has a few problematic models or years that get some (un)warranted criticism but you really have to question a manufacturer's QA when the problems are across the board, as evidenced by Tesla using low grade parts across the board. That's one thing what sets Tesla apart from the competition: they cut costs and corners anywhere they can to ship more units, because that's what drives share prices up.
And you'll never see Renault's CEO claim that their cars will cure cancer starting next year and a cult following of people hyping and believing it, or vitriolically pouncing on any unsuspecting commenter or article criticizing Renault. That's the second thing that sets Tesla apart from their competition: they can afford to consistently ship compromised products because their image can absorb anything.
On any of the rash of EV sites that popped up over the past few years just linking to an NHTSA or IIHS report contradicting the article's premise could get you an instant ban. So the articles stay there with the misleading claims because that's what comes up in an internet search.
I think it's kind of like the "risk homeostasis" thing people talk about. If you have really good engineers, then you don't just let them design the greatest vehicle possible, you ask them to cut costs from that design. And because they're really good, they can make things work that others couldn't, and so they get pushed harder, until something goes wrong on a large scale, because there were tradeoffs that couldn't be eliminated.
The blind trust in apples ability to build a car is not well reasoned as it appears.
If the story from Apple was true, I.e., they actually designed almost all parts of the phone, and give out instructions to implement, then they'll need to be able to design a car as if they were build them, to match iPhone's legendary quality.
Can apple do that? With enough time, sure. But can they catch up Tesla in a meaningful time frame, say 5 years? I doubt it.
The other factor to consider is Tesla is always on the ropes, so they will do whatever to survive. I wouldn’t be surprised if the eMMC was selected based on what oem would give them credit.
It’s a shady company that skirts or ignores the law, that has to live in the edge to survive.
I've owned a Model S since before the Model 3 arrived and have always fretted that the level of service I received wasn't sustainable at scale.
They AGGRESSIVELY attacked any problems and replaced parts, even when that's not what I asked for.
It's kinda nuts. Just the things I can remember:
- All 4 door handles were replaced with newer versions
- I complained about a corner of stitching on the driver's seat and got an entirely new driver's seat
- Glovebox cover replaced (to address latch)
- Steering wheel controls replaced
- Driver display unit replaced entirely
- Sunroof replaced
Granted, that's A LOT of parts to address, but their attitude was we'll do anything to make it right. Never once pushed back or questioned. Usually told me that they wanted to replace the part (rather than fix) because Tesla had engineered an improved version. And, to their credit, the new parts they put in seemed to have meaningful improvements and have worked really well.
As a 2013 Model S owner and now a 2018 and 2021 Model 3 owner (Tesla only family), I agree 100%. I assumed there were a few reasons for the Model S service I received:
1. Small scale
2. Desire to keep early adopters happy
3. 2x the cost of the Model 3
However, my Model 3 service has been really poor.
In my new 2021 the charge port bolt was loose and wouldn't charge. Known issue that took several days and many phone calls to get fixed while traveling. Eventually a service manager got involved and the fix took 5 minutes.
Then they installed Homelink and forgot to plug the front sensors back in. Full self-driving was unavailable for 3 weeks until they returned to plug in the sensors.
Also ordered snow tires from Tesla and the TPMS sensors don't work in my car (2021 uses BLE sensors). They're ordering new sensors and I will need to take my wheels to a tire store to have them installed.
It's been less than 2 months since I took delivery.
Love the car, love the company, an enthusiastic shareholder, but service needs to be fixed.
Of course if they made a Tesla with fewer issues and better service, I'd buy it. My 2018 Model 3 has had almost no issues. No other car company is making anything close so I'll put up with the service for now.
I considered a Ford plug-in hybrid and read on the internet how people had fires start because of a flaw with the charging cord, which was probably due to an extremely small cost-cutting choice.
Actually it wasn't too expensive - something like half of that (I don't remember exact number, but mid-100s).
It was more of a feeling that I shouldn't be paying for it at all that displeased me :( (and reddit is full of stories when it happened with others (with the same service centers' experience))
One more thing that has no solution now but I'm sure Tesla will not fix it at all or will charge a lot for it:
White interior and black seatbelts. Despite Musks's promises how durable white interior is, seatbelts they use are dyed with I don't know what and they mar the seats. It's permanently baked in and cannot be removed.. Tesla responded long time ago that they are looking for solution, but whatever they will change will definitely not apply to any existing (screwed-up) white interior owners.
This is exactly my experience with the jeep my family had. However, there's a big caveat: the jeep engine itself was perfect. Over the 15 or so years we had it, that engine was solid. Everything else in the jeep failed at some point or another, and had nothing to do with driving. I have a few examples off the top of my head: The heated seats died after a few months. The insides of the car doors became detached because the glue wasn't strong enough. One by one the dashboard knobs all split apart at one point or another. We were a normal family, there were no crazy kids or pets in it that went nuts on the inside. Our jeep just kind of disintegrated around us, and none of the problems had to do with actually driving the jeep. So yes, we didn't even need to drive it for it to fall apart.
My Jeep Wrangler does not have any physical interference issues regarding seats breaking mirrors and the only issue I have had is the rear main seal started leaking and they happily replaced under warranty at 4 years and 6 months no questions asked.
> 4) asking the owners to pay for the replacement due to tesla's own screwup.
I had an email from Tesla yesterday about this recall and as far as I can see I will not be expected to pay anything. It even says that if someone had the part replaced at their own cost that they might be able to get a refund. I live in Norway and have a 2015 S 70D.
The email says:
"Tesla har besluttet å frivillig og proaktivt tilbakekalle enkelte Model S og Model X som er bygget før mars 2018 og som er utstyrt med et 8 GB innebygd MultiMediaCard (eMMC) i MCU som kan oppleve en funksjonsfeil på grunn av akkumulert slitasje. ... Hvis du allerede har betalt for reparasjoner som er tilknyttet til denne tilbakekallingen, kan du være kvalifisert for en refusjon."
Google Translate does a good job on it:
"Tesla has decided to voluntarily and proactively recall some Model S and Model X that were built before March 2018 and that are equipped with an 8 GB built-in MultiMediaCard (eMMC) in the MCU that may experience a malfunction due to accumulated wear. ... If you have already paid for repairs associated with this recall, you may be eligible for a refund."
Hackaday's 'article' seems to be conflating what Tesla wanted to happen (and what they actually did before) with what is happening now.
Or is Tesla in the US expecting owners to pay now as well as before?
I mean, on your first point, it's an automotive grade part. I think any eMMC put under these conditions (75% full and the other 25% written heavily to with log files) is going to have a relatively short life expectancy.
What baffles me is that who is reading these logs? Can't they just turn then off (pipe to /dev/null) and then have cars that need troubleshooting put into a debug mode at some point?
I think even stock Ubuntu it's not that hard (famous last words) to disable/redirect logs to /dev/null. It would take a competent engineer probably 1 day to configure it, even if they need to read Stack Overflow. They either didn't have the time, or they didn't even know it would be a problem.
> (75% full and the other 25% written heavily to with log files)
Does an eMMC even know how full it is? If it were designed for write once it could certainly track that, but I doubt they are designed for such little usage. After having every block written once they would need TRIM to know what is in use, but to my knowledge eMMCs don't support that. (Not an expert in the area, just a random software developer.)
So what, automotive developers never make silly bugs?
Case in point: Audi Concert/Chorus radios used in early 2000s. Turning volume knob worked like that: after every knob turn save updated value to EEPROM, then immediately read EEPROM and send that value to audio chip.
Inevitably, after many years EEPROM cells storing volume died, causing that turning volume knob just set volume to totally random value. It was impossible to repair, as EEPROM wasn't separate chip but was integrated in MCU. And you couldn't easily replace MCU because it contained mask ROM code. The hack that was sometimes done was: add custom microcontroller, connect it to the knob, splice I²C wires to audio chip, and patching volume set command on the fly...
The sad truth is you might be right. At a former company we had wear leveling in the firmware of IoT devices that would cost under 20$/piece so the fact that a car lacks such a basic feature, is shocking and tells me all I need to know about the development practices at Tesla.
As a firmware engineer with automotive background, having now seen how the "sausage" is made there makes me stay away from them for the safety of my own life.
I know the rest of he car manufacturers aren't as innovative or as desirable but at least some I worked for have their firmware development quality processes down to a religion, and, as boring as that may be, it's the kind of mentality you want from something you put your family into every day.
It did have wear leveling. They were just writing such an ungodly amount to it (more than the 8GB capacity) per day it defeated the purpose of wear leveling.
> choosing that particular eMMC, likely not something specced for very high write endurance, or not speccing a larger amount of flash to spread the write endurance out over a period of up to 20, 25 years.
The problem is that there isn't any wear leveling happening at all, so if you have an event that happens at every boot up and is logged, it wears the same block every time.
> not making that a socketed/removable part.
It's a very integrated module, I don't think socketed ICs have been a thing in automtive for a long time.
edit: I'm wrong, daughter post has a nice run down
If you're curious the very long forum thread is a lovely read.[1] You can see the chip levels itself, because just soldering in a bigger chip with the same code on it makes a big difference to many customers.
> The problem is that there isn't any wear leveling happening at all, so if you have an event that happens at every boot up and is logged, it wears the same block every time.
This isn't true. Very nearly all eMMC devices have hardware wear leveling. They aren't like MTD (raw flash) devices or very cheap USB/SD where you have to rely on a block remapping layer or special filesystem because there is no hardware wear leveling. But like any flash device, when they are completely full there are fewer free blocks for the wear leveling to operate. Tesla also may have had them grossly misconfigured, but I haven't seen any evidence that it was anything other than good-old-fashioned log churn on a very full filesystem. I also understand Google kind of had them over a barrel with policies related to the caching of map data and aerial imagery too: since they could not retain this for very long, caches were constantly invalidated and the same data was being redownloaded over and over. At this point, I don't even think they cache map imagery to storage anymore.
Regardless of the technical details, it's clear that Tesla gave little consideration to this problem during design and did even further damage by wasting their small remaining space by filling it up with Elon's glorious vision of fart sounds, fireplace videos, and games -- Which are great fun, not gonna lie, but in hindsight a terrible decision that greatly exacerbated this problem.
I'm not a professional embedded systems person by any stretch, but I was watching out for this problem even in my hobby projects back in the 90's which used early flash memory products like DiskOnChip modules. To this day, I still buy SSD's that are a minimum of twice as large as I think I will need for every application? For things like all-flash SAN I double it yet again. Anyone who would like to understand why can look at my fat stacks of dead Intel SSDs that all died basically because they filled up. My point is that this sort of thing is really well known and there must have been hundreds of times people within Tesla had questioned this internally.
I didn't mean literally socketed, but in 2012 things like msata interface cards (a predecessor of m.2) absolutely were a thing. Or implemented soldered on a custom small pcb to edge pin interface not very different in size and slot from a laptop sodimm.
1) choosing that particular eMMC, likely not something specced for very high write endurance, or not speccing a larger amount of flash to spread the write endurance out over a period of up to 20, 25 years.
2) linux/operating system/software engineering implementation, for a certain volume of MB of writes per day, that would be known to wear out the flash write endurance over a short number of years. possibly a result of the software team not communicating properly with the hardware team. the software team very well may have been operating in a vacuum of assuming that they could do whatever they wanted within the CPU, I/O, RAM, and disk space limits of the architecture, and weren't even thinking at all of the consequences of their logging setup.
3) not making that a socketed/removable part.
4) asking the owners to pay for the replacement due to tesla's own screwup.