This is really cool, thanks for sharing. What's wild to me is that the program started in the late 90s and only now is the F35 fleet up to originally specified? operational capacity.
Since then I graduated high school, got a degree, got married etc etc. The time span is mind boggling. Would be interesting to see how continuity is maintained for so long. In software it feels like if a project is more than 6 months old, we throw it out and rewrite it.
> In software it feels like if a project is more than 6 months old, we throw it out and rewrite it.
I think that would be a bad way to operate, but what's worse is what we _actually_ do, which is write the project like it's gonna be replaced in 6 months and instead keep that poorly-documented untested duct-tape contraption around for a decade as the central load-bearing component of critical infrastructure.
The F-35 contract was awarded on October 26, 2001. I was in my freshman year of undergrad, 18 years old.
I started on the program in August, 2010. I was 26 years old.
The program has just completed its Initial Operational Test & Evaluation, including its runs for score in the Joint Simulation Environment. I am 40 years old.
> In software it feels like if a project is more than 6 months old, we throw it out and rewrite it.
“The Phoenix pay system is a payroll processing system for Canadian federal government employees, provided by IBM in June 2011 using PeopleSoft software, and run by Public Services and Procurement Canada… By July 2018, Phoenix has caused pay problems to close to 80 percent of the federal government's 290,000 public servants through underpayments, over-payments, and non-payments.“
That situation was (is?) absolutely mind-boggling. I personally know government employees that were being underpaid with no recourse for months on end because the software wasn't working and the government apparently had no alternate way to pay. Some people weren't getting paid at all. And as you quoted, it affected 80% of the workforce, hundreds of thousands of people.
Ironically, many of the agile development practices which are widely used today were pioneered in the Chrysler Comprehensive Compensation (C3) payroll application. It was never able to produce an accurate payroll for Chrysler and couldn't replace the legacy system, although the project was considered at least a partial success in other ways.
My employer still had the balls to pay IBM after the Phoenix snafu to come in and tell us how we could become more efficient and guess what? It was basically word for word what the frontline employees had been screaming for for YEARS that management ignored. And guess what? They ignore the consulting report and did nothing anyways. #MunicipalLife
You write shit down and you have career engineers that enforce continuity
It's trendy in software to complain about doing annoying work like writing reports and documenting things. But most hard tasks require writing reports and documenting things.
And this isn't limited to aerospace. My wife has spent a career in pharma (drug save & pharmacovigilance specifically) and it's the same way there. People complain about rigidity and sluggishness in these industries but there absolutely is an ingrained attitude of documentation and process compliance that pervades. At one point -- and this was just last year -- my wife took over running a monthly safety report that involves manipulating a bunch of data in Excel. Even that has a 9 page instruction guide, and since she now owns the output she also owns maintaining the manual.
Too often in the land of software we underestimate the potential negative impact the traditional "move fast and break things" approach to product development can have when it comes to real world use in mission critical systems.
On the other side this unwillingness and mental non-acceptance of those reports/manuals/etc. as a wasteful activity frequently comes from the understanding that there are more efficient ways of doing things, and that drives the "software eating the world" effect. While I naturally don't know the details of the case you mention and pharma is far from the domains I've been in, yet in many business/enterprise situations the software approach is to code the many-page guide into business logic, including ETL-ing the data instead of manual import, etc.
Move fast and break things brings you to the Moon in a decade using primitive tech, where is total process compliance can't do that even in 50 years using much more advanced tech.
So, an amusing anecdote related to your second paragraph - one reason it's taking so long the second time around is everything has to be repeated. They lost the knowledge of how to make rocket stages and engines of that size, and had to re-learn those lessons.
It's also quite important to remember how many lives were lost (or nearly lost) because of "breaking things" in the Apollo program. Something that's not nearly as acceptable today than it was at the height of the cold war. Something that directly implies moving more slowly and being more sure that everything works the first time, every time.
Seconded. People burned alive until we learned. Surely there is a middle ground that will let us speed up while staying fairly safe, but it's important to remember that outside of software, many rules are written in blood.
I don’t hold a strong opinion either way - in terms of process and documentation versus freestyling it - but that fire was predicted, and I think the concerns were documented.
It can and did happen again, twice, on the shuttle project. Both the O rings and the ice damage were documented.
Ultimately, any process (or lack of process) can be subverted by a bad culture. And unreasonably excessive process - as perceived by the participants - can damage culture as much as not enough.
The problem is that culture is ineffable, so we try to nail it to the ground with whatever we can think of.
a lot of people died in germany, the ussr, and the us making those rockets work. and in exchange for that we planted a flag there and have a handful of rocks in a glass viewing box.
move fast and break things worked real well for the folks who got literally creamed while they were viewing the titanic.
Heck, even maintaining my computers at home requires documenting things! I have lost count of the number of hours I’ve lost trying to rediscover how or why I set things up the way I did.
I work on software that has a lifetime once installed of about 30 years, and if a safety critical error is found during that time, ideally it needs to fixed with a minimal patch, so we have to maintain the capability to do so.
I guess the ethos is quite different to top tech company. We don't get the pay or perks that you would get in Silicon Valley, but we are unionised, and it's a viable option to spend your entire career just on one project so it's very stable.
Partly it depends on documentation, but also on thinking long term. There are certain people who are the technical authority for a particular area, and they know that about 5 years before they retire or move on they need to find someone who can take on their role for at least the next decade, to keep their knowledge rolling forward.
Not OP, but I work on medical devices. One product at my last job had an expected service life of 20 years. FDA requires that the manufacturer maintain the ability to support and service a medical device for, IIRC, 5 years after market exit.
In the 10 years after release that I was on that project, we went through multiple OS upgrades from Windows NT to XP Embedded, to Windows Embedded Industry (replacement for XP Embedded) and a number of replacement x86 CPU boards had to be qualified as one manufacturer after another exited the market. Since the device is validated as a complete system, we often had to buy a year or two stockpile of existing product to give us time to start the Validation process for replacement hardware.
You usually have plenty of warning from a supplier that a product (Windows or a CPU board) is going EOL at a certain point, so you need to start validating whatever the next replacement will be well ahead of time.
Was there ever any consideration given to building a "testing harness" to physically simulate the F35 landing? Something like the "dead load" testing that the EMALS undergoes. Just in reverse. Anyway, that was great read.
There was a lot of static load testing done, and things like a drop test [0] of a full scale article. But to my knowledge, the only way to test the dynamics of a carrier arrestment is to actually do an arrestment. We do them on land; NAS Patuxent River and NAS Lakehurst (among others) have a full set of Mark 7 arresting gear like you would find on a Nimitz class. Lakehurst also has the advanced arresting gear present on the Ford class.
How much of a difference is there between dry land arresting and carrier arresting? I would guess some since the carrier represents a somewhat dynamic surface, and flight conditions might likewise vary. Is there enough that a second round of carrier based testing is required that might trigger significant changes?
All of this was done as a work up to a carrier deployment. In software terms, trying the arrestments on land is deploying to test, doing them on a carrier is production. There were three separate developmental test deployments to carriers for the F-35C. Each deployment sought to expand the understood envelope and and handling procedures. The hook redesign happened before the first deployment. The hard landing story in the post happened during the work up to the third and final deployment.
Had the hard landing occurred on a carrier, how would the gentle, flared landing be accomplished? Try to reach an airport on land, ditch the plane in the water, maybe catch the plane with the net on the deck? Our just do a normal carrier landing and hope for the best?
The Navy developmental test community does carrier suitability testing of every new airframe, and there's a whole program of nominal and off-nominal arrestments they have to test in order to prove the jet can recover in all expected scenarios.
The model provided by -the manufacturer- correction NAVAIR (thanks OP!), stated that the cable will bounce up after having been hit by the landing gear. Thus the hook design made sense. The cable jumps up and over the hook. Plane arrested.
Instead, again as the article states, the cable is actually being pressed tightly against the flight deck and the elevated hook nose makes the entire hook get thrown up in the air when drawn over the tight cable, back towards the plane and would even destroy some parts of the monitoring mechanisms, so violently did that happen.
They also provide the new design, which is basically the old design and that is also why the techs that saw the new hook for the very first time (and know about the cable I presume) instantly said "That ain't gonna work!".
Minor correction, the wire dynamics model was provided by Naval Air Systems Command (NAVAIR), the Navy engineering organization in control of research, development, test, evaluation, and sustainment of Navy aircraft.
I would actually love to know if someone on the hook design team questioned the model. I guess we won't know but I also it doesn't hurt to ask.
Like did someone go: odd, why would that cable go up and not tighten when waves are sent through it towards the outward attachments? But was inevitably shut down and didn't have "access to the customer" to ask/verify.
Like one of the first things to ask for when having to design this that comes to my mind is: I want high speed camera footage of current arrestor in action at the customer site!
Even if two different aircraft have the same space constraints for the hook (which is a pretty big if), they have different mass and deceleration characteristics (i.e. minimum and maximum approach velocity) during landing- changing the force exerted on the hook. Designing a lighter hook for the lower loaded aircraft is VERY desirable for high tech fighter jets- every ounce saved is better range, better agility, etc.
As far as the little lip at the very tip of the hook- it looks to me like the initial design was trying to minimize any risk of digging into the flight deck and causing damage- this is just a guess though.
“After the LSO finished what he had to say and left the ready room my B/N allowed that he might fly with me again. Me, I was still shaking inside.
The next morning I went up on the flight deck before flight ops started and walked to the aft edge of the deck. I was looking for something and found it.
About one foot from the end, there was a single, shiny, brand new, solitary hook imprint in the deck.”
Due to the planes and to the rest of the tailhook (the shank, etc.), they could hit at different angles, speeds, etc. That's just a guess, however.
Each plane costs ~$100 million and the entire program will cost over $1 trillion when it's done. Performance needs are extreme: They need to land in all sorts of adverse, imperfect conditions - damage to the plane, the carrier, the wire, the personnel; bad weather; bullets and missiles flying around. It seems worthwhile to design the highest-performing tailhook for this plane, rather than to save a few bucks.
Also, IME people doing something this sophisticated don't miss those really simple, obvious issues that we happen to be able to observe and grasp from the outside.
They designed for the F-35B as the "baseline" with carrier requirements secondary. Also, the engineers knew but, "their concerns would have just as likely been ignored." This reference was 2012, when they knew it was a problem but before OP was fixing it.
A more interesting question is do the cables (size, positioning, tension) vary by aircraft? Can any carrier-capable aircraft land on any carrier in the US fleet?
I actually still have your personal email address. Expect something from me soon. It took a few years but I work as a developer now (C++, electronic warfare simulator)
First of all, thank you for the super interesting read!
Now, as a Ukrainian I do have a philosophical question of sorts. What we have seen here in a real full-scale combat is that some of the modern machines are way too delicate for actual operations on the ground. For example, I have heard some feedback about the Abrams tank: way too finicky for real use, not durable, not reliable. The same goes about many other western items. (Some hardware demonstrated exceptional reliability, like Bradleys and HIMARS)
My question is about modern fighter jets like F-35.
Does that level of engineering and the amount of delicate electronics somewhat limit the durability and reliability of the airplane compared to much simpler designs?
The F-35 was never really intended to be durable or reliable for ongoing use in long wars of attrition. It was designed to be survivable on penetrating strike missions against near-peer adversaries. Essentially to "kick the door in" and destroy high value targets such as air-defense systems during the first few days of a conflict. War games and simulations have shown that simpler designs can't accomplish that mission anymore so concerns over F-35 durability and reliability are somewhat misplaced (although there is certainly room to improve mission readiness rates).
No one is even remotely contemplating sending F-35s to Ukraine. Besides and costs and security risks, the Ukrainians unfortunately have nowhere near the infrastructure and logistics to sustain such a complex platform.
Ah, so in that case it looks like reliability and durability are not exactly important or desired features for that kind of missions. Did I get it right?
Yes, there are pendants that are supposed to keep the wire above the deck, but the short space between the F-35C main landing gear and the tail hook point means that there's not enough time for the pendants to raise the wire above the deck in the manner that the original (erroneous) wire dynamics model would have suggested.
> “Boss,” he says to me, “This fucker ain’t gonna work. Look at this thing. It’s short, it’s too close to the wheels, and look at this dumbass hook shoe they got on it. If the wire don’t hit it exactly right, it’s just gonna go under the hook and you’ll bolter.”
Did nobody with practical experience with arrested landings look at the arresting hook design prior to this? Obviously computer models can and do predict extremely novel solutions to existing problems, but it's worth double-checking the model when someone with practical experience says "it will never work"
In this case, it seems like a simple slow-motion video of an arresting wire going under the wheels of an F-18 would have been enough to debunk the model.
Random thought: this is a case where someone's intuition matched what actually happened, making us think "why don't they listen to people with common sense?".
But what about the many other cases where someone with "common sense" said "this fucker ain't gonna work" but the thing worked as predicted by simulations? Surely they must have happened too.
My point is that when models predict counterintuitive results (which they often correctly do; See e.g. Eurisco in the Traveller TCS championship, or the shape of the F-117 compared to contemporary stealth aircraft), it's worth double-checking.
> Did nobody with practical experience with arrested landings look at the arresting hook design prior to this?
I mean... it's very likely that the answer is no. The last new carrier aircraft made was the Super Hornet - and that design was basically done by 1995 (the F-35 tests in question were in 2011/2012). That expertise would also be at McDonald Douglas/Boeing. Northrop Grumman has a long history of carrier aircraft development, but it would have been long dormant by that point.
I'm sure there's all sorts of reasons the model's inaccuracy wasn't caught before hand, but sometimes... if you're given a model that's someone says that's been V&V'd, and it produces a result that's only a little weird, you just go with it. There are only so many things you can add extra testing onto in a project. Sometimes you choose wrong.
Anyhow, consider that the model results were probably exactly what they were expecting. Remember that the designers would be honing in on the shorter tailhook. You can imagine their mental model going - "ok on legacy aircraft, we have flatter tailhooks because there's enough time for the cable to settle". And then going "ok, with a shorter tailhook, there won't be enough time to settle". And then their model comes out and say "ya, with the shorter tailhook, it won't have enough time to settle - it'll be UP IN THE AIR". Whereas reality is "ya, with the shorter tailhook, it won't have enough time to settle - it'll still be displaced DOWN".
Unfortunately, my decade plus as a military aircraft tech has taught me that no, practical knowledge does not make it through the system nearly as fast as engineering "expertise".
Same, but different industry. Lots of re-engineering of the wheel after the original designers give their warnings and recommendations, but everyone's too smart to try the simple thing that already works first.
The F-18 tailhook geometry is far different than the F-35 tailhook geometry. F-18 hooks are much farther back from the main landing gear, and are also much longer.
They all came out of a book that anyone can buy called "F-35: From Concept to Cockpit". That book is a compilation of papers presented at an AIAA conference in 2018.
Yeah that missile fused. It was a normal AIM-9X. Missile war cranium detonations are underwhelming (there's not that many pounds of explosives involved) because they are trying to kill aircraft via shrapnel of some kind. Two common designs are a continuous rod that expands in a circle around the detonation, or a hashed metal casing that produces hundreds of small fragments.
> The balloon was brought down with a missile that had the primary explosive removed
The missile fused and was a normal AIM-9X.
You can see result of the war cranium detonation in the videos as a cloud of black smoke. It's just that things are happening so fast that it looks like the missile just sails right on through.
> This being the F-22's first air-to-air kill after nearly 2 decades in service, it's not like these missiles are a precious resource to be conserved.
The first clause of your sentence here is not logically related to the second clause. These acquisitions programs are separate. While missile purchase quantities are adjusted based on projected needs, it is not as if there is a quantity in inventory earmarked specifically for the F-22 only.
> It is possible that the F-22 simply could not get up high enough to engage with guns even if that was desirable
> You can see result of the war cranium detonation in the videos as a cloud of black smoke.
That's the tiny self destruct fuse going off after it's already gone through the balloon. An AIM-9X actually going off is substantially larger.
> The first clause of your sentence here is not logically related to the second clause. These acquisitions programs are separate. While missile purchase quantities are adjusted based on projected needs, it is not as if there is a quantity in inventory earmarked specifically for the F-22 only.
There is most certainly an inventory of munitions earmarked for the F-22. They are stored on the bases with the F-22, the pilots train with these munitions, if there was a need to deploy them they would deploy with these munitions. They could potentially be used by other planes, but only if they were urgently needed, which was the actual point I was making: these are air-to-air missiles, used only for an aircraft to shoot down another aircraft, something which has been a very rare occurrence for quite some time. Not only was this the first A2A kill for the F22, it was merely the second A2A kill for the US in the 21st century. This lack of aerial combat is the reason why the F-22 has been so rarely used.
Yeah, there isn't any explosion even that large. Perhaps you are mistaking the cloud of vapor as the balloon pops?
Again, the goal was to preserve as much intact as possible, shrapnel production runs completely counter to that objective. I'm basing my comments off this source [0], which admittedly is just some guy on the internet, though I have not been able to find any source claiming the missile carried a munition, if you have one please share it.
> Perhaps you are mistaking the cloud of vapor as the balloon pops?
No. Two clouds are visible. One (white) clearly from the balloon envelope, and one (darker) clearly from the missile. https://imgur.com/a/y386kLU
It's also very clear from the video that they targeted/hit below the balloon's envelope.
> shrapnel production runs completely counter to that objective
So does missing it, and they're already dropping it from 65k feet. They'll be prepared to piece things together.
> I have not been able to find any source claiming the missile carried a munition...
This is a pretty silly inversion of burden of proof. Between "some guy on the internet said something no one else reputable is corroborating" and "the Pentagon typically doesn't use dummy rockets to shoot enemy aircraft down", I know which side I fall on.
The problem with being an expert on the F-35 is, you can't talk about most of the things you know. So internet hot takes dominate and you can't refute them without revealing things that you shouldn't reveal.
That goes both ways. The problem of being an expert on the F-35 is that you can't warn friendly nations of the critical problems classified as confidential.
"“Details of [deficiencies] — even unclassified [deficiencies] — are not publicly releasable because the information is operationally sensitive, and its release could be detrimental to U.S. and international war fighters operating F-35s worldwide,”... Seal noted that all remaining critical deficiencies are classified as category 1B issues, which represent a “critical impact on mission readiness.” The more serious category 1A problems indicate a risk to the operator’s life..."
"...In June 2019, Defense News published an investigation into the F-35 that detailed all 13 category 1 deficiencies on the books at the time — the first and only time a full list of F-35 critical deficiencies has been publicly released."
Since Finland selection was done way before, how many of these were disclosed to them?
The F-22 was that advanced. The avionics suite of the F-35 is significantly more advanced than the F-22. The F-22 hasn't been produced in over a decade and is going to be phased out in the early 2030s.
Even if there weren't a law still on the books banning the F-22's export, we don't have production and no one would buy one when they could have a still-in-production F-35.
In the present time, there is an air defense network around Washington DC. It uses the NASAMs system. Anything that is determined to be a threat can be engaged with surface launched AIM-120C missiles.
That Pentagon shaped pool really caught my eye, then the long covered building, had to look it up. Now I know all about the David Taylor Model Basin and the Naval Surface Warfare Center's Explosive Test Pond.
I've literally been that engineer, manning that computer, in that room at White Sands, for that instrument on SDO. I was the telemetry/power systems engineer for one of their launches to do underflight calibration on the EVE instrument. It was an absolute nightmare of a job.
The PCs were Dolch lunchbox style single board computers. I had to scrounge the one together for the mission I worked in 2013 by assembling it from a couple of others that were broken. Then I got the joy of installing DOS, to run good old TDP502.exe to handle the telemetry stream coming in. Every fifteen seconds I had to hit "Print Screen" to make the tractor feed printer attached to the machine do a print out of my screen. Kind of insane given that the ground station was making a Chapter 10 format recording of the same data, but NASA allowed no deviation from the way they had always done things.
You're right about the tone of those statements. But just how much dystopian bureaucracy can one take before expressing displeasure at the dystopian bureaucracy?
It’s important to understand that in order to navigate the world we live in, sometimes you’ll need help from someone “on the inside” and often those people don’t have direct control over the outcome but can be very valuable allies. Taking your frustration out on those people is directly against your own interest.
Being angry at what is essentially a front line customer service rep will do absolutely nothing at all to end the dystopia and just make one human’s day worse.
If you’re not mad at the person who actually has the power to fix the problem then you’re wasting your breath. And to get out ahead of it, your anger doesn’t “go up the chain” either. Like all service workers, dealing with angry customers is a standard workplace hazard.
But the service worker here (Anthony) has the power to approve the app, he is just incompetent or hostile to the author. The anger is warranted (though the statements are harmless), this is targeted sabotage.
Not sabotage, just incompetence. Anthony is probably overworked, and has seconds to approve or reject an app. He may also not be a native english speaker, and be unsure about the meaning of the words used in this app.
Sure, he has every "right" to be angry, but that strategy will only make things worse.
Quietly accepting the abuse, and getting angry are two sides of the same coin, both of which are basically "just embracing the dystopia." Notice that the same people that do the first, will eventually do the second once pushed far enough, with ultimately, the same results.
It's perfectly fine to express displeasure in a calm and collected way, the same way you would patiently correct a child... e.g. "I am really disappointed that ____, can you please ____?" (without being sarcastic or condescending)
But losing your cool doesn't correct anything, it just expresses emotional weakness and makes the person dislike you and not want to help you. Someone customer facing at a place like Google sees angry upset customers all day long, and probably makes fun of them to their friends and family, but doesn't go out of the way to help them.
What they don't often see is people showing strength and calm in the face of this BS. Someone that can endlessly adapt and redirect things back to their own clear goals regardless of what is thrown at them, without even needing to be adversarial. Think Bruce Lee's "Be like water." In most cases, this will be off script for them, and they actually won't know how to deal with that other than giving you what you want.
