> But at the rate we were going, we were still roughly five years from finishing.
Everyone has seen how this sort of thing comes about: it's meaningful work for engineering but never business/product critical so it just drags along.
Seems like this time around someone just went "I wonder if we could do it this way" and it worked. Perfect example of ditching sunk-cost and starting from scratch. Great outcome for them.
This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting.
> Back in 2022, we set out to migrate Asana's frontend test suite off Enzyme, our aging testing library, and onto React Testing Library (RTL).
Your telling me they had a full team of engineers at Asana, doing nothing but rewriting tests for 4 years until AI came along and did the last year in a couple days? I'm extremely doubtful. I don't doubt for a minute they were one-track to take 5 years, but not because it was 5 years of engineering effort for humans.
Far more likely they finally cleared up some tech debt they had been plugging at off and on for 4 years, then a PR flack got ahold of it and it became a breathless "AI did 5 years of work in a couple days".
I'm not sure I agree with the extremely easy to verify part. How do you validate that your new test suite covers the exact same edge cases as your old?
But yes, LLMs are great at tests. I'm not doubting that an LLM migrated some legacy tests, or that it was a task taking a long time. I am doubting the way it was presented in the article, that this was taking a full team of engineers dedicated to nothing but this, 5 years to accomplish (and presumably already spent 4 million working on this, since the total estimate was 6 million, and they've been at it for 4 year).
I think some PR flack got ahold of the fact that an LLM wrapped up migrating some legacy tests that a team had been slowly chipping away at for 4 years, between their normal feature work, and were on track to finish in 5, then wrote it up like it was that teams entire focus instead of a piece of tech debt.
That makes far more sense to me than spending millions for a team of engineers dedicated to nothing but rewriting an existing test suite.
> Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify
I don't think that's true. All you see is all the test pass. You don't know that the tests still cover everything that they used to.
LLMs excel where there is an excellent test suite, and you ask them to modify the thing that the test suite tests, and forbid them from changing the tests.
How is it bad faith to question corporate marketing blog posts? That's not bad faith, that's table stakes for critical thinking.
We absolutely should be skeptical when an AI company makes big claims. The fact that the company the AI company is talking about also claims the same thing doesn't change that. OpenAI is getting awareness and marketing out of this, and I'm sure Asana is getting something out of it too.
And I'm not even saying OpenAI or Asana are necessarily lying. Asana might not find out for months or years that some tests had been rewritten poorly, and don't sufficiently test the thing they were supposed to test anymore. For example. If they truly had 5 years of work, then I find it hard to believe that in two weeks of the LLM churning, they had the time to review all the new tests. They spot-checked, at best.
Maybe everything is great. Maybe the LLM did a wonderful job, and this was awesome for Asana. But we have no idea, and we're unlikely to ever find out. Unless, of course, it's in Asana's interest from a marketing perspective to tell us.
(Not sure what the URLs you posted in your comment are supposed to prove. They're unrelated to the issue at hand.)
That’s not feedback, just a non-constructive opinion. As someone else wrote above, if you’re so precious about it then help out the devs of the project and submit design PRs.
Here is a constructive one then: delete it. It’s better for everyone. If I want to read AI slop about your open source code, I can ask LLMs directly. It’s cheaper for you, and everybody.
This is absurd. No one has an obligation to help anyone nor do they owe you any standard of help. Opinion on quality of output is constructive to the wise and an opportunity to scream "If you can do better than do it" for the small minded.
I'll offer a different perspective on this: for years we (product/c-suite people) constantly got push back that giving a goal was no longer enough, we had to articulate the path to the goal in excruciating detail.
If I need to think about the solution that hard, I may just as well take it all the way, remove the middleman and get some efficiency back.
There's still a way out of this mess though. All it takes is for folks to take shared ownership for hitting the goal and bring their expertise to the table to help draw the path to the goal.
I'd ground what I say in sharing your assessment of the observations or insight, but with a different takeaway.
If I need to think about the solution that hard, I may just as well take it all the way, remove the middleman and get some efficiency back.
Totally. And the takeaway is that we by and large should be doing this much more. That's what's needed from leaders. More vision, less delegation of vision. The stronger the vision, the less need for delegation anyway. There's something to be said about the difference between big picture and small picture, but at this point it's relatively intuitive that the big picture divorced from the small makes for worse big pictures. I'm not talking about how handing the big picture to the middle person tends to create interpretation/translation errors as well as execution errors to the extent the subordinate is less capable. Those factors are prominent and _in addition_ to the issue that the big picture is a worse big picture when these duties get stretched across two (or more) people. And thus, we should own these duties "vertically" and if needed reduce duties "horizontally".
Leaders (and companies) should be doing less horizontally. It runs the normal risk of stretching to thin, but their tenth and eleventh ideas are also filtered towards worse. Perhaps more fundamentally, the bar for adding a product/project should be higher. When the leader can "just" add another leader to take on the work—particularly one by nature with less power, ability, autonomy, and perhaps accountability—it's getting set up for a lower chance of success. In order: (1) the project probably shouldn't start at all; (2) if "started" at all, it shouldn't really look like starting, it should be a person validating for the leader who will then start it (many product teams pretend this is what they're doing); (3) if started, it makes quite a bit more sense for the higher (and likely more capable, but especially more empowered) leader to take on that project and hand the existing, stable, better trodded project where the team has institutional knowledge and support to the other leader.
In practice, sharing ownership is superficial, whether between leaders at different levels or between project team members. Why share? Own it or have someone else own it. Or if there isn't trust in the other person, cut it. If the leader doesn't have the time/interest and doesn't think they have someone capable of doing it, it should be as much a sign as a low quality idea.
It's not hard to find problems with middle managers—many are with the sorting that picks them, many are with the conditions of the middle. Structures and strategies to remove them, as you suggest, are a great idea. The main way is for the leaders, when faced with the scenario of "if I need to think hard", to not come away with "let's do it, but not me." We should be default no. Green lighting some work to validate it to a Yes works, but these are tasks such that the leader knows what kind of validation is needed—and is assigned to a person with those skills. An engineer, marketer, business analyst, user researcher, data scientist is going to validate things that a PM or a director have less or no training in. Leaders tend to appoint the latter though, sometimes for empire-building reasons, but I think more basically out of fear of control and legibility. Those are counter productive reasons if understandable.
You’re absolutely right, but just to a point. It should be easy to clearly quantify the desired financial outcome of a sprint, but not of its components. I don’t want to spend a single minute figuring out the financial outcome of a single ticket.
what are the plans to support design systems? no one seems to be able to do this. prototyping doesn't happen in a vacuum, I'll always want to use the design system we spent months building in Figma.
Yes! We have a feature for reusing components where you can prompt "use my @design-library/button." But right now @design-library needs to be re-created in Magic Patterns. This is behind a feature flag, but let me know your account email or DM me alex [at] magicpatterns.com and we can add you to it!
We're looking into importing from Storybook too, but it can get fairly custom and we want to make sure it scales. It gets tricky because these systems are expecting a certain format - in our case React + Tailwind - and so if the design system isn't that, the LLM won't handle it well.
Once you figure out how it can understand all my Figma components and their styles (maybe via some “design”MCP).. it will make me 100% switch my design proto workflow!
exactly! I do not understand the British obsession for preserving insane amounts of crooked-walled buildings with horrible insulation ratings. zero historical interest, 100% NIMBY-ism. then the general populace mopes about the lack of housing, and dreams 15-minute cities without realising the only way to solve both is tear down what’s taking up space and build tall instead.
Key statement:
> But at the rate we were going, we were still roughly five years from finishing.
Everyone has seen how this sort of thing comes about: it's meaningful work for engineering but never business/product critical so it just drags along.
Seems like this time around someone just went "I wonder if we could do it this way" and it worked. Perfect example of ditching sunk-cost and starting from scratch. Great outcome for them.