I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Our monolith flagship web app. And, there’s more systems at LinkedIn with MM lines of code and long builds example is the mid tiers, the offline data stacks, the metrics systems, KafkaKafkaKafka.
Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.
I worked at LinkedIn in infra. Their internal tools are a nightmare. The true definition of Jugaad.
The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing.
I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.
Seeing you use 'Jugaad' make me unreasonably happy.
I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution.
Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.
LinkedIn thinks so. Their tech stack is so over engineered and spaghetti that they spent a hundred million and years trying to move into Azure and were beaten by how bloated and unmovable the tech stack is.
Theyre still using python 2 and centOS in tons of systems.
Yea, give everything 3x estimate because everything that can break will break along the way.
Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.
In my first SWE job, I was perplexed, I never felt much impact for solving complex bugs, even if they had side effects like reducing latency for everyone in our infra. Anyways, I realized it's like you say - you get rewarded for pushing features and shipping stuff. This incentive can easily become a 'tragedy of the commons' situation.
My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that.
As opposed to making builds faster, or build infra faster/cheaper.
There are mostly only a few ways to make builds faster.
* Ship less code (Very hard to get a large org to do)
* Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code.
* Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building.
These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.
I mean both. Smaller pieces but also less of those pieces. It is my experience that many systems code size is less purely a function of what they need to do than a complex set of layered components and systems in the name of ease and fear of NIH syndrome.
Sometimes doing the same thing with simpler components and fewer components is just better.
The top 1/3rd of Eng don’t see it as immutable. But the stack can consist of 10 other teams tools, and you’re not rewarded for the amount of time you’re spending to fix other people’s stuff. So it goes on
Just for another perspective, I've worked at a few companies including LinkedIn as a backend developer and I think LinkedIn is probably around 70-80th percentile in code quality.
There was significant emphasis on code quality at least on the team I was on, and an ever improving culture.
I did work on one voyager task though and I remember it being a nightmare
They're trying to be a site that makes a lot of money from ads. To do that they need to get people to spend a lot of time there. To do that they need an infinitely scrolling feed that has addictive characteristics. What else are people going to spend a lot of time on there? Scrolling through their contacts?
Oh, you meant what are they trying to be that's helpful to the user? Doesn't matter.
I wonder, is "being facebook" is that an organic goal LinkedIn came up with, or did they pick that up from their users?
I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.
In what way? Users/customers do not generally drive development. Indirect measurements of users do, such as measuring "engagement".
At best, what likely happened was A/B testing showed "what users want", which rarely and only by chance intersected with what users actually want, and instead showed over and over that socials media patterns (light and dark) hijacking attention drives engagement.
If people didn't want to use twitter, they wouldn't and it would be gone. But users do use it, and even the folks who tell me they're upset ABOUT Twitter, most choose to be there.
Whatever the reason they make that choice, at some point that's on them.
Nah, basically every other social type thing got screwed by FB convincing everyone that MAU's (i.e. the metric FB looked best on) was the right metric, when it clearly wasn't.
Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).
Some users love it, the type who want to sell their every thought online to anyone who can tolerate them.
Other users (almost everyone I know) absolutely loathe it. It is hands-down the worst service I have an account with, but it's also practically required to get a job if you don't have lucky personal connections. I was hoping TFA was actually about leaving as a user, that it might be inspiration for me to free myself.
They made loads of product changes to get people to engage in that way. As an example, showing who viewed your profile in the email vs needing to log in.
Part of that might be the "reward for spamming" - any "social media" site is always amazing at the beginning, because only the die-hards know about it and the spammers have no return on investment.
LinkedIn has slightly different style of spammer, but it's basically the end game for anything.
It's because LinkedIn is old in terms of web-age. Really old. They, along with thousands of other companies, had the need for a reactive web front-end when JavaScript had not yet matured to where it is now.
this plus lack of performance eval/culture/strategy to grow responsibly and enable future success
its a great learning experience though despite what people say about the inability to learn the new hot hot tech. the nuance of software development is the decisions that other people make, that is inescapable and is a skill worth developing. i'm not buying the "just do a startup" because i think its a cop out.
In my experience, a good platform team will increase velocity a bit while a bad platform team will tank it. Perpetual migrations and re-writes, "simplified" platforms that fail to meet future developer needs, half-abandoned NIH systems as people move teams, ownership moats if product teams try to touch platform code, etc, etc. So if a company has been burned a couple times they may decide to just not bother.
Preface: I’m not in front end or flagship teams so I’m saying hearsay.
The initiative Chris refers to is still alive but likely hasn’t made any meaningful progress. These things tend to have a lot of fan fair and then suddenly leave the front conscious of the company as we get a new GDPR/DMA fire to put out. at that point it will be dead. Or, it will stay alive for 2+ years as slowly but surely 50% of clients migrate to it and see even poorer performance.
There was a blog here a few months ago about migration from restli to grpc. It’s still going on, and nearly every app i see is still restli.
1. Your reply to parent said dams’ output is small
2. parent refutes the data
3. You reply again with same message
Can you address the parent’s concern that the data you’re reporting is incorrect? (Solar metrics are based on max / theoretical output and not real output)
Parent did not refute the data, parent said they did not care that a vastly greater quantity of renewables were being installed than the hydro capacity being removed, due to concerns about capacity factor. My second comment therefore showed that the hydro capacity being removed is insignificant relative to hydro alone, without reference to other renewables.
“Comparing max wind capacity to actual outputs from dams is nonsensical. Wind typically produces 30% or less of its max capacity over any period of time and it does so unreliably. The energy from damns is much more valuable.”
Still absolute amount is a factor. A small river dam with 0.09% power can absolutely not be worth it if it is old, in the wrong spot, has huge detrimental effects on downstream areas etc. Everything has to be taken into account.
Sure hydro powers well with wind and solar, but a dam can still be overall a bad thing. And I say this as a guy who grew up next to a dam which is one of 12 dams on that river and I don't have a problem with any single one of them.
Building structures that guide rivers is never a thing where the thing you did decades ago will turn out to be the most clever, most efficient and environmentally best solution. That means people who live with and around rivers have to be able to adapt their plans to what they learned. And sometimes that means realizing a particular dam in a particular place does more harm than good, while this is the polar opposite for some other structure.
The person you quote was mistaken; I cited nameplate capacity, not production output, for both hydro and wind. (My wording could have been clearer.) Furthermore, capacity factors are almost identical: US wind production in 2022 was 35.9% of capacity, while the corresponding figure for hydro was 36.3%.
your mind also conjectures many unreasonable and illogical solutions to things. then separately your mind has an "evaluator" to deduce a set of solutions down to things that make sense.
Is anyone saying LLMs in isolation and without contextual tooling are miraculous? Even the text box for chatGPT is a form of wrapping LLM in a UX that is incrementally more miraculous.
> double the amount of the gas than in New Mexico, per unit of production, since 2019.
when we look at data at scale, we use p99 metrics to get more insight into the average. because this article, i'm assuming, is just using average, the metric here does not have enough information. There could be just one incident in texas over 4 year period that affects the average per unit of production.
> Despite increasing its own oil production in recent years, New Mexico has no site with repeated methane leaks, unlike in Texas,
Why not deep dive into root cause of a few of these abusers? Blameless post mortem to assess what must be changed with very specific examples of issues.
Are we wrong to request a higher bar for journalism / research from Kayrros?
I know the hackernews crowd is very pro-regulation, so before downvoting please consider responding so we can have a discussion and share experience together.
"It seems the regulation in New Mexico has had an impact without hurting business"
Assuming that oil has some net affect on humanity -- pros and cons. Then we need to consider the net affect of oil produced by texas on humanity, and consider how that net affect is changed with more regulation.
Simply saying "yeah it seems like New Mexico doin real good with more laws" is extremely dangerous, because most regulation DOES harm and destroy market affects and competitive businesses. The authors on the study here should AT LEAST report with due diligence on this before requesting for increased regulation and attempting to sway public opinion -- potentially having a net NEGATIVE affect on humanity.
One possible example of not considering the net negative affect of regulation, is, for example, if there are many competing energy companies in texas, and there is a consumer demand for more responsible operations, then those competing energy companies are forcing each other to tighten their operations wrt environmental impact. However, if regulation destroys small producers and benefits big corporations, then we would see LESS competition and LESS Priority from companies to tighten their operations, negatively affecting the environment.
I don't think "hackernews is pro-regulation" is accurate, but that might be an easy mistake if you are "anti-regulation". I'm neither thing. I like regulation that works well and dislike it if it doesn't, which I dare say is the rational POV. I'm going to repost my comment from a thread on a recent story flagged as dupe:
It's not the amount of regulation that matters, it's the intent. Is it written to benefit a greater good, or written to benefit a political party's benefactor? The same applies to removing regulations. Where did the intent lie? New regulation and repeal are most often shades of grey --bad mixed with good in terms of overall consequences. Many changes disproportionately help a few at some expense to the many. That's what happens when minority interests with a lot of money are overrepresented in government, lobbyists write bills, etc.
My conclusion on hackernews being "pro-regulation" is based on observance that posts discussing the downside of regulation are often downvoted.
I agree with what you said. Knowing that repealing regulation is much less frequent than producing new regulation -- we should assume all new regulation is high risk, since its' impact is known to have tradeoffs, is to some degree created from perverse incentives and minority interests, and once in place is immutable. Regulation can be good and it can be bad.
So if we know that regulation carries high risk, we need to be very apprehensive to anyone calling for more regulation such as this article via "https://www.kayrros.com/". We don't know who they are, what they want, and what their suggested change in policy affects in net to humanity over time.
I guess what I want is for companies like "https://www.kayrros.com/" to be held to a higher bar and be more responsible for the outcome of their research and conclusions.
> because most regulation DOES harm and destroy market affects and competitive businesses
I don't see that as a problem, when the alternative seems to be essentially zero oversight over people who don't care if they -- through their own negligence -- release chemicals into the atmosphere that contribute to the destruction of our planet's biosphere.
"Business interests" are irrelevant if our planet becomes unlivable for humans.
Regardless:
> Simply saying "yeah it seems like New Mexico doin real good with more laws" is extremely dangerous
Is anyone actually saying that, though? Sounds like a straw man to me.
in addition there is a scaling factor. it's easy to control mistakes when you have 1 well. hard to control mistakes when you have 10,000,000 wells. thus the regulation has to be mapped appropriately.
also, I'd be interested in seeing p90, p99, etc. to see if outliers affect the average that is reported -- because as texas probably has larger 100x production, average is exposed to more large incidents.
i would be MOST interested to compare against other similar "sovereign entity" with comparable production scale and comparable challenges e.g. geography and infra.
the US has a huge environmental benefit in producing oil vs entity's like China, Saudi Arabia, because as a transparent democracy we are able to hold our producers accountable to regulation where offshore producers have no transparency and accountability.
I have 90 PRs to Mypy configured repos this year, across 10+ python repos which are a mix of services and background jobs. All of them have some mix of pyright/jedi+ruff+black. I have implemented these tools either via CI or in my ide like experience in NVim.
TLDR; BE WARNED. this basket of bandaids is NOT a replacement for a strongly typed language. I would NOT recommend Python to teams or for services. No amount of mypy, pydantic, pyright, ruff, black, can convert python into a production language. This collection of bandaids is complex to maintain, upgrade, standardize, and has infinite edge cases where type annotation errors are just wrong.
The worst part about this basket is that you get all of the above features for free and are REQUIRED with go via gopls and gofmt. Gopls "is the official Go language server developed by the Go team." good luck achieving parity of this in python.
If you implement a service/app/anything production, use anything but a dynamically typed language like python.
It doesn't take an expert to notice that enforced types make a language more reliable in production, easier to debug, and faster to refactor.
You should be horrified by Dropbox's blog post on python in production: https://dropbox.tech/application/our-journey-to-type-checkin.... because this problem only existed and required $Ms of dollars to fix (even hiring the core team behind Mypy) because Python is being used instead of a typed language.
I'm rejecting all dynamically typed languages *for production*. Where production means anything that needs to exist for a long time and is supported by a team. A litmus test for whether an app is production is if you actually need mypy, pyright, jedi, to effectively work on the repo.
Python is great for single developer experience and going fast as a single developer. I use it every day, willingly, in jupyter notebooks. But at scale there are no success stories. You are zoomed in on the present success of "engineering miracles" and ignoring the big picture that the application would have moved faster if they had the foresight to choose a typed language.
Hence, why i'm saying be warned on following the path of mypy+black+pyright+ruff+etc., since you will be doomed to patching on top of a fundamental flaw of the language.
I highly recommend to avoid Python for production apps especially if it is your free will (and not a matter of refactoring or rewriting existing mistakes).
yup! along the dimensions of feature velocity and developer experience with regard to cost.
of course python/ruby/javascript have succeed in bringing value to the world in general -- but in this thread we're focusing on production reliability, debugging, refactoring.
as we see, dropbox is proud to admit that they hired Mypy core developers and fund the project, but guess how much they spent on the mistake of using python?
twitter, famously an advocate for Ruby, spent an ungodly amount of engineering to work around ruby, hit a wall, and switched to JVM.
Just the fact that mypy and typescript exist and have exploded in adoption is proof that eng. teams very quickly hit a sand pit when using dynamic typed languages. Mypy being funded by dropbox, Typescript funded by microsoft, shows the level of investment needed to support these dynamic languages.
So the thousands of developers using Python either doesn't know their own best or are just in perpetual pain in their use of python? There isn't the slight possibility that your view on what is a useable programming language in production is subjective and not objective?
Yep! I think all of those repos will either proverbially die a hero or live long enough to see themselves as a villain.
And thats because of the fundamental tradeoff of dynamic typing vs static typing. That part no one can disagree on.
The only disagreeable part I believe I'm saying is where does the dynamic typing scale limit happen. As a polyglot developer, working a lot with junior developers, I can say that dynamic typing hits a scale limit of feature velocity / production support / cost much faster than the general folk are aware of.
by no means is go/java/rust/c++ a silver bullet. it is a way to raise the bar for debugging, reliability, support. it is more effective than bundling python with 5 other type bandaids. in the medium/long term, lets guess at 2 KLoC or >3 person team, these typed langs will result in faster feature development and lower issues in production.
its comfortable, easy, appealing in the short term. just like why people overeat their junk food.
there's this common idea in management that new grads, SREs, "less experienced devs", can jump in and contribute. it's a falacy because the code becomes unmaintainable without a large investment (more unittests, refactoring the "buggy" code blocks with mypy type annotations, some SME who continues fixing prod bugs). a Python developer will shove their head in the sand with repl'ing, testing after deploy, and settle with this dev/ex while being completely unaware of the benefits of compiler errors and warnings.
Python has the same level of complexity as another language but you'll only see all of that dirty baggage in production as runtime exceptions.
As a former LinkedIn engineer, let me ask, what happened? LI was using protobufs and BJSON like ten years ago. That was part of the whole conversion from Java serialization to rest APIs. This article implies that tossed everything, went back to text based JSON, and then said, “Well somabitch! Binary formats are faster than uncompressed text after all!”
There are 19,000 LinkedIn employees world wide according to my cursory Kagi. That means more than two endpoints for every employee. I feel like regardless of micro or macro that’s an enormous sprawl of entropy.
The term “endpoint” is pretty fuzzy. Often it’s meant how many different url handlers there are (eg a service exposing “customers” might have dozens of endpoints for searching them, listing them, creating, updating and deleting them etc.
So number of endpoints can quickly grow to a big number.
More seriously though, I wish we used generic terms instead of using brands. "Look up" (or lookup as a noun) or "search" can't be too hard to use. We don't need to turn our thoughts into ads at every corner.
(so, yes, I'll take your nope for my whynotting)
Now, if you are launching a search engine, you need to name it wisely for when people verb it. As a brand you probably want this to happen. Duck Duck Go was not very wise.
Our interlinked system of distributed monoliths includes one in AWS region af-south-1, another on the Moon, and a third orbiting Jupiter. All of these services are yours, except for the fourth on Europa. Attempt no POSTing there.
There's certainly a spectrum, but you can have huge clusters of even the most monolithic monolith.
The real semantic fun begins when you have monolithic codebase/deployments that could do everything, but each instance gets assigned a specialized role it serves to its peers _
(I think I've seen someonevdescribing that approach somewhere). I'd consider that micro if there are more than a handful of "roles", perhaps with some qualifier, but it would hopefully be easy to agree than no description is entirely wrong.
I haven't, but if I had and if I was trying to place it in the conference circuit, I'd call it morphoservice.
(and the actual me who has never reached beyond consumer side in conferences sure hopes that conferences would resist unless a less ridiculous name was found)
I have seen 5 table micro services. A solo backend developer had created "3 micro services" and as I joined the team, their argument was that we're doubled in strength so there were a series of architectural meetings to split everything nicely into 5 different services.
Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.
Anyways, AMA