<this comment doesn't really add anything, but thought I'd share anyways for whatever reason>
I forgot the editor (maybe TextMate?) that was in vogue during the peak of the Ruby on Rails era, but there was such a feeling of magic to using what was a fairly basic editor that still had syntax highlighting.
Was this feeling of magic purely because I was younger? Or perhaps we did peak in terms of the ergonomics of human-controlling-machine without too many aids?
Fighter pilots used to fly with skill and instincts, but now are assisted by all sorts of high tech equipment that has removed much of the "flying skill" and replaced it with "equipment skill". It's not that fighter pilots are worse now. I'm sure they are better at achieving the outcomes desired, while commanding much more complex equipment. But the perhaps the art of flying is less emphasized.
In the same way, perhaps the era of software engineering is changing too?
A lot of this is new to the open source world. Proprietary systems have had this for decades. In a lot of ways, the stuff we use for things like javascript are a huge step downwards from the tooling available for Java, C#, and Visual Basic.
Visual Studio is an absolutely incredible piece of software. Two decades ago, you could drag and drop GUIs. You could write callback functions on buttons and never see the any of the code around that. You could write entire programs this way.
Vibe coding has existed since visual basic for applications escaped from the deep dark depths were it was wrought. If we want to go back further, look at fourth generation languages–the unholy realm where SQL came from. ;)
What we are seeing is wider adoption of old ideas. That wider adoption may be sufficient to cause a new era of engineering.
This sentiment peaked in popularity in urban areas ~4 years ago. Since then I've noticed support for this position slowly eroding, and my hypothesis is that the general population has slowly had enough interactions with someone who is on fent.
it’s always a little weird to me how out of touch this comments section is, especially on the topic of certain social ills that people wish to normalize. I really think the constant pressure on these subjects has been counterproductive to that goal.
As someone who used SoftLayer shortly after it was acquired (and it was still pretty much untouched by IBM) - SoftLayer was pretty bad to begin with. And I'm not only comparing it to AWS. It was bad even when compared to RackSpace.
I left Amazon a few years back because this ending was the inevitable outcome. Amazon had a chance to reinvent themselves as the scrappy startup that they claim to want, but instead they went full IBM.
Basically, the management class despises SDE worker class, and thinks of them as overhead. Recent statements by the aws head about chatGPT replacing SDEs is along the same lines.
SDEs are tools that just do what mgmt tells them. mgmt holds the decision-making and all the cards.
periodically there is a whipping (pipping) in the form of a layoff to keep the troops in fear.
I’m curious: what, in your experience, was the root cause of this contempt? Other skilled professions can make decent money as SDEs do. Is it a love/hate thing? Feeling like the tools could easily not need them if they had sufficient gumption and will?
you touch on a raw nerve that could be the subject of a long post.
in summary, the attitude of management in many large companies is that code is just work that needs to get done, and any engineer who can type on a keyboard can do it equally well (cue in ai-coder). so, the smarts is embedded in defining requirements and managing execution of said-code which resides in management.
The problem with this is many-fold.
1) it encourages a culture of top-down decision making including technical decisions and the person making designs is not the one doing the work
2) as tech evolves, the org is unable to catch up since the decision makers are the elite few.
in short, a manufacturing line mentality where the supervisor holds the cards and workers are tools.
It becomes apparent as you move higher up the IC ladder that technical work becomes slightly taboo. And it is precisely because of the attitude that the actual work is something for line employees, not them. This belief is passed down not overtly, but through gentle nudges: "you could fix that in 15 minutes, but is that really maximizing your impact?" Or the ever-present, "but how does this improve your promo packet?"
It all feels like a giant psyop on staff engs to constantly gaslight themselves into immolating themselves to the god of Impact. Little wonder they exhibit much higher rates of burnout: they're told to be responsible for things without the full range of authority.
I was $TOO_OLD when I learned that most of the actual coding on Google's systems was done by L4/L5's.
Agreed. It’s classism institutionalized in MBAs. The managerial class despises low level execution and instead considers strategy to be the most important thing. And they consider workers as replaceable pawns and not partners.
Essentially the problem is that Kafka doesn't have a way of managing acknowledgements on a per-message basis. This means that consumers from Kafka topics are assigned exclusive access on a per-partition basis and the consumer group manager only tracks acknowledged offset of each partition.
As such you end up with a few main problems. The first is head of line blocking, what this means is that if a consumer reads a message from a topic it's unable to process or will take an inordinate amount of time to process it can't move forward without potentially having to replay every message since the problematic message if it doesn't want to risk not replaying a message that wasn't processed correctly. Secondly it means that you can end up with hot partitions if the "cost" of messages isn't uniformly distributed across partitions because load isn't balanced across consumers, i.e there is no work stealing or other mechanism for other consumers to help out processing a hot partition.
Log systems with queue/subscription overlays like Pulsar and GCP Pub/Sub solve this by doing per-message acknowledgement (sometimes referred to as selective acknowledgement vs cumulative acknowledgement that Kafka does) usually by layering a persistent subscription abstraction over the top of the underlying log.
This is in contrast to pure queue systems like RabbitMQ, SQS etc that use a heap or mailbox approach where messages are simply emptied out as they are processed and don't share the log style struction of systems like Kafka.
So TLDR. If you use Kafka like a job queue you will end up in situations where queue processing gets stuck behind a single or patches of unprocessable messages.
The mitigations for it aren't pretty. They either involve building your own selective acknowledgement layer or a series of retry queues that messages are pushed onto using Kafka transactions with a final dead-letter queue at the end etc. Instead either wait for https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A... if you really want Kafka or use something that already does this, i.e Pulsar.
I forgot the editor (maybe TextMate?) that was in vogue during the peak of the Ruby on Rails era, but there was such a feeling of magic to using what was a fairly basic editor that still had syntax highlighting.
Was this feeling of magic purely because I was younger? Or perhaps we did peak in terms of the ergonomics of human-controlling-machine without too many aids?
Fighter pilots used to fly with skill and instincts, but now are assisted by all sorts of high tech equipment that has removed much of the "flying skill" and replaced it with "equipment skill". It's not that fighter pilots are worse now. I'm sure they are better at achieving the outcomes desired, while commanding much more complex equipment. But the perhaps the art of flying is less emphasized.
In the same way, perhaps the era of software engineering is changing too?