No, it's not hard. Like most topics in software engineering there are 50+ years of pretty successful development, backed by science, backed by software from the 70s, backed by all seniority levels of engineers.
The problem nowadays is that people don't want to learn the proven stuff. People want to learn the new hip framework. Learning Kubernetes and React.js is much more fun than learning how actual routing in TCP/IP works, right?
The problem is that something can only be hip for 1-5 years. But really stable complex systems like a distributed network can only be developed in a time frame of 5-10 years. Therefore most of the hip stuff is unfinished or already flawed in its basic architecture. And usually if you switch from one hip stuff to another, you get the same McDonalds software menues, just with differently flavored ketchup and marketing.
So if you feel something is hard it might be because you are not dealing with the actual problem. For instance you might think about doing Kafka, and that's fine. But be aware that email is shipping more messages than Kafka and its doing it longer than Kafka.
If you don't know your topology it might be star or mesh. But it's still a topology or a mix of multiple topologies.
And if you develop a distributed system you really need to think about how your network should be structured. Then if you know which topology fits your use case you can go and figure out the way this topology works and what the drawbacks are. Star networks (like k8s) for instance are easy to setup but have a clear bottle neck in the center. A Bus (like Kafka) is like a street. It works fine until it is full, and there are sadly some activity patterns were an overloaded Bus will cascade and the overload is visible even weeks later (although you have reduced the traffic already), if you don't reset it completely from time to time.
It's not magic. You can look all of it up on wikipedia if you know the keywords. Also there is not a single "good" solution. It depends always on how well the rest of your system integrates with the pros and cons of your topology choice. And if you use multiple topologies in parallel you have a complexity overhead at some point, which is why working in big corps is usually so slow.
> The problem nowadays is that people don't want to learn the proven stuff. People want to learn the new hip framework. Learning Kubernetes and React.js is much more fun than learning how actual routing in TCP/IP works, right?
That's an awfully short-sighted comparison.
There are far more job offers for deploying and managing systems with kubernetes and to develop front-ends with React than there are for developing TCP/IP infrastructure. It's fun to earn a living and enjoy the priviledges of having a high salary, and the odds you get that by studying solved problems that nowadays just work are not that high.
If my livelyhood depends on doing bullshit then of course I will also do bullshit. But that doesn't stop me from applying for other jobs or from creating random HN accounts and complain about it. ;-)
I also found it's not bad everywhere though. If you treat the smart people around you well here and there you will get some opportunity to actually change something.
So what I also try to do is getting people out of the mindset that they are actually doing something reasonable when they do this bullshit circus to pay the rent. It's possible when you are really frustrated to spend a few hours here and there to learn the actual stuff instead of the new hip stuff. And over time you will thereby be able to solve more and more problems with actual solutions.
An example from my own life: At one point I really learned about Ansible, Chef, Puppet etc. Then I learned about actual configurations, improved my knowledge of ssh, bash-scripting etc, and in the end I wrote bash scripts that replaced all the Ansible I had used. The results where more flexible, the error messages more readable (thanks to set -x you were able to see what actually went wrong), it was a lot less than 1000 lines of code, and it was a lot of fun to do some actual problem solving for a change.
No, it's not hard. Like most topics in software engineering there are 50+ years of pretty successful development, backed by science, backed by software from the 70s, backed by all seniority levels of engineers.
The problem nowadays is that people don't want to learn the proven stuff. People want to learn the new hip framework. Learning Kubernetes and React.js is much more fun than learning how actual routing in TCP/IP works, right?
The problem is that something can only be hip for 1-5 years. But really stable complex systems like a distributed network can only be developed in a time frame of 5-10 years. Therefore most of the hip stuff is unfinished or already flawed in its basic architecture. And usually if you switch from one hip stuff to another, you get the same McDonalds software menues, just with differently flavored ketchup and marketing.
So if you feel something is hard it might be because you are not dealing with the actual problem. For instance you might think about doing Kafka, and that's fine. But be aware that email is shipping more messages than Kafka and its doing it longer than Kafka.
For instance topologies: There is no point-to-point. There's star, meshed, bus etc. See here: https://en.wikipedia.org/wiki/Network_topology
If you don't know your topology it might be star or mesh. But it's still a topology or a mix of multiple topologies.
And if you develop a distributed system you really need to think about how your network should be structured. Then if you know which topology fits your use case you can go and figure out the way this topology works and what the drawbacks are. Star networks (like k8s) for instance are easy to setup but have a clear bottle neck in the center. A Bus (like Kafka) is like a street. It works fine until it is full, and there are sadly some activity patterns were an overloaded Bus will cascade and the overload is visible even weeks later (although you have reduced the traffic already), if you don't reset it completely from time to time.
It's not magic. You can look all of it up on wikipedia if you know the keywords. Also there is not a single "good" solution. It depends always on how well the rest of your system integrates with the pros and cons of your topology choice. And if you use multiple topologies in parallel you have a complexity overhead at some point, which is why working in big corps is usually so slow.