if I don't have hands on experience on designing large scale applications How can I be good at it? Any open source projects for the same or the books/videos..Thanks
In addition to the responses talking about getting better with software specifically, I think it's helpful to work get better about thinking in a systems way, generally. There are systems all around us, providing plenty of opportunity to practice systems thinking. For instance right now looking out my window at my backyard I see a drainage basin with frozen water there. I can start asking questions like "where did that water come from," "how long will it stay in that drainage basin", "where does it go when it leaves," etc. Or looking at the next house over and seeing the steam coming out a chimney -- where does that steam come from? What is the heat source? How did that heat get to that house? Etc.
System design is, primarily, a way of thinking -- identifying how given things are actually connected, thinking through how they are connected, what happens up and down those connections as things change. Software systems are just one particular example of this (and, as a bonus, when you start to include non-software parts, such as the humans actually using the software, your software system design will be stronger).
I'm sure others will cite this too, but I don't see it yet at the time I'm writing this, so for a specific book let me recommend Thinking in Systems by Donella Meadows[0]
Adding one more thing to my comment -- I think an important skill is the ability to put details in boxes and then kind of forget about them. Eg when thinking about a software system, being able to look at it with the appropriate "zoom level." The shape of the boxes, and how they relate to each other, are what you need to look at. Crucially, though, this doesn't mean details aren't important, it means they're not important _right now_. That's why I picked the analogy of the boxes; you will at some point need to open up the boxes again.
Or another way to try and explain it -- you can't keep everything in your mind's "RAM" -- when trying to think systematically, you need to offload the details onto disk for later retrieval. Don't throw them away! People who are "vision people" and brag about not sweating the details become really annoying to work with as a software engineer -- but you won't be able to grasp the system if your mental RAM is cluttered up with those details all the time.
Please don't bother with the whole charade of you have to do it to learn it. This kind of advice places more importance on people working at internet companies which have achieved scale. If you are at these companies then you have nothing to worry, experience will teach you. If you are not, follow @cppr's advice and read through DDIA. It will give you good enough introduction to data systems - not just databases. Then I would suggest to learn how message queues work since that part isn't covered in the book. Do a lot of mock designs and attack your designs with different load parameters - how much data do I store, how many machines (if more than one is necesary), can I do it simpler. Honestly if you can estimate storage and throughput well then you are fine. Pick up some distributed systems papers if you are very interested. Once you have these down you would be able to pick more topics yourself. Good luck. Keep learning.
One topic I haven't seen widely discussed in books but happens in the real world is back pressure. I think you're getting to this with "attack your designs", though I find it hard to predict.
Have you any thoughts, books or resources you turn to, to make sure the whole pipeline is balanced - or doesn't fall over when one part is overloaded?
You need to see what happens in you data processing chain. The pipeline is only "balanced" based on what Rps you want. Throughput calculations are easier to do. Latencies are harder to predict but honestly latency calculations without sensible benchmarks are horseshit.
Back pressure is refusing to let the queue unbounded or take in more requests than slots available. That's pretty much it.
I learned that it was wrong to casually point a firearm at someone before I had the experience of firing a gun. I also first learned to direct my visual focus to the front sights. Knowing that rule enabled me to then gain more nuanced skill and helped me avoid building bad habits.
After 8 years in the software industry, I dearly wish that I’d been able to find such guidance about software estimation when I was starting out. At this point, I’m going to need some sort of therapy to be able to give software estimates.
First, you didn’t actually learned those things, you trusted them.
Second, im not speaking about trivial common knowledge. There’s obviously always something useful to learn from theory in any area.
But its subtle things that make all the difference between average and really good in something, these ones you can’t really learn without gettyhands dirty and making own mistakes. Even though often it seems different looking backwards, but it’s nothing more than a curse if knowledge.
> But its subtle things that make all the difference between average and really good in something, these ones you can’t really learn without gettyhands dirty and making own mistakes.
I strongly agree here. It was the subtle things about breath control and muscle-tension control were what I learned while practicing marksmanship.
It is simultaneously true that a person starting out can sometimes see they need guidance to learn the 'trivial common knowledge' that others in a field are already familiar with. These things that seem 'trivial' to experts are things that help them avoid the "many ways to fail at getting good at something while doing it a lot."
The book System Design Interview is quite good and pragmatic. Personally, I found Designing Data-Intensive Applications less useful because it works on a much lower level. Like if you want to know how database internals work, then read DDIA. If you want to understand how to author a scalable API with layers of caching and distributed data storage, read SDI.
In addition to the great resources already mentioned here, we’ve been making hands-on videos and lessons at Exponent that show you system design in practice, with interview prep as the main focus but advice and discussions with people across the industry.
Check out some of the free videos on our site[0] and YouTube channel.[1]
I think that good analytical skills are prerequisite. You must know how each component/module will interact /interfere with other components. I don't know if there are any labs that might help you. Which existing application do you consider to be large scale application?
my advice is that you need to both consume and produce system design.
don’t rely on interviewing people to test yourself. you have to practice talking or writing and stop settling with the feeling that you just learned something and will have the judgement to know when to use xyz when problem abc comes in front of you.
i built enumerable.co so that i can evaluate myself, to practice producing the knowledge i’m consuming, and reach the edge of my understanding.
there’s no database and i built over a few days, so it’s just a single player way to figure out how much you really don’t know
Learn ho to modularize and how to keep cohesion high and coupling low. Avoid horizontal dependencies, don't mix abstraction layers, learn to isolate, apply tagless final.
Remember that large scale, complicated systems that work invariably start out as small simple systems that work, so build it virtually first, where you can break and re-build to your hearts content.
https://www.youtube.com/c/SystemDesignInterview
https://www.educative.io/courses/grokking-the-system-design-...