Please don't discourage people without having the full view. There are many good low-level teams with software first people. At least in the bay area there are many exciting projects for people with this skill set. Self driving cars (Waymo, Tesla etc) , VR / AR headsets (Microsoft, Facebook etc.), Phones (Qualcomm, Google etc.) , Wearables (Apple, Fitbit) GPUs (Nvidia) and startups AirWare etc..

Here is a sample opening at Waymo - https://careers.google.com/jobs/#!t=jo&jid=/waymo/embedded-s...

Sure my experience may be skewed too but there are definitely great jobs with rewarding experiences of shipping tangible products for low level programmers. Not many people get the joy of shipping something your friends and family use and literally say "wow".

I fully agree with you. It's not just infrastructure but also devices and the surrounding ecosystem. I work in this space and let me assure you there are many jobs with GREAT pay.

I"ll go on and make another bold claim. Focusing on lower level stuff and systems concepts lay an excellent foundation for designing complex systems regardless of the language used. Once you understand how a program is executed from the ground up i.e. right from the program counter to page table lookups to cache hits to cache coherency, all abstractions are easier to deconstruct and understand. I work up and down the stack and this has been my, quite possibly just anecdotal, experience.

how much we talkin?

Writing and designing code that is mindful of latencies (Amdahl's law). Temporal / Spatial locality, avoiding needless copies etc.. This approach needs to start from design and also follow into implementing the code. Moreover, this is done without compromising on readability.

Umm, Amdahl's law is actually a negative result. It's saying that if you infinitely improve some magical hotspot that is 1% of your workload, the best you can only ever get to is 99%. If anything, Amdahl is telling us not to grind on trivialities.

If a system performance is profiled as being dominated by 80% A , 19% B and 1 % C. I'd focus on working on A first to get maximum gains. Amdahl's law gives you the backing as to why you should do this.

As a real world example, if an operation involves a network call and you see the RTT dominating the time. You may want to think of ways to avoid the call (caching etc..) if possible to get really good gains.

That's not how I understood it - I thought it was about the upper limit of theoretical parallelism.

But I'm an EE, so maybe I misunderstood some finer points?

An upper limit on the benefits. To me, an upper limit is a negative result. When Leon (in Blade Runner) finds out that he only has 4 years to live (an upper bound) he takes it pretty hard.

Amdahl:

  A fairly obvious conclusion which can be drawn at this
  point is that the effort expended on achieving high
  parallel processing rates is wasted unless it is
  accompanied by achievements in sequential processing
  rates of very nearly the same magnitude.

http://www-inst.eecs.berkeley.edu/~n252/paper/Amdahl.pdf

A point made when we read the paper was that Seymour Cray always made sure that his computers were also the fastest scalar computers even though they were sold as vector processors.

Thank you for the excellent explanation; I will consider it more thoroughly.

I think mathematicians consider an upper limit a positive bound - positive, in the sense of being well defined; you're using negative in the other sense? I actually like that quite a bit.

A self-driving car, as proposed by Waymo and many others, in no way relies just on GPS. These beasts have fusion of many sensors with built in redundancy.

Technology improves and evolves. If it didn't you wouldn't be so reliant on Google Maps. We convince people of progress by a safe track record. Google maps may fail occasionally but more often than not it gets you to your destination at the estimated time.

Yes self-driving cars can't be "more often than not" safe. But starting with smaller areas and showing the difference it makes on people's lives and the environment can be a good start.

"There are only two hard problems in CS : 1. Naming things. 2. Cache invalidation. "

"There are only two hard problems in CS : 1. Naming things. 2. Cache invalidation. 3. Off-by-one errors"

Only you got the order wrong. One of those distributed systems problems.

- Allow an unambiguous, never "played with", chronological timeline. Have a separate view that's your ML playground.

I disagree. The alternate to a vision you may not agree with is not having two visions. Use data to support one not both.

I loathe the non-chronological timeline. I abandoned the official Twitter app for Twidere, which lacks many features - such as being able to display full threads most of the time - but preserves my timeline in chronological order.

But you're totally right. For me, the "you might like X" or "here's what you missed" is garbage someone is flinging in my lap. But for other people, whose feed may be too chaotic or moving too fast, it's probably the perfect way to consume twitter.

I liked the article but I feel the CAP theorem is again misquoted; it often leads to misunderstanding. From the article - "but they would still confront the reality of the CAP theorem: your system can be consistent, available, or partition-tolerant, and you can only pick two"

The CAP theorem states that in the event of a network-partition you have to choose one of C or A. More intuitively, any delay between nodes can be modeled as a temporary network partition and in that event you have but two choices either wait to return the latest data at a peer node (C) or return the last available data at a peer node (A).

Edit: Switched C and A

Agree that the CAP theorem isn't as strict as that statement implies. The trade-off only occurs at certain points. Sorry it was a little unclear!

No need to be sorry. I liked the rest of the article. Thanks for that.

Edit: I also wrote my comment because the article is beginner friendly. A lot of beginners will get the right idea about CAP once they go through comments and then back again to your article.

Couldn't agree more. This kind of misunderstanding is also described in Brewer's paper "CAP twelve years later: How the "rules" have changed" (http://ieeexplore.ieee.org/document/6133253/).

This is a fantastic article that every engineer who touches a distributed system should read. The simplistic CAP understanding hurts systems by forcing the choice before it needs to be made; i.e. we can (sometimes) have our cake and eat it too.

Good point. I often heard that it makes no sense to NOT deal with network partitions and CAP as an "illusion of choice"[1]

[1] http://bravenewgeek.com/cap-and-the-illusion-of-choice/

What a wonderful explanation! You've cleared up hours of confusion I've experienced in my life, with just 2 sentences!

I'm motherf*ing amazzzzzed!!!!!!!

Thanks :)

AFAIK, this is how docker CoW works. Your read-only parts i.e. base OS / starting point of the image is always shared. As you make changes a per-container R/W layer records those changes. This link explains it well -

https://docs.docker.com/engine/userguide/storagedriver/image...

Okay, so on reading through that it looks like the answer to my question is "it depends":

* On-disk, the layered approach always saves space, as expected

* In memory, it depends on which storage backend you use: apparently btrfs can't share page cache entries between containers, while aufs/overlayfs/zfs can - I'm not sure if this is due to btrfs or docker's btrfs backend.

From looking at the relevant sources, it looks like (but I could be wrong if I looked in the wrong places) both exec() and dlopen() end up mmap-ing the executable/libraries into the calling process's address space, which should mean they just reuse the page cache entries.

So, if I understand correctly, as long as you pick a filesystem which shares page cache entries between containers, then you do indeed only end up with one copy of (the read-only sections of) executables/libraries in memory, no matter how many containers are running them at once. That's good to know!

Yes, as long as the back end for the container supports it, RO sections of shared libraries will be shared and pulled from the same cache when available. The functionality that enables shared memory (and L* cache access in general) is implemented in silicon in the MMU so as long as the backend properly updates the page tables, you can share pages across any container or VM (except when prohibited by other virtualization hardware).

It's not something that happens automatically though because each kernel is responsible for telling the MMU how it should map memory for its child processes only. Any cross container page sharing has to be implemented at the host level where the kernel has unrestricted access to all guest memory.

This is essentially same as processes sharing (via Page Table mapping) one .so file AKA Dynamic Linking :).

"Seriously? In a populace of a billion the death of 100 people for such a huge economic move is close to nothing."

This might be a stat for you but if it was your kin maybe you won't feel the same way.

It's not a stat for me. I do sympathize. But stretching this argument to say that demonetization is a failure is ridiculous. In that way, in India, the deaths due to accidents are close to 1,37,000 in 2013 alone. Should we call transportation a failure and put a halt to every mode of transportation (be it autos, cars, buses, trains, planes etc) just because of this statistic or should we work towards improving the system?

I compared demonetization carried out by India with demonetization carried out by other countries. When I see that there were large-scale riots and coup d'état in other countries but no such thing happened in India I will definitely take this as a success. You want to have a narrow negative perspective of 100 people dying you can continue to have that. I have a wider positive perspective that no large-scale riots or coup d'état happened that would have sent the Indian economy in a downward spiral or worse: civil war.

"Should we call transportation a failure and put a halt to every mode of transportation (be it autos, cars, buses, trains, planes etc) just because of this statistic or should we work towards improving the system?"

Are you actually comparing currency to transportation ? People have a choice in the mode of transport. Your villager however can't use bitcoins because his Rs.500 note isn't legal tender.

"I have a wider positive perspective that no large-scale riots or coup d'état happened that would have sent the Indian economy in a downward spiral or worse: civil war."

Your standards for a successful policy are shockingly low. And stop lying, you have no sympathy, you just called a 100 deaths as having a "narrow perspective".

Do you also disagree with the countless economists that have criticized the move based on theory and facts. Or is that "foreign propaganda" for you ?

Do you really think people will install and use PayTm apps overnight in villages ?. Was there any research done on mobile literacy or making things easier in rural areas ?

Also, you wish to ignore facts and reasoning posted by economists all in the name of "disruption". And FYI The Planning Commission of India AKA "Policy Pundits" consists of renowned economists.

Mobile literacy? Was such a research done before the Mobile and Telecom revolution? Spare me your ignorance please.

Forget the Planning Commission of India, we have had the worst experience with a Prime Minister, who was an Economist, and had no spine to take on the corrupt or even implement his own policies. India practically lost 10 years of it's growth to this useless economist. So no please, we don't need more economists. We have more than enough Economists. We need someone who can take tough policy decisions.

The 90 reforms were by the same person you are criticizing.

Yes and I already acknowledged it when I said he was "possibly the best economist India had". How did I qualify he was "the best economist"? Because of his reforms in the 90s which were spearheaded by the then Prime Minister Narasimha Rao. Would he have been able to conduct the same reforms if he was the Prime Minister instead of Narasimha Rao? Never.

As I said earlier, economists have a place. He was great as a Finance Minister and RBI Governor. But the moment he became a policy decision maker (rather than just a policy maker) he failed miserably. Do you understand this basic difference? This is the reason I say that economists cannot take policy decisions. They can make the right policies if they are dictated by someone higher up the chain but are incapable to take the same decisions if they are given a chance.

The evidence for that is the GST reform. GST was the brainchild of Manmohan Singh and his Planning Commission when he was the Prime Minister for 10 years. But he failed to get the damn reform passed in the Indian Parliament mainly because he did not know or understand political maneuvering. How is it that the same GST reform, which was Manmohan's brainchild, passed by Narendra Modi (who is his opponent) without any troubles?

Google works with a subsidiary of the Railways called RailTel. A lot of the infrastructure / backbone was supplied by RailTel which allowed the project to move fast.