Nicely written... but looking at the last section "Bringing it Home" I'd like to point out that if you were to, say, do a make -j<#cores> in the Linux source tree (or any other bloated GNU monstrosity) you'll get a 15-min load of well over the 70% desired :) But it's not a bad thing... it just means Firefox will run like crap for a while. Also, don't do that on your web server, which is probably what he was talking about anyway.
"it just means Firefox will run like crap for a while."
Put an "ionice -c 3" on that job and you probably won't notice the performance effect on Firefox anymore. (You probably don't need conventional "nice" because compile jobs tend to get their priorities dropped anyhow because they are using a lot of CPU without yielding, but dropping the scheduler that hint can still be helpful in some cases.)
(Annoyingly, unlike nice, ionice requires the specification of the class; I wish it would just default to -c 3 like nice has a reasonable default.)
IO Nice will just affect the io scheduling (obviously), but you can also do the same thing to the CPU schedulers.
schedtool -B -e ionice -c3 make -j10
That's the idiom I commonly use on long large compile jobs. means that anything else will always get the cpu or io time (maybe with some increased latency, which usually isn't too bad) that it needs while all idle time is taken up by the larger compile job. This makes for a very happy desktop system when upgrading (gentoo user here).
Seriously. Nobody _ever_ does "nice make", unless they are seriously repressed beta-males (eg MIS people who get shouted at when they do system maintenance unless they hide in dark corners and don't get discovered). It just doesn't happen.