I had actually started to warm to the idea of systemd, thinking that it couldn't end up the same clusterf*ck of mismatched reinventions that didn't really solve any problems that pulseaudio was. I guess it's time to move to Debian/kFreeBSD and ignore this crap.
I'm not a Linux expert, but why are other services even allowed the possibility of writing to the kernel log? I seems that this should simply not be possible. If you make it possible, someone will surely abuse it sooner or later.
Systemd is soon to be integrated as the default init daemon for Ubuntu, one of the most user-friendly Linux distributions available.
It worries me that Sievers, and the systemd team aren't approaching testing with an almost paranoid attitude. Linux is finally getting a foothold in consumer desktops, it'd be a shame for people to go back to other operating systems purely because "it broke one day, something about systemd".
Yeah I'm considering switching to FreeBSD or even OpenBSD (if it would be possible to play some of my games on it). I don't want to switch really, but if systemd is going to be the default init system from here on and it continues to be plagued by bullshit like this then I don't really want to be bothered by it. (FYI I currently use Arch Linux, one of the early adopters of systemd).
Someone with better freebsd knowledge should chime in, but until then... It is my understanding that the launchd port was one guys google summer project, he worked on it for the summer, put it aside for a couple years and picked it up recently.
They're fucking around well beyond PID 1. Default binary log formats. This odd binary messaging bus with god awful reverse domain names, a shifting binary protocol format that injects itself into the kernels standard text-based messaging formats.
I find the whole dependency system completely bizarre, because to truly implement it, you have to hook into _everything_ and make it part of this odd binary mess they've created above.
I really don't understand the route they are taking, I see the system becoming less capable and more fragile with this type of engineering being encouraged.
> it'd be a shame for people to go back to other operating systems purely because "it broke one day, something about systemd".
FWIW, exactly this happened to me on an upgrade one day on another Linux distribution, after a few other headaches. The switch to systemd broke smbd/nmbd, at the time there were not yet tools for spitting the new binary log format into text, and trying to data mine that logging system to figure out what the hell broke was a real adventure.
I was so skeeved I immediately dumped that and fled to Debian, because it had a reputation for stability.
Imagine how excited I was when Debian voted to adopt systemd.
I, also, have started looking at the BSDs again recently. herbstluftwm looks pretty nifty.
I haven't used linux on a desktop in several years, but my experience was that "it broke many days, something about everything". Why should this one component be held on a pedestal as make-or-break for the whole OS? Is a broken init daemon worse than a broken xorg.conf from a desktop user's perspective?
right up until a ubuntu kernel update breaks the binary gpu driver, and you spend the next two days trying to figure out which binary or open source driver allows you to run everything the way you were at the beginning of the week. Repeat this every 6 months.
Why not give it another try now then? I have used it for years and, unlike windows, I never had to reinstall it just because something broke. And things only broke when I messed around with them, but it was always fixable. If you also use virtual machines or linux containers you will never break anything.
> "Hmm, a user adds to the kernel command line "debug" and systemd starts spitting out so much crap that the system doesn't boot anymore? That sounds like a major regression to me. Note this is a kernel command line, not a systemd command line. Userspace tools should not be using the same kernel parameters that are defined by the kernel. That's just broken and wrong.
> This bugzilla is the poster child of why people hate systemd and do not trust the developers that work on it."
> Userspace tools should not be using the same kernel parameters that are defined by the kernel. That's just broken and wrong.
My understanding is that this is not the issue and, in fact, the command line parameters are provided to userspace in the way that they are exactly so that applications can do this kind of thing.
Rather, the real issue is that the flag IS commonly used this way, and systemd's response to it being set was so aggressive that it prevented it from being used for anything else. A not-uncommonly-used system configuration, having "debug" set, was being broken unnecessarily and in an unintuitive way.
Yes. Linus explains that the kernel will ignore flags it doesn't know for that reason; however, systemd should not be doing anything with the debug flag as that is meant for the kernel. The systemd flag should be namespaced.
> No, we very much expose /proc/cmdline for a reason. System services are supposed to parse it [...] And yes, that does include "quiet" and "debug". Parsing them and doing something sane with them is not a bug, it's a feature.
I think as always the kernel should be made robust enough (in this case through rate limiting) to handle such abuse.
In a way this Sievers guy is right but he could have been more polite in asking the kernel devs for this feature. And he needs to co-operate by cutting down the log traffic until the kernel is fixed.
While it would be nice for the kernel to handle any kind of bad behaviour by default, it isn't always realistic.
I mean, it would be nice if the kernel by default could handle a simple forkbomb, but it doesn't without restricting your limits.conf fairly severely. You can blow up the system in a variety of ways that are outside of the kernel's responsibility. It's not up to the kernel devs to hack around your bad behaviour.
Please see my response to tdsamardzhiev. I have replied to some of your points there. I agree this borders on bad behaviour since the author knows the kernel does not have rate limiting in place. Ideally he should have put in a feature request with the kernel devs and should have self regulated the log traffic using some workaround (like checking for debug.systemd) until the kernel stepped up its game. If I were Linus I would have taken this up as a challenge (as I am sure he will too).
If you read through the kernel thread, it seems that's exactly what they're doing.
While systemd is obviously doing something stupid, no userspace program should be able to dump so much to the log that the machine becomes unusable. It seems like they were going to apply per-file descriptor limiting the way to they seem to limit in kernel logs on a per-log site basis.
I think it is. It is the kernel's job to protect itself from user-space mischief. It always has been. For example why are processes isolated from each other? Why does it guard against invalid memory/resource accesses etc? The kernel should be able to protect itself from denial of service attacks and must remain functional at all costs. A fork bomb is equally bad it is no excuse for the current failure mode.
BTW Linus agrees rate limiting is desirable here. This is the reason why I said systemd has to hold off its fire hose until the kernel can take care of it.
The kernel does quite a lot of work to prevent user-space programs from doing stupid stuff to each other, or to at least ameliorate the impact of such stupidity. One of the Unix philosophies has always been to prevent a runaway program from rendering the system inoperable.
systemd has apparently elevated itself somewhat above the typical "user space" level, but it's still not a bad idea to harden the interface between systemd and the kernel where possible.
Linus will not be merging any code from Kay Sievers into the kernel until Kay changes his pattern of not fixing problems in code he previously submitted.
It does become a problem when you have a system service developer who
thinks the universe revolves around him, and nobody else matters, and
people sending him bug-reports are annoyances that should be ignored
rather than acknowledged and fixed. At that point, it's a problem.
It looks like Greg has stepped in as a baby-sitter for Kay, and things
are going to be fixed. And I'd really like to avoid adding hacky code
to the kernel because of Kay's continued bad behavior, so I hope this
works. But it's really sad that things like this get elevated to this
kind of situation, and I personally find it annoying that it's always
the same f*cking primadonna involved.
Steven, Borislav, one thing that strikes me might be a good idea is to
limit the amount of non-kernel noise in dmesg. We already have the
concept of rate-limiting various spammy internal kernel messages for
when device drivers misbehave etc. Maybe we can just add rate-limiting
to the interfaces that add messages to the kernel buffers, and work
around this problem that way instead while waiting for Gregs fix to
percolate? Or are the systemd debug messages going to so many other
places too that that wouldn't really help?
Linus
They then went into a rate limiting discussion. Later in the thread it seems to have come up again, leading to the post 0x006A summarized.
Although I've heard a lot of others disagree, I like the fact that Linus' posts are straight and to the point. No sugar-coating or euphemisms, just telling the hard reality like it is.
...Is a good way to turn otherwise-skilled people with some problematic but fixable habits into people who refuse to work with you ever again.
Brutal honesty works with computers, but it's not always the best policy for keeping humans productive. Linus gets away with it because he's a celebrity, and moreover the right kind of celebrity. These comments would look very different if the public mockery was coming from, say, Steve Jobs.
He's been this way long before he was a celebrity coder. This is just how he runs Linux, and I think that is evidence enough that the process works for him.
The team itself might be a self-selecting group of people who can work with Linus. But, overall, I'd say that his brutal honesty works "well enough". It might not be a good way to run a corporate project, but for an OS kernel, the benevolent dictator model seems to work pretty well.
I wonder how the kernel groups at Apple and Microsoft work though... or how FreeBSD is organized... that'd be an interesting comparison.
Then again, a hard question is: Do we want developers with problematic habits working on the linux kernel?
I'd put the core linux system into a situation special enough to make it the correct decision to toss everyone but the best of the best of the linus-compatible developers out. It's not a pretty decision, but after a certain point of importance, I can understand that decision.
I tend to think of Linus as being similar to Steve Jobs - brilliant but arrogant srseholes whose products I enjoy using, but I would never work with them without some very serious compensation for putting up with the shit. Which is where this becomes a problem for Linus. Jobs could hey away with it, because he could pay people enough that they accepted the bad treatment. Linus, not so much.
It's clear that that the kernel dev guys and Kay have a history. I wonder if Kay would have been more open to accepting change requests to systemd without Linus's bad behaviour. It bears thinking about...
No you're not getting it. Linus is not like this often. And he's not like this to people who are innocent bystanders. Linus will only chew you up if you're an engineer he knows personally that should know better.
What you have to understand is that these people are not on Linus' payroll, so he has little direct power over them. Screaming at people is one of only two forms of power that Linux can exert, the other being withholding merging of code, which is even more extreme. And these incidents actually come up rather seldom when you think about how much code is submitted to the Linux kernel, which I think is a testament to how well Linus' management is working.
Because as with many things, it's a matter of perspective. For Kay, this code works as intended. So if you want him to change it, don't open a bug, discuss it on the mailing list. This does not seem like a completely unreasonable request on Kay's part. Note also that Kay does not get personal in this response. Compare and contrast with Linus then calling Kay a "fking primadonna". Now, whether you agree with Kay or not, do you really think that Linus's approach is going to get him what he wants?
Again, I have the greatest respect for Linus's achievements, but I personally feel that he uses the power those achievements have bought him to be a bully, and that gets no respect from me.
For anyone interested in what ensued, Greg KH submitted a patch to systemd to fix this issue. Linus then replied (among some other words) to Greg's confirmation of his patch with:
"It does become a problem when you have a system service developer who thinks the universe revolves around him, and nobody else matters, and people sending him bug-reports are annoyances that should be ignored rather than acknowledged and fixed. At that point, it's a problem.
It looks like Greg has stepped in as a baby-sitter for Kay, and things are going to be fixed. And I'd really like to avoid adding hacky code to the kernel because of Kay's continued bad behavior, so I hope this works. But it's really sad that things like this get elevated to this kind of situation, and I personally find it annoying that it's always the same f*cking primadonna involved."
This is quite confusing after reading Sievers' reply on Google+ (third comment on the post) [1]
> my last kernel patch is more than a year old, my last non-trivial kernel patch 2 years old. i stopped working on the upstream kernel "long ago" for reasons i cannot stand the attitude of these guys, i decided to work with grown up or funny, or grown up and funny people instead and i enjoy it a lot more. not sure what this childish blackmail attempt relates to.
I'm guessing Linus is referring to udev (which is broadly tied to the kernel). For no good reason, udev -- a separate daemon -- was pulled into systemd.
I don't get why there are so many egos in kernel and low-level development circles. I get that it's a bit more challenging than usual application development but the egos seem way too big relative to what's involved.
The reason is, in the typical contentious low-level development human interaction, one developer is an idiot with insane scrublord opinions, while the other isn't. So the first developer has an inflated ego, and the second doesn't, and people standing on the side who don't have a clue which is which. In this case, we have one who thinks it's reasonable for the debug flag to bring down the system, and the other is a sane reasonable person who's not going to tolerate this crash-the-planet type behavior any more.
I was looking at devkmsg_open() because of Linus's patch, and I noticed that user->prev initialization is missing in that function and a few other places where seq/idx gets reset.
This is at best a minor bug since it only affects the first line read from /dev/kmsg, but I thought I'd point it out if anyone wants to fix it.
- Kay claims a system not booting due to a casual kernel log flood from systemd is not an issue.
- Linus agrees the "debug" flag (along with other generic kernel flags) should be used by services like systemd, yet blasts Kay for using it, because it "usually isn't a problem", but here "it's a problem".
- Linus threatens not to accept kernel patches from Kay.
- Kay has not written a non-trivial kernel patch for over two years, and doesn't want to write any.
- Linus agrees rate limiting should be applied on the kernel's side, but that's not a solution somehow here, because reasons.
- Kay says Linus is involved in a "childish power play".
- Linus calls Kay a "primadonna who thinks the world revolves around him".
"- Linus agrees rate limiting should be applied on the kernel's side, but that's not a solution somehow here, because reasons."
Wouldn't it be better for the entity writing the messages to prioritize them and select the most relevant so that it doesn't write so many that the system cannot start, rather than let an entity that doesn't know the meaning of the messages arbitrarily discard most of them?
Rate limiting seems a last resort not a proper solution.
Good overview, but the kick off was that a simple bug in systemd was flooding the kernel log. Rather than ask for that bug to be fixed (possibly assuming it was intended behaviour) they asked for systemd to ignore a flag that is expressly designed to be read by things other than the kernel.
The fact that the original flood of messages was as simple bug renders the whole "territorial pissing" thing somewhat more pathetic than it appears at first.
https://bugs.freedesktop.org/show_bug.cgi?id=76935
I had actually started to warm to the idea of systemd, thinking that it couldn't end up the same clusterf*ck of mismatched reinventions that didn't really solve any problems that pulseaudio was. I guess it's time to move to Debian/kFreeBSD and ignore this crap.