On Systemd and devops
If it's not broken , don't fix it.
Those who don't understand Unix are doomed to reinvent it, poorly
Complexity is the enemy of reliability.
Are some of the more frequently heard arguments in the systemd discussion. Indeed I see and hear a lot of senior Linux people react openly and probably way to late against the introduction of systemd in a lot of our favorite Linux distributions.
To me this is a typical example of the devops gap. The gap between developers writing code and operations needing to manage that code on production platforms at scale.
Often developers writing code that they think is useful and relevant while they are not listening to their target audience , in this case not the end users of the systems but the people that are maintaining the platforms. The people that work on a daily base with these tools.
I have had numerous conversations with people in favor and against systemd, till today I have not found a single general purpose use case that could convince me of the relevance of this large change in our platforms. I've found edge cases where it might be relevant. but not mainstream ones. I've also seen much more people against it than in favor. I've invited speakers to conference to come and teach me. I've probably spoken to the wrong people,
But this is not supposed to be yet another systemd rant.. I want to tackle a bigger problem. The problem that this change and some others have been forced upon us by distributions that should be open, and listen to their users, apparently both Debian and Fedora/RHEL failed largely but somehow fail to listen to their respective communities. Yes we know that e.g Fedora is the development platform and acts as a preview of what might come up in RHEL and thus CentOS later , but not everything eventually ends up in RHEL. So it's not like we didn't have an 'acceptance' platform where we could play with the new technology. The main problem here is that we had no simple way to stop the pipeline, it really feels like that long ago Friday evening rush deploy. Not like a good conversation between developers and actual ops on the benefits and problems of implementing these changes. This feels like the developers of the distributions deciding what goes in from their own little silo and voting in 'private' committee.
It also feels like the ops people being to busy to react, "Someone else will respond to this change, it's trivial this change is wrong , someone else will block this for sure",
And the fact that indeed Operating System developers, like Fedora and Debian friends kinda live in their own silo. (specifically not listing CentOS here..)
So my bigger question is .. how do we prevent this from happening again.. how do we make sure that distributions actually listen to their core users and not just the distribution developers.
Rest assured, Systemd is not the only case with this problem .. there's plenty of cases where features that were used by people, sometimes even the something people considered the core feature of a project got changed or even got ripped out by the developers because they didn't realize they were being used, sometimes almost killing that Open Source project by accident.
And I don't want that to happen to some of my favourite open source projects ..
Comments
#1 Michael : It is not gonna happen
To be honest, it is not gonna happen for a while. Customer service and marketing are specific skills for a reason, and as long as we will think this is less useful than just having technical people in the project, things will not improve. There is no proper feedback loop for users to people who do the work. So in the end, people on the distribution side have to take the raw feedback from users, and having been there, I can tell you that
1) there is too much feedback to handle ( I counted that spending 5 minutes on every mail received the first day of mageia project would have required me to work 5 days non stop, 8h per day ). It is fine if you do that full time, but in a free software project, either you answer to everybody, or you produce the software. So you need to do some sampling, which bring to the 2nd point
2) some users are just abusive. Sometime, that's because they do not realize. Speaking by mail, in a foreign language, not being sure, all are factor that result into misunderstanding. And sometime, no. Sometime, you just see people forgetting a human is on the other side. This post from Sam Kottler is quite spot on : http://shk.io/2014/08/27/open-source/
3) since you can't listen to everybody due to lack of ressources, you have to select. And since you will select based on who is shouting the most often or who is making people react, and since all of them will say that they are more important, so they must be listened, you end with a perception that part of your users are highly annoying, while they are not when you stop, step back, and reflect.
So the net result is that after a while, you either stop to do anything, or you start to drop all empathy for people. The astute reader will see that this also look closely to the symptom of burnout...
I know very few people wanting to be front line support. but with free software, when you say "listening to users", that's exactly what happen. And while at least people on support or marketing can avoid being too engaged, since that's mostly just a job, this is not the case for free software developpers or distributions makers. For you, that's just a software. For them, they sometime put lots of efforts in it.
Being in ops, you are protected from that ugly side, you do not see people complaining on how they would do better the network and this kind of stuff. But if you want to see what it mean and how it work, I would suggest to try and participate to a distribution. Even better start by forums, read stuff there, and see how exhausting it can be.
Then maybe this will bring a idea that others didn't have. Maybe we need people to do that job, so the idea would be to recruit people for that. Maybe we need better software to handle the feedback. Or maybe we should start to nurture a community of people to bring more than technical stuff, by bringing more diversity.
#2 Kenneth Martins : I can understand your point
I can understand your point of view, I also ask myself the same questions...
#3 Robyn Bergeron : I have a thousand things to
I have a thousand things to say... surprising, right? :) ... but better suited to a blog post than in a comment. So I'll do that in the next day or so. :)
#4 dirk dierickx : redhad & others
I can somewhat understand that RH is going this way, they developed it and seem to be on some mission to replace the unixy parts of linux with something else (for unknown reasons).
I don't get it why all other distros are jumping on this bandwagon. Why are debian, arch, ... doing this? Ok, Gnome3 also builds further on systemd, but;
1. gnome 3 is of no interest on the server
2. less people would cry foul if gnome 3 would no longer be available compared to the number of people complaining about systemd being the default init.
#5 Paul Cobbaut : Debian
I am still hoping for Debian to make a systemd-less version of their distro... idle hope probably.
p