SystemD Service Hardening
Much better article with very real tips about what options to try than yesterday's (weirdly flagged/dead?) post on the topic. Which while I really enjoyed lacked substance; I was in the comments trying to provide a more useful basis with some real examples, but this is an exemplary list of awesome ways systemd can easily quickly readily provide aassive boost to isolation & security. Great write up!
Yesterday's, just in case: https://us.jlcarveth.dev/post/hardening-systemd.md https://news.ycombinator.com/item?id=44928504
Maybe fix the certificate issue on the site. Some browser doesnt event let one go forward with a bad cert.
Why don't distros flip more of these switches? Are there cons of being more aggressive with these settings? It's really a lot for many people to tinker with.
> Are there cons of being more aggressive with these settings?
well, the con is you might unknowingly break some setups. take NetworkManager: after tightening it down, did you check both IPv4 and IPv6 connectivity? did you check that both the `dns=systemd-resolved` and `dns=default` modes of operation (i.e. who manages /etc/resolv.conf) work? did you check its ModemManager integrations, that it can still manage cellular connections? did you check that the openvpn and cisco anyconnect plugins still work? what about the NetworkManager-dispatcher hooks?
> Why don't distros flip more of these switches?
besides the bit of "how many distro maintainers actually understand the thing they're maintaining well enough to know which switches can be flipped without breaking more than 0.01% of user setups", there's the bit of "should these flags be owned by the distro, or by the upstream package?" if the distro manages these, they'll get more regressions on the next upstream release. if the upstream sets these, they can't be as aggressive without breaking one or two of their downstreams.
This question is sorta similar to "Why don't distros enable restrictive MAC policies by default"
Maintainers _could_ take the time to lock down sshd and limit the damage it can do if exploited, but there are costs associated with that:
1. Upfront development cost
2. Maintenance cost from handling bug reports (lots of edge cases for users)
3. Maintenance cost from keeping this aligned with upstream changes
It would be cool to have a repo with suggested hardening for common services, since there's so many different hardening options. One of the things you might notice from lots of users using common suggestions, is that the permissions often need to be opened up more than you'd think, to support edge cases.
> It would be cool to have a repo with suggested hardening for common services
From packaging stuff for nixpkgs, a distro that often is without upstream support, it is usually very useful to look at how mainstream distro package services.
Those hardening steps also tend to be well tested even if sometimes a bit lax. If you want to find out how, e. G., postgresql can be hardened, consider looking at the Debian, Ubuntu and/or RHEL packages as a starting point.
Distros don't usually do security hardening, unless the distro is security-specific. They slap something generic on like AppArmor or SELinux and call it a day. (This article is the proof of that... all the default services are not hardened). Usually this is a good thing, as it prioritizes usability, and lets the user harden as they wish.
Thanks for sharing this. It looks like you can also use "systemd-analyze" with the "--user" flag to inspect systemd user units as well ("systemd-analyze --user security"). I've started using systemd more now that I've transitioned my containers to Podman, and this will be a helpful utility to improve the security of my systemd unit/container services.
Another great security feature systemd provides is credential management[1], which allows you to expose credentials to an application in a more secure way than, say, an environment variable or a file in the filesystem.
When Vault is not available, if I’m working on a side project, for example, that’s what I always go for. Even wrote a small Go package[2] to get said credentials when your application is running inside a service with that feature.
I see we've decided to take the path of nodejs and npm, where 2-liners deserve their own package.
dir, err := os.Getenv("CREDENTIALS_DIRECTORY")
cred, err := os.ReadFile(filepath.Join(dir, "name"))
I thought the go culture was that dependencies are bad, and abstractions (i.e. any function calls) are confusing and bad, so it's better to just inline stuff like this and write it fresh each time you need it.
For what it's worth this will not work properly, you have different environment variables for user and system units. Proper error handling and graceful fallback for these cases are probably worth a module here (though it could be a 10-20 liner instead too).
I was writing the same few lines for every project, so why not make it its own package?
For my projects I can just include the dependency, as I wrote it and don’t mind using it. Other people can copy it instead, since the proverb goes “a little copying is better than a little dependency”.
Editor macros/snippets?
Didn’t think of that ¯\_(ツ)_/¯
... how exactly does it prevent inheritance to forked children?
And that's something that's impossible to do with old init scripts, that are all unique in their way and not uniform at all.
You can ofcourse achieve all these things in your init scripts which are unique in their way and not uniform at all, just to give credit where credit is due. But systemd makes it practical to use our beloved kernel and it's features in an uniform and standard way... :)
I started my Linux journey so late I can't imagine living without systemd, the few systems I've encountered without systemd are such a major PITA to use.
I recently discovered "unshare" which I could use to remount entire /nix RW for some hardlinking shenanigans without affecting other processes.
systemd is so good, warty UX when interacting with it but the alternative is honestly Windows in my case.
systemd is very complex software. Alternative is very simple software with complex scripting which will reimplement parts of systemd in a buggy way (and that's not necessarily a bad thing). systemd probably is inspired by Windows and other service managers, while old sysv init is just a tiny launcher for script tree.
Just an example of systemd limitation is that systemd does not support musl, so if you want to build a tiny embedded sysroot, you already have some limitations.
>Just an example of systemd limitation is that systemd does not support musl, so if you want to build a tiny embedded sysroot, you already have some limitations.
OpenEmbedded has carried a patchset to build systemd against musl for use in Yocto for a long time.
postmarketOS already got approval from Poettering to make a musl-linked systemd more officially supported.
So you might say that... any sufficiently complicated init system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of systemd?
systemd is heavily inspired by macOS' launchd.
More information at https://news.ycombinator.com/item?id=2565780
Not really, AIUI pottering just thought launchd's socket activation/inetd like functionality was neat: https://0pointer.de/blog/projects/systemd.html. Upstart is more of a direct ancestor:
> Why didn't you just add this to Upstart, why did you invent something new?
> Well, the point of the part about Upstart above was to show that the core design of Upstart is flawed, in our opinion. Starting completely from scratch suggests itself if the existing solution appears flawed in its core. However, note that we took a lot of inspiration from Upstart's code-base otherwise.
> If you love Apple launchd so much, why not adopt that?
> launchd is a great invention, but I am not convinced that it would fit well into Linux, nor that it is suitable for a system like Linux with its immense scalability and flexibility to numerous purposes and uses.
launchd is horrible though, the folks complaining about systemd would be up in arms if they had to write poorly typed XML key/value files
And Solaris SMF. There basically seem to be ~three generations of unix init:
1. Agglutination of shell scripts
2. "Oh wow, this is getting annoying"-phase: Wrapper for scripts (SRC SMC openrc etc pp)
3. Service supervision daemons (SMF, launchd, systemd)
Probably. I don't think systemd is a mere "Service supervision daemon", but I'm not in the mood for a can of worms today.
Yeah, I'd probably call systemd something like "an event- and graph-based orchestrator."
Aside, I'm quite surprise that systemd won out, given all the hate for so many years. You got nearly every forum and HN thread that even tangently mention systemd, flooded with people proclaiming systemd to be the spawn of the devil "forced down their throat". At the same time I noticed that a lot of regular Linux users (like developers that just want to deploy stuff) seemed to like systemd unit files. I guess all those complainers, however vocal, were neither the majority, nor the people who actually maintained distributions.
This feeling is particular striking for me, because I once worked on a Linux project with the aim of improving packaging and software distribution. We also got a lot of hate, mainly for not being .deb or.rpm, and it looked to me as if the hate was a large reason for the failure of the project.
Systemd won because for all the deserved hate it solves real problems that nobody else was really trying (or their attempts were worse). There are some really good points to systemd - you just have to takes the bad with it.
My feeling, without evidence, is that a lot of the hate is vocal people who don't like this thing that does stuff differently to what they're used to. I get the feeling, I still understand /etc/init.d and runlevel symlinks and /var/log better than systemd, but that's because I have many more years of experience interacting with it, breaking it, fixing it. Whenever I have to do stuff with systemd it's a bit annoying to have to go learn new stuff, but when I do I find it reasonably straightforward. Just different, and most likely better.
A lot of the hate was because the main developers interactions wiht just about everyone was horrible, the code was opaque at best most of the time and constantly changed, scope creep, and the feeling that it was a massive power grab by Red Hat.
It can be argued that it didn't solve very many problems and added a huge amount of complexity.
One of my earliest memories of using systemd involved logs being corrupted. journalctl would just say the file is corrupted but wouldn't even print the name of the file -- I had to resort to strace. That left a real bad taste in my mouth.
Having the backing of RedHat certainly didn't hurt... I don't care either way, although I still think OpenRC style scripts are much easier to debug and sometimes have more elegant solutions (templates vs symbolic links).
I can't say I know what OpenRC really does, but at it's core systemd is just a glorified script runner as well? You have ExecCondition, ExecStartPre, ExecStart, ExecStartPost, ExecReload, ExecStop, ExecStopPost which are all quite self explanatory, this alongside a billion optional parameters which I wouldn't know how to implement in scripts without endless hours of dread.
I remember an old Debian machine with /etc/init.d/something [start|stop|reload|restart] but I can't recall being able to automatically restart services or monitor status easily. (I didn't speak $shell well back then either)
systemd tries to avoid scripts as much as possible.
/etc/init.d/whatever were all shell scripts, and they all had to implement all the features themselves. So `/etc/init.d/foo restart` wasn't guaranteed to work for every script, because "restart" is something that each script handled individually. And maybe this one just didn't bother implementing it.
There's no good status monitoring in sysV because it's all convenience wrappers, not a coherent system.
I've been reading through how many NixOS modules use systemd units and there's a lot of scripts being executed, the final line execs the service (if there is one, NixOS uses systemd for maintenance tasks, refreshing certificates and many more things). While NixOS doesn't speak for the broader community what I'm trying to say is that it can execute anything, if that's a script or a daemon doesn't matter as long as it works for you.
Thanks for the sysV explanation, it sounds worse to me.
I use runit for some non-system-level stuff. It's extremely simple, possibly too simple. It doesn't manage load order - if a dependency hasn't loaded yet, you just exit the script and the service manager tries again in 2 seconds. Service scripts are just shell scripts.
There are two ways to design a system: so simple that it has obviously no bugs, and so complex that it has no obvious bugs.
That's because systemd knew who the target users of it were: people making distributions, and professional users with little desire to be woken up at 3 AM to troubleshoot a stuck PID file.
Most of the complainers weren't really relevant. They weren't making the decisions on what goes in a distro, and an init system is overall a fairly minor component most users don't use all that often anyway.
> This feeling is particular striking for me, because I once worked on a Linux project with the aim of improving packaging and software distribution. We also got a lot of hate, mainly for not being .deb or.rpm, and it looked to me as if the hate was a large reason for the failure of the project.
I think that's a good deal trickier because packaging is something a Linux user does get involved with quite often, and packaging systems don't mix well. A RPM based distro with some additional packager grafted on top is a recipe for disaster.
Still, I think it's also a case of the same thing: sell it to the right people. Find people making new distros suffering problems with DEB and RPM and tell them your tool can save them a lot of pain. The users can come in later.
> Still, I think it's also a case of the same thing: sell it to the right people. Find people making new distros suffering problems with DEB and RPM and tell them your tool can save them a lot of pain.
To quote one of my favorite Clone Wars episodes: Fifty tried, fifty died [1].
There have been so, so many attempts at solving the "how to ship binary builds for Linux" question... both deb and rpm have their good and their bad, and on top of that you got `alien`, flatpak, Docker images, the sledgehammer aka shipping everything as a fully static binary (e.g. UT2004 did this) or outright banning prebuilt binaries (the Gentoo and buildroot way). But that's not the actual problem that needs solving.
The actual problem is dependency hell. You might be lucky to be able to transplant a Debian deb into an Ubuntu installation and vice versa, or a SLES rpm to RHEL, but only if the host-side shared libraries that the package depends on are compatible enough on a binary level with what the package expects.
That suddenly drives up the complexity requirements for shipping software even for a single Linux distribution massively. In contrast to Windows, where Microsoft still invests significant financial resources into API-side backwards compatibility, this is not a thing in any Linux distribution. Even if you're focusing just on Debian and Ubuntu, you have to compile your software at least four different times (one each for Debian Stable, Debian Testing, Ubuntu <current rolling release> and Ubuntu <current LTS>), simply because of different versions of dependencies. Oh and in the worst case you might need different codepaths to account for API changes between these different dependency versions.
And even if you had some sort of DSL that generated the respective package manager control files to build packages for the most common combinations of package manager, distributions and actively supported releases of these, there's so, so much work involved in setting up and maintaining the repositories. Add in actually submitting your packages to upstream (which is only possible for reasonably-ish open source packages in the first place), and the process becomes even more of a nightmare.
And that's all before digging into the topics of autotools, vendoring (hello nodejs/php/python ecosystems), digital signature keyrings, desktop manager ecosystems and god knows what else. Oh, and distribution bureaucracy is even more of a nightmare... because you now have to deal with quirks in other people's software too, and in the worst case with a time span of many years of your own releases plus the distribution release cadence!
Shipping software that's not fully OSS on Linux sucks, shipping closed source software for Linux sucks even more. Windows has had that sort of developer experience figured out from day one. Even if you didn't want to pirate or pay up for InstallShield, it was and is trivial to just write an executable, compile it and it will run everywhere.
[1] https://starwars.fandom.com/wiki/Mystery_of_a_Thousand_Moons
IMO, binary compatibility on Linux isn't really solvable. There's just a thousand tiny projects that make up the Linux base that aren't on the same page, and that's not about to change.
I do think packaging can be improved. I hate almost everything about how dpkg works, it's amazing. So I'm squarely in the RPM camp because I find the tooling a lot more tolerable, but still surely further improvements can be made.
I only started packaging relatively recently. Using OBS definitely made things easier, but it's crazy how much nicer RPM is than dpkg. So much better to have more-or-less everything inside a spec file with macros, versus dpgk's mess of static, purpose-specific files.
Anyway, the ecosystem stays heathy because of code contributions. So what’s the point of binary compatibility (from the point of view of the people actually making Linux work: Open Source developers and repo maintainers)?
> So what’s the point of binary compatibility (from the point of view of the people actually making Linux work: Open Source developers and repo maintainers)?
Want to see Linux on the desktop actually happen? Then allow a hassle free way for commercial software that is not "pray that WINE works good enough" aka use win32 as an ABI layer.
Of course we can stay on our high horses and demand that everything be open source and that life for closed source developers be made as difficult as possible (the Linux kernel is particularly and egregiously bad in that perspective), but then we don't get to whine about why Linux on the desktop hasn't fucking happened yet.
I don’t really know what the point of this “Linux on the desktop” event would be, or even what it is. (Clearly it isn’t just Linux on desktops, because that’s been working fine forever).
The whole point of my comment was to keep in mind the incentives of different sub-groups. If “Linux on the desktop” doesn’t benefit the people that make Linux work, I don’t see what the big deal is.
Great article. I really appreciate the list of properties and the "check the man pages, good luck" advice. systemd is really a great piece of software I would enjoy deploying on my hosts!
Nitpick and title correction: The proper spelling of systemd is systemd, not SystemD. According to their brand page:
Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd.
Interesting! I’ve mostly seen it written as systemD. I wonder why that seems to be so popular…
Many of the more juvenile systemd haters used to deliberately spell it that way.
As an insult, it was rather less successful than the "Micro$oft" / "Slowaris" / "HP-SUX" mockery from the 1990s - but it did manage to sow enough confusion that that it still pops up regularly today, even in contexts that are otherwise neutral or positive about it.
That’s funny, I’m not even sure what the D is supposed to expand to, in that insult. What a silly and lame thing to use as an insult.
I’ve been using it because having some random letter capitalized seems to be the totally unsurprising for this sort of plumbing software. (And by plumbing, I mean: very useful and helpful boring stuff that deals with messy problems that I’m happy not to care about, just to be clear that I mean it positively, haha).
Nice tip on debugging syscall issues!
I'd hate to restrict Docker like that - depending on what you run inside of Docker, it would be very hard to narrow it down to the right security tuning settings. In that case, it's actually safer and more predictable to run it in systemd (arguably).
What would be so hard about it? Also, this is not docker--it's podman. Which has a much simpler execution model than Docker. With it, it shouldn't be any harder to narrow down what the problem is, compared to running a non-containerized service.
color me surprised to see an article mostly constructive and positive about systemd using the wrong capitalization of the project name.
Normally the rule is that people mis-capitalizing the name are usually critical of the project.
It's systemd, not SystemD
So, no more DEFAULT-DENY for SystemD?
The unreasonable effectiveness of writing a security scanner. People will do anything it takes to make the scanner give a perfect score, regardless of whether it makes sense.
these Hardening variables have been discussed some years back[1].
this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user (I've yet to see maintainers embrace shipping locked down files). Maybe they will? But this same approach hasn't worked with apparmor so why should it work with systemd? Who will do the job?
If you consider apparmor maintainers provide skeleton-templates in many cases that will make the parser stop complaining. ("look I have a profile so apparmor shuts up, but don't take too close a look OK")
Then there is firejail, which some argue[2] is snake-oil considering the high level of administrative glue compared to its massive attack-surface (also it's a setuid binary).
I didn't mention SElinux since I don't know a single person who had the joy (or pain depending on perspective) of working with it. But again, seems the expectation to implement security with it is shifted to the user.
> this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user (I've yet to see maintainers embrace shipping locked down files).
https://fedoraproject.org/wiki/Changes/SystemdSecurityHarden...
It says the change was dropped? I guess at least they tried.
> Change is dropped because of inactivity. Owner is welcome to resubmit if the work is picked up again.
thanks for the link, this is great news.
> this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user
Maybe your point is that this isn't done by the vendor in practice. And I'm sure there's room for lots of improvement. However, one of the great things about how systemd units can be provided by the vendor and seamlessly tweaked by the administrator is that the vendor (i.e. packager and/or distro) can set these up easily.
There definitely are packages that ship with locked-down files. Tor and powerdns (pdns) are two off the top of my head.
→ Overall exposure level for pdns.service: 1.9 OK
→ Overall exposure level for tor.service: 7.1 MEDIUMI think it should be done by the maintainer of the software not by the distro. My concern is that these features are available since at least 5 years and it has not yet caught on (regardless of what this blog article recommends).
It would be great to see it implemented but for now at least on Debian/sid the situation is as follows:
UNIT EXPOSURE PREDICATE
ModemManager.service 6.3 MEDIUM
NetworkManager.service 7.8 EXPOSED
alsa-state.service 9.6 UNSAFE
anacron.service 9.6 UNSAFE
atop.service 9.6 UNSAFE
atopacct.service 9.6 UNSAFE
avahi-daemon.service 9.6 UNSAFE
blueman-mechanism.service 9.6 UNSAFE
bluetooth.service 6.0 MEDIUM
cron.service 9.6 UNSAFE
dbus.service 9.3 UNSAFE
dictd.service 9.6 UNSAFE
dm-event.service 9.5 UNSAFE
dnscrypt-proxy.service 8.1 EXPOSED
emergency.service 9.5 UNSAFE
exim4.service 6.9 MEDIUM
getty@tty1.service 9.6 UNSAFE
irqbalance.service 1.2 OK
lvm2-lvmpolld.service 9.5 UNSAFE
polkit.service 1.2 OK
rc-local.service 9.6 UNSAFE
rescue.service 9.5 UNSAFE
rtkit-daemon.service 7.2 MEDIUM
smartmontools.service 9.6 UNSAFE
systemd-ask-password-console.service 9.4 UNSAFE
systemd-ask-password-wall.service 9.4 UNSAFE
systemd-bsod.service 9.5 UNSAFE
systemd-hostnamed.service 1.7 OK
systemd-journald.service 4.9 OK
systemd-logind.service 2.8 OK
systemd-networkd.service 2.9 OK
systemd-timesyncd.service 2.1 OK
systemd-udevd.service 7.1 MEDIUM
tor@default.service 6.6 MEDIUM
udisks2.service 9.6 UNSAFE
upower.service 2.4 OK
user@1000.service 9.4 UNSAFE
wpa_supplicant.service 9.6 UNSAFE> I think it should be done by the maintainer of the software not by the distro
Why would you say that? I would agree that the developer likely has better insight into what the software needs. But the security boundary exists at the interface of the application and the system, so I think that both application devs and system devs (i.e. distros) have something to contribute here.
And because systemd allows for composition of these settings, it doesn't have to be a one-or-the other situation--a distro can do some basic locking down (e.g. limiting SUID, DynamicUser, etc.), and then the application dev can do syscall filtering.
In any case, I agree that I'd like to see things get even more locked down. But it's worth remembering that, before systemd, there was basically no easy-to-use least-privilege stuff available beyond Unix users and filesystem permissions. The closest you had (afaik) was apparmor and selinux. In both of those cases, the distro basically had to do all the work to create the security policy.
Also, n.b., that pdns.service I noted is provided by PowerDNS themselves.
imho adding a few lines to the systemd unit file is waaaaaay easier than apparmor
and really, these should be written by the developers, not distro maintainers
poking around on my Ubuntu machine, a few daemons have some hardening, chronyd looks pretty good
It would be nice to be possible to do the hardening of services via allowlisting instead. E.g. AllowNothing=true and then start adding what is allowed to make the service function.
I think that pledge[0] offers that functionality
Quoting https://brand.systemd.io/#:~:text=Yes,%20it%20is%20written%2...
"Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. [...]"
At some point, if you have to write articles about the proper spelling of the name, maybe you should just accept the alternative names as well. Also looking at you Datadog and Cloudflare. (The employees of the second one are especially allergic to CloudFlare for some reason)
We tried very hard to convert FastMail to Fastmail and... it's been about 90% successful but there's definitely a bunch of things out there spelled the old way. We just joke about BIG M occasionally.
[TIL - it's not even as old as me!] https://australianfoodtimeline.com.au/1978-launch-of-big-m/
And I tried telling people "news-teller" (or "news teller") is a better word than "herald", and I got told off, so I dunno.
By the way, I love that the OP used the Text Fragment (Scroll-to-Text Fragment) feature. I hope it is going to catch on more, quite helpful / useful.
I guess you could say the same about pronouns.
Anyway, why would someone use the spelling with upper D beats me. It's not proper English.
this is ... very low quality, and very low density. why did someone feel it was useful to post to HN?
Could you please expand on the "very low quality" claim? What's missing for you here? How would you raise a bar on this piece to make it more of a high quality, security focused content? Genuine question.
The security issues with Linux will only be solved once lennart pulls the entire userland into the systemd repo
Best systemd hardening is switching to OpenRC or runit
Do you have any references for doing similar system hardening under either of those?
No, switching to Qubes OS is the real hardening.
An unbootable system is indeed harder to exploit!
/s
Why would OpenRC or runit be any less likely to boot?
Automatic systemd service hardening guided by strace profiling
https://github.com/desbma/shh
A nice thing I found is that if you do (which I see they did not in the examples)
you can do And essentially just including the binary and the path you want available. ProtectSystem= is currently not compatible with this behavior.EDIT: More info here: https://github.com/systemd/systemd/issues/33688