As discussed in the previous issue of Open Horizons Magazine, SLES 12 brings about some major changes. Some of these you will never notice unless you go looking for them, but others will change the way that you perform even the most basic administration task!
The first of these major changes is the move away from the venerable SysV init daemon. Yes, that's right, all the stuff that you've learned over the years about runlevels, scripts in /etc/init.d and the symlinks in rc3.d are a thing of the past (well, mostly, but we'll come back to that later). What do we have instead? The answer is systemd, a shiny new (to SLES) daemon that does everything that init did, but better, and a whole lot more besides.
So, where do we start with systemd? Well, we'll start with a quote from my good friend Richard Brown, now chairman of the opensuse board! He describes systemd as a "seven stage initialisation daemon". These stages are as follows:
- Denial
- Pain
- Anger
- Depression
- Working Through
- Acceptance
- Excitement
Now, obviously, this list is a little tongue in cheek, but there's also some truth in it. What I want to do is to move you through these stages as quickly as possible to get to the good bits.
Compatibility with init (Denial)
To ease the transition from init to systemd, SUSE have provided a number of backwardly compatible commands that you can use if you really don't want to leave the init world. These commands do the same job as the ones that you've been using for years, but don't necessarily work in exactly the same way!
Some examples are
rcapache2 stop/start/status
service apache2 stop/start/status
reboot
shutdown
While these commands have familiar syntax, their behaviour is now to do the systemd equivalent of the original command. One thing to watch out for in particular is stop/start. These no longer give any status information to say whether the requested action actually happened! Instead, you're now going to have to run a status command as well to see what happened and, of course, the status output is completely different, so any kind of scripting that relies on these commands will need to be modified.
What do we take from this? Yes, you've got it - time to move on and learn the new commands to understand what's really happening under the hood.
So, how does this newfangled thingummy work? (Pain, Anger, Depression)
At this point, it's best if you just unlearn everything you knew about init and start with a blank brain. Well, not blank, but you get the idea. Let's start with the basics.
Systemd is based on the idea of units. A unit is a configuration file that defines something. I'm not being deliberately vague here, it's just that this is such a big change that it's going to take a bit of getting used to. Remember in Linux, everything is a file? Well in systemd, everything is a unit. A file system mounted from fstab is mounted by a unit; a TTY text console is started by a unit; a service is started by a unit; the equivalent of a runlevel (a target in system land) is defined by a unit.
At the most basic level, a unit is a configuration file (not a script) that defines how the unit should act. For simplicity, I'm only going to deal with services and targets (runlevels) in this article. Unit files have file extensions that define the type of unit, e.g. apache2.service, poweroff.target. When referring to these units in systemd commands, you can drop the .service part for simplicity. Inside these configuration files, the syntax harks back to the .ini style that you'll be familiar with if you've ever looked at a Samba configuration file. The basic structure is:
[Section]
variable = value
variable = value
Let's look at the unit file for the ntpd time daemon service:
[Unit]
Description=NTP Server Daemon
Documentation=man:ntpd(1)
After=nss-lookup.target
After=network.target
Wants=network.target
[Service]
Type=forking
PIDFile=/var/run/ntp/ntpd.pid
ExecStart=/usr/sbin/start-ntpd start
RestartSec=11min
Restart=always
[Install]
WantedBy=multi-user.target
You can see that the [Unit] section deals with information about the service being defined. There’s info on the service itself, and also variables that define what is required by this service (Wants), and ordering clues (After). The [Service] section deals with starting/stopping the service, as well as options that specify how to restart the service if it crashes! Lastly, the [Install] section specifies where in the configuration ntpd should be installed.
That deals with services, but what about runlevels? In systemd, targets replace runlevels. They do the same job but work very differently. In SysV init, each runlevel was defined in its own directory by creating symlinks to the actual scripts, prepended with S/K and a number used for ordering. A target is defined very differently. In the systemd world, a target is a defined state that the system should be in, rather than a list of things to be done in the hope of achieving a state.
This leaves systemd free to make decisions on how to get to there, and it may make decisions that surprise you. At this point I will give you some advice: don't fight it - you will almost certainly lose and end up battered and bruised! systemd is designed to get you to the system state that you've requested, don't worry about the nitty gritty details of how it gets there - it usually does the right thing... As with some of the commands above, SUSE have created some targets with familiar names to give you something to cling on to, but you've already deleted all SysV knowledge haven't you?? The main targets defined in SLES 12 are shown in Table 1
How is a target defined then? There is a basic configuration file e.g. multiuser.target:
[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes
This file doesn't really show much about which services should run though. Services are attached to a target either within their unit file, as seen with the install line in the ntpd unit file above, or using a symlink in the wants directory:
linux-u1kg:~ # ls /usr/lib/systemd/system/multi-user.target.wants/
after-local.service plymouth-quit-wait.service
dbus.service systemd-logind.service
getty.target systemd-update-utmp-runlevel.service
plymouth-quit.service systemd-user-sessions.service
The main thing to notice here is that each target only defines a very small part of the system. Unlike init, there will be many targets running at the same time, each doing their part:
linux-u1kg:~ # systemctl list-units -t target
UNIT LOAD ACTIVE SUB DESCRIPTION
basic.target loaded active active Basic System
bluetooth.target loaded active active Bluetooth
cryptsetup.target loaded active active Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network.target loaded active active Network
nss-lookup.target loaded active active Host and Network Name Lookups
nss-user-lookup.target loaded active active User and Group Name Lookups
paths.target loaded active active Paths
remote-fs-pre.target loaded active active Remote File Systems (Pre)
remote-fs.target loaded active active Remote File Systems
slices.target loaded active active Slices
sockets.target loaded active active Sockets
sound.target loaded active active Sound Card
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
time-sync.target loaded active active System Time Synchronized
timers.target loaded active active Timers
Where is everything and how do I use it (acceptance, finding your way)
There are two main directories used by systemd; /usr/lib/systemd/system and /etc/systemd/system. All unit files are defined in these directories. There is a very important rule to follow here, and it's very very simple: never touch anything in /usr/lib/systemd/system. These files will be overwritten by patches/service packs etc. If you want to make changes, you should use copies of the unit files in /etc/systemd/system. I'll cover that in the next section. Just a quick note before I move on: systemd also has a SysV init compatibility layer built in - if your third party backup software doesn't have a systemd unit file shipped along with it, don't worry, systemd will also look in /etc/init.d for service scripts.
Now that you have an idea how systemd works, I'd better show you the commands to use to interact with it. The main command that you'll use on a daily basis is systemctl. It will let you start, stop and check the status of services:
linux-u1kg:~ # systemctl stop ntpd
linux-u1kg:~ # systemctl start ntpd
linux-u1kg:~ # systemctl status ntpd
ntpd.service - NTP Server Daemon
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled)
Active: active (running) since Thu 2014-10-16 18:00:13 CEST; 5s ago
Docs: man:ntpd(1)
Process: 2497 ExecStart=/usr/sbin/start-ntpd start (code=exited, status=0/SUCCESS)
Main PID: 2510 (ntpd)
CGroup: /system.slice/ntpd.service
└─2510 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -i /v...
Oct 16 18:00:13 linux-u1kg ntpd[2510]: proto: precision = 0.201 usec
Oct 16 18:00:13 linux-u1kg ntpd[2510]: ntp_io: estimated max descriptors: 1...16
Oct 16 18:00:13 linux-u1kg ntpd[2510]: Listen and drop on 0 v4wildcard 0.0....23
Oct 16 18:00:13 linux-u1kg ntpd[2510]: Listen and drop on 1 v6wildcard :: U...23
Oct 16 18:00:13 linux-u1kg ntpd[2510]: Listen normally on 2 lo 127.0.0.1 UDP 123
Oct 16 18:00:13 linux-u1kg ntpd[2510]: Listen normally on 3 eth0 172.16.17....23
Oct 16 18:00:13 linux-u1kg ntpd[2510]: Listen normally on 4 lo ::1 UDP 123
Oct 16 18:00:13 linux-u1kg ntpd[2510]: Listen normally on 5 eth0 fe80::20c:...23
Oct 16 18:00:13 linux-u1kg ntpd[2510]: peers refreshed
Oct 16 18:00:13 linux-u1kg ntpd[2510]: Listening on routing socket on fd #2...es
As you can see, the systemctl command is a properly quiet Linux command; it doesn't give you much feedback unless you get the syntax wrong. You can see however that the status check is significantly more verbose than the SysV equivalent. As well as the basic status info, it also shows the unit file that was loaded to start the service, process ID/control group info and the most recent log messages that the daemon generated.
You can control the desired state of a service using systemctl:
systemctl enable ntpd
This sets the service to start automatically. To disable it use:
systemctl disable ntpd
However, a word of warning here; disabling a service does not stop it from running, it only sets it not to start automatically - if a service that depends on ntpd is started, then ntpd will be started as well! If you really want to stop a service from starting at all, use
systemctl mask ntpd
Bear in mind though that this can have repercussions if you try to start another service that needs ntpd.
To affect the overall state of the server, you're going to need to use target commands. Now, it's tempting to use:
systemctl start multi-user.target
However, this may not quite achieve what you want. If you're running the graphical target, then the multi-user target is also running, and the above command will have no effect at all! The command that you probably meant was
systemctl isolate mult-user.target
This tells systemd to stop any services that are not required by the multi-user target, i.e the graphical desktop.
Lastly, you might need to shut down or reboot the server. Try:
systemctl reboot
systemctl poweroff
The next stage (Working through, acceptance, Excitement)
Now that we know what the basics of systemd look like, it's time to show you a couple of really cool features. The first of these is the ability to modify unit files in a completely safe way, well two ways really!
Let's assume we'd like to change the behaviour of the ntpd service. Remember I said that you never modify anything in /usr/lib/systemd/system? Well all you need to do is to make a copy of the unit file into /etc:
cp /usr/lib/systemd/system/ntpd.service /etc/systemd/system
You can now go ahead and modify anything you like in that copy of the unit file. Systemd will use it in preference to the /usr/lib/systemd/system version. You can change any/all settings in there, and systemd will use it.
However, by doing this, you're overriding the whole unit file. What happens if a patch fixes something in the /usr/lib version? Well, those changes will be ignored, and your copy will always be used. There is a better way of course. Most of the time, we only want to change a couple of settings, and there's a much cleaner way of doing this.
systemd has the concept of drop-in overrides. Rather than redefining a unit, you can create a config file that overrides one or more settings. So, let's assume that we'd like ntp to be restarted quickly if it crashes. The default setting is to restart it after 11 minutes - that's a long time!
To create a drop-in override, you must first create a directory that matches the name of the unit file with .d appended:
mkdir /etc/systemd/system/ntpd.service.d
You can now create any number of config files in there, the only restriction is that they must end with .conf. To solve the slow ntp restart issue, create a file in this directory with the name fast-restart.conf with only the following lines:
[Service]
RestartSec = 10s
Once that's saved, we need to tell systemd that we've created it:
systemctl daemon-reload
Or stop and start the service.
Now when we look at the status of ntpd, we'll see that it has picked up our new file:
linux-u1kg:~ # systemctl status ntpd
ntpd.service - NTP Server Daemon
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled)
Drop-In: /etc/systemd/system/ntpd.service.d
└─fast-restart.conf
…
I hope that you can see the flexibility of drop-in overrides - no more wondering why a patch didn't fix what it was supposed to, and no hunting for .rpmnew files and trying to merge your changes back in!
The final cool features that we're going to investigate in this article allow us to analyse the boot process to get some idea of what happened.
To find out if any services failed during boot try:
systemctl --failed
This sure beats trolling through /var/log/boot.msg and /var/log/messages to see what happened!
The last three commands for this article let you look at the timings of what happened during boot.
Firstly
linux-u1kg:~ # systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
graphical.target @9.735s
└─multi-user.target @9.735s
└─cron.service @9.735s
└─postfix.service @9.444s +290ms
└─nss-lookup.target @9.442s
└─network-online.target @9.442s
└─network.target @9.440s
└─wicked.service @2.452s +6.987s
└─wickedd-nanny.service @2.413s +37ms
└─wickedd.service @2.360s +52ms
└─wickedd-dhcp4.service @2.189s +169ms
└─dbus.service @2.024s
└─basic.target @2.023s
└─timers.target @2.023s
└─systemd-tmpfiles-clean.timer @2.023s
└─sysinit.target @2.022s
└─apparmor.service @1.733s +289ms
└─systemd-tmpfiles-setup.service @1.716s +16ms
└─local-fs.target @1.715s
└─boot-efi.mount @908ms +131ms
└─dev-disk-by\x2duuid-FB80\x2dB57A.device
Secondly, let's name and shame the services that took up too much time
linux-u1kg:~ # systemd-analyze blame | head
6.987s wicked.service
2.006s kdump.service
1.348s systemd-udev-settle.service
1.013s display-manager.service
403ms rsyslog.service
297ms accounts-daemon.service
290ms postfix.service
Ah, wicked.service obviously sloped off for a quick smoke while everything else was going on.
Lastly, no article is complete without a fancy graphic (figure 1). To generate a timeline for the boot use:
linux-u1kg:~ # systemd-analyze plot > boot.svg
I hope that this has given you a good overview of systemd. There's a whole lot more to investigate of course so keep your eyes peeled for future deep dives on systemd.
