{"id":3885,"date":"2024-02-27T02:15:00","date_gmt":"2024-02-27T07:15:00","guid":{"rendered":"https:\/\/www.both.org\/?p=3885"},"modified":"2024-02-27T08:02:49","modified_gmt":"2024-02-27T13:02:49","slug":"systemd-11-managing-resources-with-cgroups-in-systemd","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=3885","title":{"rendered":"systemd — #11: Managing resources with cgroups in systemd"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

Image by: Image by Mapbox Uncharted ERG, CC-BY 3.0 US<\/a><\/p>\n\n\n\n

Cgroups manage resources per application rather than by the individual processes that make up an application.<\/h2>\n\n\n\n

There is little more frustrating to me as a sysadmin than unexpectedly running out of a computing resource. On more than one occasion, I have filled all available disk space in a partition, run out of RAM, and not had enough CPU time to perform my tasks in a reasonable amount of time. Resource management is one of the most important tasks that sysadmins do.<\/p>\n\n\n\n

The point of resource management is to ensure that all processes have relatively equal access to the system resources they need. Resource management also involves ensuring that RAM, hard drive space, and CPU capacity are added when necessary or rationed when that is not possible. In addition, users who hog system resources, whether intentionally or accidentally, should be prevented from doing so.<\/p>\n\n\n\n

There are tools that enable sysadmins to monitor and manage various system resources. For example, top<\/a> and similar tools allow you to monitor the use of memory, I\/O, storage (disk, SSD, etc.), network, swap space, CPU usage, and more. These tools, particularly those that are CPU-centric, are mostly based on the paradigm that the running process is the unit of control. At best, they provide a way to adjust the nice number\u2013and through that, the priority\u2014or to kill a running process. (For information about nice numbers, see Monitoring Linux and Windows hosts with Glances<\/a><\/em>.)<\/p>\n\n\n\n

Other tools based on traditional resource management in a SystemV environment are managed by the \/etc\/security\/limits.conf<\/code> file and the local configuration files located in the \/etc\/security\/limits.d<\/code> directory. Resources can be limited in a fairly crude but useful manner by user or group. Resources that can be managed include various aspects of RAM, total CPU time per day, total amount of data, priority, nice number, number of concurrent logins, number of processes, maximum file size, and more.<\/p>\n\n\n\n

Using cgroups for process management<\/h2>\n\n\n\n

One major difference between systemd and SystemV<\/a> is how they handle processes. SystemV treats each process as an entity unto itself. systemd collects related processes into control groups, called cgroups<\/a> (short for control groups), and manages system resources for the cgroup as a whole. This means resources can be managed per application rather than by the individual processes that make up an application.<\/p>\n\n\n\n

The control units for cgroups are called slice units. Slices are a conceptualization that allows systemd to order processes in a tree format for ease of management.<\/p>\n\n\n\n

Viewing cgroups<\/h2>\n\n\n\n

I’ll start with some commands that allow you to view various types of information about cgroups. The systemctl status <service><\/code> command displays slice information about a specified service, including its slice. This example shows the at<\/code> daemon:<\/p>\n\n\n\n

[root@testvm1 ~]# systemctl status atd.service\n\u25cf atd.service - Deferred execution scheduler\n     Loaded: loaded (\/usr\/lib\/systemd\/system\/atd.service; enabled; vendor preset: enabled)\n     Active: active (running) since Wed 2020-09-23 12:18:24 EDT; 1 day 3h ago\n       Docs: man:atd(8)\n   Main PID: 1010 (atd)\n      Tasks: 1 (limit: 14760)\n     Memory: 440.0K\n        CPU: 5ms\n     CGroup: \/system.slice\/atd.service\n             \u2514\u25001010 \/usr\/sbin\/atd -f\n\nSep 23 12:18:24 testvm1.both.org systemd[1]: Started Deferred execution scheduler.\n[root@testvm1 ~]#<\/code><\/pre>\n\n\n\n
This is an excellent example of one reason that I find systemd more usable than SystemV and the old init program. There is significantly more information here than SystemV could provide. The cgroup entry includes the hierarchical structure where the system.slice<\/code> is systemd (PID 1), and the atd.service<\/code> is one level below and part of the system.slice<\/code>. The second line of the cgroup entry also shows the process ID (PID) and the command used to start the daemon.<\/p>\n\n\n\n
The systemctl<\/code> command shows multiple cgroup entries. The --all<\/code> option shows all slices, including those that are not currently active:<\/p>\n\n\n\n
[root@testvm1 ~]# systemctl -t slice --all\n  UNIT                             LOAD   ACTIVE   SUB    DESCRIPTION                     \n  -.slice                          loaded active   active Root Slice                      \n  system-getty.slice               loaded active   active system-getty.slice              \n  system-lvm2\\x2dpvscan.slice      loaded active   active system-lvm2\\x2dpvscan.slice     \n  system-modprobe.slice            loaded active   active system-modprobe.slice           \n  system-sshd\\x2dkeygen.slice      loaded active   active system-sshd\\x2dkeygen.slice     \n  system-systemd\\x2dcoredump.slice loaded inactive dead   system-systemd\\x2dcoredump.slice\n  system-systemd\\x2dfsck.slice     loaded active   active system-systemd\\x2dfsck.slice    \n  system.slice                     loaded active   active System Slice                    \n  user-0.slice                     loaded active   active User Slice of UID 0             \n  user-1000.slice                  loaded active   active User Slice of UID 1000          \n  user.slice                       loaded active   active User and Session Slice          \n\nLOAD   = Reflects whether the unit definition was properly loaded.\nACTIVE = The high-level unit activation state, i.e. generalization of SUB.\nSUB    = The low-level unit activation state, values depend on unit type.\n\n11 loaded units listed.\nTo show all installed unit files use 'systemctl list-unit-files'.\n[root@testvm1 ~]#<\/code><\/pre>\n\n\n\n
The first thing to notice about this data is that it shows user slices for UIDs 0 (root) and 1000, which is my user login. This shows only the slices and not the services that are part of each slice. This data shows that a slice is created for each user at the time they log in. This can provide a way to manage all of a user’s tasks as a single cgroup entity.<\/p>\n\n\n\n
Explore the cgroup hierarchy<\/h2>\n\n\n\n
All is well and good so far, but cgroups are hierarchical, and all of the service units run as members of one of the cgroups. Viewing that hierarchy is easy and uses one old command and one new one that is part of systemd.<\/p>\n\n\n\n
The ps<\/code> command can be used to map the processes and their locations in the cgroup hierarchy. Note that it is necessary to specify the desired data columns when using the ps<\/code> command. I significantly reduced the volume of output from this command below, but I tried to leave enough so you can get a feel for what you might find on your systems:<\/p>\n\n\n\n
[root@testvm1 ~]# ps xawf -eo pid,user,cgroup,args\n    PID USER     CGROUP                      COMMAND\n      2 root     -                           [kthreadd]\n      3 root     -                            \\_ [rcu_gp]\n      4 root     -                            \\_ [rcu_par_gp]\n      6 root     -                            \\_ [kworker\/0:0H-kblockd]\n      9 root     -                            \\_ [mm_percpu_wq]\n     10 root     -                            \\_ [ksoftirqd\/0]\n     11 root     -                            \\_ [rcu_sched]\n     12 root     -                            \\_ [migration\/0]\n     13 root     -                            \\_ [cpuhp\/0]\n     14 root     -                            \\_ [cpuhp\/1]\n<SNIP>\n 625406 root     -                            \\_ [kworker\/3:0-ata_sff]\n 625409 root     -                            \\_ [kworker\/u8:0-events_unbound]\n      1 root     0::\/init.scope              \/usr\/lib\/systemd\/systemd --switched-root --system --deserialize 30\n    588 root     0::\/system.slice\/systemd-jo \/usr\/lib\/systemd\/systemd-journald\n    599 root     0::\/system.slice\/systemd-ud \/usr\/lib\/systemd\/systemd-udevd\n    741 root     0::\/system.slice\/auditd.ser \/sbin\/auditd\n    743 root     0::\/system.slice\/auditd.ser  \\_ \/usr\/sbin\/sedispatch\n    764 root     0::\/system.slice\/ModemManag \/usr\/sbin\/ModemManager\n    765 root     0::\/system.slice\/NetworkMan \/usr\/sbin\/NetworkManager --no-daemon\n    767 root     0::\/system.slice\/irqbalance \/usr\/sbin\/irqbalance --foreground\n    779 root     0::\/system.slice\/mcelog.ser \/usr\/sbin\/mcelog --ignorenodev --daemon --foreground\n    781 root     0::\/system.slice\/rngd.servi \/sbin\/rngd -f\n    782 root     0::\/system.slice\/rsyslog.se \/usr\/sbin\/rsyslogd -n\n<SNIP>\n    893 root     0::\/system.slice\/sshd.servi sshd: \/usr\/sbin\/sshd -D [listener] 0 of 10-100 startups\n   1130 root     0::\/user.slice\/user-0.slice  \\_ sshd: root [priv]\n   1147 root     0::\/user.slice\/user-0.slice  |   \\_ sshd: root@pts\/0\n   1148 root     0::\/user.slice\/user-0.slice  |       \\_ -bash\n   1321 root     0::\/user.slice\/user-0.slice  |           \\_ screen\n   1322 root     0::\/user.slice\/user-0.slice  |               \\_ SCREEN\n   1323 root     0::\/user.slice\/user-0.slice  |                   \\_ \/bin\/bash\n 498801 root     0::\/user.slice\/user-0.slice  |                   |   \\_ man systemd.resource-control\n 498813 root     0::\/user.slice\/user-0.slice  |                   |       \\_ less\n   1351 root     0::\/user.slice\/user-0.slice  |                   \\_ \/bin\/bash\n 123293 root     0::\/user.slice\/user-0.slice  |                   |   \\_ man systemd.slice\n 123305 root     0::\/user.slice\/user-0.slice  |                   |       \\_ less\n   1380 root     0::\/user.slice\/user-0.slice  |                   \\_ \/bin\/bash\n 625412 root     0::\/user.slice\/user-0.slice  |                   |   \\_ ps xawf -eo pid,user,cgroup,args\n 625413 root     0::\/user.slice\/user-0.slice  |                   |   \\_ less\n 246795 root     0::\/user.slice\/user-0.slice  |                   \\_ \/bin\/bash\n 625338 root     0::\/user.slice\/user-0.slice  |                       \\_ \/usr\/bin\/mc -P \/var\/tmp\/mc-root\/mc.pwd.246795\n 625340 root     0::\/user.slice\/user-0.slice  |                           \\_ bash -rcfile .bashrc\n   1218 root     0::\/user.slice\/user-1000.sl  \\_ sshd: dboth [priv]\n   1233 dboth    0::\/user.slice\/user-1000.sl      \\_ sshd: dboth@pts\/1\n   1235 dboth    0::\/user.slice\/user-1000.sl          \\_ -bash\n<SNIP>\n   1010 root     0::\/system.slice\/atd.servic \/usr\/sbin\/atd -f\n   1011 root     0::\/system.slice\/crond.serv \/usr\/sbin\/crond -n\n   1098 root     0::\/system.slice\/lxdm.servi \/usr\/sbin\/lxdm-binary\n   1106 root     0::\/system.slice\/lxdm.servi  \\_ \/usr\/libexec\/Xorg -background none :0 vt01 -nolisten tcp -novtswitch -auth \/var\/run\/lxdm\/lxdm-:0.auth\n 370621 root     0::\/user.slice\/user-1000.sl  \\_ \/usr\/libexec\/lxdm-session\n 370631 dboth    0::\/user.slice\/user-1000.sl      \\_ xfce4-session\n 370841 dboth    0::\/user.slice\/user-1000.sl          \\_ \/usr\/bin\/ssh-agent \/bin\/sh -c exec -l bash -c \"\/usr\/bin\/startxfce4\"\n 370911 dboth    0::\/user.slice\/user-1000.sl          \\_ xfwm4 --display :0.0 --sm-client-id 2dead44ab-0b4d-4101-bca4-e6771f4a8ac2\n 370930 dboth    0::\/user.slice\/user-1000.sl          \\_ xfce4-panel --display :0.0 --sm-client-id 2ce38b8ef-86fd-4189-ace5-deec1d0e0952\n 370942 dboth    0::\/user.slice\/user-1000.sl          |   \\_ \/usr\/lib64\/xfce4\/panel\/wrapper-2.0 \/usr\/lib64\/xfce4\/panel\/plugins\/libsystray.so 6 23068680 systr\nay Notification Area Area where notification icons appear\n 370943 dboth    0::\/user.slice\/user-1000.sl          |   \\_ \/usr\/lib64\/xfce4\/panel\/wrapper-2.0 \/usr\/lib64\/xfce4\/panel\/plugins\/libpulseaudio-plugin.so 8 2306\n8681 pulseaudio PulseAudio Plugin Adjust the audio volume of the PulseAudio sound system\n 370944 dboth    0::\/user.slice\/user-1000.sl          |   \\_ \/usr\/lib64\/xfce4\/panel\/wrapper-2.0 \/usr\/lib64\/xfce4\/panel\/plugins\/libxfce4powermanager.so 9 2306\n8682 power-manager-plugin Power Manager Plugin Display the battery levels of your devices and control the brightness of your display\n 370945 dboth    0::\/user.slice\/user-1000.sl          |   \\_ \/usr\/lib64\/xfce4\/panel\/wrapper-2.0 \/usr\/lib64\/xfce4\/panel\/plugins\/libnotification-plugin.so 10 2\n3068683 notification-plugin Notification Plugin Notification plugin for the Xfce panel\n 370948 dboth    0::\/user.slice\/user-1000.sl          |   \\_ \/usr\/lib64\/xfce4\/panel\/wrapper-2.0 \/usr\/lib64\/xfce4\/panel\/plugins\/libactions.so 14 23068684 acti\nons Action Buttons Log out, lock or other system actions\n 370934 dboth    0::\/user.slice\/user-1000.sl          \\_ Thunar --sm-client-id 2cfc809d8-4e1d-497a-a5c5-6e4fa509c3fb --daemon\n 370939 dboth    0::\/user.slice\/user-1000.sl          \\_ xfdesktop --display :0.0 --sm-client-id 299be0608-4dca-4055-b4d6-55ec6e73a324\n 370962 dboth    0::\/user.slice\/user-1000.sl          \\_ nm-applet\n<SNIP><\/code><\/pre>\n\n\n\n
You can view the entire hierarchy with the systemd-cgls<\/code> command, which is a bit simpler because it does not require any complex options.<\/p>\n\n\n\n
I have shortened this tree view considerably. as well, but I left enough to give you some idea of the amount of data as well as the types of entries you should see when you do this on your system. I did this on one of my virtual machines, and it is about 200 lines long; the amount of data from my primary workstation is about 250 lines:<\/p>\n\n\n\n
[root@testvm1 ~]# systemd-cgls\nControl group \/:\n-.slice\n\u251c\u2500user.slice \n\u2502 \u251c\u2500user-0.slice \n\u2502 \u2502 \u251c\u2500session-1.scope \n\u2502 \u2502 \u2502 \u251c\u2500  1130 sshd: root [priv]\n\u2502 \u2502 \u2502 \u251c\u2500  1147 sshd: root@pts\/0\n\u2502 \u2502 \u2502 \u251c\u2500  1148 -bash\n\u2502 \u2502 \u2502 \u251c\u2500  1321 screen\n\u2502 \u2502 \u2502 \u251c\u2500  1322 SCREEN\n\u2502 \u2502 \u2502 \u251c\u2500  1323 \/bin\/bash\n\u2502 \u2502 \u2502 \u251c\u2500  1351 \/bin\/bash\n\u2502 \u2502 \u2502 \u251c\u2500  1380 \/bin\/bash\n\u2502 \u2502 \u2502 \u251c\u2500123293 man systemd.slice\n\u2502 \u2502 \u2502 \u251c\u2500123305 less\n\u2502 \u2502 \u2502 \u251c\u2500246795 \/bin\/bash\n\u2502 \u2502 \u2502 \u251c\u2500371371 man systemd-cgls\n\u2502 \u2502 \u2502 \u251c\u2500371383 less\n\u2502 \u2502 \u2502 \u251c\u2500371469 systemd-cgls\n\u2502 \u2502 \u2502 \u2514\u2500371470 less\n\u2502 \u2502 \u2514\u2500user@0.service \u2026\n\u2502 \u2502   \u251c\u2500dbus-broker.service \n\u2502 \u2502   \u2502 \u251c\u25001170 \/usr\/bin\/dbus-broker-launch --scope user\n\u2502 \u2502   \u2502 \u2514\u25001171 dbus-broker --log 4 --controller 12 --machine-id 3bccd1140fca488187f8a1439c832f07 --max-bytes 100000000000000 --max-fds 25000000000000 --max->\n\u2502 \u2502   \u251c\u2500gvfs-daemon.service \n\u2502 \u2502   \u2502 \u2514\u25001173 \/usr\/libexec\/gvfsd\n\u2502 \u2502   \u2514\u2500init.scope \n\u2502 \u2502     \u251c\u25001137 \/usr\/lib\/systemd\/systemd --user\n\u2502 \u2502     \u2514\u25001138 (sd-pam)\n\u2502 \u2514\u2500user-1000.slice \n\u2502   \u251c\u2500user@1000.service \u2026\n\u2502   \u2502 \u251c\u2500dbus\\x2d:1.2\\x2dorg.xfce.Xfconf.slice \n\u2502   \u2502 \u2502 \u2514\u2500dbus-:1.2-org.xfce.Xfconf@0.service \n\u2502   \u2502 \u2502   \u2514\u2500370748 \/usr\/lib64\/xfce4\/xfconf\/xfconfd\n\u2502   \u2502 \u251c\u2500dbus\\x2d:1.2\\x2dca.desrt.dconf.slice \n\u2502   \u2502 \u2502 \u2514\u2500dbus-:1.2-ca.desrt.dconf@0.service \n\u2502   \u2502 \u2502   \u2514\u2500371262 \/usr\/libexec\/dconf-service\n\u2502   \u2502 \u251c\u2500dbus-broker.service \n\u2502   \u2502 \u2502 \u251c\u25001260 \/usr\/bin\/dbus-broker-launch --scope user\n\u2502   \u2502 \u2502 \u2514\u25001261 dbus-broker --log 4 --controller 11 --machine-id \n<SNIP>\n\u2502   \u2502 \u2514\u2500gvfs-mtp-volume-monitor.service \n\u2502   \u2502   \u2514\u2500370987 \/usr\/libexec\/gvfs-mtp-volume-monitor\n\u2502   \u251c\u2500session-3.scope \n\u2502   \u2502 \u251c\u25001218 sshd: dboth [priv]\n\u2502   \u2502 \u251c\u25001233 sshd: dboth@pts\/1\n\u2502   \u2502 \u2514\u25001235 -bash\n\u2502   \u2514\u2500session-7.scope \n\u2502     \u251c\u2500370621 \/usr\/libexec\/lxdm-session\n\u2502     \u251c\u2500370631 xfce4-session\n\u2502     \u251c\u2500370805 \/usr\/bin\/VBoxClient --clipboard\n\u2502     \u251c\u2500370806 \/usr\/bin\/VBoxClient --clipboard\n\u2502     \u251c\u2500370817 \/usr\/bin\/VBoxClient --seamless\n\u2502     \u251c\u2500370818 \/usr\/bin\/VBoxClient --seamless\n\u2502     \u251c\u2500370824 \/usr\/bin\/VBoxClient --draganddrop\n\u2502     \u251c\u2500370825 \/usr\/bin\/VBoxClient --draganddrop\n\u2502     \u251c\u2500370841 \/usr\/bin\/ssh-agent \/bin\/sh -c exec -l bash -c \"\/usr\/bin\/startxfce4\"\n\u2502     \u251c\u2500370910 \/bin\/gpg-agent --sh --daemon --write-env-file \/home\/dboth\/.cache\/gpg-agent-info\n\u2502     \u251c\u2500370911 xfwm4 --display :0.0 --sm-client-id 2dead44ab-0b4d-4101-bca4-e6771f4a8ac2\n\u2502     \u251c\u2500370923 xfsettingsd --display :0.0 --sm-client-id 261b4a437-3029-461c-9551-68c2c42f4fef\n\u2502     \u251c\u2500370930 xfce4-panel --display :0.0 --sm-client-id 2ce38b8ef-86fd-4189-ace5-deec1d0e0952\n\u2502     \u251c\u2500370934 Thunar --sm-client-id 2cfc809d8-4e1d-497a-a5c5-6e4fa509c3fb --daemon\n\u2502     \u251c\u2500370939 xfdesktop --display :0.0 --sm-client-id 299be0608-4dca-4055-b4d6-55ec6e73a324\n<SNIP>\n\u2514\u2500system.slice \n  \u251c\u2500rngd.service \n  \u2502 \u2514\u25001650 \/sbin\/rngd -f\n  \u251c\u2500irqbalance.service \n  \u2502 \u2514\u25001631 \/usr\/sbin\/irqbalance --foreground\n  \u251c\u2500fprintd.service \n  \u2502 \u2514\u2500303383 \/usr\/libexec\/fprintd\n  \u251c\u2500systemd-udevd.service \n  \u2502 \u2514\u2500956 \/usr\/lib\/systemd\/systemd-udevd\n<SNIP>\n  \u251c\u2500systemd-journald.service \n  \u2502 \u2514\u2500588 \/usr\/lib\/systemd\/systemd-journald\n  \u251c\u2500atd.service \n  \u2502 \u2514\u25001010 \/usr\/sbin\/atd -f\n  \u251c\u2500system-dbus\\x2d:1.10\\x2dorg.freedesktop.problems.slice \n  \u2502 \u2514\u2500dbus-:1.10-org.freedesktop.problems@0.service \n  \u2502   \u2514\u2500371197 \/usr\/sbin\/abrt-dbus -t133\n  \u251c\u2500sshd.service \n  \u2502 \u2514\u2500893 sshd: \/usr\/sbin\/sshd -D [listener] 0 of 10-100 startups\n  \u251c\u2500vboxservice.service \n  \u2502 \u2514\u2500802 \/usr\/sbin\/VBoxService -f\n  \u251c\u2500crond.service \n  \u2502 \u2514\u25001011 \/usr\/sbin\/crond -n\n  \u251c\u2500NetworkManager.service \n  \u2502 \u2514\u2500765 \/usr\/sbin\/NetworkManager --no-daemon\n  \u251c\u2500switcheroo-control.service \n  \u2502 \u2514\u2500787 \/usr\/libexec\/switcheroo-control\n <SNIP><\/code><\/pre>\n\n\n\n
This tree view shows all of the user and system slices and the services and programs running in each cgroup. Notice the units called “scopes,” which group related programs into a management unit, within the user-1000.slice<\/code> in the listing above. The user-1000.slice\/session-7.scope<\/code> cgroup contains the GUI desktop program hierarchy, starting with the LXDM display manager session and all of its subtasks, including things like the Bash shell and the Thunar GUI file manager.<\/p>\n\n\n\n
Scope units are not defined in configuration files but are generated programmatically as the result of starting groups of related programs. Scope units do not create or start the processes running as part of that cgroup. All processes within the scope are equal, and there is no internal hierarchy. The life of a scope begins when the first process is created and ends when the last process is destroyed.<\/p>\n\n\n\n
Open several windows on your desktop, such as terminal emulators, LibreOffice, or whatever you want, then switch to an available virtual console and start something like top<\/code> or Midnight Commander<\/a>. Run the systemd-cgls<\/code> command on your host, and take note of the overall hierarchy and the scope units.<\/p>\n\n\n\n
The systemd-cgls<\/code> command provides a more complete representation of the cgroup hierarchy (and details of the units that make it up) than any other command I have found. I prefer its cleaner representation of the tree than what the ps<\/code> command provides.<\/p>\n\n\n\n
With a little help from my friends<\/h2>\n\n\n\n
After covering these basics, I had planned to go into more detail about cgroups and how to use them, but I discovered a series of four excellent articles by Red Hat’s Steve Ovens<\/a> on Opensource.com’s sister site Enable Sysadmin<\/a>. Rather then basically rewriting Steve’s articles, I decided it would be much better to take advantage of his cgroup expertise by linking to them:<\/p>\n\n\n\n
    \n
  1. A Linux sysadmin’s introduction to cgroups<\/a><\/li>\n\n\n\n
  2. How to manage cgroups with CPUShares<\/a><\/li>\n\n\n\n
  3. Managing cgroups the hard way\u2014manually<\/a><\/li>\n\n\n\n
  4. Managing cgroups with systemd<\/a><\/li>\n<\/ol>\n\n\n\n
    Enjoy and learn from them, as I did.<\/p>\n\n\n\n
    Other resources<\/h2>\n\n\n\n
    There is a great deal of information about systemd available on the internet, but much is terse, obtuse, or even misleading. In addition to the resources mentioned in this article, the following webpages offer more detailed and reliable information about systemd startup. This list has grown since I started this series of articles to reflect the research I have done.<\/p>\n\n\n\n