{"id":3840,"date":"2024-02-06T01:15:00","date_gmt":"2024-02-06T06:15:00","guid":{"rendered":"https:\/\/www.both.org\/?p=3840"},"modified":"2024-01-30T13:40:46","modified_gmt":"2024-01-30T18:40:46","slug":"systemd-3-using-the-systemctl-command-to-manage-systemd-units","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=3840","title":{"rendered":"systemd &#8212; #3: Using the systemctl command to manage systemd units"},"content":{"rendered":"<div class=\"pld-like-dislike-wrap pld-template-1\">\r\n    <div class=\"pld-like-wrap  pld-common-wrap\">\r\n    <a href=\"javascript:void(0)\" class=\"pld-like-trigger pld-like-dislike-trigger  \" title=\"\" data-post-id=\"3840\" data-trigger-type=\"like\" data-restriction=\"cookie\" data-already-liked=\"0\">\r\n                        <i class=\"fas fa-thumbs-up\"><\/i>\r\n                <\/a>\r\n    <span class=\"pld-like-count-wrap pld-count-wrap\">    <\/span>\r\n<\/div><\/div>\n<p class=\"has-small-font-size\">Image by: Opensource.com<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Units are the basis of everything in systemd<\/h2>\n\n\n\n<p>In the first two articles in this series, I explored the Linux systemd startup sequence. In the first article, <a href=\"https:\/\/www.both.org\/?p=3822\" data-type=\"link\" data-id=\"https:\/\/www.both.org\/?p=3822\" target=\"_blank\" rel=\"noreferrer noopener\">Learning to Love systemd<\/a>, I looked at systemd&#8217;s functions and architecture and the controversy around its role as a replacement for the old SystemV init program and startup scripts. In the second article, <a href=\"https:\/\/www.both.org\/?p=3835\" data-type=\"link\" data-id=\"https:\/\/www.both.org\/?p=3835\" target=\"_blank\" rel=\"noreferrer noopener\">Understanding Linux startup with systemd<\/a>, I examined two important systemd tools, systemctl and journalctl, and explained how to switch from one target to another and to change the default target.<\/p>\n\n\n\n<p>In this third article about systemd, we&#8217;ll explore systemd units in more detail and how to use the <strong>systemctl<\/strong> command to explore and manage units. I&#8217;ll also explain how to stop and disable units and how to create a new systemd mount unit to mount a new filesystem and enable it to initiate during startup.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"preparation\">Preparation<\/h2>\n\n\n\n<p>All of the experiments in this article should be done as the root user (unless otherwise specified). Some of the commands that simply list various systemd units can be performed by non-root users, but the commands that make changes cannot. Make sure to do all of these experiments only on non-production hosts or virtual machines (VMs).<\/p>\n\n\n\n<p>One of these experiments requires the sysstat package. As root, install it before you move on. For Fedora and other Red Hat-based distributions you can install sysstat with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>dnf -y install sysstat<\/strong><\/code><\/pre>\n\n\n\n<p>The sysstat RPM installs several statistical tools that can be used for problem determination. One is <a href=\"https:\/\/en.wikipedia.org\/wiki\/Sar_%28Unix%29\" target=\"_blank\" rel=\"noreferrer noopener\">System Activity Report<\/a> (SAR), which records many system performance data points at regular intervals (every 10 minutes by default). Rather than run as a daemon in the background, the sysstat package installs two systemd timers. One timer runs every 10 minutes to collect data, and the other runs once a day to aggregate the daily data. In this article, I will look briefly at these timers but wait to explain how to create a timer in a future article.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"systemd-suite\">systemd suite<\/h2>\n\n\n\n<p>The fact is, systemd is more than just one program. It is a large suite of programs all designed to work together to manage nearly every aspect of a running Linux system. A full exposition of systemd would take a book on its own. Most of us do not need to understand all of the details about how all of systemd&#8217;s components fit together, so I will focus on the programs and components that enable you to manage various Linux services and deal with log files and journals.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"practical-structure\">Practical structure<\/h2>\n\n\n\n<p>The structure of systemd\u2014outside of its executable files\u2014is contained in its many configuration files. Although these files have different names and identifier extensions, they are all called &#8220;unit&#8221; files. Units are the basis of everything systemd.<\/p>\n\n\n\n<p>Unit files are ASCII plain-text files that are accessible to and can be created or modified by a sysadmin. There are a number of unit file types, and each has its own man page. Figure 1 lists some of these unit file types by their filename extensions and a short description of each.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><th>systemd unit<\/th><th>Description<\/th><\/tr><tr><th>.automount<\/th><td>The <strong>.automount<\/strong> units are used to implement on-demand (i.e., plug and play) and mounting of filesystem units in parallel during startup.<\/td><\/tr><tr><th>.device<\/th><td>The <strong>.device<\/strong> unit files define hardware and virtual devices that are exposed to the sysadmin in the <strong>\/dev\/directory<\/strong>. Not all devices have unit files; typically, block devices such as hard drives, network devices, and some others have unit files.<\/td><\/tr><tr><th>.mount<\/th><td>The <strong>.mount<\/strong> unit defines a mount point on the Linux filesystem directory structure.<\/td><\/tr><tr><th>.scope<\/th><td>The <strong>.scope<\/strong> unit defines and manages a set of system processes. This unit is not configured using unit files, rather it is created programmatically. Per the <strong>systemd.scope<\/strong> man page, \u201cThe main purpose of scope units is grouping worker processes of a system service for organization and for managing resources.\u201d<\/td><\/tr><tr><th>.service<\/th><td>The <strong>.service<\/strong> unit files define processes that are managed by systemd. These include services such as crond cups (Common Unix Printing System), iptables, multiple logical volume management (LVM) services, NetworkManager, and more.<\/td><\/tr><tr><th>.slice<\/th><td>The <strong>.slice<\/strong> unit defines a \u201cslice,\u201d which is a conceptual division of system resources that are related to a group of processes. You can think of all system resources as a pie and this subset of resources as a \u201cslice\u201d out of that pie.<\/td><\/tr><tr><th>.socket<\/th><td>The <strong>.socket<\/strong> units define interprocess communication sockets, such as network sockets.<\/td><\/tr><tr><th>.swap<\/th><td>The <strong>.swap<\/strong> units define swap devices or files.<\/td><\/tr><tr><th>.target<\/th><td>The <strong>.target<\/strong> units define groups of unit files that define startup synchronization points, runlevels, and services. Target units define the services and other units that must be active in order to start successfully.<\/td><\/tr><tr><th>.timer<\/th><td>The <strong>.timer<\/strong> unit defines timers that can initiate program execution at specified times.<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\"><sup>Figure. 1: Some systemd unit file types<\/sup><\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"systemctl\">systemctl<\/h2>\n\n\n\n<p>systemd provides the <strong>systemctl<\/strong> command that is used to start and stop services, configure them to launch (or not) at system startup, and monitor the current status of running services. In a terminal session as the root user, ensure that root&#8217;s home directory ( <strong>~<\/strong> ) is the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Pwd\" target=\"_blank\" rel=\"noreferrer noopener\">PWD<\/a>. To begin looking at units in various ways, list all of the loaded and active systemd units. <strong>systemctl<\/strong> automatically pipes its <a href=\"https:\/\/en.wikipedia.org\/wiki\/Standard_streams#Standard_output_(stdout)\" target=\"_blank\" rel=\"noreferrer noopener\">stdout<\/a> data stream through the <strong>less<\/strong> pager, so you don&#8217;t have to do that yourself. I&#8217;ve trimmed down the output data stream a bit but left enough so you can see the wide range of systemd units.<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>&#91;root@testvm1 ~]# <strong>systemctl<\/strong>\n  UNIT                                                        LOAD   ACTIVE SUB       DESCRIPTION      >\n  proc-sys-fs-binfmt_misc.automount                           loaded active waiting   Arbitrary Executa>\n&lt;SNIP>\n  sys-devices-platform-serial8250-tty-ttyS12.device           loaded active plugged   \/sys\/devices\/plat>\n  sys-devices-platform-serial8250-tty-ttyS13.device           loaded active plugged   \/sys\/devices\/plat>\n  sys-devices-platform-serial8250-tty-ttyS14.device           loaded active plugged   \/sys\/devices\/plat>\n  sys-devices-platform-serial8250-tty-ttyS15.device           loaded active plugged   \/sys\/devices\/plat>\n&lt;SNIP>\n  lightdm.service                                             loaded active running   Light Display Man>\n  lm_sensors.service                                          loaded active exited    Hardware Monitori>\n  lvm2-monitor.service                                        loaded active exited    Monitoring of LVM>\n  mcelog.service                                              loaded active running   Machine Check Exc>\n  ModemManager.service                                        loaded active running   Modem Manager\n  NetworkManager-wait-online.service                          loaded active exited    Network Manager W>\n  NetworkManager.service                                      loaded active running   Network Manager\n &lt;SNIP>\n  sshd.service                                                loaded active running   OpenSSH server da>\n&lt;SNIP>\n  dev-zram0.swap                                              loaded active active    Compressed Swap o>\n  basic.target                                                loaded active active    Basic System\n  cryptsetup.target                                           loaded active active    Local Encrypted V>\n  getty.target                                                loaded active active    Login Prompts\n  graphical.target                                            loaded active active    Graphical Interfa>\n  integritysetup.target                                       loaded active active    Local Integrity P>\n  local-fs-pre.target                                         loaded active active    Preparation for L>\n  local-fs.target                                             loaded active active    Local File Systems\n  multi-user.target                                           loaded active active    Multi-User System\n  network-online.target                                       loaded active active    Network is Online\n  network-pre.target                                          loaded active active    Preparation for N>\n  network.target                                              loaded active active    Network\n  sysinit.target                                              loaded active active    System Initializa>\n  timers.target                                               loaded active active    Timer Units\n  veritysetup.target                                          loaded active active    Local Verity Prot>\n  dnf-makecache.timer                                         loaded active waiting   dnf makecache --t>\n  logrotate.timer                                             loaded active waiting   Daily rotation of>\n  plocate-updatedb.timer                                      loaded active waiting   Update the plocat>\n  raid-check.timer                                            loaded active waiting   Weekly RAID setup>\n  sysstat-collect.timer                                       loaded active waiting   Run system activi>\n  sysstat-summary.timer                                       loaded active waiting   Generate summary >\n  systemd-tmpfiles-clean.timer                                loaded active waiting   Daily Cleanup of >\n  unbound-anchor.timer                                        loaded active waiting   daily update of t>\n\nLOAD   = Reflects whether the unit definition was properly loaded.\nACTIVE = The high-level unit activation state, i.e. generalization of SUB.\nSUB    = The low-level unit activation state, values depend on unit type.\n200 loaded units listed. Pass --all to see loaded but inactive units, too.\n<\/code><\/pre>\n\n\n\n<p>As you scroll through the data in your terminal session, look for some specific things. The first section lists devices such as hard drives, sound cards, network interface cards, and TTY devices. Another section shows the filesystem mount points. Other sections include various services and a list of all loaded and active targets.<\/p>\n\n\n\n<p>The <strong>sysstat<\/strong> timers at the bottom of the output are used to collect and generate daily system activity summaries for SAR. SAR is a very useful problem-solving tool. (You can learn more about it in Chapter 13 of my book <em><a href=\"https:\/\/www.both.org\/?page_id=2208\" data-type=\"link\" data-id=\"https:\/\/www.both.org\/?page_id=2208\" target=\"_blank\" rel=\"noreferrer noopener\">Using and Administering Linux: Volume 1, Zero to SysAdmin: Getting Started<\/a><\/em>.) Near the very bottom, three lines describe the meanings of the statuses (loaded, active, and sub). Press <strong>q<\/strong> to exit the pager.<\/p>\n\n\n\n<p>Use the following command (as suggested in the last line of the output above) to see all the units that are installed, whether or not they are loaded. I won&#8217;t reproduce the output here, because you can scroll through it on your own. The systemctl program has an excellent tab-completion facility that makes it easy to enter complex commands without needing to memorize all the options:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl list-unit-files<\/strong><\/code><\/pre>\n\n\n\n<p>You can see that some units are disabled. Table 1 in the man page for systemctl lists and provides short descriptions of the entries you might see in this listing. Use the <strong>-t<\/strong> (type) option to view just the timer units:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl list-unit-files -t timer<\/strong>\nUNIT FILE                    STATE   \nchrony-dnssrv@.timer         disabled\ndnf-makecache.timer          enabled \nfstrim.timer                 disabled\nlogrotate.timer              disabled\nlogwatch.timer               disabled\nmdadm-last-resort@.timer     static  \nmlocate-updatedb.timer       enabled \nsysstat-collect.timer        enabled \nsysstat-summary.timer        enabled \nsystemd-tmpfiles-clean.timer static  \nunbound-anchor.timer         enabled<\/code><\/pre>\n\n\n\n<p>You could do the same thing with this alternative, which provides considerably more detail:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl list-timers<\/strong>\nThu 2020-04-16 09:06:20 EDT  3min 59s left n\/a                          n\/a           systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service\nThu 2020-04-16 10:02:01 EDT  59min left    Thu 2020-04-16 09:01:32 EDT  49s ago       dnf-makecache.timer          dnf-makecache.service\nThu 2020-04-16 13:00:00 EDT  3h 57min left n\/a                          n\/a           sysstat-collect.timer        sysstat-collect.service\nFri 2020-04-17 00:00:00 EDT  14h left      Thu 2020-04-16 12:51:37 EDT  3h 49min left mlocate-updatedb.timer       mlocate-updatedb.service\nFri 2020-04-17 00:00:00 EDT  14h left      Thu 2020-04-16 12:51:37 EDT  3h 49min left unbound-anchor.timer         unbound-anchor.service\nFri 2020-04-17 00:07:00 EDT  15h left      n\/a                          n\/a           sysstat-summary.timer        sysstat-summary.service\n\n6 timers listed.\nPass --all to see loaded but inactive timers, too.\n&#91;root@testvm1 ~]#<\/code><\/pre>\n\n\n\n<p>Although there is no option to do <strong>systemctl<\/strong> list-mounts, you can list the mount point unit files:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl list-unit-files -t mount<\/strong>\nUNIT FILE                     STATE    \n-.mount                       generated\nboot.mount                    generated\ndev-hugepages.mount           static   \ndev-mqueue.mount              static   \nhome.mount                    generated\nproc-fs-nfsd.mount            static   \nproc-sys-fs-binfmt_misc.mount disabled \nrun-vmblock\\x2dfuse.mount     disabled \nsys-fs-fuse-connections.mount static   \nsys-kernel-config.mount       static   \nsys-kernel-debug.mount        static   \ntmp.mount                     generated\nusr.mount                     generated\nvar-lib-nfs-rpc_pipefs.mount  static   \nvar.mount                     generated\n\n15 unit files listed.\n&#91;root@testvm1 ~]#<\/code><\/pre>\n\n\n\n<p>The STATE column in this data stream is interesting and requires a bit of explanation. The &#8220;generated&#8221; states indicate that the mount unit was generated on the fly during startup using the information in <strong>\/etc\/fstab<\/strong>. The program that generates these mount units is <strong>\/lib\/systemd\/system-generators\/systemd-fstab-generator,<\/strong> along with other tools that generate a number of other unit types. The &#8220;static&#8221; mount units are for filesystems like <strong>\/proc<\/strong> and <strong>\/sys<\/strong>, and the files for these are located in the <strong>\/usr\/lib\/systemd\/system<\/strong> directory.<\/p>\n\n\n\n<p>Now, look at the service units. This command will show all services installed on the host, whether or not they are active:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl --all -t service<\/strong><\/code><\/pre>\n\n\n\n<p>The bottom of this listing of service units displays 166 as the total number of loaded units on my host. Your number will probably differ.<\/p>\n\n\n\n<p>Unit files do not have a filename extension (such as <strong>.unit<\/strong>) to help identify them, so you can generalize that most configuration files that belong to systemd are unit files of one type or another. The few remaining files are mostly <strong>.conf<\/strong> files located in <strong>\/etc\/systemd<\/strong>. Unit files are stored in the <strong>\/usr\/lib\/systemd<\/strong> directory and its subdirectories, while the <strong>\/etc\/systemd\/<\/strong> directory and its subdirectories contain symbolic links to the unit files necessary to the local configuration of this host.<\/p>\n\n\n\n<p>To explore this, make <strong>\/etc\/systemd<\/strong> the PWD and list its contents. Then make <strong>\/etc\/systemd\/system<\/strong> the PWD and list its contents, and list the contents of at least a couple of the current PWD&#8217;s subdirectories. Take a look at the <strong>default.target<\/strong> file, which determines which runlevel target the system will boot to. In the second article in this series, I explained how to change the default target from the GUI (<strong>graphical.target<\/strong>) to the command-line only (<strong>multi-user.target<\/strong>) target. The <strong>default.target<\/strong> file on my test VM is simply a symlink to <strong>\/usr\/lib\/systemd\/system\/graphical.target<\/strong>.<\/p>\n\n\n\n<p>Take a few minutes to examine the contents of the <strong>\/etc\/systemd\/system\/default.target<\/strong> file:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 system]# <strong>cat default.target<\/strong> \n#  SPDX-License-Identifier: LGPL-2.1+\n#\n#  This file is part of systemd.\n#\n#  systemd is free software; you can redistribute it and\/or modify it\n#  under the terms of the GNU Lesser General Public License as published by\n#  the Free Software Foundation; either version 2.1 of the License, or\n#  (at your option) any later version.\n\n&#91;Unit]\nDescription=Graphical Interface\nDocumentation=man:systemd.special(7)\nRequires=multi-user.target\nWants=display-manager.service\nConflicts=rescue.service rescue.target\nAfter=multi-user.target rescue.service rescue.target display-manager.service\nAllowIsolate=yes<\/code><\/pre>\n\n\n\n<p>Note that this requires the <strong>multi-user.target<\/strong>; the <strong>graphical.target<\/strong> cannot start if the <strong>multi-user.target<\/strong> is not already up and running. It also says it &#8220;wants&#8221; the <strong>display-manager.service<\/strong> unit. A &#8220;want&#8221; does not need to be fulfilled in order for the unit to start successfully. If the &#8220;want&#8221; cannot be fulfilled, it will be ignored by systemd, and the rest of the target will start regardless.<\/p>\n\n\n\n<p>The subdirectories in <strong>\/etc\/systemd\/system<\/strong> are lists of wants for various targets. Take a few minutes to explore the files and their contents in the <strong>\/etc\/systemd\/system\/graphical.target.wants<\/strong> directory.<\/p>\n\n\n\n<p>The <strong>systemd.unit<\/strong> man page contains a lot of good information about unit files, their structure, the sections they can be divided into, and the options that can be used. It also lists many of the unit types, all of which have their own man pages. If you want to interpret a unit file, this would be a good place to start.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"service-units\">Service units<\/h2>\n\n\n\n<p>A Fedora installation usually installs and enables services that particular hosts do not need for normal operation. Conversely, sometimes it doesn&#8217;t include services that need to be installed, enabled, and started. Services that are not needed for the Linux host to function as desired, but which are installed and possibly running, represent a security risk and should\u2014at minimum\u2014be stopped and disabled and\u2014at best\u2014should be uninstalled.<\/p>\n\n\n\n<p>The <strong>systemctl<\/strong> command is used to manage systemd units, including services, targets, mounts, and more. Take a closer look at the list of services to identify services that will never be used:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl --all -t service<\/strong>\nUNIT                           LOAD      ACTIVE SUB        DESCRIPTION                             \n&lt;snip>\nchronyd.service                loaded    active running    NTP client\/server                       \ncrond.service                  loaded    active running    Command Scheduler                       \ncups.service                   loaded    active running    CUPS Scheduler                          \ndbus-daemon.service            loaded    active running    D-Bus System Message Bus                \n&lt;SNIP>\n\u25cf ip6tables.service           not-found inactive dead     ip6tables.service                   \n\u25cf ipset.service               not-found inactive dead     ipset.service                       \n\u25cf iptables.service            not-found inactive dead     iptables.service                    \n&lt;SNIP>\nfirewalld.service              loaded    active   running  firewalld - dynamic firewall daemon\n&lt;SNIP>\n\u25cf ntpd.service                not-found inactive dead     ntpd.service                        \n\u25cf ntpdate.service             not-found inactive dead     ntpdate.service                     \npcscd.service                 loaded    active   running  PC\/SC Smart Card Daemon<\/code><\/pre>\n\n\n\n<p>I have pruned out most of the output from the command to save space. The services that show &#8220;loaded active running&#8221; are obvious. The &#8220;not-found&#8221; services are ones that systemd is aware of but are not installed on the Linux host. If you want to run those services, you must install the packages that contain them.<\/p>\n\n\n\n<p>Note the <strong>pcscd.service<\/strong> unit. This is the PC\/SC smart-card daemon. Its function is to communicate with smart-card readers. Many Linux hosts\u2014including VMs\u2014have no need for this reader nor the service that is loaded and taking up memory and CPU resources. You can stop this service and disable it, so it will not restart on the next boot. First, check its status:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl status pcscd.service<\/strong>\n\u25cf pcscd.service - PC\/SC Smart Card Daemon\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/pcscd.service; indirect; vendor preset: disabled)\n   Active: active (running) since Fri 2019-05-10 11:28:42 EDT; 3 days ago\n     Docs: man:pcscd(8)\n Main PID: 24706 (pcscd)\n    Tasks: 6 (limit: 4694)\n   Memory: 1.6M\n   CGroup: \/system.slice\/pcscd.service\n           \u2514\u250024706 \/usr\/sbin\/pcscd --foreground --auto-exit\n\nMay 10 11:28:42 testvm1 systemd&#91;1]: Started PC\/SC Smart Card Daemon.<\/code><\/pre>\n\n\n\n<p>This data illustrates the additional information systemd provides versus SystemV, which only reports whether or not the service is running. Note that specifying the <strong>.service<\/strong> unit type is optional. Now stop and disable the service, then re-check its status:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl disable -- now pcscd<\/strong>\nWarning: Stopping pcscd.service, but it can still be activated by:\n  pcscd.socket\nRemoved \/etc\/systemd\/system\/sockets.target.wants\/pcscd.socket.\n&#91;root@testvm1 ~]# systemctl status pcscd\n\u25cf pcscd.service - PC\/SC Smart Card Daemon\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/pcscd.service; indirect; vendor preset: disabled)\n   Active: failed (Result: exit-code) since Mon 2019-05-13 15:23:15 EDT; 48s ago\n     Docs: man:pcscd(8)\n Main PID: 24706 (code=exited, status=1\/FAILURE)\n\nMay 10 11:28:42 testvm1 systemd&#91;1]: Started PC\/SC Smart Card Daemon.\nMay 13 15:23:15 testvm1 systemd&#91;1]: Stopping PC\/SC Smart Card Daemon...\nMay 13 15:23:15 testvm1 systemd&#91;1]: pcscd.service: Main process exited, code=exited, status=1\/FAIL>\nMay 13 15:23:15 testvm1 systemd&#91;1]: pcscd.service: Failed with result 'exit-code'.\nMay 13 15:23:15 testvm1 systemd&#91;1]: Stopped PC\/SC Smart Card Daemon.<\/code><\/pre>\n\n\n\n<p>The short log entry display for most services prevents having to search through various log files to locate this type of information. Check the status of the system runlevel targets\u2014specifying the &#8220;target&#8221; unit type is required:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl status multi-user.target<\/strong>\n\u25cf multi-user.target - Multi-User System\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/multi-user.target; static; vendor preset: disabled)\n   Active: active since Thu 2019-05-09 13:27:22 EDT; 4 days ago\n     Docs: man:systemd.special(7)\n\nMay 09 13:27:22 testvm1 systemd&#91;1]: Reached target Multi-User System.\n&#91;root@testvm1 ~]# systemctl status graphical.target\n\u25cf graphical.target - Graphical Interface\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/graphical.target; indirect; vendor preset: disabled)\n   Active: active since Thu 2019-05-09 13:27:22 EDT; 4 days ago\n     Docs: man:systemd.special(7)\n\nMay 09 13:27:22 testvm1 systemd&#91;1]: Reached target Graphical Interface.\n&#91;root@testvm1 ~]# systemctl status default.target\n\u25cf graphical.target - Graphical Interface\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/graphical.target; indirect; vendor preset: disabled)\n   Active: active since Thu 2019-05-09 13:27:22 EDT; 4 days ago\n     Docs: man:systemd.special(7)\n\nMay 09 13:27:22 testvm1 systemd&#91;1]: Reached target Graphical Interface.<\/code><\/pre>\n\n\n\n<p>The default target is the graphical target. The status of any unit can be checked in this way.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"mounts-the-old-way\">Mounts the old way<\/h2>\n\n\n\n<p>A mount unit defines all of the parameters required to mount a filesystem on a designated mount point. systemd can manage mount units with more flexibility than those using the <strong>\/etc\/fstab<\/strong> filesystem configuration file. Despite this, systemd still uses the <strong>\/etc\/fstab<\/strong> file for filesystem configuration and mounting purposes. systemd uses the <strong>systemd-fstab-generator<\/strong> tool to create transient mount units from the data in the <strong>fstab<\/strong> file.<\/p>\n\n\n\n<p>I&#8217;ll create a new filesystem and a systemd mount unit to mount it. If you have some available disk space on your test system, you can do it along with me. <\/p>\n\n\n\n<p><em>Note that the volume group and logical volume names may be different on your test system. Be sure to use the names that are pertinent to your system.<\/em><\/p>\n\n\n\n<p>You will need to create a partition or logical volume, then make an EXT4 filesystem on it. Add a label to the filesystem, <strong>TestFS<\/strong>, and create a directory for a mount point <strong>\/TestFS<\/strong>. To try this on your own, first, verify that you have free space on the volume group. Here is what that looks like on my VM where I have some space available on the volume group to create a new logical volume:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# lsblk\nNAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT\nsda             8:0    0  120G  0 disk \n\u251c\u2500sda1          8:1    0    4G  0 part \/boot\n\u2514\u2500sda2          8:2    0  116G  0 part \n  \u251c\u2500VG01-root 253:0    0    5G  0 lvm  \/\n  \u251c\u2500VG01-swap 253:1    0    8G  0 lvm  &#91;SWAP]\n  \u251c\u2500VG01-usr  253:2    0   30G  0 lvm  \/usr\n  \u251c\u2500VG01-home 253:3    0   20G  0 lvm  \/home\n  \u251c\u2500VG01-var  253:4    0   20G  0 lvm  \/var\n  \u2514\u2500VG01-tmp  253:5    0   10G  0 lvm  \/tmp\nsr0            11:0    1 1024M  0 rom  \n&#91;root@testvm1 ~]# vgs\n  VG   #PV #LV #SN Attr   VSize    VFree  \n  VG01   1   6   0 wz--n- &lt;116.00g &lt;23.00g<\/code><\/pre>\n\n\n\n<p>Then create a new volume on VG01 named TestFS. It does not need to be large; 1GB is fine. Then create a filesystem, add the filesystem label, and create the mount point:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>lvcreate -L 1G -n TestFS VG01<\/strong>\n  Logical volume \"TestFS\" created.\n&#91;root@testvm1 ~]# <strong>mkfs -t ext4 \/dev\/mapper\/VG01-TestFS<\/strong>\nmke2fs 1.45.3 (14-Jul-2019)\nCreating filesystem with 262144 4k blocks and 65536 inodes\nFilesystem UUID: 8718fba9-419f-4915-ab2d-8edf811b5d23\nSuperblock backups stored on blocks: \n        32768, 98304, 163840, 229376\n\nAllocating group tables: done                            \nWriting inode tables: done                            \nCreating journal (8192 blocks): done\nWriting superblocks and filesystem accounting information: done\n\n&#91;root@testvm1 ~]# <strong>e2label \/dev\/mapper\/VG01-TestFS TestFS<\/strong>\n&#91;root@testvm1 ~]# <strong>mkdir \/TestFS<\/strong><\/code><\/pre>\n\n\n\n<p>Now, mount the new filesystem:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>mount \/TestFS\/<\/strong>\nmount: \/TestFS\/: can't find in \/etc\/fstab.<\/code><\/pre>\n\n\n\n<p>This is what I did and it won&#8217;t work because I didn&#8217;t have an entry in <strong>\/etc\/fstab<\/strong>. You can mount the new filesystem even without the entry in <strong>\/etc\/fstab<\/strong> using both the device name (as it appears in <strong>\/dev<\/strong>) and the mount point. Mounting in this manner is simpler than it used to be\u2014it used to require the filesystem type as an argument. The mount command is now smart enough to detect the filesystem type and mount it accordingly.<\/p>\n\n\n\n<p>Try it again:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>mount \/dev\/mapper\/VG01-TestFS \/TestFS\/<\/strong>\n&#91;root@testvm1 ~]# <strong>lsblk<\/strong>\nNAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT\nsda               8:0    0  120G  0 disk \n\u251c\u2500sda1            8:1    0    4G  0 part \/boot\n\u2514\u2500sda2            8:2    0  116G  0 part \n  \u251c\u2500VG01-root   253:0    0    5G  0 lvm  \/\n  \u251c\u2500VG01-swap   253:1    0    8G  0 lvm  &#91;SWAP]\n  \u251c\u2500VG01-usr    253:2    0   30G  0 lvm  \/usr\n  \u251c\u2500VG01-home   253:3    0   20G  0 lvm  \/home\n  \u251c\u2500VG01-var    253:4    0   20G  0 lvm  \/var\n  \u251c\u2500VG01-tmp    253:5    0   10G  0 lvm  \/tmp\n  \u2514\u2500VG01-TestFS 253:6    0    1G  0 lvm  \/TestFS\nsr0              11:0    1 1024M  0 rom  \n&#91;root@testvm1 ~]#<\/code><\/pre>\n\n\n\n<p>Now the new filesystem is mounted in the proper location. List the mount unit files:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl list-unit-files -t mount<\/strong><\/code><\/pre>\n\n\n\n<p>This command does not show a file for the <strong>\/TestFS<\/strong> filesystem because no file exists for it. The command <strong>systemctl status TestFS.mount<\/strong> does not display any information about the new filesystem either. You can try it using wildcards with the <strong>systemctl status<\/strong> command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl status *mount<\/strong>\n\u25cf usr.mount - \/usr\n   Loaded: loaded (\/etc\/fstab; generated)\n   Active: active (mounted)\n    Where: \/usr\n     What: \/dev\/mapper\/VG01-usr\n     Docs: man:fstab(5)\n           man:systemd-fstab-generator(8)\n\n&lt;SNIP>\n\u25cf TestFS.mount - \/TestFS\n   Loaded: loaded (\/proc\/self\/mountinfo)\n   Active: active (mounted) since Fri 2020-04-17 16:02:26 EDT; 1min 18s ago\n    Where: \/TestFS\n     What: \/dev\/mapper\/VG01-TestFS\n\n\u25cf run-user-0.mount - \/run\/user\/0\n   Loaded: loaded (\/proc\/self\/mountinfo)\n   Active: active (mounted) since Thu 2020-04-16 08:52:29 EDT; 1 day 5h ago\n    Where: \/run\/user\/0\n     What: tmpfs\n\n\u25cf var.mount - \/var\n   Loaded: loaded (\/etc\/fstab; generated)\n   Active: active (mounted) since Thu 2020-04-16 12:51:34 EDT; 1 day 1h ago\n    Where: \/var\n     What: \/dev\/mapper\/VG01-var\n     Docs: man:fstab(5)\n           man:systemd-fstab-generator(8)\n    Tasks: 0 (limit: 19166)\n   Memory: 212.0K\n      CPU: 5ms\n   CGroup: \/system.slice\/var.mount<\/code><\/pre>\n\n\n\n<p>This command provides some very interesting information about your system&#8217;s mounts, and your new filesystem shows up. The <strong>\/var<\/strong> and <strong>\/usr<\/strong> filesystems are identified as being generated from <strong>\/etc\/fstab<\/strong>, while your new filesystem simply shows that it is loaded and provides the location of the info file in the <strong>\/proc\/self\/mountinfo<\/strong> file.<\/p>\n\n\n\n<p>Next, automate this mount. First, do it the old-fashioned way by adding an entry in <strong>\/etc\/fstab<\/strong>. Later, I&#8217;ll show you how to do it the new way, which will teach you about creating units and integrating them into the startup sequence.<\/p>\n\n\n\n<p>Unmount <strong>\/TestFS<\/strong> and add the following line to the <strong>\/etc\/fstab<\/strong> file:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/dev\/mapper\/VG01-TestFS  \/TestFS       ext4    defaults        1 2<\/code><\/pre>\n\n\n\n<p>Now, mount the filesystem with the simpler <strong>mount<\/strong> command and list the mount units again:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>mount \/TestFS<\/strong>\n&#91;root@testvm1 ~]# <strong>systemctl status *mount<\/strong>\n&lt;SNIP>\n\u25cf TestFS.mount - \/TestFS\n   Loaded: loaded (\/proc\/self\/mountinfo)\n   Active: active (mounted) since Fri 2020-04-17 16:26:44 EDT; 1min 14s ago\n    Where: \/TestFS\n     What: \/dev\/mapper\/VG01-TestFS\n&lt;SNIP><\/code><\/pre>\n\n\n\n<p>This did not change the information for this mount because the filesystem was manually mounted. Reboot and run the command again, and this time specify <strong>TestFS.mount<\/strong> rather than using the wildcard. The results for this mount are now consistent with it being mounted at startup:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl status TestFS.mount<\/strong>\n\u25cf TestFS.mount - \/TestFS\n   Loaded: loaded (\/etc\/fstab; generated)\n   Active: active (mounted) since Fri 2020-04-17 16:30:21 EDT; 1min 38s ago\n    Where: \/TestFS\n     What: \/dev\/mapper\/VG01-TestFS\n     Docs: man:fstab(5)\n           man:systemd-fstab-generator(8)\n    Tasks: 0 (limit: 19166)\n   Memory: 72.0K\n      CPU: 6ms\n   CGroup: \/system.slice\/TestFS.mount\n\nApr 17 16:30:21 testvm1 systemd&#91;1]: Mounting \/TestFS...\nApr 17 16:30:21 testvm1 systemd&#91;1]: Mounted \/TestFS.<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"creating-a-mount-unit\">Creating a mount unit<\/h2>\n\n\n\n<p>Mount units may be configured either with the traditional <strong>\/etc\/fstab<\/strong> file or with systemd units. Fedora uses the <strong>fstab<\/strong> file as it is created during the installation. However, systemd uses the <strong>systemd-fstab-generator<\/strong> program to translate the <strong>fstab<\/strong> file into systemd units for each entry in the <strong>fstab<\/strong> file. Now that you know you can use systemd <strong>.mount<\/strong> unit files for filesystem mounting, try it out by creating a mount unit for this filesystem.<\/p>\n\n\n\n<p>First, unmount <strong>\/TestFS<\/strong>. Edit the <strong>\/etc\/fstab<\/strong> file and delete or comment out the <strong>TestFS<\/strong> line. Now, create a new file with the name <strong>TestFS.mount<\/strong> in the <strong>\/etc\/systemd\/system<\/strong> directory. Edit it to contain the configuration data below. The unit file name and the name of the mount point <em>must<\/em> be identical, or the mount will fail:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># This mount unit is for the TestFS filesystem\n# By David Both\n# Licensed under GPL V2\n# This file should be located in the \/etc\/systemd\/system directory\n\n&#91;Unit]\nDescription=TestFS Mount\n\n&#91;Mount]\nWhat=\/dev\/mapper\/VG01-TestFS\nWhere=\/TestFS\nType=ext4\nOptions=defaults\n\n&#91;Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\n<p>The <strong>Description<\/strong> line in the <strong>[Unit]<\/strong> section is for us humans, and it provides the name that&#8217;s shown when you list mount units with <strong>systemctl -t mount<\/strong>. The data in the <strong>[Mount]<\/strong> section of this file contains essentially the same data that would be found in the <strong>fstab<\/strong> file.<\/p>\n\n\n\n<p>Now enable the mount unit:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 etc]# <strong>systemctl enable TestFS.mount<\/strong>\nCreated symlink \/etc\/systemd\/system\/multi-user.target.wants\/TestFS.mount \u2192 \/etc\/systemd\/system\/TestFS.mount.<\/code><\/pre>\n\n\n\n<p>This creates the symlink in the <strong>\/etc\/systemd\/system<\/strong> directory, which will cause this mount unit to be mounted on all subsequent boots. The filesystem has not yet been mounted, so you must &#8220;start&#8221; it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl start TestFS.mount<\/strong><\/code><\/pre>\n\n\n\n<p>Verify that the filesystem has been mounted:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@testvm1 ~]# <strong>systemctl status TestFS.mount<\/strong>\n\u25cf TestFS.mount - TestFS Mount\n   Loaded: loaded (\/etc\/systemd\/system\/TestFS.mount; enabled; vendor preset: disabled)\n   Active: active (mounted) since Sat 2020-04-18 09:59:53 EDT; 14s ago\n    Where: \/TestFS\n     What: \/dev\/mapper\/VG01-TestFS\n    Tasks: 0 (limit: 19166)\n   Memory: 76.0K\n      CPU: 3ms\n   CGroup: \/system.slice\/TestFS.mount\n\nApr 18 09:59:53 testvm1 systemd&#91;1]: Mounting TestFS Mount...\nApr 18 09:59:53 testvm1 systemd&#91;1]: Mounted TestFS Mount.<\/code><\/pre>\n\n\n\n<p>This experiment has been specifically about creating a unit file for a mount, but we&#8217;ll applied to other types of unit files as well in upcoming articles. The details will be different, but the concepts are the same. Yes, I know it is still easier to add a line to the <strong>\/etc\/fstab<\/strong> file than it is to create a mount unit. But this is a good example of how to create a unit file because systemd does not have generators for every type of unit.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"summary\">In summary<\/h2>\n\n\n\n<p>This article looked at systemd units in more detail and how to use the systemctl command to explore and manage units. It also showed how to stop and disable units and create a new systemd mount unit to mount a new filesystem and enable it to initiate during startup.<\/p>\n\n\n\n<p>In the next article in this series, I will take you through a recent problem I had during startup and show you how I circumvented it using systemd.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-wide\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"resources\">Resources<\/h2>\n\n\n\n<p>There is a great deal of information about systemd available on the internet, but much is terse, obtuse, or even misleading. In addition to the resources mentioned in this article, the following webpages offer more detailed and reliable information about systemd startup.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The Fedora Project has a good, practical <a href=\"https:\/\/docs.fedoraproject.org\/en-US\/quick-docs\/understanding-and-administering-systemd\/index.html\" target=\"_blank\" rel=\"noreferrer noopener\">guide<\/a> <a href=\"https:\/\/docs.fedoraproject.org\/en-US\/quick-docs\/understanding-and-administering-systemd\/index.html\" target=\"_blank\" rel=\"noreferrer noopener\">to systemd<\/a>. It has pretty much everything you need to know in order to configure, manage, and maintain a Fedora computer using systemd.<\/li>\n\n\n\n<li>The Fedora Project also has a good <a href=\"https:\/\/fedoraproject.org\/wiki\/SysVinit_to_Systemd_Cheatsheet\" target=\"_blank\" rel=\"noreferrer noopener\">cheat sheet<\/a> that cross-references the old SystemV commands to comparable systemd ones.<\/li>\n\n\n\n<li>For detailed technical information about systemd and the reasons for creating it, check out <a href=\"http:\/\/Freedesktop.org\" target=\"_blank\" rel=\"noreferrer noopener\">Freedesktop.org<\/a>&#8216;s <a href=\"http:\/\/www.freedesktop.org\/wiki\/Software\/systemd\" target=\"_blank\" rel=\"noreferrer noopener\">description of systemd<\/a>.<\/li>\n\n\n\n<li><a href=\"http:\/\/Linux.com\" target=\"_blank\" rel=\"noreferrer noopener\">Linux.com<\/a>&#8216;s &#8220;More systemd fun&#8221; offers more advanced systemd <a href=\"https:\/\/www.linux.com\/training-tutorials\/more-systemd-fun-blame-game-and-stopping-services-prejudice\/\" target=\"_blank\" rel=\"noreferrer noopener\">information and tips<\/a>.<\/li>\n<\/ul>\n\n\n\n<p>There is also a series of deeply technical articles for Linux sysadmins by Lennart Poettering, the designer and primary developer of systemd. These articles were written between April 2010 and September 2011, but they are just as relevant now as they were then. Much of everything else good that has been written about systemd and its ecosystem is based on these papers.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/systemd.html\" target=\"_blank\" rel=\"noreferrer noopener\">Rethinking PID 1<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/systemd-for-admins-1.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part I<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/systemd-for-admins-2.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part II<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/systemd-for-admins-3.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part III<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/systemd-for-admins-4.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part IV<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/three-levels-of-off.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part V<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/changing-roots\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part VI<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/blame-game.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part VII<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/the-new-configuration-files.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part VIII<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/on-etc-sysinit.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part IX<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/instances.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part X<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/0pointer.de\/blog\/projects\/inetd.html\" target=\"_blank\" rel=\"noreferrer noopener\">systemd for Administrators, Part XI<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Units are the basis of everything in systemd so we&#8217;ll explore them in more detail and use the systemctl command to investigate and manage units. <\/p>\n","protected":false},"author":2,"featured_media":3579,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[160],"tags":[258,242,176,257],"class_list":["post-3840","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-systemd","tag-mount-unit","tag-systemctl","tag-systemd","tag-systemd-unit-files"],"modified_by":"David Both","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/3840","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3840"}],"version-history":[{"count":13,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/3840\/revisions"}],"predecessor-version":[{"id":3938,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/3840\/revisions\/3938"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/3579"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3840"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3840"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3840"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}