{"id":9363,"date":"2025-02-04T01:20:00","date_gmt":"2025-02-04T06:20:00","guid":{"rendered":"https:\/\/www.both.org\/?p=9363"},"modified":"2025-01-31T11:03:36","modified_gmt":"2025-01-31T16:03:36","slug":"how-i-reverted-to-name-service-switch-nss-for-name-resolution-and-why","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=9363","title":{"rendered":"How I reverted to Name Service Switch (NSS) for name resolution &#8212; And why"},"content":{"rendered":"<div class=\"pld-like-dislike-wrap pld-template-1\">\r\n    <div class=\"pld-like-wrap  pld-common-wrap\">\r\n    <a href=\"javascript:void(0)\" class=\"pld-like-trigger pld-like-dislike-trigger  \" title=\"\" data-post-id=\"9363\" data-trigger-type=\"like\" data-restriction=\"cookie\" data-already-liked=\"0\">\r\n                        <i class=\"fas fa-thumbs-up\"><\/i>\r\n                <\/a>\r\n    <span class=\"pld-like-count-wrap pld-count-wrap\">    <\/span>\r\n<\/div><\/div>\n<p><strong>There are problems with the new systemd-resolved resolver and that gives us some good reasons to keep or revert to the NSSwitch resolver.<\/strong><\/p>\n\n\n\n<p>The Domain Name Services (DNS)<sup data-fn=\"e1f5c219-39d8-4ea7-a6d6-53ce5be5201b\" class=\"fn\"><a href=\"#e1f5c219-39d8-4ea7-a6d6-53ce5be5201b\" id=\"e1f5c219-39d8-4ea7-a6d6-53ce5be5201b-link\">1<\/a><\/sup> system provides the database used in the translation of Internet locations from human readable hostnames, such as www.example.net, to IP Addresses, like 54.204.39.132, so that our Internet connected computers and other devices can access them. Without these name resolver services it would be nearly impossible to surf the web as freely and easily as we do. As humans, we tend to do better with names like opensource.org while computers do much better with numbers like 104.21.84.214. So we need a translation service to convert the names that are easy for us to the IP addresses that are easy for our computers.<\/p>\n\n\n\n<p>Every computer needs its own resolver service so that it can locate hosts on the local network and the Internet. But it\u2019s complicated.<\/p>\n\n\n\n<p>In this article we look at the details of the historical NSS resolver, tools and techniques for managing it, and how to revert to it when there are problems with the newer systemd-resolved resolver that implements mDNS.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">resolv.conf<\/h2>\n\n\n\n<p>We start with the \/etc\/resolv.conf file because it is the key to determining how the systemd-resolved.service works. This file is an ASCII plain text file that contains a list of up to three domain name servers that are be used to perform hostname resolution into IP addresses. Under systemd-resolved, it&#8217;s actually a link to the \/run\/systemd\/resolve\/stub-resolv.conf file. <\/p>\n\n\n\n<p>The resolv.conf file also contains the domain name to search when a fully qualified domain name (FQDN) isn\u2019t appended to the hostname. For example a fully qualified domain name would be host1.example.com. This can be searched without a problem. But suppose I just use a hostname line host1 and not the domain name. In that case the domain name specified for searches is appended to the hostname.<\/p>\n\n\n\n<p>A typical resolv.conf file under systemd-resolved looks like that in Figure 1. In this case it&#8217;s a link to the \/run\/systemd\/resolve\/stub-resolv.conf file and contains the search domain as well as the IP addresses of the name servers provided by the DHCP server.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># This is \/run\/systemd\/resolve\/stub-resolv.conf managed by man:systemd-resolved(8).\n# Do not edit.\n#\n# This file might be symlinked as \/etc\/resolv.conf. If you're looking at\n# \/etc\/resolv.conf and seeing this text, you have followed the symlink.\n#\n# This is a dynamic resolv.conf file for connecting local clients to the\n# internal DNS stub resolver of systemd-resolved. This file lists all\n# configured search domains.\n#\n# Run \"resolvectl status\" to see details about the uplink DNS servers\n# currently in use.\n#\n# Third party programs should typically not access this file directly, but only\n# through the symlink at \/etc\/resolv.conf. To manage man:resolv.conf(5) in a\n# different way, replace this symlink by a static file or a different symlink.\n#\n# See man:systemd-resolved.service(8) for details about the supported modes of\n# operation for \/etc\/resolv.conf.\n\nnameserver 127.0.0.53\noptions edns0 trust-ad\nsearch both.org<\/code><\/pre>\n\n\n\n<p class=\"has-text-align-left has-medium-font-size\"><em>Figure 1. A typical resolv.conf file when using systemd-resolved.<\/em><\/p>\n\n\n\n<p>The nameserver line of the resolv.conf file is the important one for this article. The IP address 127.0.0.53 is designated as the local host name service resolver. Port 53 on DNS servers is the DNS port and this mDNS resolver only works for the localhost on that IP address. The presence of this line in resolv.conf is one easy way to determine that your host is using mDNS.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Name service strategies<\/h2>\n\n\n\n<p>There are currently three strategies available for use in resolving domain names into IP addresses. Each has its own tools, advantages, and best use cases. Two of these tools require work on the part of the SysAdmin. mDNS, though requiring almost no administrative work, is quite chatty and creates a significant amount of network traffic. It also has its share of problems.<\/p>\n\n\n\n<p>We\u2019ll take a quick peek at the \/etc\/hosts file and Multicast DNS (mDNS). Then we&#8217;ll discuss nss-DNS and reverting to it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><a><\/a>The \/etc\/hosts file<\/h3>\n\n\n\n<p>The \/etc\/hosts<sup data-fn=\"ced33552-9ea0-46a3-903a-33fe6e8ce63e\" class=\"fn\"><a href=\"#ced33552-9ea0-46a3-903a-33fe6e8ce63e\" id=\"ced33552-9ea0-46a3-903a-33fe6e8ce63e-link\">2<\/a><\/sup> file is an ASCII plain-text file that can list the IP addresses of all hosts on the local network and was the first tool used for local network name resolution. It can also be used for non-local hosts.<\/p>\n\n\n\n<p>In small networks the \/etc\/hosts file on each host can be used as a simple local name resolver. The SysAdmin can add and manage entries in the hosts file. Maintaining copies of this file on several hosts can become very time-consuming and errors can cause much confusion and wasted time before they\u2019re found. Although the hosts file can have non-local domains such as www.example.net added to it, if the IP addresses can be discovered, it is a labor intensive tool best suited for use in small local networks and limited to internal hosts.<\/p>\n\n\n\n<p>A default hosts file such as the one in Figure 2 is always present but it would normally contain only the lines needed to enable internal services and commands to translate the localhost hostname to IPV4 address 127.0.0.1 and IPV6 address ::1 \u2013 this is an explicitly defined standard to enable Linux services and commands to work with the local host.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Loopback entries; do not change.\n# For historical reasons, localhost precedes localhost.localdomain:\n127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4\n::1 localhost localhost.localdomain localhost6 localhost6.localdomain6\n# See hosts(5) for proper format and other examples:\n# 192.168.1.10 foo.mydomain.org foo\n# 192.168.1.13 bar.mydomain.org bar<\/code><\/pre>\n\n\n\n<p class=\"has-text-align-left has-medium-font-size\"><em>Figure 2. The default hosts file contains the two standard entries, some additional examples, and a suggestion for obtaining more information.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">mDNS<\/h2>\n\n\n\n<p>Multicast<sup data-fn=\"c85aa7d5-b209-4c29-b246-2e9149a0ee2e\" class=\"fn\"><a href=\"#c85aa7d5-b209-4c29-b246-2e9149a0ee2e\" id=\"c85aa7d5-b209-4c29-b246-2e9149a0ee2e-link\">3<\/a><\/sup> DNS (mDNS<sup data-fn=\"1960d36e-20ec-4258-9679-5a8156b00ff2\" class=\"fn\"><a href=\"#1960d36e-20ec-4258-9679-5a8156b00ff2\" id=\"1960d36e-20ec-4258-9679-5a8156b00ff2-link\">4<\/a><\/sup>) is a relatively new addition to name service resolution. Intended to provide name resolution for local networks that have no internal, central name resolver, mDNS requires no user intervention. In addition to automatic discovery of local hosts, it also uses more traditional name services for access to the Internet. mDNS is now the default name resolution strategy for Fedora and other distributions.<\/p>\n\n\n\n<p>Multicast services like mDNS send out broadcast (multicast) packets that are received and examined by every host on the network. The packet is a request to the computer with the hostname it wants to communicate with, to respond with its IP address so the requesting computer can send further packets directly to that host. Since only one computer (hopefully) has that hostname, only that computer will respond with its IP address and the requesting host enters that hostname\/IP address into its local database. Other computers on the network that use mDNS can also add that data to their own mDNS databases. With mDNS each host keeps its own database. Entries in the database have a TTL (Time To Live) so the entries will expire at the end of their TTL. This means that the host must make another mDNS broadcast request to the network in order to obtain that IP address again.<\/p>\n\n\n\n<p>The cost of this level of automation for local host discovery is a significant amount of network traffic from each host that is intended to discover other hosts on the network. This type of chatty protocol sucks up network bandwidth, uses host system resources, and is not fast relative to the more traditional nss-DNS protocols. The speed penalty was why I first noticed this service was less efficient than nss-DNS.<\/p>\n\n\n\n<p>The \/etc\/resolv.conf and \/etc\/nsswitch.conf files are symbolic (soft) links to the created files. Figure 3 shows these files. The links keep the same date and time as when they were originally included in the installation package. The \/etc\/authselect\/nsswitch.conf file was created during system installation. The ..\/run\/systemd\/resolve\/stub-resolv.conf was created during the most recent boot.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>ll resolv.conf nsswitch.conf<\/strong>\nlrwxrwxrwx. 1 root root 29 Jan 29 10:16 nsswitch.conf -&gt; \/etc\/authselect\/nsswitch.conf\nlrwxrwxrwx. 1 root root 39 Oct 24 10:53 resolv.conf -&gt; ..\/run\/systemd\/resolve\/stub-resolv.conf\n\n# <strong>ll \/etc\/authselect\/nsswitch.conf ..\/run\/systemd\/resolve\/stub-resolv.conf<\/strong>\n-rw-r--r--. 1 root            root            671 Jan 29 10:16 \/etc\/authselect\/nsswitch.conf\n-rw-r--r--. 1 systemd-resolve systemd-resolve 927 Jan 30 09:01 ..\/run\/systemd\/resolve\/stub-resolv.conf<\/code><\/pre>\n\n\n\n<p class=\"has-text-align-left has-medium-font-size\"><em>Figure 3. The resolver files used to configure DNS client services.<\/em><\/p>\n\n\n\n<p>Actually, two versions of the resolv.conf file are created in the \/run\/systemd\/resolve\/ directory each time the host is booted. Figure 4 shows both of these.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@testvm1:\/run\/systemd\/resolve# <strong>cat resolv.conf<\/strong>\n# This is \/run\/systemd\/resolve\/resolv.conf managed by man:systemd-resolved(8).\n# Do not edit.\n#\n# This file might be symlinked as \/etc\/resolv.conf. If you're looking at\n# \/etc\/resolv.conf and seeing this text, you have followed the symlink.\n#\n# This is a dynamic resolv.conf file for connecting local clients directly to\n# all known uplink DNS servers. This file lists all configured search domains.\n#\n# Third party programs should typically not access this file directly, but only\n# through the symlink at \/etc\/resolv.conf. To manage man:resolv.conf(5) in a\n# different way, replace this symlink by a static file or a different symlink.\n#\n# See man:systemd-resolved.service(8) for details about the supported modes of\n# operation for \/etc\/resolv.conf.\n\nnameserver 192.168.0.52\nnameserver 8.8.8.8\nnameserver 8.8.4.4\nsearch both.org\n\n\nroot@testvm1:\/run\/systemd\/resolve# <strong>cat stub-resolv.conf<\/strong>\n# This is \/run\/systemd\/resolve\/stub-resolv.conf managed by man:systemd-resolved(8).\n# Do not edit.cat stub-resolv.conf\n#\n# This file might be symlinkevd as \/etc\/resolv.conf. If you're looking at\n# \/etc\/resolv.conf and seeing this text, you have followed the symlink.\n#\n# This is a dynamic resolv.conf file for connecting local clients to the\n# internal DNS stub resolver of systemd-resolved. This file lists all\n# configured search domains.\n#\n# Run \"resolvectl status\" to see details about the uplink DNS servers\n# currently in use.\n#\n# Third party programs should typically not access this file directly, but only\n# through the symlink at \/etc\/resolv.conf. To manage man:resolv.conf(5) in a\n# different way, replace this symlink by a static file or a different symlink.\n#\n# See man:systemd-resolved.service(8) for details about the supported modes of\n# operation for \/etc\/resolv.conf.\n\nnameserver 127.0.0.53\noptions edns0 trust-ad\nsearch both.org<\/code><\/pre>\n\n\n\n<p class=\"has-text-align-left has-medium-font-size\"><em>Figure 4. The contents of the two resolver files in the \/run\/systemd\/resolve directory.<\/em><\/p>\n\n\n\n<p>The top file in Figure 4 is a more traditional resolv.conf file but it is not used by default. It defines three external servers for name resolution. This information can be obtained from the DHCP server for the network, or from NetworkManager network connection files<sup data-fn=\"4676f49f-e927-46fb-9957-be0073800a39\" class=\"fn\"><a href=\"#4676f49f-e927-46fb-9957-be0073800a39\" id=\"4676f49f-e927-46fb-9957-be0073800a39-link\">5<\/a><\/sup> for static configurations.<\/p>\n\n\n\n<p>The bottom file in Figure 4 is the systemd-resolved version, stub-resolv.conf and it is the target of the \/etc\/resolv.conf symlink. This file defines the local host as the name server for this client host. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">nss-DNS<\/h2>\n\n\n\n<p>This is the historical name service and resolver combination. The Name Service Switch (NSS)<sup data-fn=\"3b93df35-ba82-40b6-accb-5b6cef2d7bc2\" class=\"fn\"><a href=\"#3b93df35-ba82-40b6-accb-5b6cef2d7bc2\" id=\"3b93df35-ba82-40b6-accb-5b6cef2d7bc2-link\">6<\/a><\/sup> resolver performs the client tasks of requesting IP addresses from the global Domain Name Service distributed database. It has the attributes of being fast, easy to use, mature, and well known.<\/p>\n\n\n\n<p>Originally developed by Sun Microsystems as part of their Solaris<sup data-fn=\"b4aae0f8-16ec-40b3-bb81-15dac2619923\" class=\"fn\"><a href=\"#b4aae0f8-16ec-40b3-bb81-15dac2619923\" id=\"b4aae0f8-16ec-40b3-bb81-15dac2619923-link\">7<\/a><\/sup> operating system, NSS was rewritten for the GNU utilities and tools. It\u2019s code is embedded in the GNU glibc<sup data-fn=\"9e709d0c-c9d6-4321-9a03-e71fe18fc702\" class=\"fn\"><a href=\"#9e709d0c-c9d6-4321-9a03-e71fe18fc702\" id=\"9e709d0c-c9d6-4321-9a03-e71fe18fc702-link\">8<\/a><\/sup> library so is always present on all Linux systems. This makes it easy to revert to it when necessary.<\/p>\n\n\n\n<p>Two ASCII plain text files are used to provide the primary configuration for name services. These files, nsswitch.conf and resolv.conf have historical origins having been around since the earliest versions of name service resolvers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">NSSwitch<\/h3>\n\n\n\n<p>As its name implies, the nsswitch \u2013 short for Name Service Switch \u2013 is used to define the database sources and order in which name-service information is obtained. It&#8217;s used for much more than just name services but we&#8217;ll ignore those other tasks in this article.<\/p>\n\n\n\n<p><strong>Unicast<\/strong> services like nss-DNS use a single server that maintains the entire database. If a host needs the IP address of another host on the network, it sends a unicast packet only to the name server requesting the IP address from it. The name server responds only to the requesting host with a packet containing that IP address.<\/p>\n\n\n\n<p>The NSS facility is a tool that is used by a number of services that need name resolver data. Using NSS based on the data in the \/etc\/nsswitch.conf configuration file, it aids them in locating the appropriate configuration and name resolution sources in a specified sequence.<\/p>\n\n\n\n<p>The sequences listed for each service in the nsswitch.conf file can be changed, and can differ between distributions. This file can also be modified to meet local needs. Until relatively recently, I&#8217;ve never needed to change anything about this file and it is a good place to start problem determination if there seems to be a problem with name resolution that can\u2019t be otherwise explained.<\/p>\n\n\n\n<p>Figure 5 shows the modern default version of nsswitch.conf. This file is created during Linux startup. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Generated by authselect\n# Do not modify this file manually, use authselect instead. Any user changes will be overwritten.\n# You can stop authselect from managing your configuration by calling 'authselect opt-out'.\n# See authselect(8) for more details.\n\n# In order of likelihood of use to accelerate lookup.\npasswd:     files sss systemd\nshadow:     files\ngroup:      files &#91;SUCCESS=merge] sss &#91;SUCCESS=merge] systemd\nhosts:      files mdns4_minimal &#91;NOTFOUND=return] resolve &#91;!UNAVAIL=return] myhostname dns\nservices:   files sss\nnetgroup:   files sss\nautomount:  files sss\n\naliases:    files\nethers:     files\ngshadow:    files\nnetworks:   files dns\nprotocols:  files\npublickey:  files\nrpc:        files<\/code><\/pre>\n\n\n\n<p class=\"has-text-align-left has-medium-font-size\"><em>Figure 5. The default version of nsswitch.conf.<\/em><\/p>\n\n\n\n<p>Look at the hosts database entry in the data stream. The first entry is \u201cfiles\u201d which means that the resolver is to search first the local database. The database isn\u2019t explicitly specified here but it is the \/etc\/hosts file that we experimented with above. <\/p>\n\n\n\n<p>If a match is not found, the resolver moves on to the next entry which is \u201cmyhostname.\u201d This provides name resolution for the locally configured system hostname as contained in the $HOSTNAME environment variable.<\/p>\n\n\n\n<p>The myhostname entry in this line is new and is intended to provide fail-safe resolution of any local hostnames such as localhost and localhost.localdomain. This is built into the resolver code and is not dependent on the entries in the \/etc\/hosts file. <\/p>\n\n\n\n<p>Because all of these entries are sequence-sensitive, if an entry is found for a host name in the \/etc\/hosts database, that takes precedent over any other, later entries.<\/p>\n\n\n\n<p>The nsswitch.conf file in Figure 5 mentions a tool called authselect that&#8217;s used to determine the content of the nsswitch.conf file. It also describes the simple command used to opt-out of of using authselect to manage the hosts name services; we&#8217;ll use that as we extract ourselves from systemd-resolved. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fedora name resolution fails when using systemd-resolved<\/h2>\n\n\n\n<p>When I first installed Fedora 33, one of the major changes in that release, a switch from the nss resolver to systemd-resolved caused me a significant amount of trouble and borked my entire network. <\/p>\n\n\n\n<p>The change from the venerable nsswitch and NetworkManager to systemd-resolved damaged and slowed name services. The result of this resolver change was apparent in a number of symptoms. Inability to find the addresses of many remote servers resulting in timeouts was the most noticeable. When the connections were made, They were very slow to respond. I didn\u2019t really understand how much slower until after I fixed the problem.<\/p>\n\n\n\n<p>I run my own name server using BIND. I started this soon after I began learning Linux as a way to overcome the horrible name services provided by my series of ISPs. They were very slow and would fail intermittently, always at the most inopportune times for me. It was far less trouble for me to start my own name service and that has been the case \u2014 until systemd-resolved forced its way onto my Fedora systems. All of them.<\/p>\n\n\n\n<p>I also found many named errors in the systemd journal. A small sample is shown in Figure 6.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Sep 15 01:41:27 yorktown.both.org named&#91;1464]: SERVFAIL unexpected RCODE resolving 'dns-02.as49870.net\/A\/IN': 116.203.70.186#53\nSep 15 02:00:14 yorktown.both.org named&#91;1464]: loop detected resolving 'ns6.pinterest.com\/A'\nSep 15 02:00:14 yorktown.both.org named&#91;1464]: loop detected resolving 'ns5.pinterest.com\/A'\nSep 15 02:24:03 yorktown.both.org named&#91;1464]: REFUSED unexpected RCODE resolving '218.67.58.103.in-addr.arpa\/PTR\/IN': 103.58.117.2#53\nSep 15 02:34:38 yorktown.both.org named&#91;1464]:   validating in-addr.arpa\/SOA: got insecure response; parent indicates it should be secure\nSep 15 02:34:53 yorktown.both.org named&#91;1464]: REFUSED unexpected RCODE resolving '29.140.3.106.in-addr.arpa\/PTR\/IN': 101.251.253.10#53<\/code><\/pre>\n\n\n\n<p><em>Figure 6. A sampling of the name service errors from the systemd journal indicates resolver problems.<\/em><\/p>\n\n\n\n<p>Several on-line resources indicate that these errors are caused by configuration issues for the target domain\u2019s name services. The comments on these articles suggest that the domain admins should fix their problems but the commenters recognize that\u2019s unlikely to happen. <\/p>\n\n\n\n<p>We must implement our own changes to fix someone else\u2019s problem. This is not as uncommon as you might hope for SysAdmins. We&#8217;re always fixing other people&#8217;s problems. <\/p>\n\n\n\n<p>To summarize the root cause of several related resolver problems, much of the progress towards the goal of \u201cLinux on the desktop,\u201d is to make things easier for the end user. This objective has resulted in many changes introduced along with various systemd services that perform a more automatic configuration of the host\u2019s network connection. This has repeatedly introduced new problems into the name resolution process. I\u2019m pretty sure that these problems would cause all but the most technical desktop users to abandon any attempt to use Linux.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Resolving the problem<\/h2>\n\n\n\n<p>It takes several steps to resolve this problem. This section describes each step and why it\u2019s needed as part of the complete solution.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Stop and disable the Avahi service<\/strong><\/li>\n<\/ol>\n\n\n\n<p>The Avahi web site describes Avahi better than I can.<\/p>\n\n\n\n<p>&#8220;<em>Avahi is a system which facilitates service discovery on a local network via the mDNS\/DNS-SD protocol suite. This enables you to plug your laptop or computer into a network and instantly be able to view other people who you can chat with, find printers to print to or find files being shared. Compatible technology is found in Apple MacOS X (branded \u201cBonjour\u201d and sometimes &#8216;Zeroconf&#8217;).<\/em>&#8220;<\/p>\n\n\n\n<p>Avahi is the basis for many of the good things that end user simplification can support, however it\u2019s not going to be needed when we disable some of the other services that it supports.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>systemctl disable --now avahi-daemon.service<\/strong><\/code><\/pre>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li><strong>Stop and disable the Avahi daemon<\/strong><\/li>\n<\/ol>\n\n\n\n<p>The Avahi daemon socket is a part of the Avahi service. When a program requests Avahi services, it does so through the daemon rather than directly to the service itself. The socket then send the request to the service. Other systemd services also work this way. This won\u2019t be required since we\u2019ve disabled the Avahi service. A socket like this could also cause the service it belongs to to start even though the service is disabled. We don\u2019t want to allow that to happen.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>systemctl disable --now avahi-daemon.socket<\/strong><\/code><\/pre>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li><strong>Stop and disable the systemd-resolved service<\/strong><\/li>\n<\/ol>\n\n\n\n<p>The systemd-resolved service is the root cause of the problems we\u2019re having so we disable it. The systemd-resolved man page states its purpose succinctly.<\/p>\n\n\n\n<p>&#8220;<em>systemd-resolved is a system service that provides network name resolution to local applications. It implements a caching and validating DNS\/DNSSEC stub resolver, as well as an LLMNR and MulticastDNS resolver and responder.<\/em>&#8220;<\/p>\n\n\n\n<p>The man page then proceeds to describe the interfaces it exposes to programs and a high-level statement about how to access it as a resolver. This service is the root cause of the problem and we disable it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>systemctl disable --now systemd-resolved.service<\/strong><\/code><\/pre>\n\n\n\n<ol start=\"4\" class=\"wp-block-list\">\n<li><strong>Delete the \/etc\/resolv.conf link<\/strong><\/li>\n<\/ol>\n\n\n\n<p>The NetworkManager service examines \/etc\/resolv.conf file to determine which servers to use for name resolution. Up to three servers are supported in a simple list format. This file also defines the name of the domain in which to search for host names if a simple hostname is provided, i.e., host, rather than the FQDN (Fully Qualified Domain Name) , i.e., host.example.com.<\/p>\n\n\n\n<p>Only one nameserver is specified in this file; the local host. The systemd-resolved service and Avahi search the local network for other local hosts using systemd-resolved and can configure name resolution so that the hosts can talk amongst themselves. If there is a name server found, such as that provided on wired or wireless routers, it can use that to perform name resolution for external hosts such as www.both.org.<\/p>\n\n\n\n<p>If there\u2019s no locally accessible name server, external name resolution is not possible. This is what happened to me at the beach. The local name server at the hotel was intermittent so no name resolution was possible. I could. however, still ping remote hosts such as www.both.org using the IP address. Yes \u2014 this is an edge case. But it clearly does happen.<\/p>\n\n\n\n<p>So we delete the existing \/etc\/resolve.conf link. We won\u2019t create a new resolv.conf file because once we get the rest of this mess sorted, NetworkManager will create a usable one. The NetworkManager service is responsible for creating the \/etc\/resolv.conf file at boot time if it doesn\u2019t exist. If systemd-resolved is running, the default link is created, which is not the one we want.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>rm -f \/etc\/resolv.conf<\/strong><\/code><\/pre>\n\n\n\n<ol start=\"5\" class=\"wp-block-list\">\n<li><strong>Delete the \/etc\/nsswitch.conf link<\/strong><\/li>\n<\/ol>\n\n\n\n<p>The man page for nsswitch.conf provides a brief description of the uses for this file.<\/p>\n\n\n\n<p>&#8220;<em>The Name Service Switch (NSS) configuration file, \/etc\/nsswitch.conf, is used by the GNU C Library and certain other applications to determine the sources from which to obtain name-service information in a range of categories, and in what order. Each category of information is identified by a database name.<\/em>&#8220;<\/p>\n\n\n\n<p>My testing determined that the \/etc\/nsswitch.conf file shown at the beginning of this article is directly responsible for the slow resolution speeds I encountered. If you look back at that file, the logic in the hosts line seems to be the cause.<\/p>\n\n\n\n<p>We don\u2019t need \u2014 or really want \u2014 to delete the actual nsswitch.conf file. We\u2019ll just delete the symbolic link (symlink) in \/etc.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>rm -f \/etc\/nsswitch.conf<\/strong><\/code><\/pre>\n\n\n\n<ol start=\"5\" class=\"wp-block-list\">\n<li><strong>Create a revised nsswitch.conf<\/strong><\/li>\n<\/ol>\n\n\n\n<p>Since we deleted the symlink to this file in the previous step, we need to create a new version of this file, but it won\u2019t be a symlink. After the next step, it won\u2019t be changed or overwritten. I copied the original from \/etc\/authselect\/nsswitch.conf to \/etc so that it\u2019s not a symlink. I made my changes to this file. Note that some lines are wrapped. I deleted the first line shown in Figure 5 &#8212; since we are going to make changes to this file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Do not modify this file manually, use authselect instead. Any user changes will be overwritten.\n# You can stop authselect from managing your configuration by calling 'authselect opt-out'.\n# See authselect(8) for more details.\n\n# In order of likelihood of use to accelerate lookup.\npasswd:     files sss systemd\nshadow:     files\ngroup:      files &#91;SUCCESS=merge] sss &#91;SUCCESS=merge] systemd\n# hosts:      files mdns4_minimal &#91;NOTFOUND=return] resolve &#91;!UNAVAIL=return] myhostname dns\nhosts:      files myhostname dns\nservices:   files sss\nnetgroup:   files sss\nautomount:  files sss\n\naliases:    files\nethers:     files\ngshadow:    files\nnetworks:   files dns\nprotocols:  files\npublickey:  files\nrpc:        files<\/code><\/pre>\n\n\n\n<p><em>Figure 7. The revised nsswitch.conf file with the changes shown in bold.<\/em><\/p>\n\n\n\n<p>Then I commented out the original hosts line and added a new one with the desired changes. This leaves an easy option for returning to the original configuration.<\/p>\n\n\n\n<ol start=\"7\" class=\"wp-block-list\">\n<li><strong>Opt out of authselect<\/strong><\/li>\n<\/ol>\n\n\n\n<p>In order to prevent authselect from changing \/etc\/nsswitch, we opt out.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>authselect opt-out<\/strong><\/code><\/pre>\n\n\n\n<p>You can safely ignore the first line of the file and make changes to it manually.<\/p>\n\n\n\n<ol start=\"8\" class=\"wp-block-list\">\n<li><strong>Restart NetworkManager<\/strong><\/li>\n<\/ol>\n\n\n\n<p>The last step is to restart the NetworkManager service. This will create a new \/etc\/resolv.conf, and utilize the new nsswitch.conf file we created.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># <strong>systemctl restart NetworkManager.service<\/strong><\/code><\/pre>\n\n\n\n<p>Every time it\u2019s restarted, whether at Linux startup or a command line restart, NetworkManager creates the new \/etc\/resolv.conf using the data provided by the DHCP server for the network or from NetworkManager Connection Files. For many stand-alone systems in home and office, this DHCP server is usually the wired\/wireless router.<\/p>\n\n\n\n<p>The revised resolv.conf file for my VM is shown in Figure 8 and it contains the information obtained from my DHCP server.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Generated by NetworkManager\nsearch both.org\nnameserver 192.168.0.52\nnameserver 8.8.8.8\nnameserver 8.8.4.4<\/code><\/pre>\n\n\n\n<p class=\"has-text-align-left has-medium-font-size\"><em>Figure 8. The resolv.conf file generated by NetworkManager from the connection files I configured for this interface.<\/em><\/p>\n\n\n\n<p>The first name server in the list is my internal name server. The second and third are fallback external name servers. I use Google name servers because I trust them more than my ISP\u2019s whichever ISP I have been using at a given time. <\/p>\n\n\n\n<p>The server at 192.168.0.52 is my internal server. It handles name services for the local network with zone files and uses the top level DNS servers for external network name resolution. If you want to override the network configuration provided by a DHCP server, you can explicitly configure the network interface using NetworkManager Connection Files.<\/p>\n\n\n\n<p>At this point name services are using NSSwitch with a decent and reliable resolv.conf file. I tested this using a few pings to internal and external hosts. I always use the domain example.com for external testing like this, even if it&#8217;s only a few pings.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">My thoughts about nsswitch<\/h2>\n\n\n\n<p>Based on my experimentation, the nsswitch.conf file generated by authselect, and dependence on the Avahi daemon to locate services such as network configuration and other hosts on the local network, slows the entire process to the point of uselessness. I think aiming at Linux on the desktop is an admirable goal. While this may work \u2014 once the problems are resolved \u2014 for minimally-technical users, its can cause issues for those of us SysAdmins who\u2019ve had things well-configured and working for years.<\/p>\n\n\n\n<p>In previous attempts to fix the resolver problems, I was able to resolve the issues at hand, but after this last round of extreme symptoms, I finally realized the extent of the multiple root causes. Part of the issue is that various systemd name service tools have been added over a period of time. Here we&#8217;re able to consider all of the currently known root causes for name service resolution issues related to systemd-resolved and to resolve them.<\/p>\n\n\n\n<p>I hadn\u2019t realized how lengthy the delays in name resolution were until after resolving this problem. Web pages that took minutes to load \u2014 and some never did with all the external links they use to load pictures and advertisements \u2014 now take only a second or so. Tests using the <strong>dig<\/strong> command show name resolution times of around 100 milliseconds (msec) for sites that were not currently in the cache of my name server.<\/p>\n\n\n\n<p>I suggest reading the man pages for each of the files mentioned in this article as there is additional important information about each that can be very helpful.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Summary<\/h2>\n\n\n\n<p>Name resolution is a critical service for hosts connected to the Internet, and systemd-resolved is the most current implementation of a resolver for many Linux distributions. Its intended use case is for small networks with no internal name service server, usually because no SysAdmin is available to create a DNS or DHCP server.<\/p>\n\n\n\n<p>While it can work well in that limited use case, it has failed miserably in my use case in which I have both DNS and DHCP services on my internal server. Reverting to the nss-DNS resolver has resolved my name resolution problems.<\/p>\n\n\n\n<p>Don\u2019t misunderstand me. I\u2019m not suddenly saying I hate systemd. That\u2019s not it at all. What I am saying is that the unintended consequences of these decisions can cause SysAdmins pain as they try to determine what\u2019s changed and how to fix it. In this case it\u2019s simply that what\u2019s good for one set of users is not necessarily good for other sets of users. My use case is significantly different from that of non-technical users.<\/p>\n\n\n\n<p>Networks of any size with DHCP or static network configurations are much better suited to nss-DNS name resolution services.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n<ol class=\"wp-block-footnotes\"><li id=\"e1f5c219-39d8-4ea7-a6d6-53ce5be5201b\">Both, David, <em>Introduction to Domain Name Services (DNS)<\/em>, <a href=\"https:\/\/www.both.org\/?p=4759\">https:\/\/www.both.org\/?p=4759<\/a> <a href=\"#e1f5c219-39d8-4ea7-a6d6-53ce5be5201b-link\" aria-label=\"Jump to footnote reference 1\">\u21a9\ufe0e<\/a><\/li><li id=\"ced33552-9ea0-46a3-903a-33fe6e8ce63e\">Wikipedia, <em>Hosts file<\/em>,<a href=\"https:\/\/en.wikipedia.org\/wiki\/Hosts_(file)\">https:\/\/en.wikipedia.org\/wiki\/Hosts_(file)<\/a> <a href=\"#ced33552-9ea0-46a3-903a-33fe6e8ce63e-link\" aria-label=\"Jump to footnote reference 2\">\u21a9\ufe0e<\/a><\/li><li id=\"c85aa7d5-b209-4c29-b246-2e9149a0ee2e\">Wikipedia, <em>IP Multicast<\/em>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/IP_multicast\">https:\/\/en.wikipedia.org\/wiki\/IP_multicast<\/a> <a href=\"#c85aa7d5-b209-4c29-b246-2e9149a0ee2e-link\" aria-label=\"Jump to footnote reference 3\">\u21a9\ufe0e<\/a><\/li><li id=\"1960d36e-20ec-4258-9679-5a8156b00ff2\">Wikipedia, <em>Multicast DNS<\/em>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Multicast_DNS\">https:\/\/en.wikipedia.org\/wiki\/Multicast_DNS<\/a> <a href=\"#1960d36e-20ec-4258-9679-5a8156b00ff2-link\" aria-label=\"Jump to footnote reference 4\">\u21a9\ufe0e<\/a><\/li><li id=\"4676f49f-e927-46fb-9957-be0073800a39\">Both, David, <em>NetworkManager on Linux: #3 \u2014 How I migrated to NetworkManager Connection Files for configuration<\/em>, <a href=\"https:\/\/www.both.org\/?p=4863\">https:\/\/www.both.org\/?p=4863<\/a> <a href=\"#4676f49f-e927-46fb-9957-be0073800a39-link\" aria-label=\"Jump to footnote reference 5\">\u21a9\ufe0e<\/a><\/li><li id=\"3b93df35-ba82-40b6-accb-5b6cef2d7bc2\">GNU.org, <em>System Databases and Name Service Switch<\/em>, <a href=\"https:\/\/www.gnu.org\/software\/libc\/manual\/html_node\/Name-Service-Switch.html\">https:\/\/www.gnu.org\/software\/libc\/manual\/html_node\/Name-Service-Switch.html<\/a> <a href=\"#3b93df35-ba82-40b6-accb-5b6cef2d7bc2-link\" aria-label=\"Jump to footnote reference 6\">\u21a9\ufe0e<\/a><\/li><li id=\"b4aae0f8-16ec-40b3-bb81-15dac2619923\">Wikipedia, <em>Oracle Solaris<\/em>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Oracle_Solaris\">https:\/\/en.wikipedia.org\/wiki\/Oracle_Solaris<\/a> <a href=\"#b4aae0f8-16ec-40b3-bb81-15dac2619923-link\" aria-label=\"Jump to footnote reference 7\">\u21a9\ufe0e<\/a><\/li><li id=\"9e709d0c-c9d6-4321-9a03-e71fe18fc702\">General C Library. A collection of basic Linux functions that are always present. <a href=\"#9e709d0c-c9d6-4321-9a03-e71fe18fc702-link\" aria-label=\"Jump to footnote reference 8\">\u21a9\ufe0e<\/a><\/li><\/ol>\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>There are problems with the new systemd-resolved resolver and that gives us some good reasons to keep or<\/p>\n","protected":false},"author":2,"featured_media":6962,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":"[{\"content\":\"Both, David, <em>Introduction to Domain Name Services (DNS)<\/em>, <a href=\\\"https:\/\/www.both.org\/?p=4759\\\">https:\/\/www.both.org\/?p=4759<\/a>\",\"id\":\"e1f5c219-39d8-4ea7-a6d6-53ce5be5201b\"},{\"content\":\"Wikipedia, <em>Hosts file<\/em>,<a href=\\\"https:\/\/en.wikipedia.org\/wiki\/Hosts_(file)\\\">https:\/\/en.wikipedia.org\/wiki\/Hosts_(file)<\/a>\",\"id\":\"ced33552-9ea0-46a3-903a-33fe6e8ce63e\"},{\"content\":\"Wikipedia, <em>IP Multicast<\/em>, <a href=\\\"https:\/\/en.wikipedia.org\/wiki\/IP_multicast\\\">https:\/\/en.wikipedia.org\/wiki\/IP_multicast<\/a>\",\"id\":\"c85aa7d5-b209-4c29-b246-2e9149a0ee2e\"},{\"content\":\"Wikipedia, <em>Multicast DNS<\/em>, <a href=\\\"https:\/\/en.wikipedia.org\/wiki\/Multicast_DNS\\\">https:\/\/en.wikipedia.org\/wiki\/Multicast_DNS<\/a>\",\"id\":\"1960d36e-20ec-4258-9679-5a8156b00ff2\"},{\"content\":\"Both, David, <em>NetworkManager on Linux: #3 \u2014 How I migrated to NetworkManager Connection Files for configuration<\/em>, <a href=\\\"https:\/\/www.both.org\/?p=4863\\\">https:\/\/www.both.org\/?p=4863<\/a>\",\"id\":\"4676f49f-e927-46fb-9957-be0073800a39\"},{\"content\":\"GNU.org, <em>System Databases and Name Service Switch<\/em>, <a href=\\\"https:\/\/www.gnu.org\/software\/libc\/manual\/html_node\/Name-Service-Switch.html\\\">https:\/\/www.gnu.org\/software\/libc\/manual\/html_node\/Name-Service-Switch.html<\/a>\",\"id\":\"3b93df35-ba82-40b6-accb-5b6cef2d7bc2\"},{\"content\":\"Wikipedia, <em>Oracle Solaris<\/em>, <a href=\\\"https:\/\/en.wikipedia.org\/wiki\/Oracle_Solaris\\\">https:\/\/en.wikipedia.org\/wiki\/Oracle_Solaris<\/a>\",\"id\":\"b4aae0f8-16ec-40b3-bb81-15dac2619923\"},{\"content\":\"General C Library. A collection of basic Linux functions that are always present.\",\"id\":\"9e709d0c-c9d6-4321-9a03-e71fe18fc702\"}]"},"categories":[376,5,576,89],"tags":[694,695,698,693,699,700,696,256],"class_list":["post-9363","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-configuration","category-linux","category-name-services","category-system-administration","tag-etc-hosts","tag-hosts","tag-mdns","tag-name-resolution","tag-nss-dns","tag-nsswitch-conf","tag-resolv-conf","tag-systemd-resolved"],"modified_by":"David Both","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/9363","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9363"}],"version-history":[{"count":83,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/9363\/revisions"}],"predecessor-version":[{"id":9483,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/9363\/revisions\/9483"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/6962"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9363"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9363"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9363"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}