{"id":3876,"date":"2024-02-22T02:17:00","date_gmt":"2024-02-22T07:17:00","guid":{"rendered":"https:\/\/www.both.org\/?p=3876"},"modified":"2025-01-21T06:53:43","modified_gmt":"2025-01-21T11:53:43","slug":"systemd-9-using-systemd-journals-to-troubleshoot-transient-problems","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=3876","title":{"rendered":"systemd — #9: Using systemd journals to troubleshoot transient problems"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

Find the root cause of hard-to-track errors by diving into your systemd journals.<\/h2>\n\n\n\n

Problem determination can be as much an art as a science, and sometimes, it seems even a little magic can be useful. Everyone has encountered situations where a reported failure could not be reproduced, which is always frustrating for both the user and the system administrator. Even home appliances and automobiles can be obstinate and refuse to fail when the service person shows up.<\/p>\n\n\n\n

Anthropomorphism aside, sysadmins have some tools that can show what has been happening in their Linux computers with varying degrees of granularity. There are tools, like top, htop, glances, sar, iotop, tcpdump, traceroute, mtr, iptraf-ng, df, du, and many more, all of which can display various aspects of a host’s current state, and several of which can produce logs with wide ranges of detail.<\/p>\n\n\n\n

While these tools can be used to find ongoing problems, they are not particularly helpful for transient problems or those with no directly observable symptoms\u2014not observable, at least, until some major and possibly catastrophic problem occurs.<\/p>\n\n\n\n

An important tool I use for problem determination is the system logs\u2014and with systemd, the system journals. The systemd journal is always one of the first tools I turn to when solving a problem, especially the problems that don’t seem to happen when I am watching. It took me a long time at the beginning of my sysadmin career to realize the wealth of information in the log files, and this discovery greatly improved my speed in resolving problems.<\/p>\n\n\n\n

You have already seen some uses of the journalctl<\/code> command in many of the previous articles in this series. In this article, I explore some details about the systemd journal, how it works, and ways to use journalctl<\/code> to locate and identify problems.<\/p>\n\n\n\n

About journals<\/h2>\n\n\n\n

The purpose of any log or journal is to maintain a time-sequenced history of the normal activities of the services and programs that run on a host and to record any errors or warning messages that occur. The log messages used to be maintained in separate files in \/var\/log<\/code>, usually one file for the kernel and separate ones for most of the services running on the host. Unfortunately, the large number of log files could spread out necessary information and delay the discovery of a problem’s root cause. This could be especially time-consuming when you’re trying to determine what was happening in a system when an error occurred.<\/p>\n\n\n\n

The old \/var\/log\/dmesg<\/code> file was usually used for the kernel, but that file was discontinued several years ago in favor of using the dmesg<\/code> command to display the same information and integrating those messages (and more) into the \/var\/log\/messages<\/code> file. This merger of other logs into the messages<\/code> file helped speed problem determination by keeping much of the data in one file, but there were still many services whose logs were not integrated into the more central messages<\/code> file.<\/p>\n\n\n\n

The systemd journal is designed to collect all messages into a single, unified structure that can show a complete picture of everything that happened in a system at and around a specific time or event. Because the events, regardless of the source, are in one place and in time sequence, it is possible to see at a glance everything happening at a specific point or range of times. In my opinion, this is one of the main benefits of systemd journaling.<\/p>\n\n\n\n

About the systemd journal<\/h2>\n\n\n\n

The systemd journaling service is implemented by the systemd-journald<\/code> daemon. According to the systemd-journald<\/code> man page<\/a>:<\/p>\n\n\n\n

\n

systemd-journald<\/code> is a system service that collects and stores logging data. It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources:<\/em><\/p>\n\n\n\n