{"id":9089,"date":"2025-01-02T06:16:08","date_gmt":"2025-01-02T11:16:08","guid":{"rendered":"https:\/\/www.both.org\/?p=9089"},"modified":"2025-01-02T06:16:08","modified_gmt":"2025-01-02T11:16:08","slug":"processing-files-with-find-and-xargs","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=9089","title":{"rendered":"Processing files with ‘find’ and ‘xargs’"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

I manage several websites, including Technically We Write<\/a> and Coaching Buttons<\/a>, and recently I wanted to see how much I had written over the last year. I wanted more than just a count; I also was curious to know how many words I had written for each article, and in total.<\/p>\n\n\n\n

I manage these websites using\u00a0 a static website generator, which means the website content is saved in plain text files. That makes the website very fast, but it also lets me use standard Linux commands to examine the content, including the word count. Here’s how I counted words for the articles I wrote in 2024:<\/p>\n\n\n\n

Finding my articles<\/h2>\n\n\n\n

The website content is stored in a directory for the year, such as 2024<\/strong> for all articles published during 2024. Every article is saved in its own directory, which also contains some plain text metadata; one file is called author<\/strong> and contains the author’s username. My username is jhall<\/strong>.<\/p>\n\n\n\n

With this information, I could look for all files called author<\/strong> under the 2024<\/strong> directory that contained the text jhall<\/strong>, using this find<\/strong> command:<\/p>\n\n\n\n

$ find 2024 -type f -name author -exec grep -q 'jhall' {} \\; -print<\/code><\/pre>\n\n\n\n
The find<\/strong> command operates on a set of files, but usually on a directory tree, to match files and directories that match a pattern. In this command, find<\/strong> looks for all files (-type f<\/code>) with a specific name (-name author<\/code>). For each matching file, find<\/strong> runs the grep<\/strong> command to look for text in the file (-exec grep -q 'jhall' {} \\;<\/code>).<\/p>\n\n\n\n
The grep<\/strong> command runs silently (-q<\/code>) and returns a “success” value if it finds jhall<\/strong> in the file. Note that find<\/strong> uses {}<\/code> as a placeholder for whatever filename matches the earlier pattern, and the command executed by the -exec<\/code> action must end with a semicolon, which I’ve protected from Bash interpretation by using a backslash (\\;<\/code>).<\/p>\n\n\n\n
Putting everything together, I can use find<\/strong> to locate every author<\/strong> file that contains jhall<\/strong>, and prints the filename (the -print<\/code> action). With this find<\/strong> command, and the wc<\/strong> command to count lines in the output, I can see that I wrote about half of the articles for Technically We Write in 2024:<\/p>\n\n\n\n
$ cd technicallywewrite\n\n$ find 2024 -type f -name author -exec grep -q 'jhall' {} \\; -print | wc -l\n47\n\n$ find 2024 -type f -name author -print | wc -l\n104<\/code><\/pre>\n\n\n\n
Running the same set of commands in the other website, I counted that I wrote a third of the articles for Coaching Buttons:<\/p>\n\n\n\n
$ cd coachingbuttons\n\n$ find 2024 -type f -name author -exec grep -q 'jhall' {} \\; -print | wc -l\n24\n\n$ find 2024 -type f -name author -print | wc -l\n76<\/code><\/pre>\n\n\n\n
Counting my words<\/h2>\n\n\n\n
For each matching author<\/strong> file, I wanted to count the words in the article. Each article’s main content is stored in an HTML file called content.html<\/strong>, saved in the same directory as the author<\/strong> metadata file. To get a list of all content files I first save a list of the matching author<\/strong> files, then replace the author<\/strong> text with content.html<\/strong> on each line of the list.<\/p>\n\n\n\n
For example, running this command in the Technically We Write website prints all author<\/strong> files that contain my username, then uses sed<\/strong> to change author<\/strong> (but only at the end of a line) to content.html<\/strong>, before saving the list in a plain text file:<\/p>\n\n\n\n
$ find 2024 -type f -name author -exec grep -q 'jhall' {} \\; -print | sed -e 's\/author$\/content.html\/' > ~\/tww.list<\/code><\/pre>\n\n\n\n
I can also run the same command from the Coaching Buttons website:<\/p>\n\n\n\n
$ find 2024 -type f -name author -exec grep -q 'jhall' {} \\; -print | sed -e 's\/author$\/content.html\/' > ~\/cb.list<\/code><\/pre>\n\n\n\n
To count the total words, I only need to run the wc<\/strong> command against each file in the list. One way to do that is with the $()<\/code> Bash expansion to print the contents of the list of filenames as options to the wc<\/strong> command, showing that I wrote almost 59,600 words in 47 articles for Technically We Write in 2024:<\/p>\n\n\n\n
$ wc -l < ~\/tww.list \n47\n\n$ wc -w $(cat ~\/tww.list) | tail -1\n 59592 total<\/code><\/pre>\n\n\n\n
But running a command with a long list of files (especially where each file might have a long path) can overload the command line. To avoid this, the more typical way to run a command with a list from a file is the xargs<\/strong> command. This runs a command as though you specified each filename on the command line. If the command line gets too long, xargs<\/strong> can automatically break up the list and run the other command multiple times.<\/p>\n\n\n\n
To accommodate possibly running wc<\/strong> more than once (which will result in multiple total<\/strong> output lines) I’ll add the --total=never<\/code> command line option to suppress the total, and pass the output through gawk<\/strong> to print the sum of the word counts:<\/p>\n\n\n\n
$ wc -l < ~\/tww.list \n47\n\n$ xargs wc -w --total=never < ~\/tww.list | gawk '{tot += $1} END {print tot}'\n59592<\/code><\/pre>\n\n\n\n
Running the same commands from the Coaching Buttons website shows that I wrote over 22,000 words in 24 articles during 2024:<\/p>\n\n\n\n
$ wc -l < ~\/cb.list \n24\n\n$ xargs wc -w --total=never < ~\/cb.list | gawk '{tot += $1} END {print tot}'\n22047<\/code><\/pre>\n\n\n\n
Processing files with ‘find’ and ‘xargs’<\/h2>\n\n\n\n
A core tenet of the Linux Philosophy is to store everything in plain text files<\/a>. This makes it easy to work with them using the Linux command line, which provides a ton of useful utilities to process text. Two powerful commands that I used here are find<\/strong> to locate matching files and directories and print the results as a list, and the xargs<\/strong> command to run a command against a list of files.<\/p>\n\n\n\n
If you look at how I’ve written my commands, you can see this in action. I used find<\/strong> to match files that contained my name, and saved the results to a file. Then I used that list with xargs<\/strong> to count the words in all the files, and print the result. This is made possible because each command does one thing<\/em> and operates on plain text<\/em>, making the overall process a series of small steps.<\/p>\n","protected":false},"excerpt":{"rendered":"
Here’s how I used ‘find’ and ‘xargs’ to locate the articles I wrote and count the words.<\/p>\n","protected":false},"author":33,"featured_media":2818,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[100,5],"tags":[104,91],"class_list":["post-9089","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-command-line","category-linux","tag-command-line","tag-linux"],"modified_by":"David Both","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/9089","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9089"}],"version-history":[{"count":2,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/9089\/revisions"}],"predecessor-version":[{"id":9091,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/9089\/revisions\/9091"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/2818"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9089"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9089"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9089"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}