{"id":13156,"date":"2026-01-07T03:00:00","date_gmt":"2026-01-07T08:00:00","guid":{"rendered":"https:\/\/www.both.org\/?p=13156"},"modified":"2025-12-29T18:59:58","modified_gmt":"2025-12-29T23:59:58","slug":"counting-words-from-online-articles","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=13156","title":{"rendered":"Counting words from online articles"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

I recently wrote a week-long series of articles for “DOScember,” highlighting some things you can do with FreeDOS such as editing files with an Emacs-like editor, programming in BASIC, and listening to music. When I was done, I was curious: how much did I write?<\/p>\n\n\n\n

One way that I could tally how much I wrote in that article series is to copy and paste each article<\/em> into a word processor like LibreOffice, and add up the word counts for each article to get a total. And for just seven articles, that might not take too long. But I prefer to follow the Linux philosophy and automate things where I can<\/a>. Here’s how I did it using the command line.<\/p>\n\n\n\n

Start with a list<\/h2>\n\n\n\n

The first step was the only manual one: I copied the URLs for each article and pasted them into a text file, with each URL on a separate line:<\/p>\n\n\n\n

https:\/\/www.both.org\/?p=12944
https:\/\/www.both.org\/?p=12955
https:\/\/www.both.org\/?p=12967
https:\/\/www.both.org\/?p=12971
https:\/\/www.both.org\/?p=12976
https:\/\/www.both.org\/?p=12978
https:\/\/www.both.org\/?p=12986<\/pre>\n\n\n\n
I saved it as list<\/code>, which is a very plain and obvious filename.<\/p>\n\n\n\n
Convert to plain text<\/h2>\n\n\n\n
Linux includes the wc<\/strong> (“word count”) command, which is a standard Unix utility that counts words, lines, and characters in a text file. You can process one or more files at a time; if you examine more than one file, wc<\/strong> will also print a total word count, which is what I wanted.<\/p>\n\n\n\n
But websites transmit data as HTML, which includes a lot of extra markup that will skew my word count. Instead, I preferred to process the articles in plain text so I could get a more accurate word count for each.<\/p>\n\n\n\n
For that, I used pandoc<\/strong>, an open source universal document converter<\/a> that can convert from and to all kinds of formats, including from HTML<\/em> and to plain text<\/em>. If you don’t have pandoc<\/strong> installed on your system already, you can install it using your distribution’s package manager, such as dnf<\/strong> on Fedora:<\/p>\n\n\n\n
$ sudo dnf install pandoc<\/code><\/pre>\n\n\n\n
A neat feature in pandoc<\/strong> is that it can read directly from a website, without having to fetch the HTML document separately. For example, I can use this command to convert just one of the URLs to plain text:<\/p>\n\n\n\n
$ pandoc --from html --to plain https:\/\/www.both.org\/?p=12986 -o f.txt<\/code><\/pre>\n\n\n\n
This saves the content of the Edit text with this Emacs-like editor<\/a> article into a plain text file called f.txt<\/code> with 971 words in it:<\/p>\n\n\n\n
$ wc -w f.txt\n971 f.txt<\/code><\/pre>\n\n\n\n
Count the overhead<\/h2>\n\n\n\n
The plain text file is a complete copy of the text components from the web page, including the website’s header and footer. You can see the extra “header” text by using the head<\/strong> command to view the first 20 lines:<\/p>\n\n\n\n
$ head -20 f.txt\nHome\n\nEdit text with this Emacs-like editor\n\n[]\n\nFreeDOS Fun Text Editors\n\nEdit text with this Emacs-like editor\n\n[]\n\nJim Hall\n\nDecember 27, 2025December 16, 2025\n\nOn Linux, I often use the GNU Emacs editor to write the source code for\nnew programs. I learned GNU Emacs long ago when I was an undergraduate\nstudent, and I still have the \"finger memory\" for all the keyboard\nshortcuts.<\/code><\/pre>\n\n\n\n
Similarly, you can see the extra “footer” text by using the tail<\/strong> command, although I found it easier to use the less<\/strong> command to view the file interactively. I discovered that my text<\/em> was between the “December 27, 2025” article date and the “Bluesky” link, which is the first link in the website’s footer.<\/p>\n\n\n\n
Using the awk<\/strong> command, I stripped out the extra header and footer text, to count just the text from my article:<\/p>\n\n\n\n
awk '\/December 27, 2025\/ {p=1} \/Bluesky\/ {p=0} p==1 {print}' f.txt | wc -w\n760<\/code><\/pre>\n\n\n\n
This is a three-part awk<\/strong> script written on one line: each instruction is written in pattern-action pairs<\/a>. The first pair sets the variable p<\/code> to the value 1 when it matches the text “December 27, 2025” on a line. The second pair sets p<\/code> to 0 when it finds the line with “Bluesky” in it. The third pair prints any line whenever the value of p<\/code> is 1.<\/p>\n\n\n\n
This effectively prints just my text from the article, and uses the wc<\/strong> command to count the words. In the end, my article’s word count was about 760 words. Subtracting 760 from 971 (the total word count) means the “overhead” is about 181 words:<\/p>\n\n\n\n
$ expr 971 - 790\n181<\/code><\/pre>\n\n\n\n
Count all files at once<\/h2>\n\n\n\n
Now that I know the “overhead” for each article, I can tally the total words across all of my articles in the “DOScember” series. I used a for<\/strong> loop at the Bash prompt to convert each article in the list to a plain text file, then used wc<\/strong> to count the words:<\/p>\n\n\n\n
$ n=1; for url in $(cat list); do pandoc --from html --to plain $url -o article$((n++)).txt; done\n$ wc -w article?.txt\n  973 article1.txt\n  976 article2.txt\n 1019 article3.txt\n 1013 article4.txt\n 1009 article5.txt\n 1804 article6.txt\n  971 article7.txt\n 7765 total<\/code><\/pre>\n\n\n\n
The Bash line has several neat features that I should explain:<\/p>\n\n\n\n
    \n
  • Use $( )<\/code> for command substitution<\/em>, to expand a command line into a list. For example, using $(cat list)<\/code> includes the list of URLs in the file named list<\/code><\/li>\n\n\n\n
  • The $(( ))<\/code> performs arithmetic expansion<\/em>. Bash variables are expanded withing having to use an extra $<\/code>, so $((n++))<\/code> means “print the current value of n<\/code> then increment it by one” which allows the for<\/strong> loop to increment the value of the n<\/code> variable as it processes each URL<\/li>\n<\/ul>\n\n\n\n
    Adjust the count by the “overhead”<\/h2>\n\n\n\n
    The total word count is actually off by about 181 words for each article. To calculate the total without the “overhead,” I can use the wc<\/strong> command in a loop, and subtract the extra 181 from each article’s word count while working out the total:<\/p>\n\n\n\n
    $ total=0; for f in article?.txt; do count=$(wc -w --total=only $f); total=$((total + count - 181)); done; echo $total\n6498<\/code><\/pre>\n\n\n\n
    Again, this Bash line uses command substitution and arithmetic expansion. The $(wc -w --total=only $f)<\/code> command substitution runs the wc<\/strong> command to count words from the file in the $f<\/code> variable; the --total=only<\/code> option is a GNU wc<\/strong> extension to only print the total, not the filename. The running total uses $((total + count - 181))<\/code> to add the word count for each article, minus the extra “overhead.”<\/p>\n\n\n\n
    In the end, I found that I had written about 6,498 words for all seven articles. Doing this from the command line was very easy, requiring only the pandoc<\/strong> command to convert the web pages to plain text, and wc<\/strong> to count the words. A few clever Bash commands later, and I had the total.<\/p>\n\n\n\n
    I had my answer in about a minute, including the time to copy and paste the list of URLs, and figuring out what commands to run. In contrast, copying and pasting each article<\/em> into a word processor, counting the words in each, and using a calculator or spreadsheet to tally the total word counts would have taken much longer.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Here’s how I automated counting words from a series of articles.<\/p>\n","protected":false},"author":33,"featured_media":4464,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[100,5],"tags":[104,91],"class_list":["post-13156","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-command-line","category-linux","tag-command-line","tag-linux"],"modified_by":"Jim Hall","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/13156","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=13156"}],"version-history":[{"count":7,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/13156\/revisions"}],"predecessor-version":[{"id":13164,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/13156\/revisions\/13164"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/4464"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=13156"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=13156"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=13156"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}