{"id":10802,"date":"2025-05-30T03:00:00","date_gmt":"2025-05-30T07:00:00","guid":{"rendered":"https:\/\/www.both.org\/?p=10802"},"modified":"2025-05-28T20:47:42","modified_gmt":"2025-05-29T00:47:42","slug":"extracting-text-with-awk","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=10802","title":{"rendered":"Extracting text with awk"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

The awk<\/strong> script interpreter is a very handy tool for systems administrators, and anyone else who uses Linux at the command line. With awk, you can solve a tricky problem with a 1-line script.<\/p>\n\n\n\n

Recently, I needed to extract content from an HTML document. The details of the HTML page are not important, only that I wanted to get just the body text<\/em> from the document. In HTML, the body<\/em> is defined by the <body><\/code> and <\/body><\/code> tags, in a larger document that might look like this:<\/p>\n\n\n\n

<\/a><!DOCTYPE html>\n<\/a><html lang=\"en\">\n<\/a>  <head>\n<\/a>    <title>...<\/title>\n<\/a>    ...\n<\/a>  <\/head>\n<\/a>  <body>\n<\/a>\n<\/a>  ...\n<\/a>\n<\/a>  <\/body>\n<\/a><\/html><\/code><\/pre>\n\n\n\n
A sample HTML file<\/h2>\n\n\n\n
Let\u2019s say I have a very simple HTML document that contains only a few lines of text in the body. I can simulate this with the pandoc<\/strong> command, which generates over 170 lines of text (mostly stylesheet information):<\/p>\n\n\n\n
$ echo 'Hello there!' | pandoc --from markdown --to html --standalone --metadata title='Hello there' | wc -l\n177<\/code><\/pre>\n\n\n\n
But the <body><\/code> section is at the end of that long file. We can preview it by sending the output through the tail<\/strong> command:<\/p>\n\n\n\n
$ echo 'Hello there!' | pandoc --from markdown --to html --standalone --metadata title='Hello there' | tail\n    .display.math{display: block; text-align: center; margin: 0.5rem auto;}\n  <\/style>\n<\/head>\n<body>\n<header id=\"title-block-header\">\n<h1 class=\"title\">Hello there<\/h1>\n<\/header>\n<p>Hello there!<\/p>\n<\/body>\n<\/html><\/code><\/pre>\n\n\n\n
Extracting the text<\/h2>\n\n\n\n
In my case, I wanted to print only the lines between<\/em> the <body><\/code> and <\/body><\/code> tags. I decided to write a short awk script to do this. To write it, I applied a simple pattern-and-action pairing:<\/p>\n\n\n\n
    \n
  1. When awk finds <body><\/code> or <\/body><\/code>, increment a counter variable called body<\/code><\/li>\n\n\n\n
  2. This means the body<\/code> variable will be zero for any content before <body><\/code><\/li>\n\n\n\n
  3. .. and body<\/code> will have the value 1<\/strong> for content between <body><\/code> and <\/body><\/code><\/li>\n\n\n\n
  4. .. and body<\/code> will have the value 2<\/strong> for any content after <\/body><\/code><\/li>\n<\/ol>\n\n\n\n
    To print the content between <body><\/code> and <\/body><\/code>, the awk script needs to print lines when the variable body<\/code> is equal to 1<\/strong>:<\/p>\n\n\n\n
    $ echo 'Hello there!' | pandoc --from markdown --to html --standalone --metadata title='Hello there' | awk '\/<\\\/?body>\/ {body++} body==1 {print}'\n<body>\n<header id=\"title-block-header\">\n<h1 class=\"title\">Hello there<\/h1>\n<\/header>\n<p>Hello there!<\/p><\/code><\/pre>\n\n\n\n
    For what I needed to do, that was good enough. I didn\u2019t mind that the script did not print the trailing <\/body><\/code> tag. Unfortunately, there\u2019s no simple way to modify this<\/em> version of the script to either print both<\/em> tags, or print no<\/em> tags. If we swap the body++<\/code> and print<\/code> statements, we won\u2019t print the <body><\/code> tag but will print <\/body><\/code> at the end:<\/p>\n\n\n\n
    $ echo 'Hello there!' | pandoc --from markdown --to html --standalone --metadata title='Hello there' | awk 'body==1 {print} \/<\\\/?body>\/ {body++}'\n<header id=\"title-block-header\">\n<h1 class=\"title\">Hello there<\/h1>\n<\/header>\n<p>Hello there!<\/p>\n<\/body><\/code><\/pre>\n\n\n\n
    If you don\u2019t want to include the <body><\/code> and <\/body><\/code> tags, you\u2019ll need to modify this awk script to detect the opening and closing tags separately. Let\u2019s modify this script to set body<\/code> to 1<\/strong> when it finds the <body><\/code> line, and back to zero when it finds the <\/body><\/code> line. We can strategically place the print<\/code> statement to not print the tags:<\/p>\n\n\n\n
    $ echo 'Hello there!' | pandoc --from markdown --to html --standalone --metadata title='Hello there' | awk '\/<\\\/body>\/ {body=0} body==1 {print} \/<body>\/ {body=1}'\n<header id=\"title-block-header\">\n<h1 class=\"title\">Hello there<\/h1>\n<\/header>\n<p>Hello there!<\/p><\/code><\/pre>\n\n\n\n
    Awk is a versatile tool<\/h2>\n\n\n\n
    This is just one way to use awk to extract text from a document. I find that awk is a versatile tool that I can apply to most problems that involve text. If I need to print or manipulate text based on a pattern, awk is usually the right tool for the job.<\/p>\n","protected":false},"excerpt":{"rendered":"
    The awk script interpreter is a very handy tool for systems administrators, and anyone else who uses Linux<\/p>\n","protected":false},"author":33,"featured_media":3579,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[100,5],"tags":[104,91],"class_list":["post-10802","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-command-line","category-linux","tag-command-line","tag-linux"],"modified_by":"Jim Hall","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/10802","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10802"}],"version-history":[{"count":1,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/10802\/revisions"}],"predecessor-version":[{"id":10803,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/10802\/revisions\/10803"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/3579"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10802"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10802"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10802"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}