{"id":3041,"date":"2023-12-25T02:30:00","date_gmt":"2023-12-25T07:30:00","guid":{"rendered":"https:\/\/www.both.org\/?p=3041"},"modified":"2023-12-29T09:29:58","modified_gmt":"2023-12-29T14:29:58","slug":"programming-bash-4-using-loops","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=3041","title":{"rendered":"Programming Bash #4: Using Loops"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

The ability to iterate over a section of code using various types of loops is one of the most important tools we have for performing repetitive tasks. We\u2019ve already seen the for<\/code> loop but there are others available to us in Bash, the while<\/code> and until<\/code> structures also provide iterative looping.<\/p>\n\n\n\n

Using for Loops<\/h2>\n\n\n\n

Every programming language I have ever used has some version of the for<\/code> command. The Bash implementation of this structure is, in my opinion, a bit more flexible than
most because it can handle non-numeric values while the standard C language for loop, for example, can only deal with numeric values.<\/p>\n\n\n\n

The basic structure of the Bash version of the for<\/code> command is simple \u2013 for Var in list1 ; do list2 ; done<\/code>. This translates to: for each value in list1, set the variable $Var to that value and then perform the program statements in list2 using that value; when all of the values in list1 have been used, we are done so exit the loop. The values in list1 can be a simple explicit string of values, or it can be the result of a command substitution as we have seen in the previous article of this series<\/a>.<\/p>\n\n\n\n

I use this construct frequently so we\u2019ll explore it in some detail. First, make ~\/testdir6 the PWD. Then try this simple example of a for loop.<\/p>\n\n\n\n

[dboth@testvm1 testdir6]$ for I in a b c d 1 2 3 4 ; do echo $I ; done<\/strong>\na\nb\nc\nd\n1\n2\n3\n4\n[dboth@testvm1 testdir6]$<\/code><\/pre>\n\n\n\n
Here\u2019s a bit more useful version along with a more meaningful variable name for the task of creating directories for multiple departments. We\u2019ll just start by ensuring that the basic loop works as expected but won\u2019t create the directories.<\/p>\n\n\n\n
Note that it is necessary to enclose the $Dept variable in quotes in the echo and mkdir statements or the two part department names such as \u201cInformation Technology\u201d will be treated as two separate departments. That highlights a best practice that I like to follow and that is that all file and directory names should be a single unbroken string with no spaces or other whitespace in them. Although most modern operating systems can deal with spaces in those names, it takes extra work for SysAdmins to ensure that those special cases are considered in scripts and CLI programs. But they almost certainly should be considered, even if they\u2019re annoying, because you never know what files you\u2019re actually going to have.<\/p>\n\n\n\n
[dboth@testvm1 testdir6]$ for Dept in \"Human Resources\" Sales Finance \"Information Technology\" Engineering Administration Research ; do echo \"Department $Dept\" ; done<\/strong>\nDepartment Human Resources\nDepartment Sales\nDepartment Finance\nDepartment Information Technology\nDepartment Engineering\nDepartment Administration\nDepartment Research\n[dboth@testvm1 testdir6]$<\/code><\/pre>\n\n\n\n
Now let\u2019s actually make some directories and show some progress information while doing so. Then list the contents of the testdir6 directory.<\/p>\n\n\n\n
[dboth@testvm1 testdir6]$ for Dept in \"Human Resources\" Sales Finance \"Information Technology\" Engineering Administration Research ; do echo \"Working on Department $Dept\" ; mkdir \"$Dept\" ; done ; ll<\/strong>\nWorking on Department Human Resources\nWorking on Department Sales\nWorking on Department Finance\nWorking on Department Information Technology\nWorking on Department Engineering\nWorking on Department Administration\nWorking on Department Research\ntotal 28\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Administration\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Engineering\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Finance\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08 'Human Resources'\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08 'Information Technology'\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Research\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:08  Sales\n[dboth@testvm1 testdir6]$ <\/code><\/pre>\n\n\n\n
Delete everything in testdir6 and recreate the directories using names with no spaces. We\u2019ll use dashes instead.<\/p>\n\n\n\n
[dboth@testvm1 testdir6]$ for Dept in Human-Resources Sales Finance Information-Technology Engineering Administration Research ; do echo \"Working on Department $Dept\" ; mkdir \"$Dept\" ; done ; ll<\/strong>\nWorking on Department Human-Resources\nWorking on Department Sales\nWorking on Department Finance\nWorking on Department Information-Technology\nWorking on Department Engineering\nWorking on Department Administration\nWorking on Department Research\ntotal 28\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Administration\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Engineering\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Finance\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Human-Resources\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Information-Technology\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Research\ndrwxr-xr-x 2 dboth dboth 4096 Nov 21 12:18 Sales\n[dboth@testvm1 testdir6]$<\/code><\/pre>\n\n\n\n
Suppose someone asks for a list of all RPMs on a particular Linux computer and a short description of each. This happened to me while I worked at the State of North Carolina. Since Open Source was not \u201capproved\u201d for use by state agencies at that time and I only used Linux on my desktop computer, the pointy haired bosses (PHBs) needed a list of each piece of software that was installed on my computer so that they could \u201capprove\u201d an exception. How would you approach that?<\/p>\n\n\n\n
Here is one way, starting with the knowledge that the rpm \u2013qi command provides a complete description of an RPM including the two items we want, the name and a brief summary. We will build up to the final result one step at a time. Note that this can be done as a non-privileged user since we are not going to add or remove any packages. First we list all RPMs.<\/p>\n\n\n\n
[dboth@testvm1 testdir6]$ rpm -qa<\/strong>\nfonts-filesystem-2.0.5-11.fc38.noarch\ngoogle-noto-fonts-common-20230201-1.fc38.noarch\nxkeyboard-config-2.38-1.fc38.noarch\ntzdata-2023c-1.fc38.noarch\ngoogle-noto-sans-vf-fonts-20230201-1.fc38.noarch\nliberation-fonts-common-2.1.5-4.fc38.noarch\nhyperv-daemons-license-0-0.40.20220731git.fc38.noarch\nhunspell-filesystem-1.7.2-3.fc38.x86_64\nabattis-cantarell-fonts-0.301-9.fc38.noarch\nweb-assets-filesystem-5-19.fc38.noarch\nmobile-broadband-provider-info-20221107-2.fc38.noarch\nfedora-logos-38.1.0-1.fc38.noarch\nadwaita-cursor-theme-44.0-1.fc38.noarch\njs-jquery-3.6.3-2.fc38.noarch\npython-systemd-doc-235-2.fc38.x86_64\nliberation-mono-fonts-2.1.5-4.fc38.noarch\ngoogle-noto-sans-mono-vf-fonts-20230201-1.fc38.noarch\ngoogle-noto-serif-vf-fonts-20230201-1.fc38.noarch\nadobe-source-code-pro-fonts-2.030.1.050-14.fc38.noarch\ndejavu-sans-mono-fonts-2.37-20.fc38.noarch\n<SNIP><\/code><\/pre>\n\n\n\n
This gives the list of RPMs installed on the host, so we can use this as the input list to a loop that will print all of the details of each RPM.<\/p>\n\n\n\n
[david@testvm1 testdir6]$ for RPM in `rpm -qa | sort | uniq` ; do rpm -qi $RPM ; done<\/strong><\/code><\/pre>\n\n\n\n
This code produces way more data than was desired because it displays all of the data about each RPM. We only need the name and summary lines.<\/p>\n\n\n\n
Note that our loop is complete. The next step is to extract only the information that was requested. To do that we add the grep -Ei<\/code> command.1<\/a><\/sup> The -E option is used to specify extended regular expressions will be used on the pattern. In this case it is used to select either ^Name or ^Summary. Thus any line with Name or Summary at the beginning of the line (the carat ^ specifies the beginning of the line) is displayed. The -i option tells grep to ignore case in the strings being checked. The vertical bar ( | ) is a logical or<\/strong>.<\/p>\n\n\n\n
[david@testvm1 testdir6]$ for RPM in `rpm -qa | sort | uniq` ; do rpm -qi $RPM ; done | grep -Ei \"^Name|^Summary\"<\/strong><\/code><\/pre>\n\n\n\n
Our final command sequence looks like this. The only change here is to add redirection so the data stream is stored in the RPM-summary.txt file.<\/p>\n\n\n\n
[david@testvm1 testdir6]$ for RPM in `rpm -qa | sort | uniq` ; do rpm -qi $RPM ; done | grep -Ei \"^Name|^Summary\" > RPM-summary.txt<\/strong><\/code><\/pre>\n\n\n\n
This command line program uses pipelines, redirection, and a for loop \u2013 all on a single line. It redirects the output of our little CLI program to a file that can be used in an email or as input for other purposes. This process of building up the program one step at a time allows you to see the results of each step and to ensure that it is working as you expect and provides the desired results.<\/p>\n\n\n\n
Note that the PHBs received a list of over 1,900 separate RPM packages. I seriously doubt that anyone actually read that list. But I gave them exactly what they asked for and I never heard another word from them about this.<\/p>\n\n\n\n
Other loops<\/h2>\n\n\n\n
There are two more types of loop structures available in Bash. The while and until structures, which are very similar to each other in both syntax and function. The basic syntax of these loop structures is simple.<\/p>\n\n\n\n
while [ expression ] ; do list ; done<\/code><\/p>\n\n\n\n
and<\/p>\n\n\n\n
until [ expression ] ; do list ; done<\/code><\/p>\n\n\n\n
The logic of these reads as follows. \u201cWhile the expression evaluates as true, execute the list of program statements. When the expression evaluates as false, exit from the loop.\u201d And \u201cUntil the expression evaluates as true, execute the list of program statements. When the expression evaluates as true, exit from the loop.\u201d<\/p>\n\n\n\n
while<\/h3>\n\n\n\n
The while loop is used to execute a series of program statements while (so long as) the logical expression evaluates to true. Your PWD should still be ~\/~.<\/p>\n\n\n\n
The simplest form of the while loop is one that runs forever. In the following form we use the true statement to always generate a \u201ctrue\u201d return code. We could use a simple \u201c1\u201d and that would work just the same but this illustrates use of the true statement. Using the head statement displays only the first 10 results. If we don\u2019t use the head statement the count will continue until it reaches the maximum integer size for the Bash implementation.2<\/a><\/sup><\/p>\n\n\n\n
[david@testvm1 testdir6]$ X=0 ; while [ true ] ; do echo $X ; X=$((X+1)) ; done | head<\/strong>\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n[dboth@testvm1 testdir6]$<\/code><\/pre>\n\n\n\n
This CLI program should make more sense now that we have studied its parts. First we set $X to zero just in case it had some leftover value from a previous program or CLI command. Then, since the logical expression [ true ] always evaluates to 1, which is true, the list of program instructions between do and done is executed forever \u2013 or until we press Ctrl-C or otherwise send a signal 2 to the program. Those instructions are an arithmetic expansion that prints the current value of $X and then increments it by one.<\/p>\n\n\n\n
One of the tenets of \u201cThe Linux Philosophy for SysAdmins\u201d is to strive for elegance and that one way to achieve elegance is simplicity. We can simplify this program by using the variable increment operator, ++. In this first instance the current value of the variable is printed and then the variable is incremented. This is indicated by placing the ++ operator after the variable.<\/p>\n\n\n\n
[david@testvm1 testdir6]$ X=0 ; while [ true ] ; do echo $((X++)) ; done | head<\/strong><\/code><\/pre>\n\n\n\n
Now delete \u201c| head\u201d from the and of the program and run it again. Press Ctrl-C when you tire of watching the count.<\/p>\n\n\n\n
In this next version, the variable is incremented before its value is printed. This is specified by placing the ++ operator before the variable. Can you see the difference?<\/p>\n\n\n\n
[david@testvm1 testdir6]$ X=0 ; while [ true ] ; do echo $((++X)) ; done | head<\/strong><\/code><\/pre>\n\n\n\n
We have reduced two statements into a single one that both prints the value of the variable and increments that value. There is also a decrement operator, \u2013.<\/p>\n\n\n\n
We need a method for stopping the loop at a specific number. To accomplish that we can change the true expression to an actual numeric evaluation expression. So let\u2019s have our program loop to 5 and stop. In the code below you can see that -le is the logical numeric operator for \u201cless than or equal to.\u201d This means that so long as $X is less than or equal to 5, the loop will continue. When $X increments to 6 the loop terminates.<\/p>\n\n\n\n
[david@testvm1 testdir6]$ X=0 ; while [ $X -le 5 ] ; do echo $((X++)) ; done<\/strong>\n0\n1\n2\n3\n4\n5<\/code><\/pre>\n\n\n\n
until<\/h3>\n\n\n\n
The until command is very much like the while command. The difference is that it will continue to loop until the logical expression evaluates to true. Let\u2019s look at the simplest form of this construct.<\/p>\n\n\n\n
[david@testvm1 testdir6]$ X=0 ; until false ; do echo $((X++)) ; done | head<\/strong><\/code><\/pre>\n\n\n\n
We can also use a logical comparison to count to a specific value.<\/p>\n\n\n\n
[dboth@testvm1 testdir6]$ X=0 ; until [ $X -eq 5 ] ; do echo $((X++)) ; done<\/strong>\n0\n1\n2\n3\n4\n[dboth@testvm1 testdir6]$ X=0 ; until [ $X -eq 5 ] ; do echo $((++X)) ; done<\/strong>\n1\n2\n3\n4\n5\n[dboth@testvm1 testdir6]$<\/code><\/pre>\n\n\n\n
Be sure to note the difference between these two forms of the until statement.<\/p>\n\n\n\n
Summary<\/h2>\n\n\n\n
We have explored the use of many powerful tools that we can use to build command line programs and Bash shell scripts. Despite the interesting things we have done in this article, Bash command line programs and shell scripts can do so much more. We have barely scratched the surface. The rest is up to you.<\/p>\n\n\n\n
I have discovered over time that the best way to learn Bash programming is to actually do it. Find a simple project that requires multiple Bash commands and make a CLI program out of them. SysAdmins do many tasks that lend themselves to CLI programming this way so I am sure that you will easily find tasks to automate.<\/p>\n\n\n\n
Many years ago, despite being familiar with other shell languages and Perl, I made the decision to use Bash for all of my SysAdmin automation tasks. I have discovered that \u2013 perhaps with a bit of searching \u2013 I have been able to accomplish everything I need.<\/p>\n\n\n\n
\n\n\n\n
Series Articles<\/h2>\n\n\n\n
This list contains links to all eight articles in this series about Bash.<\/p>\n\n\n\n
    \n
  1. Programming Bash #1 \u2013 Introducing a New Series<\/a><\/li>\n\n\n\n
  2. Programming Bash #2: Getting Started<\/a><\/li>\n\n\n\n
  3. Programming Bash #3: Logical Operators<\/a><\/li>\n\n\n\n
  4. Programming Bash #4: Using Loops<\/a><\/li>\n\n\n\n
  5. Programming Bash #5: Automation with Scripts<\/a><\/li>\n\n\n\n
  6. Programming Bash #6: Creating a template<\/a><\/li>\n\n\n\n
  7. Programming Bash #7: Bash Program Needs Help<\/a><\/li>\n\n\n\n
  8. Programming Bash #8: Initialization and sanity testing<\/a><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
    The ability to iterate over a section of code using various types of loops is one of the<\/p>\n","protected":false},"author":2,"featured_media":2814,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[149,5,150],"tags":[151,152],"class_list":["post-3041","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bash","category-linux","category-programming","tag-bash","tag-programming"],"modified_by":"David Both","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/3041","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3041"}],"version-history":[{"count":3,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/3041\/revisions"}],"predecessor-version":[{"id":3251,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/3041\/revisions\/3251"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/2814"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3041"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3041"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3041"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}