Redirectors — More than you think

If you use the command line at all, you’re probably familiar with the basic concept of redirection using Standard IO (STDIO). One common usage is redirecting STDOUT that would normally go to the display, to a file instead.

ls -l > files.txt

STDIO

There are three STDIO data streams, each of which is automatically opened as a file at the startup of a program — well those programs that use STDIO. Each STDIO data stream is associated with a file handle that is just a set of metadata that describes the attributes of the file. File handles 0, 1, and 2 are explicitly defined by convention and long practice as STDIN, STDOUT, and STDERR, respectively.

STDIN, File handle 0, is standard input that is usually input from the keyboard. STDIN can be redirected from any file including device files instead of the keyboard. It is less common to redirect STDIN than STDOUT or STDERR, but it can be done just as easily.

STDOUT, File handle 1, is standard output that sends the data stream to the display by default. It is common to redirect STDOUT to a file or to pipe it to another program for further processing.

STDERR, File handle 2, is the data stream for STDERR. It’s also usually sent to the display.

Each of the STDIO file handles has a corresponding device in the /dev directory.

• /dev/stdin
• /dev/stdout
• /dev/stderr

# ll | grep std
lrwxrwxrwx.  1 root root            15 May 18 17:11 stderr -> /proc/self/fd/2
lrwxrwxrwx.  1 root root            15 May 18 17:11 stdin -> /proc/self/fd/0
lrwxrwxrwx.  1 root root            15 May 18 17:11 stdout -> /proc/self/fd/1

If STDOUT is redirected to a file, STDERR continues to be displayed on the screen. This ensures that when the data stream itself is not displayed on the terminal, that STDERR is, thus ensuring that the user will see any errors resulting from execution of the program. STDERR can also be redirected to the same or passed on to the next filter program in a pipeline.

The Bash man page has a long section on redirection that covers the basics like redirecting STDOUT and STDERR to a file, STDIN from a file, as well as some more esoteric combinations of those. But there’s much more to redirection than is usually covered in any class or book I’ve ever read.

The redirectors I use most frequently

Let’s explore a few of the redirectors. These are the ones I use most frequently.

The pipe: |

The redirector I use most is the pipe ( | ). This allows me to redirect the STDOUT of one command to the STDIN of another command. Pipes are the syntactical glue, the operator, that is used connect command-line utilities together. Pipes allow the Standard Output from one command to be “piped,” i.e., streamed from Standard Output of one command to the Standard Input of the next command.

Pipes are critical to our ability to do the amazing things on the command line, so much so that I think it is important to recognize that they were invented by Douglas McIlroy during the early days of Unix. The Princeton University website used to contain a fragment of an interview with McIlroy in which he discusses the creation of the pipe and the beginnings of the Unix philosophy. I’ve left the link here in case it ever shows up again.

Notice the use of pipes in the simple command-line program shown next, which lists each logged-in user a single time, no matter how many logins they have active.

$ w | tail -n +3 | awk '{print $1}' | sort | uniq
dboth
root
[student@studentvm1 ~]$

The result of this command produces two lines of data that show that the user’s root and student are both logged in. It does not show how many times each user is logged in. Your results will almost certainly differ from mine.

The |& operator can be used to pipe the STDERR along with STDOUT to STDIN of the next command. This is not always desirable, but it does offer flexibility in the ability to record the STDERR data stream for the purposes of problem determination.

A string of programs connected with pipes is called a pipeline, and the programs that use STDIO are referred to as filters.

STDOUT to a file: > and >>

The “greater than” ( > ) character, aka “gt”, is the syntactical symbol for redirection of STDOUT to a file. Redirecting the STDOUT of a command can be used to create a file containing the results from that command.

$ df -h > diskusage.txt

There is no output to the terminal from this command unless there is an error. This is because the STDOUT data stream is redirected to the file and STDERR is still directed to the STDOUT device, which is the display.

When using the > symbol to redirect the data stream, the specified file is created if it does not already exist. If it does exist, the contents are overwritten by the data stream from the command. You can use double greater-than symbols, >>, to append the new data stream to any existing content in the file.

$ df -h >> diskusage.txt

STDIO and STDERR to a file: &>

Standard Error (STDERR) is a separate data stream that’s normally sent to the display so that errors can be viewed by the SysAdmin, even as the normal STDOUT data stream is sent through more filter programs or to a file. I use this when I need to log the STDERR data stream to a file along with STDOUT so that I can review it later in context.

[command] &> log.file

This simple form is the functional equivalent of the more complex [command] 2>&1. I’m not sure why, but this complex form is what I was first taught.

Less frequently used

I use these redirectors but less frequently than the ones in the preceding section.

STDIN from a file: <

The < (less than) symbol redirects data to the STDIN of the program. You might want to use this method to input data from a file to STDIN of a command that does not take a filename as an argument but that does use STDIN. Although input sources can be redirected to STDIN, such as a file that is used as input to grep, it is generally not necessary as grep also takes a filename as an argument to specify the input source. Most other commands also take a filename as an argument for their input source.

Here’s an example of using redirection to STDIN is with the od command. The -N 50 option limits the output to the specified number of lines. You could use Ctrl-C to terminate the output data stream if you don’t use the -N option to limit it. This command illustrates the use of redirection as input to STDIN.

$ od -c -N 200 < /dev/urandom
0000000  \n 335 366 315   f   V 267 343 236 375   H   @ 250 252 364  \r
0000020 375 314   $       F 320 021   j   R   =   9 215 266   - 231 237
0000040   ,   i   G   ' 325 330 251 250 345 341 212 322 360     232   Q
0000060 374 347   4   f   F   u   R   o 237 240 004 310   F 354 265   2
0000100 365 030 267 305   Z 231 307 264 322  \t 211 347 241   \   X   -
0000120   ] 257 310   ? 315 330   V   ,   l 020 356 265 034 204 177 317
0000140 034   4   :  \n 373   D 350 215 001   7   Y   { 325   D   6   9
0000160 220  \t 202   e 205   {   8   m   L   M   #   > 362 334 241   7
0000200 326 033   <   X 347 036   +   + 363   V   - 025   }  \0   @   9
0000220   F 006 265   a 002 213   9 005 220 304 373 200 314 230 363   F
0000240 274 364 354   a   p 247 203 337 237 271 366 037 325 231   ,   m
0000260  \n 357 301   4 305 302   * 306 261 347 207 016   x   T 005   &
0000300   f 020   V 257 202  \f   0 266
0000310
tuser1@testvm1:~$

Redirection can be the source or the termination of a pipeline. Redirection is seldom needed as input, but is usually used as termination of a pipeline by sending STDOUT to a file.

STDOUT or STDERR stream to a file: X> and X>>

I use this when I want to redirect STDOUT or STDERR to a file while leaving the other stream to STDOUT. Remember that STDOUT is file handle 1 and STDERR is file handle 2. The first line is my typical usage. the first instance redirects STDERR to the error.file.

$ [command] 2> error.file
$ [command] 1> log.file 

The second instance redirects STDIO to the log.file. The command [command] >> log.file appends the data stream to an existing file or creates the file if it doesn’t already exist.

Here documents

I never thought of “here documents” as a form of redirection, yet is very much is. A here document is a way to embed a short text document in a shell script. I use this in a couple scripts I need occasionally and it’s quite powerful. One use for this is simply create a text file on the storage device or to STDOUT.

A more interesting use is to create a series of inputs to a command. In this example I use a script to perform a series of tasks, one of which is to create a partition on a storage device. Experimenting with other options showed them to be too complex, so I created the following bit of code in my script.

# Create new Linux partition
echo "Creating new partition on $Device"
fdisk /dev/$Device << EOF
g
p
n
1



p
w
EOF

sync ; sync

This code issues a string of input to the fdisk command that would normally be performed interactively. The fdisk command does not have any capability to be used in scripts, so I used this here document as a circumvention for that. There are commands like cfdisk and sfdisk that can be used in scripts, but I found that they’re difficult to work with and the arguments must be precise or the partition won’t be created.

The text on each line — a single character in this case plus the end of line (EOL) character — is one command to fdisk. For example, ‘g’ tells fdisk to create a GUID partition table on the first cylinder of the storage device. The ‘p’ prints the current partition list, which should be empty, to STDOUT, ‘n’ creates a new partition, and ‘1’ specifies the new partition to be number 1 on the device. The next few lines contain only the EOL and are used to ensure to select the default beginning and end cylinders, the first and last ones available on the device, and then that fdisk is back to the main menu. Once again, ‘p’ prints the partition list, ‘w’ writes the data to the storage device.

I use EOF (End Of File) as the delimiter for the here document, but any string of characters can be used. I think EOF makes it easy to spot the beginning and end of the here document itself.

Other redirectors

There are a number of other I/O redirectors, but they’re all rather esoteric and, in my experience at least, seldom used. I suggest exploring the REDIRECTORS section of the BASH man page. You might find something that can help solve a problem.