Talk Tech to Me: Become a Linux Foo Master with Piping and Redirection

Our Linux guru, Jason Eckert, is back to “Talk Tech to Me” with a few crash courses on some of the hottest Linux topics. Here’s his second in the three-part series of articles full of commands and step-by-step instructions to help you master Linux.

Read more from Jason in:
All About Linux and Linux+
3 Ways CompTIA Linux+ Helps New Grads Land Jobs
Talk Tech to Me: Using Linux BASH on Windows 10
Talk Tech to Me: Configuring ZFS on Linux

UNIX systems have always been powerful by design – programs and commands on these systems can be combined using piping and redirection in different ways to perform unusual or advanced tasks. This practice is often called UNIX foo (which is a playful short term for UNIX kung fu). And since the Linux operating system evolved from UNIX, it boasts the same piping and redirection features. As a result, you can call it Linux foo on a Linux system.

How Does Piping and Redirection Work?

Each command on a UNIX or Linux system has the ability to display output to the screen after being run; this output is called standard output (stdout). Additionally, many UNIX and Linux commands can take the stdout from another command as input to be processed; this is called standard input (stdin). To send the stdout from one command to another, you simply need to use a pipe ( | ) symbol between them. Consequently, we call this feature piping.

Piping is often useful if you want to narrow down a large amount of stdout that has been generated from a command.

For example, if you want to view all the processes on the system using the ps -ef command, several pages of output will be displayed.

To view this output page-by-page, you could send this output directly to the less command (a command that is used to display text files page by page) by running the ps -ef | less command.

Similarly, to view only the httpd processes running on your system, you could use the ps -ef | grep httpd command. This will send the list of all processes to the grep httpd command, which will only display lines that have httpd in them.

If you want to save the stdout of a command to a file, you could use the output redirection symbol (>).

For example, to save a list of httpd processes on the system to the /root/processes file, you could use the ps -ef | grep httpd > /root/processes command.

Alternatively, to redirect a file to become the stdin for a command, you can instead use the < redirection symbol. For example, to view only the lines in the /root/processes file that have the word apache in them, you could use the grep apache < /root/processes command.

Performing Linux Foo Using Piping and Redirection

The above examples are fairly basic. If you combine piping and redirection with more advanced commands in unusual ways, then you’re performing Linux foo!

For example, let’s examine how to perform Linux foo using the ssh command, which is normally used to administer a remote Linux or UNIX computer. If you run the ssh [email protected] command, you will connect to the computer with the IP address 192.168.1.140 as the root user. After supplying the password for the root user on the remote system, you’ll obtain a command prompt in the remote root user’s home directory and will be able to run commands on the remote system as if you were logged in locally to that system.

Now, let’s perform some Linux foo! We’ll use redirection to do something uncommon with the ssh command: copy files between your computer and a remote computer.

To copy the lala.bin file from the root user’s directory on a remote computer (IP=192.168.1.140) to the current directory on your local system (keeping the same filename), you could use the following command and supply the remote computer’s root user password when prompted:

ssh [email protected] cat lala.bin > lala.bin

Similarly, you could copy the lala.bin file from your current directory to the root user’s home directory on a remote computer using the following command (supplying the root user’s password when prompted):

ssh [email protected] cat <lala.bin ">" lala.bin

But Linux foo doesn’t stop there – you can get even more creative!

Say, for example, that you have a Linux system on the network called OOBLA (IP address 192.168.1.100). This second hard disk on this system (/dev/sdb) contains key data (e.g., user files, databases) that must be backed up each day. As a result, you scheduled the following shell script each day to back up the /dev/sdb filesystem using the dd command and copy it to another Linux server called BACKUPSERVER (IP address 192.168.1.200):

#!/bin/bash

if dd if=/dev/sdb of=/tmp/oobla-backup.img

then

mount -t nfs 192.168.1.200:/backups /remoteNFSshare

cp -F /tmp/oobla-backup.img /remoteNFSshare

rm /tmp/oobla-backup.img

fi

So, what happens if the /dev/sdb filesystem on OOBLA gets corrupted? Well, you could copy the oobla-backup.img from the BACKUPSERVER back to your /tmp directory and then use the dd command to reimage it back to /dev/sdb.

However, Linux foo using piping can provide an easier solution. You could simply log into BACKUPSERVER and run the following command to remotely overwrite the /dev/sdb filesystem on OOBLA from the backup file across the network (supplying OOBLA’s root user password when prompted):

dd if=oobla-backup.img | ssh 192.168.1.100 dd of=/dev/sdb