Efficient Linux at the Command Line

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Daniel J. Barrett

Efficient Linux at the Command Line

by Daniel J. Barrett

Copyright © 2022 Daniel J. Barrett. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

- Acquisitions Editor: John Devins

- Development Editor: Virginia Wilson

- Production Editor: Caitlin Ghegan

- Copyeditor: TK

- Proofreader: TK

- Indexer: TK

- Interior Designer: David Futato

- Cover Designer: Karen Montgomery

- Illustrator: Kate Dullea

- October 2022: First Edition

Revision History for the Early Release

- 2021-09-20: First release

See http://oreilly.com/catalog/errata.csp?isbn=9781098113407 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Efficient Linux at the Command Line, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-11340-7

[LSI]

Preface

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the Preface of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at vwilson@oreilly.com.

This book will take your Linux command-line skills to the next level, so you can work faster, smarter, and more efficiently.

In the 30-plus years that I’ve been teaching Linux, I’ve met many users who learned its basic commands by trial and error or by skimming an intro book. These folks had more important work to do, like developing software or maintaining a network, and they learned just enough command-line skills to feel productive. As the years passed, they picked up some cool command-line tricks but never developed a solid understanding of how and why things worked. Their Linux skills, as a result, were relatively slow and inefficient and they had no idea. For example, I’ve watched experienced programmers lose hours of productivity by exiting their text editor in between compiles, when savvy use of job control (Chapter TK) would make them more productive almost instantly.

This book aims to change things by teaching core skills and the core concepts behind them. Think of it as “a second book on Linux” that takes you beyond the basics. You’ll learn to create and run complex commands that solve practical problems, efficiently process and retrieve information, automate manual tasks, tailor your Linux environment, and organize your files for rapid access. Most of all, you’ll learn general best practices so no matter which Linux tools you use, you can become more effective in daily use and more competitive on the job market.

In short, this is the book I wish I had when I learned Linux.

The Command-Line Skills You Need

Every Linux command is an act of creation. You’re presented with nothing but a prompt, which is an offer to run any command you may know:

$

What happens next is up to you and your creative mind. There are no friendly icons or buttons to click here. No menus full of convenient operations to choose from. It’s just you and a prompt.

Each time you type a command and press Enter, you’re solving a business problem, whether it’s “Show me my files” or “Track my FedEx package” or “Convert these 20,000 PNG files to JPEG format.” Your solutions may be simple, like this directory listing command:

$ ls

or complex, like this brash one-liner:1

$ paste <(echo {1..10}.jpg | sed 's/ /\n/g') <(echo {0..9}.jpg | sed 's/ /\n/g') \

| sed 's/^/mv /' \

| bash

If you’re staring at the preceding command and thinking, “What the heck is that?” or “I would never need such a complicated command,” then this book is for you.

What You’ll Learn

Have you ever met a Linux wizard who could produce and run long, intricate commands, like the earlier paste command, as fast as they could type? These folks have a kind of wisdom about commands and how they combine. This book will share that wisdom and make you faster and more effective at three essential skills:

1. Recalling or inventing commands to solve the problem at hand

2. Running those commands efficiently

3. Navigating the Linux filesystem with ease, so you can launch commands from anywhere

By the end, you’ll understand what happens behind the scenes when you run a command, so you can better predict the results (and not develop superstitions). You’ll see a dozen different methods for launching commands and learn when to use each one for best advantage, including ordinary execution, lists, pipelines, subshells, command substitution, process substitution, bash -c, xargs, and more. You’ll also tailor your Linux account for speed and productivity.

Along with general best practices, you’ll learn individual tips and tricks to make you more productive, such as:

- Building complex commands out of simpler ones, step by step, to solve real-world problems, like managing passwords or generating 10,000 test files

- Automating tasks with single-use scripts that you throw away afterward, and understanding why this is a good thing

- Saving time by organizing your home directory intelligently so you don’t have to hunt for files

- Treating any text file like a database that can be queried and transformed by Linux commands

- Controlling point-and-click features of Linux from the command line, such as copying and pasting with the clipboard, and retrieving and processing web data, without lifting your hands from the keyboard

Audience and Prerequisites

This book assumes you have some Linux experience; it’s not an introduction. It’s designed for users with beginning or intermediate skills at the command line, such as students, system administrators, software developers, site reliability engineers, test engineers, and general Linux enthusiasts. Advanced Linux users may find some useful material as well, especially if they learned by trial and error over the years and want to strengthen their conceptual understanding.

To benefit most from this book, you should already be comfortable with the following topics. If not, see Appendix A (forthcoming) for a quick refresher.

- Logging into a Linux system

- Creating and editing text files with a text editor such as vim, emacs, nano, or pico

- Basic file-handling commands like cp (copy), mv (move or rename), rm (remove or delete), and chmod (change file permissions)

- Basic file-viewing commands like cat (view an entire file) and less (view one page at a time)

- Basic directory commands like cd (change directory), ls (list files in a directory), mkdir (create directory), rmdir (remove directory), and pwd (display your current directory name)

- The basics of shell scripts: storing Linux commands in a file, making the file executable (with chmod 755 or chmod +x), and running the file

- Viewing Linux’s built-in documentation, known as manpages, with the man command (example: man cat displays documentation on the cat command)

- Becoming the superuser with the sudo command for full access to your Linux system (example: sudo nano /etc/hosts edits the system file /etc/hosts which is protected from ordinary users)

If you also know common command-line features like pattern-matching for filenames (with the * and ? symbols), input/output redirection (< and >), and pipes (|), you’re ahead of the game.

This book is not a comprehensive reference for the command line — there are hundreds of commands and features that I don’t mention. This book is about expertise. It teaches a carefully-selected set of command-line wisdom in a practical order to build your skills. For a more reference-style guide, try my previous book, Linux Pocket Guide.

Conventions Used in This Book

I assume your Linux shell is bash, which is the default shell for most Linux distributions. Whenever I write “the shell,” I always mean bash. Most of the ideas I present apply to other shells too.

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

TIP

This element signifies a tip or suggestion.

NOTE

This element signifies a general note.

WARNING

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/oreillymedia/mastering_linux_command_line.

If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Book Title by Some Author (O’Reilly). Copyright 2012 Some Copyright Holder, 978-0-596-xxxx-x.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

O’Reilly Online Learning

NOTE

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

- O’Reilly Media, Inc.

- 1005 Gravenstein Highway North

- Sebastopol, CA 95472

- 800-998-9938 (in the United States or Canada)

- 707-829-0515 (international or local)

- 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://www.oreilly.com/catalog/9781098113391.

Email bookquestions@oreilly.com to comment or ask technical questions about this book.

For news and information about our books and courses, visit http://oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

TK

1   We’ll uncover this mystery command’s purpose in Chapter 8. The final backslash on each line is a line-continuation character, which permits long commands to be split across several lines.

Part I. Core Concepts

The first four chapters start you off with immediately useful concepts and techniques to increase your efficiency in several domains. You’ll learn to combine commands with pipes, understand the responsibilities of the Linux shell, rapidly recall and edit commands from the past, and navigate the Linux filesystem efficiently.

Chapter 1. Combining Commands

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st Chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at vwilson@oreilly.com.

When you work in Windows, MacOS, and most other operating systems, you probably spend your time running applications like web browsers, word processors, spreadsheets, and games. A typical application is packed with features: everything that the designers thought their users would need. So, most applications are self-sufficient. They don’t rely on other apps. You might copy and paste between applications from time to time, but for the most part, they’re separate.

The Linux command line is different. Instead of big applications with tons of features, you work with thousands of small commands with very few features. The command cat, for example, prints files on the screen and that’s about it. ls lists the files in a directory, mv renames files, and so on. Each command has a simple, fairly well-defined purpose.

What if you need to do something more complicated? Don’t worry. Linux makes it easy to combine commands so their individual features work together to accomplish your goal. This way of working yields a very different mindset about computing. Instead of asking “Which app should I launch?” to achieve some result, the question becomes “Which commands should I combine?”

In this chapter, you’ll learn how to arrange and run commands in different combinations to do what you need. To keep things simple, I’ll introduce just six Linux commands and their most basic uses, so you can focus on the more complex and interesting part — combining them — without a huge learning curve. It’s a bit like learning to cook with six ingredients, or learning carpentry with just a hammer and a saw. We’ll add more commands to your Linux toolbox in Chapter 5.

You’ll combine commands using pipes, a Linux feature that connects the output of one command to the input of another. As I introduce each command (wc, head, cut, grep, sort, and uniq), I’ll immediately demonstrate its use with pipes. Some examples will be practical for daily Linux use, while others are just toy examples to demonstrate an important feature.

Input, Output, and Pipes

Most Linux commands read input from the keyboard, write output to the screen, or both. Linux has fancy names for this reading and writing:

stdin (pronounced “standard input” or “standard in”)

The stream of input that Linux reads from your keyboard. When you type any command at a prompt, you’re supplying data on stdin.

stdout (pronounced “standard output” or “standard out”)

The stream of output that Linux writes to your display. When you run the ls command to print filenames, the results appear on stdout.

Now comes the cool part. You can connect the stdout of one command to the stdin of another, so the first command feeds the second. Let’s begin with the familiar ls -l command to list a large directory, such as /bin:

$ ls -l /bin

total 12104

-rwxr-xr-x 1 root root 1113504 Jun  6  2019 bash

-rwxr-xr-x 1 root root  170456 Sep 21  2019 bsd-csh

-rwxr-xr-x 1 root root   34888 Jul  4  2019 bunzip2

-rwxr-xr-x 1 root root 2062296 Sep 18  2020 busybox

-rwxr-xr-x 1 root root   34888 Jul  4  2019 bzcat

...

-rwxr-xr-x 1 root root    5047 Apr 27  2017 znew

This directory contains far more files than your display has lines, so the output quickly scrolls off-screen. It’s a shame that ls can’t display the filenames one screenful at a time, pausing until you press a key to continue. But wait: another Linux command has that feature. The less command displays a file one screenful at a time.

$ less myfile

You can connect these two commands and achieve your goal, because ls writes to stdout and less can read from stdin. Use a pipe to send the output of ls to the input of less:

$ ls -l /bin | less

This combined command displays the directory’s contents one screenful at a time. The vertical bar (|) between the commands is the Linux pipe symbol. It connects the first command’s stdout to the next command’s stdin. Any command line containing pipes is called a pipeline.

Commands generally are not aware that they’re part of a pipeline. ls believes it’s writing to the display, when in fact its output has been redirected to less. And less believes it’s reading from the keyboard when it’s actually reading the output of ls.

WHAT’S A COMMAND?

The word command has three different meanings in Linux, shown in Figure 1-1.

Figure 1-1. Programs, simple commands, and combined commands are all referred to as “commands”

A program name

An executable program named by a single word, such as ls, or a similar feature built into the shell, such as cd (called a shell builtin).1

A simple command

A program name (or shell builtin) optionally followed by arguments, such as ls -l /bin.

A combined command

A combination of simple commands treated as a unit, such as the pipeline ls -l /bin | less.

In this book, I’ll use the word “command” in all of these ways. Usually the surrounding context will make clear which one I mean, but if not, I’ll use one of the more specific terms.

Six Commands To Get You Started

Pipes are an essential part of Linux expertise. Let’s dive into building your piping skills with a small set of commands, so no matter what Linux commands you encounter later, you’re ready to combine them.

The six commands — wc, head, cut, grep, sort, and uniq — have numerous options and modes of operation that I’ll largely skip for now to focus on pipes. To learn more about any command, run the man command to display full documentation. For example:

$ man wc

To demonstrate our six commands in action, I’ll use a file named animals.txt that lists some O’Reilly book information.

$ cat animals.txt

python Programming Python 2010 Lutz, Mark

snail SSH, The Secure Shell 2005 Barrett, Daniel

alpaca Intermediate Perl 2012 Schwartz, Randal

robin MySQL High Availability 2014 Bell, Charles

horse Linux in a Nutshell 2009 Siever, Ellen

donkey Cisco IOS in a Nutshell 2005 Boney, James

oryx Writing Word Macros 1999 Roman, Steven

Each line contains four facts about an O’Reilly book, separated by a single tab character: the animal on the front cover, the book title, the year of publication, and the name of the first author.

Command #1: wc

The wc command prints the number of lines, words, and characters in a file.

$ wc animals.txt

7  49 324 animals.txt

wc reports that the file animals.txt has 7 lines, 49 words, and 324 characters. If you count the characters by eye, including spaces and tabs, you’ll find only 317 characters, but the count also includes the invisible newline character that ends each line.

The options -l, -w, and -c print only the number of lines, words, and characters, respectively:

$ wc -l animals.txt

7 animals.txt

$ wc -w animals.txt

49 animals.txt

$ wc -c animals.txt

324 animals.txt

Counting is such a useful, general-purpose task that the authors of wc designed the command to work with pipes. It reads from stdin if you omit the filename, and it writes to stdout. Let’s use ls to list the contents of the current directory and pipe them to wc to count lines. This pipeline answers the question, “How many files are in my current directory?”

$ ls -1

animals.txt

myfile

myfile2

test.py

$ ls -1 | wc -l

4

The option -1, which tells ls to print its results in a single column, is not strictly necessary here. To learn why I used it, see the sidebar “ls Changes Its Behavior When Redirected”.

wc is the first command you’ve seen in this chapter, so you’re a bit limited in what you can do with pipes. Just for fun, pipe the output of wc to itself, demonstrating that the same command can appear more than once in a pipeline. This combined command reports that the number of words in the output of wc is four: the number of lines, number of words, number of characters, and the filename:

$ wc animals.txt

7  49 324 animals.txt

$ wc animals.txt | wc -w

4

Why stop there? Add a third wc to the pipeline and count lines, words, and characters in the output “4”:

$ wc animals.txt | wc -w | wc

1       1       2

The output indicates one line (containing the number 4), one word (the number 4 itself), and two characters. Why two? Because the line “4” ends with an invisible newline character.

That’s enough silly pipelines with wc. As you gain more commands, the pipelines will become more practical.

LS CHANGES ITS BEHAVIOR WHEN REDIRECTED

Unlike virtually every other Linux command, ls is aware of whether stdout is the screen or whether it’s been redirected (to a pipe or otherwise). The reason is user-friendliness. When stdout is the screen, ls arranges its output in multiple columns for convenient reading:

$ ls /bin

bash        dir         kmod      networkctl       red     tar

bsd-csh     dmesg       less      nisdomainname    rm      tempfile

...

When stdout is redirected, however, ls produces a single column. I’ll demonstrate this by piping the output of ls to a command that simply reproduces its input, such as cat:

$ ls /bin | cat

bash

bsd-csh

bunzip2

busybox

...

This behavior can lead to strange-looking results, as in the following example:

$ ls

animals.txt   myfile   myfile2    test.py          A single line

$ ls | wc -l                                     Count the lines

4

The first command prints all filenames on one line, but the second command reports that ls produced four lines. If you aren’t aware of the quirky behavior of ls, you might find this discrepancy confusing.

ls has options to override its default behavior. You can force ls to print a single column on the screen with the -1 option, or force multiple columns in a pipeline with the -C option.

Command #2: head

The head command prints the first lines of a file. Print the first 3 lines of animals.txt using the option -n:

$ head -n3 animals.txt

python Programming Python 2010 Lutz, Mark

snail SSH, The Secure Shell 2005 Barrett, Daniel

alpaca Intermediate Perl 2012 Schwartz, Randal

If you request more lines than the file contains, head prints the whole file (like cat does). If you omit the -n option, head assumes 10 lines (-n10).

By itself, head is handy for peeking at the top of a file when you don’t care about the rest of the contents. In addition, head writes to stdout, making it useful in pipelines. Count the number of words in the first three lines of animals.txt:

$ head -n3 animals.txt | wc -w

20

head can also read from stdin for more pipeline fun. A common use is to reduce the output from another command when you don’t care to see all of it, like a long directory listing. Print the first five files in a listing of the /bin directory.

$ ls /bin | head -n5

bash

bsd-csh

bunzip2

busybox

bzcat

Command #3: cut

The cut command prints one or more columns from a file. For example, print all book titles from animals.txt, which appear in the second column:

$ cut -f2 animals.txt

Programming Python

SSH, The Secure Shell

Intermediate Perl

MySQL High Availability

Linux in a Nutshell

Cisco IOS in a Nutshell

Writing Word Macros

cut provides two ways to define what a “column” is. The first way is to cut by field (-f), which means that the input consists of strings (fields) separated by a single tab character. Conveniently, that is exactly the format of the file animals.txt. The preceding cut command prints the second field of each line, thanks to the option -f2.

To shorten the output, pipe it to head to print only the first three lines:

$ cut -f2 animals.txt | head -n3

Programming Python

SSH, The Secure Shell

Intermediate Perl

You can also cut multiple fields, either by separating their field numbers with commas:

$ cut -f1,3 animals.txt | head -n3

python 2010

snail 2005

alpaca 2012

or by numeric range:

$ cut -f2-4 animals.txt | head -n3

Programming Python 2010 Lutz, Mark

SSH, The Secure Shell 2005 Barrett, Daniel

Intermediate Perl 2012 Schwartz, Randal

The second way to define a “column” for cut is by character position, using the -c option. Print the first three characters from each line of the file, which you can specify either with commas (1,2,3) or as a range (1-3):

$ cut -c1-3 animals.txt

pyt

sna

alp

rob

hor

don

ory

Now that you’ve seen the basic functionality, try something more practical with cut and pipes. Imagine that the animals.txt file is thousands of lines long, and you need to extract just the authors’ last names. First, isolate the fourth field, author name:

$ cut -f4 animals.txt

Lutz, Mark

Barrett, Daniel

Schwartz, Randal

...

Then pipe the results to cut again, using the option -d to change the delimiter character to a comma instead of tab, to isolate the authors’ last names:

$ cut -f4 animals.txt | cut -d, -f1

Lutz

Barrett

Schwartz

...

SAVE TIME WITH COMMAND HISTORY AND EDITING

Are you retyping a lot of commands? Press the up arrow key instead, repeatedly, to scroll through commands you’ve run before. (This shell feature is called command history.) When you reach the desired command, press Enter to run it immediately, or edit it first using the left and right arrow keys to position the cursor. (This feature is command-line editing.)

I’ll discuss much more powerful features for command history and editing in Chapter ???.

Command #4: grep

grep is an extremely powerful command, but for now I’ll hide most of its capabilities and say it prints lines that match a given string. (More detail will come in Chapter 5.) For example, the following command displays lines from animals.txt that contain the string Nutshell:

$ grep Nutshell animals.txt

horse Linux in a Nutshell 2009 Siever, Ellen

donkey Cisco IOS in a Nutshell 2005 Boney, James

You can also print lines that don’t match a given string, with the -v option. Notice the lines containing Nutshell are absent:

$ grep -v Nutshell animals.txt

python Programming Python 2010 Lutz, Mark

snail SSH, The Secure Shell 2005 Barrett, Daniel

alpaca Intermediate Perl 2012 Schwartz, Randal

robin MySQL High Availability 2014 Bell, Charles

oryx Writing Word Macros 1999 Roman, Steven

In general, grep is extremely useful for finding text in a collection of files. This command looks for lines that contain the string Perl in files with names ending in .txt:

$ grep Perl *.txt

animals.txt:alpaca      Intermediate Perl       2012    Schwartz, Randal

essay.txt:really love the Perl programming language, which is

essay.txt:languages such as Perl, Python, PHP, and Ruby

In this case, grep found three matching lines, one in animals.txt and two in essay.txt.

grep reads stdin and writes stdout, making it great for pipelines. Suppose you want to know how many subdirectories are in the large directory /usr/lib. There is no single Linux command to provide that answer, so we construct a pipeline. Begin with the ls -l command:

$ ls -l /usr/lib

drwxrwxr-x  12 root root    4096 Mar  1  2020 4kstogram

drwxr-xr-x   3 root root    4096 Nov 30  2020 GraphicsMagick-1.4

drwxr-xr-x   4 root root    4096 Mar 19  2020 NetworkManager

-rw-r--r--   1 root root   35568 Dec  1  2017 attica_kde.so

-rwxr-xr-x   1 root root     684 May  5  2018 cnf-update-db

...

Notice that ls -l marks directories with a d at the beginning of the line. Use cut to isolate the first column, which may or may not be a d:

$ ls -l | cut -c1

d

d

d

-

-

...

Then use grep to keep only the lines containing d:

$ ls -l | cut -c1 | grep d

d

d

d

...

Finally, count lines with wc, and you have your answer: /usr/lib contains 144 subdirectories.

$ ls -l | cut -c1 | grep d | wc -l

144

Command #5: sort

The sort command reorders the lines of a file into ascending order (the default):

$ sort animals.txt

alpaca Intermediate Perl 2012 Schwartz, Randal

donkey Cisco IOS in a Nutshell 2005 Boney, James

horse Linux in a Nutshell 2009 Siever, Ellen

oryx Writing Word Macros 1999 Roman, Steven

python Programming Python 2010 Lutz, Mark

robin MySQL High Availability 2014 Bell, Charles

snail SSH, The Secure Shell 2005 Barrett, Daniel

or descending order (with the -r option):

$ sort -r animals.txt

snail SSH, The Secure Shell 2005 Barrett, Daniel

robin MySQL High Availability 2014 Bell, Charles

python Programming Python 2010 Lutz, Mark

oryx Writing Word Macros 1999 Roman, Steven

horse Linux in a Nutshell 2009 Siever, Ellen

donkey Cisco IOS in a Nutshell 2005 Boney, James

alpaca Intermediate Perl 2012 Schwartz, Randal

sort can order the lines alphabetically (the default) or numerically (with the -n option). I’ll demonstrate this with pipelines that cut the third field in animals.txt, the year of publication:

$ cut -f3 animals.txt                         Unsorted

2010

2005

2012

2014

2009

2005

1999

$ cut -f3 animals.txt | sort -n               Ascending

1999

2005

2005

2009

2010

2012

2014

$ cut -f3 animals.txt | sort -nr              Descending

2014

2012

2010

2009

2005

2005

1999

To learn the year of the most recent book in animals.txt, pipe the output of sort to the input of head and print just the first line:

$ cut -f3 animals.txt | sort -nr | head -n1

2014

MAXIMUM AND MINIMUM VALUES

sort and head are powerful partners when working with numeric data, one value per line. You can display the maximum value by piping the data to:

... | sort -nr | head -n1

and display the minimum value with:

... | sort -n | head -n1

As another example, let’s play with the file /etc/passwd, which lists the users that can run processes on the system.2 You’ll generate a list of all users in alphabetical order. Peeking at the first five lines, you’ll see:

$ head -n5 /etc/passwd

root:x:0:0:root:/root:/bin/bash

daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin

bin:x:2:2:bin:/bin:/usr/sbin/nologin

smith:x:1000:1000:Aisha Smith,,,:/home/smith:/bin/bash

jones:x:1001:1001:Bilbo Jones,,,:/home/jones:/bin/bash

Each line consists of strings separated by colons, and the first string is the username, so you can isolate the usernames with the cut command:

$ head -n5 /etc/passwd | cut -d: -f1

root

daemon

bin

smith

jones

and sort them:

$ head -n5 /etc/passwd | cut -d: -f1 | sort

bin

daemon

jones

root

smith

To produce the sorted list of all usernames, not just the first five, replace head with cat:

$ cat /etc/passwd | cut -d: -f1 | sort

To detect if a given user has an account on your system, match their username with grep. Empty output means no account.

$ cut -d: -f1 /etc/passwd | grep -w rutabaga         (produces no output)

$ cut -d: -f1 /etc/passwd | grep -w jones

jones

The -w option instructs grep to match full words only, not partial words, in case your system also has a username that contains “jones” such as sallyjones2.

Command #6: uniq

The uniq command detects repeated, adjacent lines in a file. By default, it removes the repeats. I’ll demonstrate this with a simple file containing capital letters:

$ cat letters

A

A

A

B

B

A

C

C

C

C

$ uniq letters

A

B

A

C

Notice that uniq reduced the first three A lines to a single A, but it left the last A in place because it wasn’t adjacent to the first three.

You can also count occurrences with the -c option:

$ uniq -c letters

3 A

2 B

1 A

4 C

I’ll admit, when I first encountered the uniq command, I didn’t see much use in it, but it quickly became one of my favorites. Suppose you have a tab-delimited file of students’ final grades for a university course, ranging from A (best) to F (worst).

$ cat grades

C Geraldine

B Carmine

A Kayla

A Sophia

B Haresh

C Liam

B Olivia

B Emma

A Elijah

D Noah

F Ava

You’d like to print the grade with the most occurrences. (If there’s a tie, print just one of the winners.) Begin by isolating the grades with cut and sorting them:

$ cut -f1 grades | sort

A

A

A

B

B

B

B

C

C

D

F

Next, use uniq to count adjacent lines:

$ cut -f1 grades | sort | uniq -c

3 A

4 B

2 C

1 D

1 F

Then sort the lines in reverse order, numerically, to move the most frequently-occurring grade to the top line:

$ cut -f1 grades | sort | uniq -c | sort -nr

4 B

3 A

2 C

1 F

1 D

and keep just the first line with head:

$ cut -f1 grades | sort | uniq -c | sort -nr | head -n1

4 B

Finally, since you want just the letter grade, not the count, isolate the grade with cut:

$ cut -f1 grades | sort | uniq -c | sort -nr | head -n1 | cut -c9

B

and there’s your answer, thanks to a six-command pipeline — our longest yet.

Case Study: Detecting Duplicate Files

Suppose you’re in a directory full of JPEG files and you want to know if any are duplicates.

$ ls

image001.jpg

image002.jpg

image003.jpg

...

You can answer this question with a pipeline. You’ll need another command, md5sum, which examines a file’s contents and computes a 32-character string called a checksum:

$ md5sum image001.jpg

75c8eb99afc71c32023488421f45e15f  image001.jpg

A given file’s checksum, for mathematical reasons, is very, very likely to be unique. If two files have the same checksum, therefore, they are duplicates. Here, md5sum indicates the first and third files are duplicates:

$ md5sum image001.jpg image002.jpg image003.jpg

75c8eb99afc71c32023488421f45e15f  image001.jpg

592edfb3a77dad54392edefd368722d2  image002.jpg

75c8eb99afc71c32023488421f45e15f  image003.jpg

Duplicates are easy to see with only three files, but what if you have three thousand? It’s pipes to the rescue. Compute all the checksums, use cut to isolate the first 32 characters of each line, and sort the lines to make any duplicates adjacent:

$ md5sum *.jpg | cut -c1-32 | sort

167413e8b7ce2a17c24b7b6a44e12705

1fd5505e1917e62ce3a6b0890bceb102

1fd5505e1917e62ce3a6b0890bceb102

2b11565d85da178b3a1942a22d20c624

...

Now add uniq to count repeated lines:

$ md5sum *.jpg | cut -c1-32 | sort | uniq -c

1 167413e8b7ce2a17c24b7b6a44e12705

2 1fd5505e1917e62ce3a6b0890bceb102

1 2b11565d85da178b3a1942a22d20c624

...

If there are no duplicates, all of the counts produced by uniq will be 1. Sort the results numerically from high to low, and any counts greater than 1 will appear at the top of the output:

$ md5sum *.jpg | cut -c1-32 | sort | uniq -c | sort -nr

3 f7d53f25b67715fd4959eb7787de7902

2 1fd5505e1917e62ce3a6b0890bceb102

2 75c8eb99afc71c32023488421f45e15f

1 2b11565d85da178b3a1942a22d20c624

...

Now let’s remove the non-duplicates. Their checksums are preceded by six spaces, the number one, and a single space. We’ll use grep -v to remove these lines:3

$ md5sum *.jpg | cut -c1-32 | sort | uniq -c | sort -nr | grep -v "      1 "

3 f7d53f25b67715fd4959eb7787de7902

2 75c8eb99afc71c32023488421f45e15f

2 e243255b6cf3b9403df53cb9cd6176e1

Finally, you have your list of duplicate checksums, sorted by the number of occurrences, produced by a beautiful six-command pipeline. If it produces no output, there are no duplicate files.

This command would be even more useful if it displayed the filename belonging to each checksum, but that operation requires features we haven’t discussed yet. You can still identify the files having a given checksum by searching with grep:

$ md5sum *.jpg | grep 75c8eb99afc71c32023488421f45e15f

75c8eb99afc71c32023488421f45e15f  image001.jpg

75c8eb99afc71c32023488421f45e15f  image003.jpg

and cleaning up the output with cut:

$ md5sum *.jpg | grep 75c8eb99afc71c32023488421f45e15f | cut -c35-

image001.jpg

image003.jpg

Summary

You’ve now seen the power of stdin, stdout, and pipes. They turn a small handful of commands into a collection of composable tools, proving that the whole is greater than the sum of the parts. Any command that reads stdin or writes stdout can participate in pipelines.4 As you learn more commands, you can apply the general concepts from this chapter to forge your own powerful combinations.

1   The POSIX standard calls this form of command a utility.

2   Some Linux systems store the user information elsewhere.

3   Technically, you don’t need the final sort -nr in this pipeline to isolate duplicates because grep removes all the non-duplicates.

4   Some commands do not use stdin/stdout and therefore aren’t useful in pipelines. Examples are mv and rm.

Chapter 2. Introducing the Shell

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd Chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at vwilson@oreilly.com.

So, you can run commands at a prompt. But what is that prompt? Where does it come from, how are your commands run, and why does it matter?

That little prompt is produced by a program called a shell. It’s a user interface that sits between you and the Linux operating system. Linux supplies several shells, and the most common (and the standard for this book) is called bash.

Bash and other shells do much more than simply run commands. For example, when a command includes a wildcard (*) to refer to multiple files at once:

$ ls *.py

data.py    main.py    user_interface.py

the wildcard is handled entirely by the shell, not by the program ls. The shell evaluates the expression *.py and invisibly replaces it with a list of matching filenames before ls runs. In other words, ls never sees the wildcard. From the perspective of ls, you typed the following command:

$ ls data.py main.py user_interface.py

The shell also handles the pipes you saw in Chapter 1. It redirects stdin and stdout transparently so the programs involved have no idea they are communicating with each other.

Every time a command runs, some steps are the responsibility of the invoked program, such as ls, and some are the responsibility of the shell. Expert users understand which is which. That’s one reason they can create long, complex commands off the top of their head and run them successfully. They already know what the command will do before they press Enter, in part because they understand the separation between the shell and the programs it invokes.

In this chapter, we’ll launch your understanding of the Linux shell. I’ll take the same minimalist approach I used for commands and pipes in Chapter 1. Rather than cover dozens of shell features, I’ll hand you just enough information to carry you to the next step of your learning journey:

- Pattern matching for filenames

- Variables to store values

- Redirection of input and output

- Quoting to disable certain shell features

- The search path for locating programs to run

- Saving changes to your shell environment

Pattern Matching for Filenames

In Chapter 1, you worked with several commands that accept filenames as arguments, such as cut, sort, and grep. These commands (and many others) accept multiple filenames as arguments. For example, you can search for the word “Linux” in 100 files at once:

$ grep Linux chapter1 chapter2 chapter3 chapter4 ... chapter100

Listing multiple files by name is a tedious time-waster, so the shell provides special characters as a shorthand. Many folks call them wildcards, but the more general concept is called pattern matching or globbing. Pattern matching is one of the two most common techniques for speed that Linux users learn. (The other is pressing the up arrow to recall the shell’s previous command, which I’ll cover in Chapter 3.)

The shell uses a few special characters to refer to files or directories with similar names. Most Linux users are familiar with the star or asterisk character (*) which matches any sequence of zero or more characters (except for a leading dot)1 in file or directory paths:

$ grep Linux chapter*

Behind the scenes, the shell (not grep!) expands the pattern chapter* into a list of 100 matching filenames. Then the shell runs grep.

TERMINOLOGY: EVALUATING EXPRESSIONS AND EXPANDING PATTERNS

Strings that you enter on the command line, such as hello world or chapter*, are called expressions. (An entire command is an expression too.)

When the shell interprets and handles special characters in an expression, such as stars and pipe symbols, we say that the shell evaluates the expression.

Pattern matching is one kind of evaluation. When the shell evaluates an expression that contains pattern-matching symbols, such as chapter*, and replaces it with filenames that match the pattern, we say that the shell expands the pattern.

Many users have also seen the question mark (?) special character, which matches any single character (except a leading dot). For example, you could search for the word Linux in chapters 1 through 9 only, by providing a single question mark to make the shell match single digits:

$ grep Linux chapter?

or in chapters 10 through 99, with two question marks to match two digits:

$ grep Linux chapter??

Fewer users are familiar with square brackets ([]), which request the shell to match a single character from a set. For example, you could search only the first five chapters:

$ grep Linux chapter[12345]

Equivalently, you could supply a range of characters with a dash:

$ grep Linux chapter[1-5]

You could also search even-numbered chapters, combining the star and the square brackets to make the shell match filenames ending in an even digit:

$ grep Linux chapter*[02468]

Any characters, not just digits, may appear within the square brackets for matching. For example, filenames that begin with a capital letter, contain an underscore, and end with an @ symbol would be matched by the shell in this command:

$ ls [A-Z]*_*@

Patterns are acceptable wherever you’d supply a file or directory path on the command line. For example, you can list all files in the directory /etc with names ending in .conf:

$ ls -1 /etc/*.conf

/etc/adduser.conf

/etc/appstream.conf

...

/etc/wodim.conf

Be careful using a pattern with a command that accepts just one file or directory argument, such as cd. You might not get the behavior you expect:

$ ls

Pictures   Poems    Politics

$ cd P*                                   Three directories will match

bash: cd: too many arguments

If a pattern doesn’t match any files, the shell leaves it unchanged to be passed literally as a command argument. In the following command, the pattern *.doc matches nothing in the current directory, so ls looks for a filename literally named *.doc and fails.

$ ls *.doc

/bin/ls: cannot access '*.doc': No such file or directory

When working with file patterns, two points are vitally important to remember. The first, as I’ve already emphasized, is that the shell, not the invoked program, performs this pattern-matching. I know I keep repeating this, but I’m constantly surprised by how many Linux users don’t know it and develop superstitions about why certain commands succeed or fail.

The second important point is that shell pattern-matching applies only to file and directory paths. It doesn’t work for usernames, hostnames, and other types of arguments that certain commands accept. You also cannot type (say) s?rt at the beginning of the command line and expect the shell to run the sort program. (Some Linux commands such as grep, sed, and awk perform their own brands of pattern-matching, which we’ll explore in Chapter 5.)

PATTERN-MATCHING AND YOUR OWN PROGRAMS

The shell expands a command’s file patterns before running the command. Therefore, programs that accept filenames as arguments automatically “work” with patterns (because the shell evaluates the patterns before the program runs). This is true even for programs and scripts you write yourself. For example, if you write a program that accepts multiple filenames on the command line and translates the files from English to Swedish, you can instantly run it with pattern-matching:

$ english2swedish *.txt

Evaluating Variables

A running shell can define variables and store values in them. A shell variable is a lot like a variable in algebra — it has a name and a value. An example is the shell variable HOME. Its value is the path to your Linux home directory, such as /home/smith. Another example is USER, whose value is your Linux username, which will assume is smith throughout this book.

The command printenv prints a variable’s value on stdout:

$ printenv HOME

/home/smith

$ printenv USER

smith

When the shell evaluates a variable, it replaces the variable name with its value. Simply place a dollar sign in front of the name. For example, $HOME evaluates to the string /home/smith. Supply a variable anywhere on the command line where its value would be appropriate.

The easiest way to watch the shell evaluate a command line is to use the echo command, which simply prints its arguments (after the shell is finished evaluating them):

$ echo My name is $USER and my files are in $HOME    Evaluating variables

My name is smith and my files are in /home/smith

$ echo ch*ter9                                       Evaluating a pattern

chapter9

Where Variables Come From

Variables like USER and HOME are predefined by the shell. They are set automatically when you log in. (More on this process later.) Traditionally, such predefined variables have uppercase names.

You also may define or modify a variable anytime by assigning it a value. For example, if you work frequently in the directory /usr/src/linux, you can assign its name to a variable:

$ work=/usr/local/etc

and use it as a handy shortcut with cd:

$ cd $work

$ pwd

/usr/local/etc

You may supply $work in any command that expects a directory:

$ cp myfile $work

$ ls $work

myfile

When defining a variable, no spaces are permitted around the equals sign. If you forget, the shell will assume the first word on the command line is a program to run:

$ work = /usr/local/etc

work: command not found

A user-defined variable like work is just as legitimate and usable as a system-defined variable like HOME. The only practical difference is that some Linux programs change their behavior internally based on the values of HOME, USER, and other system-defined variables. For example, a Linux program with a graphical interface might retrieve your username from the shell and display it. Such programs don’t pay attention to an invented variable like work because they weren’t programmed to do so.

Variables and Superstition

When you print the value of a variable with echo:

$ echo $HOME

/home/smith

you might think that the echo command examines the HOME variable and prints its value. That is not the case. echo knows nothing about variables. It just prints whatever arguments you hand it. What’s really happening is that the shell evaluates $HOME before running echo. From echo’s perspective, you typed:

$ echo /home/smith

This behavior is extremely important to understand, especially as we delve into more complicated commands. The shell evaluates the variables in a command — as well as patterns and other shell constructs — before executing the command.

Case Study: Patterns and Variables

Let’s test your understanding of pattern and variable evaluation. Suppose you’re in a directory with two subdirectories, mammals and reptiles, and the mammals subdirectory contains files named lizard.txt and snake.txt.

$ ls

mammals   reptiles

$ ls mammals

lizard.txt  snake.txt

In the real world, lizards and snakes are not mammals, so the two files should be moved to the reptiles subdirectory. Here are two proposed ways to do it. One works and one does not.

mv mammals/*.txt reptiles

FILES="lizard.txt snake.txt"

mv mammals/$FILES reptiles

TK

TK

Method 1 works because patterns match an entire file path. See how the directory name mammals is part of both matches for mammals/*.txt:

$ echo mammals/*.txt

mammals/lizard.txt mammals/snake.txt

So, method 1 operates as if you’d typed the following correct command:

$ mv mammals/lizard.txt mammals/snake.txt reptiles

Method 2 uses variables, which evaluate to their literal value only. They have no special handling for file paths.

$ echo mammals/$FILES

mammals/lizard.txt snake.txt

So, method 2 operates as if you’d typed the following problematic command:

$ mv mammals/lizard.txt snake.txt reptiles

This command looks for the file snake.txt in the current directory, not in the mammals subdirectory, and fails:

$ mv mammals/$FILES reptiles

/bin/mv: cannot stat 'snake.txt': No such file or directory

To make a variable work in this situation, use a loop that prepends the directory name mammals to each filename:

for f in $FILES

do

mv mammals/$f reptiles

done

Redirecting Input and Output

The shell controls the input and output of the commands it runs. You’ve already seen one example: pipes, which direct the stdout of one command to the stdin of another. The pipe syntax, |, is a feature of the shell.

Another shell feature is redirecting stdout to a file. For example, if you use grep to print matching lines from the animals.txt file from Chapter 1, the command writes to stdout by default:

$ grep Perl animals.txt

alpaca Intermediate Perl 2012 Schwartz, Randal

You can send that output to a file instead, using a shell feature called output redirection. Simply add the symbol > followed by the name of a file to receive the output:

$ grep Perl animals.txt > outfile                         No output is displayed

$ cat outfile

alpaca Intermediate Perl 2012 Schwartz, Randal

You have just redirected stdout to the file outfile instead of the display. If the file outfile doesn’t exist, it’s created. If it does exist, redirection overwrites its contents. If you’d rather append to the output file rather than overwrite it, use the symbol >> instead:

$ grep Perl animals.txt > outfile              Create or overwrite outfile

$ echo There was just one match >> outfile     Append to outfile

$ cat outfile

alpaca Intermediate Perl 2012 Schwartz, Randal

There was just one match

Output redirection has a partner, input redirection, that redirects stdin to come from a file instead of the keyboard. Use the symbol < followed by a filename to redirect stdin.

Many Linux commands that accept filenames as arguments, and read from those files, also read from stdin when run with no arguments. An example is wc for counting lines, words, and characters in a file:

$ wc animals.txt                            Reading from a named file

7  49 324 animals.txt

$ wc < animals.txt                          Reading from redirected stdin

7  49 324

It’s very important to understand how these two commands differ in behavior.

- In the first command, wc receives the filename animals.txt as an argument, so wc is aware that the file exists. wc deliberately opens the file on disk and reads its contents.

- In the second command, wc is invoked with no arguments, so it reads from stdin, which is usually the keyboard. The shell, however, sneakily redirects stdin to come from animals.txt instead. wc has no idea that the file animals.txt exists.

The shell can redirect input and output in the same command:

$ wc < animals.txt > count

$ cat count

7  49 324

and even use pipes at the same time. Here, grep reads from redirected stdin and pipes the results to wc, which writes to redirected stdout, producing the file count.

$ grep Perl < animals.txt | wc > count

$ cat count

1       6      47

You’ll dive deeper into such compound commands in Chapter 8 and see many other examples of redirection throughout the book.

STANDARD ERROR (STDERR) AND REDIRECTION

In your day-to-day Linux use, you may notice that some output cannot be redirected by >, such as certain error messages. For example, ask cp to copy a file that doesn’t exist and it produces this error message:

$ cp nonexistent.txt file.txt

cp: cannot stat 'nonexistent.txt': No such file or directory

If you redirect the output (stdout) of this cp command to a file, errors, the message still appears on screen:

$ cp nonexistent.txt file.txt > errors

cp: cannot stat 'nonexistent.txt': No such file or directory

and the file errors is empty (has zero size):

$ ls -l errors

-rw-r--r-- 1 smith smith 0 Jul  6 17:06 errors

Why does this happen? Linux commands can produce more than one stream of output. In addition to stdout, there is also stderr (pronounced “standard error” or “standard err”), a second stream of output that is traditionally reserved for error messages. The streams stderr and stdout look identical on the display but internally they are separate. You can redirect stderr with the symbol 2> followed by a filename:

$ cp nonexistent.txt file.txt 2> errors

$ ls -l errors

-rw-r--r-- 1 smith smith 61 Jul  6 17:07 errors

$ cat errors

cp: cannot stat 'nonexistent.txt': No such file or directory

and append stderr to a file with 2>> followed by a filename:

$ cp nonexistent.txt file.txt 2> errors

$ cp another.txt file.txt 2>> errors

$ cat errors

cp: cannot stat 'nonexistent.txt': No such file or directory

cp: cannot stat 'another.txt': No such file or directory

To redirect both stdout and stderr to the same file, use &> followed by a filename:

$ echo This file exists > goodfile.txt               Create a file

$ cat goodfile.txt nonexistent.txt &> all.output

$ cat all.output

This file exists

cat: nonexistent.txt: No such file or directory

Disabling Evaluation with Quotes and the Escape Character

Normally the shell uses whitespace as a separator between words. The following command has four words — a program name followed by three arguments:

$ ls file1 file2 file3

Sometimes, however, you need the shell to treat whitespace as part of a filename such as my cool file.

$ ls -l

-rw-r--r-- 1 smith smith 27 Aug  9 22:12 my cool file

The following command fails because the shell treats the space characters in the filename as separators:

$ cat my cool file

cat: my: No such file or directory

cat: cool: No such file or directory

cat: file: No such file or directory

To force the shell to treat the spaces as part of the filename, you have three options: single quotes, double quotes, and backslashes.

$ cat 'my cool file'

$ cat "my cool file"

$ cat my\ cool\ file

Single quotes tell the shell to treat every character in a string literally, even if the character ordinarily has special meaning to the shell:

$ echo '$HOME'

$HOME

Double quotes tell the shell to treat all characters literally except for certain dollar signs and a few others you’ll learn later.

$ echo "Notice that $HOME is evaluated"                  Double quotes

Notice that /home/smith is evaluated

$ echo 'Notice that $HOME is not'                        Single quotes

Notice that $HOME is not

A backslash, also called the escape character, tells the shell to treat the next character literally. This command escapes the dollar sign:

$ echo \$HOME

$HOME

Backslashes act as escape characters even within double quotes:

$ echo "The value of \$HOME is $HOME"

The value of $HOME is /home/smith

but not within single quotes:

$ echo 'The value of \$HOME is $HOME'

The value of \$HOME is $HOME

Use the backslash to print a double quote character within double quotes:

$ echo "This message is \"sort of\" interesting"

This message is "sort of" interesting

A backslash at the end of a line disables the special nature of the invisible newline character, allowing shell commands to span multiple lines:

$ echo "This is a very long message that needs to extend \

onto multiple lines"

This is a very long message that needs to extend onto multiple lines

Final backslashes are great for making pipelines more readable, like this one from Chapter 1:

$ cut -f1 grades \

| sort \

| uniq -c \

| sort -nr \

| head -n1 \

| cut -c9

When used this way, the backslash is sometimes called a line continuation character.

Locating Programs to Be Run

When the shell first encounters a simple command, such as +ls *.py+, it’s just a string of meaningless characters. Quick as a flash, the shell splits the string into two words, “ls” and “*.py”. In this case, the first word is the name of a program on disk, and the shell must locate it.

The program ls, it turns out, is an executable file in the directory /bin. You can verify its location with this command:

$ ls -l /bin/ls

-rwxr-xr-x 1 root root 133792 Jan 18  2018 /bin/ls

or you can change directories with cd /bin and run this lovely, cryptic-looking command:

$ ls ls

ls

which uses the command ls to list the executable file ls.

How does the shell know to find ls in the /bin directory? Behind the scenes, the shell consults a prearranged list of directories that it holds in memory, called a search path. The list is stored as the value of the shell variable PATH:

$ echo $PATH

/home/smith/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/java/bin

Directories in a search path are separated by colons (:). For a clearer view, convert the colons to newline characters by piping the output to the tr command, which translates one character into another (more details in Chapter 5):

$ echo $PATH | tr : "\n"

/home/smith/bin

/usr/local/bin

/usr/bin

/bin

/usr/games

/usr/lib/java/bin

The shell consults directories in your search path from first to last when locating a program like ls. “Does /home/smith/bin/ls exist? No. Does /usr/local/bin/ls exist? Nope. How about /usr/bin/ls? No again! Maybe /bin/ls? Yes, there it is! I’ll run /bin/ls.” This search happens too quickly to notice.2

The search path is a great example of taking something mysterious about Linux and showing it has an ordinary explanation. The shell doesn’t pull commands out of thin air or locate them by magic. It methodically examines directories in a list until it finds an executable file that matches a word on the command line.

Environments and Init Files, the Short Version

A running shell maintains a bunch of important information: the search path, the current directory, your preferred text editor, your customized shell prompt, and more. Collectively, this information is called the shell’s environment. Much of the environment is held in variables such as PATH. All of it is destroyed when the shell exits.

It would be extremely tedious to define every shell’s environment by hand. The solution is to define the environment once, in shell scripts called startup files and init files, and have every shell read and run these scripts on startup. The effect is that certain information appears to be “global” or “known” to all of your running shells.

I’ll dive into the gory details in a later chapter. For now, I’ll teach you about one init file so you can get through the next few chapters. It’s located in your home directory and named .bashrc (pronounced “dot bash R C”). Because its name begins with a dot, ls doesn’t display it by default.

$ ls $HOME

apple   banana   carrot

$ ls -a $HOME

.bashrc   apple   banana    carrot

If ~/.bashrc doesn’t exist, create it with a text editor. Commands you place in this init file will run automatically when a shell starts up,3 so it’s a great place to define variables and other things important to the shell. Here is a sample ~/.bashrc file. Lines beginning with # are comments.

# Set the search path

PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin

# Set the shell prompt

PS1='$ '

# Set your preferred text editor

EDITOR=emacs

# Start in my work directory

cd $HOME/Work/Projects

# Offer a hearty greeting

echo "Welcome to Linux, friend!"

Any changes you make to ~/.bashrc do not affect any running shells, only future shells. You can force a running shell to reread ~/.bashrc with either of the following commands:

$ source ~/.bashrc                     Uses the builtin "source" command

$ . ~/.bashrc                          Uses a dot

This process is known as sourcing the init file. If someone tells you to “source your dot-bash-R-C file,” they mean run one of the preceding commands.

WARNING

In real life, do not put all of your shell configuration in ~/.bashrc. Once you’ve read the details in a later chapter, examine your ~/.bashrc and move commands to their proper startup or init files as needed.

Summary

I’ve covered only a tiny number of bash features and their most basic uses. You’ll see many more in the chapters that follow, particularly in Chapter 6.For right now, your most important job is to understand:

- The shell exists and has important responsibilities

- The shell evaluates the command line before running any commands

- Commands can redirect stdin, stdout, and stderr

- Quoting and escaping disable the special nature of special shell characters

- The shell locates programs using a search path of directories

- You can save your changes to the shell’s behavior by adding commands to the file ~/.bashrc, which runs automatically on shell startup (more fully explained in a later chapter)

The better you understand the division between the shell and the programs it invokes, the more that the command line will make sense, and the better you can predict what will happen before you press Enter to run a command.

1   That’s why the command ls * doesn’t display dot files.

2   Some shells memorize (cache) the paths to programs as they’re located, cutting down on future searches.

3   This statement is oversimplified and explained in detail in later chapters.

Chapter 3. Rerunning Commands

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd Chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at vwilson@oreilly.com.

Suppose you’ve just executed a lengthy command with a detailed pipeline:

$ md5sum *.jpg | cut -c1-32 | sort | uniq -c | sort -nr

and you want to run it a second time. Don’t retype it! Instead, ask the shell to reach back into history and rerun the command. Behind the scenes, the shell keeps a record of the commands you invoke so you can easily recall and rerun them with a few keystrokes. This shell feature is called command history. Expert Linux users make heavy use of command history to speed up their work and avoid wasting time.

Similarly, suppose you make a mistake typing the preceding command before you run it, such as misspelling “jpg” as “jg”:

$ md5sum *.jg | cut -c1-32 | sort | uniq -c | sort -nr

To fix the mistake, don’t press the Backspace key dozens of times and retype everything. Instead, change the command in place. The shell supports command-line editing for fixing typos and performing all sorts of modifications like a text editor can.

This chapter will show you how to save lots of time and typing by leveraging the shell’s command history and command-line editing. As usual, I won’t attempt to be comprehensive — I’ll focus on the most practical and useful parts of these shell features.

LEARN TO TOUCH TYPE

All the advice in this book will serve you better if you can type quickly. No matter how knowledgeable you are, if you type 40 words per minute and your equally-knowledgeable friend types 120, they’re set up to work three times as fast as you. Search the web for “typing speed test” to measure your speed, then search for “typing tutor” and build a lifelong skill. Try to reach 100 words per minute. It’s worth the effort.

Viewing the Command History

A command history is simply a list of previous commands that you’ve executed in an interactive shell. To see a shell’s history, run the history command, which is a shell builtin. The commands appear in chronological order with ID numbers for easy reference. The output looks something like this:

$ history

1000  cd ~/Music

1001  ls

1002  mv jazz.mp3 jazzy-song.mp3

1003  play jazzy-song.mp3

...                                   Omitting 479 lines

1481  cd

1482  firefox https://google.com

1483  history                         Includes the command you just ran

The output of history can be hundreds of lines long (or more). Limit it to the most recent commands by adding an integer argument, which represents the number of lines to print:

$ history 3                      Print the 3 most recent commands

1482  firefox https://google.com

1483  history

1485  history 3

Since history writes to stdout, you also can process the output with pipes. For example, view your history a screenful at a time:

$ history | less

or print only the historical commands containing the word cd:

$ history | grep -w cd

1000  cd ~/Music

1092  cd ..

1123  cd Finances

1375  cd Checking

1481  cd

1484  history | grep -w cd

To clear (erase) the history for the current shell, use the -c option:

$ history -c

FREQUENTLY-ASKED QUESTIONS

How many commands are stored in a shell’s history?

The maximum is five hundred or whatever amount is stored in the shell variable HISTSIZE, which you can change:

$ echo $HISTSIZE

500

$ HISTSIZE=10000

Computer memory is so cheap and plentiful that it makes sense to set HISTSIZE to a large number, so you can recall and rerun commands from the distant past. (A history of 10,000 commands occupies only about 200K of memory.) Or be daring and store unlimited commands by setting the value to -1.

What text is appended to the history?

The shell appends exactly what you type, unevaluated. If you run ls $HOME, the history will contain “ls $HOME” not “ls /home/smith”. (There’s one exception: see “History Expressions Don’t Appear in the Command History”.)

Are repeated commands appended to the history?

The answer depends on the value of the variable HISTCONTROL. By default, if this variable is unset, then every command is appended. If the value is ignoredups (which I recommend), then repeated commands are not appended if they are consecutive. See man bash for other values.

$ HISTCONTROL=ignoredups

Does each shell have a separate history, or do all shells share a single history?

Each interactive shell has a separate history.

I launched a new interactive shell and it already has a history. Why?

Whenever an interactive shell exits, it writes its history to a file, ~/.bash_history or whatever path is stored in the shell variable HISTFILE.

$ echo $HISTFILE

/home/smith/.bash_history

New interactive shells load this file on startup, so they immediately have a history. It’s a quirky system if you’re running many shells because they all write $HISTFILE on exit, so it’s a bit unpredictable which history a new shell will hold.

The variable HISTFILESIZE controls how many lines of history are written to the file. If you change HISTSIZE to control the size of the history in memory, consider updating HISTFILESIZE as well.

$ echo $HISTFILESIZE

500

$ HISTFILESIZE=10000

Recalling Commands from the History

I’ll show you three ways to work efficiently by recalling commands from a shell’s history:

1. Cursoring, which is extremely simple to learn but often slow in practice

2. History expansion, which is harder to learn (frankly, it’s cryptic) but can be very fast

3. Incremental search, which is both simple and fast

Each method is best in particular situations, so I recommend learning all three. The more techniques you know, the better you can choose the right one in any situation.

Cursoring Through History

To recall your previous command in a given shell, press the up arrow key. It’s that simple. Keep pressing the up arrow to recall earlier commands in reverse chronological order. Press the down arrow to head in the other direction (toward more recent commands). When you reach the desired command, press Enter to run it.

Cursoring through the command history is one of the two most common speedups that Linux users learn. (The other is pattern-matching filenames with *, which you saw in Chapter 2.) Cursoring is efficient if your desired command is nearby in the history — no more than two or three commands in the past — but it’s tedious to reach commands that are further away. Whacking the up arrow 137 times gets old quickly.

The best use case for cursoring is recalling and running the immediately previous command. On many keyboards, the up arrow key is near the Enter key, so you can press the two keys in sequence with a quick flick of the fingers. On a full-sized American QWERTY keyboard, I place my right ring finger on the up arrow and my right index finger on Enter to tap both keys efficiently. (Try it.)

History Expansion

History expansion is a shell feature that accesses the command history using special expressions. The expressions begin with an exclamation point, which traditionally is pronounced “bang.” For example, two exclamation points in a row (“bang bang”) evaluates to the immediately previous command:

$ echo Hello world

Hello world

$ !!                           "Bang bang" = previous command

echo Hello world               The shell helpfully prints the command being run

Hello world

To rerun the most recent command that began with a certain string, place an asterisk in front of that string. To rerun the most recent grep command, run “bang grep”:

$ !grep

grep Perl animals.txt

alpaca Intermediate Perl 2012 Schwartz, Randal

To rerun the most recent command that contained a given string somewhere, not just at the beginning of the command, surround the string with question marks as well:1

$ !?grep?

history | grep -w cd

You can also retrieve a particular command from a shell’s history by its absolute position — the ID number to its left in the output of history. For example, the expression !1203 (“bang 1023”), means “the command at position 1023 in the history”:

$ history | grep hosts

1203  cat /etc/hosts

$ !1203                         The command at position 1023

cat /etc/hosts

127.0.0.1       localhost

127.0.1.1       example.oreilly.com

::1             example.oreilly.com

A negative value retrieves a command by its relative position in the history, rather than absolute position. For example, !-3 (“bang minus three”) means “the command you executed three commands ago”:

$ history

4197  cd /tmp/junk

4198  rm *

4199  head -n2 /etc/hosts

4199  cd

4200  history

$ !-3                         The command you executed three commands ago

head -n2 /etc/hosts

127.0.0.1       localhost

127.0.1.1       example.oreilly.com

History expansion is quick and convenient, if a bit cryptic. It can be risky, however, if you provide a wrong value and execute it blindly. Look carefully at the preceding example. If you miscounted and typed !-4 instead of !-3, you’d run rm * instead of the intended head command and delete files in your home directory by mistake! To mitigate this risk, append the modifier :p to print the command from your history but not execute it.

$ !-3:p

head -n2 /etc/hosts               Printed, not executed.

The shell appends the unexecuted command (head) to the history, so if it looks good, you can run it conveniently with a quick “bang bang”:

$ !-3:p

head -n2 /etc/hosts              Printed, not executed, and appended to history

$ !!                             Run the command for real

head -n2 /etc/hosts              Printed and then executed

127.0.0.1       localhost

127.0.1.1       example.oreilly.com

HISTORY EXPRESSIONS DON’T APPEAR IN THE COMMAND HISTORY

The shell appends commands to the history verbatim — unevaluated — as I mentioned in “Frequently-Asked Questions”. The one exception to this rule is history expansion. Its expressions are always evaluated before they’re added to the command history.

$ ls               Run any command

hello.txt

$ cd Music         Run some other command

$ !-2              Use history expansion

ls

song.mp3

$ history          View the history

1000  ls

1001  cd Music

1002  ls          "ls" appears in the history, not "!-2"

1003  history

This exception makes sense. Imagine trying to understand a command history full of expressions like !-15 and !-92 that refer to other history entries. You might have to trace a path through the whole history by eye to understand a single command.

Some people refer to history expansion as “bang commands,” but expressions like !! and !grep are not commands. They are string expressions that you can place anywhere in a command. As a demonstration, use echo to print the value of !! on stdout without executing it, and count the number of words with wc:

$ ls -l /etc | head -n3       Run any command

total 1584

drwxr-xr-x  2 root     root       4096 Jun 16 06:14 ImageMagick-6/

drwxr-xr-x  7 root     root       4096 Mar 19  2020 NetworkManager/

$ echo "!!" | wc -w           Count the words in the previous command

echo "ls -l /etc | head -n3" | wc -w

6

This is just a toy example to demonstrate that history expansions have more uses than executing commands. You’ll see a more practical, powerful technique in “Case Study: Never Delete the Wrong File Again”. It uses the history expansion !$, which means “the final word that you typed in the previous command.” Read the case study to discover why this might be the most beloved history expansion of all.

I’ve covered only a few features of command history here. For full information, run man history.

Case Study: Never Delete the Wrong File Again

Have you ever meant to delete files using a pattern, such as *.txt, but accidentally mistyped the pattern and wiped out the wrong files? Here’s an example with an accidental space character after the asterisk:

$ ls

123  a.txt   b.txt   c.txt  dont-delete-me  important-file  passwords

$ rm * .txt       DANGER!! Don't run this! Deletes the wrong files!

The most common solution to this hazard is to alias rm to run rm -i so it prompts for confirmation before each deletion:

$ alias rm='/bin/rm -i'                  Often found in a shell configuration file

$ rm *.txt

/bin/rm: remove regular file 'a.txt'? y

/bin/rm: remove regular file 'b.txt'? y

/bin/rm: remove regular file 'c.txt'? y

As a result, an extra space character needn’t be fatal, because the prompts from rm -i will warn that you’re removing the wrong files:

$ rm * .txt

/bin/rm: remove regular file '123'?      Something is wrong -- kill the command

The alias solution is cumbersome, however, because most of the time you might not want or need rm to prompt you. It also doesn’t work if you’re logged into another Linux machine without your aliases. I’ll show you a better way to avoid matching the wrong filenames with a pattern. The technique has two steps and relies on history expansion.

1. Verify. Before running rm, run ls with the desired pattern to see which files match.

$ ls *.txt

a.txt   b.txt   c.txt

2. Delete. If the output of ls looks correct, run rm !$ to delete the same files that were matched.2

$ rm !$

rm *.txt

The history expansion !$ (“bang dollar”) means “the final word that you typed in the previous command.” Therefore, rm !$ here is a shorthand for “delete whatever I just listed with ls,” namely *.txt. If you accidentally add a space after the asterisk, the output of ls will make it obvious — safely — that something is wrong:

$ ls * .txt

/bin/ls: cannot access '.txt': No such file or directory

123  a.txt   b.txt   c.txt  dont-delete-me  important-file  passwords

It’s a good thing you ran ls first instead of rm! You can now modify the command to remove the extra space and proceed safely. This two-command sequence — ls followed by rm !$ — is a great safety feature to incorporate into your Linux toolbox.

TIP

A related technique is peeking at a file’s contents with head before you delete it, to make sure you’re targeting the right file, and then running rm !$.

$ head myfile.txt

(first 10 lines of the file appear)

$ rm !$

rm myfile.txt

The shell also provides a history expansion !* (“bang star”) which matches all arguments you typed in the previous command, rather than just the final argument:

$ ls *.txt *.o *.log

a.txt   b.txt   c.txt   main.o   output.log   parser.o

$ rm !*

rm *.txt *.o *.log

In practice, I use !* much less often than !$. Its asterisk carries the same risk of being interpreted as a pattern-matching character for filenames (if you mistype something), so it’s not much safer than typing a pattern like *.txt by hand.

Incremental Search of Command History

Wouldn’t it be great if you could type a few characters of a command and the rest would appear instantly, ready to run? Well, you can. This speedy feature of the shell, called incremental search, is similar to the interactive suggestions provided by web search engines. In most cases, incremental search is the easiest and fastest technique to recall commands from history, no matter how long ago you ran them. I highly recommend adding it to your toolbox.

1. At the shell prompt, press ctrl-R (the “R” stands for reverse incremental search)

2. Start typing any part of a previous command — beginning, middle, or end

3. With each character you type, the shell displays the most recent historical command that matches your typing so far

4. When you see the command you want, press Enter to run it

For example, if you typed the command cd ~/Finances/Bank a while ago and you want to rerun it, press ctrl-R at the shell prompt. The prompt changes to indicate an incremental search:

(reverse-i-search)`':

Start typing the desired command. For example, type c:

(reverse-i-search)`': c

The shell displays its most recent command that contains the string c, highlighting what you’ve typed:

(reverse-i-search)`': less /etc/hosts

Type the next letter, d:

(reverse-i-search)`': cd

The shell displays its most recent command that contains the string cd, again highlighting what you’ve typed:

(reverse-i-search)`': cd /usr/local

Continue typing the command, adding a space and a tilde:

(reverse-i-search)`': cd ~

The command line becomes:

(reverse-i-search)`': cd ~/Finances/Bank

This is the command you want. Press Enter to run it, and you’re done in five quick keystrokes.

I’ve assumed here that cd ~/Finances/Bank was the most recent matching command in the history. What if it’s not? What if you typed a whole bunch of commands that contain the same string? If so, the preceding incremental search would have displayed a different match, such as:

(reverse-i-search)`': cd ~/Music

What now? You could type more characters to hone in on your desired command, but instead, press ctrl-R a second time. This keystroke causes the shell to jump the next matching command in the history:

(reverse-i-search)`': cd ~/Linux/Books

Keep pressing ctrl-R until you reach the desired command:

(reverse-i-search)`': cd ~/Finances/Bank

and press Enter to run it.

Here are a few more tricks with incremental search:

- To recall the most recent string that you searched for and executed, press ctrl-R twice in a row

- To stop an incremental search and continue working on the current command, press the Escape key, or ctrl-J, or any key for command-line editing (the next topic in this chapter) such as the left and right arrow keys

- To quit an incremental search and clear the command line, press ctrl-G or ctrl-C.

Take the time to become expert with incremental search. You’ll soon be locating commands with incredible speed.3

Command-Line Editing

There are all sorts of reasons to edit a command, either while you type it or after you’ve run it:

- To fix mistakes

- To create a command out of order, like typing the end of the command first, then moving to the beginning of the line and typing the beginning

- To construct a new command based on a previous one from your command history (a key skill for building up complex pipelines, as you’ll see in Chapter 8)

In this section, I’ll show you three ways to edit a command to build your skill and speed:

1. Cursoring, which is (again) the slowest and least powerful method but simple to learn

2. Caret notation, a form of history expansion

3. Emacs or Vim-style keystrokes to edit the command line in powerful ways

As usual, I recommend that you learn all three techniques for flexibility.

Cursoring Within a Command

Simply press the left-arrow and right-arrow keys to move back and forth on the command line, one character at a time. Use the Backspace and Delete keys to remove text, and then type any corrections you need. Table 3-1 summarizes these and other standard keystrokes for editing the command line.

Cursoring back and forth is easy but inefficient. It’s best when the changes are small and simple.

Table 3-1. Cursor Keys for Simple Command-Line Editing

Keystroke

Action

Left arrow

Move left by one character

Right arrow

Move right by one character

Ctrl + left arrow

Move left by one word

Ctrl + right arrow

Move right by one word

Home

Move to beginning of command

End

Move to end of command

Backspace

Delete one character before the cursor

Delete

Delete one character beneath the cursor

History Expansion with Carets

Suppose you’ve mistyped and run the following command by typing jg instead of jpg:

$ md5sum *.jg | cut -c1-32 | sort | uniq -c | sort -nr

md5sum: '*.jg': No such file or directory

To run the command properly, you could recall it from the command history, cursor over to the mistake and fix it, but there’s a quicker way to accomplish your goal. Just type the old (wrong) text, the new (corrected) text, and a pair of carets (^), like this:

$ ^jg^jpg

Press Enter, and the correct command will appear and run:

$ ^jg^jpg

md5sum *.jpg | cut -c1-32 | sort | uniq -c | sort -nr

...

The caret syntax, which is a type of history expansion, means, “In the previous command, instead of jg, substitute jpg.” Notice that the shell helpfully prints the new command before executing it, which is standard behavior for history expansion.

This technique changes only the first occurrence of the source string (jg) in the command. If your original command contained jg more than once, only the first instance would change to jpg.

MORE POWERFUL SUBSTITUTION WITH HISTORY EXPANSION

If you’re familiar with using the commands sed or ed to change a source string into a target string:

s/source/target/

the shell also supports it as a more powerful technique than what the carets provide. Begin with an expression for history expansion to recall a command, such as !!. Then add a colon, and end with a sed-style substitution. For example, to recall the previous command and replace jg by jpg (first occurrence only), just as caret notation does, run:

$ !!:s/jg/jpg/

You may begin with any history expansion you like, such as !md5sum to recall the most recent command beginning with md5sum, and perform the same replacement of jg by jpg.

$ !md5sum:s/jg/jpg/

This notation may be complicated-looking, but sometimes it’s quicker for achieving your goal than other command-line editing techniques. Run man history for full details.

Emacs or Vim-Style Command-Line Editing

The most powerful way to edit a command line is with familiar keystrokes inspired by the text editors Emacs and Vim. If you’re already skilled with one of these editors, you can jump into this style of command-line editing right away. If not, Table 3-2 will get you started with the most common keystrokes for movement and editing. Note that the Emacs “Meta” key is usually Escape (pressed and released) or Alt (pressed and held).

The shell default is Emacs-style editing. If you prefer Vim-style editing, run the following command (or add it to your ~/.bashrc file and source it):

$ set -o vi

To edit a command using Vim keystrokes, first press the Escape key to switch from insertion mode to command mode. Then use keystrokes from the table. To switch back to Emacs-style editing, run:

$ set -o emacs

Now practice, practice, practice until the keystrokes (either Emacs’s or Vim’s) are second nature. Trust me, you’ll quickly be paid back in saved time.

Table 3-2. Keystrokes for Emacs or Vim-Style Editing

Action

Emacs

Vim

Move forward by one character

Ctrl-f

h

Move backward by one character

Ctrl-b

l

Move forward by one word

Meta-f

w

Move backward by one word

Meta-b

b

Move to beginning of line

Ctrl-a

0

Move to end of line

Ctrl-e

$

Transpose (swap) two characters

Ctrl-t

Ctrl-r

Transpose (swap) two words

Meta-t

n/a

Capitalize next word

Meta-c

n/a

Upper-case next word

Meta-u

n/a

Lower-case next word

Meta-l

n/a

Change case of the current character

n/a

~

Insert the next character verbatim, including control characters

Ctrl-v

Ctrl-v

Delete forward by one character

Ctrl-d

x

Delete backward by one character

Backspace  or Ctrl-h

X

Cut forward by one word

Meta-d

n/a

Cut backward by one word

Meta-Backspace  or Ctrl-w

Ctrl-w

Cut from cursor to beginning of line

Ctrl-u

Ctrl-u

Cut from cursor to end of line

Ctrl-k

D

Delete the entire line

Ctrl-e Ctrl-u

dd

Paste (yank) the most recently-deleted text

Ctrl-y

p

Paste (yank) the next deleted text (after a previous yank)

Meta-y

n/a

Undo the previous editing operation

Ctrl-_

u

Undo all edits made so far

Meta-r

U

Switch from insertion mode to command mode

n/a

Escape

Switch from command mode to insertion mode

n/a

i

Abort an edit operation in progress

Ctrl-g

n/a

Clear the display

Ctrl-l

Ctrl-l

For more details, see these documents:

Emacs-style editing

https://www.gnu.org/software/bash/manual/html_node/Bindable-Readline-Commands.html

Vim-style editing

https://catonmat.net/ftp/bash-vi-editing-mode-cheat-sheet.pdf

Summary

Practice the techniques in this chapter and you’ll speed up your command-line use immensely. Three of the techniques in particular transformed the way I use Linux, and I hope they will for you too:

- Deleting files with !$ for safety

- Incremental search with ctrl-R

- Emacs-style command-line editing

1   You can omit the trailing question mark here — !?grep — but in some cases it’s required, such as sed-style history expansion (see “More Powerful Substitution with History Expansion”).

2   I’m assuming that no matching files were added or removed behind your back after the ls step.

3   While writing this book, I frequently reran version-control commands like git add, git commit, and git push. Incremental search made rerunning these commands a breeze.

Chapter 4. Cruising the Filesystem

In the movie The Adventures of Buckaroo Banzai Across the 8th Dimension, a classic cult comedy from 1984, the swashbuckling title character offers the following Zen-like words of wisdom: “Remember, no matter where you go… there you are.” Buckaroo could very well have been talking about the Linux filesystem:

$ cd /usr/share/lib/etc/bin              No matter where you go...

$ pwd

/usr/share/lib/etc/bin                   ...there you are.

It’s also the case that wherever you are in the Linux filesystem — your current directory — you will eventually go somewhere else (to another directory). The faster and more efficiently you can perform this navigation, the more productive you can be.

The techniques in this chapter will help you navigate between directories more quickly with less typing. They have enormous bang for the buck, with small learning curves, deceptive simplicity, and big payoffs.

- Moving quickly to a specific directory

- Returning rapidly to a directory you’ve visited before

For a quick refresher on Linux directories, see Appendix A.

Visiting Specific Directories Efficiently

If you ask 10 Linux experts what is the most tedious aspect of the command line, seven of them say, “Typing long directory paths.”1 After all, if your work files are in /home/smith/Work/Projects/Apps/Neutron-Star/src/include, your financial documents are in /home/smith/Finances/Bank/Checking/Statements, and your videos are in /data/Arts/Video/Collection, it’s no fun to retype these paths over and over. In this section, you’ll learn techniques to navigate to a given directory as efficiently as possible.

Jump to Your Home Directory

We’ll begin with the basics. No matter where you are in the filesystem, you can return to your home directory by running cd with no arguments:

$ pwd

/etc                              Start somewhere else

$ cd                            Run cd with no arguments...

$ pwd

/home/smith                       ...and you're home again

To jump to subdirectories within your home directory from anywhere in the filesystem, refer to your home directory with a shorthand rather than its absolute path (/home/smith). One shorthand is the shell variable HOME:

$ cd $HOME/Work

Another is a tilde:

$ cd ~/Work

Both $HOME and ~ are expressions expanded by the shell, a fact that you can verify by echoing them to stdout:

$ echo $HOME ~

/home/smith /home/smith

The tilde can also refer to another user’s home directory if you place it immediately in front of their username:

$ echo ~jones

/home/jones

Move Faster With Tab Completion

When you’re entering cd commands, save typing by pressing the Tab key to produce directory names automatically. As a demonstration, visit a directory with subdirectories, such as /usr:

$ cd /usr

$ ls

bin  games  include  lib  local  sbin  share  src

Suppose you want to visit the subdirectory share. Type sha and press the Tab key once:

$ cd sha<Tab>

The shell completes the directory name for you:

$ cd share/

This handy shortcut is called tab completion. It works immediately when the text that you’ve typed matches a single directory name. When the text matches multiple directory names, your shell needs more information to complete the desired name. Suppose you had typed only s and pressed Tab:

$ cd s<Tab>

The shell cannot complete the name share (yet) because other directory names begin with s too: sbin and src. Press Tab a second time and the shell prints all possible completions to guide you:

$ cd s<Tab><Tab>

sbin/  share/  src/

and waits for your next action. To resolve the ambiguity, type another character, h, and press Tab once:

$ cd sh<Tab>

The shell completes the name of the directory for you, from sh to share:

$ cd share/

In general, press Tab once to perform as much completion as possible, or press twice to print all possible completions. The more characters you type, the less ambiguity and the better the match.

Tab completion is great for speeding up navigation. Instead of typing a lengthy path like /home/smith/Projects/Website/src/include, type as little as you want and keep pressing the Tab key. You’ll get the hang of it quickly with practice.

TAB COMPLETION VARIES BY PROGRAM

Tab completion may behave differently for different commands. When the command is cd, the Tab key completes directory names. For other commands that operate on files, such as cat, grep, and sort, tab completion expands filenames too. If the command is ssh (secure shell), it completes hostnames. If the command is chown (change the owner of a file), it completes usernames. You can even create your own completion rules for speed, as we’ll see in Example 4-1. Also see man bash and read the topic “programmable completion.”

Hop to Frequently-Visited Directories Using Aliases

If you visit a faraway directory frequently, such as /home/smith/Work/Projects/Website/src/include, create an alias that performs the cd operation:

# In a shell configuration file:

alias work="cd $HOME/Work/Projects/Website/src/include"

Simply run the alias anytime to reach your destination:

$ work

$ pwd

/home/smith/Work/Projects/Website/src/include

EDIT FREQUENTLY-EDITED FILES WITH AN ALIAS

Sometimes, the reason for visiting a directory frequently is to edit a particular file. If that’s the case, consider defining an alias to edit that file by absolute path without changing directory. The following alias edits $HOME/.bashrc no matter where you are in the filesystem — no cd required.

$ alias rcedit='$EDITOR $HOME/.bashrc'

If you regularly visit lots of directories with long paths, you can create aliases for each of them. This approach has some disadvantages, however:

- It’s hard to remember all those aliases

- You might accidentally create an alias with the same name as an existing command, causing a conflict

An alternative is to create a shell function like the one in Example 4-1, which I’ve named qcd (“quick cd”). This function accepts a string key as an argument, such as work or recipes, and runs cd to a selected directory path.

Example 4-1. A function for cd-ing to faraway directories

# Define the qcd function

function qcd {

# Accept 1 argument that's a string key, and perform a different

# "cd" operation for each key.

case "$1" in

work)

cd $HOME/Work/Projects/Website/src/include

;;

recipes)

cd $HOME/Family/Cooking/Recipes

;;

video)

cd /data/Arts/Video/Collection

;;

beatles)

cd $HOME/Music/mp3/Artists/B/Beatles

;;

*)

# The supplied argument was not one of the supported keys

>&2 echo "qcd: unknown key '$1'"

return 1

;;

esac

# Helpfully print the current directory name to indicate where you are

pwd

}

# Set up tab completion

complete -W "work recipes video beatles" qcd

Store the function in a shell configuration file such as $HOME/.bashrc, source it, and it’s ready to run. Type qcd followed by one of the supported keys to quickly visit the associated directory:

$ qcd beatles

/home/smith/Music/mp3/Artists/B/Beatles

As a bonus, the script’s final line runs the command complete, a shell builtin that sets up customized tab completion for qcd, so it completes the four supported keys. Now you don’t have to remember qcd’s arguments! Just type qcd followed by a space and press the Tab key twice, and the shell will print all the keys for your reference, and you can complete any of them in the usual way.

$ qcd <Tab><Tab>

beatles  recipes  video    work

$ qcd v<Tab><Enter>                       Completes 'v' to 'video'

/data/Arts/Video/Collection

Make a Big Filesystem Feel Smaller with CDPATH

The qcd function has limitations: it handles only the directories that you specify and may need updates as your needs change. The shell provides a more general cd-ing solution without these shortcomings, called a cd search path. This shell feature transformed how I navigate the Linux filesystem.

Suppose you have an important subdirectory that you visit often, named Photos. It’s located at /home/smith/Family/Memories/Photos. As you cruise around the filesystem, anytime you want to get to the Photos directory, you may have to type a long path, such as:

$ cd ~/Family/Memories/Photos

Wouldn’t it be great if you could shorten this path to just Photos, no matter where you are in the filesystem, and reach your subdirectory?

$ cd Photos

Normally, this command would fail:

bash: cd: Photos: No such file or directory

unless you happen to be in the correct parent directory (~/Family/Memories) or some other directory with a Photos subdirectory by coincidence. Well, with a little setup, you can instruct cd to search for your Photos subdirectory in locations other than your current directory. The search is lightning-fast and looks only in parent directories that you specify. For example, you could instruct cd to search $HOME/Family/Memories in addition to the current directory. Then, when you type cd Photos from elsewhere in the filesystem, cd will succeed:

$ pwd

/etc

$ cd Photos

/home/smith/Family/Memories/Photos

A cd search path works like your command search path, $PATH, but instead of finding commands, it finds subdirectories. Configure it with the shell variable CDPATH which has the same format as PATH: a list of directories separated by colons. If your CDPATH consists of these four directories, for example:

$HOME:$HOME/Projects:$HOME/Family/Memories:/usr/local

and you type:

$ cd Photos

then cd will check the existence of the following directories in order, until it finds one or it fails entirely:

1. Photos in the current directory

2. $HOME/Photos

3. $HOME/Projects/Photos

4. $HOME/Family/Memories/Photos

5. /usr/local/Photos

In this case, cd succeeds on its fourth try and changes directory to $HOME/Family/Memories/Photos. If two directories in $CDPATH have a subdirectory named Photos, the earlier parent wins.

NOTE

Ordinarily, a successful cd prints no output. When cd locates a directory using your CDPATH, however, it prints the absolute path on stdout to inform you of your new current directory.

Fill CDPATH with your most important or frequently used parent directories, and you can cd into any of their subdirectories from anywhere in the filesystem, no matter how deep they are, without typing most of the path. Trust me, this is awesome, and the following case study should prove it.

Case Study: Organize Your Home Directory for Fast Navigation

We’ll now use CDPATH to simplify the way you navigate your home directory. With a little configuration, we’ll make many directories within your home directory easily accessible with little typing, no matter where you are in the filesystem. This technique works best if your home directory is well-organized with at least two levels of subdirectories, as in Figure 4-1.

Figure 4-1. Two levels of subdirectories in the directory /home/smith

The trick is to set up your CDPATH to include, in order:

1. $HOME

2. Your choice of subdirectories of $HOME

3. The relative path ..

By including $HOME, you can jump immediately to any of its subdirectories (Family, Finances, Linux, Music, and Work) from anywhere else in the filesystem without typing a leading path:

$ pwd

/etc                                   You're outside your home directory

$ cd Work

/home/smith/Work

$ cd Family/School                   You jumped 1 level below $HOME

/home/smith/Family/School

By including subdirectories of $HOME in your CDPATH, you can jump into their subdirectories in one shot:

$ pwd

/etc                                   Anywhere outside your home directory

$ cd School

/home/smith/Family/School              You jumped 2 levels below $HOME

All the directories in your CDPATH so far are absolute paths in $HOME and its subdirectories. By including the relative path .. however, you empower new cd behavior in every directory. No matter where you are in the filesystem, you can jump to any sibling directory (../sibling) by name without typing the two dots, because cd will search your current parent. For example, if you’re in /usr/bin and want to move to /usr/lib, all you need is cd lib:

$ pwd

/usr/bin                              Your current directory

$ ls ..

bin   include   lib   src             Your siblings

$ cd lib

/usr/lib                              You jumped to a sibling

Or, if you’re a programmer working on code that has subdirectories src, include, and docs:

$ pwd

/usr/src/myproject

$ ls

docs   include   src

you can jump between the subdirectories concisely:

$ cd docs                            Change your current directory

$ cd include

/usr/src/myproject/include           You jumped to a sibling

$ cd src

/usr/src/myproject/src               Again

The CDPATH for the tree in Figure 4-1 would contains seven items:

# In a shell configuration file:

export CDPATH=$HOME:$HOME/Work:$HOME/Family:$HOME/Finances:$HOME/Linux:$HOME/Music:..

Source your configuration file to load its settings into your current shell:

$ . $HOME/.bashrc

Now you can cd to a large number of important directories without typing long directory paths, just short directory names. Hooray!

This technique works best if all subdirectories beneath the CDPATH directories have unique names. If you have duplicate names, such as $HOME/Music and $HOME/Linux/Music, you might not get the behavior you want. The command cd Music will always check $HOME before $HOME/Linux and consequently will not locate $HOME/Linux/Music by search.

To check for duplicate subdirectory names in the first two levels of $HOME, try this brash one-liner. It lists all subdirectories and sub-subdirectories of $HOME, isolates the sub-subdirectory names with cut, sorts the list, and counts occurrences with uniq.

$ cd

$ ls -d */ && (ls -d */*/ | cut -d/ -f2-)) | sort | uniq -c | sort -nr | less

You may recognize this duplicate-checking technique from “Case Study: Detecting Duplicate Files”. If the output displays any counts greater than 1, you have duplicates.

Returning to Directories Efficiently

You’ve just seen how to visit a directory efficiently. Now I’ll show you how to revisit a directory quickly when you need to go back.

Toggle Between Two Directories With “cd -”

Suppose you change directory with cd:

$ pwd

/home/smith/Finances/Bank/Checking/Statements

$ cd /etc

and then think, “No, wait, I want to go back to the Statements directory where I just was.” Don’t retype the long directory path. Just run cd with a dash as an argument.

$ cd -

/home/smith/Finances/Bank/Checking/Statements

This command returns to your previous directory and helpfully prints its absolute path so you know where you are.

To jump back and forth between a pair of directories, run cd - repeatedly. This is a timesaver when you’re doing focused work in two directories in a single shell. There’s a catch, however: the shell remembers just one previous directory at a time. For example, if you are toggling between /usr/local/bin and /etc:

$ pwd

/usr/local/bin

$ cd /etc                 The shell remembers /usr/local/bin

$ cd -                    The shell remembers /etc

/usr/local/bin

$ cd -                    The shell remembers /usr/local/bin

/etc

and you run cd without arguments to jump to your home directory:

$ cd                      The shell remembers /etc

the shell has now forgotten /usr/local/bin as a previous directory:

$ cd -                    The shell remembers /home

/etc

$ cd -                    The shell remembers /etc

/home

The next technique overcomes this limitation.

Toggle Between Many Directories With pushd and popd

The cd - command is helpful for toggling between two directories, but what if you have three or more to keep track of? Suppose you’re creating a local website on your Linux computer. This task often involves four or more directories:

- The location of live, deployed web pages, such as /var/www/html

- The webserver configuration directory, often /etc/apache2

- The location of SSL certificates, often /etc/ssl/certs

- Your work directory, such as /home/smith/Work/Projects/Website/src

Believe me, it’s tedious to keep typing:

$ cd /home/smith/Work/Projects/Website/src

$ cd /var/www/html

$ cd /etc/apache2

$ cd /home/smith/Work/Projects/Website/src

$ cd /etc/ssl/certs

If you have a large, windowing display, you can ease the burden by opening a separate shell window for each directory. But if you’re working in a single shell (say, over SSH), take advantage of a shell feature called a directory stack. It lets you quickly travel among multiple directories with ease, using the built-in shell commands pushd, popd, and dirs. The learning curve is maybe 10 minutes, and the huge payoff in speed lasts a lifetime.2

A stack is a special kind of list with two operations, push and pop. “Pushing” adds an item to the beginning of the list, which is traditionally called the top of the list. “Popping” removes an item from the top. A directory stack is, as its name implies, a stack of directory names. Initially, it contains only your current directory, but you can add (push) and remove (pop) directories and rapidly cd among them.

NOTE

Every shell maintains its own directory stack. It’s part of the shell’s local environment.

Push a Directory Onto the Stack

The command pushd (short for “push directory”) does all of the following:

1. Adds a given directory to the top of the stack

2. Performs a cd to that directory

3. Prints the stack from top to bottom for your reference

I’ll build a directory stack of four directories, pushing them onto the stack one at a time:

$ pwd

/home/smith/Work/Projects/Website/src

$ pushd /var/www/html

/var/www/html /home/smith/Work/Projects/Website/src

$ pushd /etc/apache2

/etc/apache2 /var/www/html /home/smith/Work/Projects/Website/src

$ pushd /etc/ssl/certs

/etc/ssl/certs ww/etc/apache2 /var/www/html /home/smith/Work/Projects/Website/src

$ pwd

/etc/ssl/certs

When the shell prints the stack after each pushd operation, the current directory is the leftmost (top) directory.

View a Directory Stack

Print a shell’s directory stack with the dirs command. It does not modify the stack.

$ dirs

/etc/apache2 /home/smith/Work/Projects/Website/src /var/www/html

If you prefer a top-to-bottom display of the stack, use the -p option:

$ dirs -p

/etc/ssl/certs

/etc/apache2

/var/www/html

/home/smith/Work/Projects/Website/src

and even pipe the output to the command nl to number the lines from zero onward:

$ dirs -p | nl -v0

0  /etc/ssl/certs

1  /etc/apache2

2  /var/www/html

3  /home/smith/Work/Projects/Website/src

If you prefer this top-down format, consider making an alias:

# In a shell configuration file:

alias dirs='dirs -p | nl -v0'

Pop a Directory From the Stack

The popd command (“pop directory”) is the reverse of pushd. It:

1. Removes one directory from the top of the stack

2. Performs a cd to the new top directory

3. Prints the stack from top to bottom for your reference

For example, if your stack has four directories:

$ dirs

/etc/ssl/certs /etc/apache2 /var/www/html /home/smith/Work/Projects/Website/src

then repeatedly running popd will traverse these directories from top to bottom:

$ popd

/etc/apache2 /var/www/html /home/smith/Work/Projects/Website/src

$ popd

/var/www/html /home/smith/Work/Projects/Website/src

$ popd

/home/smith/Work/Projects/Website/src

$ popd

bash: popd: directory stack empty

$ pwd

/home/smith/Work/Projects/Website/src

TIP

The pushd and popd commands are such timesavers that I recommend creating two-character aliases that are as quick to type as cd:

# In a shell configuration file:

alias gd=pushd

alias pd=popd

Swap Directories on the Stack

Now that you can build and empty the directory stack, let’s focus on practical use cases. pushd with no arguments swaps the top two directories in the stack and navigates to the new top directory. Let’s jump between /etc/apache2 and your work directory several times by simply running pushd. See how the third directory /var/www/html remains in the stack as the first two directories swap positions.

$ dirs

/etc/apache2 /home/smith/Work/Projects/Website/src /var/www/html

$ pushd

/home/smith/Work/Projects/Website/src /etc/apache2 /var/www/html

$ pushd

/etc/apache2 /home/smith/Work/Projects/Website/src /var/www/html

$ pushd

/home/smith/Work/Projects/Website/src /etc/apache2 /var/www/html

Notice that pushd behaves similarly to the cd - command, toggling between two directories, but it does not have the limitation of remembering just one directory.

Turn a Mistaken cd Into a pushd

Suppose you are jumping among several directories with pushd and you accidentally run cd instead:

$ dirs

/home/smith/Work/Projects/Website/src /var/www/html /etc/apache2

$ cd /etc/ssl/certs

$ dirs

/etc/ssl/certs /var/www/html /etc/apache2

Oops, the accidental cd command removed /home/smith/Work/Projects/Website/src from the stack, replacing it with /etc/ssl/certs. But don’t worry. You can reverse this regrettable mistake and add /home/smith/Work/Projects/Website/src back to the stack without typing that long directory name. Just run pushd twice, once with a dash argument and once without:

$ pushd -

/home/smith/Work/Projects/Website/src /etc/ssl/certs /var/www/html /etc/apache2

$ pushd

/etc/ssl/certs /home/smith/Work/Projects/Website/src /var/www/html /etc/apache2

Let’s dissect why this works:

- The first pushd returns to your previous working directory, /home/smith/Work/Projects/Website/src, and pushes it onto the stack. pushd, like cd, accepts a dash as an argument to mean “go back to my previous directory.”

- The second pushd command swaps the top two directories, bringing you back to /etc/ssl/certs. The end result is that you’ve restored /home/smith/Work/Projects/Website/src to the second position in the stack, exactly where it would have been if you hadn’t made your mistake.

This “oops, I forgot a pushd” command is useful enough that it’s worth an alias. I call it slurp because in my mind, it “slurps back” a directory that I lost by mistake.

# In a shell configuration file:

alias slurp='pushd - && pushd'

Go Deeper Into the Stack

What if you want to cd between directories in the stack other than the top two? pushd and popd accept a positive or negative integer argument to operate further into the stack. The command:

$ pushd +N

shifts N directories from the top of the stack to the bottom, and then performs a cd to the new top directory. A negative argument (-N) shifts directories in the opposite direction, from the bottom to the top, before performing the cd.3

$ dirs

/etc/ssl/certs /home/smith/Work/Projects/Website/src /var/www/html /etc/apache2

$ pushd +1

/home/smith/Work/Projects/Website/src /var/www/html /etc/apache2 /etc/ssl/certs

$ pushd +2

/etc/apache2 /etc/ssl/certs /home/smith/Work/Projects/Website/src /var/www/html

In this manner, you can jump to any other directory in the stack with a simple command. If your stack is long, however, it may be difficult to judge a directory’s numeric position by eye. So, print the numeric position of each directory with nl, as you did in “View a Directory Stack”:

$ dirs -p | nl -v0

0  /etc/apache2

1  /etc/ssl/certs

2  /home/smith/Work/Projects/Website/src

3  /var/www/html

To shift /var/www/html to the top of the stack (and make it your current directory), run pushd +3.

Summary

All of the techniques in this chapter are easy to grasp with a bit of practice and will save you lots of time and typing. The techniques I’ve found particularly life-changing are:

- CDPATH for rapid navigation

- pushd and popd for rapid returns

- The occasional cd -

1   I made this up, but it’s surely true.

2   An alternative is to opens multiple, virtual displays using command-line programs like screen and tmux, which are called terminal multiplexers. They’re more effort to learn than directory stacks but worth checking out.

3   Programmers may recognize this operation as rotating the stack.

Table of Contents

Preface

The Command-Line Skills You Need

What You’ll Learn

Audience and Prerequisites

Conventions Used in This Book

Using Code Examples

O’Reilly Online Learning

How to Contact Us

Acknowledgments

I. Core Concepts

1. Combining Commands

Input, Output, and Pipes

Six Commands To Get You Started

Command #1: wc

Command #2: head

Command #3: cut

Command #4: grep

Command #5: sort

Command #6: uniq

Case Study: Detecting Duplicate Files

Summary

2. Introducing the Shell

Pattern Matching for Filenames

Evaluating Variables

Where Variables Come From

Variables and Superstition

Case Study: Patterns and Variables

Redirecting Input and Output

Disabling Evaluation with Quotes and the Escape Character

Locating Programs to Be Run

Environments and Init Files, the Short Version

Summary

3. Rerunning Commands

Viewing the Command History

Recalling Commands from the History

Cursoring Through History

History Expansion

Case Study: Never Delete the Wrong File Again

Incremental Search of Command History

Command-Line Editing

Cursoring Within a Command

History Expansion with Carets

Emacs or Vim-Style Command-Line Editing

Summary

4. Cruising the Filesystem

Visiting Specific Directories Efficiently

Jump to Your Home Directory

Move Faster With Tab Completion

Hop to Frequently-Visited Directories Using Aliases

Make a Big Filesystem Feel Smaller with CDPATH

Case Study: Organize Your Home Directory for Fast Navigation

Returning to Directories Efficiently

Toggle Between Two Directories With “cd -”

Toggle Between Many Directories With pushd and popd

Summary