Practical Linux System Administration

A Guide to Installation, Configuration, and Management

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Ken Hess

Practical Linux System Administration

by Ken Hess

Copyright © 2022 Hess Media and Consulting, LLC. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Acquisitions Editor: John Devins

- Development Editor: Jeff Bleiel

- Production Editor: Gregory Hyman

- Interior Designer: David Futato

- Cover Designer: Karen Montgomery

- Illustrator: Kate Dullea

- September 2022: First Edition

Revision History for the Early Release

- 2021-08-31: First Release

- 2021-10-29: Second Release

- 2021-12-09: Third Release

- 2022-01-24: Fourth Release

- 2022-03-03: Fifth Release

- 2022-05-19: Sixth Release

- 2022-06-28: Seventh Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098109035 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Practical Linux System Administration, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-10903-5

Chapter 1. Getting Started with Linux

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the first chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

Linux system administration means different things to different people. Administration, for this book, means the daily actions that a Linux system administrator must take to manage and support users, maintain system health, implement best practices for security, install software, and perform housekeeping tasks. This chapter covers Linux installation, initial setup, and system exploration using simple shell commands.

You’ll spend a significant portion of your time at the command line, also known as the command line interface (CLI). Linux system administrators rarely install or use graphic user interfaces (GUIs) on their supported server systems. This chapter introduces you to the CLI and some simple commands to navigate the filesystem, to locate important files, and to familiarize yourself with the Linux CLI.

Installing Linux

One of the first things every Linux system administrator (sysadmin) learns is how to install Linux. There’s no single correct way to install Linux but there are a few guidelines and suggestions that will make your life easier in the future as your user’s needs change.

While this section won’t go into step-by-step detailed instructions on how to install Linux, the basic steps are outlined here. For the majority of junior-level sysadmins, system installation generally takes place via automated means such as Kickstart or other enterprise-level delivery system.

Preparing Your System for Linux

If this is your first time to install Linux, I suggest that you install it into a virtual machine (VM) so that you don’t have to dedicate an entire piece of hardware for a learning system and so that you don’t potentially render your system inoperable by attempting to install Linux in parallel to your current system creating a multi-boot computer. (Setting up multi-booting is a more advanced concept and out of the scope of this book.)

A good place to start with virtualization, if you don’t already have it installed, is to download and install the latest version of VirtualBox (https://www.virtualbox.org/). VirtualBox is an application that allows your current computer to act as a virtual machine host system where you may install virtual guests, such as Linux into what amounts to a separate functioning computer system. Virtualbox runs on many different host operating systems (OSs) and supports a variety of guest operating systems including Linux. The host OS and guest OS can be different from one another. Your computer (host OS) can be a Windows, Mac, or Linux-based system but have guest Linux systems installed on it as VirtualBox VMs.

Downloading and Installing Linux

Next, you’ll need to select a Linux distribution (distro) to install so that you can practice issuing commands, changing configurations, rebooting, installing software, creating users, and so on. I suggest that you should select a Linux distro based on the one that your current employer uses. If your company doesn’t use Linux yet or you’re not employed in a system administrator role, then select from one of the popular distributions described below:

Debian

Debian is a top-level distribution from which many other distributions are derived. Debian is community-supported, open source, and free.

OpenSUSE

OpenSUSE is a community-supported, top-level distribution that has many faithful followers worldwide. Its commercial version SUSE Linux Enterprise has widespread adoption.

Red Hat Enterprise Linux

Red Hat is a commercially-supported Linux that enjoys worldwide enterprise adoption and is now owned by IBM.

Ubuntu

Ubuntu is a very popular, Debian-derived community and commercially-supported distribution. Ubuntu also offers ready-made VirtualBox (and other) virtual machines to help you get a quick start.

The downloaded ISO file is a bootable Linux image. You don’t have to do anything to it if you use it to create a virtual machine. A virtual machine will boot from the ISO image and begin the installation process. After you’ve configured your virtual machine in VirtualBox, select Settings from the Oracle VM VirtualBox Manager, as shown in Figure 1-1.

Figure 1-1. The Oracle VirtualBox Manager application and a configured virtual machine.

Then, select Storage, as shown in Figure 1-2.

Figure 1-2. Virtual machine settings with Storage settings selected.

Select the empty optical disk drive under the IDE Controller in the Storage Devices pane and then select the optical disk icon in the Attributes pane to browse for your ISO image file. See Figure 1-3.

Figure 1-3. Select the ISO image from the list.

Once you’ve selected your ISO image, click OK to proceed. Now, when you start your VM, it will boot from this ISO image to begin installation onto your VM’s virtual disk.

When your system boots, you can accept the default settings. If you have some experience installing Linux, then you can change the default settings to suit your needs. Create yourself a user account when prompted to do so. If your distribution prompts you to give the root account a password, do so. You must remember this password because without it you’ll have to reinstall your Linux VM or try to recover it. Installing Linux can take several minutes and a reboot is required at the end of the installation process.

Getting to Know Your New Linux System

After installation, the first you need to do is to login using the username and password that you created during installation. Upon login, you’re placed into your home directory inside a shell or operating environment. Your home directory is a subdirectory of the /home directory. The Linux filesystem is a hierarchical filesystem, similar to Microsoft Windows. At the very top level, there is the root directory, which is represented by the / symbol. Windows uses a drive letter, such a C: for the root directory. On Windows, you can have many drive letters that all have their own root levels, such as C:, D:, E:, and so on. In Linux, there is only one root directory, /. All other directories are subdirectories of the root directory. Figure 1-4 is an illustration of the Linux root directory and its subdirectories.

Figure 1-4. The Linux Hierarchical Filesystem showing the root directory (/) and all of its subdirectories.

Note that there are only directories under the / (root) filesystem and no individual files. All files are kept in directories. You’ll explore many of these subdirectories throughout the rest of the book. The following table gives you a brief overview of the files and information contained in each directory.

Directory

Description

/

The root filesystem only contains other directories but no individual files.

/bin

The binaries directory contains executable files. Points to /usr/bin.

/dev

The device directory contains device files to address peripherals.

/etc

Contains system configuration files for users and services.

/home

User’s home directories.

/lib

System libraries files. Points to /usr/lib.

/media

Directory for mounting media such as USB drives or DVD disks.

/mnt

The mount directory for mounting remote filesystems.

/opt

Directory in which third-party software is installed.

/proc

A virtual filesystem that tracks system processes.

/root

The root user’s home directory.

/run

Variable and volatile run-time data.

/sbin

System binary (executable) files.

/srv

Might contain data from system services.

/sys

Contains kernel information.

/tmp

Temporary file directory for session information and temporary file storage.

/usr

Programs and libraries for users and user-related programs.

/var

Variable files such as logs, spools, and queues.

System files are protected from user modification. Only the root (administrative) user can modify system configuration files and settings. Users generally only have write access to their own home directories, the /tmp directory, and shared directories specifically created and modified by the administrator.

In the next section, you learn how to interact with your new Linux system at the command line.

Learning the Command Line Interface

The command line interface or CLI is how most system administrators interact with their Linux systems because server systems don’t typically have a graphic interface. In Microsoft Windows terminology, a CLI only system, such as a Linux server, would be equivalent to a Windows Server Core system where you only have access to command line utilities.

As the name suggests, you interact with the Linux system using commands that you enter with a keyboard or standard input (stdin). The source from standard input can also be file redirection, programs, and other sources, but in the context for this book, stdin refers to keyboard input unless otherwise noted. Many commands are informational and display data about the system or system activities to the screen or standard output (stdout). Sometimes you’ll receive an error from the system that’s also known as standard error or stderr. You’ll see the shorthand and the long versions of these references used interchangeably throughout this text and in other Linux-related documentation that you’ll find elsewhere.

You must learn a few commands to successfully interact with the filesystem. And by learn, I mean know them without looking them up online. There’s only a handful of commands like this and there are few options that you should also commit to memory so that your interaction with the system becomes natural and efficient. And don’t worry about harming the system with any command that I cover. I’ll warn you when a command should be used with care or caution.

There are a few things you need to know before jumping into issuing commands. The first is that Linux doesn’t use file extensions. This means that the file, filename.exe has no more meaning to Linux than does the file, Financial_Report.txt or Resume.doc. They are all files and might or might or not be executable or text files. In Linux, you can name a file anything you want to.

The second thing to know about Linux is that filenames are case-sensitive. In other words, the two files, filename.txt and filename.TXT are two different files. I will prove this later in the chapter. For now, take my word for it. The third thing to know is that a file’s permission determines whether you can execute the file, edit the file, or even look at the file’s contents. Fourth, all Linux locations are stated from the / (root) directory. For example, if you mention the password file in Linux, it’s always shown as /etc/passwd. This is known as the absolute path and is the standard convention for speaking of or referring to files on the system.

Fifth, Linux assumes you know what you want to do and that you’ve spelled everything correctly when you issue a command, so be careful because some actions are irreversible.

Finally, Linux, like Unix, or more generally *nix systems are not “chatty” like the Windows® operating systems are. Linux systems, for example, don’t prompt you with an “Are you sure?” message when you remove (delete) files. The Linux system assumes that you want to execute the command that you issued and that you correctly spelled all parts of the command. Spelling counts at the Linux command line.

Commands for Filesystem Navigation

Navigating the Linux filesystem means exploring the various system directories, learning to return to your home directory, and listing directory contents in different ways. If you’re a Windows user and you’ve worked at the command prompt (CMD window) on that platform, then the Linux command line will be familiar to you.

The following short list of commands will acquaint you with the Linux filesystem, files, and the contents of your home directory.

pwd

The pwd or print working directory command displays where you are on the filesystem. If you issue the pwd command now, followed by the ENTER key, the command responds with /home/<your login name>. Always assume that you press the ENTER key after each command so that you can receive a response.

$ pwd

/home/student1

The $ is your shell prompt that shows you’re logged into the system as a user. You’ll use this command more later in the chapter.

cd

The cd, change directory, or current directory command places you into a new directory, returns you to your home directory, moves you to a higher level, or to a subdirectory. The cd command is analogous to the Windows cd command.

$ cd /etc

$ pwd

/etc

Simply entering cd returns you to your home directory regardless of where you are on the filesystem. From the example above.

$ pwd

/etc

$ cd

$ pwd

/home/student1

When you cd to a directory, use its absolute path.

$ cd /usr/bin

You can cd to a subdirectory without the absolute path if you’re currently in the parent directory.

$ cd

$ pwd

/home/student1

$ cd /usr

$ cd bin

$ pwd

/usr/bin

The cd command is one that you’ll use every day that you connect to a Linux system.

ls

The ls or list command displays a list of files and directories of the location you specify. If you don’t specify a location, ls displays the list of files and directories in your current directory.

$ cd

$ pwd

$ /home/student1

$ ls

You have no visible files on the system yet because you haven’t created any and none exist by default. However, you can list files from other directories by specifying the absolute path to the directory list you wish to see.

$ ls /usr/bin

a2x               getcifsacl                   p11-kit                   snmpping

a2x.py            getconf                      pack200                   snmpps

ac                getent                       package-cleanup           snmpset

There are too many files to list here in the /usr/bin directory, so I’ve truncated it to these few.

There are actually files in your home directory but they’re hidden because of the way they’re named. Files that begin with a period (.) are hidden from a regular ls command. To see these files, you must use a command option to allow you to see all files.

$ ls -a

.  ..  .bash_history  .bash_logout  .bash_profile  .bashrc  .gnupg  .zshrc

Your listing might vary slightly from this one but understand that all directories and files whose names begin with a (.) are hidden from standard file lists. You can cd into hidden directories or list files within them.

$ ls .gnupg

private-keys-v1.d  pubring.kbx

The ls command is one you’ll certainly use every time you login to your Linux system and is certainly one that you want to commit to memory. I will use the ls command throughout this book and introduce you to many more options for it along the way. Now that you’ve learned the basics of filesystem navigation, it’s time to cover rebooting and shutting down your system.

Starting, Rebooting, and Shutting Down a Linux System

The most basic tasks facing any Linux sysadmin is that of starting, restarting, and shutting down a system. If you have experience with computers, you know that powering off a system without issuing a shutdown command is bad. It’s bad because doing so can, for example, corrupt open files, it can leave open files open, it disrupts running services, and it can cause problems with database transaction logs--possibly resulting in data loss.

Proper knowledge of how to start, restart, and shutdown a system is of great value to sysadmins.

Starting a System

For physical systems, you press the power button and release to start a system. This begins the power-on self tests (POST) and boot process. Watching the console during boot is important because the system notifies you and logs any issues on the way up. Your goal is to watch the screen for any errors or anomalies along the way. The worst message of all is a “kernel panic” which will be covered later in the book. Hopefully all is well with your system and the process ends with a login prompt.

The startup process is quite short but can identify systemic problems such as memory, disk, filesystem, and network issues. I’ll cover troubleshooting in a later chapter, but be aware now that you should watch the boot process carefully noting any problems for later investigation.

Restarting a System

Restarting or rebooting a system is a standard sysadmin practice. Although you might read or hear the contrary, there’s nothing wrong with rebooting your system. In fact, it’s recommended that you do so on a regular basis for all of the reasons stated under the previous section. Restarting a system clears memory, refreshes connections, and ensures that the system is healthy. A good reboot can cure certain nagging problems, such as an application that drains your system’s memory, but only temporarily.

Any issues that are resolved with a reboot should be investigated more thoroughly after the system is up and stable. Restarting a system gives you an opportunity to do some troubleshooting before application problems, logging problems, or network problems place the system back into a state when it requires another reboot.

Shutting Down a System

Shutting down means issuing a command that gently and appropriately closes all programs and eventually powers off the system. This gentle shutdown also has the effect of warning users that the system is going down, so that everyone can save their work and logoff.

System shutdown should be reserved for hardware maintenance, relocation, or decommissioning. Some enterprise policies require that systems go through a full shutdown once a year to identify any hardware issues that might not manifest themselves except through a complete system failure. Technicians and sysadmins usually take this opportunity to perform hardware maintenance or hardware checks at the same time.

Summary

This first chapter gets you up and running with a live Linux system, covers some Linux basics, explores a few essential commands, and instructs you on the how and why of system startup, rebooting, and shutdown. In Chapter 2, you learn more about using the command line interface using commands to create, remove, and modify files. You also learn about Linux permissions, how to set them, how to interpret them, and how to set a global default permission for users.

Chapter 2. Diving Deeper into the Command Line Interface

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the second chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

For system administrators (sysadmins), the command line interface (CLI) is home. Typing at the keyboard is standard fare for sysadmins. You’ll need to become comfortable with the command line, its idiosyncrasies, and its shortcuts--yes, there are command line shortcuts. There’s a lot for you to learn about Linux at the command line. There are dozens of commands each with dozens of options. Sure, you’ll only use a handful of commands and a limited number of options for each command but you need to know how to find the options you need and how to use them when you need to.

The true power of the CLI is in its ease of use. The CLI was the first interface that users and programmers had with which to address their operating environments. The fact that the CLI is still in use some 50 years later is a testament to its power and its usefulness to the sysadmin and user alike. In this chapter, you learn to work at the command line as a regular user and as the root user. You also learn to set and modify file permissions, and the effects that those permissions have on files.

Working as a Regular User

There are two user types on a Linux system: regular users and the root user. Regular users each receive their own home directory and a somewhat limited use of the system. Users have unlimited power in their own home directories to create, modify, remove, and manipulate files but have almost no power outside of that single location. Many system commands are available to regular users while other commands are restricted to those who either have been granted limited root user access through the sudo command or through direct access to the root user account.

The general, and most security conscious, rule is that you should always work as a regular user unless some task requires administrative (root user) access, which is covered in the next section.

Working as the Root User

The administrator or root user is the all-powerful account on any Linux system. The root user can create, edit, move, or remove any file on the system. The root user can reboot, change runlevels, and shutdown the system. There are three methods of becoming the root user.

- Login as the root user

- Use the su (substitute user) command

- Use the sudo command

Login as Root

You can directly login as the root user on a system either via SSH across the network, or interactively at the console. It’s not recommended to SSH to a system and login as root. The primary reason is that if you allow across-the-network root access to a system, then it’s possible that malicious actors can attempt to brute force a root login. You don’t want this to happen. Later in the book, I’ll show you how to prevent SSH root logins. Some Linux distributions prevent SSH root logins by default, while others leave it up to the administrators to decide.

You shouldn’t directly login as root at the console either because doing so prevents system logging from recording who has logged in and become root. Recording who uses the root user account is important because when something goes wrong, you want to know which administrator performed the actions. This record-keeping’s purpose is not to lay blame but is necessary to meet some regulatory requirements and to correct actions of a system administrator who needs a teachable moment or some advanced training. The next two options we discuss are better, safer ways to become the root user.

su to Root

One of the appropriate methods of becoming the root user is to use the su (substitute user) command. The caveat with using su is that the user must know the root user password. If administrative users know the root password, then it’s difficult to prevent those same administrators from directly logging in as root. Using the su command to become root is acceptable but only if the root password changes after each use. In larger enterprises, security groups maintain root passwords and system administrators are allowed to check out the root password on a temporary basis to perform maintenance.

The root user may su to any other user account on the system without knowing the user’s password. This power gives administrators the capability to login as any user for troubleshooting purposes because it’s often difficult for users to accurately describe problems they’re experiencing. It also prevents a user from having to reveal their password to an administrator, which should force the user to change their password.

To su to another account is a simple procedure. Issue the su command and the user account you wish to su to. For this example, I use the full prompt rather than just the $ to demonstrate the user change.

[bjones@server1] $ su root

Password:

#

The # prompt informs you that you are now logged in as the root user. The user prompt is $ and the root is # to distinguish a standard user’s prompt from the root user’s prompt. Any command you issue now is done so with root privilege, which means that you must be careful because there are no restrictions on the account.

The better method of using su is to do so using the su - command because the - means that you also want to take on the root user’s full environment rather than just the account privilege. The display is much too long to show here but if you issue the env command, you’ll see the original user’s environment variables rather than root’s.

# env

Use the exit command to return to the original user account.

# exit

[bjones@server1] $

And now issue the su command with the - option. You don’t have to specify root in this command because the default is root.

[bjones@server1] $ su -

Password:

# env

The root user’s environment variables are now displayed. Using the su - command is the equivalent of logging into the console as the root user.

Actually, any user may su to any other user account but to do so requires one to know a user’s password.

[bjones@server1] $ su cdavis

Password:

[cdavis@server1] $

Use the sudo Command

The best method to obtain root access is to use the “substitute user do” or “execute a command as another user” (sudo) command. The sudo command allows an appropriately configured user account to issue individual commands as the root user. The sudo command must precede each command issued. On first use, the sudo command requires that the sudoer (A user account configured for sudo use) supply their own password. Knowledge of the root password isn’t required.

$ sudo env

[sudo] password for bjones:

bjones is not in the sudoers file.  This incident will be reported.

[bjones@server1 ~]$

The warning that the user is not in the sudoers file means that the user account, bjones, is not configured in the /etc/sudoers file. In the next section, I demonstrate how to setup a user to be a sudoer.

NOTE

The sudo command, the /etc/sudoers file, and the sudoer user label are interesting because they have their own unique pronunciations. The accepted pronunciation for sudo is ‘soodoo’ and sudoer is ‘soodooer.’ Some sysadmins pronounce them as ‘soodoe’ and ‘soodoe-ers’ and no one takes issue with either pronunciation.

Creating a sudoer

You must have root user access to edit the /etc/sudoers file and you must use the visudo utility, whose only purpose is to edit the /etc/sudoers file. You can’t successfully edit it directly with vi, emacs, or any other text editor. To edit the /etc/sudoers file, issue the visudo command as root with no options.

# visudo

The /etc/sudoers file is a simple text file that describes users, groups, and commands allowed to work with root privileges. You can create a sudoer with very restrictive permissions (i.e., to run a single command as root) or permissive: Run any command as root without entering a password to do so. I prefer to configure a mixture of the two by creating sudoers who can run any command as root but must supply their password to do so.

There are hundreds of possible configuration scenarios for the /etc/sudoers file and for sudoers. It’s out of the scope of this book to explore more than what’s given here as examples. In this first example, I demonstrate how I set up my own user account to use sudo.

## Allow root to run any commands anywhere

root    ALL=(ALL)       ALL

khess   ALL=(ALL)       ALL

I copy the root user’s setting and insert my user account in its place. The setting takes effect immediately. It’s not recommended to set up a user account to use sudo without issuing a password. Using a password when issuing a command is an attempt to make it more difficult to make mistakes while wielding root privilege. The same can be said of the sudo command itself. The theory is that if an administrator has to issue the sudo command that they will make fewer mistakes as root because it requires the user to think specifically about their command action and its results.

Setting and Modifying Permissions

In this section you learn how to read, set, and modify file permissions. You must learn file permissions so that you can appropriately set and modify access to files and directories. Knowing file permissions also helps with troubleshooting when users can’t access a file or directory.

File permissions are simple but central to Linux security. Their simplicity can make them susceptible to neglect and misconfiguration. Frustrated sysadmins sometimes open permissions up to solve a problem but never return to the issue or reset the permissions to their proper settings.

Read, Write, and Execute

The three Linux file permissions or modes are read (r), write (w), and execute (x).

Read

View a file or list directory contents.

Write

Create and modify a file or copy, move, and create files in a directory.

Execute

Execute/run a file or cd into a directory.

As mentioned in Chapter 1, filenames have nothing to do with file actions. File actions and capabilities are all permission-based.

Using the rwx designations in permissions is known as symbolic mode. The symbolic mode is one of two methods of identifying permissions. The other is the numeric mode which assigns values to each of the rwx permission modes.

Numerical Permission Values

Each of the permission modes has its own assigned numerical value. This shortcut method makes setting permissions easier for sysadmins.

The read permission has a value of 4, write has a value of two, and execute has a value of 1. Permission values can range from 0 to 7. A zero permission value means no permission.

- Read - 4

- Write - 2

- Execute -1

- None - 0

In the next section, you find out how these permissions work together with group permissions to create a simple, but complete file security system.

Group Permissions

There are four file group permissions: user, group, others, and all. The all group includes user, group, and others. It is a shorthand method of globally assigning permissions to a file or directory. Each group has a shorthand designation as well.

- User - u

- Group - g

- Other - o

- All - a

Users and sysadmins may set permissions on files for each group individually or for all groups at once. Each Linux file and directory is assigned read, write, and execute permissions for each group. The next section brings all of the permissions settings together .

Bringing Permissions Settings All Together

The examples in this section will use the file, file.txt. If you want to follow along with the example, issue the following command to set up your file:

$ touch file.txt

This command creates an empty file named file.txt for you. Next, issue the ls command with the -l (long) option to see file permissions.

$ ls -l

-rw-rw-r--. 1 khess khess 0 Jun 19 17:35 file.txt

Figure 2-1 illustrates the positions and their designations (in red). The first position is for special file types such as directories that will have a d in that position. Regular files have a - to show that they’re not directories or other special files. The next nine positions are user, group, and other permissions locations. The first “triad” or three positions are for the user, the second for group, and the final three are for other users.

Figure 2-1. Special character position and user, group, and other permission locations

Figure 2-2 shows you the numerical permissions for each listing and then an explicit label for the user (u), group (g), and other (o) triads.

Figure 2-2. Numerical permission values and user, group, and other (ugo) designations.

As you can see from Figure 2-2 above, numerical permissions are additive for each triad to create a permission profile for a file. For example, a file with -rw-rw-r-- permissions has a numeric permission value of 664. The read permission has a value of 4 added to the value of the write permission, which is 2, equals six. All three possible permissions, rwx, yield a value of seven.

If a file has a permission equal to 750, the rwx representation is -rwxr-x---. This means that others outside of the designated user and group have no permissions for the file.

NOTE

The other group is often referred to as world. For example, if permissions for a file are -rw-rw-r--, this file is referred to as world readable rather than as other readable. Permissions for the other group are especially sensitive because allowing write or execute permissions to files and directories to others can be a security risk.

In the next section, you learn how to set and change file permissions using multiple methods.

Changing File Permissions

Setting and changing file permissions or modes is a common system administrator task. Each file on the filesystem has its own individual permissions that either allows or denies access to users, groups, and others. To change file permissions, you use the chmod (Change mode) command. You can set or modify permissions with the chmod command in multiple ways. You don’t have to be consistent. You can use chmod with numeric, rwx, or ugo designations, but not both together in the same command. I demonstrate several possibilities and practical examples in the following sections.

NOTE

Some sysadmins find the symbolic (rwx and ugo) method easier to grasp than the numeric (0, 1, 2, 4) method. You can use either or both methods because they are equivalent to one another.

Symbolic Mode

Changing permissions using the symbolic mode method is quite simple. Referring back to the original file.txt file that you created in a previous example, view the original permissions with the ls -l command.

$ ls -l

-rw-rw-r--. 1 khess khess 0 Jun 19 17:35 file.txt

The current file permissions aren’t adequate. You need to restrict anyone else but yourself from even reading this file. How do you do it? You remove the read permission from others. Removing is equivalent to subtraction because you are subtracting a permission from the current ones given to the file. So, to remove read permission from the file, you subtract read from others using the chmod command.

$ chmod o-r file.txt

$ ls -l

-rw-rw----. 1 khess khess 0 Jun 19 17:35 file.txt

You have removed read permission from the file for others. Now, no one but you can read (or write to) this file. When you create a shell script and attempt to execute it with ./file.sh and nothing happens, you should check the file’s permissions to see if you’ve added the execute permission to it.

NOTE

To execute an executable file or script that is not in your path, you must provide the absolute path to the file. If the file is in your current directory, you must tell the shell that it is in your current directory and that you’d like to execute it. Use ./script_name.sh to inform the shell that the file is executable and in your current directory. Here, script_name.sh is the name of the file you wish to execute.

$ touch file.sh

$ echo “echo Hello” > file.sh

$ ./file.sh

-bash: ./file.sh: Permission denied

Permission denied? But, I just created the file in my home directory. Checking permissions reveals the problem.

$ ls -l

-rw-rw-r--. 1 khess khess  11 Jun 29 19:58 file.sh

The file, file.sh, although named with a .sh extension, you recall that extensions have no effect in Linux and realize that file.sh isn’t executable because it doesn’t have the execute permission and therefore you receive the “Permission denied” message. To fix the problem, add the execute permission for yourself.

$ chmod u+x file.sh

$ ls -l

-rwxrw-r--. 1 khess khess 11 Jun 29 19:58 file.sh

Now, file.sh is executable.

$ ./file.sh

Hello

You can add or subtract multiple permissions from a file and even add and subtract permissions within the same command. Here are some examples of each action. The first command removes (deletes) (rm file.txt) the file from any previous example.

To add multiple permissions to a file:

$ rm file.txt

$ touch file.txt

$ ls -l

-rw-rw-r--. 1 khess khess   0 Jun 29 20:13 file.txt

$ chmod ug+x,o+w file.txt

-rwxrwxrw-. 1 khess khess 0 Jun 29 20:13 file.txt

To subtract multiple permissions from a file:

$ ls -l

-rwxrwxrw-. 1 khess khess 0 Jun 29 20:13 file.txt

$ chmod a-x,o-rw file.txt

$ ls -l

-rw-rw----. 1 khess khess 0 Jun 29 20:13 file.txt

Now add execute permission for all groups and remove read permission for others:

$ rm file.txt

$ touch file.txt

$ ls -l

-rw-rw-r--. 1 khess khess   0 Jun 29 20:13 file.txt

$ chmod a+x,o-r file.txt

$ ls -l

-rwxrwx--x. 1 khess khess 0 Jun 29 20:23 file.txt

WARNING

Be careful that you explicitly define which group you want to add or subtract permissions. The reason is that simply supplying a +x or -r defaults to all.

If you don’t specify to which groups you wish to add permissions or subtract permissions from, the default behavior is for the system to assume the intended group is all. This can be dangerous from a security perspective. Never grant a particular permission to all groups unless that is what you intended to do. The execute permission is granted to all groups because you didn’t explicitly define which group should receive it.

$ rm file.txt

$ touch file.txt

$ ls -l

-rw-rw-r--. 1 khess khess   0 Jun 29 20:34 file.txt

$ chmod +x file.txt

$ ls -l

-rwxrwxr-x. 1 khess khess 0 Jun 29 20:35 file.txt

The execute permission was granted to all groups because you didn’t explicitly define which group should receive it.

Numeric Mode

For the purposes of clarity and comparison, the examples in this section are duplicates of the examples in the previous section. But here, we use the numeric mode (rather than the symbolic mode) of changing permissions. Create a new file and check its permissions.

$ rm file.txt

$ touch file.txt

$ ls -l

-rw-rw-r--. 1 khess khess   0 Jun 29 21:12 file.txt

Remove the read permission from the other group using the numeric method. First, calculate the current permission value of the file and then what you want the new value to be. Currently, the file’s permission value is 664. The desired value is 660.

$ chmod 660 file.txt

$ ls -l

-rw-rw----. 1 khess khess 0 Jun 29 20:12 file.txt

Using the numeric method, there’s no adding or subtracting of permissions. You simply reassign a value to the file. In a previous example, you added the execute permission to all and subtracted the read permission from others.

$ rm file.txt

$ touch file.txt

$ ls -l

-rw-rw-r--. 1 khess khess   0 Jun 29 20:13 file.txt

$ chmod a+x,o-r file.txt

$ ls -l

-rwxrwx--x. 1 khess khess 0 Jun 29 20:23 file.txt

The numeric equivalent is to reassign the value of the original file (664) to the new one (771).

$ rm file.txt

$ touch file.txt

$ ls -l

-rw-rw-r--. 1 khess khess   0 Jun 29 20:13 file.txt

$ chmod 771 file.txt

$ ls -l

-rwxrwx--x. 1 khess khess 0 Jun 29 20:23 file.txt

Either method of changing permissions is perfectly acceptable. There’s no stigma with using one over the other. Like most sysadmins, I use both methods interchangeably. It depends more on context and how quickly I want to do something. The process of changing permissions will become automatic to you with some practice and a few mistakes along the way.

Default Permissions Explained: umask

You might have noticed that when you create a new file, it’s created with specific permissions: 664 or -rw-rw-r--. For the root user the default permissions for a new file are 644 or -rw-r--r--. You might now wonder how this happens. There is a global setting called a umask or user file-creation mask that masks or filters certain permissions from being given to files by default. The execute permission is never given by default so it’s not explicitly masked by the umask setting. To find out what your system umask is, use the umask command.

$ umask

0002

You might now wonder why the umask reports four digits and we’ve only worked with three so far. The first (leftmost) digit is for special permissions such as setuid, setgid, and sticky, which I’ll cover in a later chapter. For now, focus on the other three digits: 002. These three digits correspond to rwx. When you create a new file, certain permissions are filtered out. In the case of the 002 umask, the write (w) permission is filtered out, so that new files are created as -rw-rw-r--. The 2 is for write. When a new file is created, the write permission is masked from the other group and therefore isn’t given to the new file.

For the root user, the umask is 0022. The write (w) permission is masked from both group and other. The reason for a umask is security. When a regular user creates a file, you don’t want everyone else to be able to write to it. You must explicitly grant this permission. For the root user, the umask prevents the root group and others from writing to files by default. This security feature prevents daemons or programs that might run as root from writing to certain sensitive files such as the /etc/passwd file. Everyone may read the file but only the root user may write to it.

You can change your own umask value by issuing the umask command and a new value. This temporarily changes the umask during your current login session.

$ umask 006

$ touch test.txt

$ ls -l test.txt

-rw-rw----. 1 khess khess 0 Jun 29 22:16 test.txt

To make this change permanent, do the following to append the new umask to the end of the .bashrc file:

$ echo umask 006 >> .bashrc

$ source .bashrc

$ umask

0006

Now, every time you login your umask will be set to 006 or 0006, which yields a more secure -rw-rw---- new default file permission.

Summary

In this chapter, you gained more experience working at the command line, you learned some new commands, and perhaps more importantly, you learned to read, set, and change file permissions. In Chapter 3, you learn some file editing basics and how to modify the user’s default environment.

Chapter 3. Customizing the User Experience

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the third chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

This chapter covers customizing the user experience for yourself and for your users. System administrators are often called upon to make minor changes to a user’s environment, or the environment for an entire system (the latter is known as a global change). As long as any requested alterations and enhancements don’t compromise system security or violate corporate policy, then there’s really no harm in making a few changes that accommodate the user’s needs and workflows. Our duty as sysadmins is, after all, to the company first and then to the user. The user is your customer.

Customizing the user environment, on a global basis, changes the environment for everyone on the system. However, you or the user can override some global parameters. You actually made such an override in Chapter 2 when you added the new umask to your user account. By setting your personal umask preference after the global one was set, you superseded the one set by the system. It’s a common practice for users to customize their own environments that they have control over.

This chapter covers customizing your environment and your user’s environments through the editing of key files located in each user’s home directory. As a system administrator, you’ll also explore the “master” versions of these environment files that can be changed or added to so that you can create a specific experience for your users.

Altering Home Directory Options

In every user’s home directory, there are a few hidden files that control most of a user’s environment. Since many Linux users use Bash, the Bourne Again Shell, that is the focus of the user environment discussion in this chapter. (Other shells such as ash, zsh, csh, and ksh are also available to you and the hidden, user-editable files are similar in name, function, and structure.)

Because some users don’t have adequate skills to make the changes they need, you might have to make them on the user’s behalf. The relevant files are listed below:

- .bash_profile

- .bashrc

- .bash_logout

- .bash_history

You won’t have to make changes to all of them. For example, the .bash_history file doesn’t require any changes. It’s a log of issued commands and there are no user configurable items in it. When you login to a Linux system, the .bashrc file executes first and then the .bash_profile executes. The .bash_logout file executes upon logout.

WARNING

Be careful about which (and how many) programs, scripts, and messages you place in your startup files because if they’re corrupt, damaged, or half-open, you might find that your login either is delayed or that you’re completely unable to login and will have to be rescued by another sysadmin. Pass this warning on to your users too.

When you login to a Linux system interactively, there is a group of files that automatically execute to build your user environment, as described in the following sections.

Login Versus Non-login Shells

You’ll read and hear of two types of interactive shells: login and non-login. An interactive login shell is one in which you SSH to or directly login to a Linux system supplying a username and password or SSH key. An interactive non-login shell is one where you call a subshell from your command line:

$ bash

$ echo $SHLVL

2

The $SHLVL is a variable that tracks your shell level. When you first login to an interactive shell either with a username and password or a key, your $SHLVL is 1. Subshells that you call after the first one increment the SHLVL variable. A subshell called in this way is an interactive, non-login shell because it is interactive but doesn’t require a new login.

/etc/bashrc

When you login interactively to a Linux system, /etc/bashrc is the first personalization file to execute. The /etc/bashrc file also executes on interactive non-login shells. The /etc/bashrc file is a global personalization file that provides your login shell (bash) with functions and aliases. This file should be left as is--meaning that even as a system administrator, you shouldn’t edit it.

The following information and warning are reprinted from the /etc/bashrc file. You’ll see these warnings if you attempt to edit the file.

# System wide functions and aliases

# Environment stuff goes in /etc/profile

# It's NOT a good idea to change this file unless you know what you

# are doing. It's much better to create a custom.sh shell script in

# /etc/profile.d/ to make custom changes to your environment, as this

# will prevent the need for merging in future updates.

The /etc/bashrc file is analogous to the .bashrc file in your home directory. If you need to change functions and aliases, change them there.

/etc/profile

The /etc/profile file is a system-wide startup file. It provides generic variables, paths, and other settings to all users. There is also a warning in this file, shown below, that states that you shouldn’t edit this file unless you know what you’re doing. Again, as stated in the warning, it’s better to edit the personalization files in the user’s home directory or create other global personalization files under /etc/profile.d. Information and warning from the /etc/profile file:

# System wide environment and startup programs, for login setup

# Functions and aliases go in /etc/bashrc

# It's NOT a good idea to change this file unless you know what you

# are doing. It's much better to create a custom.sh shell script in

# /etc/profile.d/ to make custom changes to your environment, as this

# will prevent the need for merging in future updates.

However, these global settings can be overwritten by the individual personalization startup files located in the home directory. It is the second environment personalization file that executes when you login to a Linux system. Strangely, it also calls the /etc/bashrc file so /etc/bashrc is executed twice.

The /etc/profile file is analogous to the .bash_profile file in your home directory. Change any environment settings there.

.bashrc

The .bashrc file is a hidden file in your home directory. It’s hidden because you don’t want direct access to it either when renaming or deleting files with a wildcard. You have full ownership of the file and you can edit it at will. This file is for setting and including any functions you might need, and setting aliases for commands. After the global personalization files execute, the .bashrc file executes. The .bashrc file, like its global analog, executes in interactive non-login shells. The following is a listing of an unchanged .bashrc file. You should augment or change your PATH in this file so that your shell behaves the same for login and non-login instances.

# .bashrc

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

# User specific environment

if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]

then

PATH="$HOME/.local/bin:$HOME/bin:$PATH"

fi

export PATH

# Uncomment the following line if you don't like systemctl's auto-paging feature:

# export SYSTEMD_PAGER=

# User specific aliases and functions

Aliases are useful shortcuts to commands and their options. For example, if you want a “chatty” version of common commands, create the following aliases in your .bashrc file:

alias rm='rm -i'

alias cp='cp -i'

alias mv='mv -i'

The -i option means interactive and asks you to verify before the command completes its action. When you issue the new aliased rm command, you get a verification message. This option is especially handy for destructive commands such as rm because Linux isn’t “chatty” and provides you with no feedback when you remove a file, for example. Using the -i option provides you with a more Windows command-like experience by prompting you to confirm your actions.

$ rm file4.txt

rm: remove regular empty file 'file4.txt'?

Aliases are handy if you use the same commands and options many times. For example, on every system that I use, I create a few default aliases for myself. My most useful one is for long lists:

alias ll=’ls -l’

You can also create aliases on a per session basis. This means that if you login to a system, you can simply issue the alias command at the command line without saving it to a file and it remains in effect until you logoff.

TIP

Many sources recommend placing all customizations in the .bashrc file because it’s executed for interactive login and non-login shells. Making your shell customizations in the .bashrc file guarantees shell consistency.

.bash_profile

The last file to execute on an interactive login is the .bash_profile file in your home directory.

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

You may place customizations in this file that will run in interactive login shells but they will not be available for interactive, non-login shells.

.bash_logout

The .bash_logout file executes upon exiting the shell. This file is optional. It only exists to provide users with the capability to clean up temporary files on exit. One could also use it to log time using the shell or to send a message upon logout.

In the next section, you learn the origins of these shell personalization files and how you can include your own for each user.

The /etc/skel Directory

The /etc/skel directory is a special directory for holding files that you want every user to receive in their home directories as you create their user accounts. The files that you create there don’t have to be hidden although the default ones are. These files are copied to the user’s home directory during account creation. These are the master copies of the environment personalization files. The list of /etc/skel default files on Red Hat Enterprise Linux-based systems is shown below.

# ls -la /etc/skel

total 28

drwxr-xr-x.   2 root root   76 Jul  4 12:45 .

drwxr-xr-x. 144 root root 8192 Jul  4 11:07 ..

-rw-r--r--.   1 root root   18 Apr 21 10:04 .bash_logout

-rw-r--r--.   1 root root  141 Apr 21 10:04 .bash_profile

-rw-r--r--.   1 root root  376 Apr 21 10:04 .bashrc

-rw-r--r--.   1 root root  658 Mar  3  2020 .zshrc

If you create a file in the /etc/skel directory, it’s copied to each user’s home directory during new account creation. Existing users won’t receive files placed in /etc/skel after you have created their accounts. You’ll have to manually copy those to each user’s home directory and change the permissions so that the user has full control of them.

NOTE

Linux has a local help system known as manual pages or man pages for short. If you need help for a command, configuration, or system setting, enter man and the keyword to see if the documentation exists for it. For example, for help with the ls command, you’d use: $ man ls. You can move through the man pages using vi (Vim) navigation commands.

Customizing the Shell Prompt

Generally, users accept the default prompt presented by the system, which is the standard. Generically, it looks like: [username@hostname pwd]$ or specifically in my case: [khess@server1 ~]$. The tilde (~) represents the user’s home directory. For example, in the previous sections discussing personalization scripts, they are often represented as: ~/.bashrc and ~/.bash_profile to illustrate that these files are located in the user’s home directories.

To set a custom prompt, there is a list of escaped characters that represent locations, username, time, carriage returns, etc. These are shown below in the listing from the Bash man page. For example, the default prompt is given by the following code for the prompt environment variable (PS1): PS1="[\u@\h \W]\\$ “. This default prompt is set by the system in the /etc/bashrc file. You may override it with an entry in your own .bashrc file.

Bash allows these prompt strings to be customized by inserting a number of backslash-escaped special characters that are decoded as follows:

\a

an ASCII bell character (07)

\d

the date in “Weekday Month Date” format (e.g., “Tue May 26”)

\D{format}

the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation. The braces are required.

\e

an ASCII escape character (033)

\h

the hostname up to the first `.’

\H

the hostname

\j

the number of jobs currently managed by the shell

\l

the basename of the shell’s terminal device name

\n

newline

\r

carriage return

\s

the name of the shell, the basename of $0 (the portion following the final slash)

\t

the current time in 24-hour HH:MM:SS format

\T

the current time in 12-hour HH:MM:SS format

\@

the current time in 12-hour am/pm format

\A

the current time in 24-hour HH:MM format

\u

the username of the current user

\v

the version of bash (e.g., 2.00)

\V

the release of bash, version + patch level (e.g., 2.00.0)

\w

the current working directory, with $HOME abbreviated with a tilde (uses the value of the PROMPT_DIRTRIM variable)

\W

the basename of the current working directory, with $HOME abbreviated with a tilde

\!

the history number of this command

\#

the command number of this command

\$

if the effective UID is 0, a #, otherwise a $

\nnn

the character corresponding to the octal number nnn

\\

a backslash

\[

begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt

\]

end a sequence of non-printing characters

The default prompt is pretty good but some users and sysadmins prefer something a little different, so they’re free to change it. Some clever users have devised codes for colorful and artistic prompts such as this Christmas themed prompt:

PS1="🎅\[\e[33;41m\][\[\e[m\]\[\e[32m\]\u\[\e[m\]\[\e[36m\]@\[\e[m\]\[\e[34m\]\h\[\e[m\]\[\e[33;41m\]]\[\e[m\]🎄 "

Search online for “fun Linux prompts” and enjoy yourself. After you’ve played a bit with that, you can logoff/login to reset your prompt or enter the following code to return to the default: PS1="[\u@\h \W]\\$ “.

Summary

In this chapter, you learned how to edit user environments through personalization files and the default location and configuration for adding more files to new accounts. You also learned their locations and the order they’re loaded and which ones you should edit for a particular effect. Finally, you had a brief overview of the shell prompt and how to alter it.

In chapter four, you’ll get an overview of user management from user account creation to managing users through groups to how to grant access to resources.

Chapter 4. Managing Users

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the fourth chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

As a system administrator, a significant portion of your time will be spent on managing users. And then there’s tending to tickets that require troubleshooting user problems that aren’t account or permissions related such as connectivity problems, broken applications, data corruption, training issues, security issues, and user-caused problems.

Managing users covers the following tasks:

- Creating users accounts

- Modifying user accounts

- Removing user accounts

- Granting access to files and directories

- Restricting access to files and directories

- Enforcing security policies

- Setting permissions on files and directories

Some user management tasks can be learned from this book, while others are purely experiential and on-the-job training for you. No two user environments are exactly alike and no two user experiences are exactly alike. In this chapter, you learn some methods of pre-emptive user management but there are still problems that occur. The techniques you learn in this chapter will get you started in being able to handle an array of user-related problems.

Creating User Accounts

Like most tasks in Linux, there’s more than one way to create user accounts. For this book, I stick with the mainstream methods of creating accounts of which there are two: adduser and useradd.

Adding Users with adduser

On some Linux distributions, adduser is a symbolic link to useradd.

lrwxrwxrwx. 1 root root 17 Oct 26 2020 /usr/sbin/adduser -> /usr/sbin/useradd

On other distributions, however, adduser is an interactive Perl script that steps you through adding a new user, and useradd is a separate utility with its standard switches and arguments.

Adding Users with useradd

In this section, I demonstrate how to create user accounts with the useradd command. This is the standard command line Linux method of adding new users to a system. The useradd command is very simple and all you really need to supply is a username as an argument.

# useradd jsmith

This creates the home directory, /home/jsmith, fills it with the default complement of hidden environment files, and places an entry into /etc/passwd. When I create a user account with useradd, I supply a single argument and bit of information (The user’s full name) that otherwise requires me to edit the /etc/passwd file.

# useradd -c “Jane Smith” jsmith

The -c option writes information you supply to it to the fifth field of the /etc/passwd file. If you wish to supply more information such as phone number, email address, or whatever you wish to include, use commas (,) to separate the information.

# useradd -c “Jane Smith,Room 26,212-555-1000,jsmith@domain.com” jsmith

The new user’s /etc/passwd entry:

jsmith:x:1007:1007:Jane Smith,Room 26,212-555-1000,jsmith@domain.com:/home/jsmith:/bin/bash

The individual fields in a user’s /etc/passwd entry are as follows (from left to right):

- Username

- /etc/shadow password field

- User ID

- Primary group ID

- The comment field

- Home directory

- Default shell

Passwords are no longer stored in the /etc/passwd file. The /etc/shadow field refers to the /etc/shadow file that contains each user’s encrypted password and is only readable by root. Note the /etc/shadow file’s permissions are 000.

----------. 1 root root 1547 Jul 17 10:55 /etc/shadow

Although Jane Smith’s user account is created, her home directory exists, and there’s an entry in the /etc/passwd file for the account, Jane can’t login to the system. Do you know why? It’s because the account has no password. As the sysadmin, you have to supply an initial password to Jane so that the user can login. Before you supply a password for the account, the /etc/shadow entry shows that there is no password:

jsmith:!!:18825:0:99999:7:::

Use the passwd command to supply a password to the account.

# passwd jsmith

Changing password for user jsmith.

New password:

Retype new password:

passwd: all authentication tokens updated successfully.

Now, you have to give the password to Jane before she can successfully login on the system.

Modifying User Accounts

There’s rarely such a thing as a static user account. Therefore, the usermod command exists to assist you in making those required changes without editing the /etc/passwd, a home directory, or configuration files of any kind. The usermod command is the catchall for changing all things user account related. The following is an abbreviated list of modifications that you can make with the usermod command:

- Add the user to a supplementary group

- Change the user’s comment field in /etc/passwd

- Change the user’s home directory

- Set an account expiry date

- Remove an expiry date

- Change a user’s login name (username)

- Lock/unlock a user’s account

- Move the contents of a user’s home directory

- Change a user’s login shell

- Change a user’s ID

Some of these options are more frequently used than others are. For example, it’s completely reasonable to change a user’s login shell, or lock and unlock an account, to set or remove an expiry date, and add a user to a supplementary group. It’s rare to change a user’s ID once it’s set during account creation or to relocate a user’s home directory.

In the following sections, I give you examples of the most commonly requested user account changes. If you need to alter other aspects of a user’s account, check the man page for details.

Adding a Supplementary Group

When you create a new user account, the system assigns the user a user ID (UID) and a primary group ID (GID). They can be the same sequential number but that isn’t always the case. For example, for the account created earlier for Jane Smith, the UID is 1007 and the GID is 1007.

jsmith:x:1007:1007:

Jane’s primary GID is 1007 but she might also work in an area of the company, such as IT, Engineering, or Application development that requires her to have access to a group-owned directory. For this exercise, Jane works in the Engineering department as an associate engineer. The Engineering department’s shared directory GID is 8020. Using usermod, here’s how to grant Jane access to that group’s shared directory.

# usermod -a -G 8020 jsmith

This adds Jane’s user account to the engineering group in the /etc/group file.

engineering:x:8020:bjones,kdoe,vkundra,adundee,jsmith

Now, Jane can access the Engineering group’s shared directory. Use the -a (append) and the -G (supplementary group) together. The reason is that, if you don’t use the -a option, then your user will be removed from any other supplementary groups and only added to the one you specify. For example, if you now want Jane to also have access to the finance department’s shared directory, you have to append her to that group. You either can use the GID number or the name of the group when adding a user.

# usermod -a -G finance jsmith

A user can be a member of several other groups as well. For example a user might be in the finance department (GID=8342), but also require access to human resources (GID=8901) information. You also can add a user to more than one group at once.

# usermod -a -G 8342,8901 jsmith

This command adds Jane Smith to the finance and to the human resources group with a single command.

Changing the User Comments Field

Changing the user comments (GECOS) field is a common task. You can edit the /etc/passwd file directly although it comes with significant risk to do so. The rule of thumb is that if there’s a tool to perform some action or task, you should use it rather than directly editing configuration files. You can easily change the GECOS field using the usermod command and the -c option.

Let’s say the company has recently hired a second person named Jane Smith, so you need to make a distinction between them by adding a middle initial to the first Jane Smith’s GECOS field.

# usermod -c “Jane R Smith” jsmith

This command replaces Jane Smith with Jane R Smith.

The -c option tells the usermod command that you’re editing the “comments” field. You can also change this information using the chfn command.

# chfn -f “Janie Smith” jsmith

The chfn command changes your finger information. Finger is an old daemon that ran on early Unix systems and some Linux systems that supplied information about users. Almost no one uses it these days because of security issues but the information is still referred to as finger information. The -f option changes the user’s full name field for the specified account. There are other options for office (-o), office phone (-p), and home phone (-h). Generally, only the user’s full name or the service’s name and purpose are used for the GECOS field.

Setting an Expiration (Expiry) Date on an Account

If a user gives notice at a company, is preparing to leave a group, or goes on parental leave, sysadmins might decide to disable an account, for security reasons, until the person returns or prior to removing the user account from a system.

# usermod -e 2021-07-23 rsmith

Rob Smith’s account will be disabled (expired) on the specified date in the format: YYYY-MM-DD. The -e option sets the account for expiration.

Changing a User’s Login Shell

The default Linux shell is Bash, but some users don’t like using it, so they request that their default shell be changed to one of the other many available shell options. There are three methods of changing a user’s default shell: usermod, chsh, and directly editing /etc/passwd. Direct editing of the /etc/passwd file is not recommended.

The usermod command method uses the -s option, the new shell, and the username to make the change.

# usermod -s /bin/sh jsmith

The updated /etc/passwd file:

jsmith:x:1002:1002:Janie Smith:/home/jsmith:/bin/sh

Only a user with root privileges may edit the /etc/passwd file or use the usermod command. However, any user may change their own shell with the chfn command.

$ chsh -s /bin/zsh

Changing shell for jsmith.

Password:

Shell changed.

The resulting /etc/passwd entry:

jsmith:x:1002:1002:Janie Smith:/home/jsmith:/bin/zsh

For other changes, consult the man page for usermod.

NOTE

At the end of any man page, you’ll find a list of related alternative commands, links to external documentation, and configuration files referenced in the SEE ALSO section. These are handy to explore and might prove more efficient for making some changes. The following is the SEE ALSO section excerpted from the usermod man page:

SEE ALSO

chfn(1), chsh(1), passwd(1), crypt(3), gpasswd(8), groupadd(8), groupdel(8), groupmod(8), login.defs(5), subgid(5), subuid(5), useradd(8), userdel(8).

The next section provides some context around why UIDs and GIDs begin numbering at 1000.

User and Group ID Numbering Conventions

User account UID and GID numbers typically begin at 1000 and increment up by one for each new account. There are some guidelines associated with creating and maintaining user accounts on Linux systems. These aren’t hard and fast rules but they’re generally followed on most corporate systems. The UID and GID for the root user are always 0. No other user on the system has these user and group IDs. System and service accounts aren’t human user accounts and typically don’t have an interactive shell associated with them. These accounts are given UIDs and GIDs in the range from 1-999. Standard human users’ account UIDs and GIDs begin at 1000. These separations make system housekeeping much easier than randomly assigning UIDs and GIDs to user accounts.

UID

GID

Description

0

0

root

1-999

1-999

System/Service accounts

1000+

1000+

User accounts

This convention is optional on your systems but it is a best practice to use it, and you’ll find it in use in almost every industry. You’ve learned to create and modify user accounts, now you’ll find out how to remove user accounts in the next section.

Removing User Accounts

Fortunately, system administrators and developers who name commands name them in such a way that they’re easy to remember and their names describe their functions. The useradd command is one such example. To remove a user account from a system, you use the userdel command. And the userdel command is just as easy to use as useradd is.

To remove a user account from your system, issue the userdel command and supply the username for the account.

# userdel jsmith

This command removes the user’s entry from /etc/passwd and from /etc/shadow but leaves the user’s home directory (/home/bsmith) intact. Why do you think that’s a good option? Sysadmins often leave a user’s home directory intact after a user has separated from a company or has changed jobs within the company but no longer requires access to a system. Retaining the user’s home directory ensures that only the root user can access any documents left by the user that might be important to the company.

If you make nightly backups of user’s home directories, then you don’t necessarily need to retain the user’s home directory. The following userdel command removes the user’s home directory and all files inside it.

# userdel -r jsmith

WARNING

Destructive Linux commands, such as userdel and rm, are irreversible and can’t be undone once executed. Always be sure that you have the correct user account before pressing the ENTER key.

When it’s time to change passwords, you need to know how to force users to do it. Our next section shows you how.

Forcing Password Changes

It’s a matter of trust that the user will change their password when you provide them with an initial password. Sysadmins who regularly audit their user’s passwords realize that this “honor system” level of trust doesn’t work 100 percent of the time. You can easily audit a user’s account settings using the chage command. The -l option lists the current settings for the specified user account.

# chage -l rsmith

Last password change : Jul 17, 2021

Password expires : never

Password inactive : never

Account expires : never

Minimum number of days between password change         : 0

Maximum number of days between password change         : 99999

Number of days of warning before password expires : 7

As you can see, this account’s password never expires, which is a security violation that needs to be fixed. In addition to a regular forced change, you should also set a minimum change period. For example below, I set the rsmith account to force a password change every 90 days (-M 90) with a minimum number of days between password changes to one (-m 1). Setting a minimum number of days ensures that users don’t change their passwords ten times (or whatever the number of the system’s remembered passwords is set to) to reset it to their original password, which amounts to no net password change.

# chage -m 1 -M 90 rsmith

# chage -l rsmith

Last password change : Jul 17, 2021

Password expires : Oct 15, 2021

Password inactive : never

Account expires : never

Minimum number of days between password change         : 1

Maximum number of days between password change         : 90

Number of days of warning before password expires : 7

The expiration date set by the system is 90 days after the last password change. If the last password change is in the past, the user will be required to change their password on the next login.

NOTE

From the chage man page: The chage command changes the number of days between password changes and the date of the last password change. This information is used by the system to determine when a user must change their password.

Passwords are not the best form of authentication because they can be guessed, cracked, or read in plain text if users write them down. Because passwords are a weak form of authentication, care must be taken to ensure that passwords are changed often and not reused.

Handling Service Accounts

Nothing stirs up controversy like the mere mention of service accounts. I’m not sure what all the controversy is really about because every Linux system has more than 30 service accounts.

An example of a service account is one such as the nobody account, which is the Kernel Overflow User account.

nobody:x:65534:65534:Kernel Overflow User:/:/sbin/nologin

Generally, you can spot a service account because in the /etc/passwd file, you see that the user account has no assigned shell. Service accounts have /sbin/nologin where the user’s shell should be. This means that the service accounts have no interactive shell and no passwords to use. That doesn’t mean that their passwords are blank or null, they simply don’t exist. In other words, if a user account has /sbin/login as their shell, they can’t login on the system with any password. And no user, not even the root user, can su or sudo to those accounts.

# su - nobody

This account is currently not available.

Because service accounts have no interactive shell and no method of switching to one via su or sudo (even by the root user), there’s no security violation associated with having service accounts on a system. The controversy comes from the fact that some system administrators don’t know that service accounts don’t usually have interactive shells or passwords. It’s possible that there are services that require an interactive shell account for their service to function. For those services, extreme scrutiny and other security measures should be in place to thwart potential clandestine logins on those accounts.

Managing Groups Rather Than Users

Sysadmins agree that managing groups is much easier and more efficient than managing users. Group management allows you to:

- Manage permissions on assets such as folders and files

- Manage permissions according to job function

- Change permissions for a large number of users rather than for each user individually

- Easily add users to and remove users from a group’s shared folders and files

- Restrict permissions to sensitive folders and files

It’s very difficult to manage permissions for individual users because if those permissions need to change, you have to trace permissions for that user on every system. Managing permissions for groups allows sysadmins to manage fine-grained user access on a more global level.

For example, if you have a user who works in the Human Resources (HR) department and then moves to the Finance department, it’s easy to remove that user from the hr group and then add them to the finance group. The user immediately has access to all shared files and folders that other finance group members do. And the user no longer has access to HR files and folders.

You learned earlier in this chapter how to add users to supplementary groups. You should practice adding directories to the system, adding a group account, setting that particular group ownership to the directory, and then adding users to the group. You can then su - <username> becoming that user and test your permissions settings.

The following is one possible scenario for you to work through to learn group management. This example assumes you already have a finance group and users assigned to it. Notice that during this example the root user exits and returns to a regular user account that is part of the finance group. And users can change group ownership and permissions of files they own.

# mkdir /opt/finance

# chgrp finance /opt/finance

# ls -la /opt

total 0

drwxr-xr-x.  3 root root     21 Aug 11 21:56 .

dr-xr-xr-x. 18 root root    239 Aug 11 21:08 ..

drwxr-xr-x.  2 root finance   6 Aug 11 21:56 finance

# chmod 770 /opt/finance

# ls -la /opt

total 0

drwxr-xr-x.  3 root root     21 Aug 11 21:56 .

dr-xr-xr-x. 18 root root    239 Aug 11 21:08 ..

drwxrwx---.  2 root finance   6 Aug 11 21:56 finance

# exit

logout

$ cd /opt/finance

$ touch budget.txt

$ ls -la

total 0

drwxrwx---. 2 root  finance 24 Aug 11 21:58 .

drwxr-xr-x. 3 root  root    21 Aug 11 21:56 ..

-rw-rw-r--. 1 khess khess    0 Aug 11 21:58 budget.txt

$ chgrp finance budget.txt

$ ls -la

total 0

drwxrwx---. 2 root  finance 24 Aug 11 21:58 .

drwxr-xr-x. 3 root  root    21 Aug 11 21:56 ..

-rw-rw-r--. 1 khess finance  0 Aug 11 21:58 budget.txt

$ chmod 660 budget.txt

$ ls -la

total 0

drwxrwx---. 2 root  finance 24 Aug 11 21:58 .

drwxr-xr-x. 3 root  root    21 Aug 11 21:56 ..

-rw-rw----. 1 khess finance  0 Aug 11 21:58 budget.txt

If you understand everything that’s going on in this example, then you’re ready to move on to the next chapter. If you haven’t yet mastered these concepts, work through the chapter examples again and return to this exercise. Remember that creating users, groups, directories, and changing permissions are daily tasks for sysadmins and practicing these skills is the only way to acquire them and become comfortable using them.

Summary

This chapter covered a lot of topics and concepts that are probably unfamiliar to you unless you have some previous experience with Linux systems. In this chapter, you learned how to create, remove, and change user accounts. You also learned how to set up service accounts and had a brief overview of managing groups. In the next chapter, you learn about Linux networking from the basics of why networks are important to more complex concepts such as network troubleshooting.

Chapter 5. Connecting to a Network

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the fifth chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

A standalone Linux system is still powerful, but the scope of what the system is ultimately capable of is limited. Only a single operator or administrator can use a standalone system. While it can still run multiple workloads, its limited visibility to other users and computers severely weakens the whole purpose of a Linux system, which is to host multiple users and their tasks, scripts, documents, applications, and data.

Linux is a multi-tasking, multi-user operating system. One of its most outstanding values is being networked with other computers to allow multiple users to run various workloads on it simultaneously. Connecting a Linux system to a network enables it to become part of a local network, a grid, a cloud, or the global internet.

Plugging into a Network

There’s no great skill required to plug a server system into an existing network. These days as soon as a new system comes online, a dynamic host configuration protocol (DHCP) server provides an IP address, subnet mask, gateway, domain name system (DNS) servers, and some basic routing information to it. There are two “schools” of thought concerning DHCP and servers. The first claims that all server systems should have their own static IP addresses and the second asserts that all systems should use DHCP for IP address distribution and management. I’ve always configured servers, printers, and networking equipment with static IP addresses, near the lower end of the IP address pool. For me, it makes good organizational sense to do so rather than relying on DHCP and DNS services to maintain order for services that users rely on so heavily.

In the following sections, I describe the two IP addressing schemes—static and dynamic—and the advantages and disadvantages of each. The example network for each has the following IP addressing information:

- Network: 192.168.1.0/24

- Gateway: 192.168.1.254 (Static IP address)

- DNS: 192.168.1.1, 192.168.1.2 (Static IP addresses)

This example network is for learning purposes and is only practical for the smallest of companies or groups of users. The reason that this scheme will only work for small companies is that there are only 251 usable IP addresses on this network. A user might have as many as five devices that consume IP addresses and it’s easy to see that it doesn’t take long to exhaust such a small pool.

Assigning a static IP address varies among Linux distributions. For these examples, I use Debian and CentOS distributions.

Static IP Addressing

A static IP address is a “hard-coded” IP address embedded into a configuration file. The IP address does not change unless it’s changed manually. Static IP addresses do not depend on or use DHCP services. When assigning static IP addresses to systems, you should decide on a reserved block of IP addresses that you use for servers, network equipment (switches, routers, WiFi access points), printers, and any IP-capable system that is not mobile. Workstations, laptops, tablets, phones, and other mobile devices should use DHCP.

For example, using the network parameters given above, I would reserve 192.168.1.1 through 192.168.1.25 for servers, network equipment, printers, and other stationary, IP-capable systems. You can exclude a range of IP addresses in your DHCP configuration so that these addresses are not part of the available IP address pool.

One of the advantages of using a static pool of IP addresses is that network services are stable. If you set up some sort of inventory service and monitoring on your network, which I highly recommend, having those statically addressed systems will save some headaches for you. You’ll be able to compare inventories over time and you can see what’s changed, measure growth, and plan for changes. Another advantage is that you don’t have to rely on DNS and DHCP services to find critical systems. A static IP address gives you the stability you need irrespective of any other service’s status.

Some network services require static IP addresses for the server systems on which they reside. Services such as FTP, web, VPN, database, Active Directory, and DNS either require static IP addresses or they’re recommended.

The primary disadvantage and argument against static IP addressing is management. The argument is that managing static IP addresses is labor-intensive and might cause conflicts elsewhere on the network.

I don’t see this as a problem for most sysadmins because equipment that you assign static IP addresses to lasts for years and their locations are stable. You don’t typically move a server to different locations within a company and you never move them offsite to use and then return them as you do laptops, tablets, and mobile phones. Having a reserved pool that’s excluded from your DHCP pool prevents any potential conflicts with other systems.

Dynamic IP Addressing

Using a DHCP service removed a lot of the IP address management that plagued system administrators in the days before DHCP came on the scene. Assigning static IP addresses and keeping up with them for more than a few systems becomes a management nightmare. When users only had desktop systems, static IP addresses made a little more sense. But these days with laptops, tablets, and mobile phones, a static IP address would likely mean that the user would only have network access on their corporate network, they would experience IP conflicts if they ever connected to another network, or they would have to know how to change their systems from static to dynamically assigned IP addresses each time they connected to a different network.

For example, if your laptop has a static IP address for your corporate network of 192.168.1.50 and you take that laptop to a remote location, such as your home, to work, then your home would have to use the same IP addressing scheme as your office. If it does, then your laptop might conflict with a dynamically assigned IP address for your television, someone’s mobile phone, or some other IP-capable gadget. You might be able to work around the issue at home, but when you take that same laptop to a hotel and attempt to connect to your corporate VPN, chances are very good that you wouldn’t be able to because of that statically assigned IP address. The hotel might use a 10.0.1.0 IP addressing scheme and therefore your laptop would never connect.

Device mobility is one great advantage of DHCP. The user never has to reconfigure anything no matter where the laptop connects to another network or the internet. The other advantage is that once you configure a DHCP pool of addresses, there’s really not much maintenance for you to do. You’ll have to determine which lease duration best suits your users. I’ve seen lease durations range from 24 hours to 30 days. I don’t have a particular preference but if you use a long lease duration (more than a few days), then you might have some occasional cleanup to do by removing duplicates, stale leases, and so on. DHCP is supposed to clean up after itself but doesn’t always.

Whether to use static IP addressing, DHCP or a mixture of the two is really up to your personal preference for IP address management. You can still use a pure DHCP scheme and reserve IP addresses by MAC address making them statically assigned. The only time there’s a problem is if you change network interface cards (NICs). You have to remember to go into your DHCP reservations list and enter the new MAC address.

TIP

If you make a change to an adapter, such as changing from DHCP to a static IP address, you need to restart the adapter for those changes to take place. Restart the adapter by issuing the commands:

$ sudo ifdown <adapter_name>

$ sudo ifup <adapter_name>

The adapter name varies by system but some examples are eth0 and enp0s3.

The next section moves into a discussion of the security-related repercussions of placing systems on the network and how you can apply best security practices to your systems.

Networking and Security

Two types of Linux systems can be considered secure: One powered off and the other powered on but not connected to a network. The powered-off system is safe from over-the-network attacks, as are non-networked systems. The powered-off system’s only security vulnerability is physical security. Someone who has physical access to the system could steal, dismantle, or damage it. Non-networked systems are, as stated above, still somewhat useful but have limited use beyond a single operator.

Once you network a system, you’ve exposed it to over-the-network attacks. Malicious actors continuously scan IP address ranges, searching for vulnerable systems to exploit. While many servers are indirectly exposed to the internet and inside a corporate or home router and firewall-protected network, those systems are also susceptible to attack. Once inside your network, a malicious actor performs automated scans of all connected systems, searching for vulnerabilities.

Additionally, creating user accounts on your system decreases security because of weak passwords, the potential for man-in-the-middle (MITM) attacks, and social engineering exploits that might compromise one’s credentials to a malicious actor. For these reasons, administrators must take care to only grant what’s required for users to work, enforce strong security policies for passwords, keys, or multi-factor authentication, regularly patch and update systems, and perform periodic security audits on all systems, network equipment, and devices that access corporate resources.

Preparing a System for Network Connectivity

When a system administrator provisions a new system and installs it into a rack, plugging in the network cables is standard practice. We often rely on others to vet the systems’ function, purpose, and security options. However, this is not always the case. New system installation is often automatically performed by provisioning a “standard build” from a prepared operating system image that could be months or years old. Using old images is a poor security practice. If the system is immediately connected to a network before it’s fully updated and secured, it’s vulnerable to attack and compromise before it begins its regular duties on the network.

The solution is to provision new systems on a private network where they can receive updates, patches, and secure configurations from an internal repository before being placed into a production network.

Pruning Your Systems

Pruning means removing any unnecessary services and daemons from your systems. There’s no need to create problems for yourself by running a production system with multiple services that no one uses and could leave your system vulnerable to attack. Only install what you specifically need to provide services to your users or to other systems.

At a minimum, you need to have an SSH daemon running on your systems so that you can login and manage them remotely. If you find that some users require a special service that’s only used occasionally, or that leaves a system in a less than secure state, either turn on the service when needed and turn it off when it’s no longer being used or place the service on a secure network that can only be accessed from a restricted number of systems.

The next section covers using secure service protocols and other techniques to secure your systems.

Securing Network Daemons

Deciding which network daemons to install and support is generally easy because you know its intended purpose each time you build a system. If it’s a web server, you know that you must install a web service such as Apache or NGINX. If the system lives its life as a database server, you must install MySQL, MariaDB, or some other database software onto the system. The issue is that for a system to be useful, it must expose the corresponding TCP ports for services it provides. These network daemons are vulnerable to attack and therefore must be protected.

There are multiple methods of securing network daemons and services, but the simplest is to install secure versions of the services you want to provide. For example, if your new system is a DNS server, then use DNSSEC. If you configure an LDAP server, use LDAPS. And always use HTTPS for web servers secured with a certificate. The following table shows a partial list of secure services:

Protocol

Port

Description

https

443/tcp

http protocol over TLS/SSL

https

443/udp

http protocol over TLS/SSL

ldaps

636/tcp

LDAP over SSL

ldaps

636/udp

LDAP over SSL

imaps

993/tcp

IMAP over SSL

imaps

993/udp

IMAP over SSL

pop3s

995/tcp

POP-3 over SSL

pop3s

995/udp

POP-3 over SSL

Using secure protocols and encryption doesn’t guarantee security but it’s better than using non-secure protocols with no encryption. Vulnerabilities frequently appear even when you use secure applications and protocols. Keeping your systems updated and patched helps prevent security breaches. Security patches are generally available before widespread damage can be done but not always, so you have to remain vigilant and give your full attention to maintaining security.

The Secure Shell (SSH) daemon

The most common network daemon available on almost every Linux system is the secure shell or SSH daemon. SSH provides a secure (encrypted) connection to a Linux system over the network. Although the SSH daemon (SSHD) has built-in security through its encrypted channel, its communications are still vulnerable to attack. There are multiple methods of securing the SSHD so that attacks are less fruitful for an attacker.

Limiting access to SSHD from specific hosts

There are two files that an administrator can use to limit access to any daemon: /etc/hosts.allow and /etc/hosts.deny. They require no service restart because they’re not configuration files, and the system checks them each time a client accesses a service. The /etc/hosts.allow file is the more important of the two because its settings override those in the /etc/hosts.deny file.

An /etc/hosts.allow file entry to enable SSH connectivity from a single IP address (192.168.1.50) is as follows:

sshd: 192.168.1.50

sshd: ALL: DENY

This entry will only accept SSH connections from 192.168.1.50 and will deny them from all other IP addresses.

If you find that setting such an entry in /etc/hosts.allow doesn’t work for you, then you need to check for tcp_wrapper integration in sshd using the following command:

$ sudo ldd /path/to/binary | grep libwrap

If you receive no response, your sshd wasn’t compiled with tcp_wrappers enabled and is likely deprecated for your distribution. This is, unfortunately, the case with many packaged sshd installations. The response you’re looking for is similar to the following:

$ sudo  ldd /usr/sbin/sshd |grep libwrap

libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007fc6c2ab2000)

You have three options if your sshd doesn’t have tcp_wrappers support:

- Use other methods to secure sshd (firewall rules, iptables, nftables).

- Compile openssh_server with tcp_wrappers enabled.

- Find and replace your current sshd with an openssh_server package that has tcp_wrappers enabled.

To give you options for firewalld, iptables, and nftables, consider the following commands that perform similar functions to adding entries into /etc/hosts.allow and /etc/hosts.deny.

Firewalld Rules:

First, delete the ssh service from firewalld’s rules:

$ sudo firewall-cmd --permanent --remove-service=ssh

Add a new zone rather than using the default zone:

$ sudo firewall-cmd --permanent --new-zone=SSH_zone

$ sudo  firewall-cmd --permanent --zone=SSH_zone --add-source=192.168.1.50

$ sudo firewall-cmd --permanent --zone=SSH_zone --add-service=ssh

You must reload the firewall to make the new configuration active:

$ sudo firewall-cmd --reload

If you use iptables, you can restrict ssh access to a single IP address with a single command:

$ sudo iptables -A INPUT -p tcp -s 192.168.1.50 --dport 22 -j ACCEPT

For netfilter (nft), you add a new rule:

$ sudo nft insert rule ip filter input ip saddr 192.168.1.50 tcp dport 22 accept

However, if you receive an error that reads:

Error: Could not process rule: No such file or directory.

either you must create a new table and chain for the rule, or use an existing chain. Use the following commands to create a new table and chain named input:

$ sudo nft add table ip filter # create table

$ sudo nft add chain ip filter input { type filter hook input priority 0\; } # create chain

$ sudo nft insert rule ip filter input ip saddr 192.168.1.50 tcp dport 22 accept

Note that chain names are case-sensitive. For example, if you had used the existing input chain, INPUT, you would not have received the error:

$ sudo nft insert rule ip filter INPUT ip saddr 192.168.1.50 tcp dport 22 accept

Restart nftables after you’ve made your changes.

The nftables system replaces iptables and combines functionality from iptables, ip6tables, arptables, and ebtables into a single utility. You can read more about nftables on the netfilter homepage.

$ sudo systemctl restart nftables

You now have multiple methods of limiting access to the ssh daemon from random hosts. If you don’t want to single out a specific ip address, you can isolate the target system by subnet using 192.168.1.0/24 in place of the individual ip address.

NOTE

Be aware that opening access from an entire subnet might still place your system in significant danger if an intruder infiltrates your network. Ideally, you should limit access from one or two hosts so that logs and monitoring systems are likelier to detect a security breach.

You can also configure the ssh daemon to limit access to certain users using the /etc/ssh/sshd_config file. The one user that you must prevent from using SSH is the root user. You learn how to prevent root ssh access in the next section.

Denying SSH access for the root user

You should, upon installation, deny SSH access to the root user. Some Linux systems deny root logins via SSH by default, while others allow it. The root user should never login via SSH to any system. A regular user should login via ssh and then become root or use the sudo command to perform tasks as the root user.

You’ll have to check your /etc/ssh/sshd_config for the line that reads:

PermitRootLogin yes

Change the “yes” to “no” and restart the sshd:

$ sudo systemctl restart sshd

The root user is unable to login via ssh. The root user may directly login to the console.

Using keys rather than passwords for authentication

Another method of making your system more secure is to remove password authentication, which is the least secure method of authentication. Using key files is much more secure and more efficient. You must make the following changes to the /etc/ssh/sshd_config file and restart sshd for the new configuration to take effect:

PasswordAuthentication yes

Change the “yes” to “no” and then look for the following two entries to ensure they’re uncommented and set as shown:

PubkeyAuthentication yes

AuthorizedKeysFile .ssh/authorized_keys

Restart sshd to enable the new settings:

$ sudo systemctl restart sshd

On the client-side (local system), users need to do the following to set up key pair authentication:

Create the public/private key pair. This example is for the user “tux”:

$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/Users/tux/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /Users/tux/.ssh/id_rsa.

Your public key has been saved in /Users/tux/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:NVweugZXvDitzl0JGypcWTJOww/F54vmHUbX7r4U2LQ tux@server1

The key's randomart image is:

+---[RSA 2048]----+

|         . +=    |

|         .B=+..  |

|        .o*@.+ ..|

|         +*o* * +|

|       .S.o+ B E |

|        o.o + * o|

|         + + + + |

|          o o o .|

|               oo|

+----[SHA256]-----+

Copy the generated keys to a remote host:

$ ssh-copy-id tux@192.168.0.99

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/tux/.ssh/id_rsa.pub"

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

tux@192.168.0.99's password:

Number of key(s) added:        1

Now try logging into the target system, with “ssh ‘tux@192.168.0.99’” and check to ensure that only the key(s) you wanted exist on the target system:

$ ssh tux@192.168.0.99

Last login: Sun Sep 26 13:48:19 2021 from 192.168.0.10

[tux@server1 ~]$

User tux has successfully logged into the remote host, server1 (192.168.0.99), using a secure key pair rather than a password.

Remote connectivity: Client to server

When connecting to a service running a secure protocol, your client software communicates on a secure channel with the service daemon. You don’t have to have users do anything special to negotiate a secure communications link between the client and the service daemon.

It’s just as important to keep client software up to date as it is for you, as an administrator, to update the software on servers. I suggest that you set up cron jobs on each user system to automatically download and install updates and configure each new system to receive regular updates with no user interaction required. Be sure to schedule any required reboots for nighttime or during times when the user’s system is idle.

Summary

In this chapter, you learned about selecting an IP addressing scheme for your network and some advantages and disadvantages of each. You also now know the security implications of connecting systems to your network. You should understand the dangers and how to, as much as is possible, prevent security breaches by implementing some good security practices such as using secure protocols, turning off unnecessary services and daemons, and keeping systems patched and updated.

In the next chapter, you learn how to install software via a package manager, keep your system updated, and install software from source code.

Chapter 6. Installing and Uninstalling Software

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the sixth chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

Installing and uninstalling software are basic system administrator tasks. You might not perform them on a daily basis but these are regular tasks for you and your team to complete. Most often, you’ll install updates, which can be automated. Any new software you install should be accompanied by a business justification, a change control record, and a written understanding of security implications, if any from the requesting party. Installing software with known vulnerabilities is an easy pathway for malicious actors to compromise your systems.

Uninstalling software also requires a change control record because of the potential danger of removing a package, directory, or library required by some other critical system or service function.

There are three methods of installing software: From repositories, from individual packages, and from source code. This chapter covers all three methods. There are two standard methods of uninstalling software: Using the package management tool and, in the case of compiled software, using an uninstall process. A third, non-standard uninstall method is to manually uninstall software by removing directories, libraries, and binaries. Manual removal of software is a tedious task that only senior-level sysadmins should perform. Each section teaches you first how to install software by a particular method and then how to uninstall software by that same method.

NOTE

All demonstrations and examples in this chapter use CentOS 8.3 (server1) and Ubuntu Server 20.04 LTS (server2). I perform all tasks on server1 first and then on server2 noting any differences between the two systems. The software package used in the examples is lynx, which is a lightweight text-based browser.

Updating Your System

I’ve mentioned keeping your system updated multiple times in this book so far. It’s an important task to remember. It should be one of your top priorities. Updates are a part of standard maintenance. Many system administrators apply updates on a weekly basis, which is a good practice. But don’t hesitate to apply patches, updates, and upgrades as needed to mitigate vulnerabilities. Security is your top priority. The following two sections illustrate how to apply updates to your systems.

Applying Red Hat Enterprise Linux-Based System Updates

Red Hat Enterprise Linux-based systems use the YUM/DNF utility to maintain updates and software installation from repositories. From the official Red Hat documentation:

YUM/DNF (yum/dnf) is the primary tool for getting, installing, deleting, querying, and managing Red Hat Enterprise Linux RPM software packages from official Red Hat software repositories, as well as other third-party repositories. YUM/DNF is used in Red Hat Enterprise Linux versions 5 and later.

DNF is the latest incarnation of the utility and that’s why I’ve combined the two. DNF is YUM version 4 according to the documentation and is the tool to use from Red Hat Enterprise Linux version 8 onward.

$ sudo yum update

Last metadata expiration check: 1:52:37 ago on Sun 07 Nov 2021 07:14:51 PM CST.

Dependencies resolved.

==================================================================================

Package                                Architecture     Version                                     Repository           Size

==================================================================================

Installing:

kernel                                 x86_64           4.18.0-305.25.1.el8_4                       baseos              5.9 M

kernel-core                            x86_64           4.18.0-305.25.1.el8_4                       baseos               36 M

kernel-modules                         x86_64           4.18.0-305.25.1.el8_4                       baseos               28 M

Upgrading:

NetworkManager                         x86_64           1:1.30.0-13.el8_4                           baseos              2.6 M

NetworkManager-config-server           noarch           1:1.30.0-13.el8_4                           baseos              129 k

NetworkManager-libnm                   x86_64           1:1.30.0-13.el8_4                           baseos              1.8 M

NetworkManager-team                    x86_64           1:1.30.0-13.el8_4                           baseos              146 k

...truncated output…

Removing:

kernel                                 x86_64           4.18.0-193.28.1.el8_2                       @BaseOS               0

kernel-core                            x86_64           4.18.0-193.28.1.el8_2                       @BaseOS              60 M

kernel-modules                         x86_64           4.18.0-193.28.1.el8_2                       @BaseOS              20 M

Transaction Summary

==================================================================================

Install   21 Packages

Upgrade  256 Packages

Remove     3 Packages

Total download size: 404 M

Is this ok [y/N]:

Agree to the installation here to have your target packages upgraded to the latest stable versions. If you want to automate subsequent updates, use the -y option to answer ‘yes’ to any prompts:

$ sudo dnf -y update

This automatically accepts the installation and does not prompt you interactively. This is a great option for scripted solutions. The next section provides you with the equivalent update action on Debian-based systems.

Applying Debian-Based System Updates

You apply updates to Debian-based systems with an analogous command to the DNF one you used for Red Hat Enterprise Linux-based systems, using the Debain apt utility. You’ll also receive a similar response if your system requires updates.

$ sudo apt update

If your system doesn’t require updates, your response will look similar to the following for either system type:

Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease

Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]

Get:3 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]

Get:4 http://us.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]

Fetched 328 kB in 52s (6,311 B/s)

Reading package lists... Done

Building dependency tree

Reading state information... Done

All packages are up to date.

A system reboot is not required, even if the update includes a newer kernel version. There are no special update commands for patches, security updates, or application version upgrades; this single command takes care of all updates regardless of type or priority on both systems. Your system checks for updates from all configured repositories and applies them when available. It’s a good practice to check at least weekly for updates and apply them during a scheduled maintenance window, or as necessary for critical security updates.

The rest of this chapter focuses on installing software as on-demand service requests from your users, your management, or other sources.

Installing Software from Repositories

Installing software from a repository is by far the easiest method of installing software on a Linux system. The reason that it’s the easiest method is that the repository automatically meets your dependencies without you having to do anything except request an installation. For example, if you want to install the Apache web server on a system, there are several dependencies that you must satisfy before it will install. The repository contains all of your dependent packages, gathers, and installs them as they’re called for in support of your primary package.

Installing an Application

For the CentOS system:

$ sudo yum install lynx

Last metadata expiration check: 1:00:58 ago on Fri 05 Nov 2021 10:50:41 AM CDT.

No match for argument: lynx

Error: Unable to find a match: lynx

If you receive this error, try the following steps to resolve it:

$ sudo dnf install dnf-plugins-core

This installs three packages: dnf-plugins-core, python3-dnf-plugins-core, and yum-utils. Then, use the following command to enable the PowerTools repository where lynx resides:

$ sudo dnf config-manager --set-enabled powertools

Now, proceed with the installation of lynx and its dependencies:

$ sudo dnf install lynx

CentOS Linux 8 - PowerTools                                                                    2.4 kB/s | 2.4 MB     16:50

Last metadata expiration check: 0:14:39 ago on Fri 05 Nov 2021 12:09:08 PM CDT.

Dependencies resolved.

==================================================================================

Package                            Architecture             Version                        Repository                    Size

==================================================================================

Installing:

lynx                               x86_64                   2.8.9-2.el8                    powertools                   1.6 M

Installing dependencies:

centos-indexhtml                   noarch                   8.0-0.el8                      baseos                       246 k

Transaction Summary

==================================================================================

Install  2 Packages

Total download size: 1.8 M

Installed size: 6.5 M

Is this ok [y/N]:

Agree to the installation to continue:

Is this ok [y/N]: y

Downloading Packages:

(1/2): lynx-2.8.9-2.el8.x86_64.rpm                                                             208 kB/s | 1.6 MB     00:07

(2/2): centos-indexhtml-8.0-0.el8.noarch.rpm                                                    30 kB/s | 246 kB     00:08

----------------------------------------------------------------------------------

Total                                                                                          207 kB/s | 1.8 MB     00:09

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

Preparing        :                                                                                                       1/1

Installing       : centos-indexhtml-8.0-0.el8.noarch                                                                     1/2

Installing       : lynx-2.8.9-2.el8.x86_64                                                                               2/2

Running scriptlet: lynx-2.8.9-2.el8.x86_64                                                                               2/2

Verifying        : centos-indexhtml-8.0-0.el8.noarch                                                                     1/2

Verifying        : lynx-2.8.9-2.el8.x86_64                                                                               2/2

Installed:

centos-indexhtml-8.0-0.el8.noarch                                   lynx-2.8.9-2.el8.x86_64

Complete!

Installation of the lynx application and its dependency, centos-indexhtml, is complete.

For the Ubuntu system:

$ sudo apt install lynx

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following additional packages will be installed:

libidn11 lynx-common

The following NEW packages will be installed:

libidn11 lynx lynx-common

0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.

Need to get 1,586 kB of archives.

After this operation, 5,731 kB of additional disk space will be used.

Do you want to continue? [Y/n]

Note the two dependencies: libidn11 and lynx-common. The system installs the dependencies before the target package. Continuing from above by responding yes (y) to the prompt:

Do you want to continue? [Y/n] y

Get:1 http://us.archive.ubuntu.com/ubuntu focal/main amd64 libidn11 amd64 1.33-2.2ubuntu2 [46.2 kB]

Get:2 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 lynx-common all 2.9.0dev.5-1 [914 kB]

Get:3 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 lynx amd64 2.9.0dev.5-1 [626 kB]

Fetched 1,586 kB in 4min 29s (5,890 B/s)

Selecting previously unselected package libidn11:amd64.

(Reading database ... 107982 files and directories currently installed.)

Preparing to unpack .../libidn11_1.33-2.2ubuntu2_amd64.deb ...

Unpacking libidn11:amd64 (1.33-2.2ubuntu2) ...

Selecting previously unselected package lynx-common.

Preparing to unpack .../lynx-common_2.9.0dev.5-1_all.deb ...

Unpacking lynx-common (2.9.0dev.5-1) ...

Selecting previously unselected package lynx.

Preparing to unpack .../lynx_2.9.0dev.5-1_amd64.deb ...

Unpacking lynx (2.9.0dev.5-1) ...

Setting up libidn11:amd64 (1.33-2.2ubuntu2) ...

Setting up lynx-common (2.9.0dev.5-1) ...

Setting up lynx (2.9.0dev.5-1) ...

update-alternatives: using /usr/bin/lynx to provide /usr/bin/www-browser (www-browser) in auto mode

Processing triggers for libc-bin (2.31-0ubuntu9.2) ...

Processing triggers for man-db (2.9.1-1) ...

Processing triggers for mime-support (3.64ubuntu1) ...

The apt package manager installed the lynx package and its dependencies. That’s all there is to installing from a repository. Use the package installer and name the application packages that you want to install and the package manager takes care of everything for you. In the next section, you learn how to uninstall a software package.

Uninstalling an Application

This simple process uninstalls a target package and the second step removes any dependencies that are no longer used by other packages. The rpm (Red Hat Package Manager) installs, uninstalls, and queries packages. The -e option erases (removes) target packages from the system. As you’ll see below, when there are no errors from removing a package, the system gives no response. The autoremove step automatically removes unused dependencies. Your system might have more than one unused dependency. It’s safe to remove them.

$ sudo rpm -e lynx

$ sudo dnf autoremove

Last metadata expiration check: 0:31:32 ago on Fri 05 Nov 2021 01:52:28 PM CDT.

Dependencies resolved.

==================================================================================

Package                             Architecture              Version                        Repository                  Size

==================================================================================

Removing:

centos-indexhtml                    noarch                    8.0-0.el8                      @baseos                    505 k

Transaction Summary

==================================================================================

Remove  1 Package

Freed space: 505 k

Is this ok [y/N]: y

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

Preparing        :                                                                                                       1/1

Erasing          : centos-indexhtml-8.0-0.el8.noarch                                                                     1/1

Verifying        : centos-indexhtml-8.0-0.el8.noarch                                                                     1/1

Removed:

centos-indexhtml-8.0-0.el8.noarch

Complete!

This process removed the lynx package and its dependency, centos-indexhtml.

For the Ubuntu system, the process is a single step using Ubuntu’s apt with the purge option:

$ sudo apt purge lynx

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following packages were automatically installed and are no longer required:

libidn11 lynx-common

Use 'sudo apt autoremove' to remove them.

The following packages will be REMOVED:

lynx*

0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.

After this operation, 1,992 kB disk space will be freed.

Do you want to continue? [Y/n] y

(Reading database ... 108095 files and directories currently installed.)

Removing lynx (2.9.0dev.5-1) ...

(Reading database ... 108082 files and directories currently installed.)

Purging configuration files for lynx (2.9.0dev.5-1) ...

The purge option removes the lynx package but not the dependencies. In the dialog above, you must run sudo apt autoremove to erase those files from the system. If you use the remove option, then apt only removes the binaries from the system, which leaves configurations and other files intact. In the next section you learn how to install and remove individual packages using a system’s package manager.

Installing and Uninstalling Individual Software Packages

There are software packages from other sources such as vendor websites, GitHub, and SourceForge that you need that aren’t part of any repository. You must install these individual packages manually at the command line. Rather than using repository commands to install these packages, you use the local package manager utilities such as rpm and dpkg.

TIP

Be sure to read any documentation that accompanies your target package before attempting to install it. Check for dependencies, configurations, and any security warnings. You’ll need to satisfy dependencies before installing the target package.

Installing an Individual Software Package Manually

To simplify the process of locating a package, the examples in this section make use of the download-only option for packages from repositories. It doesn’t matter what the source is for your packages, only that you have them downloaded to your system and you’re installing them manually at the command line.

For the CentOS system:

$ sudo dnf --downloadonly install lynx

Last metadata expiration check: 0:20:15 ago on Sat 06 Nov 2021 08:11:31 AM CDT.

Dependencies resolved.

==================================================================================

Package                            Architecture             Version                        Repository                    Size

==================================================================================

Installing:

lynx                               x86_64                   2.8.9-2.el8                    powertools                   1.6 M

Installing dependencies:

centos-indexhtml                   noarch                   8.0-0.el8                      baseos                       246 k

Transaction Summary

==================================================================================

Install  2 Packages

Total download size: 1.8 M

Installed size: 6.5 M

DNF will only download packages for the transaction.

Is this ok [y/N]:y

Downloading Packages:

(1/2): lynx-2.8.9-2.el8.x86_64.rpm                                                             287 kB/s | 1.6 MB     00:05

(2/2): centos-indexhtml-8.0-0.el8.noarch.rpm                                                    37 kB/s | 246 kB     00:06

----------------------------------------------------------------------------------

Total                                                                                          252 kB/s | 1.8 MB     00:07

Complete!

The downloaded packages were saved in cache until the next successful transaction.

You can remove cached packages by executing 'dnf clean packages'.

When you download packages using this method they’re stored in a subdirectory of the /var/cache/dnf directory. The subdirectory where packages download to depends on the repository that the packages originate from. For example, packages can download to any of the following on my CentOS system:

appstream-a520ed22b0a8a736

AppStream-a520ed22b0a8a736

baseos-929b586ef1f72f69

BaseOS-929b586ef1f72f69

epel-6519ee669354a484

epel-modular-95d9a0c53e492cbd

extras-2770d521ba03e231

powertools-25a6a2b331e53e98

In this example, the lynx package downloaded to: /var/cache/dnf/powertools-25a6a2b331e53e98/packages and centos-indexhtml downloaded to /var/cache/dnf/baseos-929b586ef1f72f69/packages.

I’m going to attempt to install the lynx package first, which will fail because of its dependency on the centos-indexhtml package:

$ sudo rpm -i lynx-2.8.9-2.el8.x86_64.rpm

error: Failed dependencies:

redhat-indexhtml is needed by lynx-2.8.9-2.el8.x86_64

NOTE

Since CentOS is a Red Hat Enterprise Linux binary compatible distribution, our centos-indexhtml package is equivalent to the redhat-indexhtml package and will work because the package names are analogous.

Heeding the preceding error, install the centos-indexhtml package first and then proceed to install the lynx package:

$ sudo rpm -i centos-indexhtml-8.0-0.el8.noarch.rpm

The package installs without error.

$ sudo rpm -i lynx-2.8.9-2.el8.x86_64.rpm

The lynx package installed successfully. The rpm switch (-i) means install.

For Ubuntu systems, the manual download and install process proceeds as follows:

$ sudo apt install --download-only lynx

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following additional packages will be installed:

libidn11 lynx-common

The following NEW packages will be installed:

libidn11 lynx lynx-common

0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.

Need to get 960 kB/1,586 kB of archives.

After this operation, 5,731 kB of additional disk space will be used.

Do you want to continue? [Y/n]y

Get:1 http://us.archive.ubuntu.com/ubuntu focal/main amd64 libidn11 amd64 1.33-2.2ubuntu2 [46.2 kB]

Get:2 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 lynx-common all 2.9.0dev.5-1 [914 kB]

Get:2 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 lynx-common all 2.9.0dev.5-1 [914 kB]

Fetched 214 kB in 2min 24s (1,491 B/s)

Download complete and in download only mode

On Debian-based systems such as Ubuntu, files downloaded this way reside in /var/cache/apt/archives as .deb packages and you can install them from that location. Note that the apt utility dialog stated that the packages “will be installed” but the final message, “Download complete and in download only mode” means that the packages downloaded but didn’t install. In the following example, I attempt to install lynx and ignore the dependencies that downloaded with it.

$ sudo dpkg -i lynx_2.9.0dev.5-1_amd64.deb

Selecting previously unselected package lynx.

(Reading database ... 108026 files and directories currently installed.)

Preparing to unpack lynx_2.9.0dev.5-1_amd64.deb ...

Unpacking lynx (2.9.0dev.5-1) ...

dpkg: dependency problems prevent configuration of lynx:

lynx depends on libidn11 (>= 1.13); however:

Package libidn11 is not installed.

lynx depends on lynx-common; however:

Package lynx-common is not installed.

dpkg: error processing package lynx (--install):

dependency problems - leaving unconfigured

Errors were encountered while processing:

lynx

The system won’t allow you to install without the dependencies that downloaded into the same directory with lynx:

$ ls /var/cache/apt/archives

libidn11_1.33-2.2ubuntu2_amd64.deb  lynx_2.9.0dev.5-1_amd64.deb  lynx-common_2.9.0dev.5-1_all.deb

Install the dependencies first and then install lynx:

$ sudo dpkg -i libidn11_1.33-2.2ubuntu2_amd64.deb

Selecting previously unselected package libidn11:amd64.

(Reading database ... 108039 files and directories currently installed.)

Preparing to unpack libidn11_1.33-2.2ubuntu2_amd64.deb ...

Unpacking libidn11:amd64 (1.33-2.2ubuntu2) ...

Setting up libidn11:amd64 (1.33-2.2ubuntu2) ...

Processing triggers for libc-bin (2.31-0ubuntu9.2) …

$ sudo dpkg -i lynx-common_2.9.0dev.5-1_all.deb

Selecting previously unselected package lynx-common.

(Reading database ... 108044 files and directories currently installed.)

Preparing to unpack lynx-common_2.9.0dev.5-1_all.deb ...

Unpacking lynx-common (2.9.0dev.5-1) ...

Setting up lynx-common (2.9.0dev.5-1) ...

Processing triggers for mime-support (3.64ubuntu1) ...

Processing triggers for man-db (2.9.1-1) …

$ sudo dpkg -i lynx_2.9.0dev.5-1_amd64.deb

(Reading database ... 108136 files and directories currently installed.)

Preparing to unpack lynx_2.9.0dev.5-1_amd64.deb ...

Unpacking lynx (2.9.0dev.5-1) over (2.9.0dev.5-1) ...

Setting up lynx (2.9.0dev.5-1) ...

update-alternatives: using /usr/bin/lynx to provide /usr/bin/www-browser (www-browser) in auto mode

The -i switch for the dpkg command means install, just as it does for the rpm utility. Now to uninstall the same packages manually.

Uninstalling Individual Software Packages

To uninstall a manually installed package, you have to reverse the process. This means that you uninstall in the exact opposite order beginning with the last installed package first. If there are dependencies, the system will instruct you which ones they are.

On CentOS:

$ sudo rpm -e centos-indexhtml

error: Failed dependencies:

redhat-indexhtml is needed by (installed) lynx-2.8.9-2.el8.x86_64

$ sudo rpm -e lynx

$ sudo rpm -e centos-indexhtml

You have successfully uninstalled lynx and its dependency, centos-indexhtml. In other words, you cannot uninstall a dependency before you install a package because the installed package depends on that package to function correctly.

On Ubuntu:

You’ll notice that when you attempt an uninstall of a package that you manually installed, there are no warnings about dependencies. To install lynx on Ubuntu, you required three packages, lynx and its two dependencies: libidn11 and lynx-common.

Look at the differences when attempting to individually uninstall lynx and its dependencies:

$ sudo apt purge lynx

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following packages will be REMOVED:

lynx*

0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.

After this operation, 1,992 kB disk space will be freed.

Do you want to continue? [Y/n]

$ sudo apt purge lynx-common

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following packages will be REMOVED:

lynx* lynx-common*

0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.

After this operation, 5,481 kB disk space will be freed.

Do you want to continue? [Y/n]

$ sudo apt purge libidn11

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following packages will be REMOVED:

libidn11* lynx*

0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.

After this operation, 2,242 kB disk space will be freed.

Do you want to continue? [Y/n]

And, if you only remove lynx from the system, it leaves behind lynx_common and libidn11. Issuing the command sudo apt autoremove won’t remove the unused dependencies as it did when you installed from a repository. The following section describes how to find dependencies for a specific software package.

Finding Package Dependencies

It helps to find dependencies before you install a particular software package:

$ dnf deplist lynx

CentOS Linux 8 - AppStream                                                                      37 kB/s | 9.6 MB     04:28

CentOS Linux 8 - BaseOS                                                                         39 kB/s | 8.5 MB     03:44

CentOS Linux 8 - Extras                                                                        7.8 kB/s |  10 kB     00:01

CentOS Linux 8 - PowerTools                                                                    121 kB/s | 2.4 MB     00:20

Extra Packages for Enterprise Linux Modular 8 - x86_64                                          20 kB/s | 955 kB     00:48

Extra Packages for Enterprise Linux 8 - x86_64                                                  29 kB/s |  11 MB     06:08

package: lynx-2.8.9-2.el8.x86_64

dependency: libc.so.6(GLIBC_2.15)(64bit)

provider: glibc-2.28-151.el8.x86_64

dependency: libcrypto.so.1.1()(64bit)

provider: openssl-libs-1:1.1.1g-15.el8_3.x86_64

dependency: libcrypto.so.1.1(OPENSSL_1_1_0)(64bit)

provider: openssl-libs-1:1.1.1g-15.el8_3.x86_64

dependency: libdl.so.2()(64bit)

provider: glibc-2.28-151.el8.x86_64

dependency: libncursesw.so.6()(64bit)

provider: ncurses-libs-6.1-7.20180224.el8.x86_64

dependency: libssl.so.1.1()(64bit)

provider: openssl-libs-1:1.1.1g-15.el8_3.x86_64

dependency: libssl.so.1.1(OPENSSL_1_1_0)(64bit)

provider: openssl-libs-1:1.1.1g-15.el8_3.x86_64

dependency: libtinfo.so.6()(64bit)

provider: ncurses-libs-6.1-7.20180224.el8.x86_64

dependency: libz.so.1()(64bit)

provider: zlib-1.2.11-17.el8.x86_64

dependency: redhat-indexhtml

provider: centos-indexhtml-8.0-0.el8.noarch

dependency: rtld(GNU_HASH)

provider: glibc-2.28-151.el8.i686

provider: glibc-2.28-151.el8.x86_64

As you know from reading this chapter, the system already has most of the required dependencies installed. The required one, centos-indexhtml, doesn’t stand out in any particular way. The only way I know of to isolate any dependencies that your system requires is to attempt an installation of the target package. The following listing shows you the same query on the Ubuntu system:

$ sudo apt show lynx

Package: lynx

Version: 2.9.0dev.5-1

Priority: extra

Section: universe/web

Origin: Ubuntu

Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>

Original-Maintainer: Debian Lynx Packaging Team <pkg-lynx-maint@lists.alioth.debian.org>

Bugs: https://bugs.launchpad.net/ubuntu/+filebug

Installed-Size: 1,992 kB

Provides: news-reader, www-browser

Depends: libbsd0 (>= 0.0), libbz2-1.0, libc6 (>= 2.15), libgnutls30 (>= 3.6.12), libidn11 (>= 1.13), libncursesw6 (>= 6), libtinfo6 (>= 6), zlib1g (>= 1:1.1.4), lynx-common

Recommends: mime-support

Conflicts: lynx-ssl

Breaks: lynx-cur (<< 2.8.9dev8-2~), lynx-cur-wrapper (<< 2.8.8dev.8-2)

Replaces: lynx-cur (<< 2.8.9dev8-2~), lynx-cur-wrapper (<< 2.8.8dev.8-2)

Homepage: https://lynx.invisible-island.net/

Download-Size: 626 kB

APT-Manual-Installed: yes

APT-Sources: http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages

Description: classic non-graphical (text-mode) web browser

In continuous development since 1992, Lynx sets the standard for

text-mode web clients. It is fast and simple to use, with support for

browsing via FTP, Gopher, HTTP, HTTPS, NNTP, and the local file system.

If you’re a Linux purist and like to compile your software so that you have maximum control, the next section provides you with a demonstration of installing a software package from source code.

Installing Software from Source Code

There are system administrators who prefer to install all software from source code (source) because it is the most flexible method of software installation. Installing from source allows you to customize the installation for your specific needs. You can change installation paths, enable features, disable features, and make minute adjustments to every possible configuration option available for an application.

There are some downsides to installing from source. The major downside is that you have to satisfy dependencies for the software you’re installing. This can be frustrating, time-consuming, and tedious. I have personally chased dependencies to the point where I’ve actually forgotten what the original software was that I needed to install. Another downside is that you have to install a full complement of development tools onto your systems, which consumes considerable disk space. Another downside is that upgrading to a newer version of the software installed from source is just as difficult and time-consuming as installing the original version. And if a previous version isn’t fully overwritten or removed, you can experience version conflicts that are often quite difficult to resolve.

Satisfying Prerequisites: Building a Development Environment

Before you begin installing any application from source code, you’ll need to set up a development environment by installing a code compiler and supporting software. The easiest method of doing this is to install a group of software packages from your Linux vendor’s repositories.

For Red Hat Enterprise Linux-based systems, using the groupinstall option and identifying “Development Tools” as the target group to install is the best choice. Unfortunately, this group selection installs a lot of unnecessary and potentially non-secure packages for compiling source code at the command line, such as a list of graphical tools. For this reason, it’s often desirable to set up a specific system that’s dedicated to software development.

$ sudo dnf groupinstall "Development Tools"

Last metadata expiration check: 2:41:45 ago on Sun 07 Nov 2021 09:57:05 AM CST.

Dependencies resolved.

========================================================================

Package                                     Architecture  Version                                      Repository        Size

========================================================================

Upgrading:

automake                                    noarch        1.16.1-7.el8                                 appstream        713 k

binutils                                    x86_64        2.30-93.el8                                  baseos           5.8 M

cpp                                         x86_64        8.4.1-1.el8                                  appstream         10 M

elfutils-libelf                             x86_64        0.182-3.el8                                  baseos           216 k

elfutils-libs                               x86_64        0.182-3.el8                                  baseos           293 k

gcc                                         x86_64        8.4.1-1.el8                                  appstream         23 M

...output truncated

xorg-x11-font-utils                         x86_64        1:7.5-40.el8                                 appstream        103 k

xorg-x11-fonts-ISO8859-1-100dpi             noarch        7.5-19.el8                                   appstream        1.1 M

xorg-x11-server-utils                       x86_64        7.7-27.el8                                   appstream        198 k

zlib-devel                                  x86_64        1.2.11-17.el8                                baseos            58 k

zstd                                        x86_64        1.4.4-1.el8                                  appstream        393 k

Installing weak dependencies:

gcc-gdb-plugin                              x86_64        8.4.1-1.el8                                  appstream        117 k

kernel-devel                                x86_64        4.18.0-305.25.1.el8_4                        baseos            18 M

Enabling module streams:

javapackages-runtime                                      201801

Installing Groups:

Development Tools

Transaction Summary

========================================================================

Install  159 Packages

Upgrade   23 Packages

Total download size: 219 M

Is this ok [y/N]:

On Ubuntu systems, the equivalent installation uses a “group” named build-essential to install all the necessary development tools on your system:

$ sudo apt install build-essential

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following additional packages will be installed:

binutils binutils-common binutils-x86-64-linux-gnu cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9 gcc-9-base

libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan5 libatomic1 libbinutils libc-dev-bin

libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-9-dev libgomp1

libisl22 libitm1 liblsan0 libmpc3 libquadmath0 libstdc++-9-dev libtsan0 libubsan1 linux-libc-dev make manpages-dev

Suggested packages:

binutils-doc cpp-doc gcc-9-locales debian-keyring g++-multilib g++-9-multilib gcc-9-doc gcc-multilib autoconf automake

libtool flex bison gdb gcc-doc gcc-9-multilib glibc-doc bzr libstdc++-9-doc make-doc

The following NEW packages will be installed:

binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9

gcc-9-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan5 libatomic1 libbinutils

libc-dev-bin libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl libfakeroot libfile-fcntllock-perl

libgcc-9-dev libgomp1 libisl22 libitm1 liblsan0 libmpc3 libquadmath0 libstdc++-9-dev libtsan0 libubsan1 linux-libc-dev make

manpages-dev

0 upgraded, 41 newly installed, 0 to remove and 0 not upgraded.

Need to get 43.0 MB of archives.

After this operation, 189 MB of additional disk space will be used.

Do you want to continue? [Y/n]

Confirm the installation and continue. Installing groups of packages can take several minutes. Once your development environment is set up, you need to download the source code for lynx from https://invisible-mirror.net/archives/lynx/tarballs/lynx2.8.9rel.1.tar.gz. The instructions for installing from source are the same for any Linux distribution, however, I’m going to attempt this installation on both the CentOS and on the Ubuntu systems and note any differences and errors in the text. Any user can download, compile, and make the binaries but you must be root to install binaries to the system directories.

Download, Extract, Compile, and Install Your Software

Download your compressed source code using a method such as wget:

$ wget https://invisible-mirror.net/archives/lynx/tarballs/lynx2.8.9rel.1.tar.gz

Extract the source code from the compressed archive:

$ tar zxvf lynx2.8.9rel.1.tar.gz

Change directory into the lynx source tree created by the extracting process:

$ cd lynx2.8.9rel.1

Before running configure, take a few minutes to look for and read the README file that always exists in source code trees. This file has valuable instructions and information about the source code and installation instructions. This README file refers you to the INSTALLATION file that describes installation options. For demonstration purposes, I’m accepting all the defaults and simply running the configure script. If all dependencies are met, then the configuration checks will go without error, create the makefile, and then drop you back at your shell prompt. This process can take a few minutes to complete.

$ ./configure

Both configure scripts failed with the following error:

checking for screen type... curses

checking for specific curses-directory... no

checking for extra include directories... no

checking if we have identified curses headers... none

configure: error: No curses header-files found

When you encounter errors, the configure script stops but keeps its place so that as you satisfy dependencies, you may continue where you left off by running the configure script again. To satisfy the current dependency, I installed the ncurses-devel package on the CentOS system. The configure script completed successfully. For the Ubuntu system, I installed the lib32ncurses-dev package and the configure script completed successfully.

According to the INSTALLATION file, you now must run the make command to compile the sources. This process will take a few minutes to complete:

$ make

After satisfying the failed dependencies in the configure script, both compilations proceeded to successful completion. To install lynx to its proper location and set the correct permissions, run sudo make install:

$ sudo make install

/bin/sh -c "P=`echo lynx|sed 's,x,x,'`; \

if test -f /usr/local/bin/$P ; then \

mv -f /usr/local/bin/$P /usr/local/bin/$P.old; fi"

/usr/bin/install -c lynx /usr/local/bin/`echo lynx|sed 's,x,x,'`

/usr/bin/install -c -m 644 ./lynx.man /usr/local/share/man/man1/`echo lynx|sed 's,x,x,'`.1

** installing ./lynx.cfg as /usr/local/etc/lynx.cfg

** installing ./samples/lynx.lss as /usr/local/etc/lynx.lss

Use make install-help to install the help-files

Use make install-doc to install extra documentation files

At this point, most instructions advise you to run make clean to remove all of the object code and other temporary files from your system:

$ make clean

rm -f WWW/Library/*/*.[aoib]

rm -f WWW/Library/*/.created

cd ./WWW/Library/Implementation && make  DESTDIR="" CC="gcc" clean

make[1]: Entering directory '/home/khess/lynx2.8.9rel.1/WWW/Library/Implementation'

rm -f core *.core *.leaks *.[oi] *.bak tags TAGS

rm -f dtd_util

rm -f ./*.o

make[1]: Leaving directory '/home/khess/lynx2.8.9rel.1/WWW/Library/Implementation'

cd ./src && make  DESTDIR="" CC="gcc" clean

make[1]: Entering directory '/home/khess/lynx2.8.9rel.1/src'

rm -f lynx core *.core *.leaks *.i *.o *.bak tags TAGS test_*

cd chrtrans && make  DESTDIR="" CC="gcc" clean

make[2]: Entering directory '/home/khess/lynx2.8.9rel.1/src/chrtrans'

rm -f makeuctb *.o *uni.h *uni2.h *.i

make[2]: Leaving directory '/home/khess/lynx2.8.9rel.1/src/chrtrans'

make[1]: Leaving directory '/home/khess/lynx2.8.9rel.1/src'

rm -f *.b ./src/lynx *.leaks cfg_defs.h LYHelp.h lint.*

rm -f help_files.sed

rm -f core *.core

It’s a matter of preference to run this command. If you need to remove compiled software from your system, the next section steps you through the process of doing so.

Uninstalling a Source-Installed Software Package

If your source trees exist with your original makefile intact, you can uninstall a package quite easily but you must have the makefile to do so. If you don’t want to keep all of your source trees on your system for every compiled program, make a backup of the makefile such as copying it to a backup directory with the name makefile.lynx289r1 or similar. The makefile must be in your current directory when uninstalling:

$ sudo make uninstall

rm -f /usr/local/bin/`echo lynx|sed 's,x,x,'`

rm -f /usr/local/share/man/man1/`echo lynx|sed 's,x,x,'`.1

rm -f /usr/local/etc/lynx.cfg

rm -f /usr/local/etc/lynx.lss

/bin/sh -c 'if test -d "/usr/local/share/lynx_help" ; then \

WD=`cd "/usr/local/share/lynx_help" && pwd` ; \

TAIL=`basename "/usr/local/share/lynx_help"` ; \

HEAD=`echo "$WD"|sed -e "s,/${TAIL}$,,"` ; \

test "x$WD" != "x$HEAD" && rm -rf "/usr/local/share/lynx_help"; \

fi'

/bin/sh -c 'if test -d "/usr/local/share/lynx_doc" ; then \

WD=`cd "/usr/local/share/lynx_doc" && pwd` ; \

TAIL=`basename "/usr/local/share/lynx_doc"` ; \

HEAD=`echo "$WD"|sed -e "s,/${TAIL}$,,"` ; \

test "x$WD" != "x$HEAD" && rm -rf "/usr/local/share/lynx_doc"; \

fi'

/bin/sh -c 'if test -d "/usr/local/share/lynx_help" ; then \

WD=`cd "/usr/local/share/lynx_help" && pwd` ; \

TAIL=`basename "/usr/local/share/lynx_help"` ; \

HEAD=`echo "$WD"|sed -e "s,/'${TAIL}'$,,"` ; \

test "x$WD" != "x$HEAD" ; \

cd "/usr/local/share/lynx_help" && rm -f COPYING COPYHEADER ; \

fi'

If you don’t have your makefile or it isn’t in your current directory, you receive the following error:

$ sudo make uninstall

make: *** No rule to make target 'uninstall'.  Stop.

You can re-create the makefile if you extract the same version of the source tree that you previously used and can remember your configure options. Otherwise, you can use the above uninstall results to guide you in uninstalling manually.

Summary

This chapter stepped you through the process of installing software from different sources: repositories, local package files, and from source code. You’ll find that installing software is easy enough with a little practice. Please remember that the responsibility of keeping your systems running smoothly and securely rests with you. Just because installing software is quick and easy to do doesn’t mean that you should ignore the impact of it on system performance, security, and disk usage.

The next chapter teaches you how to add disk space to your systems. Adding space, mounting filesystems, creating an appropriate layout, and formatting with the correct filesystem type are some of the covered topics.

Chapter 7. Managing Storage

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the seventh chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

In this chapter, you learn how to add disk space to your systems beginning with adding a new disk drive to the system and making it available for use. You also explore the logical volume manager and how to manipulate logical volumes. You learn disk formatting, partitioning, and mounting as well as how to decide between standard mechanical disks and solid-state disks for your systems.

In the first section, I cover some general concepts related to disks, filesystems, volumes, partitions, directories, and filesystem mounting.

Administering Linux Storage

The price of disk space has decreased so much in recent years that space is no longer a high-value commodity. You can buy multi-terabyte disks for a few dollars per gigabyte. So system administrators rarely have to threaten to implement quotas or other arbitrary limits on disk space. Workstations and laptops often have as much space as servers, so space is no longer at a premium, and managing it is far less of a problem than it was just a few years ago. For example, many sysadmins now bypass old types of backup methods such as tape for faster and cheaper disk-to-disk backups.

But even though disk space is cheap and available, sysadmins still need to keep an eye on users’ disk usage. You don’t want individuals filling up shared disk spaces or home directories with their personal music, videos, or other large files because these files waste corporate-owned space and prolong backup times. In this section, we provide a high-level discussion of disk-related terminology and how Linux system administrators interpret those terms. The specifics of how to work with each of these items are covered later in the chapter.

Disks

Disks are the devices that we refer to as hard drives or hard disk drives (HDDs), but disks can also refer to solid-state drives (SSDs) and USB thumb drives. System administrators make entire disks available to Linux systems using internal connections, USB connectivity, or an over-the-network technology such as Ethernet or fiber optic cabling. Before accessing a disk on Linux systems, a system administrator must mount the disk on a directory. For example, to mount a new disk identified by the system as /dev/sdd, the sysadmin creates a new directory, such as /software, and mounts the entire disk on that directory or mount point.

$ sudo mount /dev/sdd1 /software

TIP

The entire disk device is /dev/sdd but initializing the disk requires at least one partition, so the entire disk’s partition is now /dev/sdd1. Details of this process appear later in this chapter.

Once the administrator prepares the disk by partitioning and establishing a filesystem on it, users may access space made available to them.

Filesystems

A filesystem is an organizational construct that allows for file storage and retrieval for an operating system. The Linux Documentation Project (tldp) defines a filesystem as follows:

A filesystem is the methods and data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk. The word is also used to refer to a partition or disk that is used to store the files or the type of the filesystem. Thus, one might say “I have two filesystems” meaning one has two partitions on which one stores files or that one is using the “extended filesystem,” meaning the type of the filesystem.

Contemporary Linux systems give administrators a broad choice of filesystems, although most sysadmins either stick with XFS or EXT4 when creating new partitions. There are many other filesystems available for specific needs and applications.

Mounting and mount points

Only the root user, or a user with sudo privileges, may mount a filesystem. Mounting a filesystem on a directory is roughly analogous to assigning a drive letter to a disk in Windows. Linux uses mount points rather than drive letters. In the Disks section example, a new disk, /dev/sdd was mounted on the directory, /software. The directory, /software, is the mount point.

Linux systems provide a generic mount point, /mnt, onto which you may temporarily mount disks. You shouldn’t use the /mnt directory for a permanent mount point because another system administrator might mount another filesystem over it hiding the original contents.

Mount points need not be directories off of the root directory. They can be subdirectories. For example, you could mount the /dev/sdd disk onto /opt/software.

$ sudo mount /dev/sdd1 /opt/software

TIP

A mount point (directory) must exist before mounting a disk or filesystem onto it.

There’s nothing special about a mount point directory. It’s the same as any other directory on the filesystem. Create a new directory, set its permissions, and mount the filesystem or disk onto it.

WARNING

Don’t mount filesystems or disks onto existing system directories such as /tmp, /var, /usr, /etc, and so on. Doing so will cause erratic system behavior and possibly catastrophic failure.

To permanently mount a disk or filesystem, you must create an entry in the /etc/fstab file that includes the new disk or filesystem and the mount point. For example, to add the /dev/sdd1 on /opt/software, the /etc/fstab entry looks like the following:

UUID=324ddbc2-353b-4221-a80e-49ec356678dc /opt/software xfs defaults 0 0

If you don’t create this entry, then the /dev/sdd1 won’t mount automatically upon reboot. This isn’t a problem if that’s your intent or if you only have a few systems, but if you manage dozens or hundreds of systems, you need to set up /etc/fstab for each filesystem or disk that you wish to mount automatically.

Physical and logical volumes

When sysadmins speak of physical and logical volumes they’re referring to logical volume management (LVM). Figure 7-1 shows a visual reference of the Logical Volume Manager. A physical volume is a partition or disk that’s managed by a logical volume. The physical volume looks exactly like a disk partition. For example, the partition /dev/sdd1 is also the physical volume /dev/sdd1.

A volume group contains physical volumes. A logical volume is equivalent to a partition on a disk but you create logical volume partitions from volume groups. Logical volumes contain filesystems that are named and those names can be descriptive.

Another way of thinking about logical block devices is that volume groups are analogous to disks and logical volumes are analogous to disk partitions.

Figure 7-1. The Logical Volume Manager

The advantages of abstracting physical volumes into logical ones are that you have the flexibility of spanning multiple disks to create very large volume groups and that you can dynamically resize (Shrink and grow) logical volumes without taking the system offline to do so.

Checking space

Sysadmins should keep a close eye on disk space usage. If something goes terribly wrong on a system, log files can grow to fill filesystems. Users will fill filesystems and shared spaces with non-work files. Developers also often download gigabytes, or even terabytes, of code and other files without discussing their needs with the sysadmins or anyone else.

$ df -h

Filesystem           Size  Used Avail Use% Mounted on

/dev/mapper/cl-root  6.2G  3.3G  3.0G  53% /

/dev/sdb1            1.5G   43M  1.4G   3% /opt/software

/dev/sda1            976M  250M  660M  28% /boot

The df (Disk free) command lets you know at a glance how much available space there is on your mounted filesystems. The (-h) switch means human-readable, which means M for megabytes, G for gigabytes, etc. Monitoring systems or your own scripts can alert you to filesystems that fill beyond a given threshold. Setting a 90% threshold, for example, would trigger an alert that a particular filesystem is at or over 90% capacity.

For checking individual directories, the du command is very handy and provides you with a breakdown of what’s consuming space. For example, check the space consumed by the /var/log directory.

$ sudo du -h /var/log

0 /var/log/private

0 /var/log/samba/old

0 /var/log/samba

36K /var/log/sssd

28K /var/log/tuned

2.3M /var/log/audit

0 /var/log/chrony

3.3M /var/log/anaconda

10M /var/log

You can see at a glance which directories contain the most files. This is important if you’re auditing your system’s disk space before removing unneeded files.

Swap space

Swap space is a special type of Linux disk partition that extends a system’s memory beyond the limits of physical random access memory (RAM). Your system’s kernel makes use of swap space to write non-active programs from memory to disk freeing up memory for active programs. If a user or process activates those swapped programs, the system writes them from disk back into memory.

Swap space is not a remedy for memory problems. If your system has memory constraints, the solution is to add more physical RAM rather than increasing swap space or adding another swap partition. A system’s overuse of swap space creates a condition known as thrashing. Thrashing occurs when there are too many programs running, the swap partition is too small, or the system has insufficient physical RAM to support its processes.

I discuss how to create and manage swap space later in this chapter.

RAM-based temporary space (ramfs, tmpfs)

Ramfs and tmpfs are both filesystems whose files exist in virtual memory and are not written to disk. The tmpfs system is the newer and preferred RAM-based temporary filesystem and tmpfs is the default for all contemporary Linux distributions. One reason for the transition away from ramfs to tmpfs is that ramfs allowed itself to fill to capacity. Tmpfs has limit checking to prevent reaching its maximum capacity. Tmpfs has the added feature of using swap space to save resources.

The following listings show the tmpfs mount information for both CentOS and Ubuntu, respectively.

$ mount |grep tmpfs

devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=397220k,nr_inodes=99305,mode=755)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)

tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)

tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=82808k,mode=700,uid=1000,gid=1000)

$ mount |grep tmpfs

udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=456144k,nr_inodes=114036,mode=755)

tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=100480k,mode=755)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)

tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=100480k,mode=755)

tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=100480k,mode=700,uid=1000,gid=1000)

The purpose of tmpfs is to write temporary files and caches to memory rather than to disk because of the speed differences between RAM and disk. RAM is many times faster than the fastest SSD. The downside of tmpfs is that if you reboot or unmount the tmpfs, the data you have stored there is lost. Programs, processes, and users may all write to this temporary space. Similar to disk-based filesystems, a warning of “no space left on device” occurs when files, from whatever sources, have filled the allotted space.

Adding a New Disk to a System

In this section, I show you the steps required to add a new disk to a system and prepare it for use. The first sections describe how to add a disk to a physical system. The second section demonstrates how to perform the same procedure but using a virtual disk. You’ll also learn how to create a filesystem on the disk and mount it as a single usable directory. I also demonstrate how to set up a logical volume on a disk.

WARNING

You must be root or have sudo access to perform these system-level functions.

Installing the disk

For physical systems that have hot-swappable disk interfaces, you can simply attach a raw hard drive into the system without powering it down. If your system doesn’t have hot-swappable interfaces, then shutdown your system before adding a new disk to it. Once you’ve physically added the disk, power on the system (if required) and login to the console or connect remotely via SSH to set up the disk.

For those of you who use virtual machines (VMs), shut down your VM, add a new disk to it, and restart it. From this point on, the process is the same for physical and virtual systems.

Prepping the disk for use

The first task you have to perform is to determine the new disk’s device name. The system assigns the device name automatically. Use the fdisk command to display all attached disks and partitions.

$ sudo fdisk -l

Disk /dev/sda: 8 GiB, 8589934592 bytes, 16777216 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x38e117ab

Device     Boot   Start      End  Sectors Size Id Type

/dev/sda1  *       2048  2099199  2097152   1G 83 Linux

/dev/sda2       2099200 16777215 14678016   7G 8e Linux LVM

Disk /dev/sdb: 1.5 GiB, 1550843904 bytes, 3028992 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0xd268486b

The system identifies the raw disk device as /dev/sdb. Now that you’ve identified the disk’s device name, you can begin to initialize it using fdisk.

$ sudo fdisk /dev/sdb

Welcome to fdisk (util-linux 2.32.1).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Command (m for help): n

Partition type

p   primary (0 primary, 0 extended, 4 free)

e   extended (container for logical partitions)

Select (default p): <ENTER>

Partition number (1-4, default 1): <ENTER>

First sector (2048-3028991, default 2048): <ENTER>

Last sector, +sectors or +size{K,M,G,T,P} (2048-3028991, default 3028991): <ENTER>

Created a new partition 1 of type 'Linux' and of size 1.5 GiB.

Command (m for help): w

The partition table has been altered.

Failed to add partition 1 to system: Device or resource busy

The kernel still uses the old partitions. The new table will be used at the next reboot.

Syncing disks.

$ sudo fdisk -l

Disk /dev/sda: 8 GiB, 8589934592 bytes, 16777216 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x38e117ab

Device     Boot   Start      End  Sectors Size Id Type

/dev/sda1  *       2048  2099199  2097152   1G 83 Linux

/dev/sda2       2099200 16777215 14678016   7G 8e Linux LVM

Disk /dev/sdb: 1.5 GiB, 1550843904 bytes, 3028992 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0xd268486b

Device     Boot Start     End Sectors  Size Id Type

/dev/sdb1        2048 3028991 3026944  1.5G 83 Linux

You can ignore the message, Failed to add partition 1 to system: Device or resource busy. You can see that the partition, /dev/sdb1, does exist on the system. You have initialized the disk and created partition /dev/sdb1 on it. Now, you must create the filesystem. The following command formats the /dev/sdb1 partition using XFS.

$ sudo mkfs.xfs /dev/sdb1

meta-data=/dev/sdb1              isize=512    agcount=4, agsize=94592 blks

=                       sectsz=512   attr=2, projid32bit=1

=                       crc=1        finobt=1, sparse=1, rmapbt=0

=                       reflink=1

data     =                       bsize=4096   blocks=378368, imaxpct=25

=                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0, ftype=1

log      =internal log           bsize=4096   blocks=2560, version=2

=                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

List all block devices and filesystems.

$ sudo lsblk -f

NAME        FSTYPE      LABEL UUID                                   MOUNTPOINT

loop0       squashfs                                                 /var/lib/snapd/snap/asciinema/16

loop1       squashfs                                                 /var/lib/snapd/snap/core/10577

loop2       squashfs                                                 /var/lib/snapd/snap/core/10583

sda

├─sda1      ext4              1825fee2-65b0-43d6-8de9-1db26560935c   /boot

└─sda2      LVM2_member       PqHcgi-XAam-jsSw-xF2P-6eBc-IqiF-uCcDL0

├─cl-root xfs               41b93d34-8c91-40de-b054-d47b82846e53   /

└─cl-swap swap              6b31789a-6e77-4d86-baa8-3031d5d5dbf4   [SWAP]

sdb

└─sdb1      xfs               ca2701e0-3e75-4930-b14e-d83e19a5cffb

sr0

The lsblk command also displays the device’s universally unique identifier (UUID), which you need for the last step of the disk preparation process: mounting the filesystem.

TIP

You can also display the UUID by issuing the blkid command with the device name as the argument: $ sudo blkid /dev/sdb1

Next, create a directory on which you want to mount the new partition. In this example, I use /opt/software as the mount point.

$ sudo mkdir /opt/software

Mount the /dev/sdb1 partition onto the /opt/software directory.

$ sudo mount /dev/sdb1 /opt/software

Check to see if the mount command worked correctly.

$ mount |grep sdb1

/dev/sdb1 on /opt/software type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

The partition /dev/sdb1 successfully mounted on /opt/software. Now, you need to make this mount permanent, which means that it will survive a system reboot. Edit the /etc/fstab file and enter the following information into it.

UUID=ca2701e0-3e75-4930-b14e-d83e19a5cffb /opt/software xfs defaults 0 0

Save the file. The /dev/sdb1 partition will now mount automatically each time you reboot the system. Debian-derived systems handle this /etc/fstab syntax a bit differently. On the Ubuntu system, this /etc/fstab entry looks like the following:

/dev/disk/by-uuid/ca2701e0-3e75-4930-b14e-d83e19a5cffb /opt/software xfs defaults 0 0

The only task that’s left is to alter the permissions of /opt/software to allow users to access it or create subdirectories with appropriate permissions.

Implementing Logical Volumes

The main reason for implementing logical volumes is to have flexibility in allocating disk space. Having the capability of resizing live partitions is a huge feature because it means that adding or removing disk space from a partition doesn’t require any system downtime. You should still schedule a maintenance window when resizing a logical volume because there is still potential for something to go wrong during the process. If you make a mistake, you could lose data or possibly have to rebuild the entire logical volume. Disasters are somewhat unlikely but it does happen.

Resizing isn’t the only notable feature of logical volumes. You can span disks–meaning that you can create a very large logical volume from multiple disks. Only a few years ago, administrators were reluctant to span disks because spinning, mechanical hard drives are prone to failure. But using SSDs deprecates the “don’t span” rule. Yes, SSDs fail too but their lifespans and reliability make it much more reasonable to span when necessary.

In this section, I demonstrate how to set up a logical volume from a raw disk. (You can convert a currently used disk to a logical volume but you lose all the data on it in the process.)

Identifying available disks

To check your system for available (unused) disks or for disks that you want to convert to logical volumes, use the lsblk command.

$ lsblk

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

loop0         7:0    0  8.5M  1 loop /var/lib/snapd/snap/asciinema/16

loop1         7:1    0 97.9M  1 loop /var/lib/snapd/snap/core/10577

loop2         7:2    0 97.9M  1 loop /var/lib/snapd/snap/core/10583

sda           8:0    0    8G  0 disk

├─sda1        8:1    0    1G  0 part /boot

└─sda2        8:2    0    7G  0 part

├─cl-root 253:0    0  6.2G  0 lvm  /

└─cl-swap 253:1    0  820M  0 lvm  [SWAP]

sdb           8:16   0  1.5G  0 disk

sr0          11:0    1 1024M  0 rom

From the listing, you can see that the disk, sdb, is available for use.

sdb           8:16   0  1.5G  0 disk

First, you must create the physical volume (PV), which is the basic block device onto which you’ll build logical volumes. Use the pvcreate command and the device name to initialize the disk as a physical volume.

$ sudo pvcreate /dev/sdb

WARNING: dos signature detected on /dev/sdb at offset 510. Wipe it? [y/n]: y

Wiping dos signature on /dev/sdb.

Physical volume "/dev/sdb" successfully created.

Check the newly created physical volume with the pvs (PV Show) command.

$ sudo pvs

PV         VG Fmt  Attr PSize  PFree

/dev/sda2  cl lvm2 a--  <7.00g    0

/dev/sdb      lvm2 ---   1.44g 1.44g

Use the pvdisplay command to see details about your physical volume.

$ sudo pvdisplay /dev/sdb

"/dev/sdb" is a new physical volume of "1.44 GiB"

--- NEW Physical volume ---

PV Name               /dev/sdb

VG Name

PV Size               1.44 GiB

Allocatable           NO

PE Size               0

Total PE              0

Free PE               0

Allocated PE          0

PV UUID               yl6k0u-cymt-p4jd-VXro-Lit1-EkLe-Jqjc94

WARNING

If your disk or filesystem is mounted when you attempt to use pvcreate, you’ll receive the following error:

$sudo pvcreate /dev/sdb

Can't open /dev/sdb exclusively.  Mounted filesystem?

You’ll need to unmount /dev/sdb before proceeding.

The second step is to create a Volume Group (VG) from the physical volume /dev/sdb using the vgcreate command.

$ sudo vgcreate vgsw /dev/sdb

Volume group "vgsw" successfully created

$ sudo vgs

VG   #PV #LV #SN Attr   VSize  VFree

cl     1   2   0 wz--n- <7.00g    0

vgsw   1   0   0 wz--n-  1.44g 1.44g

$ sudo vgdisplay vgsw

--- Volume group ---

VG Name               vgsw

System ID

Format                lvm2

Metadata Areas        1

Metadata Sequence No  1

VG Access             read/write

VG Status             resizable

MAX LV                0

Cur LV                0

Open LV               0

Max PV                0

Cur PV                1

Act PV                1

VG Size               1.44 GiB

PE Size               4.00 MiB

Total PE              369

Alloc PE / Size       0 / 0

Free  PE / Size       369 / 1.44 GiB

VG UUID               AsQkWT-UpSu-3dYk-5EWT-DbQR-SovE-2oAmlR

Referring back to Figure 7-1, you see that you create volume groups from physical volumes. In this demonstration, I only use one physical volume. Now that you have a volume group, you must create logical volumes from it. Creating the logical volume is the third step in this process.

For this demonstration, I begin by deciding how much space to allocate to the logical volume. I decide to use 1 GB of the 1.5 GB disk. It’s now time to create the logical volume using the lvcreate command.

The general syntax of the lvcreate command is as follows:

$ sudo lvcreate -L size -n lvname vg

You must provide the size parameter in G for gigabytes or M for megabytes. The lvname is the name you want to use for this logical volume (software-lv) and you must supply the volume group from which you want to create the logical volume (vgsw).

$ sudo lvcreate -L 1G -n software-lv vgsw

WARNING: xfs signature detected on /dev/vgsw/software-lv at offset 0. Wipe it? [y/n]: y

Wiping xfs signature on /dev/vgsw/software-lv.

Logical volume "software-lv" created.

From the above message, you see that the disk or partition had previously held an xfs signature, which means it’s not a new disk or partition but a recycled one.

NOTE

There’s no problem with using a previously used disk but realize that all the information on it will be overwritten and unrecoverable in this process.

Use the lvs and lvdisplay commands to list details about your logical volumes.

$ sudo lvs

LV          VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

root        cl   -wi-ao----  <6.20g

swap        cl   -wi-ao---- 820.00m

software-lv vgsw -wi-a-----   1.00g

$ sudo lvdisplay /dev/vgsw/software-lv

--- Logical volume ---

LV Path                /dev/vgsw/software-lv

LV Name                software-lv

VG Name                vgsw

LV UUID                ebB3ST-3E7k-BShG-8oPi-sj0c-yXXr-C7EgAw

LV Write Access        read/write

LV Creation host, time server1, 2021-12-10 07:34:56 -0600

LV Status              available

# open                 0

LV Size                1.00 GiB

Current LE             256

Segments               1

Allocation             inherit

Read ahead sectors     auto

- currently set to     8192

Block device           253:2

NOTE

You must use the full device name of the logical volume when you filter by the logical volume name.

$ sudo lvs /dev/vgsw/software-lv

LV     VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

software-lv vgsw -wi-a----- 1.00g

The fourth step in this process is to create a filesystem on your logical volume. You perform this task just as you would for any partition.

$ sudo mkfs.xfs /dev/vgsw/software-lv

meta-data=/dev/vgsw/software-lv  isize=512    agcount=4, agsize=65536 blks

=                       sectsz=512   attr=2, projid32bit=1

=                       crc=1        finobt=1, sparse=1, rmapbt=0

=                       reflink=1

data     =                       bsize=4096   blocks=262144, imaxpct=25

=                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0, ftype=1

log      =internal log           bsize=4096   blocks=2560, version=2

=                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

Create a mount point for your filesystem.

$ sudo mkdir /sw

Mount the filesystem onto the mount point.

$ sudo mount /dev/vgsw/software-lv /sw

$ mount |grep software

/dev/mapper/vgsw-software--lv on /sw type xfs  \

(rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Check available space on the device.

$ df -h /sw

Filesystem                     Size  Used Avail Use% Mounted on

/dev/mapper/vgsw-software--lv 1014M   40M  975M   4% /sw

The final step is to add the filesystem and its mount point to /etc/fstab to make the mount persistent. My entry looks like the following:

/dev/vgsw/software-lv /sw       xfs     defaults        0 0

NOTE

You don’t use the UUID for defining logical volumes in /etc/fstab. Instead, you use the device name.

The logical volume will mount automatically on reboot. The next section describes how to increase the size or extend a logical volume.

Extending a Logical Volume

For the logical volume that I configured in this scenario, I used 1 GB of the 1.5 GB total disk size for /dev/vgsw/software-lv. To extend this volume, you can use one the following general commands:

$ sudo lvextend -L +size(M or G) lvname

$ sudo lvextend -l +100%FREE lvname

In this example, I’ll use the -l (extents) option rather than a specific size so that I can consume the rest of the free space on the device.

$ df -h /sw

Filesystem                     Size  Used Avail Use% Mounted on

/dev/mapper/vgsw-software--lv 1014M   40M  975M   4% /sw

$ sudo lvextend -l +100%FREE /dev/vgsw/software-lv

$ sudo lvextend -l +100%FREE /dev/vgsw/software-lv

Size of logical volume vgsw/software-lv changed from 1.00 GiB (256 extents) to 1.44 GiB (369 extents).

Logical volume vgsw/software-lv successfully resized.

The lvextend command extends the logical volume to its maximum capacity but a df shows the same amount of available space.

$ df -h /sw

Filesystem                     Size  Used Avail Use% Mounted on

/dev/mapper/vgsw-software--lv 1014M   40M  975M   4% /sw

You have extended the logical volume but not the filesystem. Now resize the filesystem using the xfs_growfs command with no size parameter. Not specifying a size parameter with -D size, xfs_growfs will extend the filesystem to its maximum value.

$ sudo xfs_growfs /dev/vgsw/software-lv

meta-data=/dev/mapper/vgsw-software--lv isize=512    agcount=4, agsize=65536 blks

=                       sectsz=512   attr=2, projid32bit=1

=                       crc=1        finobt=1, sparse=1, rmapbt=0

=                       reflink=1

data     =                       bsize=4096   blocks=262144, imaxpct=25

=                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0, ftype=1

log      =internal log           bsize=4096   blocks=2560, version=2

=                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

data blocks changed from 262144 to 377856

$ df -h /sw

Filesystem                     Size  Used Avail Use% Mounted on

/dev/mapper/vgsw-software--lv  1.5G   43M  1.4G   3% /sw

NOTE

You cannot shrink an XFS volume.

You have successfully extended the filesystem and your logical volume now has its full capacity (1.5 GB) available for use. The actual available size is 1.44 GB because of filesystem overhead.

Decommissioning and Disk Disposal

System decommissioning involves the wiping or destruction of disks prior to disposal. The decommissioning process differs from company to company but generally follows the steps below.

- Notification

- “Scream” test

- Power down

- Disk wiping

- Unracking

- Palletizing

- Disposal

The timeline for each of these steps varies. The following sections provides details for each step.

Notification

Stakeholders, system administrators, network administrators, storage administrators, and management all receive multiple decommissioning notifications for a list of systems. Large companies typically send the list out once a week for a period of three to four weeks. These notifications give stakeholders and others a chance to take possession of a system or to prevent its decommissioning via email and then a discussion during a governance meeting. If no one speaks up on behalf of any listed system, the process proceeds to the “scream” test phase.

Scream test

The so-called “scream” test is a period of two or more weeks where a system administrator or data center staff member unplugs from the network but doesn’t power down the listed systems. The plan is that during this time if someone screams about a system being down, that system would be plugged back into the network and its operations would continue as before. The next governance meeting would remove the system from the decommissioning list.

Power down

The next milestone in the decommissioning process is the power down phase which lasts two or more weeks. System administrators power down all listed systems. This period is a second chance for interested parties to claim a system or to notify the governance committee that the system needs to remain in operation.

Disk wiping

After several weeks of notifications and waiting, the governance committee finalizes the list and submits it to system administrators for disk wiping. The system administrator powers on each system and uses a disk wiping utility to overwrite every local disk. Leveraged storage such as SAN, NAS, or other non-local disk is not included in this process. This process ensures that no data is left on a system’s internal storage.

NOTE

I have relied on Darik’s Boot and Nuke (DBAN) utility for years to wipe disks for decommissioning and disposal. You can find it at dban.org. It is a free, open source utility for hard disk drives (HDDs). This product is not for solid-state drives (SSDs).

Unracking and palletizing

Once a system’s disks have been wiped, the list goes to data center personnel for unracking. Technicians remove the systems from data center racks and place them onto shipping pallets. Once a pallet is full, the technician labels it and the pallet goes into a disposal queue.

Disposal

The disposal process usually consists of a bidding process where companies bulk purchase palletized systems for sale or redeployment. Sometimes the systems go to a recycling facility where technicians remove disks, CPUs, memory, and other salvageable components for individual or bulk resale. System disposal can also mean that those whole systems are disposed of by crushing or shredding and then sold as recyclable material.

Summary

This chapter covered multiple aspects of disk management including general disk-related information, adding a new disk to a system, logical volume management, and decommissioning and disposal.

In the next chapter, I cover system health and housekeeping, which are two important tasks for system administrators. Also covered are preventative maintenance, security updates, and periodic health checks.

Chapter 8. Maintaining System Health

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the eighth chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at LSAbookfeedback@gmail.com.

Maintaining system health is a broad topic that includes preventative maintenance, housekeeping, patching, security tasks, user maintenance, and monitoring and mitigating various types of sprawl. Maintaining the health of your systems is an active task, not a passive one.

Monitoring can help. Automating periodic cleanup of certain areas can help. Automating updates can help, but you still must actively watch logs, check space and sprawl, and take care of user maintenance tasks. Many of these tasks require eyes on screens and hands on keyboards. If you could automate every aspect of system health maintenance, far fewer system administrators would be required.

This chapter covers both automated and manual system health maintenance tasks.

Keeping your system clutter-free

Housekeeping is something that no one loves to do. It’s tedious, time-consuming, and can anger users to the point of reporting your behavior to management, which is never a good thing. As long as you comply with corporate policy and don’t apply any of your own, you’ll have material to refer your angry users to. The rules for decluttering a system are the same for any maintenance you do: Have a good backup available if you remove a file or directory that you need to replace. The following sections detail how to keep your systems clutter-free.

Cleaning the /tmp directory

The /tmp directory is a shared directory. It’s shared with all users, applications, and with the system itself. Anyone may write to this directory, which is bad for administrators because having unlimited access to a system directory can have fatal consequences. If the original administrator didn’t create the /tmp directory as a separate filesystem with access to limited space, users or applications can potentially fill up all of the space. The /tmp directory should never be a part of the same partition as / for that reason alone.

The /tmp directory is tricky from a housekeeping perspective for sysadmins because there are no restrictions on users, applications, or the system. Because any user may write to it, they might expect that it’s extra, available free space where they can store downloads and other files. To make this directory a less desirable location to store files, you should create a cron job to run every night, after backups, to remove any non-system files. The tricky part is that hidden files aren’t removed with a standard rm command. Clever application developers who direct their applications to write into the /tmp directory should write their temporary files as hidden ones. Writing to a special log directory keeps well-meaning sysadmins and automated cleanup scripts from removing them.

NOTE

In many enterprises, automated scripts remove everything from the /tmp directory every night, and most exclude /tmp directories from backups.

The solution to the /tmp dilemma if you don’t want to create a separate filesystem and mount it on /tmp is to enable tmp.mount. This service creates a temporary filesystem (tmpfs) and mounts it as /tmp. One of the features is that it’s volatile storage (RAM) and filling it up won’t cause stability issues on your system.

To enable tmp.mount on your system, follow these steps:

Check your /tmp directory mount point to see if you need to configure tmp.mount.

$ df -h /tmp

Filesystem           Size  Used Avail Use% Mounted on

/dev/mapper/cl-root  6.2G  3.3G  3.0G  53% /

As you can see, my /tmp has the flaw of being a subdirectory of the / filesystem. Enable tmp.mount and then start it.

$ sudo systemctl enable tmp.mount

Created symlink /etc/systemd/system/local-fs.target.wants/tmp.mount → \ /usr/lib/systemd/system/tmp.mount.

$ sudo systemctl start tmp.mount

$ df -h /tmp

Filesystem      Size  Used Avail Use% Mounted on

tmpfs           405M     0  405M   0% /tmp

Now, the /tmp directory is no longer a subdirectory of /. Enabling tmp.mount makes this setting persistent, which means it is a permanent change and will survive a reboot.

The next section deals with keeping the /home directory usable for everyone.

Making /home a livable space for everyone

The /home directory is sort of a shared directory because it contains all system user’s home directories. For example, suppose you have users tux, penguin, and gentoo on your system. Their home directories will look like the following:

/home/tux

/home/penguin

/home/gentoo

On systems where the /home directory is on the same filesystem as the / directory, there is a potential that users can fill up / and cause system problems, much the same as filling up /tmp, discussed in the previous section. Because of the nature of files contained in /home, you can’t create and mount a volatile filesystem for it. The /home directory must be a permanently mounted filesystem, preferably not part of /. If it is, there is a procedure to correct it that either involves shrinking an existing LVM filesystem, creating a partition and filesystem, and mounting that filesystem as /home or installing a new disk, creating a new partition, and mounting that partition as /home. For the demonstration in this chapter, I’ll use the second approach of using a new disk for /home.

The steps to complete this are as follows:

- Create or install a new disk

- Create a new partition on the disk

- Make the filesystem on the new partition

- Mount the new partition on /mnt

- Copy all files from /home to /mnt

- Remove all files from /home

- Unmount the new partition from /mnt

- Mount the new partition on /home

- Add an entry for /home in /etc/fstab

I created a 1 GB disk on my virtual machine to demonstrate these steps.

Create a new partition on the disk

Identify the new disk by listing all disk block devices on the system.

$ lsblk |grep disk

sda                   8:0    0    8G  0 disk

sdb                   8:16   0  1.5G  0 disk

sdc                   8:32   0    1G  0 disk

The sdc (/dev/sdc) device is the new disk. To create a new partition on the disk, use the fdisk utility.

$ sudo fdisk /dev/sdc

Welcome to fdisk (util-linux 2.32.1).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Device does not contain a recognized partition table.

Created a new DOS disklabel with disk identifier 0x3c239df6.

Command (m for help): n

Partition type

p   primary (0 primary, 0 extended, 4 free)

e   extended (container for logical partitions)

Select (default p): p

Partition number (1-4, default 1):

First sector (2048-2097151, default 2048):

Last sector, +sectors or +size{K,M,G,T,P} (2048-2097151, default 2097151):

Created a new partition 1 of type 'Linux' and of size 1023 MiB.

Command (m for help): w

The partition table has been altered.

Calling ioctl() to re-read partition table.

Syncing disks.

List the sdc block device to verify that the partition /dev/sdc1 exists.

$ lsblk |grep sdc

sdc                   8:32   0    1G  0 disk

└─sdc1                8:33   0 1023M  0 part

The /dev/sdc1 partition exists and is ready to have a filesystem created on it.

Create the filesystem

For the /home directory, I’m using the ext4 filesystem type.

$ sudo mkfs.ext4 /dev/sdc1

mke2fs 1.45.6 (20-Mar-2020)

Creating filesystem with 261888 4k blocks and 65536 inodes

Filesystem UUID: bcd18fcc-3774-4b94-be0a-e2abf8aa4d31

Superblock backups stored on blocks:

32768, 98304, 163840, 229376

Allocating group tables: done

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information: done

Verify that you correctly created the ext4 filesystem on /dev/sdc1.

$ lsblk -o NAME,FSTYPE,SIZE |grep sdc

sdc                                1G

└─sdc1              ext4        1023M

Mount the new partition

To use the new partition, you must mount it. I use /mnt as the temporary mount point so that I can copy files from /home to the new partition that I will eventually mount as /home. If I mount /dev/sdc1 onto /home now, I can’t copy the files because mounting on /home hides its files from view.

$ sudo mount /dev/sdc1 /mnt

$ mount |grep sdc

/dev/sdc1 on /mnt type ext4 (rw,relatime,seclabel)

Copy files from /home to /mnt

Copy all files from the /home directory recursively (-R) and preserve permissions and timestamps (-p).

$ sudo cp -pR /home/* /mnt

$ ls -la /home

total 0

drwxr-xr-x.  4 root  root   32 Nov  3  2020 .

dr-xr-xr-x. 19 root  root  258 Dec 10 08:37 ..

drwx------.  2  1001  1001  62 Nov 13  2020 diane

drwx------.  7 khess khess 273 Dec 14 21:03 khess

$ ls -la /mnt

total 28

drwxr-xr-x.  5 root  root   4096 Dec 15 08:33 .

dr-xr-xr-x. 19 root  root    258 Dec 10 08:37 ..

drwx------.  2  1001  1001  4096 Nov 13  2020 diane

drwx------.  7 khess khess  4096 Dec 14 21:03 khess

drwx------.  2 root  root  16384 Dec 15 08:17 lost+found

Remove all files from /home

The reason that I remove all files from the original /home directory is that it releases disk space back to the / filesystem, which is one of the reasons why we’re going through this exercise.

$ sudo rm -rf /home/*

This removes all files under /home without removing the /home directory. There’s no need to remove /home.

Unmount /dev/sdc1 from /mnt

You must unmount /dev/sdc1 from /mnt so that you can mount it onto the /home directory.

$ sudo umount /mnt

Mount the new partition on /home

$ sudo mount /dev/sdc1 /home

The true test of this exercise is to login as a user and check if everything works as expected.

NOTE

You might find that this process has broken existing ssh key associations. If it does, each user must use the following command to re-establish the passwordless connection: $ ssh-copy-id <hostname> and enter their password to complete the process.

Create the /etc/fstab entry

The /home directory won’t mount automatically after a system reboot because /home is now a different filesystem. You must create an /etc/fstab entry for it.

My /etc/fstab entry for /dev/sdc1 is:

/dev/sdc1  /home        ext4  defaults  0 0

This /etc/fstab entry makes the mount persistent.

The next section deals with archiving and removing “stale” and duplicated files from shared directories.

Decluttering shared directories

Keeping shared directories free of clutter is difficult. It might be impossible, but there are some things you can do to confront the issue using system tools and some planning. When users complain or report you to management, your best defense is to refer to the employee manual, handbook, or corporate policy manual. Some exceptions are likely to occur, even if you’re right and you have the corporate policy manual at your disposal.

This section explores some utilities that can make decluttering and housekeeping a little easier.

Deduplicating files with fdupes

Fdupes is a popular deduplication utility. Fdupes finds duplicate files in a given set of directories. The fdupes man page describes its function as:

Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte comparison.

You can install fdupes from your distribution’s repositories but it’s better to grab the patched version from a github repository using the following steps:

$ git clone https://github.com/tobiasschulz/fdupes

$ cd fdupes

$ make fdupes

$ sudo make install

This patched version has more options such as the one you’re most likely to use as a system administrator in a production environment. When you use fdupes on a directory, you can just look at the duplicates, delete the duplicates (not recommended), or remove the duplicates but provide links to one of the files (recommended). The only problem with this solution is if the original file no longer exists. But providing a link to one of the files is far less invasive than actually removing the duplicates, which will undoubtedly result in tickets, phone calls, and reprimands. Here are two good examples of how to use fdupes to find duplicates, generate a report, and then provide links to one of the files.

Example 8-1. List duplicate files and their sizes.

$ fdupes -rS /opt/shared

29 bytes each:

/opt/shared/docs/building/test.rtf

/opt/shared/docs/building/test.txt

/opt/shared/a/b/list.txt

/opt/shared/a/b/c/stuff.doc

/opt/shared/a/todo.list

/opt/shared/one/foo.lst

/opt/shared/x/y/listed.here

20 bytes each:

/opt/shared/docs/hr/list.txt

/opt/shared/x/got.txt

/opt/shared/a/b/none.doc

The -r option is recursive which means check the specified directory and all of its subdirectories. The -S option means to report the file sizes.

Example 8-2. Generate a duplicate file report using the -m option.

$ fdupes -rm /opt/shared

8 duplicate files (in 2 sets), occupying 214 bytes.

Example 8-2 might scare you a bit because it uses the options -rm, which ordinarily would mean remove, but in this case, they mean recursive and summary (report).

Example 8-3. Provide links to one of the files.

$ fdupes -rL /opt/shared

[+] /opt/shared/docs/building/test.rtf

[h] /opt/shared/docs/building/test.txt

[h] /opt/shared/a/b/list.txt

[h] /opt/shared/a/b/c/stuff.doc

[h] /opt/shared/a/todo.list

[h] /opt/shared/one/foo.lst

[h] /opt/shared/x/y/listed.here

[+] /opt/shared/docs/hr/list.txt

[h] /opt/shared/x/got.txt

[h] /opt/shared/a/b/none.doc

In Example 8-3, fdupes replaces duplicate files with hard links using the -L option. Only the first file in each group of duplicates remains intact.

Using the selective delete option, -d, you’re prompted for which files you want to keep. Again, deleting files is highly discouraged. Proceed at your own risk.

$ fdupes -rd /opt/shared

[1] /opt/shared/a/b/goo.txt

[2] /opt/shared/a/false.doc

[3] /opt/shared/one/two/three/three

[4] /opt/shared/docs/building/test.rtf

[5] /opt/shared/x/y/new.txt

[6] /opt/shared/docs/building/test.txt

[7] /opt/shared/a/b/list.txt

[8] /opt/shared/a/b/c/stuff.doc

[9] /opt/shared/a/todo.list

[10] /opt/shared/one/foo.lst

[11] /opt/shared/x/y/listed.here

Set 1 of 2, preserve files [1 - 11, all]:

This option immediately removes files except for the ones you preserve and does not create links. Check the online help and man pages for fdupes for other options.

Tackling /home file sprawl with quotas

Some system administrators and most users find that using quotas is a harsh solution to dealing with file sprawl. I find it to be a good solution that works well in various situations to help deal with users who chronically use up more than their fair share of disk space. You can implement quotas on any directory but you certainly want to consider it for shared directories and /home.

The first task is to install quota, if your system doesn’t have it.

$ sudo yum install quota

$ sudo apt install quota

If not already complete, create an xfs filesystem and have it ready and mounted. Mine, for example, is /home. Create an entry in /etc/fstab for the xfs filesystem. Here is my /etc/fstab entry for /home where I want to enforce quotas.

/dev/sdc1  /home        xfs  defaults,usrquota,grpquota  0 0

Next, create two new files on /home: quota.group and quota.user

$ sudo touch /home/quota.group /home/quota.user

Then, enable quotas with the quotaon command.

$ sudo quotaon /home

quotaon /home

quotaon: Enforcing group quota already on /dev/sdc1

quotaon: Enforcing user quota already on /dev/sdc1

quotaon: Enable XFS project quota accounting during mount

The system is now ready to apply quotas to users’ accounts. The following command sets a very low quota value for demonstration purposes. The soft limit is 50 MB, and the hard limit is 80 MB. You receive a warning when you exceed the soft limit, but the quota system prevents you from exceeding the hard limit. You can also limit the number of inodes (file pointers) that a user may consume. The inode limit is a bit harsher because a user might generate thousands of tiny text files that only amount to a few megabytes, but it limits the number of files users can create. There are hard and soft limits for inodes as well.

[root@server1 home1]# sudo xfs_quota -x -c 'limit -u bsoft=50m bhard=80m isoft=60 \ ihard=80 djones' /home

The following is an example of what the user sees when they exceed a quota.

$ su - djones

Password:

Last login: Mon Jan 10 20:16:49 CST 2022 on pts/0

[djones@server1 ~]$ head -c 51MB /dev/urandom > fillit.txt

[djones@server1 ~]$ head -c 51MB /dev/urandom > fillit1.txt

head: error writing 'standard output': Disk quota exceeded

$ ls -l

total 81892

-rw-rw-r--. 1 djones djones 32854016 Jan 10 20:21 fillit1.txt

-rw-rw-r--. 1 djones djones 51000000 Jan 10 20:21 fillit.txt

You can see that when the user reached their hard limit, the system truncated file they attempted to create (fillit1.txt). If there had been no quota placed on the user, they would have had two 51 MB files.

Quotas prevent users from consuming more than a reasonable amount of space on a system, but should only be enforced on users who violate company policy or eggregiously store personal files on company systems. You can easily remove a quota limit from a user’s account by setting all limits to 0.

$ sudo xfs_quota -x -c 'limit -u bsoft=0 bhard=0 isoft=0 ihard=0 djones' /home

[sudo] password for khess:

[khess@server1 home]$ su - djones

Password:

Last login: Mon Jan 10 20:20:41 CST 2022 on pts/0

[djones@server1 ~]$ head -c 51MB /dev/urandom > fillit2.txt

[djones@server1 ~]$ ls -l

total 131700

-rw-rw-r--. 1 djones djones        0 Jan 10 20:21 blah.txt

-rw-rw-r--. 1 djones djones 32854016 Jan 10 20:21 fillit1.txt

-rw-rw-r--. 1 djones djones 51000000 Jan 10 22:37 fillit2.txt

-rw-rw-r--. 1 djones djones 51000000 Jan 10 20:21 fillit.txt

Now you see no warnings or truncations about exceeding limits. The djones account no longer has quota limits imposed on it.

That’s a good introduction to quotas, what they can do for you as an administrator, and how they can keep your users in check with their space usage. The next topic to explore is patching, an essential task for any administrator wanting to keep their system running smoothly.

Patching your way to a healthy system

Patching, to the uninitiated, might sound like you’re putting duct tape on a system to hold it together temporarily while you locate the real fix for a problem. Nothing could be further from the truth. Patching is an essential task and perhaps is a misnomer for what it is: updating system utilities, daemons, and applications, with fixes supplied by developers. A software patch fixes a specific problem. A good example is applying a patch to the SSH daemon to resolve some newly discovered vulnerabilities that can render a system susceptible to attack and compromise.

Patching isn’t always security-focused. Often, patches fix a stability problem, plug a memory leak, or repair a broken feature. Remember that patching a Linux system rarely means that you’ll have to initiate a reboot, but if you update a service, you’ll want to restart that service unless the patching process takes care of that for you. To be safe, I often reboot after a major patch event to ensure that the system and its services receive a refresh.

A major patch event means one that includes multiple services and a kernel update. If you are inside your maintenance window and you’ve updated several services, the kernel, or other critical system software, I suggest that you reboot the system. An earlier chapter states that no one receives medals for long uptimes. Rebooting is a good thing, so don’t shun it.

Some system administrators automate their patching by setting up a cron job to check for and install patches periodically. That will work, but you should have a test system or two that you manually update so that you can see what’s to be updated, how the system reacts to those updates, and how the system responds after a reboot. You don’t want or need any surprises from a bad patch session.

Patching a Red Hat Enterprise Linux-based system

To initiate patches on a RHEL-based system, use yum or dnf. For this system, I used yum. DNF works the same way and is the preferred command for versions 8 and above.

$ sudo yum update

Last metadata expiration check: 0:23:24 ago on Sun 30 Jan 2022 08:27:15 AM CST.

Dependencies resolved.

========================================================================

Package         Arch    Version                 Repo      Size

========================================================================

Installing:

kernel          x86_64  4.18.0-348.7.1.el8_5    baseos    7.0 M

kernel-core     x86_64  4.18.0-348.7.1.el8_5    baseos    38 M

kernel-modules  x86_64  4.18.0-348.7.1.el8_5    baseos    30 M

Upgrading:

NetworkManager  x86_64  1:1.32.10-4.el8         baseos    2.6 M

*** truncated output ***

openssh         x86_64  8.0p1-10.el8            baseos    522 k

openssh-clients x86_64  8.0p1-10.el8            baseos    668 k

openssh-server  x86_64  8.0p1-10.el8            baseos    485 k

openssl         x86_64  1:1.1.1k-5.el8_5        baseos    709 k

openssl-libs    x86_64  1:1.1.1k-5.el8_5        baseos    1.5 M

*** truncated output ***

yum             noarch   4.7.0-4.el8            baseos    205 k

yum-utils       noarch   4.0.21-3.el8           baseos    73 k

Installing dependencies:

libbpf          x86_64   0.4.0-1.el8            baseos    110 k

Removing:

kernel          x86_64   4.18.0-240.1.1.el8_3   @BaseOS   0

kernel-core     x86_64   4.18.0-240.1.1.el8_3   @BaseOS   62 M

kernel-modules  x86_64   4.18.0-240.1.1.el8_3   @BaseOS   21 M

Transaction Summary

========================================================================

Install    4 Packages

Upgrade  287 Packages

Remove     3 Packages

Total download size: 558 M

Is this ok [y/N]:

As you can see from the truncated listing above, it’s been a while since I last updated this system. I included the parts I want you to see such as a new kernel update and the new OpenSSH server (SSHD) to demonstrate that I will reboot this system after the updates. I prefer to install manually by not providing the -y option so that I can view everything that needs an upgrade before I proceed.

NOTE

The default behavior for YUM/DNF updates is to not install them. Is this ok [y/N]:

The updates will not install if you press ENTER. You have to explicitly approve the installation by answering with a ‘y’ here.

It’s generally OK to proceed with updates on production systems if you’ve installed and verified them on test systems first. I don’t think it’s a good idea to blindly install updates without having actual eyes on them and to check them first on test systems. Doing so might cause stability problems or application/library conflicts, especially if you run older versions of some programs.

Patching a Debian-based Linux system

Patching a Debian-based system is similar to patching a RHEL-based system with a couple of subtle differences. The first difference is that you use the apt command to install updates to the system.

$ sudo apt update

Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease

Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]

Get:3 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]

Get:4 http://us.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]

Fetched 336 kB in 1s (407 kB/s)

Reading package lists... Done

Building dependency tree

Reading state information... Done

20 packages can be upgraded. Run 'apt list --upgradable' to see them.

khess@server2:~$ sudo apt upgrade

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

The following packages will be upgraded:

cloud-init command-not-found libasound2 libasound2-data libnetplan0 libssl1.1 linux-firmware netplan.io openssh-client

openssh-server openssh-sftp-server openssl python3-commandnotfound python3-software-properties rsync

software-properties-common ubuntu-advantage-tools ufw update-notifier-common wget

20 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Need to get 120 MB of archives.

After this operation, 20.3 MB of additional disk space will be used.

Do you want to continue? [Y/n]

The other difference is in the approval procedure. Note that the Do you want to continue? [Y/n] option. The Y is capitalized, which means that if you press the ENTER key, the updates will install to your system. Remember that when updating a RHEL-based system, if you press the ENTER key, the updates don’t install. But, also note that apt doesn’t have an automatic approval option (-y) like yum or dnf do.

NOTE

Unattended-upgrades is a package that you can install that will allow you to update your Debian-based system automatically.

As stated earlier, you should always install updates and upgrades on test systems first. If you set up automated updates, this is more difficult to do unless you create a clever schedule for updating your systems similar to the following:

- Test systems: Manual updates every Tuesday.

- Development systems: Manual updates every other Thursday.

- Production systems: Monthly updates last Sunday of the Month.

This schedule, or one similar to it, allows you to check automated updates before installing them to your production systems. If you have an automation tool or some enterprise management tool that allows you to turn off cron jobs in case of a problem, then you can better control your automated patching. If you don’t have such a tool and you have more than a handful of systems to manage, I suggest that you explore an automation tool such as Ansible to deliver new configuration files, scripts, and so on to your systems.

An automation system also helps when applying zero-day or other emergency patches to your systems. Patching, like backups, needs your close attention. Keeping your systems running smoothly and securely by regularly applying the latest patches is a major part of your system administrator duties.

Securing your systems

While regular patching should be part of an overall security plan, there are some basic security settings, unrelated to the topic of patching, that I will cover in this section. Again, an automated system such as Ansible can help greatly in saving time and effort in maintaining the security of dozens, hundreds, or thousands of systems.

And not all systems can be treated equally when applying security measures. For example, your DMZ systems, your database systems, your web servers, your application servers, and your file storage systems each (as a group) need different security settings. You can rarely apply a single security policy covering every type of system. There are a few, but to state that you’re going to set up a single, generic security plan is not realistic. You need to apply security configurations to each system type to focus on specific vulnerabilities.

For example, can you think of particular security settings that apply to web servers and not file servers? In the case of a web server, you’ll need to implement certificates to secure your web communications. On a file server, you need to concentrate your efforts on securing shared directories and user accounts. Are you going to allow Samba shares on your file servers? Are you going to allow uploads onto your web servers? How you and your users interact with a system dictates its security concerns, not the fact that it’s a generic Linux system.

The following table shows some example security guidelines for your Linux systems with specific purposes and workloads:

Purpose/Workload

Description

Security Measure

Web Server

Apache, NGINX, IHS, IIS, others

TLS, certificates, HTTPS

Database Server

MySQL, PostgreSQL, others

Tunnel over SSH, restrict connections to localhost

File Server

Samba shares, NFS shares, SSH

/etc/hosts.allow and /etc/hosts.deny restrictions, Active Directory integration for Samba, 2FA

Application Server

Tomcat, WebSphere, others

HTTPS, TLS, certificates

DMZ Server

Services exposed to internet

Firewall, SELinux, SSH, /etc/hosts.allow and /etc/hosts.deny restrictions

Mail Server

SMTP, IMAP, POP3 protocols

Use secured protocols and mail servers

All Servers

Miscellaneous services

SSH, /etc/hosts.allow and /etc/hosts.deny restrictions, Firewall, SELinux

As you can see from the list above, there are certain security features that you should implement on every system regardless of purpose unless there’s some vendor-related reason that you shouldn’t. You should lockdown every system and be as restrictive as possible while maintaining functionality and accessibility for your users.

Maintaining user and group accounts

User account maintenance goes beyond adding and removing accounts. It also includes developing an account naming convention, creating policies that set standards for how long to disable accounts before they’re removed, how long to retain a user’s home directory after they’ve left the group or the company, retiring of group accounts, and the retiring or reusing of user and group IDs. All of these activities prevent user account sprawl. Here are some examples of account sprawl:

- Disabled accounts left on systems past policy limits

- Home directories left on systems past policy limits

- Active, terminated user accounts left on systems

- Group accounts with no members

- Disabled or removed accounts that remain in /etc/sudoers

- Active temporary accounts left on systems

- Active service accounts with shells or passwords

Setting up a naming convention

One of the easiest methods of circumventing account sprawl is to create a user account naming convention. A naming convention prevents sprawl by setting a standard by which you and your team create user accounts. For example, you now have a working system if you set your naming convention as the first initial and first seven characters of the last name. The exception is when two or more people have the same first and last names. In these cases, you’d opt for using the first initial of the first name, a middle initial, and the first six characters of the last name.

First, here are examples of the standard naming convention:

First Name

Last Name

Username

José

Alvarez

jalvarez

Paula

Anderson

panderso

Vivek

Kundra

vkundra

Sylvia

Goldstein

sgoldste

And, here are examples of repeated name exceptions:

First Name

Last Name

Username

José

Alvarez

jalvarez

José

Alvarez

jqalvare

Paula

Anderson

panderso

Paula

Anderson

pmanders

If your users have no middle names, then you can work backward from the end of the alphabet:

First Name

Last Name

Username

Paula

Anderson

panderso

Paula

Anderson

pzanders

Paula

Anderson

pyanders

Paula

Anderson

pxanders

You can certainly develop your own naming convention, but I’ve seen this one work at other enterprises, so it’s worth a mention here. I’ve also seen other companies use a number system to avoid duplicate usernames. For example, if you have two users named Vivek Kundra, the first one created on a system is created as vkundra and the second Vivek Kundra is vkundra2.

The whole point of a naming convention is to develop a system and then stick to it. Having a naming convention prevents administrators from creating random user accounts that could duplicate other accounts.

For example, suppose you create a user account for Sylvia Goldstein as sgoldste and another system administrator creates an account as sylviag. You now have a security problem because you have a single user with two accounts on the same system, plus the problem of user account sprawl.

User account sprawl is not a minor issue. Suppose that Sylvia logs in and uses both accounts to create files, install software, and possibly create dependencies for groups that she’s a member of. Now, the problem is far more complex than simply moving files from one account to the other, changing user and group ownership, and removing the lesser-used account from the system.

If the administrators of the system had a naming convention, the second administrator wouldn’t have created the second account. Next, I discuss creating an account retention policy, an additional method of preventing account sprawl.

Creating account retention policies

An account retention policy is a good security practice as well as a good housekeeping practice. Nothing flags security scans quite like active accounts that no one has accessed in more than 90 days. Some highly secure environments will force a password change every 45 days, disable accounts after 90 days of inactivity, and remove accounts that remain inactive for six months. This type of aggressive account retention policy won’t fit every environment but it certainly protects and challenges users of sensitive systems to maintain their accounts or let them go.

I’m not suggesting that you adopt a policy as strict as this one. Still, you should have some type of retention policy in place that you provide to your users as part of an overall security and employment education program.

Your account retention policy should be a written policy as well as a system-level policy. By system-level policy, I mean that you must configure system-wide settings for inactive account lockout and removal.

Changing inactive account status

To work toward a system solution, audit the default value for the number of days after a password expires that the system disables the account, which is also known as the INACTIVE variable.

$ sudo useradd -D | grep INACTIVE

INACTIVE=-1

The -1 means that there’s no default inactive value set.

Set the inactive value according to your company’s security policy. If you have no security policy that addresses this value, then set it to 15 days. Some sysadmins set this value to 30. The purpose of this setting is to disable inactive accounts and protect system security.

$ sudo useradd -D -f 15

This sets the system-wide default inactive value to 15 days. If a user receives the message that they need to update their password, they have 15 days to do so before the system disables the account.

$ sudo useradd -D | grep INACTIVE

INACTIVE=15

If a user attempts to login to a system and fails, you can check their account status:

$ sudo passwd -S <username>

$ sudo passwd -S ndavis

ndavis PS 2022-02-13 0 99999 7 15 (Password set, SHA512 crypt.)

This user is in good standing. Perhaps they had the wrong system or typed in the wrong password. However, another user, asmith, contacted you complaining that they couldn’t login to the same system.

$ sudo passwd -S asmith

asmith LK 2022-02-12 0 99999 7 15 (Password locked.)

User asmith is locked out of the system. You will have to unlock their account before they can login again.

$ sudo passwd -u asmith

passwd: Success

The user may login again. Next, I will demonstrate how to take account protection further with the chage command.

Protecting user accounts

The chage command has several options for further protecting your user accounts by forcing regular password changes, setting a minimum password change duration, and so on. In this section, I show you how to alter those settings to protect your users and your systems.

There are three parameters that I change for all users on the systems that I manage. I change them according to company policy if one exists, otherwise, I change them to my “defaults,” which are:

- Inactive: 15

- Minimum number of days between password changes: 1

- Maximum number of days between password changes: 90

$ sudo chage -m 1 -M 90 --inactive 15 asmith

$ sudo chage --list asmith

Last password change                               : Feb 13, 2022

Password expires                                   : May 14, 2022

Password inactive                                  : May 29, 2022

Account expires                                    : never

Minimum number of days between password change     : 1

Maximum number of days between password change     : 90

Number of days of warning before password expires  : 7

This sets the user’s password to expire every 90 days. If they don’t change it their account will be disabled 15 days after the change password date. If they do change their password when prompted, they won’t be able to change it again until one day later.

You can only change one user at a time using the above chage command. This can become cumbersome if you have more than a few users on a few servers. A scripted version that covers all users is far more efficient. I’ve provided my script below.

#!/bin/bash

egrep ^[^:]+:[^\!*] /etc/shadow | cut -d: -f1 | grep -v root > user-list.txt

for user in `more user-list.txt`

do

chage -m 1 -M 90 -I 15 $user

done

The script must be run as root to use the chage command.

In the next section, I demonstrate how to retire an unused group account.

Retiring group accounts

Group accounts that remain on the system once they’re empty and inactive is a common source of account sprawl, but one that’s easily remedied. The groupmems command can save you a lot of time and frustration by answering who, if anyone, is a member of a specific group.

$ sudo groupmems -g operations -l

ajones bhaas

$ sudo groupmems -g hr -l

asmith

The groupmems command options used in the example are (-g) for the group name and (-l) for list.

If there are no group members, the command returns nothing.

$ sudo groupmems -g engineering -l

$

On this system, the engineering group should be retired (removed from the system). But, before removing the group, you must find out if the engineering group left any files that they own.

$ sudo find / -group engineering

find: ‘/proc/3368/task/3368/fd/7’: No such file or directory

find: ‘/proc/3368/task/3368/fdinfo/7’: No such file or directory

find: ‘/proc/3368/fd/6’: No such file or directory

find: ‘/proc/3368/fdinfo/6’: No such file or directory

/shared/engineering

/shared/engineering/one

/shared/engineering/two

/shared/engineering/three

/shared/engineering/four

/shared/engineering/five

/shared/engineering/six

/shared/engineering/seven

/shared/engineering/eight

/shared/engineering/nine

/shared/engineering/ten

You should transfer the files to another user or change ownership to root to secure them and then remove the empty group account.

$ sudo groupdel engineering

You have successfully removed the engineering account. You can recreate it in the future if you need to.

The next section departs from sprawl discussions to monitoring your system’s health.

Monitoring system health

System monitoring doesn’t explicitly fall under the sprawl umbrella, but it is part of maintaining system health. There are dozens of commercial monitoring tools that you can purchase, but sysstat is a free one native to Linux. It’s easy to install, and on Red Hat-based systems, it self-configures and begins collecting data immediately. On Debian-based systems, you have to enable sysstat to collect data manually.

To enable sar to begin collecting data, edit the /etc/default/sysstat file and change the line that reads:

ENABLED="false"

To:

ENABLED="true"

And restart the sysstat service:

$ sudo service sysstat restart

Give the system some time to begin generating activity reports. You’ll learn how to generate activity reports later in this chapter.

The sysstat package contains new binaries for checking performance and formatting system statistics information. The following table lists the binaries and their functions. I copied the descriptions of each binary from their respective man pages.

Binary

Name

Description

cifsiostat

CIFS Statistics

The cifsiostat command displays statistics about read and write operations on CIFS filesystems.

iostat

CPU and device I/O Statistics

The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates.

mpstat

Processor-related Statistics

The mpstat command writes to standard output activities for each available processor, processor 0 being the first one.

pidstat

Task Statistics

The pidstat command is used for monitoring individual tasks currently being managed by the Linux kernel.

sadf

sar Data Formatting

The sadf command is used for displaying the contents of data files created by the sar command. But unlike sar, sadf can write its data in many different formats (CSV, XML, etc.)

sar

System Activity Reporter

The sar command writes to standard output the contents of selected cumulative activity counters in the operating system.

tapestat

Tape Statistics

The tapestat command is used for monitoring the activity of tape drives connected to a system.

If you’re familiar with the vmstat command, you know how the other “stat” commands work. For example, vmstat, which is on every Linux system, requires that you provide a time delay interval between snapshots in seconds and a specific number (count) of snapshots, with five being the typical value used.

vmstat [options] [delay [count]]

The first run of the vmstat command gives averages since the last reboot.

$ vmstat 5 5

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

3  0  44616 343888    244 334676    0    0     1     1    1   24  0  0 100  0  0

0  0  44616 343768    244 334676    0    0     0     0   48   91  0  0 100  0  0

0  0  44616 343768    244 334676    0    0     0     2   53  104  0  0 100  0  0

1  0  44616 343768    244 334676    0    0     0     4   78  141  0  0 100  0  0

0  0  44616 343768    244 334676    0    0     0     0   75  141  0  0 100  0  0

This vmstat command ran every five seconds with five snapshots. The other “stat” commands work the same way. You provide a delay and a count. The following is a iostat command example with two snapshots and a five-second delay.

$ iostat 5 2

Linux 4.18.0-348.7.1.el8_5.x86_64 (server1)   02/13/2022   _x86_64_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

0.01    0.00    0.24    0.01    0.00   99.74

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

sda               0.16         0.98         1.34    1592993    2187576

scd0              0.00         0.00         0.00          1          0

sdc               0.00         0.01         0.00       8975       2786

sdb               0.00         0.00         0.00       2921       2048

dm-0              0.17         0.94         1.33    1529818    2170543

dm-1              0.02         0.02         0.06      38232      94228

loop0             0.00         0.00         0.00       1192          0

loop1             0.00         0.00         0.00        253          0

loop2             0.00         0.00         0.00       1202          0

dm-2              0.00         0.00         0.00       1223       2048

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

0.00    0.00    0.50    0.00    0.00   99.50

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

sda               0.00         0.00         0.00          0          0

scd0              0.00         0.00         0.00          0          0

sdc               0.00         0.00         0.00          0          0

sdb               0.00         0.00         0.00          0          0

dm-0              0.00         0.00         0.00          0          0

dm-1              0.00         0.00         0.00          0          0

loop0             0.00         0.00         0.00          0          0

loop1             0.00         0.00         0.00          0          0

loop2             0.00         0.00         0.00          0          0

dm-2              0.00         0.00         0.00          0          0

Gathering system activity reports

The sar command is comprehensive and versatile in its collection, reporting, and saving of system activity information. You can use it like the “stat” commands by supplying it with an interval and a count, you can issue an option such as -b to display I/O statistics, or you can combine the two to display your selected performance counter at a specific interval.

I’ll begin with an example of using sar with a switch. Staying with the -b example, here is a partial report of I/O statistics:

$ sar -b

Linux 4.18.0-348.7.1.el8_5.x86_64 (server1)   02/13/2022   _x86_64_  (1 CPU)

12:00:15 AM       tps      rtps      wtps   bread/s   bwrtn/s

12:10:15 AM      0.22      0.03      0.19      2.32      4.30

12:20:15 AM      0.10      0.00      0.10      0.00      0.79

12:30:15 AM      0.16      0.00      0.16      0.00      1.76

12:40:15 AM      0.08      0.00      0.08      0.00      0.5

***truncated output***

12:20:15 PM      0.08      0.00      0.08      0.00      0.73

12:30:15 PM      0.10      0.00      0.10      0.00      0.87

12:40:15 PM      0.08      0.00      0.08      0.00      0.78

12:50:15 PM      0.11      0.00      0.11      0.00      0.90

Average:         0.25      0.11      0.14      6.29      2.35

As you can see from the report, data display begins at midnight, which is why I had to truncate it. The following is an example of I/O statistical data using -b with a count (5) and a time delay (5 seconds):

$ sar -b 5 5

Linux 4.18.0-348.7.1.el8_5.x86_64 (server1)   02/13/2022   _x86_64_  (1 CPU)

12:57:13 PM       tps      rtps      wtps   bread/s   bwrtn/s

12:57:18 PM      0.00      0.00      0.00      0.00      0.00

12:57:23 PM      0.00      0.00      0.00      0.00      0.00

12:57:28 PM      0.00      0.00      0.00      0.00      0.00

12:57:33 PM      0.00      0.00      0.00      0.00      0.00

12:57:38 PM      0.00      0.00      0.00      0.00      0.00

Average:         0.00      0.00      0.00      0.00      0.00

And finally, you can issue sar with a count and an interval.

$ sar 5 5

Linux 4.18.0-348.7.1.el8_5.x86_64 (server1)   02/13/2022   _x86_64_  (1 CPU)

01:02:11 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle

01:02:16 PM     all      0.00      0.00      0.20      0.00      0.00     99.80

01:02:21 PM     all      0.00      0.00      0.40      0.00      0.00     99.60

01:02:26 PM     all      0.00      0.00      0.20      0.00      0.00     99.80

01:02:31 PM     all      0.00      0.00      0.20      0.00      0.00     99.80

01:02:36 PM     all      0.00      0.00      0.20      0.00      0.00     99.80

Average:        all      0.00      0.00      0.24      0.00      0.00     99.76

NOTE

The default output from the sar command is CPU statistics. To display other statistics, you must use a switch, such as -b for I/O data.

The next section demonstrates how to save sar data into files.

Formatting system activity reports

Displaying sar data enables you to see snapshots and performance from the last 24 hours, but what if you want to capture that data to a file in a specific format such as CSV or XML? The sadf command is available to you for this purpose. My personal favorite sadf command uses the -d switch to create a comma-separated file that’s perfect for ingesting into a database.

$ sadf -d

# hostname;interval;timestamp;CPU;%user;%nice;%system;%iowait;%steal;%idle

server1;600;2022-02-13 06:10:15 UTC;-1;0.01;0.00;0.24;0.01;0.00;99.74

server1;600;2022-02-13 06:20:15 UTC;-1;0.00;0.00;0.21;0.01;0.00;99.78

server1;600;2022-02-13 06:30:15 UTC;-1;0.01;0.02;0.23;0.01;0.00;99.74

server1;600;2022-02-13 06:40:15 UTC;-1;0.00;0.00;0.22;0.01;0.00;99.77

The system activity logs reside in /var/log/sa and have the label: sa(date). For an system activity log for the tenth of the month, the datafile is sa10. You can specifying an activity log datafile from another date:

$ sadf -d /var/log/sa/sa10 -- -d

# hostname;interval;timestamp;DEV;tps;rkB/s;wkB/s;areq-sz;aqu-sz;await;svctm;%util

server1;600;2022-02-10 06:10:15 UTC;dev8-0;0.35;8.46;2.17;30.81;0.00;0.62;0.81;0.03

server1;600;2022-02-10 06:10:15 UTC;dev11-0;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:10:15 UTC;dev8-32;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:10:15 UTC;dev8-16;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:10:15 UTC;dev253-0;0.30;8.35;2.27;35.42;0.00;0.90;0.79;0.02

server1;600;2022-02-10 06:10:15 UTC;dev253-1;0.03;0.11;0.00;4.00;0.00;1.00;0.87;0.00

server1;600;2022-02-10 06:10:15 UTC;dev7-0;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:10:15 UTC;dev7-1;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:10:15 UTC;dev7-2;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:10:15 UTC;dev253-2;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

NOTE

Sysstat logs reside in /var/log/sa on some Linux distributions but on others those logs reside in /var/log/sysstat.

The -- option tells the sadf command that the options that follow it are sar command options and not sadf options. In the command sadf -d /var/log/sa/sa10 -- -d, the -d option following the -- will display sar device information. Perhaps a less confusing example is to display network information from the sar command.

$ sadf -d /var/log/sa03 -- -n DEV

# hostname;interval;timestamp;IFACE;rxpck/s;txpck/s;rxkB/s;txkB/s;rxcmp/s;txcmp/s;rxmcst/s;%ifutil

server1;600;2022-02-10 06:10:15 UTC;lo;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:10:15 UTC;enp0s3;0.50;0.01;0.12;0.00;0.00;0.00;0.01;0.00

server1;600;2022-02-10 06:20:15 UTC;lo;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:20:15 UTC;enp0s3;0.42;0.00;0.10;0.00;0.00;0.00;0.01;0.00

server1;600;2022-02-10 06:30:15 UTC;lo;0.00;0.00;0.00;0.00;0.00;0.00;0.00;0.00

server1;600;2022-02-10 06:30:15 UTC;enp0s3;0.47;0.00;0.10;0.00;0.00;0.00;0.01;0.00

The -n is the sar option for network activity. DEV means the network device, such as lo (loopback), eth0, enp0s3, and so on.

The sadf options and sar options might confuse you but to help keep things straight, remember that sadf options are first in the command and then sar options follow the double hyphen. Refer to the man pages for all available options for both commands.

I’ve only scratched the surface of system health and this chapter could be an entire book by itself. Health monitoring is extremely important to maintaining SLAs for your customers. Keeping track of system health is an important system administrator task that affects capacity and performance, security, and standard system housekeeping. It’s so important that some large enterprises dedicate entire teams to it.

In Chapter 9, I cover non-health system monitoring and network inventory systems.

About the Author

Kenneth Hess has been a Linux system administrator for more than 25 years and a technology writer and journalist for the past 20 years. Ken has written hundreds of articles covering desktop Linux, virtualization, databases, and the general topic of system administration.

Table of Contents

1. Getting Started with Linux

Installing Linux

Preparing Your System for Linux

Downloading and Installing Linux

Getting to Know Your New Linux System

Learning the Command Line Interface

Commands for Filesystem Navigation

Starting, Rebooting, and Shutting Down a Linux System

Starting a System

Restarting a System

Shutting Down a System

Summary

2. Diving Deeper into the Command Line Interface

Working as a Regular User

Working as the Root User

Login as Root

su to Root

Use the sudo Command

Creating a sudoer

Setting and Modifying Permissions

Read, Write, and Execute

Numerical Permission Values

Group Permissions

Bringing Permissions Settings All Together

Changing File Permissions

Symbolic Mode

Numeric Mode

Default Permissions Explained: umask

Summary

3. Customizing the User Experience

Altering Home Directory Options

Login Versus Non-login Shells

/etc/bashrc

/etc/profile

.bashrc

.bash_profile

.bash_logout

The /etc/skel Directory

Customizing the Shell Prompt

Summary

4. Managing Users

Creating User Accounts

Adding Users with adduser

Adding Users with useradd

Modifying User Accounts

Adding a Supplementary Group

Changing the User Comments Field

Setting an Expiration (Expiry) Date on an Account

Changing a User’s Login Shell

User and Group ID Numbering Conventions

Removing User Accounts

Forcing Password Changes

Handling Service Accounts

Managing Groups Rather Than Users

Summary

5. Connecting to a Network

Plugging into a Network

Static IP Addressing

Dynamic IP Addressing

Networking and Security

Preparing a System for Network Connectivity

Pruning Your Systems

Securing Network Daemons

Summary

6. Installing and Uninstalling Software

Updating Your System

Applying Red Hat Enterprise Linux-Based System Updates

Applying Debian-Based System Updates

Installing Software from Repositories

Installing an Application

Uninstalling an Application

Installing and Uninstalling Individual Software Packages

Installing an Individual Software Package Manually

Uninstalling Individual Software Packages

Finding Package Dependencies

Installing Software from Source Code

Satisfying Prerequisites: Building a Development Environment

Download, Extract, Compile, and Install Your Software

Uninstalling a Source-Installed Software Package

Summary

7. Managing Storage

Administering Linux Storage

Disks

Filesystems

Mounting and mount points

Physical and logical volumes

Checking space

Swap space

RAM-based temporary space (ramfs, tmpfs)

Adding a New Disk to a System

Installing the disk

Prepping the disk for use

Implementing Logical Volumes

Decommissioning and Disk Disposal

Notification

Scream test

Power down

Disk wiping

Unracking and palletizing

Disposal

Summary

8. Maintaining System Health

Keeping your system clutter-free

Cleaning the /tmp directory

Making /home a livable space for everyone

Decluttering shared directories

Deduplicating files with fdupes

Tackling /home file sprawl with quotas

Patching your way to a healthy system

Patching a Red Hat Enterprise Linux-based system

Patching a Debian-based Linux system

Securing your systems

Maintaining user and group accounts

Setting up a naming convention

Creating account retention policies

Retiring group accounts

Monitoring system health

Gathering system activity reports

Formatting system activity reports

About the Author