Andrew Mallett

Salt Open

Automating Your Enterprise and Your Network

1st ed.

Andrew Mallett

Peterborough, UK

ISBN 978-1-4842-7236-7    e-ISBN 978-1-4842-7237-4

https://doi.org/10.1007/978-1-4842-7237-4

© Andrew Mallett 2021

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Apress imprint is published by the registered company APress Media, LLC part of Springer Nature.

The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.

To Lori: You were a flower so beautiful lighting the life of everyone you met and now so sadly missed as you were picked oh so early.

Introduction

For many years, we have seen the importance of being able to manage your servers and desktops. Configuration management systems are abounding in this arena with so many to choose from. For enterprises that need to automate the configuration of their varied network devices, the options have been few and far between.

The SaltStack project from Utah in the United States has created not only a world-beating configuration management suite with Salt, but has included the Salt Proxy subsystem that can allow your network devices to be upgraded to first-class citizens in the world of configuration management. Not wanting to be held back by outdated thought processes, SaltStack continued their growth into the cloud, and the free open source tools include Salt Cloud to manage the virtual machine instances, both in the public and private clouds. We now have one tool, Salt, to manage your Windows and Linux servers. Matching this, we have the single tool, Salt Proxy, to manage your network devices independent of their vendor and the single Salt Cloud tool to manage public and private clouds independent of the hypervisor. If your enterprise needs vendor-agnostic tools to manage the entire infrastructure, then SaltStack is very much your only answer.

In this book, we take you through the SaltStack suite using different Linux distributions allowing to experience it in a true multi-vendor playing field. Starting with configuration management with Salt, we implement the server-client architecture to manage your server and desktop systems, moving quickly forward into an agentless environment using Salt SSH. You are not limited with the possibilities presented by the Utah-based team. It is your choice to use agents or not. As the number of systems managed by Salt increases, we show how, using Salt Syndic, you can manage remote sites and load-balance connections to the main server. The book closes by showing how, using Salt and Salt Proxy, your network devices can have the same status as your servers, being managed as easily as everything else with the same agnostic approach.

By the time you reach the end of this book, you will have received information on the most important elements of SaltStack, being able to write your own Salt State (SLS) files in YAML (YAML Ain't Markup Language), Jinja, and raw Python. You will be able to set up true orchestration using reactors and beacons, ensuring that the system you have built becomes self-healing. You will have learned to use Salt SSH to connect to remote systems and even use it to deploy the Salt Minion on your device without ever needing to leave your main console. Culminating in managing Arista routers, you will have become the consummate professional with your entire estate managed using the Salt Open suite.

Table of Contents

Chapter 1: Understanding Salt and Configuration Automation

Partnerships and Open Source Projects

Open Source Configuration Management Projects

Salt

Ansible

Puppet

Chef

Salt Products

What Can Salt Do

Summary

Chapter 2: Installing Salt

Configuring the Lab Environment

Configuring the Salt Master

Installing Salt

Python Installer

Bootstrap Installer

Accepting the Minion Key

Review Minion Configuration

Everything Is Python!

Installing the First Independent Salt Minion

Salt-Key and Key Files

Minion Keys

Salt Master Firewall Requirements

Summary

Chapter 3: Adding and Removing Salt Minions

Salt Minion Configuration

Regenerating Salt Minion Keys

Accepting Keys

Supported Salt Minion Platforms

Summary

Chapter 4: Targeting Salt Minions

Salt Minion Targeting

Default Minion ID Globbing

Lists

Targeting Using PCRE

Targeting Using Grains

Targeting Nodegroups

Targeting Subnets

Targeting Using Pillar Data

Using Compound Matches

Discovering the Mechanics of Matching

Supported Salt Platforms

Summary

Chapter 5: Working with Remote Execution Modules in Salt Open

Execution Modules

Accessing Help on Execution Modules

Using Web-Based Help

Implementing Configuration Changes Using Execution Modules

Installing Software Packages

Having to Understand the Target System and Version Changes

Investigating Agnostic Behavior

Managing Services

Managing Files

Summary

Chapter 6: Writing YAML Files

YAML?

Gain Practice Writing YAML

Editing YAML Files

Using Nano

Using Vim

Persisting YAML Configuration for SLS Files

Using Graphical Editors/IDEs

Summary

Chapter 7: Writing Salt State Files

State vs. Flow

State File Location

Configuring File Roots

Creating Salt State Files

Configuring the Time Zone from a State File

Applying State Files

Previewing Actions

Salt Rendering Engines

Applying State Files Locally

Configuring Packages and Services

Configuring Files Using State Files

Debugging State Files

Using Salt Formulas

Summary

Chapter 8: Building an Effective State Tree

Salt Top File

Simple Top File

Viewing the Assigned States

Organizing Content

Debugging the State Tree

Summary

Chapter 9: Creating Reusable State Files

Working with Grains

Using Simple Jinja Logic

Streamlining Jinja Logic

Using YAML Map Files

Implementing Custom Grains

Creating a File System Structure

Saving Custom Grains

Implementing Custom Grains

Adding Reliability to Our Application Using Require Statements

Linking the init.sls

Using Pillar Data

Using Other Rendering Engines

Native Pure Performance

View Rendering Actions

Summary

Chapter 10: Implementing Reactors and Beacons

The Beating Heart of Salt: The Event Bus

Creating Schedules on Salt Minions

Creating Reactors

Investigating Events Needing Reactors

Reactor and State Configuration

Configuring Beacons

Creating the Beacon

Viewing Beacon Activity

Reacting to Beacons

Testing the Beacon

Debugging Reactors

Summary

Chapter 11: Using Salt SSH

Salt SSH

Vagrant and Password Authentication

Managing Nodes Using Salt SSH

Agentless Operation

Creating State Files

Summary

Chapter 12: Deploy Virtual Machines Using Salt Cloud

Installing Salt Cloud

Configuring Salt Cloud

Cloud Providers

Cloud Profiles

Managing Virtual Machine Instances Using Salt Cloud

Provisioning Virtual Machines

Deleting Instances

Summary

Chapter 13: Scaling Configuration Management Using Salt Syndic

Danger: Redevelopment in Progress

Rebuilding the Lab Systems

Installing the Salt Master

Installing Salt Syndic

Installing the Additional Salt Minion

Verify the Salt Syndic Infrastructure

Download the Formula and Configure File Roots

Deploy the Vim Formula

Summary

Chapter 14: Automating Network Infrastructure with Salt Proxy

AWS Network Virtualization

Salt Cloud Profile

Configuring the Arista Router

Salt Proxy Topology

NAPALM: Network Agnostics

Installing NAPALM

Using the NAPALM CLI

NAPALM Python Scripts

Configuring Salt Proxy

Salt Proxy Configuration

Configuring Salt Pillar

Using GPG to Encrypt Sensitive Data

Adding the Cipher Text to Pillar

Running Salt Proxy

Starting Salt Proxy Initially

Restart Salt Proxy in the Foreground

Managing Network Devices

Reading Router Configuration

Writing Router Configuration

Gaining Help

The Super Proxy

Summary

Index

About the Author

Andrew Mallett

is a well-known Linux consultant and trainer. His YouTube channel has over 65K subscribers and more than 1000 videos. Working mainly online now, Andrew has authored courses on both Pluralsight and Udemy and regularly teaches classes online to a worldwide audience. Andrew is familiar with Linux and UNIX and has worked with them for over 20 years. Scripting and automation is one of his passions as he is inherently lazy and will always seek the most effective way of getting the job done. The Urban Penguin, his alter ego, is a UK-based company where his work is created from and currently employs five people.

About the Technical Reviewer

Patrick

is passionate about the things that he most cares about: family and the ability to make a difference in what he does. He is excited about technologies that open new possibilities and overcome challenges that our world faces today. As an open source advocate, he has been known to use a variety of open source projects in the most remote regions of Africa.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_1

1. Understanding Salt and Configuration Automation

Andrew Mallett[1]

(1)

Peterborough, UK

We all know and use salt, that condiment that sits on every table and within every packet of crisps adding that desirable taste tuned to exactly what we require. When we talk about Salt in terms of automating our IT enterprise, we are less concerned with sodium chloride and more focused on the tuning element. We need to “salt” our systems so they are tuned to perfection and meet the desired state that we seek.

Founded in 2011 by Thomas Hatch, the SaltStack project developed the configuration management system Salt, based on Python and YAML states. This book will guide you through managing your servers and switches using Salt Open from SaltStack. Salt is available as an open source product from the SaltStack project, or where support is required, there is SaltConfig, part of the vRealize suite from VMware who acquired SaltStack in October 2020. In the main, SaltConfig is the same as Salt Open but with the addition of a web front end, which helps especially where reporting is concerned. The big advantage with the commercial version is the world-class support from VMware, often essential in an enterprise.

To be clear on terminology that we will use throughout this book

- SaltStack: Refers to the open source project based in Utah and is now owned by VMware.

- Salt: When used with an uppercase S, we refer to the Salt Open configuration management system from SaltStack.

- salt: When used with a lowercase s, we refer to the salt command, which is executed on the Salt Master (server) to publish jobs to Salt Minions (clients).

Note   The SaltStack project was founded by Hatch in the US state of Utah. The project remains there despite the acquisition, and Thomas Hatch is now employed by VMware and based in Utah within the project team. This guarantees stability in both the open source project and the ongoing enterprise versions of SaltConfig for VMware.

Partnerships and Open Source Projects

Acquiring additional quality projects is a major goal of many large enterprise IT organizations. It is not just SaltStack that has seen their jewels valued highly in the marketplace. VMware is responding to the success that has been seen in Red Hat’s purchase of Ansible, originally developed by Michael DeHaan. Red Hat has huge resources that they have made available to the good folks at the Ansible open source project, giving Ansible a huge boost and a lot of market share in IT automation. Seeing that success, VMware is hoping to replicate similar benefits to the SaltStack project.

Open Source Configuration Management Projects

As we have already mentioned, we have SaltStack and Ansible, which are widely used in configuration management. Having choice is always a good thing and keeps competition good between the vendors. Ultimately, that is a benefit to us as consumers. We are by no means limited to just these two systems either. There are more that you can choose from, and it is always wise to make an informed choice. Bearing that in mind, we list the major players in configuration management.

Salt

As we have mentioned, Salt is a configuration management system created and maintained by the SaltStack project now owned by VMware. Based in Utah, the project has fresh funding to help it move forward and maintain its market share and prominence. Salt uses a server and client architecture where the server is known as the Salt Master and the client is the Salt Minion. Both the server and client are written in Python, as is the complete suite. With the focus on Python, the extensibility of the product is immense as Python is so widely known and supported. From its initial inception by Thomas Hatch in 2011, Salt has been designed to be incredibly quick and scalable.

The speed element comes from reduced latency. Salt Minions connect to the Salt Master via a high-speed message bus, ZeroMQ. Jobs are published in the form of Salt State files to the message bus and collected by the client immediately. There is no time delay and zero latency. The Salt Master only publishes the State file to the message bus. These files are written in YAML and take little resource to be delivered via the network to the waiting Salt Minions. On receiving the State file, the tasks are executed on the Salt Minion providing the scalability factor as the work is distributed to the Minions with little resource utilization on the Salt Master.

The Salt Master can support thousands of connected devices without needing huge resources to be expended on the system. Connections to the message bus do not consume huge resources, and with the jobs being executed remotely, there is little or no CPU utilization.

Ansible

Dating back to 2011, Ansible and Salt share the same vintage and, additionally, both systems utilize Python and YAML. The architecture of Ansible, though, does differ, where Ansible does not require clients or long-running services. In fact, Ansible only needs to be installed on the controller node and connects to the managed nodes using SSH. There is no server service or client service other than SSH. The managed nodes require Python to be installed, but there is no other specialist agent service that is needed. Both Ansible and Salt use Python, but Ansible delivers the required Python module to the managed nodes where it is executed. This puts more load on the controller node as not only does the controller need to deliver the Python module but it is required to read and interpret the State file. These State files are known as Playbooks in Ansible terminology. Both Salt States and Playbooks are based on YAML, making it easy to migrate between systems.

Puppet

The initial release of Puppet from Puppet Labs was in 2005, much earlier than either Salt or Ansible. Like Salt, Puppet makes use of a client-server architecture, but clients do not maintain a constant connection to a message bus. The clients check in on a regular basis that can lead to latency in configuration while we wait for the client to check in. Based on Ruby rather than Python, Puppet continues to be different. Strangely enough, Python predates Ruby although Puppet predates both Salt and Ansible. Ruby was first released in 1995 and Python in 1991. The State files are known as Puppet Manifests and are in their own format, which is not dissimilar to Ruby.

Chef

Founded in 2008 in Seattle by Adam Jacob. Also based on Ruby and utilizing a client-server architecture. These are obvious similarities with Puppet. But as well as the year of the pandemic, 2020 was also the year for acquisitions. Progress made the purchase of Chef in the same year that Salt was acquired by VMware. I think we all made a few online purchases during the lockdown caused by the pandemic, but really USD 220M for Chef is a big purchase. The required configuration of a client, or State, is created in a Cookbook written in Ruby format.

Note   VMware has not declared the financial details of the SaltStack acquisition.

Salt Products

Included with Salt Open, yes, the free suite, we have a range of tools that we can use to help automate the IT infrastructure that we need to manage. Each of these tools brings its own unique attributes to help Salt become one of the most powerful configuration management and orchestration systems available:

- salt: Used on the Salt Master, the command salt is a Python script used to publish Salt State (SLS) files to the message bus for Salt Minions.

- salt-ssh: Where a long-running agent process is not possible or desirable on the managed nodes, salt-ssh can be used to remotely execute SLS files on the managed nodes.

- salt-proxy: The salt-proxy runs on a standard Salt Minion and can receive commands destined for your network infrastructure devices such as routers, switches, and WAPs. These devices typically cannot have the client installed but can still be managed via the salt-proxy. The proxy will connect to the managed device via REST or SSH depending on the system.

- salt-syndic: Used to traverse a slower network connection, the salt-syndic will run on a Salt Minion in a remote office. Other Minions will connect to the Syndic, which will forward requests to the Salt Master across the network connection.

- salt-cloud: Used to manage your cloud infrastructure. This could be local cloud services such as VMware ESX servers or public cloud entities such as AWS and Google Cloud. The single command salt-cloud can be used to provision and decommission instances in any supported cloud, no matter if private or public.

- Reactors: Reactors are configured on the Salt Master to read events on the message bus and mitigate the events by executing State files. Making up part of the orchestration capabilities of Salt, reactors are a crucial feature of Salt and a major differentiator.

- Beacons: Part of the Salt Minion configuration, beacons can be used to monitor conditions on the Minion to alert the reactor system if required, extending the orchestration features in Salt.

- Salt Pillar: Pillar data allows secure storage of variables and configuration required by Minions. The data is stored centrally on the Salt Master and encrypted when sent to Minions that require the data.

- Salt Mine: A central database of information that can be shared between Minions. Salt Minions mine the data from the Salt Master where it is stored.

- Salt States: SaLt State or SLS files represent the desired state or configuration of our nodes. The default renderer of these files is #!jinja|yaml, meaning that we render Jinja template data first and output to YAML for further rendering. Many State files will be pure YAML without the need of templating, but for some files, this will be needed. Other rendering engines are available, and a Salt State file could be written in pure Python if required. The file extension is “.sls,” and we will become very familiar with these files.

- Salt Grains: Grains are specific facts about the Salt Minion or managed device. One of the many important grains is os_ family, helping Salt to identify the running system and cater for differences between package names, service names, configuration locations, and so forth. By identifying the os_family, we can install the correct software package for the Apache web server, as an example. The required package differs between systems. On Red Hat we need the httpd package; on Debian we need the apache2 package; and, if installing on Windows, we would need the apache-httpd. Salt handles these differences by reading Grains from the target system.

- Salt Formula: A formula in Salt is a prewritten collection of State files for a given task. Community formulas are stored in GIT, https://github.com/saltstack-formulas, and they can be downloaded to the Salt Master or accessed by the Salt Master in place using the GIT file system available in Salt. As an example, you might download the vim-formula that can be used to install and configure the Vim text editor to work correctly with Salt State files. You can find the formula here: https://github.com/saltstack-formulas/vim-formula.

What Can Salt Do

What can Salt do? This is what we are going to learn as we progress further into this book, but as a quick taster, even before we have Salt up and running, we can look at the capabilities it offers us and, crucially, why we want to use this.

Configuring a single system is relatively easy and does not offer too many challenges. We only need to learn one set of commands, and very often, we will only need to run those commands once to configure the system. Unfortunately, life in the real world is unlikely to be as simple. As an administrator, we are likely to be working with more than one Linux distribution and version, perhaps Red Hat Enterprise Linux 8, CentOS 7, and Ubuntu 20.04 systems. All of which will require the execution of different commands and configuration files, services, and software packages. Throwing something else into the mix, we will almost certainly need to include Windows servers in our configuration needs. A single State could install and configure the Apache web server across all of these systems. You could even download a formula to achieve this: https://github.com/saltstack-formulas/apache-formula. Listing 1-1 is an extract from the Apache formula illustrating how we can cater for differences between systems that make up the Red Hat family and those in the Windows family.

Windows:

pkg:

name: apache-httpd

service:

name: apache

RedHat:

pkg:

name: httpd

service:

name: httpd

Listing 1-1  Extract of YAML from the Apache Formula

We are not limited to managing traditional server and desktop operating systems. Salt is one of the most powerful systems in managing network infrastructure devices. These too have their differences, but Salt can cater for those differences in much the same way. Although network devices connect to a Salt Proxy rather than directly to the Salt Master, we are still able to use Salt Grains to identify the device characteristics. In Listing 1-2, we have a template file that can be used to configure the hostname of network devices. The settings differ between Juniper systems running JunOS and Arista systems running EOS. With the one file, we are able to configure either system. It is this agnostic view of your enterprise that makes Salt so desirable as a configuration management system.

{%- set vendor = grains.vendor|lower %}

{%- set hostname = pillar.proxy.host %}

{%- if vendor == 'juniper' %}

system {

host-name {{hostname}}.lab;

}

{%- elif vendor in ['cisco','arista'] %}

hostname {{ hostname }}.lab

{%- endif %}

Listing 1-2  Catering for Differences Between Juniper and Arista Devices

Summary

Do you know what? You are amazing. You now know what Salt is and where and when it was born. This foundational knowledge in configuration management and the background of Salt and SaltStack is important to help you understand future chapters where we start to introduce more configuration to you.

If you are in the market for a configuration management system, it is important that you have an understanding of the products that are available. You have been very lucky that we have been able to introduce to you not only Salt but competitive products such as Ansible, Puppet, and Chef and how they match up to Salt. Of course, our focus is Salt within this book. The title may have given you a clue on that, but I am very familiar with both Ansible and Puppet. You may also be interested in my Apress-published book Red Hat Certified Engineer (RHCE) Study Guide, which looks at Ansible automation on RHEL 8.

The product range is immense within Salt from SaltStack. All as part of the Salt Open suite, we have not only the Salt Master and Salt Minion but salt-ssh, allowing us to operate more like Ansible where we need to configure systems that do not have the Salt Minion installed but are accessible via SSH. We also have the ability to seamlessly manage our private and public cloud instances with the single CLI tool: salt-cloud. Enabling us to manage the network infrastructure centrally, we have salt-proxy, which runs on our Salt Minions that allows the configuration to traverse to the network device.

With new investment in the Utah-based SaltStack project from VMware, Salt is now prepared to take on the world and provide leading cloud, configuration management, and orchestration to everyone. Now that you are reading this book, you are preparing yourself to take on the world of administration and DevOps with world-class tools and newly forged skills that you are now learning.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_2

2. Installing Salt

Andrew Mallett[1]

(1)

Peterborough, UK

To be able to learn Salt, perhaps we are best advised to install the product, building a fine foundation for our education. In this chapter, we will establish the lab setup being used in the book. You, naturally, do not need to use exactly the same setup, but you get to see and understand what I am using. Once we have the lab complete, we can identify the different options available to install Salt. We will be using the bootstrap installer script to configure both the Salt Master and Salt Minion. Later we will see how we can add repositories to other systems remotely using salt-ssh, adding a third system to manage with Salt. When we look at salt-proxy, I will be using a local Aruba physical switch running ArubaOS and a virtual Arista router running EOS. These are optional if you do not want to manage network devices.

As a quick guide to this chapter, we have put together the following list of topics that you are going to learn if you can progress to the end of the chapter:

- Configure the lab environment.

- Configure network time synchronization on the Ubuntu 20.04 system that we will use as the Salt Master.

- Install the Salt Master and Salt Minion on the master Ubuntu 20.04 system, and install a Salt Minion independently on the CentOS 8 Stream.

- Authorize Salt Minions using salt-key.

- Implement basic tests to check the operation of Salt.

- Review firewall requirements for the Salt Master.

Configuring the Lab Environment

For the demonstration used in this book, we will be using VirtualBox and Vagrant . You can use systems of your own choice, but we have chosen VirtualBox and Vagrant as they are available on most platforms without charge. We do not cover the installation of either VirtualBox or Vagrant, but instructions can be found for your platform from the supplier. For VirtualBox this is Oracle (www.virtualbox.org/wiki/Downloads), and for Vagrant it is HashiCorp (www.vagrantup.com/):

- Salt Maste r: This will be an Ubuntu 20.04 system with two NICs, one using NAT to the Internet and a private host-only network allowing access to the host and other lab systems.

- Independent Salt Minion : This will run on CentOS 8 Stream, again with the same NIC configuration.

- Second independent Minion: This will be used later in the course as a target for Salt SSH. The Salt Minion will be installed remotely on this system. It will use the same NIC configuration as the other systems and is based on Ubuntu 20.04.

- Virtual Arista router : Running in VirtualBox, we will step you through the setup of this system toward the end of the book when we demonstrate Salt Proxy. The same NIC system is used.

- Physical Aruba switch : We can manage physical as well as virtual systems, and I demonstrate management of a local physical switch.

Using Vagrant from HashiCorp, we can use a Vagrantfile defining the virtual machines. The file configures all systems other than the Arista router. We will step you through the process of setting up your virtual router later in this book. We recommend creating a new subdirectory to place the file in and then creating a file called Vagrantfile (in the exact same case) within the newly created directory (Listing 2-1).

# -*- mode: ruby -*-

# vi: set ft=ruby :

Vagrant.configure("2") do |config|

config.vm.define "master" do |master|

master.vm.box = "ubuntu/focal64"

master.vm.network "private_network", ip: "192.168.33.100"

master.vm.hostname =  "master"

master.vm.provider "virtualbox" do |vb|

vb.memory = "1024"

end

end

config.vm.define "minion1" do |minion1|

minion1.vm.box = "centos/stream8"

minion1.vm.network "private_network", ip: "192.168.33.101"

minion1.vm.hostname = "minion1"

minion1.vm.provider "virtualbox" do |vb|

vb.memory = "1024"

end

end

config.vm.define "minion2" do |minion2|

minion2.vm.box = "ubuntu/focal64"

minion2.vm.network "private_network", ip: "192.168.33.102"

minion2.vm.hostname = "minion2"

minion2.vm.provider "virtualbox" do |vb|

vb.memory = "1024"

end

end

end

Listing 2-1  Vagrantfile Configuring Lab Systems

To start the virtual systems, we can move to the directory where we have stored the Vagrantfile, and we can use the command vagrant up.

Configuring the Salt Master

Prior to the installation of Salt, we can configure the Salt Master host system, the Ubuntu 20.04 virtual machine on which we will install the Salt Master. Using Vagrant, we work at the command line of our host system. We are using macOS as our host system. Ensuring that we are working in the directory where we had defined the Vagrantfile, we are able to list the running systems and connect using SSH (Listing 2-2).

$ cd vagrant/salt-book

vagrant/salt-book $ vagrant status

master                    running (virtualbox)

minion1                   running (virtualbox)

minion2                   running (virtualbox)

vagrant/salt-book $ vagrant ssh master

vagrant@master:~$

Listing 2-2  Connecting to the Master Virtual Machine

As always, especially when we are configuring new systems for services, we update the currently installed software. The Salt Master system we are using is based on Ubuntu 20.04 and uses the apt package manager. From the command line of the Salt Master system, we can execute the software update (Listing 2-3).

vagrant@master:~$ sudo apt update

vagrant@master:~$ sudo apt -y upgrade

vagrant@master:~$ hostnamectl | grep 'Operating'

Operating System: Ubuntu 20.04.2 LTS

Listing 2-3  Updating the Salt Master

Installing the Salt Master package on our system will require that we have accurate time. Certificate requests are signed by the Salt Master, and we need an assurance that date stamps are accurate. This is especially important on the Salt Master, which signs these requests. It is not difficult or resource intensive to configure a time client (systemd-timesyncd) on the Salt Master before installing Salt, which initiates the certificate system that we use. Initially, we can check the status of time on our system using the timedatectl command (Listing 2-4).

vagrant@master:~$ timedatectl

Local time: Fri 2021-05-14 12:16:25 UTC

Universal time: Fri 2021-05-14 12:16:25 UTC

RTC time: Fri 2021-05-14 12:16:23

Time zone: Etc/UTC (UTC, +0000)

System clock synchronized: no

NTP service: inactive

RTC in local TZ: no

Listing 2-4  Checking the Time Status of the Salt Master

This is a very thorough command and can identify many issues for us. Looking at the output line by line, we can analyze the results and configuration of the system:

- Local time: The time displayed on the system.

- Universal time: The time displayed from the UTC time zone.

- RTC time: The real-time clock or hardware clock on the local system.

- Time zone: The configured time zone for the system. This is a downloaded VM and set to UTC.

- System clock synchronized: This indicates that we are not synchronized with network time servers.

- NTP service: There is no active time client on the system.

- RTC in local TZ: This should always be set to no in Linux. The real-time clock or hardware clock should be set to UTC time no matter the physical location of the system. The time zone is an offset to UTC time, and the local time will be adjusted to the offset. This will not happen if you set the RTC to the local time zone and does not easily allow for daylight savings time or change of time zones due to travel.

We need to set the time zone to match our location. For us, this will correct the local time, which is an hour out due to daylight savings. We can also enable time services from the command timedatectl (Listing 2-5).

#Working as root

root@master:~# timedatectl set-timezone Europe/London

root@master:~# timedatectl set-ntp true

root@master:~# timedatectl

Local time: Fri 2021-05-14 13:34:02 BST

Universal time: Fri 2021-05-14 12:34:02 UTC

RTC time: Fri 2021-05-14 12:34:03

Time zone: Europe/London (BST, +0100)

System clock synchronized: yes

NTP service: active

RTC in local TZ: no

Listing 2-5  Configuring Time on Systemd-Based Systems

The output is now correct indicating the correct time zone and that we are synchronized with NTP. Note also we have not needed to change the RTC; it remains in the UTC time zone with local time adjusting to the current time zone. Enabling the NTP client using timedatectl, we are enabling the systemd-timesynd.service.

Installing Salt

Salt utilities are all built in Python, which does mean that you can download the packages using pip3, the Python package installer. Natively, Ubuntu does not have any of the Salt packages within its own Ubuntu repositories, but we can add the SaltStack software repository using the bootstrap installer. This is achieved seamlessly for us as administrators.

Python Installer

Installing Python packages with pip3 is simple and quick. All of the SaltStack tools and services are Python scripts making this a viable option. The downside is that the systemd service units are not created, meaning that we have to create the service units to automate the start of both the Salt Master and Salt Minion services. For this reason, we tend toward the bootstrap installer. Should you want to install using pip3, the process is listed here for Ubuntu 20.04, but we strongly suggest using the bootstrap installer:

vagrant@master:~$ sudo apt -y install python3-pip

vagrant@master:~$ sudo pip3 install salt

Bootstrap Installer

Perhaps a more conventional mechanism to install Salt packages is utilizing the bootstrap installer. This is a Linux shell script that will add the SaltStack repository and GPG (GNU Privacy Guard) key and then will proceed with the installation of either the Salt Master or the Salt Minion (or both). The details of downloading the installer can be found at https://repo.saltstack.com/. The following shows how we can download the installer using curl:

vagrant@master:~$ curl -o install_salt.sh -L https://bootstrap.saltproject.io

vagrant@master:~$ sh install_salt.sh --help

On this system, we want to install the Salt Master, utilizing the option -M. This will also install the Salt Minion. If we did not need the Salt Minion to be installed on the Salt Master, that is to say the Salt Master does not need to be managed, we would include the option -N. We also include the option -L to install Salt Cloud, which we can use later in the book. Once installed, we need to configure the name of the Minion, known as the Minion ID. If it is not configured, the system FQDN (fully qualified domain name) is used. We may prefer to set the Minion ID during the install utilizing the option -i followed by the ID we want to use. This is not the only configuration that we need for the Salt Minion. We need to set the location of the Salt Master to connect to. This can be resolved from DNS if there is a host record “salt.” If that is not possible, then we can pass the option -A followed by the IP address or hostname of the Salt Master. Finally, we need to consider the version that we want to install. The default is the latest stable release. The Salt Master should always have the same version as the Minions or later (Listing 2-6).

vagrant@master:~$ sudo sh install_salt.sh -M -L -i master

-A 192.168.33.100 stable 3003

vagrant@master:~$ salt --version

salt 3003

Listing 2-6  Installing Salt on the Master Ubuntu System

In this example, we are setting the Minion ID to master. The Salt Master’s address is 192.168.33.100. Use the correct address of your Salt Master, and we are installing the stable version of 3003. This would have been the default had we not added the stable option at the time of the installation. Later versions may be available when you come to install Salt. Especially when installing the Master, we recommend using the latest available. If installing a new Minion, we recommend selecting the same version running on the Salt Master .

Accepting the Minion Key

Each Salt Minion will connect to the Salt Master presenting its own public key for signing. The Salt Master needs to accept this request and return the signed key to the Salt Minion authorizing its connection. This signed key is used to authenticate the Salt Minion to the Salt Master. On the Salt Master, we can use the command salt-key to manage these keys (Listing 2-7).

Note   Even though the Salt Master and the Salt Minion, in this case, are on the same system, we still need to authorize the request.

vagrant@master:~$ sudo salt-key -L

Accepted Keys:

Denied Keys:

Unaccepted Keys:

master

Rejected Keys:

vagrant@master:~$ sudo salt-key -y -a master

The following keys are going to be accepted:

Unaccepted Keys:

master

Key for minion master accepted.

Listing 2-7  Listing Keys and Signing Requests

Now that the key is accepted, the Salt Minion named master can be managed by Salt.

Review Minion Configuration

Using the options -i and -A, we configured the Salt Minion with the Minion ID and Salt Master address, respectively. The configuration directory for Salt is /etc/salt/. The minion_id file is used to name the Minion if set before the Salt Minion is started. The Salt Minion is configured in the file minion.d/99-master-address.conf (Listing 2-8).

vagrant@master:~$ cat /etc/salt/minion_id

master

vagrant@master:~$ cat /etc/salt/minion.d/99-master-address.conf

master: 192.168.33.100

vagrant@master:~$ sudo salt '*' test.ping

master:

True

Listing 2-8  Verify Salt Minion Configuration

Configuring the Salt Minion during the installation means that it is ready to connect to the Salt Master immediately. In many instances, the Salt Master can start without any further configuration. The final command that was issued in Listing 2-8 uses the command salt on the Salt Master to test connectivity to all Salt Minions. Currently we only have the one. This is not an ICMP ping but a simple request to the Salt Minion for a response.

Everything Is Python!

Salt is entirely based on Python, and all of the commands and services are Python scripts (Listing 2-9).

vagrant@master:~$ for f in salt salt-key salt-master salt-minion; do

> file $(which $f)

> done

/usr/bin/salt: Python script, ASCII text executable

/usr/bin/salt-key: Python script, ASCII text executable

/usr/bin/salt-master: Python script, ASCII text executable

Listing 2-9  List Python Files

Installing the First Independent Salt Minion

We now have the Salt Master installed with the accompanying Salt Minion and the addition of Salt Cloud. We will now move on to configuring the first independent Salt Minion, which is based on CentOS 8 Stream. Using a different Linux distribution is important to help us understand the agnostic nature of Salt. Please open up another terminal on your host system, move to your directory containing the Vagrantfile, and connect to the minion1 virtual machine (Listing 2-10).

iMac:~$ cd vagrant/salt-book

vagrant/salt-book $ vagrant ssh minion1

[vagrant@minion1 ~]$ hostnamectl | grep 'Operating'

Operating System: CentOS Stream 8

[vagrant@minion1 ~]$ sudo yum -y update

[vagrant@minion1 ~]$ curl -o install_salt.sh -L https://bootstrap.saltproject.io

[vagrant@minion1 ~]$ sudo sh install_salt.sh -i minion1 \

> -A 192.168.33.100 stable 3003

Listing 2-10  Connect to minion1 and Install the Salt Minion

Walking through the preceding commands, we connect to the virtual machine running CentOS 8 Stream, and we update the system before downloading the installer and installing the Salt Minion. We set the Minion ID to minion1 and use the same address as before for the Salt Master. It is important to set the version to the same as the one the Salt Master is running; it is not recommended to run Salt Minions with versions later than that of the Salt Master that they connect to. The installer will start the services, so we now need to return to the Salt Master to authorize the keys (Listing 2-11).

[vagrant@master ~]$ sudo salt-key -y -a minion1

The following keys are going to be accepted:

Unaccepted Keys:

minion1

Key for minion minion1 accepted.

[vagrant@minion1 ~]$ sudo salt '*' test.version

master:

3003

minion1:

3003

Listing 2-11  On the Salt Master, We Accept the New Key and Test

From the Salt Master, we can run the command salt. It is only the Salt Master that has this command, which is used to remotely configure the Salt Minions. On this occasion, we run the test.version function across all Salt Minions. The returned data from each Salt Minion is the version of the Minion software that each is running, really useful as a quick audit of your Salt Minions.

Salt-Key and Key Files

We have been amazingly productive so far ; we have quickly installed one instance of the Salt Master and two instances of the Salt Minion and accepted the keys. We have seen a little of the configuration of the Salt Minion, and now we are going to look more at these illusive keys. Firstly, we will stay on the Salt Master and investigate the salt-key command and where keys are stored (Listing 2-12).

[vagrant@master ~]$ sudo salt-key -L

Accepted Keys:

master

minion1

Denied Keys:

Unaccepted Keys:

Rejected Keys:

3003

[vagrant@master ~]$ sudo apt install -y tree

[vagrant@master ~]$ sudo tree /etc/salt/pki/master

/etc/salt/pki/master/

├── master.pem

├── master.pub

├── minions

│   ├── master

│   └── minion1

├── minions_autosign

├── minions_denied

├── minions_pre

└── minions_rejected

Listing 2-12  On the Salt Master, We List Keys

Walking through the preceding command listing, we begin by listing the keys using salt-key. We can see the two accepted keys. If you can’t, please book for an eye exam ☺. The other key categories are all empty, which is a good thing. We have just two Salt Minions, and both the keys are accepted. Each key status category, including Accepted Keys, represents a subdirectory within /etc/salt/pki/master. You can see that we install the tree package to provide access to the tree command, which is amazing at listing directories for us. You see this when we list the aforementioned master directory. Each of the subdirectories listed below /etc/salt/pki/master is described in the following:

- minions: This directory holds the public keys from Salt Minions that have been accepted and listed as Accepted Keys from salt-key.

- minions_autosign: A transitioning directory used temporarily when the Salt Master has been configured to auto-sign keys. Keys will be in this directory briefly, and we are never likely to see keys here, even if auto-sign is enabled.

- minions_denied: This directory represents Salt Minion keys that have been automatically denied usually because of duplicate Minion IDs or keys being regenerated on a Salt Minion where the original key was never deleted. Keys can be deleted with sudo salt-key -d <minion-id>. This directory is shown by salt-key as Denied Keys.

- minions_pre: When keys are waiting to be accepted, they are placed in this directory and are represented by salt-key as Unaccepted Keys.

- minions_rejected: If we choose to reject keys rather than accept them, they are placed in this directory. To reject a key, we use sudo salt-key -r <minion-id>. The command salt-key represents this directory as Rejected Keys.

It is very important to use the salt-key command to manage key requests, rather than moving key files between directories. For example, when deleting a key with salt-key, it does not just delete the key from the minions subdirectory. In addition, the encryption key used to communicate securely with all Salt Minions is regenerated to ensure the deleted Salt Minion is denied access. Ongoing Salt Minions have to reauthenticate, seamlessly, to receive the new AES encryption key. For more information on salt-key, you can use the simple help or read the man page:

[vagrant@master ~]$ salt-key --help

[vagrant@master ~]$ man salt-key

Minion Keys

The key structure on a Salt Minion is simpler. Using the /etc/salt/pki/minion directory, we just store the public and private keys of the Salt Minion along with the original Salt Minion public key, signed by the Salt Master. We demonstrated this on the independent CentOS 8 Stream Salt Minion, although we could use either Salt Minion. I have chosen to use the CentOS Stream 8 system to demonstrate this, so we understand that this is the Salt Minion and not the Salt Master.

[vagrant@minion1 ~]$ sudo yum install -y tree

[vagrant@minion1 ~]$ sudo tree /etc/salt/pki/minion

/etc/salt/pki/minion/

├── minion_master.pub

├── minion.pem

└── minion.pub

Listing 2-13  Listing Keys on the Salt Minion

Salt Master Firewall Requirements

For Salt Minions, there are no specific firewall requirements; however, if you do have a host-based firewall running on the Salt Master or between the Salt Master and Salt Minions, then TCP ports 4505 and 4506 need to open inbound to the Salt Master. Our Salt Master is not running a host-based firewall, and we do not have firewalls blocking access from our Salt Minions. So long as the Salt Minions are able to access the Salt Master to offer the public key. In the following, we see that the Uncomplicated Firewall (UFW) is not running on the Ubuntu Salt Master. Additionally, we list the listening TCP ports and observe that 4505 and 4506 are both listening.

[vagrant@master ~]$ sudo ufw status

Status: inactive

[vagrant@minion1 ~]$ ss -ntl

LISTEN 0 1000 0.0.0.0:4505 0.0.0.0:*

LISTEN 0 1000 0.0.0.0:4506 0.0.0.0:*

Listing 2-14  Checking Ubuntu Firewall

Summary

You continue to amaze me with your rapid progress. You have passed this chapter with flying colors, and not only that, you have Salt installed on two systems, and we are ready to progress. Thank you for your participation so far.

We mentioned that when installing Salt we could use the Python installer, pip3. This is not the best way as we would need to create service unit files for the services. With that in mind, we chose the installation provided by the bootstrap installer. We can download this from the https://repo.saltstack.io website. The installer is a shell script that automates the configuration of the Salt Minion, as well as installing the Salt Master and Salt Minion software as required.

Securing access to the Salt Master, we are required to grant access by accepting keys from the Salt Minions using salt-key. We also can later delete or reject keys using the same tool. Incidentally, we did see that all these commands are Python scripts.

The ZeroMQ message bus that the Salt Master uses listens on TCP ports 4505 and 4506. These ports need to be accessible by the Salt Minions and opened in the ingress direction to the Salt Master.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_3

3. Adding and Removing Salt Minions

Andrew Mallett[1]

(1)

Peterborough, UK

It may have escaped your attention, but you did so well in the last chapter that we were able to install both the Salt Master and Salt Minion. We have two Salt Minions, one on the Ubuntu Salt Master and the other independently on the CentOS 8 Stream virtual machine. By streamlining the installation with the use of additional options, we were able to configure the Salt Minion simultaneously alongside the installation. The option -i was used to set the Salt Minion ID, and the use of -A allowed us to set the IP address of the Salt Master. In this chapter, we will look a little more at the Salt Minion configuration file and security related to the key-based authentication model.

As your quick reference for this chapter, we have put together the following list of topics that you are going to learn as you progress to the end of the chapter:

- Review Salt Minion configuration.

- Regenerate Salt Minion keys.

- Review salt-key on the Salt Master.

- Delete Salt Minions.

- Configure auto-accept on the Salt Master.

- Salt Minion platforms.

- Salt Minion configuration.

Salt Minion Configuration

During the installation of the Salt Minion, we set the Minion ID and the Salt Master location. Both of these configuration settings can always be set later in the configuration file for the Salt Minion. Any changes made to the configuration while the Salt Minion service is running will require a restart of the service. You will find the Salt Minion’s main configuration in the file /etc/salt/minion; however, there is an extension directory, /etc/salt/minion.d/. The former file is heavily commented and documents the default settings used by the Salt Minion out of the box. Although we could use this file for our custom configurations, the extension directory is recommended for two reasons:

It prevents package updates overwriting custom changes.

It becomes an easy mechanism to view custom changes to the Salt Minion.

We can create additional configuration files with an extension of .conf within this directory implementing changes that we want to use. We would recommend consideration of the name /etc/salt/minion.d/local.conf as a starting point. Perhaps we may use additional files where we may have multiple settings such as for beacons (/etc/salt/minion.d/beacons.conf), which we will see later. The configuration files can merge to produce a combined configuration.

In general and as a minimum, we need to consider configuring the Minion ID and the location of the Salt Master. We avoided the explicit definitions ourselves by using options -i and -A for the installer as previously noted. If searching the /etc/salt/minion file for information on configuring the Minion ID, use the key id: for your search, as shown in Listing 3-1.

vagrant@minion1:~$ sudo grep -B 5 id: /etc/salt/minion

# Explicitly declare the id for this minion to use, if left commented the id

# will be the hostname as returned by the python call: socket.getfqdn()

# Since salt uses detached ids it is possible to run multiple minions on the

# same machine but with different ids, this can be useful for salt compute

# clusters.

#id:

Listing 3-1  Minion ID Setting (Working on the CentOS 8 Stream Minion)

We can see the commented key id: without a value. This represents the default setting. If we leave the value blank, the system will try to make use of the system’s FQDN or hostname entries for the 127.0.0.1 address that are not simple localhost. If the Salt Minion has previously set the Minion ID using the FQDN, the value is cached to the /etc/salt/minion_id file. The installer also populates this file with the name passed to the option -i. Even though the id: is not actually set, we can make use of the value from the cache (Listing 3-2).

vagrant@minion1:~$ sudo cat /etc/salt/minion_id

minion1

Listing 3-2  minion_id File

In addition to the Minion ID, we also need to set the address of the Salt Master or be able to resolve a host named “salt” or “salt.<our domain name>.” If we are configuring the entry via the configuration file, we need to research the key master: (Listing 3-3).

vagrant@minion1:~$ sudo grep -B 2 '#master:' /etc/salt/minion

# Set the location of the salt master server. If the master server cannot be

# resolved, then the minion will fail to start.

#master: salt

Listing 3-3  Researching the master: Key in /etc/salt/minion

The default setting is commented, but the hostname of salt is set. We can use either a resolvable hostname or IP address for the master: key. Using the installer to set this value, the setting is made in the file /etc/salt/minion.d/99-master-address.conf. For the moment, we will leave this setting in place, but we will add an additional local.conf file containing just the id: key (Listing 3-4).

vagrant@minion1:~$ echo 'id: minion1' | \

sudo tee /etc/salt/minion.d/local.conf

id: minion1

vagrant@minion1:~$ sudo rm /etc/salt/minion_id

vagrant@minion1:~$ sudo systemctl restart salt-minion.service

vagrant@minion1:~$ sudo salt-call test.ping

local:

True

Listing 3-4  Explicitly Setting the Minion ID (Working on minion1, the CentOS 8 Stream System)

In the preceding listing, we first populate the local.conf file. As we no longer need the cache file, we go ahead and delete it before restarting the Salt Minion service. To test the operation from the Salt Minion, we can use the command salt-call rather than salt, which can only be run on the Salt Master. The command salt is used on the Salt Master to target selected Salt Minions; the command salt-call is executed on a Salt Minion and targeting operations to be run locally.

Moving back to the command line on the Salt Master, we will now see how we can easily add host records to both systems using Salt configuration management. Yes, even at this early stage, we can have an overview on how we can use Salt. We will see much more on these operations as we progress, but as a quick overview, we will add a host record for “salt” to each system (Listing 3-5).

vagrant@master:~$ sudo salt '*' hosts.add_host 192.168.33.100 salt

master:

True

minion1:

True

vagrant@master:~$ sudo salt '*' cmd.run 'ping -c1 salt'

master:

PING salt (192.168.33.100) 56(84) bytes of data.

64 bytes from salt (192.168.33.100): icmp_seq=1 ttl=64 time=0.011 ms

--- salt ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.011/0.011/0.011/0.000 ms

minion1:

PING salt (192.168.33.100) 56(84) bytes of data.

64 bytes from salt (192.168.33.100): icmp_seq=1 ttl=64 time=0.276 ms

--- salt ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.276/0.276/0.276/0.000 ms

Listing 3-5  Adding Host Records Using Salt (Working on the Salt Master)

As we can now resolve the hostname for “salt,” we can discard the setting for the master: key within our Salt Minion configuration and make use of the default from the /etc/salt/minion file. We can also delete the 99-master-address.conf file centrally, using salt.

Note   Do make sure that both Minions are able to ping the Salt Master using the hostname of “salt” (Listing 3-6).

vagrant@master:~$ sudo salt '*' \

file.remove /etc/salt/minion.d/99-master-address.conf

master:

True

minion1:

True

vagrant@master:~$ sudo systemctl restart salt-minion

vagrant@master:~$ sudo salt '*' test.ping

master:

True

minion1:

True

Listing 3-6  Deleting Superfluous Configuration (Working on the Salt Master)

Note   We cannot reliably restart the Salt Minion service using Salt itself. To test connectivity, we restart the service manually from the system we are currently working on. We can be confident the restart will be successful on both systems.

Regenerating Salt Minion Keys

If the keys to the Salt Minion have become compromised, we will need to regenerate the key pair for the Minion. You will find these keys are stored in the /etc/salt/pki/minion directory:

minion.pem: The Salt Minion private key

minion.pub: The Salt Minion public key

minion_master.pub: The signed certificate request from the Salt Master

Using salt-call, we can regenerate the keys locally on the Salt Minion. Ideally, we would do this from the Salt Master as we would also need to ensure the currently accepted key was deleted. For us, we want to observe the behavior by not deleting the existing key from the Salt Master. We will start working at the command line of the CentOS 8 Stream Salt Minion (Listing 3-7).

vagrant@minion1:~$ sudo salt-call saltutil.regen_keys

local:

None

vagrant@minion1:~$ sudo salt-call test.ping

[CRITICAL] The Salt Master has rejected this minion's public key!

Listing 3-7  Regenerating Salt Minion Keys on the CentOS 8 Stream Minion

To repair this issue, delete the public key for this Minion on the Salt Master and restart this Minion.

Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

Hmmmmm! This does not look good, does it? We did warn you that we really should have targeted the minion1 Salt Minion from the Salt Master from where we could have deleted the previously accepted key. But let’s not become too stressed and resolve these minor issues on the Salt Master (Listing 3-8).

vagrant@master:~$ sudo salt-key -L

Accepted Keys:

master

minion1

Denied Keys:

minion1

Unaccepted Keys:

Rejected Keys:

vagrant@master:~$ sudo salt-key -y -d minion1

Listing 3-8  Resolving Issues with Keys on the Salt Master

The following keys are going to be deleted:

Accepted Keys:

minion1

Denied Keys:

minion1

Key for minion minion1 deleted.

Key for minion minion1 deleted.

The newly generated key became a duplicate entry on the Salt Master. Not a good situation, but was resolved by adding the new key to the Denied Keys category. Resolving this is a simple matter of deleting the Minion, which deletes both the old and new keys. The remote Salt Minion will have to be restarted to connect again. For the moment, we will leave it idle while we investigate how we can auto-accept keys.

Accepting Keys

We have already seen that an incoming key request will need to be accepted by the Salt Master using salt-key. Perhaps, in a very controlled environment, we could allow the Salt Master to accept key signing requests automatically. This is generally a bad idea as it opens to the possibility of any system, authorized or not, being able to use your configuration system. Where limited access to the network is guaranteed, auto_accept: can be configured and help your system become active very quickly. When teaching Linux in a classroom environment, we are happy for this automation. We now write to the Salt Master configuration, adding to the extension directory similar to that of the Salt Minion (Listing 3-9).

vagrant@master:~$ echo "auto_accept: True" | \

sudo tee /etc/salt/master.d/local.conf

auto_accept: True

vagrant@master:~$ sudo systemctl restart salt-master

Listing 3-9  Enabling Auto-Accept on the Salt Master

Restarting the Salt Master will not cause an issue to other Salt Minions, the ZeroMQ message bus is robust, and Salt Minions will simply reconnect again. Like Salt Minions, the Salt Master needs to be restarted if the configuration has changed. Now that we have enabled the auto_accept: key, restarting the Salt Minion on CentOS 8 Stream will see the key accepted seamlessly on the Salt Master. Returning to minion1, we restart the Salt Minion service and test connectivity back to the Salt Master, proving key acceptance (Listing 3-10).

vagrant@minion1:~$ sudo systemctl restart salt-minion

vagrant@minion1:~$ sudo salt-call test.ping

local:

True

Listing 3-10  Testing Auto-Accept from minion1

Having a positive response indicates that we have been able to send the results of the test.ping function through to the Salt Master and our key has been authorized, albeit automatically.

Supported Salt Minion Platforms

The Salt Master is supported on mainstream Linux distributions , meaning to use Salt, we need at least one Linux system. The Salt Minion is supported additionally on other platforms including Windows, UNIX, and macOS:

Mainstream Linux: RHEL/Ubuntu/Debian/Raspbian/SLES (also supports Salt Master)

macOS: 10.12 and later

Solaris: 10,11

Windows: 2008, 2012, 2016, 2019 and 10

AIX: 7.1, 7.2

Summary

Yes! You must be pleased as you have swathed a massive path through this chapter, and you are now well on your way to becoming a Salt guru.

We are hopeful that by now you are a little more familiar with the initial configuration of Salt Minions and the state of the key authentication system. We have also seen a little of the operation of Salt, where we use the command salt on the Salt Master to configure Salt Minions. Using the command salt-call, we can configure the local Salt Minion. Flexible is the name of our Salt game.

A Salt Minion requires a Minion ID to present to the Salt Master. This must be unique in the context of the Salt Master without duplicates. We have seen that the duplicate key will be denied. Where a great naming standard exists for FQDNs, we can use the fully qualified domain name of a system as the Minion ID. The first time this is discovered it will be cached in the file /etc/salt/minion_id to save retrieving the FQDN each time the Salt Minion is restarted. To explicitly set the Minion ID, we can utilize the id: key within the Salt Minion configuration.

Connectivity to the Salt Master is also a must for the Salt Minion. The location of the Salt Master can be set either by resolving the “salt” hostname or having the master: key configured in the Salt Minion’s configuration. We saw that using the function hosts.add_hosts to configure this centrally on the Salt Master is possible where we already have connected Salt Minions. More often, this would be an entry in DNS.

Salt Minions use their signed public key to gain access to the Salt Master. It may be necessary to regenerate these keys if the key has been compromised. The correct procedure is to use the function saltutil.regen_keys targeting just the single Salt Minions and then deleting the existing key for the Salt Minion on the Salt Master using salt-key. Automating the acceptance of new Salt Minion keys is possible in controlled environments. This is configured using the Salt Master configuration.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_4

4. Targeting Salt Minions

Andrew Mallett[1]

(1)

Peterborough, UK

Running the command salt from the command line of the Salt Master, we are required to target the Salt Minions, or more simply, we have to specify the Salt Minions that should execute the specified module and function. In the examples we have seen so far, we have not gone into many details or filtered on the Salt Minions. Using the ‘*’ as our point of aim, we are targeting using “globbing” all Salt Minions. The method of “globbing ” is the default using the Minion ID or part thereof as the target. The asterisk metacharacter used on its own is representing any Minion ID.

A quick overview of what we mean by “targeting” is shown in the following and describes your journey through this chapter:

- Targeting Minions

- Previewing targets

- Using the match.<target> module and function

- Using Salt Grains

- Creating Nodegroups

Salt Minion Targeting

Configuring Salt Minions from the command line using salt from the Salt Master will require a list of Salt Minions to target. As previously mentioned, we have used the simple ‘*’ as the specified target, expanding to all Salt Minions. For some tasks, this is perfect, as further targeting can be specified at other levels of the Salt infrastructure, which we will see later when looking at building an effective top file when using Salt States. There are other times when we will want to be more targeted, and we can then use a more restrictive “globbing” target or even implement other targeting methods. These targets include

- Minion ID shell-style globbing : This is the default matching that is used by Salt and utilizes globbing metacharacters used in the shell to match on a Minion ID or part thereof. Any globbing metacharacter used must be enclosed in single quotes to prevent expansion by the shell prior to being evaluated by Salt.

- Lists : We can create a list of Minion IDs dynamically as the target for Salt.

- Regular expressions : Where matching with a shell-style glob is not sufficient, we may drill down even further on the Minion ID using the PCRE (Perl Compatible Regular Expressions) regular expression engine, allowing for more metacharacters than globbing.

- Grains : Salt Grains are facts about the target system and can be used to help identify Salt Minions by using elements such as the os or os_ family as an identifier.

- Pillar data : Pillars of Salt or Pillar data is stored on the Salt Master for specific Salt Minions and can be used as custom targeting of Salt Minions. As a Salt administrator, we have complete control over the Pillar keys and values, which create the Pillar data but, unlike custom Grains, are not manageable by local administrators on a Salt Minion.

- Subnet/IP address : You will probably not be surprised to find out that we can target IP addresses or specified subnets as our target.

- Nodegroups : We can define groups of Salt Minions within the Salt Master configuration allowing targeting these Minion groups.

- Compound matching : It is also possible to match Minions by combining targeting methods to produce better differentiation. Methods can be combined using the Boolean values of and, not, and or.

Having learned the target types that we can implement, we are going to learn how to compose these targets and include some examples that you can copy in your own lab.

Default Minion ID Globbing

If we only want to match on the Minion ID , we do not need to use any additional command line options with the salt command as this is the default target. To allow for the administrator to see the result of the target processing, we can add the option --preview-target. This option can be used without any execution function, allowing for safe resolution of the target without any operational impact. Listing 4-1 illustrates some examples of globbing on the Minion ID and the expansion of the glob.

#Match only on minion1

vagrant@master:~$ $ sudo salt 'minion1' --preview-target

- minion1

#Match on minion followed by any single character

#The ? represents any single character

vagrant@master:~$ sudo salt 'minion?' --preview-target

- minion1

#Match on minion1 or minion2

#The [] enclose a list or range to match

vagrant@master:~$ sudo salt 'minion[12]' --preview-target

- minion1

#Match on any Minion ID starting with m

#The * represents zero or more characters

vagrant@master:~$ sudo salt 'm*' --preview-target

- master

- minion1

#Match on any Minion ID ending with 1

vagrant@master:~$ sudo salt '*1' --preview-target

- minion1

Listing 4-1  Default Salt Minion Targeting

Lists

If we cannot differentiate accurately enough using the default globbing target, we can create lists at the command line. This may help us, but the reality is Nodegroups act as persistent lists when the list may be needed more than once. Lists at the command line are transient and do not persist. Creating a list from the command line, we specify a list of Minion IDs without metacharacters. The option -L defines the list being used, and list items are comma delimited (Listing 4-2).

#Matching on a list of Minion IDs

vagrant@master:~$ sudo salt -L 'master,minion1' --preview-target

- master

- minion1

Listing 4-2  Matching Lists of Minion IDs

Targeting Using PCRE

Perl Compatible Regular Expressions can be used with salt implementing the option -E.. If you haven’t come across regular expressions before, they are similar to globbing metacharacters but far more powerful. Although regular expressions can be off-putting when you come across them, they are worth working with as they are common to many tools and languages. Learning the basics once will pay off in the long run. Some examples are shown in Listing 4-3 to get you started with at least the PCRE fundamentals.

#Matching on a Minion ID starting with minion and finishiing with any single digit

vagrant@master:~$ sudo salt -E 'minion\d{1}' --preview-target

- minion1

#Match on minions starting with 'm' and ending with '1' with any amount of characters in between the first and last character

vagrant@master:~$ sudo salt -E '^m.*1$' --preview-target

- minion1

#Match on any Minion ID that starts with an m

vagrant@master:~$ sudo salt -E 'm' --preview-target

- master

- minion1

#Match on master or minion followed by any single digit

vagrant@master:~$ sudo salt -E \

'^(master|minion\d{1})$' --preview-target

- master

- minion1

Listing 4-3  Matching the Minion ID Using PCRE

Targeting Using Grains

Using Salt Grains as our target unleashes a huge potential and power to us. Grains are facts about the remote systems and can be expanded using custom Grains defined by ourselves and stored on the Salt Minion.

First, we look at comparing “salt” and “salt-call.” The first is used only from the Salt Master to target Salt Minions; the latter is used without the need of a target. We will begin by using “salt-call” as this is an easy shortcut to list all Grains and their values (Listing 4-4).

#Using -g shortcut

vagrant@master:~$ sudo salt-call -g

local:

----------

biosreleasedate:

12/01/2006

biosversion:

VirtualBox

...

#Listing Grains and values longhand

vagrant@master:~$ sudo salt-call grains.items

local:

----------

biosreleasedate:

12/01/2006

biosversion:

VirtualBox

...

#Listing just the Grains without values

vagrant@master:~$ sudo salt-call grains.ls

local:

- biosreleasedate

- biosversion

- cpu_flags

- cpu_model

#Returning the the command salt, we can list selected Grains

vagrant@master:~$ sudo salt '*' grains.item os os_family

master:

----------

os:

Ubuntu

os_family:

Debian

minion1:

----------

os:

CentOS Stream

os_family:

RedHat

#Using the function get we can access a single grain, returning only the value without the key. Useful in templates.

vagrant@master:~$ sudo salt '*' grains.get os

master:

Ubuntu

minion1:

CentOS Stream

Listing 4-4  Listing Grains Using salt-call and salt

All of the preceding commands could have been run with “salt-call.” It just makes more sense to compare values between the two systems when less data is returned. Now that we have an understanding of Grains, we can use them as targets making use of the option -G. Using the os_family grain, we can target Ubuntu- and Debian-based systems. When targeting Debian as the os_ family, we include Debian and Ubuntu as well as other distributions such a Raspbian running on the Raspberry Pi. When using Red Hat, we target CentOS, CentOS Stream, as well as RHEL (Listing 4-5).

vagrant@master:~$ sudo salt -G 'os_family:Debian' --preview-target

- master

vagrant@master:~$ sudo salt -G 'os_family:RedHat' --preview-target

- minion1

Listing 4-5  Targeting Using Grains

Targeting Nodegroups

Using lists, we dynamically create lists from the command line; however, these lists do not persist and are said to be transient. Implementing Nodegroups as part of your Salt Master’s configuration allows for persistent list definitions. Nodegroups can be created in any .conf file within the /etc/salt/master.d/ directory. We already have the local.conf file; however, as we are likely to have multiple Nodegroups, we will create a new file, /etc/salt/master.d/groups.conf. We can use any text editor that we are familiar with. Ubuntu has both nano and vim. We use vim to edit the file, which is in YAML format. This means that the leading whitespace is significant. Each group name is indented two spaces below the nodegroups: entry (Listing 4-6).

Note   There is no requirement to restart the Salt Master when defining or editing Nodegroups.

vagrant@master:~$ sudo vim /etc/salt/master.d/groups.conf

nodegroups:

webservers: 'L@master,minion1'

ubuntu: 'G@os_family:Debian'

centos: 'G@os_family:RedHat'

vagrant@master:~$ sudo salt -N webservers --preview-target

- minion1

- master

vagrant@master:~$ sudo salt -N ubuntu --preview-target

- master

vagrant@master:~$ sudo salt -N centos --preview-target

- minion1

Listing 4-6  Defining Nodegroups and Targeting Groups

Using Nodegroups, we have access to the same targeting methods that we may use from the command line using the option without the dash followed by the @ symbol. So we get to see “L@” and “G@” and so on.

Targeting Subnets

Targeting Salt Minions on a specific subnet may be really useful where the subnet can identify an office, department, or function by the network address. The targets consist of an IP address or CIDR range and are targeted using the option -S. Adding the entries as Nodegroups can also prove useful (Listing 4-7).

vagrant@master:~$ sudo salt -S 192.168.33.100 --preview-target

- master

vagrant@master:~$ sudo salt -S 192.168.33.0/24 --preview-target

- master

- minion1

vagrant@master:~$ sudo vim /etc/salt/master.d/groups.conf

#Append to groups.conf

nodegroups:

webservers: 'L@master,minion1'

ubuntu: 'G@os_family:Debian'

centos: 'G@os_family:RedHat

office: 'S@192.168.33.0/24'

vagrant@master:~$ sudo salt -N office --preview-target

- minion1

- master

Listing 4-7  Targeting IP Address

Targeting Using Pillar Data

As yet, we have not implemented Pillars of Salt or Pillar data in Salt. We will later, but simply put, Pillar data is similar to Grains but entirely created by the Salt administrator. Pillar data is stored centrally on the Salt Master rather than being stored in the Salt Minion, avoiding the potential pitfalls of local administrators being able to modify local custom Grains. Pillar data is assigned to Salt Minions from the Salt Master and can be used like Grains as variables in templating where differences would exist in such things as service names and packages. To make use of Pillar data in targeting, the option -I is used. In the examples, we would not expect Salt Minions to be returned where we have not defined Pillar data:

vagrant@master:~$ sudo salt -I "location:hq" --preview-target

ERROR: No return received

Using Compound Matches

Finally, we can look at combining targeting mechanisms, known as compound matching. Implementing the -C option, we can look at these matches. In Listing 4-8, we look for Ubuntu web servers using Grains and Nodegroups, before moving on to the second example where we combine two groups together specifying that targets must belong to both.

Note   Compound matching does not work well with the space in the OS Grain value for CentOS Stream.

vagrant@master:~$ sudo salt -C 'G@os:Ubuntu and N@webservers' \

--preview-target

- master

vagrant@master:~$ sudo salt -C "N@centos and N@webservers" \

--preview-target

- minion1

Listing 4-8  Compound Matching

Discovering the Mechanics of Matching

Using the command line on the Salt Master, we have been able to see that we are able to use many different methods in targeting Salt Minions, but it is the Salt Minion that decides if it matches or not. The scalability of Salt is very much dependent on the distributed workload sent to Salt Minions. Tasks that are published to the ZeroMQ message bus are read by each Minion before a match is made. The Salt Minion examines the published target to see if a match can be made. Internally, the Salt Minion runs a function based on the target type. The match module has individual functions to match each potential target. We can demonstrate this process by using “salt” on the Salt Master targeting all Salt Minions but executing the match module (Listing 4-9).

vagrant@master:~$ sudo salt '*' match.ipcidr '192.168.33.0/24'

minion1:

True

master:

True

vagrant@master:~$ sudo salt '*' match.ipcidr '192.168.33.100'

master:

True

minion1:

False

vagrant@master:~$ sudo salt '*' match.grain 'os:CentOS Stream'

master:

False

minion1:

True

Listing 4-9  Matching Targets

Looking at the resulting returned data from each Minion, we can determine if they matched or not. The match module is not something we would often run but is used internally by the Salt Minions to determine which jobs are destined for execution by them.

Supported Salt Platforms

The Salt Master that is used with Salt has to be installed on mainstream Linux platforms, meaning you have to have at least one Linux server within your mix. The Salt Minions though can be extended into UNIX, macOS, and Windows devices. Targeting becomes important to make sure we are running the right commands on the right Salt Minions. Grains can be especially important in checking the running OS.

Summary

Being able to run the correct command on the correct system is fundamental to your success as an administrator or DevOps engineer. Making it to the end of this chapter, you have proved that you have what it takes to become a great Salt administrator.

Using the command “salt,” we need to be able to target the Salt Minions, and we have a variety of targeting methods at our disposal. We are hoping by now you are a little more familiar with those methods, and we can be a little more targeted than just using ‘*’ as we had previously seen. The default target is the Minion ID or part thereof. If specifying just part of the ID, we can use shell-style globbing metacharacters. If this is not enough, we can use option -E and PCRE, giving us many more metacharacters and control. To access Salt Minions on a remote network, we may want to use the option -S targeting a single IP address or CIDR range. Lists can be defined at the command line with the option -L. To persist the lists, Nodegroups defined on the Salt Master are an option. Once defined, we can access the group using the option -N.

Although the targets are defined for use with the “salt” command on the Salt Master, it is the role of Salt Minions to determine if they match the target criteria. We can view the internal matching mechanism by executing the match Salt module.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_5

5. Working with Remote Execution Modules in Salt Open

Andrew Mallett[1]

(1)

Peterborough, UK

We are now able to target selected Salt Minions, and, while that is exciting, we now need to learn to execute configuration commands. This is where Remote Execution modules step into the Salt configuration management stadium. When using test.ping, we made use of the test execution module and the ping function from within that module. All of these Salt modules are written in the Python scripting language and exist as an integral part of the Salt Minion; Remote Execution modules are both stored and executed on the Salt Minion and installed with the Salt Minion package.

In this chapter, we are going to learn to locate these modules and find help from both the Web and the command line. We will be able to identify modules that we can use easily from the CLI to configure Salt Minions and, crucially, why they are used:

- Remote Execution: State vs. flow

- Locating modules

- SaltStack All Module Index

- Using command line help

- Installing packages

- Managing services

- Creating directories

- Managing service configuration files

- State vs. flow

Salt Remote Execution modules fit into a configuration management style that SaltStack calls “flow.” The project team describes both Flow and State that equate to Remote Execution modules and State modules.

Remember: A module is a Python script, and these scripts contain function definitions. Accessing the module by name, we can reference the function we need using the dot separator between the module and function.

Flow : When using Remote Execution modules directly from the command line, we have the flexibility to run commands as and when we want, but repetition is not easy, making flow management ad hoc and not reliably repeatable. If we have to run six or seven Remote Execution commands against Salt Minions, we have to consider if we could remember to execute the exact same commands in the exact same order each time. Using flow at this stage of the book is a quick and easy way to understand Salt and to quickly see results from our installation.

State: Using State modules explicitly declares that the associated modules and functions must be executed from a Salt State file. A State file lists all of the State modules that must be executed and their associated functions and arguments. State modules, as are all Salt modules, are still written in Python but are executed from within the State file rather than directly from the CLI. The State configuration management style allows for reliability in enforcing a Salt Minion’s state, and the configuration is repeatably correct. Not only this, a State is designed to be a self-documenting configuration where careful thought is given to State names or IDs.

In simple terms, flow allows for quick and easy configuration of Salt Minions, whereas State provides for a long-term defined state that requires enforcing.

Execution Modules

We have become familiar with the test module and using functions defined therein, such as ping or version. As we keep mentioning, all Salt modules are Python scripts sitting as files within the file system, and, as such we can search for and locate the source file. Using the native Linux command find, we search for the test.py file (Listing 5-1).

vagrant@master:~$ find /usr/lib -name 'test.py' -type f

/usr/lib/python3/dist-packages/salt/engines/test.py

/usr/lib/python3/dist-packages/salt/states/test.py

/usr/lib/python3/dist-packages/salt/runners/test.py

/usr/lib/python3/dist-packages/salt/modules/test.py

Listing 5-1  Locating Salt Modules in the File System

Reviewing the resulting output, we can see that we have four test modules. We can resolve their purpose from the directory that the module is located within. Execution modules are found in the salt/modules directory. The early versions of Salt were heavily dependent on Remote Execution modules, and this explains the simple name of the salt/modules subdirectory. The salt/states subdirectory houses modules to be executed from Salt State files; so even though the module names may be the same, they are differentiated by the context in which they are executed. Knowing where the module is located allows us to search for the ping function. Python scripts are text files. Extracting the function will display the help and source code of the function, helping us to understand the code as well as access documentation. You do not need to know Python to use and administer Salt, but learning a little Python is useful. Function definitions within a module (.py file) use the def statement. Searching the test module for def ping() will show the matching line, but not the rest of the definition. Wanting to include the definition itself, we show an additional 21 lines after the match using the grep command in Linux (Listing 5-2).

vagrant@master:~$ cd /usr/lib/python3/dist-packages/salt/modules

vagrant@master:modules$ grep -A21 'def ping()' test.py

def ping():

"""

Used to make sure the minion is up and responding. Not an ICMP ping.

Returns ``True``.

CLI Example:

.. code-block:: bash

salt '*' test.ping

"""

if not salt.utils.platform.is_proxy():

log.debug("test.ping received for minion '%s'", __opts__.get("id"))

return True

else:

ping_cmd = __opts__["proxy"]["proxytype"] + ".ping"

if __opts__.get("add_proxymodule_to_opts", False):

return __opts__["proxymodule"][ping_cmd]()

else:

return __proxy__[ping_cmd]()

Listing 5-2  Investigating Salt Modules Making Part of the Salt Minion Package on Ubuntu 20.04

Accessing Help on Execution Modules

By using grep, as we saw previously, we can see not only the code making up the function, but we also see the documentation, enclosed within triple quotes. If we purely need to access the documentation, the task is made simple through the sys Remote Execution module. Using salt-call, from any Salt Minion, we can access detailed help on functions within a module using either a shortcut or longhand version of the sys.doc function. The longhand version color-codes the output, whereas the shortcut does not. Additionally, the shortcut is only available with the salt-call command and not from salt. The longhand version can be used with either command, not forgetting that salt will always need a target and salt-call does not. In Listing 5-3, we use the test module for display purposes, but any Remote Execution module can be used, and we will see more later.

vagrant@master:~$ sudo salt-call -d test

...

vagrant@master:~$ sudo salt-call sys.doc test

...

Listing 5-3  Listing Help on All Functions in a Specified Module

We can see on our own displays that the output is quite verbose, prompting us to think of something a little more useful. To list all of the functions in a module, we can try this:

vagrant@master:~$ sudo salt-call sys.list_functions test

...

We can also list all functions by omitting the name of the module. Although less specific, this may be useful when you are looking for a specific function but unsure of the module it is located in:

vagrant@master:~$ sudo salt-call sys.list_functions

...

Having been able to list modules and functions, we should be a little more targeted on the exact function that we need help on now. Knowing the module and function that we want to use allows us to target the specific help we are needing (Listing 5-4).

vagrant@master:~$ sudo salt-call sys.doc test.ping

local:

----------

test.ping:

Used to make sure the minion is up and responding. Not an ICMP ping.

Returns ``True``.

CLI Example:

salt '*' test.ping

Listing 5-4  Specific Help on test.ping

We can see that we are extracting the help that we initially saw from the module. Moving away to a more practical help example, let’s look at managing the time zone (Listing 5-5).

vagrant@master:~$ sudo salt-call sys.doc timezone

...

vagrant@master:~$ sudo salt-call sys.doc timezone.get_zone

...

vagrant@master:~$ sudo salt-call sys.doc timezone.set_zone

...

vagrant@master:~$ sudo salt-call sys.doc timezone.get_hwclock

Listing 5-5  Working with the Time Zone Function

With this bad boy under our belt, we are able to use what we have learned. We know when setting up the Salt Master we did configure the time zone, but not on the Salt Minion. We can begin by reviewing the current settings and moving back to the salt command allowing targeting of all Salt Minions (Listing 5-6).

vagrant@master:~$ sudo salt '*' timezone.get_hwclock

minion1:

UTC

master:

UTC

vagrant@master:~$ sudo salt '*' timezone.get_zone

minion1:

UTC

master:

Europe/London

Listing 5-6  Reviewing Time Zone Configuration

These current settings can give us inconsistencies when viewing log files where the systems are configured in differing time zones. Even though the time will be correct on both systems, the displayed times will not be correct. Both Salt Minions have their hardware clock set to UTC time, which is desired, no matter where in the world the system is located. The offset to UTC is controlled by the configured time zone. We can configure both systems to the correct time zone, and it will only be adjusted on the Salt Minion needing the change (Listing 5-7).

vagrant@master:~$ sudo salt '*' timezone.set_zone 'Europe/London'

minion1:

True

master:

True

vagrant@master:~$ sudo salt '*' timezone.get_zone

master:

Europe/London

minion1:

Europe/London

Listing 5-7  Correcting Time Zone Configuration

This suits the Salt flow or Remote Execution model as we need a quick fix to configuration issues. Knowing that our systems should all be in the single time zone allows us to confidently run the command across ‘*’ targeting all Salt Minions. Those Salt Minions already meeting the configuration requirement silently ignore the configuration request. Configuring the correct time zone should be noted to be included as part of our main configuration applied to all Salt Minions via Salt State files to enforce this newly discovered configuration drift.

Using Web-Based Help

Browsing help on the Internet can prove useful and, for some, more appealing and readable. The SaltStack project maintains an index of all Salt modules that you can find at https://docs.saltproject.io/en/latest/py-modindex.html.

This is an indexed list, and to locate Remote Execution modules, we need to look under M for modules, just as in the file system where we needed to look in the salt/modules directory (Figure 5-1).

Figure 5-1  SaltStack Remote Execution Module Index

The Internet is so accessible, and, as we can see, it is easy to locate the module we need and the required documentation. By clicking the hyperlink on the module name, we are able to see all the functions within the module within the right screen panel. Figure 5-2 shows the managing local host entries with the hosts module.

Figure 5-2  Function Listing from the Hosts Remote Execution Module

Digging a little further, we may want to look at adding a new host record to each system, so we will look at the add_host help (Figure 5-3). Just click this in the panel.

Figure 5-3  Graphical Help on the add_host Function, Part of the Hosts Remote Execution Module

Implementing Configuration Changes Using Execution Modules

Having seen a little more of the Salt Remote Execution modules, we can dive a little deeper into our configuration arena and see what we can configure on our systems and the agnostic approach of Salt to differing systems. Salt specifies what to do in the form of the desired configuration but, crucially, not how to do it. That is down to the expertise of the local operating system.

Installing Software Packages

One of the big three configuration management items is the installation of software packages. Using the command line or scripts to try and automate this will be problematic for many reasons:

Errors may occur if the software is already present and installed.

Packaging tools differ between platforms:

1. 1.
RPM based on Red Hat–based systems

2. 2.
DPKG based on Debian-based hosts

3. 3.
PKG based on Arch Linux

4. 4.
Chocolatey on Windows-based systems

Having to Understand the Target System and Version Changes

Using Salt to perform these actions, we are abstracted from the target system, and we can use uniform commands to install packages. What we do have to be concerned with is the package name. If that name differs between systems, then we can use templates and variables to cater for those differences.

Using both an Ubuntu system and a CentOS system allows us to display the agnostic character found in Salt. Here, we look at installing the command line utility tree, which is great for listing directories and helping document your system. The package name is consistent across Linux distributions (Listing 5-8).

vagrant@master:~$ sudo salt '*' pkg.install tree

master:

----------

minion1:

----------

Listing 5-8  Checking the Installation of Tree

The empty returned data from both Salt Minions indicates that the package tree was installed on both systems. We can verify this, should we want, using pkg.info_installed for full information. For a simple listing of all installed packages, we can use pkg.list_pkgs (Listing 5-9).

vagrant@master:~$ sudo salt master sys.doc pkg.info_installed

vagrant@master:~$ sudo salt '*' pkg.info_installed tree

minion1:

----------

tree:

----------

arch:

x86_64

build_date:

2019-05-11T15:54:52Z

build_date_time_t:

1557590092

build_host:

x86-01.mbox.centos.org

description:

The tree utility recursively displays the contents of directories in a

tree-like format.  Tree is basically a UNIX port of the DOS tree

utility.

group:

Unspecified

install_date:

2021-05-14T18:14:08Z

install_date_time_t:

1621016048

license:

GPLv2+

packager:

CentOS Buildsys <bugs@centos.org>

release:

15.el8

relocations:

(not relocatable)

signature:

RSA/SHA256, Tue Jul  2 01:53:45 2019, Key ID 05b555b38483c65d

size:

111603

source:

tree-1.7.0-15.el8.src.rpm

summary:

File system tree viewer

url:

http://mama.indstate.edu/users/ice/tree/

vendor:

CentOS

version:

1.7.0

master:

----------

tree:

----------

architecture:

amd64

description:

displays an indented directory tree, in color

Tree is a recursive directory listing command that produces a depth indented

listing of files, which is colorized ala dircolors if the LS_COLORS environment

variable is set and output is to tty.

group:

utils

install_date:

2021-05-14T17:35:03Z

name:

tree

packager:

Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>

source:

tree

url:

http://mama.indstate.edu/users/ice/tree/

version:

1.8.0-1

vagrant@master:~$ sudo salt master sys.doc pkg.list_pkgs

vagrant@master:~$ sudo salt '*' pkg.list_pkgs | grep -A1 tree

tree:

1.8.0-1

tree:

1.7.0-15.el8

Listing 5-9  Checking Installed Packages

The package at is a command scheduler and does not appear to be listed as installed on the CentOS system used in the demonstration. We can check this from grep as we saw before; however, we need to be more accurate on the search term as the string at is in many package names. We now search for the name starting with any optional whitespace character followed by the package name at and a colon. Regular expressions, such as the one used in Listing 5-10, are not the most friendly, but are very powerful.

vagrant@master:~$ sudo salt '*' pkg.list_pkgs | grep -A1 '^\s*at:'

at:

3.1.23-1ubuntu1

vagrant@master:~$ sudo salt '*' pkg.install at

master:

----------

minion1:

----------

at:

----------

new:

3.1.20-11.el8

old:

Listing 5-10  Installing Packages on Systems

As the package was already installed on the Ubuntu system, we only needed to add it to the CentOS Salt Minion, but there were no errors including both Salt Minions within the configuration target. Finally, we can see the result of installing the zsh shell on each Salt Minion, as it is not on either by default (Listing 5-11).

vagrant@master:~$ sudo salt '*' pkg.list_pkgs | grep -A1 'zsh:'

vagrant@master:~$ sudo salt '*' pkg.install zsh

master:

----------

zsh:

----------

new:

5.8-3ubuntu1

old:

minion1:

----------

zsh:

----------

new:

5.5.1-6.el8_1.2

old:

Listing 5-11  Allowing the System to Detect Which Salt Minions Require Packages

Investigating Agnostic Behavior

Viewing the agnostic behavior is simple enough. The previous commands illustrate that well. To understand the behavior fully, we are much better off moving to the command salt-call. If you recall, salt is used to publish the job to the event bus. This is for the Salt Minions to collect from the Salt Master. Each Salt Minion is connected to the publishing port on the Salt Master, TCP 4505. It is the Salt Minion that does all of the heavy lifting allowing for the immense scalability of Salt. First, executing the removal of zsh from Ubuntu and then from CentOS, we can see how the same Salt function is executed differently on each system by using the log level of INFO (Listing 5-12).

vagrant@master:~$ sudo salt-call pkg.remove zsh -l info

[INFO    ] Executing command dpkg-query in directory '/root'

[INFO    ] Executing command systemd-run in directory '/root'

[INFO    ] Executing command dpkg-query in directory '/root'

local:

----------

zsh:

----------

new:

old:

5.8-3ubuntu1

[vagrant@minion1 ~]$ sudo salt-call pkg.remove zsh -l info

[INFO    ] Executing command rpm in directory '/root'

[INFO    ] Executing command systemd-run in directory '/root'

[INFO    ] Executing command rpm in directory '/root'

local:

----------

zsh:

----------

new:

old:

5.5.1-6.el8_1.2

Listing 5-12  Removing Software with salt-call Increasing Log Level for Visibility

We can see the running salt-call on the master Salt Minion used the command dpkg-query and when run on the CentOS minion1, we saw the execution of the rpm command .

Managing Services

Having viewed package management in Salt , we can make use of Salt to manage services in a similar way. For example, having just installed the package at, there is its associated service: atd. This needs to be running and enabled to be effective (Listing 5-13). On Debian-based systems, the installation of the package will also enable and start the service; this is not the case on Red Hat–based systems such as CentOS.

Note   We could equally run the same command targeting all Salt Minions. Using the grain helps us focus on the previously seen targeting methods.

vagrant@master:~$ sudo salt -G 'os_family:RedHat'  service.enable atd

minion1:

True

vagrant@master:~$ sudo salt -G 'os_family:RedHat'  service.start atd

minion1:

True

Listing 5-13  Starting and Enabling Services

Even though natively we have a common command managing services on both Ubuntu and CentOS, the systemctl command attaching to systemd, we can view the underlying command used by Salt just in the same way as with packages by using salt-call and raising the log level to include INFO (Listing 5-14).

[vagrant@minion1 ~]$ sudo salt-call service.enable atd -l info

[INFO    ] Executing command systemctl in directory '/root'

[INFO    ] Executing command systemd-run in directory '/root'

local:

True

Listing 5-14  Systemctl and Salt

Managing Files

The final element in our big three is the management of files . Files include all file types including directories. We will start with a quick example in creating the directory /srv/salt on all systems. We will see this later when using the Salt State system that uses this directory to house State files, usually on the Salt Master but possibly also on Salt Minions (Listing 5-15).

vagrant@master:~$ sudo salt '*' file.mkdir '/srv/salt'

master:

True

minion1:

True

Listing 5-15  Creating Directories Across Many Systems

Verifying the creation is equally easy as we have access to system commands also using Salt. Here, we list the newly created directories using the native ls command but across all systems (Listing 5-16).

vagrant@master:~$ sudo salt '*' cmd.run 'ls -ld /srv/salt'

minion1:

drwxr-xr-x. 2 root root 6 May 23 12:09 /srv/salt

master:

drwxr-xr-x 2 root root 4096 May 23 12:09 /srv/salt

Listing 5-16  Listing Directories

This is a great start but very simplistic. We can raise the power of this command, especially if you are using the Vagrant boxes put forward by this book. You will, no doubt, soon find that password-based authentication is disabled on the SSH server, the Secure Shell server. Using Salt, we can edit the associated configuration file for the SSH service to allow passwords and restart the service to effect the change. The file.replace command is what we need to look at, but first, we can check the existing configuration setting in each file using file.grep (Listing 5-17).

vagrant@master:~$ sudo salt '*' file.grep \

'/etc/ssh/sshd_config' '^PasswordAuthentication'

master:

----------

pid:

26562

retcode:

0

stderr:

stdout:

PasswordAuthentication no

minion1:

----------

pid:

6685

retcode:

0

stderr:

stdout:

PasswordAuthentication no

Listing 5-17  Searching Files for Regular Expressions

To change this, we can edit each file simultaneously as previously mentioned, using the function file.replace (Listing 5-18).

vagrant@master:~$ sudo salt '*' file.replace '/etc/ssh/sshd_config' \

pattern='^PasswordAuthentication no' \

repl='PasswordAuthentication yes'

master:

---

+++

@@ -55,7 +55,7 @@

#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!

-PasswordAuthentication no

+PasswordAuthentication yes

#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with

minion1:

---

+++

@@ -67,7 +67,7 @@

# To disable tunneled clear text passwords, change to no here!

#PasswordAuthentication yes

#PermitEmptyPasswords no

-PasswordAuthentication no

+PasswordAuthentication yes

# Change to no to disable s/key passwords

#ChallengeResponseAuthentication yes

vagrant@master:~$ sudo salt '*' file.grep \

'/etc/ssh/sshd_config' '^PasswordAuthentication'

master:

----------

pid:

27054

retcode:

0

stderr:

stdout:

PasswordAuthentication yes

minion1:

----------

pid:

6695

retcode:

0

stderr:

stdout:

PasswordAuthentication yes

Listing 5-18  Editing Files Using Salt

Having made the change to the files, the SSH service will need to be restarted. This too is easy, and we return to the service module (Listing 5-19).

vagrant@master:~$ sudo salt '*' service.restart sshd

minion1:

True

master:

True

Listing 5-19  Restarting the SSH Service

Summary

In this chapter, which you have passed with flying colors, we have extended our knowledge of Remote Execution modules beyond the test module we have previously been introduced to.

Firstly, we used the native Linux command find to locate the Salt modules that we use. Modules are Python files making use of the .py extension. This introduced to us the concept of module types, finding the test.py file within the salt/modules, salt/states, salt/runners, and salt/engines subdirectories. Each Salt subsystem has its own modules representing how they will be accessed. The subdirectory structure is seen from the file system directory hierarchy and is replicated in the SaltStack project documentation website. When referencing the module index, we need to check under M for Remote Execution modules and under S for State modules and so on.

Documentation is also accessible directly from the command line using either salt or salt-call. The latter, salt-call, often is better for a couple of reasons: the first being that we only need to read the documentation from a single system and the second being the shortcut salt-call -d.

Having read our fill of documentation – and let’s admit it we didn’t read anything – we took that knowledge boldly forward into the configuration arena that is Salt. We saw that configuring the correct time zone, installing packages, managing services, and editing files were simple even across mixed Linux distributions.

Although what we have seen may interest you, this is just the start of your journey, and we must remind you of the concept of flow vs. State relating to Salt configuration management, the latter being commands that are run ad hoc and not required to be repeatable. In the next chapter, we will introduce YAML files, the basis of Salt State files and repeatable configurations enforcing system state.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_6

6. Writing YAML Files

Andrew Mallett[1]

(1)

Peterborough, UK

Creating Salt State files requires an understanding of YAML, and additionally, Salt Master and Salt Minion configuration files are also written in YAML, making it hard to avoid. This chapter is going to introduce you to formatting YAML documents and getting you started with the basics.

Predominately, the focus will be on these main areas:

- YAML Ain’t Markup Language

- Using online parsers to create YAML

- Keys, lists, and dictionaries in YAML

- Editing YAML documents using Nano

- Editing YAML documents using Vim

- Using graphical editors and IDEs (Integrated Development Environments)

YAML?

Of course, we have to know the meaning of this. We simply cannot continue without divulging the meaning of the acronym. Firstly, it is a specific acronym type called retronym or a recursive acronym. Originally developed in 2001 extracting concepts from Python, Perl, and C: YAML Ain’t Markup Language! Originally it was Yet Another Markup Language but evolved to distinguish its purpose as a data-oriented language rather than a document markup language.

YAML is easily written and readable by us, as simple humans. Omitting the normal array of parentheses, brackets, and braces, YAML makes use of colons, whitespace, and dashes. This does make it easy to read and to write; the JSON format may still be the preferred data structure for computers to process, but we are happy to leave that to the microchips while we can enjoy the cleaner format of YAML:

- Mappings : In YAML, key-value pairs are known as mappings, using a colon and space in the mapping and before the value, for example, name: fred.

- Sequences : In YAML, lists are known as sequences and denoted with the dash and space. We can have a list of values, such as

- sales

- it

- hr

This often would expand into a mapping, now the double-space indent used before each dash to say that they belong to the departments mapping:

departments:

- sales

- it

- hr

We can have nested lists, or sequences of sequences. A nested sequence makes use of the brackets:

- [ fred, it, 25 ]

- [ sally, hr, 31 ]

We can omit the brackets by creating a mapping for each list:

user01:

- fred

- it

- 25

Adding readability, the YAML document could include additional mappings:

user01:

name: fred

dept: it

age: 25

- Multiline strings : To preserve embedded newlines in a string value, we can use the |. Again take note of the indented text showing its association with the message mapping:

message: |

This is the text and

newlines are preserved

Where embedded newlines are not required in YAML, we use >:

message: >

Text is not folded on embedded newlines

This is a single YAML line.

Gain Practice Writing YAML

Testing some of these rules, we can visit https://yaml-online-parser.appspot.com/, an online YAML parser that allows us to input YAML documents and view the default JSON output. Practice some of the examples previously seen but also try to break YAML so you really understand the language.

Editing YAML Files

Using Linux, which is obligatory for the Salt Master, you have the choice of command line editors, and you should use whichever editor you are most familiar with. If you create your YAML files on your desktop system and transfer them to the Salt Master later, you also have the choice of graphical editors that may come with YAML plugins. We will start by looking at nano on the Ubuntu 20.04 Salt Master.

Note   In the ideal world, a State file would be developed locally before being synchronized with a GIT repository and used directly from GIT or downloaded to the Salt Master.

Using Nano

The nano text editor is available on both Ubuntu and CentOS systems having a short learning curve, meaning that you can pick up the basics quickly. Like many software applications, nano includes the use of the ~/.nanorc that allows for a script to run on starting the application. It is this file that we can use to customize the editor. The file we need to edit to make these changes will be in your user account home directory, or when using Salt, root access is required so we would edit the file from /root, the home directory of the root user. The editor nano does have preconfigured files but not for YAML files. You can find these files by listing the directory /usr/share/nano/. We won’t add syntax highlighting, but we will add indenting options to help create and edit YAML files and we show this in Listing 6-1:

- set tempfile: Nano will save the file without additional prompting.

- set autoindent: When a line return is entered, the cursor will return to the last indent level. Really useful when creating related items in YAML. They all need to be at the exact same level.

- set tabsize 2: The Tab key is interpreted as two spaces.

- set tabstospaces: We HAVE to use spaces and not tabs in YAML. Using the Tab key is great to get the correct indentation, but we need to store tabs as spaces and not as tabs.

vagrant@master:~$ cd

vagrant@master:~$ pwd

/home/vagrant

vagrant@master:~$ nano .nanorc

set tempfile

set autoindent

set tabsize 2

set tabstospaces

vagrant@master:~$ nano my.yml

users:

- fred

- sally

vagrant@master:~$ od -a my.yml

0000000   u   s   e   r   s   :  nl  sp  sp   -  sp   f   r   e   d  nl

0000020  sp  sp   -  sp   s   a   l   l   y  nl

0000032

Listing 6-1  Sample .nanorc File for YAML

When editing the new sample YAML file, we need to use the Tab key to indent before adding the list item fred. After entering the fred item, the current indent level is maintained so the list item sally can be added without further indentation. Using the od command (octal dump), we can verify that spaces are used on the file and not tabs. It is a bit geeky, but at least we can see the raw characters used in the file .

Using Vim

The vim text editor is more powerful having a much longer learning curve. If you are not used to using vim, it is going to take a little more time to learn especially if you are not constantly using it. Repetition is the main key to learning in many areas including using vim. Similarly to Nano, Vim can use the ~/.vimrc to customize the experience:

- syntax on: Set syntax highlighting to be on.

- set bg=dark: If using a dark background, use this for the highlighting color scheme. If you are using a light background, then use set bg=light.

- set autoindent: As with Nano, we return to the last used indent level.

- set tabstop=2: Set the Tab key to use two spaces.

- set expandtab: Store tabs as spaces.

- set shiftwidth=2: This sets the indent level used by auto-indent.

- set cursorcolumn: Highlight the current column, which is really useful to check indenting is consistent down the document.

These settings can be combined onto a single line if required and are illustrated in the demonstration (Listing 6-2).

vagrant@master:~$ cd

vagrant@master:~$ pwd

/home/vagrant

vagrant@master:~$ vim .vimrc

syntax on

set bg=dark

set autoindent shiftwidth=2 tabstop=2

set expandtab cursorcolumn

Listing 6-2  Sample .vimrc File

Using vim, we can go beyond this. These settings may not work for use in every file we edit. For example, Python files are recommended to use a four-space indentation level. Let’s edit this again so these settings are used only for YAML files. We can also abbreviate the configuration settings (Listing 6-3).

vagrant@master:~$ vim .vimrc

syntax on

set bg=dark autoindent

autocmd FileType yaml setlocal sw=2 ts=2 et cuc

Listing 6-3  Improved .vimrc

We can test that this is all working the way that it should be by re-editing the my.yml file we created earlier. Do check that the auto-indent is working and the tab spacing is the same. From the screenshot shown in Figure 6-1, we can additionally gain the benefit of the cursorcolumn (cuc) setting.

Figure 6-1  Vim Editing Using cursorcolumn

Note   When editing files such as the /etc/salt/minion.d/local.conf, we need to remember that these are also YAML files using the .conf extension. Using the key sequence ESC :set ft=yaml allows the file to be read in YAML format. Make sure that the .vimrc file is located in the /root/ directory too as the file will be edited as the user root.

Persisting YAML Configuration for SLS Files

Salt State files use the extension SLS rather than YAML or YML, meaning that they would be associated with the YAML editing configuration. This is the same as the CONF files that we have just mentioned. Unlike .conf files, which will not always be in YAML format, SLS files are most likely to be YAML allowing us to persist the association. We will add a final line to the .vimrc file (Listing 6-4).

vagrant@master:~$ vim .vimrc

syntax on

set bg=dark autoindent

autocmd FileType yaml setlocal sw=2 ts=2 et cuc

autocmd BufEnter *.sls :setlocal filetype=yaml

vagrant@master:~$ vim my.sls

insall_at:

pkg.installed:

- name: 'at'

Listing 6-4  Final .vimrc File

With the extra line added, we can maintain the syntax highlighting associated with YAML as well as the indent settings we have chosen to use. The SLS file we create to test this is a working example that can be used to ensure that the at package is installed on targeted systems.

Using Graphical Editors/IDEs

There are many graphical editors with Integrated Development Environments that understand YAML. Many are free including Visual Studio Code and PyCharm. They may not easily interpret SLS files directly, but some do or allow you to associate extensions with YAML. For example, Microsoft’s Visual Studio Code has a plugin for Salt State files (Figure 6-2).

Note   We do not cover IDEs in detail here, but you are welcome to try your own.

Figure 6-2  YAML Files in PyCharm

Summary

Becoming familiar with YAML is a prerequisite for learning Salt as it is for other systems such as Ansible. Although the whitespace sensitivity may seem scary to start with, it soon becomes natural.

YAML is a data structure language being able to present complex data structures in an easy-to-read-and-write format – easily accessible to humans. YAML is one of the default renderers used in Salt State files as well as the Salt configuration files.

Using command line editors, we can customize their behavior to work well with YAML files. The simplest editor is Nano, but that also limits how much nano can be customized. vim is the most complex editor; it allows for complex configuration. If you are going to be using the Linux command line very frequently, then Vim is worth learning. If it is unlikely that you will have a lot of time at the command line editing files, then Nano has a shorter learning curve.

Graphical IDEs can be a great option if you work mainly on your desktop before uploading files to the server. Many options include free versions that work very well with YAML and/or SLS files.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_7

7. Writing Salt State Files

Andrew Mallett[1]

(1)

Peterborough, UK

Having gained the mastery of YAML files, we (and by “we” I mean “you”) can begin looking at creating Salt States and having repeatable configurations. Salt State files have the SLS extension as we have already seen when configuring vim.

Predominantly, the focus in this chapter will be on these main areas:

- State vs. flow

- Rendering engines and the default Jinja|YAML

- Installing common software packages

- Managing services with State files

- Delivering and editing files using Salt

- Using Salt Formulas

- Syntax checking State files

State vs. Flow

The ad hoc nature of Salt’s flow management style does have its place; however, when we need repeatable configurations, then we will use State management, both defining the configuration and enforcing it. Where we detect configuration oversights, we may quickly mitigate them using flow and Remote Execution modules, but we should always consider adding them to State files, allowing us to enforce them across our estate.

State File Location

State files, as we know, are written in YAML and are required to be located in specific locations, the default location being /srv/salt on the Salt Master. This is defined as the default, but there is nothing stopping us from checking the default entry and adding an explicit entry of our own using the same value. Firstly, we can check the default setting from the file /etc/salt/master, as shown in Listing 7-1.

Note   Salt can use State files written in other languages, even in raw Python. The default rendering is Jinja and then YAML.

vagrant@master:~$ sudo grep -A2 '^#file_roots:' /etc/salt/master

#file_roots:

#  base:

#    - /srv/salt

Listing 7-1  The Default file_roots Setting

Although commented, these display the default settings for the file_roots, the location used by the Salt Minions as the file server to retrieve States and files. The environment base: represents the default environment. We could add other environments, perhaps presenting a test or development environment. In this way, we can deploy State files that are not quite polished enough for the main production environment of base:. This environment is followed by a list of directories; here, we have the single entry /srv/salt, but we could have more than a single directory in the list should we need. We will see examples of this later when adding in Salt Formulas.

Configuring File Roots

Having our own explicit setting allows us to be more aware of the default and is more easily adjusted should the defaults need to be changed. We will also become familiar to Salt configuration files being in YAML format (Listing 7-2).

Note   Don’t forget that when editing files with Vim, we can temporarily associate the .conf file with YAML using ESC :set ft=YAML.

vagrant@master:~$ sudo cp .vimrc /root/

vagrant@master:~$ sudo vim /etc/salt/master.d/file_roots.conf

file_roots:

base:

- '/srv/salt/'

vagrant@master:~$ sudo systemctl restart salt-master

Listing 7-2  Configuring file_roots on the Salt Master

From the demonstration, you can see that we made sure that we copied the previously created .vimrc file to the root user’s home directory. The same would apply to the .nanorc file if using the nano text editor. On editing the file, we need to ensure that we indent two spaces before the base entry and four spaces before the list entry - '/srv/salt'. It is not required to quote strings, but it is good practice, ensuring that we don’t forget to quote strings that DO require quoting. These include strings containing special characters that are interpreted by YAML with special meaning. In general, editing the Salt Master configuration will require a restart of the service.

We created the directory /srv/salt earlier using a Remote Execution module, file.mkdir. If the directory does not exist on your system, you can create it using the command sudo mkdir /srv/salt executed from the Salt Master.

Creating Salt State Files

Having a directory to house our State files is going to allow us to create our first State file. As noted, we require that all of our systems are in the correct time zone. For me, this is Europe/London, but you can configure the correct time zone for your location. We will NOT be using the same Remote Execution module; we will make use of State modules within State files. These are found in the salt/states subdirectory of Python (Listing 7-3).

vagrant@master:~$ find /usr/lib -type f -name 'timezone.py'

/usr/lib/python3/dist-packages/salt/states/timezone.py

/usr/lib/python3/dist-packages/salt/modules/timezone.py

Listing 7-3  Locating the timezone.py Module

Help is accessed in a slightly different way, using now sys.state_doc:

vagrant@master:~$ sudo salt-call sys.state_doc timezone.system

...

Figure 7-1  Similarly, Help Is Accessed from the Web Index Using S and this is shown in Figure 7-1

Configuring the Time Zone from a State File

Using vim or nano, we can create the timezone.sls file ensuring correct configuration on our systems. We provide two examples. It is the second example that we use as, although more verbose, it provides human-readable explanation of what is being configured (Listing 7-4).

vagrant@master:~$ sudo vim /srv/salt/timezone.sls

'Europe/London':

timezone.system:

- utc:

Listing 7-4  Sparse SLS File Configuration of Time Zone (Not So Good)

The first line of the SLS file defines the State ID – the name of the State in the State file that can contain many States. The State ID can also be used as the - name: mapping used in the function, effectively acting as a shortcut allowing as to omit - name:. This though does not help describe the purpose of the State. Ideally a nontechnical person should be able to read the State ID and gain understanding of what it should be doing. Compare the first example with Listing 7-5, a more verbose version.

vagrant@master:~$ sudo vim /srv/salt/timezone.sls

'Configure UK Time Zone on Salt Minions':

timezone.system:

- utc: True

- name: 'Europe/London'

Listing 7-5  Verbose SLS File Configuration of Time Zone (Incredibly Good)

The State ID is more verbose now, describing fully the purpose of the State. We now are required to add the specific time zone that we wish to use now using the - name: mapping. In both files, we have the mapping -utc:, which is used to ensure that the hardware clock on the system is set to UTC time. This should be correct no matter the geographic location of the system.

Applying State Files

The State file we created must exist in our configured file_roots, in the directory /srv/salt. To apply the State file, we use a Remote Execution module state.sls referencing the name of the State file and omitting the extension .sls (Listing 7-6).

vagrant@master:~$ tree /srv/salt

/srv/salt

└── timezone.sls

vagrant@master:~$ sudo salt '*' state.sls timezone

master:

----------

ID: Configure UK Time Zone on Salt Minions

Function: timezone.system

Name: Europe/London

Result: True

Comment: Timezone Europe/London already set, UTC already set to Europe/London

Started: 11:49:22.122513

Duration: 8.774 ms

Changes:

Summary for master

------------

Succeeded: 1

Failed:    0

------------

Total states run:     1

Total run time:   8.774 ms

minion1:

----------

ID: Configure UK Time Zone on Salt Minions

Function: timezone.system

Name: Europe/London

Result: True

Comment: Timezone Europe/London already set, UTC already set to Europe/London

Started: 11:49:22.210925

Duration: 24.198 ms

Changes:

Summary for minion1

------------

Succeeded: 1

Failed:    0

------------

Total states run:     1

Total run time:  24.198 ms

Listing 7-6  Applying the Time Zone State File

Wow! This output is verbose. Looking at the details though, we can see that we go way beyond the Remote Execution commands, and the Comment: entry explains what was done, or in this case, both systems met the desired configuration state. Additionally, having set a verbose State ID, we can see what is happening by reviewing the ID: entry. Configuration management systems are idempotent, meaning that you can apply the same State many times and you will always receive the same result whether the configuration is in place or not.

Previewing Actions

Previewing the resulting actions before applying changes is a useful feature of Salt. Rather than making the changes, we are able to see what would have happened had we not been in a preview mode. We can change the time zone on the local Ubuntu Salt Minion and then preview what would happen if we applied the configuration, a kind of sanity check if you like (Listing 7-7).

vagrant@master:~$ sudo timedatectl set-timezone 'America/Denver'

vagrant@master:~$ sudo salt '*' state.sls timezone test=True

master:

----------

ID: Configure UK Time Zone on Salt Minions

Function: timezone.system

Name: Europe/London

Result: None

Comment: Timezone Europe/London needs to be set

Started: 13:19:15.705871

Duration: 8.477 ms

Changes:

Summary for master

------------

Succeeded: 1 (unchanged=1)

Failed:    0

Listing 7-7  Previewing Changes

Within the demonstration, we only display output from the master Salt Minion as it is the only system needing changes. The comment shows that the time zone needs to be configured but it has NOT been configured. When we are satisfied that the change is what we want, we can implement the file again, omitting the test=True option.

Salt Rendering Engines

The default rendering engine in Salt is to process Jinja statements and variables first and piping the output of Jinja to YAML. This can be set in a header known as the shebang . Jinja is a templating language that we will investigate in more detail in a later chapter. Even though we have not included any Jinja language elements, the Jinja renderer will still load and render the document with Jinja unless we will tell it not to. We can save a very small amount of processing time by explicitly setting YAML-only rendering. Although the CPU time saved will be negligible, it is still better to explicitly specify the renderer that is required. Not only does this save CPU time but we help document the file, stating that it is written in native YAML. To render only in yaml rather than !#jinja|yaml, we add the explicit shebang line as the topmost line in the SLS file (Listing 7-8).

vagrant@master:~$ sudo vim /srv/salt/timezone.sls

#!yaml

'Configure UK Time Zone on Salt Minions':

timezone.system:

- utc: True

- name: 'Europe/London'

Listing 7-8  Setting the Rendering Engine

Note   The shebang is written entirely in lowercase.

Applying State Files Locally

Having seen how States can record and replay configuration requirements, we should be able to understand better the operation of salt-call and local operations. Firstly, working on the CentOS Salt Minion, minion1, we run salt-call without any additional options other than the function:

[vagrant@minion1 ~]$ sudo salt-call state.sls timezone

Although the State file is not local, salt-call requests the timezone.sls file from the Salt Master file server. The request is sent to TCP port 4506 on the Salt Master. To operate in a true masterless environment, the State file must be in the local file_roots directory and the --local option used. To copy the file from the Salt file server to the Minion, we can use the salt-cp command on the Salt Master. The directory /srv/salt must exist on the CentOS Minion. This too would have been created earlier if you followed the demonstrations (Listing 7-9).

Listing 7-9. salt-call Executing State Locally on the Salt Minion

vagrant@master:~$ sudo salt-cp minion1 \

/srv/salt/timezone.sls /srv/salt/timezone.sls

minion1:

----------

/srv/salt/timezone.sls:

True

[vagrant@minion1:~]$ tree /srv/salt

/srv/salt

└── timezone.sls

[vagrant@minion1 ~]$ sudo  salt-call state.sls timezone --local

local:

----------

ID: Configure UK Time Zone on Salt Minions

Function: timezone.system

Name: Europe/London

Result: True

Comment: Timezone Europe/London already set, UTC already set to Europe/London

Started: 14:00:56.872754

Duration: 18.577 ms

Changes:

Summary for local

------------

Succeeded: 1

Failed:    0

------------

Total states run:     1

Total run time:  18.577 ms

Once the State file is available locally, then we can use the option --local to salt-call to ensure that all operations are isolated on the Minion with no communications back to the server. The operation would fail if the file was not found. Provided that the State files are available, then this becomes a viable build option for new systems without the need of the Salt Master. Having decentralized State files is not ideal for future developments but can ensure that a system is built to the required standard.

Configuring Packages and Services

Having already seen that we can install software packages and manage services using Salt with Remote Execution modules, we will now learn to do the same with States. First, we look at installing some standard packages, which we would want on all Linux systems. This is likely to be many packages, and we can use - names: as a list mapping rather than a single mapping or scalar mapping as we have used before with -name: (Listing 7-10).

vagrant@master:~$ sudo vim /srv/salt/software.sls

'Install Common Software Packages Salt Minions':

pkg.installed:

- names:

- 'nano'

- 'bash-completion'

- 'tree'

- 'git'

vagrant@master:~$ sudo salt '*' state.sls software

...

Listing 7-10  Installing Common Software Packages

Note   The function is installed with the State module and install with the Remote Execution pkg module.

We also know that we wanted the package at installed along with the associated atd service being enabled and started. We will do this in another State file as we will manage both the package and service on the one file (Listing 7-11).

vagrant@master:~$ sudo vim /srv/salt/at.sls

#!yaml

'Install at and start the associated atd service':

pkg.installed:

- name: 'at'

service.running:

- name: 'atd'

- enable: True

vagrant@master:~$ sudo salt '*' state.sls at

...

Listing 7-11  Managing the at Command Scheduler with Salt States

With this State file, we have a single State using two independent functions. The two functions will appear at the same two-space indent from the left margin.

Configuring Files Using State Files

When configuring our Salt Minions using Remote Execution modules, we also wanted to ensure that password authentication was enabled on the SSH server. In Vagrant systems, it is disabled by default. This too can be enforced reliably from a State file (Listing 7-12).

vagrant@master:~$ sudo vim /srv/salt/ssh.sls

#!yaml

'Install Open SSH Server':

pkg.installed:

- name: 'openssh-server'

'Ensure SSH Server is Running and Enabled':

service.running:

- name: 'sshd'

- enable: True

- watch:

- file: '/etc/ssh/sshd_config'

'Ensure SSH Password Authentication is enabled':

file.replace:

- name: '/etc/ssh/sshd_config'

- pattern: '^PasswordAuthentication no'

- repl: 'PasswordAuthentication yes'

- append_if_not_found: True

vagrant@master:~$ sudo salt '*' state.sls ssh

Listing 7-12  Configuring SSH Using State Files Part 1

In this file, we have three independent States. They are listed in the following:

- Ensure SSH is installed: A simple package install of the openssh-server.

- Ensure service is running: Even though this is listed second, it will run after the file that it is watching. The watch: mapping will restart the service if the configuration is modified by the referenced State file.

- Edit the configuration file: We look for password authentication being disabled and enable it. Changes to this file will automatically cause the SSH service to restart due to the previously mentioned watch mapping in the service.

Changing the configuration file locally on the Ubuntu Salt Master, we can verify that this is working correctly. The file should be changed back by applying the State file, and the service should be restarted. For ease we use sed to edit the file locally (Listing 7-13).

vagrant@master:~$ sudo sed -i \

's/^PasswordAuthentication yes/PasswordAuthentication no/' \

/etc/ssh/sshd_config

vagrant@master:~$ sudo systemctl restart sshd

vagrant@master:~$ sudo salt '*' state.sls ssh

master:

----------

ID: Install Open SSH Server

Function: pkg.installed

Name: openssh-server

Result: True

Comment: All specified packages are already installed

Started: 17:17:17.766719

Duration: 20.813 ms

Changes:

----------

ID: Ensure SSH Password Authentication is enabled

Function: file.replace

Name: /etc/ssh/sshd_config

Result: True

Comment: Changes were made

Started: 17:17:17.790517

Duration: 4.662 ms

Changes:

----------

diff:

---

+++

@@ -55,7 +55,7 @@

#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!

-PasswordAuthentication no

+PasswordAuthentication yes

#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with

----------

ID: Ensure SSH Server is Running and Enabled

Function: service.running

Name: sshd

Result: True

Comment: Service restarted

Started: 17:17:17.818033

Duration: 235.547 ms

Changes:

----------

sshd:

True

Summary for master

------------

Succeeded: 3 (changed=2)

Failed:    0

------------

Total states run:     3

Total run time: 261.022 ms

Listing 7-13  Configuring SSH Using State Files Part 2

This is beginning to look totally amazing. The output shown is only for the master Salt Minion, but we can see the edit to the file and that the service is restarted. If you run the State again, no changes are required, and the service is not restarted.

Debugging State Files

We have previously seen that we can use the option test=True to the state.sls function. This will also let us know of any syntax errors that we need to resolve. We can also use $ sudo salt-call --local state.show_sls ssh to test the ssh.sls file locally on the Salt Minion running on the Salt Master. Running salt-call on the Minion running on the Master ensures that the ssh.sls is available to the local Salt Minion. We can also install the salt-lint Python script to check for style violations. These will not cause errors, but we should try to adhere to style guidelines by not adding trailing whitespace to lines and having a clear empty line at the end of an SLS file. No output from the linter is good, but add the option -v to display a check status (Listing 7-14).

vagrant@master:~$ sudo pip3 install salt-lint

vagrant@master:~$ salt-lint -v /srv/salt/*.sls

Examining /srv/salt/at.sls of type sls

Examining /srv/salt/software.sls of type sls

Examining /srv/salt/timezone.sls of type sls

Examining /srv/salt/ssh.sls of type sls

Listing 7-14  Linting Salt State Files

It seems that all my files were created matching style recommendations. There has to be a first time , but if I re-edit one of the files and add a trailer space at the end of a line, we can see the linter at its best (as shown in Listing 7-15).

vagrant@master:~$ salt-lint -v /srv/salt/*.sls

Examining /srv/salt/software.sls of type sls

Examining /srv/salt/at.sls of type sls

Examining /srv/salt/ssh.sls of type sls

Examining /srv/salt/timezone.sls of type sls

[201] Trailing whitespace

/srv/salt/ssh.sls:3

pkg.installed:

Listing 7-15  Linting Salt State Files Not Matching Style Guidelines

Reviewing the output, we easily see that it was the ssh.sls file that was edited and line 3 of that file has the trailing whitespace. The erroring line is also displayed for ease of editing. I can now re-edit the file, removing the extra space at the end of line 3.

Using Salt Formulas

There is also a huge resource of community-created content that you can use without having to write code. Now you are interested! Salt Formulas are maintained in GitHub and can be found at https://github.com/saltstack-formulas. Although you can configure SaltStack to use the GitHub repository directly, for simplicity, we will download the code locally and add to our file_roots mapping on the Salt Master. We will learn that using the vim-formula we can configure the vim text editor for SLS files using prewritten configurations. We can also ensure that vim is installed on both CentOS and Ubuntu. In Ubuntu the package is vim and in CentOS it is vim-enhanced. The formula will handle these differences for us using Jinja, and we don't need to lift a finger (Listing 7-16).

Create directory for formulas

vagrant@master:~$ sudo mkdir /srv/formulas

Move into the directory

vagrant@master:~$ cd /srv/formulas

Clone the code from the repository

vagrant@master:formulas$ sudo git clone \

https://github.com/saltstack-formulas/vim-formula.git

Add the vim directory to the file_roots of the Salt Master

vagrant@master:formulas$ sudo vim /etc/salt/master.d/file_roots.conf

file_roots:

base:

- '/srv/salt/'

- '/srv/formulas/vim-formula'

Don't forget to restart the Salt Master

vagrant@master:formulas$ sudo systemctl restart salt-master

Applying the vim/salt state file from the /srv/formulas/vim-formula directory will install vim and add specifications for SLS files. The period is used to separate directories rather than the slash in the state hierarchy

vagrant@master:formulas$ sudo salt '*' state.sls vim.salt

We can now delete the root users customizations using the one added by Salt

vagrant@master:/srv/formulas$ sudo rm /root/.vimrc

Finally we test that SLS file editing still works with vim

vagrant@master:/srv/formulas$ sudo vim /srv/salt/ssh.sls

Listing 7-16  Using Salt Formulas

Summary

This is where you want to be a DevOps engineer, sitting with a beer at the end of a long day, admiring what you have achieved without needing to put hours of overtime in. You have this covered.

Having grasped YAML amazingly well, we took you straight into using Salt State files and configuring the systems reliably and in a well-documented way. We were able to configure the time zone, install software, manage services, and edit service configurations. Not only this, we also saw how we could test the State files using test=True to prevent little accidents.

Dropping into the Salt Master configuration, we explicitly set the file_roots mapping, becoming especially useful when needing to add the formulas directory later in the chapter.

When managing service configuration files, we can tell a service to watch a State file for changes to enforce a restart of the service when a file changes. We used this to manage the SSH server configuration to allow password-based authentication. Changing the configuration would need the restart of the SSH service.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_8

8. Building an Effective State Tree

Andrew Mallett[1]

(1)

Peterborough, UK

Having already mastered State files, we have a collection of files that do need to be applied and applied reliably. The difficulty that we now face is being able to apply the correct State to the correct system and in the correct order. I am already starting to stress. Far from solving the issues created by Remote Execution commands, we just seem to be facing the same problems but more so as the system has grown along with our confidence. As we start to do more with Salt, we will collect more and more State files, excited by what is possible, and this system is going to rapidly become an absolute mess. We need to get a housekeeper!

Well, thank you, Andrew. You have led me down this avenue, and it now has become a jungle path! Don’t worry. We will now learn to organize the content within the file roots and design a State top file. We will learn to design and implement an effective State tree hierarchy, and NO, we don’t need a housekeeper. We will find out in this chapter how to

- Create a top file.

- Target Minions within a top file.

- View assigned State files.

- Organize related content.

- Apply multiple State files.

Salt Top File

Creating a top file or the top.sls within the file roots, we can easily assign and apply multiple State files in a single action. Currently, we can list the /srv/salt directory on the Salt Master seeing that we have four State files. These files need to be applied to all Salt Minions that we have, at least so far. We can target all Minions from the command line, and that is valid now, but perhaps not later if we employ Windows Minions. Making use of the top file, we can further drill down to Salt Minions that we need to target differently (Listing 8-1).

root@master:~# ls -l /srv/salt

total 16

-rw-r--r-- 1 root root 172 May 25 15:08 at.sls

-rw-r--r-- 1 root root 150 May 25 14:56 software.sls

-rw-r--r-- 1 root root 450 May 25 18:46 ssh.sls

-rw-r--r-- 1 root root 112 May 25 14:57 timezone.sls

Listing 8-1  Current State on the Salt Master

Simple Top File

The top.sls is the top file and should be located in the root directory of the file roots for the Salt Master. For us this will be /srv/salt. Within the top file (Listing 8-2), we define three items:

1. 1.
Environment: By default, we have the base or our main production environment.

2. 2.
Target: The Minions that we want to target and possibly the targeting method.

3. 3.
State files to apply: A list of State files.

#Working as root

root@master:~# vim /srv/salt/top.sls

#!yaml

base:

'*':

- at

- software

- ssh

- timezone

Listing 8-2  Example Simple Top File

Within this file, we assign all of the State files to all Minions using the globbing and default targeting method. This works for us just now, but it is not the best mechanism. We know we can do better and stay accurate.

Viewing the Assigned States

Having made the assignment in the /srv/salt/top.sls file, we can verify that this is correct. We could always apply the top file. Think about it. That could be disastrous. Avoiding potential disaster is always a good option. Independently viewing the State assignment is possible and good practice, using the well-used state module, but this time using the show_states function (Listing 8-3).

root@master:~# salt '*' state.show_states

minion1:

- at

- software

- ssh

- timezone

master:

- at

- software

- ssh

- timezone

Listing 8-3  Viewing State Assignment

Reviewing the output, we easily see that we have no stray Minions. We still just have the Ubuntu master and the CentOS minion1. They have the correctly assigned States. Within the top file though, we do run a risk by using the global assignment of '*'; what if we bring in Windows Minions? We might want to change this to a grain that targets just Linux systems. The grain, kernel, allows us to target correctly Linux systems (Listing 8-4).

root@master:~# salt -G 'kernel:Linux' --preview-target

- minion1

- master

Listing 8-4  Targeting Using the Kernel Grain

Using this grain, we can more accurately target the required systems. Reflecting this, we can re-edit the top.sls (Listing 8-5).

root@master:~# vim /srv/salt/top.sls

#!yaml

base:

'kernel:Linux':

- match: grain

- at

- software

- ssh

- timezone

root@master:~# salt '*' state.show_states

master:

- at

- software

- ssh

- timezone

minion1:

- at

- software

- ssh

- timezone

Listing 8-5  Targeting More Accurately

Note that we now must include the match type as we move away from the default globbing. Valid values that we can use with the match include

- glob

- pcre

- grain

- grain_pcre

- list

- pillar

- pillar_pcre

- pillar_exact

- ipcidr

- data

- range

- compound

- nodegroup

Organizing Content

The next matter to address is the amount of content that we have. So far, we have just four State files, but this could grow. Organizing the content based on the purpose will help the growth stay manageable. As our current State files are, vaguely, common content, we can create a common subdirectory for this. To make the assignment less verbose in the top file, we can assign the directory to Minions, requiring that we have a file named init.sls in the directory. Let’s take a look, but be prepared we have a few tasks to complete. Perhaps a coffee break is required but gibe Listing 8-6 a go!

Create the new directory

root@master:~# mkdir /srv/salt/common

Create the init.sls including files that we need

root@master:~# vim /srv/salt/common/init.sls

#!yaml

include:

- common.at

- common.software

- common.timezone

- common.ssh

Move States into the common directory, exclude the top file

root@master:~# mv /srv/salt/[!t]*.sls /srv/salt/common

We can mode the timezone independently

root@master:~# mv /srv/salt/timezone.sls /srv/salt/common

The tree command can list the contents of the file roots

root@master:~# tree /srv/salt

/srv/salt

├── common

│   ├── at.sls

│   ├── init.sls

│   ├── software.sls

│   ├── ssh.sls

│   └── timezone.sls

└── top.sls

Editing the Top File

root@master:~# vim /srv/salt/top.sls

#!yaml

base:

'kernel:Linux':

- match: grain

- common

root@master:~# salt '*' state.show_states

master:

- common.at

- common.software

- common.timezone

- common.ssh

minion1:

- common.at

- common.software

- common.timezone

- common.ssh

Listing 8-6  Organize Content

A few things to point out. Within the init.sls file, we need to specify the path through to the included files, as they are not within the root directory. The path separator we use is the dot and not the forward slash.

Note   We could also use .at instead of common.at and so forth, if we prefer.

The init.sls file is used to represent the directory common. We can see, though, that the structure is tidier and the top file is kept tidy. Moving forward, we have a solid platform on which to build our Salt Solution. “But wait!” I hear you cry. “We haven’t applied the changes.” How do we apply the top file? This is simple, and we can use state.apply or state.highstate over state.sls (Listing 8-7).

root@master:~# salt '*' state.apply

OR

root@master:~# salt '*' state.highstate

Listing 8-7  Apply the Highstate

Debugging the State Tree

Using our newly designed tree structure, we can investigate the use of the assignment of the common directory, which is obviously not an .sls file. Debugging like this is often best performed with salt-call so that we are able to see how the Minion accesses the State tree. The debug output will be verbose, but toward the top of the output, we should be able to see the resolution of the name common.sls:

root@master:~# salt-call state.apply -l debug

[DEBUG   ] Could not find file 'salt://common.sls' in saltenv 'base'

[DEBUG   ] In saltenv 'base', looking at rel_path 'common/init.sls' to resolve 'salt://common/init.sls'

Being able to debug potential issues within State files in this way is always going to be a skill worth learning, so please take the time to practice this.

Summary

Now we really are starting to make progress having organized the State file content. No longer do we have a mess of State files at the root of the /srv/salt directory. We have tidied up our sock drawer, albeit metaphorically, and the content now is something to be proud of.

Creating the top file, top.sls, we can easily apply configurations to "*" (all Salt Minions) without fear of issues. Targeting further inside of the top.sls file, we can assign the correct State files to the correct systems. Using the kernel grain, we are able to identify all Linux platforms for deployment of common elements. As your system grows, targeting becomes crucial.

Using a single State file, we are able to use the include: statement to run other States, keeping assignments simple and as tidy as the rest of the system. Starting your Salt deployment in an organized manner early on saves a lot of work refactoring later. We caught this just in time. Well done.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_9

9. Creating Reusable State Files

Andrew Mallett[1]

(1)

Peterborough, UK

So far, we have seen that Salt has been pretty easy to use, the underlying Python subsystems helping us differentiate the various needs of the underlying OS. However, we have chosen carefully to install packages with matching names on both Ubuntu and Red Hat–based systems, like CentOS Stream. We will have encountered more problems ensuring that the Vim text editor was installed. The package to install is simply vim on Debian-based systems and vim-enhanced on Red Hat–based systems. This extends into many services too; if we want to install the Apache web server, we need the packages apache2 on Debian and httpd on Red Hat. The service names also differ on each system. For simplicity, we have avoided these issues, but it is time to face these problems head-on and see how using templating languages like Jinja can help to resolve the issues. We will also learn how we can use other renderers and discover that we can write our State files in Python directly.

We will find out in this chapter how to

- Create custom grains.

- Use Pillar data.

- Include Jinja within State files.

- Review rendered data.

- Simplify templates using maps.

- Deploy Apache web server across Ubuntu and CentOS Stream.

- Write Python States.

Working with Grains

Grains can be used for targeting, as we have already seen, but we can also use Salt Grains within templates, allowing us to further cater for differences. Creating custom grains can really help by removing the need for logic statements within the State files. The os_family grain is always a good starting point when dealing with Linux differences and allows us to produce even simpler States (Listing 9-1).

root@master:~# salt '*' grains.item os_family

minion1:

----------

os_family:

RedHat

master:

----------

os_family:

Debian

Listing 9-1  Reviewing the os_family Grain

Using Simple Jinja Logic

The CentOS Stream Minion is listed first and is part of the Red Hat family, and the Ubuntu Minion is part of the Debian family. Package and service differences will often be resolvable by using this Grain value. Using conditional statements in a State file, we can install the Vim text editor across both systems even though the package name differs between Ubuntu and CentOS Stream. These conditional statements are written in Jinja, the default rendering engine in Salt. This is written in the shebang as #!jinja|yaml (Listing 9-2). We can exclude the shebang line when using the defaults, but we have to include it as the first line where we want to use different rendering engines. Later we will see using a YAML-only shebang as well as native Python.

root@master:~# vim /srv/salt/common/vim.sls

#!jinja|yaml

{% if salt['grains.get']('os_family') == 'Debian' %}

{%  set pkg = 'vim' %}

{% elif salt['grains.get']('os_family') == 'RedHat' %}

{%  set pkg = 'vim-enhanced' %}

{% else %}

{%  set pkg = 'vim' %}

{% endif %}

'Install Vim text editor':

pkg.installed:

- name: {{ pkg }}

Listing 9-2  Using Jinja Logic Statements to Install Vim

In this example, we use the full logic statement and cater for Debian and Red Hat systems. We default to using the package name vim if we are connecting from another os_family. Reading through the file, we can explain the State file line by line:

#!jinja|yaml: The default shebang line reading jinja and piping to YAML. Remember that the default rendering engine is jinja|yaml so this just aids the file documentation

{% if salt['grains.get']('os_family') == 'Debian' %}: Jinja statements start with {% and end with %}. Each statement is on a single line. In this condition we check the os_family grain being equal to Debian

{%  set pkg = 'vim' %}: If the statement is true, then we set the variable pkg with the value vim

{% elif salt['grains.get']('os_family') == 'RedHat' %}: We now check the grain to be equal to RedHat

{%  set pkg = 'vim-enhanced' %}: The RedHat package name should be set to vim-enhanced

{% else %}: If we have not matched a value so far, we can use the else condition

{%  set pkg = 'vim' %}: Where we set the pkg variable to be vim in the else block we are not from Debian or RedHat and we will try to install the package named vim

{% endif %}: The if statement is closed with endif

- name: {{ pkg }}: The name of the package to be installed within the YAML State is set to the variable. Jinja variables use the double brace-bracket notation.

This does look quite verbose, and it can be annoying adding each Jinja statement, but it is self-contained as it does work! We have stored this file within the common subdirectory, so the sls file is referenced as common.vim. We can test the deployment first allowing us to see the package names prior to any action:

root@master:~# salt '*' state.sls common.vim test=True

root@master:~# salt '*' state.sls common.vim

Using the Vagrant systems, Vim is already installed on Ubuntu but not on the CentOS Stream system. We should see that the package vim is already installed on the master node, and the package vim-enhanced will be installed on the CentOS Stream node. If the test was successful, we can proceed with the proposed changes by executing again without the test option.

Streamlining Jinja Logic

Finding out better ways to write code is always a challenge and the illusive nirvana of coding in any language. Starting with a simpler example is great. In general, simple files are easier to read even if a little verbose. Streamlining the code begins by simplifying the logic statement by using grains.filter_by . The lookup requires a supplied dictionary and defaults to work with the os_family grain . The dictionary is denoted using the single brace brackets as the first supplied argument to the filter_by function (Listing 9-3). A dictionary is a collection of keys and values, the keys being the values from the os_family lookup and the values being the package that we want to set for each family. We have supplied only the dictionary as an argument to the filter_by function, meaning that we would look up the values of the os_family grain . If we needed to work with another grain, we would need to supply this as another argument.

root@master:~# vim /srv/salt/common/vim.sls

#!jinja|yaml

{% set pkg =  salt['grains.filter_by']({

'Debian': 'vim',

'RedHat': 'vim-enhanced'}) %}

'Install Vim text editor':

pkg.installed:

- name: {{ pkg }}

Listing 9-3  Implementing Streamlined Lookups Using filter_by

The code is simpler to read and less verbose, allowing us to discard the cumbersome if statement; however, we are still able to maintain the independence of our code and simplify it one step further using map files.

Using YAML Map Files

Describing the mapping of the os_family key to its value requires the Jinja dictionary and the use of brace brackets. Those who are conversant with scripting languages may find this very comfortable, but for others, this is just another hurdle. Understanding YAML, we know that we can create dictionaries easily without the need of braces. Creating an external YAML map and importing it takes our code to the next level of simplicity. Listing 9-4 looks at our version 3 of this file, creating the new map file and editing the existing SLS file.

root@master:~# vim /srv/salt/common/vim-map.yaml

Debian: vim

RedHat: vim-enhanced

root@master:~# vim /srv/salt/common/vim.sls

#!jinja|yaml

{% import_yaml 'common/vim-map.yaml' as osmap %}

{% set pkg =  salt['grains.filter_by'](osmap) %}

'Install Vim text editor':

pkg.installed:

- name: {{ pkg }}

Listing 9-4  Using YAML Maps

Finalizing the use of independent code, where we have a self-contained solution without relying on external data from the Salt Minion, we have the State file and the YAML file, creating a streamlined and effective solution. The code is very concise and allows easy editing and growth; adding a new os_family is a simple edit of the YAML file. We have created something very powerful here.

Implementing Custom Grains

Having seen the solution of installing the package vim across our different Linux distributions using independent code that does not need any reliance on external data, we will look at using custom Grains. Later we will see something similar using Pillar data. Even though Salt Grains were used in the previous example, we used the standard os_family Grain, which we know is going to be present on all systems. Customs Grains have to be created, and, as such, we are reliant on the data being present on the Salt Minion. However, using custom Grains provides for a simple solution in the coding of our enforcements. We will look at a different scenario now learning to install Apache across both Salt Minions.

Creating a File System Structure

Firstly, we will need to organize the content; we have our common content, and we now need the web content. Creating the /srv/salt/www directory will allow this organizational structure we should always consider:

root@master:~# mkdir -p /srv/salt/www/files

root@master:~# echo "My Site" > /srv/salt/www/files/index.html

As well as creating the directory www, we create the files subdirectory where we can add the web content for the site.

Saving Custom Grains

Needing to install and manage the Apache web server, we have to install either the apache2 or httpd package based on the distribution, and the services have the same name as the packages. We need to ensure the service is enabled and running as well as delivering the content. To cater for the differences without the need of logic in the State file, we can use custom Salt Grains. Making use of the grains.setval function and ensuring we target the correct system, we are able to set the required values as we demonstrate in the following (Listing 9-5).

root@master:~# salt -G 'os:Ubuntu' grains.setval apache_pkg apache2

master:

----------

apache_pkg:

apache2

root@master:~# salt -G 'os:Ubuntu' grains.setval apache_svc apache2

master:

----------

apache_svc:

apache2

root@master:~# salt -G 'os:CentOS Stream' grains.setval apache_pkg httpd

minion1:

----------

apache_pkg:

httpd

root@master:~# salt -G 'os:CentOS Stream' grains.setval apache_svc httpd

minion1:

----------

apache_svc:

root@master:~# salt  '*' grains.item apache_svc apache_pkg

master:

----------

apache_pkg:

apache2

apache_svc:

apache2

minion1:

----------

apache_pkg:

httpd

apache_svc:

httpd

Listing 9-5  Saving Custom Grains

Having set the custom Grains, they are immediately available to use and are persisted in the file system of the targeted Salt Minion. Custom Grains are stored in the file /etc/salt/grains. We can use Salt to read the file from each Salt Minion ensuring we have configured this correctly (Listing 9-6).

root@master:~# salt '*' cmd.run 'cat /etc/salt/grains'

master:

apache_pkg: apache2

apache_svc: apache2

minion1:

apache_pkg: httpd

apache_svc: httpd

Listing 9-6  Verify Custom Grains

Implementing Custom Grains

Having expertly curated the custom Grains, we can see how we can use these Grains in reusable State files. The importance to us is that we do not need the complex logic within the State now. We can discard much of the ugly Jinja code. Ideally, creating State files that do minimal tasks allows for the best modularity of the enforcement application. We will create individual State files to install Apache, manage the service, and deliver the web content, allowing the desired modularity (Listing 9-7).

root@master:~# vim /srv/salt/www/install.sls

#!jinja|yaml

'Install the Apache Web Server':

pkg.installed:

- name: {{ grains.apache_pkg }}

root@master:~# salt '*' state.sls www.install test=true

root@master:~# vim /srv/salt/www/service.sls

#!jinja|yaml

'Ensure the Apache Web Server is running and enabled on Startup':

service.running:

- name: {{ grains.apache_svc }}

- enable: True

root@master:~# salt '*' state.sls www.service test=true

root@master:~# vim /srv/salt/www/content.sls

#!jinja|yaml

'Deliver web content':

file.managed:

- name: '/var/www/html/index.html'

- source: 'salt://www/files/index.html'

root@master:~# salt '*' state.sls www.content test=true

Listing 9-7  Creating the Apache Application and Implementing a Simple Test

Adding Reliability to Our Application Using Require Statements

Creating each of the three required State files allows us to independently manage each component – this was our stated objective. Testing the solution using the test=True option displayed what would happen, showing the correct variables are being used and the States have been created correctly. However, if we tried to apply the service.sls without the test before the package has been installed, it would generate errors and fail. We can add reliability to the application by requiring the package to be installed before we manage the service or add web content. This dictates the need for us to include the install.sls as any requirements have to exist in the execution context of the running State. This sounds very complex, but in reality, this is extremely easy, as shown in Listing 9-8.

root@master:~# vim /srv/salt/www/service.sls

#!jinja|yaml

include:

- .install

'Ensure the Apache Web Server is running and enabled on Startup':

service.running:

- name: {{ grains.apache_svc }}

- enable: True

- require:

- pkg: {{ grains.apache_pkg }}

Listing 9-8  Adding Requirements to the Service

Using the include statement, we can either put the full path of www.install or the relative path of .install where the install.sls is in the same directory as the service.sls. When requiring the pkg:, we can use either the State ID or the name key of the package. As we have included the install.sls within the service.sls, the package has been made available to the execution context. The reality is we also need to protect the execution of the contents.sls by ensuring that the content is only added after the package is installed, even when the content is accessed independently of the install.sls (Listing 9-9).

root@master:~# vim /srv/salt/www/content.sls

#!jinja|yaml

include:

- .install

'Deliver web content':

file.managed:

- name: '/var/www/html/index.html'

- source: 'salt://www/files/index.html'

- require:

- pkg: {{ grains.apache_pkg }

Listing 9-9  Adding Requirements to the Contents

Note   It is always a good idea to test after each change, making sure that we can easily re-edit the file we have just changed if errors occur. Using state.sls on the edited State file with the test=True option is always a good way to verify the State (Listing 9-10).

root@master:~# salt 'master' state.sls www.content test=true

master:

----------

ID: Install the Apache Web Server

Function: pkg.installed

Name: apache2

Result: None

Comment: The following packages would be installed/updated: apache2

Started: 10:25:44.613750

Duration: 25.968 ms

Changes:

----------

ID: Deliver web content

Function: file.managed

Name: /var/www/html/index.html

Result: None

Comment: The file /var/www/html/index.html is set to be changed

Note: No changes made, actual changes may

be different due to other states.

Started: 10:25:44.642514

Duration: 26.528 ms

Changes:

----------

newfile:

/var/www/html/index.html

Summary for master

------------

Succeeded: 2 (unchanged=2, changed=1)

Failed:    0

------------

Total states run:     2

Total run time:  52.496 ms

Listing 9-10  Ensure Correct Requirement Enforcement

Linking the init.sls

Being able to independently manage the service, contents, and package is a useful feature; however, most often, we imagine that you would want the Apache server installed and started and the content added. Creating the init.sls file in /srv/salt/www allows the reference of the www State. Including the other States in the init.sls will give us that all-important centralized control (Listing 9-11).

root@master:~# vim /srv/salt/www/init.sls

#!yaml

include:

- .install

- .service

- .content

root@master:~# tree /srv/salt/www/

/srv/salt/www/

├── content.sls

├── files

│   └── index.html

├── init.sls

├── install.sls

└── service.sls

root@master:~# salt -N 'webservers' state.sls www

Listing 9-11  Creating the Init State File

Note   The webservers group was previously defined when we investigated the targeting methods that we could use. This is defined in the Salt Master configuration. If this is not complete on your system, then you can still use the ‘*’ assignment if you prefer.

Adding www to the top file is always a good idea ensuring that we dont need to explicitly apply the state (Listing 9-12). Future Salt Minions added to the webservers groups will automatically collect the correct configuration via the top file or highstate.

root@master:~# vim /srv/salt/top.sls

#!yaml

base:

'kernel:Linux':

- match: grain

- common

'webservers':

- match: nodegroup

- www

root@master:~# root@master:~# salt  '*' state.apply

Listing 9-12  Editing the Top File

Using Pillar Data

We have seen how using custom Grains can save us additional work in creating logic statements, needing only the Jinja variables. The downside in using Grains is that data is stored locally on the Salt Minion, allowing editing, either malicious or accidental. Where more Salt administrative control is required, Salt Pillar is the required option, storing data only on the Salt Master. Pillar data is transferred securely to each Salt Minions cache but is not persisted on the Salt Minion. The default pillar_root is the directory /srv/pillar but, like the file_roots, will not exist after a Salt installation. The Pillar top file makes the State assignment in much the same way as the standard top file as well see in Listing 9-13.

root@master:~# mkdir /srv/pillar

root@master:~# vim /srv/pillar/top.sls

#!yaml

base:

'os:Ubuntu':

- match: grain

- ubuntu-apache

'os:CentOS Stream':

- match: grain

- centos-apache

root@master:~# vim /srv/pillar/ubuntu-apache.sls

apache_pkg: apache2

apache_svc: apache2

root@master:~# vim /srv/pillar/centos-apache.sls

apache_pkg: httpd

apache_svc: httpd

root@master:~# salt '*' saltutil.pillar_refresh

root@master:/srv/pillar# salt '*' pillar.items

master:

----------

apache_pkg:

apache2

apache_svc:

apache2

minion1:

----------

apache_pkg:

httpd

apache_svc:

httpd

Listing 9-13  Configuring Pillar Data

Using Pillar data, not only is the security improved, but we only need to manage the data on the Salt Master making for easier long-term management. To change the State files to match Pillar data rather than custom Grains, I would recommend using the stream editor sed. This allows for noninteractive editing across all SLS files within the stated directory. Take a look at how awesomely easy this edit is, just replacing grains with Pillar data (Listing 9-14).

root@master:/srv/pillar# sed -i 's/grains/pillar/' /srv/salt/www/*.sls

root@master:/srv/pillar# grep 'pillar' /srv/salt/www/*.sls

/srv/salt/www/content.sls:    - pkg: {{ pillar.apache_pkg }}

/srv/salt/www/install.sls:    - name: {{ pillar.apache_pkg }}

/srv/salt/www/service.sls:    - name: {{ pillar.apache_svc }}

/srv/salt/www/service.sls:    - pkg: {{ pillar.apache_pkg }}

root@master:/srv/pillar# salt '*' state.apply

Listing 9-14  Editing SLS Files Using sed

Using Other Rendering Engines

We are not limited to using Jinja and YAML rendering engines . State files, though normally written in Jinja and YAML or even just YAML, can also be inscribed in pure Python as well as other languages. Pure Python is attractive because it matches the language executed by Salt and will be quickest and, additionally, is well known and supported. If you already write in Python, you may well be happy to create your States in Python, especially those needing Jinja statements. We will revisit the State file to install Vim, writing another version in Python (Listing 9-15).

root@master:~# vim /srv/salt/vim-python.sls

#!py

from distro import linux_distribution

os = linux_distribution()[0].lower()

if os == 'ubuntu':

editor = 'vim'

elif os == 'centos stream':

editor = 'vim-enhanced'

def run():

return {"Installing VIM": {

"pkg.installed": [

{ "name": editor }]

}}

Listing 9-15  Writing Pure Python State Files

Native Pure Performance

For performance, we will win 100% of the time using pure Python States as we are writing in the language of Salt; writing in both Jinja and YAML will require rendering first to YAML from Jinja and then from YAML to Python. Using Python, we are always good to go. Comparing the runtime of the original Jinja and YAML State to the Python State, we can see a slight performance benefit in using Python (Listing 9-16). In a more complex State, we would see more performance gains.

root@master:~# salt 'master' state.sls common.vim | grep 'run'

Total states run:     1

Total run time:  60.281 ms

root@master:~# salt 'master' state.sls vim-python | grep run

Total states run:     1

Total run time:  58.609 ms

Listing 9-16  Comparing Runtimes

View Rendering Actions

To observe the rendering process and see the time to render each language, we can use salt-call and the debug log level again. The output will be verbose, so scroll carefully to see the rendering actions (Listing 9-17).

Note   Python rendering does not show the results of rendering as we are already in the native language.

root@master:~# salt-call  state.sls common.vim -l debug

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/common/vim.sls' using 'jinja' renderer: 0.01957988739013672

[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/common/vim.sls:

#!jinja|yaml

'Install Vim text editor':

pkg.installed:

- name: vim

[DEBUG   ] Results of YAML rendering:

OrderedDict([('Install Vim text editor', OrderedDict([('pkg.installed', [OrderedDict([('name', 'vim')])])]))])

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/common/vim.sls' using 'yaml' renderer: 0.00038623809814453125

root@master:~# salt-call  state.sls vim-python -l debug

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/vim-python.sls' using 'py' renderer: 0.00018644332885742188

Listing 9-17  View Rendering Actions

Filtering a little further, we can see much less information by using the profile log level. This does not show the rendered output but does show the rendering time and is more easily read without the additional debug lines (Listing 9-18).

root@master:~# salt-call  state.sls common.vim -l profile

[PROFILE ] Time (in seconds) to render import_yaml 'common/vim-map.yaml': 0.004618167877197266

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/common/vim.sls' using 'jinja' renderer: 0.013263940811157227

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/common/vim.sls' using 'yaml' renderer: 0.0003185272216796875

root@master:~# salt-call  state.sls vim-python -l profile

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/vim-python.sls' using 'py' renderer: 0.0001227855682373047

Listing 9-18  View Rendering Times

Although these times are not huge, neither is the code that we have created. As your code grows, then so does the processing time. You do not need to know Python to use Salt, but it does help and allows you to tune your States.

Summary

This has been a real milestone in your learning, and I truly hope you have been impressed with your progress. You are now able to create reusable State files using code and variables. We have learned to write this in both Jinja and YAML as well as native Python. Like everything, this will need a lot of practice, so please take the time to work through the examples included as well as creating your own.

Learning the Jinja templating language, which is common to both Ansible and Salt, we were able to use both variables and flow control operation. We saw that using this allows us to manage different package names and service names. We also introduced the idea of the shebang to state the rendering languages used in the State.

Moving to Grains and Pillar data, we were able to ditch the complex control statements for lookups in the required subsystem. Custom grains are stored locally on the Salt Minion, and centrally, the Salt Master holds Pillar data. Where Python does not scare you, we are able to create State files written in Python, the language native to Salt. This will provide both speed and familiarity where we already know a little Python.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_10

10. Implementing Reactors and Beacons

Andrew Mallett[1]

(1)

Peterborough, UK

You have now progressed to the next level of Salt administration, having conquered configuration management, the key to your IT estate management. The next level is orchestration, where we can automate even more and become proactive and less reactive. We will begin by learning to view the event bus and the activity, which will be surprisingly minimal. Being able to view events helps us debug Salt, giving us another alternative view from under the hood of the Salt operation. As a simple next step, we will learn the schedule activity on our Salt Minions before moving on to the real subject we want for this chapter: reactors and beacons.

We will find out in this chapter how to

- View events on the message bus.

- Automate configuration by scheduling activity on Salt Minions.

- Create reactors on the Salt Master to respond to events.

- Use beacons on Salt Minions to alert the Salt Master and reactors.

The Beating Heart of Salt: The Event Bus

Living at the heart of Salt, we have the event bus, the ZeroMQ message bus connecting Salt Minions to the Salt Master. When using the command salt, we are publishing tasks for the Salt Minions to execute. Having executed these tasks, Salt Minions respond to the Salt Master, and these responses can be viewed from the event bus. Administrative tasks can also be seen here, not just events for the Salt Minions. Deleting authorized Salt Minion keys using the command salt-key will create events and trigger the regeneration of the AES256 encryption key used to communicate with Salt Minions. This is the reason why we need to manage keys using salt-key and not just move files in the file system. The AES key regeneration ensures that previously authenticated Salt Minions whose keys have been deleted have no mechanism to read events on the Salt Master, or at least not until their keys have been reaccepted. To view the event bus, we must use the salt-run command on the Salt Master. Having two consoles open to the Master will aid this as the event bus consumes one terminal until you exit from the viewer.

Within the following demonstration, we delete the master Salt Minion key from the Salt Master. This has already been authorized. When viewing the events, we see the deletion and the key rotation. Each event has a timestamp allowing us to see the exact timings. The AES key is rotated within a minute of the key's deletion. Having auto-accept enabled on the Salt Master, the deleted Minion just needs to present its public key again to continue normal activity. salt-call test.ping executed on the Salt Minion will cause this to happen. We see this with the key being accepted by the Salt Master on the event bus (Shown in Listing 10-1).

console 1: root@master:~# salt-run state.event pretty=True

console 2: root@master:~# salt-key -yd master

The following keys are going to be deleted:

Accepted Keys:

master

Key for minion master deleted.

console 1:

salt/key        {

"_stamp": "2021-06-07T08:33:02.685633",

"act": "delete",

"id": "master",

"result": true

}

key     {

"_stamp": "2021-06-07T08:33:52.937943",

"rotate_aes_key": true

}

console 1:

salt/auth       {

"_stamp": "2021-06-07T08:34:47.648099",

"act": "accept",

"id": "master",

"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxAMzMl+SciBBklFm0oAZ\nws11ySxjGtLPVl/Oxit04ff+I1QCLf7xc4wYLsxqS9AQsI4aMLEDhCnrGyLYgVIF\nRHophcTVHb/dRql8iCt46O+ebJCTBf/HDJx/n4YV0U642hyAhlx8b0e+sn1oMQPL\nXz5PCgF0M8Gzke0cESg3w8/mQ5ElyHh2EXMIb2EdfWMZZd0lsOqZVLts5Paug7Pq\nWDDak03z/gt1a2vqwWJAFKEAq0dyS/r80BcNaEw9bT5Fw7iCrNswvBX6CJuwIsSy\n0Ub4l10o5TVkkFutVFUUt7wK6LamgtpWEcauAT7sNoqmnIYZ6aFG9C8MjNanqt0w\nkwIDAQAB\n-----END PUBLIC KEY-----",

"result": true

}

minion/refresh/master   {

"Minion data cache refresh": "master",

"_stamp": "2021-06-07T08:34:47.754883"

}

Listing 10-1  Viewing Events

To end the viewing session, use the CTRL+C key combination when you have gained focus on the console running the event viewer. That, my friends, is your quick introduction to the event viewer. Later, we will learn how we can react to these events and execute code when events occur. Moving on, we will disable the auto-accept that was previously configured on the Salt Master and quickly look at scheduling (Shown in Listing 10-2).

root@master:~# vim /etc/salt/master.d/local.conf

auto_accept: False

root@master:~# systemctl restart salt-master

Listing 10-2  Disable Auto-Accept

Creating Schedules on Salt Minions

You may have already noticed that there is an additional configuration file created automatically in the /etc/salt/minion.d/ directory, the schedule.conf file. Like all .conf files in this directory, it makes up part of the Salt Minion’s configuration and where scheduled executions are stored. By default, there is a single hidden scheduled task configured.

Note   Hidden task names start with an underscore.

Hidden tasks do not show in a standard listing but can be seen in the file. Including the show_all option will also display these hidden tasks when listing the schedules (Shown in Listing 10-3).

root@master:~# salt 'master' schedule.list

master:

----------

schedule:

----------

root@master:~# salt 'master' schedule.list show_all=True

master:

schedule:

__mine_interval:

enabled: true

function: mine.update

jid_include: true

maxrunning: 2

minutes: 60

name: __mine_interval

return_job: false

run_on_start: true

root@master:~# cat /etc/salt/minion.d/_schedule.conf

schedule:

__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,

minutes: 60, return_job: false, run_on_start: true}

Listing 10-3  Listing Schedules

The default hidden task is used to update the Salt Mine database on the Salt Master. We will create our own scheduled task to ensure that the main top file is applied at least once a day to our Salt Minions, automating the enforcement of the required system state and reducing configuration drift (Shown in Listing 10-4) .

root@master:~# salt '*' schedule.add apply_highstate \ function='state.apply' days=1

root@master:~# salt '*' schedule.show_next_fire_time apply_highstate

master:

----------

next_fire_time:

2021-06-08T10:07:15

result:

True

minion1:

----------

next_fire_time:

2021-06-08T10:07:15

result:

True

root@master:~# salt 'master' schedule.list

master:

schedule:

apply_highstate:

days: 1

enabled: true

function: state.apply

jid_include: true

maxrunning: 1

name: apply_highstate

Listing 10-4  Creating Scheduled Events

Having already created the /srv/salt/top.sls file, we can apply the highstate daily or whenever we decide. We can use seconds, minutes, hours, or, as we implemented, days. We can also build a schedule using when: and a list of days and times, or, when needing to run a task on a single occasion, we have once:. Creating the schedule using the schedule.add function, tasks are persisted in the _schedule.conf, and no Salt Minion restart is needed. Using the schedule.show_next_fire_time function, we display the exact time of the next execution. Finishing, we can list the schedule again. To list all of the functions, do not forget the help commands. Yes, please do read the documentation! The following listing will list all functions that are available in the schedule module:

root@master:~#  salt 'master' sys.list_functions schedule

Creating Reactors

Reactors are configured on the Salt Master, and, as the name suggests, they react to events seen on the event bus. As a simple example, we will create a reactor to auto-accept the presented key from the master Salt Minion but only from that Minion. The reactor will accept the key for us rather than relying on the explicit auto_accept configuration setting.

Investigating Events Needing Reactors

The first learning task will be to view the signing requests on the event bus. This will help us understand the events to look out for. Having ensured that auto_accept is disabled on the Salt Master, we now stop the Salt Minion running on the master node, delete the key, and restart the Salt Minion while viewing the event bus. Start the lab with two consoles to the Salt Master allowing us to view events and issue commands (Shown in Listing 10-5).

console 2: root@master:~# systemctl stop salt-minion

console 2: root@master:~# salt-key -yd master

The following keys are going to be deleted:

Accepted Keys:

master

console 1: root@master:~# salt-run state.event pretty=True

console 2: root@master:~# systemctl start salt-minion

console 1:

salt/auth       {

"_stamp": "2021-06-07T09:34:31.120740",

"act": "pend",

"id": "master",

"pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxAMzMl+SciBBklFm0oAZ\nws11ySxjGtLPVl/Oxit04ff+I1QCLf7xc4wYLsxqS9AQsI4aMLEDhCnrGyLYgVIF\nRHophcTVHb/dRql8iCt46O+ebJCTBf/HDJx/n4YV0U642hyAhlx8b0e+sn1oMQPL\nXz5PCgF0M8Gzke0cESg3w8/mQ5ElyHh2EXMIb2EdfWMZZd0lsOqZVLts5Paug7Pq\nWDDak03z/gt1a2vqwWJAFKEAq0dyS/r80BcNaEw9bT5Fw7iCrNswvBX6CJuwIsSy\n0Ub4l10o5TVkkFutVFUUt7wK6LamgtpWEcauAT7sNoqmnIYZ6aFG9C8MjNanqt0w\nkwIDAQAB\n-----END PUBLIC KEY-----",

"result": true

}

Listing 10-5  View Pending Requests

The key will be presented to the Salt Master every 10 seconds or until it is accepted. Annoying, isn't it! The critical information has been highlighted in the event data for you:

- salt/auth: The event type and subsystem being used.

- "act": "pend": The data of the event includes the action of pending, indicating that we are waiting for the key to be accepted.

- "id": "master": We can check the Salt Minion ID, so we only accept the correct key.

For the moment, we will stop the event viewer using CTRL+C. With the event viewer stopped, we turn our attention to defining the reactor configuration on the Salt Master.

Reactor and State Configuration

Reactors are configured as part of the Salt Master configuration. We will create a new configuration file within /etc/salt/master.d/. Don't forget that these configurations, both for the Salt Master and Salt Minions, are YAML files and require the correct indentation levels (Shown in Listing 10-6).

root@master:~# vim /etc/salt/master.d/reactor.conf

reactor:

- 'salt/auth':

- '/srv/reactor/accept_key.sls'

root@master:~# systemctl restart salt-master

Listing 10-6  Reactor Configuration

Having created the configuration, the heavy lifting will be done by the accept_key.sls . We have specified that this should be located in the /srv/reactor directory, which will need to be created. We will also make sure that the event view is back up and running (Shown in Listing 10-7).

console 1: root@master:~# salt-run state.event pretty=True

console 2: root@master:~# mkdir /srv/reactor

console 2: root@master:~# vim /srv/reactor/accept_key.sls

#!jinja|yaml

{% if 'act' in data and data['act'] == 'pend' and data['id'] == 'master' %}

'add_pending_master_minion':

wheel.key.accept:

- match: {{ data['id'] }}

{% endif %}

console 1:

salt/wheel/20210607100039108548/ret     {

"_stamp": "2021-06-07T10:00:39.122259",

"fun": "wheel.key.accept",

"fun_args": [

"master",

{

"include_denied": false,

"include_rejected": false

}

],

"jid": "20210607100039108548",

"return": {

"minions": [

"master"

]

},

"success": true,

"user": "Reactor"

}

Listing 10-7  Reacting to Events

As soon as the State file is saved, the reactor will accept the pending key request for the Salt Minion named master, as defined in the Jinja code we have created. Take care creating the code, but don't forget you can download the code samples from Apress. We can CTRL+C on the event viewer when we are ready.

Configuring Beacons

Beacons make part of the Salt Minion configuration and can be used to create events on the event bus. If necessary, reactors can respond to these events as we have previously seen. Having created the schedule to apply the highstate daily, we know that we meet the configuration requirement at least daily. This includes ensuring that the Apache web server is running. It is possible that an administrator may stop the service creating an outage for up to 24 hours. Using a beacon, we can monitor the state of the web server. The beacon can trigger a reactor into starting the service, and we return to peace and tranquility.

Creating the Beacon

First, we will define the beacon on the Ubuntu Salt Minion. Having created the beacon, we can also list it. As part of the Salt Minion configuration, the Salt Minion service needs to be restarted (Shown in Listing 10-8).

root@master:~# vim /etc/salt/minion.d/beacons.conf

beacons:

service:

- services:

apache2:

onchangeonly: True

root@master:~# systemctl restart salt-minion

root@master:~# salt 'master' beacons.list

master:

beacons:

service:

- services:

apache2:

onchangeonly: true

Listing 10-8  Defining Beacons

The created beacon uses the salt.beacon.service module to monitor running services for the apache2 service changing state. This could be either starting or stopping of the service.

Viewing Beacon Activity

In itself, the beacon does not do anything but monitor the service and send data to the event bus. Even without further configuration, we can see the beacon if it is triggered by changing the service state. We will need two consoles again, and we will start the event viewer, allowing us to see the beacon when we change the web server state (Shown in Listing 10-9).

console 1: root@master:~# salt-run state.event pretty=True

console 2: root@master:~# systemctl restart apache2

console 1:

salt/beacon/master/service/apache2      {

"_stamp": "2021-06-07T11:16:12.441073",

"apache2": {

"running": false

},

"id": "master",

"service_name": "apache2"

}

salt/beacon/master/service/apache2      {

"_stamp": "2021-06-07T11:16:13.319367",

"apache2": {

"running": true

},

"id": "master",

"service_name": "apache2"

}

Listing 10-9  Viewing Beacon Activity

The restart will both stop and start the service, triggering the beacon on both events. We see this in the event viewer and two instances of the beacon event. The service beacon delivers two dictionaries: the event itself and, nested within this, the apache2 dictionary where we can see the service state. This will affect how the data is accessed from the reactor State file. First, let's stop the viewer using CTRL+C.

Reacting to Beacons

To really orchestrate the Salt environment, we can configure the Salt Master to react to this beacon if the Apache service is not running. In this way, we can attempt to keep the web server running in the event it is stopped (Shown in Listing 10-10).

root@master:~# vim /etc/salt/master.d/reactor.conf

reactor:

- 'salt/auth':

- '/srv/reactor/accept_key.sls'

- 'salt/beacon/*/service/apache2':

- /srv/reactor/start_www.sls

root@master:~# systemctl restart salt-master

root@master:~# vim /srv/reactor/start_www.sls

#!jinja|yaml

{% if data[data['service_name']]['running'] == False %}

'start_apache_web_server':

local.service.start:

- args:

- name: {{ data['service_name'] }}

- tgt: {{ data['id'] }}

{% endif %}

Listing 10-10  Creating Reactors for Beacons

Testing the Beacon

Now that the reactor back end has been created, the beacon will now cause the reactor to trigger when the Apache service is not running. Working with the event viewer again and two consoles, we view the actions (Shown in Listing 10-11).

console 1: root@master:~# salt-run state.event pretty=True

console 2: root@master:~# systemctl stop apache2

console 1:

salt/beacon/master/service/apache2      {

"_stamp": "2021-06-07T11:43:13.530473",

"apache2": {

"running": false

},

"id": "master",

"service_name": "apache2"

}

20210607114313579137    {

"_stamp": "2021-06-07T11:43:13.580810",

"minions": [

"master"

]

}

salt/job/20210607114313579137/new       {

"_stamp": "2021-06-07T11:43:13.581309",

"arg": [

{

"__kwarg__": true,

"name": "apache2"

}

],

"fun": "service.start",

"jid": "20210607114313579137",

"minions": [

"master"

],

"missing": [],

"tgt": "master",

"tgt_type": "glob",

"user": "root"

}

salt/job/20210607114313579137/ret/master        {

"_stamp": "2021-06-07T11:43:13.869361",

"cmd": "_return",

"fun": "service.start",

"fun_args": [

{

"name": "apache2"

}

],

"id": "master",

"jid": "20210607114313579137",

"retcode": 0,

"return": true,

"success": true

}

salt/beacon/master/service/apache2      {

"_stamp": "2021-06-07T11:43:14.321378",

"apache2": {

"running": true

},

"id": "master",

"service_name": "apache2"

}

Listing 10-11  Viewing Beaconing Triggers

Viewing the event bus shows that the Apache service was stopped at 11:43:13 and one second later started again at 11:43:14. We are able to maintain the integrity of the web server starting the service as soon as it is stopped. We have now finished with the event viewer and will end it again using CTRL+C. We should be quite used to this by now.

Debugging Reactors

If all does not go well and the reactors appear not to work, then the event viewer may help, becoming an easy check to make and to familiarize yourself with. Another useful tool is to stop the Salt Master service and start it from the command line manually, rather than as a systemctl service. Starting the service with the -l debug log level option allows you to see the processing of any reactor State file.

Note   Start the Salt Master in debug mode:# systemctl stop salt-master# salt-master -l debug

Summary

Salt is often thought of as mainly a configuration management system, but it goes way beyond that, and we will see more on this later. In this chapter, we have seen the orchestration element of Salt where you learned to schedule tasks, view events, create reactors, and create beacons.

Each Salt Minion can be configured with its own schedules, helping automate the configuration of the system. The schedule makes up part of the Salt Minion configuration, but, if added using the schedule.add function, a restart can be avoided.

A runner module is used to view the events on the message bus. Making use of the salt-run command, these modules only execute on the Salt Master. Viewing events provides you the advanced insights and diagnostics that come with true understanding.

Reactors are defined on the Salt Master and react to events on the event bus. The Salt Master configuration for reactors will direct events through to State files.

Salt Minions are where we define beacons, which are used to send information to the event bus, perhaps for reactors to attend to. Working with beacons and reactors, we could ensure the reliability of the Apache web server.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_11

11. Using Salt SSH

Andrew Mallett[1]

(1)

Peterborough, UK

Where a long-running Salt Minion service is not required or desired, we can use Salt SSH with Salt to connect using SSH to remote devices that have Python installed. You may also want to use this as a method of installing the Salt Minion remotely. We will use Salt SSH to enlist the additional Ubuntu systems as a full Salt Minion. This is just as easy with over 20 devices as it is with the one device that we will use. Standardizing communication with SSH and no specific node requirements as Python is universally available in Linux distributions, Salt SSH is a simple alternative to the full Salt Minion.

Working though this chapter, we will find out how to

- Install Salt SSH.

- Configure the Salt Roster.

- Deploy Configurations Using Salt SSH.

Salt SSH

Salt SSH can be installed on the Salt Master and operates much like the standard salt command using Remote Execution and State modules. In Salt, the Salt Master uses the authorized keys as a list of nodes to manage, whereas using Salt SSH, we do need to create a roster that contains the connectivity details for each device we need to manage. Working on the Salt Master, we can install Salt SSH and edit the sample roster with the connectivity details for the remote Ubuntu system. This is the third system we have not used so far without the Salt Minions installed. We show this is Listing 11-1.

root@master:~# apt install salt-ssh

root@master:~# vim /etc/salt/roster

# Sample salt-ssh config file

#web1:

#  host: 192.168.42.1 # The IP address or DNS hostname

#  user: fred         # Remote executions will be executed as user fred

#  passwd: foobarbaz  # The password to use for login, if omitted, keys are used

#  sudo: True         # Whether to sudo to root, not enabled by default

#web2:

#  host: 192.168.42.2

minion2:

host: 192.168.33.102

user: vagrant

Listing 11-1  Installing Salt SSH on the Salt Master

The default roster file includes comments, which we have left in place appending the new device to the end. The roster file is also written in YAML, so maintenance of the indentation is required.

Vagrant and Password Authentication

By default, the Vagrant system uses key-based authentication only and is configured to have password authentication disabled for the SSH server. Temporarily, we need password authentication enabled and to set a password for the vagrant user, which will not have had a password assigned. It is this Vagrant account that we will use to authenticate as to the remote Ubuntu system. Once we have connected initially with salt-ssh and a password, we will be able to use key-based SSH authentication; until then, it is easiest to set password authentication. Connecting from your host operating system to the minion2 Ubuntu system, we can use Vagrant internal keys. We make the edit to the SSH server configuration using the stream editor sed. This may seem a little complex, but it is easier than finding the correct line in the file, believe me . If you are new to sed, the options used are shown in the following:

- -E: Enhanced regular expression is needed because we use grouping using parentheses.

- -i: In-place edit, or quite simply, we edit the file.

- 's/: The command s represents a substitution, where we replace the first listed string with the second.

- (PasswordAuthentication) no/: This is the string we search for and want to replace. The parentheses create a numbered group allowing us to repeat the group using just the group number in the replacement text. This saves us having to type the key again.

- \1 yes/': This is the replacement text, \1 being the first grouping in the search string. The replacement text expands to PasswordAuthentication yes.

- /etc/ssh/sshd_config: The SSHD configuration file that we need to edit.

We will now use sed to edit the file on the minion2 system (Listing 11-2). We show this is Listing 11-2:

hostOS:~$ vagrant up minion2

hostOS:~$ vagrant ssh minion2

vagrant@minion2:~$ sudo sed -Ei \

's/(PasswordAuthentication) no/\1 yes/' /etc/ssh/sshd_config

vagrant@minion2:~$ sudo systemctl restart sshd

vagrant@minion2:~$ sudo passwd vagrant

New password:

Retype new password:

passwd: password updated successfully

vagrant@minion2:~$ exit

logout

Connection to 127.0.0.1 closed.

Listing 11-2  Enable Password Authentication on minion2

Managing Nodes Using Salt SSH

Having configured the remote Vagrant system , we are ready to deploy the full Salt Minion. Initially though, running a simple test.ping will be wise. At least this way we know Salt SSH is working by testing the roster and exchanging keys. During the first execution, salt-ssh will generate an SSH public and private key pair in /etc/salt/pki/master/ssh. These keys are used to authenticate Salt to the remote systems, but we need to have password authentication initially. In this way, if the keys are not preinstalled, we can copy the SSH key across while using password authentication. Salt delivers the keys automatically; this compares well with Ansible, which would require administrators to deliver the key with ad hoc commands or Playbooks. We show this is Listing 11-3:

Note   Vagrant can run configuration scripts to provision the system the way you need. This can include enabling password authentication for SSH and setting the password for the user (Listing 11-3).

root@master:~# salt-ssh -i '*' test.ping

Permission denied for host minion1, do you want to deploy the salt-ssh key? (password required):

[Y/n] y

Password for vagrant@minion2:

minion2:

True

root@master:~# salt-ssh '*' test.ping

minion2:

True

root@master:~# ls /etc/salt/pki/master/ssh

salt-ssh.rsa  salt-ssh.rsa.pub

Listing 11-3  Testing SSH Connectivity to minion2 from the Salt Master

In the first test.ping execution , we include the option -i for an interactive login should the key authentication fail, and it will the first time around. In the second execution, we can omit the -i option and go straight to the key-based authentication that will now be working, the key having been transparently copied to the remote vagrant user’s authorized_key file allowing key-based authentication. We can now easily manage the remote system or systems without being prompted for more passwords. Having just a single node in the roster file is a good demonstration, but we could easily add more in real life.

Agentless Operation

Having no requirement for the Salt Minion allows for so-called agentless operation . Just needing SSH and Python, execution remains on the remote nodes as in Salt, but the Salt SSH now has to deliver the modules to be executed. The module is still executed remotely but no longer is installed with the agent as there is no agent. Using the exact same module and functions though as we do in Salt makes the transition easy, a system we have become used to. We DO need to consider execution privileges as part of this strategy. Relieving ourselves of the long-running Salt Minion service also means that we are not executing through the service that runs as the root user. Commands issued via Salt SSH are executed in the user context that is configured in the roster, for us the vagrant user. To elevate rights, we can use the option --sudo with salt-ssh, which elevates the remote Vagrant account. The remote sudo configuration must allow password-less access to sudo for the user vagrant. This is standard on Vagrant systems in any case (Listing 11-4). We show this is Listing 11-4:

root@master:~# salt-ssh --sudo '*' pkg.install tree

minion2:

----------

tree:

----------

new:

1.8.0-1

old:

Listing 11-4  Remote Execution Commands Using Salt SSH

As we have only the one system in the roster, it is easy to use the asterisk glob metacharacter, but we can target the individual device also, but advanced targeting such as Grains is not available:

root@master:~# salt-ssh --sudo  'minion2' timezone.set_zone 'Europe/London'

minion2:

True

Creating State Files

Using State files with Salt SSH allows for more complex configuration, just as they do in Salt. Installing the Salt Minion on the remote system requires adding the SaltStack repository, adding a host record allowing resolution of the name “salt” to the IP address of the Salt Master before installing the Salt Minion. The State files need to be in the standard file_roots, /srv/salt, and we can organize the content creating a subdirectory and four new SLS files (Listing 11-5). We show this is Listing 11-5:

root@master:~# mkdir /srv/salt/minion/

root@master:~# vim /srv/salt/minion/repo.sls

#!yaml

'add_saltstack_repo':

pkgrepo.managed:

- name: deb https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest focal main

- humanname: SaltStack

- file: /etc/apt/sources.list.d/saltstack.list

- key_url: https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest/salt-archive-keyring.gpg

root@master:~# vim /srv/salt/minion/salthost.sls

#!yaml

'add_record_for_host_salt':

host.present:

- name: salt

- ip: 192.168.33.100

- clean: True

root@master:~# vim /srv/salt/minion/install_minion.sls

#!yaml

include:

- .repo

'Install Salt Minion':

pkg.installed:

- name: salt-minion

- require:

- pkgrepo: add_saltstack_repo

root@master:~# vim /srv/salt/minion/init.sls

#!yaml

include:

- .repo

- .salthost

- .install_minion

root@master:~# salt-ssh --sudo  'minion2' state.sls minion

root@master:~# salt-key -L

Accepted Keys:

master

minion1

Denied Keys:

Unaccepted Keys:

minion2

Rejected Keys:

root@master:~# salt-key -ya minion2

The following keys are going to be accepted:

Unaccepted Keys:

minion2

Key for minion minion2 accepted.

Listing 11-5  Installing the Salt Minion

Having not forgotten anything that we have learned, we have crafted an amazing solution to enroll the remote system into Salt. Creating the individual State files allows those configurations to be applied separately or as one using the init.sls. Remembering that the Salt Minion will need a repository to be in place, we make sure to include the repo.sls and make that a requirement. We can use the name or the State ID in the requirement. Here the ID was the easiest choice.

Now that the key is accepted on the Salt Master, we can revert to using Salt to manage all systems and apply the highstate. As we have not added the new Salt Minion to the webservers groups, the new Minion will match on the common.sls only from the /srv/salt/top.sls:

root@master:~# salt '*' state.apply

Summary

On a more standard Linux system, rather than Vagrant, we could have deployed the Salt Minion without ever needing to connect to the node directly. Just stop and think about that for the moment. We could have also used scripts when provisioning Vagrant allowing password authentication so we could enjoy the remote Minion installation more, but we did not want to complicate the installation. That said, I do think you can see the power of Salt SSH as an alternative to the full Salt Minion. Using standard protocols and requiring only Python to be installed, we have a great way to manage systems that either do not have the Salt Minion yet or will never have the Salt Minion. They are not out of our management scope, and we can manage them with standard Salt modules and State files. Salt SSH can be installed on the Salt Master and executed similarly to the command salt. Devices are configured in the /etc/salt/roster file acting as the host inventory. This is an important part of Salt and competes well with both Ansible and Puppet Bolt, which use similar technologies.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_12

12. Deploy Virtual Machines Using Salt Cloud

Andrew Mallett[1]

(1)

Peterborough, UK

As well as sending configurations to your current systems, Salt can be used to manage virtual machines without regard to their location. These virtual machines may be cloud based or on-prem. Imagine this, you need to deploy a virtual machine to test website development. You can easily deploy the system, and once deployed the web server and current content can be delivered reliably with Salt, locally or to the cloud.

Salt Cloud is an additional suite of tools that can be deployed with or without an existing Salt Master. The Salt Master deploys as a dependency to Salt Cloud if not already present. Salt Cloud is used to manage virtual machine instances and, optionally, works with Salt delivering Salt Minion software and the address of the Salt Master as part of the VM deployment. In this chapter, we will

- Install Salt Cloud.

- Configure cloud providers.

- Configure cloud profiles.

- Deploy AWS cloud systems.

- List cloud systems.

- Delete cloud systems.

Installing Salt Cloud

For this demonstration, we will work with the Salt Master system adding in the Salt Cloud Python tools. We will use AWS for our cloud system. Free accounts are available for 12 months on AWS as well as other cloud platforms.

Installing the Salt Master using the option -L from the bootstrap installer will also add Salt Cloud and the associated Python cloud libraries. If you did not include the option when installing the Salt Master, then you can install Salt Cloud easily from the command line. It will do no harm to try the install in any case. Once installed, the option --version will show the version of Salt Cloud, whereas the option -V displays the full version report. This is shown in Listing 12-1.

root@master:~# apt install salt-cloud

root@master:~# salt-cloud --version

salt-cloud 3003

root@master:~# salt-cloud -V

Salt Version:

Salt: 3003

Dependency Versions:

Apache Libcloud: 2.8.0

cffi: Not Installed

cherrypy: Not Installed

dateutil: 2.7.3

docker-py: Not Installed

gitdb: 2.0.6

gitpython: 3.0.7

Jinja2: 2.10.1

libgit2: Not Installed

M2Crypto: Not Installed

Mako: Not Installed

msgpack: 1.0.2

msgpack-pure: Not Installed

mysql-python: Not Installed

pycparser: Not Installed

pycrypto: 2.6.1

pycryptodome: 3.10.1

pygit2: Not Installed

Python: 3.8.5 (default, May 27 2021, 13:30:53)

python-gnupg: 0.4.5

PyYAML: 5.3.1

PyZMQ: 22.0.3

smmap: 2.0.5

timelib: Not Installed

Tornado: 4.5.3

ZMQ: 4.3.4

System Versions:

dist: ubuntu 20.04 focal

locale: utf-8

machine: x86_64

release: 5.4.0-74-generic

system: Linux

version: Ubuntu 20.04 focal

root@master:~# file $(which salt-cloud)

/usr/bin/salt-cloud: Python script, ASCII text executable

Listing 12-1  Installing Salt Cloud on the Salt Master

Emphasizing the relationship with Python, Salt Cloud, as with the other subsystems, is written in Python.

Configuring Salt Cloud

Default configuration settings for Salt Cloud are displayed in the file /etc/salt/cloud. Changes to the defaults can be made with any .conf file placed in the /etc/salt/cloud.conf.d/ directory. This is the same idea as implemented with the Salt Minion and Salt Master, as we have already seen. Additionally, the Salt Cloud configuration requires a list of cloud providers and cloud profiles.

Cloud Providers

Cloud providers describe how connections should be managed to your public and private clouds. This could even be Vagrant should you want; however, for Vagrant to work, the host system that you have Vagrant installed on must be a Minion enrolled to the Salt Master that Salt Cloud runs on. I prefer to keep my host OS separate from any of my labs that it runs. For that reason, I will demonstrate Salt Cloud with AWS. In Listing 12-2, you will see a sample AWS provider configuration for AWS.

root@master:~# vim /etc/salt/cloud.providers.d/aws.conf

aws:

driver: ec2

id: <your aws id>

key: <your aws authentication key>

securitygroup: AllOpen

keyname: awslondon

private_key: /root/.ssh/awslondon.pem

location: eu-west-2

Listing 12-2  AWS Cloud Provider

As is the normal, configuration files of this type are written in YAML. The dictionary aws configures the name of the provider. This can be any name, but aws does seem a useful name. The driver though has to be set to ec2 to use AWS as a cloud provider.

The id and key are obtained from your account details in AWS. Log in to you EC2 Console. From your account details from the top right of the AWS page, choose “My Security Credentials.” From the Security Credentials page, select the tab “Access keys (access key ID and secret access key).” From here you can create a new id and key that you use as authentication credentials. Having created a provider – and of course, we can have multiple providers within the same file or even different files – we can list configured providers. This is shown in the following command with output:

root@master:~# salt-cloud --list-providers

aws:

----------

ec2:

----------

This shows the provider that we have named aws using the Salt Cloud driver ec2. The driver ec2 that we have selected in a Python module exists as a file called ec2.py. In the following output from the Linux find command, we see the Salt Cloud module that we reference but that in turn will use the libcloud driver ec2.py:

root@master:~# find /usr/lib -name 'ec2.py'

/usr/lib/python3/dist-packages/salt/cloud/clouds/ec2.py

/usr/lib/python3/dist-packages/libcloud/compute/drivers/ec2.py

Note   We can configure the provider even if you do not have an AWS account. You can still create and list the provider using the dummy details provided here.

Cloud Profiles

Salt Cloud profiles are configured within the /etc/salt/cloud.profiles.d/ directory. Creating files with the .conf extension in this directory will allow us to define profiles or templates used to provision virtual machine instances. The format will differ slightly depending on the cloud provider used. The size attribute is defined by available sizes from the provider, and the image attribute depends on the available images. Salt Cloud can deploy the Salt Minion software and configure the Salt Minion settings. The profile example in Listing 12-3 does not deploy the Salt Minion.

root@master:~# vim /etc/salt/cloud.profiles.d/profiles.conf

ubuntu:

#Ubuntu 20.04

provider: aws

image: ami-096cb92bb3580c759

size: t2.micro

ssh_username: ubuntu

deploy: False

del_all_vol_on_destroy: True

del_root_vol_on_destroy: True

Listing 12-3  Example Profile Without Agent

This again is in YAML format , and we define the profile, which we have called ubuntu. The provider attribute is used to link to our named provider, aws. The deploy attribute defaults to True; we have explicitly set it to False to prevent deployment of the Salt Minion for this profile. We will create another profile that will deploy the same Ubuntu system and the Salt Minion. We append to the same file, adding the additional profile as shown in Listing 12-4.

root@master:~# vim /etc/salt/cloud.profiles.d/profiles.conf

ubuntu:

#Ubuntu 20.04

provider: aws

image: ami-096cb92bb3580c759

size: t2.micro

ssh_username: ubuntu

deploy: False

del_all_vol_on_destroy: True

del_root_vol_on_destroy: True

ubuntu-salt:

#Ubuntu20.04

provider: aws

image: ami-096cb92bb3580c759

size: t2.micro

ssh_username: ubuntu

ssh_interface: public_ips

del_all_vol_on_destroy: True

del_root_vol_on_destroy: True

minion:

master: 18.130.247.42

id: minion3

Listing 12-4  Profile Deploying Salt Minion

Each profile needs a unique name. Here we set it to ubuntu-salt. We have added the attribute ssh_interface. This can be set to public_ips or private_ips and defines how the Salt Cloud system should connect when deploying the bootstrap installer. Using another AWS system for Salt Cloud, you can use private_ips; else, you will need public_ips to be able to connect from outside of AWS. We also add the Minion configuration within the deployment, setting the address of the Salt Master and the Minion ID to use. If we are deploying more than one system, I would not set the ID, allowing the Salt Minion to use the FQDN of the system.

To retrieve a list of profiles, we are able to use the --list-profiles option with the argument of all or the named provider to search. Listing 12-5 provides the configured AWS profiles .

root@master:~# salt-cloud --list-profiles aws

ubuntu:

----------

aws:

----------

ubuntu-salt:

----------

aws:

----------

Listing 12-5  Listing Profiles from Salt Cloud

Managing Virtual Machine Instances Using Salt Cloud

Salt Cloud is much more than just this configuration. It allows for the creation and ongoing management of your virtual machine instances. These instances may be in-house or in the public cloud. Either way, they can be managed with the single command line tool salt-cloud.

Provisioning Virtual Machines

Our first port of call is to create or provision the virtual machine instances. To create a simple single virtual machine is really easy. The following demonstrates this but requires a working AWS account and configured profile:

root@master:~# salt-cloud -p ubuntu vm101

The option -p is the profile we want to use, and the argument vm101 is the name of the virtual machine instance that we are creating. The output will be quite verbose, and it takes a little time for the public IP address to be detected. Once complete, we are able to see the complete instance details. We can also create multiple instances at the same time, as shown in the following listing:

root@salt-cloud:~# salt-cloud -p ubuntu vm102 vm103

Later, we can review these settings, or the most important setting using the option -Q (Listing 12-6).

root@master:~# salt-cloud -Q

aws:

----------

ec2:

----------

vm101:

----------

id:

i-03be1da36a64d23a0

image:

ami-096cb92bb3580c759

name:

vm101

private_ips:

172.31.35.36

public_ips:

18.133.76.113

size:

t2.micro

state:

running

vm102:

----------

id:

i-0d544b63e2042c74d

image:

ami-096cb92bb3580c759

name:

vm102

private_ips:

172.31.46.110

public_ips:

18.132.12.184

size:

t2.micro

state:

running

vm103:

----------

id:

i-050f48582978f46d7

image:

ami-096cb92bb3580c759

name:

vm103

private_ips:

172.31.42.236

public_ips:

3.8.166.121

size:

t2.micro

state:

running

Listing 12-6  Query VM Instances

We can now see both the internal and public IP addresses of the three virtual machines. Outside of AWS we can connect using the public address and from inside we can use either address .

Deleting Instances

When we are finished with the instances, we can delete them as easily as they were created. The option -d is used to delete the instances. Adding the additional option -y enters yes to all prompts. The output in Listing 12-7 has been included allowing you to see that the instance name changes during the deletion to include DEL in the name.

root@salt-cloud:~# salt-cloud -yd vm101 vm102 vm103

aws:

----------

ec2:

----------

vm101:

----------

currentState:

----------

code:

32

name:

shutting-down

instanceId:

i-03be1da36a64d23a0

newname:

vm101-DEL0338e0f5bc0c43c48b7fbc3668b5b542

previousState:

----------

code:

16

name:

running

vm102:

----------

currentState:

----------

code:

32

name:

shutting-down

instanceId:

i-0d544b63e2042c74d

newname:

vm102-DEL1608407b409347d0863244db64f3e43a

previousState:

----------

code:

16

name:

running

vm103:

----------

currentState:

----------

code:

32

name:

shutting-down

instanceId:

i-050f48582978f46d7

newname:

vm103-DELd6ce01eaabd1431b8a07d4e600029a88

previousState:

----------

code:

16

name:

running

Listing 12-7  Using Salt on New Minion

Summary

Being able to manage virtual machines with a simple interface without regard to the hosting system used is both desirable and deliverable when using Salt Cloud. In addition to this simplicity, installing the Salt Minion can be included as part of the instance creation, making the new systems manageable by Salt very quickly after the installation and adding to the reasons to use Salt as your main configuration management system.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_13

13. Scaling Configuration Management Using Salt Syndic

Andrew Mallett[1]

(1)

Peterborough, UK

Your Salt Minions may be in a single office location with well-connected high-speed networks; others will find that they are employing WANs interconnecting branch offices with the main location. It is possible to decentralize configuration with Salt adding Salt Masters to each remote location. Is this a good idea? Probably not, unless you can link those Salt Masters back to the central Salt Master! Installing Salt Syndic will allow this, making the centralized Salt Master as a Master of Masters or MoM .

Salt Syndic is installed onto a Salt Master within the remote location. Installing Salt Syndic allows our Salt Masters to cooperate. They are no longer independent of each other, and the Salt Master in the centralized office still provides complete configuration control over all Salt Minions. The Salt Master of the Salt Syndic system can be used to manage its own local Salt Minions. In this chapter, we will cover the following topics:

- Rebuild Vagrant labs.

- Install the Salt Master (Master of Masters).

- Install the Salt Master and Salt Syndic (remote Master).

- Install the Salt Minion or third system to use Salt Syndic as the local Master.

- Sign keys on Salt Masters.

- Test Salt from the central Master of Masters.

Danger: Redevelopment in Progress

As you know, we have been using Vagrant as our hypervisor management system. The Vagrantfile describes the configuration, and like any great management system, Vagrant is absolutely brilliant. If we want to start over again, we can easily destroy the current virtual machines and recreate the same clean and fresh systems again, in exactly the same way as before using the exact same IP addresses and the same operating systems. Although we are not implementing a WAN, we can still implement Salt Syndic on minion2 and target Salt Syndic on minion2 as the Salt Master for minion3. We can control all Minions from the Master of Masters installed on the main Salt Master system, which we can refer to as the MoM.

Even in a single location, Salt Syndic helps by reducing the number of Salt Minions that connect to a single Salt Master. Implementing Salt in the simple topology allows for a single Salt Master, and all Salt Minions connect to the Salt Master. This is illustrated in Figure 13-1.

Figure 13-1  Simple Salt Topology

Configuring our new lab, we will include a Salt Syndic system, allowing Salt Minions to connect to a local Salt Master or load-balance connections to the Salt Master. In the next image, we illustrate a simple scenario using three Salt Syndic systems, each having three Salt Minions connected. While we in no way suggest that we need Salt Syndic to support just nine Salt Minions, Figure 13-2 shows how we can scale to many thousands of Salt Minions and build WANs into the topology.

Figure 13-2  Implementing Salt Syndic

Rebuilding the Lab Systems

To rebuild the Vagrant lab, we can destroy the current system and bring the system back up. The destroy sub-command deletes the virtual machines rather than just halting them as we have used before.

Important   Back up files you want to keep to your host system before destroying the lab.

Working on the host system, we navigate to the directory where we create the Vagrantfile. From here we can target our managed systems. We show the commands you will need in Listing 13-1.

hostOS: ~ % cd vagrant/salt-book

hostOS: salt-book % vagrant destroy

minion2: Are you sure you want to destroy the 'minion2' VM? [y/N] y

==> minion2: Destroying VM and associated drives...

minion1: Are you sure you want to destroy the 'minion1' VM? [y/N] y

==> minion1: Destroying VM and associated drives...

master: Are you sure you want to destroy the 'master' VM? [y/N] y

==> master: Destroying VM and associated drives...

hostOS: salt-book % vagrant up

Bringing machine 'master' up with 'virtualbox' provider...

Bringing machine 'minion1' up with 'virtualbox' provider...

Bringing machine 'minion2' up with 'virtualbox' provider...

Listing 13-1  Example Profile Without Agent

The virtual machines will be freshly cloned from the associated Vagrant boxes or golden images we initially downloaded when first creating the lab. The systems are now back up and running but lack the Salt installation.

Installing the Salt Master

The Salt Master, or what will become the Master of Masters (MoM) , will still need to be installed on the virtual machine we have named master. We will connect to the system with Vagrant and run though the fresh install and demonstrate this in the next listing (we can remain as the Vagrant account, elevating privileges as required) (Listing 13-2).

hostOS: salt-book % vagrant ssh master

vagrant@master:~$ curl -L https://bootstrap.saltproject.io \

-o install.sh

vagrant@master:~$ sudo sh install.sh -M -i master -A 127.0.0.1

vagrant@master:~$ sudo salt-key -ya master

The following keys are going to be accepted:

Unaccepted Keys:

master

Key for minion master accepted.

vagrant@master:~$ sudo salt '*' test.ping

master:

True

Listing 13-2  Installing the Salt Master (Master of Masters)

We can also prove the point that Salt Minions can connect directly to the Salt Master, even after we implement one or more Salt Syndic systems. Salt Minions can connect to either the Salt Master or a Salt Syndic.

Installing Salt Syndic

We will connect to the virtual machine that we have named minion1 to install the Salt Master, Salt Syndic, and Salt Minion packages. The configuration of Salt Syndic will occur on both the Salt Master (MoM) and the Salt Syndic system. The key from the Salt Syndic Minion needs to be accepted on both the Salt Syndic system and the Salt Master (MoM).

The Salt Minion connects to the local Salt Master and needs to authenticate with the key. The Salt Syndic service connects to the MoM and needs the key authorized on the MoM.

Working on the Salt Master (MoM) console, we can manage all Salt Minions passing requests to the Salt Syndic system when required. Working on the console of the Salt Syndic system, we can manage all Salt Minions connected to the Syndic system.

Using the bootstrap installer and the option -S, we install the Salt Syndic package. This is all shown for you within Listing 13-3.

hostOS: salt-book % vagrant ssh minion1

Minion1

[vagrant@minion1 ~]$ curl -L https://bootstrap.saltproject.io \

-o install.sh

[vagrant@minion1 ~]$ sudo sh install.sh -M -S -i salt_syndic \

-A 127.0.0.1

[vagrant@minion1 ~]$ sudo systemctl stop salt-*

[vagrant@minion1 ~]$ sudo vi /etc/salt/master.d/syndic.conf

syndic_master: 192.168.33.100

Listing 13-3  Installing Salt Syndic

Note   We use vi rather than vim as it is not currently installed on CentOS Stream.

Master

vagrant@master:~$ sudo vim /etc/salt/master.d/syndic.conf

order_masters: True

vagrant@master:~$ sudo systemctl restart salt-master

Minion1

[vagrant@minion1 ~]$ sudo systemctl start salt-master salt-syndic \salt-minion #Ensure the correct order in starting services

[vagrant@minion1 ~]$ sudo salt-key -ya salt_syndic

The following keys are going to be accepted:

Unaccepted Keys:

salt_syndic

Key for minion salt_syndic accepted.

Master

vagrant@master:~$ sudo -i #you'll need full shell to accept key

root@master:~# salt-key -ya salt_syndic

The following keys are going to be accepted:

Unaccepted Keys:

salt_syndic

Key for minion salt_syndic accepted.

root@master:~# salt '*' test.ping

master:

True

salt_syndic:

True

Installing the Additional Salt Minion

On the Vagrant minion2 system, we will install just the Salt Minion package. The Salt Master for this Salt Minion will be the new Salt Syndic system that we have just created. Using the bootstrap installer, the option -A specifies the address of the Salt Master to connect to. We will use the IP address of the Vagrant minion1 system that is the newly created Salt Syndic. Listing 13-4 demonstrates this.

hostOS: salt-book % vagrant ssh minion2

Minion2

vagrant@minion2:~$ curl -L https://bootstrap.saltproject.io \

-o install.sh

vagrant@minion2:~$ sudo sh install.sh -i minion2 -A 192.168.33.101

Listing 13-4  Installing Salt Minion

Note   We use the IP address of the minion1 system when installing minion2.

Minion1

[vagrant@minion1 ~]$ sudo salt-key -ya minion2

The following keys are going to be accepted:

Unaccepted Keys:

minion2

Key for minion minion2 accepted.

Note   The key only needs to be accepted on minion1 and not the MoM. There is no direct communication from minion2 to MoM.

Master

root@master:~# salt '*' test.ping

master:

True

salt_syndic:

True

minion2:

True

Using the salt command on the Salt Master (MoM), we are able to configure and manage all systems. Using the salt command on the Salt Syndic system, minion1, we can only manage minion1 and minion2, those Salt Minions that report directly to the Salt Syndic system running on minion1. As a quick example, let’s run the test.ping to all Minions from the Salt Syndic CLI:

[vagrant@minion1 ~]$ sudo salt '*' test.ping

minion2:

True

salt_syndic:

True

Verify the Salt Syndic Infrastructure

Having already tried using the vim text editor on the CentOS system, we know we have vi and not vim. We may well want to ensure that vim is available on all systems, both Debian-based systems such as Ubuntu and Red Hat–based systems such as CentOS. We also know that using Salt Formulas, we can easily install vim across all systems without writing our own templates.

Download the Formula and Configure File Roots

All Salt Masters, the MoM and Salt Syndic, will need the Salt Formula or, in fact, any State file that you want to deploy across your estate. If we have many Salt Syndic systems, we would develop a Salt State file to ensure we have the required States and formulas. As we have just the Master and one Salt Syndic, we will do this on each system, starting first with the Salt Master (MoM) shown in Listing 13-5.

root@master:~# mkdir /srv/{formulas,salt}

root@master:~# cd /srv/formulas

root@master:/srv/formulas# git clone \

https://github.com/saltstack-formulas/vim-formula.git

root@master:/srv/formulas# vim /etc/salt/master.d/file_roots.conf

file_roots:

base:

- /srv/salt

- /srv/formulas/vim-formula

root@master:/srv/formulas# systemctl restart salt-master

Listing 13-5  Configuring the MoM with the Formula

With this complete on the MoM, we can move our attention to the Salt Syndic system. We will work as the root user as we need to run a few elevated tasks. We will also need to install the package git as it is not installed by default in the CentOS Stream Vagrant image. Listing 13-6 steps you through the process.

[root@minion1 ~]# mkdir -p /srv/{salt,formulas}

[root@minion1 ~]# yum install -y git

[root@minion1 formulas]# cd /srv/formulas

[root@minion1 formulas]# git clone \

https://github.com/saltstack-formulas/vim-formula.git

[root@minion1 formulas]# vim /etc/salt/master.d/file_roots.conf

file_roots:

base:

- /srv/salt

- /srv/formulas/vim-formula

[root@minion1 formulas]# systemctl restart salt-master

[root@minion1 formulas]# systemctl restart salt-syndic

Listing 13-6  Configuring the Salt Syndic with the Formula

Deploy the Vim Formula

Having added the formula to each Salt Master system, we are ready to go. We would also need to add State files and Pillar data to each Master; but with the configuration in place, it is easy to push these out with Salt. We only need to restart the services where we change the configuration of the Salt Master. From the MoM system, we can now deploy Vim to all systems using the following command:

root@master:/srv/formulas# salt '*' state.sls vim

Summary

As your estate grows and, especially, when WANs are traversed, implementing Salt Syndic can aid the performance of the Salt system. Salt Syndic is part of the free Salt Open suite and installed along with the Salt Master but communicates to the central Salt Master or Master of Masters. When restarting the Salt Master on a Salt Syndic system, Salt Syndic should be restarted also, and the Salt Master should be running before the Salt Syndic service.

State files and other resources such as formulas need to be available in the correct location on each Salt Master. Clever use of Salt and targeting with Nodegroups becomes a simple solution to mitigate this.

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

A. MallettSalt Open

https://doi.org/10.1007/978-1-4842-7237-4_14

14. Automating Network Infrastructure with Salt Proxy

Andrew Mallett[1]

(1)

Peterborough, UK

Having concentrated our Salt management toward Linux so far, we can turn our attention to managing our IT operations infrastructure and network hardware. Salt is very proficient at managing your entire IT suite, which includes routers, switches, access points, and load balancers. Should you really want to reach out, there are Salt Proxy modules for Philips Hue lighting systems! However, I have to admit I have not tried them.

In this chapter, we learn to implement Salt Proxy to act as an intermediary between the Salt Master and network devices. This is needed when the device OS does not allow the installation of the Salt Minion. If your device runs Linux, then you can use the Salt Minion directly. The specific steps we investigate include the following:

- Deploy a cloud-based AWS virtual Arista router.

- Install the NAPALM Proxy module on the Salt Minion.

- Manage the Arista router using NAPALM CLI tools and scripts.

- Configure Salt Proxy and Pillar data for network management.

- Use Salt to manage the Arista router.

- Deploy salt-proxy to the Salt Master to manage the Arista router.

AWS Network Virtualization

The demonstrations we show will use a virtual Arista router hosted in AWS running EOS, the Arista Extensible Operating System. We can continue to use the existing Salt system carried forward from the previous chapter, but you will need a virtual EOS or physical EOS device to follow through the demonstration yourself. My recommendation is to use the AWS system, but only run for the time you actually need. This cloud-based system is not available on the free tier even if you are still within your first year of AWS membership.

Salt Cloud Profile

You do not need to use Salt Cloud to deploy the EOS router, however; if Salt Cloud is up and running, you can easily add a profile for EOS. You will need to search and subscribe to the Arista EOS image. You can retrieve the image ID from the image details. It is recommended to be used with the c5n.xlarge-sized machine template. This is where most of the running costs will come from. The user account for this image is ec2-user, and you will need to authenticate from your id and key within your cloud provider configuration for AWS. The provider configuration will need to ensure that both SSH and HTTPS ports are open via the security group configuration. This configures the AWS firewall settings. Listing 14-1 shows the profile settings that I have used.

eos:

provider: aws

image: ami-0d549eba7f4bddd04

size: c5n.xlarge

ssh_username: ec2-user

deploy: False

del_all_vol_on_destroy: True

del_root_vol_on_destroy: True

Listing 14-1  SSH Cloud Profile for Arista EOS

Where you prefer not to use Salt Cloud, then you can deploy the router using the EC2 Management Console web interface. Ensure that a security group is deployed that allows HTTPS and SSH inbound traffic. We will use HTTPS from the Salt Proxy system to manage the Arista router AWS network virtualization:Salt Cloud Profile.

Configuring the Arista Router

Once the Arista router has booted, you will need to record the public IP address of the system and note the SSH key used when deploying this system. The key used in my AWS profile is available on my host OS, and I use the SSH client from my host OS to configure the router. Listing 14-2 demonstrates the connection to the remote EOS system.

hostOS: ssh -i .ssh/awslondon.pem ec2-user@18.169.132.180

The authenticity of host '18.169.132.180 (18.169.132.180)' can't be established.

ECDSA key fingerprint is SHA256:sOHLVONXybWlGtb8lH7xqd/dpkuYIoiOv+OjBmqcl6I.

Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Warning: Permanently added '18.169.132.180' (ECDSA) to the list of known hosts.

Last login: Fri Jun 18 08:28:54 2021 from 81.97.43.189

Please register for a CVaaS account at https://www.arista.io/cv!

localhost>

Listing 14-2  Connecting to the EOS Router

Note   Connect to the public IP address of your EOS router. The IP used here is for demonstration purposes only.

Having successfully connected to the system, we need to configure the router. Only the basics, but we will need to create an administrative user account and enable the HTTPS Management API that will be used by Salt Proxy to manage this router. Arista and Aruba use the HTTPS API. Other systems may use protocols such as SSH such as IOS from Cisco. In Listing 14-3, we demonstrate the initial configuration, creating the admin user with a password of “[Password1]” and enabling the HTTPS API.

localhost>enable

localhost#configure terminal

localhost(config)#username admin secret [Password1]

localhost(config)#management api http-commands

localhost(config-mgmt-api-http-cmds)#no shutdown

localhost(config-mgmt-api-http-cmds)#exit

localhost(config)#copy running-config startup-config

Copy completed successfully.

localhost(config)#exit

localhost#exit

Listing 14-3  Configuring the Arista Router

We can close this connection now as we have finished the initial configuration that we need to make on the router. Keep the system running and make a note of the IP address of your router.

Salt Proxy Topology

Salt Proxy is installed alongside the Salt Minion and makes up part of the Salt Minion package. This means that we do not need to run through the Salt Proxy installation. We have the Salt Minion installed on all three of our devices. For simplicity, we will use Salt Proxy as part of the Minion on our master VM device. To understand the Salt Proxy topology, there is absolutely no communication from the managed network device to the Salt Master service. Salt Proxy represents the network device and acts as an intermediary between the Salt Master and the managed device. To help understand the topology map used, Figure 14-1 describes the communications path.

Figure 14-1  Salt Proxy Topology

NAPALM: Network Agnostics

The Network Automation and Programmability Abstraction Layer with Multi-vendor Support, or NAPALM for short, allows for simple communication to supported network operating systems from Salt without regard to the underlying OS – much in the same way as Salt does with differing Linux distributions. Within the core NAPALM libraries, there is support for common network devices:

- Arista IOS

- Juniper JunOS

- Cisco IOS

- Cisco IOS-XR

- Cisco NXOS

Community-authored NAPALM drivers include

- Aruba

- F5

For a full list, the following URL lists all supported community drivers: https://github.com/napalm-automation-community.

Installing NAPALM

Communication to the managed network device is from the Salt Minion running Salt Proxy. It is on this system that we will need to install the NAPALM libraries. These are Python libraries and installed using the Python installer pip3. We will need to make sure that this is installed first, before we continue with the install of NAPALM. The steps are demonstrated in Listing 14-4, working as root on the master VM.

root@master:~# apt install -y python3-pip

root@master:~# pip3 install napalm

Listing 14-4  Installing NAPALM on the Salt Minion

Note   Although this system is also the Salt Master, the NAPALM drivers for Salt Proxy need to be installed on the Salt Minion where Salt Proxy will execute. An error may show relating to the JunOS module, but this will not affect us where we are using an Arista and not a Juniper device.

Using the NAPALM CLI

NAPALM provides the abstraction layer we need to communicate with multi-vendor network devices. NAPALM works with Salt, Ansible, and other systems, as well as being able to be used directly from the CLI. As a simple example of this, we can show how the command allows us to read information from the remote device. This is immediately available to you once NAPALM is installed and without further configuration. Listing 14-5 demonstrates this, making sure that you use the IP address of your EOS system.

root@master:~# napalm --user 'admin' --password '[Password1]' \

--vendor 'eos' 18.169.132.180 call get_facts

{

"hostname": "localhost",

"fqdn": "localhost",

"vendor": "Arista",

"model": "vEOS",

"serial_number": "D7614C173C0CCDCE68BB1ADD08F18402",

"os_version": "4.24.3.1FX-CloudEOS-cloud-20225864.42431FXCloudEOS",

"uptime": 12239,

"interface_list": [

"Ethernet1",

"Ethernet100"

]

}

root@master:~# napalm --user 'admin' --password '[Password1]' \

--vendor 'eos' 18.169.132.180 call get_users

{

"admin": {

"role": "network-operator",

"level": 1,

"password": "",

"sshkeys": []

},

"ec2-user": {

"role": "network-operator",

"level": 1,

"password": "",

"sshkeys": [

"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy5+H8HsNr4VsIokJfXHVN7vQQrt/HgS8vlVzu/JdYxYrhaQPi3SwCm6+i+Mcck539Ceg1hQY51gxG57rdBdAOpa9oCsl/KrRM8nDSoGQBk3pbrJCksqdr4OHw6ZIJcy7Bny2Me65RlNCXmYH+xr/k8palNzqUSzHqdp0oZkdULMTJzcOccqD5p2EAK2UxjWiDFrUlfFDPHz9yV1xIf9kde+qIWidKSybG3QtUi6KtGiXfiaQ/UxK/z//h1uOlehE+Y66tHV0AhUoeS29AYbu8Pw/4IaaYlRScULsS/ohZyDhzdrJr5Ia4nPbYlkyfoz2yifMmsDKUYxKQWhVXp3YV awslondon"

]

}

}

Listing 14-5  Using the NAPALM CLI Tool

Note   We configured the Arista router with an administrative account named admin who can authenticate with the password of [Password1]. This password starts and ends with the square brackets and includes the square brackets.

NAPALM Python Scripts

Similarly to the NAPALM CLI tool, we can use NAPALM as a module within our own Python scripts. This still will not be using Salt illustrating what could be done if you wanted to use Python and native NAPALM (Listing 14-6).

root@master:~# vim arista.py

#!/usr/bin/env python3

import napalm

driver = napalm.get_network_driver('eos')

with driver ('18.169.132.180', 'admin', '[Password1]') as device:

print(device.get_facts())

root@master:~# chmod +x arista.py

root@master:~# ./arista.py

{'hostname': 'localhost', 'fqdn': 'localhost', 'vendor': 'Arista', 'model': 'vEOS', 'serial_number': 'D7614C173C0CCDCE68BB1ADD08F18402', 'os_version': '4.24.3.1FX-CloudEOS-cloud-20225864.42431FXCloudEOS', 'uptime': 13249, 'interface_list': ['Ethernet1', 'Ethernet100']}

Listing 14-6  NAPALM Python Scripts

Having shown that NAPALM is working and we are able to communicate with the remote device, we will now start configuring Salt Proxy.

Configuring Salt Proxy

We will now move to the main configuration of the Salt system allowing Salt to manage the Arista device, both reading and writing to the configuration. As we have already mentioned, we do not need to install Salt Proxy as it is part of the Salt Minion package. We will be using the Salt Minion that is installed onto the master VM. This should be the same system where we just installed NAPALM.

Salt Proxy Configuration

The first step in the configuration will be to set the defaults for each Salt Proxy to be used on the Salt Minion. A single Salt Minion can run multiple Salt Proxy services. We will add this setting to the file /etc/salt/proxy.d/local.conf as is shown in Listing 14-7.

root@master:~# vim /etc/salt/proxy.d/local.conf

master: 127.0.0.1

multiprocessing: True

Listing 14-7  Salt Proxy Default Setting

When using SSH to manage remote devices with Salt, multiprocessing should be turned off. For other systems such as Arista, Aruba, and Cisco NXOS, SSH is not used, and multiprocessing for Salt Proxy can be enabled. The address of the Salt Master is needed by the Salt Proxy service. For us the Salt Master also runs on the same system.

Configuring Salt Pillar

Details of the devices to connect to and the credentials to use will be stored as Pillar data on the Salt Master and targeted to the unique Salt Proxy ID used by each Salt Proxy service. The credentials stored in Salt Pillar will include the username and password. Initially we will store this as clear text. Later we will use GPG to encrypt this and offer better security. In Listing 14-8, we see creation of the initial Pillar data.

root@master:~# mkdir -p /srv/pillar

root@master:~# vim /srv/pillar/top.sls

base:

'arista':   #ProxyID

- arista  #State containing configuration

root@master:~# vim /srv/pillar/arista.sls

proxy:

proxytype: 'napalm'

driver: 'eos'

host: '18.169.132.180'

username: 'admin'

password: '[Password1]'

Listing 14-8  Configuring Pillar Data for Salt Proxy

The Pillar top file being quite simple defines the Proxy ID that we have called arista and associates the arista.sls file with this device and nothing else. Each managed device will need its own configuration and unique Proxy ID. The arista.sls file defines the parameters needed for Salt Proxy to connect to the device:

- proxytype : The proxy module to use to initiate the connection.

- driver : For NAPALM, we need the name of the Python module matching the network OS used on our network device. For Arista, EOS is part of the core NAPALM installation.

- host : The IP address or resolvable hostname of the device.

- username: The user account to use.

- password: The password for the user account. This will be encrypted later but will work as clear text. Obviously, for a production environment, encryption would be used.

Using GPG to Encrypt Sensitive Data

Storing passwords in clear-text files is certainly not going to be the best way forward. Although the configuration will work and demonstrates the construction of the Pillar data, we will need to encrypt the password. Salt has access to the GPG (GNU Privacy Guard) suite and can use the GPG renderer to decrypt secured data. The directory where the Salt Master will check for GPG keys is /etc/salt/gpgkeys. We will need to create this directory and add a key pair for Salt to use. The directory needs to be secured to the user only. We use the option -m 700 to set the mode when creating the directory. In Listing 14-9, we demonstrate securely creating the directory and key pair.

root@master:~# mkdir -m 700 -p /etc/salt/gpgkeys

root@master:~# gpg --homedir /etc/salt/gpgkeys --generate-key

gpg (GnuPG) 2.2.19; Copyright (C) 2019 Free Software Foundation, Inc.

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

gpg: keybox '/etc/salt/gpgkeys/pubring.kbx' created

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: saltmaster

Email address: salt@example.com

You selected this USER-ID:

"saltmaster <salt@example.com>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit?o

Listing 14-9  Creating GPG Key Pair for Salt

Entering the letter o in uppercase or lowercase will allow the key generation to proceed; however, the private key should be protected with a passphrase. This is not practical when used by a service such as the Salt Master, so we enter a blank passphrase when prompted. We are prompted twice for this, and we additionally need to confirm that a passphrase is not required on each occasion.

Note   Use a full screen for your console window as the dialogue box will not display if the console is not large enough and you can become stuck in a loop being prompted for a passphrase endlessly.

We are now ready to encrypt the password that we want to use in the Pillar data. The command is shown in Listing 14-10. The encrypted output follows in Figure 14-2.

root@master:~# echo -n '[Password1]' | gpg \

--homedir /etc/salt/gpgkeys  --batch --armor --trust-model always \

--encrypt -r "saltmaster"

Listing 14-10  Encrypting the Password

Note   Copy the output including the BEGIN and END lines to your clipboard.

The output of the gpg encryption should be copied to your clipboard. For clarity, the output is shown in Figure 14-2.

Figure 14-2  Cipher Text

Adding the Cipher Text to Pillar

The encrypted password is within the Cipher Text. This should be within our clipboard ready for pasting into the Pillar data. We can re-edit the /srv/pillar/arista.sls file adding in the encrypted password.

The SLS file must start with the correct shebang for YAML and GPG rendering:

#!yaml|gpg

To indicate the multiline string password, the password key starts with password: |

The vertical bar indicates YAML multiline text follows on the next line. This must not be quoted, and each line must be indented into the password key.

root@master:~# vim /srv/pillar/arista.sls

Note   For clarity, the edited file is shown in Figure 14-3 to maintain formatting.

Figure 14-3  The arista.sls with Encrypted Password

If you have taken a little care, then your Salt Proxy will be configured now and ready to go. Let’s move on and look at the next steps.

Running Salt Proxy

The Salt Proxy service can be run by systemd in the background much in the same way as any other service; however, we may prefer to check the operation in the foreground initially, allowing us to check for errors.

Starting Salt Proxy Initially

On the first run of Salt Proxy, we will need to accept and sign the key for the proxy. The service is started and identifies the Proxy ID that we want to use; it is this ID that is used in the signing process. In Listing 14-11, we show the initial start. Once we see the “wait for the key acceptance,” we can use CTRL+C to stop Salt Proxy, allowing us to accept the key.

root@master:~# salt-proxy --proxyid=arista

[ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

^C[WARNING ] ProxyMinion received a SIGINT. Exiting.

The Salt ProxyMinion is shutdown. ProxyMinion received a SIGINT. Exited.

root@master:~# salt-key -ya arista

The following keys are going to be accepted:

Unaccepted Keys:

arista

Key for minion arista accepted.

Listing 14-11  Starting Salt Proxy and Key Acceptance

Restart Salt Proxy in the Foreground

Now that we have accepted the key for the Proxy ID arista, the startup can continue. It is not until the key has been accepted that the Pillar data can be read. The successful startup of Salt Proxy requires that we can locate the proxy key in Salt Pillar for the Proxy ID used by the service. This means a correctly formatted Pillar top file referencing the arista Proxy ID and a correctly formatted proxy key in the arista.sls file. Take care and check indentation levels and missing colons. Anyway, let’s give it a go, but this time we can view more by starting the service in the debug logging level (Listing 14-12).

root@master:~# salt-proxy --proxyid=arista -l debug

...

[DEBUG   ] LazyLoaded config.merge

[DEBUG   ] schedule.handle_func: Removing /var/cache/salt/proxy/arista/proc/20210618165516679011

[DEBUG   ] LazyLoaded status.proxy_reconnect

[DEBUG   ] Subprocess Thread-46-Schedule-__mine_interval

Listing 14-12  Having Accepted the Key, the Proxy Is Restarted

The screen should fill quickly with debug data, but if successful, it will settle quite quickly to output similar to Listing 14-12. At this stage, we can CTRL+C. If it was successful, we will start it as a background service. If the screen just kept scrolling, check errors after the CTRL+C. Starting Salt Proxy as a standard background service with systemd requires the use of the @ symbol setting the Proxy ID:

root@master:~# systemctl start salt-proxy@arista

With the service running in the background, we have a running Salt Proxy, and we can start managing the Arista router.

Managing Network Devices

Working on the Salt Master, we are able to manage Salt Minions and Salt Proxy devices. For some functions, such as test.ping , they can be executed across all devices. In Listing 14-13, we can see that arista Proxy responds as do the three other Minions.

root@master:~# salt '*' test.ping

arista:

True

master:

True

salt_syndic:

True

minion2:

True

Listing 14-13  Testing Salt Proxy

We can be more accurate in targeting the Arista device. Using the os grain, we can target just eos devices:

root@master:~# salt -G 'os:eos' test.ping

arista:

True

Reading Router Configuration

To become more specific to Arista and other network devices, we can begin checking the configuration. To see the complete router configuration, we can use the following command. The output is verbose and has been excluded:

root@master:~# salt -G 'os:eos' net.config source=running

For something a little less verbose, we can check the time settings from the command line of the Arista device (Listing 14-14).

root@master:~# salt -G 'os:eos' net.cli 'show clock'

arista:

----------

comment:

out:

----------

show clock:

Fri Jun 18 17:17:00 2021

Timezone: UTC

Clock source: local

result:

True

Listing 14-14  Checking Time Settings

Seeing that we are in the UTC time zone and not configured with an NTP server, we know we need some changes. Additionally, we will want to set a hostname as this was not set initially.

Writing Router Configuration

If we have a simple one-line configuration that needs to be written to our device, we do not need a configuration file or any complexity. Salt allows this to be written directly from the CLI. As the hostname will be unique to each device, as such, it is not worth adding to a shared configuration file that could be loaded across all Arista routers. This could be templated, but for the moment, we will see the simple configuration (Listing 14-15)

root@master:~# salt -G 'os:eos' net.load_config \

text='hostname arista.example.com'

arista:

----------

already_configured:

False

comment:

diff:

@@ -10,6 +10,7 @@

!

service routing protocols model ribd

!

+hostname arista.example.com

ip name-server vrf default 172.31.0.2

!

spanning-tree mode mstp

loaded_config:

result:

True

Listing 14-15  Setting the Hostname

Where we have many settings to make, especially where they apply to many systems, we can create a text configuration file to upload. In Listing 14-16, we see the file that we create and how it is uploaded to configure the time zone and set the NTP server to synchronize with.

root@master:~# vim arista.cfg

clock timezone Europe/London

ntp server uk.pool.ntp.org

root@master:~# salt -G 'os:eos' net.load_config \ filename='/root/arista.cfg'

arista:

----------

already_configured:

False

comment:

diff:

@@ -13,6 +13,8 @@

hostname arista.example.com

ip name-server vrf default 172.31.0.2

!

+ntp server uk.pool.ntp.org

+!

spanning-tree mode mstp

!

aaa authentication policy on-success log

@@ -23,6 +25,8 @@

username admin secret sha512 $6$Pu7g3kJS2I33KI.Q$J5cciN4lrtZ7wRflB7KjxfZNpT.Q5wUB8MS5GbWDzc6.KdnexcxpOSzXuOhV/TH3zaOt6HhmS7.Z0otW25S1P/

username ec2-user nopassword

username ec2-user ssh-key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy5+H8HsNr4VsIokJfXHVN7vQQrt/HgS8vlVzu/JdYxYrhaQPi3SwCm6+i+Mcck539Ceg1hQY51gxG57rdBdAOpa9oCsl/KrRM8nDSoGQBk3pbrJCksqdr4OHw6ZIJcy7Bny2Me65RlNCXmYH+xr/k8palNzqUSzHqdp0oZkdULMTJzcOccqD5p2EAK2UxjWiDFrUlfFDPHz9yV1xIf9kde+qIWidKSybG3QtUi6KtGiXfiaQ/UxK/z//h1uOlehE+Y66tHV0AhUoeS29AYbu8Pw/4IaaYlRScULsS/ohZyDhzdrJr5Ia4nPbYlkyfoz2yifMmsDKUYxKQWhVXp3YV awslondon

+!

+clock timezone Europe/London

!

interface Ethernet1

no switchport

loaded_config:

result:

True

Listing 14-16  Setting Shared Configuration Values from a File

We can provide the full path to the configuration file that we need to load. The NTP server synchronization will take a good 20 minutes, but we can read the process so far using the following command:

root@master:~# salt -G 'os:eos' ntp.stats

The output is verbose, but we are interested mainly in

synchronized:

False

This should change to True in about 20 minutes; a point to note is that you will only see output from ntp.stats when NTP is enabled, and it is not by default. Check it again in 20 minutes. When time has become synchronized, we can review the output of the show clock command again. This will show the new time zone and NTP as the time source (Listing 14-17).

root@master:~# salt -G 'os:eos' net.cli 'show clock'

arista:

----------

comment:

out:

----------

show clock:

Fri Jun 18 20:51:10 2021

Timezone: Europe/London

Clock source: NTP server (85.199.214.100)

result:

True

Listing 14-17  Time Is Now Synchronized

Gaining Help

NAPALM modules differ slightly in functionality between different systems and are accessed with virtual names. To read the documentation from the CLI, you will need to target the correct OS. In Listing 14-18, we first list functions from the net module targeting the Salt Minion master and then the arista device.

root@master:~# salt 'master' sys.list_functions net

master:

root@master:~# salt 'arista' sys.list_functions net

arista:

- net.arp

- net.blockreplace

- net.cancel_commit

- net.cli

- net.commit

- net.compare_config

- net.config

- net.config_changed

- net.config_control

- net.confirm_commit

- net.connected

- net.discard_config

- net.environment

- net.facts

- net.interfaces

- net.ipaddrs

- net.lldp

- net.load_config

- net.load_template

- net.mac

- net.optics

- net.patch

- net.ping

- net.replace_pattern

- net.rollback

- net.save_config

- net.traceroute

Listing 14-18  Correctly Target Virtual Modules

Many modules like this are virtual modules. To use the web help, you will need to use their full name before they have been virtualized with a target in mind. The module names and what appears in the web help are shown in Listing 14-19.

# find /usr/lib/python3/dist-packages/salt/modules/ -name 'napalm*.py'

/usr/lib/python3/dist-packages/salt/modules/napalm_netacl.py

/usr/lib/python3/dist-packages/salt/modules/napalm_users.py

/usr/lib/python3/dist-packages/salt/modules/napalm_network.py

/usr/lib/python3/dist-packages/salt/modules/napalm_bgp.py

/usr/lib/python3/dist-packages/salt/modules/napalm_formula.py

/usr/lib/python3/dist-packages/salt/modules/napalm_mod.py

/usr/lib/python3/dist-packages/salt/modules/napalm_probes.py

/usr/lib/python3/dist-packages/salt/modules/napalm_yang_mod.py

/usr/lib/python3/dist-packages/salt/modules/napalm_route.py

/usr/lib/python3/dist-packages/salt/modules/napalm_snmp.py

/usr/lib/python3/dist-packages/salt/modules/napalm_ntp.py

Listing 14-19  Full NAPALM Module Names

The Super Proxy

Having many network devices is very possible considering that these devices can include switches, wireless access points, routers, and network load balancers. I have worked with Salt in organizations with 2.5K devices that needed to be managed. A simple check of the status of the salt-proxy service on my system shows it using 129M of RAM:

root@master:~# systemctl status salt-proxy@arista.service

...

Memory: 129.4M

...

Having over 2000 of these services running 24/7 is an excessive waste of resources when they are used perhaps once a week. Although RAM is cheap, it is still a colossal amount of wastage. Another alternative is to use the Salt Super Proxy developed by Mircea Ulinic, one of the main and original NAPALM authors. Installing salt-sproxy onto the master along with NAPALM, the Super Proxy executes as a Salt Runner on the Salt Master without the need of any extra service, just the normal Salt Master. Maintaining the Pillar data, we can dispense with the Salt Proxy services; these can be stopped or even masked if required. In Listing 14-20, we stop the Proxy service and install salt-sproxy. Using the salt-sproxy command in place of salt, we can still access the network device without the need of the Proxy service.

root@master:~# systemctl stop salt-proxy@arista.service

root@master:~# pip3 install salt-sproxy

root@master:~# salt-sproxy 'arista' ntp.servers

arista:

----------

comment:

out:

- uk.pool.ntp.org

result:

True

Listing 14-20  Salt Sproxy as an Alternative to Salt Proxy

Using salt-sproxy in this way, we can access devices by the ID but not by using globs such as the asterisk. To enable this, we need to create a little Salt Master configuration and a roster file listing the names of the devices. We only have the single device, but the roster would normally reference all devices. Their configuration can remain in the Salt Pillar. Listing 14-21 shows the configuration and access devices using glob names.

root@master:~# vim /etc/salt/master.d/roster.conf

roster: file

roster_file: /etc/salt/sproxy-roster

root@master:~# vim /etc/salt/sproxy-roster

arista: {}

root@master:~# salt-sproxy '*' ntp.servers

arista:

----------

comment:

out:

- uk.pool.ntp.org

result:

True

Listing 14-21  Salt Sproxy Roster File

Summary

With this chapter, we close the book. By this stage, you will be well on your way to expertise in Salt and able to manage your servers and, now, your network devices. We have managed the network devices using the NAPALM abstraction layer. This could be directly from the CLI or from Python scripts. However, when used with Salt, you allow your configuration management system the true power of your complete IT infrastructure and simplified management of your networking equipment. Thank you for your time in reading this, and I look forward to meeting you again soon.

Index

A

Agentless operation

Agnostic behavior

Ansible

apache2

Arista routers

auto_accept configuration setting

AWS cloud provider

AWS network virtualization

Arista router

Salt Cloud profile

B

Beacons

creation

defining

reactors, creation

testing

triggers viewing

viewing activity

Bootstrap installer

C

CentOS 8 Stream virtual machine

CentOS Stream

CentOS system

Cipher text

Cloud profiles

Cloud providers

Compound matching

Custom Grains

adding requirements

file system structure

linking, init.sls

requirement enforcement

saving

verify

D

Debian-based systems

Debugging

reactors

state files

State tree

Default file_roots setting

Default Minion ID globbing

Default pillar_root

Default rendering engine

Default roster file

deploy attribute

Directories

Driver

E

Event bus

F

File roots configuration

Files management

filter_by function

Formulas

G

Globbing

GNU Privacy Guard (GPG) suite

GPG key pair

Grains

grains.filter_by

grains.setval function

Graphical editors

H

Host

I

image attribute

Independent salt minion

Init State file

Installation

Bootstrap

independent salt minion

list python files

minion key

Python

review minion configuration

J

Jinja logic

statements to install Vim

streamlining

YAML Map files

K

Kernel Grain

Key files

L

Lab environment, configuration

M

Managing services

Master of Masters (MoM)

Matching targets

Minion

accepting keys

configuration

installation

installing NAPALM

keys

platforms support

regenerating keys

schedules creation

Modules

N

nano text editor

NAPALM

CLI

community-authored NAPALM drivers

installation

module names

network devices

Python scripts

Native pure performance

Network devices management

NAPALM modules

router configuration

reading

writing

Nodegroups

O

Open source configuration management projects

Ansible

Chef

Puppet

Salt

Organizing content

os_family grain

P, Q

Packages

Password authentication

Physical Aruba switch

Pillar data

configuration

State files

Pillar top file

Profile Deploying Salt Minion

provider attribute

Provisioning virtual machines

Proxytype

Python State files

R

Reactors

configuration

debugging

events

investigating events

Recursive acronym

Red Hat–based systems

Regular expressions

Remote sudo configuration

Remote Execution modules

accessing help

agnostic behavior

configuration changes

flow

function

investigating Salt Modules

locating Salt Modules

management of files

managing services

Salt Minions configuration

SaltStack index

software packages installation

state

web-based help

Rendering engines

Require statements

contents

service

Reusable State files

Grains

native pure performance

Pillar data

rendering engines

S

Salt

products

working

salt.beacon.service module

salt-call test.ping

Salt Cloud

configuration

cloud profiles

cloud providers

installation

listing profiles

virtual machine instances management

deleting instances

provisioning

SaltConfig

Salt grains

Salt-Key

Salt Master

configuration

enabling auto-accept

firewall requirements

installation

mainstream Linux distributions

resolving Issues with Keys

Salt Cloud installing

time status

updating

Salt Proxy

configuration

Cipher Text to Pillar

default setting

GPG to encrypt sensitive data

Pillar data

running

restart

starting and key acceptance

super proxy

testing

topology

Salt rendering engines

salt-run command

Salt Sproxy roster file

Salt SSH

enable password authentication

installation

managing nodes

remote execution commands

State files, creation

vagrant and password authentication

SaltStack

SaltStack Remote Execution Module Index

Salt State files

Salt Syndic

configuration

formula

implementing

installation

vim formula

Salt topology

Schedules

events, creation

listing

Shebang

show_states function

Simple Top File

size attribute

SSH Cloud Profile

state.apply

State configuration management style

State files, creation

applying time zone State file

configuring files

configuring packages and services

previewing actions

rendering engines

salt-call executing state locally

SSH configuring

time zone configuration

timezone.py Module, locating

State vs. Flow

file roots configuration

state file location

state.highstate

State modules

Subnets

Super proxy

T

Targeting IP Address

Targeting Salt Minions

compound matches

Default Minion ID Globbing

globbing

Grains

lists

Minion ID shell-style globbing

Nodegroups

PCRE

Pillar Data

regular expressions

subnet/IP address

Target virtual modules

test.ping

timedatectl

Time Zone function

U

Ubuntu system

V

Vagrant

rebuilding lab systems

Salt Master installation

Salt Minion installation

Vagrant system

View pending requests

View rendering actions

View rendering times

vim text editor

Virtual Arista router

VirtualBox

W, X

Web-based help

Y

YAML

definition

graphical editors/IDEs

rendering engines

mappings

multiline strings

nano text editor

PyCharm

sequences

SLS files configuration

writing

vim text editor

Z

ZeroMQ message

Table of Contents

Front Matter

1. Understanding Salt and Configuration Automation

2. Installing Salt

3. Adding and Removing Salt Minions

4. Targeting Salt Minions

5. Working with Remote Execution Modules in Salt Open

6. Writing YAML Files

7. Writing Salt State Files

8. Building an Effective State Tree

9. Creating Reusable State Files

10. Implementing Reactors and Beacons

11. Using Salt SSH

12. Deploy Virtual Machines Using Salt Cloud

13. Scaling Configuration Management Using Salt Syndic

14. Automating Network Infrastructure with Salt Proxy

Back Matter