Kubernetes – An Enterprise Guide

Copyright © 2021 Packt Publishing

This is an Early Access product. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the content and extracts of this book may evolve as it is being developed to ensure it is up-to-date.

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

The information contained in this book is sold without warranty, either express or implied. Neither the author nor Packt Publishing or its dealers and distributors will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Early Access Publication: Kubernetes – An Enterprise Guide

Early Access Production Reference: B17950

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK

ISBN: 978-1-80323-003-0

www.packt.com

Table of Contents

1. Kubernetes – An Enterprise Guide, Second Edition: Effectively containerize applications, integrate enterprise systems, and scale applications in your enterprise

2. Docker and Container Essentials

1. Technical requirements

2. Understanding the need for containerization

1. Understanding Kubernetes Deprecating Docker

2. Introducing Docker

3. Understanding Docker

1. Containers are ephemeral

2. Docker images

3. Image layers

4. Persistent data

5. Accessing services running in containers

4. Installing Docker

1. Preparing to install Docker

2. Installing Docker on Ubuntu

3. Granting Docker permissions

5. Using the Docker CLI

1. docker help

2. docker run

3. docker ps

4. docker start and stop

5. docker attach

6. docker exec

7. docker logs

8. docker rm

6. Summary

7. Questions

3. Deploying Kubernetes Using KinD

1. Technical requirements

2. Introducing Kubernetes components and objects

1. Interacting with a cluster

3. Using development clusters

1. Why did we select KinD for this book?

2. Working with a base KinD Kubernetes cluster

3. Understanding the node image

4. KinD and Docker networking

4. Installing KinD

1. Installing KinD – prerequisites

2. Installing the KinD binary

5. Creating a KinD cluster

1. Creating a simple cluster

2. Deleting a cluster

3. Creating a cluster config file

4. Multi-node cluster configuration

5. Customizing the control plane and Kubelet options

6. Creating a custom KinD cluster

7. Installing Calico

8. Installing an Ingress controller

6. Reviewing your KinD cluster

1. KinD storage objects

2. Storage drivers

3. KinD storage classes

4. Using KinD's storage provisioner

7. Adding a custom load balancer for Ingress

1. Installation prerequisites

2. Creating the KinD cluster configuration

3. Deploying a custom HAProxy container

4. Understanding HAProxy traffic flow

5. Simulating a Kubelet failure

8. Summary

9. Questions

4. Kubernetes Bootcamp

1. Technical requirements

2. An overview of Kubernetes components

3. Exploring the control plane

1. The Kubernetes API server

2. The Etcd database

3. kube-scheduler

4. kube-controller-manager

5. cloud-controller-manager

4. Understanding the worker node components

1. kubelet

2. kube-proxy

3. Container runtime

5. Interacting with the API server

1. Using the Kubernetes kubectl utility

2. Understanding the verbose option

3. General kubectl commands

6. Introducing Kubernetes objects

1. Kubernetes manifests

2. What are Kubernetes objects?

3. Reviewing Kubernetes objects

7. Summary

8. Questions

1. Cover

2. Table of contents

Kubernetes – An Enterprise Guide, Second Edition: Effectively containerize applications, integrate enterprise systems, and scale applications in your enterprise

Welcome to Packt Early Access. We’re giving you an exclusive preview of this book before it goes on sale. It can take many months to write a book, but our authors have cutting-edge information to share with you today. Early Access gives you an insight into the latest developments by making chapter drafts available. The chapters may be a little rough around the edges right now, but our authors will update them over time. You’ll be notified when a new version is ready.

This title is in development, with more chapters still to be written, which means you have the opportunity to have your say about the content. We want to publish books that provide useful information to you and other customers, so we’ll send questionnaires out to you regularly. All feedback is helpful, so please be open about your thoughts and opinions. Our editors will work their magic on the text of the book, so we’d like your input on the technical elements and your experience as a reader. We’ll also provide frequent updates on how our authors have changed their chapters based on your feedback.

You can dip in and out of this book or follow along from start to finish; Early Access is designed to be flexible. We hope you enjoy getting to know more about the process of writing a Packt book. Join the exploration of new topics by contributing your ideas and see them come to life in print.

1. Chapter 1: Docker and Container Essentials

2. Chapter 2: Deploying Kubernetes Using KinD

3. Chapter 3: Kubernetes Bootcamp

4. Chapter 4: Services, Load Balancing, and External DNS

5. Chapter 5: Integrating Authentication into Your Cluster

6. Chapter 6: RBAC Policies and Auditing

7. Chapter 7: Deploying a Secured Kubernetes Dashboard

8. Chapter 8: Extending Security Using Open Policy Agent

9. Chapter 9: Node Security with Gatekeeper

10. Chapter 10: Auditing Using Falco and EFK

11. Chapter 11: Backing up Workloads

12. Chapter 12: Istio Deployment

13. Chapter 13: Building and Deploying Applications on Istio

14. Chapter 14: Provisioning a Platform

Docker and Container Essentials

Before we begin, you may have read that Kubernetes will be deprecating Docker as a compatible runtime in an upcoming release. We will explain this change in more details in the next section, but for now, we will continue with the introduction to containers and the advantages that they deliver.

Containers are one of the most transformational technologies that we have seen in years. Technology companies, corporations, and end users have all adopted it to handle everyday workloads. Increasingly, common off-the-shelf (COTS) applications are transforming from traditional installations into fully containerized deployments. With such a large technology shift, it is essential for anyone in the Information Technology realm to learn about containers.

In this chapter, we will introduce the problems that containers address. After an introduction to why containers are important, we will introduce the runtime that launched the modern container frenzy, Docker and explain its relationship to Kubernetes. We’ll also cover how Kubernetes’ recent deprecation of support for Docker as a runtime impacts the use of Docker and why you should still be familiar with how to use it. By the end of this chapter, you will understand how to install Docker and how to use the most common Docker CLI commands.

In this chapter, we will cover the following topics:

- Understanding the need for containerization

- Kubernetes Deprecating Docker

- Understanding Docker

- Installing Docker

- Using the Docker CLI

Let's get started!

Technical requirements

This chapter has the following technical requirements:

- An Ubuntu 18.04+ server with a minimum of 4 GB of RAM, though 8 GB is suggested

- You can access the code for this chapter by going to the following GitHub repository: https://github.com/PacktPublishing/Kubernetes-and-Docker-The-Complete-Guide.

Understanding the need for containerization

You may have experienced a conversation like this at your office or school:

Developer: "Here's the new application. It went through weeks of testing and you are the first to get the new release."

….. A little while later ….

User: "It's not working. When I click the submit button, it shows an error about a missing dependency."

Developer: "That's weird; it's working fine on my machine."

This is one of the most frustrating things a developer can encounter when delivering an application. Often, the issues that creep up are related to a library that the developer had on their machine, but it wasn't included in the distribution of the package. It may seem like an easy fix for this would be to include all the libraries alongside the release, but what if this release contains a newer library that overwrites the older version, which may be required for a different application?

Developers need to consider their new releases, as well as any potential conflicts with any existing software on the user's workstations. This often becomes a careful balancing act that requires larger deployment teams to test the application on different system configurations. It can also lead to additional rework for the developer or, in some extreme cases, full incompatibility with an existing application.

There have been various attempts to make application delivery easier over the years. First, there are solutions such as VMware's Thinapp, which virtualizes an application (not to be confused with virtualizing an operating system). It allows you to package the application and its dependencies into a single executable package. This packaging eliminates the issues of an application's dependencies conflicting with another application's dependencies since the application is in a self-contained package. This provided application isolation not only eliminates dependency issues but also provides an enhanced level of security and eases the burden of operating system migrations.

You may or may not have heard of application streaming before reading this book. It sounds like a great solution to the "it worked on my machine" issue. There are many reasons it hasn't taken off as expected, though. For starters, most offerings are paid solutions that require a substantial investment. Besides licensing, they require a "clean PC," which means that for every application you want to virtualize, you need to start with a base system. The package you want to create uses the differences between the base installation and anything that was added after the initial system snapshot. The differences are then packaged into your distribution file, which can be executed on any workstation.

We've mentioned application virtualization to highlight that application issues such as "It works on my machine" have had different solutions over the years. Products such as Thinapp are just one attempt at solving the problem. Other attempts include running the application on a server running Citrix or Remote Desktop, Linux containers, and even virtual machines.

Understanding Kubernetes Deprecating Docker

In December 2020, Kubernetes announced the deprecation of Docker as a supported container runtime. We thought it would be important to explain how the announcement effects any reason for using, or not using, Docker.

The announcement is only related to using Docker as the container runtime in a cluster – It is important to note that this is the only impact that removing Docker will have. You can still create new containers using Docker and they will run on any runtime that supports the OCI (Open Container Initiative) specification. When you create a container using Docker, you are creating a container that is OCI compliant, so it will still run on Kubernetes clusters that are not using Docker as the runtime.

To fully explain the impact and the alternatives that will be supported, we need to understand what a container runtime is. A high-level definition would be that a container runtime is the software layer that runs and manages containers. Like many components that make up a Kubernetes cluster, the runtime is not included as part of Kubernetes – it is a pluggable module that needs to be supplied by a vendor, or by you, to create a functioning cluster.

There are many technical reasons that led up to the decision to deprecate Docker, but at a high level, the main concerns were:

- Docker contains multiple pieces inside of the Docker executable to support its own remote API and user experience (UX). Kubernetes only requires one component in the executable, dockerd, which is the runtime process that manages containers. All other pieces of the executable contributed nothing to using Docker in a Kubernetes cluster.

- Docker does not conform to the Container Runtime Interface (CRI) standard, which were introduced to create a set of standards to easily integrate container runtimes in Kubernetes. Since it didn’t comply, the Kubernetes team has had extra work that only caters to supporting Docker.

When it comes to local container testing and development, you can still use Docker on your workstation or server. Building on the previous statement, if you build a container on Docker and the container successfully runs on your Docker runtime system, it will run on a Kubernetes cluster that is not using Docker as the runtime.

Removing Docker will have very little impact to most users of Kubernetes in new clusters. Containers will still run using any standard method, as they would with Docker as the container runtime. If you happen to manage a cluster, you may need to learn new commands when you troubleshoot a Kubernetes node – you will not have a Docker command on the node to look at running containers, or to clean up volumes, etc…

At the time of updating this chapter, Kubernetes will support the following runtimes in place of Docker:

- ContainerD

- Rocket (rkt)

- CRI-O

- Frakti

Since we are focusing on general containers and we will be using Docker as our runtime to create KinD clusters, we will not go into too many details on the alternative runtimes. They are only being mentioned here to explain the alternatives that can be used on a cluster.

For more details on the impact of deprecating Docker, reference the article called Don't Panic: Kubernetes and Docker on the Kubernetes.io site at https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/

Now, let’s introduce Docker and how you can use it to create and manage containers.

Introducing Docker

The industry and even end users needed something that was easier and cheaper – enter Docker containers. Containers are not a new technology; they have been used in various forms for years. What Docker did was make them accessible to the average developer.

Docker brought an abstraction layer to the masses. It was easy to use and didn't require a clean PC for every application before creating a package, thus offering a solution for dependency issues, but most attractive of all, it was free. Docker became a standard for many projects on GitHub, where teams would often create a Docker container and distribute the Docker image or Dockerfile to team members, providing a standard testing or development environment. This adoption by end users is what eventually brought Docker to the enterprise and, ultimately, what made it the standard it has become today.

While there are many books on Docker, this book focuses on the base topics of Docker that are used to interact with containers. This book will be focusing on what you will need to know when trying to use a local Kubernetes environment. There is a long and interesting history of Docker and how it evolved into the standard container image format that we use today. We encourage you to read about the company and how they ushered in the container world we know today.

While our focus is not to teach Docker inside-out, we felt that those of you who are new to Docker would benefit from a quick primer on general container concepts. If you have some Docker experience and understand terminology such as ephemeral and stateless, feel free to continue to the Installing Docker section.

Understanding Docker

This book was created with the assumption that you have some basic knowledge of Docker and container concepts. We realize that not everyone may have played with Docker or containers in the past, so we wanted to present a crash course on container concepts and using Docker.

Important Note

If you are new to containers, we suggest reading the documentation that can be found on Docker's website for additional information: https://docs.docker.com/.

Containers are ephemeral

The first topic to understand is that container images are ephemeral.

For those of you who are new to Docker, the term ephemeral means short-lived. By design, a container can be destroyed at any time and brought back up with no interaction from a user. In the preceding example, someone interactively added files to a web server. These added files are only temporary since the base image does not have these files included in it.

This means that once a container is created and running, any changes that are made to the image will not be saved once the container is removed, or destroyed, from the Docker host. Let's look at an example:

- You start a container running a web server using NGINX on your host without any base HTML pages.

- Using a Docker command, you execute a  copy  command to copy some web files into the container's filesystem.

- To test that the copy was successful, you browse to the website and confirm that it is serving the correct web pages.

- Happy with the results, you stop the container and remove it from the host. Later that day, you want to show a co-worker the website and you start your NGINX container. You browse to the website again, but when the site opens, you receive a 404 error (page not found error).

What happened to the files you uploaded before you stopped and removed the container from the host?

The reason your web pages cannot be found after the container was restarted is because all containers are ephemeral.

Whatever is in the base container image is all that will be included each time the container is initially started. Any changes that you make inside a container are short-lived.

If you needed to add permanent files to the existing image, you would need to rebuild the image with the files included or, as we will explain in the Persistent data section later in this chapter, you could mount a Docker volume in your container. At this point, the main concept to understand is that containers are ephemeral.

But wait! You may be wondering, "If containers are ephemeral, how did I add web pages to the server?". Ephemeral just means that changes will not be saved; it doesn't stop you from making changes to a running container.

Any changes made to a running container will be written to a temporary layer, called the container layer, which is a directory on the local host filesystem. The Docker storage driver is in charge of handling requests that use the container layer. This location will store any changes in the container's filesystem so that when you added the HTML pages to the container, they will be stored on the local host. The container layer is tied to the container ID of the running image and it will remain on the host system until the container is removed from Docker, either by using the CLI or by running a Docker prune job.

If a container is ephemeral and the image cannot be written to, how can you modify data in the container? Docker uses image layering to create multiple linked layers that appear as a single filesystem.

Docker images

At a high level, a Docker image is a collection of image layers, each with a JSON file that contains metadata for the layer. These are all combined to create the running application that you interact with when a container image is started.

You can read more about the contents of an image on Docker's GitHub at https://github.com/moby/moby/blob/master/image/spec/v1.md.

Image layers

As we mentioned in the previous section, a running container uses a container layer that is "on top" of the base image layer, as shown in the following diagram:

Figure 1.1 – Docker image layers

The image layers cannot be written to since they are in a read-only state, but the temporary container layer is in a writeable state. Any data that you add to the container is stored in this layer and will be retained as long as the container is running.

To deal with multiple layers efficiently, Docker implements copy-on-write, which means that if a file already exists, it will not be created. However, if a file is required that does not exist in the current image, it will be written. In the container world, if a file exists in a lower layer, the layers above it do not need to include it. For example, if layer 1 had a file called  /opt/nginx/index.html  in it, layer 2 does not need the same file in its layer.

This explains how the system handles files that either exist or do not exist, but what about a file that has been modified? There will be times where you'll need to "replace" a file that is in a lower layer. You may need to do this when you are building an image or as a temporary fix to a running container issue. The copy-on-write system knows how to deal with these issues. Since images read from the top down, the container uses only the highest layer file. If your system had a  /opt/nginx/index.html  file in layer 1 and you modified and saved the file, the running container would store the new file in the container layer. Since the container layer is the topmost layer, the new copy of  index.html  would always be read before the older version in the image layer.

Persistent data

Being limited to ephemeral-only containers would severely limit the use cases for Docker. It is very likely that you will have some use cases that will require persistent storage, or data that will remain if you stop a container.

This may seem like we are contradicting our earlier statement that containers are ephemeral, but that is still true. When you store data in the container image layer, the base image does not change. When the container is removed from the host, the container layer is also removed. If the same image is used to start a new container, a new container image layer is also created. So, the container is ephemeral, but by adding a Docker volume to the container, you can store data outside of the container, thus gaining data persistency.

Accessing services running in containers

Unlike a physical machine or a virtual machine, containers do not connect to a network directly. When a container needs to send or receive traffic, it goes through the Docker host system using a bridged NAT network connection. This means that when you run a container and you want to receive incoming traffic requests, you need to expose the ports for each of the containers that you wish to receive traffic on. On a Linux-based system,  iptables  has rules to forward traffic to the Docker daemon, which will service the assigned ports for each container.

That completes the introduction to base containers and Docker. In the next section, we will explain how to install Docker on a host.

Installing Docker

The hands-on exercises in this book will require that you have a working Docker host. You can follow the steps in this book, or you can execute the script located in this book's GitHub repository, in the  chapter1  directory, called  install-docker.sh .

Today, you can install Docker on just about every hardware platform out there. Each version of Docker acts and looks the same on each platform, making development and using Docker easy for people who need to develop cross-platform applications. By making the functions and commands the same between different platforms, developers do not need to learn a different container runtime to run images.

The following is a table of Docker's available platforms. As you can see, there are installations for multiple operating systems, as well as multiple CPU architectures:

Figure 1.2 – Available Docker platforms

Important Note

Images that are created using one architecture cannot run on a different architecture. This means that you cannot create an image based on x86 hardware and expect that same image to run on your Raspberry Pi running an ARM processor. It's also important to note that while you can run a Linux container on a Windows machine, you cannot run a Windows container on a Linux machine.

The installation procedures that are used to install Docker vary between platforms. Luckily, Docker has documented many of the installation procedures on their website: https://docs.docker.com/install/.

In this chapter, we will install Docker on an Ubuntu 18.04 system. If you do not have an Ubuntu machine to install on, you can still read about the installation steps, as each step will be explained and does not require that you have a running system to understand the process. If you have a different Linux installation, you can use the installation procedures outlined on Docker's site at https://docs.docker.com/. Steps are provided for CentOS, Debian, Fedora, Ubuntu, and there are generic steps for other Linux distributions.

Preparing to install Docker

Before we start the installation, we need to consider what storage driver to use. The storage driver is what provides the union filesystem, which manage the layers of the container and how the writeable layer of the container is accessed.

In most installations, you won't need to change the default storage driver since a default option will be selected. If you are running a Linux kernel that is at least version 4.0 or above, your Docker installation will use the  overlay2  storage driver; earlier kernels will install the  AUFS  storage driver.

For reference, along with the  overlay2  and  AUFS  drivers, Docker supports the  btrfs  storage driver. However, these are rarely used in new systems and are only mentioned here as a reference.

If you would like to learn about each storage driver, take a look at the following Docker web page, which details each driver and their use cases: https://docs.docker.com/storage/storagedriver/select-storage-driver/.

Now that you understand the storage driver requirements, the next step is to select an installation method. You can install Docker using one of three methods:

- Add the Docker repositories to your host system.

- Install the package manually.

- Use a supplied installation script from Docker.

The first option is considered the best option since it allows for easy installation and making updates to the Docker engine. The second option is useful for enterprises that do not have internet access to servers, also known as air-gapped servers. The third option is used to install edge and testing versions of Docker and is not suggested for production use.

Since the preferred method is to add Docker's repository to our host, we will use that option and explain the process we should use to add the repository and install Docker.

Installing Docker on Ubuntu

Now that we have finished preparing everything, let's install Docker:

1. The first step is to update the package index by executing  apt-get update :

sudo apt-get update

2. Next, we need to add any packages that may be missing on the host system to allow HTTPS apt access:

sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release

3. To pull packages from Docker's repository, we need to add their keys. You can add keys by using the following command, which will download the  gpg  key and add it to your system:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

4. Now, add Docker's repository to your host system:

echo “deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

5. With all the prerequisites completed, you can install Docker on your server:

sudo apt-get update && sudo apt-get install docker-ce docker-ce-cli containerd.io

6. Docker is now installed on your host, but like most new services, Docker is not currently running and has not been configured to start with the system. To start Docker and enable it on startup, use the following command:

sudo systemctl enable docker && systemctl start docker

Now that we have Docker installed, let's get some configuration out of the way. First, we'll grant permissions to Docker.

Granting Docker permissions

In a default installation, Docker requires root access, so you will need to run all Docker commands as root. Rather than using  sudo  with every Docker command, you can add your user account to a new group on the server that provides Docker access without requiring  sudo  for every command.

If you are logged on as a standard user and try to run a Docker command, you will receive an error:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied

To allow your user, or any other user you may want to add to execute Docker commands, you need to create a new group and add the users to that group. The following is an example command you can use to add the currently logged on user:

sudo groupadd docker && sudo usermod -aG docker $USER

The first command creates the  docker  group, while the second command adds the user account that you are currently logged in with to the  docker  group.

To add the new membership to your account, activate the group changes using the  newgrp  command:

newgrp docker

Finally, you can test that it works by running the standard hello world image (note that we do not require  sudo  to run the Docker command):

docker run hello-world

You should see the output shown below, which verifies that your user has access to Docker:

Figure 1.3 –Output for hello-world

Now that we've granted Docker permission to run without  sudo , we can start unlocking the commands at our disposal by learning how to use the Docker CLI.

Using the Docker CLI

You used the Docker CLI when you ran the  hello-world  container to test your installation. The Docker command is what you will use to interact with the Docker daemon. Using this single executable, you can do the following, and more:

- Start and stop containers

- Pull and push images

- Run a shell in an active container

- Look at container logs

- Create Docker volumes

- Create Docker networks

- Prune old images and volumes

This chapter is not meant to include an exhaustive explanation of every Docker command; instead, we will explain some of the common commands that you will need to use to interact with the Docker daemon and containers. Since we consider volumes and networking to be very important to understand for this book, we will go into additional details on those topics.

You can break down Docker commands into two categories: general Docker commands and Docker management commands. The standard Docker commands allow you to manage containers, while management commands allow you to manage Docker options such as managing volumes and networking.

docker help

It's common to forget an option or the syntax for a command, and Docker realizes this. Whenever you get stuck trying to remember a command, you can always use the  docker help  command to refresh your memory.

docker run

To run a container, use the  docker run  command with the provided image name. Before executing a  docker run  command, you should understand the options you can supply when starting a container.

In its simplest form, an example command you can use to run a NGINX web server would be  docker run bitnami/nginx:latest . While this will start a container running NGINX, it will run in the foreground:

Figure 1.4 – NGINX container startup

To run a container as a background process, you need to add the  -d  option to your Docker command, which will run your container in detached mode. Now, when you run a detached container, you will only see the container ID, instead of the interactive, or attached, screen:

Figure 1.5 – Docker run output

By default, containers will be given a random name once they are started. In our previous detached example, the container has been given the name  silly_keldysh :

Figure 1.6 – Docker naming example

If you do not assign a name to your container, it can quickly get confusing as you start to run multiple containers on a single host. To make management easier, you should always start your container with a name that will make it easier to manage. Docker provides another option with the  run  command: the  --name  option. Building on our previous example, we will name our container  nginx-test . Our new  docker run  command will be as follows:

docker run --name nginx-test -d bitnami/nginx:latest

Just like running any detached image, this will return the containers ID, but not the name you provided. In order to verify the container ran with the name  nginx-test , we can list the containers using the  docker ps  command.

docker ps

Every day, you will need to retrieve a list of running containers or a list of containers that have been stopped. The Docker CLI has an option called  ps  that will list all running containers, or if you add an extra option to the  ps  command, all containers that are running and have been stopped. The output will list the containers, including their container ID, image tag, entry command, the creation date, status, ports, and the container name. The following is an example of containers that are currently running:

Figure 1.7 – Currently running containers

This is helpful if the container you are looking for is currently running. What if the container was stopped, or even worse, what if you started the container and it failed to start and then stopped? You can view the status of all containers, including previously run containers, by adding the  -a  option to the  docker ps  command. When you execute  docker ps -a , you will see the same output from a standard  ps  command, but you will notice that the list may include additional containers.

How can you tell what containers are running versus which ones have stopped? If you look at the  STATUS  field of the list, the running containers will show a running time; for example,  Up xx hours , or  Up xx days . However, if the container has been stopped for any reason, the status will show when it stopped; for example,  Exited (1) 3 days ago .

Figure 1.8 – Docker PS output

A stopped container does not mean there was an issue running the image. There are containers that may execute a single task and, once completed, the container may stop gracefully. One way to determine whether an exit was graceful or if it was due to a failed startup is to check the logs of the container.

docker start and stop

To stop a running container, use the  docker stop  option with the name of the container you want to stop. You may wish to stop a container due to the resources on the host since you may have limited resources and can only run a few containers simultaneously.

If you need to start that container at a future time for additional testing or development, execute  docker start container name , which will start the container with all of the options that it was originally started with, including any networks or volumes that were assigned.

docker attach

You may need to access a container interactively to troubleshoot an issue or to look at a log file. One method to connect to a running container is to use the  docker attach container name  command. When you attach to a running container, you will connect to the running containers process, so if you attach to a container running a process, you are not likely to just see a command prompt of any kind. In fact, you may see nothing but a blank screen for some time until the container outputs some data to the screen.

You must be careful once you attach to the container – you may accidentally stop the running process and, in turn, stop the container. Let's use an example of attaching to a web server running NGINX. First, we need to verify that the container is running using  docker ps :

Figure 1.9 – docker ps output

Using the  attach  command, we execute  docker attach bbadb2bddaab :

Figure 1.10 – docker attach output

As shown in the preceding screenshot, once you attach to the running container process, it appears that nothing is happening. When you attach to a process, you will only be able to interact with the process, and the only output you will see is data being sent to standard output. In the case of the NGINX container, the  attach  command has attached to the NGINX process. To show this, we will leave the attachment and curl to the web server from another session. Once we curl to the container port, you will see logs outputted to the attached console:

Figure 1.11 – STDOUT output from the container

Attaching to a running container has varying benefits, depending on what is running in the container.

We mentioned that you need to be careful once you attach to the container. Those who are new to Docker may attach to the NGINX image and assume that nothing is happening on the server or the attach failed. Since they think that there may be an issue, since it's just sitting there, they may decide to break out of the container using the standard Ctrl + C keyboard command. This will send them back to a bash prompt, where they may run  docker ps  to look at the running containers:

Figure 1.12 – docker ps output

Where is the NGINX container? We didn't execute a  docker stop  command, and the container was running until we attached to the container. Why did the container stop after the attachment?

When an attachment is made to a container, you are attached to the running process. All keyboard commands will act in the same way as if you were at a physical server that was running NGINX in an interactive shell. This means that when the user used Ctrl + C to return to a prompt, they stopped the running NGINX process. If a container's running process stops, the container will also stop, and that's why the  docker ps  command does not show a running container.

Rather than use ctrl-c to return to a prompt, the user should have used Ctrl + P, followed by Ctrl + Q.

There is an alternative to the  attach  command: the  docker exec  command. The  exec  command differs from an  attach  command since you supply the process to execute on the container.

docker exec

A better option when it comes to interacting with a running container is the  exec  command. Rather than attach to the container, you can use the  docker exec  command to execute a process in the container. You need to supply the container name and the process you want to execute in the image. Of course, the process must be included in the running image – if you do not have the bash executable in the image, you will receive an error when trying to execute bash in the container.

We will use a NGINX container as an example again. We will verify that NGINX is running using  docker ps  and then using the container ID or the name, we execute into the container. The command syntax is  docker exec <options> <container name> <process> :

Figure 1.13 – docker exec example

The option we included is  -it , which tells  exec  to run in an interactive TTY session. Here, the process we want to execute is bash. Notice how the name changed from the original user and hostname. The host name is  Blade , while the container name is  0a7c916e7411 . You may also have noticed that the current working directory changed from  ~  to  /app  and that the prompt is not running as a root user, as shown by the  $  prompt.

You can use this session the same way you would a standard SSH connection; you are running bash in the container.

Since we are not attached to the container, ctrl-c will not stop any process from running. To exit an interactive session, you only need to type in  exit , followed by Enter, to exit the container. If you then run  docker ps , you will notice that the container is still in a running state:

Figure 1.14 – docker ps output

Next, let's see what we can learn about Docker log files.

docker logs

The  docker logs  command allows you to retrieve logs from a container using the container name or container ID that you retrieved using the  docker ps  command. You can view the logs from any container that was listed in your  ps  command; it doesn't matter if it's currently running or stopped.

Log files are often the only way to troubleshoot why a container may not be starting up, or why a container is in an exited state. For example, if you attempted to run an image and the image starts and suddenly stops, you may find the answer by looking at the logs for that container.

To look at the logs for a container, you can use the  docker logs <container ID or name>  command.

To view the logs for a container with a container ID of  7967c50b260f , you would use the following command:

docker logs 7967c50b260f

This will output the logs from the container to your screen, which may be very long and verbose. Since many logs may contain a lot of information, you can limit the output by supplying the  logs  command with additional options. The following table lists the options available for viewing logs:

docker rm

Once you name a container, the assigned name cannot be used to start a different container unless you remove it using the  docker rm  command. If you had a container running called  nginx-test  that was stopped and you attempted to start another container with the name  nginx-test , the Docker daemon would return an error, stating that the name is in use:

Figure 1.15 – Docker naming conflict error

This container is not running, but the daemon knows that the container name was used previously and that it's still in the list of previously run containers.

If you want to reuse the same name, you need to remove the container before starting another container with that name. This is a common scenario when you are testing container images. You may start a container only to discover an issue with the application or image. You stop the container, fix the image/application issue, and want to redeploy using the same name. Since the name was in use previously and is still part of the Docker history, you will need to remove the image before reusing the name.

We haven't discussed volumes yet, but when removing a container that has a volume, or volumes, attached, it's a good practice to add the  -v  option to your remove command. Adding the  -v  option to the  docker rm  command will remove any volumes that were attached to the container.

Summary

In this chapter, you learned how Docker can be used to solve common development issues, including the dreaded "It works on my machine" problem. We also presented an introduction to the most commonly used Docker CLI commands that you will use on a daily basis. We closed out this chapter by looking and how to handle persistent data for a container and customizing container networking.

Questions

A single Docker image can be used on any Docker host, regardless of the architecture used.

1. True

2. False

What does Docker use to merge multiple image layers into a single filesystem?

1. Merged filesystem

2. NTFS filesystem

3. EXT4 filesystem

4. Union filesystem

Kubernetes is only compatible with the Docker runtime engine?

1. True

2. False

When you edit a container's filesystem interactively, what layer are the changes written to?

1. Operating system layer

2. Bottom-most layer

3. Container layer

4. Ephemeral layer

Assuming the image contains the required binaries, what Docker command allows you to gain access to a container's bash prompt?

1. docker shell -it <container> /bin/bash

2. docker run -it <container> /bin/bash

3. docker exec -it <container> /bin/bash

4. docker spawn -it <container> /bin/bash

When a container is stopped, the Docker daemon will delete all traces of the container.

1. True

2. False

What command will show you a list of all containers, including any stopped containers?

1. docker ps -all

2. docker ps -a

3. docker ps -list

4. docker list all

Deploying Kubernetes Using KinD

One of the largest obstacles to learning Kubernetes is having enough resources to create a cluster for testing or development. Like most IT professionals, we like to have a Kubernetes cluster on our laptops for demonstrations and for testing products in general.

Often, you may have a need to run multiple clusters for a complex demonstration, such as a multi-cluster service mesh or testing  kubefed2 . These scenarios would require multiple servers to create the necessary clusters, which, in turn, would require a lot of RAM and a hypervisor.

To do full testing on a multiple cluster scenario, you would need to create six nodes for each cluster. If you created the clusters using virtual machines, you would need to have enough resources to run 6 virtual machines. Each of the machines would have an overhead including disk space, memory, and CPU utilization.

But what if you could create a cluster using just containers? Using containers, rather than full virtual machines, will give you the ability to run additional nodes due to the reduced system requirements, create and delete clusters in minutes with a single command, script cluster creation, and allow you to run multiple clusters on a single host.

Using containers to run a Kubernetes cluster provides you with an environment that would be difficult for most people to deploy using virtual machines, or physical hardware due to resource constraints. To explain how to run a cluster using only containers locally, we will use KinD to create a Kubernetes cluster on your Docker host. We will deploy a multi-node cluster that you will use in future chapters to test and deploy components such as Ingress controllers, authentication, RBAC, security policies, and more.

In this chapter, we will cover the following topics:

- Introducing Kubernetes components and objects

- Using development clusters

- Installing KinD

- Creating a KinD cluster

- Reviewing your KinD cluster

- Adding a custom load balancer for Ingress

Let's get started!

Technical requirements

This chapter has the following technical requirements:

- A Docker host installed using the steps from Chapter 1, Docker and Container Essentials

- Installation scripts from this book's GitHub repository

You can access the code for this chapter by going to this book's GitHub repository: https://github.com/PacktPublishing/Kubernetes-and-Docker-The-Complete-Guide.

Note

We thought it was important to point out that this chapter will reference multiple Kubernetes objects, some without a lot of context. Chapter 3, Kubernetes Bootcamp, goes over Kubernetes objects in detail, many with commands you can use to understand them, so we thought having a cluster to use while reading about this would be useful.

Most of the base Kubernetes topics covered in this chapter will be discussed in future chapters, so if some topics may be a bit foggy after you've read this chapter, don't fear! They will be discussed in detail in later chapters.

Introducing Kubernetes components and objects

Since this chapter will refer to common Kubernetes objects and components, we wanted to provide a short table of terms that you will see and a brief definition of each to provide context.

In Chapter 3, Kubernetes Bootcamp, we will go over the components of Kubernetes and the base set of objects that are included in a cluster. We will also discuss how to interact with a cluster using the kubectl executable:

Table 2.1 – Kubernetes components and objects

While these are only a few of the objects that are available in a Kubernetes cluster, they are the main objects we will mention in this chapter. Knowing what each object is and having basic knowledge of their functionality will help you understand this chapter and deploy a KinD cluster.

Interacting with a cluster

To test our KinD installation, we will interact with the cluster using the kubectl executable. We will go over kubectl in Chapter 3, Kubernetes Bootcamp, but since we will be using a few commands in this chapter, we wanted to provide the commands we will use in a table with an explanation of what the options provide:

Table 2.2 – Basic kubectl commands

In this chapter, you will use these basic commands to deploy parts of the cluster that we will use throughout this book.

Next, we will introduce the concept of development clusters and then focus on one of the most popular tools used to create development clusters: KinD.

Using development clusters

Over the years, various tools have been created to install development Kubernetes clusters, allowing admins and developers to perform testing on a local system. Many of these tools worked for basic Kubernetes tests, but they often had limitations that made them less than ideal for quick, advanced scenarios.

Some of the most common solutions available are as follows:

- Docker Desktop

- minikube

- kubeadm

- K3s

Each solution has benefits, limitations, and use cases. Some solutions limit you to a single node that runs both the control plane and worker nodes. Others offer multi-node support but require additional resources to create multiple virtual machines. Depending on your development or testing requirements, these solutions may not fill your needs completely.

It seems that a new solution is coming out every few weeks, and one of the newest options for creating development clusters is a project from a Kubernetes in Docker (KinD) Kubernetes SIG.

Using a single host, KinD allows you to create multiple clusters, and each cluster can have multiple control plane and worker nodes. The ability to run multiple nodes allows advanced testing that would have required more resources using another solution. KinD has been very well received by the community and has an active Git community at https://github.com/kubernetes-sigs/kind, as well as a Slack channel (#kind).

Note

Do not use KinD as a production cluster or expose a KinD cluster to the internet. While KinD clusters offer most of the same features you would want in a production cluster, it has not been designed for production environments.

Why did we select KinD for this book?

When we started this book, we wanted to include theory, as well as hands-on experience. KinD allows us to provide scripts to spin up and spin down clusters, and while other solutions can do something similar, KinD can create a new multi-node cluster in minutes. We wanted to separate the control plane and worker nodes to provide a more "realistic" cluster. In order to limit the hardware requirements and to make Ingress easier to configure, we will only create a two-node cluster for the exercises in this book.

A multi-node cluster can be created in a few minutes and once testing has been completed, clusters can be torn down in a few seconds. The ability to spin up and spin down clusters makes KinD the perfect platform for our exercises. KinD's requirements are simple: you only need a running Docker daemon to create a cluster. This means that it is compatible with most operating systems, including the following:

- Linux

- macOS running Docker Desktop

- Windows running Docker Desktop

- Windows running WSL2

Important note

At the time of writing, KinD does not offer support for Chrome OS.

While KinD supports most operating systems, we have selected Ubuntu 20.04 as our host system. Some of the exercises in this book require files to be in specific directories and selecting a single Linux version helps us make sure the exercises work as designed. If you do not have access to an Ubuntu server at home, you can create a virtual machine in a cloud provider such as GCP. Google offers $300 in credit, which is more than enough to run a single Ubuntu server for a few weeks. You can view GCP's free options at https://cloud.google.com/free/.

Now, let's explain how KinD works and what a base KinD Kubernetes cluster looks like, before we move on to creating our first cluster.

Working with a base KinD Kubernetes cluster

At a high level, you can think of a KinD cluster as consisting of a single Docker container that runs a control plane node and a worker node to create a Kubernetes cluster. To make the deployment easy and robust, KinD bundles every Kubernetes object into a single image, known as a node image. This node image contains all the required Kubernetes components to create a single-node cluster, or a multi-node cluster.

Once a KinD cluster is running, you can use Docker to exec into a control plane node container and look at the process list. In the process list, you will see the standard Kubernetes components for the control plane nodes running:

Figure 2.1 – Host process list showing control plane components

If you were to exec into a worker node to check the components, you would see all the standard worker node components:

Figure 2.2 – Host process list showing worker components

We will cover the standard Kubernetes components in Chapter 3, Kubernetes Bootcamp, including  kube-apiserver ,  kubelets ,  kube-proxy ,  kube-scheduler , and  kube-controller-manager .

In addition to standard Kubernetes components, both KinD nodes have an additional component that is not part of most standard installations: Kindnet. Kindnet is the included, default CNI when you install a base KinD cluster. While Kindnet is the default CNI, you have the option to disable it and use an alternative, such as Calico.

Now that you have seen each node and the Kubernetes components, let's take a look at what's included with a base KinD cluster. To show the complete cluster and all the components that are running, we can run the  kubectl get pods --all-namespaces  command. This will list all the running components for the cluster, including the base components we will discuss in Chapter 3, Kubernetes Bootcamp. In addition to the base cluster components, you may notice a running pod in a namespace called  local-path-storage , along with a pod named  local-path-provisioner . This pod is running one of the add-ons that KinD includes, providing the cluster with the ability to auto-provision  PersistentVolumeClaims :

Figure 2.3 – kubectl get pods showing local-path-provisioner

Most development cluster offerings provide similar, common functions that people need to test deployments on Kubernetes. They all provide a Kubernetes control plane and worker nodes, and most include a default CNI for networking. Few offerings go beyond this base functionality, and as Kubernetes workloads mature, you may find the need for additional plugins such as  local-path-provisioner . We will leverage this component heavily in some of the exercises in this book because without it, we will have a tougher time creating some of the procedures.

Why should you care about persistent volumes in your development cluster? Most production clusters running Kubernetes will provide persistent storage to developers. Usually, the storage will be backed by storage systems based on block storage, S3, or NFS. Aside from NFS, most home labs rarely have the resources to run a full-featured storage system.  local-path-provisioner  removes this limitation from users by providing all the functions to your KinD cluster that an expensive storage solution would provide.

In Chapter 3, Kubernetes Bootcamp, we will discuss a few API objects that are part of Kubernetes storage. We will discuss the  CSIdrivers ,  CSInodes , and  StorageClass  objects. These objects are used by the cluster to provide access to the backend storage system. Once installed and configured, pods consume the storage using the  PersistentVolumes  and  PersistentVolumeClaims  objects. Storage objects are important to understand, but when they were first released, they were difficult for most people to test since they aren't included in most Kubernetes development offerings.

KinD recognized this limitation and chose to bundle a project from Rancher called  local-path-provisioner , which is based on the Kubernetes local persistent volumes that were introduced in Kubernetes 1.10.

You may be wondering why anyone would need an add-on since Kubernetes has native support for local host persistent volumes. While support may have been added for local persistent storage, Kubernetes has not added auto-provisioning capabilities. CNCF does offer an auto-provisioner, but it must be installed and configured as a separate Kubernetes component. KinD makes it easy to auto-provision since the provisioner is included in all base installations.

Rancher's project provides the following to KinD:

- Auto-creation of  PersistentVolumes  when a PVC request is created

- A default  StorageClass  named standard

When the auto-provisioner sees a  PersistentVolumeClaim  request hit the API server, a  PersistentVolume  will be created and the pod's PVC will be bound to the newly created PVC.

local-path-provisioner  adds a feature to KinD clusters that greatly expands the potential test scenarios that you can run. Without the ability to auto-provision persistent disks, it would be a challenge to test many pre-built deployments that require persistent disks.

With the help of Rancher, KinD provides you with a solution so that you can experiment with dynamic volumes, storage classes, and other storage tests that would otherwise be impossible to run outside a data center. We will use the provisioner in multiple chapters to provide volumes to different deployments. We will point these out to reinforce the advantages of using auto-provisioning.

Understanding the node image

The node image is what provides KinD the magic to run Kubernetes inside a Docker container. This is an impressive accomplishment since Docker relies on a  systemd  running system and other components that are not included in most container images.

KinD starts off with a base image, which is an image the team has developed that contains everything required for Docker, Kubernetes, and  systemd . Since the base image is based on an Ubuntu image, the team removes services that are not required and configures  systemd  for Docker. Finally, the node image is created using the base image.

If you want to know the details of how the base image is created, you can look at the Dockerfile in the KinD team's GitHub repository at https://github.com/kubernetes-sigs/kind/blob/controlplane/images/base/Dockerfile.

KinD and Docker networking

Since KinD uses Docker as the container engine to run the cluster nodes, all clusters are limited to the same network constraints that a standard Docker container is limited to. These limitations do not limit testing your KinD Kubernetes cluster from the local host, but they can lead to issues when you want to test containers from other machines on your network.

When you install KinD, a new Docker bridge network will be created, called kind. This network configuration was introduced in KinD v0.8.0 which resolved multiple issues from previous versions that used the default Docker bridge network. Most users will not notice this change, but it’s important to know this, as you start to create more advanced KinD clusters with additional containers that you may need to run on the same network with KinD. If you have a requirement to run additional containers on the KinD network, you will need to add the --net=kind to your Docker run command.

Along with the Docker networking considerations, we must consider the Kubernetes Container Network Interface (CNI) as well. Officially, the KinD team has limited the networking options to only two CNIs: Kindnet and Calico. Kindnet is the only CNI they will support but you do have the option to disable the default Kindnet installation, which will create a cluster without a CNI installed. After the cluster has been deployed, you can deploy a CNI manifest such as Calico.

Many Kubernetes installations for both small development clusters and enterprise clusters use Tigera's Calico for the CNI and as such, we have elected to use Calico as our CNI for the exercises in this book.

Keeping track of the nesting dolls

Running a solution such as KinD can get confusing due to the container-in-a-container deployment. We compare this to Russian nesting dolls, where one doll fits into another, then another, and so on. As you start to play with KinD for your own cluster, you may lose track of the communication paths between your host, Docker, and the Kubernetes nodes. To keep your sanity, you should have a solid understanding of where each component is running and how you can interact with each one.

The following diagram shows the three layers that must be running to form a KinD cluster. It's important to note that each layer can only interact with the layer directly above it. This means that the KinD container in layer 3 can only see the Docker image running in layer 2, and the Docker image can see the Linux host running in layer 1. If you wanted to communicate directly from the host to a container running in your KinD cluster, you would need to go through the Docker layer, and then to the Kubernetes container in layer 3.

This is important to understand so that you can use KinD effectively as a testing environment:

Figure 2.4 – Host cannot communicate with KinD directly

As an example, consider that you want to deploy a web server to your Kubernetes cluster. You deploy an Ingress controller in the KinD cluster and you want to test the site using Chrome on your Docker host or a different workstation on the network. You attempt to target the host on port 80 and receive a failure in the browser. Why would this fail?

The pod running the web server is in layer 3 and cannot receive direct traffic from the host or machines on the network. In order to access the web server from your host, you will need to forward the traffic from the Docker layer to the KinD layer. Remember that in Chapter 3, Understanding Docker Networking, we explained how to expose a container to the network by adding a listening port to the container. In our case, we need port 80 and port 443. When a container is started with a port, the Docker daemon will forward the incoming traffic from the host to the running Docker container:

Figure 2.5 – Host communicates with KinD via an Ingress controller

With ports 80 and 443 exposed on the Docker container, the Docker daemon will now accept incoming requests for 80 and 443 and the NGINX Ingress controller will receive the traffic. This works because we have exposed ports 80 and 443 in two places on the Docker layer. We have exposed it in the Kubernetes layer by running our NGINX container using host ports 80 and 443. This installation process will be explained later in this chapter, but for now, you just need to understand the basic flow.

On the host, you make a request for a web server that has an Ingress rule in your Kubernetes cluster:

1. The request looks at the IP address that was requested (in this case, the local IP address).

2. The Docker container running our Kubernetes node is listening on the IP address for ports 80 and 443, so the request is accepted and sent to the running container.

3. The NGINX pod in your Kubernetes cluster has been configured to use the host ports 80 and 443, so the traffic is forwarded to the pod.

4. The user receives the requested web page from the web server via the NGINX Ingress controller.

This is a little confusing, but the more you use KinD and interact with it, the easier this becomes.

To use a KinD cluster for your development requirements, you need to understand how KinD works. So far, you have learned about the node image and how the image is used to create a cluster. You've also learned how KinD network traffic flows between the Docker host and the containers running the cluster. With this base knowledge, we will move on to creating a Kubernetes cluster using KinD.

Installing KinD

The files for this chapter are located in the KinD directory. You can use the provided files, or you can create your own files from this chapter's content. We will explain each step of the installation process in this section.

Note

At the time of writing, the current version of KinD is .0.11.0, supporting Kubernetes clusters up to 1.21.1

Installing KinD – prerequisites

KinD can be installed using a few different methods, but the easiest and quickest way to start building KinD clusters is to download the KinD binary and the standard Kubernetes kubectl executable to interact with the cluster.

Installing Kubectl

Since KinD is a single executable, it does not install  kubectl . If you do not have  kubectl  installed and you are using an Ubuntu 20.04 system, you can install it by running a snap install, or you may download it from Google directly.

To install kubectl using snap, you only need to run a single command:

sudo snap install kubectl --classic

To install kubectl from Google, you need to download the executable, give it the execute permission, and move it to a location in your systems path variable. This can be completed using the steps outlined below.

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

Now that you have kubectl, we can move on to downloading the KinD executable.

Installing the KinD binary

Installing KinD is an easy process; it can be done with a single command. You can install KinD by running the included script in this book's repository, located at  /chapter2/install-kind.sh . Alternatively, you can use the steps below to install it manually.

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-linux-amd64

chmod +x ./kind

sudo mv ./kind /usr/bin

Once installed, you can verify that KinD has been installed correctly by typing  kind version  into the prompt:

kind version

This will return the installed version:

kind v0.11.1 go1.16.4 linux/amd64

The KinD executable provides every option you will need to maintain a cluster's life cycle. Of course, the KinD executable can create and delete clusters, but it also provides the following capabilites:

- The ability to create custom build base and node images

- Can export  kubeconfig  or log files

- Can retrieve clusters, nodes, or  kubeconfig  files

- Can load images into nodes

Now that you have installed the KinD utility, you are almost ready to create your KinD cluster. Before we execute a few  create cluster  commands, we will explain some of the creation options that KinD provides.

Creating a KinD cluster

Now that you have met all the requirements, you can create your first cluster using the KinD executable. The KinD utility can create a single-node cluster, as well as a complex cluster that's running multiple nodes for the control plane with multiple worker nodes. In this section, we will discuss the KinD executable options. By the end of the chapter, you will have a two-node cluster running – a single control plane node and a single worker node.

Important note

For the exercises in this book, we will install a multi-node cluster. The simple cluster configuration is an example and should not be used for our exercises.

Creating a simple cluster

To create a simple cluster that runs the control plane and a worker node in a single container, you only need to execute the KinD executable with the  create cluster  option.

Let's create a quick single-node cluster to see how quickly KinD creates a fast development cluster. On your host, create a cluster using the following command:

kind create cluster

This will quickly create a cluster with all the Kubernetes components in a single Docker container by using a cluster name of  kind . It will also assign the Docker container a name of  kind-control-plane . If you want to assign a cluster name, rather than the default name, you need to add the  --name <cluster name>  option to the  create cluster  command:

Creating cluster "kind" ...

✓

Ensuring

node

image

(kindest/

node:v

1.21.1)

🖼

✓

Preparing

nodes

📦

✓

Writing

configuration

📜

✓

Starting

control-plane

🕹

️

✓

Installing

CNI

🔌

✓

Installing

StorageClass

💾

Set kubectl context to "kind-kind"

You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/

The  create  command will create the cluster and modify the kubectl  config  file. KinD will add the new cluster to your current kubectl  config  file, and it will set the new cluster as the default context.

We can verify that the cluster was created successfully by listing the nodes using the kubectl utility:

kubectl get nodes

This will return the running nodes, which, for a basic cluster, are single nodes:

NAME               STATUS   ROLES                  AGE  VERSION

kind-control-plane Ready    control-plane,master   32m  v1.21.1

The main point of deploying this single-node cluster was to show you how quickly KinD can create a cluster that you can use for testing. For our exercises, we want to split up the control plane and worker node so that we can delete this cluster using the steps in the next section.

Deleting a cluster

When you have finished testing, you can delete the cluster using the  delete  command:

kind delete cluster –name <cluster name>

The  delete  command will quickly delete the cluster, including any entries in your  kubeconfig  file.

A quick single-node cluster is useful for many use cases, but you may want to create a multi-node cluster for various testing scenarios. Creating a more complex cluster requires that you create a config file.

Creating a cluster config file

When creating a multi-node cluster, such as a two-node cluster with custom options, we need to create a cluster config file. The config file is a YAML file and the format should look familiar. Setting values in this file allows you to customize the KinD cluster, including the number of nodes, API options, and more. The config file we'll use to create the cluster for the book is shown here – it is included in this book's repository at  /chapter4/cluster01-kind.yaml :

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

networking:

apiServerAddress: "0.0.0.0"

disableDefaultCNI: true

apiServerPort: 6443

kubeadmConfigPatches:

- |

apiVersion: kubeadm.k8s.io/v1beta2

kind: ClusterConfiguration

metadata:

name: config

networking:

serviceSubnet: "10.96.0.1/12"

podSubnet: "10.240.0.0/16"

nodes:

- role: control-plane

extraMounts:

- hostPath: /dev

containerPath: /dev

- hostPath: /var/run/docker.sock

containerPath: /var/run/docker.sock

- role: worker

extraPortMappings:

- containerPort: 80

hostPort: 80

- containerPort: 443

hostPort: 443

- containerPort: 2222

hostPort: 2222

extraMounts:

- hostPath: /dev

containerPath: /dev

- hostPath: /var/run/docker.sock

containerPath: /var/run/docker.sock

Details about each of the custom options in the file are provided in the following table:

Config Options

Option Details

apiServerAddress

This configuration option tells the installation what IP address the API server will listen on. By default it will use 127.0.0.1, but since we plan to use the cluster from other networked machines, we have selected to listen on all IP addresses.

disableDefaultCNI

This setting is used to enable or disable the Kindnet installation. The default value is false, but since we want to use Calico as our CNI, we need to set it to true.

kubeadmConfigPatches

This section allows you to set values for other cluster options during the installation. For our configuration, we are setting the CIDR ranges for the for the  ServiceSubnet  and the  podSubnet  .

Nodes

This section is where you define the nodes for the cluster. For our cluster, we will create a single control plane node, and a single worker node.

- role: control-plane

The role section allows you to set options for nodes. The first role section is for the control-plane. We have added options to map the local hosts  /dev  and  /var/run/docker.sock  , which will be used in the Falco chapter, later in the book.

- role: worker

This is the second node section, which allows you to configure options that the worker nodes will use. For our cluster, we have added the same local mounts that will be used for Falco, and we have also added additional ports to expose for our Ingress controller.

extraPortMappings

To expose ports to your  KinD  nodes, you need to add them to the  exrtaPortMappings  section of the configuration. Each mapping has two values, the container port, and the host port. The host port is the port you would use to target the cluster, while the container port is the port that the container is listening on.

extraMounts

The  extraMounts  section allows you to add extra mount points to the containers. This comes in handy to expose mounts like  /dev  and  /var/run/docker.sock  that we will need for the Falco chapter.

Table 4.3 – KinD configuration options

If you plan to create a cluster that goes beyond a single-node cluster without using advanced options, you will need to create a configuration file. Understanding the options available to you will allow you to create a Kubernetes cluster that has advanced components such as Ingress controllers or multiple nodes to test failure and recovery procedures for deployments.

Now that you know how to create a simple all-in-one container for running a cluster and how to create a multi-node cluster using a config file, let's discuss a more complex cluster example.

Multi-node cluster configuration

If you only wanted a multi-node cluster without any extra options, you could create a simple configuration file that lists the number and node types you want in the cluster. The following  config  file will create a cluster with three control plane nodes and three worker nodes:

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

- role: control-plane

- role: control-plane

- role: worker

- role: worker

- role: worker

Using multiple control plane servers introduces additional complexity since we can only target a single host or IP in our configuration files. To make this configuration usable, we need to deploy a load balancer in front of our cluster.

KinD has considered this, and if you do deploy multiple control plane nodes, the installation will create an additional container running a HAProxy load balancer. During the creation of a multi-node clustrer, you will see a few additional lines regarding configuring an extra load balancer, joining additional control-plane node and joining extra worker nodes – as shown below:

Creating cluster "multinode" ...

✓

Ensuring

node

image

(kindest/

node:v

1.21.1)

🖼

✓

Preparing

nodes

📦

📦

📦

📦

📦

📦

✓

Configuring

the

external

load

balancer

⚖️

✓

Writing

configuration

📜

✓

Starting

control-plane

🕹

️

✓

Installing

StorageClass

💾

✓

Joining

more

control-plane

nodes

🎮

✓

Joining

worker

nodes

🚜

Set kubectl context to "kind-multinode"

You can now use your cluster with:

kubectl cluster-info --context kind-multinode

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

If we look at the running containers from a multi-node config, we will see six node containers running and a HAProxy container:

Container ID

Image

Port

Names

d9107c31eedb

kindest/haproxy:v20200708-548e36db

0.0.0.0:6443

multinode-external-load-balancer

03a113144845

kindest/node:v1.21.1

127.0.0.1:44445->6443/tcp

multinode-control-plane3

9b078ecd69b7

kindest/node:v1.21.1

multinode-worker2

b779fa15206a

kindest/node:v1.21.1

multinode-worker

8171baafac56

kindest/node:v1.21.1

127.0.0.1:42673->6443/tcp

multinode-control-plane

3ede5e163eb0

kindest/node:v1.21.1

127.0.0.1:43547->6443/tcp

multinode-control-plane2

6a85afc27cfe

kindest/node:v1.21.1

multinode-worker3

Table 4.4 – KinD configuration options

Since we have a single host, each control plane node and the HAProxy container are running on unique ports. Each container needs to be exposed to the host so that they can receive incoming requests. In this example, the important one to note is the port assigned to HAProxy, since that's the target port for the cluster. If you were to look at the Kubernetes config file, you would see that it is targeting  https://127.0.0.1:42673 , which is the port that's been assigned to the HAProxy container.

When a command is executed using  kubectl , it is sent to directly to the HAProxy server. Using a configuration file that was created by KinD during the cluster's creation, the HAProxy container knows how to route traffic between the three control plane nodes. In the HAproxy container, we can verify the configuration by viewing the config file, found at  /usr/local/etc/haproxy/haproxy.cfg .

# generated by kind

global

log /dev/log local0

log /dev/log local1 notice

daemon

resolvers docker

nameserver dns 127.0.0.11:53

defaults

log global

mode tcp

option dontlognull

# TODO: tune these

timeout connect 5000

timeout client 50000

timeout server 50000

# allow to boot despite dns don't resolve backends

default-server init-addr none

frontend control-plane

bind *:6443

default_backend kube-apiservers

backend kube-apiservers

option httpchk GET /healthz

# TODO: we should be verifying (!)

server multinode-control-plane multinode-control-plane:6443 check check-ssl verify none resolvers docker resolve-prefer ipv4

server multinode-control-plane2 multinode-control-plane2:6443 check check-ssl verify none resolvers docker resolve-prefer ipv4

server multinode-control-plane3 multinode-control-plane3:6443 check check-ssl verify none resolvers docker resolve-prefer ipv4

As shown in the preceding configuration file, there is a backend section called  kube-apiservers  that contains the three control plane containers. Each entry contains the Docker IP address of a control plane node with a port assignment of 6443, targeting the API server running in the container. When you request https://127.0.0.1:32791, that request will hit the HAProxy container, then, using the rules in the HAProxy configuration file, the request will be routed to one of the three nodes in the list.

Since our cluster is now fronted by a load balancer, we have a highly available control plane for testing.

Note

The included HAProxy image is not configurable. It is only provided to handle the control plane and to load balance the API servers. Due to this limitation, if you needed to use a load balancer for the worker nodes, you will need to provide your own.

An example use case for this would be if you wanted to use an Ingress controller on multiple worker nodes. You would need a load balancer in front of the worker nodes to accept incoming 80 and 443 requests that would forward the traffic to each node running NGINX. At the end of this chapter, we have provided an example configuration that includes a custom HAProxy configuration for load balancing traffic to the worker nodes.

Customizing the control plane and Kubelet options

You may want to go further than this to test features such as OIDC integration or Kubernetes feature gates. KinD uses the same configuration that you would use for a kubeadm installation. As an example, if you wanted to integrate a cluster with an OIDC provider, you could add the required options to the configuration patch section:

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

kubeadmConfigPatches:

- |

kind: ClusterConfiguration

metadata:

name: config

apiServer:

extraArgs:

oidc-issuer-url: "https://oidc.testdomain.com/auth/idp/k8sIdp"

oidc-client-id: "kubernetes"

oidc-username-claim: sub

oidc-client-id: kubernetes

oidc-ca-file: /etc/oidc/ca.crt

nodes:

- role: control-plane

- role: control-plane

- role: control-plane

- role: worker

- role: worker

- role: worker

For a list of available configuration options, take a look at Customizing control plane configuration with kubeadm on the Kubernetes site at https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/.

Now that you have created the cluster file, you can create your KinD cluster.

Creating a custom KinD cluster

Finally! Now that you are familiar with KinD, we can move forward and create our cluster.

We need to create a controlled, known environment, so we will give the cluster a name and provide the config file that we discussed in the previous section.

Make sure that you are in your cloned repository under the  chapter2  directory. If you want to create the entire cluster using our supplied script, you can simply execute the create-cluster.sh script in the  chapter2  directory.

If you want to create a KinD cluster interactively, with our required options, you need to run the KinD installer with the following config file, which is found in the  chapter2  directory:

kind create cluster --name cluster01 --config cluster01-kind.yaml

The option  --name  will set the name of the cluster to cluster01 and the  --config  tells the installer to use the config file  cluster01-kind.yaml .

When you execute the installer on your host, KinD will start the installation and tell you each step that is being performed. The entire cluster creation process should take less than 2 minutes.

Creating cluster "multinode" ...

✓

Ensuring

node

image

(kindest/

node:v

1.21.1)

🖼

✓

Preparing

nodes

📦

📦

📦

📦

📦

📦

✓

Writing

configuration

📜

✓

Starting

control-plane

🕹

️

✓

Installing

StorageClass

💾

✓

Joining

worker

nodes

🚜

Set kubectl context to "kind-multinode"

You can now use your cluster with:

kubectl cluster-info --context kind-multinode

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

The final step in the deployment creates or edits an existing Kubernetes config file. In either case, the installer creates a new context with the name  kind-<cluster name>  and sets it as the default context.

While it may appear that the cluster installation procedure has completed its tasks, the cluster is not ready yet. Some of the tasks take a few minutes to fully initialize and since we disabled the default CNI to use Calico, we still need to deploy Calico to provide cluster networking.

Installing Calico

To provide networking to the pods in the cluster, we need to install a Container Network Interface, or CNI. We have elected to install Calico as our CNI and since KinD only includes the Kindnet CNI, we need to install Calico manually.

If you were to pause after the creation step and look at the cluster, you would notice that some pods are in a pending state:

coredns-6955765f44-86l77  0/1  Pending  0  10m

coredns-6955765f44-bznjl  0/1  Pending  0  10m

local-path-provisioner-7  0/1  Pending  0  11m 745554f7f-jgmxv

The pods listed here require a working CNI to start. This puts the pods into a pending state, where they are waiting for a network. Since we did not deploy the default CNI, our cluster does not have networking support. To get these pods from pending to running, we need to install a CNI – and for our cluster, that will be Calico.

To install Calico, we will use the standard Calico operator deployment, requiring only two manifest. To deploy the Calico operator, use the following commands:

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml

Then deploy the custom resources. Since we are setting the CIDR range to a custom range, we have included a custom resource manifest in the repository, in the  chapter2  directory.

kubectl create -f calico.yaml

As it deploys, you will see that that a number of Kubernetes objects are created.

The installation process will take about a minute and you can check on its status using  kubectl get pods -n kube-system . You will see that three Calico pods were created. Two are  calico-node  pods, while the other is the  calico-kube-controller  pod:

NAME                    READY STATUS RESTARTS AGE

calico-kube-controllers  1/1  Running    0    64s -5b644bc49c-nm5wn

calico-node-4dqnv        1/1  Running    0    64s

calico-node-vwbpf        1/1  Running    0    64s

If you check the two CoreDNS pods in the  kube-system  namespace again, you will notice that they have changed from the pending state, from before we installed Calico, to being in a running state:

coredns-6955765f44-86l77   1/1  Running   0  18m

coredns-6955765f44-bznjl   1/1  Running   0  18m

Now that the cluster has a working CNI installed, any pods that were dependent on networking will be in a running state.

Installing an Ingress controller

We have a chapter dedicated to Ingress to explain all the technical details. Since we are deploying a cluster and we require Ingress for future chapters, we need to deploy an Ingress controller to show a complete cluster build. All these details will be explained in more detail in Chapter 4, Services, Load Balancing, and External DNS.

Installing the NGINX Ingress controller requires only two manifests, which we will pull from the internet to make the installation easy. To install the controller, execute the following command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/kind/deploy.yaml

The deployment will create a few Kubernetes objects that are required for Ingress in a namespace called  ingress-nginx :

namespace/ingress-nginx created

serviceaccount/ingress-nginx created

configmap/ingress-nginx-controller created

clusterrole.rbac.authorization.k8s.io/ingress-nginx created

clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created

role.rbac.authorization.k8s.io/ingress-nginx created

rolebinding.rbac.authorization.k8s.io/ingress-nginx created

service/ingress-nginx-controller-admission created

service/ingress-nginx-controller created

deployment.apps/ingress-nginx-controller created

validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created

serviceaccount/ingress-nginx-admission created

clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created

clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created

role.rbac.authorization.k8s.io/ingress-nginx-admission created

rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created

job.batch/ingress-nginx-admission-create created

job.batch/ingress-nginx-admission-patch created

Since this manifest has been created for KinD, you do not need to edit or patch the deployment for any reason, it comes preconfigured to integrate with KinD by default.

Congratulations! You now have a fully functioning, two-node Kubernetes cluster running Calico with an Ingress controller.

Reviewing your KinD cluster

With a Kubernetes cluster now available, we have the ability to look at Kubernetes objects first-hand. This will help you understand the previous chapter, where we covered many of the base objects included in a Kubernetes cluster. In particular, we will discuss the storage objects that are included with your KinD cluster.

KinD storage objects

Remember that KinD includes Rancher's auto-provisioner to provide automated persistent disk management for the cluster. In Chapter 3, Kubernetes Bootcamp, we went over the storage-related objects, and now that we have a cluster with a storage system configured, we can explain them in greater detail.

There is one object that the auto-provisioner does not require since it uses a base Kubernetes feature: it does not require a  CSIdriver . Since the ability to use local host paths as PVCs is part of Kubernetes, we will not see any  CSIdriver  objects in our KinD cluster.

The first objects in our KinD cluster we will discuss are our  CSInodes . In the bootcamp, we mentioned that this object was created to decouple any CSI objects from the base node object. Any node that can run a workload will have a  CSInode  object. On our KinD clusters, both nodes have a  CSInode  object. You can verify this by executing  kubectl get csinodes :

NAME                      DRIVERS   AGE

cluster01-control-plane   0         20m

cluster01-worker          0         20m

If we were to describe one of the nodes using  kubectl describe csinodes <node name> , you would see the details of the object:

Name:               cluster01-worker

Labels:             <none>

Annotations:           storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/cinder

CreationTimestamp:  Sun, 27 Jun 2021 00:12:03 +0000

Spec:

Events:  <none>

The main thing to point out is the  Spec  section of the output. This lists the details of any drivers that may be installed to support backend storage systems. Since we do not have a backend storage system, we do not require an additional driver on our cluster.

Storage drivers

As we already mentioned, your KinD cluster does not have any additional storage drivers installed. If you execute  kubectl get csidrivers , the API will not list any resources.

KinD storage classes

To attach to any cluster-provided storage, the cluster requires a  StorageClass  object. Rancher's provider creates a default storage class called standard. It also sets the class as the default  StorageClass , so you do not need to provide a  StorageClass  name in your PVC requests. If a default  StorageClass  is not set, every PVC request will require a  StorageClass  name in the request. If a default class is not enabled and a PVC request fails to set a  StorageClass  name, the PVC allocation will fail since the API server won't be able to link the request to a  StorageClass .

Note

On a production cluster, it is considered a good practice to omit assigning a default  StorageClass . Depending on your users, you may have deployments that forget to set a class, and the default storage system may not fit the deployment needs. This issue may not occur until it becomes a production issue, and that may impact business revenue or the company's reputation. If you don't assign a default class, the developer will have a failed PVC request, and the issue will be discovered before any harm comes to the business.

To list the storage classes on the cluster, execute  kubectl get storageclasses , or use the shortened version by using  sc  instead of  storageclasses :

NAME                  PROVISIONER    RECLAIMPOLICY     VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION

standard (default)   rancher.io/local-path      Delete            WaitForFirstConsumer   false

Next, let's learn how to use the provisioner.

Using KinD's storage provisioner

Using the included provisioner is very simple. Since it can auto-provision the storage and is set as the default class, any PVC requests that are coming in are seen by the provisioning pod, which then creates  PersistentVolume  and  PersistentVolumeClaim .

To show this process, let's go through the necessary steps. The following is the output of running  get pv  and  get pvc  on a base KinD cluster:

kubectl get pv

No resources found

Remember that  PersistentVolume  is not a namespaced object, so we don't need to add a namespace option to the command. PVCs are namespaced objects, so I told Kubernetes to show me the PVCs that are available in all the namespaces. Since this is a new cluster and none of the default workloads require persistent disk, there are no PV or PVC objects.

Without an auto-provisioner, we would need to create a PV before a PVC could claim the volume. Since we have the Rancher provisioner running in our cluster, we can test the creation process by deploying a pod with a PVC request like the one listed here:

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

name: test-claim

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 1Mi

---

kind: Pod

apiVersion: v1

metadata:

name: test-pvc-claim

spec:

containers:

- name: test-pod

image: busybox

command:

- "/bin/sh"

args:

- "-c"

- "touch /mnt/test && exit 0 || exit 1"

volumeMounts:

- name: test-pvc

mountPath: "/mnt"

restartPolicy: "Never"

volumes:

- name: test-pvc

persistentVolumeClaim:

claimName: test-claim

This PVC request will be named  test-claim  in the default namespace and it is requesting a 1 MB volume. We do need to include the  StorageClass  option since KinD has set a default  StorageClass  for the cluster.

To create the PVC, we can execute a  create  command using kubectl, such as  kubectl create -f pvctest.yaml  – Kubernetes will return, stating that the PVC has been created, but it's important to note that this does not mean that the PVC is fully working. The PVC object has been created, but if any dependencies are missing in the PVC request, it will still create the object, though it will fail to fully create the PVC request.

After creating a PVC, you can check the real status using one of two options. The first is a simple  get  command; that is,  kubectl get pvc . Since my request is in the default namespace, I don't need to include a namespace value in the  get  command (note that we had to shorten the volume's name so that it fits the page):

NAME       STATUS   VOLUME         CAPACITY   ACCESS MODES STORAGECLASS   AGE

test-claim Bound    pvc-b6ecf50…   1Mi        RWO          standard   15s

We know that we created a PVC request in the manifest, but we did not create a PV request. If we look at the PVs now, we will see that a single PV was created from our PVC request. Again, we shortened the PV name in order to fit the output on a single line:

NAME        CAPACITY  ACCESS MODES RECLAIM POLICY STATUS   CLAIM            pvc-b6ecf…  1Mi       RWO          Delete         Bound    default/test-claim

With so many workloads requiring persistent disks, it is very important to understand how Kubernetes workloads integrate with storage systems. In this section, you learned how KinD adds the auto-provisioner to the cluster. We will reinforce our knowledge of these Kubernetes storage objects in the next chapter, Chapter 3, Kubernetes Bootcamp.

Adding a custom load balancer for Ingress

Note

This section is a complex topic that covers adding a custom HAProxy container that you can use to load balance worker nodes in a KinD cluster. You should not deploy these steps on the KinD cluster that we will use for the remaining chapters.

We added this section for anybody that may want to know more about how to load balance between multiple worker nodes.

KinD does not include a load balancer for worker nodes. The included HAProxy container only creates a configuration file for the API server; the team does not officially support any modifications to the default image or configuration. Since you will interact with load balancers in your everyday work, we wanted to add a section on how to configure your own HAProxy container in order to load balance between three KinD nodes.

First, we will not use this configuration for any of chapters in this book. We want to make the exercises available to everyone, so to limit the required resources, we will always use the two-node cluster that we created earlier in this chapter. If you want to test KinD nodes with a load balancer, we suggest using a different Docker host or waiting until you have finished this book and deleting your KinD cluster.

Installation prerequisites

We assume that you have a KinD cluster based on the following configuration:

- Any number of control plane nodes

- Three worker nodes

- Cluster name is  cluster01

- A working version of Kindnet or Calico (CNI)

- NGINX Ingress controller installed

Creating the KinD cluster configuration

Since you will use an HAProxy container exposed on ports 80 and 443 on your Docker host, you do not need to expose any ports in your cluster  config  file.

To make a test deployment easier, you can use the example cluster config shown here, which will create a simple six-node cluster with Kindnet disabled:

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

networking:

apiServerAddress: "0.0.0.0"

disableDefaultCNI: true

kubeadmConfigPatches:

- |

apiVersion: kubeadm.k8s.io/v1beta2

kind: ClusterConfiguration

metadata:

name: config

networking:

serviceSubnet: "10.96.0.1/12"

podSubnet: "192.168.0.0/16"

nodes:

- role: control-plane

- role: control-plane

- role: control-plane

- role: worker

- role: worker

- role: worker

You need to install Calico using the same manifest that we used earlier in this chapter. After installing Calico, you need to install the NGINX Ingress controller using the steps provided earlier in this chapter.

Once you've deployed Calico and NGINX, you should have a working base cluster. Now, you can move on to deploying a custom HAProxy container.

Deploying a custom HAProxy container

HAProxy offers a container on Docker Hub that is easy to deploy, requiring only a config file to start the container.

To create the configuration file, you will need you to know the IP addresses of each worker node in the cluster. In this book's GitHub repository, we have included a script file that will find this information for you, create the config file, and start the HAProxy container. It is located under the  HAProxy  directory and it's called  HAProxy-ingress.sh .

To help you better understand this script, we will break out sections of the script and detail what each section is executing. Firstly, the following code block is getting the IP addresses of each worker node in our cluster and saving the results in a variable. We will need this information for the backend server list:

#!/bin/bash

worker1=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' cluster01-worker)

worker2=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' cluster01-worker2)

worker3=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' cluster01-worker3)

Next, since we will use a bind mount when we start the container, we need to have the configuration file in a known location. We elected to store it in the current user's home folder, under a directory called  HAProxy :

# Create an HAProxy directory in the current users home folder

mkdir ~/HAProxy

Next, the following part of the script will create the  HAProxy  directory:

# Create the HAProxy.cfg file for the worker nodes

tee ~/HAProxy/HAProxy.cfg <<EOF

The  global  section of the configuration sets process-wide security and performance settings:

global

log /dev/log local0

log /dev/log local1 notice

daemon

The  defaults  section is used to configure values that will apply to all frontend and backend sections in the configuration value:

defaults

log global

mode tcp

timeout connect 5000

timeout client 50000

timeout server 50000

frontend workers_https

bind *:443

mode tcp

use_backend ingress_https

backend ingress_https

option httpchk GET /healthz

mode tcp

server worker $worker1:443 check port 80

server worker2 $worker2:443 check port 80

server worker3 $worker3:443 check port 80

This tells HAProxy to create a frontend called  workers_https  and the IP addresses and ports to bind for incoming requests, to use TCP mode, and to use a backend named  ingress_https .

The  ingress_https  backend includes the three worker nodes that are using port 443 as a destination. The check port is a health check that will test port 80. If a server replies on port 80, it will be added as a target for requests. While this is an HTTPS port 443 rule, we are only using port 80 to check for a network reply from the NGINX pod:

frontend workers_http

bind *:80

use_backend ingress_http

backend ingress_http

mode http

option httpchk GET /healthz

server worker $worker1:80 check port 80

server worker2 $worker2:80 check port 80

server worker3 $worker3:80 check port 80

This  frontend  section creates a frontend that accepts incoming HTTP traffic on port 80. It then uses the list of servers in the backend, named  ingress_http , for endpoints. Just like in the HTTPS section, we are using port 80 to check for any nodes that are running a service on port 80. Any endpoint that replies to the check will be added as a destination for HTTP traffic, and any nodes that do not have NGINX running on them will not reply, which means they won't be added as destinations:

EOF

This ends the creation of our file. The final file will be created in the  HAProxy  directory:

# Start the HAProxy Container for the Worker Nodes

docker run --name HAProxy-workers-lb -d -p 80:80 -p 443:443 -v ~/HAProxy:/usr/local/etc/HAProxy:ro HAProxy -f /usr/local/etc/HAProxy/HAProxy.cfg

The final step is to start a Docker container running HAProxy with our created configuration file containing the three worker nodes, exposed on the Docker host on ports 80 and 443.

Now that you have learned how to install a custom HAProxy load balancer for your worker nodes, let's look at how the configuration works.

Understanding HAProxy traffic flow

The cluster will have a total of eight containers running. Six of these containers will be the standard Kubernetes components; that is, three control plane servers and three worker nodes. The other two containers are KinD's HAProxy server, and your own custom HAProxy container:

Figure 2.6 – Custom HAProxy container running

There are a few differences between this cluster output versus our two-node cluster for the exercises. Notice that the worker nodes are not exposed on any host ports. The worker nodes do not need any mappings since we have our new HAProxy server running. If you look at the HAProxy container we created, it is exposed on host ports 80 and 443. This means that any incoming requests to the host on port 80 or 443 will be directed to the custom HAProxy container.

The default NGINX deployment only has a single replica, which means that the Ingress controller is running on a single node. If we look at the logs for the HAProxy container, we will see something interesting:

[NOTICE] 093/191701 (1) : New worker #1 (6) forked

[WARNING] 093/191701 (6) : Server ingress_https/worker is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure (Connection refused)", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

[WARNING] 093/191702 (6) : Server ingress_https/worker3 is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure (Connection refused)", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

[WARNING] 093/191702 (6) : Server ingress_http/worker is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

[WARNING] 093/191703 (6) : Server ingress_http/worker3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

You may have noticed a few errors in the log, such as SSL handshake failure and  Connection refused . While these do look like errors, they are actually failed checked events on the worker nodes. Remember that NGINX is only running in a single pod, and since we have all three nodes in our HAProxy backend configuration, it will check for the ports on each node. Any nodes that fail to reply will not be used to load balance traffic. In our current config, this does load balance, since we only have NGINX on one node. It does, however, provide high availability to the Ingress controller.

If you look carefully at the log output, you will see how many servers are active on a defined backend; for example:

check duration: 0ms. 1 active and 0 backup servers left.

Each server pool in the log output shows 1 active endpoint, so we know that the HAProxy has successfully found a NGINX controller on both port 80 and 443.

To find out what worker the HAProxy server has connected to, we can use the failed connections in the log. Each backend will list the failed connections. For example, we know that the node that is working is  cluster01-worker2  based on the logs that the other two worker nodes show as  DOWN :

Server ingress_https/worker is DOWN Server ingress_https/worker3 is DOWN

Let's simulate a node failure to prove that HAProxy is providing high availability to NGINX.

Simulating a Kubelet failure

Remember that KinD nodes are ephemeral and that stopping any container may cause it to fail on restart. So, how can we simulate a worker node failure since we can't simply stop the container?

To simulate a failure, we can stop the kubelet service on a node, which will alert  kube-apisever  so that it doesn't schedule any additional pods on the node. In our example, we want to prove that HAProxy is providing HA support for NGINX. We know that the running container is on  worker2 , so that's the node we want to "take down."

The easiest way to stop  kubelet  is to send a  docker exec  command to the container:

docker exec cluster01-worker2 systemctl stop kubelet

You will not see any output from this command, but if you wait a few minutes for the cluster to receive the updated node status, you can verify the node is down by looking at a list of nodes:

kubectl get nodes.

You will receive the following output:

Figure 2.7 – worker2 is in a NotReady state

This verifies that we just simulated a kubelet failure and that  worker2  is in a  NotReady  status.

Any pods that were running before the kubelet "failure" will continue to run, but  kube-scheduler  will not schedule any workloads on the node until the kubelet issue is resolved. Since we know the pod will not restart on the node, we can delete the pod so that it can be rescheduled on a different node.

You need to get the pod name and then delete it to force a restart:

kubectl get pods -n ingress-nginx

This will return the pods in the namespace, for example:

nginx-ingress-controller-7d6bf88c86-r7ztq

Delete the ingress controller pod using kubectl:

kubectl delete pod nginx-ingress-controller-7d6bf88c86-r7ztq -n ingress-nginx

This will force the scheduler to start the container on another worker node. It will also cause the HAProxy container to update the backend list, since the NGINX controller has moved to another worker node.

If you look at the HAProxy logs again, you will see that HAProxy has updated the backends to include  cluster01-worker3  and that it removed  cluster01-worker2  from the active servers list:

[WARNING] 093/194006 (6) : Server ingress_https/worker3 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

[WARNING] 093/194008 (6) : Server ingress_http/worker3 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

[WARNING] 093/195130 (6) : Server ingress_http/worker2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

[WARNING] 093/195131 (6) : Server ingress_https/worker2 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

If you plan to use this HA cluster for additional tests, you will want to restart the kubelet on  cluster01-worker2 . If you plan to delete the HA cluster, you can just run a KinD cluster delete and all the nodes will be deleted.

Summary

In this chapter, you learned about the Kubernetes SIG project called KinD. We went into details on how to install optional components in a KinD cluster, including Calico as the CNI and NGINX as the Ingress controller. Finally, we covered the details of the Kubernetes storage objects that are included with a KinD cluster.

Hopefully, with the help of this chapter, you now understand the power that using KinD can bring to you and your organization. It offers an easy to deploy, fully configurable Kubernetes cluster. The number of running clusters on a single host is theoretically limited only by the host resources.

In the next chapter, we will dive into Kubernetes objects. We've called the next chapter Kubernetes Bootcamp since it will cover the majority of the base Kubernetes objects and what each one is used for. The next chapter can be considered a "Kubernetes pocket guide." It contains a quick reference to Kubernetes objects and what they do, as well as when to use them.

It's a packed chapter and is designed to be a refresher for those of you who have experience with Kubernetes, or as a crash course for those of you who are new to Kubernetes. Our intention for this book is to go beyond the base Kubernetes objects since there are many books on the market today that cover the basics of Kubernetes very well.

Questions

1. What object must be created before you can create a  PersistentVolumeClaim ?A. PVCB. DiskC.  PersistentVolume D.  VirtualDisk

2. KinD includes a dynamic disk provisioner. What company created the provisioner?A. MicrosoftB.  CNCF C.  VMware D.  Rancher

3. If you create a KinD cluster with multiple worker nodes, what would you install to direct traffic to each node?A. Load balancerB. Proxy serverC. NothingD. Network load balancer

4. True or false: A Kubernetes cluster can only have one  CSIdriver  installed.A.  True B.  False

Kubernetes Bootcamp

We are sure that many of you have used Kubernetes in some capacity—you may have clusters running in production or you may have kicked the tires using kubeadm, Minikube, or Docker Desktop. Our goal for this book is to go beyond the basics of Kubernetes and, as such, we didn't want to rehash all of the basics of Kubernetes. Instead, we added this chapter as a bootcamp for anyone that may be new to Kubernetes or might have only played around with it a bit.

Since it is a bootcamp chapter we will not get in-depth on every topic, but by the end, you should know enough about the basics of Kubernetes to understand the remaining chapters. If you have a strong Kubernetes background, you may still find this chapter useful as a refresher, and we will get into more complex topics starting in Chapter 4, Services, Load Balancing, and External DNS.

In this chapter, we will cover the components of a running Kubernetes cluster, which include the control plane and the worker node(s). We will detail each Kubernetes object and its use cases. If you have used Kubernetes in the past and are comfortable using  kubectl  and fully understand Kubernetes objects (such as  DaemonSets ,  StatefulSets ,  ReplicaSets , and so on…), you may want to jump to Chapter 4, Services, Load Balancing, and External DNS, where we will install Kubernetes using KinD.

In this chapter, we will cover the following topics:

- An overview of Kubernetes components

- Exploring the control plane

- Understanding the worker node components

- Interacting with the API server

- Introducing Kubernetes objects

Technical requirements

This chapter has the following technical requirements:

- An Ubuntu 18.04 server with a minimum of 4 gigabytes (GB) of random-access memory (RAM)

- A KinD Kubernetes cluster

You can access the code for this chapter at the following GitHub repository: https://github.com/PacktPublishing/Kubernetes-and-Docker-The-Complete-Guide.

An overview of Kubernetes components

In any infrastructure, it is always a good idea to understand how the systems work together to provide services. With so many installer options out there today, many Kubernetes users have not had the need to understand how Kubernetes components integrate.

A few short years ago, if you wanted to run a Kubernetes cluster, you needed to install and configure each component manually. It was a steep learning curve to install a functioning cluster, which often led to frustration, causing many people and companies to say Kubernetes is just too difficult. The advantage of installing manually was that you truly understood how each component interacted, and if your cluster ran into issues after installation, you knew what to look for.

Nowadays, most people will click a button on a cloud provider and have a fully functioning Kubernetes cluster in minutes. On-premise installations have become just as easy, with options from Google, RedHat, Rancher, and more, removing the complexities of installing a Kubernetes cluster. The issues we see occur when you run into a problem or have questions after the installation. Since you didn't configure the Kubernetes components, you may not be able to explain to a developer how a Pod is scheduled on a worker node. Lastly, since you are running an installer provided by a third party, they may enable or disable features that you are not aware of, leading to an installation that may be against your company's security standards.

To understand how Kubernetes components work together, you must first understand the different components of a Kubernetes cluster. The following diagram is from the  Kubernetes.io  site and shows a high-level overview of a Kubernetes cluster component:

Figure 5.1 – Kubernetes cluster components

As you can see, the Kubernetes cluster is made up of several components. As we progress through the chapter, we'll discuss these components and the role they play in a Kubernetes cluster.

Exploring the control plane

As its name suggests, the control plane controls every aspect of a cluster. If your control plane goes down, you can probably imagine that your cluster will encounter issues. Without a control plane, a cluster will not have any scheduling abilities, which means that workloads that are running will remain running unless they are stopped and restarted. Since the control plane is extremely important, it is always suggested that you have at least three master nodes. Many production installations run more than three master nodes, but the number of installed nodes should always be an odd number. Let's look at why the control plane and its components are so vital to a running cluster by examining each one.

The Kubernetes API server

The first component to understand in a cluster is the  kube-apiserver  component. Since Kubernetes is application programming interface (API)-driven, every request that comes into a cluster goes through the API server. Let's look at a simple  get nodes  request using an API endpoint, as follows: https://10.240.100.100:6443/api/v1/nodes?limit=500

One common method users of Kubernetes deploy to interact with the API server is the kubectl utility. Every command that is issued using kubectl calls an API endpoint behind the scenes. In the preceding example, we executed a  kubectl get nodes  command, which sent an API request to the  kube-apiserver  process on  10.240.100.100  on port  6443 . The API call requested the  /api/vi/nodes  endpoint, which returned a list of the nodes in the cluster, as shown in the following screenshot:

Figure 5.2 – List of Kubernetes nodes

Without a running API server, all requests to your cluster will fail. So, as you can see, it is very important to have a  kube-apiserver  component running at all times. By running three or more master nodes, we can limit any impact of losing a master node.

Note

When running more than one master node, you need to have a load balancer in front of the cluster. The Kubernetes API server can be fronted by most standard solutions, including F5, HAProxy, and Seesaw.

The Etcd database

It's not a stretch to say that Etcd is your Kubernetes cluster. Etcd is a fast and highly available distributed key-value database that Kubernetes uses to store all cluster data. Each resource in a cluster has a key in the database. If you logged in to the node—or Pod—running Etcd, you could use the  etcdctl  executable to look at all of the keys in the database. The following code snippet shows an example from a cluster running KinD:

EtcdCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key –

cert=/etc/kubernetes/pki/etcd/server.crt get / --prefix --keys-only

The output from the preceding command contains too much data to list it all in this chapter. A base KinD cluster will return approximately 317 entries. All keys start with  /registry/<object> . For example, one of the keys that were returned is the  ClusterRole  for the  cluster-admin  key, as follows:  /registry/clusterrolebindings/cluster-admin .

We can use the key name to retrieve the value using the  etcdctl  utility by slightly modifying our previous command, as follows:

EtcdCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt get /registry/clusterrolebindings/cluster-admin

The output will contain characters that cannot be interpreted by your shell, but you will get the idea of the data stored in Etcd. For the  cluster-admin  key, the output shows us the following:

Figure 5.3 – etcdctl ClusterRoleBinding output

The reason we explain the entries in Etcd is to provide a background on how Kubernetes uses it to run a cluster. You have seen the output for the  cluster-admin  key directly from the database, but in everyday life you would query the API server using  kubectl get clusterrolebinding cluster-admin -o yaml , which would return the following:

Figure 5.4 – kubectl ClusterRoleBinding output

If you look at the output from the  kubectl  command and compare it with the output from the  etcdctl  query, you will see matching information. When you execute  kubectl  commands, the request goes to the API server, which then queries the Etcd database for the object's information.

kube-scheduler

As the name suggests, the  kube-scheduler  component is in charge of scheduling running Pods. Whenever a Pod is started in a cluster, the API server receives the requests and decides where to run the workload, based on multiple pieces of criteria including host resources and cluster policies.

kube-controller-manager

The  kube-controller-manager  component is actually a collection of multiple controllers that are included in a single binary. Including the four controllers in a single executable reduces complexity by running all four in a single process. The four controllers included in the  kube-controller-manager  component are the node, replication, endpoints, and service account and token controller.

Each controller provides a unique function to a cluster, and each controller and its function is listed here:

Each controller runs a non-terminating (never-ending) control loop. These control loops watch the state of each resource, making any changes required to normalize the state of the resource. For example, if you needed to scale a deployment from one to three nodes, the replication controller would notice that the current state has one Pod running, and the desired state is to have three Pods running. To move the current state to the desired state, two additional Pods will be requested by the replication controller.

cloud-controller-manager

This is one component that you may not have run into, depending on how your clusters are configured. Similar to the  kube-controller-manager  component, this controller containers four controllers in a single binary. The included controllers are the node, route, service, and volume controllers—each controller is responsible for interacting with their respective cloud service provider offering.

Understanding the worker node components

Worker nodes, as the name implies, are responsible for running workloads. When we discussed the  kube-scheduler  component of the control plane, we mentioned that when a new Pod is scheduled, the  kube-scheduler  component will decide which node to run the Pod on. It does this using information that has been sent to it from the worker nodes. This information is constantly updated to help spread Pods around a cluster to utilize resources efficiently. Here is a list of the worker node components.

kubelet

You may hear a worker node referred to as a  kubelet .  kubelet  is an agent that runs on all worker nodes, and it is responsible for running the actual containers.

kube-proxy

Contrary to the name,  kube-proxy  is not a proxy server at all.  kube-proxy  is responsible for routing network communication between a Pod and the outside network.

Container runtime

This is not represented in the picture, but each node also needs a container runtime. A container runtime is responsible for running the containers. The first thing you might think of is Docker. While Docker is a container runtime, it is not the only runtime option available. Over the last year, other options have become available and are quickly replacing Docker as the preferred container runtime. The two most prominent Docker replacements are CRI-O and containerd.

For the book exercises, we will create a Kubernetes cluster using KinD. At the time of this writing, KinD only offers official support for Docker as the container runtime, with limited support for Podman.

Interacting with the API server

As we mentioned earlier, you interact with the API server using either direct API requests or the  kubectl  utility. We will focus on using  kubectl  for the majority of our interaction in this book, but we will call out using direct API calls where applicable.

Using the Kubernetes kubectl utility

kubectl  is a single executable file that allows you to interact with the Kubernetes API using a command-line interface (CLI). It is available for most major operating systems and architectures, including Linux, Windows, and Mac.

Installation instructions for most operating systems are located on the Kubernetes site at https://kubernetes.io/docs/tasks/tools/install-kubectl/. Since we are using Linux as our operating system for the exercises in the book, we will cover installing  kubectl  on a Linux machine. Follow these steps:

1. To download the latest version of  kubectl , you can run a  curl  command that will download it, as follows:

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

2. After downloading, you need to make the file executable by running the following command:

chmod +x ./kubectl

3. Finally, we will move the executable to your path, as follows:

sudo mv ./kubectl /usr/local/bin/kubectl

You now have the latest  kubectl  utility on your system and can execute  kubectl  commands from any working directory.

Kubernetes is updated every 3 months. This includes upgrades to the base Kubernetes cluster components and the  kubectl  utility. You may run into a version mismatch between a cluster and your  kubectl  command, requiring you to either upgrade or download your  kubectl  executable. You can always check the version of both by running a  kubectl version  command, which will output the version of both the API server and the  kubectl  client. The output from a version check is shown in the following code snippet:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2020-01-14T00:09:19Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

As you can see from the output, the  kubectl  client is running version  1.17.1  and the cluster is running  1.17.0 . A minor version difference in the two will not cause any issues. In fact, the official supported version difference is within one major version release. So, if your client is running version 1.16 and the cluster is running 1.17, you would be within the supported version difference. While this may be supported, it doesn't mean that you won't run into issues if you are trying to use any new commands or objects included in the higher version. In general, you should try to keep your cluster and client version in sync to avoid any issues.

Through the remainder of this chapter, we will discuss Kubernetes objects and how you interact with the API server to manage each object. But before diving into the different objects, we wanted to mention one commonly overlooked option of the  kubectl  utility: the  verbose  option.

Understanding the verbose option

When you execute a  kubectl  command, the only outputs you will see by default are any direct responses to your command. If you were to look at all Pods in the  kube-system  namespace, you would receive a list of all Pods. In most cases this is the desired output, but what if you issued a get Pods request and received an error from the API server? How could you get more information about what might be causing the error?

By adding the verbose option to your  kubectl  command, you can get additional details about the API call itself and any replies from the API server. Often, the replies from the API server will contain additional information that may be useful to find the root cause of the issue.

The  verbose  option has multiple levels ranging from 0 to 9; the higher the number, the more output you will receive. The following screenshot has been taken from the Kubernetes site, detailing each level and what the output will include:

Figure 5.5 – Verbosity description

You can experiment with the levels by adding the  -v  or  --v  option to any  kubectl  command.

General kubectl commands

The CLI allows you to interact with Kubernetes in an imperative and declarative manner. Using an imperative command involves you telling Kubernetes what to do—for example,  kubectl run nginx –image nginx . This tells the API server to create a new deployment called  nginx  that runs an image called  nginx . While imperative commands are useful for development and quick fixes or testing, you will use declarative commands more often in a production environment. In a declarative command, you tell Kubernetes what you want. To use declarative commands, you send a manifest to the API server, usually written in YAML Ain't Markup Language (YAML), which declares what you want Kubernetes to create.

kubectl  includes commands and options that can provide general cluster information or information about an object. The following table contains a cheat-sheet of commands and what they are used for. We will use many of these commands in future chapters, so you will see them in action throughout the book:

With an understanding of each Kubernetes component and how to interact with the API server using imperative commands, we can now move on to Kubernetes objects and how we use  kubectl  to manage them.

Introducing Kubernetes objects

This section will contain a lot of information and, since this is a bootcamp, we will not go into deep details on each object. As you can imagine, each object could have its own chapter, or multiple chapters, in a book. Since there are many books on Kubernetes that go into detail on the base objects, we will only cover the required details of each to have an understanding of each one. In the following chapters, we will include additional details for objects as we build out our cluster using the book exercises.

Before we go on to understand what Kubernetes objects really are, let's first explain Kubernetes manifests.

Kubernetes manifests

The files that we will use to create Kubernetes objects are referred to as manifests. A manifest can be created using YAML or JavaScript Object Notation (JSON)—most manifests use YAML, and that is the format we will use throughout the book.

The content of a manifest will vary depending on the object, or objects, that will be created. At a minimum, all manifests require a base configuration that include the  apiVersion , object  KinD , and  metadata  fields, as can be seen here:

apiVersion: apps/v1

KinD: Deployment

metadata:

labels:

app: grafana

name: grafana

namespace: monitoring

The preceding manifest alone is not complete; we are only showing the beginning of a full deployment manifest. As you can see in the file, we start with the three required fields that all manifests are required to have: the  apiVersion ,  KinD , and  metadata  fields.

You may also notice that there are spaces in the file. YAML is very format-specific, and if the format of any line is off by even a single space, you will receive an error when you try to deploy the manifest. This takes time to get used to, and even after creating manifests for a long time, formatting issues will still pop up from time to time.

What are Kubernetes objects?

When you want to add or delete something from a cluster, you are interacting with a Kubernetes object. An object is what a cluster uses to keep a list of a desired state. The desired state may be to create, delete, or scale an object. Based on the desired state of the object, the API server will make sure that the current state equals the desired state.

To retrieve a list of objects a cluster supports, you can use the  kubectl api-resources  command. The API server will reply with a list of all objects, including any valid short name, namespace support, and supported API group. There are approximately 53 base objects included with a base cluster, but an abbreviated list of the most common objects is shown in the following screenshot:

Figure 5.6 – Kubernetes API resources

Since this chapter is a bootcamp, we will offer a brief review of many of the objects in the list. In order to ensure that you can follow the remaining chapters, we will provide an overview of each object and how to interact with them. Some objects will also be explained in greater detail in future chapters, including  Ingress ,  RoleBindings ,  ClusterRoles ,  StorageClasses , and more.

Reviewing Kubernetes objects

To make this section easier to follow, we will present each object in the order they were provided by the  kubectl api-services  command.

Depending on which version of Kubernetes you’re running, you may not see a  metadata.selfLink  attribute. This attribute was removed in 1.20 and has broken many controllers that rely on that link. The example below will assume that the selfLink attribute is not available.

Most objects in a cluster are run in a namespace, and to create/edit/read them, you should supply the  -n <namespace>  option to any  kubectl  command. To find a list of objects that accept a namespace option, you can reference the output from our previous  get api-server  command. If an object can be referenced by a namespace, the namespaced column will show  true . If the object is only referenced by the cluster level, the namespaced column will show  false .

ConfigMaps

A ConfigMap stores data in key-value pairs, providing a way to keep your configuration separate from your application.  ConfigMaps  may contain data from a literal value, files, or directories.

Here is an imperative example:

kubectl create configmap <name> <data>

The  name  option will vary based on the source of the  ConfigMap . To use a file or a directory, you supply the  --from-file  option and either the path to a file or an entire directory, as shown here:

kubectl create configmap config-test --from-file=/apps/nginx-config/nginx.conf

This would create a new ConfigMap named  config-test , with the  nginx.conf  key containing the content of the  nginx.conf  file as the value.

If you needed to have more than one key added in a single  ConfigMap , you could put each file into a directory and create the  ConfigMap  using all of the files in the directory. As an example, you have three files in a directory located at  ~/config/myapp . In the directory are three files, each containing data, called  config1 ,  config2 , and  config3 . To create a  ConfigMap  that would add each file into a key, you need to supply the  --from-file  option and point to the directory, as follows:

kubectl create configmap config-test --from-file=/apps/config/myapp

This would create a new  ConfigMap  with three key values called  config1 ,  config2 , and  config3 . Each key would contain a value equal to the content of each file in the directory.

To quickly show a  ConfigMap , using the example to create a  ConfigMap  from a directory, we can retrieve the  ConfigMap  using the get command,  kubectl get configmaps config-test , resulting in the following output:

NAME DATA AGE config-test 3 7s

We can see that the  ConfigMap  contains three keys, shown as a  3  under the  DATA  column. To look in greater detail, we can use the same  get  command and output the value of each key as YAML by adding the  -o yaml  option to the  kubectl get configmaps config-test -o yaml  command, resulting in the following output:

Figure 5.7 – kubectl ConfigMap output

Looking at the preceding output, you can see each key matches the filenames, and the value for each key contains the data in each respective file.

One limitation of  ConfigMaps  that you should keep in mind is that the data is easily accessible to anyone with permissions to the object. As you can see from the preceding output, a simple  get  command shows the data in cleartext. Due to this design, you should never store sensitive information such as a password in a  ConfigMap . Later in this section, we will cover an object that was designed to store sensitive information, called a Secret.

Endpoints

An endpoint maps a service to a Pod or Pods. This will make more sense when we explain the  Service  object. For now, you only need to know that you can use the CLI to retrieve endpoints by using the  kubectl get endpoints  command. In a new KinD cluster, you will see a value for the Kubernetes API server in the default namespace, as illustrated in the following code snippet:

NAMESPACE   NAME     ENDPOINTS         AGE

default   kubernetes 172.17.0.2:6443   22h

The output shows that the cluster has a service called  kubernetes  that has an endpoint at the Internet Protocol (IP) address  172.17.0.2  on port  6443 . Later, you will see when looking at endpoints that they can be used to troubleshoot service and ingress issues.

Events

The  Events  object will display any events for a namespace. To get a list of events for the  kube-system  namespace, you would use the  kubectl get events -n kube-system  command.

Namespaces

A namespace is an object to divide a cluster into logical units. Each namespace allows granular management of resources, including permissions, quotas, and reporting.

The  namespace  object is used for namespace tasks, which are cluster-level operations. Using the  namespace  object, you can execute commands including  create ,  delete ,  edit , and  get .

The syntax for the command is  kubectl <verb> ns <namespace name> .

For example, to describe the  kube-system  namespace, we would execute a  kubectl describe namespaces kube-system  command. This will return information for the namespace, including any labels, annotations, and assigned quotas, as illustrated in the following code snippet:

Name: kube-system

Labels: <none>Annotations: <none>

Status: Active

No resource quota.

No LimitRange resource.

In the preceding output, you can see that this namespace does not have any labels, annotations, or resource quotas assigned.

This section is only meant to introduce the concept of namespaces as a management unit in multi-tenant clusters. If you plan to run clusters with multiple tenants, you need to understand how namespaces can be used to secure a cluster.

Nodes

The  nodes  object is a cluster-level resource that is used to interact with the cluster's nodes. This object can be used with various actions including  get ,  describe ,  label , and  annotate .

To retrieve a list of all of the nodes in a cluster using  kubectl , you need to execute a  kubectl get nodes  command. On a new KinD cluster running a simple one-node cluster, this would display as follows:

NAME STATUS ROLES AGE VERSION

KinD-control-plane Ready master 22h v1.17.0

You can also use the nodes object to get details of a single node using the  describe  command. To get a description of the KinD node listed previously, we can execute  kubectl describe node KinD-control-plane , which would return details on the node, including consumed resources, running Pods, IP classless inter-domain routing (CIDR) ranges, and more.

Persistent Volume Claims

We will describe Persistent Volume Claims (PVCs) in more depth in a later chapter, but for now you just need to know that a PVC is used by a Pod to consume persistent storage. A PVC uses a persistent volume (PV) to map the storage resource. As with most other objects we have discussed, you can issue  get ,  describe , and  delete  commands on a PVC object. Since these are used by Pods, they are a  namespaced  object, and must be created in the same namespace as the Pod(s) that will use the PVC.

PVs

PVs are used by PVCs to create a link between the PVC and the underlying storage system. Manually maintaining PVs is a messy task and in the real world it should be avoided, since Kubernetes includes the ability to manage most common storage systems using the Container Storage Interface (CSI). As mentioned in the PVC object section, we will discuss how Kubernetes can automatically create the PVs that will be linked to PVCs.

Pods

The Pod object is used to interact with the Pods that are running your container(s). Using the  kubectl  utility you can use commands such as  get ,  delete , and  describe . For example, if you wanted to get a list of all Pods in the  kube-system  namespace, you would execute a  kubectl get Pods -n kube-system  command that would return all Pods in the namespace, as follows:

Figure 5.8 – All Pods in the kube-system namespace

While you can create a Pod directly, you should avoid doing so unless you are using a Pod for quick troubleshooting. Pods that are created directly cannot use many of the features provided by Kubernetes, including scaling, automatic restarts, or rolling upgrades. Instead of creating a Pod directly, you should use a Deployment, or in some rare cases a  ReplicaSet  object or replication controller.

Replication controllers

Replication controllers will manage the number of running Pods, keeping the desired replicas specified running at all times. If you create a replication controller and set the replica count to  5 , the controller will always keep five Pods of the application running.

Replication controllers have been replaced by the  ReplicaSet  object, which we will discuss in its own section. While you can still use replication controllers, you should consider using a Deployment or a  ReplicaSet  object.

ResourceQuotas

It is becoming very common to share a Kubernetes cluster between multiple teams, referred to as a multi-tenant cluster. Since you will have multiple teams working in a single cluster, you should consider creating quotas to limit any potential of a single tenant consuming all the resources in a cluster or on a node. Limits can be set on most cluster objects, including the following:

- Central processing unit (CPU)

- Memory

- PVCs

- ConfigMaps

- Deployments

- Pods, and more

Setting a limit will stop any additional objects being created once the limit is hit. If you set a limit of 10 Pods for a namespace and a user creates a new Deployment that attempts to start 11 Pods, the 11th Pod will fail to start up and the user will receive an error.

A basic manifest file to create a quota for memory and CPU would look this:

apiVersion: v1

kind: ResourceQuota

metadata:

name: base-memory-cpu

spec:

hard:

requests.cpu: "2"

requests.memory: 8Gi

limits.cpu: "4"

limits.memory: 16Gi

This will set a limit on the total amount of resources the namespace can use for CPU and memory requests and limits.

Once a quota has been created, you can view the usage using the  kubectl describe  command. In our example, we named the  ResourceQuota   base-memory-cpu . To view the usage, we would execute the  kubectl get resourcequotas base-memory-cpu  command, resulting in the following output:

Name: base-memory-cpu

Namespace: default

Resource Used Hard

-------- ---- ----

limits.cpu 0 4

limits.memory 0 16Gi

requests.cpu 0 2

requests.memory 0 8Gi

ResourceQuota  objects are used to control a cluster's resources. By allocating the resources to a namespace, you can guarantee that a single tenant will have the required CPU and memory to run their application, while limiting the impact that a poorly written application can have on other applications.

Secrets

Earlier we described how to use a  ConfigMap  object to store configuration information. We mentioned that  ConfigMap  objects should never be used to store any type of sensitive data. This is the job of a Secret.

Secrets are stored as Base64-encoded strings, which aren't a form of encryption. So, why separate Secrets from  ConfigMap  objects? Providing a separate object type offers an easier way to maintain access controls and the ability to inject sensitive information using an external system.

Secrets can be created using a file, directory, or from a literal string. As an example, we have a MySQL image we want to execute, and we would like to pass the password to the Pod using a Secret. On our workstation, we have a file called  dbpwd  in our current working directory that has our password in it. Using the  kubectl  command, we can create a Secret by executing  kubectl create secret generic mysql-admin --from-file=./dbpwd .

This would create a new a Secret called  mysql-admin  in the current namespace, with the content of the  dbpwd  file. Using  kubectl , we can get the output of the Secret by running the  kubectl get secret mysql-admin -o yaml  command, which would output the following:

apiVersion: v1

data:

dbpwd: c3VwZXJzZWNyZXQtcGFzc3dvcmQK

kind: Secret

metadata:

creationTimestamp: "2020-03-24T18:39:31Z"

name: mysql-admin

namespace: default

resourceVersion: "464059"

uid: 69220ebd-c9fe-4688-829b-242ffc9e94fc

type: Opaque

Looking at the preceding output, you can see that the  data  section contains the name of our file and then a Base64-encoded value, which was created from the content of the file.

If we copy the Base64 value from the Secret and pipe it out to the  base64  utility, we can easily decode the password, as follows:

echo c3VwZXJzZWNyZXQtcGFzc3dvcmQK | base64 -d

supersecret-password

When using the  echo  command to Base64-encode strings, add the  -n  flag to avoid adding an additional  \n . Instead of  echo 'test' | base64 , use  echo -n 'test' | base64 .

Everything is stored in Etcd, but we are concerned that someone may be able to hack into the master server and steal a copy of the Etcd database. Once someone has a copy of the database, they could easily use the  etcdctl  utility to look through the content to retrieve all of our Base64-encoded Secrets. Luckily, Kubernetes added a feature to encrypt Secrets when they are written to a database.

Enabling this feature can be fairly complex for many users, and while it sounds like a good idea, it does present some potential issues that you should consider before implementing it. If you would like to read the steps on encrypting your Secrets at rest, you can view these on the Kubernetes site at https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/.

Another option to secure Secrets is to use a third-party Secrets management tool such as HashiCorp's Vault or CyberArk's Conjur.

Service accounts

Kubernetes uses service accounts to enable access controls for workloads. When you create a Deployment, you may need to access other services or Kubernetes objects. Since Kubernetes is a secure system, each object or service your application tries to access will evaluate role-based access control (RBAC) rules to accept or deny the request.

Creating a service account using a manifest is a straightforward process, requiring only a few lines in the manifest. The following code snippet shows a service account manifest to create a service account for a Grafana Deployment:

apiVersion: v1

kind: ServiceAccount

metadata:

name: grafana

namespace: monitoring

You combine the service account with role bindings and roles to allow access to the required services or objects.

When you create a ServiceAccount a  Secret  will also be created that stores a static token, the CA certificate for the cluster, and the name of the  Namespace  the  Secret  is in. This token does not have an expiration and should not be used outside of the cluster.

Services

In order to make an application running in a Pod(s) available to the network, you need to create a service. A service object stores information about how to expose the application, including which Pods are running on the application and the network ports to reach them.

Each service has a network type that is assigned when they are created, and they include the following:

- ClusterIP : A network type that is only accessible inside the cluster itself. This type can still be used for external requests using an ingress controller, which will be discussed in a later chapter.

- NodePort : A network type that exposes the service to a random port between ports  30000 - 32767 .This port becomes accessible by targeting any worker node in a cluster on the assigned  NodePort . Once created, each node in the cluster will receive the port information and incoming requests will be routed via  kube-proxy .

- LoadBalancer : This type requires an add-on to use inside a cluster. If you are running Kubernetes on a public cloud provider, this type will create an external load balancer that will assign an IP address to your service. Most on-premise Kubernetes installations do not include support for the  LoadBalancer  type, but some offerings such as Google's Anthos do offer support for it. In a later chapter, we will explain how to add an open source project called  MetalLB  to a Kubernetes cluster to provide support for the  LoadBalancer  type.

- ExternalName : This type is different from the other three. Unlike the other three options, this type will not assign an IP address to the service. Instead, this is used to map the internal Kubernetes Domain Name System (DNS) name to an external service.

As an example, we have deployed a Pod running Nginx on port  80 . We want to create a service that will allow this Pod to receive incoming requests on port  80  from within the cluster. The code for this can be seen in the following snippet:

apiVersion: v1

kind: Service

metadata:

labels:

app: nginx-web-frontend

name: nginx-web

spec:

ports:

- name: http

port: 80

targetPort: 80

selector:

app: nginx-web

In our manifest, we create a label with a value of  app  and assign a value of  nginx-web-frontend . We have called the service itself  nginx-web  and we exposed the service on port  80 , targeting the Pod port of  80 . The last two lines of the manifest are used to assign the Pods that the service will forward to, also known as endpoints. In this manifest, any Pod that has the label of  app  with a value of  nginx-web  in the namespace will be added as an endpoint to the service.

CustomResourceDefinitions

CustomResourceDefinitions (CRDs) allow anyone to extend Kubernetes by integrating your application into a cluster as a standard object. Once a CRD is created, you can reference it using an API endpoint, and it can be interacted with using standard  kubectl  commands.

DaemonSets

A  DaemonSet  allows you to deploy a Pod on every node in a cluster, or a subset of nodes. A common use for a  DaemonSet  is to deploy a log forwarding Pod such as FluentD to every node in a cluster. Once deployed, the  DaemonSet  will create a FluentD Pod on all existing nodes. Since a  DaemonSet  deploys to all nodes, any additional nodes that are added to a cluster will have a FluentD Pod started once the node has joined the cluster.

Deployments

We mentioned earlier that you should never deploy a Pod directly, and we also introduced the  ReplicationContoller  object as an alternative to creating Pods directly. While both of these will create your Pods, each comes with the following limitation: Pods created directly cannot be scaled and cannot be upgraded using rolling updates.

Pods created by a  ReplicationController  can be scaled, and can perform rolling updates. However, they do not support rollbacks, and upgrades cannot be done declaratively.

Deployments offer you a few advantages, including a way to manage your upgrades declaratively and the ability to roll back to previous revisions. Creating a Deployment is actually a three-step process executed by the API server: a Deployment is created, which creates a ReplicaSet object, which then creates the Pod(s) for the application.

Even if you don't plan to use these features, you should use Deployments by default so that you can leverage the features at a future date.

ReplicaSets

ReplicaSets can be used to create a Pod or a set of Pods (replicas). Similar to the  ReplicationController  object, a  ReplicaSet  object will maintain the set number of Pods defined in the replica count of the object. If there are too few Pods, Kubernetes will reconcile the difference and create the missing Pods. If there are too many Pods for a ReplicaSet, Kubernetes will delete Pods until the number is equal to the replica count set in the object.

In general, you should avoid creating ReplicaSets directly. Instead, you should create a Deployment, which will create and manage a  ReplicaSet .

StatefulSets

StatefulSets  offer some unique features when creating Pods. They provide features that none of the other Pod creation methods offer, including the following:

- Known Pod names

- Ordered Deployment and scaling

- Ordered updates

- Persistent storage creation

The best way to understand the advantages of a  StatefulSet  is to review an example manifest from the Kubernetes site, shown in the following screenshot:

Figure 5.9 – StatefulSet manifest example

Now, we can look at the objects that the  StatefulSet  object created.

The manifest specifies that there should be three replicas of a Pod named  nginx . When we get a list of Pods, you will see that three Pods were created using the  nginx  name, with an additional dash and an incrementing number. This is what we meant in the overview when we mentioned that Pods will be created with known names, as illustrated in the following code snippet:

NAME READY STATUS RESTARTS AGE

web-0 1/1 Running 0 4m6s

web-1 1/1 Running 0 4m2s

web-2 1/1 Running 0 3m52s

The Pods are also created in order – w eb-0  must be fully deployed before  web-1  is created, and then finally  web-2 .

Finally, for this example, we also added a PVC to each Pod using the  VolumeClaimTemplate  in the manifest. If you look at the output of the  kubectl get pvc  command, you would see that three PVCs were created with names we expected (note that we removed the  VOLUME  column due to space), as illustrated in the following code snippet:

NAME STATUS CAPACITY ACCESS MODES STORAGECLASS AGE

www-web-0 Bound 1Gi RWO nfs 13m

www-web-1 Bound 1Gi RWO nfs 13m

www-web-2 Bound 1Gi RWO nfs 12m

In the  VolumeClaimTemplate  section of the manifest, you will see that we assigned the name  www  to the PVC claim. When you assign a volume in a  StatefulSet , the PVC name will combine the name used in the claim template, combined with the name of the Pod. Using this naming, you can see why Kubernetes assigned the PVC names  www-web-0 ,  www-web-1 , and  www-web-2 .

HorizontalPodAutoscalers

One of the biggest advantages of running a workload on a Kubernetes cluster is the ability to easily scale your Pods. While you can scale using the  kubectl  command or by editing a manifest's replica count, these are not automated and require manual intervention.

HorizontalPodAutoscalers (HPAs) provide the ability to scale an application based on a set of criteria. Using metrics such as CPU and memory usage, or your own custom metrics, you can set a rule to scale your Pods up when you need more Pods to maintain your service level. After a cooldown period, Kubernetes will scale the application back to the minimum number of Pods defined in the policy.

To quickly create an HPA for an  nginx  Deployment, we can execute a  kubectl  command using the  autoscale  option, as follows:

kubectl autoscale deployment nginx --cpu-percent=50 --min=1 --max=5

You can also create a Kubernetes manifest to create your HPAs. Using the same options as those we did in the CLI, our manifest would look like this:

apiVersion: autoscaling/v1

kind: HorizontalPodAutoscaler

metadata:

name: nginx-deployment

spec:

maxReplicas: 5

minReplicas: 1

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

name: nginx-deployment

targetCPUUtilizationPercentage: 50

Both options will create an HPA that will scale our  nginx-deployment nginx  Deployment up to five replicas when the Deployment hits a CPU utilization of 50%. Once the Deployment usage falls below 50% and the cooldown period is reached (by default, 5 minutes), the replica count will be reduced down to 1.

CronJobs

If you have used Linux cronjobs in the past, then you already know what a Kubernetes  CronJob  object is. If you don't have a Linux background, a cronjob is used to create a scheduled task. As another example, if you are a Windows person, it's similar to Windows scheduled tasks.

An example manifest that creates a  CronJob  is shown in the following code snippet:

apiVersion: batch/v1

kind: CronJob

metadata:

name: hello-world

spec:

schedule: "*/1 * * * *"

jobTemplate:

spec:

template:

spec:

containers:

- name: hello-world

image: busybox

args:

- /bin/sh

- -c

- date; echo Hello World!

restartPolicy: OnFailure

The  schedule  format follows the standard  cron  format. From left to right, each  *  represents the following:

- Minute (0 – 59)

- Hour (0 -23)

- Day (1 -31)

- Month (1 – 12)

- Day of the week (0 – 6) (Sunday = 0, Saturday = 6)

Cron jobs accept step values, which allow you to create a schedule that can execute every minute, every 2 minutes, or every hour.

Our example manifest will create a  cronjob  that runs an image called  hello-world  every minute and outputs  Hello World!  in the Pod log.

Jobs

Jobs allow you to execute a specific number of executions of a Pod or Pods. Unlike a  cronjob  object, these Pods are not run on a set schedule, but rather they will execute once created. Jobs are used to execute a task that may only need to be executed at the initial Deployment stage.

An example use case would be an application that may require the creation of Kubernetes CRDs that must exist before the main application is deployed. The Deployment would wait until the job execution completed successfully.

Events

Events objects store information about events for Kubernetes objects. You do not create events; rather, you can only retrieve events. For example, to retrieve events for the  kube-system  namespace, you would execute  kubectl get events -n kube-system , or to show events for all namespaces, you'd execute  kubectl get events --all-namespaces .

Ingresses

You may have noticed that the  Ingress  object was listed twice in our  api-server  output. This will happen to objects as Kubernetes upgrades are released and objects changed in the API server. In the case of Ingress, it was original part of the extensions API and was moved to the  networking.k8s.io  API in version 1.16. The project will wait a few releases before deprecating the old API call, so in our example cluster running Kubernetes 1.21, using either API will work. In version 1.18, the Ingress extensions were deprecated.

NetworkPolicies

NetworkPolicy  objects let you define how network traffic can flow through your cluster. They allow you to use Kubernetes native constructs to define which Pods can talk to other Pods. If you've ever used Security Groups in Amazon Web Services (AWS) to lock down access between two groups of systems, it's a similar concept. As an example, the following policy will allow traffic on port  443  to Pods in the  myns  namespace from any namespace with the  app.kubernetes.io/name: ingress-nginx  label on it (which is the default label for the  nginx-ingress  namespace):

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: allow-from-ingress

namespace: myns

spec:

PodSelector: {}

policyTypes:

- Ingress

ingress:

- from:

- namespaceSelector:

matchLabels:

app.kubernetes.io/name: ingress-nginx

ports:

- protocol: TCP

port: 443

A  NetworkPolicy  object is another object that you can use to secure a cluster. They should be used in all production clusters, but in a multi-tenant cluster they should be considered a must-have to secure each namespace in the cluster.

PodSecurityPolicies

PodSecurityPolicies  (PSPs) are how your cluster protects your nodes from your containers. They allow you to limit the actions that a Pod can execute in a cluster. Some examples include denying access to the  HostIPC  and  HostPath , and running a container in a privileged mode.

We'll get into the details of PSPs in Chapter 10, Creating Pod Security Policies. The key point to remember about PSPs is that without them, your containers can do almost anything on your nodes.

ClusterRoleBindings

Once you have defined a  ClusterRole , you bind it to a subject via a  ClusterRoleBinding . A  ClusterRole  can be bound to a  User ,  Group , or  ServiceAccount .

We'll explore  ClusterRoleBinding  details in Chapter 8, RBAC Policies and Auditing.

ClusterRoles

A  ClusterRole  combines a set of permissions for interacting with your cluster's API. A  ClusterRole  combines a verb or action with an API group to define a permission. For instance, if you only want your continuous integration/continuous delivery (CI/CD) pipeline to be able to patch your Deployments so that it can update your image tag, you might use a  ClusterRole  like this:

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

name: patch-deployment

rules:

- apiGroups: ["apps/v1"]

resources: ["deployments"]

verbs: ["get", "list", "patch"]

A  ClusterRole  can apply to APIs at both the cluster and namespace level.

RoleBindings

The  RoleBinding  object is how you associate a Role or  ClusterRole  to a subject and namespace. For instance, the following  RoleBinding  object will allow the  aws-codebuild  user to apply the  patch-openunison  ClusterRole to the  openunison  namespace:

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

name: patch-openunison

namespace: openunison

subjects:

- kind: User

name: aws-codebuild

apiGroup: rbac.authorization.k8s.io

roleRef:

kind: ClusterRole

name: patch-deployment

apiGroup: rbac.authorization.k8s.io

Even though this references a  ClusterRole , it will only apply to the  openunison  namespace. If the  aws-codebuild  user tries to patch a Deployment in another namespace, the API server will stop it.

Roles

As with a  ClusterRole , Roles combine API groups and actions to define a set of permissions that can be assigned to a subject. The difference between a  ClusterRole  and a  Role  is that a  Role  can only have resources defined at the namespace level and applies only within a specific namespace.

CsiDrivers

Kubernetes uses the  CsiDriver  object to connect nodes to a storage system.

You can list all CSI drivers that are available on a cluster by executing the  kubectl get csidriver  command. In one of our clusters we are using Netapp's SolidFire for storage, so our cluster has the Trident CSI driver installed, as can be seen here:

NAME CREATED AT

csi.trident.netapp.io 2019-09-04T19:10:47Z

CsiNodes

To avoid storing storage information in the node API object, the  CSINode  object was added to the API server to store information generated by the CSI drivers. The information that is stored includes mapping Kubernetes node names to CSI node names, CSI driver availability, and the volume topology.

StorageClasses

Storage classes are used to define a storage endpoint. Each storage class can be assigned labels and policies, allowing a developer to select the best storage location for their persistent data. You may create a storage class for a backend system that has all Non-Volatile Memory Express (NVMe) drives, assigning it the name  fast , while assigning a different class to a Netapp Network File System (NFS) volume running standard drives, using the name  standard .

When a PVC is requested, the user can assign a  StorageClass  that they wish to use. When the API server receives the request, it finds the matching name and uses the  StorageClass  configuration to create the volume on the storage system using a provisioner.

At a very high level, a  StorageClass  manifest does not require a lot of information. Here is an example of a storage class using a provisioner from the Kubernetes incubator project to provide NFS auto-provisioned volumes, named  nfs :

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

name: nfs

provisioner: nfs

Storage classes allow you to offer multiple storage solutions to your users. You may create a class for cheaper, slower storage while offering a second class that supports high throughput for high data requirements. By providing a different class to each offering, you allow developers to select the best choice for their application.

Summary

In this chapter, you were thrown into a Kubernetes bootcamp that presented a lot of technical material in a short amount of time. Try to remember that this will all become easier as you get into the Kubernetes world in more depth. We realize that this chapter had a lot of information on many objects. Many of the objects will be used in later chapters, and they will be explained in greater detail.

You learned about each Kubernetes component and how they interact to create a cluster. With this knowledge, you have the required skills to look at errors in a cluster and determine which component may be causing an error or issue. We covered the control plane of a cluster where the  api-server ,  kube-scheduler ,  Etcd , and control managers run. The control plane is how users and services interact with a cluster; using the  api-server  and the  kube-scheduler  will decide which worker node to schedule your Pod(s) on. You also learned about Kubernetes nodes that run the  kubelet  and  kube-proxy  components, and a container runtime.

We covered the  kubectl  utility that you will use to interact with a cluster. You also learned some common commands that you will use on a daily basis, including  logs  and  describe .

In the next chapter, we will create a development Kubernetes cluster that we will use as the base cluster for the remaining chapters. Throughout the remainder of the book, we will reference many of the objects that were presented in this chapter, helping to explain them by using them in real-world examples.

Questions

1. A Kubernetes control plane does not include which of the following components?

1. api-server

2. kube-scheduler

3. Etcd

4. Ingress controller

2. What is the name of the component that keeps all of the cluster information?

1. api-server

2. Master controller

3. kubelet

4. Etcd

3. Which component is responsible for selecting the node that will run a workload?

1. kubelet

2. api-server

3. kube-scheduler

4. Pod-scheduler

4. Which option would you add to a  kubectl  command to see additional output from a command?

1. Verbose

2. -v

3. –verbose

4. -log

5. Which service type creates a randomly generated port, allowing incoming traffic to any worker node on the assigned port to access the service?

1. LoadBalancer

2. ClusterIP

3. None—it's the default for all services

4. NodePort

6. If you need to deploy an application on a Kubernetes cluster that requires known node names and a controlled startup of each Pod, which object would you create?

1. StatefulSet

2. Deployment

3. ReplicaSet

4. ReplicationController