Operating OpenShift

An SRE Approach to Managing Infrastructure

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Rick Rackow and Manuel Dewald

Operating OpenShift

by Rick Rackow and Manuel Dewald

Copyright © 2022 Rick Rackow and Manuel Dewald. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Acquisitions Editor: John Devins

- Development Editor: Corbin Collins

- Production Editor: Clare Jensen

- Interior Designer: David Futato

- Cover Designer: Karen Montgomery

- Illustrator: Kate Dullea

- November 2022: First Edition

Revision History for the Early Release

- 2022-05-16: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098106393 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Operating OpenShift, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-10632-4

Chapter 1. Introduction

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the development editor at ccollins@oreilly.com.

Operating distributed software is a difficult task. This task requires humans with a deep understanding of the system they maintain. No matter how much automation you create, it will never replace highly skilled operations personnel.

OpenShift is a platform, built to help software teams develop and deploy their distributed software. It comes with a large set of tools that are built-in or can be deployed easily. While it can be of great help to its users and eliminate a lot of traditionally manual operations burdens, OpenShift itself is a distributed system as well, that needs to be deployed, operated, and maintained.

Many companies have platform teams that provide development platforms based on OpenShift to software teams so the maintenance effort is centralized and the deployment patterns are standardized across organizations. These platform teams shift more and more into the direction of Site Reliability Engineering (SRE) teams, where software development practices are applied to operations tasks. Scripts are replaced by proper software solutions that can be tested easier and deployed automatically using CI/CD systems. Alerts are turned from simple cause-based alerts like “a high amount of memory is used on Virtual Machine 23” into symptom-based alerts based on Service Level Objectives (SLO) that reflect customer experience, like “processing of requests takes longer than we expect it to”.

OpenShift provides all the tools you need to run software on top of it with SRE paradigms, from a monitoring stack to an integrated CI/CD system that you can use to observe and run both the software deployed to the OpenShift cluster, as well as the cluster itself. But still building the automation, implementing a good alerting strategy, and finally debugging issues that occur when operating an OpenShift cluster is a difficult task that requires skilled operations or SRE staffing.

Even in SRE teams, traditionally a good portion of the engineers’ time is dedicated to manual operations tasks, often called toil. The operations time should be capped, though, as the main goal of SRE is to tackle the toil with software engineering. In Site Reliability Engineering, the authors suggest a 50% cap for working on toil.

Traditional Operations Teams

The goal of having an upper limit for toil is to avoid shifting back into an operations team where people spend most of the time working down toil that accumulates with both the scale of service adoption and the advancement of the software.

A part of the accumulating toil while the service adoption grows is the number of alerts an operations team gets if the alerting strategy isn’t ready for scaling. If you’re maintaining software that creates one alert per day per tenant, keeping one engineer busy running 10 tenants, you will need to scale the number of on-call engineers linearly with the number of tenants the team operates. That means in order to double the number of tenants, you need to double the number of engineers dedicated to reacting to alerts. These engineers will effectively not be able to work on reducing the toil created by the alerts while working down the toil and investigating the issues.

In a traditional operations team that runs OpenShift as a development platform for other organizations of the company, onboarding new tenants often is a manual task. It may be initiated by the requesting team to open a ticket that asks for a new OpenShift cluster. Someone from the operations team will pick up the ticket and start creating the requirements, kick off the installer, configure the cluster so the requesting team gets access, etc. A similar process may be set up for turning down clusters when they are not needed anymore. Managing the lifecycle of OpenShift clusters can be a huge source of toil, and as you can see it will naturally scale with the adoption of the service.

In addition to being a toil-packed process, manual lifecycle and configuration management are error-prone. When an engineer runs the same procedure several times during the week, as documented in a team-managed Wiki, chances are they miss an important step or pass a wrong parameter to any of the scripts, resulting in a broken state that may not be discovered immediately.

When managing multiple OpenShift clusters, having one that is slightly different from the others due to a mistake in the provisioning or configuration process, or even due to a customer request is dangerous and usually generates more toil. Automation that the team generated over time may not be tailored to the specifics of a single snowflake cluster. Running that automation may just not be possible, causing more toil for the operations team. In the worst case, it may even render the cluster unusable.

Automation in a traditional ops team can often be found in a central repository, that can be checked out on engineer devices so they can run the scripts they need as part of working on a documented process. This is problematic not only because it still needs manual interaction and hence doesn’t scale well, but also do engineer devices often be configured differently. They can differ in the OS they use, adding the need to support different vendors in the tooling, e.g. by providing a standardized environment like a container environment to run the automation.

But even then, the version of the scripts to run may differ from engineer to engineer or the script to run hasn’t been updated when it should’ve been as a new version of OpenShift has been released. Automated testing is something that is seldomly implemented for operations scripts made to quickly get rid of a piece of toil. All this makes automation in scripts that are running on developer machines brittle.

How Site Reliability Engineering Helps

In an SRE team, the goal is to replace such scripts with actual software that is versioned properly, has a mature release strategy, a continuous integration and delivery process, and runs from the latest released version on dedicated machines, e.g. an OpenShift cluster.

OpenShift SRE teams treat the operations of OpenShift clusters, from setting them up to tearing them down as a software problem. By applying evolved best practices from the software engineering world to cluster operations, many of the problems mentioned earlier can be solved. The software can be unit-tested to ensure new changes won’t break existing behavior. Additionally, a set of integration tests can ensure it works as expected even when the environment changes, e.g. when a new version of OpenShift is released.

Instead of proactively reacting to more and more requests from customers as the service adoption grows, the SRE team can provide a self-service process that can be used by their customers to provision and configure their clusters. This also reduces the risk of snowflakes, as there is less manual interaction needed by the SRE team. What can and cannot be configured should be part of the UI provided to the customer, so requests to treat a single cluster differently from all the others should turn into a feature request for the automation or UI. That way, it will end up as a supported state rather than a manual configuration update.

To ensure the alerting strategy can scale, SRE teams usually move from a cause-based alerting strategy towards a symptom-based alerting, ensuring that only problems that risk impacting the user experience reach their pager. Smaller problems that do not need to be resolved immediately can move to a ticket queue to work on as time allows.

Shifting towards an SRE culture means allowing people to watch their own software taking away the operations burden from the team one step at a time. It’s a shift that will take time, but it’s a rewarding process. It will turn a team that runs software someone else wrote into a team that writes and runs software they’re writing themselves, with the goal of automating the lifecycle and operations of the software under their control. An SRE culture enables service growth by true automation and observation of customer experience rather than the internal state.

OpenShift as a Tool for Site Reliability Engineers

This book will help you to utilize the tools that are already included with OpenShift or can be installed with only little effort to operate software and OpenShift itself the SRE way.

It will show you the different options for installing OpenShift, helping you to automate the lifecycle of OpenShift clusters as needed. Lifecycle management doesn’t only include installing and tearing down clusters but also managing the configuration of your OpenShift cluster in a GitOps fashion. Even if you need to manage the configuration of multiple clusters, you can use Argo CD on OpenShift to manage the configuration of a multitude of OpenShift clusters.

This book will show you how to run workloads on OpenShift using a simple example application. You can use this example to walk through the chapters and try out the code samples. However, you should be able to use the same patterns to deploy more serious software, like automation that you built to manage OpenShift resources, for example, an OpenShift operator.

OpenShift also provides you the tools you need to automate the build and deployment of your software, from simple automated container builds, whenever you check in a new change to version control, to full-fledged custom pipelines using OpenShift Pipelines.

In addition to automation, the SRE way of managing OpenShift clusters includes proper alerting that allows you to scale. OpenShift comes with a lot of built-in alerts, that you can use to get informed when something goes wrong with a cluster. This book will help you to understand the severity levels of those alerts and how to build your own alerts, based on metrics that are available in the OpenShift built-in monitoring stack.

Working as OpenShift Site Reliability Engineers at Red Hat together for more than two years, we both learned a lot about all the different kinds of alerts that OpenShift emits, and how to investigate and solve problems. The benefit of working close to OpenShift Engineering is, that we can even contribute to alerts in OpenShift if we find problems with them during our work.

Over time, a number of people have reached out, being interested in how we work as a team of SREs. We realize there is a growing interest in all different topics related to our work: From how we operate OpenShift to building custom operators, people show interest in the topic at conferences or reach out to us directly.

This book aims to help you take some of our learnings and use them to run OpenShift in your specific environment. We believe that OpenShift is a great distribution of Kubernetes that brings a lot of additional comfort with it. Comfort that will allow you to get started quickly and strive at operating OpenShift.

Individual Challenges for Site Reliability Engineering Teams

OpenShift comes with a lot of tools that can help you in many situations as a developer or operator. This book can only cover a few of those tools and does not aim to provide a full overview of all features of OpenShift. Instead of trying to replicate the OpenShift documentation, it focuses on highlighting the things which we think will help you get started operating OpenShift. With more and more features being developed and added to OpenShift over time, it is a good idea to follow OpenShift blog and the OpenShift documentation for a more holistic view of what’s included in a given release.

Many of the tools this book covers are under active development, so you may find them behaving slightly differently from how they worked when this book was published. The goal of this book is not to replace the documentation of any of the components or addons of OpenShift. Each section will reference the documentation for a more detailed explanation of how to use a specific component. These documentations are usually updated frequently, so you can find up-to-date information there.

When you use Kubernetes as a platform, you probably are aware that many things are automated for you already: You only need to tell how many resources you need in your deployment, Kubernetes will care for finding a node to place it. You don’t need to do a rolling upgrade of a new version of your software manually, Kubernetes can handle that for you if you tell it your preferred strategy.

OpenShift, being based on Kubernetes, adds some more convenience, like routing traffic to your webservice from the outside world: Exposing your service at a specific DNS name and routing traffic to the right place is done via the OpenShift router.

These are only a few examples of tasks that used to be done by operations personnel but can be automated in OpenShift by default.

However, depending on your specific needs and the environment you’re running OpenShift in, there are most probably some very specific tasks that you need to solve on your own. This book cannot tell you step-by-step what you need to do in order to fully automate operations. If it was that easy to fit every environment, it would most probably be part of OpenShift already. So, please treat this book as an informing guideline, but know that you will still need to solve some of the problems to make OpenShift fit your operations strategy.

Part of your strategy will be to decide how and where you want to install OpenShift. Do you want to use one of the public cloud providers? While this may be the easiest to achieve, you probably need OpenShift to run in your own data center.

The first step towards operating OpenShift is setting it up, and when you find yourself in a place where you’ll need to run multiple of those, you probably want to automate this part of the cluster lifecycle. Chapter 2 gives you an overview of different ways to install an OpenShift cluster, from running it on a developer machine, which can be helpful to develop software that needs a running OpenShift cluster during development, to a public reachable OpenShift deployment using a public cloud provider.

Chapter 2. Installing OpenShift

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the development editor at ccollins@oreilly.com.

As with any piece of software, the story of OpenShift starts by installing it. You need to get going somehow. This chapter walks you through some possible scenarios and go from small to scale in the way that we install. We also talk about what scale actually means and how to plan for it.

While we understand that certain aspects of what we are going to talk about in this chapter and throughout the book may seem abstract to you. In order to make it easier for you to relate, we provide you with a real world scenario.

The example is a startup game developer specialized in building and distributing arcade games. As a growing company, they will have ever-changing needs. This example allows this book to illustrate the different scaling models using a small “real world” business.

OKD, OCP and Other Distributions

OpenShift can be considered as distribution of Kubernetes, it itself is available in different ways. We are going to go over each of them in this section and draw a small comparison and point out how they relate to one another.

OKD

You are probably wondering “What does that stand for?”. The answer is simple while it’s not at the same time. OKD is not an acronym. Before a rebranding, OKD used to be called “OpenShift Origin”. Now it’s OKD and, that is how it is supposed to be referred to, for trademark reasons. Namely, the Linux Foundation does not allow Red Hat to use “Kubernetes” in products or projects further the referencing it.

“OKD is a distribution of Kubernetes optimized for continuous application development and multi-tenant deployment. OKD also serves as the upstream code base upon which Red Hat OpenShift Online and Red Hat OpenShift Container Platform are built.”1 To lay that out a bit, OKD is where upstream Kubernetes is vendored and the core of OpenShift starts to exist, and it serves as the base for everything else that is OpenShift.

OCP

OCP stands for OpenShift Container Platform. This is what most commonly people (especially inside Red Hat) refer to when “OpenShift” is mentioned. It is positioned downstream of OKD and includes all features that you know (or will get to know throughout the course of this book). There is a set of different support levels available like for most products, but we don’t want to turn this into a sales pitch. What you need to know however, is that you can try it out for free during an evaluation period. All you need in advance is a Red Hat account.

OCP is what we will refer to in this book as well. Unless explicitly called out, if there is a difference between how OCP and OKD work, we will default to OCP.

OSD, ROSA, ARO

In addition to a self-hosted and self-installed OpenShift, Red Hat also offers Openshift-as-a-Service on a variety of different platforms. Those are fully managed offerings on AWS, Azure and GCP. We will not go into much detail with those, as you wouldn’t really need to read this book if you were to buy a subscription for any of those, but let’s still cover the naming real quick for future reference:

Acronym

Name

Available On

OSD

OpenShift Dedicated

AWS, GCP

ROSA

Red Hat OpenShift Service on AWS

AWS

ARO

Azure Red Hat OpenShift

Azure

Summary

To conclude this small section, we can say that all of those a viable options for anyone who wants to run production workloads on OpenShift as they are all very closely connected to one another with direct dependencies like

OKD ⇒ OCP ⇒ OSD,ROSA, ARO

Long story short, it depends on your needs in terms of support, environment, ease of use, ease of operation as well as cost per cluster which one you decide to go with and each has its place. We have decided to default to OCP for this book as we find it to be nicely placed in the middle (not just literally): It is more feature complete than OKD, offers support but not to a level of a fully managed solution, like OSD or the other managed solutions would. One important aspect mentioned before, should be clarified again at this point: since you can create a cluster during your evaluation period, there is no cost to follow along any of the examples or exercises in this book, other than those invoked by your cloud provider.

Local Clusters with CodeReady Containers

Before we dive into all the fun and joy of OpenShift, how about we take a look at a possibility for you to run an OpenShift cluster locally. Let’s look at CodeReady Containers.

CodeReady Containers or in short CRC is, by the time of writing this book, the easiest way to launch a full OpenShift cluster locally. If you have touched Kubernetes before, you have probably heard of Minikube2 and CRC is in very simple words the OpenShift equivalent.

Of course this isn’t really much of an explanation, so what is CRC?

It’s developers describe it as “OpenShift 4 on your laptop” 3 and that is what it comes down to. You deploy a local OpenShift cluster. Opposing the neat statement, you can install it not just on laptops but almost everywhere: Workstations, Cloud VMs or in fact, Laptops. At it’s core CRC is a virtual machine, that serves as both OpenShift Worker and Master. Now that we know what we are dealing with, the logical next step is, to kick the tires with it.

Real World Scenario

Here we are at the very beginning of the arcade game developer. Their team is looking at evaluating different options on how to proceed. They have looked at Kubernetes by running Minikube locally ,but they are looking for a more enterprise-ready option and thus have started to read on OpenShift. Now it’s time for them to play around and start the first evaluation. They are looking for an easy option, which in the best case scenario, is cheap as well. After all the arcade gaming company is still just a small startup and not a multi-million dollar company (yet).

Getting started with CodeReady Containers

WARNING

CodeReady Containers are ephemeral by nature and should not be used for production use cases.

First things first: the documentation4 is your best friend. Make sure to consult it whenever you get stuck. It is the condensed start to finish guide for CRC, and it’s Open Source like most things coming out of Red Hat. What this means for you, is that it’s frequently update, and even you can contribute to it, in case you find something along the way, that you think isn’t covered enough yet.

After we got that out of the way, head on over to Open Cluster Manager. We will reference this page a couple of times throughout the cheaper, specifically when we are going to talk about the installers. It serves as your single pane of glass for all clusters, that you registered, regardless if they are CRC, OCP or managed clusters.

Sign in with your Red Hat account. If you don’t have one, just go ahead and create one. Once you have done that, you should be presented with a view similar to the one below. Unless you create a lot of clusters before or work at Red Hat your view will hold significantly fewer clusters, but the general idea is the same.

Click the "Create cluster" button and then choose “Local” in the next view.

You will end up at the CRC pane.

Now you will just choose the platform that we want to install CRC on. Note that it has your current platform auto-selected, based on your browser’s user agent. So in my case in our example, we are on macOS, and it is auto-selected.

Once that is done, we can download the archive by clicking "Download CodeReady Containers“.

While you wait for download to finish, also make sure to download and save your “Pull secret”.

After finishing the download, you can now extract the archive into any location that is in your $PATH.

A possible command for this is

$ tar -xJvf crc-macos-amd64.tar.xz

Since you have extracted into your $PATH, you will now be able to just the included binaries right away. Namely, you get the crc binary as well as oc.

Those two together allow you to effectively set up and manage your CRC cluster, as well as to interact with it afterward like you would interact with any other OpenShift cluster.

The basic interaction with your cluster will be to set it up at first. This can be done as follows

$ crc setup

CodeReady Containers is constantly improving and we would like to know more about usage (more details at https://developers.redhat.com/article/tool-data-collection)

Your preference can be changed manually if desired using 'crc config set consent-telemetry <yes/no>'

Would you like to contribute anonymous usage statistics? [y/N]: y

Thanks for helping us! You can disable telemetry with the command 'crc config set consent-telemetry no'.

INFO Checking if running as non-root

INFO Checking if podman remote executable is cached

INFO Checking if admin-helper executable is cached

INFO Caching admin-helper executable

INFO Using root access: Changing ownership of /Users/rrackow/.crc/bin/admin-helper-darwin

Password:

INFO Using root access: Setting suid for /Users/rrackow/.crc/bin/admin-helper-darwin

INFO Checking minimum RAM requirements

INFO Checking if HyperKit is installed

INFO Setting up virtualization with HyperKit

INFO Using root access: Changing ownership of /Users/rrackow/.crc/bin/hyperkit

INFO Using root access: Setting suid for /Users/rrackow/.crc/bin/hyperkit

INFO Checking if crc-driver-hyperkit is installed

INFO Installing crc-machine-hyperkit

INFO Using root access: Changing ownership of /Users/rrackow/.crc/bin/crc-driver-hyperkit

INFO Using root access: Setting suid for /Users/rrackow/.crc/bin/crc-driver-hyperkit

INFO Checking file permissions for /etc/hosts

INFO Checking if CodeReady Containers daemon is running

INFO Checking file permissions for /etc/resolver/testing

INFO Setting file permissions for /etc/resolver/testing

INFO Using root access: Creating dir /etc/resolver

INFO Using root access: Creating file /etc/resolver/testing

INFO Using root access: Changing ownership of /etc/resolver/testing

INFO Checking if CRC bundle is extracted in '$HOME/.crc'

INFO Checking if /Users/rrackow/.crc/cache/crc_hyperkit_4.7.5.crcbundle exists

INFO Extracting bundle from the CRC executable

INFO Ensuring directory /Users/rrackow/.crc/cache exists

INFO Extracting embedded bundle crc_hyperkit_4.7.5.crcbundle to /Users/rrackow/.crc/cache

INFO Uncompressing crc_hyperkit_4.7.5.crcbundle

crc.qcow2: 10.13 GiB / 10.13 GiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00%

Your system is correctly setup for using CodeReady Containers, you can now run 'crc start' to start the OpenShift cluster

The output if just a bunch of info-level logging. The only thing that is really noteworthy is that you opt in or out to sending telemetry data.

This is a very limited set of on-cluster data that gets forwarded to Red Hat. Luckily even that bit is open source, so that you can actually see the full list of what gets send online5 and there is nothing particularly sensitive in there, in case you are worried about that.

NOTE

Opting out of sending telemetry data can impact certain features in OpenShift Cluster Manager, that use said telemetry data.

Now that the setup is done, we can go ahead and launch our cluster. It is as simple as running one command.

$ crc start

INFO Checking if running as non-root

INFO Checking if podman remote executable is cached

INFO Checking if admin-helper executable is cached

INFO Checking minimum RAM requirements

INFO Checking if HyperKit is installed

INFO Checking if crc-driver-hyperkit is installed

INFO Checking file permissions for /etc/hosts

INFO Checking file permissions for /etc/resolver/testing

CodeReady Containers requires a pull secret to download content from Red Hat.

You can copy it from the Pull Secret section of https://cloud.redhat.com/openshift/create/local.

? Please enter the pull secret

At this point, paste the content of the pull secret you downloaded earlier.

The pull secret will allow you to pull the required images from Red Hats container registry as well as associate the cluster to your Red Hat user, which ultimately also will make it show up in OpenShift Cluster Manager.

The process will go on and complete. After that you have an up and running OpenShift cluster on your local machine. Congrats!

This should be it for the installation part of CRC since there are a lot of different possibilities as to how to go from here: you can use the oc on the command line or the web console and there a many ways to authenticate. All of them are covered in the official documentation6. However, CRC is only the first way to dip your toe in the waters of OpenShift, and we want to render our focus on production environments. We do still encourage you to play around with CRC just to familiarize yourself more with the oc command line tool as well as the web console. Remember that after all this is your easiest way to get toe to toe with OpenShift and how it feels and works.

Planning Cluster Size

Now that our first experiment with OpenShift has been a success, it is about time to go ahead and deploy our first cluster to production. Before doing so, there are some considerations and one of the most important ones is planning the cluster’s size and capacity.

Real World Scenario

The developers of the arcade gaming company have successfully experimented with a local cluster using CRC and are ready to move ahead. OpenShift is going to be its Kubernetes distribution of choice. Now a small startup has does not have unlimited financial capabilities and as such needs to carefully consolidate its resources. The developers of the arcade game company want to ensure that their cluster has all the capacity it needs to deploy its workloads, while at the same time making sure that they plan for success. Meaning that they want to be able to scale in the future without wasting crucial time to restructure their whole setup.

Instance Sizing Recommendations

OpenShift documentation has some pointers as to how to scale your clusters’ Instances. We want to take a look this and shine some light as to what you want to be looking out most, what potential issues you can run into if you scale too small. We can safely assume that scaling too big is not an issue, other than cost , and we don’t want to get too deep into this topic, however we will make some small remarks for cost comparison.

The instance size is directly related to your workloads and master and nodes behave similarly to some extent. Meaning, that the more workload you plan to run, the bigger your instances have to become. However, the way they scale is fundamentally different. While nodes directly relate to workload almost linearly , master behave rather differently. Luckily, we need to add. Image you had to scale your master nodes linearly with the workloads you want to run on cluster and how that would impact your cost.

Node Sizing Recommendations

We mentioned before already that nodes are almost scaled linearly alongside workloads that you want to run on your cluster. To better understand that, let’s look at an example.

Think of a cluster of 3 Nodes, ignore the masters for now, and each of them is an AWS m5.xlarge so 4 vCPU and 8GB of ram. That gives us a total cluster capacity of 12 vCPU and 24GB ram. In this virtual scenario we can try to run workloads that in perfect distribution use all the resources up, and then we will either need to scale nodes to bigger instances (horizontally) or more of them (vertically). Add another instance and our cluster capacity grows linearly. Now we have 16 vCPU and 32GB for our workloads.

So far things are relatively straight forward. The above scenario disregards a small but important detail that we need to keep in mind: system reserved and kube reserved capacity. Luckily this isn’t all that much to worry about. Since OpenShift release 4.8 we can let OpenShift do the heavy lifting for us by automatically scale the reserve as needed. This is done by adding a little portion to our KubeletConfig

Example 2-1. Machine Configuration Yaml

apiVersion: machineconfiguration.openshift.io/v1

kind: KubeletConfig

metadata:

name: dynamic-node

spec:

autoSizingReserved: true

If you already have an OpenShift cluster and still want to benefit from that ease, it is possible to adjust the setting post-install. This is taking a lot of pain away from cluster operators. Before you were forced to make a compromise between getting the most out of your nodes and also making them resilient. Then scale this out, because the kube-system requires more resources as you scale out the cluster as a whole and also with more pods per node. Now we just apply this setting and let OpenShift do the rest of it for us. A first step in that direction had happened before in OpenShift 4.7 with OpenShift already reserving half of a CPU core by default. This goes to show, that stability of OpenShift cannot be guaranteed without at least a certain set of resources being reserved.

Think of it like this: you have 10 pods running on that m5.xlarge node and each of those pods has a requested set of resources of 0.4 CPU, and they actually use that. Just naturally your system process get into trouble and that node gets unstable. Now the worst case, the node becomes unresponsive and crashes, the workloads on it get reallocated to other nodes and overloading those and so on. You end up with a chain reaction and your whole cluster becoming unresponsive. From that perspective it’s a small price to pay to sacrifice some of that precious capacity to ensure cluster stability.

The learning here is to absolutely use the automatic capacity reservation.

So we know that nodes scale linearly with their workloads and that we need to add a bit of reserved capacity on top of that. The missing bit here is the fundamental question of “How big should my node be?”. It depends. Probably not a particularly helpful answer so let’s look a bit further into it. We have to consider a small set of questions: - How big is our single biggest workload? - How much can we utilise a big node? - How fast can we deploy more nodes?

Our single biggest workload determines the minimum size of a node. The explanation is relatively simple: if we can’t fit the workload on a node, we have a problem, because we obviously want to be able to deploy all our workloads to our cluster.

The contrary to that is, the efficiency that we want to achieve. Having a node idle around at only 50% usage all the time, is really just burning money. We want to find the sweet spot between being able to fit all our workloads, while at the same time making the most of our nodes. Those two points together lead to the point that we want to be using nodes as small as possible and if we need more, deploy another one, so the utilisation per node is still high even with an extra node added to the cluster.

The factor that can make us go down a different path, is time. The time it takes for us to deploy another node, in case we hit capacity. We will talk about deployment models for OpenShift later in this chapter, but we need to quickly touch it here already. There are certain ways to deploy, that are faster than others. For example having set up automation to a certain degree, that allows us to deploy another node to our cluster within 5 minutes, makes a great difference from having to manually provision a new blade in our datacenter and waiting for it for a day until the datacenter team has mounted and connected it, and we’re ready to go with it.

The rule of thumb here is the slower you can provision new nodes, the bigger a single node needs to be and the earlier you have to provision new nodes. The time to new node directly works against the max utilisation you want to aim for per node.

Master Sizing Recommendations

Nodes are important for giving a home to your workloads, but what masters are where the heart of OpenShift is at.

The masters or control plane nodes, are what keeps the cluster running. They are hosting.

- etcd

- API server (kube and OpenShift)

- controller manager (kube and OpenShift)

- OpenShift Oauth API server

- OpenShift Oauth Server

- HA proxy

HA proxy needs to get a special remark. This is only the case, if we decide to use OpenShift’s default router configuration. There are other options available, but for the majority of use cases and scenarios HA Proxy is just fine, so let’s take that as a granted default option.

Back to the original topic, with the masters not directly running workloads, they behave differently when it comes to scalability. As opposed to the linear scalability needs of nodes, that are dependent on the workloads, the master capacity has to be scaled alongside the number of nodes.

Another difference to the node scalability is that we need to look at vertical scaling over horizontal scaling. Hence, there is no master count mentioned in the above chart. The reason is that we cannot simply scale out master nodes horizontally because some components that run on masters require a quorum as well as replication. The most prominent case is etcd. The central store for state, secrets, etc. (no pun intended) is just one of the components to name. We theoretically could use almost any arbitrary number of masters in our OpenShift cluster as long they can form quorum, which on its own, can become rather tricky for example with an even number of nodes like “4” or “2”. With those two options out of the way, the question is “Why not just 1?” and the answer to that is the clusters’ resilience. We cannot risk our whole clusters, which is basically unusable without masters, by having only a single point of failure. Imagine a scenario where you had one master instance, and it crashes because of a failure in the underlying infrastructure. Our whole cluster is completely useless at this point and recovery is hard from that kind of failure. The next smallest option is “3” and that is also our recommendation. In fact the official documentation states that “Exactly three master nodes must be used for all production deployments.” 7.

With the count set, we have the option of vertical scaling left. However, with masters being the heart of the cluster, we have to take into account into what fragile state we take our cluster, when we purposely shut one by one down to adjust their size afterwards. The following charts gives a small insight into how to scale master nodes size as a guidance.

Nodes

CPU

Memory

25

4

16GB

100

8

32GB

250

16

96GB

Make sure to plan for growth. If you plan to have 20 nodes at the very beginning in order to have room for your workloads, choose the next bigger size master instances. This comes at a small price point but will save you massive amount of work and risk by avoiding a master scaling operation.

NOTE

There are a lot of details skipped in this section. What exactly is the kubelet doing? What actions inside etcd are causing a higher resource consumption? Those and others are simply out of scope for this book. The “How does OpenShift work?” could be a book on its own and is therefore not further touched in this book on purpose.

Infra Nodes

We want to briefly touch on this additional concept as it is a practice that we recommend to you, the further along you are on your journey to maturing your installation.

We talked about masters and nodes, so what are “Infra nodes”. The very simplified version is that they are worker nodes with an extra label. Other than that they are just regular OpenShift nodes. So if they’re “just” nodes, why would we give them the extra label? There’s two broader categories of reasons. The cost and the cluster resilience. The easy one is the cost: certain infrastructure workloads don’t cause subscription costs with Red Hat. What that means is, that if you have a node, that exclusively runs infrastructure workloads, you don’t have to pay your subscription fee for that node. Seems like an easy way to safe money. For the sake of completeness the full list of components that don’t require a node subscriptions can be found in the latest documentation8. Some of them are components that run on masters and also need to be there, like the OCP control plane. Others however can be moved around. So we create a new set of nodes, with the infra label.

Reason number two: the clusters’ resiliency. Regular workloads as well as infra workloads don’t make a difference to OpenShift when they’re on the same node. Imagine a regular cluster with just masters and nodes. We deploy all our applications as well as the infra workloads that come out of the box to nodes. Now when the unfortunate situation happens that we run out of resources, it may just as well be that an “infra” workload gets killed as a “regular” application workload. Which is of course not the best situation to be in. On the other hand when all our infrastructure related workloads are safely placed on their own set of nodes, the “regular” applications don’t impact them at all, better resilience and better performance. Good candidates to be moved around are - in-cluster monitoring (configmap) - routers (IngressController) - default registry (Config). Moving them is also relatively simple by adding a label to the corresponding elements, that are noted inside the parenthesis. The following example shows how it is done for the in-cluster monitoring solution.

apiVersion: v1

kind: ConfigMap

metadata:

name: cluster-monitoring-config

namespace: openshift-monitoring

data:

config.yaml: |+

alertmanagerMain:

nodeSelector:

node-role.kubernetes.io/infra: ""

prometheusK8s:

nodeSelector:

node-role.kubernetes.io/infra: ""

prometheusOperator:

nodeSelector:

node-role.kubernetes.io/infra: ""

grafana:

nodeSelector:

node-role.kubernetes.io/infra: ""

k8sPrometheusAdapter:

nodeSelector:

node-role.kubernetes.io/infra: ""

kubeStateMetrics:

nodeSelector:

node-role.kubernetes.io/infra: ""

telemeterClient:

nodeSelector:

node-role.kubernetes.io/infra: ""

openshiftStateMetrics:

nodeSelector:

node-role.kubernetes.io/infra: ""

thanosQuerier:

nodeSelector:

node-role.kubernetes.io/infra: ""

Add that to your already existing configmap or create a new one, with just this. For the latter option, we would create the file above in our favourite editor and apply it like follows:

$ oc create -f cluster-monitoring-configmap.yaml

It will take a hot minute to get done, and we can follow it up with the following command.

$ watch 'oc get pod -n openshift-monitoring -o wide'

A last note on the scaling of infra nodes: They scale almost the same way as master nodes. The reason they need to be scaled vertically in the first place is that Prometheus as part of the in-cluster monitoring solution requires more memory with more metrics it is storing.

NOTE

We will not print a scaling chart at this point, as the official numbers are based on scaling tests and are constantly subject to change, as they are being updated. We don’t want to give you numbers that are almost outdated by the time of this book being released.

Basic OpenShift Installations

In this section we are going to talk about the first way to install an actual production OpenShift cluster. There are essentially two different ways here that come in different shapes but do the same thing, just for your respective infrastructure. In the following we present the real world scenario and then talk about the two main ways. Since we have a whole chapter that is dedicated to advanced installation models, we will only briefly touch on the user-provisioned infrastructure and then come back to it in chapter 10.

NOTE

If you are already advanced beyond this point, feel free to skip this section and look at chapter 10, where we talk about advanced installation methods.

Real World Scenario

While analyzing their needs, the developers of the arcade gaming company have found out that to get started a small cluster will do the job for them, since they assume fast node creation times and low workloads at the beginning. They want to get a cluster with 3 master and 3 nodes in a single AWS region.

NOTE

There is a vast amount of cloud providers and even more bare metal infrastructure options available that we cannot possibly all cover. Therefore, we are only going to look at Google Cloud Platform as an example. This way it will also be easier to compare the two basic installation options.

Installer-provisioned Infrastructure

Think of this as an all-in-one solution. The installer creates the underlying infrastructure, networking infrastructure and OpenShift cluster on the cloud provider of your choice (or compatible bare metal options). Basically you run a single command, pass in your credentials and what you get back is an up and running OpenShift cluster. If that gets you hooked, good, but stick with us for a little longer as we want to explore what is actually happening under the hood as well as what exactly you need to do and walk through an example installation.

We start off at the OpenShift Cluster Manager landing page, that we already know from the CRC section.

Click the "Create cluster" button again, but this time choose your cloud provider of choice, in our case Google Cloud Platform (GCP). This takes us to the next page where we choose “Installer-provisioned infrastructure”. That last bit was relatively obvious, given which section we are in, but better safe than sorry.

This next page should seem somewhat familiar from the CRC section as well. In the first part we have all required artifacts. Part two is giving us the absolute basic installation command and part three contains some minor information about subscriptions. image::images/OCM-create-gcp.png[]

Let’s download the installer by clicking the "Download Installer" button to do just that and while we’re on it also download the pull secret and oc binary. Same thing as for CRC: the distro we currently use is auto-selected, if you require a different one, just choose the appropriate platform from the list.

Unpack the archive with the binaries to somewhere in your $PATH to have easy access to them on the command line. This should look something like below.

$ tar -xzvf openshift-client-mac.tar.gz

x README.md

x oc

x kubectl

Now also unpack the installer in the same way:

$ tar -xzvf openshift-install-mac.tar.gz

x README.md

x openshift-install

You can either move openshift-install into a directory in your $PATH too, in case you want plan to access it rather frequently for example. Otherwise, just keep it in a location that suits you and reference it by absolute or relative path.

In our example we unpacked in the ~/Downloads directory, so we can access the installer as follows:

$ ./Downloads/openshift-install

Pre-requisites

Before we do in fact execute the installer, we need to make sure that we have our cloud provider setup and ready. The installer would have told us at some point, but we know, so we tackle that up front and go ready into the actual installation process. There is a whole section in the documentation9 that only targets the set-up of the pre-requisites, but we want to go over it anyway, just to be sure we have a good overview of what we need.

To begin, we need a project. That can be created from the console or from the CLI by running the following

WARNING

Everything from here on, will invoke costs. Be vary of your budget.

gcloud projects create openshift-guinea-pig

You can create that in an automated way or itself with a service account or in subfolders and so on and so forth, but we will just look at this super simple case as it’s just one example and would look completely different on other cloud providers.

NOTE

Your GCP project must use the Premium Network Service Tier if you are using installer-provisioned infrastructure. The Standard Network Service Tier is not supported for clusters installed using the installation program. The installation program configures internal load balancing for the api-int.<cluster_name>.<base_domain> URL; the Premium Tier is required for internal load balancing.

In the project we just created, we also need a certain set of APIs to be enabled. Below is a chart to show you which ones, we’re going to need.

API service

Console service name

Compute Engine API

compute.googleapis.com

Google Cloud APIs

cloudapis.googleapis.com

Cloud Resource Manager API

cloudresourcemanager.googleapis.com

Google DNS API

dns.googleapis.com

IAM Service Account Credentials API

iamcredentials.googleapis.com

Identity and Access Management (IAM) API

iam.googleapis.com

Service Management API

servicemanagement.googleapis.com

Service Usage API

serviceusage.googleapis.com

Google Cloud Storage JSON API

storage-api.googleapis.com

Cloud Storage

storage-component.googleapis.com

We can leverage the gcloud CLI tool again to enable all of those or any other method that you prefer.

$ gcloud services enable compute.googleapis.com cloudapis.googleapis.com \

cloudresourcemanager.googleapis.com \

dns.googleapis.com \

iamcredentials.googleapis.com \

iam.googleapis.com \

servicemanagement.googleapis.com \

serviceusage.googleapis.com \

storage-api.googleapis.com \

storage-component.googleapis.com

Operation "operations/acf.p2-1044842249801-71a76ac6-67d5-4f6b-9625-91a9fd12a64b" finished successfully.

Make sure that you have enough quota in your project. You can find current required project in the OpenShift documentation.10

Additionally, to that you will also need a dedicated public DNS zone in your project. This needs to be authoritative for the domain. If you don’t have a domain, you can purchase one from your preferred registrar. We shall refrain from dropping names here just because there are a lot of them, and we don’t want to seem like we favour any of them.

Now create the managed zone like this but with your domain

$ gcloud dns managed-zones create ocp-cluster \

--description=openshift-cluster \

--dns-name=operatingopenshift.com \

--visibility=public

Created [https://dns.googleapis.com/dns/v1/projects/innate-attic-182119/managedZones/ocp-cluster].

Get the authoritative name servers from the hosted zone records:

$ gcloud dns managed-zones describe ocp-cluster

creationTime: '2021-04-22T11:13:17.236Z'

description: openshift-cluster

dnsName: operatingopenshift.com.

id: '9171610950957705760'

kind: dns#managedZone

name: ocp-cluster

nameServers:

- ns-cloud-d1.googledomains.com.

- ns-cloud-d2.googledomains.com.

- ns-cloud-d3.googledomains.com.

- ns-cloud-d4.googledomains.com.

visibility: public

The last step here is to point your registrar to the name servers that you just extracted as authoritative.

Now on to the final bit: the service account.

Same as with all the other resources we created, we will use the gcloud CLI.

$ gcloud iam service-accounts create ocp-cluster \

--description="Service account for OCP cluster creation" \

--display-name="OCP_CREATOR"

Created service account [ocp-cluster].

to create the service account. Afterwards we need to assign it some roles in order to get the needed permissions. The list of required permissions is in the documentation11 but if this is not yet your full production deployment, you can also just assign it the “owner” role.

WARNING

Do not do this in a production environment. Follow the path of the least privileges in production or long-lived clusters.

$ gcloud projects add-iam-policy-binding innate-attic-182119 \

--member="serviceAccount:ocp-cluster@innate-attic-182119.iam.gserviceaccount.com" \

--role="roles/owner"

Updated IAM policy for project [innate-attic-182119].

bindings:

- members:

- serviceAccount:ocp-cluster@innate-attic-182119.iam.gserviceaccount.com

role: roles/owner

etag: BwXAjkFSyZw=

version: 1

The last step before we can actually install our cluster, is to get our local environment ready. We need an ssh keypair that we can later on also use to access the instances, as well as the service account credentials of the account that we created previously.

Create a key-pair and add it to your ssh-agent (after you enabled the agent).

$  ssh-keygen -t ed25519 -N ''

Generating public/private ed25519 key pair.

Enter file in which to save the key (/Users/rrackow/.ssh/id_ed25519):

Your identification has been saved in /Users/rrackow/.ssh/id_ed25519.

Your public key has been saved in /Users/rrackow/.ssh/id_ed25519.pub.

The key fingerprint is:

SHA256:c0y9aLQMnv6lBd51Hdrw4q4muNwAeExxdWvauvhwTtk rrackow@MacBook-Pro

The key's randomart image is:

+--[ED25519 256]--+

|     . ... .     |

|      o   ...    |

|     .  . oo.. . |

|    +  . B+o .= o|

|   . +  S.O..o.oo|

|    . .. =+o.... |

|       oo=.E+.   |

|      ..Oo.=.    |

|       +o==...   |

+----[SHA256]-----+

$ eval "$(ssh-agent -s)"

Agent pid 49003

$ ssh-add /Users/rrackow/.ssh/id_ed25519

Identity added: /Users/rrackow/.ssh/id_ed25519 (rrackow@MacBook-Pro)

Now create a key-file and download it. Once that is done, export its path.

$ gcloud iam service-accounts keys create servicce-account-keys \

--iam-account=ocp-cluster@innate-attic-182119.iam.gserviceaccount.com

created key [b8879741ba8850edcadd9840996e882adc05e228] of type [json] as [servicce-account-keys] for [ocp-cluster@innate-attic-182119.iam.gserviceaccount.com]

$ export GOOGLE_APPLICATION_CREDENTIALS='~/service-account-keys'

Now we have everything together and can get into the actual installation of our OCP cluster.

Installation

We are going to pick this up, where we left of: running the installer. The installer, if you don’t pass in any arguments, works in an interactive mode, which looks something like the following, where it will prompt you for choices, and you can move around with the arrow keys and make an appropriate selection with the return key.

$ ./Downloads/openshift-install create cluster --dir='ocp-cluster-install'

? SSH Public Key  [Use arrows to move, enter to select, type to filter, ? for more help]

> /Users/rrackow/.ssh/id_ed25519.pub

/Users/rrackow/.ssh/libra.pub

/Users/rrackow/.ssh/openshift-gcp.pub

/Users/rrackow/.ssh/rpi-ocp-discovery.pub

/Users/rrackow/.ssh/rrackow_private.pub

/Users/rrackow/.ssh/rrackow_redhat_rsa.pub

<none>

? Platform  [Use arrows to move, enter to select, type to filter, ? for more help]

aws

azure

> gcp

openstack

ovirt

vsphere

INFO Credentials loaded from file "/Users/rrackow/.gcp/osServiceAccount.json"

? Project ID  [Use arrows to move, enter to select, type to filter, ? for more help]

> openshift-guinea-pig (innate-attic-182119)

? Region  [Use arrows to move, enter to select, type to filter, ? for more help]

europe-west6 (Zürich, Switzerland)

northamerica-northeast1 (Montréal, Québec, Canada)

southamerica-east1 (São Paulo, Brazil)

> us-central1 (Council Bluffs, Iowa, USA)

us-east1 (Moncks Corner, South Carolina, USA)

us-east4 (Ashburn, Northern Virginia, USA)

us-west1 (The Dalles, Oregon, USA)

? Base Domain  [Use arrows to move, enter to select, type to filter, ? for more help]

> operatingopenshift.com

rackow.io

? Cluster Name ocp-cluster

? Pull Secret [? for help] *****************

INFO Creating infrastructure resources...

INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp-cluster.operatingopenshift.com:6443...

INFO Install complete!

INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/myuser/install_dir/auth/kubeconfig'

INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp-cluster.operatingopenshift.com

INFO Login to the console with user: "kubeadmin", and password: "4vYBz-Ee6gm-ymBZj-Wt5AL"

INFO Time elapsed: 36m22s

NOTE

You don’t have to write the credentials down, as you can find them in your install dir for example ocp-cluster-install/.openshift_install.log.

Each option will collapse once you made a selection, so don’t be confused if it looks slightly different for you. The last two require a manual input.

After you made your last selection, the installer will work its magic. This commonly takes around 45 minutes.

Self-provisioned Infrastructure

If the fully automated installation is not your jam, we can of course also install on our infrastructure. The advantages are pretty obvious. You are in full control of absolutely everything. This also allows for a better incorporation in any sort of pipeline. Imagine you ran a pipeline, with just an create cluster command, and it fails at some point. Probably not very pretty to sort out what went wrong and even worse to actually automate error handling.

1   https://docs.okd.io/

2   https://minikube.sigs.k8s.io/docs/

3   https://github.com/code-ready/crc#red-hat-codeready-containers---openshift-4-on-your-laptop

4   https://code-ready.github.io/crc/

5   https://github.com/openshift/cluster-monitoring-operator/blob/master/Documentation/timeseries.txt

6   https://code-ready.github.io/crc/#accessing-the-openshift-cluster_gsg

7   https://docs.openshift.com/container-platform/4.7/architecture/control-plane.html#defining-masters_control-plane

8   https://docs.openshift.com/container-platform/4.7/scalability_and_performance/recommended-host-practices.html#infrastructure-components_

9   https://docs.openshift.com/container-platform/4.7/installing/installing_gcp/installing-gcp-account.html

10   https://docs.openshift.com/container-platform/4.7/installing/installing_gcp/installing-gcp-account.html#installation-gcp-limits_installing-gcp-account

11   https://docs.openshift.com/container-platform/4.7/installing/installing_gcp/installing-gcp-account.html#installation-gcp-permissions_installing-gcp-account

Chapter 3. Running Workloads on OpenShift

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the development editor at ccollins@oreilly.com.

At this point you should already have an OpenShift cluster that you can use to deploy applications. It may be a cluster running on VMs provisioned by a cloud provider or even a small cluster on your notebook using CodeReady Containers (CRC). You can access the console and are able to log in to the cluster with the oc command-line utility. But how do you get an application that your team built deployed to the cluster?

Most applications running on OpenShift clusters are web-based. Such applications are usually accessed by users via a web browser, or as back-ends by apps installed to user-owned devices. For the sake of this chapter you can use an arranged deployment consisting of three different services to practice deploying application code to your OpenShift cluster. A small CRC cluster should provide enough capacity to deploy this application. However, to follow some parts of the chapter you will need a cluster that is accessible externally.

The application used in this chapter is the arcade gaming platform of a fictitious game publisher. It consists of the following components:

- Games, each running in its own service (for now there is only one game)

- A highscore service where the scores of every game and player can be shown

- The platform service, used as entry point where customers can browse, start, and purchase games.

Figure 3-1 gives you an overview of the involved components and how they interact.

Figure 3-1. Components of the arcade platform example application.

The code is organized in a git repository on GitHub, where each developer of the company can contribute to every service when necessary. All three services of this small sample application are located in the same git repository. This is so you only need to look at one repository and do not need to clone several different ones. The code from this example is used in all of the following sections. If you want to follow along with this example code, use the following command to check out the latest version:

$ git clone https://github.com/OperatingOpenShift/s3e

Deploying Code

To have all services you want to run on your OpenShift cluster contained in the same namespace, first create a new project:

$ oc new-project arcade

To have all further commands target the created project without mentioning it in every command, use the following command to switch to the “arcade” project:

$ oc project arcade

A project in OpenShift is a namespace with additional annotations. In most cases the differentiation between project and namespace is not relevant for the examples in this book, so the two terms are mostly interchangeable.

Instead of running the oc project command before subsequent commands you can also execute all the commands against a certain namespace by selecting the namespace in each command. All oc commands support the -n flag, which can be used to specify a namespace to run the command in.

In practice, when you know you’ll execute a number of commands against the same namespace, switching to it using oc project saves you some typing time and also saves you from executing commands against the “default” namespace and wondering where all your resources went.

Deploying Existing Container Images

The quickest way to start a container in the new project is using oc run. Since the game service of the application you want to deploy is already built into a container image, you can start it on the cluster using the following command:

$ oc run game --image=quay.io/mdewald/s3e

pod/game created

This will spin up a new pod on the cluster. Use the following command to observe it while it’s starting up. As soon as it’s ready you should see the status “Running”:

$ oc get pods

NAME   READY   STATUS    RESTARTS   AGE

game   1/1     Running   0          24s

At this point you’re probably curious to take a look at the game you just deployed. However, the oc run command just spins up a pod without an exposed endpoint so you need to find a way to access the game UI (which is exposed at port 8080 in this container image). A quick and simple approach to see the UI is working is to forward the port from the container to your local machine. To do so, run the following command:

$ oc port-forward game 8080

Forwarding from 127.0.0.1:8080 -> 8080

Forwarding from [::1]:8080 -> 8080

Visiting http://localhost:8080/s3e in a web browser should provide you with a simple, yet familiar game UI shown in Figure 3-2. To stop the port forwarding, simply stop the running process by pressing Ctrl+C.

Figure 3-2. Example game, accessed using port-forwarding.

NOTE

The webserver configuration specifies the location /s3e for the game. This is necessary to achieve the service locations shown in the service landscape in Figure 3-1.

Creating Deployments

While oc run is a quick and easy way to verify that the cluster can access your built container image, it is not the best way to run an application on your cluster. The standard way to deploy an application is a deployment resource. Deployments provide additional features to plain pods. For example, they can be used for rolling upgrades, or to run multiple instances distributed across nodes. To create a Deployment with the same container image, run oc create deployment and oc get pods to observe the pod coming up:

$ oc create deployment game --image=quay.io/mdewald/s3e

deployment.apps/game created

$ oc get pods

NAME                   READY   STATUS    RESTARTS   AGE

game                   1/1     Running   0          13m

game-c6fb95cc6-bk6zp   1/1     Running   0          78s

Security Context Constraints

When deploying a container using oc create deployment the pod will run with different parameters. One difference is the annotation openshift.io/scc. Compare the output of the following two commands, adjusted to the pod generated for your deployment:

$ oc get pod game \

-o "jsonpath={.metadata.annotations['openshift\.io/scc']}"

anyuid

$ oc get pod game-c6fb95cc6-bk6zp \

-o "jsonpath={.metadata.annotations['openshift\.io/scc']}"

restricted

The restricted security context constraint (SCC) means the pods of this deployment will not be able to run privileged containers or mount host directories, and containers must use a UID from the allowed range. That means, for applications running a web server (in this example, NGINX), that they need to be configured accordingly. They cannot run on port 80 or specify a UID that will be mapped automatically to a high UID within the range configured by the project.

See the NGINX documentation for an explanation on how to configure NGINX to serve on a specific port.

You can now scale the game Deployment using oc scale deployment. You will see additional pods coming up immediately.

$ oc scale deployment game --replicas=3

deployment.apps/game scaled

$ oc get pods

NAME                   READY   STATUS              RESTARTS   AGE

game                   1/1     Running             0          16m

game-c6fb95cc6-bk6zp   1/1     Running             0          3m24s

game-c6fb95cc6-bmxzd   0/1     ContainerCreating   0          3s

game-c6fb95cc6-q8bp8   0/1     ContainerCreating   0          3s

To access those different instances, you need to create a service resource and tell it to expose port 8080 from your pods. To create the service run the following command:

$ oc expose deployment game --port=8080

service/game exposed

$ oc get service

NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE

game   ClusterIP   172.25.113.82   <none>        8080/TCP   6s

$ oc get endpoints

NAME   ENDPOINTS                                            AGE

game   10.116.0.57:8080,10.116.0.59:8080,10.116.0.60:8080   22s

As you can see from the output of oc get endpoints, OpenShift has registered 3 different endpoints for the service, one for each instance running. To test the connection, you can again forward port 8080 to localhost, this time using the service instead of the pod:

$ oc port-forward service/game 8080

Forwarding from 127.0.0.1:8080 -> 8080

Forwarding from [::1]:8080 -> 8080

To get the second service of the arcade platform application deployed, repeat the preceding steps for the platform service:

$ oc create deployment platform --image=quay.io/mdewald/s3e-platform

$ oc expose deployment platform --port=8080

Use port-forwarding again to check if the service is accepting requests:

$ oc port-forward service/platform 8080

As you have probably already realized, this is not how your users would want to access your service. Before we dive into exposing the services to the outside of the cluster in “Accessing Deployed Services”, the following section takes a look at a third way to deploy your application.

Deploying Applications from git Repositories

The arcade platform contains a service that collects the scores per user of all games. The service is written in Go and can be found in the highscore subfolder of the git repository. In order to deploy this service, this example does not use an already existing image from a container registry but instead uses OpenShift’s built-in build infrastructure.

To deploy the application right from the git repository, run the following command:

$ oc new-app https://github.com/OperatingOpenShift/s3e \

--context-dir=highscore

--name=highscore

--> Found container image 28f6e27 (13 days old) from Docker Hub for

"alpine:latest"

* An image stream tag will be created as "alpine:latest" that will track

the source image

* A Docker build using source code from https://github.com/OperatingOpenShift/s3e

will be created

* The resulting image will be pushed to image stream tag

"highscore:latest"

* Every time "alpine:latest" changes a new build will be triggered

--> Creating resources ...

imagestream.image.openshift.io "alpine" created

imagestream.image.openshift.io "highscore" created

buildconfig.build.openshift.io "highscore" created

deployment.apps "highscore" created

service "highscore" created

[...]

Git repository containing the application

Subfolder in the repository to deploy

Name of the application used in resources

Resources created for the application

When reading the output of this command you can see OpenShift does a lot of work for you in maintaining this application. [Link to Come] takes a closer look at OpenShift’s built-in build system.

What’s important for now is that OpenShift created a build pod, that checked out the git repository and built a container image using the Dockerfile in the highscore subfolder. It automatically created a service for the application in the same step.

It will take some time to finish the build. When running oc get pods you will see a build pod running, and after the state of this pod turns to “Completed” the application pod will come up:

$ oc get pods

NAME                         READY   STATUS      RESTARTS   AGE

game                         1/1     Running     0          33h

game-c6fb95cc6-vj2qh         1/1     Running     0          20h

highscore-1-build            0/1     Completed   0          4m12s

highscore-56656f848c-k542p   1/1     Running     0          2m57s

There is no owning resource for all the resources created by oc new-app. To clean them up, you can run the following command, or delete the project. However, the following sections still use the deployments to expose them to the outside of the cluster:

$ oc delete all --selector app=highscore

service "highscore" deleted

deployment.apps "highscore" deleted

buildconfig.build.openshift.io "highscore" deleted

build.build.openshift.io "highscore-1" deleted

imagestream.image.openshift.io "alpine" deleted

imagestream.image.openshift.io "highscore" deleted

Accessing Deployed Services

After deploying all three services of the arcade platform application as described in the previous section, you should now have three services running in the arcade namespace:

$ oc get services

NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

game        ClusterIP   172.25.113.82    <none>        8080/TCP   35h

highscore   ClusterIP   172.25.32.245    <none>        8080/TCP   45s

platform    ClusterIP   172.25.170.245   <none>        8080/TCP   6s

All three services expose port 8080 of the pods. For game and platform you used your knowledge of the services to expose the right port. In case of the highscore service, OpenShift detected the exposed port from the container it built.

Accessing services from other pods

All three services are of type ClusterIP which allows other components of the cluster to access it. This is helpful for services that are only used by components communicating to each other within the cluster. To test this, you can deploy a pod to interact with the services:

$ oc run curl --image=curlimages/curl --command sleep 30h

This command will create a pod in the cluster which you can use to query one of the services using the curl command. The hostname of the service is the name you gave the service, so in this case you can query http://platform:8080 to reach the platform web service:

$ oc exec curl -- curl -s http://platform:8080

<html>

<head>

[...]

The preceding oc run command created a pod in the namespace arcade, where all the services of the arcade platform are deployed as well. That’s why you can access the service just by specifying the service name as hostname. If you create the curl pod in another namespace, for example the default namespace, this would not be possible as the following snippet shows:

$ oc -n default run curl --image=curlimages/curl --command sleep 30h

$ oc -n default exec curl -- curl -s platform:8080

command terminated with exit code 6

As you can see, the curl pod in the default namespace cannot resolve the hostname platform. However, we still can query a service in a different namespace by specifying the full internal domain name of the service:

$ oc -n default exec curl -- curl -s platform.arcade.svc.cluster.local:8080

<html>

<head>

[...]

The internal DNS name of OpenShift services is set to <service-name>.<namespace>.svc.cluster.local.

Distribution of Requests

In the previous section you scaled the game deployment up to three running pods. If you have not done this until now or scaled it back down, use the following command to scale it up:

$ oc scale deployment game --replicas=3

deployment.apps/game scaled

OpenShift will distribute the requests across all the endpoints of the service. To make this visible, the game deployment writes a header instance-ip to responses which you can query from your curl pod. Use the following command to list all endpoints of the game service:

$ oc get endpoints game

NAME   ENDPOINTS                                            AGE

game   10.116.0.62:8080,10.116.0.63:8080,10.116.0.64:8080   35h

The following command runs an endless loop with curl commands to send HTTP requests to the game service:

$ oc exec curl -- sh -c \

'while true; do curl -si game:8080 | grep instance-ip; sleep 1s; done'

instance-ip: 10.116.0.62

instance-ip: 10.116.0.63

instance-ip: 10.116.0.62

instance-ip: 10.116.0.64

instance-ip: 10.116.0.63

instance-ip: 10.116.0.64

instance-ip: 10.116.0.63

[...]

The -i flag tells curl to print response headers. Each output of the curl command is filtered with grep to only print the response header instance-ip. This results in a list, showing the distribution of requests.

As you can see in the output of the command, the requests are distributed randomly to all three deployed pods.

NOTE

The “instance-ip” header is a custom header added for the purpose of this chapter. If you want to replicate this with your own application you can add the following line to your NGINX configuration:

add_header instance-ip $server_addr always;

However, this is not something we recommend for production deployments, but just to visualize which endpoint receives the request.

Exposing Services

So far you’ve seen how to access services from within the cluster using the hostname or the cluster-internal DNS name of a given service. To access a service from your local machine for debugging you can use port-forwarding. In most cases, however, you want your users to reach the web services, or at least parts of them, via the network, for example using their web-browser. For that, you need to expose your services. OpenShift provides easy-to-use tooling to create a public DNS name as subdomain of the cluster domain that can be reached from outside of the cluster. To use it, you can create route resources for the services you want to expose to the network or internet.

Route by Auto-generated DNS Names

The first service to expose is the main entrance point of the arcade gaming platform, the platform service. To do so, just run oc expose again, this time specifying the service you want to expose to the outside world:

$ oc expose service platform

route.route.openshift.io/platform exposed

After running this command, a route resource has been created in the “arcade” namespace. Use the following command to see the route that has been generated:

$ oc get routes

NAME       HOST/PORT                           PATH   SERVICES   PORT

platform   platform-arcade.apps-crc.testing           platform   8080

Next, expose the game service. Run oc expose again and inspect the routes that OpenShift created in the namespace:

$ oc expose service game

route.route.openshift.io/game exposed

$ oc get routes

NAME       HOST/PORT                           PATH   SERVICES   PORT

game       game-arcade.apps-crc.testing               game       8080

platform   platform-arcade.apps-crc.testing           platform   8080

You can now see that the different routes for the services each got a unique DNS name assigned. Open a browser to verify the two web pages can be reached. Figure 3-3 shows how the arcade gaming platform page should look. If you’re running CRC those will be http://platform-arcade.apps-crc.testing and http://game-arcade.apps-crc.testing/s3e. Remember the game service only serves the /s3e path.

Figure 3-3. Example application: Arcade gaming platform front-end.

Route by Path

From the platform page, you will notice that neither the link to the highscore page, nor the button to the game are currently working. This is because the highscore service is not yet exposed, and because the game service is currently exposed with a different domain name. By default, OpenShift creates unique subdomains for each exposed service, composed from namespace and service name. You can see them in the output of the preceding oc get routes command. However, you can tell OpenShift to route the requests based on the path in a URL instead of generating unique names per service. If you look back at the architecture of the example application in Figure 3-1, routing by path using the same domain name is what you need to get the application running.

You can reuse the domain name generated for the platform service, platform-arcade.apps-crc.testing for the complete application, specifying paths which should be routed to the different services. Since the platform service is meant as main entrypoint to the application and expects requests at /, you don’t need to alter this route. Expose the highscore service at /highscore with the following command:

$ oc expose service highscore \

--hostname=platform-arcade.apps-crc.testing --path=/highscore

route.route.openshift.io/highscore exposed

To change the hostname of the game service, you can edit the generated route with the following command. It opens up an editor where you can adjust the generated hostname to platform-arcade.apps-crc.testing and set the path to /s3e:

$ oc edit route game

apiVersion: route.openshift.io/v1

kind: Route

metadata:

[...]

name: game

namespace: arcade

spec:

host: platform-arcade.apps-crc.testing

path: /s3e

port:

targetPort: 8080

to:

kind: Service

name: game

weight: 100

wildcardPolicy: None

status:

[...]

After saving your changes and exiting the editor you can get a list of the routes again. All three routes should now be assigned to the same hostname:

$ oc get routes

NAME        HOST/PORT                          PATH         SERVICES    PORT

game        platform-arcade.apps-crc.testing   /s3e         game        8080

highscore   platform-arcade.apps-crc.testing   /highscore   highscore   8080-tcp

platform    platform-arcade.apps-crc.testing                platform    8080

When you visit the main page http://platform-arcade.apps-crc.testing in your browser again, the game button should work. The link to the highscore page should work as well, which will look similar to Figure 3-4 after finishing some games.

Figure 3-4. Example application: Arcade gaming platform highscore.

External Load Balancers

Instead of using the OpenShift router to expose and access your services, you can use infrastructure-provided load balancers. For example, say you have deployed your OpenShift cluster to Google Cloud Platform (GCP), you can create GCP load balancers which will distribute the requests to your service across the OpenShift nodes.

On all nodes of the cluster there is will be a port open that listens for traffic directed to a specific service. The load balancer will distribute the requests to the nodes at this specific port, which is called . Requests to the load balancer IP will be forwarded to the right pod. This concept is called NodePort in OpenShift.

OpenShift will take care for configuring the load balancer in the infrastructure. All you need to do is specify the type LoadBalancer when exposing the deployment.

NOTE

To follow the examples of this section, you will need an OpenShift cluster that is deployed on infrastructure that provides load balancers, like GCP or Amazon Web Services (AWS). Although you can execute all the commands and create services of type LoadBalancer on CRC, it won’t be different from a service of type NodePort.

Use the following command to create a new service with name “game-lb” to expose the game deployment with a service of type LoadBalancer. Afterwards inspect the created service with oc describe:

$ oc expose deployment game --type=LoadBalancer --port=8080 --name=game-lb

service/game-lb exposed

$ oc describe service game-lb

Name:                     game-lb

Namespace:                arcade

Labels:                   app=game

Annotations:              <none>

Selector:                 app=game

Type:                     LoadBalancer

IP:                       172.30.248.198

LoadBalancer Ingress:     34.71.94.126

Port:                     <unset>  8080/TCP

TargetPort:               8080/TCP

NodePort:                 <unset>  30547/TCP

Endpoints:                0.116.0.62:8080,10.116.0.63:8080,10.116.0.64:8080

Session Affinity:         None

External Traffic Policy:  Cluster

Events:

Type    Reason                Age   From                Message

----    ------                ----  ----                -------

Normal  EnsuringLoadBalancer  46s   service-controller  Ensuring load balancer

Normal  EnsuredLoadBalancer   8s    service-controller  Ensured load balancer

You can compare this service to the original game service.

$ oc describe service game

Name:              game

Namespace:         arcade2

Labels:            app=game

Annotations:       <none>

Selector:          app=game

Type:              ClusterIP

IP:                172.25.113.82

Port:              <unset>  8080/TCP

TargetPort:        8080/TCP

Endpoints:         10.116.0.62:8080,10.116.0.63:8080,10.116.0.64:8080

Session Affinity:  None

Events:            <none>

You can see that OpenShift has created a service of type LoadBalancer as opposed to ClusterIP. Additionally, a NodePort has been allocated for this service which can now be used to access the service. You should be able to query any node at this port and reach the game-lb service, which will distribute the requests across the same endpoints as the game service.

Get an IP address of any of your nodes from oc get nodes -o wide or oc describe nodes. If your nodes are reachable externally and the ports are exposed, you can use the external IP to query the service right from your computer. In most cases the nodes are not exposed to the internet directly or the ports are behind a firewall so they cannot be reached from the internet. However, you can still deploy a pod inside the cluster and query the internal IP to observe the service behavior.

$ oc describe node | grep InternalIP

InternalIP:  192.168.126.11

[...]

$ oc run curl --image=curlimages/curl --command sleep 30h

$ oc exec curl -- sh -c \

'while true; do curl -si 192.168.126.11:32167 | \

grep instance-ip; sleep 1s; done'

instance-ip: 10.116.0.63

instance-ip: 10.116.0.62

instance-ip: 10.116.0.62

instance-ip: 10.116.0.63

instance-ip: 10.116.0.62

instance-ip: 10.116.0.64

[...]

In the preceding command one node is queried using the allocated port of the service. From the headers you can see that the requests are distributed to the same endpoints as the requests you directed to the game service before.

In addition to the NodePort, OpenShift exposed the service using a load balancer in the infrastructure you deployed your cluster on.

For this example, an OpenShift cluster on GCP has been created. In the GCP console, you can find the load balancer that is associated to this service by comparing the public IP associated with the load balancer. Figure 3-5 shows an example configuration of a load balancer, created by OpenShift.

Figure 3-5. Load balancer details in the GCP console.

Just as you queried the NodePort before, you can use curl from your local machine to access the service behind the load balancer using the external IP:

$ while true; do curl -si 34.71.94.126:8080/s3e/ | \

grep instance-ip; sleep 1s; done

instance-ip: 10.116.0.63

instance-ip: 10.116.0.62

instance-ip: 10.116.0.63

instance-ip: 10.116.0.62

instance-ip: 10.116.0.62

instance-ip: 10.116.0.64

Be aware that using infrastructure-provided load balancers for your services comes with additional costs for your project. We recommend using the OpenShift router as a default (also for its ease of use). Only if it doesn’t provide the performance you need, or you need an external load balancer for a different reason, should you consider creating one for your service.

Securing Services with TLS

When following the examples in this chapter so far, you probably have realized or even been warned by your web browser that the web services you’ve been accessing were insecure. Most browsers indicate that you’re looking at a website that uses HTTP over HTTPS in some way, be it a small icon or a full-screen warning that you need to click your way through before you’re able to visit the web page. This is a good reminder for everyone that all web pages should use secure connections. It may not be important when deploying a web service that on your local device, but in most cases you will have real users on another device, accessing your web service via the internet. In those cases, you should provide an HTTPS connection to your users so the data they enter is secured on its way to your web server. The warnings that web browsers show also tell the user in one glance whether you care for their privacy or not.

Luckily, providing officially signed certificates is very easy using Let’s Encrypt. Let’s Encrypt is a service run by the Internet Security Research Group (ISRG). It aims to provide trusted, free TLS certificates to anyone who runs a website on any public domain. Let’s Encrypt certificates are valid for three months, which is much shorter than certificates provided by many paid providers, so the renewal process needs to be automated or you’ll find yourself renewing certificates every three months. Luckily, Let’s Encrypt itself provides good options to automate the certificate renewal, and there are even web servers like Caddy hiding the certificate renewal process from administrators (or SREs).

For services deployed using the OpenShift router, you can provide TLS certificates in the route configuration. In the following examples, you will first deploy fake certificates and later request trusted certificates with Let’s Encrypt. You can use a very similar procedure to provide custom certificates from trusted issuers with your route.

Specifying TLS Certificates

Before looking into automating the renewal of certificates issued by Let’s Encrypt, take a look at how TLS routes are configured in OpenShift: For that, first, create a self-signed certificate for the DNS name you use. This example is created on a CRC cluster using the base domain name apps-crc.testing, so for domain names of routes in this cluster, it is not possible to generate a publicly trusted certificate anyhow. You can run this example with any domain name, but please be advised that self-signed certificates should only be used to help understanding the process and must not replace trusted certificates for your web services.

All the routes you deployed in this chapter use the same domain name platform-arcade.apps-crc.testing, so you only need to create and provide a single certificate:

$ oc get routes

NAME        HOST/PORT                          PATH         SERVICES    PORT

game        platform-arcade.apps-crc.testing   /s3e         game        8080

highscore   platform-arcade.apps-crc.testing   /highscore   highscore   8080-tcp

platform    platform-arcade.apps-crc.testing                platform    8080

To create a self-signed certificate, you can use the openssl command-line tool. Run the following command to generate a certificate:

$ openssl req -x509 -newkey rsa:4096 -nodes \

-keyout tls.key -out tls.crt -days 90

This will create a certificate and a key file that you can use to encrypt the traffic to your services. For this purpose, recreate the game route with the following two commands:

$ oc delete route game

route.route.openshift.io `game` deleted

$ oc create route edge \

--cert=tls.crt --key=tls.key \

--hostname=platform-arcade.apps-crc.testing \

--port=8080 --path=/s3e --service=game

route.route.openshift.io/game created

Specify the route type edge, which means the router will handle the TLS handshake.

Enter the two files you generated above.

You just specified the certificate and key to use for HTTPS traffic to the game service. You can see them now in the route specification:

$ oc get route game -o yaml

apiVersion: route.openshift.io/v1

kind: Route

metadata:

labels:

app: game

name: game

namespace: arcade

spec:

host: platform-arcade.apps-crc.testing

path: /s3e

tls:

certificate: |

-----BEGIN CERTIFICATE-----

[...]

-----END CERTIFICATE-----

key: |

-----BEGIN PRIVATE KEY-----

[...]

-----END PRIVATE KEY-----

termination: edge

to:

kind: Service

name: game

weight: 100

wildcardPolicy: None

status:

[...]

Accessing the service should now be possible using the HTTPS scheme. Open a web browser to check if you can access your service. In the preceding example, it is accessible at https://platform-arcade.apps-crc.testing/s3e. Though it now uses HTTPS, the web browser still warns you that the web page is using a self-signed certificate.

Redirecting Traffic to TLS Route

Accessing the service with HTTP instead of the HTTPS scheme will not be possible anymore in the current configuration. This allows you to deploy a second route using HTTP instead of HTTPS. You can also redirect the user automatically to the secure route, which should be the preferred option for most cases. To do so, set the insecureEdgeTerminationPolicy to Redirect. This setting has three different options:

- None: The default configuration. Only connection to HTTPS service is possible for this route.

- Allow: Allow both, traffic to HTTP and HTTPS.

- Redirect: When accessing the HTTP scheme, the web server redirects the client to HTTPS.

To update this setting, edit the route as shown:

$ oc edit route game

apiVersion: route.openshift.io/v1

kind: Route

metadata:

[...]

name: game

namespace: arcade

spec:

[...]

tls:

[...]

termination: edge

insecureEdgeTerminationPolicy: Redirect

to:

[...]

status:

[...]

The new configuration will also show up in the TERMINATION column in oc get route:

$ oc get route game

NAME   HOST/PORT                        [...] PORT    TERMINATION     WILDCARD

game   platform-arcade.apps-crc.testing [...] 8080    edge/Redirect   None

You can test and observe the redirect workflow now using either the network section of the debug tools of your web browser, or using curl, as shown in the following listing:

$ curl -ikL http://platform-arcade.apps-crc.testing/s3e/

HTTP/1.1 302 Found

Cache-Control: no-cache

Content-length: 0

Location: https://platform-arcade.apps-crc.testing/s3e/

HTTP/1.1 200 OK

Server: nginx/1.19.7

Date: Mon, 05 Apr 2021 09:49:28 GMT

Content-Type: text/html

Content-Length: 7085

[...]

Cache-control: private

<!DOCTYPE html>

<html xmlns='http://www.w3.org/1999/xhtml' lang='' xml:lang=''>

<head>

[...]

Use -k to tell curl to trust the self-signed certificate, -i to show headers and -L to follow redirects.

The first response is a 302, asking the client to execute the same request with a different URL.

This URL is returned by the server with the HTTP scheme.

The next request is sent to the URL returned by the server (3) and returns status code 200 (ok).

The web page containing the game is returned.

Redirecting your users to the secure HTTPS scheme is a good practice especially for services that are directly user facing as in the example of the gaming platform. With redirection, the scheme used by the web service is nothing the user needs to care or think about. For example need don’t need to worry about typing http:// or https://.

Now that you enabled TLS encryption for one of the routes, the other two routes still use unencrypted HTTP. Since they all share the same hostname, you can reuse the same certificate and do not need to create or request new ones.

Use the following commands to extract the TLS property from the game route and apply it to the platform and highscore route, or recreate the routes as you did before with the game route:

$ oc patch route platform \

--patch="{\"spec\":{\"tls\":$(oc get route game -o json | jq .spec.tls)}}"

route.route.openshift.io/platform patched

$ oc patch route highscore \

--patch="{\"spec\":{\"tls\":$(oc get route game -o json | jq .spec.tls)}}"

route.route.openshift.io/highscore patched

Now you should see all routes are showing “edge/redirect” in the “TERMINATION” column and a check with the browser should show you’re redirected to the right scheme. Since you’re still using the self-signed certificate, a warning will be present.

$ oc get routes

NAME        HOST/PORT                        [...] PORT     TERMINATION

game        platform-arcade.apps-crc.testing [...] 8080     edge/Redirect

highscore   platform-arcade.apps-crc.testing [...] 8080-tcp edge/Redirect

platform    platform-arcade.apps-crc.testing [...] 8080     edge/Redirect

Let’s Encrypt Trusted Certificates

Now that you know how to configure a certificate for a given route with a self-signed certificate, you can use the same process to use an officially signed and trusted certificate in your routes. However, updating certificates by hand when they are getting close to expiry - or even worse, when somebody first notices they’re expired - is a tedious process, even if the certificates are valid for more than 90 days, as the self-signed certificates are in the previous example. Imagine your cluster serves more than a couple of routes, or you maintain more than one cluster and need to care for all those certificates. A typical SRE task would be to automate this process so you don’t ever again have to renew the certificates manually. If you decide to use Let’s Encrypt, a lot of the automation process is already provided either by the Let’s Encrypt service itself (automatic requesting of certificates) or by the community, as this service is quite popular.

For OpenShift, you can use the open source openshift-acme operator to automatically request certificates for all routes you want to configure for TLS encrypted traffic. To use it, you should first remove the self-signed certificates you set above from the routes. This makes sure the operator recognizes there are no certs set, so it will request new ones:

$ oc patch route game --type=json \

-p='[{"op": "remove", "path": "/spec/tls"}]'

$ oc patch route platform --type=json \

-p='[{"op": "remove", "path": "/spec/tls"}]'

$ oc patch route highscore --type=json \

-p='[{"op": "remove", "path": "/spec/tls"}]'

NOTE

Using Let’s Encrypt with the openshift-acme operator requires a cluster with a public-accessible domain name to perform the certificate renewal. Using CRC for it may be possible but difficult so to practice this part you need to deploy a cluster that is publicly accessible using one of the methods described in Chapter 2.

The openshift-acme operator can be deployed in different modes, either to operate on the whole cluster, on a single namespace, or on a list of namespaces. For this example deployment, enable the operator for all routes you or another user creates, so anybody can easily create secure routes. To deploy the operator into its own namespace, you can first create a separate project and afterwards run the commands provided by the documentation of the operator:

$ oc new-project openshift-acme-operator

$ oc apply -f https://raw.githubusercontent.com/tnozicka/\

openshift-acme/master/deploy/cluster-wide/\

{clusterrole,serviceaccount,issuer-letsencrypt-live,deployment}.yaml

$ oc create clusterrolebinding openshift-acme --clusterrole=openshift-acme \

--serviceaccount="$( oc project -q ):openshift-acme" --dry-run -o yaml | \

oc apply -f -

The operator will now scan all routes deployed to the cluster for the annotation kubernetes.io/tls-acme. If set to true it will start to manage the certificates for this routes using Let’s Encrypt. It will request new certificates and set the tls property on the route, and when the certificate reaches the end of its validity the operator will renew the certificate without anybody noticing. Enable the operator for the first route by setting the annotation:

$ oc edit route platform

apiVersion: route.openshift.io/v1

kind: Route

metadata:

name: platform

namespace: arcade

annotations:

kubernetes.io/tls-acme: "true"

[...]

It takes the operator a certain amount of time to detect and process the changes, perform the renewal protocol, and update the route. Afterwards, the tls property will be back at the route object, this time with a signed certificate, issued by Let’s Encrypt. The default behavior of openshift-acme is to redirect users from the HTTP to the HTTPS scheme in the same way you configured it before.

Listing the routes again after the renewal process is done shows that the platform route now uses edge termination and will redirect users:

NAME        HOST/PORT             [...] PORT       TERMINATION     WILDCARD

game        platform-arcade.apps. [...] 8080                       None

highscore   platform-arcade.apps. [...] 8080-tcp                   None

platform    platform-arcade.apps. [...] 8080       edge/Redirect   None

Visiting the route in the browser will show you’re using a trusted certificate.

Figure 3-6. Using a trusted certificate for the example application.

Now you need to update the other two services to add the same annotation, so the operator updates them as well with a TLS certificate:

$ oc get routes

NAME        HOST/PORT                    [...] PORT     TERMINATION     WILDCARD

game        platform-arcade.apps.arcade. [...] 8080     edge/Redirect   None

highscore   platform-arcade.apps.arcade. [...] 8080-tcp edge/Redirect   None

platform    platform-arcade.apps.arcade. [...] 8080     edge/Redirect   None

NOTE

At the time of writing, the OpenShift acme-operator does not support multiple routes with the same hostname. To work around the issue, copy the TLS certificate manually as you did it with the self-signed certificate. In production you would want to take an option that works automatically, so for now you would need to adjust the application to use different DNS names per route.

Encrypted Communication to the Service

The previous section talked about the encryption of the communication between a client outside a cluster and the cluster. TLS has been terminated on the router (edge termination) and the requests were forwarded to the web services in an unencrypted fashion. In some cases this is fine, but in many cases you also want the communication between the router and the pods of the OpenShift cluster to be encrypted - that is, the router should use HTTPS to forward the requests. To achieve this, you have two different options. You can either not do the TLS termination on the router and rather forward the HTTPS requests to the target service (passthrough) or terminate the request and send a new HTTPS request to the target service (reencrypt). Compare the different termination modes in Figure 3-7.

Figure 3-7. Comparison of TLS termination modes

Passthrough

A service that uses the passthrough TLS termination mode must handle the TLS termination on its own. The router will not decrypt the requests and passes them right to the receiving service. That means the web service you’re using must support communicating via HTTPS and you must provide the certificate to the service. If you’re using Let’s Encrypt with the openshift-acme operator, you can leave the requests and renewal of certificates to openshift-acme operator and have it export the certificates as secrets, which you can then mount into your web service’s pods.

For example, for a passthrough route you can use the platform service, which supports TLS communication when a certificate is mounted at a predefined path.

To configure openshift-acme to export the TLS certificate as secret, add the annotation “acme.openshift.io/secret-name” to the platform route:

$ oc edit route platform

apiVersion: route.openshift.io/v1

kind: Route

metadata:

annotations:

acme.openshift.io/secret-name: "public-cert"

[...]

The value of the annotation is the name of the secret where the TLS certificate will be stored. The secret will be created in the same namespace as the route. The operator takes care of updating the secret when a new certificate is requested. Now you can mount that secret at the path /etc/nginx/certs/tls.crt and /etc/nginx/certs/tls.key in the platform pod, where the platform NGINX service expects the certificates to exist.

$ oc edit deployment platform

apiVersion: apps/v1

kind: Deployment

[...]

spec:

template:

spec:

containers:

- image: quay.io/mdewald/s3e-platform

[...]

volumeMounts:

- mountPath: /etc/nginx/certs

name: tls-certs

readOnly: true

volumes:

- name: tls-certs

secret:

defaultMode: 420

secretName: public-cert

The volumeMounts specify where the certificates should be mounted. Mount the certificates into the folder /etc/nginx/certs which is where the container expects them.

The secretName references the secret you also specified in the annotation of the route.

NOTE

How does the service know it should use those certificates? The platform service is based on NGINX and uses the following configuration to listen on HTTP (port 8080) and HTTPS (port 4443) at the same time. When no certificate is mounted, the service uses a self-signed default certificate which is included in the container image. Without those files, the service would not start:

[...]

ssl_certificate     certs/tls.crt;

ssl_certificate_key certs/tls.key;

server {

listen 0.0.0.0:8080;

listen 0.0.0.0:4443 ssl;

[...]

}

The next step is to expose the port used for HTTPS communication in the platform pod instead of the HTTP port. Change port and target port in the platform service from 8080 to 4443:

$ oc edit service platform

apiVersion: v1

kind: Service

metadata:

name: platform

namespace: arcade

spec:

ports:

- port: 4443

protocol: TCP

targetPort: 4443

[...]

Finally, update the target port in the platform route:

$ oc edit route platform

apiVersion: route.openshift.io/v1

kind: Route

metadata:

name: platform

namespace: arcade

spec:

port:

targetPort: 4443

[...]

NOTE

It’s sufficient to change the “targetPort” setting on the service only so you don’t have to update the route as well to switch the port. However, consistency in port numbers helps understand and debug a system, so you should also update the port used for the communication between router and service.

To verify that the route now uses TLS to communicate with the pod, the platform service modifies the log output to print the protocol used. You can follow the logs of the service and send requests to the route from a browser or a second terminal and should see output similar to the following:

$ oc logs -lapp=platform -f

[...]

10.116.0.1 - [..] TLSv1.2 "GET / HTTP/1.1" 200 587 "-" "curl/7.68.0" "-"

The string TLSv1.2 tells you that TLS has been terminated in the pod. Without the settings you changed in this section the log line would look similar to the following:

10.116.0.1 - [...] - "GET / HTTP/1.1" 200 587 "-" "curl/7.68.0" "192.168.130.1"

NOTE

NGINX by default doesn’t log if TLS is used or not. The variable variable $ssl_protocol is added to the default log line in the NGINX configuration specifically to visualize the differences between the termination settings.

Reencrypt

For the edge as well as passthrough termination modes there is only one certificate in use for the encryption of the request from the client to the place where it’s terminated. The termination happens either in the router, or the pod running the service itself. For the reencrypt termination mode a second certificate comes into play. The certificate presented to the clients will remain the same, but now the router will communicate with the service’s pod via HTTPS, using a different TLS certificate.

To communicate with the target web server, the router needs to trust the certificate presented by the web server. That means you need to sign the certificate you want to use with a CA and tell the router to trust it. For the self-signed certificate you created earlier, that means you first need to create a CA and sign the certificate with it.

To create a CA certificate with OpenSSL, you can run the following two commands1:

$ openssl genrsa -out ca.key 4096

$ openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt

Now you need to create a signed certificate using this CA with the following commands:

$ openssl req -new -key tls.key -out tls.csr

$ openssl x509 -req -in tls.csr -CA ca.crt \

-CAkey ca.key -CAcreateserial -out tls.crt -days 90 -sha256

Next, tell the router that it should trust the newly created CA when communicating with the web service. This example assumes the platform route uses a trusted certificate already, for example issued by Let’s Encrypt. Edit the route and add the contents of the CA. Also, change the termination to reencrypt:

$ oc edit route platform

[...]

destinationCACertificate: |

-----BEGIN CERTIFICATE-----

... paste contents of ca.crt here ...

-----END CERTIFICATE-----

termination: reencrypt

Finally, you need to create a secret with the certificate and mount it into the platform pod as described in the previous section:

$ oc create secret tls self-signed --cert=./tls.crt --key=./tls.key

secret/self-signed created

$ oc edit deployment platform

apiVersion: apps/v1

kind: Deployment

[...]

spec:

template:

spec:

containers:

- image: quay.io/mdewald/s3e-platform

[...]

volumeMounts:

- mountPath: /etc/nginx/certs

name: tls-certs

readOnly: true

volumes:

- name: tls-certs

secret:

defaultMode: 420

secretName: self-signed

Querying the service while observing the logs should now again show you the requests are using TLS:

$ oc logs -lapp=platform -f

[...]

10.116.0.1 - [...] TLSv1.2 "GET / HTTP/1.1" 200 587 "-" "curl/7.68.0" "-"

Troubleshooting routes that use reencrypt termination is usually time intensive, as a small failure in the certificate-requesting process will result in the inability to access the web service. The router by default does not provide a lot of debugging information when something is wrong in the TLS communication.

If you have a single public certificate per route, we recommend using passthrough if you want to have the traffic encrypted until they reach the destination service. That makes it easy to handle the renew process as renewing public certificates can be automated easily, for example using Let’s Encrypt. When using custom certificates for the cluster-internal communication, be aware that the renewal of those certificates is a manual action you need to run on your own. It usually contains tasks like reconfiguring many routes and recreating a lot of services to get certain certificates rotated. Best prepare or create automation for those tasks right from the beginning rather than think about that problem when the certificates are close to expiry.

Summary

In this chapter you successfully deployed different services onto an OpenShift cluster of your choice, using different deployment methods. You are now able to access any web service on a cluster whether it is exposed as a service, route, or not all. Even if you won’t use port-forwarding too often to access your services, some services provide a maintenance UI, logging, or monitoring interfaces that you can quickly access when debugging an issue.

You now are able to expose a web service using the built-in OpenShift router and even encrypt traffic to your web services with TLS certificates that renew themselves automatically. You are even able to decide and implement different places where TLS is terminated: On the edge, or in the destination service.

You tied different services together, and figured out how to reach a service on the cluster using the internal domain name. To access services from outside of the cluster, you now know how to route requests based on an automatically generated hostname or a predefined path using the same domain name.

A lot of SRE work is about eliminating manual work so you can scale as an organization and manage more services with a relatively small team. While following along this chapter you still did many things manually to get to understand the concepts. When you start working on the automation of manual tasks, as you will do in [Link to Come], you will be quickly confronted with the question which user account will execute a certain task. [Link to Come] will talk about service accounts, a concept that allows non-user accounts to exist in a cluster to run certain tasks and how both, users and service accounts get the permissions they need for their job using fine-grained Role-based Access Control (RBAC).

1   Commands are based on this quick guide: https://gist.github.com/fntlnz/cf14feb5a46b2eda428e000157447309

About the Authors

Rick Rackow is a seasoned professional who has worked on cloud and container adoption throughout his career. As Site Reliability Engineer on Red Hat’s OpenShift Dedicated team, Rick manages and maintains countless OpenShift clusters at scale and ensures their reliability every day by developing and following the best practices the reader will learn in this book.

Manuel Dewald is working as a site reliability engineer at Red Hat since 2019. In his role, he’s keeping the lights on of OpenShift clusters on top of the major public clouds.

Previously he was working as a software engineer, participating in DevOps practices in different development teams. With all the challenges of this setup, he decided to change into a role that is focussed on operating software, while keeping the focus on software engineering to solve operations challenges.

He’s passionate about open source software and good software engineering practices. From time to time you can find him complaining about missing tests even on the smallest projects, which are only solving a little pain of the operations team.

Manuel is speaking at conferences and meetups about utilizing the OpenShift build infrastructure, best practices for implementing Kubernetes operators, and agile practices in SRE teams. He is writing blog posts about Kubernetes operators, agile practices, and Raspberry Pi projects on https://blog.redhat.com, and https://opensource.com.

Table of Contents

1. Introduction

Traditional Operations Teams

How Site Reliability Engineering Helps

OpenShift as a Tool for Site Reliability Engineers

Individual Challenges for Site Reliability Engineering Teams

2. Installing OpenShift

OKD, OCP and Other Distributions

OKD

OCP

OSD, ROSA, ARO

Summary

Local Clusters with CodeReady Containers

Real World Scenario

Getting started with CodeReady Containers

Planning Cluster Size

Real World Scenario

Instance Sizing Recommendations

Basic OpenShift Installations

Real World Scenario

Installer-provisioned Infrastructure

Self-provisioned Infrastructure

3. Running Workloads on OpenShift

Deploying Code

Deploying Existing Container Images

Creating Deployments

Deploying Applications from git Repositories

Accessing Deployed Services

Accessing services from other pods

Distribution of Requests

Exposing Services

Route by Auto-generated DNS Names

Route by Path

External Load Balancers

Securing Services with TLS

Specifying TLS Certificates

Redirecting Traffic to TLS Route

Let’s Encrypt Trusted Certificates

Encrypted Communication to the Service

Summary

About the Authors